Organization finder in spaCy I am trying to find the parties between which the agreement is executed from the statement below.

AGREEMENT, dated as of January 10, 2000, is entered into by and between ABC-EFG GROUP Inc. having an address at 418 Mona Drive, Prominade 34, Florida 34673, United States of America and Rob Cummins, an individual residing in the state of Florida, and having an address at 13 test Dr, Arosa, FL 43566

To get this, I tried the following code:
for chunk in doc.noun_chunks: # after loading "nlp = spacy.load("en_core_web_sm")" and doc = nlp("string")
   print(chunk.text, chunk.root.text, chunk.root.dep_,chunk.root.head.text)

which gives me:
AGREEMENT AGREEMENT nsubjpass entered
January January pobj of
ABC-EFG GROUP Inc. Inc. pobj between
an address address dobj having
418 Mona Drive Drive pobj at
United States States conj Drive
America America pobj of
Rob Cummins Cummins conj America
an individual individual appos States
the state state pobj in
Florida Florida pobj of
an address address dobj having
13 test test pobj at
Dr Dr ROOT Dr
FL FL appos Arosa

I don't understand how I am supposed to find the two parties from this output.
I am not specific to spaCy. A suggestion of any other model or method also will work.
Please note that I have already tried NER but the results are very poor.

Using dependency parsing alone will not give you what you need. You may be able to get your answer by interpreting the dependency tree. For instance, in this case ABC-EFG GROUP Inc. is a pobj of between, which could infer that it is party 1. In this particular case, the dependency parsing isn't fully correct and party 2 (Rob Cummins) is difficult to find.

I would recommend you try the following things:

It seems that the parties are either organization or people, so you could narrow down your candidates to proper nouns and you could use NER to see whether any ORG or PER are detected. That will not tell you exactly whether they are the parties, but they could give you useful hints.

In this case, it feels like a plain regular expression could work. Whatever proper noun occurs after between seems to be party 1, and whatever follows and seems to be party 2. Of course, a regular expression would only work for cases written with the same pattern, but it is always wise to consider. Perhaps, you are already a fair amount of your scenarios with a solution that will always work. You could also use the presence of words like "between" and "and" as a feature.

If you don't have a lot of training examples, you could take a look at libraries like mimesis and faker. They allow you to generate company names, addresses, people names, etc.


Image classification problem using convolutional neural networks I am trying to solve this problem by using a convolutional NN to classify an image data set to check the type of disease it is. I have reached task 1b and trying to implement the training loop. However, I am getting an error and can't understand how to implement the training loop.
I am sharing the google drive link.
The code in task 1b where I am getting an error:
import datasets

ds = datasets.LesionDataset('/content/data/img',
                            '/content/data/img/train.csv')
input, label = ds[0]
print(input)
print(label)

error:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-27-dcb680f1f34a> in <module>()
      4 ds = datasets.LesionDataset('/content/data/img',
      5                             '/content/data/img/train.csv')
----> 6 input, label = ds[0]
      7 print(input)
      8 print(label)

/content/drive/MyDrive/DL Assignment/datasets.py in __getitem__(self, idx)
     34   def __getitem__(self, idx):
     35
---> 36         image = self.img_dir[idx]
     37         label = self.labels_fname[idx]
     38

'datasets' is a .py file currently in the same folder in the link shared which has the following code: You can open the link to understand it better.
import collections
import csv
from pathlib import Path

import numpy as np
import torch
import torchvision.transforms as transforms
from PIL import Image
import pandas as pd

# TODO Task 1b - Implement LesionDataset
from torch.utils.data import Dataset
class LesionDataset(Dataset):
#The __init__ function should have the following prototype
  def __init__(self, img_dir, labels_fname):
    # is the directory path with all the image files
    # is the csv file with image ids and their corresponding labels
      self.img_dir = Path(img_dir)

      #image = Image.open('ISIC_0024306.jpg')
      #
      self.labels_fname = pd.read_csv(labels_fname)

      image = self.labels_fname['image']

#list of images store in list
  def __len__(self):

        return len(self.labels_fname)

  def __getitem__(self, idx):

        image = self.img_dir[idx]
        label = self.labels_fname[idx]

        return image, label

# TODO Task 1e - Add augment flag to LesionDataset, so the __init__ function
#                now look like this:
#                   def __init__(self, img_dir, labels_fname, augment=False):

# TODO Task 2b - Implement TextDataset
#               The __init__ function should have the following prototype
#                   def __init__(self, fname, sentence_len)
#                   - fname is the filename of the cvs file that contains each
#                     news headlines text and its corresponding label.
#                   - sentence_len the maximum sentence length you want the
#                     tokenized to return. Any sentence longer than that should
#                     be truncated by the tokenizer. Any shorter sentence should
#                     padded by the tokenizer.
#                We will be using the pretrained 'distilbert-base-uncased' transform,
#                so please use the appropriate tokenizer for it. NOTE: You will need
#                to include the relevant import statement.

I am stuck at this part. Can anyone suggest how to implement it?

In the __getitem__ method you should return both the image and the label, in your current example you are trying to get the image by indexing the img_dir variable or your class, which won't work since this is of type Path. The general steps for the __getitem__ method is that you should (1) get the path of image, (2) load the image and (3) convert the image to a tensor for pytorch to use. This would something like this:
from torch.utils.data import Dataset
import torchvision.transforms as transforms
from pathlib import Path
import pandas as pd
from PIL import Image

class LesionDataset(Dataset):
    def __init__(self, img_dir, labels_fname):
        self.img_dir = Path(img_dir)
        self.img_paths = self.img_dir.glob("*")
        self.labels_fname = pd.read_csv(labels_fname)

    def __len__(self):
        return len(self.labels_fname)

    def __getitem__(self, idx):
        img_path = self.img_paths[idx]
        image = Image.open(img_path)
        image = transforms.ToTensor()(image)
        label = self.labels_fname[idx]
        return image, label


Restricting the output of a model didn't improve the loss value of the model evaluation There is a deep model for prediction.

The outputs are some numbers between 0 and 80. (In the dataset the outputs are 0-80)

The model Loss value is 70 and I would like to reduce it.

I printed the outputs after evaluating the model by test values and some of the predicted values are more than 80 or less than 0.

I decided to set up the final layer to predict just in 0-80 in the training step, therefore I set a lambda layer after final Dense layer to clip output values.

The codes:

def relu_advanced(x):
return K.relu(x, max_value=80)

def createModel4():
model = models.Sequential()
model.add(Conv2D(256,(3, 3),
                 activation='relu',
                 input_shape=(320,20,1), padding='same'))
model.add(MaxPooling2D((2, 2)))

model.add(Conv2D(128, (3, 3), activation='relu', padding='same'))
model.add(MaxPooling2D((2, 2)))

model.add(Conv2D(64, (3, 3), activation='relu', padding='same'))
model.add(MaxPooling2D((2,2)))

model.add(Flatten())
#model.add(Dense(5*320, activation= 'relu'))
model.add(Dense(5*320))
model.add(Lambda(relu_advanced))

model.summary()
return model

I tested the model with and without the relu_advanced and unfortunately, the Loss value is increased with advanced_relu!

While there is no value much than 80 or less than zero, I don't know what may happen that the Loss is increased?

Thank you

I suggest you normalise your labels so that it is scaled between 0 and 1, rather than 0 and 80. Once you have a trained model then multiply your output at the end. The network should find it easier to learn values between 0 and 1 (Andrew Ng's coursera course has a good lecture on this).

Go back to using the standard ReLU: The problem with yours is that it cannot learn if it tries to output a value greater than 80, as there is no gradient with which to update the parameters.

I would also consider putting in a much smaller Dense layer at the end (eg 10 nodes).


How can I extract the residual array from the Scikit Learn PCA routine? The Nipals PCA algorithm calculates the scores and loadings of a data array iteratively.  If only (say) 3 scores and loadings are calculated from a data array with more than 3 variables, there is a residual matrix created (called E).  How can I extract this matrix from the SciKit Learn PCA algorithm so that I can create contribution charts?

I'm not super familiar with PLS/NIPALS, but I think you need to construct it yourself. Multiply the scores by the loadings and subtract the resulting matrix from the data to get the residual matrix. See here for a more detailed discussion: https://learnche.org/pid/latent-variable-modelling/projection-to-latent-structures/how-the-pls-model-is-calculated


Predicting the missing word using fasttext pretrained word embedding models (CBOW vs skipgram) I am trying to implement a simple word prediction algorithm for filling a gap in a sentence by choosing from several options:

Driving a ---- is not fun in London streets.

Apple
Car
Book
King

With the right model in place:

Question 1. What operation/function has to be used to find the best fitting choice? The similarity functions in the library are defined between one word to another word and not one word to a list of words (e.g. most_similar_to_given function). I don't find this primitive function anywhere while it is the main operation promised by CBOW (see below)! I see some suggestions here that are not intuitive! What am I missing here?

I decided to follow the head first approach and start with fastText which provides the library and pre-trained datasets but soon got stuck in the documentation:

  fastText provides two models for computing word representations: skipgram and cbow ('continuous-bag-of-words'). The skipgram model
  learns to predict a target word thanks to a nearby word. On the other
  hand, the cbow model predicts the target word according to its
  context. The context is represented as a bag of the words contained in
  a fixed size window around the target word.

The explanation is not clear for me since the "nearby word" has a similar meaning as "context". I googled a bit and ended up with this alternative definition:

  In the CBOW model, the distributed representations of context (or surrounding words) are combined to predict the word in the middle.
  While in the Skip-gram model, the distributed representation of the
  input word is used to predict the context.

With this definition, CBOW is the right model that I have to use. Now I have the following questions:

Question 2. Which model is used to train fastText pre-trained word vectors? CBOW or skipgram?

Question 3. Knowing that the right model that has to be used is CBOW, can I use the pre-trained vectors trained by skipgram model for my word prediction use case?

Question 1:
To do so, I would use the Gensim wrapper of FastText because Gensim has a predict_output_word which does exactly what you want. Given a list of context words, it provides the most fitting words.
Question 2:
It is up to the user. FastText isn't inherently CBOW or Skipgram. See this
Question 3:
Yes, even though CBOW and SkipGram are different training procedures, they share a common goal. Both will generate word embeddings where (hopefully) words that are semantically close also have embeddings that are close. The main difference between SkipGram and CBOW is the inherent heuristic used for semantic closeness.


Signs there are too many class labels I have a classification problem, where I initially started with 100+ class labels. My intuition told me this is too many labels for a model to predict with good accuracy. I thought grouping the labels together will help the situation and it did, there is now only 19 class labels. Is 19 still too many class labels? What are the typical signs that there are too many labels for a model to predict and how can this be addressed, if there is too many class labels?

A class taxonomy should:

Serve the business needs
Be learnable

There is a potential tradeoff here. The more exact and specific the taxonomy, the more you'll know about the entities and you'll be able to server better the business needs. However, for a large taxonomy the classifier will have to model more complex rules, will have less samples for each case and the boundary samples will have more influence.

Check the number of samples per label in your dataset. A class label with not enough samples will be hard to learn. If you have many class label, the ratio of some of them will also be low, introducing an imbalance learning problem which is also harder.

Using a cost matrix, you can evaluate the extra cost due to merging some classes. merge classes when the cost on not distinguishing among them is low.

Building a taxonomy of class might can help working in many resolutions. Take for example animals, you can create a taxonomy of animals -> reptiles -> crocodiles and then try to distinguish reptiles from other animals or crocodiles from other reptiles.

In many some cases you might get many class labels due to a Cartesian product of some dimensions. Using the animals examples again, you might be interested in dimension like "is_flying?", "is_carnivore?" and learn them instead of the animal classes. These dimensions will probably be more balanced and can be learn with a certain degree on independence (reusing samples). The down side is the a flying carnivore will be a bird of prey and not necessarily a falcon.


Deep learning facial recognition research project I'm a second year pure maths, applied maths and computer science student. I have taken up a research course and was given the topic to focus on facial recognition using deep learning.

I have done a fair deal of reading on the topic (looking at older methods such as eigenfaces and fisherfaces all the way to newer methods such as deepface). I also feel comfortable with the maths and algorithms underlying ANNs.

I was wondering whether someone in the community could maybe point me in the direction in what I (as an undergraduate) could take a deeper look at in the category of facial recognition using deep learning?

I thought it might be interesting to pursue recognising faces given a few images of a face i.e. < 5. Maybe someone has a better suggestion?

First, tensorflow would be a great resource as it is gaining a userbase with exponential growth. Tensorflow has gpu support for single node calculations, and has recently added a distributed version for larger calculations.  There are some tutorials on convolution ANN, which will probably be the direction you will go with a computer vision problem. Also, some books are coming soon and some are in alpha.  But enough of plugging a particular library!

A perspective based facial recognition problem would be interesting if data were available e.g. with two perspectives on faces, do recognition on a third perspective.
A relative based facial recognition problem could be interesting, again, if data were available e.g. given two parents, try to find the child.  One could probably scrape facebook to get a reasonable data set for this one.
Blending and aging faces and controlling for sex (hair length, facial hair, etc) for problems where a kid has been abducted and you have parents pictures and sibling pictures and the child has grown up.
A simple aging recognition problem could be interesting... given baby pictures, can you recognize the adult or vice versa.

Just some thoughts, the possibilities are endless. I suggest starting with a very simple stub of a problem and then slowly adding sophistication.

Hope this helps!


Deep learning theory: why are hidden layers necessary? For this question, I’ll refer to the popular YouTube video by 3Blue1Brown on deep learning applied to recognition of written numbers.
The video describes a neural network with the following layers:

Input (individual grayscale pixels)

Small-scale feature determination (e.g., the top quarter of the loop in the number 9)

Larger-scale features (e.g., the entire loop in the number 9)

The output layer, which shows the probability of an input image being each of the numbers 0-9

I’m also going to try to read through the entire wiki section here, and I’m currently on the neural networks page.
I particularly like the explanation of coefficients... “A node combines input from the data with a set of coefficients, or weights, that either amplify or dampen that input, thereby assigning significance to inputs with regard to the task the algorithm is trying to learn; e.g. which input is most helpful is classifying data without error?“
Essentially, it’s saying that each input has some level of importance to each output, which to me begs the question... are component features/hidden layers necessary at all?  In the handwriting example, couldn’t every input node be connected to every output node without the use of hidden layers?  The idea is that all of the high-weight input pixels or a given output would still have high weights for that output, but the network would skip the feature/aggregation stages.  Is this just a matter of training efficiency (i.e., to prevent duplication by effectively extracting the same features more than once)?
Also, do the connections between various nodes need to specifically chosen so that the number of nodes and number and selection of connections are intelligently chosen?
Is it accurate to say that a sufficiently deep neural network looks to find the significance of all relevant combinations of input values and that’s basically all it’s doing?

Every input node can be connected to every output node, but that wouldn't account for non linearity of features and relationships amongst each other that are predictive. A neural network without any hidden layers, is just a regression. I think this is a great intro to deep learning.
In answer to your 2nd question: The number of nodes and connectivity are hyperparameters that should be optimized. Also, people use dropout (ie dropping out nodes) as a form of regularization.


Dataset used for regression Using Linguistic Cues for the Automatic Recognition of Personality in Conversation and Text
www.aaai.org › JAIR-3012PDF
Web results
Using Linguistic Cues for the Automatic Recognition of Personality in ...
While going through the paper I found out the scoring have veen done via regression model. However I can t find the data containing the assessment scores. I can see the essays dataset containing 1/0 (classification) of text.
Please let me know if I am missing something here.
Thanks,

Sorry I'm not going to read the paper, but in general it's perfectly possible to use binary values 0/1 as numerical regression targets. A regression model doesn't care about having a continuous sample in the training set, it just calculates the relationship between the features and the target variable. For example a linear regression needs only two points to learn a model in theory. Of course this doesn't work if there are specific patterns either between the two target values or outside them, the model can only generalize on the full continuous range. As a result it will mostly predict values in the range [0,1] but it can also predict values outside of it.


SMOTE and multi class oversampling I have read that the SMOTE package is implemented for binary classification. In the case of n classes, it creates additional examples for the smallest class. Can I balance all the classes by running the algorithm n-1 times?

I am pretty sure that the SMOTE package in python can also be used for multi-class as well.
Just you can check its implementation for iris dataset-

http://contrib.scikit-learn.org/imbalanced-learn/stable/auto_examples/plot_ratio_usage.html
-Please correct me if I am wrong.


How to reload all attributes in WEKA Is there a way to reload all attributes after having removed someones without reopening the data file ?

Any help please ?

Judging from the screenshot, you are currently looking at the data in the preproces tab from the explorer module. In the menu above the top menu in your screenshot should be an undo option (5th option from the left).


Replacing dataloader samples in training pytorch Initially, a data loader is created with certain samples. While training I need to replace a sample which is in dataloader. How to replace it in to dataloader.
train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)

for sample,label in train_dataloader:
        prediction of model
        select misclassified samples and change them in train_dataloader but      how to change sample in train_dataloader

While training, the misclassified samples need to be modified.
So How to replace a sample within train_dataloader?

Process per sample
Usually, you would process data at the sample level by creating your own class inheriting torch.utils.data.Dataset and do the processing inside __getitem__ method.
from torch.utils.data import Dataset
class MyDataset(Dataset):
  def __init__(self, *args, **kwargs):
    super().__init__(*args, **kwargs)

  def __getitem__(self, i):
    # Call the __getitem__ from Dataset class
    sample = super().__getitem__(i)
    # ...Do some processing on the sample here
    return sample

train_data = MyDataset()
train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)

Process per batch
You can use the collate_fn parameter of torch.utils.data.DataLoader to specify a custom function which processes your batch.
from torch.utils.data import Dataset, DataLoader

def collate_fn(samples):
  # samples is a list of samples you get from the __getitem__ function of your torch.utils.data.Dataset instance
  # You can write here whatever processing you need before stacking samples into one batch of data
  batch = torch.stack(samples, dim=0)
  return batch

train_data = Dataset()
train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size, collate_fn=collate_fn)


Reusing a portion of a neural network (with shared weights) in Keras I have a partial neural network with several layers of various types, with weights $\theta$; let's call it $F_{\theta}$; and the dimensions of input and output arrays are the same, implemented in Keras with Tensorflow backend. Within this framework, how do train a model of the form $y = F_{\theta}(F_{\theta}(x))$, or more generally some $n$-th iteration of $F_{\theta}$? In other words, I reuse an entire subset of the neural network, with all the same weights, so the full model is deeper but does not have more parameters than the shallower model.

When you build your Keras model using the functional interface, you can also build additional models on any subset of the paths through the network by reusing the intermediary functions. Then you can train on just portions of the network (given that you have targets for the outputs). I haven't tried to train on sub-networks of a network, but I do use these intermediary models to propagate activations between internal layers. The Python package conx that is built on top of Keras will build these intermediary models for you, by the way. For more, see: http://conx.readthedocs.io/en/latest/Getting%20Started%20with%20conx.html?highlight=propagate_from#Propagation-functions


Yarn timeline recovery not enabled error upgrading via ambari Using the automated upgrade, when I try to upgrade I get:

“YARN Timeline state preserving restart should be enabled
Reason: YARN should have state preserving restart enabled for the Timeline server. The yarn-site.xml property yarn.timeline-service.recovery.enabled should be set to true”

However, using Ambari, whenever I go to change the setting in yarn-site.xml it gets me to create a new configuration group and changes it there. I have moved all of the servers into that config group but the check still fails.

How do I modify the default yarn-site.xml?
Alternatively, does anyone know the actual script (or how to find it) that is kicked off when the automated upgrade is invoked?

Run the following from the ambari server's shell:

/var/lib/ambari-server/resources/scripts/configs.sh set mace gamma yarn-site "yarn.timeline-service.recovery.enabled" "true"

Where mace is the ambari server's host name and gamma is the cluster name


Getting Validation Accuracy of 99% with MNIST with less than 10000 parameters CNN Given MNIST dataset in keras,the challenge is to develop a CNN neural net model with less than 10k parameters with 99% validation accuracy.

I tried making the model for the same but am getting accuracy as 98.71.
Please find below the code for the same.

    def create_model():
        lay1=Conv2D(2,kernel_size=(1,1),activation='relu',padding='same')(inputs)
        lay1=Conv2D(2,kernel_size=(7,7),strides=(2,2),activation='relu',padding='same')(lay1)
        lay1=Conv2D(2,kernel_size=(1,1),activation='relu',padding='same')(lay1)
        lay1=MaxPooling2D(pool_size=(7,7),strides=(2,2),padding='same')(lay1)
        lay2=Conv2D(4,kernel_size=(1,1),activation='relu',padding='same')(inputs)
        lay2=Conv2D(4,kernel_size=(7,7),strides=(2,2),activation='relu',padding='same')(lay2)
        lay2=Conv2D(4,kernel_size=(1,1),activation='relu',padding='same')(lay2)
        lay2=MaxPooling2D(pool_size=(7,7),strides=(2,2),padding='same')(lay2)
        lay3=Conv2D(6,kernel_size=(1,1),activation='relu',padding='same')(inputs)
        lay3=Conv2D(6,kernel_size=(7,7),strides=(2,2),activation='relu',padding='same')(lay3)
        lay3=Conv2D(6,kernel_size=(1,1),activation='relu',padding='same')(lay3)
        lay3=MaxPooling2D(pool_size=(7,7),strides=(2,2),padding='same')(lay3)
        fc=concatenate([lay1,lay2,lay3])
        fc=Flatten()(fc)
        fc=Dense(10,activation='relu')(fc)
        outputs=Dense(10,activation='softmax')(fc)
        model=Model(input=inputs,output=outputs)
        model.compile(loss=keras.losses.categorical_crossentropy,
          optimizer=keras.optimizers.Adadelta(),
          metrics=['accuracy'])
        return model

The total parameters coming are 8,862 and the batch size used for the above is 32 and the number of epochs are 10.

Can you please suggest ways to improve the model with the constraints on the number of parameters so that the validation accuracy is 99% or above?

Policy distillation is an excellent way to efficiently reduce number of parameters and maintain the quality of the original (large) network.

According to the paper, while having orders of magnitude less numbers, distilled models can actually improve score from 84% up to 124% and above.

The basic idea is that we train a new, smaller network from scratch, to predict what the original (large) network would produce.

An analogy would be: it's a lot easier to assign the desired mindset to a young child, than to update an old person's way of thinking.


Encode missing data and unseen data Let's assume that I have a classification problem and all my features are categorical data.

I have missing data (and I do not want to do any imputation).

Also, I know that I will have some unseen data (at my test data) at some of my features.

My question is the following:

Should I encode the missing and unseen data (of the test set) into the same class or to different ones?

Which is the most common practice and why?

This depends at least a little on your model type and encoding strategy.  It sounds like you'll be one-hot encoding, with possible level/dummy for "missing" and/or for "new".  Given you say you won't impute, I guess you'll be using tree-based models?

If you have a dedicated level for unseen categories, then the trained model will have no idea what to do with it.  Results will vary among errors in model build, zero input of the dummy feature, and random contributions from the dummy feature.

Assigning new categories to the missing category is a little better, but will depend on how you treat missing values in your model training (but you've said you don't want to impute).  At least it gives a consistent result.  This may even be the best choice, depending on your data.

In your elaboration you mention bank names, which suggests an alternative (maybe several).
Perhaps these unseen banks will have some commonality with banks in the training set (they're small and local maybe?); if so, you could try to coarsen the categories in the train set so that these new ones will fall nicely into those categories (and then encode all future unseen banks into that new category; make a note to reexamine this process when you retrain!).
If your training data has lots of small levels, it can also make sense to lump the rare categories together (to help avoid overfitting); in that case you can just drop the new levels into that one.  Again depending on your data this might work well (small and local would also be rare, so this accidentally recovers the last idea in this case).


How to find correlation between categorical data and continuous data I'm working on imputing null values in the Titanic dataset.
The 'Embarked' column has some. I do NOT want to just set them all to the most common value, 'S'.
I want to impute 'Embarked' based on its correlation with the other columns.
I have tried applying this formula to the 'Embarked' column:
def embark(e):
        if e == 'S': return 1
        if e == 'Q': return 2
        if e == 'C': return 3
        else: return 4

This allows me to check out data.corr(), but I think it's trickier than that since I'll get a different correlation with different value assignments (right??). I also thought about using a four-dimensional (for S,Q,C,NaN) one-hot vector, but I doubt that would work.
Is there a skLearn method that does this some way? Any further insights on the matter?

For your specific case I would recommend the "grouped mode" because that would be the value you are interested in imputing (I did the same for this kaggle challenge).
On more general terms we have to understand the scale of each variable. We often talk about categorical data but in more detail we have to differentiate between "nominal data" and "ordinal data".
Income brackets are ordinal, that means there is a clear numerical hierarchy, while other data such as the "Embarkment" here is more nominal, that means there is no order or numerical relation.
Therefore a correlation is impossible and it is more helpful to think in terms of grouped distributions.
What does that mean for imputation
Finally besides simply imputing the grouped mode (that is the most frequent value per group) you could try imputation algorithms like MICE which are in a way predicting the missing value based on other variables. This is possible for nominal as well as ordinal categorical factors.
Please be aware that imputation is highly depended on the question of why a value is missing. You have to make sure that the value is not MNAR before you try imputation at all.


How to retrieve column names from applying a wrapper method in feature selection? This question probably has a simple answer to it, so I will get to the point...
How do I retrieve the names of the columns from applying a wrapper method in feature selection?
Code I have used:
from mlxtend.feature_selection import SequentialFeatureSelector as SFS

X = df[['A','B','C','D']].values
y = df[['F']].values

classifier = KNeighborsClassifier(n_neighbors=7)
code = SFS(classifier,
           k_features=5,
           forward=True,
           floating=False,
           verbose=8,
           scoring='accuracy'
)

code.fit(X, y)
code.k_feature_names_

Output:
Features: 3/3 -- score: 0.78

('1' '2', '3')

Therefore, how do I retrieve the names of the features ('1' '2', '3')?

Based on this scikit-learn documentation, you can get a boolean mask (in the same order) of the input features, via the get_support method:


How is error back-propagated in a multi-layer RNN Let's say I have a 2 layer LSTM cell, and I'm using this network to perform regression for input sequences of length 10 along the time axis.

From what I understand, when this network is 'unfolded', it will consist of 20 LSTM cells, 10 for each layer. So the 10 cells corresponding to the first layer receive the network input for t = 1 to 10, whereas the 10 cells corresponding to the second layer receive the first layer's output for t = 1 to 10. In other words, the output from the cell in layer 1 corresponding to t = 1 goes to (1) the 'next' cell in layer 1 corresponding to t = 2, and (2) the cell in layer 2 corresponding to t = 1.

So when the error is back-propagated, will there not be two derivatives coming into each cell in layer 1? If so, how is weight update performed? Is the sum or mean of both derivatives used or is there something else going on?

As you can see here, derivatives will be propagated by the chain rule although they are stacked. Actually, there will be two main paths. The first one will be backpropagation through time and the next one will be the backpropagation from the output of each unrolled cell which can directly be connected to the output or can be connected to the stacked unrolled cells. Also, take a look at here.


Cosine similarity versus dot product as distance metrics It looks like the cosine similarity of two features is just their dot product scaled by the product of their magnitudes. When does cosine similarity make a better distance metric than the dot product? I.e. do the dot product and cosine similarity have different strengths or weaknesses in different situations?

Think geometrically. Cosine similarity only cares about angle difference, while dot product cares about angle and magnitude. If you normalize your data to have the same magnitude, the two are indistinguishable. Sometimes it is desirable to ignore the magnitude, hence cosine similarity is nice, but if magnitude plays a role, dot product would be better as a similarity measure. Note that neither of them is a "distance metric".


Where can I download historical market capitalization and daily turnover data for stocks? There are plenty of sources which provide the historical stock data but they only provide the OHLC fields along with volume and adjusted close. Also a couple of sources I found provide market cap data sets but they're restricted to US stocks. Yahoo Finance provides this data online but there's no option to download it ( or none I am aware of ).

Where can I download this data for stocks belonging to various top stock exchanges across countries by using their ticker name ?
Is there some way to download it via Yahoo Finance or Google Finance ?

I need data for the last decade or so and hence need some script or API which would do this.

Quant SE is better place for questions related to getting financial data:

What data sources are available online
Market capitalization data | Quantitative Finance


How to use K.function with two inputs and a concatenate layer? In Keras, I try to compute use the K.function between some layers. But I get an error when I use a Concatenate layer.

Here is a minimal code you can try yourself:

import numpy as np
from keras.layers import *
from keras.models import Model
def test_model(concat, conv2_dim=32, kernel=(3,3), dropout=0,
                     optimizer='adam', extra_conv=False, padding='valid',
                     width_dense=1024, batch_norm=False):
    img_shape = (64,64,1)

    image_input = Input(shape=img_shape)
    entropy_vector = Input(shape=(1,))

    conv = Conv2D(conv2_dim, kernel_size=kernel,
                     activation='relu', padding=padding,
                     input_shape=img_shape)(image_input)
    conv = MaxPooling2D(pool_size=(2, 2), padding=padding,)(conv)

    final_conv = Flatten()(conv)
    dense = Dense(width_dense, activation='relu')(final_conv)

    concat_layer = Concatenate()([dense, entropy_vector])

    if(concat): #Skipping
        flat = Dense(width_dense//4, activation='relu')(concat_layer)
    else:
        flat = Dense(width_dense//4, activation='relu')(dense)

    output = Dense(1, activation='sigmoid')(flat)

    model = Model(inputs=[image_input, entropy_vector], outputs=[output])
    model.compile(loss='binary_crossentropy',
                  optimizer=optimizer,
                  metrics=['accuracy'])
    return model

X = [np.ones((1,64,64,1)), np.ones(1,)]
m = test_model(concat=False) # you can try with False
fn = K.function(m.input, [m.output])

m.predict(X)
fn(X)

But I get this error if concat=True in the test_model function:

InvalidArgumentError: ConcatOp : Ranks of all input tensors should match: shape[0] = [1,1024] vs. shape[1] = [1]
     [[{{node concatenate_18/concat}} = ConcatV2[N=2, T=DT_FLOAT, Tidx=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](dense_112/Relu, _arg_input_44_0_1/_4327, concatenate_18/concat/axis)]]
     [[{{node dense_114/Sigmoid/_4329}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_68_dense_114/Sigmoid", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]


Why should the initialization of weights and bias be chosen around 0? I read this:

  To train our neural network, we will initialize each parameter W(l)ijWij(l) and each b(l)ibi(l) to a small random value near zero (say according to a Normal(0,ϵ2)Normal(0,ϵ2) distribution for some small ϵϵ, say 0.01)

from Stanford Deep learning tutorials
at the 7th paragraph in the Backpropagation Algorithm

What I don't understand is why the initialization of the weight or bias should be around 0?

Assuming fairly reasonable data normalization, the expectation of the weights should be zero or close to it. It might be reasonable, then, to set all of the initial weights to zero because a positive initial weight will have further to go if it should actually be a negative weight and visa versa. This, however, does not work. If all of the weights are the same, they will all have the same error and the model will not learn anything - there is no source of asymmetry between the neurons.

What we could do, instead, is to keep the weights very close to zero but make them different by initializing them to small, non-zero numbers. This is what is suggested in the tutorial that you linked. It has the same advantage of all-zero initialization in that it is close to the 'best guess' expectation value but the symmetry has also been broken enough for the algorithm to work.

This approach has additional problems. It is not necessarily true that smaller numbers will work better, especially if the neural network is deep. The gradients calculated in backpropagation are proportional to the weights; very small weights lead to very small gradients and can lead to the network taking much, much longer to train or never completing.

Another potential issue is that the distribution of the outputs of each neuron, when using random initialization values, has a variance that gets larger with more inputs. A common additional step is to normalize the neuron's output variance to 1 by dividing its weights by $sqrt(d)$ where $d$ is the number of inputs to the neuron. The resulting weights are normally distributed between $\left[\frac{-1}{\sqrt{d}}, \frac{1}{\sqrt{d}}\right]$


ValueError: Cannot convert a partially known TensorShape to a Tensor: (?, 256) I'm working on a sequence to sequence approach using LSTM and a VAE with an attention mechanism.

p = np.random.permutation(len(input_data))
input_data = input_data[p]
teacher_data = teacher_data[p]
target_data = target_data[p]
BUFFER_SIZE = len(input_data)
BATCH_SIZE = 64
embedding_dim = 100
units = 256
vocab_in_size = len(input_lang.word2idx)
vocab_out_size = len(target_lang.word2idx)

# Create the Encoder layers first.

encoder_inputs = Input(shape=(None,))
encoder_emb = Embedding(input_dim=vocab_in_size, output_dim=embedding_dim)
encoder_lstm =LSTM(units=units, return_sequences=True, return_state=True)
encoder_outputs, state_h, state_c =
encoder_lstm(encoder_emb(encoder_inputs))
encoder_states = [state_h, state_c]
 #################### Adding VAE #######################
latent_dim =256
# output layer for mean and log variance
z_mu = Dense(latent_dim)(encoder_outputs)  #remplacer h
z_log_var = Dense(latent_dim)(encoder_outputs)
def sampling(args):
     batch_size=1
     z_mean, z_log_sigma = args
     epsilon = K.random_normal(shape=(batch_size, latent_dim),
                          mean=0., stddev=1.)
     return z_mean + K.exp(z_log_sigma) * epsilon

z = Lambda(sampling, output_shape=(latent_dim,))([z_mu, z_log_var])
state_h= z
state_c = z
encoder_states = [state_h, state_c]

def vae_loss(y_true, y_pred):

    recon = K.sum(K.binary_crossentropy(y_pred, y_true), axis=-1)
    kl = 0.5 * K.sum(K.exp(z_log_var) + K.square(z_mu) - 1. - z_log_var,
       axis=-1)
     return recon + kl[:, None]

 ##########################

# Now create the Decoder layers.
decoder_inputs = Input(shape=(None,))
decoder_emb = Embedding(input_dim=vocab_out_size, output_dim=embedding_dim)
decoder_lstm = LSTM(units=units, return_sequences=True, return_state=True)
decoder_lstm_out, _, _ = decoder_lstm(decoder_emb(decoder_inputs),
    initial_state=encoder_states)

# Attention layer
attn_layer = AttentionLayer(name='attention_layer')
attn_out, attn_states = attn_layer([encoder_outputs, decoder_lstm_out])

When I execute this code I get this error at the last line:

  ValueError: Cannot convert a partially known TensorShape to a Tensor: (?, 256)

I tried all solutions given for this error, no one solved my problem, if any one can help, I'll be so thankfull.

The problem is inside the sampling functions. I had the same problem and found out the answer in the tutorial here.

my original code is:

def sampling(args):
     z_mean, z_log_sigma = args
     epsilon = K.random_normal(shape=z_mean.shape)
     return z_mean + K.exp(z_log_sigma) * epsilon

with this sampling method, I got the same error with yours.

the codes fixed the problem is:

class Sampling(layers.Layer):
  def call(self, inputs):
    z_mean, z_log_var = inputs
    batch = tf.shape(z_mean)[0]
    dim = tf.shape(z_mean)[1]
    epsilon = tf.keras.backend.random_normal(shape=(batch, dim))
    return z_mean + tf.exp(0.5 * z_log_var) * epsilon

It uses tf.shape to get the required shape parameters. It somehow fixes the problem.


How does SelectKBest work? I am looking at this tutorial: https://www.dataquest.io/mission/75/improving-your-submission

At section 8, finding the best features, it shows the following code.

import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif

predictors = ["Pclass", "Sex", "Age", "SibSp", "Parch", "Fare", "Embarked", "FamilySize", "Title", "FamilyId"]

# Perform feature selection
selector = SelectKBest(f_classif, k=5)
selector.fit(titanic[predictors], titanic["Survived"])

# Get the raw p-values for each feature, and transform from p-values into scores
scores = -np.log10(selector.pvalues_)

# Plot the scores.  See how "Pclass", "Sex", "Title", and "Fare" are the best?
plt.bar(range(len(predictors)), scores)
plt.xticks(range(len(predictors)), predictors, rotation='vertical')
plt.show()

What is k=5 doing, since it is never used (the graph still lists all of the features, whether I use k=1 or k="all")?
How does it determine the best features, are they independent of the method one wants to use (whether logistic regression, random forests, or whatever)?

The SelectKBest class just scores the features using a function (in this case f_classif but could be others) and then "removes all but the k highest scoring features". http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectKBest.html#sklearn.feature_selection.SelectKBest

So its kind of a wrapper, the important thing here is the function you use to score the features.

For other feature selection techniques in sklearn read: http://scikit-learn.org/stable/modules/feature_selection.html

And yes, f_classif and chi2 are independent of the predictive method you use.


How to use CNN to build ROC curves I know we can use SVMs probabilities after predicting validation data in order to build ROC curves. However, for CNNs, I have a binary classification problem and so the sigmoid activation function will give me probabilities of both classes. So, which probability should I use to build the ROC curve? what happens if the probabilities are close to each other?

For binary classification problem, naturally, classes are called positives and negatives. Positive is usually a class of detection something that we are interested in, for example, spam or disease detection. According to documentation for sklearn.metrics.roc_curve, it takes as argument probabilities for positive class. For most binary classifiers, output probability usually refers to the probability of positive class (which is marked as 1 in the data). So, use probabilities of the class marked as 1 in the data.


Genetic Algorithm to find best parameter values of an estimaor I am making some stochastic training ensemble classes in Python, and I want to get hyperparameters values. Grid search will take too long for moderate data sets, because in my stochastic training I train different learners on different dropouts, as in cross-validation, then I average or weight-average on best error for dropout. Having this set, I want to get best parameter values at every iteration, so I thought to use a genetic algorithm approach.

I found one example on Github, but failed to install, and I am looking for more references of genetic algorithms for doing this. Can anyone help me?

Having used genetic algorithms extensively in both professional projects and for my own entertainment, I think it is best to just code your own set of functions/classes for the specific project.  As you have seen, there are quite a few well-intended open-source GA packages out there. However, many of the fine details and tweaks necessary for a good GA are problem-specific.  It is also more convenient to work directly with your existing data structures, rather than adapt to someone else's worldview.  In other words, cost-benefit does not seem to work out when applying someone else's GA package.  You need a function for creating a random instance, a function for checking it is valid, a function for scoring it, and a mating function.  That's pretty much it!


Is the number of iterations in gradient tree boosting just the number of trees? I have been searching for a while and I just can't find any indication. When people talk about iterations in algorithms like XGBoost or LightGBM, or Catboost, do they mean how many decision trees i.e. base learners will be built? I.e. XGboost m=100 means the algorithm will build a total of 100 base learners, each calculating and optimizing towards the residual value of the previous prediction?
Or is it more like 1 epoch in deep learning?

Your first interpretation is correct. One base learner will be added per boosting iteration/round and that is probably what people are referring to when talking about iterations.

From wiki:

  One natural regularization parameter is the number of gradient
  boosting iterations M (i.e. the number of trees in the model when the
  base learner is a decision tree).

Iterations take place in other parts of the algorithm, for instance in the gradient descent, but I don't think that is what is discussed if it is simply referred to as "iterations".

This was really helpful when I was trying to understand GBMs


Why some papers write false positive rate per case instead of percentage rate? In some published works, especially in medical image analysis, instead of writing FP rate as percentage, they write it per case, for example, FP: 128.52 [/case].
What is the meaning of this? Is it different from percentage rate? How to calculate it per case?

In general a false positive rate is abbreviated FPR, so it's likely that "FP: 128.52 [/case]." is not the rate but simply the number of false positive instances. In this case they should also mention the total number of instances by case, and you can easily obtain the rate: FPR = FP/total


How to model a Bimodal distribution of target variable I want to regress on this target, have tried multiple transformations to bring it to normal but its not helping, read some stuffs online but none of the suggestions have worked till now.

I am attaching the residual histogram as well, somehow the residuals are normally distributed.

Thanks in Advance.

To my understanding you should be looking for something like a Gaussian Mixture Model - GMM or a Kernel Density Estimation - KDE model to fit to your data.

There are many implementations of these models and once you've fitted the GMM or KDE, you can generate new samples stemming from the same distribution or get a probability of whether a new sample comes from the same distribution.

In python an example would be like this:(directly taken from here)

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm
from sklearn.neighbors import KernelDensity

# Plot a 1D density example
N = 100
np.random.seed(1)
X = np.concatenate((np.random.normal(0, 1, int(0.3 * N)),
                    np.random.normal(5, 1, int(0.7 * N))))[:, np.newaxis]

X_plot = np.linspace(-5, 10, 1000)[:, np.newaxis]

true_dens = (0.3 * norm(0, 1).pdf(X_plot[:, 0])
             + 0.7 * norm(5, 1).pdf(X_plot[:, 0]))

fig, ax = plt.subplots()
ax.fill(X_plot[:, 0], true_dens, fc='black', alpha=0.2,
        label='input distribution')

for kernel in ['epanechnikov', 'tophat', 'gaussian']:
    kde = KernelDensity(kernel=kernel, bandwidth=0.5).fit(X)
    log_dens = kde.score_samples(X_plot)
    ax.plot(X_plot[:, 0], np.exp(log_dens), '-',
            label="kernel = '{0}'".format(kernel))

ax.text(6, 0.38, "N={0} points".format(N))

ax.legend(loc='upper left')
ax.plot(X[:, 0], -0.005 - 0.01 * np.random.random(X.shape[0]), '+k')

ax.set_xlim(-4, 9)
ax.set_ylim(-0.02, 0.4)
plt.show()

In the end the kde model, could be used for sampling new data points or predicting the probability of a new sample to have been generated from this distribution.

You should play around with different kernels in KDE models or number of base distributions in GMMs, along with other parameters to get optimal results for your data.


Is there a ready-made classifier to determine if webpage is an article or not? I am trying to solve a problem, where I have a set of URLs and I need to filter out only those that can be classified as an article (i.e. typically not include privacy policies, terms of usage, webpage for unsubscribing from a feed etc.).
Is there a trained classifier that can be used to filter these webpages? Every webpage classification material I found was about classifying articles into categories, but this doesn't concern me. I need to go a step back and figure out if the webpage is actually an article or not.
If there's not ready made classifier for this, what would be the best approach to create such classifier?
UPDATE: To clarify some questions below. The problem is not to identify an article only by the url, but to find a way of labelling a webpage as article/non-article based upon its contents (html, visual representation?, some other metadata perhaps). From the research I've done since asking the question, I believe this to be a variant of something called blog identification. I understand that the definition of an article is an important piece here and very ambiguous requirement, however that is exactly the hard part I have been struggling with while searching for some algorithm to do just that. If I had a fixed dataset, I should be able to identify common patterns and come up with a definition of an article by myself. But since the set of webpages that need to be filtered is very dynamic and unknown beforehand, I don't want to rely on my own custom rules for defining how an article looks like but would much rather use a method of identification that might have some research or at least a field-test validation behind it. It just seems to me, that this is a problem that someone had to be solving before me and that there might be solutions based upon that already.

I'm going on the assumption that you "have a set of URLs" that you need to pre-filter prior to doing something else with.  Based on this assumption you could use urlparse and some string patterns to filter out X percentage of the URLs in your set that are most likely non-articles.
import re as regex
from urllib.parse import urlparse

urls =['https://candid.org/privacy-policy',
       'https://happify.com/health/privacy-policy',
       'https://www.abstract.com/legal/customer-terms-of-service',
       'https://pesa.org.au/membership/terms-of-service-and-privacy-statement',
       'https://www.cnn.com/2021/01/14/economy/unemployment-benefits-coronavirus/index.html',
       'https://www.nasa.gov/content/nasa-rss-feeds',
       'https://www.sciencedaily.com/newsfeeds.htm',
       'https://www.ndtv.com/rss']

patterns = ['privacy-policy', 'customer-terms-of-service', 'terms-of-service-and-privacy-statement', 'rss-feeds',
            'newsfeeds', 'rss']
for url in urls:
    split_url = urlparse(url)
    possible_article = [pattern for pattern in patterns if regex.findall(pattern, split_url.path)]
    if not possible_article:
        print(f'Possible article: {url}')
        # output
        Possible article: https://www.cnn.com/2021/01/14/economy/unemployment-benefits-coronavirus/index.html

You could potentially expand the regular expression above to flag other URLs that meet common article URLs that can date strings or common keywords, such news or wirestory.
import re as regex
from urllib.parse import urlparse

urls =['https://www.ndtv.com/rss',
       'https://candid.org/privacy-policy',
       'https://www.sciencedaily.com/newsfeeds.htm',
       'https://happify.com/health/privacy-policy',
       'https://www.nasa.gov/content/nasa-rss-feeds',
       'https://www.bbc.com/news/technology-55675826',
       'https://www.abstract.com/legal/customer-terms-of-service',
       'https://pesa.org.au/membership/terms-of-service-and-privacy-statement',
       'https://www.cnn.com/2021/01/14/economy/unemployment-benefits-coronavirus/index.html',
       'https://www.cnet.com/news/samsungs-galaxy-s21-upgrades-likely-wont-spell-an-end-to-galaxy-fe-or-note-lines-yet',
       'https://abcnews.go.com/Politics/wireStory/trump-leave-washington-morning-bidens-inauguration-75278801?cid=clicksource_4380645_6_heads_hero_live_headlines_hed']

non_article_patterns = ['privacy-policy', 'customer-terms-of-service', 'terms-of-service-and-privacy-statement',
                        'rss-feeds', 'newsfeeds', 'rss']

known_article_patterns = ['\d{4}\/\d{2}\/\d{2}', 'news', 'wireStory']

for url in urls:
    split_url = urlparse(url)
    non_article = [pattern for pattern in non_article_patterns if regex.findall(pattern, split_url.path)]
    if non_article:
        pass
    else:
        possible_article = [pattern for pattern in known_article_patterns if regex.findall(pattern, split_url.path)]
        if possible_article:
            print(f'Possible article: {url}')
            Possible article: https://www.bbc.com/news/technology-55675826
            Possible article: https://www.cnn.com/2021/01/14/economy/unemployment-benefits-coronavirus/index.html
            Possible article: https://www.cnet.com/news/samsungs-galaxy-s21-upgrades-likely-wont-spell-an-end-to-galaxy-fe-or-note-lines-yet
            Possible article: https://abcnews.go.com/Politics/wireStory/trump-leave-washington-morning-bidens-inauguration-75278801?cid=clicksource_4380645_6_heads_hero_live_headlines_hed

BUT as Erwan pointing out in his comment it's not clear "how you define the difference between article and not article", so my answer is only part of a potential solution to your problem of URL classification.
It's hard to make any solid recommendation for any "out-of-the-box" solution that can meet an undefined use case.
Here are some links that might help you:

paper: Fast webpage classification using URL features
paper: A Comprehensive Study of Features and Algorithms for URL-Based
Topic Classification
paper: Web Page Classification: Features and Algorithms
commercial: Website Categorization API

I'm still of the mindset that your use case will likely require a multi-prong approach and will need to be refined as more URLs are classified as either "an article or non-article."


My models performs better with the arbitrary random feature. How can I interpret this? I am training 6 different classifiers 'Decision Tree', 'Random Forest', 'Logistic regression' and 'SVM' with different kernels. There are about 80 dependent variables including categorical and numerical variables.
For my experiment, I added a 'random' column which is generated by any arbitrary random numbers, but all the model performs better on both validation set and test set. Is there any good explanation about this phenomenon?

It's a good sanity check, but the fact it fails means there must be a mistake somewhere:

Hypothesis 1: mistake in the interpretation of the results. Is the performance improvement significant? If not, it might simply be due to chance. That would mean that none of the models actually uses the random feature, one happens to be slightly better by chance. However the chances that this would happen with 6 distinct classifiers are very low ($1/2^6$ to be precise)
Hypothesis 2: mistake in the generation of the the random feature. If it actually contributes to the prediction, then there must be a regular pattern so it's not truly random. Manually inspect the models (especially the decision tree one, it's the easiest to interpret) to see what happens. Then try to reproduce the result with a different random sequence and observe the model again: if the model uses the random feature in the same way then it's not really random.
Hypothesis 3: mistake in the split between training and test set. We always underestimate how easy it is to make a stupid mistake, so my money is on this one ;) More seriously, the fact that the performance is better with the random feature points to this direction: assuming it's truly random and it has mostly distinct values, an overfit model could use it as an id for an instance that was seen in the training set. Try re-sampling and/or cross-validation.


Detecting anomalies with neural network I have a large multi dimensional dataset that is generated each day.
What would be a good approach to detect any kind of 'anomaly' as compared with previous days? Is this a suitable problem that could be addressed with neural networks?
Any suggestions are appreciated.
Additional information: there are no examples, so the method should detect the anomalies itself

From the formulation of the question, I assume that there are no "examples" of anomalies (i.e. labels) whatsoever. With that assumption, a feasible approach would be to use autoencoders: neural networks that receive as input your data and are trained to output that very same data. The idea is that the training has allowed the net to learn representations of the input data distributions in the form of latent variables.

There is a type of autoencoder called denoising autoencoder, which is trained with corrupted versions of the original data as input and with the uncorrupted original data as output. This delivers a network that can remove noise (i.e. data corruptions) from the inputs.

You may train a denoising autoencoder with the daily data. Then use it on new daily data; this way you have the original daily data and an uncorrupted version of those very same data. You can then compare both to detect significant differences.

The key here is which definition of significant difference you choose. You could compute the euclidean distance and assume that if it surpasses certain arbitrary threshold, you have an anomaly. Another important factor is the kind of corruptions you introduce; they should be as close as possible to reasonable abnormalities.

Another option would be to use Generative Adversarial Networks. The byproduct of the training is a discriminator network that tells apart normal daily data from abnormal data.


How would you quantify an experience into a score without labeled data How would you approach a scenario where you have to quantify an abstract notion like “customer experience” without having any labeled data? So basically what you have are bunch of variables that you know, more or less, how they affect the experience, but you don't know the "weights" of importance of each one.

So e.g. if it is an experience from a food delivery service, then I have the ETAs for the order, ratings (not very reliable by the day), cancelation reasons, $ amounts, etc.

To continue with your example, a domain expert (or business expert) would have to determine which of these is more important and therefore should have higher weight.

They may care about ETAs for the order because their customers appreciate getting a fast delivery, but may not care as much about the cancellations due to a high amount of orders they have anyways. Now this is purely guessing.

In order to build a model that will suit the business's needs - this should ideally be discussed with someone who understands the domain well and knows what they want to achieve.


Service Request classification, questionnaire filling and call logging I am very new to machine learning. I just went through some of the tutorials in Azure and completed one practice workflow(car price prediction). I hope I can ask basic questions here.

Scenario : We get service request from our customers via email. This has fields like customer name, user name, email id, Equipment affected, type of call and Issue experienced(this is a free text area).

The employee reads this email, mainly the issue experienced. Based on the issue experienced section, s/he takes the appropriate actions. We will have 4-6 fields(type of request , a few questionnaire etc). The issue experienced is a free text area where customer can write anything about the issue.

Does this qualify as a AI model if we have last 2-3 years data. If yes, Is multi class classification the solution? If not, which ML algorithm needs to be used here. Can I rely on Azure for this or do we need to build a new model/algorithm for this?

Sorry if it is a too basic question

He reads this and in another application he logs a call. There he has to fill the request type and a number of questionnaires.

This is a "multi class" classification problem.

You can start with a library that makes it easy to mix text , categorical and numerical attributes.

Text : "issue experienced" attribute   (Library should be able to apply NLP on text )
Categorical : "type of request" and similar attributes
Numerical : "price" and similar attribute (if applicable)

One such library is : Uber Lugwig

If you have data in CSV, it should not take more than a few hours to train the model.

Documentation : https://uber.github.io/ludwig/user_guide/

Introduction : https://hackaday.com/2019/02/25/ludwig-promises-easy-machine-learning-from-uber/


What exactly is convergence rate referring to in machine learning? My understanding of the term "Convergence Rate" is as follows:

Rate at which maximum/Minimum of a function is reached, so in logistic
regression rate at which gradient decent reaches global minimum.

So by convergence rate I am guessing it is measure of:

time measured from start of gradient descent until it reaches global maximum.
average number of distance our model went downhill(Do not know technical term...) for each iteration.

Can someone verify whether or not one of my guess is True if not explain what it means?

A rate is always a gain per some time/step. A rate can exist even if the maximum is never reached. In supervised learning a loss function is defined, which is expected to have a global maximum, that we try to reach by gradient descent. How much closer we get with each timestep/iteration/epoch/batch is the rate of convergence.


Predicting house price using linear regression I'm trying to predict a house price using linear regression method. I gather the real data from a real estate website. I have some features and two numerical value in which the price is the target variable to be guessed. I have about 3000 data in which first column is provinces, area field of the house as meter square, following how many salon + rooms, and the other features as 0 or 1. What I try to obtain is a formula(coefficients). However, the Orange Toolkit which I use shows far strange guessing. Is there any wrong step or omitted step(s)? Can be the guesses improved? By the way, via the Box link the dataset can be downloaded.

https://app.box.com/s/0tjroz2tn8h710n6l1q5n5w4y0htn8lt


XGB custom objective function - small change to default regression squared error objective function Where can I find the code for the default squared error objective function? I just want to make a small change to re-weight certain datapoints?

You can find a good explanation and example on creating a custom objective function here:
https://xgboost.readthedocs.io/en/latest/tutorials/custom_metric_obj.html
https://github.com/dmlc/xgboost/blob/master/demo/guide-python/custom_objective.py
https://github.com/dmlc/xgboost/blob/master/demo/guide-python/custom_rmsle.py
Original XGB code repo is here, you may have to do some digging to find the code for your objective function:
https://github.com/dmlc/xgboost


What is the differences between normal equation and gradient descent for polynomial regression I'm new to machine learning and willing to study and work with machine learning. It just that I still don't get to understand the benefits of using the normal equation in some occasion in comparison with gradient descent. I use Andrew Ng's course on Coursera but the notation really makes me a hard time to understand.

I want to know more about the derivation of the cost function $J(\theta)$ for polynomial regression and the reason why he uses the transpose of vector $x(i)$

You want to solve $X \times \theta = Y$. Actually, you have to find the parameters $\theta$. To find it, you should multiply both sides by $X^{-1}$ but it may have not an inverse. Consequently, you multiply each side by $X^t$ due to the fact that $X^tX$ has the inverse. After that, you multiply each side by $(X^tX)^{-1}$ which leads to a closed form equation for finding $\theta$. It should be mentioned that $X^tX$ does not always have an inverse. If you want to be sure it has it, you have to exploit data samples which are more than the number of features and they should not be linearly dependent otherwise the constructed matrix will not inverse and you will not be able to use it.


What Introductory Statistics book would you recommend? Is there a particularly good book or good publisher for teaching Statistics from beginner level for someone interested in studying Data Science?

I would recommend Elements of Statistical Learning, by Trevor Hastie and Rob Tibshirani. That link gets the book directly from their Stanford website.

In addition, there are also accompanying videos (they follow the structure of a similar book), which are really helpful!

They offer a simpler set of books and online courses, which you can find listed here.


What method/algorithm for constrained multi-target regression I am working with three dimensional measurement data and want to model them using a multivariate linear regression.
I have already implemented a simple gradient descent algorithm to solve the classic regression problem
$y = \beta_0 + \beta_1x_1 + \beta_2x_2 + \beta_3x_3.$
However my ultimate goal is to fit three target variables with a hard condition:
$y_1 = \beta_{10} + \beta_{11}x_1 + \beta_{12}x_2 + \beta_{13}x_3,\\
y_2 = \beta_{20} + \beta_{12}x_1 + \beta_{22}x_2 + \beta_{23}x_3,\\
y_3 = \beta_{30} + \beta_{31}x_1 + \beta_{32}x_2 + \beta_{33}x_3,\\
\text{subject to }\beta_{11}+\beta_{22}+\beta_{33}=0.$
Or in other words
$\underline{y} = \underline{\underline{B}}\cdot\underline{x} + \underline{\beta_0},\\
\text{subject to } \mathrm{tr}(\underline{\underline{B}})=0.$
I've searched high and low for multi-target or multi-output regression methods but I could not find out how I would possibly go on to implement a hard condition into them. Happy for any kind of recommendations, even just keywords for further research.

After some further research and inspiration I have figured out the following method:
I am using the vector notation of my problem with
$\underline{y}=\begin{pmatrix}y_1\\y_2\\y_3\end{pmatrix},~\underline{x}=\begin{pmatrix}1\\x_1\\x_2\\x_3\end{pmatrix},~\underline{\underline{B}}=\begin{pmatrix}\beta_{10}&\beta_{11}&\beta_{12}&\beta_{13} \\ \beta_{20}&\beta_{21}&\beta_{22}&\beta_{23} \\ \beta_{30}&\beta_{31}&\beta_{32}&\beta_{33}  \end{pmatrix}$
This results in the optimisation problem with $N$ examples
$\underline{\underline{\hat{B}}}=\underset{\beta \in C}{\operatorname{argmin}}L(\underline{\underline{B}})=\underset{\beta \in C}{\operatorname{argmin}}\sum_{i=1}^{N}\lVert \underline{y}_i - \underline{\underline{B}}\cdot\underline{x}_i \rVert^2,~~~~\text{with}~~~~C= \lbrace \underline{\underline{B}}\in \mathbb{R}^{3\times4} \mid \beta_{11}+\beta_{22}+\beta_{33}=0\rbrace$
For solving this, I am using projected gradient descent where $\underline{\underline{B}}$ for every iteration is projecet onto $C$. Normally the Projection operator would require another optimisation process $\operatorname{proj}_C(x)=\operatorname{argmin}_u \big\lbrace \chi_C(u) + \frac{1}{2}\lVert u-x\rVert^2\big\rbrace$ with $\chi_C$ the characteristic function.
Now comes the neat part: Since the constraint $C$ describes a flat subspace in a 12-dimensional vector space, the projection can be described as a simple linear map.
Using a reduced version of $\underline{\beta}=\begin{pmatrix}\beta_{11}\\\beta_{22}\\\beta_{33}\end{pmatrix}$ (in the projection step $\underline{\underline{B}}$ will be flattened to a vector to avoid working with 3D tensors), the projection $\underline{\beta}'=\operatorname{proj}_C(\underline{\beta})$ can be represented as
$\underline{\beta}' = \underline{\underline{P}}\cdot\underline{\beta}~~~~\text{with}~~~~ \underline{\underline{P}}=\frac{1}{3}\begin{pmatrix}2&-1&-1\\-1&2&-1\\-1&-1&2\end{pmatrix}$
The vector $\underline{\beta}$ and projection map $\underline{\underline{P}}$ can then be arbitrarily augmented with the remaining entries of $\underline{\underline{B}}$, as they will not be affected by the projection.
With the above method, my constrained gradient descent is only slightly more expensive than a usual gradient descent method by computing one extra linear operation for each iteration.
The results I have obtained so far with the method seem promising


Importance/intuition behind stacking RNNs Nowadays there's a trend towards using architectures of "deep"  RNNs i.e. vertically stacked RNNs. RNN chapter from Bengio's bookThese networks seem to work well in practice.

What's the intuition around using vertically stacked layers of RNNs (beyond the obvious fact that they increase the capacity by increasing the # parameters)?

All neural networks can increase expressiveness and representational capacity by stacking layers. Each later layer can learn to non-linearly weigh the earlier layers. These non-linearities allow any function to be approximated. In the case of Recurrent Neural Network (RNN), it is functions over time. Stacked RNNs have increased abilities to learn functions over time.


A good machine learning approach for distribution of a whole? So I had done with different classification, regression and clustering approaches for predictions of values etc. I was wondering if there is a machine learning approach for distribution of a whole based on on some features (I do not know if there is an approach for that I just could not find one with my research).

An easy example might be lets consider we have height and weight data of many children and we have to distribute a given number of pizza slices amongst them so that skinny children gets more pizza as compared to obese ones because pizza is more beneficial for skinny as compared to obese. So might have to find out the optimum number of slices for each child out of the total number of slices so that each child gets maximum possible nutrients. A more complex version could incorporate more features like age, overall health, blood sugar content, physical activity index, daily calorie consumption and others.

A similar example might be to find out the optimal value of fuel to be allocated to each vehicle if we have a total of 100 gallons. Features might be distance they have to travel, mpg, driver competency, engine horse power etc so that all of them might travel the maximum distance possible.

So can we achieve a task like this with machine learning/deep learning approaches? If not what are the hurdles achieving this?

The example you describe looks more like an optimization problem, possibly related to operations research but I'm not sure.

Naturally ML can be used to approximate the optimal solution(s) for this kind of problem. For example genetic algorithms are a common way to find the "approximately best" solution to an optimization problem.

This kind of problem can also sometimes be formalized as a regression problem, depending on the exact constraints to satisfy.


Image segmentation training set labeling I am new to pytorch and Deep learning. I am trying to do image segmentation.
But , I am stuck at how to label training set images.

Can anyone please help me ?

This is one of my training image

I have two kinds of plants here - one is weed and another one is a good crop. I need to label them.

Can anyone tell me how can I do this ?

If you have the labeled training data with you then, you will have to label each pixel in the training set, compute the loss and backpropagate it. The link given below should help.

Suprevised Image Segmentation


Many things behave differently in high dimensional space It turns out that many things behave very differently in high dimensional space. The below paragraph is picked from a book. I need extra help to understand.
The book says,

  if you pick a random point in a unit square (a $1\times 1$ square), it will have only about a $0.4\%$  chance of being located less than $0.001$ from a border( In other words, it is very unlikely that a random point will be extreme along any dimension). But in a $10,000$-dimensional unit hypercube (a $1\times 1\times 1\times 1\times 1\times 1\ldots \times 1$ cube with ten thousand 1s), this probability is greater than $99.99999\%$. Most points in a high dimensional hypercube are very close to the border.

Q) What is the author trying to convey from the above paragraph?

If you pick two points randomly in a unit square, the distance between these two points will be, on average roughly $0.52$. If you pick two points in a unit 3D cube, the average distance will be roughly $0.66$. Now If you two points picked randomly in a $1,000,000$ dimensional hypercube, the average distance will be about approx $408$?

How is it possible?

Along each direction for a unit cube, we have $2$ boundaries. To be less than $0.01$ from a bounday in a $d$-dimensional unit cube, it is not inside the cube of side length $1-2\times 0.001$ sharing the same centroid as the original cube.

The corresponding volume for that is $(1-2\times 0.01)^d$ where $d$ is the coresponding dimension.

$$\left(1-\left( 1-2\times 0.001\right)^2\right) \times 100 \% \approx 0.3996\%$$

$$\left(1-\left( 1-2\times 0.001\right)^{10,000}\right) \times 100 \% \approx 99.999999798\%$$

When we first computed the answer for $2$ dimension, we might carelessly conclude that most points are not closed to the boundary. The moral of the story is be careful not to generalize what we believe without proof to higher dimension.

Your second question is also known as Hypercube line picking of which its computation is non-trivial. We know that

$$\frac13 \sqrt{d} \le \Delta(d) \le \sqrt{\frac{d}{18}\left( 1+2\sqrt{1-\frac{3}{5d}}\right)}$$


gradient descent for non convex function like $-x^2$ I know how to calculate gradient descent for a convex function where there is only one global minima. Also, I know methods to handle cases where the function is a non-convex function.
What is really bothering me that for a non-convex function like $y = -x^2$, how gradient descent is actually calculated as here descent would go to minus infinity instead of directly converging to global maxima.
Thus, it contradicts the point of getting stuck in a saddle point for a function like $(x^2 - y^2)$.

The calculation of the gradient is the same no matter the optimization algorithm you use. Gradient descent is a method to find (local) minima of a function. If no such minimum exists, any algorithm to find it will render wrong results or never converge. $-x^2$ only has a maximum, so looking for a minimum does not make much sense and to find the maximum you would need to run gradient ascent, i.e. going uphill.

Getting stuck in a saddle point is a different issue. For $f(x,y) = x^2 - y^2$ the gradient is $\nabla f(x,y) = 2[x, -y]^T$ and thus equals zero at $x=y=0$, so gradient ascent/descent would just sit there and not move. But $f(0,0) = 0$ is neither a minimum nor a maximum ($f$ actually doesn't have either of these).


Parallel hyperparameter optimization techniques? Most hyperparameter optimization technique want to evaluate points one by one.
I have an expensive optimization problem, but i can run hundreds of evaluations in parallel.
The dimension of the problem is around 20-30.
My variables are mostly continuous.

Is there any technique with open source, documented implementation available for this kind of problem?

Bayesian optimisation is sequential in the sense that you need to know the value of the function for n point to decide through an acquisition criteria the next point to evaluate.

Maybe you could customize it to your problem so that the acquisition returns not one point but a batch of them, which you distribute at the next step.

You can also use an hybrid method. First run a classic grid search, distributed, and evaluate the function at many many points. Feed all this knowledge (points and objective value at these points) to a classic bayesian optimiser which will pick points one by one and finer tune the optimisation here. Not as optimal as the former, but less implementation work here.


First steps on a new cleaned dataset What is the very first thing you do when you get your hands on a new data set (assuming it is cleaned and well structured)? Please share sample code snippets as I am sure this would be extremely helpful for both beginners and experienced.

I think this is a reasonable question. Here is what I do:

Peak at the first few rows
Visualize the distribution of the features I care about (histograms)
Visualize the relationship between pairs of features (scatterplots)

I downloaded the abalone dataset from the UCI Machine Learning repository here. Let's say I care about how height and diameter can be used to predict whole weight. For completeness, I've included the step of reading the data from file.

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

data = pd.read_csv("abalone.data", header=False)
data.columns = ["sex", "length", "diameter", "height",
                "whole_weight", "shucked_weight",
                "viscera_weight", "shell_weight", "rings"]

Now we can take a peak at the first few rows:

data.head()

Now, I know that the variables I care about are floating point values and they can be treated as continuous. I want to take a look to see how these three variables are distributed:

fig = plt.figure(figsize=(20,5))
plt.subplot(1, 3, 1)
plt.hist(data['diameter'], normed=True)
plt.title("Diameter")
plt.subplot(1, 3, 2)
plt.hist(data['height'], normed=True)
plt.title("Height")
plt.subplot(1, 3, 3)
plt.hist(data['whole_weight'], normed=True)
plt.title("Whole Weight")
plt.show()

Great! Now, I know that diameter and whole weight are skewed left and right (respectively). I also know that there are some outliers in terms of height (which is why matplotlib gives me extra room to the right of the distribution). Finally, I'd like to see if I can find any visual patterns between my predictors and outcome variable. I use a scatter plot for this:

plt.figure(figsize=(15,5))
plt.subplot(1, 2, 1)
plt.plot(data['diameter'], data['whole_weight'], 'o')
plt.title("Diameter vs. Whole Weight")
plt.ylabel("Whole Weight")
plt.xlabel("Diameter")
plt.subplot(1, 2, 2)
plt.plot(data['height'], data['whole_weight'], 'o')
plt.title("Height vs. Whole Weight")
plt.ylabel("Whole Weight")
plt.xlabel("Height")
plt.show()

Here, I see there is a non-linear relationship between diameter and whole weight and I'm going to have to deal with my height outliers. Now, I'm ready to do some analysis!


Doc2vec for text classification task Can I use doc2vec for classification big documents (500-2000 words, 20000 total documents, classication for three classes)? Is it a problem that the documents are large enough and contain many common words? Can I train my data together with Wikipedia articles (using unique tag for every article) for a more accurate calculation word-embedings, or it can't give positive effect?

Can I use doc2vec for classification big documents (500-2000 words,
  20000 total documents, classification for three classes)?

Yes. This amount of documents is considered small to medium. It is worth noting that the original Doc2Vec paper experimented with 75K documents.

  Is it a problem that the documents are large enough and contain many common words?

No, it is OK. Consider the fact that both common and distinctive words are increasing with the size of documents, and distinctive words are what that matters. That is, larger texts are easier to distinguish.

  Can I train my data together with Wikipedia articles (using unique tag
  for every article) for a more accurate calculation word-embeddings, or
  it can't give positive effect?

It depends and is worth the try. If Wikipedia documents are close in format and content to your documents, they could definitely help. In the Doc2Vec paper, authors used 25K labeled documents (IMDb reviews with positive and negative labels) and 50K unlabeled ones for training.

Also, this is a nice tutorial on using doc2vec. It reports 1-2 hours for 100K documents.


Probability of similarity of two clusters For example, there are 4 objects. Each object has some elements. There are 3 clusters. Each element of each object belongs to some cluster. Clustering was done before, and we know only how many elements of each object belong to some cluster.
How we can calculate the porbability of similarity of two objects?

         Cluster1  Cluster2  Cluster3
Object1    500       300        200
Object2    200       200        100
Object3    250       2500       1250
Object4    190       210        300

You can consider that the clusters are defined by a multinomial distribution with 4 classes. Thus you would identify the cluster as a vector of 4 probabilities, one for each class, and the sum of probabilities being 1.

In this form there are some tests and distance functions which can prove to be useful for you.

You can ask questions like "is the cluster 1 is different than the cluster 2?". In order to do that you can employ hypothesis testing, and specifically chi square independence test (wikipedia page here). You select from your table only the first 2 columns, compute the marginal totals, compute the estimated counts as $E_{ij} = rowsum_i * colsum_j / grandtotal$. Denoting with $O_{ij}$ the cell counts you can compute a test statistic with
formula:

$$chi = \sum_{i=1}^4\sum_{j=1}^2\frac{(E_{ij}-O_{ij})^2}{E_{ij}}$$

This is known to has a chi square distribution with $(rows-1)(cols-1)=3$ degrees of freedom. Thus using the cumulative distribution of chi square you can compute the p-value as $1-cdf_{\chi_{3}^2}(chi)$. Now if p-value is smaller than any critical value you choose before test (like 0.05, 0.01 or so), you have a statistical proof that the clusters have a different distribution (this follows from the fact that your procedure tests for the independence of cluster and object distributions, which with short derivation is the same as saying the clusters have the same distribution).

To answer to the same type of question you can also employ G-test (wikipedia page here) or Fisher exact test (wikipedia page here).

Note also that all those tests can't prove that two clusters are the same. The logic behind the tests can be formulated as "we consider the distributions the same, as a natural state, accepted without other proofs, and if we see some exceptionally obvious differences, we can bring that as a proof that they are different indeed".

A different approach would be to build a distance between those two distributions. The advantage of a distance function would be that it could be used for comparisons and eventually some further algorithms based on distances. The drawback of a distance function is that it gives you no hint to answer the first type of questions like "are the differences between 2 distributions small enough in order to be explained by the natural noise in data?". Thus I have two candidates in mind.

First distance is the chi square distance. This type of distance gives a measure to the distance between two normalized histograms (or, put it simply, between two vectors of probabilities of the same size). Thus, if we denote with $cluster_i = (n_{i1}/n_i, n_{i2}/n_i, .., n_{ik}/n_i] = (p_{i1},p_{i2},..p_{ik})$, the distance between cluster i and j would be:

$$d(i,j) = \frac{1}{2}\sum_{k=1}^{4}\frac{(p_{ik}-p_{jk})^2}{p_{ik}-p_{jk}}$$

This distance function has the advantage that it is symmetric regarding $i$ and $j$. To be honest I rarely used it since I did not fully understand it's meaning. In general I use another function, described below.

The second distance function is mutual information (see wikipedia page here). The idea is similar with Kulback-Leibler divergence, however the KL distance is an oriented measure (measures how a distribution can be expressed through another one). The MI distance is a measure of mutual independence between two distributions. In your case the two distributions are two clusters, and the MI measures how dependent they are. I give formula just to illustrate the idea behind it.
$$MI(x,y) = \sum_{x,y}p(x,y)log(\frac{p(x,y)}{p(x)p(y)})$$
You can see from the formula that when the two variables are independent, the quantity inside the logarithm is $1$, and as a consequence, the MI is $0$. As the difference grows, the function also grows. I will not derive what p means, see what I explained already for chi square test, since is very similar.


LOF gives same number of outliers irrespective of parameters I am running lof algorithm for around 100k 2d points. Each time, I run the lof algorithm with different n_neighbours parameter, I get the same number of points as outliers. It's always 10% of the points as outliers. Is this how this algorithm is supposed to work? Why does it occur this way?

As per sklearn documentation, you have default parameter contamination set at 0.1.

contamination : float in (0., 0.5), optional (default=0.1)
The amount of contamination of the data set, i.e. the proportion of outliers in the data set. When fitting this is used to define the threshold on the decision function.

No matter how many times you will run the algorithm with different n_neighbours, 10% of your dataset will be retained as outliers.
Keep in mind that, when running a LOF, you are first executing a k-nearest-neighbour algorithm, which is dependent of the n_neighbours parameter. You then evaluate the local density of your points (respectively with their nearest neighbours), which returns a numerical value. However, you have yet to decide over which threshold value a point should be considered as outlier, and this is precisely where the contamination parameter intervenes. It allows you to calculate the threshold with respect to the contaminated proportion you have specified.
I suggest you visualize the results of the LOF to determine whether your contamination proportion is relevant, or ought to be adjusted.


Making outputs of sub-models the inputs of other models LSTM I'm trying to create a neural network that is composed of different sub-models. Each sub-model have their own inputs, which some of them can be shared between the sub-models.

For example, sub-model 1 could be something like this:

S1 = Input(shape=(1,50),name="Stats")
S2 = LSTM(300,return_sequences = True)(S1)
S3 = Dropout(0.5)(S2)
S4 = LSTM(200,return_sequences = False)(S3)
S5 = Dropout(0.5)(S4)
S6 = Dense(100,kernel_initializer='uniform',activation='relu')(S5)
S7 = Dense(1,kernel_initializer='uniform',activation='relu')(S6)

And then, the output of this sub-model I want it to go as an input of a hidden layer. The thing is that the hidden layer I'm trying to do is also with LSTM and I read in some forums that using keras concatenate can do this.

So, it would go like this:

merge = concatenate([S7,A7,B7,...],name="conca")

At last, I tried to gave this merge object to a new model that would be treated as a "hidden layer".

It would go like this:

hidden_1_1 = Input(shape=(1,6),name="Hidden_Layer_1")(merge)
hidden_1_2 = LSTM(300,return_sequences = True)(hidden_1_1)
hidden_1_3 = Dropout(0.5)(hidden_1_2)
hidden_1_4 = LSTM(200,return_sequences = False)(hidden_1_3)
hidden_1_5 = Dropout(0.5)(hidden_1_4)
hidden_1_6 = Dense(100,kernel_initializer='uniform',activation='relu')(hidden_1_5)
hidden_1_7 = Dense(1,kernel_initializer='uniform',activation='relu')(hidden_1_6)

And I'm getting this error: TypeError: 'Tensor' object is not callable.

So...how can I make that the outputs of the sub-models to be the inputs of the other models?

In an image, what I'm trying to do is something like this, but the thing is that instead of a neuron, each neuron would be a model.

Thank you in advance!

Try removing the Input layer

hidden_1_2 = LSTM(300,return_sequences = True)(merge)
hidden_1_3 = Dropout(0.5)(hidden_1_2)
hidden_1_4 = LSTM(200,return_sequences = False)(hidden_1_3)
hidden_1_5 = Dropout(0.5)(hidden_1_4)
hidden_1_6 = Dense(100,kernel_initializer='uniform',activation='relu')(hidden_1_5)
hidden_1_7 = Dense(1,kernel_initializer='uniform',activation='relu')(hidden_1_6)


Couple PCA plot and clusters to labels I am trying my first 'project' concerning machine learning and I am a bit stuck.
However, I am not sure if it's even possible but here goes my question.

What I want to achieve is clustering user groups based on the amount of visits a user does on a certain website.
So I started out with this feature matrix:

USER    abc.be  abc.be/a    abc.be/b    xyz.be  xyz.be/a
123      0        0           0            0      1
456      1        0           1            0      0
789      2        3           1            0      0
321      1        0           1            0      1
654      1        1           1            1      1
987      0        1           0            3      0

So I got in this example 5 features (my 5 different websites).
So then I used PCA to come to 2 dimensions, so I could plot it and see how it went.

My feature matrix (in my example) is 5 columns * 6 rows.

My PCA matrix is 2 columns * 6 rows.

I came to this plot (please note that this plot uses different data then the example but the idea is the same)

The green points are my PCA points
The red circles are my K-Means centroids.

But the part I am struggling with is this: so I got my clusters (red circles) but how can I use this to say:"Looks like most users go to either site A or site B)?

So how can I couple my clusters to a feature label from my feature matrix?

Or how does one approach this?

Any help is appreciated :)

So how can I couple my clusters to a feature label from my feature matrix?

Principal Components are not intuitive features. What seems common here is to cluster users based on PCs and investigate clusters based on original features afterward i.e. extract different clusters and plot data based on different subsets of features and use different colors for different clusters. It might give you some intution.

These two paths can be seen in many PCA results where the information varies with e.g time.

For instance in your case a beginner user visits less than an old one and the number of new users are most probably higher than old ones so data will be more dense around origin and far it gets from the origin less dense it becomes. Such a phenomenon affects your PCs as well so you'll see some paths along different lines in PC space.

Hope it helps :)


How to handle unseen class feature in test dataset feature   Definition    Key
survival    Survival    0 = No, 1 = Yes
pclass   Ticket class   1 = 1st, 2 = 2nd, 3 = 3rd
sex       Sex           M/F
Age     Age in years

feature notes given below

pclass: A proxy for socio-economic status (SES)
1st = Upper
2nd = Middle
3rd = Lower

age: Age is fractional if less than 1. If the age is estimated, is it in the form of xx.5

Suppose I have dataset same as mentioned above.

How to handle a case where I can have a unseen value for a class. For example : In our dataset we have 3 different value for pclass. But how to handle a case where pclass value may have 4rth class say "elite" which did not appear in training dataset but appeared in the test dataset

Depends on the model.  I would say most of the time the new level is dropped or mapped to a NA level.  If possible, you should try to do a stratified sample for your training set to ensure that you get every level you care about.


Difference between rulefit & skope-rules Python Packages RuleFit is an implementation of a rule-based prediction algorithm based on the rulefit algorithm from Friedman and Popescu (PDF).
skope-rules
Skope-rules is a Python machine learning module built on top of scikit-learn and distributed under the 3-Clause BSD license.
Both the packages generate rules. Do both the packages rely on The RuleFit algorithm by Friedman and Popescu (2008), and implement the same algorithm?

In the Skope rules poster: http://2018.ds3-datascience-polytechnique.fr/wp-content/uploads/2018/06/DS3-309.pdf
They made themselves a differentiation

However, our approach mainly differs in the way that decision rules are chosen: semantic deduplication based on variables composing each rule as opposed to L1-based feature selection (RuleFit).

In my personal opinion and after seeing the semantic deduplication package, I will be quite careful as it made some strong modifications of logical rules by default.