How Sklearn-crfsuit interpret text features As we see here, to build an NER model we can pass text features (parts of the word, pos tag, structure of the word etc.) to Sklearn-CRF. I was wondering how does this package convert the text features to numerical features? What strategy do they use - any specific type of embedding?

CRF models are not DL models, they use text features in the traditional way as categorical data (equivalent to one-hot-encoding). However they are different from other supervised ML methods because they exploit the sequential information of the text.
Typically the text sequence is represented word by word, and each word can have several additional "features" usually represented as columns. For example:
The     DET   the
dog     NOUN  dog
chases  VERB  chase
a       DET   a
cat     NOUN  cat

The real features used by the CRF are computed based on the rules defined before training the model. They can involve the position of the current word or any other position relative to the current one. The features are usually simple binary values, for examples "doc[n-1][2] == 'DET'" would represent the condition "the previous word is a determiner": "previous word" is indicated as n-1 in the sequence, column 2 indicates the POS column. The exact syntax to specify the rules may differ with different CRF implementations but the principle is the same.


Role of mode in bottleneck function in unet network I have come across the script that belongs to a person in kaggle. The snippet is given below.

 def bottleneck(x,filters_bottleneck,mode='cascade', depth=6,kernel_size=(3,3),activation='relu'):
   dilated_layers = []
   if mode == 'cascade':
      for i in range(depth):
        x = Conv2D(filters_bottleneck,kernel_size,activation=activation,padding='same',dilation_rate=2**i)(x)
        dilated_layers.append(x)
      return add(dilated_layers)
  elif mode == 'parallel':
     for i in range(depth):
       dilated_layers.append(Conv2D(filters_bottleneck,kernel_size,
                            activation=activation,padding='same',dilation_rate=2**i)(x))
       return add(dilated_layers)

To understand what the function bottleneck does, A pictorial explanation is given below. The highlighted portion is the bottleneck. This part of the network is between the contracting and expanding paths.

The function bottleneck accepts a parameter mode. Am confused with the parameter mode. Is the parameter mode part of the vocabulary of deep learning, if so, can you help me by providing additional resources to understand. And the same applies to the cascade in mode parameter too.

That mode parameter is simply referring to how (what way) the bottleneck should be, what type of bottleneck to use.

There is nothing fancy to it. There is at least one usage in statistics of mode that I know of, as a way to describe some distributions, but this doesn't have anything to do with it.

As to the difference between cascade and parallel, in the former the layers pass the input to each other (like in the diagram), while in the latter the layers operate on the same input.

In general, cascade refers to the application of operations sequentially (e.g. in a cascade classifier). Parallel means that operations are performed independently and could, theoretically, be ran simultaneously (think about parallel operations for threads). Here, those layers are parallel since they all operate on the same input, and their output is then collected by subsequent layers.


Random Forest in R with only character variables I am new to using random forest in R an my goal is to identify the independent variables which have the highest impact on the dependent variables. I am looking at sales data, and sales is my dependent variable (1 vs. 0) I have other variables which have different levels such as professional status (retired, employed, unemployed), searching for (myself, parent, other) and region (north, west, south), etc...

summary(data) provides me with the information that the class of my variables is character (dependent variable shows min, 1st Qu, Media - so I assume R reads it as continuous?) and I believe that a character variable needs to be factored before I can run the randomForest command. Is there a single command that transforms all character into factors?

My second questions is whether I should remove the id of the customer from my imported table, or whether it will affect the results if I keep it in the RF model?

You can check the class of the class(df$dependent). You are expecting it to be numeric.

To convert multiple columns to factors, you can do something like this

factor_cols <- c("col_1","col_7"),
df[factor_cols] <- lapply(df[factor_cols], as.factor)

If you keep the customer id,  then you will have a problem when applying your model to a new customer.


removing special character from CSV file I read my csv file as pandas dataframe. Originally it's a dict with multiple entries per keys. Its looks like this after reading as pandas dataframe:

aad,"[1,4,77,4,0,0,0,0,3]"
bchfg,"[4,1,7,8,0,0,0,1,0]"
cad,"[1,2,7,6,0,0,0,0,3,]"
mcfg,"[0,1,0,0,0,5,0,1,1]"

so I want to firstly remove the double quotes" symbol from the file and then want to create a new csv file from previous one with consecutive three entries in each row.

aad,1,4,77
aad,4,0,0
aad,0,0,3
bchfg,4,1,7
bchfg,8,0,0
bchfg,0,1,1
cad,1,2,7
cad,6,0,0
cad,0,0,3
mcfg,0,1,0
mcfg,0,0,5
mcfg,0,1,1

from pandas import read_csv, concat
from ast import literal_eval
df = read_csv('file.csv',header=None,names=['name','value'])
split = df.value.apply(literal_eval).apply(Series).set_index(df.name)
part1 = split.ix[:,:2]
part2 = split.ix[:,3:5]
part3 = split.ix[:,6:]
part2.columns=part3.columns=range(3)
stacked = concat([part1,part2,part3])

Note that this yields a different order than what you requested:

aad    1  4  77
bchfg  4  1   7
cad    1  2   7
mcfg   0  1   0
aad    4  0   0
bchfg  8  0   0
cad    6  0   0
mcfg   0  0   5
aad    0  0   3
bchfg  0  1   0
cad    0  0   3
mcfg   0  1   1


GE Predix's machine learning (anomaly detection) capabilities Background:
I am investigating time-series anomaly detection for industrial machine data, and have stumbled upon GE Predix. It seems like a promising tool, however, I am not familiar with their approaches for anomaly detection.

Question:
Does anyone have experience with GE Predix, in the anomaly detection space? How would one rate its anomaly detection capability? What technical approaches (algorithms) does it use? Any other literature about GE Predix that is openly available that I have missed (besides References below)? Are there any available benchmarks (e.g., versus Splunk + Prelert)? Who are its competitors?

References:
1) https://www.predix.com/sites/default/files/predixplatformbrief-march_2016.pdf
2) https://www.predix.com/sites/default/files/GE-Software-Modernizing-Machine-to-Machine-Interactions.pdf

(full disclosure, I do work for GE)

GE itself has been using their GE Predix Anomaly detection algorithms to establish a relationship between anomalies & machine failure for their factories, in an effort to establish a preventive maintenance framework. The algorithms have been heavily tested to catch any point anomalies from a machine data.  Go ahead and test it out for your datasets: The algorithms are available for registered developers (in catalog) to use in their applications.

Technical approaches (algorithms): Univariate/Multi-variate anomaly detection algorithms.  It detect Shift anomalies , Unexpected trends in physical system behaviors, Anomalous patterns & detect changes in relationship among different variables. It also detects data with very high dimensions (>100). Mann-Kendall, Mann-Whitney, Kernel density estimate etc are some of the key algorithms used.


Multiple output size in neural network In the paper "A NOVEL FOCAL TVERSKY LOSS FUNCTION WITH IMPROVED ATTENTIONU-NETFOR LESION SEGMENTATION" the author use deep supervision by outputing multiple outputmask which have different scale.
I do not understand how it can work with regards to the loss function. y_pred and y_true doesnt share the same dimension exept for the final output.
model = Model(inputs=[img_input], outputs=[out6, out7, out8, out9])

the input seems to only be the one with real resolution

I checked the code (https://github.com/nabsabraham/focal-tversky-unet/blob/master/newmodels.py) and I haven't see anything special that would make it works. The loss function doesn't explicitly handle it neither.

As you can see in lines 286-296 in newmodels.py the model can use two different loss functions for the four different outputs.
loss = {'pred1':lossfxn,
        'pred2':lossfxn,
        'pred3':lossfxn,
        'final': losses.tversky_loss}

loss_weights = {'pred1':1,
                'pred2':1,
                'pred3':1,
                'final':1}
model.compile(optimizer=opt, loss=loss, loss_weights=loss_weights,
              metrics=[losses.dsc])

The first three outputs of the model, out6, out7, and out8 uses the loss function, lossfxn, which is given as the third argument to the attn_reg function. In isic_train.py, this happens to be the Focal Tversky Loss. For the final output of the UNet (out9) the Focal Tversky Loss is always used. The total loss for the model is then the weighted sum of the different losses for the four model outputs, which is simply equal to the sum given that all weights are set to 1 in loss_weights.


Should I use a decision tree or logistic regression for classification? I am working on a classification problem. I have a dataset containing equal numbers of categorical variables and continuous variables. How do I decide which technique to use, between a decision tree and logistic regression?
Is it right to assume that logistic regression will be more suitable for continuous variables and that decision trees will be more suitable for both continuous and categorical variables?

Long story short: do what @untitledprogrammer said, try both models and cross-validate to help pick one.

Both decision trees (depending on the implementation, e.g. C4.5) and logistic regression should be able to handle continuous and categorical data just fine. For logistic regression, you'll want to dummy code your categorical variables.

As @untitledprogrammer mentioned, it's difficult to know a priori which technique will be better based simply on the types of features you have, continuous or otherwise. It really depends on your specific problem and the data you have. (See No Free Lunch Theorem)

You'll want to keep in mind though that a logistic regression model is searching for a single linear decision boundary in your feature space, whereas a decision tree is essentially partitioning your feature space into half-spaces using axis-aligned linear decision boundaries. The net effect is that you have a non-linear decision boundary, possibly more than one.

This is nice when your data points aren't easily separated by a single hyperplane, but on the other hand, decisions trees are so flexible that they can be prone to overfitting. To combat this, you can try pruning. Logistic regression tends to be less susceptible (but not immune!) to overfitting.

Lastly, another thing to consider is that decision trees can automatically take into account interactions between variables, e.g. $xy$ if you have two independent features $x$ and $y$. With logistic regression, you'll have to manually add those interaction terms yourself.

So you have to ask yourself:

what kind of decision boundary makes more sense in your particular problem?
how do you want to balance bias and variance?
are there interactions between my features?

Of course, it's always a good idea to just try both models and do cross-validation. This will help you find out which one is more likely to have better generalization error.


Accelerate deep learning model training on several GPUs I have a deep learning model that can be trained in one GPU, however, is very slow. Is there a way to accelerate the training by parallelizing it across several GPUs? How would be the training process?

I can use any framework (pytorch, tensorflow, etc). And I know they accept multiple GPU and how to do it.  I am more interested in how to spread, conceptually, a single model between GPUs to gain performance. I do mostly NLP models with RNN but I am also interested in CNN.

What kind of Framework are you using? If you are using Keras, the newest Version of it supports multi-gpu How can i run a keras model on multi gpus

If you have an RNN network, using GPU wont give you much more performace. Most of the times CPU learning is faster in my experience.

Please provide more informations to your problem.

Update

If you have a more complex architecture in Tensorflow or Keras you can give each process Step in your Model a specific GPU/CPU which this part of the Model will be working on. You can do it with

with tf.device('/gpu:1'):

I think you just have to try it out. It depends on the Model you have( which you did not provide, so the complexity is not known) and on the volume of the data. Is your whole dataset in the RAM or do you generate each batch individually?
If you have several branches in your architecture and they are concatenating at a certain point. Which branch is the slowest?

First of all I would try to train your Model on the CPUs before doing anything else.

This post gives a quite good overview how you can use multi GPU.


Pretrained word embeddings vs model weights I'm trying to understand the relationship between pretrained word vectors and pretrained  weights when using pretrained word vectors in another neural network.

Are the vectors themselves the weights or do the weights need to be transferred in addition to selecting embedded representations? What's the high level conceptual process in using pretrained vectors?

Thanks!

Pre-trained word embedding methods such as word2vec, are used in NLP for the initialization of the first layer of a neural network before training for a new task. It is a single layer of weights (known as embeddings) and any prior knowledge from the word embeddings is only present in the first layer of the network, so the entire network would still need to be trained for a new target task.


Dummy encoding the categorical variables using the changed version of OneHotEncoder This is my code, I was trying to dummy encode the first column of X using OneHotEncoder but it was showing error and the documentation page of OneHotEncoder says that it has been changed and I wasn't able to figure out how I can do it using the changed version. the data type of X is numpy.ndarray.

from sklearn.preprocessing import LabelEncoder, OneHotEncoder
onehotencoder=OneHotEncoder(categorical_features = [0])
X[:, 0]=onehotencoder.fit_transform(X).toarray()

Did you try using

X = pd.get_dummies(X)

?

Otherwise make sure that your X is either a 2D Array or a dataframe for OneHotEncoding to work.


I got 100% accuracy on my test set,is there something wrong? I got 100% accuracy on my test set when trained using decision tree algorithm.but only got 85% accuracy on random forest

Is there something wrong with my model or is decision tree best suited for the dataset provided.

Code:

from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.20)

#Random Forest

from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(n_estimators = 1000, random_state = 42)
rf.fit(x_train, y_train);
predictions = rf.predict(x_test)
cm = sklearn.metrics.confusion_matrix(y_test,predictions)
print(cm)

#Decision Tree

from sklearn import tree
clf = tree.DecisionTreeClassifier()
clf = clf.fit(x_train, y_train)
predictions = clf.predict(x_test)
cm = sklearn.metrics.confusion_matrix(y_test,predictions)

Confusion Matrix:

Random Forest:

[[19937  1]
 [    8 52]]

Decision Tree:

[[19938  0]
 [    0 60]]

There may be a few reason this is happening.

First of all, check your code. 100% accuracy seems unlikely in any setting. How many testing data points do you have? How many training data points did you train your model on? You may have made a coding mistake and compared two same list.
Did you use different test set for testing? The high accuracy may be due to luck - try using some of the KFoldCrossValidation libraries that are widely available.
You can visualise your decision tree to find our what is happening. If it has 100% accuracy on the test set, does it have 100% on the training set
?


A weird result from a recommender system Say there're the top 10 most popular items among 100 sales products and about 100k users regularly purchase items on daily basis.

A = has been purchased by 100k users.
B = has been purchased by 30k users.
C = has been purchased by 20k users.
D = has been purchased by 18k users.
E = has been purchased by 10k users.
F = has been purchased by 8k users.
G = has been purchased by 7k users.
H = has been purchased by 4k users.
I = has been purchased by 3k users.
J = has been purchased by 1k users.

X = never bought by anyone.
Y = never bought by anyone.
Z = never bought by anyone.

So basing on this fact, the training data is going to have more than 50m rows of data like this.

User Id  |  User Name  |  Item Id  |  Item Name  |  label  |  Purchase Date  |
1           Thomas        1           A             true      12, Mar 2019
1           Thomas        1           A             true      13, Mar 2019
1           Thomas        1           A             true      14, Mar 2019
1           Thomas        1           A             true      15, Mar 2019
1           Thomas        2           B             true      11, Mar 2019
1           Thomas        3           C             true      09, Mar 2019
1           Thomas        4           D             true      07, Mar 2019
2           Angelica      1           E             true      12, Mar 2019
.
.
.

The preferences of users will be like this, they might be countless but let's take one example.

Thomas bought A, B, C, D
Angelica bought A, B, C, D
Gloria bought A, B, C, D
Jennifer bought A, B, C, D and I

Using the user based collaborative filtering, it is quite obvious that Thomas, Angelica, Gloria are likely to get the item I as a recommended item because Jennifer likes I item and also has the exact same purchase pattern as the others do.

With this in mind, I was starting to think that if I have another two users who bought the unpopular items X,Y,Z, the predictions on them will result in recommending the unsold items.

So I added dummy data manually before training the model like this.

User Id  |  User Name  |  Item Id  |  Item Name  |  label  |  Purchase Date  |
1           Thomas        1           A             true      12, Mar 2019
1           Thomas        1           A             true      13, Mar 2019
1           Thomas        1           A             true      14, Mar 2019
1           Thomas        1           A             true      15, Mar 2019
1           Thomas        2           B             true      11, Mar 2019
1           Thomas        3           C             true      09, Mar 2019
1           Thomas        4           D             true      07, Mar 2019
2           Angelica      1           E             true      12, Mar 2019
.
.
.
100001      Andrew        24          X             true      19, Mar 2019
100001      Andrew        25          Y             true      19, Mar 2019
100002      Andy          24          X             true      19, Mar 2019
100002      Andy          25          Y             true      19, Mar 2019
100002      Andy          26          Z             true      19, Mar 2019

As I mentioned above, I thought Andrew will get Z as a recommended item because Andrew has a common in the item preference with Andy and he bought Z as well, even if the purchase data for X,Y and Z has a extremely small portion of training data ( only 5 records exist among the 10M records of data ).

But the result was totally unexpected.

Every users have X, Y, Z in their recommended list, although the prediction score is very low compared to the others and what's more puzzling is that Andrew and Andy have no outstanding scores on the unpopular items even if they actually bought them!

I don't know why this happens, do I misunderstand the user based collaborative filtering concept?

Increasing the latent dimension value was the key here.

My recommendation system was implemented via ML.NET. And the framework's default setting for the latent dimension value was 20, which seems pretty small considering the volume of my training data.

Increasing the hidden feature count makes my system perform better, successfully predicting the X,Y,Z items as false candidates for other existing users except  Andy and Andrew. Below is how to set the value. This is based on the example code in ML.NET

var pipeline = mlContext.Transforms.Text.FeaturizeText(outputColumnName: "userIdFeaturized", inputColumnName: nameof(MovieRating.userId))
       .Append(mlContext.Transforms.Text.FeaturizeText(outputColumnName: "movieIdFeaturized", inputColumnName: nameof(MovieRating.movieId))
       .Append(mlContext.Transforms.Concatenate(DefaultColumnNames.Features, "userIdFeaturized", "movieIdFeaturized"))
       .Append(mlContext.BinaryClassification.Trainers.FieldAwareFactorizationMachine(
           new string[] {DefaultColumnNames.Features}
           , (e) => { e.latentDim = 200; }) // set custom value here.
           )
       );

To my best knowledge

When decomposing matrices, SVD extracts hidden features from the matrix of the training data and the hidden layers will be directly related to each users and a set of items in the middle of each entities, which is referred as dimension reduction here. I guess too small latent dimension generalizes the variety of recommendation items. That's why, I reckon, increasing the value solves my problem.

Any correction or comment would be appreciated. I definitely don't want to deliver false belief.


Why is the variance of my model predictions much smaller than the training data? I trained a GRU model on some data and then created a bunch of predictions on a test set.
The predictions are really bad, as indicated by a near zero R2 score.
I notice that the variance of the model predictions are much smaller than the actual training data. i.e it seems like the model is overfit to the mean:

But why is this? I made sure to stop training/use hyperparameters where there was no model overfitting, so why are model predictions centred around the mean and less dispersed than the actual variance of the data set?
My model, if it is relevant:
model = Sequential()
model.add(GRU(100, activation='relu', input_shape=(3, 280), recurrent_dropout = 0.2, dropout = 0.2))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse', metrics=['mae'])
model.summary()

# fit model
history = model.fit(X_train, y_train, epochs=40, verbose=1, validation_split=0.33)

The first thing you can do here is to scale your output data before training the model. For instance
$$
\dfrac{x_i - \bar{x}_{train}}{\sigma_{x_{train}}}
$$
If that does not work, then you can have a look at the model and / or training parameters.


How to prune the features using recursive feature elimination? I have some kind of spatial data for nearly 1000 locations and at each location around 5000 features. I am doing neighborhood analysis to identify which features are predominant in local neighborhood.
I read that using recursive feature elimination only strong features can be kept and weak features can be eliminated. For some particular location I get neighborhood data X of numpy shape (14, 5000) where 5000 is features and 14 are local neighbors.
I also get target vector of 14 length having 3 class labels.
I tried to perform following things for pruning the features but getting value error. I am new to ML community and not sure whether doing the right things or not. Any suggestion will be helpful.
from sklearn.model_selection import StratifiedKFold
from sklearn.feature_selection import RFECV
from sklearn.ensemble import RandomForestClassifier

#X.shape (14, 5000)
#target.shape (14,)
#len(np.unique(target)))  3

rfc = RandomForestClassifier(random_state=101)
rfecv = RFECV(estimator=rfc, step=1, cv=StratifiedKFold(5), scoring='accuracy')
rfecv.fit(X, target)
print('Optimal number of features: {}'.format(rfecv.n_features_))

error
ValueError: n_splits=5 cannot be greater than the number of members in each class.

StratifiedKFold tries to keep balance between representation of classes in each fold. You have 5 folds and one or more of your classes have less than 5 instances in the data i.e. they can not be spread among 5 folds. So you can not use it.
That was the minor problem. ML-wise you have very few data points so the right way to cross-validate is leave-one-out.


Link prediction on network embeddings A picture is worth a thousand words, so I decided to illustrate how I imagine the procedure of link prediction on network embeddings. In the figure below the LR model stands for the "Logistic regression model".

Train and test networks are composed of an equal number of positive and negative edges (training examples).

My question is: should I learn embeddings on a network defined by positive edges only or should I also consider negative edges in the network embedding process?

Any idea is appreciated.

The short answer is:
You are correct, in order to compute node embeddings you only need the "true" train edges. These edges span a training graph which is the input we give NE methods to learn the node embeddings. Non-edges are only used in the binary classification problem.

The longer answer is:
The most effective way to compute link predictions from embeddings is via binary classification. The node embeddings first need to be transformed into edge embeddings (e.g. averaging the embeddings of i and j to get the embedding of e=(i,j)) and then solve a binary classification problem. The binary classifier (usually Logistic regression) will take the edge embeddings as examples and try to learn a mapping to their corresponding "label" where these labels are 1 for the "true" edges and 0 for the non-edges. Therefore, in addition to the edge embeddings of "true" graph edges, we need to compute the edge embeddings of a set of non-edges in the graphs (those with label 0). This is what we usually refer to when we talk about negative sampling for LP.

I believe the confusion here comes from the way some popular NE algorithms compute the embeddings. For instance, methods base on matrix factorization directly take the train graph you give as input, compute the adjacency matrix, factorize it and return the node embeddings. Other methods e.g. Deepwalk, Node2vec or LINE have a special way of learning embeddings (Skip-gram from work embedding literature) and require "negative sampling" in the embedding learning process itself. These methods will select, from that train graph you give them, a set of non-edges or paths of non-connected edges to learn which node vectors should be put close together and which should be far. The NE methods will select these samples of non-edges themselves and in different ways, so the user generally doesn't have any control over it.

I hope this clarifies your question.


Types of Recurrent Neural Networks I have a question about types of RNN. Ian Goodfellow in his book Deep Learning writes:

  Some examples of important design patterns for recurrent neural
  networks include the following:

  • Recurrent networks that produce an output at each time step and have
  recurrent connections between hidden units, illustrated in figure
  10.3.

  • Recurrent networks that produce an output at each time step and have recurrent connections only from the output at one time step
  to the hidden units at the next time step, illustrated in figure 10.4

  • Recurrent networks with recurrent connections between hidden units,
  that read an entire sequence and then produce a single output,
  illustrated in figure 10.5.

  CHAPTER 10. SEQUENCE MODELING: RECURRENT AND RECURSIVE NETS

Next I read about RNN article by Andrej Karpathy http://karpathy.github.io/2015/05/21/rnn-effectiveness/ and his describe rnn architecture like relation model.

My question is: description of types RNN by Ian Goodfellow are equal to Andrej Karpathy? If not what it the diffrence beetween this descriptions?

They match up fairly well.

The first Goodfellow description is Karpathy's final "many to many" image. The output at each time step is based on the previous hidden state of the net and the input.

The third Goodfellow description directly corresponds to Karpathy's "many to one" image. This model reads an entire input sequence, and then produces one output.

The only difference is that the second description from Goodfellow's text isn't captured by Karpathy's image. Here's my rendition of what that description states.


How to find appropliate algorithm to bulid a model for natural language based two data What I would like to do
I would like to create a model to infer nationality from name and created the below data frame combining two dataset from Kaggle.
Titanic: Machine Learning from Disaster （input/titanic/train.csv）
titanic-nationalities
    PassengerId Nationality Name
0   1   CelticEnglish   Braund
1   2   CelticEnglish   Cumings
2   3   Nordic  Heikkinen
3   4   CelticEnglish   Futrelle
....

Problem
How can I find algorithm to build a first model using these two data: Nationality and Name?
Since both natural language, so I can understand that it is essencial to make them vectors and this problem would be multi-value classification.
However, I have no idea how to find algorithm to train this dataset.

There's no algorithm intended specifically for this task, you need to design the process yourself (like for most tasks btw).
Given that the goal would be to use a person's name as an indication, I'd suggest you represent a name as a vector of characters n-grams in the features.
Example with bigrams ($n=2$):
"Braund" = [ #B, Br, ra, au, un, nd, d# ]

Intuitively the goal is for the model to find the sequences of letters which are more specific to a nationality. You could try with unigrams, bigrams or trigrams (the higher $n$, the more data you need for training).
Once the names are represented as features this way, you can train any type of supervised model, for example Decision Tree or Naive Bayes.


What is the problem with this architecture ? I am getting error negative dimensions. I want to avoid dense layers and dropouts model = Sequential()
model.add(Conv2D(64, (1, 1), activation='relu', input_shape=(64,64,3)))
model.add(Conv2D(64, (5,5), strides=(2,2), padding='same'))
model.add(MaxPooling2D(pool_size=3, strides=(2,2)))
model.add(Conv2D(64, (1, 1),
                          strides=(1,1),
                          activation='relu',
                          padding='valid' ))
model.add(Conv2D(64, (3, 3),
                          strides=(1,1),
                          activation='relu',
                          padding='valid' ))
model.add(Conv2D(64, (1, 1),
                          strides=(1,1),
                          activation='relu',
                          padding='valid' ))
model.add(Conv2D(64, (3, 3),
                          strides=(1,1),
                          activation='relu',
                          padding='valid' ))
model.add(Conv2D(64, (1, 1),
                          strides=(1,1),
                          activation='relu',
                          padding='valid' ))
model.add(Conv2D(64, (3, 3),
                          strides=(1,1),
                          activation='relu',
                          padding='valid' ))
model.add(Conv2D(128, (1,1),
                strides=(2,2),
                activation='relu',
                padding='valid'))
model.add(Conv2D(128, (3,3),
                strides=(1,1),
                activation='relu',
                padding='valid'))
model.add(Conv2D(128, (1, 1),
                          strides=(1,1),
                          activation='relu',
                          padding='valid' ))
model.add(Conv2D(128, (3,3),
                strides=(1,1),
                activation='relu',
                padding='valid'))
model.add(Conv2D(128, (1,1),
                strides=(1,1),
                activation='relu',
                padding='valid'))
model.add(Conv2D(128, (3,3),
                strides=(1,1),
                activation='relu',
                padding='valid'))
model.add(Conv2D(128, (1,1),
                strides=(1,1),
                activation='relu',
                padding='valid'))
model.add(Conv2D(128, (3,3),
                strides=(1,1),
                padding='valid'))

model.add(AveragePooling2D(pool_size=2, strides=(2,2), padding='valid'))
model.add(Conv2D(1,1,200))
model.add(Flatten())
model.add(Activation('softmax'))

You are using too many layers and you run out of spatial space.

Most of your convolutional layers use "valid" padding, meaning that the convolution is performed only on actual "pixels" without any padding and as a result the spatial dimensions of the output are smaller than the input.

I've marked down where it happens in your script:

model = Sequential()
model.add(Conv2D(64, (1, 1), activation='relu', input_shape=(64,64,3)))
model.add(Conv2D(64, (5,5), strides=(2,2), padding='same'))
model.add(MaxPooling2D(pool_size=3, strides=(2,2)))
model.add(Conv2D(64, (1, 1),
                          strides=(1,1),
                          activation='relu',
                          padding='valid' ))
model.add(Conv2D(64, (3, 3),
                          strides=(1,1),
                          activation='relu',
                          padding='valid' ))
model.add(Conv2D(64, (1, 1),
                          strides=(1,1),
                          activation='relu',
                          padding='valid' ))
model.add(Conv2D(64, (3, 3),
                          strides=(1,1),
                          activation='relu',
                          padding='valid' ))
model.add(Conv2D(64, (1, 1),
                          strides=(1,1),
                          activation='relu',
                          padding='valid' ))
model.add(Conv2D(64, (3, 3),
                          strides=(1,1),
                          activation='relu',
                          padding='valid' ))
model.add(Conv2D(128, (1,1),
                strides=(2,2),
                activation='relu',
                padding='valid'))
model.add(Conv2D(128, (3,3),
                strides=(1,1),
                activation='relu',
                padding='valid'))
model.add(Conv2D(128, (1, 1),
                          strides=(1,1),
                          activation='relu',
                          padding='valid' ))
model.add(Conv2D(128, (3,3),
                strides=(1,1),
                activation='relu',
                padding='valid'))
model.add(Conv2D(128, (1,1),
                strides=(1,1),
                activation='relu',
                padding='valid'))

model.summary() # This is where it happens - The output of this layer is of shape (1,1,128)

model.add(Conv2D(128, (3,3),
                strides=(1,1),
                activation='relu',
                padding='valid'))
model.add(Conv2D(128, (1,1),
                strides=(1,1),
                activation='relu',
                padding='valid'))
model.add(Conv2D(128, (3,3),
                strides=(1,1),
                padding='valid'))

model.add(AveragePooling2D(pool_size=2, strides=(2,2), padding='valid'))
model.add(Conv2D(1,1,200))
model.add(Flatten())
model.add(Activation('softmax'))

You can use the Keras "summary" method to investigate your model. For example, the output from the script I've written here is:

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d_1 (Conv2D)            (None, 64, 64, 64)        256
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 32, 32, 64)        102464
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 15, 15, 64)        0
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 15, 15, 64)        4160
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 13, 13, 64)        36928
_________________________________________________________________
conv2d_5 (Conv2D)            (None, 13, 13, 64)        4160
_________________________________________________________________
conv2d_6 (Conv2D)            (None, 11, 11, 64)        36928
_________________________________________________________________
conv2d_7 (Conv2D)            (None, 11, 11, 64)        4160
_________________________________________________________________
conv2d_8 (Conv2D)            (None, 9, 9, 64)          36928
_________________________________________________________________
conv2d_9 (Conv2D)            (None, 5, 5, 128)         8320
_________________________________________________________________
conv2d_10 (Conv2D)           (None, 3, 3, 128)         147584
_________________________________________________________________
conv2d_11 (Conv2D)           (None, 3, 3, 128)         16512
_________________________________________________________________
conv2d_12 (Conv2D)           (None, 1, 1, 128)         147584
_________________________________________________________________
conv2d_13 (Conv2D)           (None, 1, 1, 128)         16512
=================================================================
Total params: 562,496
Trainable params: 562,496
Non-trainable params: 0


Passing tuples (key, value) into parameterized SQL query in Python NOTE: I am new to this community, so please let me know if I can help you help me!

I built a scraper in python that creates a tuple of (key, value).

I would like to pass both key and value as parameterized variables in an SQL query that updates a table in my sqlite database.

I have exhausted researching online and trial and error and hope you guys can help. Is this even possible?

Example of what I want to achieve:

    cur.execute(''' UPDATE Ads SET ?=?''', (tup.keys(),tup.values()))

Columns in my SQLite3 database (identical to tuple.keys() ):

Marke, Kilometerstand, Erstzulassung, Kraftstoffart, Leistung (PS), Getriebe

Example of my tuple:

{'Marke': 'Weitere Automarken', 'Kilometerstand': '88.888 km', 'Erstzulassung': 'Juli 1963', 'Kraftstoffart': 'Diesel', 'Leistung (PS)': '69 PS', 'Getriebe': 'Manuell'}

Generally speaking, you need a string to be executed that contains placeholders to accept the values contained within the tuple. Also, what you are calling your "tuple" is actually a Python dictionary, but that isn't a problem.

From the sqlite3 documentation, we can see there are two ways to correctly use the execute() method:

an example command is given as:

# This is the qmark style:
cur.execute("insert into people values (?, ?)", (who, age))

# And this is the named style:
cur.execute("select * from people where name_last=:who and age=:age", {"who": who, "age": age})

And the second example, the named style is exactly what would work with your situation. You can try the following, using the keys of the dictionary as the named placeholders:

cmd = """insert into YOUR_TABLE values Marke=:Marke, Kilometerstand=:Kilometerstand, Erstzulassung=:Erstzulassung, Kraftstoffart=:Kraftstoffart, "Leistung (PS)"=:"Leistung (PS)", Getriebe=:Getriebe"""
cur.execute(cmd, your_dictionary)

So each of the keys of your dictionary matches the name of the corresponding column in the target table.

Notice that I put the entire cmd in triple " to make a string literal that can itself contain the " character. I think it will be required, because you have a column name in your table that contains a space: Leistung (PS).


recurrent weights on LSTM Data-gate? (the one with tanh) Looking at the literature, there are 2 distinct approaches to LSTM

Some people use recurrent weights with Input, Forget, Output - notice, their equations don't even mention dataGate, they start from describing the $f$ or $i$ gate (1), Wikipedia: (2)

Lke this:

Other people use recurrent weights with dataGate (z), Input, Forget, Output (1) (2)  (3), (4)

Like this:

Having tried the first approach (I still used standard weights on dataGate but no recurrent weights in that place) seems fine but a little odd - network converges well, but very rarely jumps over local mimima, as if too high learning rate.

Personally I like the second approach (recurrent weights on all 4 entry points), which one do I use?

Edit:

to make things worse, the "LSTM Peephole paper", page 121 describes 3 connections (no peephole for dataGate).

This destroys the equal-sizes of my matrices, because now 3 gates use the Usual, Recurrent and Peephole weights, but the very first gate only uses the Usual weights

There's more than two or three variants with regards to LSTM.

A paper that explores these variants can be read here:

LSTM:A Search Space Odyssey


Incremental Learning with sklearn: warm_start, partial_fit(), fit() I have built an ML model with the goal of making predictions for targets of the following week. In general, new data will come in and be processed at the end of each week and be in the same data structure as before. In other words, the same number of features, same classes for classification, etc.
Instead of re-training the model from scratch for each week's predictions, I am considering applying an incremental learning approach so that past learning is not entirely discarded and the model would (presumably) increase in performance over time. I'm working with sklearn  on Python 3. There were only a handful of posts on StackOverflow regarding this, but many of the answers seem inconsistent (possibly due to updates with sklearn's API?).
The documentation here and here suggests that incremental/online learning is possible with certain ML implementations - implying that the new datasets could be thought of as "mini-batches" and incrementally trained by saving/loading the model and calling .partial_fit() with the same model parameters.

Although all algorithms cannot learn incrementally (i.e. without seeing all the instances at once), all estimators implementing the partial_fit API are candidates. 1
Unlike fit, repeatedly calling partial_fit does not clear the model, but updates it with respect to the data provided. The portion of data provided to partial_fit may be called a mini-batch. Each mini-batch must be of consistent shape, etc. In iterative estimators, partial_fit often only performs a single iteration. 2

However, the documentation here is throwing me off.

partial_fit also retains the model between calls, but differs: with warm_start the parameters change and the data is (more-or-less) constant across calls to fit; with partial_fit, the mini-batch of data changes and model parameters stay fixed. 3
There are cases where you want to use warm_start to fit on different, but closely related data. 3

For the problem I am tackling, ideally model parameters should be adjusted based on cross-validation and new datasets should  be weighted more heavily than old ones due to concept drift. However  ignoring this for now,

In (3), what exactly does "(more-or-less) constant...different but closely related data" mean? Since the data structure of new datasets are the same, should i be calling estimator(warm_start=True).fit(#new df) or estimator.partial_fit(#new df)?
For iterative estimators such as sklearn.linear_model.SGDClassifier, only one epoch is run when using .partial_fit(). If I want $k$ epochs, would calling it on the same dataset repeatedly be the same as calling .fit() with $k$ epochs to begin with?
Do dedicated libraries such as creme offer any advantage for incremental learniing?

Answering my own question after some investigation:

warm_start=True and calling .fit() sequentially should not be used for incremental learning on new datasets with potential concept drift. It simply uses the previously fitted model's parameters to initialize a new fit, and will likely be overwritten if the new data is sufficiently different (i.e. signals are different). After a few mini-batches with large enough sample size (datasets in my case), the overall performance converges to exactly that of simply re-initializing the model. My guess is that this method should be used for the primary purpose of reducing training time when fitting the same dataset, or when there is no significant concept drift in new data.
partial_fit on the other hand has an effect and can be used for incremental learning (especially for datasets too large to fit into memory and feeding in mini-batches). However, in datasets with potential concept drift or high noise, it performs worse than disregarding past observations and simply fitting on each dataset without any incremental learning.
For SGDClassifier, calling partial_fit repeatedly makes a difference.


What probability distribution would be more appropriate for monthly rate of going to the store? Part of a model I am making includes the frequency with which people go shopping for a given good (e.g. people on average go to the supermarket some n times a month).  I am trying to figure out what probability distribution is most appropriate for the task.  I think that this can be modeled by a poisson process, because it involves a rate across time, but am unsure if occurrences of shopping are independent of each other in this case. I considered a normal distribution with the mean centered around some rate but am also unsure about that.

I think you were right with your initial consideration of Poisson. I think that you can assume that the frequency with which people go shopping for a given good can be considered independent. Of course this assumption might not be perfect (like maybe you could argue that Person A buys item X every 3 days. However, if there is enough people I don't think this should matter too much).  Overall, sometimes I think if you need to use a distribution to approximate a process, you need to pick the best option and it seems like Poisson is best here.
Also, It can't be normal since its bounded (ie the lowest number of people that can go to a store is 0)


Model the predictive relationship between images Hello fellow machine learners,

We have numerous pairs of 64 x 64 (or other dimensionality) images (maps). In each pair, the first image demonstrates a physical parameter, e.g. wind speed, at each pixel; the second shows another physical or financial parameter, e.g. temperature or insurance loss, at each pixel. We want to model the predictive relationship from the first image to the second. Our previous models typically consisted of characterizing the field using domain-specific knowledge for dimension reduction. Now my colleague wants to explore the possibility of using statistical learning methods alone.

I did a shallow research and found this could fit in a multi-output regression problem. Online forums suggested papers like Melki and Cano's "Multi-Target Support Vector Regression Via Correlation Regressor Chains", and many Python libraries. However I am a little concerned about the extreme dimensionality, i.e. 4096, and the potential failure of exploiting the spatial structure of our problem if we go down that road.

I know CNNs are suitable for image recognition tasks as such. Has anyone encountered a similar problem and attacked it with neural nets? Any suggestion would be much appreciated!

Thank you!

To answer my own question. A naïve approach is to train a convolutional neural net with an output layer of 4096 neurons $x_1,x_2,\ldots, x_{4096}$. Assume values in the response image are $y_1,y_2,\ldots,y_{4096}$. Let the loss function be $L=\sum_{i=1}^{4096}(x_i-y_i)^2$.


Can my neural network learn conditional rules when classifying? I'm concerned that I'm attempting the impossible with my neural network. This is the scenario:
I have a 2D square world. In it, I create five circles of random size and position. I then classify one of them as the correct answer, based on the following rules:

If any circle's radius is > THRESHOLD, I choose the largest circle
Otherwise, I choose the circle with the origin nearest the center

I send the inputs as serial coordinates, like this: [X0, Y0, RADIUS0, X1, Y1, RADIUS1, ...].
The output is a one-hot array, e.g. [0, 0, 1, 0, 0].
I've modeled this in TensorFlow without success. My best scoring result appears to always choose the largest circle, ignoring the else clause of the arbitrary rule.
Am I fundamentally misunderstanding the capabilities of neural networks? I've tried many (many) different configurations (layer counts, node counts, activation functions ... you name it). All of my networks have been feed-forward, so far.
Thanks in advance for any insight!

Here are some details of my network and data:

I have tried with up to 500k cases. I separate 10% for generalization checks after training, and train on the remaining 90% with a 50/50 validation split.
I've tried with the test data weighted 75% toward rule A, 50/50, and 75% toward ruleB.
I've tried 0-10 hidden layers, and neuron counts from 2 to 256 (each hidden layer gets the same number of neurons).
I change the number epochs as time allows, but generally it's 10-100. My longest runs have been several hours (with giant case numbers, and dropouts to prevent overfitting).
I've used batch sizes of 1-50.
I've tried learning rates of 0.0001 - 0.1.
I'm currently using ReLU activation, initializing bias to const(0.1) and kernel w/ heNormal. I have tried several other approaches for all three.
I standardize the inputs to center on zero w/ variance of one.
The loss function is categoricalCrossentropy.
The optimizer is Adam.

Centering the data on zero and scaling to variance of one helps with a lot of classification problems but in this case it would remove information that's needed to solve your problem as I understand it.
Another possible problem is the loss function, which you don't mention at all in your question. I would suggest something that stays fairly high when your neural network is only learning one of the two rules.


How to overcome the class imbalance issue in a text classification problem? The dataset has 5 labels A,B,C,D,E

A,B,C are majority and D&E are minority.The penalty to misclassify D&E are huge. How can I implement a cost sensitive learning. The input to the model will be english sentences like user comments.

Option 1: play with the thresholds for classifying something – perhaps you could set a threshold for classifying something as a minority class should even if it doesn't have the highest probability/score, or a ratio of probabilities/thresholds.

Option 2: upsample your data (take resamples with replacement of the minority classes until they have as many observations as the others).

Option 3: use some cost-sensitive algorithm (e.g. some forms of classification trees) - probably won't translate into very accurate results.


How does Alexa utterance parsing work? What are the basic principles/tools necessary to make something like Alexa utterance parsing?

For reference, Alexa allows a designer to define phrases with "placeholders" that will be filled in. For example, the phrase "what is the horoscope for gemini" would match the underlying model (below), and return Sign=gemini.

what's the horoscope for {Sign}
what is the horoscope for {Sign}
give me the horoscope for {Sign}
tell me the horoscope for {Sign}

To clarify: I am interested in the theory behind how the language model and parsing (with regard to theory and algorithms) work, so I can build my own version.

If you're interested more generally in speech understanding or speech-to-text, some approaches to natural language parsing and speech-to-text use recurrent neural networks or Hidden Markov processes for learning, as well as a number of signal-processing algorithms to extract more data from the input stream that just raw audio. Keep in mind people have spent their whole careers on this work, so it's not a good problem to just pick up and run with unless you're a MS/PhD candidate looking for a capstone/dissertation project. Here's the iconic paper from Bell Labs that inspired a lot of the DFA/HMM solutions. I've yet to find a paper that does a good job of explaining how to actually implement the RNN style solutions, but here's one in case you're interested.

It's likely that Alexa uses some combination of these methods, but I doubt you're going to get any good answer out of anyone here. After all, it's an important Amazon project and it's not like their engineers are going to come on Stack Overflow and start giving away trade secrets.


custom loss in keras, problem with batch size I am trying to create a custom loss function,custom_loss(y_true, y_pred). I understand that y_pred is calculated by my model but I want to deliver, two kinds of y_true, such as y_true1 and ytrue2(these are pre-calculated, not delivered in model.fit(), and used as follows: (y_pred -  y_true1/ y_pred - ytrue2). The problem is, I have batch size mis-match problem, because, my custom y_true1 and y_true2 is created on my total dataset. How to make their batch size as y_pred?

One approach could be:

class CustomLoss():
        def __init__(self, steps_per_epoch):
            super().__init__()
            self.steps_per_epoch = steps_per_epoch
            self.step = 0

        def calc_custom_loss(self, y_true, y_pred):
            y_true1 = get_y_true1(self.step)
            y_true2 = get_y_true2(self.step)

            self.step += 1
            self.step %= self.steps_per_epoch

This way you only need to provide get_y_true1() and get_y_true2() functions which receive step index. That index is used to generate appropriate batch.

To use it in code:

# Assuming there are 100 batches in one epoch ...
custom_loss = CustomLoss(steps_per_epoch=100)
model.compile(loss=custom_loss.calc_custom_loss, ...)

Be carefull if you also provide validation data during training as it might change your self.step value ... If you want to use validation data, you will need to handle the step appropriately.


amount of data required for a good ann mdel I am new to deep learning and started with ANN. I have a dataset with 15 parameters and 2100 rows. r2_score with Multi linear regression and random forest models is around 85% but when i try an ANN with this dataset r2_score is very less, around 32% . Is it because the number of rows are less? If so atleast what number of rows are required to go for ANN. I have added the code too for reference

import pandas as pd

dataset = pd.read_csv('chiller-2_runningdata_withcommon_parameters.csv')

dataset = dataset.drop(['DateTime','delta','KWH','RunStatus','OP Hours'], axis=1)

X = dataset.iloc[:,:-1 ].values
y = dataset.iloc[:,15:16].values
# Splitting the dataset into the Training set and Test set
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)

# Feature Scaling
from sklearn.preprocessing import StandardScaler

sc_X = StandardScaler()

sc_Y = StandardScaler()

X_train = sc_X.fit_transform(X_train)

X_test = sc_X.transform(X_test)

y_train = sc_Y.fit_transform(y_train)

# Importing the Keras libraries and packages
import keras
from keras.models import Sequential
from keras.layers import Dense

# Initialising the ANN
classifier = Sequential()

# Adding the input layer and the first hidden layer
classifier.add(Dense(units = 8, kernel_initializer = 'uniform', activation = 'relu', input_dim = 15))

# Adding the second hidden layer
classifier.add(Dense(units = 8, kernel_initializer = 'uniform', activation = 'relu'))

# Adding the output layer
classifier.add(Dense(units = 1, kernel_initializer = 'uniform'))

# Compiling the ANN
classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])

# Fitting the ANN to the Training set
classifier.fit(X_train, y_train, batch_size = 32, epochs = 500)

# Predicting the Test set results
y_pred = classifier.predict(X_test)
y_pred = sc_Y.inverse_transform(y_pred)
y_pred_train = classifier.predict(X_train)
y_pred_train = sc_Y.inverse_transform(y_pred_train)
y_train = sc_Y.inverse_transform(y_train)

#calculate r2_score
from sklearn.metrics import r2_score
score_train = r2_score(y_pred_train,y_train)
score_test = r2_score(y_pred,y_test)

You seem to use a loss function for classification problems, binary crossentropy, for a regression problem.

Try to change your compile function to something like

model.compile(optimizer='adam', loss='mse', metrics=['mae'])

Also, you can change your fit function to

model.fit(X_train, y_train, epochs=500, validation_data=(X_test, y_test), verbose=2)

to track your accuracy during training. This might help you detect overfitting and change the amount of epochs accordingly.

Don't worry about your amount of data, try understanding the difference between classification and regression problems instead.


What is the correct formula for Jaccard coefficient with integer vectors? I understand the Jaccard index is the number of elements in common divided by the total number of distinct elements. But it seems to be some discrepancy or terminology confusion about Jaccard being applied to "binary vectors", meaning a vector with binary attributes (0, 1), or, "integer vectors", meaning any vector with integer values (2, 5, 6, 8).
There are two formulas depending on the type of elements in the vector?
This answer comments about "binary vectors" which "they can be interpreted as sets of indices with value 1". However there are examples where Jaccard Coefficient is calculated with an integer vectors, so it seems to be valid. Besides, scikit-learn seems to define 3 cases:
Binary vectors
y_true = np.array([[0, 1, 1],
                   [1, 1, 0]])
y_pred = np.array([[1, 1, 1],
                   [1, 0, 0]])

Multilabel cases
(what is a Multilabel case is not defined in the scikit-learn documentation)
"Multiclass problems are binarized and treated like the corresponding multilabel problem"
y_pred = [0, 2, 1, 2]
y_true = [0, 1, 2, 2]

An additional test with a R library which uses the equation form

TP / (TP + FP + FN)

results in an undefined behavior:
library("ClusterR")
pc <- c(0, 1, 2, 5, 6, 8, 9)
tc <- c(0, 2, 3, 4, 5, 7, 9)
external_validation(pc, tc, method = "jaccard_index")
[1] NaN

Is using the set based formula only suitable for binary vectors?
$$
J(A,B) = {{|A \cap B|}\over{|A \cup B|}} = {{|A \cap B|}\over{|A| + |B| - |A \cap B|}}
$$

The Jaccard coefficient (or Jaccard similarity) is defined on two sets $A$ and $B$:
$$
J(A,B) = {{|A \cap B|}\over{|A \cup B|}} = {{|A \cap B|}\over{|A| + |B| - |A \cap B|}}
$$
There is a single definition for this coefficient, but note that Jaccard is a general similarity measure, it is not specifically designed as an evaluation measure. So assuming one wants to use it as an evaluation measure, they need to decide (design) how. More precisely confusion may arise from the following questions:

Which pair of sets does one want to compare?
Which data structure does one use to represent a set?

Let's start with the second question: a binary vector is a common and convenient way to represent (encode) a set. Given a set of possible elements (universe) $U=\{x_1, ...,x_n\}$, any subset $s\subseteq U$ can be represented as a binary vector of length $n$ where the $i^{th}$ boolean is 1 if $x_i\in s$, 0 otherwise. It follows from this representation that the Jaccard coefficient is the number of indexes where both vectors contain 1 divided by the the number of indexes where at least one of the two vectors contain 1.
Example from scikit documentation:
>>> y_true = np.array([[0, 1, 1],
...                    [1, 1, 0]])
>>> y_pred = np.array([[1, 1, 1],
...                    [1, 0, 0]])
>>> jaccard_score(y_true[0], y_pred[0])
0.6666...

The vectors compared are [0, 1, 1] and [1, 1, 1] (meaning for instance sets $\{b,c\}$ and $\{a,b,c\}$), so the result is 2/3.
Multi-label classification is when an instance can be assigned any number of classes. In the linked scikit example they apply the Jaccard coefficient to the two sets of classes for every instance (optionally aggregating the results across instances). This is an example of a design choice as mentioned in my first question above. Note: in the case of the multiclass problem, I don't even understand how they obtain this result for the third value so I can't comment about it.
The formula $TP / (TP + FP + FN)$ is another example of what does one choose as sets. This formula only makes sense for a binary classification task: TP is the intersection of true and predicted positive instances, $TP+FP+FN$ is the union of all positive cases (whether true or predicted). This is why it doesn't work with integer vectors, since these cannot represent the result of binary classification.
Conclusion: one must define which sets they want to compare. The most simple way with integer vectors is to consider them as sets of integers, in this case they could be represented as binary vectors as follows:
[0, 1, 2, 5, 6, 8, 9] -> [1,1,1,0,0,1,1,0,1,1]
[0, 2, 3, 4, 5, 7, 9] -> [1,0,1,1,1,1,0,1,0,1]

But going through the binary representation is not a must, it's only needed if one wants to use a function which takes this format as input. One could as well define their own function using a set as data structure for instance.


How to extract specific information from raw , unstructured text using NLP and Deep Learning? I have data coming from different sources having similar information like the below example where different sources want to specify the age criteria. Is there a NLP or Deep learning based approach which I can use to extract the age rule as shown below from raw unstructured text.

A Criteria
Applicants should be above 21 years of age and up to 65 years or less at the time of maturity.

B Criteria
You are between 25-58 years of age.

C Criteria
Age of the all applicants should be between 21 and 65 .

A Rule
age >=21 AND age <= 65

B Rule
age >=25 AND age <= 58

C Rule
age >=21 AND age <= 65

You can create your own named-entity recognition through a pre-trained model like Spacy. https://spacy.io/usage/linguistic-features#section-named-entities

these keywords in your example should be considered as an age entity

 above 21 years
 65 years or less

try to play its linguistic features to get what you need.
Hope this helps.


Meaning of "TRUE" column in R RandomForest output for Importance()? I want to assess the importance of variables in my model using the Importance() function of R RandomForest package. I have a binary response variable / class and binary feature values.

mytree.rf <- randomForest(class ~ ., data=mydata, ntree=1500,keep.forest=FALSE,importance=TRUE)
importance(mytree.rf)

The output matrix contains the MeanDecreaseAccuracy and MeanDecreaseGini. I understand those two.

My problem is with two other columns in the output. One simply says "TRUE" the other one "FALSE". Neither in the documentation nor online I was able to find an answer what those values are and how they are calculated...

Can anyone help me out?

Cheers

Edit 1: Thanks to Davids answer I realized TRUE and FALSE are my class "names". I still don't understand how the value given in the matrix is calculated though... Can anyone help with that?

Edit 2: Thanks to David again, it turns out the answer is in the documentation. But it cannot be found in the chapter about the importance() function, but rather in the description of objects of class randomForest. Importance is one of these objects.

[...] a matrix with nclass + 2 (for classification) or two (for regression) columns.
For classification, the first nclass columns are the class-specific measures computed
as mean descrease in accuracy. The nclass + 1st column is the mean
descrease in accuracy over all classes. The last column is the mean decrease
in Gini index.

I don't see any columns like that. Here's a simple example:

library(randomForest)
data(iris)
iris.rf <- randomForest(Species ~ ., data=iris, importance=TRUE)
colnames(importance(iris.rf))

# [1] "setosa"               "versicolor"
# [3] "virginica"            "MeanDecreaseAccuracy"
# [5] "MeanDecreaseGini"

If you're seeing columns named "TRUE" and "FALSE" in the matrix returned by importance, my best guess is that those are the names of columns in your training data.


How are the channels handled in CNN? Is it independently processed or fused? Let's assume that we are talking about 2D convolutions applied on images.

In a grayscale image, the data is a matrix of dimensions $w \times h$, where $w$ is the width of the image and $h$ is its height. In a color image, we normally have 3 channels: red, green and blue; this way, a color image can be represented as a matrix of dimensions $w \times h \times c$, where $c$ is the number of channels, that is, 3.

A convolution layer receives the image ($w \times h \times c$) as input, and generates as output an activation map of dimensions $w' \times h' \times c'$. The number of input channels in the convolution is $c$, while the number of output channels is $c'$.

My confusion is that will CNN operate on the fused representation of the data if $c =2$ or 3 or 4 etc? Or does it operate on each channel at a time and then stacks the results? Say I have 4 channels each channel is a 2D matrix then would CNN internally form a fusion of the 4 channels and make some sort of a representation?

Let $n$ be a convolutional layer with dimensions $w' \times h' \times c'$. Then each of its $c'$ filters is connected to all $c$ filters (or channels*) of the previous layer.

I find it helpful to look at the number of weights here: A single filter of that convolutional layer $n$ with kernel size $k'\times k'$ will have $c \times k' \times k'$ weights. And since layer $n$ has $c'$ of these filters it has a total of $c \times k' \times k' \times c'$ weights.

In a toy example with a 3 channel input layer followed by a conv. layer with 5 channels this would look as follows (not showing the bias here for simplicity):

As you can see from the drawing each feature map of the conv. layer receives all input channels as an input (and the same would apply if this was not an input layer but a conv. layer with 3 feature maps).

*Note that it does not make a difference whether the previous layer is a conv. layer too or the input layer - in the first case you call its depth "filters" and in the second you call it "channels" but that does not change how it is connected to the following conv. layer.


How do I perform multi-step forecasting on LSTM trained with multiple observations of a sequence? I have a trained sequence-to-sequence LSTM that predicts a person's velocity trace (Δx, Δy) given their position (x, y) as they walk along a path. I trained the LSTM using 200 examples of different people walking along the path. Now, I would like to use multi-step forecasting to predict the future velocity of several new people after they have walked 1/3 of the path. Is this possible? If so, how can I implement this in Python or MATLAB?

Train = 200 sequence observations, 2 input features (x, y), time lengths of observations vary

Test = 5 partial sequence observations (1/3 of the sequence), I would like to predict the last 2/3 of the sequence for each person.

ConvLSTM, keras has an implementation, therefore you can go with Python itself.

If you want multiple outputs from the LSTM, you can have look at return_sequences and return_state feature in LSTM layers. Default values for them are None, But if you give True you can get multiple outputs for each timestep, and for everyone.

This excellent blog post helped me understand them.


Multi-label classification based on single-label dataset I'm looking for a solution to detect different moods/styles expressed by an image. Unfortunately, there is no multi-labeled dataset for this task.

The scenario of defining a multi-label classification model based on single labeled data doesn't seem too absurd to me but still I couldn't find any publications or other sources addressing this problem.

So I'm very thankful for any hint how scenarios like this could be tackled (e.g. deriving a multi-labeled dataset from available single label datasets in a weakly supervised manner).

I would train it using the loss appropriate for multi-label (sigmoid activation/binary cross-entropy loss instead of softmax activation/categorial cross-entropy loss).
The model will give you probabilities per label, so it's up to you to decide how to interpret them in your application.

Having said that, some multi-label problems have mostly single labels, much fewer double labels, and almost no labels beyond that. For problems like this, you could try a single-label model and see if it works better than a multi-label one. Or have an ensemble :)


How to add more weight to certain features? I have extracted features from two types of signals. Prior to merging them to create one feature vector, I have computed an importance score of every feature within that type of signal.
I would like to weight the features according to those scores. Would the best way to do this be by multiplying every feature with its score and then concatenate the features of both signals, and should the data be normalized again after multiplication? Or, is there a different way to assign feature weights based on some precomputed score. (I have used multihead self attention to compute scores that I want to use as weights, for every feature within the modality). The model that I will use after merging the features, will be GRU.

The features should be first normalized before doing any kind of combination.
There are two ways to combine: concatenation and addition. You can try these both. Addition is only possible if the features sizes are same, though.


Fast k-means like algorithm for $10^{10}$ points? I am looking to do k-means clustering on a set of 10-dimensional points.  The catch: there are $10^{10}$ points.
I am looking for just the center and size of the largest clusters (let's say 10 to 100 clusters); I don't care about what cluster each point ends up in.  Using k-means specifically is not important; I am just looking for a similar effect, any approximate k-means or related algorithm would be great (minibatch-SGD means, ...).  Since GMM is in a sense the same problem as k-means, doing GMM on the same size data is also interesting.
At this scale, subsampling the data probably doesn't change the result significantly: the odds of finding the same top 10 clusters using a 1/10000th sample of the data are very good.  But even then, that is a $10^6$ point problem which is on/beyond the edge of tractable.

k-means is based on averages.

It models clusters using means, and thus the improvement by adding more data is marginal. The error of the average estimation reduces with 1/sqrt(n); so adding more data pays off less and less...

Strategies for such large data always revolve around sampling:

If you want sublinear runtime, you have to do sampling!

In fact, Mini-Batch-Kmeans etc. do exactly this: repeatedly sample from the data set.

However, sampling (in particular unbiased sampling) isn't exactly free either... usually, you will have to read your data linearly to sample, because you don't get random access to individual records.

I'd go with MacQueen's algorithm. It's online; by default it does a single pass over your data (although it is popular to iterate this). It's not easy to distribute, but I guess you can afford to linearly read your data say 10 times from a SSD?


turn off parts of features in a neural network? Suppose I have a neural network which accepts two sets of features as inputs and generates corresponding outputs, for instance, generate average final grade from: 1. working hours for N students in a class, 2. mid-term grade for N students in the same class.  During training, the neural network is trained with many different classes. What I would like to during inference is to turn off one set of the features (i.e. feed only working hours as inputs) and predict outputs using the same neural network.  Obviously, setting the mid-term grades to be all 0s would not be a good option. I wonder if anyone know what would be a good way to do this?

Thanks!

As far as I know, you cannot do that.
First :

  Obviously setting the mid-term grades to be all 0s would be be a good
  option

No, actually that is a really bad option, neural networks do not understand magical parameters, meaning, if you put a 0 there it will think that the mid-term grades is 0, therefore it will probably give you a very low final grade prediction for you.

My advise, if you are very interested in doing that, create boolean factors, training inputs:

working_hours
mid-term_grade
mid-term_grade_present

mid-term_grade_present should be a 1 when you have the mid-term_grade training data, and 0 when you do not have it. BTW, yes, generate training data without the mid-term_grade.

If you are working with non linear machine learning algorithms that should be enough, if you are using linear algorithms you should one hot encode the mid-term_grade_present and multiply it by the mid-term_grade, ending with something like this:

working_hours
mid-term_grade_present_1 * mid-term_grade
mid-term_grade_present_0 * mid-term_grade


Efficient solution of fmincg without providing gradient? I'm working on multiclass logistic regression model with a large number of features (numFeatures > 100). Using a Maximum Likelihood Estimation based on the cost function and gradient, the fmincg algorithm solves the problem quickly. However, I'm also experimenting with a different cost function and do not have a gradient.

Is there a good way to speed up the calculation process? E.g., is there a different algorithm or fmincg setting that I can use?

If you do not have a gradient available, but the problem is convex, you can use the Nelder-Mead simplex method. It is available in most optimization packages, for example in scipy.optimize.


Pandas dataframe, create columns depending on the row value I get a csv that if you read it, looks like:

import pandas as pd
df = pd.DataFrame([['de,ch,fr', '1,2,3'],['fr,ch,dk', '3,4,5']], columns=['countries', 'numbers'], index=['abc', 'bcd'])

I want to make it look like this:

df = pd.DataFrame([[1,2,3,0], [0,4,3,5]], columns=['de_number', 'ch_number', 'fr_number', 'dk_number'], index=['abc', 'bcd'])

Meaning exploding the countries column and getting for every value in index the number for every country in a separate column. I have the list of all the countries for this dataframe beforehand (meaning that I knew beforehand that I'm going to have the values ['de', 'ch', 'fr', 'dk'])

Is there a nice clean way of doing it? Everything that comes into my mind is quite messy.

First we use DataFrame.explode to unnest your lists to rows.

Then we use DataFrame.pivot_table to pivot your dataframe from rows to column to get your desired result:

dfn = df.assign(countries=df['countries'].str.split(',')).explode('countries')
dfn['numbers'] = df.assign(numbers=df['numbers'].str.split(',')).explode('numbers')['numbers']

dfn = (
    dfn.pivot_table(index=dfn.index,
                    columns='countries',
                    values='numbers',
                    aggfunc=lambda x: x,
                    fill_value=0)
       .rename_axis(None, axis='columns')
       .add_suffix('_number')
)

Output

    ch_number de_number dk_number fr_number
abc         2         1         0         3
bcd         4         0         5         3


Sklearn Aggregating Multiple Fitted Models Into A Single Model? (binary classification) My problem context:

dataset too big to fit into memory.
binary classification [0,1]
30 csv files in a directory with exactly 30,000 samples (rows)
each file contains 15,000 0 class and 15,000 1 class (no unbalance)
model is xgboost.XGBClassifier()

If I iterate through these 30 files, fit a model, and pickle the model to disk; is it possible to aggregate these 30 fitted models into a single model? I am aware of sklearn.ensemble.VotingClassifier but that is used for:

sklearn.ensemble.VotingClassifier
Soft Voting/Majority Rule classifier for unfitted estimators.

Despite the downvote, the question is clear, and a common one I'm sure most stumble across after doing machine learning work for some time.

The goal was to make a stronger predictive model from multiple trained models.

Quote from my question:

  is it possible to aggregate these 30 fitted models into a single
  model?

Answer: yes but there's no good functionality that allows you to do this in sklearn.

Verbose answer:

Imagine you have 30 CSV files that contain 15,000 0 class and 15,000 1 class samples. In other words, an equally balanced number of binary responses (no class imbalance). I generated these files myself because 1) the size of the data I'm working with is too big to fit into memory (point #1 of my original question) and 2) contains 97% class 0 and 3% class 1 (large class imbalance). My goal is to see if it's even possible to distinguish between a 0 or a 1 if the class imbalance issue was removed from the equation. If there were distinguishing features found, I'd want to know what those features were.

To generate each batch, I grabbed 15,000 1 responses, and randomly sampled from the 97% 15,000 more samples, joined into one dataset (30,000 samples total), then shuffled them at random.

I then went through each batch and trained an XGBClassifier() using optimal parameters found from GridSearchCV() for each. I then saved the model to disk (using Python's pickle capabilities).

At this point, you have 30 saved models.

/models/ directory looks like this:

['model_0.pkl', 'model_1.pkl', 'model_10.pkl', 'model_11.pkl', 'model_12.pkl',
 'model_13.pkl', 'model_14.pkl', 'model_15.pkl', 'model_16.pkl', 'model_17.pkl',
 'model_18.pkl', 'model_19.pkl', 'model_2.pkl', 'model_20.pkl', 'model_21.pkl',
 'model_22.pkl', 'model_23.pkl', 'model_24.pkl', 'model_25.pkl', 'model_26.pkl',
 'model_27.pkl', 'model_28.pkl', 'model_29.pkl', 'model_3.pkl', 'model_4.pkl',
 'model_5.pkl', 'model_6.pkl', 'model_7.pkl', 'model_8.pkl', 'model_9.pkl']

Again, going back to the original question - is it possible to aggregate these into a single model? I wrote a pretty basic python function for this.

def xgb_predictions(X):
  ''' returns predictions from 30 saved models '''

  predictions = {}
  for pkl_file in os.listdir('./models/'):
    file_num = int(re.search(r'\d+', pkl_file).group())
    xgb = pickle.load(open(os.path.join('models', pkl_file), mode='rb'))
    y_pred = xgb.predict(X)
    predictions[file_num] = y_pred

  new_df = pd.DataFrame(predictions)
  new_df = new_df[sorted(new_df.columns)]

  return new_df

It iterates through each file in a directory of saved models and loads them one at a time casting its own "vote" so-to-speak. The end result is a new dataframe of predictions.

Since this new dataset is smaller (X.shape x 30), it will fit into memory. I iterate through each batch file calling xgb_predictions(X), getting a dataframe of predictions, add the y column to the dataframe, and append it to a new file. I then read the full dataset of predictions and create a "level 2" model instance where X is the prediction data and y is still y.

So to recap, the concept is, for binary classification, create equally balanced class datasets, train a model on each, run through each dataset and let each trained model cast a prediction. This collection of predictions is, in a way, a transformed version of your original dataset. You build another model to try and predict y given the predictions dataset. Train it all in memory so you end up with a single model. Any time you want to predict, you take your dataset, transform it into predictions, then use your level 2 trained model to cast the final prediction.

This technique performed much better than any other single model did or even stacking with UNTRAINED prior models.


Why is xgboost so much faster than sklearn GradientBoostingClassifier? I'm trying to train a gradient boosting model over 50k examples with 100 numeric features. XGBClassifier handles 500 trees within 43 seconds on my machine, while GradientBoostingClassifier handles only 10 trees(!) in 1 minutes and 2 seconds :( I didn't bother trying to grow 500 trees as it will take hours. I'm using the same learning_rate and max_depth settings, see below.

What makes XGBoost so much faster? Does it use some novel implementation for gradient boosting that sklearn guys do not know? Or is it "cutting corners" and growing shallower trees?

p.s. I'm aware of this discussion: https://www.kaggle.com/c/higgs-boson/forums/t/10335/xgboost-post-competition-survey but couldn't get the answer there...

XGBClassifier(base_score=0.5, colsample_bylevel=1, colsample_bytree=1,
gamma=0, learning_rate=0.05, max_delta_step=0, max_depth=10,
min_child_weight=1, missing=None, n_estimators=500, nthread=-1,
objective='binary:logistic', reg_alpha=0, reg_lambda=1,
scale_pos_weight=1, seed=0, silent=True, subsample=1)

GradientBoostingClassifier(init=None, learning_rate=0.05, loss='deviance',
max_depth=10, max_features=None, max_leaf_nodes=None,
min_samples_leaf=1, min_samples_split=2,
min_weight_fraction_leaf=0.0, n_estimators=10,
presort='auto', random_state=None, subsample=1.0, verbose=0,
warm_start=False)

Since you mention "numeric" features, I guess your features are not categorical and have a high arity (they can take a lot of different values, and thus there are a lot of possible split points). In such a case, growing trees is difficult since there are [a lot of features $\times$ a lot of split points] to evaluate.

My guess is that the biggest effect comes from the fact that XGBoost uses an approximation on the split points. If you have a continuous feature with 10000 possible splits, XGBoost consider only "the best" 300 splits by default (this is a simplification). This behavior is controlled by the sketch_eps parameter, and you can read more about it in the doc. You can try lowering it and check the difference it makes. Since there is no mention of it in the scikit-learn documentation, I guess it is not available. You can learn what XGBoost method is in the their paper (arxiv).

XGBoost also uses an approximation on the evaluation of such split points. I do not know by which criterion scikit learn is evaluating the splits, but it could explain the rest of the time difference.

Adressing Comments

Regarding the evaluation of split points

  However, what did you mean by "XGBoost also uses an approximation on the evaluation of such split points"? as far as I understand, for the evaluation they are using the exact reduction in the optimal objective function, as it appears in eq (7) in the paper.

In order to evaluate the split point, you would have to compute $L(y,H_{i-1}+h_i)$ where $L$ is the cost function, $y$ the target, $H_{i-1}$ the model built until now, and $h_i$ the current addition. Notice that this is not what XGBoost is doing; they are simplifying the cost function $L$ by a Taylor Expansion, which leads to a very simple function to compute. They have to compute the Gradient and the Hessian of $L$ with respect to $H_{i-1}$, and they can reuse those number for all potential splits at stage $i$, making the overral computation fast. You can check Loss function Approximation With Taylor Expansion (CrossValidated Q/A) for more details, or the derivation in their paper.

The point is that they have found a way to approximate $L(y,H_{i-1} + h_i)$ efficiently. If you were to evaluate $L$ fully, without insider knowledge allowing optimisation or avoidance or redundant computation, it would take more time per split. It this regard, it is an approximation. However, other gradient boosting implementations also use a proxy cost functions to evaluate the splits, and I do not know whether XGBoost approximation is quicker in this regards than the others.


Formal definition for parameter setting in data mining context While reading this material on decision trees, I came across the following statement:

  The construction of decision tree classifiers does not require any domain knowledge or parameter setting, and therefore is
  appropriate for exploratory knowledge discovery.

I have couple of doubts related to this statement. First one is the definition of parameter setting in this context and another one is how being without parameter setting is apt for exploratory knowledge discovery?

First part about parameter setting regards to the fact that you don't need to define any parameters of the model, which is the case in per se linear regression which takes the form of ŷ =θ0+θ1x where $θ_0,θ_1$ are parameters. Decision trees create those n-dimensional boundaries for future classification (regression) based on certain criteria.

If you are familiar with statistical inference, a good analogy would be good old parametric vs non-parametric tests.


Proper derivation of dz[1] expression for backpropagation algorithm For backpropagation algorithm, is it true to have weight, $w$ transposed in the expression of $dz^{[1]} = w^{[2]T}dz^{[2]} * g^{[1]'}(z^{[1]}) $ ? Could anyone show me why $w^{[2]T}$ instead of just $w$ ?

I have read other articles, but the transposed dimension of $w$ is ignored when chain derivation is applied.

Screenshot from Andrew Ng deeplearning coursera course video:

For backpropagation algorithm, is it true to have weight, $w$ transposed in the expression of $dz^{[1]} = w^{[2]T}dz^{[2]} * g^{[1]'}(z^{[1]}) $ ?

Yes it is correct. You can show it is with some different arguments:

Checking correct dimensions

Although this is not a robust theoretical argument, checking dimensions is actually what I do in practice when I am confused when implementing NNs.

Here's the rough logic of a dimension check on the equation in your question:

$W^{[2]}$ is a $n^{[2]} \times n^{[1]}$ matrix for the forward propagation to work.
$dz^{[2]}$ is a $n^{[2]} \times 1$ column vector (because it is the gradient of cost function with respect to $z^{[2]}$, so it has to have the same dimension as it)
$dz^{[1]}$ is a $n^{[1]} \times 1$ column vector
It is $dz^{[2]}$ that is being multiplied - so the multiplying matrix must be $\text{something} \times n^{[2]}$.
The output needs to be same dimensions as $dz^{[1]}$ - so the matrix must also be $n^{[1]} \times \text{something}$.
That means the multiplying matrix must have dimensions $n^{[1]} \times n^{[2]}$ - which $W^{[2]T}$ has (and no other component of the network could match)

Theory from item-by-item calculation, converted to matrix math

But why is it $W^{[2]T}$ and not some other matrix with the desired dimensions? For that we need to go back to the chain rule. Note that the expression you have written is actually two steps of the chain rule combined - first we derive $da^{[1]}$ from $dz^{[2]}$, then we derive $dz^{[1]}$ from $da^{[1]}$. It is that first step that introduces terms from $W^{[2]}$, so I will just show that. Also, deriving this directly using matrix notation is more complex, so we'll just drop into using the index values, and use the "official" partial derivative notation.

To calculate $da^{[1]}$ for a specific $i^{th}$ neuron, you have to sum over the gradients $dz^{[2]}$ for all the neurons it links to:

$$ \frac{\partial J}{\partial a_i^{[1]}} = \sum_j \frac{\partial J}{\partial z_j^{[2]}} \frac{\partial z_j^{[2]}}{\partial a_i^{[1]}} = \sum_j \frac{\partial J}{\partial z_j^{[2]}} W_{ij}^{[2]} $$

Doing this for each $i$ value in turn gets your the full $da^{[1]}$ vector.

Compare this to how the forward propagation works in the same network using the same notation and indexing:

$$z^{[2]}_j = b^{[2]}_j + \sum_i W_{ij}^{[2]} a_i^{[1]}$$

And you can see that the index used to drive the sum is different between forward and backward calculations:

When we write $W^{[2]}_{ij}$, then $i$ is index of matrix row, $j$ is index of matrix column from $W^{[2]}$.
When multiplying forward, we sum over $i$, i.e. each row of $W^{[2]}$ multiplied by each matching element of $a^{[1]}$, and this is summed up.
When you look at the gradient, you can see the equivalent sum is over $j$.
To turn that gradient calculation into a normal matrix multiply (just so the matrix notation works, and you don't have to write out the sum) you have to swap rows and columns - which is what a transpose is.

This is partly a definitions and notation issue. We could define a different kind of matrix multiply that worked with sums column-by-column. But this is not usually done, because multiplying by the transpose does the same thing.


How/What to initialize the hidden states in RNN sequence-to-sequence models? In an RNN sequence-to-sequence model, the encode input hidden states and the output's hidden states needs to be initialized before training.

What values should we initialize them with? How should we initialize them?

From the PyTorch tutorial, it simply initializes zeros to the hidden states.

Is initializing zero the usual way of initializing hidden states in RNN seq2seq networks?

How about glorot initialization?

For a single-layer vanilla RNN wouldn't the fan-in and fan-out be equals to $(1 + 1)$ which gives a variance of $1$ and the gaussian distribution with $mean=0$ gives us a uniform distribution of $0$s.

for-each input-hidden weight
  variance = 2.0 / (fan-in +fan-out)
  stddev = sqrt(variance)
  weight = gaussian(mean=0.0, stddev)
end-for

For single layer encoder-decoder architecture with attention, if we use glorot, we'll get a very very small variance when initializing the decoder hidden state since the fan-in would include the attention which is mapped to all possible vocabulary from the encoder output. So we result in a gaussian mean of ~= 0 too since stdev is really really small.

What other initialization methods are there, esp. for the use on RNN seq2seq models?

It is important to clear up the difference between hidden state initialization and weight initialization. Glotrot (Xavier), Kaiming etc. are all initialization methods for the weights of neural networks.

Since your question is asking about hidden state initialization:
Hidden states on the other hand can be initialized in a variety of ways, initializing to zero is indeed common. Other methods include sampling from Gaussian or other distributions. In relation to RNN's this defines what a RNN starts with as its 'memory'. Two common approaches seem to be either a noisy initialization (from some sort of distribution or a random number generator), or a learned initialization.

To synthesize the link above; initializing hidden states with zeros can lead to the network learning to adapt from a zero hidden state, rather than minimizing the loss for a long sequence (it follows that this is more of a problem for short sequences). If there are enough sequences it can make sense to have the initial state be a trained variable that is a function of the error during back propagation.


What's wrong with RF/SVM with word embedding (GloVe)? I searched many times in google for examples on word embedding (specifically GloVe) with Random forest and I couldn't find any single example. For GloVe, it was all either LSTM or CNN. Maybe there's something I don't understand, but why only these 2 algorithms are used with GloVe, what's wrong with the other algorithms?

Usually these are highly dimensional matrices, and all of the information can not be modeled by less-complex models like RF and SVM.


Word2Vec vs. Sentence2Vec vs. Doc2Vec I recently came across the terms Word2Vec, Sentence2Vec and Doc2Vec and kind of confused as I am new to vector semantics. Can someone please elaborate the differences in these methods in simple words. What are the most suitable tasks for each method?

Well the names are pretty straight-forward and should give you a clear idea of vector representations.

The  Word2Vec Algorithm builds distributed semantic representation of words. There are two main approaches to training, Distributed Bag of Words and The skip gram model. One involves predicting the context words using a centre word, while the other involves predicting the word using the context words. You can read about it in much detail in Mikolov's  paper.

The same idea can be extended to sentences and complete documents where instead of learning feature representations for words, you learn it for sentences or documents. However, to get a general idea of a SentenceToVec, think of it as a mathematical average of the word vector representations of all the words in the sentence. You can get a very good approximation just by averaging and without training any SentenceToVec but of course, it has its limitations.

Doc2Vec extends the idea of SentenceToVec or rather Word2Vec because sentences can also be considered as documents. The idea of training remains similar. You can read Mikolov's Doc2Vec paper for more details.

Coming to the applications, it would depend on the task. A Word2Vec effectively captures semantic relations between words hence can be used to calculate word similarities or fed as features to various NLP tasks such as sentiment analysis etc. However words can only capture so much, there are times when you need relationships between sentences and documents and not just words. For example, if you are trying to figure out, whether two stack overflow questions are duplicates of each other.

A simple google search will lead you to a number of applications of these algorithms.


Should I remove features that occur very rarely to build a model? I am trying ML techniques in language processing.
I have got 3000 short texts and I extract features(words and phrases) from all of them and build a vocabulary. I end up with 6000 od features and most of them occurs once or twice.
So for example from texts:

0: One text here
1: Another text there

I got
    One Another  Text  Here  There   Target
  0   1       0     1      1     0     True
  1   0       1     1      0     1    False

So if word "one" occured once I have got as a column and it is False for all other 2999 texts.
Should I drop these columns? Or should use different technique?
This amount of columns make me some problems becuase it takes a lot of time to build a classify.

Short answer:

yes, if they occur so rarely, they can only lead to overfit, so it's better to ignore them as features.

Longer answer:

Usually one puts all those unique occurrence in a single feature/token, and that's the way I suggest you to proceed. So if you have two features that appears only once, you can join them to create a feature that appears twice.

But anyway, 3000 short text are too few to create a NLP model that can work well.
To mitigate your lack of data, you can use pre-trained word embeddings like the ones available here. Doing so you can also keep most of the single occurring words/features, because in those word embeddings they already have a defined semantic meaning.


How to Access a Text File from IBM DSX Scala-Spark Notebook I have recently faced problem in accessing text files that I have uploaded from my local computer to the IBM DSX Cloud Object Storage so that I can use it in my Spark-Scala notebooks. It appeared to me that the general configuration of the platform is focused on CSV format; so is most of the help material. In fact, I figured a way around when I changed the extension of my .txt file to .csv. Through this post and the answer followed, I am trying to share my experience with others.

This post is useful, if:

You have created a IBM DSX notebook with “Language=Scala” and “Spark version=2.1 OR 2.0″[did not try with other versions]
AND You are trying to access a text file [did not try other formats] that you have uploaded from your personal computer to the IBM Cloud Object Storage

In order to access files that are in Cloud Object Storage, you need to provide the path of the file in the form of a CloudObjectStorage instance. Add following lines, after the file credentials, to your code:

var soc = new CloudObjectStorage(sc, credentials, null, “bluemix_cos”) var myFile = sc.textFile(soc.url(“your_bucket”,”data.txt”))

In the second code line above, “your_bucket” is the value of “BUCKET” in the file credentials.

There is one last thing we need to do before we are good to go. The CloudObjectStorage constructor accepts credentials in a way that is for the CSV files. Luckily the main difference between the two formats is the HashMap key names; hence, we can easily manage this problem by changing the above credentials manually, as mentioned below:

“IBM_API_KEY_ID” as “apiKey”

“IAM_SERVICE_ID” as “serviceId”

“ENDPOINT” as “endPoint”

“IBM_AUTH_ENDPOINT” as “iamServiceEndpoint”

For further details, you can visit the link below:
http://rehmans.org/2018/01/31/how-to-access-a-text-file-from-ibm-dsx-scala-spark-notebook/


How to learn spam email detection? I want to learn how a spam email detector is done. I'm not trying to build a commercial product, it'll be a serious learning exercise for me. Therefore, I'm looking for resources, such as existing projects, source code, articles, papers etc that I can follow. I want to learn by examples, I don't think I am good enough to do it from scratch. Ideally, I'd like to get my hand dirty in Bayesian.

Is there anything like that that? Programming language isn't a problem for me.

First of all check this carefully. You'll find a simple dataset and some papers to review.

BUT as you want to start a simple learning project I recommend to not going through papers (which are obviously not basic) but try to build your own bayesian learner which is not so difficult.

I personally suggest Andrew Moore's lecture slides on Probabilistic Graphical Models which are freely available and you can learn from them simply and step by step.

If you need more detailed help just comment on this answer and I'll be glad to help :)

Enjoy baysian learning!


Check similarity between time series I have time series of parameters A, B, C and D. All of them are under influence of the same major conditions, but each one has minor differences. They are placed in different locations, A, B, C are in local1 and D is in local2.

I would like to know which one (A, B, C) has the major similarity to D. How should I approach this issue?

Sounds like a job for Dynamic Time Warping, there are implementations in Python and R.