Logistics regression with polynomial features vs neural networks for classification I am taking Andrew Ng's Coursera class on machine learning. He mentions that training a logistic regression model with polynomial features would be very expensive for certain tasks compared to training a neural network. Why is that though? I mean, when we talk about neural networks we're usually looking at a model with a very large number of parameters so why would logistic regression be more computationally expensive?

PS: Here's some context (at the beginning of an exercise on neural networks):

  In the previous part of this exercise, you implemented multi-class logistic regression to recognize handwritten digits. However, logistic regression cannot form more complex hypotheses as it is only a linear classifier (You could add more features, such as polynomial features, to logistic regression, but that can be very expensive to train).

I expect what he's referring to is the combinatorial explosion in the number of terms (features)
as the degree of the polynomial increases.

Let's say you have $N$ measurements/variables you're using to predict some other variable.  A $k$th degree polynomial of those $N$ variables has $ N+k \choose k$ terms (see here).  This increases very quickly with $k$.

Example:

Say we have N = 100 variables and we choose a third degree polynomial, we'll have
 $ 103\choose3$ = 176,851 features. For a five-degree polyomial it goes to:
$105 \choose 5$ = ~96 million features.  You'll then need to learn as many parameters as you have features.

Compare to a NN:

Compare this to using a fully connected NN
say we choose K fully connected hidden layers with $M$ units each.  That gives:
$ (NM) + (K-1)(MM) + M$ parameters.  This is linear in $K$ (though the $MM$ term attached to it might be big).

For N = 100 variables again, two hidden layers, and 350 features nodes per layer we get 157,580 parameters - less than we'd need for logistic regression with third degree polynomial features.

Representational power

(Added this section after seanv507's great comment)  * See caveat below

The argument above was just a numbers game - how big of a NN can you get while still having the same parameter count as logistic regression with polynomail features.  But you're getting at something when you say in your question:

  I mean, when we talk about neural networks we're usually looking at a
  model with a very large number of parameters so why would logistic
  regression be more computationally expensive?

Well said.

Efficiency

Neural nets are universal function approximators, and we know that polynomials can approximate functions too.  Which is better? What should one use? I'd bet that, for a given parameter "budget", a NN could better approximate "more" functions than a polynomial with the same number of parameters.  It wouldn't surprise me if there was theory to back it up, but I don't know it off hand.  seanv507's statement

  One possibility is to assume that the nonlinear activation function is quadratic...and identify what range of polynomials the NN could represent.

is an interesting idea.  Empirically, NN's have done better for many hard tasks than polynomial representations, and that's pretty strong evidence.

*Caveat

As seanv507 says - this is the hard part.  The above statements won't always be true - I'd argue they're probably mostly true.  If a low-degree polynomial basis nicely and reliably separates your classes, then it's probably worth using / trying polynomial features.


You've really got a classification problem on your hands, not a regression problem. Your target is not continuous, and Pearson correlation measures a relationship between continuous variables really. That's problematic enough to start.

Low correlation means there's no linear relationship; it doesn't mean there's no information in the feature that predicts the target.

I think you're really looking for mutual information, in this case between continuous and categorical variables. (I assume your other inputs are continuous?) This is a little involved; see https://stats.stackexchange.com/questions/29489/how-do-i-study-the-correlation-between-a-continuous-variable-and-a-categorical

If you're attempting to do feature selection then you could perform a logistic regression with L1 regularization and select features based on the absolute value of their coefficients.


Multi-Output Regression with neural network in Keras I have got an .xlsx Excel file with an input an 2 output columns. And there are some coordinates and outputs in that file such as:
x= 10 y1=15 y2=20
x= 20 y1=14 y2=22 ...
I am trying to do that regression using tensorflow. But somehow i can't manage to do it. I am leaving my code here, I would appraciate it if someone could help! I also have test datas ready as well.

training_data = pd.read_excel(...\training_data.xlsx',sheet_name="i1-o2")

training_data_X = training_data['i1']

training_data_Y = training_data[['o1','o2']]

testing_data = data = pd.read_excel(....\testing_data.xlsx',sheet_name="i1-o2")

testing_data_X = testing_data['i1']

testing_data_Y = testing_data[['o1','o2']]

model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(6, activation='linear')
])

model.compile(optimizer='adam',
              loss='mean_squared_error',
              metrics=['accuracy'])
model.fit(training_data_X,training_data_Y,epochs=10,batch_size=100)

val_loss,val_acc = model.evaluate(testing_data_X,testing_data_Y)
print(val_loss,val_acc)

I found some mistakes:

input data must be numpy objects, not pandas
this Network has 6 output nodes, not 2
the number of layers is completely exagerated IMHO
the Flatten() layer at the beginning is not correct
the way you called ReLU's is not correct

This should be enough:

from tf.keras.models import Sequential
from tf.keras.layers import Dense
from tf.keras.activations import relu

model = Sequential([
    tf.keras.layers.Dense(128, activation = relu),

    tf.keras.layers.Dense(128, activation = relu),

    tf.keras.layers.Dense(2, activation = None)
])

Check if the loss works at this point. Alternatively, you need to write your own custom loss function using Keras backend functions.


Which type of clustering algorithm to use to identify the "same" item in different data sets? I'm trying to find a solution for a data quality problem - specifically, identifying which items in different data sets are used to represent the same things.

As an example, assume that we're a retailer and we buy out a couple of other retailers.  In the process, we also get their systems and databases.  This might lead to some overlap - different systems can represent the same items, customers, etc. in different ways, but with no single unique identifier.

What would the best approach be to determine which rows represent the same thing across data sets in order to come up with a single view of 'unique' entities?

I've done a machine learning course, and I understand the bare minimum.  I believe that the solution to this problem requires a clustering algorithm, but what type?  I may be dealing with a multitude of features in the data - dimensions, names, dates, contact details - and some of those would probably require a higher 'weight' in matching.

Examples (Items):

A:  ABC Notebook, Large, Released: 2018-02-20, 150mm x 100mm.

B:  Notebook (ABC), L, Date: 18/02/2018, 150mm x 100mm

I'd expect these to be treated as the same item.

Examples (Customer):

A:  Doe, Jane, DOB 1970-06-23, 123 ML Ave, F

B:  John Doe, DOB 1971-04-33, 123 ML Avenue, M

C:  J. Doe, Born '71

I'd expected B and C to be identified as the same person, but not A.

For this scenario, I don't expect to come up with something with a 100% accuracy, but I would like to be able to come up with a (narrow) list of possible matches that someone can check.

Can some please point me in the right direction?  Any case studies that I should look at?

A: ABC Notebook, Large, Released: 2018-02-20, 150mm x 100mm.
B: Notebook (ABC), L, Date: 18/02/2018, 150mm x 100mm

It doesn't seem like a clustering problem. Rather, it's more like a text mining problem. E.g., you extract "ABC" and "Notebook" from the names of both A and B, and then decide they are the same. Besides that, there are also some format detection needed, like mapping date string to datetime format, L to Large.
After you extracted these features and constructed a feature matrix, you can simply determine if two entries are different items/people by their commonly existing features in the matrix (since there will be NAs in your matrix). You don't really need a machine learning clustering algorithm.

A: Doe, Jane, DOB 1970-06-23, 123 ML Ave, F
B: John Doe, DOB 1971-04-33, 123 ML Avenue, M
C: J. Doe, Born '71

E.g., A and B are not the same since there are mismatches in DOB and sex. B and C can be treated as same, as they both have 2 features - name and DOB - show up in your data set, and both 2 features matched.
You might need a bit more complex algorithm, since there might be a D: John Doe, DOB 1971-03-12, F, with which you should leave B, C, D alone. But that's all it need, I guess.


Linear regression model with (categorical) predictor variables I used LM model with (categorical) predictor variables on my data in r like this (I have count variable as dependent/target variable):

# Fit linear model to total viewing:
fit <- lm(
  total_viewing_minutes ~ panelist + sex + internet + education,
  data = cln
)

If I understood correctly in r the first category is always the reference one so has by definition a coefficient of 0.

So by looking at the model result, is 68.3 minutes (intercept)  average time viewing of

Panelist0 + sex1 + internet1 + education1

and by taking into account the term Panelist1
the 68.4 + 17.6 is avg time viewing of

 Panelist1 + sex1 + internet1 + education1

is that correct interpretation?

Your interpretation is correct. I would also add that if Panelist1 increases by 1 unit and the other predictors are held constant, then the average increase in the response variable is 17.557 minutes.


Graduate Degree Choices for Data Science I'm currently finishing up a B.S. in mathematics and would like to attend graduate school (a master's degree for starters, with the possibility of a subsequent Ph.D.) with an eye toward entering the field of data science. I'm also particularly interested in machine learning.

What are the graduate degree choices that would get me to where I want to go?

Is there a consensus as to whether a graduate degree in applied mathematics, statistics, or computer science would put me in a better position to enter the field of data science?

Thank you all for the help, this is a big choice for me and any input is very much appreciated. Typically I ask my questions on Mathematics Stack Exchange, but I thought asking here would give me a broader and better rounded perspective.

Why not do an MSc in ooh... Data Science?

I wrote a quick review of UK Data Science Masters' offerings recently. That should help you get an idea what is offered. Mostly they are mashups of stats and computing, but there are specialisms (health, finance for example) that might interest you.

Note that list was compiled for courses that have already started, so some of those courses might not be available for starting next October, or have different syllabus contents.


Python strategies for handing categorical variables I am currently working on a binary classification task where the class is imbalanced.

I have the following categorical attributes with different levels:

time_slot: 8 levels
product_type: 3 levels
state: 40 levels
due_day: 6 levels (Mon - Sat)
lead_time: numerical in days (0-100)

Now, I am planning to use three algorithms to start with:

Logistic Regression, Decision Tree and Random Forest

I am confused as to what sort of encoding strategy is best when it comes to categorical variables?

LabelEncoder, OneHot, BinaryEncoding?

Also, I am thinking of creating bins for lead_time

any pointers/tips will be useful.

The best option for encoding - OneHot, because if you use Label encoding you indicate that categorical values are comparable(for example label 1 < label 2), which most probably it's not true. One hot encoding create columns for each specific value in the column, moreover, these columns are linearly independent, so you don't create fake order between categorical values. Unfortunately, you got a lot of columns so learning of algorithms could be greedy for time and resource.

More details you will find there


Why ROC value area under curve of two models is different whereas accuracy, precision, recall, f1-score and confusion matrix is same I am applying logistic regression and support vector machines on the extacly same dataset with 70% data for training and 30% for test. Both perform exactly the same have the same precision, recall, f1-score and confusion matrix. The confusion matrix and classification report for both can be seen below.

       [1180    0]
       [  17    0]]

         precision    recall  f1-score   support

   0       0.99      1.00      0.99      1180
   1       0.00      0.00      0.00        17

   accuracy                               0.99      1197
   macro avg          0.49      0.50      0.50      1197
   weighted avg       0.97      0.99      0.98      1197

I applied 10-fold cross validation to both models too both achieve the same mean accracy of 98.57% both failing to achieve any True Positives across all 10 folds, also have same number of TP,TN,FP,FN in each of the 10-fold

However, the problem is when I compute the roc area under curve values they are different as seen below:

Logistic Regression: 0.86
Support Vector Machines: 0.23

These values indicate logistic regression is much better as compared to support vector machines although there confusion matrix and classification report is the same I am really confused as all the other reuslts indicated same performance by both models ?? Can someone please help why both get different values of roc whereas everything else is the same.

Those scorings looks strange for me, but beside that you must remember that F1, accuracy, confusion matrix, etc depends on the chosen threshold, while AUC is threshold-independent (it is an integral over all of the thresholds from 0 to 1).

Your models return some probabilities of being a member of class 1. If you choose to label by '1' only those, which have probability higher than 99% (this is a threshold) then you get a recall at almost 100% and a precision near 0%.


Large Scale Personalization - Per User vs Global Models I'm currently working on a project that would benefit from personalized predictions.  Given an input document, a set of output documents, and a history of user behavior, I'd like to predict which of the output documents are clicked.

In short, I'm wondering what the typical approach to this kind of personalization problem is.  Are models trained per user, or does a single global model take in summary statistics of past user behavior to help inform that decision?  Per user models won't be accurate until the user has been active for a while, while most global models have to take in a fixed length feature vector (meaning we more or less have to compress a stream of past events into a smaller number of summary statistics).

The answer to this question is going to vary pretty wildly depending on the size and nature of your data. At a high level, you could think of it as a special case of multilevel models; you have the option of estimating a model with complete pooling (i.e., a universal model that doesn't distinguish between users), models with no pooling (a separate model for each user), and partially pooled models (a mixture of the two). You should really read Andrew Gelman on this topic if you're interested.

You can also think of this as a learning-to-rank problem that either tries to produce point-wise estimates using a single function or instead tries to optimize on some list-wise loss function (e.g., NDCG).

As with most machine learning problems, it all depends on what kind of data you have, the quality of it, the sparseness of it, and what kinds of features you are able to extract from it. If you have reason to believe that each and every user is going to be pretty unique in their behavior, you might want to build a per-user model, but that's going to be unwieldy fast -- and what do you do when you are faced with a new user?


Neural network q learning for tic tac toe - how to use the threshold I am currently programming a q learning neural network tha does not work. I have previously asked a question about inputs and have sorted that out. My current idea to why the program does not work is to do with the threshold value. this is a neural network - q learning specific variable.

basically the theshold is a value that is between 0 and 1, you then make a random number between 0 and 1, if this random number is larger than the threshold then you pick a completely random choice, otherwise the neural network chooses by finding the largest q value.

My question is that with this threshold value, i am currently implementing it as starting at almost 0, then increasing linearly until it reaches 1 by the time the program has reached the final  iteration. Is this correct?

The reason i suspect this is incorrect is that when plotting an error graph from training the neural network, the program doesnt not learn at all, but when the threshold reaches almost 1, it starts to learn very fast, and if you run more iterations after it reaches 1, the all the game sets in the replay memory become the same and the error is basically 0 from their on in.

Any feedback is greatly appreciated and if this question in unclear in anyway just let me know and i will try and fix it. Thank you to anyone who helps out.

You are effectively implementing $\epsilon$-greedy action selection.

The usual way to represent this in RL, at least that I am familiar with, is not as a "threshold" for probability of choosing the best estimated action, but as a small probability, $\epsilon$, of not choosing the best estimated action.

For consistency with RL literature that I know, I will use the $\epsilon$-greedy form, so instead of considering what happens as your threshold rises from 0 to 1, I will consider what happens when $\epsilon$ drops from 1 to 0. It is the same thing. I hope you can either adjust to using $\epsilon$ or mentally convert the rest of this answer so it is about your threshold . . .

When monitoring Q-Learning, you have to be careful how you measure success. Monitoring the behaviour on the learning games will give you slightly off feedback. The agent will make exploratory moves (with probability $\epsilon$), and the results from a learning game might involve the agent losing even though it already has a policy good enough to not lose from the position where it started exploring. If you want to measure how well the agent has learned the game, you have to stop the training stage and play some games with $\epsilon$ set to $0$. I suspect this could be one problem - that you are measuring results from behaviour during training (note this would work with SARSA)

In addition, choosing values that are too high or low for your problem will reduce the speed of learning. High values interfere with Q-learning because it has to reject some of data from exploratory moves, and the agent will rarely see a full game played using its preferred policy. Low values stifle learning because the agent does not explore different options enough, just repeating the same game play when there might be better moves that it has not tried. For Tic Tac Toe and Q-learning I would suggest picking a value of $\epsilon$ between $0.01$ and $0.2$

In fact, with Q-learning there is no need to change the value of $\epsilon$. You should be able to pick a value, say $0.1$, and stick with it. The agent will still learn an optimal policy, because Q-learning is an off-policy algorithm.


How do i calculate prediction probability of a class in Java Weka Api? I am developing a prediction model using Java Weka api. I can predict class for the new instance using the following code:

double predictClass = classifer.classifyInstance(instance)

However, I need class probability instead of class value. Thanks in advance for your support.

Welcome to the community!
You can replace your code by the following code.

double[] prediction=classifier.distributionForInstance(instance);

    for (int k<prediction.length; k++){
    System.out.println("Probability of class "+
      trains.classAttribute().value(k)+
       " : "+Double.toString(prediction[k]));
      }

This loop prints all the four values.
Hope it will help you.


About seq2seq networks I have read that seq2seq is a network, similar to other networks types (CNN, RNN, ...). However, in my opinion it is actually an architecture for RNNs. Isn't that?

For example, when input and output lengths are not equal, LSTM is not applicable. LSTM is an architecture for RNNs. Instead another architecture is used consisting two units: encoder and decoder and they are built with RNNs.

So, is seq2seq a subset of RNN? a sibling of LSTM? Or it is a sibling of RNN and CNN?

seq2seq is a type of problem, that can be solved with LSTM, RNN etc...

Generally one thinks about sequences of data that are inter connected and these relationships in sequences should be modeled (which can be achieved with for example LSTM etc.)


Why this sequential model is not starting? I am using following code:

input_shape = (75, 75, 3)
x = Input(input_shape)
model = BatchNormalization(axis = 3)(x)

Above code works all right. However, following code does not work:

from keras.models import Sequential
input_shape = (64,64,3)
model = Sequential()
model = model.add(InputLayer(input_shape=input_shape))
model = model.add(BatchNormalization(axis = 3))

But at last line, I get error:

AttributeError: 'NoneType' object has no attribute 'add'

If I change to:

model = model.add(Input(input_shape))

I get following error:

TypeError: The added layer must be an instance of class Layer.
Found: Tensor("input_1:0", shape=(?, 64, 64, 3), dtype=float32)

Where is the problem and how can it be solved?

(PS: If you find this question to be interesting/important, please upvote it.)

from keras.models import Sequential
from keras.layers import InputLayer

model = Sequential()
model.add(InputLayer(input_shape))
model.add(BatchNormalization(axis = 3))

This should work. The first error was because the model was getting reassigned. The second error was because 'Input' is a function of 'layers' class but not a class, 'InputLayer' is a class.


Can I use an array as a model feature? Problem
I have data that includes multiple different text inputs as well as floats, categories, etc. Therefore I need to pass several different data types as features, including text which is an int array when tokenized.

Question
Say I tokenize the several text inputs; can I pass the tokenized text array as a feature alongside my floats and categories? If not, how is this done?

Background
When I've done NLP models, my code looks similar to this:

...
tokenizer = Tokenizer(num_words=max_features)
tokenizer.fit_on_texts(df['Stem'])
list_tokenized_train = tokenizer.texts_to_sequences(df['Stem'])
X_train = pad_sequences(list_tokenized_train, maxlen=10)
y_train = df['TotalPValue']
...

So, the text input becomes an array of int tokens padded with zeroes, e.g. [0 0 0 0 0 0 0 1 12 52].

This is not enough to solve my problem. I want to instead use multiple tokenized string inputs and floats as features. I want to first tokenize and pad each text input like above and put them in the same input array, like this: X_train = [[0 0 0 0 0 0 0 1 12 52], [0 0 0 0 0 0 0 42 12 23], 0.0425672].

I want to then start my model like this:

model = Sequential()
model.add(Embedding(max_features, embedding_vector_length, input_length=3))

Will it work if implemented like this?

My attempts
I searched for a while but couldn't find anyone else doing it like this. Surprising to me that I couldn't find anything since it seems like a basic problem.

Just wanted to know if I have the right idea, since - as a beginner -  implementation will cost a lot of time if this isn't the right way of doing it. Thanks so much for the insight!

Found a fantastic answer here that avoids high-level black box libraries.

Essentially, numerical floats are categorized into bins based on boundary values which are determined by their distribution. Text tokens are hashed with column ID, and then concatenated with other columns' hashed tokens via an interaction array. All hashed tokens, whether or not they have in interaction, are fed as inputs into the model.

That's the gist of it.


In which predictive tasks are the ARIMA model useful? I'm working on a project where I have to predict the total quantities sold at the ITEM/DAY level.

As a model, I thought of using an ARIMA (I'm using R), but I wanted to get a confirmation of whether it is a good choice or not. If not, can I get the contexts where the ARIMA model is the most useful?

A popular and widely used statistical method for time series forecasting is the ARIMA model. ARIMA is an acronym that stands for AutoRegressive Integrated Moving Average. It is a class of model that captures a suite of different standard temporal structures in time series data. You can read about it more here.

Based on your problem statement, it definitely looks like ARIMA model can help as you are predictions are based of previous time series data. However, as a rule of thumb in Machine learning, there is no one model that can solve the problem out of the box. There is a saying in statistics that goes something like this :

  All models are wrong, some are useful.

What this implies is, you can start with ARIMA but you need evaluate your model to see if ARIMA can solve the problem for you or you need to explore further.


NLP - How to perform semantic analysis? I'd like to perform a textual/sentiment analysis.

I was able to analyse samples with 3 labels: (positive, neutral, negative) and I used algorithms such as SVM, Random Forest, Logistic Regression and Gradient Boosting.

My script works correctly and with the cross validation I can take the best algorithm among the 4.

I use supervised algorithms with the python function "Countvectorizer"

But my boss typed "NLP" on the internet and looked at some articles.

He told me : "These 3 outputs are not enough, I want a complete semantic analysis that can explain the global meaning of the sentence"

He didn't seem to have a preference between supervised and unsupervised algorithms.

He told me that he wanted an algorithm able to tell that "The company president is behind bars" is equivalent to "the CEO is in jail".

So do you have any idea how one could perform that ? And how to implement it in Python? I guess we need a great database full of words, I know this is not a very specific question but I'd like to present him all the solutions.

What scares me is that he don't seem to know a lot about it, for example he told me "you have to reduce the high dimension of your dataset" , while my dataset is just 2000 text fields.

Thank you very much for your answers :)

With your three labels: positive, neutral or negative - it seems you are talking more about sentiment analysis. This answer the question: what are the emotions of the person who wrote this piece of text?

Semantic analysis is a larger term, meaning to analyse the meaning contained within text, not just the sentiment. It looks for relationships among the words, how they are combined and how often certain words appear together.

To gain a deeper insight into your text, you could read about topics such as:

Semantic Analysis in general might refer to your starting point, where you parse a sentence to understand and label the various parts of speech (POS). A tool for this in Python is spaCy, which words very nicely and also provides visualisations to show to your boss.
Named Entity Recognition (NER) - finding parts of speech (POS) that refer to an entity and linking them to pronouns appearing later in the text. An example is to distinguish between Apple the company, and apple the fruit.
Embeddings - finding latent representation of individual words e.g. using Word2Vec. Text is processed to produce a single embedding for individual words in the form of an n-dimensional vector. You can then compute similarity measures (e.g. cosine similarity) between the vectors for certain words to analyse how they are related.
Lemmatisation - this method reduces many forms of words to their base forms, which means they appear more regularly and we don't consider e.g. verb conjugations as separate words. As an example, tracking, tracked, tracker, might all be reduced to the base form: track.

Your next step could be to search for blogs and introductions to any of those terms I mentioned.

Here is an example parse-tree from spaCy:

Reducing dimensions

This is something that would then refer to the vectors, which describe each of your words. Generally, the Word2Vec vectors are something like 300-dimensional. You might want to visualise the words, plotting them in 2d space. You can try a method like t-SNE, which will map the 300d vectors to 2d space, allowing nice plots showing relationships, while retaining as much of the original relationships described in the 300d space. There will, of couse, be some information loss, but you could not have visualised the 300d vectors in the first place!

Using the vectors for your words, you can compute things like the similarity (on a scale between 0 and 1) between president and CEO is something like 0.92 - meaning they are almost synonyms!


What's wrong with my gradient descent? (simple, one variable) # calculating the cost function
theta = np.array([[0], [0]])
def h(X, theta):
    return np.matmul(X, theta)

def cost(X, y, theta, m):
    return 1/(2* m) * np.sum((h(X, theta)-y)**2)

print(cost(X, y, theta, m))
# 1-variable gradient decent
cost_vals = []
def grad_decent(X, y, theta, m, alpha):
    theta1 = theta[0, 0]
    theta2 = theta[1, 0]
    for i in range(0, iter):
        cost_vals.append(cost(X, y, np.array([[theta1], [theta2]]), m))
        theta1 = theta1 - alpha/m * np.sum((h(X, theta)-y) * X[:, 0])
        theta2 = theta2 - alpha/m * np.sum((h(X, theta)-y) * X[:, 1])
        # print(f"Iteration {i}: ", theta1, ", ", theta2)

    return np.array([[theta1], [theta2]])

As one can see I'm doing gradient descent with only one feature, i.e. I want to fit a line of the form theta1 + theta2 * x to my data. I know that for such small feature sizes one would definitely use the normal equation, but I wanted to implement it with gradient descent. My learning rate alpha is 0.01 and I'm following the machine learning course of andrew ng on coursera. However, instead of converging, my cost values increase so there is some mistake in the algorithm which I can't find. X is of the shape 97x2 where the first column is just ones since there is only one feature but I want to fit a polynomial with two coefficients.

You set your model parameters using np.array([[0], [0]]), meaning that your parameter array is of shape (2, 1). Changing this to np.array([0, 0]) (with shape (2,)) decreases the loss over time as expected using a learning rate $\alpha$ of 0.01. Looking more closely at the difference of using the two, I see that your way returns way higher losses (exactly 100x). Using a smaller $\alpha$, e.g. 0.0001, with your implementation also works:

Therefore the main reason of your implementation is your choice of $\alpha$, which is too high, preventing the model to reach the optimum.
EDIT: The issues is twofold, the first is indeed the shape or your theta array. The second has to do with the fact that you are using the theta array in your for loop which always contains the original theta parameters, as you are not storing and using the updated theta1 and theta2. The following code seems to give a good result of $\theta_1 \approx -3.2$ and $\theta_2 \approx 1.12$ using 1000 iterations with $\alpha=0.01$
import numpy as np

# load and transform data
data = np.genfromtxt("data.txt", delimiter=",")
X = np.stack((np.ones(data.shape[0]), data[:,0]), axis=-1)
y = data[:,1]

m = X.shape[0]
iter = 1000

# calculating the cost function
theta = np.array([[0.3], [0.5]])
def h(X, theta):
    return np.matmul(X, theta)

def cost(X, y, theta, m):
    return 1/(2* m) * np.sum((h(X, theta)-y)**2)

print(cost(X, y, theta, m))
# 1-variable gradient decent
cost_vals = []
def grad_decent(X, y, theta, m, alpha):
    theta1 = theta[0, 0]
    theta2 = theta[1, 0]
    for i in range(0, iter):
        theta = np.array([theta1, theta2])
        cost_vals.append(cost(X, y, np.array([theta1, theta2]), m))
        theta1 = theta1 - alpha/m * np.sum((h(X, theta)-y) * X[:, 0])
        theta2 = theta2 - alpha/m * np.sum((h(X, theta)-y) * X[:, 1])
        # print(f"Iteration {i}: ", theta1, ", ", theta2)

    return np.array([[theta1], [theta2]])

grad_decent(X, y, theta, m, 0.01)
# 1740.4098044406999
# array([[-3.20070025],
#        [ 1.12320526]])


Time series binary classification probability smoothing Problem
Suppose we have trained binary classifier and want to predict value of [x1, ..., x5] with associated timestamps [t1, ..., t5]. We get the prediction as following: [0.25, 0.99, 0.1, 0.75, 0.79].
Assume that I have the domain knowledge to say that probability of positive class must not change abruptly. Jumps like from 0.99 at t2 to 0.1 at t3 cannot occur in real application.
Questions

Can I enforce smooth output constraint on (any/some) classifier?
Does applying moving average on the prediction probability to smooth it make sense?

You can use a total variation regularizer (https://en.wikipedia.org/wiki/Total_variation_denoising), it's a penalty for abrupt changes of neighbor values. It's usually used for images, that's why its TF version (https://www.tensorflow.org/api_docs/python/tf/image/total_variation) operates with 4D tensors, but if you're writing your model in pytorch for instance, it's easy to implement that regularizer yourself. Also possibly you don't need it if you've got enough data and target values there are already smooth. Your ML algorithm would just learn that smoothness from data, the only 2 cases you'd need it is when your dataset is small or when your training targets aren't smooth, but testing targets should be smooth.


Node Importance in scikit learn I'm trying to understand exactly how feature_importances in scikit-learn's RandomForestClassifier works. I managed to find this helpful link explaining most of the process: https://towardsdatascience.com/the-mathematics-of-decision-trees-random-forest-and-feature-importance-in-scikit-learn-and-spark-f2861df67e3
I just have two questions about the equation for ni_j (equation for node importance, first equation in the section on feature importance):

What is meant by weighted number of samples reaching node j? Are they running all the training samples through the tree and counting how many reach node j? If so, why is the word 'weighted' necessary?
Can we be mathematically certain that ni_j >= 0?

Edit: Looking at the code for forest.py, I see this rather mystifying function:
def feature_importances_(self):
        """Return the feature importances (the higher, the more important the
           feature).
        Returns
        -------
        feature_importances_ : array, shape = [n_features]
        """
        check_is_fitted(self, 'estimators_')

        all_importances = Parallel(n_jobs=self.n_jobs,
                                   backend="threading")(
            delayed(getattr)(tree, 'feature_importances_')
            for tree in self.estimators_)

        return sum(all_importances) / len(self.estimators_)

I can see it's calculating a feature importance at each tree, and then taking the mean over all trees. But how it manages to calculate the feature importance at a tree without calling any functions to count samples or compute impurities, I don't understand.

weighted number of samples reaching node j

We have to compare the Parent to two children, then the children's impurity must be weighted.
Let's say the Parent has 100 samples and impurity=5
Children has the same data as [20, 2] and [80, 4]
Dip = 5 - (0.2 * 2 + 0.8 * 4)
=> 5 - 3.6
=> Dip = 1.4
"Weighted" means applying the appropriate ratio i.e. 0.2 and 0.8 here.


PCA, why variance of eigen values is measure of its utility? Source - Murphy, 12.3

Heuristic for assessing applicability of PCA.

Let the empirical covariance matrix Σ have eigenvalues λ1≥λ2≥···≥λd>0, with mean λ. Explain why the variance of the eigen values, σ2=(1/d)∑(λi−λ)2 is a good measure of whether or not PCA would be useful for analysing the data (the higher the value of σ2 the more useful PCA)

What is important is that the covariance matrix eigenvalues represent the variance explained by each component of the PCA.

PCA can be considered useful when it allows to reduce the dimension without removing too much variance from the dataset; that is, when there are a lot of low eigenvalues. As the sum of the eigenvalues is fixed because it is the total variance of the dataset (this is a property of the eigenvalues decomposition of a matrix: the trace of a matrix is equal to the sum of its eigenvalues), low eigenvalues for some eigenvectors means that there are other high eigenvalues to compensate; that is a high variance in the eigenvalues.

In a more mathematical way, if we denote by $\sigma_{tot}^2$ the total variance over the dataset (not the variance of eigenvalues), we know that the mean of eigenvalues will be $\frac{\sigma_{tot}^2}{n}$. As all eigenvalues are non-negative (property of the covariance matrix), and based on the trace conservation property, we deduce that all eigenvalues are lower or equal to $\sigma_{tot}^2$. Then, the convexity of the square function implies that maximum variance among eigenvalues is reached when one of the eigenvalues is equal to $\sigma_{tot}^2$, and all other are null. Hence, eigenvalues maximum variance is reached when the first component wears the total dataset variance, which is obviously the optimal PCA case (all features reduced to a single dimension).

Oppositely, the case with minimal variance of eigenvalues is the one where all eigenvalue are equal, meaning that there is no particular "principal" component (the projection of the dataset onto any direction gives the exact same variance). If I'm not mistaken in the math, this is only possible when the covariance matrix is a multiple of the identity matrix, meaning the dataset is already completely uncorrelated. In that case, PCA is simply useless.


Approach to creating a user profile in music web application I am working on a use case, and I'm unsure of the best way to proceed: in order to analyze the behavior of users of a web-based music application, we retain all songs each has played since 2009. We store this information in flat files, each containing the songs played in a day. Each file contains 50M of lines, and we have  19M users. Our entire song catalogue consists of 35M tracks.

The format of these files is as follows:

id-user | country | id-artist | id-track

Question: I would like to represent each user by the songs he or she has played; this profile would be used by the production site. Does anyone have a suggestion for what would be the best way to process the whole chain?

The first question is: what to do you want to see in user profile?

Top-10 tracks, top-10 artist by user?
How many tracks/artists a user listens to in a day on average (may be, in last month)?

May be you want to get some general information related to the whole user base:

Which artist/track is the most popular among users from different countries (top-N of them)?

The second: you store and want to make aggregates for millions of records. It's not a text file deal. Make a database. Create a table id-user | country | id-artist | id-track. Create another tables with some aggregates from #1, update it regularly and diplay on the front end.


Loss Function - Decreasing a lot at beginning of the epoch I noticed something when looking at the output of the verbose. When I train my model, in the early part of the epoch (first 20 %), the loss is decreasing a lot. And then in the rest of the epoch (last 80%), the loss is very stable and doesn't change that much until the next epoch. It does the same thing.

I build a model that is training a kind of large dataset (60000 entries). I am using Keras and Tensorflow and my model is just a simple regression model with conv2d and dense layers. I am trying to get the best loss function possible. The loss function that I am using is just a simple Mean Squared Error (MSE). I am also using Adam optimizer.

I don't understand how the loss is decreasing a lot at the beginning of the epoch and less for the rest of the training. Should I reduced the size of my database ? Is it considered overtraining then ?

Is there a way to speed up this process (like early stopping but just for the epoch) ? As I see it most of the training is done for the first part of the data. Maybe I am wrong.

Have you already plotted the training and validation loss over multiple epochs? What you would expect is something that looks like an exponential decay.

Chances are your models learns the mean of the target variable in the first few batches, which reduces the MSE-loss already considerably and only later learns the subtle differences in your data. You can try to use different batch sizes as well.

Are you generally happy with the performance on your model? Does it perform well on the test/validation set? If this is the case there might be no reason to be worried about.


How to maintain accuracy with more classes? I'm working on a CNN model (implemented by keras) that detects landmarks from images. Every landmark (for example Acropolis, Big Ben, Eiffel Tower etc) is as a separate class and is divided into own directory.
The training dataset contains 1600 images, validation 400 images and test 500 images. The training process has 80 iterations.

I noticed that with two or three classes the accuracy is quite good (3 classes => accuracy: 0.8353) but by every next class it's getting to decrease.

Is it so that adding more classes increases features and complexity? What could be the solution for that? Adding more layers or data? I have tried to add more layers but it's rather worsened the result than made it better. Is it because of vanishing gradient? Adding more data is big problem because I just can't find more unique images.
I also changed other config parameters (optimization algorithm, dropout rate etc) but with no result.

The used config parameters are:

NUMBER_OF_CLASSES = 4
BATCH_SIZE = 32
EPOCHS = 80
DROPOUT_RATE = 0.3
LOSS = 'categorical_crossentropy'
OPTIMIZER = 'rmsprop'
METRICS = ['accuracy']
CLASS_MODE = 'categorical'

And constructed model:

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3), input_shape=input_shape))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2), strides=2))

model.add(Conv2D(64, kernel_size=(3, 3),))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2), strides=2))

model.add(Conv2D(128, kernel_size=(3, 3),))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2), strides=2))

model.add(Conv2D(256, kernel_size=(3, 3),))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2), strides=2))

model.add(Flatten())

model.add(Dense(256))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(Dropout(0.3))

model.add(Dense(256))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(Dropout(0.3))

model.add(Dense(NUMBER_OF_CLASSES))
model.add(Activation('softmax'))

--------------
Total params: 721,604
Trainable params: 719,620
Non-trainable params: 1,984

There can be multiple explenations.

But here are some:

Images in train test and valid that are poor for some classes are not representative in train (poperties of these objects are not to be found hence you cant predict on something that you did not atleast partially learn on)- covariate shift

You say adding data is problematic and I understand but just note that more data beats everything else by a mile. Seriously. Hence augment the data, A LOT!. And make sure that for every class properties are representative and to be found in every train test and valid set.

I can start talking about architecture but these are marginal benefits at this point, make sure you have this point right. Without that every discussion about architeture if futile.


How to retrieve the clustering results of rpart I am using rpart package in order to create a segmentation of my data using decision tree. As final result I want to obtain a classification of my data. For exemple, if the rpart devide data into 3 classes, I want to divide my data onto this three subsets such that I know that row n°1 is in subset 1, row n°2 in subset 3, etc. I can't find how could I get this information from the results of rpart object?

tree.res <- rpart(x ~ ., data, method="class", parms=list(split = "gini"))

I know that I can retrieve the results of this classification as follows:

plot(tree)    #plot the tree
text(tree)
printcp(tree) #Displays CP table for Fitted Rpart Object
predict(tree) #displays prediction results of x variable

How can I get the information on the segmentation of my data set? As first attempt I used :

predict(tree, type = "vector")

Is this a correct approach?

I found an answer to my question using the package partykit, I post it to help others.

tree_party <- as.party(tree) #coercing the rpart.object into partykit

tree_fit <- fitted(tree_party) #a two-column data.frame with the fitted node numbers and the observed responses on the training data.

tree_pred <- predict(tree_party, newdata = data, type = "node")


building a model that calculate effect of an event Say I have a bike sharing company and want to know how events affect number of rides per day.
events vary on:

number of coupons sent per user
expiry date
amount for each coupon.

I have created a df that looks something like this:
deploy  duration  avg_temp  rain  register_count   ...   rent_count
   33     131113      23      2.2       12                   33
   523    931143      25      0         322                  756
   63     231153      26      0         111                  124
   123    566363      22      1.2       334                  345

where rent_count is our target variable.
If I want to add columns that describe events(num_coupons, duration, etc...) how should I design columns?
So at the end I want to build a model that would look something like this:
Y = b0 + b1x1^n + .... + bnxn^n.

therefore I can calculate how much event variable is contributing to Y.

This is a classic example of Causality, where we need to find how much one variable is effected by changing other variable.
You can infer the causal graph from data and then select the variables which best describe your model.
Intervention can be done to represent particular event and then check the effect on target variable.
Refer:
Causalnex ,
DoWhy


Keras functional API Layer name not captured with TimeDistributed wrapper class Net2:
    @staticmethod
    def build_cat_branch(inputs,category_size):
        x = TimeDistributed(Dense(category_size))(inputs)
        x = Activation('softmax', name="cat_output")(x)
        return x

    @staticmethod
    def build_t_branch(inputs):
        x = TimeDistributed(Dense(1, activation='relu', name="t_output"))(inputs)
        return x

    @staticmethod
    def build_full_model(timestep_len,hidden_size,category_size,num_features,dropout,rec_drop):
        inputs = Input(shape=(timestep_len,num_features),name="Input")
        bn = BatchNormalization()(inputs)
        lstm = LSTM(hidden_size, return_sequences=True, dropout=dropout, recurrent_dropout=rec_drop,name="LSTM")(bn)
        bn2 = BatchNormalization()(lstm)
        cat_branch = Net2.build_cat_branch(bn2,category_size)
        t_branch = Net2.build_t_branch(bn2)
        model = Model(inputs=inputs,outputs=[cat_branch,t_branch],name="Net2")
        return model

When I try to compile this model I get:

ValueError: Unknown entry in loss dictionary: "t_output". Only expected the following keys: ['cat_output', 'time_distributed_2']

(in my model summary, my layer that I name "t_output" has the name "time_distributed_2" instead)

So, basically, my question is what's going on with the TD wrapper that causes the name attribute to not be part of the object returned by the build_t_branch function?  Clearly, the "cat_output" name is stored correctly, as the loss dictionary recognizes it, but the output layer I have inside a TimeDistributed wrapper is not saving the user defined layer-name.  I know I can get around this by just having all layers defined in a single function without the "branch" functions, but that is besides the point here.  Is this a bug in Keras?  Any way to get around this without the above mentioned fix?

The reason is you should name the TD instead of Dense. Be cautious with the bracket. Change it to:

  x = TimeDistributed(Dense(1, activation='relu'), name="t_output")(inputs)


Data set with multiple tables I want to check an algorithm that learns the connections between different tables in a database.

Where can I find datasets that are compound of multiple tables (i.e. more than 5) that have connections between them?

All I can find on the web (mainly on Kaggle.com) are datasets with 1 or 2 tables.

I'm aware that my question is kind of off topic but still I'm sure it can help people that have the same problem as I do

There are many places:

this one is very good: https://relational.fit.cvut.cz/

there are also:

https://datahub.io/search

https://www.reddit.com/r/bigquery/wiki/datasets


Is there a straightforward way to run pandas.DataFrame.isin in parallel? I have a modeling and scoring program that makes heavy use of the DataFrame.isin function of pandas, searching through lists of facebook "like" records of individual users for each of a few thousand specific pages. This is the most time-consuming part of the program, more so than the modeling or scoring pieces, simply because it only runs on one core while the rest runs on a few dozen simultaneously.

Though I know I could manually break up the dataframe into chunks and run the operation in parallel, is there any straightforward way to do that automatically? In other words, is there any kind of package out there that will recognize I'm running an easily-delegated operation and automatically distribute it? Perhaps that's asking for too much, but I've been surprised enough in the past by what's already available in Python, so I figure it's worth asking.

Any other suggestions about how this might be accomplished (even if not by some magic unicorn package!) would also be appreciated. Mainly, just trying to find a way to shave off 15-20 minutes per run without spending an equal amount of time coding the solution.

Unfortunately, parallelization is not yet implemented in pandas. You can join this github issue if you want to participate in the development of this feature.

I don't know any "magic unicorn package" for this purposes, so the best thing will be write your own solution. But if you still don't want to spend time on that and want to learn something new – you can try the two methods built into MongoDB (map reduce and agg framework).  See mongodb_agg_framework.


Gradient Boosting Tree: "the more variable the better"? From the tutorial of the XGBoost, I think when each tree grows, all the variables are scanned to be selected to split nodes, and the one with the maximum gain split will be chosen. So my question is that what if I add some noise variables into the data set, would these noise variables influence the selection of variables (for each tree growing)? My logic is that because these noise variables do NOT give maximum gain split at all, then they would never be selected thus they do not influence the tree growth.

If the answer is yes, then is it true that "the more variables the better for XGBoost"? Let's not consider the training time.

Also, if the answer is yes, then is it true that "we do not need to filter out non-important variables from the model".

Thank you!

My logic is that because these noise variables do NOT give maximum gain split at all, then they would never be selected thus they do not influence the tree growth.

This is only perfectly correct for very large, near infinite data sets, where the number of samples in your training set gives good coverage of all variations. In practice, with enough dimensions you end up with a lot of sampling noise, because your coverage of possible examples is weaker the more dimensions your data has.

Noise on weak variables that ends up correlating by chance with the target variable can limit the effectiveness of boosting algorithms, and this can more easily happen on deeper splits in the decision tree, where the data being assessed has already been grouped into a small subset.

The more variables you add, the more likely it is that you will get weakly correlated variables that just happen to look good to the split selection algorithm for some specific combination, which then creates trees that learn this noise instead of the intended signal, and ultimately generalise badly.

In practise, I have found XGBoost quite robust to noise on a small scale. However, I have also found that it will sometimes select poor quality engineered variables, in preference to better-correlated data, for similar reasons. So it is not an algorithm where "the more variables the better for XGBoost" and you do need to care about possible low-quality features.


Datasets understanding best practices I am a CS master student in data mining. My supervisor once told me that before I run any classifier or do anything with a dataset I must fully understand the data and make sure that the data is clean and correct.

My questions:

What are the best practices to understand a dataset (high dimensional with numerical and nominal attributes)?
Practices to make sure the dataset is clean?
Practices to make sure the dataset doesn't have wrong values or so?

There are basic things you can do with any set of data:

Validate values (String length tolerance, data type, formatting masks, required field presence, etc.)
Range correctness (Does this seemingly correct data fall within expected ranges of values)
Preliminary processing (If I attempt to analyze this data, can I perform the basics without running into errors)
Preliminary reporting (run a report against a data set and ensure that it passes a sanity test)
Defining null vs. empty vs. zero vs. False for any given column of data
Identifying data that is out of place (numeric values dramatically different than other values in a data set, string values that look like they might be misspelled, etc.)
Eliminating or correcting obviously errant data

Understanding data to identify errors is a whole different ball game, and it is very important.

For instance, you can have a rule that says a serial number must be present in a given data set and that serial number must be alphanumeric with a maximum string length of 255 and a minimum string length of 5.

Looking at the data, you may find one particular serial number value reads "PLEASE ENTER SERIAL"  It's perfectly valid, but wrong.

That's kind of an obvious one, but say you're processing stock data and you had a price range for 1000 stocks that was under a dollar.  A lot of people would not know that a stock price so low is invalid on certain exchanges and perfectly valid on others.  You need knowledge about your data to understand if what you are seeing is problematic or not.

In the real world, you don't always have the luxury of understanding your data intimately.

The way I avoid problems is by leveraging the people around me.  For small data sets, I can ask someone to review the data in it's entirety.  For large ones, pulling a set of random samples and asking someone to do a sanity check on the data is more appropriate.

Further, questioning the source of the data and how well that data source can be trusted is imperative.  I often have multiple conflicting sources of data and we create rules to determine the "source of truth".  Sometimes one data set has great data in a given aspect, but other data sets are stronger in other areas.

Manually entered data is usually what I'm most skeptical about, but in some cases it is stronger than anything that can be acquired through automation.


a neural network can be used as an optimizer? I was wondering if a neural network can be used as an optimizer.
In other words, a network2 used as an optimizer takes the loss value of network1 and based on that it predicts the best weights for network 1.
as a discriminator in architecture, gan imitates a function of loss.

Theoretically yes, pratically no.
just think About what you Need for this second Network. You Need a dataset X of real numbers representing loss and you Need y representing weight plus bias.
You can already see that weights of a slightly different Architecture will be different. So creating such dataset is just infeasable given that you Need a lot of datapoints for your Network to learn.


User-to-item and item-to-user recommendations I'm currently creating a recommender system and there are different types of the systems.
Does anyone know something about the user-to-item and item-to-user recommendations? I have been searching for days, but I couldn't find anything about them. I have only found info about the user-to-user and item-to-item ones.

You might try several techniques to achieve your goal, I have researched and used these before, so it might help you.
Semi-Supervised Learning
Basically you need some historical data linking the products the users, therefore you can establish a primary knowledge of the relationship between some products and the users after that you cluster the same products/users and you label the rest of unlabeled products based on that.
Association rule learning
is a rule-based machine learning method for discovering interesting relations between variables in large databases. It is intended to identify strong rules discovered in databases using some measures of interestingness.
Collaborative learning/Federated learning
It's better to use it when you have informations about the buyers/sellers, therefore you can study the relationship between the product/(buyers/sellers), then products/shops, then you can supervise the learning between the products/shops.You need a high content quality data.
Cooperative Learning
You run diffrent test on diffrent combination of users/items, and you derive diffrent results from them, then you compare them against each other. For this you need some pre-determined rules to evaluate your own decision making results. You need some historical data of the products/users.
Combine collaborative learning and Cooperative Learning
This is what Google search engine is doing, he know what other people are looking for and build patterns for which is called collaborative learning, but if they give the results of it, it will not be specific for you as a user, so they add Cooperative Learning to emphasize your own search results experience. As interesting as it might sound it not that easy to code.


Bib number recognition using Keras I want to implement a racing bib number recognition application (for study purposes) using Keras API. I can manage training a CNN model to detect different numbers but how to localize only specific regions from the image and make sure that this region is a bib number?

Steps for this task:

Digit localizer - ?
Digit recognizer - trained CNN model using MNIST dataset.

What solution would be the best choice for localizing only specific regions? Should I use Faster R-CNN, YOLO Framework, Semantic Segmentation or smth else?

We can break down the task into two parts.

In the first part, our aim will be to localize regions where the bibs could be found. This is an object detection task, so you can use YOLO, Faster RCNN etc. YOLO could be a better choice as it is fast and efficient. We don't need semantic segmentation here. A number of GitHub repositories are available which enable to train a custom dataset for detecting bibs in images.

In the second part, as mentioned in the question, a CNN trained on MNIST will not be enough. We will need an OCR ( Optical Character Recognition ) which would get the text ( number ) from the image.

  The localizer will produce bounding box coordinates. The image will be
  then cropped by those coordinates. This cropped image could be passed
  into an OCR engine and finally, you will get the number present on the
  bib.

For training YOLO on a custom dataset, see here.

For an OCR engine, see here.


Why is the testing set standardized using fit values from the training set I was wondering, in the feature scaling part of data preprocessing, why is the data in the testing set standardized using the fit values from the training set? Why aren't the fit values recalculated separately for the testing set and then used for transforming the testing set? Here is the code I am using as reference:

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
x_train[:, 3:] = sc.fit_transform(x_train[:, 1:3])
x_test[:, 3:] = sc.transform(x_test[:, 1:3])

In this example, x_train is the training set and x_test is the testing set.

Thanks!

You could do that. Assuming your train/test data are from the same distribution, which is in fact what you assume, they should have a similar mean and standard deviation. The result would be similar, in theory, so by the same token, why would you refit the scaling on test data?

In practice, the mean/variance estimate from the presumably smaller test set is probably not as accurate. You will be transforming the test data differently than the training data, and the model may not fit as well.

Taking it to a logical conclusion, how would you then predict for a single future instance? you can't scale one instance based on stats from one instance; it has no variance.

The principle here is to fit a model on training data and then apply that fixed model, of which scaling may be a part, to future data. So there is no particular upside to refitting this specially, and some potential downside.


Custom loss for low false positive rate (higher precision) I am working with a scenario where I need to minimize the false positive rate for the minority class. Additionally my dataset is imbalanced. (10% minority class, 90% majority class). I am using the class_weight in the fit function of keras.

Additionally, I would also like to try a custom loss function to see if this makes a difference. A number of solutions online Keras custom loss function as True Negatives by (True Negatives plus False Positives) discuss a specificity/precision etc loss function. However this cannot be derived, so I don't think this can work.

Any suggestions?

Thanks in advance!

You have two options here that I can see:

1) quick and very very dirty - rebalance your data so that negatives are favored in the training data. Then your model will prefer to predict negative rather than positive, and false positives will be suppressed. To see this, consider what happens if all training data were negative. Tune the data balance until you get the desired result (I feel like a bad person for saying this...)

2) use a custom loss function such as the $F_\beta$ score (https://en.wikipedia.org/wiki/F1_score). However, this may be numerically tricky to implement for training as the gradient of the naive function generally behaves badly.


Build train data set for natural language text classification? I have extracted ~550 video scripts (subtitles) from 11 free courses on the Coursera platform. I have pre-processed them in terms of punctuation removal, stop words removal, tokenization, stemming and lemmatization. Now, I've been advised that for my task I can attempt to use a simple Bag of Words. However I am not sure how exactly would that help me towards classifying my text into one out of six categories. The categories are related to the intent a video material was created with and more precisely, which part explains a concept, which part discusses an example, which part gives practical advice etc. Below are my categories:

ConceptDescription-> Explanation of the Main concept(s)
ConceptMention-> Mentioning of a concept, related to the main concept
Methodology / Technique-> To achieve something, what should one do
Summary-> Summary of the discussed material or of the whole course
Application-> Practical advise for the concept
Example-> Concept example

By manually reading several files from 3 of the courses, I created a dictionary, containing spoken language words, that may help me identify which class a specific sentence/paragraph falls into. However I do NOT have a train dataset for a classifier. So my idea was to use that dictionary to label my data, e.g. sentence 1 as Summary, sentence 4 as ConceptDescription and sentence 12 as Example and then marking sentences 2 and 3 the same as 1, sentence 5-11 like sentence 4 etc.

My question is, is this idea too lame? And is there a way to create at least an average quality training dataset in a way that is not manual? Or if manual check is the only option, is there an option where I would need to do manual labeling on only a small fraction of the files, say 50 out of 550 and classification would still produce average to good result? I don't aim at perfect result, but I aim at something less time-consuming due to limited time.

I also played with tf-idf which outputs terms, but of course, not really what I need, so that was a bit random.

Thanks in advance for your help. Any specific ideas and algorithms would be very welcome.

It would help to do some analysis of the scripts to identify aspects that distinguish the various categories. Once you do this manually for some examples, you could consider writing some rules based on the observations. The rest of the examples can be labeled using the rules. For a model-based approach, if you label a small set of examples (~50), then a simple model (Naive Bayes, etc.) can potentially be trained on these.


How to evaluate multi label image retrieval model I'm using a deep hashing model to search most similar images in a database (most similar to the image given as a query). I'm doing this on the coco dataset which has multiple labels per image. I'd like to evaluate the performance of the model but I'm not sure what type of metric should be used here.

If it was just a single label per image, I'd go for something like mean average precision (given a query image of a dog, check how many dog images the system retrieved, evaluate MAP). But this obviously cannot be used on the multi label task (given a query image of three classes, system retrieved image with just one of them, it's not totally correct but it's not incorrect either). So are there any commonly used metrics to evaluate this kind of tasks? If so, please refer me to them. Or do I have to come up with something own (maybe some kind of a weighted MAP)?

First consider analyzing the results per class (with normal accuracy/F-score/whatever) - this way you can have a good idea of where the algorithm is doing well and where it may be improved.
Consider only the exact match. It's a "hit" if the algorithm got the exact labels that it should have. You can then use the hit count to calculate precision, recall and F-score.
Hamming Loss: this is a favorite of researchers in my experience, it represents the how many of the total labels were misclassified

The relevant Wikipedia article seems to agree with me and adds the Jaccard index to the mix.

The paper that presents the famous Classifier Chains method (READ J. et al, Classifier Chains for Multi-label Classification, 2009) uses four different evaluation methods: a accuracy variation that closely resembles Jaccard distance, a similarly changed F1-score, and a log-loss function. The fourth method they use to evaluate is the area under the Precision-Recall curve, but that's one I'd argue should not be used (see the work on Precision-Recall-Gain curves by Peter Flach).


Discrimination vs Calibration - Machine Learning Models I came across a new term called Calibration while reading about prediction models.

Can you please help me understand how different it is from Discrimination.

We build ML models to discriminate two/more classes from one another

But what does calibration mean and what does it mean to say that "The model has good discriminative power but poorly calibrated/calibrative power`?"

I thought we usually only look for separation only between 2 classes.

Can help me with this with a simple example please?

Discrimination is the separation of the classes while calibration gives us scores based on risk of the population.

For example, there are 100 people that we’d like to predict a disease for and we know that only 3 out of 100 people have this disease. We get their probabilities from our model. Due to good predictability power, our model predicts probabilities between 0-0.05 for 70 people and 0.95-1 for 30 people. This is a good discrimination between classes. We now know that 30 people are at high risk considering only discrimination. But we also know that only 3 out of 100 people get the condition which is 3% prevalence. We use the 3% prevalence to calibrate our scores which will give the actual risk based on population of 100. That means, 0.95 x 0.03 = 0.0285 is their actual risk to the disease.

This is a very crude approach, there are advanced techniques like Kernels, Platt Scaling etc.,


Theano logistic regression example I am trying to understand some simple neural net case using theano.
The deeplearning.net site gives the following simple code for implementing a logistic regression application to a simple case:

import numpy
import theano
import theano.tensor as T
rng = numpy.random

N = 400
feats = 784
D = (rng.randn(N, feats), rng.randint(size=N, low=0, high=2))
training_steps = 10000

# Declare Theano symbolic variables
x = T.matrix("x")
y = T.vector("y")
w = theano.shared(rng.randn(feats), name="w")
b = theano.shared(0., name="b")
print("Initial model:")
print(w.get_value())
print(b.get_value())

# Construct Theano expression graph
p_1 = 1 / (1 + T.exp(-T.dot(x, w) - b))   # Probability that target = 1
prediction = p_1 > 0.5                    # The prediction thresholded
xent = -y * T.log(p_1) - (1-y) * T.log(1-p_1) # Cross-entropy loss function
cost = xent.mean() + 0.01 * (w ** 2).sum()# The cost to minimize
gw, gb = T.grad(cost, [w, b])             # Compute the gradient of the cost
                                          # (we shall return to this in a
                                          # following section of this tutorial)

# Compile
train = theano.function(
          inputs=[x,y],
          outputs=[prediction, xent],
          updates=((w, w - 0.1 * gw), (b, b - 0.1 * gb)))
predict = theano.function(inputs=[x], outputs=prediction)

# Train
for i in range(training_steps):
    pred, err = train(D[0], D[1])

print("Final model:")
print(w.get_value())
print(b.get_value())
print("target values for D:")
print(D[1])
print("prediction on D:")
print(predict(D[0]))

I understand most of it, p_1 is the logistic regression function, the prediction is whether the value will be in the 0 class or 1 class, xent is the loss function, i.e. how far from correct is our prediction. I do not understand the next line, the cost. Shouldn't the cost be equal to the xent, i.e. the loss?
What is the cost function representing here?
Also, why is the bias initially set to 0 and not a random number like the weights?

I do not understand the next line, the cost. Shouldn't the cost be equal to the xent, i.e. the loss? What is the cost function representing here?

The cost is the error (xent.mean()) + some regularization (0.01 * (w ** 2).sum())

  Why is the bias initially set to 0 and not a random number like the weights?

It is possible and common to initialize the biases to be zero, since the asymmetry breaking is provided by the small random numbers in the weights.

More details here.


Strategies for continuously assessing and improving model performance I am building a supervised machine learning model to generate forecast.

So I would have historic data like this:

SKU, Month, .... other features, Actual Volume

That I can use a model to generate forecast, using the actual volume as the label.

Of course, there would be a variance between the forecast volume and the actual volume

What are the proper ways to leverage such data, without generating any data leakage, to incorporate such info to train the model to minimize the variance?

Should the data be fed back to the data with moving average, etc. and retrain, or is there other better strategy to properly assess the performance of the model and learn from it?

The data will be time series data with various features such as exchange rate, salesperson, etc.

I'm glad to see this question because this site gets such few questions on models that are actually in a production state.

If I was in your position, I would start to think about how I'm going to use this algorithm on a go-forward basis and start to log everything. Every new prediction that your algorithm makes is also a new data point for training.

So you need to take the time to create a feedback loop and (1) take new datapoints, put them into training, (2) re-train your algorithm, (3) deploy a new version and then (4) use that revised algorithm. You repeat this entire process on a continual basis throughout the life of your project. From the technical side, it's an exercise in re-inforcement learning since your algorithm won't start from a base of zero knowledge. Also, most of the steps I described will most likely require you to write new code, most of which will be related to process and workflow, not necessarily data science or algorithmic.

Finally, as an FYI, it is said that there are multiple algorithms out there that carry out these types of cycles on a continual basis, literally updating on a minute-by-minute (or faster!) basis, always providing answers on the latest available data points.


How can I determine how many signals there are in a mixture? Let's say that I have 9 sensors arranged in a 3 by 3 grid. I have multiple objects which emit the same signal and move past the grid of sensors, which are picking up the signals. I have a CSV file which contains readings from the 9 sensors collected every 0.01s for up to a minute and I don't know how many objects are in there. How can I find out how many there were?
I think each of the 9 sensors will basically collect very similar readings to each other, but at different time-points (since an object moves past each sensor at a different time). Any ideas on a general approach to these sorts of problems would be welcome too!

It is fourier transformation problem. All you need to do is to do a fourier transformation of your discrete signal and it will tell all the different frequencies it consists of. Numpy.fft class will be helpful.


How to Prepare data for LSTM I'm having difficulties to wrap my head around how I can prepare my dataset to train an LSTM.
Below is a screenshot of a subset representation of my dataset.

There are several other feature not included in this screenshot. The last column is inhospital_mortality which is 0 or 1 for each row.
Each feature was taken at a certain time x. features with the same feature_1,2,3 were taken at the same time.
My idea is that I will need to break every row (sample) as the example below: So in this case, each row would become 6 new rows.
| tc_tb1 | spo2_tb1 | g1_tb1| inhospital_mortality (label 0 | 1) |
| tc_tb2 | spo2_tb2 | g1_tb2| inhospital_mortality (label 0 | 1) |
| tc_tb3 | spo2_tb3 | g1_tb3| inhospital_mortality (label 0 | 1) |
...
| tc_tb6 | spo2_tb6 | g1_tb6| inhospital_mortality (label 0 | 1) |

Am I correct here? If so, how could I accomplish this dataframe manipulation in a more straight forward way? Perhaps there's a better way to reshape turn this dataset into the format I want. I wasn't able to accomplish it.
Thank you.

Welcome to the community.
About your problem, you say that your last column inhospital_mortality is binary so your goal is to classify, at a certain time with some info about them, patients likely to die in hospital if I am right.
Assuming this is your goal, I think you can try a classification approach without reshaping your dataset to a time series format, since you are not interested in forecasting a sequence (or a point in that sequence), you have independent buckets of x measurements per patient, without temporal connection in your training history.
You can use certain lag values (the -1, 2, 3... you already have) as informative features, and proceed with a binary classifier. Other option is, with this approach and per patient, calculate some statistical values like min, max, mean... of the 8 measurements for each type of patient indicator, while keeping the same number of rows but aggregating the attributes.
Of course you can also reshape it by having a unique date-time index with a value for each attribute type as you say, but you end up with less input features per row to predict your inhospital_mortality target value.
If you are interested in actually forecasting and eventually use an LSTM as you suggest, you might want to have a look at other answers about this type of problem like this


Sklearn train_test_split() error: Found input variables with inconsistent numbers of samples I am fitting a regression model on randomly generated X1,x2 and Y be the sum of x1, x2 but I am getting this error

  ValueError: Found input variables with inconsistent numbers of samples: [2, 10000000]

Note:- I am doing this only for learning purposes

My code:-

X = np.random.random_integers(100000000,size=(2,10000000))
X=(X-(100000000/2))/(100000000/2) # Scaling [-1,1]
Y = X[0]+X[1]

regr = linear_model.LinearRegression()
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.2,
random_state=0)
regr.fit(X_train,Y_train)

You need the shape of your training data to be (num_samples, num_features) or in your case (10000000, 2). An easy way to fix this is to transpose X before feeding it to the train test split:

X_train, X_test, y_train, y_test = train_test_split(X.T, Y, test_size=0.2, random_state=0)
regr.fit(X_train, y_train)  # <-- also 'y_train' not 'Y_train' here


Ranked or Ordered Class Classification My friend was asked this question in an interview for analytics and I cannot figure out the answer so I would like to see how could this data science problem be solved. Here's the problem:

Let's say the keyword "Superman" has a search volume of 250k per
month. Other related search terms can be "Superman Logo" (55k),
"Batman vs Superman" (60k), "Superman returns" (50k) etc. using which
users search for a certain topic in Google in a country (say US).
However, there may be thousands of keywords with the word "Superman"
in it, and we don't always have exact data for many of these keywords
with low volume (less than 1000). In order to find the volume of these
topics we use predictions based on the data that we already have for
high volume keywords.  For example, we may say that the term "Future
Superman" will not be searched by a lot of people and its volume can
be medium (close to 300).
Similarly, terms "Superman image" and "Superman jacket" will have
medium volume and terms that people rarely search will have low volume
like "Long Sleeve Superman Shirt" (100) and then there are terms like
"cheap superman shirt" or "Superman college" which will have very low
volume. Can you think of an approach on how you can classify keywords
into "Medium", "Low" and "Very Low" by using some logic when we only
know the high volume keywords?
Your answer should be descriptive and backed by reason. (Hint- your
approach may be to classify certain words which if present will make
the keyword "low" volume or "very low" volume. Or your approach can be
based on number of words in a keyword, spellings, where exactly a term
appears in a phrase and other common human psychology that works
during a search).

This looks like it could be framed either as Ordinal Regression or Ranking using Discounted cumulative gain

Ordinal regression is useful when you are trying to classify classes that have some inherent order like "Medium", "Low" and "Very Low". Used in psychology and marketing extensively it's a fit modeling procedure for this particular problem. The implementation is quite complicated but in a nutshell can be understood as overlapping distributions that can be understood as probabilities of them belonging to said class. R has an Ordinal Regression package.

Another one is Ranking, most machine learning packages include that in the form of Discounted Cumulative Gain which is a measure of rank quality and it is often used to measure effectiveness of web search engine algorithms or related applications. In xgboost it's found as rank:ndcg.

A naive approach but that can be accurate too is a simple multi-class classification. Information about the order will not be included in the algorithm but surprisingly its results are similar in these kinds of problems because the algorithms underneath like neural nets are now quite powerful specially when you have a big dataset.


How to construct confidence bound for Time Series Prediction? I have some time series data and am using some Deep Learning techniques to get its prediction. Now, I would like to construct confidence bounds for it.

I calculated the residuals, their mean and standard deviation. Then I plotted prediction vs residuals and calculated mean + 3$\sigma$ and mean - 3$\sigma$, and plotted them on my graph. So, I have something like this

Now, I would like to use a similar technique so that I can have confidence bounds on time series plots.

There is a technique called "Quantile Regression" which allows you to calculate a percentile of your $Y$ variable, the way a quantile regression works is by giving more penalty to the data far to the desired percentile.

Quantile Regression

As you used deep learning (I don't know in which software), maybe you could tailor a penalty function which does the same as quantile regression does or maybe the code you are using has it implemented.


What type of consideration can be made using clustering? I am clustering my data to see how information look like and which group may be identified.
Since clustering is an unsupervised algorithm, I cannot test the accuracy of the classification.
So I was wondering what type of consideration I can make after using clustering.
For example, if I had many emails, with no flag or label for spam/not spam, how could I use clustering to group them into two groups and test the ‘accuracy’ of the clustering?
To give more context on what I am trying to do:  I have different files (csv) having fields like date, users, emails’ subjects and emails’ bodies.
I would like to run some analysis but, in order to do this, I would need to classify emails into spam/not spam.
I have 23000 emails so it is very difficult to do this manually. I already included in a list of words the common words used as flag for spam (ads, buy, offer, porn, promotion,...) but, since the most of emails has no these words in a title or  in the body, this first step can assign ‘spam’ flag to around 100 emails. Very low! I have tried with topic classification (lda) but it is not so accurate. I thought then to use k-means clustering to assign these labels, once labelled manually around 300 emails.
I do not know if this is the right way to proceed for assignignig labels, so comments and answers would be greatly appreciated.

This is basic architecture of spam filter :

Statistically,spam bear lower entropy ( i.e., higher similarities) than legitimate emails.
We could use bisect k-means clustering after doing topic modelling. In k-means we had to specify k which lead to drastic change in results and it also leads to empty clusters.
I would recommend going through this paper as it highlight this approach.


Energy consumption time series forcasting Is there a good java library for doing time series energy consumption forecasting based on weather data and other variables?

If you just want to get a forecast, use automatic forecasting software like Autobox, ForecastPro, SCA, etc.

Comparison of different software: https://stats.stackexchange.com/questions/68253/expert-forecasting-software-evaluation
Worked example: http://people.duke.edu/~rnau/autocomp.htm

If you are doing research on forecasting techniques, there are some Java packages, for example:

JMotif http://code.google.com/p/jmotif/
JTSA http://sourceforge.net/projects/jtsa/

However, Java is not very widely used for this; if you must use Java you will likely have a feeling of being out in the cold.   R and Python would have much more packages, examples, etc.  A tiny sample of what is available in R:

http://www.amazon.com/Introductory-Time-Series-Paul-Cowpertwait/dp/0387886974/
http://a-little-book-of-r-for-time-series.readthedocs.org/en/latest/
http://www.r-bloggers.com/time-series-analysis-and-mining-with-r/


What algorithms will stuck in the local minimum? Algorithms like neural network are easily getting stuck in local minimum because the shape of the loss function (so there are parameters like momentum are designed to solve this type of problem). Logistic regression will always find global minimum because log-loss is a convex function (please feel free to correct me if I miss anything here).

Then I am wondering for other algorithms like RandomForest, GradientBoostTree, or SVM, which one would face the problem of stuck in local minimum, which one would not? And why? Thanks!

Any greedy algorithm -- which try to maximize your objetive by choose whichever is locally best -- may get stuck in local minima.

These include, but are not limited to:

gradient descent optimization like neural networks, which try to gradually minimize the loss function by making small changes to the weights in the direction that minimize the loss. This also includes algorithms like gradient boosting. You can mitigate such problems by using second-order derivatives (Hessian) or smart heuristics.
one look-ahead algorithms like decision trees, which locally choose the feature $x$ and threshold $t$ whose split $x<t$ maximizes/minimizes a given metric like gini or entropy (and, as a subsequent, random forest) -- if a decision tree was able to see the final impact of all possible combinations, then it might make different choices, but that would be infeasible.
expectation-maximization (as used by e.g. k-means) is also heavily influenced by the initialization.

I am being a little imprecise in my listing. What defines if an optimization problem may get stuck in a local minima is the function that you are trying to maximize. If you use gradient descent to solve the SVM optimization problem, then you'll always converge to the global minimum. More important than the algorithm is the surface of the optimization problem (in SVM, the objective is a nice quadratic surface).

Also notice that algorithms like Random Forests (when the number of trees is very, very high) become almost deterministic, but they still suffer from getting stuck in local minima. I am point this out because some people confuse optimizations being stuck in local minima with the algorithms being stochastic -- they reason that they can mitigate local minima by making multiple runs of the algorithm. But the algorithm being vulunerable to local minima $\neq$ the algorithm being stochastic. You'll never e.g. fix the one look-ahead problem by running a decision tree or a random forest several times. For algorithms like k-means, it is indeed a good idea to run it several times, as most implementations do behind-the-scenes.


Minimizing an upper bound of objective function In many machine learning problems, you often have an objective function (e.g., cost, loss, error) that you want to minimize.  Instead of directly minimizing this objective function, I sometimes see papers that derive the upper bound of the objective function, and then minimize the upper bound version instead.  The reason for this seems to be that the original objective cannot be obtained by the data we have at hand, or just hard to minimize for some reasons.

My question is, does it make sense to minimize the upper bound?  Are there guarantees that they lead to the same solution?  My concern is that the minimizer can obviously become different in the two cases.  If this is the case, do we know how close the two can be?  Are there good textbooks or papers that explain these issues?

Thank you very much for your help!

In many cases, the real function that you want to minimize is something that you cannot reasonably compute explicitly (e.g. in Bayesian statistics, you oftentimes have a formula integrating over all the possible combinations of something, which are too much for even the best supercomputer), or for which there is no known formula or procedure that can minimize it directly  (e.g. in discrete optimization).

In that case, if you cannot compute or minimize the function itself, it makes sense to minimize something that either correlates highly with the function (e.g. a sampled estimate, or some approximation that replaces true values with other estimations), or an upper-bound (as long as it is a reasonable) - in the second case, you know that if you take this upper bound from something like $10^{20}$ to $10^-3$, the initial parameters would very likely have resulted in a worse value for the function that the ones you obtained by minimizing the upper bound.

In some cases, it can be proved that the optimal value of the function is within some percentage of the solution that maximizes the upper bound (quite common in discrete optimization), but this is oftentimes not the case (e.g. in Bayesian statistics), and what you get is NOT BY ANY MEANS the true global optimum for the original function, but something that is nevertheless better than random and feasible to calculate on your computer.

Now, you might argue that there are black-box methods such as evolutionary search that could minimize pretty much any function that you can evaluate directly, but these oftentimes lead to worse solutions than minimizing the upper bound, require more function evaluations, are much, much slower, and impractical if you have too many parameters, while the upper bounds can oftentimes be reasonably minimized with faster methods that rely on assumptions such as differentiability, smoothness, Lipschitz continuity, etc.

In short: the optimal solution for the original function and for the upper bound are indeed different, but if you cannot minimize the original function directly, you can still get a decent solution (not the best one) by minimizing the upper bound.


Is the Cross Entropy Loss important at all, because at Backpropagation only the Softmax probability and the one hot vector are relevant? Is the Cross Entropy Loss (CEL) important at all, because at Backpropagation (BP) only the Softmax (SM) probability and the one hot vector are relevant?

When applying BP, the derivative of CEL is the difference between the output probability (SM) and the one hot encoded vector. For me the CEL output, which is very sophisticated, does not play any roll for learning.

I´m expecting a fallacy in my reasoning, so could somebody please help me out?

During backpropagation we take the derivative of loss function with respect to all the weight parameters. For cross entropy loss this derivative is simply the difference between output probability and predicted probability. Upto this you're correct.

But you missed what loss function represents. The loss function represents the surface manifold of the network. The purpose of learning is to find the minima of this loss function.

We use CE loss because the surface define by it is smooth and continuous, hence we can find the derivative easily.

Mathematically that derivative comes out to be a simple difference, but theoretically we're differentiating the surface defined by the loss function, which is important.


How do NLP tokenizers handle hashtags? I know that tokenizers turn words into numerics but what about hashtags? Are tokenizers design to handle hashtags or should I be filtering the "#" prior to tokenizing? What about the "@" symbol?

The answer depends on what you want to do with the hashtags/words and also on what tokenizer you are using.

Consider this example tweet:

  Hi, we need you!

  #Hi #Weneedyou

If you use TreeBank or WordPunct tokenizers the output will be:

 ['Hi', 'we', 'need', 'you', '!', '#', 'Hi', '#', 'Weneedyou']

However if you use Whitespace Tokenizer, the result is:

 ['Hi', 'we', 'need', 'you!', '#Hi', '#Weneedyou']

Similar discrepancies can be found in terms such as can't or pre-order for instance.

Additionally, you need to consider what is a token for your task at hand. In my previous example Hi make sense either with or without #. However Weneedyou without the hash is just a poorly written word. Maybe you want to keep the hashtags in your corpus or maybe not, but this depends on what you want to do with it.

So first you need to know how your tokenizer handles these cases and decide if you want to remove them beforehand or later to keep hashtags in your corpus or only (sometimes weird) words.

In the case of the @ is exactly the same, you can keep it, remove it or maybe delete the whole instance @user as you don't want to keep user names in your corpus. As I said, it all depends on your task.

PS: In case you want to play around with different tokenizers, try this.


Possible reasons for word2vec learning context words as most similar rather than words in similar contexts I am observing my word2vec model learning context words as most similar rather than words in similar contexts. I don't understand why it (word2vec in general, not my model in particular) can behave like that and would like to know why.

I have implemented the original word2vec in keras. I chose the variant with the dot product layer rather than the hierarchical softmax and trained the model on a Wikipedia dump that I split into 5-grams. For each word I construct 8 pairs with a binary target label as training items. I use the 4 context words with the label True and choose 4 random words that are not one of the context words with the label 0.

Intuitively this model should learns similar representations for words in similar context, because it modifies the representation of these words in a similar manner, as it optimizes them independently with similar context words.
So these similar words are not directly made similar but rather indirectly, as they are subject to similar nudges due to their similar contexts.

The model is this:

input_target = Input((1,))
input_context = Input((1,))

embedding = Embedding(vocab_size, vector_dim, input_length=1, name='embedding')

target = embedding(input_target)
target = Reshape((vector_dim, 1))(target)
context = embedding(input_context)
context = Reshape((vector_dim, 1))(context)

dot_product = Dot(axes=1)([target, context])
dot_product = Reshape((1,))(dot_product)

output = Dense(1, activation='sigmoid')(dot_product)

model = Model(inputs=[input_target, input_context], outputs=output)
model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

However, what I observe from training this model is, that when I rank the most similar words given a certain word, that I get words that appear in the context of that word, rather than words that appear in similar context, as the most similar words.

For example:

words most similar to "plant":

rank |   word

0    |   amount
1    |   surface
2    |   electron
3    |   mass
4    |   plant # also: Why is plant not most similar to plant? How can that happen?
5    |   fluid
6    |   air
7    |   metal
8    |   molecule
9    |   cell
10   |   electric
11   |   per
12   |   oxygen
13   |   demonstrate
14   |   smooth

To me that looks a lot more like words that appear in the context of plant rather than words that appear in similar context as it.

The function to compute these is:

def get_most_similar(word_vector, embeddings, n=15):
    """
    find the `n` words that are most similar to `word_vector` in `embeddings`
    measured by their cosine similarity
    """
    v = word_vector
    m = embeddings
    cosines = (np.dot(v, m.T))/(np.linalg.norm(m.T, axis=0)*np.linalg.norm(v))
    ranked_by_similarity = np.argpartition(cosines, -n)[-n:]
    return reversed(ranked_by_similarity)

Is there a simple reason for that?

I have the following other parameters:

word vector size: 300
batch size: 128
vocabulary size: 169161 (distinct lemmas)
training sample count: 27793586 (5-grams, overlapping within sentences)

I trained the model for 1 epoch and did only observe marginal further convergence into the second epoch.

Neural networks are different from other Machine Learning techniques in that they have to learn by repetition.  In Neural Network parlance, the repetition is called an epoch. During an epoch, each instance, in your case each word, is evaluated and the error is applied backwards through the layers and the weights are adjusted.  The changes to the weights can happen more or less aggressively based on the hyperparameters, but they tend to only move a little at a time.

You have trained your word2vec model for two epochs.  I didn't see how you initialized the weights for your model, but typically they are set to random values or set to a zeros. Given your results look a little random, I am sure it is because the model has barely begun to learn.

In order to get a well trained word embedding model, or any other Neural Network, you may need to train for hundreds, thousands, or even millions of epochs.  This is why the pre-trained word embedding models are so valuable even though they are freely available.  If you just want a well trained embedding model, download it. If you need to incorporate some domain specific usage, you can further train the downloaded embeddings.

HTH