splitting mechanism with one hot encoded variables (tree based/boosting) I am using xgboost and have a categorical unordered feature with 25 levels. So when i apply one hot encoding i have 25 columns. This introduces alot of sparsity. Even more unusual, my feature importance report shows 5 of these one hot encoded column in top 10, with one of them appearing at the top.
I tried to see if there was a diffrence in percentage of these categories between my binary classes (1, 0) but there isn't so i am a little perplexed as to why it is assigning such a high feature importance to them.
i have read online that if we have a categorical variable with q levels, the tree has to choose from ((2^q/2)-1) splits. For a dummy variable, there is only one possible split and this induces sparsity
i am not sure i understand this, say i have a column called color: red, green, blue,yellow , and i implement one hot encoding so is the number of splits that happen is 2^4/2 -1 = 3? if this increases as i have e.g. 2^25/2 -1, more splits means the tree is more likely to find a 'good split' for data at hand and lead to overfitting? But what i don't understand is how this splitting chages with dummy variables.. does that equation hold or not for one hot endoded variables.
am i interpreting this correctly?
sources elemts of statisicatl learning:
https://towardsdatascience.com/one-hot-encoding-is-making-your-tree-based-ensembles-worse-heres-why-d64b282b5769#:~:text=For%20every%20tree%2Dbased%20algorithm,a%20feature%20and%20a%20value.&text=The%20trees%20generally%20tend%20to,values%20(0%20or%201).

i have read online that if we have a categorical variable with q levels, the tree has to choose from ((2^q/2)-1) splits. For a dummy variable, there is only one possible split and this induces sparsity
i am not sure i understand this, say i have a column called color: red, green, blue,yellow , and i implement one hot encoding so is the number of splits that happen is 2^4/2 -1 = 3?...

You have the order of the operations wrong there (probably because the "((2^q/2)-1)" you quote above is misleading, but compare to the ESL quote in your image):  it's
$$ \frac{2^4}{2}-1 = 2^{4-1}-1 = 7$$
possible splits, namely:
(red green blue) vs (yellow)
(red green yellow) vs (blue)
(red blue yellow) vs (green)
(green blue yellow) vs (red)
(red green) vs (blue yellow)
(red blue) vs (green yellow)
(red yellow) vs (green blue)

if this increases as i have e.g. 2^25/2 -1, more splits means the tree is more likely to find a 'good split' for data at hand and lead to overfitting?...

It is certainly true there are more possible splits, and that increases the model's capacity and hence perhaps increases overfitting.  This is particularly worrisome if some of the levels are quite rare, and less so if you have a lot of data.

But what i don't understand is how this splitting chages with dummy variables.. does that equation hold or not for one hot endoded variables.

No, when you one-hot encode such a feature, the tree now must split on only one of those dummy variables (at a time).  So, considering all the $q$ levels' new indicator variables, you have exactly $q$ splits to consider.  In your example, they are the first four splits given above:
is_yellow=0 vs is_yellow=1
is_blue=0 vs is_blue=1
is_green=0 vs is_green=1
is_red=0 vs is_red=1


Can i pursue a career in data science with a business analytics degree? I had a question regarding about a career as a data scientist.  Im pursuing a degree in business analytics with a minor computer science.  Could i still become a data scientist with a bachelors degree in business analytics. Or would i have to pursue a masters degree in data science in order to become a data scientist.

(I have a good programming background in data structures and algorithm in java, plus web development, python)

You can. Nobody is going to demand a degree, but they will expect you to pass the interviews. I would recommend creating a blog and do some Kaggle competitions to show off your skills. I would also recommend taking more serious maths courses from the statistics department.


Preparing data, choosing algorithm First off, I am new to machine learning, so these questions may be trivial.

Basically I am trying to tune an object with numeric knobs and numeric outputs. By doing a brute force tuning (permutations), there is a solution to find ideal values of the outputs, but it takes time. I'm trying to take a stab at using ML to at least shorten the process of tuning.

I have a dataset for a large number of good units that were tuned successfully, at the various number of attempts though.

For example:
   passing requirement: X = near 10, Y = near 5, Z = near 4

Object 1:

Try 1 => A = 1, B = 2, C = 3 ; X = 1, Y = 0, Z = 1  => not good
Try 2 => A = 1, B = 1, C = 1 ; X = 10, Y = 5, Z = 4 => good enough

Object 2:

Try 1 => A = 1.4, B = 2.6, C = 3.8 ; X = 10, Y = 5, Z = 3.9 => lucky!!!

Object 3:

...
Try 10 => A = 1.4, B = 2.6, C = 3.8 ; X = 10, Y = 5, Z = 3.9 => took a while!!!

I'm wondering how do I prepare the data for training and testing with this type of problem, as each object had a variable number of tries before it got successfully tuned.
Should I just take the last successful combination for each object, and maintain the same columns (A, B, C, X, Y, Z). Or take them all (multiple rows per object)?

Or, for each object record, append another set of columns, such that per object, I have only one row. For example (A1,B1,C1,X1,Y1,Z1, A2,B2,C2,X2,Y2,Z2,... An,Bn,Cn,Xn,Yn,Zn)

As for the algorithm choice, I can only articulate that this isn't a classification problem (or a binary outcome). Something like regression, or decision tree, or hill climbing if there's such a thing?

I don't know that I'd call this a machine learning problem per se...though others may disagree, this sounds like an optimization problem.

Your description is vague, but let's pretend it's a guitar that you're trying to tune and you have 6 strings. The trick is to define an objective function that quantifies, with a single number, how 'good' the tuning is. (The program can independently manipulate each string's tension). Perhaps it's some deviance in power from an ideal FFT, and you sum the deviance from each string.

Optimization routines such as simplex, sub plex, Nelder-Mead, GA, etc., are ways of manipulating the tension in 'smart' ways that obviate the need for a brute force (or 'grid search') solution. So long as you can define an objective function, you can start trying it out, though the local max/min problem will need to be dealt with.

I'd look into some tutorials on minimization algorithms


Fields list of YearPredictionMSD (identify year of music from Million Song Dataset) The YearPredictionMSD dataset, is a machine learning practice dataset extracted from the Million Song Dataset and found in the UCI Machine Learning Repository. I couldn't find the field list that describes the features. Does anyone know where I can find them?

Also, I'm confused why there are only 54 features in the original data from LabRosa but 90 in the subset of the data on UCI.

You don't get a full data dictionary with this dataset. Instead the explanation (from your link) is:

  90 attributes, 12 = timbre average, 78 = timbre covariance
  The first value is the year (target), ranging from 1922 to 2011.
  Features extracted from the 'timbre' features from The Echo Nest API.
  We take the average and covariance over all 'segments', each segment
  being described by a 12-dimensional timbre vector.

This is basically features derived direct from the audio. This is typical for problems in signal analysis, and the UCI dataset has been cut down so that it is purely about matching audio summaries to year of production.

The data dictionary from the original MSD dataset shows much more metadata. There are 54 feature types, but actually thousands of features - each type can be repeated many times. The UCI subset is not only subset by row, but by columns too, and only uses some of the MSD data.


Using GANs to generate synthetic tabular data to improve supervised learning One topic I see some people trying is using GANs to generate synthetic tabular data for supervised learning. Also as a way to oversample the minority class in a binary classification.
For me creating synthetic data is a bit dangerous.
In practice, all the experiments that I have seen to generate new training data using GANs have failed.
Is there any theoretical reason behind?

GANs have many known problems. The main ones are:

Lack of convergence.
Vanishing gradients when discriminator is "too good", leading to stagnation of the generator.
Mode collapse: the diversity of the generated samples tends to be very low, generating always the same values.

GANs for image generation have been studied extensively. Other domains, like speech filtering, have also been studied, but not so extensively. In other domains, like text generation, GANs are not very successful. For tabular data generation via GANs, the amount of released work is scarce: medGAN, VeeGAN, ehrGAN, TableGAN, CTGAN.
I think that one of the main problems preventing us from devising better GANs in non-image domains is the evaluation. With images, you can eyeball the results and quickly determine if they are of good quality and diverse. However, with other domains, it is not easy to evaluate both the quality and diversity of the generated data.
I think most people nowadays stick to classical oversampling methods to generate tabular data.


Feature Scaling both training and test data It is stated that for:
Feature Normalization -

  The test set must use identical scaling to the training set.

And the point is given that:

  Do not scale the training and test sets using different scalars: this
  could lead to random skew in the data.

Could someone explain what that means?

Generally speaking, best practice is to use only the training set to figure out how to scale / normalize, then blindly apply the same transform to the test set.

For example, say you're going to normalize the data by removing the mean and dividing out the variance. If you use the whole dataset to figure out the feature mean and variance, you're using knowledge about the distribution of the test set to set the scale of the training set - 'leaking' information.

The right way to do this is to use only the training set to calculate the mean and variance, normalize the training set, and then at test time, use that same (training) mean and variance to normalize the test set.

As for the point in your question, imagine using the training mean and variance to scale the training set and test mean and variance to scale the test set. Then, for example, a single test example with a value of 1.0 in a particular feature would have a different original value than a training example with a value of 1.0 (because they were scaled differently), but would be treated identically by the model.  This is where the bias would come from.

Hope this helps!


Is there a disadvantage to letting a model train for a large number of epochs? I created a model to solve a time series forecasting problem.  I had a limited amount of time series with which I could train the model therefore I decided to augment the data.  The data augmentation strategy I used is quite basic but has shown to increase the accuracy of my model.

I wrote my own data_generator which I use to train my model with, using the fit_generator function in keras.  Essentially it takes in the whole training data set that I have, shuffles all the time series and the augmentation process takes place specifically in each batch.  In each batch I randomly pick, per time series in the batch, start and end points, so that each batch contains varying length slices of each series within the batch.  This creates obviously an almost endless stream of data but it is entirely reliant on the number of epochs the model is run for as the dataset is not augmented upfront.  No noise or anything is applied to the data set, the augmentation is purely from varying the lengths of the time series and the start and end points of the series.

I observe that my loss continues to decrease over time I have tried 100, 500, 1000, 5000 and 10,000 epochs.  In general the accuracy of the model predictions does get better but at some point with diminishing returns.  It is hard to say when as I am still tuning the model architecture and hyperparameters.

Does such an augmentation strategy affect how I can interpret the loss of model?  As the longer I train the model for the more "new" data it sees instead of constantly seeing the same data and training on it.

If you can be sure that the model is not seeing the same instances repeatedly then there is very good chances that your model is not overfitting and that is precisely what you can measure from your validation set, you should see continuously downward loss which will eventually plateau at the local optimum that has been eventually reached by your model, that is the best possible solution attained given the starting conditions. If your model would be overfitting then you should see the loss function start increasing for the validation.

The best way to know if you are overfitting is to take your model and then apply it to a completely new dataset and measure the performance. If the performance is good, then you are fine.

p.s. To overcome falling inside a bad optimum you can train multiple models each with different starting conditions for the model parameters and then make an ensemble classifier.


Multiclass classification with Neural Networks Let’s suppose I wanted to classify some input as one of three categories using a simple neural network.
The output of my network are three columns (one for each possible category I assume) with values between 0 and 1. Moreover, the single rows add up to precisely one when adding the three columns together. Is is possible to interpret the output as the probability of my input belonging to each single category?

Indeed, this is the standard interpretation of continuous classifier outputs, not only for neural networks, but for the more general case called Softmax Regression.

Thus, provided that you have used softmax activation on the final layer (in order, among other things, to ensure that your outputs indeed sum up to 1), you can interpret the continuous outputs as the respective probabilities of a particular data sample belonging to each one of your classes.

See also the discussion in this (rather unfortunately titled) discussion at SO:

How to convert the output of an artificial neural network into probabilities?


Is it meaningful to use word2vec for non-string inputs like time series analysis? I am working on a project that detects anomalies in a time series. I wonder if I can use word2vec for anomaly detection for non-string inputs like exchange rates?

The goal of Word2vec is to represent each element of a sequence into an "embedded space", i.e. a lower dimensional space where "similar" elements are located closer to each other.

I think this is a bit off topic for time series analysis. If you want to detect outliers, you might try looking at the normalized distance of each datapoint from the trend, and set a threshold.

Alternatively, if you really want to stick with Neural Networks, you can use Autoencoders. They can be applied to outlier detection, but it's not the simplest model to implement.


Neural networks with non-negative weights Could you tell me, are there any techniques for building neural networks with non-negative weights?

One possible method to approach neural networks with non-negative weights is using Feedforward neural network. We can construct the neural network using optimization techniques with the constraint of non-negative weights, as opposed to the normal back propagation method.

A modified Matlab example taken for here is as follows:

load iris_dataset

% Number of neurons
n = 4;

% Number of attributes and number of classifications
[n_attr, ~]  = size(irisInputs);
[n_class, ~] = size(irisTargets);

% Initialize neural network
net = feedforwardnet(n);

% Configure the neural network for this dataset
net = configure(net, irisInputs, irisTargets); %view(net);

fun = @(w) mse_test(w, net, irisInputs, irisTargets);

% Add 'Display' option to display result of iterations
ps_opts = psoptimset ( 'CompletePoll', 'off', 'Display', 'iter', 'MaxIter', 100); %, 'TimeLimit', 120 );

% There is n_attr attributes in dataset, and there are n neurons so there
% are total of n_attr*n input weights (uniform weight)
initial_il_weights = ones(1, n_attr*n)/(n_attr*n);
% There are n bias values, one for each neuron (random)
initial_il_bias    = rand(1, n);
% There is n_class output, so there are total of n_class*n output weights
% (uniform weight)
initial_ol_weights = ones(1, n_class*n)/(n_class*n);
% There are n_class bias values, one for each output neuron (random)
initial_ol_bias    = rand(1, n_class);
% starting values
starting_values = [initial_il_weights, initial_il_bias, ...
                   initial_ol_weights, initial_ol_bias];

% alter the patternsearch function with the appropriate constraints, in your case we would change it so that the lower bounds of the weights are zero
[x, fval, flag, output] = patternsearch(fun, starting_values, [], [],[],[], zeros(size(starting_values)), 1e10, ps_opts);

where the mse_test.m function is as follows:

function mse_calc = mse_test(x, net, inputs, targets)
net = setwb(net, x');
y = net(inputs);
[row col] = size(y);
mse_calc = sum(sum((y - targets).^2))/(row * col);
end


High Level Understanding of Neural Machine Translation I am currently studying Deep Learning based Machine Translation systems but I'm not sure in my understanding the logic of the process. I understand that the source and destination language translation sentence pairs must be represented as word2vec vectors, but why is it necessary to apply two (encoder-decoder) Recurrent Neural Networks? My first idea would be applying only one RNN, where the input is the source language examples (in the form of word2vec vectors) and the output is simply a word2vec sequence of the destination language. Why is it necessary to use another RNN?

My additinal question is if this system is flexible enough to cope with synonimes, word order variations and other disambiguities? Is it capable of approximate the correct meaning of a new to-be-translated source language sentence?

And last but not least: how could one evaluate such a model where many translations can be correct at the same time?

In encoder-decoder architecture, we first represent the input sequence by a fixed vector. It is assumed that this fix vector represents the complete meaning of the sentence. Now decoder uses this fix vector to generate the output sequence.

Answers of your questions:

If you use 1 recurrent unit, it outputs the value at each time instant which is not suitable for machine translation as translation of first word may be in the last or somewhere else in target language.(I am not sure how will you map the output and input in that case.)
Regarding synonyms, system can handle synonyms as words which are synonyms have very close word vectors.
Regarding Word-Order, it can learn the word order if you have enough training data. Decoder also uses beam search to find the best possible translation. I will suggest you to read this paper thoroughly.
https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf
Regarding evaluation, there are various metrics which are designed to evaluate the machine translation system. BLEU is one of the most famous one out of all. It uses various reference translations. You can read more about BLEU in the paper by Kishore Papineni et. al.(I can not post more than two link in answer as my reputation is less than 10.)

I will also suggest you to read the below paper regarding neural machine translation.
https://arxiv.org/pdf/1409.0473.pdf

Hope it helps!!


K Means giving poor results I have several user names and their salaries.
Now I need to cluster user based on their salaries.
I am using KMeans clustering and following is my code

from sklearn.cluster import KMeans
from sklearn.preprocessing import LabelEncoder
import pandas as pd

le = LabelEncoder()
data = pd.read_csv('kmeans.data',header=None, names =['user', 'salary'])

# Numerical conversion
data['user'] = le.fit_transform(data['user'])
km = KMeans(n_clusters=4, random_state= 10, n_init=10, max_iter=500)
km.fit(data)

data['labels'] = le.inverse_transform(data['user'])
data['cluster'] = km.labels_

print data

But my results are bad and there are lot of overlapping salaries.

Is there anything wrong in the code ? How to improve the results ?

Or whether clustering is not a right approach here ?
Then how can I cluster users only based on salary ?

 km.fit(data['salary'])

EDIT:

I figured out a way to solve my problem using numpy.reshape

km.fit(data['salary'].reshape(-1,1))

K-means is based on the assumption that the data is "translation invariant" (more precisely: variance does, and k-means is variance minimization).

In other words, it assumes that a difference of d=(x-y)^2 is of the same importance everywhere. Because of this, k-means does not work on skewed data. Furthermore, because of the square, it is sensitive to outliers and other extreme values.

For salaries and other monetary values, this usually does not hold. The difference between \$0 and \$1000 is massive, and not the same as a salary difference of \$100000 to \$101000. Salaries are usually rather skewed, and you often have some extreme values.

Converting the "user" attribute to a numerical value is outright statistical nonsense. What's variance worth in this attribute? K-means is for continuous numerical data only, and converting data does not chnage the nature, only the encoding - it's still inappropriate.


Meaning of stratify parameter I'm training a Neural Network and I'm trying to divide my data into training and testing sets. I have a lot of output classes and for some of them I have as little as 2 examples, so I would like to have, in that case, 1 example in training and 1 example in testing. From what I've read, this is using the stratify parameter, but what does stratify mean?

I'm divifing my data into training and testing:

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.1, random_state=42, stratify=y)

So, from my understanding, this divides into two sets: training (90% of data) and testing (10% of data) but making sure that there are at least 1 of each class in each set?

stratify parameter will preserve the proportion of target as in original dataset, in the train and test datasets as well.

So if your original dataset df has target/label as [0,1,2] in the ratio say, 40:30:30. That is, for every 100 datasets, you can find 40, 30 and 30 observations of target 0,1 and 2 respectively.

Now when you split this original using the train_test_split(x,y,test_size=0.1,stratify=y), the  methods returns train and test datasets in the ratio of 90:10. Now in each of these datasets, the target/label data proportion is preserved as 40:30:30 for the classes [0,1,2].

Often, we want to preserve the dataset proportions for better prediction and reproduceability of results


How/What Correlation for Position Ranking? I'm newbie to Data Science. I'm trying to understand how to correlate the position of an app in the app store, e.g. 1-10, to the number of backlinks, e.g. #1 = 250 links, #8 = 50 links, to the app store listing page in Python.

I've manage to correlate both rows entirely, which obviously gives me a nonsense figure.

Could someone point me in the right direction to allow me to fill this knowledge gap. I'm not entirely sure of the correct terminology for this, making Googling it pretty impossible.

Sounds like you are looking for the Pearson correlation coefficient between the two variables. You can compute that by using Scipy's pearsonr method.


How to use text as an input for a neural network - regression problem? How many likes/claps an article will get I am trying to predict the number of likes an article or a post will get using a NN.
I have a dataframe with ~70,000 rows and 2 columns: "text" (predictor - strings of text) and "likes" (target - continuous int variable). I've been reading on the approaches that are taken in NLP problems, but I feel somewhat lost as to what the input for the NN should look like.
Here is what I did so far:

Text cleaning: removing html tags, stop words, punctuation, etc...
Lower-casing the text column
Tokenization
Lemmatization
Stemming

I've assigned the results to a new column , so now I have "clean_text" column with all the above applied to it. However, I'm not sure how to proceed.
In most NLP problems, I have noticed that people use word embeddings, but from what I have understood, it's a method used when attempting to predict the next word in a text. Learning word embeddings creates vectors for words that are similar to each other syntax-wise, and I fail to see how that can be used to derive the weight/impact of each word on the target variable in my case.
In addition, when I tried to generate a word embedding model using the Gensim library, it resulted in more than 50k words, which I think will make it too difficult or even impossible to onehot encode. Even then, I will have to one hot encode each row and then create a padding for all the rows to be of similar length to feed the NN model, but the length of each row in the new column I created "clean_text" varies significantly, so it will result in very big onehot encoded matrices that are kind of redundant.
Am I approaching this completely wrong? and what should I do?

from what I have understood, it's a method used when attempting to predict the next word in a text.

Not really, Word2vec is a technique to represent the (relative) meaning of words in a way that can be fed into a ML model. You can use them as language models, i.e. for the prediction of the next word in a sequence, but that's just one of the possible uses of it. You can train a model with word embeddings for whatever other task. Word2vec is suitable in this case.

Learning word embeddings creates vectors for words that are similar to each other syntax-wise, and I fail to see how that can be used to derive the weight/impact of each word on the target variable in my case.

I don't know your problem well enough, but I'd say you need additional information. For example: informations on the account such as number of followers, likes, shares/retweets, you name it.

In addition, when I tried to generate a word embedding model using the Gensim library, it resulted in more than 50k words, which I think will make it too difficult or even impossible to onehot encode.

Gensim requires a list of words/tokens (in the form of strings), and the word2vec model will take care of anything for you. You don't have to manually one-hot encode all the words. Don't worry about that.

I will have to one hot encode each row and then create a padding for all the rows to be of similar length to feed the NN model, but the length of each row in the new column I created "clean_text" varies significantly, so it will result in very big onehot encoded matrices that are kind of redundant.

I don't really know what you did here, but I'm pretty sure it's not correct. You don't have to manually one-hot encode anything. More importantly, one-hot encoding rows doesn't make sense, why would you do that?


Machine Learning algorithm for predicting number of cases in pandemic I’m giving my first steps with AI and Machine Learning so I have the following issue. I’m trying to predict an outcome from COVID-19 number of day vs confirmed cases using scikit-learn library. I mean, my input is the number of days since the pandemic started in my country and my output is the number of confirmed cases in that corresponding date. However both using GradientBoosting and RandomForest I get the same output values for the test values…I post below the code in Python as it is very short…

import numpy as np

from sklearn import ensemble

import pandas
datos = pandas.read_csv('covid.csv',";")

entrada = np.array(datos['ORDEN']).reshape(-1,1)

salida = datos["CASOS"]

regr = ensemble.GradientBoostingRegressor(random_state=0,n_estimators=500).fit(entrada,salida)

test = np.array([i for i in range(63,70)]).reshape(-1,1)

print(regr.predict(test))

regr = ensemble.RandomForestRegressor(random_state=0,n_estimators=500).fit(entrada,salida)

print(regr.predict(test))

My output is this:

[1782.99976513 1782.99976513 1782.99976513 1782.99976513 1782.99976513
 1782.99976513 1782.99976513]
[1773.99 1773.99 1773.99 1773.99 1773.99 1773.99 1773.99]

What am I doing wrong?? Thanks in advance.

It will depend completely on your feature engineering so I can think that in this case your model is maybe only predicting the mean or median of your target.

Also, it might help try using other kinds of models since you are trying to predict the counts of an event on a determined period of time, so it might be useful to use Poisson models that are in experimental phase in sklearn, nonetheless, the documentation might help to understand how the model works


Negatively correlated features Is it ok to use negatively correlated features in data modeling? Say I have features A and B that have a correlation coefficient of 0.2 and features C and D with -0.2 correlation coefficient, is it fine to use features C and D in the model, since they have a low negative correlation? Also, does this have different effects on a regression vs. classification problem?

There's no problem in having negative correlations, in fact, if you change C to -C the correlation will be positive and every machine learning model I know should be equivalent trained on a variable or the same one with the sign changed.
So no, in any supervised case, regression or classification, it doesn't matter if there are negative correlations among features, and it happens almost every time.


Is there a library that would perform segmented linear regression in python? There is a package named segmented in R. Is there a similar package in python?

No, currently there isn't a package in Python that does segmented linear regression as thoroughly as those in R (e.g. R packages listed in this blog post). Alternatively, you can use a Bayesian Markov Chain Monte Carlo algorithm in Python to create your segmented model.

Segmented linear regression, as implemented by all the R packages in the above link, doesn't permit extra parameter constraints (i.e. priors), and because these packages take a frequentist approach, the resulting model doesn't give you probability distributions for the model parameters (i.e. breakpoints, slopes, etc). Defining a segmented model in statsmodels, which is frequentist, is even more restrictive because the model requires a fixed x-coordinate breakpoint.

You can design a segmented model in Python using the Bayesian Markov Chain Monte Carlo algorithm emcee. Jake Vanderplas wrote a useful blog post and paper for how to implement emcee with comparisons to PyMC and PyStan.

Example:

Segmented model with data:

Probability distributions of fit parameters:

Link to code for segmented model.
Link to (large) ipython notebook.


IndexError: list index out of range I'm implementing a sequence-2-sequence model with RNN-VAE architecture, and I use an attention mechanism. I have problem in the decoder part.
I'm struggling with this error:   IndexError: list index out of range
When I run this code:

decoder_inputs = Input(shape=(len_target,))
decoder_emb = Embedding(input_dim=vocab_out_size, output_dim=embedding_dim)
decoder_lstm = LSTM(units=units, return_sequences=True, return_state=True)
decoder_lstm_out, _, _ = decoder_lstm(decoder_emb(decoder_inputs),
             initial_state=encoder_states)

print("enc_outputs", encoder_outputs.shape) # ==> (?,256)
print("decoder_lstm_out", decoder_lstm_out.shape)# ==> (?,12,256)
print("zzzzzz", z.shape) # ==> (?,256)

attn_layer = AttentionLayer(name='attention_layer')
attn_out, attn_states = attn_layer([z,z], decoder_lstm_out)

The error is raised at the last line, and the traceback given:

Traceback (most recent call last):
     File "malek_tuto.py", line 197, in <module>
           attn_out, attn_states = attn_layer([z,z], decoder_lstm_out)
     File "C:\Users\lightland\Anaconda3\lib\site-
       packages\tensorflow\python\keras\engine\base_layer.py", line 728, in
       __call__    self.build(input_shapes)
     File "D:\PFE\Contribution\modele\layers\attention.py", line 24, in
        build shape=tf.TensorShape((input_shape[0][3], input_shape[0][3])),
     File "C:\Users\lightland\Anaconda3\lib\site-
       packages\tensorflow\python\framework\tensor_shape.py", line 615, in
         __getitem__   return self._dims[key]
     IndexError: list index out of range

in AttentionLayer class , build function id defined by:

 def build(self, input_shape):

      assert isinstance(input_shape, list)
      print("hhhhhhhhhh",input_shape)
      print("jjknkjnjk")
      # Create a trainable weight variable for this layer.

       self.W_a = self.add_weight(name='W_a',
                               shape=tf.TensorShape((input_shape[0][2],
                                input_shape[0][2])),
                               initializer='uniform',
                               trainable=True)
       self.U_a = self.add_weight(name='U_a',
                               shape=tf.TensorShape((input_shape[1][2],
                               input_shape[0][2])),
                               initializer='uniform',
                               trainable=True)
       self.V_a = self.add_weight(name='V_a',
                               shape=tf.TensorShape((input_shape[0][2], 1)),
                               initializer='uniform',
                               trainable=True)

       super(AttentionLayer, self).build(input_shape)

If someone can help me I'll be so thankful, I cannot understant where the problem is, and how to resolve it.

Thanks in advance

Maybe you are not using the attention.py you think you are using. The error states that the problematic line in Attention build is:

shape=tf.TensorShape((input_shape[0][3], input_shape[0][3])),

but the code of attention.py you shared with us does not seem to contain such line.
It is possible you also just forgot to save your changes to the new attention.py file.


How would keras model learn at every epoch? I know concept of Epochs, batch size and iteration.

let's say,

Total_data = 6400

Batch_size = 64

Iteration = 100

In this, basically we are taking in 64 data points to computer memory and calculate and at each iteration we get weight updated. so after 100 iterations, we will fulfill one epoch.

My question is, after one epoch, we are using again the same 6400 data. How is it different from the first epoch in terms of learning? does the model select different 64 data point than first epoch in second epoch and try to learn? how does it internally really work?

I wish I can get some clear answers.

Thank you people in advance.

The data points are the same, but the model is different at each epoch. Weights are updated at each iteration so they will be different and in turn, the loss function will be different for the same data point at different epochs. What training process does is to change the weights slowly at each run to minimize the loss for the same data smaller and smaller.

Weights are modified in a way to minimize the error for the data points using an algorithm such as backpropagation in gradient descent. You might also want to check the explanation of backpropagation given by 3B1B in this youtube video.


Fractions or probabilities as training labels This it a problem that has come on my path a few times now and I don't have a satisfying solution yet. The goal is to predict probabilities or fractions based on some $x$ where our training $y$ has these probabilties or fractions and thus is in the domain $[0,1]$ as opposed to $\{0,1\}$. My question is with regards to my loss function. In the case of fractions, if the error between 0.4 and 0.5 and the error between 0.89 and 0.99 is the same I can just use MSE if I want to predict the expected value.

In case of probabilities where we want to approach it similarly as classification problems, where the difference between 0.89 and 0.99 is much bigger than that of 0.4 and 0.5, we want to put this in our loss function. Does cross entropy still work properly if I feed it fractions in $y$?

$\mathcal{L}(y,\hat{y})=-y\log(\hat{y}) - (1-y)\log(1-\hat{y})$

Let's say our $y=0.5$ and our current prediction is $\hat{y}=0.6$ we would get:

$\mathcal{L}(0.5,0.6)=-0.5\log(0.6) - 0.5\log(0.4)$

I don't really see why this would go wrong? The function is still convex. Everywhere it says that the target should be in $\{0, 1\}$ however. Maybe my math is lacking or I'm missing something obvious, why is this a bad idea?

Cross-entropy loss still works with probabilities in $[0,1]$ as well as $\{0,1\}$. Most importantly, $\hat{y} = y$ is still a stationary point (although it will not equal $0$).

It is also the case that the possible improvement in loss (and immediate gradient) is larger for $\hat{y} = 0.99, y = 0.89$ than for $\hat{y} = 0.4, y = 0.5$. If you use a sigmoid output, then the gradient at the logit $\hat{y} - y$ still applies - the larger gradient of the loss function scales inversely to the lower gradient of the sigmoid at that point.

So, in short, yes use binary cross-entropy loss for single-class probabilities, even when they are not strictly in $\{0,1\}$.


What would I prefer - an over-fitted model or a less accurate model? Let's say we have two models trained. And let's say we are looking for good accuracy.
The first has an accuracy of 100% on training set and 84% on test set. Clearly over-fitted.
The second has an accuracy of 83% on training set and 83% on test set.

On the one hand, model #1 is over-fitted but on the other hand it still yields better performance on an unseen test set than the good general model in #2.

Which model would you choose to use in production? The First or the Second and why?

There are a couple of nuances here.

Complexity question very important - ocams razor
CV - is this trully the case 84%/83% (test it for train+test with CV)

Given this, personal opinion: Second one.

Better to catch general patterns. You already know that first model failed on that because of the train and test difference. 1% says nothing.


What does the embedding mean in the FaceNet? I am reading the paper about FaceNet but I can't get what does the embedding mean in this paper? Is it a hidden layer of the deep CNN?
P.S. English isn't my native language.

Assume you have features wich lie in a $R^n$ space, e.g. your input is a picture with $28 \times 28$ pixels, then $n$ would be $28 \times 28 = 784$.

Now you can "embedd" your features into another $R^d$ space, where often $d < n$. This way you learn a rich representation of your input. When you compress your $784$ input-pixels to, lets say $64$ you compressed your input by more than a factor $10$ and can elimnate redundant/useless features.

This embedding learning is of course done in such a way, that you could fully restore your original $784$ pixels out of your $64$ compressed pixels.


regex to remove repeating words in a sentence I am new to regex. I am working on a project where i need to replace repeating words with that word. for example:

  I need need to learn regex regex from scratch.

I need to change it to:

  I need to learn regex from scratch.

I can identify the repeating words using the following regex
\b(\w+)\b[\s\r\n]*(\l[\s\r\n])+

For substituting it, I need the word in the repeated word phrase.
pattern.sub(sentence, <what do i write here?>)

Since you were working with RegEx, I will ofer a RegEx solution.

I will also show that you need to also take care to first remove punctuation. (I will not go down the rabbit-hole of re-sinserting the punctuation back where it was!)

A RegEx solution:

import re
sentence = 'I need need to learn regex... regex from scratch!'

# remove punctuation
# the unicode flag makes it work for more letter types (non-ascii)
no_punc = re.sub(r'[^\w\s]', '', sentence, re.UNICODE)
print('No punctuation:', no_punc)

# remove duplicates
re_output = re.sub(r'\b(\w+)( \1\b)+', r'\1', no_punc)
print('No duplicates:', re_output)

Returns:

No punctuation: I need need to learn regex regex from scratch
No duplicates: I need to learn regex from scratch

\b : matches word boundaries
\w : any word character
\1 : replaces the matches with the second word found - the group in the second set of parentheses

The parts in parentheses are referred to as groups, and you can do things like name them and refer to them later in a regex. This pattern should recursively catch repeating words, so if there were 10 in a row, they get replaced with just the final occurence.

Have a look here for more detailed definitions of the regex patterns.

The more pythonic (looking) way

It has to be said that the groupby method has a certain python-zen feel about it! Simple, easy to read, beautiful.

Here I just show another way of removing the punctuation, making use of the string module, translating any punctuation characters into None (which removes them):

from itertools import groupby
import string

sentence = 'I need need to learn regex... regex from scratch!'

# Remove punctuation
sent_map = sentence.maketrans(dict.fromkeys(string.punctuation))
sent_clean = sentence.translate(sent_map)
print('Clean sentence:', sent_clean)

no_dupes = ([k for k, v in groupby(sent_clean.split())])
print('No duplicates:', no_dupes)

# Put the list back together into a sentence
groupby_output = ' '.join(no_dupes)
print('Final output:', groupby_output)

# At least for this toy example, the outputs are identical:
print('Identical output:', re_output == groupby_output)

Returns:

Clean sentence: I need need to learn regex regex from scratch
No duplicates: ['I', 'need', 'to', 'learn', 'regex', 'from', 'scratch']
Final output: I need to learn regex from scratch
Identical output: True

Benchmarks

Out of curiosity, I dumped the lines above into functions and ran a simple benchmark:

RegEx:

In [1]: %timeit remove_regex(sentence)
8.17 µs ± 88.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

groupby:

In [2]: %timeit remove_groupby(sentence)
5.89 µs ± 527 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

I had read that regex would be faster these days (using Python3.6) - but it seems that sticking to beautiful code pays off in this case!

Disclaimer: the example sentence was very short. This result might not scale to sentences with more/less repeated words and punctuation!


How to improve an existing machine learning classifier in python? I have a big dataset (1million x 50) to which I want to predict a particular class. I have thought of segregating the dataset in batches of 20k. And then train a classifier (lets say random forest or a basic SVM). How do I then improve that classifier by providing it with additional dataset. In other words, how do I preserve the random forests created in iteration i and use as starting model in interation i+1 to improve the model in python?

The simplest, and likely best, way is to just rebuild the model every time you add more data.

If you want to build on existing models, one way is to average multiple random forest models.  So you build a model on 20k, test it, then decide to build on it by building another random forest model on a separate set of 20k and averaging the model parameter results with the previous model.  This is basically a poorly described ensemble method.  Check out this: http://scikit-learn.org/stable/modules/ensemble.html.


When does it makes senses to use Dot-Product as similarity measure instead of Cosine? When does it makes senses to use Dot-Product as similarity measure instead of Cosine?. I have seen there is already question asked about this, However that merely explains the difference between calculation of dot-product & cosine and it does not focus on when should we use one vs another with real world example.

Borrowing from this Quora answer -- which includes some concrete examples of when you might prefer one measure over the other -- it comes down to whether you care about taking the magnitude of the vectors into account. This is highly domain-specific, but the example used there is in information retrieval, where magnitude represents the length of the documents in question. The dot product will take the document length into account, whereas the cosine similarity will not.

In general, you will want to have good reasons specific to your problem space to use one over the other.


How to handle preprocessing (StandardScaler, LabelEncoder) when using data generator to train? So, I have a dataset that is too big to load into memory all at once. Therefore I want to use a generator to load batches of data to train on.

In this scenario, how do I go about performing scaling of the features using LabelEncoder + StandardScaler from scikitlearn?

Some more context:

I have 10million+ samples of data with 23 features and 1 label column in a database.

My setup used to be (when it was ~3 million samples) to load in pandas with sql, perform some more feature extractions, use LabelEncoder on some features, do train/test split and then use StandardScaler on the training features. And then fit my keras model.

However this workflow is no longer possible on my machines because of the amount of data. (MemoryErrors.)

I'm looking into using keras.utils.Sequence to load batches of data instead of everything in memory at once, this way i would only need to have the complete list of indexes, and one full batch in memory at a time.

However how would I go about LabelEncoding and more importantly: How would I go about feature scaling in this scenario? And given the context, is this a correct approach?

It is a correct approach to standardize on your training features. In that way, you ensure not to give any information from the testing set to the training set.
About features scaling, if you have too many samples to fit your scaler at once, you could use the partial_fit (see here) method of StandardScaler in sklearn. Load sequentially your training features and do the partial_fit. Once finished, your scaler is ready to be used on your training/testing batches.
About label encoding, either you already have an array containing all the labels, so you can fit your LabelEncoder, or you would have to load sequentially all your data to get all the different labels before fitting (LabelEncoder does not have partial_fit method).


How do i get dummies for this dataset I am using an udemy course for MachineLearning and I am trying to form a dummy for my variable the column is Country
I want to change to   France   Germany   Spain
- France                             1        0       0
- Spain                              0        0       1
- Germany                            0        1       0  etc

I tried this but I got this error
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
labelencoder_X = LabelEncoder()
X[:, 0] = labelencoder_X.fit_transform(X[:, 0])
onehotencoder = OneHotEncoder(categorical_values = [0])
X = onehotencoder.fit_transform(X).toarray()

Traceback (most recent call last):

  File "<ipython-input-9-f7654080cf6a>", line 4, in <module>
    onehotencoder = OneHotEncoder(categorical_values = [0])

TypeError: __init__() got an unexpected keyword argument 'categorical_values'

As the error says: there is no categorical_values parameter for OneHotEncoder.  It was removed at the same time that OneHotEncoder was extended to deal with strings directly, and you may want to use ColumnTransformer for selecting out the categorical column(s).  For example, https://datascience.stackexchange.com/a/57383/55122


How to detect if one tweet is agreeing with another I want to detect tweet text agreement. Suppose someone posts some subjective opinion in twitter. Other users will post reply either agreeing or opposing the original tweet. I want to estimate the amount of agreement. Is there any algorithm/library in any language to do that or any labeled dataset?

Not sure there is anything for that, you could check

is it a verbatim retweet
does it have the same sentiment
is the edit distance low

Or you can train your own model, where you label (agree) by hand and then build features.


Python sklearn model.predict() gives me different results depending on the amount of data I train my XGBoostClassifier().
If my testing set has:
0: 100
1: 884

It attempts to predict 210 1's. Around 147 are wrong (False positives) and 63 1's correctly predicted (True positives).
Then I increase my testing sample:
0: 15,000
1: 884

It attempts to predict 56 1's. Around 40 are wrong (False positives) and 16 1's correctly predicted (True positives).
Am I missing something? some theory? some indication on how to use model.predict(X_test)?
Does it say somewhere - if you try to predict 10 items is gonna try harder than if you try to predict 10000 items? In what situation model.predict(X_test) would give me a different result for Joe Smith if his prediction is accompanied by 8000 more rows?
The code I use is the following:
from xgboost import XGBClassifier
xgb = XGBClassifier(subsample=0.75,scale_post_weight=30,min_child_weight=1,max_depth=3,gamma=5,colsample_bytree=0.75)
model = xgb.fit(X_train,y_train)
y_pred_output = model.predict(X_test)
cm = confusion_matrix(y_test, y_pred_output)

y_pred_output2 = model.predict(X_test2) #contains the same 884 1's plus 14500 more rows with 0's as the target value
cm = confusion_matrix(y_test2, y_pred_output2)

it produces two different matrices:
#Confusion matrix for y_test with 15000 0's and 884 1's
[[14864   136]
 [  837    47]]

#Confusion matrix for y_test with 500 0's and 884 1's
[[459  41]
 [681 203]]

Notice that the same 884 positive class items are being used across both attempts. Why would the true positives go down to 47 just because we now have more Negatives on the X_test?

If XGBoostClassifier is fed the same input data over and over again it will yield the same results. There is no inherent randomness in this classifier that would different results for the same input. Additionally - there should be no difference in the result of an individual prediction if it's requested in a smaller batch versus a larger batch (again the result will be identical).
On the other hand - if you train XGBoost on different data their outputs will definitely be different. If you add new data to the underlying dataset & train with that - new & different patterns will emerge that XGBoost will try to take advantage of and the entire tree network will be fit very differently.
I suspect what you are observing is a bug in structuring your input data that you are then feeding to the .predict() method. If you share a sample of your code maybe we can drill-down on the issue.


Is it Possible to Use Machine Learning for Ranking Alternatives? Right now, I’m working on road and street safety analysis. I have a dataset of dangerous points in four regions of a city. Some of the available variables  are road lighting status, ITS, latitude, longitude, longitudinal protection status, type of point control, as well as the number of injuries and deaths for the past two years. Also, there are some dummy variables(road lighting status, ITS, longitudinal protection status, type of point control). I want to know whether I could rank these points from the most dangerous one to the least dangerous one based on their potential danger using machine learning methods. If possible, which methods could help? I would appreciate it if you could share relevant links or references. Thanks in advance.

Yes, you can rank those data points.

First, you need to define a quantitative measure of "potential danger" called a target variable. That could be a single variable (e.g., number of accidents) or weighted collection of variables, frequently called an index.

Then a regression model can be trained to predict that continuously-measured target variable and the data points can be rank-ordered.


predicting next jobtitle I have a dataset of which has 30M rows each like [current_jobtitles, nextjobtitles].
[['junior software programmer', 'senior software programmer'],
 ['senior software programmer', 'lead software programmer'],
 ['sales associate', 'regional sales associate']]

I want to build a deep learning model to predict the nextjobtitle when a currenttitle is given. Are there any ways that I could acheieve this using some deep learning model? if yes, what kind of model ?
Can we use any of the text generation models for this scenario ? I have seen some examples, but I cannot relate to my solution for the problem. Any suggestions please ..!
update:
I have sequences of jobtitles in each row.
[['junior programmer', 'senior programmer', 'lead programmer'],
 ['sales associate', 'sales regional associate', 'sales manager'],
 ['chairman', 'ceo'],
 ['associate', 'intern', 'junior', 'student']]

for the orginal data (in update) i have modifed data as above (before update) by just taking the nextjob title.
Can I train model with the original sequence data to predict the next jobtitle?
** - all the rows will be of different lengths.

I want to build a deep learning model to predict the next job title when
  a current title is given. Are there any ways that I could achieve this
  using some deep learning model?

I think that approaching this problem in the classification way (input:- current job embedding, output:- getting next job title as a class) can somewhat be time-consuming (not necessarily hard) to train.

The reason behind this:-

Assuming that each of the job-title pair is unique in your dataset,
given the volume of your dataset, there can be many classes, so much that even deep learning models may take time(for processing, optimization, etc.) to train.
Also as there can be many to many relationships in the dataset, hence using the classic approach might not be a good option.

  what kind of model?

As per your given examples, the kind of model that we need here is one, that can learn the hierarchy of the job titles from the text itself, except for some cases where the sample denotes that the individual of that job title has changed their domain of work(this can be attributed as noise).

  Can we use any of the text generation models for this scenario?

A deep learning model such as seq2seq(https://www.geeksforgeeks.org/seq2seq-model-in-machine-learning/) or noisy autoencoders(https://towardsdatascience.com/convolutional-autoencoders-for-image-noise-reduction-32fce9fc1763). I know that there is not much content of 'noisy autoencoders for nlp' on the internet, but we can always try. These models can learn the complex relationships between the words of the job title and the job title itself.

I would recommend noisy-autoencoders as I assume (a strong one) that only some if not all corresponding words in the current and next job title are different, and this is the purpose of noisy autoencoders:-

They can accommodate noise very well(many to one and one to many
relationships between current and next job title)
They keep most of the content in input and output is the same, which I think you are gonna need because of the strong assumption I mentioned above.

I hope it helps, thanks!!


YOLO Dense Prediction I have two questions about dense prediction in YOLOv4 paper

What does it mean by the (hard negative, online hard) example mining method is not applicable to one-stage object detector, because this kind of detector belongs to the dense prediction architecture ?
Why dense prediction does not belong to two-stage detector ?

It is kind of unfortunate that no further explanations were given in the paper about this. In my opinion Hard negative mining is actively used by architectures like SSD, actually by boosting its performance with the use of selective training.
I could speculate about what the authors meant, namely that a one-stage detector does not have per construction any element capable of performing the hard negative mining, whereas Faster-RCNN (sparse predictor) does as it relies on two different stages (classification/bounding box regression + RoI generation) to discriminate the results actively.

Dense prediction means that RoIs are a dense sample of possible boxes around points of interest (image or feature maps). Object Detectors whose output is the subset of the top relevant RoIs (and output features) out of these are dense predictors. They are planned as a self-contained framework that runs in a one-stage manner. Dense prediction is also part of two-stage detectors as you can see in the image, but specific boxes are used for further processing (e.g. a box refinement method).


Permutation test on two groups I am trying to use a permutation test to test my hypothesis.
I want to make sure I am understanding concept of permutation correctly.
I have control and experimental group. Then I combine them and resample from combined dataset randomly calculating desired statistics. Do this N time and distribute them on a histogram to see where my observed statistics stand.
In the book "practical statistics for data scientists" it says that from combined data (without replacing) resample same size as groupA in my case control group. Then since I have only two groups rest are simply groupB(experimental group)?
So my question simply put, when using permutation test on two groups first you resample size of groupA then rest automatically becomes resample of groupB?

Yes it does, you resample from the whole population without replacement $N_A$ samples, which are associated to group $A$, and the rest of the samples are associated to group $B$. As I understand, you compute the statistic for every possible permutation of the total population and you measure how the actual statistic compares in the distribution of all the permutations.
If your statistic is not extreme in the distribution among all possible permutations, then it's likely that $H_0$ is true. Otherwise, if your statistic is an outlier of that distribution it's likely that $H_0$ is false.


Machine learning PhD Interview technical questions I'm Software Engineer who applied to grad school for Machine Learning/Computer Vision PhD and currently waiting for interview calls. I'm brushing up Linear algebra/ ML topics. What kind of technical questions do professors generally ask in Ph.D interviews ?

Thanks in advance.

In general, linear algebra and basic concepts in statistics (probability distributions, marginalization) and machine learning (you should be familiar with terns like Maximum-A-Posteriori estimation and Maximum-Likelihood-Estimation, maybe Markov chains and SVMs, neural-networks, etc.).

Best practice would be to find out who is interviewing you and than look at what courses he is leading and what papers he has published in recent years. He is most likely to ask specific question from his field of expertise and interests.


Is there any kaggle competition for finding the feature for affecting revenue? I have a question about finding the most significant feature of the data that affects the company revenue. I have data set that contains car share company data (car model, completed trips, lead_time, trip length, trip revenue, delivery_fee, net_revenue ...) columns.

Is there any kaggle competition out there doing EDA (Explotary data analysis) not prediction for finding the most significiant feature that affects the net_revenue or sales ? Could not find and uber of lyft data competition for this question!

I would appreciate If you happen to know such data set and share the link with me!

Thanks in advance!

Is there any kaggle competition out there doing EDA (Explotary data analysis) not prediction for finding the most significiant feature that affects the net_revenue or sales ?

Although it is hard to prove a negative, I would say "no" to this.

Kaggle competitions are based on continuous metrics that can be ranked, such as getting best log loss, mean average precision etc.

Identifying the "most significant feature", although a potentially useful question to answer from a data set, is not something that can be ranked automatically and objectively in a competition.

It is possible you will find competitors performing that kind of analysis within a prediction challenge. It is fairly common to rank features by relevance, especially when using ML techniques that support doing so, such as random forests or xgboost. I am pretty sure I have seen forum posts in Kaggle discussing these things and searches for a "golden feature" which is a similar concept - usually some processed combination of one or more original features.

In addition, there have been a few competitions where the entrants were expected to produce supporting documentation. Discussion of feature relevance could be an important part of such entries. These are much rarer competitions at Kaggle than the more direct finding best predictive model.

So you may find competitions where identifying a "best feature" is part of the winning strategy, but the goal of the competion will likely be getting the best mean log loss on the test set.


How to compare different similarity measurements in text clustering? I have a dataset which contains vectors generated from subtitles (each column represents a genre, each row is a movie name), my purpose is to find the most similar movie titles, I want to use different distance/similarity measurements and compare them, what is the best method to use?

For now, I tried L1 distance, cosine similarity, Euclidean distance, Mahalanobis Distance, I got the results of top n most similar titles, but all the results seem very reasonable, how can I compare them to see which method perform best?

I also tried to do k-means, when I implement K-means clustering, it used Euclidean distance by default, how to use other distance to implement K-means?Also any suggestions about other similarity measurements? Many thanks

I got the results of top n most similar titles, but all the results seem very reasonable, how can I compare them to see which method performs best?

I'm afraid there's no meaningful definition of "best" unless you specify some concrete metric.  If the results in each case seem reasonable, then all you can do is be educated about the various distance metrics and decide which makes the most sense for your use-case.  (For example, Euclidean distance measures spatial difference, while cosine similarity measures directional difference.)

  I also tried to do k-means, when I implement K-means clustering, it used Euclidean distance by default, how to use other distance to implement K-means?

Keep in mind that k-means is designed for Euclidean distance, which is why many implementations (such as scikit-learn) do not allow the use of any other distance function.  For many other distance metrics, such as cosine similarity, taking the mean of the points in the cluster is not appropriate, so you'd also need to replace the centroid-estimation function as well.

If you really want to, then the K-Means implementation from NLTK allows you to specify a custom distance metric.  Just keep in mind the GIGO principle.  If you violate the assumptions of the model, don't be surprised if you get poor results.


What does the below phrase in the lstm blog mean? - Data Science I am a newbie to data science. I was reading this blog

When I was half way through, I came into this sentence

  Further, each series of data has been partitioned into overlapping
  windows of 2.56 seconds of data, or 128 time steps. These windows of
  data correspond to the windows of engineered features (rows) in the
  previous section.

What does 'series' means here and what is window partitioning?

Further I would like to ask all the data science enthusiasts there,What do I lack here.Is it comprehension or logical skills needed .What can I do to improve upon this.I encounter such doubts often and what I do is search for the terms I dont understand or leave it and get ahead.When I search for the terms I get broad answers leaving me more confused.If i leave it like that I dont understand the rest.What should I do here?I cant post every small doubts in stack exchange.It would be great If you could share some light on this as well.

I think, with "series", the authors refer to time series data in their dataset. In this case, it should be x, y, z, angular velocity and maybe other features for each participant.

"Splitting data into fixed windows of 2.56 seconds (128 data points) with 50% overlap." I would translate to:
Every 2.56 seconds aggregate each series of the data over the next 5.12 seconds.

You can imagine windowing basically as a timestamp-based data aggregation technique that is often used for stream data processing to reduce data and fix granularity of the data. The stream analytics windowing function documentation contains visualizations, which should help you to understand what is meant. I think, in the blog, the authors mean what Microsoft calls "hopping windows".

I read that windowing is a time series feature engineering topic. So maybe, it might help, if you learn more about feature engineering.


Can RNN be used for feature extraction? I was reading this paper and my question is related to it. What I am stuck at is the intuition behind using two CNNs for feature extraction. Can just RNN not be used for feature extraction as well as for capturing temporal relationship?

In theory, any neural network can be used for feature extraction since neural nets perform it by themselves.

Whether or not using CNNs vs RNNs is a different question. CNNs are typically more computationally efficient whilst RNNs are more flexible but typically slightly harder to train.


Machine Learning - How to predict set of fixed fields based on past features I have quite a large dataset (> 100k rows), which contains information for logistical shipments. (export shipments)

The dataset looks like this:

|shipper|consignee                    |origin|destination                                  |
|-------|-----------------------------|------|---------------------------------------------|
|6409292|288882                       |USSFO |CNPVG                                        |
|6409292|288882                       |USSFO |CNPVG                                        |
|6409292|182724                       |USSFO |HKHKG                                        |
|6409292|182724                       |USSFO |HKHKG                                        |
|8201922|948292                       |USSFO |FRCDG                                        |
|8201922|948292                       |USSFO |FRCDG                                        |
|8201922|948292                       |USSFO |FRNIC                                        |
|8201922|291222                       |USEWR |AEDXB                                        |

So what we have here is a list of past shipments. It shows the relationship between shipper and consignee, and from where the shipment was from and where it was sent to.

Based on this past data, I wish to be able to predict when a new shipment is added by looking at the consignee code and origin.

Example

Take below new booking as an example:

|shipper|consignee                    |origin|destination                                  |
|-------|-----------------------------|------|---------------------------------------------|
|1234567|948292                       |USMOB |?                                            |

How can I train a model to predict the destination? And what is this area in ML referred to?

It's a supervised classification problem: you're trying to predict the destination (class) based on some categorical features (input columns). I would suggest starting with some simple algorithms such as decision trees or Naive Bayes.

However I'm guessing that logistical shipments can evolve over time: maybe a shipper business grows with country X but decreases with country Y, etc. If this is relevant, it might make sense to look into more advanced methods which could take the chronological evolution into account (time series).


How (and why) are random forests able to represent both linear and non linear data From reading articles I've read that random forests are suited for representing both linear and non linear data quite well, but I can't find an explanation as to how and why it has his flexibility?

Any information to clarify this would be great!

Decision Trees are able to create both linear and nonlinear boundaries and so are Random Forests.

This is because of how they cluster the data based on nested "if-else" statements. These statements draw vertical/horizontal lines between the samples and cluster them in rectangles. Consecutively, rectangles of the same class can be far away from each other (with other class rectangles in between) but still they belong to the same class. This is how nonlinear relations are modeled.


There is no guarantee that you get the same subset of features from any feature selection algorithm. There might be multiple subsets that all perform approximately equally.  For instance, perhaps feature C is highly correlated with feature I.  So, there is no reason to be surprised by what you are observing.  There is no notion of "too low".


Tensorflow neural network TypeError: Fetch argument has invalid type I am making a simple neural network using tensorflow, with data I collected myself, however, it's not cooperating :P I have run into an error that I can't fix or find the fix for and I would love your help.
The errormessage:
TypeError: Fetch argument 2861.6152 of 2861.6152 has invalid type <class 'numpy.float32'>, must be a string or Tensor. (Can not convert a float32 into a Tensor or Operation.)

The error refers to the following line in my code:
_, cost = tf_session.run([optimizer, cost], feed_dict = {champion_data: batch_input, item_data: batch_output})

I have already figured out that the error doesn't occur when I comment out the following lines in my code:
prediction = neural_network_model(champion_data)
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(prediction, item_data))
optimizer = tf.train.AdamOptimizer().minimize(cost)
_, cost = tf_session.run([optimizer, cost], feed_dict = {champion_data: batch_input, item_data: batch_output})

So somewhere one of these lines is getting something that doesn't look exactly like it expects it to look. I have already tried the obvious (removing the np.array() from batch_input and batch_output or replacing it by list()) but that doesn't solve the problem. My current hypothesis is that the output of neural_network_model(champion_data) is somehow of the wrong shape or type, however I'm not sure how to test that or how to solve it if that turns out to be the case.
The full code can be found here.
Edit: I have verified that the champion data that are input into the neural_network_model, the prediction and the cost are all tensors. I've been trying to solve the problem using the hypothesis that the problem somehow lies in the feed_dict = {} part of the code, but not getting anywhere so far

The problem lay in using the name 'cost' on two occasions, the problem was solved by changing this:

_, cost = tf_session.run([optimizer, cost], feed_dict = {champion_data: batch_input, item_data: batch_output})

to this:

_, c = tf_session.run([optimizer, cost], feed_dict = {champion_data: batch_input, item_data: batch_output})

This way the name of the variable 'c' doesn't clash anymore with the [optimizer, cost] part of the code.


Display Images (url) Inside Pandas Dataframe I would like to display images (mostly jpg and png formats) directly from their url link inside a pandas dataframe. Imagine I already have the following dataframe:

id image_url
1  http://www.nstravel.ro/wp-content/uploads/2015/02/germany-profile_6023_600x450.jpg
2  https://www.nh-hotels.com/multimedia/images/cityscape-of-berlin_retoque_600x458-tcm41-132903-32.jpg
3  https://www.scandichotels.com/imagevault/publishedmedia/8q33xlshjl4o9d5ftp3r/germany-berlin-berliner-dom-cathedral-in-the-eveni.jpg

I would like to have the actual image displayed in another column (properly resized to fit).

I am aware that I could do it differently, e.g. not using pandas and go with grids using matplotlib, but for ease of illustration and presentation I would prefer to show it within a table (dataframe).

I have found these solutions: solution1, solution2. These suggested solutions while seem very relevant but did not work for me. Even I tried with images from my local disk, but I get this using the solution2:

Python: 3.6.5, Pandas: 0.23.0, Jupyter: 4.4.0

Thanks!

Actually the solution2 worked; I just had to be a bit more patient. I am posting it here in case someone have difficulties, like me, getting this to work:

import pandas as pd
from IPython.display import Image, HTML

def path_to_image_html(path):
    '''
     This function essentially convert the image url to
     '' format. And one can put any
     formatting adjustments to control the height, aspect ratio, size etc.
     within as in the below example.
    '''

    return ''

HTML(df.to_html(escape=False ,formatters=dict(column_name_with_image_links=path_to_image_html)))

Note that df is your actual dataframe name, column_name_with_image_links is the column name that contains all image urls, and path_to_image_html is the above function.


Are single input single output neural networks possible? This might be a weird question but I'm trying to have a deep understanding of how neural networks work theoretically.

I was doing some tests with my perceptron and I decided to test it on a single input single output dataset. What I was looking for was 100% accuracy since what I was testing on was a trivial separable dataset with binary output. However, the accuracy was below 50% (it wasn't balanced). I realized that this is due to the fact that there is a single weight that when trained is either going to be =>0 or <0. So after going through the activation function (e.g. hardlim), unless the input is scaled to be [-1,1] range, it's always going to give the same output.

Does my explanation make sense? Are there any theoretical aspects I'm missing?

Also, are neural networks useful at all for single input problems? They seems useless to me since we're basically classifying by putting a point or points (for multiclass problems) on a single line to separate outputs, which is a pretty simple problem that doesn't need the intricacy of neural networks.

You have saturation of the weights which results in constant predictions.
There are several ways to fix that:

Replace hardlim with a standard activation function like ReLU or sigmoid
Pick a different random initialization
Normalize inputs


Why would a fake feature with random numbers get selected in feature importance? I'm using a sklearn.ensemble.RandomForestClassifier(n_estimators=100) to work on this challenge:
https://kaggle.com/c/two-sigma-financial-news

I've plotted my feature importance:

I created a fake feature called random which is just numbers pulled from np.random.randn(). Unfortunately, it seems to have quite significant feature importance.

How am I supposed to interpret this? I had expected it to be at the bottom.

PS xgboost seems to discard this feature, as it should.

Scikit-learn's random forest feature importance is based on mean decrease in impurity, which is fast to compute and faithful to the original creation of the Random Forest method. In short, the default feature_importances_ gives a numerical justification of the Random Forest's representation of the feature importance using its native metric of construction. Like you saw, this metric has the drawback that it can say noise is an important feature. So you may want to consider other feature importance methods, like permutation feature importance, which will give you a more apples-to-apples comparison with other models you will test. There are pros and cons to many of these methods so be aware of them.


How can I get a measure of the semantic similarity of words? What is the best way to figure out the semantic similarity of words? Word2Vec is okay, but not ideal:

# Using the 840B word Common Crawl GloVe vectors with gensim:

# 'hot' is closer to 'cold' than 'warm'
In [7]: model.similarity('hot', 'cold')
Out[7]: 0.59720456121072973

In [8]: model.similarity('hot', 'warm')
Out[8]: 0.56784095376659627

# Cold is much closer to 'hot' than 'popular'
In [9]: model.similarity('hot', 'popular')
Out[9]: 0.33708479049537632

NLTK's Wordnet methods appear to just give up:

In [25]: print wn.synset('hot.a.01').path_similarity(wn.synset('warm.a.01'))
None

What are other options?

Word2vec does not capture similarity based on antonyms and synonyms. Word2vec would give a higher similarity if the two words have the similar context. Eg
The weather in California was _____ .
The blank could be filled by both hot and cold hence the similarity would be higher. This concept is called Paradigmatic relations.
If you are interested to capture relations such as hypernyms, hyponyms, synonyms, antonym you would have to use any wordnet based similarity measure. There are many similarity measures based on wordnet. You may check this link


prediction() returning mistakenly false positives I do not know how to interpret the result of:

prediction(c(1,1,0,0),
           c(1,1,0,0))

prediction() functino comes from prediction {ROCR}  it has this site: http://rocr.bioinf.mpi-sb.mpg.de/

The above is a working example. As per the documentation the first parameter is 'predictions' and the second 'labels' (they would be the true values).

The output is this, which I do not fully understand, specially why there is a '2' in "fp". :

An object of class "prediction" Slot "predictions": [[1]] [1] 1 1 0 0

Slot "labels": [[1]] [1] 1 1 0 0 Levels: 0 < 1

Slot "cutoffs": [[1]] [1] Inf   1   0

Slot "fp": [[1]] [1] 0 0 2

The slot "fp" counts how many false positives there are at each choice of classification cutoff (which can be found in the "cutoff" slot). The cutoff represents at what value you set the threshold to binarize the numerical values into classes. Your output already appears to be binary classes, so the concept of a threshold doesn't really make much sense, but the package still tries, setting the potential thresholds at Infinity, 0, and 1. When you set the threshold at 0, everything gets classified positive, including the 2 actual negative samples - when the classification threshold is 0, you get 2 false positives.


Big difference in randomTree accuracy with train and test sets I've created a model using randomForest for the following dataset: https://archive.ics.uci.edu/ml/datasets/Contraceptive+Method+Choice

The thing i'm questioning is that the results of the model when used on my training and testing sets are vastly different.

library(randomForest)
library(caret)
df <- read.csv('cmc.csv')

Changing values to factors

df$Wife.s.education <- as.factor(df$Wife.s.education)
df$Husband.s.education <- as.factor(df$Husband.s.education)
df$Wife.s.religion <- as.factor(df$Wife.s.religion)
df$Wife.s.now.working. <- as.factor(df$Wife.s.now.working.)
df$Husband.s.occupation <- as.factor(df$Husband.s.occupation)
df$Standard.of.living.index <- as.factor(df$Standard.of.living.index)
df$Media.exposure <- as.factor(df$Media.exposure)
#add string representation for readiblilty
df[df$Contraceptive.method.used == 1,]$Contraceptive.method.used <- "No-use"
df[df$Contraceptive.method.used == 2,]$Contraceptive.method.used <- "Long-term"
df[df$Contraceptive.method.used == 3,]$Contraceptive.method.used <- "Short-term"
df$Contraceptive.method.used <- as.factor(df$Contraceptive.method.used)

Splitting data:

set.seed(47)
ind <- sample(2, nrow(df), replace = TRUE, prob = c(0.7,0.3))
inTrain <- createDataPartition(y=df$Contraceptive.method.used, p=.7, list = FALSE)
training = df[ind==1,]
testing = df[ind==2,]

Creating model:

model <- randomForest(Contraceptive.method.used ~., data = training, proximity=TRUE)

#Prediction & Confusion Matrix - training data
p1 <- predict(model, training)
confusionMatrix(p1, training$Contraceptive.method.used)

Confusion Matrix Results (training):

          Reference
Prediction   Long-term No-use Short-term
  Long-term        208      9         18
  No-use             4    408          5
  Short-term        16     14        338

Overall Statistics

               Accuracy : 0.9353
                 95% CI : (0.9184, 0.9496)
    No Information Rate : 0.4225
    P-Value [Acc > NIR] : < 2e-16

                  Kappa : 0.9002

 Mcnemar's Test P-Value : 0.09773

#Prediction & Confusion Matrix - testing data
p2 <- predict(model, testing)
confusionMatrix(p2, testing$Contraceptive.method.used)

Confusion Matrix Results (testing)

    Reference
Prediction   Long-term No-use Short-term
  Long-term         42     11         23
  No-use            27    122         47
  Short-term        36     65         80

Overall Statistics

               Accuracy : 0.5386
                 95% CI : (0.4915, 0.5853)
    No Information Rate : 0.4371
    P-Value [Acc > NIR] : 8.972e-06

                  Kappa : 0.2788

 Mcnemar's Test P-Value : 0.005869

As we can see theres is a massive change in the two results, if i print my model I get the following results:

     Type of random forest: classification
                     Number of trees: 500
No. of variables tried at each split: 3

        OOB estimate of  error rate: 46.96%
Confusion matrix:
           Long-term No-use Short-term class.error
Long-term         76     63         89   0.6666667
No-use            45    282        104   0.3457077
Short-term        72    106        183   0.4930748

This makes me belief that the test results are correct however I'm not sure why there is such a big difference when used on the training set, is this due to overfitting? If so how can this be handled?

Any guidance would be awesome.

Overfitting is a common problem in random forest, you can use cross Validation methods (Cross validation is a technique to build models that are not prone to overfitting) example - K-fold cross validation, stratified k fold..

Try this using train package-
control <- trainControl(method = "cv", number = 5)
and add this as a control parameter to the model.


Train on multi-domains, then fine-tune on specific domain Would it make sense to first train a model on images from multiple domains, and then do "fine-tuning" on one specific domain to improve its performance on it?

For instance, one could train an object detector based on cars camera recorded in NYC, Paris and Beijing, then continue training on Paris only.
For a model that would be deployed on Paris only, should we favor diversity or specificity? And does this training method has a name?

Generally speaking, training with a broad distribution of data and then fine tuning on a specific domain generally leads to better performance and faster training time.
There are techniques such as meta-learning, domain invariance using techniques (ex. Gradient Reversal) or minimizing distance between the mean domain feature space using Maximum Mean Discrepency metrics. You could also familiarize yourself with empirical risk minimization and develop an intuition grounded on experiments. You could also check out 'A theory of learning from different domains' by Ben-David et al.


Role derivative of sigmoid function in neural networks I try to understand role of derivative of sigmoid function in neural networks.

First I plot sigmoid function, and derivative of all points from definition using python. What is the role of this derivative exactly?

import numpy as np
import matplotlib.pyplot as plt

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def derivative(x, step):
    return (sigmoid(x+step) - sigmoid(x)) / step

x = np.linspace(-10, 10, 1000)

y1 = sigmoid(x)
y2 = derivative(x, 0.0000000000001)

plt.plot(x, y1, label='sigmoid')
plt.plot(x, y2, label='derivative')
plt.legend(loc='upper left')
plt.show()

The use of derivatives in neural networks is for the training process called backpropagation. This technique uses gradient descent in order to find an optimal set of model parameters in order to minimize a loss function. In your example you must use the derivative of a sigmoid because that is the activation that your individual neurons are using.

The loss function

The essence of machine learning is to optimize a cost function such that we can either minimize or maximize some target function. This is typically called the loss or cost funtion. We typically want to minimize this function. The cost function, $C$, associates some penalty based on the resulting errors when passing data through your model as a function of the model parameters.

Let's look at the example where we try to label whether an image contains a cat or a dog. If we have a perfect model, we can give the model a picture and it will tell us if it is a cat or a dog. However, no model is perfect and it will make mistakes.

When we train our model to be able to infer meaning from input data we want to minimize the amount of mistakes it makes. So we use a training set, this data contains a lot of pictures of dogs and cats and we have the ground truth label associated with that image. Each time we run a training iteration of the model we calculate the cost (the amount of mistakes) of the model. We will want to minimize this cost.

Many cost functions exist each serving their own purpose. A common cost function that is used is the quadratic cost which is defined as

$C = \frac{1}{N} \sum_{i=0}^{N}(\hat{y} - y)^2$.

This is the square of the difference between the predicted label and the ground truth label for the $N$ images that we trained over. We will want to minimize this in some way.

Minimizing a loss function

Indeed most of machine learning is simply a family of frameworks which are capable of determining a distribution by minimizing some cost function. The question we can ask is "how can we minimize a function"?

Let's minimize the following function

$y = x^2-4x+6$.

If we plot this we can see that there is a minimum at $x = 2$. To do this analytically we can take the derivative of this function as

$\frac{dy}{dx} = 2x - 4 = 0$

$x = 2$.

However, often times finding a global minimum analytically is not feasible. So instead we use some optimization techniques. Here as well many different ways exist such as : Newton-Raphson, grid search, etc. Among these is gradient descent. This is the technique used by neural networks.

Gradient Descent

Let's use a famously used analogy to understand this. Imagine a 2D minimization problem. This is equivalent of being on a mountainous hike in the wilderness. You want to get back down to the village which you know is at the lowest point. Even if you do not know the cardinal directions of the village. All you need to do is continuously take the steepest way down, and you will eventually get to the village. So we will descend down the surface based on the steepness of the slope.

Let's take our function

$y = x^2-4x+6$

we will determine the $x$ for which $y$ is minimized. Gradient descent algorithm first says we will pick a random value for $x$. Let us initialize at $x=8$. Then the algorithm will do the following iteratively until we reach convergence.

$x^{new} = x^{old} - \nu \frac{dy}{dx}$

where $\nu$ is the learning rate, we can set this to whatever value we will like. However there is a smart way to choose this. Too big and we will never reach our minimum value, and too small we will waste soooo much time before we get there. It is analogous to the size of the steps you want to take down the steep slope. Small steps and you will die on the mountain, you'll never get down. Too large of a step and you risk over shooting the village and ending up the other side of the mountain. The derivative is the means by which we travel down this slope towards our minimum.

$\frac{dy}{dx} = 2x - 4$

$\nu = 0.1$

Iteration 1:

$x^{new} = 8 - 0.1(2 * 8 - 4) = 6.8 $
$x^{new} = 6.8 - 0.1(2 * 6.8 - 4) = 5.84 $
$x^{new} = 5.84 - 0.1(2 * 5.84 - 4) = 5.07 $
$x^{new} = 5.07 - 0.1(2 * 5.07 - 4) = 4.45 $
$x^{new} = 4.45 - 0.1(2 * 4.45 - 4) = 3.96 $
$x^{new} = 3.96 - 0.1(2 * 3.96 - 4) = 3.57 $
$x^{new} = 3.57 - 0.1(2 * 3.57 - 4) = 3.25 $
$x^{new} = 3.25 - 0.1(2 * 3.25 - 4) = 3.00 $
$x^{new} = 3.00 - 0.1(2 * 3.00 - 4) = 2.80 $
$x^{new} = 2.80 - 0.1(2 * 2.80 - 4) = 2.64 $
$x^{new} = 2.64 - 0.1(2 * 2.64 - 4) = 2.51 $
$x^{new} = 2.51 - 0.1(2 * 2.51 - 4) = 2.41 $
$x^{new} = 2.41 - 0.1(2 * 2.41 - 4) = 2.32 $
$x^{new} = 2.32 - 0.1(2 * 2.32 - 4) = 2.26 $
$x^{new} = 2.26 - 0.1(2 * 2.26 - 4) = 2.21 $
$x^{new} = 2.21 - 0.1(2 * 2.21 - 4) = 2.16 $
$x^{new} = 2.16 - 0.1(2 * 2.16 - 4) = 2.13 $
$x^{new} = 2.13 - 0.1(2 * 2.13 - 4) = 2.10 $
$x^{new} = 2.10 - 0.1(2 * 2.10 - 4) = 2.08 $
$x^{new} = 2.08 - 0.1(2 * 2.08 - 4) = 2.06 $
$x^{new} = 2.06 - 0.1(2 * 2.06 - 4) = 2.05 $
$x^{new} = 2.05 - 0.1(2 * 2.05 - 4) = 2.04 $
$x^{new} = 2.04 - 0.1(2 * 2.04 - 4) = 2.03 $
$x^{new} = 2.03 - 0.1(2 * 2.03 - 4) = 2.02 $
$x^{new} = 2.02 - 0.1(2 * 2.02 - 4) = 2.02 $
$x^{new} = 2.02 - 0.1(2 * 2.02 - 4) = 2.01 $
$x^{new} = 2.01 - 0.1(2 * 2.01 - 4) = 2.01 $
$x^{new} = 2.01 - 0.1(2 * 2.01 - 4) = 2.01 $
$x^{new} = 2.01 - 0.1(2 * 2.01 - 4) = 2.00 $
$x^{new} = 2.00 - 0.1(2 * 2.00 - 4) = 2.00 $
$x^{new} = 2.00 - 0.1(2 * 2.00 - 4) = 2.00 $
$x^{new} = 2.00 - 0.1(2 * 2.00 - 4) = 2.00 $
$x^{new} = 2.00 - 0.1(2 * 2.00 - 4) = 2.00 $

And we see that the algorithm converges at $x = 2$! We have found the minimum.

Applied to neural networks

The first neural networks only had a single neuron which took in some inputs $x$ and then provide an output $\hat{y}$. A common function used is the sigmoid function

$\sigma(z) = \frac{1}{1+exp(z)}$

$\hat{y}(w^Tx) = \frac{1}{1+exp(w^Tx + b)}$

where $w$ is the associated weight for each input $x$ and we have a bias $b$. We then want to minimize our cost function

$C = \frac{1}{2N} \sum_{i=0}^{N}(\hat{y} - y)^2$.

How to train the neural network?

We will use gradient descent to train the weights based on the output of the sigmoid function and we will use some cost function $C$ and train on batches of data of size $N$.

$C = \frac{1}{2N} \sum_i^N (\hat{y} - y)^2$

$\hat{y}$ is the predicted class obtained from the sigmoid function and $y$ is the ground truth label. We will use gradient descent to minimize the cost function with respect to the weights $w$. To make life easier we will split the derivative as follows

$\frac{\partial C}{\partial w} = \frac{\partial C}{\partial \hat{y}} \frac{\partial \hat{y}}{\partial w}$.

$\frac{\partial C}{\partial \hat{y}} = \hat{y} - y$

and we have that $\hat{y} = \sigma(w^Tx)$ and the derivative of the sigmoid function is $\frac{\partial \sigma(z)}{\partial z} = \sigma(z)(1-\sigma(z))$ thus we have,

$\frac{\partial \hat{y}}{\partial w} = \frac{1}{1+exp(w^Tx + b)} (1 - \frac{1}{1+exp(w^Tx + b)})$.

So we can then update the weights through gradient descent as

$w^{new} = w^{old} - \eta \frac{\partial C}{\partial w}$

where $\eta$ is the learning rate.


Training neural nets: is it important that the data is randomly sorted? Let's say I have a neural network that classifies data into A, B or C.
I've heard that it's bad to train the network with data from one class at a time, e.g. A,A,A,...,B,B,B,...,C,C,C. Now, is there a problem with other patterns in the data? For example A,B,C,A,B,C,A,B,C,... and so forth.

Yes it is important that the data is randomly ordered. You want to remove patterns of how you descent along the cost function because such patterns can increase the likelihood of overfitting. It's easiest to explain on A,A,A, ... B,B,B ... pattern. Here the network is given an opportunity to adapt to peculiarities of the first training example before moving further. And the goal of training is not to adapt to peculiarities but to find features that generalize well. So it seems that a better idea is to expose the network to a different example in each iteration.

Most research works that I've seen simply randomize the order of examples in each epoch. But it seems sensible to think that a useful pattern might in fact exist. Perhaps one that focuses on maximizing the variety of features that the network is exposed to with each subsequent iteration.


Neural Network not learning when more than 1 training data is given I am very new to neural networks and data science in general and wanted to try getting my hand in making a simple neural network in python.

I tried to make a neural network from scratch hoping to start off with one that can learn to draw a regression line.

Here is my code, its pretty messy.

import numpy as np
import matplotlib.pyplot as plt
import random

class Layer: #format is input multiplied weights
    def __init__(self,n_inputs,n_neurons):
        self.n_inputs = n_inputs
        self.n_neurons = n_neurons
        self.weights = np.random.randn(n_inputs,n_neurons)
        self.biases = np.zeros((1,n_neurons))

    def forward(self, inputs):
        self.prevX = np.array(inputs)
        self.output = np.dot(inputs,self.weights)+self.biases
        self.ReLU = np.maximum(0,self.output)

    def Correction(self,target,Lrate,ReLUTrue = True):
        if ReLUTrue:
            output = self.ReLU.copy()
        else:
            output = self.output.copy()
        self.biases += 2*(target-output)*Lrate
        dWeights = np.dot(self.prevX.T,2*(target-output))
        dX = np.zeros((1,self.n_inputs))
        for j in range(self.n_neurons):
            dX += 2*(target[0][j]-output[0][j])*self.weights.T[j]
        self.targetprevX = self.prevX + dX*Lrate/self.n_neurons
        self.weights += dWeights*Lrate

class Network:
    def __init__(self):
        self.layers=[]

    def appendLayer(self,layerObject):
        self.layers.append(layerObject)

    def forward(self,inputs):
        for i in range(0,len(self.layers)-1):
            self.layers[i].forward(inputs)
            inputs = self.layers[i].ReLU.copy()
        self.layers[-1].forward(inputs)

    def backProp(self,target,Lrate):
        for i in range(-1,-len(self.layers)-1,-1):
            self.layers[i].Correction(target,Lrate)
            target = self.layers[i].targetprevX.copy()

    def UpdateLayers(self):
        for layer in self.layers:
            layer.Update()

    def Output(self):
        return self.layers[-1].output

neunetwork = Network()
neunetwork.appendLayer(Layer(1,8))
neunetwork.appendLayer(Layer(8,8))
neunetwork.appendLayer(Layer(8,1))

for iter in range(0,100):
    X = random.uniform(-5,5)
    Y = X**2
    neunetwork.forward([[X]])
    print("error:",neunetwork.Output()-Y)
    neunetwork.backProp([[Y]],0.001)

plotXs=[]
plotYs=[]
for points in range(0,100):
    X=random.uniform(-5,5)
    neunetwork.forward([[X]])
    Y=neunetwork.Output()[0][0]
    plotXs.append(X)
    plotYs.append(Y)
plt.plot(plotXs,plotYs,'ok')
plt.show()

The programme converges to the correct value when a single constant value is parsed but never converges when multiple different values are parsed. My understanding of neural networks is very limited and am just doing this as a little project for my own learning so pardon me for any incorrect use of terms.

Thank you for your time.

The input has the following shape:

X = random.uniform(-5,5)
neunetwork.forward([[X]])

print(np.array([[X]]).shape) # == (1,1)

You could use numpy to generate X and y:

X = random.uniform(-5,5,size=(10,1)
Y = X**2

It would help to get a copy of the error message to make sure this is the problem.

The forward/backprop looks right in terms of accumulating gradients. I'm not sure about the following piece of code though:

dX = np.zeros((1,self.n_inputs))
for j in range(self.n_neurons):
        dX += 2*(target[0][j]-output[0][j])*self.weights.T[j]
self.targetprevX = self.prevX + dX*Lrate/self.n_neurons

I think you can utilize a dot product here.


Isolation Forest: simple example Can some one please explain Isolation Forests more clearly? Everywhere I search, I find the same explanation:

  Isolation Forest ‘isolates’ observations by randomly selecting a
  feature and then randomly selecting a split value between the maximum
  and minimum values of the selected feature.

Let's take an example to solve this:

x1 = [2, 1, 4, 6, 4, 2, 1, 2, 3, 4, 19]

How would I say that 19 is an outlier?

The idea is that the faster you can isolate a sample the higher the chance it is an outlier/anomaly.

Here is one possible scenario for your example:

x1 = [2, 1, 4, 6, 4, 2, 1, 2, 3, 4, 19]

Draw a random value between min=1 and max=19; e.g. 10
Split according to selected value:

x1_left = [2,1,4,6,4,2,1,2,3,4]
x1_right = [19]

Repeat on subsets

We already isolated 19 after one step whereas the other samples "to the left" are not yet isolated and haven't changed much so far. Any threshold sampled greater than 6 would isolate 19 in a single split which has a probability close to 70%.

To make it more obvious, say x1=[1,2,3,4,5,10000000000]. Sampling a value between 1 and 10000000000 will almost surely give a result > 5 and therefore immediately isolate 10000000000.


Gradient boosting vs logistic regression, for boolean features I have a binary classification task where all of my features are boolean (0 or 1).  I have been considering two possible supervised learning algorithms:

Logistic regression
Gradient boosting with decision stumps (e.g., xgboost) and cross-entropy loss

If I understand how they work, it seems like these two might be equivalent.  Are they in fact equivalent?  Are there any reasons to choose one over the other?

In particular, here's why I'm thinking they are equivalent.  A single gradient boosting decision stump is very simple: it is equivalent to adding a constant $a_i$ if feature $i$ is 1, or adding the constant $b_i$ if feature $i$ is 0.  This can be equivalently expressed as $(a_i-b_i)x_i + b_i$, where  $x_i$ is the value of feature $i$.  Each stump branches on a single feature, so contributes a term of the form $(a_i-b_i)x_i + b_i$ to the total sum.  Thus the total sum of the gradient boosted stumps can be expressed in the form

$$S = \sum_{i=1}^n (a_i-b_i) x_i + b_i,$$

or equivalently, in the form

$$S = c_0 + \sum_{i=1}^n c_i x_i.$$

That's exactly the form of a final logit for a logistic regression model.  That would suggest to me that fitting a gradient boosting model using the cross-entropy loss (which is equivalent to the logistic loss for binary classification) should be equivalent to fitting a logistic regression model, at least in the case where the number of stumps in gradient boosting is sufficient large.

You are right that the models are equivalent in terms of the functions they can express, so with infinite training data and a function where the input variables don't interact with each other in any way they will both probably asymptotically approach the underlying joint probability distribution.  This would definitely not be true if your features were not all binary.

Gradient boosted stumps adds extra machinery that sounds like it is irrelevant to your task.  Logistic regression will efficiently  compute a maximum likelihood estimate assuming that all the inputs are independent.  I would go with logistic regression.