Is the role of the validation set in a deep learning network is only for Early Stopping? In the "deep learning crash course" given by Leo Isikdogan in lecture 4 https://www.youtube.com/watch?v=ms-Ooh9mjiE&list=PLWKotBjTDoLj3rXBL-nEIPRN9V3a9Cx07&index=4
Overfitting, Underfitting, and Model Capacity
, He suggests that the data should be split in train, validation and test sets. The Train set is used to train the model, the validation set to optimize the hyperparameters and the test set to give an unbiased estimation of the generalization error.
When I look at how people implement the design, they usually use gridseachCV to evaluate the deep learning neural network to configure some of the hyperparameters such as the number of neurons, learning rate, optimizer, etc. After that, they used the validation set to perform the early stopping.

I think that it is possible to obtain a hyperparameter configuration from gridsearchCV that gave the best performance because it is already overfitted. Is it correct?

Is there another option to tune the hyperparameters with a validation set. Why they do not try different configurations in the validation set?

The validation set's purpose is to track your level of overfitting while you are experimenting with your network. When we try something we want to know how well it works on data it hasn't seen before.

By looking on the score on the training set we can't tell what is good from what is overfitting. If you instead use your test set to evaluate the results of all your experiments, then your test set stops being a reliable final evaluation since you have included it in your decisions. The test set is only for final evaluations.

To fill this need we set aside a part of our data as a validation set. A dataset that we do not directly train on, but instead the use to check results of training different layer/unit configurations, tuning hyperparameters, feature engineering and all the other things we possibly want to try. Early stopping is just one of the possible uses.

Regarding your comments about hyperparameters:
You don't use the validation set directly when you are tuning hyperparameters (with grid search or any other method). You still use the training set, but then you use the validation set to test what gave the best result.


Why the estimated Lasso coefficients of almost all variables are equal to zero? I would greatly appreciate if you could guide me. In fact, I used "Bayesian Optimization " to tune hyper-parameters  of Lasso but the estimated Lasso coefficients of almost all variables are equal to zero.

 X = np.array(pd.read_csv('my_X_table1-1c.csv',header=None).values)
 y = np.array(pd.read_csv('my_y_table1-1c.csv',header=None).values.ravel())

 ln=X.shape

names = ["x%s" % i for i in range(1,ln[1]+1)]

# split data into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

def Lassocv(alpha):
    return cross_val_score(Lasso(alpha=float(alpha), random_state=42),
                           X_train, y_train, scoring='neg_mean_absolute_error', cv=5).mean()

if __name__ == "__main__":

    LassoBO = BayesianOptimization(Lassocv, {'alpha': (0, 8)})
    LassoBO.maximize(init_points=2, n_iter = 30)
    print('Final Results')
    print('Lasso: %f' % LassoBO.res['max']['max_val'])
    alpha=LassoBO.res['max']['max_params']['alpha']

lasso = Lasso(alpha=alpha)
lasso.fit(X_train, y_train)
print('train Score:', lasso.score(X_train, y_train))
print('test Score:', lasso.score(X_test, y_test))
print('train MSE:', mean_squared_error(y_train, lasso.predict(X_train)))
print('test MSE:', mean_squared_error(y_test, lasso.predict(X_test)))
print('train MAE:', mean_absolute_error(y_train, lasso.predict(X_train)))
print('test MAE:', mean_absolute_error(y_test, lasso.predict(X_test)))

print("Features sorted by their score:")

# A helper method for pretty-printing linear models
def pretty_print_linear(coefs, names=None, sort=False):
    if names == None:
        names = ["X%s" % x for x in range(len(coefs))]
    lst = zip(coefs, names)
    if sort:
        lst = sorted(lst, key=lambda x: -np.abs(x[0]))
    return " + ".join("%s * %s" % (round(coef, 3), name)
                      for coef, name in lst)

print("Lasso model:", pretty_print_linear(lasso.coef_,names=names,sort=True))

output:

Final Results
Lasso: -0.786422
train Score: 0.476819501615
test Score: 0.459836314561
train MSE: 1.10511096023
test MSE: 1.02388727356
train MAE: 0.681164633717
test MAE: 0.663119930613
Features sorted by their score:
Lasso model: 0.001 * x25 + 0.0 * x54 + 0.0 * x48 + -0.0 * x17 + 0.0 * x31 + 0.0 * x12 + -0.0 * x15 + -0.0 * x34 + 0.0 * x1 + 0.0 * x2 + -0.0 * x3 + -0.0 * x4 + 0.0 * x5 + 0.0 * x6 + 0.0 * x7 + 0.0 * x8 + 0.0 * x9 + 0.0 * x10 + 0.0 * x11 + 0.0 * x13 + -0.0 * x14 + 0.0 * x16 + 0.0 * x18 + -0.0 * x19 + 0.0 * x20 + -0.0 * x21 + 0.0 * x22 + 0.0 * x23 + -0.0 * x24 + -0.0 * x26 + 0.0 * x27 + -0.0 * x28 + -0.0 * x29 + 0.0 * x30 + 0.0 * x32 + 0.0 * x33 + 0.0 * x35 + 0.0 * x36 + -0.0 * x37 + 0.0 * x38 + 0.0 * x39 + -0.0 * x40 + -0.0 * x41 + -0.0 * x42 + 0.0 * x43 + 0.0 * x44 + 0.0 * x45 + 0.0 * x46 + -0.0 * x47 + -0.0 * x49 + -0.0 * x50 + -0.0 * x51 + 0.0 * x52 + 0.0 * x53 + 0.0 * x55 + 0.0 * x56 + 0.0 * x57 + 0.0 * x58 + 0.0 * x59 + 0.0 * x60

This was due to the fact that I should normalize my data before passing it into the algorithm. For example,

scaler = StandardScaler()
scaler.fit(X_train)
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)

Or, the "normalize" parameter of Lasso should be set as True.

Lasso(alpha=float(alpha), normalize=True, random_state=42)


Softmax: Different output scikit-learn and TensorFlow I'm trying to learn a simple linear softmax model on some data. The LogisticRegression in scikit-learn seems to work fine, and now I am trying to port the code to TensorFlow, but I'm not getting the same performance, but quite a bit worse. I understand that the results will not be exactly equal (scikit learn has regularization params etc), but it's too far off.

total = pd.read_feather('testfile.feather')

labels = total['labels']
features = total[['f1', 'f2']]

print(labels.shape)
print(features.shape)

classifier = linear_model.LogisticRegression(C=1e5, solver='newton-cg', multi_class='multinomial')
classifier.fit(features, labels)
pred_labels = classifier.predict(features)

print("SCI-KITLEARN RESULTS: ")
print('\tAccuracy:', classifier.score(features, labels))
print('\tPrecision:', precision_score(labels, pred_labels, average='macro'))
print('\tRecall:', recall_score(labels, pred_labels, average='macro'))
print('\tF1:', f1_score(labels, pred_labels, average='macro'))

# now try softmax regression with tensorflow
print("\n\nTENSORFLOW RESULTS: ")

## By default, the OneHotEncoder class will return a more efficient sparse encoding.
## This may not be suitable for some applications, such as use with the Keras deep learning library.
## In this case, we disabled the sparse return type by setting the sparse=False argument.
enc = OneHotEncoder(sparse=False)
enc.fit(labels.values.reshape(len(labels), 1)) # Reshape is required as
Encoder expect 2D data as input
labels_one_hot = enc.transform(labels.values.reshape(len(labels), 1))

# tf Graph Input
x = tf.placeholder(tf.float32, [None, 2]) # 2 input features
y = tf.placeholder(tf.float32, [None, 5]) # 5 output classes

W = tf.Variable(tf.zeros([2, 5]))
b = tf.Variable(tf.zeros([5]))

# Construct model
pred = tf.nn.softmax(tf.matmul(x, W) + b) # Softmax

clas = tf.argmax(pred, axis=1)

# Minimize error using cross entropy
cost = tf.reduce_mean(-tf.reduce_sum(y*tf.log(pred), reduction_indices=1))
# Gradient Descent
optimizer = tf.train.GradientDescentOptimizer(0.01).minimize(cost)

# Initialize the variables (i.e. assign their default value)
init = tf.global_variables_initializer()

# Start training
with tf.Session() as sess:

   # Run the initializer
   sess.run(init)

   # Training cycle
   for epoch in range(1000):
      # Run optimization op (backprop) and cost op (to get loss value)
    _, c = sess.run([optimizer, cost], feed_dict={x: features, y: labels_one_hot})

# Test model
correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))
class_out = clas.eval({x: features})

# Calculate accuracy
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
print("\tAccuracy:", accuracy.eval({x: features, y: labels_one_hot}))
print('\tPrecision:', precision_score(labels, class_out, average='macro'))
print('\tRecall:', recall_score(labels, class_out, average='macro'))
print('\tF1:', f1_score(labels, class_out, average='macro'))

The output of this code is

(1681,)
(1681, 2)
SCI-KITLEARN RESULTS:
    Accuracy: 0.822129684711
    Precision: 0.837883361162
    Recall: 0.784522522208
    F1: 0.806251963817

TENSORFLOW RESULTS:
    Accuracy: 0.694825
    Precision: 0.735883666192
    Recall: 0.649145125846
    F1: 0.678045562185

I inspected the result of the one-hot-encoding, and the data, but I have no idea why the result in TensorFlow is much worse.

The problem turned out to be silly, I just needed more epochs, a smaller learning rate (and for efficiency I turned to AdamOptimizer, results are now equal.

(1681,)
(1681, 2)
SCI-KITLEARN RESULTS:
    Accuracy: 0.822129684711
    Precision: 0.837883361162
    Recall: 0.784522522208
    F1: 0.806251963817

TENSORFLOW RESULTS:
    Accuracy: 0.82213
    Precision: 0.837883361162
    Recall: 0.784522522208
    F1: 0.806251963817


Efficient algorithm to find the lowest value cluster in a series of values Let's say I have a list of numeric values that tend to be grouped into some number of clusters of values that are close to one another.
I'm aware of things like k-means to group these into groups of similar values, but I'm curious as to whether there's a faster and perhaps simpler algorithm to simply find all the values in the first or lowest valued cluster.
Example: take this sequence:
9,8,22,45,46,41,20,19,7,10,11,20,21,24

There are three clusters: one near about 10, one near 20, and one near 42. The algorithm in question might return:
8,9,7,10,11

These values are part of the lowest valued cluster of grouped values. Whether the results are sorted or not doesn't matter.
I can think of some easy improvised ways but I'm wondering if there's a robust general solution.

Sort your data. That is the fast trick that you cannot do on multivariate data. Then solid for r by three largest gaps or using KDE.

Clustering is more interesting for multivariate data.


LSTM Sequential Model question re: ValueError: non-broadcastable output operand with shape doesn't match broadcast shape This is probably a very simplistic question but I have not been able to find resources that directly address this. I know I must be understanding this incorrectly; I'm not quite sure how.
I've noticed that if the number of units in the last Dense output layer of my LSTM sequential model does not equal the number of features (columns), I get an error.
If you wanted to output 1 feature in the output (Dense) layer, and you had several input features, how would you:

Do that without errors
identify which feature is being outputted, or does Keras provide outputs for each feature and you have to identify which one you want?

I want to train the model with multiple features, but I'm only interested in one feature's prediction.
Example:  I have data with 'open', 'low', 'high' and 'close' and 'volume columns (5 features). If I set the number of units in the last Dense layer to anything other than 5, I get a broadcast error telling me I have inconsistent shapes in the model.  If I put 5 units in the last Dense output layer, I get no errors.
Example:
def create_model(self, epochs, batch_size):

    model = Sequential()

    # Adding the first LSTM layer and some Dropout regularisation
    model.add(LSTM(units=128, return_sequences=True,
                   batch_size=batch_size, input_shape=(TIME_STEP, self.X_train.shape[2])))

    # Adding a second LSTM layer and some Dropout regularisation
    model.add(LSTM(units=128, return_sequences=True))
    model.add(Dropout(DROPOUT))

    # Adding a third LSTM layer and some Dropout regularisation
    model.add(LSTM(units=128, return_sequences=True))
    model.add(Dropout(DROPOUT))

    # Adding a fourth LSTM layer and some Dropout regularisation
    model.add(LSTM(units=128, return_sequences=False))
    model.add(Dropout(DROPOUT))

    # Adding the output layer
    model.add(Dense(units=5))
    model.summary()

    # compile model
    adam = optimizers.Adam(lr=LR)
    model.compile(optimizer=adam, loss='mae')
    model.fit(self.X_train, self.y_train, epochs=EPOCHS, batch_size=BATCH_SIZE)

If I enter 'units=1' in the Dense layer, I get the following error:

ValueError: non-broadcastable output operand with shape (11784,1) doesn't match the broadcast shape (11784,5)

The final dense layer's units should be equal to the number of features in your y_train.
Suppose your y_train has shape (11784,5) then dense layer's units should be 5 or if y_train has shape (11784,1), then units should be 1. Model expects final dense layer's units equal to the number of output features.
You have to identify which features you need in input and output.
'open', 'low', 'high' and 'close' and 'volume' - these are your features.
What do you want to predict? Is this a classification or regression problem? What is your problem statement?
Based on these, identify your input and output features.
Put input features in x_train and output features in y_train.
Now, in the final dense layer, you should use units equal to the number of features in y_train.


In natural language processing, why each feature requires an extra dimension? I am reading Machine Learning by Example. I am trying to understand natural language processing. The book used Scikit-learn's fetch_20newsgroups data as an example.

The book mentioned that the text data in the 20 newsgroups dataset that we downloaded from fetch_20newsgroups data is highly dimensional. I do not understand this statement.
It is my understanding that dimension is used to describe axies that an array has.
For example,

import numpy as np
np.random.seed(0)
x1 = np.random.randint(10, size=6)
print("x1",x1) # 1 dimensions
np.random.seed(0)
x2 = np.random.randint(10, size=(3,4))
print("x2",x2) # 2 dimensions
np.random.seed(0)
x3 = np.random.randint(10, size=(3,4,5))
print("x3",x3) #3 dimensions

How does no. of axies relates to feature in NLP? Why one feature equals to one dimension? Please explain. Thanks.

Below is the code from the book that used to download the data for your reference.

from sklearn.datasets import fetch_20newsgroups
groups = fetch_20newsgroups()

You can think of each feature running along its own axis on a graph. Just because we store all feature e.g. in a single DataFrame – one feature per column – it doesn't mean the data's structure is just 2d (rows and columns). This is not the case only in NLP, but in most contexts involving statistics and modelling.

We can see this with your example data. There are text blocks, which you should match to a category (as far as I can tell).

The initial dataset contains other meta-data, such as a desctiption of the dataset, the names of the target categories and also the location of each sample's file. We don't really care about these for the pure modelling part. So there are only text blocks, called data, and the target categories, called target. Your input is then 1d - the text blocks.

I will show how to put that into a dataframe, being very verbose about dimensions and features:

from sklearn.datasets import fetch_20newsgroups
groups = fetch_20newsgroups()

import pandas as pd                    # needed to use a dataframe

# Get the desired parts from "groups"
desired = ['data', 'target']           # we don't care about the 'filenames' ona so on

# make a new dictionary with only desired key-value pairs
only_data = {k: v for k, v in groups.items() if k in desired}

Now we put this into a dataframe

df = pd.DataFrame.from_dict(only_data)

# Check the shape of the dataframe
df.shape
(11314, 2)

So there are 11314 samples of 1 feature, to 1 target variable. This is therefore 1-dimensional input data (we don't count the target variable).

When we have e.g. 50 features, explaining some target variable, it may be referred to as a 50-dimensional input space. People then may use dimensionality reduction techniques, such as Principal Components Analysis, which will attempt to squeeze the 50 dimensions into a lesser number (you can choose how many to use).

In your data, you will likely pre-process the text samples to create more features. These will just be new columns in the dataframe, whose shape could become e.g. (11314, 40) if you add another 38 features, by doing things like counting words or constructing some word-embeddings.


Neural Networks for time series I have an understanding problem. I am a beginner in machine learning and have also a little experience in modelling NNs but not for time series.

But I cannot imagine how to use Neiural Networks for time series.

So if I want to train a Multilayer NN what is the input?

I read several papers about such ANN to predict time series. But they did not explain how actually.

Can I imagine it as follows: Is every input node a specific time stamp?

If I want to predict the value for time t+1, and I use 15 input neurons, so I use the values at each time stamp from t-15 upt to t?

How do I train such a NN?

I am a little confused.

The type of neural networks you are looking for to predicting timeseries are called recurrent networks; the previous neuron states in the network affect the new neuron states. Examples of such recurrent networks are LSTM, GRU and NARX.

If you have the data from t0 to t10, and you want to predict t11, then you need to input the values from t0- t10 one by one into the network. You have to train the network so that the output of tn is tn+1. So after you have inputted t10, the output will be the values predicted for t11.

Real life example: I want a network to keep on decreasing it's output until it has reached 1, and then make it output 0 to start over again:

in: 0.0, out: 0.2
in: 0.2, out: 0.4
in: 0.4, out: 0.6
in: 0.6, out: 0.8
in: 0.8, out: 1.0
in: 1.0, out: 0.0

And that is what your training data should look like

If it is still to complicated, I recommend you to play around with this front-end neural network library for the browser: Neataptic. It is very easy to fiddle around with. Example


Similarity Threshold Standards When using similarity measures (eg. Resnik Information Content, Cosine Similarity, etc.) for any type of data, are there any standard similarity thresholds that are used, or does it all depend on the situation? A similarity threshold would be the value X in [0,1] such that all pairs with similarity score greater than X are "connected" while ones with similarity score below X are not.
Also, are low similarity thresholds (~0.15) acceptable when higher thresholds simply do not produce enough "connected" pairs and having a low similarity threshold still works well in practice?

I don't think there's any standard, but there might be some exceptions in very specific cases where the distribution of the scores is known precisely.
There's no standard because in general the optimal value of the threshold strongly depends on the task and the data. That's why thresholds are usually determined empirically based on the desired outcome. In other words, a threshold can be seen as an hyper-parameter: its optimal value can be found by maximizing the performance of the target task on a training set (or validation set).


How can I train a many-to-one RNN with an array of 2D matrices? I have eye tracking data for every word of a novel. Features for every word is given separately. I want to take groups of 100 words to make a sample and then use each of these samples as a single training instance (instead of a single word). My samples are 2D matrices of shape (no. of words, features). How can I train my many-to-one RNN with these samples?

you can use CNN just before inputting your raw matrix into RNN. An other option, which is just made for that purpose is ConvLSTM. Several research papers show that ConvLSTM can be used to process in sequential frames of video which is similar to your input type.


Weird distribution of neural network outputs I've faced an unusual behavior during training a neural network.

The problem is to predict if a sample of 1st class or 2nd class. (2-class classification). Classes are imbalanced (~ 5 / 95). I use weighted crossentropy.

What do I see is probabilities of two classes for each sample, and it's seems weird to me that no matter how long I wait (number of epochs), the probabilities are in range [0, 0.9) and there are no samples close to probability 1.0. I guess, the imbalance and weighted loss + ADAM with low starting learning rate could led to this, but I am not sure.

Has anyone faced the same?

It's not that weird. You probably have an activation function on output that only reach 0 or 1 at infinity. To reach that 0 or 1 you would need to have your last hidden layer to output +/- infinity. Which is difficult given the weight structure. The best it can do will be to output extremely large positive or negative values.
It may be a problem because during the calibration process (which is an optimisation problem) you will spend significant ressources and time in trying to reach +/- infinity. Notably this is one of the causes of the famous exploding gradient problem.
How to deal with that ? In Efficient Backpropagation (LeCun and others, 1998), the author propose to use an activation function that is able to reach desired outputs. It states that a good activation function needs to have the following properties :

$f(\pm1)=\pm1$

The second derivative is a maximum at $x=1$

The effective gain is close to 1

Your problem is that you don't have something similar to the first condition. To respect those conditions LeCun proposes to use the activation function: $1.7159*tanh(\frac{2}{3}x)$. you might not need to use it in all your layers. Now your problem would probably be to find a framework that allows for custom activation functions.


Predicting with multiple time series Say you had a set of users, tens of thousands.  You have time series of each of their behaviors using this app.  How might you use these time series to predict future behavior of new users?

The intuitive solution is to feature engineer behavior of users and use that to train a model, such as average weekly minutes in app and things like that.  My issue with this approach is that you lose a great deal of information.

What I'm wondering is if there's a better technique for aggregating a large set of time series data to build a predictive model.  Perhaps an LSTM would work but that seems like it wouldn't capture the nuances of the dataset and I don't believe they're typically used to aggregate predictions on a variety of individual samples.

I would recommend looking into FBProphet.
It's a good starting point for automating the creation of forecasts. It's very easy to use, and often produces better results than classical forecasting methods (ARIMA, Holt-Winters, etc.) right out of the box.
The default settings offer an additive or multiplicative model, comprised of trend and seasonality. This can be a combination of daily, weekly and yearly seasonality patterns, with the option to add custom patterns if required. Holiday effects can also be tracked using the built in holiday calendars, which again can be customised.
This is generally used for data with no external regressors, just a datetime and a measurement. However, external regressors can be configured, to influence the outcome of the forecast.


pairwise_distances with Cosine and weighting Is there a way to get a weight into the
pairwise_distances(X, metric='cosine')

Potentially using **kwrds?
from sklearn.metrics import pairwise_distances

In the scipy cosine distance it's possible to add in an array for weights, but that doesn't give a pairwise matrix.
a = np.array([9,8,7,5,2,9])
b = np.array([9,8,7,5,2,2])
w = np.array([1,1,1,1,1,1])

distance.cosine(a,b,w)

Where w is the weights.

Instead of using pairwise_distances you can use the pdist method to compute the distances. This will use the distance.cosine which supports weights for the values.
import numpy as np
from scipy.spatial.distance import pdist, squareform

X = np.array([[5, 4, 3], [4, 2, 1], [5, 6, 2]])
w = [1, 2, 3]

distances = pdist(X, metric='cosine', w=w)

# change the result to a square matrix
distances = squareform(distances)

Result:
array([[0.        , 0.05508882, 0.04898252],
       [0.05508882, 0.        , 0.07833123],
       [0.04898252, 0.07833123, 0.        ]])


Which other algorithms fit residuals like XGBoost? XGBoost and standard gradient boosting train learners to fit the residuals rather than the observations themselves. I understand that this aspect of the algorithm matches the boosting mechanism which allows it to iteratively fit errors made by previous learners.

Which other algorithms or also train single or multiple learners to fit residuals?
Does this method only make sense for learners built in a sequence? Or also for any ensemble methods?
Is there a deep significance to fitting residuals or is this a mathematical convenience that allows for iterative improvement from learners built in succession?

Training of XGBoost is based on a boosting model, which is a general ensemble method creating a strong model from a number of weak models. This process is performed by building a model from the training dataset, then, creating a second model that attempts to correct the errors from the first model. Models are added until the training set is predicted perfectly or a maximum number of models are added.
All machine learning models based on boosting almost follow the above procedure. For example, AdaBoost  is one of the boosting algorithms developed for binary classification. Methods like LightGBM and Catboost use this algorithm.
Unlike the boosting algorithm, in the bagging algorithm, the models are independent and each model is directly fitted to a subset of the original training data.


Train CNN's / Rotation and Mirroring i am thinking about using a CNN to classify certain images from industrial production, like scratches, stain, particles etc.

the problem is we don't have many images. We only get around 10 parts with certain defects to detect. I thought about using a database with different defects on it.

I use the example of scratches, there are around 100 images of scratches. Now my question:

Does it make sense to rotate the images in all 4 orientation, and to mirror them as well? So i would get 8 images (the original + 7 uniformly transformed images), i know the "value" of the rotated and mirrored images is not as high as from real images, but it should still help the CNN to abstract and find features.

What do you think about it?

Thanks in advance!

Dog_Fish

I am currently doing a project with similar industrial image processing task. I am detecting defects in a rectangular product image and from my experience, horizontal and vertical flipping of images has helped me a lot to augment my dataset and increase F1 score. Since I am not sure regarding your picture orientation I am unable to answer regarding other rotations. But if it has equal height and width then rotation of dataset makes sense. Try incorporating residual units into your CNN which has helped me massively since I had to use deeper networks to extract intricate features.


plotting a decision tree based on gridsearchcv i was trying to plot the decision tree which is formed with GridSearchCV, but its giving me an Attribute error.

AttributeError: 'GridSearchCV' object has no attribute 'n_features_'

However if i try to plot a normal decision tree without GridSearchCv, then it successfully prints.

code [decision tree without gridsearchcv]

# dtc_entropy : decison tree classifier based on entropy/information Gain
#plotting : decision tree on information/entropy  based

from sklearn.tree import export_graphviz
import graphviz

feature_names = x.columns

dot_data = export_graphviz(dtc_entropy, out_file=None, filled=True, rounded=True,
                                feature_names=feature_names,
                                class_names=['0','1','2'])
graph = graphviz.Source(dot_data)
graph                           ### --------------> WORKS

code [decision tree with gridsearchcv]

#plotting : decision tree with GRIDSEARCHCV (dtc_gscv)  on information/entropy  based
from sklearn.tree import export_graphviz
import graphviz

feature_names = x.columns

dot_data = export_graphviz(dtc_gscv, out_file=None, filled=True, rounded=True,
                                feature_names=feature_names,
                                class_names=['0','1','2'])
graph = graphviz.Source(dot_data)
graph                            ##### ------------> ERROR

Error

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-201-603524707f02> in <module>()
      6 dot_data = export_graphviz(dtc_gscv, out_file=None, filled=True, rounded=True,
      7                                 feature_names=feature_names,
----> 8                                 class_names=['0','1','2'])
      9 graph = graphviz.Source(dot_data)
     10 graph

1 frames
/usr/local/lib/python3.6/dist-packages/sklearn/tree/_export.py in export(self, decision_tree)
    393         # n_features_ in the decision_tree
    394         if self.feature_names is not None:
--> 395             if len(self.feature_names) != decision_tree.n_features_:
    396                 raise ValueError("Length of feature_names, %d "
    397                                  "does not match number of features, %d"

AttributeError: 'GridSearchCV' object has no attribute 'n_features_'

code for decision-tree based on GridSearchCV

dtc=DecisionTreeClassifier()

#use gridsearch to test all values for n_neighbors
dtc_gscv = gsc(dtc, parameter_grid, cv=5,scoring='accuracy',n_jobs=-1)

#fit model to data
dtc_gscv.fit(x_train,y_train)

One solution is taking the best parameters from gridsearchCV and then form a decision tree with those parameters and plot the tree.

However is there any way to print the decision-tree based on GridSearchCV.

While I don't have a module named graphviz I can still try to help. Reading the documentation for GridSearchCV, I can see that there's a attribute called best_estimator_ that provides the estimator that was chosen by the search. By applying .best_estimator_ to your sample code, it seems be working fine.

clf.fit(iris.data, iris.target)
dot_data = export_graphviz(clf, out_file=None, filled=True, rounded=True,
                                class_names=['0','1','2'])
-------------------
AttributeError: 'GridSearchCV' object has no attribute 'tree_'

With .best_estimator_

clf.fit(iris.data, iris.target)
dot_data = export_graphviz(clf.best_estimator_, out_file=None, filled=True, rounded=True,
                                class_names=['0','1','2'])

I get no error. I hope this helps


sklearn RandomForestRegressor oob_score_ looks wrong? I am a newbie to datascience. I installed the Jupyter notebook and was trying to create model for the kaggle titanic dataset. Below is the code I wrote -

from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import roc_auc_score
import pandas as pd
X = pd.read_csv("train.csv")
y = X["Survived"]
X["Age"].fillna(X.Age.mean(),inplace = True)
numeric_variables = list(X.dtypes[X.dtypes != "object"].index)
model = RandomForestRegressor(n_estimators=100, oob_score = True, random_state = 42)
model.fit(X[numeric_variables],y)
model.oob_score_

I am getting the oob_score_ output as 1.0 while the actual output as per tutorial should be 0.13420....
After that I did

y_oob = model.oob_prediction_
print "c,stat:", roc_auc_score(y,y_oob)

The output of this as well I am getting 1.0. Is there something wrong in my code or setup of Jupyter?

You are using Random Forest Regressor, but since you have a binary response where a passenger survives or not, the Random Forest Classifier should be used.  If this doesn't fix the issue, please send the link to the tutorial.


adding non-failure data to failure one I have a dataset containing features of different engines showing when they failed. I want to build supervise learning model to predict whether an engine with a certain mileage is going to fail or not. Unfortunately, I have data of engines that they never fail and I need to add them to my dataset (it is true these engines never fail but they will help model to have a better view on lifetime an engine).
Could any one help me how to add these  non-failure data to failure one.

Best,

As far as I understand, you want to combine two datasets to use both for training??

If yes, you need to append one dataframe to the other. You need to make sure, that all columns/features needed for training are present in both DFs. After that you can simply append the DFs.

Python example:

>>> df = pd.DataFrame([[1, 2], [3, 4]], columns=list('AB'))
>>> df
   A  B
0  1  2
1  3  4
>>> df2 = pd.DataFrame([[5, 6], [7, 8]], columns=list('AB'))
>>> df.append(df2)
   A  B
0  1  2
1  3  4
0  5  6
1  7  8

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.append.html


I know which forums specific user read - how to cluster the data? I have a dataset as follows. For each user, I have a separate row with the forum he reads.
There is up to 100 different forums.

I would like to cluster this data, so each user will be assigned to one of the groups (I do not know how much of groups there could be) based on the forums he read.
Do you know if there is a ready algorithm that I can use?
For now, I'm thinking if I can calculate the vector from the list of all the forums and use k-means on that.
Second thing would be to transform the data as follows:

I guess that if I would only use 1, if user read the forum, or 0 if not, that would not work with k-means. I can extract number of posts, reputation or upvotes instead of "1". Will it work with k-means?

There are different approaches from easy to complicated
Similarity-Based

Distance or similarity matrix is calculated for different topics (forums) and if two topics are more similar than a threshold, you suggest them to the users of the other one. For this, you need to gather all texts of different forums and vectorise them (from TF-IDF to Neural Embedding). Then define a similarity score which tells you how related topics are. Then suggest them to users.
Second way of doing the same is to calculate similarity of different topics based on the number of mutual readers. Then doing the same as above.

Community Detection (Clustering) in Bipartite Graphs
Familiar yourself with the concepts of Community Detection and Bipartite Graphs. Then apply this clustering on users based on the forums they read. Select top topics inside each cluster of users and suggest them to all members of that cluster.
Link Prediction in Bipartite Graphs
Familiarize yourself with concept of Link Prediction. Applying link prediction on your problem gives you most probable links between users and forums.


How to work with Log-transformation? I'm beginning my data science journey and I've faced a challenge that confuses me a bit.
I have a set with few features and a target variable whose raw distribution is highly skewed.

I've read that it's possible to use a log transformation to normalize the target variable (loss in $) and thus increase the accuracy.

When I train my model with "y_raw", using MAE I get an error of 306k. When I log-transform
y = y.transform(np.log) I get MAE accuracy of around 2 (log-transformed units I suppose?), which is e^2 = 7.39 (y_raw). This is a significant drop from 306k to only 7.39 ($) (or am I getting it wrong?), so I am a bit suspicious about it.

So here are my questions:
1) Did I get it correct that the error rate drop from 306k to only 7.39 is real and is valid?
2) How do I make a predictions from there? If I feed a sample to my model, receive a log-transformed output, lets say it returned a prediction of y_log = 10. Do I then simply use an inverse of it by placing e^10 = 22,026.5 and will it be my final prediction?

Taking the log doesn't result in a normally-distributed target; it would tend to if the target was log-normally distributed, and you have something normalish there, not quite. But, this distribution isn't actually what matters.

What taking the log does is change your model of how errors arise when fitting a regressor. You're now saying that the target values are $e^{P + \epsilon}$ where $P$ is your model's prediction and $\epsilon$ is Gaussian noise. Or: $e^P e^\epsilon$. That part directly interacts with the assumptions in your regressor.

So what you're finding is that on average the predictions are wrong by a factor of 7.39, not +/- $7.39.

What you really want to do is evaluate MAE on actual target values vs $e^P$ . You probably have a better model but not that much better.


Find Net Reclassification Improvement/Index metric using Python I am working on a binary classification problem with ~5k records and class proportion of 33:67.

I have 60 features/variables in my dataset and finally I have come to about 10 variables based on multiple feature selection algorithms and domain understanding.

Now I would like to experiment with this 10 features and get the best of these 10 which gives the high performance.

For example, Having 6 features might give an AUC of 84% whereas adding one new feature (7 features) might give an AUC of 85%.

But adding two features (8 features), might give an AUC of 83%.

I read in several posts and F Harrel blog posts that AUC is not always a good metric to compare model performance.

q1) So what are the other best metrics that you usually use?

q2) How to calculate Net Reclassification Improvement? Read it's better than AUC.

Can you people please help me?

Just stick to F1.

Yes NRI is cools but there are edgecases and its not a magic bullet. Read this paper

IF you really want to implement it you can embedd r programs in python.

Since there is already implementation in r what you could do is something like this. Just change it to use NRI.


Unformatted data entries I have been working recently on an independent project using a database for Cybersecurity Attack classification. I imported the database using Pandas (Python) and before starting the processing step, I have noticed that some of the entries contain the following symbols: "-", "0x000b", "0xc0a8", and many others that it is difficult to track them and see how many of these unformatted data are present, specially when the database is so big. Is there a way to take the whole dataframe, spot all the possible unformatted and error data and substitute them by NaN, to treat them later as missing values?

Thanks in advance!

Those symbols you are talking about are hex values. They could actually be useful to keep in the dataset, as the tcpdump tool outputs them if the right options are set.

Despite that, you can replace occurrences of them using pandas and a simple regular expression. A regular expression that matches hex values is: 0[xX][0-9a-fA-F]+ - see a live demo of that here.

I downloaded one of the CSV files and did it like this:

import pandas as pd

filename = "./UNSW-NB15_1.csv"
df = pd.read_csv(filename, nrows=5000)     # read first 5000 lines
df.to_html("with_hex.html")                # open in a browser to inspect (e.g. line 211)

Now use the DataFrame replace and set the regex flag, and insert whatever you want to replace the matches with (here "NaN").

clean = df.replace("0[xX][0-9a-fA-F]+", value="NaN", regex=True)

I used this to have a look at the cleaned versions of the data quickly in a browser (notice line 211):

clean.to_html("clean.html")

You can use the same trick to replace almost anything, so your fields with '-' can be replaced with:

# "inplace" means it overwrites the values, no new dataframe is returned
clean.replace("-", "NaN", regex=False, inplace=True)


How do local minima occur in the equation of loss function? In gradient descent, I know that local minima occur when the derivative of a function is zero, but when the loss function is used, the derivative is equal to zero only when the output and the predicted output are the same (according to the equation below).
So, when the predicted output equals the output, that means the global minima is reached! So, my question is: How can a local minima occur, if zero gradient occurs only for the "perfect" fit?
$$\theta_j := \theta_j - {\alpha \over m} \sum_{i=1}^M (\hat y^i-y^i)x_j^i$$

The equation you used for gradient descent isn't general; it's specific for linear regression.  In linear regression, there is indeed only a single global minimum and no local minima; but for more complex models, the loss function is more complex, and local minima are possible.


Sklean pipeline order Is there a correct order I should put data transformations into a pipeline using Sklearn?

Currently I have these items in my pipeline;

Feature selection, skew removal, scaling, outlier removal, oversampling and the estimator.

Is there a right or wrong way to do it? Or is there no free lunch?

The most common approach is a combination of these two strategies:

Domain expertise - Given knowledge about the dataset and the goal of the model, choose the order that best manipulates the data to accomplish the goal of the project.
Empirical evidence - Permutation the order and benchmark results. Pick the permutation that has the highest performance on the benchmarks.


Lightweight data provenance tool One of the problems I often encounter is that of poor data provenance.

When I do research I continuously make modifications to my code and rerun experiments. Each time I'm faced with a number of questions, such as: do I save the old results somewhere, just in case? Should I include the parameter settings in the output filenames or perhaps save them in a different file? How do I know which version of the script was used to produce the results?

I've recently stumbled upon Sumatra, a pretty lightweight Python package that can capture Code, Data, Environment (CDE) information that can be used to track data provenance. I like the fact that it can be used both from the command line and from within my Python scripts and requiring no GUI. The downside is that the project seems inactive and perhaps there's something better out there.

My question is: what is a good lightweight data provenance solution for my research? I'm coding small projects mostly in Python in the terminal on a remote server over SSH, so a command line solution would be perfect for me.

EDIT 2019: The Sumatra project seems inactive and other, more mature project have come along. DVC looks very promising and I've been in contact some of its authors, who have proven very helpful and supportive.

Yes, you should save result files before you make major mods to the code.  Disk space is cheap, so you're unlikely to run into issues unless your results sets are prolific.  I would suggest storing old results sets with folder names that include a time stamp of when they were generated.

As far as time shots of your code, using github (or some other code repository tool) is as easy as can be, will save version information, allows for collaboration, and is an all around great way to backup and version your code.

The combination of these two things, you'll effectively have an easy way of mapping a result set to a specific version of the code.


What kind of neural network is the best for classifying a series of vectors I am currently working on a project where I decided that it might be useful to use machine learning. I have a good understanding of pytorch and I know how to build my neural network within this framework.
However, currently I am struggling to find the perfect kind of neural network to solve my problem.
Let me describe the problem:
You have a game which you have developed and you want to develop an anti cheat for all sorts of cases. Now just for an example i will choose "fly-hacking".
What i am trying to do is record all of a players data over a period of time.
Let's say within 1 second i will take n "snapshots" of a current players state over e.g. 5 seconds.
This data includes the position vector in a 3-dimensional space (x,y,z), and some data of his surroundings which will be just split up in small areas (e.g. 3x3x3 matrix), now if i know there are solid objects in his surroundings they will be represented by 1 and if an area is empty it is 0.
Now my input data could be looking like this for a standing person:
(4.2, 10.2, 3.2, 0, 0, 0, 0, [....], 0, 1, 1, 1) which would just be representing a player who is standing on a plane with nothing else arround him.
And if i collect this data over 5 seconds with e.g. 3 snapshots per second i would end up with an array of 15 elements.
I can collect a fair amount of these vectors and labels them with "flying" and "not-flying" categories. I just need a fitting neural network model which would fit my problem best.
In the end I think it is just a classification problem between "flying" and "not flying" but I am struggling with the idea of which model would learn the fastest and easiest from this data.
Maybe you can help me out here and give some advice where to start :)

First, let me point that knowing how to use PyTorch is not enough to work with neural networks.
Second, your data points have about 450 elements from which 405 are binary (solid object or not), neural networks will require you to place then as real elements for training, dealing with floating points where you could make a faster processing using models that can deal with this kind of data. After training you can speed up predictions by changing the implementation of your layers to avoid the trivial multiplications with 1 and 0.
Third, neural networks are not a silver bullet, you should try simpler models first, just to check it you actually need the power of a neural network and it's non-linearity. Check some alternatives and guides for time-series analysis. I woul go with a Random Forest or a decision tree, you should be able to make them fairly simple and then do an analysis of what's relevant for solving your problem.
That said:
First Step:
Your data is composed of vectors that can be flattened to $R^{450}$ ($15 \times (3 \times 3 \times 3 + 3)$) which is a fairly small vector, few fully connected layers (two or three should be enough) may solve your problem, that should be your first try (in neural network).
Second Step
If that doesn't suffice, you can try more complex models:
Your data has the following format:
$P = (x,y,z,t)$, which is the position of a player in time.
$D_{geo}(P) \in R^{3 \times 3 \times 3}$, which is a tensor describing their surroundings.
Since, $P$ resembles a time-series, LSMTs are usually indicated for this kind of analysis. Since you plan on using fixed time-steps, a CNN could be used, keeping data format as a tensor ($T \in R^{3 \times 3 \times 3 \times t}$) and using few CNN layers to extract information from the geographical information and only then joining the positions with it (assuming you want a fast processing model).
The actual shape of your CNN layers could will have to be tunned with a grid search.


Best way for correlation analysis I am looking for an alternative way of analysing my data which will be used for ML. I am using Matlab to implement my code. Using my code, I obtain arrays which are used to describe signals. Each array represents a new and different signal.

I have used the corrcoeff method in Matlab and it delivered what I needed. But, I need another alternative way to verify the correlation between all the signals. Is there a better way to do this which is easy to interpret?

Dumb question: is it better to compare the images of the signal generated by Matlab or to compare the arrays?

Forgive me if this sounds badly formulated but I hope it is clear.

Thank you in advanced!

Based on the current details about your question, I'm not sure what you seek, but I will give it a try.

I assumed you already tried Pearson's coefficient. There are other metrics, which are more or less difficult to implement if not already existing in matlab.

Spearman's rank
Coefficient of determination
Hilbert-Schmidt independence criterion

All those will provide some insight about dependencies within your dataset.


Factorization Machine - prevent over fitting I was recently asked this question in an interview and wondered what the answer would be - "How do Factorization Machines get around the overfitting problem when using second-order interactions?"

Here are some excerpts from the original paper that I think are key to understanding the question:

  Instead of using an own model parameter for each interaction,
  the FM models the interaction by factorizing it. We will see later on,
  that this is the key point which allows high quality parameter
  estimates of higher-order interactions under sparsity.

  Factorization machines can estimate interactions even in these
  settings (sparse data) well because they break the independence of the interaction
  parameters by factorizing them. In general this means that the data
  for one interaction helps also to estimate the parameters for related
  interactions.

In other words, instead of fitting an independent parameter for every second order interaction, it factorizes the parameters which reduces the parameter space and the model complexity, and thus making the model less prone to over fitting.

Hope this helps understand some aspects of the question.


Relative merits of different open source natural language generators Does anyone know what (from your experience) is the best open source natural language generators (NLG) out there? What are the relative merits of each?

I'm looking to do sophisticated text summarization and would like to use theme extraction/semantic modeling in conjunction with NLG tools to create accurate, context-aware, and natural-sounding text summaries.

In terms of open source NLG components, I'm most familiar with Mumble and FUF/SURGE. They've got both similarities and differences, so it's hard to say which is better...

Mumble:

written in Lisp
EPL license
based on tree-adjoining grammar
focuses on linguistic message planning

FUF/SURGE:

written in Lisp
GPL license
based on functional unification grammar
focuses on syntactic realization

Since it sounds like you're interested in abstractive summarization (which is much harder than traditional extractive summarization), I'd recommend the following academic papers:

Text Generation for Abstractive Summarization
Framework for Abstractive Summarization using Text-to-Text Generation
Towards a Framework for Abstractive Summarization of Multimodal Documents -- full disclosure: I'm the author of this one

Also, consider checking out this textbook to get started: Building Natural Language Generation Systems


Data wrangling for a big set of docx files advice! I'm looking for some advice on a data wrangling problem I'm trying to solve. I've spent a week solid taking different approaches and nothing seems to be quite perfect. Just FYI, this is my first big (for me anyway) data science project, so I'm really in need of some wisdom on the best way to approach it.

Essentially I have a set (200+) of docx files that are semi-structured. By semi-structured I mean the information I want is organized into tables (it's a form, with different tables which contain different info to fill out), but unfortunately these tables are not consistently formatted. Sometimes after people enter data into them they accidentally hit backspace to stick the tables together. Or sometimes they accidentally break the tables apart, for example.

My first attempt used python-docx to extract the data using document.tables[0] etc. I could then pull this into a big python dictionary for each document. It was quite neat but hit a snag - the table formatting problem above.

I then used python-docx again and tried to use the headings of each table as a marker (picked them out using regex) for when a sub data set should begin or end, and iterated across all text in the document. This sort of works, and is more flexible, but picks up a lot of text from outside of the table which makes it difficult to manage and clean.

Anyway, I'm interested in how an experienced data scientist would approach the problem.

The end goal is to extract the data from one of these documents into an SQL database.

If you're interested in the problem, let me know and I can send you the template documents I'm working with and some samples. If it's helpful, I can also post the code I've written so far (haven't done so because it's long).

The big problem with docx files is that they have a ton of content related to formatting that most people find irreverent when scraping docx files.  Hence, one approach is to convert the docx files to something more friendly like json that excludes all the irrelevant bits, and then use existing frameworks like JSON schema for pattern matching and jmespath for data extraction.  There is a python package called simplify-docx for converting Docx files to JSON and a toy website where you can try it out.  (Disclaimer: I'm the author of both)


How to keep the test data from leaking into the training process of a machine learning algorithm? I read in many different sources that I need to split my data into a training set and a test set. Then I have to make sure that the algorithm is trained only on the training data, and do my best to keep the test data from leaking into the training process.

To avoid learning insignificant details of the data (which will increase the algorithm's generalization ability) I can further split my training data into the proper training set and a validation set multiple times, and choose the parameters of the algorithm that give the best average performance for all such splittings.

Finally I evaluate my algorithm on the test set, and get some numbers: MSE, RMSE, etc. But do these numbers really show, how good is my algorithm, and aren't they affected by the test set?

Of course, I didn't use my test data during training but the parameters of the algorithm that I got are valid only for this particular splitting into the training and test sets. If I split my data differently, I will get different MSE, RMSE, and different optimal parameters.

I can then do many different splittings into training/test, and calculate the error for each split. I that case how is it different from the cross-validation that I did during the first split? Do I need to further divide my training data into the proper data set and a validation set?

If I do many different splittings into the training an test sets, doesn't it mean that my test data leak into the training process because some data were in the test set for one splitting but in the training set for another splitting?

Just to clarify (and I think you've got this right, but I'm just being careful), it is best practice to:

1: Split your data into train and test

2: Split train into train and eval

3: Grid search over hyperparameters, for each combination, train on train, evaluate on eval. Select the hyperparameters which allow you to get the best score on the eval set

4: Using the best model (as ascertained in step 3), calculate the test loss, using the test data set create in step 1. Use your answer in step 4 to get the best estimate of how well your model will perform in production.

To be very clear, step 4 is not about hyperparameter selection, the hyperparameters are fixed by step 3. Step 4 is all about getting a better estimate of production performance, as what you get from step 3 is likely to be overly optimistic, as the particular hyperparameter combination used to overly optimised (overfit) to that particular train-eval split.

You can repeat steps 1-4 many times, but what you should not do, is try to use this repetition in any way, to find the best hyperparameters, you are using it to answer the question "how well is my model likely to perform on unseen data". For example, you might run it ten times and get a spread of accuracies from 65%-67%. You can then tell your boss, that if you deployed this model in production, it would likely have an accuracy of 65%-67% (or you're a bit more quantitative and you say it's the mean $\pm$ the standard error)

This leaves the question, "how do I select my best hyperparameters?". After all, you've run steps 1-4 many times and step 3 has given you a different best hyperparameter combination every time.

This is where you need to run step 3 one more time, but slightly differently. This time, you don't need a train-test split. You simply take all of your data, and do a train-eval split. You then run one further hyperparameter grid search, and that's the model you use in prod (or you could combine the train and eval sets, and train one more time, using those hyperparamters, depending on whether you're using a model that needs an eval set for early stopping like a neural net/xgboost or one that does not like a Random Forest). However, at this stage, you ignore the accuracy/peformance metrics that the training procedure is telling you. You already estimated your production accuracy/performance when you ran steps 1-4 multiple times, and that was the point of doing this.

I hope this answers your question, both on how to do the splitting, as well as illustrates why test data at no point leaks into the training data


What is the simplest ReLU network that can learn an upside-down v pattern? Input: $x \in \mathbb{R}^+$

Expected output:

$f(x) = \begin{cases}x &\text{for } x \leq 5\\10 -
 x &\text{for } x > 5\end{cases}$

So basically you have a neural network with one input neuron and one output neuron. Let's say the output has the identity as an activation.

What is the simplest network with ReLU activations that fits this function?

In order to find the network, I tried a couple of different small networks:

One hidden layer with two neurons

This would be the function

$$out = \max(a_1 x + b_1, 0) + \max(a_2x+ b_2, 0)$$

$$a_1 = 1, b_1 = 0, a_2=1, b_2 = -1$$

$$a_1 = 1, b_1 = -5, a_2=-1, b_2 = 5$$

2 hidden layers, 3 neurons in total

This is pretty close:

I guess the pattern is right, but the values are just not quite right. Also, I'm not totally sure if only one hidden layer with 3 neurons (or more) might also work.

I always include biases.

The solution

    / o (ReLU) \
IN o            o (linear)
    \ o (ReLU) /

It is basically the 1 hidden layer with 2 neurons, but also using the scaling (-1?) and the bias (+5?) of the output layer.

$$(-1) \cdot (\max(x - 5, 0) + \max(-x + 5, 0))+5$$


Oversampling only balances the training set, what about the testing set? In a case of imbalanced data classification, I know that we only oversample the training set (to prevent data leakage from training to testing subsets), but what if there are no positive data points in my testing set? The testing set is still highly skewed and has only 1% of my positive class. I am using XGBoost, Random Forest, Logistic Regression, and KNN for the classification task.

Also, I have tried SMOTE, SMOTE-NC, and Class_weight to oversample my training set. To increase the chance of having more data from the minority class, I changed the 10-fold to 5-fold cross-validation (when developing the models), no improvement!

PS: I have >100K data points in my dataset.

Use a stratified split (as pointed out by louic in the comments). It will distribute your classes evenly across all folds. It can be done using sklearn's StratifiedKFold.


Train a deep learning model in chunks/sequentially to avoid memory error How do I train/fit a model in chunks so as to escape the dreaded memory error?

def TFIDF(X_train, X_test, MAX_NB_WORDS=75000):
    vectorizer_x = TfidfVectorizer(max_features=MAX_NB_WORDS)
    X_train = vectorizer_x.fit_transform(X_train).toarray()
    X_test = vectorizer_x.transform(X_test).toarray()
    print("tf-idf with", str(np.array(X_train).shape[1]), "features")
    return (X_train, X_test)

# In[3]:

def Build_Model_DNN_Text(shape, nClasses, dropout=0.5):
    """
    buildModel_DNN_Tex(shape, nClasses,dropout)
    Build Deep neural networks Model for text classification
    Shape is input feature space
    nClasses is number of classes
    """
    model = Sequential()
    node = 512  # number of nodes
    nLayers = 4  # number of  hidden layer
    model.add(Dense(node, input_dim=shape, activation='relu'))
    model.add(Dropout(dropout))
    for i in range(0, nLayers):
        model.add(Dense(node, input_dim=node, activation='relu'))
        model.add(Dropout(dropout))
    model.add(Dense(nClasses, activation='softmax'))
    model.compile(loss='sparse_categorical_crossentropy',
                  optimizer='adam',
                  metrics=['accuracy'])
    return model

# In[18]:

df = pd.read_csv("ExtractedData.csv")

# In[19]:

df = df.dropna()
X_train, X_test, y_train, y_test = train_test_split(df['body'], df['user_id'],
                                                    test_size=0.3, random_state=42, shuffle=True)
X_train_tfidf, X_test_tfidf = TFIDF(X_train, X_test)
y_train_tfidf, y_test_tfidf = TFIDF(y_train,y_test)
output_nodes = len(list(set(df['user_id'])))
model_DNN = Build_Model_DNN_Text(X_train_tfidf.shape[1], output_nodes)
model_DNN.fit(X_train_tfidf, y_train_tfidf,
              validation_data=(X_test_tfidf, y_test_tfidf),
              epochs=10,
              batch_size=256,
              verbose=2)

predicted = model_DNN.predict(X_test_tfidf)
print(metrics.classification_report(y_test, predicted))

When I run the above, I keep getting the memory error: Unable to allocate 37.9 GiB for an array with shape (67912, 75000) and data type float64. I know I can send the data in chunks pd.read_csv( , chunksize= 5000), but what I don't know is how do I implement it here. My data has 98000 rows and 4 columns. Thank you.

Pandas is not memory efficient for data loading.

It appears you are using Keras / TensorFlow package. It would be better to use that package's data loading for csv which will better manage memory.


Which NLP library has the most mature Chinese language models? I am trying to do some NLP on Simplified Chinese texts (needing to extract sentence structure and to do named entity recognition).  I've used spaCy previously for English texts, but I see the notes on the Chinese models suggest they are a work in progress, and the NER extraction accuracy has been poor for the examples I've tried.
Which NLP library has the most mature pre-built Chinese language models? Ideally Python based.

There are no common libraries that support high quality named entity recognition for Chinese.
Other options include Information-Extraction-Chinese on GitHub or adapting a paper with code.


Grid search or gradient descent? Assume we have a neural network and one if its activation functions is a function of parameter a. We want to find the weights and parameter a that leads to the minimum loss on the validation set which one is better?:

  1) Treat a as a hyperparameter. Do grid search for a: consider range for a and evaluate the loss for a few points in the range. Pick the best a in the range along with the weights leading to the minimum loss.

  2) Treat a as a parameter. Since loss of the network is a function of parameter a, use gradient descent to update weights as well as parameter a at each iteration.

To me, the second option is better since it can lead to the optimal point but the grid cannot lead to the optimal point, it can lead to the points around the optimal point depending on your selection. But, is the second method right?

Is there a better method?

Lastly, in grid search, how do we pick the range?

The main distinction to be made here is between a parameter and a hyperparameter; once we have this clarified, the rest is easy: grid search is not used for tuning the parameters (only the hyperparameters), which are tuned with with gradient descend.

Now, roughly speaking, a parameter is something that changes during training; in a neural network, the only parameters are the weights and the biases, and they are tuned with gradient descend.

A hyperparameter can be thought as something "structural", e.g. the number of layers, the number of nodes for each layer (notice that these two determine indirectly also the number of parameters, i.e. how many weights and biases there are in our model), i.e. things that do not change during training. Hyperparameters are not confined to the model itself, they are also applicable to the learning algorithm used (e.g. optimization algorithm, learning rate etc). A specific set of hyperparameters defines a family of models, which differ among themselves in the exact values of their parameters; in contrast, a specific parameter set (e.g. weights & biases in a NN) define a unique model.

Having clarified the above, it should be easy to see that a in your example above is a hyperparameter, and not a parameter; as such, it would normally be optimized using grid search.

  To me, the second option is better since it can lead to the optimal point

Not so fast; gradient descend is not applicable everywhere: there are mathematical prerequisites for a function in order to be eligible for optimization using gradient descend, i.e. to be continuous & defferentiable. This is indeed the case for the loss as a function of the weights & biases, but not clear if it is also the case for the loss as a function of your parameter a. And if it isn't, we simply cannot use gradient descend for tuning a.

On the contrary, techniques like grid search do not have such prerequisites, thus they can be used for a much wider range of cases.

  in grid search, how do we pick the range?

As we pick almost everything else in ML & DL: empirically, and with trial & error (there is actually not much theory behind it).

UPDATE (after comment):

  Now assume that the prerequisites are met for a so that we can use gradient descent. Which one is better now? Grid search or gradient descent?

Sorry, I didn't realize that this was a "Superman vs. Batman" question... Well, gradient descend is a disciplined mathematical operation, which is guaranteed to find the global minimum for convex functions (although NN loss functions are not convex), while grid search is actually just an ad hoc, quick-and-dirty procedure which does not guarantee anything (as you have already correctly suspected), so...

Beware though, especially if this is part of any homework or exam: the question, despite its phrasing, is not actually asking you which one is "better" in an abstract & hypothetical context, but which one is better in a very specific context, i.e a neural network model; and if something is not even applicable, it cannot be actually better, right?

In other words, if someone is actually testing your ML & NN knowledge here, they are certainly more interested in the exposition above, rather that which one is "better" in general; and choice (2) would be a huge mistake (after all, assume that the NN loss is a continuous & differentiable function of a is not part of the question)...


Mathematic explanation needed for a univariate classification method based on solving a quadratic equation I read a piece of codes on classifying image hue values into three classes with derived thresholds. The thresholds are calculated by simply using a quadratic formula. The related documentations for the codes are as follows:

/*
*       (v_B - v_A)x^2 + 2(m_B*v_A - m_A*v_B)x + (m_A^2*v_B - m_B^2*v_A) = 0
*
*       v_A : variance of group A
*       m_A :   mean of group A
*       v_B : variance of group B
*       m_B :   mean of group B
*
*       Quadratic formula
*       ax^2 + bx + c = 0
*       a = v_B - v_A
*       b = 2(m_B*v_A - m_A*v_B)
*       c = (m_A^2*v_B - m_B^2*v_A)
*
*       x1 = (-b+sqrt(b^2-4ac))/2a
*       x2 = (-b-sqrt(b^2-4ac))/2a
*
*/

Here above is to find a threshold to delimit the class A and B, based on solving a quadratic equation involving the mean and variance of the two groups, and the threshold is one of the solutions (i.e., x1 and x2) depending on their signs.

I haven't seen such a quadratic equation based classification method before. Anyone know what formal name for the method? It, on the first sight, seems to me a simplified case of quadratic discriminant analysis, but not sure about that. I need a little bit mathematic background for the method. Educate me if you know that. Thanks!

Actually, it is very straightforward to derive.

Hint 1: where would you most likely find mean and variance used? Gaussian!

Hint 2: how would you do classification using Gaussians? Simple, you fit a Gaussian to each group and compare the PDFs of new data $x$. Even simpler, you fit a Gaussian to each group and calculate the Mahalanobis distance of $x$ to each group.

A threshold is called a threshold because at this value you cannot tell which class it is in, aka, the Mahalanobis distances to two groups equal:

$$\frac{(x-\mu_A)^2}{\sigma_A^2}=\frac{(x-\mu_B)^2}{\sigma_B^2}$$

Expanding this, you will get the quadratic formula.


How to Obtain Output of Intermediate Model in Keras I'm creating a neural architecture using the functional API as follows:

x2 = layer1(x1, name='layer1')
x3 = layer2(x2, name='layer2')
m1 = Model(x1, x3)
x5 = layer3(x4, name='layer3')
x6 = TimeDistributed(m1)(x5)
m2 = Model(x5, x4)

I've trained m2 (with m1 getting implicitly trained). I can access the output of the layers of m2 (say layer3) by creating an intermediate model:

layer_name = 'layer3'
intermediate_layer_model = Model(input=model.input, output=model.get_layer(layer_name).output)
intermediate_output = intermediate_layer_model.predict(input)

Since m1 is a nested model, I'm not able to access the layers of m1 through m2. How do I obtain the output x3 (layer2's output) when you use the trained m2 to predict on a new data point.

The key is to first do .get_layer on the Model object, then do another .get_layer on that specifying the specific layer, THEN do .output:

layer_output = model.get_layer('Your-Model-Object').get_layer('the-layer-contained-in-your-Model-object').output


Classifying boat images I trying to get some experience by exploring this Kaggle dataset.
It consists of 1500 pictures of boats classified in 9 categories. The data is as follows :
#x_train consists of 1159 images, with 80% of images from each category

x_train.shape = (1159,200,200,3)

y_train contains the number-label for each boat

y_train.shape = (1159,)

I have tried many variations of models like the following one but without any success.
model = Sequential()

model.add( Conv2D(32, (3,3),  input_shape = x_train.shape[1:] , activation='relu') )
model.add(MaxPooling2D(pool_size=(3,3)))

model.add(Flatten())
model.add(Dense(4, activation='relu'))

model.add(Dense(2, activation='softmax'))

model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['accuracy'])

h = model.fit(x_train, y_train, epochs=50,
              batch_size = 64,
              validation_data = (x_val, y_val) )

Could you give me any advice on how to get a model with decent test_accuracy?

By looking at your code snippet, I realize you are training your CNN from scratch.

Use Transfer Learning Instead. Training a new model (choice of model architecture i.e. how deep your model should be, hyperparameters etc.) is very difficult if not impossible with only 1500 images. You can achieve great results quickly by using  an already-trained model (aka Transfer Learning). If you are not quite familiar with the subject, read this article Transfer learning from pre-trained models, or this one First steps with Transfer Learning for custom image classification with Keras. There are codes included that helps to get started faster. One of the recent advances in Transfer Learning is efficientnet, you may want to jump using that one! But I would guess boats would be easy even with earlier models.


Would keeping all punctuation make any sense in word2vec? I am trying to learn how word2vec works to get to more complicated stuff like LSTMs.
Because I will use the same training data (so with the same vocabulary) and I want to predict punctuation too, I decided to keep it.

Punctuation is obviously limited to .!,?, anything else is discarded (The sentence itself to avoid loss of meaning). Also, every symbol is converted into a string-like representation to avoid any kind of problems with encoding.

Now, does it make sense to keep punctuation? Should I discard it completely (if I do discard it, how can I generate punctuation for seq2seq model later?)?

You are correct to keep punctuation if you want to be able to predict it.

Tokenization of your input should actually work for any character, be it a letter or punctation. In fact there have already been exmaples of people modelling mathematics using Word2Vec and then generating very realistic maths, via $\LaTeX$ !  Look at the subsection called Algebraic Geometry in Karpathy's now famous blog post.

There is a good note on the matter here, whcih a specific example given in seq2seq learning (basically translation within the realm of NLP). Be sure to read the comments on the accepted answer there.

to answer your final question, I don't think it would be possible to use your generated model to place to required punctuation back into your model, as something like an LSTM would not have a representation for, say a comma, as it had never seen one.


Pandas - Create a Integer Primary Key based in string columns - python I need to create a primary key based in string columns in my dataframe
Month       Name                   ID
01/01/2020  FileName1 - Example    1
01/02/2020  FileName2 - Example    2
01/03/2020  FileName3 - Example    3

I'm using the hash, but its generating the largest values, I would like that ID was the integer numbers.
This is my code
all_data['unique_id'] = all_data._ID.map(hash)

where  _ID is :
all_data['_ID'] = all_data['Month'].astype(str) + all_data['Name']

This group by return 0 for all rows
all_data['sequence']=all_data.groupby('_ID').cumcount()

You could simply drop the index to create and use it as a primary key column.
all_data.reset_index(inplace=True)
all_data.head()


How to build in symmetry of inputs into a Deep Neural Network? I have a Deep Neural Network that takes $n$ inputs $X = [X_1, \ldots, X_n]^T$ and gives $n$ ouputs $Y = [Y_1, \ldots, Y_n]^T$. Normally, I can just do a standard deep neural network with a few fully connected hidden layers. However, I want to build into the network the fact that there should be permutational symmetry. That is to say, if $\pi(\cdot)$ is a particular permutation, then $\pi(X)$ should output $\pi(Y)$. So each input should be treated symmetrically in some sense.

A simple example would be I get $n$ IID sensor readings, and I want to build an auto-encoder for these sensor readings.

Is there some way of building this symmetry into the neural network, perhaps through some parameter sharing or a special architecture?

This might not be what you're looking for, but you can use data augmentation to do this: from each sample, you can generate multiple samples by cloning it, generating a random permutation and permuting the input and the expected output (for an autoencoder, this should achieve the same goal). This doesn't guarantee complete permutation invariance but if you do it enough times for each sample it should get you pretty close.


Learning ordinal regression in R? I'm working on a project and need resources to get me up to speed.

The dataset is around 35000 observations on 30 or so variables.  About half the variables are categorical with some having many different possible values, i.e. if you split the categorical variables into dummy variables you would have a lot more than 30 variables.  But still probably on the order of a couple of hundred max.  (n>p).

The response we want to predict is ordinal with 5 levels (1,2,3,4,5).  Predictors are a mix of continuous and categorical, about half of each.  These are my thoughts/plans so far:
1.  Treat the response as continuous and run vanilla linear regression.
2.  Run nominal and ordinal logistic and probit regression
3.  Use MARS and/or another flavor of non-linear regression

I'm familiar with linear regression.  MARS is well enough described by Hastie and Tibshirani.  But I'm at a loss when it comes to ordinal logit/probit, especially with so many variables and a big data set.

The r package glmnetcr seems to be my best bet so far, but the documentation hardly suffices to get me where I need to be.

Where can I go to learn more?

I suggest this tutorial on ordered logit: http://www.ats.ucla.edu/stat/r/dae/ologit.htm

It showcases the use of polr in the MASS package, and also explains the assumptions and how to interpret the results.


What if Training and testing dataset comes from the same source? I am working on a classification problem in which I have to distinguish between healthy and damaged plates. when I use the combination of k-means clustering and SVM algorithm together with 10-fold cross validation, I can achieve the accuracy up to 95%. All the training and validation datasets come from the experiment.
For the testing, can I get the datasets after repeating same experiments with same specimen or I have to use different sets of specimens?

You have to use a different set of specimens. Or you can keep one or two specimens from the original set aside and use them as test. Use data augmentation and transfer learning in that case.


This has been a discussion for quite a while. For a more theoretical point of view you can find a good summary here.

From a practical point of view I'd look at it as follows. With increasing $k$ two things happen:

Your $k-1$ training folds increase in size
Your validation folds decreases in size

From the first point you can draw the conclusion that your $k$ models become more similar since your training data becomes more similar since you splitt off less data for the validation sets in the $k$-th fold. Which might lead to less between model-variance.

Moreover, since you are training on more data, the relative model complexity (i.e. compared to your data) decreases. In the bias-variance-trade-off graph (taken from [1])

this means we are moving towards the left, i.e. we trade less model variance for more model bias (please note that model variance here has a more general and conceptual meaning than the calculated variance or standard deviation between folds). The reason is that we are fitting a model with constant complexity to more data as $k$ increases, i.e. it becomes harder to learn the training data.

However, we are not only increasing the size of our training set with increasing $k$. We are also decreasing the validation set in size (see point two from above). Therefore, there might be higher between-fold variance with regards to our validation sets. This might be more relevant if the overall dataset carries higher variance or more outliers.

[1] "The Elements of Statistical Learning" by Hastie et al


Autoencoder: using cosine distance as loss function I'm trying to train an autoencoder (in PyTorch) to reconstruct gene profiles. At the moment I'm using the Mean Squared Error (MSE) loss for training: the model is not overfitting and both the training and validation loss are decreasing. The problem is that the cosine similarity on the validation set between original and reconstructed vectors has a mean of 0.4. I was thinking of using the cosine similarity as loss function instead of MSE.

At the following link (slide 18), the author proposes the following loss:

$$
l(x_1, x_2, y) =
\begin{cases}
max(0, cos(x_1, x_2) - m) & \text{if $y$ == -1} \\
1 - cos(x_1, x_2) & \text{if $y$ == 1}.
\end{cases}
$$

I'm not entirely sure whether this is the right approach, but I'm having some difficulties even understanding the formula. What is $y$ (the cosine similarity between $x_1$ and $x_2$?) and why it's an input to the loss?

Hey so the Keras implementation of Cosine Similarity is called as Cosine Proximity. It just has one small change, that being cosine proximity = -1*(Cosine Similarity) of the two vectors. This is done to keep in line with loss functions being minimized in Gradient Descent.

To elaborate,
Higher the angle between x_pred and x_true. lower is the cosine value. This value approaches 0 as x_pred and x_true become orthogonal.
And if the angle between x_true and x_pred is small the value approaches 1.

So it logically follows that the cosine proximity loss value approaches -1 as the model converges.

As for the usage, I would recommend using this loss only when the magnitude of the generated vectors isn't as important as the angle between them. And that really depends on what you are trying to solve.


Models to deal with huge possible outputs My question is what kind of machine learning models could be used in the case we need to predict something from a large pool of possibilities. For example in a kaggle competition( Instacart challenge ) , they ask to predict what products will a user reorder , based on prior orders.

In this case the spectrum of possible outputs is huge ( there are +40000 possible products )

I have studied some models like CNN and usually the output for this models, is a logits tensor holding the probabilities, the size of the tensor usually corresponds to the possibilities in the classification problem. That are generally small ( maximum something like a hundred in cifar-100

But in this case, seems to me that this is not a good approach, because of the number of possible items.

So this CNN models are appropriated in this kind of tasks? what other approaches could be used?

Thanks.

There is something called "sampled softmax" (e.g. tensorflow's implementation), which simply partitions the output space and at each training step only takes into account one of the partitions (see section 3 from this article to learn the math). Sampled softmax is only meant to speed up training for very large output spaces; at inference time you use normal softmax.

Also, if your output is of hierarchical nature, you can use hierarchical softmax. See this blog post for a review of different methods to handle very large categorical space.


How to use InceptionV3 without retraining I know I can retrain Inception to label images, but can I just provide an image to an untrained Inception and get a label back? Is there an API for this? If there is, then why does Google open source this if they're trying to make money off of google cloud vision?

can I just provide an image to an untrained Inception and get a label back?

You will get a random output from the network because it hasn't been trained and has been initialized with random values for the weights.

  Is there an API for this?

Cloud Vision is an API with already trained models. If you want to train your own model you can use tensorflow, theano, caffe... there are several frameworks for it. None of them is easy.

  why does Google open source this if they're trying to make money off of google cloud vision?

Open sourcing tools and models gives to the community a lot of knowledge, in exchange the researchers community use the Google tools to improve the models, more people use the models, so it is easier for Google to hire Machine-Learning developers and, most important, because the real value of machine learning today is in the data and the infrastructure you have. The more data you have the better your models. The better the infrastructure the faster and cheaper you can train and deploy your models. And people will pay you to use your infrastructure.


Building a machine learning model based on a set of timestamped features to predict/classify a label/value? I'm trying to apply machine learning to pharmaceutical manufacturing to predict whether batches of drug products manufactured are good or not. for the sake of relatability, let's use coffee brewing as an analogous process. Let's imagine that I'm trying to predict the acidity of the coffee that I've brewed.

The dataset that I have contains features such as temperature of water, stirring speed and pressure that are constantly measured (say..on a per second basis) over a variable amount of time (the first cup may be brewed in 5 minutes, the second in 10 etc).

What kind of preprocessing should I perform on such a multidimensional dataset? One stumbling block is that for each observation, the duration is different, which may complicate dimension reduction? Once preprocessed, is there any specific model that would suit the task at hand? I'm looking at something like a regression but alternatively, classifiers seem to be fine as well if I split the acidity(pH) into "<5.5" or ">5.5"?

I hope to get some general directions and if you can paste a few links to texts or examples that'll be good! Also, I'm more familiar with python and scikit learn, so if you can point me in the right section in the documentation that'll be great too!

You can use panda date time function to create various features representing time; Total time, day etc.  Then use these features as your independent variable to train your model


PCA for dimensionality reduction with simultaneous clustering so, let's say I have a set of 3D points. Let's say these points lie more or less on a plane that is embedded in the 3d space, then I can use PCA to 'compress' these 3D points to 2D coordinates on that plane, such that they still aproximate the original data well.
let's say half of the 3d points don't lie close to that plane, but instead close to some other plane.
If I just do PCA and reduce to 2 dimensions, I won't get a good aproximation.
If the algorithm however would 'see' that some of the 3d points compress well onto one plane, and others compress well on another plane and label each point and do PCA separately for each set (and compress them to points with 2 coordinates plus one bit that says which set it belongs to) it will aproximate the original data much better.
What's the name for such a PCA algorithm that is also capable of splitting the input data into maximally N sets (probably with some penalty on the number of sets), such that for each set dimensionality reduction yields a much better fitting than if all data points would be reduced together?
// Edit:
adding an example. If one would only cluster by distance in the high-dim space one would arrive at the bad clustering where there are more clusters and each cluster would have a higher error when projected down.
the good example uses fewer clusters and they project better on their 2 dimensional sub-spaces (the green cluster being able to even compress to a 1D space)

Your task is achieved by Subspace Clustering


Classification problem in pytorch with loss function CrossEntropyLoss returns negative output in prediction I am trying to train and predict SVHN dataset (VGG architecture). I get very high validate/test accuracy by just getting the largest output class. However, the output weights are of large positive and negative numbers. Are they supposed to parsed as exp(output)/sum(exp(output)) to be converted to probability? Thank you!

The numbers to which you are referring are probably the logits of the NN. The logits have to be transformed using a activation function which could be :

Softmax ( which you shown ) for classification tasks.
Linear activation function for regression tasks.

Softmax activation function :

The mathematical notion is as follows :

$\Large S( y_i ) = \frac{ e^{y_i}}{ \sum e^{y_i} }$

It converts the logits to class probabilities which sum upto 1. It is widely used in multiclass classification problems.

CNNs for classification need to produce class probabilities which are produced by a softmax activation function at the output layer.


Optimal input setup for character-level text classification RNN I want to classify 500-character long text samples as to whether they look like natural language using a character-level RNN. I'm unsure as to the best way to feed the input to the RNN. Here are two approaches I've thought of:

Provide the whole 500 characters (one per time step) to the RNN, and predict a binary class, $\{0,1\}$.
Provide shorter overlapping segments (e.g. 10 characters) and predict the next (e.g. 11th) character. Convert this to classification by taking the test input and calculate the joint probability of the observed characters based on predicted next-character distributions.

The first approach seems sub-optimal as I don't believe that the 1st character is going to have any effect on the prediction of the 500th character. The second approach gives me diminishingly small likelihoods when you calculate the joint probability.
I'm aiming for a more nuanced language model akin to n-gram frequency counting. I'm using simple RNNs for now but intend to swap to either LSTM or GRU.

For character-level NLP, one-dimensional convolutions are often used to shrink the number of hidden states to a reasonable number.
For instance, ELMo first tokenizes the input into words and then uses a CNN with max-pooling to obtain character-based word embeddings that are used as an input to an LSTM.
In machine translation, there is an approach that shrinks the character-level input into pseudo-word hidden states. This is a one-dimensional CNN over the character input (here, without segmenting the input into words), the CNN is followed by max-pooling with window step 5 (which is the average word length) and a bunch of highway-layers. These pseudo-word states are then used again as an input to an LSTM.


What is ChunkParserI in nltk.chunk ? What exactly it has been called for? from nltk.chunk import ChunkParserI
from nltk.chunk.util import conlltags2tree
from nltk.corpus import gazetteers

class LocationChunker(ChunkParserI):
    def __init__(self):
        self.locations = set(gazetteers.words())
        self.lookahead = 0
        for loc in self.locations:
            nwords = loc.count(' ')
        if nwords > self.lookahead:
            self.lookahead = nwords

What is ChunkParserI  in nltk.chunk ? What exactly it has been called for?
Also, please explain the code.
What is the difference between chunking and parsing?

Parsing is the process of decomposing a string into it's constituent symbols (if the string is a word or a sequence of characters) or syntactic components (if the string is a meaningful textual entity like a short story, a scientific abstract or a sentence). In an NLP context, when one talks about parsing, he/she usually refers to the latter interpretation.

Chunking (in an NLP context) is a specific form of parsing in that it extracts groups of words in so-called 'chunks'. These groups of words or chunks  are 'meaningful short phrases from the sentence (tagged with Part-of-Speech). Chunks are thus made up of words and the kinds of words are defined using the part-of-speech tags. One can even define a pattern or words that can't be a part of chunk and such words are known as 'chinks''1. The latter can be defined with chunking rules.

I assume the code you posted comes from "Natural Language Processing: Python and NLTK" by Hardeniya et al.2? From there i can find that the LocationChunker class 'starts by constructing a set of all locations in the gazetteers corpus. Then, it finds the maximum number of words in a single location string so it knows how many words it must look ahead when parsing a tagged sentence.' (cf. Chapter 5, p. 319)


Why do we need to discard one dummy variable? I have learned that, for creating a regression model, we have to take care of categorical variables by converting them into dummy variables. As an example, if, in our data set, there is a variable like location:

Location
----------
Californian
NY
Florida

We have to convert them like:

1  0  0
0  1  0
0  0  1

However, it was suggested that we have to discard one dummy variable, no matter how many dummy variables are there.

Why do we need to discard one dummy variable?

Simply put because one level of your categorical feature (here location) become the reference group during dummy encoding for regression and is redundant. I am quoting form here "A categorical variable of K categories, or levels, usually enters a regression as a sequence of K-1 dummy variables. This amounts to a linear hypothesis on the level means."

This is already discussed at this very nice stats.stackexchange answer.

I was told there is an advanced course by Yandex in Coursera that covers this subject in more details if you still have doubts, see here. Note you can always audit the course content for free. ;-)

Another nice post if you want a thorough explanation with lots of examples with statistical perspective and not being limited to only dummy coding, see this from UCLA (in R)

Note that if you using pandas.get_dummies, there is a parameter i.e. drop_first so that whether to get k-1 dummies out of k categorical levels by removing the first level. Please note default = False, meaning that the reference is not dropped and k dummies created out of k categorical levels!


Imbalance in observable data I am studying the performance, over 10 years, of high school students that enrolled in a school district. My Objective is to make inferences about factors leading to poor performance in school exams. I have data and stats on all those who were dropouts. Drop outs are a big group from among the poor performers. For example if there are 1000 students, 250 are poor performers. Of these 250, 200 are drop outs and there is no data on those. I plan to run a logistic regression model and draw inferences, but how do i remedy the imbalance that this missing data is creating?

So You have 750 : 50 ratio of positive and negative class. Try running LR on this data. Class imbalance would not be a problem. Even if so, you under sample the positive class and then re-calibrate the probabilities after model execution.
I would also suggest you to be active on cross-validation stack-exchange community. There are similar questions answered.


How to group identical values and count their frequency in Python? Newbie to analytics with Python so please be gentle :-) I couldn't find the answer to this question - apologies if it is already answered elsewhere in a different format.

I have a dataset of transaction data for a retail outlet. Variables along with explanation are:

section: the section of the store, a str;
prod_name: name of the product, a str;
receipt: the number of the invoice, an int;
cashier, the number of the cashier, an int;
cost: the cost of the item, a float;
date, in format MM/DD/YY, a str;
time, in format HH:MM:SS, a str;

Receipt has the same value for all the products purchased in a single transaction, thus it can be used to determine the average number of purchases made in a single transaction.

What is the best way to go about this? I essentially want to use groupby() to group the receipt variable by its own identical occurrences so that I can create a histogram.

Working with the data in a pandas DataFrame.

EDIT:

Here is some sample data with header (prod_name is actually a hex number):

 section,prod_name,receipt,cashier,cost,date,time
 electronics,b46f23e7,102856,5,70.50,05/20/15,9:08:20
 womenswear,74558d0d,102857,8,20.00,05/20/15,9:12:46
 womenswear,031f36b7,102857,8,30.00,05/20/15,9:12:47
 menswear,1d52cd9d,102858,3,65.00,05/20/15,9:08:20

From this sample set I would expect a histogram of receipt that shows two occurrences of receipt 102857 (since that person bought two items in one transaction) and one occurrence respectively of receipt 102856 and of receipt 102858. Note: my dataset is not huge, about 1 million rows.

From this sample set I would expect a histogram of receipt that shows
  two occurrences of receipt 102857 (since that person bought two items
  in one transaction) and one occurrence respectively of receipt 102856
  and of receipt 102858.

Then you want:

  df.groupby('receipt').receipt.count()

receipt
102856    1
102857    2
102858    1
Name: receipt, dtype: int64


How does BERT deal with catastrophic forgetting? In the ULMFit paper authors propose a strategy of gradual unfreezing in order to deal with catastrophic forgetting. That is, when the model starts be fine-tuned according to a downstream task, there is the danger of forgetting information on lower layers. Although Google's BERT is also a pre-trained language model, which makes use of fine-tuning for downstream tasks, authors don't mention this phenomenon. Why is that the case? Is BERT immune to it? Or does it deal with this in another way?

As far as I know, no neural network can be immune to catastrophic forgetting during fine-tuning (which is essentially controlled retraining).

The key is to not fine tune the pretrained model for longer epochs, or with higher learning rates. This ensures that the learned knowledge from the lower layers is preserved more or less intact, while also helping the model learn from the new data used for fine-tuning, as mentioned here : https://github.com/huggingface/transformers/issues/1019


unable to plot contour/surf graph of cost vs parameters for multiple linear regression.In python After running gradient descent I have three arrays theta1 theta2 and J all three of size num_iterationx1. I tried plotting contours using the contour function in Axes3D and 3d surface plot using surf but there is always a dimension error or no figure gets plotted.

Can anyone tell me how to plot this. In python

Try using a trisurf plot! It is very simple to get a nice surface plot.

Have a look at this Matplotlib 3d documentation that includes many examples.

Here is one of the examples, where I additionally print the size of the three arrays used to plot. We see that they are all single vectors of the same length, therefore your data should just work fine using the code below:

from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
import numpy as np

n_radii = 8
n_angles = 36

# Make radii and angles spaces (radius r=0 omitted to eliminate duplication).
radii = np.linspace(0.125, 1.0, n_radii)
angles = np.linspace(0, 2*np.pi, n_angles, endpoint=False)

# Repeat all angles for each radius.
angles = np.repeat(angles[..., np.newaxis], n_radii, axis=1)

# Convert polar (radii, angles) coords to cartesian (x, y) coords.
# (0, 0) is manually added at this stage,  so there will be no duplicate
# points in the (x, y) plane.
x = np.append(0, (radii*np.cos(angles)).flatten())
y = np.append(0, (radii*np.sin(angles)).flatten())

# Compute z to make the pringle surface.
z = np.sin(-x*y)

# Inspect the array shapes
print(x.shape)                    # gives: (289,)
print(y.shape)                    # gives: (289,)
print(z.shape)                    # gives: (289,)

# Create the plot and show it

fig = plt.figure()
ax = fig.gca(projection='3d')

ax.plot_trisurf(x, y, z, linewidth=0.2, antialiased=True)

plt.show()