autoencoder for features selection I m using a data set with 41 features numerics and nominals the 42 one is the class (normal or not) first I changed all the nominals features to numeric since the autoencoder requires that the imput vector should be numeric. so the number of features incresed from 42 to 122. I removed the class colomn because AE use unlabelled data and I used it to reduce dimensionality from 121 to 10 ( 121> 50->10->50-121) now I want to build a MLP to classify the data I divided the data set into 3 parts: train, validate and test set, I want to put the 10 features selected by the AE instead of the 121 but I dont know how (code?). and how add the class colomn again to the data set to do a supervised classification with MLP?

An autoencoder is meant to do exactly what you are asking. It is a means to take an input feature vector with $m$ values, $X \in \mathbb{R}^m$ and compress it into a vector $z \in \mathbb{R}^n$ when $n < m$. To do this we will design a network that is compressed in the middle such that it looks this.

We train this network by comparing the output $X'$ to the input $X$. This will cause $X'$ to tend towards $X$, thus despite the feature compression in the network, the output will preserve sufficient information about the input such that the input $X$ can be recovered.

Once this network is trained, we can then truncate everything after the layer which outputs the vector $z$. Then you can use the feature vector $z$ as the input features to train a different neural network which you can use to classify your instances as normal or not. During this process you will NOT tune any of the weights of the autoencoder. It will only be used as a feed-forward network.

The autoencoder

We will train an autoencoder on the MNIST dataset.

First we will download all the data

from keras.datasets import mnist
import numpy as np

(x_train, y_train), (x_test, y_test) = mnist.load_data()

x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.

print('Training data shape: ', x_train.shape)
print('Testing data shape : ', x_test.shape)

  Training data shape:  (60000, 28, 28) Testing data shape :  (10000,
  28, 28)

Let us see the distribution of our output classes for the MNIST data

import matplotlib.pyplot as plt
%matplotlib inline

training_counts = [None] * 10
testing_counts = [None] * 10
for i in range(10):
    training_counts[i] = len(y_train[y_train == i])/len(y_train)
    testing_counts[i] = len(y_test[y_test == i])/len(y_test)

# the histogram of the data
train_bar = plt.bar(np.arange(10)-0.2, training_counts, align='center', color = 'r', alpha=0.75, width = 0.41, label='Training')
test_bar = plt.bar(np.arange(10)+0.2, testing_counts, align='center', color = 'b', alpha=0.75, width = 0.41, label = 'Testing')

plt.xlabel('Labels')
plt.xticks((0,1,2,3,4,5,6,7,8,9))
plt.ylabel('Count (%)')
plt.title('Label distribution in the training and test set')
plt.legend(bbox_to_anchor=(1.05, 1), handles=[train_bar, test_bar], loc=2)
plt.grid(True)
plt.show()

Let us look at some examples of the MNIST dataset

import matplotlib.pyplot as plt
%matplotlib inline

# utility function for showing images
def show_imgs(x_test, decoded_imgs=None, n=10):
    plt.figure(figsize=(20, 4))
    for i in range(n):
        ax = plt.subplot(2, n, i+1)
        plt.imshow(x_test[i].reshape(28,28))
        plt.gray()
        ax.get_xaxis().set_visible(False)
        ax.get_yaxis().set_visible(False)

        if decoded_imgs is not None:
            ax = plt.subplot(2, n, i+ 1 +n)
            plt.imshow(decoded_imgs[i].reshape(28,28))
            plt.gray()
            ax.get_xaxis().set_visible(False)
            ax.get_yaxis().set_visible(False)
    plt.show()

show_imgs(x_train, x_test)
print('Training labels: ', y_train[0:10])
print('Testing labels : ', y_test[0:10])

  Training labels:  [5 0 4 1 9 2 1 3 1 4] Testing labels :  [7 2 1 0 4 1 4 9 5 9]

These are some imports we will use or not for making our model. Note: not all of these are needed but I'm too lazy to sift through and pick the useful ones.

from __future__ import print_function
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras.callbacks import ModelCheckpoint
from keras.models import model_from_json
from keras import backend as K
from keras.layers import Input, Dense, Conv2D, MaxPooling2D, UpSampling2D, Flatten, Reshape
from keras.models import Model
from keras import backend as K

Let us build our model. Notice we have the encoder, this maps the input from the higher dimension to the constrained dimension in the middle of the network. It goes from a vector of dimension 784 at the input to a vector $z$ of dimension 128. Then we have a decoder that is a mirror of the encoder which will try to decompress the vector $z$.

input_img = Input(shape=(28, 28, 1))  # adapt this if using `channels_first` image data format

input_img = Input(shape=(28, 28, 1))  # adapt this if using channels_first image data format

x = Conv2D(16, (3, 3), activation='relu', padding='same')(input_img)
x = MaxPooling2D((2, 2), padding='same')(x)
x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)
x = MaxPooling2D((2, 2), padding='same')(x)
x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)
encoded = MaxPooling2D((2, 2), padding='same')(x)

# at this point the representation is (4, 4, 8) i.e. 128-dimensional

x = Conv2D(8, (3, 3), activation='relu', padding='same')(encoded)
x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)
x = UpSampling2D((2, 2))(x)
x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)
x = UpSampling2D((2, 2))(x)
x = Conv2D(16, (3, 3), activation='relu')(x)
x = UpSampling2D((2, 2))(x)
decoded = Conv2D(1, (3, 3), activation='sigmoid', padding='same')(x)

autoencoder = Model(input_img, decoded)
autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')

You can see a description of the model using

autoencoder.summary()

Now let us train this model

from keras.callbacks import TensorBoard
epochs = 50
batch_size = 128

autoencoder.fit(x_train_reshaped, x_train_reshaped, epochs=epochs, batch_size=batch_size,
               shuffle=True, validation_data=(x_test_reshaped, x_test_reshaped), verbose=1,
               callbacks=[TensorBoard(log_dir='/tmp/autoencoder')])

This will take a while. I will get back to you when it is done training. However, once this is done we will take the autoencoder model and we will separate the encoder and decoder part. We will trash away the decoder and only use the encoder.

Then we will use that as a forefront to any model we want to use to classify these digits. We will pass every image through our encoder, to get this compressed information vector $z$ and we will use that as the input to our classification model.

The classifier

The classification model will then look something like

# The known number of output classes.
num_classes = 10

# Input image dimensions
img_rows, img_cols = 28, 28

# Channels go last for TensorFlow backend
x_train_reshaped = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)
x_test_reshaped = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)
input_shape = (img_rows, img_cols, 1)

# Convert class vectors to binary class matrices. This uses 1 hot encoding.
y_train_binary = keras.utils.to_categorical(y_train, num_classes)
y_test_binary = keras.utils.to_categorical(y_test, num_classes)

The classification model

model  ==  Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
                 activation='relu',
                 input_shape=input_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

We will train the classification model

# Save the model# Save t
model_json = model.to_json()
with open("weights/model.json", "w") as json_file:
    json_file.write(model_json)

# Save the weights using a checkpoint.
filepath="weights/weights-improvement-{epoch:02d}-{val_acc:.2f}.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
callbacks_list = [checkpoint]

epochs = 100
batch_size = 128
# Fit the model weights.
model.fit(x_train_reshaped, y_train_binary,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          callbacks=callbacks_list,
          validation_data=(x_test_reshaped, y_test_binary))

score = model.evaluate(x_test_reshaped, y_test_binary, verbose=0)
print('Model accuracy:')
print('Test loss:', score[0])
print('Test accuracy:', score[1])

print('Predict the classes: ')
prediction = model.predict_classes(x_test_reshaped[0:10])
show_imgs(x_test)
print('Predicted classes: ', prediction)


Data driven project for a self-catering accommodation business I want to apologise in advance for my ignorance but I'm hitting my head in a wall here as I'm not sure how to proceed with my project and I've got no experience in that field even though I'm doing a masters in Data Science (don't slag me too much pls)
So I've started a project with this small business that offer self-catering accommodation like Airbnb and what they want is to increase their occupancy. Basically they have properties which perform well and some that doesn't and they want to know why. I've got a list with variables but no clue how to use any other software than weka and I am not sure if it will be appropriate. I am also clueless how to proceed with the analysis so I would be grateful if you have any ideas. I've added a screenshot of the variables to give you an idea.
My university supervisor is not very helpful so I've decided to reach out for help here and any tips will be much appreciated.

Would it be more helpful to provide you with the excel sheet? I am not sure how to upload it

I would start with a simple decision tree regression with the occupancy rate as target value. By visualizing the tree after training you can see which features impact the occupancy rate. You can do this in Weka with M5P regression: http://weka.sourceforge.net/doc.stable/weka/classifiers/trees/M5P.html


decision -tree regression to avoid multicollinearity for regression model? I read in comments a recommendation for decision tree´s instead of linear models like neural network, when the dataset has many correlated features. Because to avoid multicollinearity.
A similar question is already placed, but not really answered.
https://stats.stackexchange.com/questions/137573/do-classification-trees-need-to-consider-the-correlation-between-attributes
or here
In supervised learning, why is it bad to have correlated features?
https://www.quora.com/Is-multicollinearity-a-problem-in-decision-trees#:~:text=Decision%20trees%20follow%20the%20non%20parametric%20approach.&text=Though%20single%20tree%20leads%20to,robust%20to%20the%20multi%20collinearity%20.
My problem:
I have a dataset of about 30 columns. 10 columns have a high correlation with the target/dependend variable. Data are numerical. I would like to do a prediction (regression model) include all variables if possible?
One big problem is to avoid multicollinearity.

Is there a decision tree regression model good when 10 features are high correlated? (if I follow the answers of the links, but there is no really good explanation for that).
Is there a scientific or math explanation or recommendation (to use decision tree regression)?

To answer your questions directly, first:

Is there a decision tree regression model good when 10 features are
high correlated?

Yes, definitely. But even better than decision trees, is many decision trees (RandomForest, Gradient Boosting (xGBoost is popular). I think you'd be well served by learning about how decision trees split, and how they naturally deal with collinearity. Maybe try this video Follow the logic until the 2nd tier of splits, and you'll be able to imagine how the correlated variables are suddenly not important because they're relative to the split above them.

Is there a scientific or math explanation or recommendation (to use
decision tree regression)?

The math explanation of why collinearity is "bad" for linear models, comes down to the coefficients and how you interpret them. One of the side effects is that they can undermine the statistical significance of a variable, as well as flip their coefficients the wrong direction. It usually doesn't affect the accuracy of the model very much, but most people want linear models so that they can interpret the coefficients (which is totally messed up with collinearity). I suggest reading maybe this article to start.
One of the things that you mentioned, include all variables if possible? is not really something you should be concerned with. The goal of a model is to explain the most, with the least. If you're forcing as many variables as possible into the model, then it's possible that you'll be fooled into thinking a model is good, when in fact it isn't if you were to test it on new data. In fact, sometimes less variables will give you a better model. This is exactly the kind of problem that multicollinearity causes with linear models - that you can't really judge very well what variables are significant or not. Stepwise selection doesn't work very well when there are correlated features.
In general, I think decision trees - especially Random Forests - will be a good start for you. But remember not to force all of the variables into the model just for the sake of it. Experiment with using less variables and manipulating the tree structure such as leaf size and max depth. And as always - test your model on validation data and holdout data so that you don't overfit a model and fool yourself into thinking it's a strong model.


Architecture for Machine Learning resource management The problem: We have several people that do a lot of ML work in our lab and have more people asking to use our equipment. We have several machines that have Ubuntu on them that people share. This causes a problem for resource contention. PersonA had all of their code/libraries on MachineA or a person is waiting on MachineB which has 4GPUs instead of 1GPU.

Looking for:
What I’m looking for is a way of creating a cluster with these machines that will allow people to utilize and share the resources. This would allow people to scale out and make better use of the resources that we already have (e.g. waiting for a specific machine that is already used). This could be used by interactively running code in their IDE or submitting jobs to the cluster like some type of scheduler (slurm, pbs). At this point the end user could look at this as a "pool" of resources. (e.g. we have 100GPUs, personA requests 4GPUs, we have 96GPUs available for someone else to request). Would also like to avoid the user from having to anything about the cluster. For example UserA simply requests 4GPUs, they don't know and don't care which machines it gets launched on.

They use a mix of Tenserflow and mxnet with keras and some theano.

Tensorflow/mxnet seem to have clustering option but don’t think this would handle the scheduling problem? or would it?

If you have successfully deployed some type of machine learning cluster with GPU nodes, I would really be interested in seeing your architect, tools, and software you used.

If this is out of scope for this forum please feel free to move this to another channel.

It depends on how the cluster (i.e., compute resources) are being managed. For example, Apache Mesos is an open-source cluster management tool.

One established option is Horovod. It is designed to manage workloads for
TensorFlow and Keras across multiple cores in a single machine or across multiple machines.


percentage of confidance on desion trees results I am looking towards a solution where classification algorithms produce output with some confidence value. but I am confused whether classification algorithms are able to produce results with percentage of confidence?
Thanks

Some classification algorithms can indeed return a probability distribution over the considered classes (see Wikipedia on probabilistic classification).

In the topic of your question you're asking about Decision Trees. Well, these have their limitations in terms of providing probability estimates (see this paper on probability estimates from decision trees).

In case you would like to play around with this, it's very easy to start with scikit-learn:

import numpy as np
from sklearn.tree import DecisionTreeClassifier

dtc = DecisionTreeClassifier()

# training samples
X = np.array([[1, 1, 0],
              [1, 1, 0],
              [0, 0, 1],
              [0, 0, 1]])

# target values for training samples
y = np.array([0, 0, 0, 1])

dtc.fit(X, y)

print 'Class probabilities for training samples:'
print dtc.predict_proba(X)
print 'Probabilities for previously unseen samples:'
for sample in ((1, 0, 1), (0, 0, 1), (1, 1, 1), (0, 0, 0)):
     print 'Sample {}. Result: {}'.format(sample, dtc.predict_proba(sample))

This code returns following results:

Class probabilities for training samples:
[[ 1.   0. ]
 [ 1.   0. ]
 [ 0.5  0.5]
 [ 0.5  0.5]]
Probabilities for previously unseen samples:
Sample (1, 0, 1). Result: [[ 0.5  0.5]]
Sample (0, 0, 1). Result: [[ 0.5  0.5]]
Sample (1, 1, 1). Result: [[ 0.5  0.5]]
Sample (0, 0, 0). Result: [[ 1.  0.]]

At this scale the results are easily interpretable:

a sample of features (1, 1, 0) is classified as 0 with 100% probability
a sample of features (0, 0, 1) is classified 50/50 as 0 or 1. etc.

This brings us to an important question of how accurate is your model?


How to use random forest with large number of categorical features and categories? I have 2 features productName and productCategory , both of them are strings. I have a category named supplier. There are 4000 suppliers and 500,000 items in test data. I don't think one hot encoding will be a good approach to deal with such a big data. How should i handle these using random forest?

You have a few options on your hand:
Option 1 - Augment you dataset with additional Supplier-related features
Do you have additional features about the supplier that makes sense to augment into your dataset ? For e.g. you might be able to replace the Supplier field with something like "Years in Business" , "Primary Industry", "Reputation Score" etc.
That way you convert the categorical variable into something more meaningful that can be compared across the universe of Suppliers.
Go this route if and only if you have meaningful and easily accessible features for your problem domain.
Option 2 - Encode each Supplier with a unique integer
I know this sounds stupid - but it really works for Random Forest models (wouldn't work on simple Linear Regression for sure).
Random Forest models are scale invariant. They won't get biased towards suppliers with a larger numeric Supplier Id. They'll figure out the best way to split your Supplier Ids to meet your objective function.
I would recommend Option 2 to start with since that is the simplest.


Does anyone know how google play music's recommendation engine works? What sort of models do they use?  Presumably some flavor of neural net.  Do they do a lot of feature engineering?  Or do they throw in huge raw matrices of oscilloscope output?  Given that it must be supervised learning, what is their target?  A thumbs up?  How do they deal with heterogeneity in the population? We're not independent nor are we identically distributed.

It seems like it works better lately.  Curious how it works.

Deep neural networks, probably with tens of internal layers, distributed across many many machines. Each user probably receives scores from many deep networks for various applications, and each network is probably automatically scored to determine when its particular model is most helpful. This is the pattern used across the industry. Music companies have the additional complexity of trying to parse sound data. Spotify does this with their Release Radar recommender.

As far as targets go, they probably build models to predict all forms of engagement. You probably get scores for listens, scores for time listening, scores for thumbs up or down, scores for add-to-list, and so on. Ensemble methods work well in practice and Google probably gives you on the order of dozens to hundreds of different machine-learned predictions.


How to train a neural network for high recall? I would like to train a neural network for named entity recognition to tag an unlabeled dataset of texts. The generated labels will then be checked via a crowdsourcing platform. The goal is to annotate the dataset. Therefore, the neural net should find all possible entities in the text, i.e. have high recall rather than precision.

What would be the best way to train a neural network for high recall, i.e. assigning lower cost to false positives than to false negatives? Could the loss function be changed from negative log likelihood to something else to encourage high recall?

There is a very simple way of increasing the recall of a network without requiring retraining:

A network's output is the probability it gives that a sample belongs to a class, let's say class $0$ or class $1$. The output of the net would look like this $(0.53, 0.47)$. This means that the net gives a $53\%$ probability of the sample being class $0$ and a $47\%$ probability of it being class $1$. Normally we check which class has the highest probability and consider that to be the network's prediction. However, you could set a threshold over which the sample would be considered to be class $1$.

This way you would get more samples predicted to class $1$, which would increase the class' recall (at the cost of its precision though).


What is the use of fit method in sklearn.preprocessing.Normalizer()? According to the documentation of fit(self, X[, y]) method of sklearn.preprocessing.Normalizer(), it does nothing and return the estimator unchanged.
I understand that if I intend to normalize data I can simply pass the data to the Normalizer() function. so what is the use of use of fit method. Moreover, normalization is not a learning algorithm so why is there a fit() method?

In scikit-learn, many preprocessing operations follow the Estimator API (i.e. having fit and transform methods).  The benefit of conforming to the Estimator API is that the object can be included in a data transformation pipeline.  Some of the benefits of pipelines are described in the docs:

  Pipeline can be used to chain multiple estimators into one. This is useful as there is often a fixed sequence of steps in processing the data, for example feature selection, normalization and classification. Pipeline serves multiple purposes here:

  Convenience and encapsulation:
  You only have to call fit and predict once on your data to fit a whole sequence of estimators.

  Joint parameter selection:
  You can grid search over parameters of all estimators in the pipeline at once.

  Safety:
   Pipelines help avoid leaking statistics from your test data into the trained model in cross-validation, by ensuring that the same samples are used to train the transformers and predictors.

Because the Normalizer estimator is stateless, its fit method is a no-op.  But if it was missing the fit method, then it could not be used in scikit-learn Pipelines.


List of NLP challenges Is there any comprehensive list of past, current and future NLP challenges?

E.g. for NLP conferences, Joel Tetreault's unofficially official conference calendar and WikiCFP are pretty good.

The "Competitions and Challenges" page on the ACL wiki quite incomplete.

List of NLP competitions on Kaggle by popularity [number of teams]:

Two Sigma Connect: Rental Listing Inquiries [2709 teams]
https://www.kaggle.com/c/two-sigma-connect-rental-listing-inquiries
Home Depot Product Search Relevance [2125 teams]
https://www.kaggle.com/c/home-depot-product-search-relevance
Quora Question Pairs [2123 teams]
https://www.kaggle.com/c/quora-question-pairs
What's Cooking? [1388 teams] https://www.kaggle.com/c/whats-cooking
Crowdflower Search Results Relevance [1326 teams]
https://www.kaggle.com/c/crowdflower-search-relevance
Bag of Words Meets Bags of Popcorn [578 teams]
https://www.kaggle.com/c/word2vec-nlp-tutorial
Transfer Learning on Stack Exchange Tags [380 teams]
https://www.kaggle.com/c/transfer-learning-on-stack-exchange-tags/data
Facebook Recruiting III - Keyword Extraction [367 teams] https://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction

New entrants are prohibited, you won't be able to download data for this competition

Dato: Truly Native? [274 teams] https://www.kaggle.com/c/dato-native
The Allen AI Science Challenge [170 teams]
https://www.kaggle.com/c/the-allen-ai-science-challenge


Why is there a trade-off between bias and variance in supervised learning? Why can't we have best of both worlds? The bias-variance trade-off is like a law in machine learning. You cannot have the best of both worlds. What is it about supervised learning in machine learning that makes it impossible to satisfy the two at the same time?

The tradeoff between bias and variance summarizes the "tug of war" game between fitting a model that predicts the underlying training dataset well (low bias) and producing a model that doesn't change much with the training dataset (low variance).

What statisticians/mathematicians a while ago realized is that any model can be made to perfectly fit the dataset at hand (i.e. have zero bias). Look at this picture, for instance from the Wikipedia page on overfitting:

The graph depicts two binary classifiers that are trying to distinguish between the blue class and the red class. The classifiers have been trained on all of the blue and red observations shown in the graph to generate their boundaries. Notice that the green line has zero bias; it perfectly separates the blue and red classes whereas the black line clearly has a few errors on the training set and therefore has higher bias.

The problem? Observations in datasets are realizations of random variables that follow some unknown probability distribution. Thus, observed data will always have some sort of noise, containing observations that don't actually represent the underlying probability distribution. These anomalies can be seen in the graph above in which there are red and blue observations that appear to be "crossing" the implied boundary between the two classes. Clearly, the majority of observations are easily distinguishable in this simple example except for a few cases. These few cases can be viewed as noise, which is inherently random and therefore, cannot be predicted.

The green line, as a result, is essentially fitting random noise which by definition is unpredictable and non-representative. The result is that if we were to train both classifiers again using a new training dataset (from the same process), the "green boundary model" will be expected to generate a very different boundary compared to what is shown above, since it was influenced by data that did not represent the actual underlying data generating process. This is overfitting = high variance. The model associated with the black line, which has refrained from fitting the noise of the data, should be expected to remain relatively stable since it was not influenced by data that was not actually representative of the population.

What does this all mean? The model associated with the black line is closer to reality (the "truth"), and therefore, if we were to pull new observations from "reality" that neither classifier has seen it will, on average, be expected to produce lower overall error despite having higher bias. This is the bias-variance tradeoff, which is really just a way of stating the tradeoff between underfitting and overfitting. I don't have another graph to illustrate this, but imagine if I straight up drew a diagonal line from the bottom left corner to the top right corner, and called this another classifier. Now, my "model" has zero variance; in fact, it is completely independent of the training dataset. But now, I haven't even fit the training dataset. I now have zero variance for tons of bias and large overall error.

The decomposition of MSE into bias, variance, and irreducible error encapsulates the tradeoff. When you hope to fit a model that generalizes well to the population, simply having a model with low bias is not enough and minimizing variance is also important, if not more important. But, if we are to take variance into consideration, we now have to sacrifice some ability to predict the training set well (through regularization) in hopes of obtaining a model that is closer to the truth.

Almost all ML methods use regularization to effectively trade bias (again, worse performance on the training dataset) for hopefully, lower variance. Some examples:

L1/L2/elastic nets introduce extra terms to the unbiased maximum likelihood estimators for the regression coefficients in GLM's. The introduction of these terms make the estimators biased, but the idea is to hopefully penalize in an effective way such that we get bigger gains in reducing variance, which in turn results in lower overall error.
Support vector machines (see the picture above) have a cost parameter that controls how smooth or "wiggly" the boundary is, which as you can see in the graph above trades bias by purposely missing some observations in the training dataset for a model that generalizes better to the population overall.
Neural networks have dropout layers that effectively introduce bias by simply removing hidden units from the model at random. The goal here is to purposely sack parts of the model that have learned from the training dataset in hopes of preventing complex co-adaptions that are highly specific to the training set and not to the general population.
Decision trees can be controlled through pruning (cost complexity, removing branches that do not significantly reduce in sample error/lead to large enough information gains) and through limiting the depth of the tree. The result of both methods? Again, higher bias (removing rules near the bottom of the tree that have potentially been created to better predict non-representative noise observations, leading to sparse terminal nodes) but hopefully, lower variance.


Group similar words under one topic and assign them a title I am working on a natural language processing data problem and I have selected some keywords from it as features. I want to group them under one heading. But I can't find any method or algorithm to do that? I tried topic modelling using MALLET. It groups similar words into a cluster. I want a heading or subject attached to them.

Is there a way to group similar words under one topic and assign them a title?

There are a number of different features that need to get one topic like picture quality camera selfie resolution that should come under camera. Need something that groups these things I enter words and it gives me the topic.

If you have already selected the keywords you want grouped, why not write a function that finds all occurrences of words in the list and replaces it with your one, core word?

What you're describing sounds like a more advanced version of stemming.


the library 'transformers' works also with older version of Tensorflow? i am working with Tensorflow version 1.14 and i would like to use the bert embedding.
In order to do so, i was thinking to use the transformers library( https://pypi.org/project/transformers/)  but i am not sure if that will work with my tensorflow version.

It does work but I recommend using TF2 for any new developments. In the end, if something does not work, you will have to port it back to TF2. Even though there are not significant difference in the semantics of the framework


What is the hypothesis space of decision tree learning? Could you please explain what the hypothesis space for decision tree learning look like?

And what is the cardinality of this space?

As per Tom Mitchell's,

".....For example, consider the space of
hypotheses that could in principle be output by the above checkers learner. This
hypothesis space consists of all evaluation functions that can be represented by
some choice of values for the weights wo through w6. The learner's task is thus to
search through this vast space to locate the hypothesis that is most consistent with
the available training examples....."

Hence , Basically all possible combination of distinct trees makes the hypothesis space.

Lets say if you have chosen to represent your function to be a linear line then all possible linear lines which go through the data (given input, output) makes up your hypothesis space.

Each tree= Single hypothesis , that says this tree shall best fit my data and predict the correct results

therefore combination of all such possible tress= hypothesis space.

Here is the snippet of PPT from lecture


Relation between Cross Validation and Confidence Intervals I've read from a source which I forgot where that 'In cross validation, the model with best scores at 95% confidence interval is picked'.
But according to my stat knowledge, in order for CI (confidence interval) to works, you need normality assumption about the sampling statistics of the experiment.

But how come from that unknown source it seems to simply use results from each flow to construct the sample mean & the confidence interval. It seems to me that neither checking if central limit theorem testing at all. And it seems to me this is what people are doing as well:
i) automatically assume normality in sampling MEANS (instead of sampling distribution)
ii) CLT automatically satisfied.
May I know if it's my misunderstanding or the industry is adopting a norm which is too loose? Thanks.

It depends how on the confidence interval (CI) is generated. The most common method is on a sample mean with the assumption that the samples are drawn from a normal distribution . However, a CI can be generated from any statistic from observed data. An alternative method would be through bootstrapping, resampling the statistic, which does not require the normality assumption.


Why does my cost either increase or go down very slowly def hypothesis(W, B, X):
  Z = np.dot(W.T, X) + B
  return sigmoid(Z)

def sigmoid(Z):
  return(1/(1+np.exp(-Z)))

def cost(A, Y):

  return (-Y * np.log(A) - (1 - Y) * np.log(1 - A)).mean()

def derivatives(W, B, X, Y):

  A = hypothesis(W, B, X)

  return np.dot(X.T, (A - Y)) / Y.shape[0] , (A - Y) / Y.shape[0]

def updateParameters(W, B, X, Y, learning_rate):
  # print(W.shape, X.shape, Y.shape )

  dW, dB = derivatives(W, B, X, Y)
  W -= learning_rate * dW
  B -= learning_rate * dB

  return W, B

def LogisticRegression(X, Y):
  W = np.zeros((X.shape[0], 1))
  B = np.zeros((X.shape[0], 1))

  for i in range(0, 1000000):
    print(cost(hypothesis(W, B, X), Y))
    W, B = updateParameters(W, B, X, Y, 0.0000003)

LogisticRegression(X, Y)

I'm a beginner trying to implement logistic regression from scratch.

However, my cost always goes up, and when I reduced my learning rate in case that was too high, it started going down but only extremely slowly.

Could someone point out if there is any issue in the implementation of the algorithm? Any help would be greatly appreciated

I would refer you to this website that provides a very extensive and detailed walk through the math behind the logistic regression and it's cost function.
https://www.internalpointers.com/post/cost-function-logistic-regression

And here is a complete tutorial on how to manually code the whole thing in python:

https://towardsdatascience.com/building-a-logistic-regression-in-python-301d27367c24

Finally, I don't know the reason you need to implement this manually, there are pretty good ready-to-use functions that have been used and tested and that would save you a lot of time.

Good luck !


Search and Replace in specific column of csv with python I created a program that search and replaces over an entire csv file but I need to make so it is column specific. Is there a way to only search via a column. Here is my code, I am pretty new to python so I apologize if this is an easy fix.

import csv

ifile = open('testbook.csv', 'rb')
reader = csv.reader(ifile,delimiter='\t')
ofile = open('output.csv', 'wb')
writer = csv.writer(ofile, delimiter='\t')

findL = ['Australia', 'Mexico', 'United States', 'hob']
replaceL = ['Kangaroo', 'Spider Monkey', 'Eagle', 'Test']

rep = dict(zip(findL, replaceL))

def findReplace(find, replace):
    s = ifile.read()
    for item, replacement in zip(findL, replaceL):
        s = s.replace(item, replacement)
    ofile.write(s)

for item in findL:
    findReplace(item, rep[item])

ifile.close()
ofile.close()

You can do it easily by using pandas, like this:

import pandas as pd

data = [['Austria', 'Germany', 'hob', 'Australia'],
        ['Spain', 'France', 'Italy', 'Mexico']]

df = pd.DataFrame(data, columns = ['A','B','C','D'])

# Values to find and their replacements
findL = ['Australia', 'Mexico', 'United States', 'hob']
replaceL = ['Kangaroo', 'Spider Monkey', 'Eagle', 'Test']

# Select column (can be A,B,C,D)
col = 'C';

# Find and replace values in the selected column
df[col] = df[col].replace(findL, replaceL)


What is difference between attention mechanism and cognitive function? How are attention mechanism used in different deep learning algorithms different from the cognitive function attention?

Are you reffering to biological "cognitive function attention"?
If yes than it could be argued its different.
TL;DR
Fixed-length Context vector in the encoder decoder architecture could not remember long term dependencies. Attention could learn  these long term dependencies, hence have a long-term memory. If we speak about human attention it is usually short spaned (relative).
On the other hand I believe it was Yoshua who first talked in the terms of attention and one could argue that we could implicitly focus (our attention) on some older parts of text and hence extract deeper meaning.
TL;DR TL;DR One can make a case its actually more natural to talk about long-term memory when discussing attention-ML-mechanism


How to create a matrix from two given vectors in R in RStudio? Suppose, $c(1, 2, 3, 4)$ and $c(2, 4, 5, 6)$ are two vectors. Then in R or RStudio,

  How to create a $4\times 2$ matrix from these two vectors?

Also,

  how to add another vector $c(8, 9, 10, 11)$ as a column to the previous matrix?


Minimal (numerical) example for simple neural net I‘m looking for a good minimal numerical example for teaching how a neural net is solved. Basically the idea is to see how the hidden layers (i.e. the weights) work using „pen and paper“.

Back in Uni I did similar things „by hand“ for OLS regression on a 3x3 matrix.

Does anyone know about a good source/example for neural nets?

Please check the work of Andrew Trask in this book.

Most detailed work on this that I have seen.

grokking Deep Learning


I am trying to figure out the stationarity of time series data Here, this plot shows the number of customers served per day from 1 jan 2018 to 31 dec 2019.
I grouped the entire data by each month and calculated the average and variance per month.
This is the average

This is the variance

I also ran an augmented dickey-fuller test and got the following result
ADF Statistic: -26.031963

p-value: 0.000000

Since, the mean and variance do not vary too much, should I infer that the data is stationary?

There are certainly more tests you could try out but by definition of the test, you can conclude that your data is stationary.
On the other hand I would be concerned with your test results. Getting a p of exactly zero, and a test statistic value of way below -3 or -4  I would look at your implementation again and check if it's right.


Training CNN on a huge data set I am trying to train an AlexNet image model on the RVL-CDIP Dataset. The dataset consists of 320,000 training images, 40,000 validation images, and 40,000 test images.
Since the dataset is huge I started training on 500 (per class) sample from the training set. The result is below:
we can see from the graph above that the validation loss started decreasing at a much slower rate around epoch 20 while training loss continued decreasing the same. This means our model started overfitting the data? I assume that this is probably because the data i have in the training set is not enough to get better results on the validation set? (validation data is also a 500 (per class) sample from the whole validation set)
is it a correct approach to train the model on a small sample (eg. 500 images per class), save the model, load the saved model weights and then train again with a larger sample (eg 1000 images)? My intution is that this way the model would have new data every new run that helps it to learn more about the validation set. And if this approach is correct, when training the model for the second time with a larger sample, should the training sample include images (some or all) that were trained in the first model?
You can find the full code with results here

It reminds me of this question, the training loss is decreasing faster than the validation loss. I understand there is some overfitting, as the model is learning some patterns that are only in the training set, but the model is still learning some patterns that are more general, as the validation loss is decreasing as well. To me it would be more of an issue if the validation loss increased, but it is not the case.
Edit
Usually neural networks are trained with all the data, training by using mini-batch gradient descent already does what you mention in your approach without the need of storing the model in memory. So, I would train with as much data as possible, to have a model with the least possible variance. If you are not feeding the data using generators and the whole dataset doesn't fit into memory, I recommend to use them, or train with a model which is as big as possible given your memory limitations.


ValueError: y should be a 1d array, got an array of shape (1045, 5) instead I have just started Python and working on training models. The task that I have been assigned is to train a dataset named "austin_Weather"
Original Dataset
y attribute
After having done some manipulations (following the article), these are my new datasets for training and testing,
X Train
y Train
Now, I tried to apply Naive-Baysen algorithm on these two datasets, but I got this following error
ValueError: y should be a 1d array, got an array of shape (1045, 5) instead.

So, I tried searching it up online, and I found one solution that I should reshape these arrays, so I tried the following code,
X_train = X_train.reshape(-1, 1)
y_train = y_train.reshape(-1, 1)
X_test = X_test.reshape(-1, 1)
y_test = y_test.reshape(-1, 1)

But I got this error on this,
AttributeError: 'DataFrame' object has no attribute 'reshape'

I tried to fix it up by searching it up online, but I couldn't understand most of the things due to my lack of knowledge in Python syntax. If anyone can help resolve this and tell me what exactly is the problem here, I would be very thankful.

Your data is multilabel: each row can have more than one label. But sklearn's Naive Bayes doesn't support that format of problem. You can use MultiOutputClassifier to wrap the Naive Bayes classifier, effectively training one model for each of the labels.
See the User Guide, especially the MultiOutput classification section. You may also want to consider the ClassifierChain (in the next section).

The last error you see is because you're using pandas structures, but reshape is a numpy method. Anyway, as @Oxbowerce observed, reshaping won't help; you need to deal with having multiple labels.


Fixing misaligned rows using Python I'm trying to read a csv file into a dataframe in Python but when examining the file I noticed they are many misaligned rows . Does anyone know how I can shift these rows so my Dataframe's size is consistent.
data= pd.read_csv('MyData.csv')

What is your seperator in the csv- a comma? It would help if you could open the raw csv file in a text editor and post the result here. You could then check if this seperator can appear in the input data. You then have two options: Change the seperator (if possible) or create a small python script/ manually replace all cases in which this can happen.


Are RNN or LSTM appropriate Neural Networks approaches for multivariate time-series regression? Dear Data Science community,

For a small project, I've started working on Neural networks as a regression tool, but I am still confused about possibilities of some variants. Here's what I am aiming to do:

I have multiple input data time series $X(t)=[X_1(t), X_2(t), X_3(t),X_4(t)]$, and multiple target data time series which I want to modelize $Y(t)=[Y_1(t), Y_2(t)]$. All data are available for training.
I aim to train my model/regression on an interval $[t_0,t_n]$, and then be able to apply it on a larger different interval.
I know that relation between my $Y$ and $X$ are non-linear, but also that in need to take in account lag, or inertia. For example, $Y_1(t)$ is dependant of $X_1([t-dt_1,t])$ and $X_2([t-dt_2,t])$. All $dt_n$ are different, but I have an approximate idea of how 'far' I need to reach.

With this in mind, through some research I have been guided to focus on Recurrent Neural Networks (RNN) and Long Short Term Memory (LSTM) networks. I aim to use TensorFlow/Keras to work on this. However, after some reading, I'm getting confused with those solutions. Many people present them in prediction applications, which supposedely means that those networks use data from a time interval (either $X$ or $Y$ in my case) on an interval $[t_0,t_n]$ to predict $Y(t_{n+1})$. But my objective is to use $X([t_0,t_n])$ to modelize $Y([t_0,t_n])$ (on the same time interval). I am getting confused with this notion of "prediction".

Therefore, are RNN and LSTM networks appropriate solutions for my multivariate time series regression/model project? Or am I already going the wrong way?

As a beginner in this field, any reference or link to ressources/tutorial, or demo, is also gladly welcome.

Here is a really good source to begin multivariate time-series forecasting in Keras using LSTMs.

  I aim to train my model/regression on an interval $[t_0,t_n]$ and then be able to apply it on a larger different interval.

There's no harm in this as long as you perform the right kind of multi-step forecasting. If your problem requires you to train on $[t_0, t_n], \text{for some } n < 100$ and produce $y_1(t_{n+1}), y_2(t_{n+1})$ as outputs then APPLY it on $[t_n, t_n+100]$ then there will be issues in implementation, as most ML models will require you to provide the same shape as you did when you were training your model. Here's when sliding across time-series will help you. Simply put,

$$
\begin{align}
\text{Train on: }& [t_0, t_n] &\text{ Output: }& y_1(t_{n+1}), y_2(t_{n+1}) \\
\text{Predict on: }& [t_1, t_{n+1}] &\text{ Output: }& y_1(t_{n+2}), y_2(t_{n+2})\\
\text{Predict on: }& [t_2, t_{n+2}] &\text{ Output: }& y_1(t_{n+3}), y_2(t_{n+3})\\
& \vdots & & \vdots\\
\end{align}
$$

  I know that relation between my Y and X are non-linear, but also that in need to take into account lag or inertia. For example, Y1(t) is dependent of X1([t−dt1,t]) and X2([t−dt2,t]). All dtn are different, but I have an approximate idea of how 'far' I need to reach.

Well, you can model your time-series data as $X_1(t-k),\cdots,X_1(t), X_2(t-d),\cdots,X_2(t)$ as $X$ (input) and $X_1(t+1), X_2(t+1)$ as $y$ (target), where $k$ and $d$ refer to different lags for each variable.


How to train or approach the image datasets with different resolution in Deep Learning Image classification:

I am having a data set of image collection more than 10k but even though all are the same image but taken in different sizes (pixels into pixels) some are in square and some are in a rectangle. When I am trying to make them similar, I have planned to zoom to the image exactly where my learning is concentrated but I don't know how to do this.

Note these are color images.

I don't whether this is the good approach or any approach is appreciable.

You don't need to change their size. You can zero pad the images when you feed them into the Network. Zero padding creates a "frame of zeros" around each image, so that they all take equal shape.

conv layers in Keras / TensorFlow 2.0 come with already built-in zero padding arguments.


matplotlib graph to plot values and variance I am really new to the world of matplotlib graphing as well as using those graphs to understand data.

I have written a simple python code where I read a .csv file in and then store the values of one column into a variable. Then plotting them similar to the code bellow:

dev_x= X   #storing the values of the column to dev_x
plt.plot(dev_x)
plt.title('Data')

The graph looks like this, which seems quite messy and hard to understand. So, I am asking for some advice on how to make more cohesive graphs.

This is what my .csv column looks like. It is just many other other rows.

['40' '20' '10' '0' '10' '30' '50' '70' '90' '110' '130' '150' '170' '200'
 '240' '290' '40' '20' '10' '0' '10' '30' '50' '70' '90' '110' '130' '150'
 '170' '200' '240' '290' '40' '20' '10' '0' '10' '30' '50' '70' '90' '110'

At the end of the day I would like a way to display these in a better way so I can also find the variance of this column.

You have currently stored your numbers as strings causing matplotlib to treat your variable as categorical, hence the y-axis is not ordered as expected. Before plotting you should therefore first convert them to integers like this:

x = [float(i.replace(",", ".")) for i in dev_x]

You can then use plt.plot(x) once again to plot the values, this should give you the following plot:

Edit:

Using the csv file you've provided, I am using the following code to read in the data and create the plot:

import matplotlib.pyplot as plt
import pandas as pd

# Read in csv file
df = pd.read_csv("DATA.csv")
# Set figure size
plt.figure(figsize=(15, 5))
# Create plot
plt.plot(df["DATA"])

This should give the following plot:


What is the best performance metric used in balancing dataset using SMOTE technique I used smote technique to oversample my dataset and now I have a balanced dataset. The problem I faced is that the performance metrics; precision, recall, f1 measure, accuracy in the imbalanced dataset are better performed than with balanced dataset.

Which measurement can I use to show that balancing dataset may improve performance of the model?

NB: roc_auc_score is better in balanced datset than roc_auc_score with imbalanced dataset
Can it be considered as a good performance measurement?
after the explanation i implemented code and i got this results

import pandas as pd
import numpy as np
from sklearn import preprocessing
import matplotlib.pyplot as plt
plt.rc("font", size=14)
from sklearn.svm import LinearSVC
from sklearn.svm import SVC
from sklearn.cross_validation import train_test_split,StratifiedShuffleSplit,cross_val_score
import seaborn as sns
from scipy import interp
from time import *
from sklearn import metrics
X=dataCAD.iloc[:,0:71]
y= dataCAD['Cardio1']
# Split the dataset in two equal parts
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=0)
print(y_test.value_counts())
model=SVC(C=0.001, kernel="rbf",gamma=0.01, probability=True)
t0 = time()
clf = model.fit(X_train,y_train)
y_pred = clf.predict(X_test)
t = time() - t0
print("=" * 52)
print("time cost: {}".format(t))
print()
print("confusion matrix\n", metrics.confusion_matrix( y_test, y_pred))
cf=metrics.confusion_matrix(y_test, y_pred)
accuracy=(cf.item((0,0))/50)+(cf.item((1,1))/14)
print("model accuracy \n",accuracy/2)
print()
print("\t\tprecision_score: {}".format(metrics.precision_score( y_test, y_pred, average='macro')))
print()
print("\t\trecall_score: {}".format(metrics.recall_score(y_test, y_pred, average='macro')))
print()
print("\t\tf1_score: {}".format(metrics.f1_score(y_test, y_pred, average='macro')))
print()
print("\t\troc_auc_score: {}".format(metrics.roc_auc_score( y_test, y_pred, average='macro')))

Results:

Name: Cardio1, dtype: int64
====================================================
time cost: 0.012008905410766602

confusion matrix
 [[50  0]
 [14  0]]
model accuracy
 0.5

        precision_score: 0.390625

        recall_score: 0.5

        f1_score: 0.43859649122807015

        roc_auc_score: 0.5

For balanced dataset

X_train1,y_train1 = sm.fit_sample(X_train, y_train.ravel())
df= pd.DataFrame({'Cardio1': y_train1})
df.groupby('Cardio1').Cardio1.count().plot.bar(ylim=0)
plt.show()
print(X_train1.shape)
print(y_train1.shape)
#model=SVC(C=0.001, kernel="rbf",gamma=0.01, probability=True)
model=SVC(C=10, kernel="sigmoid",gamma=0.001, probability=True)
t0 = time()
clf = model.fit(X_train1,y_train1)
y_pred = clf.predict(X_test)
t = time() - t0
print("=" * 52)
print("time cost: {}".format(t))
print()
print("confusion matrix\n", metrics.confusion_matrix(y_test, y_pred))
cf=metrics.confusion_matrix(y_test, y_pred)
accuracy=(cf.item((0,0))/50)+(cf.item((1,1))/14)
print("model accuracy \n",accuracy/2)
print()
#print("\t\taccuracy: {}".format(metrics.accuracy_score( y_test, y_pred)))
print()
print("\t\tprecision_score: {}".format(metrics.precision_score( y_test, y_pred, average='macro')))
print()
print("\t\trecall_score: {}".format(metrics.recall_score(y_test, y_pred, average='macro')))
print()
print("\t\tf1_score: {}".format(metrics.f1_score(y_test, y_pred, average='macro')))
print()
print("\t\troc_auc_score: {}".format(metrics.roc_auc_score( y_test, y_pred, average='macro')))

Results:

(246, 71)
(246,)
====================================================
time cost: 0.05353999137878418

confusion matrix
 [[ 0 50]
 [ 0 14]]
model accuracy
 0.5

        precision_score: 0.109375

        recall_score: 0.5

        f1_score: 0.1794871794871795

        roc_auc_score: 0.5

I found  no efficient results. Should i implement the model using cross validation?

First of all, just to be clear, you shouldn't evaluate the performance of your models on the balanced data set. What you should do is to split your dataset into a train and a test set with ideally the same degree of imbalance. The evaluation should be performed exclusively on the test set, while the balancing on the training set.

As for your question, any macro averaged metric should do just fine for proving that your balancing technique is effective. To calculate such a metric (let's say accuracy for simplicity), you just need to compute the accuracies of each class individually and then average them.

Example:
We trained two models m1 and m2, the first without balancing the dataset and the second after using SMOTE to balance the dataset.

Actual values :  0, 0, 0, 0, 0, 0, 0, 0, 1, 1
Predicted m1: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0  <-- only predicts majority class
Predicted m2: 1, 0, 0, 1, 0, 1, 0, 0, 1, 1

How would we normally calculate accuracy?

$acc = \frac{correct \, predictions}{total \, predictions}$

How do our two models perform on this metric?

$acc_1 = \frac{8}{10} = 80\%$
$acc_2 = \frac{7}{10} = 70\%$

According to this performance metric, m2 is better than m1. However, this isn't necessarily the case as m1 just predicts the majority class! In order to show how m2 is better than m1, we need a metric that treats the two clases as equals.

We' ll now try to calculate a macro-averaged accuracy. How? First we'll calculate the accuracy for each class separately, and then we'll average them:

For m1:
$acc_1^0 = \frac{8}{8} = 100\%$ <-- m1's accuracy on class 0
$acc_1^1 = \frac{0}{2} = 0\%$  <-- m1's accuracy on class 1
$macro\_acc_1 = \frac{acc_1^0 + acc_1^1}{2} = \frac{100\% + 0\%}{2} = 50\%$
For m2:
$acc_2^0 = \frac{5}{8} = 62.5\%$ <-- m2's accuracy on class 0
$acc_2^1 = \frac{2}{2} = 100\%$  <-- m2's accuracy on class 1
$macro\_acc_2 = \frac{acc_2^0 + acc_2^1}{2} = \frac{62.5\% + 100\%}{2} = 81.25\%$

Notes:

Macro averaging can be applied to any metric you want, however it is most common in confusion matrix metrics (e.g precision, recall, f1).
You don't need to implement this by yourself, many libraries already have it (e.g. sklearn's f1_score has a parameter called average, which can be set to "macro")


How do you create an optimized walk list given longitude and latitude coordinates? I am working on a political campaign where dozens of volunteers will be conducting door-knocking promotions over the next few weeks. Given a list with names, addresses and long/lat coordinates, what algorithms can be used to create an optimized walk list.

As Steve Kallestad has said, this is a TSP problem, and there are wonderful free solvers to find approximate solutions.

It may be too much work for what you are looking for, but you may try use one of those solvers in combination with the Google Maps API, to find real walking distances beetwen your coordinates:
https://developers.google.com/maps/documentation/directions/#DirectionsRequests

(I have never used this API, so I don't know how easy or effective it would be)


Doubt in Derivation of Backpropagation I was going through the derivation of backpropagation algorithm provided in this document (adding just for reference). I have doubt at one specific point in this derivation. The derivation goes as follows:
Notation:

The subscript $k$ denotes the output layer
The subscript $j$ denotes the hidden layer
The subscript $i$ denotes the input layer
$w_{kj}$ denotes a weight from the hidden to the output layer
$w_{ji}$ denotes a weight from the input to the hidden layer
$a$ denotes an activation value
$t$ denotes a target value
$net$ denotes the net input

The total error in a network is given by the following equation
$$E=\frac12 \sum_{k}(t_k-a_k)^2$$
We want to adjust the network’s weights to reduce this overall error:
$$\Delta W \propto -\frac{\partial E}{\partial W}$$
We will begin at the output layer with a particular weight.
$$\Delta w_{kj} \propto -\frac{\partial E}{\partial w_{kj}}$$
However error is not directly a function of a weight. We expand this as follows.
$$\Delta w_{kj} = -\epsilon \frac{\partial E}{\partial a_k} \frac{\partial a_k}{\partial net_k} \frac{\partial net_k}{\partial w_{kj}}$$
Now the document individually calculates above three partial derivatives to get following final equation for weight change rule for a hidden to output weight [Section 4.4 in the attached document]:
$$\Delta w_{kj} = \epsilon (t_k-a_k)a_k(1-a_k)a_j$$
or, $$\Delta w_{kj} = \epsilon \delta_k a_j$$
I followed till above equation.
Now, the document continues the derivation as follows (I am adding exact wordings from the document) [Section 4.5 in the document attached]

Weight change rule for an input to hidden weight
Now we have to determine the appropriate weight change for an input to hidden weight. This is more complicated because it depends on the error at all of the nodes this weighted connection can lead to.
$$\Delta w_{ji} \propto -[\sum_{k}\frac{\partial E}{\partial a_k} \frac{\partial a_k}{\partial net_k} \frac{\partial net_k}{\partial a_j}]\frac{\partial a_j}{\partial net_j}\frac{\partial net_j}{\partial w_{ji}}$$

I couldn't follow that why there is this extra $\sum_{k}$ in above equation as we have already considered sum over all $k$ when we wrote error $E=\frac12 \sum_{k}(t_k-a_k)^2$
Any lights?

For a while I thought you were right, but the equation is correct.

If you think about it conceptually,  then I think you will see that it has to be correct. When updating $w_{ij}$, you have to make the change that improves the total error $E$, and not just the error $E_k$ of output node $k$, right? So you have to be looking for the weights that give you the lowest $E$, that is, where $\partial E / \partial a_k$ is 0.

If you write out the sum over $k$ in the equation for $E$, you get one term for $a_1$, one for $a_2$, and so on. The sum over $k$ in the equation that confuses you, just makes sure to first get the $a_1$ term from $E$, then the $a_2$ term, and so in. It first differentiates $E$ to $a_1$, which is $t_1 + a_1$ if I did that correctly, and gets the correct tion for that. Then for $a_2$, until the end, and finally you have to sum over $k$ to add up the corrections from all terms.


Spliting keras model into multiple GPU's Dear fellow Data Scientists.
I'm having a problem with splitting model into multiple GPU's. I have read something about "towering" in native tensorflow but my whole architecture is already written in keras (tensorflow backend of course). Keras as far as I know only supports data paralellism which is useless while operating on images bigger than 1760x1760 in my case(Yolo architecture). I'm asking for advice, how could I achieve this without using native tensorflow I must run this model on 4500x4500 images and I can use up to 4 Tesla k40 (11GB) GPU's.

EDIT:
I'm already using batch = 1

One graph - multiple GPUs

In order to specify specific GPU for layers, this post can be helpful. Just specify for layer the GPU device.

Multiple copies of graph - multi_gpu_model()

There is a multi_gpu_model() function in Keras which will make your training be distributed on multiple GPUs on one machine.

But, as it is stated in the documentation, this approach copies the graph on multiple GPUs and splits the batches to those multiple GPUs and later fuses them.

When using this multi_gpu_model() be aware that you might encounter a problem with saving such model, but it can be easily solved by saving the original model as stated is this answer.


How to benefit Data augmentation when it yields to different classes I'm trying to classify rooftop sky images orientations, whether it is horizontal or vertical. Knowing that the most obvious feature here is known: orientation. I can simply augment each class by rotating it 90° so it belongs to the other class. It is like dynamic augmentation to different classes.

Using keras image classification capabilities, obviously defeats the purpose:

datagen = ImageDataGenerator(
        zoom_range=0.2, # randomly zoom into images
#       rotation_range=90,  # mistaken the model
        horizontal_flip=True,  # randomly flip images
        vertical_flip=False)  # randomly flip images

I am using keras for classification but open to other libraries, and techniques other than deep learning.

You could rotate images manually (without using ImageDataGenerator) and save it to disk.
That way you would know which images you have rotated - so you would know which images have changed the class.

After it, when using ImageDataGenerator, you need to set rotation_range to small value in order to be sure that it won't change the classes of images.

Examples how to rotate image manually can be found here: https://stackoverflow.com/a/43864937/5997950


is numpy isclose function returning bad answer? Using numpy isclose in the following example:

import numpy as np
np.isclose(1533761040,1533748023.0, atol=1)

Returns True, which is False.

No, it uses both an absolute and relative tolerance, and the default of rtol is nonzero, adding some more tolerance. See https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.isclose.html


Is ML a good solution for identifying what the user wants to do from a sentence? I am learning machine learning and I'm trying to implement a solution for a real problem: predict from a human sentence what programming function he/she is trying to do.

I have a series of programming functions related with a series of descriptions (there can be n > 0 descriptions for each unique function).

I created a neural network and a bag of words model trying to convert a human sentence "we get the data from the database" to a programming function. So far it works with very easy examples but not with my real data.

Something like this works:

"description"     | programming function
lala lolo lulu ka | function1
lala lolo lulu ke | function1
lala lolo lulu ko | function1
lala lele lili ka | function2
lala lele lili ki | function2
lala lele lili ko | function2

Every word in the description is converted to an neuron-input (with value of 1 if present and 0 if not present) and every possible function is converted to a neuron-output.

I'm using pyBrain with back propagation and an error threshold of 0.005. The neural network has three layers, and the middle one has lenght: number of possible words + number of possible programming functions (this is kind of arbitrary).

I know full text search or auto-complete is probably a better alternative for this task, but I'm just experimenting with machine learning and I'd like this to work if possible. In my real data I have 1000 descriptions related with ~500 functions.

So my question is:

Is bag of words + neural networks a good approach for solving this?
Maybe Word2vec is a better option?
If neither is good, is there any known machine learning approach that could work with something like that?

Yes this problem is extremely well suited for machine learning. However, I think you should be careful as to which algorithms you tend to use.

A machine learning algorithm should be structured as follows: feature extraction and then your model. These are two things that should be done separately.

Feature Extraction

This is the bag of words, n_grams and word2vec. These are all good choices for text examples. I think bag of words is a good choice in your case. However, if this generates a sparse matrix then maybe n_grams can be better. You can test all 3 methods.

The Model

Theoretically, the more parameters in your model the more data you need to train it sufficiently otherwise you will retain a large amount of bias. This means a high error rate. Neural networks tend to have a very high number of parameters. Thus they require a lot of data to be trained.

But, you have 1000 instances!!! Yes. But, you also have 500 classes. So imagine you have a very young child and you want him to be able to correctly classify 500 different types of images. Then you can't just show the kid 2 different examples of each class for him to truly understand what each class really means.

As a very general rule of thumb, the number of instances you need to train a model increases exponentially with number of classes. So you will need a MASSIVE amount of data to properly train a neural network model.

I would suggest a less intensive model. Moreover, looking at your example, it seems that the classes should be linearly separable. So you can use something really simple, linear regression, logistic regression, naive bayes or knn. These methods would do MUCH MUCH better than a neural network.

My Suggestion

I would start with bag of words and then use knn. This should be a good starting point.

A neural network is 0% recommended for the amount of data you have.


Pandas - Get feature values which appear in two distinct dataframes I have a Pandas DataFrame structured like this:

    user_id movie_id    rating
0   1       1193        5
1   2       1193        5
2   12      1193        4
3   15      1193        4
4   17      1193        5
5   18      1193        4
6   19      1193        5
7   24      1193        5
8   28      1193        3

Each row corresponds to a rating event performed by the user user_id for the movie movie_id. For instance, the first row says that user 1 rated the movie 1193 with a rating of 5.

This data comes from the MovieLens project.

My goal is to find all the users who satisfy these two conditions:

rated movie 588 with a rating of 5
rated movie 3578 with a rating of 3

I came up with two filtered DataFrame objects for each of the above conditions:

ratings_588_5 = data[(data.movie_id == 588) & (data.rating == 5]
ratings_3578_3 = data[(data.movie_id == 3578) & (data.rating == 3)]

Which result in, respectively:

>>> ratings_588_5
user_id movie_id    rating
438     588         5
758     588         5
913     588         5
1024    588         5
1214    588         5

>>> ratings_3578_3
user_id movie_id    rating
45      3578        3
321     3578        3
467     3578        3
758     3578        3
1024    3578        3
1381    3578        3

In Pandas, how can I compute a list of all user_id which appear in both DataFrames?

In this example, the result I want to obtain is:

[758, 1024]

you can use numpy.intersect1d() method:

In [277]: np.intersect1d(a.user_id, b.user_id).tolist()
Out[277]: [758, 1024]

or pd.core.common.intersection() method, but it seems to be slow (at least on my notebook for aa and bb DataFrames [see setup below...]):

In [307]: pd.core.common.intersection(a.user_id, b.user_id).tolist()
Out[307]: [1024, 758]

Timing for aa DF (50K rows) and bb DF (60K rows):

In [294]: aa = pd.concat([a] * 10**4, ignore_index=True)

In [295]: bb = pd.concat([b] * 10**4, ignore_index=True)

In [296]: aa.shape
Out[296]: (50000, 3)

In [297]: bb.shape
Out[297]: (60000, 3)

In [298]: %timeit aa.ix[aa.user_id.isin(bb.user_id),'user_id'].tolist()
10 loops, best of 3: 41.8 ms per loop

In [299]: %timeit np.intersect1d(aa.user_id, bb.user_id).tolist()
100 loops, best of 3: 5.36 ms per loop

In [300]: %timeit pd.merge(aa, bb, on='user_id').user_id.tolist()
...
skipped
...
MemoryError:

In [308]: %timeit pd.core.common.intersection(aa.user_id, bb.user_id).tolist()
10 loops, best of 3: 52.8 ms per loop

PS original answer


What are Hybrid Classifiers used in Sentiment Analysis? What are Hybrid classifiers used for sentiment analysis? How are they built? Please suggest good tutorial/book/link for reference. Also how are they different from other classifiers like SVM and Naive Bayes?

In sentiment analysis you may want to combine a number of classifiers. Let's say: a separate classifier for emoticons, another one for emotionally loaded terms, another one for some special linguistic patterns and - let's say - yet another one to detect and filter out spam messages. It's all up to you.

You can use either SVM, Naive Bayes, or anything else that best suits your problem. You may use majority voting, weights (for example based on cross validation results), or any other more advanced technique to decide which class is the most appropriate one.

Also, googling for hybrid sentiment returns tons of papers containing answers to the questions that you have stated. Please, don't ask us to rewrite this papers here.


Advantage Function - Variance Reduction When explaining advantage function, it is usually claimed that using a baseline reduces the variance. I have not found any specific reference to justify this.

Is this an application of control variates or something similar?

Could anyone provide some reference or formal justification for the variance reduction?

I assume that you are referring to policy gradient estimates. Adding any kind of function to your policy estimation,  which is dependent on the state of the environment, first of all, does not bias your gradient estimator (Proof Here).

The basic idea of subtracting a baseline from your action value function (and thus forming the advantage function) is that an unbiased estimator of your policy gradient is still unbiased if a constant is subtracted from that estimator. Then, that constant can be chosen appropriately in order to reduce the variance of the new estimator by optimization. If you have access, you can find a very good explanation in the book Statistical Reinforcement Learning: Modern Machine Learning Approaches in section 7.2.2. Also [2] and section 3 in [3].

As you mention it can be viewed as a control covariate addition [4] which is used to reduce variance in Monte Carlo estimations. A good choice for that function is to use the usual value function ($V(s)$) which reduces the variance of your estimate.

Hope it helps!


Identify given patterns in unstructured data like text files I wasn't sure if I had to ask it here or in Stackoverflow, but since I am also seeking research papers/algorithms and not only code, I decided to do it here.

When I have a text, I can manually write a regex to find all the possible outputs from what I want to extract from the file. What I want to do, is to find an algorithm or a research, which can let you highlight (set the input) different positions of the same (repeated) data you want to extract in the text file, train the algorithm and then identify all the others under the same contentions of those you set.

For example, let's say that I have a text with several titles which are following with \n\n\n and starting with \n\n. It is easy with regex, but I want to do it dynamically.

An idea is to build an algorithm which will take examples and create regex automatically. But I am not aware of any research like this and maybe there are also other techniques that you can achieve it.

Any ideas?

That is exactly what the Trifecta product does (in addition to other features). It uses the Wrangle language which is a DSL (domain specific language) designed for data manipulation. There is a much earlier research project called Wrangler from the same people. The Wrangler papers might give you ideas.


Macro- or micro-average for imbalanced class problems The question of whether to use macro- or micro-averages when the data is imbalanced comes up all the time.

Some googling shows that many bloggers tend to say that micro-average is the preferred way to go, e.g.:

Micro-average is preferable if there is a class imbalance problem.
On the other hand, micro-average can be a useful measure when your dataset varies in size.

A similar question in this forum suggests a similar answer.

However, this seems quite counter-intuitive. For example if we have a data set with 90%-10% class distribution then a baseline classifier can achieve 90% mico-averaged accuracy by assigning the majority class label.

This is corroborated by books, e.g. An Introduction to Information Retrieval says (page 282) "Microaveraged results are therefore really a measure of effectiveness on the large classes in a test collection.  To get a sense of effectiveness on small classes, you should compute macroaveraged results."

In the end the real decision about which measure to use should be based on the relative mis-classification costs for the classes. But a quick look at the internet seems to suggest use of micro-averaging. Is this correct or misleading?

The choice of a metric depends on how you rank the importance of your classes and what you value from a classifier. Let's look at your example:

  For example if we have a data set with 90%-10% class distribution then a baseline classifier can achieve 90% accuracy by assigning the majority class label.

One minor correction is that this way you can achieve a 90% micro-averaged accuracy. If your goal is for your classifier simply to maximize its hits and minimize its misses, this would be the way to go.

However, if you valued the minority class the most, you should switch to a macro-averaged accuracy, where you would only get a 50% score. This metric is insensitive to the imbalance of the classes and treats them all as equal.

In many applications the latter is preferable. Imagine a classification problem aiming at diagnosing a disease that appears in 1% of the population. What good is a classifier that would always predict that the patient was healthy, even if it could achieve a 99% micro-averaged accuracy on the task?

The reason why micro-averaging is prevalent is because in most tasks, we would be interested in simply maximizing the number of correct predictions the classifier makes. In these tasks no class is more important than the others.


What does it mean to "share parameters between features and classes" When reading this paper there is a line which says "linear  classifiers  do not  share  parameters  among  features  and  classes." What is the meaning of this statement? Does it mean that linear classifiers such as logistic regression need features that are mutually independent?

I will try to answer this question through logistic regression, one of the simplest linear classifiers.

The simplest case of logistic regression is if we have a binary classification task ($y \in\{0,1\})$and only one input feature ($x \in R$). In this case the output of logistic regression would be:

$$
\hat y = σ(w \cdot x + b)
$$
where $w$ and $b$ are both scalars. The output of the model $\hat y \in [0,1]$ corresponds to the probability that $x$ will be of class $1$.

We'll try to break down the phrase "linear classifiers do not share parameters among features and classes" into two parts. We will examine the cases of multiple features and multiple classes separately to see if logistic regression shares parameters for any those tasks:

Do linear classifiers share parameters among features?

In this case, for each example, $y$ is a scalar that takes binary values (like before), while $x$ is a vector of length $N$ (where $N$ is the number of features). Here, the the output is a linear combination of the input features (i.e. a weighted sum of these features plus the biases).

$$
\hat y = σ \left(\sum_i^N{(w_i \cdot x_i)} + b\right) \;\; or \;\; σ( \mathbf w \cdot \mathbf x + b)
$$
where $ \mathbf x $ and $ \mathbf w $ are vectors of length $N$. The product $\mathbf x \cdot \mathbf w$ produces a scalar. As you can see from above there is a separate weight $w_i$ for each input feature $x_i$ and these weights are independent by all means. From this we can conclude that there is no parameter sharing among features.

Do linear classifiers share parameters among classes?

In this case $x$ is a scalar, however $y$ is a vector of length $M$ (where $M$ is the number of classes). To tackle this, logistic regression essentially produces a separate output $y_j$ for each of the $M$ classes. Each output is a scalar $y_j \in [0,1]$ and corresponds to the probability of $x$ belonging to class $j$.

$$
\mathbf{ \hat y} = w \cdot \mathbf x + \mathbf b, \;\; where \;\; \mathbf{ \hat y} = {\hat y_1, \hat y_2, ..., y_M}
$$

The easiest way to think of this is as $M$ simple independent logistic regressions each with an output of:

$$
\hat y_j = σ(w_j \cdot x + b_j)
$$

From the above it is obvious that no weights are shared among the different classes.

multi-feature and multi-class:

By combining the two cases above we can finally reach the most general case of multiple features and multiple classes:

$$
\mathbf{ \hat y} = σ( \mathbf W \cdot \mathbf x + \mathbf b)
$$
where $\mathbf{ \hat y}$ is a vector with a size of $M$, $\mathbf x$ is a vector with a size of $N$, $\mathbf b$ is a vector with a size of $M$ and $W$ is a matrix with a size of $(N \times M)$.

In any case, linear classifiers do not share any parameters among features or classes.

To answer your second question, linear classifiers do have an underlying assumption that features need to be independent, however this is not what the author of the paper intended to say.


Python: advice on which machine learning algorithm to use for a problem which involves lots of randomness I'm new to machine learning so I'll summarize my problem with two examples without getting technical (because I can't).
The dog vs. cat classification example is solvable, in the sense that a human can tell you whether its a dog or a cat with certainty. Many machine learning algorithms are able to replicate the human performance and identify dog or cat with near certainty.
For my problem, there is no certainty, only a slight-better-than-random prediction. I am trying to predict whether a person who was recently released from incarceration will commit a crime within the next year. Lets assume the actual chances of re-offending are about 50/50. If I could use machine learning to make a modestly-better-than-random prediction, that would be a huge win for me. More specifically, if 50/50 is a random guess, then if I could achieve 55% to 60% success rate, that would be considered wildly successful.
I know this task is possible since I have used a dataset (with around 50 features and 100,000 observations) to make a "man-made" linear regression which achieves around 52% out of sample.
I have tried SKLearn's logistic regression and XGBoost but their performance has been lower than my "man made" attempt. I'm assuming that is because these algorithms aren't meant to deal with a prediction of an event that is mostly random.
Given that I am dealing with the prediction of an event which is mostly random and I am only looking to achieve slightly-better-than-random predictions, is there a machine learning algorithm / strategy you could recommend to best tackle this problem?

It sounds like you have good data - 50 columns and 100,00 rows!
I would do exploratory data analysis (EDA) and look for variables (columns) that are correlated with the response variable (re-offending) but NOT correlated with each other. If you can find a handful (~10) of these then you can build an excellent regression model.
Other techniques to try could include random forests and cluster analysis. Both can be done really quickly in Python so you can compare lots of different hyperparameter iterations.


Treating missing data in categorical features I have a dataset with one of the  categorical columns having a considerable number of missing values. The interesting thing about this column is that it has values only for a particular category in "another" column .
For eg :
column 1                        column2
========================================
Google                             -
Google                             -
Google                             -
Google                             -
Facebook                        Image
Facebook                        Video
Facebook                        Image

My column of interest has values only for one category (Facebook) that is present in another column. Therefore, the missing values for google cannot be imputed with average, cannot be predicted and those rows cannot be ignored either.
In such a situation, is it wise to consider the missing values '-' as a separate category in one-hot encoding? Or will this affect my machine learning model badly?

You could break the column 2 from your example into number of columns :
Image,Video....
So the new features will be like:
Column1  Image  Video
Google     0      0
Google     0      0
Facebook   1      0
Facebook   0      1


How to use Inception v3 in Tensorflow I am trying to import Inception v3 in TensorFlow. I wish to apply it after reading this tutorial on object detection.

Keras, now fully merged with the new TensorFlow 2.0, allows you to call a long list of pre-trained models. If you want to create an Inception V3, you do:

from tensorflow.keras.applications import InceptionV3

That InceptionV3 you just imported is not a model itself, it's a class. You now need to instantiate an InceptionV3 object, with:

my_model = InceptionV3()

at this point, my_model is a Keras Sequential() model with the architecture and trained weights of Inception V3, that you can re-train, freeze, save, and load as you need.

Check also the full list of models available in the module, it's great.


Can I train two stacked models end-to-end on different resolutions? Is it possible to stack two networks on top of each other that operate on different resolutions of input data?

So here's my usecase: like Google, I want to recognize text in images. Unlike Google, I have access to very limited computational resources. My solution for this is that instead of using a sequence-to-sequence network on the whole scene image, I first run object detection using YOLO and then pass the crop to a sequence-to-sequence model, Attention-OCR.

To further reduce processing time, I run the object detection on low resolution, and crop the detection result in the higher resolution input image, so I still have access to the high resolution input when using sequence recognition on my crop. To detect that there is text, the low resolution is adequate, but to read the text, the network needs the higher resolution input.

This all works fine, but I suspect I would get better performance if I could train the whole system end-to-end, so the crop gets directly optimized to result in the best reading of the text. I could stack the sequence-to-sequence model on top of the object detection model to do this, but then the text reading operates on the same low resolution input that is used for the text detection.

Does anyone have an idea how to solve this, or can anyone point me to research that is related to this?

I would suggest you to split your problem into two:

How to train?
How to make inference when resources are limited?

Very common pattern is to train model on large data (by renting AWS server, for instance) and then use its predictions to train much smaller network.
Here is fundamental paper on the subject from Hinton et al.
This way you can stick to more conventional object detection method without stacking involved and then deploy 'squeezed' network to your device.
But back to your question. If you are rigid in your approach for various reasons and can't do like I described above, then lots of end-to-end approaches are available in literature, take a look at this to get started.
And even code.
I will just cite the paper:

A unified deep neural network, denoted the multi-scale CNN (MS-CNN),
is proposed for fast multi-scale object detection. The MS- CNN
consists of a proposal sub-network and a detection sub-network.
In the proposal sub-network, detection is performed at multiple output
layers, so that receptive fields match objects of different scales.
These complementary scale-specific detectors are combined to produce a
strong multi-scale object detector. The unified network is learned
end-to-end, by optimizing a multi-task loss.
Feature upsampling by deconvolution is also explored, as an alternative to input upsampling, to reduce the memory and computation costs.
State-of-the-art object
detection performance, at up to 15 fps, is reported on datasets, such
as KITTI and Caltech, containing a substantial number of small
objects.

But nevertheless feel free to experiment with network compression (take a look at this). It typically has high degree of redundacy, and by being creative in exploring this redundacy it's possible to have many gains


What could be possible features of a textual word? I wish to construct feature vectors of words in a document and then calculate their linkage distance to detect anomalies. My question is how can I model these features ? If possible please give an example so I can understand better. Further how distance calculated will vary from the text that is anomalous and the one that is normal ? (Please refer to D. Guthrie, PhD. thesis for more clarification).

That is a rather broad question. If you are on a document level, you could (after removing stop words) determine the most important words in your corpus by calculating the tf/idf measure https://en.m.wikipedia.org/wiki/Tf–idf
Then you could define a vector over the top n words and represent every document as a vector of the relative frequencies of the top n words in that document and work on these vectors.
On a sentence level that could work as well, on a word level obviously not. Anyway, information retrieval and computational linguistics are a rather large research area. Maybe the best way would be to start with a book that provides a good overview.

Since you asked for some more specific recommendations:

If you can get access to (or actually want to buy it), definitely this one: Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition.

You can also take a shot this: Sentiment Analysis and Opinion Mining this is probably not your focus, but it covers feature representation to some extent and is for free.

More like a handbook with practical approaches is: Programming Collective Intelligence, look for it at Google, it is also available for free.

Otherwise if you want to look for yourself for books at a library or so, your keywords are "Computational Linguistics" and "Natural Language Processing".

Hope this helps


are OneHotEncoder and keras To_categorical same? The length of human_vocab is 18377.
The length of input X is 1000

I'm trying to run to_categorical

np.array(list(map(lambda x: to_categorical(x, num_classes=len(human_vocab)), X)))

Is this the same if i apply:

onehot_encoder = OneHotEncoder(sparse=False)
onehot_encoder.fit_transform(X)

the output of onehot_encode is (1000, 9739) shouldn't it be (1000,18377) ?

human_vocab contains the corpus data that X doesnt contain
I'm trying to find the replacement of to_categorical due to memory issues to create hot vector

To answer your question:

the output of onehot_encode is (1000, 9739) shouldn't it be (1000,18377) ?

The OneHotEncoder instance will create a dimension per unique word seen in the training sample. Here you are only showing it 9739 different words at training so it does not need more dimensions to perform one hot encoding.

One way to have it accommodate the entire vocabulary is

onehot_encoder = OneHotEncoder(sparse=False, categories=human_vocab)
onehot_encoder.fit_transform(X)


Classifying hand written digits dataset using Probabilistic neural network (PNN)? I am trying to apply Probabilistic Neural Network (PNN) on MNIST dataset using MATLAB on 28000 samples. I tried to build the model but it gave an error saying matrix dimension exceeds memory showing it will require 5.08 GB (28000x28000) matrix.
I want to know should I decrease the number of samples as input for training from 28000 to 200 samples as I read in research paper.
Should I reduce the sample size or is there something else wrong?
is pnn only applicable on small data sets?

Reducing the sample size is pretty much up to you.  PNN have the drawback that the size of the network after training is equal to the size of the training dataset.  Therefore it's a bit impractical to use for large datasets.  I also believe that there is evidence that suggest the performance may also decrease when the layers are big.

I don't have access to the paper but I imagine a big reason why they reduced the sample was to avoid the memory error that you're encountering right now.  I also suspect they carefully selected a good representation for the sample.


How to speedup message passing between computing nodes I'm developing a distributed application, and as it's been designed, there'll be a great load of communication during the processing. Since the communication is already as much spread along the entire process as possible, I'm wondering if there any standard solutions to improve the performance of the message passing layer of my application.

What changes/improvements could I apply to my code to reduce the time spent sending messages? For what it's worth, I'm communicating up to 10GB between 9 computing nodes, and the framework I'm using is implemented with OpenMPI.

Firstly, I would generally agree with everything that AirThomas suggested. Caching things is generally good if you can, but I find it slightly brittle since that's very dependent on exactly what your application is. Data compression is another very solid suggestion, but my impression on both of these is that the speedups you're looking at are going to be relatively marginal. Maybe as high as 2-5x, but I would be very surprised if they were any faster than that.

Under the assumption that pure I/O (writing to/reading from memory) is not your limiting factor (if it is, you're probably not going to get a lot faster), I would make a strong plug for zeromq. In the words of the creators:

  We took a normal TCP socket, injected it with a mix of radioactive
  isotopes stolen from a secret Soviet atomic research project,
  bombarded it with 1950-era cosmic rays, and put it into the hands of a
  drug-addled comic book author with a badly-disguised fetish for
  bulging muscles clad in spandex. Yes, ØMQ sockets are the world-saving
  superheroes of the networking world.

While that may be a little dramatic, zeromq sockets in my opinion are one of the most amazing pieces of software that the world of computer networks has put together in several years. I'm not sure what you're using for your message-passing layer right now, but if you're using something traditional like rabbitmq, you're liable to see speedups of multiple orders of magnitude (personally noticed about 500x, but depends a lot of architecture)

Check out some basic benchmarks here.


Terminology - cross-validation, testing and validation set for classification task Confusion1) If k=10 then does this mean that 90% is for training and 10% for testing? So always we have k% for testing?

Confusion2) In the following code I have used 10-fold cross-validation for training a Support Vector Machine (SVM). In general a data set will be split into (a) Training set, meas(trainIdx,:) (b) Testing set, meas(testIdx,:) c) Validation set.
In the cross-validation approach I am building the SVM learner by training and validating inside the loop. Based on my understanding, the validation data must be completely different from the training and testing. But, in many online resources it is said that after cross-validation, one must re-train on the entire data set which in this example would be the meas(:,1:end). If so, then the learned model svmModel inside the cross-validation is lost. Have I misunderstood completely wrong?

Can somebody please show what is the next step in the classification once the cross-validation is over?

Confusion 1)

From wikipedia :

  k-fold cross validation is a technique for assessing how the results
  of a statistical analysis will generalize to an independent data set

They also say :

  In k-fold cross-validation, the original sample is randomly partitioned into k equal sized subsamples. Of the k subsamples, a single subsample is retained as the validation data for testing the model, and the remaining k − 1 subsamples are used as training data. The cross-validation process is then repeated k times, with each of the k subsamples used exactly once as the validation data. The k results can then be averaged to produce a single estimation. The advantage of this method over repeated random sub-sampling (see below) is that all observations are used for both training and validation, and each observation is used for validation exactly once. 10-fold cross-validation is commonly used,[7] but in general k remains an unfixed parameter.

So you now see that you don't have k% for the testing but you always use 1/k % of the your dataset as test set. Note : you can choose to keep 2/k or more but it will be a lot more complicated to code.

Confusion 2)

In the scikit learn they above all refer to this tool as an "evaluating performance tool" counter to what wikipedia authors may suggest. The point is that CV allows you to assess how robust and reliable your prediction will be according to your initial dataset.

The final mean obtained at the end of the CV is a mean score on the k testing sets. It is often good to have a look at all the intermediary results to assess the variances of them that can be a good explanatory estimation in case of bad generalization capability of your model.

Edit : why running another training after CV

Cross validation can also be used as an optimization tool to find the best hyper-parameters of your model. In this case, you should take the better hyper-parameters (among the k different parameters; one for each fold) and use them to do prediction on the full set to see if the optimized (i.e. choosen from CV) hyper-parameters are good on the full dataset. The notion of "best" parameter can be seen as the hyper-parameters from the model who gave the best score in your CV process.

Note you can still put aside a validation set from the dataset, on which you won't do CV. This validation set can be used as the last test of your model prediction's quality. See also here

Finally, you can use each k-fold model to predict an estimation and then take the mean of them as the final prediction of your model as Wikipedia authors have suggested, but this idea is closer to ensemble learning or a kind ofBootstrap method without replacement, than to CV

hope it helps


Why convolution over volume sums up across channels? A simple question about convolution over volume .

Say we have an image with dimensions $(n, n, 3)$ and we apply a filter of dimension $(k, k, 3)$ this outputs an matrix of dimension $(n-k+1, n-k+1)$.

Why do we sum across channels in this case. Don't we lose information by mixing different channels. In case of images, this implies mixing information in R, G, B channels? For ex. when trying to detect traffic signal lights, such mixing can be fatal.

There are multiple points that I try to explain them.

First, each filter for convolutional networks for images is a 3d volume. Consequently, whenever it is said we have $n$ filters, means we have $n$ volumes of those 3d filters.

Second, you can consider each convolutional layer as an MLP which is applied to small regions of the input. These are applied to different regions of the input to investigate whether a typical pattern is in that region or not. These patterns are going to be learned by means of cost functions. You can easily consider that for each filter which is a volume, you are concretely doing a summation over weighted inputs, exactly as MLPs.

Third, as the result, your trained filters will decide to choose the information of which channels depending on the task using the cost function. They may be in a single plane or they may be among multiple of them.


Smart sentence segmentation not splitting on abbreviations Sentencer from SpaCy and NLTK does not catch the fact that typical abbreviations (e.g. Mio. for Million in German) and the resulting sentence split is not correct. I understand that sentencers are supposed to be simple and quick but I am wondering if there is a better one that takes into account something more than uppercased words and punctuation? Alternatively, how to make SpaCy / NLTK / ... sentencer work for such sentences?
I am interested primarily with sentencers with Python API.

Neural tools trained on Universal Dependencies corpora use learned models for tokenization and sentence-spliting. Two I know of are:

UDPipe – developed at Charles University in Prague. Gets very good results (at least for parsing), but has a little unintuitive API.

Stanza – developed at Stanford University. The API is quite similar to Spacy.

However, they are quite slow compared to regex-based sentence-spliting.