Feature scaling worsens performance? I noticed that feature scaling can destroy completely neural networks performance in some cases. Below are my results that you can reproduce easily.

I use a neural network to approximate function $g$ on $[0,100]$ as follows:

$\mathcal{N}\mathcal{N}(X; \theta) \approx g(X)$

where $X \sim \mathcal{U}([0,100])$. Then I randomly sample some $X$ ($(X_i)_i$) to be my training set, and use $Y_i=g(X_i)$ as targets.

Typically $\mathcal{N}\mathcal{N}$ is a Keras Sequential model, and has a few hidden layers with 15-20 neurons (elu), and a single output layer with no activation function. Parameters are initialized using kernel_initializer='normal'.

Now I noticed the following while testing my approximations in 1dim (before trying to generalize to higher dimensions):

For $g(x)=\max(0, X-50)$, scaling $X$ by dividing it by 100 does not really change the performance of the $\mathcal{N}\mathcal{N}$, which estimates $g$ relatively well.

However, when I use other functions such as $g(x)=1-\mathbb{I}_{x\in[40,60]}$ or $g(x)=(x-50)^2$, it fails completely.

Do you have an explanation?

This is very disturbing because how can you trust your network to approximate high-dimensional functions if it fails completely for simple 1-dimensional examples?

Below is an example implementation in Python:

from keras.models import Sequential
from keras.layers import Dense
import numpy as np
import matplotlib.pyplot as plt

d=1
nn_value_batch_size=128
nn_value_epochs=50
nn_value_training_size=10000

def nn_value_get_architecture(layers_sizes):
        nn_value = Sequential()
        nn_value.add(Dense(layers_sizes[0], input_dim=d, #d=1
                        kernel_initializer='normal', activation='elu'))

        # hidden layers
        for nb in layers_sizes[1:]:
            nn_value.add(Dense(nb, kernel_initializer='normal',
                               activation='elu'))

        # output layer
        nn_value.add(Dense(1, kernel_initializer='normal'))

        nn_value.compile(loss="mean_squared_error", optimizer='adam')
        return nn_value

def nn_value_optimize(nn_value, objective_function,
                          x_normalize=True):
        # fit the model
        x_distribution = lambda size: np.random.uniform(0, 100, size)
        for epoch in range(nn_value_epochs):
            X_train = x_distribution(nn_value_training_size)
            y_train = objective_function(X_train)
            if x_normalize==True:
                X_train = X_train/100
            nn_value.fit(X_train, y_train,
                         batch_size=nn_value_batch_size,
                         verbose=1)

        return nn_value

def g(X):
        #return 1-(np.maximum(0., X-40)/(X-40) - np.maximum(0., X-60)/(X-60))
        return (X-50)**2
        #return np.maximum(0., X-50) #only example that works

nn_value = nn_value_get_architecture(
                layers_sizes=[20,15,15,15,15])
g = np.vectorize(g)
normalize=True
nn_value = nn_value_optimize(nn_value, g, normalize)

# plot results
x = np.arange(0,101,1)
if normalize==True:
  plt.plot(x, nn_value.predict(x/100))
else:
  plt.plot(x, nn_value.predict(x))
plt.show()

No scaling:

With scaling:

By scaling X but not y, and having small weights at initialization, I think you are making it hard for the NN to bring the weights up to the scale they need to be.  Possibly if you let it train for more epochs it would find the right scale, but quick experimentation suggests it gets stuck into an oscillating loss that is quite large.
I had success in reaching a parabolic curve by either (1) scaling y as well, to land roughly in $[0,1]$, or (2) setting the initial weights to be larger, e.g. from keras import initializers; nn_initializer=initializers.RandomNormal(std=0.5) and using kernel_initializer=nn_initializer throughout.  (The default for std in RandomNormal is 0.05).
For the indicator function target, obviously scaling y isn't the issue.  But interestingly, scaling up the initial weights does work.  Maybe because you need to get those two large-slope bits at 40 and 60?
One other thing I'd suggest trying (I haven't here) is varying the learning rate, dependent on the scale of things.  Especially when the initial weights are far below what we know they'll need to end up as, maybe a large initial learning rate with some decay would help.


What is GridSearchCV doing after it finishes evaluating the performance of parameter combinations that takes so long? I'm running GridSearchCV to tune some parameters. For example:

params = {
    'max_depth':[18,21]
}

gscv = GridSearchCV(
    xgbc,
    params,
    scoring='roc_auc',
    verbose=50,
    cv=StratifiedKFold(n_splits=2, shuffle=True,random_state=42)
)

gscv.fit(df.drop('LAPSED', axis=1), df.LAPSED)
print('best score: ', gscv.best_score_, 'best params: ', gscv.best_params_)

All fine. Because I've specified some verbosity, it outputs some stuff about what it's doing, like this:

Fitting 2 folds for each of 2 candidates, totalling 4 fits
[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.
[CV] max_depth=18 ....................................................
[CV] ........... max_depth=18, score=0.9453140690301272, total= 8.2min
[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:  8.3min remaining:    0.0s
[CV] max_depth=18 ....................................................
[CV] ........... max_depth=18, score=0.9444119097669363, total= 7.9min
[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed: 16.3min remaining:    0.0s
[CV] max_depth=21 ....................................................
[CV] ........... max_depth=21, score=0.9454705777130412, total= 8.4min
[Parallel(n_jobs=1)]: Done   3 out of   3 | elapsed: 24.8min remaining:    0.0s
[CV] max_depth=21 ....................................................
[CV] ........... max_depth=21, score=0.9443863821843195, total= 8.3min
[Parallel(n_jobs=1)]: Done   4 out of   4 | elapsed: 33.2min remaining:    0.0s
[Parallel(n_jobs=1)]: Done   4 out of   4 | elapsed: 33.2min finished

However once it's finished running all the folds, it takes a very long time (at least as long as it takes to fit and evaluate one fold for one combination of parameters) for it to return the output to print('best score: ', gscv.best_score_, 'best params: ', gscv.best_params_), even though I could literally manually calculate that easily from the data it outputs as part of that fitting process. I presume this means that the algorithm is hung up doing something else after it finishes fitting and evaluating the different models, but I'm not sure what that might be.

n.b. actually, the thought just occurred to me that this might be time spent retraining the model on the parameters it identified as giving the best performance, so that it's available to the .predict() etc methods. I'm just checking that now by passing refit=False to prevent that from happening, and if it works I'll answer my own question.

Yep I figured it out. The answer is that by default GridSearchCV's last act is to expose the API of the estimator object you passed so that you can directly call things like .predict() or .score() on the GridSearchCV object itself. It does this by retraining the estimator against the best parameters it found during cross validation. If you want to skip this step (because, for example, you're going to go on to do more development or cross-validation afterwards) then you can pass refit=False to prevent that from happening.


Is converting a numerical target to binary helpful? I'm trying to solve an ML problem where the target variable is numeric, let's say the pollution level in a city. But the client is not interested in predicting the actual amount of pollutants, they are just interested in knowing whether the pollution level is high or low based on an agreed upon threshold. (High if the PM2.5 level is above 200, Low otherwise).

Should I treat it as a regression problem and take the numeric PM2.5 levels as target or as a classification problem where I make another feature of high/low pollution level based on the threshold and use that binary variable as a target? What are the advantages and disadvantages of both and What impact it can have on accuracy, if any?

When you convert a variable from numerical to binary, what happens is that you lose information about the magnitude of the variable and summarise it into a >=X variable.

It has advantages and disadvantages depending on the use you will give to your model:

If your model has to determine the exact amount of "how high" is your pollution, then is not a good idea to convert it because you are loosing crucial information, but if the city major declares a curfew if the pollution is bigger than x, then a binary output is not only good idea but necessary for the decision you are achieving with it.

When you convert the variable to binary, the hypothetical model you are achieving has different kind of information that the model with numerical variables:

You will not be comparing exact values (measuring RMSE), you will be measuring membership to a set (measuring AUC).
If your client wants to measure wheter you trespassed or not a threshold, you should explain to him what is achievable with that model and ask him it that meets the requirements of his job.


sklearn.accuracy_score(y_test, y_predict) vs np.mean(y_predict == y_test) What is the difference between these two methods for finding model accuracy?

I have used both methods in python3 and i normally get identical results. However in few cases i get completely different results, so I am trying to figure out the possible reason for this.

Both of these methods act differently. You will only get the same results in very few cases or if you are testing only one row at a time.

np.mean(y_test==y_pred) first checks if all the values in y_test is
equal to corresponding values in y_pred which either results in 0 or 1. And then takes the mean of it (which is still 0 or 1).
accuracy_score(y_test, y_pred) counts all the indexes where an element
of y_test equals to an element of y_pred and then divide it with the total number of elements in the list.

For example-

  import numpy as np
  from sklearn.metrics import accuracy_score
  y_test = [2,2,3]
  y_pred = [2,2,1]
  print(accuracy_score( y_test, y_pred))
  print(np.mean(y_test==y_pred))

This code returns -

  0.6666666666666666
  0.0

You will get the same result from both the method if you have only one sample/element to test. You can find more details here on accuracy_score and np.mean.

Also, accuracy_score is only for classification data. As mentioned in the first line here.


TF-IDF Features vs Embedding Layer Have you guys tried to compare the performance of TF-IDF features* with a shallow neural network classifier vs a deep neural network models like an RNN that has an embedding layer with word embedding as weights next to the input layer? I tried this on a couple of tweet datasets and got surprising results: f1 score of~65% for the TF-IDF vs ~45% for the RNN. I tried the setup embedding layer + shallow fully connected layer vs TF-IDF + fully connected layer but got almost same result difference. Can you guys give some opinion on how TF-IDF features can outperform the embedding layer of a deep NN? Is this case common? Thanks!

I've used unigrams and bigrams to produce the TF-IDF features

It is common for TFIDF to be a strong model. People constantly get high places in Kaggle competitions with TFIDF models. Here is a link to the winning solution that used TFIDF as one of its features (1st place Otto product classification). You will most likely get a stronger model if you combine the TFIDF and RNN into one ensemble. Other results from Kaggle:

2nd place: https://www.kaggle.com/c/stumbleupon/discussion/6184
4th place: https://www.kaggle.com/c/avito-demand-prediction/discussion/59881
3rd place: https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/52762

https://www.kaggle.com/c/avito-demand-prediction/discussion/56897:

  A good number of kernels are going the traditional route with
  CountVectorizer/TF-IDF, and some brave souls (I say brave because
  training is slower and the results don't seem as spectacular so far)
  have been experimenting with embeddings, as per the previous
  competitions.


Should there be a flat layer in between the conv layers and dense layer in YOLO? Should there be a flat layer in between the conv layers and dense layer in YOLO?

It's something not specified in the paper, but I see most implementations of YOLO on github do this. In my implementation, I do not flatten the 7*7*1024 feature map and directly add a Dense(4096) layer after it (I'm using keras with tensorflow backend).

Code in question:

x = Conv2D(filters=1024, kernel_size=(3,3), padding='same', activation=leaky_relu)(x)
# x = Flatten()(x) # not using this right now
x = Dense(4096, activation=leaky_relu)(x)
x = Dense(30)(x)

My output of model.summary is as follows:

conv2d_294 (Conv2D)          (None, 14, 14, 1024)      9438208
_________________________________________________________________
conv2d_295 (Conv2D)          (None, 7, 7, 1024)        9438208
_________________________________________________________________
conv2d_296 (Conv2D)          (None, 7, 7, 1024)        9438208
_________________________________________________________________
conv2d_297 (Conv2D)          (None, 7, 7, 1024)        9438208
_________________________________________________________________
dense_14 (Dense)             (None, 7, 7, 4096)        4198400
_________________________________________________________________
dense_15 (Dense)             (None, 7, 7, 30)          122910

I feel that this implementation is more correct than the flat one (unless otherwise specified by the author) because on flattening the output we lose the spatial information needed to predict bounding boxes and confidence scores for each element in the grid.

It doesn't matter, with or without flattening, a Dense layer takes the whole previous layer as input. The spatial structure information is not used anymore. Some Neural Network implementations might not be able to map a spatial structure directly into a dense layer, which is why you would need the Flatten in between. Mathematically it is exactly the same in this case.
EDIT: As I expected, in the source of Keras it mentions that if the rank is higher like with convolutions, they implicitly Flatten it first.


What are prototypes in RBF networks? I am currently reading Boosting the Performance of RBF Networks with Dynamic Decay Adjustment by Michael R. Berthold and Jay Diamond (online) to understand how Dynamic Decay Adjustment (DDA; a constructive trainining algorithm for RBF networks). Doing so, I stumbled over the word prototype a couple of times:

  Unfortunately PRCE networks do not adjust the standard deviation of their prototypes individually, using only one global value for this parameter.

  [...]

  This paper introduces the Dynamic Decay Adjustment (DDA) algorithm which utilizes the constructive nature of the PRCE algorithm together with independent adaptation of each prototype's decay factor.

  [...]

  PNNs are not suitable for large databases because they commit one new prototype for each training pattern they encounter, eeffectively becoming a referential memory scheme.

I've tried to find it in the only resource they referenced in this context (D.L. Reilly, L.N. Cooper, C. Elbaum: "A Neural Model for Category Learning"), but sadly I don't have access to that one.

I found an explanation on https://chrisjmccormick.wordpress.com/2013/08/15/radial-basis-function-network-rbfn-tutorial/:

  An RBFN performs classification by measuring the input’s similarity to examples from the training set. Each RBFN neuron stores a “prototype”, which is just one of the examples from the training set. When we want to classify a new input, each neuron computes the Euclidean distance between the input and its prototype. Roughly speaking, if the input more closely resembles the class A prototypes than the class B prototypes, it is classified as class A.

So a prototype is just the parameters (center and radius, assuming Gaussians are used) of an RBFs neuron?

Rephrasing the first quoted sentence, does it mean that the RBF networks usually only learn the center and the radius is fixed?

My question is if I understood it correct. Please add a reference (which is not a random blog article) which makes it more clear.

Kernel based methods work by constructing a feature vector based on some distance metric of an input vector to one or more example vectors, which are determined during training.These are the prototypes in your case (also known as regressors in some contexts). They are essentially vectors in the space of the independent variables.

Radial Basis functions essentially "convert" the (Euclidean) distance between input vector and prototype to a similarity value, usually  [0,1]. Some of them, such as the Gaussian, have parameters such as the bandwidth.

In RBF Networks, there are a number of different options of determining the prototypes, for instance by Clustering at the space of independent variables, or by Orthogonal Least Squares (references at the bottom).

Like you correctly assumed, the first sentence is explaining that in many cases of training algorithms (the ones mentioned above included), the bandwidth of the RBF is constant and determined a-priori What is subject to determining is the actual location of the prototypes. In this context, the prototype would refer to the value composition in the input space, but not to the radius (which is fixed anyway).

I have not read the paper in question, but it seems that the authors are proposing some method to individually adjust the bandwidths of the RBFs. In this case, the radius may have been included as well into the notion of the prototype, but it is specific to the research in question.

References

J. Moody and C. J. Darken, “Fast Learning in Networks of Locally-Tuned Processing Units,” Neural Comput., vol. 1, no. 2, pp. 281–294, Jun. 1989.

S. Chen, C. F. N. Cowan, and P. M. Grant, “Orthogonal least squares learning algorithm for radial basis function networks,” IEEE Trans. Neural Networks, vol. 2, no. 2, pp. 302–309, Mar. 1991.


Are there any Python libraries for predicting the closest value to a correct label out of a variable-size list of possible label values? Imagine I have the following dataset:

|---------------------|------------------|------------------|
|      Feature 1      |  Feature 2       | possible labels  |
|---------------------|------------------|------------------|
|          1          |         5        |    {1, 4}        |
|---------------------|------------------|------------------|
|          2          |         2        |     {2}          |
|---------------------|------------------|------------------|
|          4          |         3        |  {3, 5, 6}       |
|---------------------|------------------|------------------|

and my goal is to be able to predict a value that is the closest to the set of possible labels. That is, this is a regression problem.

(A sub-optimal solution I would also accept as an answer would be for my goal to be to predict a value that belongs to the set of possible labels, not taking into account the distance between the predicted value and the elements of the set.)

Does this have a name? Are there any Python libraries (for deep learning, if possible) that solve this?  Or should I define my custom loss function? If I have to define my custom loss function, could you suggest me a library/point me in the right direction? My main problem with that is that the set of possible labels has variable size! Thanks!

This could be treated as a multiclass & multilabel classification problem. If you are able to get your label lists in some good shape, you could check the Keras functional API for multitarget regression/classification models.

The main linitation is that you would need a list of outputs with equal length. I guess the best way is to do one-hot (aka dummy encoding) of your labels and to predict "classes" (labels). However, if the number of possible labels/classes is really large, you may get into trouble with this approach.

Using the Keras functional API is relatively straight forward. You simply need to define your outputs. Here is a minimal example. In order to do classification you would need to change a few things (loss function etc) but this should be no problem:

import numpy as np
import pandas as pd
from keras.datasets import boston_housing
(train_data, train_targets), (test_data, test_targets) =  boston_housing.load_data()

# Standardise data
mean = train_data.mean(axis=0)
train_data -= mean
std = train_data.std(axis=0)
train_data /= std
test_data -= mean
test_data /= std

# Add an additional target (just add some random noise to the original one)
import random
train_targets2 = train_targets + random.uniform(0, 0.1)
test_targets2   = test_targets + random.uniform(0, 0.1)

# https://keras.io/models/model/
from keras import models
from keras import layers
from keras.layers import Input, Dense
from keras.models import Model
from keras import regularizers
from keras.layers.normalization import BatchNormalization

# Input and model architecture
Input_1=Input(shape=(13, ))
x = Dense(1024, activation='relu', kernel_regularizer=regularizers.l2(0.05))(Input_1)
x = Dense(512, activation='relu', kernel_regularizer=regularizers.l2(0.05))(x)
x = Dense(256, activation='relu', kernel_regularizer=regularizers.l2(0.05))(x)
x = Dense(128, activation='relu', kernel_regularizer=regularizers.l2(0.05))(x)
x = Dense(8, activation='relu', kernel_regularizer=regularizers.l2(0.05))(x)

# Outputs
out1 = Dense(1)(x)
out2 = Dense(1)(x)

# Compile/fit the model
model = Model(inputs=Input_1, outputs=[out1,out2])
model.compile(optimizer = "rmsprop", loss = 'mse')
# Add actual data here in the fit statement
model.fit(train_data, [train_targets,train_targets2], epochs=500, batch_size=4, verbose=0, validation_split=0.8)

# Predict / check type and shape
preds = np.array(model.predict(test_data))
#print(type(preds), preds.shape)
# is a 3D numpy array

# get first part of prediction (column/row/3D layer)
preds0 = preds[0,:,0]
# second part
preds1 = preds[1,:,0]

# Check MAE
from sklearn.metrics import mean_absolute_error
print(mean_absolute_error(test_targets, preds0))
print(mean_absolute_error(test_targets2, preds1))


How to evaluate clustering success in a completely unsupervised system? The algorithm in question is Kohonen's SOM. But the question could also apply to PCA and some others.

When the umatrix (or the codebook?) is examined, is there a way to tell how successful clustering was?

And would it be a good idea to apply GA's to optimize size, lattice structure, learning rate, and the learning degradation functions as well as the epoch count for clustering, or is there a danger of overfitting in this instance?

Assume that the SOM data is coming from a demonstrably weak PRNG and that the very first attempt shows a distinct structure. Is there some statistical property or algorithm that can evaluate the presence and degree of structure to be used for a GA fitness function?

First, PCA is not a clustering method. It is a dimensionality reduction scheme.  You can assess the performance of PCA through analysis of the percent of variance in the dataset that is retained as you decrease the number of dimensions.  Retaining 99%,95%, or 90% is usually ideal, depending on your problem.

With regards to clustering, you probably want to start with the Silhouette Coefficient.  This combines assessments of both the cohesion (how tight a cluster is) and separation (how well separated each cluster is from other clusters).

Calculate a = average distance of i to the points in its cluster
Calculate b = min (average distance of i to points in another cluster)
The silhouette coefficient for a point is then given by:
s =1 – a/b, if a < b

It is typically between 0 and 1 with larger numbers being "better".  You can average the coefficients over a cluster or the entire region to get an assessment of the cluster or the entire clustering procedure for your data.

More generally, try googling "assessing clustering" or "cluster validity" to read about all of the other ways that you can score your clustering algorithm.  Here is a very complete treatment of the topic.

Hope this helps!


Construct word2vec (CBOW) training data from beginning of sentence When constructing training data for CBOW, Mikolov et al. suggest using the word from the center of a context window. What is the "best" approach to capturing words at the beginning/end of a sentence (I put best in quotes because I'm sure this depends on the task). Implementations I see online do something like the this:
for i in range(2, len(raw_text) - 2):
    context = [raw_text[i - 2], raw_text[i - 1],
               raw_text[i + 1], raw_text[i + 2]]

I see two issues arising from this approach.

Issue 1: The approach gives imbalanced focus to the middle of the sentence. For example, the first word of the sentence can only appear in 1 context window and will never appear as the target word. Compare this to the 4th word in the sentence which will appear in 4 context windows and will also be a target word. This will be an issue as some words appear frequently at the beginning of sentences (i.e. however, thus, etc.). Wouldn't this approach minimize their use?
Issue 2: Sentences with 4 or fewer words are completely ignored, and the importance of short sentences is minimized. For example, a sentence with 5 words can only contribute one training sample while a sentence of length 8 will contribute 4 training samples.

Can anyone offer insight as to how much these issues affect the results or any alternative approaches for constructing the training data? (I considered letting the first word be the target word and using the next N words as the context, but this creates issues of it's own).
Note: I also asked this question on Stack Overflow: https://stackoverflow.com/questions/63747999/construct-word2vec-cbow-training-data-from-beginning-of-sentence

Here is a great answer to this question. I'll summarize:

The code example was taken from a "buggy" repository on GitHub and is not typical of robust solutions.
Robust solutions actually do use the first word as a target word. If the context window is length 10, then the method uses the next 5 words as the context and the first word as the target (it won't actually have a context of size 10 since the first half of the context doesn't exist).
Even though the first few words in the sentence are used as target words, they still will not appear in as many contexts. This issue is mitigated because they appear in smaller contexts. Since the contexts they appear in are smaller, they have more impact on the context and, therefore, more significance in back propagation.
Many of the more robust implementations will use an entire paragraph or document as opposed to a sentence (some even include punctuation). This make sense because the ending of one sentence may give context for the beginning of another sentence. When this approach is implemented, there are far fewer start/ending words, which reduces the issue.

The answer linked above has some other helpful details and is worth reading.


Detecting spammers with artificially generated target class I have a bunch of data related to my company's incoming and outgoing calls, and I'm trying to detect spam callers in order to blacklist them. The features I've extracted from the data are a bit simple right now:

Caller Number: Who is calling
N_Unique: The amount of different phones called
N_Answered: The amount of times the call was connected
N_Missed: The amount of times the call was ignored/missed
Avg_Duration: The average duration answered calls lasted

Initially my superior sent me the dataset with a field called "IsSpammer", which was binary, and after a very simple Random Forest (1000 trees, default scikit-learn everything else) I got a 100% accuracy even after cross-validation. I took a closer look at the data and realized anyone who called more than 20 different numbers inside the company or had average call duration less than 20 seconds where flagged as spammers (which may be true), but then I asked supervisor how this field was generated, and he told me he set it under that rule the random forest discovered.

I then realized every ML problem I've faced came from a Kaggle contest that gave me a clear field to predict, and a training set that had "correct" information on the fields to predict. I don't know how to proceed with ML if I don't have the information of Spammer/NotSpammer, since apparently if I come up with a clever (or not) rule, then the best I can hope to discover is that rule I'm using in the first place... I'm not sure if getting more features and making and obscure rule using all of them would help. Obviously I lack experience in this, so I figured I would ask.

Is this a problem that could benefit from using machine learning?

Machine learning is the process of automatically discovering (inductively) the formula/rule that models a certain random variable. If you can discover the rule by yourself, you don't need the machine to aid you.

You (or your boss), as the domain expert, has written a rule to model the target class. As long as this rule is deemed good enough for the business, there's no need to spend resources to generate another model.


Which regularization in convolution layers (conv2D) I am using Keras for a project. I would like to know if it makes any sense to add any kind of regularization components such as kernel, bias or activity regularization in convolutional layers i.e Conv2D in Keras.

If yes, then which regularization is most useful for conv2d layers

Kernel
Bias
Activity

As explained here the regularization techniques are useful for the fully connected(dense) layers. Any such intuition/logic for conv2D?

Nowadays, people don't tend to add much regularisation like $L_2$ to convolutional networks. Usually, regularisation is achieved through the use of techniques like dropout and batch normalisation (although exactly how this regularisation actually happens is not well understood).

Having said that, it may be different for your particular problem. I recommend that after experimenting with dropout and batch norm, you should experiment with $L_2$ and see how the predictive performance is affected on a validation set.


How to split up my dataset in a train and testset, in order to prevent data leakage? I realize that this could be considered a duplicate of this question, Is using samples from the same person in both trainset and testset considers being a data leakage?, where it is stated that "The testing data should not be linked to the training data in any way" in order to prevent data leakage.
However, how should I continue if it is not possible to split the dataset, in a train and testset, where there is no link between them in any way?
My dataset exists of 839 time series samples which i want to classify according to 14 classes. However, this is an unbalanced dataset as can be seen in Figure 1.
The dataset is composed of the recordings of 16 volunteers. Unfortunately, for a number of these classes, data is only available for a number of volunteers. Figure 2 shows that for some classes only data of one specific volunteer is available. In the figure, the number gives the amount of volunteers of which samples are available for that specific class.
I want to validate my machine learning models using a seperate testset. My initial idea was to leave the data of one volunteer out for the testset. But as stated above, this is not possible since for two classes there is only data available from one volunteer.
The second best approach that I could come up with is leaving the data from one volunteer (which has samples available for 12 of the 4 classes) out alongside a selected few samples of the two minority classes that are not present in the samples of this specific volunteer.
This means that my testset would consist of samples which are completely seperated from the trainset for 12 of the 14 classes. The samples of the testset for the 2 other classes would be from the same volunteer on which the models are trained (since there is only data available of one volunteer for these classes).
As stated above, I know that this will introduce some form of data leakage. Is there a good alternative that won't introduce data leakage (besides adding more data)?
If not, would the StratifiedKFold method from sklearn be a good alternative for validating my model?
(With the StratifiedKFold i would use a pipeline in order to perform preprocessing only on the training part of the folds.)
I would be really glad for an answer and some more explanation on data leakage for this specific case.
Class imbalance
Data availability of volunteers per class

In multi-patient datasets a typical cross validation split is leave-subject-out, check this out: What is difference between leave one subject out and leave one out cross validation
Basically, each subject used as test subjects in different folds.
Your dataset has a few issues. Firstly, depending on your use case there might or not be data leakage. For example, if it is possible to firstly observe enough time series samples of a subject for training, say the first half, then it's ok to use the second half for testing.
Secondly, two of your classes appear on two single subjects. For these two, you can't design an algorithm on based on other subject's data.
Thirdly, and more importantly, most classes don't have enough samples. For example, the eat_soup class has only 3 samples.
In summary, you should use leave-subject-out but on the classes that appear on a substantial number of subjects and there is enough samples in total.


Rainbow vs A3C ...too unfair? In Deep Mind's Rainbow paper, how come A3C algorithm be so slow? twice slower than DDQN... Was it trained on a single actor? :D

It's on page 1 of the paper

Wasn't A3C supposed to be something a lot more powerful?

For example as splendid as this, taken from here:

Keep in mind that the DQN based method have replay memory that is sampled and used to optimize the model at every time step while A3C is only optimized when the episode finishes. So for 4000 timesteps, and 1000 timestep per episode, A3C optimized 4 times while DQN could have optimized 3000+ times depending on the implementation. This may explain why given the same number of timesteps, DQN is learning faster.


How to propagate error back to previous layer in CNN? I have a convolutional layer (link) with an input 5x5x2 (width, height, depth):

The layer has 3 filters with dimensions 3x3x2, it produces an output with dimensions 3x3x3.

I have completed the forward pass:

In a backward pass I calculated gradients of weights $dL/dw$, but I am not sure how to calculate $dL/dh$ (gradients for the previous layer).

First or all, is my calculation of $dL/dw$ valid ? (please refer to external spread sheet linked to the question) And most importantly, how do I propagate deltas to the previous layer ?

Since calculating the backward gradients that too in CNN is very math heavy and it's a bit different from normal MLP as in CNN'S we have Tensors not scalar values to multiply so People use Correlation to Calculate the Convolution Operation...

Just adding some links which I refer to from time to time thinking that someday I will be able to do this from scratch (seems impossible with deepening of Layers though...)

Reference 1
Reference 2
CS231N Slides (Lecture 4-6 specially, the computational graphs construction are very helpful to understand the Back-propagation Algorithm..)

It would be great if Someone can transfer the Equations into Latex here
So that the question can be answered formally..

Based on the above Links, here's what I got,(Came to know that Convolutions are equivalent to Correlation)

Image 1

Image 2

Stack doesn't allow me uploading image so using gdrive..(do rotate one of the images, the first one actually)

Hope it will give you an idea...


Use forecast weather data or actual weather data for prediction? I'm planning to use weather data to predict crime in Chicago.
I'm confused which to use as features: historical weather forecast data or historical actual weather data?
And what is the underlying logic here?
I'm assuming later only forecast data is available for prediction. Thank you for any input.

You could you both forecast and actual weather, then see which provide better features for forecasting? Obviously there will be a lot of correlation between the forecast and the actual weather, but you could use feature selection to choose the most important?


Methods to improve neural network to distinguish red from blue? I have a lot of data where im trying to distinguish red data from blue, but all the features look like this basically (see imgur).

They overlap a lot, and I can only get roc auc score of 0.65 .
So far in terms of data-processing ive only used class weights to balance it. I have tried different kinds of dense/dropout networks, with the different features but only 0.65 is the best I can get, most hover around 0.63-0.65, no matter how many layers or epochs I add.
For example a simple DNN/dropout yields 0.65:
"""
Dense(512, activation='relu'),
Dropout(0.3),
Dense(512, activation='relu'),
Dropout(0.3),
Dense(512, activation='relu'),
Dropout(0.3),
Dense(512, activation='relu'),
Dropout(0.3),
Dense(512, activation='relu'),
Dense(1, activation="sigmoid")
"""
Some resnet modification I found online also only provided 0.65.
So i have tried CNN and dense layer networks only, so far.
Any hints or ideas I could do to get better AUC scores?

I think the other comments about improved feature engineering and sharing more about your problem statement are correct.
I would just add that one helpful tip when thinking through these kinds of problems could be how you as an expert would classify these points by hand. If you would have trouble manually separating the two classes (which the distribution you have shown would lead us to believe), then almost any ML model will have the same trouble.
Without knowing more I'd say that this is potentially related to the concept of Bayes error rate (the amount of intrinsic error which cannot be further reduced with better modeling). If you flip a fair coin, sometimes you will get heads and sometimes you will get tails. There are no better features which will help you decide how to map that action (flipping the coin) to a value (heads/tails), there will always be some intrinsic error because the process itself is random.
In case you are interested: https://en.wikipedia.org/wiki/Bayes_error_rate


handling missing values for LightGBM model I have read that LightGBM handles missing values defaultly. And there certain parameters to change the consideration of missing values like zero_as_missing etc..,
I have seen some people using negative values (-1, -999) in the place missing values.
So, my question is, which is better 1) leaving it to model to handle or 2) manually replacing the values with negative values?
Thanks in advance

The default behavior allows the missing values to be sent down either branch of a split. Replacing with a negative value that is less than all your data forces the (originally) missing values to take the left branch, and so your model has (slightly) less capacity.  That may be a good or a bad thing, depending on where you land on the bias-variance curve.
So, I think the best answer here is "it depends on your data."  If your missing values actually behave like lesser values, then encoding them as large negative numbers enforces that, reducing capacity in a probably-beneficial way.  But without digging into exploratory data analysis, or complex imputation methods (MICE), I'd personally stick to the default behavior.


Is linear regression suitable for these data? I have a data set predicting a continuous variable, $Y$. I have $15$ to $20$ potential feature variables most of which are categorical, some of which are ordinal or categorical. These have been converted to numerical values. I have two questions.

Is linear regression suitable in this case?
If the variables do not show linear relationships with $Y$, is linear regression still suitable? Otherwise, which algorithms, hopefully existing in scikit-learn, might work?

You can employ the linear regression algorithm even for categorical data. The point is that whether your data is learnable or not. For instance, take a look at your data, and see whether an expert can really find the output by taking a look at the input vector. If it's possible, your task can be learnt using linear regression method.

About linearity, the point is that linear regression can also learn nonlinear mappings. You just have to provide enough higher order polynomials of the current feature space you have which is not an easy task. For instance, you can expand your current feature space by adding the square of each feature to the current feature space. You will observe that it may have better performance than the usual case if your mapping is not linear, but you may still have error. Consequently, you have to supply more polynomial features, but you do not know which to use.

An alternative to linear regression which does not need to add extra features is multi layer neural networks (MLP). You can simply use them which can learn nonlinear mappings. You can take a look at the official page of SKlearn for applying them. Furthermore, you can take a look at here for applying them.


Algorithm for Binary classification I have a data set with huge number of features ( Approximately 3000)  and a binary target variable . The reason I have too many features is because of one hot encoding  many categorical variables in my data set .
I think logistic regression might only work with small number of features .
So , given that I have many features , which algorithm should I use for better classification score ?
My aim is to increase the ROC-AUC metric for this classification task .
Is it better to use SVM or Neural networks ?

First thing that comes to my mind is to do different encodings. There are some ways to deal with high cardinality categorical data such as: Label Encoding or the famous target encoding. Before anything else I will recommend changing the encoding type.
But, since your question about which predictor use with small and space data. I will go still with logistic regression, decision tree or SVM. When data is small all algorithms tend to work quite similar.
Things like Random Forest might perform well since they do bootstrapping what tends to be a way to sample your data with replacement.


Classifier and Technique to use for large number of categories I am designing a scikit learn classifier for a sequence labelling task which has 5000+ categories and training data is at least 80 million and may grow upto an additional 100 million each year. I have already tried with all the categories but it generates classifiers in the order of 100s of GBs binary file. So I think that having one classifier for each category would be helpful and would also help me to fine tune features for each category thereby improving accuracy, but this means 5k+ classifiers for each of these categories. So how to handle this large data requirements and which incremental classifiers to use for this case , considering the fact that I will keep on getting additional training data as well as may discover new categories?

The number of features are about 100+ and due to the sequence labelling task contiguous sequence of training samples share same features values. The feature values are mostly text based and most are categorical with long text based values  with large cardinality i.e many features may have huge number of possible values.

The available RAM IS 32gb with 8 core CPU. On a small scale I tried Multinomial NB and linear SGD with sparse matrices which are extremely sparse. Used the scikit learns Dictvectorizer to vectorize the feature dictionary. Also will pandas dataframes help to optimize the overall configuration?

Due to the scale of data involved I have only used about 20% of the data. And the features itself are 56GB of data from the 500MB input data. For the restricted training data, The precision too is very low does not reach above 12% with a very low  recall too(1-2%). Have gone through vowpal wabbit and  found the SEARN task but it seems that it is now no longer available in the latest version .

This sounds like a pretty gnarly problem, so a lot of finesse will be needed to solve it.  @Jérémie Clos has some good points, but I wanted to add some more general thoughts...

With the size of your problem, you might want to think about a scalable framework like Mahout or H2O rather than scikit-learn, which is an awesome shared memory library, but does have scalability limitations.

With a categorical feature with cardinality of 5000 and 80 million training cases, a uniform distribution will only give you 16,000 training cases for each positive result.  However, the distribution probably isn't uniform.  I suggest that you eliminate the least common categories until you have thrown out 5% of the data and assess the cardinality of the remaining 95%.  This may give you a better clue about how to proceed i.e. what percent of the cardinality remains?

Most multiclass classification algorithms are just lots of binary classifiers that are combined to produce a composite result, so will essentially do the work for you of splitting up the problem, binarizing the data (aka one-hot encoding), and training the classifiers, so you may not need to explicitly perform this splitting step.  Two exceptions to this, i.e. classifiers that can handle the multi-class problem intrinsically are decision trees and random forests (Mahout version). I would try both of these and hold them as benchmarks moving forward.

In terms of handling the addition of categories as data is added to the system, you will have to retrain sometimes.  This is sometimes referred to as the stability-plasticity dilemma.  The best way to handle this is probably only to make predictions on categories that comprise some threshold of the data and throw out the rest as outlier categories that are not powered well enough to predict on e.g. similar to the sanity check I suggest above.

Hope this helps!


Why ML model produces different results despite random_state defined? And how to set global random seed for sklearn I have been running few ML models on same set of data for a binary classification problem with class proportion of 33:67.
I had the same algorithms and same set of hyperparamters during yesterday and today's run.
Please note that I also have the parameter random_state in each estimator function as shown below

np.random.seed(42)
svm=SVC()  # i replace the estimator here for diff algos
svm_cv=GridSearchCV(svm,op_param_grid,cv=10,scoring='f1')
svm_cv.fit(X_train_std,y_train)

q1) Why does this change happens even when I have random_state configured?
q2) Is there anything else that I should do to reproduce the same results every time I run?
Please find below the results that are different? Here auc-Y denotes yesterday's run

Not every seed is the same.
Here is a definitive function that sets ALL of your seeds and you can expect complete reproducibility:

def seed_everything(seed=42):
    """"
    Seed everything.
    """
    random.seed(seed)
    os.environ['PYTHONHASHSEED'] = str(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.deterministic = True

You have to import torch, numpy etc.
UPDATE: How to set global randomseed for sklearn models:
Given that sklearn does not have its own global random seed but uses the numpy random seed we can set it globally with the above :
np.random.seed(seed)

Here is a little experiment for scipy library, analogous would be sklearn (generating random numbers-usually weights):
import numpy as np
from scipy.stats import norm
print('Without seed')
print(norm.rvs(100, size = 5))
print(norm.rvs(100, size = 5))

print('With the same seed')
np.random.seed(42)
print(norm.rvs(100, size = 5))
np.random.seed(42) # reset the random seed back to 42
print(norm.rvs(100, size = 5))

print('Without seed')
np.random.seed(None)
print(norm.rvs(100, size = 5))
print(norm.rvs(100, size = 5))

outputing and confirming

Without seed
[100.27042599 100.9258397  100.20903163  99.88255017  99.29165699]
[100.53127275 100.17750482  98.38604284 100.74109598 101.54287085]
With the same seed
**[101.36242188 101.13410818 102.36307449  99.74043318  98.83044407]**
**[101.36242188 101.13410818 102.36307449  99.74043318  98.83044407]**
Without seed
[101.2933838  100.52176902 101.38602156 100.72865231  99.02271004]
[100.19080241  99.11010957  99.51578106 101.56403284 100.37350788]


Which are the worse machine learning models for text classifications? I was looking at text classification, and for curiosity I was searching online for which were the best models for text classifications. About this, I found that they are linear support vector machines and naive bayes.
But which are the worse models to use in text classification? And, if possible, why?

First, the question is too broad because there are many different kinds of text classification tasks. For example one wouldn't use the same approach for say spam detection and author profiling (e.g. predict the gender of the author), two tasks which are technically text classification but have little in common (and there are many others).
Second, even with a more specific kind of problem, the question of the type of model is misleading because a lot of what makes a ML system perform better than another in text classification is due to other things: the type and amount of training data of course, but also crucially the features being used. There are many options in terms of representing text as features, and these different options usually have a massive impact on performance. I even think that most of the time the choice of a type of classification model does not matter as much as the design of the features.
Finally I'm actually going to answer the question but probably not in the way OP expects: the worst model in any classification task is exactly like the best model, but it swaps the answers in order to have as many wrong predictions as possible (e.g. class 1 -> class 2, class 2 -> class 3, .., class N -> class 1).
Since it's a lot of work to implement the best classifier just to obtain the worst one, a close to worst one can be done with a minority baseline classifier: just predict every instance as the least frequent class in the training data.
I hope a few of the things I said will be helpful, even though it's probably not what OP wished for! :)


What is the appropriate name for this dataset? I'll assume most of us know about cross-validation and are accustomed to using Training and Test sets. However, I've also seen references to a third set that you use to test a completed model. It's the test to see how your model does against a dataset that it hasn't seen before; helps determine if your model has been overly-fit to the training data.

I have seen this dataset referred to everything from "the third dataset", "the wild dataset", "the real-world dataset". So what is the appropriate name for this third dataset?

Well, I would not call this "Test" set you defined as "Test"! I would say: Train, Validation, Test. And this "Test" is the dataset that your model has not seen yet. This is pretty standard even explained in details in Andrew Ng courses like this very useful and important lesson on splitting data and the significance of their distributions.


Is it necessary to normalise data for XGBoost? MinMaxScaler in scikit_learn is used for data normalization (a.k.a feature scaling). Data normalisation is not necessary for decision trees. Since XGBoost is based on decision trees, is it necessary to do data normalisation using MinMaxScaler for data to be fed to XGBoost machine learning models?

Your rationale is indeed correct: decision trees do not require normalization of their inputs; and since XGBoost is essentially an ensemble algorithm comprised of decision trees, it does not require normalization for the inputs either.
For corroboration, see also the thread Is Normalization necessary? at the XGBoost Github repo, where the answer by the lead XGBoost developer is a clear:

no you do not have to normalize the features


Implementing a neural network: low accuracy I'm training my neural network implementation on the MONK's problems data set. I read that it can be possible to achieve almost a 0 loss with the hyper-parameters I'm using (chapter 9). These are my results on training set after 90 epochs, they aren't good as you can see:

Training error after 0 iterations: 0.26427015361
Training error after 20 iterations: 0.231331617693
Training error after 40 iterations: 0.216396390383
Training error after 60 iterations: 0.198107908553
Training error after 80 iterations: 0.166181613583
TrErr: 0.155389641876

And here is the code of my neural network. I think the code is quite simple and readable for a expert, tell me if you need any clarification.

# paramaters
etas = [0.1]
hidden_dim = 2
alpha = 0.5
reg_lambda = 0.0

input_dim = 17
output_dim = 1

# read and parse MONK data
def parse_data(text):
    with open(text) as f:
        content = f.readlines()
    t = np.zeros((len(content), output_dim))
    X = np.zeros((len(content), input_dim))
    for i,val in enumerate(content):
        s = val.split(" ")
        t[i] = float(s[1])
        for idx in xrange(17):
            X[i][idx] = 0.0
        j = -1
        # |0 |1 |2 |3 |4 |5 |6 |7 |8 |9 |10|11|12|13|14|15|16
        # |a1|a1|a1|a2|a2|a2|a3|a3|a4|a4|a4|a5|a5|a5|a5|a6|a6
        X[i][int(s[2])+j] = 1.0 # a1
        j = 2
        X[i][int(s[3])+j] = 1.0 # a2
        j = 5
        X[i][int(s[4])+j] = 1.0 # a3
        j = 7
        X[i][int(s[5])+j] = 1.0 # a4
        j = 10
        X[i][int(s[6])+j] = 1.0 # a5
        j = 14
        X[idx][int(s[7])+j] = 1.0 # a6

    return (X, t)

def sigmoid(x):
    output = 1/(1+np.exp(-x))
    return output

def sigmoid_output_to_derivative(output):
    return output*(1-output)

# initialize weights
W0 = np.random.uniform(-0.7,0.7,size=(input_dim,hidden_dim))
W1 = np.random.uniform(-0.7,0.7,size=(hidden_dim,output_dim))
b0 = np.zeros((1, hidden_dim))
b1 = np.zeros((1, output_dim))
dW0 = 0
dW1 = 0

for j in xrange(90):
    # forward propagation
    l0 = X
    l1 = sigmoid(np.dot(l0,W0) + b0)
    l2 = sigmoid(np.dot(l1,W1) + b1)

    # back propagation
    l2_error = t - l2
    l2_delta = l2_error*sigmoid_output_to_derivative(l2)

    l1_error = l2_delta.dot(W1.T)
    l1_delta = l1_error*sigmoid_output_to_derivative(l1)

    if (j% 20) == 0:
        print "Training error after", j, "iterations:",np.mean(np.square(l2_error))

    dW1 = eta*(l1.T.dot(l2_delta) + reg_lambda*W1) + alpha*dW1
    dW0 = eta*(l0.T.dot(l1_delta) + reg_lambda*W0) + alpha*dW0
    db1 = eta*(np.sum(l2_delta, axis=0))
    db0 = eta*(np.sum(l1_delta, axis=0))

    W1 += dW1
    W0 += dW0
    b1 += db1
    b0 += db0

It seems like you just got a bug @ your parse_data. (see my comment: replace X[idx][int(s[7])+j] = 1.0 # a6 to X[i][int(s[7])+j] = 1.0 # a6).

I altered the parameters. You will see that they have a big influence. I guess this problem is easily stuck on a local minima.

Could you run my code please (Only difference is that it is python 3, but should run on python 2)

This is the link of the data saved in data1.txt:
http://mlearn.ics.uci.edu/databases/monks-problems/monks-2.test

import numpy as np
    import os
    import matplotlib.pyplot as plt

abspath = os.path.abspath(__file__)
dname = os.path.dirname(abspath)
os.chdir(dname)

# paramaters
eta = 0.05
hidden_dim = 2
alpha = 0.8
reg_lambda = 0.0
epochs = 100

file = 'data1.txt'

input_dim = 17
output_dim = 1

# read and parse MONK data
def parse_data(text):
    with open(text) as f:
        content = f.readlines()
    t = np.zeros((len(content), output_dim))
    X = np.zeros((len(content), input_dim))
    for i,val in enumerate(content):
        s = val.split(" ")
        t[i] = float(s[1])
        # for idx in range(17):
        #   X[i][idx] = 0.0
        j = -1
        # |0 |1 |2 |3 |4 |5 |6 |7 |8 |9 |10|11|12|13|14|15|16
        # |a1|a1|a1|a2|a2|a2|a3|a3|a4|a4|a4|a5|a5|a5|a5|a6|a6
        X[i][int(s[2])+j] = 1.0 # a1
        j = 2
        X[i][int(s[3])+j] = 1.0 # a2
        j = 5
        X[i][int(s[4])+j] = 1.0 # a3
        j = 7
        X[i][int(s[5])+j] = 1.0 # a4
        j = 10
        X[i][int(s[6])+j] = 1.0 # a5
        j = 14
        X[i][int(s[7])+j] = 1.0 # a6

    return (X, t)

X, t = parse_data(file)

print(np.sum(t)/len(t))

def sigmoid(x):
    output = 1/(1+np.exp(-x))
    return output

def sigmoid_output_to_derivative(output):
    return output*(1-output)

# initialize weights
seed = 1
np.random.seed(seed)
W0 = np.random.normal(0, 1/input_dim, size = (input_dim,hidden_dim))
W1 = np.random.normal(0, 1/hidden_dim, size = (hidden_dim,output_dim))
b0 = np.zeros((1, hidden_dim))
b1 = np.zeros((1, output_dim))
dW0 = 0
dW1 = 0

error_list = []
for j in range(epochs):
    # forward propagation
    l0 = X
    l1 = sigmoid(np.dot(l0,W0) + b0)
    l2 = sigmoid(np.dot(l1,W1) + b1)

    # back propagation
    l2_error = l2 - t
    l2_delta = l2_error*sigmoid_output_to_derivative(l2)

    l1_error = l2_delta.dot(W1.T)
    l1_delta = l1_error*sigmoid_output_to_derivative(l1)

    error = np.mean(np.square(l2_error))
    error_list.append(error)
    if (j% 20) == 0:
        print("Training error after", j, "iterations:",error)

    dW1 = -eta * (l1.T.dot(l2_delta) + reg_lambda * W1) + alpha * dW1
    dW0 = -eta * (l0.T.dot(l1_delta) + reg_lambda * W0) + alpha * dW0
    db1 = -eta*(np.sum(l2_delta, axis=0))
    db0 = -eta*(np.sum(l1_delta, axis=0))

    W1 += dW1
    W0 += dW0
    b1 += db1
    b0 += db0

plt.plot(error_list)
plt.show()


How to treat outliers in a time series dataset? I've read the following article about how to treat outliers in a dataset: http://napitupulu-jon.appspot.com/posts/outliers-ud120.html

Basically, he removes all the y which has a huge difference with the majority:

def outlierCleaner(predictions, ages, net_worths):
    """
        clean away the 10% of points that have the largest
        residual errors (different between the prediction
        and the actual net worth)

        return a list of tuples named cleaned_data where
        each tuple is of the form (age, net_worth, error)
    """

    #calculate the error,make it descend sort, and fetch 90% of the data

    errors = (net_worths-predictions)**2
    cleaned_data =zip(ages,net_worths,errors)
    cleaned_data = sorted(cleaned_data,key=lambda x:x[2][0], reverse=True)
    limit = int(len(net_worths)*0.1)

    return cleaned_data[limit:]

But how may I apply this technique to a time series dataset if its rows are correlative?

Decide how auto-correlative your usual event in the time series is.
For example, "I'm tracking temperature over time and it rarely changes more
than 30 degrees F in an hour".
Throw out or smooth any values where the observed value changes
more than that. In other words, "If ever I see the temperature
changing more than 30 degrees in an hour, I'm going to ignore that value and substitute the average of the prior and the next value because that must be a sensor malfunction".

Once you were comfortable with doing that, use something like the standard deviation of the data over a rolling window instead of an absolute, arbitrary value like I did.


How to interpret a random variable in the variable importance? I have a problem, for simplicity let's say it is a binary classification problem.

I am trying to solve this problem using XGBoost.

A standard output plot for any ML algorithm, is the feature importance, and I want to look at the top n features.

But how to decide on n ?

A method would be to add a random variable (a vector with numbers from uniform distribution) in the XGBoost model, and then see in which place this random variable lands, and only look at the variables above the random variable.

If I do that, and then I re-run the model, including only the variables that score above the random one, then the accuracy score in the test set, decreases. Why ?

I'll go through your questions one by one:

  ... how to decide on n ?

This is a completely arbitrary choice that depends on your subjective preferences, the nature of your data, your problem at hand, the amount of computational power you can afford, you name it... There is no rule-of-thumb on this. What is the purpose of your analysis?

--

  If I do that, and then I re-run the model, including only the
  variables that score above the random one, then the accuracy score in
  the test set, decreases. Why ?

The variables that have an explanatory power below a given threshold are comparatively less relevant but not useless, they can capture some of the variance of your dependent variable. Removing them means renouncing to a little bit of predictive power. That's why accuracy decreases.

Hope this helps, otherwise let me know.


Batch convert json links to csv I have a batch of links with the following naming convention:

https://api.cilabs.net/v1/conferences/ws15/info/attendees?page=1; and the page number incrementing.

I would like to extract the data from those web pages into a CSV file.

I guess what you are looking for is a crawler, which can crawl through the web pages and scrape the data for you into a csv file.

I really don't like the idea of giving away the full code. So, I would tell you how it's done. I would use Python for the example.

Define the url of the web page. Leave the last part (page=), as it can go into a loop, so that as it needs to be incremented after every page gets scrapped.

Now, identify the xpaths of the data which you want to scrape. Store each of them in an array, and keep writing(and appending into a csv)

This is the code of one such scrapper which I have written for a project of mine. Refer to it in case of a road-block. I have used the lxml library of Python.
You can use other fancy libraries like Scrapy, BeautifulSoup. But, they are just fancy wrappers over lxml, so it's faster.


increase performances in neural networks I am starting being interested in neural networks, and I am writing some code about it. But, differently from methods like support vector machines, random forests,..etc., to me it seems more like a black box which I can't control.

So my question is:

What are methods to increase accuracy and other performances in a neural network?

Thanks in advance.

basically you should try to use regularization/dropout/batch-normalization and find out the optimal hyper-parameters to improve performance of your model.

After applying these optimizations you can use Randomized Search to find optimal hyper-parameters


Using class weights in Keras with multiple binary outputs which are not simply one-hot-encoded My labels are binary vectors of length 5, e.g., [0, 0, 1, 1, 1].

My label set is very biased, 1-to-50, where the case [0, 0, 0, 0, 0] is very common while all other combinations are not. I'd like to weight the uncommon versions using the class_weight argument in the model.fit() function in Keras. I get an error message which says class_weight must contain all classes in the data.

  The classes {0, 1, 2, 3, 4} exist in the data but not in class_weight.

Based on this I suspect Keras expects that I only have classes of the form [1, 0, 0, 0, 0], [0, 1, 0, 0, 0], [0, 0, 1, 0, 0], etc., i.e., only 1 in each entry in the vector. And that I'm supposed to supply the weights as follows:

weight_classes = {
    0: 50.0,        # for [1, 0, 0, 0, 0]
    1: 1.0,         # for [0, 1, 0, 0, 0]
    # etc.
}

Is there a way to tell Keras to give weights in this fashion:

weight_classes = {
    (0, 0, 0, 0, 0): 1.0,
    (0, 0, 1, 1, 1): 50.0,
    (0, 0, 0, 0, 1): 50.0,
    # etc.
}

Or is there some other way to just say that the all zero case is less important?

I've consider just transform the label set to conform to the one-hot-encoding type of representation. But I'd rather not, because the labels will become very large, and there are interdependence between the labels I think might be better represented by having them encoded as I've done. They are interdependent similar to how words in a sentence are dependent, but where individual predictions are valuable independently.

I have also consider just sticking with sub/supersampling, and skip this whole class weighting.

The solution to this question is to use sample_weight in the model.fit() function, (and as the third tuple entry in validation_data if you're using it). For each label sample you simply classify if it is rare or common using some algorithm, and set the weight accordingly.

For me this could be done with the following:

sample_weight = np.max(label_data, axis=1)) * 49 + 1

Here I took advantage of the fact that the common case is all zero per label row, e.g.:

     label_rows                             weights   label type
max([0, 0, 0, 0, 0])*49+1 = 0*49+1 =  0+1 =  1      # common
max([0, 0, 1, 1, 1])*49+1 = 1*49+1 = 49+1 = 50      # rare
max([0, 0, 0, 0, 0])*49+1 = 0*49+1 =  0+1 =  1      # common
max([0, 0, 0, 0, 1])*49+1 = 1*49+1 = 49+1 = 50      # rare
max([0, 0, 0, 1, 1])*49+1 = 1*49+1 = 49+1 = 50      # rare
...

That said, I ultimately recoded my labels so that there were only one positive entry in each label row, making it possible to set class weights. Not because weighting of the samples, but because it gives a better interpretation as a probabilistic output (using softmax). The following function does the weighting:

def create_class_weights(label_dim, rare_weight=10.0, common_weight=1.0):
    class_weight = {
        idx: rare_weight for idx in range(label_dim)
    }
    class_weight[label_dim-1] = common_weight
    return class_weight

It assumes the common case is the last per label row.


Square Root Regularization and High Loss I am testing out square root regularization (explained ahead) in a pytorch implementation of a neural network. Square root regularization, henceforth l1/2, is just like l2 regularization, but instead of squaring the weights, I take the square root of their absolute value. To implement it I penalize the loss as such in pytorch:

for p in model.parameters():
  loss += lambda * torch.sqrt(p.abs()).sum()

p.abs() is the absolute value of p, i.e the weights, torch.sqrt() is the square root and .sum() is the sum of the result for the individual weights. lambda is the penalization factor.
With no regularization, the loss settles around 0.4. With lambda=100, the loss settles somewhere around 0.45, if I use l2 or l1 regularization. Interestingly enough with lambda=0.001, the final value for the loss is around 0.44. Now if I use l1/2 with the same lambda it settles around 5000! This just does not make sense to me. If the regularization is such an overwhelming factor in the loss, then SGD has got to bring the (absolute value) of the weights down until, the penalty from the regularization is balanced with the actual classification from the Cross Entropy Loss I'm using - I know this is not the case, because the train and validation accuracies, are around the same of the original network (without regularization) at the end of training. I also know that the regularization is indeed happening in all three cases, as the loss in the early epochs differ significantly).
Is this to be expected or is this some error in my code or pytorch's SGD?

One more note If use l1/2 with a small lambda like 0.001, the loss comes down to around 0.5 and then becomes nan around epoch 70. For lambda=0.01, it becomes ~1.0 and then nan around the same epoch. For lambda=0.1, loss becomes 5 but no nan anymore for this lambda or any value higher (in 120 epochs total). For lambda=1.0, loss settles at ~50- as expected: apparently the weights settle down at a point where the sum of their square roots equals ~ 50, regardless of lambda...

The problem with using $L_p$ norm with $p < 1$ for regularization is the gradient.

Regularization term is used to force the parameters to be closer to zero. For this to work, when a parameter goes closer to zero, the gradient of regularization term, i.e. its contribution in updating the parameter, should decrease as well or at least remain constant.

However, by going closer to zero, the gradient increases unboundedly for $p < 1$, which causes radical changes in parameters, leading to your observations. Here is a visual plot for root $r$, absolute $a$, and squared $s$ regularization terms in one dimension (drawn here):

As you can see, for root regularization, the gradient explodes to infinity as we get closer to zero which is in conflict with our intention to bring the parameters close to zero.

For the sake of completeness, here is what a 1D parameter update with root regularization looks like:
$$\theta_{n+1} = \theta_n + \alpha \frac{d L(\theta)}{d \theta}\Bigr|_{\theta=\theta_n} = \theta_n + \alpha \left(\frac{d l(\theta)}{d \theta}\Bigr|_{\theta=\theta_n} \pm \lambda \frac{1}{\color{red}{\sqrt{|\theta_n|}}}\right)$$
where $L(\theta) = l(\theta) + \lambda \sqrt{|\theta|}$.

You can spot the trouble! The more we get close to zero, the less we get close to zero!


SVM SVC: Metric for parameter optimization on imbalanced data I trained a multiclass SVC with RBF kernel on a down-sampled (and therefore balanced) dataset.
Now I want to perform grid search to find best cost and gamma.

What performance metric should I optimize for?

I have a highly imbalanced test set. There might be a factor of over 100 between the number of instances of different classes.
I am classifying 3D points (car, facade, human) - so I think one could assign equal weight to all classes.

Using resampling methods to fix the imbalanced databases a good method. This can be completed by oversampling, downsampling or creating synthetic instances.

In this kind of problem, precision is not necessarily a good evaluation metric. Instead evaluating True Positives, True Negatives, False Positives and False Negatives is better way to evaluate this.

For this evaluation following techniques can be utilized from Sklearn library.

Recall score
Accuracy score
F1 score
AUC Score

In Sklearn the method caled "classification_report" gives a summary of those evaluations (except AUC Score from the list). That can be a good evaluation metric to check the performance of the model.


forecasting - likelihood of customers participating in next month sales I have historical transaction information of customers for the last 2 years and other information about the customers like what type of card (gold/platinum) they used for transactions etc. is also there in the dataset. Using this dataset, I will have to forecast the likelihood of each customers transacting next month. What are the different approaches that I can analyze before I choose one?

EDIT

More info on the problem:

I have customer credit information and transaction history of the cards for an online ticket booking portal. From this portal customers can book flights, cruises and cars.
I have to predict the customers who are most likely to book something (flights, cruises and cars) in the next month. Now what possible approaches can I take for this?

Forecasting is about predicting a variable as a function of time, such as the number of sales during coming months). Your problem is a regression problem, where the output is between 0 (no chance) and 1 (certain).

Any method to solve a regression problem is valid, but I would recommend starting with the simplest thing you think may work, to establish a baseline performance that you can try to improve.

This problem is often phrased as churn prediction, which is just the opposite chance from what you need (although definitions of churn vary), so you can google and see how people do that.

One easy approach that allows you to calculate this is to use the RFM model, where R, F and M stand for recency (time since last purchase), frequency (of purchases) and monetary (total money spent). You could use these features and a logistic regression as your baseline model.


Please help me out with this Python error - 'invalid syntax' Code:

import matplotlib.pyplot as plt
import numpy as np
from sklearn import datasets, linear_model
house_price = [245, 312, 279, 308, 199, 219, 405, 324, 319, 255]
size = [1400, 1600, 1700, 1875, 1100, 1550, 2350, 2450, 1425, 1700]
size2 = np.array(size).reshape((-1, 1))
#fitting into the model
regr = linear_model.LinearRegression()
regr.fit(size2, house_price)
print("Coefficients: \n", regr.coef_)
print("intercept: \n", regr.intercept_)
#############################
#formula obtained for the trained model
def graph(formula, x_range):
   x = np.array(x_range)
   y = eval(formula)
   plt.plot(x, y)
#plotting the prediction line
graph('regr.coef_*x + regr.intercept_', range(1000, 2700))
print(regr.score(size2, house_price))
plt.scatter (size,house_price, color='black')
plt.ylabel('house price')
plt.xlabel('size of house')
plt.show()

Error

**Error Line:print regr.predict([2000])**
Error: File "<ipython-input-4-9afa91ca7f9e>", line 1
    print regr.predict([2000])
             ^
SyntaxError: invalid syntax

In your current line

print regr.predict([2000])

This will not work. The first error is the lack of brackets around the contents of your print statement which is required in Python 3. Change this first to

print(regr.predict([2000]))

However, you will see that this does not work either. I suspect you are attempting to evaluate the price for a new $size = 2000$. You will need to reshape the input to your regression for this to work.

new_size = np.array([2000]).reshape((-1, 1))
print(regr.predict(new_size))

  [ 317.78380528]


Performance of CNN based deep models with number of classes How does a given deep cnn model performance vary with number of classes in tasks such as classification, object detection segmentation?

For example mobilenet v2 gives around 90% accuracy on classification task using PASCAL VOC; now if we train the same network with just 2 classes (say human and dog) how will it effect accuracy, speed and model size?(say, using tensorflow).Also what will be the case in tasks like object detection and semantic segmentation(in particular)?

Can we make the network smaller to learn from the smaller dataset(2 classes) to achieve same or better accuarcy & speed?

You might want to have a look at the hierarchical classification idea I described in my masters thesis.

In short:

If I had a problem where I needed to distinguish 42 breeds of dogs and 30 breeds of cats, I would very likely just create one classifier for the $42+30=72$ classes. Making a two-step approach (first cat vs dog, then cat-breed / dog-breed) seems not to give better results and for sure is more complicated
Analysis of errors becomes simpler the less classes you have
A single accuracy score for more than a handfull of classes / skewed data is not meaningful. For example, if you have two models with both 90% accuracy, one might be way better than the other one.


Could anyone explain these terms, "input space", "feature space", "sample space", "hypothesis space", "parameter space" with a concrete example? People use these terms "input space", "feature space", "sample space", "hypothesis space", "parameter space" in machine learning.

Could anyone explain these terms with a concrete example, such as sklearn MNIST dataset?, which has 1797 Samples, 10 Classes, 8*8 Dimensionality and 17 Features.

Please do NOT talk about in general.

For example, in this particular case, is the feature space a set of 17 elements {0, 1, ..., 16}?

We'll discuss each of the terms.
Input Space
It contains all the possible inputs for a model. Suppose the model takes in a vector, $input = [ x_1 , x_2]$ , where $x_1 , x_2 \in [ 1 , 10 ]$, then we can have $10^{2}$ inputs. This constitutes the "input space". See here.
For the MNIST dataset, the dimensions of the image are 8 * 8 meaning 64 points. Now each point can have a value lying in the interval $[ 0 , 16 ]$, so it can have 16 values. So the input space has a size of $16^{64}$.
Feature Space
The multidimensional space in which are features is defined. Considering the above examples, we can have three samples,
$a_1 = [ 2 , 3 ] \\ a_2 = [ 7 , 4.5 ] \\ a_3 = [ 3.67 , 2 ]$
These vectors could be included in an n-dimensional space ( here n=2 for our case ). Hence, in our case, the 2D space where we can plot our features constitutes our "feature space".
For the MNIST dataset, the input vector has 64 elements which correspond to a 64-dimensional space ( feature space ).
See this answer.

Difference between input space and feature space.
Input spaces include all possible inputs for our model. Feature
spaces, on the other hand, include the feature vectors from a given
set of data. They may not contain all the possible inputs for a model.

Hypothesis Space
Space which contains all the functions produced by a model. The functions map the inputs to their respective outputs. A model can output various functions ( or rather relationships between the inputs and outputs ) based on its learning. If you have a larger hypothesis space, the model cannot find the "best" one. See this answer.
For the MNIST dataset, as we calculated earlier, the size of the input space is $16^{64}$. Each one of them can have any one of the 10 labels ( classes ). Hence, the size of the hypothesis space is $10^{16^{64}}.$
Parameter Space
For each model in ML, we have some parameters for the model. The space in which we can define these parameters ( or hyperparameters ) is our "parameter space". From Wikipedia's example, we can understand it,
The parameter space would differ for every model.

In a sine wave model ${\displaystyle y(t)=A\cdot \sin(\omega t+\phi
> ),}y(t)=A\cdot \sin(\omega t+\phi )$, the parameters are amplitude A >
0, angular frequency ω > 0, and phase φ ∈ S1. Thus the parameter space
is ${\displaystyle R^{+}\times R^{+}\times S^{1}}$.


What is the formula to calculate the precision, recall, f-measure with macro, micro, none for multi-label classification in sklearn metrics? I am working in the problem of multi-label classification tasks. But I would not able to understand the formula for calculating the precision, recall, and f-measure with macro, micro, and none. Moreover, I understood the formula to calculate these metrics for samples. Even, I am also familiar with the example-based, label-based, and rank-based metrics.

For instance,

import numpy as np
from sklearn.metrics import hamming_loss, accuracy_score, precision_score, recall_score, f1_score
from sklearn.metrics import multilabel_confusion_matrix
y_true = np.array([[0, 1, 1 ],
                   [1, 0, 1 ],
                   [1, 0, 0 ],
                   [1, 1, 1 ]])

y_pred = np.array([[0, 1, 1],
                   [0, 1, 0],
                   [1, 0, 0],
                   [1, 1, 1]])

conf_mat=multilabel_confusion_matrix(y_true, y_pred)
print("Confusion_matrix_Train\n", conf_mat)

Confusion matrix output:

 [[[1 0]
  [1 2]]

 [[1 1]
  [0 2]]

 [[1 0]
  [1 2]]]

Macro score

print("precision_score:", precision_score(y_true, y_pred, average='macro'))
print("recall_score:", recall_score(y_true, y_pred, average='macro'))
print("f1_score:", f1_score(y_true, y_pred, average='macro'))

Macro score output:

precision_score: 0.8888888888888888
recall_score: 0.7777777777777777
f1_score: 0.8000000000000002

Micro score

print("precision_score:", precision_score(y_true, y_pred, average='micro'))
print("recall_score:", recall_score(y_true, y_pred, average='micro'))
print("f1_score:", f1_score(y_true, y_pred, average='micro'))

Micro score output:

precision_score: 0.8571428571428571
recall_score: 0.75
f1_score: 0.7999999999999999

Weighted score

print("precision_score:", precision_score(y_true, y_pred, average='weighted'))
print("recall_score:", recall_score(y_true, y_pred, average='weighted'))
print("f1_score:", f1_score(y_true, y_pred, average='weighted'))

Weighted score output:

precision_score: 0.9166666666666666
recall_score: 0.75
f1_score: 0.8

Samples score

print("precision_score:", precision_score(y_true, y_pred, average='samples'))
print("recall_score:", recall_score(y_true, y_pred, average='samples'))
print("f1_score:", f1_score(y_true, y_pred, average='samples'))

Samples score output:

precision_score: 0.75
recall_score: 0.75
f1_score: 0.75

None score

print("precision_score:", precision_score(y_true, y_pred, average=None))
print("recall_score:", recall_score(y_true, y_pred, average=None))
print("f1_score:", f1_score(y_true, y_pred, average=None))

None score output:

precision_score: [1.         0.66666667 1.        ]
recall_score: [0.66666667 1.         0.66666667]
f1_score: [0.8 0.8 0.8]

Thanks in advance for your help.

Generally, the scoring metrics you are looking at are defined as following (see for example Wikipedia):

$$precision = \frac{TP}{TP+FP}$$
$$recall= \frac{TP}{TP+FN}$$
$$F1 = \frac{2 \times precision \times recall}{precision + recall}$$

For the multi-class case scikit learn offers the following parameterizations (see here for example):

  'micro':
  Calculate metrics globally by counting the total true positives, false negatives and false positives.

  'macro':
  Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account.

  'weighted':
  Calculate metrics for each label, and find their average weighted by support (the number of true instances for each label). This alters ‘macro’ to account for label imbalance; it can result in an F-score that is not between precision and recall.

  'samples':
  Calculate metrics for each instance, and find their average (only meaningful for multilabel classification where this differs from accuracy_score).

And none does the following:

  If None, the scores for each class are returned.

TLDR: "micro" calculates the overall metric, "macro" derives an average assigning each class an equal weight and "weighted" calculates an average assigning each class a weight based on the number of ocurences (its support).

Accordingly, the calculations in your example go like this:

Macro

$$precision_{macro} = \sum_{classes} \frac{precision\text{ }of \text{ }class}{number\text{ }of\text{ }classes} = \frac{(2/2) + (2/3) + (2/2)}{3} \approx 0.89$$

$$recall_{macro} = \sum_{classes} \frac{recall\text{ }of \text{ }class}{number\text{ }of\text{ }classes} = \frac{(2/3) + (2/2) + (2/3)}{3} \approx 0.78$$

$$F1_{macro}= \ \sum_{classes} \frac{F1\text{ }of \text{ }class}{number\text{ }of\text{ }classes} = \frac{1}{3} \times \frac{2 \times (2/2) \times (2/3)}{(2/2) + (2/3)} + \frac{1}{3} \times \frac{2 \times (2/3) \times (2/2)}{(2/3) + (2/3)} + \frac{1}{3} \times \frac{2 \times (2/2) \times (2/3)}{(2/2) + (2/3)} \approx 0.80$$

Note that macro means that all classes have the same weight, i.e. $\frac{1}{3}$ in your example. That is where the $\times \frac{1}{3}$ to calculate the F1 score comes from.

Micro

$$precision_{micro} = \frac{\sum_{classes} TP\text{ }of \text{ }class}{\sum_{classes} TP\text{ }of\text{ }class + FP\text{ }of\text{ }class } = \frac{2+2+2}{2+3+2} \approx 0.86$$

$$recall_{micro} = \frac{\sum_{classes} TP\text{ }of \text{ }class}{\sum_{classes} TP\text{ }of\text{ }class+FN\text{ }of\text{ }class} = \frac{2+2+2}{3+2+3} = 0.75$$

$$F1_{micro}= 2\times \frac{recall_{micro} \times precision_{micro}}{recall_{micro} + precision_{micro}} \approx 0.8$$

Weighted
$$precision_{weighted} = \sum_{classes}{weight\text{ }of \text{ }class \times precision\text{ }of\text{ }class} = \frac{3}{8}\times\frac{2}{2} + \frac{2}{8}\times\frac{2}{3} + \frac{3}{8} \times \frac{2}{2} \approx 0.92$$

$$recall_{weighted} = \sum_{classes}{weight\text{ }of \text{ }class \times recall\text{ }of\text{ }class} = \frac{3}{8} \times \frac{2}{3} + \frac{2}{8}\times\frac{2}{2} + \frac{3}{8} \times \frac{2}{3} = 0.75$$

$$F1_{weighted} = \sum_{classes}{weight\text{ }of \text{ }class \times F1\text{ }of\text{ }class} = \frac{3}{8} \times \frac{2 \times (2/2) \times (2/3)}{(2/2) + (2/3)} + \frac{2}{8} \times \frac{2 \times (2/3) \times (2/2)}{(2/3) + (2/3)} + \frac{3}{8} \times \frac{2 \times (2/2) \times (2/3)}{(2/2) + (2/3)} = 0.8$$

None

$precision_{class 1} = \frac{2}{2} = 1.0$

$precision_{class 2} = \frac{2}{2+1} \approx 0.67$

$precision_{class 3} = \frac{2}{2} = 1.0$

$recall_{class 1} = \frac{2}{2+1}  \approx 0.67$

$recall_{class 2} = \frac{2}{2} = 1.0$

$recall_{class 3} = \frac{2}{2+1} \approx 0.67$

$F1_{class 1} =  \frac{2 \times 1 \times \frac{2}{3}}{1 + \frac{2}{3}} = 0.8$

$F1_{class 2} =  \frac{2  \times \frac{2}{3}\times 1}{\frac{2}{3} + 1} = 0.8$

$F1_{class 3} =  \frac{2 \times 1 \times \frac{2}{3}}{1 + \frac{2}{3}} = 0.8$

Samples

$$Precision_{samples}= \frac{1}{Number\, of\, examples} \sum_{examples} \frac{TP\,of\,example}{TP\,of\,example + FP\,of\,example} = \frac{1}{4}[\frac{2}{2}+\frac{0}{1}+\frac{1}{1}+\frac{3}{3}] = 0.75$$

$$Recall_{samples}= \frac{1}{Number\, of\, examples} \sum_{examples} \frac{TP\,of \,example}{TP\,of\,example + FN\,of\,example} = \frac{1}{4}[\frac{2}{2}+\frac{0}{2}+\frac{1}{1}+\frac{3}{3}] = 0.75$$

$$F1_{samples}= 2\times \frac{recall_{samples} \times precision_{samples}}{recall_{samples} + precision_{samples}} = 0.75$$


Twitter tweet classification I am trying to do a small project on my own to find out job openings using twitter data. I saved data using flume and converted it to .csv for analysis. My problem is i don't know how to classify tweets, whether it is a job vacancy or just some news on say machine-learning.I read online about neural networks and word2vec but i am not sure if it will solve my problem. Can anyone suggest some ways to do this based on tweet text and hashtags.I don't have training and test data, i just have tweets stored using flume.Also what kind of analysis will i be able to do with it.I am new to data science :/

You should be reading about text classification with supervised learning technique. You could chose a neural network such as CNN implementation and use it to your data. But you have to first prepare your data by cleaning it up and labelling it with respect to your desired category. For example,

You should collect the tweet data and label each tweet or set of words as job or news with respect to their original category.
Then you should use the prepared data and labels to train the chosen network.
Read about k fold cross validation and segregate data in such a way so that systematic iterative training and validation can be done and tweaks to the network could be made.
Finally once the training is completed you could use the model to test it on unseen test data.

These are common steps involved in text classification. It should be customized according to your need.  You could google profanity filter which is mostly developed with a NLP type classifier. That would help you a lot.


How does the Naive Bayes algorithm function effectively as a classifier, despite the assumptions of conditional indpendence and bag of words? Naive Bayes algorithm used for text classification relies on 2 assumptions to make it computationally speedy:

Bag of Words assumption: the position of words is not considered
Conditional Independence: words are independent of one another

In reality, neither of those conditions often holds, yet Naive Bayes is quite effective. Why is that?

The main reason is that in many cases (but not always) the model obtains enough evidence to make the right decision just from knowing which words appear and don't appear in the document (possibly also using their frequency, but this is not always needed either).

Let's take the textbook example of topic detection from news documents. A 'sports' article is likely to contain at least a few words which are unambiguously related to sports, and the same holds for many topic as long as the topics are sufficiently distinct.

In general tasks which are related to the general semantics of the text work reasonably well with unigrams (single words, unordered) as features, whether with NB or other methods. It's different for tasks which require taking syntax into account, or which require a deeper understanding of the semantics.


How can I change not only the value of the custom field but the what the field shows itself? I need to be able to write custom fields in Power BI but the values of the fields change based off of company. For example, Twitter vs Instagram - field 1 for Twitter should show company name and field 1 for Instagram should show company revenue. So not only does the field change but the value of the field changes too. I have an excel sheet that lists around 30 custom fields for 2 different companies so I was wondering what the easiest way around that would be. Thanks!

I'm by no means a PowerBI expert, so there may be options I'm not aware of. I don't think that there is a tidy solution to this in PowerBI, as this is not quite what BI tools are usually intended to do. However, it can be done-- you'll just have to accept a degree of hack/kludge character to your projects.

The most straightforward approach I can think of is to make a new table for your presentation values, with something like a column for each company and then another column for "meaningful data to present". That latter column should be set to something like text, so that it will display different values with the correct conventions (dollar signs, comma placement, etc.). Then, when drawing on data to populate the report, set the card (or whatever structure you're using) to draw from that presentation table. Since field1 will always contain the correct data to display for a given company, and that field is formatted as text, you won't have to worry about mismatched data types.

This approach will require some preprocessing of your data, so that the presentation table is populated correctly for each distinct company whose data you are presenting, but it's nothing that a SQL query can't produce pretty easily. This can be done within PowerBI or externally, with the presentation table imported. Importantly, you will not have the option of using aggregations or similar operations (that would be the case anyways, with this type of report output, but it's worth mentioning explicitly). This will be, essentially, a display-only report.

The key is that field1 (using the naming convention in the question) is representing something unstable in the data itself (different data types in the same field), but relevant in an "above the data" sense.

As a final note, I'll encourage you to consider whether or not this data really belongs in a single, conglomerated report. That might legitimately be the case, or it may be that this structure is demanded by a superior (whether it makes sense or not). But maintaining a single, kludge-filled report like this might ultimately be more work (and less valuable) than multiple reports which fit better with PowerBI's design favoring strongly-typed data fields.


Google's election results data visualization Can someone explain the data visualization of Google's live election results to me?

My understanding is that the states they've "called" have a check mark that is blue if a Democrat (Clinton) victory or red if a Republican (Trump) victory. The results additionally show the number of votes report thusfar as a count and as a percent. The states they consider too early to call based on the % of precincts reporting do not have a check mark.

However, in the case of Washington it has been "called" as a Democrat (Clinton) victory, but shows a higher count and percent for Trump (?).

View 1:

View 2:

Note: There were some smaller third party candidates cut off in the screenshot above, like 2 Socialist candidates.

Possibly because google was predicting based on changes from previous elections, possibly combined with opinion polls, on a precinct-basis. If all the pro-Trump precincts report in first, you'd still see a blue box because of the very low probability of historically Democrat precincts voting Trump.

"Calling" something in a US election is not an official result, its a confident prediction by an agency. Sometimes these are wrong. Search for "Dewey Defeats Truman" for a classic example.


How to deal with word length variability while using char-level one-hot encoding? I am trying to run some experiments on internal word structure of morphologically rich language (Russian).

Each valid character is 1-hot encoded and then fixed-length vectors are concatenated to produce word vector:

 c             a             t

(0... 1... 0) (0... 1... 0) (0... 1... 0)

This approach produces vectors of different length for words of different length. To unify them I have applied zero-padding:

c a t 0 0 0 0 0 0 0 0

k i t t e n 0 0 0 0 0

t r a n q u i l i t y

h a p p i n e s s 0 0

Does this approach sounds valid? What are best practices on the described issue?

Your padded vector should be OK as a starting model. You could take it a step further and have a specific "no character" encoding, as that will be a clearer learning signal.

You say that you are using a feed-forward neural network. In that case, there is a possible change and improvement to your model, which is to use Recurrent Neural Network (RNN), probably a long short-term memory (LSTM) variation, because that has had some success in language models.

In a RNN, instead of having the character encoding repeated N times, you have it represented just once, and run the network N times for each sample, feeding in each character in turn. Instead of a "no character" symbol, you would have an "end of sequence" symbol. For an open-ended model with no specific target, it is common to have the output of the network be a prediction of the next character. This allows you to explore what the model has learnt by sampling from its predictions - a lot like using n-grams to generate realistic-looking words.

The advantage of a RNN for character-level language modelling is that the model is built around the assumption that each input is somehow equivalent - the same encoding is used at each time step. This is clearly true for character strings, so it can lead to better models from less training data.

The main disadvantage of using a RNN is that it is more complex to understand, build and train such a model.


Preference Matching Algorithm There's this side project I'm working on where I need to structure a solution to the following problem.

I have two groups of people (clients). Group A intends to buy and group B intends to sell a determined product X. The product has a series of attributes x_i, and my objective is to facilitate the transaction between A and B by matching their preferences. The main idea is to point out to each member of A a corresponding in B whose product better suits his needs, and vice versa.

Some complicating aspects of the problem:

The list of attributes is not finite. The buyer might be interested in a very particular characteristic or some kind of design, which is rare among the population and I can't predict. Can't previously list all the attributes;
Attributes might be continuous, binary, or non-quantifiable (ex: price, functionality, design);

Any suggestion on how to approach this problem and solve it in an automated way?

I would also appreciate some references to other similar problems if possible.

Great suggestions! Many similarities in to the way I'm thinking of approaching the problem.

The main issue on mapping the attributes is that the level of detail to which the product should be described depends on each buyers. Let’s take an example of a car. The product “car” has lots and lots of attributes that range from its performance, mechanical structure, price etc.

Suppose I just want a cheap car, or an electric car. Ok, that's easy to map because they represent main features of this product. But let’s say, for instance, that I want a car with Dual-Clutch transmission or Xenon headlights. Well there might be many cars on the data base with this attributes but I wouldn't ask the seller to fill in this level of detail to their product prior to the information that there is someone looking them. Such a procedure would require every seller fill a complex, very detailed, form just try to sell his car on the platform. Just wouldn't work.

But still, my challenge is to try to be as detailed as necessary in the search to make a good match. So the way I'm thinking is mapping main aspects of the product, those that are probably relevant to everyone, to narrow down de group of potential sellers.

Next step would be a “refined search”. In order to avoid creating a too detailed form I could ask buyers and sellers to write a free text of their specification. And then use some word matching algorithm to find possible matches. Although I understand that this is not a proper solution to the problem because the seller cannot “guess” what the buyer needs. But might get me close.

The weighting criteria suggested is great. It allows me to quantify the level to which the seller matches the buyer’s needs. The scaling part might be a problem though, because the importance of each attribute varies from client to client. I'm thinking of using some kind of pattern recognition or just asking de buyer to input the level of importance of each attribute.

My first suggestion would be to somehow map the non-quantifiable attributes to quantities with the help of suitable mapping functions. Otherwise, simply leave them out.

Secondly, I don't think that you need to assume that the list of attributes is not finite. A standard and intuitive approach is to represent each attribute as an individual dimension in a vector space. Each product is then simply a point in this space. In that case, if you want to dynamically add more attributes you simply have to remap the product vectors into the new feature space (with additional dimensions).

With this representation, a seller is a point in the feature space with product attributes and a buyer is a point in the same feature space with the preference attributes. The task is then to find out the most similar buyer point for a given seller point.

If your dataset (i.e. the number of buyers/sellers) is not very large, you can solve this with a nearest neighbour approach implemented with the help of k-d trees.

For very large sized data, you can take an IR approach. Index the set of sellers (i.e. the product attributes) by treating each attribute as a separate term with the term-weight being set to the attribute value. A query in this case is a buyer which is also encoded in the term space as a query vector with appropriate term weights. The retrieval step would return you a list of top K most similar matches.