Dynamic Time Warping is outdated? At http://www.speech.zone/exercises/dtw-in-python/ it says

  Although it's not really used anymore, Dynamic Time Warping (DTW) is a
  nice introduction to the key concept of Dynamic Programming.

I am using DTW for signal processing and are a little surprised: What is used instead?

I wouldn't consider DTW to be outdated at all. In 2006 Xi et al. showed that

  [...] many algorithms have been proposed for the problem of time series
  classification. However, it is clear that one-nearest-neighbor with
  Dynamic Time Warping (DTW) distance is exceptionally difficult to
  beat.

The results of this paper are summarized in the book "Temporal Data Mining" by Theophano Mitsa as follows:

  In [Che05a], a static minimization–maximization approach yields a maximum error of 7.2%. With 1NN-DTW, the error is 0.33% with the same
  dataset as in the original article.
  In [Che05b], a multiscale histogram approach yields a maximum error of 6%. With 1NN-DTW, the error (on the same data set) is 0.33%.
  In [Ead05], a grammar-guided feature extraction algorithm yields a maximum error of 13.22%. With 1NN-DTW, the error was 9.09%.
  In [Hay05], time series are embedded in a lower dimensional space using a Laplacian eigenmap and DTW distances. The authors achieved an
  impressive 100% accuracy; however, the 1NN-DTW also achieved 100%
  accuracy.
  In [Kim04], Hidden Markov Models achieve 98% accuracy, while 1NN-DTW achieves 100% accuracy.
  In [Nan01], a multilayer perceptron neural network achieves the best performance of 1.9% error rate. On the same data set, 1NN-DTW’s rate
  was 0.33%.
  In [Rod00], first-order logic with boosting gives an error rate of 3.6%. On the same dataset, 1NN-DTW’s error rate was 0.33%.
  In [Rod04], a DTW-based decision tree gives an error rate of 4.9%. On the same dataset, 1NN-DTW gives 0.0% error. • In [Wu04], a
  super-kernel fusion set gives an error rate of 0.79%, while on the
  same data set, 1NN-DTW gives 0.33%.

Please see the original book for a list of the mentioned references.

An important thing to note here is the fact that Xi et al. even managed to beat the performance of an MLP back in 2006. Even though the situation might look a bit different these days (as we have better and faster Deep learning algorithms at hand), I would still consider DTW a valid option to look into when it comes to signal classifications.

Update

I would also like to add a link to a more recent paper called "The Great Time Series Classification Bake Off: An Experimental Evaluation of Recently Proposed Algorithms" from 2016. In this paper, the authors "have implemented 18 recently proposed algorithms in a common Java framework and compared them
against two standard benchmark classifiers (and each other)". The following quotes from the paper stress that DTW is (or at least was in 2016) indeed still relevant:

  Many of the algorithms are in fact no better than our two benchmark classifiers, 1-NN DTW and Rotation Forest.
  For those looking to build a predictive model for a new problem we would recommend starting with DTW, RandF and RotF as a basic sanity
  check and benchmark.
  Received wisdom is that DTW is hard to beat.


Retrain image classifier using MobileNet v2 I am using my own dataset to retrain mobilenet_v2_100_224 model, I currently have 4 classes where each class have more than 100 images still I'm observing overfitting even though I've used --random_scale and --random_brightness parameters, how should I overcome this overfitting problem when there's no such regularization technique available in the retrain.py script? following is my TensorBoard visualization:

Retrain command:

python retrain.py --tfhub_module https://tfhub.dev/google/imagenet/mobilenet_v2_100_224/feature_vector/2 --how_many_training_steps 500 --random_scale=5 --random_brightness=10 --output_graph=./retrained_graph.pb --output_labels=./retrained_labels.txt --image_dir ./data --summaries_dir /tmp/log

I do not see any overfitting there. Overfitting is when your validation loss becomes worse with time, while training loss improves. Just because your training error is less than the validation (which is expected) does not mean it overfits.

But you can try to freeze the first layers (up to the fully connected one) to ensure the network does not lose generalization.


Understanding scipy sparse matrix types I am trying to select the best scipy sparse matrix type to use in my algorithm. In that, I should initialize data in a vij way, then I should use it to perform matrix vector multiplication. Eventually I have to add rows and cols. Trying to select the best for my problem, I want to understand which are the best cases to use each of this types: lil_matrix, coo_matrix, csr_matrix, csc_matrix, dok_matrix. Can someone explain me? Its not necessary to show examples of all the types in the same answer.

Ok, I was looking for an answer and now I have it clearer:
Scipy documentation does not elaborate too much on the explanation, but wikipedia article is much more clear. For those who are looking for an answer, there are two major groups of sparse matrices:

a) Sparse types used to construct the matrices:

DOK (Dictionary Of Keys): a dictionary that maps (row, column) to the value of the elements. It uses a hash table so it's efficient to set elements.
LIL (LIst of Lists): LIL stores one list per row. The lil_matrix format is row-based, so if we want to use it then in other operations, conversion to CSR is efficient, whereas conversion to CSC is less so.
COO (COOrdinate list): stores a list of (row, column, value) tuples.

b) Sparse types that support efficient access, arithmetic operations, column or row slicing, and matrix-vector products:

CSR (Compressed Sparse Row): similar to COO, but compresses the row indices.  Holds all the nonzero entries of M in left-to-right top-to-bottom ("row-major") order (all elements in the first row, all elements in the second row, and so). More efficient in row indexing and row slicing, because elements in the same row are stored contiguously in the memory.
CSC (Compressed Sparse Column): similar to CSR except that values are read first by column. More efficient in a column indexing and column slicing.

Once the matrices are build using one of the a) types, to perform manipulations such as multiplication or inversion, we should convert the matrix to either CSC or CSR format.


High train and val results. Bad test and predict results For my thesis project I've been trying to make a CNN for some challenging data.
There's four classes with the following amount of images respectively
[410, 410, 269, 206] = 1,295 total.

Now I know that this is not perfect, both a small and a unbalanced dataset. I am using this tutorial as an example, resulting in this code:

# dimensions of our images.
img_width, img_height = 200, 200

top_model_weights_path = 'bottleneck_fc_model.h5'
train_data_dir = '/content/output/train'
validation_data_dir = '/content/output/val'
nb_train_samples = 760
nb_validation_samples = 240
epochs = 30
batch_size = 10

def save_bottlebeck_features():

    # build the VGG16 network
    model = applications.VGG16(include_top=False, weights='imagenet', input_shape=(img_width, img_height, 3))

    train_datagen = ImageDataGenerator(rescale=1. / 255,
                                      horizontal_flip=True,
                                      vertical_flip=True)

    generator = train_datagen.flow_from_directory(
        train_data_dir,
        target_size=(img_width, img_height),
        batch_size=batch_size,
        class_mode='categorical',
        classes = folders,
        shuffle=False)

    bottleneck_features_train = model.predict_generator(generator, nb_train_samples // batch_size)
    np.save(open('bottleneck_features_train.npy', 'wb'),
            bottleneck_features_train)

    datagen = ImageDataGenerator(rescale=1. / 255)

    generator = datagen.flow_from_directory(
        validation_data_dir,
        target_size=(img_width, img_height),
        batch_size=batch_size,
        class_mode='categorical',
        classes = folders,
        shuffle=False)

    bottleneck_features_validation = model.predict_generator(
        generator, nb_validation_samples // batch_size)

    np.save(open('bottleneck_features_validation.npy', 'wb'),
            bottleneck_features_validation)

save_bottlebeck_features()

#def train_top_model():
train_data = np.load(open('bottleneck_features_train.npy', 'rb'))
train_labels = to_categorical(np.array(
    [0] * (nb_train_samples // 2) + [1] * (nb_train_samples // 2)), 4)

print(train_data.shape)

validation_data = np.load(open('bottleneck_features_validation.npy', 'rb'))
validation_labels = to_categorical(np.array(
    [0] * (nb_validation_samples // 2) + [1] * (nb_validation_samples // 2)), 4)

print(validation_data.shape)

model = Sequential()
model.add(Flatten(input_shape=train_data.shape[1:]))
model.add(Dense(256, activation='relu',
                kernel_regularizer=regularizers.l2(0.0005),
                activity_regularizer=regularizers.l1(0.0005)))
model.add(Dropout(0.5))
model.add(Dense(4, activation='sigmoid'))

model.compile(optimizer='SGD',
              loss='categorical_crossentropy', metrics=['accuracy'])

history = model.fit(train_data, train_labels,
          epochs=epochs,
          batch_size=batch_size,
          validation_data=(validation_data, validation_labels),
          verbose=2)
model.save(top_model_weights_path)

Now my training and validation results are good:

But when I try to evaluate my model with test data which I held back, I get bad results:
Found 261 images belonging to 4 classes.

test acc: 0.458248479333526

test loss: 3.629304162353702

This is just a little better then guessing.

    .......precision    recall  f1-score   support

    0kpa       0.30      0.48      0.37        82
   40kpa       0.16      0.50      0.24        42
   80kpa       0.00      0.00      0.00        82
  160kpa       0.00      0.00      0.00        55

  accuracy                         0.23      261
  macro avg     0.12      0.24     0.15      261
  weighted avg  0.12      0.23     0.15      261

I notice that the last two classes get very bad results, what could cause this? I do shuffle the data before I split it up in train/val/test.
I also noticed that when i fed the model.evaluate_generator(..) with training data instead of test data it performed as bad.. that seems very odd to me..
Can anyone tell me what's going on!? Thanks in advance!

I figured it out.

First: I was making the labels the wrong way.

Instead of this:

train_labels = to_categorical(np.array(
[0] * (nb_train_samples // 2) + [1] * (nb_train_samples // 2)), 4)

do this:

train_labels = to_categorical(training_generator.classes, 4)

The first snippet wrongly generated a 4-ary array, where only the first and second element got a 1 assigned.

Second:

I found out my method of evaluation was wrong. I tried to use model.evaluate_generator which yielded the wrong results. I think it was because I tried to feed the model raw images, instead of creating bottleneck features from the raw test images and then feeding them to the model.

so instead of this:

from keras.models import load_model
conv_base = applications.VGG16(include_top=False, weights='imagenet', input_shape=(img_width, img_height, 3))
loaded_model = load_model('bottleneck_fc_model.h5')

model = Sequential()
model.add(conv_base)
model.add(loaded_model)

model.compile(optimizer='SGD',
              loss='categorical_crossentropy', metrics=['accuracy'])

test_datagen = ImageDataGenerator(rescale=1./255)

test_generator = test_datagen.flow_from_directory(
        '/content/output/test',
        target_size=(img_width, img_height),
        batch_size=batch_size,
        class_mode='categorical')

test_loss, test_acc = model.evaluate_generator(test_generator, 216 // batch_size)
print('test acc:', test_acc)
print('test loss:', test_loss)

I did this (Which worked):

from keras.models import load_model
model = applications.VGG16(include_top=False, weights='imagenet', input_shape=(img_width, img_height, 3))

bottleneck_features_test = model.predict_generator(
    test_generator, 261 // batch_size)

loaded_model = load_model('bottleneck_fc_model.h5')

model = Sequential()
model.add(loaded_model)

model.compile(optimizer='SGD',
              loss='categorical_crossentropy', metrics=['accuracy'])

test_datagen = ImageDataGenerator(rescale=1. / 255)

test_generator = test_datagen.flow_from_directory(
    '/content/output/test',
    target_size=(img_width, img_height),
    batch_size=batch_size,
    class_mode='categorical',
    classes = folders,
    shuffle=False)

bottleneck_features_test = conv_base.predict_generator(
    test_generator, 261 // batch_size)

test_labels = to_categorical(test_generator.classes, 4)

(test_eval_loss, test_eval_accuracy) = model.evaluate(
   bottleneck_features_test[:256], test_labels[:256], batch_size=batch_size, verbose=1)

print("[INFO] test accuracy: {:.2f}%".format(test_eval_accuracy * 100))
print("[INFO] test Loss: {}".format(test_eval_loss))

I hope this helps someone who's struggling with the same!


Simple Excel Question: VLookup Error My data looks like this:

Why is this error showing up?

This is a common result of imprecise matching, such as with whitespace problems.

For example, if you imagine that the grey shading used in codeblocks on StackExchange represents the whitespaces in your Excel cells, then
  AK
is technically the same as

AK

You can trim and/or use wildcard chars to allow matching to skip spaces. For example:

=VLOOKUP(CONCATENATE("*",TRIM(A2),"*"), $E$2:$F$5, 2,FALSE)


Feauture selection for clustering regarding zero-correlated feature I want to cluster a 5 feature data-set. Firstly to explore the data I did a correlation matrix to see if some features where highly correlated so I could reduce them. Then I saw a feature that have close to zero correlation against all the other features. This got me wondering if I should exclude this parameter since it acts as a kind of "noise" relatively to all the other features. What's your opinion?

Lack of correlation with other features is not a reason to omit a feature. On the contrary, it is usually a reason to keep the feature because it may provide unique information. Typically, highly correlated features provide redundant information and feature reduction techniques (e.g., Principal Components Analysis) are used to remove the redundancy.

While it is possible that the uncorrelated feature is noise, you should not make that assumption. It could be that the uncorrelated feature is the only one containing information and the other 4 features are all correlated noise.


Keras: visualizing the output of an intermediate layer I have read the docs here and I understand the general idea. I am able to visualize the weights of the intermediate layers. However, I'm having trouble visualize the activations. Here's what I have:

I trained my model and saved the weights in a file called weights_file.

Thanks to this jupyter notebook, I got the values of the weights. First I defined my model:

def mlp_model(hid_dim=10):

        model = Sequential()
        model.add(Dense(units=hid_dim, input_dim=X.shape[1], activation='relu'))
        model.add(Dense(Y.shape[1], activation='softmax'))
        model.load_weights(weights_file)
        model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])

        return model

model_created = mlp_model(hid_dim=15)

To get the weights, I did this:

W = model_created.layers[0].kernel.get_value(borrow=True)
W = np.squeeze(W)
print("W shape : ", W.shape) #(153, 15)

W_out = model_created.layers[1].kernel.get_value(borrow=True)
W_out = np.squeeze(W_out)
print('W_out shape : ', W_out.shape) #(15, 8)

From there I could create Hinton diagrams using this. However, when I try to work with the activations:

get_first_output = theano.function([model_created.layers[0].input], [model_created.layers[1].output])
layer_out = get_first_output([X[0,:]])[0]

I get this error:

TypeError: ('Bad input argument to theano function with name "mlp1_visualize_weights.py:131" at index 0 (0-based).  \nBacktrace when that variable is created:\n\n  File "mlp1_visualize_weights.py", line 213, in <module>\n    mlp_repeat(X, Y, Xtest, Ytest, params_to_use, weights_file)\n  File "mlp1_visualize_weights.py", line 125, in mlp_repeat\n    model_created = mlp_model(hid_dim=hid_val, lr=lrate, reg_val=reg, momentum=moment, nest=nestval, optimizer=optim)\n  File "mlp1_visualize_weights.py", line 105, in mlp_model\n    model.add(Dense(units=hid_dim, input_dim=X.shape[1], kernel_initializer=\'he_uniform\', activation=\'relu\', W_regularizer=l2(reg_val), b_regularizer=l2(reg_val)))\n  File "/mnt/data/caugusta/pkgs/anaconda2/lib/python2.7/site-packages/keras/models.py", line 426, in add\n    dtype=layer.dtype, name=layer.name + \'_input\')\n  File "/mnt/data/caugusta/pkgs/anaconda2/lib/python2.7/site-packages/keras/engine/topology.py", line 1392, in Input\n    input_tensor=tensor)\n  File "/mnt/data/caugusta/pkgs/anaconda2/lib/python2.7/site-packages/keras/engine/topology.py", line 1303, in __init__\n    name=self.name)\n  File "/mnt/data/caugusta/pkgs/anaconda2/lib/python2.7/site-packages/keras/backend/theano_backend.py", line 184, in placeholder\n    x = T.TensorType(dtype, broadcast)(name)\n', 'TensorType(float32, matrix) cannot store accurately value [array([ 0.        ,  0.2037037 ,  0.20138889,  0.21100917,  0.62962963,\n        0.6875    ,  0.61206897,  0.44660194,  0.31168831,  0.17391304,\n        0. ...

I would like to look at just one input example, and find the activation and the weights from just that input example. Essentially I'm trying to figure out which features of the data each hidden unit is picking up.

Can anyone explain how to get the activations of intermediate layers in Keras?

Consider this network

model = Sequential()

model.add(Convolution2D(32, 3, 3, input_shape=(1,28,28)))
convout1 = Activation('relu')
model.add(convout1)
convout2 = MaxPooling2D()
model.add(convout2)

model.add(Flatten())

model.add(Dense(128))
model.add(Activation('relu'))
model.add(Dropout(0.2))
model.add(Dense(10))
model.add(Activation('softmax'))

Now you can visualize the activation using this function.

def layer_to_visualize(layer):
    inputs = [K.learning_phase()] + model.inputs

    _convout1_f = K.function(inputs, [layer.output])
    def convout1_f(X):
        # The [0] is to disable the training phase flag
        return _convout1_f([0] + [X])

    convolutions = convout1_f(img_to_visualize)
    convolutions = np.squeeze(convolutions)

    print ('Shape of conv:', convolutions.shape)

    n = convolutions.shape[0]
    n = int(np.ceil(np.sqrt(n)))

    # Visualization of each filter of the layer
    fig = plt.figure(figsize=(12,8))
    for i in range(len(convolutions)):
        ax = fig.add_subplot(n,n,i+1)
        ax.imshow(convolutions[i], cmap='gray')

# Specify the layer to want to visualize
layer_to_visualize(convout1)

If you want you can refer my jupyter notebook.


Demoing data science work To data scientists who use python for development, how do you demo your work?

I used to demo my work in Shiny as my developments was mainly in R. Now I develop 99% in Python but not sure of the best way to demo work (predictive modelling, data analytics side, pattern detection,..etc)?

Currently I use Dash, but the amount if time I spend on html and css makes it less appealing.

Use a Jupyter notebook. Write good descriptions in Markdown between blocks of code. Here are some examples showing machine learning in Python:

https://github.com/ogrisel/notebooks

https://github.com/ogrisel/notebooks/blob/master/Labeled%20Faces%20in%20the%20Wild%20recognition.ipynb


GD for logistic regression isn't stable. Why? Here's my (incomplete) implementation for linear regression using GD:

from enum import Enum
import numpy as np
from math import exp

class GDType(Enum):
  GD = 1
  SGD = 2

class LogisticRegression:
  def __init__(self, gd_type=GDType.GD, learning_rate=0.1, num_iterations=1000):
    self.gd_type = gd_type
    self.learning_rate = learning_rate
    self.num_iterations = num_iterations

  def __odds(self, x, b):
    return exp(b.dot(x)) / float(exp(1 + b.dot(x)))

  def __gd_step(self, X, y):

    grad = np.zeros(self.p)

    for j in range(self.p):
      for i in range(self.m):
        grad[j] += self.__odds(X[i], self.b) - y[i]
      grad[j] *= X[i, j]

    return grad

  def __gd(self, X, y):
    for i in range(self.num_iterations):
      dl = self.__gd_step(X, y)
      self.b -= (self.learning_rate / float(self.m)) * dl
      print(self.b)

  def fit(self, X, y):
    X = np.column_stack((np.ones(X.shape[0]).T, X)) # 1's column for the bias
    self.m = X.shape[0]
    self.p = X.shape[1]
    self.b = np.zeros(self.p)
    self.__gd(X, y)
    print(self.b)

I'm testing it on the iris data, as follows:

from sklearn.datasets import load_iris
iris = load_iris()
X = iris.data[:, :2]
y = (iris.target != 0) * 1

lr = LogisticRegression()
lr.fit(X, y)

The problem is that the coefficients are getting larger and larger instead of converging. Why?

I doubled checked (correct me if wrong) and it seems to me that __gd_step is correct.

Your implementation seems not correct. For example:

(1) in your __gd_step() function, the line

grad[j] *= X[i, j]

which uses the index i, is outside your for loop: for i in range(self.m).

(2) in your __odds() function, the denominator exp(1 + b.dot(x)) seems incorrect. 1 should be outside of exp().


Format for proposing a new algorithmic project? I am working for an organization that is still new to implementing data science projects. I have an idea for a data spider/algorithm project that will require various pieces of code as well as datasets that I will need to gain access to.

I know that this is something that I should just start doing instead of proposing, but unless I come up with a formal document, the org might see it as a "waste of time". So are there any generally accepted formats and/or project templates for proposing a new data science effort?

To be clear: I am looking for something that I can present to management and be able to "sell" them on a new data science project idea. So the template I'm looking for isn't necessarily too technical, it's more high-level than that.

There you go cookiecutter template. I use this, with all the necessary modifications. Each project can be really versatile in Data Science. Nonetheless, I would agree that is a general standard.


Which of the scikit learn classification algorithms accept Sparse matrices? If I have to use scikit-learn(sklearn) library for classification and the feature matrix is a sparse matrix then which of the classification algorithms of this library can be used by me?

From this kaggle discussion, the classification algorithms from scikit-learn that support sparse matrices are at least:

linear_model.LogisticRegression()
svm.SVR()
svm.NuSVR()
naive_bayes.MultinomialNB()
naive_bayes.BernoulliNB()
linear_model.PassiveAggressiveClassifier()
linear_model.Perceptron()
linear_model.Ridge()
linear_model.Lasso()
linear_model.ElasticNet()
linear_model.SGDClassifier()

Also, from this quora question, you can check in the implementation of the algorithm if they import scipy.sparse.csr_matrix .


Select two best classifier using F1-score,Recall and precision I have three classifiers that classify same dataset with these results:

classifier A:
              precision    recall   f1-score
  micro avg       0.36      0.36      0.36
  macro avg       0.38      0.43      0.36
  weighted avg    0.36      0.36      0.32

classifier B:
              precision    recall   f1-score
   micro avg       0.55      0.55      0.55
   macro avg       0.60      0.60      0.56
   weighted avg    0.61      0.55      0.53

classifier C:
               precision    recall   f1-score
   micro avg       0.34      0.34      0.34
   macro avg       0.36      0.38      0.32
   weighted avg    0.39      0.34      0.32

I want two select two best of them, and I know F1-score is a parameter for compare the classifiers because of its harmony between precision and recall.
So, at first I select classifier B for its best F1-score. for next, both A and C have a same F1-measure,
I want to ask how can I select between them?

f1-score combines precision and recall in a single figure. As both are pretty similar in A and C cases, f1-score is similar too.

Your choice depends on what it is less harmful in your categorization: false positives or false negatives.

I do recommend you to read the 3rd chapter of "DEEP LEARNING:From Basics to Practice" volume 1 by Andrew Glassner. There you have the three concepts (precision, recall and f1-score) described in a very illustrative way.


Finding value of theta in linear classification I have an examing coming up, and I'm practicing with exams from previous years. However, the answers to the questions are not provided unfortunately.
I'm currently doing the question below, and the intuitive answer I get is 2. However, this seems too easy. Am I correct?

Since it is an exam question, I will give you hints and leading questions instead of answers.
Note that $x^T w$ must be well-defined, and $x$ is 2-dimensional. What is the dimension of $w$?
How is bias incorporated into the model?
The margin satisfies $\widehat{y} = 0.$ What is the equation of the margin (as a function of x and bias)?
P.S. I expect a typo/vagueness in this question.


My corr() function in Python keeps resulting in an "ValueError: The truth value of a Series is ambiguous..." I am a very inexperienced programmer, this is my first question on the Data Science StackExchange, I sorry if it is formatted poorly or comes across as basic. For some strange reason, in Python, whenever I try to run a correlation function on the population density & total cases per million columns of my COVID-19 DataFrame (which I imported/read into Spyder as a csv), I keep getting the same long error message, namely, "ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()." regardless of whether I use the correlation function from Pandas or Numpy.
My first thought was that this error was caused by the presence of null values in those columns, so I used df.dropna(), then ran the correlation function again but I got the same "ValueError", so I have no idea what is going on here, I was able to run a correlation on those same columns just fine in RStudio which I am equally unskilled and inexperienced with.

You should look at the documentation.  You do not pass the column names as an argument.
subset_df = df[['col1', 'col2']]
subset_df.corr()

That should solve this for you.


For a square matrix of data, I achieve $R^2=1$ for Linear Regression and $R^2=0$ for Lasso. What's the intuition behind? For a square matrix of random data, N columns and N rows. I am fitting two models, linear regression and Lasso.
For the linear regression, I achieve a perfect score in train set, while in the Lasso I achieve a score of 0.
import pandas as pd
import numpy as np
from sklearn import linear_model

N = 100
X = pd.DataFrame(np.random.rand(N,N))
y = np.random.randint(20, size=N)

lin = linear_model.LinearRegression().fit(X,y)
lasso = linear_model.Lasso().fit(X,y)

print('Linear regression score',lin.score(X,y))
print('Lasso score',lasso.score(X,y))

Linear regression score 1.0
Lasso score 0.0
My questions will be:

Can someone give me an intuitive idea why for a square matrix, Lasso regression achieve different results from Linear regression?

What is the intuition behind the Lasso shrinking?

Note: In this case, the matrix is squared and the data/target is created randomly.

A few things going on here:

Your matrix is 100x100. So you have no degrees of freedom left in a linear model, which will cause $R^2=1$. See this post.
You use random numbers. Thus, they should make little sense in terms of explaining your dependent variable (it's basically noise). Since Lasso "shrinks" parameters which are not useful (and none is useful given perfect noise), all parameters are set to zero. This in turn also gives you an $R^2=0$. In this case the model consists only of an intercept (should be equal to 0.5), which is essentially the mean.

Change the matrix to have less columns than rows and you will get a low (but positive) $R^2$ for linear regression. Also look at the Lasso coefficients, they are all zero.

Have a look at the predicted values as well. You will see that the linear model does not work well. So the R-squared gives no indication about model fit in this case.

You can also try to change the random pattern to have some trend and you will find that Lasso does not shrink all coefficients to zero.

import pandas as pd
import numpy as np
from sklearn import linear_model

n = 1000
k = 5

X = pd.DataFrame(np.random.rand(n,k))
y = np.random.randint(2, size=n)

lin = linear_model.LinearRegression().fit(X,y)
lasso = linear_model.Lasso().fit(X,y)

print(lin.coef_)
print(lin.intercept_)

print(lasso.coef_)
print(lasso.intercept_)

print('Linear regression score',lin.score(X,y))
print('Lasso score',lasso.score(X,y))

Output:

[-0.00446436 -0.04509905  0.07870413 -0.00165025 -0.03951546]
0.5010650787133537
[-0. -0.  0. -0. -0.]
0.495
Linear regression score 0.003094912043144493
Lasso score 0.0


What's the difference between GridSearchCrossValidation score and score on testset? I'm doing classification using python. I'm using the class GridSearchCV, this class has the attribute best_score_ defined as "Mean cross-validated score of the best_estimator".
With this class i can also compute the score over the test set using score.
Now, I understand the theoretical difference between the two values(one is computed in the cross validation, the other is computed on the test set), but how should I interpret them? For example, if in case 1 I get these values (respectively as best_score_ and as score on the test set) 0.9236840458731027 0.8483477781024932, and in case 2 these other values 0.8923046854943018 0.8733431353820776, which one should I prefer and why? Why can the difference between the two values vary so much?

Gridsearch cross-validation can be used to learn the hyper-parameters of a prediction function. Consider that learning and testing of the model on the same data is a big mistake. The chance of having a perfect score but failing to predict anything useful on yet-unseen data (i.e., overfitting) is very high when using the same data for learning and testing. It is common practice when training a model to hold out a part of the data as a test set to prevent overfitting and measure the performance of the model. Also, note that the best hyper-parameters can be determined by grid search techniques and the score resulted from grid search should not be used as a criterion to measure the performance of the model. Please refer to this page for more information
That being said, best_score_ from GridSearchCV is the mean cross-validated score of the best_estimator. For example, in the case of using 5-fold cross-validation, GridSearchCV divides the data into 5 folds and trains the model 5 times. Each time, it puts one fold aside and trains the model based on the remaining 4 folds. Then, it measures the performance of the model based on the left-out fold.  Finally, it returns back the mean of the performance of 5 models as the final score.
Now, let's answer this question: what does the best estimator mean? it is the estimator that was chosen by the search or the estimator which gave the highest score (or smallest loss if specified) on the left-out data. GridSearchCV's goal is to find the optimal hyperparameters.  It receives a range of parameters as input and it finds the best ones based on the mean score explained above. Grid search trains different models based on different combinations of the input parameters and finally returns the best model or the best estimator. Hence,  best_score_ is the mean score of the best estimator.
It is notable that tuning hyperparameters with cross-Validation in the above context is one of the methods that helps you to prevent overfitting.
In your case, 0.8923046854943018 is the mean score of the best estimator. Let's call this score cross-validation score.  For your case, I would go with the second case, because in that case there is no overfitting and the cross-validation and test scores are almost the same. In the first case, the cross-validation is significantly higher than the unseen test score and there is overfitting. It means that the model works very well on the train data but not on the unseen data.


How to structure the output layer of an MLP that finds the quadrant of an arbitrary point in a rectangle? I'm trying to write a neural network that outputs the quadrant of a rectangle that an arbitrary point lies in. This rectangle has its upper left at {0, 0} and its lower right at {1, 1} (e.g. point {0.25, 0.25} should return 'upper left', and point {0.75, 0.75} should return 'lower right').
I'm confused about the structure of the output layer.
I've seen examples that suggest two nodes with Boolean "isLeft" and "isTop".
I've also seen examples that suggest four nodes with probabilities for UL, UR, LL and LR.
Is one or the other considered the proper structure in this case? Are both approaches valid?
I want to do this exercise as simply as possible, as I work to learn Multi Layer Perceptron. If there is a subtopic I need to focus on, please direct me to it.
Thanks in advance!

You need only two hyperplanes to solve this.
Thus you need two neurons in the hidden layer.
You can use two or four neurons in the output layer.
Both options result in the correct solution (theoretically).
You can use perceptrons. With perceptrons, the output is a boolean vecotor, not a probability.


Detect different "sections" of stationarity in time-series data? I'm working with some time-series data that is clearly non stationary. At first glance, it looks like there are 3 sections of my data. Within each section, the data have similar mean and variance, but between each section the variance of the data is quite different.

Is there a well known algorithm to detect and label different sections of stationarity in time-series data?

I hear my business forecasting professor in my head right now, suggesting the Cumulative Sum (CUSUM) Control Chart for this. Basically, this is a method borrowed from control/logistics studies that tracks a value over time. If a quantity is stable over time, then the series is "in control", and it falls "out of control" when the quantity starts to deviate systematically (that is, it stops looking like noise).

In the context of time series, you might compute a windowed stationarity metric from $t_{(0)}$ to $t_{(i)}$, then $t_{(i+1)}$ and so on until $t_{(n)}$, and create a CUSUM chart to watch for when the series seems to break stationarity. The same process might be applied to tracking ARIMA model parameters over time.

Some of this thinking has already been done, of course, and it seems that R Bloggers got a relatively nice result pretty quickly by using the changepoint package. You can dig into the details in the links below.

I applied this method to see if health insurance membership changed drastically across the implementation of the Affordable Care Act in the USA. This would answer the question:

  If we had models built on past membership data in 2012, would we have given an acceptable forecast for post-ACA membership?

References:

https://www.r-bloggers.com/changepoint-analysis-of-time-series/
https://cran.r-project.org/web/packages/changepoint/changepoint.pdf
https://www.itl.nist.gov/div898/handbook/pmc/section3/pmc323.htm
https://pdfs.semanticscholar.org/97d5/0c538be1d366f85826d440006d8de65a4f84.pdf


How to get out of local minimums on stochastic gradient descent? I'm not programming a neural network but I'm looking at it from a non-hands-on, theoretical point of view and I'm currently wondering how to escape a local minimum and how to get to a global minimum.

If you start at a point, for instance: (red)

When you compute the gradient of the error function and step in the direction of greatest descent, you'd end up in that immediate local minimum. AFAIK, you'd get stuck there. How do neural network trainers go about this? Do they start from a new random configuration of weights at each batch and see if the cost is smaller, or is there some way of immediately getting to the global minimum?

I've heard of a method resetting the learning rate to 'pop' out of local minimums but I'm not sure how that works when the gradient is 0. I've also heard that stochastic gradient descent is more reliable than gradient descent at finding global minimums, but I don't know how using the training data in batches rather than all at once allows it to steer around local minimum in the example, which is clearly steeper than the path to the global minimum behind it.

Stohastic gradient descent loss landscape vs. gradient descent loss landscape

  I don't know how using the training data in batches rather than all at once allows it to steer around local minimum in the example, which is clearly steeper than the path to the global minimum behind it.

So, stochastic gradient descent is more able to avoid local minimum because the landscape of batch loss function is different than the loss function of whole dataset (the case when you calculate the losses on all data and then update parameters). That means the gradient on the whole dataset could be 0 at some point, but at that same point, the gradient of the batch could be different (so we hope to go in other direction than the local minimum).

Neural network architecture and loss landscape

In order to escape the local minimum, your neural architecture can also help. For example, see this work: Visualizing the Loss Landscape of Neural Nets. It shows that skip connections can smoothen your loss landscape and, hence, help the optimizers to find the global minimum more easily.

Local minimums vs global optimum

Finally, there are some works suggesting that the local minimums have almost the same function value as the global optimum. See this question and answer.


Date prediction - periodic recurrence If I have some data regarding the occurence of an event on a certain date and some other variables regarding it (think fe.: I have data on which dates it rained, and some addtitional data like temperature, atmospheric pressure etc.), which is the most appropriate model for predicting on which day the event is going to happen again? Or, to be more precise, I'd like to predict the frequency of said event, to know in how many days it's going to occur again. I mostly use Python with the numpy, sklearn libraries, and I'm interested which of its models fits my use case best.

Thank you!

You should read that :

http://www.analyticsvidhya.com/blog/2016/02/time-series-forecasting-codes-python/

and take a look at that :

http://statsmodels.sourceforge.net/0.6.0/generated/statsmodels.tsa.arima_model.ARIMA.html

The endog argument is your time serie and the exog is your other data


Dealing with unseen data/categories in machine learning models for stream data I want to build a machine learning model (xgb and lgbm) that has to handle streaming data on a weekly basis. The models are trained on a bi-weekly basis. The data includes order information and I want to predict the likelyhood that the order will be delivered. The orders are entered in the system and a week after one can say if the orders were indeed delivered or not.
For nominal data like supplier I use pd.get_dummies() for transformation. However, lets say I receive my order data for the orders that arrive next week. There is a new supplier that the trained model doesn't know yet as the column supplier_new_unkown_supplier does not exist in the saved model parameters.
Does anyone know how to deal with such cases?

If you want to make an inference on an order where a categorical variable was not seen in the training data, you could train the model on a hash bucket representation of that variable.
If using tensorflow, you can leverage:
https://www.tensorflow.org/api_docs/python/tf/feature_column/categorical_column_with_hash_bucket
Or implement yourself.


How to interpret coefficients from logistic regression? I ran a logistic regression (statsmodel) on my data with 60 features using the below code

import statsmodels.api as sm
logit_model=sm.Logit(y_train,X_train_std)
result=logit_model.fit()
print(result.summary())

I was able to see that few variables had negative coefficient and few had positive coefficients.

Am I right to understand that irrespective of sign of coefficients, all the below variables are significant predictors that influence the outcome?

Or does negative coefficient mean they don't have any influence on the model outcome? But p-value is significant. Am a bit confused. Can you help in simple terms please

The below output shows the records whose p-values were less than 0.05

They are all significant but for certain thing.
What do I mean? You are predicting evidence, i.e. first column of the following picture:

In other words you have "linear regression part"+ instead of y you have evidence. So changing values of independnet variable X (positive or negative) will influence different binary class (0 or 1), hence different values are significant for different thing. (they add some info)


Using python and machine learning to extract information from an invoice? Inital dataset? DISCLAIMER: I have absolutely no background with machine learning/data science, and am unfamiliar with the general lingo of data science, so please bear with me.

I'm trying to make a machine learning application with Python to extract invoice information (invoice number, vendor information, total amount, date, tax, etc.). As of right now, I'm using the Microsoft Vision API to extract the text from a given invoice image, and organizing the response into a top-down, line-by-line text document in hopes that it might increase the accuracy of my eventual machine learning model. My current situation is strictly using string parsing, and this method works pretty well for invoice number, data, and total amount.

However, when it comes to vendor information (name, address, city, province, etc.) string parsing hardly ever works due to my script relying on the detection of an address (because addresses have a pretty distinct format compared to the rest of the information on an invoice). Tax information is difficult to parse because of the amount of numeric values that appear on an invoice.

So (and here's where I get lost)
I envision a machine learning model that will have an input of a single invoice image from a user, and it's output will be the extracted invoice information. I am currently looking into Azure Machine Learning Studio because the plug 'n play aspect of it appeals to me and it seems easy enough to use and experiment with. BUT I have no clue what the requirements are for an initial dataset! Should I just fill my dataset (in a CSV format btw) with the necessary information (invoice number, total amount, date, ...) from a bunch of sample invoices? If not, what other information should I include in my dataset? I was thinking x-y coordinate pairs of where the important information occurs on the image.

One last question related to this problem scope, which algorithm (regression, classification, clustering) could even "extract" (or help with it) information from the input text? As far as I know, regression predicts numeric values (i.e. 2 + x = 10 so 8 + x could be 16) and classification "labels" things (i.e. this image contains a tree, this one does not). I'm not too familiar with clustering, although I think it could be useful to identify the structure of the input text.

To summarize: what might be some features of an invoice that I can fill an initial dataset with to initialize a model? How could a clustering algorithm be used to identify the structure of an invoice? How could I utilize a regression/classification algorithm in this problem, and what inputs would these algorithms need to deliver meaningful results?

Any help/guidance/feedback would be appreciated to the moon and back. Sorry for my lack of knowledge, but this field is very interesting and need some help wrapping this all around my head. Thanks:)

I'm more of a beginner as well, but wanted to possibly help guide you towards next steps based on some of my experiences.

First, what you've illustrated here sounds a little along the lines of how a traditional EDI system does in Supply Chain & Procurement processing, where it will receive a document and enter the information into transaction-based operational systems to create records as data. I'm not entirely sure how those work, and if they can only extract standardized formats from documents but that's something that you could possibly look into with a few searches.

Now as far as creating a CSV data set, that is a great idea for testing the accuracy of your algorithm on a set of invoices to train your model. Training a model is pretty self-explanatory, but essentially you'd be using a supervised machine learning strategy, where the system actively uses a training data set where the correct answers are known. By comparing the models results to the data set, you could then know if the algorithm is appropriately retrieving the information. I'd recommend that you include the information the application should be getting from each invoice, and compare that to what the machine did retrieve. That way you effectively known the error of the model. (x,y) coordinates would work well for preliminary attempts to get a better feel for location-based assessment of the invoices, so that's a pretty good idea in my opinion as well.

Regarding algorithm types, this one will almost always be "it depends". Many times, I'll use a couple and compare the error that returns for each model to find the one that works the best for what I'm predicting. That said, Clustering can be useful for continuous, numerical data and "grouping" items based on the location of the coordinates. I think clustering could be useful if you're breaking down the coordinates of the information on an image, but wouldn't be of much use to gather and extract the text from the invoice. And to not go too deep, K-Means would probably not be great at all because the locations for the cluster centroids is entirely randomized and you must also declare a number of n clusters, which would be impracticable for the purpose at hand.

I'd recommend looking into the different types of Neural Networks(NN's). These are used extensively to take images of some kind, and break them down to best estimate the contents of an unknown input. It's the type of algorithm Google uses to do image searches and things along those lines. It's also the type of software that tells an autonomous vehicle that the sign ahead is a stop sign. There are different types of NN's out there, and it can get pretty overwhelming pretty quickly, so I recommend looking into basic videos to explain it. If you do decide to go this route, Google Cloud has a pretty solid platform with there google.vision and other machine-learning-in-a-box libraries that are really easy to import and play with.

as far as inputs, define the information that you want extracted, and record that data for use of testing and training as I mentioned above. Then play around with a few methods in sciket-learn and TensorFlow!

Unfortunately, I can't give you a straight-forward answer on the many trade-offs with different algorithms or inputs, but I hope that this gave you a few ideas about next steps or provided you with a better understanding of how you can begin the project. Good luck!


when will the mlp give constant prediction? I have a regression task.(to predict price for finanical market)
I build a mlp to do the regression.
I found mlp will stop at giving a constance prediction. which i think it's useless.
Does this mean, i cant do a better prediction than constant value? but i tried catboost, it give a meaningful prediction.
So, if the mlp give the constant value, is there any common reason? and how to improve that?

This means that your features do  ot have predictive power to estimayevthe target. Hence the MLP has learnt to predict the mean of the target distribution so as to minimize the loss.
Try checking if the constant value is indeed the mean of the target variable.


Python: How to test a RandomForest regression model for Overfitting? I'm a beginner in this area so maybe I'm doing something wrong here.
I'm using RandomForest for a regression model and wanted to see if my model is overfitting. Here is what I did:
EDIT:
I use GridSearchCV for hyperparameter tuning:
rf = RandomForestRegressor(random_state=0)
rf_params = {'n_estimators': [100, 500, 1000], 'max_depth': [3, 6, 9, None],
             'min_samples_leaf': [2, 5, 10], 'max_features':['auto', 'sqrt']}

gs_rf = GridSearchCV(rf, rf_params, scoring = 'neg_mean_absolute_error', cv = 10)

gs_rf.fit(x_train, y_train.values.ravel())
b = gs_rf.best_params_

Then I create a RandomForestRegressor with those parameters:
RF = RandomForestRegressor(n_estimators=b['n_estimators'], max_depth=b['max_depth'], min_samples_leaf=b['min_samples_leaf'], max_features=b['max_features'], random_state=0)

Then I fit the model using the train dataset:
model = RF.fit(x_train, y_train.values.ravel())

Then I predict with the test dataset:
y_pred = model.predict(x_test)

Then I did the exact same with x_train instead of x_test:
y_pred = model.predict(x_train)

Here are the results that I achieve:
Test Data:
MAE: 15.11
MAPE: 26.98%

Train Data:
MAE: 6.17
MAPE: 10.97%

As you can see there is a pretty significant difference.
Do I have a big problem with overfitting or am I doing something wrong when using x_train to predict?
Any help is much appreciated!

Predicting on x_train and comparing the error with the prediction error of x_test is a good approach. Here, it seems that you are overfitting since you model is better to predict what it learned than new data. It would be a good thing to use a cross-validation approach to ensure that the problem does not come from your split.
Here are some leads I would follow:

Apply a cross-validation to evaluate your model
If using cross-validation in hyperparameters tuning, it is important to keep a validation dataset to evaluate your final model after the tuning. Doing that, you ensure to test your model on unseen data.
Have a look at your hyperparameters. You could for instance decrease max_depth, that may lead to reduce overfitting.
Plot your outputs: plot the excepted values and the predicted ones. MAE is the Mean Absolute Error, so you may discover unexpected effects (such as your model having high troubles to predict specific range values and not others)

Hope it helps.
Note
I would not say overfitting is a problem, rather its consequences could be a problem. Overfitting occurs when the model learns too much from the training data and so does not generalize well. If you constructed your pipeline well, the metric that should lead your performance is the score you choose (in your case MAE)


How to find the relationship between two categorical variables? I'm building a model and trying to get the relationship between two multi-level categorical variables.

For example, I want to know the relationship between race and likelihood of graduate from collage, we have 5 races and YES or NO for graduation. How to find the which race have higher likelihood or which have lower?

Thanks

Traditional statistics like Chi-squared tests and Cramer's V can be used to determine relationship between two categorical features.


Is there a way to overlay stem plot over line plot in python? Date        Day_Perc_Change No. of Trades
2017-05-15  0.000000        40593
2017-05-16  -0.019184       22027
2017-05-17  -0.008550       15709
2017-05-18  -0.017699       47159
2017-05-19  -0.021748       85921

I visualized "Day_Perc_Change" in stem plot and "No. of Trades" in line plot. But now i want to compare these two plots by overlaying them on each other.. How do i visualize that?

x = df["Date"]
y = df["Day_Perc_Change"]

plt.stem(x, y, bottom=0)
plt.show()

df.plot(x="Date", y="No. of Trades", figsize=(10,8))

i Want to overlay these two on each other to compare them..is there a way to do that?

Part 2: post trying the answer mentioned below:
After trying the below code by @ElBurro
i get the following output

The expected output should look like this

I think what you are looking for is called twinx. I just used one of the examples given and slightly modified the input parameters:

import numpy as np
import matplotlib.pyplot as plt

date_name = "Date"
stem_name = "Day_Perc_Change"
plot_name = "No. of Trades"
t = df[date_name]
data1 = df[stem_name]
data2 = df[plot_name]

fig, ax1 = plt.subplots()

color = 'tab:blue'
ax1.set_xlabel(date_name)
ax1.set_ylabel(stem_name, color=color)
ax1.stem(t, data1)
ax1.tick_params(axis='y', labelcolor=color)

ax2 = ax1.twinx()  # instantiate a second axes that shares the same x-axis

color = 'tab:red'
ax2.set_ylabel(plot_name, color=color)  # we already handled the x-label with ax1
ax2.plot(t, data2, color=color)
ax2.tick_params(axis='y', labelcolor=color)

fig.tight_layout()  # otherwise the right y-label is slightly clipped
plt.show()


Vanishing gradient problem In a neural network, does gradient vanish during a great number epochs as well, rather that only vanishing through different layers?

Gradient vanishing means drastic decrease of gradients when backpropagating through many layers. This problem is also known for recurrent neural networks as they are mathematically equivalent to very deep networks.
That is true that gradients decrease during training. However, that is not gradient vanishing. That is the sign that the network has been trained, i.e. that is what you normally expect in the end of network training.


Should I standardize first or generate polynomials first? Recently I am dealing a classification problem with some algorithms, say logistic regression.

When I preprocess my data, I standardize all my features and then generate polynomial features based on them.

from sklearn.preprocessing import PolynomialFeatures, StandardScaler

and I do

# features is my entire features dataset, labels excluded
features = poly.fit_transform(features)
features = std.fit_transform(features)

After finishing training my model, the accuracy is, say about 80%. Then I invert the two line of preprocessing code to

features = std.fit_transform(features)
features = poly.fit_transform(features)

I have read this post but it seems the answers is not strong enough to help me figure it out.

Should I standardize my data first or generate polynomials from original features first?

Depends on what are you using the standardization for and how the features relate to your problem, but I guess for most situations you’re better off standardizing them afterwards.

If you are looking for some standardization in order to make optimization of some algorithm faster (e.g. neural networks rather than logistic regression), it might be a better idea to standardize afterwards, as otherwise it defies the purpose.

If you are doing it in order to introduce some regularization, then standardizing them to 0-1 first will make the polynomials smaller overall and thus more likely to be given weights compared to the original features, which is probably not what you want.


Complex Chunking with NLTK I am trying to figure out how to use NLTK's cascading chunker as per Chapter 7 of the NLTK book. Unfortunately, I'm running into a few issues when performing non-trivial chunking measures.

Let's start with this phrase:

"adventure movies between 2000 and 2015 featuring performances by daniel craig"

I am able to find all the relevant NPs when I use the following grammar:

grammar = "NP: {<DT>?<JJ>*<NN.*>+}"

However, I am not sure how to build nested structures with NLTK. The book gives the following format, but there are clearly a few things missing (e.g. How does one actually specify multiple rules?):

grammar = r"""
  NP: {<DT|JJ|NN.*>+}          # Chunk sequences of DT, JJ, NN
  PP: {<IN><NP>}               # Chunk prepositions followed by NP
  VP: {<VB.*><NP|PP|CLAUSE>+$} # Chunk verbs and their arguments
  CLAUSE: {<NP><VP>}           # Chunk NP, VP
  """

In my case, I'd like to do something like the following:

grammar = "MEDIA: {<DT>?<JJ>*<NN.*>+}
           RELATION: {<V.*>}{<DT>?<JJ>*<NN.*>+}
           ENTITY: {<NN.*>}"

It occurs to me that a CFG might be a better fit for this, but I only became aware of NLTK's support for this function about 5 minutes ago (from this question), and it does not appear that much documentation for the feature exists.

So, assuming that I'd like to use a cascaded chunker for my task, what syntax would I need to use? Additionally, is it possible for me to specify specific words (e.g. "directed" or "acted") when using a chunker?

your grammar is correct!

grammar = """MEDIA: {<DT>?<JJ>*<NN.*>+}
           RELATION: {<V.*>}
                     {<DT>?<JJ>*<NN.*>+}
           ENTITY: {<NN.*>}"""

by specifying

RELATION: {<V.*>}
          {<DT>?<JJ>*<NN.*>+}

you are indicating that there are two ways to generate the RELATION chunk i.e. {<V.*>} or {<DT>?<JJ>*<NN.*>+}

so

grammar = """MEDIA: {<DT>?<JJ>*<NN.*>+}
               RELATION: {<V.*>}
                         {<DT>?<JJ>*<NN.*>+}
               ENTITY: {<NN.*>}"""
    chunkParser = nltk.RegexpParser(grammar)
    tagged = nltk.pos_tag(nltk.word_tokenize("adventure movies between 2000 and 2015 featuring performances by daniel craig"))

    tree = chunkParser.parse(tagged)

    for subtree in tree.subtrees():
        if subtree.label() == "RELATION":
            print("RELATION: "+str(subtree.leaves()))

gives

RELATION: [('featuring', 'VBG')]


Binary classification model for unbalanced data I have a dataset with the following specifications:

Training dataset with 193,176 samples with 2,821 positives
Test Dataset with 82,887 samples with 673 positives
There are 10 features.

I want to perform a binary classification (0 or 1). The issue I am facing is that the data is very unbalanced. After normalization and scaling the data along with some feature engineering and using a couple of different algorithms, these are the best results I could achieve:

mean square error : 0.00804710026904
Confusion matrix : [[82214   667]
                   [    0     6]]

i.e only 6 correct positive hits. This is using logistic regression. Here are the various things I tried with this:

Different algorithms like RandomForest, DecisionTree, SVM
Changing parameters value to call the function
Some intuition based feature engineering to include compounded features

Now, my questions are:

What can I do to improve the number of positive hits ?
How can one determine if there is an overfit in such a case ? ( I have tried plotting etc. )
At what point could one conclude if maybe this is the best possible fit I could have? ( which seems sad considering only 6 hits out of 673 )
Is there a way I could make the positive sample instances weigh more so the pattern recognition improves leading to more hits ?
Which graphical plots could help detect outliers or some intuition about which pattern would fit the best?

I am using the scikit-learn library with Python and all implementations are library functions.

edit:

Here are the results with a few other algorithms:

Random Forest Classifier(n_estimators=100)

[[82211   667]
[    3     6]]

Decision Trees:

[[78611   635]
[ 3603    38]]

Since you are doing binary classification, have you tried adjusting the classification threshold? Since your algorithm seems rather insensitive, I would try lowering it and check if there is an improvement.
You can always use Learning Curves, or a plot of one model parameter vs. Training and Validation error to determine whether your model is overfitting. It seems it is under fitting in your case, but that's just intuition.
Well, ultimately it depends on your dataset, and the different models you have tried. At this point, and without further testing, there can not be a definite answer.
Without claiming to be an expert on the topic, there are a number of different techniques you may follow (hint: first link on google), but in my opinion you should first make sure you choose your cost function carefully, so that it represents what you are actually looking for.
Not sure what you mean by pattern intuition, can you elaborate?

By the way, what were your results with the different algorithms you tried? Were they any different?


"DeepTree" network with binary classifiers at each node I need to do a project where I build a "DeepTree Network". Let's say I  have a 3-classification problem with classes A, B, and C.

My tree should then look like this:

first node: CNN binary classifier classifying between class C, and superclass A+B
second node: CNN binary classifier classifying between class A and B

Does anyone know if there are any existing papers on this subject? Or any implementation? The leaves don't need to be CNNs necessarily, any sort of tree implementation like this in TensorFlow would help me a lot.

You can try the SVM with your decision function :

https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html#sklearn.svm.SVC

it can handle muilt classfication problem with default ovr decision function

if you stick with tensorflow you can only try linver svc : https://www.tensorflow.org/api_docs/python/tf/contrib/learn/SVM


How can one generate future forecasts from probabilistic events? I have an event "whether an item sold will be returned or not" which I can predict with a certain probability based on information gathered at the time that the purchase occurs (product features, customer information, time and place, etc). So:

P(Return | transaction information) = x% for a specific unit sold

I also have a historical time series of total units sold for that item, and a future forecast of sales of that item over the next few weeks. Assuming I gave the relevant transaction data for each historical sale that occurred, is there away to generate a future forecast from the return probability, so that I can state with some confidence that I will get 15% total returns on the item next week, 10% the week after etc?

If you have historical data on percentage of return for few weeks after the sales, I believe you can do it. It will be a multiple-output regression problem e.g. you will have to formulate the target values (i.e. output matrix) in a way so that %-return in first week will be in column one, %-return in second week will be in columns two and so on. After this, plug in all your input variables and run a multivariate multiple output regression algorithm. Such an algorithm on scikit-learn can be found (sklearn.multioutput.MultiOutputRegressor: here)


Correlations, p-values and features selection By using correlation matrix, I got some results:
Count_words          -0.098857
Count_numbers        -0.008305
Count_symbols        -0.025853
Count_question       -0.031649
Count_equal           0.224223
Count_characters      0.09

I used this line of code (in case you are familiar with Python):     df.drop("Target", axis=1).apply(lambda x: x.corr(df.Target))
If I understand correctly, the above results should suggest that there is not correlation between the variables considered.
Since I would like to add the above variables (or some of them) in a model already built with other features (textual), I would like to know if I can include all of them based on the fact that they are not correlated to each other and that the p-value is less than 0.05. My doubt is  if the above results do not make sense and do not suggest that these variables can be used in the model.
I hope you can give me some advices on that. Thanks

The fact that a feature has low correlation with the target variable shows that it's not a good indicator on its own, but that doesn't mean that it can't be useful for the model when combined with the other features.
The only way to know if these features are useful is to use them to train a model, then evaluate on a validation set and see if it improves performance.

the p-value is less than 0.05

Is this the result of a correlation significance test? It depends on the test but in general a p-value lower than 0.05 means that there is a significant difference, i.e. in this case it probably means that the correlation is truly not zero. Anyway imho this wouldn't prove anything with respect to using these features or not.


Once a predictive model is in production, how it can be evaluated? I have a data science project, predicting customer's next purchase day. Customer's one year behavioral data was split to 9 and 3 months for train and test, using RFM analysis, I trained a model with different classifiers and the best one's result is as follow:
Accuracy of XGB classifier on training set: 0.93
Accuracy of XGB classifier on test set: 0.68
This is my school's project, and I was wondering, in real world projects, how can we evaluate a model's performance after it's on production. How can I measure how successful my model was? What if the performance measures in production are much lower than my test result?

This is in fact a very good question.
The answer is simple, but depends on the case.
In general, what we do after pushing a model to production we apply an audit process. Let me explain: in reality machine learning models that are being pushed to production are pushed to replace another process (e.g, manual process- this is the case of automation). At the beginning everything predicted by machine learning model are audited through another process (e.g, manual), we call this stage the pilot stage. By comparing the model performance to the manual process we establish the quality of the model. Once we are happy, we start reducing the audit percentage from 100% to 5% or so ( there is some math behind what should be the audit percentage). This audit will never go away and will always be used to measure the performance of the model and to establish ground truth data for new samples that can be added to the training set.
In fact, training models in theory is something and using them in production is something else. It is really a complex process. Just to mention: we also like to implement some protection mechanisms to protect the model. For example data drift detection, uncertainty detection and so on.


Can I use categorical data and Decision Trees to regress a continuous variable? Is there a way to take a set of data that consists of discrete values and predict a continuous value? Take for instance data that looks like:

sample matrix of jewel data
 color |  size | shape
['red' ,'large','square']
['blue','small','circle']
['blue','small','square']

sample array of price labels
[9.99, 7.00, 6.37]

Can I do Decision Tree Regression on this to predict the price of a jewel with a given set of features? What if some of the data is continuous? Also is there any way I can/should pre-process the categorical data other than onehot encoding?

Yes, most software implementations of trees will allow you to predict a continuous target variable with all binary predictors. This is because the predictors are only used as splits, and the prediction comes from the average value at a given terminal node. The predictions will not be truly continuous across all terminal nodes in the same way that linear regression is continuous, but in practice, this is generally not a problem. If your tree is under-fitting (not continuous enough) you can always add more terminal nodes. Also, one-hot encoding should be sufficient.


suitable algorithms for very high dimensional ”binary pattern" recognition? I have a dataset consisting of 3000 binary features and one binary ouput. subset of these binary features form binary patterns. these subsets could be neighbouring features or from different regions (which one is the case is unknown). also these patterns can show up anywhere within the feature vector.
any ideas what algorithms are best suited for a problem like this? I have done some searching and seem like logical neural networks, CNNs and HMMs be good candidates. also feature reduction is something I am currently looking at. data points are not images but rather 1D arrays.

For feature selection, you may want to look at PCA and I would suggest that you try Naive bayes Classifier and Random Forests.


Best CNN architecture for binary classification of small images with a massive dataset The title has it all...

Any tip is welcomed.

Should I use a very deep convolutional neural network ?

Use a large amount of filters ?

Parallel layers ?

Dataset examples:

1) "Good"

2) "Bad"

It all depends on the dataset, there is no single model can be the best.

I would prefer to try a transfer learning Resnet34 or resnet50 with a custom last layer for the number of class of classification.


Similarity between bordermode and zero padding in Keras In Keras, the border_mode = 'valid' doesn't zero pad the input. Thus, we subsequently get an output feature map that is not the same size as the input. Likewise, setting border_mode = 'same' gives output feature maps as the input. My question is if we set border_mode = 'same', should we also perform zero padding using the zero_padding function?

I think I found the answer to this here https://github.com/fchollet/keras/issues/1984. If the stride is 1, border_mode = 'same' does the job of padding to ensure that the output feature maps are the same size as the input.


Should unique vectors (SIFT descriptors) be used in K-Means Clustering? I'm doing image classification by extracting SIFT features, clustering them and then finding BOVW histogram and classifying.

I have around 180 training images from which I'm extracting SIFT descriptors. I need to cluster these features using k-Means clustering. Now, among the SIFT descriptors, some are duplicate. Before applying K-Means, should I remove these duplicate vectors or should I not? Or does it not make any difference?

Thanks!

It does make a difference.

If you have many duplicates, you can merge them into weighted vectors. It's straightforward to add to existing code.


Tensorflow Conv3D with variable input size I have a hypotethical question: Is it possible to train Conv3D with variable input size?
Sample dim = Length x Width x Depth ; Depth are fixed per each samples, let's say 500. However Length x Width can vary, e.g.:
Sample 1 = 50 x 4 x 500
Sample 2 = 7 x 7 x 500
Sample 3 = 10 x 13 x 500
.....
Sample n = 5 x 32 x 500

These are for classification problems, the next class could have a different sample size, e.g.:
Sample 4 = 6 x 8 x 500 (from class 2)
Sample 5 = 3 x 32 x 500 (from class 2)
....
Sample m = 10 x 11 x 500 (also from class 2)

Thanks in advance.

In Keras, you should specify the shape of your inputs and that shape should be fixed. Then you probably have to somehow resize all of your samples to a fix size $m \times n \times 500$.

You can use similar ideas in image resizing (e.g. interpolation, Nearest-neighbor interpolation, Bilinear and bicubic algorithms, Box Sampling, ...).


Displaying date in SAS I am trying to display a date in my result after running the below program in SAS. It runs properly but in the sas data table under DOB column I don't get anything except a period .

Below is my code what am I doing wrong?

data sample;
Input ID name $ Dob;
Format DOB mmddyy10. ;
datalines;
1 abc 22jan1996
2 xyz 25aug1996
;
run;
Proc print data = sample;
run;

Also I would like to know what does a period means at the end of this line before the semicolon:

Format DOB mmddyy10. ;

Try the following:

data sample;
Input ID name $ Dob;
informat DOB date9. ;
format DOB mmddyy10. ;
datalines;
1 abc 22jan1996
2 xyz 25aug1996
;
run;
Proc print data = sample;
run;

I have added the informat line. The problem with the original code is that you have not told SAS what is the format that DOB will come in. In this case: 22jan1996 is of date9. format, so I added the informat telling SAS that the data will come in this way.

The format DOB line tells SAS to display the data as mmddyy10. which makes 22jan1996 look like 01/22/1996.

Lastly, the . before the ; in the format/informat lines specifies that it is a format type. There are character formats: $8. or numeric formats Best12. etc...


Null-invariant measures of association in R A usual way to find association rules in R is the "arules" package, which easily let's use calculate some rules based on the apriori algorithm.

However, for the data i'm using, I have a lot of NULL cases (baskets where no product A o B is present). This means that I need to calculate some null-invariant measures (kulcynski, for instance).

Does anyone know of any package or workable code that let's me implement this as opposed to writing from scratch the entire algorithm?

interestMeasure function from arules v1.5-5 let’s you find additional measures and also some invariant ones. Here’s the link to the R documentation: https://www.rdocumentation.org/packages/arules/versions/1.5-5/topics/interestMeasure


Algorithms inherently supporting multilabel classification In the documentation of sklearn, it says that several algorithms inherentrly support multilabel classification, such as RandomForest or MLP : https://scikit-learn.org/stable/modules/multiclass.html

Does it come from the implementation or the structure of the models? Moreover, how does it works for these algorithm? Is is a One-vs-the-Rest strategy or something else?

Thanks.

If the algorithm inherently supports multi-label classification, then it's usually an implicit feature of the algorithm rather than an implementation detail.

For example, MLPs inherently support multi-label classification because the output layer has a perceptron for each class, and each of these perceptrons output a probability for that class.  The vector of outputs will predict an example's membership among all labels.  Similarly, the the leaves of a Random Forest (or any other tree-based algorithm) can contain arbitrary-length vectors that describe the probability of the example belonging to each label.

The one-vs-rest strategy is used generalize binary classifiers (e.g. logistic regression) to multinomial problems.  Multinomial problems are distinct from multi-label problems.

Edit:

Clarified my answer w.r.t. the differences between multinomial and multi-label problems.


Fixed-Width vs Adaptive Binning I have some continuous variable in my data that I wish to apply binning for. The values range from 0 to 800 but I got motivated by the fact that the data distribution was left skewed as you could see in the following figure:

However, I have read this amazing blog about binning data this whereby the author claims that adaptive binning is better than fixed-width binning. I understand the idea behind this because the some of the bins that we define in the fixed-width approach may have too little data distribution in comparison to other bins, which won't be a fair game to play whereby the adaptive approach, motivated by the idea of quantiles is better. Are there any more arguments, or more in depth analysis, of this hypothesis ?

I took a look at that link.  It's pretty informative.  As you already know, The cut function is used to specifically define the bin edges.  There is no guarantee about the distribution of items in each bin.  In fact, you can define bins in such a way that no items are included in a bin or nearly all items are in a single bin.  The qcut function is slightly different.  The qcut function tries to divide up the underlying data into equal sized bins.  The function defines the bins using percentiles based on the distribution of the data, not the actual numeric edges of the bins.  In conclusion, if you want equal distribution of the items in your bins, use qcut . If you want to define your own numeric bin ranges, then use cut.

See the link below for more details.

https://pbpython.com/pandas-qcut-cut.html


What does it mean if a high or low number of my componenets describe a percentage of the cumulative explained variance? In the following code run after PCA i can see that X number of components explain Y % of cumulative explained variance (CEV).
I would like to know

1- What percentage of the CEV is typically acceptable e.g. 95% or 99%? (Or is it a case by case basis?)

2- If 20 out of 200 components explain 95% of the CEV, what does this say about my data, what about when 200 out of 200 explain 95% ?

pca = PCA().fit(X_train)
plt.plot(np.cumsum(pca.explained_variance_ratio_))
plt.xlabel('Number of components')
plt.ylabel('cumulative explained variance');

I always think that Principal component analysis is a very interesting tool. Many know its applications, few knows exactly what is going on.
If you care about the math, you need to study eigen values and eigen vectors, but here let me explain pca in a very simple way. Imagine that you have a point in a 2-d space (x, y), this same point can be represented by using another two values, let us say (r, theta), where r is the distance from the origin and theta is the angle with respect to the +v x-axis.
Given any point (x,y), you can find the point (r, theta) corresponding to the point (x,y). You should be able to move between the two coordinates easily. Now let us ask the following question:
If I decide to ignore the theta dimension (for example set it to zero) how many points in the (x,y) you still can recover? All the points across the +ve x-axis can be recovered correctly. Now what about the r dimension, if we set it to zero, we can recover only one point which is the origin point.

Principal component analysis helps find this new coordinate system depending on the data, number of dimensions and variation across each dimension. The higher variation across dimension the more important to reserve this dimension. Very important to know that pca proposes both new coordinates and a measure of how much each dimension is important to recover the points in the original coordinate.

For example, let us take these points,
(1,1)
(2,2)
(3,3)

Both x and y dimensions are important and they experience the same variation.
If we transform, these data points into a new coordinate system (k,m), such that
K=x.  And.  M=0
Then. K is very important 100% and m is not important at all 0%
You can recover the original points by using, x=k. And y=k. Note that the variation across the m dimension is zero and it is high across the k dimension.

How much you want to reserve depends mainly on the application, usually > 95% is good.
For the second question, it is possible to recover less than 100% when you use less dimensions, but when you use 200 of 200 you should be able to recover 100%.


What is auxiliary loss in Character-level Transformer model? I am reading Character-Level Language Modeling with Deeper Self-Attention from Rami Al-Rfou. In the second page, they had mentioned about Auxiliary Losses which can speed-up the model convergence and as an additional regularizer. They said they had 3 kinds of auxiliary losses:

Auxiliary losses at intermediate sequence positions
Auxiliary losses from intermediate hidden representations
Auxiliary losses at target positions multiple steps.

However, I cannot find any information or reference explaining auxiliary losses. I would like to know:

What is auxiliary losses? Is it an unsupervised prediction model using part of the data?
How can I calculate the auxiliary losses?

Auxiliary losses are additional terms added to your global loss to induce learning further upstream in your model and can be seen as a way to combat vanishing gradients.

An early reference would be how Google's Inception model uses auxiliary classifiers. The general idea is that you take an intermediate output of your model and use it as the prediction in a separate loss function. So during training, your model is acting as both a global model (all parameters subject to that final global loss) and a model composed of sub-models with specific terms applying exclusively to your auxiliary tasks.

In practical terms, you can follow this example. If the link ever dies, here's a generic view:

# let's say we have two auxiliary losses, and thus two auxiliary outputs
final_task_prediction, predictions = model.forward(x)

global_loss = loss(final_task_prediction, y)
intermediate_loss_1 = aux_loss_1(predictions[0], aux_y_1)
intermediate_loss_2 = aux_loss_2(predictions[1], aux_y_2)

# can weight auxiliary losses however we choose
total_loss = global_loss + .4*intermediate_loss_1 + .8*intermediate_loss_2
total_loss.backward()

As an example, say we have an input sequence ABCDEF. To quickly run through the auxiliary losses in the Al-Rfou et al. paper

Multiple positions works as an auxiliary task of "given A, predict B, then given B (and our internal memory of A), predict C" (etc)
Intermediate hidden representations works the same as multiple positions, except intermediate layers also get their own loss terms
Multiple predictions breaks the target prediction sequence into multiple subsequences, so our auxiliary task is "given A predict B,C ; given B, predict C,D" (etc)

Hope this helps!


What is the outcome of a Cox regression in xgboost? I am creating a model using xgboost. Regarding its parameters, its objective is survival:cox and its eval_metric is cox-nloglik. The output Y ranges from -800 to 800. However, the predicted values are way to large (in range from 10^3 to 10^13). Why is this? What is the outcome of a Cox regression in xgboost?

In the documentation, you can find that the predictions are returned on the hazard ratio scale:

  survival:cox Cox regression for right censored survival time data (negative values are considered right censored). Note that predictions are returned on the hazard ratio scale (i.e., as HR = exp(marginal_prediction) in the proportional hazard function h(t) = h0(t) * HR).

In other words, in a Cox proportional hazards rate model we have:

$h(t)=h_0(t)×X$

where $X$ in a traditional linear model is of the form $exp(b_1x_1+b_2x_2+...+b_px_p)$, and $h_0(t) = $ baseline hazard rate function. In the xgboost case with tree base learners, $X$ is the exponentiated prediction generated by the weighted sum of individual trees trained on the negative gradient of a loss function designed for survival data in particular.

In other words, the predicted values are not times of failure.

I find the xgboost documentation on the survival:cox setting to be extremely sparse and not well described. To my knowledge, I do not believe there exists an internal method to derive $h_0(t)$ for example.


Is this cluster analysis / prediction? I have a series of seemingly random data dripping in one value at a time through time. Although it appears to be random, the data forms clusters when certain attributes are analysed which the charts show. I'm trying to avoid the fallacy of seeing patterns that are not there, but across several datasets the data does indeed appear to form clusters through time. I'm interested in trying to:

detect the clusters of either the blue squares and / or the white squares in the data I have
ignoring data which are outliers
predicting the future shape of the clusters as data is dripped in one column at a time. (There appears to be a certain momentum through time of the cluster shapes where the current cluster areas of the new data is in nearby region of the preceding clusters)

They appear to take on regular semi-predictable shapes which (to my eyes at least) follow a momentum from the data before. How could I detect this?

My questions are:

is this cluster analysis, and
what would be the best way to detect the white squares and / or the blue areas. I'm guessing the blue areas would be the easiest to cluster?
the white squares appear fairly close to the previous y-values and may continue at a certain angle up or down in value. How is the best way to predict this?

I think your process is not really a cluster analysis problem, but instead a time series analysis problem. You have some process that is measuring a value ($Y$ variable) at different points in time ($X$ variable). I would suggest using a Shewhart Control Chart or something similar to understand the process that you have. You should also do analysis on the distribution to ensure it is normal, or use the CLT to enable the data to move towards normality.


What makes you confident in your results? At what point do you think you can present your work to tech illiterate superiors? I understand that the models are only as good as the data you get, and bad design can generate really bad data. Nonrandom sampling, unbalanced/incomplete designs, confounding, can make data analysis really hard.

At what point should one be confident that they ran a useful model? Do you just do a cross-validation with a training/test dataset and call it a day? Obviously "all models are wrong, some are useful" but at some point the tradeoffs with excluding too many parameters by LASSOing and strange transformations by getting BIC down become glaring.

tl;dr at the end of the day what makes you go "I did the right thing for my company/project, and this should work"

Hey Welcome to the Site!

What you are saying is right, Data Science din't reach to the stage where it has some standard methods for achieving this(standard procedures, don't know we would be able to reach that stage in near future). But we have some general standards like:

Forecasting: ETS,ARIMA,SARIMA etc
Prediction: Linear Regression,Random Forest, GLM, Neural Network etc
Classification: Logistic Regression, Random Forest etc

When you go to granular level it is hard to generalize, as every business problem is different and one single method cannot be used for solving all the business problems.

So, to answer the next question how do you get confidence that the outcome is good enough, I assume that you have heard about RMSE,MAPE and many more for predictions and confusion matrix for classification problem. We use these metrics to see access the models performance, for example if you are trying to classify whether the given cell is cancer cell or not, there are 100 records in which 90 are non-cancer cells and 10 are cancer cell, your model gives 99% accuracy but could classify 5 out of 9 literally 55% of the total in such scenarios you need to look cannot use accuracy, you need to use F1 score etc. As you were asking about a model right, all models are not useful. True not all models built are going to go for production level, you would choose the best one and productionize it. You can re-train your model on a basis(Daily, Weekly,Monthly based on business requirement). Would you call it a day off post completion of validation? I wouldn't, I would go to the Subject Matter Expert present him the results ask him/her for their insights, if they both are inline then I would do a Beta testing on some actual data and then productionize it.

Now to address your last question, There is no standard saying that this is good or bad, if it works for you, your Business then that is a Good Model. To convenience your mangers and subject matter(Data) experts, you need to dig deep into the data try all different scenarios ask as many questions as possible. Try understanding the data very well. So, you can answer Business Questions with data supportive answers(This is possible only when you are well-worse with data). As they are very good with business they would be asking questions with respect to business, you need to be ready with all such scenarios by understanding business and data well.

Finally, I do have a feeling like you do. I did alot of things but nothing worked but you shouldn't be unhappy as you understood that these are the ways which would lead you to Unsuccessful results(best example is Thomas Alva Edison has used 1000 diff metals before using Tungsten to make a bulb). Similarly all the methods which we have tried are different steps you have tried to get the solution. My funda is, did I try something different/new everyday or not. Crucial part of this process is, maintenance of clear documentation at each and every step. Which would come in handy in the near future.

Anything in R&D is never a waste it is just an other try or experiment, so your work is never waste. Your are trying to build a solid base for the bright future of your company.


How to quantify the performance of the classifier (multi-class SVM) using the test data? I am working on a traffic sign recognition code in MATLAB using Belgian Traffic Sign Dataset. The dataset consists of training data and test data.

I resized the given images and extracted HOG features using the VL_HOG function from VL_feat library and then trained a multi class SVM using all of the signs inside the training dataset.

I am particularly interested detecting 8 signs. Question 1: Am I suppose to train the model using only the training set of these 8 signs or train the model with the entire training dataset and ignore the signs that I am not interested in at the detection stage?

Upon training the multi-class SVM, I want to test the classifier performance using the test data. Question 2: Can someone guide me through how to do this? I found something similar to what I am looking for available here. It would be great if I can obtain quantitative data such as a confusion matrix in percentage form.

Thanks!

Machine Learning process usually follows three steps:

Training (on training data): this data is used in the learning stage. The model optimally adjusts the weights in order to minimize the loss function over this dataset.
Evaluation (on evaluation data): it serves to fine-tune the model, for instance you can decide if your model overfits
Testing (on test data): tells you about the inference power of your model

The intersection of these three datasets has to be zero, and they should come from the same disttibution.

Recalling that, the first thing is the naming convention for your two datasets:

If training and test datasets come from the same distribution (I guess yes, since they are all Belgian sign traffics) you can name it Training and Validation datasets.
If the same condition holds, you can perform cross-validation over your Training dataset, and carry on naming it Training and Tets datasets.

  I am particularly interested detecting 8 signs. Question 1: Am I suppose to train the model using only the training set of these 8 signs or train the model with the entire training dataset and ignore the signs that I am not interested in at the detection stage?

It depends on whether the practical application you are designing the algorithm for will encounter those remaining signs. If so, I will give these signs a label, for examle 'Don't care'. The algorithm should have the capability of recognizing your 8 signs and to know that the others are of non importance, but do not fall on your desired targets. On the contrary, if the algorithm will be always fed with these 8 signs, you can use only those images. There would be other approaches: you can train a previous SVM in which you set two groups of signs (binary classifier or Support Vector Data Description), one containing the target 8, and other with the remaining ones. Then, another SVM can recognize the sign only if it has been previously classified among the 8 desired one (this SVM is only trained with the 8 signs).

  Upon training the multi-class SVM, I want to test the classifier performance using the test data. Question 2: Can someone guide me through how to do this? I found something similar to what I am looking for available here. It would be great if I can obtain quantitative data such as a confusion matrix in percentage form.

Taking it into account, I guess you have to name your datasets as Training and Evaluation. You first train the model with the training data, and then you evaluate how it generalizes with the evaluation data. First you have to decide which metrics you will use, I recommend precision and recall, per class. Once you have your trained model, you perform predictions over the evaluation test and you'll get the metrics for this dataset. For instance, if your results over the evaluation data are worse than on over the training data, your model is overfitting the training dataset.

However, I strongly recommend you to use cross-validation on the Training-dataset, visit this link


oversampling plus down sampling using smote not working on random forests I am trying to solve a classification problem on a highly imbalanced data set. I am using SMOTE to over sample the minority samples and down sample the majority ones. After creating a balanced data set, I applied the random forest model. But, the prediction error for the minority class is extremely high even after using a balanced data set. What could possibly be going wrong?

library(DMwR)
new.data <- SMOTE(Clicked ~ ., train, perc.over = 600, perc.under =  80)
table(new.data$Clicked)

rand.forest <- randomForest(Clicked ~., data=new.data, mtry = 7,
                            importance = TRUE, proximity=TRUE, ntree = 1000
                            )

#confusion matrix
table(yhat.rf, test$Clicked)

yhat.rf   0   1
      0 889  47
      1  57   7

In my experience, giving weights to observations (if the algorithm in use supports it) generally works better for highly imbalanced  classification problems. Since, your are using RandomForests I would suggest you to try that.


How hidden layer is made binary in Restricted Boltzmann Machine (RBM)? In RBM, in the positive phase for updating the hidden layer(which should also be binary), [Acually consider a node of h1 ∈ H(hidden layer vector)] to make h1 a binary number we compute the probability of turning on a hidden unit by operating activation function over total input (after the activation function operation, we would be getting values in the range between 0 and 1, since activation function I am using - sigmoid).
My doubt is that how do we make it binary by leveraging the probability computed. I don't think if P>=0.5, make it 1 else 0 is a proper method to work on.

By few literature reviews,
I found this document (by Hinton), in section 3.1: he has stated "the hidden unit turns on if this probability is greater than a random number uniformly distributed between 0 and 1". What does this actually mean? And also in this link, they say "Then the jth unit is on if upon choosing s uniformly distributed random number between 0 and 1 we find that its value is less than sig[j]. Otherwise it is off." I actually didn't get this. Whether the random number generated is same for all h ∈ H ? Another query is, what about the random in next sampling iteration?

I saw this video. Just watch the video from that point as per the link. How do you get that sampled number? Whether we have to just run rand() in Matlab and obtain it? Should it would be different for each h(i) (oh nooo! I don't think the machine will learn properly)? Whether the random number should be different for each iteration or the same random number can be used for all iteration to compare?

As you correctly say, we calculate the probability of a hidden unit $h_j$
being one and then make it binary. That probability is given by
$$p(h_j=1) = \sigma\left(b_j + \sum_{i=1}^V w_{ij}v_i \right)$$
where $\sigma$ is the sigmoid function, $b_j$ is the bias of hidden unit $h_j$, $V$ is the number of visible units, $v_i$ is the (binary!) state of visible unit $i$, and $w_{ij}$ are the weights.

So, your MATLAB code for obtaining the probabilities hidden_probs is something like this (we write the sum implicitly by making a matrix multiplication):

hidden_probs = sigmoid(hidden_bias + data * weights)

Now, we have the probability $p(h_j=1)$ for each hidden unit $j \in [1,H]$. Now, this is only a probability. And we need a binary number, either 0 or 1. So the only thing we can do is pick a random sample from the probability distribution of $h_j$, which is a Bernoulli distribution.

As all hidden units are independent, we need to get one sample for each hidden unit independently. And also, in each training step, we need to draw new samples.

To draw these samples from the Bernoulli distribution, you can use the built-in functions of e.g. MATLAB (binornd) or Python (numpy.random.binomial). Note that these functions are to sample from a binomial distribution, but the Bernoulli distribution is just a special case of the binomial distribution with N=1. In MATLAB, that would be something like

hidden_states = binornd(1, hidden_probs)

which would create vector hidden_states which contains either 0 or 1, drawn randomly for each probability in hidden_probs.

As you probably have noticed, nobody does that! E.g. describes it in his Practical Guide to Training RBMs, as

  the hidden unit turns on if this probability is greater than a random number uniformly distributed between 0 and 1.

That is exactly what Hinton does in his RBM code: he gets a random number for each hidden unit using rand, i.e. randomly sampled from the uniform distribution between [0,1]. He then does the comparison:

hidden_states = hidden_probs > rand(1, H)

This is equivalent to using binornd, but is probably faster. For example to generate a random number that is 1 with p=0.9, you get a random number from [0,1]. Now, in 90% of the cases, this random number is smaller than 0.9, and in 10% of the cases it is larger than 0.9. So to get a random number that is 1 with p=0.9, you can call 0.9 > rand(1) - which is exactly what they do.

tl;dr: Pick a new random number from the range [0,1] for each hidden unit in each iteration. Compare it to your probability with hidden_probs > rand(1,H) to make it binary.


SVM with polynomial kernel: strange results with high degree? Currently I'm working in WEKA, using the SMO classifier (an implementation of SVM).
For an assignment I am requested to use a polynomial kernel, and report the results for degrees varying from 1 up to 50. I need to comment on overfitting, and my understanding is that higher degrees lead up to a higher risk of overfitting. (And hence, the percentage of correctly classified instances, using 10-fold cross validation should increase as the degree increases).

My intuition is right for the 'first' degrees, resulting in a 99.6% rate at degree=10. However, when I set the degree to 50, WEKA reports a 9% rate.

How is it possible that this rate drops (and by so much)?

Setting the polynomial kernel degree to 50 is likely causing the SVM to severely overfit to the data, which would explain the 9% you are seeing.  Increasing the degree helps the SVM make an appropriate generalization, but when you start to see the validation/test accuracy decrease, then the SVM is starting to overfit.