Features from images using opencv in Python I want to start   with  image processing (image classification).
I installed  opencv  and plan  to use it    with Python.
Being starter  at   this, I am looking   for references   regarding  trivial  issues like:
I just  saved some images  and  want to  get their pixel  values  as  features ,  hoewever   when I try loading it  in opencv,  I am    getting differently  shaped   3-dimmensional  objects.
Some   references   or suggestion regarding this   issue  will be highly  appreciated.

Your numpy arrays are 3-dimensional because they are colored images. The first dimension is the y-dimension, the second dimension is your x-dimension and the third dimension is the different color channels. What order they are in depends on your reader, but I think OpenCV uses BGR (check the documentation here). A lot of the feature extraction methods only work on grayscale arrays which are mapped back to two dimensions, any of the well known image packages can do this for you (OpenCV, scikit-image).


How to limit a Julia notebook to a single core? I'm running a Julia 1.4 notebook, via Jupyter lab. I need to limit the notebook to a single logical CPU core, on average.
In Python I use the following incantation to restrict Numpy's CPU usage:
from threadpoolctl import threadpool_info,threadpool_limits
with threadpool_limits(limits=1, user_api='blas'):
    >> code here <<

Would anyone happen to know whether its possible to similarly limit Julia's CPU usage from within a notebook?

If the high CPU usage is due to parallelized linear algebra routines, limiting the CPU usage of the linear algebra backend helps. For Julia 1.4 compiled to use BLAS, the following limits it to a single core:
using LinearAlgebra;
BLAS.set_num_threads(1);

(Many forum posts recommend blas_set_num_threads(1) for older versions of Julia. This seems to have been depricated)


How to print nullity correlation matrix I've a trainingset which has 400 features and most of them have null value.

I tried to draw the heatmap of nullity correlation matrix by means of Python and missingno, but the heatmap is unreadable due to high number of features.

How can I print the nullity correlation matrix, instead of draw it?

Using pandas, the nullity correlation matrix seems to be obtained by df.isnull().corr() (this is how it is done is missingno), and this makes sense.

missingno package also states, in the heatmap function documentation, that for large datasets the dendogram view is better. However, it does not tell me if "large" means many features or entries.


Working as data scientist for a nonprofit company Hello I work as data scientist for a private company.
I am interested in working for a nonprofit company, such as a research institute (public or private) or a company that takes care of  issues such as  environment, public health,  social improvements. Even an internet company like Wikipedia can be interesting.
Does anybody know if nonprofit companies hire data scientists?

I see at least five ways to approach this problem of finding a data scientist position/work specifically at non-profit, non-governmental or similar organizations, as I describe below. I hope that this is helpful.

First, and the most obvious, way is to search major job portals, such as indeed.com, dice.com, monster.com, CareerBuilder, Glassdoor and others, for data scientist or similar positions, such as data analyst, data engineer, quantitative analyst, statistical modeler, or even market researcher.
The second and also obvious way is to tap your professional social networks and research or inquire your contacts for any potential data science work opportunities in the areas of your interests.
The third way is to search web sites, focused on the non-profit and related topics (they usually have a job listings or similar section) as well as specialized non-profit job portals. For example: http://encore.org, http://idealist.org, http://bridgespan.org, http://cgcareers.org, http://opportunityknocks.org, http://foundationcenter.org, http://thenonprofittimes.com, http://philanthropy.com, http://ynpn.org, http://philanthropyjournal.org, http://nonprofit-jobs.org, CoF Jobs (section isn't easy to find), http://careers.councilofnonprofits.org, http://nonprofittalentmatch.com.
The fourth way is to perform Internet search and research on non-profit or similar organizations that you might be interested in working with (criteria might vary from organization's size to industry focus or geographical locations). Based on the information presented  on their websites, make notes and then approach those organizations, directly inquiring or applying for positions of interest.
The fifth way is to consider various non-profit, for-profit and otherwise social good-themed data science-focused organizations, initiatives and Kaggle-like competitions, such as DataKind, DrivenData, DataLook and Data Science for Social Good.


Tensorflow: Unable to Save checkpoint after every 2 global steps during training the SSD model object detection python models/object_detection/train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/ssd_inception_v2_coco.config

INFO:tensorflow:Restoring parameters from /home/rahul/Downloads/ssd_inception_v2_coco_2018_01_28/model.ckpt
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Starting Session.
INFO:tensorflow:Saving checkpoint to path training/model.ckpt
INFO:tensorflow:Starting Queues.
INFO:tensorflow:global_step/sec: 0
INFO:tensorflow:Recording summary at step 0.
INFO:tensorflow:global step 1: loss = 14.1708 (73.929 sec/step)
INFO:tensorflow:global step 2: loss = 13.3957 (26.779 sec/step)
INFO:tensorflow:global_step/sec: 0.0207936
INFO:tensorflow:Recording summary at step 2.
INFO:tensorflow:global step 3: loss = 13.2996 (34.331 sec/step)
INFO:tensorflow:global step 4: loss = 12.6129 (27.737 sec/step)
INFO:tensorflow:global step 5: loss = 12.0835 (28.638 sec/step)
INFO:tensorflow:global step 6: loss = 11.9736 (29.535 sec/step)
INFO:tensorflow:global_step/sec: 0.0333812
INFO:tensorflow:Recording summary at step 6.
INFO:tensorflow:global step 7: loss = 11.3325 (35.411 sec/step)
INFO:tensorflow:global step 8: loss = 10.9632 (28.500 sec/step)
INFO:tensorflow:global step 9: loss = 10.8758 (27.419 sec/step)
INFO:tensorflow:global step 10: loss = 11.1301 (25.544 sec/step)
INFO:tensorflow:Stopping Training.
INFO:tensorflow:Finished training! Saving model to disk.
rahul@rahul-pc:~/crack_detection_ssd_inception$

For above it is saving checkpoint at 0th & 10th step but I want to save checkpoint after every 2 global steps for which i re-trained my model only for 10 num_steps (this is only for testing).
from where we can modify the checkpoint saving?

I tried to change save_checkpoints_steps, save_checkpoints_secs in object_detection/model_main.py & object_detection/model_lib_test.py

Use the saver class:

saver = tf.train.Saver()
with tf.Session() as sess:
    # Run your ops here with sess.run()
    saver.save(sess, 'my-model', global_step=2)

This will create checkpoint files with prefix my-model at every 2 time steps.

Reference: https://www.tensorflow.org/guide/saved_model


Pivot reshape dataframe I need help reshaping a DataFrame that i got from a csv file.

In this file I have the first column as ID and then 97 columns that represent sale units of 4 products during time.

the data is like

id    p1_201501   p1_201502 ......  p1_201812  p2_201501..... p2_2018012   p3_..

x12     125         12        ......  06          500     ..... 14         2...
nz15     250         16        ......  600         423     ..... 312        56...
....

The Id represent the area in which the product is sold. And I have 700 rows (i.e 700 df['id'].nunique() ) And so I need to find the yearly sales trend of a given product. So i thought the solution is to change the dataframe in order to have years as rows and the quantity for each product (p1, p2, p3)

Something like:

id    date        p1   p2    p3
x12   201501     12   500   32
      201502    ....
      201503    ....
      ....
      201812    .....
nz15  201501    .....
      201502    .....
      .....
      201812    .....

But I don't know if it is the right choice to find the saled trend in this case and if so How to change the first dataframe to be like the second I know some about pivot reshape. But I can't find how to do it and how to keep the id (the area in which the product is sold)

Any help please?

my intial dataframe :

     id  p1_201501  p1_201502  p1_201803  p2_201801  p2_201812
0   x12        125         12          6        500         14
1  nz15        250         16        600        423         32
2  qz15        350          4         20        223         32

my program:

import pandas as pd

df = pd.melt(df, id_vars=["id"], var_name="product", value_name="values").reset_index(drop=True)
df['date'] = df['product'].str.split('_',n = 1, expand=True)[1]
df['year'] = df['date'].astype(str).str[:-2]
df['month'] = df['date'].astype(str).str[4:]

#drop unused columns
df.drop(['product','date'],axis=1,inplace=True)

df = df[['id', 'year','month','values']]
df = df.sort_values(['id', 'year','month'], ascending=[True,True,True]).reset_index(drop=True)
df.set_index(['id','year','month'])

#just sum each annual total sales for each id
df = df.groupby(['id','year'])['values'].sum()

print(df)

final result:

id    year
nz15  2015     266
      2018    1055
qz15  2015     354
      2018     275
x12   2015     137
      2018     520


Why Decision trees performs better than logistic regression I'm working on a machine learning project, a classification of (100 x 100) Images (every pixel contains 0 or 255), my training set contains 10000 examples (which I split into 2 parts 80% training/20% for testing). When I applied Logistic regression algorithm I had 22 % of accuracy and when I applied a decision trees algorithm I had 55 % of accuracy, but I can't understand why the second algorithm gives a better result.

Have you considered using a neural network for this? The challenge with the two approaches you're using is that both models are not great at handling a large number of features and you, essentially, have 10,000 features. So, my guess is that even with the decision tree,  you're probably just picking up some background pixels that are inflating your accuracy rate. There's probably something there that the decision tree is picking up and then "locking" you into that branch of the model, which is less than ideal. Using a neural network can help you create a model that can do more nuanced choices


Implementation of Stochastic Gradient Descent in Python I am attempting to implement a basic Stochastic Gradient Descent algorithm for a 2-d linear regression in Python. I was given some boilerplate code for vanilla GD, and I have attempted to convert it to work for SGD.
Specifically -- I am a little unsure as to if I correctly implemented the loss function and partial derivatives, since I am new to regressions in general.
I do see that the errors tend to "zig zag" as expected. Does the following look like a correct implementation or have I made any mistakes?
    #sample data
    data  = [(1,1),(2,3),(4,3),(3,2),(5,5)]

    def compute_error_for_line_given_points(b, m, points):
        totalError = 0
        x = points[0]
        y = points[1]
        return float(totalError + (y - (m * x + b)) ** 2)

    def step_gradient(b_current, m_current, points, learningRate):
        N = float(1)
        for i in range(0, 1):
            x = points[0]
            y = points[1]
            b_gradient = -(2/N) * (y - ((m_current * x) + b_current)) #this is the part I am unsure
            m_gradient = -(2/N) * x * (y - ((m_current * x) + b_current)) #here as well
        new_b = b_current - (learningRate * b_gradient)
        new_m = m_current - (learningRate * m_gradient)
        return [new_b, new_m]

    err_log = []
    coef_log = []
    b = 0 #initial intercept
    m = 0 #initial slope

    iterations = 4
    for i in range(iterations): #epochs
        for point in data: #one point at a time for SGD
            err = compute_error_for_line_given_points(b,m, point)
            err_log.append(err)
            b,m = step_gradient(b,m,point,.01)
            coef_log.append((b,m))
```

There is only one small difference between gradient descent and stochastic gradient descent. Gradient descent calculates the gradient based on the loss function calculated across all training instances, whereas stochastic gradient descent calculates the gradient based on the loss in batches. Both of these techniques are used to find optimal parameters for a model.

Let us try to implement SGD on this 2D dataset.

The algorithm

The dataset has 2 features, however we will want to add a bias term so we append a column of ones to the end of the data matrix.

shape = x.shape
x = np.insert(x, 0, 1, axis=1)

Then we initialize our weights, there are many strategies to do this. For simplicity I will set them all to 1 however setting the initial weights randomly is probably better in order to be able to use multiple restarts.

w = np.ones((shape[1]+1,))

Our initial line looks like this

Now we will iteratively update the weights of the model if it mistakenly classifies an example.

for ix, i in enumerate(x):
   pred = np.dot(i,w)
   if pred > 0: pred = 1
   elif pred < 0: pred = -1
   if pred != y[ix]:
      w = w - learning_rate * pred * i

This line is the weight update w = w - learning_rate * pred * i.

We can see that doing this process continuously will lead to convergence.

After 10 epochs

After 20 epochs

After 50 epochs

After 100 epochs

And finally,

The code

The dataset for this code can be found here.

The function which will train the weights takes in the feature matrix $x$ and the targets $y$. It returns the trained weights $w$ and a list of historical weights encountered throughout the training process.

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

def get_weights(x, y, verbose = 0):
    shape = x.shape
    x = np.insert(x, 0, 1, axis=1)
    w = np.ones((shape[1]+1,))
    weights = []

    learning_rate = 10
    iteration = 0
    loss = None
    while iteration <= 1000 and loss != 0:
        for ix, i in enumerate(x):
            pred = np.dot(i,w)
            if pred > 0: pred = 1
            elif pred < 0: pred = -1
            if pred != y[ix]:
                w = w - learning_rate * pred * i
            weights.append(w)
            if verbose == 1:
                print('X_i = ', i, '    y = ', y[ix])
                print('Pred: ', pred )
                print('Weights', w)
                print('------------------------------------------')

        loss = np.dot(x, w)
        loss[loss<0] = -1
        loss[loss>0] = 1
        loss = np.sum(loss - y )

        if verbose == 1:
            print('------------------------------------------')
            print(np.sum(loss - y ))
            print('------------------------------------------')
        if iteration%10 == 0: learning_rate = learning_rate / 2
        iteration += 1
    print('Weights: ', w)
    print('Loss: ', loss)
    return w, weights

We will apply this SGD to our data in perceptron.csv.

df = np.loadtxt("perceptron.csv", delimiter = ',')
x = df[:,0:-1]
y = df[:,-1]

print('Dataset')
print(df, '\n')

w, all_weights = get_weights(x, y)
x = np.insert(x, 0, 1, axis=1)

pred = np.dot(x, w)
pred[pred > 0] =  1
pred[pred < 0] = -1
print('Predictions', pred)

Let's plot the decision boundary

x1 = np.linspace(np.amin(x[:,1]),np.amax(x[:,2]),2)
x2 = np.zeros((2,))
for ix, i in enumerate(x1):
    x2[ix] = (-w[0] - w[1]*i) / w[2]

plt.scatter(x[y>0][:,1], x[y>0][:,2], marker = 'x')
plt.scatter(x[y<0][:,1], x[y<0][:,2], marker = 'o')
plt.plot(x1,x2)
plt.title('Perceptron Seperator', fontsize=20)
plt.xlabel('Feature 1 ($x_1$)', fontsize=16)
plt.ylabel('Feature 2 ($x_2$)', fontsize=16)
plt.show()

To see the training process you can print the weights as they changed through the epochs.

for ix, w in enumerate(all_weights):
    if ix % 10 == 0:
        print('Weights:', w)
        x1 = np.linspace(np.amin(x[:,1]),np.amax(x[:,2]),2)
        x2 = np.zeros((2,))
        for ix, i in enumerate(x1):
            x2[ix] = (-w[0] - w[1]*i) / w[2]
        print('$0 = ' + str(-w[0]) + ' - ' + str(w[1]) + 'x_1'+ ' - ' + str(w[2]) + 'x_2$')

        plt.scatter(x[y>0][:,1], x[y>0][:,2], marker = 'x')
        plt.scatter(x[y<0][:,1], x[y<0][:,2], marker = 'o')
        plt.plot(x1,x2)
        plt.title('Perceptron Seperator', fontsize=20)
        plt.xlabel('Feature 1 ($x_1$)', fontsize=16)
        plt.ylabel('Feature 2 ($x_2$)', fontsize=16)
        plt.show()


Don't know how to preprocess my dataset for image classification I'm trying to do image classification using CNN. The exact model isn't important but I decided to try use AlexNet and I'm getting abysmal accuracy.

I believe the issue might be with my data preprocessing. My dataset directory contains a Training and Test folder but no validation folder (I have to split the dataset myself) and they are layed out like this:

Training
├── class0
│   ├── image1
│   ├── ....
│   └── image20
│
├── ....
│   ├── image1
│   ├── ....
│   └── image20
│
└── class9
    ├── image1
    ├── ....
    └── image20

Here is my AlexNet model, the code is below:

model = tf.keras.models.Sequential([
    #Conv_1          #original model was built for input shape of 224X224
    tf.keras.layers.Conv2D(96, (11,11),strides=4, padding='valid', activation='relu', input_shape=(224, 224, 3)),
    # Pooling_1
    tf.keras.layers.MaxPooling2D((2, 2), strides=(2,2),padding='valid'),
    # Batch Normalisation_1
    tf.keras.layers.BatchNormalization(),
    # Conv_2
    tf.keras.layers.Conv2D(256, (11,11),strides=1, padding='valid', activation='relu'),
    # Pooling_2
    tf.keras.layers.MaxPooling2D((2, 2), strides=(2,2),padding='valid'),
    #Batch Normalisation_2
    tf.keras.layers.BatchNormalization(),
    # Conv_3
    tf.keras.layers.Conv2D(384, (3,3),strides=1, padding='valid', activation='relu'),
    # Batch Normalisation_3
    tf.keras.layers.BatchNormalization(),
    # Conv_4
    tf.keras.layers.Conv2D(384, (3,3),strides=1, padding='valid', activation='relu'),
    # Batch Normalisation_3
    tf.keras.layers.BatchNormalization(),
    #conv_5
    tf.keras.layers.Conv2D(256, (3,3),strides=1, padding='valid', activation='relu'),
    #pooling_3
    tf.keras.layers.MaxPooling2D((2, 2), strides=(2,2),padding='valid'),
    #Batch Normalization_4
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Flatten(),
    #Dense layer_1
    tf.keras.layers.Dense(4096, activation='relu'),
    tf.keras.layers.Dropout(0.5),
    tf.keras.layers.BatchNormalization(),
    #Dense layer_2
    tf.keras.layers.Dense(4096, activation='relu'),
    tf.keras.layers.Dropout(0.5),
    tf.keras.layers.BatchNormalization(),
    #Dense layer_3
    tf.keras.layers.Dense(1000, activation='relu'),
    tf.keras.layers.Dropout(0.5),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Dense(1, activation='sigmoid')
    ])

As for what I did to preprocess my data:

train_dir = '/content/gdrive/My Drive/x/Training'
test_dir = '/content/gdrive/My Drive/x/Test'

train_datagen = ImageDataGenerator(
      rescale=1./255,
      rotation_range=40,
      width_shift_range=0.2,
      height_shift_range=0.2,
      shear_range=0.2,
      zoom_range=0.2,
      horizontal_flip=True,
      fill_mode='nearest',
      )

test_datagen = ImageDataGenerator(rescale=1./255, validation_split=0.4)

train_generator = train_datagen.flow_from_directory(
        train_dir,
        target_size=(224, 224),
        batch_size=20,
        class_mode='binary')

validation_generator = test_datagen.flow_from_directory(
        test_dir,
        target_size=(224, 224),
        batch_size=20,
        class_mode='binary',
        subset='validation')

model.compile(loss='binary_crossentropy',
              optimizer=RMSprop(lr=1e-4),
              metrics=['acc'])

history = model.fit_generator(
      train_generator,
      steps_per_epoch=100,
      epochs=10,
      validation_data=validation_generator,
      validation_steps=10,
      #callbacks = [checkpoint],
      verbose=2)

Any and all help will be immensely appreciated! Thank you to anyone who may reply.

Find a working Keras setup for a pretrained model with custom data generator here.

Note that validation data should not be augmented!

from keras.preprocessing.image import ImageDataGenerator

train_datagen = ImageDataGenerator(
      rescale=1./255,
      rotation_range=40,
      width_shift_range=0.2,
      height_shift_range=0.2,
      shear_range=0.2,
      zoom_range=0.2,
      horizontal_flip=True,
      fill_mode='nearest')

# Validation data should not be augmented!
test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
        # This is the target directory
        train_dir,
        # All images will be resized to 150x150
        target_size=(150, 150),
        batch_size=20,
        # Since we use binary_crossentropy loss, we need binary labels
        class_mode='binary')

validation_generator = test_datagen.flow_from_directory(
        validation_dir,
        target_size=(150, 150),
        batch_size=20,
        class_mode='binary')

model.compile(loss='binary_crossentropy',
              optimizer=optimizers.RMSprop(lr=lr_first), # 2e-5
              metrics=['acc'])


custom delimiter for dat file I've a dat file in the following format which I'm trying to load using pandas.

28::Persuasion (1995)::Romance
29::City of Lost Children, The (1995)::Adventure|Sci-Fi
30::Shanghai Triad (Yao a yao yao dao waipo qiao) (1995)::Drama
31::Dangerous Minds (1995)::Drama
32::Twelve Monkeys (1995)::Drama|Sci-Fi
33::Wings of Courage (1995)::Adventure|Romance
34::Babe (1995)::Children's|Comedy|Drama

Code

movies = pd.read_csv('movies.dat')

I'm getting this error

ParserError: Error tokenizing data. C error: Expected 1 fields in line 11, saw 2

Line 11:

11::American President, The (1995)::Comedy|Drama|Romance

I understand that it's because of the comma. One option I found is to skip these lines, which I don't want to. Is it possible to set newline as the delimiter here? Any help will be truly appreciated.

You have to tell

pd.read_csv

the separator that you want to use by doing:

pd.read_csv('movies.dat', sep = '::')

This should work.


Training set Distribution and Activation function/Loss function correlation How should the probability distribution of the training set influence the choice of the activation function / loss function?
For instance if I have a Multinoulli distribution, which activation function should I choose? And why?
I can't get this correlation between the probability distribution of the training set and the choice of the activation function / loss function.

The probability distribution of the training set has normally nothing to do with the activation function/loss function. Instead, the activation function of the last layer and the loss function are directly defined by what you are trying to predict.
For instance:

If you have a regression problem where the output values are not bounded, you would probably use no activation in the last layers and a mean squared error as loss function.
If you want your network to perform binary classification, you would use a sigmoid activation in the last layer (which outputs a value between 0 and 1, associated with the probability of belonging to one class or the other), with a binary cross-entropy as loss function.
If you want your network to select among N elements (e.g. multiclass classification), you would probably want your network to predict a probability distribution over those N discrete elements. This is precisely a categorical/multinoulli distribution over a discrete output space (i.e. only one element is selected from N possible discrete alternatives). In this case, the activation of the last layer of your network should be a softmax, and the loss must be the categorical cross-entropy, also referred to as negative log-likelihood. Be careful, however, with frameword-specific details, because it is frequent that this pair of activation and loss is implemented with numerical stability considerations and you need to select the appropriate implementations. For instance, Pytorch, when you use NLLLoss, the last layer activation must be a LogSoftmax, not a normal Softmax. Alternatively, you can use CrossEntropyLoss which combines NLLLoss and LogSoftmax into a single class. These aspects are normally described in the framework documentation.


RandomizedSearchCV() not scoring all fits I'm experiencing an issue with a RandomizedSearchCV grid that is not able to evaluate all of the fits. 50 of the 100 fits I'm calling do not get scored (score=nan), so I'm worried I'm wasting a bunch of time trying to run the gridsearch. I'm wondering how to troubleshoot this and haven't found anything in the past few days and I'm hopeful that the community can help me squash this bug. Now, the details:
I have constructed a XGBClassifier model as such:
xgb_clf = xgb.XGBClassifier(tree_method="exact", predictor="cpu_predictor", verbosity=1,
                            objective="binary:logistic", scale_pos_weight= 1.64)

# my trainingset is imbalanced 85k majority class, 53k minority class

Currently, I am attempting to use the hashing trick to encode my categorical variables, as they are all nominal. I do this after splitting my training set into X and y variables
ce_hash = ce.HashingEncoder()
hashed_new = ce_hash.fit_transform(X)
hashed_X = hashed_new

I then conduct my train_test_split as normal, then instantiate a RandomizedSearchCV with a parameter grid, code is as such:
X_train, X_test, y_train, y_test = tts(hashed_X, y, test_size=.25)

# create my classifier
xgb_clf = xgb.XGBClassifier(tree_method="exact", predictor="cpu_predictor", verbosity=1,
                            objective="binary:logistic", scale_pos_weight= 4)

# Create parameter grid
params = {"learning_rate": [0.2, 0.1, 0.01, 0.001],
               "gamma" : [10, 12, 14, 16],
               "max_depth": [2, 4, 7, 10, 13],
               "colsample_bytree": [ 0.8, 1.0, 1.2, 1.4],
               "subsample": [0.8, 0.85, 0.9, 0.95, 1, 1.1],
               "eta": [0.05, 0.1, .2, ],
               "reg_alpha": [1.5, 2, 2.5, 3],
               "reg_lambda": [0.5, 1, 1.5, 2],
               "min_child_weight": [1, 3, 5, 7],
               "n_estimators": [100, 250, 500]}

from sklearn.model_selection import RandomizedSearchCV

# Create RandomizedSearchCV Object
xgb_rscv = RandomizedSearchCV(xgb_clf, param_distributions=params, scoring='precision',
                              cv=10, verbose=3)

# Fit the model by running ten fits on ten 'folds', or 100 individual fits.
model_xgboost = xgb_rscv.fit(X_train, y_train)

However, during 50% of the 100 fits, I will get a score that looks like this:
[CV]  subsample=0.8, reg_lambda=2, reg_alpha=3, n_estimators=100, min_child_weight=3, max_depth=10, learning_rate=0.001, gamma=16, eta=0.1, colsample_bytree=1.4, **score=nan**, total=   0.1s

When this occurs, it occurs in sections of ten, so 10 straight fits will all generate a score of nan. The 50 nan scores don't always occur in the same order, but there are always 50 that don't get scored correctly.
Would anyone know how I can attempt to correct this and ensure that all 100 fits get scored? Is this happening because I'm using a hashed feature set?
Thanks!

Some of your hyperparameter values aren't allowed (colsample_bytree and subsample cannot be more than 1), so probably xgboost errors out and sklearn helpfully moves on to the next point, recording the score as NaN.
Half of your values for colsample_bytree are disallowed, which supports seeing half of your scores as NaN; and that will happen regardless of the fold, which explains why you always see them in groups of 10.


Hesitate to drop a feature I used to tried making a fake correlation in my mocking dataset and found that if the score is more than 0.5 I can reduce feature to avoid singularity

In the given example they are many correlated variables. Therefore I am thinking about dimensionality reduction. The one feature here that I plan to remove is Hum-6D05 because it has strong correlation between Hum-6D12, Hum-6F04, and Hum-6F14

Question:
Am I correct?
Any comment or advice are welcomed

That isn't easy to answer, as it highly depends on your taks. The only time where it is definitely recommended to remove a feature is when two attributes correlate perfectly (e.g. price and tax of a product).

Removing attributes that do not correlate perfectly could worsen your accuracy as information gets lost. But keeping them could affect the computation time. If you don't worry about computation time, I would highly recommend you to keep every bit of information about the data.

If you are concerned about overfitting there are way better techniques such as regularization to solve this problem.

Note: Dimensionality reduction is mostly used to visualize high dimensional data, so that a human eye can detect patterns. For example by shrinking the dimension to 3D or even 2D one could observe an exponential dependency.


What is a better solution for text classification than use of perplexity To classify some texts, I train a language model over a training set and then select the model which has the lowest perplexity on a given test sample as the class of that sample.
I would like to know what other methods or measures except perplexity can be used.

You can evaluate pre-built language models. Advantage is :

These models are usually trained on very large text corpora (Such a a snapshot of  text from billions of Web pages).
It requires very little development effort for testing models for a given classification problem.

Examples :

https://spacy.io/usage/examples

https://uber.github.io/ludwig/examples/


Calculating Rank Ordering Error Metric for implicit recommendation I'm reading Collaborative Filtering for Implicit Feedback Datasets. On page 6 they detail their evaluation strategy, which they define as mean Expected Percentile Ranking with the following formula:

$$\overline{\text{rank}} = \frac{\sum_{u,i} r^t_{ui} \text{rank}_{ui}}{\sum_{u,i} r^t_{ui}}$$

This is the same formula that Datacamp defines as the appropriate error metric for implicit recommendation engines, except they call it "Rank Ordering Error Metric". I'm implementing the system in Spark, so I've defined a test dataset to try things out:

test_df = spark.createDataFrame(
  [
    ("A", "Fish", 1, 1),
    ("A", "Dogs", 2, 2),
    ("A", "Cats", 3, 3),
    ("A", "Elephants", 4, 4),
    ("B", "Fish", 1, 1),
    ("B", "Dogs", 2, 2),
    ("B", "Cats", 3, 3),
    ("B", "Elephants", 4, 4)
  ], ["Customer", "Item", "ImplicitRating", "PredictedRating"]
)

rankWindow = Window.partitionBy("Customer").orderBy(desc("PredictedRating"))
test_df\
  .withColumn("RankUI", percent_rank().over(rankWindow))\
  .withColumn("RankUIxRating", col("RankUI") * col("ImplicitRating"))\
  .show()

and the output is:

+--------+---------+--------------+---------------+------------------+------------------+
|Customer|     Item|ImplicitRating|PredictedRating|            RankUI|     RankUIxRating|
+--------+---------+--------------+---------------+------------------+------------------+
|       B|Elephants|             4|              4|               0.0|               0.0|
|       B|     Cats|             3|              3|0.3333333333333333|               1.0|
|       B|     Dogs|             2|              2|0.6666666666666666|1.3333333333333333|
|       B|     Fish|             1|              1|               1.0|               1.0|
|       A|Elephants|             4|              4|               0.0|               0.0|
|       A|     Cats|             3|              3|0.3333333333333333|               1.0|
|       A|     Dogs|             2|              2|0.6666666666666666|1.3333333333333333|
|       A|     Fish|             1|              1|               1.0|               1.0|
+--------+---------+--------------+---------------+------------------+------------------+

I'm effectively modelling a perfect prediction here by setting the Predicted "Rating" to match the ImplicitRating. My problem is that plugging those values into the formula above gives me...

$$\overline{\text{rank}} = \frac{\sum_{u,i} r^t_{ui} \text{rank}_{ui}}{\sum_{u,i} r^t_{ui}} = \frac{0.0+1.0+1.\dot{33}+1.0+0.0+1.0+1.\dot{33}+1.0}{4+3+2+1+4+3+2+1} = \frac{6.\dot{66}}{20} = 0.\dot{33}$$

Given the paper is explicit in saying that lower values of $\overline{\text{rank}}$ are better and that they achieved values as low as ~ 8%, I'm confused as to how that can be given my experience in this experiment.

What am I doing wrong?

I found a video called "Evaluating Implicit Ratings Models" by Datacamp. It is an explanation of evaluation of recommendation engines that use Implicit Ratings. Although they refer to the metric differently by calling it Rank Ordering Error Metric, the formula they give is identical to the one defined in the paper I linked in my question. Following along with the examples in the video and plugging in the values from their examples, I get the same result they do. I guess it seems slightly less than intuitive but I appear to be understanding how it works correctly anyway.


Create an Algorithm, that translates human language into code I am an experienced programmer and want to get into ML. My final goal in ML is to create an algorithm, that converts written human language into a query language like SQL.

Example:

Input:
"Give me all clients, with an email"

Expected output:

SELECT * FROM Clients where email != null

Where should I look into, to achieve something like this sometime in the future?

A simple approach would be just syntactic conversion, as done in precompilers/preprocessors.

A second step might be interpretation of some sort of controlled language where you allow more complex phrases, as long as they are not ambiguous.

A not very different, but data-driven, approach is represented by form-filling chat bots (couldn't quickly find a generic link, but plenty of examples in search results) where you aim to identify/locate the necessary bits to build what you expect to be your queries.

A more advanced approach that might give you some interesting ideas is this screenshot-to-HTML builder, based on recurrent neural networks, namely LSTM and GRU.

This could go further with named entity extraction, context interpretation, error-correction, and so on. The list continues infinitely.

The last, simply because, as many do, you have left out much of your problem definition implicit, which reminds me of our little friend Alice in Wonderland:

Alice: Would you tell me, please, which way I ought to go from here?
The Cheshire Cat: That depends a good deal on where you want to get to.
Alice: I don't much care where.
The Cheshire Cat: Then it doesn't much matter which way you go.
Alice: ...So long as I get somewhere.
The Cheshire Cat: Oh, you're sure to do that, if only you walk long enough.


CNN computing time on good CPU vs cheap GPU I am a researcher working on my first deep learning project, which consists of using a CNN (pre-trained VGG16+2 densely connected layers) to classify drone imagery of vegetation.

In trying to hack computing times for both training and prediction of new images, I am considering asking my employer for money to buy a (cheap) NVIDIA GPU.

Being a biologist and not a computer scientist, I do not have any sense of the upgrade this would get me. Searching online I found contradicting opinions.

I am currently working with Keras+Tensorflow on a desktop PC with i7, 3.6 ghz, 32 Gb RAM.

Question: how good a GPU would I need to get a sensible performance increase?

Thanks a lot!

According to this guy, he got a 15x increase from Intel i7 to GeForce 1070.

You also may consider using AWS. You can use a machine 100x as powerful (as a single 1070) and your employer may find it attractive because the upfront sunk cost is zero.


Multiple search/replace on pandas series I have a pandas dataframe with school names as one of the columns. However, there is quite a bit of misspelling in the school names, for example:

'Abernethy Elem School', 'Abernethy Elementary School', 'BOISE/ELIOT ELEM SCHOOL', 'Boise/Eliot Elementary School', 'Boise-Eliot Elementary School' ...

I am trying to clean up the names by doing this:

school_perf_report["SCHLNM"] = school_perf_report["SCHLNM"].str.lower().str.replace(r"elementary","elem").str.replace(r"/"," ").str.replace("-"," ").str.replace(r"\s+"," ")

Is there some cleaner way to perform the same operation? Basically

Change "elementary" to "elem"
Remove / or - and replace them with space.
Remove multiple spaces
Lower case everything

Thanks

Create a definition for string replacements:

def change_string(x):
    return x.replace('|', '_').replace("elementary","elem").replace('old_value', 'new_value)

Then use map command to perform operation on entire column:

# You could create a new column to check whether the output is as expected
# If it is as expected, you could modify the original column
school_perf_report["SCHLNM_MODIFIED"] = school_perf_report["SCHLNM"].map(lambda x: change_string(x))

Hope this helps.


Minimum number of samples on a CNN Related to this paper, could it also be applied to convolutional neural networks? I ask this because FNN and CNN share many characteristics
This is the formula:
$(N_e+1)\cdot N_C + (N_C+1)\cdot N_S < 0,1N_{ej}$
Where Ne it is Entry Layers, Nc it is Hidden Layers, Ns it is Output Layers and Nej it is the minimum number of samples needed for training.

CNN is a feed forward neural network that is generally used for Image recognition and object classification. So, the answer it is yes.


Hinge Loss understanding and proof I hope this doesn't come off as a silly question, but I am looking at SVMs and in principle I understand how they work. The idea is to maximize the margin between different classes of point (within any dimension) as much as possible.

So to understand the internal workings of the SVM classification algorithm, I decided to study the cost function, or the Hinge Loss,  first and get an understanding of it...

$$L=\frac{1}{N} \sum_{i} \sum_{j \neq y_{i}}\left[\max \left(0, f\left(x_{i} ; W\right)_{j}-f\left(x_{i} ; W\right)_{y_{i}}+\Delta\right)\right]+\lambda \sum_{k} \sum_{l} W_{k, l}^{2}$$

Interpreting what the equation means is not so bad. For all classes we find the difference between all classes and the class we want to interpret (with a difference of delta) and sum all of them that are greater than 0. By looking at this it seems we are trying to figure out how off a classification is for every point.

However, I do not actually understand why the Hinge Loss works, and why it is useful for SVMs. I cannot find any proofs of the Hinge Loss, nor any visual representation as to why it works.

I understand that we are trying to minimize the Hinge Loss during training, but I have no idea how the equation of the Hinge Loss came to be, and why it works. I am looking for some kind of mathematical proof of the Hinge Loss in relation to SVMs.

Understanding

In order to calculate the loss function for each of the observations in a multiclass SVM we utilize Hinge loss that can be accessed through the following function, before that:

The point here is finding the best and most optimal w for all the observations, hence we need to compare the scores of each category for each observation. Imagine, we have an observation j which we know its label. the score of its actual label is $s_(y_j)$ or based upon what you wrote $f(x, w)_(y_i)$.

We also know the scores of other classes for this observation, in fact, we have a vector of scores, namely, $f(x, w)_j$ What do we need to do now? calculate the differences between the actual class score and other class scores (the score vector $f(x, w)_j$) obviously not for the actual score ($j != y_i$) and sum them up.

Ultimately, we would have $L_i$ that is the loss value for each observation. now if the difference between the actual class's score and the other scores is high enough, then the prediction is ok, yeah? so we put 0 as the loss value, consider that we put a safety margin like 1 or your $\delta$ and if otherwise, the actual class's score is not higher or equal than the incorrect class's score plus that $\delta$ we put their difference as loss value, meaning $s_j - s_(y_i) + 1$

then we take the mean of loss values ($L_i$)s for all of the observations and set it as Loss value, L.

The second part of your formula is a $L_2$ regularization value in order to work better for the test set by impeding the model to be too complex.

If the score for the actual class is high enough, higher than the score for other classes plus the safety margin, the loss value for that observation is 0, hence we have a better classification. In fact, we want to have the weights in such a way that for each observation, the score of it's actual class is enough more than the scores of other classes.

Formula

(i is each of the observations in the dataset)

$$Li = \sum_{j != y_i} something$$

and this something is $0$ if $s_(y_i) >= s_j + 1$ and $s_j - s_(y_i) + 1$ otherwise.

in fact, $L_i$ can be written as:

$$Li = \sum_{j != y_i} max(0 , s_j - s_(y_i) + 1)$$

and then, in order to find out the general cost function for all of the observations in the dataset, we need to calculate their average, hence:

$$L = \frac{1}{N} \sum_{i}^n Li $$

Plot

This Hinge name comes from the plot:

(the values and slope might differ)

Here the $x-axis$ is the actual class's score and the $y-axis$ is the Loss value. As seen in the plot, we set a sample $s_j$ for a class, for this $s_j$, if the real class's score is something more than 0, then since $s_j - s_(y_i) + 1$ is negative, the loss value would be 0 and so on.

Inspired by Justin Johnson's lecture in Stanford Machine Learning course.


ValueError: cannot reshape array of size 136415664 into shape (2734,132,126,1) I have a data set I loaded with cv2 but when I try to format it I get the above error. I start by moving the data into X_train and X_test (the loaded data is in x_train and x_test).
X_train = []
X_test = []

# Image matrices are different sizes so I am making them the same size
for i in range(len(x_train)-1):
  resized = cv2.resize(x_train[i], (img_width, img_height))
  X_train.append(resized)
for i in range(len(x_test)-1):
  resized = cv2.resize(x_test[i], (img_width, img_height))
  X_test.append(resized)

# Convert to numpy arrays
X_test = np.array(X_test)
X_train = np.array(X_train)

# Gather statistics
print(X_train.shape)    # -> (2734, 132, 126, 3)
print(X_train.size)     # -> 136415664
print(len(X_train))     # -> 2734

# Convert to black and white
X_train = X_train/ 255.
X_test = X_test/ 255.

# First line throws error
X_train = np.reshape(X_train, (len(X_train), img_height, img_width, 1))
X_test = np.reshape(X_test, (len(X_test), img_height, img_width, 1))

What am I doing wrong?

You need $2734 \times 132\times 126\times 1=45,471,888$ values in order to reshape into that tensor. Since you have $136,415,664$ values, the reshaping is impossible. If your fourth dimension is $4$, then the reshape will be possible.


How can I detect blocks of text from scanned document images ORIGINAL IMAGE:

GOAL:

I want to separate texts into individual paragraphs by placing bounding boxes over them (as shown above).

I tried it do this via traditional computer vision approach using opencv.

I plotted character level bounding box
Next, I gray-scaled the image, binarized it.
Applied dilation
And finally placed bbox over the dilated image.

This is what I get:

> #Morphological Transformation

kernel = np.ones((3,4),np.int8)

dilation = cv2.dilate(im_bw, kernel)

cv2.imwrite('dilated.png', dilation)

Plotting rectangular box

ret,thresh = cv2.threshold(im_bw, 127,255,0)
image, contours,hierarchy = cv2.findContours(thresh,cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE )

for c in contours:
    rect = cv2.boundingRect(c)
    if rect[2] < 50 or rect[3] < 50 : continue

    print (cv2.contourArea(c))
    x,y,w,h = rect
    cv2.rectangle(im_new,(x,y),(x+w,y+h),(0,255,0),2)

cv2.imwrite('sample_res_inner.jpg',im_new)

Since the image is scanned image plus the line-spaces between them are small I couldn't able to segment them based on paragraphs.

How can I get my desired result?

There are two options :

Scan the images with a higher DPI. This should accentuate vertical separation between paragraphs.
Train a Deep learning model for Text Detection in scene. Examples : https://github.com/qjadud1994/CRNN-Keras and https://github.com/mvoelk/ssd_detectors


Over fitting in Transfer Learning with small dataset I am using Transfer Learning to perform image classification.
Base model used : Resnet50 using ImageNet dataset
class_1 and class_2 are the classes each having 1000 samples each (small dataset). And the dataset is not similar to ImageNet dataset.
Number of FC layers used here are 3 with [1024, 512, 256].
I have used a drop out of 0.5 to reduce over-fitting.
When I trained the model with 100 epochs, I could clearly see the model over-fits with training accuracy of 0.9985 and testing accuracy of 0.875.
Is the number of FC layers used is too many which is causing this over-fit problem?
How can I make the model more generalised?
The code used is as given below:
from keras.applications.resnet50 import ResNet50, preprocess_input
from keras.preprocessing.image import ImageDataGenerator
from keras.layers import Dense, Activation, Flatten, Dropout
from keras.models import Sequential, Model
from keras.optimizers import SGD, Adam
from keras.callbacks import TensorBoard
import keras
import matplotlib.pyplot as plt

HEIGHT = 300
WIDTH = 300
TRAIN_DIR = "/home/ubuntu/dataset/training_set/"
TEST_DIR = "/home/ubuntu/dataset/test_set/"
BATCH_SIZE = 8
class_list = ["class_1", "class_2"]
FC_LAYERS = [1024, 512, 256]
dropout = 0.5
NUM_EPOCHS = 100
BATCH_SIZE = 8

def build_finetune_model(base_model, dropout, fc_layers, num_classes):
    for layer in base_model.layers:
        layer.trainable = False

    x = base_model.output
    x = Flatten()(x)
    for fc in fc_layers:
        print(fc)
        x = Dense(fc, activation='relu')(x)
        x = Dropout(dropout)(x)
    preditions = Dense(num_classes, activation='softmax')(x)
    finetune_model = Model(inputs = base_model.input, outputs = preditions)
    return finetune_model

base_model = ResNet50(weights = 'imagenet',
                       include_top = False,
                       input_shape = (HEIGHT, WIDTH, 3))

train_datagen = ImageDataGenerator(preprocessing_function = preprocess_input,
                                   rotation_range = 90,
                                   horizontal_flip = True,
                                   vertical_flip = False)

test_datagen = ImageDataGenerator(preprocessing_function = preprocess_input,
                                  rotation_range = 90,
                                  horizontal_flip = True,
                                  vertical_flip = False)

train_generator = train_datagen.flow_from_directory(TRAIN_DIR,
                                                    target_size = (HEIGHT, WIDTH),
                                                    batch_size = BATCH_SIZE)

test_generator = test_datagen.flow_from_directory(TEST_DIR,
                                                  target_size = (HEIGHT, WIDTH),
                                                  batch_size = BATCH_SIZE)

finetune_model = build_finetune_model(base_model,
                                      dropout = dropout,
                                      fc_layers = FC_LAYERS,
                                      num_classes = len(class_list))

adam = Adam(lr = 0.00001)
finetune_model.compile(adam, loss="categorical_crossentropy", metrics=["accuracy"])

filepath = "./checkpoints" + "RestNet50" + "_model_weights.h5"
checkpoint = keras.callbacks.ModelCheckpoint(filepath, monitor = ["acc"], verbose= 1, mode = "max")
cb=TensorBoard(log_dir=("/home/ubuntu/"))
callbacks_list = [checkpoint, cb]

print(train_generator.class_indices)

history = finetune_model.fit_generator(generator = train_generator, epochs = NUM_EPOCHS, steps_per_epoch = 100,
                                       shuffle = True, callbacks=callbacks_list, validation_data = test_generator)

Update :

Weight file generated from the model after training is 2.7 GB. Is it normal considering the complexity of the model?

How would I select the steps_per_epoch value? Is there any standard?

First of all:

I think you should reduce the number of FC layers and number of nodes of FC layers, for example, one FC with 256 or 512, or 2 FC with 256 and 512. Try this.

Try to make your batch size 30, and decrease number of epochs to nearly 10 or 20. 100 epochs are too many for your small size dataset.

Secondly, there is more than one way to reduce overfitting:
1- Enlarge your data set by using augmentation techniques such as flip, scale, etc.
2- Using regularization techniques like dropout (you already did it), but you can play with dropout rate. Try more or less than 0.5.
3- One of the good techniques in your case is to do early stopping. In any epoch when you see that the model goes to overfit, stop it.
4- Using cross-validation to train/test your model.
and many more.

For more, see here, or here, or here.

Feel free to ask any further questions.


Is 25-75 a good case-control ratio for a neural network? I have 50,000 DNA samples from a biobank. Only 2,200 samples have the label (disease) that I want to work with for supervised learning in a neural network.

So my question is, how many of those non-diseased samples do I need to include in my training data? If the diseased samples are my cases, how big should my control group be? Twice as big? The same size? What would a good ratio be?

The feature space is insanely big, so I am worried about training times. It feels unnecessary to include all the other 48,000 controls.

And yes, yes I know it will depend on the accuracy that I am seeing in the model, but what’s a good starting ratio?

I don't think there really is an optimal answer to a question like this. I would advise that the control group should be at least 50%, but anything more than that would be better. For a NN model the more examples you give it to train on, the more patterns it will learn and the more accurate it will be. If you could use the entire set, then do!

It's really down to time constraints and processing power. Try starting by using a small sample, maybe a few dozen or a few hundred samples and see how long that takes. Then try a sample roughly twice the size and check it again. You can then extrapolate the size of the sample you could use for given time constraints.

If you use this approach make sure to use a stratified k-fold split for cross validation. That will keep the relative distributions in your samples. Hope this helps.


Daily access pattern of a real cloud storage for a current research project I would need file access traces of real world cloud storages. Unfortunately I was not able to find any yet.

More precisely I need real world traces with at least the following information: filename, file size, access time, operation (put, post, get, delete). The kind of data (if the files are images, textdata or something else) does not matter. Furthermore, it is also ok if it is not a cloud storage, but for example an online image gallery, as long as all 4 operations (put, post, get, delete) are included its fine for me.

Any help would be appreciated.

Try these links from UCI Machine Learning Repository:

Anonymous Microsoft Web Data
MSNBC.com Anonymous Web Data


How to best visualise two sets of data over several years? I have a set of data on crime statistics by police force and year, and another set of data showing police workforce numbers over the same period. I am trying to determine the best way to show this data. I have tried three alternatives but looking for preference out of these three or altnernatives that would show the data better (using Tableau but it's the visualisation more than the tool that matters).
First, i created a pair of charts shown one above the other so the X axis is common with separate Y axis.

Then i combined these two into a dual axis chart

And my last attempt was to use a KPI used by the police in some of their reports - the crime rate per 1000 population and create a similar metric for police staff per 1000 population.

So for 2020, this is showing on average across England and Wales, there were 86 crimes per 1000 people and 3.34 police staff.  Dropdowns on the dashboard of each allow changing this to a specific police force and to specific crime groups. The crime rate per 1000 dont match up with the police force reported numbers as they seem to use just the census 2011 figures whereas i have rolledup the population estimates per year (census 2011 + births + immigration - deaths - emmigration). It's still an estimate but should be more accurate than using 10 year old figures.
Any suggestions on how to better present this data ? Thanks in advance.
The full dashboard (still work in progress and draft so apologies for the performance until i determine the right data vis), is on tableau public
https://public.tableau.com/profile/trevor.north#!/vizhome/police-reported-crime-workforce-population-public-v1_12/PoliceWorkforceandReportedCrime-EnglandWales-2007-2020
After some further feedback, i have created two more versions.  One without the map

And one retaining the map but removing one of the panes (that showed %change of workforce).

There is a problem with the map.  I used colour to show the number of offences so London shows as darker colour.  Without modifying the data, all the rest were at other end of the scale.  To rectify, i used a LOG function on the number of offences.  This then shows more colour variation between the different police force regions.  But i now cannot show the legend since that shows a range upto about 7000 whereas the number of offences is over a million in London.  Think i may have to create an information button and an extra layer and manually include a legend on that for the map. Is there a better way ?

Statistics of this nature are commonly normalised with respect to the population sizes by presenting them as per X. In this case per 1,000 or 100,000 people.
Absolute number of crimes makes no sense and lacks context. In terms of display, as both KPI ranges are similar (3 vs 19) a common X and Y axis is possible with two different lines.
If you want to communicate trends relative to a baseline year (e.g. 2014) you could divide all values relative to this year and express it as a % change.  I.e. crime went up 10% but police numbers went down by 5%.


ValueError: Expected 2D array, got 1D array instead I would like to extract the 20 most informative features of a very large set of features $X$ coming from a dataset containing clinical data  by using the RFE function from scikit-learn in Python.

$X$ is a 68 x 1140 matrix where

Each row represents a recorded session.
For each subject, there are 4 recorded sessions.
Then, there are 17 subjects in the dataset.

My idea is to use 70% of the dataset (i.e. 70% x 1140 random features from each recording) and extract 50 features out of the whole dataset.

$Y$ represents a ranking from 0 to 2.

In other words, my data looks like this:

And my implementation in the code is the following:

## X = features
## Y = labels
p = 0.7
n_perc = round(X.shape[1]*X.shape[0]*p) #70% of the data -> number of elements (height x width x 70%)
rand_idx = np.random.randint(X.shape[1]*X.shape[0], size=n_perc) #random indices (70% of the data)
X_rnd = X.flatten()[rand_idx] #select that 70% in X
Y_rnd = np.repeat(Y,round(X.shape[1]*p)) #we match the dimensions for X_rnd - Y_rnd
selector = RFE(estimator=LogisticRegression(C=1),n_features_to_select=20) #run RFE
selector.fit(X_rnd.reshape,Y_rnd) #select best features

The idea is that I flatten all the values from X and I get only 70% random elements from $X$, i.e $X_{rnd}$ (and also adapt $Y$ accordingly, i.e. $Y_{rnd}$).

ValueError: Expected 2D array, got 1D array instead:
array=[-0.25367578  0.8069118  -0.63161352 ...  0.5500815  -0.37418711
0.2580666 ]. Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

But some reason I'm getting this error, which I don't understand. It says that I should reshape the array if I have either one feature or one sample, but it's not my case.

Does anybody know what I should do? Is this how I should approach the problem? Should I reshape $X$ in another manner?

Thanks.

selector.fit(X_rnd.reshape,Y_rnd) #select best features

X_rnd should be a 2D matrix.

I believe this error came when "fit" method was executed with X_rnd
but your code is showing X_rnd.reshape. In this case, error should be -

    ValueError: Expected 2D array, got scalar array instead:
    array=<built-in method reshape of numpy.ndarray object at 0x000000000C28EAD0>.
    Reshape your data either using array.reshape(-1, 1) if your data has a single                         feature or array.reshape(1, -1) if it contains a single sample.


Apply GroupByKey to identify the Frequently Products Purchase Together I've a data set with this fields:

Transaction_ID
Customer_ID
Department
Product_ID

And I'm trying to obtain a tuple with the products associated with each customer transaction. Like:

Transaction_1 -> Product_ID 1, Product_ID 2, Product_ID 3
Transaction_2 -> Product_ID 1, Product_ID 2, Product_ID 4
....

I've this code: But It not return the dataset as I want:

case class transactions (Transaction_ID: String, Customer_ID: String, Department: String, Product_ID: String)

def csvToMyClass(line: String) = {
    val split = line.split(',')
    transactions(split(0),split(1),split(2),split(3))
}

val  csv = sc.textFile("FILE").map(csvToMyClass)
csv.take(10)
csv.saveAsTextFile("PATH/output.csv")

How can I obtain the list of products associated group by Transaction_ID??

Since you are using Spark 1.6, I'd rather do these kind of transformations with DataFrame as it's much easier to manipulate.

You'll need to use SQLContext implicits for this :

import sqlContext.implicits._ // not need in spark-shell

Now, let's create some dummy data just to follow the code snippet that you have provided :

scala> val data = sc.parallelize(Seq("1,2,3,4", "2,3,4,5", "1,3,4,5", "1,6,6,7"))
// data: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[13] at parallelize at <console>:27

Back to your code, we define a case class Transactions and the CSV converter :

scala> case class Transactions(Transaction_ID: String, Customer_ID: String, Department: String, Product_ID: String)
// defined class Transactions

scala> def csvToMyClass(line: String) = { val split = line.split(','); Transactions(split(0), split(1), split(2), split(3)) }
// csvToMyClass: (line: String)Transactions

We will use implicits now to convert our data into a DataFrame after converting it to Transactions :

scala> val df = data.map(csvToMyClass).toDF("Transaction_ID", "Customer_ID", "Department", "Product_ID")
// df: org.apache.spark.sql.DataFrame = [Transaction_ID: string, Customer_ID: string, Department: string, Product_ID: string]

Let's take a look at the DataFrame :

scala> df.show
// +--------------+-----------+----------+----------+
// |Transaction_ID|Customer_ID|Department|Product_ID|
// +--------------+-----------+----------+----------+
// |             1|          2|         3|         4|
// |             2|          3|         4|         5|
// |             1|          3|         4|         5|
// |             1|          6|         6|         7|
// +--------------+-----------+----------+----------+

Now all we have to do is a simple group by and perform a collect_list aggregation on the first dataframe :

scala> val df2 = df.groupBy("Transaction_ID").agg(collect_list($"Product_ID"))
// df2: org.apache.spark.sql.DataFrame = [Transaction_ID: string, collect_list(Product_ID): array<string>]

We can check the content of our new DataFrame df2 :

scala> df.groupBy("Transaction_ID").agg(collect_list($"Product_ID")).show
// +--------------+------------------------+
// |Transaction_ID|collect_list(Product_ID)|
// +--------------+------------------------+
// |             1|               [4, 5, 7]|
// |             2|                     [5]|
// +--------------+------------------------+

I hope that this answers your question.

Note: If you wish to know what's the difference between RDD and DataFrames, I advice you to read Databrick's blog entry about it here.


Evaluating information extraction from structured documents I'm trying to find metrics to evaluate multiple algorithms for key information extraction from already OCRed invoices. For instance, such an algorithm, given an invoice, could find that:

{
    "company": "STARBUCKS STORE #10208",
    "date": "14/03/2015",
    "address": "11302 EUCLID AVENUE, CLEVELAND, OH (216) 229-0749",
    "total": "4.95",
}

Formally, I defined the task as a function f that for a given invoice document D returns a dictionary of entries: {field1:value1,field2:value2,...}.

Initially, I thought about using precision, recall and F1 measures, assuming that each dictionary entry is marked as correct if both a field and its value matches exactly the ground truth. I borrowed this approach from ICDAR 2019 Robust Reading Challenge on Scanned Receipts OCR and Information Extraction.

However, the problem with metrics based on exact string match is that they penalize even small errors: missing/redundant words, single differences in characters and spacing. Those errors may stem not only from the imperfect information extraction algorithm but also from faulty OCR algorithm. Unfortunately, I am not able to test information extraction in isolation (i.e., given 100% perfectly OCRed input).

Therefore, I am looking for some more relaxed metrics. What are my possibilities? What has been used in this field?

Some ideas that come to my mind:

calculate distance between two texts using Levenstein distance on character/word
consider two texts matching if distance is smaller than threshold X.
or include information about the distance in the final measures, e.g. normalize it and micro-average over each document.

I have worked with Structured text using OCR. OCR is prone to errors even while reading the content and a slight change in string arrangement would lead to false positives. I used Cosine Similarity and LEvenstein distance. I curated the data with pip install ftfy, Flashtext fuzzywuzzy ,chardet and set a threshold of 97% giving with 2% buffer for errors which might be unintentional/Typos or Inability of OCR to read the character etc. With this approach you are just relaxing on the 2% error and you can still use the same Precision, Recall or Specificity and Accuracy metrics. And we based on new data, you can gather observations and adjust your threshold on run time. And yes, Make your threshold configurable than hard coded which will be easier to run multiple experiments.

I hope this helps.


Groupby product, return tuple I have found out the most profitable products in my dataframe by using:

df.groupby('ProductName')['ProfitPerOrder'].sum().sort_values().tail()

It gives me the output below. each ProductName has a ProductCategory. How do I display the category next to the product name in the output below?

ProductName
WFS Shoes           29033.887659
RDL Suit            45845.253318
Tennis Suit         46848.513342
Davenport Shoes    127103.915707
Halter Dress       314155.742025
Name: ProfitPerOrder, dtype: float64

Add more columns when you are doing group by in the first parentheses..

First we should understand why it's giving this result..

It's similar to Sql,we are applying an aggregate function on a grouped by value,
That's why it's giving only one value,

If you want to have further information,
Add the columns name in the first parentheses in a $list$

It will be something like this

df.groupby([Col1,Col2...,Coln])(another col).sum().sort_values().tail()

Edit -1

Recently enough, I came across that we can cascade group by together by playing smart and using the .join(second group by(). aggregate ())

Here's the working example (ignore the column names...it's relevant to my dataset)

df.groupby('Year_of_Release')[['Global_Sales']].sum().join( df.groupby('Year_of_Release')[['Name']].count())


Machine Learning algorithm to predict an outcome where output is not known I have a very unique problem that I would like to solve using machine learning. I have a set of 90 or so unique options. Each of these options has a unique set of features (5 to be specific) that vary over time and that can be represented mathematically as either scalars or booleans. I want to train a model to tell me what the best option at that moment is given the complete set of inputs.

At first, I figured a simple feedforward neural network would do the trick but I don't have a ton of experience with ML models so I ran into the following issue. I know my problem isn't really a classification problem since I am trying to pick the best option. I figured it could be easily modelled as a regression problem (as some mathematical combination of the inputs) which is fine but then I don't know what the expected outcome would be. For example, if the expected output is the weighted average of the inputs, I don't know what the final value should be. All I know is that option B should be suggested over option A.

I started reading about random forests and they seem like they might help my situation but I wanted to get other opinions first.

What I was looking for is called ordinal regression or ranking learning (https://en.wikipedia.org/wiki/Ordinal_regression)

Edit: I will elaborate more on why this solved my issue
What I was having a problem with was finding a statistical model to appropriately predict the ordinality of multiple independent options. What I cared about was the order of the options and less so about what the value of the options was.

Ordered Logit/Probit describe such models. The provided wikipedia link explains them in far more detail than I can.


Keep track of trainings, datasets eetc After searching quite some time for it on Google I could not find a sufficient software/toolbox that can manage trainings of neural networks. I thought of a program that combines visualization techniques without the need to write code as well as having the possibility to compare several trainings of neural networks and be able to store them easily.
Does a program like this exist?

Regards
Lukas

It does, it's an API called Keras, written on top of TensorFlow.
With that you can rapidly prototype any "standard" NN architecture, such as feedforward NN, RNN, or CNN. With it you can export the trained model with one line of code and save it for later use. As for visualization, Keras offers integration with pydot (GraphViz) to plot the whole NN as a graph for you to inspect.


Using non-negative matrix factorization (NMF) for facial recognition I'm interested to find out how to implement NMF for facial recognition. I understand that the NMF works by taking V, which is a matrix of face images (n resolution x m persons), and factorize V = WH, where we get r basis vectors. I presume that this step is done using the training set. But I don't know how to proceed from here.

With PCA, after building the eigenfaces from the training set, we project the test image onto the face-space, then classify the image by comparing the weight coefficients between the test image and training images. But what about NMF? Do I use the basis vectors? What do I do with them?

One major issue with this approach is that if you represent your image as a vector of pixel intensities, you have zero translation-invariance.  Two pictures that are exactly the same but shifted over 10 pixels will likely look like dissimilar vectors.  That is why people try to use other methods that handle translation, like convolutional neural networks or SIFT features.


Is there any data tidying tool for python/pandas similar to R tidyr tool? I'm working on a Kaggle challenge where some variables are represented by rows instead of columns (Telstra Network Disruption). I am currently searching for the equivalent of gather(), separate() and spread(), which can be found in R tidyr tool.

I'd start with the melt() function in pandas.  I wrote an article about it:
https://www.ibm.com/developerworks/community/blogs/jfp/entry/Tidy_Data_In_Python?lang=en


How to approach the PlantVillage dataset? I'm working on the PlantVillage dataset and i want to predict the type of the disease from the image of a leaf. The dataset is labeled in pairs (Type of the plant,Healthy/name of the disease). I'm wondering which approach from the following is better ? Should I build 2 neural nets, one for recognizing the type of the plant, and the second for the name of the disease. Or, should i build one neural net that predicts at the same time the name of the plant and the disease.
In my opinion the first approach is better because i'll have more data for each task while in the second approach i'll have more classes and less data for each class. However, there might be a correlation between the plant and the disease that might improve the performance of the model.

I think this approach should work. So we need to clasify the plant and its disease. As mentioned, the disease maybe be associated with the plant's species.

Build a standard CNN or Siamese CNN for identifying the plant species. The model should be able classify the name/species of the plant even when it has a disease of some another plant ( The classification should not be baised upon the plant's disease ).
So, now, after the above model is ready, we build a 2nd model for disease classification. But the classification of the disease maybe be dependent on the plant species. So, for the 2nd model we will have 2 features namely, the image ( raw sample ) and the output of the 1st model ( plant name ). So, the 2nd model's prediction will be based on these two features.

The basic idea will be:

Model 1: input = image, output = plant name.
Model 2: input = ( image, plant name ), output = disease name.

So this is a system of models. That's easy if you are working with Keras or PyTorch.

Hope that helps.


Neural Network Optimization steps order I have a very basic question on the optimization algotithm, when I'm adjusting weights and biases in a NN, should I:

Forward propagate and backpropagate to calculate gradient descent (DC) for each batch once and then repeat for iterations_number times.

or

Forward propagate and backpropagate to calculate gradient descent (DC) one batch for iterations_number times and then continue with the next batch.

One iteration means that you do one forward pass and one backward pass for a single batch containing batch_size examples. Then you move on to the next batch. Also see this post on SE SO.
Note that this is not identical to an epoch. An epoch is only completed when all examples of your dataset have been passed through your network. And as long as your batch size is less than the number of datapoints you will need $\frac{n}{\text{batch size}}$ (with $n$ being the sample size) iterations to complete an epoch. Also see this answer on SE DS.


Is there an API to extract questions and labels from yahoo answers? I want to work on a ML project which involves the family and relationship category from yahoo answers. I want to extract questions from yahoo answers (only the title) with the label as the category it is in on yahoo answers. I want to use it in csv form to train my naive bayes model. Is there an API or something similar to do so ?

It seems as though there is a lot of advice on the Yahoo Developer Site.

Basic steps will be:

get yourself a standard Yahoo account
using that account, get yourself an Application ID (basically an API key)
(probably) get a unique password to go with your API key

This will all allow you to authenticate yourself when making requests for data.

There are daily rate limits for data requests, they say:

  Each service may have different limits on the number of results returned by each query, as well as other variations in supported parameters. The documentation for each service specifies the default number and maximum number you can receive.

I should also point out that the service is not intended for commercial use, so if you plan to make a product and sell it, based on Yahoo data, you need to probably discuss the API usage with them.

The results will be in a form that requires parsing, such as XML, as it seems to be a standard web requests API. This means you can look at tools such as BeautifulSoup4 or the xml ElementTree. You can then choose the specific parts of the received data from Yahoo's API.


Switching To Data Science Field I have some 14 years of work experience in operations management and education areas as Instructor of statistics and operations. I have little better command over statistical learnings. I do have a keen interest in data science learnings. Basically, want to perform a role of analyst on big data.  Will it right to make venture in Data Science field with this experience? There is touch on coding is with but do not project experiences.

Welcome to the site!

As you have mentioned that you are an Instructor of Statistics I think you can directly start with different methodologies/techniques available.

As @Emre has suggested it would be better if you can work on different project available on Kaggle, Analytics Vidya. You would be exposed to most of the necessary things in Data Science field. Following R-Bloggers, will expose you to the latest packages. Ofcourse following Data Science Exchange.

You can doing small projects on different datasets on UCI. For experimenting using different techniques.

These are just couple of things which would help you to be updated.

Follow them for atlest next 2-3 months then you can access yourself.


About sklearn.metrics.average_precision_score documentation There is a example in  sklearn.metrics.average_precision_score documentation.

import numpy as np
from sklearn.metrics import average_precision_score
y_true = np.array([0, 0, 1, 1])
y_scores = np.array([0.1, 0.4, 0.35, 0.8])
average_precision_score(y_true, y_scores)
0.83

But when I plot precision_recall_curve

precision, recall, _ = precision_recall_curve(y_true, y_scores)
plt.plot( recall,precision)

I got the picture:

why the area under the precision_recall_curve is not 0.83?

According to the documentation, the value is not exactly the area under curve, it is
$$\text{AP} = \sum_n(R_n - R_{n-1})P_n.$$
which is a rectangular approximation.

For your specific example, i.e.

      R     P
  1   0.0   1.0
  2   0.5   1.0
  3   0.5   0.5
  4   0.1   0.66

it is calculated as
$$\begin{align*}
\text{AP} & = \overbrace{(0.5 - 0.0)\times1.0}^{(R_2 - R_1)P_2} + \overbrace{(0.5 - 0.5)\times 0.5}^{(R_3 - R_2)P_3} + \overbrace{(1.0 - 0.5) \times0.66}^{(R_4 - R_3)P_4} \\
&= 0.5 + 0.00+ 0.33 = 0.83
\end{align*}$$
which is the area under the red curve as illustrated below:

compared to $$\text{AUPR}=0.5 + \overbrace{\frac{0.5 + 0.66}{2}  \times 0.5}^{\text{trapezoid area}} = 0.79$$
which is the area under the blue curve.


Positively skewed target label in regression I have a dataset where the target label is positively skewed and produces a long tail, and currently I have a high residual on these values when experimenting with some linear, tree-based and neural-network regression models.

I see the same problem with the Boston Housing prediction dataset, and recommendations to apply a log transformation to the target label. This has given some small improvement but not enough. Additionally I've tried to randomly duplicate values within the tail to shift the mean, although I'm not overly comfortable with method.

Are there any alternative transformations to apply, or any models that can put a higher cost weighting on labels with high residuals?

Something that might work is normalizing/standardizing the output, e.g. with target over [0,1] (see min-max scaling). Changing the distribution isn't ideal, and I can't see how changing the mean would lead to increased performance. If it makes sense, you can also potentially experiment with changing from a strict continuous linear regression to a categorical interpretation (see ordinal regression). Classification is generally easier than regression, so if you're able to frame it in that way sometimes it helps performance, especially if the end goal in mind is, for example, only to determine a binary decision (buy/not buy).

For example, for income, regressing over strict amount will lead to imbalance, and the difference between 100 billion net worth and 500 billion net worth is negligible to the mean. If you can accept classifying might be useful, you can use logistic regression, SVM etc and experiment with generating a classification instead of a direct prediction.


How to create a historical timeline using Pandas Dataframe and matplotlib So I've seen a few answers on here that helped a bit, but my dataset is larger than the ones that have been answered previously. To give a sense of what I'm working with, here's a link to the full dataset. I imported this into a jupyter notebook using pandas. I've included a picture of one attempted solution, which was found at this link, right here:
Example Picture. The issue is that 1. This is difficult to read and 2. I don't know how to flatten it out so that it looks like a traditional timeline. The issue becomes more apparent when I try and work with larger segments, such as this one: It's basically unreadable. Here's the code I used to produce both of these (I just modified the included code in order to change which section of the overall dataset was used).

event = Xia['EnglishName']
begin = Xia['Start']
end = Xia['Finish']
length = Xia['Length']

plt.figure(figsize=(12,6))
plt.barh(range(len(begin)), (end-begin), .3, left=begin)
plt.tick_params(axis='both', which='major', labelsize=15)
plt.tick_params(axis='both', which='minor', labelsize=20)
plt.title('Xia Dynasty', fontsize = '25')
plt.xlabel('Year', fontsize = '20')
plt.yticks(range(len(begin)), "")
plt.xlim(-2250, -1750)
plt.ylim(-1,18)
for i in range(18):
    plt.text(begin.iloc[i] + length.iloc[i]/2, i+.25, event.iloc[i], ha='center', fontsize = '12')

This code semi-works, but I'd prefer if the bars were either closer together or differently colored and all on the same y-value. I appreciate any and all help. I've been trying to figure this out for about two weeks now and am hitting a brick wall.

Sorry for the late reply. I don't know whether the below kind of plot suffices for what you are looking.

If yes is the case, you might like matplotlib.hlines. I am providing a sample code for generating a picture like the above.

import matplotlib.pyplot as plt
import random

for x in range(0, 10, 2):
    color = random.choice(['red', 'green', 'blue', 'yellow'])
    plt.hlines(1, x, x + 2, colors=color, lw=5)
    plt.text(x + 1, 1.01, color, ha='center')
plt.show()

Let me know if this was helpful!


Do generative model produce varying outputs for same input I am new to data sciences. I believe the generative model generate responses on-the-fly for a valid user input. Is it correct to assume that such models would generate different responses for the same question?
For e.g: if we trained the model on say medical data. Now if user 1 asked "what is fever" and user 2 asked the same question, could be that user 1 and 2 will receive different answers? if this is so then how to circumvent this problem?
thanks in advance

This depends entirely on the specific model. There are generative models, like most Generative Adversarial Networks (GANs) that receive a random number and generate data. There are other generative models that generate a probability distribution over the output space (e.g. text generation models), and therefore whether the model generates data deterministically depends on the inference procedure (e.g. greedy, sampling, beam search).
If you want your model to generate outputs deterministically, you just select a model and inference method that assures that.
In your example, you may have a normal seq2seq model (e.g. Transformer) and use beam search for decoding, and the outputs will be the same given the same input.


Sequentially Training Certain Layers/Sub-Networks in Keras Functional API Suppose we have a stacked neural network architecture with a
layer that is to be shared between two "sub-networks".

Example:

from keras.layers import Input, Dense
from keras.models import Model

main_input = Input(shape=(5, ))

## Model A: main_input -> A_output
layer_A1 = Dense(10, name='A1')(main_input)
layer_A2 = Dense(10, name='A2')(layer_A1)
layer_A3 = Dense(10, name='A3')(layer_A2)
A_output = Dense(1, name='A_output')(layer_A3)

## Model B: main_input -> layer_A2 -> B_output
layer_B1 = Dense(10, name='B1')(layer_A2)
B_output = Dense(1, name='B_output')(layer_B1)

model = Model(inputs=main_input,
              outputs=[A_output, B_output],
)

model.compile(optimizer='adam',
             loss={
                        'A_output':'mean_squared_error',
                        'B_output':'mean_squared_error'
                    },
             )

The goal is to train model A first so that model B can learn from the pre-trained weights of Layer A2. However calling fit in the current architecture will train both simultaneously and sum up the losses.

How can I change the architecture so that model A is trained first without creating separate models? Ultimately, I'll need to call model.predict(new_sample) where new_sample is of shape (5,) in the example.

Figured it out. To train certain sub-networks first and re-use the trained weights to initialize other layers:

from keras.layers import Input, Dense
from keras.models import Model

main_input = Input(shape=(5, ))

## Model A: main_input -> A_output
layer_A1 = Dense(10, name='A1')(main_input)
layer_A2 = Dense(10, name='A2')(layer_A1)
layer_A3 = Dense(10, name='A3')(layer_A2)
A_output = Dense(1, name='A_output')(layer_A3)

# Train Model A first
model_A = Model(inputs=main_input, outputs=A_output)
model_A.compile(# args)
model_A.fit(# args)

## Model B: main_input -> layer_A2 -> B_output
layer_B1 = Dense(10, name='B1')(main_input)
B_output = Dense(1, name='B_output')(layer_B1)

model_B = Model(inputs=main_input, outputs=B_output)

# Set weights of layer_B1 and freeze; essentially a copy of pre-trained layer_A2
trained_A2_weights = model_A.layers[2].get_weights()
model_B.layers[1].set_weights(trained_A2_weights)
model_B.layers[1].trainable = False

model_B.compile(# args)
model_B.fit(# args)
```


Gradient boosting how can accuracy increase when we lower the depth of tree? What I don't understand about gradient boost is, doesn't lowering height of the tree means we use fewer features in our model?

From my model I get the highest accuracy when the depth is one. Meaning there is just root node at my trees, and uses one feature. How can a model that uses one feature gives such accuracy?

In boosting, a low biased & low variance model is ensembled by additively combining high biased & low variant models. Here, biases are gradually diminished while keeping variances low. Therefore, a tree with a single depth, that is highly biased, isn't an issue because additively errors will be reduced.


Micro Average vs Macro Average for Class Imbalance I have a dataset consisting of around 30'000 data points and 3 classes. The classes are imbalanced (around 5'000 in class 1, 10'000 in class 2 and 15'000 in class 3). I'm building a convolutional neural network model for classification of the data. For evaluation I'm looking at the AUC and ROC curves. Because I have three classes I have to either use micro- or macro-average.
To calculate the micro- and macro-averaged AUC and ROC curve, I use the approach described here: https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html
The micro-averaged AUC / ROC is calculated by considering each element of the label indicator matrix as a binary prediction and the macro-averaged AUC / ROC is calculated by calculating metrics for each label, and find their unweighted mean. In my case micro-averaged AUC is usually higher than macro-averaged AUC.
If we look at the sklearn.metrics.roc_auc_score method it is written for average='macro' that

This does not take label imbalance into account.

I'm not sure if for micro-average, they use the same approach as it is described in the link above.
Is it better to use for dataset with class imbalance micro-average or macro-average? That means which metric is not affected by class imbalance?  In my case micro-averaged AUC (0.85) is higher than macro-averaged AUC (0.79). When I look at the confusion matrix, the majority class is very well predicted (because the network probably learns to predict the majority class) but the minority classes are poorly predicted (almost as many false negatives as true positives). So, overall the AUC should not be that high I think.

The question is actually about understanding what it means to "take imbalance into account":

Micro-average "takes imbalance into account" in the sense that the resulting performance is based on the proportion of every class, i.e. the performance of a large class has more impact on the result than of a small class.
Macro-average "doesn't take imbalance into account" in the sense that the resulting performance is a simple average over the classes, so every class is given equal weight independently from their proportion.

Is it actually a good idea to "take imbalance into account"? It depends:

With micro-average, a classifier is encouraged to focus on the largest classes, possibly at the expense of the smallest ones. This can be considered a positive because it means that more instances will be predicted correctly.
With macro-average, a classifier is encouraged to try to recognize every class correctly. Since it is usually harder for the classifier to identify the small classes, this often makes it sacrifice some performance on the large classes. This can be considered a positive in the sense that it forces the classifier to properly distinguish the classes instead of lazily relying on the distribution of classes.

One could say that it's a kind of quantity vs. quality dilemma: micro-average gives more correct predictions, macro-average gives attention to actually distinguishing the classes.
Very often one uses macro with strongly imbalanced data, because otherwise (with micro) it's too easy for the classifier to obtain a good performance by relying only on the majority class. Your data is not strongly imbalanced so it's unlikely this would happen, but I think I would still opt for macro here.


Features selection/combination for random forest I am working on using random forest to predict 1 or 0. I have about 20 variables available for modeling. I realized that if I put different variables will have different accuracy/sensitivity/specificity. I am wondering if there is a test or method can tell me which variables combination has the highest accuracy? Or which variable combination has the highest sensitivity and specificity respectively?

Thanks in advance!

The Random Forest model in sklearn has a feature_importances_ attribute to tell you which features are most important. Here is a helpful example.

There are a few other algorithms for selecting the best features that generalize to other models such as sequential backward selection and sequential forward selection. In the case of sequential forward selection, you begin by finding the single feature that provides you with the best accuracy. Then, you find the next feature in combination with the first that gives you the best accuracy. This pattern continues until you find $k$ features, where $k$ is the number of features you want to use. Sequential backward selection is just the opposite, where you start with all of the features and remove those which inhibit your accuracy the most. You can find more information on these algorithms here.


Artificial neural network on ternary content addressable memory One of the applications of Ternary content addressable memory(TCAM) is artificial neural network. How does TCAM do prediction or recognition? How TCAM is related to artificial neural network?

It is mostly a performance benefit thing. You can run a neural network on a computer (slow or not). A TCAM is the only real hardware suited to the problem of seeing the cause and supplying the effect, TCAM can perform hopfield net operations very natively in pure hardware.

It puts the network in a hardware spreadsheet formfactor.  It's really simple, and can be bewilderingly so, you have a linear amount of patterns, (which I like to call bullets, because it's like a machine gun ready to fire off and enter in patterns really damn fast, faster than a minigun)    and requires a lot of miniaturization to have a large amount, say a billion bullets,  more so than just ordinary RAM.

If you ever wanted to see a "real" robot in action,  it's going to use hardware TCAM.


How to resample one dataset to conform to the distribution of another dataset? I have two datasets with 20 features, but with different feature distributions (DS_A and DS_B). How can I sample the DS_A to make its distribution similar to DS_B, with respect to multiple features??

I check the similarity/difference of two datasets by checking individual features from DS_A against DS_B, in shape, and percentiles. Features are mostly numerical, some binary, some normalized.

Background:

Some time ago I trained a model using dataset DS_B as ground truth. Now, I want to retrain the model with more recent data and see if the performance improves. The new ground truth data I collect is DS_A, but due to practical reasons, new data is collected somewhat differently, and hence the feature distribution in the new data set is different from the old data set.

One simple way is to transform your distribution linearly. That should work fine if the distribution of your data has changed approximately linearly.

  Question is how to change the distribution of DS_A to match
  distribution of DS_B, with respect to multiple features?

That being said you could transform your feature distribution DS_A by

Substructing mean(DS_A) and add the mean(DS_B)
Divide with the standard deviation of DS_A and multiply with the standard deviation of DS_B

Long story short:You change the mean and the standard Deviation of the DS_A distribution to match the DS_B.

Here is a code in python that apply this transformation to two gaussian distributions

import seaborn as sns
import numpy as np
import matplotlib.pyplot as plt

m1, m2 = 0, 10
s1, s2 = 1, 3

x1 = np.random.normal(m1,s1, 1000)
x2 = np.random.normal(m2,s2, 1000)

sns.distplot(x1, hist_kws=dict(alpha=0.1), color='red', label='Distribution 1')
sns.distplot(x2, hist_kws=dict(alpha=0.1), color='green', label='Distribution 2')

estimated_x1_mean = np.mean(x1)
estimated_x1_sd   = np.std(x1)
estimated_x2_mean = np.mean(x2)
estimated_x2_sd   = np.std(x2)

x2_new = (x2 - estimated_x2_mean + estimated_x1_mean)  * estimated_x1_sd / estimated_x2_sd
sns.distplot(x2_new, color='blue', hist_kws=dict(alpha=0.1, edgecolor='black'), label='Distribution 2 after Transformation')
plt.legend()

And the result


Ways to speed up Python code for data science purposes Although it might sound like a pure techie question, I would like to know which ways you usually try out, for very data science-like processes, when you need to speed up your processes (given that the data retrieval is not a problem and that it also fits in memory etc). Some of those could be the following, but I would like to receive feedback about any other else:

good practices as always using Numpy when possible on numeric operations and not loops...
more good practices like using 'apply', 'applymap'... instead of loops when applying functions to elements of lists, dataframes, etc
Numba applied on native python loops, numpy arrays...
multiprocessing with multiprocessing library depending on the available logical cores

This is motivated by the fact that, if we mainly use Python with all its advantages, we do not want to switch to other languages like Scala or Julia, unless there is no alternative.

Things I care about a lot:

list comprehensions instead of loops
use apply + lambda functions when forced to iterate operations on pandas dataframes
use @tf.function decorator on top of TensorFlow functions to speed up computation
use as much SQL as possible when importing data from databases, to avoid doing the same stuff in Python


Choosing a distance metric and measuring similarity I am trying to decide which particular algorithm would be most appropriate for my use-case.
I have dataset of about 1000 physical buildings in a city with feature space such as location, distance, year built and other characteristics etc. For each new data point, a building, I'd like to find 3-5 buildings that are most similar based on feature space comparison.
I define similarity as weighted comparison of features. I'd like to iterate over entire feature space (w/ filter like location) and choose 3-5 most similar buildings matching the new building data point.
Here's what my data looks like:

I'm wondering what similarity measure would make sense? I work in python, so prefer a pythonic/sci-kit learn way of doing this.

It appears to me that what you're looking for in your use-case is not clustering - it's a distance metric.
When you get a new data point, you want to find the 3-5 most similar data points; there's no need for clustering for it. Calculate the distance from the new data point to each of the 'old' data points, and select the top 3-5.
Now, which distance metric to pick? There are options. If you're using SKLearn, I'd look over this page for example of distance(/similarity) metrics.
If your features are continuous, you can normalize them and use cosine similarity; Start with this, and see if it fits.


What are the Most Dissimilar MNIST Digits? Using whatever definition of dissimilarity over sets that you'd like, what are the most dissimilar two digits in MNIST? I was thinking that a reasonable approach to answering the question would be to pass the two sets through some state-of-the-art VAE and then check the distances in the latent space.  The dissimilarity metric could be distance to/from centroids of the clusters (easiest) or every point to every point in every other cluster ($O(n^2)$), or one of the other set measures of difference.
Has there been any research done on this (to the extent you even need research to answer the question)?  Does anyone have an idea about what the answer might be? I've seen TSNE plots of the latent space for some VAEs, but they're not exactly rigorous measures of distance or dissimilarity.

Not sure if this constitutes a "study" but I have investigated using PCA to decompose the MNIST dataset to visualize in 2D:

pca = PCA(n_components=2)
pca.fit(ziptrain[:, 1:])
Z_train = pca.transform(ziptrain[:, 1:])

fig, ax = plt.subplots()
for digit in np.unique(ziptrain[:, 0]):
    x = Z_train[ziptrain[:, 0]==digit, 0]
    y = Z_train[ziptrain[:, 0]==digit, 1]
    ax.scatter(x, y, label=int(digit),
               alpha=1.0, edgecolors='none', marker='${}$'.format(int(digit)))

ax.legend()

This yielded the following plot:

Visually I would say that 1's and 0's are very different. I'm sure you could continue the study by clustering them & measuring the cartesian distance from the centroids of their respective clusters.
It is important to note though that your definition of what "different" means will change the outcome of such an investigation and that this is simply one approach.


Cross validation techniques What are the advantages vs disadvantages of cross validation types? Like k-fold, leave one out, etc.

As you probably know by now, cross-validation is a method of:

partition dataset into 'train set' and 'test set'
fit model to train set, get the prediction error on test
repeat (1-2) in a bunch of different, 'pre-defined' ways, storing the prediction error each time.
average all the prediction errors to understand how your model will behave in the wild!

All the 'magic' of different cross-validation (CV) types happens at (3), basically. Say your dataset has $n$ observations.

leave $p$ out CV separates your dataset into a train set of size $n-p$ and a test of $p$, gets the prediction error, and repeats this process for every single possible $p$-sized subset of the dataset.
leave $1$ out CV is a special case of leave $p$ out where $p=1$
$k$-fold cross CV partitions your dataset into $k$ equal sized partitions, trains on the first $k-1$ partitions and tests on the last partition, and gets the prediction error. The result at the end is ten $k$ prediction errors to average over. (Note that $k=n$ is the same as leave $1$ out CV.)

As for disadvantages and advantages, it's basically just a trade-off between exhaustiveness and speed. And I guess it depends on your dataset. The upper limit of exhaustiveness is basically doing leave $p$ out CV for all $p=\{1,2,...,n\}$. Probably depends too on the linearity of your dataset. In my experience, I haven't had trouble using $k$-fold cross CV.

Reference


How to justify a predictor in influencing the outcome? I am working on a prediction (binary classification) problem

Currently I get an AUC score of 85-86 and F1-score of 81

Questions

1) The above performance is based on 6 well-known features

2) Let's say I add another feature and I see my AUC and F1 score improve by 1 or 2 points? So is this the only way to know that it really helps/value addition to the model?

3) How do I justify that these features really help in predicting the output?

4) Is there anyway to prove or validate that adding this feature really helps my model and improves the outcome? Is it only using changes in AUC score?

5) For instance, I can add multiple features and my AUC increases by few decimal points, so can I say that they are important or useful or drive predictions? Yes ofcourse, but adding 10 features (where 6 features really impact the outcome and rest 4 increase the auc only by few decimal points). doesn't really overfit. Am I right? Because my prediction score doesn't go beyond 86

Cross Validation

You had a post where we discussed causality, but with ML models assumption is that data represents your problem entirely and has all the information in it. In other words every pattern that you pickup in your train data you can expect it to behave pretty similar in production, hence with this assumption what you want is to "evaluate the entire train" (thats what you can do with CV.) and if it score good on average over all folds you want to add this change (be it a new feature)