How to predict multiple images from folder in python Here is the code for the Prediction of multiple images from the folder. But getting the same label(class) for all the images.I'm not able to find out why every image shows the same label.
# import the necessary packages
from tensorflow.keras.models import load_model
import argparse
import pickle
import cv2
from tensorflow.keras.preprocessing.image import img_to_array
from tensorflow.keras.applications.imagenet_utils import decode_predictions
import numpy as np
import logging, os
import sys
from keras.preprocessing import image
import tensorflow as tf
import math
import operator
from pathlib import Path

# disable the warnings
logging.disable(logging.WARNING)
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3"

image_path = "test_image_folder"

images = []

# load all images into a list
for img in os.listdir(image_path):
        img = os.path.join(image_path, img)
        img = image.load_img(img, target_size=(64,64))
        img = image.img_to_array(img)
        img = np.expand_dims(img, axis=0)
        # normalize the image
        processed_image = np.array(img, dtype="float") / 255.0
        images.append(processed_image)

images = np.vstack(images)

# relative paths to the model and labels
model_path = os.path.join("Output", 'VGG_model.h5')
label_file_path = os.path.join("Output", 'labels')

# load the model and the label encoder
model = load_model(model_path)
lb = pickle.loads(open(label_file_path, "rb").read())

# make a prediction on the image
images_data = []
filenames = []
for filename in os.listdir(image_path):
    pred_result = model.predict(images)
    images_data.append(pred_result)
    filenames.append(filename)

#sorts attributes according to confidence score (how probable attribute exists)
top_k = []
pred = []
for i in range(len(images_data)):
            rank = images_data[i][0].argsort()[-len(images_data[i][0]):][::-1]
            top_k.append(rank)
            top = top_k[i][:15]
            print(filenames[i])
            for node_id in top:
                human_string = label_file_path[node_id]
                score = images_data[i][0][node_id]
                print('%s (score = %.5f)' % (human_string, score))

You are looping on a folder to predict each image -
for filename in os.listdir(image_path):
    pred_result = model.predict(images)
    images_data.append(pred_result)
    filenames.append(filename)

But the argument of the predict function is not changing. Its a stacked value defined above as -
images = np.vstack(images)

This same prediction is being appended into images_data
Assuming your prediction is not failing, it means every prediction is the prediction on all the images stacked in the images_data.
So, for every iteration for i in range(len(images_data)):
This images_data[i][0] is returning you the 1st prediction only.
Changing to for i in range(len(images_data[0])): and images_data[i] should work


How to create visualisation on medical data? I want to create a data visualization on medical data i.e. patient's medical history, allergies one is having, any chronic patient's name must be highlighted, etc. And, there can be a separate visualisation for the medicines, which shows medicines' availability, their expiry date can be highlighted by using mouse_hover_on functionality.  It will be useful for the doctors and the medical staff as well.

Problems:

bl.ocks.org has variety of base visualization models to work upon, but the schema of database I have (given by my mentor) is a bit confusing.
Also, he won't provide us with the actual data of our college due to its confidential nature, so can we get dummy data on the net?

Experience:

I have done work on Weka earlier to understand about data mining, and many different algorithms and classifiers.
I have built a generic data visualization model (for hierarchical data) using the D3.js
I have good working knowledge of Python. So, if anyone could please tell whether there is any hope for doing it(the data visualisation) in python.

If you have good working knowledge in Python than you are good for it. D3.js actually has a Python comparable called Bokeh. You question implies a lot of ramifications and does explain fully what you are trying to achieve so let's go by parts.

Some 2D Scientific Plotting Python Libraries:

Matplotlib
ggplot
Bokeh
Chaco
pyQtGraph (This one has some interesting features in volume slicing if it suits you)

Some UI development Python libraries:

PyQt5
PyQt4 (there are significant changes to PyQt5, thus the suggestion)
PySide (also a Qt port)
wxPython

Some Python numerical libraries:

Pandas
Numpy
Scipy
Scikit-learn (You can do some serious data analysis with this)

Using combinations of the above you can build some very powerful software solutions. Any of the UI libraries suggested has spreadsheet widgets. There are libraries to read csv and excel, among others (JSON, HDF5, etc.).

The data problem can be resolved by either building your own randomized data or by using resources on the net like this or this.


How to Iterate over rows in a dataframe So I've been trying to iterate over rows of my dataframe and my goal is to find matching rows based on two particular columns (say C and P). Then i need to do some manipulations as well in the data for the rows. I've read quite a few answers here telling to use iterrows() or itertuples() but that doesnt serve my purpose because I cannot manipulate my data using them. Same goes for functions like groupby since it only allows manipulation on the whole groups not elements of those groups(correct me if I am wrong here because thats what I have seen in the articles on groupby(). What approach should I use to match rows in my data frame to each other based on columns and then manipulate them.

"What approach should I use to match rows in my data frame to each other based on columns and then manipulate them."
Use pandas.DataFrame.loc:

Setting values
Set value for all items matching the list of labels

df.loc[['viper', 'sidewinder'], ['shield']] = 50
df
            max_speed  shield
cobra               1       2
viper               4      50
sidewinder          7      50

Set value for rows matching callable condition

df.loc[df['shield'] > 35] = 0
df
            max_speed  shield
cobra              30      10
viper               0       0
sidewinder          0       0


Knowledge Graph as an input to a neural network I want to create a neural network that takes as an input a knolwedge subgraph(different types of nodes and different types of edges) to predict some properties. For instance an input in the graph can be :
:Mike :likes :chocolate;
      :studies :Biology.

Assuming we have n nodes and r relation types. How can I provide this as an input in a neural network and how will the architecture of the neural network look like.

You can represent this type of knowledge graph as a binary $n \times r \times n$ tensor.  (You can think of this as a 3D matrix if it helps.)
The first dimension is for the node on the left side of the relationship, the second dimension is for the relation type, and the third dimension is for the node on the right side of the relationship.  Then you can represent any relationship between two nodes by inserting ones in the correct indices.
To demonstrate, suppose we have 4 nodes: Mike at index 0, Sully at index 1, chocolate at index 2, and Biology at index 3.
And suppose we have 2 relations: likes (index 0), studies (index 1).
We would first create a $4 \times 2 \times 4$ tensor filled with zeros:
import torch

graph = torch.zeros([4, 2, 4], dtype=torch.bool)

""" Output:
tensor([[[False, False, False, False],
         [False, False, False, False]],

        [[False, False, False, False],
         [False, False, False, False]],

        [[False, False, False, False],
         [False, False, False, False]],

        [[False, False, False, False],
         [False, False, False, False]]])
"""

To represent the relationship :Mike :likes :chocolate; we would insert a one at the position [0, 0, 2]  (0 for Mike, 0 for likes, 2 for chocolate).
To represent :Mike :studies :Biology, we have a one at [0, 1, 3].
# :Mike :likes :chocolate
graph[0, 0, 2] = 1

# :Mike :studies :Biology
graph[0, 1, 3] = 1

""" graph:
tensor([[[False, False,  True, False],
         [False, False, False,  True]],

        [[False, False, False, False],
         [False, False, False, False]],

        [[False, False, False, False],
         [False, False, False, False]],

        [[False, False, False, False],
         [False, False, False, False]]])
"""

Of course we can represent any relationship between 2 nodes. Let's add a few more:
# Mike and Sully are friends

# :Sully :likes :Mike
graph[1, 0, 0] = 1

# :Mike :likes :Sully
graph[0, 0, 1] = 1

# I suppose chocolatogoly is a subject in Biology
# :Biology :studies :chocolate
graph[3, 1, 2] = 1

Concerning architecture, there are at least two viable ways to feed this representation to a neural network.  You could flatten the whole graph and treat it like a 1-dimensional input.  Another option is to retain the 3 dimensions and use convolutional layers (or something else) to extract features (3D convolution works just like 2D convolution).
As for the rest of the architecture, that's up to you!


Shouldn't L2 regularization be normalized for the number of nodes in a layer? I'm naively thinking it would be better to normalize L2 regularization to the number of elements in a tensor, but I don't see anyone doing that. What am I missing?

I'd say in a multi layer fully connected NN:

x  = input            # vector length n_in
w1 = weights          # shape n_in x n_1
b_1 = bias            # shape n_1
h_1 = Relu(w1 * x + b_1)

w2 = weights          # shape n_1 x n_2
b_2 = bias            # shape n_2
h_2 = Relu(w2 * h_1 + b_2)

Then the L2 loss is normally defined as:

L2 = lambda * ( sum((w1) ^2) + sum((w2) ^2) )

If the first layer is much bigger than the second layer, than the first term in above L2 definition is much bigger, effectively making the second term useless. I'm thinking this would be better:

L2 = lambda * ( sum((w1) ^2)/(n_in*n_1) + sum((w2) ^2)/(n_1*n_2) )

This would also make the hyper parameter lambda independent of the number of nodes you choose. However, I don't see anybody doing this so I suspect I'm missing something. What is it?

However, I don't see anybody doing this so I suspect I'm missing something. What is it?

You are concentrating on how this extra loss term affects the absolute value of the loss function. This is not really relevant. Many forms of regularisation are not expressed as changes to loss function at all - framing L2 regularisation as part of the loss function is convenient because it allows us to re-use the existing weight adjustment logic. However, functionally it is almost identical to a weight decay term applied per batch independently of loss function, e.g. multiply all weights by 0.99 every batch.

If you use L2 regularisation loss, then you will almost certainly want to track a separate metric such as mean square error (i.e. your original loss function before considering regularisation), when comparing test results between different levels of regularisation.

That said, there is no reason that you cannot have a different L2 weight param per layer, and many frameworks will support this. Sometimes this could result in better generalisation. It is not done much in practice because it adds yet more dimensions to search when optimising hyper-parameters to a problem.

There is no reason expect better performance by aiming for roughly equal loss values from regularisation per layer. However, if that is your goal, then a naive scaling by inverse layer size $\frac{1}{N_{in} \times N_{out}}$ would not do it. That is because the weights in a trained network will already tend to scale due to the training targets so that the mean squared value is proportional to $\frac{1}{N_{in} + N_{out}}$. So your actual scaling factor to make the L2 regularisation loss per layer roughly the same would be $\frac{N_{in} + N_{out}}{N_{in} \times N_{out}}$. In code this might look like:

L2 = lambda * (
  sum((w1) ^2)*(n_in+n_1)/(n_in*n_1) + sum((w2) ^2)*(n_1+n_2)/(n_1*n_2)
)

(and of course matching changes to loss gradients). Although I will stress again that I do not expect this change to result in better generalisation.


Getting different chi square values than sklearn function I have a dataset which looks like this-

I am trying to use chi2 as a feature selction algorithm on it.Here is the code

dataset = pd.read_csv('small_dataset.csv')
print(dataset.shape)

X = dataset.iloc[:, 1:106].values
y = dataset.iloc[:,0].values

print(sklearn.feature_selection.chi2(X, y))

I tried to validate the answers from sklearn and the formula which is

so I took the first feature and computed the A B C D which are 0 19 73 73 and N is 146.So the X2 value coming is 21 but in sklearn its coming 579.

What am I doing wrong?

I had a similar problem using Scipy's chi2_contingency. It turns out that the Scipy function uses by default correction=True, and the statistic you showed doesn't use the correction. Perhaps sklearn's chi2 uses the correction as well, and that's the reason you don't get the same results. I suggest trying Scipy's with both correction set to true and false to see if it matches the two results.


overfit random walk using ANN in Keras I am trying to build a neural net that will overfit random walk path. So, far I wasn't able to get a neural net that we shatter/overfit. I was wondering which parameters I should explore, or which guide lines I should follow to achieve this (wierd) task.
Code:
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import SGD,Adam
from keras import regularizers
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import random
import math

model=Sequential()
num_units=100
model.add(Dense(num_units,input_shape=(1,),activation='tanh'))
model.add(Dense(num_units,activation='tanh'))
model.add(Dense(num_units,activation='tanh'))
model.add(Dense(num_units,activation='tanh'))
model.add(Dense(1, activation='tanh')) #output layer 1 unit
model.compile(Adam(),'mean_squared_error',metrics=['mse'])

num_of_steps=3000;
step_size=0.01
x_all=list(range(0,num_of_steps))

##########################random walk functino
random_walk=[0]
for i in x_all[1:]:
    f=np.random.uniform(0,1,1)
    if f<0.5:
        random_walk.append(random_walk[i-1]+step_size)
    else:
        random_walk.append(random_walk[i-1]-step_size)
########################

x1=list(range(1,1000,2))
x2=list(range(2,1000,2))
y1=[random_walk[x] for x in x1]
y2=[random_walk[x] for x in x2]

model.fit(x1,y1,epochs=2500,verbose=1)
fit1=model.predict(x1)
fit2=model.predict(x2)
plt.plot(x1,y1,'k')
plt.plot(x2, y2, 'r')
plt.scatter(x1, fit1, facecolors='none', edgecolors='g') #plt.plot(x_value,sample,'bo')
plt.scatter(x2, fit2, facecolors='none', edgecolors='b') #plt.plot(x_value,sample,'bo')

this is an example of the results I am getting:

And I would want to get a better fit (to my sample data: x1,y1).

Try changing your kernel regularizer. Either reduce its effect or shut it off entirely, then retrain and see what happens.

EDIT: I expect that the kernel regularizer is your best bet. You could also experiment with changing the depth of the network (number of layers) and how many units you give each layer. I would expect if you put more units into the layer that currently has 10, you'd see more overfitting.


Detecting if a sentence contains a numeric series I have many different strings of text. These strings of text are labels for particular things. But these labels are sloppy, sometimes one label is used for many different things. For example:

"Brown foxes edition 1999 series 1-6 EDI"

"Light [old] seasons 1,2,3,4 other gibberish"

I would like to answer the question: "If the label contains a series, does that serie contain the value N?" For the examples above 6 would be included in the first one, but not the second.

Initially I thought of using regexes but that quickly grew out of hand. Digits appear everywhere, people can get very creative with separators and the location of the series in the label is not fixed.
There are many different ways the labels denote series.

What I can do, however, generate labels with series. I'll just grab a bunch of separators, a start and an end digit and iterate. This gives me a nice labeled training set.

Naive Bayes comes to mind for this problem but I'm not sure what good features would be.

update

Let me try to clarify.
Given a label and given a number determine if that number is contained within that label.

For example: Given I'm looking for season 2 (the number). Does "Pioneer One 2011 seasons 1-3" contain season 2?

So there are many ways to denote a series.  How are you going to parse the series down to determine the values if you don't know the format?

Determining if the label has a series does not get you to the specific numbers in the series.

2,3,5,7  parses out to 4 numbers

Is 6 in 1996?  I assume that is one number and 1996 != 6

"55,56,57" is series with a 6 but but not the number 6

Does 7-9 parse out to 2 numbers or 3 number
Is 6 in 7-9?
If 6 is in 7-9 identifying that as a series does not answer that question.

How many ways can there be represent a series that regex got out of hand?  For each format of series you also need to parse the values.  You need to know the format of the series to parse out the numbers.  I would have a set of regex mapped to set of parsers.

Maybe use machine learning to identity new series formats but you are still going to need to parse out the series.


Calculating confidence score in NER I am working on a problem on Named Entity Recognition. Given a text, my model is detecting the Named Entities and extracting that info for the end user. Now the ask is end user needs a confidence score along with the extracted entity. For example, given text is: XYZ Bank India Limited is a good place to invest your money - Our model is detecting XYZ Bank as an Org, but India as a Location (which is wrong - the whole XYZ Bank India Limited is the name of the organization). Our model also gives a probability score for each token it classifies. But the end user wants to know the confidence of the model that it did not mistake to detect the subsequent tokens as the parts of the organization name.
Question is - how can we efficiently measure that in a given sequence our model is detecting a certain sub-sequence as an Organization name (or a Location or something else) correctly or not? How can we say that it did not miss out any subsequent or preceding token which actually a part of the named entity (like it missed India Limited in the above example)?

Named Entity Recognition is traditionally evaluated using precision/recall and F1 score [https://towardsdatascience.com/entity-level-evaluation-for-ner-task-c21fb3a8edf] - the medium article gives a low down on how to achieve this
I recently happened to read this article on a new approach for the same. Please see the details in the attached medium link : [https://towardsdatascience.com/a-pathbreaking-evaluation-technique-for-named-entity-recognition-ner-93da4406930c]  but havent tried this out yet though


What is the difference between ImageNet and ImageNet1k? How to download it? Some papers mention just ImageNet and some papers mention ImageNet 1k database?
What is the difference between these 2? Are they same or is the latter one subset of the former one?

I'm working on Generative Adversarial Nets. I wanted to train it on ImageNet Database. How to download ImageNet 1k? I went to ImageNet site and created & verified my account. Then there were several links. Which one to select?

Thanks!

The ImageNet dataset consists of more than 14M images, divided into approximately 22k different labels/classes. However the ImageNet challenge is conducted on just 1k high-level categories (probably because 22k is just too much).

ImageNet Stats

When people mention results on the ImageNet, they almost always mean the 1k labels (if some paper uses the original 22k labels, they would surly mention it). So basically ImageNet=ImageNet-1k.

Regarding downloading the dataset. Since you are downloading ImageNet for your personal usage (GAN training) and not to participate in one of the challenges, it doesn't really matter, so just download the latest dataset labeled "Download links to ILSVRC2017 image data".


Overfitting and cross-validation In machine learning, an overfitted model fits training set very well but cannot generalize to new instances. I evaluated my model using cross-validation and my accuracy drops when setting the maximum number of splits of my decision tree beyond a certain number. Can this be associated with overfitting?

UPDATE
Here is the result. I am doing Human Activity Recognition in MATLAB with 16 features.

It definitely can be associated with over-fitting. I would suggest plotting the training and validation accuracy in a single graph. So you see how both depend on the number of splits. If it is over-fitting one would expect that the training accuracy continues rising, while the validation accuracy gets to a maximum and then drops.

However, you have to make sure that your training and validation sets are OK. I assume you do something like k-fold cross validation. Where every sample is k-times in a training set and one time in a validation set. And you have enough samples to train your model etc.


What cost function and penalty are suitable for imbalanced datasets? For an imbalanced data set, is it better to choose an L1 or L2 regularization?

Is there a cost function more suitable for imbalanced datasets to improve the model score (log_loss in particular)?

If you have an imbalanced dataset you usually want to make it balanced to begin with, since that will artificially affect your scores.
Now, you want to be measuring precision and recall, since those can capture a bit better the imbalanced dataset biases.
L1 or L2 won't perform particularly better in a balanced or unbalanced dataset, what you want to do is call elastic nets (which is a combination of the two) and do cross validation over the coefficients of each of the regularizers.
Also, doing grid search is very odd, you are better using just cross validation and see what parameters work better.
They even have ElasticNetCV, which does that part for you


How to find the driver features towards a particular result in Classification problems In a classifier model, we can predict the outcome class, but here I need to find out the features that drive towards a particular result in a classification problem, that are a strong indicator of a particular result eg: driver features for loan default or features indicating for a successful sale and so on. Is it the same as finding the important features for the classifier model, if not, how do I proceed towards this. Also which model could be the most useful in such a case. Thanks in advance.

If you want to know what features are important to predict some outcome, you speak about feature importance (FI). FI can differ between models, e.g. between a random forest and logistic regression. This is because the models take a different perspective on how to predict some outcome.

A mathematically sound way to assess FI in binary classification would be to run a logistic regression with a lasso or ridge penalty. This is easy and the results (the log odds) can be directly interpreted. Larger positive or negative values indicate a larger positive or negative FI et vice versa.

Here is a good R example: https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html

See „Introduction to Statistical Learning“ Ch. 6.2 for some background and demo: https://www-bcf.usc.edu/~gareth/ISL/


The right join for the right situation? (Please pardon the pun in the question title.)

I'm currently teaching an introductory class to data science.

I understand the differences between
a left join, a right join, an inner join and a full join.
However, I am not sure how to explain to students when
a particular join (say an inner join)
is the appropriate join for that particular situation.

I am looking for three real-life examples:

In example A, an inner join is the most appropriate.
In example B, a left/right join is the most appropriate.
In example C, a full join is the most appropriate.

I don't need all the details,
but I do want the big picture ideas that show that
in that real-life example, this join is most appropriate.

Note: I am teaching using the R package dplyr,
so I have used the names of the dplyr join functions.

You're an online retailer. Like Amazon.
You keep your purchase data for different categories of items in different tables, but all website users have one account with one ID.

Inner Join:
You have two datasets, one with User IDs and purchases of clothing data, the second dataset has User IDs and purchases of books data. You want to find out who purchases both clothes and books from your site.

You inner join to find the User IDs and purchases for people who bought clothing and books. Any users who didn't buy both of these items will be 'dropped' from the final table.

Left Join
You have one dataset with User IDs and account info(e.g. age, name) of all users. You want to build out this table to include some sales data, without dropping people from your complete list of users. So you would left join your purchase dataset to your account dataset (with User ID as the key). Any user who has made no purchases won't be dropped from your final table.

Right Join
The opposite of left join. E.g. you want to keep all purchase data, even if for some reason your Account Info table does not contain the ID of the user who made that purchase.

Full (Outer) Join
You want to Join account info and purchases tables, but you don't want to lose any entries in either table. So your final table will include Users who have made no purchases, and (perhaps mysteriously) purchases who have no users associated with them.


Emotion Recognition with Multi-task Learning I am beginner in Data Science and currently working on a learning project aimed at emotion recognition from a bio-medical sensor dataset. I want to use a Multi-task Learning to generate the output of three emotion levels - Arousal, Dominance & Valance.
The dataset consists of 8 sensors data from 20 subjects, here I have attached a screenshot of very small part of the dataset to give youbetter insight about it:

So as you can see, the Dataset is about:

Multiple subjects and for each subject there are 11 columns of data.
Columns 1-8 are raw sensor data (EEG, EOG, GSR, Skin Temperature, IR Response,... ) at 250 Hz.
Columns 9-11 are the corresponding labels.

I suppose your question is .... "How to do multi-task learning?"

Should you take a neural network approach, for instance:
A simple example with the above diagram is to pass Columns 1-8 into the "Shared Layers".
The task specific layers would each be related to your three targets
ie. Task 1 - Arousal,
Task 2 - Dominance &,
Task 3 - Valance.
Do have a look at this review article on multi-task learning and its various types. A more accessible article with some code examples here.


Does statsmodels compute R2 and other metrics on a validation-/test- set? Does statsmodels compute R2 and other metrics on a validation set?
I am using the OLS from the statsmodels.api
when printing summary, an r2 and r2_asjusted are presented.
I did not trust those 0.88 and computed an own adjusted R2 with scikit-learn r2_score and the adjusted r2 function from this answer resulting in 0.88 as well. So the question arose.

No.
There's another question over on Stack Overflow concerning wrapping statsmodels models to use the sklearn conventions, so that you can use sklearn cross-validation, hyperparameter searches, etc.


Extending a trained neural network for a larger input I have a seq2seq conversational model (based on this implementation) trained on the Cornell movie dialogs.
Now I want to fine-tune it on a much smaller dataset. The new data comes with the new words, and I want UNKs for as few new words as possible. So I'm going to create a new network with respect to the new input/output sizes, and I'm going to initialize its submatrices with learned weights I have at hand.

Could you say if this method can cause problems with the resulting model's performance? E.g. are the softmaxes likely to be affected significantly with these new initially untrained weights?
And if it's OK, do you have some examples on how to do it with the least pain in tensorflow's seq2seq setup?

Its okay as long as the nerwork you are planning to create has the same number of layers and units i.e the dimensions of your network must be compatible with the weights that you are borrowing from the trained model. Also it would be better if you follow the second blog post of suriyadeepan practical seq2seq where he trains a conversation model on twitter chat. The code is much simpler and easier to understand, also it is on a smaller dataset, also he mentioned that the bot trained on cornell movie dialog corpus wasnt performing so well. Mainly to use the pre-trained weights all you have to do is load the model, create placeholders for the weights, assign thr weights from the loaded model to the placeholders and run a forward pass. This blog and this question might help you with this task


Vertical and horizontal lines appearing on large confusion matrix? I have produced a large heatmap-like confusion matrix and am seeing horizontal and vertical lines on it, so I'm trying to determine:

What they mean
Why they are there
How I can improve on this

Approach

I am relatively new to ML and in the early stages of of a multi-class text classification problem.  I may be a little verbose so you can ensure I'm on track and my question isn't due to a flaw in my approach.

I have 90,000+ samples that I'd like to be able to classify into one of 412 classes.  I've taken a basic look at the data in terms of its class distribution and the unigrams and bigrams that are selected for each class.  Continuing exploration, I trained 4 classifiers on the data, receiving the following levels of accuracy:

LinearSVC                 0.547190
LogisticRegression        0.530063
MultinomialNB             0.368121
RandomForestClassifier    0.200568

Having had a lot of trouble plotting a confusion matrix this large with Seaborn or Matplotlib, I used used the following python code to produce a confusion matrix in CSV:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split
from sklearn.svm import LinearSVC

def make_confusion_matrix(a,p,c):
    cm = pd.DataFrame(0,index=c,columns=c)
    for count in range(len(p)):
        cm[int(a[count])][int(p[count])]+=1
    return cm

tfidf = TfidfVectorizer(sublinear_tf=True, min_df=5, norm='l2', encoding='latin-1', ngram_range=(1, 2), stop_words='english')
features = tfidf.fit_transform(df['DetailedDescription'])

model = LinearSVC()
X_train, X_test, y_train, y_test, indices_train, indices_test = train_test_split(features, df['BreakdownAgency'], df.index, test_size=0.33, random_state=0)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

cm = make_confusion_matrix(y_test.tolist(),y_pred,labels_df['TOOCS Breakdown Agency'])
cm.to_csv('ConfusionMatrix.csv')

I was finally able to view the confusion matrix in a heatmap style by using Excel conditional formatting, which produced the matrix above.

Interpretation

Given that the X axis is actual and y axis is predicted:

I interpret the horizontal lines as showing incorrect bias of predictions towards a class with a disproportionately large number of samples?

I interpret the vertical lines as showing incorrect predictions away from a class with a disproportionately large number of samples?

Does this show that the model is both overfitting and underfitting the data?
Or that the samples within my classes are overly diverse?

Action

I'm contemplating:

Manually adding samples to the classes that have very few (a minimum of 10?).
Using SMOTE to oversample small classes (knn=6).
Potentially removing some samples that are atypical or incorrect.

Any help on my Interpretation or Action would be greatly appreciated!

If the $x$-axis is actual and the $y$-axis is predicted, then vertical lines means a given input of the class $x$ is not being discriminated sufficiently and as a result is randomly mapped to a number of classes. Whereas a horizontal line signifies that many classes are mapped as belonging to a particular class.

Class imbalance

Many reasons can lead to these lines in your confusion matrix. The most common is class imbalance. If you train your model using data that is not well balanced across the different classes this will result in both horizontal and vertical lines. Consider a model trained to distinguish cats, dogs and foxes with 1000, 1000, and 10 instances each. Evidently, you can see how many future instances of foxes may be classified as dogs or cats.

You can remedy this by obtaining more data. This is often not possible, so you can try and weight the model to account for the class imbalance. You can do this by weighting the loss function. Or by oversampling the underrepresented class. Furthermore, you can try and synthesize new instances of the underrepresented class. This can be done with a GAN (hard, requires lots of data, unstable), or you can use k-NN type techniques to find out where the data is concentrated and add novel points to add density to this set. The new technique SMOTE is also highly recommended for its simplicity.

Overlapping distributions

Another big reason is a distribution that overlaps. This is when a class' distribution encompasses one or more class distributions. For example, if you are classifying between cats, dogs, shiba inu, corgi. You can see that any amount of training will always cause conflicts between the class dog and shiba inu, corgi. Since these are dogs.

This problem is easy to distinguish for human perceptive classification, like images, but often times we are working with data where the span of these distributions is not as obvious. You can use clustering methods to see if certain classes fall within the distribution cloud of others. I use percent overlap as a metric for this task.


K-means clustering on big data stored on multiple nodes on HDFS I have a very basic question about how the k-means clustering algorithm works when the data is stored in HDFS.
Does the training take place on the subset of data on each node for a fixed value of K( say 3)? But then we will have 3*number of nodes centroids?

How would it be different if we applied linear regression? Will it only train on data stored on each node locally?

Posting details from the link in comments above.

The data can be partitioned among multiple processors (or streams or threads). Each processor can read the previous iteration's cluster centers and assign the rows on the processor to clusters. Each processor then calculates new centers for its of data. Each actual cluster center (for the data across all processors) is then the weighted average of the centers on each processor.

  In other words, the rows of data do not need to be combined globally.
  They can be combined locally, with the reduced set of results combined
  across all processors. In fact, MapReduce even contains a "combine"
  method for just this type of algorithm.

  You postulate a shared file that has the centroids as calculated for
  each processor.  This file contains:

The iteration number.
The cluster id.
The cluster coordinates.
The    number of rows assigned to the cluster.

This is the centroid file.

An iteration through the algorithm is going to add another set of
   rows to this file. This information is the only information that
   needs to be communicated globally.

  There are two ways to do this in the MapReduce framework. The first
  uses map, combine, and reduce. The second only uses map and reduce.

link with all details


How to use Machine Learning to discover important biomarkers in an unbalanced small data set I have a project which I am just starting out, I am only just learning machine learning and statistics so I am somewhat unsure as to what approaches will be good to start off with, and I am sorry if this does not belong here.

The data set is of  different patients carrying a certain disease and each patient has different biomarkers and  physical measurements such as heart rate at different time points, until death, if they do die. I was told that the goal was to identify the key features, which would be associated with a a patient dying.

I only have 33 patients, and only 16 of them have died. But disregarding patient the biomarkers came from I have 300 odd time slots, I first tried to approach it as a binary classification problem,  classifying the 'death' point from other points. The problems were:

The data imbalance and,
How to you interpret the models to discover most important features.

For imbalance, I tried SMOTE oversampling with didn't work as I thought, then I randomly under-sampled, which gave decent results but the data set was even smaller, so I wasn't sure if its a good idea.

Simple binary classification models like Gaussian Naive Bayes and Logistic Regression did okay even with the imbalanced data, but they don't (at least as far as I know) give a way to discern feature importance..

So my main questions are:

What's the best way to approach this problem, or in general what kind of approaches work when you want to identify most influential features (data measurements).
If I do want to approach it as a binary classification problem what approaches can I take to combat class imbalance?

If your goal is to identify important features, I would say go for a Decision Tree which inherently calculates importance/separation capability of the features while selecting them for splitting the internal nodes. You can also go for an ensemble of decision trees such as RandomForest which will return feature importance based on their average impurity reduction throughout all its trees.

This article can help you set up a basic experiment.


What's the difference of stateless LSTM and a normal feed-forward NN? From what I understand, the whole point of LSTM is for the network to establish long-term dependencies in the data, i.e. an event happening now may be in some way determined by something that happened sometime in the past, and that may not be in the batch of data currently being presented, but many sequences or batches previously.

So I understand that a stateful LSTM does this, and the network will retain the state until reset_states() is called.  But with stateless LSTM, where the state isn't retained between batches, then how does that differ from a normal feed-forward perceptron network?

I'm assuming that even though in stateless LSTM you don't control the states as you would in a stateful model, that some state is still retained at passed on between sequences?  Is it passed on between batches?

I can understand why someone would use stateful LSTM, but why would someone use stateless LSTM instead of a feed-forward perceptron network?  Or vice-versa, why would someone use a regular FF instead of a stateless LSTM?

Theoretically a stateless LSTM gives the same result as a statefull LSTM, but there are few pros and cons between them. A stateless LSTM requires you to structure your data in a particular way, in turn it is vastly more performant, while a statefull LSTM you can have varying timesteps, but at a performance penalty.

The stateless LSTM does have state, it's just implemented differently. Instead of managing it yourself by constantly calling reset_states() your data is structure in such a way that it is automatically reset when the end of one series of timesteps is done.

Let's create an explicit example. Say you have some label data y[i] where i is the index, and input data x[i, t] where i corresponds to your label's index, and t is at what timestep this was observed. (The x and ys can be vectors, matrices, or any type of tensor really.)

Say you have 4 output labels (not 4 classes), and 2 to 4 timesteps for each of them. Your data might look like this:

                   x[0, 0], x[0, 1]  ->  y[0]
          x[1, 0], x[1, 1], x[1, 2]  ->  y[1]
 x[2, 0], x[2, 1], x[2, 2], x[2, 3]  ->  y[2]
                   x[3, 0], x[3, 1]  ->  y[3]

This is the kind of data a statefull LSTM can handle. But at the end of each label, you would have to call reset_states() to indicate that your no longer dependent on the previous series of xs. This is slow.

With a stateless LSTM you would pad the x so that you have a rectangular matrix:

 0        0        x[0, 0], x[0, 1]  ->  y[0]
 0        x[1, 0], x[1, 1], x[1, 2]  ->  y[1]
 x[2, 0], x[2, 1], x[2, 2], x[2, 3]  ->  y[2]
 0        0        x[3, 0], x[3, 1]  ->  y[3]

In this case, there is an implicit reset of state for each label. This way of doing it is vastly more performant. But there might occur issues with bogging down your data with empty data. Personally I tend to add a binary flag to each x vector to indicate to the network in the LSTM that it should probably ignore the data.

I'm not sure about the specifics of how stateless LSTMs are implemented, but I think it might be about unrolling the LSTM units.

Ultimately, both stateless and statefull LSTM's do the same thing, they are just implemented differently.

  why would someone use stateless LSTM instead of a feed-forward perceptron network?

Well, ironically it's because stateless LSTM have state, which means it can take into account the previous timestep input values when doing predictions.


When to use mean vs median I'm new to data science and stats, so this might seems like a beginner question.

I'm working on a dataset where I've user's Twitter followers gain per day. I want to measure the average growth he had over a period of time, which I did by finding the mean of growth. But someone is suggesting me to use median for this.

Can anyone explains, in which use-case we should use mean and when to use median?

It depends what question you are trying to answer. You are looking at the rate of change of a time series, and it sounds like you are trying to show how that changed over time. The mean gives the reader one intuitive insight: they can trivially estimate the number of followers at any date $d$ days since the start by multiplying by the mean rate of change.

The downside to this single metric is that it doesn't illustrate something which is very common in series such as this: the rate of change is not fixed over time. One reasonable metric for giving readers an idea of whether the rate of change is static is giving them the median. If they know the minimum of the series (presumably zero in your case), the current value, the mean and the median, they can in many cases get a "feel for" how close to linear the increase has been.

There is a great cautionary tale in Anscombe's quartet - four completely different time series which all share several important statistical measures. Basically it always comes back to what you are trying to answer. Are you trying to find users which are likely to become prominent soon? Users which are steadily accruing followers year by year? One hit wonders? Botnets?

As you've probably guessed, this means it's not possible to universally call mean or median "better" than the other.


What is an appropriate machine learning technique to analyse development of status over time? I have a dataset as follows (not the actual data, but representative):

User   Age  Salary  Status_2000  Status_2001  Status_2002  Status_2003  ... Status_2019
John   30   10000   A            A            A            B            ...  A
Mary   25   20000   A            B            B            B            ...  C
Jacob  35   30000   E            F            F            F            ...  A
Jenny  28   22000   A            F            F            A            ...  F

I have over 50,000 rows and I want to perform an analyse the development patterns of the Status from 2000 to 2019. However, the Status are encoded with letters from A to J. I would like to ask whether there are any machine learning methods or clustering methods which could help in this analysis? I initially thought of K-Means but I'm not sure if that is appropriate even with Label Encoding.

In the same dataset, there are also the following columns:

User   Age  Salary  2000_%  2001_%  2002_%  2003_%  ... 2019_%
John   30   10000   0%      0%      20%     25%     ... 100%
Mary   25   20000   0%      10%     40%     60%     ... 80%
Jacob  35   30000   0%      80%     100%    80%     ... 80%
Jenny  28   22000   0%      0%      0%      20%     ... 60%

where the data represents % of the maximum amount from 2000 to 2019, and I want to perform an analysis on the development of the %. Are there any techniques suitable for such analysis?

Thank you all for your help in advance!

Yes, K-means is an option. But you need to do some data processing first.

My first concern is about quantifying those statuses (are they linearly depended, like does B means something in the middle of A and C)? then numerate them (A to 1 and B to 2 and C to 3 etc)
Represent your data set in a dimensional way, melt years columns into three columns

Year (with values of 2000, 2001, 2003 etc) so you get a time series dimension.
Year_status (status value).
Year_percentage (percentage of the year).

Try to treat pattern by moving window (kernel), and try different sizes.. for example size of 3 will cluster every 3 followed statuses into patterns centroids (for example.. having a common occurrence of A, E, A)


Using t-SNE to track progress of a word vector embedding model. Pitfalls? I've been training a word2vec/doc2vec model on a large amount of text. I recently stumbled across the t-SNE package, and am finding it wonderful at finding hidden structure in high-dimensional data.

Can t-SNE be used as a way of tracking the progress of a hard machine learning task like this - where the model's understanding goes from unintelligible nonsense to something with hidden structure?

I have seen examples of the MNIST data set on t-SNE where all the individual numbers cluster well with each other. (as explained in this answer)

As I increase the number of vectors in the doc2vec model and the size of the training set, I start to see clumping (if you squint) in the t-SNE plot. So far, these clumps are mainly associated with posts of very similar wording (one clump is mainly "Good morning/evening!" tweets). (Picture was generated with perplexity of 400)

How much additional clumping can I expect to see as the model is improved? Is this indicative that the model is, in fact, improving and learning deeper connections between words/phrases? Or have these t-SNE plots settled into the form they'll always take?

EDIT: I have since realised that the apparent lack of clumping could be due to the data itself. MNIST separates out cleanly because there are generally no weird glyphs that look like halfway mutations between numbers. My dataset (twitter sentiment, 1.6 million tweets) is, for a lack of a better word, filled with unclassifiable drivel, and it seems entirely probable that the homogeneous forest of points in the centre of the plot represents these sorts of tweets.

T-SNE is extremely useful for visualizing high-dimensional data in lower-dimensional space. However, t-SNE can have several gotchas, including comparing cluster sizes. The t-sne algorithm tries to even out cluster sizes by expanding dense clusters and contracting sparse clusters. Thus, it is not straightforward to directly compare clusters across different runs.

"How to Use t-SNE Effectively"  goes into greater detail about common pitfalls of using the technique.


Recommending products to buy I have the following dataset where each row represents a customer, and each column is a product. So for example, the first customer (row 0) buys product 2 and product 7. The second customer (row 1) buys products 3,6 and 7. And so on...

Basically I want to recommend customers who are already buying certain products, some other products. Here is an example:

Let's say I choose all customers who are buying products 1 and 4. How would I recommend them the next best...let's say 3... products based on what other 1 and 4 customers are buying?

I'd like to use keras neural networks or random forests for this kind of task. Thanks :)

product 1.......product 2.........product 3....... product 4...... product 5........ product 6..... product 7

This is a classic case of recommendation problem.
There are a couple of steps in which a recommendation is made:

Candidate Generation - What to show to a customer?
Candidate Ranking - How to show the items to the customer?
Personalisation - How to show the items to each customer?

Step 1 and 2 are general that means they talk about the overall trend and customer base, meaning we perform these steps by saying in general what would happen.

Step 3 is more personalised, where the ranking (order) in which the items are shown can be different for different customers.

For starters you should experiment with Singular Value Decomposition and Matrix Factorisation for recommendation.

If you have already tried your hand at these, then you should go with word2vec or doc2vec for building the step 1 and 2 and some kind of neural network at the end that ties to it in the end.


Change priority of cell execution in jupyter notebook? I am sometimes in the following situation:

I want to execute two cells: Cell 1 takes on the order of 10 minutes or hours. Cell 2 will take 1 second.
I'd rather execute cell 2 first and see the result, but I didn't know that beforehand and hence already started cell 1.
cell 1 is already significantly into the computation, so it would be wasteful to just abort it.

How do I first pause cell 1, then start cell 2, and then start cell 1 again? Or even better, how do I pause cell 1, start cell 2, and have cell 1 automatically start again when cell 2 is done?

I don't think Jupyter notebooks (even via extensions) currently offer pausing/restarting cell blocks. I would suggest putting the code of both cells into a single cell and using Python logic to determine the order of execution.

In general, however,  you cannot strictly pause the execution of something and come back to it later. If you abort the execution of the function, the intermediate results are lost (because they are stored within the namespace of the function you effectively just killed). This is because Python only runs one single process at a time (key term: Global Interpreter Lock).

The only thing you might be able to do is create a cell 0, which performs some kind of check, testing how long cell 1 might take to run, then just put cell 1 and cell 2 in and if/else construction that gives the correct order to use.

Other approaches

You might want to look into something like the multiprocessing library. There you can create a group of worker processors, to which you can send the contents of cell 1 and at the same time the contents of cell 2. They will be computed at the same time, using two different processes i.e. two different instances of the Python interpreter. This can be easy to achieve if there is no direct dependency between (in your case) cell 1 and cell 2. Multiprocessing is particularly useful in the case that your long-running cell is compute bound, meaning is has to perform a large computation.

Yet another option would be to investigate the threading module and concurrent programming in general, but this get a little more complicated and is probably beyond what you want in your situation (it also requires more effort to get working that multiprocessing). This allows Python to kind of do two things at once, but with shared state - so each running thread can change variables that the other thread might also be changing, which can require a lot of work to make safe. This approach is however beneficial when you tasks are IO bound, i.e. there is no big computation, but rather you send data e.g. to a website and wait for a return - most your time is spent waiting.


How can I approach this problem? Let's say I have a dataset with pricing information for the same flight during the past year. So, for a flight departing on day D, I have the available price from D-130 up to D (departure day). Then the same for flights on the other 365 days of the year for the same departing time every day. Does this make sense? I want to see if today's price for any departure day in the future is higher or lower than expected, hence if it's expected to rise or not.

Plotting this data on a chart, the X axis are the days prior to departure, and the Y axis the price, I get the following:

How can I shape the data so I can train a model for price prediction? I can't see a clear trend. The price just below 90 means that it's the max price and it's not discounted.

You certainly need to add at least one other variable representing the time of year, because from your graph it's clear that the fare can't be predicted accurately using only the time until departure: for the same day you can have many points representing different fares. That makes sense, since the fares are going to be very different depending if the flight occurs during holiday season or not. There might also be other variables of interest but since it's always the same flight I can't think of any.

I would suggest plotting a graph where x is the time of year and the time before departure is represented for instance with colour or gradient, that should make things clearer.


How to store complex tables and structures? I'm wondering about general approaches to storing complex tables and structures. For example, imagine I have a table like this:

               A1 A2 A3 A4 A5 B1 B2 B3 B4 B5 C1 ....Z5
 individuals
 1             .  .  .  .  . .  .  .  .  .
 2             .  .  .  .  . .  .  .  .  .
 3             .  .  .  .  . .  .  .  .  .
 4             .  .  .  .  . .  .  .  .  .
 5             .  .  .  .  . .  .  .  .  .
 ...           .  .  .  .  . .  .  .  .  .

(I'm just showing the row and column names (variables), but the table would be filled, or could be even much larger, with A1 to A999, B1 to B999 colums.)

I can want to perform some calculation on all $Ax$ columns, or on all $x3$ ones. Since it can be difficult to deal with so many columns, it may be better to write this as:

 i  subcase  A B C ... Z
 1  1      .  .  .  .
 1  2      .  .  .  .
 1  3      .  .  .  .
 1  4      .  .  .  .
 1  5      .  .  .  .
 1  6      .  .  .  .
 1  7      .  .  .  .
 1  8      .  .  .  .
 1  9      .  .  .  .
 2  1      .  .  .  .
 2  2      .  .  .  .
 ...       .  .  .  .

I remember seeing that kind of structure several times, sometimes referred to as long format. Later, one can perform arbitrary operations on all rows meeting some condition on "case". This has some advantages, but could be confusing.

One could average or crate the differences of all the $\text{subcase}==1$ rows, only from $\text{individual}==1$, or when $\text{individual}==subcase$,  for example.

But what would be the best way to store the resulting numbers--columnwise or rowise? If I wanted to store the average of all $\text{subcase}==1$ rows, I would need to repeat the number on every row meeting that condition.

Another option would be to keep the wide format structure, but having 2 rows for the headers—one for the letters and a second one for the numbers. I think it could be a natural way, but I've never seen it before.

Are there other structures people can recommend for this problem, or for more complex situations, such as trees?

In third normal form

IndvidualID
CatID
CaseID
Value

Primary Key:  IndvidualID, CatID, CaseID

If you need to constrain CatID and/or CaseID then have a FK

From that structure you can easily loop and create long format or multidimensional array.
It is also space efficient as a null is simply no row.

In sql an avg

select CatID, avg(value)
  from table
 where CaseID = x
 group by CatID

Or you can just loop on those same constraints
Or a tool like LINQ


Applying machine learning to cache algorithm I have an application that receives ~10k of requests per day; each request has multiple parameters and goes through a pipeline with multiple steps and they finish in about 1 hour.
Let's say the requests are like bake_cake [flavor=chocolate] [topping1=strawberry] [topping2=cream]

Given a history of requests from previous days, is there a ML framework that can help me predict the "100 most like requests to arrive today" so I can cache their results? Or some other similar strategy that can help me delivering part of those requests faster?

Ideally it would be based on how much I gain by delivering faster, how much I lose by processing a request that might not come, etc.; but for now even simpler algorithms could be of much help since currently there's no optimization.

You need to validate your assumptions based on data driven analysis

Types of requests received on a day
Types of requests received on a holiday
Pattens for different times (Morning / Evening / Night)
Seasonality / Monthly / Weather based patterns

Post Analysis

Do you see correlation from Data & patterns
Do you see correlation from Data & Location
Do you see correlation from Data & Age / Gender / Locality

ML Models

Do you see correlation from Data & patterns
With the variables we can identity and model a forecast algo

ML Model is not the start. It is the outcome of detailed data and domain driven analysis


Modern Feature Selection Review/Resources I found this review paper by Guyon and Elisseeff in a 2003 JMLR publication but, although not outdated, it is quite old. Is there a more recent review or resource on the topic of feature selection?

JMLR Review 2003

I looked into it quite recently and found these papers:

Saeys et al. (2007)  (also already a bit dated, but a good overview, and I think the 'key' paper in this area).
Dougherty et al. (2009)  (interesting paper discussing various difficulties such as classifier dependence, label distribution and sample size).
Bolòn-Canedo et al. (2014) .
Dessì & Pes (2015).

However, I do agree with you that there is a need for an updated review regarding this topic.


Association Rule Mining across two market baskets I am quite familiar with Association Rule mining but I need to use it to associate ACROSS two market baskets instead of finding support WITHIN a market basket.

Imagine customers come to a Store A and buy a certain number of products. The same customers go to Store B and buy another set of products. I want to associate between the two Stores and not within the Store.

So I want to make "A --> B" statements like

  "Customers that Bought x and y from Store A also purchased z from
  Store B"

I could lump all the purchases as one market basket and run some association mining algo on it but in that case the association algo will not control for where the items were purchased.

Clearly one alternative is to find all rules using one market basket and then exclude the ones where the A isn't pure in the A -> B relationship.

Any other ideas would be great.

Option 1

What a good option can be is to group those purchases into individual itemsets and explore by customer.
Consider the following example:

Customer 001 goes to store A and buys: {Banana, Milk}. Then he goes to store B and buys {Eggs, Oranges}.
Next, call {Banana, Milk} as product 01 and {Eggs, Orange} as product 001.

Customer 002 goes to store A and buys: {Banana, Milk}. Then he goes to store B and buys {Apple, Watermelon}.
Now, using the previous customer's data, you see that {Banana, Milk} already exists and you also call it product 01, whereas {Apple, Watermelon} is a new product that you call product 002.

You perform this for all of the itemsets for all customers.

In the above example, your Customer 001 total receipt includes: product 01, product 001. Your Customer 002 total receipt includes: product 01, product 002.

Note: product with one leading zero (01) comes from store A, product with two leading zeros (001, 002) comes from store B.

Now you can perform the regular Market Basket Analysis using association rules in Python and get the rules you are looking for by identifying store products with leading zeros.

Option 2

Your other option is to encode products before hand. Using the above example, Banana and Milk can be coded as 1A, 2A (from store A). And Eggs, Orange, Apple, Watermelon can be coded 1B, 2B, 3B, 4B (from store B).

Then you combine it all in one dataset, and create a unique receipt number including all items so they are on one transaction.
For Customer 001, it will be {1A, 2A, 1B, 2B}, for Customer 002, it will be {1A, 2A, 3B, 4B}.

Now you can perform the regular Market Basket Analysis using association rules in Python and get the rules you are looking for by identifying store products by character A or B to find in which store a customer made a purchase.


What are some options to add or remove nodes from a multiclass classification model? I'm building a classification model that will need to classify into one of many possible outputs. I know in advance that I will need to add and subtract nodes from the output layer as circumstances change. Please refer me to any resource you are aware of that can help me understand potential approaches. I can retrain the model from scratch every time, of course. I'm looking for alternatives. I am only aware of two approaches:

Transfer learning: Only retraining the final layer with the new desired outputs.
Stacking one binary classifier for every potential output and streaming my data through all of them.

I need a level of specificity that is pretty granular, for example, I'd like to be able to identify a document that is primarily focused on a topic like "Changes in goat-herding practices in Uzbekistan." This is a made-up requirement, but the point is that it may have very specific features. Please provide input on the pros and cons of the approaches that I mentioned and add any approaches that I haven't thought of.

@DPCII, I don't think modifying output nodes at runtime will help you. This is because,

A neural network is trained on a specific dataset and to predict predefined variables only. In backpropogation ( used for optimization ), the gradients are for each weight and bias ( or any other parameters ) are calculated using the loss function ( also the objective function ).

For the changing the number of output nodes, you have remove some the connections coming from the previous layer which is $L_N$, suppose. But during training, the parameters of $L_{N-1}$ were optimized using the gradients of the activation functions of $L_N$.

Hence, removing output nodes at inference time would affect all the previous layers which in turn will affect the final output ( of the remaining nodes ).

Instead, I would suggest, to construct a neural network which predicts all the possible classes. Train this NN on various samples wherein you'll require the predictions of some desired nodes. Let me explain this by an example.

Suppose, we have a neural network which predicts classes $C_1, C_2,C_3,...,C_N$. For a specific sample, we only need the predictions for classes $C_1$ and $C_2$ only, given a sample $X$. For this sample, we set the label as $[ y_1 , y_2 , 0 , 0 , 0 , ... , 0].$ We only set the values for classes $C_1$ and $C_2$ and rest all the classes are set to zero.

Similarly, we assign labels for various samples and set only those values whose predictions are needed. Rest all values are set to zero.

This is actually the common classification we do for images. For a given sample of a cat, I would like to predict an array of $[1,0]$ and for a sample of dog, $[0,1]$.

Hence, we could apply this ideology instead of removing nodes and disturbing the NN.


What are some good models to test the speed of a data science machine? I'm writing a battery of tests (in Python) for the purpose of measuring the speed of my company's different computational instances. The goal is to see how fast different AWS EC2 instances are at running different ML models or common data science tasks.

I'm reaching out to ask if anyone knows of any time-consuming (but realistic) models that can be built using only the standard Anaconda packages and either the built in datasets, or a popular dataset known in the industry?

The goal would be to give end users a sense of how much computational power they need, and I think using some popular data sets or models to compare relative time to completion would be the best way for them to choose the right amount of power for their needs.

MLPerf is a set of benchmarks designed just for that purpose. It tests a variety of common machine learning tasks across a variety of possible hardware.


How should I evaluate writing quality to compare two articles(which article is better suited/written for a topic ) according to their content? I am trying to come up with a platform which can Synthesize Quality Content among many articles found about some particular "topic" on the internet, the algorithm should be able to say recommend top 10 articles found on the internet about the topic. What algorithms and methods should be used to evaluate writing quality and give some kind of a score on many other parameters like "source credibility". I have taken care of crawling and finding out the content, looking for algorithms and methods to evaluate this. Please suggest resources(course,research papers etc..) for the same.

"Source credibility" of Internet articles is best calculated through the Page Rank algorithm.

Algorithmically determining writing quality might be intractable. However Page Rank could be a proxy. If an article is a hub then it is the authority on the topic and can be assumed well written (or at least very useful).


Which metric to use for classify a binary logistic regression I still novice in machine learning. My problem is as follow :

I use two binary logistic regression models for making comparaison. Additionally to LogLoss and accuracy, I would like to use for metric : precision / recall/ F1-mesure.

I have used the formulat :

print (classification_report(y_test, y_pred.round()))

My question is: which one, for the tree metric value, should I used. is it the weighted average ore the parametres of the true values (number 1 in picture)

The performance measure should be related to the actual real world problem you face. (Imagine, you are a football trainer. You are seeking for young talents to introduce them to your very expensive training camp. It might be more important to you to prevent expenses on kids, that proof not to be as talented as hoped for, than reaching all hidden talents with the program. So you might be interested in strong precision of beforehand assessement. But if you dont have to care about funding, your priorities might change. You could be more interested in finding any last talented kid. Then you would be more interested in recall.)

If you want good probabilty predictions, use logloss. If you want the prediction score to separate very well among classes, rather use area under RO curve. Your choice should depend on your real world cost for missclassification or bad predictions.

The averages are merely an aggregation of the class specific statistics. This is mostly useful for multi class problems. For two class problems I recommend to look at the class specific statistics instead. Anywasy, how averageing is conducted, you can read here: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html


bad regression performance on imbalanced dataset My current dataset has a shape of 5300 rows by 160 columns with a numeric target variable range=[641, 3001].

That’s no big dataset, but should in general be enough for decent regression quality. The columns are features from different consecutive process steps.

The project goal is to predict the numerical variable, with the satisfactory object to be very precise in the area up too 1200, which are 115 rows (2,1%). For target variables above 1200 the precision can be lower than in the area [640, 1200]. The target-variable is normally distributed with its mean ~1780 (25%: 1620, 75%: 1950) and variance of 267.5.

prediction vs actual:

residual plot:

My problem is (see plots above), that no matter what I try, the range of predictions (y_hat) is very limited and rather random (Training RMSE ~300, Test RMSE ~450), best test-mean-abs-error for y-values <= 1200  ~= 120.

I’ve already tried:

feature cleaning
process step wise addition of features to compare model performance/information gain
feature generation
derive new features (by business logic)
generate features

cross-product of features
differences to previous rows
differences between features
differences per feature to mean
durations based on timestamps

normalizing, scaling
log-transformation of target variable
Over- &/ Under-Sampling
various algorithms (using GridSearchCV for hyper-parameter tuning):
sklearn [SVR, RandomForrestRegressor, LinearRegression, Lasso, ElasticNet]
xgboost
(mxnet.gluon.Dense)

What would be your approach? Do you have any advice what technique I could try or what I've probably missed? Or if it's more likely that the training data simply doesn't fit well on the target variable?

Your residuals are huge, which is not surprising, given that your data is very variable, a linear model may not be the best choice for this task. You could try transforming your data (log, sqrt) depending on the nature of your data to reduce the variability, but as I said, your variability is huge.

Alternatively you could try modeling the variance with a mixed model if it makes sense for your data, given some additional knowledge of some variable.

Other then that you could try a different algorithm for this task.


Why does categorical_crossentropy work in when input is not 1-hot encoded? I'm going through lessons on the REINFORCE algorithm to solve Cartpole/Pong/etc (using AIGym) and every one uses categorical_crossentropy as the loss function.
What's confusing me is that categorical_crossentropy expects a 1-hot encoded setup as its true parameter. The states being passed in are not 1-hot encoded, they're just the numerical representation of the state of the universe at that moment. Why does this work?

The algorithm converts your categorical labels into one hot encode before calculating loss. So you don't have to carry the burden.


Creative Analysis - What are the possible approaches? We're analyzing some creatives from past advertising campaigns and are looking for the best way to approach. Currently we've broken down each creative into 20+ objective characteristics, for example:

CTA button (Y/N)
Exclusive discount mentioned (Y/N)
etc

For each campaign we have two main data points: CTR and conversions.
What are some possible ways to approach analyzing this data? We're looking to find out how each feature affects CTR/conversions.

If you are looking to find effect of each feature on CTR then a correlation matrix can help you with it and you can do statistical test to prove it.
But if you want to find hidden causes then it a classical problem of prediction vs causation.
Suppose we find that giving discount help increase our campaign you may conclude that giving discount helps but it may actually happen that sales are usually higher that time of year and so giving more discounts necessarily doesn't help. Causal analysis can help us find hidden causes which will be hard to find.
For doing Causal Analysis you can refer Dowhy package.


LSTM claim found in paper I read the following claim in this paper https://arxiv.org/pdf/1803.06305.pdf:
In one gate/cell, $W_{∗x} x_t + W_{∗r} y_{t−1}$ can be combined/fused
in one matrix-vector multiplication by concatenating the matrix
and vector as $W_{∗(x r)}[x_t, y_{t−1}]$ (page 3).
How is this the case? I would have thought since $W_{∗x}$ and $W_{∗r}$ are different matrices, the computations would have to be done separately? And does this combination (if it's really possible) save computational time?

It's a usual trick in deep learning. Matrix multiplications can be done in parallel:
$$
(A.B)_{ij} = \sum_{k} A_{ik} B_{kj}
$$
since you have to do this sum for each component, then you can divide your computing time by the number of processes you use - i.e. compute these components several at a time rather than one by one. Assuming you use a naive matrix computation then you have a $O(\dfrac{n^3}{K})$ with K number of processes. That's why GPUs are so much better for deep learning (usually have hundreds if not thousands of cores) which allows you to divide by a factor of several 100s at worst the time to compute the matrix multiplication.
In this particular case, you can use block matrix multiplications (we expend the rules of matrix multiplications to matrices):
$$
\mathbf{}_{∗()}[_,_{−1}]= \begin{bmatrix}
  \mathbf{}_{∗} & \mathbf{0} \\
  \mathbf{0} & \mathbf{}_{∗y}
\end{bmatrix}[_,_{−1}]
$$
if you apply the traditional matrix multiplications (keeping in mind matrix sizes), you get:
$$
\mathbf{}_{∗()}[_,_{−1}]=\mathbf{}_{∗}_ + \mathbf{0} * _{−1} + \mathbf{0} * _ + \mathbf{}_{∗y} * _{−1} =\mathbf{}_{∗}_ + \mathbf{}_{∗y} * _{−1}
$$


Classification: Dealing with unknown class balance in the prediction dataset I'm working on a classification problem that predicts if a grant application will be accepted. The data I'm training on is from 2005 to 2008. I'de like to predict any time after 2008.

The issue I'm running into is that the ratio of successful grant applications differs over time. For example, the success rate in the training set is 20%. But depending on the period I use after 2008, the success rate could be anywhere from 10 to 30%. This results in under or over predicting.

Since the success rate of grant applications is not constant, how do I account for this in my model?

This phenomenon is called Covariate Shift and it can affect features,target, you name it.

This happens when the test data distribution is different from the training data. Or your problem is a time-series one where time is the biggest independent variable.

First case : No model can adapt to the target shifting in its behavior. You have to either wait until you collect enough data from the current times, and retrain a fairly satisfying model, or do online learning which means that every datapoint that you predict for , is fed into the model to be trained on. That way; if changes in the target occur, your model will get updated with every new datapoint.

Second case : Lets say your target variable has a tendency to go up every 2 years, and then goes down. This is an assumption i'm making. Your task is to detect that shift in time that affects your target and model it. You create a model that takes into account that every 2 years, your target goes down. and so on. It's what we call a time-series problem where time is the biggest contributor to the target's changes.

If the second case doesn't make sense to you, and doesn't relate to your problem, then look at first case.


Handling a regularly increasing feature set I'm working on a fraud detection system. In this field, new frauds appear regularly, so that new features have to be added to the model on ongoing basis.

I wonder what is the best way to handle it (from the development process perspective)? Just adding a new feature into the feature vector and re-training the classifier seems to be a naive approach, because too much time will be spent for re-learning of the old features.

I'm thinking along the way of training a classifier for each feature (or a couple of related features), and then combining the results of those classifiers with an overall classifier. Are there any drawbacks of this approach? How can I choose an algorithm for the overall classifier?

In an ideal world, you retain all of your historical data, and do indeed run a new model with the new feature extracted retroactively from historical data. I'd argue that the computing resource spent on this is quite useful actually. Is it really a problem?

Yes, it's a widely accepted technique to build an ensemble of classifiers and combine their results. You can build a new model in parallel just on new features and average in its prediction. This should add value, but, you will never capture interaction between the new and old features this way, since they will never appear together in a classifier.


Is it possible to make a CS:GO Machine Learning AI? I am not an expert on Machine Learning, Neural Networks or NEAT. In fact, I probably have no clue what I'm talking about. My question is if you can make a learning AI that learns to play complex multiplayer games and possibly outpreform humans. If it is possible, could you also recommend a language or languages to make this AI in? (I know I'll probably have to take a VACation for botting, but it's something I feel like we should try.)

The answer is yes:

Example of a neural network outplaying human players in DOTA.  I haven't been able to find much regarding what kind of neural network but here is what is on the OpenAI website.  If you're a beginner you can learn some basic architecture and design principles in neural networks using Python and Keras (a neural network library).


How to compensate for class imbalance in prediction model? I'm trying to run a prediction model on a customers' data set to predict the likelihood that a new customer would be interested in buying product X, offered by a company that sells products X,Y and Z. E.g. would this guy, non-customer, of this age and salary, be interested in product X?

To train the model, I have a basin of 100K company customers, of which only 5K bought product X - the remaining 95K bought other products. Any prediction model guesses 'nobody will buy product X' accepting those ~5% false negatives.

How can I compensate for this skewness of the data? i.e. 95% vs. 5%? Thanks

There is a lot of information and techniques for rare event or imbalanced classes. Sorry to post links but (https://stats.stackexchange.com/search?q=rare+event and https://stats.stackexchange.com/search?q=imbalanced) but I do not want to duplicate all of the other work.

In my experiences, often I left the data set alone. I usually got good results. If that is the true ratio, then my model should know that. Sometimes I downsampled the majority class. I played with SMOTE (https://arxiv.org/pdf/1106.1813.pdf) before as well.

For your questions above, yes - changing the data with weights or over/unsampling may bias the results.  Need to check. If you downsample, make sure you are not tossing out signal from the majority class.


What is the difference between AI, ML, NN and DL? What is the difference between the following four categories:

Artificial Intelligence (AI)
Machine Learning (ML)
Neural Network (NN)
Deep Learning (DL)
Data Science

My current understanding is that each of these encapsulates some set of algorithms. I feel like, amongst ML, NN, and DL, Machine Learning (ML) is the largest set, which contains NN, which in turn contains DL. See figure below. Data Science is the largest set amongst all 5. Is this correct? Where would AI be on this figure?

AI: a super general term, means a bit of everything... and nothing at the same time. It's all about building intelligent machines, even though its meaning is not fully developed. It's not used in a rigorous way at all. In fact, scientists prefer to use more technical terms like the others you listed.

ML: it's an approach to data, and it's all based on building models that improve automatically from experience.

DL: Deep Learning, it's the subset of ML that makes use of Deep Neural Networks. So technically it's still Machine Learning, currently its most advanced and promising branch.

NN: The general class of models that is employed in DL. DL is the field of application of NN's.

Personally, I don't really like that Venn diagram you posted. DL being a subset of NN... that doesn't make sense to me.


Can closer points be considered more similar in T-SNE visualization? I understand from Hinton's paper that T-SNE does a good job in keeping local similarities and a decent job in preserving global structure (clusterization).

However I'm not clear if points appearing closer in a 2D t-sne visualization can be assumed as "more-similar" data-points. I'm using data with 25 features.

As an example, observing the image below, can I assume that blue datapoints are more similar to green ones, specifically to the biggest green-points cluster?. Or, asking differently, is it ok to assume that blue points are more similar to green one in the closest cluster, than to red ones in the other cluster? (disregarding green points in the red-ish cluster)

When observing other examples, such as the ones presented at sci-kit learn Manifold learning it seems right to assume this, but I'm not sure if is correct statistically speaking.

EDIT

I have calculated the distances from the original dataset manually (the mean pairwise euclidean distance) and the visualization actually represents a proportional spatial distance regarding the dataset. However, I would like to know if this is fairly acceptable to be expected from the original mathematical formulation of t-sne and not mere coincidence.

I would present t-SNE as a smart probabilistic adaptation of the Locally-linear embedding. In both cases, we attempt to project points from a high dimensional space to a small one. This projection is done by optimizing the conservation of local distances (directly with LLE, preproducing a probabilistic distribution and optimizing the KL-divergence with t-SNE). Then if your question is, does it keep global distances, the answer is no. It will depend on the "shape" of your data (if the distribution is smooth, then distances should be somehow conserved).

t-SNE actually doesn't work well on the swiss roll (your "S" 3D image) and you can see that, in the 2D result, the very middle yellow points are generally closer to the red ones than the blue ones (they are perfectly centered in the 3D image).

An other good example of what t-SNE does is the clustering of handwritten digits. See the examples on this link:https://lvdmaaten.github.io/tsne/


out of memory error when consrtucting 2d list from 2 numpy arrays I am working on lung CT images from luna16 dataset, the dataset have a 3d lung image and a label from CSV file, I have a code for constructing 2d list from 3d array 25x25x25 (the 3d image) and a label [0,1] or [1,0] from CSV file, after creating the 2d list I want to save it in numpy file, below is my code for creating the 2d list and saving it in numpy file:

def getIDlist(csv_Dist,Data_Dist):
    # receive marked coords and ID in annotations.csv, and return the distination with coords.
    print('loading')
    data = np.loadtxt(csv_Dist, delimiter = ',', dtype = 'str')
    # delete the header file via 1:0, and receive the ID, x, y, z, r via 0:5 to a list.
    ID_coords = data[1:,0:5][0:10000] # get list of 'seriesuid' 'coordX' 'coordY' 'coordZ' 'class' (without header).
    # define the output file.
    ID_dist = []

    print('strat finding')
    process_bar = ShowProcess(len(ID_coords))

    for ID,x,y,z,label in ID_coords:
        ID = ID +'.mhd'
        found = 0
        for parent, dirnames, filenames in os.walk(Data_Dist):
            for filename in filenames:# loop inside all files
                if ID == filename: # ID + .mhd in csv equal to filename in files
                    process_bar.show_process()
                    ID = parent + '\\' + ID# ID gets full path of the founded file
                    ID_dist.append([ID,x,y,z,label])# ID_dist gets info of founded files
                    found = 1
                    #print("found: ", found)
                    break
            if found == 1:
                break
        if found == 1:
            continue

    process_bar.close()
    return ID_dist

def get3Dmatrix(ID_dist):

    print('preparing the 3d matrix')
    matrixlist = []
    for Dist, xcoords, ycoords, zcoords, label in tqdm(ID_dist):
        # read the image
        imagearray,origin,spacing = load_itk_image(Dist)
        # resample in to 1mm*1mm*1mm
        imagearray = resample(imagearray,spacing,(1,1,1))

        # transfer world coordinates to voxel-coordinates, divide new spacing 1mm
        z = int(round((float(zcoords)-float(origin[0]))/1))
        y = int(round((float(ycoords)-float(origin[1]))/1))
        x = int(round((float(xcoords)-float(origin[2]))/1))

        # get the 3D array with shape 25*25*25
        imagearray = imagearray[z-13:z+12,y-13:y+12,x-13:x+12]

        #converting the label number into a one-hot-encoding
        if int(label) == 1:
            label=np.array([0,1])
        elif int(label) == 0:
            label=np.array([1,0])

        # put it into output file
        matrixlist.append([imagearray,label])# 2d list consist of 3d array + label of all cases.
    return matrixlist

 def main():
    start_time = time.time()
    # get ID_list from the csv and data dist.
    ID_list = getIDlist(candidates_V2_Dist, Data_Dist)# nested list - get file name with dist + x,y,z,class
    # Data_set[i][0] is the 3D array, Data_set[i][1] is the label
    Data_set = get3Dmatrix(ID_list) # 2d list consist of 3d array + label of all cases.
    print("Begin saving in numpy file")
    np.save(output_path+'np_ds(10000)-25-25-25(zyx)_one_hot.npy', Data_set)
    print("%s time takes in seconds" % (time.time() - start_time))

if __name__ == "__main__":
    main()

my problem is:

1- with approximatly 550 samples, the RAM gets fulled and I get memory error, I am working on dell inspiron core i7 with 16 gb ram laptop.

2- it takes 34 seconds for creating each sample, and I see this is huge amount of time for only one sample.

I did a lot of search in google and asked a question in some other forums but didn't get any answer, can anyone help me, please? realy I am confused with that error.
image below is the error message:

I would recommend breaking the problem down a little bit to reduce the memory usage at any one time.

The first part of your main function gets the IDs using getIDList. That seems fine, so leave it there.

I would then break that list down into smaller chunks, calling get3Dmatrix on each chunk in turn. Altering your code, it might look something like this:

# Get number of entries in ID list
N = len(ID_list)

# break it down into a number of chunks e.g. 4, based on your progress bar
import numpy as np    # should already be imported

N = len(ID_list)
num_chunks = 4           # you can play with this number, making it larger until you don't get emmory errors
chunks = np.linspace(0, N, num_chunks)

for i in range(len(chunks) - 1):
    this_sublist = ID_list[chunks[i] : chunks[i + 1]]
    sub_data_set = get3Dmatrix(this_sublist)

    # At this point, either save this sub_data_set, or try appending it to another list toi make one final numpy matrix at the end before saving

...

print("Begin saving in numpy file")
np.save(output_path+'np_ds(10000)-25-25-25(zyx)_one_hot.npy', Data_set)
print("%s time takes in seconds" % (time.time() - start_time))

Even from the traceback you added, it is hard to say where exactly in your code that is happening.

Roughly looking at the dimensions you mention, it also doesn't seem plausible the a 16Gb machine is running out of memory - so I must not completely understand just how many images/patches are being saved.


Why do we scale down images before feeding them to the network? I have been seeing a lot where images are generally scaled down to either $64\times64$, $32\times32$ or other lower resolutions. Can someone please help me with this and answer a few questions:

Don't we lose image details by doing so?
What would be the consequences if we scale down the image to some other higher resolution such as $512\times512$ or $1024\times1024$ or something?
Can we feed the network without $1:1$ square images?

Because it saves lots of computation time. Yes, we do loose image details, but it depends on your task whether the details are really important. Let's say your task is to detect and count the total number of circles in an image, So you are only concerned with the shape circle, not its dimension. Hence, reducing image resolution a bit would be more computationally efficient.


What is slowing down classic DBSCAN algorithm How to apply CSR Matrix on DBSCAN algorithm in python without using any libraries?

Update: Matrix size (8580, 126356)

I have given a shot and implemented the algorithm. It runs rather slow. I guess its because regionQuery function calculates the euclidean distance  between a chosen point and every other point in the dataset.

Is there any way this problem can be solved?

Write the DBSCAN code yourself.
Run the code.
Observe that your code likely will be a lot slower than the libraries code.


Is Vector in Cosine Similarity the same as vector in Physics? I'm new to Data Science. I'm trying to understand cosine similarity and it seems like the equation is about finding the distance between two vectors. From what I've Googled, a vector needs to have magnitude and direction. But in CS, it seems like it's a 1-dimensional array. Is vector in CS the same as vector in Physics? If so, what is the direction of a vector. And if a vector is like this [1, 0, 1, 0] what is the magnitude of this vector?

As you ask specifically for the Cosine Similarity technique, it has magnitude and direction, and similar to a vector which is used in Physics, as Cosine Similarity deals with vectors in an inner product space.

So, the magnitude of vectors is exactly the same as the formula in Physics (summating over the squares of the vector elements.)


What best/correct algorithm/procedure to cluster a dataset with a lot 0's? I'm new to statistics so sorry any major lack of knowledge in the topic, just doing a project for graduation.

I'm trying to cluster a Health dataset containing Diseases(3456) and Symptoms(25) grouping them considering the number of events occurred.

My concern is that a lot of the values are 0 simple because some diseases didn't show that particularly symptom, for example (I made up the values for now):

So, I was wondering what was the best way to cluster this dataset. I was looking for and found hierarchical and kmeans, but don't know if i can properly apply to this cenario.
First of all, I switched the absolute values of occurrences to % of total, does that make possible to deal with the 0's ? I thought about it but at the same time 1% is close to 0%, I don't know if the algorithm can also understand as 'flags', since 1% represents that actually that symptom occurr(even at lower rates) and in another disease don't occurr at all.

I heard about PCA to reduce number of variables, and I was also curious if:
1-PCA is applicable to this cenario (dataset with a lot of sparse 0's)
2-PCA could solve my problem(because I think that even when a reduce to 2 3 variables, some rows could still be 0 for that particular column(symptom).

Any help/guidance would be extremely helpful and I thanks all in advance,
Sorry for some english error as well!

Have a great week!

Maybe I am not understanding your question properly, but k-means clustering is not sensitive to the zeros. More accurately zeros are valid values for features to cluster on.

While PCA may enable you to reduce the number of features to train on, it may complicate the clarity of the clusters from k-means. PCA may be beneficial in its own right, but AI would resist the urge to feed PCA data into k-means simply to reduce the number of zeros in your data.

HTH


Weight training breakdown in machine learning I'm not sure if this exists. Is there such a situation where weights in gradient descent fail to work or break up? If so, how and when?

There is a situation called "exploding gradients" where very large error gradients accumulate and result in very large updates to neural network model weights during training.

This can result in an unstable model that cannot learn from the training data or even result in NaN weight values that can no longer be updated.


How to plot mean_test score and mean_train score of GridSearchCV How to plot mean_train_score and mean_test_score values in GridSearchCV for C and gamma values of SVM?

You could visualize them as a heatmap.

For example you could use the C values as the rows, the gamma values as the columns and the color intensity of each element in the heatmap array would correspond to the mean_test_score.

To implement this you first need to create a pandas.DataFrame like this:

$$
\begin{array}{c | c c c}
  & C   & gamma & mean\_test\_score \\ \hline
1 & 0.1 & 0.001 & 0.798 \\
2 & 1   & 0.001 & 0.813 \\
3 & 1   & 0.01  & 0.801 \\
4 & 10  & 0.001 & 0.787 \\
\end{array}
$$

To do this you need to store each run you make in a different line, which will contain all necessary hyper-parameters and the result. Then you will need to make a pivot table which will use C as the rows, gamma as the columns and mean_test_score as the values.

pivot = pd.pivot_table(df, values=df['mean_test_score'])

This pivot will be the array that will form your heatmap. Now you should select your aesthetic parameters (e.g. colormap) and proceed to make the heatmap.

sns.heatmap(pivot) # plus any other aesthetic parameters you wish


What is the minimum size of the test set? The mean of a population of binary values can be sampled with about 1000 samples at 95% confidence, and 3000 samples at 99% confidence.

Assuming a binary classification problem, why is the 80/20% rule always used, and not the fact that with a few thousand samples the mean accuracy can be estimated with > 95% confidence?

This is a great question since it is illuminating to examination the best practices of both about traditional statistics and machine learning as they are brought together.

Two Separate Rules/Best Practices -

First, the two items that you mentioned should be examined separately and not conflated i.e. they should be carefully combined as you suggest in your question.

Significance: You have estimated that you want greater than 1000 cases to have statistical significance and would further benefit from greater than 3000 test cases.

Cross validation: Cross validation is performed by splitting the data (often 80% train-20% test) so that tests for bias (~underfitting) and variance (~overfitting) can be assessed.

Combining significance with cross validation: Now we know that we want to have significance in our tests so we want greater than 3000 records.  We also want to perform cross validation, so in order for both the testing and training data to return significant results we want both to have a minimum of 3000 records.  The best scenario for this would be to have 15,000 total records. This way, the data can be split 80/20 and the testing set is still significant.

Lets assume you only have 4000 records in your data set. In this case, I would opt to make my training data significant while allowing the testing set to drop to lower significance.

More rigor: Everything above has been quite hand wavey and lacks statistical rigor. For this, I refer you to a couple of papers referenced in another Stack Exchange question -

Dietterich, "Approximate Statistical Tests for Comparing
Supervised Classication Learning Algorithms"

Salzberg, Data Mining and Knowledge Discovery, 1, 317–327 (1997), "On Comparing Classifiers: Pitfalls to Avoid and a
Recommended Approach".

Hope this helps!


numpy array quirks: does A.T return a view? I am trying to understand the numpy array object, and a little mystified by the following:
A = np.array([[1,2,3],[4,5,6]]) # A is a 2x3 matrix
B = A

A = A.T # A is now the matrix [[1,4],[2,5],[3,6]]
A[0,1]= -1 # A is now the matrix [[1,-1],[2,5],[3,6]]

print(B)

I get for B:
[[1,2,3],[-1,5,6]]

So changing A using the assignment A[0,1]= -1 changed B (which is to be expected since numpy arrays are mutable objects, and python assignments are by reference). But what is mystifying is that A = A.T did not transpose B. Furthermore, if I ask (after the A = A.T statement):
A is B
I get False. What exactly is A.T returning? It seems to return a view of the data, but a copy of the alignment of data into a matrix.

You are right that B points to the original array and A (at the end) is a transposed view of the array.
The key point here is something that trips up a lot of people: whenever you assign a new value to a variable by using = with no indexing (e.g., A = <anything>), it causes that variable to point to a new location, but it doesn't alter the data previously held in that variable.
Here is an annotated version of what is happening:
A = np.array([[1,2,3],[4,5,6]])
# create the array [[1,2,3],[4,5,6]] at some location in memory
# (let's call it loc 1)
# point variable A to loc 1

B = A
# point variable B to the same location as A (loc 1)

A = A.T
# right-hand side:
#    - get the array that A currently points to (from loc 1)
#    - create a transposed view of this array and store the view
#      somewhere in memory (let's call it loc 2)
# left-hand side: set variable A to point to loc 2
# Note: the `=` assignment does not alter the object at loc 1; instead,
# it points A to the new object that was created on the right-hand side

A[0,1]= -1
# Push the value -1 into position (0, 1) of the object that A
# currently points to, i.e., the view at loc 2. This updates
# the A view, but also propagates back to the original,
# un-transposed array at loc 1 (which B still points to).

# Note that the last two lines are equivalent to this:
X = A.T     # doesn't transpose B
X[0,1]= -1  # propagates through view back to B
# i.e., the A.T could have been assigned to anything, e.g. a new
# variable X. When you do an assignment with `=`, it creates a new
# variable every time, even if it happens to have the same name as
# an old one, as in your case.

This is the general issue of changing mutable data and getting unexpected side effects. The easiest way to think about it is that a bare variable assignment (A = <something>) never alters the underlying data that A currently points to. Instead, it just causes A to point somewhere new. On the other hand, assigning A[index] = <something> or A.attribute = <something> or using A.data_altering_method(...) will alter the underlying object that A points to, in place, so anything else that points to the same object will also see the alteration.
In your case, the only thing you did that changed the underlying data was A[0,1]= -1, and that did indeed propagate to B. A = A.T did not alter the underlying data, it just pointed A to something new (which happened to be a new view of the original data). So the change to the A variable didn't propagate to B. However, it is a little surprising that  A = A.T creates a view rather than a copy, so that later changes to A affect B. Your middle two lines are equivalent to B.T[0,1] = -1.
These answers also help:

https://stackoverflow.com/a/7838047/3830997
https://stackoverflow.com/a/38925257/3830997