What to optimize for when asked to find the most important features I have a binary classification problem, let's say people can buy or not buy a certain product. Now unlike a standard prediction task, I only want to find which features are the most important for the person's decision to buy.

Which metric should I use to optimize the algorithm? Maximize out of sample accuracy like when I would be interested in making the best prediction? Or maximize fit and don't care about overfitting? A mixture of both?

I am using xgboost.

Ideally you should select features in the same way you select the best model hyperparameters, with a validation set.

You are interested in how the features work on unseen data, not your training data.

Depending on your feature selection algorithm, feature search will be intractable with larger data.  Naive feature selection is O(2^n) AND some model hyperparameters could be dependent on your choice of features.


Is there a particular order in which to do feature selection and sampling? I want to use feature selection and observation subsampling on my data, for several reasons:

feature selection for the usual motivations (reduce noise, decrease running time, etc.)
observation subsampling because I have strongly imbalanced data, and I want not to introduce bias towards the most prevalent class in downstream classifiers

My question is: is there a specific order in which I should do feature and observation selection? E.g. first feature selection then subsampling?

In my opinion, the methodological correct way to do it would be to first randomly select the observations (e.g., with stratified random sampling to sustain the class balances as in the original data) and then do any model building (feature selection is a part of model building) based on the sected examples.

You should asses the quality of your model with remaining observations that have not been used for either selecting features or training a model.

Hope this helps.


Can you recommend a machine learning challenge that is suitable for novices? I am looking for a challenge that is suitable for a group of novices who want to learn the basics of data science and machine learning. The challenge should match the following criteria:

is based on a real application or is at least realistic
has a clearly defined goal and partial progress is measurable
includes a machine learning component, but also other aspects of data science
should be doable within 3 to 6 weeks
is suitable for novices
it should be an actual challenge in the sense that you cannot just look up near-optimal solutions from the internet

You've already answered yourself by tagging kaggle.

Let me share the two competitions that I have every new hire in my team go through- and then have them keep improving their solurion for the next six months. These are a very well curated set of problems for a budding data scientist!

Titanic: Machine Learning from Disaster
https://www.kaggle.com/c/titanic
This one helps bridge the gap between an analyst and a data scientist, as well as eases you into the world of code from Excel.
Digit Recognizer
https://www.kaggle.com/c/digit-recognizer
The MNIST dataset has prepared almost every data scientist for the real world of dirty data, in a sweet, cushy manner. It makes very real the fact that not always will we have structured and organized data, but it should not deter us from gaining insight from it!

Happy Coding!


SMOTE train test split with validation data Would like to ask, in which way to use SMOTE?
My dataset is imbalanced and a multiclass problem. As I read in many posts, use SMOTE method only for the training dataset (X_train and y_train). Not for the test dataset (X_test and y_test). There I include validation data. How do you handle SMOTE with validation data?
df = pd.read_excel...

X=df.drop('column1',axis=1)
y=df.column1

#Training part
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, shuffle=True, random_state=42)

smote = SMOTE(random_state=42)
X_train_smote, y_train_smote = smote.fit_resample(X_train, y_train)

#validation part
X_train_smote, X_val, y_train_smote, y_val = train_test_split(X_train_smote, y_train_smote, test_size=0.5, random_state=42)

Is this correct?
and is it right, that the validation datasets (X_val and y_val) have both SMOTE inside?  or should I make it out of the normal train test split:
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.5, random_state=42)?
Im confused.

Problem in applying smote on data and than applying the split (test/or Validation does not matter) is that you could suffer from data leakage. Meaning that some Information from the Train could Spill over to the future and falsly give good predictions. I would advise seperating smote data Generation process for all 3 data sets. So do the splits, than do the data Generation.


Neural network using Tensorflow Is there any possibility that cost function might end up in local minima rather than global minima while implementing the Neural network using Tensorflow ?

Most of the critical points in a neural network are not local minima, as it can be seen in this question. Although it is not impossible to fall into a local minimum, the probability of it happening is so low that in practice it does not happen, except from very special cases as a single-layer perceptron.

Being local minima so hard to find, this means that it is highly unlikely to come across the global minimum in your optimization method. All that we do in deep learning is decrease the loss function to find fairly good parameters, but finding local and global minima is extremely unlikely.


How can I find contextually related words and classify into custom tags/labels? PROBLEM:

Suppose if I have a small dataset containing some words and their tags/labels.
The main task is to provide tags to other words(which are not in the dataset) based on their contextual relationship with the words already in the dataset.

Let's say, for example, my custom dataset includes

              Soap --> label__(cleaning_agent)
              pencil--> label__(stationary_item)
              mobile--> label__(electronics)
              washingmachine--> label(electronics)
              and so on.

I want my program to be able to correctly predict the label of an unknown word, e.g.

         washing powder to its correct category
         label__(cleaning_agent)
         radio to label__(electronics) etc.

ACTIONS:

Now the main problem is to find the relation between two words based on context, but I am not able to decide what can be the parameters for finding that.

I have tried a naive approach using datamuse API and fastText library.

    Naive approach is as follows->
        step 1-> find all the related words of the given word(let's say W) e.g. pencil using datamuse API.
        step 2-> combine them into a string(let's say S) with spaces in between them
        step 3-> use the label name, W, S as the training dataset for fastText.

NOTE: fastText requires label name, word, sentence(can be from news articles, blogs, Wikipedia etc ) as a context for that word.

RESULTS:
fastText is not providing any reliable results.
I am thinking of building a neural network kind of thing for this purpose.
But I am not able to decide what can be the input parameters for our data.

The main problem is about custom word tagging. Our program should be able to tag unknown(not in training dataset) words to their most probable classes based on some score.

As I am new to NLP, I want to know what can be the next move forward.

You are working with word classification so you really don't have any contextual information that you can leverage. So the best thing would be to use freely available contextual information called as word2vec. I would suggest using the pretrained GLOVE Word2Vec as they are much cleaner than the Google ones. I am going to suggest two approaches for doing this:

Naive Approach

Load the pretrained word vectors.
For every new word calculate cosine similarity with every item in your training list.
Predict the label for the word which is closest to your new word.
Alternatively you can calculate cosine similarity with the labels and just predict the label which comes the closest.

Better Approach

Load the pretrained word vectors.
Extract the vectors for every item in the training set.
One hot encode the labels.
Train a neural net or any multiclass classification algorithm using the vectors as features and save the model.
For every new word extract the vector and run the model on this vector.
The predicted label is your output.

You can use the first approach to run quick tests and check the efficacy of the approach and then move on to the second one if you found it fit to your use.


Predicting which drug is most appropriate for which patient returns an accuracy of almost 0 I have a dataframe that looks like this:
data = {'age': [54, 21, 7, 18],
        'sex': [0, 1, 1, 0],
        'disease_type': ['A', 'B', 'A', 'F'],
        'change_in_pain': [-0.54, -0.89, 0.07, -0.01],
        'drug': ['drug_1', 'drug_7', 'drug_1', 'drug_89'],
        }
df = pd.DataFrame(data)

=>
   age  sex disease_type  change_in_pain     drug
0   54    0            A           -0.54   drug_1
1   21    1            B           -0.89   drug_7
2    7    1            A            0.07   drug_1
3   18    0            F           -0.01  drug_19
...

The real df has > 10000 rows (=patients) and 34 different drugs but seemingly I cant upload a csv here for a more usable example?
I would like to train a model that predicts which drug is most effective for which patient given the patient’s age, sex, disease type and how much the pain was reduced (a more negative “change_in_pain” column is better).
In this simple example “drug_1” woud work for disease A only if the patient is older and female.
I wrote the following code but the mean accuracy is returned as almost 0 :
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier

# shuffle
df = df.sample(frac=1.0).reset_index(drop=True)

X = df[['age', 'sex', 'disease_type', 'change_in_pain']]
y = df['drug']

# convert categorical variable into dummy/indicator variables.
X_OHE = pd.get_dummies(X)
y_OHE = pd.get_dummies(y)

X_train, X_test, y_train, y_test = train_test_split(X_OHE, y_OHE, test_size=0.20)

scaler = StandardScaler()
scaler.fit(X_train)

X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)

knn = KNeighborsClassifier(5)
knn.fit(X_train, y_train)
score = knn.score(X_test, y_test)
print('mean accuracy: {:2.2f}'.format(score))

I also tested: RandomForestClassifier(max_depth=5, n_estimators=10, max_features=1), KNeighborsClassifier(3), DecisionTreeClassifier(max_depth=5), MLPClassifier(alpha=1, max_iter=1000) but again the mean acc is around zero.
What am I doing wrong?
EDIT:
Doing it more slowly using:
knn.fit(X_train, y_train)  # X_train: 8000x11, y_train: 8000x34
y_pred = clf.predict(X_test)  # X_test: 2000x11, y_pred: 2000x34
acc = accuracy_score(y_test, y_pred)

shows that y_pred seems to contain only zeros - but why?

I can't say exactly why you get null accuracy, but I have some comments that may help:

you are mixing continous data and categorical data. You might already be aware:

when standardizing your data: you are although standardizing your categorical data (sex, disease_type), just be sure that it makes sense (depending on the classifier you're using)
depending on the classifier, mixing continous and binary data may lead to undesirable results. This post explains it in the case of KNN

Otherwise I don't see anything suspicous after a first look. Could you share more data so we can reproduce your result ?


Real-time or offline Data Augmentation for segmenting microscopy images? I'm doing semantic segmentation(for cells) using microscopy images. I'm exploring U-net and FCN DenseNets for the task. In the U-net paper the authors have trained their model only from 30 images but have used data augmentation extensively specially elastic deformations. I too have very little data, 15-20 annotated images. Thus I too plan to use data augmentation. I'm also using elastic deformations.

Other techniques I'm using:

Flipping, Rotation, Translation, Shearing
Random crops, zoom-out, stretching
Image Contrast and Gaussian noise

This will increase my dataset 10-20 folds.

Q.1 My question is shall I use offline augmentation or on-the-fly(Real time) augmentation?

From what I've read so far: use on-the-fly augmentation if dataset if large so to not make the dataset explode in size. I know this is very generalised. I've also seen this on Kaggle: For online data augmentation, the model see one random generated sample only once hence generalises better.

Q.2 Also I'm using so many data augmentation techniques will that be even useful as the data set so small, or shall I restrict myself to only some of these techniques?

Any help is highly appreciated.

I think you have way too less of data assuming that you would have to separate it out further to training, test and validation sets. Your model will not generalize well. Consider procuring some more data (around 1500-2000 samples atleast)


Which is the best algorithm for entity extraction for unstructured document I have unstructured documents from which I have to extract the information like
let buyer name, seller name, expiry date, buying date etc. I had planned to use spacy(Custom entity recolonization(Followed this blog https://medium.com/@manivannan_data/how-to-train-ner-with-custom-training-data-using-spacy-188e0e508c6)). But it seems sometimes buyer name predict as seller name and vice-versa and also sometimes got multiple predicted data wrongly in single entity when I passed whole document content. FYI.. This documents have approx 2-20 pages. so it has large content.

Can someone share if we can use any other packages for higher accuracy? if not how I need to train the model so that accuracy will be higher? Thanks in advance

Try to clean your document and use the flair library, it's a user friendly library from Zalando Research that allows you do do all sorts of nlp tasks very quickly. Especially NER.


Clustering method for 2-D data that self-detects number of clusters and takes care of outliers Assuming I have data that looks something like that:

I'm looking for a method or algorithm that can perform the clustering (e.g. as shown in the picture), that automatically determines the optimal number of clusters while also accounting for outliers.
Would appreciate suggestions of methods or relevant implementations/libraries/pipelines, preferably in Python.

DBSCAN is the algorithm of choice for this task.
This a density based algorithm which will look for clusters according to two main parameters, epsilon and min samples.
It will also identify those samples that do not form any cluster according to prior parameters as outliers.
From Scikit-learn documentation:

The DBSCAN algorithm views clusters as areas of high density separated by areas of low density. The central component to the DBSCAN is the concept of core samples, which are samples that are in areas of high density. A cluster is therefore a set of core samples, each close to each other (measured by some distance measure) and a set of non-core samples that are close to a core sample (but are not themselves core samples). There are two parameters to the algorithm, min_samples and eps, which define formally what we mean when we say dense. Higher min_samples or lower eps indicate higher density necessary to form a cluster.

Additional material worth to be checked:
https://www.naftaliharris.com/blog/visualizing-dbscan-clustering/


How to split train/test datasets having equal classes proportion I would like to know how I can split in an equal number the following
Target
0    1586
1     318

in order to have the same proportion of 0 and 1 classes in a dataset to train, if my dataset is called df and includes 10 columns, both numerical and categorical.
I would consider the following
y=df['Target']
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.1, stratify=y)

so to do a stratification, but I do not know if it is right and I would appreciate if you could confirm it or provide an alternative to do that.
Sample of data
Fin                 Eco   Target
High percentage     12      1
Low percentage      5       0
Medium percentage   48      0
NA                  3       1
TBC                 NA      1
Low percentage      25      0
Medium percentage   12      0

How can I check if it is actually splitting in equal classes proportion my dataset?
I think the best way to train a model should be having an equal proportion of 0 and 1 values. Right now I have 5 times data with Target=0.

If what you want to do is keep the same proportions across the splits, what you are doing is right.
In order to validate properly your model, the class distribution should be constant along with the different splits (train, validation, test).
In the train test split documentation , you can find the argument:

stratifyarray-like, default=None
If not None, data is split in a stratified fashion, using this as the class labels.

One step beyond will be using Stratified K-Folds cross-validator.

This cross-validation object is a variation of KFold that returns stratified folds. The folds are made by preserving the percentage of samples for each class.

There are more splitting techniques in scikit learn that you can use, have a look.
In order to check what is the function doing just calculate the percentages in the splits:
 np.unique(y_train, return_counts=True)
 np.unique(y_val, return_counts=True)

But this will make you have the same proportions across the whole data, if your original label proportion is 1/5, then you will have 1/5 in train and 1/5 in test
If what you want is have the same proportion of classes 50% - 0 and 50% - 1. Then there is two techniques oversampling and undersampling. But I wont recommend you this for your problem. Your label seems fairly good balanced 1/5 its a great proportion.
Still this library has some implementations to do that
In this blog you can see an overview of imbalanced datasets but yours is not. Choosing a proper metric is more important.


Name Tagger in Stanford NLP Is there any good library in stanfordNLP which is meant for name parsing? Basically i want to separate first name, last name, nickname, surname, givenname from the full name. Something similar to HumanName and parserator.tag, parserator.parse? Lets say i have a name Mr George "Gob" Bluth II and if i use parserator.parse, it gives me below output. Is there something like this in stanford NLP?

>>> probablepeople.parse('Mr George "Gob" Bluth II')
[('Mr', 'PrefixMarital'), ('George', 'GivenName'), ('"Gob"', 'Nickname'), ('Bluth', 'Surname'), ('II', 'SuffixGenerational')]

Thanks!

Name parsing does not appear to built-in to Stanford CoreNLP.
.

One option is writing a series of Regular Expression using Stanford TokensRegex to parse and label name tokens.

Another option is using a third party package, such as nameparser in Python.


normalization of probabilities in predicting a poly-neuron output in neural nets When predicting a poly-neuron output in neural nets, say, predicting multiple handwritten digits and giving an output neuron vector (0.1,...,0.9,0.1,...), many use sth like softmax (or sth like the energy dependent probability exponential formula in statistical mechanics) to normalize the output vector such that all the components of the output vector sum up to 1, and that the normalized output vector becomes a probability vector. I doubt the necessity of this normalization, for without which I can equally well predict as per the biggest vector component. Is there anything I overlooked?

You are correct, but this is not called normalization. You can simply use the highest probability output for category. This is what softmax does for you. For example 2 output neurons can have 0.1 for dog and 0.9 for cat as the loss. Softmax will it just convert it to [0,1] meaning no dog but a cat is on the image.


Does Homoscedasticity applies only for linear regression models? In statistics, a sequence of random variables is homoscedastic if all its random variables have the same finite variance. This is also known as homogeneity of variance (Wikipedia).
Is this assumption only for linear regression models?
If not, is there any relative method is available to check for non-linear models?

It also applies to other methods, i.e. not just linear regression. For example, ANOVA and T-test also depend on homogeneity of variance.
One method to check the homogeneity of variance, compatible with the one-way ANOVA, is the Barlett's test.


How to force weights to add to $1$ in Linear regression I am using a linear regression using scikit-learn in python. However, I would like to force the weights to add to $1$. Is there a way to do this? I know that I need to add a constraint but am not able to figure out how. My regression looks something like this $Y= a_0 + bX_1 + (1-b)X_2$

Thanks in advance

Suppose the problem formulation is

$$\min_{a,b} \sum_{i=1}^n(y_i -a_0-\sum_{j=1}^pb_jx_{ij} )^2$$

subject to $$\sum_{j=1}^p b_j=1$$
$$b \ge 0$$

then this is an instance of a quadratic programming problem.

There are solvers such as Gurobi (free for academia) and CVXOPT (freeware) that can manage a quadratic programming problem. An example of CVXOPT code to solve a QP can be found here.

Note that the objective function can be written as

\begin{align}&\|Y-a_0e-Xb\|^2\\&=\left\|Y-\begin{bmatrix}e & X \end{bmatrix}\begin{bmatrix}a_0 \\ b\end{bmatrix}\right\|^2\\
&=\begin{bmatrix}a_0 & b \end{bmatrix}\begin{bmatrix} e^T\\X^T\end{bmatrix}\begin{bmatrix} e &X\end{bmatrix}\begin{bmatrix}a_0 \\ b \end{bmatrix}-2Y^T\begin{bmatrix} e &X\end{bmatrix}\begin{bmatrix}a_0 \\ b \end{bmatrix}+\|Y\|^2\end{align}

Also, you might like to check with people in your field what is the common package that they use.


ideas for variable with branching data? I had no idea how to explain this in the title but anyway... Let's say I have a data points like this:

John, Happy | Greedy | Smart | Funny, 0.8
Ann, Smart | Sad | Funny, 0.6
Joel, Greedy | Prideful | Stupid, 0.2

Where the first part is the name the second is there characteristics and the third is their overall character score (how nice they are to be around or something). Is there a good way to work with this data so that I can work out what the best possible combinations are? Assume I have a large enough data set. There may be say 30 of those characteristics and any person can have any number of them and every characteristic is equally valid to every person.

Hopefully that explains it. Essentially I am want a way to organise the traits so that I can say "smart and happy" make a better combination than "sad and greedy". I also need to be able to asses the ultimate combination and be able to compare any two possible combinations.

The typical way to address this is to transform those characteristics into a representation where you compose a vector with one component for each possible characteristic, with a $1$ in the position of each characteristic owned by the individual, or zero otherwise.

This way, you would have:

name | greedy | smart | funny | sad | prideful | stupid | ... | score
----------------------------------------------------------------------
John |   1    |   1   |   0   |  0  |    0     |   0    | ... | 0.8
Ann  |   0    |   1   |   1   |  1  |    0     |   0    | ... | 0.6
Joel |   1    |   0   |   0   |  0  |    1     |   1    | ... | 0.2


How to choose input variables for ML Let's say I have a huge database with 100K records and 60 columns. Let's say one of the column is "min_p". What I do is apply some logic/rule to determine the output label for this record. Basically I look at previous two records and next two records of this min_p. If the condition is satisfied, I will mark the label as 1 else I will mark it as 0.

Now my question, since I have directly derived the label from this called "min_p", should I retain it as one of my predictors in my final dataset? Since I have used that derive the label,  I didn't include them in my dataset as a input variable thinking that it is incorrect

Can you help me with this?

[edited, I misread the question in the first version]

The fact that the label is determined from a combination of values from this feature is not a problem in itself: if it makes sense, it's always better to give the best indicators to the learning algorithm. So the only questions are:

whether it makes sense for your problem to have the feature provided as input for any new instance: if yes, then there's no reason to remove it.
whether it's useful to apply ML to your problem: if the label can be determined directly from a single feature, it's simply not useful to train a model.

You mention that the label is based on information from the previous/next two records. Keep in mind that the model needs to predict its target for any individual instance as input, unless you're using a sequential model (for instance with times series).


How can I have the same initialization for all my networks? I want to have the same weights for layer initializations in all my networks, so that when I'm comparing their first epoch loss they all start from the same value. Is there a way in keras to do this?

I have set the random seed for the numpy and tensorflow, but still I get different results in initializations.

You need to specify the seed in the initializer, e.g:

from keras.initializers import RandomUniform

seed = 0

model.add(Dense(64, kernel_initializer = RandomUniform(minval = -0.05,
                                                       maxval =  0.05,
                                                       seed = seed)))


What are latent representations? I am reading some research papers about graph convolutional neural networks and I have seen the term "latent representations" used a lot. For instance, "the model was able to learn latent representations of the nodes of the graph".
What does the word "latent" means here? Is it the same as latent features?

The latent representation is the simplified model of your input data, for example, created by a neural network.
Considering an autoencoder, the central layer of this network (after training) will contain a simplified representation of the input data (i.e. summary of key features), which can be used to reconstruct the output.
If we take a dictionary definition of Latent: present and capable of emerging or developing but not now visible, obvious, active, or symptomatic, we can see how this describes the somewhat non-existence of the state, rather instead only a latent representation of the input data.
This image is a nice description. The latent representation is key features of the input data (here: the ears, nose, eyes of the animals.)

So yes, the latent representation is the sum of the latent features.


What is the depth of an image in Convolutional Neural Network? I am learning cs231n Convolutional Neural Networks for Visual Recognition. The lecture notes introduce the concepts of width, height, depth. For example,

  In CIFAR-10, images are only of size 32x32x3 (32 wide, 32 high, 3
  color channels)

However, in another example,

  a volume of size [55x55x96] has 96 depth slices, each of size [55x55]

What does width 96 means? Does it mean 96 color channels? Why can we have more than 3 color channels?

It means that the number of filters (a.k.a. kernels, or  feature detector) in the previous convolutional layer is 96. You may want to watch the video of the lecture, and in particular this slide, which mentions that a filter is applied to the full depth of your previous layer:


Doc2vec(gensim) - How to calculate the most similar sentence and get its label? I'm trying to use doc2vec(gensim) to identify the most similar sentence and get its label. That is, for example, when the data is composed of 36 types of TVs (each sentence explains a specific product and its labeled to that product), the doc2vec categorizes the user input and decides what TV the user is referring to.

I only know how to get the most similar word: model.most_similar('red/noun')
How can you, instead of words, get the most similar sentence and its label?

Doc2Vec - How to label the paragraphs (gensim) (this tells that the above method is actually possible in doc2vec)

Thank you :)

As far as I understood you are using type of TV as tag of particular sentence , and you are using doc2vec model for future classification . So As above answer is suggesting that model will learn semantic meaning of type of TV(tag).
let's suppose s is your future sentence for prediction. then you use infer vector.

model = Load_model('model.doc2vec')
infer_vector = model.infer_vector(s)
similar_documents = model.docvecs.most_similar([infer_vector], topn = 1)

here similar document is list of tuples. where first element is label.
let me know if this help you.


I have n dimensional data and I want to check integrity, can I downgrade to 2 dimensional feature space via PCA and do so? Say I have n dimensional data samples. I want to check the integrity of the features, if they are good representation of the respective classes, i.e. these features are good or not.

My plan is:

I use PCA and convert this to 2 dim data. Plot this data. See if they're separable enough.

Does the above plan sound okay for testing to see if features are any good?

By good I assume you're trying to see if the features are different enough? It depends on your data set actually. As smallchess has pointed out, PCA is commonly used and may work with your dataset. Try it out and see if the different classes are distinct? If not, you can try using higher dimensional visualisation tools.

Besides, with PCA you're reducing the dimensions to new features that may or may not be intuitive in meaning. That makes interpretation more complicated. If you're looking to visualise data, why not go with higher dimension visualisations? In case you're not aware of the different available tools..
Here is a question with great answers on Quora about higher dimension visualisation

Of the tools recommended, Andrews plot and parallel coordinates plots are very popular. I would suggest using these tools instead of using PCA and subsequently plotting for the purpose you're describing. If you have no need to visualise the data, sometimes you can just get a statistical summary/description of the datasets (mean, variance, min, max, etc). Below is an example of an Andrews Plot. You can plot the various classes into different colours to see if there's any discernible differences between the classes.

If that's not what you're looking for, to visualise the data to gauge the representation, just proceed to try out and build a model with the features and see if you get a good score?

Edit: Here are some shortcomings of PCA..if there's information that's misrepresented, do let me know!
One thing to note is that with PCA, its transformation is orthogonal. Sometimes your data requires non-orthogonal representation. This is very dependent on how your data looks like. To give an example, take a look at the diagram below.

In the above example, PCA fails to find PCs that maximises the variance due to orthogonality. Independent Factor Analysis may work better. Also, if you're planning to use the transformed data to build your model upon, there may be limitations with the linearly transformed data. Sometimes we need non-linear PCs. See the below example from the scikit webpage.

In such a case, a kernel PCA (utilizing a kernel trick similar in SVMs) may work better by getting a projection to get better linear distinction.


Overfitted model produces similar AUC on test set, so which model do I go with? I was trying to compare the effect of running GridSearchCV on a dataset which was oversampled prior and oversampled after the training folds are selected. The oversampling approach I used was random oversampling.

Understand that the first approach is wrong since observations that the model has seen bleed into the test set. Was just curious about how much of a difference this causes.

I generated a binary classification dataset with following:

# Generate binary classification dataset with 5% minority class,
# 3 informative features, introduce noise with flip_y = 15%

X, y = make_classification(n_samples=5000, n_features=3, n_informative=3,
                            n_redundant=0, n_repeated=0, n_classes=2,
                            n_clusters_per_class=1,
                            weights=[0.95, 0.05],
                            flip_y = 0.15,
                            class_sep=0.8)

I split this into 60/40% train/test split and performed GridSearchCV with both approaches on a random forest model. Ended up with following output based on best_estimator_ from both approaches:

Best Params from Post-Oversampled Grid CV:  {'n_estimators': 1000}
Best Params from Pre-Oversampled Grid CV:  {'classifier__n_estimators': 500}
AUC of Post-Oversampled Grid CV - training set:  0.9996723239846446
AUC of Post-Oversampled Grid CV - test set:  0.6060618701968091
AUC of Pre-Oversampled Grid CV - training set:  0.6578310812852065
AUC of Pre-Oversampled Grid CV - test set:  0.6112671617024038

As expected, the Post-Oversampled Grid CV AUC is very high due to overfitting. However, evaluating both models on the test set lead to very similar results on AUC (60.6% vs 61.1%).

I had two questions. Why is this observed? I didn't assign a random_state to any of these steps and retried it many times, but still end up with the same results. In such a case, what becomes the better model to progress with since they're producing similar results on test set?

For oversampling and handling it through the pipeline, I made use of imblearn:

# imblearn functions
from imblearn.over_sampling import RandomOverSampler
from imblearn.pipeline import Pipeline as Imb_Pipeline

Happy to share my code if needed. Thanks.

If you don't provide any int for random_int for train_test_split it doesn't mean there isn't going to be anything. It defaults to None, check the documentation of train_test_split it says,

  If None, the random number generator is the RandomState instance used by np.random.

Now let's look at RandomState instance from the documentation of np.random. Take a look at these points:

  defaults to None. If size is None, then a single value is generated and returned.

and

  If seed is None, then RandomState will try to read data from /dev/urandom (or the Windows analogue) if available or seed from the clock otherwise.

what these mean is, you can achieve the results, the very same several times. But no guarantee it will work for others. There is nothing called random, everything uses pseudo-random. Somehow a "random_state" is generated.

My best guess would be to try: train:valid:test::80:10:10, or 60:20:20 after doing all this GridSearchCV compare your validation accuracy then try the best estimator on the test set. But this assumes you need to have more data.

Hope this helps. If anyone finds any mistakes, I'd be happy to get corrected.


Python is reading my data with NANS and Infs, but they don't have any I'm having an issue in Python where it says that the dataframe I have loaded through pandas.read_csv() cannot be scaled using StandardScaler()because of the presence of Inf values or values too big for dtype(float) I checked in R (I am more comfortable with this language, but have to use Python for this project) and it shows that the dataframe does not have any Inf values or NAs.

Are there restrictions in Python that converts large numbers or small numbers to NA/NaN/Inf?

Thanks.

You should use pandas.DataFrame.describe() to find the values that are present in the feature and see which specific values might be an issue.


2-label dataset for 3-label classifier? I have a dataset containing headlines and sentiment related to those headlines. The headlines have been filtered out from another bigger dataset using the following criteria: keep the ones that have a very negative or a very positive sentiment. At the end, I have a dataset with a very positve and a very negative sentiment headlines.
My goal is to create a deep learning classifier using tf and keras in order to classify new observations into three class: positive sentiment, negative sentiment and neutral sentiment. In other words, my goal is to use a binary labelled dataset to create a classifier that outputs a 3-label classification and I want to do it by predicting the probability of a headline to be positive or negative.
If the predicted probabilities of a headline are:
p(positive) = 80%
p(negative) = 20%

than the headline is positive. But if:
p(positive) = 50%
p(negative) = 50%

than the headline is neutral.
What do you think?

Yes this is a good strategy. The only thing you need to figure out is the threshold value for negative to neutral, and neutral to positive.
What you could do is decide on some precision value (say 95%) that you want to achieve when classifying a positive article as positive and a negative article as negative. Based on this desired precision you can calculate the threshold you would need to set by evaluating your trained model on a holdout set (if the positive threshold is too low maybe you won't be able to acheive 95% precision, or if it's too high it might be too restrictive and label most headlines as neutral).
The design decision for these thresholds depends on the "business problem" you are trying to solve or the ideal client/use of your tool.


How to shuffle only a fraction of a column in a Pandas dataframe? I would like to shuffle a fraction (for example 40%) of the values of a specific column in a Pandas dataframe.

How would you do it? Is there a simple idiomatic way to do that, maybe using np.random, or sklearn.utils.shuffle?

I have searched and only found answers related to shuffling the whole column, or shuffling complete rows in the df, but none related to shuffling only a fraction of a column.

I have actually managed to do it, apparently, but I get a warning, so I figure even if in this simple example it seems to work, that is probably not the way to do it.

Here's what I've done:

import pandas as pd
import numpy as np

df = pd.DataFrame({'i':range(20),
                   'L':[chr(97+i) for i in range(20)]
                  })

df['L2'] = df['L']

df.T

    0   1   2   3   4   5   6   7   8   9   10  11  12  13  14  15  16  17  18  19
i   0   1   2   3   4   5   6   7   8   9   10  11  12  13  14  15  16  17  18  19
L   a   b   c   d   e   f   g   h   i   j   k   l   m   n   o   p   q   r   s   t
L2  a   b   c   d   e   f   g   h   i   j   k   l   m   n   o   p   q   r   s   t

For now, L2 is simply a copy of column L. I keep L as the original, and I want to shuffle L2, so I can visually compare both. The i column is simply a dummy column. It's there to show that I want to keep all my columns intact, except for a fraction of L2 that I want to shuffle.

n_rows=len(df)
n_shuffle=int(n_rows*0.4)
n_rows, n_shuffle

(20, 8)

pick_rows=np.random.permutation(list(range(n_rows)))[0:n_shuffle]
pick_rows

array([ 3,  0, 11, 16, 14,  4,  8, 12])

shuffled_values=np.random.permutation(df['L2'][pick_rows])
shuffled_values

array(['l', 'e', 'd', 'q', 'o', 'i', 'm', 'a'], dtype=object)

df['L2'][pick_rows]=shuffled_values

I get this warning:

C:\Users\adumont\.conda\envs\fastai-cpu\lib\site-packages\ipykernel_launcher.py:1: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  """Entry point for launching an IPython kernel.

df.T

I get the following, which is what I expected (40% of the values of L2 are now shuffled):

    0   1   2   3   4   5   6   7   8   9   10  11  12  13  14  15  16  17  18  19
i   0   1   2   3   4   5   6   7   8   9   10  11  12  13  14  15  16  17  18  19
L   a   b   c   d   e   f   g   h   i   j   k   l   m   n   o   p   q   r   s   t
L2  e   b   c   l   i   f   g   h   m   j   k   d   a   n   o   p   q   r   s   t

You can see the notebook here (it's rendered better on nbviewer than here): https://nbviewer.jupyter.org/gist/adumont/bc2bac1b6cf7ba547e7ba6a19c01adb6

Thanks in advance.

I don't think there is any idiomatic way of doing this since it's quite unusual operation, normally the whole row or column should be shuffled. What you are doing looks like a good approach.

The error SettingWithCopyWarning you get is a common warning that you could be operating on a copy of the original data and not a view (the origianl). For more information I would recommend checking the answers here: https://stackoverflow.com/questions/20625582/how-to-deal-with-settingwithcopywarning-in-pandas.

To avoid the error and make the code more compact you could do it as follows:

import random

fraction = 0.4
n_rows = len(df)
n_shuffle=int(n_rows*fraction)

pick_rows = random.sample(range(1, n_rows), n_shuffle)

df.loc[pick_rows, 'L2'] = np.random.permutation(df.loc[pick_rows, 'L2'])

Note that the use of loc here will make sure that no copy is created and everything is done on the origianl dataframe (i.e. this will not give the SettingWithCopyWarning warning).


Why is word prediction an obsession in Natural Language Processing? I have heard how great BERT is at masked word prediction, i.e. predicting a missing word from a sentence.

In a Medium post about BERT, it says:

  The basic task of a language model is to predict words in a blank, or it predicts the probability that a word will occur in that particular context. Let’s take another example:

  “FC Barcelona is a _____ club”

Indeed, I recently heard about SpanBERT, which is "designed to better represent and predict spans of text".

What I do not understand is: why?

I cannot think of any common reason that a human would need to do this task, let alone why it would need to be automated.
This does not even seem to be a task where it is particularly easy to evaluate the success of a model. For example,

  My ___ is cold

This could reasonably be a number of possible words. How can BERT be expected to get this right, and how can humans or another algorithm be expected to evaluate whether "soup" is a better answer than "coffee"?

Clearly there are a lot of smart people who think that this is important, so I accept my lack of understanding is likely based on my own ignorance. Is it that this task itself is not important, but it's a proxy for ability at other tasks?

What am I missing?

The first line of the BERT abstract is

  We introduce a new language representation model called BERT.

The key phrase here is "language representation model". The purpose of BERT and other natural language processing models like Word2Vec is to provide a vector representation of words, so that the vectors can be used as input to neural networks for other tasks.

There are two concepts to grasp about this field; vector representations of words and transfer learning. You can find a wealth of information about either of these topics online, but I will give a short summary.

  How can BERT be expected to get this right, and how can humans or another algorithm be expected to evaluate whether "soup" is a better answer than "coffee"?

This ambiguity is the strength of word prediction, not a weakness. In order for language to be fed into a neural network, words somehow have to be converted into numbers. One way would be a simple categorical embedding, where the first word 'a' gets mapped to 1, the second word 'aardvark' gets mapped to 2, and so on. But in this representation, words of similar meaning will not be mapped to similar numbers. As you've said, "soup" and "coffee" have similar meanings compared to all English words (they are both nouns, liquids, types of food/drink normally served hot, and therefore can both me valid predictions for the missing word), so wouldn't it be nice if their numerical representations were also similar to each other?

This is the idea of vector representation. Instead of mapping each word to a single number, each word is mapped to a vector of hundreds of numbers, and words of similar meanings will be mapped to similar vectors.

The second concept is transfer learning. In many situations, there is only a small amount of data for the task that you want to perform, but there is a large amount of data for a related, but less important task. The idea of transfer learning is to train a neural network on the less important task, and to apply the information that was learned to the other task that you actually care about.

As stated in the second half of the BERT abstract,

  ...the pre-trained BERT model can be finetuned with just one additional output layer
  to create state-of-the-art models for a wide
  range of tasks, such as question answering and
  language inference, without substantial taskspecific architecture modifications.

To summarize, the answer to your question is that people DON'T care about the masked word prediction task on its own. The strength of this task is that there is a huge amount of data readily and freely available to train on (BERT used the entirety of wikipedia, with randomly chosen masks), and the task is related to other natural language processing tasks that require interpretations of the meaning of words. BERT and other language representation models learn vector embeddings of words, and through transfer learning this information is passed to whatever other downstream task that you actually care about.


Classification targets with heterogenous meanings I am training a classification model on a dataset of users on a website and each has 100 different measurements of their behaviour on the platform.

Most of these users are dormant but about 10% will reawken. We are interested in purchasing activity once awake.

We have separated the customers into two classes:

Those who reawaken and purchase products 5+ times (positive class)
Those who remain dormant and those who rewaken but purchase less than (or including) 4 times.

We also have a separate model that sorts customers into groups

Will reawaken.
Remain dormant

My question is:

If we output the probability of being in the positive class for both of these classifiers, what will the product of these two probabilities mean semantically?

Will it be P(5+ and reawaken), or will it be something else?

Will it be P(5+ and reawaken), or will it be something else?

The events are not independent, so one cannot assume that p(A and B) = p(A) * p(B).

Let's denote the events as follows:

A = reawakens (not A = remains dormant)
B = purchases at least 5 times

The event "reawakens and purchases more than 5 times" is "A and B". In general we have:

p(A and B) = p(A/B) p(B)
p(A and B) = p(A/B) (p(A and B) + p(not A and B))

But it's impossible for a customer to stay dormant and buy anything, so:

p(not A and B) = 0

Which gives us:

    p(A and B) = p(A/B) p(A and B)
    p(A/B) = 1

Then we obtain:

p(A and B) = p(B)

Note that this makes sense intuitively: the probability of buying at least 5 times is the same as awakening and buying at least 5 times.

Currently the labels of the two models overlap so I don't think you can infer much from combining their outputs. A way to make it usable would be to make the first model consider only awakening customers in order to avoid the overlap. But it might be more useful to train a single joint model to get a clear picture of your data. In general such a model would classify between 4 categories:

not A and not B
not A and B
A and not B
A and B

But since p(not A and B) = 0 there are actually only 3 labels corresponding to:

stays dormant
awakens and buys less than 5 times
awakens and buys at least 5 times


splitting of training examples into the mini batch: what to do with the rest tiny mini-batch? Lets assume I have 103 training examples. I want a mini-batch to be of the size 16. That means that there will be 6 mini-batches of the size 16 and one mini-batch of the size 7.

In the tensor flow one needs to specify the shape of the input:

x = tf.placeholder(tf.int32, shape=[batch_size], name='x')

which of course led to the following error:

Cannot feed value of shape (7,) for Tensor 'x_10:0', which has shape '(16,)'

So, what do I do with the mini-batch of size 7? Should I find the corresponding mini-batch size, that will create equal mini-batches? If so, how can I follow then the advice to create mini-batches of the power of two? or I should disregard the last mini batch?

A few basic approaches come to mind. It depends on your context and intention which might work best for you. Here are some ideas assuming that working with a partial batch is not an option:

Drop the remainder. This might make more sense when you have many complete batches, and you consider the set of examples to be enough.
Complement the final batch with duplicated samples from the other batches. E.g. randomly get one example from each batch.
If something like that is feasible at all, generate additional (possibly synthetic) examples to complete the final batch.


Searching prediction from 4 datasets The fourth dataset contains (train_data, test_data, previous_data, and information_history_data). The goal is to search for a user's rating on the loan to the bank. I am confused about the first step for doing this because there are many datasets (4). If the first step is to preprocess the data, which dataset will be preprocessed first? I am using the R language.

Thank you.

I am assuming this is competition data?

Basically, start with the previous_data and information_history data which seemingly provide extra information for observations in the training and testing datasets. You are probably given lookup ids that allow you to relate any particular row of these extra datasets to that of the training/testing datasets.

Most people will aggregate these extra variables so that one row = one unique lookup id (i.e., aggregate these datasets by id) by using various sample statistics; means, maxes, mins, variances, kurtosis, percentiles (like the median), etc. I've even seen people attach the target variable from the training dataset on to the "extra" datasets so that they can run regressions/classifiers on these datasets to get more features that give better aggregations. One extra suggestion: information_history and previous_data sounds like there is a time element. Perhaps be mindful of using more recent data, perhaps having features like "x mean in past k time periods", or "y median in n most recent loans", etc. I don't know what the datasets are but that is just a suggestion.

After that is done, then simply join these aggregated summary statistics to the training/testing data sets by the lookup = id column. In R, this is quite easy to do with the tidyverse and left_join/right_join, for example.

I hope this answers your question.


How to decide to go with BOW or TFIDF I know that there are methods that help in selecting features such as Matual Info, and Info Gain, etc.
But for datasets with thousands of records and thousands of features it is time consuming to train the model in BOW and TFIDF to decide which method is better.
is there a way to decide which method to choose without the need to spend all this time?

It depends on the problem you are trying to solve. If you know the signal in the dataset already, the words which decide your decision then go with Bag of Words. This is useful when you are doing something like text classification.
On the other hand, TF-IDF is useful when you don't know the signal in the dataset. If you want to do text similarity, then, this is a good option.


is it ok to get 100% accuracy in random forest classifier algorithm? while i was building the model to predict the performance of machine using the features like OEF,working time,performance/head etc...

I splitted the training data using

x_trainset, x_testset, y_trainset, y_testset = train_test_split(x, y, test_size=0.3)

then built the model

#Import Random Forest Model
from sklearn.ensemble import RandomForestClassifier

# Create the model with 100 trees
forest=RandomForestClassifier(n_estimators=100)

#Train the model using the training sets
train1=forest.fit(x_trainset, y_trainset)
y_pred=train1.predict(x_testset)

y_pred

o/p:

array(['level2', 'level5', 'level2', 'level3', 'level3', 'level2',
       'level3', 'level2', 'level4', 'level2', 'level2', 'level3',
       'level4', 'level3', 'level3', 'level2', 'level2', 'level3',
       'level3', 'level3', 'level3', 'level3', 'level2', 'level5',
       'level3', 'level3', 'level2', 'level5', 'level4', 'level3',
       'level3', 'level3', 'level2', 'level2', 'level3', 'level3',
       'level3', 'level2', 'level4', 'level2', 'level2', 'level3',
       'level4', 'level3', 'level1', 'level3'], dtype=object)

y_testset

o/p:

120    level2
104    level5
117    level2
91     level3
4      level3
10     level2
133    level3
18     level2
119    level4
61     level2
71     level2
30     level3
27     level4
103    level3
14     level3
59     level2
50     level2
55     level3
53     level3
22     level3
78     level3
114    level3
70     level2
60     level5
6      level3
1      level3
83     level2
82     level5
26     level4
67     level3
62     level3
145    level3
74     level2
11     level2
107    level3
29     level3
138    level3
49     level2
150    level4
8      level2
20     level2
95     level3
51     level4
56     level3
72     level1
102    level3
Name: levels, dtype: object

then found the accuracy and i got like,

from sklearn import metrics #Import scikit-learn metrics module for accuracy calculation
print("Accuracy:",metrics.accuracy_score(y_testset, y_pred))

o/p:

Accuracy: 1.0

In x_trainset,x_testset i have only the features(7) and in y_trainset,y_testset i have only column "levels" which is needed to be predicted

dataset size is about 153

I don't know whether this is correct.Please can anyone answer this as soon as possible

There could be many reasons why you achieved 100% accuracy.One of them could be:Duplicates in your data which are repetitive in both train and test data.I would suggest you to try the following steps:
1.Check if there are any duplicates in the original data
2. Try a different Train-Test split like 80-20
3.Try k-Fold cross validation
4.Checkout for Precision and Recall apart from accuracy.
Hope this helps!


Your histogram is valid, but it has too many bins to be useful.

If you want a number of equally spaced bins, you can simply pass that number through the bins argument of plt.hist, e.g.:

plt.hist(data, bins=10)

If you want your bins to have specific edges, you can pass these as a list to bins:

plt.hist(data, bins=[0, 5, 10, 15, 20, 25, 30, 35, 40, 60, 100])

Finally, you can also specify a method to calculate the bin edges automatically, such as auto (available methods are specified in the documentation of numpy.histogram_bin_edges):

plt.hist(data, bins='auto')

Complete code sample

import matplotlib.pyplot as plt
import numpy as np

# fix the random state for reproducibility
np.random.seed(19680801);

# sum of 2 normal distributions
n = 500;
data = 10 * np.random.randn(n) + 20 * np.random.randn(n) + 20;

# plot histograms with various bins
fig, axs = plt.subplots(1, 3, sharey=True, tight_layout=True, figsize=(9,3));
axs[0].hist(data, bins=10);
axs[1].hist(data, bins=[0, 5, 10, 15, 20, 25, 30, 35, 40, 60, 100]);
axs[2].hist(data, bins='auto');


Docker for data science I recently started to read articles about Docker.
To me, in data science, Docker is useful because:

1) You have a totally different environment, which protect you against libraries and dependencies problems.

2) If your application modify, for example, the database of your company, you want first to make sure that the code works fine, and won't have bad consequences on the database. Thus, you use Docker to test your code first.

my questions:

Am I right if I say that only the second reason is about sand boxing? The first reason has nothing to do with sand boxing, right?
Are there other reasons why Docker is useful in Data science??
I don't find a lot of interesting research papers about Docker for data science. Do you know some famous ones?

Instead of focusing on the technology term, will provide generalized answer and use the term “containers”.

Containers run only what it supposed to run, assuming anything unknown is untrusted so what only reside in container only for the life of container, so modification of a code in database to test will be same approach in VMs(sandboxing) or containers(docker) and biggest difference is kn resource consumption and time for provisioning VMs vs spinning up containers/pods in couple seconds for application.

  More details:

  Part1_ From application point of view

Containers are very important when does come to data science world from following points:

Minimizing conflicting library versions for different development projects(each project has own container/pod to run).
Consistency across environments and developers against specific design criteria.
Rapid Prototyping with Edge Node On Demand.
Avoiding a need to reinstall everything on new hardware after a refresh or failure.
Portability of your compute environment, develop in your machine and if you need more resources like CPU, RAM or GPU you can port your environment to cloud, powerful clusters, powerful machine, group of containers that can be orchestrated with Kubernetes or Dockerized cluster called Swarm.
Maximize collaboration clarity across groups. Consistent libraries, result sets, etc.
Extending on-premises autonomy/agility to cloud scale and reach.
Project workspaces powered by Docker containers for control over environment configuration. You can install new packages or run command-line scripts directly from the built-in terminal.
Resources used by containers are very few comparing with VM or Bare metal ... you can have multiple containers running in VM or Bare metal so this will greatly reduce license cost for OS and projects/apps will scale based on resources needed and not full resources of the host( here to note: you will run multiple apps/models in a machine when using containers comparing without containers you’ll use full machine resources for 1 single app/model and this is waste of resources. In a cluster environments you can spin up multiple pods/containers(apps/models across the cluster and with zero cost of the pod comparing with VMs you pay for OS of each host and you run all resources to that task).
easy to have new image with new packages installed and share with everyone instead to deal with virtual environments into VMs and have conflict of packages and many more(would like to see virtual environments to be moved from traditional approach to containers, that will be a good improvement and will help a lot for Data scientists to bypass the configuration of each project individually with all conflicts and activate/deactivate then search for requirements when putting in production when with containers all you need is a config file, this is my point of view from what I see daily in Data Science world).
Infrastructure independent platform allows data scientists to run the the analytics applications on the infrastructure best optimized for the applications
Empower data scientists to build models with the tools and packages best suited for the research without worry of application and environment conflicts.
Most important Research Reproducibility: The SCALE provided by containers, you scale easily with containers and will be identical across all environments and you don’t care about the host OS, ensuring repeatability of analysis and research with immutable containers and eliminating issues with different environments
Secure Collaboration: Integrated security in the platform and lifecycle allows for collaboration without risk of tampering and data integrity(more details in Part2&3.

Here comes in play Cloudera with  Cloudera Data Science Workbench  and Anaconda with Anaconda Enterprise both does use containers so can bring quick results for the business and deploy easy the models from dev to QA and ultimately to production.

Why is important last statement? is to have portability from dev to prod without any changes to environments and without cost DevOps part to operationalize.

  Part2_ From security point of view

One notorious security advantage is that the host OS security is separate from the container, meaning that is patched separated, patching host OS wouldn’t impact your containerized application(how many times we have the issue when we do patch the OS and does affect the app  and service in that OS(paths, ports, services etc)?

For example: if you have a malicious library on your app/code than that malware will reside only on that container for the time of container being live, containers does run as endpoint all the time and didn’t saw a case that will spread the malware into network.
monitoring a container node is robust and you can put add-on or service to monitor behaviors of application or node, and will only replicate new node/container only and only based on config file.

Comparing VMs vs containers: With containers is a different story, you take care of OS separate from containers(containers is separate task when security is in place).

Docker security does provide detailed information of major security points.

Docker standards and compliances provides a full list of security compliances and standards available for containers.

"Docker container with a well crafted seccomp profile (which blocks unexpected system calls) provides roughly equivalent security to a hypervisor."

Folder sharing. With containers, you can share a folder by setting up a shared mount, and since the Docker/kernel enforces file permissions that is used by containers, the guest system can't bypass that restrictions. This is very important for Data Science applications and users because in corporate is used most of the time sensitive/restricted data and multiple layers of security or restrictions are in place, one approach to solve this security problem is using VDI or VMs that with AD group in place to restrict/share data access and does become problematic and costly to maintain and allocate resources.

When does come to debugging applications or OS with all services and logs generated I think containers are winning and evolving now to NLP approach: An experimental natural language processing (NLP) utility is also included, for proofreading security narratives.

  Here I will quote Jianing Guo from Google: A properly secured and
  updated VM provides process level isolation that applies to both
  regular applications as well as container workloads, and customers can
  use Linux security modules to further restrict a container’s attack
  surface. For example, Kubernetes, an open source production-grade
  container orchestration system, supports native integration with
  AppArmor, Seccomp and SELinux to impose restrictions on syscalls that
  are exposed to containers. Kubernetes also provides additional tooling
  to further support container isolation. PodSecurityPolicy allows
  customers to impose restriction on what a workload can do or access at
  the Node level. For particularly sensitive workloads that require VM
  level isolation, customers can use taint and toleration to help ensure
  only workloads that trust each other are scheduled on the same VM.

On top of all, kubernetes cluster has following additional security features:

Use Transport Layer Security (TLS) for all API traffic
API Authentication
API Authorization
Easy remediation of security flaws and minim impact on application/environment compared with VMs or dedicated Bare metal.

  Part3_CIS benchmarks for hardening containers: Docker & Kubernetes.

First step making containers more secure is to prepare the host machine that is planed to use for executing containweized workloads. By securing containers host and following infrastructure security best practices would build a solid and secure foundation for executing containerized workloads.
Stay up to date on Docker updates, vulnerabilities in the software.
Have a specific dedicated directory for Docker related files and allocate just enough space to see for containers to run(default path is /var/lib/docker but change to other mounting point and monitor at OS level using auditd or aide services for any changes or size/unwanted workload run, keep the logs and configure according to the needs.

Note: The best part about step 2 is that with VMs you need to monitor way more locations for your data science project(libraries in different location, multiple versions of packages, locations/path for even python, run cron jobs or systemd to ensure some process does run, log all logs etc but with containers is single point for all this jobs to run and monitor only a path instead of multiple ones).

Verify all the time the users into docker group so you prevent unauthorized elevated access into the system(Docker allows the share of directory between the Docker host and a guest container without limiting the access rights of the container) so remove any untrusted users from the docker group and do not creat a mapping of sensitive directories frim the host to container volumes. Here I would say to use a separate user for the installation and specific container tasks "NEVER use root for containers run dedicate a PID only for this task(will have elevated access but will be task based, I use gravitational for the cluster and when installing I NEVER use root).
Audit all Docker daemon activities(be aware that does take space the logs in containerized "world" so prepare separate partition with decent space to hold the logs and required configuration(rotation and period to store the logs).
Audit all Docker files and docker.service , in etc, var and what else is aplicable.
Restrict all inter-container communication, Link specific containers together that are require communication(best will be to create a custom network and only join containers that need to communicate to that custom network). This hardening approach will prevent unintended and unwanted disclosure of information to other containers.
All applications in containerized infrastructure should be configured or at least to have th eoption to Encryp All Sensitive Information(this is very importan for Data Scientist because most of the time we do login to platforms to get data, including sensitive data for the company.
Have option to have Encrypted All Sensitive Infomration in Transit.
Uses only specific approved Ports, Protocols and Services, VMs has more open surface when does come to run an app/project, with containers you specify only what will be used and not wondering for all other ports listening, protocls and services that does run at OS level to protect or monitor, this does minimize the "attack surface".
Sensitive information stored on systems is encrypted at rest and does require secondary authentification mechaninsm , not integrated into operating system, in order to access the information.
Does allow to be enabled Operating System Anti-Exploitation Features/Deploy Anti-Exploit Technologies: such as Data Execution Prevention(DEP) or Address Space Layout Randomization (ASLR).
The best simple security difference between VMs and Containers is: when updating or running a project you don't need elevated access to do it over entire VM or network, you just run as a defined user and if is elevated access then does exist only for the time of the container and not shared across of the host(here does come Data Science libraries install, update, executing projects code etc).

  Part4_ More resources(on top of links incorporated in Part1-3) related to containers for Data Science:

Data Science on docker
Conda, Docker, and Kubernetes: The cloud-native future of data science (sponsored by Anaconda)
https://devblogs.nvidia.com/making-data-science-teams-productive-kubernetes-rapids/
Why Data Scientists Love Kubernetes
A very good book  to understand docker usage for Data Science: Docker for Data Science: Building Scalable and Extensible Data Infrastructure Around the Jupyter Notebook Server.


How to deal with with rows with zero in every feature while clustering? I am working on a clustering problem which has 13000 observations and 15 features. Around 3000 observations in the dataset has  zero in every features ( i.e all values zero in 3000 rows). I am trying to do clustering on top of it. What is a better way to deal with it ? I have few things in my mind but would like to get clarity on :

Check for number of rows with all zero and remove them ?
Include the rows with zero value in every feature and let the clustering
algorithm handle the same?

Also,please suggest if there are any better way to handle the same.

Note : I am using k-means clustering.

It's a matter of data quality so it depends how the dataset was built:

Either these instances are meaningful, i.e. it makes sense that an observation would have zeros for all the features and that it would happen that often.
Or these are the result of an error, typically the complete absence of measurement for these observations.

Naturally one wants to keep the instances in the data in the former case but (usually) not in the later case, because the values don't represent an actual data point so they would introduce a massive bias for the clustering algorithm.


tensorflow pseudo inverse doesn't work for complex matrices! The Tensorflow documentation here says that:

  tf.linalg.pinv is ''analogous to numpy.linalg.pinv. It differs only in
  default value of rcond''.

  However, tf.linalg.pinv requires the matrix to be float type while
  np.linalg.pinv could be used with complex matrices.

I was wondering why they would only create it for float types and if there is a straightforward way to modify tf.linalg.pinv to be used with complex matrices.

I just faced the same situation. If you need to explicitly build the inverse, check this paper:

https://pdfs.semanticscholar.org/f278/b548b5121fd0d09c2e589439b97fad16ece3.pdf

In particular, given a Matrix M that you need to invert, you can just do:

    A = tf.math.real(M)
    C = tf.math.imag(M)

    r0  = tf.linalg.pinv(A) @ C
    y11 = tf.linalg.pinv(C @ r0 + A)
    y10 = -r0 @ y11

    M_inverse = tf.cast(tf.complex(y11,y10), dtype = M.dtype)

The complexity is a bit higher than the pure-complex implementation, but so far it has proven to be pretty stable for me.

(just copying my answer from your other post:
https://stackoverflow.com/questions/60025950/tensorflow-pseudo-inverse-doesnt-work-for-complex-matrices/60128892#60128892)


statistics or robust statistics for identifying multivariate outliers For the single variate data sets, we can use some straightforward methods, such as box plot or [5%, 95%] quantile to identify outliers. For multivariate data sets, are there any statistics that can be used to identify outliers?

Multivariate outlier detection can be quite tricky and even 2D data can be difficult to visually decipher at times.  You are spot-on in looking for robust statistical treatments analogous to 95% quantiles.

Where as normally distributed data naturally aligns with the chi square distribution, the gold standard for robust statistics in n dimensions would be to use Mahalanobis distances and then eliminate data beyond 95% or 99% quantiles in Mahalanobis space.

Plug and play capabilities are available in scikit-learn and in R.

Here is an excellent theoretical and practical treatment of the methodology:

And here is a big picture viewpoint with some heuristics.

Additionally there is a very sophisticated treatments called PCOUT for outlier detection that instead rely on principal component decomposition.  There is a corresponding R package, but the theoretical treatment is behind a paywall:

P. Filzmoser, R. Maronna, M. Werner. Outlier identification in high dimensions, Computational Statistics and Data Analysis, 52, 1694-1711, 2008

Hope this helps!


How close is close enough, with regression? When exploring different techniques in machine learning (neural networks), I like to use binary classification problems as a test-bed, because it's very easy to understand how well the technique is working: 50% training/test accuracy is no better than chance, 70% is okay, 98% is very good, etc.
However, sometimes I need to use regression problems instead, and here I struggle to interpret my results. If I draw a scatter plot of training and test mean squared error before and after training, and before training I get around 0.1 loss, and afterwards I get 0.01 loss... is that... good? How do I recognize the difference between success, and barely doing any better than chance?
Typically I'm working with synthetic problems, by the way: fitting polynomials or other kinds of mathematical functions on random vector inputs.

I think it is a far question, since we should always try to translate any metric result to a useful interpretation.
When talking about regression, we can judge the success of our model by checking whether that MAE or RMSE exceed what we are willing to accept.
For instance, if we want to predict the temperature for the next days, and our result (in this case in a time series problem type) is a MAE of about 2ºC (after re-scaling our data to the natural units like celsius degrees), is it enough for our goal? Is a mean (absolute) error of 2ºC above or below the real temperature enough for the case being studied? Maybe it is for an approximate model, but maybe it is not a good result if we want to use it in an environment where tempreature could have more impact on whatever being studied...
Another additional chance you have is estimating confidence intervals (see this answer) for your model coefficents and, more precisely, prediction intervals (see this other validated answer) to have an estimate for your prediction together with an interval.


Dense text representations I did a lot of Googling but could not find a paper that presents an algorithm which will produce dense feature vectors for short text input. I would be happy to find feature extraction algorithm which would perform at least as good as sparse word unigram and bigram feature vectors.

Currently I am exploring the idea of using LDA (Latent Dirichlet Allocation) but there are problems with processing short text (2-7 words per document).

The task at hand is short text classification. The number of classes for my data ranges from 10 to 20 classes. The classes are fairly well represented and the word unigram and bigram features work well. I would like to compute dense feature vectors for other experiments.

Any pointers to papers, preferrably simple to implement, would be appreciated.

At my work we've had some success just using averages of pre-trained embeddings (e.g. the GloVe vectors) for classifying short texts. Have you tried that?


Can FTRL be applied on linear least squares? or is it just for logistic regression models? I'm exploring follow-the-regularized-leader FTRL proximal gradient descent:
paper, reference implementation.

Everywhere FTRL is mentioned, the loss surface for the gradient decent is the LogLoss, and the model for prediction is Logistic regression.

Can I use the same algorithm for a linear least squares model? I have a problem I want to model with a linear model and define the loss by least squares, and then do FTRL to find the optimal solution - do you see any problem with that?

Thanks.

As we can see, FTRL is nothing to do with our model, and it only serves as a way of approximating the optimal solution , just as SGD does.

Logloss is just a kind of loss pattern in classification problem,and there are other alternatives for it , such as ExpLoss. Least-squres loss is one of all loss patterns in regression problem.

The reason why FTRL was mentioned everywhere ,which mainly treated Logloss as a example, is that it was often used in classification problem , such as CTR model in online advertisement scenario.

Hopes this can helps you ,and good luck !


What is the difference between BatchNorm and Adaptive BatchNorm (AdaBN)? I understand that BatchNorm (Batch Normalization) centers to (mean, std) = (0, 1) and potentially scales (with $ \gamma $) and offsets (with $ \beta $) the data which is input to the layer. BatchNorm follows this formula:

(retrieved from arxiv-id 1502.03167)

However, when it comes to 'adaptive BatchNorm', I don't understand what the difference is. What is adaptive BatchNorm doing differently? It is described as follows:

(retrieved from arxiv-id 1603.04779)

I think the original batch normalization paper proposes to use mean and standard deviation estimated on the train set. The adaptive batch normalization simply re-estimates them on the target domain (could be the test set, or some unlabeled data from the target domain).

Please correct me if I am wrong.


How to run two different models in single frame? I have mask_detector.model and yolov3 social distancing weights.
I want to run them simultaneously with a single webcam stream.
how can I run them both i.e. detecting mask and social distancing model together?

Use ray or Numba for Parallel Computation. Question is "are not you able to do transfer learning to detect both in one single model"?


How does combining two linear perceptrons create non-linear boundaries? I don't understand the equation that you get from combining the two linear perceptrons is non-linear?

The video starts with two linear perceptrons with the equations:
$$e1 = 5x_1 -2x_2 - 8 = 0 \hspace{10ex} e2 = 7x_1 - 3x_2  + 1 = 0 $$

  Note: The bias unit signs flip between the written equation and the
  neural network diagram. I am using the negative sign since it is
  continued throughout the rest of the video

Then we go on to combine them with respective weights and biases as follows:
$$ 7e_1 + 5e_2 -6 = 0$$
When I do the math, I get:
$$ 7e_1 + 5e_2 - 6 = 0 $$
$$ 7(5x_1 -2x_2 - 8) + 5(7x_1 - 3x_2 + 1) - 6 = 0$$
$$ 35x_1 - 14x_2 - 56 + 35x_1 -15x_2 +5 -6 = 0$$
$$ 70x_1 - 29x_2 - 57 = 0$$

The resulting equation is very much linear, however, the idea is that that this generates a non-linear equation (and model).

What am I doing incorrectly in the way I am combining the two models?

You are correct that stacking two layers with a linear activation function on top of each other does not do anything that a single layer could not do (i.e. it is still a linear combination of terms).

This changes, once you use other activation functions. Then, once you combine neurons from the previous layer in the next layer e.g. as $w_0 + w_1 f(e_1) +w_2 f(e_2)$ for some suitable activation function $f$ such as ReLU, the sigmoid function etc., and then stack several such layers on top of each other, you can get substantially more complex functional relationships.


high precision and recall but low cross validated accuracy I'm using a random forest classifier (scikit learn). When I do a 5-fold cross validation, the average accuracy score is around 70%. However, when I look at precision_score and recall_score, they are both 1. Is that possible? If it's wrong, what could be the problem? Here is the code:

clf = RandomForestClassifier(n_estimators=100)
clf.fit(X, Y.values.ravel())
scores = cross_val_score(clf, X, Y.values.ravel(), cv=5, scoring="accuracy")
Y_pred = clf.predict(X)
precision = precision_score(Y, Y_pred)
recall = recall_score(Y, Y_pred)

First of all, you should use cross_val_predict to get you predictions vector, so that you followed approximately the same validation scheme to get them :

Y_pred = cross_val_predict(clf, Y, cv=5)

Then, it is totally possible to have 1 for recall or precision given that these are the scores for the considered positive class for scikit. In fact scikit assigns by default one of your class as the 'positive' one and to metrics inherent to classes are computed according to it.

However, as you said you can't have 1 for both and 70% accuracy.

The fact is that your clf.predict gave you 100% accuracy while during cross validation it was not the case because in your second scheme:

clf.fit(X, Y.values.ravel()) and Y_pred = clf.predict(X)

you train and predict on the same data ! So you obtain 100% accuracy which is common, while during cross validation :

scores = cross_val_score(clf, X, Y.values.ravel(), cv=5, scoring="accuracy")

you trained and tested on different folds which logically will lead to worse results (but closer to real condition experiment).

Besides, try to use cross_val_predict and then precision_recall_fscore_support(Y, Y_pred) in order to have a detailed report of metrics for each class.


Recommendation system depend on user rating and favorite list In my project I have a database of Japanese Sake(rice wine).Each Sake has following attributes which has direct link to the taste of the Sake:

classification (enumeration 1-5 integer)
alcohol percentage (1-100 integer)
polish percentage (1-100 integer)
acidity (double)

Users of the system can :

Rate Sake (1-5 double value)
Add Sakes to his/her favorite list.

Depend on these user data, I am thinking of developing a  recommendation system for sake.
I am looking at Collaborating Filtering methods,but guess it doesn't fit this scenario?

What kind of algorithm/techniques can be used for this kind of user recommendations?

[Edit] I am looking for a recommendation algorithm not only considering user rating,but also the taste attributes.

You can use collaborative filtering. From the user ratings you can define a ratings matrix with missing ratings that your algorithm will predict to recommend items to users.

The items data and user's whishlist can be included as auxiliary information (basically you cluster Sakes given their alcohol percentage, polish percentage and acidity and users given their whishlist using a K-mean algorithm). Have a look at this paper:

https://pdfs.semanticscholar.org/a0bc/3d1c05da5b57bee066368ffd0839247832c8.pdf


You have a data frame with 4 columns, each columns has 25 rows. You want to replace every 5th element with an outlying value (the mean of the column).

apply(x,2,function(i){
    i[seq(5,25,5)]=i[seq(5,25,5)]+mean(i)
})


When would not normalizing input values have higher accuracy? Right now I'm training a deep neural network for a binary classification problem, with a feature set of winrates. As such, each winrate is bigger or equal to 0 but smaller than 100.

I've been getting promising results without normalizing the input data, until I normalized it and got staggeringly worse accuracy.

The input feature is a 2d matrix of size 20, and the network has four layers with differing numbers of nodes in each layer. I'm using sgd optimizer and ReLU activation for the hidden layers, and the softmax activation function for the output layer.

The thing I'm wondering is why I'm getting better results with the neural network without the normalization? Is it because the optimal hyperparameters required for the network with the normalized input are different from when it is not normalized?

The thing I'm wondering is why I'm getting better results with the neural network without the normalization?

From a theoretical standpoint, normalizing your input shouldn't affect accuracy, in a sense that a NN can converge to any range of input values.

The reason why we normalize our data when using NNs is that it offers some useful properties, mainly regarding convergence and convergence speed. You can read more details here.

  Is it because the optimal hyperparameters required for the network with the normalized input are different from when it is not normalized?

Yes, I think that is your problem. Some hyperparameters might be optimized for the regular input and might not work well for the normalized one. An example of this could be a higher learning rate, which might work for a higher range of input values but not for a normalized one.


How to cluster words automatically? I have a problem where I have a list of n words with truly k different ones (k is unknown) because some may be malformed or contracted. I would like to automatically cluster them.
I thought about using something like a dendogram with a distance between words (I don't know which), how can this be handled efficiently?

You can get a dendrogram from any hierarchical clustering method. The tricky thing here is how to compute the distances between the words. If efficiency is your main concern, I would consider using HDBSCAN clustering.
The Jaro-Winkler distance was originally designed for such tasks. There is an efficient implementation in the python Levenshtein package, but it still might be too slow for computing the distance between all pairs of words.
If this is still too slow, you can opt for some more approximative methods and represented the strings as vectors - bags of characters (or character n-grams). You lose some information about how exactly characters are ordered in the string, but you will get them represented as vectors and vector comparison is fast. In this way, you can use algorithms that do not require comparing all pairs of data points such as K-Means.


Autoencoder for cleaning outliers in a surface I have been looking at autoencoders from the keras blog here: https://blog.keras.io/building-autoencoders-in-keras.html

I was wondering, what motifications would be necessary in order to be able to give it different surfaces i.e. 2-dimensional vectors, of which some of them have large spikes.

For example here we see a surface that looks clean:

How could a neural network look like, if I wanted to remove individual spikes from this surface?

Am I right in thinking that a normal fully connected feed forward propagation would be sufficient? If so, is there any way to control thresholds when spikes are should be eliminated?

Also, would you agree that the training principle would still be the same as show in the keras blog? Would it work if I simply trained it with many good examples of clean surfaces to recognize themselves?

A possible approach would be a denoising autoencoder. It is like a normal autoencoder but instead of training it using the same input and output, you inject noise on the input while keeping the expected output clean. Hence, the autoencoder learns to remove it. This kind of autoencoders are also described in the blog post you linked to.

In your case, you could just train your denoising autoencoder injecting to the inputs spikes of the height you expect to be removed.

About what kind of architecture (e.g. fully connected, convolutional), only actual tests can tell you what is appropriate and what is not.


importing csv data in python I have a csv file with around 130 columns and 6000 rows

what is the best way to import them into python, so that I can later use them in a classification algorithm(columns are the labels and rows are individual samples)

For small data, I think pandas.read_csv is the way to go.

For "medium" data, I recommend dask.read_csv

And for big data, I recommend spark.read.csv


Binary Classification - One Hot Encoding preventing me using Test Set I have a preprocessing pipeline that includes replacing missing values and onehotencoding for the categorical variables.

When I try to use my model on the test set, it explains that the number of columns it expects differs. This is due to one hot encoding

One option I considered was passing the full dataset into the pipeline and then seperating into test and split. However, this causes data leakage as the missing values it capturing values from the testset.

Please let me know how to prevent this.

Thanks,

You can use handle_unknown parameter of sklearn while encoding training data.

sklearn.preprocessing.OneHotEncoder(handle_unknown='ignore')

When this parameter is set to ‘ignore’ and an unknown category is encountered during transform, the resulting one-hot encoded columns for this feature will be all zeros.

Note : I assumed you are using scikit-learn.

Source : sklearn.preprocessing


pandas apply function with multiple condition? if i want to apply lambda with multiple condition how should I do it?

df.train.age.apply(lambda x:0 (if x>=0 and x<500))

or is there much better methods?

Alternatively, you can use loc:

import pandas as pd

df = pd.DataFrame({"age": [-100, 300, 400, 500, 600, 700]})
df["age"].loc[(df["age"] < 500) & (df["age"] >= 0)] = 0

Now your df looks like this:

    age
0   -100
1   0
2   0
3   500
4   600
5   700


Can BRAT be installed on Microsoft Windows? I would like to create some annotations on some texts using BRAT (brat rapid annotation tool). Can BRAT be installed on Microsoft Windows? If so, how?

The installation instructions do not rule out Microsoft Windows:

  The brat server is a Python (version 2.5+) program that runs by default as a CGI application, and the installation script assumes a UNIX-like environment. If you are setting up brat server in a compatible environment with an existing web server that supports CGI, the quick start instructions for using CGI should work. If you don't have a web server installed and want to try brat out locally, you may want try out the quick start instructions for the standalone server. (For security reasons, we strongly recommend serving brat via a full web server such as Apache in production environments.)

However they don't seem to give any instructions for Microsoft Windows.

I don't want to use some in emulation of Unix such as virtual machines or Cygwin.

If interested, to use BRAT with Cygwin: https://github.com/Franck-Dernoncourt/NeuroNER/blob/master/install_windows.md

  After installing Cygwin, run Cygwin.bat (by default located on
  C:\cygwin64\ if you used the 64-bit installation for Cygwin). Then,
  in the terminal that just opened, run:

mkdir brat
cd brat
wget http://weaver.nlplab.org/~brat/releases/brat-v1.3_Crunchy_Frog.tar.gz
tar xzf brat-v1.3_Crunchy_Frog.tar.gz
cd brat-v1.3_Crunchy_Frog
./install.sh -u
python standalone.py

  BRAT should now be accessible through the web browser at
  http://127.0.0.1:8001.

The installation documentation shows that the easiest way to run a brat server on Windows is in a virtual machine running a UNIX-like operating system such as Ubuntu. Also, there's an issue (still opening) about running brat server on windows. I think you're on your own right now.


Can we make two separate models vs one for classification? Suppose I have a binary classification problem and my data is imbalanced, I can build a classification model using any of the algorithms and use an oversampling or undersampling technique to handle the class imbalance.

What If I were to make two separate models independently, each model trained only for one class of data. Will this be a right approach?

That doesn't make much sense, no. If you provide all the data to both models, they should perform exactly the same. If you only provide positive data to each model, you're making two incomplete models that (supposedly) won't learn as well as using all the available data. Making multiple models to a single response variable can make sense, but not when the response is binary.

If you happen to also have unlabeled then it's a different thing, because you can use techniques such as the two-view models. But in practice, it's often more useful to spend resources labeling data than developing a semi-supervised model.


Why does Siamese neural networks use tied weights and how do they work? Reading this paper on one-shot learning "Siamese Neural Networks for One-shot Image Recognition" I was introduced to the idea of Siamese Neural Networks.

What I did not fully grasp was what they meant by this line:

  This objective is combined with standard
  backpropagation algorithm, where the gradient is additive
  across the twin networks due to the tied weights.

Firstly, how exactly are they tied? Bellow, I believe I've provided the formula by which they compute the gradient. T is the epoch, $\mu_j$ is the momentum, $\lambda_j$ the regularization, $\eta_j$ the learning rate, $w_{kj}$ I believe to be the weight between neuron k and in one layer and j in another but correct me if I'm wrong.

\begin{equation}\begin{array}{c}
\mathbf{w}_{k j}^{(T)}\left(x_{1}^{(i)}, x_{2}^{(i)}\right)=\mathbf{w}_{k j}^{(T)}+\Delta \mathbf{w}_{k j}^{(T)}\left(x_{1}^{(i)}, x_{2}^{(i)}\right)+2 \lambda_{j}\left|\mathbf{w}_{k j}\right| \\
\Delta \mathbf{w}_{k j}^{(T)}\left(x_{1}^{(i)}, x_{2}^{(i)}\right)=-\eta_{j} \nabla w_{k j}^{(T)}+\mu_{j} \Delta \mathbf{w}_{k j}^{(T-1)}
\end{array}\end{equation}

My other question is why this is even desirable?
Why not just reuse the same network twice?
Or perhaps the two networks will be identical after training?
If the networks are identical after training, why would you set it up like this?
What benefits does it have?

Have a look at this post: https://datascience.stackexchange.com/a/43820/98414
which should provide the intuition behind it.


Handling imbalanced data by deleting over represented rows vs. adding under represented rows I am currently working with a very imbalanced data set
(frauded credit card data from kaggle, which has 492 rows of frauded cards and over 280,000 rows of non-frauded cards).

As much as I know, there are 2 ways to handle imbalanced data without adjusting penalty:

over-sampling the under-represented data - by creating a lot of copies of the frauded data
under-sampling the over-represented data - by not using a lot of the non-frauded data

Right now I go with under-sampling (since my computer is not that strong and I do not want to waste hours for answers).

Technically what I do, is create a 50-50 dataframe, which has all of the under-represented rows and randomly selected over-represented rows.

Problem is, every time I run the notebook, the algorithms use different random okay data and there are different results. When tweaking xgboost, there are different optimal parameters every time I run the notebook.

What should I do? Should I save as csv one of the balanced df I created and call it a day? Should I creat a lot of copies from the frauded cards? What is the best way to tackle the situation?

Technically what I do, is create a 50-50 dataframe, which has all of the under-represented rows and randomly selected over-represented rows.

You could use a random seed for the selection from the non-frauded cards dataset so that every time it selects the same rows. This way you are able to track and compare the results.

import numpy as np
from numpy import random

arr = np.arange(5)  # [0, 1, 2, 3, 4]
random.seed(1)  # Reset random state
random.shuffle(arr)  # Shuffle!
print(arr)

Just an example on how to use random seed from numpy, you'll have to come up with your own way to use it for your need (suggestion: create arr in range(0, len(frauded_card)) and shuffle it, then use arr as indexes for the frauded_card sample to use)


Classifier that optimizes performance on only a subset of the data? I'm working on machine learning problem where I'm only interested in getting high accuracy within a narrow band of my predicted likelihoods. Specifically, I want an algorithm that will score very accurately when it predicts a likelihood above a specific threshold.

A motivating example:

I'm presented with 1 million closed boxes, 10 percent of which contain a gold coin. The likelihood of a box containing a gold coin is associated with physical characteristics of the boxes. I'm allowed to open 500K boxes, chosen at random, to see which contain the coins.  After this, I'm allowed to choose 100 of the remaining 500K boxes and keep any gold that I find.

I'm looking to build a classifier to solve this problem.  Specifically, I'm looking for the correct way to incorporate the constraint that I am only able to open 100 boxes in the test sample. Put another way, my measure of performance is limited to a subsample of my test data, so I need a classifier that is highly accurate when it predicts the highest likelihoods. Prediction errors outside of these 100 boxes are of zero concern.

I'm aware of cost-weighted loss functions that penalize false positives/negatives differently, but I'm not familiar with any that solve this specific problem of optimizing for a localized accuracy.

Specifically, I'm trying to answer two questions:
1) Are there specific classification algorithms well suited for this type of problem?
2) Can I modify the loss function used on my training set to produce a high localized accuracy within these specific 100 boxes?

I want to clarify that the fact that you can pick only 100 boxes from the test dataset is NOT a limitation (like you describe it). You just need to see it from this perspective:

  How to pick the 100 boxes (out of 500K) that MOST LIKELY contain gold?

A very suitable algorithm would be a Decision Tree classifier. By using the Information Gain (based on Entropy Reduction), it can learn which features can split the dataset more accurately into two classes, i.e. the one that contains gold and the one that doesn't, and neglect (prune) the weakest attributes. After that, you will not only have a trained classifier but also gain insight on which features are the most important to make the decision (very important information).