Processing data in the right manner in data science From what i have learned, people say that it is more correct if i preprocess data after splitting it to train/test dataset. My questions are

1.Does it mean we detect flaws of the data + preprocess it after the splitting? if yes/no, why?

2.Is it okay if i detect flaws of the data before splitting it and then preprocess the flaws after splitting it? if it is/ no it is not, why?

The goals of all these methodological guidelines is to avoid data leakage.

Example: let's imagine we want to classify short messages (e.g. tweets). When inspecting the data we find various kinds of smileys: :-), :|, :-/... At preprocessing stage we replace all smileys found in the data with a special token like <smiley> (or something more specific).

If the detection/replacement is done on the whole data, every occurrence of a smiley in the test set is replaced with <smiley>.
If the detection/replacement is done on the training set, even after preprocessing there might be a few smileys left in the test, because some uncommon ones didn't appear in the training set.

In the first case there is data leakage: we fixed some issues in the test set even though this wouldn't have been possible with actual fresh data (here the variants of smiley that were not seen in the training set). In the second case the test set is "imperfect", i.e. it's exactly as if it was made of "fresh" unseen data, therefore the evaluation will be more realistic.

This example shows why it's always safer to separate the data first, design the preprocessing steps on the training data, then apply exactly the exact same preprocessing steps to the test data.

In practice there can be cases where it's more convenient to apply some general preprocessing to the whole data. The decision depends on the task and the data: sometimes the risk of data leakage is so small that it can be neglected. However it's crucial to keep in mind that even the design of the preprocessing can be a source of data leakage.


Get a prediction from the new data inputted against the model, but an error is produced, how to adapt the R code for it to work? In the R code below, I included the sentences when looking to compare the manually classified with lexicon dictionary results by positive, negative and neutral (in matrixdata1), the algorithms results for the model produces different outcome in the tables, which is good. However, when executing..

results2 = classify_models(container2, models)

..when feeding in new data (matrixdata2) against the model it produces an error message:

Error in predict.svm(model, container@classification_matrix, prob = TRUE,  :
  test data does not match model !

In checking the datasets, I understand the train set's sentences used to create the model contains specific words, but the new data fed against the model include new words not recognised in the train set. I did create a few sentences that contained only words that appeared in the creation of the model. I fed in this few sentences with its labelled sentiment (new data) in against the model, but it still produced the same error message above. I do not understand why this is the case as these words are recognised in the trainset. However, when I used one of the same sentences in the new data to feed in against the model, it worked, so from what I can tell is if the sentence does not exactly match whats in the trainset, then it produces the error. I am still unsure how to adapt the R code to rectify the issue.

Please can you help me adapt the the R code below to overcome the error?

#Load Libraries
library(RTextTools) #RTextTools available for 3.4.1
library(e1071)
library(gmodels)

setwd(directory/path)
text= read.csv("matrixdata1.csv", header = FALSE)

# build dtm
matrix= create_matrix(text[,1:2])
mat = as.matrix(matrix)

# build the data to specify response variable, training set, testing set.
container = create_container(mat, as.numeric(as.factor(text[,3])),
                             trainSize=1:672, testSize=673:840,virgin=FALSE)

models = train_models(container, algorithms=c("MAXENT" , "SVM", "RF", "BAGGING", "TREE"), set_heldout = 168)

#container1
results1 = classify_models(container, models)

text2 = read.csv("matrixdata2.csv", header = FALSE)
matrix2= create_matrix(text2[,1:2])
mat2 = as.matrix(matrix2)

container2 = create_container(mat2, labels=NULL, trainSize=1:500,testSize=NULL, virgin=TRUE)

#Results from feeding in new data against the model
#When running this code below, it produces the error message outlined above in the description of the problem.
results2 = classify_models(container2, models)

matrixdata1.csv on GitHub

matrixdata2.csv on GitHub

This is a quite common mistake, you transformed the test and training data separately which messes with factor levels.

There are multiple solutions to this:

1) Create a common transformer

If you there is no "new" data but simply a test set (e.g. like in kaggle competitions or similar problems) you should create a function that transforms / tidies ALL data at once and only split the data after it already has the proper form for the modelling (including OHE, etc.).

2) Save your factor levels!

For all other case you should save your factor levels and create a transformer accordingly. Here is a code example:

text_levels <- levels(train$text)

levels(test$text) <- text_levels

Realize also, that any word or factor level not present in the data you train your model on cannot be considered in the model and therefore is useless! All above solutions simply add the appropriate factor levels from the training model if they are missing in the test data.

If you have new words in the test data you will simply need to drop them before transforming the data as they are not used in the prediction model.


What I need to write in the line level of torch.vison for 21 classes? In this code I found , line labels = torch.ones((records.shape[0],), dtype=torch.int64) ,that there is only one class and 0 in the case of  Faster RCNN is reserved for the Background. What would be for 21 classes?
def __getitem__(self, index: int):

file_name = self.file_names[index]
records = self.data[self.data['file_name'] == file_name]

image = np.array(Image.open(file_name), dtype=np.float32)
image /= 255.0

if self.transform:
    image = self.transform(image)

if self.mode != "test":
    boxes = records[['xmin', 'ymin', 'xmax', 'ymax']].values

    area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
    area = torch.as_tensor(area, dtype=torch.float32)

    labels = torch.ones((records.shape[0],), dtype=torch.int64)

    iscrowd = torch.zeros((records.shape[0],), dtype=torch.int64)

    target = {}

    target['boxes'] = boxes
    target['labels'] = labels
    target['image_id'] = torch.tensor([index])
    target['area'] = area
    target['iscrowd'] = iscrowd
    target['boxes'] = torch.stack(list((map(torch.tensor, target['boxes'])))).type(torch.float32)

    return image, target, file_name
else:
    return image, file_name

I solved my problem. Here are the steps I changed :

add a 'class_name' column to the "marking" DataFrame
replace 'source' with 'class_name' in data splitting (df_folds…)
modify load_image_and_boxes() function to also return labels modify load_cutmix_image_and_boxes() function to also return labels (most work to do here)
replace labels = torch.zeros((boxes.shape[0],), dtype=torch.int64) with torch.tensor(labels, dtype=torch.int64)
in get_net() function, set config.num_classes = number of classes in your dataset


What are similarity and distance metrics in classification? I have an assignment to train a model to classify text data, the brief for the assignment mentions that any for any learning model used I have to provide a reasoning for the similarity or distance metric used. What does this refer to?

My initial thought was that, for example, in logistic regression, the normalisation used would be L1 (Minkowski) or L2 (Euclidean). Would this be correct?

The choice of the metric is somewhat dependent on how you are representing the text.

A common metric for vector space models would be cosine similarity.

Cosine similarity: K(X, Y) = < X, Y > / (||X||*||Y||)

  Blockquote My initial thought was that, for example, in logistic regression, the normalisation used would be L1 (Manhattan) or L2 (Euclidean). Would this be correct?

In vector space models for text representation, the vectors are from a high dimension and sparse. When using L2 or L1 norm, there will be a term for every dimension which has a term in either vector. However, for consine similarity, since it is a dot product, there will only be a term when both elements in either vector is non-zero. The sparsity renders the consine similarity a better choice for such comparisons.


SMOTE and standardisation I have an unbalanced dataset X.
I split it between data and labels, then I standardize the data.
Then I use train_test_split to split between train and test data and I output the result.

Now I want to compare to see what I would get if I used smote to upsample the minority class, but I want to keep the comparison meaningful. To do that, I keep the same test data as before, and I add the new synthetic samples only to the train data. How should I handle the standardization?

Should I

Simply assume that since I used points from the already standardised data that the new synthetic samples will also be standardised? Therefore just add the synthetic samples to the train data and be done with it? (Without touching the test data)
Create the synthetic samples from the pre-standardised data, standardise the synthetic samples and add them to the train data? (Without touching the test data)
Create the synthetic samples from the pre-standardised data, add them to the train data, and standardise the whole set? (Without touching the test data)
Or ... ?

I am getting very different results doing these three techniques, what is the best way to get a meaningful result and comparison?

Solution 1 and 3 look really similar but I would go for 1 when thinking about the programming paradigm.

Seeing the classification as a big pipeline using standardization, balancing, and classification, one would not like to re-standardize something in the balancing. Furthermore, the assumption made in 1 is correct.

In a practical way, you could use the pipeline offer by the imbalanced API to use SMOTE with a scikit-learn normalizer


mean average precision - pseudo code There are many blog posts and threads concerning the computation of mAP, both for PASCAL and for COCO. Still, they do not answer some fundamental questions.

I am listing here the flow as I understand it, and will be happy for people's answers and fixes:

Flow:

You run your detection and recognition model through all your testing images, for which you already have ground-truth annotations. You apply no threshold.  Question: Should each detection be duplicated |classes| times and saved for each class with its corresponding confidence? For example, a detection that is 60% dog, 40% cat and 0% for all the rest, should be saved once as dog with 0.6, once as cat with 0.4 and once for each other class with 0?
For each class, the ground-truth boxes of this class are matched with the detected boxes that intersect with them with $IoU > iou\_threshold$, where $iou\_theshold \in [0,1]$ is a predefined parameter, and which share the same class. The matches will be used in order to compute precision and recall for the class. Question: Is this process performed image-wise or over all the images at once? i.e., are precisions and recalls computed per image and then averaged, or are they computed over the total detections in all the images in one step?
The detections are sorted according to their confidence. The threshold starts from 0 and is each time raised to exclude the next detection(s) with the next lowest confidence. The remaining detections already have correct/incorrect fields, from the previous step, which are now used to compute precision and recall with the current confidence threshold (the "next threshold" is the confidence of the detection(s) that we just excluded)
The precision values are computed in certain predefined recall points, as the highest precision in all the points with recall > recall_point. If there are no such points, then we use 0 as precision. The class' average precision (AP) is then computed as simply the mean of all the precisions we found for the set of certain recalls.
mAP is computed as the mean of the classes' APs
In COCO, this whole process will be repeated with a certain set of IoU thresholds and will be averaged again.

Firstly for your question 2. The precision and recall is calculated across all images, for a single class. Not on a per-image basis and averaging across images. The "A" in "mAP" means averaging across different recall levels, as you stated in step 4.

For your question 1. Whether to report a same bounding box to both the cat and the dog class is a decision that your detection algorithm should make, not a decision that the evaluation function should make. You can imagine that input to the evaluation script to be a ranked list for each class, for example:

class [dog]                        |  class[cat]                        |  class[foo] ...
image=17, score=0.98, bbox=[...]   |  image=41, score=0.75, bbox=[...]  |  ...
image=33, score=0.60, bbox=[...]*  |  image=07, score=0.45, bbox=[...]  |  ...
image=93, score=0.55, bbox=[...]   |  image=33, score=0.40, bbox=[...]* |  ...
...                                |  ...                               |...

The evaluation script will calculate the AP for each class and it won't care whether the same bounding box (*) appears in different classes.

So for your detection algorithm, you can try different strategies and see which one achieves higher mAP.


Classification of non-stationary acoustic signals I'm currently working on a task of classification of short non-stationary audio signals with length of 1024 elements and sampled at 120 kHz.

I was wondering if there are any special techniques or models for classification of such signals. As far as I know, most of solution for this kind of tasks relies on computation of FFT/MFCC or  spectrograms. But due to  non-stationary nature of features, I can't compute statisticaly reliable frequency based features.

Also approach with CNN as described in the article Raw Waveform-based Audio Classification Using Sample-level CNN Architectures shows promising results, but still doesn't work good enough.

Can anyone advice me another approaches or thoughts where to start from? Will appreciate any help!

As you try to classify sequential data, you can try simple recurrent neural network or their advance version LSTMs - but as you have a short signal, RNN should work fine.

You can read this paper on sound classification with LSTM and this  medium article


Data scheduling for recommender I do at the moment some data experiments with the Graphlab toolkit. I have at the first next SFrame, with the three columns:

Users Items Rating

The pair in the same row from every Users and Items values build the unique key and the Rating is the corresponded float value. These values are not normalised. First of all, I do someself next normalisation:

Division of every rating value of specific user by the rating maximum from this user (scale between 0 and 1)
Take the logarithm by every rating value

Afterward I create a recommender model and evaluate the basic metrics for it.

In this topic I invite everybody to discuss another interesting normalisation methods. If anybody could tell some good method for data preparation, it would be great. The results could be evaluated because of the metrics and I can publish it here.

PS

My dataset is comming from some music site, the users rated some tracks. I have approximately 100 000 users and 300 000 tracks. Total number of ratings is over 3 millions (actually the matrix is sparse). This is the most simple data set, which I analyze now. In the future I can (and will) use some additional information about the users and tracks (f.e. duration, year, genre, band etc). At the moment I just interest to collect some methods for rating normalisation without to use additional information (users & items features). My problem is, the data set doesn't have any Rating at the first. I create someself the column Rating, based on the number of events for unique User-Item pair (I have this information). You can of course understand that some users can hear some tracks many times, and another users only one time. Consequently the dispersion is very high and I want to reduce it (normalise the ratings value).

Another good normalization is zScore normalization. Was already implemented in python in the module scipy.stats


PCA or cluster table of experimental fitness scores I need to find patterns experimental data.

The columns are "experiments" which are chemical treatments for growth experiments.  The rows are individual gene names, the values are a fitness-defect score, which reflect the genes contribution to growth.

I would like to find patterns that are reflected across all experiments using some type of PCA or clustering.  I have been trying to use sklearn but have not been successful in applying a model.

The data looks like:

gene     SGTC_1                   SGTC_2                 SGTC_3
YAL002W  3.56420220283773        1.80774301690328       0.431491057210906
YAL004W -0.885645399324204      -1.76020417788351       0.883034190306176

....

There are 4000 rows for genes and 30 columns for experiments.

Any suggestions would be greatly appreciated.

PCA is a dimensionality reduction algorithm - it projects your high dimensional data onto a lower dimensional plane. This is useful for either visualisation (if you reduce to 2 or 3 dimensions to plot), or for training machine learning models.

You say you want to find patterns in your data - I’m not quite sure what you mean by this. Do you want to visualise your data, train a model on it and make some prediction, or something else?

To visualise high dimensional data, you could use either the tSNE (t-stochastic neighbour embedding) algorithm, or PCA.

Depending on the type of data you have, you can “find patterns” in different ways.

If your data is unlabelled (you don’t know the classes of each sample or there is no dependent variable) you can use unsupervised learning algorithms such as K-means clustering, K nearest neighbours, Gaussian mixture model. If your data has dependent variables, depending on whether your dependent variable is categorical or continuous, you could use classification algorithms for the former and regression algorithms for the latter. Classification algorithms include logistic regression or decision trees. Regression models include linear regression.


How can I get the ImageNet ILSVRC 2012 data used for the classification challenge? I would like to see if I can reproduce some of the image net results. However, I could not find the data (the list of URLs) used for training / testing in the ILSVRC 2012 (or later) classification challenges. I only found http://www.image-net.org/download-imageurls.

Where is the data used for the ImageNet ILSVRC 2012 (or later) classification challenge?

You can use the site academic torrents. The imagenet data you are looking for is shared there.


Causes of inconsistent results with neural network? I've created my own neural network in python. I am tracking the performance of each run. What are some things I can investigate when my performance results are inconsistent? When I compare my network results against my expect results (during cross-validation) I can get everything from 0.46 to 0.51 all the way up to 0.56 and various numbers in between.

Is this a sign of bad data? Not enough training epochs? Need for a different learning rate?

Before briefing about the other things, I would like to suggest you something good. It is always better to set a seed number in machine learning and deep learning problems to get a reproducible result and by that I mean that every time when you try to run your model anywhere, it can produce the same best result which you got on your machine while experimenting. So, you should always set numpy.random.seed(*anynumber*).

Now how to compare your result to your expected result? There are a number of parameters that you can look upon during training your model. Some of these are:

Validation accuracy: It is the one of the most important parameter to look upon for evaluating your model's performance. This gives you a fair idea if your model is overfitting or not. More the validation accuracy, the better is the model.
Validation loss: If your model is really good, then validation loss must decrease, at least initially until your best epoch hasn't been obtained. Lower the validation loss, better the model.
Optimizers: There are a number of optimizers like 'rmsprop', 'sgd', etc. which can be used for different types of problems but not all give the same performance. You should grid-search for a number of optimizers for your model if you are not specific about which is best for your model.
Number of training epochs: Depending upon the size of dataset, no. of training epochs differ but if you are not giving enough training to your model, then your model performance will suffer.
Training data: If you think your model is fine, but you are not achieving expected results, then you can try to train your model on a large dataset. NN require large dataset for achieving awesome performance.


Explained definition of the norm in Ordinary Least Squares I have recently started learning Scikit-learn and I am not able to understand the below equation. Could anybody please explain?

The norm indicates the Eucleadan norm, which gives the ordinary distance between the points.
https://en.wikipedia.org/wiki/Norm_(mathematics)

The subscript 2 with norm sign indicates "square root of the sum of squares".
The superscript 2 indicates sign power.


How do you classify groups when the inputs are paired? My data is like this: there is a choice between two alternatives and the customer chooses one. Each time the alternatives are different. I would like to characterize a desirable alternative. I don't want to use a normal classification algorithm because I don't want to lose the information that one alternative was chosen over the other in the pair. Does anyone have suggestions for ways to do this, or types of algorithms for solving this problem? Thanks!

That is called two-alternative forced choice (2AFC) task, a common technique in Psychology research. The Diffusion Decision Model (DDM) is used to analyze the data.


Choosing k value in KNN classifier? I'm working on classification problem and decided to use KNN classifier for the problem.

so if k=131 gave me auc of 0.689 and k=71 gave me auc of 0.682 what should be my ideal k?

Does choosing higher k means more usage of computational resource? if that's the case can I go with k=71. (or) should I always use K with maximum score no matter what?

I was taught the best way is to find the error for each k then plot them and look for the "elbow" on the plot.


Deep neural network models merging Recently I am working on the neural network deep learning algorithms, just curious to ask is it possible to merge two neural network models and to output one model that contains all the learned knowledge from the two models?
For example:
Model A can work on feature [A, B]
Model B can work on feature [C, D]
(Model type is the same for model A and model B if this makes things easier.)
After model merging, we get model C which is capable of working on feature [A, B, C, D].
I have looked into the transfer learning and Siamese network, but to my knowledge, I don't think these techniques can help me achieve my goal? (please correct me if I am wrong).
So any ideas will be much appreciated for merging the two models, or any possible technique, terminologies are welcomed.

The concept of Ensemble Learning is very applicable to your scenario: given two machine learning models that can predict well (or even just better than a random choice), combine the outputs of both models (with another ML model) to produce a unified model (let's call it Model C) that is more predictive than the individual models alone.
A straightforward application to your scenario could involve using the outputs of Models A and B along with some additional features from the original feature set as inputs to Model C. This larger set of features would allow for model C to learn for example when to the signal from Model A is more informative than B based on the current context provided by additional features from the original set.
It may also be possible to do this within a deep learning framework by adding both models into the same graph and adding a layer that ensembles the signals from the two models for an output prediction.


Launch Orange3 core dump for any >3.17 Updated to 3.19 using Anaconda navigator, from CLI I receive these messages

orange-canvas
realloc(): invalid old size
Aborted (core dumped)

python3 -m Orange.canvas
Segmentation fault (core dumped)

Using Anaconda navigator got this log

/home/davide445/.anaconda/navigator/scripts/orange3.sh: line 3:  8261 Aborted                 (core dumped) orange-canvas > /home/davide445/.anaconda/navigator/scripts/orange3-out-1.txt 2> /home/davide445/.anaconda/navigator/scripts/orange3-err-1.txt

Same problem with 3.20.1 or 3.18, is working using 3.17. Working on Kubuntu 18.04 LTS.
Any suggestion for troubleshooting, many thanks.

I have the same problem as you, and I just used pip to reinstall the 3.17 version.

pip install Orange3==3.17


K-means sensitivity to outliers? I'm studying K-means, and one important drawback of K-means is the lack of robustness to outliers. My question is: are there any cases when the lack of robustness to outliers may be considered not as a defect of K-means but as a virtue instead?

yes, in detecting abnormal cases.

In the k-means based outlier detection technique, the data are partitioned into k groups by assigning them to the closest cluster centers.

Once assigned we can compute the distance or dissimilarity between each object and its cluster center, and pick those with largest distances as outliers.

k-means clustering for Outlier detection


How much more data does my classifier need? I have a dataset of ~100k news articles, and I'm trying to build a classifier based on the headlines and leads of every article. The dataset is not pre-labeled so I'm manually labelling a subset of the articles to train on. So far, I have 9 categories with ~600 labeled articles for each category. I'm using doc2vec to create document vectors and the SVC classifier from sklearn to make predictions.

My cross validation score (with 10 splits and shuffling of data) is hovering around 0.89.

When I plot a learning curve for the training set and test set, I interpret it as the classifier suffering from high variance, and that I need to collect more data. But is there any way I can approximate how much more data is needed to get a cross validation score of say, 0.95?

Here is my learning curve and the mean scores:

Training scores: 1.0, 0.99919679, 0.9991984 , 0.99791667, 0.99615385, 0.99550155, 0.99487179.

Testing scores: 0.40769912, 0.72283119, 0.78529511, 0.83461723, 0.85527486, 0.86151579, 0.86471912.

EDIT: I performed a grid search on the C and gamma parameters to the SVC model, and adjusted gamma to 0.015 which made the two lines converge a bit more. Adding plot and new scores.

New training scores: 0.97758621, 0.98703072, 0.98177172, 0.97540872, 0.96717739, 0.96479244, 0.96241702.

New testing scores: 0.45800254, 0.78917394, 0.83526468, 0.853326, 0.86849859, 0.87232875, 0.87508134.

I think you already have enough data. Your problem seems to be one of generalization, which is in general a tricky one to solve.
For SVM, depending on what kind you have, you could try tinkering with the parameters and kernels.
Have a look here. -- https://stats.stackexchange.com/questions/35276/svm-overfitting-curse-of-dimensionality?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa


XGBoost Linear Regression output incorrect I am a newbie to XGBoost so pardon my ignorance. Here is the python code :

import pandas as pd
import xgboost as xgb

df = pd.DataFrame({'x':[1,2,3], 'y':[10,20,30]})
X_train = df.drop('y',axis=1)
Y_train = df['y']
T_train_xgb = xgb.DMatrix(X_train, Y_train)

params = {"objective": "reg:linear"}
gbm = xgb.train(dtrain=T_train_xgb,params=params)
Y_pred = gbm.predict(xgb.DMatrix(pd.DataFrame({'x':[4,5]})))
print Y_pred

Output is :

[ 24.126194  24.126194]

As you can see the input data is simply a straight line. So the output I expect is [40,50].  What am I doing wrong here?

It seems that XGBoost uses regression trees as base learners by default. XGBoost (or Gradient boosting in general) work by combining multiple of these base learners. Regression trees can not extrapolate the patterns in the training data, so any input above 3 or below 1 will not be predicted correctly in your case. Your model is trained to predict outputs for inputs in the interval [1,3], an input higher than 3 will be given the same output as 3, and an input less than 1 will be given the same output as 1.

Additionally, regression trees do not really see your data as a straight line as they are non-parametric models, which means they can theoretically fit any shape that is more complicated than a straight line. Roughly, a regression tree works by assigning your new input data to some of the training data points it have seen during training, and produce the output based on that.

This is in contrast to parametric regressors (like linear regression) which actually look for the best parameters of a hyperplane (straight line in your case) to fit your data. Linear regression does see your data as a straight line with a slope and an intercept.

You can change the base learner of your XGBoost model to a GLM (generalized linear model) by adding "booster":"gblinear" to your model params :

import pandas as pd
import xgboost as xgb

df = pd.DataFrame({'x':[1,2,3], 'y':[10,20,30]})
X_train = df.drop('y',axis=1)
Y_train = df['y']
T_train_xgb = xgb.DMatrix(X_train, Y_train)

params = {"objective": "reg:linear", "booster":"gblinear"}
gbm = xgb.train(dtrain=T_train_xgb,params=params)
Y_pred = gbm.predict(xgb.DMatrix(pd.DataFrame({'x':[4,5]})))
print Y_pred

In general, to debug why your XGBoost model is behaving in a particular way, see the model parameters :

gbm.get_dump()

If your base learner is linear model, the get_dump output is :

['bias:\n4.49469\nweight:\n7.85942\n']

In your code above, since you tree base learners, the output will be :

['0:[x<3] yes=1,no=2,missing=1\n\t1:[x<2] yes=3,no=4,missing=3\n\t\t3:leaf=2.85\n\t\t4:leaf=5.85\n\t2:leaf=8.85\n',
 '0:[x<3] yes=1,no=2,missing=1\n\t1:[x<2] yes=3,no=4,missing=3\n\t\t3:leaf=1.995\n\t\t4:leaf=4.095\n\t2:leaf=6.195\n',
 '0:[x<3] yes=1,no=2,missing=1\n\t1:[x<2] yes=3,no=4,missing=3\n\t\t3:leaf=1.3965\n\t\t4:leaf=2.8665\n\t2:leaf=4.3365\n',
 '0:[x<3] yes=1,no=2,missing=1\n\t1:[x<2] yes=3,no=4,missing=3\n\t\t3:leaf=0.97755\n\t\t4:leaf=2.00655\n\t2:leaf=3.03555\n',
 '0:[x<3] yes=1,no=2,missing=1\n\t1:[x<2] yes=3,no=4,missing=3\n\t\t3:leaf=0.684285\n\t\t4:leaf=1.40458\n\t2:leaf=2.12489\n',
 '0:[x<3] yes=1,no=2,missing=1\n\t1:[x<2] yes=3,no=4,missing=3\n\t\t3:leaf=0.478999\n\t\t4:leaf=0.983209\n\t2:leaf=1.48742\n',
 '0:[x<3] yes=1,no=2,missing=1\n\t1:[x<2] yes=3,no=4,missing=3\n\t\t3:leaf=0.3353\n\t\t4:leaf=0.688247\n\t2:leaf=1.04119\n',
 '0:[x<3] yes=1,no=2,missing=1\n\t1:[x<2] yes=3,no=4,missing=3\n\t\t3:leaf=0.23471\n\t\t4:leaf=0.481773\n\t2:leaf=0.728836\n',
 '0:[x<3] yes=1,no=2,missing=1\n\t1:[x<2] yes=3,no=4,missing=3\n\t\t3:leaf=0.164297\n\t\t4:leaf=0.337241\n\t2:leaf=0.510185\n',
 '0:[x<2] yes=1,no=2,missing=1\n\t1:leaf=0.115008\n\t2:[x<3] yes=3,no=4,missing=3\n\t\t3:leaf=0.236069\n\t\t4:leaf=0.357129\n']

Tip : I actually prefer to use xgb.XGBRegressor or xgb.XGBClassifier classes, since they follow the sci-kit learn API. And because sci-kit learn has so many machine learning algorithm implementations, using XGB as an additional library does not disturb my workflow only when I use the sci-kit interface of XGBoost.


Implementing single variable Linear Regression in python I'm trying to implement LMS algorithm in python. I have the following code:

import os
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

def compute_cost(theta, X , y):
    inner = np.power(X.dot(theta) - y, 2)
    return np.sum(inner/2*len(X))

if __name__ == '__main__':
    path = os.getcwd() + '/data/ex1data1.txt'
    data = pd.read_csv(path, header=None, names=['Population', 'Profit'])
    data.plot(x='Population', y='Profit', kind='scatter', figsize=(10, 10))
    plt.show()
    # append a ones column to the front of the data set
    data.insert(0, 'Ones', 1)

    # set X (training data) and y (target variable)
    cols = data.shape[1]
    X = data.iloc[:, 0:cols - 1]
    y = data.iloc[:, cols - 1:cols]
    theta = np.array([0,0])
    print(X.shape, theta.shape, y.shape)
    print(compute_cost(theta, X, y))

The first print statement (printing X.shape, theta.shape, y.shape) prints the following:

(97, 2) (2,) (97, 1)

When I try to compute the cost function I get the following:

Profit    0.0
0         0.0
1         0.0
2         0.0
3         0.0
         ...
92        0.0
93        0.0
94        0.0
95        0.0
96        0.0
Length: 98, dtype: float64

However, I'm supposed to (according to the exercise) get 32.07
I think that the bug is related to the theta shape but I tried initializing theta like this:

theta = np.zeros(shape=(2, 1))

And still this doesn't work..
To be clear, I'm not looking for help writing the algorithm simply find the syntax bug, understand why it happened and fix it.

The Equation for Least Square method shall be as below-

theta(0)+theta(1).X , since you have 1 variable.

if theta(0) =0 and theta(1)=0 since you are adding it theta = np.zeros(shape=(2, 1)).

the value of Y shall be 0 hence error is 0.

To breakdown nicely you can add it like-

n = X.shape[1]
theta = np.zeros((1, n))

Now in the next run the values of theta shall change as per learning rates and in then the errors shall change too, that part you haven't reached yet, by looking at your code if I might add.

Here have a look-

https://towardsdatascience.com/linear-regression-using-python-b136c91bf0a2


What does normalizing and mean centering data do? Are there any concerns to normalizing data to be within the range 0 - 1 and mean centering the data as well?

Does it matter which comes first?

If you do one, is the other not required?

If you don't center before you normalize, you don't take advantage of the full [-1,1] range if your input is non-negative. The combination of centering and normalization is called standardization.

Sometimes one normalizes by the standard variation, and other times by just the range (max-min). The latter is called feature scaling. The effect is much the same. Normalizing by the range is easier computationally. Normalizing by the standard deviation fixes the sample variance, which is nice from a statistical perspective. When using the standard deviation, the subtraction is usually against the sample mean rather than the minimum.

There are several reasons for performing standardization. Sometimes we are interested in relative rather than absolute values. Standardization achieves invariance to these irrelevant differences. By explicitly preprocessing the data to reflect this disinterest, we relieve the model from having to learn it, allowing us to use a simpler one. Another reason is computational; it reduces the condition number -- you can think of this as the skewness or niceness of the loss surface -- making optimization easier and faster.


How do data types influence hardware (CPU / GPU / TPU) performance? I am currently dealing with a relatively big data set, for which I have some memory usage concerns. I am dealing with most of the different data types : floats, integers, Booleans, characters strings for which some were changed to factors.
From a memory usage (RAM) standpoint it is pretty clear what is happening when I switch column type from float64 to float32: the memory usage is divided by two. For string conversion to factor, it is a bit less clear but I get the general idea that regarding to memory theiy are delt with as small integer and remapped if necessary.  It doesn't really matter, as they generally delt with in pre-processing by being encoded as numerical columns some way or another.
Now what I am wondering is how do that data types influence calculations further down the pipeline (pre-processing, calibration). I somehow got the idea that changing data types from float 64 to float 32 would allow for twice faster calculations on CPU, but this seems to be varying wildly. So, I am wondering, for what that matters (pre-processing, calibration), how does data types impact the hardware performance ?
Some clarifications / simplyfiying assumptions :

I have the intuition that some weirds things may happen in the CPU if you mix data types or regarding convergence if you go too low on memory usage (using float16). So let's assume we are just dealing with float64 to float32 conversion.

I am trying to get a langage / library / model agnostic answer. I am mostly using Python + pandas if that matter (to be honest I've never really had to handle data types in R), Python + cudf for GPU. If that matter I am using tensorflow + keras to build a NN. Notably I've called tf.keras.backend.set_floatx('float32'). Similarly I've carrefully made sure that the pre-processing pipeline is handling float32 data types correctly and that code is vectorized.

As I am getting involved in Kaggle competitions with differents accelerator (GPU / TPU), I'd like to get an answer that cover the main kind of hardware (CPU / GPU / TPU). I hope that the answer don't really depends on brands.

The variation in performance gains you are seeing from reduction precision might be do to different frameworks using different types. Even after downcasting data types, some operations will automatically upcast types. You mention using Pandas and TensorFlow / Keras. Mixing these frameworks leads to unwanted recasting of data types. It is better to use a single framework to avoid recasting.
There is no software (i.e., language, library, or model) agnostic answer. Hardware utilization is software specific. In order to get the reduced precision gains, the same software stack should be throughout the entire modeling process. That software should be designed for the specific chip (i.e., CPU, GPU, or TPU).


Technique to determine variation in metric due to varying parameters So basically I have a large set of features corresponding to a metric - like many ML problems.

What I want to know is: can we correlate the variance of metric with the variation in each feature.

ex:

I have features x, y, z that produce an output say 10. When I vary x, no matter how much I vary it, the output stays relatively close to 10. However, when I vary y the output is heavily influenced.

Is there a good technique to be able to assign a value correlating x and/or y to the metric?

I'm mostly looking for direction here.. i.e. techniques or relevant papers. In my experience I haven't really come across this problem. I don't have a good solution in my toolbelt.

Thanks!

If the variation in values of feature then should not you remove the column? Your model will not learn from it if the variance is low.

you can Q-Q plots when you vary the features to check how close are the 2 datasets comparing their distribution


System Requirements to train a Image Recognition Neural Network I have 6000 Images to be trained on a Neural Network.

My current PC Specs :-

  32GB RAM, i5 2 core Processor, Standard GPU (No work going on GPU),
  1TB Hard Disk

My Neural Network Specs :-

  3000 Epoch Size, 6 Batch Size

Process is getting killed with these specifications.

  How much Hardware is required to trained it well? What would be the
  ideal Batch Size and Epoch Size?

This is not really a data science specific question, so you might want to ask elsewhere; in any case, you need to provide more information!

What error message are you receiving? Are you running out of memory? What size are the images?

Unless your images are really really large, I would not expect you to be running out of memory, given you have 32Gb. however, if you have built a massive network, where e.g. many ConvNet filters are then passed to an extremely wide FullyConnected layer (Dense layer), you will have a huge number of weights, which may result in a memory error.

Try creating your model (or a similar one) using Keras instead of pure Tensorflow, and then using model.summary on the compiled model. This will show you a nice overview of the number of weights in each layer.


How accurate does my machine learning model need to be? Is there any way to determine upfront how accurate a model should be before it's worth putting into production? Just aiming for the highest accuracy possible doesn't tell us anything about the money a company would save with a 75% versus 80% model.

Say we needed to predict the number of mechanical failures. If the national average is 100/year but our company is getting 125, how accurate does our model need to be (how many correct predictions does it need to make) to get our company back down the national average?

The accuracy level that should satisfy you depends on the reliability that your customer would have on it and the costs matrix of false predictions.

He should put into consideration the costs of the unpredicted failures (low recall of your model) causing him production delays and the costs of over/false maintenance alarms (low precision of your model). If you know these costs you can minimize the expected cost equation which considers the precision recall and accuracy of your model:

Expected cost
  = p(true positive) × "maintenance cost"
  + p(false positive) × "inspection cost"
  + p(true negative) × 0 (no action)
  + p(false negative) × "mechanical failure cost"


Python Pandas - Interpolation with fixed value Say we have a pandas series with the following values

[np.nan, np.nan, 1, np.nan, 2, np.nan]

What is the most efficient way fill the nan value with 0 in the middle. so we have

[np.nan, np.nan, 1, 0, 2, np.nan]

In other word, how to we do interpolation with a fixed value, or a .fillna operation but ignore the nan at the beginning and end of the array.

Current solution, I am using

def interpolate_with_fixed(s, value=0):
    i = s.first_valid_index()
    j = s.last_valid_index()
    s.loc[i:j].fillna(value, inplace=True)
    return s


Understanding Locally Weighted Linear Regression I'm having problem understanding how we choose the weight function. In Andrew Ng's notes, a method for calculating a local weight, a standard choice of weights is given by:
What I don't understand is, what exactly is the x here? Apparently

  Note that the weights depend on the particular point x at which we’re trying
  to evaluate x.

But I don't get it.

Take the example of housing prices predicted by number of rooms and size in square feet. So each x^(i) is a [roomnum, size] array. So what's in x? I guess that also should be a [roomnum, size] array, but what's in it? Is it even a vector? Or is it the target variable? If so, why isn't it marked with y? I don't get it, please help!

EDIT
Ok so what I want is to make create a regression line like this:

How would I choose the x-es? What would they be in an algorithm? Do I have to make guesses for each x? How can I produce a line like this?

Locally weighted linear regression is a non-parametric method for fitting data points. What does that mean?

Instead of fitting a single regression line, you fit many linear regression models. The final resulting smooth curve is the product of all those regression models.
Obviously, we can't fit the same linear model again and again. Instead, for each linear model we want to fit, we find a point x and use that for fitting a local regression model.
We find points that closest to x to fit each of our local regression model. That's why you'll see the algorithm is also known as nearest neighbours algorithm in the literature.

Now, if your data points have the x-values from 1 to 100: [1,2,3 ... 98, 99, 100]. The algorithm would fit a linear model for 1,2,3...,98,99,100. That means, you'll have 100 regression models.

Again, when we fit each of the model, we can't just use all the data points in the sample. For each of the model, we find the closest points and use that for fitting. For example, if the algorithm wants to fit for x=50, it will put higher weight on [48,49,50,51,52] and less weight on [45,46,47,53,54,55]. When it tries to fit for x=95, the points [92,93,95,96,97] will have higher weight than any other data points.

Do you see the pattern? The points closer the where you want to fit have higher weight and the points further have lower weight (zero if too far). That's what the weight function is for.

To answer your question directly:

  x are the data points for each local regression model. They are usually (but not always) the data points in your sample.


Clustering initialization I'm running into a problem while working on clustering. I work on data with white Gaussian noise. All of the methods I have come across use some sort of random initialization to set up the mean and covariance matrix of the clusters.

My question is: Since the initialization is random, there is a chance that I get a really bad starting point which gives me bad results. How do I deal with this?

One specific initialization I'm considering is the K-Means++ which is better than strictly random because it at least attempts to use the data to make informed initialization, but it too is random in the end.

Do people usually do multiple runs and take the best initialization?

What about that for streaming data?

You have two options:

1) Let the K-means algorithm run for a large number of iterations (if on sklearn, change the max_iter parameter value for sklearn.cluster.KMeans). It will eventually converge to a good result (but it will take more time)

2) Make and "educated guess" for the initial starting point. One way to do that is to transform your data in a space where you know your points can only lie within a specific region: from there, you can evaluate the best seeds where to start the K-means. For a clearer explanation, see this article (for a quick overview, look at the related slides, in particular at step 3, slide number 10)


How to quantitatively compare two or more complex data sets I have a series of datasets that are composed of 100 or so variables and a corresponding response variable. I am often faced with the question of trying to attribute differences in the response variable to the 100 or so variables.

My problem is that it is often difficult to compare two or more datasets and attribute the response variable to the 100 or so variables, due the large number of variables.

What do people do when faced with such a problem? What are some common analyses for doing this kind of work? It's preferable if the solution can be explained to people with a non-technical background.

One option is to calculate a distance metric between the data sets.
Choosing a distance metric depends on the properties of the data. If the data is binary, the Hamming distance can work. If the data are sets, the Jaccard distance can work. Other data typee require other distance metrics.


Similarity measure for ordered binary vectors I would like to ask your opinion on how to choose a similarity measure. I have a set of vectors of length N, each element of which can contain either 0 or 1. The vectors are actually ordered sequences, so the position of each element is important. Suppose I have three vectors of length 10, x_1 x2, x3: x1 has three 1 at positions 6,7,8 (indexes start from 1. Both x2 and x3 have an additional 1, but x2 has it in position 9 while x3 has it in position 1. I am looking for a metric according to which x1 is more similar to x2 than to x3, in that the additional 1 is closer to the "bulk" of ones. I guess this is a relatively common problem, but I am confused on the best way to approach it.
Many thanks in advance!

One thing you could do is fuzzify your vectors: replace each 1 by (for example) 0.4 in its position, 0.2 in the neighbouring positions, and 0.1 in the second position over. Then add up what's in each position.
With these fuzzified vectors, you can apply a similarity metric either based on a distance or one like cosines similarity.
Your example would produce: (showing only first decimal)

0000011100 -> 0001378731

0000011110 -> 0001378873

1000011100 -> 4211378731

cos(x1, x2) = 0.9613,  cos(x1,x3) = 0.9469


Unsupervised Learning - Using the Outcome of Learning My understanding in Unsupervised Learning is that -- when you want a computer to learn on its own by examining a large dataset. The goal is to establish some form of cluster or association-based grouping of data from the given input aka "Find patterns - known/unknown"
So my question is - "Now what do we do with it?" - e.g. I have now identified a pattern using Unsupervised Learning. Do I now look for Supervised Learning to empower my Data Insights work? Or, do I make decision as a human/team/committee to do certain things based on those?

Unsupervised Learning is a collection of tools and you can use those tools for many purposes.
Sometime, it's just for data visualisation / data exploration (see UMAP for example), to better understand the problem and get the idea how to tackle it. Sometimes it will lead to business decision like building processes for a specific cluster or building submodels for specific clusters.
Sometime it helps deal with sub-problems of data-science, like outliers identification (see Hdbscan or isolation forests for example). What to do with those outliers depends on your business case.
Sometimes it helps define intermediate 'targets' to perform semi-supervised or supervised learning, like using some self-organisaed map, identify a cluster and use that cluster as a target. It help when you are not sure you have identified all your positive targets or when you don't wnat to label everything.


In neural networks model, which number of hidden units to select? In the neural networks model, how many numbers of hidden units need to keep to get an optimal result, as per Cybenko theorem which demonstrates that only one hidden layer is sufficient to solve any regression/classification problem but the selection of the number of units in a hidden layer is very important because it impacts the model performance. Is there a theory to tell us how to choose the optimal number of units for a hidden layer?

Unfrtunately not, there is not theory to tell us what is the right number of units to choose. As there is not theory for the number of hidden layers to choose. On this respect, Deep Learning is still more an art than a science.

It's true that with one hidden layer we could theoretically solve any problem, but most of the problems are complicated enough to require unimaginable amounts computation.

I think in the end it all boils down to two main issues:

Your specific task, i.e. how large and deep a Network must be so that your model works as you need.
The compute power at your disposal.

When it comes to this, I strongly suggest to prioritize depth over width. Deeper Networks (the ones with many hidden layers) are much more efficient and powerful than large Networks (the ones with fewer, larger layers). It seems they are better at producing abstractions on the input data, to transform and process the signal in more sophisticated ways.


Keras: extreme spike in loss during training I am training an LSTM for time series forecasting and it has produced an extremly high loss value during one epoch:

Epoch 00043: saving model to /...
904/904 - 2s - loss: 0.7537 - mean_absolute_error: 0.5772 - val_loss: 1.4430
- val_mean_absolute_error: 0.7124

Epoch 00044: saving model to /...
904/904 - 2s - loss: 240372339275.7649 - mean_absolute_error: 56354.0078
- val_loss: 4.6229 - val_mean_absolute_error: 1.5681

Epoch 00045: saving model to /...
904/904 - 2s - loss: 1.3348 - mean_absolute_error: 0.7894 - val_loss: 2.2875
- val_mean_absolute_error: 1.1510

My model:

model = keras.Sequential()
model.add(keras.layers.LSTM(360, activation='relu', input_shape=(N_STEPS, n_features)))
model.add(keras.layers.Dropout(0.1))
model.add(keras.layers.Dense(1, activation='relu'))
model.compile(optimizer='adam', loss='mse', metrics=['mae'])

What is the cause of this?

Theoretically, it shouldn't be able to have such a high loss unless it outputs very high values for that epoch. Which is strange since the model's output makes sense during other epochs.

Sorry I couldn't comment as it requires 50 Reputation. On Epoch 44 there is a huge spike in the loss. It is entirely possible that the model may have come across new data and it may have learned a few tricks up its sleeve. Try to plot loss of train & validation vs epoch to see if it underfits or overfits.


Tools for automatic anomaly detection on a SQL table? I have a large SQL table that is essentially a log. The data is pretty complex and I'm trying to find some way to identify anomalies without me understanding all the data. I've found lots of tools for Anomaly Detection but most of them require a "middle-man" of sorts, ie Elastic Search, Splunk, etc.

Does anyone know of a tool that can run against a SQL table which builds a baseline and alerts of anomalies automagically?

This may sound lazy but I've spent dozens of hours writing individual reporting scripts as I learn what each event type means and which other fields go with each event and I don't feel any closer to being able to alert on real problems in a meaningful way. The table has 41 columns and just hit 500 million rows (3 years of data).

If you need SQL code that runs various outlier detection methods against any arbitrary table, check out my series of articles and code samples geared towards SQL Server. I provide some preliminary code for Grubb's Tests, Z-Scores and Modified Z-Scores, Interquartile Range, Dixon's Q-Test, GESD, the Tietjen-Moore Test, Pierce's Criterion, Chauvenet's Criterion, the Modified Thompson Tau Test, Cook's Distance, Mahalanobis Distance, Benford's Law and various visual plots. Please note that I'm an amateur at this and that this is some of my older code, so I'm sure corrections will be needed; this at least provides a starting point so that you can run these tests against any database table you have permissions for. It's also in T-SQL, so you'll need to adjust the code for PL/SQL or whatever other brand of SQL you might be using, if you happen to be on some other platform like Oracle or MySql. This should get you started though. Just works out the kinks and set the stored procedures to run periodically on a schedule and you're good to go. Let me know if you have any feedback on the routines, I'm always looking to improve my SQL writing.


How to draw Deep learning network architecture diagrams? I have built my model. Now I want to draw the network architecture diagram for my research paper. Example is shown below:

I recently found this online tool that produces publication-ready NN-architecture schematics. It is called NN-SVG and made by Alex Lenail.

You can easily export these to use in, say, LaTeX for example.

Here are a few examples:

AlexNet style

LeNet style

and the good old Fully Connected style


How to find whether a dataset is blanced or imbalanced? I have few dataset to experiment classification(Multi-class). These datasets are about 400GB. I wanted to know whether the dataset is balanced or imbalanced. How to know that dataset is balance or imbalanced using any scientific way?

Typically, the representation of each class in a multi-classification problem should be equal. Say if there are 4 classes, then the ratio of count of samples in each class should ideally be n:n:n:n, most classification data sets do not have exactly same number of sample count in each class, which is fine and a lit bit of difference often does not matter. But if the difference is huge, say for example 100:5:9:13 then it matters and it is an imbalanced dataset.

coming to 400 GB of data to read - Depending on the type of your file, you can read it in chunks and then read and save the target variable( the one which has multi class labels) in another variable.

You can visualize this variable (containing target) using a bar chart which will show you the count of variables for each class. Along with that you can also calculate the distribution of your classes to get better understanding of data.


CART vs Random Forest Considering that individual trees in random forests use the cart algorithm(or can be configured to) , if cart fails to predict anything(empty tree) ,why should random forest perform better on the same dataset.
Edit:
I am using R (rpart vs randomForest) ; the data is daily financial data of a stock. CART gave empty trees but random forest achieved decent accuracy even on unseen data.

The goal of having a "forest" (baggging ensemble) of trees is to make the prediction more solid. Individual decision tree tend to overfit, and with Random Forest the sampling, features selection and bagging helps to make a more robust score.
Its weird that you have an empty tree, since decision tree are greedy and they will fit to anything. Even if there is just random noise in your data they will make splits (depends a bit on how you have configurated it).
If you have an empty tree, random forest wont do nothing since is just a bunch of decision trees.
With out seeing your code, or nothing else I would guess that you have a coding error somewhere.


Valid actions in OpenAI Gym Why don't the gym environments come with "valid actions"? The normal gym environment accepts as input any action, even if it's not even possible.

Is this a normal thing in reinforcement learning? Do the models really have to learn what valid actions are all the time? Would it not be much nicer to have a env.get_valid_actions() functions so that the model knows what actions are doable? Or is this somehow possible and I'm missing it?

In general, if the agent is simply not able to take non-valid actions in a given environment (e.g. due to strict rules of a game, like chess), then it is standard practice to have the environment support that by providing some kind of function or filter for $\mathcal{A}(s)$, the set of actions available in state $s$.

However, it does seem like the basic Gym interface does not support it, and has no plans to support it.

It is still possible for you to write an environment that does provide this information within the Gym API using the env.step method, by returning it as part of the info dictionary:

next_state, reward, done, info = env.step(action)

The info return value can contain custom environment-specific data, so if you are writing an environment where the valid action set changes depending upon state, you can use this to communicate with your agent. The caveat is that both your environment and agent would be working to a convention you had invented in order to do this, and the same approach would not extend to other environments or be used by developers of other agents.

Alternatively, you can support calculating what the valid actions are in any other custom way that is not covered by the Gym API - a new method on the environment, a separate "rules engine" for a game etc. Both the agent and the environment could call that in order to perform their roles correctly.

The situation without an official API to do this may be subject to change, but from the linked Github issue it seems that Open AI developers consider a generic interface for this, that accounted for all the different kinds of action space, was more effort than it was worth.


Choosing the right model for predicting demand We have a data set of 300,000+ records that looks something like the following:

Item ID, Quantity 2017, Quantity 2016, Quantity 2015, Quantity 2014, Quantity 2013
1111, 100, 50, 25, 10, 0
2222, 0, 10, 100, 500, 1000
3333, 10, 0, 5, 2, 4

We are currently attempting to find the best model to predict Quantity 2017 based on the previous quantities for each record. We have tried Decision Tree Regression, Multiple Linear Regression, and Random Forest Regression (10, 100, 1000 trees) but our results were a bit too far off for approval. We are using 80% of our data for training and 20% for testing.

Are there any models that are better suited for this type of calculation?

We are also concerned that perhaps our data structure is part of the problem and we should re-evaluate. Is there another structure that might be better?

It looks to me that the data are in fact a Time Serie. Therefore, you can treat them as such.

Now, there are many approaches with each having each own strengths/weaknesses. I'd start with simple models such as ARMA or SARIMA. Possibly also GARCH or VAR. If you wish to go for something more advanced, Recurrent Neural Networks are also used in Time Series prediction.

Beside that, I am surprised that for example Random Forest have not given sufficient performance. Have you tried thorough feature selection? It may be that you have too high expectations.


Do I Need Pretrained Weights For Keras VGG16? As context, I am relatively new to the world of machine learning and I am attempting a project with a goal of classifying plays in an NBA game. My inputs are a sequence of 40 frames from each play in an NBA game and my labels are 11 all-encompassing classifications for a given play.

The plan is to take each sequence of frames and pass each frame into a CNN to extract a set of features. Then each sequence of features from a given video would be passed onto an RNN.

I am currently using Keras for most of my implementation and I chose to use a VGG16 model for my CNN. Here is some of the relevant code below:

video = keras.Input(shape = (None, 255, 255, 3), name = 'video')
cnn = keras.applications.VGG16(include_top=False, weights = None, input_shape=
(255,255,3), pooling = 'avg', classes=11)
cnn.trainable = True

My question is - would it still be beneficial for me to initialize the weights of the VGG16 ConvNet to 'imagenet' if my goal is to classify video clips of NBA games? If so, why? If not, how can I train the VGG16 ConvNet to get my own set of weights and then how can I insert them into this function? I have had little luck finding any tutorials where someone included their own set of weights when using the VGG16 model.

I apologize if my questions seem naive but I would really appreciate any help in clearing this up.

It would be beneficial to use pretrained weights trained on ImageNet because ImageNet is a huge dataset, and is fairly general. So you're more likely to get good image features from it. You might have to play around with from which layer you want to extract the features from.

If you want to create your own embeddings, you need to use the same code and instead of loading the weights, train it instead on the images you have. I do not recommend this for the following reasons

You most likely have too small a dataset for the model to converge
VGG16 does classification, what will be the labels for the individual frames is a design choice and being a novice, it might be frustrating for you to get it right
It takes a lot of time and resources, not an ideal situation

P.S, if you still feel that ImageNet does not sufficiently cover your use case, you can try the pretrained Inception V3 model trained on the Open Images dataset


Time series forecasting using multiple time series as training data I am trying to forecast the total attendance (ie. the number of entrances, which is also the number of tickets bought) of a festival just two days after it started. That is, knowing how many people went to the event during the first two days, how to predict the total number of people that will have visited the festival?

I do know it seems difficult at first glance, and that I theoretically could only make pretty bad forecasts, but here's the deal : I have organized more than thirty festivals in the past, and have collected data on each of them. Specifically, on each of these festivals, I have a daily time series where I know:

the number of tickets bought daily
whether the day was a weekday or during the week-end
whether the day was a day of public and school holidays
the daily weather

I have observed that all of these time series follow multiple trends. For instance, attendance always is at its best on saturdays, and at its worse on tuesdays... Likewise, there seems to always be more people coming in the very last days of the event than in beginning. These trends are the same for almost all the festivals. When decomposing the time series, I observe close trends, and close seasonal values.

Another thing, which I guess is no good news, is that the events had different timespans : some lasted 4 days, others 5, 6, 7 and even 8 days. Some started on a monday, others on a saturday.

So my questions is: how could I use these time series as a training data to try to forecast total attendance at the event knowing the attendance of the very first days. That is to say, which model could I use to predict total attendance of the event knowing I have all of this data ? I was of course thinking of machine-learning (or deep-learning) since I have a lot of training data, but I'm unsure whether it can be easily implemented in R or Python...

In order to do forecasts, I do of course know, for the on-going festival, how long it will last, whether it is going to take place during public and school holidays or not, whether it is going to span over a week-end, and I have the weather forecasts for each day.

First  cluster the events that have the most similarities. Then use a comparable (or more than one of  then ) to forecast the sales of the new events that you do not have historical data. Use all other information you have as regressor. Here is a code to do the forecast in R. You will be able to combine different forecasting models with this code:

choose_model<-function(x,h,reg,new_reg,end_train,start_test){
      library(forecast)
      library(forecastHybrid)
      library(tidyverse)

  #train data

  x_train <- window(x, end = end_train )

  x_test <- window(x, start = start_test)

  #train and test for regressors

  reg_train <- window(reg, end = end_train )

  reg_test <- window(reg, start = start_test)

  h1=length(x_test)

  #model1

  stlf(x_train , method="arima",s.window= nrow(x_train),xreg = reg_train, newxreg = reg_test, h=h1)-> fc_stlf_xreg

  #model2
  auto.arima(x_train, stepwise = FALSE, approximation = FALSE,xreg=reg_train)%>%forecast(h=h1,xreg=reg_test) -> fc_arima_xreg

  #model3
  set.seed(12345)#for nnetar model
  nnetar(x_train, MaxNWts=nrow(x), xreg=reg_train)%>%forecast(h=h1, xreg=reg_test) -> fc_nnetar_xreg

  #model4
  stlf(x_train , method= "ets",s.window= 12, h=h1)-> fc_stlf_ets

  #Combination

  mod1 <- lm(x_test ~ 0 + fc_stlf_xreg$mean + fc_arima_xreg$mean + fc_nnetar_xreg$mean + fc_stlf_ets$mean)
  mod2 <- lm(x_test/I(sum(coef(mod1))) ~ 0 + fc_stlf_xreg$mean + fc_arima_xreg$mean + fc_nnetar_xreg$mean + fc_stlf_ets$mean)

  #model1

  stlf(x, method="arima",s.window= 12,xreg=reg, newxreg=new_reg, h=h)-> fc_stlf

  #model2
  auto.arima(x, stepwise = FALSE, approximation = FALSE,xreg=reg)%>%forecast(h=h,xreg=new_reg) -> fc_arima

  #model3
  set.seed(12345)#for nnetar model
  nnetar(x, MaxNWts=nrow(x), xreg=reg)%>%forecast(h=h, xreg=new_reg) -> fc_nnetar

  #model4
  stlf(x , method= "ets",s.window= 12, h=h)-> fc_stlf_e

  #Combination

  Combi <- (mod2$coefficients[[1]]*fc_stlf$mean + mod2$coefficients[[2]]*fc_arima$mean +
              mod2$coefficients[[3]]*fc_nnetar$mean + mod2$coefficients[[4]]*fc_stlf_e$mean)

  return(Combi)
}


What is the advantage of using RNN with fixed timestep length over Neural Network? More often than not, I see RNNs being used with fixed length timesteps. So what is the difference between the following two networks?

RNN with timestep length of 3 over sequence Xt.
NN with inputs x(t-2), x(t-1), x(t)

They both get 3 timesteps of sequence at each turn. Thus, (I know I am wrong but) these two networks have the same capability. They both use previous 3 samples to predict the next sample. What is the difference, then?

It's true that they both use 3 samples to predict the next sample. But the difference is obviously how they learn it. Deep Neural Nets will treat time-distributed samples such as they were single sample at once. However, RNN will learn it as a sequence by its sequential nature, part by part.

Each activation layer of (uni-directional) RNN will take a part of a sequence, make prediction and feed the prediction on that part forward to the next activation layer which will take the next part of the sequence as input. Important part is that the prediction is feeded forward so that the next neuron learning the next part of the sequence will also affected from the prediction of the previous neuron as well as the input sequence, unlike classical neural networks. It can get easy to understand when you look at the visual example below:

Hope I could help.


How can i use Hellinger Distance on array of different length? I have to use Hellinger distance to compare arrays that are not the same length.

How do you do this correctly? Putting a zero in the missing fields for the shorter array does not sound like the best method to me.

Some more info on my data:

Most array dimensions are (1,58), but a some others are (1,28). Arrays contain numbers from 1 to 3.

Example:

Array1=[1 1 3 2 3]

Array2=[2 3 1 1]

One possible solution: newArray2=[2 3 1 1 0]

Is possible to use Hellinger Distance in this case? Is there any other distance function that could solve my problem?

I'm using Helliger in K-means because it's what the author of a paper I'm reading used. So, I would like to solve this issue using Hellinger.

Thanks.

What are you trying to do?

Don't blindly for together functions without thinking about the underlying math!

Hellinger distance is usually applied to histograms, and your before don't look like histograms. So something is wrong I'm your approach... Go back to the drawing board, not to the code.

P.S. k-means will also need vectors of the same length, and doesn't minimize Hellinger, I'd assume...


CNN - imbalanced classes, class weights vs data augmentation I have a set of data with a few strongly imbalanced classes, eg the smallest class is about 54 times smaller than the largest. Therefore, data augmentation in order to equalize the size of classes seems to me a bad idea (in the example above each image would have to be augmented 54 times on average). So I thought that I could do less augmentation of minority classes, and then use class weights in the loss function. Is this approach better than the mere augmentation or just the use of class weights ?

Is this approach better than the mere
  augmentation or just the use of class weights ?

Note that data augmentation is the process of changing the training samples (e.g. for images, flipping them, changing their luminosity, adding noise, etc.) and adding them back into the set. It is used for enriching the diversity of training samples, thus, in this aspect it cannot be replaced with class weighting. However, it is related to over-sampling since we can both enrich and increase the size of a class via data-augmentation. For now we are only focusing on the size increasing part, i.e. data over-sampling.

Data-oversampling and class weighting are equivalent. Copying the samples of a class 3X is equivalent to assigning a 3X weight to the class. However, the weighting is better from storage and computational point of view since it avoids working with a larger data-set.

Note that based on this equivalency we can mix and match. Increasing the size of a class 6X $\equiv$ increasing the weight 6X $\equiv$ increasing the size 3X and increasing the weight 2X.

Class weights vs over-sampling in more detail

I copied my answer from here, since the solutions are almost the same but questions are the reverse of each other.

This challenge can be tackled in two places:

Data: as you mentioned, this is done by artificially increasing the number of samples for under-represented class $uc$. This produces the same effect as data-sets that are naturally balanced,
Model: this is generally done by over-penalizing the miss-classification of $uc$ compared to other classes. One place for this modification is the loss function. A frequently used loss function in classification is cross entropy. It can be modified for this purpose as follows. Let $y_{ik}$ be $1$ if $k$ is the true class of data point $i$, and be $0$ otherwise, and $y'_{ik} \in (0, 1]$ be the corresponding model estimation. The original cross-entropy can be written as:
$$H_y(y')=-\sum_{i}\sum_{k=1}^{K}y_{ik}log(y'_{ik})$$
which can be weighted as
$$H_y(y')=-\sum_{i}\sum_{k=1}^{K}\color{blue}{w_{k}}y_{ik}log(y'_{ik})$$
For example, by setting $w_{uc} = 10$ and $w_{k \neq uc}=1$, we are essentially telling the model that miss-classifying $1$ member from $uc$ is as punishable as miss-classifying $10$ members from other classes. This is roughly equivalent to increasing the ratio of class $uc$ $10$ times in the training set using method (1).
As a concrete example,  suppose ratio of three classes are $c_1=10\%$, $c_2=60\%$, and $c_3=30\%$, thus, we can set $w_{c_1}=6$, $w_{c_2}=1$, and $w_{c_3}=2$, which implies $w_{c_i} \times P(c_i)=w_{c_j}\times P(c_j)$ for all class pairs $c_i$ and $c_j$.


Can I use the test dataset to select a model? I'm trying to understand if the test dataset can be used to select a final trained model. Let's assume this scenario:
I first split the whole dataset: 70% training, 30% test. Then I fit several models (let's say NN, RandomForest, AdaBoost,..) on the training dataset with cross-validation and tune the hyperparameters to get the best performance on the train data. I know that these scores are biased, since I was tuning the hyperparameters on this data.
Then I use the test dataset to get the true performance on the unbiased data and select which model performs the best.
Is this a correct way to use the test dataset? Some confusion comes from the internet definition of the test dataset:

The sample of data used to provide an unbiased evaluation of a final model fit on the training dataset.

It seems like it should only be used to get the performance of the one final trained model. My teacher told me I can't select a trained model based on the scores of the test dataset and quoted the definition above. I'm struggling to believe that she is correct. Which dataset should be used to select the model then?

Your teacher is correct. The test dataset is unseen data. You could not select the final model using the test set. In competitions, to be fair, the test dataset is not revealed until you submit the final trained model.

To select the hyperparameters of the final model (e.g., activation functions, the number of hidden layers, the number of units, learning rate, dropout, ...), you should use the validation set.


F1 maximization with convolutional neural net. for an imbalanced dataset I'm dealing with an imbalanced dataset for binary classification (about 70% to 30%). I was wondering what is the best way to optimize the F1 score for such a task when using a convolutional neural net.

As of now, I'm sampling the dataset in order to create a balanced training set and am using the mean of softmax_cross_entropy_with_logits (from tf) with a regularization term as my loss.

How can I optimize for the F1 score? As it isn't convex I can't just plug it in as my loss, right? Most papers I found mentioned finding the best threshold. However, these were relatively old, is there something better which can be done for convolutional neural nets ?

I have a paper where we try to maximize F1 score by using different techniques. We hoped that a ranking algorithm like RankNet would be able to get better F1 score than the others. But as you can see looking at our table, regular neural networks, without even using a cost matrix, were good enough. Creating synthetic samples until classes were balanced and even using MetaCost was also largely irrelevant.

We did have a F1 gain when using RankNet, but that was likely due to our post-processing where we convert the ranking score into classes. But that is a pain. I think you can get similar results by, instead of using 0.5 as your threshold for choosing the positive class, choosing a threshold that maximizes the F1 score by just using the training data.

After working on class imbalance for awhile, I think this topic is a great way to get many publications easily, but, in the real world, a post-processing step and/or using a cost matrix to balance priors is more than enough.

ps: Please do not take this results as holy grail. Neural networks are notoriously difficult to optimize, especially across a range of very different datasets. Some cross validation was performed as mentioned in the paper, but, due to lack of time, not as much as we should have. And we did not allow that many iterations. I would try in your own code to introduce weights and some simple things to see if it makes a difference. Do let me know how it works please.


test_train_split with stratify integer overflow I'm trying to do a stratified split for a skewed dataset with target variable 'b'. The target variable is a bit value (either 0 or 1). Here's an example:
df = pd.DataFrame(data={'a': np.random.rand(100000), 'b': 0})
df.loc[np.random.randint(0, 100000, 1000), 'b'] = 1
tr, ts = train_test_split(df, test_size=.2, stratify=df['b'])
print(tr.shape, ts.shape)

This code returns the following:
(93105, 2) (38, 2)

My problem is that the returned train/test arrays do not meet the set split ratio of 20%.
My setup:

Python 3.7.0 (32bit)
Sklearn 0.20.3
Pandas 0.23.4

I discovered that the problem is resulting from an integer overflow in the underlying split function.
How can I resolve this issue and is this a known bug? I couldn't find anything helpful.

How about this?

# Split the data between the Training Data and Test Data
xTrain , xTest , yTrain , yTest = train_test_split(X , y ,
                                                  test_size = 0.30 ,
                                                  random_state = 0,
                                          ----->  stratify = y)


How do I replace NaN values using group by pivot_table in pandas DataFrame? I am working on a machine learning practice problem, from https://datahack.analyticsvidhya.com/contest/practice-problem-big-mart-sales-iii/#ProblemStatement
I want to replace the null values in the column 'Item_Weight' and for that I am using the mean values given by a pivot_table where I calculated the mean of 'Item_Weight' and grouping the mean by column 'Item_Identifier' of the dataset.
item_weight_mean = ds.pivot_table(values='Item_Weight',columns='Item_Identifier')
loc2 = ds['Item_Weight'].isnull()
ds.loc[loc2, 'Item_Weight'] = ds.loc[loc2, 'Item_Identifier'].apply(lambda x: item_weight_mean[x])

I am getting an error for the same code.
(key)
-> 2902             indexer = self.columns.get_loc(key)
   2903             if is_integer(indexer):
   2904                 indexer = [indexer]

D:\Important Applications\anaconda3\lib\site-packages\pandas\core\indexes\base.py in get_loc(self, key, method, tolerance)
   2895                 return self._engine.get_loc(casted_key)
   2896             except KeyError as err:
-> 2897                 raise KeyError(key) from err
   2898
   2899         if tolerance is not None:

KeyError: 'FDN52'

How do I remove this error?

Consider using df.fillna(). That has a  variety of options to replace nans with values of choices. You can apply it on either the complete data frame or on specific columns (I.e Series).


Reduce the risk of numerical underflow We use log-likelihood (called as lambda) to reduce the risk of numerical underflow (in context of sentiment analysis using Naive Bayes).
What does "reduce the risk of numerical underflow" means?

Arithmetic underflow can happen if the result of a calculation is a number smaller than absolute value than the computer can actually represent as a fixed length (fixed precision) binary digit. Instead returning the actual result of the calculation, the computer returns a zero.
Arithmetic underflow can happen in many statistical and machine learning models when many small likelihoods are multiplied together.
Taking the log of each likelihood increases the absolute value so arithmetic underflow is less likely to happen.


Why I get a very low accuracy with LSTM and pretrained word2vec? I'm working on a reviews classification model with only two categories 0 (negative) and 1 (positive). I'm using pre-trained word2vec from google with LSTM. The problem is I get an accuracy of around 50% where it should be around 83% according to this paper. I tried many different hyperparameters combination and still gets a horrible accuracy. I also tried to change the data preprocessing techniques and tried stemming but it hasn't resolved the problem

here's my code

X, y = read_data()
X = np.array(clean_text(X)) #apply data preprocessing
tokenizer = Tokenizer()
tokenizer.fit_on_texts(X)

#converts text to sequence and add padding zeros
sequence = tokenizer.texts_to_sequences(X)
X_data = pad_sequences(sequence, maxlen = length, padding = 'post')

X_train, X_val, y_train, y_val = train_test_split(X_data, y, test_size = 0.2)

#Load the word2vec model
word2vec = KeyedVectors.load_word2vec_format(EMBEDDING_FILE, binary=True)

word_index = tokenizer.word_index
nb_words = min(MAX_NB_WORDS, len(word_index))+1

embedding_matrix = np.zeros((nb_words, EMBEDDING_DIM))
null_words = []
for word, i in word_index.items():
    if word in word2vec.wv.vocab:
        embedding_matrix[i] = word2vec.word_vec(word)
    else:
        null_words.append(word)

embedding_layer = Embedding(embedding_matrix.shape[0], # or len(word_index) + 1
                            embedding_matrix.shape[1], # or EMBEDDING_DIM,
                            weights=[embedding_matrix],
                            input_length=701,
                            trainable=False)

model = Sequential()
model.add(embedding_layer)
model.add(LSTM(100))
model.add(Dropout(0.4))
model.add(Dense(2, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

I also tried other optimizers like AdaMax and MSLE loss function. I'm just confused if the problem isn't with the model and preprocessing where could it be? Thanks

Its huge discrepancy, I suspect a lie.

While +33% can be achieved, you said that you tried very different architectures and you did not get even close. Dont expect that one tweak, one layer, one xyz can give you all of a suddenly such a huge increase. If you did not get closer using suggestions from the paper there is also a possibility (not saying they did, but it has been done before in ML papers- lying about accuracy achieved without providing code) that they lied.


Does the training set of one topic will be useful to predicate the sentiment for any other topic? I am working on Twitter sentiment analysis. Now, I should prepare a training data set that fits for any kind of Twitter data that predicts the sentiment of a tweet as pos, neg, and neu.

From googling, I found an air-line tweets training data set with needed labels (pos, neg, neu, and sentiment). When I checked the sentiment of tweets with that training data set, it gave me mixed results. I found that some positive tweets are given as negative and negative as positive.

Is the method that I am following correct or not?

Please suggest your views for preparing a training data set that can predict any kind of the twitter data.

Since you have trained your model on air-line tweets, the model will learn the characteristics of the air-line tweets. There could be words used in air-line tweets which are positive and contain great weightage but never even used in other tweets leading to negative results.
I suggest you to try

Get a dataset which contains all kinds of tweets which you desire essentially similar to what your task ahead is.
Look into Transfer learning.


Is the activation function the only difference between logistic regression and perceptron? As far as I know, logistic regression can be denoted as:

$$
f(x) = \sigma(w \cdot x + b)
$$

A perceptron  can be denoted as:

$$
f(x) = \operatorname{sign} (w \cdot x + b)
$$

It seems that the only difference between logistic regression and a perceptron model is the activation function.

Is this correct?

TL;DR: Yes and No; they're both similar decision function models but there's more to each model than their main formulation

One could use the logit function as the activation function of a perceptron and consider the output a probability.  Yet, that value would likely need a probability calibration.

As with most ML models several things are very similar from model to model, on the other hand varying tiny parameters can result in a different model.  Let's take on both sides:

Logit Regression and Perceprton similarities

The logit function is used in logit regression for its properties of being an S-curve, by default valued between 0 and 1.  The sign activation function in the perceptron is also shaped like an "S-curve" (with very rough edges - so mathematically not an S-curve by its definition but with similar properties) valued "between" -1 and 1.  Another activation function often used with perceptron is hyperbolic tangent (tanh), which is another S-curve - very similar to the sign function but with a rounded shape (and also valued between -1 and 1).  We can say that tanh is similar to sign because:

$$
\texttt{sign}(x) \approx \texttt{tanh}(kx) \qquad \text{for k >> 0}\\
or\\
\texttt{sign}(x) = \texttt{tanh}(kx) \qquad k \to \infty
$$

So it makes sense to compare tanh with logit as an analogy to compare sign with logit.

Now the logit function is (the one used in logistic regression that is - not the statistical $ln(x/(1- x))$ one):

$$
\texttt{logit}(x) = \frac{L}{1 + e^{-k(x - b)}}
$$

Where $L$, $k$ and $b$ are parameters that we can steer. (Since $b$ is a bias term, an unknown constant, it does not matter if we write $- b$ or $b$)

And the hyperbolic tangent is:

$$
\texttt{tanh}(x) = \frac{e^{2x} - 1}{e^{2x} + 1}
$$

But wait, if we set the parameters of the logit as $L = 1$, $k = 1$ and $b = 0$, then:

$$
2 \cdot \texttt{logit}(2x) - 1 = 2 \cdot \frac{1}{1 + e^{-2x}} -1
= \frac{2e^{2x}}{e^{2x} + 1} - 1
= \frac{2e^{2x}}{e^{2x} + 1} - \frac{e^{2x} + 1}{e^{2x} + 1}
= \frac{2e^{2x} - e^{2x} - 1}{e^{2x} + 1}
= \frac{e^{2x} - 1}{e^{2x} + 1} = \texttt{tanh}(x)
$$

So tanh, which is more-or-less a rounded sign, is a scaled and shifted special case of logit.  Not only the model differ by the activation function alone but the activation functions are very similar to each other.

Logit Regression and Perceprton differences

Log probabilities (this is probably the major difference and the most important one)

We looked at the logit function above but in reality the logit regression takes the logarithm of the logit instead of plain values for the probabilities. Or, more exactly, in statistics the logit is defined as the natural logarithm of what in most ML implementations is defined as a logit decision function. This is very different from the perceptron which performs the output directly from the activation function.

Regularization

Both models (logit regression and perceptron) are often trained with some form of gradient descent optimizer (be it SGD, have it momentum, or even something else). This training will optimize all parameters ($w$ in your representation or $k$ in mine for logit and the weights in a perceptron) but the model itself will also be given a bunch of hyper-parameters.  And in terms of hyper-parameters things do differ:

The perceptron will be trained by optimizing the weights (including a bias) and can be regularized with $L_2$ or $L_1$ (or a combination of both). One may or may not add a bias term.

The actual model of the logistic regression will optimize the logarithm but there's more: most implementations will include a scaling hyper-parameter ($C$) which will multiply the log probabilities. One can use $C$ to regularize the model apart from $L_1$ and/or $L_2$.

Multi-class

The perceptron is always a binary classifier.  One sets an output threshold and that works as the decision function (for the sign or tanh functions the threshold is often $0$). For multi-class classification one must build One-Vs-Rest or One-Vs-One groups of models.

Logistic Regression can be used as a binary classifier and in this case can be used for multi-class classification with One-Vs-Rest and One-Vs_one methods. But, there exist a formulation of logistic regression for direct multi-class classification:

$$
\texttt{multinomial logit}(x|y) = \frac{e^{-k_y(x - b)}}{\sum^{K}_{k=0} e^{-c(x - b)}} \qquad \text{for y in }\{0,1,\dots\}
$$

i.e. whether $x$ belongs to class $y$. This is performed for each class and the result can be passed to a softmax function.  This multi-class formulation can be performed because logit regression deals with log probabilities instead of direct inputs (contrary to a perceptron).


Are ICD codes forward compatible? The ICD code scheme has gone through many revisions: https://en.wikipedia.org/wiki/International_Statistical_Classification_of_Diseases_and_Related_Health_Problems

Is there any compatibility between the different versions? (e.g. can ICD-9 code be directly used where ICD-10 codes are expected?)

9 and 10 have different schema. For example, a displaced transverse fracture of the right femur would be

821.01RT in 9
S72.321 in 10

10 is also more specific. In the example, 9 doesn't tell you that the fracture is transverse and displaced.

Since ICD10 is basically a better organized and more precise superset of the data represented by ICD9, someone has probably created a mapping. However, I haven't looked into that.


Applying convolutional neural network over text documents using 1-D tf-idf feature vectors I want to apply a CNN over documents. I have tf-idf vectors of documents with me (one vector per document).

My question is, is 1D CNN applicable in this case? The reason I am asking this question is that I have read 1D convolutions being applied to sequence of vectors and I have sequence of scalars (tf-idf vector) as input in this case, so is it applicable?

In TFIDF representation "neighborhood" is not interpretable. The 1st number in a document vector has just as much relation to the 2nd and to the 100th.
While in case of a standard time series or sequence analysis "neighborhood" is interpretable.
On the other hand, if you could re-order the TFIDF representation in a way, that makes sense from the analysis perspective (eg: ordering based on the "sentiment" of the word), maybe 1D CNN would show some good results.


Given one language ngram model, how do I compare likelihoods of two texts of different length? Let's say I have conditional probabilities estimates for N-grams and I want to find out which of the two sequences of different length 'looks more natural' in terms of the given model.

How does one do this? Is it something to do with perplexity?

Would also be glad if someone posted a reference to a reasonable Python implementation of the suggested method [a code snippet or a link to nltk/spacy/whatever method].

Thanks in advance.

I didn't find any good explanations or papers on this topic, other than things about category based models and multigrams (parts of words). So, I came up with one myself. I'm using Java, but here's the code for the length normalization translated to Python. Also, this is assuming that you're adding the log probabilities for each word together, like KenLM does for the total sentence (or phrase) score.

# Get likelihood normalized by length and number of words.
# Since the likelihoods of each word are added together, phrases with more words tend to
# have a lower probability.
def getNormLikelihood(phrase):
    score = getLikelihood(phrase) # The normal score
    numWords = phrase.count(' ')+1 # The number of words in the phrase.
    # Get the absolute difference between the
    #   average word length of all words and the average word length of the phrase.
    # Then, weight that weight by the reciprocal of the normal score, such that the
    #   lenWeight is less if the overall score is less.
    # Intuitively, this means that we care less about normalizing by word length the more
    #   common the phrase is.
    # This effectively solves the problem of giving a greater likelihood for long words
    #   that aren't even in the language model (and are given the default "<unk>" score).
    lenWeight = abs(len(phrase) / numWords - avgLen) * (-1/score)
    return score / (1/lenWeight)

How you want to use this normalized score, and how effective it is, depends on your application. In my case (splitting words), I used this normalized score to score each splitting option, but then also added (logarithm) the normal score of the option with context, which you could consider the prior for the option (in Naive Bayes terms). I then added an overall prior (manually picked, but could be tuned) to the original, unsplit option, for the probability of needing to split a word. The normalized score greatly helped, and with these probabilities together, I was able to get 100% accuracy for my test cases.


Measure correlation for categorical vs continous variable Given a variable which is categorical that depends on continuous variables, I would like to know how to check wether these continous variable explain the categorical one.
So:
Y = cagetorical
X1 = continous
X2 = continous
X3 = continous

I'd start with a correlation but which? I've seen How to get correlation between two categorical variable and a categorical variable and continuous variable? but there it is explained wether there is a difference in categorical variables explaining a continous variable, so I think it's another topic?
I'm fine with tool advices in R and python as well.
edit: I'm not sure wether cateogrical is correct here. The values of $ Y $ are $ 0, 1, 2, 3 $ but I could also use $ A, B, C, D $. They represent a classification of the measure of cleanliness of a room.

By saying you want to "explain Y by X" it sounds that you try to build a classifier F that can map X values into expected Y: F(X) --> Y.
If so, you don't have to search for "correlation" necessarily.
There are various methods to build such a classifier. You can use logistic regression \ SVM \ Neural network \ etc.
Besides, if it make more sense for you, you can always first discretize the continuous variables into categorical vars and than use also other methods such as decision trees \ Naive Bayes and more.


Imagining a Linear Regression model with more than 3 dimensions I'm just getting started with Machine Learning and this is really bugging me now.
Assuming we could use more than 2 feature variables to train a Multiple Linear Regression model, how can we imagine the geometry of the model?
Let me elaborate.
For a linear regression model with a feature variable and a response variable, we come up with an equation of a line (y = mx + c) and for a multivariate linear regression model with 2 predictor variables, we can figure out the equation of a plane (a(x-x1)+ b(y-y1) + c (z-z1) = 0).
How can we imagine the model in 4 dimensions or more, since we are humans and as a layman, we cannot imagine higher dimensions like physicists?

Humans live in a 3 dimensional world. Consequently it can be hard for us to visualise anything beyond 3 dimensions. For 4 dimensions, we can perhaps think about time as the next dimension, so we might think about how a 3 dimensional image changes over time. Even that is quite tricky, so for 5 dimensions and above it is basically impossible.
Rather than trying to visualise a model in terms of geometry, it may be better to understand how models work by using linear algebra (ie matrix algebra)


Interpreting the results of randomized PCA in scikit-learn I'm using scikit-learn to do a genome-wide association study with a feature vector of about 100K SNPs. My goal is to tell the biologists which SNPs are "interesting".

RandomizedPCA really improved my models, but I'm having trouble interpreting the results. Can scikit-learn tell me which features are used in each component?

Yes, through the components_ property:

import numpy, seaborn, pandas, sklearn.decomposition
data = numpy.random.randn(1000, 3) @ numpy.random.randn(3,3)
seaborn.pairplot(pandas.DataFrame(data, columns=['x', 'y', 'z']));

sklearn.decomposition.RandomizedPCA().fit(data).components_

> array([[ 0.43929754,  0.81097276,  0.38644644],
       [-0.54977152,  0.58291122, -0.59830243],
       [ 0.71047094, -0.05037554, -0.70192119]])

sklearn.decomposition.RandomizedPCA(2).fit(data).components_

> array([[ 0.43929754,  0.81097276,  0.38644644],
       [-0.54977152,  0.58291122, -0.59830243]])

We see that the truncated decomposition is simply the truncation of the full decomposition. Each row contains the coefficients of the corresponding principal component.


How to discard trash topics from topic models? I am undertaking text analysis of some twitter data. In the end I want to have a data that is interpretable. And so in the end I would like to reduce the data to relevant unit of analysis. Topic models seem a nice fit since they reduce the noise significantly, but still at the end there are quite a lot of trash topic lists which do not mean anything.What would be the best way to remove these topics. Is there a working example of this?

I am using LDA. Probably I will use Gensim. Its going to be unsupervised and its a simple twitter data. So the whole corpora will be 1500 tweets of 160 characters.

I assume you have inspected your topic-by-word matrix, and that is why you say you have 'trash' topics.  If no words are strongly associated with a given topic, it may not be a useful topic.  If you find many of these, you can re-run the experiment with fewer topics.

If instead there are words strongly associated with these topics, you just don't see the cohesive meaning behind the words associated with the topic, then these are not exactly "trash" topics.  They are still capturing the latent structure of the data, it just isn't something easily deemed a "topic" in the traditional sense of the word.

Another thing to remember is that the hyperparameters also control the shape of your resulting distributions.  This answer does a great job explaining alpha and beta.