How cross validation works for regression? For regression type problem we know the result is a continuous value, so how is it be cross validated?
In classification type problem we know the class label so easy to compare, but how is it compare in regression type problem?

The idea behind cross validation is to understand the performance of some measure of your model's performance on unseen data. This can be applied to loads of different statistics, not just ones relevant to classification like accuracy. Common measures for holdout performance of regression models include MSE and MAPE, either (or both) of which can be cross-validated over.


regularization error vs over fitting I have a collected data from 50 unique blocks, and then merged data from 49 blocks into one data set, and saved the data from 1 block for testing purpose.
I then split the merged data set from 49 blocks using train_test_split(sklearn). Then used training data to train a random forest regressor using cross validation and get a good model score(R^2 score from sklearn random forest regressor model) on train (0.99) and test set(0.94). But when I use the trained model on the reserved data from the 1 block, the performance is very bad (-1.0).
If I merge the data from all 50 blocks, then use train test split and keep 60% data as training set, 20% as test set, and 20% (reserved set), I get good scores from all three sets. Training set score(0.98), test set(0.93) and the reserved set(0.96).
Any intuition regarding what could be causing this? And any suggestions on how to improve the model score for the 1 block of unseen data?

Welcome to the community!
There are points coming to my mind:

Check the amount of data in each block and then their distribution. First experience might be due to the lack of enough data in reserved block (i.e. you literally just trained but did not validated resulting to a complete overfitting) or having a totally different distribution (i.e. the difference between blocks is NOT random).
Be sure about wht you called "Cross Validation". If you splitted 49 blocks into a training and a testing set, then it is minimal version of Cross Validation. 49 blocks should be splitted to $k$ different set, each time one of them plays the test set role and then the final error is averaged over all $k$ runs. That is statistically more robust (first experience shows problem in robustness of statistical inference)
Second experiment uses 100 blocks instead of 50 (where were those other 50 in first experiment?) so first, you have more data, and second, you did not split according to "blocks" (whatever they are) but merged all and randomly chose train, validation and test. This procedure reduces the danger of distributional bias within train/valid/test and also might be an indicator of the fact that statistically those "blocks" are different (their distribution)

hope it helped. Please share your results so everyone can learn from it.
Good Luck!


Patch wise training vs Full Convolutional Training in semantic segmentation As mentioned in the title, what are those 2 methods? I already checked this question: Patchwise and Full training, (and the mentioned paper) but i can't really understand the meaning and the process of those 2 methods.

What's the difference between the two? When should i prefer one instead of the other one?

In Patch wise training, the CNN is trained on a small patch of the image instead of the whole image. For example, if the input image is of size 6000-by-6000 pixels, it would be difficult to train a CNN on it. You could downscale the image but then you would lose relevant information. Instead, if you perform patch based semantic segmentation, the model would pick up patches from the image, say of size 200-by-200 pixels and train on those patches as if they were separate images.

This feature retains information as well as provides good accuracy.A link to a paper discussing the relevance of patch-based CNN can be found here.


Eliminate input in gradient by clever choosing of cost function in neural networks In http://neuralnetworksanddeeplearning.com/chap3.html

The author explains that for for a single neuron in neural net, by choosing cost function as cross entropy we can eliminate the derivative of activation function in the gradient term, if the activation function is choose to be sigmoid function.

In the problem, he asks why we cannot eliminate the input to neuron term x in gradient of cost function with respect to weights.

I had the following reasoning, in order to compute gradient of Cost, we use chain rule, and derive cost with respect to activation, and activation with respect to  (w * x + b), and the sum with weight.

For a weight Wi, the derivative of (Summation w * x + b) w.r.to Wi is always xi, and the derivative of activation function cannot know that, so it can never eliminate Xi unless its zero.

Or is there any other subtle reasoning?

The main reason is that the chain rule is hidden in your justification. The fact that the inner function is linear w.r.t weights then derivative w.r.t weight will always result into input $x$ will be somewhere in $\dfrac{\partial C}{\partial w}.$


Multi-Class classification with CNN using keras - trained model predicts object even in a fully white picture I built an multi classification in CNN using keras with Tensorflow in the backend. It nicely predicts cats and dogs. However, when it comes to an image which does not have any object-white background image-, it still finds a dog ( lets say probability for dog class 0.75…, cats 0.24… ). I am a quite newbie learner in learning built with neural network.

Sorry if I am asking a silly question, even though I have searched the internet I could not find any answer.

What is my exception from the case of the white background image as an input to prediction method, is 0 probability for dog and cat classes.

Any suggestion would make me so happy.

The below is how I implemented the training.

classifier = Sequential()

classifier.add(Conv2D(32, 3, 3, input_shape=(64, 64, 3), activation='relu'))
classifier.add(MaxPool2D(pool_size=(2, 2)))

classifier.add(Conv2D(32, 3, 3, activation='relu'))
classifier.add(MaxPool2D(pool_size=(2, 2)))

classifier.add(Flatten())

classifier.add(Dense(units=128, activation='relu'))

classifier.add(Dense(units=2, activation='softmax'))

# Metrics will be categorical_accuracy
classifier.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

from keras.preprocessing.image import ImageDataGenerator

train_datagen = ImageDataGenerator(
    rescale=1. / 255,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True)

test_datagen = ImageDataGenerator(rescale=1. / 255)
training_set = train_datagen.flow_from_directory(
    '/Users/ozercevikaslan/Desktop/Convolutional_Neural_Networks/dataset/training_set',
    target_size=(64, 64),
    batch_size=32,
    class_mode='categorical')

test_set = test_datagen.flow_from_directory(
    '/Users/ozercevikaslan/Desktop/Convolutional_Neural_Networks/dataset/test_set',
    target_size=(64, 64),
    batch_size=32,
    class_mode='categorical')

classifier.fit_generator(
    training_set,
    steps_per_epoch=8000,
    epochs=25,
    validation_data=test_set,
    validation_steps=2000)

As I mentioned in my question post, the post is a bit silly even for a new learner. In this case, the world has only 2 classes which are dogs and cats so the output must be either a dog or a cat.


what is the reason behind the bad outputs gained by RNN, LSTM when using GloVe pretrained model in text classification? the problem is with the results gained for accuracy and f1  afer training our model via pretrained models such as GloVe.
when I apply CNN as a classifier, the result are good as follows:

acc: 0.9345 - val_loss: 0.1513

but when I apply RNN and LSTM as a classifier the results will be as follows:

24931/24931 [==============================] - 188s 8ms/step - loss: 7.9559 - acc: 6.0166e-04 - val_loss: 7.9904 - val_acc: 0.0000e+00
Epoch 2/4
24931/24931 [==============================] - 189s 8ms/step - loss: 7.9645 - acc: 0.0000e+00 - val_loss: 7.9904 - val_acc: 0.0000e+00

the above result is reached both via RNN and LSTM.

the problem is that I use the same data set and the same structure and the same GloVe but I reach acc: 0.9345 for CNN and gain acc: 0.0000e+00 for both LSTM and RNN.
It is worth noting that I have changed optimizer but still get the same result. the applied dataset contains 41,399 items, totaling 60.3 MB and also is binary.
any guidance will be appreciated as I am a beginner in working with GloVe.

I apply keras with tensorflow backend with python 3.5 in ubuntu.

I have already dealt with RNN and LSTM codes and finally I found tow solutons to achieve better result for accuracy measure which are as follows:
at first I change the activation function, optimizers and learning rate but it is worth noting that activation function has the most impact and the second trick which I have used is to removing stop words, now the code works fine.
I hope it is useful.


Estimating time to travel between two lat/longs I'm trying to create an offline estimator for how long it would take to get from one lat/long to another. Two approaches I have come across are the Haversine distance and the Manhattan distance. What I'm thinking of doing is calculating both of them and then using the average between the two as the distance and then use some average speed to calculate time.

Since this value will be used as an estimator for drivers in a city a straight line distance will grossly underestimate the actual distance drivers will have to travel.

Any thoughts would be greatly appreciated.

This example is carried out using R.

Suppose there exists a data frame with latitude and longitude coordinates for two locations:

coors <- data.frame(
  lat = c(48.8566,52.5200),
  long = c(2.3522,13.4050),
  stringsAsFactors = FALSE
)

The distance between the two points is calculated as follows:

> p=0.017453292519943295
> a = 0.5 - cos((coors$lat[2] - coors$lat[1]) * p)/2 + cos(coors$lat[1] * p) * cos(coors$lat[2] * p) * (1 - cos((coors$long[2] - coors$long[1]) * p)) / 2
> distance <-12742*sin(sqrt(a))
> distance #Distance in kilometers
[1] 876.0783


data science student I need to replace a specific value in a particular column with NaN. Here is what I tried but it didn't work.

# Names of the columns we're searching for missing values
columns = ['median', 'p25th', 'p75th']

# Take a look at the dtypes
print(recent_grads.columns.dtype)

# Find how missing values are represented
print(recent_grads["median"].unique())
print(recent_grads["p25th"].unique())
print(recent_grads["p75th"].unique())

# Replace missing values with NaN
for column in columns:
    recent_grads.loc[recent_grads.columns == 'UN', column] = np.nan

If I understood your question correctly, you can do something like:

import numpy as np

columns = ['median', 'p25th', 'p75th']
df[columns] = df[columns].replace(r'UN', np.nan, regex=True)


Can I create a word cloud of crowdfunding donors using word cloud? I have a table like this:

FirstName    SecondName     Amount
Lorenzo      Perone         100
Mario        Rossi          25
...          ...            ...

I'd like to create a "word cloud" using "Amount" as weight, is it possible using the "word cloud" tool?
Thanks.

I will give you a simple solution using R which requires the wordcloud package. Of course there are many other solutions which do not require any programming skills.

The solution is a slight varient of this R-Bloggers tutorial. Feel free to have a look there for further formatting.

library(wordcloud)

words = c('Paolo Gentiloni', 'Matteo Renzi',
          'Enrico Letta', 'Mario Monti',
          'Silvio Berlusconi', 'Romano Prodi')
freq = c(100, 25, 50, 70, 95, 20)

wordcloud(words = words, freq = freq, min.freq = 1,
          max.words=200, random.order=FALSE, rot.per=0.35,
          colors=brewer.pal(8, "Dark2"))

It produces the following output:

Btw. @Lorenzo Perone: Where the ones listed Italien names? I was not sure about that.


Is GridSearchCV computing SVC with rbf kernel and different degrees? I'm running a GridSearchCV with a OneVsRestClasssifer using SVC as an estimator. This is the aspect of my Pipeline and GridSearchCV parameters:

pipeline = Pipeline([
    ('clf', OneVsRestClassifier(SVC(verbose=True), n_jobs=1)),
    ])

parameters = {
    "clf__estimator__C": [0.1, 1],
    "clf__estimator__kernel": ['poly', 'rbf'],
    "clf__estimator__degree": [2, 3],
}

grid_search_tune = GridSearchCV(pipeline, parameters, cv=2, n_jobs=8, verbose=10)
grid_search_tune.fit(train_x, train_y)

According to the documentation of SVC the degree parameter is only used by the poly kernel:

http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html

  degree : int, optional (default=3)

  Degree of the polynomial kernel
  function (‘poly’). Ignored by all other kernels.

but when I see the output of my GridSearchCV it seems it's computing a different run for each SVC configuration with a rbf kernel and different values for the degree parameter.

[CV] clf__estimator__kernel=poly, clf__estimator__C=0.1, clf__estimator__degree=2
[CV] clf__estimator__kernel=poly, clf__estimator__C=0.1, clf__estimator__degree=2
[CV] clf__estimator__kernel=rbf, clf__estimator__C=0.1, clf__estimator__degree=2
[CV] clf__estimator__kernel=rbf, clf__estimator__C=0.1, clf__estimator__degree=2
[CV] clf__estimator__kernel=poly, clf__estimator__C=0.1, clf__estimator__degree=3
[CV] clf__estimator__kernel=poly, clf__estimator__C=0.1, clf__estimator__degree=3
[CV] clf__estimator__kernel=rbf, clf__estimator__C=0.1, clf__estimator__degree=3
[CV] clf__estimator__kernel=rbf, clf__estimator__C=0.1, clf__estimator__degree=3

Shouldn't all values of degree be ignored, when the kernel is set to rbf?

From what I understand, you'll be able to pass different values of degree even when you're using kernels that are not the polynomial kernel, just that it will not be used. I believe the score will come out to be similar for the other kernels even with different degrees. Will you be able to confirm this?

To avoid additional computation time due to redundant searches, you can fine tune the GridSearchCV by specifying two grids. Try out the below code and passing it into the param_grid argument.

param_grid = [
  {'C': [1, 10, 100, 1000], 'kernel': ['linear']},
  {'C': [1, 10, 100, 1000], 'gamma': [0.001, 0.0001], 'kernel': ['rbf']},
 ]

You can explore the GridSearchCV documentation for the specific examples. Take a look at the example here (look at 3.2.1)


XGBoost: How to set the probability threshold for multi class classification I am using the XGBoost for classification of text data.
There are the 3 different classes in the training dataset.

classifier = Pipeline([
    ('features', FeatureUnion([
        ('text', Pipeline([
            ('colext', TextSelector('Issue')),
            ('tfidf', TfidfVectorizer(tokenizer=Tokenizer, stop_words='english',
                     min_df=.0025, max_df=0.25, ngram_range=(1,3))),
            ('svd', TruncatedSVD(algorithm='randomized', n_components=300)), #for XGB
        ]))
    ])),
    ('clf', XGBClassifier(n_estimators=300,max_depth=3, learning_rate=0.1)),
#    ('clf', RandomForestClassifier()),
    ])

As per the classification results, the class for which prediction probability is highest is assigned to the data point.
For example, if the prediction probability for class A is .67, then that data point is assigned to that category(Class A).

predictionProbability=classifier.predict_proba(X_test)

But the requirement is to assign the data point to the 4th Category that is "UnDetermined" if the prediction probability for the data point does not differ much among the classes. For example, if the prediction probability of the datapoint for three classes is .32,.33,.35, then can we mark it as Undetermined.
So that the user can review the undetermined category and assign that to the appropriate class.

But I am not sure how to set the cutoff probability for multiclass classification problem?  PLease let me know how to identify the cutoff probability. So that i can mark the data points with prediction probability less than cutoff probability as "Undetermined"

You don't set it in xgboost. Its job is to return probabilities in predict_proba. predict does the logical thing and tells you the most likely class.

If you want to interpret the probabilities differently, you'd have to write code to do so. It depends on what "does not much differ" means. For example if you simply mean "the most likely class is has probability < 0.5", in numpy it's something like np.argwhere(probabilities.max(axis=1) < 0.5). Those rows are what are undetermined. For the rest the prediction is the argmax of the row.

Of course, that's not the only definition of when it's undetermined, so you need to figure out what the rule is that you're meant to implement. You could base it on entropy too, not simply the max class probability.


Reducing (Variance) | the gap between my weights I have ML ready samples. And each sample has a weight.
The weights distribute between [0-1]
My problem arise because there are a lot of samples which are 0.001, 0.00x
And a lot of samples which are 0.997, 0.99x
I am going to sample data based on these weights. And samples with 0.99x will overshadow the other samples in the data set while 0.00x samples will have 0 significance.
The solution I am looking for, is some kind of function over those weights that will balance them a little bit / reduce those huge gaps (Therefor reducing the variance) AND still preserve their order
So if 0.997 turned into 0.88, 0.996 will turn into something < 0.88

For example:
in [0.01, 0.1, 0.4, 0.6, 0.8, 0.997]
I would want something like:
[0.15, 0.2, 0.42, 0.6, 0.78, 0.9] (Just an example)

Try the softmax function:
weights = [0.01, 0.1, 0.4, 0.6, 0.8, 0.997]

temperature = 1
weights = np.array(weights) / temperature
new_weights = np.exp(weights) / np.exp(weights).sum() # softmax function

You can tweak the temperature hyperparameter.
Higher the value - the "new_weights" come closer - yet order is preserved


Changes in CNN architecture How can I see whether a CNN layer is overfitting, underfitting or whether the activation function, normalisation requires some changes?

We usually say a model is overfitting, not a layer or a neuron. Basically, there is not a standard way to set the number of convolutional layers. Iff you see your model is not learning well enough, increase the number of filters. Iff it is overfitting, decrease the number of filters or use the common approaches like drop out to overcome the problem. There are standard ways for interpreting the number of neurons is dense layers. Take a look at How to set the number of neurons and layers in neural networks.

If by activation normalisation you mean batch normalisation, there is an answer. Batch normalisation is a technique for avoiding covariat shift. It does not allow the distribution of activations to change. It is needed for very deep networks and it does help. If you have a very deep network, use it and you will see that it is very helpful most of the time. Simple networks may be able not to have them. These are usual facts but you may be able not to use them in dense layers, something like AlexNet, and still have a good result.


Arguments in python fast.ai function that are not in the function definition? I have been coming across function calls that use arguments that are not in the function definition. I would like to know how that works (i.e. how the compiler interprets this).

For example, this function call:

 interp.plot_confusion_matrix(figsize=(12,12), dpi=60)

uses the variables "figsize" and "dpi", but neither of them turn up in the definition of plot_confusion_matrix: help(interp.plot_confusion_matrix) gives:

plot_confusion_matrix(normalize: bool = False, title: str = 'Confusion matrix',
cmap: Any = 'Blues', norm_dec: int = 2, slice_size: int = None, **kwargs) -> None
method of fastai.vision.learner.ClassificationInterpretation instance
    Plot the confusion matrix, with `title` and using `cmap`.

Why is it not an error to use "figsize" and "dpi" in the function call if they are not arguments of the function? How does this work in python?

Those parameters are taken care of by **kwargs in the function definition.
You can look how this is dealt with by the code in the fastai repo and looking for **kwargs in the function definition.
Some explanations about how this works can be found on SO: https://stackoverflow.com/questions/3394835/use-of-args-and-kwargs.


Do precision-recall curves have a constant shape/pattern? I know ROC curve always looks like a stair shape and that I can evaluate AUC of ROC. And I know I can compute AUC of ROC curve to compare which model is better. What I wonder is:

Does precision-recall curves have a constant shape/pattern?
Can I compute AUC of PR curve to compare which model is better?
If not, how can you compare models by PR curves?

I wonder if PR curve(Precision-Recall curve) has a constant shape(pattern)?

For a model that is better than random (having a ROC above tpr = fpr line), which is mostly the case, precision and recall roughly have an inverse relationship. Therefore, we should expect that a (precision, recall) curve generally decreases (checkout this post on the relationship between precision and recall).

  Can I compute AUC of PR curve to compare which model is better?

Yes. It is even preferred over AUC (of ROC) when classes are imbalanced in the test set (for example, check out this blog, this Kaggle notebook, and this 2015 paper). Here is an image from the referenced blog

which shows that PRC (right), unlike ROC (left), is robust to imbalanced classes.

And here is a quote from the referenced Kaggle notebook (texts in brackets are added by me):

  ROC curve is not a good visual illustration for highly imbalanced
  data, because the False Positive Rate ( False Positives / Total Real
  Negatives ) does not drop drastically [i.e., does not produce a lower curve with smaller AUC] when the Total Real Negatives is
  huge [i.e., negative class is much larger than positive class].

  Whereas Precision ( True Positives / (True Positives + False
  Positives) ) is highly sensitive to False Positives and is not
  impacted by a large total real negative denominator.

which is illustrated as below

emphasizing on the black model (black curve) that is shown to be powerful in ROC  but not that much in PRC; given that negative to positive ratio is around 600 (the ratio should be 1 when classes are balanced).


Keras/Theano custom loss calculation - working with tensors I'm struggling to write some tensor manipulation code for a custom loss function I'm using in Keras.

Basically, I'm trying to modify a binary_crossentropy loss by adding a weight that is calculated from a particular feature.

First thing I do is pass in my extra feature data to the custom loss by appending it to the y_true like this:

y_trainModded = numpy.append(y_train, MyExtraData, axis=1)

Which is then passed to the fit function like:

model.fit(X_train, y_trainModded, epochs=2500, .....)

Then extracted to make it usable like this:

def myCustomLoss(data, y_pred):
    y_true = data[:,:2]
    MyExtraData = data[:,2]
    ...
    ...

So far, that all works fine.  However, I'm struggling with a section where I want to only select the MyExtraData where I predicted '1'.  Intuitively, this would simply be something like:

ExtraDataWherePredicted1 = MyExtraData[y_pred > 0]

However, we're dealing with tensors, not numpy arrays. I tried casting to numpy arrays using eval(), but that didn't work. I also tried various approaches using keras.backend operations such as:

WherePredicted1 = K.greater( y_pred,0)
ExtraDataWherePredicted1 = tf.boolean_mask(MyExtraData, WherePredicted1)

Which I could then use to weight my loss such as:

return K.mean(K.binary_crossentropy(y_pred,y_true), axis=-1)-(K.mean(ExtraDataWherePredicted1))

But anything I try throws out various errors...I just can't figure out how to calculate ExtraDataWherePredicted1. I'm also finding it super hard to debug the loss function because I can't print() anything inside it, so it's very hard to double check to see if the arrays/tensors are what I expect them to be.

Any help would be appreciated!

I think I might have finally just solved this myself.

1) I changed my Keras backend to use TensorFlow instead of Theano, so that I could use:

tf.boolean_mask

This command was not available under the Theano backend and thus giving me errors.

2) I had to change my code slightly to work with the correct dimensions. It now reads:

WherePredicted1 = K.greater( y_pred[:,1],0.5)
ExtraDataWherePredicted1 = tf.boolean_mask(MyExtraData, WherePredicted1)

Still finding it hard to debug/test a custom loss function, but it's looking like feasible values to might well be correct.


How do I pick "K" for precision at K and recall at K? I understand precision at k and recall at k.  It is a more useful metric for evaluating the success of a binary classifier when the positive class is overwhelmingly out-weighed by the negative class.

I'm wondering how to choose an appropriate "k" value.  According to resources like this the recall at "k" is bounded by the number of positive examples, so it is not a useful metric to use when evaluating the success of severely imbalanced classes.

It also seem to me like this:  Precision at K is also limited by the number of positive examples at K.  Saw we have 100 examples total, and only 3 are positive examples.  Say we rank these

Scenario 1: we choose k=10.  Then, the precision at K can at most be 3/10 = 0.3. And, the recall at k will be 0.03 because there are 3 in the entire dataset of 100.

Scenario 2: we choose k=3.  Then, the precision at K would be 3/3 = 1.0 !!!And, the recall at k will STILL be 3/100 = 0.03.

Even though our binary classifier is performing perfectly, it's perfect performance only is reflected when we choose k=3?

So, my question how do I choose K correctly?

Precision and Recall can be dubious at times. It depends on the point on the ROC Curve. Therefore, pick the point as per your need. The best way is to give more information about your results; i.e., Accuracy measure and Confusion matrix.


ValueError: operands could not be broadcast together with shapes while using two sample independent t test I am trying to perform two sample t test. My data set consists of 744 rows and 186 columns for which I have calculated total sum and mean. I need to perform two sample t test. My csv looks like this from which I have to calculate ttest and rank sum test for each row as individual row denotes separate ID and have the corresponding values :

SRA ID  ERR169499            ERR169498           ERR169497
Label   1                    0                   1
TaxID   PRJEB3251_ERR169499  PRJEB3251_ERR169499 PRJEB3251_ERR169499
333046  0.05                 0.99                99.61
1049    0.03                 2.34                34.33
337090  0.01                 9.78                23.22

The labels 0 and 1 are for case and control respectively. So far I have done this:

import pandas as pd
import numpy as np
from scipy.stats import ttest_ind
from scipy.stats import ranksums

def transposer(filename):
file = open(filename, 'rt')
pd.read_csv(file).T.to_csv(str(filename).split("/")
[-1].split(".")[0]+'_transposed.csv',header=False)

pd.read_csv('project.csv').T.to_csv('transposed.csv', header=False)

file = open('transposed.csv', 'rt')
out = open('final_out.csv', 'w')
meta = open('Meta3251.csv', 'rt')
contents = {}
for ids in meta:
    contents[ids.split(',')[1]]=ids.split(',')[-1]
count = 0
for row in file:
    if count == 0:
    out.write('SraID, Label,'+row)
    count=1
else:

    try:
        pid = row.split(',')[0].split('_')[1]
out.write(pid.replace('\n','')+','+contents[pid].replace('\n','')
+','+str(row))
        out.flush()
    except:
        print(pid)
        pass
file.close()
out.close()
transposer('final_out.csv')
file1 = open('final_out_transposed.csv','rt')
label = []
data = {}

x = open('final_out_transposed.csv','rt')
for r in x:
    datas = r.split(',')
    if datas[0] == ' Label':
        label.append(r.split(",")[1:])
label = label[0]
label[-1] = label[-1].replace('\n','')
counter = len(label)
for row in file1:
    content = row.split(',')
if content[0]=='SraID' or content[0]== 'TaxID' or content[0]==' Label':
    pass
else:
    dt = row.split(',')
    dt[-1] = dt[-1].replace('\n','')

    data[dt[0]]=dt[1:]
keys = list(data)
sum_file = open('sum.csv','w')
sum_file.write('TaxId,sum_case,sum_ctrl,case_count,
ctrl_count,case_mean,ctrl_mean,\n')
for key in keys:
    sum_case = 0
    sum_ctrl = 0
    count_case = 0
    count_ctrl = 0
    mean_case = 0
    mean_ctrl = 0
for i in range(counter):
    if label[i] == '0':
        sum_case=np.float64(sum_case)+np.float64(data[key][i])
        count_case = count_case+1
        mean_case = sum_case/count_case
    else:
        sum_ctrl = np.float64(sum_ctrl)+np.float64(data[key][i])
        count_ctrl = count_ctrl+1
        mean_ctrl = sum_ctrl/count_ctrl
sum_file.write(key+','+str(np.float64((sum_case)))+','

+str(np.float64((sum_ctrl)))+','+str(np.float64((count_case)))
+','+str(np.float64((count_ctrl)))+','+str(np.float64((mean_case)))
+','+str(np.float64((mean_ctrl)))+'\n')
sum_file.flush()
sum_file.close()

df  = pd.read_csv('final_out_transposed.csv', header=[1,2], index_col=[0])
case = df.xs('0', axis=1, level=0).dropna()
ctrl = df.xs('1', axis=1, level=0).dropna()
(tt_val, p_ttest) = ttest_ind(case, ctrl, equal_var=False)
print (tt_val)
print (p_ttest)

I am getting the error:

  ValueError: operands could not be broadcast together with shapes (92,) (95,)

How can I handle this error. I cannot change my data.

The answer to this question would be :
The objects created by the xs method of the Pandas DataFrame look like two-dimensional arrays. These must be flattened to look like one-dimensional arrays when passed to ttest_ind. The values attribute of the Pandas objects gives a numpy array, and the ravel() method flattens the array to one-dimension.
It would go like :

df  = pd.read_csv('final_out_transposed.csv', header=[1,2], index_col=[0])
case = df.xs('0', axis=1, level=0).dropna()
ctrl = df.xs('1', axis=1, level=0).dropna()
(tt_val,p_ttest ) = ttest_ind(case.values.ravel(), ctrl.values.ravel(),
equal_var=False)
print (tt_val)
print(p_ttest)


Deep Q-learning, how to set q-value of non-selected actions? I am learning Deep Q-learning by applying it to a real world problem. I have been through some tutorials and papers available online but I counldn't figure out the solution for the following problem statement.
Let's say we have $N$ possible actions in each state to select from. When in state $s$ we make a move by selecting an action $a_i, i=1\dots N$, as the result we get a reward $r$ and end up in a new state $s^\prime$. In order to update the Neural Network with this experience $(s, a, r, s^\prime)$, the only ground truth q-value that we have is for action $a_i$. In other words we do not have any ground truth q-values for all other possible actions ($a_j, j=1\dots N, j\neq i$). Then how should we feed this training data sample to the Neural Network?
Following are the possibilities I was thinking about,

Set other q-values to don't cares. In this scenario we do not update the weights of the last hidden layer that connects to the output values of all $a_j$'s. However, due to interconnections between the earlier layers, any update to the weights will affect the q-values of $a_j$'s.

Set other q-values to the current predicted values by the Neural Network. This makes the error for those q-values zero, but like above solution the change in the weights will affect the $a_j$'s q-values eventually.

Use one Neural Network for each possible actions. This perfectly makes sense as the solution to me, but Deep Q-learning uses one network to predict the q-values of all possible actions (regardless of having two networks, one for policy and the second as target).

If someone who has experience and knowledge to help me with this to understand, I would highly appreciate it.

Consider these two depictions of a simple Deep Q Learning Algorithm:

Look at the gradient update steps in both images: we are only taking the gradient with respect to the Q value of the one action, $a_i$. The network outputs $N$ nodes, but since we only care about one of them, you can probably see that all of the computations still work out, for the ground truths of the other output nodes are never used in this gradient update step.
I hope this helps.
Edit: looking at you first solution you listed, this is correct, but i feel you aren’t understanding the actual computation of the gradient. If you work it out, the ground truth values of the other nodes aren’t actually used because the loss function is only written in terms of the $i$th node.


neural network to find a very simple linear model (scikit-learn) I'm trying to test different machine learning algorithm to try to find correlation between various data on MRI scans.
Since I'm dealing with medical data, I don't have access to many events, but still I'm trying to see what a simple fully connected NN while provides me.
By debugging, it looks like even if I enter the output almost as is in the input, a simple NN using the default scikit-learn NN regressor is unable to find a satisfactory model.
I tried to reproduce the effect and I'd like to share with you the following code. I guess I'm just doing something wrong, because, basically, I'd like the NN to produce a simple model where label=X2-X1 and it doesn't work to me which is really surprising me.

here is the code:

import sklearn.neural_network as NN
import matplotlib.pyplot as plt
import numpy as np
N_events=100000 # number of rows of dataset
N_features = 2 # number of features
X = np.random.rand(N_events,N_features) # chose the data randomly
labels = X[:,1]-X[:,0] # label = X2-X1

C = NN.MLPRegressor(hidden_layer_sizes=(2,2),max_iter=500000,random_state=1) # very simple scikit-learn NN regressor
n = N_events // 2 # half data for training, half for test

print(X[0:10,1]-X[0:10,0]) # check that indeed label=X2-X1
print(labels[0:10])
C.fit( X[0:n,:] , labels[0:n]) # train the model

# plot y_predicted vs y_true for the test set...
plt.figure()
y_real = labels[n:] #
y_pred = C.predict(X[n:,:])
plt.plot(y_real,y_pred,'.')

# plot y_predicted vs y_true for the training set... even this doesn't work
plt.figure()
y_real = labels[0:n] #
y_pred = C.predict(X[0:n,:])
plt.plot(y_real,y_pred,'.')

I was expecting the NN with 2x2 = 4 degrees of freedom to easily find such a simple model (note that I even don't add unused features which could disturb the fit, since N_features is 2 here), so I guess I'm just not using the code correctly.
Can anyone helps me ?

Many thanks

The problem is relu which is the default activation function. It zeros the inputs smaller than 0. Problem will be solved by using other functions. For example:

C = NN.MLPRegressor(activation="identity", hidden_layer_sizes=(1,1),max_iter=25,random_state=1)

Note that even 1 hidden layer with 1 neuron is enough to find the $X_2 = X_1$ decision boundary.


How to use "tree boosting" with a data-driven loss function We have a problem which has a data-driven (non-analytical) loss function. Our target contains whole numbers between 0 and 20 (the target is inherently discrete), although larger values are possible, just not present in our dataset. The fact that we have a very precise loss function leaves us with some serious issues when using algorithms like XGBoost:
The loss function is generally non-convex. It's not easily fitted by a convex function since its shape is data-driven and can vary drastically. For example, this means that a large punishment is inevitably given for predictions further from the part of the function that is well-fitted, where no large punishment is required. If we interpolate instead of fit, the hessian can be negative (see attached picture), which is a problem for determining leaf weights (right?).

We think we can adapt something like the XGBoost algorithm (I use this algorithm as an example because I'm both familiar with the paper and the API) by swapping out its dependance on the gradient en hessian with a brute-force method for finding the optimal leaf weights and best gain. However, this will slow down the algorithm massively, perhaps cripplingly so.
My questions are: is the some default way of dealing with complex loss-functions within existing algorithms? Is the an algorithm that is suited for dealing with these problems? Is there anything else you could suggest to solve the above issues?
Thanks in advance.

First some prior and known aclarations (that you probably already know).
Metric is what we want to optimize.
Optimization Loss is what the model optimizes.
Obviously, we would like the Metric and the optimization loss to be the same, but this is always not possible. How to deal with this?

Run the right model. Some models can optimize different loss functions. In the case of XGBoost you have two loss functions, the one of the decision tree and the one of the boosting.

Preprocess the target and optimize another metric, this will be for example transforming the  target to the logarithmic of the target and then in that space applying a known loss function

Optimize another loss function and metric and then post-process the predictions.

Write your own cost functions. For xgboost we implement a single function that takes predictions and target values and computes the first and second-order derivatives.

Optimize another metric and use early stopping.

The last one almost always works.
In general for complex algorithms Neural Networks tend to work better due to the flexibility of the loss functions (more than in normal ML).


Evaluating machine learning explainers? I'm working on a project where multiple machine learning explainers (LIME and SHAP, potentially more coming) are applied to pre-trained models (neural networks) to help explain the predictions of those 'black boxes'. The explainers assign each feature an importance value for each prediction.

The thing is, those feature importance values differ from explainer to explainer, and it is rather tricky to evaluate which explainer actually performs best and produces the most sensible outcome. The authors of SHAP have tested the explainers' consistency with human intuition, assuming that:

  [...] good model explanations should be consistent with explanations from
  humans who understand that model.

Right, that makes sense. But how to evaluate the performances of different explainers in a more generic and scalable way, without the need for human intervention?

Some authors have recently attempted to come up with evaluation metrics. I found one approach to be interesting. The authors worked with the following intuition:

  [...] a feature-based local explanation has high quality only if it is
  difficult to change the prediction when the explanation features are
  kept intact. The quality of an explanation is quantified by the
  minimum amount of perturbation on features that are not in the
  explanation region in order to alter the prediction.

That makes sense, too. So I am planning to build up on that and focus on tabular data. The idea would be to, for each instance (row) in our test data:

Run the neural network and observe the predicted outcome.
Run the explainer and identify each feature's importance value.
Keep the features with the highest importance value intact and alter the remaining features.
Run the neural network again. If the predicted outcome hasn't changed, go back to step 3 and alter the features some more until a switch in the outcome.

Then, the explainer for which the necessary amount of alteration to obtain a change of outcome was the highest (aggregated across all instances), would be the best.

I was hoping to open a debate about quantitative techniques that can evaluate explainers. What do you think? Also, speaking of 'altering' the features, what would be the appropriate way to do so (when working with tabular data)? I'm open for comments and thoughts!

I would just like to add Microsoft's interpret to the list of explainers which uses boosting, and has some really cool graphics for explaining a model's behavior.


How to determine the most active time of a user from active directory logs I have user data login logs from Active directory data that lists when a user was logged in to a server. I am looking to find a deviation in the normal behavior of the user. For example if a user is usually active from say 9am to 5pm, if there is some activity outside this, it should be detected.
I am unsure if the min(first logon) and the max(last logon) should be considered.
Any input will be appreciated.

As always, start by looking at the data.  Pick a user, and look at their typical login times.  Maybe plot a histogram to show the distribution of time of day of their login.  Pick another user, and look at them.  Pick a few more.  Do you see any patterns?  Do they have a common shape?  Do users fall into a few clusters?  Try to understand the data well.

Then, once you have a feeling for what normal behavior tends to look like, and how it varies from user to user, you can start to think about how to build a model that captures this random behavior and helps you distinguish normal vs abnormal behavior (perhaps by some kind of anomaly detection).

But as always: start by visualizing the data.  Asking how to approach the problem or what machine learning method to use is premature, until you have a better understanding of the data.


alternative to one hot encoding Hi I am working on a movie recommendation system and I have like 80.000 movies in my data set. The problem is there are like thousands of directors which is a problem when I represent a movie as a vector. One of the features of a movie is its director and I do not want my movie vector to be like 5000 in length. One hot encoding is advised everywhere but I want to do something else that will make my movie vectors small in length. I heard integer encoding but I have never seen an example of it nor it is advised. One thing that comes to my mind is to create a small interval like from 0 to 5 and let's say I have 6000 directors and I will assing first director to 0 and the next director will be assigned to 0+6/6000 and third one will be assigned to 0+2×6/6000 etc. However, I have a lot of doubts. what can you suggest? I am a rookie and not well-versed on the topic.

There are many other options instead of doing one-hot encoding. You can do target encoding, count encoding and many others. Here is a nice library for encoding categorical variables. Explore all the operations in here. Here's the link.
Target encoding would be my bet.


what is the interest of TimeDistributed after an LSTM layer? I've already seen several similair questions but I did not understand anything,
what is the interest of TimeDistributed?
why we need to insert a TimeDistributed layer after LSTM to establish the time series prediction

Ok let's say you have an LSTM() layer with return_sequences = True set. That means each LSTM cell in it is outputting its value. The output of the layer is therefore a sequence of outputs, not just the final one. That means the output is a processed time series, with sequential information in it.

You want Dense() layers to take this information and use it to process the signal further. In particular, you want some Dense architecture to process each of these sequential outputs one by one. By using the TimeDistributed() wrapper it's like if you're iterating the same Dense() architecture on this sequential output.

I typically use TimeDistributed(Dense()) at the bottom of a seq2seq Decoder, in order produce the final output sequence.


Python encode piano notes in 88 bits - int too large to convert to C long The piano keyboard has 88 keys:

I would like to encode one note or one chord in 88 bit array. I do this, for example:
A4 is 0000000000000000000000000000000000000000000000000000000001000000000000000000000000000000

Next I convert this number to decimal. When I have chord the final number is really big. The problem is when I put this big number into array from NumPy I gets:

note_data = np.array(note_data, dtype=np.int64)
OverflowError: Python int too large to convert to C long

My point is to feed recurrent neural netowrk this kind of data or maybe is better representation on music notes into numbers?

EDIT:

Another example of encode C3 note:

C3 = np.array([0000000000000000000000000001000000000000000000000000000000000000000000000000000000000000], dtype=int)

and error sitll the same:

OverflowError: Python int too large to convert to C long

It's possible to reduce this error?

That seems like a bad way to encode the information. A C chord and a C7 chord would be very different numbers even though they are similar conceptually.

I would take your idea, but instead of smashing the information into a single int, make your input a binary vector of length 88 created in the way you describe. In this way, a C and C7 will have similar inputs, differing in only one location of the vector.

As a simplified one octave example going from C to B:

C : [1,0,0,0,1,0,0,1,0,0,0,0]
C7: [1,0,0,0,1,0,0,1,0,0,1,0]

This kind of encoding is making use of “dummy variables”. It is like the “Bag of Words” encoding used a lot in Natural Language Processing.


Tools to preprocess a big data for dashboards? I have a complex dataset with more than 16M rows coming from pharmaceutical industry. Regarding the data, it is saved in a sql server with multiple (more than 400) relational tables. Data got several levels of hierachies like province, city, postal code, person, and antigens measures, etc. I would like to create many dashboards in order to observe the changes & trends happening. I can use Pentaho, R (shiny) or Tableau for this purpose. But the problem is data is so huge, and it take so long to process it with dashboard softwares. I have a choice of making cube and connect it to dashboard. My question here is whether there are  any other solutions that I can use instead of making a cube? I don't want to go through the hassle of making & maintaining a cube. I would like to use a software where I specify relationships between tables, so the aggregation/amalgamation happens smoothly and output processed tables that can connect  to dashboards. I hear Alteryx is one software that can do it for you (I haven't tried it myself, and it is an expensive one!). I understand this task needs two or more softwares/tools. Please share your input & experience. Please mention what tools do you use, size of your data, and how fast/efficient is the entire system, and other necessary details.

We have dashboards that show information about some processes, which have billions of rows in the database. It's not queried directly though, but instead from pre-aggregated data.

We have automated scripts running in database that populate the aggregated data tables specifically for dashboards. In most extreme cases the raw data is so large and comes in at such a high pace that there's a two-tier aggregation in place. First tier will de-normalize the data and 2nd tier will do the actual sum/count type of aggregation.

So you don't need two tools since you can do aggregation purely in database.

edit (answering question from comment): Our dashboards are in Tableau. Data is all in PostgreSQL databases. Automation is done with unix cronjob executing a database function. The function in turn queries which tasks it has to run and runs them. There is a few moving pieces to the whole system, but the architecture isn't too complicated. Write down your ideas and go talk to some architects / engineers - they will know the best way to approach this.


What algorithm could be used to fuzzy merge multiple datasets? Problem Description

I have several tables that are related but do not share any unique key. I've come across this problem several times with customer data in separate source systems that needs to be compared together.

Lets say my data is multiple tables, Table A through Z.

There may be columns where I'm 100% certain on a match. For example table A and B have the column tax ID which is a certain match joining A to B. Both A and B alone cannot match to C, but using columns from A and B can make a certain match to C.

There are columns that are likely matches like matching first name, last name, birthday, etc "John Smith", but may result in false positives. Additional fuzzy matches may increase likelihood of match at different magnitudes.

I would like to feed an ML engine a set of tables that have been verified to be related to the same person, and see what rules it can come up with for matching the tables. I may get only a subset of tables at a time, and I want to be able to match what is possible and await more data. What algorithm could be used?

Thoughts

I know decision trees are great for this type of problem but probably many of the fuzzy checks I could make could better be expressed by some certainty magnitude. I've also never used a decision tree across a set of datasets where I could only be dealing with a subset.
I know Naive Bayes is often used for fuzzy matching but I also want to combine logic for 100% matches
Since this is a case where there can be limited full information via manually tracking down data, evolutionary algorithms could potentially work

The first thing is to define a distance metric to say how close potential keys are. If the data is text, Levenshtein distance  is a common distance metric. If the data is numeric, then Euclidean or Manhattan distance could work. If the data is geographic, then Haversine is a good choice.

After defining a distance, a threshold needs to be set to for merging. The threshold could be picked by a person or learned by a machine learning algorithm. Typically, there is not enough data to train a machine learning algorithm. It is often good enough to just pick a threshold based on domain knowledge.

If you can use R, there is the fuzzyjoin package.

Given that you have to search the cartesian product of the space, evolutionary algorithms will take too much time.


Extract data from facebook I am learning about social media analysis. I am aware that we can extract the data from twitter using hashtags and API. Ex; If I use #covid19, I will get all tweets that contain this hashtag for the duration that I specify.
Similarly, let's say I visit a public Facebook page https://www.facebook.com/A123
Will I be able to extract only the posts which contain the term covid in their text (from all of their posts) using my API credentials?
Is it possible? I don't like to extract all the posts from the page A123. However, I only wish to extract only the posts which contain the term covid in their text (from all of their posts)
Can you help me on how can it be done?
Any tutorial/tools available to extract this info?

Yes, its called graph-api.
https://developers.facebook.com/docs/graph-api/reference/v7.0/page/feed
You should be aware that scraping data from facebook not using this api its
violation of the regulations of fb laws and could you be ended with ban. So be careful.


Categorization of approaches to deal with imbalanced classes What is the best way to categorize the approaches which have been developed to deal with imbalance class problem?

This article categorizes them into:

Preprocessing: includes oversampling, undersampling and hybrid
methods,
Cost-sensitive learning: includes direct methods and
meta-learning which the latter further divides into thresholding and
sampling,
Ensemble techniques: includes cost-sensitive ensembles and data
preprocessing in conjunction with ensemble learning.

The second classification:

Data Pre-processing: includes distribution change and weighting the
data space. One-class learning is considered as distribution change.
Special-purpose Learning Methods
Prediction Post-processing: includes threshold method and cost-sensitive post-processing
Hybrid Methods:

The third article:

Data-level methods
Algorithm-level methods
Hybrid methods

The last classification also considers output adjustment as an independent approach.

Thanks in advance.

The way I see it all three categorizations agree in many things. For example, all three have a category for pre-processing steps.

I would tend to mostly agree on the third categorization as its more generic and encompasses more things.

The data-level category includes any pre-processing steps dealing with class imbalance (e.g. over/under sampling).
The algorithm-level could be considered to include the second categories of the first two articles. Any change to the algorithm that deals with class imbalance would go here (e.g. class weighting).
Finally, a hybrid category for combining the two.

The only thing missing from the first two articles are the post-processing steps, which to be honest, aren't used in practice as often as the other.


How to plot train test error for classification models like Support Vector Classification(SVC) How to plot train test error for classification models like Support Vector Classification(SVC). I am using SVC from sklearn module, not able to get train and test errors to plot

Well you haven't defined what "error" means, so I'll just assume that you want the log loss.

First, you need to create your SVC object telling it you'll want probability estimations:

model = sklearn.svm.SVC(probability=True)

Then you can compute the log loss given said estimation:

probs = model.predict_proba(x)
loss = sklearn.metrics.log_loss(y_true=y, y_pred=probs)


Why encode pitch as one-hot encoding instead of ordinal encoder? looking at the state-of-the-art publications on deep learning for synthesizing audio one can see that they always resort to encoding pitch as a one-hot vector. I'm curious what the advantage is on doing this, since pitch (frequency) fits naturally as a scalar value and one could reduce network dimensionality by encoding it as such.

The papers I've studied are:

GANSynth: Adversarial Neural Audio Synthesis

Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders

Because average of encoded pitches is not the encoded class.

Look at this example:

LabelEncoder can turn [dog,cat,dog,mouse,cat] into [1,2,1,3,2], but then the imposed ordinality means that the average of dog and mouse is cat. Still there are algorithms like decision trees and random forests that can work with categorical variables just fine and LabelEncoder can be used to store values using less disk space.


Do Activation Functions map to Higher Dimensions?: I just started learning tensorflow and I have a question regarding activation functions used in neural networks, I watched a 3b1b video a while ago and it seems it squished the value into an interval like sigmoid does so by squishing it between 0 and 1 so we could make more concrete comparisons however while watching a tutorial today the instructor said that it projects the data points into higher dimensional spaces. I didn't really get how that's the case as it seems it's being converted to a scalar. Is there an interpretation/example for the latter claim?
This is the timestamped URL he talks about it during the couple of minutes following this.

We explain AI with intuition rather than maths most of the time, so everyone has its own explanation and representation of things, here is how I would explain activation functions (I'll try to make is clearer instead of adding another version to the already 2 you know):
Just in case you don't know what a basis is, you should have a look at Wikipedia since it is the core concept of your problem (to my eyes at least).

You need to consider the whole input values, not just their values. All input values are reals, so their dimension is 1, yet if we consider them all together, their dimension is N (number of inputs). And the basis of this space is $(Input1, Input2, ..., InputN)$.
Neurons are made of 2 parts :

linear combination of the inputs.
activation function

Let's imagine we use neurons that are only a linear combination of inputs $(Output = WA + B)$. Then we project our initial points into a space that has exactly the same basis as the input space (since all outputs are linear combination of the input if we forget about biases). So this action is just a remapping of the inputs into the same space, but with a different basis. This may help if your problem is linearly separable, but doesn't suffice if your problem is not That's why we use activation function.
If we consider neurons with an activation function now. The particularity of activation functions is that they are non-linear. So the use of activation functions maps your inputs into a different space, basis needs to be different since all outputs are not anymore linear combination of the inputs. So this time the output space is different from the initial one.
The way I see it is that activation function do not generate information from nothing, but they allow to remap the available information into a higher dimension space where the problem is easier to solve.
Hope this helps, feel free to ask if you have any remaining question.


How to optimize client's portafolio with analytical models? I have a model in which we want to optimize the probability of an outcome depending on a election of some product (a personalized product for every client amongst three posibilities). The product is the second product elected after the same client has a first.

I have never done an optimization of a probability with analytical models before so I am a little blind in the topic.

What I have for now is this:

I have a dataset with clients with ONE product, the optimization is intended for THEM, they have a first product and I want to model which is the next best product for every one of them - Application dataset.
I have another dataset with clients with TWO OR MORE products, the training could be done with them (I know which product was purchased first) - Training dataset.
Both datasets have exogenous variables which could help modelling: i.e. Age, gender, etc.

I have thought these posible ways to achieve the objective:

Split the training dataset depending on the first purchased product, then obtain a personalized probability on the basis the first product is purchased, the "sign" and coefficient of the second possible products would tell me which is the next best one for every first product (subset).
Model the dataset without separating anything, read the results looking for optimal combinations.
Model with interactions first-second products to look for the best combinations.
Model with white-box techniques (decision tres, logistic regression), so the combinations are traceable.
Something different.

Your problem could be framed as a recommendation system. Given historical information about preferences, what is the best item to recommend next to a user?


is this a classification or clustering problem? Suppose, I am building a hotel recommendation system, that learns user profile based on his/her interaction with the system.

I have two classes, "Like" and "Dislike". For example, a user likes 5 "5-star" hotels, likes 3 "1-star" hotels and dislikes 10 "1-star" hotels. Now, a Naive Bayes Classifier will put a new 5 star hotel in the "Like" class and a new 1 star hotel in "Dislike" class. Now, what I am trying to achieve is a bit different, I don't want to classify, instead I want to extract features/attributes more closely related with the "Like" class. For example, I need "5-star" attribute from the "Like" class.

Now, obviously, I have many more attributes, than the hotel star rating. What I want to know is that, can Naive Bayes be modified to extract features from a class. Or are there any specialized algorithms that are more suited for this task?

I would have data in this form

user_id | hotel_id | hotel_rating | hotel_attr1 | .... | hotel_attr(n)| preference

1       1        5 star        some value   some value    like

1       2        1 star        some value   some value    dislike

Another thing, If I want to extract features such as "5 star" and "1 star" from a class, Is the data formatted above correct? or instead of hotel_rating as the feature, I need 5_star and 1_star as the features, instead of them being values of the hotel_rating feature?

To me, what you described, sounds very much like collaborative filtering. In collaborative filtering, the features of the user(usually individual preferences) and features of the item(e.g. type and attribute of hotels) are trained collaboratively, gradually improving each other through iterations and converge to an optimal. Take a look at Andrew Ng's lecture, you can get a very quick idea of how it works.

In my opinion, collaborative filtering is a relatively basic technique for providing recommendations. Decision tree based models are more generally used in real life recommendations by big companies.


Does the choice of error function impact the model parametrs? Suppose I have trained a multi variate linear regression model on a particular training set, and the model parameters $\theta=[\theta_1,\theta_2,\ldots, \theta_n]$ were determined by minimizing a cost function, say MSE (Mean Squared Error).

Now, if for some reason I decide to change my choice of error function from MSE to MAE (Mean Absolute Error), will this change my model parameter $\theta$ or will it remain the same?

Yes, it will impact because when you change the loss function, the numerical value of the loss function will change. So, this will change gradient values of the parameters during the back propagation. Therefore, change in loss function will impact the parameters.


Jupyterlab Inline Interactive plot I am trying to make my inline plots in jupyterlab interactive. So far, I have tried a suggestion as pointed out here, among others:

https://stackoverflow.com/questions/50149562/jupyterlab-interactive-plot

# %matplotlib notebook - does not work : Javascript Error: IPython is not defined
# %matplotlib widget - works, but plots are overwritten

The widget magic works in making the plots interactive, but unfortunately, my plots are overwritten. Subsequent cells render plots on top of the output of cell 1 as below:

plt.scatter(trainData['x'], trainData['y'], color='Red', s=0.5)
plt.show()

plt.hist(trainData['x'])
plt.xlabel("Values of X")
plt.ylabel("Frequency")
plt.show()

plt.hist(trainData['y'])
plt.xlabel("Values of Y")
plt.ylabel("Frequency")

This does not look right, have I missed an option? I am looking for an IPython magic that can help with having several inline plots (after each cell)?

I am using jupyterlab v 0.35.4, run from Anaconda navigator.

You either need to create a new figure before plotting by plt.figure(), or you need to use subplots. Otherwise, plt.show() will act on the last-created figure.


Clipping threshold of softmax layer I recently came across a paper on using (rather simple version of) LSTM for sentiment classification, and it describes its network settings as:

  We randomize the parameters with uniform distribution U(-0.003,
  0.003), set the clipping threshold of softmax layer as 200 and set learning rate as 0.01.

I am trying to reproduce their results with my code written in Tensorflow. I have to say I find it a bit confusing by the meaning of "set the clipping threshold of softmax layer as 200". 200???

Can someone explain this to me please so I know how to implement this using Tensorflow? Thanks so much!!!

Update, ok so I found the their code and below is how they use this clipping threshold of softmax layer:

for(int k = 0; k < softmax.outputG.length; k++)
    softmax.outputG[k] = 0.0;
softmax.outputG[goldPol] = 1.0 / softmax.output[goldPol];

// if ||g|| >= threshold, then g <- g * threshold / ||g||
if(Math.abs(softmax.outputG[goldPol]) > clippingThreshold)
{
    if(softmax.outputG[goldPol] > 0)
        softmax.outputG[goldPol] =  clippingThreshold;
    else
        softmax.outputG[goldPol] =  -1.0 * clippingThreshold;
}

I still can't say I fully understand the use of this threshold..

I think what they were doing is just gradient clipping. It keeps the gradient of softmax layer between [-200,200]


NLP: Getting the top 5 or top 10 predictions I am working on a social networking application and I have to make its news feed better. For example: If someone searches for 'suggest me some good books',  it should yield some names.

Now, I have used the Infersent algo (to begin with) in order for my model to be able to answer questions.

I am getting only the best output that my model could predict viz., 'Alchemist'.

I want at least 4 or 5 other outputs, other words, the top five predictions.

I know that Xgboost has the ability to do this activity in some sorts, but I am not sure how I should use that in my problem.

Any heads up?

My apologies, I cannot share any code but I would really appreciate ideas and suggestions.

Thank you,

There's predict_proba method in xgboost. You get probabilites for each class, sort them and take top 5.


Support Vector Classification kernels ‘linear’, ‘poly’, ‘rbf’ has all same score I build a classification model based on SVM and getting same results after running different kernels. Can you please let me know if is mistake ? also recall for all are identical. Thank you for help.

Adding the location for the notebook and data.
SVC repo with the notebook and data

You will love the answer to this one...

Take a look at your code and notice that you are calling the scoring function and each time you are passing in the exact same values i.e. they are all spitting out the lin_svc.score().  Try interweaving the four scoring calls below the four respective fit calls and you should see the desired variation in the result.

# we create an instance of SVM and fit out data. We do not scale our
# data since we want to plot the support vectors
#rbf is for gaussian
C = 1.0  # SVM regularization parameter

svc = svm.SVC(kernel='linear', C=C).fit(X_train, y_train)
print svc.score(X_test, y_test)

rbf_svc = svm.SVC(kernel='rbf', gamma=0.7, C=C).fit(X_train, y_train)
print rbf_svc.score(X_test, y_test)

poly_svc = svm.SVC(kernel='poly', degree=3, C=C).fit(X_train, y_train)
print poly_svc.score(X_test, y_test)

lin_svc = svm.LinearSVC(C=C).fit(X_train, y_train)
print lin_svc.score(X_test, y_test)

Similarly, you are doing the same thing below also.

Hope this helps!


How to measure execution time on distributed system I'm planning to run experiments with large datasets on distributed system in order to evaluate efficiency gains in comparison with previous proposals.

I have limited number of machines nearly ten machines having 200 GB of free space on hard disk on each. On the contrary, I wished to perform experiments on more than available nodes in order to measure scalability, more precisely. Since I don't have any, I thought about using a commodity cluster. However, I'm not sure about the policies of usage, and I need to reliably measure execution times.

Are there commodity services which will grant me that only my application would be running at a given time?

I have some thoughts about your question. I hope it may help you to solve your problem.

  I'm planning to run some experiments with very large data sets, and I'd like to distribute the computation.

In one of my posts, I have done research on topic of evaluation methods of Data Science. With Learning Curve, you can evaluate your experiments learning ability. To talk a bit more, you will fix commodity configuration, and then will run same experiment on the same number of machines with different size of data set you have (from starting from small chunk in size, incrementally increase the size until you reach the whole data set).

To point on, you should avoid having power distribution for result of performance test being run with different size of data sets. To avoid, you should carefully choose step size (step size = amount of increments).

  I have about ten machines available, each with 200 GB of free space on hard disk. However, I would like to perform experiments on a greater number of nodes, to measure scalability more precisely.

For this type question, I have intuitively searched and read materials; afterwards, published as a blog post. At the end of the post, I have briefly talked about how to test your hypothesis on real complex system. If you let me, I want to briefly talk about;

First of all, base requirement should be formed in order to run data set as a whole. The minimum requirement will build your baseline evaluation score which is calculated with one/combination of evaluation metrics you have chosen, or with one/combination of methods being used to calculate Running Time = Computation complexity + Communication cost + Synchronization cost.

After those steps, with an evaluation strategy, add new elements, e.g. new node, to the system you have doing scalability test; meanwhile, for each addition, measure performance w.r.t new system configuration.

Just to note, evaluation strategy must be planned along with considerations of default behavior of parallel and distributed systems. For example, what I mean by behavior is that adding just more cores will, after some point, automatically drop performance of the system not due to your algorithm characteristics. It is because more cores need more RAMs, more hard driver, or etc. In other words, there is a N-Way relationship between hardware components. As a second example, adding more nodes to the distributed system will punished you with more communication and synchronization costs.

As a last step, you will sketch two different graphs with your evaluation results via data analysis program or language (As a recommendation, use GNU Plot or R programming language). Print out and put those results at your desktop, and start to examine them, carefully. According to your investigation, modify/erase + rebuild evaluation strategy and re-do the performance test.

  Are there commodity services which would grant me that only my application would be running at a given time? Has anyone used such services yet?

I have no much experiment on commodity services, but I can easily say whether it grants or not depends on your configuration of services. If you configured say Hadoop to your node as an only service, Hadoop will grant your code will be only running at any time.


Growth function of a 6-dimensional linear classifier In our course, we are dealing with a d-dimensional classification problem
($\chi = \mathbb{R}^{d}$ as our input space, and $y = \{-1,+1\}$). Our hypothesis class $H$ consists of all hypotheses of the following form:
$h(x) = a\cdot \text{sign}(x_i - b)$, where $i = \{1,2,\dots,d\}$, $a \in \{-1,+1\}$, and $b\in\mathbb{R}$.

We have already shown that the growth function $m_{H}(3) = 2^3$ for $d=2$ by showing all 8 possible dichotomies for three chosen points. We further know that for a $d$-dimensional linear perceptron, the VC-dimension is always equal to $d+1$.

We know want to show that for $d=6$, $m_{H}(7) < 2^7$, i.e. that the VC-dimension of our hypothesis class is lower than 7.

Could you help us out with this?
Thanks a lot!

Suppose we have $n$ points in $\chi$. We have $d$ choices of $i$, two choices of $a$, and at most $n+1$ effective choices of $b$ so the number of possible combinations of outputs on those $n$ points is at most $2d(n+1)$. I'm sure you can take it from here.


Algorithm Suggestion For a Specific Problem I'm working on a problem where in I have some data sets about some power generating units. Each of these units have been activated to run in the past and while activation, some units went into some issues. I now have all these data and I would like to come up with some sort of Ranking for these generating units. The criteria for ranking would be pretty simple to start with. They are:

Maximum number of times a particular generating unit was activated
How many times did the generating unit ran into problems during activation

Later on I would expand on this ranking algorithm by adding more criteria. I will be using Apache Spark MLIB library and I can already see that there are quite a few algorithms already in place.

http://spark.apache.org/docs/latest/mllib-guide.html

I'm just not sure which algorithm would fit my purpose. Any suggestions?

You can use a clustering algorithm such as k-means to divide the generators into groups.  You never know what kind of groups you'll get until you try it.  Try and assess the character of each group of generators as you increase the number of clusters.  At some point you should find a meaningful division of generators.  The inputs to your k-means algorithm will be the criteria you mentioned in your post: the number of times it was activated, the number of activation problems, and so forth.  When you are finished, the group a generator belongs to is its ranking.  This method will not generate a ranking of 1-1000 if you have 1000 generators.  Rather it will give you, for example with k=3: a group of 243 outstanding generators, 320 average generators, and 446 terrible generators.


PySpark Crossvalidation error I want to do a very simple cross validation using LogisticRegression.
Here is my code:
logreg = LogisticRegression(labelCol = "churn", featuresCol = "features")

pipeline = Pipeline(stages = [logreg])
paramGrid = ParamGridBuilder().addGrid(logreg.regParam, [.1, .01]).build()

crossval = CrossValidator(
    estimator = pipeline,
    estimatorParamMaps = paramGrid,
    evaluator = BinaryClassificationEvaluator(),
    numFolds = 2)

bestLogReg = crossval.fit(df_train)

When I run this, I get the following error on bestLogReg = crossval.fit(df_train):
IllegalArgumentException: label does not exist. Available: features, churn, CrossValidator_764038c00edc_rand, rawPrediction, probability, prediction
Here is my df_train dataset's schema:
root
 |-- features: vector (nullable = true)
 |-- churn: integer (nullable = true)

I have fit this to a LogisticRegression before and it predicts fine.
Can you help me figure out what I did wrong?

For some reason in cross validation we also need to set the label column of the evaluator (even tho it's already set for the estimator. So all you need to do is change BinaryClassificationEvaluator() into BinaryClassificationEvaluator().setLabelCol("churn") where "churn" is the name of your target variable.


How to calculate a specific time complexity of inverse calculation of matrix? I am a green-hand in calculating the time complexity. Given a calculation as follows:

\begin{equation}
\mathbf{x}=\mathbf{A^T}(\mathbf{AA^T}+\lambda\mathbf{I}_n)^{-1}\mathbf{b}
\end{equation}
where $\mathbf{A}\in R^{n\times p}$, $\mathbf{I}_n\in R^{n\times n}$, $\mathbf{b}\in R^{n}$ and $\lambda\in R$.

Could some one kindly give me the complexity of the above formula in details? Thanks a lot.

If you need complexity of this calculation in big O notation - it is: $$O(n^3)$$
Why? Because matrix inverse needs $$O(n^3)$$ operations, and it is biggest complexity here.

Multiplication matrix by its transpose is $$O(n^2p)$$ (Because for computing every value in the resulting matrix of size NxN you have to compute p multiplications).
Matrix transpose is $$O(np)$$ But you can ignore any complexities lesser than $$O(n^3)$$ (like matrix sum, transpose, or multiplying by constant) because their order of growth much lesser than order of growth of inverse operation.

Computational_complexity_of_mathematical_operations


Predicting a word using Word2vec model Given a sentence:
"When I open the ?? door it starts heating automatically"
I would like to get the list of possible words in ?? with a probability.
The basic concept used in word2vec model is to "predict" a word given surrounding context.
Once the model is build, what is the right context vectors operation to perform my prediction task on new sentences?
Is it simply a linear sum?
model.most_similar(positive=['When','I','open','the','door','it','starts' ,
                   'heating','automatically'])

Word2vec works in two models CBOW and skip-gram. Let's take CBOW model, as your question goes in the same way that predict the target word, given the surrounding words.

Fundamentally, the model develops input and output weight matrices, which depends upon the input context words and output target word with the help of a hidden layer. Thus back-propagation is used to update the weights when the error difference between predicted output vector and the current output matrix.

Basically speaking, predicting the target word from given context words is used as an equation to obtain the optimal weight matrix for the given data.

To answer the second part, it seems a bit complex than just a linear sum.

Obtain all the word vectors of context words
Average them to find out the hidden layer vector h of size Nx1
Obtain the output matrix syn1(word2vec.c or gensim) which is of size VxN
Multiply syn1 by h, the resulting vector will be z with size Vx1
Compute the probability vector y = softmax(z) with size Vx1, where the highest probability denotes the one-hot representation of the target word in vocabulary.
V denotes size of vocabulary and N denotes size of embedding vector.

Source : http://cs224d.stanford.edu/lecture_notes/LectureNotes1.pdf

Update: Long short term memory models are currently doing a great work in predicting the next words. seq2seq models are explained in tensorflow tutorial. There is also a blog post about text generation.


The Area Under an ROC Curve (AUC) vs Confusion Matrix for classifier evaluation? When should I use The Area Under an ROC Curve (AUC) or the Confusion Matrix for classifier evaluation? The clasifier evaluation is for example the prediction of customers for possible future sales.

A confusion matrix can be used to measure the performance of a particular classifier with a fixed threshold. Given a set of input cases, the classifier scores each one, and score above the threshold are labelled Class 1 and scores below the threshold are labelled Class 2.

The ROC curve, on the other hand, examines the performance of a classifier without fixing the threshold. Given a set of input cases, the classifier scores each one. The ROC curve is then generated by testing every possible threshold and plotting each result as a point on the curve.

The ROC curve is useful when you want to test your classifier over a range of sensitivities/specificities. This may or may not be a desirable thing to do. Perhaps you want very high sensitivity and don't care much about specificity - in this case, the AUC metric will be less desirable, because it will take into account thresholds with high specificity. The confusion matrix, on the other hand, could be generated with a fixed threshold known to yield high sensitivity, and would only be evaluated for that particular threshold.

A confusion matrix evaluates one particular classifier with a fixed threshold, while the AUC evaluates that classifier over all possible thresholds.


Model that predicts probability of correctness of another model Problem:
Given a neural network for image classification with $1000$ classes, the objective is to create another model which will output the probability of the neural network giving the correct prediction for a specific input image.
Thoughts:
My ideas so far have been:

Creating a convolutional network and training it with raw images together with a label which indicates if the first NN predicted it correct or not.
Creating a fully connected network and training it with outputs of hidden layers/features of the first NN instead of the raw image, together with the labels as before.
Creating a fully connected network and training it with the top-k outputs of the softmax layer of the first NN together with the labels.

The first method yielded an accuracy of $0.51$, the second $0.58$ and the last one $0.79$.
Can you suggest another method (or a modification of one from the above) which can achieve greater accuracy?

This is a problem known as quality estimation (QE) in the application of Machine Translation (there's a regular Shared Task about it). The goal is to train a model to predict the quality of the output of a ML system.
There is an important theoretical obstacle to this kind of task: if such a QE model was able to predict perfectly, then in theory it would be possible to create a perfect model for the original task by trying many different models/parameters until the QE system predicts a high level of quality. What this means practically is that a QE model can only be as good as the system it's trying to estimate since it relies mostly on the same information, otherwise the system is clearly sub-optimal. Essentially the best the QE system can do is to determine how hard it is to correctly predict a particular input image, but it cannot predict with any certainty whether the actual prediction is correct or not.


Uploading images folder from my system into Google Colab I want to train a deep learning model on a dataset containing around 3000 images. Since the dataset is huge, I want to use Google colab since it's GPU supported. How do I upload this full image folder into my notebook and use it?

Method 1 :

zip the file
Upload the zipped file, there is an Upload button under the Files Section.
Unzip it using the command on colab :
!unzip level_1_test.zip

Method 2 :

upload the zip file to the google drive account.
The only difference is in step 2 where in place of the GUI upload option you can run the google code_snippets to upload download your zip file from the google drive to Colab account .
Unzip it using the command on colab :
!unzip level_1_test.zip


Probability calibration is worsening my model performance I'm using RandomForest and XGBoost for binary classification, and my task is to predict probabilities for each class. Since tree-based models are bad with outputting usable probabilities, i imported the sklearn.calibration CalibratedClassifierCV, trained RF on 40k, then trained CCV with a separate 10k samples ( with cv="prefit" option ), my metric ( Area Under ROC ) is showing a huge drop in performance. Is it normal for probability calibration to alter the base estimator's behavior?

Edit : Since i'm minimizing logloss on my XGBClassifier, the output probabilities aren't that bad compared to RF's outputs.

The probability calibration is just stacking a logistic or isotonic regression on top of the base classifier.  The default is logistic, and since the sigmoid is a strictly increasing function, the rank-ordering of samples will be unaffected, and so AUC should not change at all.

(With isotonic regression, it's actually piecewise-constant, so within a span where the function is constant, all samples will have their scores made equal, and so your ROC curve will become more coarse, which would affect the AUC; but these effects ought to be small, as long as the isotonic fit produces sufficiently many/short constant segments.)

Also, gradient boosting like XGBoost produces scores biased toward the extremes, not away from them like random forests, so the logistic calibration is unlikely to work out well.


Why real-world output of my classifier has similar label ratio to training data? I trained a neural network on balanced dataset, and it has good accuracy ~85%. But in real world positives appear in about 10% of the cases or less. When I test network on set with real world distribution it seems to assign more positive labels than needed tending to balanced proportion as in training set.
What can be the reason of such behavior and what should I do solve it?
I'm using Keras and combination of LSTM and CNN layers.

What can be the reason of such behavior?

A classifier only tries to capture the features-label relationship as accurate as it can, it does not learn nor does it guarantee that the ratio of predicted labels to be close to the true ratio. However, if sampled classes (even balanced) are good representatives of true class regions and classifier finds good decision boundaries, the closeness of ratios will happen naturally. Therefore, the explanation is that the sampled negative class is not a good representative of its true occupied regions, meaning some regions are not sampled well, and/or classifier is finding bad decision boundaries. Here is a visual illustration (drawn by myself):

In the case of good training set, predictions resemble the real-world ratio (2:1), even though model is trained on a balanced set (1:1). In the case of bad training set and/or bad decision boundary, predictions are completely incompatible (1:2) with the real-world ratio (2:1).

  What should I do to solve it?

If the problem is related to bad negative representatives

Use more negative samples in the training set, i.e. imbalanced training (you can nullify the effect of imbalanced samples with class weights), or
Train with balanced classes but change the decision threshold (a post-training solution). That is, instead of assigning instances with output > 0.5 to positive class, use a harder-to-pass threshold like output > 0.8 to decrease the number of positive predictions.

If the problem is related to classifier performance, we should come up with a better classifier which is an open-ended endeavor.

However, in my opinion, you should not select models based on the ratio of positive predictions. You should ecide based on a metric like macro-f1 (or any other one). Therefore, by using a validation set, a model that produces more positive samples and has a higher macro-f1 should be preferred over a model that produces less positives but has a lower macro-f1.

EDIT:

As @BenReiniger pointed out in another post, a hidden assumption here (specially in the sketch) is that classes are "clearly separable". This assumption becomes more justified in higher dimensions. As an example, dogs and cats are clearly separable based on their images (high dimensions) compared to their length (one dimension).


Gradient Descent or Normal Equation? Suppose you have a dataset with m = 50 examples and n = 15 features for each example. You want to use multivariate linear regression to fit the parameters theta to our data. Should you prefer gradient descent or the normal equation and why?

If you can, it is preferable to use the normal equation to estimate the coefficients for multivariate linear regression. Since the normal equation is a closed-form expression, it will be faster than gradient descent.

Given you have relatively few examples and features, inverting the matrix is not an issue.


Encoding categorical variables using likelihood estimation I am trying to understand how I can encode categorical variables using likelihood estimation, but have had little success so far.

Any suggestions would be greatly appreciated.

I was learning this topic too, and these are what I found:

This type of encoding is called likelihood encoding, impact coding or target coding
The idea is encoding your categorical variable with the use of target variable (continuous or categorical depending on the task). For example, if you have regression task, you can encode your categorical variable with the mean of the target. For every category, you calculate the corresponding mean of the target (among this category) and replace the value of a category with this mean.
If you have classification task, you calculate the relative frequency of your target with respect to every category value.
From a mathematical point of view, this encoding means a probability of your target, conditional on each category value.
If you do it in a simple way, how I described above, you will probably get a biased estimation. That's why in Kaggle community they usually use 2 levels of cross-validation. Read this comment by raddar here. The corresponding notebook is here.

The quote:

  It's taking mean value of y. But not plain mean, but in cross-validation within cross-validation way;

  Let's say we have 20-fold cross validation. we need somehow to calculate mean value of the feature for #1 fold using information from #2-#20 folds only.

  So, you take #2-#20 folds, create another cross validation set within it (i did 10-fold). calculate means for every leave-one-out fold (in the end you get 10 means). You average these 10 means and apply that vector for your primary #1 validation set. Repeat that for remaining 19 folds.

  It is tough to explain, hard to understand and to master :) But if done correctly it can bring many benefits:)

Another implementation of this encoding is here.
In R library vtreat they have implementation of impact encoding. See this post.
In CatBoost library they have a lot of options for categorical variable encoding including target encoding.
There is no such encoding in sklearn yet.

UPDATE: There is a nice package for sklearn models and pipelines! https://github.com/scikit-learn-contrib/category_encoders


Missing at random vs missing not at random: What if it is both? (Does one imply the other?) My understanding is that:

Missing at random: Whether or not a variable's value is missing is dependent on the values of the other variables.
Missing not at random: When the propensity for a variable's value to be missing depends on the value.

But what about when the variables are correlated, as they often are?
To make things more concrete, let us consider an experiment where we are collecting data on temperature, humidity, and CO2, and let us suppose that there relationship between these is T = H = C.
Say that we are missing all CO2 variables below 50, because the sensor freezes.
In this case, it is
Missing at random: Because the propensity of CO2 to be missing is dependant on the value of temperature and humidity.
Missing not at random: Because all CO2 values below 50 are missing.
Since the variables are interlinked, missing at random => missing not at random.
Or have I made a mistake in my reasoning somewhere?

Missing at random (MAR) means the NA frequency of the variable is never depended on the value of the variable itself.
Therefore in your example the data would be Missing not at random (MNAR)!
Why is this distinction important?
Because when the data is MNAR we have to identify the relationship between missingnes and the value however if the data was truly MAR or MCAR we could ignore the NAs or impute them with simple methods like mean-imputation.
Correctly identifying wether data is MCAR,MAR or MNAR is the only way to correctly identify how to deal with it!
But how can something be MAR and not MNAR?
Imagine you are asking senior citizens their birth month but also measure whether they have Alzheimers or similar memory impairments.
Birth month would likely be MAR as the fact whether it is missing correlates with the Alzheimer variable yet I have no information about the actual birth month from this fact.
I can predict from other variables whether the data is not missing but not what the actual value of it is!


Do I need to standardize my one hot encoded labels? I'm trying to do a simple softmax regression where I have features  (2 columns) and a one hot encoded vector of labels (two categories: left = 1  and Right = 0). Do I need to standardize just the vector of features or also the vector of labels? when I do that all my zeros and ones transform in different numbers and also I don't know how to identify who is the Left or the Right category. I'm using mxnet and gluon.
Here is how I standardize:
labels = (labels - labels.mean()) / (labels.max() - labels.min())

labels before standardization:
[0. 1. 1. 1. 1. 1.

labels after standardization:
[-0.5633803   0.43661973  0.43661973  0.43661973  0.43661973  0.43661973 ...

How can I  after identify (with strings) if my prediction is actually giving me Left or Right?

NO, you do not standardize labels
The purpose of standardization is to bring features with disparate ranges into a standard range. When the data is not standardized, features with large numerical values will tend to have a larger influence (weight) than those that are smaller numerically.

Consider the Automobile Data Set (https://archive.ics.uci.edu/ml/datasets/automobile)  from UC Irvine. Among the features, it has height, lenght, width, weight, # cylinders, along with a number of other label and numerical features. Height ranges from from 47.8 to 59.8 and weight ranges from 1488 to 4066. You'll want to standardize them, but not label features such as body-style or engine-type.


Is Oracle 11g is able to ingest big data? What is the sizing limitation for Oracle 11g? Can I use it as big data platform? From the following link I understand that there is no limitation for the records, only the columns are bounded to 1000. So if my big data has less than 1000 per table but contains many records can I use Oracle 11g as big data platform?

There is no reason why you cannot use Oracle 11g to create large databases. The 1000-column limit is a theoretical one, not a practical one imo. The theoretical database size limit is in petabytes. You should question why you would want to put that much data in a single database and be prepared to pay for the infrastructure to support it. This is a good summary link - http://awads.net/wp/2010/02/15/oracle-database-limits-you-may-not-know-about/

Having administered and tuned dbs of 50TB and more, the limits you should be more concerned with are for query performance. Ask Oracle about licensing as well before you proceed.


Solutions for big data preprecessing for feeding deep neural network models built with TensorFlow 2.0? Currently I am using Python, Numpy, pandas, scikit-learn to do data preprocessing (LabelEncoder, MinMaxScaler, fillna, etc.), and then feeding the processed data to DNN models built with Tensorflow 2.0. This input pipeline meets my needs when data is small enough to fit a PC's RAM.

Now I have some large datasets, more than 10GB, some are larger. I also plan to deploy the models in a production environment, which means there will be new data coming everyday. For DNN model training there is distributed strategy of tensorflow 2.0. But for data preprocessing obviously I cannot use pandas, scikitlearn on the large datasets with one PC. It seems to me I need to use a for-loop where I repeatedly fetch a small part of the data and use it for training?

I am wondering what do people typically use in either experiment or production environment for big data preprocessing?
Should I use Spark(PySpark) and Tensorflow input pipeline?

If most of your data and machine learning pipeline is in Python, then Dask project for Python is a very good candidate. It allows you to scale (certain types of) data-frame operations to datasets more than memory. The good thing about Dask is that it is also easy to scale down if required. Most of the code stays in Python, you don't pay the serialization overhead (which is paid in PySpark for Python -> JVM).


Visualizing results of a classification problem, excluding confusion matrices? A classification system I built is going to go into production soon (it'll be part of a larger dashboard), and I'm looking for ways to better visualize and convey to business folks the results of a classification.

Basically, given "old" data on which the model was trained, I predict the classes of "new" data, with the goal being to show whether the class distributions for the new data are statistically indifferent from the class distributions observed in the old data. So, if there are three classes A, B, and C in the old data, with proportions of 50%, 30%, and 20%, respectively, I compare the classification distribution for the new data with those original, observed proportions.

Outside of a confusion matrix (which I think is probably inappropriate for most dashboard users), how else can I effectively present these results? I was thinking of a bar chart like this:

Here are a couple of options. I prefer the stacked column chart, which could also be annotated with the actual proportions if needed.


DTW (Dynamic Time Warping) requires prior normalization? I'm trying DTW from mlpy, to check similarity between time series.

Should I normalize the series before processing them with DTW? Or is it somewhat tolerant and I can use the series as they are?

All time series stored in a Pandas Dataframe, each in one column. Size is less than 10k points.

I am glad you asked ;-)

In 99% of cases, you must z-normalize.

Want to know why? I wrote a tutorial on this, page 46
http://www.cs.unm.edu/~mueen/DTW.pdf


How do you think about neural networks and ways to design new models? I'm currently learning about neural networks, and it seems to me that there usually is no good theoretical explanation given for why certain architectures work; there is most of the times, no formal argument as to why certain architectures work. The explanation give by professors seems to be more that we devise architectures based on what we think might work; not on some formalized notion. As an example, we think GRUs work because they allow us to remember information from the past, and allows us to forget and remember as we chose. Well, who's to say simple skip level connections don't allow that?
basically, is it the case that ANNs will remain blackboxes, and new techniques are basically architectures coming from gut feeling as to what might work?

You are right to guess that there is no solid theoretical way to design neural models. This is in part consistent with the free lunch theorem which says that there is no universal algorithm that can solve all problems efficiently.

The performance of neural models not only depends on the idea but also on the data. Batchnorm has a valid reason for normalising the input to each hidden layer in neural networks but it doesn't always improve the performance of models. One can find the limitations of current neural architecture for a given data and can come up with a fix that might improve the performance but it doesn't always mean that it will prove to be better.

As of now the field is more or less driven by an empirical approach rather than a theoretical one when it comes to designing new neural architecture.


Can the 'bin size' in a histogram be thought of as a regularity constraint? When thinking about a histogram as an estimate of the density function, is it reasonable to think of the bin size as a parameter that constrains the local structure of that function?

Also, is there a better way to articulate this reasoning?

Yes, I think that the idea is to think of a histogram as a particular case of Kernel Density Estimation. In this case, the kernel function for a KDE to be a histogram is an indicator function. In this case, the bin size is equivalent to the bandwidth of the KDE.

The bandwidth of the KDE is indeed a regularity measure: KDEs with very high bandwidth have huge bias (they say that the distribution is uniform), while KDEs with very low bandwidth have huge variance (they adapt to the training data but do not generalize well). As the bandwidth of a KDE is equivalent to the bin size, we can say that the bin size is a regularity measure.