How to decide on using xgboost with imputation or without it and keeping missing values? I have a large genetic dataset that I am using xgboost on to score most likely disease causing genes - giving the genes a score between 0-1 of likelihood.

I try to avoid features with a lot of missing data but this can be hard for genetic data, the largest amount of missingness I have for a feature is roughly half of values in a feature column are missing.

Currently I run my xgboost model in 2 versions, one with random forest imputation of missing values and one without where xgboost is handling the missing data directly. The imputation model performs at an r2 of 0.7 and the model with missing values performs at 0.8 on nested cross-validation.

My question is how do I choose which version to take on to further work? Can I trust that the higher 0.8 r2 with missing data is because xgboost is finding patterns in the missingness and finding this useful? Are there rules around missing data I should be trying to abide by? I have a biology background so I'm unsure what is best practice from a data science perspective for handling missing data, most resources about this that I've found online conclude this is a case by case problem, which I find hard to interpret into what I should be specifically looking into. Any help would be appreciated.

The first question about missing data is always why is it missing?
Have you checked or know why the data is missing and whether it is MAR, MCAR or not missing at random?
If your data is MCAR imputation is generally fine and your lower test metric might simply indicate a suboptimal imputation strategy. In this case you could try MICE or similar more advanced imputations than simple median imputation.
That not imputing missing values actually improves your prediction might indicate that your data isn't missing completely at random however. In this case coding missing values might improve your performance and therefore be the best course of action.
As described besides simply looking at the performance metric of a properly validated test set also try to understand why data is missing and what that would mean.


Different Time series Modeling techniques? I started learning Time series modeling and bit confused when I started learning the advanced techniques like Box-Jenkins and Holts Winter techniques. What is the difference between ARIMA, Box-Jenkins and Holt's Winter Models.

ARIMA is a parameterized model that consists of a few parts. First the I is for integrated, and it has to do with a technique to remove non-stationarity from the data, by looking at differences between sequential values instead of the values itself, the parameter indicates how often we want to do this. AR stands for auto-regressive, it's a regression model that regresses on previous values of itself (lag features), the order determines how far back we look. The MA means Moving Average, which is referring to a linear combination of error terms, this is also parameterized by an order.

Box-Jenkins is a method that uses ARIMA, but it first involves a number of checks before fitting the model, and afterwards does criticism to see how well the model works. This allows for iteration into a better fitting model.

Holt-Winters methods are used for smoothing out time series. Sometimes there is a lot of noise involved in time series, while you are only interested in trends and how values relate to each other. By smoothing it out via for example moving averages or exponential smoothing hopefully you gain more information by trying to filter out noise than you lose by throwing away relevant information.


Importance of features It is common to say in ML feature selection that features that are irrelevant in isolation can be important in combination with other features. Is there a simple example (one or two features) to demonstrate this?

I'm not sure irrelevant is the right word, but here is an example where it may not be an easy trend.

You're trying to predict blood pressure.

If you're only given weight, there may be some correlation, but someone who is 150lbs could be 5'10 or 5'4, and that would make a large difference to their overall health.

If you have both, you'll likely have a much stronger correlation to work with.

But as all things in ML you should test it yourself usingany real world examples as theory doesn't mean much if it doesn't hold true with real data.


Correlation vs Multicollinearity I have been taught to check correlation matrix before going for any algorithm.
I have a few questions around the same:

Pearson Correlation is for numerical variables only.

What if we have to check the correlation between a continuous and categorical variable?
I read some answer where Peter Flom mentioned that there can be scenarios where correlation is not significant but two variables can be multi-collinear?
Removing the variable is the only solution?

I was asked in an interview if we are removing one of the correlated variables, then how this multi-collinearity come?

since pandas.corr() checks correlation between all variables.
How is multi-collinearity is different from correlation?

Great answer by Leevo, just let me point out one thing: Perfect multicollinearity means that one variable is a linear combination of another. Say you have $x_1$ and $x_2$, where $x_2 = \gamma x_1$. This causes various problems as discussed in this post.

The main takeway (to put it simple) is, that $x_1$ and $x_2$ basically carry the same information (just "scaled" by $\gamma$ in the case of $x_1$). So there is no benefit of including both. In fact there is a problem with including both since multicollinearity will "confuse" the model because there is no unique effect of $x_1, x_2$, when considered jointly, on some outcome $y$.

Look the following situation (R code):

y = c(5,2,9,10)
x1 = c(2,4,6,8)                           ### = 2   * x2
x2 = c(1,2,3,4)                           ### = 0.5 * x1
cor(x1, x2, method = c("pearson"))

The correlation between $x_1$ and $x_2$ equals 1 (so of course a linear combination). Now when I try to make a simple linear OLS regression:

lm(y~x1+x2)

The result is:

Coefficients:
(Intercept)           x1           x2
        1.0          1.1           NA

The second term has been dropped by R (due to perfect multicollinearity).

We can run a regression on each term separately:

Call:
lm(formula = y ~ x1)

Coefficients:
(Intercept)           x1
        1.0          1.1

...and...

Call:
lm(formula = y ~ x2)

Coefficients:
(Intercept)           x2
        1.0          2.2

Now you can see that the coefficient for $\beta_2$ is simply $2\beta_1$ because $x_1$ is $2 x_2$. So nothing to learn from including both, $x_1, x_2$ since there is no additional information.

Basically the same problem can occur if correlation between $x_1,x_2$ is really high. See some more discussion in this post. Thus given strong correlation, one should be cautious to include both variables. The reason is that in this case, your model cannot really tell apart the effect of $x_1$ and $x_2$ on some outcome $y$, so that you may end up with weak predictions (among other problems).


Why do we need to handle data imbalance? I would like to know why we need to deal with data imbalance. I know how to deal with it and different methods to solve the issue - by up sampling or down sampling or by using SMOTE.
For example, if I have a rare disease 1 percent out of 100, and lets say I decided to have a balanced data set for my training set which is:  50/50 sample
won't that make the machine think 50% of patients will have disease? even though the ratio is 1 of 100.
So

Why do we need to deal with data imbalance?
What is the recommended ratio to have balance set?

You need to deal with imbalanced data set when the value of finding the minority class is much higher than that of finding the majority.

Let say that 1% of the population have that rare disease. Suppose that you assign the same cost to saying that a healthy man is sick or saying that a sick man is healthy. Provide a model that say that everybody are healthy, get 99% accuracy and go home early.
The problem with such a model is that though it has high accuracy, it will probably not what you are looking for. Most of the time you want to find the people with the disease (giving high weigh to false negatives) much more than you are afraid to send an healthy person to unneeded test (low weight to false positives). In a real world health problem the ratio between the weight can easily be 1 to 1,000.

The imbalance in the distribution fails most algorithms from finding a proper solution.

You are correct that just balancing the distribution isn't the optimal solution. Indeed, an algorithm that is trained on a balanced distribution is not fitted to the natural distribution on which it will be evaluated.
My favorite method is adapting it back, as you can see here.
For a discussion, see here.

Just setting the ratio to some other problem won't work since you will have the same problem.

Smote is working in a different way, which didn't work as well when I tried it, but it might fit your problem.


How to use an existing model as in input into a new model We have a click-model which is currently being used for search ranking in production, and I want to create a new model which takes the old model click probability as one input and adds some other variables in too. The problem is that training data will positionally biased by the fact that the probability of a click is correlated to the old model's prediction.

My plan is to introduce a penalty factor on the original model's prediction to ensure that it doesn't dominate the new model (eyeballing the results to decide on an appropriate penalty factor). Is this approach valid or would there be a better way to approach this?

Note that I don't want to rebuild the old model with the new variables because

The existing model takes a long time (days) to build
The new model and old model will be deployed separately, i.e. the old model will be scored offline/batch whereas the new model will be scored real-time

This depends a  bit on the used case but you could also create the 2nd model independent and then use a third model to combine them. This has the advantage that there should be no leakage between the two models (if variables are independent) and you can merge them in a very controlled way.

What is unclear for me is what advantage you would have from taken input from a model that can not be used in the same context. (offline vs real time)

If you want to use them combined and Model A as input for B you would need to guess what the output for Model A is.


unique predictions for "multi-label multi-output" classification task Let’s assume that four participants (A, B, C and D) take on five sport-challenges (e.g. swimming, running, ...). Our goal is to predict the placement of each participant for each challenge. Moreover, let’s assume we have appropriate predictors. We know that each placement (1 to 4) is unique for each challenge (only one winner ...).

My questions:

I think this prediction task is a multi-label multi-output classification, right?

Are there any algorithms, which provide unique forecasts (for each class). In other words, the algorithm should classify each person in one unique class (1 to 4). Obviously, we know a priori that same placements are very unlikely.

Thank you!

Greets

2 Questions.

I think this prediction task is a multi-label multi-output classification, right?

Yes. But you could consider splitting these joint classification tasks into seperate ones (i.e. for swimming whats the placement of A,B,C,D etc) IF you data allows it.

Are there any algorithms, which provide unique forecasts (for each class)?

Think in terms of probabilities. The strongest (most discriminatory one) is the one that has the highest-not-so-much-overlapping probabilities to different outputs. So taking these probabilities and "rounding" (possibly forcing some predictions if model is highly uncertain---meaning, for example, that he predicts 25% for every placement, here some other heuristics should be used. For example create a new variable that says how did competitor fare in other competitions etc) that up will give unique classes.


What are "belief maps" and "affinity maps"? When evaluating Nvidia-Deep Object Pose Estimation, I came across these terms, belief maps and affinity maps. I haven't been able to find a satisfying answer online on what these terms mean.

In computer vision, pose estimation is the detection of an object's orientation and positioning.

Belief Maps

A team from Nvidia have proposed "Deep Object Pose Estimation", which has two main components:

Detect objects in image and generate 2D keypoint mappings
Project 2D mapping to 6D

From what I can see, "belief maps" are generated by the first component of their system and describe elements of the 2D keypoint mappings that are then projected to 6D by the second component. From the article:

  The feedforward network takes as input an RGB image of size w×h×3 and
  branches to produce two different outputs, namely, belief maps and
  vector fields. There are nine belief maps, one for each of the
  projected 8 vertices of the 3D bounding boxes, and one for the
  centroids. Similarly, there are eight vector fields indicating the
  direction from each of the 8 vertices to the corresponding centroid...
  to enable the detection of multiple instances of the same type of object.

Affinity Fields

I've not come across affinity maps, but affinity fields are referred to above as inspiration for their approach to pose estimation, specifically in "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields". That article aims to solve the problem of how to attribute limbs to people in video containing more than one person.

Affinity maps are "an explicit nonparametric representation of the keypoints association that encodes both position and orientation of human limbs", which you can see a demo of in their video (which is pretty cool).

So bringing it back to the first article, affinity fields are analogous to the vector fields mentioned above. They're used to associate body parts with a particular person in 5, or to associate vertices of belief maps to a centroid representing an object in 4.

So belief maps represent the features extracted by their deep CNN, and vector/affinity fields help associate those vertices with an item that's been detected.


How can I create a space in IBM Cloud? I want to try the IBM speech to text API. I created an IBM cloud account and went to https://console.bluemix.net/catalog/services/speech-to-text. I see the error message:

  Please create a space in the selected org or choose another org.

Where can I create such a space? I wasn't able to find it yet.

It seems that Watson doesn't allow some accounts to access some regions. Just choose another region, like "US South", when creating the resource, and the problem will be gone.


Multivariate time series forecasting with LSTM I have the following problem:

$\mathbf{Y}(t)$ = real valued random vector of observations at time t, $Y_i(t) \in R_{(0, 1)}$

$\mathbf{X}(t)$ = real valued random vector of observations at time t, $X_i(t) \in R$

I want to forecast $\mathbf{Y}(t+1)$, given $\mathbf{X}(t+1)$ and $\mathbf{Y}$ history.

To sum up I'd like to forecast a vector of observations given its history and another vector of observations as the same timestamp.

To make it clearer: let's say that I want to forecast daily temperatures in London and Dublin, knowing the same day temperatures in Manchester and Liverpool and London and Dublin (and Manchester and Liverpool) historical data. It's a bit like a kalman filter where you want to estimate the state variables given some output.

How would you set up this problem in an LSTM?

1)First and most important, do not give Yi(t) history as feature. You will just end in a model that replicates the previous input to minimize the error, a cheating model. For more detail, you can have a look at my explanations at two different questions:

https://stackoverflow.com/questions/52778922/stock-prediction-gru-model-predicting-same-given-values-instead-of-future-stoc/52786399?noredirect=1#comment93125020_52786399

https://stackoverflow.com/questions/52252442/how-to-handle-shift-in-forecasted-value/53141558#53141558

2)Create your labels by sliding your Y(t) one step forward so that your each sample will have a label of Y(t+1). That means you will delete sample #1 as a result.

3)Use a time-window for each sample for your features. Do not provide just your features as x(t) for the label y(t+1). For example, with a window size w = 10, provide x(t-9),x(t-8),.....,x(t-1),x(t+1) as a single sample for label y(t+1). Then you will be boosting the sequential nature of LSTM, possibly acquiring greater performance.

4)You can use Keras for your LSTM regression task, have a look at simple code piece from my old works, I modified it for your task:

nn = Sequential()
nn.add(LSTM(80, batch_input_shape=(64,11,20), return_sequences=True, recurrent_dropout = 0.1))
nn.add(LSTM(60, recurrent_dropout = 0.2))
nn.add(Dropout(0.2))
nn.add(Dense(40, activation='relu'))
nn.add(Dropout(0.2))
nn.add(Dense(10, activation='relu'))
nn.add(Dense(1))
nn.compile(loss= 'mse, optimizer= 'adam')
nn.fit(x_train, y_train, epochs=15, batch_size=64, shuffle=True, validation_data=(x_dev, y_dev))
y_pred = nn.predict(x_dev, batch_size=64)

where batch_input_shape=(64,10,25) explains that your batch size is 64 (trains 64 samples per gradient descent), your window size is 10, and you have 25 different features. You can delete anything about dropout in the code if you get confused; they are just for preventing overfitting.

Note: Do not forget to normalize your numeric input data at the start!

Hope I could help. Good Luck!


How do write the following piece of code in R? I am trying to write a piece of code for R to fetch data from an api with the following command:

data <- read.csv("https://...../api/dataservice?sql = select * from db where name IN (?)")

The parenthesis are where I need to input 218 different name tags. How can I do this programmatically? It is possible to write down few of them, but over 200 is not possible. Is there a way this? Can I use a character vector containing the names?

If you already have the names in R, you can iterate over the vector of names, and use the paste function. Here's the help page for it. sprintf would be another option, if you prefer that approach.

An example of this would be

names <- c("a", "b", "c")
querystring_a <- "https://...../api/dataservice?sql = select * from db where name IN "
querystring_b <- '\\'
querystring_c <- "\'"
for(n in names){
    cat(paste(querystring_a, querystring_b, querystring_c, n, querystring_c, sep=""), sep="\n")
}

Return:

https://...../api/dataservice?sql = select * from db where name IN \'a'
https://...../api/dataservice?sql = select * from db where name IN \'b'
https://...../api/dataservice?sql = select * from db where name IN \'c'


Post processing with Random Forest I understand that by filtering out the instances with labels that Random Forest trees are uncertain upon with their decisions, and model these with another classifier could give a better overall result. My question is, how can I "combine" two (or more) classifiers' classification on a single unlabeled dataset?

Here's a solution using the caret package for R. A Random Forest is first trained on the data. All observations for which the probability (from the voting) is less than 99% are then passed to model 2, linear discriminant analysis. Only the probabilities from unseen resampling observations are used, since the Random Forest will otherwise fit the training data perfectly. That is what caret is needed for.

The accuracy is a little higher for the uncertain cases, but this is probably overfitting since I have tried several different models and the data set is small.

I'd like to know if this really improves the performance in your application for the out of sample test data. Are there any papers that recommend this approach? This approach seems to resemble boosting. I tried it on some of my data but could not improve the out of sample performance I got from model 1 (a Random Forest).

data(iris)
library(caret)
myTrainControl <- trainControl(method = "cv", number = 10,
                               savePredictions = T,
                               classProbs = TRUE)
set.seed(4213) # To get the same resamples every time
M1 <- train(y = iris$Species,
            x = iris[, !(names(iris) == "Species")],
            method = "rf",
            trControl = myTrainControl)

M1pred <- M1$pred[M1$pred$mtry == 2, ]
confusionMatrix(M1pred$pred, M1pred$obs)
# Accuracy 96%

# Inspect the predicted class probabilities:
probs <- cbind(M1pred$setosa,
                   M1pred$versicolor,
               M1pred$virginica)
colnames(probs) <- c("setosa", "versicolor", "virginica")
maxCol <- max.col(probs)
probs <- cbind(probs, maxCol)
maxProbs <- apply(probs, 1, function(x) x[x["maxCol"]])
summary(maxProbs)
# Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
# 0.5120  0.9805  0.9980  0.9645  1.0000  1.0000

# Let's define anything below 99% as uncertain:
uncertain <- which(maxProbs < 0.99)
length(uncertain) # 45
# Find rows in iris data that belong to 'uncertain' cases:
uncertainIndex <- M1pred$rowIndex[uncertain]
    confusionMatrix(M1pred$pred[uncertain],
                reference = M1pred$obs[uncertain])
# M1 Performance in uncertain cases:
#              Reference
# Prediction   setosa versicolor virginica
# setosa          5          0         0
# versicolor      0         17         3
# virginica       0          3        17
#
# Overall Statistics
# Accuracy : 0.8667

# Train new model on uncertain data only:
irisUncertain <- iris[uncertainIndex, ]
set.seed(4213) # To get the same resamples every time
M2 <- train(y = irisUncertain$Species,
            x = irisUncertain[, !(names(irisUncertain) == "Species")],
            method = "lda",
            trControl = myTrainControl)

M2pred <- M2$pred

confusionMatrix(M2pred$pred, reference = M2pred$obs)
#              Reference
# Prediction   setosa versicolor virginica
# setosa          4          0         0
# versicolor      1         16         0
# virginica       0          4        20
#
# Overall Statistics
# Accuracy : 0.8889
# (Small discrepancy: Why does caret report an accuracy of 88,5% for M2?)

# For new data predictions can be made as follows:
# (just as an example from the original data again)
# Some 'uncertain' cases are 'certain' now using the full M2 model
newdat <- irisUncertain[30:34, -5]
M1maxProbs <- apply(predict(M1, newdat, type = "prob"), 1, max)
ifelse(M1maxProbs < 0.99,
       paste("M2:", predict(M2, newdat)),
       paste("M1:", predict(M1, newdat)))
# 85               94              107              126
# "M2: versicolor" "M1: versicolor"  "M2: virginica"  "M1: virginica"
# 71
# "M2: virginica"


Trade-off between variable stability and discriminatory model quality It is very normal for data scientists and modeling professionals to be concerned with the stability of the model. It basically means that if a variable is important today, it cannot change its importance over time because this instability would be a demonstration of the model's weakness.
It is difficult to argue that stability is good. But in the area of credit modeling (which produces credit scores, for example) people are cracked by stability, to the point of preferring models with less discrimination to a more stable one.
My question is what is the true utility of stability and what is the correct way to make this trade-off with the discriminatory metric of the model (KS, Gini, AUC, etc.).
I would guess (personal guess) that stability somehow ties the model to a less than ideal metric because it does not allow behaviors to change over time. If they don't really change, there will be no problems, but if the effects of the variables are changing over time, it is normal to assume that this is reflected in less stability and analysts work around this by placing simpler models or removing the variables.

I think there might be a bit of confusion here: what is usually called a stable model is a model for which the performance doesn't vary (or not significantly) when sampling a different subset of training data or test data. In other words a model is stable if chance doesn't affect its performance. Typically one can use cross-validation in order to assess the stability of a model: if the variance is high between different splits then the model is unstable. Unstability is often a sign of overfitting, so one should be very cautious with an unstable model even if it seems to perform well: the risk is that the model might obtain good performance on the test set by chance, which means that it will actually perform poorly on new data but there will be no way to detect the problem. This is why people might favor a stable model to a high performance in some applications, because the consequences of deploying an unstable model in production would be costly.
But what you are talking about is not general model stability, it's stability of the performance across time for time-based data. That's a completely different question, it's about whether the model can represent time variations accurately. There are many types of models which can do that, but of course the more complex the model the more data it needs and the higher the risk of overfitting.


Pre-process data images before training OneClassSVM and decrease number of features I want to train a OneClassSVM() using sklearn, and I have a set of around 800 images in my training set.

I am using opencv to read the images and resize them to constant dimensions (960x540) and then adding them to a numpy-array. The images are RGB and thus have 3-dimensions. For that, I am reshaping the numpy array after reading all the images:

#Assume X is my numpy array which contains all the images before reshaping
#Now I reshape X
n_samples = len(X)
X = X.reshape(n_samples, 950*540*3)

As you can see, the number of features is huge (1,539,000 to be exact).

Now I try to train my model:

model = OneClassSVM(kernel='rbf', gamma=0.001)
model.fit(X)

After running my code, it crashed due to MemoryError. If I'm not mistaken this is obvious due the large number of features? So, is there a better way to pre-process the images before fitting them, or to decrease the number of features?

You should try converting them to Principle Components using PCA. Please refer this Analytics Vidya PCA, this should give give you a good understanding.

PCA converts n large vector into p Principle components.

  First principal component is a linear combination of original predictor variables which captures the maximum variance in the data set

Second PC is also a linear combination of original predictor variables which captures remaining variance. All the succeeding ones follow the same concept.

This way you can select top Principle components which explains good enough cumulative variance in your data


how to get the classe names in an image classifer after predection? I made an image classifier of 80 classes of handwritten numbers then I tested my model and it worked pretty fine, the only problem that I have now is the display of the correct names of these classes.
Dataset:
2 folders: [Train Folder===> 80 folders each has 110 images, Validation folder===> 80 folders each has 22 images]

Bellow the code I used for training, saving and testing my model:

from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D
from keras.layers import Activation, Dropout, Flatten, Dense
from keras import backend as K

# dimensions of our images.
img_width, img_height = 251, 54
#img_width, img_height = 150, 33

train_data_dir = 'C:/Users/ADEM/Desktop/msi_youssef/PFE/test/numbers/data/train'
validation_data_dir = 'C:/Users/ADEM/Desktop/msi_youssef/PFE/test/numbers/data/valid'
nb_train_samples = 8800 #10435
nb_validation_samples = 1763 #2051
epochs = 30 #20 # how much time you want to train your model on the data
batch_size = 32 #16

if K.image_data_format() == 'channels_first':
    input_shape = (3, img_width, img_height)
else:
    input_shape = (img_width, img_height, 3)

model = Sequential()
model.add(Conv2D(32, (3, 3), input_shape=input_shape))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(32, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(64, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Flatten())
model.add(Dense(64))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(80)) #1
model.add(Activation('softmax')) #sigmoid

model.compile(loss='sparse_categorical_crossentropy',optimizer='rmsprop',metrics=['accuracy'])#categorical_crossentropy #binary_crossentropy

# this is the augmentation configuration we will use for training
train_datagen = ImageDataGenerator(
    rescale=1. / 255,
    shear_range=0.1,
    zoom_range=0.05,
    horizontal_flip=False)

# this is the augmentation configuration we will use for testing:
# only rescaling
test_datagen = ImageDataGenerator(rescale=1. / 255)

train_generator = train_datagen.flow_from_directory(
    train_data_dir,
    target_size=(img_width, img_height),
    batch_size=batch_size,
    class_mode='binary')

validation_generator = test_datagen.flow_from_directory(
    validation_data_dir,
    target_size=(img_width, img_height),
    batch_size=batch_size,
    class_mode='binary')

model.fit_generator(
    train_generator,
    steps_per_epoch=nb_train_samples // batch_size,
    epochs=epochs,
    validation_data=validation_generator,
    validation_steps=nb_validation_samples // batch_size)

model.save('testX_2.h5') #first_try

last epoche resulat

Epoch 30/30
275/275 [==============================] - 38s 137ms/step - loss: 0.9406 - acc: 0.7562 - val_loss: 0.1268 - val_acc: 0.9688

how I tested my model:

from keras.models import load_model
from keras.preprocessing import image
import matplotlib.pyplot as plt
import numpy as np
import os

#result = [10,7]

def load_image(img_path, show=False):

    img = image.load_img(img_path, target_size=(251, 54))

    img_tensor = image.img_to_array(img)
    img_tensor = np.expand_dims(img_tensor, axis=0)
    img_tensor /= 255.

    if show:
        plt.imshow(img_tensor[0])
        plt.axis('off')
        plt.show()

    return img_tensor

if __name__ == "__main__":

    # load model
    model = load_model('C:/Users/ADEM/Desktop/msi_youssef/PFE/other_shit/testX_2.h5')

    # image path
    img_path = 'C:/Users/ADEM/Desktop/msi_youssef/PFE/dataset/1.75/eeza.png'

    # load a single image
    new_image = load_image(img_path)

    # check prediction
    pred = model.predict_classes(new_image)
    print(pred)

it gives me this result instead of given the name of folder:

[7]

I get your problem,  the point is that its correct what the model does, but you have to build a look-up table for its answer. Your ground-truth, looks somethink like that [0,0,0,0,1], a one-hot vector for example. You, the human know what this code stands for, for example cats.  just like that you have to build an numpy array, listing the word-embeddings in the correct order and afterwards calling it like: class_names[prediction],prediction being your CNN-result ->[7].

To sum it up, the Dense-layer in the end is giving you with softmax-activation a propability desnity function P. You are using this in comparing it with your ground-truth desnity function q, and calculate the difference. You just use numbers, not words, so you have to write an interpretation of the models answeres, for example in form of a lookup-table.

An example could be like:

pred = model.predict_classes(new_image)
labels=np.array(["cats","dogs","cars","humans"])
print(labels[pred[0]])

-> >>> cats


Delete/Drop only the rows which has all values as NaN in pandas I have a Dataframe, i need to drop the rows which has all the values as NaN.

ID      Age    Gender
601     21       M
501     NaN      F
NaN     NaN      NaN

The resulting data frame should look like.

Id     Age    Gender
601     21      M
501    NaN      F

I used df.drop(axis = 0), this will delete the rows if there is even one NaN value in row. Is there a way to do as required?

The complete command is this:

df.dropna(axis = 0, how = 'all', inplace = True)

you must add inplace = True argument, if you want the dataframe to be actually updated. Alternatively, you would have to type:

df = df.dropna(axis = 0, how = 'all')

but that's less pythonic IMHO.


Clustering of sparse matrix with many co-variates I have a 2M x 2000 sparse matrix where rows represent an item and columns represent dimensions. I want to understand whether there are meaningful clusters in the data and I started to explore the dimensions to transform and normalise the data.

Of the 2000 attributes to an item, many are co-variant (rho > .5).
Are there clustering techniques that handle co-variants well automatically, without having to remove them manually?

You need to apply PCA and reduce your data to lower dimensions, then applying a classic clustering technique (e.g. k-means or DBSCAN) works depending on how your samples are distributed. I strongly recommend you visualize data in 2 or 3 dimensions and have a brief visual inspection. Gives you an insight about what is going on there. However the final number of dimensions you get out of PCA might be chosen to be more than 2 or 3 (it is usually).

Steps

Normalize features if needed
Apply PCA and take the number of PCs which explains 85% of variance
Optional: Visualize embedded data in 2 or 3 dimension to get a feeling about distribution of samples (it does not anything more than just an intuition! Even that intuition can not be validated)
Apply a clustering algorithm on the result of (2)


Should estimated probabilities from multi class classification sum to 1 I am using a neural network with sigmoid activation function $h(z) = 1 / {(1+e^{-z})} $ in order to classify image data into 6 categories.  When running the trained neural network over new image data, I noticed that the sums of the estimated probability from the hypothesis output for all 6 classes do not always sum to 1.  For example given an input image, the hypothesis output for each class might be:

  Class 1  --- 0.10

  Class 2  --- 0.11

  Class 3  --- 0.12

  Class 4  --- 0.13

  Class 5  --- 0.14

  Class 6  --- 0.15

I interpret this image as having a $13%$ probability of being classified into class 6.  However, the sum across all classes is < 1.

My intuition says the probability of each class should sum to 1 but again, I am very new to the machine learning world.

Could there be a bug in my code or is this a 'normal' output?

Summing up to one or not both can have their special meaning that I'll try to explain them. If you have classes that are not mutually exclusive, say you have dog and cat classes and they both can exist in an image. In such cases, you should use sigmoid nonlinearity as the output of each class and interpret each one separately. Each one that has a value greater than half can explain the existence of the corresponding label. On the contrary, your inputs may be mutually exclusive which means in each input you may just have a cat or a dog. In this case, you should employ softmax nonlinearity and yes, it sums up to one. The winner would be the one with the highest value.


making predictions from detrended data I am building a machine learning based model (random forest in scikit-learn) to predict maize yields in the U.S. based on data on historical maize yields and temperature and precipitation information. Since there is a trend in maize yields, I detrend the yield data first, then fit the model and then predict for current year. However, I assume that the predictions from the model are detrended as well. How do I go about making predictions that are not detrended? Any pointers and literature reference would be helpful

I'm just guessing but I can think of two options:

Train the model from the original data, including features which represent the current trend so that the model can integrate it. It makes the job of the model more complex so it might not work very well.
Since you can untrend the data I assume that you can calculate the trend function. So you could train an independent model for this trend function and apply it on the predictions obtained with the untrended model. In this case performance depend how accurately the trend is predicted.


Is confusion matrix possible in one column I am performing anomaly detection using K-Means. I am working with only one column, plotting those values and then within this column I am adding some anomalies.

My question is

if it is possible to perform confusion matrix for this ? If so what is the approach ?
Asking this since I have seen that confusion matrix needs y axis and x axis

Confusion matrix needs groun truth values and predicted values.

You have _____, so you need other part for this multiclass confusion matrix.


Explanation of random forest performance difference to when using categories and when using dummy variables I have some hand coded feature which is a category with values "High", "Low", and "Normal".
I created this feature myself and my problem performance (classification) increased dramatically when using it with expanding these by dummy variables.
Now since I'm trying random forest, I thought I change "High, Low, Normal" to 1, -1, 0 instead.
Now the same model doesn't learn at all.
I thought it should become easier actually for it to split.
Does this have to do with me putting normal to 0?
Thank you for any explanation helping me to understand this.

It should work: the variable is ordinal so using numerical values makes sense.
So there's a bug somewhere, here are a few suggestions of things to look at:

Possibly a type conversion error somewhere: make sure the variable is interpreted as numerical.
Check whether the model actually uses the variable: if not then it's likely some type error; if yes then I would investigate what goes wrong: for example it might help to plot this variable vs. target in the two cases where the variable is categorical or numerical.
Maybe some difference between the preprocessing of the training and test set: apply the model on the training set, if the performance is good then it's likely that there's something wrong in the preprocessing of the test set.


Does it make sense to repeat calculating AUC in logistic regression? I have a question regarding logistic regression models and testing its skill.
I am not quite sure if I understand correctly how the ROC Curve is established.
When calculating the ROC curve, is a train test split happening and then the skill of a model based on the training split is tested on the test split? or is a model based on the ENTIRE data just tested on the ENTIRE data?
If the first is the case, would it make sense to do repeated random train test splits and average out the Area under the curve? would that bring about any additional certainty about the model's skill?
Thank you.

The calculation of ROC curve and the AUC based off of that curve is simply a comparison of the predictions from your model (logistic regression) and the actual values on some set of data. This can occur with predictions on the training set or predictions on a test set. Best practice is to do this comparison on a test set as this will best represent the performance of the model on new data.
Averaging of the AUC based on repeated train-test splits, with retraining of the logistic regression on the training set and calculation of the ROC curve and AUC on the test set can give a better estimate of the performance of the model. In addition, the distribution of the AUC across each of the splits can give a sense of the stability of the model's performance.
The repeating of train-test splits is commonly called cross-validation and most commonly performed through k-fold cross-validation where you split the data into k sets, use one as the test set and the rest as the training set. You can then repeat this process using each of the k sets as the test set.


svm.LinearSVC: larger max_iter number doesn't always increase the accuracy/precision/recall Background:

Supervised machine learning
Data shape

10+ features, target = 1 or 0 only, 100,000+ samples (so should be no issue of over-sampling)

80% training, 20% testing

train_test_split(X_train, Y_train, test_size=0.2)
Use svm.LinearSVC(max_iter = N).fit( ) to train labelled data

Scaling not applied yet (all feature values are around 0-100 (float64))
Other parameters (e.g., c = ) use default value

Results:

print("Accuracy:", metrics.accuracy_score(y_test, y_pred))
print("Precision:", metrics.precision_score(y_test, y_pred))
print("Recall:", metrics.recall_score(y_test, y_pred))

Question:

I increased max_iter = from 1,000 to 10,000 and 100,000, but above 3 scores don't show a trend of increments. The score of 10,000 is worse than 1,000 and 100,000.

For example, max_iter = 100,000

Accuracy: 0.9728548424200598
Precision: 0.9669730040206778
Recall: 0.9653096330275229

max_iter = 10,000

Accuracy: 0.9197914270378038
Precision: 0.9886761615689937
Recall: 0.8093463302752294

max_iter = 1,000

Accuracy: 0.9838969404186796
Precision: 0.964741810105497
Recall: 0.9962729357798165

What could be the reason?
Do I need to test different max_iter values and select the best performance? For example, use GridSearchCV( )

When trying to find the optimum number of iterations it's normally quite useful to visualise how the increasing iteration effect the accuracy (can identify over-fitting and when you should stop fitting).

# Import libraries used
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import accuracy_score
from sklearn.svm import SVC

# Create a template lit to store accuracies
acc = []

# Iterate along a logarithmically spaced ranged
for i in np.logspace(0,5, num = 6):
    # Print out the number of iterations to use for the current loop
    print('Training model with iterations: ', i)
    # Create an SVC algorithm with the number of iterations for the current loop
    svc = SVC(solver = 'lbfgs', multi_class = 'auto', max_iter = i, class_weight='balanced')
    # Fit the algorithm to the data
    svc.fit(X_train, Y_train)
    # Append the current accuracy score to the template list
    acc.append(accuracy_score(Y_test, logreg.predict(X_test)) * 100)

# Convert the accuracy list to a series
acc = pd.Series(acc, index = np.logspace(0,5, num = 6))
# Set the plot size
plt.figure(figsize = (15,10))
# Set the plot title
title = 'Graph to show the accuracy of the SVC model as number of iterations increases\nfinal accuracy: ' + str(acc.iloc[-1])
plt.title(title)
# Set the xlabel and ylabel
plt.xlabel('Number of iterations')
plt.ylabel('Accuracy score')
# Plot the graph
acc.plot.line()
plt.show()

This will produce a graph where the number of iterations has been logarithmic increased (note that it might take a bit of time experimenting with np.logspace to create whole number iteration steps).

If the accuracy increases then keep following the trend, if it plateus stop theres probably no point wasting your time, if it drops go back to the highest value (you've over-fitted to the training data).


How to find if any variable which is normally distributed in pandas dataframe Im having age column and how can i plot and say this is normally distributed or not

You can do like so:
from scipy.stats import normaltest

def normal_test(frame, significance = .01, plot = False):
    """
    Function to perform ks test and test against normal distribution using
    D’Agostino, R. B. (1971), “An omnibus test of normality for moderate and large sample size”

    frame: a pandas dataframe
    significance: float. Alpha level for which the null hypotesis will be rejected (H0: series comes from a normal distribution)
    plot: Boolean, whether or not plot a histogram for resulting columns

    returns a dataframe with only those columns that follow a normal distribution according to test.
"""

    columns = frame.columns.tolist()

    for col in columns:
        aux = frame[col]

        _, p = normaltest(aux)

    if p <= significance:
        frame.drop(col, axis = 1, inplace = True)

    if plot:
        fig, ax = plt.subplots(1,1, figsize = (12,7))

    frame.hist(ax = ax, color = "darkred", edgecolor = "black");

    return frame

Test data:
 A = pd.DataFrame({"normal":np.random.normal(size = 1000),
                  "exponential":np.random.exponential(size = 1000),
                  "poisson":np.random.poisson(size = 1000),
                  "normal_2":np.random.normal(size = 1000)})

result = normal_test(A)


How to select particular column in Spark(pyspark)? testPassengerId = test.select('PassengerId').map(lambda x: x.PassengerId)

I want to select PassengerId column and make RDD of it. But .select is not working. It says 'RDD' object has no attribute 'select'

You could try the following,

testPassengerID = test.select('PassengerID').rdd

this would select the column PassengerID and convert it into a rdd


How to apply multiple filter in Data Frame? How to implement multiple filters for checking data cell in a range ?
Suppose, I have a list of numbers like,

range_1 = [ 70 ,15,5,7,3,7,8,3,2, 63  ]

# and

range_1 = [ 50, 56, 80, 61, 83, 87, 13, 58, 43, 24, 84, 54, 64,36, 48  ]

And I want to check any column values exist within these two lists.

Any suggestion would be appreciated

data.loc[ data['age'].isin(range_1) & data['height'].isin(range_2) ]


How to make machine learning specifically for an individual in a group when we have the data on the group? Lets specify the question with the help of the figure below: We know that one part of the behaviour (our target Y) will depend on common parameters (for the group). It is represented by the grey zone on the figure. And that one part is parameters specific to each individual. ( It is represented by the pink and blue zone )

The precise question is:

Knowing that we have data from the entire group. How to use all the data of the groups, to create a specific model for an individual of this group? The idea is to get a solid model/result because it is based on all the available data, but still specific to an individual.
I imagine an answer in the form of a short list of techniques to achieve that.

Let’s illustrate the question with an example:

We do a study of 100 people. Let’s name the people [0, 1, 2, 3, 4, 5 … 99]
For each person, we do a study of 30 days. Every day, we do 5 measurements of the emotion of the person (X). And then a measure of the quantify of calories in the evening meal (Y).

In this example, the goal, using machine learning, is knowing X on day 55 (after the tests), for person number 3, to be able to predict Y (on day 55) for a given person. I often came across this problem. With my experiments, tests and research, I see two possibilities:

The first option is to take all the samples. 100 people * 30
measurements (of 5 samples x and y) = 3000 points. We create a model
that connects x to y. Then we take the new x (for person number 3),
and ‘predict’ y. We then have an answer that takes into account all
the observations but is not the specificity of the person number 3.
The model and the answer are kind of an ‘average behavioural
response’ for the population. This model corresponds more to the grey
zone of the figure you see above.
The second option is to take only information from person 3 to
create the model. We have 30 points. We create a model that connects
x to y. The model is really specific to person 3. Since the model
uses fewer points, it is less accurate. We did not use our total
knowledge/data of the grey zone.

I have the deep intuition that there is a way to do better than these two options. I tried a lot but without satisfying success.

I tried, with random forest algorithms, to use all data at the same time and add an id in X vectors. This id represents the individual. It does not seem to have worked. The internet searches I have done on this topic give me unrelated results. Any help or keywords on this topic are welcome.

Is it useful or necessary that I share an example dataset?

There are a number of other options.  The way to select between them is to ask yourself: How are you going to deploy this classifier?  What data will be available to the classifier?

It's possible to build a single model that still has the ability to personalize for a particular person.  You do that by adding another attribute that identifies the person; typically, this would be encoded as a one-hot vector, so you have 3000 data points, and each data point has some features realted to their emotion measurement(s), plus 100 more features for the one-hot-encoded identity vector.

The exact details depend on what data will be available at test time.  Suppose that you will try to predict the number of calories in the evening meal based on the emotion measurements over the past week.  Then you have $100 \times (30-6) = 2400$ data points (the first 6 days from each person aren't usable because you don't have a week of emotion measurements yet).  Each data point has 135 features: 35 features for the emotion measurements over the past 7 days, and 100 features for the one-hot-encoded vector.  Then you train a model on this.

If you use this kind of approach, you need to be careful about how you split the data into training, validation, and test sets.  I suggest you split by person not by data point; so if you want a 60/20/20 split, you pick 60 people to be in the training set (leading to $60 \times 24$ data points in the training set, 20 people to be in the validation set ($20 \times 24$ data points), and 20 people for the test set ($20 \times 24$ data points).  This is based on a deployment scenario where you train a single model once based on some people's data, and then to predict calories for a new person, you don't retrain it based on historical data from the new person.

Hopefully you can see how to adjust this based on the deployment scenario.  The basic principle is to figure out how the classifier will be used in practice, and then choose an evaluation strategy that matches how it will be used.

Another option is to first try to cluster the users into a few clusters, and build a classifier to predict which cluster each person is in.  Then, build one model per cluster.

A third option is to try to identify some additional attribute of the person that will help you predict their calory consumption.  Add this as an additional feature (W).    For instance, this might be gender or demographics or anything else that you think has predictive power.


Data Leakage when preprocessing categorical features? I am fairly new to machine learning. I came across the concept of Data Leakage. The article says that always split the data before performing preprocessing steps.
My question is, do steps such as discretization, grouping categories to a single category to reduce cardinality, converting categorical variables to binary variables, etc. lead to Data Leakage?
Should I split the data to train and test set before applying these steps?
Also, which are the main preprocessing steps I really need to be cautious of in order to avoid data leakage?

Think of the data being organized in rows (instances) and columns (features). Any pre-processing step which mixes information between/across rows can lead to data leakage.
A typical example is standardization or normalization. Applying min-max-scaling, for example, on the whole dataset would leak information since it is an aggregation across rows/instances. Another typical example is encoding categorical variables. When you train your encoder before the split you're taking into considerations values which you should not have seen before. Because when applied on real data, you might be presented unknown data values too.
The guiding principle here is, that your test strategy should resemble the real application in order to estimate the ability of your model to generalize to unseen data.
The safest way to avoid data leakage is to split the data before applying any pre-processing. Moreover, this approach supports designing the pipeline in a way that you can apply it to validation/test and later production data the same way you applied it on training data.


NLP food review classification I have a dataset consisting of a bunch of food reviews. I'm trying to build a model that determines if a given review talks about a food safety issue. The dataset has a lot of features, but relevant to my goal there are only two features of importance --  food reviews and their corresponding numerical ratings.
I'd love a pointer or two to figure out how to approach this problem.
As a first pass attempt, I'm thinking of pipelining all reviews into a sentiment analsyis filter and then look at the ones have keyword related to food safety (eg: "high cholesterol", "unsafe", etc)

The best way to solve your problem would be to collect the right labels for your task. Currently, you have the input i.e. text of a review and the rating which can be used to predict the sentiment of the review, but not whether it mentions safety issues. To collect the right labels, you should look at each review and assign it a binary label - 0 means food safety issues are not mentioned, 1 means they are mentioned. The feasibility of such data collection depends on the size of the dataset and the importance of your project. How big is your dataset? How much time/money are you willing to commit to that project? Once you collect the labels, you can train and evaluate a neural network based on them. If this answer helps you, feel free to ask for elaboration, I don't know what's your level of data science expertise and don't want to state the obvious.


Help Using Backprop Formulas from Deep Learning Book I am trying to see how the fundamental formulas of backpropagation from this online book work on a small example.

The author lists these 4 fundamental equations (where I believe from the reading that the cost (or loss) function C as well as the derivative of the activation function $\sigma$ are general and work with any cost or activation function):

If I take a small example network:

and assume....

C = $\frac{1}{2}(a^3-y)^2$  (squared error)
The activation (a) for each node is just the logistic function sigmoid

From here we have the following:

$z^2_1=w^1_{11}x_1+w^1_{12}x_2+b_1$
$z^2_2=w^1_{21}x_1+w^1_{22}x_2+b_2$
$z^2_3=w^1_{31}x_1+w^1_{32}x_2+b_3$
$a^2_1 =\sigma(z^2_1)$
$a^2_2 =\sigma(z^2_2)$
$a^2_3 =\sigma(z^2_3)$
$z^3_1=w^2_{11}a^2_1+w^2_{12}a^2_2+w^2_{13}a^2_3$
$a^3=\sigma(z^3_1)$

If I want to calculate $\frac{\partial C}{\partial w^2_{11}}$ it seems from the book that I should calculate $a^1_1\delta^2_1$ where $\delta^2 = ((w^{2+1})^T\delta^{2+1})\odot\sigma'(z^2) $.

Since $W^3$ does not exist I assume it is ignored and we can calculate $\delta^2_1= \delta^{2+1}_1\odot\sigma'(z^2_1)$.

$\delta^3_1 = \frac{\partial C}{\partial z^3_1} = (a^3-y)\sigma'(z^3_1)$
so $\frac{\partial C}{\partial w^2_{11}}= a^1_1*((a^3-y)\sigma'(z^3_1))*\sigma'(z^2_1) $. $a^1_1$ doesnt exist except if this would mean $x_1?$

However, if I just calculate $\frac{\partial C}{\partial w^2_{11}}$ I get $\frac{\partial }{\partial w^2_{11}}\frac{1}{2}([\sigma(w^2_{11}a^2_1+w^2_{12}a^2_2+w^2_{13}a^2_3)]-y)^2=a^2_1(a^3_1-y)\sigma'(z^3_1)$

Where am I going wrong?

My mistake was silly. I did not notice in the book that the weight matrices are numbered as the layer they are going into..so my $w^2$ above should be $w^3$ and then the result from the book is $\frac{\partial }{\partial w^3_{11}}=a^2_1(a^3_1-y)\sigma'(z^3_1)$ exactly as calculated by just doing the derivatives.


Are stacked NN the second generation of NN? Spiking Neural Networks are said to be the NN's third generation. Feed-Forward NN are the first. What is the second Generation? Stacked NN?

No.

Kumar, Satish. Neural networks: a classroom approach. Tata McGraw-Hill Education, 2004.:

  The first generation of neural network models employed McCulloch-Pitts TLN type neuron. They were essentially switching networks. The second generation of neural network models employed a smooth sigmoids. The sigmoidal function was motivated by the increase in firing rate of a neuron with the net input, which is saturates at a certain maximum frequency. The third generation of neural network models will use spiking neurons which encode information in spike trains.


Multiple variable as input and output I'm trying to predict the possible diagnosis given a consultation reason. I have ID's for all the data. So my data kind of looks like below

Reason            | Diagnosis
------------------------------
448, 124          | 9
551, 448, 122     | 9, 12
111, 110          | 32
143               | 43

There can be up to 10 reasons and upto 5 diagnoses in my training data.

What I'm looking for in the algorithm or model is that it accepts 1 - 10 reason_ids as input and returns top 5 possibilities for diagnosis with % of probability.

I'm good at python so if there is any open source model or code I can look at it will be great.

You could approach this in two ways:

Association-rule learning

Intuitively, it's like trying to infer rules such that: if the reasons are X and Y, the diagnosis will likely be Z.

The most common algorithm is Apriori, which is easy to implement.

Classification/Supervised learning

Your features could be binary (0 if reason X is not applied, 1 if it is). You could have one for each reason, and even combination of reasons etc. You might run into some issues if you have many possible reasons (your features will be very sparse).

If this is the case, I'd look into dimensionality reduction to make your features denser.


How do I integrate Github files to Orange for ML? I am currently trying to make a facial recognition workflow with Orange but since the widgets are still prototypes they are not all available in Orange. I found the Github files for the widgets but don't know how to install this so that I can see and use the widgets in Orange.

There are three ways to install Orange add-ons:

The Options -> Add-ons menu of Orange.

Use pip: pip3 install orange3-prototypes

Install from the source:
Terminal commands:
git clone https://github.com/biolab/orange3-prototypes.git
cd orange3-prototypes
python3 setup.py install


Difference between machine learning and artificial intelligence Is there any difference between machine learning and artificial intelligence? Or do these terms refer to the same thing?

The subject areas Artifical Intelligence and Machine Learning (plus Data Science) are loosely defined, such that it is hard to make strict statements about how they relate. In the general case, it seems that there are parts that overlap, but that they are quite far from being "the same subject with two different names" as suggested in the question.

The term Artifical Intelligence has many possible meanings and interpretations - which version to refer varies by time and by the source using it. Textbooks on artificial intelligence will often cover topics such as search algorithms, logical deduction and other things which are clearly not machine learning as it is practised today.

For instance, we could take it to refer to Artificial General Intelligence (or "hard AI"), and it should be clear in this case that at least some form of learning algorithm(s) would be required to meet the goals of AGI. However, it is far less clear how much of AGI can be solved by combining machine learning into complex structures.

The term Machine Learning has a few different working definitions, but this is a popular one:

  A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E.

This is far more tightly defined than Artificial Intelligence, but still has a lot of scope.

The trend to conflate AI and ML appears to be a media and marketing issue, not a technical one. I suspect this is in part due to advances in the last 5-10 years in neural networks. Neural network models have made strong progress, especially in signal processing of images, video, audio. There is also an analogy with biological brains which can be compelling - especially when the subject matter is simplified for consumption by mainstream media.

It is worth mentioning Data Science too. Like Artificial Intelligence, the term is somewhat fuzzily defined. Also like AI, Data Science has more to it than just Machine Learning. To Data Science practitioners, ML is part of a toolkit to achieve goals - for some people it is a large part of what they do, for others it is just one part of a wider scope (actually training and refining a ML model might take only a small fraction of a professional data scientist, analyst or statistician's time). I think it is reasonable to state that Artificial Intelligence and Data Science relate to Machine Learning in a similar way.


Approaches to high dimension pattern matching problem My apologies in advance as I am new to this. I have searched the internet and tried various processes and nothing seems to work or address this situation.

I have a dataset of 30,000 transactions and 500,000 items. Average item size for a transaction is 50. The dataset is sparse, so the support number must be set quite low. Furthermore, the rules become more valuable the larger the number of items in the rule.

I have tried running this in arules and the tests fail after exceeding 64 gb of RAM (the limit of the machine). I have tried reducing items and transactions to smaller subsets, but still hit this memory limit.

Ultimately, I am looking for ways to cluster large groups of similar accounts by selection of items and generate confidence and lift of various next items selected from those clusters.

My question: are there alternative, more efficient ways to do this, or other approaches to consider?

Thank you.

According to the following discussion on StackOverflow, a situation like that you've described can occur, when one of the variables in a data set is of unexpected type (for example, a factor instead of a character): https://stackoverflow.com/q/7246412/2872891.

Also, consider using package bigmemory, recommended in the accepted answer, or similar packages for big data analysis. For the latter, please see section "Large memory and out-of-memory data" in CRAN Task View High-Performance and Parallel Computing with R.

Finally, an additional note. There is an ecosystem of R packages, built around the arules package, which includes supporting packages for algorithms (arulesNBMiner), applications (arulesSequences, arulesClassify) and visualization (arulesViz). You are likely aware of that, but I have decided to include this good-to-know fact just in case and for the sake of completeness.


Keras loading images in incorrect format So I was working with the the vgg16 model for dogs vs cats classification and I noticed that keras is not loading images in correct color format.
The code is as follows:

import cv2
import matplotlib.pyplot as plt
import numpy as np
from PIL import Image
from keras.preprocessing import image

path='data/dogscats/sample/train/dogs/dog.1402.jpg'

imgkeras=image.load_img(path)
imgkeras=image.img_to_array(imgkeras)

plt.imshow(imgkeras)
plt.show()

The output of the following code is

Where as the original image is

Can someone explain why is this happening? , also when the image is loaded through opencv and fed into vgg16 the predicted label is more accurate for this particular image than when it is loaded through keras as above,is the improper color format affecting that?

This is caused due to the img_to_array method which converts the image to a float32 array.

x = np.asarray(img, dtype=K.floatx())

Matplotlib interprets NxMx3 uint8 array as a standard image (0..255 components) in which case there is no preprocessing. Otherwise the pixels are multiplied by 255(without checking the range) and then cast into uint8, which I guess leads to this behaviour.

Check this:https://stackoverflow.com/questions/39925420/bizzare-matplotlib-behaviour-in-displaying-images-cast-as-floats

To answer the second part of your question, I guess the imagenet competitors used OpenCV to load images in BGR format to train vgg16 and hence the pretrained weights work well with images opened in BGR format.


Reason for capping Learning Rate (alpha) up to 1 for Gradient Descent I am learning to implement Gradient Descent algorithm in Python and came across the problem of selecting the right learning rate.

I have learned that learning rates are usually selected up to 1 (Andrew Ng's Machine Learning course). But for curiosity reasons, I have tried alpha = 1.1 and alpha = 1.2.

I can see in the case of alpha = 1.2, we reach the lower cost faster than the other learning rates (simply because the curve touches the bottom first). Is it safe to say that alpha = 1.2 is the best rate?

I plugged in the theta values, where alpha = 1.2, to predict the price of an item, my implemented function provided the same answer as Sklearn's LinearRegression() in lesser iterations than it did with alpha = 1.0. Using lower alpha rates would increase the number of iterations.

So, why is the learning rate capped at 1? Is it mandatory or suggested?

Should I forget about selecting learning rates and let functions like LinearRegression() take care of it automatically in the future?

I am new to machine learning and I want to understand the reasoning behind the algorithms rather than calling the functions blindly and playing around with parameters using high-level libraries.
Feel free to correct me if I have understood the concepts wrong.

Setting a hard cap on the learning rate, for example at alpha = 1, is certainly not mandatory. It is also not necessarily advisable to set such a cap, as the merits of using different values for the learning rate are highly dependent on the exact function upon which you are performing gradient descent, what you hope to achieve in doing so, and what measures you will use to measure the relative success of one value choice over another.

I think the information you provided demonstrates this concept well. For example, if all you care about is moving towards some local minimum of your cost function, ultimately finding parameters for your model that achieve a cost less than say .01, and all else being equal accomplishing these tasks in the least number of iterations possible, we can see that among the values you tried alpha = 1.2 is indeed the best value (among the runs you showed us, it reached the cost of .01 in the least number of iterations). However, many people care about other properties of their gradient descent algorithms. For example, one may prefer a learning rate which is more likely to arrive at whichever (if any) local minima is nearest to the initialized parameters; lower learning rates seem better suited for this goal, since a high learning rate has a higher potential of 'overshooting' one minimum and landing in the basin of another. Or one may prefer a learning rate which produces a very smooth looking cost over time graph; lower learning rates seem better suited for this goal too (for an anecdotal example, your alpha = .03 learning curve looks smoothest).

There are many resources and methods available for choosing "ideal" learning rates and schedules out there, and I think it is worthwhile to read up on them to get a flavor for what people typically do. Most suggestions are heuristic, and not guaranteed to be meaningful in any particular example. Setting a cap of alpha = 1 is one such heuristic, and is probably suggested because it has been useful for many people with a lot of experience. Since many people have devoted significant time to studying this question, I don't think it is necessarily a bad idea to postpone thinking too hard on the topic when one first uses gradient descent, and instead just use the defaults in things such as scikit-learn's implementations, or take suggestions such as never setting alpha larger than 1. Personally, though, I share your desire to not blindly use defaults when I have the time to think on alternatives, and think it would be informative (if potentially not useful) to spend time investigating exactly how learning rate choices affect the goals you have in your gradient descent implementation.


Tensorflow MirroredStrategy() looks like it may only be working on one GPU? I finally got a computer with 2 gpus, and tested out https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html and https://github.com/tensorflow/models/tree/master/tutorials/image/cifar10_estimator and confirmed that both gpus are being utilized in each(The wattage increases to 160-180 on both, Memory is almost maxed out on both, and GPU-Util increased to about 45% on both at the same time).

So I decided I would try out tensorflow's MirroredStrategy() on an exitsting neural net I had trained with one GPU in the past.

What I don't understand is that the wattage increases in both, and the memory is pretty much maxed in both but only one GPU looks like it is being utilized at 98% and the other one just chills at 3%.  Am I messing something up in my code?  Or is this working as designed?

strategy = tensorflow.distribute.MirroredStrategy()
with strategy.scope():
    model = tensorflow.keras.models.Sequential([
        tensorflow.keras.layers.Dense(units=427, kernel_initializer='uniform', activation='relu', input_dim=853),
        tensorflow.keras.layers.Dense(units=427, kernel_initializer='uniform',activation='relu'),
        tensorflow.keras.layers.Dense(units=1, kernel_initializer='uniform', activation='sigmoid')])
    model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
    model.fit(X_train, y_train, batch_size=1000, epochs=100)

nvidia-smi:

Fri Nov 22 09:26:21 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 435.21       Driver Version: 435.21       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  TITAN Xp COLLEC...  Off  | 00000000:0A:00.0 Off |                  N/A |
| 24%   47C    P2    81W / 250W |  11733MiB / 12196MiB |     98%      Default |
+-------------------------------+----------------------+----------------------+
|   1  TITAN Xp COLLEC...  Off  | 00000000:41:00.0  On |                  N/A |
| 28%   51C    P2    64W / 250W |  11736MiB / 12187MiB |      3%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      2506      C   python3                                    11721MiB |
|    1      1312      G   /usr/lib/xorg/Xorg                            18MiB |
|    1      1353      G   /usr/bin/gnome-shell                          51MiB |
|    1      1620      G   /usr/lib/xorg/Xorg                           108MiB |
|    1      1751      G   /usr/bin/gnome-shell                          72MiB |
|    1      2506      C   python3                                    11473MiB |
+-----------------------------------------------------------------------------+

I'm seeing the same thing, here I enabled just the first 2 GPU's.
If I force it to use just 1 GPU the memory utilization drops but the amount of time per epoch is unchanged.


Non-linear regression line fit I performed a regression analysis with two datasets, each of which has size 50. One dataset is called Spatial % and the other Min values, and I wanted to check whether the two are correlated. I did the analysis in SPSS and the resulting scatterplot is as follows:

I am not that much experienced but it seems to me that a line is not the perfect fit for this scatterplot. Would a power line fit better? Or what else do you suggest?

I tried to estimate a few of your data values from the scatter plot you provided.

Then I performed a power model regression and came up with

$y = (5.777 \cdot 10^{-16}) \cdot 1.404^{x}.$

The estimated values I used are below.

$(70, 0.01), (75, 0.012), (80, 0.015), (90, 0.025), (95, 0.075), (98, 0.15), (99, 0.20), (99.5, 0.25), (99.9, 0.32)$

Of course, your actual model will differ because you have the actual data set. I just eyeballed a few points so I could test the power fit.


Bayesian network in Python: both construction and sampling For a project, I need to create synthetic categorical data containing specific dependencies between the attributes. This can be done by sampling from a pre-defined Bayesian Network. After some exploration on the internet, I found that Pomegranate is a good package for Bayesian Networks, however - as far as I'm concerned - it seems unpossible to sample from such a pre-defined Bayesian Network. As an example, model.sample() raises a NotImplementedError (despite this solution says so).

Does anyone know if there exists a library which provides a good interface for the construction and sampling of/from a Bayesian network?

Please use the function from_samples() to build a Bayesian n/w from the data.


How to classify named entities of the same type? I am doing a project where I am extracting date/time entities from text. I'm using a rule-based system to extract the temporal expressions and ground them to an actual date/time.

The second part of the problem I hope to solve is label the role of each entity discovered. For example, consider the following text: "Leaving at 2pm and back at 4pm". I correctly identified 2pm and 4pm as date/time entities. However, I'm unable to say whether the entity is "start-time", "end-time", or neither.

The question is how do I do this?

I'm new to NLP and ML. Here is an idea I have please tell me if I'm going the right direction:

The plan is to train a logistic regression (or naive bayes?) classifier  using the following features:

The average of the word embedding for each word within a window of the date/time phrase.
The POS tags for each word within a window of the date/time phrase??(Not sure how to pass this in to a logistic regression classifier but just a thought)
The word shapes of the words in the temporal expression??

I'm a little confused as to where to start and would really appreciate some pointers on how to select my features and what classifier would be appropriate.

I'm also open to suggestions on learning resources. There's a lot of NER resources online but not many on how to "role classify" found entities.

You might be interested in resources built around TimeML, I think there are some corpora and specific parsers specialized in extracting the time details of events. I don't remember any specifics but I tried to google "timeml extract time" and found a few related resources, that might give you at least some inspiration about how people have dealth with similar problems.

If you don't find anything that suits your need, in general the best approach would be to train a custom NER using your own annotated dataset with "start-time", "end-time", "neither" labels.


What is the top-5 error rate in ILSVRC challenge? While I'm reading the deep learning book, I came across something as "top-5  error rate". I want to know what is it really about and how it is evaluated in   ILSVRC (ImageNet Large Scale Visual Recognition challenge) ?

The Top-5 error rate is the percentage of test examples for which the correct class was not in the top 5 predicted classes.

So, for example, if a test image is a picture of a Persian cat, and the top 5 predicted classes in order are [Pomeranian (0.4), mongoose (0.25), dingo (0.15), Persian cat (0.1), tabby cat (0.02)], then it is still treated as being 'correct' because the actual class is in the top 5 predicted classes for this test image.

Remember to keep in mind that ImageNet has 1000 classes, so a low top-5 error rate is still not extremely easy to achieve.


Misclassification Rate for Random Forest Plateauing too Early Using R, I have created 5 different random forest models using 5 different numbers of trees (3,10,30,100,300). My intention was to compute the misclassification rates of each of these models and plot the rates against the number of trees to illustrate the idea that generally, an increase in trees in a random forest model correlates with a decreasing misclassification rate.

I had a few colleagues run this same model in Python and with all of them, their model reached a misclassification rate of ~0.08 with the 300-tree model. However, When I run my models in R, the misclassification rate seems to level out around ~0.2 at the 100-tree model, and does not get any lower with the ~300 tree model. I'm curious as to what may be causing this discrepancy. I've provided my code below.

madelon_train <- data.frame(madelon_train_data, madelon_train_labels)
for(i in c(3,10,30,100,300)){
    assign(paste("madelonforest", i, sep = ""),
    randomForest(as.factor(madelon_train$V1.1) ~ ., data = madelon_train, ntree =
    i, mtry = sqrt(500), replace = FALSE))
}

modellist <- vector(mode="list", length=5)
for(i in c(3,10,30,100,300)){
    modellist[[i]] <- eval(as.name(paste("madelonforest", i, sep = "")))
}

#Use models to predict training data and compute misclassification error

classerrlisttrain <- vector(mode="list", length=5)
for(i in c(3,10,30,100,300)){
    err <-table(as.numeric(as.character(predict(modellist[[i]],
    madelon_train_data, type = 'class', OOB = TRUE))) - madelon_train_labels)
    classerrlisttrain[[i]] <- assign(paste("misclassification", i, sep = ""),
err[names(err)==0])
}

for(i in c(3,10,30,100,300)){
    classerrlisttrain[[i]] = as.double(classerrlisttrain[[i]])
    classerrlisttrain[[i]] = 1 -
classerrlisttrain[[i]]/length(madelon_train_labels$V1)
}

#Use models to predict test data and compute misclassification error

classerrlisttest <- vector(mode="list", length=5)
for(i in c(3,10,30,100,300)){
    err <-table(as.numeric(as.character(predict(modellist[[i]],
    madelon_valid_data, type = 'class'))) - madelon_valid_labels)
    classerrlisttest[[i]] <- assign(paste("misclassification", i, sep = ""),
err[names(err)==0])
}

for(i in c(3,10,30,100,300)){
    classerrlisttest[[i]] = as.double(classerrlisttest[[i]])
    classerrlisttest[[i]] = 1 -
classerrlisttest[[i]]/length(madelon_valid_labels$V1)
}

#Plot misclassification errors vs Tree Depth

plot(c(3,10,30,100,300), classerrlisttrain[c(3,10,30,100,300)], type = 'l',
xlab = 'Number of Trees', ylab = 'Misclassification Rate', xlim = c(1,300),
ylim = c(0,0.5), col = "red")
lines(c(3,10,30,100,300), classerrlisttest[c(3,10,30,100,300)], type = 'l',
col = "blue")
legend(1,0.1,legend = c("Train Data", "Test Data"), col =
c("red","blue"),lty=1, cex=0.8)

If you and your colleagues ran the same model on the same data you should get the same results (give or take a stochastic error). Did your colleagues use the same environment, same packages and same versions?

Also, it is known that building more trees gives better performance and if possible you should build more not less, as RF does not overfit with more trees, the error / accuracy stabilizes at some point. What that point is (number of trees) varies from data to data, so you cannot really determine this beforehand.


Reinforcement Learning control with known dynamic equation I know there is model-based reinforcement learning. But all the approaches assume an MDP.

If I want to do a feedback control of a system (i. e. control an inverted pendulum) it's quite easy to find the nonlinear differential equation. Can I somehow feed this knowledge into RL-algorithms or are there ways to transform a dynamic system to an MDP?

Reinforcement learning (RL) is completely based around MDPs, to the point where its definition is essentially "RL is a collection of algorithms that can learn about action choices within a MDP environment".

Outside of RL, you can work with control systems using differential equations more or less directly, and some are solvable analytically. In principle these direct solutions are more robust and require no learning, compared to RL. However, they usually rely on having simple to describe goals - typically static control of maintaining some important value (speed, position, temperature). This works fine for cruise control in cars, thermostats and industrial processes. It also works great for simple environments like inverted pendulum which has been solved without RL for decades.

The analytical non-RL approaches are far more trusted than trial-and-error statistical learners such as RL. But they are limited in terms of describing goals and complexity of environments. They start to fail at the level of the mountain car environment where the correct action may be to move further from a goal state before moving towards it. Mountain car can still of course be fully described by relatively simple differential equations.

If you have equations for a dynamic system in an environment with goals that are too complex to solve analytically, then you can easily convert to discrete MDP form: Use the equations to simulate the environment, choosing a discrete time step for action choice. There are also RL methods that can work with continuous  control and variable time steps, which would also benefit from such a model. You could use such a model to learn in a simulated environment, or use it to help with planning in a real environment (or have a learning + planning algorithm in a simulated environment).

If you are starting with differential equations, then typically you would convert them into some non-differential form in order to apply them. If you can do this fully analytically - e.g. change something in the form $a\frac{d^2x}{dt}  + b\frac{dx}{dt} +c = 0$ into some $x = \alpha e^{\beta t}\text{sin}(\gamma t)$ - then this would make the most accurate predictions and simulations. Otherwise you can use some method of approximation in order to resolve the differential equation into something that predicts next state from current state.


Replacing mean by median over batch-size to lessen the impact of outliers In the case of training a Neural Network on a regression task. Assuming the data has a significant amount of outliers. Provided that the error needs to be RMS and not MAE. Can it be better (as in less sensitive to the outliers) to replace the average over batch size in the weights update by a median over batch size computation?

For a batch size large enough, this should lessen the impact the contribution from the outliers. It does not seem to be common though, at least to current knowledge. What are the shortcomings of this approach?

One shortcoming is that the median is often more computational expensive to calculate than the mean. The median can be calculated with a variation of quickselect which is linear worst-case performance. Calculating the mean only requires the sum and count of the numbers.


Understanding the Shuffle and Split Process in a Neural Network Code I need to understand how the splitting of labels in the following code happened :

import keras
import librosa
import librosa.feature
import librosa.display
import glob
import numpy as np
from keras.models import Sequential
from keras.layers import Dense , Activation
from keras.utils.np_utils import to_categorical

def extract_features_song(f):
    y, _ = librosa.load(f)

    # get mfcc
    mfcc = librosa.feature.mfcc(y)

    # make value between 1 -1
    mfcc /= np.amax(np.absolute(mfcc))
    return np.ndarray.flatten(mfcc)[:25000]

def extrac_features_and_labels():
    all_features = []
    all_labels = []

    genres = ['blues' , 'classical', 'country' , 'disco' , 'hiphop', 'jazz', 'metal' , 'pop', 'reggae', 'rock']
    for genre in genres:

        sound_files = glob.glob('genres/'+genre+'/*.au')

        print ('prcoessing %d songs in %s genre'% (len(sound_files), genre))

        for f in sound_files:
            features =extract_features_song(f)
            all_features.append(features)
            all_labels.append(genre)

    # one hot encoding
    label_uniq_ids , label_row_ids = np.unique(all_labels, return_inverse= True)
    label_row_ids = label_row_ids.astype(np.int32, copy= False)
    onehot_labels = to_categorical(label_row_ids, len(label_uniq_ids))

    return np.stack(all_features), onehot_labels

features , labels = extrac_features_and_labels()

print (np.shape(features))
print (np.shape(labels))

training_split = 0.8

alldata = np.column_stack(features , labels)

np.random.shuffle(alldata)
splitidx = int(len(alldata))*training_split
train , test = alldata[:splitidx,:], alldata[splitidx:, :]

print (np.shape(train))
print (np.shape(test))

# the concerned part: begin
train_input = train [:,:-10]
train_labels = train [:,-10:]

test_input = test [:,:-10]
test_labels = test [:,-10:]
#the concerned part: end

print (np.shape(train_input))
print (np.shape(train_labels))

the output is as follows:

(1000, 25000)
(1000, 10)
(800, 25010)
(200, 25010)
(800, 25000)
(800, 10)

Now, when he - code instructor - stacked both arrays together and shuffled them, he can't be sure that the last ten elements are the labels, am I right? If so how did he do it like this, this would cause an error.


What is the relationship between clustering and association rule mining? Cluster analysis or clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense or another) to each other than to those in other groups (clusters).

Source: Wikipedia

  Association rule learning is a method for discovering interesting relations between variables in large databases.

Source: Wikipedia

So both, clustering and association rule mining (ARM), are in the field of unsupervised machine learning. Clustering is about the data points, ARM is about finding relationships between the attributes of those datapoints.

However, I wonder if there are more relationships. For example, given a clustering, can this enhance / simplify ARM or vice versa?

First of all, the problem of finding frequent itemsets (ARM) differs from the similarity search (clustering).

If you think about market baskets, clustering tries to find objects that have a large fraction of their baskets in common, the absolute number of those objects (basket occurences) is not of interest.
With ARM we are interested in the absolute number of baskets that contain a particular set of items.

CLustering:  Allocates objects in such a way that objects in the same group (called a cluster) are more similar (given a distance metric) to each other than to those in other groups (clusters).

ARM: Given many baskets (could be actual supermarket baskets) find which items inside a basket predict another item in the basket.

Sources


What is the most common practice of generating (X,Y) from an arbitrary CDF or PDF? So I can generate X from arbitrary CDF F(x) by the procedure above. Can it be generalized to two variables? How, exactly? If not, what's the best way to generate (X,Y) from arbitrary CDF F(x,y) or PDF f(x,y)?

I am no expert in the field but I can imagine 3 different cases :

Variables are independent in which case you can use your formula on each
You have access to some chain rules decomposition that allows you to easily generate each variable one at a time using your formula
If none of the above : https://appsilon.com/how-to-sample-from-multidimensional-distributions-using-gibbs-sampling/


NLP data cleaning and word tokenizing I am new to NLP and have a dataset that has a bunch of (social media) messages on which I would like to try some methods like latent Dirichlet allocation (LDA). First, I need to clean the data of things like punctuation, emojis, etc. I'm not sure how to go about doing this in the most efficient and accurate manner. My code right now is this:
import pandas as pd
import re

class TopicModel():
    def __init__(self, data_path = "data.csv"):
        self.data_path = data_path
        self.data = pd.read_csv(self.data_path, low_memory=False)

    def clean_data(self):
        self.remove_message_na()
        self.remove_emojis()
        self.remove_punctuation_and_lower()
        self.remove_url()
        self.remove_empty_messages()

    def remove_message_na(self):
        self.data = self.data.loc[~pd.isna(self.data['message'])]

    def remove_emojis(self):
        self.data['message'] = self.data['message'].str.encode("ascii", "ignore").str.decode("utf8")

    def remove_punctuation_and_lower(self):
        p = re.compile('''[!#?,.:";]''')
        self.data['cleaned_data'] = [p.sub("", ii).lower() for ii in self.data['message'].tolist()]

    def remove_empty_messages(self):
        self.data = self.data.loc[self.data['cleaned_data'] != ""]

    def remove_url(self):
        self.data = [re.sub(r"http\S+", "", ii) for ii in self.data['message'].tolist()]

I don't want to remove contractions, which is why I left out ' from my punctuation list, but I think optimally, the contractions would be reformatted as two separate words. I also wonder about the other punctuation marks when dealing with social media data, e.g., #. I know this question is a bit general, but I'm wondering if there is a good python library for performing the kind of data-cleaning operations that I want, prior to perform topic analysis, sentiment analysis, etc. I'd also like to know which libraries can efficiently perform these data-cleaning operations on a pandas data frame.

I summarize your questions and then try to answer under each bullet point:

How to remove punctuation marks (e.g. # for hashtags which is used in social media)

The first goto is a regular expression that is used in data preprocessing very frequently. But if you are looking for all your punctuation to be removed from the text you can use one of these two approaches:
import string
sentence = "hi; how*, @are^ you? wow!!!"
sentence = sentence.translate(sentence.maketrans('', '', string.punctuation))
print(sentence)

output: 'hi how are you wow'
or use a regular expression, example:
import re
s = "string. With. Punctuation? 3.2"
s = re.sub(r'[^\w\s]','',s)
print(s)

output: 'string With Punctuation 32'

What are the possible python library for data cleaning?

generally, NLTK and SciPy are very handly. But for specific purposes, other libraries also exist. For example library contractions, inflect.

Which libraries are efficient in data cleaning when using pandas dataframes?

You can apply any function from any library to your pandas dataframe using the apply method.
Here is an example:
mport pandas as pd
s = pd.read_csv("stock.csv", squeeze = True)

# adding 5 to each value
new = s.apply(lambda num : num + 5)

code source


Using BERT for the first time, what are the two columns of my test_results.tsv? I followed the steps to feed in both dev, test, train.tsv to the model, trained it, then tried to classify test data, and I only have 1 feature, and the classification is binary, 1 or 0. I assumed my test_results.tsv would have just 1 column, representing the estimated classification. Any advice? Here is a picture of my output.  I thought it would be a column of 1's and 0's showing me its estimated classification.

If you look at your output you will see that the two values in every row add up to 1. This means that the first value is the probability of the output having value 0 in your classification, and the second value the probability of the out having value 1. If you want to only have the 0/1 classification instead of the probabilities you can use a simply threshold for the probabilities (i.e. if value < threshold then 0, else 1). The simplest way to set the threshold could be 0.5, but you could try changing threshold and see how the performance on your test set changes.


Problem building dictionary from series I have a pandas dataframe with a column CAS_BRM_IDA of type category (even if its values seem of foat type ... but they are not meaningful per se)

I built the following serie from that dataframe :

p = df.groupby( [ 'CAS_BRM_IDA' ] ).mean()[ 'TOP_FRD']
print( p )

Result displayed :

CAS_BRM_IDA
10.0       0.001131
13.0       0.000000
15.0       0.002038
17.0       0.000000
20.0       0.003802
...
missing    0.019549

I then tried to build a dictionary using that serie :

mydic = dict( [ ( i , p[i] ) for i in p.index ] )

But I got the following error message :

TypeError: cannot do label indexing on
  <class 'pandas.core.indexes.category.CategoricalIndex'>
   with these indexers [10.0] of <class 'float'>

Some details about the p index :

CategoricalIndex(
    [10.0, 13.0, 15.0, 17.0, 20.0, 21.0, 30.0, 31.0, 40.0, 43.0,
     50.0, 51.0, 56.0, 'missing'],
     categories=[10.0, 13.0, 15.0, 17.0, 20.0, 21.0, 30.0, 31.0, ...],
     ordered=False, name='CAS_BRM_IDA', dtype='category')

I can't figure out where is the problem.  '10.0' value seems to have been interpreted as float which I thought it's not.

How can I build a dict from a Series with a Categorical Index which contains floats?

Try the intrinsic pandas conversion:

p.to_dict()


Learning character sequences and predicting sequences I'm novice to deep learning and sorry if these questions may look very basic.

******************* First Question *******************

I need to predict the next k-characters given some initial text input, where the algorithm is learnt on a piece of text (character sequence).
I've done some work on predicting only 1-character at a time (using LSTM and Keras wrapper), but my question is how to generate k-characters and not only one. Of course I can generate k-characters recursively using only 1-character prediction algorithm, but I'm wondering if this problem (input text to k-characters) is well known and there exist a state-of-the-art solution.

What I learnt from reading relevant topics, is that this problem is (similar) to "sequence-to-sequence" learning and of course there are many solutions and implementations available (theano, tensorflow, etc).
Am I correct?

For example, given a string of letters:

seq_1 = 'aaaaabababcbcbbababbbdcbbbaaabbbaad...'

the algorithm should predict (given a randomly picked input sequence):

aaabbb -> aad

(the last 9 characters from the aforementioned example  )

******************* Second Question *******************

I know, that neural network algorithms are very "data hungry" and in order to make meaningful learning and prediction accuracy, one has to provide a lot of data (compared to shallow learning methods)
If I have only relatively short strings of letters (50-100 characters), does it make any sense to train seq2seq algorithm (LSTMs network) on such short sequences, but with many bootstrapped random instances?

Are there any theoretical (or some rules of thumb) bounds on the size of the training set and the reliability of the trained algorithm? (overfitting)

I'm quite motivated by the example of learning temporal patterns in alphabet with LSTM RNNs:

The author got very nice results, by random sampling of sequences of varied length and showing them to LSTM RNN. The idea was to randomly generate a relatively large (~1000 instances) data set and then train the network.

Can anyone comment on this way (bootstrapping) of creating a training set from a small sequence? Is it a standard approach? Are there any statistical flaws?
Many thanks!!

I think you might want to check Andrej Karpathy's work around charRNNs some pretty cool work is being done. The github link has all the relevant code too.
If you are looking for a more applied way - you can check my blog on deep learning gender from name which effectively uses character level LSTM RNNs to learn patterns :
medium.com/@prdeepak.babu/deep-learning-gender-from-name-lstm-recurrent-neural-networks-448d64553044

Hope this help.


Confusion matrix - determine the values of FP FN TP and TN After running my code ,I get the values of accuracy, precision and recall and I want t determine the values of FP FN TP and TN from these metrics. I tried to calculate it using the formula of each metric but I couldn't. Is there any way to do this?

You can!

The trick is that you actually know two other critical variables: the number of positive and negative examples (P and N). You can then use them to algebraically solve for the confusion matrix:

$ recall=\frac{TP}{TP+FN}=1-\frac{FN}{P}\Rightarrow $

$ FN = P(1-recall) $

$ recall=\frac{TP}{TP+FN}\Rightarrow (recall)(TP+FN)=TP\Rightarrow TP(1-recall)=FN\Rightarrow$

$ TP=\frac{FN}{1-recall} $
or simply: $ TP=P-FN $

$ accuracy = \frac{TP+TN}{P+N}\Rightarrow (accuracy)(P+N)=TP+TN\Rightarrow $

$ TN=(accuracy)(P+N)-TP $

$ accuracy = \frac{TP+TN}{TP+TN+FP+FN}\Rightarrow (accuracy)(TP+TN+FP+FN)=TP+TN\Rightarrow $

$ FP=\frac{TP+FN}{accuracy}-TP-TN-FN $
or simply: $ FP=N-TN $


Which method to use to remove trend from time series? From what I understand, differencing is necessary to remove the trend and seasonality of a time series. So I assumed it basically does the same thing as signal.detrend from the scipy library.

But I tried differencing and then, separately, used signal.detrend and my time series looked completely different.

Original:

Differencing:

Imported libraries:

The x axis represents months and the y axis is sales. The colours on the first two charts just represent three different years.

Detrend does a least squares fit (linear or constant) and subtracts this from your data points. You can look this up in the docs.

Simply taking the difference between consecutive data points will in general lead to other results.

In general the regression based detrending seems to be more reasonable. You could also think about using random sample consensus (RANSAC) to be more robust to outliers.


Why ELMo's word embedding can represent the word better than glove? I have read the code of ELMo.
Based on my understanding, ELMo first init an word embedding matrix A for all the word and then add LSTM B, at end use the LSTM B's outputs to predict each word's next word.
I am wondering why we can input each word in the vocab and get the final word representation from the word embedding matrix A after training.
It seems that we lost the information of LSTM B.
Why the embedding can contains the information we want in the language model.
Why the training process can inject the information for a good word representation into the word embedding matrix A?

I am wrong. ELMo also use the output of LSTM for context-dependent representation.

The output only from word embedding is the context-independent representation.

Why the representation is useful?

I think it is because, it is learning the difference between words and the representation is not the real meaning for the word.