Categorical cross-entropy works wrong with one-hot encoded features I'n struggling with categorical_crossentropy problem with one-hot encoding data. The problem is in unchanged output of code presenting below:
            inputs = keras.Input(shape=(1190,), sparse=True)
            lay_1 = layers.Dense(1190, activation='relu')
            x = lay_1(inputs)
            x = layers.Dense(10, activation='relu')(x)
            out = layers.Dense(1, activation='sigmoid')(x)
            self.model = keras.Model(inputs, out, name='SimpleD2Dense')
            self.model.compile(
                optimizer=keras.optimizers.Adam(),
                loss=tf.losses.categorical_crossentropy,
                metrics=['accuracy']
            )

Epoch 1/3
1572/1572 - 6s - loss: 5.7709e-08 - accuracy: 0.5095 - val_loss: 7.0844e-08 - val_accuracy: 0.5543
Epoch 2/3
1572/1572 - 6s - loss: 5.7709e-08 - accuracy: 0.5095 - val_loss: 7.0844e-08 - val_accuracy: 0.5543
Epoch 3/3
1572/1572 - 7s - loss: 5.7709e-08 - accuracy: 0.5095 - val_loss: 7.0844e-08 - val_accuracy: 0.5543

Few words about data: 1190 features (10 actual features with 119 categories). The inputs are a dataframe rows with 1190 values per sample. Output is a binary value 0 or 1.
Attempts done before: binary_crossentropy used with satisfying results, however, number of samples is not enough to get good results on validation data. Tried to use different activations and layer sizes.
Main question is why categorical_crossentropy is not working and how to use it in right way.
Also, one concern appears about data representation is it right way to use in one rare row of straightforward one-hot encoded data?

For it to work -

Change output neurons count to 02
Activation of output to Softmax
Keep all the vectors of OHE output

This is how Keras is designed internally. Same has been written on the official documentation page

BinaryCrossentropy class
Use this cross-entropy loss when there are only two label classes (assumed to be 0 and 1). For each example, there should be a single floating-point value per prediction.
In the snippet below, each of the four examples has only a single floating-pointing value, and both y_pred and y_true have the shape [batch_size]

CategoricalCrossentropy class
The shape of both y_pred and y_true are [batch_size, num_classes]

And we know that to keep the Classification multi-class you need to make all the num_class output relative to each other, so we use softmax
Ref
Keras official page
Similar SE thread
Similar SE thread


How to calculate Temperature variable in softmax(boltzmann) exploration Hi I am developing a reinforcement learning agent for a continous state/discrete action space. I am trying to use boltmzann/softmax exploration as action selection strategy. My action space is of size 5000.

My implementation of boltzmann exploration:

def get_action(state,episode,temperature = 1):
    state_encod = np.reshape(state, [1, state_size])
    q_values = model.predict(state_encod)
    prob_act = np.empty(len(q_values[0]))

    for i in range(len(prob_act)):
        prob_act[i] = np.exp(q_values[0][i]/temperature)

    #numpy matrix element-wise division for denominator (sum of numerators)
    prob_act = np.true_divide(prob_act,sum(prob_act))

    action_q_value = np.random.choice(q_values[0],p=prob_act)
    action_keys = np.where(q_values[0] == action_q_value)
    action_key = action_keys[0][0]
    action = index_to_action_mapping[action_key]
    return action

If my temperature variable is 200, after 100 episodes I get an error

ValueError: probabilities contain NaN

If my temperature is 1 in very few episodes i get NaN error.

Why is this happening. Am I doing something wrong here? How to select the temperature variable? Can someone help me with this.

Thanks.

the reason you get error is that when temperature annealed, the softmax would obtain a big value which the exponential cannot calculate. The softmax would approximate the value into nan or 0 or 1 which causes the error. I got the same problem and still fixing it. In my case, I just make sure the reward is not too big which can guarantee the softmax a not big q value.


CNN to learn and visualize 2d features In the paper Large-Margin Softmax Loss for Convolutional Neural Networks the author has a figure as below:

He is claiming that he is using only 2d features to classify MNIST with a CNN. How is he doing that? Is he reducing the dimension of MNIST images to 2 and how? How do you visualize such figures in 2D if the classification is for 10 classes?

I think I got my answer from the following post:
http://unixnme.blogspot.com/2018/04/dimensionality-reduction-and-scattered.html


Suggestions for improvement? Time series of variation in relative frequency of emotion-related words in academic psychology over time First time plotting and interpreting time series data and I have used a line plot for ease of use. I am aware this is incredibly basic, but any input/ recommendations would be much appreciated (e.g., is anything unclear?).

My main concern is whether I have adequately displayed the data and
whether I can do anything useful to improve (e.g., moving average)?
Additionally, whether I have interpreted this time series data appropriately:

"The relative frequency of affect-related tokens (counts per 10,000 tokens in psychology abstracts) increased from 3.51% in 1980 to 4.87% in 2017—-an overall relative increase of 39%. The relative frequency of affect-related tokens shows an increase at a rate of approximately 0.037 units per year (over 37 years). Overall, this displays a rapid growth trend in academics' use of emotion-related terms in psychology abstracts over time."

I am aware this is incredibly basic, but any input/ recommendations would be much appreciated (e.g., is anything unclear?).

In general it is strongly recommended to communicate knowledge in the most simple way possible, as long as it's accurate. Yes a line plot is simple but there's nothing wrong with that, in this case I can't think of any better way to convey the observation accurately.

My main concern is whether I have adequately displayed the data and
whether I can do anything useful to improve (e.g., moving average)?

It perfectly shows the increasing trend which is your main point:

A moving average would not really help in this case: it's useful only when there's too much noise/variation in the data and the general trend is hard to see. Here you apparently have enough data and/or the data is stable enough, so there's simply no need.
You took care of making the plotted value a proportion per 10,000 tokens, and this is also a good idea since an increase in absolute value could be biased: if the total number of abstracts increases, the proportion of abstracts containing the target terms would naturally increase as well even though this doesn't show anything.

Overall this looks very good to me. Just a small note, in your description you could mention that the rate by year is an average: "approximately 0.037 units per year in average".


Can i use deep learning in my agriculture PHD? My PHD is about yield of soybeans and it is typical agriculture theme, but I am pretty good with programming and Python programming language and I have already some deep learning programs which I made by myself successfully. Problem is that model is not very easy to show on the paper.

Can I show model just like a chart or graph? And maybe somebody already solved issue with transporting deep learning model file to "paper"? Is it normal to use deep learning in my agriculture PHD instead common middle age Linear Regression etc?

Inputs and outputs is ok on paper but deep learning model is a little bit controversial in my scientific circles and I don't know how to present it so I'm asking maybe somebody already has successful experience by presenting deep learning model on the paper in science.

Please note that the following is my personal opinion; I still hope that you find it useful.

If you cannot properly explain a concept on paper, then it should not be part of your PhD thesis. It is possible to implement Deep Learning algorithms without any conceptual knowledge. While this may be (somewhat) acceptable in the industry, it will get you into trouble in academia.

I have no doubt that your algorithms are well-programmed and solve many issues that are challenging from a programmer's perspective. But that is not what the reviewers of your thesis are interested in. They are also not interested in the prediction accuracy of your algorithm.
As scientists, they want to know why something happens; and to my knowledge, Deep Learning algorithms provide next to nothing on this question.

And this is where 'middle age' linear regression has its strengths (by the way: a quick way to make an enemy is to tell him that the method he has been using all his life is too old to be useful). Yes, linear regression is unexciting, and it has many shortcomings. But it is extremely well understood. Most importantly, there are tons of books and papers about the connection between assumptions and valid interpretations: we know under which assumptions and in which sense the linear regression estimator is optimal; we know which assumptions lead to which kind of confidence intervals for the estimator; we know about robustness with respect to model misspecification; etc.

As long as you cannot address these issues convincingly and on paper (That is: In what sense is my result optimal? How confident can I be about my result? How robust is my model?), you should not use Deep Learning in research as an alternative to classical statistical methods.


How to incorporate the uncertainty of the model coefficients in the prediction interval of a multiple linear regression I'm dealing with the modeling of small experimental data sets. As most experimental work does not generate thousands of samples, but rather a handful, I need to be inventive in how to deal with this small number of data sets (say 10-20). I've been building a nice framework to do just this, and at this point I am interested in generating error-bars with the predicted values.

In a rough outline, this is what the happens in the framework (e.g. when applying a multi-linear model):

Create an ensemble of N data sets.
On each data set a regression gives rise to a (linear) model as given in Eq.1 below. This gives rise to N values for each of the coefficients $\beta$.
The mean is calculated for each of the three sets of $\beta$'s. (The mean could also be another function, but for now assume it's the mean)
These three mean $\beta$'s are the coefficients of the model to be used (again Eq.1).
The goal: find the prediction interval (PI) for the model in Eq.1 taking into account the fact that the coefficients $\beta$ are calculated from numerical distributions.

So take for example the following multiple linear regression model:
$$
y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 \tag{1}
$$
and I'm looking for an algebraic equation to calculate (numerically) the prediction interval (PI) for a new prediction $y_0$ (Confidence interval would be OK as well as it is related to the PI).

So far, my searches have only been able to provide me with answers which deal with the statistical nature of the data set ($x_i$'s). These provide me with an error component:
$$
\hat{V}_f=s^2\cdot\mathbf{x_0}\cdot\mathbf{(X^TX)^{-1}}\cdot\mathbf{x_0^T} + s^2 \tag{2}
$$
which can be used to calculate the PI, via:
$$
y=y_0 \pm t_{\alpha/2,n-k}\cdot\sqrt{\hat{V}_f} \tag{3}
$$

In contrast to those examples, each of the model coefficients ($\beta_0, \beta_1$ and $\beta_2$) in this case have an error-bar (extracted via bootstrapping from a distribution, with the distributions being numerical in nature not analytic, and the distributions are specific for each of the three coefficients).
Is there a way to incorporate the uncertainty of the $\beta_i$'s (c.q. the "error-bars") in the calculation of the PI (and CI).

Note
I Know, one could create an ensemble of the various model instances with the $\beta_i$ drawn from their respective distributions, and based on the distribution of obtained $y_0$ calculate the CI of the $y_0$, but this is not really computationally efficient and brings a lot of other issues which I would like to avoid.

One possible solution is Bayesian linear regression. Bayesian linear regression estimates a posterior distribution for each coefficient. From that posterior distribution, a credible interval can be calculated.


no decrease loss and val_loss I try to train a neural network for time series. I use some data from Covid, mainly the goal is knowing 14 days of number of people at hospital to predict the number at J+1.
I have use some early stopping to not over fit, but almost one time over two the learning stop at patience+1 and there is no decrease of loss and val_loss. I have tries to move hyperparameters like learning rate but the problem is always here. Any guess?
The main code is below and the whole code with data :https://github.com/paullaurain/prediction
import os
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
from keras.models import Sequential
from keras.layers import Dense
from keras.callbacks import EarlyStopping
from keras.callbacks import ModelCheckpoint
from keras.models import load_model
from keras.optimizers import Adam
from sklearn.preprocessing import MinMaxScaler

# fit a model
def model_fit(data, config):
# unpack config
    n_in,n_out, n_nodes, n_epochs, n_batch,p,pl = config
# prepare data
    DATA = series_to_supervised(data, n_in, n_out)
    X, Y = DATA[:, :-n_out], DATA[:, n_in:]
    X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.1)
# define model
    model = Sequential()
    model.add(Dense(4*n_nodes, activation= 'relu', input_dim=n_in))
    model.add(Dense(2*n_nodes, activation= 'relu'))
    model.add(Dense(n_nodes, activation= 'relu'))
    model.add(Dense(n_out, activation= 'relu'))
    model.compile(loss='mse' , optimizer='adam',metrics=['mse'])
# fit
    es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=p)
    file='best_modelDense.hdf5'
    mc = ModelCheckpoint(filepath=file, monitor='loss', mode='min', verbose=0, save_best_only=True)
    history=model.fit(X_train, y_train, validation_data=(X_test,y_test), epochs=n_epochs, verbose=0,batch_size=n_batch, callbacks=[es,mc])
    if pl:
        plt.plot(history.history['loss'])
        plt.plot(history.history['val_loss'])
        plt.title('model loss')
        plt.ylabel('loss')
        plt.xlabel('epoch')
        plt.legend(['train', 'test'], loc='upper left')
        plt.show()
    saved_model=load_model(file)
    os.remove(file)
    return history.history['val_loss'][-p], saved_model

# repeat evaluation of a config
def repeat_evaluate(data,n_test, config, n_repeat,plot):
    # rescale data
    scaler = MinMaxScaler(feature_range=(0, 1))
    scaler = scaler.fit(data)
    scaled_data=scaler.transform(data)
    scores=[]
    for _ in range(n_repeat):
        score, model= model_fit(scaled_data[:-n_test], config)
        scores.append(score)
    # plot the prediction id asked
        if plot:
            y=[]
            x=[]
            for i in range(n_test,0,-1):
                y.append(float(model.predict(scaled_data[-14-i:-i].reshape(1,14))))
                x.append(scaled_data[-i])
            X=scaler.inverse_transform(x)
            plt.plot(X)
            Y=scaler.inverse_transform(np.array([y]))
            plt.plot(Y.reshape(10,1))
            plt.title('result')
            plt.legend(['real', 'prdiction'], loc='upper left')

            plt.show()
    return scores

# summarize model performance
def summarize_scores(name, scores):
# print a summary
    scores_m, score_std = mean(scores), std(scores)
    print( '%s: %.3f RMSE (+/- %.3f)' % (name, scores_m, score_std))
    # box and whisker plot
    pyplot.boxplot(scores)
    pyplot.show()

#setting variable
n_in=14
n_out=1
n_repeat=5
n_test=10
# define config n_in, n_out, n_nodes, n_epochs, n_batch, pateince, draw loss
config = [n_in, n_out, 10, 2000, 50, 200,True]
# compute scores
scores = repeat_evaluate(data,n_test, config, n_repeat, True)
print(scores)
# summarize scores
summarize_scores('mlp ', scores)

Result:

Because the data is time series while only Dense layers are used in the model, the problem is caused by model initialization. A model with a 'bad' initialization will constantly predict zero, as you will see by running the script below.
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler
from tensorflow.keras.models import Model,Sequential
from tensorflow.keras.layers import Dense

import numpy as np
import tensorflow as tf

# fix keras random state
# https://stackoverflow.com/a/52897216/8366805
seed_val = 94
np.random.seed(seed_val)
tf.set_random_seed(seed_val)
# Configure a new global `tensorflow` session
session_conf = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)
sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)
tf.compat.v1.keras.backend.set_session(sess)

# main
def series_to_supervised(data,n_in,n_out):

    df = pd.DataFrame(data)
    cols = list()
    for i in range(n_in,0,-1): cols.append(df.shift(i))
    for i in range(0, n_out): cols.append(df.shift(-i))
    agg = pd.concat(cols,axis=1)
    agg.dropna(inplace=True)

    return agg.values

n_in,n_out = 14,1
data = np.load('data.npy')
scaler = MinMaxScaler(feature_range=(0, 1))
scaler = scaler.fit(data)
scaled_data=scaler.transform(data)
DATA = series_to_supervised(scaled_data[:-10], n_in, n_out)
X, Y = DATA[:, :-n_out], DATA[:, n_in:]
X_train,X_test,y_train,y_test = train_test_split(X,Y,test_size=0.1,random_state=49)

model = Sequential()
n_nodes = 10
model.add(Dense(4*n_nodes,activation='relu',input_dim=n_in,name='dense_0'))
model.add(Dense(2*n_nodes,activation='relu',name='dense_1'))
model.add(Dense(n_nodes,activation='relu',name='dense_2'))
model.add(Dense(n_out,activation='relu'))
model.compile(loss='mse',optimizer='adam',metrics=['mse'])
# fit
history = model.fit(X_train,y_train,validation_data=(X_test,y_test),epochs=20,)

pred = model.predict(X,)
print('model prediction, mean %.3f, std %.3f' % (np.mean(pred),np.std(pred)))

for ind in range(3):
    intermediate_layer_model = Model(inputs=model.input,outputs=model.get_layer('dense_%i' % ind).output)
    pred = intermediate_layer_model.predict(X)
    print('layer %i, mean %.3f, std %.3f, min %.3f, max %.3f' % (ind,np.mean(pred),np.std(pred),np.min(pred),np.max(pred)))

In this script, I saved the array data in OP's post, to data.npy, which can be found in this repo, for simplicity. Besides, I fixed the random seeds of keras and train_test_split, therefore, you will reproduce the scenario in which the trained model constantly predicts zero.
In fact, as you mentioned in your post, similar scenarios are not rare (and try a shallower model does not help), I think the problem is Dense is simply not capable of dealing with time series, you need LSTM instead. Try the code below, in which I replaced Dense with LSTM+Dropout, besides, I changed the activation function of the output layer to tanh.
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler
from tensorflow.keras.models import Model,Sequential
from tensorflow.keras.layers import Dense,LSTM,Dropout

from matplotlib import pyplot as plt
import numpy as np
import tensorflow as tf
from tensorflow import keras

# main
def series_to_supervised(data,n_in,n_out):

    df = pd.DataFrame(data)
    cols = list()
    for i in range(n_in,0,-1): cols.append(df.shift(i))
    for i in range(0, n_out): cols.append(df.shift(-i))
    agg = pd.concat(cols,axis=1)
    agg.dropna(inplace=True)

    return agg.values

n_in,n_out = 14,1
data = np.load('data.npy')
scaler = MinMaxScaler(feature_range=(0, 1))
scaler = scaler.fit(data)
scaled_data=scaler.transform(data)
DATA = series_to_supervised(scaled_data[:-10], n_in, n_out)
X, Y = DATA[:, :-n_out], DATA[:, n_in:]
X_train,X_test,y_train,y_test = train_test_split(X,Y,test_size=0.1,random_state=49)
X_train = X_train[:,None,:]
X_test = X_test[:,None,:]

wrong_ind = 0
for ind in range(100):
    print('working on %i' % ind)
    keras.backend.clear_session()
    model = Sequential()

    model.add(LSTM(4,name='lstm_0'))
    model.add(Dropout(0.2,name='dropout_0'))
    model.add(Dense(n_out,activation='tanh'))
    model.compile(loss='mse',optimizer='adam',metrics=['mse'])
    # fit
    n_epoch = 5 if ind < 99 else 200
    history = model.fit(X_train,y_train,validation_data=(X_test,y_test),epochs=n_epoch,verbose=0)
    val = history.history['val_loss']
    if np.abs(val[0] - val[-1]) < 1e-4:
        print(ind,val)
        wrong_ind += 1

print(wrong_ind)

fig,ax = plt.subplots(nrows=1,ncols=2,figsize=(12,8))
ax[0].plot(history.history['val_loss'],'r')
ax[0].plot(history.history['loss'],'b')
ax[1].plot(model.predict(X[:,None,:]),'r')
ax[1].plot(Y,'b')
plt.show()

the output is


How to calculate coverage in recommender systems? I am trying to calculate coverage metrics for a recommender system that I have designed. This  blog post talks about how to do it. I had some difficulties in understanding the same. It says that These metrics will be for user u of U users over n items out of N potential items in a recommendation list L. Each item has a content vector C of length c ... My difficulty arises in understanding N and U. Is U the total number of users? Secondly, what does N potential items mean? Is it the list of all items that are present or is it a subset of all the items present?

I am using the Python LightFM package to generate the recommendations

Right, the capital letters denote the total available. In that blog post: U means all users, N means all items but in other places is usually written I, and L means all top-n recommendation lists. "Top-n" means that the recommender system outputs a ranked list of n items, so if you had 1000 users all getting a Top-10 list, you'd have L length of 1000*10.

I suggest you read Ge, Mouzhi, Carla Delgado-Battenfeld, and Dietmar Jannach. "Beyond accuracy: evaluating recommender systems by coverage and serendipity." Proceedings of the fourth ACM conference on Recommender systems. ACM, 2010, at http://ls13-www.cs.tu-dortmund.de/homepage/publications/jannach/Conference_RECSYS10b.pdf instead. It's a more clearly written definition of metrics and uses more traditional recommender system notation.


Wrong output multiple linear regression statsmodels I recently moved to python for data analysis and apparently I am stuck on the basics. I am trying to regress the parameters of the following expression: z=20+x+3*y+noise, and I get the right intercept but the x and y parameters are clearly wrong. What am I doing missing? Code below:

import numpy as np
import pandas as pd
import statsmodels.formula.api as smf

# generate true values, and noise around them
np.random.seed(5)
x = np.arange(1, 101)
y = np.arange(1, 101)
z = 20 + x + 3* y + np.random.normal(0, 20, 100)

data = pd.DataFrame({'x':x, 'y':y, 'z': z})

lm = smf.ols(formula='z ~ x + y', data=data).fit()

# print the coefficients
lm.summary()

returns

where the x and y parameters are both 1.5, instead of being 1 and 3. What's wrong?

I think that you are seeing what you are seeing because the model sees the relationship of each set of points in your data frame, which are governed by the equation:

 z = 20 + x +3*y + noise

But all the model sees is the resulting Z, NOT that equation which you know created z.

So it attempts to build a model which considers how Z was accomplished without knowing there was noise, while KNOWING that both x and y were in this equation because you explicitly told it the are in the model.

Based on this data. (at least this is what I got without a seed, when I ran your data..so it is close with probably different Z due to different noise)

x   y   z
1   1   32.824550
2   2   21.382597
3   3   80.615424
4   4   30.958157
5   5   42.192197
6   6   75.649622
7   7   29.815352
8   8   40.167267
9   9   59.752065
10  10  53.402601

Because x and y are the same for each point  and because your formula has x + 3*y + noise ,  z is also equal  to 4*x+ noise or 4*y+ noise or 2*x +2*y+ noise for each row. There are many ways to get the same change to Z with x & y in exact proportion plus some noise.

So the regressors are being assigned equal influence plus an equal share of the noise. It is the most parsimonious evaluation of x and y.

It is not what you expect knowing the equation, but it is the answer you should get. If you use a linear model which reduces variables of zero influent, you might even get a 0 or NA value for y.

To test that lm is working, simply reverse the order of y changing the relationship between x & y within that equations and you have a completely different result. I think the one you were looking to find.

y = np.arange(100,0,-1)

            coef    std err    t    P>|t|   [0.025  0.975]
Intercept   0.0439  0.000   119.009 0.000   0.043   0.045
x           1.2184  0.038   32.471  0.000   1.144   1.293
y           3.2130  0.038   85.629  0.000   3.139   3.288

You built your model right, but in this case the data set was not created in a way to test for what you hoped to see...but it was not wrong.


Understanding Learning Curves I would like to clarify my understanding of learning curves with two example plots below. I am experimenting with small data sets here between 500 and 1500 samples to clarify my understanding.
My understanding from the learning curve below is that underlying data must have a lot of noise and therefore the learning algorithm is not able to generalize the underlying function. This learning curve indicates neither high bias or variance. Just the overall data samples are not a good indicator of the outcome. Is my interpretation of this learning curve above correct? Would getting more data help?

From the learning curve below, I am interpreting this as a good learning curve. Because both the training and cross validation misclassification errors are going down as we add more samples. Around ~240 samples we get the least difference between the training and validation sets. Is this indicative of high variance since the training and validation never converge even though the misclassification error delta is pretty low between the two?

I wouldn't interpret the first graph the same way. As far as know (and bear in mind that nobody knows everything), there's no clear way to distinguish between noisy features and bias (underfitting). But in either case there's no reason why the validation/CV loss would be much higher than the training loss, this is actually a clear sign of overfitting: if the model was not able to generalize, the training loss would be as high as the validation loss. Instead the model succeeds in representing some patterns, but it turns out that the patterns it captures don't apply (at least not as well) on the validation set. In other words, the model captures patterns which happen by chance in the training set: it overfits.
It's possible that adding more instances would help reduce the overfitting, as the CV loss appears to still decrease significantly in the 800-900 zone. But the difference is very large, so it's not sure that it can be solved only with more data.
I agree that the second graph shows some overfitting (as well), but not as much as the first one. In my opinion, the fact that the lowest difference between the two curves is not at the end of the curve points to either a model which is not stable yet (needs more data) or maybe there are simply not enough points on the curve to observe the real trend (i.e. the low difference at 240 and/or the higher one at 300 might happen by chance). This is indeed variance, but it's not very high
in my opinion.


User Location Prediction I am trying to predict the user location for a mobile app for a certain timestamp. So far I've gathered training data throughout different days. The data contains the latitude, longitude and the timestamp.

What would be the best algorithm/best way in order to train a model that takes a timestamp as an input and gives as output a latitude and longitude pair?

KNN:

The prediction of Lat/Lng from a given timestep could be performed by analysing the previous data of the user. This is a totally naive approach but could be tried,

Take an input timestamp from the user. Let's call it $x$.
From the previous data of the user, search for the timestamp which is the closest to $x$.
Once you find the closest timestamp, output the Lat/Lng associated with that timestamp.

  This approach is called K-NN ( K Nearest Neighbours ) classification.
  It will work well if the user is mostly at a common place at a given
  time ( timestamp ).

Clustering method:

This method will work on making clusters of Lat/Lng. For some a given time-interval, suppose, you will get a common timestep ( which will be the centre of the cluster ).

Once you have a cluster of timesteps, find the timestep ( centre of the cluster ) which is the closest to the input $x$.
For this timestep, you will get Lat/Lng which close to one another.
For predicting the Lat/Lng, use the centre of the cluster and output the associated Lat/Lng values.

  Basically, you are clustering similar data points which help in prediction. You can find where a user's location is clustered at given time intervals.


Why do RNNs usually have fewer hidden layers than CNNs? CNNs can have hundreds of hidden layers and since they are often used with image data, having many layers captures more complexity.

However, as far as I have seen, RNNs usually have few layers e.g. 2-4. For example, for electrocardiogram (ECG) classification, I've seen papers use LSTMs with 4 layers and CNNs with 10-15 layers with similar results.

Is this because RNNs/LSTMs are harder to train if they are deeper (due to gradient vanishing problems) or because RNNs/LSTMs tend to overfit sequential data fast?

CNNs and RNNs feature extraction methods:

CNNs tend to extract spatial features. Suppose, we have a total of 10 convolution layers stacked on top of each other. The kernel of the 1st layer will extract features from the input. This feature map is then used as an input for the next convolution layer which then again produces a feature map from its input feature map.

Likewise, features are extracted level-by-level from the input image. If the input is a small image of 32 * 32 pixels, then we will definitely require fewer convolution layers. A bigger image of 256 * 256 will have comparatively higher complexity of features.

RNNs are temporal feature extractors as they hold a memory of the past layer activations. They extract features like an NN, but RNNs remember the extracted features across timesteps. RNNs could also remember features extracted via convolution layers. Since they hold a kind-of memory, they persist in temporally/time features.

In case of electrocardiogram classification:

On the basis of the papers you read, it seems that,

  ECG data could be easily classified using temporal features with the help of RNNs. Temporal features are helping the model to classify the ECGs correctly. Hence, the usage of RNNs is less complex.

The CNNs are more complex because,

  The feature extraction methods used by CNNs lead to such features
  which are not powerful enough to uniquely recognize ECGs. Hence, the larger number of convolution layers is required to extract those minor features for better classification.

At last,

  A strong feature provides less complexity to the model whereas a
  weaker feature needs to be extracted with complex layers.

Is this because RNNs/LSTMs are harder to train if they are deeper (due to gradient vanishing problems) or because RNNs/LSTMs tend to overfit sequential data fast?

This could be taken as a thinking perspective. LSTM/RNNs are prone to overfitting in which one of the reasons could be vanishing gradient problem as mentioned by @Ismael EL ATIFI in the comments.

I thank @Ismael EL ATIFI for the corrections.


Custom POS tagger for health issues I am new to NLP, I have a bunch of raw data that is not tagged at all of medical questions, I need to extract from them what are the health issues stated in those texts.
I was thinking I need to create two custom POS tags for NER:
-the location on the body
-the problem itself
So if someone asked 'my head hurts' it would understand that the location is the head and the problem is that it hurts, but if someone asked 'my skin is red around my abdomen' it would understand that the location is the abdomen and the problem is that the skin is red.
After I extract this data I need to recommend medical articles based on what that user asked.
I have some questions:
1.Am I on the right path?
2.How would you implement it?
3.Do I need a custom pos tag for the location and the health issue or can it be done easier? How would you extract those informations?
4.I guess I have to manually tag questions right?
5.What framework would you use?
6.To create a recommender system I need to extract the same informations from medical articles?
7.How would you create a recommender system?
As I said, I am new to NLP and I didn't decide on the framework yet but the questions are not in english, I have found however on github a WordNet clone and a Named Entity Corpus for my language, so please keep in mind when recommending frameworks.

A few comments:

The term "POS" (Part Of Speech) has the specific meaning of "grammatical category" so it's not really what you mean here, you should probably say "custom entities tags/categories/labels".
Your plan makes sense, but be aware that that you're addressing a really difficult problem so even with a really good dataset and method it's not going to work perfectly (probably far from it).
You're probably going to need a very large amount of annotated data for this to work, because there are many different ways to express a medical problem so the model will need many examples to extract the relevant patterns.
The standard way to address this kind of problem is a bit different when there are terminology  resources for the language: the medical terms can be annotated directly simply by pattern matching. If you have such resources and they are complete enough, this saves you the need to manually annotate and train a custom model.
Doing a recommender system on top of that is a second problem, and I think you need to think carefully about the goal:

if it's meant to be used by non-experts, it wouldn't make sense to recommend research articles.
The vast majority of the research literature is published in English, so there would be an additional problem to address here.
In case you're considering the whole medical literature, you might have a complexity issue because it's huge (PubMed contains around 30 millions abstracts, and that's without considering the full articles in PMC)
What would be the basis for recommendation? Is it based only on matching the terms in the query with the documents?


Sentiment analysis using python I have some text files containing movie reviews I need to find out whether the review is good or bad. I tried the following code but it's not working:
import nltk
with open("c:/users/user/desktop/datascience/moviesr/movies-1-32.txt", 'r') as m11:
    mov_rev = m11.read()
mov_review1=nltk.word_tokenize(mov_rev)
bon="crap aweful horrible terrible bad bland trite sucks unpleasant boring dull moronic dreadful disgusting distasteful flawed ordinary slow senseless unoriginal weak wacky uninteresting unpretentious "
bag_of_negative_words=nltk.word_tokenize(bon)
bop="Absorbing Big-Budget Brilliant Brutal Charismatic Charming Clever Comical Dazzling Dramatic Enjoyable Entertaining Excellent Exciting  Expensive Fascinating Fast-Moving First-Rate Funny Highly-Charged Hilarious Imaginative Insightful Inspirational Intriguing Juvenile Lasting Legendary Pleasant Powerful Ripping Riveting Romantic Sad  Satirical Sensitive  Sentimental Surprising Suspenseful Tender Thought Provoking Tragic Uplifting Uproarious"
bop.lower()
bag_of_positive_words=nltk.word_tokenize(bop)
vec=[]
for i in bag_of_negative_words:
    if i in mov_review1:
        vec.append(1)
    else:
        for w in bag_of_positive_words:
            if w in moview_review1:
                vec.append(5)

So I am trying to check whether the review contains a positive word or a negative word. If it contains a negative word then a value of 1 will be assigned to the vector vec else a value of 5 will be assigned. But the output I am getting is an empty vector.
Please help. Also, please suggest others way of solving this problem.

Try to search from the databases of official "bad words" that google publishes in this link Google's official list of bad words.
Also, here is the link for the good words Not the official list of good words
For the code, I would do it like this:
textArray = file('dir_to_your_text','r').read().split()

#Bad words should be listed like this for the split function to work
# "*** ****** **** ****" the stars are for the cenzuration :P
badArray = file('dir_to_your_bad_word_file).read().split()
goodArray = file('dir_to_your_good_word_file).read().split()

# Then you use matching algorithm from difflib on good and bad word for every word in an array of words
import difflib

goodMachingCouter = 0;
badMacihngCouter = 0;

for iGood in range(0, len(goodArray)):
    for iWord in range(0, len(textArray)):
        goodMachingCounter += difflib.SequenceMatcher(None, goodArray[iGood], textArray[iWord]).ratio()

for iBad in range(0, len(badArray)):
    for iWord in range(0, len(textArray)):
        badMachingCounter += difflib.SequenceMatcher(None, badArray[ibad], textArray[iWgoodord]).ratio()

goodMachingCouter *= 100/(len(goodArray)*len(textArray))
badMacihngCouter *= 100/(len(badArray)*len(textArray))

print('Show the good measurment of the text in %: '+goodMachingCouter)
print('Show the bad measurment of the text in %: '+badMacihngCouter)
print('Show the hootnes of the text: ' + len(textArray)*goodMachingCounter)

The code will be slow but accurate :) I didn't run and test it please do it for me and post the correct code :) because I wanna test it too :)


What is the typical accuracy of masked language models during BERT pretraining? I was reading the BERT paper but I didn't find any tables concerning the performance of the masked language models during pretraining. Does anyone know the accuracy of BERT's masked language model?

It is very hard to tell and researchers usually don't measure it because it's not a comparable number such as GLUE score. It depends on size of your vocabulary and also the masking strategy (for example newer BERT successors use span-masking). BERT is a subword language model (uses WordPiece tokenizer) so if you use large vocab then sentences are usually tokenized to "smaller" subword units that are easier to predict. I trained a BERT-like model for Czech language, which is morphologically more complex than English, and got top@1 test accuracy ~0.55, top@3 test accuracy ~0.68 after 1.5M steps with 30K vocab size.


Is it possible to create feature groups in Orange? Background:
I am trying to use Orange as to classify if a patient has TB based on their coughing sounds.

In the dataset, there are say 100 patients and for each patient we have 10 coughs. For each cough, we have a full feature vector (170 features).

Giving Orange this dataset and training various learning algorithms is fairly straightforward, but the issue that I have is that Orange will consider each feature vector to be independent of another feature vector, which means it will consider every cough of each patient to be independent, and they aren't.

So my question is: Is there a way to tell orange that all 10 coughs of a patient belong to that patient, and when performing leave-one-out or cross validation methods, all the coughs from each patient should be excluded in each fold?

This is not Orange-specific, but IIUC, you could preprocess your data (e.g. in Python or Excel) to have each of the 10 coughs pertaining to a patient on the same patients line. Thus you would have: 100 rows of patients with each row (10*170 + other patient data) attributes wide.


Is there a way to measure the "sharpness" of a decision boundary of a CNN? It is commonly seen as something bad if the decision boundary of a neural network is too sharp, meaning if slight changes in the input change the class prediction completely.

Given a trained CNN, is it possible to measure / calculate the "sharpness" of its decision boundaries? Did somebody do that already?

You might enjoy looking into the literature on "adversarial examples".  Given an instance $x$ with a label $y$, an adversarial example is a (typically carefully constructed) instance $x'$ that is very similar to $x$, but whose label differs from $y$.  The research literature suggests that it is often possible to find adversarial examples that are very close to the original $x$.  You could use the distance $d(x,x')$ as a measure of sharpness of the decision boundary near $x$; or you could average this over many $x$ to get a global measure of sharpness.

There are many methods for finding adversarial examples.  A standard simple one is the gradient sign method, originally described in the following paper:

Ian J. Goodfellow, Jonathon Shlens, Christian Szegedy. Explaining and harnessing adversarial examples. arXiv:1412.6572, 2014.

Since then there have been numerous improvements that find even closer adversarial examples, e.g., iterative gradient sign (arxiv:1607.02533), Deepfool (arxiv:1511.04599), and others.  You might also be interested in  Cleverhans, a software library to assist with finding adversarial examples.


NoSQL vs SQL backend for semi structured data I have a corpus of job descriptions and another corpus of CVs of applicants. I plan to implement a matching system using machine learning algorithms, to find top 5 or top 10 applicants for each job description. Should I store the data in a document oriented NoSQL db (MongoDB) or stick to SQL.

Given that the data I have is semi-structured at best, I feel a NoSQL db will offer more flexibility. I would appreciate opinions on this.

I would use SQL and create a set of structured fields that are common across all applications (name, school, years of experience, job they are applying for, etc.) with a field containing the raw application for you 'semi-structured' part of the data. You can always do something fancy with the raw application field, but if you want to get some summary statistics quickly then SQL is the proper route.

In general, thinking carefully about your schema up front will pay big dividends in the future. I would recommend NoSQL only if you really have no idea how to structure things and are pressed for time. Even getting a list of the keys in a collection requires MapReduce.


How to get R Table Column Names I could not find anywhere how I can do this seemingly simple task (Retrieving column names in R was not helpful).
I have an R table defined as sports_by_sex <- table(ais$sex, ais$sport). This is using DAAG's ais dataset. If I want to get the name of the sport at sports_by_sex[, 2], how would I do that? When I print(sports_by_sex), this is my result:
    B_Ball Field Gym Netball Row Swim T_400m T_Sprnt Tennis W_Polo
  f     13     7   4      23  22    9     11       4      7      0
  m     12    12   0       0  15   13     18      11      4     17

So for the example of sports_by_sex[, 2], for example, I would like to get Field back.

Found the solution - access the index you want on colnames():
colnames(sports_by_sex)[2], for example.


How to choose layer from which to unfreeze image classification model I'm wondering what steps do you take to
decide on the part of the model to unfreeze. Do you do
multiple experiments? Since the use of GPU is expensive, you must
have some guidelines.

Note: I know the relationship between size of dataset, how close
dataset is to the original dataset and how much that impacts whether
or not we train more layers. However is there a rule of thumb
involving the depth of the model to get the approximate layer?

Example:  Try unfreezing model starting from Layer number 169,
or  layers between 70-100
How much does one need to know specifics of the pretrained model? Can I use it without knowing the architecture?

Thank you for your help!

Regarding #1: As you mentioned, the following amount of fitting you do on a pre-trained (PT) network depends on the relative size and similarity of your data and the data used to train the PT model.

If our dataset is smaller than the PT data, we should unfreeze fewer layers to avoid overfitting.
If our dataset is of a similar size as the PT data, we can unfreeze more layers since overfitting is less of a concern.

Essentially, more frozen layers means less overfitting, since those layers will retain the relatively generic features of the PT network. As such, you can use hyperparameter tuning (and the results of your cross-validation) to optimize this like you would any other hyperparameter (grid search, etc).

Regarding #2: Yes, you certainly can use the PT model to fit and make predictions. But, the more you understand the model's architecture, the better you will be able to tune the model, and ultimately, improve your results.


Terminology question In Machine Learning, is the definition of the Model just the algorithm that was selected for the problem domain, or is the Model the algorithm and the training data?

Thanks.

The model is that process or algorithm which you are using to train it with training or labeled data. After the training of that model, it will be dubbed trained model.


Calibration Curve Error I want to calibrate probability outputs of a model. I'm using Isotonic Regression. After calibration, when I called calibration_curve function of sklearn calibration module I got this error: ValueError: 'list' argument must have no negative elements. However when I checked results that I obtain from Isotonic Regression there are no negative values and all values are in [0,1] range. There are no problems in targets too.
from sklearn.isotonic import IsotonicRegression

i_reg = IsotonicRegression().fit(X_train, y_train)
res = i_reg.predict(X_test)
prob_true, prob_pred = calibration_curve(y_test, res, n_bins=200, strategy="quantile")

Error log:
ValueError                                Traceback (most recent call last)
<ipython-input-36-5a2bd10e64b2> in <module>
----> 1 prob_true_model_test , prob_pred_cal_test = calibration_curve(y_test, res, n_bins=200, strategy="quantile")

/environment/lib/python3.6/site-packages/sklearn/utils/validation.py in inner_f(*args, **kwargs)
     70                           FutureWarning)
     71         kwargs.update({k: arg for k, arg in zip(sig.parameters, args)})
---> 72         return f(**kwargs)
     73     return inner_f
     74

/environment/lib/python3.6/site-packages/sklearn/calibration.py in calibration_curve(y_true, y_prob, normalize, n_bins, strategy)
    590     binids = np.digitize(y_prob, bins) - 1
    591
--> 592     bin_sums = np.bincount(binids, weights=y_prob, minlength=len(bins))
    593     bin_true = np.bincount(binids, weights=y_true, minlength=len(bins))
    594     bin_total = np.bincount(binids, minlength=len(bins))

<__array_function__ internals> in bincount(*args, **kwargs)

ValueError: 'list' argument must have no negative elements

It's hard to say without your data.  I'd comment this, but it's a bit long and better formatted as an answer.
The part of the source code that's relevant is quite short, so you can go through step by step to see what's wrong:
bins = np.linspace(0., 1. + 1e-8, n_bins + 1)
binids = np.digitize(y_prob, bins) - 1
bin_sums = np.bincount(binids, weights=y_prob, minlength=len(bins))

digitize can return 0 if an element is to the left of all the bins; that shouldn't happen here, but maybe some computational precision error has crept in?  Check what values you get for binids above.
See also this StackOverflow question, where NaNs were the culprit.


Lightgbm confidence interval I want to compute a confidence interval for each sample for a lightgbm model I've trained.
If the model was a random forest, it'd be quite easy, just take all the trees and compute the standard deviation of the predictions.
I wonder if a similar think can be done with lightgbm. In particular:

Does it make sense to compute the standard deviation of a boosting model by taking the predictions of all trees, or something similar?
Can it be done in scikit-learn?
Are there alternatives to obtain confidence intervals for lightgbm predictions, appart from the alternative that would be training quantile models?

If you are looking for a statistical trick, I don't know, but
Recently Andrew NG team recently published about NGBoost.

NGBoost is a new boosting algorithm, which uses Natural Gradient Boosting, a modular boosting algorithm for probabilistic predictions.

In this Towards Data Science toy example you can see how to use the Python API:
Quoting the TDS author:

NGBoost’s one of the biggest differences from other boosting algorithms is can return probabilistic distribution of each prediction. This can be visualized by using pred_dist function.

In their web they say "NGBoost can be used with any base learner, any family of distributions with continuous parameters, and any scoring rule." So it seems pretty flexible to do whatever you consider.


Error on custom RNN/LSTM with multiple inputs I want to implement a custom RNN/LSTM model similar to this. The model should take two separate vectors as input and process them. I was following keras tutorial to implement a custom keras layer and inputting two vectors a and b as a list [a,b] to the layer as shown below.

import keras
from keras.layers.recurrent import RNN
import keras.backend as K

class MinimalRNNCell(keras.layers.Layer):
    def __init__(self, units, **kwargs):
        self.units = units
        self.state_size = units
        super(MinimalRNNCell, self).__init__(**kwargs)
    def build(self, input_shape):
        print(type(input_shape))
        self.kernel = self.add_weight(shape=(input_shape[0][-1], self.units),
                                      initializer='uniform',
                                      name='kernel')
        self.recurrent_kernel = self.add_weight(
            shape=(self.units, self.units),
            initializer='uniform',
            name='recurrent_kernel')
        self.built = True
    def call(self, inputs, states):
        prev_output = states[0]
        h = K.dot(inputs[0], self.kernel)
        output = h + K.dot(prev_output, self.recurrent_kernel)
        return output, [output]

# Let's use this cell in a RNN layer:
cell = MinimalRNNCell(32)
a = keras.Input((None, 5))
b = keras.Input((None, 5))
layer = RNN(cell)
y = layer([a,b])

But I am getting the error TypeError: 'NoneType' object has no attribute '__getitem__' at

self.kernel = self.add_weight(shape=(input_shape[0][-1], self.units),
                                          initializer='uniform',
                                          name='kernel')

Also the type of the input_shape is showing as <type 'tuple'>, not a list.
Whats wrong I am doing and how to overcome this error. Please help

I think, input_shape[0] is None. Based on my understanding, you will pass a matrix of size (Batch_size, Time steps, dim of each time step data) to an RNN cell. And while creating your architecture, input_shape[0] will give the batch size (if you mentioned what is the batch size in input layer, if not you will get None). Even if you mention batch size, it will be an integer and you can't index the integers so you are getting that error.


Why are sigmoid/tanh activation function still used for deep NN when we have ReLU? Looks like ReLU is better then sigmoid or tanh for deep neural networks from all aspects:

simple
more biologically plausible
no gradient to vanish
better performance
sparsity

And I see only one advantage of sigmoid/tanh: they are bounded. It means that your activations won't blow up as you keep training and your network parameters won't take off to the sky.

Why should not we forget about sigmoid/tanh for deep neural networks?

In certain network structures having symmetric activation layers has advantages (certain autoencoders for example)
In certain scenarios having an activation with mean 0 is important (so tanh makes sense).
Sigmoid activation in the output layer is still important for classification

In 95% of the cases ReLU is much better though.


is it possible to output more than 2 nodes away from a node in a decision tree? if yes, how to do that with sklearn? usually a decision tree has one root node, some nodes, and some leaves.

lots tutorial illustrate this as something like binary tree.

is it possible more than 2 nodes away from a node in a decision tree?

this image comes from this post

by "more than 2 nodes", i mean there are more than 3 splits (in this case, 3, Low, Med, High) away from the root node.

if it is reasonable in real life application, plz provide an open dataset on which a decision tree would spit more than 2 nodes, and a piece of sklearn code.

It is possible to make more than a binary split in a decision tree. Chi-square automatic interaction detection (CHAID) is an algorithm for doing more than binary splits.

However, scikit-learn only supports binary splits for many reasons. One primary reason to limit to just binary splits is that the library can support as many splitting criteria as possible with the same API. For example, Gini Impurity only supports binary splits.

In practice, only supporting binary splits is not an issue because a series of binary splits can model any number of simultaneous splits.


Looking for "outliers" will not give you the appropriate answer because you have a trend.
Moreover the trend is not linear, so the answer will depend on the trend model.

What you should do is :

(1) fit the trend. I choose a logistic function with a MAD but an exponential with LSE growth would have lead the same results

(2) compute the residuals $e(t)$ as the observed data minus the model.

(3) search for the outliers on the residuals. I tool Tuckey's fence with $|e(t)|$ greater than 1.5 times the interquartile.

The attached graph shows that obviously you should choose an exponential growth with a multiplicative error, thus minimizing the MAPE and computing the errors in percentage.


Keras ImageDataGenerator.flow_from_directory doesn't find images I'm working on a deep learning (CNN) problem. I have structured my images into folders correctly (I think), like this:

train_dir:
   - class 1
   - class 2
validation_dir:
   - class 1
   - class 2
test_dir:
   - class 1
   - class 2

The images are PNG in a (196,256,3) format but will I'd like to convert them to (196,256,1) because they are grayscale. I do this with the flow_from_directory argument color_mode='grayscale'

As such the code I have so far to get on with this is:

train_datagen = ImageDataGenerator(rescale=1./255) # re-scale pixel values between [0,1]
test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
        train_dir,
        target_size = (196,256),
        color_mode='grayscale',
        batch_size=20,
        class_mode='binary')

validation_generator = test_datagen.flow_from_directory(
        validation_dir,
        target_size = (196,256),
        color_mode='grayscale',
        batch_size=20,
        class_mode='binary')

With train_dir and validation_dir being strings with the equivalent path of the correct folders I mentioned earlier above.

However I get this error when running this section of the code:

Found 0 images belonging to 2 classes

After trying for sometime I can't figure out what's wrong. Does anyone have a clue?

SOLVED!

Always be sure you have '.png' or whatever other file extension you are using at the end of your filename


Given two large corpora of text from different sources, is there an accepted way to get differences in vocabulary (n-grams) between them? Given two large corpora of text from different sources, is there an accepted way to get differences in vocabulary (n-grams) between them?

That is, to get results which say that, for example, the bigram "hello world" is much more common in corpus A than corpus B (ideally with some kind of measure of how much more common).

TF-IDF examples use a larger number of documents to highlight "important" words in each, but I am not sure if that would work in this case?

As far as I understood your question you can work with NLTK.

Have a look on the following functions:

Tokenize
Stemmers
pos_tags (if you wanna categorize by Verb, Noun etc. beforehand)

And to come to your question of similarity or relative word/phrase usage you have FreqDistfunction in NLTK. Using this Output you could set each occurance relative to the length of the list that is returned by tokenize


memory error in matrix cosine_similarity I have (20905040, 7)
of a dataset to recommend 10 different product to the user
it could be larger than that but anyway I got memory error when processing the

cosine_sim = cosine_similarity(normalized_df,normalized_df)

  --------------------------------------------------------------------------- MemoryError                               Traceback (most recent call
  last)  in
        1 get_ipython().run_line_magic('time', '')
  ----> 2 cosine_sim = cosine_similarity(normalized_df,normalized_df)

  ~/venv/lib/python3.6/site-packages/sklearn/metrics/pairwise.py in
  cosine_similarity(X, Y, dense_output)    1034     1035     K =
  safe_sparse_dot(X_normalized, Y_normalized.T,
  -> 1036                         dense_output=dense_output)    1037     1038     return K

  ~/venv/lib/python3.6/site-packages/sklearn/utils/extmath.py in
  safe_sparse_dot(a, b, dense_output)
      140         return ret
      141     else:
  --> 142         return np.dot(a, b)
      143
      144

  MemoryError:

questions
1. when I have too much rows how do I apply cosine similarity?
2. do they talking about ram memory? or what memory error ?
3. is there way to use gpu for cosine similarity training?
4. any good idea?

This is talking about RAM. There are a few factors that will decide how many rows/columns you can use. Instead of rows/columns, it is maybe easier to just think in total number of elements: num_rows * num_cols. The memory you will require is going to have a relationship to this number.

There are ways that might take less working memory to solve the problem - usually memory and speed are part of the trade-off. If you use a lot less memory while computing the result, you can do fewer at a time and so it takes longer.

If you have floating point numbers (those with decimals), then Pandas usually uses the data type float64 by default. You could try using float32 instead. It offers about half of the accuracy, but also only uses half of the memory. You can do this by simply adding this line before you compute the cosine_similarity:

import numpy as np

normalized_df = normalized_df.astype(np.float32)
cosine_sim = cosine_similarity(normalized_df, normalized_df)

Here is a thread about using Keras to compute cosine similarity, which can then be done on the GPU. I would point out, that (single) GPUs will generally have less working memory available than your computer itself.

Here is a blog that speaks about how to scale up your computation to use tools like Spark, for distributed computing. That would allow you to deal with much larger matrices, provided you have several computers available.


Can we apply to GridSearchCV to Logistic regression .? When I apply GridSearchCV to my model Logistic Regression, it's continuously throwing below error.
I understand that it's trying to convert string to float. But that's was my data. So how can I convert my x value to float? Can any please hep me with this issue. Thank you.

Error: could not convert string to float: 'great im readi go home'

Sample data

Airline_sentiment     text

0     said

1     plu youv ad commerci experi tacki

2     didnt today must mean need take anoth trip

penalty = ['l1', 'l2']

C = np.logspace(0, 4, 10)

hyperparameters = dict(C=C, penalty=penalty)

gs = **GridSearchCV**(lr,hyperparameters,cv=5)

I believe that the first thing you should do is to convert your data to a numeric type.

Maybe this starter tutorial can help you
https://towardsdatascience.com/introduction-to-natural-language-processing-nlp-323cc007df3d


Cross Validation with Artificial Neural Network in Keras I needed to use cross-validation in the Neural network.
I used
 kf = KFold(n_splits=5, shuffle=True)

and
  for train_index, test_index in kf.split(train):
    train_X, train_y = train_XX[train_index], train_yy[train_index]

but I couldn't manage to run. One of the errors I see is
ValueError: Cannot have number of splits n_splits=5 greater than the number of samples: n_samples=2.

I would be glad to have your suggestions if you ever tried CV with in Neural network.

It is likely that your train variable in kf.split(train): is a list of two lists e.g. train_x and train_y or something similar. I am guessing this because the KFold API is only detecting only two entries in it, which it is unable to partition in 5 subsets (folds).


Chi-square test - how can I say if attributes are correlated? I am experimenting a course's teorical contents on this dataset. After data cleaning, I am trying to use chi-square test. I wrote the following code:

chisq.test(chocolate$CompanyMaker, chocolate$Rating, simulate.p.value = TRUE)
chisq.test(chocolate$SpecificBeanOriginOrBarName, chocolate$Rating, simulate.p.value = TRUE)
chisq.test(chocolate$CompanyLocation, chocolate$Rating, simulate.p.value = TRUE)
chisq.test(chocolate$BeanType, chocolate$Rating, simulate.p.value = TRUE)
chisq.test(chocolate$BroadBeanOrigin, chocolate$Rating, simulate.p.value = TRUE)

chisq.test(chocolate$CompanyMaker, chocolate$CocoaPerc, simulate.p.value = TRUE)
chisq.test(chocolate$SpecificBeanOriginOrBarName, chocolate$CocoaPerc, simulate.p.value = TRUE)
chisq.test(chocolate$CompanyLocation, chocolate$CocoaPerc, simulate.p.value = TRUE)
chisq.test(chocolate$BeanType, chocolate$CocoaPerc, simulate.p.value = TRUE)
chisq.test(chocolate$BroadBeanOrigin, chocolate$CocoaPerc, simulate.p.value = TRUE)

And these are my results:

RATING

CompanyMarker   = 0.29
Specific...     = 0.6267
CompanyLocation = 0.1819
BeanType        = 0.5372
BroadBeanOrigin = 0.1534

COCOA PERC

CompanyMarker   = 0.0004998
Specific...     = 0.902
CompanyLocation = 0.04748
BeanType        = 0.8136
BroadBeanOrigin = 0.8356

Online, I read about significance level, but i didn't quite understand it. In particular, is it at 0.5 or 0.05? Which values are "ok"?

From what I understood, I should say that CompanyMarker, CompanyLocation and BroadBeanOrigin are related to Rating, while CompanyMarker and CompanyLocation are related to cocoaPercent.

Is this right?
If not, can you write or link me an example or a guide to do it right?
Thanks.

Chi-Square is used to determine which of the attributes are most informative. Its used in feature Selection.

So, if you have an attribute A ,B and C and output Y, we are trying to know Y depends on A or B or C ? A or B or C might be independent also i.e. has no affect on output Y.

So Chi-Sqaure is a statistical test to find out which attribute is independent and can be removed.

A contingency tables is created for each attribute value and its frequencies/occurrence is recorded and p_values above/below threshold determines if its relevant or not.

More about it here-
https://machinelearningmastery.com/chi-squared-test-for-machine-learning/

  Online, I read about significance level, but i didn't quite understand
  it. In particular, is it at 0.5 or 0.05? Which values are "ok"?

Please read about p-value


normalization/denormalization for linear regression problem My question is simple actually, I have two features that have big difference in scale. So I used a simple normalization by dividing the scale=np.max(array) for both data and lables. Then after prediction, I mulitiplied this scale value back.

But since I used a DNN, wouldn't the nonlinear change the scale so make the multiply not valid? e.g.

given input data: X, label: y;
y' = y/scale
X' = X/scale
predicted = f(X')
predicted_update = predicted * scale

Anyone could provide some advice on whether I could do this or it's actually not correct? How do we handle this kind of problem?

I think it is ok, as long as your training and test data have the same maximum values for every feature, approximately. The idea is that the scaling has to be done with the training set (remember that using the test set for anything that is not testing is illegal, not even for scaling).

So, you actually fit $y'$ as a function of $X'$, and you have a model that maps properly $y' = f(X')$. When you get your test data, you just obtain the predictions by doing $f(X_{test}')$. As the paragraph before states, if you have that $scale \approx scale_{test}$, then you can just recover $y_{test}$ by doing $y_{test} = scale \cdot f(X_{test}')$.

Edit: Don't worry about nonlinearities

Even if the function $f$ is highly nonlinear, it is a function capable of mapping $X'$ to $y'$. If you trust this function and trust the fact that $y = y' \cdot scale$, then there is no need to worry about the way $f$ acts, as function composition makes sense for all kind of functions, both linear and nonlinear.


Understanding decision tree concept I am referring Ehthem Alpaydin, 'Introduction To Machine Learning' book.

Under the chapter 'Decision Trees', I need help with understanding the concept of discriminant and how it is being used in this paragraph:

  Each $f_m(x)$ defines a discriminant in the d-dimensional input space dividing it into smaller regions that are further subdivided as we take a path from the root down. $f_m(.)$ is a simple function and when written down as a tree, a complex function is broken down into a series of simple decisions. Different decision tree methods assume different models for $f_m(.)$, and the model class defines the shape of the discriminant and the shape of regions. Each leaf node has an output label, which in the case of classification is the class code and in regression is a numeric value.

In the above image the big square is your $f_m(x)$.

The big square is made up of small partitions. Those partitions are your smaller functions(Partitions) which performs the comparison between your variables and try to find out best possible split.

Here each partition has nearly same labels for classification and nearest numerical values for regression.

The square representation is for 2 dimensions if you want to get for higher dimensions refer the last one with 3 dimensions.

Reference: Elements of Statistical Learning, Basics of Decision trees


What is the job of "RepeatVector" and "TimeDistributed"? I read about them in Keras documentation and other websites, but I couldn't exactly understand what exactly they do and how should we use them in designing many-to-many or encoder-decoder LSTM networks?

I saw them used in the solution of this problem here.

model = Sequential()
model.add(LSTM(input_dim=1, output_dim=hidden_neurons, return_sequences=False))
model.add(RepeatVector(10))
model.add(LSTM(output_dim=hidden_neurons, return_sequences=True))
model.add(TimeDistributed(Dense(1)))
model.add(Activation('linear'))
model.compile(loss='mean_squared_error', optimizer='rmsprop', metrics=['accuracy'])

tf.keras.layers.RepeatVector

According to the docs :

  Repeats the input n times.

They have also provided an example :

model = Sequential()
model.add(Dense(32, input_dim=32))
# now: model.output_shape == (None, 32)
# note: `None` is the batch dimension
model.add(RepeatVector(3))
# now: model.output_shape == (None, 3, 32)

In the above example, the RepeatVector layer repeats the incoming inputs a specific number of time. The shape of the input in the above example was ( 32 , ). But the output shape of the RepeatVector was ( 3 , 32 ), since the inputs were repeated 3 times.

tf.keras.layers.TimeDistributed()

According to the docs :

  This wrapper allows to apply a layer to every temporal slice of an input.
  The input should be at least 3D, and the dimension of index one will be considered to be the temporal dimension.

You can refer to the example at their website.

TimeDistributed layer applies a specific layer such as Dense to every sample it receives as an input. Suppose the input size is ( 13 , 10 , 6 ). Now, I need to apply a Dense layer to every slice of shape ( 10 , 6 ). Then I would wrap the Dense layer in a TimeDistributed layer.

model.add( TimeDistributed( Dense( 12 , input_shape=( 10 , 6 ) )) )

The output shape of such a layer would be ( 13 , 10 , 12 ). Hence, the operation of the Dense layer was applied to each temporal slice as mentioned.


What should I do to test the confidence of my deep learning model? I've recently fine-tuned a deep learning framework/model BERT for a sentiment classification task. I had a 80/10/10 train/validation and test set. After running several experiments, I've gotten a decent model that I'd like to eventually productionize. However, I'm looking to create an experiment to test the robustness/reliability/confidence of the model before putting it into production. What are some ways/experiments that can be conducted to test the robustness/reliability/confidence of this model or its predictions?
For example, are there statistically sound principles behind calculating the standard error for binary predictions on a new datapoint?

For binary predictions, it is standard to evaluate models based on their ROC and PRC curves. Some metrics are also useful, namely MCC, which is probably the most holistic scalar metric.
Using these metrics, you should evaluate the model via cross-validation. For deep models that take significant time to train, k-fold cross-validation is often sufficient. If you want, you can also do repeated k-fold cross validation, if time permits.
Lastly, while not always possible, many consider the use of different datasets to be one of the best indicators of reliability. Splitting a single dataset still risks "leaking" a common bias in both the training and testing sets. When using two or more datasets, biases that the model learns in the training dataset probably wouldn't manifest in a completely different dataset, hence a more objective evaluation methodology that mimics a production environment. Such biases include, among many things, data acquisition methods, preprocessing/data cleaning methods, etc..


Masters thesis topics in big data I am looking for a thesis to complete my master M2, I will work on a topic in the big data's field (creation big data applications), using hadoop/mapReduce and Ecosystem ( visualisation, analysis ...), Please suggest some topics or project that would make for a good masters thesis subject.
I add that I have bases in data warehouses, databases, data mining, good skills in programming, system administration and cryptography ...

Thanks

Since it's a master's thesis, how about writing something regarding decision trees, and their "upgrades": boosting and Random Forests? And then integrate that with Map/Reduce, together with showing how to scale a Random Forest on Hadoop using M/R?


Machine learning algorithms for correct words formation from jumbled words https://www.google.com/search?q=jumbled+words&oq=jumbled&aqs=chrome.1.69i57j0l4.3399j0j9&client=ms-android-lava&sourceid=chrome-mobile&ie=UTF-8
Can Machine learning algorithms solve the input dataset of jumbled words and form the correct words from them?

from itertools import permutations
import string
permutation_list = []
s = "BOX"
a = string.ascii_letters
p = permutations(s)

# Create a dictionary
d = []
for i in list(p):

    # Print only if not in dictionary
    if (i not in d):
        d.append(i)
        permutation_list.append(''.join(i))
        print(''.join(i))

import nltk
nltk.download('words')

from nltk.corpus import words
for word in permutation_list:
    print(word.lower() in words.words())

output -
BOX True
BXO False
OBX False
OXB False
XBO False
XOB False


Amount of multiplications in a neural network model I'm currently reading this book and want someone to tell me if what currently I'm assuming about neural network is right or not.   https://www.pyimagesearch.com/deep-learning-computer-vision-python-book/

If a layer has 30 neurons: In a feedforward meural network each neuron of the first layer multiplied with all the neurons of the second layer. That would be 30 neurons of the first layer multiplied by the 30 neurons of the second layer that would give a total of 900 (30*30 = 900) multiplications (is this is correct?)

But those 900 multiplications is only for the first layer and the second layer of the neural network(nn). So if I have a feedforward nn that has 3 layers I would have to do 900 more multiplications because they are the multiplications of the output of the first layer (that they are the input of the second layer) with the weight of the third layer.

So to recap what I said so far I have a feedforward nn with 3 layer with 30 neurons in the first two layers multiply each neuron with all the neurons of the second layer it would be 30 * 30 = 900 multiplications for each additional layer that I add, it adds 900 additional multiplications.

Then for a model with three (3) fully connected layer would be 900 multiplications in the first two layer because for what I said earlier and 900 multiplications in the second and third layer for a total of 1,800 (900 + 900)  multiplications excluding an activation function and this is only for a forward pass, is my understanding correct?

And in addition to a forward pass in a typical Neural network they also have a backward pass that because of my calculations doing earlier they are 1,800 derivatives (gradient) for the entire backward pass. Am I correct assuming this for a Neural network?. That's why a CPU computer takes so long to train a model because it has to do about 3,600 (1,800 + 1,800 ) mathematical operations.

Essentially you are correct, there are a lot of calculations necessary to process inputs and train neural networks.
You have some terminology a bit wrong or vague. E.g.

In a feedforward meural network each neuron of the first layer multiplied with all the neurons of the second layer.

The neurons do not multiply together directly. A common way to write the equation for a neural network layer, calling input layer values $x_i$ and first hidden layer values $a_j$, where there are N inputs might be
$$a_j = f( b_j + \sum_{i=1}^{N} W_{ij}x_{i})$$
where $f()$ is the activation function $b_j$ is the bias term, $W_{ij}$ is the weight connecting $a_j$ to $x_i$.
So if you have $M$ neurons in the hidden layer, you have $N\times M$ multiplications and $M$ separate sums/additions over $N+1$ terms, and $M$ applications of the transfer function $f()$

And in addition to a forward pass in a typical Neural network they also have a backward pass that because of my calculations doing earlier they are 1,800 derivatives (gradient) for the entire backward pass.

It doesn't work quite so directly, and there as a small factor of more calculations involved (you do not calculate each derivative with a single multiplication, often there are a few, some results are re-used, and other operations may be involved). However yes you do need to calculate a derivative for each weight and bias term, and there are roughly that number of weights in your network that require the calculations done.
Your suggested numbers are actually quite small compared to typical neural networks used for image problems. These typically perform millions of computations for a forward pass.

That's why a CPU computer takes so long to train a model because it has to do about 3,600 (1,800 + 1,800 ) mathematical operations.

Actually that is a trivial number of calculations for a modern CPU, and would be done in less than a millisecond. But multiply this out by a few factors:

You must do this for each and every example in the training data

Your example network is small, think bigger

This does not include the activation function calculations - typically slower than a multiply

Your rough estimate ignores some of the necessary operations, so as a guesstimate, multiply number of CPU-level operations by 3 or 4 from your analysis.
. . . and the number of operations does start to get to values where CPUs can take hours or days to perform training tasks in practice.


Neural network: does bias equal to zero, is the same as, a layer without bias? Question as in the title. Does bias equal to zero, is the same as, removing bias from the layer? Here's a pytorch implementation to showcase what I mean.
class MLP_without_bias(torch.nn.Module):
    def __init__(self):
        super().__init__()

        # Bias set as False
        self.linear = torch.nn.Linear(5, 3, bias = False)

        # Xavier initialization
        torch.nn.init.xavier_uniform_(self.linear.weight)

    def forward(self, x):
        return self.linear(x)

class MLP_with_bias_zero(torch.nn.Module):
    def __init__(self):
        super().__init__()

        # Default bias set as True
        self.linear = torch.nn.Linear(5, 3)

        # Xavier initialization
        torch.nn.init.xavier_uniform_(self.linear.weight)

        # Bias initialized as zero
        torch.nn.init.zeros_(self.linear.bias)

    def forward(self, x):
        return self.linear(x)

No, they are not the same:

In MLP_without_bias the bias will be zero after training, because of bias=False.

In MLP_with_bias_zero the bias is zero at initialization, but this will not prevent it from being updated during training.


Why doesn't class weight resolve the imbalanced classification problem? I know that in imbalanced classification, the classifier tends to predict all the test labels as larger class label, but if we use class weight in loss function, it would be reasonable to expect the problem to be solved. So why we need some approaches like down sampling or up sampling for imbalanced classification problem?

Class weights do help with the imbalance problem ("resolve" seems too much), but upsampling has a certain advantage on it.

If you think about it, downsampling/upsampling the number of samples in each class to balance the dataset is almost exactly the same as using class weights.

For example, say you have a dataset containing 3 samples divided into 2 classes:  and you are training with an MSE loss.

You can choose to upsample the number of samples from class B, which will get you the following cost function over a single epoch:

Here is where the small difference comes into effect, if you are training in a batch gradient descend (a single weights update per epoch), the prediction for the 2 identical B1 samples will be the same, so the loss function can be written as:

Which is exactly the same as using a weighted loss function. However if you are using a mini-batch gradient descend (as most model our days), the 2 different samples may appear in different mini-batches, and so the predictions for them at the same epoch won't be the same (because one of them will pass thru a model that was already updated once).

This is a small difference but sometimes it is important. It means that with weighted classes, the effective learning rate varies between mini-batches. In some cases that can make learning unstable. So, when possible, upsampling is the better approach (practically it produces slightly better results).

The problem is that you can't always upsample/downsample without any worry. If we go back to our example but this time we have a dataset of 5 samples, divided into 2 classes:

Downsampling is problematic - which of the 3 A class samples do we ignore?

Upsampling is problematic - which of the 2 B class samples do we duplicate?

It can be solved by randomly choosing which samples to keep/ignore at each epoch, however with very big datasets, this can lead to slower processing... So the weighted classes is still a valid option.


How to extract trees in XGBoost? I want to extract each tree so that I can feed it with any data, and see the output.

dump_list=xg_clas.get_booster().get_dump()
num_t=len(dump_list)
print("Number of Trees=",num_t)

I can find number of trees like this,

xgb.plot_tree(xg_clas, num_trees=0)
plt.rcParams['figure.figsize']=[50, 10]
plt.show()

graph each tree like this. When I do something like:

  dump_list[0]

it gives me the tree as a text. But I couldn't find any way to extract a tree as an object, and use it.

https://github.com/dmlc/xgboost/issues/117#ref-commit-3f6ff43
I found this but didn't really understand what is suggested.

Progress: I tried to somehow turn

  dump_list[0]

string object into a sklearn DecisionTreeClassifier object. Still no luck.

I uploaded my notebook if you want to check it out: https://github.com/sciencelove11/Question

This is an open feature request (at time of writing):
https://github.com/dmlc/xgboost/issues/2175
https://github.com/dmlc/xgboost/issues/3439
There, a very wasteful but working solution is mentioned: predict using ntree_limit for each number of trees of interest.  I've put together a quick demonstration Colab notebook here.
It also has been asked several times over at SO, see e.g.
https://stackoverflow.com/questions/51681714/extract-trees-and-weights-from-trained-xgboost-model
https://stackoverflow.com/questions/37677496/how-to-get-access-of-individual-trees-of-a-xgboost-model-in-python-r
and their Related questions.
In the first link, another workaround is mentioned: by dumping to text/PMML, you should be able to reload each individual tree (or subsets thereof) and make the predictions.  It's not clear how to make this work though: XGB itself doesn't have an easy way to load a model except from its own binary format.  You might be able to do it by parsing the output (JSON seems most promising) into another library with tree models.


How to determine the abnormality of a specific variable by taking into account all the other variables in the data? I have an issue of machine learning/anomaly detection. Indeed, I have a variable Y and several other variables X. The purpose is to quantify the degree of abnormality of the data on Y but I have to take into account the values on the other variables (the relationship between Y and X).
Normally, an anomaly detection algorithm would find anomalies but on the whole data (Y + X), but in my case I want to zoom in on Y because it is a very important variable. If I wanted to quantity the abnormality on all my variables (Y + X), Y would be lost in the middle of all the variables.
It is not something strange because when you apply a linear regression Y ~ X, you can calculate the Cook distance which is a kind of "abnormality score" and it took into account the relationship between Y and X.
I hope it is clear!

If you want to focus on the outliers wrt the class, you can do as follows:
Using Isolation Forest
import pandas as pd
import numpy as np

from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler, FunctionTransformer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import IsolationForest
from sklearn.decomposition import PCA
from sklearn.pipeline import Pipeline

import plotly.express as px

X, y = load_iris(return_X_y= True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 42)

out_model = Pipeline([("model", IsolationForest(random_state= 42))]).fit(X_train)

visualizer = Pipeline([("scaler",StandardScaler()),
                ("decomposer",PCA(n_components= 3)),
                ("framer", FunctionTransformer(lambda x: pd.DataFrame(x, columns = ["p1","p2","p3"])))]).fit(X_train)

outliers = out_model.predict(X_train)
X3D = visualizer.transform(X_train)

px.scatter_3d(data_frame=X3D, x = "p1", y = "p2", z = "p3", color = y_train, symbol=outliers )

pd.crosstab(index = y_train, columns= outliers, margins= True, normalize= 0 )

Output:

We can see that ~ 35% of observations that belong to class 2 are marked as outliers
This is a good starting point to analyze the reasons behind this.
Hope it helps


Change aliases of filter items in Tableau Data sample contains a single feature: random integer number from 1 to 4.

Is it possble to change 1,2,3,4 representation on the filter card to some custom names, say: Type1,Type2,Type3,Type4? (not changing data set)

Found it myself.

Go to context menu right clicking to the dimension field.
Go to Aliases... and change the labels.


What exactly is .csv in machine learning? I already have dataset of dogs and cats , so do i need to make .csv file or can i directly use the dataset for classification

A csv file is a way to store an array (similar to an excel table).
With a code like this one, you can save a pandas dataframe to a csv file:

 import pandas as pd
 df = pd.DataFrame({'c1':[1, 2, 3, 4], 'c2':['a', 'b', 'c', 'd']})
 df.to_csv('df.csv')

which will look like that

,c1,c2
0,1,a
1,2,b
2,3,c
3,4,d


Supervised Learning Quiz website? What are some good websites which  provide quiz questions on supervised learning and machine learning in general?
I have a quiz coming up and I would like to be prepared for it.

Depends on which Quiz you are talking about. But these might help.

This is for AWS Machine learning exam:

https://d1.awsstatic.com/training-and-certification/docs-ml/AWS%20Certified%20Machine%20Learning%20-%20Specialty_Sample%20Questions.pdf

And this is for Microsoft Azure Machine Learning exam:

https://www.itexams.com/exam/70-774

Take a look.


Extracting MM-YYYY from python date and creating a new column with the same I want to extract the month and year from one of my date columns in the dataset and create a new column in the data-frame with the new MM-YYY format.

My current solution is working fine but its way to long. I am looking for an efficient way to do this.

Date point looks like this: 2017-07-26 and its in datetime64[ns] format and I want the output to look like this: Jul-2017

import datetime
df1['Month'] = df1['booking_date'].apply(lambda x: datetime.datetime.strftime(x, "%B-%Y"))


Problems with decision tree labeling of nodes Decision trees as we know assigns label to the node based on majority class voting. I am curious to find that what could be the problems with such labeling schemes? Does it lead to overfitting the data?

Decision Tree does assign the label based on majority given the attribute test condition and its value.

Regarding the class label assignment-

In case DT has a longer  depth, there might not be enough instance left for a certain branch/test condition/node . then this might not be the reliable estimation of the class label statistically. This is also called Data fragmentation problem.

so a DT with 50 nodes, at dept 10, for day = Humid there is only 1 instance left which is -ve. So Its assigned as -ve but there is not enough data ideally to support this.

One way to solve this is to dis-allow to grow the tree beyond a certain threshold in terms of number of node i.e. stopping condition.

Which also brings us to Over-fitting, Regarding Over-fitting- There is this classic Error vs number of nodes graph on train and test to show how over-fitting happens in DT.

As you can see in below graph, tree with more number of nodes has lower training error but while its being tested error is higher. The gap between test and training error is telling us that the tree is over-fitting/has captured the noise when tree size is growing.

Now Random Forest is a Assembly/forest of multiple Decision Trees. While classifying the example we take majority voting out of Trees.


Does shuffling the training data matter in a K Nearest Neighbors Classifier model? I am new to machine learning and I have a couple of questions about a project.
So, I created a classifier using the MNIST data set for a ML project that I was working on. I augmented the data by shifting each original point 2 pixels up, down, left, and right (so I have 240,0000 new images in addition to the 60,000 original ones). I was wondering if I should bother retraining my model with a shuffled training set, since I think (please correct me if I am very wrong) that the order of the training data in a KNN classifier does not play a factor in the final model, since predictions are made on nearest neighbors and shuffling the dataset will not change these neighbors. *As a side note, I am using the classifier/model on user drawn images, and the one with the augmented data set (unsurprisingly) performs better than the model trained only with the original data.
However, I seem to be getting the same accuracy score (the exact same to the 4th decimal place) on my test set with my augmented data set (30,000 total images) as I did with my original data set (60,000 total images). Is this something to be concerned about/did I do something wrong?
Thanks so much for your help!

If you are talking about K-nn algorithm per se, then, you do not have any advantage in shuffling your training set. Indeed, as you said, K-nn is just looking for the K nearest neighbors and does not care at all about the order of the samples, the algorithm will scan the entire training set for a single prediction, no matter what (unless you are using some efficient implementation like ball-tree or kd-tree).
Regarding the second point of your question, I am wondering what your pipeline is: are you augmenting the dataset before splitting in train-validation-test? If so, I think you are injecting some bias in the evaluation of you algorithm, because you are evaluating your algorithm in samples that do not exist. You should augment only the training set and then validate/test your algorithm on real data.


Wanting to overfit model to say if data has been seen before I would like to train a scalable model that has as an input row of a database and has an output of either 1 or 0 depending upon whether it has seen this entry of the database before or not. The purpose of this is to then be able to check against a similar database and determine which entries are the same in the first database, without ever having to share the database. I can't hash the entries and hash-match or any other techniques like that.

That is, let's say two organizations are both using a person database with just floats and with columns = ["name" : float, "age": float, "net_worth": float]. Now, imagine that I am the first group, and I train a model that will overfit on my data, 'memorizing' it in some way. Then, what I want to be able to do is send that model to the other organization, who could apply it to each element of their table. This will then tell the other organization which entries of their dataset are also shared by me.

I understand that this is a unique (if not downright weird) way to try and solve this problem, but this is the approach that I'm trying to take. Does anyone have any solutions they can think of?

I'm not sure if a machine learning approach is a good choice for such a task. I am tempted to say no. The reason is mainly, that ML usually is used for the discovery of patterns or in other words of similarities while they are designed to generalize so they still recognize a pattern even if it looks different due to some randomness (noise etc.).

In your case however it sounds more like you want to discover exact matches.

You also have to consider, that you cannot store your data in an arbitrarily compact form without reducing information. So even if you would find a suitable ML model for your purpose, you cannot expect it to consume much less space as the part of the database for which you want to find matches in the other database.


NLP: To remove verb and find the match in a sentence Is there a NLP method like stemming, lemmatisation to figure out the below?

1=2
3=4

1) provide technical documentation
2) technical documentation
3) use software design patterns
4) software design patterns

As indicated by @hssay, your way seems to be PoS tagging and then removing verbs. If you don't want to get your hands dirty, you might prefer to use the off-the-shelf Google natural language web API. If you try the UI, click on the Analyse button, and then under the Syntax tab, look for Part of Speech and VERB.

Another similar toolkit that you could use as an API is StanfordNLP.


How do various statistical techniques (regression, PCA, etc) scale with sample size and dimension? Is there a known general table of statistical techniques that explain how they scale with sample size and dimension? For example, a friend of mine told me the other day that the computation time of simply quick-sorting one dimensional data of size n goes as n*log(n).

So, for example, if we regress y against X where X is a d-dimensional variable, does it go as O(n^2*d)? How does it scale if I want to find the solution via exact Gauss-Markov solution vs numerical least squares with Newton method? Or simply getting the solution vs using significance tests?

I guess I more want a good source of answers (like a paper that summarizes the scaling of various statistical techniques) than a good answer here. Like, say, a list that includes the scaling of multiple regression, logistic regression, PCA, cox proportional hazard regression, K-means clustering, etc.

Most of the efficient (and non trivial) statistic algorithms are iterative in nature so that the worst case analysis O() is irrelevant as the worst case is 'it fails to converge'.

Nevertheless, when you have a lot of data, even the linear algorithms (O(n)) can be slow and you then need to focus on the constant 'hidden' behind the notation. For instance, computing the variance of a single variate is naively done scanning the data twice (once for computing an estimate of the mean, and then once to estimate the variance). But it also can be done in one pass.

For iterative algorithms, what is more important is convergence rate and number of parameters as a function of the data dimensionality, an element that greatly influences convergence. Many models/algorithm grow a number of parameters that is exponential with the number of variables (e.g. splines) while some other grow linearly (e.g. support vector machines, random forests, ...)


Gerrymandering - Geospatial optimization to maximize votes in R I am looking for a practical guide/tutorial preferably in R to show how to do gerrymandering. (I was looking for it also in CRAN but didn't find such package) Gerrymandering is the manipulation of the boundaries of electorial districts in order to gain political advantage for one party. If there is an analogy/similar process in another area it would be also interesting.

I have detailed historical  election results. I can see the results from each "voting office" and also the geographical area they cover. These results are aggregated to electorial district level. I'd like to see how moving voting offices to different electorial districts can influence the results of the election and how this process can be optimized for one party. Certainly there should be some constraints not to create really weird shaped districts at the end.

I found a number of interesting articles on mathematical analyses of gerrymandering. Hopefully some of these might give you some ideas on detecting / optimising against gerrymander.

Gerrymandering and Legislator Efficiency
Gerrymander and Convexity
Recursive Algorithm with Examples
Mary Solbrig's Automated District Model


How to load this data from .dat into dataframe using python 1 1:31080.410200 2:2.871828 3:5.862267 4:7.100850 5:8.283706 6:-5.427875 7:-6.667087 8:-8.888233 9:28898.943400

Can someone please tell me how to load this data into a dataframe from .dat file. The data is given such that attribute number:value. I want only the values into  the dataframe

Given just one line of the data, it's a little hard to go off of, but I'm assuming you're trying to get at the number after each colon, and the number before it refers to the column name?

If so, you can use read_csv with a little tweaking:

import pandas as pd
from pandas.compat import StringIO

temp='1 1:31080.410200 2:2.871828 3:5.862267 4:7.100850 5:8.283706 6:-5.427875 7:-6.667087 8:-8.888233 9:28898.943400'
#after testing replace StringIO(temp) to filename

df = pd.read_csv(StringIO(temp),
                 sep="\s+", #separator whitespace
                 index_col=0,
                 header=None)

for c in df.columns.values:
    df[c] = df[c].apply(lambda x: float(str(x).split(':')[1]))

df.head()

Which will output:

    1           2           3           4       5           6           7           8           9
1   31080.4102  2.871828    5.862267    7.10085 8.283706    -5.427875   -6.667087   -8.888233   28898.9434


What algorithm can be used to divide a time series Hi everyone

I have a data as on the image. I would like to derive a line that separates the data (like the red line), but not sure what algorithm can be used.

I thought about k-means in 1-d data, but then the high points (i.e. y-value ~0.01 are not really in the same cluster with the data point of 0.002).

Thanks a lot.

P/S: In fact, the values here are reconstructed MSE from h2o Autoencoder, and now I want to find an automatic way to determine the threshold, to say a point is an anomaly or not.

First of all, there is nothing wrong with using a threshold.

If you can't manually choose the threshold, use standard statistics. For example trim the largest 10% of points. Then compute the mean, standard deviation, and use 2*stddev as threshold.


Anyone have fruit disease dataset? I am doing a project on fruit disease recognition and classification. Anyone have an existing dataset of fruit diseases? Can you help me to find one?

I work on plant disease image recognition for my Master's thesis. Until now, I did not find any good database with fruit images, but there are two quite large databases which contain leaf-images of infected plants. One is the Plantvillage-dataset which has pictures taken under laboratory conditions and the other one is the Digipathos-dataset which contains pictures that are nearer to real-life scenarios with multiple objects in focus and difficult backgrounds.

You can download the Plantvillage-dataset from this Github-Repo. For the Digipathos-dataset, I wrote a downloader which you can also find on Github.


Identifying parts of speech for individual words (not texts) Please forgive the potential ignorance of this question. I am not a complete virgin regarding NLP:

I am trying to assign parts of speech to individual words. I realize that many words can serve as multiple parts of speech, but I am looking for the POS that a human would be most likely to interpret the word as, seeing it out of context.

Is there any database that has this info in it for a reasonably large set of words? Or, is there any large corpus with POS tags where I could feed it  words, and get frequency distributions of the POS that each word is playing in the corpus.

Thanks.

A POS tagged dataset that might help: link