Analyzing country data. Does duplicating observations by population for regression make sense? Question: While analyzing country happiness data via OLS regression, should I duplicate observations based on country population?

Example: If duplicating per million, the U.S. would have 327 observations and Denmark would have 6. Currently every country has 1.

If it matters...

y: Happiness score

X: GDPPerCapita, InfantMortalityRatePer1000, UnemploymentPercent, GiniIndex, etc.

Function to duplicate observations based on population:

def create_population_weighted_df(df):
    df_pop = pd.DataFrame(columns=df.columns)
    for index, row in df.iterrows():
        for i in range(int(row['PopulationInMillions'])):
            df_pop = df_pop.append(row)
    return df_pop

Thanks.

I see what you're doing, and understand the thinking behind it, but don't think it makes sense with this problem. By assigning 300 variables the same values, you're effectively saying that each of these million people have the average economic output (i.e., if you're using GDPpercapita) for all the people in your dataset. That's a pretty misleading understanding of the distribution of that variable, and will lead to poor analysis. Same goes for other aggregate factors.

If you're looking to do a regression where those are incorporated into your analyses, using the unscaled population data as one of your features will include that information as one of the coefficients you return. So you can get back the result on happiness of having a million more/less people. No need to duplicate the data.


Identify same product I am new to ML and still learning it.

My problem is to identify duplicate products. I have a dataset containing product details such as name, colour, size, description, features etc (there are roughly 70 columns).

I need to remove duplicate products.

I just completed some of the supervised ML model(classification and regression) and unsupervised clustering(K means and HC). I am also on the way of learning w2v and d2v.

But due to time constrain, I need to deliver a solution to the above problem statement. I am unsure as to how to proceed.

Any help and guidance would be appreciated

This problem is called record linkage, there are various techniques which can be used, usually involving some distance measure between record and/or approximate string matching between string fields.

Fyi it's a quite complex problem, especially if quality deduplication is expected and the volume of data is high.


Why is predict_generator is returning an empty array? I am trying to print the predicted labels of my test data but the predict_generator() function is returning an empty array.

My Model:

from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Dense, Flatten

train_directory = 'D:\D_data\Rock_Paper_Scissors\Train'
training_datgagen = ImageDataGenerator(rescale = 1./255)
training_generator = training_datgagen.flow_from_directory(
    train_directory,
    target_size = (28,28),
    class_mode = 'categorical')

validation_directory = 'D:\D_data\Rock_Paper_Scissors\Test'
validation_datagen = ImageDataGenerator(rescale= 1./255)
validation_generator = validation_datagen.flow_from_directory(
    validation_directory,
    target_size = (28,28),
    class_mode = 'categorical'
    )

filenames = validation_generator.filenames
nb_samples = len(filenames)

model = Sequential()
model.add(Flatten(input_shape = (28,28,3)))
model.add(Dense(128,activation = 'relu'))
model.add(Dense(64, activation = 'relu'))
model.add(Dense(16, activation = 'relu'))
model.add(Dense(3, activation = 'softmax'))

model.compile(optimizer = 'adam', loss = 'categorical_crossentropy',metrics = ['accuracy'],)

model.fit_generator(training_generator,epochs=20,validation_data = validation_generator)
predict = model.predict_generator(validation_generator,steps = nb_samples)

print(predict)

The output :

Found 1748 images belonging to 3 classes.
Found 0 images belonging to 0 classes.
Epoch 1/20
55/55 [==============================] - 4s 64ms/step - loss: 0.9951 - accuracy: 0.4908
Epoch 2/20
55/55 [==============================] - 4s 73ms/step - loss: 0.7280 - accuracy: 0.7248
Epoch 3/20
55/55 [==============================] - 6s 109ms/step - loss: 0.5783 - accuracy: 0.7895
Epoch 4/20
55/55 [==============================] - 5s 85ms/step - loss: 0.4649 - accuracy: 0.8387
Epoch 5/20
55/55 [==============================] - 4s 71ms/step - loss: 0.3955 - accuracy: 0.8707
Epoch 6/20
55/55 [==============================] - 4s 74ms/step - loss: 0.3595 - accuracy: 0.8804
Epoch 7/20
55/55 [==============================] - 4s 70ms/step - loss: 0.3247 - accuracy: 0.8919
Epoch 8/20
55/55 [==============================] - 4s 66ms/step - loss: 0.3435 - accuracy: 0.8827
Epoch 9/20
55/55 [==============================] - 4s 65ms/step - loss: 0.2740 - accuracy: 0.9102
Epoch 10/20
55/55 [==============================] - 3s 60ms/step - loss: 0.2141 - accuracy: 0.9336
Epoch 11/20
55/55 [==============================] - 3s 61ms/step - loss: 0.1836 - accuracy: 0.9462
Epoch 12/20
55/55 [==============================] - 3s 63ms/step - loss: 0.1722 - accuracy: 0.9416
Epoch 13/20
55/55 [==============================] - 3s 62ms/step - loss: 0.1788 - accuracy: 0.9394
Epoch 14/20
55/55 [==============================] - 3s 63ms/step - loss: 0.1331 - accuracy: 0.9571
Epoch 15/20
55/55 [==============================] - 4s 68ms/step - loss: 0.1343 - accuracy: 0.9537
Epoch 16/20
55/55 [==============================] - 4s 65ms/step - loss: 0.1033 - accuracy: 0.9680
Epoch 17/20
55/55 [==============================] - 3s 62ms/step - loss: 0.1001 - accuracy: 0.9651
Epoch 18/20
55/55 [==============================] - 3s 62ms/step - loss: 0.1209 - accuracy: 0.9565
Epoch 19/20
55/55 [==============================] - 3s 61ms/step - loss: 0.1187 - accuracy: 0.9559
Epoch 20/20
55/55 [==============================] - 3s 63ms/step - loss: 0.0834 - accuracy: 0.9737
[]

The code looks OK to me... I can only think that you have somehow exhausted the ImageDataGenerator object, because you use the same on during training and then for the prediction. (I do realise this should never happen).

To rule this out, you could create a separate generator for testing.

This would also have the advantage of using some of your data for a hold-out test set, so your final predictions are out-of-sample, giving you a better idea of performance  :)

For example, you might change your code to be like this:

validation_directory = 'D:\D_data\Rock_Paper_Scissors\Test'
holdout_directory =  'D:\D_data\Rock_Paper_Scissors\Holdout'   # make this new directory

Move e.g. 10 files into the new holdout directory

validation_datagen = ImageDataGenerator(rescale= 1./255)
validation_generator = validation_datagen.flow_from_directory(
    validation_directory,
    target_size = (28,28),
    class_mode = 'categorical'
    )

# new generator for predictions
holdout_generator = validation_datagen.flow_from_directory(
    holdout_directory,
    target_size = (28,28),
    class_mode = 'categorical'
    )

model.fit_generator(training_generator, epochs=20, validation_data=validation_generator)

Set the number N accordingly, e.g. to the 10 files in the new directory

predict = model.predict_generator(holdout_generator, batch_size=N)

print(predict.shape)    # this should return the shape of the numpy array: (N, H, W, C)
print(predict()        # the actual rpedictions


Continuous VS Categorical variable In the dataset I have a continuous variable AGE and categorical variable AGE_CATEGORY as well. They are both highly correlated.

Which method should I use to identify the feature to be dropped AGE OR AGE_CATEGORY?

If your goal is to use them to train a supervised machine learning, the best solution is to find out which one is more efficient in predicting your output.

AGE has more information than AGE_CATEGORY. So, If I were to remove one of them I would remove AGE_CATEGORY.

In Addition, if your goal is to train tree-based models AGE_CATEGORY is not gonna be that much efficient.

You can use A/B test to find out which feature is more efficient in predicting your output.


Is dimension reduction helpful to select features for a classification problem? Let's say I have a data set but I don't know what features are relevant to solve a classification/regression problem.

In this case, is it worth/good to use a dimension reduction algorithm and then apply a classification algorithm? Or can I  just select "randomly" my features by using my common sense and then try to tune my algorithm next?

Also if someone has some explanation of a dimension reduction "in real life with real use case" it would be great because I feel my comprehension of dimension reduction is wrong!

Well, let's say it depends on the distribution of your data. In approaches like PCA the approach does not care about the labels of the data in hand. This is why PCA may lead to data which are sometimes difficult to be separated or vice versa. PCA just cares about which direction leads to more variance and take that direction as a new basis. Not caring about the labels is why you cannot say it may lead to a better space for classification or not. You have to employ that and after that, investigate whether it's helpful or not. Approaches like LDA or other variants of that take care of the labels but they are linear classifiers which are not strong at least in a current feature space where you've not done any feature engineering.


How to formulate reward of an rl agent with two objectives I have started learning reinforcement learning and trying to apply it for my use case. I am developing an rl agent which can maintain temperature at a particular value, and minimize the energy consumption if equipment by taking different actions that are available for it to take.

I am trying to formulate a reward function for it.

energy and temp_act can be measured

energy_coeff = -10
temp_coeff = -10

temp_penalty = np.abs(temp_setpoint - temp_act)

reward = energy_coeff * energy + temp_coeff * temp_penalty

This is the reward function I am using, but intuitively , I feel it should be better. because absolute value of enenrgy and temp_penalty are on different scales. How do i take into count the scaling problem, while structuring a reward.

In general it is not possible to simultaneously optimise two separate objective functions. Your approach of adding weights (your coefficients) to each objective, then summing the scaled objectives, is a standard way of resolving that.

As your penalties are on different scales and in different units, it is your task as the engineer setting the objective to provide the conversion to a single scale. That is what the coefficients represent - you can even think of them as $points/Joule$ for energy and $points/(\Delta K)$ for temperature difference.

Sometimes analysis will show you that there is a natural combined scale. For instance, in business settings it may be possible to frame compromises as financial costs, e.g. your coefficients might be $\text{GBP}/Joule$ for energy and $\text{GBP}/(\Delta K)$ for temperature difference. Then you have one clear objective to minimise cost or maximise profit.

If that is not possible - a financial cost for exceeding temperature bounds may be hard if this is about human comfort in a building - deeper analysis might lead to thinking about longer-term outcomes. Perhaps your initial rewards are too focused on immediate numerical issues (that appear easy to collect, but don't represent your true goals), and a re-framing of the problem could work. For instance, perhaps it is more reasonable that both the temperature and energy costs stay within strict bounds over a year of varying external temperatures and system workload, with a scaling penalty depending on how badly these are exceeded.


How to load the pre-trained BERT model from local/colab directory? Hi i downloaded the BERT pretrained model (https://storage.googleapis.com/bert_models/2018_10_18/cased_L-12_H-768_A-12.zip) from here and saved to a directory in gogole colab and in local .

when i try to load the model in colab im getting "We assumed '/content/drive/My Drive/bert_training/uncased_L-12_H-768_A-12/config.json" . tried to laod the model in local machine and getting same error .

this is how i loaded the model:
from transformers import BertForMaskedLM
BertNSP=BertForMaskedLM.from_pretrained('/content/drive/My Drive/bert_training/uncased_L-12_H-768_A-12/')

is this the correct way of loading model from the directory when i have downloaded the pretrained model ?
Im getting error " '/content/drive/My Drive/bert_training/uncased_L-12_H-768_A-12/config.json' "
the downloaded model had these naming conventions where file name start with bert_ but the BertForMaskedLM class is expecting the file name to be config.json .

bert_config.json
bert_model.ckpt.data-00000-of-00001
bert_model.ckpt.index vocab.txt
bert_model.ckpt.meta

FULL ERROR:
Model name '/content/drive/My Drive/bert_training/uncased_L-12_H-768_A-12/' was not found in model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased). We assumed '/content/drive/My Drive/bert_training/uncased_L-12_H-768_A-12/config.json' was a path or url to a configuration file named config.json or a directory containing such a file but couldn't find any such file at this path or url.

when i renamed the above 4 files by removing bert from all 4 file names , i get this error even though the "model.ckpt.index" files exist

ERROR:
"OSError: Error no file named ['pytorch_model.bin', 'tf_model.h5', 'model.ckpt.index'] found in directory /content/drive/My Drive/bert_training/uncased_L-12_H-768_A-12/ or from_tf set to False"

You can import the pre-trained bert model by using the below lines of code:
pip install pytorch_pretrained_bert

from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForNextSentencePrediction

BERT_CLASS = BertForNextSentencePrediction

# Make sure all the files are in same folder, i.e vocab , config and bin file
PRE_TRAINED_MODEL_NAME_OR_PATH = '/path/to/the/files/containing/models/files'

model = BERT_CLASS.from_pretrained(PRE_TRAINED_MODEL_NAME_OR_PATH, cache_dir=None)


PCA and k-means for categorical variables? I have a clustering task at hand. The data that I have contains only categorical variables. So, k-modes seemed like the best option. But I am not sure what are the data pre processing steps required for the same ?
What I am doing right now is the following:

label encoding features which have ordinal values.

one hot encoding the others.

and that's all I am doing as part of data preprocessing steps. My feature space gets to 50 from original 4 after doing above mentioned  steps. I am getting 17 clusters as best number of clusters for silhouette score 0.60.
Also, I think doing Principal Component Analysis (PCA) to reduce dimensions and feature scaling doesn't make sense here as if I do this, I might as well use K-means. Would it be a good decision to run PCA and then use k-means for categorical variables ?

No - PCA and k-means can not be used on categorical variables. Both PCA and k-means require numerical variables.


Computing Item-to-Item Similarity Using Cosine I have a "User x Item" matrix as below:

user item1 item2 item3
u1   2     0     3
u2   1     2     0
u3   4     3     1
u4   0     2     2

I want to computer the similarity between items based on users.
For example, to calculate similarity between items i1 and i2, I only choose users who have assigned values to both of these items.

The cosine similarity between two items (i1 and i2) is as follows:

cos(v1,v2) = (1*2 + 4*3)/sqrt[(1 + 4)*(16+9)]

My question is for item-to-item similarity, should I consider all users, or just common users who assigned values to the items?

Filtering users will create bias in your training data. This may be good or bad, depending on your data and goals. The best way to find out, for your specific system, is to try and test both methods, optimizing for whatever metric you choose is best.

Honestly, I think you should consider all users. Say you have a system for recommending movies. In this system, i1 represents a children's movie, that is rated by users u1 through u5, with u5 being a parent and u1, u2, u3 and u4 children. Similarly, i2 represents a terror movie, that is rated by users u5 through u9, all adults. In such a system, u5 being the only common ground between i1 and i2, if this user rated both positively and you only consider them, then you will have a high similarity between the items; but if you consider all users that rated either movie, you will have a very low similarity.

Or in other words, I think intuitively it's best to consider a missing rating as a false/negative rating. Excluding terms where both ratings are zero makes no difference in the cosine similarity anyway. But again, you should run and test it.


Cosine Similarity but with weighting for vector indexes I am very new to NLP and although this seems like a basic question I don't know how to search for an answer online.
This is my problem: I have extracted and ranked keywords from 2 text sources:

A rank of 1, means this keyword is more important than a keyword with rank 5. Some keywords may not exist in one text but do in the other. In this case, where the keyword does not exist, there is no rank, therefore, Nan.
What method do I need to use to extract the similarity between the keyword ranks? I want to find out how similar the 2 texts are based on what keywords it contains and the ranks of these keywords.
I have tried cosine similarity by removing rows that contain Nan values and then treating text1Rank and text2Rank as vectors like this:

the 2 columns are the vectors I pass into the cosine similarity formula.
However, I do not think that this method weighs higher-ranked keywords more than lower-ranked keywords. Am I correct in thinking this?
If so, what method should I use to compare the 2 sets of ranks of keywords?

Cosine similarity won't work very well because it's only based on whether the rank at position $i$ is the same in vector 1 and vector 2.
For instance the vectors [3,2,4,1,5] vs [2,3,5,1,4] will have very low similarity because 4 positions out of 5 are different, even though there are only two swaps between (2,3) and (4,5).
A much better way to measure the similarity between two rankings is Spearman Rank Correlation.
Note that (if I'm not mistaken) in this case you could also directly use Pearson correlation, since the numeric values are already ranks. Spearman just ranks the values before applying Pearson correlation. So normally the two will give you the same result.
Note also that this method won't give more weight to the top of the ranking than the bottom. I'm not aware of a measure which does this, other than defining a custom weighted measure.


Understanding LSTM Keras implementation So I understand what LSTM units are. But I have trouble understanding the implementation / function in Keras framework.

Let's say, I add a layer

model.add(LSTM(5, ...))

does this mean, I have

five different LSTM units, that are independent, or
one LSTM unit, that is unfolded into five time steps?

I assume that the second answer is right, because else a lot of models I saw would not make much sense.

When I have as first hidden layer a LSTM layer with size 5, does this mean my input should be a time series with 5 time steps?

Please correct me if I'm wrong. I am confused because I thought one LSTM would use its own state in the next iteration.

What is the correct meaning of LSTM in Keras?

I am afraid both of your description is incorrect but the first one is the closest.

Each cell/timestep will actually produce a hidden state representation. However, this is surpressed by default on keras. You can re. As LSTM without Bidirectional wrapper produce a one-way sweep through your sequence, your output from LSTM(5) is the hidden state of the final LSTM cell(cell output not cell state).

Now to enable outputting the sequence of hidden output similar to what you might have learned about LSTM, simply supply return_sequence=True.

To compare output shape:

Without return_sequence=True, Input shape from [batch_size,sequence_length,previous_hidden_dim] to [batch_size,hidden_dim]

With return_sequence=True,
Input shape from [batch_size,sequence_length,previous_hidden_dim] to [batch_size,sequence_length,hidden_dim]

What you are referring as "folding" of LSTM is actually iautomatically based on sequence length of previous input.

In case of any doubt on my explanation, please read the material I give you below. You can verify my description by calling model.summary().

So basically what led to your confusion is due to keras LSTM default behaviour of only outputting the final cell output.

LSTM link


What does a blue empty cell mean [ ] in Anaconda if the cell above is still [*]? I've been running a machine learning algorithm for two days now, I have a large dataset and a slow machine.
I noticed today that the cell below the "fit model" cell turned from green to blue, however, the cell above (the one with fit) is still running [*]

The cell with [*] means it is currently running, you can queue other cells for execution but until this cell finishes they will not run.
When it is empty [] it means you haven't run this cell.


How to group by ID, and output without repeating IDs? Input

UserID  Profile
101     apple
102     banana
103     pie
101     banana
102     pine
103     berry

Output

UserID  Profile
101     apple
        banana
102     banana
        pine
103     pie
        berry

This problem will by solved by group by function.

Whatever tool you are using (R or SQL), you need to just group the data by UserID.


Implementation of a distributed data mining paper I have a project about distributed data mining and I need to do some implementations, So I've searched and found this paper. The address of dataset is mentioned in the paper and I've downloaded it. For the process I should split the dataset into 10 smaller datasets. And the other task is using Weka4WS (weka for web services) for the process (For clustering part).

So my questions:
1. How can I split the dataset using python code?
2. What is Weka4WS and how should I use it, I mean the implementation (because it's not something like apps like the Weka itself)?

Splitting your dataset

First you can load your data set to memory:

# Load the data to memory
with open('file.txt', 'r', encoding='utf-8') as file:
    header = file.readline()
    data = []
    for line in file.readlines():
        data.append(line)

Next, you use KFold to split your data into $10$ splits and save to $10$ different files.

# Split the data and save the data
from sklearn.model_selection import KFold
kf = KFold(n_splits=10)

split = 1
for _, split_index in kf.split(data): # Iterate over the splits

    with open('file_{}.txt'.format(split), 'w+', encoding='utf-8') as file: # Open a file with name file_{split number}
        file.write(header) # write the header of the file
        for index in split_index: # Write each point
            file.write(data[index])
    split += 1 # Increase split number

Weka4WS

The Weka4WS is an extension to Weka itself. Yes, it is like an application like Weka, with a Graphical User Interface. From the docs:

  Thus, Weka4WS also extends the Weka GUI to enable the invocation of the data mining algorithms that are exposed as Web Services on remote Grid nodes.

However, it is now as easy to run as the normal Weka. You need some additional steps to use it.

The official installation is here: http://scalab.dimes.unical.it/weka4ws/howtos/installation/

Here is a tutorial to run the application on Windows: http://scalab.dimes.unical.it/weka4ws/howtos/run-the-client-on-windows/

This is how the app looks like, this is the KnowledgeFlow screen:


Understanding of number of cells in layers of sequential models I am trying to teach myself RNN, but I have a question.
And so, imagine 2 layers: an input layer with three neurons $(x1, x2, x3)$ and a classic recurrent layer with 2 neurons and an activation function f. I will write out the outputs of each neuron of the recurrent layer. $ht1 = f (W * [x1, [0, 0, N]] + b) ht2 = f (W * [x2, ht1] + b)$. It turns out that $x3$ is not used, what to do in this case?
And also, let's imagine a slightly different RNN architecture.
An input layer with two neurons $(x1, x2)$ and a classic recurrent layer with 3 neurons and an activation function f. I will write out the outputs of each neuron of the recurrent layer. $ht1 = f (W * [x1, [0, 0, N] + bias]) ht2 = f (W * [x2, ht1] + bias)$. It turns out that the 3rd neuron of the RNN layer is not used, what to do in this case?
Please help me figure out how the neural network works in these cases. Thanks!
UPD:
I realized that i don't know how recurrent neural  networks work if  number of neurons in recurrent layer doesn't equal(!=) number of inputs
I have one thought:number of inputs has to always be equal number of neurons in RNN layer.
But code below contradicts with my guess.
model = Sequential()
model.add(Embedding(maxWordsCount, 256, input_length = inp_words))
model.add(SimpleRNN(128, activation='tanh'))
model.add(Dense(maxWordsCount, activation='softmax'))
model.summary()

That's model for predicting next word.

RNN cell versus neuron
You are hesitating between RNN cells and neurons. I understand you refer to RNN cells in your question. And so, traditionally any layer of a sequence model will always have the same number of cells as the length of your sequence or embedding size.
See difference between cell and neuron here: Difference between cell state and hidden state
See input size for sequence models here: How can I picture an unfolded RNN as a normal Feed Forward Network?
Re; code snippet
In SimpleRNN, the first argument is the number of units in each cell i.e. number of neurons in each cell. The number of cells is not an argument in sequential layers, and always is equal to the length of your input sequences or size of embedding layer.
model = Sequential()
model.add(Embedding(maxWordsCount, 256, input_length = inp_words))
# Below, 128 is the neurons of each cell, and relates to the cell memory capacity.
model.add(SimpleRNN(128, activation='tanh'))


Difference between packaged sentiment analysis tools (TextBlob/NLTK) and training your own classifier? I'm new to ML and training classifiers in practice, so I was just wondering what the difference was between the built-in sentiment tools of packages such as NLTK and TextBlob as compared to manually creating a classifier (training, testing, etc). I think I read in a comment somewhere that Textblob/NLTK's existing sentiment analysis tools basically just tokenize the text and count the number of positive/negative words to determine an overall sentiment rating (not sure how accurate this is). Does anyone know if using a custom classifier would, in general, be a better way to doing sentiment analysis of text (I'm looking at analyzing the sentiments expressed in hotel reviews)?

In general, training your own classifier is likely to perform better but it's going to take more time and effort:

Probably the NLTK system was trained on some generic data which might be very different from your target text. Since any supervised system assumes the same distribution between the training and test set, it's always better to train on a sample of your target data.
By implementing your own method you can adapt it more specifically to your use case.

So it's a trade-off between the level of quality and the time you're ready to spend on annotating your own data and tuning your own classifier.


How to create a multi-dimensional softmax output in Tensorflow? I came across this research paper released by YouTube, on how they use deep learning neural networks for recommendations. It's located here: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45530.pdf

In the paper, the candidate generation neural network model outputs a softmax with 256 dimensions, which acts as an "output embedding" of each of the 1M video classes.

How is this possible to implement in tensorflow, for example? Isn't softmax supposed to be only 1-Dimensional. If the model outputs an "embedding" like this, as they say it does, how would the training data's labels be formatted as 256-dimensional? In other words, how do they compute the 256-dimensional vector for each of the videos in their training dataset?

Thank you so much for your time and help, guys!

You are confusing "dimensions" with "order of tensor". A softmax with 256 different categories is a 256 dimensional vector, but is also a tensor with order 1 (whilst a matrix is a tensor of order 2). The paper is using the technical terms correctly, so the 256 dimensional vector is just a normal vector with 256 scalar entries.

Therefore a 256-dimensional softmax in TensorFlow is typically an output layer that looks something like this:

y = tf.nn.softmax(tf.matmul(h, W) + b)

where h is the last hidden layer, W is the weight matrix n x 256, and b is the bias 1 x 256 vector.

  In the paper, the candidate generation neural network model outputs a softmax with 256 dimensions, which acts as an "output embedding" of each of the 1M video classes

That is a description of the training process that compresses 1M different inputs to 256-dimensional output for use as an embedding for recommendation matches. The softmax is at the output, and as far as I can see is just a normal softmax classifier output as seen in many other classifier networks (except the result is not technically being used to classify anything). I am not clear on what supervision data was used or on what the input representation was. However, I don't think it likely that 1M "classes" ever appear as e.g. 1-hot encoding, because that would not scale out usefully to the many other millions of videos - the point of the embedding is to turn disparate features of the videos into something that be used as a similarity measure, that can be run on any video stored in YouTube.


unsupported operand type(s) for -: 'list' and 'list' using python Here I have a data file and I designed neural network to predict value. I have a three inputs. These three inputs affect to predict value bysubtarcting and adding. If my three inputs are x1,x2,x3 . X1 and X2 add together and that  value  will subtract with the X3 value and give the predict value. This process will happened in x_train. I wrote a code. But it gives me this error. Can nyone help me to solve this problem?
My code is,

data = pd.DataFrame(data,columns=['X1','X2','X3'])
data = data.values
a = []
for i in range(0,len(data)):
    X1 = data[i][0]
    a.append([X1])
b = []
for i in range(0,len(data)):
    X2 = data[i][1]
    b.append([X2])
c = []
for i in range(0,len(data)):
    X3 = data[i][2]
    c.append([X3])
x_train =[]
x =[]
for i in range(0,len(data)):
    x_train.append([a[i] ,b[i], c[i]])
    x.append((a[i] - b[i] + c[i]))
x_train =np.array(x_train)
x = np_array(x)

You are facing the error on the line x.append((a[i] - b[i] + c[i])) due to expression a[i] - b[i] + c[i]. Let us see the how it is being calculated. Suppose I have the following data:

array([[1, 2, 3],
       [4, 5, 6],
       [7, 8, 9]])

Then the values of a, b and c are:

a
Out[20]: [[1], [4], [7]]

b
Out[21]: [[2], [5], [8]]

c
Out[22]: [[3], [6], [9]]

Let us take i=0 and print a[0]:

Out[23]: [1]

Hence what you are calculating is [1] - [4] + [7] instead you should be calculating 1 - 4 + 7. To do this change the line with error to:

x.append((a[i][0] - b[i][0] + c[i][0]))


Pandas: Assign back to table from grouping by column and index I am trying to implement Exponential Moving Average calculation on a DataFrame. The formula is

An additional complication is that my table is grouped and there is a unique bin number per group. This is what I tried

import numpy as np
import numpy.random as rand

n = 5
groups = np.array(['one', 'two', 'three'])
data = pd.DataFrame({
    'price': rand.random(3 * n) * 10,
    'group': np.repeat(groups, n),
    'bin': np.tile(np.arange(n),3)}, index=np.arange(3 * n))

print(data)

       price  group  bin
0   1.601310    one    0
1   3.190662    one    1
2   4.419421    one    2
3   3.817510    one    3
4   2.440774    one    4
5   6.832265    two    0
6   5.636502    two    1
7   4.630515    two    2
8   5.856423    two    3
9   0.916452    two    4
10  4.247134  three    0
11  7.146746  three    1
12  8.049161  three    2
13  7.852168  three    3
14  0.246720  three    4

This is how I am trying to implement the EMA calculation;

data['EMA'] = np.zeros(len(data.index))
data.loc[data['bin'] == 0, 'EMA'] = data.loc[data['bin'] == 0, 'price']

a = 2 / (n + 1)
for _, group in data.groupby('group'):
    for index, row in group.iloc[1:].iterrows():
        prev = group[group['bin'] == row['bin'] - 1].iloc[0]
        row['EMA'] = a * row['price'] + (1 - a) * prev['EMA'] # nope
        data.loc[index, 'EMA'] = a * row['price'] + (1 - a) * prev['EMA'] # nope

Unfortunately neither of these last lines update the values in the group. On the second iteration, the value of prev['EMA'] is still 0. What is the correct way to assign the values back to the table such that it is updated dynamically? Do I need to write out to a temporary array and write it back afterwards?

Additionally, I could not think of an elegant way to do this using assign or transform. If someone can solve that it might be a very good alternative.

Response

Thank you @DaFanat for your help. Unfortunately your code doesn't work. I tried the following

data.loc[:, 'EMA2'] = map(lambda x, y: x if pd.isnull(y) else x*a + (1-a) * y,
data['price'], data.groupby('group')['price'].shift(1))

But I get an error TypeError: object of type 'map' has no len(). I tried changing it to this

data['EMA2'] = list(map(lambda x, y: x if pd.isnull(y) else x*a + (1-a) * y,
data['price'], data.groupby('group')['price'].shift(1)))

And I do indeed get some results, but they do not look correct;

       price  group  bin       EMA      EMA2
0   5.407722    one    0  5.407722  5.407722
1   0.495734    one    1  3.770393  3.770393
2   7.911491    one    2  5.150759  2.967653
3   1.085836    one    3  3.795785  5.636272
4   7.326432    one    4  4.972667  3.166035

I am inclined to believe my implementation, since the price goes up from 0.5 to 7.9, how can the moving average go down? I think the indices are getting lost and it is putting the values against the wrong cells. How do I retain the indices when performing this calculation?

Solution

Thanks @DaFanat, you took me a long way. I finally figured out how to do it with a slight modification of your original method;

data['EMA2'] = data.groupby('group') \
                   .apply(lambda x: x['price'] * a + x['EMA'].shift(1) * (1-a)) \
                   .reset_index(level=0, drop=True)
data.loc[data['bin'] == 0, 'EMA2'] = data.loc[data['bin'] == 0, 'price']

print(data)

       price  group  bin       EMA      EMA2
0   3.556171    one    0  3.556171  3.556171
1   5.637241    one    1  4.249861  4.249861
2   3.278771    one    2  3.926164  3.926164
3   7.343718    one    3  5.065349  5.065349
4   6.128884    one    4  5.419861  5.419861

Not using the list(map()) ensures that the result is a DataFrame with indices retained, so it knows where to insert the individual rows.

EDIT:
Couldnt find a clean solution but this function should do the trick
def calc_ema(cur, *prevs):
    rele_prevs = [x for x in prevs if not pd.isnull(x)]
    rele_prevs_cnt = len(rele_prevs)
    if rele_prevs_cnt == 0:##First item
       res = cur
    else:
       x0 = rele_prevs[rele_prevs_cnt-1]
       prev_items_sum = sum([x * ((1-a) ** (i+1)) for i, x in
       enumerate(rele_prevs)])
       st = a * (cur + prev_items_sum)
       res =  st + ((1-a) ** (rele_prevs_cnt+1)) * x0
    return res

You should use it as follows:
grpd = data.groupby('group')
data['EMA3'] = list(map(calc_ema, data['price'], *[grpd['price'].shift(i) for i in range(1, n + 1)]))

Original Answer:
-- Dosnt answer OP's needs --
You can use the shift function on a grouped by object.
This will let you calculate on $Y_t$ and $Y_{t-1}$
data.sort_values(by='bin', inplace=True)
a = 0.1
data.loc[:, 'ema'] = map(lambda x, y: x if pd.isnull(y) else x*a + (1-a) * y,
data['price'], data.groupby('group')['price'].shift(1))

##python 3
 data.loc[:, 'ema'] = list(map(lambda x, y: x if pd.isnull(y) else x*a + (1-a) * y,
data['price'], data.groupby('group')['price'].shift(1)))


Generate new sentences based on keywords For example, for a domain specific neural network in Fashion, with the Keywords light, dress, orange, cotton. It could output: This gorgeous orange summer dress is great for wearing on sunny camping days. It's cotton fabric makes it very comfortable to wear.

Can someone please suggest the simplest way to achieve this?

This is area of NLG . You can use template based text generation techniques, wherein you have defined structure of output text and fill in required blank areas based on keywords. This technique is used in reports generation. An example is narrative science company.

Other approach can be to use OpenAI GPT . Example is Generate Text using OpenAIGPT2 in Python
 . You may have to tweak the code as per your requirement.

Paraphrasing can be another technique. An example of paraphrasing is - https://github.com/vsuthichai/paraphraser


Low accuracy using Multi Layer perceptron network on CIFAR10 dataset? I  trained Muti layer perceptrons using keras to classify cifar-10 dataset the results I got shows that there is something wrong in the code ,because all the epoch are identical here is the code :
In this I flatten the data from 32x32x3 to 3072 array.

   #nn layers
import numpy
from keras.datasets import cifar10
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout
from matplotlib import pyplot
from scipy.misc import toimage

#load data
(Xt,Yt),(Xts,Yts)=cifar10.load_data()

num_pix=Xt.shape[1]*Xt.shape[2]*Xt.shape[3]
X_tr=Xt.reshape(-1,3072).astype('float32')
X_ts=Xts.reshape(-1,3072).astype('float32')
#feature scaling
X_tr= X_tr / 255
X_ts = X_ts / 255
#one hot encoding
from keras.utils import np_utils
Ytr=Yt.reshape(-1,50000)
Yt=np_utils.to_categorical(Ytr)
Yts=Yts.reshape(-1,10000)
Yts=np_utils.to_categorical(Yts)
num_class=Yts.shape[1]

def base_model():
    model=Sequential()
    #build layers Dense(10, init="normal", activation="relu"
    model.add(Dense(num_pix,input_dim=num_pix,activation='relu'))#input layer
    model.add(Dense(num_class,activation='softmax'))#output layer
    #compile model
    model.compile(loss='squared_hinge',optimizer='adam',metrics=['accuracy'])
    return model
    #build the model
model=base_model()
#Fit model
model.fit(X_tr,Yt,validation_data=(X_ts,Yts),epochs=10,batch_size=200,verbose=2)
#Final assesment
score=model.evaluate(X_ts,Yts,verbose=0)
print("Baseline Error: %.2f%%" % (100-score[1]*100))

  Results :

 Train on 50000 samples, validate on 10000 samples
    Epoch 1/10
    134s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000
    Epoch 2/10
    97s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000
    Epoch 3/10
    99s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000
    Epoch 4/10
    96s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000
    Epoch 5/10
    96s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000
    Epoch 6/10
    96s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000
    Epoch 7/10
    96s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000
    Epoch 8/10
    95s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000
    Epoch 9/10
    95s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000
    Epoch 10/10
    95s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000
    Baseline Error: 90.00%

Any help will be appreciated.can my model be overfitted?

CIFAR10 has 10 class label. So by random guessing, you should achieve an accuracy of 10%. And this is what you are getting. This means your algorithm is not learning at all. The most common problem causes this is your learning rate. Reduce your learning rate by replacing your line,

model.fit(X_tr,Yt,validation_data=(X_ts,Yts),epochs=10,batch_size=200,verbose=2)

with

optimizer = keras.optimizers.Adam(lr=1e-4)
model.compile(loss='squared_hinge',optimizer=optimizer,metrics=['accuracy'])

I just did a test run, it is learning slowly.


How to train and evaluate machine learning models with growing/changing datasets over time Assume that you have a classification machine learning model, and you start with an initial dataset that contains 3 classes. You split the initial dataset into training/testing spits, you train the initial model and evaluate it, and then overtime, you collect more data for your dataset. Now you have more data that you want to add to you initial training dataset. The question is: How do you you organize you dataset and model training regiment so you can effectively quantify possible improvement between the initial model and the new model?
One possible solutions: if you split the initial training dataset into training/testing data, once you have more data, you just split that data and then add the respective new training/testing splits into the original splits. But this approach feels a little too simplistic, and it could potentially lead to drift in the dataset itself.
This is similar to doing regression testing but for machine learning models, but I am not sure im using the correct technical jargon for it.

The solution you described is definitely an option.
The other motivation for retraining is that the distribution of new data has drifted away from that of the original dataset and that warrants retraining to maintain the same level of model performance.
In such situations, It is possible that you may actually throw away some of your older training data because they may have become stale and include a part of newly available data for training. The remainder new data can then be used for out of sample testing and validation.
You also want to be able to figure out when to trigger the retraining. This is usually done by an independent drift detection process, which keeps track of some metric related to your Business and triggers retraining if it dips below a predefined threshold.


How to compute f1_score for multiclass multilabel classification I have used one hot encoder [1,0,0][0,1,0][0,0,1] for my functional classification model.
The predicted probabilities for test data yprob = model.predict(testX) gives me :
yprob = array([[0.18120882, 0.5803128 , 0.22847839],
       [0.0101245 , 0.12861261, 0.9612609 ],
       [0.16332535, 0.4925239 , 0.35415074],
       ...,
       [0.9931931 , 0.09328955, 0.01351734],
       [0.48841736, 0.25034943, 0.16123319],
       [0.3807928, 0.42698202, 0.27493873]], dtype=float32)

I would like to compute the Accuracy, F1 score and the confusion matrix from this.
The sequential api offers a predict_classes function to do it.
yclasses = model.predict_classes(testX) and using the f1_score function of sklearn we could compute all those values.
How could I apply it to predict probabilities of test data for multiclass multilabel classification ?
My second question is to know if the highest value of each array of yprob = model.predict(testX) corresponds to the predicted class ? for example, [0.18120882, 0.5803128 , 0.22847839] is the first element in the array. The highest value is  0.5803128. Does it mean that it corresponds to the one hot encoder [0, 1, 0], so the second label because it is the second element in the array ?

There seems to be a confusion between multiclass and multilabel classification:

Multiclass is the regular case where the task consists in predicting among N possible classes. For example an image can be either a dog or a horse or a cat, but always exactly one among these three animals.
Multilabel is the when the task consists in predicting a set. For example an image can be any subset of {dog, horse, cat}: it could be {dog, cat}, it could be {horse}, it could be {dog, horse, cat}, it could even be the empty set (no animal at all).

Practically in the multilabel case you predict each possible animal independently as a binary problem, so for every image the system answers 3 questions:

does this image contain a dog? (y/n)
does this image contain a horse? (y/n)
does this image contain a cat? (y/n)

Because each question is predicted independently, it doesn't make sense to pick the class which has the max probability. In effect there are 3 independent binary classification problems and 3 corresponding confusion matrices.
Apparently you didn't mean to take into account the multilabel case and you don't have any image labelled with multiple animals, right? If so you should change the system to solve a regular multiclass problem.
It looks like the confusion might be due to the one-hot-encoding of the class: maybe you thought that the class being categorical, it would be a mistake to encode the class as a numerical value. This is true for categorical features but actually not for the target, you can perfectly use for instance LabelEncoder to represent the class as a single target. This is much simpler and probably more appropriate for your problem.
One difference you will notice is that the probabilities obtained as predictions sum to one, because the classifier doesn't consider the classes as independent (as opposed to what happens in your current experiment: the probs don't sum to 1).


How to merge personalized models together Let's say I'm building an app like Uber and I want to predict the user's most likely destination based on the user's past history, current latitude-longitude, and current date and time.

We have millions of users, but each user needs will be probably too unique for generalization. I.e., everyone's commute is so different that what you have learned from other users are probably not applicable to other users.

In conclusion, I have to build millions of models, how can I merge these models together for a better deployment process?
If not, what are the best practices to deploy millions of models?

Feature engineering is the name of the game when it comes to this cases. I stumbled upon a similar problem a few years ago and it can be baffling to have a model nor generalize well for all cases. However, one model for one user is never the way to go, after all you, in many cases have only one data point for that user in particular. Additionally, you will never be able to make predictions if a new user appears.

Therefore, you need to use one model for as many as possible users. There are many more approaches but these are the ones worth considering  in my opinion.

You can cluster users by location, city, neighborhood, region. Usually people in the same areas frequent the same places. If one model is incapable of generalization for all possible users then splitting the data into similar clusters is an approach.
In the case a model can generalize well for most cases but there is a big cluster when it cannot generalize, use a second model for the hard cases.
Basic feature engineering: users living nearby are going to go to similar places based on their current location and are going to use the same ways. There is a lot of information similar users will share. Use starting location as a feature as well.
Advanced feature engineering: Use old targets from the same users along with time features in a meaningful way as training features to predict future targets.
Use users ids: user ids can be used to predict targets in modern machine learning algorithms, sparse matrices do the trick, think of them as NLP words. Some might be useful others don't but let the algorithm do its work. Search for algorithms that support sparse matrices if you take this approach.
Research GIS (Geographic Information System) forecast: there are other tricks and feature engineering methods in that field that are not applicable for other ML problems which are useful for forecasting these kind of problems.
Try different targets. Longitude - -latitude is often not possible but a raw estimate of the region can be done. You can treat the targets as a multilabel preferably (not multiclass) classification with each region/neighborhood/block be one label. Remember, algorithms can handle many targets like in the case of image classification, research those approaches.
Consider simpler methods if you don't have enough data: If user Bob goes to spot A every Monday then probably Bob will be at A the next Monday as well.
Use dates/days of the week/holidays as features. People change transit behaviors during holidays.

tl.dr. Find a way to use many users for a model that generalizes well because training one model for one person never works due to users having often just one point of data. Because, all algorithms need massive amounts of data to generalize well.


Image Segmentation with a challenging background I'm working on an animal classification problem, with the data extracted from a video feed. The recording was made in a pen, so the problem is quite challenging with a dark background and many shadows:

Initially I tried scikit-image, but then someone helped me with an advanced tool called crf-rnn (http://crfasrnn.torr.vision/) that does a great job segmenting and labelling objects in an image. I did the following:

import caffe
net = caffe.Segmenter(MODEL_FILE, PRETRAINED)
IMAGE_FILE = '0045_crop2.png'
input_image = caffe.io.load_image(IMAGE_FILE)
from PIL import Image as PILImage
image = PILImage.fromarray(np.uint8(input_image))
image = np.array(image)
mean_vec = [np.mean(image[:,:,vals]) for vals in range(image.shape[2])]
im = image[:, :, ::-1]
im = im - reshaped_mean_vec
cur_h, cur_w, cur_c = im.shape
pad_h = 750 - cur_h
pad_w = 750 - cur_w
print(pad_h, pad_w, "999")
im = np.pad(im, pad_width=((0, max(pad_h,0)), (0, max(pad_w,0)), (0, 0)), mode = 'constant', constant_values = 255)
segmentation = net.predict([im])
segmentation2 = segmentation[0:cur_h, 0:cur_w]

The resulting image segmentation is rather poor (although two cows are recognized correctly):

I use a trained crf-rnn (MODEL_FILE, PRETRAINED), which works well for other problems, but this one is harder. I would appreciate any suggestions on how to pre-process this sort of image to extract the shape of most cows.

It would be appreciated if you could explain precisely what your goal is:

you want to identify what animal is in your picture ?
you want to count the number of animals ?
you want to get the position of each animal in the picture ?

In any case, I know that you can get some already trained neural nets from google or anywhere else. This neural net can be used with caffe as it is the case in this google deepdream stuff on github (look at ):

https://github.com/google/deepdream/blob/master/dream.ipynb

Then, if you want to highlight or identify the positions of your animals, you'll find this article inspiring:

http://www.matthewzeiler.com/pubs/cvpr2010/cvpr2010.pdf

It explain how to reverse convolutional networks to identify what part of your image helped to recognize what is inside. The found projection get you something similar to your second picture (called a mask), but depending on the neural net you use, you can get better results.


Can I use a GAN to increase my Dataset used for Image detection? I am currently working on a machine learning project where I use the YOLO Algorithm to detect an object inside of a picture or video. The problem I face is that the specific image set (side-scan sonar) that I am working with is mostly classified, thus there is not a wide range of images available to the public to be used for training. Would I be able to implement a GAN to produce a larger data-set of side-scan sonar imagines to be used for training an image detection Algorithm? I understand that there may be image distortion from using a GAN, but for the purpose of having a larger training set would this be possible?

In short: no

If you don't have enough images now then you almost certainly dont have enough to successfully train a GAN (it takes more than you think). If you can't train a GAN to reproduce good images, then you don't have images that are going to give you high accuracy for your YOLO effort. The most likely scenario is that you would end up with an algorithm that is really, really good at detecting GAN images that looking nothing like your side scan sonar validation data :)


What interactive graph package do you use for timeseries data? I have millions of sensor readings taken over time.  I'd like to create an interactive plot in Google colab where I can easily zoom in on sections.  I've done something similar with the Plotly package.  However, it fails when the amount of data gets over ~ 700,000.

Are there interactive plotting packages that work with Google colab and handle millions of data points?

Thank you.

I didn't get an answer so here's my take.  Ideally, I would use holoviews and the pyviz packages...to some extent i can, so i will continue to try this method.  However, many of the examples do not work.


Why does the parent connection matter during a NEAT crossover when both connections are identical? I've been reading through NEAT (NeuroEvolution of Augmenting Topologies)'s whitepaper, and one thing is particularly confusing:

Innovation numbers are supposedly used to track the origin of a mutation, so that crossing over is easily done. From what I understand, two identical connections should always have the same innovation number, because their numbers are indexed. However, according to this illustration:

when the two genomes being crossed over share connections with the same innovation number, the resulting innovation is taken from the more fit parent (or random if they are equal fitness).

If the two connections are identical (they have the same innovation number) then why does it matter which parent it comes from, since the connection is the same either way? Or is there something about them that is not the same and I am understanding this wrong?

The genes shown in the diagrams have been simplified so that they only show topology and innovation index number. In addition, the full genome contains weights for each connection and bias for each node.

The innovation numbers are not tracking changes with connection weights or biases, just the rough time of inserting a specific connection between two nodes. So each parent can have matching connections/innovations, but with different weights associated. The full genome is still subject to cross-over and mutation, the excerpt you quote helps explain how the genome is aligned between parents with different architectures, so that GA crossover and mutation can occur in each connection and node.

Which parent is used to create each offspring gene matters because of the additional connection weight data associated with each parent's gene.


logic behind weighted KNN I am reading about KNN

So I made another example to make things clearer

In this example (Image attached)

You can see there are in total 5 Greed Circles and 20 Blue Squares

by standard KNN (k=3) , X should be a Blue Square

This is obvious 2 Blue Squares vs 1 Green Circle.

But in weighted KNN things are difference

In this case we have to calculate the weight (Likelihood) for each instance

Each Green Circle likelihood is $\frac{1}{5} $ , we have 5 Green Circles

While for Blue Squares it is  $\frac{1}{20} $ , we have 20 Blue Squares

Therefore the weights around X will be $\frac{1}{5} $  Green Circle, and $\frac{2}{20} $ Blue Squares.

which means $\frac{1}{5} > \frac{2}{20} $

Then X is Green Circle

But if try to think of it logically then there are more Blue Squares than Green Circles which means X more likely to be Blue Square than Green Circle.

My question is :

Am I doing anything wrong here? Can someone explain why the equation is showing Green Circle while logic says Blue Square?

Logically speaking, I think "X is Green Circle" is a reasonable conclusion. I find the idea in the paper in your question is quite similar to this paper: KRNN: k Rare-class Nearest Neighbour Classification.

Intuitively, for example, if a new data point is close to one rare class' point and one common class' point, it is more likely to belong to the rare class.

There's no conflict here because a data point will be much more likely to be close to a common class' point. However, once it's already close to a rare class's data point, it's more likely to belong to this class.

That said, I didn't check your calculation, I just don't think the "X is Green Circle" conclusion is illogical for this algorithm.

[Update]

After reconsidering this problem, I think Weighted kNN want to put an emphasis on the rare class data point because that is the class of interest (e.g. Anomaly detection).

It's possible that Accuary is not the metrics here but a Weighted Accuracy metrics that penalize a misclassified rare class data point harder so that we can detect more rare class data point.


Interpreting vertical and horizontal parts of ROC curve It's not clear to me how I can interpret vertical and horizontal parts of the ROC curve. What important information can I gain from this? This is a text from the book "Human-in-the-Loop Machine Learning" by Robert Monarch:

In this example, we can see that the line of the ROC curve is
near-vertical for the first 20%. This tells us that for the 20% most
confident predictions, we are almost 100% accurate. The ROC curve is
almost horizontal at 1.0 for the final 30%. This tells us that by the
time we get to the 30% least most confident predictions for a label,
there are few very items with that label remaining.

The explanation follows from understanding what a ROC curve is made of.
First, a reminder: a ROC curve represents the performance of a soft binary classifier, i.e. a classifier which predicts a numerical value (usually a probability) which represents the likelihood of the instance to be positive.
The points of a ROC curves correspond to what happens when splitting the instances at different thresholds of their predicted value/probability. For example let's say we have 10 instances with the following predictions and status, ordered by decreasing predicted value:
predicted value   gold status
  1.0                  P
  0.95                 P
  0.95                 P
  0.85                 N
  0.70                 P
  0.55                 P
  0.52                 N
  0.47                 P
  0.26                 N
  0.14                 N

If the threshold is set at 0.60, we have 4 TP, 1 FP, 2 FN and 3 TN.
If the threshold is set at 0.75, we have 3 TP, 1 FP, 3 FN and 3 TN.
etc.

If the classifier does a decent job, the top part should contain only gold positive instances and the bottom part only negative instances, since the classifier predicts a very high (resp. very low) value for these instances. By contrast the middle part is more likely to contain errors.
This means that with a decent classifier, we encounter only positive instances until a certain threshold from the top: in my example, there are 3 P instances above threshold = 0.85, so any threshold chosen between 0.85 and 1 will have no FP instances. This implies FPR=0 with TPR ranging from 0 to 3/10, hence a vertical line on the ROC curve from the origin to (0,0.3).
Applying the same observation for the lowest values in my example, we see than when the threshold is between 0 and 0.47 (excluded) we have no FN, so we have a perfect TPR=1 and therefore an horizontal line from (1,1) to (0.8, 1).
These two extreme parts of the ranking are what the author calls "most confident predictions" and "least most confident predictions". Btw the latter is not a very good term because actually the bottom part is where the classifier is very confident about the instance being negative. If anything, the part where the classifier is the least confident is around the middle of the ranking.
[edit: forgot to answer this question]

What important information can I gain from this?

So these two straight lines show how far the classifier can go:

without making any false positive error (bottom left vertical line)
without making any false negative error (top right horizontal line)

It can be useful in applications where one wants to have perfect precision at the cost of recall, or the converse. However in general (optimizing F1-score for instance) the optimal threshold is somewhere in the middle of the curve.
Of course the longer the line, the better the classifier in general. Geometrically, it's clear that when these two lines are long then the area under the curve (AUC) is large. Note that in theory it's possible for a classifier to have a few errors close to the extremes (i.e. short lines) while still performing well in the middle range, but in practice it's quite rare.


How to make forecast in LSTM with specific/defined feature values I am analyzing the avocado dataset to predict the future prices of avocado depending on the region and type (organic/conventional).

I've trained my model  which seems to be working. The test results look good.

Here is what my data looks like after processing: the date (which I turned into the index column), total volume, type (label encoded to 1s and 0s), and 54 columns for the 54 regions that I one-hot encoded.

However, I'm not sure how to generate data for future prediction given specific X feature values. Basically I want the end user to provide a (future) date, a type, a region, and my model would predict the price of the avocado. For example, I want to predict the price of organic avocado for Albany in 3 months. I know that my model could forecast for a certain date, based on the prices of the past date. But how would it work when I assign the region and type feature to a specific value? Is it even feasible? Should I be using another model? I know that I could train a LSTM model for each of the 54 regions, but I'm sure there is a better way...

Sorry if this is a trivial question, I'm a beginner and I've been stuck on this for days and would really appreciate any help/guidance!

I'm not really sure if I understood you correctly, but let's talk about it.

You are trying to predict only one value, that is price, right?
And what is your feature set? is it only date, type and region?

Remember, to predict a value you should always have known feature set. If you have a feature that is not known, then this is not the right way to do it.

What you can also do is, predict multiple values. Not sure if this is helpful in your case, but you can use something like MultiOutputRegressor to predict multiple times based on a model. So you can have date, type and region as features and predict anything else that is missing. Be aware though that this would usually reduce the prediction accuracy.

See here: sklearn.MultiOutputRegressor

So if you want to predict for future dates, based on previous dates. You can either have feature sets of one month data, and always predict for the next one month. So you have a feature set of 30 days, and you predict 30 times, for each days in the future.

Or you can predict only for one day, and train with your predicted data, then predict the next day and so on. So one by one.

You should try both of these ways to see which one would give you a better results.

Hope these helps.


DQN with decaying epsilon I'm new to reinforcement learning. I'm studying DQN with decaying epsilon. I came across such example:

EPISODES = 91
GAMMA = 0.2
EPSILON_DECAY = 0.999
MIN_EPSILON = 0.01
MAX_EPSILON = 1

My questions are:

Is it correct if epsilon doesn't reach MIN_EPSILON?
Is there something wrong with the reward - the reward is not higher and higher but it is behaving otherwise - it decreases in time?

If you set epsilon decay to 0.999 you will need
$$
\epsilon_{max} \cdot \epsilon_{decay}^x = \epsilon_{min} \\
1 \cdot 0.999^x = 0.01 \\
x \approx  4603
$$
4603 episodes to reach minimum epsilon. After 91 episodes you will reach
$$
\epsilon_{current} = \epsilon_{max} \cdot \epsilon_{decay}^{episodes} = 1 \cdot 0.999^{91} \approx 0.913
$$
which is exactly what you can see in your plot. It's not a problem but remember that this model still makes over 91% moves randomly.
Average reward should not decrease over time. It can mean a few things for example error in dqn algorithm or too high learning rate in your model. The best way to debug is to start with as simple environment as possible and let your model learn to play it and only then increase the difficulty.


Get multiple output from Keras I have a regression problem which I have to predict 3 numerical values from a provided data. For example let's say I have a data set containing X1,X2,X3,X4,X5,X6...X100,Y1,Y2,Y3 columns. In this case I have to predict Y1,Y2,Y3 values. Is there are any way to construct the model to get all the outputs at the same time using Keras.

it's quite easy to make! One thing to note I'm using the Keras functional API.

from keras.layers import Input, Dense

Input_1= Input(shape=(shape, ))

x = Dense(100, activation='relu')(Input_1)
x = Dense(100, activation='relu')(x)
x = Dense(100, activation='relu')(x)

out1 = Dense(1,  activation='linear')(x)
out2 = Dense(1,  activation='linear')(x)
out3 = Dense(1,  activation='linear')(x)

model = Model(inputs=Input_1, outputs=[out1,out2,out3])
model.compile(optimizer = "rmsprop", loss = 'mse')


Is there a documentation where it is explained why scikit-learn does not provide p-values? Is there a documentation, paper etc. where it is explained why scikit-learn does not provide p-values/confidence levels (1, 2, 3, 4)? Note: I'm not asking about opinions, but about documentation. For example the R package lme4 does not provide p-values either, but has a FAQ point  about this. On the official scikit-learn website I found no explanation for this desicion.

There is a scoring function in sklearn package which is used in a feature selection procedure. This function first computes F-score and then converts it to p-value.


One scaler for all features or one scaler per feature? I have a time series with more than 30 features. For preprocessing with scikit learn do you usually use one scaler per feature or one scaler for all features that should be standardized/normalized?

Sklearn scaler works on feature/column (and thats why you want)

Imagine if it did not. Than you would shift your mean and std in weird-determined-by distribution-of-the-whole-set-kind of way.


What does it mean to 'gradient check a few dimensions for every separate parameter'? In lecture notes for cs231n while discussing checking analytical gradient with numerical gradient the paragraph says this:

Check only few dimensions. In practice the gradients can have sizes of million parameters. In these cases it is only practical to check some of the dimensions of the gradient and assume that the others are correct.
Be careful: One issue to be careful with is to make sure to gradient check a few dimensions for every separate parameter. In some applications, people combine the parameters into a single large parameter vector for convenience. In these cases, for example, the biases could only take up a tiny number of parameters from the whole vector, so it is important to not sample at random but to take this into account and check that all parameters receive the correct gradients.

I understand to check only a few dimensions but what does the part after be careful mean. I fail to understand that warning. can somebody explain it?

As you might know, biases are used in the CNN only when there is no batch normalization layer is included because this automatically adds a type of bias to the activation outputs.

So, this is what I understood:

Let us assume a layer of CNN (without batch norm) with 50 kernels each of size of 5 x 5. So, total weight updates will be 5 x 5 x 50 = 1250 and total biases updates will be 50 (a bias for each kernel).

For the part "...people combine the parameters into a single large parameter vector for convenience. In these cases, for example, the biases could only take up a tiny number of parameters from the whole vector..." :-

If you merge all these parameters into one single vector and randomly sample the parameters to check gradients for, you may end up checking very few or none of the gradient updates for biases as they make a very less portion of that vector (only 50 out of 1300).

For the part "One issue to be careful with is to make sure to gradient check a few dimensions for every separate parameter..." :-

On the other hand, you can choose some kernels and take all parameters (weights and biases) of only those kernels to check gradient for and so in this case you will definitely be checking gradients of some biases which they say is a good way of doing gradient checking.


How to get significance level for ranked features? I am aware of below approaches of feature selection

a) Feature Importance methods which are available in tree based models like Random Forest and Xgboost,GradientBoost etc.

b) statsmodel.logistic regression which in it's summary output provide us the results which contains whether variables are significant or not (P-value)

c)  SelectKbest which uses ANOVA, Chi-square etc to compute the influence of input variable on target attribute

But unfortunately with methods b and c, it doesn't consider the feature interaction. Am I right? It works by considering each column to the target variable

Whereas with methods a it returns the ranking but we aren't sure about whether they are significant or not.

Is there anyway to know from Feature Importance whether the Features are significant or not? I understand features occurring in top 4-5 places could be significant but is there anyway to test/validate this?

Or is it like I pick each feature (out of say top 20 assuming they have a role) from feature importance result and do a SelectKbest test or statsmodel summary?

How can I know that the features that I select from Feature importance model are significant?

1. univariate feature importance (that is c) in your list)

You are correct. Univariate statistics to estimate feature importance does not capture feature interactions. But they are fast and simple.

2. model-based feature importance (that are a) and b) in your list)

On the other hand model-based feature importance estimates can capture interactions as long as the model is capable of doing so (see "Introduction to Machine Learning with Python"; Mueller, Guido; 2017; p. 238/239). Which is not the case for linear regression.

For model-based feature importance estimates using trees there are ways to derive p-values. And at least R does have some implementations for that. Have a look at section "2.5 Importance testing procedures" in this paper The revival of the Gini importance?.


Best way to store financial data together with news My question is if there is a DBMS that allows storing financial data (i.e. timestamp+value) together with small text fragments.

These small fragments will not be very long, but it might help an algorithm to predict changes in the future. In summary, I wonder if you can suggest me what it is the best way to massively store timestamps+numeric value+(optional)text fragment.

Thank you in advance.

The short answer is that (almost) any DBMS will be able to store that information for you.

I use Microsoft SQL and you can store the record with a specified or system calculated date-time, a column for the numeric value and you can store the story text in one of four ways:

Plain text as a NVARCHAR
structured text as XML (or JSON if you are on SQL2016 or above)
Filestream if the data format is binary (normally an image, audio or video)
As a reference to a file on disk

MSSQL is an RDBMS (relational database management system) Others in this classification are Oracle, MySql, Postgress.  There is another school of database classified as NOSQL or Document Databases and these include Mongo, CouchDB, DocumentDB and RavenDB.  These differ from RDBMS in the way they store data and manage the relationships (e.g. find all the articles relating to company X in the time period 1st April to 6th June)

I would suggest that if you don't know how you are going to want to process the data in the future, look at storing it in a NOSQL database as it provides much greater flexibility of record structure.  Most can be installed on-premise with a free / open source licence and most are also available on a hosted platform for a relatively small sum.  Many of the RDBMS (esp Oracle and MSSQL) are exceeding expensive for this sort of project.  That said, Azure SQL - the hosted version of SQL can be had for as little as $8 per month depending on your storage requirements.


What would be a good way to use clustering for outlier detection? For simplicity let's assume the feature space is the XY plane.

A very robust clustering algorithm against outliers is PFCM from Bezdek.
In this paper Bezdek proposes Possibilistic-Fuzzy-C-Means which is an improvement of the different variations of fuzzy posibilistic clustering. This algorithm is particularly good at detecting outliers and avoiding them to influence the clusterization. So using PFCM you could find which points are identified as outliers and at the same time have a very robust fuzzy clustering of your data.


Using of 100s of Binary features in regression model I have 100s of columns with binary values [0, 1] plus some extra columns without binary values. I am trying to do regression model but the model performance is very low. For non-binary features, I have used PCA to decrease the dimension of it. I don't think its appropriate to use PCA for binary values. I am guessing, its because of the large number of binary columns, the model isnt doing great. What can be done on this kind of situation ? I have tested with almost all of the regression model available in sklearn.

What approach I could take to improve the model performance ? Any suggestions.

An approach worth exploring for dimensional reduction of these binary features would be Multiple Correspondence Analysis (Python libraries such as Prince and mca are available), which is similar to PCA but deals with nominal categorical variables. Here is an excellent answer on applicability of PCA and factor analysis to binary features and another one on the same topic.


Boundaries of Reinforcement Learning I finally developed a Game Bot that learns how to play the videogame Snake with Deep Q-Learning. I tried with different neural networks and hyper-parameters, and I found a working set-up, for a specific set of rewards.  The problem is is: when I reward the agent for going in the right direction - positive rewards in case the coordinates of the agent increase or decrease accordingly to the coordinates of the food - the agent learns pretty fast, obtaining really high scores. When I don't reward the agent for that, but only negative rewards for dying and positive for eating the food, the agent does not learn. The state takes into account if there's any danger in proximity, if the food is up, down, right or left and if the agent is moving up, down, right or left.
Here's the question: is rewarding the agent for going into the right direction a "correct approach" in Reinforcement Learning? Or it's seen as cheating, cause the system needs to learn that by itself? Is passing the coordinates of the food as state an other way of "cheating"?

Here's the question: is rewarding the agent for going into the right direction a "correct approach" in Reinforcement Learning?

It depends on what you are hoping the agent is capable of learning by itself. This is an issue for you here, because you have a "toy" problem where you can control a lot more of the environment and alter the meaning of what it means to win.

In general, then yes this is "cheating", at least in terms of claiming to have written an RL agent that solves the game. The academically ideal basic RL agent is rewarded by the gain of something meaningful in the context of the problem being solved, and is not helped by interim rewards. In a game of snake, and any other arcade-style game, it should really be the official points scored in the game and nothing else.

  Is passing the coordinates of the food as state an other way of "cheating"?

Again it depends on what you expect the agent to learn from. If, in your target production environment, this data was easy to obtain, and you intended to use it to write a game bot working from the trained policy, then this is fine. There is no requirement that you do one thing or another if you have a practical problem to solve.

However, learning from a pixel-only state, as in the DQN original papers, is of academic interest, because that is a generic state representation that applies to many problems, whilst the distance from the snake to food is a specific feature that you have engineered that makes learning easier in a smaller set of games.

The main issue here is again that your goal is not really to put a "snake bot" into a production system, but to learn how RL works. RL is tricky, and often doesn't work as well as you expect - or at all, for many combinations of algorithm and problem.

It is worth reading this article: Deep Reinforcement Learning Doesn't Work Yet - it may put disappointing results from basic DQN into perspective.

I would encourage you to strip back your Snake problem to remove "helpful" rewards and state, and instead look into extensions to the core DQN algorithm, or different learning agents such as A3C.


Applying dimensionality reduction on OneHotEncoded array I have a really large data set with mixed variables. I have converted categorical variables to numerical using OneHotEncoding and it has resulted in more than a couple of thousand different features, combined that is.

Is it possible to apply dimensionality reduction algorithms on OneHotEncoded data which looks like [[1. 0. 1. 0.]...[0. 0. 0. 0.]] or should it be done by merging with the original data set?

Following your example, you have different points in a 4-dimensional space. So, yes! you can use any dimensionality reduction technique, from PCA to UMAP.

In general, if your data is in a numeric format (and one-hot actually is), all the elements have the same dimensionality, and you don't have undefined values (NAN, inf), you can always use dimensionality reduction.


Machine learning - features engineering from date/time data What are the common/best practices to handle time data for machine learning application?

For example, if in data set there is a column with timestamp of event, such as "2014-05-05", how you can extract useful features from this column if any?

Thanks in advance!

I would start by graphing the time variable vs other variables and looking for trends.

For example

In this case there is a periodic weekly trend and a long term upwards trend.  So you would want to encode two time variables:

day_of_week
absolute_time

In general

There are several common time frames that trends occur over:

absolute_time
day_of_year
day_of_week
month_of_year
hour_of_day
minute_of_hour

Look for trends in all of these.

Weird trends

Look for weird trends too.  For example you may see rare but persistent time based trends:

is_easter
is_superbowl
is_national_emergency
etc.

These often require that you cross reference your data against some external source that maps events to time.

Why graph?

There are two reasons that I think graphing is so important.

Weird trends
While the general trends can be automated pretty easily (just add them
every time), weird trends will often require a human eye and knowledge
of the world to find.  This is one reason that graphing is so
important.
Data errors
All too often data has serious errors in it.  For example, you may find that the dates were encoded in two formats and only one of them has been correctly loaded into your program.  There are a myriad of such problems and they are surprisingly common.  This is the other reason I think graphing is important, not just for time series, but for any data.


How to fix class imbalance in training sample? I was very recently asked in a job interview about solutions to fix an imbalance of classes in the training dataset. Let's focus on a binary classification case.

I offered two solutions: oversampling the minority class by feeding the classifier balanced batches of data, or partitioning the abundant class such as to train many classifiers with a balanced training set, a unique subset of the abundant and the same set of the minority. The interviewers noded, but I was later cut off and one of the knowledge gaps they mentioned was this answer. I know now that I could have discussed changed the metric..

But the question that  pops in my mind now is: is it really a problem to train a classifier with 80% class A if the testing set will have the same proportion? The rule of thumb of machine learning seems to be that the training set needs to be as similar as possible to the testing for best prediction performance.

Isn't just in the cases with have no idea (no prior) about the distribution of the testing that we need to balance the classes? Maybe I should have raised this point in the interview..

Actually what they mentioned is right. The idea of oversampling is right and is one of, in general, Resampling methods to cope with such problem. Resampling can be done through oversampling the minorities or undersampling the majorities. You may have a look at SMOTE algorithm as a well-stablished method of resampling.
But about your main question: No it's not only about the consistency of distributions between test and train set. It is a bit more.
As you mentioned about metrics, Just imagine accuracy score. If I have a binary classification problem with 2 classes, one 90% of the population and the other class 10%, then with no need of Machine Learning I can say my prediction is always the majority class and I have 90% accuracy! So it just does not work regardless of the consistency between train-test distributions. In such cases you may pay more attention to Precision and Recall. Usually you would like to have a classifier which minimizes the mean (usually harmonic mean) of Precision and Recall i.e. the error rate is where FP and FN are fairly small and close to each other.
Harmonic mean is used instead of arithmetic mean because it supports the condition that those errors are as equal as possible. For instance if Precision is $1$ and Recall is $0$ the arithmetic mean is $0.5$ which is not illustrating the reality inside the results. But harmonic mean is $0$ which says however one of the metrics is good the other one is super bad so in general the result is not good.
But there are situations in practice in which you DO NOT want to keep the errors equal. Why? See the example bellow:
An Additional Point
This is not exactly about your question but may help understanding.

In practice you may sacrifice an error to optimize the other one. For instance diagnosis of HIV might be a case (I am just mocking-up an example). It is highly imbalanced classification as, of course, the number of people with no HIV is dramatically higher than the ones who are carrier. Now let's look at errors:

False Positive: Person does not have HIV but test says they have.
False Negative: Person does have HIV but test says they don't.

If we assume that wrongly telling someone that he got HIV simply leads to another test, we may take much care about not wrongly say someone he is not a carrier as it may result in propagating the virus. Here your algorithm should be sensitive on False Negative and punishes it much more than False Positive i.e. according to the figure above, you may end up with higher rate of False Positive.
Same happens when you want to automatically recognize people faces with a camera to let them enter in an ultra-security site. You don't mind if the door is not opened once for someone who has permission (False Negative) but I'm sure you don't want to let a stranger in! (False Positive)
Hope it helped.


Gumbel-Softmax trick vs Softmax with temperature From what I understand, the Gumbel-Softmax trick is a technique that enables us to sample discrete random variables, in a way that is differentiable (and therefore suited for end-to-end deep learning).

Many papers and articles describe it as a way of selecting instances in the input (i.e. 'pointers') without using the non-differentiable argmax-function. The thing that confuses me is that this effect can be achieved without randomness by just using Softmax with temperature:

Softmax with temperature
$$y_=\frac{exp(\frac{_}{\tau})}{\sum_{}exp(\frac{_}{\tau})}$$

Gumbel-Softmax
$$y_=\frac{exp(\frac{log(\pi_)+g_i}{\tau})}{\sum_{}exp(\frac{log(\pi_j)+g_j}{\tau})}$$

My question

From a practical and theoretical perspective, when is it beneficial to incorporate Gumbel noise into a neural network, as opposed to just using Softmax with temperature?

A couple of observations:

When the temperature is low, both Softmax with temperature and the Gumbel-Softmax functions will approximate a one-hot vector. However, before convergence, the Gumbel-Softmax may more suddenly 'change' its decision because of the noise.
When the temperature is higher, the Gumbel noise will get a larger significance and the distribution will become more uniform. Why is this desired?

My best guess is that the introduction of the Gumbel noise enforces stronger exploration before convergence, but I can't recall reading any papers that use this as a motivation to bring in the extra randomness.

Does anyone have any experience or insights on this? Maybe I've completely missed the key point of Gumbel-Softmax :)

Let's say you have two states, $X_1$ and $X_2$, and you have a model, $M$, that produces a score $M(X_i)$ for each state (i.e, the logits). Next you can use the logits to compute some distribution

$$P = softmax(\{M(X_1), M(X_2)\})$$

and take the state with the highest probability

$$X=argmax_{X_i}(P)$$

But what if you actually want to sample from $P$ instead of just taking the argmax - and you want the sample operation to be differentiable! This is where the Gumbel Trick comes in - instead of softmax, you compute

\begin{equation}
\ X = argmax_{X_i}(\{M(X_i)+Z_i\})
\end{equation}

Where $Zi$ are i.i.d Gumbel(0,1). It turns out that $X$ will be equal to $X_1$ about $P(X_1)$ of the times and to $X_2$ about $P(X_2)$ of the times. In other words, the equation above samples from $P$.

But it still not differentiable because of the argmax operation. So instead of doing that we'll compute the Gumbel-Softmax distribution. Now if the temperature is low enough then the Gumbel-Softmax will produce something very close to one hot vector, where the probability of the predicted label will be 1 and other labels will have a probability of zero. So for example, if the Gumbel-Softmax gave the highest probability to $X_1$, you can do:

\begin{equation}
\ X = \sum_{x_i} P_g(X_i)*X_i = 1*X_1 + 0*X_2 = X_1
\end{equation}

Where $P_g$ is the Gumbel-Softmax operation. No argmax is needed! So with this cool trick we can sample from a discrete distribution in a differentiable way.


Can I use categorical and numerical data variable at the same time in randomForest in r? I have data in which few columns contain categorical data whereas remaining columns contain numerical data. I want to use random forest regressor from the randomForest library in r. So does this library work with such data out of the box (assuming I have preprocessed the data and is ready to directly feed in the regressor)?

DATA (something along this line) :

Target | product | status | revenue | MS
  0.8      abc       NC      1000    0.5
  0.5      abc       UR      200     0.2
  0.2      pqr       NC      800     0.04

In general you can always transform categorical data into numeric data. Many experienced STATA users need to do routinely. If you have gender variable coded as "male/female" you need one binary gendervvariable recorded as 0/1. If you have four levels variable such as main role in a football team "goalkeeper/defense/midfield/attack"  you could take midfield as reference and generate the goalkeeper, midfield and attack variables all coded as 0/1.
In caret, the dummyVars function allow to create easily dummy variables.
Here an example with cylinders transformed as factors and then dummyfied:

 library(caret)
 mtcars$cyl <- as.factor(mtcars$cyl)
 dmy<- dummyVars(" ~ .", data = mtcars, fullRank=T)
 dummy.mtcars <- data.frame(predict(dmy, newdata = mtcars))
 head(dummy.mtcars)


Using Spark for finding similar users to a user? I read about
https://mahout.apache.org/users/algorithms/intro-cooccurrence-spark.html

but couldn't find a spark library for this implementation.

I have columnar string dataset.

I have a dataset with around data of 15-20 million users with their show_watched, times_watched, genre, channel and some more columns, I need to calculate lookalike/s for a user(or 100k users).

How do I find lookalikes for them within less time,

I have tried by indexing data in Solr, and then using Solr MLT for finding similar users, but that takes a lot of time, also it uses TF-IDF for MLT and I need users which have times_show_watched close to that user's times_show_watched.

Can anyone recommend a better approach for this, maybe using any other framework for faster processing?

I also tried to implement clustering using Spark MLLIB and later search in which cluster a user belongs so that search space is less, but I couldn't get this approach finished.

I am open to any approaches which would be efficient.

Thanks!

PMC from Mahout here- we're in the middle of a site re-org at the moment, and things are... well they're a mess.

Here's a link to something I think is more useful. A tutorial on Co-Occurance in Spark.

http://mahout.apache.org/docs/latest/tutorials/cco-lastfm/

Re "A Spark Library" well, mahout IS the spark library.

To use Mahout (Scala only, sorry if you're a Python-phile, however the syntax, especially for Mahout is very pleasant), you either need to download mahout and run ./mahout spark-shell from the bin/ directory. Or if you like GUIs notebooks and Apache Zeppelin, check out this tutorial for setting up Mahout+Spark on Zeppelin

http://mahout.apache.org/docs/latest/tutorials/misc/mahout-in-zeppelin/

(If you are compiling a Jar, you just add Mahout as a dependency.)


Why are ROC curves better for imbalanced datasets? I have recently read this:

" AUC(Area Under Curve) is good for classification problems with a class imbalance. Suppose the task is to detect dementia from speech, and 99% of people don’t have dementia and only 1% do. Then you can submit a classifier that always outputs “no dementia”, and that would achieve 99% accuracy. It would seem like your 99% accurate classifier is pretty good, when in fact it is completely useless. Using AUC scoring, your classifier would score 0.5. "

Can someone please explain why does it reach 0.5? If 99% are negative and we output always "no", wouldn't that mean that the TruePositiveRate will be very high and the FalsePositiveRate very low, resulting in an Area Under Curve close to one?

The AUC reaches 0.5 because of this:

$Sensitivity = \frac{True Positives}{True Positives + False Negatives}$

$Specificity = \frac{True Negatives}{True Negatives + False Negatives}$

In your case:

$Sensitivity = \frac{99%}{99%+0%}=1$

$Specificity = \frac{0%}{0%+1%}=0$

$1-Specificity = 1$

Remember that the X axis is 1-Specificity.

The point $(1,1)$ is in the diagonal line that makes the AUC equal to 0.5

If you try this exercise with any other rule like "Saying 'yes' to dementia for all patients" you will get the point $(0,0)$ and joining these points you will have the diagonal line.


DecisionTree split quality using chi-squared statistic In most of the popular packages (rpart etc.), the decision tree split is based on information gain, entropy or Gini index.

Is there a popular decision tree package ( either 'R' or python) where the node split quality can be specified using chi-squared statistic?

Base on another post about significance test for stopping decision tree in stats stackexchange website. CHAID, QUEST, GUIDE are three options. The QUEST was provide in binary format, but not available now.

The usage for an implementation of CHAID in R is expained here.
To install it on R

install.packages("partykit")
install.packages("CHAID", repos="http://R-Forge.R-project.org")


Logistic regression on biased data I am currently working on a dataset to predict customer attrition based on past data and transactions of the customers.

There are 2,40,000 customers in total out of which around 1,77,000 customers are active(as of today) while the remaining ones are inactive (6300).
This is how sample headers look like :

custID|custAge|custGender|TQuantity|TVolume|TValue|TAmount|HolidayStatus|...

Overall, I have 40 predictors which include customer details, transaction details, item details etc.

The data obviously has more active customers than inactive customers i.e. inactive customers form only 2.6% of the entire customer base. Due to this, there are more transactions conducted by active customers(25million/32million) than by inactive (previously active) (6million/32million) ones.

Despite this, I created a logistic regression model using random data (shuf -n 500000 data.csv). The model achieves 96.69% base accuracy in predicting when fed with random data.

The problem: How to make the model predict with greater accuracy on such a biased dataset? or How do I sample the data more appropriately?

Model prediction: With 99.7% probablity, it predicts that the customer will be active whereas the customer is inactive

PS: Changing threshold won't help much

Background

I'll start with some background to help you research the solution yourself and then will add some specifics.  What you refer to as "biased data" is more commonly known as unbalanced classes in the data science world.  Also "customer turnover" is often referred to as churn.

Metrics

As hoards of Ng'ian devotees will undoubtably point out you need to start by designing a set of metrics that work better with unbalanced classes than accuracy. Accuracy does a poor job in testing the quality of predictions for unbalanced classes e.g. a cancer test for a cancer that occurs in 0.05 % of the population is 99.95% accurate if it always predicts "no cancer". I suggest using the F1-score as the key metric in cross-validating your model.  The F1-score is the harmonic mean of precision and recall and tends to work both for balanced and unbalanced classes.  There are other rations of harmonic mean, that could work in special cases, so be aware of these.

There are other metrics you should learn about also. ROC-AUC is likely at the top of the list for other metrics you should understand and know about.

Model Selection and Cross-Validation

Beginning a classification task with Logistic Regression is a fantastic strategy. I make a point to always use a linear regression for regression tasks and a logistic regression for classification tasks. The linear model provides significant insight into the feature importance and helps frame the problem.

But following this initial survey you should move on to other, more sophisticated models. Many will give you a litany of things to try. You should perhaps focus on one or two and develop the model while paying very careful attention to bias and variance as you cross-validate and test your model. A full bias-variance decomposition may be unnecessary, once you develop better intuition, but is a great place for newbs to start.

I suggest starting with an SVM and also eventually trying a random forest or naive Bayes model as this will traverse several regimes of model types (analogy, decision trees, bagging, Bayesian).

Finally... Unbalanced Classes

There are two typical methods for dealing with unbalanced classes. These include oversampling the minority class, and fixing the model by altering the hyperplane (SVM) or changing priors (Bayes).

There are lots of summaries of this problem and solution if you search for "unbalanced classes". But, it can still be a tricky problem despite all the literature.  Good luck...

Hope this helps!


Syntax of the inputs for LightGBM binary build (without python API) I built the LightGBM binaries from https://github.com/microsoft/LightGBM. The build was successful, but the python API did not work for me. I am using the light gbm binary directly to run for time being.

I tested the binaries on LightGBM/examples/regression. The train and predict work fine using

lightgbm config=train.conf
lightgbm config=predict.conf

The outputs are _model.txt and _predict_result.txt

The problem I am facing is I cannot make out which in the input file regression.train is the target variable.

Also, I am looking for some documentation that explains how to run LightGBM without python API.

Thanks for the help.

The first column in the input file is considered to be the target variable. I found this by trial and error.

EDIT

There is a parameter called label_column which can take an integer value from 0 to N, to specify which column is the target variable. (source: https://lightgbm.readthedocs.io/en/latest/Parameters.html)


Difference between sklearn make_pipeline and imblearn make_pipeline Can anybody please explain the difference between sklearn.pipeline.make_pipline and imblearn.pipeline.make_pipline.

The imblearn package contains a lot of different samplers for easy over- or under-sampling of data.
These samplers can not be placed in a standard sklearn pipeline.

To allow for using a pipeline with these samplers, the imblearn package also implements an extended pipeline. This pipeline is very similar to the sklearn one with the addition of allowing samplers.

If you want to include samplers in the pipeline, use the imblearn pipeline. Otherwise, use the sklearn one.

The code for the imblearn pipeline can be seen here and the sklearn pipeline code here. Note that make_pipeline is just a convenient method to create a new pipeline and the differnece here is actually with the pipelines themselves.