Custom weight initialization in Keras From the comments in my previous question, I'm trying to build my own custom weight initializer for an RNN. Based on the code given here (careful - the updated version of Keras uses 'initializers' instead of 'initializations' according to fchollet), I've put together an attempt.

import numpy as np
import pandas, math, sys, keras
from keras import optimizers
from keras import initializers
from keras.models import Sequential
from keras.layers import Dense, SimpleRNN
from keras.regularizers import l2
import numpy as np

def rnn_model(hid_dim=10, ker_reg=0.01, rec_reg=0.01, optimizer="sgd", w_mean = 0., w_var = 0.05):
        my_init = lambda shape: initializers.TruncatedNormal(mean=w_mean, stddev=w_var)
        model = Sequential()
        model.add(SimpleRNN(units=hid_dim, activation='relu', kernel_initializer=my_init, recurrent_initializer=my_init, input_shape = (X.shape[1], X.shape[2]), kernel_regularizer=l2(ker_reg), recurrent_regularizer = l2(rec_reg), return_sequences = False))
        model.add(Dense(Y.shape[1], activation='softmax'))
        model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
        print 'fitting a model'
        return model

When I call rnn_model later, I get an error:

    model = rnn_model(hid_dim=hid_val, ker_reg=ker_reg_best, rec_reg=rec_reg_best, optimizer=optim, w_mean=ave_weights, w_var=var_weights)
  File "rnn.py", line 187, in rnn_model
    model.add(SimpleRNN(units=hid_dim, activation='relu', kernel_initializer=my_init, recurrent_initializer=my_init, input_shape = (X.shape[1], X.shape[2]), kernel_regularizer=l2(ker_reg), recurrent_regularizer = l2(rec_reg), return_sequences = False))
  File "/user/pkgs/anaconda2/lib/python2.7/site-packages/keras/models.py", line 430, in add
    layer(x)
  File "/user/pkgs/anaconda2/lib/python2.7/site-packages/keras/layers/recurrent.py", line 257, in __call__
    return super(Recurrent, self).__call__(inputs, **kwargs)
  File "/user/pkgs/anaconda2/lib/python2.7/site-packages/keras/engine/topology.py", line 551, in __call__
    self.build(input_shapes[0])
  File "/user/pkgs/anaconda2/lib/python2.7/site-packages/keras/layers/recurrent.py", line 478, in build
    constraint=self.kernel_constraint)
  File "/user/pkgs/anaconda2/lib/python2.7/site-packages/keras/engine/topology.py", line 384, in add_weight
    weight = K.variable(initializer(shape), dtype=K.floatx(), name=name)
  File "/user/pkgs/anaconda2/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 288, in variable
    v = tf.Variable(value, dtype=_convert_string_dtype(dtype), name=name)
  File "/user/pkgs/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 197, in __init__
    expected_shape=expected_shape)
  File "/user/pkgs/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 274, in _init_from_args
    initial_value(), name="initial_value", dtype=dtype)
TypeError: __call__() takes at least 2 arguments (1 given)

Does anyone know how to initialize a Keras model using custom parameters for the initializer?

Check this example (a reproduction of the IRNN experiment with pixel-by-pixel sequential MNIST in "A Simple Way to Initialize Recurrent Networks of Rectified Linear Units" by Quoc V. Le, Navdeep Jaitly, Geoffrey E. Hinton).

Relevant code:

model.add(SimpleRNN(hidden_units,
                    kernel_initializer=initializers.RandomNormal(stddev=0.001),
                    recurrent_initializer=initializers.Identity(gain=1.0),
                    activation='relu',
                    input_shape=x_train.shape[1:]))


Array of categorical variables vs one-hot encoding I have some JSON data to be transformed to a machine-learning friendly format. Every object in my data, which will eventually become an instance in my dataset, has the exact same fields (in this case, foo, bar and array). The array field contains a variable number of sub-objects (from 0 to 10). Each of these sub-objects has one categorical field with a massive range and some other fields which can be safely ignored. All categorical fields in these sub-objects belong to the same range.

Example (massively simplified object):

{
"foo": 1
"bar": 0.5
"array" : [
    {
        "categorical": "Lorem"
        "other": 34
        "stuff": 56
    },
    {
        "categorical": "Ipsum"
        "other": 53
        "stuff": 12
    },
    {
        "categorical": "Dolore"
        "other": 6
        "stuff": 101
    }
]
}

Obviously foo and bar are easily represented as numeric attributes. I would now like to represent this array (array) as one large one-hot (or several-hot) vector. Assuming I don't care about any other field of the sub-objects except for the category field categorical, here is my question:

Question

Is it valid and possible to set more than one bit in a one-hot vector (which would make it, I assume, a several-hot vector) to represent all categories present in this instance? If not, how could it be done?

Yes, it will work. Basically by creating the encoding, scikit-learn's label encoder does the same thing, you are creating such more features each representing the presence or absence of that level in your sample. To represent d levels you need d-1 variables, so in your case that is what is exactly happening. However since you said, the number of levels can be massive it will not be a good idea to use a vector that massive. It would be a good idea to run some preliminary analysis and check if some levels are totally useless and dont relate to the response as well, some might have the exact same effect and hence those levels can be combined.


Transformer-based architectures for regression tasks As far as I've seen, transformer-based architectures are always trained with classification tasks (one-hot text tokens for example). Are you aware of any architectures using attention and solving regression tasks? Could one build a regressive auto-encoder for example? How would normalization fit into this (as LayerNorm destroys some of the information from the input)?

In the simplest case, doing regression with Transformers is just a matter of changing the loss function.
BERT-like models that use the representation of the first technical token as an input to the classifier. You can replace the classifier with a regressor and pretty much nothing will change. The error from the regressor will get propagated to the rest of the network and you can both train the regressor and fine-tune/train the underlying Transformer.
Also, I don't think that layer normalization causes severe information loss. It is already there when the network is trained, so the rest of the network parameters need to take care of that, which should not be a problem because the gradients "know very well" that there was a normalization layer.


Generative adversarial networks for multiple distribution noise removal I am working on a project where I need to denoise images, and my dataset is composed of a big chunk of pairs <natural image, same image with synthetically added noise>. The fact is, I have multiple sources of noise (with labels), e.g. Gaussian noise, salt&pepper, distortion, saturation to name a few. Different noises types are on the same original images, meaning that for each undistorted image, I have one pair for each noise type, which I believe is relevant for training.

Thanks to the recent success of GANs for image translation tasks, I was looking into recent architectures and how I can adapt them for my task. The question is: do you think that is possible for a GAN to learn a many-to-one mapping between distributions, i.e. different noise distributions (the many) and the undistorted image distribution (the one) or do I need to train multiple networks for different kinds of noise?

Given that I have paired images I was looking into the pix2pix architecture. What about the recent CycleGAN? Learning the backward mapping function looks a lot tougher if there are multiple distributions, which I guess is why when they release the pre-trained models they indeed have multiple models for multiple tasks.

The task is denoising, so I think you can go with many-to-one mapping between distribution, and then I would probably try changes in generatr architecture if it won't learn properly.
With multiple models, you'd have to classify the type of noise to select correct trained model for particular noise distribution.

And about the CycleGAN...

CycleGAN is for unpaired images. It tries to learn generally more complex problem, because it does not require paired images.

Since you have paired images, you should utilize that fact and go for pix2pix which needs paired images. Also, CycleGAN mas 2 generators and 2 discriminators, so you have bigger model and thus you need more computational power.


SWF - Incremental mining I am new to this data mining. Can anyone please help me with an example for Sliding Window Filtering algorithm(SWF) for incremental mining?

There are pseudo-codes in this paper, Section 3.1 shows an example of incremental mining by this algorithm. However I can't find any projects or working source codes with this algorithm implemented.


Why is this binary state format so much more efficient than a non-binary one? I am teaching myself Deep Q Learning (and machine learning in general) using python and tensorflow to play Snake. My initial learning came mostly from this article. I have rewritten it in my own coding style and it is working well.
Now I am experimenting with different parameters and state information. The state for my game is an array of 11 values that looks like this:
0 - Danger 1 step ahead
1 - Danger 1 step on the right
2 - Danger 1 step on the left
3 - Snake is moving left
4 - Snake is moving right
5 - Snake is moving up
6 - Snake is moving down
7 - Food is on the left
8 - Food is on the right
9 - Food is on the upper side
10 - Food is on the lower side

All of these values are binary, either 0 or 1. To experiment, I wanted to switch out elements 3 through 6 with the constants I use for direction.
DIR_UP = 1
DIR_RIGHT = 2
DIR_DOWN = 3
DIR_LEFT = 4

So the state object would be an array of 8 elements, element 3 being non-binary:
0 - Danger 1 step ahead
1 - Danger 1 step on the right
2 - Danger 1 step on the left
3 - Snake movement direction
4 - Food is on the left
5 - Food is on the right
6 - Food is on the upper side
7 - Food is on the lower side

To me, this seems to have all the same information but in a different format, but the DQN apparently doesn't think so. After 150 training iterations on the first format, I get a competent snake that can regularly score 40 or more. To get the same scores on format 2 I needed to run 800 iterations with a slower epsilon decay. And in testing (without any randomness) it still takes indirect routes and needlessly crashes into the wall now and then, which I thought would have been weeded out.
So my main question is: what am I failing to understand about state format that makes my first so much more efficient than my second?
And sub-questions: are states supposed to be binary? What if I have to account for something like distance that can't be easily represented that way?
Thanks so much for you time and input!

You are increasing the state space by introducing non-binary options. This increases the complexity the model has to learn to reach the same level of performance.
In particular, you are creating a nested structure which makes it more difficult to assign reward credit to a specific option. The hierarchical options are "Should the snake move?" then "If yes, which direction?" make it slower to assign credit to the directional movement. A flat structure "Should the the snake move up?" allows credit to be more directly assigned to that decision.


Replace value of a column if the value of another column is a duplicate I have a Pandas dataframe that contains three columns: ID, name and date. The name column is not unique and may contain duplicates. I want to change the date of the duplicated name to the earliest date. For example:
Input:
id  name    date
1   Rachel  3/10/2020
2   Dave    1/5/2020
3   Rachel  1/10/2020
4   Rachel  1/1/2020
5   Jason   4/15/2020
6   Dave    7/1/2020

Output:
id  name    date
1   Rachel  1/1/2020
2   Dave    1/5/2020
3   Rachel  1/1/2020
4   Rachel  1/1/2020
5   Jason   4/15/2020
6   Dave    1/5/2020

In the sample data above, there are three dates associated to Rachel, the earliest date is 1/1/2020, therefore, in the output, all the dates associated to Rachel are changed to 1/1/2020.

A cleaner way would be:
df["date"] = df.groupby("name").date.transform("min")

Outputs:


Dropout vs weight decay Dropout and weight decay are both regularization techniques. From my experience, dropout has been more widely used in the last few years. Are there scenarios where weight decay shines more than dropout?

These techniques are not mutually exclusive; combining dropout with weight decay has become pretty standard for deep learning.

However, where weight decay applies a linear penalty, dropout can cause the penalty to grow exponentially.  This property of dropout can lead to hypothetical failures as proposed and proven in section 4.2 of this paper.

In general, research has consistently shown the benefits of dropout (with and without weight decay) for training deep networks.  A practical scenario in which weight decay is exclusively preferred over dropout would be quite the anomaly.


ML Approach for Getting List of Observations with Similar Features (Discrete+Continuous) I have a dataset with 19k observations. Each has approximately 448 features:
 - Text description turned into vectors of size 300
 - 16 categorical variables represented numerically
 - The remainder are quantitative features

Each observation also has a list pointing to 10 other observations (from the 19k) that it's most similar to. I want to train an ML model that can understand how the 448 features contribute to this "similarity". Once the model understands that, it could accurately pick the 10 closest existing observations for any new observation, based on how it understands this "similarity".

I've tried clustering with Python's scikit learn (K-Means, MBKM, Affinity Propagation) but these haven't really worked out. I'm currently trying NearestNeighbors but I'm not sure I've got it right.

Please help! Thank you.

With mixed variables, weighting is extremely important.

How you seem to have labeled training data of what should be similar. Hence, you should implement some metric learning approach yourself, where the objective is to learn a similarity metric.

I don't think there is a library that you can just use, but you are better off understanding how this works and writing the code yourself.


A/B Testing (Binomial Distribution vs Random Distribution) When performing an A/B test for the number of clicks for users viewing (each view is an impression) two variants of an ad, a binomial distribution can be assumed where each variant has a constant click through rate.
Example:
Two Ads,
-> Ad one has 1000 impressions and 20 clicks, CTR is 2%;
-> Ad two has 900 impressions and 30 clicks, CTR is 3.3%.
Test whether there is difference between Click Through Rate (CTR) between Ad one and two.
t test for two population:
Similar to the first question, we know sample mean of each experiment is the CTR we need compare. According to CTR, sample mean is limiting normal distributed. Then we have
X¯1∼N(p1,p1(1−p1)/n1)
X¯2∼N(p2,p2(1−p2)/n2)
T-test:

If the click through rate is not constant and is in fact randomly distributed (e.g. normal distributed), how does this affect the standard deviation of the sample mean CTR and consequently the validity of the test result.
My concern is that if use the above distribution and the actual standard deviation is higher we will end up accepting the alternative hypothesis incorrectly. Anecdotally, when we have run A/A tests (i.e. the variants are the same), we appear to often accept the alternative hypothesis more often than you would expect to and I want to find out whether this could be due to the assumptions around the click through rate being constant.

The question you are asking is somewhat confusing and I believe there's a few assumptions you are using which are incorrect.  Please bear with me and correct me if I'm misinterpreting.
You suggest there are different types of click thru rates, constant and random (normally distributed); This is not true.  Click thru rates are assumed to be randomly distributed in a sample.  I do not know what a constant click-thru rate is, but there is no such assumption in the T-test calculation.  Order of clicks do not matter, only the probability, which is equivalent to the actual click thru rate (which is unknown).  When you sample, through measurement, you will then calculate the estimated click thru rate.  We know that this process is probabilistic, and we can end up with estimates that are higher, or lower than the actual CTR, but it should be close.  It is the sample mean which is normally distributed as described by the Central Limit Theorem.  The standard deviation of the estimated CTR, also known as the standard error, is set by the estimated CTR.
The "distribution" of clicks in your sample is assumed to be random, but the order should not matter in the calculation, and should not affect the validity of the test results.
Running an AA test is a good way to confirm whether all the math, statistics, and logging are working out.  If you are finding statistically significant results more frequently than your significance threshold (alpha), something is wrong.  But I cannot see how the "distribution of clicks assumed to be constant" can possibly be the root cause.
Good luck and may your tests be properly powered.


Keras ANN Trained Model's Accuracy change on prediction I have trained an ANN Binary classifier using Keras. It gives 90% accuracy.
After testing when I predict same data again but pass only one class then accuracy decreases to 40%.
I have figured out that if I pass mixed classes while predicting then it will give me around 90%  accuracy and if I pass data points of only one class then accuracy decreases .As I increase the data points of other class as well then accuracy increases.
Long in short.
CASE 1:
100 samples from class 0,
100 sample from class 1,
on predicting using trained model Accuracy = 90%

CASE 2:
Same 100 samples from class 0 passes to same trained Models give me 40% accuracy.
Why accuracy changes ?

EDIT: I'm performing Standardization before predicting every time which effects the predictions. How to handle this case?
Any suggestions would be much appreciated. Thanks

This problem is solved if we use same Mean and Standard Deviation which used for standardization of training samples.


Equipment failure prediction I have a system that manages equipments. When these equipments are faulty, they will be serviced. Imagine my dataset looks like this:

ID
Type
# of times serviced

Example Data:

|ID| Type       | #serviced |
|1 | iphone     | 1         |
|2 | iphone     | 0         |
|3 | android    | 1         |
|4 | android    | 0         |
|5 | blackberry | 0         |

What I want to do is I want to predict "of all the equipments that have not been serviced, which ones are likely to be serviced" ? (ie) identify "at risk" equipments.

The problem is my training data will be #serviced > 0. Any #serviced=0 will not be frozen and dont seem to be valid candidates to include in training. (ie) When it fails, it will be serviced hence the count will go up.

Is this a supervised or unsupervised problem ? (supervised because I have serviced and not-serviced labels, unsupervised because I want to cluster not-serviced with serviced and there by identify at-risk equipments)
What data should I include in training ?

Note:

The example is obviously simplified. In reality I have more features that describe the equipment.

This is supervised learning problem. Type is a predictor. #serviced classifier is target variable. Model is trained on samples set you already have. Best guess is that any model will not have substantual predictive ability. Type is not enough.

Try including more factors (predictors) into the model. Like years_being_in_usage, equipment_model, have_been_in_service_before and so on. The more you get, the better model you can train.


What are alternatives to MLP, when you have rectangular, structured data? I have a rectangular numeric dataset, and I'm applying a multilayer perceptron to it. I'm having success, but I'm now looking to see what other architectures I can apply.

Much of deep learning seems applied to loosely-structured data -- sequences, text, images -- and everyone is having a lot of fun working with a variety of interesting models...at least when they have a problem that fits these models.

What about basic, row/column datasets. What are some of the canonical models to be used with this kind of data, apart from tweaking the layers of a basic MLP?

Fancy deep learning architectures mostly work by exploiting structure in your data. Permutation invariance or equivariance, temporal structure and spatial structure come to mind. Maybe some features are based on a set of the same objects? Most of the benefits come from sharing weights and learning a shared representation. Another potential benefit is the flexibility in output, although that is less relevant in your case. If you have a lot more labeled data on another task with the same features, you could pretrain a larger model on that task and then fine tune that network on your current task. For the rest it is difficult to say without showing some examples.


How to use the same scale with new data? - scikit learn - scikit learn How do I use the same scale used in preprocessing with new data.

Actual code:

x = df.values #returns a numpy array
min_max_scaler = preprocessing.MinMaxScaler()
x_scaled = min_max_scaler.fit_transform(x)
df_scaled = pd.DataFrame(x_scaled)

clf = tree.DecisionTreeClassifier()
clf.fit(X_train, y_train)
pred = clf.predict(X_test)

If I understand it correctly I should have included a scaler variable with the StandardScaler.

  https://stackoverflow.com/questions/38780302/predicting-new-data-using-sklearn-after-standardizing-the-training-data

Something like:

clf = tree.DecisionTreeClassifier()
clf.fit(X_train, y_train)
scaler = preprocessing.StandardScaler().fit(X_train)
pred = clf.predict(X_test)

What scaler parameters should I use for future data processing?

Thanks!

IMO, you don't need to use scaling if your classifiers are based on decision trees. Also, in your final piece of code, the variable scaler is never used, so I am not sure at all why it is defined. Nevertheless, if you insist on using a scaler, you should Pipeline it so it automatically applies to the test data the same scaling it has learned by fitting to training data:

pipeline = Pipeline([('scaler', StandardScaler()), ('classifier', DecisionTreeClassifier())])
pipeline.fit(X_train, y_train)
predictions = pipeline.predict(X_test)

Note that you'd obviously need to import the necessary modules (I was too lazy to write them here!).


K-means clustering of word embedding gives strange results I'm trying to cluster words based on pre trained embeddings. I ran a simple experiment where I obtained around 100 words relating to "food taste", obtained word embeddings from a pre-trained set, and tried to run k-means on the result.

I do get sensible results on certain clusters, but very strange results on others. E.g:

Cluster 1: [fatty, oily, greasy] -- (good)
Cluster 2: [crumbly, powdery, grainy, flakey, chalky] -- (good)
Cluster 3: [flavorful, hearty, unflavored, savory, full-bodied] -(bad)
Cluster 4: [seasoned, unseasoned] -- (bad)

Can anyone suggest as to why seemingly opposite words like (seasoned, unseasoned) and (flavorful, unflavored) would be clustered together?

What I tried:

1) Using fasttext embeddings and Glove embeddings. The latest results are from concatenating fasttext wikipedia and common crawl embeddings.

2) Normalizing the vectors to same length before doing k-means (using eucledian distnaces). I think this is somewhat similar to having cosine distances. Tried not normalizing s well, but normalizing gave better results.

3) Tried some other clustring methods like DBSCAN but k-means seems better.

4) Tried PCA to reduce the dimensionality of the word vectors - didnt change the results much. Tried PCA for the selected embeddings in my vocab that needs to be clustered- not the whole word set.

Any suggestions to improve my results are welcome. If someone came across research articles discussing similar issues please post them as well.

Thanks!
TNC

Word embeddings are trained by substitutability, not similarity.

If you consider a sentence like "This food is unflavored." Then a good substitute word would be "flavored", and the sentence will still be "correct".

In many cases, substitutability arises from similarity (crunchy, crispy) but it does also arise from opposites. You may consider "king" and "queen" to be opposites, too.

You probably should use a supervised approach then.


Specifying neural network output layout for object detection I have a question about if the modeling of the output detection affects the neural nets capability.
In my case I want to train a CNN for object recognition and classification.
As an output I want to get the object classification and a bounding box where it roughly is in the image.
So my thoughts were to model the output as a sequence of vectors:

c = vector for object classification has the length of o (total count of objects to classyify)
l = vector for the bounding box with x, y, width, height

so the complete sequence would look like:

Output = (c + l) x N
where N is the maximum count of objects i would like to detect

Does the order of the subsequence (c + l) have an affect on the effectiveness of the training? Do I have to specify an order, e.g the sequences get ordered by size?

Secondly I have another question, with the specific output format I got numbers in range from 0-1 in the vector c, but numbers from range from 0 -the  size of the image in the bounding box vector l. Does this also affect the training if the activation is set to "relu"? Would it be better to normalize the bounding box vector to the image width and height or the input width and height? So all number have an equal range?

These questions came into my mind while labeling the dataset? So I don't want to waste my time labeling the dataset the wrong way :/

Thank you very much for your help!

I'd say how you order your sequences or normalize data will have much less importance than where you get features for those classifier/detection vectors (where you attach those to your base CNN network).

Your idea falls into approach called "Single Shot Detector" (as opposed to region proposal/sliding windows) and I'd recommend looking at SSD (Liu et al) and YOLO (Redmon, Farhadi) models – there is much more to it to get it performing well.


What does it mean when we say an algorithm/metric is agnostic Problem

I have all kinds of machine learning terms that co-occur with the word "agnostic", including model-agnostic learning, model-agnostic metric. From the dictionary, it explains the word "agnostic" in the following way

  a person who holds the view that any ultimate reality (such as God) is unknown and probably unknowable.

which does not make those terms more understandable.

In some contexts, I find that "agnostic" refer to "generic" or "free of". For example, in the paper I am reading now, the authors define a threshold-agnostic metric, where they use score rather than hard 0/1 assignment for the task.

However, I am wondering if there is formal definition for the word "agnostic" in the machine learning community.

I know of the model-agnostic term, and a close meaning would be model-independent. Basically, when you are studying a machine learning problem, the underlying structure in the data may or may not be described by one type of model or the other. The model-agnostic approach consists in using machine learning models to study the underlying structure without assuming that it can be accurately described by the model because of its nature. This avoids introducing a potential bias in the interpretation.

A model-agnostic approach pretty much requires that several different techniques are used for the same task.


For logistic regression, Predict.glm() outputs $p$ or $ln(p/1-p)$? I'm performing a logistic regression in R.
I wanted to know if the function predict.glm outputs $p$ (probability of event occurring) or log odds i.e. $log(p/1-p)$?

It returns the log odds. You can see that with this basic example,

# Create a perfect correlated data.
data <- data.frame(x = c(1,0,1,0), y = c(T, F, T, F))

# Estimate the model.
model <- glm(y ~ x, data)

# Show predictions.
predict(model)
##        1         2         3         4
## 23.56607 -23.56607  23.56607 -23.56607

# Compute the inverse of the log odd to come to your prediction.
boot::inv.logit(predict(model))
##           1            2            3            4
##1.000000e+00 5.826215e-11 1.000000e+00 5.826215e-11

If logit would return probabilities you could not take the inverse to come to your predicted values.


How to view entire 5788 rows and 7 columns.? I want to see entire 5788 rows and 7 columns of my CSV data. I used below code and it is giving little data by stating [5788 rows x 7 columns] but showing only 10 rows and 5 columns in python prompt. How to get display of entire rows and column there?

import pandas as pd
df = pd.read_csv("BusinessData.csv")
print (df)

pandas has a max rows setting - https://pandas.pydata.org/pandas-docs/stable/user_guide/options.html

Though perhaps looking at a 5,000+ row csv in an editor, or a spreadsheet or some IDEs have a csv editor would be more useful.


How to archive trained PyTorch models? Currently, I am working on my thesis which is built on LSTM networks and I am using PyTorch library. However I am struggling to solve the conceptual problem of archiving trained models.

To make the question more clear; I am saving models in a archicture that I can give this form /models/model-with-loss-2.634221 as an example. But with this form, it is hard to determine which is which. I tried use more detailed form like 1-layered-100-epoch-128-batchsize-...-etc, but it is also hard to read and determine.

What is your way that you think is most productive to handle such operation?

By the way I am not sure this is the correct network ask this question on, you can drop an comment if it is not.

One option is to give each model a unique identifier (e.g., a hash value or nickname). Then store all the metadata in another file.
Another option is using the PyTorch torch-model-archiver feature.


Clustering algorithms for high dimensional binary sparse data I have a dataset with 10,000 genes like below

person gene1 gene2 ... gene10000  ethnic
  1       0      1         1       asian
  2       1      0         1       European

Each row means, whether a person has a gene in their DNA or not. We are trying to classify different ethnic groups based on the data above. But first we want to use some clustering algorithms to visualize how the cluster would look like for different ethnic groups. We are not going to use this clustering algorithms to classify groups, it will be used just to visualize how it would look like, if we have nice separate clusters or overlapping clusters etc.

Please recommend some clustering algorithms for this type of dataset. Also, the dimension is 10000. Is this going to be a problem for clustering? Should I use some dimensionality reduction algorithms first? If so please give your recommendations. Thanks in advance.

How many ethnic groups did you identify?

If I had to visualize your problem, I'd determine the key influencers for each of the ethnic groups in a Naive Bayes like approach.
These genes (gene combinations?!) (including their values) may strongly correlate to some ethnic group, while not (or inverse) correlate to another.

Place them on top of a pyramid graph. Place bars to the left and right for the correlation values.


How to obtain the optimum model using Keras? Deep Learning There are too many parameters while building an artificial neural network. Some of which that comes to my mind are:

Number of layers
Types of layers
Number of nodes in each level of layer
Activation functions in each layer
Ordering of the layers
Different types of optimizers
Different types of loss functions
Batch size while fitting
Epochs while fitting
A lot of other things I never heard of (I suppose)

The lone fact that the number of layers might be in the order of tens and the number of nodes might be in the order of thousands implies that we can end up with billions of different combinations. Considering all the parameters in the list above, how can one know that their model is close to the optimum ?
The cases I study on does not require me to be too precise in terms of r2_score. In other words, it would not make a huge difference if the r2_score is 0.85 or 0.87. But the question is how do I know if I am real close to the saturation point of the r2_score for the given data, i.e. the optimum model would yield an r2_score of 0.90 and I am not stuck at 0.80?

A lot of this comes down to just trying different configurations, it can almost be seen as more as an art than a science. However, there are certain rules of thumb that can be applied based on existing research and models. Examples of this are that the Adam optimizer works well enough for most cases (point 6) and convolutional layers generally work better than others (e.g. dense layers) when working with images (point 2). It is therefore always good to see if you can find existing papers that work on problems that are similar to yours and see which approaches are used and what (doesn't) work(s) well.


Anomaly Detection from available sensor data set? I am working on a live sensor data set and looking for abnormal patterns (leading to a machine fault condition) from the available data set.

I am learning and new to the world of data science, but comfortable with Python. I have few questions that I am looking to get suggestions.

What kind of algorithms would be best suited for this case?
What are the basic steps for doing a predictive analysis in python?

Please correct me if my questions are not correctly framed.

You are talking about anomaly detection, and there are many approaches. If you can create a training set, one-class SVM is a place to start, but even simple control charts can be useful, particularly with live streaming data.


Deep learning on cloud I am trying to implement some deep learning models with large amount of data around 10gigabyte. Although, my Laptop and Collab-free crashes when it tries to load them. Do you think it worths to buy collab-pro? Do you suggest any other solutions? But my worries are mostly about buying collab-pro is only for US and Canada while I am from Europe.
Thanks in advance.

Did you use your machine GPU?
You can use batch processing, and train you're data chunk by chunk, it will take some time to go through all of your 10GB, but it will eventually gives you the result without paying extra.
You can also create a free account on GCP and they will give a 300$ worth of credits and try to execute your models...
The same with Amazon Web Serives, 100USD promotional credits for students, 150USD for educators, 30USD for other occupations AWS
Microsoft Azure give 200USD for free too Azure


Regression - predict n numbers based on another n numbers I'd like to get a recommendation how to attack a problem of predicting multiple numbers.
Training data contains 4 columns, each says a probability of record being in the bucket. So, for example:
X = [0.25, 0.5, 0.25, 0.0]
and
corresponding output should be e.g. Y = [0.8, 0.1, 0.0, 0.1].
Each row should sum to 1.0

What type of approach would you recommend?

I already tried simple neural network with 4 neurons in the last layer and softmax activation but wondering if there is a better solution.

Thanks!

I would recommend using an encoder-decoder structure based on LSTM-RNN for sequence-to-sequence learning; it will provide you with exactly what you want. A link that I find very helpful is:

https://machinelearningmastery.com/define-encoder-decoder-sequence-sequence-model-neural-machine-translation-keras/


How to create speech commands data set I am planning to create a speech recognition network that recognize few words (voice commands) and came across Speech Commands dataset from google. Apart from available dataset I am planning to add few more words like "move", "save" etc, which are not part of the google's dataset.

Is there a way or any other source where i can create/find the dataset for the words ?

As many of the online dataset are available for sentences and speech transcripts i am thinking of writing a scripts that can go through the available transcripts and find the location of the desired word and physically cropping the audio and then padding it to make one second audio file. But this seems very very arduous process and few words like the name of my robot will not be available online.

Can you guys suggest a way ?

There are few ways to create your own dataset or to update already existing one.

By yourself

This way assumes that you have a microphone (at least one). To simplify your recording experience, record the files where you repeat each command. One unique command per one file. Extract the data then. The basic pipeline should be something like this.

Collect data

Record your voice commands in one file for each command. If you use python, pay attention on pyaudio library. While recording do pauses between the words that you repeat. You may try different microphones and different locations. Also, you may ask your friends or relatives to help you with the task.
Read the recording and split it into smaller files with only one word each. For this task you may use VAD - voice audio detector software, like webrtc-vad. There is a wrapper for Python.
To normalize your data, you would want to append your audio files to the same length. In a case of .wav files, append zeros in the beginning and end of the file.

Data augmentation

Add some noise to your data. It may be white noise, randomly generated through the code, or you may just record silence, street noise, quiet conversation etc. Add the noise to your previously recorded data afterwards.

Although, this may not be the best way, but it will work just fine in a case where you.

Extract data

For much bigger dataset, you would probably like to use Simple Audio Indexer to index audio files. It may help you to find timestamps of the specific words or phrases.

Use it on audio recordings, that are available on the internet, like audiobooks (the sound quality is usually good, but the recordings are usually denoised - not like in real life recordings) or open voice datasets (some are listed here).

In the end, you may combine the two ways to solve your problem.


It depends of the algorithm you are using. For a linear model (linear / logistic regression, SVM...), you need to create dummy variables meaning features "Sex_M" and "Sex_F" as you noticed.

However, if you are using tree-based technics, create an integer typed column with Sex in [0, 1, 2] should be sufficient for these algorithms. Reason is, contrary to linear model, tree-based technics are non-linear and will evaluate all possible splits to partition your observations.

However, the way you map your categorical variale into integers can lead to different trees structures. Below is an example. Suppose you want to predict y variable using indexed feature "Sex". On left chart, there is a slight difference between category 2 and categories 0 and 1 because categories are not oriented. It will results into two consecutive splits and should happen late in the tree building. However, on the right chart, the means difference between categories is larger that on left chart. So the split should happen earlier in the tree building. Moreover, you will only need 1 split to make the difference.

To conclude, I think simple indexing can be used with tree-based technics. I would also recommend you to sort your categorical features in an orderly way to easy tree learning.


Is it always better to use the whole dataset to train the final model? A common technique after training, validating and testing the Machine Learning model of preference is to use the complete dataset, including the testing subset, to train a final model to deploy it on, e.g. a product.

  My question is: Is it always for the best to do so? What if the
  performance actually deteriorates?

For example, let us assume a case where the model scores around 65% in classifying the testing subset. This could mean that either the model is trained insufficiently OR that the testing subset consists of outliers. In the latter case, training the final model with them would decrease its performance and you find out only after deploying it.

Re-phrasing my initial question:

  If you had a one-time demonstration of a model, such as deploying it
  on embedded electronics on-board an expensive rocket experiment, would you trust a
  model that has been re-trained with the test subset in the final step without being
  re-tested on its new performance?

Interesting question. I personally haven't seen that for products going into production, but understand the logic.

Theoretically, the more data your deployed model has seen, the better is should generalise. So if you trained the model on the full set of data you have available, it should generalise better than a model which only saw for example train/val sets (e.g. ~ 90%) from the full data set.

The problem with this (and the reason we split data into train/val/test sets in the first place!) is that we want to be able to make statistical claims as to the accuracy on unseen data. As soon as we re-train a model again on all the data, it is no longer possible to make such claims.

[Edit]

Here is a related question on Cross-Validated, where the accepted answer makes similar points to me and mentions other ways of doing things.

We loop over:

train a model
assess performance on validation set $\rightarrow$ if satisfactory, go to step 5
change model
go to step 1
assess performance on test set
Present model with test accuracy found in step 5

Eventually, if you manage to get a great score on the test set, you can claim it generalises well.  So the question as to whether re-training on the full dataset will improve performance on future unseen data is not strictly something you can test. Empirical evidence of better performance in other related problem sets would be the only source or guidance at the point in time when you must make the decision.

A sanity check would be to test the final re-trained model again on the original test set; expecting that it scores higher than it ever did when the model only saw the train/val set, because it has actually seen the test set during training. This wouldn't make me feel 100% confident that this final model is superior in all future cases, but at least it is as good as it can be with the given data.

Perhaps there are more rigorous arguments against doing what you say (probably academically motovated), however it does seem appealing for practical applications!


Naive Bayes for SA in Scikit Learn - how does it work Okay so i scrape data from the web on movie reviews. I also have already got my own 'dictionary' or 'lexicon' with words and their labels (1-poor, 2-ok, 3-good, 4-very good, 5-excellent).

SO the input are paragraphs of movie reviews and i use Scikit Learn Naive Bayes to evaluate the sentiment of each comment , which would be a paragraph. I would like to know how it works under the hood. I ASSUME it uses a bag of words concept.

So , I am describing my assumption about how Scikit Learn Naive Bayes works for SA. How close am i to the reality ?

I am neither a data scientist nor a statistician, but this is a summary of what i THINK happens in  Naive Bayes algorithms for Sentiment Analysis, in Scikit Learn.

Training Phase

Lets assume i am using labels like 1,2,3,4,5 for each paragraph in the training set. Each paragraph is one unit for training as well as evaluation.

STEPS :-

1) Drop unwanted words like THE, BUT, AND and so on

2) Read the first word say 'BEACH', pick it's label from it's parent paragraph, say '5'. So attach
5 to BEACH and put it back in the bag.

3) So add up the number of times each word matched a given label.   Same word 'BEACH' could occur with multiple labels across the input paragraphs. So keep count of each label for the given word. So maybe ['BEACH' - label 1 - 10], ['BEACH' - label 2 - 8] and so on.

4) After above step for all words, sum up the probability of getting a certain label for a given word. So "BEACH" may have 1/6 probability of label 1, 2/6
probability of 2, 1/6 probability of 3, 1/6 probability of 4 and 1/6
probability of 5. So these 5 probabilities for 'BEACH" are put back in the bag of words. Remove the Word-label-count entities from Step 3 , which were used to obtain these probabilities. So now each word can have a maximum of 5 probabilities in the bag.

Evaluating Sentiment Analysis - New Data

1) Now when we feed the real data, when it hits a word, it checks if it is
found in its bag. If not it omits it

2) Say it finds BEACH in one comment. It checks highest probability
for 'BEACH' is 2/6 for label 2. So it picks label 2 for the word
BEACH. Similarly it picks corresponding labels with highest probability for all the words in bag matching the input.

3) Now it sums up all the labels for all words for the paragraph whose SA needs to be computed. Let us say they add up to 20. It has to
convert this 20 to somewhere between 0 and 1. So it uses a logarithmic
conversion which converts 20 to somewhere between 0 and 1. Let us say
0.75.

4) So 0.75 is the weight for this comment.

Is this how it works ? Thanks for any inputs.

Collecting your data

From the comments you state that you wish to classify comments into a label (1-poor, 2-fair, 3-ok, 4-good, 5-very good). Thus you will be training a model that maps a set of words (the paragraph) to a number which represents the rating. I will assume that you also have labels for some of the comments which we will call your training set.

I do not have access to your data but I have a similar dummy example which uses comments and tries to classify them as being a review of a hotel or a car rental. I think this example would suit this purpose nicely.

Pre-process the data

There's many techniques we can use for this purpose. For example some popula rones are bag-of-words, tf-idf, and n-grams. These techniques will take a comment and vectorize it into a set of features. The features which we will use are learned from the training set and then will be the same for new comments coming in. We will use bag-of-words for this example.

Bag-of-words

This method has a dictionary of words which we identify from our training set. We take all the unique words after stemming. We then count the number of times that each word shows up in the comment, that is the vector that represents a given instance. For example if our dictionary looks like

  ['car', 'book', 'test', 'wow']

And the comment was

  Wow, this car is AMAZING!!!!!

The resulting vector would be

  [1, 0, 0, 1]

We will repeat this vectorization process for each comment. We will then be left with a matrix where the rows represents each comment and the columns represent the frequency at which these words appear in the comment. We will call this matrix $X$ it is our dataset. Each of these instances will also have a label we will call that vector $Y$. We want to map each row in $X$ to it's label $Y$.

Assume this is the data you have for 40 instances in a training dataset for the cars/hotel comments. With it's corresponding label on the right. Label 0 is for cars and label 1 is for hotels.

The columns of the matrix $X$ are

  ['car', 'passenger', 'seat', 'drive', 'power', 'highway', 'purchase',
  'hotel', 'room', 'night', 'staff', 'water', 'location']

Split the data

We will split the data in order to test the accuracy of our model

from sklearn.cross_validation import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.3)

Apply Naive Bayes

from sklearn.naive_bayes import GaussianNB

clf = GaussianNB()
clf = clf.fit(X_train, y_train)
clf.score(X_test, y_test)

This gives a result of

  1.0

Perfect classification!

How does it work?

Gaussian Naive Bayes assumes that each feature is described by a Gaussian distribution (you can pick other distributions which may be better suited for your data, such as multinomial). It will then assume that each feature is entirely independent of one another. Thus we can say that the probability of

$P(Y=0|X) = P(X_0|Y=0)P(X_0) * ... * P(X_n|Y=0)P(X_n)$

$P(Y=1|X) = P(X_0|Y=1)P(X_0) * ... * P(X_n|Y=1)P(X_n)$

for your $n$ features. The the one that is the largest is going to be our label. If $P(Y=1|X)$ is larger then we will say that the comment was that of a car. So to do this we need to train the parameters of our probability distributions $P(X_i|Y)$. To calculate this term we need to know the parameters of our distribution, in our case we are using a Gaussian distribution, thus we need the mean and the variance $\mathcal{N}(\mu, \theta)$.

We will build a dictionary of values for each label that contains the mean and variance for each feature. This is the training stage!

Homemade Naive Bayes

import csv
import random
import math
import pandas as pd
import numpy as np

class NaiveBayes(object):

    def __init__(self):
        self.groupClass = None
        self.stats = None

    def calculateGaussian(self, x, mean, std):
        exponent = np.exp(-1*(np.power(x-mean,2)/(2*np.power(std,2))))
        std[std==0] = 0.00001
        return (1 / (np.sqrt(2*math.pi) * std)) * exponent

    def predict(self, x):
        probs = np.ones((len(x), len(self.stats)))

        for ix, instance in enumerate(x):
            for label_ix, label in enumerate(self.stats):
                probs[ix, int(label)] = probs[ix, int(label)] * \
                                      np.prod(self.calculateGaussian(instance, self.stats[label][0], self.stats[label][1]))
        return np.argmax(probs, 1)

    def score(self, x, y):
        pred = self.predict(x)
        return np.sum(1-np.abs(y - pred))/len(x)

    def train(self, x, y):
        self.splitClasses(x, y)
        self.getStats()
        pass

    def splitClasses(self, x, y):
        groupClass = {}

        for instance, label in zip(x, y):
            if not label in groupClass:
                groupClass.update({label: [instance]})
            else:
                groupClass[label].append(instance)
        self.groupClass = groupClass

    def getStats(self):
        stats = {}

        for label in self.groupClass:
            mean = np.mean(np.asarray(self.groupClass[label]), 0)
            std = np.std(np.asarray(self.groupClass[label]), 0)
            stats.update({label: [mean, std]})
        self.stats = stats

clf = NaiveBayes()
clf.train(X_train, y_train)
clf.score(X_test, y_test)

Once again we get an accuracy of 1.0 on the testing set!


What is the difference between filling missing values with 0 or any othe constant term like -999? Most of the text book says to fill missing values use mean/median(numerical) and most frequent(categorical) but I working on a data set which has too many missing values and I can't remove those columns because they are important.

train.isnull().sum()

TransactionID          0
isFraud                0
TransactionDT          0
TransactionAmt         0
ProductCD              0
                   ...
id_36             449555
id_37             449555
id_38             449555
DeviceType        449730
DeviceInfo        471874
Length: 434, dtype: int64

train.shape
(590540, 434)

You can see that DeviceType and DeviceInfo has too many missing values. I am not sure if filling these with most frequent value would be a right choice. So, I am thinking about filling it with some constant term 0 or -999. Is there any difference when fill it with 0 or -999. What is the right way to fill when you have too many missing values?

First, DeviceType and DeviceInfo don't sound like naturally numeric values.  If you're going to need to encode them anyway, then it doesn't matter: just encode "missing" as another level (or the baseline all-zeros).  And if the non-missing values are nearly-unique, they may not be very useful anyway; perhaps just the fact that they exist is informative?

Tree models can deal with missing values implicitly (splitting the non-missing data into two subsets, then examining which side the rows with that feature missing should go to); however, not all implementations allow for that.  E.g., sklearn doesn't yet (but working on it?) allow missing values at all; xgboost and lightgbm do what I've mentioned above; catboost only sends in one fixed direction (https://github.com/catboost/catboost/issues/588).  Quinlan-family trees actually send missing values along all possible paths, and return a result that's a weighted sum of the possible results, weights coming from the proportion of the training data in the node that went along each path (https://stats.stackexchange.com/a/98967/232706).

In a tree-based model, imputing with $-999$ (or any value less than all your data) is the next best choice: it allows you to split the rest of the data however the tree normally would, and just always sends the "missing" rows to the left.  You may want to test imputing with a very large value as well (which will instead always send the missing rows to the right); again, see the catboost github issue above.

For a linear model, imputing with anything will distort the distribution and the model.  However, adding a missingness indicator and imputing (with anything) takes care of the model: the coefficient on the imputed feature can fit to the "real" slope, while the coefficient on the indicator prevents the imputed value from pulling that slope away from its true value.  An indicator variable may also help in a tree-based model, though that's not as certain.


Picking training data Suppose i want to have 80% training data and 20% testing data. How do i choose which 80% of the data to use for training? Should it be completely random? Like what if there is a class label with 2 clusters and the 80% training data we randomly choose only captures one of the clusters?

In order to validation you can choose different methods such as k-fold cross validation, repeated splits and Bootstrapping. It seems what you aim to do is 2 fold cross validation and yes it should be random. You should be careful, and it seems your training data and testing data are very different.In this case, you;ll not have a model with a good performance.Try to find a way and make your training data look more like the data on which you want to implement your model. In this case you might want try 10 fold cross validation (for instance)


How do you search a high dimensional for the global maxima using as few samples as possible? Suppose the value at any point in the space is defined by Y = f(x1, x2 .. xk). For simplicity, we can assume that x takes only binary values. Which means that we have a total of 2^k possible values. I want to find the point in this k-dimensional space such that Y at this point is the highest.

The constraint lies in the cost of calculating Y. Because it is very 'expensive' to calculate Y, I want to do it as infrequently as possible. That being said, I have the ability to measure Y for any point in this k-dimensional space. I'm looking for a way to use only a small subset of values n, where n << 2^k and use it build a prediction model that extrapolates to the entire space

I've been able to do this by starting off with a set of randomly selected n points and using them to construct a regression tree. The prediction error is acceptable. But the size of n is still quite high.

Is there a more intelligent way to choose my n samples? What are some alternative approaches to extrapolate n measurements into the entire feature space?

If I'm willing to settle for the local maxima, how can I find it? Seems like a straight forward problem in linear algebra. But it's been ages and I could really use some help. TIA!

I think you probably should use some randomized optimization algorithm such as randomized hill climbing or genetic algorithm. These are probably more suited for such a problem


Assumptions of linear regression In simple terms, what are the assumptions of Linear Regression?

I just want to know that when I can apply a linear regression model to our dataset.

There are three major assumptions (statistically strictly speaking):

There is a linear relationship between the dependent variables and the regressors (right figure below), meaning the model you are creating actually fits the data.

The errors or residuals of the data are normally distributed and independent from each other.

Homoscedasticity. This means the variance around the regression line is the same for all values of the predictor variable.

Update 2:: Multicollinearity is not an assumption, but it is rather a sanity check especially if interpretability of the model is important (thanks Ricardo Cruz for the comment). Multicollinearity occurs when the independent variables are not independent from each other. Multicollinearity between explanatory variables which can leads to less stable parameter fits (thanks KT. for pointing this out)). There are tests like correlation matrix (Pearson's Bivariate Correlation), Variance Inflation Factor that can be used to verify this.


How to plot using facet_wrap, over multiple pages as a .pdf files in r cran I am using ggplot, to compare 114 unique studies for a particular variable I'm interested in.
This is what I have used.
ggplot(steps, aes(x=factor(edu))) + geom_bar(aes(y = (..count..), group = id_study,)) + facet_wrap(~id_study,)

Whilst this works, all 114 studies are plotted on one page and the formatting is all squashed. How do I split this over 4x4 pages ?

Many thanks

S

edit ****

As there are 114 unique studies, I have 5 pages in total
1)
ggplot(steps, aes(x=factor(edu))) + geom_bar(aes(y = (..count..), group = id_study,))+ facet_wrap_paginate(~id_study, nrow = 5, ncol = 5, scales
 = "fixed", shrink = TRUE, labeller = "label_value", as.table = TRUE, switch = NULL, drop = TRUE, dir = "h", strip.position = "top", page = 1)

2)
ggplot(steps, aes(x=factor(edu))) + geom_bar(aes(y = (..count..), group = id_study,))+ facet_wrap_paginate(~id_study, nrow = 5, ncol = 5, scales
 = "fixed", shrink = TRUE, labeller = "label_value", as.table = TRUE, switch = NULL, drop = TRUE, dir = "h", strip.position = "top", page = 2)

3)
ggplot(steps, aes(x=factor(edu))) + geom_bar(aes(y = (..count..), group = id_study,))+ facet_wrap_paginate(~id_study, nrow = 5, ncol = 5, scales
 = "fixed", shrink = TRUE, labeller = "label_value", as.table = TRUE, switch = NULL, drop = TRUE, dir = "h", strip.position = "top", page = 3)

4)
 ggplot(steps, aes(x=factor(edu))) + geom_bar(aes(y = (..count..), group = id_study,))+ facet_wrap_paginate(~id_study, nrow = 5, ncol = 5, scales
 = "fixed", shrink = TRUE, labeller = "label_value", as.table = TRUE, switch = NULL, drop = TRUE, dir = "h", strip.position = "top", page = 4)

5)

ggplot(steps, aes(x=factor(edu))) + geom_bar(aes(y = (..count..), group = id_study,))+ facet_wrap_paginate(~id_study, nrow = 5, ncol = 5, scales
 = "fixed", shrink = TRUE, labeller = "label_value", as.table = TRUE, switch = NULL, drop = TRUE, dir = "h", strip.position = "top", page = 5)

all of these into 1 page?
thank you

Depending on the shape of your plots, it might make sense to create four separate plots. I suggest this just because it makes sense to show the axes and labels on each individual page, otherwise it could be that the reader would have to always flips back to e.g. page 1 to see what the scales are.

I would probably do something like a 6 (rows) by 5 (columns) on each page, assuming portrait page orientation. Then you need to just adjust your code to do something like this (as pseudo-code):

page1 <- ggplot(steps[0:30]), aes(x=factor(edu))) + geom_bar(aes(y = (..count..), group = id_study,)) + facet_wrap(~id_study,)

page2 <- ggplot(steps[30:60]), aes(x=factor(edu))) + geom_bar(aes(y = (..count..), group = id_study,)) + facet_wrap(~id_study,)

page3 <- ggplot(steps[60:90]), aes(x=factor(edu))) + geom_bar(aes(y = (..count..), group = id_study,)) + facet_wrap(~id_study,)

page4 <- ggplot(steps[90:114]), aes(x=factor(edu))) + geom_bar(aes(y = (..count..), group = id_study,)) + facet_wrap(~id_study,)

Then continue to export the images as you require, for example:

ggsave('./first_page_plots.pdf', page1, dpi=600)

check out more save options here.


Which is a better method Lasso regression or sequential Foward selection? I have a huge data of nearly 200 features and several of them might be correlated? In order to select best features which is preferred Lasso regression or sequential forward selection considering the correlation between features.(I can not run pca on this because feature transformation would transform features which cannot be interpreted).

Few things I understood were: Lasso is really quick and sequential
forward selection is very slow. But I am not sure how would lasso
perform under correlated features i.e. I feel that the weights may be
distributed among features(assume necessary features) which are
correlated and because of low weights due to sharing Lasso might
suggest they are irrelevant features.

I think there is no obvious advantage of one algorithm over another: I'd suggest running cross-validation to investigate which works best for the data at hand. You've suggested forward stepwise and Lasso, which both performs feature selection, you could also investigate backward stepwise and best-subset selection. Linear regression models are fast to train, so maybe you could investigate the cross-validation error for all 4 algorithms.

Hope this helps.


What to do when facial recognition fails to find a face? I am currently implementing a CNN to recognise the identity of people given a portrait picture of the person.
The objective is to maximise the clf.score function in sklearn, the database is composed of 144 images of 70 different individuals. The program will be tested on a database of images of the same 70 individuals, which I do not have access to.

For most of the pictures, my facial recognition algorithm (openCV) will detect and crop out the background, leaving only the face, but in certain cases, it does not find a face.

What solution will return the best classification score?

Options I've considered are:

Leaving the picture as is (Useless background info isn't removed)
Cropping a predefined area (Could risk cropping out parts of the face)

I guess you have a learning problem that has low number of training data. I recommend you a two-step solution.

First you need to do error analysis. Find the images that your classifier does not recognize and try to compare with those which your classifier correctly classified them. find the features that your classifier can not do well on them and try to add features that your classifier can discriminate those, I mean try to find yourself what prompts your classifier to make mistake on those images and why you don't make mistake during recognizing them. Try to add those features that you care but your classifier does not and add them as input.
Try to augment your data using data augmentation for both you mistake and those which you don't. There is an important fact here. don't change the distribution of your data. I mean if one class has two times bigger population than the other class, after augmentation try to keep this ratio. The reason is that maybe after equalizing the number of population of classes you reduce the Bayes error but at borders of classes the answers would be random.


Why does horizontal lines in plt.plot(feature, '.') mean that the data have been properly shuffled? I am following a Mooc and in this lecture about visualisation in explenatory data analysis the lecturer claims that when plotting the row indexes against feature values, if we have lines on the feature value axis it means that the data have been properly shuffled. I can't see why.

Shouldn't an index have only one value in the feature axis?
One horizontal line should mean that the feature values for all indexes have been uniformized, not randomized?

On the contrary, in the following lecture, the lecturer claims that from the absence of vertical lines, the data hasn't been properly shuffled:

I think I get it as if it was, I would have seen clear lines. But how can I bee sure there isn't more classes hidden in these subs?

Shouldn't an index have only one value in the feature axis?

Yes, that's correct. On the graph given as example this is not visible because there are too many row indexes (50000). As a consequence it's impossible to distinguish a particular index from its neighbors, but if the X axis was stretched long enough one would see a single feature value for every index.

One horizontal line should mean that the feature values for all indexes have been uniformized, not randomized?

I think there could be two different confusions here:

An horizontal line means that a single feature value is distributed uniformly across the indexes, which is equivalent to saying that the indexes are random for this feature value. In other words, the chance that this feature value appears at a particular index is the same as at any other index. This is what the author means: the order (indexes) is random for any feature value.
The values for all the features have not been uniformized, this can be seen from the fact that vertically the density of the points is different around the middle (say 0.4-0.6) and the extremes (say 0-0.2 and 0.8-1). Of course this would be more visible with a standard histogram, which would show a kind of peak in the middle but with two high bars at the extremes for 0 and 1 (it can be seen from the continuous lines for these two features values that they appear much more frequently).

One may also note on this graph that there is some kind of underlying discrete distribution of the values: very clearly for values 0 and 1, but also from all the white horizontal lines which show that some values seldom exist in the data.


Calibrating probability thresholds for multiclass classification I have built a network for the classification of three classes. The network consists of a CNN followed by two fully-connected layers. The CNN consists of convolutional layers, followed by batch normalization, a RELU activation, max pooling and drop out. The three classes are imbalanced (as can be seen in the confusion matrix below). I have optimized the parameters of the network to maximize AUC.
I'm calculating the AUC using macro- and micro-averaging. As can be seen in the ROC plot, the AUC is not that bad. On the other hand, the confusion matrix looks pretty bad, especially the first (low) class is badly predicted. The network tends to predict the majority class. As output of the network I'm getting a probability for each class. Then, I'm just taking the class according to the maximum probability for creating the confusion matrix.
I have tried to use balanced class weights while training the network (in the fit method of Keras). This helped that the network also predicts more often the minority class(es) but on the other hand the AUC was decreasing.
Is there a way to infer probability thresholds from the ROC plot? I think for two classes the optimal probability threshold can be inferred from the ROC plot by taking the max(TPR - FPR) but here I have three classes... Or is there another method?

Disclaimer: This answer describes the thoughts I had about this problem, I don't offer any guarantee about their validity so use at your own risks ;)
There are two distinct parts in this problem:

Finding an optimal threshold over three classes
Improving performance with respect to the minority class

First part: I might be wrong but as far as I know there's no way to select a particular probability threshold when there are three classes. And if there were a way, it probably wouldn't be from the ROC plot since the curves for the different plots are independent, so picking a point on the curve would correspond to different thresholds for different classes and I don't see how this would be manageable. The only way that I know is what you did: label with the class which has the maximum probability.
Second part: if you want to force the model to take care of all the classes, you could also try optimizing with macro-AUC, but that would probably lead to the same problem of decreasing the micro-AUC since there would be more errors of true neutral or high predicted as low.
The way I see it, the three way model doesn't work well: the imbalance of the minority class is not that bad, it's only 2 or 3 times less than the other classes. So the fact that the model almost completely dismisses this class is a bit strange. I also see that instances of the true class low are almost as often predicted as class high than class neutral, even though I would expect that the vast majority of errors on class low to be predicted as neutral.
So my vague intuition is that maybe the system could be designed in a way which avoids the three-way problem for the model. I can think of two options in this perspective:

A two-steps system where the first model classifies between low and neutral+high then the second one between neutral and high (or first with low+neutral vs. high and then low vs. neutral). This way each model is binary classification so you have more control over the thresholds at each step. Normally this is not recommended in classification, but here I assume that the classes are not truly categorical.
Pushing the same idea further: there seems to be an order between the classes low < neutral < high, so maybe it would be possible to treat the task as a regression problem. This might help the model avoiding these "big" errors between low and high. Importantly you would also have flexibility with the predicted values: there would be two thresholds to determine and these could be found to optimize any appropriate evaluation measure.


Counting the number of trainable parameters in a gradient boosted tree I recently ran the gradient boosted tree regressor using scikit-learn via:

GradientBoostingRegressor()

This model depends on the following hyperparameters:

Estimators ($N_1$)
Min Samples Leaf ($N_2$)
Max Depth ($N_3$)
which in-turn determine the number of trainable parameters in this model.  My question is, how can I count the number of parameters (trainable or otherwise randomly assigned) which determined the final model as a function of the above?

My guess is $N_1 \times N_2 \times N_3$ but is this correct?

Maybe that can be the size of your grid, but not the number of possible trainable  hyperparameter of the Gradient Boosting Regressor of scikit learn.
I add some more hyperparameters of GBDT Regressor:

loss{‘ls’, ‘lad’, ‘huber’, ‘quantile’},
learning_rate
criterion{‘friedman_mse’, ‘mse’, ‘mae’}
max_features{‘auto’, ‘sqrt’, ‘log2’},

This is just some of the hyperparameters needed, this is extracted from the documentation
https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html


Modelling of an environment that is stochastic in nature I have started learning reinforcement learning and have few doubts regarding model based and model free methods.

Is it possible to model an environment that is stochastic in nature?

Is it because its difficult to model such environment we use model free methods?

Is it possible to model an environment that is stochastic in nature?

Yes, a model of a stochastic environment can be one of:

A distribution model, that outputs a probability distribution for next state and reward, given current state and action. If you are reading Sutton & Barto, or similar work which uses the function $p(s',r|s,a)$, then if you can implement this function for the whole environment, it means that you have access to a distribution model.
A sampling model, that outputs a single reward and next state, given current state and action, with the same probability of any outcome as the real environment. If you can implement an accurate simulation of an environment, then you have a sampling model.

If you want to use approaches such as Dynamic Programming, which work with expected values, then this is much easier with a distribution model, and in that case you need an accurate model from the start (otherwise Dynamic Programming may converge to a non-optimal policy).

  Is it because its difficult to model such environment we use model free methods?

Not really, there is no special difficulty in modelling stochastic environments. For instance, if your environment is a dice game, it is just a matter of implementing the rules and a random number generator for the dice to create a sampling model. A distribution model is usually straightforward for basic dice rules, such as rolling a die to see how many steps you can take.

However, independently of how random the environment is, complex environments can become hard to model. Distribution models may require a lot of maths to calculate all possibilities, so sampling models (simulations) are easier. For instance, a card game is relatively easy to implement in simulation when you track the deck contents. But a distibution model for it is more complex, as you have to track what has already been played or can be figured out from other observations.

Many environments are too complex to model. For example, they may involve real world physics but don't include enough measurements to establish the full state. For instance, when an agent flies a drone it will be affected by air turbulence but cannot directly observe it. Chaotic effects such as turbulence are very hard to model, and gaining real world experience is likely going to be more accurate than any model based on a physics engine, no matter how hard you try to code one. Similarly, visualising the real world, or navigating human social environments can be very hard to model accurately.

When deciding whether or not to use a model-free method, there needs to be a cost analysis. Even in complex environments you may prefer to use a model-based method:

The advantage of a model is that it allows you to safely explore without taking a real action, and also in some cases it may be a lot faster to query the model than to take an action and wait for a result in the real world. In the time it takes for a real action to resolve a computer may be able to check 10, 100, 1000 or more simulated actions from its model.
The disadvantages of a model are:

It might not be as accurate as you like, meaning that basing decisions off it makes the agent's policy too far from optimal
Using a model adds complexity to the agent.

In some cases, the real environment is fast, safe and reliable enough that there's not much to gain by using a model. These cases include a lot of the toy problems used to study learning, where the environment is actually simulated (which is a form of model) but kept separate from the agent (so despite this, the agent is still technically model-free).


preprocessing time sequence I have a long list of event (400 unique events, sequence ~10M long). I want to train an RNN to predict next event.

The preprocessing steps i took are:
(1) turning to OneHotEncoding using pandas:

vector = pd.get_dummies(sr)

This part takes about 15 seconds.

(2) Using a sliding window of 10, i create samples and labels as follows;
i iterate the vector from (1), i take Xt as the label and Xt:t-10 as the data.

Code:

X = np.zeros((len(samples), window_size, voc_len), dtype=np.bool)
y = np.zeros((len(samples), voc_len), dtype=np.bool)

if IN_COLAB:
  loading_bar = tqdm.tqdm_notebook(enumerate(samples),desc='Build dataset',total=len(samples))
else:
  loading_bar = tqdm.tqdm(enumerate(samples),desc='Build dataset',total=len(samples))

for numpy_index, pandas_idx in loading_bar:
  x_idx = (pandas_idx, pandas_idx + window_size)
  y_idx = pandas_idx + window_size
  Xt = vector.iloc[x_idx[0]:x_idx[1]]
  yt = vector.iloc[y_idx]
  X[numpy_index] = Xt
  y[numpy_index] = yt

display(yt)

The problem is that the seconds part is VERY slow (25 minutes).
Is there a better way to do this?
a built in function of some sort?

Thx

There are a couple of design patterns that are contributing to slow code:

Pandas is not designed for large-scale, fast data processing.

Your code is using a for loop which can be slow.

You are manifesting the sliding window before the program needs it. It might be better to create a view on the data and then manifest the data in memory only when the training needs it.

It would be better to use NumPy with numpy.lib.stride_tricks.sliding_window_view


Supervised Recommendation System trained on labeled phrase segments I have a big collection of phrase segments (not whole ones) with user provided labels based on text similarity:

train_data (labeled phrase segments):
"has watched movie" => phraselabel1:"movie watch activity"
"watched with friends X,Y,Z movie" => phraselabel1:"movie watch activity"
"watched show" => phraselabel2: "show watch activity"
"is jobless" => phraselabel3: "lack of job"

input phrase segment:
"watched movie in the past" => ??

output:
recommend ordered set of phrase labels: phraselabel1 then phraselabel3 based on similarity criteria

I would like to build a (scalable) experimental recommendation system that finds the most similar phrase segments that already have phrase labels to the input unlabeled one and return the best labels based on the similarity. Labels will be added by users to subset of the data initially and the system will help future users with existing labels.

The number of phrase-labels will be way less than then total number of phrase segments but more granular than topics/tags

I was thinking of the following approaches:

Do TF-IDF (http://spark.apache.org/docs/latest/mllib-feature-extraction.html) between input phrase segment and all the labeled phrase segments. By finding the most similar I can derive the labels that are best to be suggested
Do TF-IDF between input phrase segment and all the phrase-labels.
Word2Vec?? I am not sure if it can be used for phrase segments as input and not just words..
Labeled LDA (Latent Dirichlet Allocation (https://github.com/myleott/JGibbLabeledLDA) although I don't think this is readily available in Spark
Neural Network (https://spark.apache.org/docs/latest/ml-classification-regression.html#multilayer-perceptron-classifier) trained on the words of all input phrase segments as inputs and phraselabels as outputs using the user click feedback on the selected phraselabel for back-propagation training.

I would appreciate if you could shed some light on which approach (from above or a new one) is better based on the use case..

Thanks :)

CNN-Text-classification. Try this. Keep in mind that a deep learning algorithm will require a large data-set if you want a good accuracy. Since you want to give ranked recommendations to the users, a soft-max layer at the end will give you the probability of the phrase belonging to all the classes, so you can rank the labels in the order of the probability and suggest them to the user.

Your first approach is the simplest and best given that you have finite data. I don't think it will be very scalable as your data grows. Also this approach will work best only when your phrases are so limited in their context and vocabulary. Using a convnet you can train on bigger and more ambiguous phrases and get good predictions.


Affinity propagation fit, fit_predict and predict methods I am using sklearn affinity propagation algorithm as below.

affprop = sklearn.cluster.AffinityPropagation(affinity="precomputed", damping=0.5)

I also have a similarity matrix created for the data I am using. Now I want to use my similarity matrix to use in the affinity propagation model.

In sklearn they have different methods for this such as fit, fit_predict, predict. So, I'm not sure what to use.

Is it correct if I use,

affprop.fit(my similarity matrix)

It is correct to use as you indicated.

For example, in below , I use levenshtein for make similarity matrix. then use this matrix for clustring by affinity algorithm.

lev_similarity = -1*np.array([[distance.levenshtein(w1,w2) for w1 in words]
                              for w2 in words])

af = affprop.fit(lev_similarity)


Area Under Curve with probability I'm coming across a metrics for model evaluation which I had never seen before and I don't know how to further research (since I don't know its proper name).

I'm using someone else's code, whose goal is to perform cross-validation to choose the best tree-based algorithm for a binary classification. It is probably worth saying that classes are highly skewed (93% / 7%).
The metric which is used is the following: the classifier is trained and then the probability associated with each test element is computed.

probas = probas[:,list(clf.classes_).index(1)]

Then, these probabilities are ordered from the highest to the lowest and put in the x-axis. On the y-axis, being y the cumulative sum of entries associated with each probability.
Then, they compute the area under the obtained curve, as in:

joint = zip(probas, truth)
joint = sorted(list(joint), key=lambda x:x[0], reverse=True)
probas = [x[0] for x in joint]
truth = [x[1] for x in joint]

# Calculate accumulated number of true labels at each probability point.
# Also calculate Area Under Curve (AUC) (higher is better model)
truth_cumulative = np.cumsum(truth) / np.sum(truth)
area = np.trapz(truth_cumulative, dx=1) / len(truth)

Can anyone give me an intuition of what this metric is about and a pointer to some resources to better understand it?

Thanks.

In your code the Area Under the Curve (AUC) is used to calculate the area under the Cumulative Distribution Function (CDF). Let's go through the code to see how this is done. However, in the last section, I do not agree with the two lines used to calculate the area. If all the labels have a ground truth label of 1, and we have a perfect classifier where the probability of being Class 1 is always 100%, we should have an AUC of 1. However, we will get 50%. I will show this later. You should note that the way this code is written will only work for a binary classifier, with labels 0 and 1.

We will first calculate the probability of each instance being in class 0 and class 1 by

probs = forest.predict_proba(X_train)

Then we will extract the second column, this contains the probability of each instance being in Class 1.

probas = probs[:,list(forest.classes_).index(1)]

Then we will join the probabilities of each instance with its ground truth label. Then we will sort the groupings based on their probability. The instances with a probability of 1 will be at the start of the list and those with a probability of 0 will be at the end of the list. This is done using

joint = zip(probas, truth)
joint = sorted(list(joint), key=lambda x:x[0], reverse=True)

Then we will separate the probabilities and the ground truth labels into separate lists using

probas = [x[0] for x in joint]
truth = [x[1] for x in joint]

The blue line is the probabilities and the ground truth labels is the orange line.

This is where the problems start. The next two lines do not include the probabilities but only the ground truth labels.

truth_cumulative = np.cumsum(truth)
truth_cumulative = truth_cumulative / np.sum(truth)

This means that if we had all class 1, with probability 1. We would get an AUC of 0.5. This does not make any sense.

For example, if I force it by doing this

probas[100:100000] = 1
truth[100:100000] = 1

There will only be a few 0 values left. Then after the ordering and cumulative sum we get the plot as follow, which is obviously wrong.

We need to do exactly what is written in the comment but not done through code. We will get the false positive rates for each false negative rate. Much like what is described below in the Receiver Operator Curve (ROC).

This can be done with

# Training Set
probas_train = forest.predict_proba(X_train)
fpr_t, tpr_t, thresholds_t = roc_curve(y_train, probas_train[:, 1])

# Testing Set
probas_ = forest.predict_proba(X_test)
fpr, tpr, thresholds = roc_curve(y_test, probas_[:, 1])
roc_auc = auc(fpr, tpr)

train, = plt.plot(fpr_t, tpr_t, label = 'Train')
test, = plt.plot(fpr, tpr, label = 'Test')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.legend()
plt.show()

print('Area Under the Curve (AUC): ', roc_auc)

  Training Set  Area Under the Curve (AUC):  0.998140359403  Testing Set
  Area Under the Curve (AUC):  0.937729144675

Receiver Operator Curve (ROC)

This was invented during WWII as a means to detect aircraft using radar. For example, if we have a sensor which requires some kind of threshold to detect planes. We can determine the true positive rate (TPR) and false positive rate (FPR) which results in our experiment. We will then plot these as the x-axis as the false positive rate, and the y-axis as the true positive rate.

$TPR = \frac{\text{True Positives}}{\text{Positives}}$

$FPR = \frac{\text{False Positives}}{\text{Negatives}}$

If the sensor is perfect we will always have $0$ false positives and always a $100\%$ true positives. This results in a curve which looks like the blue curve. If the sensor was completely random and garbage, then you would have a random guess which is the diagonal line.

A good classifier will be as close to the blue line as possible. Note, that a line below the diagonal is also good, you should just invert your decision. You have a consistent classifier but you are inverting the decision.

Intuitively, consider a sensor which reads a value and it is above a threshold $\theta$ we call it a detection. If we lower the threshold of detection, we should expect more detection, however if we end up with an equal amount of true positives and false negatives, then we are going along the diagonal line. This is a bad classifier.

Here we have 2 sensors, red and yellow, we want to decide which sensor is better. Just looking at the curves is not very effective because it's hard to see which one is definitely higher. So we can devise a metric named the Area Under the Curve (AUC). This is exactly what it means in the name. Then we an say the curve with the higher AUC is the better one.


What kind of machine learning can predict a probability rather than classification? For example, I want to predict a probability of whether raining tomorrow, and I can't get the prob. of cloudy or sunny...etc day.

I only get events(attributes of weather) like temperature, humidity, sunny interval, rainy interval...etc.

Train data:
temp. -> 80F
humidity -> 30%
sunny interval -> 5hr
rainy interval -> 0hr

Label: prob. of rainy tomorrow is 20%

Test data example:
temp. -> 90F
humidity -> 10%
sunny interval -> 10hr
rainy interval -> 0hr

I only can get the prob. of rainy day, so I can't use classification algo.

Like above, I want to predict the prob. of rainy tomorrow. I'm not a professional in ML knowledge. I survey many sources and thought the most possible algorithm of ML maybe is regression!?(Cuz it can give me a value like linear regression).

But I don't know is there any other algorithms can help me to predict a probability(also can say I want to predict a value)?

Most of the classification algorithms can return a probability or a similar measure.

Usually, you don't get black or white, but nuances of grey, and then based on a threshold, you select one or the other answer.


Generative Adversarial Text to Image Synthesis Can anyone explain the meaning of this line:
"Deep networks have been shown to learn representations
in which interpolations between embedding pairs tend to
be near the data manifold".
Reference: Section 4.3 of the paper Generative Adversarial Text to Image Synthesis

There are several important concepts in this sentence so let's break them down.
"Data manifold"
This is a usual model we use in machine learning problem, that is to consider real data as a manifold.
I suggest to you this article by Christopher Olah on the topic.
"Interpolation between embedding pairs"
One way of generating new data is to sample the embedding space learned by the neural network. For example, you can take two real data samples, compute their embedding, interpolate them to obtain an intermediate embedding, then see the output of your neural network when fed with this intermediate embedding.
"Interpolations between embedding pairs tend to be near the data manifold"
Following the previous step, the output of your neural network should be realistic. In mathematical terms, it should be near the real data manifold.
This is the main point of generative models such as generative adversarial networks or variational autoencoders. They learn to fit some random distribution, usually a gaussian, to the real data distribution, and they learn to "convert" noise to real data and vice versa.
This is often referred to as disentanglement. As explained in the first paper cited by the article after the sentence,

deeper representations, when well trained, tend to do a better job at disentangling the underlying factors of variation.

In other words, we can obtain deep embeddings that isolate the factors of variation of real data. Ideally, embeddings of human faces can isolate the axis that controls color of the hair, expression of the mouth, etc. (like TL-GAN for example). But these factors of variations are not always as easily explainable.


If I train a model using only my testing data, will the accuracy be 100%? While I understand that if I train a model using the same data that I test on, then I'll certainly be overfitting, but will the accuracy of that model always be 100%?

In other words, is one method of testing whether I've created my model correctly, to test the model by training it on the testing data?

It will likely be not 100% even though it is possible. But, for example, a linear model might not be able to represent a non-linear relationship in your data. Or your features don't contain sufficient information to separate all classes.

Your other question doesn't seem to make sense. If you train your model on the test data, the test data wouldn't really be test data anymore?!

There are tons of resources on bias-variance trade-off and overfitting.


Recommendations based on other products seen I am trying to develop a basic book recommender system to get in touch with the field and start learning methods and how to prepare the data.

The Dataframe I am using is pretty plain, it has the following structure (this is a simplified example):

    number    type   username  product  publishing_dt   price   genres
0       34  access   kerrigan   130365  2019-12-10      16.99   fantasy, kids
1        1   order   kerrigan    76863  2020-01-15      4.66    action, crime
2        1   order  45michael    76863  2020-01-15      4.66    action, crime
3        1   order   kerrigan    76863  2020-01-15      4.66    action, crime
4        1   order  45michael    86833  2020-02-4       15.65   adventure
5        1   order  45michael    86833  2020-02-4       15.65   adventure
6        1   order  45michael   130365  2019-12-10      16.99   fantasy, kids
7        1   order    alicia7   130365  2019-12-10      16.99   fantasy, kids
8        1   order    alicia7   130365  2019-12-10      16.99   fantasy, kids
9        1   order      john5    86833  2020-02-4       15.65   adventure
10       1   order      john5    86833  2020-02-4       15.65   adventure
11      10  access  45michael    63767  2020-01-24      12.99   adventure, fantasy
12       1   order     uololo   830166  2019-11-03      18.45   action, war
13      25  access     7762hc    84325  2019-11-04      25.60   romance
14       4  access   adrian12   997165  2019-12-16      9.99    health, motivational
15       1   order     kylemm   537077  2019-12-25      8.55    history
16      31  access      yvera    76863  2020-01-15      4.66    action, crime
17       1   order   kerrigan  1023897  2020-02-03      7.99    adventure, humor
18       1   order     angel8    86217  2020-02-01      14.99   guide, travel

To make item-based recommendations (which only take into account metadata information) I've used Count Vectorizer and Cosine Similarity.

For purchase-history-based recommendations, for now, I'm also using Cosine Similarity, I am calculating a similarity score between users, the run order would be this one:

Calculate the most similar users to the one we want to get
recommendations for.
From those users, get their purchase history.
Delete from that list those books the user has already bought.
Return the final list with the recommendations.

In this case, I've tried to apply KNN but the truth is I just couldn't and finally discarded it, but I think that with this data Cosine Similarity is a good choice.

The last scenario I would like to contemplate is the classic "According to items you've seen..." (I don't know how to properly translate that into English, but you get the idea). That's why the number field in the Dataframe is there, it shows the number of times certain user accessed the book file but didn't purchase it. So I've been thinking of various options:

From all the books the user has accessed, take, for example, a random one and return new recommendations based on that one. This could end up being a pretty bad choice as people can make clicks on certain items without even being interested in them.
Search for similarities between the books the user has been seeing
and search for new books with those characteristics. This seems like the correct choice, but I don't really know how to take it into action, it looks like a KNN approach.
Simply show the user those items he has been seeing, ordered by
number of accesses (the more he accessed the file, the more he is
interested in it).

Everything in Python, of course, I'm here searching for some help on where to start on this last scenario specifically, and if someone can recommend me a good tutorial or guide on how to apply KNN in similar cases to this, where I don't have any kind of ratings (feature that I've been noticing it has a lot of weight on recommender systems).

For very large datasets there are more powerful methods than k-nn, try to look at what the well known companies like Google, Amazon, Netflix, etc. have published.

Since you are learning, I assume your dataset is small. You can define a graph and recommend based on all views of a user. One simple method is "user-based" recommendations. A more general perspective is that of path counting / random walk ("personalized pagerank", spectral methods, etc).

If you can read math papers, you could start with C. Cooper et al Random Walks in Recommender Systems:Exact Computation and Simulations.

For book content similarities, with a small dataset, apart from the item-based similarity itself, probably your only choice is to try to find some external metadata.


Constructing the Confusion matrix from given metrics I am given the following metrics for a certain classifier :
 -Total number of cases in the dataset = 110
 -Accuracy: 92.7%
 -Precision : 96.9%
 -Recall : 95%

Are this information enough to reconstruct the confusion matrix?

[edit thanks to comment] I'm assuming this is a binary classifier, since normally a multi-class classifier would not be evaluated with precision/recall (it would require micro/macro precision/recall).

Yes, that should be enough:

accuracy = 92.7%:

$$\frac{TP+TN}{110}=0.927 \rightarrow TP+TN=101.97$$

This means we have 102 correct predictions, so $FP+FN= 8$ incorrect predictions (since $TP+FP+TN+FN=110$).

precision = 96.9%:

$$\frac{TP}{TP+FP}=0.969 \rightarrow TP=31.258\times FP$$

recall = 95%:

$$\frac{TP}{TP+FN}=0.950 \rightarrow TP=19 \times FN$$

This gives us:

$$\frac{TP}{31.258}+\frac{TP}{19}=8 \rightarrow TP = 94.6$$

let's assume that means 95 true positive instances, so we get:

$FP = 3$
$FN = 5$
$TN = 7$