How to combine sparse text features with user smile for sentiment classification? I am trying to perform sentiment classification task where I have some text and some information about whether the user smiled or not.
Now when I use count-vectorizer to convert my text to feature set (Bag of Words) approach it results in more than 5000 features. When I add smile to these 5000 features, total number of features become 5001. However, I do not see any improvement in the classification accuracy. I think the 5000 sparse features dominate the one and only smile feature. Hence no effect on the classification. Can anyone recommend, what is the good way to merge textual feature with smile, prosody or other features?

If you are using a RNN and the 5000 features refers to size of your dictionary, I would recommend you consider an embedding of your words, eg word2vec. Each word would then result in a maybe 200-300 - sized vector, to which you can concatenate the single smile feature, and then run that through your RNN.

If that is not doing much, you can run the embedded text through an RNN, and then take the result of that (lets say 10-20 features per time-step) and concatenate the smile feature, and run those through a second RNN. The idea is the first one is a further compression over time, and the second is your 'final' RNN which works at a higher level of abstraction.

In general, this is a classic case of combining a high-dimensional feature set with a low dimensional one, to which there are many approaches, the question being, when do you join the features. Take a look at this paper, which covers a few ideas on multi-modal features aka when and how to merge them (diagram 3.2), though I think it is more instructive than useful since their features are more rich than what you have with the single additional feature.

Another thing to try is to soften the 'smile' feature if it is not so already, aka if it is a 0 or 1, make it a random (0, 0.1) and (0.9, 1) respectively.


TicTacToe Linear Regression low accuracy and R^2 score Im using the python sklearn library to attempt a linear regression TicTacToe AI.
I create my training set by simply having the computer play random 'blind' games against itself. For example... Player one plays a random segment of the board. Next player two plays a random valid segment of the board etc. This goes on until the board is full or someone has won. Each time player one wins, i store the board states leading up to the win. Every loss, i simply mark that board state (and past board states of the same game )  as a loss for player one. For every tie game(full board) I do not count it as anything. I play about 20k of these games. At the end I get my training data set which includes the board state (the feature set) and the outcome which is the percentage (a floating pint value. eg .8 is 80%) of games won for that state.
So for example going from board top left to bottom right: [1, 1, 1, 2, 0, 2, 0, 0, 0] would be:
X X X
O - O
- - -

would have a '1' or 100 percent after playing 20k random games etc.
I'm trying to predict the success rate of player one's next move. Basically the success rate of any free segment based on the board state.
However; after training sklearn linear regression with my training data, I get a very low R^2 score of .14 and any test is highly inaccurate. I'm beginning to think there is a flaw in my data? Is this how data scientists would go about creating the training set for tic tac toe?

Linear regression will not work for this problem because the relationship between the board features and target variable that you are using is not linear.

Is this how data scientists would go about creating the training set for tic tac toe?

It is not 100% clear what your goal is. For simplicity I will select that your goal as "Predict the probability of X winning eventually given the current board state and completely random play in future by both sides." That appears to be what you are doing.
As an aside, this is not a direct path to training a neural network to predict the best moves to make in a game. For this simple game, it might work acceptably if that is your eventual goal, but if you want machine learning for game playing you should probably look into reinforcement learning, and specifically self-play with reinforcement learning, as a framework to manage the training data.
Back to your question, what you are doing is acceptable for creating a data set, although I would want to check:

For every tie game(full board) I do not count it as anything

If that means you are still storing the states that lead to a tie, but with a different label, then that is ok. If you are discarding data about ties, then that will skew the dataset and might impact your predictions - unless you are also discarding ties when testing.
This is also slightly unusual:

At the end I get my training data set which includes the board state (the feature set) and the outcome which is the percentage (a floating pint value. eg .8 is 80%) of games won for that state.

This is unusual in that you have pre-processed the data into a summary row when features are identical. This skews the dataset when used with an approximation function (linear regression - like most ML statistical learners - is an approximation function), because you lose the number of times those features occurred. Any balancing the prediction function does to make itself more accurate for common states is lost when you do this. It is more normal to keep all records separate and have the ML method resolve the best way to take averages. If you measure the accuracy of your completed model by taking random samples of new played games, it could have lower accuracy than possible due to this.
For data collection of records, it is more usual to keep all observations separate and not to summarise them before training a classifier. The classifier can then fit the data allowing for the frequency of each observation.
Other than the caveats about ties (which you may well have right), and premature taking of averages, plus the limitation that your dataset will only help predict outcomes in fully random games, then the dataset collection looks ok to me. Neither of the above problems are major enough to cause the problem that you noticed. The reason your predictions are not working with linear regression is mainly due to needing non-linearity in the prediction function.
A simple fix for this would be to use a non-linear predictor such as a neural network or maybe a decision-tree algorithm like xgboost.
If you use a neural network, the following may help:

Use sigmoid activation in the output layer and binary cross-entropy loss. This should help when your output is a probability.

Use the value $-1$ instead of $2$ for marking positions in the board played by O. This is not strictly required, but neural networks tend to learn faster and more accurately when the input data in centered around zero with close to 1 standard deviation.

It is worth noting that your averaged win rate table is already quite a reasonable predictive model for game playing. For TicTacToe it should work quite well because there are a limited number of states. After 20k games with random play, you will have a record of nearly every possible state, and some will have reasonably accurate average values (for instance each initial play by X will have ~2000 sampled continuations which should give you the win rate within a few percent). The weakness of this approach is that it cannot generalise to new unseen states, but actually that is quite hard to do in board games where fine detail matters.


Is feature importance from classification a good way to select features for clustering? I have a large data set with many features (70). By doing preprocessing (removing features with too many missing values and those that are not correlated with the binary target variable) I have arrived at 15 features. I am now using a decision tree to perform classification with respect to these 15 features and the binary target variable so I can obtain feature importance. Then, I would choose features with high importance to use as an input for my clustering algorithm. Does using feature importance in this context make any sense?

It might make sense, but it depends what you're trying to do:

If the goal is to predict the binary target for any instance, a classifier will perform much better.
If the goal is to group instances by their similarity, loosely taking the binary target into account indirectly, then clustering in this way makes sense. This would correspond to a more exploratory task where the goal is to discover patterns in the data, focusing on the features which are good indicators of the target (it depends how good they actually are).


LSTM neural network for music generation I am doing a project where I have to train a LSTM neural network to generate music. The first step is of course training it. I am using Keras on top of Theano for this task and music21 for feature extraction from MIDI files.
So far the data I extract from MIDI files has the following format:

[[69, -7, 1, 0.75],
 [62, -1, 0, 0.25] ... ]

These are the 4 features that I take from the files. They are then placed in a 3D numpy array with the shape - (number of notes in sequence, 1 (time steps per sample), 4 (features) ).
I create another array with the expected output for each time step being the data in the next time step. I've built the following model: http://pastebin.com/AHCJj1wa

The problem is that when it starts training the model I get some nonsensical output. First the epochs' output looks like this:

Epoch 1/100
0s - loss: 15.2986 - acc: 0.9738
Epoch 2/100
0s - loss: -3.3534e+02 - acc: 0.8265
Epoch 3/100
0s - loss: -3.8183e+02 - acc: 0.8265
Epoch 4/100
0s - loss: -3.8218e+02 - acc: 0.8265
Epoch 5/100
0s - loss: -3.8256e+02 - acc: 0.8265

It seems to be taking 0s per epoch and the loss just becomes larger and larger or stops changing at all (differs between runs).
Also at the end of the training cycle when I call .predict it prints something like that:

[[ 0.11846249 -0.13457553  0.08614471 -0.1007532 ]
 [ 0.14596225 -0.17533173  0.10007301 -0.08799653]
 [ 0.15234099 -0.19707087  0.10395571 -0.08419057]
 [ 0.14596225 -0.17533173  0.10007301 -0.08799653]
 [ 0.14340727 -0.1664145   0.0987886  -0.09037785]
 [ 0.14596225 -0.17533173  0.10007301 -0.08799653]
 [ 0.14340727 -0.1664145   0.0987886  -0.09037785]
 [ 0.14802463 -0.17770432  0.10296948 -0.08740482]
 ..... ]

I think that there is something fundamentally wrong with my model and/or representation of the data. Any ideas on what I can do or at least some pointers to where I can read about this?

These are some suggestions you can try:

Just use 1 LSTM layer instead of 2, there might be less problem of vanishing gradient for shallower networks. In fact the dense layer of 4 is optional, since return_sequences = True.
Find a way to normalize the input in the range 0 to 1.
Try different optimizers like SGD or Adam.
Reduce your learning rate, the default learning rate might be too large for an input dimension of 4.
The loss should not be categorical_crossentropy, since this is a regression problem. The loss should mean squared error.

In order of importance 4 > 5 > 3 > 1 > 2 (i.e point 4 is most important)


How can i tell if my model is overfitting from the distribution of predicted probabilities? all,
i am training light gradient boosting and have used all of the necessary parameters to help in over fitting.i plot the predicted probabilities (i..e probabililty has cancer) distribution from the model (after calibrating using calibrated classifier) i.e. their histogram or kde. as you can see from below the probabilities for my class 1 are concentrated on the upper and lower end.
i have tried playing around with bandwith too to smooth this a little and it doesn't smooth the bumps too much. what do you think this shows about my model? isn't it a good thing that the model for class 1 (which is has cancer) is assigning a greater probability for this class?
i am unsure how to interpret this or where i could be going wrong

the red curve is positive class (has cancer) and the blue curve is hasn't. below is plot used to generate.
results = df[['label','predicted_prob']]

colors = ['b', 'r']

for label in [0, 1]:
    results[results['label'] == label]['predicted_prob'].plot.kde(bw_method=0.35,color=colors[label])
plt.xlim(0,1)

Such a plot doesn't really tell you much about overfitting.
First, check that your calibration has worked well; it's possible that an incorrect calibration has pushed the probabilities to the extremes.  Otherwise, the distribution of probabilities being so extreme suggests the data just naturally separates into a segment of easy-to-detect cancers and the rest.  Among the latter, it looks like you get reasonably good but not great rank-ordering of cases.


predict_proba() on a continuos target made binary I am building a Newtons gravitation equation model. It receives two mass and a radius and outputs what it generates from the formula.
$F = G \frac{m1 · m2}{r^2}$
I want to transform it now to a binary classification model, for that I select a threshold. If the force is above then 1 if below 0.
The question
How can I create a predict_proba function given the above problem?

You can use a Sigmoid function on the Force values(Scaled to [0-10] based on a max value)
The Threshold should become 5 after scaling.
def predict_proba(y_pred):

    y_pred = y_pred*10/100000  # scaled to [0-10]
    thresold = 5

    proba = np.exp(y_pred - threshold)/(1 + np.exp(y_pred - threshold))

    return proba

predict_proba(100000), predict_proba(80000), predict_proba(50000), predict_proba(40000), predict_proba(10000), predict_proba(2000)

(0.9933, 0.9525, 0.5, 0.2689, 0.0179, 0.0081)


too few data while too many degrees of freedom in linear regression To recognize handwritten digits, I have a fully connected network, containing only 2 layers: input layer (all pixels of the image) and output layer (0 or 1). I use the simplest linear regression for training (with gradient descent) and got excellent results.

However, I just realized that my model had far more degrees of freedom than data points. Say the data are all nn pixel pictures to recognize, so the number of degrees of freedom is nn+1. I use n=50, but I only use less than 20 data points (20 training pictures). From the point of view of linear fitting/regression, the number of data points should at least surpass that of degrees of freedom. But now in my model the reverse happens, no solution should be found. What happened

Good question. The solution lies in understanding how weights play into the model's prediction. I've glossed over a bunch of issues here, but the big picture is what is most important.

Theory

When you create a model all the weights are psuedo-randomly initialized. This solves your problem of not having enough degrees of freedom to solve a linear regression. We've interpolated our weights and our model will update only the weights it needs as it trains.
Additionally, your model will probably find that most of your weights converge to zero. If you are using sklearn's load_digits() function, I recall that about half of each image was just white pixels. It is likely that there is a significant percentage of pixels that do not matter across all images (and thus might be assigned a weight of zero). Your question of how can you trust the prediction of the model is thus answered and we can modify our statement about degrees of freedom to: There cannot exist a greater number of discovered statistical trends than there are data samples. It is hard to measure "number of discovered statistical trends", but it follows that there is a finite number and that they are less than the number of samples minus 1.

Example

With a fully-connected, shallow neural network consisting of the input layer with weights and an output layer with two neurons, the input to the activation function of each of the output neurons will be something like the following:

$$f(B_1+\sum_{i=1}^{i}W_i*{I_i}_A)=P$$

where

$P$ is the class probability,
$B_1$ is the bias of the output layer,
$W_i$ is the weight of a specific neuronal connection,
${I_i}_A$ is the output of the connected neuron if activated (in this case the pixel value), and
$f()$ is the activation function of the output layer neuron.

If we didn't initialize $W_i$ to something/anything, we would be unable to solve the equation (more variables than samples). However, by randomly initializing them, the equation immediately becomes solvable and we can use stochastic gradient descent or a different algorithm to optimize our weights. I hope this shows how pseudo-randomly initializing $W_i$ solves the mathematical problem of degrees of freedom.


Skills based recommendation system Assuming that I have a list of Users with a list of skills: (each value is a different skill)

And a list of Tasks with a list of demanded skills:

Based on a manual classification that returned: (History list)

Task1 --> Recommended: User3
Task2 --> Reccomended: User5
Task3 --> Recommended: User6
Task4 --> Recommended: User9
Task5 --> Recommended: User8

Given a new Task(6) with skills (A,B,C...):

What would be the best way or approach to build a recommendation system that learns with past data (history list) and returns a list of the "best ranked" users to perform this Task?

Why do you want to build a learning model?

Your problem seems to be a common assignment problem, all informations are known, absolute and deterministic. You don't need a ML algorithm for this.

Just vectorize your data as one-hot-vectors, define a distance function (you can look to the Hamming's distance).

Compute the distance vector beetween user/task and pick the one who minimize this distance.


Why does my GridSearchCV always break up? GridSearchCV for my Random Forest breaks up. I need to know the reason and the solution to make it work:
# Grid-Search for Random Forest
param_grid = {
    'bootstrap': [True],
    'n_estimators': [100, 200, 300, 400, 500],
    'max_depth': [50, 100, None],
    'max_features': ['auto', 200],
    'min_impurity_decrease':[0],
    'min_samples_split': [2, 5],
    'min_samples_leaf': [2, 5],
    'oob_score': [True],
    'warm_start': [True]
}

# Base-Model for improvement
rf_gridsearch = RandomForestRegressor(random_state=42)

# Grid-Search initiation
rf_gridsearch = GridSearchCV(estimator = rf_gridsearch, param_grid = param_grid,
                           scoring = 'neg_mean_absolute_error', cv = 5,
                           n_jobs = -1, verbose = 5)

# Perform the grid search for the model
rf_gridsearch.fit(X_train, y_train)
```

First, you are fitting $5 \cdot 3\cdot2\cdot2\cdot2\cdot5=600$ models and n_estimator=500 is quite big.
Of course, this depends on your dataset and in your computing power.

My first guess will be that you have not enough RAM memory on your laptop(if you are running it there) and that is why it is collapsing.

If the error is this one, I recommend sampling your data to 1/10 or less (depending on your data) and searching for the best hyperparameter there and then using your whole data for the final model.


Is it possible to have a validation error less than train error for a while followed by the reverse behaviour? I am solving for a regression (using tensorflow's DNNRegressor) problem. When I sampled out 20% data (randomly) and divided it further into train-eval (90-10%, random but mutually exclusive), I observed the following graphs-

Here, blue line corresponds to eval set loss and orange line is for train loss. The total length of data is nearly 4.8 million rows, thus 20% being nearly 1 million data points. How to interpret this behavior?

The behavior you notice in the graphs will always be the case regardless of the machine learning algorithm. Machine learning will require you to fit some parameters such that the model can capture the mappings from your input space to the desired outputs.

At the start of the training these parameters will all be assigned randomly, thus we should assume random performance, thus 50% for a 0/1 classification. As the model parameters are tuned using the training data, the model will tend to decrease both the training and evaluation loss. However, after some training there will be a point where the evaluation loss starts increasing. This is the point where the parameters are no longer generalizing for the data but is instead overfitting and specifically identifying points in the training data.

This is NOT what we want. This is overfitting. We want to stop the training at the minimum loss of the evaluation set.

Another way to think of this is if I ask a child to look through some multiplication examples,

3*2 = 6
4*3 = 12
...
...
10*1= 10

If the kid learns some general rules as to how x*y=z then he will be able to answer any new multiplication question I ask him. However, if he just memorizes the outcomes of the few examples that I showed them then they will get any new instance wrong. This is overfitting.

As another example, imagine fitting a 2nd order relationship between 2 variables with a 5th order polynomial. This will match the points too specifically and will not generalize to new data.


Mahalanobis distance between two clusters I want to calculate the Mahalanobis distance between cluster $a$ and cluster $b$, each consisting from a set of multidimensional points. Assuming no correlation, calculating the distance between a random point $p$ and cluster $b$ can be achieved using the normalized Euclidean distance formula:

$$ d(p,b)=\sqrt{ \sum_{i=1}^d \frac{(p_i-b_i)^2}{\sigma_i^2}}$$

where $d$ is the number of dimensions and $\sigma_i^2$ is the squared standard deviation for each dimension in cluster $b$.

Now, I want to estimate the Mahalanobis distance between clusters $a$ and $b$. Should I assume that cluster $a$ is a single point (i.e. $a$'s centroid)? Or should I normalize using the standard deviation of both clusters? In the latter case that would translate in the following formula:

$$ d(a,b)=\sqrt{ \sum_{i=1}^d \frac{(a_i-b_i)^2}{(\sigma_i^a)^2 \cdot (\sigma_i^b)^2}}$$

where $\sigma_i^b$ is the standard deviation for dimension $i$ in cluster $b$ and $\sigma_i^a$ is the standard deviation for dimension $i$ in cluster $a$.

Thank you in advance.

---------Please check the edits also for this answer---------------------

According to me, its very application specific, and depends on what you want to do. I will prefer second approach in a generic application because if 2 clusters between whom we are calculating distance are having high standard deviation, should have small distance.
Another approach I can think of is a combination of the 2. Calculate the Mahalanobis distance between 2 centroids and decrease it by the sum of standard deviation of both the clusters. I thought about this idea because, when we calculate the distance between 2 circles, we calculate the distance between nearest pair of points from different circles. Now think of circumference of the circle centered by centroid of circle. and the rest is obvious :)

EDIT:

As pointed out by @MarcusD in the comment, I will try to explain a bit:

I remarked "very application specific" because in some cases where our answer should remain same irrespective of the standard deviation of the data, then first approach will work better.
For reference check. Kevin murphy- ML A probabilistic approach Pg. 104, 2 Class LDA. The second approach is exactly this one. If you are not having this book, google Linear discriminant analysis.

The approach I gave is not theoretically different from LDA, but it is somewhat easier to implement in cases where the number of clusters is less.


What are the rules when extracting SVO triples from preprocessed text? If you have some already preprocessed text that is tagged, what are the rules to extract Subject-Verb-Object (SVO) triples if you want a triple like (word, word, word). Can you give the sentence as example and extract all triples? Do you just need to find all combinations without repetition from set of N words?

The goal of Subject-Verb-Object (SVO) triples is to extract a single triple for a sentence.
The sentence:

A rare black squirrel has become a regular visitor to a suburban garden.

results in the following SVO:

(squirrel, become, visitor)

Triplet Extraction From Sentences by Rusu et al. outlines how to do that. First, you'll need a parse tree ( Stanford Parser and OpenNLP are the most common). The three items then can be extracted:

The subject will be found by performing breadth first search and selecting the first descendent of NP that is a noun.

… the predicate of the sentence, a search will be performed in the VP subtree. The deepest verb descendent of the verb phrase will give the second element of the triplet.

… we look for objects. These can be found in three different subtrees, all siblings of the VP subtree containing the predicate. The subtrees are: PP (prepositional phrase), NP and ADJP (adjective phrase). In NP and PP we search for the first noun, while in ADJP we find the first adjective.


What are kernel initializers and what is their significance? I was looking at code and found this:

model.add(Dense(13, input_dim=13, kernel_initializer='normal', activation='relu'))

I was keen to know about kernel_initializer but wasn't able to understand it's significance?

The neural network needs to start with some weights and then iteratively update them to better values. The term kernel_initializer is a fancy term for which statistical distribution or function to use for initialising the weights. In case of statistical distribution, the library will generate numbers from that statistical distribution and use as starting weights.

For example in the above code, normal distribution will be used to initialise weights. You can use other functions (constants like 1s or 0s) and distributions (uniform) too. All possible options are documented here.

Additional explanation: The term kernel is a carryover from other classical methods like SVM. The idea is to transform data in a given input space to another space where the transformation is achieved using kernel functions. We can think of neural network layers as non-linear maps doing these transformations, so the term kernels is used.


Tensorflow classification - maximize the accuracy of certain classes I'm doing some experimentation and trying to train a forex trading model to classify based on three classes:

Buy
Sell
No action

Input rows are labeled as buy when the price at some later time is X% greater than the current price and labeled as sell when the future price is X% less than the current price.  All other rows are labeled as no action, which is the large majority of rows.

The problem I'm running into is that the no action items represent about 84% of the total dataset, however the model is predicting no action almost 99% of the time.  I suppose the reason for this is that the model is aiming for maximum accuracy and it doesn't realize that no action items are of no use.

So what I would like, if possible, is for the model to only care about buy and sell items in trying to maximize accuracy to hopefully increase the number of buy and sell predictions (to the extent that this helps the profitability of the model).

One way that I think I could do this is to increase the number of classes and have each class represent a roughly equal percentage of the total items, however, that would be pretty messy, so I'm curious if there's a way that directly affects the way accuracy is calculated.

EDIT:  I've added additional details here: Variability in CNN test results

But the focus is still in maximizing the accuracy of the buy/sell categories.  Looking through the Tensorflow metrics (https://www.tensorflow.org/api_docs/python/tf/metrics), I think I could use two separate binary classifiers and use the TruePositives metric, however, I'd prefer to keep things simple and get it all in one model.

The best approach for this problem would be to change the value of X. For example, if you are using -%2 and %2 as the classification limit such as  sell(<-%2), buy(>%2) and no action otherwise; you can reduce this to %1, which will in turn reduce the number of samples that fall into this class while increasing number of samples in buy and sell classes.

If you want to keep that value, the way to go is undersampling: when training you take equal number of samples from each class, meaning you ignore some portion of the data from no action class. So you training and validation sets will be balanced and you can use accuracy as a proper metric.

If this also is not a good option for you, another way would be to try changing the classification threshold for each output so that their possible outcomes are roughly equal. But this might be a little complicated to implement as you have 3 classes (for 2 classes this can be done in a relatively simple way by changing classification threshold from 0.5).

If none of these suit you, you can find a more general approach to imbalance problems here. In case you wish to use a different metric, the most popular ones are listed and discussed in detail here. You probably don't need to define a custom metric because there is a one for pretty much every choice.


Why Do Neural Networks Prefer Dense Features Reading through this tf-ranking notebook there is an interesting note that typically networks take in dense features, this is true in my general experience. But I am wondering why, if I were to pass in a categorical feature as a one hot encoded representation would it be worse?

  https://colab.research.google.com/github/tensorflow/ranking/blob/master/tensorflow_ranking/examples/handling_sparse_features.ipynb#scrollTo=aQ-VTA56sOTA

A one-hot encoded feature will act like a lookup table, picking out from the first layer in the network a dense representation for that categorical value. The first layer transforms the one-hot into a dense representation via matrix multiplication.


Working with text files in Excel I have an excel file containing a long text in column A. I am looking for the words starting by "popul" such as popular and populate . I can find these cells by the formula:

    =SEARCH("popul",A1,1)

I want a function that returns the whole words starting by popul such as popular and populate.

I'm no Excel expert, as I generally use Python or R instead, but this might get you started until an Excel expert comes along.  In the meantime, it would help if you clarified your question.  And you should be aware that search will only find you the index of the first match, not all matches in the string.  If you only need the first hit, you can use

=MID(A1,SEARCH("popul",A1,1),IFERROR(FIND(" ",A1,SEARCH("popul",A1,1)),LEN(A1)+1)-SEARCH("popul",A1,1))

although I cannot claim this is the best way to do this.  You really didn't specify where you want the results to appear, how they should look, or if you only have one cell you need to search in.  It would also help to know the version of Excel you have.  I'll also present a crude way to return all the hits in the string:

Cell A1 contains the string, B1 has no formula, and if you run out of "n/a"s you can extend columns B, C, and D by filling down.  The formulas are as follows:

B3 and below use

=IF(C2+1<LEN($A$1),C2+1,"n/a")

C2 and below use

=IFERROR(FIND(" ",$A$1,SEARCH("popul",$A$1,B2)),LEN($A$1)+1)

D2 and below use

=IFERROR(MID($A$1,SEARCH("popul",$A$1,B2),C2-SEARCH("popul",$A$1,B2)),"")

As you can see, there's little to no error checking except to deal with the match at the end of the string.  In the end though, if you're going to use Excel for this you should probably create a user defined function or utilize VBA instead of in-cell formulas.


Which ML classifier is appropriate for me if all of my features are categorical? My dataset contains four features. All of the features are categorical. There are 150 categories in the value of 1st and 2nd features. There are 8 categories in the value of 3rd and 4th feature. I replaced the categories with numeric values and applied Random forest. However, performance is not still up to the mark.

What other Machine Learning Classification algorithms I can try?

What does performance is not up to the mark mean?

To optimize performance you first need to understand it.

A typical workflow is this:

1. Define a baseline performanc

E.g. predict values with a constant (average, mode) or pick randomly from a distribution and measure the accuracy of that

2. Compare your 1st draft model (in your case your RandomForest) to that baseline

If your 1st draft model isn't better than the baseline, especially a random baseline, you have some errors in your code, data, etc. Try to find and eliminate those first.

If it is better but not by much, you have top optimize the model (see next step). If it is better by much you are either a) done or b) not satisfied with the absolute performance in which case you have to optimize.

3. Optimize your model parameters

Now you start grid searching your model parameters to get every last bit of performance out of it to make sure the problem is with the model and not the parameters.

4. Beauty contest

If your performance is still subpar now you can try other algorithms but do not expect wonders from this step. If you already tried RandomForest maybe go for:

Boosted models like XGBoost
Naive Gauss models
...

Fit them on the same data, do parameter optimization and compare results.

5. Back to the data

Most likely just picking another model did not help if the original performance was too far away from acceptable. Then you have to go back to the data, collect more, feature engineer, etc.


Correlation among variables (categorical, binary and numerical) I have the following data
 Prefix Label   Year    Symbols Tags    %_Men   Prop    Uni_Role    Uni_Role_2  #_Cars  #_App
    0   IN  0.0     1996    []        []    0.08    0         11         22           0     1
    1   IT  0.0     2000    []        []    0.00    0         2          46           1     1
    2   ES  1.0     1999    []        []    0.10    0         10         30           0     1
    3   UK  1.0     1999    [!]      [!]    0.07    1         10         25           0     1
    4   USA 0.0     2007    []        []    0.07    0         11         23           0     1
    6   Republic of San Marino 0.0  1996    []  []  0.07  0   10         20           0     1
    7   USA 0.0     2002    []        []    0.05    0         5          22           0     1
    8   IN  0.0     2008    []        []    0.03    0         4          18           0     1
    9   IN  0.0.    1997    []        []    0.04    0         9          19           1     1
    10  IN  0.0     1997    []        []    0.06    0         8          20           3     1

I would like to study correlation among variables (target variable is Label). Since there is a mix of categorical (including lists, under Symbols and Tags columns), numerical and binary variables. For categorical variables probably I should consider Chi-Square, for numerical Pearson. I do not know how to get information from binary, but I know that it should not be possible to apply Pearson.
The problem is a classification one, where I should predict the status of a citizen.
Have you had similar problems? Any idea on how I could do that?

There are some distance/dissimilarity functions for two binary/boolean vectors such as Jaccard distance and  Hamming distance. You can check them from scipy's documentation. You can calculate similarities between two boolean columns by using these distance functions.
from scipy.spatial import distance
distance.hamming([1, 0, 0], [0, 1, 0])
0.66666666666666663

from scipy.spatial import distance
distance.jaccard([1, 0, 0], [0, 1, 0])
1.0


Question about Similarity vs Dissimilarity Matrix Right now, I'm working on a coming up with a similarity vs dissimilarity matrix for a set of data points for a clustering algorithm. My question is, if I want to use one of the many clustering algorithms given in $R$, such as the K-Medoids algorithm, does it require a similarity or dissimilarity matrix as its parameter?

What's the difference between the two?

If I use the Gower Distance from the Daisy function in R, does it output a similarity or dissimilarity matrix?

Also lets assume that I have $n$ features and they are all categorical (this is just an example) I a custom distance function where when comparing two data points $G$ and $H$, I use the formula $$\sum_i^nX_i$$
where $X_i = 1 $ if feature $i$ of $G$, $G_i$ and feature i of $H$, $H_i$ are equal to each other. So, $$X_i=1$$if and only if $G_i=H_i$ for feature $i$ for all of the $n$ categorical features. Will this result in getting a similarity or dissimilarity matrix?

Also, as mentioned above, if I want to use one of the many clustering algorithms given in $R$, such as the K-Medoids algorithm, does it require a similarity or dissimilarity matrix as its parameter?

In general does the similarity or dissimilarity matrix get used for these

A similarity is larger if the objects are more similar.

A dissimilarity is larger if the objects are less similar.

This sounds trivial, but if you get the sign wrong, you suddenly search for the worst rather than the best solution...

It's easy to see that a distance is always a dissimilarity.

K-medoids could be implemented for similarities, but I am not aware of any implementation that does not expect the data to be a dissimilarity. It may be fine to simply pass -similarity to many implementations. Because all they care for is to minimize a sum of dissimilarities, which can trivially be shown then to be equivalent to maximizing the sum of similarities.


Comparing soccer teams name Which is the best way to compare soccer teams from different sites? For example, one soccer team in a site is named Academica Clinceni and in another FC Clinceni and in  another one is Acs Fc Academica Clinceni.
Moreover, for a team can exist also the female version of the team (eg. Acs Fc Academica Clinceni (W) or Acs Fc Academica Clinceni (F)). Which is the best approach to distinguish these teams? For the first problem I used the Fuzzy String Matching, in your opinion is it right?

I don’t know why your question was voted down! Fuzzy matching is such a common challenge. The best approach I’ve seen is this one: https://towardsdatascience.com/fuzzy-matching-at-scale-84f2bfd0c536 It gives similar results to something like Levenshtein distance but it’s much faster. If you augment the matching approach with hand-coded regex features to spot e.g. female versus male teams, you should be able to match teams pretty well.


metric performances varies with each training run of my deep learning model My model (a biLSTM model in Keras) gives different results for my test metric (precision) each time I train.
My metric varies between 80% and 92%. That is really bad.
What reason besides a bad kernel initialization could this be?

That difference in results could be from high variance, the error from sensitivity to small fluctuations in the training set.

Generally, the best solution to high variance is to increase the size of the training dataset.


Model localization: one big model vs two small models Suppose I have want to predict how likely people in both US and Canada will buy product A in a month. Basically, it's a binary classification problem.

Assume I have 2 millions rows of historical data with people who has bought/not bought the product (label). One millions rows data are from US, the other are from Canada.

I can build a machine learning models in following two ways:

Build a model with 2 millions rows of data. One model with all the
data.
Model localization: Build a model with US data. Build another model with Canada data.

Intuitively, it seems to me option 1 will always performs better than 2. Because there will be more data in one model. But in practice, option 2 seems always performs better than option 1.

My question is when should I use option 2 instead of option 1? How to make the decision?

When and how to choose 1 vs 2? Data exploration, understanding the problem, and testing the models. Make sure you use metrics, a cut-off level and test data that is appropriate for your business problem.

Take for example, if there is a fundamental difference between US and CA customers. Then allowing the model to focus on the signal of each country, without dealing with the noise of the other country, may produce a better model for each. By doing data exploration (plot purchases by country over time and some other important features) you might convince yourself that it is worth the effort to test multiple models.

This is the art vs science of modeling.


Measuring the uncertainty of predictions Given a multiclass classification model, with n features, how can I measure the uncertainty of the model for that particular classification?

Let's say that for some class the model accuracy is amazing, but for another it's not. I would like to find a metric that will let me decide if, for a particular sample, I would like to "listen" to the model or not.

I thought about using prediction intervals, but am not sure how to calculate them.

The model you will decide how best to get an uncertainty. If you used Bayesian optimisation (that's a great package for it in Python), for example, you get a covariance matrix along with your expected values, and so inherently get an uncertainty measure. In this case, you can make predictions as to the underlying function of your data, and the (co-)variance will provide levels of uncertainty, as shown by the width of the green bands around the line below:

So the red points show where we have some sample data... notice that we have none e.g. at X = 4 and X = -1, which is why we have high undcertainty; the 95% confidence interval is very large.

If you use a standard deep neural network tpo perform classification, for example, there is no inherent measure of uncertainty. All you really have is your test accuracy, to let you know how well the model performs on hold-out data.  I cannot remember where it is explained, but I believe it is not actually feasible to interpret the class prediction values in terms of uncertainty.

For example, if you are predicting cat or dog for an image, and the two classes receive (normalised) logit values [0.51, 0.49] respectively, you cannot assume this means very low certainty.


Need machine learning algorithm to fill in time-series data I am currently dealing with a time-series data set with cyclical gaps every 30 minutes (30 minutes of data, 30 minutes of no data). Is there a relatively simple way of using scikit-learn (or some other Python library) to predict the missing data using the available data to train it? I assume this would involve a 'supervised training' approach? I have provided a graph for visual reference, the orange line is a 20-minute centralized moving average of the gap data set, and I want to fill in the data to look more like the green 60-minute centralized moving average from a separate complete data set. Thanks!

You can approximate a time series with a polynomial of degree n_degree by using ridge regression. You can try different degree numbers (e.g. [2,3,4,5,6]) and choose the best one. Keep in mind that higher degree models, always get lower error values. So you should somehow penalize higher degrees.

from sklearn.linear_model import Ridge
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline

for count, degree in enumerate([2, 3, 4, 5, 6]):
    model = make_pipeline(PolynomialFeatures(degree), Ridge())
    model.fit(Time, y)

More details here.


How to insert two features in a model when a feature only applies to a certain group in the model I'm building a machine learning model in Python to predict soccer player values. Consider the following feature columns of the dataframe:
         [features]
---------------------------------
position | goals | goals_conceded
-------- |-------|---------------
Forward  |  23   |     NaN
Defender |   2   |     NaN
Defender |   4   |     NaN
Keeper   |   NaN |     20
Keeper   |   NaN |     43

Since keepers don't usually score goals, they'll almost always have null values in the "goals" column, but they still can have this statistic, so it would be fine to fill the NaNs with 0. On the other hand, since line players can't have "goals_conceded" stats, they'll also have null values in that column, but in this case, players will never have this statistic, since this is a keeper only stat. How do I build a machine learning model considering these two columns as features?
I thought about putting them together in one single column, but that can't happen since for a line player, the more goals he makes the better it is. For goalkeepers it's the opposite, the less goals he conceeds the better. I also can't fill the columns with zeros since it would affect the model prediction in the "goals_conceded" column for example, since 98% of the rows contain in line players info.
This happens with many of the columns in my dataframe, such as "clean sheets" (only Keepers will have this stat) and "shots at target" (only line players will have this stat). How do I deal with them?

Remember that any model will have the relationships between the features (X) and the label (y), so your ideas are correct. What is better will depend on your data, so try both, and try more ideas if you can. The best model will be the one that generalize better for new data, so take the final decision with data.
Please, note that your features are at least 3, goals, goals_conceded and position, so this last will help with the problem.
And I suggest not to fill NaN with 0, but -1 (or any other imposible/invalid integer number), because a newbie player will have 0 in the feature(s), and it's correct, and you want to identificate this.


making conclusions after sentiment analysis After performing some sentiment analysis, I have a dataset that looks like this:
For different products, using online reviews, I have obtained some values for positive/negative sentiments. However, now I am unable to figure out how to draw conclusions for this.
I had the idea of using correlation but need ideas on what features could be created & what comparisons could be made?
The dataset includes different "Features" like webcam, screen, mousepad for different products (product name).
id     Date       Website       Product Name    Brand   Stars   Feature   Sentiment  Positive     Negative       Anger      Happiness       Annoyance
0     2020.8.03   eBay    Lenovo Hi-Fi 320    Lenovo     4      Screen      NEGATIVE    0.000047    0.999851    0.000101    0.108132    0.248220

Depends what your Goal is.
But generally you see that positive and negative Sentiment probabilities are disjunct. Meaning your model focuses for one class only, negative one. ANd thats it, you conclude your data sample belongs to class "Negative". What that means, depends on the Definition of this class.


How does imblearn apply the transformations during prediction? Let's say I have a sklearn pipeline that:

Imputes the data
Randomly oversamples the minority class

from imblearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from imblearn.over_sampling import RandomOverSampler

pipeline = Pipeline(
    [('1', SimpleImputer(strategy='median'),
     ('2', RandomOverSampler(random_state=0)),
     ('estimator', <Some Logistic Regression>)
    ]
)

I can then fit this to my training set pipeline.fit(X_train, y_train) and the random oversampler should properly identify the class to sample. What if I try to predict i.e pipeline.predict(X_test)? Since there are no classes, does the random oversampler still apply? I would expect the imputer to apply regardless but what about RandomOverSampler?

Thank you

You generally shouldn't apply resampling to the test set (although there are some differing opinions on whether to do so on various levels of validation data).  imblearn has its own version of the pipeline to accomplish this; in particular, the pipeline docs say:

  The samplers are only applied during fit.


Handling actions with delayed effect (Reinforcement learning) I am working on a problem where the action that I learn (using DQN) can be executed 'now' but it's effect on the environment is delayed by 'T' units of time.

The environment however is active in that time and there are other conditions based on which rewards are computed and returned. How is this handled?

I believe the Q value function (with gamma) handles 'delayed' effects of rewards but not actions.

This is similar to the inventory management use cases. As an analogy consider that I sell cakes. As customers walk into my shop I consume cakes off the shelf. I must reorder to stock my shelf BUT this reordering can take time to take effect.

I thought of just adding the quantity reordered to the shelf at a later time and let the agent learn it's effects. Will this suffice?

As another approach I thought of Experience and Replay as a mechanism to handle this delayed effect.

Appreciate the help.

I thought of Experience and Replay as a mechanism to handle this delayed effect.

Experience Replay will not help directly here, it is a mechanism to make learning more stable and effective from limited data, but it will not address delayed returns.

Any approach which learns Q should cope fine with delayed rewards. So Monte Carlo Control, SARSA, Q-learning, DQN and all their variants are in theory capable of learning the delayed reward. The total reward is the ultimate effect of any action.

However, I'm guessing just because it is not mentioned and you are asking this question, that you may still have a problem . . .

You claim that the action has a "delayed effect on the environment". You need to change that, because it implies that your state representation is incorrect for your problem. Clearly making an order for something changes the environment. An environment with an order in progress is different, in a way that is critically important to your problem, than one without an order in progress.

Probably what you are missing is a state representation that captures what your action has actually done. Without that, there are hidden variables (orders currently being handled) that a Q function cannot learn about because it is not in the state, $s$ for $Q(s,a)$. For RL to be reliable, the state value $s$ has to capture all relevant information about how future state changes and rewards will progress. For instance, if you were writing an agent to control a swinging pendulum (a standard toy problem), then you don't just need the position of the pendulum, you also want its velocity in order to predict where the pendulum might end up before any action is taken. For your issue, the current stock is like the pendulum's position, and you need to track orders "in progress" as they are a bit like velocity in that they will cause further changes to state regardless of action.

To address this, you will need to add a representation of recent stock orders to your state. I would guess something that describes the contents of the order (similar to your current stock representation) and a countdown for how many days left until it is delivered (or if they are not that reliable, something similar that correlates with likely arrival time). That state should be changed immediately in response to the action that caused the order, otherwise the agent will not learn the association between action and its effects, and will treat orders arriving as some kind of random effect from the environment.

Once you have a representation like this, then I think that is enough, and the next steps are finding the right hyper-parameter values to learn effectively. This will be easier initially if you have a simulated environment, although you could also learn off-policy from historical data using something like DQN, provided you have a lot of historical data to work with.

  I thought of just adding the quantity reordered to the shelf at a later time and let the agent learn it's effects. Will this suffice?

I think this is similar, but probably not enough. The most important thing to do is associate change to state with the action that caused it. Adding the new stock as it arrives you should do anyway, but the agent needs to learn that the earlier action is what caused this, and that is only possible if that action actually changes the state in some way.


How is learning rate calculated in sklearn Lasso regression? I was applying different regression models to Kaggle Housing dataset for advanced regression. I am planning to test out lasso, ridge and elastic net. However, none of these models have learning rate as their parameter.

How is the learning rate calculated for these model? Is it dependent on the dataset being trained? I know these models are regularized linear regression and must use learning rate to update their model weights. Or is their a different way to update the model?

With sklearn you can have two approaches for linear regression:

1) LinearRegression object uses Ordinary Least Squares (OLS) solver from scipy, as Learning rate (LR) is one of two classifiers which have closed form solution. This is achieve by just inverting and multiplicating some matrices.

2) SGDRegressor which is an implementation of stochastic gradient descent, very generic one where you can choose your penalty terms. To obtain linear regression you choose loss to be L2 and penalty also to none (linear regression) or L2 (Ridge regression)

There is no "typical gradient descent" because it is rarely used in practise. If you can decompose your loss function into additive terms, then stochastic approach is known to behave better (thus SGD) and if you can spare enough memory - OLS method is faster and easier (thus first solution).

This answer is mostly extracted from https://stackoverflow.com/questions/34469237/linear-regression-and-gradient-descent-in-scikit-learn-pandas/34470001#34470001
The OP @Netro is and the answer is from @lejlot


Making bigram features from a particular dataset I have a folder which has a number of files which have a format like these

madvise
write
write
write
write
read
read
madvise
ioctl
ioctl
getuid
epoll_pwait
read
recvfrom
sendto
getuid
epoll_pwait

that is it is a set of words which repeat.This is how all the files are like.
Now I have created a feature vector table using unigram that is each word becomes a feature and each file becomes a row where I put the frequency of that word occuring in the respective columns.

Now I want to create a similar FVT using bigrams.I was wondering how to do that in this case.

Bigram is better to be used with sentences. In your case, files contain a list of words, as I could understand. Therefore using bigrams in your project might not yield expecting results. However, if you are still willing to do that, this is how you calculate bigrams:

Take the list of words and count the frequencies of adjacent words.
Ex:

(madvise, write) - 1

(write, write) -  3

(write, read) -  1

(read, read) -  1

(read, madvise) -  1

.

.

.

.

(sendto, epoll_pwait) -  1


Choosing the first node in a decision tree, basic example I'm wondering whether I'm understanding the process of choosing a node correctly and would like to see if this example makes sense.
using the following data :

> df
  Y A B C
1 1 0 1 1
2 0 0 0 1
3 1 1 1 0
4 1 0 1 0
5 1 1 0 1
6 1 0 0 1

I will split the data on A,B,C and evaluate the entropy of each split, where the entropy is computed using

$$
p \; \log_2(p) + (1-p)\; \log_2(1-p)
$$

where $p$ is the proportion of successes for a particular split.

When splitting on A I have

  Y A B C
3 1 1 1 0
5 1 1 0 1

and

  Y A B C
1 1 0 1 1
2 0 0 0 1
4 1 0 1 0
6 1 0 0 1

For $A = 1$ (the first data table) I have $p = 1$, then entropy is 0.

for $A=0$ (second table) I have $p=0.75$ and entropy 0.81

So for splitting on $A$ I would state that the entropy is

$$
0 + 0.81 = 0.81
$$

This is then carried out in a similar manner for $B,C$.

For $B=1$ I find $p=1$ so entropy = 1

For $B=0$ I find $p=0.66$ so entropy = 0.91

Then the entropy for splitting on $B$ is

$$
0 + 0.91 = 0.91
$$

For $C=1$ I find $p=0.75$ so entropy = 0.81

For $C=0$ I find $p=1$ so entropy = 0

Then the entropy for splitting on $C$ is

$$
0.81 + 0 = 0.81
$$

Given the above the split which has the highest entropy is $B$, therefore I would choose to split on $B$ first.

I now have a decision tree with one node, and need to select the next node for each branch of $B = 1$ and $B=0$.

This selection is carried out in the same manner as the above.

Is the computation and reasoning in the above valid?

Some corrections in the above:

While calculating entropy of a split, you should take into account the weights -  fraction of instances in each branch:

$$
entropy(A) = 0 * (2/6) + 0.81 * (4/6) = 0.54\\
entropy(B) = 0 * (3/6) + 0.91 * (3/6) = 0.45\\
entropy(C) = 0 * (2/6) + 0.81 * (4/6) = 0.54
$$

While choosing an attribute to split, you should choose the one with lowest entropy after the split. Entropy is the uncertainty, so you want to minimise it. As an example, if the split was perfect (all 0's on one side, and all 1's on the other), then entropy will be 0.

After selection of one node, the tree is built out recursively, as you mention.


Separating styles of numbers for simple digit classification I am just getting started with my first simple digit classifier, so my doubts are at a pretty low level. In every dataset of digit images I've seen so far, different variants of each digit are grouped together, for example:

All of these images represent the number 1, but are fairly different in looks. Won't simple convolutional neural networks have a hard time learning the visual pattern for 1 in such a case? Especially considering how the third image is similar to 7 in design.
My questions are these: Would it be better to create other labels such as "1", "1-alt", "1-serif" etc? The CNN can then add the probabilities of the image being a variant of 1 and then give its prediction, but I'm not sure about this.
How do professional classifiers approach this problem?
Theoretically, will this method affect performance or accuracy in any way?

Interesting question. You are right in assuming that some 1s may be confused for 7s, same with 8s and 3s for instance.
Generally creating different classes as you suggest doesn't happen, simply because it would require more annotation.
There are multiple ways to handle this.

stacked models

Anything labelled as a 1 or a 7 would be given to a model fine-tuned to differentiate these two specific digits

weighted training

It is possible to teach your model that some mistakes are worse than others. In your case, 1s for 7s is bad, so you could increase the cost of making that mistake.


CNN always predicts either 0 or 1 for binary classification I am using a Kaggle dataset on stress characteristics, derived from ECG signals, and I would like to train a CNN to recognize stress/non-stress situations.

I have built a model in Keras:

model = tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(U1, (3, 1), activation = 'relu', input_shape = (num_features, 1, 1)),
    tf.keras.layers.Conv2D(U2, (3, 1), activation = 'relu'),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(U3, activation = 'relu'),
    tf.keras.layers.Dense(1,  activation = 'sigmoid')
])

where U1, U2 and U3 are the parameters I have been changing to find the right combination to ensure the best performance.

What I did is, specifically:

divide the samples in training and test set (I don't have a validation set as the number of available samples is small);
normalize both training and validation set by dividing them by the max value found in the samples;
train the network with various combinations of U1, U2 and U3 to find the ones ensuring the best performance.

The training is done as follows:

model.compile(loss = 'binary_crossentropy' , optimizer = 'adam' , metrics = ['accuracy'])
history = model.fit(x_train, y_train,
          epochs = 50,
          batch_size = 160,
          validation_data = (x_test, y_test))
score = model.evaluate(x_test, y_test, batch_size=128)

The network performs really well on both the training and test set, achieving an accuracy of 99.3% on test set (and 99.5% on training set).

However, when applied on real data (by taking one's ECG, computing the features and normalizing them by the same normalization value used on training and test set above), the network is always predicting:

a label of 0.0 for "normal" ECGs;
a label of 1.0 for noisy ECGs (which are taken as stressed ECGs).

It bugs me that no other labels rather than 0.0 or 1.0 are never returned. It is true that sometimes the network predicts labels of exactly 0.0 or exactly 1.0 also for samples in the training and validation set, but never seeing a label different from 0.0 and 1.0 in real-world data sounds strange.

Is this a problem of the network? Or a problem of the dataset? Maybe could this be related to the fact that the real-world ECG data I am using are not extracted from the same distribution of the Kaggle dataset? I see that, for instance, real-world data have profound differences in values with respect to the data in the Kaggle dataset (so much difference that even after normalization values are not really normalized), but I don't know if this is a valid reason that justifies the problem I see.

Sigmoid functions might have saturation problems. The values that it receives are probably too far from zero, and the sigmoid is returning 'extreme' results (i.e.: 0 or 1). I suggest you to keep your signal zero centered by putting BatchNormalization() layers between each of your layers (certainly between Dense() ones, but also between conv layers if you prefer).

Additionally, sigmoid functions are thought to perform not very well in general. The main configuration for classifiers is to put one output node for each target category (in this case, 2) and use softmax activation. Your loss at this point would be BinaryCrossentropy(). This requires one-hot encoding of your target variable.

I would also try other regularization tricks, such as dropout and l1-l2 regularization, or different initializations. There no rule of thumb here, you have to play with these hyperparameters and figure out what's the best configuration for your task.


Understanding the layers in CNN I am new to NNs and I have a question about the convolutional layers in CNN.
A convolution layer is said to perform feature extraction or work as a feature extractor in CNN. The first convolutional layer learns to extract low-level features which means that the first layer will convert the original image into multiple copies of the original image (depending on the number of filters used) that contain only the low-level features, neglecting any other features in the original image, right ?
How does the next layers extract other features high level features i.e. faces if they were already neglected by the first layer ?
Why don't we have several separate layers, each of which does a specific extraction of some features and once all the needed has been extracted, the output is merged before the fully connected layer ?
I don't get the point of applying different filters (after the first layer)on the entry already filtered.

This is a good question to ask as it pertains to important neural network concepts.

"Why don't we have several separate layers, each of which does a
specific extraction of some features and once all the needed has been
extracted"

The answer is two-fold.
First, the layers depend on each other. The reason deeper layers can extract higher-level features is that they build upon the lower features extracted in the earlier layers. The layers may be different in architecture but they could simply be the same layer duplicated multiple times and initialized with the same weights.
Second, it is important to understand that the layers are not "told" what type of features to extract. Realizing that each layers represented various level of features was very important in better understanding the behaviour of neural networks and to confirm that no, it isn't all black magic. And it is important to understand that is a learning artefact which may not always occur depending on the network, data, and training strategy involved.


Loss value going down while accuracy remains constant? While I am training, it seems like my loss is going down, but my accuracy remains constant throughout training. It always seems to go towards 0.0023 no matter how I tweak my network, input data length, etc.

My input data is a sequence of sample levels before a current sample level. So basically predict the current sample level based on the previous sample levels.

Network is Sequential:

Dense(seq_length)
LSTM(seq_length*10)
Dense(1)

Validation set is 0.33 of the total input samples.

Using Tensorflow Keras on Windows.

CSV of loss, accuracy, val loss, etc:
https://pastebin.com/GPsmeUmg

Everything is fine, loss is going down, your model is learning. The problem is that you are using accuracy on a regression problem.

Accuracy should only be used for classification. It is a very common mistake to make when starting out. If you want another metric then you could use mean_absolute_error which, as the name suggest, tells you how large error your model makes on average.

The reason accuracy does not work is because it only works with exact matches where you have y_true == y_pred. In regression you are working with continuous values which makes this quite rare and will not have much to do with the performance of your model.

The reason you get any accuracy at all is likely because Keras does y_true == round(y_pred), rounding the model prediction. Otherwise accuracy would almost always be zero since the model will never get the same decimals. This question has a good answer by Esmailian that goes a bit more into details on this.


Splitting train and test in multilabel classification to avoid missing data in the train set I have a dataset (600 rows) composed of two columns:

-Summary: which contains the text of a document

-Keywords: which contains the keywords that identify that document.

                                              Summary                     KeyWords_in_Array_wo_insurance
0    court sanction scheme transfer insur reinsur b...                                       [insolvency]
1    immedi custodi sentenc month week impos direct...  [administration of justice, civil evidence, se...
2    motorist injur hit run collis car identifi dri...                         [negligence, road traffic]
3    claimant given permiss continu claim compani a...                      [insolvency, civil procedure]
4    court gave guidanc approach taken applic relea...                           [civil procedure, costs]
5    plaintiff solicitor entitl declar life critic ...                                           [trusts]
6    claimant insur establish requir standard road ...                           [personal injury, torts]
7    minimum indemn requir institut charter account...  [arbitration, civil procedure, costs, accounta...
8    applic secur cost court could take account eve...          [civil procedure, insolvency, cpr, costs]

I want to classify a summary in specific keywords. The keywords are not mutually exclusive.

My code is:

X_train, X_test, y_train, y_test = train_test_split(df_final["Summary"], df_final["KeyWords_in_Array_wo_insurance"], test_size=0.20, random_state=42)

mlb = MultiLabelBinarizer()
y_train_mlb = mlb.fit_transform(y_train)

classifier = Pipeline([
    ('vectorizer', CountVectorizer()),
    ('tfidf', TfidfTransformer()),
    ('clf', OneVsRestClassifier(LinearSVC()))])

classifier.fit(X_train, y_train_mlb)
y_predicted = classifier.predict(X_test)
all_labels = mlb.inverse_transform(y_predicted)

y_test_mlb = mlb.transform(y_test)

print("Accuracy = ", accuracy_score(y_test_mlb,y_predicted))

I am getting a low accuracy score: 20%

Therefore I am thinking that my classification is not good enough.

The reason it might be that some keywords are used only once. For example, the keywords "animal" or "partnership" or "succession" are used only in 1 row. (Meaning, they are assigned to only one summary)

I think, therefore, that when I split the dataset in training and test, some "lonely keywords" enter the test dataset but not the training one. Thus, the model will never be trained on them.

Is this the reason why my accuracy is so low?

Or am I doing something else wrong?

Accuracy is an awkward measure to use to assess a model predicting classification into multiple classes, and rare events are hard for models to predict well. Outcome categories that exist in your test set but not your training set will of course lead to lower accuracy, but it's far from the only thing that can go wrong.

And, if these outcomes are rare relative to the number of observations you have, it's unlikely that your model is highly predictive but reduced to 20% accuracy based solely on the failure to classify a handful of rare events. It's more likely that the model has fairly poor performance overall. Importantly, this is not necessarily the result of you doing something wrong. Some things are difficult to model, and the best-performing model possible may still have a poor success rate (however you want to measure that).

Without knowing more about your modelling procedure it's hard to make specific suggestions, but a few things come to mind (this is nowhere near an exhaustive list):

Summarizing the documents may not be a very reliable process.

If you have strong protocols for summarizing each, reviewing those might help to find odd edge cases where similar observations get summarized differently in some systematic way, or differing observations being summarized together improperly. If you are manually summarizing each document, it may be worth having multiple people perform the summarization and then evaluation the inter-rater reliability of the summaries.

The documents from which the summaries are derived may differ from one another.

If we're describing highly formulaic documents this might not be an issue. But if the documents are written in prose by different people, it's very possible that issues of personal style and preference on the part of the writers is producing systematic differences from other writers, making it hard to classify a document from an imaginary, generic author.

There could be a mismatch between the number of predictors and outcome classes and the number of observations available to train the model on.

600 observations may be enough for a model classifying a document into one of three categories based on the presence or absence of 15 predictor variables. It is not enough for a model classifying a document into one of 400 categories based on 600 predictor keywords.


Smoothing Proportions :: Massive User Database What are some possible techniques for smoothing proportions across very large categories, in order to take into account the sample size? The application of interest here is to use the proportions as input into a predictive model, but I am wary of using the raw proportions in cases where there is little evidence and I don't want to overfit.

Here is an example, where the ID denotes a customer and impressions and clicks are the number of ads shown and clicks the customer has made, respectively.

A simple way would be to consider Laplace Smoothing (http://en.wikipedia.org/wiki/Additive_smoothing ) or something like it.

Basically, instead of calculating your response rate as (Clicks)/(Impressions) you calculate (Clicks + X)/(Impressions + Y), with X and Y chosen, for example, so that X/Y is the global average of clicks/impressions.

When Clicks and Impressions are both high, this smoothed response rate is basically equal to the true response rate (signal dominates the prior). When Clicks and Impressions are both low, the this smoothed response rate will be close to the global average response rate - a good guess when you have little data and don't want to put much weight on it!

The absolute scale of X and Y will determine how many data points you consider "enough data". It's been argued that the right thing to do is set X to 1, and Y appropriately given that.


Subtracting grand mean from train and test images I am building an image classifier based off the VGG_face keras implementation. It is easiest for me to extract a csv file full of the representations and then try classifiers on those representations. When I got the representations, I first subtracted the mean of the entire dataset from each image. Then I realized... am I cheating, so to speak? In other words, since I included the test images when calculating the grand mean to be subtracted, does this now then overestimate my accuracy measurements?

There is a kind of bias that you are introducing, yes. You are basically extracting some statistics (i.e. the mean) from your hold-out set and using that to train, which makes your final claims of accuracy a little weaker (some people might say they are useless).

The general approach is to compute the mean of your training data, then you may subtract that from all of the data, including hold-out data.

You can do the mean subtraction, in general, using something like the ImageDataGenerator. The mean to be subtracted can be computed using all or some of the training data. That class also offers other augmentation functionalities, such as normalising the dataset too, adding rotations etc.

you mentioned you read features from a CSV file, so if you are not talking about images, as long as you can use e.g. NumPy, you can perform is manually on all data at the beginning.


Why are variables of train and test data defined using the capital letter (in Python)? I hope this question is the most suitable in this site...

In Python, usually the class name is defined using the capital letter as its first character, for example

class Vehicle:
    ...

However, in machine learning field, often times train and test data are defined as X and Y - not x and y. For example, I'm now reading this tutorial on Keras, but it uses the X and Y as its variables:

from sklearn import datasets

mnist = datasets.load_digits()
X = mnist.data
Y = mnist.target

Why are these defined as capital letters? Is there any convention (at least in Python) among machine learning field that it is better to use the capital letter to define these variables?

Or maybe do people distinguish the upper vs lower case variables in machine learning?

In fact the same tutorial later distinguish these variables like the following:

from sklearn.cross_validation import train_test_split

train_X, test_X, train_y, test_y = train_test_split(X, Y, train_size=0.7, random_state=0)

The X (and sometimes Y) variables are matrices.

In some math notation, it is common practice to write vector variable names as lower case and matrix variable names as upper case. Often these are in bold or have other annotation, but that does not translate well to code. Either way, I believe that the practice has transferred from this notation.

You may also notice in code, when the target variable is a single column of values, it is written y, so you have X, y

Of course, this has no special semantic meaning in Python and you are free to ignore the convention. However, because it has become a convention, it may be worth maintaining if you share your code.


How to perform one hot encoding on multiple categorical columns I am trying to perform one-hot encoding on some categorical columns. From the tutorial I am following, I am supposed to do LabelEncoding before One hot encoding. I have successfully performed the labelencoding as shown below

#categorical data
categorical_cols = ['a', 'b', 'c', 'd']
from sklearn.preprocessing import LabelEncoder
# instantiate labelencoder object
le = LabelEncoder()
# apply le on categorical feature columns
data[categorical_cols] = data[categorical_cols].apply(lambda col: le.fit_transform(col))

Now I am stuck with how to perform one hot encoding and then join the encoded columns to the dataframe (data).

Please how do I do this?

LabelEncoder is not made to transform the data but the target (also known as labels) as explained here. If you want to encode the data you should use OrdinalEncoder.

If you really need to do it this way :

categorical_cols = ['a', 'b', 'c', 'd']

from sklearn.preprocessing import LabelEncoder
# instantiate labelencoder object
le = LabelEncoder()

# apply le on categorical feature columns
data[categorical_cols] = data[categorical_cols].apply(lambda col: le.fit_transform(col))
from sklearn.preprocessing import OneHotEncoder
ohe = OneHotEncoder()

#One-hot-encode the categorical columns.
#Unfortunately outputs an array instead of dataframe.
array_hot_encoded = ohe.fit_transform(data[categorical_cols])

#Convert it to df
data_hot_encoded = pd.DataFrame(array_hot_encoded, index=data.index)

#Extract only the columns that didnt need to be encoded
data_other_cols = data.drop(columns=categorical_cols)

#Concatenate the two dataframes :
data_out = pd.concat([data_hot_encoded, data_other_cols], axis=1)

Otherwise :

I suggest you to use pandas.get_dummies if you want to achieve one-hot-encoding from raw data (without having to use OrdinalEncoder before) :

#categorical data
categorical_cols = ['a', 'b', 'c', 'd']

#import pandas as pd
df = pd.get_dummies(data, columns = categorical_cols)

You can also use drop_first argument to remove one of the one-hot-encoded columns, as some models require.


Choosing layer hyper-parameters of a CNN Context: I'm building a CNN on MATLAB to classify wallpaper groups.

I'm using the following network type.

  CONV -> ReLU -> POOL -> CONV -> ReLU -> POOL -> FC -> DROPOUT -> FC -> SOFTMAX

Should the parameters for the first CONV or POOL layers be the same as the second (or later) CONV or POOL layers succeeding it? If not, is there a meaningful way to choose parameters for the repeated succeeding layers?

For example, for the first CONV layer you may pick a 5x5 filter. Perhaps for the second CONV layer it's more meaningful to pick a certain different size filter?

Finding an appropriate architecture is somehow practical. Those hyper-parameters you are talking may be different. Try to use a base architecture and then train your model. If it does not learn your model try to change the hyper parameters. It is an iterative operation to find a good model. There are a lot of debates, but there is not exactly consensus for making good models in CNNs. You can think of if your model does not learn your data, it needs more features to be learned by the CNN. The behavior of the layers are somehow as follows:

Max Pooling

It is used for adding spacial invariance to the inputs of its layer. It is also used for decreasing the input size. By increasing its size, the two mentioned behavior would increase.

Convoluional Layers

These layers are used for extracting features to reduce the cost function in order to learn data. If you increase the size of these layers they behavior would be more vast, they would find features of a larger region.

Take a look at here and here.


Checking Correlation of Categorical variables in SPSS I am building a predictive model for a classification problem using SPSS. Of the Independent variables, I have both Continuous and Categorical variables. SPSS gives only correlation between continuous variables.

Is there a way in SPSS to find the correlation

a)between two categorical variables
b)between categorical and continuous variables?

For testing the correlation between categorical variables, you can use:

binomial test: A one sample binomial test allows us to test whether the proportion of successes on a two-level categorical dependent variable significantly differs from a hypothesized value.  For example, using the hsb2 data file, say we wish to test whether the proportion of females (female) differs significantly from 50%, i.e., from .5.  We can do this as shown below.
chi-square test: A chi-square goodness of fit test allows us to test whether the observed proportions for a categorical variable differ from hypothesized proportions.  For example, let's suppose that we believe that the general population consists of 10% Hispanic, 10% Asian, 10% African American and 70% White folks.  We want to test whether the observed proportions from our sample differ significantly from these hypothesized proportions.

You can find the code for doing above analysis from the link below:

Source: http://www.ats.ucla.edu/stat/spss/whatstat/whatstat.htm

I'm not sure about comparing numerical and categorical variables but one thing you can do is to transform the numerical variable into categories. For example, if it is age, then, you can transform it to: [0,10), [10,20), ... , [90,100), so that you can compare the correlation of age with other categorical variables.

Hope it helps.


Siamese networks vs Semantic similarity (may be gensim) I am trying to understand the Siamese networks . In this vector is calculated for an object (say an image) and a distance metric is applied (say manhatten) on two vectors produced by the neural network(s). The idea was applied mostly to images in the tutorials provided on internet.

If I compare it with Gensim semantic similarity, there also we have vectors of two objects (words or sentences) and then do a cosine similarity to calculate the difference. (remember example of King-man+woman=Queen).

Am I missing some aspects of Siamese networks or these are actually same?

Siamese and semantic similarity are not same.
See example here:
Siamese are effectively used for image classifications , but there is a paper publication by Cornell university for text classification as well.
Now for your question, we train siamese in a way such that one is positive and one is negative.
Lets talk about job title called java developer , we train it with java developer as input and j2ee developer as output in one network. In another network we train java developer as input and QA engineer as output but we dont change weights of the network.

Since we train a neural network with positive and negative so that siamese networks learns the positives and hence its also called one shot learning etc..

Now coming back to semantic similarity- its a unsupervised learning wherein the network will place similar objects together, the cosine distance that we get from gensim is how close one object to the other.


Scaling the activation function It is obvious that I have to scale the output data if the range of values is between say [-10;10] and the activation function of the output layer takes values in the interval [-1;1]. But I could also scale the activation function by multiplying it with the factor 10 instead. It seems to me that it is more common to scale the data and not the function. Is there a reason for it?

All it does is make a difference in the step size of the learning step. Scaling the outputs down makes the gradients smaller, so the gradient descent updates are smaller. We want that so we do not jump over a good solution with bigger steps.

Let's say the activation function is $a =\sigma(x)$ where $x$ is the input to the activation function and $a$ is the output.
The range of true outputs $y$ is on the order of 10 times of the range of $\sigma$. We can either scale $\sigma$ up, or scale $y$ down.
Scaling activation function
Lets scale the output by 10: $y' = 10a$.
The loss function $L$ compares predictions vs. truth: $L(y', y)$. We need the gradient w.r.t $x$ for backpropagation. That is, to find how the loss changes with respect to input so we can change the input:
$$
\frac{\partial L(y', y)}{\partial x} =
\frac{\partial L(y', y)}{\partial y'} \times \frac{\partial y'}{\partial a} \times \frac{\partial a}{\partial x} \\=
\frac{\partial L(y', y)}{\partial y'} \times 10 \times \frac{\partial a}{\partial x} \\ =
\frac{\partial L(10a, y)}{10\cdot \partial a} \times 10 \times \frac{\partial a}{\partial x} \\=
\frac{\partial L(10a, y)}{\partial a} \times \frac{\partial a}{\partial x} \\=
\frac{\partial L(10 \cdot (a, 0.1y))}{\partial a} \times \frac{\partial a}{\partial x}
$$
Scaling output
On the other hand, lets scale down to data, so $y_s = 0.1y$. This means we do not need to scale $a$. The loss function gradient now is:
$$
\frac{\partial L(a, y_s)}{\partial x} = \frac{\partial L(a, 0.1y)}{\partial a} \times \frac{\partial a}{\partial x}
$$

Now for both cases, note the final forms of the gradient. The only difference is the arguments of $L$. For the case where the output was scaled, the arguments of the loss function are 10 times smaller. Which means that the gradient will be smaller. Which means that the step update made to $x$ will be smaller. We usually want small updates so we can converge to an optimal solution.
But also note, we can make the step size smaller anyways by reducing the learning rate too.
So scaling the output down instead of the activation up is a nice rule of thumb to get better convergence. It is not a rule.


Experience replay in Reinforcement learning - Batch Size I am starting to understand Reinforcement Learning playing with the GYM Cart-pole environment. I would like to ask if the experience replay can slow down the training and if there is a maximum of the batch size?

Maybe I misunderstood, and when the program starts the fit function on the replay I guess it is not sending the action to the agent, but the environment is still running. Not sure if I missed a point or if the fit function run so quickly that the agent will have plenty of time to catch up with the environment for the next action during training.

Definitely using experience replay can slow down the agent processing each time step, because typically on each time step, a result is stored (possibly requiring another result to be deleted), a mini-batch number of results are chosen randomly and fetched, then the function approximator (usually a NN) has to be run forward for the mini-batch to discover max Q values (potentially multiple times per item if the NN uses action as input instead of multiple action values as output). Then it has to run a forward pass to calculate error values and back propagation and weight updates over the mini-batch. All that can take considerable amount of CPU and memory bandwidth compared to the same system running without experience replay.

However, there are a lot of factors to consider whether this is important. Here are a few that I can think of:

Typically the environments using DQN agents are not training in real time. Often they run multiple times the speed of real time in order to collect data quickly and solve the environment faster.
Obviously something without the overhead of experience replay could collect more data in the same real time if the environment is simulated, but actually the stabilising effect of experience replay is more important and the effective "wall clock" speed of learning is faster and more reliable, using less game play experience.
Due to the way mini-batch processing is parallelised for neural networks on GPUs, there is not such a high overhead for increasing mini-batch size as you may think. At least up to the limits of GPU. There's still a cost for memory transfers etc per item, so it's not completely free.

  when the program starts the fit function on the replay I guess it is not sending the action to the agent,

That is correct, the agent is updating its internal estimates of values in order to learn the optimal control actions for later actions. The mini-batch actions are not taken, just used to refine the estimates.

  but the environment is still running.

That is only correct in real-time systems. The Gym environments are not real-time as far as I know. They all run at the speed that the agent wants them to - they are effectively a turn-based game for the agent to play, although many of them are based on systems that run at a specific real-time rate. However, real-time reinforcement learning and control systems are a thing in other domains. If learning must be done online in such a system, then typically you have some choices on how to design the system:

Have fixed time steps, and allow for maximum computation time between each one (with perhaps a margin of error).
Asynchronous interactions with the environment - with fixed time steps for sampling observations and taking actions - and learning continuously sampling from recent history and updating value functions and/or policy.
Allow for variable length time steps. I don't know much about this approach, but it is an option.

Which to use depends on the problem, and a bit of guesswork.


Why Davies-Bould chose a number ob cluster higher than Silhouette or Calinsky Harabasz? I am doing use several metrics in order to know what number of clusters is correct in order to do this I selected 3 clustering algorithms and 3 internal evaluation metrics, Silhouette, Calinsky Harbasz and Davies Bouldin.
The results of this was the following:
      S    CH    DB
Kmean 3    3     9
Agglo 2    2     9
Gauss 3    3     10

The original dataset has 3 groups, and in general S and CH works well, the questions is what DB always returns a high value for the number of clusters?
Thanks

Each clustering evaluation metric follow different ideologies:
Silhouette analysis can be used to study the separation distance between the resulting clusters.Silhouette coefficients near +1 indicate that the sample is far away from the neighboring clusters. A value of 0 indicates that the sample is on or very close to the decision boundary between two neighboring clusters and negative values indicate that those samples might have been assigned to the wrong cluster.
Calinski–Harabasz rewards clusterings in which the cluster centroids are far apart and the cluster members are close to their respective centroids.
Davies-Bouldin is defined as the average similarity measure of each cluster with its most similar cluster, where similarity is the ratio of within-cluster distances to between-cluster distances. Thus, clusters which are farther apart and less dispersed will result in a better score. It prefers clusters equally-distanced from each other.
We can't really say what clustering quality measure is good or not.It depends on what you want to evaluate.You have to look it seems relevant for the kind of clustering you are doing.
For more info refer this discussion.


Working Behavior of BERT vs Transformers vs Self-Attention+LSTM vs Attention+LSTM on the scientific STEM data classification task? So I just used BERT pre-trained with Focal Loss to classify Physics, Chemistry, Biology and Mathematics and got a good f-1 macro of 0.91. It is good given it only had to look for the tokens like triangle, reaction, mitochondria and newton etc in a broader way. Now I want to classify the the Chapter Name also. It is a bit difficult task because when I trained it on BERT for 208 classes, my score was almost 0. Why? I an get that there are lots of information also like nacl: sodium chloride , bohr model 9.8 m/sec etc which I think BERT is not trained for. I want to ask few questions.

Is BERT useful in these conditions? Is it trained on scientific terms. I mean can it get the context of Schrödinger equation to Plank's Constant? If not, I don't think I should use it because I don't have enough data to re-train BERT. Anything but BERT
Can I use FastText or GloVe? Cn they get the meaning or context?
Or should I simply create my own embeddings in pytorch/keras and keep nacl,fe,ppm as they are and hope either of Transformer or Attention will capture it?

Please help. I have a data of 120K questions/data points.

A1. BERT by itself may not be useful for scientific terms. You have 2 choices, either find a pre-trained embedding specific to the scientific text database or use transfer learning and build upon BERT.
A2. Fastext or Glove will have the same issue as BERT
A3. You have mentioned that BERT works well at a high level. So I would not advice you to create embeddings from scratch and instead use transfer learning to enhance the embeddings


What does it mean when the model Learning Curve displays this behavior? I made a classification LSTM model and it seemed to be working as intended until I increased the number of epochs. That's when I noticed that the Validation and Training curves cross each other at a certain point:

I have never seen this kind of curve... also couldn't find anything like it on my searches.
Does it mean my model is over-fitting?
Should I worry with this behavior?
What can I do to avoid this?
Thanks.

Yes, this is overfitting. As you can see in your loss curve the training loss is steadily decreasing like it should, but at the same time your validiation loss is increasing. On data the model hasn't seen before it's actually getting worse!
This means your model is starting to learn the training data, and does not instead keep the learning generalized. As soon as validation loss is increasing you should stop training.
I'm answering this question although overfitting is a general concept and has been answered before because this question might want to have an answer that is specific to LSTMs.


LSTM evaluation metric MAE explanation I have a hard time understanding the LSTM model performance as I summarize my model as follow:

X_train.shape
(120, 7, 11)
y_train.shape
(120,)
X_test.shape
(16, 7, 11)
y_test.shape
 (16,)

model = keras.Sequential()
model.add(keras.layers.LSTM(100, input_shape=(X_train.shape[1], X_train.shape[2]), return_sequences = True))
model.add(keras.layers.Dropout(rate = 0.2))
model.add(keras.layers.LSTM(20))
model.add(keras.layers.Dropout(rate = 0.2))
model.add(keras.layers.Dense(1))
model.compile(loss='mean_squared_error', optimizer=keras.optimizers.Adam(0.001), metrics = ['mae'])

history = model.fit(
X_train, y_train,
epochs=60,
batch_size=5,
verbose= 0,
validation_split = 0.1,
shuffle=False
)

Based on the below plots, both MSE and MAE decrease in the training process and their corresponding values are near zero.

However the prediction is not precise enough as I realize:

y_pred = model.predict(X_test)
model.evaluate(X_test,y_test)
[0.04673878103494644, 0.15574690699577332]

So my question is, how does really my model perform? I mean how can interpret its performance,since both MSE and MAE seem to be low but the prediction values are not quite convincing.

You are getting loss near to 0 but, Your true distribution of y in the range of 0-1 so, that 0.04 loss may be high loss.
Just get random model and check the loss. You will get to know how much you decreased the loss. I will suggest to use r^2metric for evaluation.


Predicting evolution of an unknown mathematical function with machine learning? Is it possible to predict how a mathematical function evolves using ML? I am studying radioactive decay of polonium 210 and I have gathered data for its decay throughout a couple of weeks. I was wondering if I could use Machine Learning to train on the data available and predict how the Radioactive Activity would evolve after the timeframe where I gathered data. Can I predict (with ML) how the radioactive activity would evolve based on available experimental data?

You don't need any ML to do this task. If you search on Wikipedia Radioactive Decay you will lear in the "decay rate" section that there is a known behaviour of the times of the decays. In particular the decay of a radioactive nuclide follows what is called a Poisson process, hence the number of events in a given time window follows a Poisson distribution and the delay between two consecutive events follows an exponential distribution with a mean value that is the inverse of the decay constant. As you have the timestamp of each decay event you can calculate the delays between them and fit with an exponential distribution to obtain the decay rate of Po210. Otherwise you can calculate the number of events in a given time window and using the Poisson hypothesis you can calculate the same value for the decay rate.
As the radioactive decay is a true random event that follows quantum physics rules you won't be able (in any possible way) to know when the process wi take place, you can only estimate the probability that it will take place in a given window of time.


Does the model(best fitting line/curve) changes when the training data is changed in the cross validation? From my understanding - a machine learning algorithm goes through the inputs (independent variables) and predicts the output (dependent variable). I believe, what line/curve would best define the training data will depend on the training data. When it decides the best line or curve, the new inputs are plotted and based on plotting, the target variables are found.
My question is, during the cross-validation, the training data changes, due to the difference in splits, shouldn't that change the best fitting line, and shouldn't that make K different models(=number of splits)? If yes (it makes K different model), then what use would that be? If no, then where am I going wrong?

Your intuition that the model changes depending on the data input is right. What changes is namely the parameters, e.g. weights if you are performing a regression analysis.
Weights change primarily when you are training the data. You can reliably say that whenever you train the model it changes (namely its parameters) .
During cross validation, you split your datasets into an arbitrary number of subsets. You use some of them as training sets, while you use the rest as validation sets, and measure the error.
You repeat this through and through and you measure error of each. In each iteration when you choose different subsets (kth iteration uses different test-validation sets than the k+1th ones), you do change the model as you are re-training, (however this should not be important as you are doing this to have an approximation on how the model will perform with new data). Finally, you'll have an estimation on the model's error, (but you can use this process to choose features with most predictive power, if you are performing cross validation for feature selection). Edit: It is however important noting that cross validation is used primarily for getting an estimation on how the model would perform in general, not for training test selection.
This goes for numerical models, but I do think they apply to decision trees and other probabilistic models as well, because this process would change information metrics that are given to the model as an input (e.g. split criteria of the decision tree).