Popular classification algorithms over time In the book "Deep Learning with Python" by Francois Chollet (2018), in section 1.2.4 one can find:

Decisions trees learned from data began to receive significant research interest in the 2000s, and by 2010 they were often preferred to kernel methods.
...
In particular, the Random Forest algorithm introduced a robust, practical take on decision-tree learning that involves building a large number of specialized decision trees and then ensembling their outputs. Random forests are applicable to a wide range of problems—you could say that they’re almost always the second-best algorithm for any shallow machine-learning task. When the popular machine-learning competition website Kaggle (http://kaggle.com) got started in 2010, random forests quickly became a favorite on the platform—until 2014, when gradient boosting machines took over.

It sounds for me like the author trying to draw this kind of the evolution of classification methods:
| Method            | From | To           |
|-------------------|------|--------------|
| Kernel Methods    | ...  | 2000         |
| Decision Trees    | 2000 | 2010         |
| Random Forest     | 2010 | 2014         |
| Gradient Boosting | 2014 | Today (2019) |

Is Gradient Boosting the most popular library nowadays?
Is it can be universally applicable for any cases?
What do you think?

Gradient Boosting performs well for problems that have a medium amount of tabular data. Many problems on Kaggle are in that style.

Currently, Deep Learning holds many of the state-of-the-art benchmarks. Deep Learning generally performs better as the amount of data grows. Deep Learning performs better on other types of data (images and text).

No single machine learning algorithm is best in all cases, often called the "no free lunch theorem".


Transfer learning (on pre-trained inception net model) for multi label classification is giving similar probability for all labels Number of labels: 1000, Dataset size: 200000 images
Final probability for 1000 labels is in the range of 0.3 to 0.34.
I was expecting large variation in probabilities. Can someone tell me what I am doing wrong. I am following this tutorial

In my experience, the example code for a low number of classes (<200) works well. When moving to more classes the imbalance data makes the network converge to the same numbers. You have imbalance data because now each output is a binary classifier by its own, this doesn't happen with softmax. The way to mitigate the problem is to use weighted_cross_entropy_with_logits and set pos_weight to a positive number > 1 (10 works). But I still don't get very good results.


What is the meaning of the bubbles / spikes in the shap values plot? Here are an example of shap values plot from here.
How to interpret the 'bubble' or 'spikes' on this shap values plot I highlighted in yellow color?

That's to help indicate the volume of points in that region, which would be lost due to overlap if all plotted along the horizontal line.  This kind of plot is referred to as a "bee-swarm", and is somewhat similar to violin plots, strip plots, or scatter plots with jitter.


Classifying with certainty I'm trying to classify a binary sample with Keras and I would like to classify as many correctly as possible, while ignore the ones where the model is not sure.

The fully connected Nerual network currencly achieves around 65% but I would like to get a higher result of correctly classified ones, while ignoring the ones where the model is uncertain.

Is there a way to tell Keras to simply ignore the ones where the model is uncertain and achieve a higher accuracy that way? Or is there a network design that could achieve this, for example feeding the result of the network striaght into a second part of it which then decides whether the prediction is likely accurate or not?

One way I was thinking of achieving this is by building a second neural network on top of it that decides based on the result of the first network and all the input data of it, whether the classification will be correct or not. Would that work, and if yes, is there no more elegant way of achieving this in one go, such as directly having the results feeding into a second part of the network that then decides if the prediction is likley accurate or not?

Softmax output in neural networks can be misleading - often the confidence provided is higher than is intuitive.  See e.g. here:

A Baseline for Detecting Misclassified and
Out-of-Distribution Examples in Neural Networks
https://pdfs.semanticscholar.org/c26e/1beaeaa55acae7336882de5df48716afb8bb.pdf

which suggests that in practice, softmax is not helpfully interpretable as a probability but should instead be used for ranking among class options.

If you want to have an accurate probability estimate, you might consider using a Bayesian approach in which you explicitly model your estimate of each of the input variances, the output variance, etc.

Failing that, having a second phase neural network that takes the input and predict correct or incorrect classification by the first network is an interesting idea - where incorrect classification is a proxy for 'low confidence' classification.  If you try it I'd be curious to know how it works.

Edit: As @Emre said the input to the softmax would be more informative than the softmax itself because it's pre-scaled (i.e. not forced to sum to 1). So it should reflect confidence better, with values further away from 0 indicating higher confidence.


Is There any RNN method used for Object detection after reading the state of the art about object detection using CNN (R-CNN Faster R-CNN ,YOLO, SSD...) I was wondering if there is a method that use RNN's or that combine the use of CNN's and RNN's for object detection ??
Thank you

Yes, there have been many attempts, but perhaps the most noteable one is the approach described in the paper of Andrej Karpathy and  Li Fei-Fei where they connect a CNN and RNN in series (CNN over image region + bidirectional RNN + Multimodal RNN) and use this for labeling a scene with a whole sentence. Though, this one is more than just object detection as it leverages a data set of scenes and their descriptions to generate natural language descriptions of new unseen images.

Another example is Ming Liang and Xiaolin Hu's approche where they mix a CNN with an RNN and use this architecture for better object detection. As Ming and Xiaolin explained in their paper (linked above), the RNN is used to improve the CNN:

  A prominent difference is that CNN is typically a feed-forward architecture while in the visual system recurrent connections are abundant. Inspired by this fact, we propose a recurrent CNN (RCNN) for object recognition by incorporating recurrent connections into each convolutional layer.


are explanatory variables in multiple linear regression slopes? In simple linear regression the formula is: $y = m \cdot x + b$.
 When there are multiple independent variables: $y = b_0 + b_1 \cdot x_1 + b_2 \cdot x_2 + \ldots $

I know that $x_1,x_2,\ldots $ represents different independent variables but what exactly is $b_1,b_2,\ldots$?

They are the coefficients.

Suppose you keep the other variable fixed besides $x_1$. How much does increasing a single unit of $x_1$ would affect $y$? According to the formula, it would change by $b_1$. Similarly for the other variables.


Comparison with transfer learning and CNN from scratch on a small dataset I am doing image classification with CNN, and I have a training set of 3200 imges and a training set of 400 images. I have used two different approches for doing this classification : Transfer learnoing and bulding a CNN from scratch.

In the case of transfer learning I get an accuracy of 85%, and in the case of building the network from scratch I get an accuracy of 89%.

But usually with a small dataset shouldn't transfer leanring have better performances?

One thing to consider first is that I used early-stopping as regularization technique, and in the case of transfer learning this stopped the training process at the epoch 18, while for the CNN from scratch I arrived at the epoch 100, which is the number of epochs I imposed for training.

Can someone help me with my doubt? Thanks in advance.

They have better performance for now.

If you continued to re-train your pre-trained NN you would spezialise more and more for your dataset at hand and converge to 89%.

But here is the kicker, if you used some pre-trained models for example based ImageNet dataset you would be able to generalise a lot more better. In other words for any images in the test data that were different from the ones in the train (or from the holdout/new set) you would perform better with the transfer learning.


The most informative curve for imbalance datasets For the imbalanced datasets:

Can we say the Precision-Recall curve is more informative, thus accurate, than ROC curve?
Can we rely on F1-score to evaluate the skillfulness of the resulted model in this case?

Precision-recall curves are argued to be more useful than ROC curves in "The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets" by Saito and Rehmsmeier. They argue that ROC might lead to the wrong visual interpretation of specificity.
F1-score equally balances precision and recall. In some domains it might be more useful to more heavily weight precision (F < 1) or recall (F > 1).


How can I filter and balance a Windowed Tensorflow dataset with a binary classification label, based on the label? I have an unbalanced tensorflow windowed dataset with labels (over 90% negative examples) which I am trying to balance by filtering. I am having trouble filtering as my windowed dataset with labels does not fall under the cases I have come across in my searches, or the tensorflow documentation.

I'm working on a model to predict a binary classification based on time series data. I start with a time series dataframe with a number of columns (price, volume, etc.) where each row is one minute.

Currently I am still stuck on filtering the different labels. My next step after filtering would be to get the size of both filtered datasets, find the smaller size (n), and then concatenate the smaller dataset with (n) elements from the bigger dataset, after shuffling the bigger dataset. This way I would have a balanced dataset with an equal number of 1 and 0 labels. If you have a better Idea I would be happy to hear it.

EXPLAINING MY CODE: DFrame is a pandas dataframe with columns such as price, volume, etc., and each row is a different minute, with the first row being the earliest/oldest time period. The last column of DFrame is the classifier 0 or 1.

I then create a tensorflow dataset from slices with the first input being all the DFrame columns except for the label which is in the last column, and the second input (the label) being the last column which is the classifier.

I then use the window function to create windows of size (hindsight) which is currently 512, meaning (If i'm not mistaken) it takes the previous 511 minutes as well as the current minute and uses this as a rolling window to associate with the label of the current minute. So my understanding is that x is then an array of 512 arrays, from the row of the current minute to the row of 511 minutes ago, and the y is the label of the current minute. So x is an array of 512 arrays (rows for each minute, from the dataframe), and y is just one integer, 1 or 0.

Ideally I would like to be able to apply the same balancing logic to a multiclass classification problem, where I essentially add additional labels for additional price movement ranges.

The error comes from the filter. The model seems to run without that, and even trains my keras model. as explained I would actually want to add more code after the filter once I get it working to balance the dataset but I need to filter it first.

tensor= tf.data.Dataset.from_tensor_slices((tf.constant(DFrame[DFrame.columns.values[:-1]].values), tf.constant(DFrame[DFrame.columns.values[-1]].values)))

tensor = tensor.window(hindsight,1,1,True)

tensor = tensor.shuffle(1000)

tensor = tensor.filter(lambda x,y: tf.equal(y, 0))

tensor = tensor.flat_map(lambda x,y:tf.data.Dataset.zip((x.batch(hindsight), y.batch(1))))

tensor = tensor.batch(Batch_size).prefetch(1)

TypeError: Failed to convert object of type <class 'tensorflow.python.data.ops.dataset_ops._VariantDataset'> to Tensor. Contents: <_VariantDataset shapes: (), types: tf.int64>. Consider casting elements to a supported type.

Figured it out. Had to specify the position of y[0] instead of just y as I guess it's not just an integer but rather an array or tuple with one item, and the filter had to be done after the flatmap.

tensor= tf.data.Dataset.from_tensor_slices((tf.constant(DFrame[DFrame.columns.values[:-1]].values), tf.constant(DFrame[DFrame.columns.values[-1]].values)))

tensor = tensor.window(hindsight,1,1,True)

tensor = tensor.shuffle(1000)

tensor = tensor.filter(lambda x,y: tf.equal(y, 0))

tensor = tensor.flat_map(lambda x,y:tf.data.Dataset.zip((x.batch(hindsight), y.batch(1))))

tensor = tensor.batch(Batch_size).prefetch(1)

TypeError: Failed to convert object of type <class 'tensorflow.python.data.ops.dataset_ops._VariantDataset'> to Tensor. Contents: <_VariantDataset shapes: (), types: tf.int64>. Consider casting elements to a supported type.


Explanation on some steps of AdaBoost.R2 I am trying to understand AdaBoost.R2 in order to implement it and apply it to a regression problem. In this circumstances I need to understand it perfectly, however there's some step i don't really get.
The paper is available here, and Adaboost.R2 is presented in section 3:
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.31.314&rep=rep1&type=pdf

In step 4, $\operatorname{sup}|.|$ is used; I've never seen that notation, what does it mean exactly?
In step 7, "** means exponentation", in that case that would mean $w_i\beta  *\operatorname{exp}([1-L_i])$, right?

Q1. $\operatorname{sup}$ is the supremum, or least upper bound.  You may think of it as the maximum, but relaxed in case the set of values doesn't actually have a maximum.  (And the pipes are just absolute values.)  https://en.wikipedia.org/wiki/Infimum_and_supremum
Q2. No, they mean $w_i \cdot \beta^{1-L_i}$.  See also https://stats.stackexchange.com/a/299266/232706.


Inference of root mean square value in terms of house prediction The objective of the task is to predict the housing prices. A model is created based on California housing dataset to predict housing prices and is subjected to evaluation using the below code.

    from sklearn.metrics import mean_squared_error

    housing_predictions = lin_reg.predict(housing_prepared)
    error = mean_squared_error(housing_labels,housing_predictions)
    rmse = np.sqrt(error) --> prediction error

    // value I got for rmse was - 68628.1981

The average housing prices range from 120,000 - 265,000. But the rmse value is
68628.1981.

  Questions

By looking at rmse value, What should I infer?. I know the model is underfitting. I wanted to know the intuition behind the rmse value.

The RMSE is a measure for how "wrong" a regression model's predictions are on average, and is mostly useful as a relative metric for determining which of a set of models is best. If you want an absolute baseline against which to compare any given model you can always compute the standard deviation of your target, which is the RMSE of a model which predicts the average value of $y$ for every observation.

I would also argue that for something like housing prices a more meaningful error measure is root mean square logarithmic error, which is the RMSE after taking the log of $y$ (or $y + 1$ if $y$ contains zeros) and the predicted values. The reason for this is that we're interested in errors as proportions of our target (an error of \$50,000 is much more serious for a \$100,000 house compared to a \$1,000,000 house), and taking logs is the way to adjust for this. Also be sure to focus on test or cross validation error, else you will always choose your most overfit model.


Clustering of users in a dataset I have a dataset of book reviews:

| user_id | ISBN | vote | votes_for_user | average_user_vote | ISBN_categ |
   213      3242X   4.5        12                  3.4             1
   563      1245X   3.2        74                  2.3             2

where

  vote = rating given by user to a certain book
  votes_for_user = number of votes the user has in the dataset (nr of rows)
  average_user_vote = average of a user's votes
  ISBN_categ = integer categorical of the ISBN (since that is a string).

I want to apply a clustering algorithm such as DBSCAN to see how many clusters I can form with this dataset.

My question is:
Should I apply the clustering on the dataframe as it is (minus the ISBN column) or should I construct more features for every user and construct a dataframe where every user appears only once, together with their features, and cluster that?

Remember, the intent here is to cluster users (by user_id), not data points (votes).

If your objective is to find clusters of users, then you are interested in finding groups of "similar" reviewers.

Therefore you should:

Retain information which relates to the users in a meaningful way - e.g. votes_for_user.
Discard information which has no meaningful relationship to a user - e.g. user_id (unless perhaps it contains some information such as time / order).
Be mindful of fields which may contain implicit relationships involving a user - e.g. vote may be a result of the interaction between user and ISBN.


Variance of product of a matrix and vector using Python I am trying to calculate the variance of the product of matrix A and vector b. As it was mathematically discussed on another post https://math.stackexchange.com/questions/2365166/what-is-the-variance-of-a-constant-matrix-times-a-random-vector, I have tried to do the calculation in python.
To recap, we have
\begin{equation}
Y = \bf{Ab}
\end{equation}
where \begin{equation}
\bf{b} \sim N(\mu_b, \bf{I}\sigma_b^2)
\end{equation}
Therefore
\begin{equation}
Var(Y) = \bf{A}\sigma_b^2\bf{A}^T
\end{equation}
I have tried this code with
import numpy as np
import statistics

A = np.arange(12).reshape(4, 3)   # some 4 by 3 array
b = np.array([2,4,5])              # some 3 by 1 vector

Y = np.dot(A,b)
statistics.variance(Y)
>> 1815

Vb = np.dot(np.identity(b.shape[0]), statistics.variance(b))
np.dot(A.dot(Vb), np.transpose(A))
>> array([[ 10.,  28.,  46.,  64.],
       [ 28., 100., 172., 244.],
       [ 46., 172., 298., 424.],
       [ 64., 244., 424., 604.]])

However, the statistic.variance(Y) outputs a different result compared to np.dot(A.dot(Vb), np.transpose(A)).
I wonder how should I do it properly in python?
Your help is really appreciated!

Your are mixing some concepts. It's hard to explain but i will try.
statistics.variance(b) gives you the variance of n samples in variable b in 1 dimension. The variance matrix for b is a 1x1 matrix.
Vb is the variance matrix for 3 dimensional sample with several samples and zero covarinace between the dimensions. But it is not the variance matrix for b.
The definition of A for b is wrong. It works because np.dot uses automatically b.T, but conceptually is wrong.
I hope this help to understand it.


incremental learning vs transfer learning Can anyone explain me how incremental learning differs from transfer learning with example? Also does Transfer learning limited to neural networks?

Transfer Learning: for example you want to predict price of article normally we use previous data based on that we design  model .while new data came still we use that model for prediction here we are transferring the  same model for new task or in general
When you learn how to drive a car, you learn a generic skill and you will use these similar set of skills if you want to drive a truck. This transfer of your knowledge is called transfer learning.

continuous learning:for example In face recognition or image classification system, first we train our model with some data,if new person is came our model can't recognize the person in this case we train our model only with new data and attached that weight into our model. in general
If you have learnt any new skills while driving a truck, you will continuously update your knowledge and use that for improving your driving skill. This is called incremental learning (or) continuous learning (or) online learning.


How to add extra word features other then word Embedding in Recurrent Neural Network model I am building a deep learning model for NLP. I am pretty comfortable with adding word embedding from word2vec or Glove vectors as extra word features but I wanted to add other word features like POS tag of a word, NER tag of word along with embedding as features. How can I do this. Should I give these word features by concatenating their vector with the word vectors. Or is there some other method. Please suggest.

One option is to concatenate them, the second is to treat them as separate inputs. For example Keras offers such neural model: https://keras.io/getting-started/functional-api-guide/#multi-input-and-multi-output-models


Sales Prediction for Fashion Retail Data I am working on project related to prediction of sales quantity  based on previous data.
Data consists of attributes like  .MRP, Size , Discount,Color,Fabric type e.t.c
I am  having both sales quantity and Sock on hand in data .Currently I am trying to predict sales quantity using Random forest Algorithm(in R).

I did some basic feature transformations like one hotted categorical variables and performed Normalization on data and training the model.
   Here I am having some issues in data.I performed some manipulations listed below

I am having two different  datasets  one contains sales records and other has stock on hand.
we are matching and aggregating  the records week wise. In some cases stock on hand is not available but sales happened .For those records making sales quantity as stock on hand.
For unsold records we are not having any discount .To keep discount we are taking average of discounts over the same period for particular item from other store (Suggestion by client).

Some of the  records are having negative sales quantity after aggregation. we are making those negative sales to zero .
To train the model I am having sales quantity "0"(Zero) for unsold items.In data unsold items are more than sold items (like if sold items are 20  unsold are around  300)

I am not getting any considerable results .I tried the same thing with **Neural Network **  also in R.

I am beginner in predictive modelling.This is my first project .Please  correct me if anything wrong in my approach

If this is your first project start with something simple - e.g. linear regression, lm(sales ~ data). Once you get it up and running you can try something more sophisticated.

I suggest avoiding neural networks in sales prediction - neural networks are sexy, but require huge amounts of data to train properly. This is often not feasible in sales prediction environment.


Inconsistency on A. Graves' original Connectionist Temporal Classification (CTC) paper? Here's the link to the paper.

For the forward-backward algorithm, they introduce $\alpha_t(s)$ as a definition in eq. (5).
Then they give a recursive formula for it (eq. (6)), and its initialization. However, the two formulae are not equivalent.

For example, let's say we have a labelling $\mathbf{l}=[\text{c},\text{a},\text{t}]$.

$\alpha_1(1)$ using eq. (5):

$\mathcal{B}(\pi_1)=l_1=\text{c}$, so $\pi: \pi_1=\text{c}$ is the only possible path. Therefore $\alpha_1(1)=y_{l_1}^1=y_{\text{c}}^1$ and not $y_\varepsilon^1$ as they state in the initialization (where $\varepsilon$ is the blank label).

Another example. $\alpha_2(1)$ using eq. (5):

$\mathcal{B}(\pi_{1:2})=l_1=\text{c}$, so the possible paths are $\pi: \pi_{1:2}\in\{[\text{c},\text{c}],[\text{c},\varepsilon],[\varepsilon,\text{c}]\}$. Therefore $\alpha_2(1)=y_{\text{c}}^1y_{\text{c}}^2+y_{\text{c}}^1y_{\varepsilon}^2+y_{\varepsilon}^1y_{\text{c}}^2$. However, using equation (6) we get a different result:

We have $l'_1=\varepsilon$ so we are in the first case. Then $\alpha_2(1)=\bar{\alpha}_2(1)y_{l'_1}^2=(\alpha_1(1)+\alpha_1(0))y_\varepsilon^2=(y_\varepsilon^1+0)y_\varepsilon^2=y_\varepsilon^1y_\varepsilon^2$.

I tried many other cases as well, and it seems like, the way they are written, the two formulae are completely different. Am I missing something?

The lecture I take from the paper is not that equation (6) is a recursive form of (5). In (5), the equation refers to label sequence $l$, while (6) refers to $l^\prime$, which is a modified version of $l$ with blanks added to the
beginning and the end and inserted between every pair of labels. See the paragraph between equations (5) and (6):

  To allow for blanks in the output paths, we consider a modified label sequence l, with blanks added to the beginning and the end and inserted between every pair
  of labels. The length of l is therefore 2|l| + 1. In calculating the probabilities of prefixes of l we allow all transitions between blank and non-blank labels, and also those between any pair of distinct non-blank labels.


how to export the data edited to a new excel file The provided code imports the excel file and removes the not NaN values and store it in df2. Now, How do i save the changes to an another file.

df1 = pd.read_excel('file_name.xlsx')
df2 = df1[pd.isnull(df1['column_name'])]
df2

You could simply use:

df2.to_excel("output.xlsx")

I suggest you take some time to look at the various functionalities pandas offers. Will be great help to you in future!


Methods of building machine learning models I have seen pages where they mention 5 methods of building models.

1) All-in
2) Backward Elimination
3) Forward Selection
4) Bidirectional Elimination
5) Score Comparision

I usually implement a linear regression or any algorithm using sklearn

from sklearn.linear_model import LinearRegression
lr = LinearRegression()
lr.fit(X, y)
y_pred = lr.predict(X_test)

How can one implement those 5 methods of building models?

Can anyone explain the importance of this and What is most commonly used one out of these?

Strictly speaking, those are not methods of "building" models, they are feature selection strategies.

Most of them are implemented in Scikit-learn.


Using attributes to classify/cluster user profiles I have a dataset of users purchasing products from a website.

The attributes I have are user id, region(state) of the user, the categories id of product, keywords id of product, keywords id of website, and sales amount spent of the product.

The goal is to use the information of a product and website to identity who the users are, such as "male young gamer" or "stay at home mom".

I attached a sample picture as below:

There are all together 1940 unique categories and 13845 unique keywords for products. For the website, there are 13063 unique keywords. The whole dataset is huge as that's the daily logging data.

I am thinking of clustering, as those are unsupervised, but those id are ordered number having no numeric meaning. Then I don't know how to apply the algorithm. I am also thinking of classification. If I add a column of class based on the sales amount of product purchased. I think clustering is more preferred. I don't know what algorithm I should use for this case as the dimensions of the keywords id could be more than 10000 (each product could have many keywords, so does website). I need to use Spark for this project.

Can anyone help me out with some ideas or suggestions?

Thank you so much!

Right now, I only have time for a very brief answer, but I'll try to expand on it later on.

What you want to do is a clustering, since you want to discover some labels for your data.  (As opposed to a classification, where you would have labels for at least some of the data and you would like to label the rest).

In order to perform a clustering on your users, you need to have them as some kind of points in an abstract space.  Then you will measure distances between points, and say that points that are "near" are "similar", and label them according to their place in that space.

You need to transform your data into something that looks like a user profile, i.e.: a user ID, followed by a vector of numbers that represent the features of this user.  In your case, each feature could be a "category of website" or a "category of product", and the number could be the amount of dollars spent in that feature.  Or a feature could be a combination of web and product, of course.

As an example, let us imagine the user profile with just three features:

dollars spent in "techy" webs,
dollars spent on "fashion" products,
and dollars spent on "aggressive" video games on "family-oriented" webs (who knows).

In order to build those profiles, you need to map the "categories" and "keywords" that you have, which are too plentiful, into the features you think are relevant.  Look into topic modeling or semantic similarity to do so.  Once that map is built, it will state that all dollars spent on webs with keywords "gadget", "electronics", "programming", and X others, should all be aggregated into our first feature; and so on.

Do not be afraid of "imposing" the features!  You will need to refine them and maybe completely change them once you have clustered the users.

Once you have user profiles, proceed to cluster them using k-means or whatever else you think is interesting.  Whatever technique you use, you will be interested in getting the "representative" point for each cluster.  This is usually the geometric "center" of the points in that cluster.

Plot those "representative" points, and also plot how they compare to other clusters.  Using a radar chart is very useful here.  Wherever there is a salient feature (something in the representative that is very marked, and is also very prominent in its comparison to other clusters) is a good candidate to help you label the cluster with some catchy phrase ("nerds", "fashionistas", "aggressive moms" ...).

Remember that a clustering problem is an open problem, so there is no "right" solution!  And I think my answer is quite long already; check also about normalization of the profiles and filtering outliers.


Where to store dataset when creating neural network models and using github? I have been recommended to use GitHub for all my coding projects as it is a great way to demonstrate to employers what you have been working on as well as just sharing anything you have created that might be useful to others. So I am storing my code for Neural Network Models I am working in a GitHub repository but I am not sure when I download a data set to run a test with where I should download that too? Inside or outside the git repository?

Sorry if this is a dumb question but new to this. I can imagine arguments for either answer. Mainly size being too big or the results being easily reproducible.

If the dataset isn't too large, you may add datasets on GitHub itself. Generally project datasets aren't large, but again, I can't be sure as you haven't mentioned any approximate size. But if it is large, you may refer to data source in your ReadMe or within your .py file as an external hyperlink.

For example, save dataset on Google Drive & turn on Viewable option by generating shareable URL. Now that generated link can be refered to in your ReadMe file as something like: "Kindly refer this external link to access dataset". Alternatively, you may also read data directly within your .py file from that generated link, and any project reviewer would notice that external link being used to read dataset.

Hope that answers your query!


When to use t-distribution instead of normal distribution? According to the Student's t-distribution wiki article the t-distribution is used instead of the Normal distribution "when estimating the mean of a normally distributed population in situations where the sample size is small and the population standard deviation is unknown".

An unknown population standard deviation implies that it would have to be estimated from the samples itself which is inaccurate with small sample sizes. According to the Z-test wiki article a sample size >= 30 implies the use of a normal distribution, a sample size < 30 implies the use of the t-distribution. (t-test for reference.) Is this assumption common best practice? How does this relate to sample size determination (estimation of a mean)?

Student t-distribution by definition is a distribution of mean estimates from samples taken from the normally distributed population.

T-distribution has thicker tails and it gets thinner with increase of degrees of freedom, which in turn depends on sample distribution. So, at some point it closely resembles normal distribution and can be substituted by it.

As far as I remember (although, not 100% sure) this sample size of 30 threshold is valid for the t-test with α=0.05 (widely accepted Type I error level). Although, of you have much smaller α (e.g. 0.0001) you need to go much further to the tails of the distributions where the difference between t-distribution and normal distribution will be much more evident and thus you are better off using t rather than normal for larger sample size.

Another issue is standard deviation (or rather here we are talking about standard error). Normal distribution (and thus z-test) requires knowledge of population standard deviation. If you do not know it, you need to estimate it from the sample and it is obviously will be only an estimation of the population standard deviation. Student t-distribution handles estimated standard deviation better because using it with normal distribution (which should have only population standard deviation) will create extra errors (you will incorrectly estimate your type I (and Ii) errors).

So, the answer is, if you do not know population mean (which is almost always the case in the real world), use t-distribution. If you know population mean, be careful with 30 sample threshold. Depends on your application it can be significantly higher.


Is it possible to extract specific words from a sentence in Hindi/Marathi? I have seen the different options to extract words from sentences in English but when I wanted to know if its possible to do the same thing in Hindi or Marathi
eg: टोमॅटो बेचना है
where the word at position 1 is TOMATO and it needs to be extracted i.e. extraction of product name from a sentence.
Any help will be appreciated.

Based on your comment I understand that you want to identify a particular syntactic category of words, for instance nouns. The task which identifies these categories is called Part Of Speech (POS) tagging.
I'm not familiar with any specific resource for Hindi or Marathi but there are probably some software that you can use. From a quick search I found a few links below:

General resources for Indian languages
A POS tagger for Hindi
Another more recent POS tagger for Hindi


Kappa From Combined Confusion Matrices I am trying to evaluate and compare several different machine learning models built with different parameters (i.e. downsampling, outlier removal) and different classifiers (i.e. Bayes Net, SVM, Decision Tree).

I am performing a type of cross validation where I randomly select 67% of the data for use in the training set and 33% of the data for use in the testing set. I perform this for several iterations, say, 20.

Now, from each iteration I am able to generate a confusion matrix and compute a kappa. My question is, what are some ways to aggregate these across the iterations? I am also interested in aggregating accuracy and expected accuracy, among other things.

For the kappa, accuracy, and expected accuracy, I have just been taking the average up to this point. One of the problems is that when I recompute kappa with the aggregated average and expected average, it is not the same with the aggregated kappa.

For the confusion matrix, I have been first normalizing the confusion matrix from each iteration and then averaging them, in an attempt to avoid an issue of confusion matrices with different numbers of total cases (which is possible with my cross validation scheme).

When I recompute the kappa from this aggregated confusion matrix, it is also different from the previous two.

Which one is most correct? Is there another way of computing an average kappa that is more correct?

Thanks, and if more concrete examples are needed in order to illustrate my question please let me know.

I think most people would simply average a statistic like accuracy or kappa over the several runs, rather than compute one overall statistic.

While it's not necessarily true that an average (weighted by the test set size in each case) of these statistics equals the statistic computed over all of the runs at once, it happens to be true for accuracy and kappa. Hence it shouldn't matter.

I think the reason you may be getting a difference is that you're normalizing away the test set size. I don't think you should.


The cross-entropy error function in neural networks In the MNIST For ML Beginners they define cross-entropy as

$$H_{y'} (y) := - \sum_{i} y_{i}' \log (y_i)$$

$y_i$ is the predicted probability value for class $i$ and $y_i'$ is the true probability for that class.

Question 1

Isn't it a problem that $y_i$ (in $\log(y_i)$) could be 0? This would mean that we have a really bad classifier, of course. But think of an error in our dataset, e.g. an "obvious" 1 labeled as 3. Would it simply crash? Does the model we chose (softmax activation at the end) basically never give the probability 0 for the correct class?

Question 2

I've learned that cross-entropy is defined as

$$H_{y'}(y) := - \sum_{i} ({y_i' \log(y_i) + (1-y_i') \log (1-y_i)})$$

What is correct? Do you have any textbook references for either version? How do those functions differ in their properties (as error functions for neural networks)?

One way to interpret cross-entropy is to see it as a (minus) log-likelihood for the data $y_i'$, under a model $y_i$.

Namely, suppose that you have some fixed model (a.k.a. "hypothesis"), which predicts for $n$ classes $\{1,2,\dots, n\}$ their hypothetical occurrence probabilities $y_1, y_2,\dots, y_n$. Suppose that you now observe (in reality) $k_1$ instances of class $1$, $k_2$ instances of class $2$, $k_n$ instances of class $n$, etc. According to your model the likelihood of this happening is:
$$
P[data|model] := y_1^{k_1}y_2^{k_2}\dots y_n^{k_n}.
$$
Taking the logarithm and changing the sign:
$$
-\log P[data|model] = -k_1\log y_1 -k_2\log y_2 - \dots -k_n\log y_n = -\sum_i k_i \log y_i
$$
If you now divide the right-hand sum by the number of observations $N = k_1+k_2+\dots+k_n$, and denote the empirical probabilities as $y_i'=k_i/N$, you'll get the cross-entropy:
$$
-\frac{1}{N} \log P[data|model] = -\frac{1}{N}\sum_i k_i \log y_i = -\sum_i y_i'\log y_i =: H(y', y)
$$

Furthermore, the log-likelihood of a dataset given a model can be interpreted as a measure of "encoding length" - the number of bits you expect to spend to encode this information if your encoding scheme would be based on your hypothesis.

This follows from the observation that an independent event with probability $y_i$ requires at least $-\log_2 y_i$ bits to encode it (assuming efficient coding), and consequently the expression
$$-\sum_i y_i'\log_2 y_i,$$
is literally the expected length of the encoding, where the encoding lengths for the events are computed using the "hypothesized" distribution, while the expectation is taken over the actual one.

Finally, instead of saying "measure of expected encoding length" I really like to use the informal term "measure of surprise". If you need a lot of bits to encode an expected event from a distribution, the distribution is "really surprising" for you.

With those intuitions in mind, the answers to your questions can be seen as follows:

Question 1. Yes. It is a problem whenever the corresponding $y_i'$ is nonzero at the same time. It corresponds to the situation where your model believes that some class has zero probability of occurrence, and yet the class pops up in reality. As a result, the "surprise" of your model is infinitely great: your model did not account for that event and now needs infinitely many bits to encode it. That is why you get infinity as your cross-entropy.

To avoid this problem you need to make sure that your model does not make rash assumptions about something being impossible while it can happen. In reality, people tend to use sigmoid or "softmax" functions as their hypothesis models, which are conservative enough to leave at least some chance for every option.

If you use some other hypothesis model, it is up to you to regularize (aka "smooth") it so that it would not hypothesize zeros where it should not.
Question 2. In this formula, one usually assumes $y_i'$ to be either $0$ or $1$, while $y_i$ is the model's probability hypothesis for the corresponding input. If you look closely, you will see that it is simply a $-\log P[data|model]$ for binary data, an equivalent of the second equation in this answer.

Hence, strictly speaking, although it is still a log-likelihood, this is not syntactically equivalent to cross-entropy. What some people mean when referring to such an expression as cross-entropy is that it is, in fact, a sum over binary cross-entropies for individual points in the dataset:
$$
\sum_i H(y_i', y_i),
$$
where $y_i'$ and $y_i$ have to be interpreted as the corresponding binary distributions $(y_i', 1-y_i')$ and $(y_i, 1-y_i)$.


Confidence interval interpretation in linear regression when errors are not normally distributed I've read that "If the error distribution is significantly non-normal, confidence intervals may be too wide or too narrow" (source). So, can anyone elaborate on this? When are the confidence intervals narrow and when are they wide? Does it have anything to do with skewness?

In case you might want to try another way to find confidence intervals, and in addition to the nice and rigourous answer by Peter, I would also consider a resampling method like bootstrapping as a robust way to find confidence intervals.
One key advantage is that it does not assume any kind of distribution, being a distribution-free method to find your coefficients estimates.

In the case of finding a confidence interval for a linear regression, the steps would be:

Draw n random samples (with replacement) from your dataset, where n is the bootstrap sample size
Fit a linear regression on the bootstrap sample from step 1
Repeat steps 1 & 2 n_iters times, where n_iters will be the number of bootstrap samples and linear regressions made on them
Now that we have n_iters values for the linear regression coefficients, we can find the interval limits via the min, median and max percentiles (e.g. for a 95% CI: percentile 2.5, 50 and 97.5) to find the coefficient estimate together with the CI limits

Please mind the variability of the confidence intervals along the x-axis values, taking into account the sampling error of the coefficients estimates (good source of read: https://greenteapress.com/wp/think-stats-2e/)

The associated code of my example including the plot can be found here


What to give as predictors to predict future values? I am new to machine learning techniques. I was going through few supervised machine learning model examples and i have doubt in predicting future values.

I have daily time series dataset from database where my target variable is complete noise signal like this:

To train and predict the models, we divide the dataset into train set, validation set and test set to check if the model is efficient or not. I have two independent variable and one target variable. I am using linear regression,Keras LSTM and other models.

My basic question is how do i predict for future values (for next one week, one month) when i don't have independent variables for next week? what am i supposed to give as predictors in that case?

Any information is much appreciated.

Welcome to the site!

Firstly, when we use any kind of predicting algorithm, then you need to have the future values of the independent variables(features for explaining the target variable), for Prediction.

Secondly, If you don't have such values then you cannot use predicting algorithms, we have forecasting algorithms to do such task. To apply this you need to have data which is time series data. From the sample and graph I think your data is time series data.

FYI, you can also use additional features even in time series. For Feature Engineering on Time Series data you can go through the link.

If you need any additional information let me know.


Merging information of rows with the same date My data looks like this:

date, cardio_time, muscles, muscle_time, stretch_time
2018-01-01, 0, "biceps / lats", 40, 5
2018-01-02, 30, "", 0, 10
2018-01-03, 0, "lats / calf", 41, 6
2018-01-03, 30, "hamstring", 4, 5
2018-01-04, 0, "biceps / lats", 42, 8

I would like to merge those rows with the same date, and save the info from both rows. My data would look like this after transformation, note the 3rd of January has been changed:

2018-01-01, 0, "biceps / lats", 40, 5
2018-01-02, 30, "", 0, 10
2018-01-03, 30, "lats / calf / hamstring", 45, 11
2018-01-04, 0, "biceps / lats", 42, 8

I think I could use a for loop that checks if the date on row i is the same as on row i-1, and if it is not, check the next row, but if the rows do have the same date, merge the rows together by doing something like this:

# set default value to 1 exercise per row
df['nr_excercises'] = 1
# for loop
for i in range(1, T):
    if df.index[i] == df.index[i-1]:
      # set nr of nr_excercises to 2
      df.iloc[i, nr_excercises] = 2
      # create temp variables that hold info from both rows
      cardiotimetot = df[i, cardio_time] +  df[i-1, cardio_time]
      stretchtimetot = df[i, stretch_time] +  df[i-1, stretch_time]
      etc...
      # save temp variables to i
      df.iloc[i, cardio_time] = cardiotimetot
      # drop row i-1
      df = df.drop[df.index[i-1]] # I think this is correct

Question: Is this a good approach? Is there a better way to do it?

Maybe, the code would be faster if I first use .groupby(df.index).size() to find out which days have multiple entries, and then only apply the for loop to this subset of df.

As @Emre has pointed out in comments, you need a pandas custom aggregator.

So since you need a string custom  join by /. Create a custom aggregator as

foo = lambda a: "/".join(a)

(or if you need spaces around the join)

foo = lambda a: " / ".join(a)

Then make a pandas groupby as

data_.groupby(by='date').agg({'muscle_time': 'sum',
                              'stretch_time': 'sum',
                              'cardio_time': 'sum',
                              'muscles': foo}).reset_index()

This should get you the aggregated dataframe.


Which features do I select from text? Hello, I am very new to data science, machine learning, and stack overflow. Excuse me for being unclear or asking naive questions.
 My question is as follows:
From any given document, I am trying to classify it according to the emotions it evokes in readers, using a neural network. However, I am having difficulties with feature selection. I'm thinking of using NLTK and RAKE to extract keywords, but I don't know how I can translate them into features. Should I hash the keywords for one feature? Or, should I find a dictionary of english words (i.e. Wordnet), and use every word in the dictionary as a feature.

Using NLTK in python you should first Tokenize the sentences into words, even you can use Ngram for 2-Gram or 3-Gram bags of word, the reason I am suggesting N-Gram is that let's suppose you have sentence like: I am not happy with this product, then 2-Gram tokenize it as ['not happy', 'happy with', 'with this', 'this product'] here I and am are assumed as STOPWORDS. Using HashingTF you can hash the sentence into a feature vector as ['word position': frequency of word, ...] i.e highly sparse vectors, For Hashing in PySpark check this documentation.

Here below python code will help you to tokenize in bags of word

import string

from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem.porter import PorterStemmer

PUNCTUATION = set(string.punctuation)
STOPWORDS = set(stopwords.words('english'))
STEMMER = PorterStemmer()

example = ['Hello Krishna Prasad, this is test file for spark testing',
            'Another episode of star',
            'There are far and away many stars'
            'A galloping horse using two coconuts'
            'My kingdom for a horse'
            'A long time ago in a galaxy far']

def tokenize(text):
    tokens = word_tokenize(text)
    lowercased = [t.lower() for t in tokens]
    no_punctuation = []
    for word in lowercased:
        punct_removed = ''.join([letter for letter in word if not letter in PUNCTUATION])
        no_punctuation.append(punct_removed)
    no_stopwords = [w for w in no_punctuation if not w in STOPWORDS]
    stemmed = [STEMMER.stem(w) for w in no_stopwords]
    return [w for w in stemmed if w]

tokenized_word = [tokenize(text) for text in example]
for word in tokenized_word:
    print word

Out of the above code as:

$python WordFrequencyHash.py
[u'hello', u'krishna', u'prasad', u'test', u'file', u'spark', u'test']
[u'anoth', u'episod', u'star']
[u'far', u'away', u'mani', u'starsa', u'gallop', u'hors', u'use', u'two', u'coconutsmi', u'kingdom', u'horsea', u'long', u'time', u'ago', u'galaxi', u'far']

You can also use word2vec or countvectorizer for tokenization.


how to know what my data are uninformative and that machine learning will not work with it? following this question, I'm making a data analysis again because I tried to use machine learning algorithms like Random forest to predict a value from certain features but it didn't work for me. I also tried Neural Networks and the results were worse than Random Forest.

So I thought maybe is something wrong with my data as I also read that data can have bad quality somehow(I'm not sure what does this mean). Since I'm not a data scientist and don't have background in Data Science in general (I'm an Electrical Engineer), I want to ask here and maybe someone can help me.

I read that correlation between features and target that I want to predict is important, now this is not my case, my features doesn't have a strong correlation with my features, does this mean that my data is bad and I can't predict that target from those features ?

I ll add here an image of the heatmap of my data:

my target that I want to predict is xdistance and ydistance. It have negative correlation with one another but I think it is not important since I'm predicting every target on its own, but what is interesting here is that there is no feature that correlate strongly with one of the two targets. So I fear that this is an impossible task to try to predict those targets, am I right?

You may have heard

Correlation does not imply causation

but there is one more thats even more important

Without correlation there CAN BE causation. One example

SO just because they do not correlate does not mean that they are uninformative (think also about feature interactions)

I think you are looking for some test/theory that can give you some statement before hand. Without some HARD asssumptions thats impossible. Only way is to exhaustively try pretty different approaches and to reach desired metric scores.


Why using a frozen embedding layer in an LSTM model I'm studying this LSTM mode:
https://www.kaggle.com/paoloripamonti/twitter-sentiment-analysis

They use a frozen embedding layer which uses an predefined matrix with for each word a 300 dim vector which represents the meaning of the words.

As you can see here:

embedding_layer = Embedding(vocab_size, W2V_SIZE, weights=[embedding_matrix], input_length=SEQUENCE_LENGTH, trainable=False)

The embedding layer is frozen which means that the weights are not changed during training.

Why is this done?

The embedding matrix which used in the initialization of the Embedding layer is highly trained on a large corpus of text. The training and the data are so huge that the embedding has learnt a type of association between words.

  A pretrained embedding like Word2Vec will produce vectors for words like school and homework which are similar to each other in the embedding space.

Many such associations are learnt after rigorous training mostly on high-end machines and precisely calculated parameters.

Why is the Embedding layer set to trainable=false?

As mentioned in the code, we have given a pretrained embedding matrix to the Embedding layer through the weights= argument. As the word suggests, its "pretrained" and requires no additional training.

We can enjoy the benefits of such an embedding by keeping it untrainable.
Additional training in the context of our task, may result in unusual behaviour of the Embedding layer and also distort the learned associations.

In some cases, the Embedding layer is kept trainable.


VC dimension of hypothesis space of finite union of intervals I have the following concept:
$$C = \left\{\bigcup_{i=1}^{k}(a_i, b_i): a_i, b_i \in {\Bbb R}, a_i < b_i, i=1,2,..,k\right\}
$$
and was wondering how to determine the VC dimension of C?

VC dimension is defined for a hypothesis space $H$, e.g. a set of binary classifiers $C \rightarrow \{0, 1\}$. For example, hypothesis space
$$H=\{{\Bbb 1}_{x \le \theta}: \theta \in {\Bbb R}\}$$
has VC dimension $1$, because for any $C=\{a<b\}$,  it does not contain a classifier that gives $\{a \rightarrow 0, b\rightarrow 1\}$.

For example, a classifier from $H$ would be $f(x)={\Bbb 1}_{x \le a}$ that gives  $\{a \rightarrow 1, b \rightarrow 0\}$.

From C to H

As you have illustrated in the comments, we can build a hypothesis space $H$ from $C$ as follows:
$$H=\left\{{\Bbb 1}_{x \in C}: C = \left\{\bigcup_{i=1}^{k}(a_i, b_i): a_i, b_i \in {\Bbb R}, a_i < b_i, i=1,2,..,k\right\}\right\}$$
Meaning, each classifier in $H$ is a union of $k$ intervals that labels a point inside the union as $1$ and outside as $0$.

VC dimension of this $H$ is $2k$:

For VC $\geq 2k$: Let $A$ be an arbitrary set , and $A \rightarrow \{0, 1\}$ be an arbitrary labeling. By going from minimum to maximum member of $A$, we can cover all adjacent $1$s with one interval, and only need to use another interval when there is a $0$ barrier. Therefore, we need $k$ intervals to cover $k$ isolated regions of $1$s. Furthermore, a set with  $2k$ members has at most $k$ isolated $1$s (since to have $k+1$ isolated $1$s there should be $k$ $0$ barriers in-between), and thus, needs at most $k$ intervals.
For VC $< 2k+1$ by contradiction: for any ordered set $A_{2k+1}=\{a_1<...<a_{2k+1}\}$, there is labeling $a_k \rightarrow 1_{\text{k odd}}$, i.e.  $\{a_1 \rightarrow 1, a_2 \rightarrow 0,...,a_{2k+1} \rightarrow 1\}$ with  $k+1$ isolated $1$s which cannot be covered with $k$ intervals.


Reinforcement Learning with static state Can Q Learning work with a static state for each step?

What I mean by that is that the actions do not influence the following state at all. The episodes just iterate over the same data over and over again. Of course the different actions lead to different rewards, but is Q Learning the right concept to use in this case? What other types could you suggest?

The framework you describe is the Bandits framework. Bandits are algorithms that solve stateless MDPs. So the problem is not sequential in the sense that actions affect the transition to the next state. Take a look at a nice Contextual Bandits tutorial which is accompanied with Python Jupyter notebook. Please note that Contextual Bandits are a special category of Bandits but at least you will be exposed to the general Bandit idea and formalization.

You can try Q learning but with the Rescorla-Wagner rule $r(a) - Q(a)$ as the reward correction $\gamma\max_a' Q(s',a')$ does not make any sense as there is no transition to a subsequent state $s'$.


Choosing a suitable learning rate based on validation or testing accuracy? I have simulated a neural network with different learning rate, ranging from 0.00001 to 0.1, and recording each test and validation accuracy. The result i obtained is as below. There is 50 epoch for each learning rate, and i note down the validation accuracy at the last epoch, while the training accuracy is computed throughout the process.

Learning rate: 0.00001

Testing accuracy: 0.5850

Validation accuracy at final epoch: 0.5950

Learning rate: 0.0001

Testing accuracy:0.6550

Validation accuracy at final epoch: 0.6400

Learning rate: 0.001

Testing accuracy: 0.6350

Validation accuracy at final epoch: 0.6900

Learning rate: 0.01

Testing accuracy: 0.6650

Validation accuracy at final epoch: 0.6700

Learning rate: 0.1

Testing accuracy: 0.2500

Validation accuracy at final epoch: 0.2100

How does testing and validation accuracy influence which learning rate is better? Would a higher validation accuracy determine the most suitable learning rate for the model?

Hence, is it correct that 0.001 is the most suitable learning parameter since it has the highest validation accuracy at the last epoch?

You cannot select a parameter based on test accuracy, because the moment you do that, it becomes a validation accuracy as it has affected the final model. Therefore, you are always choosing based on validation accuracy.

As a result, the best result comes from learning rate 0.001, with the highest validation accuracy 0.6900. We have ignored Testing accuracy. If we select based on Testing accuracy, it becomes a validation accuracy.

Generally, a learning rate that is a looser at epoch 50, might be a winner at epoch 200. In other words, a slower convergence may lead to a higher accuracy. Therefore, this issue is worth considering too.


How to perform Logistic Regression with a large number of features? I have a dataset with 330 samples and 27 features for each sample, with a binary class problem for Logistic Regression.

According to the "rule if ten" I need at least 10 events for each feature to be included. Though, I have an imbalanced dataset, with 20% o positive class and 80% of negative class.

That gives me only 70 events, allowing approximately only 7/8 features to be included in the Logistic model.

I'd like to evaluate all the features as predictors, I don't want to hand pick any features.

So what would you suggest? Should I make all possible 7 features combinations? Should I evaluate each feature alone with an association model and then pick only the best ones for a final model?

I'm also curious about the handling of categorical and continuous features, can I mix them? If I have a categorical [0-1] and a continuous [0-100], should I normalize?

I'm currently working with Python.

Thanks a lot for your help!

In order to reduce your model down to 7 variables there are a few approaches you could take:

PCA (unsupervised): this creates "new" linear combinations of your data where each proceding component explains as much variance in the data as possible. So the first 7 components (out of 27) should be able to explain a good percentage of the variation in your data. You can then plug these seven components into your logistic regression equation. The disadvantage here is that because the components are combinations of your original variables you lose some interpretability with your regression model. It should however produce very good accuracy. This same technique applied to other dimension reduction methods such as
Another common method in regression is forward stepwise where you start with one variable and add on another each step, which is either kept or dropped based on some criteria (usually a BIC or AIC score). Backwards stepwise regression is the same thing but you start with all variables and remove one each time again based on some criteria. Based on a brief search it doesn't seem that python has a stepwise regression but they do a similar feature elimination algorithm described in this Data Science post.
Lasso Regression uses an $L_{1}$ penalization norm that shrinks the coefficients of features effectively eliminating some of them.You can include this $L_1$ norm into your logistic regression model. It seems sklearn's LogisticRegression allows you do assign the penalization you want in order to achieve this. Note: Lasso will not explicitly set variable coefficients to zero, but will shrink them allowing you to select the 7 largest coefficients.

As @E_net4 commented, your continuous question is addressed in another post.


what does the standard deviation plot around my learning curve indicate? I plotted a learning curve below. There is a thick red band around the top portion of my training score. Why is it so high at the beginning?

Below is a snippet of the code used:
train_sizes, train_scores, test_scores = learning_curve(
        estimator, X, y, cv=cv, n_jobs=n_jobs, train_sizes=train_sizes, scoring= 'neg_brier_score')
    train_scores_mean = np.mean(train_scores, axis=1)
    train_scores_std = np.std(train_scores, axis=1)
    test_scores_mean = np.mean(test_scores, axis=1)
    test_scores_std = np.std(test_scores, axis=1)
    plt.grid()

    plt.fill_between(train_sizes, train_scores_mean - train_scores_std,train_scores_mean + train_scores_std, alpha=0.1,
                     color="r")
    plt.fill_between(train_sizes, test_scores_mean - test_scores_std,
                     # + test_scores_std, alpha=0.1, color="g")
    plt.plot(train_sizes, train_scores_mean, 'o-', color="r",
             label="Training brier score")
    plt.plot(train_sizes, test_scores_mean, 'o-', color="g",
             label="Cross-validation brier score")

I only have a guess, but I suspect it could be simply due to random initialisation. This would mean that after few training samples the models would still be very different. After 400k training samples the models all converge to the same learning path. I could be wrong of course!


Basic doubt regarding "training" of a YOLO model So I have just recently started exploring machine learning, and for a project I was required to train the YOLO v5 model. I first tried it on the coco128 dataset:https://www.kaggle.com/ultralytics/coco128..
repository of the yolo v5:
https://github.com/ultralytics/yolov5
I followed this tutorial: https://github.com/ultralytics/yolov5/wiki/Train-Custom-Data step by step and managed to train the model succesfully, it is detecting objects as I intended it  to do.
I wanted to have a very basic overview an what is basically happening when we are "training" the model.
The command used for training was:
python train.py --img 640 --batch 16 --epochs 5 --data coco128.yaml --weights yolov5s.pt

From what I understand, this tells the command line to run the file "train.py" (which uses argeparse library) with the following arguments: 640 (image size), 16 (batch size for batch gradient descent), 5 (no. of epochs), coco128.yaml (the config file), and yolov5s.pt (the weights file).
Doubts

The tutorial mentions that the weights are pretrained. This is confusing me because I thought the entire objective of "training" a model is to find the correct weights for the parameters. If the weights are already supplied, what exactly is the training achieving? It has been fed a bunch of images along with the labels for the bounding boxes for that images. What does it do with these images and labels to become capable of predicting objects?
The coco128.yaml mentions a path for the "validation" folder. What exactly is a "validation" folder used for?

About Pre-trained model : This is a very common practise (especially in image recognition) and here is how we use it.
Let's imagine you want to recognize different types of food (beef, pork, vegetables, ...). You know some networks already exist that recognize all types of objects (boats, cars, food, sofas, ...). This objective of transfer learning is to use these models and adapt it to your task, so you pick an already trained model, and train it again, but only with your data (in our example types of food). Being able to recognize and distingish different objects is not useless for your task, so even if your network doesn't do the the same thing as the initial model, what the initial model has learned is used as foundations to learn your task.
We call this fine-tuning the model, it is basically using a complicated network that would be way too heavy to train because the dataset is not big enough, and just adjusting it so it performs our specialized task.
In your tutorial, the dataset Coco128 is made of 128 images, which is not enough for the training of a model as big as YOLOv5. So you use the pretrained weights and fine-tune it with your dataset. As far I as understand, the objective of the tutorial is to see how the model overfits these 128 images.

About validation, we usually splits datasets into 3 parts :

Training set : used to train the models. (~80% dataset)
Testing set : used to test the trained models, this part is the one used to estimate the accuracy of the model. (~15% dataset)
Validation set : used at the very end, to validate the fact that our model works correctly. (~5% dataset)

In many cases, Testing and validation datasets are the same (because validation dataset often is not necessary).
In your tutorial, what they call validation set is what I usually call the testing dataset, and it is the same as the training set, so bascially all your sets are the same. (it is very very very uncommon to do so, but since the goal of the tutorial is to see how we overfit this set, it makes sense here).


Clustering algorithms I have sparse vectors and found that cosine similarity is very efficient to to measure the similarity. Now I want to cluster these vectors based on similarity. Hence, can someone please suggest/recommend clustering algorithms that make use of cosine similarity?

P.S.: I do not have a predefined number of clusters beforehand and want the clustering algorithm itself to decide it.

You can see your affinity matrix as a weighted adjacency matrix of a graph and apply modularity-based community detection algorithms on that. Just note that modularity based algorithms have resolution problem i.e. finding very small communities is difficult in presence of large ones.


How to build classifier if only 1 image per class available If I have only 1 image for each of 10 classes, what is the best way to build an image classifier? The images themselves are large (1200x1600) and of good quality. For example:

Or similar images from https://www.pexels.com/search/flowers/

From what I have read, neural networks need large amount of data for training. Which other machine learning methods can be used in such a situation? Basically, image similarity needs to be assessed. Can support vector machines be used here?

One image per class in way too less data for building any classifier. Rather than image classification, image similarity matching seems a better approach and so I would suggest you a Siamese network for one-shot learning.

A Siamese network is a class of neural network that contains two or more identical network and are used to differentiate between the input images by maximizing the minimum distance among the images. This technique is helpful for what we call as One-shot learning.

But if you only want to build a classifier, here are a few steps you can think about:

Try to increase your dataset size by adding a few more images to you training set.
Apply heavy Image-augmentation.
Use a Pre-trained model to try transfer learning.


Which algorithm to use for transactional data I'm given a Dataset of transactions and asked to find insights for businesses. I'm extremely new to ML / Data science and have only been experiencing with KMeans. The dataset has the following features

merchant ID
Transaction date
Military time
Amount
card amount paid
merchant name
Town
area code
client ID
age band
gender code
province
average income 3 months
card value spending
card tapped

Ignoring NULL data, what type of analysis can I do on this data? I have done KMeans on if a customer will spend excessive amount (more than median amount).

On this data, you can perform a lot of supervised learning. If you know, supervised learning is when the machine learns with data which has labels. In supervised learning, there are two subsets. Those are regression and classification. Classification is when you predict on something which is discrete, such as male or female, or survived or not survived. On the basis of regression, you can predict non-discrete things, such as the price of a house, or GDP of a country.
Based on your dataset, I think you can do a lot of EDA(exploratory data analysis) with classification. Maybe you can predict which gender buys more. There are many things you can do with the dataset, but here are the algorithms you can use.
If you have a small dataset, Logistic Regression and Naive Bayes are the best algorithms. For starters, the k-NN(k-nearest neighbors) algorithm is the best though. If you start getting into complex data, then Decision Tree is the best algorithm.
Now after all of these, there is the most complex algorithm(which is basically a bunch of decision trees mixed together) which is Random Forest. This algorithm is for if you have a really huge dataset with many labels.
Hope this helps!


LIME Random Forest explanations: I'm using LIME to explain my random forest model. Everything is working great. However, I don't quite understand the image that is generated. Taking the example from the Readme:

How can it predict poisonous with 1.00 but still have a gill-size=broad which has 0.13 probability? Or do I misunderstand the table?

I already asked this question in the github project, but didn't get any answer.

I think the answer mostly lies in the fact that these are just approximations and they're not super exact because of the small data set and nature of decision trees. The prediction was really 1.0 (I'm guessing all trees' leaves agreed entirely on the prediction).

If gill_size != broad, it would still be 1.0, you could say (1.13 is meaningless). But maybe if gill_size != broad and odor != foul you could expect the probability to be about 1.0 + 0.13 - 0.26 = 0.87.

A little more explanation is at https://marcotcr.github.io/lime/tutorials/Tutorial%20-%20continuous%20and%20categorical%20features.html


How Box cox and other transformations convert data into Normal Distributions? How Box cox and other transformations convert data into Normal Distributions ?

Box Cox and other transformations find the exponent needed to transform the data into a normal distribution.

Transformations happen by finding the estimate of $\lambda$ and some values in the neighborhood are chosen to transform the original data.

Once the data is transformed, we assume the data is normally distributed.

The original formula proposed by Box & Cox is:

$$
y(\lambda) = \left\{
        \begin{array}{ll}
           \frac{y^{\lambda} -1 }\lambda \quad if \lambda \neq 0; \\
            log(y), \quad  if \lambda = 0.
        \end{array}
    \right.
$$

Which basically turns into choosing $\lambda$ as a model parameter. This value is found by maximizing log-likelihood.


Mean across every several rows in pandas I have a table of features and labels where each row has a time stamp. Labels are categorical. They go in a batch where one label repeats several times. Batches with the same label do not have a specific order. The number of repetitions of the same label in one batch is always the same. In the example below, every three rows has the same label.

I would like to get a new table where Var 1 and Var 2 are averaged across repetitions of labels. In this example, I would like to average every three rows as follows:

Is this possible to do with some functions in pandas or other Pyhton libraries?

You can try like below with shifting the frame,

df['shift1'] = df.var1.shift(1)
df['shift2'] = df.var1.shift(2)
df['avg'] = df[['var1','shift1','shift2']].mean(axis=1)
df.iloc[range(2,n,3)]


Find threshold in large dataset I have a problem. I have a data set with some users and their ratings in several movies. The movies are separated into 19 genres.

I want to cluster the users by their preferences(ratings in the movies). The problem is, that I want to find a $threshold ( θ )$ to do the clustering, but I do not know how to do this, because the data are discrete and I cannot use the statistics methods that I know.
The threshold is the maximum distance that two users can have to be in the same cluster, like 2 users that likes the same genre movies or have little differences in their tastes.

I've tried to find a threshold using simple statistics. For example, for a user sum all of his ratings in a genre and divide the result via the number of ratings and find some means in some genres, but I didn't got an answer.

Note: I must use BSAS

Somehow you have to come up with some sort of numerical classification system for your movie genres.
I would start by creating a relationship tree between genres.  For example action movies and then action movies with comedy and then action movies with comedy with animation etc.

You could develop a whole Forest of trees that relate movie genres to one another.  You can then test the genres paths of individuals to compare.


Estimating panel model in R How can I estimate

$y_{i,t}=β_{i}x_{i,t}+ϵ_{i,t} \\
β_{i}=γ_{i}z_{i}+η_{i}$

in R ? Moreover, if I have splitted my data set to a train set which will contain 80% of all the i's and then I want to forecast the $y_{i,t}$ for the rest 20% i's  $\forall t$, how can I do it in R ?

Thank you in advance

For linear regression you want to use R's lm() function, like this:

my.model <- lm(response.variable ~ predictor1 + predictor2, data = my.data)

Look at the model using:

summary(my.model)

You can apply this model to a "test" dataset (your 20% split) by using predict(), like this:

predict(my.model, test.data)


Scikit-learn Random Forest - model changes as result of input scaling Everything that I've read about random forests has indicated that they do not require scaling of inputs and that scaling should not affect the construction of the model.  Here's a quote from another SE question (https://stats.stackexchange.com/questions/255765/does-random-forest-need-input-variables-to-be-scaled-or-centered):

  Random Forests are based on tree partitioning algorithms.

  As such, there's no analogue to a coefficient one obtain in general
  regression strategies, which would depend on the units of the
  independent variables. Instead, one obtain a collection of partition
  rules, basically a decision given a threshold, and this shouldn't
  change with scaling. In other words, the trees only see ranks in the
  features.

  Basically, any monotonic transformation of your data shouldn't change
  the forest at all (in the most common implementations).

Here's what I'm using currently.  If I remove the weights multiplier, I get a different model (i.e., different value from model.score and different tree depths) despite setting random_state=0 in either case.

model = RandomForestRegressor(n_estimators=10, criterion='mse', random_state=0)
weights = np.arange(1,self.x_train.shape[1]+1)[None,:]
# weights = [[ 1.  2.  3.  4.  5.  6.  7.  8.  9. 10. 11. 12. 13. 14. 15.]]
model.fit(self.x_train * weights, self.y_train)

By comparison, I noticed that if I use XGBRegressor rather than RandomForestRegressor, scaling does not change the model.

Is there an obvious mistake that I'm making or is the explanation above not correct?

After trying to recreate the issue with random numbers (and failing initially), I figured out that the problem comes from the fact that the x_train data that I'm using contains columns that have a very small, near-zero values.

To recreate, the first section is only run once:

scale = 0.0001 # making this larger eliminates the issue
x_train = np.random.uniform(0,scale,size=(1000,10))
y_train = np.random.uniform(0,1,size=(x_train.shape[0]))

Then using the same values for x_train and y_train, run the section below but with use_weights set to True and then False.

use_weights = True
model = RandomForestRegressor(n_estimators=10, random_state=0)

if use_weights:
    weights = np.arange(1,x_train.shape[1]+1)[None,:]
    model.fit(x_train * weights, y_train)
    prediction = model.predict(x_train * weights)
else:
    model.fit(x_train, y_train)
    prediction = model.predict(x_train)

print(prediction[0]) # changes based on use_weights value assuming scale is very small

As a side note, the y_train values for the real dataset are also very small and I have to multiply them by 100 or more to get the model to even run.  That is, it will not create any leaves at all without expanding the scale of the y_train values (confirmed by running the get_depth method on each tree).

I'm wondering, is this purely a numerical imprecision issue or is it something unique to the random forest calculations that are happening under the hood?


Padding the sentences is consuming huge memory I prepared a lstm model using tensorflow which has a max_sequence_length of 5000 and I'm padding the small sentences with 0's. I then deployed and tested the model with multiple calls at a time. I then encountered a memory error because the max_sequence_length is consuming a lot of memory.

Is there is any alternative to writing the model more efficiently using some thing as an alternative to padding?

Assuming you have tried alternatives like storing it in csr matrices in scipy, you can move away from padding to avoid memory issues by declaring your batch_size=1 and may be create batches using a groupby for equal length sequences. If you are using keras, then please check Masking.


Train classifier on balanced dataset and apply on imbalanced dataset? I have a labelled training dataset DS1 with 1000 entries. The targets (True/False) are nearly balanced. With sklearn, I have tried several algorithms, of which the GradientBoostingClassifier works best with F-Score ~0.83.

Now, I have to apply the trained classifier on an unlabelled dataset DS2 with ~ 5 million entries (and same features). However, for DS2, the target distribution is expected to be highly unbalanced.

Is this a problem? Will the model reproduce the trained target distribution from DS1 when applied on DS2?

If yes, would another algorithm be more robust?

Is this a problem?

No. not at all.

  Will the model reproduce the trained target distribution from DS1 when
  applied on DS2?

No, not necessarily. If

Balanced set DS1 is a good representative of imbalanced (target) set DS2, and
Classes are well-separated (pointed out by @BenReiniger), which holds easier in higher dimensions,

then model will generate labels with a ratio close to imbalanced DS2 not balanced DS1. Here is a visual example (drawn by myself):

As you see, in the good training set, predictions resemble the real-world ratio, even though model is trained on a balanced set, of course, if classifier does a good job of finding the decision boundaries.


Predicting a cyclic target I'm familiar with using trigonometric functions to transform cyclic variables for use as features in training a model (most commonly hour of the day or month of the year); I'm now trying to figure out the best way of doing this for using these types of variables as the target for my model. (Imagine a model predicting the month when a particular event is most likely to occur, for instance). Neither using a strictly-increasing representation (so that January, 2018 is close to December, 2017 but very far from January, 2017) nor treating the month as a categorical variable is ideal, but the trigonometric encoding done for features requires both sin and cos parts to have a unique representation of each month, so using one of the two isn't a workable approach either.  Is there a better option that I'm missing?

The relevant terminology is "multi-output" or "multi-target" regression/classification.  A survey article from a few years ago has some good discussion. This functionality is implemented in many packages (for instance, in sklearn (where it extends the functionality of other regressors, rather than being an independent implementation) and in R.


How do we get the baseline of each fold in cross-validation? Given that I am using Scikit learn and cross-validation and want to compare my accuracy result for each fold with my baseline

I am using 10-fold cross-validation and how for each fold I return the baseline for that fold. is it possible?

clf= RandomForestClassifier(n_estimators=100, random_state=20)
# 10-Fold Cross validation
scores = cross_val_score(clf, features, labels, cv=10)
scores

Result: array([0.45454545, 0.63636364, 0.8       , 0.8       , 0.6       ,
       0.6       , 0.5       , 0.9       , 0.66666667, 0.33333333])

I think you look for: sklearn.model_selection.cross_validate, see the docs here:

  Returns:
  scores : dict of float arrays of shape=(n_splits,)
  Array of scores of the estimator for each run of the cross validation.
  A dict of arrays containing the score/time arrays for each scorer is
  returned. The possible keys for this dict are:

  test_score The score array for test scores on each cv split.

  train_score The score array for train scores on each cv split.
  [...]

Minimal example:

from sklearn import datasets, linear_model
from sklearn.model_selection import cross_validate

# Data
diabetes = datasets.load_diabetes()
X = diabetes.data[:150]
y = diabetes.target[:150]

# Model
lasso = linear_model.Lasso()
cv_results = cross_validate(lasso, X, y, cv=10)

# Print keys
print(sorted(cv_results.keys()))
# Print test_scores
print(cv_results['test_score'])

Returns:

['fit_time', 'score_time', 'test_score', 'train_score']
[ 0.34557351  0.34848715  0.26654262 -0.01126674  0.24875619  0.08731544
  0.13386583  0.14000888  0.2873109   0.00960079]


predicition for a specific month I am attempting to build a predictive model based on the past historical data. I have details of specific machine failure based on the past year data. I have data from some months of 2016 and from 2017 January to November. I am attempting to predict if the machine fails in December. I have attached the past historical data based on the data transformation I performed. I am stuck at a point on how to build a model to identify if the machine throws up any repair or replace in next one month.

I need to build a classification model which will identify if the machine fails. I don't understand on how to convert this multiple rows to single row record for each machine. Do I need to create categorical for day and month. How would I represent each sensor data column. Essentially look at the right way of representing data so that it is easier for classification to predict for next one month.

Any sort of guidance would be of great help dataset.csv
 train.csv

train.csv contains the 1 month data of December for the machines which are already predicted. The target column is to predict in the month of December, if the machine requires a repair or replacement or no issue.

In essence you want to shift your outcome variable by 1, that is place december's outcome to november. In doing so you are saying that the values in the current month (november) caused the next month's (december) outcome. As to the question of processing your data that depends entirely on your environment / software used for the analysis, however they seem good as they are now.


What is the difference between data-driven methods and machine learning? I was wondering (about a more semantic question), is there a difference between data-driven methods and machine learning? Or is it more correct to state that machine learning is a category of data-driven methods (and what then are other categories)?

Based on the quotation you have added in your comments, data-driven approaches are approaches where you use data that describes past states ("historical data") to get a (not defined) system to give a desired output.

To understand whether this definition includes machine learning or not we will have to define "machine learning", and while there could be plenty of ways to define it I expect that it will be quite difficult to come up with a definition that does not include within it "Using a system that, based on given states will give a desired output".

Note that in this last definition I use "given states" and not "past states" as to include approaches such as online learning.

Bottom line is that unless you really want to hold to a narrow definition of "past states" it seems that machine learning approaches are a subset of data-driven approaches.


How to change the threshold value in orange? How can I change the threshold value for each classifier? In the ROC analysis curve we are allowed to remove the tick from "default threshold"

I don't think you can do it for classifiers or any other widgets (for example confusion matrix).

For classifiers, workaround would be exporting the data along with the precise classification values (via "Predictions" and "Save data" widgets) and using external tool to process them.


What is Deep supervision? I'm interested in segmentation models for medical imaging purposes.
When I looked at the state of the art, I fell on a paper on a new architecture, Unet++:

UNet++: A Nested U-Net Architecture for Medical Image Segmentation from Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang
at Arizona State University

Like Unet, it is has an encoder/decoder architecture with skip connections (adding fine-grained feature maps of the encoder to the decoder). However, in Unet++, the skip connections are nested and dense so that a model can improve its ability to capture fine-grained details.
The second difference with Unet is the use of deep supervision.
The papers says that deep supervision enables:

"the model to operate in two modes: 1) accurate mode wherein the outputs from all segmentation branches are averaged; 2) fast mode wherein the ﬁnal segmentation map is selected from only one of the segmentation branches, the choice of which determines the extent of model pruning and speed gain"

I didn't understand how deep supervision works and what are the benefits in the case of Unet++.
Can you explain how it works?
Thank you in advance for your help.

Partial answer: Quoting a paper
"The advantage of such deep supervision is evident: (1)for small training data and relatively shallower networks, deep supervision functions as a strong “regularization” for classiﬁcation accuracy and learned features; (2) for large training data and deeper networks deep supervision makes it convenient to exploit the signiﬁcant performance gains that extremely deep networks can bring by improving otherwise problematic convergence behavior"


How cluster a twitter data-set? I have a twitter data-set and I wanna extract their related topics. So, I decided to classify my Tweets into clusters using an unsupervised machine learning algorithm like k-means. This choice is made due the time consuming of the training process in the supervised approaches.

So, as a first step after cleaning my tweets, I will extract features (eg. Hashtags...) from them, and enrich them with side information from knowledge bases (eg. Wikipedia). Secondly, they will be represented in a Vector space. Next, using k-means and for given K=6 clusters, my tweets already enriched will be classified into 6 clusters.

However, I don’t know how to identify automatically the topics related to these clusters. Is there any solutions?

k-means is very sensitive to noise

because it is designed as a least-squares approach. Noise deviations, when squared, become even larger.

Twitter is mostly noise

Twitter is full of spam and nonsense tweets. These will be entirely unlike any other and thus have the largest deviations.

Chances are you get one "cluster" that contains almost everything, and the other k-1 clusters consist of a few tweets with their duplicates. Clusters are not topics. They are more likely to be duplicates than topics.

An appropriate clustering algorithm for tweets should probably discard 90% of tweets and produce thousands of clusters. But it will rarely be better than finding all tweets in common - most tweets only have 2-3 usable words.


What ML/DL approach better suits this problem? We have a huge dataset with us that looks like below

Factor -|- ... -|- Rank1 -|- Rank2 -|- Calls
A11-----|- ... -|-0.1234--|-3.2345--|- Cat A
A11-----|- ... -|-1.1234--|-0.2345--|- Cat B
A12-----|- ... -|-2.1234--|-3.2345--|- Cat C
A12-----|- ... -|-2.1234--|-3.2345--|- Cat C
...
A13-----|- ... -|-0.1234--|-3.2345--|- Cat A
A13-----|- ... -|-3.1234--|-0.1345--|- Cat B
A13-----|- ... -|-2.1234--|-2.2345--|- Cat C
A14-----|- ... -|-4.1234--|-4.2345--|- Cat C

and we have about 10 million of such data points.
We also have a test set with about .2 million data points where we need to accurately call them out into different classes.

At this point, we are trying a mix of K-means & Random Forest approach coded in (python2.7-sklearn) which gives us about 90% accuracy at classification but we wanna reach for more.

I am interested to apply some kind of deep learning approach to this and that's why I'm learning about TensorFlow. But every link I've gone though in deep learning talks (tf documentation + youtube videos) about CNN and image recognition (MNIST .. etc) only and I'm not finding any idea how to start solving this.

I'm looking for your suggestions and guidance on how to approach this kind of problem / What neurons to stack or How to build models for this kind of data? I'm willing to burn my share of the midnight oil on any links and suggestions offered on this but I need to know if I'm even thinking right by trying to solve it with DL or if there's any other approach for this kind of data which may work better, or to solve this what should I study or learn?

Edit 1:
So, I managed to go through the TensorFlow documentation for MNIST an perhaps I can see some correlation.

In the sense if I pass the rank1 & rank2 as input np.array for the 2 neuron input layer and each call as a one_hot np.array, where Cat A = [1,0,0],Cat B = [0,1,0],Cat C = [0,0,1] and put 3 neuron on the outermost layer each predicting one Cat(category) through a softmax function, will that work?

Even if so, what should I use as internal/hidden layers? Can I use the other factors as well in the input layers? Do I need to convert them into numeric(int/float)? Should I pass these rank 1 and rank 2 values to the input layer just as it is or should I engineer them anyway beforehand?

Yes, you can use DL network to solve this problem. It is a easy multi-classes classification task. And just a full connected network will do this work.

To construct the network I recommend using Keras which is easy to use. And before training the network, it is better to preprocess the data(standardizing the numeric column, and embedding the string column).

EDIT:

For the data normalisation you can reference this blog: Neural Network Data Normalization and Encoding, that show the basic methods and codes.


X, Y names in data-science I am from CS background and moving towards Data Sciences, I have came to know ML is highly influenced by Statistical Inference/signal processing. The X we use in data science is called input, feature set or independent variables and Y is called target, class, label. What do you call it? Are they different in market and field among practitioners?

A few names I found for X are:

feature vector
an input variable
an independent variable
an explanatory variable
an exogenous variable
a predictor variable
a regressor
a covariate


Why are convolutions still used in some Transformer networks for speech enhancement? So I’ve read in Attention is All You Need that Transformers remove the need for recurrence and convolutions entirely. However, I’ve seen some TNNs (such as SepFormer, DPTNet, and TSTNN) that still utilize convolutions. Is there any particular reason for this? Doesn’t that defeat the purpose of Transformers?

We find some justifications in the Conformer paper:

Convolutions are better than Transformers at detecting fine-grained patterns:

While Transformers are good at modeling long-range global context, they are less capable to extract fine-grained local feature patterns. Convolution neural networks (CNNs), on the other hand, exploit local information and are
used as the de-facto computational block in vision.

Together, they Transformers and convolutions work better than separately:

Recent works have shown that combining convolution and self-attention improves over using them individually [14]. Together, they are able to learn both position-wise local features, and use content-based global interactions.


Extracting patterns from the spatiotemporal movement of objects Say I have a database with x and y coordinates (30 times per second) for some objects in a 2d plane.
Is there a way to extract patterns from these coordinates?
If so, is there a way to visualize these patterns using for example vectors?

You could take the difference between coordinates occuring consecutively. This gives your displacement vector. Then create two plots one indicating the magnitude of displacement( $((\delta x_{t})^{2}+(\delta y_{t})^{2})^{1/2}$) vs time and another plot of the direction indicated by the angle of the vector from the x axis; given by $\theta_{t} = tan^{-1}(\delta y/\delta x)$.
Hence you could visualize the magnitude and direction with respect to time.


Regression - Unbalanced Categorical Features I have a data set that has some unbalanced categorical features. I would like to build a regression model to predict a label using machine learning (ML).

How do I handle data imbalances in categorical variables to avoid any bias in my ML regression model?

First, it depends on the number of samples and the degree of imbalance:

Small number of samples may cause slightly imbalanced features' categories to be trained very few times within the training stage
High imbalances can cause some categories to only appear in the training or in the validation/test stages

  As a rule of thumb, I usually join the categories up to a point their distribution in the train and validation datasets are similar.

While this can lead to some bias, it avoids problems on handling categorical features when deploying in production or when testing in the test dataset.

Other strategy is to define a threshold (e.g. 5%) in which you join all the features categories which their distribution is below such threshold.


NLP - Retrieval-based model My goal is to predict the most appropriate answer from an utterance, in a group of 21 potential answers. (I'm not sure the "question" is called utterance though. )

Example:

Utterance: How are you today?
Answers: Answer1, 2, ..., 21.

I have a training file with this format:

Utterance:
Answers: Good answer, wrong answer1, wrong answer2,..., wrong answer20.

My problem

For the first time, we have to make a prediction from a group of possible answers, and, thus, this is a MCQ form.

Any ideas how I could start the problem?

What I've done

For the moment, the only thing I did was to choose the answers from the 21 possible answers which had the highest cosine similarity with the utterance. (So, unsupervised). It's not that bad (24% against 1/21 at random), but I'm sure there are ways to make something really better.

What I don't want to do at first

Use a generative model which predicts a full sentence. I want to choose the best candidate amongs the 21 answers, and use the training file which can allow us to do supervised learning.

Since answers change between different questions, your problem does not fit 'regular' classification problems.
And due to textual nature of your input/output, regression is not the best fit either.
This leaves me thinking K-NN is a good way utilizing supervised learning.

I don't have any good reference for this but this approach makes sense to me:

1) Embed both questions and answers into same space(using TF/IDF and PCA for example).
2) For a new(unseen) question find near neighboring labeled questions(from the training set), using K-NN.
3) Get neighboring questions answers.
4) Use K-NN(Or other distance based method) to find nearest neighbor in unseen question answer choices.


Multiple keras models parallel - time efficient I am trying to load two different keras models in parallel. I tried to use the functional API model:

input1 = Input(inputShapeOfModel1)
input2 = Input(inputShapeOfModel2)

output1 = model1(input1)
output2 = model2(input2)

parallelModel = Model([input1,input2], [output1,output2])

This works but it does not run in parallel actually. Inference time is just the sum of each model's individual inference time.

My question is should this run concurrently?
I also tried to load them in different py files with gpu memory options. Still I haven't got parallelism (inference time is x1.5 for each model)

Is there any way to get inference time of both models as close to a single's model inference time?
Is the only solution to add a second gpu?

UPDATE: in different scripts they seem to be able to run in parallel, so there must be a way to efficiently run in python/keras as well.

As was suggested by Erik van de Ven, it sounds like running each model on a different process should provide the requested parallelism.

I guess you could either run the fit function for each model in a different process
Or you could even load them on different cpu cores:

with K.device('cpu0'):
    input1 = Input(inputShapeOfModel1)
    output1 = model1(input1)

with K.device('gpu0'):
    input2 = Input(inputShapeOfModel2)
    output2 = model2(input2)

model = Model([input1, input2], [output1, output2])

I haven't tried any of these though, so i'm not sure what would provide the best result