Create a continuous heatmap Most heatmap tutorials look at discrete data, where each cell has a well-defined boundary and a single value. How do you create a heatmap of continuous data, where individual points may be very close together without actually being identical?
Example: I have a set of 1000 map coordinates, all in the range (-8192, -8192) to (8192, 8192). They might look like this:
745.2218627929688,808.1373901367188
741.4943237304688,775.325927734375
757.3713989257812,806.7517700195312
-300.1320495605469,1425.9727783203125
-1845.8055419921875,2091.180419921875
-289.85308837890625,1429.4222412109375
-306.8935241699219,1434.9525146484375
-287.78253173828125,1439.9078369140625

The locations will tend to be clustered, and the clusters are the most interesting, but outliers are still of some value. What I'd like would be to show colours on an image that indicate this density.
It's highly likely that all I need is the right term to search for, but "heatmap" keeps finding discrete data tutorials only :)

What about imshow? Check these different interpolation methods which will allow you to define the "discreteness".
Example with quadric interpolation:
import matplotlib.pyplot as plt
import numpy as np

harvest = np.array([[0.8, 2.4, 2.5, 3.9, 0.0, 4.0, 0.0],
                    [2.4, 0.0, 4.0, 1.0, 2.7, 0.0, 0.0],
                    [1.1, 2.4, 0.8, 4.3, 1.9, 4.4, 0.0],
                    [0.6, 0.0, 0.3, 0.0, 3.1, 0.0, 0.0],
                    [0.7, 1.7, 0.6, 2.6, 2.2, 6.2, 0.0],
                    [1.3, 1.2, 0.0, 0.0, 0.0, 3.2, 5.1],
                    [0.1, 2.0, 0.0, 1.4, 0.0, 1.9, 6.3]])

fig, ax = plt.subplots()
im = ax.imshow(harvest,interpolation="quadric")

Output image:


Algorithm for backpropagation through time I am reading through this article trying to understand the bptt algorithm, in the context of an RNN. However there is one part I don’t understand:

layer_1_delta = (future_layer_1_delta.dot(synapse_h.T) +
    layer_2_delta.dot(synapse_1.T)) +
    sigmoid_output_to_derivative(layer_1)

# Located on line 99 in the article linked above

I can’t seem to figure out which mathematical formula this links to in the bptt algorithm to understand what it does, so could someone please provide it? Many thanks

As far as I can make out, this is simply the error aggregated at layer_1, meaning it is the sum of the errors from the current hidden layer, plus the error from all future hidden layers (in this case, that is only one layer: layer_2) plus the error from the output (prediction) layer - sigmoid_out.

I think that line of code (#99) is performing the loss update with respect to a chosen layer and all layers that are ahead in time - so closer to the network's output layer, but as we are moving backwards during backpropagation. It corresponds then, I believe, to equation 3.24 and 3.25 in Ilya Sutskever's PhD Thesis (on page 35).

So you are summing up the gradients of all layers ahead of the current layer, as well as the gradient of the output layer (the sigmoid). Equation 3.25 from the link above looks like this:

$$
\frac{\delta L}{\delta W} = \sum_{t=1}^{T-1} \bigg( W'^T (r_{t+1} \odot (1 - r_{t+1}) \odot \frac{\delta L}{\delta r_{t+1}}) \bigg) v_t^T +
\sum_{t=1}^{T-1} \frac{\delta log \hat P (v_t | r_{t-1})}{\delta W}
$$

  ... where , where $v_t$ are the input variables and $r_t$ are the RNN’s hidden variables (all of which are deterministic). (from page 34)

The explanation given in the thesis:

  The first summation in eq. 3.25 corresponds to the use of $W$ for computing $r_t$, and the second summation arises from the use of $W$ as RBM parameters for $log \hat P(v_t |r_{t−1})$, so each $∂log \hat P(v_{t+1}|r_t)/∂W$ term is computed with CD (contrastive divergence). Computing $∂L/∂rt$ is done most conveniently with a single backward pass
  through the sequence. It is also seen that the gradient of the RTRBM ( Recurrent Temporal Restricted Boltzmann Machine) would be computed exactly if CD were replaced with the derivatives of the RBM’s log probability.

bold text added by me

The main leap here to relate to your example blog is that the it equates the log probabilities of a Restricted Boltzmann Machine to the loss of a generative RNN. If you are interested in more details, I would recommend reading either all of Chapter 3, or perhaps just sections 3.9 and 3.10.

I posted that equation because I could see a version that so closely matches the code in that blog in either of the original papers that propose backpropagation through time. Additionally, none seem to be free to read.
You can find them link on the relevant Wikipedia page.

Have a look at this related paper, written by one of the original authors of BPTT. Backpropagation Through Time: What It Does and How to Do It.


Image Embeddings - Negative Sampling and Imbalanced Class Issues I am using the negative sampling approach used in Word2Vec to train some image embeddings. From what I have read, for every positive example, we are creating a number of negative examples.

Question: Why do we use an imbalanced dataset here? Presumably we will get the normal issue where the algorithm ends up predicting the negative label to minimise the cost function? I understand that the aim isn't really to use it as a prediction model, but rather to extract the embeddings, but what is the benefit of having an imbalanced class here?

I don't think we are creating a number of negative examples for every positive example. Negative sampling is done to efficiently compute the softmax.

Word2vec tries to maximize the similarity for words in similar contexts. For e.g. given drink the NN should predict water with maximum probability.

Suppose you have V (typically > 1 million) words in your training data, so the last layer of your model has V neurons. For every word in your training data, you would have to compute output from all the V neurons (to compute softmax). It is computationally very expensive. Negative sampling is one way to address this problem. Instead of computing the all the V outputs, we just sample few words and approximate the softmax.

Negative sampling can be used to speed up neural networks where the number of output neurons is very high. Hierarchical softmax is another technique that's used for training word2vec.


Data model and algorithm for recommending "related" interests On my app, when a user selects an interest (example: ios), I'd like to show related interests (swift, xcode, apple, etc). I have a list of around 700 interests/tags (about 300 of them can be classified as tech). How do I design the system so that when a user selects one of the 700 items, the system can show other relevant items? Note that there is no "learning" involved in this. I'd just go and classify each one myself, and this list will be mostly static.

This is how I have been thinking about it:

I could assign a list of categories to each interest. These categories go from narrow to broad. The broadest category is just "tech", so if a user selects "ios", I could right away eliminate the 400 non-tech interests. Now I can fine-tune by looking at narrower categories (example: "mobile-apps", so I could recommend things that are categorized as "mobile-apps". I could go fine-tune further by looking at a narrower category like "ios-apps". So when a user selects an interest, I start by searching for items with the narrowest categories, and if I need more results, and I need to search for more, then, I go broader. To get started with this, I could simply have up to three categories (narrow, medium, broad) for each interest. Once I identify some categories, I can assign them as narrow, medium, or broad to each one. Now for any given interest, I can find out what other interests share the same narrow, medium, or broad categories.
The other approach I was thinking was to maintain a list of related interests for each interest. This requires more initial work on my part, so for each of the 700 interests, I'd find the 5-10 other interests that relate to it best, so for example, "ios" could have related interests like [swift, xcode, iphone, ipad], and so when a user enters ios, I just have to look up this list. I could maintain this list in a relevancy order. This approach is tedious as when I want to add a new interest to my list of 700 interests, I'd have to go through each and see if this new interest needs to be added as a related interest.

My question is--am I trying to reinvent the wheel? Are there any standard data models and algorithms to do this type of thing? I have been trying to Google this and the results seem to indicate things like interests graphs, cluster algorithms. I have no background in data science or machine learning. Any input is appreciated!

I think you could use these category interests as implicit ratings, and use the Implicit package to do some collaborative filtering and recommend interests based on what other users have selected.

I'd suggest you do that instead of manually filtering out interests using your own domain knowledge. A lot of graphic designers use Apple products, and could be interested in "ios" and "painting". Its best to use the data instead of building a model based on our own understanding, or what we think is right.


What should be the training frequency of a rnn model for timeseries prediction? If I use a rnn model for time series forecasting how frequently do I have to retrain the model.

It depends on any number of factors. What kind of accuracy are you currently getting? How often are you getting new predictions from your model? Do you have human intervention at some point to check your errors, relabel, and improve your training sets?

It's also possible that you don't need a schedule, per-se. Depending on your data, the pipeline and the programming of your model, you could also just have continuous learning built into your model with a new release of said model after every epoch. So, really, a lot of the answers you are seeking are very specific to your implementation.


Object tracking in 2 consequent frame I have a video that consist of a lot of objects. I detect these objects correctly in every frame. For example you can see here 2 sequence frame detection of objects:

Frame 1:

Frame 2:

I want draw the path of all objects, but I don't know how.
the main problem is:

How can I find that the object in the next frame is the same object in the previous frame?

If your objects move slow enough you can just find the nearest object on the other frame. In your case the objects move too fast. If you significantly increase the frame rate you will be able to see a one pixel movement. Then you just find the closest object on the previous frame and this identify the same object. If you want to have a movement on the scale you showed you should track the object across multiple frames until you reach the need distance. And you will not only have a movement vector, but also a full path from one point to the other.

If you cannot increase the frame rate the task becomes complicated even for humans. I had trouble identifying the same objects. It means that the task will be really hard for an algorithm. You can still try the nearest neighbor, but it will have much worse results than with a high frame rate. Any additional information to differentiate the objects will significantly improve matching, like color and shape.

After nearest neighbor you can also run an algorithm to minimize a loss function. Where your loss is a distance between the matches and a penalty for non matched objects and a huge penalty for same matching.  You minimize by something like evolutionary algorithm, or empirical hill descent. The variables are the matched object number from the other frame for each object on this frame.


Digitize graphs. Pull out data from points in a scatter plot I have a project where I would like to extract data from a series of scatter plots that are image files (Jpeg or png). The plots are similar but the axes scales are not always exactly the same. I have looked online and there are some web app solutions that are very “manual” and require a lot of clicking. This would get tiresome if there are a lot of graphs.
Does anyone know of any solutions that are partly or wholly automated? A python API solution would be ideal but other languages are also ok.

Digitizing Graph. I know a tool that can semi-automatically extract data, (scattered points and curves) from graph images within a few clicks. But it is not in Python.
It is PlotDigitizer.com. I have used the tool and based on my experience it is better than some others I have used earlier.
Here is a simple guide to use it:

Upload the image graph
Calibrate both axes graph - you have to do it manually
Pick the color of points or curves you want to extract
Select auto-extracting algorithms, e.g. points, curves

Done. You have extracted the data points.
You can find a more detailed guide on its website: plotdigitizer.com.


Skewed Data doesnt follow normal distribution My data (continuous) is highly skewed and it doesn't follow the normal distribution. Using sns.distplot I found out that exponweib fits the data better.

How to deal with this?

My end goal is to use the data for machine learning model (SVM).

To address skewed data you can do data transformation like logarithmic transformation, squared transformation etc.

Alternately you could try non-parametric machine learning algorithms that does not have any assumption of normality of x variables.

Hope this will help....


How to use the fillna method in a for loop I am working on a housing dataset.  In a list of columns (Garage, Fireplace, etc), I have values called NA which just means that the particular house in question does not have that feature (Garage, Fireplace).  It doesn't mean that the value is missing/unknown.  However, Python interprets this as NaN, which is wrong.  To come across this, I want to replace this value NA with XX to help Python distinguish it from NaN values.  Because there is a whole list of them, I want use a for loop to accomplish this in a few lines of code:

na_data = ['Alley', 'BsmtQual', 'BsmtCond', 'BsmtExposure', 'BsmtFinType1', 'BsmtFinType2', 'FireplaceQu', 'GarageType',
           'GarageFinish', 'GarageQual', 'GarageCond', 'PoolQC', 'Fence', 'MiscFeature']

for i in range(len(na_data)):
    train[i] = train[i].fillna('XX')

I know this isn't the correct way of doing it as it is giving me a KeyError: 0.  This is kinda like a pseudocode way of doing it to visualize what I'm trying to accomplish.  What is the way to automate fillna('XX') on this list of columns?

what you are looking for is replace().

And you don't need to write all the columns you can iterate over the columns name simply.

for col in train:
    train[col].replace("NA","XX",inplace=True)

You can do it on all the dataset in one line:

train.replace("NA","XX", inplace=True)

Or on specific columns:

for cols in na_data:
    train[col].replace("NA","XX",inplace=True)


How to draw convolutional neural network diagrams? I have to draw a CNN diagram similar to this:

I tried all the tools mentioned in https://datascience.stackexchange.com/a/14900, but there is no easy way to do it.
Is there any automated way to do it? Or do I have to do it manually.

In addition, is it possible to draw this?:

I found http://alexlenail.me/NN-SVG/LeNet.html, however, in this the input to each layer is assumed to be a perfect square. Thus, I cannot draw above diagram.

As to your first example most full featured drawing software should be capable of manually drawing almost anything including that diagram.
For example, the webpage "The Neural Network Zoo" has a cheat sheet containing many neural network architectures. It might provide some examples. The author's webpage says:

Djeb - Sep 15, 2016
Amazing. What software did you used to plot these figures ? Cheers !
Fjodor van Veen - Sep 15, 2016
I drew them in Adobe Animate, they’re not plots. Yes it was a lot of work to draw the lines.
Garrett Smith - Sep 15, 2016
Are your excellent images available for reuse under a particular license? Do you have an attribution policy?
Fjodor van Veen - Sep 16, 2016
As long as you mention the author and link to the Asimov Institute, use them however and wherever you like!

As for general automated plotting a commonly used package for Python is Matplotlib, more specific to AI, programs like TensorFlow use a dataflow graph to represent your computation in terms of the dependencies between individual operations.
TensorFlow computation graphs are powerful but complicated. The TensorBoard graph visualization can help you understand and debug them. Here's an example of the visualization.

A combination of automated output and manually annotating some of the details can produce sophisticated images not available from any one program.
This search turns up 1000's of diagrams from which you can obtain ideas to model your own images.


Binary classification: is there a tradeoff between predicting class 1 vs 0? I have been digging much more in detail into classification performance metrics lately to get my head around the 'dynamics' of classification algorithms. What I have noticed is that in binary classification problems, I have so far NOT seen an algorithm that does well predicting both classes at the same time.

To me, an algorithm is considered having an 'edge' for a given class if the accuracy on that class is higher than the proportion of that class in the dataset. E.g : if we have 20% Class A and 80% Class B, I would consider the model being 'performant' if accuracy on class A is >20%, or if accuracy on Class B is >80%.

The actual performance metric I calculate at in practice are: (% accuracy Class A-20%) and (% accuracy Class B-80%) for the example of a 20/80 distribution.(% accuracy Class A)=(# corrrecly predicted points A/# points A).
Any value of this metric above 0 means the model has an 'edge' predicting that class.

However, what I have noticed is that it is extremely rare that I find values above 0 for both classes. Most of the time (if not always), when there is an edge in a class, there is a somewhat equal or larger loss in the other class.

Formally, taking the sum of (% accuracy of Class A minus 20%)+(% accuracy of Class B minus 80%) results usually in 0 or less. This means you 'gain power' predicting a class, but you lose that power, or more, for the other class. Overall I like this metric because it beats accuracy in case of a majority class 'vote' for a biased model. In case of the majority class bias, this metric would return 0 (try it out you will see, it will end up being -20%+20%=0)

Anything above 0 would be a good model to me. I then thought meta ensembles are the natural progression of this, where u 'just' need to combine the powers of models that have an edge on each class. Hopefully the meta-ensemble will have a value above 0 for all classes for my made up metric.

Am I misinterpreting/understanding anythere here? Is this normal behaviour for binary classifications? Is it not possible to be able to predict both classes above their distribution in the dataset?

Thanks a lot for your enlightments.

P.S: In scikit-learn i always use class_weight='balanced' to avoid balancing/majority class issues etc.

To me, an algorithm is considered having an 'edge' for a given class if the accuracy on that class is higher than the proportion of that class in the dataset. E.g : if we have 20% Class A and 80% Class B, I would consider the model being 'performant' if accuracy on class A is >20%, or if accuracy on Class B is >80%.

Consider the following. Your 80% class is 0. You predict a constant 0 for all datapoints. You have 80% accuracy already with no modelling done and 100% accuracy on the majority class. If you fit a model to that data, you'll get an accuracy above 80% on the majority class and lesser accuracy on the minority class, and that's not a good performance which debunks your theory of a good metric.

The case above is from personal experience, seeing your results makes me think if your datapoints are really separable.
It is possible to have accuracies above the distribution marks if your features are good enough for model to use to separate each class from the other which in reality is feasable.

And again the choice of the metric in binary classification will always be a business problem, what is the cost of misclassifying a 1 as a 0 and a 0 as a 1? If the answer to these two questions is not the same, you should have some specific metrics for it.

  Hopefully the meta-ensemble will have a value above 0 for all classes for my made up metric.

It really depends on how diverse your base models are. But in general, meta-learners made out of weak learners outperform a single complex model. So a sort of a yes to your question.


Looking for an algorithm that correctly clusters visually separable clusters I have visualized a dataset in 2D after employing PCA. As 2D visualization shows in figure, there is a good separation between points (A, B). Now, I want to use a metric which can separate these points (between these 2 PC components not in main dataset) too. I mean have separation between these PCA components without visualization. I used some clustering methods but they raise false positives. I mean they miss cluster many points.

Also, as shown in histogram there is a gap between points A,B. Does this help in devising any metric?

I will be so grateful if you can introduce me any method and algorithm to be able to do separation between A and B.

With appropriate parameters, DBSCAN and single linkage hierarchical agglomerative clustering should work very well. Epsilon=0.2 or so.

But why? You know the data, just use a threshold.

If you just want an algorithm to "confirm" your desired outcome then you are using it wrong. Be honest: if you want your result to be "if $F-factor-1 > 1.5 then cluster1 else cluster2", then just say so, instead of attempting to find a clustering algorithm to fit to your desired solution!


Does a precision score increasing with a higher number of folds mean the model will improve with more data? I have been working on a pretty simple text classifying module (tfidf + Random Forest).
My manager insisted on using a simple .7/.3 split rather than doing cross validation, then was adamant about putting the model trained on 70% of the data in production (rather than the model trained on the whole thing). Her rationale was that the model would be more "predictable" and that anything we would gain from adding the remaining 30% would be negligible.

Out of curiosity, I ran a few test using cross-validation. First with 3 folds, then incrementally up to 10. With every increase in fold, the model's precision increases (first quite strongly: there's a 3 point increase in precision from 3 folds to 5 folds, the gains then become more marginal with each fold but still add up to a 5 point boost when reaching 10 folds). The variance between the cross-validation scores is always very low.

Am I right to assume that this can be interpreted as meaning the model is very likely to benefit from being trained on the whole dataset rather than on just 70%?

I understand there may be a risk of overfitting on the training data but a) I don't really see how the risk would be significantly mitigated by training on 70% of the data only, b) these are quite formal, standardized communications and unseen data is unlikely to differ significantly from the training datasets.

It may or may not indicate that more data will help.

The correct way to see if model performance increases with more data is to create a learning curve. Train your model on a small subset of your data with cross-validation, and then keep increasing the amount of data used. This is not just reducing/increasing the number of folds. It is actually limiting yourself to a small amount of data for your 10-fold cross-validation and increasing it, evaluating the performance at each increase. Here's more about a learning curve.


Techniques for Cluster Analysis of a Very Large (n=140000) Binary Dataset in Python? In essence: what techniques in Python are possible to find clusters/trends in a very large categorical dataset?

My very large dataset (140000 rows/observations, 80 variables) of categorical data has been re-coded using one-hot encoding, so they are all binary (e.g ethnicity_black, ethnicity_asian). This dataset is for instances of police use of force in the UK.

I was planning on doing agglomerative hierarchical clustering on it to find patterns (via cluster description) in use-of-force events, but I cannot accomplish this as the distance matrices are always too large and keep crashing. I was using Gower's distance since they are all dummy variables, and the gower package.

I have tried doing dimensionality reduction via MCA (like PCA but for categorical variables) but that simply reduces the number of columns, and the distance matrix is still too huge (140000x140000 requires 72.7Gigabytes of RAM).

I then tried reducing to the first 10000 rows, but that takes up 10000x10000=100000000 (100 million cells), and exceeded the recursion depth limit when constructing the dendrogram. Not sure if it is appropriate to simply take a tiny random subset of the 1400000 rows? Since that might be unrepresentative in terms of revealing patterns.

As such, I'm looking for other clustering methods besides hierarchical clustering, since K-means only works for continuous data, as does DBSCAN, since they use Euclidean distances. Was wondering if you guys have encountered similar situations before in your work, and could share about other techniques or possible workarounds in this case?

tldr; K-Means and DBSCAN seem to be for continuous data, and hierarchical clustering isn't very scalable to a large dataset due to the dissimilarity matrix required. I'm exploring K-modes but not sure how to interpret the output as well...

I did this MCA - Hierarchical Clustering technique before but in R, following the methodology used in this paper closely: https://erf.org.eg/wp-content/uploads/2019/11/Fateh-Belaid_Manuscript_FB_10-09-2019_FB.pdf

However, even that workflow doesn't work for my current dataset as it's simply too big.

Any advice would be appreciated...

Thank you very much!

Since you have binary data, you can use hamming distance as your similarity metric for clustering.
Then you have to handle the size of your data. Here are couple of options:

Rent a bigger computer or a cluster of computers from cloud computing service.

Treat the problem as a streaming data problem. Use "online" techniques to load and fit only a portion of the data at a time.


Is feature scaling at all needed for a feature set with a single feature? I understand that feature scaling is required to bring features in different magnitudes on a common scale so the model is not biased towards features with higher magnitudes. But if there is only a single feature in a feature set. For example, I have a quantity (events over time) on a time series and I'm forecasting for the future. In this case, do I need feature scaling (normalization or standardization?) or will it be unnecessary?
I searched online but couldn't find resources talking about the case where there's only a single feature in the feature set.

I understand that feature scaling is required to bring features in
different magnitudes on a common scale so the model is not biased
towards features with higher magnitudes. But if there is only a single
feature in a feature set. For example, I have a quantity (events over
time) on a time series and I'm forecasting for the future. In this
case, do I need feature scaling (normalization or standardization?) or
will it be unnecessary?

Usually we would try to do feature scalling although the idea is to remove the dependence of absolute magnitiude in affecting  convergence. it will help if the distribution is made gaussian with zero mean and unit variance. Since the gradient will be lower the stochastic descent will be a lot smoother.


How to increase sales and revenue of a Client? I was asked this in an interview for a Data Scientist position:
Lets say Holland and Barret came to you and said they'd like to increase their sales and revenue. How will you go about it?
My answer wasn't hitting the mark or touching the points the interviewer was looking for. How to go about answering this?

I think the question was asked to see how would you approach the problem. In similar questions, there is not a single answer, and the interviewer does not expect a certain answer instead expects a reasonable approach by you. It is like the famous interview question "How many golf balls can you fit into a swimming pool?". Such a question is asked to see the analytical thinking of the interviewee.
A structure of the answer may be like,

Churn analysis: understand the reason why customers "not coming" anymore, by doing so you will be able to reduce it.
Do a market analysis to understand the future state of the market to adjust the strategy and products according to the market.
Understanding customer segments (This is one of the most important implementations since you can come up with creative ideas after understanding the segments). Then, suggest the relevant products to relevant segments.
Understanding the competitors and their products. What are the differences between their strategy and ours, their products and ours?
Understanding the market size thus keep the predictions in a realistic range and to make solid steps.
Understanding costs and putting forward optimizations to reduce them.
etc.

These will make them sure that you have good knowledge over-analyzing the company, its strategies, products, etc. The above-mentioned solutions will state them you are not just able to increase the products and revenue, but also decrease the costs to maximize the profit. In this way, you can be a consultant to them.


File path encoding to feature I am trying to find some sort of encoding algorithm that would allow to transform system file paths eg. "c:/users/file1/subfile2/targetfile" into a feature that I could use in machine learning like a float value. It would be best if the folders/files located in the end of the path would have a smaller impact on the output value.
I am thinking maybe some combination of hashing and arithmetics but not sure for now.
I tried looking for something like that in scientific papers and on the internet in general but with no success.
Have anyone ever done something like that or similar?
Sorry if there are some stupid errors in my question. I am just starting my data science adventure.

You may encode each level and concatenate -
If we ignore the path till file1 as its same across all the names.
Then we need 1 digit for subfile and 2 digit for targetfile

c:/users/file1/subfile1/targetfile_0  - [0 00]
c:/users/file1/subfile1/targetfile_1  - [0 01]
c:/users/file1/subfile1/targetfile_2  - [0 10]
c:/users/file1/subfile2/targetfile_0  - [1 00]
c:/users/file1/subfile2/targetfile_1  - [1 01]
c:/users/file1/subfile2/targetfile_2  - [1 10]

Another approach can treat all paths as a feature. With this, you may try all the many encoding approaches e.g. Here.
+-----------+------------+--------------+
| Feature#1 | Feature#2  |  Feature#3   |
+-----------+------------+--------------+
| File_1    | Subfile_1  | Targetfile_1 |
| File_1    | Subfile_1  | Targetfile_2 |
| File_1    | Subfile_1  | Targetfile_3 |
+-----------+------------+--------------+


Binary Clasification Accurary newbie speaking:

Commonly we can say that accuracy is defined as total positive/ total nbr cases.

But I read that, when it is a binary classifier we should consider: TP+TN/ total nbr cases.

Can someone explain to me why?
Thanks

Accuracy is commonly defined as total nbr of correct predictions / total nbr of predictions.
Imagine a binary classification of color in: White or not White.

Since it's binary, all the white Classifications - a True Positive classification are as important for accuracy as a - True Negative classification, after all, the classification is also correct, hence,
TP+TN/ total nbr of predictions.


Python or R for implementing machine learning algorithms for fraud detection I was wondering which language can I use: R or Python, for my internship in fraud detection in an online banking system: I have to build machine learning algorithms (NN, etc.) that predict transaction frauds.
Thank you very much for your answer.

I would say that it is your call and purely depends on your comfort with (or desire to learn) the language. Both languages have extensive ecosystems of packages/libraries, including some, which could be used for fraud detection. I would consider anomaly detection as the main theme for the topic. Therefore, the following resources illustrate the variety of approaches, methods and tools for the task in each ecosystem.

Python Ecosystem

scikit-learn library: for example, see this page;
LSAnomaly, a Python module, improving OneClassSVM (a drop-in replacement): see this page;
Skyline: an open source example of implementation, see its GitHub repo;
A relevant discussion on StackOverflow;
pyculiarity, a Python port of Twitter's AnomalyDetection R Package (as mentioned in 2nd bullet of R Ecosystem below "Twitter's Anomaly Detection package").

R Ecosystem

CRAN Task Views, in particular, Time Series, Multivariate and Machine Learning;
Twitter's Anomaly Detection package;
Early Warning Signals (EWS) Toolbox, which includes earlywarnings package;
h2o machine learning platform (interfaces with R) uses deep learning for anomaly detection.

Additional General Information

A set of slides, mentioning a variety of methods for anomaly detection (AD);
A relevant discussion on Cross Validated;
ELKI Data Mining Framework, implementing a large number of AD (and other) algorithms.


same or different number of epochs for dataset of different sizes? I am trying to fit a DL model on a dataset, after a while of parameter tuning, I have determined the optimal number of epochs.  Now suppose there are new data available and I would like to include them to re-train the model, would I need to set the number of epochs to be exactly the same, or make it smaller such that I have equal number of gradient updates?  Both seem to make sense to me, but I am not sure which would usually work better.
I understand that a better way would be to use train-test splits to determine best epochs, but since data are super limited, I would prefer not to reserve any data if possible.
Any idea or suggestion would be much appreciated, thank you!

More data could potentially lead to more epochs required to find the best model. The number of epochs to reach the minimum loss will vary depending on your hyperparameters, dataset, and initial weights. I would get out of the mindset of trying to find the exact number of epochs required. Most popular deep learning libraries will allow you to stop training if the loss converges after a certain length of epochs. Additionally, if you overfit, you can create a callback function to save the best model only.


how can I convert tensor shape to numpy array? This is what my data looks like:

I want it to be numpy array, instead of a tensor, so that i can convert it to a dataframe. Is there a way to do so?
thanks in advance!

Assuming that these are pytorch tensors, you can convert them to numpy arrays using the .numpy() method. Depending on whether your tensors are stored on the GPU or still attached to the graph you might have to add .cpu() and .detach().


Do models without parameters exist? I am reading "A Course in Machine Learning" and, in chapter 2, the author says:

"For most models, there will be associated parameters. These are
the things that we use the data to decide on. Parameters in a decision
tree include: the specific questions we asked, the order in which we
asked them, and the classification decisions at the leaves."

My question is about the first sentence. Is there any model in machine learning that does not have parameters? I can't think of any. For sure there are models without hyperparameters (for instance, the linear model does not contain any hyperparameter, but it still contains 2 parameters, the slope and the y-intercept).

If such parameterless models exist, what are their purpose then? Isn't it the whole point of training to tune a model's parameters?

Is there any model in machine learning that does not have parameters?

Yes. k-nearest neighbors  is parameterless (there is only a single hyper-parameter $k$).

  If such parameterless models exist, what are their purpose then? Isn't it the whole point of training to tune a model's parameters?

Exactly: such models require no training at all. k-NN in particular relies on knowing the data set upon prediction. Anything close to "training" this model would be pushing data points to a set, but these do not count as parameters.


Skewed two class data set Is there any theory on the influence of skew in the data set on the performance of binary classifiers? At work, we are doing abuse detection, the negative population is regular logins, and the positive population is attack logins (account take over = ATO).

However, the frequency of ATO logins is 1/50,000 or less. So, we have a very skewed natural data set. Should I "unskew" my training data set by downsampling the legit logins? How much can I do that and still keep a model that will work well on the actual data? Any theory behind that?

It is typically called a class imbalance issue, where the occurrence of a label happens so infrequently that makes predictions unreliable.

For instance, if I know Vancouver, Canada rains 85% of the time in winter, I would simply predict that it is raining when it is winter + vancouver. You don't want your algorithm to favour one label over another because one label predominates.

One common strategy would be resampling. If you have enough data, downsampling could make more sense as (oversampling requires the creation of synthetic data (e.g., SMOTE, etc.)). so that the algorithm can properly learn the difference between the two classes and wouldn't favour one over another. 50-50 split between the two classes would probably be a good starting point, but it also depends on what is available. You still want your negative labels to be representative, and even an 80-20 split would be a vast improvement already.

Another common solution is to increase the penalty of incorrect predictions. The way to think about it, false positive = admin spending time investigating a false alarm; false negative = rouge activities went undetected. For different business, one cost could be more severe than another so you could potentially say getting a false negative is 1000x more costly to business, etc.

Not knowing your data, those are probably the first two things (separately or together) to try. Most ML packages could handle either strategy rather easily so that's why I think those are reasonable things to try.

There are many thorough articles and tutorials with additional strategies available. Try the keywords anomaly detection and class imbalance and it seems to give me some pretty good results.


Feature extraction from the text I am a newbie in machine learning but I have a coursework to create program that can extract some concrete features from the given text.

For example:
If I want to extract number of red apples and green apples, I will extract 3, 5 from "On that tree 3 red apples and 5 green apples grow" and 10, 1000 from "In the box there are 1000 green apples and 10 of red".
Actually the real examples can be more complicated like a extracting a car specification from the ad.

The concrete features are known before the data set processing.

Could you suggest what kind of algorithms I should use? Previously, I used only linear regression.

I guess if you know what you want to extract you can just find it using regular expressions for integers or car specifications.


Attention mechanism in Tensorflow 2 In the past days, I read up on the theory behind attention, when to apply it and what types there are. I think I have a decent first understanding of the concept, but now I would like to apply some of the insights I got to my own project and I find myself stuck with the implementation of attention in TF. (Quick Link to TF Attention)

The attention layer requires me to provide at least the queries and values. Correct me if I am wrong already, but this is my idea of what they are:

Queries: These are the hidden states of my decoder
Values: These are the hidden states of my encoder

So far so good. The thing I am struggling with is the fact that I have no idea where the hidden states of my decoder might come from. I would like to implement a self-attention mechanism. So my decoder hidden states are generated dynamically and I cannot know them before actually applying the attention layer. The example provided in the docs was not helpful for me, because it focused on a problem where I already have some query sequence.

Apart from whether the mentioned TF attention layer is applicable for self-attention, how do I interpret the different inputs?

In self-attention, it is not the decoder attending the encoder, but the layer attends itself, i.e., the queries and values are the same.

In practice, this is usually done in the multi-head setup. You can view that as every head focusing on collecting different kinds of information from the hidden states. In multi-headed attention with $H$ heads, you first linearly project the states in $H$ query vectors, $H$ key vectors, and $H$ value vectors, apply the attention, concatenate the resulting context vectors and project them back into the same dimension.


Is generalizing a model, then removing the generalization good for FFNNs? If one is training a basic FFNN (Feed-Forward Neural Network), one would apply regularizations like dropout, l1, l2 and gaussian noise, so that the model is robust and gives better results for unseen data. But my question is, once the model gives fairly good results, isn't it advisable to remove the reguarizations then train the model again for some time, so that its predictions are more accurate?

L1 and L2 regularizations matter only during training, they are a way to update the Network's weights in (hopefully) the right way. Once you use your model for prediction that doesn't matter anymore.

Dropout is active only during training. Once it's done, the Network usese all the trained nodes to make a prediction.

In other words, no need to remove regularization techniques manually.


Can anyone explain me the fisher score working I have been working on feature selection and I wanted to know what does fisher score tell us about the data which helps us in feature selection.

From Feature Selection for Classification: A Review (Jiliang Tang, Salem Alelyani and Huan Liu)

  Fisher Score: Features with high quality should assign similar values to instances
  in the same class and different values to instances from different
  classes.

From Generalized Fisher Score for Feature Selection (Quanquan Gu, Zhenhui Li, Jiawei Han)

  Fisher Score: The key idea of Fisher score is to find a subset of features,
  such that in the data space spanned by the selected
  features, the distances between data points in different classes are
  as large as possible, while the distances between data points in the
  same class are as small as possible.

If you are interested in the mathematical explanation, I would suggest the second paper, on section 2 (A Brief Review of Fisher Score)


Transitioning from a python script for data transformation to BigQuery So I have a dataset spread over multiple and ever-growing excel files all of which looks like:

email
order_ID
order_date

user1@email.com
1234
23-Mar-2021

user2@email.com
1235
23-Mar-2021

user2@email.com
1236
23-Mar-2021

user1@email.com
1237
24-Mar-2021

user3@email.com
1238
28-Mar-2021

End goal is to have two distinct datasets as:
First one being Orders: (Public. For analysis, trading emails with user_IDs for anonymity and marking returning for further analyses)

user_ID
order_ID
order_date
is_returning?

1
1234
23-Mar-2021
0

2
1235
23-Mar-2021
0

2
1236
23-Mar-2021
1

1
1237
24-Mar-2021
1

3
1238
28-Mar-2021
0

Second one being: Users (Private for retaining users_info. Have other columns besides email as well but idea is the same)

user_ID
email

1
user1@email.com

2
user2@email.com

2
user2@email.com

1
user1@email.com

3
user3@email.com

In pandas, I have following procedure to go about:

Read all files:
 input_directory = 'Data/'
 files =  os.listdir(input_directory)
 files

Combine them:
 all_data = pd.DataFrame()
 for df in data_dict.values():
     all_data=pd.concat([all_data,df]).drop_duplicates().reset_index(drop=True)
 all_data = all_data.drop_duplicates()

Assign user IDs:
 all_data['user_id'] = all_data['email'].factorize()[0]

Assign returning flag:
 all_data['is_returning'] = all_data.user_id.duplicated().astype(int)

Pushing both to BQ:
 columns = #list of private columns
 all_data.drop(columns, axis = columns).to_gbq(#parameters, table = `public_dataset.orders`)
 all_data[columns].to_gbq(#parameters, table = `private_dataset.users_db`)

Since files are continually coming, I have to reassign user_IDs again and again. (Because searching a corresponding email for each row in customers db is way more expensive than simply reassigning IDs. All analyses are anonymous and based on total numbers so it wont hurt as well. Also mantaining a local DB is also not preferrable as data is incoming in an excel format) I can have a script to push data once a file is come for only that file but cannot because of

Not being able to emulate pd.factorize()
And dupicated() on BigQuery
Not knowing in advance what rows are already there since files may have duplicated records as well,

I have to do this again and again. So how can I transition this whole pipeline to BQ effectively and cost-efficiently? Any help regarding bettering the situation is highly appreciated

Possible Solution:
Assign a UUID to each email/user instead of an auto-increment ID (first making sure email does not already exist else use the existing UUID).
UUIDs can be created anywhere - not necessarily in DB - and are almost guaranteed to be unique accross time and space.

For example, the number of random version-4 UUIDs which need to be
generated in order to have a 50% probability of at least one collision
is 2.71 quintillion, computed as follows:
$${\displaystyle n\approx {\frac {1}{2}}+{\sqrt {{\frac{1}{4}}+2\times \ln(2)\times 2^{122}}}\approx 2.71\times 10^{18}.}$$
This number is equivalent to generating 1 billion UUIDs per second for
about 85 years. A file containing this many UUIDs, at 16 bytes per
UUID, would be about 45 exabytes.

Note: User Email can also be used as unique ID and makes everything easier.


How to store and analyze classification results with Python? I'm applying ML for classification task in Python using sklearn/pandas. I'm going to try various things to get the best results, and I wonder how do I effectively store and analyze all the parameters and results of the classification? Parameters include:

Number of training examples (which can be extended as I get more labeled data).
Set of features.
Classification algorithm.
Algorithm hyperparameters.

Results include:

Precision/recall for each of the classes.
Overall precision/recall.
Support for each class, etc.

Of course, I can manually copy the parameters and results to an Excel spreadsheet every time, but it's not an optimal solution. Are there any Python libraries (or modules of sklearn/pandas) which allow to easily store and display the parameters and results for later analysis? How do you solve this task?

Sacred is a python library developed by the IDSIA lab that "facilitates automated and reproducible experimental research". It is available through pip as sacred.

For a related discussion see reddit.


How do we define the noisy-MAX canonical aggregator for Bayesian networks? In the book Probabilistic graphical models - principles and techniques, Daphne Koller and Nir Friedman introduce the noisy-OR canonical model for CPDs (in the independence of causal inference family of aggregators) and go on to say this can be extended to noisy-MAX. However the details on how to do this are omitted.

The OpenMarkov project (http://www.openmarkov.org/) notes talk about a noisy-MAX model but again the details on how this is implemented are absent.

The noisy-OR model takes in a number of binary variables $X_i$, for $i=1,2,...n$. It then transforms them to an intermediate set of binary random variables $Z_i$ such that $Z_i=1$ with probability $\lambda_i$ when $X_i=1$ and 0 otherwise. Adds in the binary variable $Z_0$, called the leak, which is 1 with probability $\lambda_0$ (and 0 otherwise). The resulting (binary) output is then the OR over all the $Z_j$, $j=0,1,2,...,n$.

It is unclear how one should extend this to MAX, since directly replacing OR with MAX will yield exactly the same as with OR. (A series of 1s and 0s OR'd is the same result as MAX'ing it.)

What is the methodology/algorithm/formula used for calculating the noisy-MAX?

Answers should also whether the inputs and output can be generalised to non-binary (or even numerical/continuous) types.

As cited in the OpenMarkov tutorial that you linked, the noisyMAX is to extend noisyOR to non-binary variables. You can read more in section 4.1.5 here.


Parameters used in GridSearchCV Slow Down in skLearn Tutorial I'm not sure why grid.fit(X,y) is correct, rather than grid.fit(X_2d, y_2d)

In this tutorial on RBF SVM Parameters, we are using GridSearchCV to find the optimum hyperparameters for an SVM.

They have the following code:

# Dataset for decision function visualization: we only keep the first
two
# features in X and sub-sample the dataset to keep only 2 classes and
# make it a binary classification problem.

X_2d = X[:, :2]
X_2d = X_2d[y > 0]
y_2d = y[y > 0]
y_2d -= 1

...

param_grid = dict(gamma=gamma_range, C=C_range)

# GridSearchCV will search the parameter space for the best parameters to use, minimizing the score function
cv = StratifiedShuffleSplit(n_splits=5, test_size=0.2, random_state=42)

grid = GridSearchCV(SVC(), param_grid=param_grid, cv=cv)

# ==================== CODE I'M INTERESTED IN  ==================>
# ===== SWITCH `grid.fit(X,y)` with grid.fit(X_2d, y_2d) ========>
grid.fit(X, y)
# ==================== ^^^^^^^^^^^^^^^^^^^  =============>

print("The best parameters are %s with a score of %0.2f"
      % (grid.best_params_, grid.best_score_))

where X_2d and y_2d are a subset of X and y.

Just some information to show what X and y are:

print(X.shape)    #(150,4)
print(y.shape)    #(150,)
print(X_2d.shape) #(100,2)
print(y_2d.shape) #(100,)
print(type(X))    #<class 'numpy.ndarray'>
print(type(y))    #<class 'numpy.ndarray'>
print(type(X_2d)) #<class 'numpy.ndarray'>
print(type(y_2d)) #<class 'numpy.ndarray'>

Why does changing the code above to grid.fit(X_2d, y_2d) not work? I'm not sure if it's taking a very long time, OR that it's incorrect. My Jupyter Notebook is just sitting, whereas grid.fit(X, y) only takes seconds.

My original thought was that we would want to fit to the actual data set we're running on which are X_2d and y_2d, rather than X and y

If you aren't sure whether the code works or not, you could turn on gridsearch output:

grid = GridSearchCV(SVC(), param_grid=param_grid, cv=cv, verbose=2)

I just run the script in your link - it worked fine for me. Are you sure you didn't change something while working on the code?


How to do boosting in model-ensembling? Boosting is a sequential technique in which, the first algorithm is trained on the entire dataset and the subsequent algorithms are built by fitting the residuals of the first algorithm, thus giving higher weight to those observations that were poorly predicted by the previous model. Examples are adaboost and GBM, etc. My question is that how to perform boosting when ensembling the base learner? Especially how to get the residual if it is a classification problem?

I know how to bagging the base learners and stacking the base learners. I just have no idea how to boosting the base learners.

Thanks!

You can use any base learner for boosting (Adaboost requires sample weighting though). Keep in mind however that the original idea is to use weak learners with strong bias and reducing that bias through boosting.

If it is a classification problem, usually logarithmic loss is used to calculating the residual/gradient for boosting.

For Python, there is a nice AdaBoost wrapper in scikit-learn (AdaBoostClassifier) which can take for example a Random Forest as base learner.


Zero-inflated independent feature in tree-based models What is the best approach to include a zero-inflated continuous independent feature (e.g., 90% of the values are Zero, 10% are >0) in a Tree-based models (DT, random forest, gradient boosting. etc). I am considering the following three options:

Option 1: Keep the zero-inflated continues feature as-is.

Option 2: Replace the continuous feature with a binary interaction feature (i.e., 0 for X=0; 1 for X>0)

Option 3: Include both the continuous and categorical features.

The main justification I have for Option 1 is that the continuous feature can be used in more than one split. I am also aware that option 3 indicates including two highly correlated independent features. Will I be losing information if I use option 2?
Update:
I found the following answer; however, I am not sure if it can be generalized for tree-based models

Basically, for categorical random variable, the decision tree will try all potential splits and select the one that minimizes the mse. For continuous random variable, the decision tree tries to find the split which minimizes the mse. You may see that categorizing the data makes fewer potential choices for the split, and, therefore, is not optimal in accuracy. On the other hand, fewer splits to consider during the training decrease running time.
My answer is as follows.
If the train time permits, I would do option 1. This way, you don't lose any data. Maybe exact value of the feature is a key. It may the point especially if your response is unbalanced. Say, this feature may be a good predictor of your response if positive. For example, radiation level is zero for most of people, small positive values are OK, but large positive values are the key. However, we don't know how large values we should look for.
Option 2 speeds up training time which may be of benefit.
Option 3 creates unnecessary redundancy.


How do you identify whether your RMSE score is good or not? Im building a XGBoost regression model to predict the values in the range of -3 to 3. Im using Root Mean Squared Error to evaluate the model. With hyper-parameter tuning and everything the best scores I could get was this:

training - 0.118
validation - 0.3207 (5 folds)
testing - 0.3018

I know for the regression task, the RMSE values should be as less as possible. But however, I couldn't reach the range of 0.2 atleast for the test score.
Im wondering does my validation and test score range of 0.3 has anything to do with my -3 to 3 continuous target range?
Any quick help here is much appreciated. Thank you

Note that RMSE is an easy to understand metric. Its the Root of the Mean Squared Error. So this is just how is the typical error.
If your target is something like how big is a building, and the mean of the target its 100m, then having an error of 0.3m its nothing. On the other hand if you predict the size of insect, and your target mean is around 0.1m then an error of 0.3m is huge.
For your case it seems like a good result, an RMSE in test of 0.3, in a range of [-3,3], but then this depends on your problem.


Multivariate Time-Series with Differing Selling Windows I have a data-set in the format below. I am attempting to forecast the target_value by Product_Id for an upcoming season, which is always 26 weeks in length. The main challenge is that in an upcoming season, a product may sell in a different group of stores than last season, and may be sold in a different set of weeks than a previous season (some products sell all 26 weeks, others sell in windows i.e. wk 6-13). I wanted to rely on product attributes to help determine how a product would sell in a different set of weeks/stores based on how similar products had sold in those weeks/stores.

The data is time-series and has many different products. It consists of categorical variables, numerical variables, as well as a store_id which indicates what store the sales are coming from.

Summary for data:

has 3 years of history, target variable is sales for a 6 month forecast horizon
multi-variate time series
consists of categorical and numerical variables
has products that are sold in differing time-frames (those time-frames may change in the future for the same product. i.e. last year it sold in weeks 3-10, but in the forecast horizon next year it's expected to sell in weeks 6-14)
has products that are sold in different stores in different time-frames (can also sell in different stores in the future forecast horizon than it has sold in in the past)
is sparse for subsets of products

My questions are:

What is the appropriate algorithm that could handle the sparse history as well as multiple variables
What is the proper way to structure the data as input into the algorithm?
--For this I had planned on converting categorical variables to dummy variables, but was unsure on whether to keep time as is or convert the week to a dummy variable also since I could be forecasting different sets of weeks for different products. I had the same question for store_id.

I'm new to structuring these problems, so please forgive any errors or missing information, and let me know what additional data is needed. Any and all feedback is appreciated!

Welcome to the site! This is a fairly broad question and impossible to answer in a single post. So, I would have you start by researching recommender systems and algorithms. On top of that, if you fully want to take the stores into account, I would start collecting a dataset that contains attributes of each store - where are they? How many hours are they open? Do they have a lot of traffic? etc.

If you proceed by using the products, then you would research a "content-based recommender system". If you want to proceed by looking at the stores, then you would research a "community based recommender system". From there you should be able to pick a model and/or choose a hybrid of the two approaches and develop a proper solution.

Good luck!


How to train a machine learning algorithm with multiple labels I have the following challenge and I very much hope that there is a solution to it. I also suspect that there is a simple approach to it. I just don't see it at the moment. Any help or advice is highly appreciated.

So, I have the following situation:

I asked persons to label about 1000 data points (each twice) on a 5-point scale, whose scores are not equi-distant. Texts were assessed with regard to several qualitative characteristics (such as comprehensibility). As was to be expected, the labelers did not always agree on the assessment. By analysing the inter-rater reliability, however, a "substantial" reliability (according to Landis and Koch) could be determined.

Now I want to use the labelled data as input for a machine learning algorithm (e.g. SVM and Random Forest). The challenge now is how to optimize the data in advance. Currently it is the case that for the same sample there are also different labels available.

The average value between different labels does not seem reasonable to me. So are there standard procedures how I can adjust the data set in advance?

You would help me a lot!

Thanks a lot in advance.

If you intend to use a summary statistics you would engineer it so it is well suited for your task, meaning captures most of the relevant information. For these things there is usually no best universal solution but it is problem specific. You did not specify what your problem is about so I can't help you there much, maybe use the median value.


Bias in Naive Bayes classifier I am building a document classifier using Naive Bayes. There are 10 classes. My question is that :

1 Should each class contain the same number of documents for training?
  What if the number of training example in each class is different?

2 Does the number of classes and classification algorithm have any relation? say is there any thump rule like if there are 100 classes shall algorithm 'X' has better performance than 'Y'

Unbalanced class distributions

First, unbalanced datasets will cause your model to have a bias towards the over-represented classes. If the distribution of the classes is not very drastic then this should not cause a significant problem with any algorithm you will employ. However, as the difference between the class distribution becomes more severe you should expect to get higher false negatives for that class. Consider this, you are trying to have the model adequately identify what it means for a specific example to belong to a class. If you do not provide sufficient examples, then the model will not be able to understand the extent of the variation which exists among the examples.

If the class distribution is very different, then I would suggest anomaly detection techniques. These techniques allow you to learn the distribution of a single classification and then identify when novel examples fall within this distribution or not.

Choosing an algorithm

More classes will result in a higher dimensional output, thus contributing to the complexity of your model. For example, if you have a model which discriminates between 2 classes with a set dataset size. Then further discrimination (increasing the number of output classes) will cause the model to have higher bias. You should thus expect to see greater test error if you do not increase the size of your dataset.

If you have a set dataset X. Then you need to find the correct balance between bias and model complexity to get the optimal results. For example, a neural network based technique (highly complex) is not a good algorithm to use for a limited dataset with many output classes. However, Naive Baye's or Random Forest would be.


Is ROC and AUC the only criteria for choosing a model? If no, what are the other criteria. Please elaborate.
What should be minimum value of AUC to select a model.

No, area under receiver operating characteristic (AUROC) is just one metric amongst very many possibilities, even assuming you just want to pick a standard approach. There are too many to list in a simple Stack Exchange answer. You can take a look at this list extracted from scikit learn documentantion on metrics for example, which is not by any means exhaustive:

Accuracy classification score.
Area Under the Curve (AUC) using the trapezoidal rule
Average precision (AP) from prediction scores
The Brier score.
The F1 score, also known as balanced F-score or F-measure
The F-beta score
The average Hamming loss.
Average hinge loss (non-regularized)
Jaccard similarity coefficient score
Log loss, aka logistic loss or cross-entropy loss.
The Matthews correlation coefficient (MCC) for binary classes
Area Under the Curve (AUC) from prediction scores
Zero-one classification loss.

What you should choose depends entirely on your model class and goals for the learning work. AUROC is a good choice for a binary classifier when you have different business costs associated with false positives and false negatives. That is because it gives you a sense of how well the classifier can be tuned to be more or less sensitive, and get the best outcomes by changing the class threshold.

There is no minimum AUC or other metric required to select a model in practice. It depends on performance of existing solutions (including non-ML ones) and what the costs vs benefits would be of using the model. Clearly a poorly-performing model (e.g. with AUROC 0.5, no better than random guessing on a balanced data set) is unlikely to gain much from being implemented in a production environment, and you would want to take a new look at the original problem and see if was reasonable to expect anything from ML at all.

To decide which metric to use, you need to define some goal for the eventual model. An ideal metric is one that can be simply converted to terms that the end users will care about. Failing that, one that can be compared to known results from other ML work in the subject area (domain) of the problem.


My LSTM doesn't pass gradient check i'm trying to implement my own LSTM network. I implemented back propagation algorithm, but it doesn't pass gradient check. Can't realize where is the mistake. Please help

Here is the code problem piece of code:

def backward_propagation(self, x, y, cache):
    # T - the length of the sequence
    T = len(y)
    # perform forward propagation
    cache = self.forward_propagation(x)

    #...

    # delta for output layer
    dy = cache['y'].copy()
    dy[np.arange(len(y)), y] -= 1. # softmax loss gradient

    for t in reversed(range(T)):
        dV += np.outer(dy[t], h[t].T)
        dh[t] += self.V.T.dot(dy[t])
        dhtmp = self.V.T.dot(dy[t])
        dctmp = dct[0] * (1.0 - ct[0]**2)

        for bptt_step in np.arange(t)[::-1]:
            # add to gradients at each previous step
            do[bptt_step] = dhtmp * ct[bptt_step]
            dct[bptt_step] = dhtmp * o[bptt_step]

            dctmp += dct[bptt_step] * (1.0 - ct[bptt_step]**2)

            di[bptt_step] = dctmp * g[bptt_step]
            df[bptt_step] = dctmp * c[bptt_step-1]
            dg[bptt_step] = dctmp * i[bptt_step]

            # backprop activation functions
            diga[bptt_step] = di[bptt_step] * i[bptt_step] * (1.0 - i[bptt_step])
            dfga[bptt_step] = df[bptt_step] * f[bptt_step] * (1.0 - f[bptt_step])
            doga[bptt_step] = do[bptt_step] * o[bptt_step] * (1.0 - o[bptt_step])
            dgga[bptt_step] = dg[bptt_step] * (1.0 - g[bptt_step] ** 2)

            # backprop matrix multiply
            dWi += np.dot(diga[bptt_step].T, h[bptt_step-1])
            dWf += np.dot(dfga[bptt_step].T, h[bptt_step-1])
            dWo += np.dot(doga[bptt_step].T, h[bptt_step-1])
            dWg += np.dot(dgga[bptt_step].T, h[bptt_step-1])

            dUi[:, x[bptt_step]] += diga[bptt_step]
            dUf[:, x[bptt_step]] += dfga[bptt_step]
            dUo[:, x[bptt_step]] += doga[bptt_step]
            dUg[:, x[bptt_step]] += dgga[bptt_step]

            # update deltas for next step
            dhtmp = np.dot(self.Wi.T, diga[bptt_step])
            dhtmp += np.dot(self.Wf.T, dfga[bptt_step])
            dhtmp += np.dot(self.Wo.T, doga[bptt_step])
            dhtmp += np.dot(self.Wg.T, dgga[bptt_step])
            dctmp += dctmp * f[bptt_step]

    return [dV, dWi, dWf, dWo, dWg, dUi, dUf, dUo, dUg]

Here 'ga' suffix is for 'gate activation' - designation of gate input before non-linear activation function is applied.

Hope somebody will safe lots of hours:

def backward_propagation(self, x, y, cache):
    # T - the length of the sequence
    T = len(y)
    # perform forward propagation
    cache = self.forward_propagation(x)

    #...

    # delta for output layer
    dy = cache['y'].copy()
    dy[np.arange(len(y)), y] -= 1.0 # softmax loss gradient
    # print("dy: ", dy)
    dhtmp = np.zeros((1, self.hidden_dim))
    dh_prev = np.zeros((1, self.hidden_dim))
    dctmp = np.zeros((1, self.hidden_dim))

    for t in np.arange(T)[::-1]:
        dV += np.outer(dy[t], h[t].T)
        dhtmp = self.V.T.dot(dy[t]) + dh_prev

        # add to gradients at each previous step
        do[t] = dhtmp * ct[t]
        dct[t] = dhtmp * o[t]

        dctmp += dct[t] * (1.0 - ct[t]**2)

        di[t] = dctmp * g[t]
        df[t] = dctmp * c[t-1]
        dg[t] = dctmp * i[t]

        # backprop activation functions
        diga[t] = di[t] * i[t] * (1.0 - i[t])
        dfga[t] = df[t] * f[t] * (1.0 - f[t])
        doga[t] = do[t] * o[t] * (1.0 - o[t])
        dgga[t] = dg[t] * (1.0 - g[t] ** 2)

        # backprop matrix multiply
        dWi += np.outer(diga[t], h[t-1])
        dWf += np.outer(dfga[t], h[t-1])
        dWo += np.outer(doga[t], h[t-1])
        dWg += np.outer(dgga[t], h[t-1])

        dUi[:, x[t]] += diga[t]
        dUf[:, x[t]] += dfga[t]
        dUo[:, x[t]] += doga[t]
        dUg[:, x[t]] += dgga[t]

        # update deltas for next step
        # here dh is accumulated as shared variable
        dh_prev = np.dot(self.Wi.T, diga[t])
        dh_prev += np.dot(self.Wf.T, dfga[t])
        dh_prev += np.dot(self.Wo.T, doga[t])
        dh_prev += np.dot(self.Wg.T, dgga[t])
        dctmp = dctmp * f[t]

    return [dV, dWi, dWf, dWo, dWg, dUi, dUf, dUo, dUg]


Correct interpretation of summary_plot shap graph While through the various resources online to understand the shap plots, I ended up slightly confused. Find below my interpretation of the overall plot given in examples -

Shap value 0 for a feature corresponds to the average prediction using all the other possible combinations of features except for the feature of interest. For e.g, shap value 0 for LSTAT corresponds to the average prediction of models having different combinations of features (excluding LSTAT)
Shap value of 4 with respect to 0 means that there is a relative increase in prediction being 1. How do we explain shap values 4, 2, etc. with respect to 0 in layman's terms?
Slight confusion in interpreting the overall graph vs the single point graph ( (

) -
In a single data point example, the base value/expected value is the average prediction in the entire dataset -> but in the overall plot (referenced at the top) we are saying it is the average prediction without that feature. So things don't add up?
Thank you for your help! :)

I think your interpretation is not entirely correct. Loosely rephrasing Lundberg et al. [arXiv:1802.03888], the SHAP value of feature $i$ is
$$ E[f(x) \mid S \cup \{i\}] - E[f(x) \mid S] $$
averaged over all possible subsets of features $S$, $i \notin S$. Here $f(x)$ is the prediction of the model for inputs $x$. Figure 2 from the preprint is a good illustration:

We are interested in the (signed) length of arrow for a particular feature, averaged over all permutations of the features. For a linear model $y(x) = w^Tx$ the SHAP value of feature $i$ would be simply $w_i x_i$.

Now to your items:

SHAP value of 0 for some feature means that in the current example the value of that feature is ignored by the model. In the linear model this feature's weight would be 0.
SHAP value of 4 means that the value of that feature in the current example increases the model's output by 4. Let me use your summary plot as an illustration. It was produced with Boston housing data, and we can check in the dataset that the maximal value of feature ‘RM’ (average number of rooms per house) is around 9. The plot shows that the brightest shade of red for this feature corresponds to SHAP values of around 3, 4, and 8. This means that having 9 rooms in a house tends to increase its price by 3, 4, or 8 thousand USD.
The summary is just a swarm plot of SHAP values for all examples. The example whose power plot you include below corresponds to the points with $\text{SHAP}_\text{LSTAT} = 4.98$, $\text{SHAP}_\text{RM} = 6.575$, and so on in the summary plot.


IPython notebook shortcut to run I am currious if there is a shortcut for runing the  selected cell in IPython notebook such as alt+F5 or ctrl+f5 or something simmiliar  that is present in Visual Studio (I don't recall exactly now ) , cause I found it annoing to go with  the mouse on  the run  button every  time I need to run a cell?

IPython notebook Keyboard shortcuts:

Shift-Enter: run cell
Ctrl-Enter: run cell in-place
Alt-Enter: run cell, insert below
Ctrl-m: This is the prefix for all other shortcuts, which consist of Ctrl-m followed by a single letter or character. For example, if you type Ctrl-m h (that is, the sole letter h after Ctrl-m), IPython will show you all the available keyboard shortcuts.


What to do after training a classifier? In terms of application, what happens after we train a classifier? What can we learn from it?

For example, if I trained a classifier to predict the success of a Kickstarter campaign with 80% accuracy, how can I apply this information to benefit my own Kickstarter campaign? Is there a way to know which attributes influence the outcome 'success' or 'failed' the most?

To see which attributes influence your model will depend on the type of model you are using. For Decision Trees, Random Forests, and Gradient Boosting based on trees you can plot feature importance metrics.

In linear models such as Logistic Regression, theoretically the coefficients will indicate how much the model will be influenced by their features.

Are you using Python? Let me know which models are you using so I can help you more.


Machine Learning to Get X, Y Coordinates From Picture of Plot I am currently on a project where we have people place sticky notes on a X-Y axis to plot their beliefs on certain topics. These sticky notes can be of different colors to reflect different levels of their beliefs. What we are then left with is a picture that we take of the various plots to base our findings on.

I am hoping to avoid plotting each point individually in a data frame based on the picture. I am also relatively new to utilizing machine learning, specifically in relation to image recognition. I have done a couple of random forest projects, but nothing image processing related.

My question is this, using R, is it possible to use a machine learning package to read these pictures and create a data frame of X-Y coordinates of these sticky notes plots?

I am open to all suggestions. Thanks for any and all help and let me know if you need any clarification.

Edit: Here is an example of what one of the images might look like. Unfortunately, I could only quickly find an X-axis only graph, but we do X and Y-axis graphs as well.

Learning means you have examples of a complex behavior and can learn the dependency of the behavior on some parameters implicitly.

In your picture, one can see the dependency rather explicitly and the dependency is rather simple: A big black disk for every sticky-note.

Solution:

Take picture data (red, green, blue and each 0..255) and for each pixel calculate total squared difference to a black disk of exactly the same diameter as the black dots in the picture (sum differences for each color). This means you compute the difference pixelwise and then square it pixelwise and then add up over the area of the disk.
Find local minima of these differences below a threshold.
The positions of these local minima are the desired positions.

Unfortunately I'm not good enough in R but I believe it can be done in a few lines if loops exist in R.


SparseCategoricalCrosstentropy vs sparse_categorical_crossentropy What is the difference between SparseCategoricalCrosstentropy and sparse_categorical_crossentropy ?
SparseCategoricalCrossentropy: Computes the crossentropy loss between the labels and predictions.
sparse_categorical_crossentropy: Computes the sparse categorical crossentropy loss.

But I am still not sure. Any loss will always be calculates between labels and predictions. SO how are these two different ?

SparseCategoricalCrossentropy is a class. So you have to define a object first then you can compute the loss using it.
scce = tf.keras.losses.SparseCategoricalCrossentropy()
scce(y_true, y_pred).numpy()

While sparse_categorical_crossentropy is merely a function which can be directly used to compute cost.
loss = tf.keras.losses.sparse_categorical_crossentropy(y_true, y_pred)

If you are to pass the loss to a Sequential API then you must pass the object ,not the function.
model.compile('sgd', loss=tf.keras.losses.SparseCategoricalCrossentropy())


How to reduce the resolution of a Image(276*276 --> 48*48) without affecting the features in it I have an image with resolution of $276*276$. I've created a Convolution Neural Network which accepts $48*48$ images. So, I want to resize that $276*276$ image to $48*48$ without reducing any features in the data. Can anyone help me solve this problem so that I will send the resized images as the input to my CNN model?

At least as far as I know reducing the size of images is always lossy and you can not retrieve the original image due to the fact that you are throwing away information. Although there are solutions for dimensionality reduction like PCA, they are linear approaches. It depends on your task that what you want to do but there is a solution if you have a classification problem. Try to use a spatial transformer module before the input of the predefined network and send the output of that module to your predefined network. For your grid sampler, you can set the size of the grid to $48 * 48$. For more details take a look at here.

Concretely, the grid generator first creates a normalized meshgrid of the same size as the input image U of shape $(H, W)$, that is, a set of indices $(x^t,y^t)$ that cover the whole input feature map (the subscript $t$ here stands for target coordinates in the output feature map). Then, since we’re applying an affine transformation to this grid and would like to use translations, we proceed by adding a row of ones to our coordinate vector to obtain its homogeneous equivalent. This is the little trick we also talked about last week. Finally, we reshape our 6 parameter $θ$ to a $2x3$ matrix and perform the following multiplication which results in our desired parametrised sampling grid... since bilinear interpolation is differentiable, it is perfectly suitable for the task at hand. Armed with the input feature map and our parametrised sampling grid, we proceed with bilinear sampling and obtain our output feature map $V$ of shape $(H’, W’, C’)$. Note that this implies that we can perform downsampling and upsampling by specifying the shape of our sampling grid. (take that pooling!) We definitely aren’t restricted to bilinear sampling, and there are other sampling kernels we can use, but the important takeaway is that it must be differentiable to allow the loss gradients to flow all the way back to our localisation network.


Can a Box plot be used for finding the useful features from the dataset? I am reading a book by professor Trevor Hastie and professor Robert Tibshirani called "Introduction to Statistical Learning". In the applied section of the chapter 4, there is a question 11(b) that says:

Explore the data graphically in order to investigate the association between "mpg01" and the other features. Which of the other features seem most likely to be useful in predicting "mpg01"? Scatterplots and boxplots may be useful tools to answer this question. Describe your findings.

Here I tried plotting those box plots but how to find the useful features by analyzing them.

I'm certain that from the correlation matrix and the scatter plots I can judge what all features could be useful, but how come a box-plot be used for that matter, could someone please share their opinion.

This is actually very simple: the more separation you see between the boxes, the stronger the 'predictive power' of the covariate.
Boxplots can be actually viewed as histograms looked at from the top. The box itself encompasses 50% of the data (from 25th to 75th percentile), and the line inside the box is the median. Whiskers show you the bounds of the data, up to 1.5 IQR (if I remember correctly). Anything over that range is an outlier.
Consider 2 plots: displacement and acceleration. In the displacement plot the boxes are completely separated, meaning that at least 50% of the data 'mass' is in a completely different place for mpg01='0' than for mpg01='1'. On the other hand, in acceleration the top of one box is around the median in another box - indicating poor separation.


Model selection and assessment using leave-one-out cross validation Assume $D$ is the training data set with both the value of the predictors $\mathbf{X}$ and the value of the response variable $Y$. I have a loss function $L$ and two models $f(\mathbf{X};\beta)$ and $g(\mathbf{X};\lambda)$, where $\beta$ and $\lambda$ are model parameters. Our goal is to estimate

\begin{equation}
e(f)=\mathbb{E}[L(Y,f(\mathbf{X};\beta))|D]~\mbox{and } e(g)=\mathbb{E}[L(Y,g(\mathbf{X};\lambda))|D]
\end{equation}
Note that it is the expectation of the generalization error of the model $f$ (and $g$) that is trained on the specific training data set $D$, where the expectation is taken based on the same distribution that generates $D$.

Now, if we do a leave-one-out procedure, specifically: let $N$ be the total number of observations in $D$, let $D_{-j}$ be the data set that removes the $j^{th}$ observation. Then,$L(Y_j,f(\mathbf{X};\beta))|D_{-j}$ should be an "almost" unbiased estimator of $e(f)$ right? Theoretically, to get $e(f)$, once should generate infinite new $(Y_i,\mathbf{X}^{(i)})$ from the distribution of $D$, then train the model on $D$ and use that model to make prediction on the new infinite data set and take the average. Now we fit the model on $D_{-j}$, which is only slightly different from $D$. So $L(Y_j,f(\mathbf{X};\beta))|D_{-j}$ should be an "almost" unbiased estimator of $e(f)$. Then you go through all $N$ data points in $D$ to obtain the value of $L(Y_j,f(\mathbf{X};\beta))|D_{-j}$, where $j=1,2,...,N$ (assume $N$ is large), then you take the average, then the result should be very close to $e(f)$ right? Then we do the same thing to model $g$. Then in this case, we get very good estimates of $e(f)$ and $e(g)$ so we can do model selection based on $e(f)$ and $e(g)$. Specifically, if $e(f)<e(g)$, then I should expect that on a LARGE new independent data set $T$, $f$ should perform better than $g$ correct? Also, the quantity $e(f)$ and $e(g)$ computed from $T$ should be very close to the quantity computed from $D$, assuming both $D$ and $T$ are large. Is that correct?

If all the above are correct, then it seems that I did model selection and model assessment in one step. But should I partition the data set into 3 pieces, i.e., first train 2 models on one piece, then apply the 2 models on another piece to do model selection, then apply the 2 models on the 3rd piece to do model assessment?

What you just said is dividing the data into three parts: training, validating and testing.
This is a very common practice in machine learning.
We use validation to help in selecting hyper parameters.
Going with this option vs. Leave one out vs. cross validation depends mainly on how many number of samples you have. If you have a lot of samples, then going with the option of splitting the data into three parts can be more efficient.
I am not sure about your statement of unbiased. You need to be careful when you make judgment


1: 10 rule in logistic regression - EPV I have a dataset with 4712 records. Label Yes - 1558 records and Label No - 3554 records.

I read online that 1:10 rule is based on the frequency of lower occurring class.

In my case, frequency of lower occurring class is 1558

According to 1:10 rule, am I right to understand that it is calculated like 1558/10 = 155.8 further equals 150 predictors

So In my logistic regression, I can use 150 variables/input features to the model without the risk of overfitting. Am I right?

By any chance do we also have to look at the frequency of the other (high occuring) class to determine the no of predictors that I can use? If yes, can you share me as to what has to be done to determine the predictor count?

I am aware that we could also use 1:20 or 1:50 rule. But my question is mainly on

1) Whether is there any other consideration for determining the number of predictors in logistic regression model?

2) How do people calculate min sample size required based on this?

Can someone help me with this?

1) Whether is there any other consideration for determining the number of predictors in logistic regression model?

The right number of predictors depends on your data and on your theories about data, and on that only. All these rules of thumb seem completely arbitrary to me, and they lack any scientific ground.

  2) How do people calculate min sample size required based on this?

What do you mean with "min sample size"?

EDIT: Of course, the number of explanatory variables cannot be larger that the number of observations. Based on this answer, a rulo of thumb is to use at least 10-20 observations for each variable. I wish also to stress one point: they must be useful variables, i.e. variables with actual explanatory power. If a variable is a linear combination of others, it won't improve the model by any means, and statistical softwares such as R would delete one automatically.

However, my suggestion is to try to always employ all the data available, don't just stick to a minimum threshold.


How to handle negative integer values in a data set? Suppose we have a data set of fields with negative integer values. So can we consider that fields with negative values for further proceedings or do we need to ignore that fields?
If we can consider those negative values then please tell me how will we process it?

The dataset is bank_data

There are no problem with negative inputs values, as long as it's mean something. As Ankit Saith said in the comment, temperatures can be negative, money too (positive is money I earn, negative the money I lose) and so on. Of course, inputs like distances should not be negative !

Furthermore, generally in deep learning, you normalize your dataset to have inputs with 0 mean and a std of 1. Then you have "small" value around 0 which can be positive or negative !


n-gram Model - Why Smoothing? I am creating an n-gram model that will predict the next word after an n-gram (probably unigram, bigram and trigram) as coursework.

I have seen lots of explanations about HOW to deal with zero probabilities for when an n-gram within the test data was not found in the training data. I understand how 'add-one' smoothing and some other techniques work.

However, I can find nothing about WHY we need to take actions such as these.

For instance, if the test data has "Peace begins with a Smile" and this was not in the training data, so when I supply the model with "Peace begins with a", it will not come up with "Smile" end word. It may provide others or none. If there are none or they have a low probability, then I would supply the shorter n-gram of "begins with a" and see what words and probabilities that provides. If that fails, then "with a" and so on.

I suspect I'm missing something but can't see what.

Please can you help?

The purpose of smoothing is to prevent a language model from assigning zero probability to unseen events.

That is needed because in some cases, words can appear in the same context, but they didn't in your train set. Smoothing is a quite rough trick to make your model more generalizable and realistic. You can also see it as a tool to prevent overfitting.


Can Reinforcement learning be applied in image classification? So my question is can Reinforcement learning be applied in image classification?

Yes, it can- if you have a criterion of optimality for your problem. Basically, if you can asses automatically, during the learning process, is your new result better than the result of the previous iteration.

Some examples:

https://arxiv.org/pdf/1802.02904.pdf

https://arxiv.org/pdf/1712.07465.pdf

http://www.maths.lth.se/sminchisescu/media/papers/mps-rlcvpr16.pdf

https://arxiv.org/pdf/1804.04603.pdf


How to convert categorical data to numerical data in Pyspark I am using Ipython notebook to work with pyspark applications. I have a CSV file with lots of categorical columns to determine whether the income falls under or over the 50k range. I would like to perform a classification algorithm taking all the inputs to determine the income range. I need to build a dictionary of variables to mapped variables and use a map function to map the variables to numbers for processing. Essentially, I would my dataset to be in a numerical format so that I can work on implementing the models.

In the data set, there are categorical columns like education, marital status, working class etc. Can someone tell me how to convert them into numerical columns in pyspark?

workclass = {'?':0,'Federal-gov':1,'Local-gov':2,'Never-  worked':3,'Private':4,'Self-emp-inc':5,'Self-emp-not-inc':6,'State-gov':7,'Without-pay':8}

I created a sample dictionary with key value pairs for work class. But, I don't know how to use this in a map function and replace the categorical data in the CSV file with the corresponding value.

wc = pd.read_csv('PATH', usecols = ['Workclass'])

df = pd.DataFrame(wc)
wcdict = {' ?':0,' Federal-gov':1,' Local-gov':2,' Never-worked':3,' Private':4,' Self-emp-inc':5,' Self-emp-n-inc':6,' State-gov':7,' Without-pay':8}
df_new = df.applymap(lambda s: wcdict.get(s) if s in wcdict else s)
print(df_new)

This is the code I have written in normal python to convert the categorical data into numerical data. It works fine. I want to do the conversion in spark context. And, there are 9 categorical columns in the data source. Is there a way to automate the dictionary update process to have a KV pair for all 9 columns?

This can be done using StringIndexer in PySpark and the reverse using IndexToString for reference please check this:

from pyspark.ml.feature import StringIndexer

df = sqlContext.createDataFrame(
    [(0, "a"), (1, "b"), (2, "c"), (3, "a"), (4, "a"), (5, "c")],
    ["id", "category"])
indexer = StringIndexer(inputCol="category", outputCol="categoryIndex")
indexed = indexer.fit(df).transform(df)
indexed.show()

For more details, please check the spark documentation


Optimization methods used in machine learning I don't have too much knowledge in the field of ML, but from my naive point of view it always seems that some variant of gradient descent is used when training neutral networks. As such, I was wondering why more advanced methods don't seemed to be used, such as SQP algorithms or interior-point methods. Is it because training a neutral net is always a simple unconstrained optimization problem, and the above-mentioned methods would be unnecessary? Any insight would be great, thanks.

In my reply here

Does gradient descent always converge to an optimum?

it is explained that standard gradient descent works well because backtracking gradient descent works well (proven in our recent paper mentioned in the post) and in the long run backtracking gradient descent behaves like the standard gradient descent.

The main issue with other methods, I think, is that they require too strong conditions for convergence or no convergence is proven at all. In both cases, these make them less applicable to realistic applications.


How to find out the percentage of contribution of a variable for another variable/feature? I have an e-commerce dataset for a month consisting of five columns where the fifth column is "Revenue". The other four columns are different factors for the revenue such as "Page-Views","Visitors","Bounces" and "Click Through Rates". Each row is basically a particular day of the month(hence 30 rows in total).

So, if I would like to find out the contribution/importance of each factor for the revenue, will correlation values between the variables be sufficient or are there better metrics ?

To explicitly get the percentage of contribution for one variable to another feature, you can take their (pairwise Pearson) correlation and square it. That is the percentage of variation in one explained by variation in the other. (This is the R-squared of a linear regression, if you just regressed one variable on the other.)

Similarly, if you want to control for other factors (e.g. you want to control for day of the week), you can run a linear regression with just the control variables, then one with those control variables and another one of your predictors. The difference in adjusted R-squared between those models will be how much of the variation in Revenue is explained by the other predictor, beyond what's explained by your controls, and with a slight overfitting penalty for adding another variable.


Interpretable xgboost - Calculate cover feature importance When trying to interpret the results of a gradient boosting (or any decision tree) one can plot the feature importance.

There are same parameters in the xgb api such as: weight, gain, cover, total_gain and total_cover. I am not quite getting cover.

  ”cover” is the average coverage of splits which use the feature where coverage is defined as the number of samples affected by the split

I am looking for a better definition of cover and perhaps some pseudocode to understand it better.

A more detailed explanation of cover can be found in the code

  cover: the sum of second order gradient of training data classified to the leaf, if it is square loss, this simply corresponds to the number of instances in that branch. Deeper in the tree a node is, lower this metric will be

You can find this here: cover definition in the code

This basically mean that for each split the second order gradient of the specified loss is computed (per sample). Then this number is scaled by the number of samples.


Batch Normalization For Federated Learning As you know, in a federated learning setting, clients train local versions of the joint global model with their Non i.i.d data and each submit an update to the global model which would be aggregated into the next joint global model.
The normalization which happens by Batch Normalization layers during the training phase is based on the local batch statistics. My question is, How should one aggregate these local statistics (batch normalization parameters) for the global model so they represent the global statistics of all the data? I am talking about beta, alpha, moving mean and variance for each batch normalization layer. Should we treat them like weight and biases of fully connected (or Conv) layers and simply average them?

One approach would be to simply average everything, as proposed in the FedAvg preprint.
Some very recent preprints suggest only relaying the learned parameters back to the central server and keeping local batch normalization (BN) statistics separate, as proposed by the SiloBN preprint.
The authors of this paper claim that:

Keeping BN statistics local permits the federated training of a model robust to the heterogeneity of the different centers, as the local statistics ensure that the intermediate activations are centered to a similar value across centers.

To paraphrase them; they distinguish BN statistics as encodings for local domain information whereas the learned parameters are to be domain-invariant. I believe their aggregation method for the relayed information is simply averaging.
Here's their first figure for reference:


Training an acoustic model for a speech-to-text engine What are the steps for training an acoustic model? The format of the data (the audio) includes its length and other characteristics. If anyone could provide a simple example of how to train an acoustic model, it would be greatly appreciated.

This paper has details on how to prepare Audio data (and merge it with Language Models) for speech-to-text :

http://slazebni.cs.illinois.edu/spring17/lec26_audio.pdf

Slide 16 has very high level description of feature engineering.

Following papers provide more detailed analysis of audio processing for speech to text :

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.701.6802&rep=rep1&type=pdf
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.689.4627&rep=rep1&type=pdf
https://www.ijariit.com/manuscripts/v4i3/V4I3-1646.pdf


How to prepare Audio-text data for speech recognition I have gathered some raw audio from all the conferences, meetings, lectures & casual conversation that I was part of. The machine transcription did not offer good results (from Azure, AWS etc.) I would transcribe it so to have both data+label (audio+text) for ML training.
My question is whether to have small (3-10 sec.) audio files (split it at silence) and then transcribe each small file? or large file with timestamps with subtitle.srt format?
What if I have a long duration audio file with text? I heard long files have more chances of errors & not accurate training. What if I add timestamp like subtitle file srt?
Do I need small audio files?
I tried Azure custom Speech to train and test but it threw errors, saying it wont process large audio files. (so small chunks is recommended.)
What other ML platforms (AWS, Watson, GCP) have it their data labelling criteria? Sorry I couldn't find other than MS Azure.
Ideally I shall have its own speech recognition system with clean slate (open to hear suggestions on model selection), but need to know what format & style the data should be created.
The way I see it, audio splitting (say, cutting a 30 mins audio into 200 parts) can be automated but then, how to separate transcripts into 200 lines? (need to check manually to linebreak.), So, not a good option to go for large datasets.
Thus it's important to decide the data format before working on it (for assigning the transcribers the proper instructions).
So the question again: with the clean slate (a) to have large audio files with timestamp transcript, or (b) to have small audio files with single line text?
& how?
Please guide. I did bit research but finally dared to post a question.

In general audio training tools requires data segmented on small chunks. It is not a big problem to segment, you can use segmentation scripts, for example
segment_long_utterances.sh as described in Kaldi group discussion. Many other segmentation tools beside that like dsalign or gentle or aeneas.
Overall, training model from scratch is pretty complicated process which requires time and data (>5000 hours of speech properly annotated). It can take months to build a new model.
You'd better investigate first the reasons your error rate is bad, it might not be solvable with training. If information is already lost, you can not really repair it. A better microphone, no compression for data and many other tricks can improve recognition accuracy significantly. You can also just adapt the language model to improve accuracy in case standard language model is not good for your content. It is easier than acoustic model training and you just need texts as described here.