How does layer normalization work exactly? As far as I understand, layer normalization normalizes across all the features for fully connected layers. Does that mean that for each batch dimension we have to learn the normalization parameters? Also, what do we do at test time if our batch dimension is different?

To understand how it works, we can have a look at the paper related to the question: "Layer Normalization", J. Lei, J. Kiros and G. Hinton.

Question: layer normalization normalizes across all the features for fully connected layers

Not exactly. What layer normalization does is to compute the normalization of the term $a_i^l$ of each neuron $i$ of the layer $l$ within the layer (and not across all the features or activations of the fully connected layers).
This term $a_i^l$  is given by the weighted sum of the activations of the previous layers:
$$ a_i^l = (w_i^l)^T\, h^l$$
Where $h^l$ is the vector of activations from the previous layer and $w_i^l$ is the vector of weights that connects the activations of the previous layer with the neuron $i$.
This normalization is computed within the layer because is done by using the mean ($\mu^l$) and standard deviation ($\sigma^l$) of the values of all the neurons in the same layer, $l$ for the current sample:
$$ \mu^l = \frac{1}{H}\sum_{i=1}^Ha_i^l\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\sigma^l=\sqrt{\frac{1}{H}\sum_{i=1}^H(a_i^l-\mu^l)^2}$$
Where $H$ is the number of neurons in the layer $l$. Hence, the normalization of the $a^l_i$ terms is given by:
$$\bar{a}_i^l = \frac{g_i^l}{\sigma^l}(a_i^l -\mu^l)$$
Where $g^i_l$ is defined by the authors as a scale (or gain) value for each neuron $i$ of the layer $l$ which has to be learnt by the NN.
Once we have obtained the normalized term $\bar{a}_i^l$, we compute the activation of each neuron, $h_i^{l+1}$ by:
$$ h_i^{l+1} = f(\bar{a}_i^l + b_i^l)$$
Where $b_i^l$ is the bias term of the neuron, and $f(\cdot)$ is the activation function.

Question: Does that mean that for each batch dimension we have to learn the normalization parameters?

Yes, that's true. For each sample we would have to recompute the terms $\mu^l$ and $\sigma^l$ in order to do the normalization of the $a^l_i$ terms.

Question: What do we do at test time if our batch dimension is different?

At test time we have to compute only the predictions (forward propagation). We aren't learning the weights, biases and the $g_i$'s terms, we are testing them. However, because the activations on each layer have to be normalized we have to compute again the terms $\mu^l$ and $\sigma^l$ for each layer and sample (irrespective of the mini-batch size).


How to compute the prediction interval for unseen data? I want to calculate the prediction interval of individual predictions without knowing what it's target value is gonna be. For example, I am doing a blood glucose prediction of individual patients and I want to predict what his/her blood glucose value is going to be after 1 hour. I want to compute where that value is gonna lie. (Predicted value: 250, lies within : 220 - 270 or something) And I don't know what his/her actual value is going to be in next hour. So is it possible to achieve this?

What you are referring to is a problem of estimation regression model uncertainty.
The uncertainty estimation method depends on the model that you are using.
Take a look at this tutorial, it provides a detailed explanation of what to do when you are using Linear Regression. It also points to the more sophisticated paper, which describes what to do when your model is more complicated.

If you are working with Regression Trees (xgboost) this tutorial provides a code that you can use.


How to Normalize a feature I have a feature that income of individual. It ranges from 10k to 116 Million. I have about 300k+ records.

Clearly, I cannot use this feature as is as it will distort the model output and there are outliers..

I was thinking of normalizing all values from 0 to 1. And also eleminating outliers. Like if most values from from 30k to 200k, then people with income of 200k would get a score around 1.

I have 2 questions?
1) Is this approach reasonable or should I think about it differently.
2) and suggestions on how to normalize and/or scale values to avoid outliers?

Normalization is rather common. I think you might want to consider the distribution of your data and whether you want to do anything with your outliers. Some popular normalization techniques: min-max and z-score. You will see tons of resources by googling them. From the limited amount of time I work with income data, sometimes you might want to test with a log scale as well as the top 1%-ers tend to skew the normalization as well.


How exactly does matrix factorization help with collaborative filtering We start with a matrix of user ratings for different movies with some elements unknow i.e the rating of a yet to be seen movie by an user. We need to fill in this gap. So

How can you decompose or factorize a matrix where all elements are not known in advance.
after you somehow decompose it, do you just multiply the matrices back to recreate the original one, but now with the missing item populated?
How do you know which factorization method like non negative,singular value,eigen to choose,without going into too much math?

Answers to your question

You factorize the matrix in order to approximate original one as closely as possible. This is generally done by starting with randomized values, and updating based on error (between product of factors and original matrix). In other words, for a given matrix A, you are trying to find matrices C & D such that Error(A - (C x D)) is lowest. The algorithm is designed to find an approximation , which might result in original missing entries being replaced by new values (recommendations or ratings).
Do you just multiply? Yes. That it is the essence of calculation. For every user and product, multiplication gives you ratings or score. Sorting by score, and picking the index you get recommendation for each user. It also allows you to now store much smaller matrices than the original one.
Choice of factorization will be also dictated by application. If your application has only positive ratings, then it is better to use non-negative matrix factorization. You may start using matrix factorization methods without knowing the implementation to start with, as long as you are aware of the overall idea (and the pitfalls in using it).

Further Comments

It is a little perplexing that you start with a matrix with many missing entries (unseen items), approximate the matrix via factorization and expect to get non-missing entries (which help in doing predictions). If the task is to approximate original entries, then recommendations won't be good since you can have missing/zero entries in the approximation and still get lowest error on approximation task. The idea: regularization imposed in the algorithm (point 1 above) ensures that noise in the original data is filtered and only patterns are detected. But this idea has its fair share of critics. The introduction section of this paper by Stefen Rendle gives a readable introduction to what is happening in simple matrix factorization methods and what can go wrong. The paper also re-formulates the task with better optimization criteria.
You can also read this post by Simon Funk which explains the mechanics in a readable language (and code), for matrix factorization applied to recommendations.


What are your favorite sources for news about Machine Learning and Data Science? Data Science and Machine Learning include a lot of different topics and it´s hard to stay up-to-date about all the news about papers, researches or new tutorials and tools.
What sources do you use to get all the information?
I use mostly Reddit as my first source and the subreddits Machine Learning and R.
But I would also Datatau and of course the great KDNuggets.

I like following certain tags on Cross Validated, as well as the r tag StackOverflow.
arXiv.org is also pretty cool, it has several subsections devoted to stats and ML.


is it possible (and/or logical) to set feature importance for xgboost? If I understand tree based methods correctly, it would be better for more important features to be toward the top of the tree.

Is there a way I can dictate this in xgboost? Similar to how I can assign weights to each sample/row, can I somehow enforce certain features to be more likely to be put on top compared to other features? (or some other way that enforces certain columns to play a more important role in the model?)

I am aware that you would want the data to guide the model rather than enforcing it but 1 logical case for this is when you have limited data and would like to try helping the process through your subject matter expertise.

You may duplicate some features in your dataframe. I certainly was looking for two things as you mention,one of them is putting some features closer to the root node, and also want some features to appear more on branches. I understand your concern is to make your trees more aware to some features. I may suggest something there. XGBoost samples each feature uniformly, which it would be nicer if we can say that some features are more important and should be used more. Short hack would be duplicating the columns while decreasing the colsample_bytree ratio. XGBoost for now doesn't support weighted features since it draws features uniformly. I've seen it on another place, there's no specific sampling technique for features(columns in XGBoost) in the documentation. Check colsample_bytree : https://xgboost.readthedocs.io/en/latest/parameter.html (Credits: Heard that technique from a webinar of https://www.kaggle.com/aerdem4 )
Generally, newbies like me starts to create a lot features via engineering, however also may discard that important features becomes minority. Although XGBoost parameters can deal with it while hyperparameter search, it seems really not beneficial creating trees with no rain data because of the domination of high number of engineered wind data when problem is predicting drought .
If your concern is specifying domain knowledge, you may also define a domain knowledge via "feature interaction constrainst" in XGBoost, documentation. I hope I'll edit the answer when I become %100 sure what I'm doing with feature interactions: https://xgboost.readthedocs.io/en/latest/tutorials/feature_interaction_constraint.html#:~:text=Feature%20interaction%20constraints%20are%20expressed,but%20with%20no%20other%20variable.
I'm aware these doesn't answer your primary question, it seems not possible for now. Also, I'm not sure that features put on the top are more important, this doesn't seem always true.
Edit: XGBoost has added feature_weights to the DMatrix, in 1.3.0! However, 1.3.0 is not stable yet. I'll edit here when I find/apply feature weights. https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.XGBRegressor.fit


Feature extraction in time series data as input variables for machine learning algorithms I have worked with time series data to predict the defect in production lines. I want to extract the feature from time series data as inputs variables for machine learning algorithms (such as support vector machine). My dataset looks like:

Timestamp  Pressure
t0           x0
t1           x1
t2           x2
.             .
.             .
.             .
tn           xn

There is a threshold b. If xt>b that means the defect happened. My goals are to extract features from time series dataset above and put it into algorithms to predict the value of time stamp tn+1, tn+2 (short-term) and long-term tn+10. However, I have not yet found the way to extract feature and how to bring the threshold into the algorithm. Could anyone suggest me how to deal with it? Thank you.

You don’t need to extract any features. You should feed the pressure data to a LSTM network. LSTM is a type of recurrent ANN that can be used to predict time series.


Multi Class + Negative Class Image Classification Strategies I have seen a recurring theme in real-world problems I've worked with, where the problem looks something like "build an image classifier that recognizes classes A, B, and C but if the input is not those classes, don't recognize it". This features in other questions here from time to time, notably here and here and here; however, I have had difficulty finding a general solution to what seems to be a common problem.

The approaches I've seen so far seem to fall into roughly two categories:

Treat it as an imbalanced class problem. Usual tricks of dataset balancing, under/oversampling, and class weight all apply here. However, the "other" class tends to be broad and diverse, essentially making the CNN try to encode all the information for "other" in the network. While this works "good enough" for some problems, I have seen it perform poorly and generally it seems like a square peg in a round hole.
One class classification/outlier detection based approaches. In particular, the use of siamese and triplet networks are in a similar line of thought by trying to separate image feature vectors more clearly. Most of the research I've seen tends to focus on image similarity search or verification rather than applying to a multi-class + negative class problem directly. However, the notion of a contrastative or triplet loss to separate a more tightly definable class from everything else seems like a much more sound approach. Perhaps something like training the feature vector and then using one class SVM or some such?

Is there a general strategy for this situation like finetuning CNN's for normal classification? It seems as though some general solution that works much like finetuning plus a contrastative loss of some sort at the feature vector layer likely exists but I'm just not aware of it.

There is no general strategy to anything in machine learning. Moreover, research is scattered over multiple domains so it becomes harder to get your head around, "the best strategy for a case". You can solve a single problem in multiple ways. Coming back to the multi class plus a negative class classification issue, this is one of "many approaches" you can follow:

Hierarchical classification: This is still an open problem. As the name suggests you perform classification on multiple levels starting from a root to internal node to leaf nodes (similar to tree or DAG structure). For eg. Given a digits data set ( 0,1,2,3...9) mixed with some letters (a,b,c....z). On a higher level you need to first classify if its a digit or not i.e. meta-class and if its a digit then you perform lower level classification task to identify the right digit.

There are different variants depending on your knowledge of labels/ groups at different level of hierarchy. For eg. if you don't have a pre-defined meta-class then you might have to perform clustering of similar classes at the higher level before final classification.

Also, you may want to tune the weights in your loss function to identify a particular class efficiently at any level of the classification.

P.S. My knowledge is limited in neural nets context. Definitely there would be more strategies specific to them.


Features & Models to compute the probability of certain customer accepting an offer/product from a bank? What are the features & models that can be used to compute the probability of a certain customer accepting an offer/product from a bank?

After some research, I came to know of what is called 'Propensity Scoring' and it's definition is very relevant to what I am trying to find.

But I failed to find any technical papers going into any appreciable detail. I tried Google Scholar but reached a dead end because the term 'Propensity modeling' is actually a marketing term not exactly academic.

I also know from my search that a broader term is 'Campaign Response Models' but this turns out to be too broad and I am looking for something very specific. Any references/links that contain some technical details about the modeling process, the features or the choice of models will be very helpful.

I don't think anyone can help you with features without more information about your particular domain, but you could do worse than logistic regression for modeling:

https://en.wikipedia.org/wiki/Discrete_choice

https://en.wikipedia.org/wiki/Logistic_regression


The best w_j confusion in xgboost from XGBoost tutorial, it described:

In this equation $w_j$ are independent with respect to each other, the form $G_j w_j + \frac{1}{2}(H_j+λ)w_j^2$ is quadratic and the best $w_j$ for a given structure $q(x)$
and the best objective reduction we can get is:
$w^∗_j = \frac{−G_j}{H_j+λ}$
$obj^∗=\frac{−1}{2}\sum_{j=1}^T\frac{G^2_j}{H_j+λ} + γT$

So my confusion is:
if we would like to minimize the $obj$ function, clearly, the best way is to set:
$G_j w_j + \frac{1}{2}(H_j + λ) w_j^2 = 0$
thus we would have the best $w_j^*$ is:
$w_j^* = \frac{-2G_j}{H_j + λ}$
which is different with the official explanation.
anyone please help to point out where am I wrong?

Usually to find an optimum you set the derivative of the function equal to 0. In your case that gives

$$ \frac{dG_j}{dw_J} = G_j + (H_j+\lambda)w_j = 0 $$

leading to

$$ w_j = \frac{-G_j}{H_j+\lambda}$$


How to represent an image as state in a Q-table I'm trying to do Q-learning with the Atari games using the gym python's package.

I want to use the image as the state of my algorithm, but I came up with a doubt: Is the state all the possibilities for all the pixels of the image?

I've seen many Q-learning implementations, and in the Q-table they always store all the possible states as rows, and the actions as columns (for example, https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/)

Without doing maths, all the possible states for, let's say, a 200x200 grayscale image are insane. How can I handle this problem without doing neural networks?

I guess there must be an approach for this scenario, but I haven't found any good example.

Thank you!!

The Atari games are some of the best examples in which you would need a function approximator (e.g. Neural Network) in order to solve them. This is because the state space explodes and thus you would need something that will generalize quite well in unseen states.

I am not quite sure what you mean by

  I want to use the image as the state of my algorithm, but I came up with a doubt: Is the state all the possibilities for all the pixels of the image?

The state at a specific timestep represents the state of the game and will be the input to your algorithm. Why you would need all the possible combinations of the pixels in that image (most of these combinations won't make any sense and the agent will never encounter them. What you need though is your agent to encounter enough game states that will enable it to generalize well in unseen states.

If you were trying to solve Atari games with a look up table then you would have to store every single image, that your agent encounters at every single time step, in a table (more precisely a reference of that image which will correspond to a $Q$ value). In other words you are trying to store all the states that your agent might encounter in the game (impossible in Atari games - unless you do some very sloppy assumptions about the nature of the states).


Performance gain of GPU when learning DNNs Currently, I learn deep neural networks on my CPU (i7-6700K) using TensorFlow without AVX2 enabled. The networks need about 3 weeks to be learned. Therefore, I am searching for a (cheap) way to speed up this process. Is it better to compile TensorFlow enabling AVX2 or to buy a cheap[1] GPU like the GeForce GTX 1650 Super (about 180€ and 1408 CUDA cores)? What is the estimated performance gain of using a cheap[1] GPU?

[1] Cheap compared to current top edge GPUs.

3 years ago the rule of thumb was: About 15 times faster

Your CPU does 113 GFlops on float operations (source) and your GPU does 3 Tflops (source).

My bet: somewhere between 15 and 30 times faster


Why are support vector machines good at classifying images? I'm trying to figure out if I can identify styles of art using support vector machines and I don't quite understand what makes them good. Perhaps I can make the question more accurate. Support vector machines are basically a way of doing classification with a linear model. You find $\vec{\beta}$ and $\alpha$ such that $sgn(\beta \cdot x + \alpha)$ determines what the class is. So I believe that in order to do image classification, you need logic. It does not suffice to find the average color of an image and decide whether it is greater or smaller than a number to decide which number an image represent. You need logic. I believe you need intermediate logic, like comparing one part of the image to another part of an image. Kind of like a multilayered support vector machine, which now looks like a neural net. Is this intuition correct?

Sincerely,
Still figuring it out.

Support vector machines have one built-in "layer" that helps with having an interpretation of the data - the kernel. You could even use output from some other image classifier, including a neural network, as the kernel. E.g. you could measure how far apart two images are in "classifier space" from a trained neural network (maybe one trained against different targets from the ones you want to classify with the SVM)

There is also an implied "layer" in image feature construction.  Common image features are histograms of pixel values, histograms of pixel differences (edge detection), bags of "visual words" and there are many more possible transformations and statistics.

Typically if you are using a classifier other than a deep neural network (which is supposed to discover the features automatically), then you will pre-process the image into a set of features. That pre-processing covers the majority of an approach being "good at images".


How to combine two time dependent datasets? I'm very new to data science so please be gentle.
I have a dataset that contains record of occurrence of fire for the past 35 years (+-700.000 rows). Each date and time can have more than one occurrence as two fires can happen at the same time in different locations. It's features are approximately as follows:
Date and Time | Borough | County | Area Burned | Type | Cause | etc.

I'm planning to combine this with a new dataset that I'm in the process of constructing. This new dataset would have the following features:
Date and Time | Latitude | Longitude | Av. Temperature | Av. Wind Speed | Av, Humidity | etc.

My idea is to try to predict the likelihood of a fire occurring and it's potential severity from a given set of atmospheric parameters. But I'm a bit unsure what would be the best way to combine them.
If I merge on dates, I'd have multiple repeated rows with fires occurring at the same time in different places, would that be the best way? The problem I see is that the locations where there was no fire would not be represented, so I'd have to add a bunch of blank rows for all locations where there was no fire to balance the set.
Any ideas on the best strategy for this?

If I merge on dates, I'd have multiple repeated rows with fires occurring at the same time in different places, would that be the best way?

Probably not, since you don't want to lose the location information. You should probably find a way to map the latitude/longitude to borough/county between the two datasets, so that you obtain a semantically consistent dataset (list of fires by both date/time and location).

The problem I see is that the locations where there was no fire would not be represented, so I'd have to add a bunch of blank rows for all locations where there was no fire to balance the set.

This depends what you plan to do with your data, but given how your first dataset is currently structured (list of fires) it doesn't make a lot of sense to add locations where there was no fire. You might want to create a dataset which lists for every place and every time whether there was a fire or not, for example.


What is the purpose of the discriminator in an adversarial autoencoder? This is specific to the generative adversarial network (GAN) proposed in A. Makhzani et al. "Adversarial Autoencoders". In a traditional GAN, the discriminator is trained to distinguish real samples in $p(x)$ from fake generated samples output by the generator. On the other hand, the variational autoencoder uses discriminators over the latent codes $z$ instead of the original samples.

What I cannot understand is how the discriminator is able to distinguish between the latent code's prior distribution $p(z)$ and the posterior distribution $q(z)$. What is the reasoning behind the discriminator model attempting to discriminate something that is not a sample $x$, as in the traditional GAN? And should this discriminator be trained before the rest of the autoencoder?

The purpose of having a prior distribution $p(z)$ in any generative adversarial network is to be able to smoothly match a latent code $z$ in a known distribution to an input $x$ in the domain and vice versa. The encoder of a simple autoencoder, without any additional measures other than the in typical pipeline
$$x \rightarrow E \rightarrow z \rightarrow D \rightarrow x'$$
would only require $x$ to approach $x' = D(E(x))$, and for that purpose the decoder may simply learn to reconstruct $x$ regardless of the distribution obtained from $E$. This means that $p(z)$ can be very irregular, making generation of new samples less feasible. Even with slight changes to the bottleneck vector, we cannot be sure that the encoder would ever be able to produce that code with any $x$.

In an adversarial autoencoder (AAE) however, the encoder's job is two-fold: it encodes inputs in $p(x)$ to the respective code in $q(z)$ so that:

it minimizes the reconstruction cost $f(x, D(E(x)))$ (where $f$ is a distance metric between samples, such as the mean squared error);
while it learns to adapt $q(z)$ to the prior distribution $p(x)$.

The latter task is effectively enforced because the discriminator receives:

positive feedback from codes in $p(z)$;
and negative feedback from codes in $q(z)$.

Even if the discriminator might not know anything about any of the two distributions at the beginning, it is only a matter of making enough iterations before it does. The ideal encoder will manage to trick the discriminator to the point of having approximately 50% accuracy in the discrimination process.

Also note that $p(x)$ may not be just a Gaussian or uniform distribution (as in, some sort of noise).
Quoting from Goodfellow's Deep Learning book (chapter 20):

  When developing generative models, we often wish to extend neural networks to implement stochastic transformations of $x$. One straightforward way to do this is to augment the neural network with extra inputs $z$ that are sampled from some simple probability distribution, such as a uniform or Gaussian distribution. The neural network can then continue to perform deterministic computation internally, but the function $f(x, z)$ will appear stochastic to an observer who does not have access to $z$.

Although denoising autoencoders rely on this aspect to learn a model that ignores noise from a sample, the same paper on AAEs (section 2.3) shows that combining noise with a one-hot encoded vector of classes can be used to incorporate label information about the sample. This information is only provided to the discriminator, but it still influences how the encoder produces $q(z)$.


Mapping between original feature space and an interpretable feature space I'm reading the following really interesting paper

https://arxiv.org/pdf/1602.04938.pdf on local interpretable model explanations

on page 3 however particularly section 3.3 Sampling for Local Exploration they mention obtaining perturbed samples $z' \in \{0,1\}^{d'}$, it then says

"we recover the sample in the original representation
$z \in \mathbb{R}^{d}$
and obtain $f(z)$ "

with no indication how this is done, surely the map is not injective? If not how would you know you recovered the correct sample? To this end, i wondering how something like this might be done in practice, moving from one feature space $\mathbb{R}^{d}$ to another $\{0,1\}^{d'}$. I'd really appreciate any help.

Welcome to the community @iaaml! I hope I understood the concept right by briefly going through your reference. This is my impression:

in 3.1, they say

  For example, a possible interpretable representation for text
  classiﬁcation is a binary vector indicating the presence or absence of
  a word.

So, I suppose the point is something like Sparse Representation (you may look for sparse representation to see the methods and examples). For instance, an image vector which is predicted as cat can be explained by a 3d explainer, namely, nose, glasses and mouth. Nose and mouth are evidence for cat while glasses is a human face feature (very naive example). So having a vocabulary for all explainers, you can come up with a sparse representation of each decision in which, significant criteria are represented with 0 or 1 to help validating prediction. It happens by examining the features which contribute the most to that class (that's why in Figure.1 they could understand that "No Fatigue" observation is against the prediction).

To obtain such representation, you can build the vocabulary (a set of features which are significant and all together cover whole or most of the space). Then you map your data on these space in which presence of each element of vocabulary is 0 or 1.

Example

I have three samples and their corresponding predictions:

a) Spanish is the main language in Buenos Aires         :Argentina
b) Apple released its new software                      :IT
c) Apple is the man agriculture product in Buenos Aires :Argentina

Using BoW for constructing the feature space, Apple becomes an important feature as it is a famous IT company, but in the last sentence it affects in a wrong way. Alongside, you can also have a map of which feature contributes the most to which class (let's say through Mutual Information or any other feature ranking method, specified on different classes) and construct the matrices below:

   BuenosAires  Apple
a     1           0
b     0           1
c     1           1

when you have what should happen according to the feature ranking for each class:

   BuenosAires  Apple
a     1           0
b     0           1
c     1           0

comparing these two gives you the probably-wrong Apple in last sentence (like "No Fatigue" in Figure.1). The first matrix is the mapping that you do, and the second is the mapping that feature ranking gives you.

Hope I understood it right!


How is the modulo number selected to build the hash table in DHP algorithm? I'm trying to understand the DHP(Direct Hashing and Pruning) algorithm and I got stuck at explaining the selection of modulo number.
The paper shows an example of using the hash function at page 7: h{{x y}) = ((order of x)*10 + (order of y)) mod 7

My questions are:

What are the basis to define the function this way?
How is the modulo number selected(7, in this example)?

One possible reason to define the function this way is that it works for their example. It is an academic paper so it incentivized to find a novel solution that works on a specific dataset. In the paper, it is a synthetic dataset.

When building hash tables it is best practice to pick a prime (7 in this case) to minimize the number of collisions when using the modulo operator. The fact that is small prime is a clue that they building a toy system.


What is the purpose of a confusion matrix in a classification problem? I am studying machine learning. After some research I understood that a typical workflow for a classification problem (after having  prepared the data) is the following:

Split data in test, train and validation sets
Train the model
Generate the confusion matrix
Analyze the metrics: accuracy, precision, recall and f1
Tune hyper-parameters based on the metric I have decided to optimize.

My question is: why do we ever need the confusion matrix? Shouldn't we already know what metric we need to optimize given the type of problem we are trying to solve?

I am asking this because, as far as I understand, if we have enough computational power, we could basically group steps 2 and 5 by applying a grid-search (which consists basically in a cross-validation for each tuning parameter) which takes as an input the metric to be measured. This would imply that you need to know the metric beforehand and furthermore you cannot get the confusion matrix.

Thanks in advance for the replies.

Ok, so let me answer some of these questions for you:

What is the purpose of a confusion matrix?

A confusion matrix is merely a visual help for you to better interpret the performance of your model. It's a way to graphically visualize the true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). Confusion matrices become more and more useful when you deal with a large number of different classes. It can give you some great insights about how your model is doing. Let's say you are training an image detection classifier. It might be good to know that your model is confused between dogs and wolves but isn't confused between cats and snakes.

Another purpose of the confusion matrix is to define a related cost matrix. In my example, being confused between dogs and wolves might be understandable and it shouldn't mean that your model is bad at what it does. However, if it is confused between classes it shouldn't be confused about, this should be properly represented in your performance metric.

Here is a good blog detailing these concepts: https://medium.com/@inivikrant/confusion-cost-matrix-helps-in-calculating-the-accuracy-cost-and-various-other-measurable-a725fb6b54e1

Shouldn't we already know what metric we need to optimize given the type of problem we are trying to solve?

Here you are confusing two things. On the one hand, yes, you should know in advance which metric you want to optimize (i.e. accuracy, precision, recall, etc.) but it doesn't mean that you know the value of that metric in advance. If you dumb down hyperparameter tuning, it's roughly this:

train model $M$ with hyperparameters $H$
evaluate the performance $P$ of model $M$
choose new hyperparameters $H$ and repeat step 1 and 2
pick model $M$ with hyperparameters $H$ such that $P$ is optimized

Computing the accuracy, precision, recall, or F1-score can be done if you know the TP, FP, TN, FP (see this link for more info). So technically, you don't have to create the confusion matrix per se, but you definitely need to compute the TP, FP, TN, FP to evaluate the performance of your model.

if we have enough computational power, we could basically group steps 2 and 5

You can only optimize your hyperparameters if you compute the performance of your model for each set of hyperparameters. You could skip step 3 as it doesn't technically affect your training process. It only helps you understand better what's going on. But you definitely cannot skip step 4.


K-Fold Cross Validation for NNs When using K-Fold CV, is it still useful to have a Train/Validation/Test split?

Or simply just a Train/Test? I.e. split up data into k bins, and leave one out for testing, train on the rest, and take average of the scores.

It depends.  If you're evaluating your model's performance without tuning hyperparameters, then a train/test split is sufficient.

If you're tuning hyperparameters, then you need a validation set.  Within each fold, you'll train on the training set (of course), using the validation set to tune hyperparameters.  Then you'll evaluate performance on the test set.


Any suitable way to describe the distributions of 2 Pandas Dataframes visually/graphically? I am aware of the fact that the Pandas Dataframe's Statistical description can easily be obtained using df.describe(). I am having 2 dataframes of the same dimensions (i.e. 102 columns and 800000 rows for both the dataframes). I would like to depict the fact visually that the 2 dataframes are very similar/have a statistically similar distribution.

I am not sure of how this can be done visually/graphically in Python. The df.describe method provides count, mean, std, min, 25 %, 50 %, 75 % and max values for the dataframes, but it is difficult to infer from in the first instance. I would prefer a visual/graphical method of any type (I am not sure if Boxplot can help?). Any suggested method along with a Minimum Workable Example would be highly appreciated. Cheers!

Presently,

# df1 is dataframe 1
# df2 is dataframe 2
df1.describe() # this gives statistical values of df1 dataframe
df2.describe() # this gives statistical values of df2 dataframe

You can use the Kolmogorov-Smirnov Test. From Wikipedia

  In statistics, the Kolmogorov–Smirnov test (K–S test or KS test) is a
  nonparametric test of the equality of continuous, one-dimensional
  probability distributions that can be used to compare a sample with a
  reference probability distribution (one-sample K–S test), or to
  compare two samples (two-sample K–S test). It is named after Andrey
  Kolmogorov and Nikolai Smirnov.

Good for us, there is already an implementation for this in Scipy

You can use this dummy code to test it

from scipy import stats
p_value = 0.05
rejected = 0
for col in range(103):
    test = stats.ks_2samp(df1.ix[col,], df2.ix[col,])
    if test[1] < p_value:
         rejected += 1
print("We rejected",rejected,"columns in total")

I made the assumption that the columns that you want to compare are on the same index in both dataframes. If this is not true, you need to find another way. Maybe if they have the same name? You can do it like that.

from scipy import stats
p_value = 0.05
rejected = 0
for col in df1:
    test = stats.ks_2samp(df1[col], df2[col])
    if test[1] < p_value:
         rejected += 1
print("We rejected",rejected,"columns in total")

If the K-S statistic is small or the p-value is high, then we cannot reject the hypothesis that the distributions of the two samples are the same.

Edit: I haven't tried it before, but maybe you can do something like that

Boxplot

from scipy import stats
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

#create 2 dataframes with random integers. I don't have data to simulate your case.

df1 = pd.DataFrame(np.random.randint(0,100,size=(10000, 102)), columns=range(1,103))
df2 = pd.DataFrame(np.random.randint(0,100,size=(10000, 102)), columns=range(1,103))

#apply the Kolmogorov-Smirnov Test

p_value = 0.05
p_values = []
for col in range(103):
    test = stats.ks_2samp(df1.iloc[col,], df2.iloc[col,])
    p_values.append(test[1])

#create the box plot

plt.boxplot(p_values)
plt.title('Boxplot of p-values')
plt.ylabel("p_values")
plt.show()

Heatmap

Another way is the heatmap.

import matplotlib.patches as mpatches

plt.rcParams["figure.figsize"] = 5,2
x = range(1,104)
y = np.array(p_values)
fig, (ax,ax2) = plt.subplots(nrows=2, sharex=True)

extent = [x[0]-(x[1]-x[0])/2., x[-1]+(x[1]-x[0])/2.,0,1]
im = ax.imshow(y[np.newaxis,:], cmap="plasma", aspect="auto", extent=extent)
ax.set_yticks([])
ax.set_xlim(extent[0], extent[1])

values = np.unique(np.round(y.ravel(),2))

colors = [ im.cmap(im.norm(value)) for value in values]
# create a patch (proxy artist) for every color
patches = [ mpatches.Patch(color=colors[i], label="Level {l}".format(l=values[i]) ) for i in range(len(values)) ]
# put those patched as legend-handles into the legend
plt.legend(handles=patches, bbox_to_anchor=(1.05, 2.2), loc=2, borderaxespad=0. )

ax2.plot(x,p_values)

plt.tight_layout()
plt.show()

Code for heatmap: stackoverflow


How to use Embedding() with 3D tensor in Keras? I have a list of stock price sequences with 20 timesteps each. That's a 2D array of shape (total_seq, 20). I can reshape it into (total_seq, 20, 1) for concatenation to other features.
I also have news title with 10 words for each timestep. So I have 3D array of shape (total_seq, 20, 10) of the news' tokens from Tokenizer.texts_to_sequences() and sequence.pad_sequences().
I want to concatenate the news embedding to the stock price and make predictions.

My idea is that the news embedding should return tensor of shape
(total_seq, 20, embed_size) so that I can concatenate it with the
stock price of shape (total_seq, 20, 1) then connect it to LSTM layers.

To do that, I should convert news embedding of shape (total_seq, 20, 10) to
(total_seq, 20, 10, embed_size) by using Embedding() function.

But in Keras, the Embedding() function takes a 2D tensor instead of 3D tensor. How do I get around with this problem?
Assume that Embedding() accepts 3D tensor, then after I get 4D tensor as output, I would remove the 3rd dimension by using LSTM to return last word's embedding only, so output of shape (total_seq, 20, 10, embed_size) would be converted to (total_seq, 20, embed_size)

But I would encounter another problem again, LSTM accepts 3D tensor not 4D so

How do I get around with Embedding and LSTM not accepting my inputs?

I'm not entirely sure if this is the cleanest solution but I stitched everything together. Each of the 10 word positions get their own input but that shouldn't be too much of a problem. The idea is to make an Embedding layer and use it multiple times. First we will generate some data:

n_samples = 1000
time_series_length = 50
news_words = 10
news_embedding_dim = 16
word_cardinality = 50

x_time_series = np.random.rand(n_samples, time_series_length, 1)
x_news_words = np.random.choice(np.arange(50), replace=True, size=(n_samples, time_series_length, news_words))
x_news_words = [x_news_words[:, :, i] for i in range(news_words)]
y = np.random.randint(2, size=(n_samples))

Now we will define the layers:

## Input of normal time series
time_series_input = Input(shape=(50, 1, ), name='time_series')

## For every word we have it's own input
news_word_inputs = [Input(shape=(50, ), name='news_word_' + str(i + 1)) for i in range(news_words)]

## Shared embedding layer
news_word_embedding = Embedding(word_cardinality, news_embedding_dim, input_length=time_series_length)

## Repeat this for every word position
news_words_embeddings = [news_word_embedding(inp) for inp in news_word_inputs]

## Concatenate the time series input and the embedding outputs
concatenated_inputs = concatenate([time_series_input] + news_words_embeddings, axis=-1)

## Feed into LSTM
lstm = LSTM(16)(concatenated_inputs)

## Output, in this case single classification
output = Dense(1, activation='sigmoid')(lstm)

After compiling the model we can just fit it like this:

model.fit([x_time_series] + x_news_words, y)

EDIT:

After what you mentioned in the comments, you can add a dense layer that summarizes the news, and adds that to your time series (stock prices):

## Summarize the news:
news_words_concat = concatenate(news_words_embeddings, axis=-1)
news_words_transformation = TimeDistributed(Dense(combined_news_embedding))(news_words_concat)

## New concat
concatenated_inputs = concatenate([time_series_input, news_words_transformation], axis=-1)


Why is the F-measure preferred for classification tasks? Why is the F-measure usually used for (supervised) classification tasks, whereas the G-measure (or Fowlkes–Mallows index) is  generally used for (unsupervised) clustering tasks?

The F-measure is the harmonic mean of the precision and recall.

The G-measure (or Fowlkes–Mallows index) is the geometric mean of the precision and recall.

Below is a plot of the different means.

F1 (harmonic) $= 2\cdot\frac{precision\cdot recall}{precision + recall}$

Geometric $= \sqrt{precision\cdot recall}$

Arithmetic $= \frac{precision + recall}{2}$

The reason I ask is that I need to decide which average to use in a NLG task, where I measured BLEU and ROUGE ( where BLEU is equivalent to precision and ROUGE to recall). How should I calculate the mean of these scores?

The Fı-score is preferred to simple classification accuracy in order to counter the problem of imbalanced datasets; if the thing you are looking for occurs only rarely anyway then a naive classifier can always say no and appear to be working very well! A variant on Fı is Fß, where

Fß = (1+ß²) × [ (P × R) ÷ ( (ß² × P) + R ) ]

Vary ß to balance precision and recall. As to the why F or G, I believe it to be empirical - you don't say if you are classifying or clustering in your own application?


what approach to use for find best customer out of data? I'm working on this project where the objective is to find certain good leads/customers from the existing customer dataset. I tried the RFM method for scoring but there is no data regarding money or any quantity.

dataset has:

customer_id

date (date when meeting is done)
next_action_date(if any next_meeting_scheduled that day)
status( done, missed, cancelled)

So based on only meetings, I need to score a lead or customer. So I started with normal control flow statement like,

if meeting was scheduled and missed- 5 points
if meeting was scheduled and and completed - 10points
if if meeting was scheduled and and cancelled -2points.

basically, lot of these cases are there by which I need to score a customer.
So how should I write a python code for this, I mean what will be the approach for this. I can get the data from an api for a customer. What I thought is to create features like,

days since last visit
no of meeting done succesfully
and more and add all these in other table.
My problem is which kind of code I have to write for this? Please explain in simple words if someone can. I'm new at this.

Your intuitions are good: in general you need your data to contain all the possible indications (features) which might help find the answer.

And your thinking is also correct about creating this "other table": the reason why you need it is because for this problem you need each instance to correspond to a customer. Your original data is not organized by customer, it's organized by meeting. So it makes sense to organize your data with features by customer such as these:

days_since_last_visit   number_meeting_scheduled   number_meetings_attended   ...
15                      4                          2
189                     3                          1
24                      2                          2
...

In general it's not recommended to assign scores yourself, because you probably don't know what is the optimal value. For example is a customer better when they schedule 10 meetings and attend 3 or schedule 4 and attend 2? It's usually better to give all the raw values you have to the ML algorithm and let it calculate the best way to use it.

An important point to define is: what is a good customer exactly? If you think your scoring is really accurate to define a good customer, you can calculate the score and rank the customers, then you're done: this is a heuristic, because you calculate the answer directly based on your knowledge of the problem.

Now assuming you are not so sure and want to use ML:

If you can have a sample with labels which say whether a customer is "good" or not, then you have a supervised classification problem: the goal will be to train a model able to predict the class (category) for any customer based on the features.
If you have a sample with numeric values indicating "how good" a customer is, then you have a supervised regression problem. Again the goal is to train a model which predicts the value for any customer based on the features.

For both cases above I'd suggest you start with simple methods such as decision trees or SVM (both can do classification or regression). The former has the neat advantage that you can manually observe the tree and understand how the classifier works.

If you don't have any labeled data, then your only option is unsupervised learning, which usually means some form of clustering: in this case there's no training stage, the algorithm is provided only with the features and it tries to group together the instances which are close to each other. A standard approach for that would be K-means, for example. Note that you're not sure to obtain the groups that you expect in this case.


SVM behavior when regularization parameter equals 0 I read on this Wikipedia page the following about soft-margin SVM:

  "The parameter $λ$ determines the trade-off between increasing the
  margin size and ensuring that the $x_i$ lie on the correct side of the
  margin. Thus, for sufficiently small values of $λ$, the second term in
  the loss function will become negligible, hence, it will behave
  similar to the hard-margin SVM, if the input data are linearly
  classifiable, but will still learn if a classification rule is viable
  or not."

I can't understand why in the case that λ=0 the algorithm will behave like hard-margin SVM. If λ=0, it seems to me that the algorithm won't have any reason to perform any optimization on the margin. Doesn't it just become a perceptron in that case, since the algorithm only "cares" about classifying all the train data correctly, while not reaching any optimal solution regarding the margin?

I'll appreciate a clarification about the issue, please.

First, the remark from the wikipedia article deals with small (positive) values of $\lambda$, not $\lambda=0$.  Indeed, if $\lambda=0$, then every separating hyperplane achieves the minimum score of 0.

If the data is linearly separable, then taking $\lambda$ small enough that the first term dominates ensures that minimizing the loss will require taking a separating hyperplane to zero out the first term, and subject to that minimizing the second term is equivalent to the original hard SVM.


Where does spacy, nltk, prodigy, sklearn fit in the AI project? Where does tools like spacy, sklearn, prodigy, nltk fit in the below AI project architecture and what are some common alternatives to these:

I am not sure you can neatly fit the above tools and libraries into your schematic. What even is the meaning of "feature" here, is it an output (like "text classification" is an output of the algorithm)? In which case I challenge the usefulness of this schematic.

Nevertheless nltk, sklearn, etc. are libraries that contain multiple and diverse tools to help you do everything along the NLP AI modeling workflow. From feature extraction to modelling and validation.

As an example all three libraries of spacy, sklearn and nltk have the ability to construct models. So they certainly fit into "ML algorithm" but they also help in feature extraction so they fit in that bucket.

Beyond that they offer you general data wrangling capabilities as well as validation measures which do not appear in your schematic.

The main difference

NLTK and spacy are mainly focused on NLP and text-based data whereas sklearn is very much multipurpose.

The Alternatives

The alternatives really depend on your use cases, NLTK and spacy do very similiar stuff so are already alternatives to each other for NLP. Sklearn is almost standard when it comes to ML in python but if you want to build deep-learning you will probably work more with Keras, pytorch, etc.


How to estimate the variance of regressors in scikit-learn? Every classifier in scikit-learn has a method predict_proba(x) that predicts class probabilities for x. How to do the same thing for regressors?

The only regressor for which I know how to estimate the variance of the predictions is Gaussian process regression, for which I can do the following:

y_pred, sigma = gp.predict(x, return_std=True)

In one dimension, I can even plot, how confident the Gaussian process regressor is about its prediction of different data points

How to estimate the variance of predictions for other regressors? For example, for kernel ridge regressor, multi-layer perceptron, ensemble regressors?

I believe it is the probabilistic nature of a model that allows you to get the variance of predictions, or more generally defined as the uncertainty of predictions, like the Gaussian process you mentioned. This is not simply avaialble in standard regressors.
I think you should be looking at Probabilistic regressors like BayesianRidge if you would like to estimate the uncertainty of your model. An implementation  is also avaialble in scikit-learn, also this nice python package based on PyMC3 or directly via PyMC3 itself for instance. In the latter there are examples like for Bayesian regression in Jupyter Notebook with a good explanation.
In principle, Bayesian Models do not return a a single estimate for the model parameters, but a distribution that make it possible to make inferences about new observations as well as to examine our uncertainty in the model. You may find this post useful.
Note: Adding a normal prior on the weights as it is done in Bayesian regression, one turn the Least-Squares problem to regularized L2 regression under the hood as well (see the full math. derivation here).
Updated Answer: I totally forgot the classical yet simple and powerful Bootstrap Sampling method to calculate confidence intervals for machine learning algorithms. A textbook definition says:

Bootstrapping is a nonparametric approach to statistical inference
that substitutes computation for more traditional distributional
assumptions and asymptotic results. A number of advantages:

The bootstrap is quite general, although there are some cases in which it fails.
Because it does not require distributional assumptions (such as normally distributed errors), the bootstrap can provide more accurate
inferences when the data are not well behaved or when the sample size
is small.
It is possible to apply the bootstrap to statistics with sampling distributions that are difficult to derive, even asymptotically.
It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples).

Reference: Fox, John. Applied regression analysis and generalized linear
models. Sage Publications, 2015.

Please note you do not need a model with probabilistic nature. See this post, or this answer or this one.


Multiple labels in supervised learning algorithm I have a corpus of text with a corresponding topics. For example "A rapper Tupac was shot in LA" and it was labelled as ["celebrity", "murder"]. So basically each vector of features can have many labels (not the same amount. The first feature vector can have 3 labels, second 1, third 5).

If I would have just one label corresponded to each text, I would try a Naive Bayes classifier, but I do not really know how should I proceed if I can have many labels.

Is there any way to transform Naive Bayes into multi label classification problem (if there is a better approach - please let me know)?

P.S. few things about the data I have.

approximately 10.000 elements in the dataset
text is approximately 2-3 sentences
maximum 7 labels per text

For starters, Naive Bayes is probably not appropriate here.  It assumes independence among the inputs (hence the "Naive") and words in a sentence are very dependent.

But, assuming you really want to run with NB as an algorithm to start your experimentation, there are two options I'd consider:

Ungraceful: Lots of NB classifiers

This would be an alternative approach.  Make a corupus of all the words observed as your vector of inputs.  Make a corpus off all the tags that are observed as your vector of outputs.  An NB classifier with multiple outputs is the equivalent of having multiple NB classifiers with one output each (so do whichever is easier to implement in whatever software framework you're using).  Treat each element as a training sample where a given input (a word) is a 1 if that word is present and a 0 if that word isn't.  Use the same binary scheme for the output.

This brute forces the application of the NB Classifier to your data, and leaves you to find meaning by still haivng to mine the huge set of classifiers you'll be left with.

More Graceful: Process your data

This is the approach I'd recommend if you want to run with one multiple-class NB Classifier.

Your goal here is to figure out how to map each set of tags to a single class.  I'm sure there is some sort of clustering scheme or network analysis (perhaps ["celebrity"] linked to ["murder"] could become a the segment ["debauchery"]) that will sensibly map your tags to one single cluster.  If you treat tags as nodes and two given tags together as links, then you'll want to look into community detection algorithms (which is where I'd start).  But, if you just want something working, then some sort of hack on the tags that converts a list of tags to only the tag that's most commonly seen in your dataset would be enough.

This method front-loads the work of cleaning your data and would make the NB Classifier's output easier to understand.


Using the cosine activation function seems to perform very badly I have created a neural network to classify the MNIST handwritten numbers dataset. It is using softmax as the activation function for the output layer and various other functions for the hidden layer.

My implementation with the help of this question seems to be passing the gradient checks for all activation functions but when it comes to the actual run with my training data for an exemplary run of 10 iterations I get an accuracy of about 87% if I use sigmoid or tanh as the activation function for the hidden layer, but if I use cosine it returns an accuracy of 9%. Training the network with more iterations (100, 200, 500) does not have any effect either and in fact my minimization function does not manage to move below 2.18xxx for the cost function no matter how many epochs pass.

Is there some pre-processing step that I need to perform before using cosine if not why is it that this activation function works so badly?

Cosine is not a commonly used activation function.

Looking at the Wikipedia page describing common activation functions, it is not listed.

And one of the desirable properties of activation functions described on that page is:

  Approximates identity near the origin: When activation functions have this property, the neural network will learn efficiently when its weights are initialized with small random values. When the activation function does not approximate identity near the origin, special care must be used when initializing the weights.

$cos(0) = 1$, a basic cosine function does not have this property. Combined with its periodic nature, this makes it look like it could be particularly tricky to get correct starting conditions and other hyper-parameters in order to have a network learn whilst using it.

In addition, cosine is not monotonic, which means that error surface is likely to be more complex than for e.g. sigmoid.

I suggest trying with a low learning rate, and initialising all the bias values to $-\frac{\pi}{2}$. Maybe reduce the variance in initial weights a little too, just to start off with things close to zero. Essentially this is starting with $sin()$. Caveat: not that I have tried this myself, just an educated guess, so I would be interested to know if that helps at all with stability.


For multi-class classification in SGDClassifier how do I tell if it is using one-vs-rest or one-vs-one by default? According to the Geron book, for multi-class classification, SGDClassifier in scikit-learn uses one-vs-rest. But how can I tell which one is used as it doesn't appear to give this information in the help file.

It's there in the User guide [Link]

"....SGDClassifier supports multi-class classification by combining multiple binary classifiers in a “one versus all” (OVA) scheme. For each of the  classes, a binary classifier is learned that discriminates between that and all other  classes. At testing time, we compute the confidence score (i.e. the signed distances to the hyperplane) for each classifier and choose the class with the highest confidence...."


Accuracy changes very little when modifying parameters in a Keras model I've seen other similar questions and followed their solutions, to little improvement. I'm making a model to identify the gender of names. As training data I'm using a list of baby names found here: https://www.ssa.gov/oact/babynames/limits.html. I extracted this data to a new data frame, keeping only one instance of those names occurring more than once, and sorted randomly.

Each name string in a column was converted to a numeric array of lenght max_len and normalized by the function:

def text_to_numeric(column, max_len):
    word_characters = []
    for word in column:
        word_characters.append([c for c in word])

    letters_kept = 25
    tokenizer = tf.keras.preprocessing.text.Tokenizer(num_words=letters_kept, oov_token='<UNK>')
    tokenizer.fit_on_texts(word_characters)

    word_sequence = tokenizer.texts_to_sequences(word_characters)
    words_pre = tf.keras.preprocessing.sequence.pad_sequences(word_sequence, maxlen=max_len,padding="pre")
    words_pre = tf.keras.utils.normalize(input_data)

    return list(words_pre)

The expected output is an array of 2 element list where [1,0] means “Male” and [0,1] means “Female”. The model, where data_file contains processed names and labels, looks like this:

input_length, input_data, output_data = data_reader(data_file)
model = tf.keras.Sequential()
model.add(tf.keras.layers.Dense(100, input_dim=input_length, activation='relu'))
model.add(tf.keras.layers.Dense(100, activation='relu'))
model.add(tf.keras.layers.Dense(2, activation='softmax'))

model.compile(loss='binary_crossentropy', optimizer="adam", metrics=['accuracy'])

model.fit(input_data, output_data, epochs=30, verbose=1, validation_split=0.1)

No matter what, I always get an accuracy of around 75%. I don't know how to choose the model parameters, but I’ve tried with many combinations and the accuracy changes little. So far I've tried: normalizing input, balancing the input dataset so there are the same number of men and women, changing the optimizer, defining an optimizer and change the learning rate, changing layer number, nodes per layer and activation function, increasing number of epochs.

All of this with no significant change in the model's accuracy. Am I missing something or doing something completely wrong? Is this accuracy as good as it gets?

When you can't improve the model any more, improve the data.

are there any unisex names in your dataframe? Those will obviously hurt your accuracy. Either toss them out for now or do multi label binary classification.
Visualize your data: are the labels clearly separated? Which are the top losses?
I just saw that the data has the popularity of a name. I think it would make sense to try adding this popularity as a weight of each datapoint when training the model. Intuitively, you want your model to understand what does it mean for a name to be feminine. And "Anna" (score=2604) is much more representative of this meaning than "Jewel" (score=5), don't you think?

PS: I just saw that "Joe" is included in the list of feminine names.


implementing forward and backward of a Linear model I'm implementing the code of this abstraction.
The forward is easy and looks like that:

I don't understand the backward path and how it fit's the abstraction in the first image:

Why is db defined as multiplication of ones of x's shape and dout ?
Why is dw defined as multiplication of ones of x.T and dout ?
Why both of them are accumulated. i.e it is used += and not = ?
Why is dw defined as multiplication of ones of dout and w.T ?

This is because the derivative wrt $b$ is $1$: $\frac{\partial E}{\partial b} = 1$
dout is the derivative of loss function wrt prediction. Using chain rule,
$$
\frac{dE}{dw} = \frac{dE}{dy}\frac{dy}{ds}\frac{ds}{dw}
$$
The last term is the vector of input features $x$. In your case dout is the combination of the first two terms. For example, for MSE loss and sigmoid activation  dout $= (y-L)y(1-y)$
This is often used in optimizers for momentum calculation
For MLPs, you need to compute gradients for coarse layers using gradients of deep layers. For example, for MLP with one hidden layer with features $\mathbf{z}$ (hence 3 in total) vector of gradients wrt weights in the input layer $\mathbf{w}^0$ would be
$$
y= \sigma(\sum_kw^1_k \cdot\sigma(\sum_jw^0_jx_j))\\
\frac{\partial E}{\partial \mathbf{w^0}} = \frac{\partial L}{\partial y} \frac{\partial y}{\partial s} \frac{\partial s}{\partial \mathbf{z}}\frac{\partial \mathbf{z}}{\partial \mathbf{w}^0}  = \frac{\partial E}{\partial \mathbf{z}}\frac{\partial \mathbf{z}}{\partial \mathbf{w}^{0}}\\
\frac{\partial E}{\partial \mathbf{w^0}} = (y-L) y(1-y) \sum_j\frac{\partial s}{\partial z_j}\frac{\partial z_j}{\partial \mathbf{w^0}} = (y-L) y(1-y) \sum_j\frac{\partial s}{\partial z_j}\frac{\partial z_j}{\partial s_j}\sum_i \frac{\partial s_j}{\partial w_{ij}}\\
\frac{\partial E}{\partial \mathbf{z}} = (y-L)y(1-y)\frac{\partial s}{\partial \mathbf{z}} = (y-L)y(1-y)\mathbf{w}^1
$$
So, in other words, in order to compute gradients for weights in the input layer, you need gradients wrt neurons in the hidden layer


Estimating X based on Y values in neural network Let's say I have a trained neural network model, preferably with couple of dense layers. It has learned to estimate Y values based on X values.
Can I somehow use the same trained model and find approximation of X values that result in a specific Y values?
In other words, I have some Y values, I'd like to find out X values which if I feed to the same model, result those Y values.

It is possible given a target value to get a range of values for the features. In classification, this is often called the decision surface or decision boundary.


Using handcrafted features in CNN What is the difference between using CNN with handcrafted features and CNN without handcrafted features?

Thank you

A CNN automatically extracts features, so hand-crafting features has become unnecessary for most applications.  It learns what features to extract via backpropagation.

You can introduce handcrafted features in a few ways to boost the performance of a CNN:

inject hand-crafted features into the fully connected layer
feed the CNN output into another classifier alongside the hand-crafted features

This paper explores such concepts.


Dealing with large data: selecting a sample I'm given a data set to create a model that would predict whether a certain supply chain would be able to deliver the goods without delay or not. I'm doing this in Python.

The data set has 93000 data points has at least 100 features as given. What's worse, most of these features are categorical data and each feature has around 200 unique categories.

I want to develop the best model I can but I find the size of the data set is too large for this.

I tried different methods of feature reduction. But even for feature reduction methods, it takes infeasible amount of time in my laptop. So I thought the best thing I could do is to first select a sub sample of the data points and find the best model and best parameters and then train the whole data set with this model.

Is selecting a sample from the data set the best thing I could do in this situation?

If not, your kind suggestions are welcome.

If it is, what's the best method of selecting these samples?

9300 cases is not a lot, this doesn't even fall within the realm of big data or anything were you would have to apply special techniques.

Also given a more restricted feature set there are almost no computational issues you should encounter unless you work on an actual brick or Atari. As an example benchmark I am able to fit a model for >4M cases with up to 20-30 features (some with ~40 levels) on my 16gb Laptop in <10mins.

So let's focus on the real issue, the complexity of the data i.e. the amount of available features/feature levels and most importantly efficient code.

Assuming you don't have the resources to simply throw some cloud processing power on this problem and call it a day we have to restrict the complexity.

Note that this does not mean "sampling the data" which would indicate using only a subset of cases, as said in the beginning this isn't your problem.

Instead let's workshop ways to cut down computational requirements:

1. Optimize your code

This sounds simple but research and make sure that you are using the best packages for the job, adjusted parameters correctly and cut fat from your code. This is hard work but it is necessary and worth it.

E.g. have you checked that all code is parallelized as much as you can and runs on all CPUs?

2. Optimize your data structure

Factor levels normally aren't a problem computationally speaking but even so >100 levels might be too hard to understand anyway. Try to transform your data by e.g. grouping certain factor levels together. Also try to pivot parts of your data e.g. by wide to long data.

3. Use your domain knowledge

You surely understand your business and this problem. Don't just throw all the data at it! Think for a minute and remove unnecessary or duplicate columns.

4. Work your way upwards

ML modeling has a simply yardstick, if the validated prediction quality is good, your model is good. If you manage to do that with two variables, nobody cares.

So start with a bare-bones data set that includes ALL cases (still keep split-sample validation in mind here) but only the 2-3 simplest, most relevant predictors (based on domain knowledge) as well as the depended variable.

Fit a model and check it's quality, is it good enough? Yeah your done! If not include the next round of variables...

5. Modulize your EDA and have patience

I cannot stress enough that your data set really isn't that big or complex, so optimize your code first. BUT if basic EDA needed to find the best features still takes too long, do the following:

Structure and modulize your code by e.g. doing a PCA for only chunks of the columns (again use all case but only a subset of variables).
Let it run one-by-one and just have patience if the code runs for an hour, sometimes that is how it goes.
Look at the results, compare and decide how to merge results.

6. Talk to your boss

Yes I said I assume you do not have the resources to throw more power on it but if this is an important business problem there should be resources for cloud computation. Fitting a model on AWS is dirt cheap and will only take seconds, so realistically we are talking about <100$ here to train multiple models.

If you don't have a boss to pay for this stuff or he doesn't want to, try free resources. There are a lot of cloud based services that offer enough free GPU or CPU time that you should be able to fit a model.


Augmentation on test dataset and validation dataset I'm training a segmentation model (computer-vision).
Thus, my dataset contains images and masks (binary segmentation of objects).
I'm augmenting the training dataset (applying random crop, rotation or shift etc.) to get a larger dataset. I don't apply augmentation on test and validation dataset.
Should I use augmentation on the validation dataset or the test dataset too ?

Your test and validation dataset should reflect the type of data you would expect when you deploy your model in the actual setting. So usually you do not apply augmentation to the validation and test dataset, since in the real setting you will not receive some strange augmented images.
Another way to think of it is if you apply augmentation to your validation dataset then you will actually measure how your model will perform on augmented data and not on 'real' data. You will use this to inform the best settings for your model so you will end up with a model tuned to perform well on augmented data.


Scientific explanation of Skip Connections I am searching for scientific work on skip-connections.

Everybody is talking about improving gradient flow through the Network, which feels like it makes sense. But I would love to understand and read what that means when it comes down to the math.

Other than the resnet paper I haven't found anything in depth about those magical skip-connections.

I am looking for any hints, papers or equations.

I recently read this paper which proposed a new technique without using skip-connections.It is proposed that the same accuracy can be achieved by increasing the width of the network rather than depth.


Using machine learning specifically for feature analysis, not predictions I'm new to machine learning and have spent the last couple months having a blast using Sci-Kit Learn to try to understand the basics of building feature sets and predictive models.

Now I'm trying to use ML on a data set not to predict future values but to understand the importance and direction (positive or negative) of each feature.

My features (X) are boolean and integer values that describe a product. My target (y) is the sales of the product. I have ~15,000 observations with 16 features a piece.

With my limited ML knowledge to this point, I'm confident that I can predict (with some level of accuracy) a new y based on a new set of features X. However I'm struggling to coherently identify, report on and present the importance and direction of each feature that makes up X.

Thus far, I've taken a two-step approach:

Use a linear regression to observe coefficients
Use a random forest to observe feature importance

The code

First, I try to get the directional impact of each feature:

from sklearn import linear_model
linreg = linear_model.LinearRegression()
linreg.fit(X, y)
coef = linreg.coef_
...

Second, I try to get the importance of each feature:

from sklearn import ensemble
forest = ensemble.RandomForestRegressor()
forest.fit(X, y)
importance = forest.feature_importances_
...

Then I multiply the two derived values together for each feature and end up with some value that maybe perhaps could be the information I'm looking for!

I'd love to know if I'm on the right track with any of this. Is this a common use case for ML? Are there tools, ideas, packages I should focus on to help guide me?

Thank you very much.

You don't need the linear regression to understand the effect of features in your random forest, you're better off looking at the partial dependence plots directly, this what you get when you hold all the variables fixed, and you vary one at a time. You can plot these using sklearn.ensemble.partial_depence.plot_partial_dependence. Take a look at the documentation for an example of how to use it.

Another type of model that can be useful for exploratory data analysis is a DecisionTreeClassifier, you can produce a graphical representation of this using export_graphviz


Still have the same error after reshaping I set my column 1 to 13 as X and column 14 as my Y.
My data set has 14 columns and 401 rows including the parameters
Bp,Sg,Al,Su,Rbc,Bu,Sc,Sod,Pot,Hemo,Wbcc,Rbcc,Htn,Class
80,1.02,1,0,1,36,1.2,137.53,4.63,15.4,7800,5.2,1,1

50,1.02,4,0,1,18,0.8,137.53,4.63,11.3,6000,4.71,0,1

80,1.01,2,3,1,53,1.8,137.53,4.63,9.6,7500,4.71,0,1

70,1.005,4,0,1,56,3.8,111,2.5,11.2,6700,3.9,1,1

80,1.01,2,0,1,26,1.4,137.53,4.63,11.6,7300,4.6,0,1

This is my code
import pandas as pd

from sklearn.metrics import accuracy_score

from sklearn.model_selection import train_test_split

from sklearn.ensemble import RandomForestClassifier

from PIL import Image

import streamlit as st

skipinitialspace = True

data = pd.read_csv("C:/Users/ADMIN/PycharmProjects/pythonProject/venv/KidneyPrediction/Kidney.csv")

st.subheader('Data Information: ')

st.dataframe(data)

st.write(data.describe())

chart = st.bar_chart(data)

X = data.iloc[:,0:13].values

X.reshape(-1, 13)

Y = data.loc[:,'Class'].values

Y.reshape(-1,1)

X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.25, random_state=0)

The error is
ValueError: Expected 2D array, got scalar array instead: array=nan. Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.
Traceback:
File "C:\Users\ADMIN\PycharmProjects\pythonProject\venv\KidneyPrediction\Kidney_Prediction.py", line 72, in <module>
    prediction = RandomForestClassifier.predict(user_input)

Dont reshape X. It was fine in the first place.
Try y = y.reshape(-1,1)
You were missing the assignment back to y, since reshape returns a new object.


Why do popular search engines not follow the usual AND, OR logic for queries? I am teaching myself Information Retrieval from Christopher Manning's book (PDF link: http://nlp.stanford.edu/IR-book/pdf/01bool.pdf). I tried Exercise 1.13:

"Try using the Boolean search features on a couple of major web search engines. For
instance, choose a word, such as burglar, and submit the queries (i) burglar, (ii) burglar
AND burglar, and (iii) burglar OR burglar. Look at the estimated number of results and
top hits. Do they make sense in terms of Boolean logic? Often they haven’t for major
search engines. Can you make sense of what is going on?"

By my knowledge of Boolean logic, the number of results should be like this:

burglar AND burglar <= burglar OR burglar = burglar

But this isn't so. In fact, on Google, it is:

burglar > burglar OR burglar > burglar AND burglar

So, what exactly is happening behind the scenes? Any pointers?

Note: This is NOT a homework problem, even though it is from the exercise of a textbook.

Nice question!

An exact answer should be given by looking in the search engine source code but here is a possible explanation.

I run the queries at Google

burglar 33,800,000
burglar AND burglar  29,200,000
burglar OR burglar 26,500,000

The results indeed do not respect the expected Boolean relation burglar AND burglar <= burglar OR burglar = burglar

However, that is since the search engine doesn't process the "and" and "or" as binary operator but just as search tokens.
Looking for them we get

And 25,270,000,000
Or  16,320,000,000

A term alone appears most times. "and" is more common than "or" so a term with "and" is more common than the term with "or"

Note that

burglar burglar 29,000,000

Apparently looking for documents in which the term appears twice.

By the way, Google's Search operators documentation claim that "OR" should indeed act as a binary operator.
You found a case in which they fail to do so.

Note that this behaviour is very specific to search engine.
In Bing you get the following results:

burglar 4,400,000
burglar AND burglar  1,610,000
burglar OR burglar 1,610,000
And 10,400,000,000
Or  3,750,000,000
burglar burglar 1,610,000

The number of results is similar for  "burglar AND burglar", "burglar OR burglar" and "burglar burglar" though we see that "And" is more popular than "OR". It seems that Bing treatments is the removal of "And" and "OR", possibly as stop words.

Bing documentation suggest the operators "&&" for "and" and "||" for "or".
 - burglar || burglar  4,400,000 = burglar
 - burglar && burglar  1,610,000 = burglar burglar

These results fit the claim that when a term appears twice in the search query it should appear at least twice in the document too.


What is the impact of changing image sources on an image recognition? I have a fairly general question pertaining to an image recognition ML model. I’ve recently developed an image recognition model using a single camera collecting more than 5000 images and then trained/developed the model.
My question is if I change the camera source (also changing resolution) would this impact the accuracy of the model? The scripts we’ve developed would still resize the image to the same size as the training set.
There is some discussion within our team about this and I’m hoping someone with a little more expertise in image recognition can add some insight.
Thanks!

In general the accuracy will change (even dramatically as a google research team found out) if only with slight changes in the input image quality.
References:

Understanding How Image Quality Affects Deep Neural Networks

Image quality is an important practical challenge that is often
overlooked in the design of machine vision systems. Commonly, machine
vision systems are trained and tested on high quality image datasets,
yet in practical applications the input images can not be assumed to
be of high quality. Recently, deep neural networks have obtained
state-of-the-art performance on many machine vision tasks. In this
paper we provide an evaluation of 4 state-of-the-art deep neural
network models for image classification under quality distortions. We
consider five types of quality distortions: blur, noise, contrast,
JPEG, and JPEG2000 compression. We show that the existing networks are
susceptible to these quality distortions, particularly to blur and
noise. These results enable future work in developing deep neural
networks that are more invariant to quality distortions.

Does Deep Learning Have Deep Flaws?

A recent study of neural networks found that for every correctly
classified image, one can generate an "adversarial", visually
indistinguishable image that will be misclassified. This suggests
potential deep flaws in all neural networks, including possibly a
human brain.


Trying to locate a COVID Dataset I have been looking for a dataset that includes:

Age
Hospitalized
ICU
Disposition (Recovered,Death)

Has anyone located one ?
Preferably from a western country as China's data is suspect.
I find it hard to believe CDC is not gathering this data.

You can find some data and maybe combine from multiple sources, starting with:

https://www.kaggle.com/kimjihoo/coronavirusdataset (what I could come up that includes patient ages from the first Google results)

And for the global stats: https://github.com/datasets/covid-19/blob/master/data/time-series-19-covid-combined.csv or


How to run Spark python code in Jupyter Notebook via command prompt I am trying to import a data frame into spark using Python's pyspark module. For this, I used Jupyter Notebook and executed the code shown in the screenshot below

After that I want to run this in CMD so that I can save my python codes in text file and save as test.py (as python file). Then, I run that python file in CMD using python test.py command, below the screen shot:

So my task previously worked, but after 3 or 4 hours later I again tried to execute the same process, with no changes made...Then unfortunately I got the following error message:
Can someone please explain why this has happened?
Many thanks in advance

First time when you executed your code, you might have executed it from the same location where your te.csv file was present.
Are you executing this code in the same location as before? If not, then I suggest you put in and reference the absolute path of te.csv in your code. This should then run without a hitch.


Should I Plot my Learning Curve Depending on Various Parameters to Diagnose Overfitting/Underfitting or is One Parameter Sufficient? I know that learning curves are a good tool to diagnose overfitting or underfitting for a model.

Their working principle is simple: The training/validation loss/accuracy is plotted depending on specific parameters, e.g., the learning rate, the model architecture complexity, the training set size, the number of training epochs, etc.

However, should I always plot many different learning curves, i.e., depending on these different parameters? Or is it sufficient, if I only plot one learning curve depending on just one parameter? Thus, would it be reliable enough, if I would just plot the learning curve based on the keras (tensorflow 2 as the backend) model.fit history, i.e., the training/validation loss depending on the training epochs.

Overfitting is a result of the entire model. It is best to visualize every part of the model to understand how the combination of elements is overfitting.

Ideally, you should visualize all elements on the same interactive figure so how the combination is overfitting. If that is not possible or the interpretation is difficult, then many single element visualizations would be an option. However, that increases the possibility of missing interactions causing overfitting.


macro average and weighted average meaning in classification_report I use the "classification_report" from from sklearn.metrics import classification_report in order to evaluate the imbalanced binary classification

Classification Report :
              precision    recall  f1-score   support

           0       1.00      1.00      1.00     28432
           1       0.02      0.02      0.02        49

    accuracy                           1.00     28481
   macro avg       0.51      0.51      0.51     28481
weighted avg       1.00      1.00      1.00     28481

I do not understand clearly what is the meaning of macro avg and weighted average? and how we can clarify the best solution based on how close their amount to one!

I have read it:
macro average (averaging the unweighted mean per label), weighted average (averaging the support-weighted mean per label)

but I still have a problem in understanding how good is result based on how close these amount to 1? How I can explain it?

Your data set is unbalanced since 28432 out of 28481 examples belong to class 0 (that is 99.8%). Therefore, your predictor almost always predicts any given sample as belonging to class 0 and thereby achieves very high scores like precision and recall for class 0 and very low scores for class 1.

In the case of weighted average the performance metrics are weighted accordingly: $$score_{weighted\text{-}avg} = 0.998 \cdot score_{class\text{ }0} + 0.002 \cdot score_{class\text{ }1}$$ Which turns out to be 1 due the class imbalances.

However, macro avg is not weighted and therefore $$score_{macro\text{-}avg} = 0.5 \cdot score_{class\text{ }0} + 0.5 \cdot score_{class\text{ }1}$$

Since your model just guesses to almost always predict class 0 these scores turn out to be poor.

Going forward I suggest start reading about unbalanced classification problems. There are many approaches how to tackle this. One important question here is whether false predictions for the two classes lead to different cost or not (which, for example, typically is the case in medical applications, spam filters or financial transactions). Because if they do not then a predictor always guessing the majority class could even make sense. But that strongly depends on the area and way of application.


A math question about solving the Lagrangian of Support Vector Machine $$\mathcal{L}(w,b,\xi,\alpha,r) = \frac12w^Tw+C\sum_{i=1}^m \xi_i-\sum_{i=1}^m \alpha_i[y^{(i)}(x^Tw+b)-1+\xi_i]-\sum_{i=1}^mr_i\xi_i$$

Here, the $\alpha_i$'s and $r_i$'s are our Lagrange multipliers (constrained to be $\ge 0$)

To maximize the Lagrangian of soft margin SVM (see the formula above), we set the derivatives with respect to $w$, $\xi$ and $b$ to $0$ respectively.

But what if we set the derivatives w.r.t $r$ to zero first? Wouldn't that result in $\xi$ being all $0$s? Meaning that the optimal solution is reached only when all the relaxing terms $\xi$ are $0$? But that doesn't seem right, does it?

Since you have an inequality constraint you need to meet the necessary Kuhn-Tucker Conditions which for non-negativity constraints are:

$\xi \geq 0$
$-r \cdot \xi = 0$

It is not enough to set the derivative w.r.t. to $r$ to zero as is the case for equality constraints and Lagrange multipliers.

Therefore, there are two cases to distinguish:

Either the condition is tight ($\xi=0$) and you can have $r \neq 0$
Or $\xi > 0$ and $r = 0$.


Tensorflow/Keras, How to convert tf.feature_column into input tensors? I have the following code to average embeddings for list of item-ids.
(Embedding is trained on review_meta_id_input, and used as look up for pirors_input and for getting average embedding)

 review_meta_id_input = tf.keras.layers.Input(shape=(1,), dtype='int32', name='review_meta_id')
 priors_input = tf.keras.layers.Input(shape=(None,), dtype='int32', name='priors') # array of ids
 item_embedding_layer = tf.keras.layers.Embedding(
     input_dim=100,      # max number
     output_dim=self.item_embedding_size,
     name='item')

 review_meta_id_embedding = item_embedding_layer(review_meta_id_input)
 selected = tf.nn.embedding_lookup(review_meta_id_embedding, priors_input)
 non_zero_count =  tf.cast(tf.math.count_nonzero(priors_input, axis=1), tf.float32)
 embedding_sum = tf.reduce_sum(selected, axis=1)
 item_average = tf.math.divide(embedding_sum, non_zero_count)

I also have some feature columns such as..
(I just thought feature_column looked cool, but not many documents to look for..)

  kid_youngest_month = feature_column.numeric_column("kid_youngest_month")
     kid_age_youngest_buckets = feature_column.bucketized_column(kid_youngest_month, boundaries=[12, 24, 36, 72, 96])

I'd like to define [review_meta_id_iput, priors_input, (tensors from feature_columns)] as an input to keras Model.

something like:

 inputs = [review_meta_id_input, priors_input] + feature_layer
 model = tf.keras.models.Model(inputs=inputs, outputs=o)

In order to get tensors from feature columns, the closest lead I have now is

fc_to_tensor = {fc: input_layer(features, [fc]) for fc in feature_columns}

from https://github.com/tensorflow/tensorflow/issues/17170

However I'm not sure what the features are in the code.
There's no clear example on https://www.tensorflow.org/api_docs/python/tf/feature_column/input_layer either.

How should I construct the features variable for fc_to_tensor ?

Or is there a way to use keras.layers.Input and feature_column at the same time?

Or is there an alternative than tf.feature_column to do the bucketing as above? then I'll just drop the feature_column for now;

I think my answer over there https://stackoverflow.com/questions/56693937/tensorflow-how-to-convert-feature-column-to-vector might help you.

Look at

feature_layer = tf.keras.layers.DenseFeatures(feature_columns)


Transforming data points to match other data points I have two sets of data points, set A and set B.
Each point of both sets has two dimensions $(x, y)$.
Consider the set $A$ to be the orange set on the left of the image below and $B$ to be the blue on the right (some points are more transparent, because the points represent a position over time; the darker the point, the more recent is the position it represents).

I can observe that there is a similarity between both sets. If I perform some rotation and scaling to set $B$, I end up with a set $B'$ more similar to set $A$.
My question is, what method can I used to find the transformation that makes set $B$ as similar to $A$ as possible? And how can I measure the similarity?
I do not need a programming solution, just some tips on possibly existing algorithms.
If there is no easy solution, I'd like to know if there is some designation for this type of problem or some keywords that could point me in the right direction.
Thanks!

For this task, you need to decide 3 things:
(1) What are the allowed transformations.
For example do you allow an arbirtrary affine transformation (rotation, scaling, translation). Then you need a matrix $f \in \mathbb{R}^{3 \times 3}$ (using homogeneous coordinates) that maps the points from the left image to the right one. Each entry of the matrix is a variable you want to determine. You could also restrict to specific affine maps (e.g. only rotations) by setting certain entries in $\mathbf{A}$ fixed.
In principle you can use  any  map $f_{w}: \mathbb{R}^{2 \times 2} \rightarrow \mathbb{R}^{2 \times 2}$, that depends on some variables $w$.
(2) You need to think about a measure (or "loss"), which evaluates how similar the two sets are. For arbitrary sets, there is the  Hausdorff distance, which ignores any temporal information. If you consider your data as two curves over time, there is the Frechet distance.
You can have a look here to see that its a difficult problem to  using the Haussdorf distance in general. In fact,  computing only the Hausdorff distance, without any optimization is already  not trivial.
However, for polygonal curves, you can compute Hausdorff distance, Frechet distance, and discrete Frechet distance efficiently.
Finally, in your specific case, you have for each point of $A$ a corresponding point in $B$, you can measure the distance for each correspondence (this gives you simply the mean squared errors in $\mathbb{R}^{2}$).
(3) Depending on the choice of (1) and (2), you need to employ a corresponding optimizer.
If you want to minimize the pairwise distance, but you do not have the correspondences available, there is the Iterative closest point.
If you simple compute the distance between corresponding points, you have a non-linear optimization problem. You can use solvers such as Levenberg–Marquardt.
As @bogovicj points out, the topic is called point-set registration.


Python and GridSearchCV how to eliminate input contains NaN error when using cross validation and decision tree classifier? I am trying to do cross validation on Decision tree classifier for kaggle's titanic dataset. The first step after cleaning data is to split into train and test sets:

from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(train, Y, test_size=0.2, random_state=0)

Then transform numbers into scaled values:

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
x_train = sc.fit_transform(x_train)
x_test = sc.transform(x_test)

In addition:

from sklearn.tree import DecisionTreeClassifier
classifier = DecisionTreeClassifier()

For the grid search, I used GridSearchCV:

#Make a grid search
from sklearn.model_selection import GridSearchCV
tree_param = [{'criterion': ['entropy', 'gini'], 'max_depth': [2,3, 4]}]

And at the end to fit the GridSearchCV classifier into data:

clf = GridSearchCV(classifier, tree_param, cv=4)
clf.fit(X=x_train, y=y_train)

The error I am getting is as follows:

  ValueError: Input contains NaN, infinity or a value too large for
  dtype('float32').

I checked my x_train and y_train sets, and they both contains numeric values within a specific range:

You should impute missing values, try with:

from sklearn.impute import SimpleImputer
imp = SimpleImputer(missing_values=np.nan, strategy='mean')
imp.fit(x_train)
x_train = imp.transform(x_train)
x_test = imp.transform(x_test)

Notice that I am fiting just in the train data, so you are not leaking information to the test.