Adding Group Average Line to Bar Chart I am trying to create a PivotChart in the form of a bar chart and I would like to add a group average line on the chart.

I have this (sample) table in my PowerPivot Data Model:

Group averages are A: 20, B: 50 and C: 80. My bar chart looks like this.

How would I do this? Do I need to add a measure? If so, how? I am not that great with DAX functions yet.

While I am not sure this is what you are looking for, here is what I think you are asking.

First, you are going to need an additional column. Two ways to get er done.

Calculated column

=CALCULATE (
AVERAGE ( Range[Value] ),
FILTER ( ALLSELECTED ( Range ), Range[Group] = EARLIER ( Range[Group] ) )
)
Measure

=CALCULATE (
AVERAGE ( Range[Value] ),
FILTER ( ALLSELECTED ( Range ), Range[Group] = MAX( Range[Group] ) )
)

Now edit your pivot table:

Now select your chart type:

For future reference you should look into Power BI.  It has features which do these sorts of calculations automatically.

If your looking for something different it would help is you provided the expected outcome.


Algorithm Suggestions for a Self Project So, I am doing a small self project on data analytics. I am collecting the android apps data from the play store sites by web scraping.
I am basically trying to collect all possible information related to a particular app, like it's type/genre and sub-genre, ratings, size, number of downloads, if it's paid/unpaid, and other possible information regarding the apps, wherever possible.

I was wondering after I'm done with the data extraction and cleaning work, what data analytics related algorithms I might be able to apply on my play store apps data to analyze it and find patterns in it, predict the links amongst the data and do other predictions and data analysis on it?

A few, I came across were :- TFID, Clustering, Decision trees, Sequence Analysis and maybe Regression.

So, please review the above ones and suggest any new algorithms, which might be useful to me for my project. Some specific algorithms under the above mentioned broad algorithm(if there are, which might be useful for this project ) please suggest them.
Details on them will very much helpful.

Thanks!

I would start with EDA, as usual. Plot stuff. For categorical data, what counts do you have? For quantitative data, perform the usual summaries and correlations. Since you don't have a question in mind already, I would then move on to unsupervised learning algorithms to suggest patterns in the data. Clustering is definitely a good option here (probably preceded by feature scaling). The unsupervised learning algorithms can lead into supervised algorithms, with all the associated statistical tests.

Without knowing your data, you simply cannot choose algorithms.


Data Selection according to Feature Values I am given a dataset consisting of 10 million molecules. Each row contains:

The average value predicted by an ensemble of regression models trained to predict a
certain property about chemicals (value of interest)
The ensemble variance of the predicted value
A heuristic value that characterizes the molecule with respect to some property (value 1)
Another heuristic value that characterizes the molecule with respect to some property
(value 2)

and the requirements:

High values for the value of interest are desired
High values for heuristic value 1 are desired
Low values for heuristic value 2 are desired

Design a protocol that filters and selects 1000 chemicals from the full dataset, and incorporates the mean value of interest, variance of the value of interest, heuristic value 1, and heuristic value 2. Comment on your rationale for the protocol. How would my protocol be adjusted if there was a constraint of diversity in the selected molecules (ie: selecting for chemicals that are very different from one another in the final set of 1000).
EDIT:
I am thinking about PCA if I select only 1 prior component. But is it possible to control that the weights of the features that are desired if higher are positive, while those that are desired if lower negative?

Ignoring for a moment the variance of the first feature, a straightforward approach is to perform a linear combination of the features $x_1$, $x_2$ and $x_3$, with each of the coefficients being a hyper-parameter you set to indicate the relative importance of that feature (perhaps normalizing the features before hand). This will be your molecule's utility measure.
$$u^{(i)} = \beta_1 x_1^{(i)} + \beta_2 x_2^{(i)} + \beta_3 x_3^{(i)} $$
You can then sort according to this utility and select the top 1000. A natural set of coefficients if each feature was of comparable importance would be $\beta_1=1$, $\beta_2=1$ and $\beta_3=-1$ (negative because we want to select for low values of heuristic value 2).
Knowing that we have the variance of the first feature, we can assume that this feature is distributed normally. The utility is now a random variable distributed according to:
$$U^{(i)} \sim \mathcal{N}(\beta_1 \mu_{x_1^{(i)}} + \beta_2 x_2^{(i)} + \beta_3 x_3^{(i)}, \beta_1^2\sigma_{x_1^{(i)}}^{2})$$
From this point, the method for selecting molecules will depend on certain considerations such as your tolerance to variance and if you are more interested in the collective utility of your selection or if each individually selected molecule must meet a certain standard. If you didn't care about variance at all and simply wanted to maximize the collective expected utility you could bypass modelling utility as a random variable all together and stick with the first approach. An alternative if the variance was a factor would be to select the top 1000 molecules according to their mean utility, excluding molecules which didn't have a high probability of having a utility higher than some threshold $P(U > C) > 0.95$. These are just some examples, the exact approach will depend on your criteria which you need to clarify.
As to your question "How would my protocol be adjusted if there was a constraint of diversity in the selected molecules". Diverse in what sense? Have you been provided with any other features to compare the molecules with or do you mean diverse in the three features already given? Again, another simple approach would be to append another feature calculated as the average distance to those molecules already selected.
One of the difficulties with your question is that it is very broad and open ended. There is not a lot of clarity surrounding what constitutes a good selection of molecules from a bad selection. For example, is it better to have a higher heuristic value 1 or a lower heuristic value 2? These sorts of questions need to be handled by a domain expert prior to applying the data science algorithm. If you were given a set A of 3 molecules and a set B of another 3 molecules do you have a clear way of determining which is the better selection? You need to be able to answer this question before you can begin applying any data science or else you are shooting in the dark.


Toolbox for handling NaNs in Python 2.7 Is there a good toolbox for handling and analyzing missing values in Python 2.7?

There is a good toolbox for doing this in Python 3.6 here (missingno): https://github.com/ResidentMario/missingno

I need to work in Python 2.7. so, this is why I ask.

First of all, visualize the missing values :

I am using Python 2.7

import pandas
import numpy
from pandas import DataFrame
import seaborn as sns

df = DataFrame({'A' : [0,1,numpy.nan, 5 ,6],'B':[30,numpy.nan,numpy.nan,8,10]})
df

     A  B
0   0.0 30.0
1   1.0 NaN
2   NaN NaN
3   5.0 8.0
4   6.0 10.0
sns.heatmap(df.isnull(),yticklabels=False,cbar=False,cmap='BuPu')

If u wanna drop rows containing missing values,

df.dropna(axis=0)

        A   B
    0   0.0 30.0
    3   5.0 8.0
    4   6.0 10.0

If u wanna drop columns containing missing values,

df.dropna(axis=1)

If you wanna fill NaN with a value:

df.fillna(0)

  A B
0   0.0 30.0
1   1.0 0.0
2   0.0 0.0
3   5.0 8.0
4   6.0 10.0

You can do a ffill/pad, bfill/backfill. Hit shit tab and expand the documentation if you are using jupyter.

For more info refer : https://pandas.pydata.org/pandas-docs/stable/missing_data.html

I hope this answers your question.


Maybe wrong values for precision and recall I'm trying to do some data mining with RapidMiner studio.
I've applied the K-nearest neighbor algorithm with different values of K.
As I expected, accuracy increase and after K=5, it decrease. But I cannot understand why value of recall for Basic increase (as I expected) but recall for Premium decrease. The same for values of precision.

Below my results:
Basic and Premium are the values of my class label

With K=5

True Basic
Class recall: 91.83%
Class precision: 81.18%

True Premium
Class recall: 32.87%
Class precision: 56.07%

With K=2

True Basic
Class recall: 81.99%
Class precision: 82.94%

True Premium
Class recall: 32.87%
Class precision: 45.20%

If you are calling a higher proportion of cases as Basic with K=5, then this will probably lower your Basic precision, increase your Basic recall, increase your Premium precision, and lower your Premium recall. This is because there is always a tradeoff between precision and recall, and Basic and Premium are opposites in your binary classifier.


Is using LSTM correct for this? I wish to predict whether the difference in Value and Growth returns is positive or negative for the next month.

To do this, I have collected data of a few features(to be specific, Macroeconomic Indicators), and the difference in Value and Growth returns for about 200 previous months. I wish the prediction to depend on the order of the previous months. Decision trees and others don't care about the order so I looked upon the net to find about LSTM. Is it correct to use them here?

About what I have understood, you need to use time series forecasting algorithms. As your data is time series, you can look at these algorithms based on Deep Learning. You can also use classical algorithms for time series forecasting. You can use LSTM also which is good, details are given in link.
Look here


How to reload cell in jupyter notebook? After OSError: [Errno 12] Cannot allocate memory I got an error:
OSError: [Errno 12] Cannot allocate memory
I deleted some files. And I have free memory. I don't want start learning again from first epoch. (I got an error in last epoch).

I use jupyter notebook.
I have 19 .h5 file in my output file.
Can I click something to save my last .h5 file?

Edit:

One option is to add TensorFlow Checkpoint which allows for the saving and reloading of training.


Finding frequencies in a noisy, "uneven" dataset I'm working on a problem where frequency analysis applies (decomposition of a signal into frequencies, that is), but it's noisy and the samples are unevenly spaced.

Specifically: given a list of items purchased at a bar/restaurant, try to estimate the number of guests on the check based on distinct "frequencies" of purchase. The logic is that if there are N guests on a check, then it's reasonable to see N frequencies of drinks being purchased, one person buying every 10 minutes, another every 15, etc. (Plenty of other properties of the check should be included, but here I'm focusing specifically on estimating distinct frequencies).

So more formally: given a noisy, unevenly spaced time series, find the smallest number of frequencies which reproduce the signal while minimizing the error (... for some sensible definition of how to minimize both the error and the number of frequencies simultaneously).

This is more a machine learning problem than signal processing. I realize it's also an open question, but can anyone point me in the right direction? Is there a particular method or algorithm that applies here?

Just to point you in one possible direction: you could treat this problem as one of probabilistic mixture modeling.

Imagine that each person's drink ordering is governed by a probability distribution. That distribution may be characterized by the time since their last drink order.  As time passes, the probability that they will order another drink increases until eventually they order another drink and the time resets.

One possible model for the time between drink orders for a single person is the exponential distribution.  If you consider many people ordering, the resultant times would likely be a mixture of exponentials. The problem then comes down to fitting a mixture of exponentials to the table's drink data.  You would likely have to supply some additional prior data as well to get a meaningful model (otherwise it's difficult to tell the difference between 5 people ordering 2 drinks a piece or 1 person ordering 10 drinks).


Running multiple times of a model is for model randomness or data randomness? When a paper report the average and std of a model on a dataset, it means that they have changed the split of training and test sets and run the model multiple times or they just run the model on constant splitting multiple times to find the randomness of model? Which one is more common?

Usually, we report mean and variance for k-fold crossvalidation and similar techniques. We run the model multiple times in different data, but that can be applied to the second case too as, for example, a Neural Network can be initialized with random weights multiple times for test in the model.

If the paper does not explicitly say that they re-splited the dataset, then is more likely that this variance comes from random initialization. You should read the experimental procedures used and try to determine all sources of variance for papers results.

So it might be either or a combination of both dataset random splits and model random initialization


Gradient Check is failing for RNN I am trying out POS tagging using RNN but not able to figure out what's wrong in my implementation because of which the gradient check is failing. Please help. I am pasting the relevant part below:

# weights
Wxh = randn(inputLayerSize, hiddenLayerSize)*0.01; # input to hidden
Whh = randn(hiddenLayerSize, hiddenLayerSize)*0.01; # hidden to hidden
Bh = zeros(1, hiddenLayerSize); # hidden bias
Why = randn(hiddenLayerSize, outputLayerSize)*0.01; # hidden to output
By = zeros(1, outputLayerSize); # output bias

function forwardRNN(x, y, h, p, hPrev)
  global Wxh, Whh, Why, Bh, By;
  cost = 0;
  for time in 1:length(x)
    if time == 1
      h[time] = tanh(x[time]'*Wxh + hPrev*Whh .+ Bh);
    else
      h[time] = tanh(x[time]'*Wxh + h[time-1]*Whh .+ Bh);
    end
    score = h[time]*Why .+ By;
    p_softmax = exp(score) / sum(exp(score));
    p[time] = vec(p_softmax);
    cost += -sum(log(y[time]'*p[time]));
  end
  return cost;
end

function backwardRNN(x, y, h, p, hPrev)
  global Wxh, Whh, Why, Bh, By;
  dWxh = zeros(size(Wxh));
  dWhh = zeros(size(Whh));
  dBh = zeros(size(Bh));
  dWhy = zeros(size(Why));
  dBy = zeros(size(By));
  dh = zeros(size(Bh));
  dhnext = zeros(size(h[1]));
  for time in length(x):-1:1
    dy = p[time] - y[time];
    dWhy = dWhy + (dy * h[time])';
    dBy = dBy + dy';
    dh = (Why*dy)' + dhnext;
    dhRaw = (1 - (h[time].*h[time])) .* dh;
    dWxh = dWxh + (x[time] * dhRaw);
    dBh = dBh + dhRaw;
    if time == 1
      dWhh = dWhh + (hPrev' * dhRaw);
    else
      dWhh = dWhh + (h[time-1]' * dhRaw);
    end
    dhnext = dhRaw*Whh;
  end
  return dWxh, dWhh, dBh, dWhy, dBy;
end

# gradient checking
function gradCheck(inputs, targets, h, p, hPrev)
  paramNameList = ["Wxh", "Whh", "Bh", "Why", "By"];
  global Wxh, Whh, Why, Bh, By;
  paramList = [x for x=(Wxh, Whh, Bh, Why, By)];
  num_checks = 2;
  delta = 1e-5;
  cost = forwardRNN(inputs, targets, h, p, hPrev);
  dWxh, dWhh, dBh, dWhy, dBy = backwardRNN(inputs, targets, h, p, hPrev);
  dParamList = [x for x=(dWxh, dWhh, dBh, dWhy, dBy)];
  for (param,dparam,name) in zip(paramList, dParamList, paramNameList)
    s0 = size(dparam);
    s1 = size(param);
    if s0 != s1
      println("Error dims dont match: ", s0," and ",s1);
    end
    println(name)
    for i in 1:num_checks
      ri = rand(1:length(param));
      old_val = param[ri];
      param[ri] = old_val + delta;
      cg0 = forwardRNN(inputs, targets, h, p, hPrev);
      param[ri] = old_val - delta;
      cg1 = forwardRNN(inputs, targets, h, p, hPrev);
      param[ri] = old_val
      grad_analytic = dparam[ri];
      grad_numerical = (cg0 - cg1) / ( 2 * delta );
      rel_error = abs(grad_analytic - grad_numerical) / abs(grad_numerical + grad_analytic);
      println(grad_numerical,", ", grad_analytic, " => ",rel_error);
      if rel_error > 1e-5
        error("Gradient check failed.");
      end
      println("Gradient check passed.")
    end
  end
end

The code is in Julia programming language and is inspired from Karpathy's min-char-rnn.py

as i understand, you have the wrong backprop gradient implementation. Here you should take into account, that rnn's hidden state h has its previous state in the equation: h[time-1]. This is also must be extracted via chain rule. For more information suggest to refer this post.
It also contains Python rnn implementation.


Are there any good general techniques for binning/histogramming arbitrary data? Let's say we want to histogram a finite set of measurements of some quantity. It is straight forward to calculate the usual statistical quantities for our sample such as the mean and the variance. Let's assume we can clean up our data by identifying outliers and moving them into underflow and overflow bins and thus, define more or less optimal min and max values for the plotting range. But how would one decide on the number and the size of bins? I would like to know if there are methods to find the optimal binning for the cases with fixed and variable bin sizes.

I don't know if this what you want but here is a way to calculate the number of bins.

Count the number of data points in your dataset.
Take the square root of the number of data points and round up to determine the initial number of bins required: $InitialNumberOfBins = \sqrt{NumberOfDataPoints}$.
Divide the specification tolerance $Max-Min\ value$ by initial number of bins: $FinalNumberOfBins = (Max-Min\ value) / InitialNumberOfBins$.


Resource and useful tips on Transfer Learning in NLP I have a few label data for training and testing a DNN. Main purpose of my work is to train a model which can do a binary classification of text. And for this purpose, I have around 3000 label data and 60000 unlabeled data available to me. My data type is related to instructions (like- open the door[label-1], give me a cup of water[label-1], give me money[label-0] etc.) In this case, I heard that Transferring knowledge from other models will help me a lot. Can anyone give me some useful resource for transfer learning in NLP domain?

I already did a few experiments. I used GLoVE as a pretrained embeddings. Then test it with my label data. But got around 70% accuracy. Also tried with embedding built using my own data (63k) and then train the model. Got 75% accuracy on the test data. My model architecture is given below-

Q1: I have a quick question will it be referred to as Transfer learning if I use GLOVE embeddings in model?

Any kind of help is welcomed. Even someone has other ideas for building a model without using transfer learning is welcomed.

If you use pre-trained models on data that are distinct from the data they were originally trained on, it's transfer learning. Your two-class sentence corpus is distinct from the data that the GloVe embeddings were generated on, so this could be considered a form of transfer learning. This might be a helpful explainer for general ideas around pre-training (and why it's a worthy pursuit).

Recent work in the NLP transfer learning space that I'm aware of is ULMFiT by Howard and Ruder of fast.ai, here's the paper if you prefer that. OpenAI also has recent work extending the Transformer model with a unsupervised pre-training, task specific fine-tuning approach.

As for your task, I think it might be helpful to explore research around sentence classification rather than digging deeply into transfer learning. For your purposes, it seems that embeddings are a means to have a reasonable representation of your data rather than prove that Common Crawl (or some other dataset) extends to your corpus.

Hope that helps, good luck!


Fitting multiple line Short version:
How can I find a function that maps X to Y when data looks like this.
Note:

For a pair of emissivity and distance relation between temperature and
raw_thermal_data is linear.

Long Version:
I am working on a project which uses thermal(IR) camera. Now we extract temperature from sensor reading (raw thermal data )
For some reason I need to find a function that maps temperature data to raw thermal data.
Now,
temperature = F ( raw_thermal_data, emissivity, distance )

I am trying to find,
raw_thermal_data = F1 ( temperature, emissivity, distance )

For a pair of emissivity and distance relation between temperature and raw_thermal_data is linear.
Looks like for every pair of emissivity and distance, intercept of the line is different.
Any thoughts?

Not knowing the data in detail, this look like a linear model with "dummys".
A standard linear model looks like ($\beta_0$ is the intercept, $\beta_1$ is the slope):
$$ y = \beta_0 + \beta_1 x + u.$$
Now, when you have two distinct "groups" for which there is a "flag" in the data, you can assign a indicator variable or "dummy" (say $d$, a vector with =1 "true" or =0 otherwise). You can add this to your linear model:
$$ y = \beta_0 + \beta_1 x + \beta_2 d + u. $$
$d$ introduces a separate intercept for the assigned group ($d=1$).
You can also add "interaction terms" to allow for a separate slope for group $d$ by simply multiplying $x$ and $d$.
$$ y = \beta_0 + \beta_1 x + \beta_2 d + \beta_3 x d + u. $$
Note that since there already is one intercept in the model (the $\beta_0$), you can only add "contrasts" to the intercept. So when you have $i$ groups for which you want to have an individual intercept, you would add $i-1$ indicators/dummys to the model. For the "reference group", the intercept will be $\beta_0$ and for the group which is identified by the "dummy" ($d$ above), the intercept would be $\beta_0 + d$.


What are the default values of nodes and internal layers in Neural Network models? What is the default number of internal layers and internal nodes in training a neural network?

My data has 62 observations with roughly 200 predictors. I have a target variable with two classes and implemented a neural network with one internal layer and one internal node without repeats. Also, I tried with two internal layers, with 5 internal nodes in one, and 2 internal nodes in second layer. I want to find the accuracy, first, on default values and then I will try to optimize the model performance.

What is the criterion to choose the number of layers and internal nodes in a neural network training model? In the case of Random Forest, we can choose to try to be roughly equal to the square root of the number of predictors.

There are websites that explain these pretty well.

Deciding on the number of neurons in the hidden layer(s)

From https://www.r-bloggers.com/selecting-the-number-of-neurons-in-the-hidden-layer-of-a-neural-network/:

  The most common rule of thumb is to choose a number of hidden neurons between 1 and the number of input variables.

Deciding on the number of layers of hidden layers

From https://stats.stackexchange.com/questions/181/how-to-choose-the-number-of-hidden-layers-and-nodes-in-a-feedforward-neural-netw:

  For most problems, one could probably get decent performance (even without a second optimization step) by setting the hidden layer configuration using just two rules: (i) number of hidden layers equals one; and (ii) the number of neurons in that layer is the mean of the neurons in the input and output layers.

Hope that answers your question!


How to export PCA to use in another program I'm trying to write a random forest classifier for a very large dataset, as such as part of the pre-processing i have applied PCA to reduce from 643 features to 5 PC's. Is it possible to export these settings so I can

pca.transform(data)

in another program.

I have been able to do this to the the scaler using pickle but when i run the transfrom i get given the error:
Traceback (most recent call last):

  File "<ipython-input-35-68c9849c2acc>", line 1, in <module>
    pca.transform(data)

TypeError: transform() missing 1 required positional argument: 'X'

Ideally PCA should not be used as a part of pre-processing feature reduction.

Anyhow regarding saving and reusing PCA model, sharing a basic code snippet which is working very fine in my case(as I'm not able to reproduce the error case).

from sklearn.decomposition import PCA
import pickle as pk
pca = PCA(n_components=2)
result = pca.fit_transform(X) # Assume X is having more than 2 dimensions
pk.dump(pca, open("pca.pkl","wb"))
.
.
.
# later reload the pickle file
pca_reload = pk.load(open("pca.pkl",'rb'))
result_new = pca_reload .transform(X)

# result and result_new same in my case


Given a list of thresholds in descending order will the corresponding FPRs and TPRs lists always be in ascending order? I've made a series of predictions with a machine learning model. I have a list y of labels and a list p of predicted probabilities such that p[i] is the predicted probability of the entry associated with y[i].

With Python's sklearn library I ran

fprs, tprs, thresholds = metrics.roc_curve(y, p)

thresholds is a list in descending order of decision thresholds, while fprs and tprs is the list of associated false positive rates and true positive rates where fprs[i] and tprs[i] are the respective rates given the decision threshold at thresholds[i].

My question is, given that thresholds is in descending order, will fprs and tprs be guaranteed to be in ascending order? I believe that as the decision threshold decreases, more and more probabilities will be marked positive, thus both rates will increase monotonically.

But I've been wrong before about things that seem obvious to me, and I'm writing a tool that must assume these lists will be sorted in ascending order, so I'd like to be sure

From sklearn Documentation: roc_curve returns fpr and tpr, which are increasing; and thresholds which is decreasing. Also check the examples below the definitions.


Choosing best predictors neural networks there,

I'm currently working on a project where my database has about 120 patterns with 39 columns and I am trying to build a predictive neural network out of it. It's regression task.

I was trying to get the best predictors ( alone or combinations of'em) in a simple network( 3 neurons only) to then use cross validation to better tune the model.
Problem are
1) the powerset of it is huge and my computer can't even handle generating the whole subset for simple fitting
2) only 3 neurons are already giving pretty poor results (r2<0)

Does somebody know a method or could please recommend a reading on choosing predictors for neural networks ?

Setup: windows 10, using MLPRegressor from scikitleran with hyperparameters
Hidden layer sizes = 3, max_iter= 5000, solver='sgd'

A very quick way is to run some Tree-based ML model on your data, such as Random Forest or XGBoost. Tree-based models can return importance coefficients, estimating the relative explanatory power of each variable. You can implement a very large and deep ensemble of trees (we don't really care about overfitting at this point) so they return you the three strongest predictors. You can then take them and feed them into a Neural Network.

Another, more time consuming method is to run the model multiple timesa and substitute each variable, in alternation, with random noise with the same mean and standard deviation of the original variable. This perturbation method will tell you how much performance decreases when one variable is replaces by noise. This is accurate but very time consuming.


Best format for recording time stamp and GPS In most data acquisition settings it is useful to tag your data with time and location. If I write the data to csv file, what are the best formats that I can use for this two variables if I want to create a heatmap on Google Maps?

As Spacedman put it, "best" is pretty subjective. However, as we have found, a good format for time is Unix time (aka POSIX time, aka Epoch time). Most databases support it and it is still pretty human readable.

For location, we like decimal degrees as it is easy to read and stored and is compatible with Google Maps API. It's also easy to convert to other formats if needed.


Input standartization for Deep Learning - Proper Scaling Typically the input to neural network (NN) is transformed to have zero mean and 1 std.

I wonder why std scale should be 1? What about other scales? 10? 100?
Doesn't it make sense to provide NN with input of wider range so that NN can separate different clusters easier and deal with loss function for each cluster in more simple and robust way?
Did someone here tried different scales and can share his experience?

If answer depends on the activation function - in my case I use Relu.

Thanks a lot!

First, it is important to keep in mind that neural networks (like many other machine learning algorithms) work over the domain of the real numbers, i.e. they assume for granted the properties of this space. Some properties that are relevant to this context:

additive identity is 0 and multiplicative identity is 1
there are inherent (intuitive to us) total order and a standard distance

A side note on the implications of this: if one is planning to use neural networks for categorical or ordinal data, one needs to consider what mapping into the space of real numbers is reasonable and whether the standard mapping (e.g. to natural numbers or via string similarity) makes sense in the context of the problem.

From theoretic viewpoint scaling is a homogeneous transformation, i.e. it preserves the properties of the space up to a scalar, thus it doesn't matter what you choose. Since 1 is the multiplicative identity (nothing changes when you multiply by one) it makes sense to use it as a scale, because it simplifies calculations.

Another side note: This is actually valid also in a more general context. For example the machine representation of floating point numbers is via the so-called scientific notation. From this perspective, 0.01, 0.1, 1, 10, 100,... all have the same significand and only the exponent changes. Thus, the difference between the examples you mention is actually quite small.


does KNN have a hypothesis space? if yes, Is there a way to clearly define the hypothesis space with the instances? I am learning this post,
"A few useful things to know about machine learning"

The author says

  A classifier must be represented in some formal language that the
  computer can handle.Conversely, choosing a representation for a learner is tantamount to
  choosing the set of classifiers that it
  can possibly learn. This set is called
  the hypothesis space of the learner.

and put KNN and SVM in the instance-based category.

I assume the instance here mean each individual example, for instance, KNN needs to compute the distance to each example.

Assume there are 300 examples to train a KNN classifier, how to understand choosing this specific representation is tantamount to choose hypothesis space?

to clarify the question.

does KNN have a hypothesis space? if yes, Is there a way to clearly define the hypothesis space with the instances?

The hypothesis in this case is a way to formally describe the model. The model could be a hyperplane, a set of conditions on input with a constant for each branch (decision trees) or whatever you like. The set of all possible hypothesis in a given representation is named hypothesis space. Knn prediction is realized using the first k instances from a training data set. So knn cannot represent a hyperplane because of the finite amount of datapoints. This is why choosing a representation restricts the hypothesis space. You are restricted to things which can be represented by instances and distance. SVM is the same because the support vectors are selected instances from the training dataset and the fitted surface is obtained using support vectors and kernelized distances from those.


Normalization and Outlier on Target variable which is continuous I have doubt that should I perform outlier analysis and normalization even on target variable which is continuous ?

You should do Outlier Analysis of your target variable to prepare your training data for the model. Most model would perform better on noiseless data, as Outlier might skew the findings of your model in one direction.

Generally, there is no need to perform normalization on target variable for model performance or accuracy. (Though it might be useful to do some analysis on target variable to get some useful business insights out of it)

Reasons behind performing normalization on input variables are as follows:
1) Feature scaling improves convergence of steepest descent algorithms
2) Helps to avoid a situation when several variables dominate other variables in magnitude

While if you normalize target variable, it, in turn, will normalize MSE and there will be no impact on results.

Cases when you might choose to do Normalization of Target Variables:

Neural Networks update weights of nodes based on the error generated by the cost function via back propagation, large errors might cause drastic change in the weights and could make learning process unstable. Results of which optimizer might not be able to settle on Optimal Minima.
The only time when you might want to normalize target is the case of floating point overflow. Sometimes the number is too large or too small that CPU memory can't handle it and will turn into INF or wrap-around to the other extreme representation.


How to transform many columns of TRUE/FALSE/NA to just TRUE/FALSE? I have a dataframe that consists of a few columns of text, and then a bunch of columns that are TRUE/FALSE or NA (they were TRUE/FALSE but I left-joined them with merge and that added NAs).

Eg:

issue # | title | body  | x     | y     | z    | lbl1 | lbl2  | lbl3  | lbl4 | lbl5
1       | blah  | blah  | blah  | blah  | blah | TRUE | FALSE | FALSE | TRUE | FALSE
2       | blah  | blah  | blah  | blah  | blah | TRUE | FALSE | FALSE | TRUE | FALSE
3       | blah  | blah  | blah  | blah  | blah | NA   | NA    | NA    | NA   | NA
4       | blah  | blah  | blah  | blah  | blah | NA   | NA    | NA    | NA   | NA
5       | blah  | blah  | blah  | blah  | blah | TRUE | FALSE | FALSE | TRUE | FALSE

I know how many columns need to not converted (and also their names), though I don't know how many label columns there are (or their names - they don't share any prefix).

I tried doing:

data[,-7] <- as.logical(isTRUE(data[,-7]))

Since this seemed to work with -1 for the same elsewhere, however my first columns all ended up as TRUE/FALSE too.

I also tried:

data[8:ncol(data)] <- sapply(data[8:ncol(data)], isTRUE)

But that resulted in everything being FALSE!

I also tried:

data[data==NA] <- FALSE

But that didn't seem to do anything (still has NAs).

I'm completely new to ML and R so please bear that in mind when answering. I don't know hardly any of the functions (or even completely understand all the syntax for selecting/replacing subsets of the dataframe as I'm trying to do here!).

I found that this works (though I'm interested if there's a better way - eg. avoiding the loop, or being able to exclude the columns by name instead of being just first 8).

for (i in 8:ncol(data)){
  data[,i] <- sapply(data[,i], isTRUE)
}


How can I transpose a high dimensional dataset? I have a (.csv) file with more than 35,000 rows and 100 columns, where the rows represent the attributes and columns represent the instances. In Excel the maximum number of columns is 16,384. Therefore, it is impossible to do this in one single Excel sheet. I need my dataset to be transposed in order to perform some machine learning algorithms in WEKA. Is there any tool allow such thing like this?

Also, is there a way to transpose (.arff) file in WEKA?

This is possible to do in WEKA, but it only works if:

the first column of your data is of string or nominal type, and
the rest of your columns are numeric.

Open the dataset in the explorer, and first remove the class attribute by choosing 'No Class' from the drop-down menu on the right. Then, apply the filter from filters.unsupervised.attribute.Transpose. After this, you should re-apply the class attribute by re-selecting the class attribute in the drop-down menu.

If your data doesn't meet the requirements listed above, I would recommend using this short python script to do it:

import pandas as pd
df = pd.read_csv('your_file_here.csv', index=False)
df.transpose().to_csv('your_new_file_here.csv', index=False)

and then loading the new file into WEKA.


Why does image segmentation benefit from fuzzy clustering? In many image processing papers, I've seen that they used fuzzy logic for segmentation
I wonder how fuzzification impact the result that made Fuzzy-C-Means better than ordinary K-Means.

PS. If possible could you provide me the sample data sets for case study

If you take aerial images for example, you might need to identify overlaps.

Lets say your goal is to segment water, vegetation and rock areas. More to be assigned to the closest centroid, there are hidden memberships to other centroids that can be identified by fuzzy c-means image segmentation.

On this thesis at https://zone.biblio.laurentian.ca/handle/10219/2616, at section (4.5 Analysis - Task 1) you can find detailed analysis using both (k-means & fuzzy c-means).


Understanding Loss function and Learning Algorithm In Keras, when specifying a loss such as Mean absolute error, does it replace the cost function in the learning algorithm (Adam or SGD) with the mean absolute error? I'm new to ML and a bit confused on this aspect.

Learning Algorithm like SGD or Adam don't have any Loss/cost function associated with them. First You have to build a Model. Then pass the output of the model and target (Your labelled output) to the cost function.Then the cost function will compare the differences in output of target and model and also and calculate the gradient of loss function with respect to parameters.Then it is the optimizer like SGD that will use the gradient from previous step along with a learning rate to update the parameters of model. Cost function and Optimizer are two distince seperate things. You can basically use any cost function (even of your own) in combination with any optimizer.
It is easy to visualize like this:
MODEL ----> LOSS FUNCTION ----> OPTIMIZER


Training the network with some batch size - code There is my "training" code below, I wrote it based on one youtube tutorial. I don't understand actually one part:  batch_X = train_X[i:i+BATCH_SIZE], batch_y = train_y[i:i+BATCH_SIZE]. How to understand this thing inside the brackets (i:i+BATCH_SIZE)?
def train(net):
            BATCH_SIZE = 32
            EPOCHS = 10
            for epoch in range(EPOCHS):
                # training loop
                net.train()
                for i in tqdm(range(0, len(train_X), BATCH_SIZE)):
                    batch_X = train_X[i:i+BATCH_SIZE].view(-1,3,224,224)
                batch_y = train_y[i:i+BATCH_SIZE]

                batch_X, batch_y = batch_X.to(device), batch_y.to(device)

                net.zero_grad()
                outputs1 = net(batch_X)
                matches1 = [torch.argmax(i)==torch.argmax(j) for i, j in zip(outputs1, batch_y)]
                acc = matches1.count(True)/len(matches1)
                loss = loss_function(outputs1, batch_y)
                loss.backward()
                optimizer.step()

            print(f"Epoch: {epoch}. Training acc: {round(float(acc),2)} Training loss: {round(float(loss),3)}")

That part of the code will select the samples that belong to a specific batch. The for loop first loops over the data in train_X in steps of BATCH_SIZE, which means that the variable i holds the first index for each batch in the training dataset. The rest of the samples for the batch are then the ones after that index up to the sample which completes the batch. This is done using train_X[i:i+BATCH_SIZE].


Is there a way to measure correlation between two similar datasets? Let's say that I have two similar datasets with the same size of elements, for example 3D points :

Dataset A : { (1,2,3), (2,3,4), (4,2,1) }
Dataset B : { (2,1,3), (2,4,6), (8,2,3) }

And the question is that is there a way to measure the correlation/similarity／Distance between these two datasets ?

Any help will be appreciated.

I would take a look at Canonical correlation Analysis.


High validation loss, high validation accuracy I'm just getting started into the field of deep learning, and I completed my first model training using PyTorch.
I decided to use a pre-build model from torchvision, more specifically the mobilenet_v2 (https://pytorch.org/docs/stable/_modules/torchvision/models/mobilenet.html#mobilenet_v2) on a custom dataset for binary classification.
I manage to get 100% accuracy for both training and test sets (this particular dataset is not a difficult one, so nothing weird about that)
What I don't understand is why the test loss seems to be several orders of magnitude higher than the train loss, while the accuracy is 100% in both cases. Could someone here explain me what is happening here?
I show you the training stats below:
Epoch: 1/15, Train Loss: 0.22990, Train accuracy: 88.48%, Test Loss: 2.10275, Test accuracy: 99.22%
Epoch: 2/15, Train Loss: 0.03988, Train accuracy: 99.41%, Test Loss: 2.08563, Test accuracy: 99.22%
Epoch: 3/15, Train Loss: 0.02211, Train accuracy: 99.41%, Test Loss: 2.05521, Test accuracy: 100.00%
Epoch: 4/15, Train Loss: 0.01173, Train accuracy: 99.80%, Test Loss: 2.05332, Test accuracy: 100.00%
Epoch: 5/15, Train Loss: 0.00499, Train accuracy: 100.00%, Test Loss: 2.04989, Test accuracy: 100.00%
Epoch: 6/15, Train Loss: 0.00381, Train accuracy: 100.00%, Test Loss: 2.04952, Test accuracy: 100.00%
Epoch: 7/15, Train Loss: 0.00171, Train accuracy: 100.00%, Test Loss: 2.04999, Test accuracy: 100.00%
Epoch: 8/15, Train Loss: 0.00072, Train accuracy: 100.00%, Test Loss: 2.04971, Test accuracy: 100.00%
Epoch: 9/15, Train Loss: 0.00045, Train accuracy: 100.00%, Test Loss: 2.04938, Test accuracy: 100.00%
Epoch: 10/15, Train Loss: 0.00035, Train accuracy: 100.00%, Test Loss: 2.04932, Test accuracy: 100.00%
Epoch: 11/15, Train Loss: 0.00029, Train accuracy: 100.00%, Test Loss: 2.04920, Test accuracy: 100.00%
Epoch: 12/15, Train Loss: 0.00025, Train accuracy: 100.00%, Test Loss: 2.04922, Test accuracy: 100.00%
Epoch: 13/15, Train Loss: 0.00022, Train accuracy: 100.00%, Test Loss: 2.04906, Test accuracy: 100.00%
Epoch: 14/15, Train Loss: 0.00020, Train accuracy: 100.00%, Test Loss: 2.04914, Test accuracy: 100.00%
Epoch: 15/15, Train Loss: 0.00018, Train accuracy: 100.00%, Test Loss: 2.04905, Test accuracy: 100.00%

In binary classification we have either 0 or 1. You must be using softmax function at the end layer to get the output . It ranges from (0,1)
Loss: Generally speaking , it is the difference between predicted and actual value.
Suppose
CASE 1:predicted value is 0.6 real answer is 1.
CASE 2:predicted value is 0.9 real answer is 1.
Although they both have the correct answers(because they both round off to become 1) , loss in case 2 is less than case 1.


Where can I find a software library for pairwise matching (ideally, Python, R, Java)? I am looking for a library that implements a pairwise ranking algorithm. For example, if I have 200 writing samples from 100 people (two samples from each individual) and I want to identify which samples belong together (i.e., were written by the same person), what library could I use?

If you can transform those sentences into number vectors (e.g. into a bag of words or tf-idf representation), I guess you could use k-Means or hierarchical clustering functionality from Orange, a GUI and machine learning library written in Python.

It also has an add-on for text mining specifically, but I cannot attest to it as I haven't tried it yet.


How to use Kaggle Api in Google Colab for directly using dataset? I know that we can use Kaggle's api directly in google colab which downloads the dataset. The commands are :

!mkdir .kaggle
!echo '{"username":"somename","key":"apikey"}' > /root/.kaggle/kaggle.json
!chmod 600 /root/.kaggle/kaggle.json
!kaggle competitions download -c bluebook-for-bulldozers -p /content

But I need to do this process of making .kaggle file and pass the apikey in google colab gpu everytime. And sometimes the echo command run saying no file called .kaggle but after say 2 mins without restarting the kernal, it works. It sounds funny but yes this is true. Can't I just make a .kaggle file(only once) and use these commands just once and download the dataset just once and store it somewhere and use it even for later purposes. I have used google drive mount process but it's hectic and takes lot of time uploading the datasets in drive. It would also be ok if I need to download the dataset everytime using just this command rather than making .kaggle file and writing the api key and username every time in it:

    !kaggle competitions download -c bluebook-for-bulldozers -p /content

The success rates of previous commands are pretty less in one go and wastes lot of time.

Step 1 - Refer the Kaggle doc to understand the API basics and to get your key
https://github.com/Kaggle/kaggle-api

Step 2 - Use these LoC -

    import os
    os.environ['KAGGLE_USERNAME'] = "jha01roshan"
    os.environ['KAGGLE_KEY'] = "xxxxxxxxxxxxxxxx"
    import kaggle

    !kaggle competitions download -c ashrae-energy-prediction

    import pandas as pd
    building_metadata = pd.read_csv("/content/building_metadata.csv")
    sample_submission = pd.read_csv("/content/sample_submission.csv.zip")


Combining multiple probabilities from a classifier. Propagating probabilities Let's say I have trained a classifier that classifies images of animals into 10 different classes. And let's say that I have 20 different images of a particular animal and because I know the photographer, I know with certainty that all 20 images are of the same animal. So I use my classifier to make a prediction on what animal it is and get 20 predictions one for each image. The model predicts all the images to be a dog with varying probabilities.
image 1: 80% dog
image 2: 90% dog
image 3: 75% dog
and so on.
What is the probability that the animal in question is a dog?
Let's say they predict cat with smaller probabilities, 5%, 2%, 4% ... What is the probability it is a cat?
I've tried a few different approaches, applying Bayes Theorem but I keep getting numbers that add up to be more than one.   Could it really be just the average?

If you feed the model with 20 images for testing then your output should look like array of [20x10]. For each row represents the probabilities of all classes(in your case 10). Let's see the example below, and 1-index is dog with 0.9 probability, so your model classified it correctly. If your model classifies 19 times dog-correct and 1 times cat-wrong then your model's test accuracy will be calculated as correct_preds / (correct_preds+wrong_preds)
[[0.0, 0.9, 0.0, 0.0, 0.0, 0.1, 0.0, 0.0, 0.0, 0.0],
 [0.0, 0.8, 0.0, 0.0, 0.0, 0.1, 0.0, 0.1, 0.0, 0.0],
 ..,
 [0.0, 0.75, 0.0, 0.0, 0.0, 0.1, 0.0, 0.1, 0.05, 0.0]]


Finding out the scale when using MinMaxScaler() I am using the MinMaxScaler() of sklearn to scale my features before using kmeans.I needed to find the scale used.

from sklearn import preprocessing
scaler = preprocessing.MinMaxScaler()
scaler= scaler.fit_transform(finance_features)
print scaler.scale_

However on using the line

print scaler.scale_

It shows an error saying no such attribute

This is because you were importing a class as name 'scalar' and modifying it by assigning value obtained after transforming. Just change

from sklearn import preprocessing
scaler = preprocessing.MinMaxScaler()
scaled = scaler.fit_transform(finance_features)
print scaler.scale_

This should work.


Deriving Prediction Intervals for Orthogonal Distance Regression using `scipy.odr` Questions

How can I derive prediction intervals for predictions based on new observations from the output of scipy.odr?
Is it also possible (or necessary) to take into account uncertainties in the new observations?

Background
I would like to perform a linear regression between two sets of variables, both of which have uncertainties associated with them, and also be able to derive prediction intervals as part of the output.
I have identified Orthogonal Distance Regression (ODR) as a possible method by which this could be done, and the scipy.odr library as an implementation of this method.
What I have been unable to find is how to use the output of this library to derive prediction intervals. I understand from this page that I need to add - in quadrature - the estimate of the standard deviation of the predicted value and the estimate of the residual standard deviation obtained when fitting the model to the data, but I am unsure how to derive these quantities from the output of scipy.odr. I am also almost completely unsure how to take into account any uncertainties in new observations, as surely these will affect the resulting prediction intervals.

[I am working on researching a complete answer, this is as far as I've got so far, I will edit as I work out more of the solution.]
Further research has turned up this paper which provides at least part of the answer.
Briefly, ordinary least squares (OLS) does not consider uncertainties on either axis, and weighted least squares (WLS) only takes into account uncertainties in the predictor variable / dependent variable / on the y-axis. The above publication advocates the use of bivariate least squares (BLS) over orthogonal distance regression (ODR), but goes on to derive formulae for prediction intervals which appear to be compatible with ODR.
In this approach, each 'data point' $\left(x_{i}, y_{i}\right)$ is considered to be the result of multiple experiments or observations under ostensibly the same conditions, thus allowing the calculation of the variance for both the predictor and response variable pair.
The predictor ($x_{i}$) and response ($y_{i}$) variables are related by the following equation
$y_{i}=b_{0}+b_{1} x_{i}+e_{i}$
where $b_{0}$ and $b_{1}$ are the estimates of the intercept and slope of the true linear model, and $e_{i}$ is the $i$th residual error. The variance of $e_{i}$ is referred to as the weighting factor and is denoted as $w_{i}$ or $s_{e_{i}}^{2}$:
$w_{i}=s_{e_{i}}^{2}=s_{y_{i}}^{2}+b_{1}^{2} s_{x_{i}}^{2}-2 b_{1} \operatorname{cov}\left(x_{i}, y_{i}\right)$
where $s_{x_{i}}^{2}$ and $s_{y_{i}}^{2}$ are the experimental variances of point $i$, and $\operatorname{cov}\left(x_{i}, y_{i}\right)$ is the covariance between the measurements for each $\left(x_{i}, y_{i}\right)$ data pair.
The variance of the response variable $y_{0}$, being the mean of the $q$ observations performed at $x_{0}$, is given by:
$s_{y_{0}}^{2}=\left(\frac{1}{q}+ X _{0}^{ T }\left( X ^{ T } W ^{-1} X \right)^{-1} X _{0}+s_{x_{0}}^{2} b_{1}^{2}\right) s^{2}$
where $s^{2}=\frac{\sum_{i=1}^{n}\frac{\left(y_{i}-\hat{y}_{i}\right)^{2}}{w_{i}}}{n-2}$ is the estimate of the true experimental variance, $X_{0}$ is a two element column vector $\left|\begin{array}{l}
1 \\
x_{0}
\end{array}\right|$, and $X$ is an $n \times 2$ matrix for which the first column is a column of ones and the second column is formed by the $n$ values of $x$ corresponding to the experimental points; $W$ is an $n \times n$ diagonal matrix whose $i$th diagonal element is the weighting factor $w_{i}$, defined above.
The variance of the predictor mean value at a given observation $y_{0}$ is
$s_{x_{0}}^{2}=\left( Y _{0}^{ T }\left( Y ^{ T } W ^{\prime -1} Y \right)^{-1} Y _{0}+s_{y_{0}}^{2} \frac{1}{b_{1}^{2}}\right) s^{\prime 2}$
where $Y _{0}$ is $\left|\begin{array}{l}
1 \\
y_{0}
\end{array}\right|$, $Y$ is an $n \times 2$ matrix for which the first column is a column of ones and the second column is the values of $y$, $W^{\prime}$ is an $n \times n$ diagonal matrix whose $i$th diagonal element is the weighting factor  $w_{i}^{\prime}=s_{x_{i}}^{2}+\frac{1}{b_{1}^{2}} s_{y_{i}}^{2}-2 \frac{1}{b_{1}} \operatorname{cov}\left(x_{i}, y_{i}\right)$, and $s^{\prime 2}$ the experimental error associated with the predictor variables, given by
$s^{\prime 2}=\frac{\sum_{i=1}^{n} \frac{\left(y_{i}-\hat{y}_{i}\right)^{2}}{w_{i}^{\prime}}}{n-2}$
The variance of the prediction of the predictor variable of a future sample at $y_{0}$, the mean of $q$ observations, is
$s_{x_{0}}^{2}=\left(\frac{1}{q}+ Y _{0}^{ T }\left( Y ^{ T } W ^{-1} Y \right)^{-1} Y _{0}+s_{y_{0}}^{2} \frac{1}{b_{1}^{2}}\right) s^{\prime 2}$
The prediction intervals of the response and predictor variables are then given by
$y_{0} \pm t_{\alpha, n-2} s_{y_{0}}$
$x_{0} \pm t_{\alpha, n-2} s_{x_{0}}$
where $t_{\alpha, n-2}$ is the $t$-value for the required significance level $\alpha$ and $n-2$ degrees of freedom.
TODO: Interface to scipy.odr output.


Reliable way to verify Pyspark data frame column type If I read data from a CSV, all the columns will be of "String" type by default. Generally, I inspect the data using the following functions which gives an overview of the data and its types

df.dtypes
df.show()
df.printSchema()
df.distinct().count()
df.describe().show()

But, if there is a column that I believe is of a particular type e.g. Double, I cannot be sure if all the values are double if I don't have business knowledge and because

I cannot see all the values (millions of unique values)
If I explicitly cast it to double type, spark quietly converts the type without throwing any exception and the values which are not double are converted to "null" - for example

Code:

from pyspark.sql.types import DoubleType

changedTypedf = df_original.withColumn('label', df_control_trip['id'].cast(DoubleType()))

What could be the best way to confirm the type of column then?

If you don't have business knowledge, there is no way you can tell the correct type, and no way you can 'confirm' it. You can at most make assumptions about your dataset and your dataset only, and you for sure have to inspect every value.

In your example, you created a new column label that is a conversion of column id to double. You could count all rows that are null in label but not null in id. If this count is zero you can assume that for this dataset you can work with id as a double. That doesn't necessarily mean that in a new dataset the same will be true for column id.


What's in an SGD classifier object? I ran some text classification on text data. When I tried to save the SGD classifier object, I was surprised to find it is more than 4GBs (even though my input is ~600 000 rows, 700MB). What's in that object that makes it so huge? Any thoughts?

I will answer, assuming that since you wanted to do text classification, your features were created using some TFIDF feature extraction technique. Next time, please specify the dimensions of your features and not only their storage.

According to the documentation (http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html), the object has a coef_ attributes that stores all of the weights of your model. Since for text, feature vectors are about the size of your vocabulary, it can be fairly large, especially if you do not preprocess your text using stemming or if you do not remove stopwords or irrelevant words.

The documentation also says that the size of the coef_ is number of classes * number of features. So depending on how many features and classes you have, this can get large quickly.

This could explain partially why the object is so large. Other factors could be that Python often stores the predictions in the object after you train it, so that you can access them quickly, so you would need to add another 600,000 numbers to your object. And for this object, it seems to also store the confidence scores, which is another 600,000.


Predict the date an item will be sold using machine learning I would like to predict the date a item will be sold using features such as:

      Product ID          Price        Days_since_first_post       Last_repost     Type       First_Post_Date         Sold_Date

How would machine learning principles be used in such a way to do this task? If so, what is the proper means by which to design such an algorithm? What would be the different steps which would be involved.

More generally, what are the logical steps that should be taken when tackling a machine learning problem.

Thank you

A machine learning problem can be separated into a few modular parts. Of course these are all massive in reach and possibility. However, every problem you encounter should be thought of in this way at first. You can then skip things where you feel necessary.

Data pre-processing and feature extraction
The model
Post-processing

Data pre-processing and feature extraction

This and feature extraction are the two most important parts of a machine learning technique. That's right, NOT THE MODEL. If you have good features then even a very simple model will get amazing results.

Data pre-processing goes from your raw data and remolds it to be better suited to machine learning algorithms. This means pulling out important statistics from your data or converting your data into other formats in order to it being more representative. For example if you are using a technique which is sensitive to range, then you should normalize all your features. If you are using text data you should build word vectors. There are countless ways pre-processing and feature extraction can be implemented.

Then, you want to use feature selection. From all the information you extracted from your data not all of it will be useful. There are machine learning algorithms such as: PCA, LDA, cross-correlation, etc. Which will select the features that are the most representative and ignore the rest.

In your case

First, let's consider the data pre-processing. You notice that type might not be an integer value. This may cause problems when using most machine learning algorithms. You will want to bin these different types and map them onto numbers.

Feature selection: besides using the techniques I outlined above, you should also notice that productID is a useless feature. It should for sure NOT be included in your model. It will just confuse the model and sway it.

As a general rule of thumb, the amount of data that is suggested to have for shallow machine learning models is $10 \times \#features$. So you are limited by the size of your dataset. Also, make sure the outputs are quite well distributed. If you have some skew in your dataset, like a lot of examples where the item was sold right away, then the model will learn this tendency. You do not want this.

The Model

Now that you have your feature-space, which might be entirely different from the original columns you posted, it is time to choose a model. You are trying to estimate the time of sale for an algorithm. Thus, this can be done in two different ways. Either as a classifier problem or as a regression problem.

The classifier problem would separate the different times of sales into distinct bins. For example

Class 1: [0 - 5] days
Class 2: [5 - 10] days
etc...

Of course the more classes you will choose to have then the harder it will be to train the model. That means the resolution of your results is limited by the amount of the data you have available to you.

The other option is to use a regression algorithm. This will learn the tendency of your curve in higher dimensional space and then estimates where along that line a new example would fall. Think of it in 1-dimension. I give you a bunch of heights $x$ and running speeds $y$. The model will learn a function $y(x)$. Then if I give you just a height, you will be able to estimate the running speed of the individual. You will be doing the same thing but with more variables.

There are really a ton of methods that can do this. You can look through some literature reviews on the subject to get a hold of all of them. But, I warn you there is A LOT. I usually start with kernel-Support Vector Regression (k-SVR).

To test your model. Separate your dataset into three parts (training, validating, testing) if you have sufficient data. Otherwise two parts (training, testing) is also fine. Only, train your model on the training set and then evaluate it using the example it has not seen yet which are reserved in the testing set.

Post-processing

This is the step where you can further model your output $y(x)$. In your case that might not be needed.


why does performance of machine learning models plateau after certain amount of training data I'm referring to the below image which I came across.

The explanation seemed intuitive at first but I don't think I understand how it works. The image says that the performance of traditional machine learning algorithms plateau after a certain amount of data while that of deep learning algorithms gets better with more data. Does it have to with the feature learning where deep learning methods automatically learn the important features as opposed to the manual feature selection for machine learning models? If so, can you please elaborate on that.

In part feature learning has a lot to do with the fact that deep learning models can learn more data distributions. Although the graph that you are showing isn't completely true. For instance you can see the No Free Lunch Theorem for machine learning, that states that no machine learning model is universally any better than any other.

Either way, the goal of machine learning is not to seek a universal learning algorithm. Instead our goal is to understand what kind of dat distributions are relevant to the the real world that an AI agent experiences, and what kinds of machine learning algorithms performs well on data drawn from the kinds of data generating distributions we care about.

In conclusion, when facing a machine learning problem you should try to find the model that best learns the kind of data you are trying to work with and this graph is poorly informative and has no  foundations.


How to model a supervised recommender system with varying data Suppose there are 2000 movies and a company wants to recommend some movies (for example, at most 5 movies) to each visitor. The objective is to learn how to predict which movie will be selected if a specific set of movies is recommended.
   option-1  option-2  option-3  option-4  option-5  Selected-Movie
1. movie1    movie3    movie4                        movie4
2. movie3    movie4    movie100  movie1000 movie1001 movie1001
3. movie4    movie5    movie34                       movie34

Based on this data set, I want to learn when sample 1 is suggested to a customer, he will visit movie4. Because the number of features can be so high (here 2000 movies), I think it would not be a good option to use on-hot-encoding. Think at most 5 movies can be recommended, I thought it might be a good option to consider a vector with size 5 and if the number of recommended movies is less than 5, blanks will be replaced with 0. However, in this situation, the perturbation of movies will be important. For example, (1,2,3,4,5) will be different from (2,1,3,4,5) and I want to consider both cases the same. In other words, all 5! perturbations should be the same and there is no difference between them. Moreover, with this representation of data, I think it will not be possible to use Decision Tree whereas some algorithms like Catboost works.
My preference are algorithms that can generate rules like Decision Tree. I would be thankful if you have any recommendation for data representation and how features should be considered.

In this task you're missing something: you don't have any features to represent a specific visitor.
This means that the best movie that your model can predict is the one which is selected the most often by any visitor. As a consequence, the only thing that the model can learn from such a dataset is to associate every possible sequence with the most frequently selected movie given this sequence. Of course it would have to generalize a bit for sequences which don't appear in the training data, but that's the most ML there is in this task. In theory the task can be done just by counting the joint frequency sequence+selected-movie in order to calculate the conditional probability selected-movie given sequence.

[edit]
The standard way to represent a set would be one hot encoding. It's doable even with 2000 features, assuming there are enough instances in the data. In this basic option you could certainly reduce the number of features by removing the movies which are never/rarely proposed or never/rarely selected. This would be unlikely to achieve performance, because if there are no or few cases where a movie is proposed/selected, then the model cannot use this information (or if it does it's going to cause overfitting).
I could think of a couple alternative approaches:

Apply clustering on the sequences of movies as OHE (without the target variable), then train a model with for every cluster: this way there are less features to deal with for every model.

Just rank all the movies by order of preference based on how often the movie is selected. Given that there is no user preference, it can be assumed that there is a total order on the movies:

the top preferred movie is always selected, whatever the other movies proposed

the 2nd top preferred movie is always selected, unless the first one is proposed as well

... the $n^{th}$ top is always selected, unless another one higher in the top is proposed as well

To predict the selected movie, just pick the highest ranked among the 3-5 proposed.


How to scale outputs from AutoEncoder from multiple models? I have a problem for which I have not been able to find any answers in my search so far.

BACKGROUND

I am working on an anomaly detection problem on machines utilising an auto-encoder. I am building a model file per machine because the machines' temporal behaviour varies quite a lot.

I have 5 features:

Numerical integer ranging between 0 and x (x varies per machine)
The other 4 features are categorical (After trying LabelEncoding, my architecture prefers One-Hot encoding)

I have tried to scale the numerical feature using Normalisation (MinMaxScaler and StandardScaler) which did not yield very good results at all.

As an alternate to scaling the inputs - I decided to scale the outputs using MinMaxScaler from scikit. This is so that I can have 1 generic threshold I can apply across the different models to identify anomalies.

Although this has yielded the best results so far - in practice, the outputs become too polarised to either the 0 or the 1 and consequently I am missing outliers that I shouldn't be.

PROBLEM

What scaling technique(s) can I use on the output from my auto-encoder such that I can apply 1 generic threshold across all models to identify anomalies?

I always recommend to use StandardScaler() rather than MinMaxScaler(). The reason is that the latter is too sensitive to outliers. Imagine shrinking the given vector in the [0, 1] interval:
[1, 2, 3, 4, 999]

Most of your data would be squeezed in a tiny subspace of the [0, 1] interval, causing absurd compression. The problem doesn't exist if you use the StandardScaler(). This looks particularly relevant given that you're trying to detect abnormal observations.
(I would use min-max scaling only if I'm sure that a given variable is bounded within a given range of values.)
But most importantly: What are the problems of your output? As far as we know, they might not be due to scaling.

UPDATE:
Lately, a scaling technique that I found very effective is robust min-max scaling. It works just like a min-max, but instead of being done between min and max value, it is done between min and a certain percentile (usually in 95-99). That's very effective when your dataset suffers from very few (but influent) outliers. This method makes sure that 95-99% of your dataset is properly scaled in [0,1], while few outliers take reach out-of-range values.


Boolean classification on strings I have a supervised learning (boolean classification) problem that involves strings.  Are there any resources where I can learn about the state of the art in techniques for this?

I'm familiar with many supervised learning techniques where we assume that each instance has a fixed number of attributes: e.g., the instance is represented by a feature vector.  But now I have something different; I have a string, i.e., a sequence of characters.  I don't know a priori what suitable features might be.  Are there any general techniques for doing machine learning in this context?

I'm especially interested in techniques that work given an existing training set (I don't have the ability to label additional data; I don't want an active learning algorithm), and that support some kind of regularization or can handle noisy labels.

Research I've done and approaches that don't meet the above requirements:

I'm familiar with regexps, finite automata, grammar induction, LearnLib, Angluin's algorithm, and similar concepts.  However, most of that work is in the active learning context, where the algorithm generates a new instance (a query) and asks the human to label that instance.  In my setting, I must work with passive learning: I have a training set $(x_i,y_i)$ where each $x_i$ is a string and $y_i$ is its class (positive or negative), but I can't obtain any more examples.
I'm also familiar with automata minimization, so I know how to find the minimal DFA that can generate all positive instances in the training set, but that's not really a good solution.  It doesn't have any notion of regularization or Occam's razor and is not robust to a small amount of error/noise in the labels.  There might be a very small DFA that predicts the proper label with 99% accuracy, but automata minimization won't find it; it is overly focused on finding an automaton that generates all positive instance, even if this yields a much larger automaton.
I'm also familiar with the notion of separating automata, i.e., the smallest DFA that has 100% accuracy on the training set (accepts all positive instances and rejects all negative instances).  However, this has the same problems as automata minimization.  Also, this seems likely to overfit and essentially end up memorizing the training set.  Finally, the research literature I've seen stops at noting that finding the smallest separating automaton is NP-hard, and doesn't concern itself with practical algorithms for finding a separating automaton that is as small as possible.
I'm familiar with recurrent neural networks, but my impression is that they tend to require very large training sets, so probably won't be useful in my setting.  Are there variants of this technique that work with training sets that aren't enormous (say, hundreds of examples in the training set)?
I'm familiar with some techniques from the natural language processing (NLP) literature, but they seem very specialized to parsing human languages.  Are there any general techniques that apply to other structured strings?  In my situation, the strings are not human-readable text written in some human language, but rather have some other content.

There are many approaches to this type of problem, which would be defined as text classification. You could create a Term Frequency or Term Frequency Inverse Document Frequency matrix based on the tokens (words) in the strings, then use standard machine learning algorithms such as SVM, Random Forest, Gradient Boosted Trees, Logistic Regression, Naive Bayes. You could also potentially use word embedding's and neural networks. In this case, each word would be represented as a vector of similar words. Finally, I have recently been experimenting with character level convolutional neural networks and this seems to work well for user generated text classification. There is link below.

https://papers.nips.cc/paper/5782-character-level-convolutional-networks-for-text-classification.pdf


Handling a feature with multiple categorical values for the same instance value I have data in the following form:

table 1
id, feature1, predict
1, xyz,yes
2, abc, yes

table2
id, feature2
1, class1
1, class2
1, class3
2, class2

I could perform a one many join and train on the resultant set- which is one way to go about it. But If I rather wanted to maintain the length of the resultant set equal length of table 1, what is the technique?

One possible approach is to perform an encoding, where each level of the feature2 corresponds to a new feature (column).
This way you may describe the 1:N relation between the feature 1 and 2

Here a small example in R

> table1  <- data.frame(id = c(1,2),  feature1 = c("xyz","abc"), predict = c(T,T))
> table2  <- data.frame(id = c(1,1,1,2), feature2 = c("class1", "class2", "class3", "class2"))
>
> ## encoding
> table(table2)
   feature2
id  class1 class2 class3
  1      1      1      1
  2      0      1      0

The new object contains the (now unique) id and setting of the feature2.
You need only to merge (join) the result to the table1 (basically same task a DB join - which variance: inner, outer or full depends on your requirements).


How to generate a sample from a generative model like a Restricted Boltzmann Machine? I am learning about the Boltzmann machine. So far, I have successfully written a code that can learn the coefficients of the energy function of a Restricted Boltzmann Machine. Now, since my model is generative (if I have understood things correctly so far) and I know for sure that RBMs can be used for inpainting in binary images at least, I want to know how I can generate a sample from my probabilistic distribution given by the Boltzmann machine. That is how a new binary image based on the training dataset is generated by my model.

However, even though mathematically the idea seems clear, I do not know how I can program a computer to sample from a distribution.

For a RBM, you run the stochastic network - forward and back from the "input" and hidden layers - multiple times. After a few steps it will converge into sampling from the data population it has learned from. If you have learned from a whole picture and want to fill in a patch, then hold the input values that are not in the patch (don't allow them to change randomly).

Other generative models may have different approaches. GANs and VAEs typically have a simpler approach where you generate a random input vector and run the generator part of network forward from that input.


Recommender systems and Machine Learning According to Mitchell:

  “A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T , as measured by P , improves with experience E” (Mitchell, 1997) .

If we put this in terms of recommender systems:

Task: recommend products

Experience: includes the experience of observing a set of examples encoded in a ratings matrix

Performance: metric i.e. rmse between real and predicted ratings

Is my approach okay?

Any suggestions are welcomed!

For experience:

observing a set of examples encoded in a rating matrix is good, before doing that perform hybrid filtering(content-based filtering + collaborative), from its results
get insights and do rating matrix if needed.

Performance:

use rmse metric for better understanding of accuracy.


Converting dates to appropriate form to train machine learning model I am trying apply linear regression on a dataset where the independent variable is a date formatted like '2-jan-08'. How should I convert the date so it can be used for model fitting?

It depends on whether the date is the single source of features data X or if the date is a feature of X, along with other features.

If the date is the only independent variable you have time-series models, such as the autoregressive (AR), integrated (I) or moving average (MA) models. Their combination results in the ARMA or in the ARIMA models. Check this tutorial on using ARIMA with Python

If the date is a feature among others, you can convert the original date into multiple date features. Check this thread for feature engineering with time data.


Can someone please explain what this sample function is upto? So there is a function in Dino_Name_Generator at Deeplearning.ai notebook

def sample(parameters, char_to_ix, seed):
    # Retrieve parameters and relevant shapes from "parameters" dictionary
    Waa, Wax, Wya, by, b = parameters['Waa'], parameters['Wax'],parameters['Wya'], parameters['by'], parameters['b']
    vocab_size = by.shape[0]
    n_a = Waa.shape[1]

    ### START CODE HERE ###
    # Step 1: Create the one-hot vector x for the first character (initializing the sequence generation). (≈1 line)
    x = np.zeros((vocab_size, 1))

    # Step 1': Initialize a_prev as zeros (≈1 line)
    a_prev = np.zeros((n_a, 1))

    # Create an empty list of indices, this is the list which will contain the list of indices of the characters to generate (≈1 line)
    indices = []

    # Idx is a flag to detect a newline character, we initialize it to -1
    idx = -1

    # Loop over time-steps t. At each time-step, sample a character from a probability distribution and append
    # its index to "indices". We'll stop if we reach 50 characters (which should be very unlikely with a well
    # trained model), which helps debugging and prevents entering an infinite loop.
    counter = 0
    newline_character = char_to_ix['\n']

    while (idx != newline_character and counter != 50):

        # Step 2: Forward propagate x using the equations (1), (2) and (3)
        a = np.tanh(np.dot(Wax, x) + np.dot(Waa, a_prev) + b)
        z = np.dot(Wya, a) + by
        y = softmax(z)

        # for grading purposes
        np.random.seed(counter+seed)

        # Step 3: Sample the index of a character within the vocabulary from the probability distribution y
        idx = np.random.choice(vocab_size, size=None, p = y.ravel())

        # Append the index to "indices"
        indices.append(idx)

        # Step 4: Overwrite the input character as the one corresponding to the sampled index.
        x = np.zeros((vocab_size, 1))
        x[[idx]] = 1

        # Update "a_prev" to be "a"
        a_prev = a

        # for grading purposes
        seed += 1
        counter +=1

    ### END CODE HERE ###

    if (counter == 50):
        indices.append(char_to_ix['\n'])

    return indices

Can someone please help and explain what benefit of returned indices over normal char_to_integer indices?

I want to understand the text processing in the link carried out before feeding into the network.

From the link you provided:

  Sample a sequence of characters according to a sequence of probability
  distributions output of the RNN

  Arguments:
      parameters -- python dictionary containing the parameters Waa, Wax, Wya, by, and b.
      char_to_ix -- python dictionary mapping each character to an index.

  Returns:
      indices -- a list of length n containing the indices of the sampled characters

You are returning the indices from a dictionary you gave as argument. Why should you use char_to_integer indices.


Accuracy improvement for logistic regression model I have achieved 68% accuracy with my logistic regression model. I want to increase the accuracy of the model. How can I apply stepwise regression in this code and how beneficial it would be for my model? What changes shall I make in my code to get more accuracy with my data set. I have attached my dataset below. Following is my code:

library(dplyr)
data1 <- read.csv("~/hj.csv", header=T)
train<- data1[1:116,]
VALUE<-as.numeric(rownames(train))
testset<- data1[1:116,]
mylogit <- glm(VALUE ~ POINT1 + POINT2 + POINT3 + POINT4 , data = data1, family ="binomial")
testset$predicted.value = predict(mylogit, newdata = testset, type="response")
for (i in 1: nrow(testset)){
  if(testset$predicted.value[i] <= 0.50)
    testset$outcome[i] <- 0
  else testset$outcome[i] <- 1
}
print(testset)
tab = table(testset$VALUE, testset$outcome) %>% as.matrix.data.frame()
accuracy = sum(diag(tab))/sum(tab)
print(accuracy)
print(tab)
table(testset$VALUE, testset$outcome)

Following is my dataset:
Link 1: http://www.filedropper.com/hj_2

Try:

  mylogit <- glm(VALUE ~ POINT1 * POINT2 * POINT3 * POINT4, data = data1, family ="binomial")

with about 72% accuracy.


Predicting on real test set gives only very high probability for 1 for a very unbalanced data Excuse me for this brief description of the problem, as I'm very bound on time, I'll try to sum up as much as I can.

I have a multivariate time-series, that I trained using an RNN, there are periods and repeating time indexes, from 2013-01 to 2016-09, steps are months, by repeating, I mean various subsets ordered from January to December, many times for the same year, for hundreds of times, and I am predicting the next year knowing other features.
Predicting using Keras on real test data expecting the same shape as train data
I trained using LSTM, on 3 years, and trying to predict also repeating time-series for the year 2017. I used fixed batch size, and one last layer for binary target value so I used such a basic neural network:

model = Sequential()
model.add(LSTM(10, batch_input_shape=(12, train_X.shape[1], train_X.shape[2]), return_sequences=True, stateful=True, activation='sigmoid', inner_activation='hard_sigmoid'))
model.add(Dropout(0.2))
# model.add(LSTM(70))
# model.add(Dropout(0.3))
model.add(Flatten())
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy',
optimizer='rmsprop',
metrics=['accuracy'])

The batch size is 12 ( I chose) for 12 months, the target in train is very unbalanced with

pd.value_counts(test_y)
1.0 163781
0.0 5551
dtype: int64

Yet, I waited for some low probability for one in other words predicts of zeros.

res = model.predict(t_e_s_t, batch_size=12)

res
array([[0.9633749 ],
[0.79078996],
[0.99266464],
...,
[0.9891131 ],
[0.7582535 ],
[0.95778626]], dtype=float32)

All values of probability are above 0.5 and near 1 that means no probability for any entry to be zero. What could be wrong?

[Edit]

I added

from sklearn.utils import class_weight

class_weights = class_weight.compute_class_weight('balanced',
                                                 np.unique(train_y),
                                                 train_y)

and passed class_weights to fit method, still zero values under 0.5.

In unbalanced scenarios such as this, often the probability estimates are skewed massively in favour of the majority class, but the model is still learning to assign lower probabilities to the minority class, so it can be a good idea to choose a threshold higher than 0.5 to determine your final classification. It would be a good idea to look at a precision-recall curve (implemented in scikit-learn) which considers all thresholds between 0 and 1 and plots the precision and recall against each other for each threshold. You should pick the threshold that gives the best trade-off for your problem.


Neural net layer that preserves spatial information I'm attempting to use a neural network as a kind of interpolator for a high-dimensional function.

We're doing this to circumvent the need for a physical model that calculates this function exactly, but is expensive to run. We have a dataset of past runs of this physical model that we can now use as the basis for interpolation.

The input is a set of 28 parameter values.

The output we're trying to estimate consists of the shape of 432 periodic functions (i.e. value of 432 different functions along the circumference of a circle). We know that the output is uniquely determined by the 28 input values; there's no noise.

For example, here is one of these periodic functions. For this example, it's only sampled with 12 points (though in production we will be sampling it more finely, perhaps 36 points around a circle).

My current neural network Keras implementation is:

Input layer: 28 units (one per input parameter)
One hidden layer: fully connected, 1000 units
Output layer: fully connected, 432 periodic functions × 12 samples = 5184 units

As illustrated above, this works okay (even before optimizing hyperparameters).

But I'm concerned that with these fully-connected layers, I'm treating the 5184 outputs as unrelated numbers, completely disjointed from one another. In reality, there's some spatial relationship between the 12 samples describing each of the 432 underlying functions in the output. But I'm just throwing this spatial information away.

Is there another kind of layer which could take advantage of this spatial relationship, and thus lend itself better to this kind of regression?

If I were going the other way round, i.e. from spatial information to distinct values, I would consider convolution layers (perhaps with a periodic boundary). But what about this reverse case?

Rather than trying to predict each value explicitly, you could pick a parametric function that you think might be expressive enough to describe your target function (for example, a polynomial of order 6), and get your network to learn the parameters of that polynomial that give you a minimum loss. This is basically polynomial regression.  Since your function is periodic you might use a more cycle-based basis set, like linear combinations of sine waves.

For example:

1) Input data to the network
2) output a set of values that you treat as parameters to your function
3) for each target point, compare it to the predicted value from your function
4) calculate a loss (like mean squared error) based on the difference between the function predictions and the targets.

Using a function will (1) restrict the space of possible outputs, and (2) make nearby outputs be self-similar.


How can I compare classes from clusterings performed on two different data sets? I have two data sets defined by real valued vectors, and I have performed clustering on both of them. Now I want to compare the classes to see how they map to each other. If I put the data sets together into a single, bigger data set, and cluster that, I see that most of, say, class 1 of the first data set and class 2 of the second (numbers are arbitrary) fall into the same class of the total clustering, which makes sense, meaning they're probably equivalent. What would be a good rigorous way of performing this kind of analysis and giving a quantitative measure of this overlap? Thanks.

I suggest using Kullback–Leibler divergence (KLD) to compare the classes. The procedure is as follows:

1- Find the mean vector and covariance matrix of each class of each dataset.

2- Calculate the KLD between each class of the first dataset and all classes of the second dataset. So, for example if you have two dataset and each of them has 5 classes, you should calculate KLD, 25 times.

3- For each class of the first dataset, find the class from the second dataset with the lowest KLD between them. Doing this, you will find the most similar classes of the two datasets.

Note that KLD measures the similarity between two probability distributions and by calculating the mean vector and covariance matrix of each class, you are fitting a normal distribution to it. To find the formula for calculation of KLD between two multivariate normal distribution, refer to here:

https://en.m.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence


What are the model parameters in PCA? I've been asked to report the number of parameters to be learned in a PCA model. This answer implies that parameters do exist in PCA, but does not explain. Software packages often report the number of parameters, but do not document what those parameters are.
What are the parameters in a PCA model? Subsidiary question: how many parameters are there?
To be clear - I am not asking about model hyperparameters.

From an algebra point of view PCA is a base change. You can write the transformation as:
$$ T = XW $$
Where $X$ is an nxp matrix (n instance, p features) and $W$ is a pxp 'weight' matrix (whose columns are eigenvectors of $X^TX$ - translating one base to another). Hence the general PCA has $p^2$ coefficients.
You can also reduce the space to m dimensions and use :
$$ T = XW_m $$
Where $W_m$ is a weight matrix constituted of m first columns of $W$. $W_m$ is a pxm weight matrix and has pxm coefficients (or easier to understand : mxp coefficients, that is number of dimension kept and parameters needed to express the new base vector in terms of the initial base). In the case of a projection onto 2 dimensions you will have 2p coefficients.


Understanding policy gradient theorem - What does it mean to take gradients of reward wrt policy parameters? I am looking for a little clarity on what the policy gradient theorem means. My confusion lies in the fact that the reward $R$ in reinforcement learning is non-differentiable in the policy parameters. As that is the case how does the central objective of policy gradients, finding the gradients of Reward $R$ wrt the parameters of policy function even make sense?

We want to find the gradient of policy "return" $V$ wrt. parameters of the policy $\theta$. Where the return $V$ could be written as "how good is an action $Q$ $\times$ probability of taking that action $\pi$".

Consider the policy gradient, $\nabla_\theta V = \sum_a Q \nabla_\theta \pi + \pi \nabla_\theta Q$

The first term tells us to adjust the action probability proportionally to how good it is. To me it could read "if an action yields good, take more". That is to move the peak of $\pi$ to match the peak of $Q$. This is a reasonable thing to do. But of course since $Q$ cannot directly guide us toward its peak, it is up to our $\pi$ to luckily stumble upon the high peak of $Q$. This emphasizes the importance of exploratory nature of $\pi$.

The second term is vice versa. That is to move the peak of $Q$ to match the peak of $\pi$. This is much harder a task because $Q$ is a function of both action and policy, $Q_{\pi_\theta}(s, a)$. We clearly don't have this in a differentiable form i.e. we don't have a universal $Q$ function over the space of all possible $\pi$.

We now have a partial gradient from the first term but we have yet to estimate the second term.

Turns out, the second term can be recursively written solely in the form of the first term but with subsequent actions and states.

$$
\nabla_\theta V_0 = \sum Q_0 \nabla_\theta \pi_0 + \sum Q_1 \nabla_\theta \pi_1 + \sum Q_2 \nabla_\theta \pi_2 + \dots
$$

That is to get good policy i.e. policy gradient we only need to move the peaks of $\pi$ to match the peaks of $Q$ not only the first (state, action) but also for all subsequent (state, action)'s. This yields the same result as if we differentiate through the $Q$.


A feature highly correlated with target variable What if one of the predictor variables is highly correlated with the target variable (say 0.9), what should we do? Should we drop it or keep it to build the prediction model(classification or regression)?

You should keep it, the higher the correlation with the target variable - the better the feature.
BUT - you should also make sure this correlation is "real", i.e. not due to data leakage.
(the answer was written using @GeoMatt22 and @Ubikuity comments.)


Ignore unseen columns with OneHotEncoder I'm working in a problem in which I'm OneHot encoding a set of feaures from a dataframe, for instance:

from sklearn.preprocessing import OneHotEncoder
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y)
oh = OneHotEncoder(handle_unknown='ignore')

print(X)

      a     b     c
1    one    m     y
2    two    m     n
36   three  f     n
113  one    f     n
31   two    m  other
....

oh.fit(X_train)

However, it could be that not all features are present in the test set. For this example, say I only have the two first columns. The encoder, in this case will raise an error:

oh.transform(X_test.loc[:,:'b'])

  The number of features in X is different to the number of features of the fitted data. The fitted data had 3 features and the X has 2 features.

Is there some way around this? Ideally I'd like to have the missing columns, either ignored (not present in the output), or set to NaN.

It is not possible. And this has nothing to do with OHE, if you choose any other encoding methods, still you are facing the same problem! Not having similar feature sets between train and test contradicts with fundamentals of developing ML models using train/test. This is already discussed in length in a stats.stackexchange question, or here concisely.

My two cents into the matter:

I would first recommend figuring our the nature of missing column in the test data. Do they miss sometimes or they are never present? If they do miss sometimes, how do they look like when they are present? If they are never present in the test, you have no choice than dropping them in the train set. However, if they are present every now and then in the test, and they contain similar content to the train, maybe imputing them is kind of possible, and having a NaN in test when not present in the test is a valid approach to try. Stil I would recommend doing so, as you may add artifacts (information) that may not represent the reality or even distort the reality so to say.
OHE: Please note that OHE is quite sensitive to its members. As it was mentioned in one of the answers and I am sure you are aware too, OHE create new column for each of its members and fill them with 0 or 1. For example, for your column a, you will have a_one, a_two, and a_three columns after OHE. And if even one of these members (one, two, three) are not present in column a in your test set, your pipeline raises an error. OHE is good for cases that you have always a known set of members for each features, otherwise you are better good choosing other categorical encoding methods like Hashing, Binary, or even recently introduced Entity Encoding.

Good luck!


Creating Domain specific Question Answering Systems I want to create a domain specific QA system.

I am working on a project to create an QA system for a given text book (on a specific domain and answer all related questions)

One approach I am considering for this is to create an Ontology/Knowledge base and then use this for answer retrieval.

For this :

How can I extract data from documents and populate this to the KB ?
The question will be in natural language, how can I use this question to query from the knowledge base ?
Is this the best approach ? because domain knowledge is required here for creating ontologies. And there are n number of domains. So shall I need to create different ontologies for each domain ?
Can a single QA model can be used to cover different domains ? Is there any other Machine learning and Deep learning approaches can be used for this ?

This is broad question, but here is the general approach. There are several sub-problems involved in a typical QA system, and each may be solved using different approaches:

Question Classification - What type of question is the user asking? This can be posed as a classification problem, assuming you have labels.
Parsing the question - Multiple NLP techniques would be required here
Converting question to canonical form - Every question would be converted into a canonical or structured format (which can be used to query). This is usually the most difficult part of the process. This is usually referred to as Semantic Parsing. One approach is to get variants of phrases and then map it to a single canonical form using some similarity metric. You can start with this paper.

If you can get from a natural language text to a canonical form, you are essentially done. Storing data and retrieval are trivial aspects which could be done in numerous ways. Coming to the question of domains, it is always going to be a trade-off between accuracy for one domain vs a general QA system (which is way too difficult to crack). A single model for every domain will most likely give you better results at the cost of extra effort.


Class imbalance in one hot encoding for CNN I am building a 2D Convolutional Neural Network for MFCC features for audio classification.

The issue I am facing is that there are 2 classes and huge imbalance between them. One class has 17687 samples while other has 67737 samples. I have done one hot encoding so I have two CNN outputs as [1,0] and [0,1].

From my research it seems that adding class_weights in model.fit only seems to work for  binary classification problems. Is there any way I can assign weights to one hot encoded results?

You might want to consider using metrics such as the f1-score in order to measure how well your model does.

Notice that if, in real life, the inbalance will be of a similar nature, then your model has to account for it anyway.


MLPRegressor Output Range I am using Scikit's MLPRegressor for a timeseries prediction task.

My data is scaled between 0 and 1 using the MinMaxScaler and my model is initialized using the following parameters:

MLPRegressor(solver='lbfgs', hidden_layer_sizes=50,
                           max_iter=10000, shuffle=False, random_state=9876,
                           activation='relu')

I am expecting output between 0 and 1 but getting values outside the bound (both negative values as well as > 1).

Non-normalized data has the same problem, I get predictions out of range!

Any idea where I could be wrong?

UPDATE:
Based on the answers below I played a bit with modifying the output activation layers and got some interesting results that I thought worth sharing. There are three scenarios, hope the captions convey the message clearly:
Legend:

Black solid line = Training Epoch

Red solid line = Test Epoch

Cyan dashed line = Network prediction over the entire data set

Output when the network is trained using 'relu' activation layer but output_activation_ set to 'logistic

Output when the network is trained using 'relu' activation layer and output_activation_ set explicitly to 'relu'

Output when the network is trained using 'relu' activation layer and output_activation is left alone

Altough your data is between 0 and 1, the predictions can be outside of this range, as with any satistical model. This shows that you are not able to represent your data properly with your model. However, this is not the important point. If you want to ensure the outputs to be between 0 and 1, you have to change the relu activation for another one, such as sigmoid or softmax. These activations ensure that your outputs will be between 0 and 1, although they might lead to other problems, such as vanishing gradients. What is usually advised is to use relu layers until the last one, which has to be a sigmoid or a softmax.