Confidence intervals for binary classification probabilities When evaluating a trained binary classification model we often evaluate the misclassification rates, precision-recall, and AUC.

However, one useful feature of classification algorithms are the probability estimates they give, which support the label predictions made by the model.

These probabilities can be useful for a variety of reasons depending on the use case. When using these probabilities it would be useful to have a confidence interval rather than a single point estimate.

So, how can we estimate a probability confidence interval given that the misclassification error may not always serve as a proxy for the error between the estimated probability and the actual probability (which is often unknown)?

I've considered using brier score but I'm sure there is a better way. Can anyone point me in the right direction or offer your own insight?

For example, If I have classes [C0, C1] and my probabilities for a given  instance $(x^{(i)}, y^{(i)})$
 are {C0: 80, C1:20} then I will classify this instance as C0. Let's suppose that C0 is the correct class label, at this point the model has done it's job and made the correct classification.

I want to go another step further and use the probabilities {C0:80, C1:20} which could be useful for a variety of reasons.

Let's say C0 and C1 respectively represent a customer keeping and closing their account with a bank.

If we wanted to create an expected value $EV$ of dollars at risk of leaving the bank we could calculate $EV$ as $P(C1) * account \space balance$. This would give us a point estimate of $EV$, which is fine but may not tell the whole story given our uncertainty in the model.

So, how can we provide a lower limit and upper limit of the probability this instance is of class C1 with 95% confidence?

I don't think there is a good way to do this for all models, however for a lot of models it's possible to get a sense of uncertainty (this is the keyword you are looking for) in your predictions. I'll list a few:

Bayesian logistic regression gives a probability distribution over probabilities. MCMC can sample weights from your logistic regression (or more sophisticated) model which in turn predict different probabilities. If the variance in these probabilities are high you are less certain about the predictions, you could empirically take the 5% quantile or something.

With neural networks you could train them with dropout (not a bad idea in general) and then instead of testing without the dropout, you do multiple forward passes per prediction and this way you sample from different models. If the variance is high, again you are more uncertain. Variational Inference is another way to sample networks and sample from these different networks to get a measure of uncertainty.

I don't know from the top of my head but I'm sure you could do something with random forests with the variance between the different end nodes where your features end up, assuming they are not deep, but this is just something I thought of.


What are the benefits and tradeoffs of a 1D conv vs a multi-input seq2seq LSTM model? I have 6 sequences, s1,..,s6. Using all sequences I want to predict a binary vector q = [0,0,0,1,1,1,0,0,0,1,1,1,...], which is a mask of the activity of the 6 sequences.

I have looked at seq2seq lstm models, but am struggling with the multiple-sequence-input and single-sequence-output architecture. Am I headed down the right path, or should I shift my focus to a convnet with 6 non-spatial dimensions, and 1 spatial dimension?

Thanks,

I suggest you transform your 6 sequences into 1 since they have the same lengths. You will have 1 sequence with more features for each timestamp.

It is hard to tell you which method would be better. But a LSTM model seems to be well suited for you.


How does sklearn KNeighborsClassifier compute class probabilites? The KNeighborsClassifier has a method for predicting class probabilities. However, I cannot find any documentation describing how these probabilities are computed.

Here is a toy example that returns class probabilites:

from sklearn.neighbors import KNeighborsClassifier
import numpy as np
N = 100
np.random.seed(1)
X = np.random.random((N,2))
Y = np.random.randint(2, size=N)
model = KNeighborsClassifier(5, weights='distance')
model.fit(X, Y)
print(model.predict_proba([[0.5, 0.5]]))
#>>> [[0.55486525 0.44513475]]

The class probabilities are the normalized weighted average of indicators for the k-nearest classes, weighted by the inverse distance.

For example: Say we have 6 classes, and the 5 nearest examples to our test input have class labels 'F', 'B', 'D', 'A', and 'B', with distances 2, 3, 4, 5, and 6, respectively.

Then the unnormalized class probabilities can by computed by:

(1/2) * [0, 0, 0, 0, 0, 1] + (1/3) * [0, 1, 0, 0, 0, 0] +
(1/4) * [0, 0, 0, 1, 0, 0] + (1/5) * [1, 0, 0, 0, 0, 0] +
(1/6) * [0, 1, 0, 0, 0, 0] =

[1/5 ,1/2, 0, 1/4, 0, 1/2]

And after normalizing we get:

[0.13793103, 0.34482758, 0.0, 0.17241379, 0.0, 0.34482758]

If you choose weights='uniform' then it becomes a simple average, which can be computed by multiplying each indicator by (1/k) and summing, with no need for additional normalization.


Do you have to normalize data when building decision trees using R? So, our data set this week has 14 attributes and each column has very different values. One column has values below 1 while another column has values that go from three to four whole digits.

We learned normalization last week and it seems like you're supposed to normalize data when they have very different values. For decision trees, is the case the same?

I'm not sure about this but would normalization affect the resulting decision tree from the same data set? It doesn't seem like it should but...

Most common types of decision trees you encounter are not affected by any monotonic transformation. So, as long as you preserve orde, the decision trees are the same (obviously by the same tree here I understand the same decision structure, not the same values for each test in each node of the tree).

The reason why it happens is because how usual impurity functions works. In order to find the best split it searches on each dimension (attribute) a split point which is basically an if clause which groups target values corresponding to instances which has test value less than split value, and on the right the values greater than equal. This happens for numerical attributes (which I think is your case because I do not know how to normalize a nominal attribute). Now you might note that the criteria is less than or greater than. Which means that the real information from the attributes in order to find the split (and the whole tree) is only the order of the values. Which means that as long as you transform your attributes in such a way that the original ordering is reserved, you will get the same tree.

Not all models are insensitive to such kind of transformation. For example linear regression models give the same results if you multiply an attribute with something different than zero. You will get different regression coefficients, but the predicted value will be the same. This is not the case when you take a log of that transformation. So for linear regression, for example, normalizing is useless since it will provide the same result.

However this is not the case with a penalized linear regression, like ridge regression. In penalized linear regressions a constraint is applied to coefficients. The idea is that the constraint is applied to the sum  of a function of coefficients. Now if you inflate an attribute, the coefficient will be deflated, which means that in the end the penalization for that coefficient it will be artificially modified. In such kind of situation, you normalize attributes in order that each coefficient to be constraint 'fairly'.

Hope it helps


What is the best way to deal with imbalanced data for XGBoost? There are a lot of way to deal with class-imbalanced data like undersampling, oversampling, changing cost function etc.

https://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/

Here is the post talking about them all.

I am working with an imbalanced multiclass classification problem and trying to solve it using XGBoost algorithm. I wanted to understand which method works best here. Since XGBoost already has a parameter called weights (which gives weight to each train record), would it be wise to directly use it instead of undersampling, oversampling, writing a cost function etc.?

I think using something like this could help in your case.

https://github.com/dmlc/xgboost/blob/master/demo/kaggle-higgs/higgs-train.R#L17
http://xgboost.readthedocs.io/en/latest/how_to/param_tuning.html

Hope this helps at least a bit!


Establishing data science programs as an independent discipline What are the benefits of establishing data science programs as an independent discipline rather than creating data scientists in different fields? (for students, universities and the society as a whole)

I am authoring a book on data science and I have mentioned some university programs as models for establishing data science programs in bachelor's, master's and doctorate level. I am thinking of some reasons to compare this model with enriching well-established programs and adding data science skills.

Advantages of setting up a dedicated data science as an independent discipline, which I can think of:

There would be more and more data scientists who can cater to the growing demand for them, in companies and industries in almost every domain.
The concept of statistics, algorithms (ML would be included in this), and Big Data don't need to exist as chapters in separate domains. Now, they can exist as part of the data science workflow in this program.
Data Scientists would be more aware how data science would be applicable in the domain of business. The concepts of business knowledge for data scientists is often found wanting. Data scientists with a knowledge of Business Statistics is in huge demand, and a domain dedicated program would help serving the needs.


Replacing missing values with mean of feature calculated from previously replaced values I am not sure how to ask this, but I will try my best.

I have replaced some missing values in a feature with the mean of the feature conditional on a second categorical feature. However, not all missing values were replaced because some categories of the second feature did not have any values in the first feature, so its mean cannot be calculated. Then, I tried again by replacing the leftover missing values with mean conditional on a third feature. The third feature also contains values that were just replaced with the mean of the first feature. So, is it alright to replace missing values with the mean calculated from values that include some that have once replaced other missing values?

So, is it alright to replace missing values with the mean calculated from values that include some that have once replaced other missing values?

There's no law against it :)

But by replacing missing values everywhere in many different ways you're probably damaging the reliability of your dataset at some point, and that could in turn mess up your experiment.

You don't give any detail about the task or the data so I assume you chose this approach on purpose, but just in case let me remind you of other options:

you could discard features for which most values are missing, unless they happen to be very important for the task
you could discard instances which have many/some missing values, especially if you have a very large dataset anyway
you could use a ML method able to deal with missing values by itself


Predict the next player's option for betting based on playing history? I have a project which is to predict in next time, which option player will select for betting. This prediction based on his history. For example, the player have time series data, the last options they selected, how much money they spend, did they loose or win...ect. There are many options for one bet, around 300. Based on that data, I want to predict next time which option (or possibility of options) they will choose for betting.

I don't know which kind of problem it is. Is it kind of product recommendation, multi-class classification or multi-label classification? Any keyword, or suggestion to solve this kind of problem? Thanks.

this is a multi class classification problem.

There are many approaches you can use to solve this problem.

You can first try Gradient Boosted Trees with libraries like xgboost or lightgbm. Both of those can be used in R or Python.

You can also use neural networks for multi class classification problems.

But there are many ways to solve a multi classification problem.

You can turn it into a multi binary classification problem. Since you have 300 options, you can train 300 binary classification algorithms on your data.

You can also try K nearest neighbours.


Always getting value one for a binary classifier I'm using keras. I have one classification problem. The output should be either 0 or 1. I trained my model and I'm getting 86.59 accuracy. But when i check the predicted output what I'm seeing is all ones. I tried creating a categorical classifier with two nodes and tried the same. The test accuracy is 86.59% but when I check the output the prediction contains only one node with value one for the entire dataset.

This is the code

from keras.models import Sequential

model = Sequential()
from keras.utils import to_categorical
y_traine = to_categorical(y_train)
y_teste = to_categorical(y_test)

from keras.layers import Dense
x_train = x_train.reshape(1300,64)
model.add(Dense(units=64, activation='relu', input_dim=64))
model.add(Dense(units=2, activation='softmax'))
model.compile(loss='categorical_crossentropy',
              optimizer='sgd',
              metrics=['accuracy'])
# x_train and y_train are Numpy arrays --just like in the Scikit-Learn API.
model.fit(x_train, y_traine, epochs=50, batch_size=32)
x_test = x_test.reshape(len(x_test),64)
loss_and_metrics = model.evaluate(x_test, y_teste, batch_size=128)
classes = model.predict(x_test, batch_size=128)
print (loss_and_metrics)
print (classes)

output

[0.32096896952051834, 0.875968992248062]

[[0.84422934 0.1557707 ]
 [0.8332991  0.16670085]
 [0.86778754 0.13221247]
 [0.9261704  0.07382962]
 [0.85143256 0.14856751]
 .
 .

What I'm doing wrong here? Why I'm getting training accuracy as 86% if my predictions are wrong?

Did you check how many samples are there for each class. I suspect imbalance class problem here. If you have majority of images with class=1. Classifier will be biased towards predicting everything as class 1. Try to balance your classes by either duplicating minor class images or deleting major class images.


Support Vector Machines (SVM) vs Minimum distance between two convex hulls Imagine that we have two sets of seperable points, X and Y, in the plane (R^2) that we want to classify. To find the optimal line (hyperplane) that separates these two sets of points we can:

Run a SVM algorithm
Find the minimum distance line segment between the convex hulls of X and Y and take the line perpendicular to the midpoint of this line.

2 can be done using an algorithm of H. Edelsbrunner in O(logn+lognm) time. But I guess creating the convex hulls of the point sets would be O(nlogn), so overall it should take O(nlogn) time. I do not know the time complexity of the SVM algorithm.
Do both algorithms achieve the same result? If not, why are they different? If yes, why would we prefer the SVM algorithm over 2?

You have to add a constraint of linear separability to make the comparison. The problem is that in reality most problem are not linearly separable. Either they need non-linear transformations (data processing or usage of kernel), or they are not separable at all (underlying randomness, noise). SVM can be adapted to these non-linearities quite well (allowing for missclasification, usage of kernels), while the second algo would need significant rework. Basically, as is, the second algo wouldn't work for most real life datasets.


Custom connections between layers Keras I would like to manually define connections in neural network between layers using TensorFlow and keras with Python. By default connections are beween all pairs of neurons. I need to make connections as in picture below.

If i understand you correctly,To join two outputds layers into nextn layer input, use the following design:

out1 = keras.layers.YourLayer1(**params**)
 out2 = keras.layers.YourLayer2(**params**)
 NextLayer = keras.layers.YourLayer3(**params**)[out1, out2]


Where can I find datasets with labeled duplicate text documents? I'm working on detecting duplicate text documents using a classifier. I am looking for training data - a corpus of text documents and corresponding metadata which lists out pairs of duplicate documents from the corpus.
Where can I find such data?

Instead of training a classifier, detecting duplicates may advantageously be done in a direct fashion: using a similarity metric like shingle signatures or a hashing/checksum function.

(A text to representation function F should preserve formal/meta elements that you already know to be identifiers or partial-identifiers, and it should also incorporate a compact representation of the content -- like sparse excerpts, checksum, or any other hashing key or canonized form. The comparison function G should then compare these text representations provided by F and define a similarity score.)

You'd still need an evaluation corpus, so looking for a corpus is still important. I don't know of any open one, but job-boards and academic databases contain lots of duplicates so you could start there.


Am I supposed to be using Mixed Effects? I ran a GLS random effects regression on some NBA data in Stata, and I was told that it was wrong because I didn't use mixed effects model. This may every well be the case, but I was quite confused by the explanation. Here is what I did using Stata:

Dependent variable is team win's and independent variables is different types of opponent shooting data.
Paneled data using xtset to account for the different NBA seasons
Ran robust xtreg for both fixed and random effects
Hausman test had a high chi^2, so I stayed with the GLS random effects regression.

If this is incorrect, please let me know so I can fixed this. I always thought that the mixed effects model was for logistic regressions when my dependent variable was categorical. I was told that I did not take into account this team grouping structure within your analysis, but the Group Variable was the teams.

I always thought that the mixed effects model was for logistic regressions when my dependent variable was categorical

This is not true at all. We use mixed effects, in particular random intercepts when we have clustered data, such as repeated measures on individuals, or sampling within clusters such as schools, or regions, or countries. It is certainly not the only approach - instead we can use Generalised Esitmating Equations, or just fit fixed effects for the grouping variable, but when there are a large number of groups, and we can justifiably consider their "effect" to be normally distributed and uncorrelated with the residual error or the fixed effects, a mixed effects model is often very attractive.


Implementation of the paper 'Perceptual Generative Adversarial Nets for small object detection' I studied the research paper on Perceptual Generative Adversarial Nets for small object detection.

There they have detailed the structure of Generator network as given in the picture below:

I am new in the field of GAN. I am having problem in designing the generator function. Can anyone help me out?

The generator function is just a CNN that maps an image to a feature map here (well, ROI pooled anyway).
There is nothing specific to it being a GAN; what makes something a GAN is its use as a density estimator and how it is trained (i.e., adversarially), not its architecture. So don't worry about the lack of familiarity with GANs.

In any case, the complex part here, in my opinion, is the presence of the object detection framework parts. To better understand it, though you probably have done so already, I suggest reading the "RCNN trilogy papers": RCNN, Fast RCNN, and Faster RCNN, or at least the middle one, which is relevant here. See also this post and this one about ROI pooling, which takes the dis-similarly sized ROIs and maps each of them to a small fixed-size feature map.

The rest of the architecture are standard convolutional layers mostly, with which you are presumably familiar. It might be useful to review ResNet though, since ResBlocks are used here.
The core ("backbone") of the pipeline relies on the VGG network; its use is explained in more detail in references 24 and 41 of the paper.

Anyway, here's what's happening. Given an image $I$ with proposals $P$ (where $n=|P|$), we first run conv1 on $I$, giving us the low-level feature map $L$. Two things then happen:

On the "top path", $L$ is run through convi for i=2:5, resulting in a new feature map $L_T$, which is then ROI pooled via $P$. This gives us a stack of proposals $S_1,\ldots,S_{n}$, each of which is a featurized image.
On the "bottom path", $L$ is run through two small conv layers and then ROI pooled as well, giving a similar proposal stack $f_1,\ldots,f_{n}$. These are then processed by $B$ standard ResBlocks, to give us $n$ residuals $r_1,\ldots,r_n$.

Finally, we get the ultimate output of the generator, which is a stack of super-resolved proposals given by $$ Y = (y_1,\ldots,y_n) = ( S_1 \oplus r_1,\ldots, S_n \oplus r_n ) $$
where $\oplus$ here just means element-wise sum (i.e., S_k + r_k in most array-based languages).

Basically, the problem is that the ROI pooled features will "look nice" for big objects but terrible for small ones, due to the fewer number of pixels in the latter. The fix for this here is to do "standard" object detection in the top path, but learn a "residue" $r_j$ in the bottom path for each ROI, so that when this residue is added to a small proposal $S_j$, it is given the details normally only present in large proposals. In other words, we are basically doing super-resolution on the small ROIs to help the detector on small objects.

As the paper authors write in the caption:

  The generator is a deep residual network which takes
  the features with fine-grained details from lower-level layer as input and passes them to 3 × 3 convolutional filters followed by 1 × 1
  convolutional filters to increase the feature dimension to be aligned with that of “Conv5”. Then B residual blocks each of which consists
  of convolutional layers followed by batch normalization and ReLU activation are employed to learn the residual representation, which is
  used to enhance the pooled features from “Conv5” for small objects to super-resolved representation through element-wise sum operation.

As well as in the main paper:

  As shown in Figure 3, the generator takes the feature
  from the bottom convolutional layer as the input that preserves many low-level details and is informative for feature
  super-resolution. The resulting feature is first passed into
  the 3 × 3 convolution filters followed by the 1 × 1 convolution filters to increase the feature dimension to be the
  same as that of “Conv5”. Then, B residual blocks with the
  identical layout consisting of two 3×3 convolutional filters
  followed by batch-normalization layer and ReLU activation
  layer are introduced to learn the residual representation between the large and the small objects, as a generative model.
  The learned residual representation is then used to enhance
  the feature pooled from “Conv5” for the small object proposal through RoI pooling [11] by element-wise sum operation, producing super-resolved representation.


After the training phase, is it better to run neural networks on a GPU or CPU? My understanding is that GPUs are more efficient for running neural nets, but someone recently suggested GPUs are only needed for the training phase and that once trained, it's actually more efficient to run them on CPUs.

Is this true?

That would depend entirely on what software platform you're running on, not on any particular characteristic of neural nets or their constituent objects. For example, Theano leverages GPUs because they're optimized for matrix math, which neural net weights and activations are often expressed as. On the other hand, they can also be expressed as sets, which makes them ideal for set-based languages like SQL; I prefer this approach, since sets are more flexible (you can, for example, operate easily on ragged dimensional sets, unlike with matrices) and SQL is a somewhat portable standard that makes it easier to think about neural net layers conceptually. As my screen name suggests, I use SQL Server, which is one of many database server platforms that can do some serious number-crunching on very large neural nets, all without the use of any GPU power. Long ago I also programmed neural nets in VB.Net, VB 6 and C# that efficiently made use of arrays, without leveraging GPUs at all.

I think the choice may come down to a combination of your existing skill set, what architectures you have access to, your portability requirements, code maintenance issues and all that, not to any explicit linkage between specific hardware components and the inner workings of neural nets. There may be slight advantages to modeling ragged matrices in SQL or to the matrix-crunching power of GPUs, but I don't think there's much practical difference in most scenarios, nor can I think of a way of testing such a comparison without getting into an apples vs. oranges conundrum. That being said, within a specific platform, it may be true that "GPUs are only needed for the training phase" and "once trained, it's actually more efficient to run them on CPUs." That I can't comment on, since I still with SQL and the .Net stack and avoid Theano etc. altogether (though I respect their capabilities). I hope that helps.


Is there a mistake in Lecture 5 of Stanford CS234 available on youtube? https://www.youtube.com/watch?v=buptHUzDKcE&list=PLoROMvodv4rOSOPzutgyCTapiGlY2Nd8u&index=5
At 53:45 Professor starts to describe temporal difference for linear value function approximation. At 56:20 on slide one can see how weights are updated. Is equation for $ \Delta w $ correct?
In my opinion thing in brackets should be multiplied by $ X(s) - \gamma X(s') $ instead of $ X(s) $ because $ \frac {\partial ( \gamma X(s')^T w )} {\partial w} $ is not zero.
Am i right?

It's the notation that might be a bit confusing. Take a look at David Silver slides: pages 10-15. He has a complete derivation. Do not forget that the term $r + \gamma V(s';w)$ is the target. She mentions that in the video and the fact that you are actually doing supervised learning with target provided by a bootstrapped value (you try to minimize the Bellman error).
In other words: you are trying to minimize the error between a target value and the estimation from your model. You do not know the target value so you estimate it with bootstrapping. Then you have $SE=(y - \hat{y}(w))^2$. The target y is considered known (as in supervised learning) so eventually you are searching for the weights $w$ that will make the output $\hat{y}$ of your model close to y. What I wrote here applies for batch training (minimizing the mean square error MSE).
So eventually yes the quantity $V(s';w)$ is considered constant and will have derivative 0.


Replace data in Pandas dataframe based on condition by locating index and replacing by the column's mode Hopefully, you don't mind me posting a question here instead of the regular stack exchange forum. I'm learning ML basics and practising pandas.

assuming the data frame is called df, column name = column name

Suppose I want to replace some 'dirty' values in the column 'column name'. There are "not known" values in this column that mean nothing so i would like to replace them with the mode.

df['columnname'].mode()

returns

0   dog
dtype: object

this code below replaces the "not known" values as NaN rather than the mode.

df.loc[df.index[df['columnname'] == "not known"].tolist(),'columnname'] = df['columnname'].mode()

what am i missing? or how should i fix the code?

Your entire code is correct except at the last point where you are equating with df['columnname'].mode(). The value here should have a dtype int or string but this has a dtype object. Just replace it with df['columnname'].mode().values and you are good to go.

Also, I see a lot of stuff that is not required here. Since you are using pd.loc, you can simply write:

df.loc[df['columnname'] == "not known",'columnname'] = df['columnname'].mode().values


Quick way to visually explore data? I have a very large csv file which Apple Numbers won't open. I can open it with TextEdit, but each row is so long it forms multiple lines in the document and makes the document difficult to understand. Is there a tool for opening a csv and exploring different rows and columns?

When you hit the limits of an application like Apple Numbers, or Excel, you need to start using a programming language like Python, R or C. Using a programming language, you write your own application, which is not constrained by arbitrary limits like 220 rows by 214 columns. (Of course, you are still constrained by the physical memory and the way that a particular language addresses it: R and Python tend to be more resource-intensive than C).

Let me give you an example using Python. You can install Python for your system from the official site. If you install one of the latest versions, you will already get pip as an installer of further Python packages. Install Pandas by starting a terminal (on OS X, there is a 'Terminal' application, on Windows this used to be a MS-DOS window ...), and typing into that terminal:

pip install pandas

You can then start Python in an interactive session by typing python in a terminal. In the Python shell, you can read the data as mentioned in @tasos 's answer:

>>> import pandas as pd
>>> df = pd.read_csv('name_of_your_file.csv')

Here you can use the describe method of the df DataFrame to get some statistics about the read data. These numbers are for a randomly generated data frame:

>>> df.describe()
          0         1         2
count  3.000000  3.000000  3.000000
mean  -0.150869  0.444380 -0.117066
std    0.751421  0.697880  0.565328
min   -1.017424 -0.356764 -0.506030
25%   -0.386514  0.206466 -0.441313
50%    0.244396  0.769696 -0.376595
75%    0.282409  0.844952  0.077416
max    0.320422  0.920208  0.531427

Then you can generate plots for different columns, one against the other, using the Pandas plotting functions. Check the many examples in the linked documentation. Here is a small example from there:

import matplotlib.pyplot as plt
import pandas as pd
import bumpy as np

df = pd.DataFrame({'a': np.random.randn(1000) + 1, 'b': np.random.randn(1000),
                   'c': np.random.randn(1000) - 1}, columns=['a', 'b', 'c'])
plt.figure()
df.plot(kind='hist', alpha=0.5)

Usually, you will need to use another call more in order to save it to a file:

plt.save('output.png')  ## or *.svg, *.pdf

In the file you should find something similar to this figure.


How do I forecast sales data down to the individual item? I have a dataset that looks like,

order datetime, customer id, product name, type of product, quantity sold

I want a model to forecast sales for each individual item.  I'm thinking of using one of the following but would like some advice, literature, or other options.

One time series forecast (ARIMA) for each product
RNN
HMM

What's the best method to predict sales of each individual item into the future?

Do you want to forecast sales by day over a series of days? That would be more like ARIMA. Do you want to forecast "How many widgets will we sell in the next month"? That would be more like a regression problem.

As an aside, if the retailer you're working with has a large assortment of products that changes over time (e.g. seasonal clothing) then you should also look into forecasting by product category and not an individual product. It will be much more accurate.


Hidden Markov Model: Forward Algorithm implementation in Python I am learning Hidden Markov Model and its implementation for Stock Price Prediction. I am trying to implement the Forward Algorithm according to this paper.

Here I found an implementation of the Forward Algorithm in Python.

import pandas as pd
import numpy as np

V = np.array([0, 1, 1, 2, 0, 1, 2, 0, 1, 0, 2])

# Transition Probabilities
a = np.array(((0.54, 0.46), (0.49, 0.51)))

# Emission Probabilities
b = np.array(((0.16, 0.26, 0.58), (0.25, 0.28, 0.47)))

# # Equal Probabilities for the initial distribution
pi = np.array((0.5, 0.5))

def forward(V, a, b, pi):
    alpha = np.zeros((V.shape[0], a.shape[0]))
    alpha[0, :] = initial_distribution * b[:, V[0]]

    for t in range(1, V.shape[0]):
        for j in range(a.shape[0]):
            alpha[t, j] = alpha[t - 1].dot(a[:, j]) * b[j, V[t]]

    return alpha

alpha = forward(V, a, b, pi)

But it seems to me that it does not include (c) and (d) steps from the algorithm. So I added them:

def forward(V, a, b, pi):
    p = 1
    alpha = np.zeros((V.shape[0], a.shape[0]))
    alpha[0, :] = pi * b[:, V[0]]

    for t in range(1, V.shape[0]):
        probability_of_observation = 0 #my code
        for j in range(a.shape[0]):
            alpha[t, j] = alpha[t - 1].dot(a[:, j]) * b[j, V[t]]
            probability_of_observation += alpha[t, j]  #my code
        p = p * probability_of_observation #my code

    return p #changed

p = forward(V, a, b, pi)  #changed

Does my code coincide with the given algorithm?

Maybe this python library could help you: hmmlearn

When I tried to build an hmm I used it and it worked well.


How to subset rows from a data frame with comparison operators in R I have a data frame (a csv file) with dimensions 100x6 and I need only the columns c("X1", "X2", "X4") and the rows in which the value of "X1" is greater than 30. So I did:

  data_frame <- read.csv ("data_frame")
  data_frame [c("X1", "X2", "X4")]

The column subset problem is solved but now I need to subset rows from data_frame [c("X1", "X2", "X4")] where the values of "X1" is greater than 30. I tried:

  data_frame [c("X1" > 30), c("X1", "X2", "X4")]

But it returned the same data frame as data_frame [c("X1", "X2", "X4")].
Also tried using the function subset() with the same approach but got the same results.

You want

data_frame[data_frame$X1 > 30, c("X1","X2","X4")]

that will just print it, you probably want to update data_frame or store it in something else:

data_frame = data_frame[data_frame$X1 > 30, c("X1","X2","X4")]

also you probably want to try asking this on StackOverflow, or reading a bit more basic R documentation because it should be well covered. Its a bit simple to be "data science".


can't get variables from TF collection Aurelion Geron's example shows how to stored selected operations in a collection, so they can then be easily accessed later:

Adding:

for op in (X, y, accuracy, training_op):
    tf.add_to_collection("my_important_ops", op)

Then retrieving:

X, y, accuracy, training_op = tf.get_collection("my_important_ops")

However when I try to do this I get this error:

X, y, accuracy, training_op = tf.get_collection("my_important_ops")
ValueError: too many values to unpack (expected 4)

What am I doing wrong?

A simple solution is to do the following:

temp = tf.get_collection("my_important_ops")
print(temp.get_shape().as_list())

Then you can evaluate the outputs.


Feature Extraction - calculate slope Having a bit of a mind-blank at the moment and am looking for some advice.

I am extracting features from time series data for input into a classification algorithm, for example I'm extracting average and variance from inputX.

For input Y,  I have graphed the data and have seen that for class A, it can be seen that there is an upwards slope, and for class B, it can be seen that there is a downward slope, for class C there is no slope, the line is more or less straight.
For Feature Extraction, how can I best describe this? Would a calculation to get positive/negative slope be best?

There are several ways to do this, here are a couple of options:

Calculate different lag values (difference between now and t time units)
Calculate a linear regression for different time windows and store the slope and the bias
You can also involve higher order models to describe what is happening, if you think for example that the acceleration also matters you could use a 2nd or 3rd degree polynomial over the past couple of observations.


How to draw neural network diagrams with this particular style? I would like to draw a neural network architecture with the follow style. Do you know which tool can be used to do this? The paper is Operation-aware Neural Networks for User Response Prediction.

I asked me something similar as well as I thought that a lot of papers use rather high quality images but it seems that all the authors generate them individually: https://graphicdesign.stackexchange.com/questions/114886/software-to-draw-schemes-quickly
For NNs, there is already an answered question: How to draw Deep learning network architecture diagrams?
Have a look at https://www.quora.com/What-tools-are-good-for-drawing-neural-network-architecture-diagrams and https://softwarerecs.stackexchange.com/questions/47841/drawing-neural-networks


Find points on a map close to given points I have a locus L of points (lat, long). And I would like to find N=10 points (let's call them warehouses) such that:

$$loss = \sum_{l \in L} maximum_{w \in W}(distance(l, w))^2 $$

is minimized.

Is there a documented algorithm or approach that solves this problem? Right now I am thinking Excel may be able to handle this task. However I have too much data for Excel and will need to implement this in Python / Pandas.

I can tell you how I would do it, but there is almost certainly a faster implementation.

Assuming you start with, for each point in $L$, the distances to each warehouse, $w \in W$. These distances should be calculated by the haversine formula.

You can find the distance to the $N$th closest point in $w$ by using the quickselect algorithm. This is very similar to the quicksort algorithm but only sorting the parts that you care about. The average case for quickselect is $O(N)$ but you'll need to repeat for each $l \in L$.

Note that, since the square is monotonic for positive distances, you only need to minimise

$$\sum_{l \in L} maximum_{w \in W}(distance(l, w)) $$

I found a handy implementation of the quickselect algorithm on KoderDojo


Transfer learning by concatenating the last classification layer Before going into an obvious XY problem, I will explain you what I'm trying to do.

I'm training a simple MobileNet pre-trained with Imagenet for multiclass classification. What I do is freeze all the convolutional part, and then create a new Prediction Layer (conv2D 1x1xN where N is the number of classes).

Let's say I trained with 5 classes, and then I got a new set of images of 5 new classes. I want to keep the knowledge from the last training and be able to train only with the new classes.

What I do is to train a new model with the same frozen pre-trained weights, but only with the new 5 classes, then concatenate the weights from the old and the new model into a last layer that outputs for 10 classes.

The concatenation works perfectly, but when I run the evaluation I got a horrible accuracy (like if it was predicted randomly). For example, for 30 clasess trained in "sets" of 5, I got an accuracy of 0.03.

Is what I'm doing viable? A problem that I think it can be causing this is that the "labels" for every prediction are not kept in the same order or something, so that even if I copy the weights, the predictions may be unordered.

It turns out that it was an issue related with the order of classes being loaded in the model.

Let's assume I have the following structure

root
  -train1
    +a
    +r
  -train2
    +b
    +y
  -all
    +a
    +r
    +b
    +y

Where a, r, b and y are classes (folders with images). I first train with the ones at train1, so the network assigns each output to a class. Then I train a second network (everything but last prediction layer frozen) with the second training folder train2.

The order of the outputs of the prediction layer will change depending on the algorithm, but assuming it's loading the paths in a sorted way:

In the first network, the first output will be for a and the second for r.

In the second network, the first output will be for b and the second one for y, and when concatenated to the first network, it will be a, r, b, y

When I load the whole (merged) model, and I load a folder will all classes inside (all), the network assumes the outputs are for a, b, r, y in this order, whilst the concatenated network had a, r, b, y as the output order.

In this example, the outputs for a and y will be in the same position, while r and b will be reversed, which will lead to a bad accuracy.

TL;DR:

The order in which classes are loaded while you do the "incremental learning" needs to be the same always, so be careful in which order your classes are being loaded.


How to resolve too many indices for array Index Error I'm performing a binary classification in Keras and attempting to plot the ROC curves. When I tried to compute the fpr and tpr metrics, I get the "too many indices for array" error. Here is my code:

#declare the number of classes
num_classes=2
#predicted labels
y_pred = model.predict_generator(test_generator, nb_test_samples/batch_size, workers=1)
#true labels
Y_test=test_generator.classes
#print the predicted and true labels
print(y_pred)
print(Y_test)
'''y_pred float32 (624,2) array([[9.99e-01  2.59e-04],
                                 [9.97e-01  2.91e-03],...'''

'''Y_test int32 (624,) array([0,0,0,...,1,1,1],dtype=int32)'''

#reshape the predicted labels and convert type
y_pred = y_pred.argmax(axis=-1)
y_pred = y_pred.astype('int32')

#plot ROC curve
fpr = dict()
tpr = dict()
roc_auc = dict()
for i in range(num_classes):
    fpr[i], tpr[i], _ = roc_curve(Y_test[:,i], y_pred[:, i])
    roc_auc[i] = auc(fpr[i], tpr[i])
fig=plt.figure(figsize=(15,10), dpi=100)
ax = fig.add_subplot(1, 1, 1)
# Major ticks every 0.05, minor ticks every 0.05
major_ticks = np.arange(0.0, 1.0, 0.05)
minor_ticks = np.arange(0.0, 1.0, 0.05)
ax.set_xticks(major_ticks)
ax.set_xticks(minor_ticks, minor=True)
ax.set_yticks(major_ticks)
ax.set_yticks(minor_ticks, minor=True)
ax.grid(which='both')
lw = 1
plt.plot(fpr[1], tpr[1], color='red',
         lw=lw, label='ROC curve (area = %0.4f)' % roc_auc[1])
plt.plot([0, 1], [0, 1], color='black', lw=lw, linestyle='--')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristics')
plt.legend(loc="lower right")
plt.show()

The shape of y-pred and Y_test are:

y_pred float32 (624,2) array([[9.99e-01 2.59e-04],
                                 [9.97e-01  2.91e-03],...

Y_test int32 (624,) array([0,0,0,...,1,1,1],dtype=int32)

Your code is broken in two places.

The first is because you took the argmax of your class probabilities from y_pred. The line

y_pred = y_pred.argmax(axis=-1)

reshapes your prediction vector into (624,) to match your vector of classes. Thus, when you try to slice your array later with y_pred[:,i] it's going to bark since you no longer have a second dimension. This isn't really the behavior you want either, since the roc_curve function is interested in the exact class probabilities your model produces!

The second is for the same reason, attempting to index the second dimension of  a one dimensional numpy array, but for the Y_test vector.

So if you're interested in capturing TPR/FPR for both classes by treating each as the positive class, you need to drop these lines

#reshape the predicted labels and convert type
y_pred = y_pred.argmax(axis=-1)
y_pred = y_pred.astype('int32')

and you need to change the first line of your for loop to:

fpr[i], tpr[i], _ = roc_curve(Y_test, y_pred[:, i])

hope this helps


Pearson correlation coefficient - is correlation estimator acceptable? As far as I know when it comes to theory, we use Pearson correlation when we want to check the correlation between two variables, which are both continuous or discrete. For a mixed case it's not so easy to use it to compute correlation  coefficient. On the other hand, we have Pearson correlation estimators, where we can calculate mixed case without any problems (based on samples). Does the Pearson correlation coefficient give deceptive results in this case ?

If the discrete variable has a lot of discrete values then it is almost the same as a continuous variable, because continuous variables are technically discrete due to the way how numbers are represented in computers (float64 for Python).

The worst case is binary, but, in my experience, Pearson coefficient work well with binary and continuous data together. I know that asymptotic distributions could lead to biased estimators of linear regression. But Pearson coefficient is the way to calculate, it is not an estimation of something, so I cannot say if it is biased or non-biased. I know that if the relationship is linear then you get a strong Pearson coefficient, and if not then you get a small number.

For continuous data and binary data it means that you need to have a strong threshold. If everything below the threshold means 0 and everything above means 1 for the binary variable, then the correlation is strong. Though if I remember correctly it will still not be 1, because you cannot explain 100% of variance of a continuous variable with one binary variable. But you probably can get a number about 80%. This all depends on where the threshold is, and how the variables are distributed. If you have two distinct clouds of continuous variable that are separated by a big distance, then you will get a number close to 100%. Similar logic could be used for discrete variables that have several values.

Pearson coefficient gives a stronger weight to where the most points are, due to the way it is calculated. So if you do not have points in particular regions then there is no way for the coefficient to reflect the correlation there. In practice it means that you usually have points in a limited range, and you calculate the linear correlation in this range.


Uploading huge dataset I have few questions:

Is there a website to upload huge research dataset (over 100GB) for free?
Which type of compression (rar, zip ... etc) is good for jpeg images?
In case of dataset of 120GB. what is the best split for this big files (eg: 20 GB each)?

Don't compress files that are already compressed (like how JPEG/PNG images or video files are). Their inherent compression is usually good enough, and you only trade some 5% in lower compressed size for a much lengthier and often non-seekable decompression that results in using twice the disk space.

If you need to batch the files together, just use tar.


How to classify/cluster clients of an application of transport like the Uber Model? I have a question about User segmentation
I have a data-set of rides of an application that works at the same model as Uber.
The attributes I have : Reservation it means the id of the Ride and statutCourse means the status of the ride cancelled or finished .....
the Id client and the entreprise is the company where the client works IDChaufeur is conductor ID and i have the geographic coordinations of the start point and the finish point i also have date and hour of the ride also real time and estimated time and estimated distance

I want to classify/cluster clients profiles and conductors profiles

My problem is that one client can make many rides and I don't know if I can use many lines of rides for the same client in one dataset and than use this dataset to classify the clients into classes or profiles ?

If you want to cluster people, you need to have one row per person.
To do this you will need to group by IDClient/IDChauffeur and compute features like meanHourRide, stdHourRide, ...

By feeding the algorithm with multiple records of the same person (with the same travels for example), the clustering algorithm might create a cluster with similar travels (time/location/distance-wise) even though it is made by only one customer.

This would result in travel clustering, which is also possible but not what you want here.


Decreasing accuracy as training progresses means that the learning rate for your model is too high. Your model weights are changing a lot due to the high learning rate and therefore moving away from the local minimum where your accuracy would be at its highest. Try decreasing the learning rate and see what happens.


Am I using optim.SGD incorrectly in pytorch? I am doing reinforcement learning in checkers.

After each game the network beats itself, I calculate the loss of every individual position in the game, call backward(), and step(). I am beginning to believe that this is not how SGD is supposed to be used, and that I should be feeding it batches, say a whole game at a time.

Strictly  in terms of code:

Do I do this by just wrapping the collective inputs in a one dimensional tensor?
If I give SGD a collection of inputs does it sample from the collection, calculate the average loss, and use that gradient?
Is SGD actually intended to be used in this way with large 'collections'/batches?

LT;DR: Executing SGD (Stochastic Gradient Descent) is almost like executing GD (Gradient Descent) but computationally cheaper, and probably the only way to execute GD on big datasets.  By evaluating at each sample - each game - you are losing the advantages of SGD and one could say you're "rolling your own" version of a computationally cheap GD.

Gradient Descent Summary

After you evaluate the error/loss function of you model, with whatever criterion you may use, you will then attempt to take the gradient of your loss function at the point where your network is.  To simply things let's assume that you are using cross-entropy as the criterion of the loss (which makes some sense if you are evaluating an entire matrix of probability results).  Let's say that the loss function is, where $H$ is cross-entropy, $f$ the NN evaluation, $X$ input data and $Y$ the labels:

$$
E = H(f(X), y)
$$

This is what nn.CrossEntropyLoss does (or any other criterion).
We want to find the minima of $E$, for that we can use $\nabla E$ but we cannot get the general formula for $\nabla E$ because we do not have the actual formula for $f$.  Yet, we can find the (vector) value of $\nabla E$ at the location we are in (where location is the value of every parameter/weight in our NN).  We can find $\nabla E$ for one specific configuration of $W$ which will be the matrix of all weights in the network.

$$
\nabla E_{W} = \frac{2}{N} X^T(XW - y)
$$

This is oversimplified - to actually find the matrix $W$ we need to perform backpropagation.  In pytorch we get that by doing a huge chain rule with E.backward().  i.e. this is what backward() does, it gets the matrix $W$.

Now we have a handful of very expensive matrix multiplications to get from $W$ to $\nabla E_W$.  Possibly prohibitively expensive if the dataset is big ($X$ is the full dataset).  Enters SGD.

Stochastic Gradient Descent

The only difference in SGD from GD is that SGD will not use the entire $X$ in the calculation above.  Instead SGD will select just a handful of samples (rows) from $X$ and use that as an estimation of $\nabla E_W$.  Often it is said that SGD takes only a single sample from $X$ but implementations vary (e.g. ASGD).

Finally step() will apply $\nabla E_W$ times the learning rate to all weights/parameters in the network.

Your Approach

Since you are going sample by sample and performing backward() and step() on each sample you are not really doing SGD (despite calling that function).  In your case the SGD optimizer has only a single sample to select from every time, therefore you are uniformly trying all samples in your dataset (as opposite to Stochastically).  (That uniformity will reduce the variance of your model, which may be dangerous in other ways, although not very relevant here)

Therefore yes: To use SGD you do need to feed batches of input, and it will "average" (sum) the  gradients across the sampled rows of $X$.  It will average/sum/accumulate (sum all components) the gradients thanks to the fact that some samples may result in opposite gradient values in some directions; and hopefully the taken samples will agree on the main direction of the gradient and increase the magnitude in that direction.

P.S. The fact that pytorch accumulates the gradients from each selected sample is also one of the reasons we need to call zero_grad() in the training loop.  You want to accumulate gradients within an SGD step but not accumulate across steps (SO question on exactly that).

Code

To actually generate batches you can use torch.randperm() I personally like this answer on SO about randperm but I'll add a summary here:

Having the matrix $X$ above as X (in code), one can create a batch of 256 samples with:

batch_size = 128
n_samples = X.size()[0]
permutation = torch.randperm(n_samples)

for i in range(0, n_samples, batch_size):

    indices = permutation[i:i+batch_size]
    batch_x, batch_y = X[indices], Y[indices]

    # nn(X), criterion(), backward(), step(), zero_grads(), ...

Which you need to perform per epoch (refer to linked answer for that and more - do not want to copy that code in full since I do not deserve the merit).

The tricky bit may be the fact that you need to generate the batch from both $X$ and $Y$, and keep the order between them.  Using indices generated with randperm allows for that.


What are some function/package in R to find similarity of individual words not in the context of sentences? what are some function/package in R/python to find similarity of individual words not in the context of sentences? As a novice, I searched and found methods like adist, cosine similarity, word2vec, but not sure what are the supposed use case for each. Appreciate any advice.

If your intent is to find compare similarity in meaning, word2vec is the only appropriate choice. adist measures the edit distance between two words, and cosine similarity compares the similarity of two documents (treating the documents as bags of words). On the other hand, transforming a word into its word2vec embedding captures some of the latent meaning of the word. See this for a detailed explanation of how this is done.


How to run a python script on GCP Compute Engine? I want to run some machine learning algorithms such as PCA and KNN with a relatively large dataset of images (>2000 rgb images) in order to classify these images.

My source code is the following:

import cv2
import numpy as np
import os
from glob import glob
from sklearn.decomposition import PCA
from sklearn import neighbors
from sklearn import preprocessing

data = []

# Read images from file
for filename in glob('Images/*.jpg'):

    img = cv2.imread(filename)
    height, width = img.shape[:2]
    img = np.array(img)

    # Check that all my images are of the same resolution
    if height == 529 and width == 940:

        # Reshape each image so that it is stored in one line
        img = np.concatenate(img, axis=0)
        img = np.concatenate(img, axis=0)
        data.append(img)

# Normalise data
data = np.array(data)
Norm = preprocessing.Normalizer()
Norm.fit(data)
data = Norm.transform(data)

# PCA model
pca = PCA(0.95)
pca.fit(data)
data = pca.transform(data)

# K-Nearest neighbours
knn = neighbors.NearestNeighbors(n_neighbors=4, algorithm='ball_tree', metric='minkowski').fit(data)
distances, indices = knn.kneighbors(data)

print(indices)

However, my laptop is not sufficient for this task as it needs many hours in order to process more than 700 rgb images. So I need to use the computational resources of an online platform (e.g. like the ones provided by GCP).

Can I simply make a call from Pycharm to Compute Engine API (after a I have created a virtual machine in it) to run my python script?

Or is it possible either to install PyCharm in the virtual machine and run the python script in it or to write my source code in a docker container?

In all, how can I simply run a python script on GCP Compute Engine without wasting time in needless things?

First, you will need to install Cloud SDK: https://cloud.google.com/sdk/downloads#apt-get

Then, the simplest way is to run your script through your Terminal (mac and I presume the instructions also work on Linux):

Configure your project: gcloud config set project insert_your_project_name
Set up SSH keys: gcloud compute config-ssh
Connect to the VM: gcloud beta compute ssh vm_name --internal-ip
Run script: python your_script.py

You can also connect PyCharm directly to GCP and run everything on your VM but you will need PyCharm Pro, otherwise the deployment option is not available. Let me know if this works.

Also, if you want to use the interactive version of setting your project then in step 1 do this instead: gcloud init


Approach to classify spatial trajectories of vessels I'm trying to create a classifier to distinguish different boats by their trajectories. I have training data of the longitude and latitude of a boat and time in seconds.

Vessels like a ferry will have a straight predictable trajectory between two points, whereas fishing vessels can have zig-zag like trajectories for example.

My initial approach is to create features for example the mean speed, standard deviation of the speed, standard deviation on the course, such that each trajectory table is distilled into 1 row of features. Then I can train something like a random forest classifier on these rows.

Is this a good approach, any other suggestions that could account for the characteristic trajectory shapes.

Thanks

Your data contains spatial and temporal data, which means that you can account for location and speed.

Not sure how global the coordinates are, but perhaps grouping them per region might be interesting. It would be like putting all coordinates that you're given and defining zones of interests (zone A, B, C, etc.). Perhaps, some boats only sail in specific regions. Indeed, yachts might be found more often in the Caribbeans than in the Arctic sea. Of course, the number of zones you want to define needs to be fine-tuned.

Assuming that your data accounts for an entire trip from port A to B, you could, of course, use the length of the journey as a feature, as I'm assuming that not all boats travel the same distances. You could also use the overall direction of the trip as the angle between port A and port B.

You've already accounted for speed by computing the mean, standard deviation, and other metrics. What you could also do is fit a polynomial of the speed, and use its coefficients as features. Perhaps certain boats have a more steady speed than other boats.

Following the same thought, you could perhaps fit a polynomial describing the course of each boat and use its coefficients as features. View the course here as the change of angle. This should capture how much "zig zagging" a boat does. Of course, you'll need to fine-tune to pick the right polynomial.

Good luck!


Random Forest Classifier cannot recognise parameter grid I am trying to run the below code to extract the feature importances of my random forest, but I'm getting the following error TypeError: init() got an unexpected keyword argument 'randomforestclassifier__max_depth'. Can anyone tell me what is wrong?
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, GridSearchCV, cross_val_score
from sklearn.pipeline import make_pipeline
from imblearn.over_sampling import RandomOverSampler
from imblearn.under_sampling import RandomUnderSampler
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.metrics import f1_score

x, y = make_classification(n_samples=10000, weights=[0.99], flip_y=0)
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.2, random_state = 1)

paramgrid_rf = {'n_estimators': [500],
                'max_depth': [4],
                'random_state': [0],
                'max_features': ['sqrt']
                }

imba_pipeline_rf = make_pipeline(RandomOverSampler(sampling_strategy=0.35, random_state=0),
                             RandomUnderSampler(sampling_strategy=0.9, random_state=0),
                             RandomForestClassifier())

new_params = {'randomforestclassifier__' + key: paramgrid_rf[key] for key in paramgrid_rf}
cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=3, random_state=0)
grid_imba_rf = GridSearchCV(imba_pipeline_rf, param_grid=new_params, cv=cv, scoring='f1',
                        return_train_score=True)
scores = cross_val_score(imba_pipeline_rf, x_train, y_train, scoring='f1', cv=cv, n_jobs=-1)
grid_imba_rf.fit(x_train, y_train)
y_pred_rf = grid_imba_rf.predict(x_test)
print('F1 score on validation data: ', f1_score(y_test, y_pred_rf))

rf_final = RandomForestClassifier(**grid_imba_rf.best_params_).fit(x_train_res,y_train_res)
rf_final.feature_importances_

Your grid search dictionary contains the argument names with the pipeline step name in front of it, i.e. 'randomforestclassifier__max_depth'. Instead, the RandomForestClassifier has argument names without the pipeline step name, i.e. max_depth. You therefore need to remove the first part of the string which denotes the name of the step in your original pipeline. You can do this using a dictionary comprehension:
# original
{'randomforestclassifier__max_depth': 4, 'randomforestclassifier__max_features': 'sqrt', 'randomforestclassifier__n_estimators': 500, 'randomforestclassifier__random_state': 0}

# splitting the key on '__' and take only the last part
{k.split("__")[-1]: v for k, v in grid_imba_rf.best_params_.items()}
# {'max_depth': 4, 'max_features': 'sqrt', 'n_estimators': 500, 'random_state': 0}

This changes one line in the original script to:
rf_final = RandomForestClassifier(
    **{k.split("__")[-1]: v for k, v in grid_imba_rf.best_params_.items()}
).fit(x_train,y_train)
rf_final.feature_importances_


What is Ground Truth In the context of Machine Learning, I have seen the term Ground Truth used a lot. I have searched a lot and found the following definition in Wikipedia:

In machine learning, the term "ground truth" refers to the accuracy of the training set's classification for supervised learning techniques. This is used in statistical models to prove or disprove research hypotheses. The term "ground truthing" refers to the process of gathering the proper objective (provable) data for this test. Compare with gold standard.
Bayesian spam filtering is a common example of supervised learning. In this system, the algorithm is manually taught the differences between spam and non-spam. This depends on the ground truth of the messages used to train the algorithm – inaccuracies in the ground truth will correlate to inaccuracies in the resulting spam/non-spam verdicts.

The point is that I really can not get what it means. Is that the label used for each data object or the target function which gives a label to each data object, or maybe something else?

The ground truth is what you measured for your target variable for the training and testing examples.

Nearly all the time you can safely treat this the same as the label.

In some cases it is not precisely the same as the label. For instance if you augment your data set, there is a subtle difference between the ground truth (your actual measurements) and how the augmented examples relate to the labels you have assigned. However, this distinction is not usually a problem.

Ground truth can be wrong. It is a measurement, and there can be errors in it. In some ML scenarios it can also be a subjective measurement where it is difficult define an underlying objective truth - e.g.  expert opinion or analysis, which you are hoping to automate. Any ML model you train will be limited by the quality of the ground truth used to train and test it, and that is part of the explanation on the Wikipedia quote. It is also why published articles about ML should include full descriptions of how the data was collected.


Get the Polynomial Equation with Two Variables in Python TL;DR
predict "price", given "length" and "wandRate"

I have some time-series data where the dependent variable is a polynomial result of 2 independent data points.
Here is a snippet:

This is past pricing data of Processed Rice Grains of a certain kind of rice.
Based on the variable "wandRate" (1st variable) which is the price for any "length" (2nd variable) over 8.2, prices of rice grains with lower lengths are calculated.
These prices are based on a long trial-and-error method of asking various experts on their "opinion" on how a certain grain of a certain length should be priced. There are other variables which can't be objectively measured, but length is the main indicator. I was wondering if it would be possible to create an objective model or find a polynomial equation in two variables to predict "price", given "length" and "wandRate"
I was led to thinking in terms of a polynomial when I plotted the data in google sheets and a sixth-degree-polynomial equation gave an intuitively correct looking trendline.

NOTE: I do not have a strong math background so simple google searches about "polynomial in 2 variables from data python equation" did not yield any implementable results.

I'm looking for some python code to accomplish this. ANY guidance on where to look would be appreciated.

So you want to fit 6-th degree polynomial in python to your data?

Main thing you should note is that it will be still linear regression, its juts that predictors are polynomial (most important is that your weights are still linear (betas in lin.regression))

You can transform your features to polynomial using this sklearn module and then use these features in your linear regression model.

> from sklearn.preprocessing import PolynomialFeatures from sklearn
> import linear_model poly = PolynomialFeatures(degree=6)
> poly_variables = poly.fit_transform(variables_length_wand_rate)
> poly_var_train, poly_var_test, res_train, res_test = train_test_split(poly_variables, results,test_size = 0.3, random_state=4)
> regression = linear_model.LinearRegression()
> model = regression.fit(poly_var_train,res_train)
> score = model.score(poly_var_test, res_test)


Interpreting Results of Multivariable Regression / how to transform variables to improve results I am working on a project that predicts the Market Cap (value) of different crypto-currencies. My data is very small (51 observations) and I initially have 18 X-variables. I was hoping to get feedback on my modeling approach and results, and suggestions on improving the model (particularly by transforming variables / with a non-linear regression) technique. I will do my best to keep the post clear and brief, and hope it can be helpful to others working on a similar analysis. The post may seem long, but a lot of it is images. Also, I am doing this in R, and can share any code on request.

1.) My first action was to log-transform the Market Cap (Y-variable) into log(Market Cap). Here are 2 graphs, of Market Cap and of log(Market Cap) of my 50 observations:

...the outlier point, with a $190B market cap, is bitcoin, and the model badly overfit to this data point if I did not log-transform. For this reason, I think this first action of log-transforming the Market Cap is a good action.

2.) 12 of the 50 observations were missing values for 6 of the X-variables. Rather than throwing these observations away, I predict missing values for these 6 X-variables by fitting, for each of these 6 X-variables, a simple linear regression between the (a) the X-variable with missing values, and (b) one of the remaining 12 X-variables with no missing values. For (b), I chose the X-variable with the highest pearson correlation to the missing-value-X-variable. I use this simple linear regression to predict missing values.

3.) I then fit an initial linear model with all 18 variables. However, since 18 variables will almost certainly overfit to 50 data points, and because there is some multicollinearity to the X-variables, I then use stepwise regression with backward elimination to fit a model with fewer variables. The R summary output of the model returned using backward elimination, with 9 variables, is here:

and the R diagnostics plots from this model are:

4.) Lastly, here is a grid with variable distributions and correlations for these 9 predictor variables + the log(Market Cap):

as well as a table of Variance Inflation Factors

My thoughts on next steps are to fix the remaining multicollinearity issue by removing additional variables, and also remove some outlier data points indicated in the Residuals vs. Leverage graph. I also plan to test the model on a test data-set (I did a 40 / 10 split), 5 times using 5-fold cross validation.

I am interested in anyone's thoughts on transforming the X-variables, which I currently do none of. The grid of histograms / correlations / scatter plots shows that many variables are right-skewed. Additionally, I have graphed the simple linear regression fit between log(Market Cap) and each of the 9 remaining variables, and received these plots (for 4 of the 9):

And noticed for the most part that none of the X-variables seem to have a great linear-fit with log(Market Cap).

Any thoughts on next-steps on my model building would be greatly, greatly appreicated. Apologies again for the long post, but I felt that thorough / lots of images / plots would be helpful here. Thanks!

Your VIF values are extreme, and your residuals plot is a right-opening megaphone (vice a null plot, which is what you want). It looks like the residual plot is being strongly affected by BTC.

If I were working this I would look into Principal Component Analysis (PCA) for two reasons:

1) It can help mitigate your multicollinearity and drive down your VIFs

2) It can reduce the number of predictor variables you're using (linear regression is better on fewer variables)

I would also consider removing the BTC data point just to see what happens. This can give you some insights that can help you build the model. Good luck!


Creating neural net for xor function It is a well known fact that a 1-layer network cannot predict the xor function, since it is not linearly separable. I attempted to create a 2-layer network, using the logistic sigmoid function and backprop, to predict xor. My network has 2 neurons (and one bias) on the input layer, 2 neurons and 1 bias in the hidden layer, and 1 output neuron.
To my surprise, this will not converge. if I add a new layer, so I have a 3-layer network with input (2+1), hidden1 (2+1), hidden2 (2+1), and output, it works. Also, if I keep a 2-layer network, but I increase the hidden layer size to 4 neurons + 1 bias, it also converges. Is there a reason why a 2-layer network with 3 or less hidden neurons won't be able to model the xor function?

Yes, there is a reason. It has to do with how you initialize your weights.

There are 16 local minimums that have the highest probability of converging between 0.5 - 1.

Here is a paper that analyses the xor problem.


Why does it speed up gradient descent if the function is smooth? I now read a book titled "Hands-on Machine Learning with Scikit-Learn and TensorFlow" and on the chapter 11, it has the following description on the explanation of ELU (Exponential ReLU).

  Third, the function is smooth everywhere, including around z = 0, which helps speed up Gradient Descent, since it does not bounce as much left and right of z = 0.

The z means the x-axis on the graph above. I understand the derivative is smooth since the z < 0 line has a curve and in that realm the derivative is no longer equal to 0.

However, why is it the case that if the function is "smooth everywhere, including around z=0", it speeds up Gradient Descent?

My guess it is due to the derivative, as ReLU has a discontinuous derivative at 0. Hence if you use the definition:

$$
f' \approx \frac{f(x+\epsilon) -f(x-\epsilon)}{2 \epsilon}
$$

and $x$ is very close to 0, you will get many of those 'jumps'.


Why do neural networks use cost minimization of loss function and not profit maximization of profit function? In neural networks, gradient descent is used to find optimum minimum value of cost function. Why this preference instead of finding maximum value of profit function?
What are the pros and cons of minimising cost function versus maximizing profit function?

There is not necessarily a difference between the two, but the most important reason we minimize a cost function instead of maximizing a profit function is because of the optimization method. Neural networks are optimized using gradient descent, in which we use the derivative of the cost function to calculate the gradients and use the gradient to adjust the parameters. Assuming that you would use the accuracy as the profit function, it is not differentiable, meaning that you cannot calculate the derivate and therefore cannot use gradient descent (or in this case gradient ascent since we want to maximize a function). We then specify a cost function that is differentiable and minimize this as the cost function generally goes down as the profit function increases. You can use gradient ascent to maximize any arbitrary profit function as long as it differentiable.


with ML/DL model Is possible predict numbers of items required? I have a dataset is regarding ambulance call data.
Data sample:
     v_type      district   gender  complaint      age  Month
0   Advanced    District 1  Male    Chest Pain      28  jan
1   Advanced    District 2  Male    Heart Problem   50  dec
2   General     District 3  Male    Cardiac Arrest  76  jun
3   Advanced    District 4  Male    Heart Problem   45  oct
4   General     District 5  Female  Cardiac Arrest  52  nov
5   Advanced    District 1  Male    Chest Pain      34  feb
6   Advanced    District 2  Male    Cardiac Arrest  44  jun
7   General     District 3  Female  Heart Problem   55  july
8   Advanced    District 4  Female  Heart Problem   86  may
9   General     District 5  Male    Heart Problem   65  aug
10  General     District 1  Male    Heart Problem   60  nov
11  Advanced    District 2  Male    Chest Pain      36  mar

In the data v_type (Vehicle type) we have Advanced (Advanced featured emergency vehicle) and General (Basic featured emergency vehicle).
How to predict how many advanced or general ambulances should arrange in a district for a particular month?
Example: If in a month (jan) and in district3 has huge complaint then predict and need to show 5 or 6.. Advanced vehicle type is required

The first thing to do is to reorganize the data in a way which suits the problem that you're trying to solve: since the goal is to predict the number of each type of vehicle by month, your data should contain a column for this number.
Here you need to count the number of rows for each month and each type of vehicle so that you obtain something like this:
v_type   district    month number
Advanced District 1  jan   1
General  District 1  jan   2
Advanced District 2  jan   0
General  District 2  jan   3
Advanced District 1  feb   2
General  District 1  feb   3
...

After that you will probably need to represent the month (and year) in a way that a regression algorithm can use, typically an integer starting at month 1.


Semantic segmentation: mean IOU in presence of missing classes It seems to me that the mean IOU is a poor metric in the presence of unbalanced classes. E.g., suppose I have 10 classes but one image has only 2 classes present in its label. Consider the prediction where the 2 classes are inverted, the IOU for these classes is 0, but the IOU for the 8 other classes is 0/0. If we consider it to be 1 (since no prediction for those classes is the correct prediction), with uniform weights the resulting mean IOU is 0.8 which makes it sound like the prediction was quite good.

It seems that the mean IOU weighted by the proportion of pixels corresponding to each class in the target image solves this problem. Are there cases where using the later over uniform mean IOU is problematic? Why is the uniform mean IOU the more popular of the two metrics?

You somehow misunderstood how mean_IOU is calculated.

First, mean_IOU is not calculated per image and then averaging over all images, but calculated per class for images and then averaging over classes. That means for a single image, if some classes are missing (in both ground truth and your predicted labels), you don't need to worry about the 0/0 situation. You need to only update the confusion matrix, until you finished with all images, then calculate the mean_IOU.

In the extreme case, some classes are missing in all images. In that case, you should adjust you num_class parameter. Even if you don't, the missing classes will be treated as IOU=0 instead of 1 (sad...)

For example, suppose your image has 5 pixels and you have 3 classes. The ground truth vector is always [0,0,1,1,1] and you always predicted [1,1,0,0,0] (note that you predicted class 0 and 1 inverted, and class 2 is missing). In this case, the mean_IOU is 0, not 1/3.

Therefore, you should understand that the IOU is already being normalized by the sum of the numbers of pixies in (the union of your predicted pixels and the ground truth pixels) across all you images. So I don't think it's really necessary to re-weight the IOU any more.


Accuracy for Kmeans clustering I am looking for accuracy python code for kmeans clustering with no labels. Is there anyone who knows about it? it is ok that is not built-in function. Manually made is also ok

Accuracy is a measure of comparing the true label to the predicted label. K-Means is an unsupervised clustering algorithm where a predicted label does not exist. So, accuracy can not be directly applied to K-Means clustering evaluation. However, there are two examples of metrics that you could use to evaluate your clusters.

Within Cluster Sum of Squares

The first is Within Cluster Sum of Squares (WCSS), which measures the squared average distance of all the points within a cluster to the center of the cluster (known as the cluster centroid).

To calculate this, you can start by finding the Euclidean distance between a given point and the cluster center which the point is assigned to. You then repeat this process for every point in the cluster, and then sum the values for the cluster and divide by the number of points. Finally calculate the average across all clusters. This will give you the average within cluster sum of squares.

This measurement can indicate the variability of the points within the cluster in terms of the average distance to the cluster center. A large sum of squares could indicate a very large a spread out cluster. A small sum of squares could indicate a small, compact cluster with little variation in the attributes of the points. This measurement is sometimes called Cohesion because it measures similarity between the data points in that cluster.

Between Clusters Sum of Squares

The second metric is Between Clusters Sum of Squares (BCSS), which measures the squared average distance between all cluster centroids.

To calculate this, you can find the Euclidean distance from a given cluster centroid to all other cluster centroids. Repeat this for all of the clusters. Then, sum all of the values together. This will give you the Between Cluster Sum of Squares. You can divide by the number of clusters to calculate the Average Between Cluster Sum of Squares.

This measurement can indicates the variation between all clusters. A large number can indicate clusters that are spread out. A small number can indicate clusters that are close to each other. This measurement is sometimes called separation because it measures the separation of the clusters.

Other Resources

You can also look into the Silhouette Coefficient, which combines both the Cohesion and Separation. And the Elbow Method can be used to help you determine the optimal K value.

Checkout the limitations of k-means clustering. And if you want an in-depth explanation of WCSS and BCSS, check out this Khan Academy video:

https://www.khanacademy.org/math/statistics-probability/analysis-of-variance-anova-library/analysis-of-variance-anova/v/anova-2-calculating-ssw-and-ssb-total-sum-of-squares-within-and-between-avi

Depending on the language you are programming in, there may be packages available to assist you in evaluating your clusters. R has kmeans, which has attributes including withinss and betweenss. For Python, Sklearn's implementation of k-means has inertia, which is the "sum of squared distances to the closest centroid for all observations in the training set".

Consider a different algorithm

If you are looking to measure the accuracy of a prediction on a given dataset, it would help to define what the ground truth of that is. If you have a ground truth, there may be a better algorithm to use to model the outcome.


sending files through a network with python I'm building a multi-agent system  with python, the first agent built a model and saved it in yaml file, it saved also the weights in hdf5 file, then it must send this 2 files to the second agent.
the files are saved in my pc, and the 2 agents also are in my pc.
how could the first agent send the 2 files to the second one?
thank you.

Since both agents are on the same computer, save both files to a directory. The second agent can read both files from the directory.


Are there cyclic decision trees? Usual decision trees are directed acyclic graphs. Are there generalizations of decision trees that contain cycles analogously to recurrent neural networks? If such trees exist, can they be applied to sequences of variable length? I could not find information about such trees. Or their existence is impossible of infeasible?

The goal of decision trees is to partition the feature space into successively smaller regions where each region is best characterized by a single label or value. Adding cyclical components would not help accomplish the goal, it would unnecessarily repeat the modeling fitting for a given region.


Pros and cons of using the zscore of a dataset before normalizing it during feature engineering? Normalization is a common feature engineering technique. However, this post used standardize(zscore) on the dataset before normalizing it.

I think that would result in losing some of the information in data.

What are the pros and cons of doing this?

Normalizing an already normalized dataset should not change anything unless for some reason a different normalization scheme is used.


Noise Elimination with majority vote filtering I have a dataset with label noise which I wan't to clean with majority/consensus vote filtering. This will mean I will divide the data in K-Folds and train an ensemble model. Than using the predictions on the data I will remove rows, which are missclassified by most (majority voting) or all (consensus voting).

I have a few questions on which I can't find the answers elsewhere:

how to decide what models to use in the ensemble
the dataset is very imbalanced. Do I need to do upsampling in the majority voting?
do I do hyperparameter tuning in the different models, or just use standard settings?

I have a few questions on which I can't find the answers elsewhere

It's probably because there is no simple answer to these three questions :)

I doubt there's any state of the art approach, in such cases I simply try to determine the answer to these questions empirically. Basically I create a list of hyper-parameters including the type of algorithm, the algorithm-specific hyper-parameters and any other potentially relevant option. The goal is to determine the optimal combination of values for the set of parameters. If practical I run all the combinations and select the best one. If not practical, I use a simple genetic algorithm to find an optimal combination. Of course it's suitable only if you have a dataset large enough and if the training/testing process is not too computer-intensive. You also need to be very careful about overfitting by using cross-validation and re-sampling.


how to run bert's pretrained model word embeddings faster? I'm trying to get word embeddings for clinical data using microsoft/pubmedbert.
I have 3.6 million text rows. Converting texts to vectors for 10k rows takes around 30 minutes. So for 3.6 million rows, it would take around - 180 hours(8days approx).

Is there any method where I can speed up the process?

My code -
from transformers import AutoTokenizer
from transformers import pipeline
model_name = "microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext"
tokenizer = AutoTokenizer.from_pretrained(model_name)
classifier = pipeline('feature-extraction',model=model_name, tokenizer=tokenizer)

def lambda_func(row):
    tokens = tokenizer(row['notetext'])
    if len(tokens['input_ids'])>512:
        tokens = re.split(r'\b', row['notetext'])
        tokens= [t for t in tokens if len(t) > 0 ]
        row['notetext'] = ''.join(tokens[:512])
    row['vectors'] = classifier(row['notetext'])[0][0]
    return row

def process(progress_notes):
    progress_notes = progress_notes.apply(lambda_func, axis=1)
    return progress_notes

progress_notes = process(progress_notes)
vectors_2d = np.reshape(progress_notes['vectors'].to_list(), (vectors_length, vectors_breadth))
vectors_df = pd.DataFrame(vectors_2d)

My progress_notes dataframe looks like -
progress_notes = pd.DataFrame({'id':[1,2,3],'progressnotetype':['Nursing Note', 'Nursing Note', 'Administration Note'], 'notetext': ['Patient\'s skin is grossly intact with exception of skin tear to r inner elbow and r lateral lower leg','Patient with history of Afib with RVR. Patient is incontinent of bowel and bladder.','Give 2 tablet by mouth every 4 hours as needed for Mild to moderate Pain Not to exceed 3 grams in 24 hours']})

Note - 1) I'm running the code on aws ec2 instance r5.8x large(32 CPUs) - I tried using multiprocessing but the code goes into a deadlock because bert takes all my cpu cores.

There are several things that could make your code faster:

Stop using Pandas. Pandas is not designed for large-scale, fast text processing. It would be better to switch to something like Apache Arrow which is designed for efficient analytic operations on modern hardware.

Refactor code to avoid casting into memory-inefficient data types. progress_notes['vectors'].to_list() converts to a Python list which will use a lot of memory.

Refactor code to avoid casting back and forth between data types. Replace the list comprehensions and ''.join with complied regex. Complied regex processing will be much faster than equivalent Python code.

Refactor code to replace functions with inline code. Python creates a new stack for each function call. Inline code will be more memory efficient.


Sample Space of Longest Run I am trying to find a way to calculate all possible combinations of a sequence that have a certain length of long run.

When answering questions regarding sequences of heads and tails, sometimes participants will consider a sample space of longest run.
For a sequence of five coin flips, this is easy enough to calculate using a brute-force method - writing out all 32 possible sequences and then categorising them based on their longest run.
So, for example, a sequence with the longest run of 5 has a probability of 1/32 as there is only one way to have a longest run of 5 with a sequence of length 5.

However, I now have a sequence of length 10. I want to find out exactly how many sequences out of the 1024 possible sequences that have a longest run of 2 or 3.
I am assuming that it doesn't matter if the longest run is of heads or tails.

Is there code that could be used to calculate this?

So far I have:

x <- c(0, 1) # heads = 0, tails = 1

p = permutations(n = 2, r = 10, v = x, repeats.allowed = T)

p.df = as.data.frame(p)

To create a data frame of all possible combinations. Now I need to find the longest run of consecutive zeros or ones in each row and count how many rows have the same longest run.

Aha! Figured it out!

Here is the code I have:

install.packages('gtools')
library(gtools)

x <- c(0, 1) # heads = 0, tails = 1

p = permutations(n = 2, r = 10, v = x, repeats.allowed = T)

p.df = as.data.frame(t(p))

q = apply(p.df, 2, function(x){
max(rle((x == 0))$lengths)
}
)

length(which(q == 10)) #where q == the length of the longest run
```


why this naming convention for padding as "Same" and "Valid" in keras I was going through CNN's and found that padding argument should be set to "Valid" if i need no padding and "Same" if i need padding. But, it doesn't make any sense to me. Why can't keras development team just put "Padding = True", is there a specific reason to choose this convention?

Using valid will essentially use as much of your input as possible, such that the dimensions continue to work. This means there is a chance some input will be trimmed (removed).

same on the other hand, will add padding to allow e.g. the number of filters/convolutions you specify to be applied.

The reason not to simply have padding=True as an option, is that there are more than two options (see snippet from the docs below). Also, that would not be very explicit: using same or valid makes it crystal clear, what the result should be using the given approach.

This is what the documentation says:

  padding: One of "valid", "causal" or "same" (case-insensitive).  "valid" means "no padding".  "same" results in padding the input such that the output has the same length as the original input

Take a look at this nice answer with a simple example.


Clustering documents - how to evaluate results? I'm using DBSCAN clustering on a set of documents. The documents' content was converted to TF-IDF matrix, and I'd like to find consistent ways to evaluate the clusters when no added information is given (labels etc.).

Metrics comparing clusters - a score for each cluster - the goal here is to figure out which clusters are 'better' than others (more dense and more unique). Thought of two metrics:

*Intra-cluster similarity - measure the similarity of documents inside each cluster - I used the cosine distances between documents in the TF-IDF representation

*Inter-cluster similarity - measure the similarity between clusters (to highlight the more unique clusters) - similarly, planning to use the cosine distances between clusters' kernels
Metrics comparing clustering algorithms - a score for the clustering technique (such as Silhouette score). Will be used for hyper-parameter tuning and choosing an algorithm.

Any ideas?

None of the internal evaluation metrics will work well on text in my experience. Probably because of the curse of dimensionality.

Furthermore, DBSCAN does not cluster everything, but can also produce noise points. Few evaluation methods (and even fewer  implementations...) handle it well that noise is not a cluster.


PCA scikit-learn - ValueError: array must not contain infs or NaNs I use PCA from from sklearn.decomposition to reduce data dimension.

    pca = PCA(n_components=pca_components, whiten=True, svd_solver='full')
    y = pca.fit_transform(x)

The numpy array shape is (512, 48), dtype is float64. The minimum value in the array is 0.0, the maximum value is 1.7976931348623157e+308. The array does not contain infs or NaNs but I get an error

ValueError: array must not contain infs or NaNs

I have looke for reasons of this error and found nothing useful for me.
I suppose that maybe the maxim value in the array is too large for PCA. But I still do not know how deal with it.
Could you advice me something. Thanks alot in advance.

My guess is the issue is a combination of whiten=True and the largest value equal to the largest possible value of a float64.

From scikit-learn's PCA docs:

  When True (False by default) the components_ vectors are multiplied by
  the square root of n_samples…

That creates an overflow issue which results in the ValueError exception.


Should we use only one-hot-vector for LSTM input/outputs? Should we convert our inputs to on-hot-vectors and expect one-hot-vectors as output?
I mean can we feed LSTM with a vector like x=[12, -234, 54 , 78 , 12 , 6], and have a label vector like this: y=[13, -230, 50, 80 , 9 , 7]? (And we don't use one-hot-vectors at all).
Will such network work properly? Or it's better to convert inputs/outputs to a one-hot-vector and this is essence of LSTM?
If feeding LSTM with one-hot-vector isn't a necessary rule, and we like to feed our network with such vectors in my previous question, should we again use softmax() function for out outputs? Or we can use better options for such problem(or even don't use any functions there)?
If we must(or better) to use softmax, how can we interpret it's result?
If it's better to convert our inputs/outputs to one-hot-vectors, can we use two or three hot vectors(I mean: x =[1,0,0,1,0,0] or x=[0,1,1,1,0,0])? Does this work properly or it disrupts the LSTM performance?

This depends on what your data is representing and what you want to predict.  My understanding of One-Hot-Encoding is that this should only be used for encoding of categorical features.  For example, if you have a feature representing a category of K classes, you should one hot encode this as well as the Y variable (if you are trying to predict this categorical variable).  Of course, have the final layer be a softmax to output a distribution of size K.
This highly depends on what your data is representing.  If categorical, see above.  If just a simple numeric, you should not one hot encode this.  You could, I guess, if this set of integers is finite and small, but there is no need to learn the extra weights. You should only be using a softmax when you want to output a vector of K dimension where the entries all sum to one (perfect for representing a probability distribution over K classes).  The final layer should output whatever it is you want to predict.  If you want to predict something as simple as a numeric variable at the next time-step, just have a dense layer of size 1, with some activation function (relu probably).  More information about what exactly you are trying to predict is needed to know how to recommend anything concrete here.
I'm not sure about this.  You could have inputs and outputs represented this way, but you wouldn't use a softmax activation at the end.  You would use a dense layer outputting  a vector the size of your X variable, that is, if that's the variable you are trying to predict.  I'm not super well trained in various types of use cases for LSTMs, but from my experience, I can't think of a reason to do this.


analogy for pearson r statistics for binary classification task I am trying to get idea how variables of my data correspond to target variable (binary class).

In regression, Pearson r statistic is quite good to get sense of variable relationship. Also I can use it for classification, treating classes 0 and 1 as real values but it's a risky trick.

My question: is there any equivalent statistic saying which variable is good for classification task ? Thanks

The Pearson Chi-Squared test may give you what you need:
https://en.wikipedia.org/wiki/Pearson%27s_chi-squared_test

This is a good resource for analyzing categorical data:  https://en.wikipedia.org/wiki/List_of_analyses_of_categorical_data

Also, confusion matrices can be very helpful:  https://en.wikipedia.org/wiki/Confusion_matrix


How to find relation between N components and predict the value of any one component using the predicted relation? I am new to Machine learning and trying to learn by practicing.

I have a situation where I am reading a set of N data. Each of N data will have independent state at any moment of time. I want to use these N data to find a relation between them. In case if I am not receiving one or more sets of data, I should be able to use available relation between N data to predict the missing data.

What type of machine learning algorithm can be used?

I understand your task can be formulated as an imputation task, see the link below for further details. Also before you proceed look up the concepts of missing data.

In order to fill in missing data you’ll have to build N distinct models using available observations. Each of N models will have N-1 inputs.

Note that because more than one component can be missing your models should be able to handle the missing data. According to my knowledge neural networks cannot handle the missing data out of the box. Tree based models, such that random forests or GBMs are more suitable for this kind of tasks.

http://www.stat.columbia.edu/~gelman/arm/missing.pdf