TypeError: unhashable type: 'numpy.ndarray' I'm trying to do a majority voting of the predictions of two deep learning models.The shape of both y_pred and vgg16_y_pred are (200,1) and type 'int64'.

max_voting_pred = np.array([])
for i in range(0,len(X_test)):
    max_voting_pred = np.append(max_voting_pred, statistics.mode([y_pred[i], vgg16_y_pred[i]]))

I run into the following error:

TypeError: unhashable type: 'numpy.ndarray'

How should I pass the data?

The problem is that you're passing a list of numpy arrays to the mode function.
It requires either a single list of values, or a single numpy array with values (basically any single container will do, but seemingly not a list of arrays).

This is because it must make a hash map of some kind in order to determine the most common occurences, hence the mode.  It is unable to hash a list of arrays.

One solution would be to simple index the value out of each array (which then means mode gets a list of integers). Just changing the main line to:

max_voting_pred = np.append(max_voting_pred, mode([a[i][0], b[i][0]]))

Let me know if that doesn't fix things.

If you want something that is perhaps easier than fixing your orignal code, try using the mode function from the scipy module: scipy.stats.mode.

This version allows you to pass the whole array and simply specify an axis along which to compute the mode. Given you have the full vectors of predictions from both models:

Combine both arrays to be the two columns of one single (200, 2) matrix

results = np.concatenate((y_pred, vgg16_y_pred), axis=1)

Now you can perform the mode on that matrix across the single rows, but all in one single operation (no need for a loop):

max_votes = scipy.stats.mode(results, axis=1)

The results contain two things.

the mode values for each row
the counts of that mode within that row.

So to get the results you want (that would match your original max_voters_pred, you must take the first element from max_votes:

max_voters_pred = max_votes[0]


Multivariate VAR model: ValueError: x already contains a constant I have already read this question and the associated answer.

I have removed any 'all zero' columns, as recommended in the answer. I have 3,169 columns remaining.

datavals_no_con = datavals.loc[:, (datavals != datavals.iloc[0]).any()]

I checked whether any were missed, for some bizarre reason:

varcon = np.asarray([np.var(datavals_no_con[datavals_no_con.columns[i]]) for i in range(len(datavals_no_con.columns))])
print np.where(varcon==0.) #empty array.

Also checked the minimum column variance value, which ended up being 4.306x10^(-7)

This was generated by a column that has no zero entries.

When I run this:

model = VAR(datavals_no_con)

results = model.fit(2)

I still get:

Traceback (most recent call last):
  File "vector_autoregression.py", line 163, in <module>
    results = model.fit(2)
  File "/user/anaconda2/lib/python2.7/site-packages/statsmodels/tsa/vector_ar/var_model.py", line 438, in fit
    return self._estimate_var(lags, trend=trend)
  File "/user/anaconda2/lib/python2.7/site-packages/statsmodels/tsa/vector_ar/var_model.py", line 457, in _estimate_var
    z = util.get_var_endog(y, lags, trend=trend, has_constant='raise')
  File "/user/anaconda2/lib/python2.7/site-packages/statsmodels/tsa/vector_ar/util.py", line 32, in get_var_endog
    has_constant=has_constant)
  File "/user/anaconda2/lib/python2.7/site-packages/statsmodels/tsa/tsatools.py", line 102, in add_trend
    raise ValueError("x already contains a constant")
ValueError: x already contains a constant

How can I resolve this?

EDIT: It occurred to me that the problem would be that x contains a constant, not that x contains all 0s. So the original answer suggested in the previous question was not entirely sufficient.

To test whether any of my columns contained 'all the same value' (e.g. a column of all 0.5), I tried this:

ptplist = []
for i in range(len(datavals_no_con.columns)):
    ptplist.append(np.ptp(datavals_no_con[datavals_no_con.columns[i]], axis=0))

ptparray = np.asarray(ptplist)
print any(ptparray==0.) #FALSE

So none of my columns are constant, unless I'm still missing something.

EDIT 2: I have found the root cause of the problem.

Suppose my input matrix (that is, my set of endogenous variables) is a 5x5 identity matrix, for the sake of argument, and that my lag value is 2 (that is, I'm looking for an AR(2) model: y_{t+1} = A + B_1y_{t} + B_2y_{t-1} + error) :

y = np.eye(5)

1 0 0 0 0 (row 1)
0 1 0 0 0 (row 2)
0 0 1 0 0 (row 3)
0 0 0 1 0 (row 4)
0 0 0 0 1 (row 5)

In the get_var_endog function in /statsmodels/tsa/util.py, under lags=2, the y matrix gets rearranged to this general idea:

[row 2, row 1] (i.e. concatenate these two)
[row 3, row 2]
[row 4, row 3]

And this new matrix could have zero columns, in places where my original data matrix did not. In fact, this is exactly what was happening. Following my example, the np.array Z in get_endog_var looks like this:

0 1 0 0 0 1 0 0 0 0
0 0 1 0 0 0 1 0 0 0
0 0 0 1 0 0 0 1 0 0

So now columns 0, 4, 8, and 9 are completely 0, which throws the ValueError.

Two possible approaches to a solution come to mind:

1) Remove the zero columns from the Z matrix.

2) Edit the original data set such that these zero columns never occur in the first place (much harder, because then the Z matrix here would never have existed, so how can you know which columns to remove...catch 22).

I chose option 1, but now I'm dealing with shape issues down the line. Because, of course, when doing the least squares fit, the shape of the parameters is going to be different from the shape of the original data set (some columns don't exist in the parameters, because I removed them, that do exist in the original data set).

Now, this looks like it should be a relatively frequent problem. A lot of the time, we're working with high-dimensional sparse data, which would generate this issue.
Does anyone have a more robust solution than what I've proposed?

It seems you may have things working, but maybe I can still help

Catching those pesky non-varying features

Assuming a is your dataframe, with only numerical types and columns as features, you can try the following, which will return rows filled with NaN values for features of zero variance i.e. the mean value is equal to the minimum value, so the value cannot vary.

a.T.where(a.T['mean'] == a.T['min'])

They check is slightly differently in the source code of the VAR model which raises your error:

result = (np.ptp(s) == 0.0 and np.any(s != 0.0))

So they check that the value doesn't have any peaks (ptp = "peak-to-peak"), so it doesn't vary - and also that the value is not equal to zero.

In any case, there is little tolerance for it. You have done right by trying to prune out features with low (near zero) variance.

Another possiblity which I didn't notice you having already tried, would be to utilise the other arguments to the fit() method of the VAR model class. It has the following signature:

VAR.fit(maxlags=None, method='ols', ic=None, trend='c', verbose=False)

If you were to change the trend argument to nc, meaning no constant - you may also get around your error. Check out the documentation.

Source code analysis

Looking through the contents of the file that raised your error, it seems it occurs while processing the "original input data" - as defined by the function add_trend (where your error was raised).

This suggest that your reasoning about the error being found on the data after lagged features are produced would be incorrect in this case (although it's a clever idea!)

Clearing my name...

You said:

  ... So the original answer suggested in the previous question was not entirely sufficient.

I would like to point out that my answer to your linked question was indeed sufficient, because I had written that there are constants - not that the values must all be equal to zero!  :-P


Resources for datasets Is there a go to source for searching for data sources? Like a way to ask google, specifically to show data files?

Specifically, I'm looking for data on the number of smart phones and tablets sold in countries around the world.

Any ideas where I can look?

Browse the following sources to see if you could find them:

UCI ML Repository
OpenML webpage (login required to download)
List of datasets for ML research (Wiki)

Hope this helps.


Accuracy and Loss in MLP I am trying to explore models for predicting whether the a team will win or lose based on features about the team and their opponent. My training data is 15k samples with 760 numerical features. Each sample represents a game between two teams and the features are long and short term statistics about each team at the time of the game (i.e. avg points over last 10 games).

My thought was to use a binary classifier as a multi layered perceptron.  Each layer has batch norm, dropout and an relu activation. At the output layer there is a sigmoid activation. I am also using principal component analysis to reduce the dimensionality of my dataset.

Im am using an automated hyper-parameter tuner with randomized search and population based training to find optimal accuracy hyper-parameters. I am able to tune the hyper-parameters such that training loss and validation loss both converge to a very small value but when I look at accuracy I am getting very minimal improvements.

When looking at the initial accuracy after one epoch of training, it seems to be around 60% across all initial hyperparams, and it only reaches a max of about 70% after 30 epochs.

Im not really sure what to do, does this imply my data is to noisy / random? Should I reconsider my architecture?  Would I get better results with a regression instead of classification? Also, since the data is sequential in nature this might lend itself to a recurrent architecture but Im not familiar with any RNN that is trained on multiple sequences as opposed to a single sequence.

It is hard to tell if it is the model or the signal/noise ratio in data. One sanity check could be to see how well a human would perform with the most important features.

I don't think you want to switch to a sequence model, I have a hard time seeing it do better for this data. Also, don't switch to regression.

Some suggestions:

Try a gradient boosting model, like LightGBM or XGBoost. It is really good on tabular data.
Do feature engineering and try to come up with features that might be good but does not exist in the data. You could aggregate to get the sequential information in there, but you already seem to be doing that to some extent.


Visualization for comparing the results of two different models I have two different mathematical models I use to predict the dollar amount in claims a group will incur over the course of a year. I want to visually juxtapose how accurate those two models are on historical data (i.e. I know what actual claims for each group are) with group-level granularity. I haven't done much with data visualization in the past, so I was wondering if there are any pre-existing data-visualization paradigms that this kind of thing might fit into. Thanks.

If you have a time series I guess plotting actual vs. predicted values in a graph would work fine. Maybe throw in the MAPE and adjusted R^2 as single values in one of the corners.


Why don't we use space filling curves for high-dimensional nearest neighbor search? Some space filling curves like the Hilbert Curve are able to map an n-dimensional space to a one dimensional line whilst preserving locality. Does that mean that we could map a dataset of high dimensional points to a line and expect the order of the nearest neighbors to be preserved?

If so, wouldn't that be more efficient than building a Ball tree?

Preserving "locality" is a desirable property in space-filling curves, especially in this context.  But we cannot hope to actually preserve the ordering of the nearest neighbors for all points.  (The discrete versions of Hilbert's curve put adjacent cells nearly on opposite ends of the curve; in wikipedia's illustrations, see the top-central region.)

This idea might still be useful, as an sort of k approximately-nearest neighbor algorithm; but I'm not convinced that producing an appropriate (discrete version of) a space-filling curve for a dataset and querying new points onto that curve will actually be substantially faster than ball-tree.  Another problem is that the error in the approximations of "nearest" are dependent on the location along the curve, so that which curve you select will have a substantial impact on the correctness of the approximations for different sample points.


What is the advantage of using Dunn index over other metrics for evaluating clustering algorithm? There are many metrics to evaluate clustering algorithm like Calinski-Harabaz Index, Dunn index, Rand index, etc. Are there any advantage of using Dunn index over other metrics for evaluating clustering algorithm (K-means in particular)? If yes, what are the advantages and disadvantages?

They will often give the same preferences.

Do not forget that these are largely heuristics. They won't have much advantages over one another. That is why there are so many.

As a guideline, look at the definitions. Choose that index, whose equation is most relevant for your problem. (Yes, you do need to understand what they do. Clustering is hard, it is easy to get meaningless results by just looking at the code and scores and forgetting the underlying math.)


Segment 5-7 min audio into sentence wise audio clips for creating speech recognition dataset I am trying to create a speech recognition dataset, especially for Indian Accents.
I am taking from colleagues to build this.
Daily I send an article link and ask them to record and upload it to google drive.
I have a problem with this approach.
All audio recordings of length 5 -7 min.
I am using the DeepSpeech model for this and it requires 10-sec audio sentences.
Suggest me any approach if possible to segment audio files into corresponding sentence phrases or to build a better with 5 min length audio files.
Suggestions are more than welcome on a better way to create a speech-to-text dataset.

The typical approach is to just cut the clips into consecutive sections, and run the model on each such section. Sometimes a bit of overlap is used, say 10%. then you have to decide what to do with potential conflicts in these overlaps.
A good model is usually robust against silence, otherwise you can try to cut silence in start and end of your 10-second window.

librosa.util.frame is a practical way of doing this in Python.


Feature scaling for MLP neural network sklearn I am working with a dataset that has multiple scales for my features. Before running sklearn's MLP neural network I was reading around and found a variety of different opinions for feature scaling. Some saying you need to normalize, some saying only standardize, others saying in theory nothing is needed for MLP, some saying only to scale training data and not testing data, the sklearn documentation says MLP is sensitive to feature scaling? This has left me very confused on which route I should take for my dataset before running the MLP model. Any clarification on these and how I should proceed would be much appreciated.

In short:

Scaling is indeed desired.
Standardizing and normalizing should both be fine. And reasonable scaling should be good.
Of course you do need to scale your test set, but you do not "train" (i.e. fit) your scaler on the test data - you scale them using a scaler fitted on the train data (it's very natural to do in SKLearn).

For example, if you're normalizing your data (like with an SKLearn StandardScaler object), you .fit it on the train data to get the mean and standard deviance from it, and you .transform both train and test data to subtract the train mean and divide by the standard deviance.


How to preprocess an ordered categorical variable to feed a machine learning algorithm? I have a categorical variable that measures the income of a family:
A: no income
B: Up to $500
C: $500-$700
…
P: $5000-$6000
Q: More than \\\$6000

It seems odd to me that I have to get dummies for this variable, since it's ordered. I wonder if it's better to map the values: {'A': 0, 'B': 1, …, 'Q': 17} so I can input it into the algorithm this values as integer numbers.
What's the proper way of preprocessing this variable to feed an algorithm such as Random Forest or a simple neural network?

One way to do is to use target encoding:

https://medium.com/analytics-vidhya/target-encoding-vs-one-hot-encoding-with-simple-examples-276a7e7b3e64
https://towardsdatascience.com/target-encoding-and-bayesian-target-encoding-5c6a6c58ae8c

(There are a million resources to learn target encoding)
This way your categories will not only be ordered by the number but for the target value (what is at the end what you want, to give better predictions)


Meaning of axes in a clustering plot If you have n time series of rainfall measurements every hour (x=time, y=amount of rain), and compute the distance matrix between each pair of time series based on Dynamic Time Warping, and then plot the clusters, what do the X and Y axes on the cluster plot represent?

I have figured out that the values in the distance matrix would have the same measurement unit as the amount of rain in the time series (mm or inches of rain).

In the k-medoids or k-means plot, is X still time, and Y still the amount of rain?

If you have $n$ time series of rainfall measurements every hour ($x$=time, $y$=amount of rain), and compute the distance matrix between each pair of time series based on Dynamic Time Warping, and then plot the clusters, what do the X and Y axes on the cluster plot represent?

They remain the same, $x$=time, $y$=amount of rain - dynamic time warping (DTW) shifts the times so the curves of the amount of rain align their peaks and valleys.

A different example might be easier to understand. In the paper by Berndt and Clifford (see below) the population of the Snowshoe Hare is plotted and subjected to DTW. You can see as the population of hares increased so did the ability to hunt Lynx. As the population of Lynx increased the population of hares declined. That affected the hunting of Lynx, which again allowed the hare population to increase. This creates a double-top peak in the data, using DTW these can be aligned.

  In the k-medoids or k-means plot, is X still time, and Y still the amount of rain?

Having aligned the time/population (similar to time/rainfall) the locations on a map can be plotted with a k-medoids or k-means plot.

K-medoids (also: Partitioning Around Medoids, PAM) uses the medoid instead of the mean, and this way minimizes the sum of distances for arbitrary distance functions. K-means clustering aims to partition n observations into k clusters in which each observation belongs to the cluster with the nearest mean, serving as a prototype of the cluster.

The time variable would be plotted on the map for a period of time, for example a month, season, or year; so while the time is not 'lost' it's also not $x$ or $y$. Instead a color on the map could represent the amount at a location for the entire duration of the period of time.

See how when k-medoids or k-means is applied "time" does not sit on $x$ or $y$, in this example location is on the $x$ and $y$:

References:

"Alignment of curves by dynamic time warping", by Theo Gasser and Kongming Wang.

"Using Dynamic Time Warping to Find Patterns in Time Series", by Donald J. Berndt and James Clifford.

"Clustering analysis applied to NDVI/NOAA multitemporal images to improve the monitoring process of sugarcane crops", by Luciana Alvim Santos Romani, Renata Ribeiro do Valle Gonçalves, Bruno Ferraz Amaral and Agma Traina.


There is no certain answer, only trial and error. Though it should help.

Let me elaborate. Feature importance shows the impact of features on the quality of the model: the number of times there was a split using this feature or gains from splitting on this feature. The better is the feature, the higher is the importance.
But some features could be important due to interactions with other features. Also if some features have a high correlation between them, the importance value could be split between them.
In general removing low importance features should have a small impact on the metric. It could be positive or negative. Just try and see :)


Unbalanced training set from balanced data I am looking to get an unbalanced training set with a given ratio of classA:classB from a dataset without regarding if it is balanced or not. The point is to analyze the influence of data imbalance on the accuracy.
I don't see any built-in function to go about this, so I was wondering what approach I should take. For now, I consider either undersampling or oversampling, but I am worried that it might skew the results too much, as I aim to get class ratios of 10/90, 20/80, ..., 80/20, 90/10.

With a straightforward approach, you can just oversample one of the classes to get the imbalance. You can achieve it with using SMOTE for example. Usually this technique is used to get a balanced set from imbalanced, but it can also work vice-versa, just oversample only once class.
Some links to check:
https://imbalanced-learn.org/stable/over_sampling.html
https://machinelearningmastery.com/random-oversampling-and-undersampling-for-imbalanced-classification/
https://towardsdatascience.com/how-to-effortlessly-handle-class-imbalance-with-python-and-smote-9b715ca8e5a7


what exactly happens during each epoch in neural network training Across different epochs, which of the following is/are updated?

initial weights (initial ConvNet filter matrices, initial fully connected weights)

hyper parameters: number of ConvNet filters, size of ConvNet filters, number of layers...

The lost function calculated from the last epoch appears to be the initial value of the lost function for the current epoch. Why?

You are updating your network parameters, that is, weights for fully connected layers, for the filters in the convolution operations, etc.
The hyperparameters are fixed once you start training your network. Hyperparameters are not intrinsic to the learning process and is something that the practitioner should tune carefully with GridSearch, Bayesian Optimization and Cross-Validation techniques.
You have just one loss function during training, and at each batch procesing you update your weights correcting your network and, at least theoretically, diminishing your loss function. So after the first epoch, you have reached a certain value, that will be update on the next epoch.
Think as you are on the top of a mountain, and you are climbing down, to no get tired, you count 10 steps and rest a little, after 10 steps you are not on the top again, you are going down from where you stopped, right? That is an analogy (I think it is bad, but if you understand it is ok haha).


Which approach to select category based on keywords I want to assign a certain category to a group of keywords.
So i.e. people can upload images or videos, when they do this they can set keywords for this.
These keywords are free to type so words can be spelled in different ways.
The amount of keywords are 95% between 0 and 20 words.

I want to create categories from these. So that I can assign a combination of keywords to a category.

The categories and the amount of categories are undefined.

From what I've researched is that this is probably a Topic modeling or clustering issue. Although with topic modeling most examples I see are based on long texts instead of a couple of keywords.

What would be a good approach to handle this?

I thought about first some simple fuzzywuzzy to find different spellings of the same words.

Create a big word list from this. Then each keyword will be matched against the list and rewritten if it matches one and added if there isn't a good match.

Then I would need to create groups and here I don't know which algorithms I should use.

I was thinking maybe do k-means clustering and then see on which k I get the best results and then assign a category to it manually by looking at which keywords are in it.

So it would be nice to just let the algo figure the amount and categories out, but it can be relaxed that I will set them before.

Does anyone have a better suggestion or are there just complete algorithms for this already available?

Does anyone have a better suggestion or are there just complete algorithms for this already available?

Apparently you want to do information retrieval (IR) but without the information part: normally an important part of the IR process is the set of documents that the user is searching for (for example text, images video). This is important because knowing the documents which correspond to a query gives semantic information about the keywords.

Now, if you only have the keywords available then there's no semantic to help with the clustering. So you are right that the only thing you can use is the spelling. i assume that by " simple fuzzywuzzy" you mean "fuzzy matching", i.e. using string similarity measures. I can think of two options:

You can compare each keyword with each other with some string similarity measure such as Levenshtein edit distance, Jaro, or characters n-gram based Jaccard, cosine etc.
You can represent each keyword as a vector based on the char n-grams in the word, then you can apply clustering with k-means for example.

Topic modeling techniques wouldn't work because you don't have large text documents.


In Python, why subsetting with or without square bracket is different? Suppose I have a data frame called quoteDF

quotesDF

              volume  shares
2017-01-03   2934300     100
2017-01-04   3381400     120
2017-01-05   2682300     140
2017-01-06   2945500     160
2017-01-09   3189900     180
2017-01-10   4118700     200

If I do,

> quotesDF.loc[1, 'shares']
120
> quotesDF.loc[1, ['shares']]
shares 120
Name: 1, dtype: object

Why the first one retunrs 120, the second one retures shares 120? In my mind, they are the same thing, except I put the second one in a vector. But, the first one is a vector stands by itself. It's just that I didn't put the square bracket on it.

Why Python give me such a confusing time?

Assuming, you have a pandas dataframe, .loc is strictly label based. Since you're using [] it accesses the column you're specifying inside the brackets and that is the reason you're getting shares 120. Read documentation for better explanation.

Here is another link that has answers similar to your question.


What does localist one-hot vector mean in cs224n NLP course? Chris said one-hot is a "localist" representation.

what does "localist" mean here?

I've searched on recommended text, didn't find explanation.

any clue?

One-hot vector is called "localist" because it contains information only about a single data point, and does not give clues about other points, in contrast to a distributed representation (e.g. result of an embedding algorithm) that contains information about other data points too.

For example, one-hot encoding of two animals and two flowers is as follows

dog   = [1, 0, 0, 0]
cat   = [0, 0, 1, 0]
rose  = [0, 1, 0, 0]
tulip = [0, 0, 0, 1]

Distance between any pair is the same, and effectively by knowing the vector of "rose" we gain no information about the related "tulip" or unrelated (or less related) "dog" and "cat", hence the name "localist".

But if we happen to find a distributed representation like

dog   = [1.0, 0.1]
cat   = [1.1, 0.2]
rose  = [0.0, 1.2]
tulip = [0.1, 1.3]

The vector of "rose" alone gives us clues about other flowers like "tulip", which are closer, and also about other entities like animals, which are farther, hence a name like "globalist".

Two notes:

Being "localist" is not a zero / one thing. The more clues a representation gives about other data points, the less it is localist.
The notion of "relatedness" is task dependent. For example, in the task of "grouping things with similar colors", a red car is more related to a red rose than a blue car.

Here is a random image of those entities for fun!

If all pairs look equally different, your brain works with "localist" vectors, thus has no clue ;)


Transition plot in R how does it work? I am trying to visualise a transition from one state (work in origin country) and another (work in destination country). I think transition plot is perfect but I can't get how does it work?

My data frame is called (Cimad) and my variables are EMPLOIPRE (origin) and EMPLOIFR (destination). Both are factors, the first count 20 levels, the second 15 levels (is it a problem, do I have to make them all match?) and I have 400 observations.

I hope you'll be able to help me because I am completely lost !
Thank you in advance
Caroline.

The problem with Sankey diagram in your case is that you will duplicate all the nodes (on the left, and on the right), that is messy.

Networks are better, but nodes are not well structured.

I suggest to try chord diagrams for such data.

One of the greatest examples of chord diagram was build with d3.js, and based on migration flows dataset. Here is an R wrapper for d3.js chords. Or you can try to reproduce this manual.


Best tool for text pre-processing, involving tokenization, lemmatization, stop-word removal, feature vector extraction? For text processing there are plenty of tools out there like CoreNLP, SpaCy, NLTK, textblob etc. each offering different suites of pre-processing functions and people recommend different tool for different tasks like NLTK for tokenization etc.

Which tool would you recommend which performs all these tasks?

Comparing two libraries or tools in terms of these things is somewhat that is opinion dependent. Some people prefer NLTK for doing almost all the tasks. spacy has also gained quite reputation. But what is better for you depends on what you want do. In my personal experience, I have found that NLTK along with gensim libraries is all that I need to do all the Natural Language Processing tasks.


How to find the dependent variables from a dataset? I am stuck at where how can I get the most dependent variables based on the mean

I have this dataset and when I try to:

df.groupby('left').mean()

It gives the output:

And one of my friends said, from that graph the dependent variables for the attribute left will be

1.Satisfaction Level

2.Average Monthly Hours

3.Promotion Last 5 Years

I am wondering How could someone guess that?

in Statistics, the independent variables are inputs over which you have control. The dependent variables are the outcome observed by altering the values of the independent variables. Therefore, the answer is "it depends".

Therefore, if you are studying how alterations of left values influence the values of satisfaction level. Then, left values is your independent variables and satisfaction is your independent variables

Instead, if your friend was concluding that variations in left values causes changes in Satisfaction Level, Average Monthly Hours and Promotion Last 5 Years, then, it is a very biased conclusion based on no significant evidence. It would be more appropriate to talk about correlations between variables for which you only know that some behavior appear together (correlated) without identifying any causal effect.


What are the differences between logistic and linear regression? I know that linear regression does "regression" and logistic regression does "classification". When we implement these two methods, the only difference I could notice is the loss function: linear regression uses a loss function like mean square error and logistic uses cross-entropy. Is there any other difference that I am not aware of?

As you have mentioned, the output of linear regression is a real value while logistic regression's represents classes(classification). Their main difference is this.

The loss function of linear regression is convex which means you always can find the optimal point using customary optimisations while if you use that for logistic regression, you may get stuck in a non-global minimum which is not optimal. Consequently, people take logarithm of that loss and call it cross entropy. For simple logistic regression tasks, it is convex.

Another difference that has to be cared about is the non-linearity which is usually applied for different tasks. For logistic regression, it is customary that people use non-linearites like Tanh or Sigmoid after the linear part, inner product of weights and inputs, for specifing the similarity to class 1 or 0 for typical binary classifications. For linear regression, people usually use Linear activation function. There is a point here. The idea of using the linear activation is not due to the need for employing a linear function. It is used because its output is not limited. Consequently, you can use other functions that are one-to-one and are not limited. Consider $y = x^3$ as an example.


(Newbie) Decision Tree Classifier Splitting precedure I have a dataset with 4 categorical features (Cholesterol, Systolic Blood pressure, diastolic blood pressure, and smoking rate). I use a decision tree classifier to find the probability of stroke. I am trying to verify my understanding of the splitting procedure done by Python Sklearn. Since it is a binary tree, there are three possible ways to split the first feature which is either to group categories {0 and 1 to a leaf, 2 to another leaf} or {0 and 2, 1}, or {0, 1 and 2}. What I know (please correct me here) is that the chosen split is the one with the highest information gain.

I have calculated the information gain for each of the three grouping scenarios:

{0 + 1 , 2} --> 0.17

{0 + 2 , 1} --> 0.18

{1 + 2 , 0} --> 0.004

However, sklearn's decision tree chose the first scenario instead of the third (please check the picture).

Can anyone please help clarify the reason for selecting the first scenario? is there a priority for splits that results in pure nodes. thus selecting such a scenario although it has less information gain?

I have added the frequency of each class/feature/category so it would be easy to calculate the Gini index

sklearn doesn't know that your feature is categorical; it's treating it as continuous, for which only splits of the form $x \leq \alpha$ are checked, so your second listed split candidate isn't actually a candidate.

In general, sklearn doesn't support categorical variables (yet?), and you'll need to encode it differently (one-hot?) if you want different behavior.

See also
https://datascience.stackexchange.com/a/52103/55122
how to make a decision tree when i have both continous and categorical variables in my dataset?


Neural Network backprop formula - Matrix dimensions won't match? I want to start by taking an example for a normal neural network with 2 input nodes, 3 hidden nodes and 2 output nodes. Let the weights between input and hidden nodes are $W_i{_j}$ (2x3) and weights between hidden nodes and output (3x2) are $W_j{_k}$. Last layer has linear activation. The gradient of error wrt:

$$W_i{_j}= W_j{_k} * g'(W_i{_j} * x) * x * err$$

But this does not satisfy the weight update equation as the product gives the size 2x1. But it should be 2x3.

It's important to note that you are storing weights towards previous layer in columns. So my example will be specifically for that case.

If you are using a different notation (weights from neuron towards previous layer are in rows of your matrix) -  swap the words "rows" and "columns" below:

For your case, any incoming gradient (from the layers above or from the Error aka Cost function) must have dimension 2.  It must be a vector. Otherwise it will not physically "match" your output nodes.

Section A

The gradient wrt to your weights (hidden-to-output layer) will be a 3x2 matrix. You can subtract this matrix from your weights, to correct them. Don't forget about learning rate.

To build this matrix, copy the incoming gradient (vec of dimension 2, already passed through the activation function) into every row of a new, empty matrix (3x2), then component-wise multiply its every column by the fwdprop output of your 3-node hidden layer. This is your GradForWeights matrix

Section B

The gradient wrt hidden nodes will be a 3d vector.

To make it, use dot product between your Weights (3x2) and the incoming gradient vector (2d).  The resulting vector will be of dimension 3, and can also be used as the "incoming gradient vector" for the layer below (of course, after you pass it through the activation function of that sub-layer).


Any Online R console? I am looking for an online console for the language R. Like I write the code and the server should execute and provide me with the output.

Similar to the website Datacamp.

R On Cloud provides a browser-embedded R-console.

Jupyter.org evolved from the IPython Project (the language-agnostic parts of IPython); supports Python 3, Julia, R, Haskell, Ruby, etc.


Correlation between products based on purchases placed around the same date Association rule learning has a fair bit of material based around the correlation of products purchased on the same order/at the same time. However I'd like to discover if there is a method for identifying such a relationship between products that are ordered near each other, but not together.

Say for example a customer purchases a pencil in week one, but later purchases an eraser in week two. Then a year later they do the same thing. But then also apply this across all customers who show similar behaviour. I would then aim to combine this with the correlation coefficient of products purchased in an order together.

I considered giving each of the dates before or after a purchase a diminishing weight (1/n days) and then comparing the aggregate date value of the customer purchase date history with a product A against product B. But this seems like a fairly drawn out method.

Just looking for a process to achieve this, but python hints would also help.

Thanks in advance.

Association rule mining is nearly always done with the apriori or eclat algorithms. There are a few papers which have utilized these in the time series context. This paper details a thorough implementation, though this is done with relation to identifying gene patterns, rather than customer purchases. The idea is still the same though, only there is slightly less complexity in your case. Unfortunately, this does not give a detailed Python implementation.

Depending on your situation (this is unlikely to be applicable from your description, but I list this just in case...), it may be useful to break from association rules and cluster your data instead, provided you have enough of it to treat it as continuous. A Python implementation of this is featured in this article.

Good luck!


GBM: small change in the trainset causes radical change in predictions I have build a model using transactions data trying to predict the value of future transactions. The main algorithm is Gradient Boosting Machine. The overall accuracy on the testset is fine and there is no sign of overfitting. However, a small change in the training set creates radical change in the model, and in the predictions. But even when the testset change a little the overall accuracy is stable.

The time period is from 2005 to today and when a single day is added to the dataset predictions change drastically (e.g. +/- 10%). If multiple training are perform on the same training set, the predictions are the same.

I have test Light GBM(2.1.0) and XGBoost(0.60) with Python 3.6 on Windows 10. A seed is set and I train the model on CPUs. I have tried to increase the number of iterations to a high number and adding a specific seed to the bagging parameters.

This blogpost discuss brefly that issu without giving any solutions.

A good way to avoid this problem is to add noise to your training dataset this will made your model more robust and less versatile.

there is a different way of adding noise, often a gaussian noise is added it depends to the kind of data you have.


Using HashingVectorizer for text vectorization Here is the sample data I have:

Tag 1(Val: X), Tag 2(Val: Y), Tag 3(Val: Z), Label (Val: P)

Tag 1(Val: A), Tag 2(Val: B), Tag 3(Val: C), Label (Val: Q)

Tag 1(Val: D), Tag 2(Val: E), Tag 3(Val: F), Label (Val: R)

Tag 1(Val: G), Tag 2(Val: H), Tag 3(Val: I), Label (Val: S)

I started by putting the Tags into a dataframe df and the Label into a separate dataframe df_label.
Then used a HashingVectorizer to prepare the text for processing by ML models (I want to hash the strings into a unique numerical value so that the ML Models can train on it)

vectorizer = HashingVectorizer()

X_train = vectorizer.transform(df)

y_train = vectorizer.transform(df_label)

clf = RandomForestClassifier(n_jobs=2, random_state=0)

clf.fit(X_train, y_train)

When I execute this, I get: ValueError: Unknown label type: 'unknown' on y_train.

I am new to both Python and ML and I am not sure whether the problem is with my basic logic or whether it is a trivial implementation issue. Appreciate your insight and support.

I was recently checking some things out. Thought would leave a working code here, in-case its helpful.

from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction import FeatureHasher
from sklearn.ensemble import RandomForestClassifier
from sklearn import metrics
import numpy as np

categories = [
    'alt.atheism',
    'talk.religion.misc',
    'comp.graphics',
    'sci.space',
]
newsgroups_train = fetch_20newsgroups(subset='train', shuffle=True,
categories=categories, random_state=91)
newsgroups_test = fetch_20newsgroups(subset='test', shuffle=True,
categories=categories, random_state=91)

vectorizer = FeatureHasher(input_type='string')
X_train = vectorizer.fit_transform(newsgroups_train.data)
X_test = vectorizer.fit_transform(newsgroups_test.data)

Y_train = newsgroups_train.target
Y_test = newsgroups_test.target
print(X_train.shape)
print(Y_train.shape)
print(X_test.shape)
print(Y_test.shape)

rf = RandomForestClassifier(n_jobs=-1, n_estimators=100)
rf.fit(X_train, Y_train)
pred = rf.predict(X_test)

score = metrics.accuracy_score(Y_test, pred)
print("accuracy: {:.3f}".format(score))


Clustering vs. Classification I am a bit new to this, but I just had a quick question about clustering vs. classification. I have a bunch of texts that I want to classify. There are 4 classes I have at the moment, but texts can belong to more than one class. What I have seen so far is to do 4 binary classifications, but I was wondering if there is a classification algorithm where I can achieve texts belonging to more than one class. Or could I do this with clustering having overlapping clusters?
I am trying to do this in Python.

Clustering is unsupervised which means that you do not know the classes and / or have no examples of correctly labeled texts.
Assuming you have some labeled texts then we are talking about a classification problem.
Your current approach of doing separate binary models for each label is very basic but still sound. What is your validation metric and score for each of the models? If the performance is already good then you might not need something else.
Another approach would be to train one model that predicts all classes and then to output the multiclass probability prediction. This would give you information about the best fitting classes but you would have to fit your loss metric accordingly, accuracy wouldn't make sense here for example.
Another approach would be to make a new factor variable which encodes all possible combinations of classes and train one model on this. However I suspect that the model would perform badly due to imbalanced cases and a high complexcity.


Starting search from rightmost column and moving left pandas I'm very new to pandas and I'm trying to implement the code to search for something on an excel spreadsheet.

Basically, the most recent price can be found on the furthest right column named "final cost", and that's what I need. However, there are multiple columns named "final cost" (as every invoice is posted on there).

What I currently have:

def find_most_recent(row_num: int, df) -> float:
    k = -1
    checker = False
    while not checker:
        if df.columns[k] == "final cost":
            if not pd.isnull(df.iloc[row_num, k]):
                checker = True
                return df.loc[row_num, df.columns[k]]
        k -= 1

where row_num is the current row that is being searched, and df is the current dataframe being searched.

However, this gets only the first encounter of "final cost". For example, on a dataset of:

I want the return to be one of the prices in column L, but I'm getting returns of values from column F instead.

Any suggestions of a method to use?

Found the answer, putting

if "final cost" in df.columns[k]:

instead of

if df.columns[k] = "final cost":

seemed to have fix it


IS there any way we can add range in hyperparameter tuning of Decision Tree? For example,
           "min_samples_leaf":[1,2,3,4,5,6,7,8,9,10],
           'criterion':['gini','entropy'],
            "max_features":["auto","log2","sqrt",None],
           "max_leaf_nodes":[None,10,20,30,40,50,60,70,80,90] }

but i want to add range of 1 to 100 in max deapth. is there any way i can do it?
Like shown below....
parameters={"max_depth" : range(1,100), ??????????
           "min_samples_leaf":[1,2,3,4,5,6,7,8,9,10],
           'criterion':['gini','entropy'],
            "max_features":["auto","log2","sqrt",None],
           "max_leaf_nodes":[None,10,20,30,40,50,60,70,80,90] }


Anomaly detection for time series data with only positive samples? I'm having a time series ECG dataset. I want to do anomaly detection (anything different from normal ECG should be abnormal).

The point is I'm having only positive samples with very few negative samples.

How to model this problem ? Is it possible to model this as probability distribution and whenever some negative samples comes just taking the divergence from positive distribution ?

If you want to model it as a regression problem, using only the normal ECG data sounds enough to me. In that setting, abnormal data would be only used for validation of your model, but even those few negative samples that you have could be valid for that.

You have a variety of options for getting that divergence, e.g. the Mahalanobis-Taguchi System. You can also model the error coming from your regression model or go fully Bayesian with something like Gaussian Processes, so that you already have a distribution for your posterior. That way, you can cross check new negative samples.


Accessing and Multiplying Individual Elements of a Layer's Output in Keras Problem Description:

I am trying to access the individual elements (i.e., scalar) from a softmax layer's output (with dimension (,2)) and multiply this with a tensor from another model, which has a dimension of (,10). A mock-up digram describing my problem is shown in the attached Figure.

I am using Keras with Tensorflow as the back-end. So far, my approach has been the following: Lets say the output dimension of the softmax layer is (,2) (i.e., a vector of size 2). First, my plan is  to access the individual elements of this vector (tensor according to the Keras/tensorflow) using the keras.backend.gather(a_k_p,0). Where, the variable a_k_p references the SoftmaxLayer.
However, gather simply gives the entire row and does not give the individual element. So, my first question is how to access individual elements of a layer's output?

Assuming I get an answer to the above question, I am describing my idea to multiply a scalar with a tensor as follows. First, replicate this scalar element to create a vector (i.e., a tensor) of size that matches with the DenseLayerA and DenseLayerB (in this case (,10)) and perform the multiplication. Now, I am not sure if this is the right approach since I am unsuccessful in retrieving the individual elements of the Sofmax Layer's output. So, is my approach correct? if not, what is the right way to solve my problem.

This line of code will select the highest value in your softmax function. If you'd like more than one value just change the 1 to whichever value you'd like. I hope this helps.

top_value = tf.nn.in_top_k(logits, y, 1)


Does keras categorical_cross_entropy loss take incorrect classification into account I was looking at keras source here which calculates cross entropy loss using:

output /= tf.reduce_sum(output,
                        reduction_indices=len(output.get_shape()) - 1,
                        keep_dims=True)
# manual computation of crossentropy
epsilon = _to_tensor(_EPSILON, output.dtype.base_dtype)
output = tf.clip_by_value(output, epsilon, 1. - epsilon)
return - tf.reduce_sum(target * tf.log(output),
                       reduction_indices=len(output.get_shape()) - 1)

target is the truth data, which is 0 or 1, and output is the output of the neural net.

So it looks like the loss is of the form

$$J_{y'} (y) = - \sum_{i} y_{i}' \log (y_i)$$

where $y_i$ is the model output for class $i$, and $y_i'$ is the truth data.

Does this mean the errors for $y_i' = 0$ do not contribute to the loss?  Why isn't the formula

$$J_{y'}(y) = - \sum_{i} ({y_i' \log(y_i) + (1-y_i') \log (1-y_i)})$$

used?

Does this mean the errors for $y_i=0$ do not contribute to the loss?

That is correct.
However, the respective weights that connect to wrong neurons will still have gradients due to the error, and those gradients will be influenced by the size of each incorrect classification. That is due to how softmax works:
$$\hat{y}_i = \frac{e^{z_i}}{\sum_j e^{z_j}}$$
(where $z_i$ is the pre-softmax value of each neuron, a.k.a. the logit) . . . weights that affect one neuron's pre-transform value affect the post-transform value of all neurons. So those weights will still be adjusted to produce a lower $z_j$ value for the incorrect neurons during weight updates.

Why isn't the formula
$$J_{y'}(y) = - \sum_{i} ({y_i' \log(y_i) + (1-y_i') \log (1-y_i)})$$
used?

It is not clear why when selecting a single class, that you would care how probability estimates were distributed amongst incorrect classes, or what the benefit would be to drive the incorrect values to be equal. For instance if $y' = [1, 0, 0, 0]$ then using the suggested formula for $J_{y'}(y)$ gives ~ 0.67 for $y = [0.7, 0.1, 0.1, 0.1]$ and ~0.72 for  $y = [0.73, 0.26, 0.05, 0.05]$, yet arguably the second result is better.
However, you would use this loss when dealing with non-exclusive classes (where the outputs would use sigmoid as opposed to softmax activation).


Pre-trained CNN model makes Poor Predictions on Test Images Dataset I have tried using several a pretrained models (MobileNet) for multiclass predictions. There are 42 classes and the distributions of the images are even across the 42 classes.
This is my code:
base_model=MobileNet(weights='imagenet',include_top=False,input_shape = (224,224,3)) #imports the mobilenet model and discards the last 1000 neuron layer.
x=base_model.output
x=GlobalAveragePooling2D()(x)
x=Dense(512,activation='relu')(x) #we add dense layers so that the model can learn more complex functions and classify for better results.
preds=Dense(42,activation='softmax')(x) #final layer with softmax activation
model=Model(inputs=base_model.input,outputs=preds)
for layer in base_model.layers[:20]:
    layer.trainable=False
for layer in base_model.layers[20:]:
    layer.trainable=True

I have freezed and unfreezed some of the trainable layers.
I let it run for 5 epochs:
Epoch 1/5
1318/1318 [==============================] - 3604s 3s/step - loss: 1.5493 - accuracy: 0.5796 - val_loss: 1.7180 - val_accuracy: 0.5361
Epoch 2/5
1318/1318 [==============================] - 3272s 2s/step - loss: 1.2174 - accuracy: 0.6641 - val_loss: 1.7562 - val_accuracy: 0.5372
Epoch 3/5
1318/1318 [==============================] - 3233s 2s/step - loss: 1.0853 - accuracy: 0.6981 - val_loss: 1.2993 - val_accuracy: 0.6498
Epoch 4/5
1318/1318 [==============================] - 3223s 2s/step - loss: 0.9918 - accuracy: 0.7224 - val_loss: 1.3455 - val_accuracy: 0.6382
Epoch 5/5
1318/1318 [==============================] - 3310s 3s/step - loss: 0.9153 - accuracy: 0.7413 - val_loss: 1.2375 - val_accuracy: 0.6660

The accuracy I got is relatively good. I have tried with other pretrained models like Xception too and the accuracies were pretty good. However when I use my model to predict on the test data, and upload my submission to Kaggle, my predictions are very bad.
I am wondering if I have imported the test images and made predictions correctly?
Here is the code:
import cv2
pred_images = []
filename= []

for image_file in os.listdir('./test/test'): #Extracting the file name of the image from Class Label folder
    filename.append(image_file)
    image = cv2.imread('./test/test'+r'/'+image_file) #Reading the image (OpenCV)
    image = cv2.resize(image,(224,224)) #Resize the image, Some images are different sizes. (Resizing is very Important)
    pred_images.append(image)

category = []
for i in range(len(pred_images)):
    pred_image = np.array([pred_images[i]])
    pred = model.predict(pred_image)
    cat = np.argmax(pred, axis = 1)
    category.extend(cat)

I get a data frame looking like this, with a lot from class 38 and 20. Is there something wrong?
final = pd.DataFrame(list(zip(filename, category)), columns = ['filename', 'category'])
final.head()

    filename                               category
0   c94de2fa9b06d67848f648e33a43475c.jpg    38
1   bbb7a2da148488bb878727556aa5914c.jpg    38
2   606256bdf3636d280bfdc3def33a57e7.jpg    38
3   7f741619b952876e7c7c419a0de1ed60.jpg    20
4   05c7592b31ceb8e14d4faa30fa21794c.jpg    20

Kaggle's Dataset is really noisy and it is really tough to get very good result with simple models. You have to do lots of preproccesing ,Data Augmentations and larger network for many more epoch to get a good result.
Things you can do :

Go to notebook section of your competetion and look out for some preprocessing and augmentations and apply them.
Use some regularizer like dropout, mixup,cutmix and many more.

3.Train your model for more epoch

Train a larger model.


Concatenating embedding and hand-designed features for logistic regression An interviewer told me that we cannot concatenate an embedding from a neural network (such as a  pre-trained image representation) and hand designed features (such as image metadata) for use in a linear model such as logistic regression. He says they must only be used with neural networks.
I understand the resulting data may not be linearly separable but that to me seems like a case-by-case empirical question.
Is he right? Can someone explain why?

I think that I may understand where they are coming from. The key difference is if we're talking about inference or training.

Inference

There is nothing stopping you from concatenating your features. Whether they are hand-designed or a neural network's output, features can be combined however you please.

Training

If you don't intend to update your embeddings through training, then, there is no problem. Your embeddings will be treated as some other features and you can train your downstream model.
It gets tricky if you want to train both your downstream model and your embedding layer. Indeed, you will only be able to train your embedding layer if your full model (NN + downstream model) is differentiable. This is for instance not possible if the downstream model is a KNN model.
I assume this is what they meant when they said you couldn't concate embeddings and hand-designed features.


Question on bias-variance tradeoff and means of optimization So I was wondering how does one, for example, can best optimize the model they are trying to build when confronted with issues presented by high bias or high variance. Now, of course, you can play with the regularization parameter to get to a satisfying end, but I was wondering whether it is possible to do this without relying on regularization.

If b is the bias estimator of a model and v of its variance, wouldn't it make sense to try to minimize b*v?

There are a lot of ways bias and variance can be minimized and despite the popular saying it isn't always a tradeoff.

The two main reasons for high bias are insufficient model capacity and underfitting because the training phase wasn't complete. For example, if you have a very complex problem to solve (e.g. image recognition) and you use a model of low capacity (e.g. linear regression) this model would have high bias as a result of the model not being able of grasp the complexity of the problem.

The main reason for high variance is overfitting on the training set.

That being said there are ways of reducing both bias and variance on a ML model. For example the easiest way of achieving this is getting more data (in some cases even synthetic data help).

What we tend to do in practice is:

First, we increase the capacity of the model in order to reduce the variance on the training set as much as possible. In other words we want to make the model overfit (even reach a loss of 0 on the training set). This is done because we want to make sure the model has the capacity of sufficiently understanding the data.
Then we try to reduce the bias. This is done through regularization (early stopping, norm penalties, dropout, etc.)


predict future value in every one hour using (t+60 minutes) LSTM neural network in python I have a data csv file including with three inputs and two output with time series. Here data took an every one hour one hour. So I need to predict my next future value at t+60 according to the previous input value and at that time period if having new input value using regression neural network. So I choose LSTM neural network to predict next future value. But I don't know how to give time period to predict my future value.
Can anyone suggest me how to solve this problem?
Can anyone give me any examples to clear out this problem?
Here that prediction value will come as input value (g).

subset of my csv file

here I upload my code;

def create_data(data, look_back=1):
dataX, dataY = [], []
for i in range(len(data) - look_back - 1):
    a = data[i:(i + look_back), :]
    dataX.append(a)
    dataY.append(data[i + look_back, 2])
return numpy.array(dataX), numpy.array(dataY)

data = pd.DataFrame(data,columns=['g','p','c'])
numpy.random.seed(7)
data = data.values
scaler = MinMaxScaler(feature_range=(0, 1))
data = scaler.fit_transform(data)
train_size = int(len(data) * 0.67)
test_size = len(data) - train_size
train, test = data[0:train_size, :], data[train_size:len(data), :]
# reshape into X=t and Y=t+1
look_back = 3
trainX, trainY = create_data(train, look_back)
testX, testY = create_data(test, look_back)
# reshape input to be  [samples, time steps, features]
trainX = numpy.reshape(trainX, (trainX.shape[0], look_back, 3))
testX = numpy.reshape(testX, (testX.shape[0],look_back, 3))
model = Sequential()
model.add(LSTM(6, input_shape=(look_back,3)))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')
history= model.fit(trainX, trainY,validation_split=0.33, nb_epoch=10,
batch_size=30)
trainPredict = model.predict(trainX)
testPredict = model.predict(testX)
trainPredict_extended = numpy.zeros((len(trainPredict),3))
trainPredict_extended[:,2] = trainPredict[:,0]
trainPredict = scaler.inverse_transform(trainPredict_extended) [:,2]
print(trainPredict)
testPredict_extended = numpy.zeros((len(testPredict),3))
testPredict_extended[:,2] = testPredict[:,0]
testPredict = scaler.inverse_transform(testPredict_extended)[:,2]
trainY_extended = numpy.zeros((len(trainY),3))
trainY_extended[:,2]=trainY
trainY=scaler.inverse_transform(trainY_extended)[:,2]
testY_extended = numpy.zeros((len(testY),3))
testY_extended[:,2]=testY
testY=scaler.inverse_transform(testY_extended)[:,2]
trainScore = math.sqrt(mean_squared_error(trainY, trainPredict))
print('Train Score: %.2f RMSE' % (trainScore))
testScore = math.sqrt(mean_squared_error(testY, testPredict))
print('Test Score: %.2f RMSE' % (testScore))
# shift train predictions for plotting
trainPredictPlot = numpy.empty_like(data)
trainPredictPlot[:, :] = numpy.nan
trainPredictPlot[look_back:len(trainPredict)+look_back, 2] = trainPredict

# shift test predictions for plotting
testPredictPlot = numpy.empty_like(data)
testPredictPlot[:, :] = numpy.nan
testPredictPlot[len(trainPredict)+(look_back*2)+1:len(data)-1, 2] =
testPredict

One option is to :

Break data into constant-frequency observations (E.g.: Assume that g,p,c,out are same for all time periods between two observations). With this, you will get samples every N minutes.
Training data will then be a set of [Last M observations till T, Observation for T + 60]

This Train and test data can be fed into a network.


Time series prediction with Lstm on patients data My data includes different time series length (depends on number of exams each patient did) as well the interval between exams is different.
How can I run lstm on this kind of data? (Where the sequence length is different)
How should I build the batch?
Should each batch hold a patient time series data?
Where do I put the target value?

LSTM seeks inputs having the same shape. Hence, you may first consider defining a sequence length and then padding the sequences whose lengths are less than the defined sequence length.


Evaluation of regression models with different evaluations (MSE, variance, VAF etc.) When comparing several regression models in terms of quality, it seems like most have agreed on the MSE.
There are also papers comparing "variance" and "variance accounted for (VAF)".
However, there seems to be a controversial opinion about the variance (R^2). Should it nevertheless be compared in a scientific paper?

And what does VAF say? Is the VAF still a good measure of regression models?

Variability accounted for (VAF) is not meaningful in regression.
VAF can be driven higher by the distribution of the independent variables which does tell you nothing about how good a regression model is.


Is it possible to use grayscale images to existing model? From tensorflow's object recognition (R-CNN)

I'm re-training the existing model with new categories: the types of clothes (jeans, pants, blouse, and so on). Since we don't need colors to determine the type of clothes that user is wearing, I want to re-train it with gray-scale images. Is it possible to use gray-scale images to train existing model (which are trained with color images)?

I'm concerned because they trained their model with color images.

Does the model just consider the grayscale image as color image? And does it still work? :)

p.s I'm generating XML and csv files to put data for training and testing.

It depends on the model. You'll have to dig into the model definition and see.

It's not uncommon for models to grayscale images in pre-processing. In that case, you will be fine.

However, many recent deep convolutional models operate on RGB images as 3D tensors (4D with the batch dimension). In this case, you should consider modifying the model so that it operates on a single color channel before training it. Multi-channel convolution is discussed in the convent's chapter of Goodfellow's Deep Learning.


What does the "randomly shuffle training samples" in stochastic gradient descent attain? What does the "randomly shuffle training samples" in stochastic gradient descent attain?

I interpreted that since the training samples are used to compute

$$\hat{y}=f(w^t x)$$

so if the order of $x$s changes, then the weights will be assigned "based on different order"?

Although, since $w^tx$ is linear (order doesn't matter), then where is the effect of this seen?

Or maybe it's not seen in $\hat{y}$ but in the LMS update rule:

$$\Delta w_{ij}^k=\lambda(\hat{y}_i^k-y_i^k)\color{red}{x^î_j} $$

If you are not training with minibatches, but just one batch per epoch, then random shuffle does nothing. However, if you are training with minibatches, and the data in the first minibatch is related and vastly different from the last minibatch, the error that is back propagated from the first could be totally the opposite of the error in the final.

By shuffling the records, the minibatches as presumably more representative of the entire training data and so the back-propagation of the errors will be, hopefully, making slow but stead progress towards the global minima, not whipsawing back and forth chasing what would essentially be a class based minima.


Import Orange 2.7 canvas in Orange 3 I have a saved canvas I created in Orange 2.7. I just installed Orange 3 and opening the old canvas gives multiple "UnknownWidgetDefinition" errors. I gather that the naming/location of some widgets has changed between versions. Is there a way to import the old canvas?

Not only the naming has changed, also the individual widget settings and layouts are completely different. Some Orange 2 widgets are not even available in Orange 3 yet, or have been consolidated into other ones. I'm afraid Orange 2 workflows are just not compatible with Orange 3.


Convert date into number - Apache PIG Imagine that I've a field called date in this format: "yyyy-mm-dd" and I want to convert to number like "yyymmdd". For that I'm trying to use this:

Data_ID = FOREACH File GENERATE CONCAT((chararray)SUBSTRING(Date,0,4),(chararray)SUBSTRING(Date,6,2),(chararray)SUBSTRING(Date,9,2));

But I'm getting a list of nulls...

Anyone knows what I'm doing wrong?

Thnaks!

I don't use PIG, but from looking online I think you should be able to use the Replace function built into PIG.

Assuming your Date field is already a string:

Data_ID = FOREACH File GENERATE(int) REPLACE(Date,'-','');

Here's the reference if it's helpful: https://pig.apache.org/docs/r0.9.1/func.html#replace


Extract features from a survey I need to use the answers from a questionnaire for training a classifier.
I discovered that some questions can have nested sub-questions..
Let's say (just an example) that I want to predict whether a person is going to buy a house based on the following questions:

1) What is your gender?
[] male
[x] female
[] I prefer not to answer

in the case the answer is female (as in the example above) a sub-question is ansked

1_female) are you pregnant?
[x] yes
[] no

Then the questionnaire continues..

How should I use these features to train my model?

Option 1)
Treat them separately and transform them with one-hot-encoding
I will have then the feature vector

gender_male - gender_female - gender_not_answered - pregnant_empty - pregnant_yes - pregnant_no
     0      -        1      -         0           -        0       -       1      -        0

Obviously the feature pregnant_empty will be coded with 1 for all the males

Option 2)
Merge the 2 answers and encoding the concatenation

gender_female_pregnant_yes - gender_female_pregnant_not - gender_male - gender_not_answered
     1                     -                 0          -      0      -     0

Other options?

Please treat this just as an example... the problem is that in a real scenario

the nested question could appear with 2 or more answers
expanding the features as in option 2 will make my feature vector explode..

I hope my question was clear enough

The simplest is to keep your features separate and add a synthetic feature, feature cross, that captures the relationship between those features you mention that can possibly be nested.

For example, in a neuronal network based classifier (e.g., TensorFlow), the model will learn the 'correct' weight for those combinations of features' values that are impossible to happen (e.g., male and pregnant), excluding wrong data cases obviously.

In the end... you just want the cartesian product among the features that you need to 'cross'. And yes, your vector will grow.


Using NLP to automate the categorization of user description I have a huge file of customer complaints about the products my company owns and I would like to do a data analysis on those descriptions and tag a category to each of them.

For example: I need to figure out the number of complaints on Software and Hardware side of my product from the customer complaints. Currently, I am using excel to do the data analysis which do seek a significant amount of manual work to get a tag name to the complaints.

Is there a way in NLP to build and train a model to automate this process? I have been reading stuffs about NLP for the past couple of days and it looks like NLP has a lot of good features to get a head start in addressing this issue. Could someone please guide me with the way I should use NLP to address this issue?

One way to handle this is to use 'supervised classification'. In this model, you manually classify a subset of the data and use it to train your algorithm. Then, you feed the remaining data into your software to classify it.
This is accomplished with NLTK for Python (nltk.org).
If you are simply looking for strings like "hardware" and "software", this is a simple use case, and you will likely get decent results using a 'feature extractor', which informs your classifier which phrases in the document are relevant.
While it's possible to implement an automated method for finding the keywords, it sounds like you have a list in mind already, so you can skip that step and just use the tags you are aware of. (If your results aren't satisfactory the first time, this is something you might try later on).
That's an overview for getting started. If you are unhappy with the initial results, you can refine your classifier by introducing more complex methods, such as sentence segmentation, identification of dialogue act types, and decision trees. The sky is the limit (or more likely, your time is the limit)!
More info here.


problems with installing tensorflow using anaconda I am trying to use anaconda 4.5.12 to install tensorflow on an Ubuntu 18.04.1 LTS system with python 3.7.1 and pycharm 2018.3.3.

I issue the following terminal command:

conda install tensorflow

And this gives me:

Solving environment: /\-

(It keeps switching between /-). This just continues indefinitely. I waited for like 20 minutes at one point. Why is this happening?

Also, I am confused by the explanation on this anaconda site, which gives the following terminal command:

conda create -n tensorflow_env tensorflow
conda activate tensorflow_env

What does this command do, and why would we use this rather than "conda install tensorflow"? I assume it creates a new environment but why do we need a new environment to install tensorflow? Can't we just install tensorflow for all environments? This seems messy, because I think it also means that I can not create an environment in PyCharm directly.

I can't comment because I'm new, so I put this as an answer but really should be a comment. I have come across this link that might help you:

https://towardsdatascience.com/tensorflow-gpu-installation-made-easy-use-conda-instead-of-pip-52e5249374bc

They say to use this command to install tensorflow with conda even on Ubuntu:

conda create --name tf_gpu tensorflow-gpu

Hope it helps!


Machine learning Classification model for binary input and output data I have a large longitudinal dataset with 5 minutes granularity for a period of around 30 months from thousands of households. I want to classify them as binary output (0/1) based on the input which is also a set of binary variables (sensors activated or not- 0/1. I have a training dataset available with the labeled binary output(0/1) with binary inputs.

I want to know which Machine learning model will be best for this type of case where both input and outputs are binary in nature.

Whether Logistic regression is one of the options or not?

Your problem is one of "sequence classification" for which Recurrent Neural Networks (RNN) e.g. Long short-term memory (LSTM) are generally used.

See here for a good example: https://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/

and here for a technical paper: https://www.cs.sfu.ca/~jpei/publications/Sequence%20Classification.pdf

Here is a specialized package for sequence classification which uses convolutional neural network (CNN): https://github.com/aajanki/sequence-classification

CPT algorithm, an accurate method for sequence prediction, can also be used here: https://www.analyticsvidhya.com/blog/2018/04/guide-sequence-prediction-using-compact-prediction-tree-python/ A continous output can easily be rounded to 0 or 1 to get binary result.


Classification of very similar images I have two groups of images, each one with 1000 samples.

The speckle pattern, in this context, is the same as a random pattern or "white noise" image. So these images are fundamentally different.

In group one, each figure is generated by considering a random function that returns something similar to a speckle pattern (see fig. 1).
In group two we follow the same procedure as group 1, but we plot a small point above that can be positioned anywhere and with any color (see fig. 2).

I want to classify both groups and I already tried to do it with simple neural networks, but I have been unsuccessful.

What is the best technique for this kind of problems?

Fig. 1:

Fig. 2:

Have you tried rule-based approaches?

Based on your example, I can think of two ways:

For each picture, get a list of all RGB values it contains.
The first contains only grays (which have RGB values (x, x, x)), the second one contains color (so some pixel with RGB value (x, y, z), where $x \neq y$ or $y \neq z$). Or - if that is faster - convert all pictures to gray scale and check if the converted picture is identical to the original.
Scan the image for areas (larger than one pixel) with identical colors. If your dots are large enough and the background noise is uncorrelated for neighboring pixels, it is very likely that these areas are the dots you are looking for.


It is helpful to normalize target variables for a regression neural network? It is customary to normalize feature variables and this normally does increase the performance of a neural network in particular a CNN. I was wondering if normalizing the target could also help increase performance? I did not notice an increase in performance with the data set I am using at the moment but was curious if anyone has tried in the past. Of course the normalization happens only on the training data.

One reason for normalising the inputs is to make gradient descent more stable, as gradients spend more time in a comfortable region with meaningful updates and less neurons 'die' during trainings - getting stuck at one of the tails of e.g. the sigmoid non-linearity.

Normalising the output distribution is perhaps not the best idea, as you are by definition altering the defition of the target. This means you are essentially predicting a distribution that doesn't mirror your real-world target (at least without some reverse non-linear transforms later on).

On this you could do would be to scale the target, instead of normalising. The shape of the distribution should remain almost identical (thinking about the shape of the distribution), but the values themselves might be more easily attainable and therefore faster to optimise for; they are all closer in magnitude to the gradients that are being computed.


Dividing data set into (almost) equal batches based on characteristics I have a large dataset containing 6 characteristics (all numerical).
I need to split this dataset into multiple batches to be processed in parallel, and ideally, the batches should be as equal in size as possible.
The catch is: I can only split based on the 6 characteristics, so I have to specify ranges of values for each batch.
Simplified example: [1,1,1,2,2,3,3,3,3,4] will be split in two by specifying numbers 1-2 go to batch 1 and 3-4 go to batch two and I end up with [1,1,1,2,2] and [3,3,3,3,4].
I managed to make a simple division algorithm based on one characteristic, unfortunately some of the characteristics might not be suitable to split on, and I have no way to reliably predict which characteristic will create a good split.
I remember from Uni that a classification algorithm might do the trick here, but I am not sure how to implement the requirement for similar size classes.
here is the simple code I wrote for one characteristic, it may help to understand the situation:
def divide_into_groups(lst: list, num_groups=2):
    length = len(lst)
    sorted_list = sorted(lst)
    optimal_group_size = length // num_groups

    initial_group_markers = [sorted_list[i * optimal_group_size] for i in range(1, num_groups)]
    marker_edges = [(sorted_list.index(marker), length - 1 - sorted_list[::-1].index(marker))
                    for marker in initial_group_markers]

    final_group_markers = []
    for i, marker in enumerate(initial_group_markers):
        if i - marker_edges[i][0] < marker_edges[i][1] - i:
            final_group_markers.append(marker - 1)
        else:
            final_group_markers.append(marker)

    distances = [abs(initial_group_markers[i] - final_group_markers[i]) for i in range(len(initial_group_markers))]
    efficiency = (length - sum(distances)) / length
    print('distances from optimals: %s' % distances)
    print('efficiency of division is %s %%' % (efficiency * 100))

    return final_group_markers

P.S. this is my first question in this forum. If I am missing some information or have made any mistake, please comment and I will fix as soon as possible.

If I got this right you want to split the dataset into train and test sets in a way that preserves the same proportions of examples in each class as observed in the original dataset?
This is called a stratified train-test split. See the stratify argument here sklearn split


How to implement feature selection for categorical variables (especially with many categories)? I've been trying to get some ideas of how I could treat categorical variables when doing feature selection. Mainly I've been running Random Forest feature importance on Python for which preprocessing could play a big part.

For some variables which come with a large number of unique values such as State, Zip Codes, Cities...one hot encoding may not make sense since it explodes your vector length. Another issue is also how representative that category would be. Eg: If you have oly one observation from MI, does it really make sense to encode a feature for this state? What is considered the most appropriate way to treat variables like this?

Some algorithms will perform feature selection inherently - e.g. elastic net regression, random forest - so you will not necessarily need to do this prior to running the algorithm.

You've identified one issue with one-hot encoding - it may create exceptionally wide data sets. Another issue is that a feature selection technique based on one-hot encoded variables will identify important feature levels and not important features. I would argue you might want to keep your features "together" and not evaluate individual levels.

Have you considered some basic statistical tests? Sometimes simple methods are just as useful as complex approaches. Useful tests include ANOVA (continuous response, categorical predictors) and Chi-squared / Cramer's V tests of independence (categorical response, categorical predictors).


Choosing attributes for k-means clustering The k-means clustering tries to minimize the within-cluster scatter and maximizing the distances between clusters. It does so on all attributes.
I am learning about this method on several datasets. To illustrate, in one the datasets countries are compared based on attributes related to their Human development Index. However some of the attributes are completely unrelated to this dimension, for example total population of countries. How to deal with this attributes? As mentioned before k-means tries to minimize the scatter based on all attributes, which would mean this additional attributes could hurt the clusters. To illustrate, I know the k-means cannot discern three clusters that are perfectly clustered around one dimension and are completely scattered around the other.
Should one just exclude some attributes based on prior knowledge? Is their perhaps a processes that identifies irrelevant attributes.

First of all, if you know that certain attributes shouldn't after the clusters, you should remove them altogether. There is no point in hoping that K-Means will figure it out on its own if that can be fixed upstream.
Second, obviously, not every attribute should affect the clusters equally. K-Means is based on the concept of distances between your points. Based on the distance matrix, the algorithm will find different clusters. The good thing is that you can tweak how the distance is calculated. You could weigh the different attributes such that differences between certain attributes are more important than others.
Third, if you want to programmatically find the "best" attributes for clustering, I don't know of any efficient ways to do it. Meaning that your best bet is to try different combinations of attributes and see how good the clustering becomes. To rate the quality of clustering, there exist metrics like the Dunn Index, or the Davies-Bouldin Index (see this link for more detailed information: https://medium.com/@ODSC/assessment-metrics-for-clustering-algorithms-4a902e00d92d)


Fasttext error while loading wiki pre-trained data I am loading the model using gensim package this way:

from gensim.models import FastText
model = FastText.load_fasttext_format('wiki-news-300d-1M-subword.bin')

as stated here.

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe6 in position 57: unexpected end of data

The .bin file is downloaded from this source.

How to load the model correctly?

You could use the load_model method instead of load_fasstext_format method. Also, you could FastText Library directly without really having to install gensim to do the same.


Features selection in KNN I have a naive question about using the K Nearest Neighbor algorithm: is feature selection more important in KNN than in other algorithms?

If a particular feature is not predictive in a neural network, the network will just learn to ignore it. But in KNN, it seems like it could make the prediction worse, right? If I'm predicting height based on weight and age and gender, my model will get worse if I now add house numbers, because people will similar house numbers to me will be closer to me.

In a less extreme example, what if a feature is weakly predictive? Rather than normalize all my features so they have an equal weight, wouldn't I want to make the highly predictive features have more weight than less predictive ones?

I'll go through your questions one by one:

  is feature selection more important in KNN than in other algorithms?

I don't think it is more important for kNN than for other kinds of algorithms.

  If a particular feature is not predictive in a neural network, the network will just learn to ignore it. But in KNN, it seems like it could make the prediction worse, right?

Correct. Neural Networks are "smarter" algorithms, they have internal weights that adjust to minimize a cost function. Less important features will be attributed comparatively lower importance with respect to highly predictive variables. This doesn't happen in kNN, in which prediction is based exclusively on distance between datapoints - and no information about relative importance of variables can be deduced from it.

  In a less extreme example, what if a feature is weakly predictive? Rather than normalize all my features so they have an equal weight, wouldn't I want to make the highly predictive features have more weight than less predictive ones?

sklearn allows to manipulate kNN weights. But this weights distribution is not endogenous to the model (such as for Neural Networks, that learn that autonomously) but exogenous, i.e. you have to specify them, or find some methodology to attribute these weights a priori, before running your kNN algorithm. If you can do that, and you have good methodological reasons for it, then changing variables' weights can improve your model, but I'd be careful about it (you can easily overfit).


Feature Engineering Lists\Vectors as values in dataframe Let's say I have a dataframe where some of the columns have lists of strings as values. I would like to use ML Algorithms on this dataframe.

In this case, I can:

I could add many columns of 1's and 0's for each string that may appear in some list, which seems to me terribly inefficient as some of the items in the lists only appear very few times. If there were maximum of 10 different items that could be listed in all of the lists , I guess it's quite all right to use/
I know some use SVD in these cases (even though I don't completely understand it yet.)
could put inside the dataframe pandas series as values like some kind of vectors.

So I ask:

what is the best way to feature engineer it assuming there are few items that could appear in the lists?
what is the best way to feature engineer it assuming there are lots of items that could appear in the lists?
Is there a certain way to do it that runs most efficiently with the popular ML\DL python packages?

I think a good starting point is what you have mentioned. For every element in a list, create a feature for that element. If that element is present in the list for a data point, then the element is denoted as a 1 in your feature vector. If it is not present in the list, then it is denoted by 0. This is very much a bag-of-words type way to create features. You could limit the number of features by only taking the top k occurring elements, where you determine k. Another variant is using counts of frequencies, if an element appears multiple times in a list, but it doesn't sound like that is the case.


(Feature selection) In which cases it is legitimate to remove features manually? I am dealing with the feature in which only one category takes up about 90%, the instances of more than 30 other categories are sparse.

Is it reasonable to remove this feature before building an estimator?
I've experimented with the case when including and excluding this feature, and the later case showed a slightly(very slightly...) better performance.
I am wondering if there is any machine learning principle, such as we should avoid 'human learning' and just let 'machine learning' do. What would be the best way to deal with this sort of features? I am currently using Scikit-learn.

I am a total beginner in data science, so any opinion will be valuable to me. :)

Is it reasonable to remove this feature before building an estimator? I've experimented with the case when including and excluding this feature, and the later case showed a slightly(very slightly...) better performance.

This, ultimately, is the best way to get an answer. Trust in cross-validation; if you're doing it robustly and it says that ditching the feature improves performance, then ditch the feature.

  I am wondering if there is any machine learning principle, such as we should avoid 'human learning' and just let 'machine learning' do. What would be the best way to deal with this sort of features? I am currently using Scikit-learn.

Nope, human selection of features plays a part too. Some algorithms are resistant to junk features and can effectively ignore them, but they always present the potential for overfitting. The best way to deal with them is what you've done; try the model both with and without and see which performs better.


At what stage are ROC curves used when building machine learning model? When developing a machine learning model, at what stage are ROC curve with AUC used?
Typically I have three data sets
train - validation - final test
I do K-Fold cross validation using the combined train + validation set
During that phase we can calculate the metrics including true positives, false positives as well as other metrics and average them to create a plot like the ROC curve.  Similar to this example from scikit-learn
However we can also get the metrics at the end by training the final model using all the data from train + validation and testing on the test set This can also give us all the metrics, classification report and ROC curve etc.
My question is, do people generally do the ROC curves twice, once during cross validation and then a second time for the final testing?  OR is it something that is used only during validation phase / hyper parameter tuning when selecting the algorithm?

The ROC-AUC curves are used to find the best threshold that optimizes True Positive Rate vs False Positive Rate. Using it in a K-Fold cross-validation is a good practice to determine the best threshold to use.
Then, your final test is here to validate that you did not overfit on some hyperparameters, including this threshold. So ROC-AUC must not be used again in final test. You should compare the results of your final test with the same threshold used in your cross-validation.
Hope it helps.
Note on threshold (EDIT):
The threshold to optimize could be the threshold to use in a binary classification problem that outputs probabilities (for instance, output of a sigmoid or a logistic regression). In that case, various threshold settings gives various the model's predictions (FPR, TPR), and so is built the ROC curve.
You could read further on sklearn guide page.


Problem in creating a Numpy Dataset for Classification problem I am trying to train a classification Model to classify Images, but my images here are Numpy arrays of 1's and 0's as shown in the Image,

For now, I have tried,

x1_data=[]
def create_array(columns=5,rows=5,randomness=.3):

    board = np.zeros([rows,columns],dtype='int64')

    for i in range(rows):
        for j in range(columns):
            if np.random.random() <= randomness:
                board[i,j] = 1
    return board

for i in range(5):
    x1_data.append(create_array())

which gives me 5 arrays `

array([[[0, 0, 1, 0, 0],
        [0, 0, 1, 0, 0],
        [1, 0, 0, 0, 0],
        [0, 0, 0, 0, 1],
        [0, 0, 0, 0, 1]],

       [[0, 0, 1, 0, 0],
        [0, 0, 0, 1, 0],
        [0, 0, 0, 1, 0],
        [0, 0, 0, 1, 0],
        [0, 1, 0, 0, 0]],

       [[1, 0, 0, 1, 0],
        [1, 0, 0, 0, 0],
        [0, 1, 0, 0, 0],
        [0, 0, 0, 0, 0],
        [0, 0, 1, 0, 0]],

       [[0, 0, 0, 0, 0],
        [1, 0, 0, 1, 1],
        [0, 0, 1, 1, 1],
        [0, 0, 0, 0, 0],
        [0, 0, 1, 0, 1]],

       [[0, 0, 0, 0, 1],
        [0, 0, 0, 0, 0],
        [0, 0, 0, 1, 0],
        [0, 1, 1, 0, 0],
        [0, 0, 0, 0, 1]],

       [[1, 0, 0, 1, 1],
        [0, 0, 1, 1, 0],
        [0, 0, 1, 0, 0],
        [1, 0, 0, 0, 0],
        [1, 1, 0, 0, 0]]]

For class A, similarly I generated 5 arrays for class B and then stacked them as a single X array.

My output y is:

y = np.random.choice([0, 1], size=(10,), p=[1./3, 2./3])
array([0, 1, 1, 1, 1, 0, 0, 0, 0, 0])

How do I preprocess this Data to train on an MLP in sklearn
I am getting this error

clf = MLPClassifier(solver='lbfgs', alpha=1e-5, hidden_layer_sizes=(5, 2), random_state=1)
clf.fit(g, h)
#Error
Found array with dim 3. Estimator expected <= 2.

If you're going to use an MLP classifier (probably not the best option for image classification: why not CNN?), you should feed it a "flat" array. Each image has 25 pixels (features), so your input must be of shape (n, 25), where n is the number of training observations (10 in your case).

Currently your input is of shape (10, 5, 5) (dimension 3, hence the error message). Try reshaping it like this: X = X.reshape(-1, 25).


How useful is Bayesian Inference Last few months, I had been exposed to Bayesian Inference in ML course
With further investigation, I come to place where there is MCMC technique to simulate the posterior distribution.
It seems interesting. However, I am not sure if it is really useful in the industry?
Does anyone has experience with Bayesian Inference in practical?
Take customer_lifetime_value as X for example
Basically, my main question is how bayesian inference can be more useful than I just plotting frequency and cumulative frequency of historical X
Because with frequency , i can estimate the mean X,
With cumulative frequency, i can estimate prob[X>x]
What is the advantage of trying to do bayesian inference

This may be an unpopular opinion to some, but in my experience Bayesian statistics is not particularly useful in data science in industry, for a couple of reasons:

A Bayesian approach is very useful when our questions are about statistical inference. However, in data science, more often than not, we are dealing with prediction. There may be some situations where a Bayesian approach works better than a frequentist approach, but I can't think of any off hand, apart from where a conjugate prior is available, in which case we are probably dealing with a very simple model.

Bayesian statistics usually requires a sampling, such as Markov Chain or Hamiltonian Monte Carlo, and this can be extremely computationally intensive, even for relatively small datasets. In industry we are often dealing with "big data" and a Bayesian model that requires MCMC or HMC just isn't practical.

Edit: To address a comment to this answer:

I have a question about this, how statistical inference is different from prediction. As for my understanding, prediction is usually about getting P(Y=y | X), where X is our data, which is similar with statistical inference

Prediction and inference are completely different. With prediction, all we really care about is the accuracy of the predictions, and that is a relatively easy thing to determine - because the test/validation procedure is minimising some measure of prediction error. On the other hand, with inference, we care about the coefficient estimates and often their standard errors. Often a researcher forms a hypothesis based on a theory of causes - eg, does drinking coffee cause cancer, and they will use a model to determine whether or not the data supports their theory, and to what extent.  This is much, much more difficult than prediction. For one thing, in causal inference the basic requrement is that we do not want the estimates for $\mathbf{X}$ to be biased - whereas in prediction we don't care if they are biased, provided that we get "good" predictions for $\mathbf{y}$, and there are many, many sources of bias in a regression model - bias can and does arise from confounding, mediation, differential selection and colliders. Usually, the crux of the issue is to decide which variables to include in the model in order to eliminate or reduce these biases. With prediction we can use automated variable selection procedure in order to choose which variables to include (feature selection). With inference this is pretty much impossible because automated procedures cannot generally handle the above mentioned biases.
There is a detailed discussion of these issues in my answer to this question:
How Do DAGs Help To Reduce Bias In Causal Inference ?


What is the meaning of an empty SHAP graph in Explainable AI? Using Python, I created a neural network to perform predictions on a binary class dataset (e.g. will a passenger survive the Titanic?). I am using the SHAP package to explain individual predictions. For all of the instances in this dataset, the visualization outputted by SHAP has an output value of 0 and the higher/lower graph is empty (there are no features listed).

shap.force_plot(k_explainer.expected_value[0], k_shap_values[0], label_test_X.iloc[0])

When I use a different dataset and run the line above, SHAP outputs a graph and lists features.

What does an empty SHAP graph mean, especially in this case where there is an empty graph for each instance of a dataset? I am reading the documentation to help understand this situation, but I appreciate any insight into this.

Thank you in advance for the help!

that means shap values were nan.Model was not able to compute shap values.


Function extrapolation I have a list
[1.0, 0.488, 0.300, 0.213, 0.163, 0.127]

Plot (dont have enough reputation to post an image)
I need to extrapolate this function for 15 more points further, asymptote is 0.
I figured out that maybe I need a custom kernel for SVR (something like y=1/x), but I havent found any examples on how to implement it.
I will be very grateful for any examples/links/or tips

Your data looks logarithmic. Try using the scipy.optimize curve_fit() function to find the approximate log coefficients. I tried several built-in python functions, but couldn't get a good fit on any of them, but they can be used as a starting point.
Finally, I ran the curve_fit() function on the data you posted, and after making a few adjustments, I was able to produce the green line on this chart. The blue line is your original data. They are so close together that it is difficult to see the difference. The yellow line is the one that the curve_fit gave me for a best fit.
I also included the extrapolated points (green line again) and the values for the next 15 points below. As you can see, the line is asymptotic to the x axis.

FYI, My adjustments included adding a np.log() to the denominator to convert the base of the log. Also, I changed the intercept to 1 as that is where your data starts.  If you want to automate this, you could write some functions to iterate over different values of the log base and the intercept. I included comments in the code for more details.
import matplotlib.pyplot as plt  # Matlab-style plotting
import numpy as np
import scipy
from scipy.optimize import curve_fit
x = [1.0,2.0,3.0,4.0,5.0,6.0]
x2 = np.arange(0, 22.0, 1.0)
y = [1.0, 0.488, 0.300, 0.213, 0.163, 0.127]
x = np.array(x)
y = np.array(y)

plt.plot(x,y)
# get the approximate values for the log function
data = scipy.optimize.curve_fit(lambda t,a,b: a+b*np.log(t),  x,  y, p0=(1, 1))
# plot those values, the fit is not very good
y1 = data[0][0] + data[0][1]*np.log(x) # yellow
plt.plot(x,y1)
# original coefficients were .91 and -.48
# adjust the 2 coefficents until you get a better fit
# first coefficient is like the intercept but for the line of x of 1, not the y axis
# I also changed the log base value by including the np.log() on the denominator,
# this gives the line a larger curve than the original
y2 = 1.0 - np.log(x2)/np.log(x2+1.9) # green
plt.plot(x2,y2)
plt.show

data

(array([ 0.91083753, -0.48242954]),
 array([[ 0.00543936, -0.00380311],
        [-0.00380311,  0.00346828]]))

y2

array([       inf, 1.        , 0.49069866, 0.30871637, 0.21896813,
       0.16675119, 0.13310187, 0.10985155, 0.09295092, 0.08018302,
       0.0702405 , 0.06230656, 0.05584666, 0.05049745, 0.04600401,
       0.04218259, 0.03889766, 0.03604721, 0.03355312, 0.03135462,
       0.02940378, 0.02766231])