Binary Classifier making only one prediction tl;dr I'm building a binary classifier that always eventually predicts all "0" or all "1" after some number of epochs and I'm looking for possible reasons why/how to proceed.

Below is all just more details on my approach and thoughts:

What I'm doing: I am building a binary classification basic feed-forward neural network for signal processing with tensorflow.

I pre-process my data by separating it into "windows" of length window_size points (say like 50), and that window/example can be considered either a "1" (positive) or a "0" (negative). The official, manually annotated labels are located at specific points (e.g. 403, 875, 1450, etc.) and are relatively evenly spaced. I decided that my example/window is positive if the window contains a label, negative otherwise. Multiple examples are generated from moving the window by some stride_length (say 1, 2, or 10 points, etc.), until my moving window reaches the end of the signal.

For the training data, since most of my examples are negative, I've tried to normalize the ratio of neg to pos examples so it's close to 1:1. I do this by deleting a calculated ratio of negative examples. For test data I skip this last step, but everything else is the same.

I run it through a 3 layer NN with RELU's on hidden layers and sigmoid on the sole output, with <0.5 predicting a negative and >0.5 predicting a positive. Cost is sum of (label - prediction)^2, using AdamOptimizer at default LR (0.001).

My issue:
No matter what parameters I adjust, my NN seems to predict everything as either negative or positive after a certain number of epochs (sometimes even immediately at the first epoch). It's pretty random too. I'll run the same thing multiple times and I'll get something different every time, about half of the predicting all positive, and the other half all negative.

Things I have tried/thought about:

I shuffled my training examples before training my model, with not much improvement.
I cut down on my negative examples as I stated above, since I had this problem before I cut down (was predicting basically all negative immediately). But now that I have close to a 1:1 ratio of pos/neg examples, it's flip-flopping between predicting all negative and all positive, so it's extremely sensitive to this ratio.
I originally had a softmax_cross_entropy_with_logits cost function with one-hot encoding, so 0 1 was positive and 1 0 was negative. (I thought maybe the cost function was rewarding extreme predictions, so I switched to sum of squares which is harsher)
I'm thinking of trying batch normalization and dropout, but I don't expect those to fix my problem.
I think the problem might be in how I am pre-processing the data. My thought process is this: if the ratio of pos/neg examples is so important that it is making my model predict all positive or all negative, then maybe there is no real trend in my data (which is wrong, since anyone with eyes can see the difference). My thought was that because I have a moving window, sometimes the label is near the left edge of my moving window and sometimes the right edge, which means the actual region of interest is getting input into different neurons each time, which confuses the neurons so it feels like there is no real trend.
However, I took a look at the weights and biases and they seem to be stabilizing after a few epochs. Shouldn't they still be jumping around wildly if there really is no trend in the data? Maybe that is wrong.
I was thinking about testing my hypothesis out by running the left-leaning label examples and right-leaning label examples through different neural networks and seeing if I get better results, but maybe it's not worth the effort it will take.

I've been trying this for a while and can't seem to think of a good explanation or approach. I would really appreciate any help I can get on this, and I'm happy to provide more information. Thank you!

I suspect a couple issues:

The cost function is unusual for classification. You would typically
use something like cross entropy which you mentioned. Make sure not to threshold the outputs when you input to the cross entropy loss. Also, I wouldn't characterize L2 loss as being more "harsh". If your predicted probability for a particular data point is 0, but its label is 1, L2 loss would be 1. But for cross entropy, the loss would be very large.
It predicts all one class on the validation set or just any particular batch?

If it is just a particular batch, maybe how you batch the data is off and all of one class is in that batch. Look at metrics like accuracy and loss.
If it is the validation set, I would look into if the training loss is consistently going down. Maybe the learning rate is too high. You are using Adam which is based off of momentum so it does not necessarily need as high of a learning rate as standard SGD.

If the weights are stabilized, it means that the network is saturated or you have 0 loss and your optimization problem is solved (probably not the latter). If the network did saturate too fast, batch norm would definitely help.


How does the trainable projection layer used in PRADO and pQRNN work? Trainable projection layers are said to be a very powerful thing but after reading:

https://www.aclweb.org/anthology/D19-1506.pdf
https://arxiv.org/pdf/2101.08890.pdf

I don't understand how it works. So how does the trainable projection layer used in PRADO and pQRNN work?

The key idea in both the papers are word embeddings are replaced by trainable activations that are computed using a nueral network. The network (referred to as bottleneck layer) takes word projections as input. The parameters of this network is shared across tokens. This results in a trainable token representation. The projection itself is not trainable, but the representation derived from projection is trainable. Hope that clarifies.


How to classify and cluster this time series data I have post already the question few months ago about my project that I'm starting to work on. This post can be see here:
Human activity recognition using smartphone data set problem

Now, I know this is based around multivariate time series analysis and tasks are to classify and cluster the data. I have gathered some materials (e-books, tutorials etc.) on this but still can't see a more detailed picture of how even I should start. Here's the tutorial that looks like it might be helpful but the thing is my data looks differently and I'm not really sure if this can be applied to my work.

http://little-book-of-r-for-multivariate-analysis.readthedocs.org/en/latest/src/multivariateanalysis.html#scatterplots-of-the-principal-components

So basically, my questions are:

How I can start on some very basic analysis? How to read data so it any meaning for me.
Any tips and advises will be much appreciated!
Note: I'm just the beginner in data science.

I have shared a number of resources on time series classification and clustering in one of my recent answers here on Data Science StackExchange: https://datascience.stackexchange.com/a/3764/2452. Hopefully, you will find them relevant to this question and useful.


How to learn irrelevant words in an information retrieval system? Right now my recommender system for information retrieval uses word embedding stogether with Tfidfs weights like written here:
http://nadbordrozd.github.io/blog/2016/05/20/text-classification-with-word2vec/

Using Tfidf improves results. But I have the problem that irrelevant keywords (high frequent words) still have a large impact.
Can I learn a system such that it learns on which words to pay attention - preferred in an unsupervised way?

What can you suggest for a better information retrieval using word embedings?

If you are working with TF-IDF then it's important to experiment with min_df and max_df parameter. I guess you are on Python since you linked a Python tutorial. Here is the TF-IDF documentation and the related text to the above parameters.

  max_df : float in range [0.0, 1.0] or int, default=1.0 When building the vocabulary ignore terms that have a document frequency
  strictly higher than the given threshold (corpus-specific stop words).
  If float, the parameter represents a proportion of documents, integer
  absolute counts. This parameter is ignored if vocabulary is not None.

  min_df : float in range [0.0, 1.0] or int, default=1 When building the vocabulary ignore terms that have a document frequency strictly
  lower than the given threshold. This value is also called cut-off in
  the literature. If float, the parameter represents a proportion of
  documents, integer absolute counts. This parameter is ignored if
  vocabulary is not None.

You might find several rules of thumb on the web. Some of them suggest using a flat number on the min_df close to 5-7 documents and a percentage on the max_df about 80-85%. Maybe even lower. With this, you will be able to get rid of garbage, misspelt or unwanted tokens. Keep in mind that you need to try different combinations to get the right balance in your model.


What is a discrimination threshold of binary classifier? With respect to ROC can anyone please tell me what the phrase "discrimination threshold of binary classifier system" means? I know what a binary classifier is.

Classifiers often return probabilities of belonging to a class. For example in logistic regression the predicted values are the predicted probability of belonging to the non-reference class or $\text{Pr}(Y = 1)$. The discrimination threshold is just the cutoff imposed on the predicted probabilities for assigning observations to each class.


Error in lmest: missing data in the covariates affecting the initial probabilities are not allowed I'm running an LM model using the LMest package available in R. The dataset contains NO missing values.

pct_miss(df_long)
[1] 0
n_miss(df_long)
[1] 0

The lmest function with no covariates works fine. However, when I added covariates in the latentFormula, I got the following error message.
"Error in lmest(responsesFormula = responseA + responseB + responseC + responseD ~  :
missing data in the covariates affecting the initial probabilities are not allowed"
My code follows:
LMmodel <- lmest(responsesFormula = responseA + responseB + responseC + responseD ~ NULL,

          latentFormula =  ~

              SEX + GRADE | SEX + GRADE,

          index = c("ID","Time"),

          data = df_long,

          k = 3,

          paramLatent = "multilogit",

          start = 0)

where the four response variables (responses A to D) are categorical, ranging from 0 to 8; SEX is binary (1 and 2 represent Male and Female, respectively); GRADE is categorical, ranging from grades 7 to 12.
Can anyone point out what went wrong and how to address this error?
Thanks so much.

I'm answering my question here. The issue was eventually addressed by reordering the structure of my dataset. In my original dataset, the response variables were placed at the first few columns, followed by a list of covariates. To use the LMest package, response variables should be placed towards the end of the dataset.


Calculate mean by decile in Svydesign object So, I´m working with ENIGH - Database, which stands for ¨National Survey of Household  Income and Expenses¨ in Spanish, this is an exercise conducted by the Mexican government and like most surveys of its kind, it works with Weights.

What I´m trying to do is to calculate the mean, maximum and minimum household income by Decile.  In other words What´s the income of each 10%, grouping household base on their income.
To be honest, I haven’t gone that far but this is what I got until now:

I need my svydesign object
Convert that into a table using svytable
Arrange using desc() on my income variable

ENIGH_design <-svydesign(id=~upm, strata=~est_dis, weights=~factor_hog, data = ENIGH)
ENIGH_table <- svytable(ing_cor, ENIGH_design)

Here is where it gets tricky, supposing I have 100 rows, I can’t take the first 10 of them because in reality, when taking weights in mind, the might be 9% or 20% (I´m just throwing numbers) of the actual population.

I could use cut() on my income variable but I would be forgetting about weights and results will only be representative of the sample, not total population.

I think that the best approach would be to use a combination of:

mutate() to create a new variable base
if() in conjugation with mutate to define on which decile each row falls to
group_by() and mean() to calculate what I´m aiming for

This way I will have an extra variable which I could use to calculate whatever I want with whatever other variable I wish to. But again, I haven´t define my groups so it´s pretty much useless.

Thank you for reading. Thank you for your help.

Database available: https://www.inegi.org.mx/programas/enigh/nc/2016/default.html#Datos_abiertos

Here is a glimpse of how my DB looks:

folioviv    foliohog    ubica_geo   est_dis  upm  factor    ing_cor
100587003      1        10010000       2     610    180     22,723
100587004      1        10010000       2     610    180     17,920
100587005      1        10010000       2     610    180     27,506
100587006      1        10010000       2     610    180     56,236
100605201      1        10010000       2     620    178     41,587
100605202      1        10010000       2     620    178     135,437
100605203      1        10010000       2     620    178     62,386
100605205      1        10010000       2     620    178     103,502
100605206      1        10010000       2     620    178     27,323
100606301      1        10010000       3     630    223     68,042
100606302      1        10010000       3     630    223     98,537
100606305      1        10010000       3     630    223     53,237
100606306      1        10010000       3     630    223     132,861
100609801      1        10010000       3     640    232     190,033
100609802      1        10010000       3     640    232     28,654
100609805      1        10010000       3     640    232     74,408
100631401      1        10010000       1     650    171     80,761
100711503      1        10010000       1     770    184     38,640
100711504      1        10010000       1     770    184     81,672

There are many more columns but they aren´t necessary for this exercise.

In the Hmisc and reldist packages, you have the function wdt.quantile(). You can calculate your quantiles with this function, and then use cut() to make your groups and then your calculations.


Support vector machines in R: Finding the equation of a hyper plane (in 6 dimensions) and showing it's correct lin.6d<-read.csv("D:/Documents/Linear-6d.csv", header=TRUE)

View(lin.6d)

plot(lin.6d,col = ifelse(lin.6d$Class == 1, "red","blue"))

library(e1071) # Package for Support Vector Machines

svm.lin.6d <- svm(Class ~ ., data = lin.6d, type = "C-classification", kernel = "linear", cost = 10, scale = FALSE)
print(svm.lin.6d)

tuned <- tune(svm, Class ~., data = lin.6d, kernel = "linear", ranges = list(cost=c(0.001,0.01,.1,1,10,100))) # Will show the optimal cost parameter
summary(tuned) # Optimal cost parameter is 10

# Find parameters of hyperplane (wx+c=0)
w <- t(svm.lin.6d$coefs) %*% svm.lin.6d$SV # In essence this finds the hyper plane that separates our points
c <- -svm.lin.6d$rho

The above code takes the data from an excel file "lin.6d" which contains vector points "X1,X2,X3,X4,X5,X6" and "Class:-1,1". So each vector point has a class 1 or -1.
I have plotted the data and colour coded each point depending on it's class (Blue if the point if it's class is -1, red otherwise).
I then used the svm function to obtain the coefficients of the hyperplane where:
w is the set of coefficients (W1,W2,...,W6) and c is the intercept.
hence I got an answer:
w=(0.02433109, 0.02803587, -0.02332039, -0.000118514, -0.009224677, -0.02626795)
c= 0.1712391
where wx+c=0 is the equation of the hyperplane separating the classes of the points.
My question is:
How do I know if this is the correct answer? How do I create a visual plot to show this can be represented as a line separating the classes?

I over complicated the answer I was looking for. If you have the equation of the hyperplane, you can test it against each point in the dataset. If all points of class 1 are ALL (above or below) the intercept and the converse is true for all points of class -1. Then the hyperplane separates the two classes


xgboostclassifier prediction error after saving the model and restoring it I have trained a xgboost model and during training, the prediction works fine. But if I stop the script and start a restoring script to restore and predict, then for the same test dataset I get every data classified into one class. The weird part is that, even with the first prediction, i restore the model as shown below,

X = df.drop(['label'], axis=1)
y = df['label']
training_count = 0

X_train, test_data, y_train, test_label = train_test_split(X, y, test_size=0.1, random_state=7)
model = XGBClassifier(learning_rate=0.5, n_estimators=250, max_depth= 5)
model.fit(X_train, y_train)
model.save_model('trained_model_full')

#validation

model = XGBClassifier(learning_rate=0.5, n_estimators=250, max_depth= 5)
booster = xgb.Booster()
booster.load_model('trained_model_full')
model._Booster = booster
model._le = LabelEncoder().fit(test_label)
start = time.time()
pred = model.predict(test_data)
end = time.time()

The above code works and gives me 99% accuracy. But if I remove the training part and just restore it like below, then it fails to work. I get 50% accuracy.

X = df.drop(['label'], axis=1)
y = df['label']
training_count = 0

X_train, test_data, y_train, test_label = train_test_split(X, y, test_size=0.1, random_state=7)

#validation

model = XGBClassifier(learning_rate=0.5, n_estimators=250, max_depth= 5)
booster = xgb.Booster()
booster.load_model('trained_model_full')
model._Booster = booster
model._le = LabelEncoder().fit(test_label)
start = time.time()
pred = model.predict(test_data)
end = time.time()

This is a strange issue. Have anyone come across something like this? if so could you help me out?

It's better to use joblib to save the model:

import joblib

#This one saves the model
joblib.dump(model, 'trained_model_full')
#This one loads the model
model= joblib.load('trained_model_full')


Why augmenting the training data with binary attributes works better for our dataset? We have a dataset with multiple features ~400 where all of the features have a histogram as you can see in the following picture (sampled only a few)

Our assumption

We thought that this looked like some categorical data where few of the times had a special negative number and other times were zero.

So we treated each feature as category of whether the value is zero or not and we created a new column with binary values, 0 or 1. We did the same thing for whether the value is this special negative number or not. We ended up with one more column.

In other words we did the same thing as if we had three categories for each attribute A, B and C and we wanted to encode them with one-hot-encoding. Where here the A is 0, B is our special negative value and C is anything else. Recall that in one hot encoding you throw away one of the three columns because it is redundant.

Experiments to test our assumption

First experiment:
Linear Regression with only original data

Second test:
Linear Regression with the binarized features (2 columns for each feature, we had twice as much features)

Third test:
Linear Regression with the binarized features plus the original (2 columns for each feature plus the original results is three times as much features)

Results

The results was that the first experiment performed worse than the rest which was expected.

If our theory is correct that these can be represented as categories then the second experiment would perform better in terms of RMSE. And indeed the second experiment performs better.

The surprising result was that when combining the initial data with our binarized ones (augmented dataset with three times more features than the original one) the RMSE was even lower!
Why is that?

Your third model has the most capacity of the three.  When you have the values $x$ with special negative value $\xi$  and indicators $\mathbf{1}_0$ and $\mathbf{1}_{\xi}$, the linear model can fit
$$ \alpha x + \beta \mathbf1_0 + \gamma \mathbf1_{\xi}$$
When $x>0$ you just get $\alpha x$, when $x=0$ you just get $\beta$, and when $x=\xi$ you get $\alpha \xi + \gamma$;
so your model can get the correct (linear) signal out of the positive values and arbitrary values for the two special values.


Super Resolution CNN vs Regular CNN I am digging into finding a solution for background subtraction and one of the requirements is to not loose in quality of input image. Found that there is a specific type of CNN like Super Resolution CNN.

I was looking to find something when specifically this type of CNN is used and what is usually its accuracy. But everything that I find is about its structure.

Can someone explain me what are the pros and cons of SRCNN vs regular CNN?

Super Resolution CNN's are used to increase the resolution of an image. These architectures are not used to predict classes or to detect objects. They are an image processing technique.

There are several approaches but usually, in a first step, the image is upsampled using Bicubic interpolation. Afterwards, the quality of the upsampled image is enhanced using a CNN. The second step yields additional improvements in image quality since Bicubic interpolation simply interpolates and a CNN, on the other hand, is able to recognize structures.

(Note: Bicubic interpolation is not a definitive requirement. This is just an example to explain why a CNN is used for resolution improvement.)

The image above is from a paper of the authors of SRCNN. As you can see, a low-resolution image is used as input for the Super-Resolution CNN. The SRCNN then generates a high-resolution image.

This short video has additional information about Super Resolution CNN's and should give a good intuitive understanding.


Which data mining or machine learning algorithm would be appropriate for learning ordered frequent patterns? I have a dataset as (var1, var2, out), where the ordered pair <var1, var2> gives out. Most of the frequent pattern mining algorithms like the Apriori and FP growth algorithms does not preserve the order of var1 and var2.

Which are some of the available pattern mining algorithms (may also be a NN trick), to find association rules between ordered pair <var1, var2> and output variable out?

Thanks.

Assuming you only have these two features (var1,  var2), you might want to:
 * Create one-hot encoded features for each variable under each position.
 * Add a column on which variable is first (e.g. two columns - likely to work with trees but not with anything else).
 * Take each possible combination of variables and use that as your only input (e.g. you then take the average of out for that combination, perhaps adding some prior or smoothing).

As the comments mentioned, if out is some discrete event, maybe you'd want to instead look at Markov models.


export R neuralnet package model to PMML when I look into the R package pmml, I found that, it is possible to directly export nnet model into PMML format.

But actually I am using neuralnet package since it has more features and flexibility.

The problem is:

How can I export neuralnet model to PMML format, I checked pmml package, it doesn't work.

Any other options like transform neuralnet model to nnet model, or some other PMML packages to use? thanks

You already mentioned the alternatives...

convert the neuralnet object to a nnet class object
implement pmml.neuralnet()

Both pmml, nnet and neuralnet packages are easy to read. You can find the source code over cran.

I think implementing pmml.neuralnet() might be the cleanest. You can just copy-paste the current pmml.nnet()  method, call it pmml.neturalnet() and make the necessary changes. Should not take more than a day's work.

ps: It would be cool if you'd sent your function to neuralnet authors afterwards of course, and link it from here. :)


One Hot Encoding vs Word Embedding - When to choose one or another? A colleague of mine is having an interesting situation, he has quite a large set of possibilities for a defined categorical feature (+/- 300 different values)
The usual data science approach would be to perform a One-Hot Encoding.
However, wouldn't it be a bit extreme to perform some One-Hot Encoding with a dictionary quite large (+/- 300 values)? Is there any best practice on when to choose Embedding vectors or One-Hot Encoding?

Additional, information: how would you handle the previous case if new values can be added to the dictionary. Re-training seems the only solution, however with One-Hot Encoding, the data dimension will simultaniously increase which may lead to additional troubles, embedding vectors, on the opposite side, can keep the same dimension even if new values appears.
How would you handle such a case ? Embedding vectors clearly seem more appropriate to me, however I would like to validate my opinion and check if there is another solution that could be more apporiate.

One-Hot Encoding is a general method that can vectorize any categorical features. It is simple and fast to create and update the vectorization, just add a new entry in the vector with a one for each new category. However, that speed and simplicity also leads to the "curse of dimensionality" by creating a new dimension for each category.
Embedding is a method that requires large amounts, both in the total amount of data and repeated occurrences of individual exemplars, and long training time. The result is a dense vector with a fixed, arbitrary number of dimensions.
They also differ at the prediction stage: a One-Hot Encoding tells you nothing of the semantics of the items; each vectorization is an orthogonal representation in another dimension. Embeddings will group commonly co-occurring items together in the representation space.
If you have enough training data, enough training time, and the ability to apply the more complex training algorithm (e.g., word2vec or GloVe), go with Embeddings. Otherwise, fall back to One-Hot Encoding.


Evaluating recommendations quality and accuracy I'm developing a recommendation system, that should provide my clients what actions they should take in order to hit certain targets.
The underlying mechanics of the process is physical - where both actions and outcomes can be measured physically.

My current algorithm is based on a predictive model that can predict expected result for a given action.
This result can be scored, and thus scoring* many outcomes can lead to finding the best action - the action that will be recommended.

*the scoring is based on distance between outcomes and targets.

My predictive model performance is good(low MAE). Its performance estimation is based on predicting already taken(historic) action and comparing its prediction to actual(historic) outcomes.

My problem arise when I try estimating the recommendations quality, since many of the recommendations are actions that were not taken, I have little ways evaluating their quality.

Ive tried using actual outcomes as 'targets' to my recommendation system, and compare the recommended action to actually taken action.
I am not sure this is the right way to go, and any advice will be appreciated.

Unraveling the causality involved in this type of recommender system is complicated, but tractable.

Given that you have a good experimental methodology, and your evaluation numbers are generated on numbers not used for training (among other concerns), it sounds like you have the best estimate that you can get (or are in the neighborhood) using past data.

The next step would be to form a new experiment, in which you (ideally) randomly provide the recommendation for one group and not provide it for another group, which otherwise are the same. You then measure the outcomes for both groups and compare them statistically to determine the presence of an effect and its size.

Proposing that study is your next step if you're in an organization, or performing it if you have the agency to do it on your own.


LSTM Model for predicting the minutely seasonal data of the day I am making a model for predicting the network traffic volume for our data center.
Let me describe my dataset first. At this time, we have the model of 90 days, on each day, we record the network traffic volume every minute. And this data also has the seasonality pattern: the network traffic volume fluctuates and remains at night, decreases in the morning and increases in the afternoon (as I could observe, this pattern happens on all the data in my dataset)

What I want at this time is a model for predicting the network traffic volume in some next minutes(25 minutes, for example), given that the data is given for all the previous minutes. The new predicted value will be contributed for the next predicting. For example, the value at minute i will be added to a window of data with a specific length to predict the value at minute i+1.

At this time, I have tried LSTM-RNN and the feature I use is the minute of the day and the network traffic volume in that minute (with normalization before inputting to the LSTM network).
However, my problem is: my model could catch the wrong pattern of the data: when the network traffic volume should increase in the afternoon, my model predicts it decreases. I have tried with different structures of LSTM network (increases the LSTMs layers, change the number of nodes in the fully-connected layers after the LSTM layers...); and also the length of the window, but that problem still remains.

So I want to ask if there is any problem with LSTM model for predicting the data with seasonal trend like my data? If no, could anybody suggest me an LSTM model or any other suitable model for my data?

Thank you in advance :-)

When represented at a minute level, the pattern is going to recur after almost 1440 minutes. So your LSTM needs to learn a really long term dependency here. And LSTMs are not great at such long term dependencies.

I'll have a contrary opinion here. Drop neural networks, fit a good linear model to get a benchmark. If the linear model (example outlined below) beats your LSTM error rate currently, invest in building a better (generalised) linear regression model instead. Also, start with a more coarse definition of output (predicting total for next 15 minutes / 1 hour).

Create hand-rolled features for your data. Start with traffic at the same hour of the day yesterday, traffic in same hour of day for past 7 days, traffic on the same hour of the day on same weekday last week.
Add more advanced features by defining better aggregation window than hour. Something like 6 - 9 PM, 12 midnight - 6 am.
Add features related to recent data (ratio of cumulative traffic today compared to cumulative traffic till same time yesterday).
Build a suitable linear model on this data, benchmark the error rate. If the traffic is being measured in number of hits, use poisson regression instead of simple linear regression.

Complex neural net models like LSTM are best used when hand-rolled features are extremely difficult to think and implement. For many time series applications, generating features yourself is intuitive, simple to achieve (most libraries like Python pandas have great date-time functionality) and good first step. It is much better than teaching LSTM to differentiate between morning and afternoon.

François Chollet's book on deep learning has a chapter on similar application. He tackles the problem of predicting weather in particular time window based on past data. He shows that many complex neural net architectures find it hard to beat simple baseline like averaging from same time period in past.


How to Deploy a Scikit-learn Model into the Cloud? I have generated a common Sklearn model for text classification which I want to make accessible in the cloud (there is no provider preference) as an API.

So far the closest solution that I managed to find is in this tutorial but it seems quite complex (getting the venv.zip dependencies package at the beginning is unclear, for instance) and specific (in my case NLTK and an external Stanford segmenter is involved for preprocessing and I cannot figure out where to put all these modules and how to invoke them).

Is there a decent and robust way to solve this challenge?

If you want to deploy a scikit-learn model to the cloud and be able to access it through an API, I am guessing that you will want to access the predict method.

The API

To build an API with python, the easiest is to use Flask or Django. I prefer Flask for these types of tasks because it has a simpler interface. Django offers you more lower-level control however.

You will have to create some route that the client will call by sending you the input data, and once you receive it on the server side, you'll just have to pass it through your Sklearn model, and send back the output to the client.

Run sklearn in the cloud

The other thing you need to take care of, is run Python on whatever cloud solution you'll use. In order to deploy your Flask/Django app, I would probably use Docker as it's well supported on many platform. Here is an example on how to deploy a Docker container on Azure


XGBoost evaluation metric unbalanced data - custom eval metric I have built a model using the xgboost package (in R), my data is unbalanced (5000 positives vs 95000 negatives), with a binary classification output (0,1).

I have performed cross validation with the evaluation metric AUC Area under the ROC curve which I now believe to be wrong since this is better used for balanced data sets.

I analyse the final results of the model using the Area Under the Precision Recal curve (AUPRC) and the Matthews Correlation Coefficient (MCC), however I now believe that I should have been evaluating the cross validation models with the AUPRC and MCC also and completely forget about the AUROC.

I cannot find much in the literature which uses CV with the evaluation metric of AUPRC and MCC.

I just want to make sure that I am thinking correctly and that my previous evaluation method is wrong and the AUPRC / MCC would be a better way to go.

I feel you are mixing few things:

1:20 is not exactly a class imbalanced data. Classification algorithms work well with this ration also. If you still feel so, you can oversample your positive class or undersample negative class.
AUROC generally to compare different models. Like one from logistic and other from Xgboost.( It also ensures robustness of models), Generally AUROC is not used for model accuracy/ getting cut-off values.
Logloss/ misclassification rate/ F1 score/ MCC any of these can be used based on your requirement of classification. If you are predicted cancer , then you might not want to miss any patient and might be Ok to false positive, then focus should be on recall. f1 gives balance of precision and recall. Log loss is used (Log Loss heavily penaltises classifiers that are confident about an incorrect classification) when you don't want to have false positive. These links might help.

https://machinelearningmastery.com/classification-accuracy-is-not-enough-more-performance-measures-you-can-use/
https://www.r-bloggers.com/making-sense-of-logarithmic-loss/
https://towardsdatascience.com/evaluation-metrics-for-classification-409568938a7d


Unable to import xgboost 0.9 When running

import xgboost as xgb
print(xgb.__version__)

I am getting the following error:

xgboost.core.XGBoostError: XGBoost Library (libxgboost.dylib) could not be loaded.
Likely causes:
  * OpenMP runtime is not installed (vcomp140.dll or libgomp-1.dll for Windows, libgomp.so for UNIX-like OSes)
  * You are running 32-bit Python on a 64-bit OS

As stated in this other question here I am running 64-bit Python since this:

import sys
print(sys.maxsize > 2**32)

returns True.

Finally, to install OpenMP I tried this: brew install libomp but the problem persists.

I had the same problem. Tried to fix both "likely" causes to no avail. Strangely, everything was working absolutely fine just a few days ago. I changed the Python version from 3.6.1 to 3.6.7 and it fixed the issue. Unfortunately, I don't yet know what the problem was, but it fixed the issue for me.

This is Python's docker image on CircleCi's linux machine. Specifying this in case the problem has to do anything with it.


Why does least squares linear regression perform so bad when switching from 2D to 3D line? The vector of coefficients that minimize least squares can be found like so:

beta = ((X'X)^-1)*X'y

Theoretically this result is true for any number of variables (columns of X).

I have made two python notebooks: one for univariate linear regression and another one for multivariate linear regression.

The code is analogous for both notebooks:

create a random line

# multivariate snippet. A part from the plotting part,
# this is the only part of the code that differs between the two notebooks.
beta = np.random.randint(1,6,(3,))
X = np.linspace([1, 3, 7],[1, 14, 23],100)
y = X @ beta

create noise

mu, sigma = 0, 1
noise = np.random.normal(mu, sigma, 100)

add noise to y

y_noise = y + noise

try to fit a line trough y_noise

beta_ = np.linalg.inv(X.T @ X) @ X.T @ y_noise
y_ = X @ beta_

compare y_ with y_noise

Why do I get such a different result when fitting a 2D line and a 3D one?

The matrix $X^TX$ in your example is close to a singular:

from numpy.linalg import inv, det

print(det(X.T @ X))

1.479949989718308e-06

which makes its inverse very large

inv(X.T @ X)

[[ 4.83118746e+11  2.66548273e+11 -1.83251938e+11]
 [ 2.66548273e+11  1.47061116e+11 -1.01104517e+11]
 [-1.83251938e+11 -1.01104517e+11  6.95093558e+10]]

This is the reason of the error in your question. The following expression

inv(X.T @ X) @ X.T @ X

should produce an identity matrix. But in your example it is

[[ 1.2265625   1.10289171  4.8378709 ]
 [ 0.02282715  0.61713325  0.95782902]
 [-0.01519775  0.25675456  0.33339437]]

So, when you multiply this matrix by beta the result is far from beta.

Another way to look at this singularity is that your 3D problem is not actually 3D. It is still 2D. That's why you have a line, not a 2D surface in your plot. It can be converted to a 2D representation by a transformation of coordinates. If you make a matrix for a proper 3D problem, say

np.random.seed(3)
X = np.hstack((np.ones((100,1)),np.random.rand(100,2)*100))

then inv(X.T @ X) @ X.T @ X gives the correct result:

[[ 1.00000000e+00 -1.24233956e-13 -6.21724894e-14]
 [-1.90819582e-17  1.00000000e+00  6.38378239e-16]
 [-8.67361738e-18  9.28077060e-16  1.00000000e+00]]

and the formula $\beta=(X^TX)^{-1}X^Ty$ works well.


How to interpret Sum of squared error I am working on ANN. I have 2497 training examples and each of them is a vector of 128. So the input size is 128. Number of neurons in hidden layer is 64 and number of output neurons is 6 (since classes are six). My Target vector looks something like this : [0 1 0 0 0 0]. This means that the example belongs to class 2. I have used sigmoid as an activation at all layers and sum of squared error is loss. SSE is computed over one epoch. Total epochs are 10k. my loss starts from around 700 and reduces to 450. Should I say that loss is 18% per example since 450 is the loss for all the 2497 examples. How do I interpret this. Is my model good enough? I know that I should test it on unseen data to be sure of its accuracy but still does this tell anything about the performance at all or not.

ps: I am implementing it in C

SSE in classification is proportional to the Brier score:

$$
SSE = \sum_n \sum_k (\hat y_{nk} - y_{nk})^2 \\
Brier = \frac{SSE}{N}
$$

For observations indexed by $n$ and classes indexed by $k$.

In your case, you obtained a Brier score of 0.18.

The Brier score can be decomposed into

$$
Brier = Reliability - Resolution + Uncertainty
$$


How to preprocess different kinds of data (continuous, discrete, categorical) before Decision Tree learning I want to use some Decision Tree learning, such as the Random Forest classifier.

I have data of different types: continuous, discrete and categorical. How do I have to preprocess data in order to have consistent results?

One of the benefits of decision trees is that ordinal (continuous or discrete) input data does not require any significant preprocessing. In fact, the results should be consistent regardless of any scaling or translational normalization, since the trees can choose equivalent splitting points. The best preprocessing for decision trees is typically whatever is easiest or whatever is best for visualization, as long as it doesn't change the relative order of values within each data dimension.

Categorical inputs, which have no sensible order, are a special case. If your random forest implementation doesn't have a built-in way to deal with categorical input, you should probably use a 1-hot encoding:

If a categorical value has $n$ categories, you encode the value using $n$ dimensions, one corresponding to each category.
For each data point, if it is in category $k$, the corresponding $k$th dimension is set to 1, while the rest are set to 0.

This 1-hot encoding allows decision trees to perform category equality tests in one split since inequality splits on non-ordinal data doesn't make much sense.


Gene innovation numbers in NEAT implementatoins In NEAT (neuroevolution through augmenting topologies) algorithm description, an innovation number, e.g. id, is assigned to each gene so that genomes can be crossed over meaningfully: genes having corresponding innovation numbers are aligned and then crossover takes place.

Is this innovation number global only within distinct genemoes ? Meaning that each genome has its own counter or is it the case that there is a unique counter for all genomes ?

Description of algorithm: https://www.cs.cmu.edu/afs/cs/project/jair/pub/volume21/stanley04a-html/node3.html

The innovation number is generated by a single, unique counter for all genomes. The purpose of the innovation number is to have an ID that indicates corresponding genes in two different genomes.

For example, a gene may originate in a genome due to mutation in generation 1. In future generations this gene may have been passed on to other genomes through crossover. So now we have multiple genomes with our gene in it. If two genomes with this gene are selected for crossover with each other, we want to know that this gene is identical in both genomes. We can tell that these genes are identical, because they have matching innovation numbers. If each genome had their own counter, then each genome would have its own unrelated gene for each innovation number. This tells us nothing about the nature of our network and negates the reason we added innovation number.

As an aside, here is a more thorough outline of NEAT straight from the author of the paper


Tensorflow 2 eager vs graph mode I've been working through the tensorflow-2.0.0 beta tutorials. In the  advanced example a tensorflow.keras subclass is used. The presence of the @tf.function decorator on train_step and test_step means the model executes in graph mode (not sure if that's the correct terminology, I mean oposite to eager mode). If I remove these decorators I can single step right into the model call function and see the input/output tensor for each layer which is neat.

My question is, is there a programatic way to enable/disable the @tf.function decorators. Commenting them out to switch between eager and graph mode doesn't seem particularly scaleable but it's certainly useful for debugging/learning)

You could always write two functions (one with the decorator and one without) and call whichever suits you...

For example

@tf.function()
def graph_function()
    # This function will operate in graph mode
    ...

def eager_function()
    # This function will operate in eager mode
   ...

if tf.executing_eagerly()
    my_function = eager_function
else:
    my_function = graph_function

# You proceed to my_function from now on

I don't know if there is a better way but I've seen this a lot being used in the tensorflow official repository on github.


Merge columns of coordinates and construct sequence I have to following data frame.

    x1     x2          x3
    0   58.33546    15.14554
    0   58.33604    15.14571
    0   58.33645    15.14581
    0   58.33681    15.14591
    0   58.33722    15.14600
    0   58.33810    15.14604
    0   58.33900    15.14599
    0   58.33966    15.14578

Consisting of long and lat, each pair belongs to a string (0,1,3,4 and so on, only the string 0 is displayed above).

I would like to first construct a new column with the pairs (x2,x3). How do I do that?

The coordinates above corresponding to 0 is actually a journey (say X), if there is another journey (Y) whose first coordinate is the same as X last, I would like to merge the sequence X and Y coordinates together and call that journey X (the continuation of X simply).

My data set is called data, for some reason merging them together wont work with

data[, x4 := paste(data$x2, data$x3)]

This will create a new column, only isplaying x2.

it probably depends on the use of your new generated column. Here I try to reproduce your example:

x1 <- c(0, 0)
x2 <- c(58.33546, 58.33604)
x3 <- c(15.14554, 15.14571)
df <- data.frame(x1, x2, x3)
df$x4 <- paste("(", as.character(df$x2), ", ", as.character(df$x3), ")")

And df$x4 will look like:

[1] "( 58.33546 ,  15.14554 )" "( 58.33604 ,  15.14571 )"

If you want to show the coordinates, the above format should be fine. If you want to do further analysis with that column, some other transformations are required.


Summing three lexicon based approach methods for sentiment analysis? I'm doing sentiment analysis using a lexicon based approach and I have a bunch of news headlines that needs to be categorized as negative, positive and neutral or within a scale ranging from -1 (very negative sentiment) to +1 (very positive sentiment).
I'm considering using three different methods: TextBlob, Vader (Valence Aware Dictionary and Sentiment Reasoner) and LIWC2015 as I have access to it. This would lead to three predicted sentiment of each headline.
I hove two question:

Which are the pros and cons of this technique?
And, once runned the 3 analisys, how can I melt all them into a single metrics? Is standardize/normalize the 3 metrics and averaging them a good solution?

What you're proposing is some very simple form of ensemble learning.
You need to have at least a sample of labelled data in order to evaluate any method. Using this labelled data you can:

evaluate each of the three methods on their own
evaluate your idea of averaging the 3 methods predictions
if you have enough labelled data, you could even train a model which combines their predictions optimally (this would be full-fledged stacked generalization)


Ranking skills depending on similarity I need to rank human skills depending on their similarity to the input skill. So if I enter "Dutch language", I want the list like this:

0.97 Dutch
0.86 Dutch lessons
0.55 Frisian
0.50 Flemish
0.27 German language

I have a database of around 4500 human skills (ranging from "programming in C" to "baking almond cake") with 600 manually categorized. I can already find corresponding article on BabelNet and pull domain, categories and related terms.

Example skill with the data from BabelNet:

name:"photography"
categories:
  0:"Photography"
  1:"French_inventions"
  2:"Optics"
  3:"1822_introductions"
manualCategory:"art & music"
domains:
   ART_ARCHITECTURE_AND_ARCHAEOLOGY:1
compounds:
  0:"digital_photography"
  1:"landscape_photography"
  2:"photographic_developing"
  3:"motion_photography"
  4:"nature_photography"
  ...
  48:"photographic_plates"
otherForms:
  0:"still_photography"
  1:"photo"
  2:"photos"
  3:"photographed"
  4:"photographers"
  ...
  20:"Photographer"

Can you suggest me the approach or at least steer in the right direction?

Pretty late but I'm surprised this wasn't answered more.  "Cosine similarity" is a great technique to try, though simply letting users search with a hard string and then ranking by popularity isn't so bad (e.g. "dutch" brings up everything with "dutch" in it, though I would discard mid-word matches, so "ball" wouldn't return "football", but would return "ball room dancing").

I'd say in any approach a main issue will be deduplicating previous (non-standardized) skills input by users that weren't quite standardized.  You could also try replacing the candidate skills with versions that have different synonyms substituted at search time, e.g. "soccer coaching" might be stored also as "football coaching" if most of your content is from Europeans.

Sometimes extreme accuracy might not be the best goal, though... You may want to encourage users to explore new skills that they never knew existed!  Not sure what your needs are...

Whatever you settle on, it might be worth building a semi-hand-crafted test set of queries and relevant results so that you can see if the performance is terrible (Google precision and recall in the context of search results).


What machine learning algorithms to use for unsupervised POS tagging? I am interested in an unsupervised approach to training a POS-tagger.

Labeling is very difficult and I would like to test a tagger for my specific domain (chats) where users typically write in lower cases etc. If it matters, the data is mostly in German.

I read about about old techniques like HMM, but maybe there are newer and better ways?

There are no unsupervised methods to train a POS-Tagger that have similar performance to human annotations or supervised methods.

The current state-of-the-art supervised methods for training POS-Tagger are Long short-term memory (LSTM) neural networks.


Closed form solution of linear regression via least squares using matrix derivatives How is the closed form solution to linear regression derived using matrix derivatives as opposed to using the trace method as Andrew Ng does in his Machine learning lectures.  Specifically, I am trying to understand how Nando de Frietas does it here.

We want to find the value of $ \theta $ that minimizes $ J(\theta)=(X\theta-Y)^{T}(X\theta-Y) $, where $\theta \in \mathbb{R}^{N \times 1},  X \in \mathbb{R}^{M \times N}$, and $Y \in \mathbb{R}^{M \times 1}$

$\nabla_{\theta}J(\theta) = \nabla_{\theta} (X\theta-Y)^{T}(X\theta-Y)$

$ = \nabla_{\theta} (\theta^{T} X^{T}-Y^{T})(X\theta-Y)$

$ = \nabla_{\theta} (\theta^{T} X^{T}X\theta-\theta^{T} X^{T}Y - Y^{T}X\theta + Y^{T}Y) $

Note that $\theta^{T} X^{T}Y$ is a scalar, so $\theta^{T} X^{T}Y = (\theta^{T} X^{T}Y)^{T} = Y^{T} X \theta$

$\nabla_{\theta}J(\theta) = \nabla_{\theta}(\theta^{T} X^{T}X\theta-Y^{T} X \theta - Y^{T}X\theta + Y^{T}Y)$

$ = \nabla_{\theta}(\theta^{T} X^{T}X\theta- 2 Y^{T} X \theta + Y^{T}Y)$

$ = \nabla_{\theta} \theta^{T} X^{T}X\theta - \nabla_{\theta} 2 Y^{T} X \theta + \nabla_{\theta} Y^{T}Y$

$ = \nabla_{\theta} \theta^{T} X^{T}X\theta - \nabla_{\theta} 2 Y^{T} X \theta $

How do I apply the matrix derivatives described in that video to solve this?  He skip steps.

Edit:
Below is the suggested strategy of removing theta by differentiating, then taking the inverse of both sides.  So looking at one term at a time, we have

$ \nabla_{\theta} \theta^{T} X^{T}X\theta = ? $
How do I differntiate this?  This is like differntiating $x\alpha_{1} \alpha_{2} x$ w.r.t. x in the scalar case. I need to combine those $\theta$ terms to hit them with the derivative. Transposing seems to result in the same expression:
$$ (\nabla_{\theta} \theta^{T} X^{T}X\theta)^{T} = \nabla_{\theta} \theta^{T} X^{T}X\theta$$

Looking at the second term, we have

$ \nabla_{\theta} 2 Y^{T} X \theta = 2 X^{T} Y$.

Putting together this, we have:
$$\nabla_{\theta} \theta^{T} X^{T}X\theta = 2 X^{T} Y$$

Knowing the solution is $\theta = (X^{T}X)^{-1}X^{T}Y$ we can reverse engineer the problem, but I am just not seeing it.  And how do we get rid of that 2 factor?

Matrix derivatives work a bit different than regular ones. The scalar parallel of $\nabla_{\theta} \theta^{T} X^{T}X\theta$ you make should be more like $\theta x^2 \theta$ which in turn is just $x^2 \theta^2$. (Note that I changed $\alpha$ for $x$ to avoid confusion with $X$.) You do not need to modify $\theta^{T} X^{T}X\theta$ so that the two $\theta$'s are beside each other, this actually is what you want.

Just like you usually have $\frac{d}{d\theta} (x^2 \theta^2) = 2 x^2 \theta$, then in matrix notation the rule is
$$\nabla_{\theta} \theta^{T} X^{T}X\theta = 2 X^{T}X \theta.$$

Hence, I feel he's not really "skipping steps", but applying a different step than the one you expected.


How to fix debuged CNN model in Keras I've trained a CNN model based on landscape and pool images. The main purpose is to make model classify if the image contains a pool or not. The accuracy on the test set was about 94% but when I used my own image to test, sometimes model did wrong predictions then I implemented GradCam and realized that the model looks only at the edges of the pool when I would classify the pool by the water. How can I fix my model so it could make better predictions and look not only at the edges?

I solved the problem, I added an additional conv layer with 64 feature maps and increased size of dense layer from 512 to 1024 also I trained the model with the increased number of epochs(from 50 to 150). The result is impressive! Everything as I wanted.


Machine learning with sklearn vs. scipy stats I've created 50 random x and y points (with slope of y = 2x-1).

First, I used Linear Regression from sklearn to fit the model onto my dataset where I got a slope of 2.0066... and intercept of -0.535...

My Question: is fitting the model to our dataset considered training? For each given x value, since it has a y-value (supervised), does our machine go through each x,y match and create line of best fit based upon that? Thus, is our model trained?

Second, I used stats.linregress(x,y) from scipy to get slope and intercept (which seem really close if not the same to the slope and intercept I've got from using sklearn Linear Regression).

My Question: If both methods give the same result, why not just use scipy to get formula for the best fit line to make predictions? What is the benefit of using machine learning?

Yes fitting the data and finding the best fitting line is called training the model.
If you look at the source code of scikit-learn linear regression you can find the its using scipy linalg.lstsq module for finding the coefficients and intercept (most cases). See the source code for more details . Machine learning is fancy word for Application of  mathematics (on data mostly) using computers (machines)


Choosing a right algorithm for template-based text generation I am doing a text generation project -- the task is to basically represent the statistical data in a readable way.

The way I decided to go about this is template-based: each data type has a template for how sentence should be formed and what synonyms can be used.

I'm torn about whether some kind of ML techinque can bolster this template-based approach. Text should be unique -- so I need an algorithm that optimises for uniqueness.

Now, there are API solutions that can give me uniqueness score at the end of the text (and even in the middle of it) -- so my first instinct was to try out reinforcement learning with sparse rewards. Templates can be represented as a tree, which the algorithm traverses, getting rewards in the end and during its trip. Input is its current options on where to go and the output is its decision on where to go.

The problem with this approach is that after it succeeded in generating an unique text, it can't generate the same one (I mean it can, but the score would be 0), which might prove difficult for the model to learn. Also many articles on the web indicate that RL is really, really hard to tune properly.

I am now in the pre-research process, so any feedback on how I should approach this task is appreciated. Maybe there is no need for ML at all?

What do you think? My instinct tells me that such problems should have established solutions and I'm just searching the wrong way.

Thanks!

You give this one a try - generation-caption-for-new-image . This has 2 parts one labeling the image and second generating text for those labeled images.
Your case is similar to this, the difference is - in your case there are templates and in this case there are images.

Please comment if you need help in understanding the code (that code is not mine, but I have gone through that in detail) or you find difficulty mapping this answer to your problem.


ValueError: The number of classes has to be greater than one; got 1 class in MultiLabel Classification problem I am working in Python in a Multilabel Classification problem. I have a dataset with texts and around 20k unique labels. I transformed the text to word embeddings and now I use that in a ChainClassifier to predict the tags that each texts might have.

from skmultilearn.problem_transform import LabelPowerset, ClassifierChain, BinaryRelevance
from sklearn.svm import SVC

df = pd.read_csv('sample_fos.csv')
df = df.dropna()
df.fos = df.fos.str.split(',')

mlb = MultiLabelBinarizer()

data = df.join(pd.DataFrame(mlb.fit_transform(df.pop('fos')),
                          columns=mlb.classes_,
                          index=df.index))

labels = mlb.classes_

X_train, X_test, y_train, y_test = train_test_split(data[features], data[labels], test_size=0.20, shuffle=True)

clf = ClassifierChain(classifier=SVC(gamma="auto"))
clf.fit(X_train,y_train)
y_pred = clf.predict(X_test)

I get the following error:

  File "<ipython-input-54-976064cc6433>", line 30, in <module>
    clf.fit(X_train,y_train)

  File "/lib/python3.7/site-packages/skmultilearn/problem_transform/cc.py", line 155, in fit
    X_extended), self._ensure_output_format(y_subset))

  File "/lib/python3.7/site-packages/sklearn/svm/base.py", line 147, in fit
    y = self._validate_targets(y)

  File "/lib/python3.7/site-packages/sklearn/svm/base.py", line 521, in _validate_targets
    " class" % len(cls))

ValueError: The number of classes has to be greater than one; got 1 class

In my knowledge, this means that one of the label columns has one class only (either 0 or 1)

However, running the following snippet, I get that all of my columns has 2 unique values

b = []
for i in labels:
    b.append(len(data[i].unique()))
print(min(b),max(b))
>> 2, 2

Could you see how many classes are in training and in test?

y_train.unique()
y_test.unique()

It can happen that at some point you got a split that has only one label, if your data is small or heavily imbalanced there are more chances. One way to fix it is stratifying the target.

train_test_split(data[features], data[labels], test_size=0.20, shuffle=True,stratify=data[labels])


Is it possible to train probabilistic model to return several distributions? I have nonlinear data of function y(x), which is let's say parabolic. At some points of x there are several y's (look at the picture).

Is it possible to train a probabilistic model to return several distributions (when needed) i.e. several means and variances. For example: when I feed a (x=a) to the model -> it returns 2 red distributions (2 means and 2 variances), and when I feed b (x=b) to the model -> it returns 1 blue distribution.

Thanks.

That data can be modeled as a statistical process, where the distributions and parameters change as a function of x. This is in contrast to modeling it as a typical statistical distribution which assumes the same distribution and parameters throughout a range.
The "a" range could be modeled as a bimodal distribution and the "b" range could be modeled as a unimodal distribution.


Any issue with "overlapping" sliding windows in time-series data analysis? I am developing some classification/regression models form accelerometry time-series data. So far, I have created datapoints by extracting features from non-overlapping sliding windows of the time-series data. I would like to try using overlapping windows as well. However, I was wondering whether it is conceptually sound or there might be some caveats to keep in mind as the data is reused in the overlapping windows.

In general I don't see anything wrong with overlapping windows, it might make perfect sense depending on your task. In fact some learning models (e.g. for sequence labeling) do use features based on past data points, which is conceptually similar to having overlaps between them.

However you need to be careful about the fact that this makes data points depend on each other, so of course the preprocessing must be done separately for the training and test set.


Tensorflow predicting same value for every row I have a trained model. For single prediction I restore the last checkpoint and pass a single image for prediction but the result is the same for every row. Does anyone have a clue of what might be wrong?

This is the code for prediction:

And this is the output:

I think I found the problem. I need my input to have shape of (1, 256, 256, 1), thats why I performed image = image[...,0][...,None]/255 but doing so changed all the values to 0


Why we can't Remove features with missing values in Data Preprocessing In a Real Time Dataset, There are many missing values available in the Dataset and also we need to deal with data preprocessing. And there are many ways to minimize the problem of missing values preprocessing.
So, Can we use mean, median, standerd deviation or we can remove those whole records?
Why many people denying to remove complete record from dataset, Why?

Missing values doesn't necessarily mean missing information. Sometime missing value represent an information in itself. For example: we have a data set which have features such as pool area, no. Of rooms and area. Now pool area have 90% of its value missing. You can create a new column called is_pool, which tells if the house has pool or not, from pool area column by using condition that if pool area is missing, make is_pool =0 otherwise 1.
This is one basic example, in my experience most difficult thing while doing EDA is identifying is missing value is really means no information or is it represents something else entirely. In short, understand why the value is missing.


Baby cry detection model using binary classification through logistic regression I need some help regarding my final year project.
I am fairly new to machine learning and I have tried a lot to understand how to train a model using logistic regression.
I have two datasets of audio clips of 5 seconds each, one of the babies crying voices and one of baby 'not crying'.
I want to train the model using logistic regression on google colab. Now I have been successful converting the audio clips to spectrogram images. But I am stuck in training the model because I am unable to understand how to train the model (the code is very complex where ever I have seen it up till now).
 What I want is to train the model using logistic regression, and get 100 weights at the output. Then what will happen is that I will record baby crying audio using Arduino. Then I will get 100 data points from newly recorded audio.
Then the previous and new 100 weights will be multiplied one to one. and then added. and then a sigmoid function will be run on it(all of this calculation after getting 100 weights, will not happen in google colab but in Arduino).
And then if the output value is greater than 0.5, it means that baby is crying (which means that the newly recorded audio is of the baby crying) and if the value is less than 0.5, it means the baby is not crying. Now the 100 data points from the Arduino recording, have a range of 0 to 1. It will be good if the 100 weights of the trained model also lie between 0 to 1.
Any help regarding the code or links to a good tutorial video on logistic regression will be much appreciated.

As your context, You convert your audio clips to spectrogram images. The spectrogram is a visual representation of the spectrum of frequencies of sound.
 Now you need to Extract Features from your dataset. There are many feature extraction technique for audio data (See this blog for knowledge).
  The state of the art features extraction is now using:  MFCC — Mel-Frequency Cepstral Coefficients.
 By using librosa library we can easily implement the MFCC.
 Here are some resources for MFCC :

Music Feature Extraction in Python - From towardsdatascience
Audio Data Analysis Using librosa - Kaggle Notebook (Check out Mel-Frequency Cepstral Coefficients(MFCCs) portion)
Stack Overflow Question : MP3 to MFCC

After you extract features (Converted into MFCC), Then you need fit them into the Logistic Regression Model as you wanted. I share some resources which experiment with MFCC Features and Logistic Regression:

Classification Logistic Regression/SVM on mfccs
Audio-Genre-Classification GitHub repo by dhruvesh13
Pitch detection - Kaggle Notebook

And for learn logistic regression, you may follow Machine Learning By Stanford, Coursera or follow this from youtube.


Suitable instance counting CNN for training on polygonal masks I have a medical dataset labeled with polygonal masks (rather than rectangle boxes). It works well for pixel annotation with UNet to generate masks of healthy vs damaged skin. Now I need to do instance counting. Most of the CNNs like YOLOv4 consumes bounding boxes, not polygons.
Which CNN should I use for instance counting given that my dataset consists of labeled polygons?

Which CNN should I use for instance counting given that my dataset consists of labeled polygons?

CNNs for instance segmentation. To start with you can try Mask-RCNN. Here are all state of the art CNNs for instance segmentation. You will also find code for most of them.


Are there known techniques to transform features X classified as C to features Y classified as C' I don't think the wording of my question is that clear myself, but I don't have any better words suitable for a title (on top of my head at least). I was wondering if given features X that is classified by a model M as class C, is there a way to find the features Y that is relatively "close" to X so that it would be classified as class C' by M.

I was thinking if some sort of clustering can help such as k-means and then getting the centroid of class C' and using that. The final idea is to get the difference between X and Y to be displayed. Does that sound reasonable? I'm not really a data scientist so want to check up on my thoughts.

If someone can suggest a paper or direction that would be much appreciated

EDIT:
For clarification. The purpose of this is that I have a set of people's skills and their jobs and I want to be able to give an advice of what skills a person needs to cultivate for their desired job.

E.g I can program, have a CS degree, experienced with unix etc. and am classified as software developer (skills are codified into numerical values not text anymore) and I want to work as a chemical engineer. I want to know the skills I would need so that I can be classified as fit to be a chemical engineer.

So X is my set of skills, C is software developer, C' is chemical engineer, and Y is the set of skills appropriate for a chemical engineer that I am looking for.

This problem can be addressed from an Economics perspective, even with existing data and without complicated algorithms.  There is some existing research in the Labor Economics field about transferability of skills between occupations.  For example, this "The estimation methods of occupational skills transferability" paper: https://link.springer.com/article/10.1007/s12651-016-0216-y
There is one method it discusses, referencing another 2016 paper by Ormiston, that could be helpful to you.  It involves calculating "the ratio of shared, standardized knowledge, skill and ability categories across occupations".
Here are some relevant excerpts:

It "employs a “skills” approach by explicitly estimating the proportion of knowledge, skills and abilities (KSAs) utilized in occupation i that can also be applied in occupation j."
"To estimate this proportion, it used data from O*NET – the U.S. Department of Labor’s occupational database – to examine the commonality of occupations across 120 standardized knowledge, skill and ability dimensions."
"O*NET examines each occupation and measures each component on the level (0–7 scale) of proficiency needed to function within the profession and the importance (1–5 scale) of each component to the occupation; multiplying these values together generates a comprehensive score, $S_{im}$ , for each component m of occupation i."

There is a formula given that

"estimates the skills transferability across 500 occupations based on the three-digit occupational codes and then aggregates these transferability estimates into 22 occupations in the two-digit level by taking the arithmetic mean within each two-digit category for the comparison with Shaw’s skills transferability. As a result, this study derives the 22 × 22 occupational skills transferability matrix based on two-digit U.S. census code, representing the estimated proportion of occupational human capital transferability."

There is an example given:

The data indicate that HR managers can apply only 16.01 of their 19.68 points utilized in their current profession into a new role as an industrial engineer. However, given differences in occupational skill portfolios, HR managers could apply all of their required knowledge in engineering and technology (1.03 points) and sales and marketing (5.99) in a new position as an industrial engineer. Summing the amount of shared points between occupations i and j (23.03) and dividing by the total points employed in occupation i (26.70) results in an estimated transferability rate of 86.30 % (t ij  = 0.863).

You could use this data and methods to then figure out the areas where someone's skills are lacking and make recommendations from those findings.


Is it ethical to claim experience with big data when the data isn't a part of the new advertising/social media/retail fad? Obviously most employers, when hiring a data scientist, would prefer experience with big data and/or data science.  But what can one safely assume they will acknowledge as experience?

Let's say someone often launches software on a computing cluster that typically generates an amount of data.  I'm not sure what the best measure of this data is for data science.  I'll call it one or two thousand rows, 200k or 300k points per row...certainly under 500k.  Then for each point, let's call it 25 or 30 values.  This amounts to 30 or 40 gig of data.  300 or 400 times of this and you can call it a study - maybe one or two studies per year.  I'm under the impression that this is much smaller than a data scientist at Google or Facebook would be used to, but it's certainly too big for my home computing systems.

If someone's been working with this for years (some people at this company have been doing it since before data science was coined/before social media existed), is it fair for them to claim big data experience?  According to this answer, it's not the amount of data but what needs to be done with the data that matters - is that a universally accepted opinion?

For what it's worth, working with this data entails manipulation/cleaning of the data with some proprietary languages, shell scripts, and a lot of Python.  A little bit of R but that's a more recent thing.  It involves tons of data visualization, drawing conclusions, and presenting to management/convincing decision makers.  Some of it involves trend determination, extrapolation, and comparisons made between data sets that aren't related in a straightforward way, so it sounds data science-ish to me.  But I will be the first to admit that I have a limited understanding of what data science currently is.

...bonus points if you can tell me whether or not this is an Easter egg or is the actual answer total for this site at the moment:

Edit:

I'll try to clarify.  What do data science employers acknowledge as experience with big data/data science?  Does the size of the data above qualify experience with it?  Or is it universally accepted among those in the field that it's not at all the size of the data, but what you need to do with the data?

Data can be big in various ways. It can be large N (observations) and small P (variables per observation), like access logs for example. It can be large P and small N, like in biostatistics where you have tens of thousands of gene expressions from tens or hundreds of people. Or it can be both N and P, like in Facebook data. Data can also be made big, by normalizing every single detail of it, meaning that you'll be able to make a massive relational database schema that actually represents simple data.

All kinds of big data needs different approaches and companies mostly care about their own version of big data. That's why the answer isn't black and white.

To answer the question you posed in title then I believe it's ethical to claim experience with big data as long as you can put it into context of the business or domain where you encountered it and can describe some situations where a different set of tools or skills was needed than is used for smaller toy examples.


How to measure the similarity between two text documents? Assume, I have 100 text documents, and I want to cluster those documents.

The first step is the construct pairwise similarity matrix 100X100 for the documents

My question is:

what are common way to measure similarity between two documents?

Thanks,

In general,there are two ways for finding document-document similarity

TF-IDF approach

Make a text corpus containing all words of documents . You have to use tokenisation and stop word removal . NLTK library provides all .
Convert the documents into tf-idf vectors .
Find the cosine-similarity between them or any new document for similarity measure.

You can use libraries like NLTK , Scikit learn ,Gensim for Tf-Idf implementation . Gensim provides many additional functionality .
See :  https://www2.cs.duke.edu/courses/spring14/compsci290/assignments/lab02.html

Word Embedding

Google's Doc2Vec ,which is available in Gensim library can be used for document similarity .Additonaly,teh Doc2Vec model itself can compute the similarity score ( no cosine or anything needed her ) . You just need the vectorise the docs by tokenizing ( use NLTK ) and make a Doc2vec model using gensim and fins similarity and many using Gensim inbuilt methods like model.n_similarity for similarity between two documents .

Other

Additionally,since your aim is to cluster documents,you can try Topic Modelling using LDA ( Latent Dirichet Allocation) or LSI ( Latent Semantic Indexing )  .


Seaborn Scatter plot I have a dataset containing of only one column, using matplotlib I was able to do the scatter plot the following way

data = pd.read_csv
plt.scatter(data.index,data.coulumn1)

I want the same graph using seaborn but I am not sure how to implement the same approach in the following line

ax = sns.scatterplot(x="total_bill", y="tip", data=tips)

ax = sns.scatterplot(x="total_bill", y="tip", data=tips)

With this format, Seaborn support Pandas DataFrame.
"total_bill" and "tip" are the name of the columns. "tips" is the pd.DataFrame.
You can plot on default Axis by
sns.scatterplot(x="X_Colname", y="Y_Colname", data=DataFrame)

#Can also pass the desired Axis to customize size etc.
sns.scatterplot(x="X_Colname", y="Y_Colname", data=DataFrame, ax=ax)


Is there any time series model which handles data at variable frequencies.? Goal: Predict the yellow points.(yellow events appear at varying frequencies)

But I'm struggling to find a good model to fit this use case.
Most of the time series algorithms are handling data which are at same frequencies(like per day/every 10 secs). I tried a lot of stuff but may be completely on the wrong approach.

Thanks a lot for any hint!!!

It looks like stock price data. You can use fbprophet(python),BATS(R),TBATS(R) to predict the data.
You said your model contains different frequencies which mean it contains more than one seasonality.
To find seasonality(frequencies) you can use Fourier transform. You should have significant data to get a good prediction


Process melspectrograms with convolutional neural network I am trying to do audio classification with a convolutional neural network. There are six classes. With librosa, I have created melspectrograms for the one second long .wav audio files. It returned 640x480 .jpg files.
My question is now how to proceed with the input, since I think it is too large as input for the network. If so, what would an adequate resolution be? Something around 60x60? Does it even have to be quadratic?

Options from my perspective:

Re-encode melspectrograms from librosa with smaller resolution
Use cv2 and simply do a cv2.resize() before passing it to the input layer.
Leave the resolution untouched, and introduce more convolutional layers.
?

Leave the resolution untouched, and introduce more convolutional
  layers.

This should be the next step. Two primary reasons for it :

This should reduce number of trainable parameters
Model can learn more abstract features


How do i cluster binarized categorial data, without knowing the number of clusters? I have a dataset of categorical data, and I need to cluster it without knowing k. I know algos for clustering data without knowing the number of centroids, like G-mean, but none works for categorial data.
I think that DBSCAN is also bad because of density.
My dataset is typically a marketing dataset, each row is a customer, and each column corresponds to an attribute, like eyes color. So, I need to binarize this dataset before any algorithm. Any ideas??

For categorical data, robust hierarchical clustering algorithm ( ROCK) will work better that employs links and not distances when merging clusters, which improves quality of clusters of categorical data. Boolean and categorical are two types of attributes that are most suited in this algorithm.

ROCK is a static model that combines nearest neighbor, relocation, and hierarchical agglomerative methods. In this algorithm, cluster similarity is based on the number of points from different clusters that have neighbors in common.

You can use CBA Package in R to perform the ROCK clustering.

Algorithm Steps:

Data----->Draw Random Sample----->Cluster with Links----->Label Data in DIsk

A random sample is drawn from the database
A hierarchical clustering algorithm employing links is applied to the samples
This means: Iteratively merge clusters Ci, Cj that maximise the goodness function
merge(point1,point2) = total number of crosslinks /expected number of crosslinks
Stop merging once there are no more links between clusters or the required number of clusters has been reached.
Clusters involving only the sampled points are used to assign the remaining data points on disk to the appropriate clusters

Hope it helps!!

For more details with examples, refer the following links:
https://www.cis.upenn.edu/~sudipto/mypapers/categorical.pdf
https://en.wikibooks.org/wiki/Data_Mining_Algorithms_In_R/Clustering/RockCluster


Verification of trained system I have trained a system in order to detect some features from a set of scenarios. Now the system can detect and classify that set. How can I validate how that system works in real world? What mathematics tools should I use?

Did you also validate the data set with a separate test set so you can see how it performs with known data?

This answer has a good run down on how to do that:
https://stats.stackexchange.com/questions/19048/what-is-the-difference-between-test-set-and-validation-set

It also is a best practice to ensure that you haven't overfit the data. The easiest way to do that is check the accuracy against the data that you used to train the data.  If its not substantially better than the test data (this data should never be used to train a model) then you have not over fit.

After that, if you eventually find out the correct classification in your system, I would compare the accuracy there to the accuracy you experienced in the training and test environments to determine if its performing as intended.


How do you know if your Autoencoder network is fully connected? I am new in deep learning and I am confused about fully connected network. Is an Autoencoder with more than one hidden layers a type of deep neural network (DNN)? Is DNN always fully connected? Let's say an Autoencoder network with three hidden layers was built using H2O R package, how do you know if the Autoencoder network that you build is fully connected?

Is Autoencoder with more than one hidden layers a type of deep neural
  network (DNN)?

Yes.

  Is DNN always fully connected?

No. For example Echo State Networks are sparsely connected DNNs.

  how do you know if the Autoencoder network that you build is fully
  connected?

I am not sure about R because I don't use it but for Keras/Python you can read the layer descriptions. The most common predefined layers (Dense, Convolutional, RNN, LSTM..) are fully connected. If you want to have a sparsely connected layer, then you can probably create a custom one using a Lambda Layer.


Linear regression with non-symmetric cost function? I want to predict some value $Y(x)$ and I am trying to get some prediction $\hat Y(x)$ that optimizes between being as low as possible, but still being larger than $Y(x)$. In other words:
$$\text{cost}\left\{ Y(x) \gtrsim \hat Y(x) \right\}  >> \text{cost}\left\{ \hat Y(x) \gtrsim Y(x) \right\}  $$

I think a simple linear regression should do totally fine.
So I somewhat know how to implement this manually, but I guess I'm not the first one with this kind of problem. Are there any packages/libraries (preferably python) out there doing what I want to do?
What's the keyword I need to look for?

What if I knew a function $Y_0(x) > 0$ where $Y(x) > Y_0(x)$. What's the best way to implement these restrictions?

If I understand you correctly, you want to err on the side of overestimating. If so, you need an appropriate, asymmetric cost function. One simple candidate is to tweak the squared loss:

$\mathcal L: (x,\alpha) \to x^2 \left( \mathrm{sgn} x + \alpha \right)^2$

where $-1 < \alpha < 1$ is a parameter you can use to trade off the penalty of underestimation against overestimation. Positive values of $\alpha$ penalize overestimation, so you will want to set $\alpha$ negative. In python this looks like def loss(x, a): return x**2 * (numpy.sign(x) + a)**2

Next let's generate some data:

import numpy
x = numpy.arange(-10, 10, 0.1)
y = -0.1*x**2 + x + numpy.sin(x) + 0.1*numpy.random.randn(len(x))

Finally, we will do our regression in tensorflow, a machine learning library from Google that supports automated differentiation (making gradient-based optimization of such problems simpler). I will use this example as a starting point.

import tensorflow as tf

X = tf.placeholder("float") # create symbolic variables
Y = tf.placeholder("float")

w = tf.Variable(0.0, name="coeff")
b = tf.Variable(0.0, name="offset")
y_model = tf.mul(X, w) + b

cost = tf.pow(y_model-Y, 2) # use sqr error for cost function
def acost(a): return tf.pow(y_model-Y, 2) * tf.pow(tf.sign(y_model-Y) + a, 2)

train_op = tf.train.AdamOptimizer().minimize(cost)
train_op2 = tf.train.AdamOptimizer().minimize(acost(-0.5))

sess = tf.Session()
init = tf.initialize_all_variables()
sess.run(init)

for i in range(100):
    for (xi, yi) in zip(x, y):
#         sess.run(train_op, feed_dict={X: xi, Y: yi})
        sess.run(train_op2, feed_dict={X: xi, Y: yi})

print(sess.run(w), sess.run(b))

cost is the regular squared error, while acost is the aforementioned asymmetric loss function.

If you use cost you get

  1.00764 -3.32445

If you use acost you get

  1.02604 -1.07742

acost clearly tries not to underestimate. I did not check for convergence, but you get the idea.


Calculating similarity where order matters How can I calculate a similarity (coefficient) where the order of the items matters and something like the Jaccard index would not be useful.

Specifically, I'm interested in comparing ingredients. Take a simplified apple pie ingredient list, for example:

Apples
Enriched Unbleached Flour
Palm Oil
High Fructose Corn Syrup
Salt
Eggs
Spices

And then compare that to:

The exact same ingredients, but in reverse order. Where spices is the top ingredient, and apples are the least used ingredient
The list, but with, palm oil replaced with vegetable oil and corn syrup replace with sugar

Even if you're naive about the fact that the replacements made in #2 are similar, the set of #2 is much more similar to the original than the set of #1. Is there an algorithm which would express that?

Not sure if this exists already, but maybe you could build a new similarity metric (let's call it $js$) with a few requirements:

If lists $A$ and $B$ are exactly the same, then $js(A, B) = 1.0$
If lists $A$ and $B$ are completely different (no items in common), then $js(A, B) = 0.0$
If all elements match but are shuffled, then $js(A, B) = 0.5$
If $|A| \neq |B|$, then the similarity metric should consider the items of the longest list (e.g. if there are few extra ingredients in $A$, but $B$ is identical otherwise, it will still have some penalization)

Then you can build a method that basically does the following (e.g. in python):

def js(A, B):
    cummulative_score = 0
    longest_list = B
    shortest_list = A
    if len(A) > len(B):
        longest_list = A
        shortest_list = B

    for index, element in enumerate(longest_list):
        if index < len(shortest_list):
            if element == shortest_list[index]:
                cummulative_score += 1
            if element != shortest_list[index]:
                if element in shortest_list:
                    cummulative_score += 0.5
    return cummulative_score / len(longest_list)

Then you would have something like this:

In [10]: print(l)
['Apples', 'Enriched Unbleached Flour', 'Palm Oil', 'High Fructose Corn Syrup', 'Salt', 'Eggs', 'Spices']

In [11]: print(l1)
['Apples', 'Enriched Unbleached Flour', 'vegetable oil', 'sugar', 'Salt', 'Eggs', 'Spices']

In [12]: print(l2)
['Spices', 'Eggs', 'Salt', 'High Fructose Corn Syrup', 'Palm Oil', 'Enriched Unbleached Flour', 'Apples']

In [13]: print(l3)
['Enriched Unbleached Flour', 'Palm Oil', 'High Fructose Corn Syrup', 'Salt', 'Eggs', 'Spices', 'Apples']

In [14]: print(l4)
['Apples', 'Enriched Unbleached Flour', 'Palm Oil', 'High Fructose Corn Syrup', 'Salt', 'Eggs', 'Spices', 'Ice Cream']

In [15]: js(l, l1)
Out[15]: 0.7142857142857143

In [16]: js(l, l2)
Out[16]: 0.5714285714285714

In [17]: js(l, l3)
Out[17]: 0.5

In [18]: js(l, l4)
Out[18]: 0.875

In [19]: js(l, ['Nutella'])
Out[19]: 0.0


Fitting as much points as possible on a line I want to fit a set of points on a 2D plane that looks like the blue points in the following picture. Instead of having a least square fit (the yellow, dotted line), I want a line that looks like the red line that either crosses the blue points as much as possible or is perpendicular to them. How can I achieve this?

I think as suggested in the comments, you should split this up in several linear regressions.

If the datapoints are ordered you could just do repeated linear regressions, always adding one point until the added point leads to a significant reduced goodeness of fit. Then you start the next linear regression until that one goes bad and you start the next one.
At the end you can compare the angles between the two lines and see if they are (roughly) 90 degrees.


Kmeans clustering with multiple columns containing strings I have the following dataset:
https://www.kaggle.com/carolzhangdc/imdb-5000-movie-dataset
What I want to find is clusters based on imdb score per genre per country. I have created a pandas data frame that contains per country for every unique genre the average imdb rating.
The dataframe looks like this:
country       object
genre         object
avgRating    float64
dtype: object

Since the columns country and genre contain strings, I can't use Kmeans for this.
Is there anyway I can achieve what I want?

After some more research we found this library: https://github.com/nicodv/kmodes.

The library k-modes is used for clustering categorical variables. It defines clusters based on the number of matching categories between data points. (This is in contrast to the more well-known k-means algorithm, which clusters numerical data based on Euclidean distance.) The k-prototypes algorithm combines k-modes and k-means and is able to cluster mixed numerical / categorical data.

Because the dataframe contains categorical data we can't visualize it in a scatterplot. So I added the number representing the cluster the row was assigned to, for every row to get some form of visualization.

Normally you can only cluster ordinal data, because clustering happens based on distance. So I don't know to what extent this is reliable.


how to calculate number of datapoints within a given time interval? I have a dataframe with one column which is a timestamp. I have converted that column to a datetime object using pandas to_datetime() method. However what I want is to count, for each time stamp, the number of timestamps that fall between this timestamp and prior 15 minutes.

  For example: If a timestamp is (2018,7,6,13,55) [year,month, day,
  hour, minute format], then I want to count the number of timestamps
  (or datapoints) that fall between this timestamp and the timestamp
  (2018,7,6,13,40).

It is easy to see what needs to be done, but it involves repeatedly going back and forth over the dataset. I have close to 750,000 datapoints and what I am thinking is not efficient.

How can I do it efficiently?

Any suggestion/hint would be highly appreciated.

Thanks.

The answer given by @Aditya is very good and educational despite the fact that the how is deprecated. So the right answer would be as follows:
df.resample("15T").agg({'count':'sum'})


Supported GPU for Pytorch Question
Which GPUs are supported in Pytorch and where is the information located?
Background
Almost all articles of Pytorch + GPU are about NVIDIA. Is NVIDIA the only GPU that can be used by Pytorch? If not, which GPUs are usable and where I can find the information?

That's correct, you need a NVIDIA GPU compatible with CUDA 8, 9 or 10: https://pytorch.org/get-started/locally/

NVIDIA has a list of compatible cards here: https://developer.nvidia.com/cuda-gpus#compute

Alternatively you could work on a GPU equipped cloud instance (or install pytorch without GPU).


For loop over a tensor in TensorFlow? I have a tensor of shape (1, M) where M is a multiple of 10. When I print the tensor it may look something like

Tensor("some_name", shape = (1, 80), dtype = float32)

This tensor is the output of a neural network which will be run in a session. I wanted to modify this tensor according to the following (broken)code:

for chunk_number in range(int(tensor.shape[1]/10)):
    this_chunk = my_fn(tensor[10*chunk_number : 10*(chunk_number+1)]) // my_fn is custom
    tf.assign(tensor[10*chunk_number : 10*(chunk_number+1)], this_chunk)

This is essentially like a custom activation function that selectively modifies the output.

However I cannot use a separate tf.Session() to run the tf.assign function because there will be a session running for the neural network. How do I go about this assignment?

Try updating to Tensorflow 2.0. With it you can fiddle with your network, when it's running.


Number of features of the model must match the input. Model n_features is `N` and input n_features is `X`. I am new to data science and trying get some results. I'm applying Decision Tree Classifier. When my train and test datasets' size are not equal I get an error `Number of features of the model must match the input. Model n_features is N (no. of entries in training datasets) and input n_features is X (no. of entries in test datasets).

If I have 100 entries in my dataset and parameter for split is test_size=0.30 as:

import pandas as pd
from pandas import Series, DataFrame
import numpy as np
from sklearn import tree
from sklearn.model_selection import train_test_split

data=pd.read_csv("ndata.csv")

X_train, X_test, y_train, y_test = train_test_split(data.dis, data.gen, test_size=0.30, random_state=42)

c = tree.DecisionTreeClassifier()

y_test_size = y_test.size
y_train_size = y_train.size
X_train = [X_train]
y_train = [y_train]
X_test = [X_test]
y_test = [y_test]

c.fit(X_train, y_train)

accu_train = np.sum(c.predict(X_train) == y_train)/y_train_size
accu_test = np.sum(c.predict(X_test) == y_test)/y_test_size

print("Accuracy on Train: ", accu_train)
print("Accuracy on Test: ", accu_test)

And the error occurs as follows:

ValueError                                Traceback (most recent call last)
<ipython-input-33-f6cc77390526> in <module>()
     24
     25 accu_train = np.sum(c.predict(X_train) == y_train)/y_train_size
---> 26 accu_test = np.sum(c.predict(X_test) == y_test)/y_test_size
     27
     28 print("Accuracy on Train: ", accu_train)

~/anaconda3/lib/python3.6/site-packages/sklearn/tree/tree.py in predict(self, X, check_input)
    410         """
    411         check_is_fitted(self, 'tree_')
--> 412         X = self._validate_X_predict(X, check_input)
    413         proba = self.tree_.predict(X)
    414         n_samples = X.shape[0]

~/anaconda3/lib/python3.6/site-packages/sklearn/tree/tree.py in _validate_X_predict(self, X, check_input)
    382                              "match the input. Model n_features is %s and "
    383                              "input n_features is %s "
--> 384                              % (self.n_features_, n_features))
    385
    386         return X

ValueError: Number of features of the model must match the input. Model n_features is 70 and input n_features is 30

Link to data file: https://gist.github.com/mutafaf/7715ad67bc3cf4e08985afefcc0ce08a#file-ndata-csv

Why do I'm getting this error. Is it necessary to have dataset size of both train and test equal?

You are supposed to pass numpy arrays and not lists as arguments to the DecisionTree, since your input was a list it gets trained as 70 features (1D list) and your test had list of 30 elements and the classifier sees it as 30 features.

Nonetheless, you need to reshape your input numpy array and pass it as a matrix

meaning: X_train.values.reshape(-1, 1) instead of X_train (it should be a numpy array not a list)

this is the entire gist:

data=pd.read_csv("ndata.csv")

X_train, X_test, y_train, y_test = train_test_split(data.dis, data.gen, test_size=0.30, random_state=42)

from sklearn import tree

c = tree.DecisionTreeClassifier()
c.fit(X_train.values.reshape(-1, 1), y_train)

accu_train = np.sum(c.predict(X_train.values.reshape(-1, 1)) == y_train)/y_train_size
accu_test = np.sum(c.predict(X_test.values.reshape(-1, 1)) == y_test)/y_test_size

print("Accuracy on Train: ", accu_train)
print("Accuracy on Test: ", accu_test)

I'm getting the following output:

Accuracy on Train:  0.8857142857142857
Accuracy on Test:  0.7333333333333333

Thanks for sharing the dataset. It was helpful for testing.


sklearn cross_validate without test/train split I'm running cross validation on my training data:

scoring = {'acc': 'accuracy',
           'prec_macro': 'precision_macro',
           'rec_macro': 'recall_macro',
           'f1_macro' : 'f1_macro'}
all_scores = cross_validate(clf, X_train, y_train, scoring=scoring, return_train_score=True)

This works fine but the output contains scores with keys "test_...". This suggests that the function is splitting the training data into train and test subsamples. This isn't what I want. I already have an independent test sample so I don't need to make a further split in this function.

Perhaps I'm just confused about the terms used by sklearn:

Case 1) The "test" scores are the result of running the metrics on the test set for only one cross-validation split(this is fine).

Case 2) or the "test" scores are the result of running the metrics on a completely independent test set(not what I want).

The "test" scores are returned as an array of three values in my case. This leads me to believe that the "test" scores are simply referring to the set of data that is independent from the training data for each cross validation split. And thus that test set is different for each cross validation split (this is fine and not a problem).

If the "test" scores are from a completely independent sample that is never used for training in this function, then it should only contain one value, right? But it could return three values if the metrics were run on this data for each independently trained cross validation model.

My gut says its the first case and that everything is fine. Can anyone clarify this for me?

Its the first case - there are 3 test_scores since the default setup is 3-fold cross validation.
Which means that the training set is split into 3 subsets and at each cross validation step the model is trained on 2 subsets and tested on the one subset left out.

Its all in the docs


Time Series Classification for 1 hour blocks I am doing some analysis on time series.
The time series would consist of 3 channels and contain 5 minute interval data.

What I want is to be able to give it a 1 hour block of 5 minute interval data and it will categorise it based on the entire one hour and picking up some patterns how the time series looks for each of the categories as per the training data.

I have many 1 hour series of 5 minute interval data which is classified to a particular category, and I want to be able to have a deep learning model which can detect the pattern between these samples and be able to determine for new samples which categories they belong to.

****Could you please recommend a type of deep learning model which is capable of this?****

Maybe I don't understand LSTM's but to my understanding they provide a prediction for each point in a time series based on the points that occur before it and would therefore give a series of predictions, where as I want 1 prediction for each hour.

I appreciate any help that you can provide to help me understand this better.

Thankyou.

Correct me if I am wrong, but the problem you describe here sounds like a classification problem, not a times series forecasting. You, just want to know to what class each 1 hour of data belongs to. If this is the case, you can try using a CNN with 1 dimensional convolutions and 3 channels.


Evaluate document similarity / content-based recommender system I'm planning on building a basic content-based recommender system with word2vec and cosine similarity.
The data consists of 300k documents in varying length.
How do I evaluate my model if I have no labels / categories whatsoever?

When you don't have label/categories then it's called Unsupervised Learning. You can solve this problem via Latent Dirichlet Allocation (LDA) model and then evaluate your model by splitting the texts in half and compare the topic assignment for each half using cosine similarity. The more similar the topic assignment, the better.
Example