Pearson correlation method using absolute values and relative values I have a dataset with election results and crime rates per city. For each variable I have an absolute value (i.e. Total votes, Total crimes) and a relative value (i.e. Percentage shares of votes).

I want to calculate the correlation coefficient for some variables, but in the process I had a question about what value I need to use, if relative values or absolute values.

First I calculated z score for absolute values and then I calculated the correlation using excel. I also used pandas.DataFrame.corr() and pearsonr from scipy.stats.stats in python, in order to corroborate results.

For example, if I use absolute values I will get a positive correlation between candidate 1 and candidate 2.

x = df['Abs Cand 1'].tolist()
y = df['Abs Cand 2'].tolist()

print (pearsonr(x,y))
(0.95209664861187004, 0.0)

However, if I use relative ones I will get a negative correlation:

x = df['Rel Cand 1'].tolist()
y = df['Rel Cand 2'].tolist()

print (pearsonr(x,y))
(-0.99704737036262991, 0.0)

I was confused when I saw both results, and now I need some orientation to understand those differences.

Thanks in advance!

In general, the correlation coefficient is "invariant to separate changes in location and scale in the two variables". In particular, you can mix relative with absolute values.

However, that only works if you scale the variables globally. You can't scale every individual data point (here on a city level). If this was a county wide election, you could scale the city values by the county population.

But it sounds like your crime rates are on a per city level. In this case you should scale the votes on a city level as well to make them comparable. This will change the correlation coefficient and give a different result than with absolute values. I think using percentages is more intuitive in your case.


What is the best way of combining audio and visual data to make predictions? I am trying to predict the probability of a disease by using audio and images, the audio and the images do not come from the same source. I am thinking of combining the outputs (maybe average them) of two models one utilizing image data to calculate the probability the other using audio data. I am feeling unsure of this approach as I will not be able to verify the final combined output. Will the outcomes be reliable, or should I use a different approach?

You can try to fuse the image and audio data. You can fuse in different ways:

1) Early fusion - fusion at feature level

2) Late fusion - fusion at output level (similar to your idea)

Here is some papers for your reference:

Look, Listen and Learn
https://arxiv.org/pdf/1705.08168.pdf

Learn to Combine Modalities in Multimodal Deep
Learning
https://arxiv.org/pdf/1805.11730.pdf


Identify members who are likely to switch where they receive drug administration I have access to medical claim data from a large health insurance company. As some of you may know there is a large delta between the price of drug X depending on where it is administered.
My company wishes to help members to reduce the cost they pay in deductibles, co pays, and coinsurance as well as reducing the cost for the health play by identifying the lowest cost of care typically home infusion or physician administration.
I am researching various approaches and customer segmentation modeling seems like the best approach to this sort of problem.
The claim data goes back 3 years and is at a member level. For each member claim, there could be many records in the DB. One line could be for 2hrs of chemo admin, another for the drugs they were provided, others for nurse's time, etc. Each row specifies where the site of care was i.e. home infusion, physician, outpatient, specialty pharmacy. Each claim also provides details on the primary diagnosis I.e. rheumatoid arthritis (R.A) etc.
Ideally, I would like to identify those members who use expensive sites of care when there are cheaper alternatives available. I was thinking for example that for primary diagnosis for R.A I could filter down to R.A members using the diagnosis code and then cluster those members based on certain input features such as demographics, distance to home infusion center/hospital/physician, level of cover, frequency of refill, dosage, cost of a drug, the current site of service, etc.
Is my approach right or is there a better way to solve this problem other than clustering?

Given that the data is labeled, just perform supervised approaches, they will almost always beat unsupervised.

Intuition why thats the case is because we dont have target function in unsupervised approach. In other words function that discriminates classes given our data set. I like to think that in unsupervised learning this function is identity function and not (what in reality is) some complex one. Given an entire data set, we are asking for a partitioning of the input space, with particular properties (the partitions contain almost all examples, each partition is not too large, the partitions are not too close to one another) but this kind of partitioning does not map to the real classes (since we are expecting classes that are clearly seperated in the dataitself but in reality they have some non-linear seperation, and this non-linear/complex behaviour is represented in terms of some function. That supervised approaches approximate (you can think of almost every supervised approaches as an optimisation))


Apply SVM on LDA in python hope someone kindly put time here,

my approach is like this:
TFIDF -> LDA -> SVM

I am using LDA to extract topics. I want to do topic modelling and use the topics as features to do document classification.

I know that I have to send the feature vector to SVM,
but my question is that how can I make this feature vector to send to svm?
Is featureValue here is the probability assigned to each word?
My question is kind of the step 3 in the below link that I dont know how to do it

svm on lda

but in this link there is no implementation, that is just explanation.

Many thanks for your time:)

Following your code in your other thread:

# Use tf (raw term count) features for LDA.
print("Extracting tf features for LDA...")
tf_vectorizer = CountVectorizer(max_df=0.95, min_df=2,
                                max_features=n_features,
                                stop_words='english')
t0 = time()
tf = tf_vectorizer.fit_transform(data_samples)
print("done in %0.3fs." % (time() - t0))

print("Topic modelling with LDA...")
lda = LatentDirichletAllocation(n_topics=n_topics, max_iter=5,
                                learning_method='online',
                                learning_offset=50.,
                                random_state=0)

lda_x = lda.fit_transform(tf)
# so lda_x is your doc-topic distribution that you can use for feature vector to your SVM model.
# lda.components_ is your topic-word distribution.

Hope this helps!


Simple Explanation of Apache Kafka Can anybody explain Apache Kafka for me in a plain language? I'd appreciate an explanation with a practical example instead of abstract theoretical definitions, then I can understand better.

What is it used for? What does messaging mean? Messaging between what?!! At which stage of a BigData analysis is it used?

And what are prerequisites for learning it?

PS:
Please explain as you would explain for a non-technical person

Can anybody explain Apache Kafka for me in a plain language?

As the official documentation of Kafka says:

  Apache Kafka is publish-subscribe messaging rethought as a distributed
  commit log. It follows the publish-subscribe messaging style, with speed and durability built in.

Now, let me give a dumbed down explanation of the same:

A publish-subscribe(commonly called pub-sub) architecture basically contains a publisher(who sends the messages, for example: A twitter stream) and a subscriber(who receives the messages Ex: the analyst/the user's activity stream)

A clearer explanation would be:

  In software architecture, publish–subscribe is a messaging pattern
  where senders of messages, called publishers, do not program the
  messages to be sent directly to specific receivers, called
  subscribers, but instead characterize published messages into classes
  without knowledge of which subscribers, if any, there may be.
  Similarly, subscribers express interest in one or more classes and
  only receive messages that are of interest, without knowledge of which
  publishers, if any, there are.

And keep in mind that Kafka is a distributed pub-sub messaging system, designed to scale.

Now,

  At which stage of a BigData analysis is it used?

It is basically used in the Extraction step of the ETL pipeline. It can contain high volumes and speed of data flow (we call it high throughput in technical terms).

It can store messages also, but it is a persistent store; which means that the data is not stored forever. The stored data has an expiry date.

  And what are prerequisites for learning it?

This is a tough question, as everyone who can program can get started with it, but if you really want to implement this in your architecture, then these are the pre-requisites which I can think of:

Know what is a persistent data store and where to use them
Know what a pub-sub messaging system is
Know what an extraction step is, in ETL. And also know how to deal with data with huge volume coming at you at high velocity.

Further reading:

Kafka has an excellent documentation, which can help to get started with it.


ARIMAX v. ARX Time Series Modeling I need to build a time series model with explanatory variables, and ARIMAX seems to be the one that comes up most frequently in practice, based on my survey of related work.

I know ARX solves a similar problem, but I'm having trouble wrapping my mind around the practical differences in what an ARX representation of my data would have compared with an ARIMAX approach.

I know ARX lacks the moving average component, but I'm curious whether anyone can point me toward some best practice for choosing one approach over the other.

Are there certain characteristics in my data I should look for to make an informed decision?

Yes, there are characteristics. You can use a correlogram to inform you as to the error structure in your data. That will tell you whether or not your data needs to account for AR and/or MA terms. Also check for unit roots to tell you whether you need to difference the time series.

This link has a good introduction to analyzing the correlogram that shows the pattern for AR(1) process. It is charecterized by a declining or oscillating but declining values. Depending on whether or not you have positive or negative autocorrelation.

The ACF for an MA(1) process is below. It is characterized by one significant value and then non-significant oscillating values thereafter.

To tell if you need to do differencing you should use a test like the augmented dickey-fuller test to check for the unit roots (i.e. your data is integrated).


How can I generate reasonable dummy/artificial data from a pre-existing time-series data? I have a dataset like this, basically all numerical time-series data.

I would like to generate dummy/artificial/fake data for future values of this, preferably in python. How can I achieve this for reasonable looking future values?
I checked some packages like faker, Timesynth; but couldn't figure out how to produce data which considers previous data.

This problem is referred to as scenario generation, commonly used in stochastic optimization. The key problem is that you have to respect correlations among the different variables, which means that you need to model the joint multivariate predictive distribution. One way to bypass this is to use Copulas, which only require modelling the marginal probability densities. An approach widely used in engineering is the following:

Model the marginal (empirical) probability distribution of each variable separately. You can also model the conditional probability distribution using probabilistic forecasting if needed.
Estimate the empirical correlation matrix between the time series.
Use the empirical correlation matrix and the marginal conditional distributions to fit a multivariate Copula.
Obtain correlated scenarios by sampling from the multivariate Copula.

I am including a standard reference from the field of wind forecasting, which models the temporal correlation from the same time series. This can be applied to the case of different time series (spatial correlation).


What are the prerequisites before running Holt Winters Model? I just read Demand-Driven Forecasting: A Structured Approach to Forecasting(Wiley and SAS Business Series) and have a few doubts in Holt-Winters Model:

1) Unlike OLS Regression Modeling technique or ARIMA, no assumptions were checked before running Holt-Winters. For instance, in ARIMA, we first make the data stationary before running ARIMA or in OLS, we check normality, auto-correlation etc. However, as per the book, no test was conducted before and after running Holt-Winters. We just calculate MAPE and check if it is acceptable. So, can someone confirm if there are any tests that we should do before and after running Holt-Winters?

2) Can Holt-Winters model incorporate causal factors such as Price, Promotion, Marketing etc?

3) When we are running Holt-Winters multiplicative or additive model, we don't need to explicitly adjust for seasonality before?

4) How to treat for missing values or outliers in time series forecasting? We obviously cannot remove missing values rows as the time interval will break. Hence, for missing values, I have thought to use CAGR for that period(weekly, monthly or yearly) in the last 5 years and use it to estimate missing value. Can someone confirm if this correct?

Also, I believe we shouldn't remove outliers and rather look why it is there in the first place. It might be because of some external factors like Price Change or Promotion which is causing the spike. This will ofcourse be not detected by Holt-Winters. So, if it is a rare promotion, then we should remove that value and insert the normal demand again by using CAGR of past data in order to maintain the historical pattern. Hence, the spike will be gone and the historical trend will be intact, thus increasing our MAPE. Can someone confirm if this correct?

1) Holt-Winters doesn't require stationarity the way an ARIMA model does, so you don't need to perform any of the steps you mention. However Holt-Winters is specifically designed for seasonal data, so don't bother applying it unless you have good reason to think your data is seasonal.

2) In the basic formulation of Holt-Winters, no, it can't. More recent implementations of Holt-Winters use the state space approach to time series modeling, and it might be possible to do so using the state space approach - although even that is still in debate. BSTS is another state space model with a structure similar to Holt-Winters, and that approach does allow for causal factors. You should check that.

3) No, you don't need to. The whole point of Holt-Winters is that it accounts for seasonality, so you don't need to adjust for it.

4) See here. Holt-Winters can't handle missing values, but other methods can. You should be careful with Outliers, since you don't know whether an outlier is a "real" outlier, a seasonal spike, or a causally driven event. For example in a sales time series, your spike might be a true outlier, or it might be an event like Black Friday, or it might be the result of a promotion and advertising campaign.


Python TypeError: __init__() got an unexpected keyword argument 'decision_function_shape' I tried creating a SVM Classifier, as:

# Create a SVM Classifier
model = SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
            decision_function_shape=None, degree=3, gamma='auto', kernel='linear',
            max_iter=-1, probability=True, random_state=None, shrinking=True,
            tol=0.001, verbose=False
            )

(Using Python 2.7)

But getting this error--

  TypeError: init() got an unexpected keyword argument
  'decision_function_shape'

Any thoughts on that? How to sort it out?

Update >>

My sklearn version is 0.16.1. I tried to install the update but it's kept on saying- No matching distribution found for the upgrade.

Your snippet is almost exactly the same as scikit-learn's example (except for kernel='rbf' and probability=False) and works fine under version 0.18, provided the needed imports are present.

Update: the version of scikit learn used is 0.16.1, and in that version, SVC did not have as many arguments as in 0.18, as per the docs. You should therefore use something like this:

model = SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0, degree=3,
            gamma=0.0, kernel='linear', max_iter=-1, probability=True,
            random_state=None, shrinking=True, tol=0.001, verbose=False)

For upgrading scikit-learn to 0.18, (assuming you use pip) do this:

pip install scikit-learn==0.18 --force-reinstall


Data visualization of frequencies of state transitions (possibly in R?) I am working on some experimental data, which can be of types A, B and C. Now I observe this data for 5 time points, and I can see them move between A to B, B to C,... etc.
I see such transitions for a number of independent data points, and I have the cumulative frequencies from all data.

For example, I have:
$$
  Period  A   B  C           \newline
  1       4   4  2
  2       1   2  7
  3       0   1  9
  4      10  0  0
  5       8   1  1
$$

I DO know the transitions from one state to another, for example from A->B, B->C so on and so forth. For example I know that from Period 1, (all A's went to C. Among the missing B's one went to A, and rest to C.)
I was thinking of what would be the best way to visually represent this time wise transitions from one state to another. I was thinking that there might be some better way than just having a transition matrix, maybe something that looks like a Markov Chain but which could accommodate all the 5 periods of transitions in a succinct way?
I myself work on a statistical software called STATA, which has limited graphical applications. IS there something on other software packages (R maybe?) which can help me in this?

Sorry for the hack representation of the data matrix.

How about a Sankey diagram with time on the x-axis and flow width representing state transition frequency. Here is a SO discussion on implementing Sankey diagrams in R.

One possible R package is {riverplot}... here is code showing the first transition in your data:

library(riverplot)
nodes <- as.character(sapply(1:2, FUN = function(n){paste0(LETTERS[1:3],n)}))
edges <- list(A1=list(C2=4), B1=list(A2=1,C2=1,B2=2), C1=list(C2=2))
r <- makeRiver( nodes, edges, node_xpos= c( 1,1,1 ,2,2,2),
                node_labels= c( A1= "A", B1= "B", C1= "C", A2="A",B2="B",C2="C" ))
plot( r )

Will produce this:


Does K - Means clustering on data reduced using PCA and the original data make any difference? I am working on clustering and I have 90 features with 13500 data points and after removing the correlated variables which had pearson correlation more than 90% my feature space reduced to 70. Also, almost all my original 90 features has lot of values as zeros (more than 70-80% of data points). What I did in terms of algorithmic implementation is :

Ran K-Means on original data with 70 features (all numerical) by selecting number of clusters based on Silhouette index.
Ran K-Means by reducing the dimensions to 2 by selecting the number of clusters based on silhouette index.

What I observed and my corresponding question is :

K-Means on pca reduced data gave better clusters. Is there any way I can use this clusters that would make sense ? Like assigning the cluster labels from pca reduced data to the original data
How is the K-Means on original data and K-Means on pca reduced data different ? I understand that the pca would have reduced the data to two dims that I chose and has preserved the components with max variance. But can I assign the cluster labels from the pca reduced data to the original data ? would it be a right approach ? I guess not.

Also, there are many implementation of K-Means like Lloyds (Python), Elkan(Python), Hartigan-Wong(R), Forgy(R), MacQueen(R). Which of these can be used for numerical vars and which one for categorical ? I think, wong is used for categorical variables not sure though. Also, which of these Implementations can I simply rule out ?

But can I assign the cluster labels from the pca reduced data to the original data ? would it be a right approach ? I guess not.

Yes, that is totally the right approach. Principal components are just the linear combinations of your original features that explain the most variance, so you can definitely use them for clustering. Moreover, since you only kept 2 of them you also removed a lot of noise from your data - since K-means is based on distances, the fact of having 70 features can be problematic, as you're equally weighting distances on useless features and distances on important ones.
Summing up - yes, assign the cluster based on the results of Kmeans you obtained on the Principal Components.
Unfortunately I cannot help you with the implementations, but remember then when mixing categorical and continuous variables you need to define an appropriate distance metric between your points. And any good implementation should allow you to do so for categorical only or continuous only data. Not sure about the mix.


Two-class model with predicted scores needed - classification or regression approach In my problem, step one is to build a model to classify cases as one of True or False (1 or 0 could also be used obviously). Once the optimum model is found, step two is to retrieve probabilities for these predictions and use these to work out the optimum threshold (based on an accuracy measure) to classify my data thereafter.

So I know for step two I need to use a classification accuracy measure e.g. f1 score to work out the optimum threshold however, I am debating on the best approach for step one. My initial thought was to do a two-class classification model. The model I'm using is a Radial SVM implemented in R. My issue with this however, is using caret, how I get predicted scores once the optimum model is found with classProbs=T (full code below). I don't know if this is ideal as I read somewhere (can't find link now sorry!), to do this, caret actually runs a second model which doesn't seem like a great idea to me. The only information on this is the package documentation seems to be

  classProbs a logical; should class probabilities be computed for classification models (along with predicted values) in each resample?

I have also searched SO for another way to get predicted values using caret but have had no success.

My other idea was to use a regression model for step one using, 0 and 1 as the only y values and then perhaps implementing a custom error function that allows for predicted values outside of 0 and 1. My thought here is that I don't necessarily want to penalise a model if the prediction is above 1 and the actual is 1.

I'm not sure which approach is better or if there is a third way I'm not thinking of, any help/suggestions would be great! I think the first approach is probably better if the class probabilities calculation is not done through using a separate model so if anyone has found further documentation on what goes on with this setting that would also be great!

svm_tests <- train(x = x_train,
                   y = y_train,
                   method = "svmRadial",
                   scale=F,
                   tuneGrid = expand.grid("C"=0.1,"sigma"= 0.05),
                   trControl = trainControl("repeatedcv",  #Do 5 fold cv, repeated 5 times with different seeds
                                            repeats = 5,
                                            number = 5,
                                            seeds = seeds,
                                            summaryFunction = m.c.c,
                                            classProbs=T, #Used to get probabilities when using predict
                   ))

scores_svm <-  predict(svm_tests,
                       x_valid,
                       type = "prob")

This is a classification problem, not a regression problem, so your first instinct was correct. For most models, you can easily get the probability estimate for each prediction. If you did tree based methods or a linear model, you would have probabilities easily. However, because of how an SVM works, it does not automatically output a probability estimate. It makes sense that Caret has to do some extra computations to get your probabilities.

For the specifics, see: https://stats.stackexchange.com/questions/335527/what-are-the-predicted-probabilities-from-an-svm


Measuring impact of missing feature on model performance Is there any general approach to approximate how a missing feature will impact the prediction performance of a regression model?

For example, if I train a model using 10 features, but want to make predictions afterwards only providing 8 of those 10 features - simply imputing the missing values -, how does that (statistically) affect my target metric. You could also think of it like some kind of confidence decline of the predictions.

I wondered if there is some approach like deriving / estimating the deterioration of performance from feature importances, or similar. I did not find any resources about that though, nor could I come up with some meaningful idea myself so far.

I would just remove each feature individually and see how that effects performance on my test set.  No need to get fancy.


How can one use a validation set to reduce overfitting Naive Bayes? What is the correct procedure for using a validation set to reduce overfitting?

Say I split the data 80:10:10 (training: validation:test). I train on the training set then get 90% accuracy. I apply this model to the validation set then get 20%. What do I do then?

How can the validation set be used to reduce overfitting especially with reference to Naïve Bayes?

It is not much about the algorithm you use. It is about the fact that you learned so much details from your training set so just tune the parameter inside a loop in which training and validation errors are calculated each time. In NB case, you do not have many parameters in that sense. Probably the features can be inspected instead of parameter. Then see the famous point at which the validation error is the minimum.

Do not use fixed splits. For each calculation in loop do $n$ times splitting + evaluation and take the mean and std of errors. Gives you better impression about the stability of the results and effectiveness of your algorithm.


After the training phase, is it better to run neural networks on a GPU or CPU? My understanding is that GPUs are more efficient for running neural nets, but someone recently suggested to me that GPUs are only needed for the training phase. Once trained, it's actually more efficient to run them on CPUs.

Is this true?

This depends on many factors, such as the neural network architecture (CNNs tend to be better optimized than RNN on GPU) as well as how many test samples you give as input to the neural network (GPUs can be even faster when given a batch of samples instead of a single sample).

As an example, here is a benchmark comparing CPU with GPU on different CNN-based architectures. The forward pass is much slower on a CPU in that case:

FYI: Benchmarks based on neural networks libraries to compare the performance between different GPUs


Is it a good idea to combine fine tuning and feature extraction techniques? I have a normal/tumor medical images dataset and, for the same patients, also the relative genomics, and my goal is to predict if a patient has a tumor by combining all the information.
To achieve this, I am using a ResNet50 with imagenet weights to extract features from images, and other methods to extract features from genes. I join the two features and use an SVM to make a prediction.
The accuracy isn't extremely bad, but I wanted to know if it can be increased by performing a fine-tuning over the same images, in place of using a network only trained on the imagenet datasets as it is.
I have researched papers but I found literally nothing, neither in favor nor in contrast with this hypothesis.
Is there some known contraindication?

The concept of multimodal learning is relevant: in this case, combining data from two modalities: 1) image signal using ResNet50 and 2) genomic features extracted from genes. Multimodal learning for extending state-of-the-art performance of pre-trained unimodal models is currently an area of active research in the literature. In the NLP domain, the paper on Multimodal Bert describes combining a pre-trained language model with additional signals from visual and auditory inputs for improved classification performance.
In the specific scenario mentioned here, combining the image and non-image signals into a single deep neural network graph would allow for additional fine-tuning. It is possible to first train the layer combining signals from both modalities, then later fine-tune all weights in the graph (with a lower learning rate) for improved performance. This procedure discussed a bit in the Keras Guide to Transfer Learning.


Finding correlation between MNIST digits What way would be correct to calculate the correlation between say digit '1' and digit '7' images from MNIST? Taking average values of all digit '1' pixels and digit '7' pixels to compute correlation between those would be a correct?

You can’t. Correlation is a measure of a variable changes as another variable changes. One goes up by a certain amount, the other usually goes up too: positive correlation. And so on.

What you can calculate is how similar or how different images of 1s are, compared to images of 7s. You could average all images of each, by summing the images to get one images with very high pixel values, and then dividing all pixel values by the number of images that you summed.

Then you can represent the average 1 and the average 7 as a long vector of 784 pixels and calculate the distance between these two as a measure of their similarity.


Entity linking vs aliasing The process of finding entity in a knowledge base (KB) that a given keyphrase in a text refers to is called entity linking. I have the
opposite problem. I have an entity in my knowledge base (KB) and I want to find all the ways people might refer to this entity. For instance, I have "Madonna" (singer) and I am looking for aliases like "Louise Ciccone", "Madonna Ritchie", "Queen of Pop", "Mo", etc.
Is it called aliasing? Or there is a better name in the literature?
I guess finding the right key words will help me find related research.

One option is to call them synonyms, a word or phrase that means exactly or nearly the same as another word or phrase.
Entity linking is mapping words of interest to corresponding unique entities in a knowledge base. It is useful to think of the unique entities in knowledge base as hashes and all surface forms as synonyms.


Why do we need XGBoost and Random Forest? I wasn't clear on couple of concepts:

XGBoost converts weak learners to strong learners. What's the advantage of doing this ? Combining many weak learners instead of just using a single tree ?
Random Forest uses various sample from tree to create a tree. What's the advantage of this method instead of just using a singular tree?

It's easier to start with your second question and then go to the first.

Bagging

Random Forest is a bagging algorithm. It reduces variance.

Say that you have very unreliable models, such as Decision Trees. (Why unreliable? Because if you change your data a little bit, the decision tree created can be very different.) In such a case, you can build a robust model (reduce variance) through bagging -- bagging is when you create different models by resampling your data to make the resulting model more robust.

Random forest is what we call to bagging applied to decision trees, but it's no different than other bagging algorithm.

Why would you want to do this? It depends on the problem. But usually, it is highly desirable for the model to be stable.

Boosting

Boosting reduces variance, and also reduces bias. It reduces variance because you are using multiple models (bagging). It reduces bias by training the subsequent model by telling him what errors the previous models made (the boosting part).

There are two main algorithms:

Adaboost: this is the original algorithm; you tell subsequent models to punish more heavily observations mistaken by the previous models
Gradient boosting: you train each subsequent model using the residuals (the difference between the predicted and true values)

In these ensembles, your base learner must be weak. If it overfits the data, there won't be any residuals or errors for the subsequent models to build upon. Why are these good models? Well, most competitions in websites like Kaggle have been won using gradient boosting trees. Data science is an empirical science, "because it works" is good enough. Anyhow, do notice that boosting models can overfit (albeit empirically it's not very common).

Another reason why gradient boosting, in particular, is also pretty cool: because it makes it very easy to use different loss functions, even when the derivative is not convex. For instance, when using probabilistic forecast, you can use stuff such as the pinball function as your loss function; something which is much harder with neural networks (because the derivative is always constant).

[Interesting historical note: Boosting was originally a theoretical invention motivated by the question "can we build a stronger model using weaker models"]

Notice: People sometimes confuse random forest and gradient boosting trees, just because both use decision trees, but they are two very different families of ensembles.


How to label a dataset for Machine Learning? I have a collection of educational dataset.
The dataset consists of a username and their review for the course.
I want to analyze the data for sentiment analysis.

How can I label the data to train the model for my supervised machine learning model?

Do I need to label manually (Positive, negative, neutral) to train the model?

If you have a smaller dataset, then I think manual labelling could work.

For larger datasets, you need to use a sentiment analyser engine which has been retained on a very large dataset.

For example, you can use TextBlob,

from textblob import TextBlob
obj = TextBlob( some_text )

print( obj.sentiment.polarity )

You can one by one load each review and sort them into three classes or continuous values.

You can classify the review into a categorical variable. For example, if the polarity is above 0 then the review is positive else it is negative. Here, you will perform classification.
Use the values from TextBlob as they are. Hence, they will form continuous labels. Here, you will perform regression.

Tip:

Also, you can search for similar packages which provide sentiment score for a text. You can use Natural Language Processing APIs.

import urllib

data = urllib.urlencode({"text": "I'm a very good boy "})
u = urllib.urlopen("http://text-processing.com/api/sentiment/", data )
the_page = u.read()

print (the_page)


Cross validation Vs. Train Validate Test cont'd Cross validation Vs. Train Validate Test

I have one further question relating to @louic answer in the post above:

"Training happens k times, each time leaving out a different part of the training set. Typically, the error of these k-models is averaged. This is done for each of the model parameters to be tested, and the model with the lowest error is chosen."

When we say "the model with the lowest error is chosen" which of the following does this mean:

we choose the parameters which has the lowest averaged error and then find the the model with the absolute lowest error from within the K models
we choose the parameters which has the lowest averaged error and then train a new model - on the entire training set, using these parameters?
we choose the parameters which has the lowest averaged error and then take the average weights of the K-models trained
Thanks

Iain

The answer in the linked question says to use approach 2. Specifically:

Run cross-validation on the train set separately for each choice of hyper-parameter values.
On each run, average the score across the k folds to produce a final cross-validation score.
Choose the hyper-parameters that give us the best cross-validation score.
Retrain a model with those hyper-parameters on the entire training set.
Test this model on the test set, which has not been used at all until now.


How to create a classification model for multi output dataset? I have a dataset where there are two target variables target-1 and target-2.
Both target variables are ordinal and thus I want to develop a multi-class classification model over these two target variables.
However, I am unsure as to where to start in the model creation process.
Therefore, if someone could give some pointers to this, that would be greatly appreciated.
Thanks in advance.

So the question asks how to create a model such that you take in input X and translate this into two simultaneous predictions, one which predicts the breed_category and pet_category.
Depending on the number of the classes within the two targets (breed_category and pet_category), you could simply concatenate the outputs as one-hot encoded vectors (https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/).
So, you would encode the categories for breed_category and pet_category into one-hot encoded vectors and then concatenate them together.
By representing the output like this, you can use a regular feedforward neural network architecture, with a final sigmoid output layer, which maps the output into values between 0 and 1. In this case you cannot use softmax, since you are now predicting two targets simultaneously.
If the number of classes from both targets is large, it might be better to have separate feedforward models, one for breed_category and pet_category.
For this, you can then use a softmax final output layer.


Predictor Variable vs. Target Variable [Update] Multiple target variables can exist.

If I find that a variable is not a predictor variable, does that mean it automatically becomes a target variable?

The question you should be really asking is,

  What is your problem statement? What are you trying to predict?

It will then be automatically clear what your target variable is.


Best Approach to Forecasting Numerical Value Based on time series and categorical data? Consider a dataset of thousands of car repairs that have been performed. In simplest of terms, the columns to consider are the time of year when it was broken (seasonal changes in demand for car repairs), type of damage to car (some damages take longer to repair than others), and the type of car (some cars are more difficult to work on).
I am inquiring as to the best fit for trying to model data of this format where you are predicting the repair time based off of timeseries and categorical data as the inputs. Keep in mind that the data does not have a constant period.
Example Column Names:
datetime | Type of Damage to Car | Type of Car | Repair Time
Any suggestions?

So the question is about to model the next repair date given the previous repairs.
If you have customer-specific data, where you have logged customer repairs to a specific customer, then it would be good idea to do time series, provided you have enough instances of customer repeatedly coming back.
If you have this scenario, then you can use something like an RNN (Recurrent Neural Network [https://towardsdatascience.com/understanding-rnn-and-lstm-f7cdf6dfc14e]) or LSTM, whereby you feed in the date time, type of damage at every time step to then get a single output date time.
If you have not logged a customer ID to the repairs to identify which repairs have been carried out for particular customers, then you could easily use a standard neural network for this.
Here you would train the model on all input data with the goal of getting a date time which equates to when the next repair is likely to happen.
In terms of data representation, you could represent the following features as the following:

Datetime: here, you could this as the day in the year, for example, Jan 1 could be represented as 1 and Dec 31 as 365. This can then be normalised (i.e. divided by 365) to reduce the scale of this feature.
Categorical variables like type of damage, etc: normally we represent these as one-hot encoded vectors (https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/).

To make the input for a given entry, we then concatenate these features into one massive vector.
Good luck!


Why Generator error increases after some epoch I'm training Realtivistic GAN on dog images. I don't understand why my generator loss increased after 100 epoch.

This is the generator loss graph.

In your question, it's pretty hard to answer specifically what is going on as you don't share any details of what you're trying to do. However, I'll try to address the general case:

In GANs (and other adversarial models) an increase of the loss functions on the generative architecture could be considered preferable because it would be consistent with the discriminator being better at discriminating.

As the discriminator is trained, it actually changes the loss landscape of the generator, which in turn increases the loss of said generator. This is probably why.

Another reason could be that the discriminator is getting too good and the generator is getting stuck, which often can be fixed by adjusting learning rates in the optimizers or increase the number of iterations of the adversarials. In this particular case I'd try decreasing the discriminator learning rate or increase the number of steps the generator is trained.

I'd suggest reading the original Generative Adversarial Nets paper and perhaps also the Improved Techniques for Training GANs paper.


Feature Selection algorithm/library for CRF I am using the Conditional Random Fields CRF suite scikit-learn wrapper algorithm. I have read on the literature various approaches for feature selection, but I cannot find any on that package or, generally, available ones for CRF. Would you know any libraries (Python preferred) or easy to implement algorithms for this purpose?

Update

I tried using the scikit-learn feature selector's library but does not work for 2 reasons: 1) the CRF takes as an input list of lists of dicts whereas the other take tabular data 2) the CRF does not have a .coef or a feature_importances_ attribute in order to perform feature selection.

Since sklearn-crfsuite provides a scikit-learn compatible estimator, any scikit-learn feature selection class that takes an estimator could work.


Problem importing CNTK in Azure jupyter notebook Vineeth Sai indicated in this that with the following code:

pip install cntk

the problem is solved. However, I am getting the error shown in attached image:

Keeping your latest edit in mind, the error in your screenshot is actually just a recommendation to upgrade pip. As suggested, please run pip install --upgrade pip, and it shall upgrade pip package in your environment and shall not re-appear unless a new version of pip get released later on.


Comparing Dataset - Should I use the same Test dataset? I am training ML CNN model. I want to compare different images dataset. The dataset all have different characteristics (Translated or not, Rotated or not, etc.).
I do not modify the ML model between the different dataset training.
Should I use the same Test dataset to compare them ? This dataset would not be changed through the testing and would contain data that can't be found else where. It would not be more suited for a specific training dataset.
Or should I use a Test dataset that has the same caracteristic as the Training Dataset ? So that I can compare them at their best ?
For example, if I want to compare the dataset A and B, should I use a combinaison of Test dataset A and Test dataset B ? or When testing dataset A, use Test dataset A and when testing dataset B, use Test dataset B.

By testing a model on the same dataset (sharing same characteristics), you will have information on how pertinent you hyperparameters are for this dataset.
Then you can test on another dataset that has other characteristics. It will give you information on how good is a model to generalize.
I would not create a mixed dataset to test the model for interpretation as it could hide some interesting information. For instance, case 1 you predict 100% of the samples from test dataset A and only 50% from test dataset B. Case 2 you predict 75% from both. Interpretation will not be the same. But If you want to compare the two models created from 2 different training datasets, then yes, compare them with a same test dataset, otherwise the test would be biased.


Explaination or Description of clusters after clustering After clustering, is there a way to explain the clusters? Or get the boundaries of the clusters?

For example:
If we have a data set of people's spending habits with columns for their spend in different categories like groceries, clothing, transportation, rent etc. And we applied a clustering algorithm (like k-means or agglomerative clustering) on it.
Can we get descriptions of clusters, like:

Cluster 1 contains people who spend

More than \$500 on groceries
Less than \$200 on transportation

Cluster 2 contains people who spend

Less than \$100 on rent
Less than \$300 on transportation
More than \$50 on transportation

Basically I need an explanation which is meaningful to a layman user.

It depends on the clustering technique you use. Since you tagged this post with k-means I will assume this is what you are using. Cluster centers should already be somewhat informative for laymans, but since you should be/are scaling this can lose some of it's interpretation.

What you could do is assign class labels to each sample based on in what cluster they ended up in. Then you could fit a multi-class decision tree to your data and use the decision rules for interpretation, like 60% of cluster 1 has $x_1 < 0.9$.


Why did Logistic regression perform better than svm? I have a data set of movies and their subtitles.My task is to classify them based on their ratings-[R,NR,PG,PG-13,G].
I have tried different ML algorithms and found that Logistic regression out performed them all, but I am unable to figure out why.My data had more features than observations.

SVM- should perform well on high dimensional data and will perform well even the there is class imbalance, but failed to show great results.
Naive Bayes-I think Naive Bayes did not perform well because of class imbalance.
Random forest-decent performance.but did not out perform logistic regression.

I am looking for an explanation as to why did one perform better than the other.

Note:The data set is sparse and it has more features/parameters than observations/examples.

There is a theorem in machine learning literature which is called “No free lunch theorem”. The essence of NFL is that there is no universal model which performs best for every problem and every dataset. So, according to NFL, you can not expect SVM to outperferm logistic regression in all situations and contexts. If your classes were linearly separable SVM would be perfect with 100% accuracy but otherwise you shouldn’t expect it to necessarily outperform ligistic regression. So, whether SVM or ligistic regression are better choice highly depends on the problem and on the available dataset.


To extract the skills required for the job given the job description Any recommendation on the libraries/methods to extract the skill set required for the job from the job description (raw text) ?
And also to extract the skillset from the resume ? The resume could be of any format raw text, pdf, docx.

I have attached the job description format.

Assuming you already have the raw text, you can do the followings.

Create train data:

You need to create set words and bigrams labeled as skills. There might be some available lists to help you out. Otherwise, generate your list using resources such as wordnet and thesaurus. You can also start with a short list based on the data you have and then expand it using word2vec or similar word embedding techniques. For example, we start with a list that contains coding as one of the skills. Then, we query the word2vec pretrained model for closest words/bigrams. There is a fair chance you will end up with programming, software coding, and computer programming.

Another approach would be to cluster the words and bigrams in your dataset using word embedding techniques. Then, look into your clusters to see which ones contain the skill set.

Note that any word/expression not on your list will not be considered as required skills. Therefore, you may need to expand your list after a few trials.

Detecting the Skills

Tokenize your raw text into words and expressions
Remove stop words
Encode your tokens using an embedding (Word2Vec, FastText, etc)
Use the list from the previous step to add labels to your data (anything on the list is True, other as False)
Train a binary classifier (Naive Bayes classifier should be good enough)
Evaluate your model, feature set, and labels. If needed, refine and repeat.


How do I cluster data that is a mix of text & categorical data? I have a set of strings, each also has soem categorical information associated with it. The categorical information isn't always great though, so I need to cluster the messages based on the text content & the categories. What is the best way to do this?

First and foremost, there is no best way to do this task in general. It will require some creativity and problem-solving

Here is one potential idea. Clustering involves some kind of similarity metric. You can compute a similarity between text documents (after some term weighting like TF-IDF you could use the cosine similarity for example). You can also compute a similarity between objects with categorical data. Matching coefficients are good option for this. For example, http://en.wikipedia.org/wiki/Simple_matching_coefficient  is about as basic as it gets, but there are many other measures of similarity for categorical data that you can find with a simple google search.

So if I have one similarity matrix A that gives the similarity based on the text, and one similarity matrix B that gives the similarity based on the categorical attributes, then I could take some weighted combination of A and B and use it as an overall similarity matrix for clustering. Perhaps a convex combination would work well and give you some flexibility to choose which measure of similarity you want to weight more heavily (since it sounds like the categorical information might not be as good):

choose some parameter $0\leq \alpha\leq 1$ and compute an overall similarity matrix S as

$$\textbf{S} = \alpha \textbf{A} + (1-\alpha)\textbf{B}$$

Then you could use any number of clustering algorithms using the similarity information. For example, $k$-means should work just fine on a similarity matrix as input. Any type of graph/spectral clustering algorithm is also well suited for this type of input.


Convert Keras model to C++ using protobuf file and Tensorflow C API I need to convert Keras model to C++ code. I found some implementations to do this: keras2cpp, another keras2cpp and frugalydeep. Only the third one was able to convert my model to cpp code. But the frugalydeep has no support for GPU. Therefore I am looking for another option to make this. My thoughts are to save Keras model to .pb(protobuf) file and then load this model with Tensorflow C API.
Do you think this will be possible? Or I will have to manually rewrite the Keras model to Tensorflow C API.
Are there any other ways to do this task?

My thoughts are to save Keras model to .pb(protobuf) file and then
load this model with Tensorflow C API.
Do you think this will be possible?

the answer is yes, absolutely. there are many ways to do this and i'll list some here so that you can pick up what suites your needs.

you can use cppflow, which is a c++ wrapper for the tensorflow C API, and actually makes it way easier to do the job than using the C API directly. they also have some examples on how to use it and in fact i think that this is the best solution out there.
you can see this wonderful repo hello_tf_c_api, especially this example.

or you can follow these articles on medium if you wish to use the c API directly:

undocumented tensorflow c api
deploying tensorflow 2.1 as c/c++ executable

there's also these two wonderful github gists that would walk you through how to do this exactly (although i recommend the c approach because it's more straightforward):

Training TensorFlow models in C
Training TensorFlow models in C++


How to group categorical columns into similar types? (Forgive me if the question is ill put. I am a novice in data science. Please comment or edit so that the question can be improved)

I have a dataset where we have to predict the future sale of a shop. "City' is one of the columns. The problem with 'City' is that data provided per city is very uneven in size(detail of only 1 shop in one city while giving detail of nearly 50 shops in another) Moreover many cities which are not present in train are present in test.

My plan is, based on other columns in train, I will find which cities are similar to each other so that I can classify those cities as a single group. (Logic is, say area, demand for an item are similar for city 1 and 3 so the sales in both of them will be similar.)

I can think of KmeansClustering or aggregated features. But I am not exactly sure how to achieve this. Please suggest some ideas, what are some good ways to deal with such cases.

although i don't quite understand your question. you don't have any business in worrying about data values in train and test sets, they are two different datasets after the split was done, as long as the features are the same, the content shouldn't be your focus. this is why scaling is done after splitting the dataset


Is it better to use a MinMax or a Log Return normalization to predict stock price movements? I am trying to use a LSTM model to predict d+2 and d+3 closing prices. I am not sure whether I should normalize the data

with a MixMax scaler (-1,+1)
using the log return
(P(n)-P(0))/P(0) for each sample

I have tried quite a lot of source code from Github and they don't seem to converge on any technique.

Log returns are symmetric compared to percentage change. Log(a/b)=log(b/a) and this (less skewness), in theory, leads to better results for most models (linear regression, neural networks).
Neural networks like lstm work better if the values are close to zero, but the difference in normalizations is usually not that big.
Any returns (log or percentage) are better than raw values because prices change according to previous prices. Their absolute (raw) values have almost negligible influence compared to previous price.

I would recommend first to convert to log returns and then normalize. If it is daily prices then I would divide the log returns by something like 0.05. Price changes have very heavy tails in a distribution so I would not suggest using minmax because then you divide by something like 0.5 (which probably was in great depression) and get all values too close to zero. Dividing by standard deviation should also be good.

But reality is different than theory, so it is better to benchmark. Maybe percentage changes are better because this is the number people see and react to. And markets are a lot about psychology.

And be prepared to see very high errors and bad models. Financial markets are badly predictable both in practice and theory. According to economic theory if they were predictable and people are rational and have unlimited credit lines then any possibility of earning additional money compared to the whole market will be closed in milliseconds. Only if you find some way to analyze data that noone is currently using only then will you be able to earn money. Neural networks were discussed in 1990s to predict financial markets. So LSTM is not really new in 2018.


Scatter plot for binary class dataset with two features in python I have my dataset that has multiple features and based on that the dependent variable is defined to be 0 or 1.
I want to get a scatter plot such that all my positive examples are marked with 'o' and negative ones with 'x'.
I am using python and here is the code for the beginning.

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

# Importing the dataset
dataset = pd.read_csv('/home/Dittu/Desktop/Project/creditcard.csv')

now I know how to make scatter plots for two different classes.

fig = plt.figure()
ax1 = fig.add_subplot(111)

ax1.scatter(x[:4], y[:4], s=10, c='b', marker="s", label='first')
ax1.scatter(x[40:],y[40:], s=10, c='r', marker="o", label='second')
plt.show()

but how to segregate both class of examples and the plot them or plot them with distinct marks without separating?

One approach is to plot the data as a scatter plot with a low alpha, so you can see the individual points as well as a rough measure of density.

from sklearn.datasets import load_iris
iris = load_iris()
features = iris.data.T

plt.scatter(features[0], features[1], alpha=0.2,
            s=100*features[3], c=iris.target, cmap='viridis')
plt.xlabel(iris.feature_names[0])
plt.ylabel(iris.feature_names[1]);

We can see that this scatter plot has given us the ability to simultaneously explore four different dimensions of the data:

the (x, y) location of each point corresponds to the sepal length and width,
the size of the point is related to the petal width, and
the color is related to the particular species of flower, i.e the Target Variable...

Multicolor and multifeature scatter plots like this can be useful for both exploration and presentation of data.


Why Logistic regression into Spark Mllib does not use Maximum likelihood estimation? During comparison estimates/ coefficients in 'R'& Spark Mllib of Logistic regression, It has been observed that estimates are not same.

On further investigation, I found that R & Mllib has different implementations for Logistic regression.

  R's glm is return a maximum likelihood estimate of the model while
  Spark's LogisticRegressionWithLBFGS is return a regularized model
  estimate.

Maximum likelihood estimates seems more efficient, as per literature available.

I am curious to know, why Spark Mllib developers has not chosen 'Maximum likelihood estimation' technique?

I think you're confused. Maximum likelihood is a general technique for giving the most likely parameter estimation. There is no closed solution for maximum likelihood for logistic regression, so both R and Spark must numerically estimate it.

How exactly the likelihood is estimated could be slightly different and might depends on the implementation. For example, Spark prefers to add regularization, R might like to use something else (you'll need to check the documentation).

Summary: Both R and Spark try to estimate ML for you. Please check the R documentation on how R do it.


How to count number of word embeddings in Gensim Word2Vec model I am trying to create a Word2Vec model of the the Pub Med Central corpus using the Gensim library and want to limit the total number of word embeddings to around 1 billion.

I have searched high and low and am unable to find out a) How to count the total number of word-embeddings in a saved model, and b) how to limit the total number of embeddings when training the model (once I hit 1 billion, then stop).

Please forgive my simpleton questions.

Alright, after digging around in methods available to my loaded Word2Vec model I believe the answer is len(model.wv.vectors)...


Understanding the softmax output in Youtube's recommender This question has been asked before, but never (that I can see) satisfactorily answered.

I'm reading Youtube's paper on their recommender system. The system has two elements, the first of which is a DNN generating 100 "candidate" videos, which are then combined with candidates from other sources and ranked by a second DNN. In the paper they say that they treat the candidate generation problem as extreme multiclass classification with Softmax. That approach is understandable; the indices of the N highest values in the softmax output become the N predicted candidates - easy peasy.

Reading further in the paper however, I began to get confused about exactly what their network is doing. They give a figure of their network's structure:

And it's apparent from that image that there's a whole other step beyond the softmax layer, which is what I'm not understanding. There's also later in the paper the following quote:

  The softmax layer outputs a multinomial distribution over the same 1M video classes with a dimension of 256 (which can be thought of  as  a  separate  output  video  embedding).

and also that:

  Since calibrated likelihoods from the softmax output layer are not needed at serving time, the scoring problem reduces to a nearest neighbor search in the dot product space for which general purpose libraries can be use.

But I'm confused about what they're actually implementing here. The assertion that the softmax layer outputs...with a dimension of 256 implies to me that they literally have a final layer in the form Dense(256, activation='softmax') instead of Dense(n_classes..., however the presence of "class probabilities" in the figure makes that impossible, since you couldn't convert such an embedding to class probabilities.

So; I'm really confused; does anyone know how to interpret exactly what's being done here?

Part of the confusion comes from different prediction processes used for training and serving.

Softmax is used during training to make predictions for the 1 million video categories. The softmax makes the training of the neural networks gradients easier and works well within deep learning libraries.

Those 1 million video categories also happen to be embedded into 256-dimensional space. During serving, prediction is now a nearest neighbor search. Nearest neighbor search is faster, since it is a dot product operation and can use general purpose libraries.


Normalizing test data I have a problem in data normalization. I have data for which I need to create an SVM. I will be using the model for real-time predictions. I know that the test tuples should be normalized using the exact same values as per the training data. However, my test tuples can have values that exceeds the maximum value of the data in the training set. For example, in the training set I have following values for a given feature.

Maximum : 20457
Minimum: 3

In the testing tuple, I sometimes get values like 35002. This is present in most of the features.

The problem would have been solved if I knew the maximum and minimum values for all feature, but it no possible. The maximum value can go up to any value.
How do I do data normalization in a scenario like this? Can someone please help me with this?

Judging from your question you are probably using this formulae for normalisation:

(x - x_mean)/(x_max - x_min). This is just an approximation of the real normalization formulae. The real one would be:

where mu is the mean and sigma is standard deviation. If your data trends is same throughout, then you can expect the mean and standard deviation to be approximately same, and thus give you a more uniform representation.

Check this Wikipedia article, Feature scaling which says about the schemes used in different ML techniques. Hope this helps!


Is it good practice to transform some variables and not others? I have a dataset with categorical variables encoded into numeric values, other variables that are continuous and have many outliers, and other continuous variable with a fairly normal distribution.
I was planning to use the sklearn preprocessing method .PowerTransformer in order to transform all of them, but maybe it might make more sense to just use it for those columns that have not normal distribution at all and many outliers?
It's for a classification problem (the Titanic machine learning one).

About the question whether to scale only a subset of features, I would tell you to do it over all the features (at least the continuous numeric ones) since the goal of data-scaling is to put these data on the same "reference scale" to be fairly compared.
Nevertheless, having mixed data types (continuous numerical, categorical...) for your classification problem looks more appropriate for scale-invariant algorithms like the ones based on decision trees. More precisely, you can have a look at XGBoost, where the author explains in this link that you do not actually have to re-scale your data.
Actually, in a recent real use case at my company, we tried re-scaling data VS not re-scaling it applying XGB, and we had better results with the second option.


recommender systems : how to deal with items that change over time? Let's say I am building a recommender system where items change through time. We suppose that each transaction is composed of  :

an item $i$ in list of items $(i_1, i_2, i_3, .., i_m)$.
a user $u$ in list of users $(u_1, u_2, u_3, ..., u_n)$.
a date $t$ in list of dates $(t_1, t_2, ... t_k)$.

We suppose that items have underlying features that change over time.For example, If we consider retail products, underlying features could be :

The discount level that is applied on the item when the customer has purchased the transaction (5%, 10%, 20%, 30%, ...).

An other example, if we consider financial stocks, underlying features that change over time could be :

The stock situation at time of the transaction (underpriced or overpriced).
The stock's central bank politics at time of the transaction (low interest rates, medium interest rates, high interest rates).

We suppose that these underlying features have a strong impact on users. It completly drives their decisions to buy or not an object. If we consider two items $i_1$ and $i_2$, at time $t_1$, a given user $u_1$ could prefer $i_1$ over $i_2$ because $i_1$'s underlying features are more interesting than $i_2$'s. If we consider a different time, maybe $u_1$ could be more interested in $i_2$ than $i_1$.

My question is : how to take into account underlying features that change over time in recommender systems such as user-user collaborative filtering, SVD, ALS... ?.

You will have look for incremental, online, or dynamic versions of classic recommender system algorithms. Those are the terms associated with changes over time.
Another option is to reinforcement learning. The reinforcement learning framework can also model changes over time. The most work has been done in multi-arm bandits for recommendations.


Good books for Hadoop, Spark, and Spark Streaming Can anyone suggest any good books to learn hadoop and map reduce basics?

Also something for Spark, and Spark Streaming?

Thanks

There's such an overwhelming amount of literature that with programming, databases, and Big Data I like to stick to the O'reilly series as my go-to source. O'reilly books are extremely popular in the industry and I've been very satisfied.

A current version of

Hadoop: The Definitive Guide,
MapReduce Design Patterns, and
Learning Spark

might suit your needs by providing high quality, immediately useful information and avoiding information overload -- all are published by O'reilly.

Spark Streaming is covered in Chapter 13 of "Learning Spark".


any efficient way to find surrounding adjective/verbs with respect to the target phrase in python [updated]? I am doing sentiment analysis on given documents. My goal is to find out the closest or surrounding adjective words with respect to the target phrase in my sentences. I do have an idea how to extract surrounding words with respect to target phrases. But how do I find out relatively close or closest adjective or NNP or VBN or other POS tag with respect to target phrase ?

Here is the sketch idea of how I may get surrounding words with respect to my target phrase.

sentence_List = {
    "Obviously one of the most important features of any computer is the human interface.",
    "Good for everyday computing and web browsing.",
    "My problem was with DELL Customer Service",
    "I play a lot of casual games online[comma] and the touchpad is very responsive"
}

target_phraseList = {
    "human interface",
    "everyday computing",
    "DELL Customer Service",
    "touchpad"
}

Note that my original dataset was given as DataFrame where the list of the sentences and respective target phrases were given. Here I just simulated data as follows:

import pandas as pd
df=pd.Series(sentence_List, target_phraseList)
df=pd.DataFrame(df)

Here, I tokenize the sentence as follow:

from nltk.tokenize import word_tokenize
tokenized_sents = [word_tokenize(i) for i in sentence_List]
tokenized=[i for i in tokenized_sents]

Then I try to find out surrounding words with respect to my target phrases by using this loot at here. However, I want to find out relatively closer or closest adjective, or verbs or VBN with respect to my target phrase.

How can I make this happen? Any idea to get this done? Thanks

POS-tagging consist of qualifying words by attaching a Part-Of-Speech to it. Part-Of-Speech is a tag that indicates the role of a word in a sentence (e.g. a noun, a transitive verb, a comparative adjective, etc.). You need this to know if a word is an adjective, and it is easily done with the nltk package you are using [source]:

>> nltk.pos_tag("The grand jury")
>> ('The', 'AT'), ('grand', 'JJ'), ('jury', 'NN')

Here, JJ means "Adjective" and "NN" means "Common Noun".

In your case, you are interested in neighbors adjectives. Does that mean "the closest adjective" in the sentence ? Or adjectives within a radius of the target, if any?
Depending on the definition, the way to do it differs.

For adjectives within a radius, as you have already selected words within a radius using the snippet you mentioned, you can POS-tag them and then select only those with a tag that indicates an adjective tag.

>> adjective_tags = ["JJ", "JJR", "JJS"]
>> close_adjectives_list = [a[0] for a in nltk.pos_tag(" ".join(close_words_list)) if a[1] in adjective_tags ]

You can look at this answer that list most of existing POS-tags.


Problem when cherry picking actions - Proximal Policy Optimization I'm using the implementation of PPO2 in stable-baselines (a fork of OpenAI's baselines) for a RL-problem.

My observation space is 9x9x191 and my action space is 144. Given a state, only some actions are "legal". If an "illegal" action is taken, the environment will return the same state. Think of it as the game of Go, where you try to place a stone on an intersection that is already occupied.

When a legal action is taken, it can result in a state that is completely different from the state before. It can also result in legal actions that are completely different. So it's not as in Go, where the stones stay on the board.

I currently feed the model a vector containing the legal actions (a multi-hot-vector, 1 for legal, 0 for illegal). When the model samples the actions, I first run the action means through a sigmoid so they all become positive but still keeps relative size order. Then I multiply the means with the legal action vector which sets the illegal actions to zero. The model then samples from only the legal actions with argmax. I do not change the underlying probabilities of the actions, so when calculating the neglogs of an action, they will be the true neglogs.

This works well for some time, but after a while I get the following error:

tensorflow.python.framework.errors_impl.InvalidArgumentError: Found Inf or NaN global norm. : Tensor had NaN values

The error is thrown when I calculate the tf.clip_by_global_norm:

grads, _grad_norm = tf.clip_by_global_norm(grads, self.max_grad_norm)

I'm guessing that the grads become infinitely large.

A general trend is that new and old neglog action probabilities starts to increase (which means that the action probabilities for the new and old policy starts to decrease). The Kullback Leiber distance also increases rapidly between them. This will lead to the ratio (r(θ) in the paper) to increase, which also will lead to a very volatile policy loss.

If I increase the number of steps before update, it can run for longer, but eventially it will crash.

I have tried to punish illegal actions instead of removing them, but the convergence is very slow.

My questions are:

What might be the reason for the grads going bananas?
Can it affect the model in an unexpected way if I cherry pick the legal actions?
Is there any other, better way that limit the action space to just legal actions? I haven't found any papers that analyze it. Do anyone know how AlphaZero handles illegal actions?

I believe that alphazero handles illegal actions by masking out the illegal logits before feeding to the final softmax output.


How to decide neural network architecture? I was wondering how do we have to decide how many nodes in hidden layers, and how many hidden layers to put when we build a neural network architecture.

I understand the input and output layer depends on the training set that we have but how do we decide the hidden layer and the overall architecture in general?

Sadly there is no generic way to determine a priori the best number of neurons and number of layers for a neural network, given just a problem description. There isn't even much guidance to be had determining good values to try as a starting point.

The most common approach seems to be to start with a rough guess based on prior experience about networks used on similar problems. This could be your own experience, or second/third-hand experience you have picked up from a training course, blog or research paper. Then try some variations, and check the performance carefully before picking a best one.

The size and depth of neural networks interact with other hyper-paramaters too, so that changing one thing elsewhere can affect where the best values are. So it is not possible to isolate a "best" size and depth for a network then continue to tune other parameters in isolation. For instance, if you have a very deep network, it may work efficiently with the ReLU activation function, but not so well with sigmoid - if you found the best size/shape of network and then tried an experiment with varying activation functions you may come to the wrong conclusion about what works best.

You may sometimes read about "rules of thumb" that researchers use when starting a neural network design from scratch. These things might work for your problems or not, but they at least have the advantage of making a start on the problem. The variations I have seen are:

Create a network with hidden layers similar size order to the input, and all the same size, on the grounds that there is no particular reason to vary the size (unless you are creating an autoencoder perhaps).
Start simple and build up complexity to see what improves a simple network.
Try varying depths of network if you expect the output to be explained well by the input data, but with a complex relationship (as opposed to just inherently noisy).
Try adding some dropout, it's the closest thing neural networks have to magic fairy dust that makes everything better (caveat: adding dropout may improve generalisation, but may also increase required layer sizes and training times).

If you read these or anything like them in any text, then take them with a pinch of salt. However, at worst they help you get past the blank page effect, and write some kind of network, and get you to start the testing and refinement process.

As an aside, try not to get too lost in tuning a neural network when some other approach might be better and save you lots of time. Do consider and use other machine learning and data science approaches. Explore the data, maybe make some plots. Try some simple linear approaches first to get benchmarks to beat, linear regression, logistic regression or softmax regression depending on your problem. Consider using a different ML algorithm to NNs - decision tree based approaches such as XGBoost can be faster and more effective than deep learning on many problems.


How data are prepared during training, testing and in production? Most of real world datasets have features with missing values. Replacing missing values with an appropriate value such as its mean, is considered as a good step in feature engineering. Some times we also standardize/normalize feature columns before feeding it to train an model.
Before modelling  we also split our dataset to training and testing sets.
My first query is how do we do feature engineering in this splitted dataset?
Do we use a global mean of the unsplitted features  to replace the missing value of those features in both training and testing set or should we use local means of those sets?
Like the above question how do we do normalization to a train, test dataset?
The last but an important question, in productions we mostly get feature values one at a time (think a row of features), how do we feature engineer such data rows?

The principle in supervised ML is quite simple: the "method" which is going to be used to predict the response variable must be fully determined from the training set and only from the training set. In other words, anything which doesn't belong to the training set cannot be used.
As a consequence, feature engineering, i.e. choosing how to prepare/represent/normalize features must be done using only the training set. This includes any feature selection/extraction step.
Note that once the final data preparation process is fully determined, it can and should be applied exactly the same way on the test set or in production. This means that for instance normalization does not involve recomputing any parameter, it uses the ones calculated on the training set.
See also a few related questions:

Data normalization before or after train-test split?
Why you shouldn't upsample before cross validation
Using PCA as features for production


Handling a Pandas Data Frame containing multiple non-ordinal categorical features I'm currently trying to analyse a dataset containing multiple non-ordinal categorical features and a binary target variable. The table looks something like this:

+------------+---------+------------+--------+
|    Col1    | ....    |    Col14   | Target |
+------------+---------+------------+--------+
| cat 1      |  cat 1  | cat 1      |    0   |
| ...        |  ...    | ...        |  ...   |
| cat 9      |  cat 50 | cat 450    |    1   |
+------------+---------+------------+--------+

The entire table is 400.000 rows x 15 columns, from which the last column is the target variable. Each feature has multiple non-ordinal categories ranging from 9 categories to multiple hundreds of categories.

My first instinct would be to one hot encode all the categorical variables. However, I'm scared that doing so will make any model prone to overfitting.

How could I handle/encode the features variables to analyse their effect on the target variable, using Python?

It looks like a case where target encoding will shine.

Target encoding replaces a category for the mean target in that category. You have to be careful not to overfit, but it is a very effective method to use categorical features with many levels.

There's a python package that implements target encoding using the syntax of Scikit Learn.


Identify important less frequent words I have set of newspaper articles. I want to identify important less frequent words in the set of newspaper articles. Currently I am using TF-IDF scores. However, it does not seem to be a good metric in my problem. Is there any better way of doing this?

Thank you in advance :)

You could try looking just at the IDF scores. Things like names/entities will score high, this may be desirable or undesirable depending on what you want.


Machine Learning algorithm for detecting anomalies in large sets of events Let's start with the following hypothetical preconditions:

There is traffic: normal and anomaly. Each traffic sample contains a list of events (of variable size)
Events happen in order, the possible events set size is ~40000  elements
Should run on relatively small amounts of memory and processing power

Having a traffic sample (of size 1000 events max), what is the best machine learning algorithm, that fits the preconditions, to identify whether it's an anomaly?

Given my limited knowledge in machine learning algorithms, here is what I came up with:

This system can be described very well as a markov process, but there are huge memory limitations in this scenario

1. Reduced Markov Chains

Store frequent pairs of events (that appeared more than 10 times in normal traffic) and then search the some pair there: if it doesn't appear, count as an anomaly. Then use some heuristic to identify if the traffic as a whole is an anomaly.

I called it reduced, because practically, we are using only a chain of two events, any other bigger size would become a huge combinatorial problem and fill any memory it's been given, which is infeasible.

2. Naive KNN

Get all the normal traffic samples (each sample may contain up to 1000 events) and analyze the number of appearances of each event in each sample.
Separate the dataset into 10 parts, compute their means to get the mean frequency for each part (basically, we now have 10 mean-frequency vectors) and use them as positive data-points inside the KNN algorithm

Do the same with anomaly traffic and add 10 data points.
Having the points, we can now use the KNN algorithm regression to compute a score and make a decision.

This is a bit tricky, because the frequency vectors are quite big, have too many of them becomes a problem. A solution would be to implement sparse vectors

Any other ideas, what am I missing?

You can also try these:

4- One Class SVM (Support vector Machine):

You can train this with the normal samples that you have and then it can label them all and detect anything that does not fall under this label. It can also scale easily, although it might get quite slow on training with large data. You can try the scikit-learn implementation:
https://scikit-learn.org/stable/modules/generated/sklearn.svm.OneClassSVM.html

5- Isolation Forest:

Isolation forest is actually an Ensamble algorithm, it is basically using Random Forest (Which by itself is using Trees) to detect Anomalies or outliers in the data. At first you can try scikit-learn implementation of it:
https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html

6- Deep One-Class Classification:

The other thing that you can try is the Neural Network implementation of One class classification. You should be able to do it easily with Keras for example. You can take a look at Keras documentation here:
https://keras.io/models/about-keras-models/

Also for implementation, these might help:
https://blog.keras.io/building-autoencoders-in-keras.html ,
https://medium.com/@curiousily/credit-card-fraud-detection-using-autoencoders-in-keras-tensorflow-for-hackers-part-vii-20e0c85301bd


Is regularization included in loss history Keras returns? I'm getting to know Keras. Right now, I'm testing with regularization and how to use them. Comparing the results of loss history for a training session with and without regularization, it seems to me that the loss history reported by Keras has the regularization term added to it, is that right?

When my model has no regularization term, the loss value starts from something less than 1 but when I retrain the model with regularization (L1L2), the same problem's loss value starts from 500. The only logical explanation I've got for it is that Keras is reporting loss value after it added the regularization term to it. And I believe the loss value without regularization is as valuable as with it, if not more valuable. Don't you think?

the regularisation values are by default computed in the loss and so you cannot see the regularisaiton values separately (as far as I know). For a brief explanation of an example using the cross-entropy loss with regularisation, have a look at the example in Michael Nielsen's book.

You can see what is returned and available by saving the results from calling model.fit().

You can train a model then also check out:

print(history.history.keys())

which will show you all available metrics for analysis after training the model.

You could perhaps define a custom function that compute the regularisation terms you are using, and execute that as either your own metric or as a callback function in a Keras model. If you use your own metric, those are by default recorded separately from the training/validation loss value.


What does it mean that classes are mutually exlcusive but soft-labels are accepeted? The Tensorflow's documentation of softmax_cross_entropy_with_logits:

  Measures the probability error in discrete classification tasks in
  which the classes are mutually exclusive (each entry is in exactly one
  class). For example, each CIFAR-10 image is labeled with one and only
  one label: an image can be a dog or a truck, but not both.

  NOTE: While the classes are mutually exclusive, their probabilities
  need not be. All that is required is that each row of labels is a
  valid probability distribution. If they are not, the computation of
  the gradient will be incorrect.

At first glance it may seen contradictory, but my guess is that, provided these conditions for the classes and for the labels, the probability distribution of the labels have always one maximum. For instance, labels = [0.5 0.5] is a valid probability distribution in a binary case but it does not comply with the exclusivity of the classes. If I'm wrong, I don't get the idea behind the documentation.

It is also pointed out that for mutually elusive probabilities, sparse_categorical_cross_entropy should be used, but I think that softmax_cross_entropy_with_logits can be used if the labels are one-hot encoded, which is a valid probability distribution (a deterministic one)

Edit:

When going around this question again, I'm wondering if in a multi-class exclusive case where the only constraint on the labels are that they have to be a valid probability distribution, labels = [0.5 0.5] should be a valid instance label. This label means that the annotator nor the net can tell if this ground-truth instance belongs to class_0 or class_1...

What does it mean that classes are mutually exclusive but soft-labels are accepted?

As it can be seen from here, tf.nn.softmax produces just the result of applying the softmax function to an input tensor. The softmax "squishes" the inputs so that sum(input) = 1; it's a way of normalizing. The shape of the output of a softmax is the same as the input - it just normalizes the values. The outputs of softmax can be interpreted as probabilities. In contrast, tf.nn.softmax_cross_entropy_with_logits computes the cross-entropy of the result after applying the softmax function (but it does it all together in a more mathematically careful way). It's mathematically careful due to the fact that $y_i$ in $log(y_i)$ can be zero. As you can read from here, A randomly-initialized softmax layer is extremely unlikely to output an exact 0 in any class. But it is possible, so worth allowing for it. First, don't evaluate log(yi) for any y′i=0, because the negative classes always contribute 0 to the error. Second, in the practical code you can limit the value to something like $log( max( y_predict, 1e-15 ) )$ for numerical stability - in many cases, it is not required, but this is sensible defensive programming. I encourage you to take a look at the answers to this question.

  NOTE: While the classes are mutually exclusive, their probabilities need not be. All that is required is that each row of labels is a valid probability distribution. If they are not, the computation of the gradient will be incorrect.

The first sentence means that your classifier may not be able to classify the labels exactly as they are, one-hot-encoded. What it does is to find the chance that how likely it is that the input belongs to each class. And this won't make a problem if it does not have mutually exclusive output vector as the algorithm implies. It just needs a vector that the sum of its entries is equal to one. If they are not, the computation of the gradient will be incorrect. I guess this line is added to announce that the outputs of this differentiable component will not be one-hot-encoded and it's because of the nature of these nets. The first layers of convolutional networks are like bases vectors and each instance of classes share these bases. All inputs are made up of these bases.

  I'm wondering if, in a multi-class exclusive case where the only constraint on the labels is that they have to be a valid probability distribution, labels = [0.5 0.5] should be a valid instance label. This label means that the annotator nor the net can tell if this ground-truth instance belongs to class_0 or class_1...

Basically, for multi-label classification, your input may have different labels. Consequently, your classes won't be mutually exclusive anymore. Moreover, in those cases, we don't use softmax as the last layer. You have to have sigmoid for each output and your cross-entropy cost function will be slightly different 1. Consequently, the output of each entry should be a valid probability, that is why sigmoid is used, and the label vectors for such tasks is a not one-hot-encoded anymore. different classes have different entries and based on the existence of instances of each class, the corresponding entry should be one.


What's the difference between multi label classification and fuzzy classification? Is it just the between academics and practitioners in term usage?

Or is theoretical difference of how we consider each sample: as belonging to multiple classes at once or to one fuzzy class?

Or this distinction has some practical meaning of how we build model for classification?

Multi-label classification (Wiki):

Given $K$ classes, find a map $f:X \rightarrow \{0, 1\}^K$.

Fuzzy classification (a good citation is needed!):

Given $K$ classes, find a map $p: X \rightarrow [0, 1]^K$ where $\sum_{k=1}^{K} p(k)=1$.

In multi-label classification, as defined, there is no "resource limit" on classes compared to fuzzy classification.

For example, a neural network with a softmax layer does fuzzy classification (soft classification). If we only select a class with the highest score, then it will become a single-label classification (hard classification), and if we select top $k$ classes, it will be a multi-label classification (again hard classification).

Fuzzy classification:        [0.5, 0.2, 0.3, 0, 0]
Single-label classification: [1,   0,   0,   0, 0]
Multi-label classification:  [1,   0,   1,   0, 0]

As another example for multi-label classification, we could have $K$ neural networks for $K$ classes with sigmoid outputs, and assign a point to class $k$ if output of network $k$ is higher than 0.5.

Outputs:                     [0.6, 0.1, 0.6, 0.9, 0.2]
Multi-label classification:  [1,   0,   1,   1,     0]

Practical considerations

As demonstrated in the examples, the key difference is the "resource limit" that exists in fuzzy classification but not in multi-label classification. Including the limit (in the first example), or ignoring it (in the second example) depends on the task. For example, in a classification task that has  mutually exclusive labels, we want to include the "resource limit" to impose the "mutually exclusive" assumption on the model.

Note that the $\sum_{k=1}^{K} p(k)=1$ restriction in fuzzy classification is merely a "definition", there is no point in arguing about a definition. We can either propose another classification, or  argue when to use - and when not to use - such classification.


How to use spectral clustering to predict? In an academic paper, they talk about using a nearest neighbour algorithm to predict the cluster of a new point. And how the number of nearest neighbours is set to 10 in their example.
What do they mean with this? The two things I could think of were:
1. Look which 10 points used in the training set (neighbours) are closest and then assign it to the cluster of which the majority of the points come from.
2. Collect one by one the closest points from the training set until you have 10 points that come from one single cluster. That is the cluster to which the point belongs.

What are the other ways to assign a(n) (existing) cluster to a new point?

If the paper didn't elaborate on this, it must mean that they do 1). They look for the 10 closest neighbors of a new point and use majority voting to assign a cluster. On a side note, strange that they use an even number of neighbors, since it may introduce ties.

"What are the other ways to assign a(n) (existing) cluster to a new point?"

There are plenty of ways but it doesn't mean that they are all good! You could randomly assign points to clusters and use the size of the clusters as weight, but that isn't such a great method.

Generally, a nearest-neighbor approach is what is used. Now, it can be done in many ways:

KNN as I explained before (with any number of K)
Calculate the mean point of each cluster, and assign to the closest cluster
For each cluster, calculate the most distant point to the new point, then assign the cluster that has the "least far" distant point
And so on.


In Orange, is it possible to analyze or visualize real time, streaming data? Using orange, I would like to be able to do real time analysis or visualizations on streaming data. I would appreciate any input on the matter!

"Realtime" is a very big word, in many regards you would only call a system realtime when there is very little latency in a system, almost no latency. It would require specialized hardware and os, for a discussion of python and realtime see https://stackoverflow.com/questions/7079864/real-time-operating-via-python.

If you describe the requirement as "continously (or frequently) updated", it looks possible to me to do some mining on streamed data with orange. I am adding my 5 ct, as I have just started looking into orange.

You would need to create a widget that opens your input stream and propagates it in chunks to the subsequent widgets in the flow, i.e. when you want to build something like a "tail -f" on a file source, reads character by character until there is a linefeed in the stream. In that case it would release a signal with the line as data to the output.

Orange widgets are receiving their inputs as signals, "widgets receive inputs at runtime with the designated handler method (specified in the OWWidget.inputs class member)." This next widget would update on that signal and propagate on its own outputs. So the continued live analysis propagation of streaming data through orange widgets seems in general possible to me.

How close to or far from real time this would be depends on many factors like load, computing power, and memory. The choice of mining algorithms might also be limited by such factors.


Quantifying a trade-off Let our data be a set of 2d points sampled from a line $ax+by=1$ (that we do not know) plus some noise $\nu$.

This is meant to represent a frontier of development - we can build a car with a powerful engine or very fuel efficient, but it is hard to do both.

How can I go about estimating the line $ax + by = 1$?

Linear regression is easy.  Or you could build some self training AI NN and do it the hard way.

Engine vs fuel efficient trade off  would not be linear and there are so many other factors that go into a final trade off.

You must trade off ALL factors to optimise the system.


How to know if classification model is predicting 1 or 0 I have used logistic regression to predict whether customer is good(1) or bad(0). I got the accuracy .80 . How do i know whether model predicted 1 or 0 .Is it related to parameter of model1.predict_proba(X_test)[:,1] (the 1 in the end in square bracket).

You can find the model predictions here:
model1.predict(X_test)

array([0, 0, 0, ..., 0, 0, 0], dtype=int64)

In a binary (good/bad or 1/0) model, predict_proba is the probability of 0 or 1 , the value that is calculated by your model, which is used along with a cutoff (which is 0.5 in the sklearn implementation and cannot be changed) to determine if the final prediction is 0 or 1
model1.predict_proba(X_test)[:,0] # probability the answer is 0

array([0.94009529, 0.96378774, 0.98951049, ..., 0.67607543, 0.97599932,
   0.82838031])

model1.predict_proba(X_test)[:,1] # probability the answer is 1

array([0.05990471, 0.03621226, 0.01048951, ..., 0.32392457, 0.02400068,
   0.17161969])

since this is a binary model, both should add up to one.
The accuracy score is how many predictions were correct.
(correct_pred_0 + correct_pred_1)/total_predictions
If you remove the [:,1], you get the entire array:
model1.predict_proba(X_test)

        #col 1       col 2

array([[0.94009529, 0.05990471],
       [0.96378774, 0.03621226],
       [0.98951049, 0.01048951],
       ...,
       [0.67607543, 0.32392457],
       [0.97599932, 0.02400068],
       [0.82838031, 0.17161969]])

First column is the probability for 0 and the second column is the probability for 1
you can check the order of the classes using this:
model1.classes_,

for binomial, the default is like this:
array([0, 1], dtype=int64)