Improving Naive Bayes accuracy for text classification I am performing document (text) classification on the category of websites, and use the website content (tokenized, stemmed and lowercased).

My problem is that I have an over-represented category which has vastly more data points than any other (roughly 70% or 4000~ of my data points are of his one category, while about 20 other categories make up the last 30%, some of which have fewer than 50 data points).

My first question:

What could I do to improve the accuracy of my classifier in this case of sparse data for some of the labels? Should I simply discard a certain proportion of the data points in the category which is over-represented? Should I use something other than Gaussian Naive Bayes with tf-idf?

My second question:

After I perform the classification, I save the tfidf vector as well as the classifier to disk. However, when I re-rerun the classification on the same data, I sometimes get different results from what I initially got (for example, if previously a data point was classified as "Entertainment", it might receive "News" now). Is this indicative of an error in my implementation, or expected?

Regarding your first question...

Do you anticipate the majority category to be similarly over-represented in real-world data as it is in your training data? If so, perhaps you could perform two-step classification:

Train a binary classifier (on all your training data) to predict membership (yes/no) in the majority class.
Train a multi-class classifier (on the rest of the training data) to predict membership in the remaining minority classes.


Difference between AWS amazonei_tensorflow and tensorflow conda environments In AWS SageMaker there are several Conda environments available based on different ML frameworks:

conda info -e

I would like to know the difference between a tensorflow and an amazonei_tensorflow environment.

The amazonei environments are for use with the Amazon Elastic Inference accelerators. You can see this mentioned in the DLAMI README file:

for TensorFlow(+Keras2) with Python3 (CUDA 9.0 and Intel MKL-DNN) ______________
____ source activate tensorflow_p36

for Tensorflow(+Amazon Elastic Inference) with Python3 _________________________
____ source activate amazonei_tensorflow_p36


Libraries for Online Machine Learning I am looking for packages (either in python, R, or a standalone package) to perform online learning to predict stock data.

I have found and read about Vowpal Wabbit (https://github.com/JohnLangford/vowpal_wabbit/wiki),
which seems to be quite promising but I am wondering if there are any other packages out there.

Thanks in advance.

You could look at scikit-learn or Orange module in Python.
Scikit-learn has a SGD classifier and regressor that could do a partial fit data in case of online learning.
In R, take a look at caret package


Bag of words and word2vec clarifications I have documents,  and I calculated the word vectors using word2vec for all the terms in my corpus.

Now I want to compute similarity between documents using the bag-of-words model.  After  creating the Bag-of-words ( consisting of  vector representations).

how do I now compute similarity between a sequence of vectors?

Do I simply just take the mean? and calculate the cosine similarity between the mean vectors?

or is there a better approach for computing similarity between two bags?

BoW is a text representation like word2vec or doc 2vec. If you already have the word2vec vectors then you can take the mean for each document and calculate the similarity but it's not the direct approach. The direct approach is to calculate doc2vec which models the documents directly and with more detailed precision than word2vec.


How important is the input data for a ML model? Last 4-6 weeks, I have been learning and working for the first time on ML. Reading blogs, articles, documentations, etc. and practising. Have asked lot of questions here on Stack Overflow as well.

While I have got some amount of hands-on experience, but still got a very basic doubt (confusion) --
When I take my input data set with 1000 records, the model prediction accuracy is say 75%. When I keep 50000 records, the model accuracy is 65%.

1) Does that mean the model responds completely based on the i/p data being fed into?

2) If #1 is true, then in real-world where we don't have control on input data, how will the model work?

Ex. For suggesting products to a customer, the input data to the model would be the past customer buying experiences. As the quantity of input data increases, the prediction accuracy will increase or decrease?

Please let me know if I need to add further details to my question.

Thanks.

Edit - 1 - Below added frequency distribution of my input data:

Edit - 2 - Adding Confusion matrix and Classification report:

To answer your first question, the accuracy of the model highly depends on the "quality" of the input data. Basically, your training data should represent the same scenario as that of the final model deployment environment.

There are two possible reasons why the scenario you mentioned is happening,

When you added more data, maybe there is no good relationship between input features and label for the new examples. It is always said that less and clean data is better than large and messy data.
If 49000 records added afterward are from the same set(i.e. have a good relationship between label and features) as that of 1000 before, there are again two possible reasons

A. If accuracy on the train dataset is small along with test dataset. e.g. training accuracy is 70% and test accuracy is 65%, then you are underfitting data. Model is very complex and dataset is small in terms of the number of examples.

B. If your training accuracy is near 100% and test accuracy is 65%, you are overfitting data. Model is complex, so you should go with some simple algorithm.

NOTE* Since you haven't mentioned about training accuracy, it is difficult to say what out of the two above is happening.

Now coming to your second question about real-world deployment. There is something called model staleness over time which is basically the problem of reducing model accuracy over time. This is the article by a product manager at Google explaining the staleness problem and how it can be solved. This will answer your second question.

Let me know if something is not clear.


Optimization based on validation and not training Hello neural network programmers,

I am currently creating a neural network with keras, as I am not that familiar with tensorflow and it's a bit more difficult.

I want my optimization to optimize the validation loss and not the training loss/accuracy.

Ideally, I would want it to look something like this:

  model.compile(optimizer=adam, loss='categorical_crossentropy', metrics=['val_accuray'])

but val_accuracy is not something I can eneter into metrics.

Any ideas on how I can implement that?

Thanks!

Welcome to the site! The optimization function in your model creation is not the best place to do what you want. What you should be doing is:

Create the model
Run as many epochs as you want during model training
During the epochs create Keras callbacks that save the best model based on validation accuracy (as opposed to other metrics)

That will allow you to still create your model in an efficient way while ending up with an epoch training process that creates the model with the highest validation accuracy.


Where in the workflow should we deal with missing data? I'm building a workflow for creating machine learning models (in my case, using Python's pandas and sklearn packages) from data pulled from a very large database (here, Vertica by way of SQL and pyodbc), and a critical step in that process involves imputing missing values of the predictors. This is straightforward within a single analytics or stats platform---be it Python, R, Stata, etc.---but I'm curious where best to locate this step in a multi-platform workflow.

It's simple enough to do this in Python, either with the sklearn.preprocessing.Imputer class, using the pandas.DataFrame.fillna method, or by hand (depending upon the complexity of the imputation method used). But since I'm going to be using this for dozens or hundreds of columns across hundreds of millions of records, I wonder if there's a more efficient way to do this directly through SQL ahead of time. Aside from the potential efficiencies of doing this in a distributed platform like Vertica, this would have the added benefit of allowing us to create an automated pipeline for building "complete" versions of tables, so we don't need to fill in a new set of missing values from scratch every time we want to run a model.

I haven't been able to find much guidance about this, but I imagine that we could:

create a table of substitute values (e.g., mean/median/mode, either overall or by group) for each incomplete column
join the substitute value table with the original table to assign a substitute value for each row and incomplete column
use a series of case statements to take the original value if available and the substitute value otherwise

Is this a reasonable thing to do in Vertica/SQL, or is there a good reason not to bother and just handle it in Python instead? And if the latter, is there a strong case for doing this in pandas rather than sklearn or vice-versa? Thanks!

My strong opinion regarding automated tasks like imputation (but, here I can include also scaling, centering, feature selection, etc) is to avoid in any way do such things without carefully inspecting your data.

Of course, after deciding what kind of imputation to apply it can be automated (under the assumption that the new data has the same shape/problems).

So, before anything, take a wise decision. I often wasted time trying to automate this things, destroying my data. I will give you some examples:
- a marketplace encoded as N/A, which I missed and considered to be North/America
- numbers like -999.0, because the data producer could not find a better replacement for missing data
- number like 0 for blood pressure or body temperature, instead of missing data (it is hard to imagine a living human with 0 blood pressure)
- multiple placeholders for missing data, due to the fact that the data was collected from various sources

After that you need to understand what kind of imputation would resemble better the information from your data for a given task. This is often much harder to do it right than it seems.

After all those things, my advice is to delay your imputation task to an upper layer where you have tools to reproduce on new data and to inspect if the assumptions for the new data are not violated (if it is possible).


What is the difference between fully observed and partially observed state features in Reinforcement learning ? When we use a function to approximate Q values or policy to state-action pairs we hear the above two terms fully and partially observed features. what is the difference . Normally in Atari Games they generate Q values by taking the current pixels distribution of the frame .

A fully observed state means that there is no hidden information. Clear examples of this are chess and Go because both players have all the information. The fact that both these games are deterministic doesn't matter. A game where the state changes are stochastic can still be fully observable. Games like poker, where both players can observe their own hand but not their opponents' are called partially observable. Other examples of this can be real time strategy games like Starcraft where you can only see in the line of sight of your units.


Understanding dimensions of Keras LSTM target I'm learning about Keras and LSTMs and came across this tutorial, but I don't understand the dimensions of the target variable. Quoting the article below:

  The training y data in this case is the input x words advanced one
  time step – in other words, at each time step the model is trying to
  predict the very next word in the sequence. However, it does this at
  every time step – hence the output layer has the same number of time
  steps as the input layer.

  To make this a bit clearer, consider the following sentence:

  “The cat sat on the mat, and ate his hat. Then he jumped up and spat”

  If num_steps is set to 5, the data consumed as the input data for a
  given sample would be “The cat sat on the”. In this case, because we
  are predicted the very next word in the sequence via our model, for
  each time step, the matching output y or target data would be “cat sat
  on the mat”.

The article than shows the following code to generate batches:

def generate(self):
    x = np.zeros((self.batch_size, self.num_steps))
    y = np.zeros((self.batch_size, self.num_steps, self.vocabulary))
    while True:
        for i in range(self.batch_size):
            if self.current_idx + self.num_steps >= len(self.data):
                # reset the index back to the start of the data set
                self.current_idx = 0
            x[i, :] = self.data[self.current_idx:self.current_idx + self.num_steps]
            temp_y = self.data[self.current_idx + 1:self.current_idx + self.num_steps + 1]
            # convert all of temp_y into a one hot representation
            y[i, :, :] = to_categorical(temp_y, num_classes=self.vocabulary)
            self.current_idx += self.skip_step
        yield x, y

I understand that the model is trying to predict the next word, and I understand that the dimensions of x are self.batch_size, self.num_steps, but I don't understand the y dimensions: self.batch_size, self.num_steps, self.vocabulary. Why not just self.batch_size, self.vocabulary? I might have misunderstood the article, but I thought each record in the batch corresponds to a different time step, or at least that's what I would have assumed for any other non-LSTM type of model. Does Keras automatically unroll the y output because it involves an LSTM?

If num_steps is set to 5, the data consumed as the input data for a given sample would be “The cat sat on the”. In this case, because we are predicted the very next word in the sequence via our model, for each time step, the matching output y or target data would be “cat sat on the mat”.

In this example, y is a sequence of words with length num_steps starting at the very next word. The input and output both have length num_steps.

Keras will return a sequence for the output when return_sequences is True. https://keras.io/layers/recurrent/#lstm

  return_sequences: Boolean. Whether to return the last output in the output sequence, or the full sequence.

In general, the input to an LSTM has dimensions (batch, time, features). The batch dimension should not be dependent on time. The output can either be a sequence or a single number, which in Keras is handled by return_sequences.


What will happen if the number of units and the time-steps are different numbers in LSTM? How will data feed the LSTM in following scenarios?
I have a data array with the shape of (100,10,3) and a LSTM like one of these:

  First scenario:

  LSTM(units = 5 , input_shape(time-steps = 5, No_of_features = 3))

  Second scenario:

  LSTM(units = 15 , input_shape(time-steps = 5, No_of_features = 3))

  Third scenario:

  LSTM(units = 5 , input_shape(time-steps = 20, No_of_features = 3))

  Fourth scenario:

  LSTM(units = 15 , input_shape(time-steps = 20, No_of_features = 3))

It depends on how you are sending the data into the LSTM. Lets say you have one data point with t timestamps. It can be of shape of 1xt or tx1, which means you will be sending all the time inputs once into LSTM and while in the second case you will sending one timestamp of data t number of times to complete one data point.

Based on the number of units, the network computes what are the sizes U, W and V matrices. units just tells what should be the hidden dimensions the network should operate at. So it does not matter what is your input sequence length is. It is going to take in one input at a time and take the output of last input you put in and computes new output, when you unroll it and see.

Edit

  First scenario:

  LSTM(units = 5 , input_shape(time-steps = 5, No_of_features = 3))

If LSTM cell does not return the intermediate cell hidden states, it will output 1x5 output at the end else it will be giving out 5*5

  Second scenario:

  LSTM(units = 15 , input_shape(time-steps = 5, No_of_features = 3))

If LSTM cell does not return the intermediate cell hidden states, it will output 1x15 output at the end else it will be giving out 5*15

  Third scenario:

  LSTM(units = 5 , input_shape(time-steps = 20, No_of_features = 3))

If LSTM cell does not return the intermediate cell hidden states, it will output 1x5 output at the end else it will be giving out 20*5

  Fourth scenario:

  LSTM(units = 15 , input_shape(time-steps = 20, No_of_features = 3))

If LSTM cell does not return the intermediate cell hidden states, it will output 1x15 output at the end else it will be giving out 20*15


Compute Jaccard distance between two lists of strings I have input like below:

input_list = [['Search' 'engines','using','machine','learning','pattern','detections'],

        ['machine','learning','helped','Google','automatically','sift','pages']]

input_list1 = ['Machine','learning','ever','evolving','technology']

Expecting Jaccard similarity distance between input_list and input_list1.

Python lib textdistance is a "python library for comparing distance between two or more sequences by many algorithms." It includes the Jaccard index. Having the similarity, you can get the distance by $Jacc_{distance}(x,y) = 1  -Jacc_{similarity}(x,y)$.

Although installing and importing the whole module wouldn't make much sense if this is all you need, reading the module's README may give you new ideas on the various distances you could use in your project, and take advantage of having all of them already implemented, so you can test how each of them performs really quick.


How to compare a machine learning model and a rule based model The difference between machine leaning models and rule based model is that you feed input and output to machine learning models to get rules where you feed input and rules to rule based models to get output.
Say you have a dataset with 10k records. You might use 8k of them to build a machine learning model and the rest to validate your model. On the other hand, you would probably use all the 10k records to create a rule based model. If this was a classification model, how should I compare the 2 models? Accuracy of the machine learning model from the test dataset (2k) and accuracy of the rule based model from all the records?

For the rule base model I would take the exact same approach as for ML model, split the data into train and test set since at the end you want to check whether or not your rule base and ml models generalize well
It is also recommended to used the same sets to train and evaluate both models in such a way that it is a fair comparison.


very important features, but rarely input The features are of capital gain and capital loss, but very small amounts of people have one or the other. As far as I can see it may not signify any great difference, though I'm not sure how I would go about even normalising or whatnot these features:

     age        wgt       sex   gain        loss         hpw
0   0.379310    0.119825    1   2174           0         -0.078664
1   0.568966    0.131898    1   0              0         -2.327679
2   0.362069    0.407596    1   0              0         -0.078664

The gain can skyrocket to 100,000 and that is definitely of significance considering I'm predicting whether someone will make  over $50k, though they occur at such a small rate, close to 3-5% that I'm not sure how I could go about implementing them. I doubt imputing data would be a good idea considering just about everyone is on 0 for both results.

Also to note is I have 15 columns total (many more with get_dummies()).

A tree-based model, especially one that allows missing data in the input, seems a good choice: the tree(s) can split large capital gains out (presumably separating a nearly pure node from the rest).

Also consider something more basic: just hard-code an exception for those with large gains, and model for the rest.


How is Data Science related to Machine learning? I went through this comparison of analytic disciplines and this perspective of machine learning, but I am not finding any answers on the following:

How is Data Science related to Machine learning?
How is it not related to Machine Learning?

Data science is much broader concept than machine learning.
It starts from simple data visualization and descriptive statistics to get insights, manipulations like cleansing to prepare data. Before you can use some ML algorithms.

Basically such huge stacks as bigdata, visualization and data preprocessing are out of machine learning scope. And they are all integral parts of "Data Science".

Large resolution image:
https://whatsthebigdata.files.wordpress.com/2013/07/datascientistmap.png


How do I use LSTM Networks for time-series classification problems? I have 2 binary outputs (1 and 0) with time series data. The dataset order is shown in the image..Can anyone suggest me how to handle this problem with LSTM? Particularly in MATLAB or Python. Thanks in advance.

This can be done with RNN/LSTM/GRU (type of Neural Networks that are well-suited for time-series).

For example : https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/

This example is quite similar to the problem mentioned in question (predict air quality based on ~10 parameters. Parameters are available as a time-series).


Loss is decreasing but val_loss not! If loss is decreasing but val_loss not, what is the problem and how can I fix it?

I get such vague result:

This indicates that model is not generalizing (it is over-fitting). Few options are :

Get more training data
Reduce complexity of model (Number of LSTM layers, complexity of dense layers)

Andrew NG has a good video on this topic :

https://www.youtube.com/watch?v=OSd30QGMl88

A tutorial specific to LSTM :

https://machinelearningmastery.com/diagnose-overfitting-underfitting-lstm-models/


Feature selection method explanation In the context of feature relevance, I am trying to understand the meaning of the correlation method for feature selection. Can somebody please explain if the following results of the correlation coefficients arise, then should I take that feature? The rule is to select the features for which corrcoeff values are greater than 0.5. Please correct me if wrong. The way I am calculating is using Matlab's corrcoeff(target,feature) where target and feature are vectors

Case1: corrcoeff returns NaN values --

Nan Nan
Nan  1

Should the feature be selected since the value is greater than 0.5?

Case2: corrcoeff returns 0 values

0 0
0  1

In this case, I should reject the feature.

Case3:

-0.3 0
0    -0.3

Negatively correlated but absolute values less than 0.5, so reject the feature

Case4: What if there is no linear relationship at all in which case corrcoeff will not work. How do I know if there is no linear relationship and in that case how to do feature selection; is there any other function or technique?

The idea is quite simple: If you find features that are correlated with your target variable but uncorrelated with each other, it will give you information with which you can predict your target variable.

The threshold of 0.5 is chosen by a person and should be changed depending on your data. For example, if you find 10 features that are uncorrelated with each other but all have a 0.3 absolute correlation coefficient with the S&P 500 than you found the stock market equivalent of the Holy Grail.

Also, as you already mentioned, the correlation coefficient only captures linear correlations. This could cause you to exclude strongly related features just because they have a more complex relationship to your target variable.

Therefore, I would strongly advise against this method. Instead, I recommend plotting the data in pairplots.


What are the input and output channels of a convolution in PyTorch? From the documentation of Pytorch for Convolution, I saw the function torch.nn.Conv1d requires users to pass the parameters "in_channels" and "out_channels". I know they refer to input channels and output channels but I am not sure about what they mean in the context of convolution. Could someone explain this to me?

in_channels is the number of channels of the input to the convolutional layer. So, for example, in the case of the convolutional layer that applies to the image, in_channels refers to the number of channels of the image. In the case of an RGB image, in_channels == 3 (red, green and blue); in the case of a gray image, in_channels == 1.

out_channels is the number of feature maps, which is often equivalent to the number of kernels that you apply to the input. See e.g. https://stats.stackexchange.com/a/292064/82135 for more info.


How to get the tagset for hindi pos tagging? I am trying to tag Hindi text using python(nltk library). I have been successful but I am unable to understand some of the tags.

I tried searching for tagset but the only information I could find is about some upenn_tagset. The tags that I get matches only some of the tags from upenn_tagset.

Does anyone know how to get the tagset for hindi?

I'm not familiar with NLTK but the tagset must come from the annotated corpus which was used to train the tagger. According to https://www.nltk.org/book/ch05.html (section 2.2), if you can find the name of the Hindi corpus you should be able to access the words with their tags with:

nltk.corpus.<corpus>.tagged_words()

Apparently this is the list of corpora available in NLTK: http://www.nltk.org/nltk_data/, it might help finding the one used to train the Hindi POS tagger.

Once you identify the corpus, it's likely that you can find explanations about the tagset and annotation process by searching for papers about its creation by the original authors.


SKLEARN GridSearchCV hinting higher accuracy than Pipeline but with same parameters as Pipeline estimators I have pipeline estimators like this:
text_clf = Pipeline([
      ('tfidf', TfidfVectorizer(max_df=0.95, min_df=2, max_features=n_features, stop_words='english')),
  ('clf', SGDClassifier(loss='hinge', penalty='l2', alpha=1e-3, random_state=random_state, max_iter=5, tol=None)),
])
text_clf.fit(dataset.data, dataset.target)

Then when evaluating the model accuracy like this
mean = np.mean(predicted == twenty_test.target)
print("mean %0.3f" % mean)

I get score 0.802.
Then when I add the GridSearch to get the best params like this:
parameters = {
     'tfidf__use_idf': (True, False),
     'clf__alpha': (1e-2, 1e-3),
}

gs_clf = GridSearchCV(text_clf, parameters, cv=5, n_jobs=-1)
gs_clf = gs_clf.fit(dataset.data, dataset.target)

(Note that i am fitting GridSearch on TRAIN data same as the pipeline - although when i first tried fit it on TEST data by mistake the result was same though.)
It reports this:
grid search best score 0.868
Best params: clf__alpha: 0.001
Best params: tfidf__use_idf: True

Note, that these params are already set on the model but the score is lower in the pipeline.
Is it because the other parameters I set in Pipeline are not kept when using GridSearch or?
Btw how the Grid Search knows the best parameters if I didn't provide any testing data.
Another problem is, that when I added more parameters to adjust in Grid search and then applied it to Pipeline, the accuracy didn't change at all. (or changed from 0.802 to 0.805, but GS hinted 0.867)

The GridsearchCV object in sklearn does a cross-validation on the data you feed it during your fit. In your case you have specified cv=5: this means GridSearchCV splits your data into train/test splits 5 times and reports on the mean performance over those 5 trials to be 0.868.
You asked why GridSearchCV knows the best parameters without feeding it testing data - this is because it's taking your training data & splitting that into a smaller subset of train/test splits (5 of them to be exact).
You are then evaluating the best model on the testing set and getting accuracy of 0.805. This can happen if the single train/test split you performed split the data in a way that causes some of the more difficult samples to predict to be put in the testing set. If you were to perform the same cross-validation your GridSearchCV performed (say with cv=5) you might find the average performance to be a little higher and closer to the 0.868 it cited.


SGDClassifier partial_fit() for online learning - is one step of gradient descent enough? I'm interested in incremental (online) learning for my logistic regression model trained with SGDClassifier. Basically updating the model as more labeled data comes in. I know I can use partial_fit() for this. The following seems to be a common example on StackOverflow:

clf = linear_model.SGDClassifier()
x1 = some_new_data
y1 = the_labels
clf.partial_fit(x1,y1)
x2 = some_newer_data
y2 = the_labels
clf.partial_fit(x2,y2)

The problem is, partial_fit() performs just one step of gradient descent:

  Internally, this method uses max_iter = 1. Therefore, it is not guaranteed that a minimum of the cost function is reached after calling it once. Matters such as objective convergence and early stopping should be handled by the user.

Would one step be enough? How do I know if I don't need to make more steps than just one per each new labeled datapoint / minibatch that comes in?

Monitor the metrics that you are attempting to optimize, and compute the change in those metrics after each epoch. Once those metrics stop improving according to certain criteria, then you can stop gradient descent. See this article for a way to keep track of the metric improvements.


Keras: apply multiple filters to each feature map in CNN I am new to Keras, and I want to do the following: take a 2D image, and apply four 2D convolution kernels to it, giving four 2D feature maps. I could accomplish this. But then I want to apply two  distinct 2D convolutions to each of those 4 maps, giving 8 feature maps. Is that possible?
Here's what I have so far:
import keras
from keras.layers import Conv2D

input_img = keras.Input(shape=(N_rows, N_cols, 1))
x = Conv2D(4, (3,3))(input_img)

But then I don't know how to apply 2 kernels to each of the 4 channels, so that I have eight 2D maps.

You may try Keras DepthwiseConv2D layer

Depthwise Separable convolutions consist of performing just the first step in a depthwise spatial convolution (which acts on each input channel separately). The depth_multiplier argument controls how many output channels are generated per input channel in the depthwise step.

It will convolute each Channel separately. As shown in this depiction.

$\hspace{6cm}$Image credit - Blog by Chi-Feng Wang
With depth_multiplier argument, you can add more Filetrs i.e. more copy of the "triplet" shown in the depiction.

depth_multiplier:
The number of depthwise convolution output channels for each input channel. The total number of depthwise convolution output channels will be equal to filters_in * depth_multiplier.


Why using L1 regularization over L2? Conducting a linear regression model using a loss function, why should I use $L_1$ instead of $L_2$ regularization?

Is it better at preventing overfitting? Is it deterministic (so always a unique solution)? Is it better at feature selection (because producing sparse models)? Does it disperse the weights among the features?

Basically, we add a regularization term in order to prevent the coefficients to fit so perfectly to overfit.

The difference between L1 and L2 is L1 is the sum of weights and L2 is just the sum of the square of weights.

L1 cannot be used in gradient-based approaches since it is not-differentiable unlike L2

L1  helps perform feature selection in sparse feature spaces.Feature selection is to know which features are helpful and which are redundant.

The difference between their properties can be summarized as :


Is the neural network in DQN used to learn like a supervised model? Is the neural network in DQN used to learn like a supervised model?

Is the neural network in DQN used to learn like a supervised model?

Yes.

In DQN, the neural network is used as a function approximator to learn the action value function $Q(s,a)$ - the neural network approximation to it is sometimes noted $\hat{q}(s,a,\theta)$ to show that this is an ongoing, approximate estimate, and that the values that it outputs depend on the NN's learnable parameters (weights and biases).

Why is this "supervised learning"? That is because in order to improve an estimate for Q, the network is presented with labelled data. In DQN this is specifically a mini-batch drawn from experience replay memory, but in all cases in RL, the format of each row of data is the same: A representation of the current state (or state+action sometimes), and a "ground truth" target value to learn to associate with that input. This learning by correct example is exactly supervised learning.

Now, there are some complications:

The "ground truth" is built by the RL algorithm on-demand, and equals the underlying real value $q_{\pi}(s,a)$ only in expectation over time. However, that is fine for any statistics-based learning algorithm that will learn a mean value.
The expected "ground truth" changes over time as RL improves the policy, so higher Q values will be expected, and the old lower ones need to be forgotten. This is why RL algorithms build the "ground truth" on demand - for single-step Q-learning, as used in basic DQN that is the $r + \gamma \text{max}_{a'}\hat{q}(s', a')$ formula that you may see in explanations, and is called the TD Target. That means that RL algorithms prefer to work with online algorithms that are biased towards most-recent data and forget older data. Neural networks and linear regression trained by gradient descent both fit this requirement.

An aside: In RL, to avoid using the term model twice, the term "function approximator" is used, where supervised learning might call the same thing a "model". That is because RL already uses the term model to refer to models of the environment (e.g. the rules of a game, or all the known transition probabilities and rewards).


How to assess nested cross validation results in comparison to non-nested results? I have a nonlinear regression model scoring genes from scores between 0 to 1 as to whether they are likely to cause disease. Training data is ~700 gene samples by 53 features.
Currently I get results with xgboost which look like:
r2 Nested CV Average: 0.807
MSE Nested CV Average: -0.016

Non-nested Results:
XGBR Train r2: 0.949 Test r2: 0.805
XGBR Train MSE: 0.002 Test MSE: 0.018
r2: 0.895
Predicted r2: 0.871

Should I be concerned that the non-nested training result is overfitting in comparison to the test result, or should I only rely on my 5-fold nested cross-validation to determine overfitting is minimised?
For reference, the XGBoost model I use is tuned like this:
xgbr = xgboost.XGBRegressor(random_state=seed, objective='reg:squarederror')
xgbr_params = {
    'max_depth': (1, 10),
    'learning_rate': (0.01, 0.5),
    'n_estimators': (20, 50),
    'reg_alpha': (1, 10),
    'reg_lambda': (1, 10),
    'gamma': (0, 0.5),
    'min_child_weight': (1, 5),
    'subsample': (0.1, 1),
    'colsample_bytree': (0.1, 1)}

#Best parameter output:
xgbr = xgboost.XGBRegressor(random_state=seed, subsample=0.8258568992489053, min_child_weight=1,
n_estimators=50, gamma=0.0, objective='reg:squarederror', colsample_bytree=1.0,
learning_rate= 0.3987519903467713, max_depth=4, reg_alpha=1, reg_lambda=10)

That is not a regression problem. Thus those results can not be interpreted.
It should be framed as probabilistic binary classification. The target is binary because there are two outcomes - disease or not. Probabilistic because it on a scale from 0 (not possible) to 1 (completely certain).


How can I change the transparency of a histogram plot in Seaborn using Pairgrid? I'm using the Kaggle Titanic dataset. One feature is "Embarked", the city the passenger embarked from. The survival rate appears to correlate with it, but I'm worried it may just be correlated with the ticket Fare (which the survival rate definitely correlates with).

I want to plot a histogram of the fares. That would be easy. However, I also want to, on the same plot, have the histograms for the three embarked values (Q,C,S), labeled by different colors.

I've searched but can't figure out how. I can achieve something relatively similar with FacetGrid:

g = sns.FacetGrid(train_df, col='Embarked')
g.map(plt.hist, 'Fare', bins=20)

which gives three separate histograms, almost what I want:

But that's not quite what I want. For example, it's not great for comparing them. I could calculate the averages, but ideally I'd see them overlaid, so the three histograms are on the same plot, and you can tell them apart because they're labeled different colors.

It seems like PairGrid is almost what I want, but it appears to be giving some strange axis for "Fare" (which is not between 0 and 1), and the legend is blank, and the colors are solid (so you can kind of only see the top layer).

g = sns.PairGrid(train_df[["Embarked","Fare"]], hue="Embarked",size=8)
g = g.map_diag(plt.hist)
g = g.add_legend()

I'm very confused and not sure how to even go about finding the right answer. It seems like Seaborn can do anything, some really cool things, but it's just impossible to find the specific thing I want to do for a given circumstance.

How could I achieve this?

With a pandas dataframe, the following should do it without seaborn:

train_df.groupby('Embarked')['Fare'].plot.hist(alpha=0.5)

If you have to use seaborn you can use FacetGrid without the col and row argument:

g = sns.FacetGrid(train_df, hue='Embarked')
g = g.map(plt.hist, 'Fare', alpha=0.5)
g = g.add_legend()


Tuning svm and cart hyperparameters I am trying to optimize the hyperparameters of SVM and CART with tune() function of e1071 R package, but I have a doubt.
Should I tune the parameters on the training data, fit the model on the training data and then test it on the test data, or may I avoid the second step?

There's two common approaches here, depending on how much data you have.

If you have plenty of data, you'll probably be fine to:

set part of the training set aside, and call it your validation set;
start with a set of hyperparameters, train a model on your training set, evaluate performance on the validation set;
repeat step 2 with different hyperparameters;
pick the hyperparameters which give you the best score on the validation set;
train your model on the training set and the validation set;
Test your model ONCE on your test set.

Otherwise, you can do the following:

start with a set of hyperparameters, evaluate your model's performance on unseen data via cross-validation on the training set;
repeat step 2 with different hyperparameters;
pick the hyperparameters which give you the best score on the validation set;
train your model on the entire training set;
Test your model ONCE on your test set.


How many trees does a Random Forest need? At first, I did a GridsearchCV and the best parameter I found was 100, i.e., a random forest with just 100 trees. My trainset has 80,000 rows and 669 columns. My test set has 20,000 rows and 669 columns.
How is it possible that such small number of trees is enough?

By other posts and this one seems what you don't have a clear intuition of the n_estimators of the random forest.

I am going to assume that you are referring to the n_estimators (from this other question). n_estimators is the number of trees that your 'forest' has. Not the depth of your tree. That is another parameter.

If you are referring to max_depth = 100, 100 splits it can be a lot. Feel free to plot one tree and see what it is doing. see this link

The number of rows and columns is not necessarily important, what is important is the complexity of your problem.


An alternative algorithm to sklearn KMeans that separates values by similarity? I have the following dataset:

        node        bc cluster
1    russian  0.457039       1
48       man  0.286875       1
155    woman  0.129939       0
3        bit  0.092721       0
5      write  0.065424       0
98       age  0.064347       0
97     escap  0.062675       0
74      game  0.062606       0

Then I perform kMeans clustering by bc value to separate the nodes into two different groups. Right now with the code below I get the result above (the clustering result is in the cluster column).

    bc_df = pd.DataFrame({"node": bc_nodes, "bc": bc_values})
    bc_df = bc_df.sort_values("bc", ascending=False)
    km = KMeans(n_clusters=2).fit(bc_df[['bc']])
    bc_df.loc[:,'cluster'] = km.labels_
    print(bc_df.head(8))

Which is pretty good, but I would like it to work slightly differently and to select the first 4 nodes into the first cluster and then the other ones in the 2nd one, because they are more similar to each other.

Can I do some adjustment to kMeans or maybe you know another algorithm in sklearn that can do that?

Welcome, Dmitry!

A very first red alarm ... If you are doing clustering, then let it tell you who is similar to who! If you already know your clusters, then why are you doing clustering indeed?!

And about the algorithms: Yes there are many more, for example, Spectral Clustering which makes use of different similarity measures (via Gaussian Kernel or K-NN connectivity matrix) to find clusters or DBSCAN which is based on similarity according to the density of data points.

If you needed more help please drop a comment here!

Good Luck!


Train model for predicting events based on other signal events I am trying to build a supervised ML-Model in the area of predictive Maintenance where the features are signal events, which can be represented with a gantt chart:

So my feature data consists of events with signal_code (each row in the gantt chart represents all events of a certain signal code), startTime, endTime, status (represented as the color in the chart). There can be up to a few hundred different signal codes.
I want to predict the likelyhood of an event (e.g. dark red in chart) say 30 minutes before it occures.
How could I detect "malicious" patterns prior to those events? Are there existing best practices for a use case like that?
I could find a lot information on time series (related question) but not for this kind of data.
Obviously i will ne to apply some dimensionality reduction techniques such as PCA or LDA, but what will a do beforehand in order to put the data into a one-dimensional representation?
I welcome any advice on which algorithms or feature extraction methods might help! :)

For somebody new to ML you're starting with a quite complex problem!
Your first job is to formalize the problem: what exactly do you want to predict and from which information?

Do you want to predict whenever a specific signal will occur, or whenever any one of several specific signals occurs, or detect any unusual pattern...? In the first two cases your task is supervised classification, detecting unusual patterns would probably be unsupervised outlier detection.
In the case of supervised classification, which seems more compatible with your description, you're going to use the specific signal to detect as target variable: your features are made of a sequence of data until time $t$, and the target variable is 0 or 1 depending on whether the signal occurs in the next 30 mns.

There's a lot of good advice that  you can use in the related question you mention. You could consider sequence labeling methods, but there are certainly a lot of other options.
I'm not so sure you will need dimensionality reduction in your case: the signals you're using as features are binary so your features are much less complex than in the related question.


Multidimensional Scaling with Categorical Data I have read the following about MDS in a book:

  using MDS requires an understanding of the individual feature's units;
  maybe we are using features that cannot be compared using the
  Euclidean metric. For instance, a categorical variable, even when
  encoded as an integer (0= circle, 1= star, 2= triangle, and so on),
  cannot be compared using Euclidean (is circle closer to star than to
  triangle?).

I accept the statement above, but it raises a few question about the application of MDA:

Given the fact that many conventional data sets contain categorical
features, does it mean that MDA cannot fit to these sets?
Maybe it would be a solution to change the distance measure type (e. g.
"Euclidean") to other, but Sklearn has no other built-in option,
not to mention R, where cmdscale has no option at all to specify
distance type. How to change this feature in general?

An additional question: I have read that PCA is a kind of MDS (or vice versa), apart from the fact that the former focuses on variance, the latter on keeping distance. Am I right that the two "converges" somehow (in case of visualization with the two first component, for instance)?

MDS only requires a distance matrix which stores the distance between each pair of data examples. How to compute that distance depends on the kind of data you are dealing with. If you only have numerical (real valued) features, you can use the Euclidean distance, but that is not always the case.

For instance, if you have both numerical and categorical variables, you may apply any metric for mixed data, like for instance this one.

Regarding your second question, I need further clarification. What do you mean by "converging? Are you asking if both algorithms converge to the same solution? Is that is your question, the answer is no, because the aim of both algorithms is different. MDS projects your data onto a 2D plane by trying to keep relative distances, whether PCA projects data by focusing on those "directions" in which there is more variability.


Building Recommender for book paragraphs I have some application which are offering a book to read. Users normally read some paragraphs of it only (it contains +6000 paragraphs).
Looking at scatter for users vs paragraphs:

Which you can see is semi equal distributed. Using SVD algorithm for matrix factoize gives a semi random predictions. I have total of 18k records of user read paragraphs. Looking for users, it seems that a user is reading semi random set (i.e. it is hard to specify common topics for a single user readings)
Can you suggest me how to produce suggestions related to each user ?

You would have to come up with a metadata set of each paragraph and then create a content-based recommendation system based on that metadata. You could also do a community-based recommendation system if you have extensive user data.

As for the metadata set, you could use something like n-grams to "summarize" each paragraph and then have an index for each. From there, you would know what people like and then algorithmically choose the n-grams that person is most interested in and find the paragraphs that correspond to those n-grams.


How differential semi-gradient Sarsa updates estimated average reward? I cannot understand the way how algorithm Differential Semi-gradient Sarsa updates its estimated average reward $\bar{R}$. The algorithm I am looking at is from Sutton's text book Reinforcement Learning:An Introduction , section 10.3.

Why does not update $\bar{R}$ using the reward $R$ got at current step like $\bar{R} = (1-\beta)\bar{R}+\beta*R$ ? Since based on definition, $\bar{R}$ is the estimated average reward. I cannot understand the reason why the updating is like this: $\bar{R} = \bar{R}+\beta*\delta$, where $\delta$ is just the TD error. Why using TD error to update the average reward?

The following figure shows the algorithm.

In a continuing task, TD error can increase w to infinity unless its expected value is zero. By subtracting the TD average estimation, the expected value of our update value is zero and w cannot go to infinity. TD error is a biased estimation of the average reward ( with a bias that goes to zero as the number of updates goes to infinity assuming that every state can be reached from every state since the average reward is independent of the state action combination with which we start).


Neural Network stacking layers I was watching Andrew Ng's video on ResNets, and he mentioned that "And in theory, as you make a neural network deeper, it should only do better and better on the training set."
Here is my understanding of neural networks, as the model progresses through the model, the parameters it will learn will become more and more sophisticated, correct? Intuitively, it should be able to recognize/discover more detailed pattern information about the training set. Is my understanding correct?

Then why, in practice, does adding excessive layers to Neural Networks actually harm the performance of the model?
Thanks in advance.

The answer given so far is wrong.
1.) In general, adding more and more layers does not lead to better training accuracy.
2.) The issue is certainly not overfitting, as Andrew Ng is talking only about the training set.
The answer becomes true if you add layers, where each additional layer $l$ is represented by a mapping $f^{(l)}_{w}: \mathbb{R}^{n_{l}} \rightarrow \mathbb{R}^{n_{l}}$ for which weights $w$ exist, such that $f^{(l)}_{w}$ is the identity map. In such a case, you can add arbitrary many layers to an existing neural network architecture and in theory optimizing this new network cannot be worse than without the additional layers (since all additional layers can become the identity map). Potentially these new layers could improve the training accuracy.
In practice this is not observed (see the ResNet paper). The reasons are mainly
1.) The employed solvers do not deliver the global optimal solution.
2.) Adding more layers makes the gradient computation more prone to numerical errors due to the involved chain rule, e.g. have a look here.


Visualizing item similarities I have an implicit dataset. It contains which user click which item. I'm doing collaborative filtering and finally i get the item similarites. So now i have data like;

Item - SimilarItem - SimilarityValue
A    - C           - 0.12
A    - R           - 0.42
A    - Y           - 0.34
A    - J           - 0.62
B    - A           - 0.16
B    - Y           - 0.83
C    - J           - 0.23

or

Item  --  Similarities
A     --  C,R,Y,J
B     --  A,Y
C     --  J,A,D

As i know all the relations between 40K item, can i turn this into a nice visualization showing some of them get together at somewhere and someothers are elsewhere. I can do this either with python or some javascript library.

What should be the way of doing this? Can you show me some examples?

You could treat your results as an undirected graph with weighted edges. Your nodes are A, B, C, etc. and your vertices are the connections between items, weighted by similarity value.

Here is a Python function that could be a great starting point. It draws thicker connections for vertices with a larger weight. I copied it and replaced the example data with your data points above:

[...]
node_list = ['A', 'B', 'C', 'D', 'J', 'R', 'Y']
[...]
G.add_edge(node_list[0],node_list[2],weight=0.12) # A - C
G.add_edge(node_list[0],node_list[5],weight=0.42) # A - R
G.add_edge(node_list[0],node_list[6],weight=0.34) # A - Y
G.add_edge(node_list[0],node_list[4],weight=0.62) # A - J
G.add_edge(node_list[0],node_list[1],weight=0.16) # A - B
G.add_edge(node_list[0],node_list[6],weight=0.83) # B - Y
G.add_edge(node_list[2],node_list[4],weight=0.21) # C - J

The result is this:

Of course, if you have 40,000 items, that would probably require additional work to make the plot comprehensible. You could begin by ignoring all connections between nodes with a similarity value of, say, 0.01 or less.


You can use the built-in date_range function from pandas library to generate dates and then add them to your dataframe. You can use it in the following way:
In [9]: import pandas as pd

In [10]: df = pd.DataFrame({'column1':[34,54,32,23,26]})

In [11]: df
Out[11]:
   column1
0       34
1       54
2       32
3       23
4       26

In [12]: df['date'] = pd.date_range(start='1/1/1979', periods=len(df), freq='D')

In [13]: df
Out[13]:
   column1       date
0       34 1979-01-01
1       54 1979-01-02
2       32 1979-01-03
3       23 1979-01-04
4       26 1979-01-05

Here I have incremented the dates day-wise per row. If you want to generate dates in some other way, check out the official documentation here.


regression to model exponential distribution I have a work order system. I found out that the work order completion times are exponentially distributed. Every ticket has some features. I understand that regression is "a line that best fits the points". In my case, I want the line to be exponential curve. Is it possible ? Or is CDF for exponential distribution better option ?

My use case is I want to predict the probability of a work order completing by deadline. I was thinking of querying sets of data through "group by" & then compute and store the lambda parameters. When a new ticket comes in, I will pull the corresponding lambda & use CDF to determine %age. I am wondering if an ML algorithm can simplify this process.

I have .net stack. I use Accord.net. I'm open to other .net libs.

There was an MIT group that got a lot of attention a few years back for predicting twitter trends and doing so with really good ROC-AUC Scores.  The problem you are trying to solve is kind of reminiscent of the twitter trending problem.  The crux of the solution is that they avoided the hassle of parametric modeling through a grouping of similar time series followed by a classification.

There is a pretty good writeup of the method for the general public in Stan Nikolov's Blog post.  Notice that there is a video of a presentation by his thesis advisor also.

The basic idea of the method is to find the most common shapes in the time evolution hold those as the dominant classes.  These would be various exponential shapes in your case (from Nikolov's blog post):

Then you can take your training data and choose the best shape for each of your cases.  You can then use this data to train a classification algorithm.  Finally you can use this to predict the outcome of your new data (from Nikolov's blog post):

The full dissertation is also available online for download.

There's actually an entire company that is using this methodology and trying to sell it to companies.  They have some pretty cool promotional videos on youtube also, which won't teach you the methodology, but show how powerful it can be.

Hope this helps!


Online Audio annotation tools I need to find a decent online annotation tool to transcribe audio.

There are some requirements for a potential tool:

I should be able to deliver audio files to a few labelers. I should
be able to track which files went to which labeler.
It should be safe in terms of data storage.

Any suggestions?

I'm not sure if it meets all your criteria (mostly because I'm not sure I understand all your criteria!), but you could have a look at ELAN:

  Description: With ELAN a user can add an unlimited number of textual
  annotations to audio and/or video recordings. An annotation can be a
  sentence, word or gloss, a comment, translation or a description of
  any feature observed in the media. Annotations can be created on
  multiple layers, called tiers. Tiers can be hierarchically
  interconnected. An annotation can either be time-aligned to the media
  or it can refer to other existing annotations. The content of
  annotations consists of Unicode text and annotation documents are
  stored in an XML format (EAF).

See also the Wikipedia page about ELAN. There is also a Wikipedia list of related software, I don't know if this is relevant.


Comparing SMOTE to down sampling the majority class in imbalanced binary classification I have a binary classification task with imbalance between the two classes.
I want to compare SMOTE vs down sizing the majority class to the size of the minority class.

I trained the classifier with 3-fold validation using the two methodologies:

SMOTE to increase the size of the minority class to the majority class size
Downsizing the majority class to the minority class size with a random subsampling

To test which methodology works better I trained my classifier (Random Forests) with a 3-fold Cross-Validation.

The confusion matrices I get from 3-fold CV seems to promote the use of SMOTE (better classification performance for the two classes). I assume that this CV can be used to choose the best methodology.

However, when I test the classifier on a real testing set (which was kept out and not used for training or validation) I don't see a real superiority of the SMOTE algorithm w.r.t. random subsample of the majority class. The minority class is better classified, but at the expense of the majority class performance.

Is this a limitation of SMOTE algorithm or my model selection methodology (using 3-fold CV) has some flaws?

It's difficult to say without the actual data.

However, I can tell you that SMOTE creates artificial instances, hence, when used in much expanse can
"deviate" from the actual minority class data. It's difficult to determine the expanse. Many factors take place, firstly the Data, then the neighbouring coefficients.

P.S. You could try boosting using many random under Samples. Hence, instead of Random Forest you could try first Adaboost for instance were each classifier is trained on a different sub sample.


Passing TFIDF Feature Vector to a SGDClassifier from sklearn import numpy as np
from sklearn import linear_model

X = np.array([[-1, -1], [-2, -1], [1, 1], [2, 1]])
Y = np.array(['C++', 'C#', 'java','python'])
clf = linear_model.SGDClassifier()
clf.fit(X, Y)
print (clf.predict([[1.7, 0.7]]))
#python

I am trying to predict the values from arrays Y by giving a test case and training it on a training data which is X, Now my problem is that, I want to change the training set X to TF-IDF Feature Vectors, so how can that be possible?
Vaguely, I want to do something like this:

import numpy as np
from sklearn import linear_model

X = np.array_str([['abcd', 'efgh'], ['qwert', 'yuiop'], ['xyz','abc'],  ['opi', 'iop']])
Y = np.array(['C++', 'C#', 'java','python'])
clf = linear_model.SGDClassifier()
clf.fit(X, Y)

You can find a nice tutorial how to achieve that on this blog
This is acctually a part II. In part I the author discusses how and what is "Term frequency".
Link to part I.


What is the name of this visualization with a circle and internal arcs? I came across this visualization where there is a circle with data points on the circle and internal lines (arcs or edges) between the points on the circle. What is the name of this type of visualization?

How would I generate it from a 2-D table like the one below? I want the points A, B, C, D, and E on the circle and the value from point i to j to be an arc between them. Would something like d3.js be able to generate it?

This is a chord diagram. It is usuallly generated from a square matrix like the one you have. This can be done with specific libraries at least in R (circlize package) and Python (plotly). It's also possible to do it with d3.js but apparently not so easily.
General advice: I find the "X Graph Gallery" websites quite convenient for exploring different types of graphs: R graph gallery, Python graph gallery and D3 graph gallery.


why use a perceptron when it seems multiple IF staments do the same thing If we are to stick with an X and Y axis, the X axis being Time and the Y axis being test scores. Where more time equates to High test scores. You can use a binary classifcation algorithm to predict success. Wouldn't 2 if statements do the same thing

If (time > someValue)
   User will probably pass

Another Scenario
Is I have papayas, and they have two labels, squishness and color.

If the squishness is greater then 7 //on a scale of 1-10
   AND the color is green
     Then it is a good papaya

Else
   It isn't ripe.

I dont understand the value of a perceptron in these scenarios.

Machine learning (perceptrons or not) is all about automatically finding generic but correct rules, be it in the form of If-Else-Rules, encoded formulas, closest occurrences, or others.

The ML algorithm is just a way to (automatically) find this knowledge, whatever the representation may be. In another words, you use it to find the someValue of your example, based on your data. You don't need ML if you can represent such knowledge yourself.


(SOLVED) Power BI, Page level filter not working with many to one relation EDIT: Solution found.

Turns out the relations were set to Cross filter direction: Single on all relations (and the one from Month table.Month to A.Month was not set to active).

Setting Cross filter direction: Both fixed the problem.

I have four tables A, B, Month table and Quarter table

A has a column A.Month, and many other columns with irrelevant data

B has a column B.Quarter, and many other columns with irrelevant data

Month table has columns Month table.Month and Month table.Quarter

Quarter table has a column Quarter table.Quarter

One to many relations are shown as this: "<"

Quarter table.Quarter < Month table.Quarter as there are multiple (3) months in one quarter

Month table.Month < A.Month as there are several rows of data for each month

Quarter table.Quarter < B.Quarter as there are several rows of data for each quarter

Here is a visualisation of the relations

On my dashboard, I have a visual for A, and a visual for B

What I'm trying to do is to apply a page level filter with Month table.Month, thinking that therefore, A should only show data for whatever month/months are selected in the filter, and B should show the data for the corresponding quarter/quarters.

Filter to January

Expectation:

A shows data for January
B shows data for Q1

Reality:

A shows data for all months
B shows data for all quarters

Switch page level filter from Month table.Month to Quarter table.Quarter and set filter to Q1

Expectation:

A shows data for January, February and Mars
B shows data for Q1

Reality:

A shows data for all months
B shows data for Q1

What am I missing?

Posting this as an answer.

Turns out the relations were set to Cross filter direction: Single on all relations (and the one from Month table.Month to A.Month was not set to active).

Setting Cross filter direction: Both fixed the problem.


Trying to run a kaggle notebook I'm found an interesting problem on kaggle and more or less solved it with my limited knowledge of machine learning. I was curious how other people solved and checked the solution with the highest vote.
The solution includes a line where is implementing and training a model.
params = {'num_leaves': 8,
         'min_data_in_leaf': 42,
         'objective': 'binary',
         'max_depth': 5,
         'learning_rate': 0.01,
         'boosting': 'gbdt',
         'bagging_freq': 5,
         'feature_fraction': 0.8201,
         'bagging_seed': 11,
         'reg_alpha': 1,
         'reg_lambda': 4,
         'random_state': 42,
         'metric': 'auc',
         'verbosity': -1,
         'subsample': 0.81,
         'num_threads': 4}
oof_lgb, scores = train_model(X, y, params=params, folds=folds, model_type='lgb', plot_feature_importance=True)

But it is nowhere mentioned what kind of model he is using, which causes the error.
NameError: name 'train_model' is not defined

Does someone of you know which model he is using, so that i can run his notebook on my computer.

The actual code is included in the notebook, if you click on the 'Code' button just above the cell with the code you provide you will see the code used to define the train_model function. If you look in that function you will see that this function can use different models based on the model_type argument, and in the line of code you linked 'lgb' means that a gradient boosting model from the lgb library is used.


How to test People similarity measure? I am doing a project on finding famous people who are similar to each other.

For this, I am extracting a bunch of features and applying a distance function on them to evaluate who is closer to whom.

Is there a way to test this out quantitatively? Take the example of actors for easy reference.

Say, one model predicts Robert Downey Jr. to be most similar to Christian Bale. While after applying more weights to the 'heights' feature, George Clooney is found to be most similar to Christian Bale.

Now how do I quantitatively assert that one is better than the other?

One might say that Robert Downey is a better result because both Christian Bale and Robert Downey have starred in cult superhero movies (albeit different ones), while someone else might say that since George Clooney and Christian Bale were both batmans at one point, George Clooney is the better result.

Is there a way to test this other than intuition?

Not unless you have an idea of what makes two people similar to each other. By choosing a distance function you are defining what it means for two people to be similar to one another. The question of whether one metric is quantitatively better than another may not be answerable without a defined objective.

For example, compare the Manhattan distance (L1 norm) to the euclidean distance (L2 norm). The former may be more effective for a pedestrian trying to navigate a city on foot, whereas the latter may be more effective for an athlete running across a sports field.

  Now how do I quantitatively assert that one is better than the other?

You would need some examples which embody what it means for two people to be similar (or not) to one another. This would have to be generated by a person or group of people.

Rather than generate a "similarity value" directly between two people, one could instead generate a ranked list of people which are most similar to a given person of interest:
For person A, the most similar other person is B, the second most similar person is C, etc.
Better yet, presented with a person of interest and a list of other people someone would just have to rank each person in the list by their similarity to the person of interest.

If more than one person generate ranks of a given list of people for a given person of interest, you could agglomerate the results by summing the ranks for each person and reordering the list by summed rank.

If you have some data in this form, then you may evaluate your similarity model by how well it recreates the ranked lists. Some possible metrics for evaluation are:

Recall (at rank n): What fraction of the top n people are returned in the model's top n results?
Precision (at rank n): What fraction of the model's top n results are in the true top n results?

Look here for other methods of evaluating ranking methods.

I hope this helps. Let me know if I'm not understanding your question.


What does the oob decision function mean in random forest, how get class predictions from it, and calculating oob for unbalanced samples I am interested in finding the OOB score for random forest using sklearn, when it is used for a binary classification task, and there are unbalanced samples. What does the oob decision function mean in random forest, and how get class predictions from it?
I read RandomForestClassifier OOB scoring method but am still not clear. Does the oob decision function provide class probabilities, and if so, do I get the class predictions by taking whichever number is higher (e.g. by doing something like pred_train = np.argmax(forest.oob_decision_function_,axis=1))?
Since my classes are unbalanced, would it be correct to say I can't used sklearn's default OOB score here, and I should do the above to get some kind of F1 score from the OOB predictions, to get a better estimate of my random forest's error?

Every Tree gets its OOB sample.
So it might be possible that a data point is in the OOB sample of multiple Trees.
oob_decision_function_ calculates the aggregate predicted probability for each data points across Trees when that data point is in the OOB sample of that particular Tree.
The reason for putting above points is that OOB will give you the mean of probability but it will not tell you anything about the standard deviation of the probability across Trees.

Does the oob decision function provide class probabilities,

Yes

and if so, do I get the class predictions by taking whichever number is higher (e.g. by doing something like pred_train = np.argmax(forest.oob_decision_function_,axis=1))?

Yes

Since my classes are unbalanced, would it be correct to say I can't use sklearn's default OOB score here

OOB score is still the default score i.e. Accuracy. So, will not help for the Imbalanced class.


What does a Word2Vec represents? Does a single word vector, trained using Word2Vec or a similar approach, carries information or meaning?

So we have that a word to vector model has been trained on a certain corpus to be able given words as inputs (one hot encoding is one way for the input) to represent the word as a vector of a usually high dimensionality of numbers.

Does it carry information? Yes, we could say that it carries the information about the words. Similar words have similar vectors. The point is that us as humans we do not define the similarity but the neural network defines the similarity for us based on the corpus that we are providing.

Does it carry meaning? No, because this representation does include grammatical or syntactical information and therefore the answer would be that it does not carry any meaning.


How to train a model on top of a transformer to output a sequence? I am using huggingface to build a model that is capable of identifying mistakes in a given sentence.
Say I have a given sentence and a corresponding label as follows ->
correct_sentence = "we used to play together."
correct_label = [1, 1, 1, 1, 1]

changed_sentence = "we use play to together."
changed_label = [1, 2, 2, 2, 1]

These labels are further padded with 0s to an equal length of 512. The sentences are also tokenized and are padded up(or down) to this length.
The model is as follows:
class Camembert(torch.nn.Module):
    """
    The definition of the custom model, last 15 layers of Camembert will be retrained
    and then a fcn to 512 (the size of every label).
    """
    def __init__(self, cam_model):
        super(Camembert, self).__init__()
        self.l1 = cam_model
        total_layers = 199
        for i, param in enumerate(cam_model.parameters()):
            if total_layers - i > hparams["retrain_layers"]:
                param.requires_grad = False
            else:
                pass
        self.l2 = torch.nn.Dropout(hparams["dropout_rate"])
        self.l3 = torch.nn.Linear(768, 512)

    def forward(self, ids, mask):
        _, output = self.l1(ids, attention_mask=mask)
        output = self.l2(output)
        output = self.l3(output)
        return output

Say, batch_size=2, the output layer will therefore be (2, 512) which is same as the target_label.
To the best of my knowledge, this method is like saying there are 512 classes that are to be classified which is not what I want, the problem arises when I try to calculate loss using torch.nn.CrossEntropyLoss() which gives me the following error (truncated):
 File "D:\Anaconda\lib\site-packages\torch\nn\functional.py", line 1838, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), igno
re_index)
RuntimeError: multi-target not supported at C:/w/1/s/tmp_conda_3.7_100118/conda/conda-bld/p
ytorch_1579082551706/work/aten/src\THCUNN/generic/ClassNLLCriterion.cu:15

How am I supposed to solve this issue, are there any tutorials for similar kinds of models?

I think you should treat this problem as a binary classification problem.  For each word in the changed sentence, you will have a binary label: correct or incorrect.  I would recommend relabeling so that "correct" words will have a label of 0 and "incorrect" words will have a label of 1.  In your example you would have:
correct_sentence = "we used to play together"
changed_sentence = "we use play to together"
labels = [0, 1, 1, 1, 0]

And instead of padding with some special value, pad with the "correct" label (which would be 0 if you use my suggestion above).
Conventionally, class labels always start at index 0, so this labeling scheme will match what PyTorch expects for binary classification problems.

Next, you will need to change the activation function for your final Linear layer.  Right now, your model ends with just a Linear layer, meaning the output is unbounded.  This doesn't really make sense for classification problems, because you know that the output should always be in the range [0, C-1], where C is the number of classes.
Instead, you should apply an activation function to make your outputs behave more like class labels.  For a binary classification problem, a good choice for the final activation is torch.nn.Sigmoid.  You would modify your model definition like this:
class Camembert(torch.nn.Module):
    """
    The definition of the custom model, last 15 layers of Camembert will be retrained
    and then a fcn to 512 (the size of every label).
    """
    def __init__(self, cam_model):
        super(Camembert, self).__init__()
        self.l1 = cam_model
        total_layers = 199
        for i, param in enumerate(cam_model.parameters()):
            if total_layers - i > hparams["retrain_layers"]:
                param.requires_grad = False
            else:
                pass
        self.l2 = torch.nn.Dropout(hparams["dropout_rate"])
        self.l3 = torch.nn.Linear(768, 512)
        self.activation = torch.nn.Sigmoid()

    def forward(self, ids, mask):
        _, output = self.l1(ids, attention_mask=mask)
        output = self.l2(output)
        output = self.l3(output)
        output = self.activation(output)
        return output

Your output will now have dimension (batch_size, 512, 1).  Each of the 512 outputs will be a number between 0 and 1.  You can treat this as the probability of each particular token being "incorrect".  If the output is greater than 0.5, then the label becomes "incorrect".  Otherwise, the label is "correct".
Finally, since you're treating the problem as a binary classification problem, you will want to use binary cross-entropy loss (torch.nn.BCELoss).  Note that you will have to unsqueeze the labels to make their dimension match the dimension of your output.
model = Camembert(cam_model)
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

input = <tokenized, padded input sequence>
labels = torch.tensor([0, 1, 1, 1, 0, . . .  , 0])
output = model(input)
loss = criterion(output, labels.unsqueeze(1))

optimizer.zero_grad()
loss.backward()
optimizer.step()


When can you reorder log operations? For example, you can reorder a softmax + nl (negative likelihood)
 to  log_softmax + nll (negative log-likelihood)

Essentially changing log(softmax(x)) to softmax(log(x))

However, what are the rules to reordering logging of things?

In general, you cannot reorder like that. The example you give is a very special case that only works because softmax is based on the exponential function, which is the inverse function of the natural logarithm.


I got your problem like this way:
You want to show labels on x and y axis on seaborn heatmap. So for that sns.heatmap() function has two parameter they are xticklabels for x-axis and yticklabels for y-axis labels.

follow below code snippet

import seaborn as sns # for data visualization
flight = sns.load_dataset('flights') # load flights datset from GitHub seaborn repository

# reshape flights dataeset in proper format to create seaborn heatmap
flights_df = flight.pivot('month', 'year', 'passengers')

sns.heatmap(flights_df)# create seaborn heatmap

Output >>>

Now, we are changing x and y axis labels using  xticklabels and yticklabels sns.heatmap() parameters.

x_axis_labels = [1,2,3,4,5,6,7,8,9,10,11,12] # labels for x-axis
y_axis_labels = [11,22,33,44,55,66,77,88,99,101,111,121] # labels for y-axis

# create seabvorn heatmap with required labels
sns.heatmap(flights_df, xticklabels=x_axis_labels, yticklabels=y_axis_labels)

Output >>>

for in depth explanation follow seaborn heatmap tutorial.


Val_accuracy (val_acc) very low We have a data set that is converted from signal data to video. We want to classify these images using convolution. We tried many different methods but val acc is consistently low. Training accuracy is 99% and val_acc is 40%. We need your help in this respect. Thank you

weight_decay=0.0005
input_ = Input(shape=(125, 125, 1))
# Block 1
x = Conv2D(64, (3, 3), strides=(1, 1), activation='relu', padding='same',kernel_regularizer='0.001')(input_)
x = BatchNormalization()(x)
x = MaxPooling2D((2, 2), strides=(2, 2), padding='same', name='pool1')(x)

# Block 2
x = Conv2D(128, (3, 3), strides=(1, 1), activation='relu', padding='same',kernel_regularizer='0.001')(x)
x = BatchNormalization()(x)
x = MaxPooling2D((2, 2), strides=(2, 2), padding='same', name='pool2')(x)

# Block 3
x = Conv2D(256, (3, 3), strides=(1, 1), activation='relu', padding='same',kernel_regularizer='0.001')(x)
x = BatchNormalization()(x)
x = Conv2D(256, (3, 3), strides=(1, 1), activation='relu', padding='same',kernel_regularizer='0.001')(x)
x = BatchNormalization()(x)
x = MaxPooling2D((2, 2), strides=(2, 2), padding='same', name='pool3')(x)

# Block 4
x = Conv2D(512, (3, 3), strides=(1, 1), activation='relu', padding='same',kernel_regularizer='0.001')(x)
x = BatchNormalization()(x)
x = Conv2D(512, (3, 3), strides=(1, 1), activation='relu', padding='same',kernel_regularizer='0.001')(x)
x = BatchNormalization()(x)
x = MaxPooling2D((2, 2), strides=(2, 2), padding='same', name='pool4')(x)
x = GlobalMaxPooling2D()(x)

x = Dense(800,kernel_regularizer='0.001')(x)
x = Dropout(0.5)(x)
x = BatchNormalization()(x)
x = Dense(800,kernel_regularizer='0.001')(x)
x = Dropout(0.5)(x)
x = BatchNormalization()(x)
x = Activation('relu')(x)
x = Dense(8)(x)
x = Activation('softmax')(x)
model = Model(inputs = input_, outputs=x)

enter image description here

my kernel and dataset
https://www.kaggle.com/ultrasonraporlama/video-kernel/

Try adding dropout layers to your network, this should work very well to reduce the amount of overfitting of your network.


Why could an overfitted CNN model have a higher validation accuracy? I am currently training a CNN model by using cifar10 images (50000 for training, another 10000 for validation).

I plot training loss, validation loss and accuracy against training iteration:

I am not sure when should I stop the training, should I stop it at around iteration 1250, since overfitting happens beyond that point? Or should I stop it at around iteration 5000, since I can get the maximum validation accuracy?

Why can the overfitted model have a higher validation accuracy?

I will try to answer your question "Why can the overfitted model have a higher validation accuracy?" with an example, then try to answer the remaining questions (TLDR at bottom):

Say we have 2 classes with a 1:1 ratio in both the training and testing sets (each 4 instances).

Perhaps our training/validation curves are just like yours (I highlighted some points for reference).

Going from point A to point B ($\overline{AB}$), we do not observe overfitting:

validation accuracy increases
validation loss decreases

$\therefore$ We might conclude an inverse relationship between validation accuracy/loss.

However, $\overline{BC}$ shows overfitting:

validation accuracy is nearly constant
validation loss increases

This appears to contradict the relationship we observed with the $\overline{AB}$ example!

The reason behind this phenomenon is simple: we cannot assume a linear dependence between the accuracy and loss functions.

Let us define our accuracy and loss functions respectively:

$accuracy:\quad 1-\frac{1}{N}\sum^N_{i=1}|[X_i]-\hat{X}_i|$

$loss \space (MSE):\quad \frac{1}{N}\sum^N_{i=1}(X_i-\hat{X}_i)^2$

$\{X \space | \space 0<X<1\},\qquad\{\hat{X} \space | \space \{0,1\}\}$

We can see that these are linearly independent (which you could verify by calculating their Wronskian). How might this affect the relationship between training/loss curves?

Let us plug-in some numbers to see. Columns {A,B,C} represent the model output vectors for these three epochs and the corresponding ground truth is in $\hat{X}$ column.

\begin{array}{|c|c|c|c|}
\hline
A   & B   & C & \hat{X}\\ \hline
0.6 & 0.3 & 0.3 & 0\\ \hline
0.2 & 0.2 & 0.2 & 0 \\ \hline
0.4 & 0.4 & 0.1 & 1 \\ \hline
0.8 & 0.8 & 0.8 & 1 \\ \hline
\end{array}

\begin{array}{|c|c|c|}
\hline
epoch & accuracy \space calculation & accuracy & loss \space calculation & loss \\ \hline
A & 1-\frac{1}{4} * (1 + 0 + 1 + 0) & 0.50 & \frac{1}{4} * (0.6^2 + 0.2^2 + 0.6^2 + 0.2^2) & 0.2000\\ \hline
B & 1-\frac{1}{4} * (0 + 0 + 1 + 0) & 0.75 & \frac{1}{4} * (0.3^2 + 0.2^2 + 0.6^2 + 0.2^2) & 0.1325 \\ \hline
C & 1-\frac{1}{4} * (0 + 0 + 1 + 0) & 0.75 & \frac{1}{4} * (0.3^2 + 0.2^2 + 0.9^2 + 0.2^2) & 0.2450 \\ \hline
\end{array}

Therefore, $C_{loss} > A_{loss},\quad$even though$\quad C_{accuracy} > A_{accuracy}$!

Of course this is a very simple example, and your situation will have many more complexities, but establishing the linear independence of the validation accuracy and validation loss and understanding these curves is a critical aspect of determining which epoch model to select.

TLDR: With this in mind, you should use the model instance at epoch B. This is the point at which overfitting starts, not at epoch 1250. (although I suggest saving many intermediate models and empirically determining their viability on another dataset).

After epoch B, your validation accuracy is not changing while your training loss decreases. Therefore, your weights are fitting more to the training data without improving your results, ergo overfitting. At epoch 1250, your model is still improving validation accuracy quite dramatically, so stopping training then would yield a very poor model.

As we can see from your validation accuracy curve, this model will probably perform similarly to models beyond epoch B, but there are a couple reasons you might prefer model B.

If you ever wanted to resume training (perhaps you add more data to your training set but do not want to restart completely), then having weights that are not overly fitted (or even vanished!) to the initial dataset would be ideal.

A very similar reason is that, even though this particular dataset shows virtually no difference between the epoch B model and models beyond epoch B, other datasets might. It would then be easier to adjust model B for these new datasets rather than an overfitted model.


Which database to use for storing machine learning data? I am currently storing my training data into HDF5 files and I want my team and I to switch for a database for two main reasons:

the data is not used only by me and the different datasets are stored in different folders at different paths etc so I want to create a single database containing all the data
we need to store a lot of metadata like hashes, insertion dates, ground truth (that may differ depending on the application) etc.

I don't know which database to use. My first thought would be something like mongodb which is not relational, but I wonder if there are databases specialized in deep learning training out there. I heard of SciDB but I am not sure what to think about it, and I searched for "databases for machine learning" in google scholar without success. That is why I am asking the question here.
Thank you in advance for your help.

I share here the results of my quick research:
I think I will create a mongodb database to store everything, because it is easy to use and accross the NoSQL databases it is the one my company knows best.
If you want to use a database, you may also consider things like parallel access, distributiveness of the database, resilience, etc.
I think more advanced solutions like shared filesystems, spark, hadoop, sciDB (in particular, sciDB seems specialized for analytics on arrays only) etc are good for data lakes and when we want to process ALL the data or a big part of it at the same time, per batch or through a pipeline for example. Therefore, I don't need it for now, and the simpler the best.


How can I find out what class each of the columns in the probabilities output correspond to using Keras for a multi-class classification problem? I'm using transfer learning to build an image recognition model using a pre-trained VGG network in Keras and excluding the final fully-connected layer to get the output weights.  I'm then using these output weights to feed into my new model which has a few layers along with a new fully-connected layer of my own that I'm training. The fully-connected layer maps to the number of output classes that I'm trying to predict.

Everything is working fine.  However, when I run:

results = model.predict(img_tensor)

I get output probabilities corresponding to each class, similar to the below:

print(results)

[[0.1426621  0.6193871  0.23795079]
 [0.11187755 0.6208466  0.2672758 ]
 [0.10050113 0.3768951  0.52260375]
 [0.1338948  0.59470254 0.27140263]
 [0.06612041 0.69726    0.2366195 ]
 [0.12080433 0.495977   0.38321865]]

My question is: How can I find out what class each of the columns in the probabilities output correspond to?

Does Keras have anything built-in to identify which column of the output probabilities corresponds to which class?  I would be shocked if nothing is provided...

What have others done to create a work-around?

The class_indices attribute in Keras’ flow_from_directory(directory) creates a dictionary of the classes and their index in the output array:

classes: optional list of class subdirectories (e.g. [‘dogs’, ‘cats’]). Default: None. If not provided, the list of classes will be
automatically inferred from the subdirectory names/structure under
directory, where each subdirectory will be treated as a different
class (and the order of the classes, which will map to the label
indices, will be alphanumeric). The dictionary containing the mapping
from class names to class indices can be obtained via the attribute
class_indices.

Source
I would be interested to see how people have incorporated this into their workflow/scripts though...


How to identify topic transition in consecutive sentences using Python? I'm new to data mining. I want to detect topic transition among consecutive sentences. For instance, I have a paragraph (this could be a collection of dozens of sentences, sometimes without transitional words) as follows:

As I really like Mickey Mouse, I was hopping to go to Florida. But my
dad took me to Nevada. Obviously, Mickey Mouse was not there. But, I
attended a camp with other children. And, I really enjoyed and learnt a lot from my
camp.

Here, I want to automatically split this into following sub-paraphs:

As I really like Mickey Mouse, I was hopping to go to Florida. But my
dad took me to Nevada. Obviously, Mickey Mouse was not there.

But, I attended a camp with other children. And, I really enjoyed and learnt a lot from my
camp.

As far as I know, this is not the sentence similarity measurement. What technique should be used here? Any example using python or tensorflow models would be greatly appreciated.

One solution could be to:

Get sentence embeddings from FastText
Compute Euclidean Distance between the consecutive sentences
If the distance between the consecutive sentences is close to 1, then, you may say the two sentences are talking about different topics.

See here how to compute sentence embeddings for the English language: https://github.com/facebookresearch/fastText/blob/5b5943c118b0ec5fb9cd8d20587de2b2d3966dfe/python/fasttext_module/fasttext/FastText.py#L127

fasttext.util.download_model('en', if_exists='ignore')  # English
ft = fasttext.load_model('cc.en.300.bin')
fasttext.util.reduce_model(ft, 20)
def get_fasttext_sentence_embedding(sentence, ft):
    if pd.isna(sentence):
        return np.zeros(20)
    return ft.get_sentence_vector(sentence)

Then, compute the euclidian distance between fast text embeddings of consecutive sentences.

The same can be done using LDA (topic model), but, that would require a lot of text to model the topics.


What do we learn from training a dataset for logistic regression What do we learn from training our dataset in Logistic Resgression?
Like in Linear Regression, with the help of training set we are able to generate a best fit line(y = mx+c) where m and c come from training our dataset.
Similarly, once we train our model for logistic regression, what is that the model learns which is then used to predict a class for a particular input?

I hope this is what you are looking for:

In Linear regression, the parameters are learned (e.g. theta1,theta2, etc.) which are then used to help us in predicting the real value ( as it is the regression problem)
But In case of logistics regression (as it is used for classification purposes) the same parameters are learned but different activation function is used like sigmoid which results in the range of 0 to 1( which actually shows the probability of an outcome). So, based on your requirement, if the result of the sigmoid or any other activation function crosses a certain threshold value which can be any between 0 to 1, then it is classified as one class otherwise the other class.

For example, If after multiplying all the parameters and input values and then applying activation function on the multiplication result, the output value is 0.7 which let say is great than our threshold value(e.g. 0.5) then it is classified as YES otherwise NO.


Thoughts on Feature Engineering of a duration_in_program Variable So I am trying to predict which customers would leave a loyalty program sponsored by X firm, using an ML classification model.

I further believe that the duration for which a customer has been in the program affects their likelihood of staying/leaving the program, for reasons such as long-term customers get more loyalty discounts etc.. which may raise the indirect cost/price of them leaving a program.

However, one issue that I am currently facing, is that I am calculating duration_in_program, based on the start_date and end_date for each customer.

However, I think one issue with this approach of coding the variable is that there are values for duration_in_program that don't map to any "stayed" outcomes. Which kind of makes sense. Like, if everyone in the program with a duration_in_program of 3 yrs left the program, then the model will just learn to always predict that as "left".

It is crucial to point that: the program allows customers to stay in the program for a maximum period of five years, after that point, they receive regular prices paid by other cable customers.

Therefore, one way I am thinking of dealing with this is that duration_in_program, is determined by an arbitrary cutoff point. For instance, if they still have not left the program by the date at which the 1st cohort completes the program (i.e. 31/08/2017), then we consider their duration_in_program equal to that. So someone that joined in (i.e. 01/09/2013), and still has not left by 31/08/2017, then we set their duration_in_program=4.

Any thoughts on my approach above?

What you are describing sounds much like a survival model, where you have observations that are still surviving (still in the program) and observations that died along the way (attrition).

You would want to look into survival methods and how to censor your data. There are basic regression methods for this where you are trying to predict time-until-event. XGBoost can also handle censored data for survival.


How to combine multiple images features I have a dataset that contains information of pets (Breed,color,age,some text descriptions) and images of that pet. One pet can have more images than others.

I want to combine these images somehow to generate a single 1x255 feature so that I can merge it with the non-image features and use for my model.

One way I can think of is to put each image of a pet through a pretrained model to get a 1x255 output, then averaging out all of the outputs to get a single feature of that pet. But I havenät seen any paper/people doing the same to backup my intuition...

I'm new to data science, so if you have any thoughts just let me know. Thanks in advance.

Your initial intuition seems pretty good to me. A pretrained image classifier like VGG19 or ResNet, as included in the Keras library will yield simple vectors if the include_top parameter is set to False upon initialization. You can then use a method like PCA to reduce the dimensionality to 255, or whatever number your algorithm can handle. Averaging the vectors before or after this step will allow you to merge the photos. You will need to test which works best for your application.


Testing independence of random variables in Python Are there any tools available in Python that allow for testing of independence of two random variables (data columns)? I have two columns of data $X$ and $Y$. They can be both discrete, with values $\{0,1\}$ or one of them can be continuous. I would like to perform some statistical test to be sure they are independent. I am using Python so it would nice to have some ready-to-use tool implemented. I my also use R if it is not something difficult to do.

As is discussed in the link to Cross-Validated SO from Mephy, this is isn't an easy thing to do.

If they are independent, you might expect a correlation between pairs of variables to be close to zero. That would mean that knowing anything about one of the two variables doesn't give you any insight as to the behaviour of the second. To this end, there is a nice answer here, which shows how to compute pairwise pearson correlation (with corresponding p-values) for all columns in a Pandas DataFrame.

The Pearson correlation does assume your random variables to be normally distributed, so keep that in mind when interpreting results. Alternatively, you could swap out that pearsonr function for the Spearman Rank correlation function: spearmanr, which does not assume normality of your variables.

Another (perhaps simpler) way using just a Pandas DataFrame is to use the built in method corr: This takes a keyword method, which allows you to specify one of three:

  method : {‘pearson’, ‘kendall’, ‘spearman’}

If you random variables are time-series (you didn't mention it), another possible tool to look at would be Granger Causality. This could also be performed pairwise (or batch-wise) across variables. It tests to see if a the future value of variable can be better predicted when historic values of the a different variable are included in the model. For example, if the price of StockA can be predicted with an accuracy of 52% using its own prices of the previous 5 days, the Granger test would have a null hypothesis that including some lags from StockB would not improve the accuracy. So if the accuracy does indeed jump up to 53% when including lagged prices of StockB, (and the test is significant), the null hypothesis is rejected and we say that StockB Granger-causes StockA.

This is implemented in the vars package in R (there are others too). As a bonus, this version can also perform the Wald test for correlation in error processes of predictor and target variables.


Using Keras how and what do I need to export to use my classifier independently? I have a basic question that I can't seem to find an answer to.

I built and trained with good results (above 90% accuracy) a NLP Log classifier that takes in a UTF-8 payload and classifies it into 32 distinct categories but I am having a hard time writing a simple script that loads all the necessary info from my training and testing session (model.h5 and ?).

This is the structure of my code.

# load data logs and split it 80-20 for training and testing
vocab_size = 500
tokenizer = text.Tokenizer(num_words=vocab_size)
tokenize.fit_to_text(trainRawLogs)
x_train = tokenize.text_to_matrix(trainRawLogs)
x_test = tokenize.text_to_matrix(testRawLogs)

encoder = labelBinarizer()
encoder.fit(trainRawLogs)

#Model build is simple ReLu - Softmax

model.compile..

model.fit..

model.evaluate..

Now here is my question.

Out of all of this process what do I need to save to build a lightweight classifier? The model? The model and the labels? Anything else? I tried loading the model

In keras you have the option to save the entire model state including the optimizer parameters or simply the model weights. In the first case all you need to do is:

model.save(model_path)
model = load_model(model_path)

In the second case you have to first create your model and then load the weights:

model.save_weights(model_weight_path)

In case model is not already specified:

model = Sequential()
model.add()
...

model.load_weights(model_weight_path)

https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model


What is an object detection problem with only one class called? Object detection is defined as the problem in which a model needs to figure out the bounding boxes and the class for each object. A lot of ML solutions for object detection base around having "two passes" - one for creating the bounding box of the region and another for classifying it.

I was wondering if there is a name for a subset of this problem where $n_{classes} = 1$.  I feel like there is an interesting opportunity here as the whole classification part of the model can (basically) be ignored. Obviously, I can just train a typical object detection model with one class, but was just interested to see if there are any more specialized methods.

If you are talking about "two-stage" obejct detectors like Faster R-CNN, note that the second phase is not only for classification, but to obtain more accurate results (https://stackoverflow.com/a/61965140/4762996).

in addition, I guess training a detector with many classes acts like a regularizer, which results in much better accuracies.

The only benefit of explicitly training with one class could be the reduzed model size (and a corresponding speedup).

Note also that there are one stage object detectors (CornerNet: https://arxiv.org/abs/1808.01244, YOLOv3: https://pjreddie.com/media/files/papers/YOLOv3.pdf, DETR: https://arxiv.org/abs/2005.12872, and many more). I just wanted to stress that just because you only have one class, it does not make sense to take a two-stage detection architecture and skip the second stage.

Finally, have a look at: https://stats.stackexchange.com/q/351079

An interesting direction would be to incorporate one-class classification approaches, e.g. as mentioned here: https://stackoverflow.com/a/61965358/4762996.