What affect will replacing words with bigrams have on TfIDF? Say I have a corpus of text documents on which I have calculated each documents TfIDF vector. With this sparse matrix representation of the corpus, I can calculate similarities between documents by calculating the cosine similarity between the documents' TfIDF vectors.

If I now compile a set of bigrams based on frequency, and replace instances of two words in each document with the concatenation of the bigram, what affect will this have on

1) Calculating TfIDF vectors?

2) Calculating document similarity?

To be explicit, lets take an example in python:

doc1 = ['the', 'car', 'drove', 'from', 'new','york', 'to', 'washington'] # a single text document
top_n = {('new', 'york'), ('drove', 'from')...} # set of top n bigrams by frequency
new_doc = replace_ngrams(doc1) # function that replaces words with concatenated bigrams
print(new_doc)
>>> ['the', 'car', 'drovefrom', 'newyork', 'to', 'washington']

TFIDF decreases as term frequency will  be decreased linearly and idf increases log linearly.

Document similarity will decrease as value of tfidf vectors should decrease as reputation of bigrams are more less than each single word.


Predict multi-steps for Multi-Time series output with Keras I have a dataset like the following one:

Each column is a different numerical feature. Each row represents a timestamp. I want to create an LSTM model that can make prediction of the future time-steps for all the features. For example, I want to use the first 2000 examples to train my model and use the next 1000 to test it. The problem is that I do not know how to proceed.

Since we do not have y's value in this dataset, I was thinking of creating them by shifting the time t+1 to t has a new column y. I explain myself: for example I will have a new column 14, with the value 996.52 for timestamp 0, which is the value at time-step 1 for the feature 0. And so on for all the time-steps and all the features.

The problem is after that I do not know how to feed my LSTM using Keras to make several steps predictions with such a dataset.

LSTM inputs should be a tensor of size (#samples, time_steps, #features). Considering your dataset; you will have #samples = 2000-1 (for the training); time_steps = 1 (this is the lag-time you choose), and #features = 14. Then for the testing/validation #samples = 1000-1. But here, I suggest you take due consideration to the autocorrelation of the dataset; it seems that you choose the lag-time = 1 (just for the sake of creating the response variables); however, check out the temporal correlation + temporal structure of the data. Similar, to other Deep Neural networks, LSTM requires large dataset to train and test; checkout if you can increase the lag-time and get more predictor data. Have a look at Multi-dimentional and multivariate Time-Series forecast (RNN/LSTM) Keras this similar question for more information.


keras flow_from_directory returns 0 images When I try to use the following snippet of code to try to predict on a batch of images, I get a message saying that no image were found.

test_datagen = ImageDataGenerator()

test_generator = test_datagen.flow_from_directory(
        directory='test/',
        target_size=(300, 300),
        color_mode="rgb",
        shuffle = False,
        class_mode='binary',
        batch_size=1)

filenames = test_generator.filenames

My directory structure is as follows.
I first have a main directory named dogs_vs_cats in which I have two sub directories train and test containing the respective images and also the notebook which contains this code.

Keras generator alway looks for subfolders (representing the classes). Images insight the subfolders are associated with a class.

So whan you work on C:\images\ and you have two classes, say C1, C2, you need to create subfolders C:\images\C1\ and C:\images\C2\. The directory insight the generator function should point to C:\images\.

See this post for the case of image prediction: https://stackoverflow.com/a/55991598/9524424


Predict customer behaviour with Transformer(attention is all you need) Please advice, am I thinking correctly: is it possible to represent customer behavior data from an online store as a sequence data? Because it is describing interactions of the customer with the shop through time.
So in this case N would be the number of users (or number of user sessions), T / time window I could set myself and D I could set also myself taking only event type (purchase, view, etc.) or something else like price, brand etc.(please see screenshot below)

Please share your opinion
Many thanks in advance

Definitely, it is a good idea and has been attempted before.
Have a look at Alibaba's paper which uses transformers:
https://www.kdnuggets.com/2019/08/order-matters-alibabas-transformer-based-recommender-system.html
https://arxiv.org/abs/1905.06874
There are also various other papers that use other types of seq2seq for recommender systems.


Outlier detection and removal using scatter-plots and histograms Suppose, we use the following code to generate scatter plots,

function res = plot2features(tset, f1, f2)
% Plots tset samples on a 2-dimensional diagram
%   using features f1 and f2
% tset - training set; the first column contains class label
% f1 - index of the first feature (mapped to horizontal axis)
% f2 - index of the second feature (mapped to vertical axis)
%
% res - matrix containing values of f1 and f2 features

    % plotting parameters for different classes
    %   restriction to 8 classes seems reasonable
    pattern(1,:) = 'ks';
    pattern(2,:) = 'rd';
    pattern(3,:) = 'mv';
    pattern(4,:) = 'b^';
    pattern(5,:) = 'gs';
    pattern(6,:) = 'md';
    pattern(7,:) = 'mv';
    pattern(8,:) = 'g^';

    res = tset(:, [f1, f2]);

    % extraction of all unique labels used in tset
    labels = unique(tset(:,1));

    % create diagram and switch to content preserving mode
    figure;
    hold on;
    for i=1:size(labels,1)
        idx = tset(:,1) == labels(i);
        plot(res(idx,1), res(idx,2), pattern(i,:));
    end
    hold off;
end

The following is its usage,

>> plot2features(train, 3,4)

This code generates the following image before removing outliers,

and following image after removing outliers,

I have the following questions,

(1) What do the 1st image tell us about the existence of outliers? I can guess that the plot at a distant position is an outlier. But, how can I find which row or column is generating outliers? according to the 1st picture, the outlier is situated at (27,375) coordinates. But, in the actual data it is situated on the train(184:188,:) rows. So, why is that difference?

(2) What do the color codes in the second picture represent?

(3) Why has the two images that much different? Why only removing 4 rows bring so radical differnce?

(4) How can we analyze the existence of outliers using histograms? Please, supply me any study material about outlier detection using histograms.

Suppose we have the following training and test data in our hands to be used in testing Bayes Classifier algorithm,

Training data

train.txt

Test data

test.txt

First column represents class. Rest of the columns represent features.

Yes. This code does nothing but to plot the points for two chosen columns and assign different color and markers to them. It does not do any outlier selection/removal.
Column 3 and 4 (which are features in your dataset) are reasonably predictive of different classes.
Plot a histogram. See if there is any clear disjoint in among the bars. Investigate a little time to find a suitable bin size.

In addition, the title of your question does not match the content. Can you please update the question?


Original k-Means Research Paper I'm having difficulty searching for the original published paper proposing k-Means as an algorithm. I have been inspired to find it as reference for similar work, inspired by this TowardsDataScience article.
I have tried wiki's references and Google Scholar but with no luck.

According to the Wikipedia article, it doesn't look like there is a definitive research article that introduced the k-means clustering algorithm.
Hugo Steinhaus had the original idea in 1956. The standard algorithm was proposed by Stuart Lloyd at Bell Labs in 1957, but apparently that wasn't published until 1982.


How to backpropogate Convolution layer padding inputs with respect to output derivative I created a convolution network with 5 Conv blocks, let discuss the issue based on Conv block 4 & 5
Conv Block 4
Input Image size         : 28 * 28,
Padding size 1           : 30 * 30 (image size after padding),
Filter size              : 3 * 3,
Convolution output image : 28 * 28,
Max pooling output       : 14 * 14
Conv Block 5
Input Image size         : 14 * 14,
Padding size 1           : 16 * 16 (image size after padding),
Filter size              : 3 * 3,
Convolution output image : 14 * 14 ,
Max pooling output       : 7 * 7
Full connected layer

Above output image  (7 * 7) has been flattened and passed to Full connected layer and the flattened input has been backpropogated(dL/dO) with respect to loss(gradient checker results are fine).

Now I am trying to backpropogate through the convolution layer as below.

Back Porp - Conv Block 5
Reverse Max pool:  7 * 7 (dL/dO) to 14 * 14 output (dL/dO)
Conv 5 Filter derivative 3 * 3 (dL/dF): (14 * 14 output (dL/dO)) * (16 * 16 Conv 5 input image with padding)
I have issue with the below
Conv 5 input derivative:
input size 16 * 16 with padding (dL/dX) =  (14 * 14 output (dL/dO)) * (3 * 3 filter) after 180 degree
The question is if I have input derivative of (16 * 16) in conv 5 how can I reverse max pool to Conv block 4 because Conv 4 max output was only (14 * 14).
I think conv 5 input derivative should be 14 * 14 so we can reverse max pool to 28 * 28 and backpropogate the Conv 4 block.
Please help me to resolve this, I am sure I am missing something.
Thanks

You have derivatives for a 16x16 image including derivatives for padding - but these are worthless. Padding is not a trainable parameter. It is also not a function of previous layers which means computing padding's derivatives is not necessary to do backpropagation for all layers. For these reasons you can just cut out the middle 14x14 matrix and pass it to the max pool layer.
To visualise it even better imagine that you have 1D convolution. Your maxpool outputs 14 values and you pad it to the left and to the right with a 0. It should look like this.

Now notice that during backprop when you have calculated your 16 derivatives to reverse concatenation you simply split your matrix to 1-14-1 and pass 14 variables to maxpool. You can pass the leftover derivatives to padding if for example they're trainable parameters but it usually isn't the case.
That's why you don't have to calculate padding derivatives at all.


how to create multiple subplots( scatterplot) in for loop I want to create scatterplot of target variable with all the other variables in dataframe. Just like we create pairplot but as pairplot with large number of variables will take lot of time and computing memory, it is required to draw scatter plot containing all the variables with target variable.
So tried implementing it in for loop, but only last one got rendered properly.
count=1
for i in df.columns:
    plt.subplot(n,1,count)
    plt.figure(figsize=[5,5])
    sns.scatterplot(df["cnt"],df[i])
    count+=1

output: only last plot is rendered properly

Just get rid of the plt.figure(figsize=[5,5])
count=1
for i in df.columns:
    plt.subplot(n,1,count)
    sns.scatterplot(df["cnt"],df[i])
    count+=1

plt.show()

or use this to make it more beautier (Adjust the first two parameters of subplot function, i.e. 3, 2, to if you have more columns. First parameter specifies the number of rows for subplots in the plot, and second parameter specifies number of columns)
count=1
plt.subplots(figsize=(10, 8))
for i in df.columns:
    plt.subplot(3,2,count)
    sns.scatterplot(df["cnt"],df[i])
    count+=1

plt.show()


How to go about text mining for suggestions/Tips in reviews for restaurants? For example for restaurants reviews usually have suggestions like "Go in the evenings", "order the so and so sauce with this dish"  or even "TIP: ask for the blah blah blah"

How can I detect such sentences? How do people usually tackle similar challenges?

Do they create classification rules like <modal_verb><preference_verb><optional_window_size_of_3><positive_sentiment_words>
Some examples of these rules are “would be great” and “could be really good”  --
 found this from here.

I guess I would have to use a tagger to categorize words?

Any blog that has attempted something similar step by step?

Any help would appreciated.

So the way I thought about it is - abstractly you are attempting to identify which sentences are, Imperative Mood (IMP) and which are not. One option is to put together a training set/validation set, where each row is a phrase's pos tags (can use some encoder to convert to integers) with a target value of 'isImperative' (binary) then train a model and restructure each review so that they resemble the training set format and have your model assign the values. Seems the easiest and most sustainable method from where I am sitting. Sorry but I don't know any easily accessible phrase data sets but I bet there is one out there somewhere. Hope this helps.


K fold cross validation algorithm Suppose I want to apply cross validation without any inbuilt function. I have multi classifier problem which is perfectly  divided in n section where n is the number of target features. If I divide my training data sets in to k folds sets then I might end up having single target feature in one set. So to solve this problem I shuffle my training set and target and I am able to achieve 0.96 accuracy compared to 0.98 when I use inbuilt method in sklearn. Is there a better approach to divide your data perfectly in to k folds?

It seems there is a small bit of ambiguity in the question. But my best guess is that you are looking for stratifiedkfold (Emre has mentioned that in a comment) or second best guess is labelkfold.
See: http://scikit-learn.org/0.17/modules/generated/sklearn.cross_validation.StratifiedKFold.html#sklearn.cross_validation.StratifiedKFold
and
http://scikit-learn.org/0.17/modules/generated/sklearn.cross_validation.LabelKFold.html#sklearn.cross_validation.LabelKFold

It would help if you specify which inbuilt cross_validation in sklearn you tried, since there are several to choose from:
http://scikit-learn.org/0.17/modules/classes.html#module-sklearn.cross_validation

Additionally, please also let us know about the number of data points available, since that will contribute to deciding on the numeric value of k. The point being that you don't want to bias your training due to an counter-productively high value of k for the available n data points. Herein lies the reason for the labelkfold approach since it will ensure that the same data point does not repeatedly occur in multiple folds, which would end up biasing your classifier, and we want to avoid this.

Finally, if your training time is of the order of a few minutes, you could try several cross_validation schemes. It will be a good learning experience and might provide some useful insights into the data :-)

--
Since, you are exploring the Iris dataset, I've gone ahead and created a small demo of the effect of different cross validation methods on classifier evaluation. I've put in a small range for relevant parameters for each of the methods considered.

I've included my code for you to experiment with, you can try more cross-validation methods. Iris data set has small set of instances and there isn't too much 'craziness' in the distribution of the classes. If you tried it on some real-world data with a lot of bias and noise, you might see a more pronounced effect of cross-validation (use error-bars).

--

    """
Evaluate cross-validation schemes for Iris dataset
"""

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn import svm
from sklearn import datasets
from sklearn.model_selection import train_test_split
from time import time
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import RepeatedStratifiedKFold

iris = datasets.load_iris()
X, y = iris.data, iris.target

""" visually inspect iris dataset """
iris_sns = sns.load_dataset('iris')
g = sns.pairplot(iris_sns, hue='species', markers=["o", "s", "D"])
plt.savefig(filename='iris_pairplot.png')
""" Dimensions of data matrix and target labels """
print(X.shape, y.shape)

"""
cross-validation: the apparent silver bullet to over-fitting
The available data and label matrices are split into set of training and validation matrices
We first begin with mere evaluation of classifier trained on different random sets
of training and validation data sampled from the entire available data.
By averaging the computed classification performance on each training subset,
we will acquire a statistically reliable estimate of how this classifier might perform on unseen test data.
Subsequently, we will train classifier on multiple subsets of training data to actually improve the
learnt classifier itself.
"""

""" The effect of train-test split sizes"""
test_sizes = [0.1 * i for i in range(1, 10)]
scores = []
for test_size in test_sizes:
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_size, random_state=round(time()))
    print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)
    clf = svm.SVC(kernel='linear', C=0.1).fit(X_train, y_train)
    scores.append(clf.score(X_test, y_test))

fig1 = plt.figure(1)
plt.plot(test_sizes, scores, 'k-d')
plt.title('Iris dataset, effect of proportion of train-test splits')
plt.xlabel('proportion of test-images')
plt.ylabel('classification score')
plt.savefig('train-test-splits.png')
""" The effect of number of cross-validations """
cvs = [i for i in range(2, 11)]
cv_scores_means = []
cv_scores_std = []
for cv in cvs:
    clf = svm.SVC(kernel='linear', C=0.1, random_state=round(time()))
    scores = cross_val_score(clf, X, y, cv=cv)
    cv_scores_means.append(scores.mean())
    cv_scores_std.append(scores.std())

fig2 = plt.figure(2)
plt.errorbar(cvs, cv_scores_means, yerr=cv_scores_std, fmt='k-o')
plt.title('Effect of number of cross-validations')
plt.xlabel('number of cross-validation')
plt.ylabel('mean classification score')
plt.savefig('number-of-cv.png')
""" The effect of k-stratified splits"""
num_splits = [i for i in range(2,11)]
scores_splits = []
for n_splits in num_splits:
    skf = StratifiedKFold(n_splits=n_splits)
    clf = svm.SVC(kernel='linear', C=0.1)
    scores = []
    for train, test in skf.split(X, y):
        clf.fit(X[train], y[train])
        scores.append(clf.score(X[test], y[test]))
    scores_splits.append(np.average(scores))

fig3 = plt.figure(3)
plt.plot(num_splits, scores_splits, 'k-s')
plt.title('Effect of number of stratified splits')
plt.xlabel('number of splits')
plt.ylabel('mean classification score')
plt.savefig('num-stratified-splits.png')
""" the effect of k-label splits"""
num_folds = [i for i in range(2, 11)]
num_repeats = [i for i in range(2, 10, 2)]
line_colors = ['r', 'g', 'b', 'k', 'y']
scores_repeats = []
for n_repeat in num_repeats:
    scores_folds = []
    for n_fold in num_folds:
        rskf = RepeatedStratifiedKFold(n_splits=n_fold, n_repeats=n_repeat, random_state=round(time()))
        clf = svm.SVC(kernel='linear', C=0.1)
        scores = []
        for train, test in rskf.split(X, y):
            clf.fit(X[train], y[train])
            scores.append(clf.score(X[test], y[test]))
        scores_folds.append(np.average(scores))
    scores_repeats.append(scores_folds)
fig4 = plt.figure(4)
handles = []
for i in range(len(num_repeats)):
    h, = plt.plot(num_folds, scores_repeats[i], line_colors[i]+'-x', label="#rep: "+str(num_repeats[i]))
    handles.append(h)
plt.title('Effect of number of splits and repeats for repeated stratified k fold')
plt.xlabel('number of splits')
plt.ylabel('mean classification score')
plt.legend(handles=[h for h in handles])
plt.savefig('num-split-repeat-stratified.png')
plt.show()


Can a classifier be trained with reinforcement learning without access to single classification results? Question: Can a classifier be trained with reinforcement learning without access to single classification results?

I want to train a classifier using reinforcement learning. However, there is one big restriction: the program does not have access to the score regularly, not even after every classification. Only after many classifications were completed (e.g. around 40-200 classifications, let's call them a batch) the final score of that batch is available. One batch can be executed rather quickly: it takes just around one second. Therefore, thousands of batches can be executed, each of them returning a score for its classifications. Every time a batch is executed, the current ML Model is given as input for the batch to use.

Other than that, of course, the feature vector is known (contains around 60 features) and the labels are known (around 6 labels).

I have never applied Reinforcement Learning before, therefore, I can not tell whether this can work. In theory, I think, it should: all data is available. The algorithm can choose some parameter values for the model, try them out, and get a score. Then try out different values and get the score again. This way it should be able to improve step by step.

Additional Notes: Although the text above should be enough to understand the problem and provide an answer (which can be general and not specific to a concrete use case), my personal use case and details about it are explained here. This might be useful to understand the problem in more detail.

Edit: Before, I used "Random Forest" as an example of a possible ML model that one could use. As a Random Forest seems to require supervised learning contrary to reinforcement learning, I have removed it from the text (not counting some special use cases, such as this one).

If every data point has a ground-truth label (i.e., one of the six labels), than any supervised learning technique can work, including Random Forest. If the labels come in batches, then the parameters of the model can be updated with each new batch. Either completely retrain the model with data up to the current time point or incrementally update the model parameters with the new data.

There is no reason to apply reinforcement learning unless it is necessary.

If every data point does not have a ground-truth label, then reinforcement learning might be appropriate. Then you can not use Random Forest or any other supervised learning technique. You'll need to pick a Reinforcement Learning technique that can handle sparsely labeled data.


Evaluating the result of topic modeling in a way that time matters I have run different topic modeling approach on my data(its clinical data related to Cognitive impairment diseases. we are going to process what thing is important that make it develop to more harsh disease).
before anything, I have divided my data into different 6-month data(from a starting point back every 6 months) and then run the topic modeling approach on every 6 months. I was going to see the difference between the derived topics of each 6 months.

For example, for the first six months there are 20 topics, then for the second six month there 20 topics and... till the tenth (5 years).
I was hopeful to see a different topic because of the use case I have in every six months or at least each 1 year. but sadly most of the words have been repeated in every 6 months. however, the number of the words has changed.

For example in the first six months word "sleeping" has been repeated 10 times in different topics but in the second 6 months, it has been repeated 4 times.

What I am going to say is that, if we look at this as a thing that times matters, I can not see any pattern visibly in my data unless I rely on the number of words changing in every six months.

Do you think analyzing my output and plotting the different words number in different 6 months makes sense at all? or its something unreliable.

Also, do you mind to let me know what other approach is there that I can apply to get insight out of the output of my topic modeling(please consider that the changing in each six months matters)?

I think the issues comes from the fact that the item you are looking at (the word "sleeping") is a rare event, so the probability that you observe one is about 0. Technically, it is called a Poisson process.

One of the way to circumvent it is what you did: aggregate on a period (in your case 6 month) so that the number of events get significant.

You do not need to cut you observation time in 6 month period though, you may use moving averages: period 1-6, 2-7, 3-8, 4-9,... If there is a temporal pattern it will be more visible.

An other way is to use cumulative data: the number of words up-to time t. If you get a logistic "S" shaped curve, then you are on an important thing.

Basically you want to see if a certain word (sleeping) is more frequent in the CI sub-population than the nonCI sub-population. You can use the t-test (unpaired observations, unequal sample size, equal variance), to check if the count of the word is significantly different in the two sub-populations.
And you can do this over time.

More adapted tests can be found in this paper treating of the similar problem of testing if documents have the same author. Here again, nothing prevent to make a test per time period, cumulative or not.


Updating weights in Adaboost I'm studying the Adaboost algorithm. This algorithm updates the weight after training.

This is the table when they explain about weight on Adaboost

I'm confused about what this "weight" means. Is it:

weight for each node?
weight for each model?
table weight column (more confusing)?

weight for each node?  $w_i$

weight for each model? $\alpha_i$

table weight column (more confusing)? The initial weight is set as $\frac{1}{M}$ where M is the number of data points. I assume the data2_1 is the update of data1_1 after being updated by a new weight for feature1 but it's a bit obscure due to lack of description about the title, etc. of the table.

weight for each node is being updated after a classifier is trained on the data and it's weight ($\alpha_i$) and the score is calculated.

$T(x_i)$ is the final classifier but seemingly is lacks a sign before the expression. which is the sum of multiplication of each classifier with it's weight $\alpha_i$


how to programmatically introduce grammatical errors in sentences I've a set of sentences in English language. I'm exploring ways to create a dataset of sentences with grammatical errors programmatically. The following options has been tried out randomly -

identify verbs, propositions etc. by POS tagging and change the tense or remove them
change the order of 2 or more words
remove commas, colons, semi-colons etc.

These are not always fool-proof. Are there any proven ways to approach this problem?

Generating artificial errors is generally risky in NLP, because it's difficult to make sure that the type and distribution of errors correspond exactly to real human errors. If the artificial errors diverge from real errors and a model is trained based on this data, the model will appear to have very good performance since it will rely on the patterns used to generate the errors. However it might not perform well with real data, and it would be difficult to detect it.
That being said, it's been a problem which has been studied for quite a while so the state of the art should help: Google Scholar gives a lot of references, probably some of these papers provide existing implementations as well. One may notice that the concerns I mentioned above are a recurrent question, with some recent papers analyzing how much artificial errors actually help.


How can I use Data Science to profoundly contribute to Humanity? Undergraduate researcher here. I worked at many traditional scientific research labs, ranging from cancer biology, to radiation medicine, and to supercapacitors. I'm thinking of switching to Statistics and Computer Science from Computational Biology to join the exciting field of Data Science.

What ways can I contribute to Humanity without a technical background in basic science? I don't know how I will feel about making better clickbaits. I've thought about continuing in scientific research but I will lack the science background. I've thought about working for the United Nations. What other ways can I contribute? Inspire me!

Since I have already answered a similar question on Data Science StackExchange site, plus some related ones, I will mention all of them here and let you decide, if you find them helpful:

on data science project ideas, including those with social impact;
on working as a data scientist for a non-profit company;
on using data science in political campaigns.


music recommender system i want to build a music recommender system. i have user_id,song,play_counts triplets as my data.I want to do it with collaborative filtering in with i will have USERS row and SONGS columns.It is very similar to movie recommendation but instead of rating,i now have play counts.So, cn i use play count as my matrix values...i will be then using linear regression to predict features of both users and songs simultaneously.
Will it work with play counts to predict music?
Please answer ...i am confused!

Short answer

Yes, you can use the counts as the target you are trying to predict. I would say that you should probably normalize this, so that your target is between 0 and 1. This will help collaborative filtering converge faster.

Long answer

Given the input data you describe, you have to remember the concept of "Garbage in, garbage out"

Now I am not saying that your input is garbage, far from that. What I mean is that if all you know is how many times somebody played a song, this is also all you will be able to learn from and eventually predict.

In your case, it's more of a matter of interpreting the output. Let's imagine that your model works extremely well and is able to predict how many times someone will listen to a song. In that case, it would indeed make sense to recommend them songs that they will indeed listen to a lot because the assumption is that you will listen to songs more often if you like them. But of course, you have to remember that this might not be entirely true


Goodness of fit metric to compare topic models NMF vs LDA Does anyone have a good idea for how to compare topic modeling done by NMF and LDA? Let's say I fit LDA to a dataset and generate topic-word and document-topic distributions--I can use perplexity, for instance, to measure the goodness of fit of the model. If I fit NMF to the same dataset and generate topic-word and document-topic distributions, I can take the resultant fitted data as $X^{Est} = HW^T$ and use $\|X^{Original} - X^{Est}\|_{Fro}$ as a goodness of fit of the model.

What is an objective metric that I can use to compare the goodness of fits to each other? Can either of these metrics be used for the other model?

NMF and LDA models produce topic-word and document-topic distributions, so you can compare these models on evaluation tasks of topic coherence (i.e. evaluating topic-word distributions), document clustering/classification (i.e. evaluating document-topic distributions) or information retrieval (i.e. evaluating topic-word and document-topic distributions together).

See some examples in "Improving topic models with latent feature word representations", "LDA-Based Document Models for Ad-hoc Retrieval".


How to handle unseen labels in test data? I get something like TF-IDF of training corpus in python with (something like) TfidfVectorizer. In test data some features (here are the words of test corpus, every word is a feature) are not seen in the training data and because of this, the shape of the test and the train matrix are not equal and the program gives an error (number of columns isn't same and some words in the test data are not seen in the train data).
How should I solve this problem? How should I handle unseen features in test set?

It depends on the problem.There is no single answer to it.
Things you can do:

Delete these features and focus on features that appear in both train and test set.
Semi-supervised learning.
You can also turn these unusual cases into a new unique category.
Find nearest neighbour and use that value for that feature.


Finding similar articles in realtime I want to build a large document (news article) searchable database, such as when adding a new article I will be able to quickly find X most similar articles from it.
What is the right tech/algorithm/Python framework to approach this?

Elasticsearch is the right tool to use if you don't want to code this yourself. Indeed, you need an indexing algorithm that is able to efficiently retrieve pieces of texts in a big database, and SQL isn't particularly good at it. Moreover, Elasticsearch is quite user friendly, so it won't be an overkill to actually install it and use it. You might discover in the process that finding most similar articles isn't that easy and that Elasticsearch is of a great.

Here is the documentation for a Python client:


Whether solution of Kernel ridge regression through inverse calculation yields global optimal solution or not? As Kernel Ridge Regression computes inverse (Not a pseudo inverse) of a square matrix, so whether obtained solution would always be global optimal or not?

In the case that the inverse exists, it is the same as the pseudo inverse. Unless the method crashes due to the fact that the inverse does not exist, the results will be the same.


How to get periodicity from timeseries data? I would like to create a recommendation system for a smart home application. I gather the data in a time-series database. The app monitors the on/off state of a smart lamp and can create daily routines. I would like to create notifications to the user like "It looks like you are usually use your lamp from 8:00 to 10:00. Would you like to create a routine for that?".
I am beginner to data science and machine learning and I would like to ask what kind of machine learning algorithm or method should I use for this problem. Someone recommended me Facebook Prophet, but as I can see that library can only predict future occurences not the periodicity of time-series data. Do you know any library or algorithm?
In the image below you can see my problem detailed. There is hourly on/off state of the lamp daily in green and in blue the result that I would like to get: the periods that the lamp is usually on.

Looking at your data - the easiest way is to create a Last-N Days hourly average of the binary indicator - and then use a threshold (based on experimentation) to binarize it.
e.g. if your Last 10 Day hourly average looks like this:
0, 0, 0.6, 0.8, 0.9, 1, 0.9, 0.7, 0, 1, 1, 1, 0
Then, a threshold of 0.8 to binarize would result in the following:
0, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0
and - then you could use the time slices associated with the 1's for the recommendation.
This might be the simplest thing you could do - There could however be more sophisticated algorithms out there.


Good "frequent sequence mining" packages in Python? Has anyone used (and liked) any good "frequent sequence mining" packages in Python other than the FPM in MLLib? I am looking for a stable package, preferable stilled maintained by people. Thank you!

I am actively maintaining an efficient implementation of both PrefixSpan and BIDE in Python 3, supporting mining both frequent and top-k (closed) sequential patterns.


LSTM: Converting to Bayesian Deep Neural Network Starting from Yarin Gal's research paper on using Dropout as a Bayesian Approximation (https://arxiv.org/pdf/1506.02142.pdf), I am trying to apply this concept to my Sequence Prediction model. My model comprises of 2 LSTM layers, followed by a relu dense layer, followed by a softmax layer. A dropout layer is added after each of the layers.

I've found some implementations for measuring the uncertainty of a deep neural network (like this one here: https://fairyonice.github.io/Measure-the-uncertainty-in-deep-learning-models-using-dropout.html), but all of them seem to be applicable to dense layers rather than LSTM layers.

So in summary, what I am asking about is:

How can I turn on Dropout during testing for LSTM layers?
How can I measure my model's uncertainty?

Firstly, I suggest looking at the following paper:

  Gal and Ghahramani, A Theoretically Grounded Application of Dropout in
  Recurrent Neural Networks, 2016

which discusses how to appropriately apply dropout as an approximate variational Bayesian model. The key is to use the same dropout mask at each timestep, rather than IID Bernoulli noise.

As for measuring model uncertainty, note that while dropout gives us an approximate variational Bayesian neural network, it does not give access to the variational posterior density, and so we cannot compute e.g. the entropy of the posterior distribution. But, we can trivially compute the predictive variance, which is closely related to model uncertainty. Simply run $n$ forward passes on an input to get outputs $Y_F = (y_1,\ldots,y_n)$. Then compute your measure (e.g., the covariance matrix in the regression case) on $Y_F$. For classification as your case seemingly is, you can use predictive entropy, for instance. For more measures see

  Gal et al, Deep Bayesian Active Learning with Image Data, 2017


Keras : switch backend in Notebook If a script is launched from command line, I use "KERAS_BACKEND" env var to switch between Theano and  Tensoflow. What can be done to switch backend if script is running in a notebook ?

You can set the environment variable via python's built in os module.

import os

os.environ["KERAS_BACKEND"] = "tf"   # or theano

Making changes in one notebook does not seem to carry over to other notebooks that are started in the same Jupyter session (run from a single terminal).

Here I try to get the environment variable in notebook 2, and we see it doesn't exist:

Create environment variable in Notebook 1:

Now make the changes in notebook 1 (after also confirming that the environment variable didn't exist - we catch the exception)

And now go back to notebook 2 to see if it picks up on the changed value:

There are also settings that should be read by Keras on import. Have a look at the documentation. If you are on windows, there are some separate steps you should check out there.


Classify the coordinate data I'd like to classify the data on coordinate.

Here are 2 example data.

data1 = [(1,1), (2,2), (3, 3), (4, 2), (5, 3), (6, 0)]

data2 = [(1,1), (2,2), (3, 10), (4, 9), (5, 10), (6, 0)]

The bold part have the same wave in above data.

The length are all different in my data set.

Is there any way that I can find the similar wave in many data like this?

Thank you.

Calcualte distance?

Distance = data2 - data1 = [(1,0), (2,0), (3, 7), (4, 7), (5, 7), (6, 0)]

And what will you treat as the same wave?

Any three (or more) nearby points that have the same distance...


How to choose a model for this cross-validation curve? I'm using GridSearchCV to tune hyperparameters for a Logistic Regression multiclass model.

I read on Kaggle that you should choose the hyperparameter that results in the lowest discrepancy between the CV-score and the training score, but in this case this leads to a very low score.

How should I choose the proper C value to ensure generalisability of the model but also high model performance based on the CV-curve below?

From my understanding opting for low discrepency between the two scores ensures the ability of the model to be generalised to unseen data. But on the other hand I want a score as high as possible on unseen data.

Thanks for any help!

Choosing the best validation accuracy is the common practice, since validation is unseen data.

Sometimes you might have over-fitting to the validation set, mainly if it is too small or no very representative of the data (for example if it has considerably more examples of one class, thus a good model would be a model that says that (almost) everything belongs to that class).

If you are worried about over-fitting, you could increase your regularization strength.


Where does the name 'LSTM' come from? Long short-term memory is a recurrent neural network architecture introduced in the paper Long short-term memory.

Can you please tell me where the name comes from?

("Memory", as the network can store information because of the recurrence - but where does the "Long short-term" come from?)

In Sepp Hochreiter's original paper on the LSTM   where he introduces the algorithm and method to the scientific community, he explains that the long term memory refers to the learned weights and the short term memory refers to the gated cell state values that change with each step through time t.

edit: quote from paper "Recurrent networks can in principle use their feedback connections to store representations of recent input events in the form of activations ("short-term memory", as opposed to "long-term memory embodied by slowly changing weights)"


Hybrid Convolutional and Conventional Neural Networks Convolutional Neural Networks are usually the best choice for classification and semantic segmentation of images. categorical/numerical data (age, height, city, etc.) on the other hand are best processed by conventional machine learning models, such as (deep) Random Forests, Support Vector Machines or Conventional Neural Networks.

Are there hybrid architectures combining both convolutional and conventional Neural Networks for classification of datasets consisting of images and categorical data? I am sure this problem has been solved before and I am specifically looking for research papers, tutorials and implementations in one of the common libraries (PyTorch, Tensorflow, Keras, etc.).

A good example for such a "hybrid" dataset is the ISIC dataset, containing thousands of images of skin growths together with information such as the age and the sex of the patient for each image. Hypothetically, a hybrid model that includes this categorical/numerical information could be more succesful in detecting skin cancer, than a model which just uses the images.

If I had to do it, I would use a transfer learning strategy:

I would train a deep learning model just with the images to solve a classification problem. In order to do this, I would need of course to have tags for the image classes. If this tags do not exist, then this approach does not make sense. It is relatively easy to make very accurate image classification, as there are lots of tools. Of course, I would do that using a CNN.

Once I have my CNN trained, I would take the last fully connected layer out. I would concatenate the inputs of this layer with the cateogrical/numerical data that you have, and obtain a feature vector using this concatenation. I would input this feature vector to another ML algorithm to carry out the objective that you need.

Did I just make this up?

No. A similar thing is done in fast ai deep learning course for text. An RNN is trained to predict the next word of a sentence and then the weights of that RNN are used to build a sentiment analysis machine. I think the idea is very similar and this is why I call it transfer learning.

What if I don't have tags for the images?

Then my advise is to use a CNN trained on image-net, take the last fully connected layer out and concatenate the inputs to that layer with the other features.


Choosing best model produced from different algorithms. Metric produced by cross-validation on the train set or metric produced on the test set? I know that choosing between models produced by one algorithm with different hyperparameters the metric for choosing the best one should be the cross-validation on train set.
But what about choosing between models that are produced by different algorithm? By which metric should I compare them, the metric produced by cross-validation on the train set or metric produced on the test set? Also why?

Your main question seems to be what about different algorithms?, and the answer to that is it makes no difference.
While hard to argue with the "test set" answer logic it is also easy to find a counter-example. If your training set has 100K examples, and you used, say 5-fold CV, and then you also have a test set with only 200 examples... the test set is 100x smaller and likely to be much less reliable.
Normally you would use cross-validation, or a test set. And if your training data is large, I would generally recommend just using a test set, because you probably have enough data and because CV takes longer to train. When training data is scarce, cross-validation makes more sense: training time probably matters less, you want to train the final model on everything you have, and also you will value the N different opinions that N-fold CV gives you, because your sample size is small.
If you have used both cross-validation and a test set for multiple models (whether same algorithm or different algorithms), then you should expect them both to agree on your choice of best model. If model A does better than model B on cross-validation, but model B does better on the test data, then you should go looking for a problem.
If that does happen, I'd first be suspicious that the test data is not representative. E.g. to exaggerate, your training data covers the period 1950 to 1970, and your test data is for the year 2020. At the other extreme, I'd check that the test data is not actually in your training data (implying that model B does a better job of memorizing).


What to do if training loss decreases but validation loss does not decrease? I am training a LSTM model to do question answering, i.e. so given an explanation/context and a question, it is supposed to predict the correct answer out of 4 options.

My model architecture is as follows (if not relevant please ignore): I pass the explanation (encoded) and question each through the same lstm to get a vector representation of the explanation/question and add these representations together to get a combined representation for the explanation and question. I then pass the answers through an LSTM to get a representation (50 units) of the same length for answers. In one example, I use 2 answers, one correct answer and one wrong answer. From this I calculate 2 cosine similarities, one for the correct answer and one for the wrong answer, and define my loss to be a hinge loss, i.e. I try to maximize the difference between the cosine similarities for the correct and wrong answers, correct answer representation should have a high similarity with the question/explanation representation while wrong answer should have a low similarity, and minimize this loss.

The problem I find is that the models, for various hyperparameters I try (e.g. number of hidden units, LSTM or GRU) the training loss decreases, but the validation loss stays quite high (I use dropout, the rate I use is 0.5), e.g.

My dataset contains about 1000+ examples. Any advice on what to do, or what is wrong?

Welcome to DataScience. This looks like a typical of scenario of overfitting: in this case your RNN is memorizing the correct answers, instead of understanding the semantics and the logic to choose the correct answers.
A typical trick to verify that is to manually mutate some labels. For instance, you can generate a fake dataset by using the same documents (or explanations you your word) and questions, but for half of the questions, label a wrong answer as correct. If you re-train your RNN on this fake dataset and achieve similar performance as on the real dataset, then we can say that your RNN is memorizing.
Note that it is not uncommon that when training a RNN, reducing model complexity (by hidden_size, number of layers or word embedding dimension) does not improve overfitting.
If it is indeed memorizing, the best practice is to collect a larger dataset. I understand that it might not be feasible, but very often data size is the key to success. If you haven't done so, you may consider to work with some benchmark dataset like SQuAD
or bAbI.


K-fold cross validation when using fit_generator and flow_from_directory() in Keras I am using flow_from_directory() and fit_generator in my deep learning model, and I want to use cross validation method to train the CNN model.

datagen = ImageDataGenerator(rotation_range=15,width_shift_range=0.2,
                             height_shift_range=0.2,shear_range=0.2,
                             zoom_range=0.2,horizontal_flip=True,
                             fill_mode='nearest')

image_size = (224, 224)
batch = 32

train_generator = datagen.flow_from_directory(train_data,
                                              target_size=image_size,
                                              batch_size=batch,
                                              classes= classes_array)

I found this Youtube video and this Tutorial, But it is not use flow_from_directory().

Do you have any idea how do I use k-fold cross validation when using fit_generator and flow_from_directory() in Keras?

So, I haven't found any solution regarding this application of cross-validation in fit_generator(), I hope it comes in one update of the Keras package, since cross-validation is an important part of training models.

What I have done so far, basically I split the dataset first then I pass the data and labels to the fit_generator. At the end of each step I save the model, at the beggining of each step I load the preciuos model to continue the training over the same model, but with a different k-fold of the dataset. I'm pasting the part of my code where I use this approach, hope it helps.

## Training with K-fold cross validation
kf = KFold(n_splits=k_folds, random_state=None, shuffle=True)
kf.get_n_splits(images_file_paths)

X = np.array(images_file_paths)
y = np.array(class_labels)

i = 1
for train_index, test_index in kf.split(X):
    trainData = X[train_index]
    testData = X[test_index]
    trainLabels = y[train_index]
    testLabels = y[test_index]

    print("=========================================")
    print("====== K Fold Validation step => %d/%d =======" % (i,k_folds))
    print("=========================================")

    trainGenerator = Generator(trainData,trainLabels,batchSize=batchSize,imageSize=imageSize,augment=True,grayMode=grayMode)
    valGenerator = Generator(testData,testLabels,batchSize=batchSize,imageSize=imageSize,augment=False,grayMode=grayMode)

    try:
        model = load_model(weights_path, compile=True)

    except Exception as OSError:
        pass

    model.fit_generator(
            trainGenerator,
            steps_per_epoch=len(trainData),
            epochs=epochs,
            validation_data=valGenerator,
            validation_steps=len(testData))
    i+=1


Doc2Vec - How to label the paragraphs (gensim) I am wondering how to label (tag) sentences / paragraphs / documents with doc2vec in gensim - from a practical standpoint.

Do you need to have each sentence / paragraph / document with its own unique label (e.g. "Sent_123")? This seems useful if you want to say "what words or sentences are most similar to a single specific sentence labeled "Sent_123".

Can you have the labels be repeated based on content? For example if each sentence / paragraph / document is about a certain product item (and there are multiple sentence / paragraph / document for a given product item) can you label the sentences based on the item and then compute the similarity between a word or a sentence and this label (which I guess would be like an average of all those sentences that had to do with the product item)?

Both are possible. You can give every document a unique ID (such as a sequential serial number) as a doctag, or a shared string doctag representing something else about it, or both at the same time.

The TaggedDocument constructor takes a list of tags. (If you happen to limit yourself to to plain ints ascending from 0, the Doc2Vec model will use those as direct indexes into its backing array, and you'll save a lot of memory that would otherwise be devoted to a string -> index lookup, which could be important for large datasets. But you can use string doctags or even a mixture of int and string doctags.)

You'll have to experiment with what works best for your needs.

For some classification tasks, an approach that's sometimes worked better than I would have expected is skipping per-text IDs entirely, and just training the Doc2Vec model with known-class examples, with the desired classes as the doctags. You then get 'doc vectors' just for the class doctags – not every document – a potentially much smaller model. Later inferring vectors for new texts results in vectors meaningfully close to related class doc vectors.


Backpropogation through Maxpool and relu Why is backpropagation through maxpool and relu needed?

Purpose of backpropagation is to update weights while on the other hand maxpool and relu only perform a simple operation on the input. They don't really have any weights or any filters. So why is there a need to backpropagate through them?

Why is backpropagation through maxpool and relu needed?

Any differentiable function (max pool and relu) through which inputs pass will have a gradient.. The nonlinear functions are still functions, so the chain rule still applies here...and hence we have the Gradients flowing through them..This gradient will have to be backpropagated...


Should I repeat the testing/training split while repeating K-fold cross validation? I'm fitting random forest regressions on my data, and using 10 K-fold cross-validation to evaluate model performance. While re-runing the cross-validation, I noticed that the results differed between each run, sometimes by a lot. So, I decided to repeat the cross-validation calculation 20 times, creating a for-loop, and then summarising the results afterwards. Just to illustrate, I'm doing something like this:
for (i in 1:20) {

trainIndex <- createDataPartition(data$response, p = .8, list = FALSE, times = 1)
data_train <- data[ trainIndex,]
data_test  <- data[-trainIndex,]

train.control <- trainControl(method = "cv",
                              number = 10)

model <- train(formula, data = data_train, method = "rf",
               trControl = train.control)

pred <- predict(model, data_test)
t<- postResample(pred = pred, obs = data_test$estimate)
t <- t[[2]]

result[[i]] <- t }

So, essentially, each re-run I am splitting my dataset into training and testing sets again. This results in a lot of variance, when ploting the resuls of each run together they vary from ~0.2 to 0.6 R.squared. If I don't do this within the loop (i.e. if I split into training/testing before the loop), then the results of the 20 runs are very similar.
Which way is the right way to go about this?

The purpose of k-fold cross validation is to fit the model multiple times and average out to results to estimate the model's predictive performance.
If having done 10-fold CV once and got one result and then repeating it again and getting a significantly different result, probably means the number of folds is too low, and the k-fold process isn't capturing the inherent variability in the data sufficiently well.
A better test is to change the number of folds and see what happens to the average and variance of the combined output.


Regression vs Random Forest - Combination of features I had a discussion with a friend and we were talking about the advantages of random forest over linear regression.

At some point, my friend said that one of the advantages of the random forest over the linear regression is that it takes automatically into account the combination of features.

By this he meant that if I have a model with

Y as a target
X, W, Z as the predictors

then the random forests tests also the combinations of the features (e.g. X+W) whereas in linear regression you have to build these manually and insert them at the model.

I am quite confused, is this true?

Also if it true then is it about any kind of combination of features (e.g. X*W, X+W+Z etc) or only for some specific ones (e.g. X+W)?

I think it is true. Tree based algorithms especially the ones with multiple trees has the capability of capturing different feature interactions. Please see  this article from xgboost official documentation and this discussion. You can say it's a perk of being a non parametric model (trees are non parametric and linear regression is not). I hope this will shed some light on this thought.


Algorithms that would benefit from variable transformations? 1- Which algorithms would benefit from data that has been transformed, so that distributions of continuous variables resemble that of a normal distribution ?

2- What would be the benefits of transforming variables in such a way ?

It's not necessary for all features, but most algorithms would benefit from making highly-skewed or heavy-tailed features more Gaussian-like.
Transforming to Gaussian makes the data symmetric and removes both heavy and long tails. That does a similar job of normalizing to $[0,1]$, without aggressively smashing the data into a bounded interval.

Side note: Transforming to standard Gaussian means that the center scale of every feature is the same. This is very helpful in making sure that your model learns "fairly", rather than putting undue influence on the feature with greater variance. See here for an example of how un-standardized data can harm K-means results. See also the comments in the other answer about regularization.


Is there a way to classify an alphanumeric string? I have data containing various items. Each item has a unique alphanumeric code associated with it (see the example below).
Is there a way to predict the item type based on the alphanumeric code?
Data:
Item          code          type
1       4S2BDANC5L3247151   book
2       1N4AL3AP1JC236284   book
3       3R4BTTGC3L3237430   book
4       KNMAT2MT1KP546287   book
5       97806773062273208   pen
6       07356196706378892   Pen
7       97807345361169253   pen
8       01008130715194136   chair
9       01076305063010CCE44 chair
etc

Find a minimal implementation of (binary) text classification based on n-grams below. It's written in base R. You first need to extract the ngrams and apply some model (Lasso in my case) to predict each ngram's class. You can easily expand this to multiclass by changing the model accordingly. Note that this is just a minimal example with room to improve.
See also here: https://github.com/Bixi81/R-ml/blob/master/NLP_ngram_short_text.R
Find a Python implementation of GLMnet here: https://web.stanford.edu/~hastie/glmnet_python/
Data:
# Dummy data
# Note the little differences in the strings, however there is a clear pattern
df = data.frame(text=c("ab13ab12ab16","ag16ag16fg16","ab12ab12ab12","fg12fg16fg16","ab16ab12af12","fg16fg16fg16"),target=c(1,0,1,0,1,0))

head(df)

          text target
1 ab13ab12ab16      1
2 ag16ag16fg16      0
3 ab12ab12ab12      1
4 fg12fg16fg16      0
5 ab16ab12af12      1
6 fg16fg16fg16      0

Ngrams:
# Set up lists to post results from loop
ngrams = list()
targets = list()
observation = list()

# Loop over rows of DF
counter = 1
for (row in 1:nrow(df)){
  # Loop over strings in first colum per row
  for (s in 1:nchar(as.character(df$text[[row]]))){
    # Get "ngram" (sequence of two letters/digits)
    substring=substring(as.character(df$text[[row]]), s, s+1)
    # Append if >1, also post row and target
    if (nchar(substring)>1){
      ngrams[[counter]]<-substring
      targets[[counter]]<-df$target[[row]]
      observation[[counter]]<-row
      counter = counter+1
    }
  }
}

# Lists to DFs
ngramdf=data.frame(ngram=matrix(unlist(ngrams), nrow=length(ngrams), byrow=T))
targets=data.frame(ngram=matrix(unlist(targets), nrow=length(targets), byrow=T))
obs=data.frame(obs=matrix(unlist(observation), nrow=length(observation), byrow=T))

# Get dummy encoding ("one hot") from all the ngrams
dummies = model.matrix(~ . -1 , data=ngramdf)

# Bind target and dummies to train DF
train = cbind(targets,dummies)

Estimate some model:
# Now apply Lasso to predict the classes
# https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html
library(glmnet)
cvfit = cv.glmnet(as.matrix(train[,-1]), as.matrix(train[,1]), family = "binomial", type.measure = "class")

# Predict class per ngram
classes = predict(cvfit, newx = as.matrix(train[,-1]), s = "lambda.min", type = "class")

# predicted probability per ngram -> needs (type="response")
probs = predict(cvfit, newx = as.matrix(train[,-1]), s = "lambda.min", type = "response")

# Calculate average prob that a sequence (so ngrams per row) belongs to some class
result = cbind(probs,obs)
# Get mean per row
result = aggregate(. ~ result$obs, result[1], mean)
# Bind original text and target
result = cbind(result,df)
colnames(result)<-c("observation","estimated_prob", "text", "original_target")

Result:
  observation estimated_prob         text original_target
1           1      0.7173567 ab13ab12ab16               1
2           2      0.3020248 ag16ag16fg16               0
3           3      0.7674571 ab12ab12ab12               1
4           4      0.2922798 fg12fg16fg16               0
5           5      0.6815783 ab16ab12af12               1
6           6      0.2393033 fg16fg16fg16               0


Counting the occurrence of each string in a pandas dataframe column I'm working with a data set of movies which has various info on them. One of the columns contains the various genres a movie may belong to like so:

What I would like to do is count how often a genre occurs in each column, in above example a corresponding series would look like (created the series myself):

How can I extract this information from the original dataframe using pandas?

Something Equivalent to this should work:

the function(pandas.Series.str.count) in this link will even make it easier...

You can use split by regex :

#using your sample
df[['class1', 'class2', 'class3',..]] = df['genre'].str.split('|', expand=True)

Alternative to the above method (but iterating the dataframe)

  l = list(df[index-key].values)
  l = (",".join(l))
  l.split('|')

And if need remove column genre add drop:

df = df.drop('genre', axis=1)

And then you can use value_counts()

But this assumes that you have same length of each genre or apply a check first and proceed accordingly..

Or

You can also try using Counter from the collections module(after splitting them, just update each key's value)

example Counter dict

>>> from collections import Counter
>>> Counter(['apple','red','apple','red','red','pear'])
Counter({'red': 3, 'apple': 2, 'pear': 1})


How can there be more true positive than positive? Currently reading  Learning from Little: Comparison of Classifiers Given Little Training

In 3 Experiment Results, the following graph is shared:

The experiment is described as follow

  We begin by examining an example set of results for the average TP10
  performance over all benchmark tasks, where the training set has P=5
  positives and N=200 negatives.We vary the number of features selected
  along the logarithmic x-axis.

I understand this as "we use a training set of 205 elements, 5 being positives and the 200 remaining being negatives".
But looking at the results, Naive Bayes using Information Gain with a few features, and Multinomial Naive Bayes using Bi-normal separation with several hundred features  both end up with 6.5 true positives in the top 10 (The TP10 metric is the number of true positives found in the 10 test cases that are predicted most strongly by the classifier to be positive).

I would have assumed that the models being closest to 5/10 would be the most accurate, but reading their results, it looks like the higher the better.
Thus, it feels like I overlooked and misunderstood something. Can somebody enlighten me on this issue ?

Thank you

Of course it is impossible to have a higher than $100\%$ true positive rate in your results.

The author does explain how he arrives at his results, but I must admit it is not at all obvious. In the subsection Performance Metrics of section 2 Experiment Protocol the author explains that the metrics are calculated as

  The TP10 metric is the number of true positives found in the 10 test
  cases that are predicted most strongly by the classifier to be
  positive.

This mean that the TP10 is only calculated using the testing set. The author also explains in the subsection Experiment Procedure of section 2 Experiment Protocol states

  The condition on the second loop ensures that there are 40 positives
  available for training plus at least 10 others in the test set
  (likewise, 200 negatives for training and at least 50 for testing)

So we see that there is actually a minimum of 10 testing instances with which the TP10 is computed.


Calculate Similarity using User's Personal Data? I want to find out which users are similar to each other using their personal/organisational data, such as department, company, site, etc.

I have this data in a boolean format, as shown below:

       Dep1 Dep2 Comp1 Comp2 Site1 Site2
    U1  0    1     0     1     0     1
    U2  1    0     0     1     1     0
    U3  1    0     1     0     1     0
    U4  0    1     0     1     0     1
    U5  0    1     0     1     1     0
    U6  1    0     1     0     0     1

I want to select a user and identify the other users that are similar to them, so that I can recommend them software/hardware that those similar users are using.

I have looked into Cosine and Jaccard similarity but I have not seen much help when it comes to calculating them on Boolean data.

I'm using Python and am not new to the language, but I am fairly new to Data Analytics and ML. Any advice is appreciated!

You can use the scipy module to calculate similarities. For your example, this would work like this:

import pandas as pd
users = ["U1", "U2", "U3", "U4", "U5", "U6"]
X = pd.DataFrame({
"Dep1": [0, 1, 1, 0, 0, 1],
"Dep2": [1, 0, 0, 1, 1, 0],
"Comp1": [0, 0, 1, 0, 0, 1],
"Comp2": [1, 1, 0, 1, 1, 0],
"Site1": [0, 1, 1, 0, 1, 0],
"Site2": [1, 0, 0, 1, 0, 1],
"Sitex": [1, 0, 0, 1, 0, 1]

}, index =users)
X

Out[2]:
    Comp1  Comp2  Dep1  Dep2  Site1  Site2  Sitex
U1      0      1     0     1      0      1      1
U2      0      1     1     0      1      0      0
U3      1      0     1     0      1      0      0
U4      0      1     0     1      0      1      1
U5      0      1     0     1      1      0      0
U6      1      0     1     0      0      1      1

from scipy.spatial.distance import pdist
from scipy.spatial.distance import squareform

# calculate the jaccard distances of the users, you can change jaccard to 'cosine'
distances = pdist(X.values, metric='jaccard')

# pdist returns a non-redundant vector of distances
# distances matrices are always symmetric, and diagonals are always zero
# so it is unnecessary to save all the information
# however if we want a matrix, we can use squareform()

distance_matrix = squareform(distances)

# the distances are bewteen 0 and 1, we get the similiarities by using 1-distances
similiarities = 1-distance_matrix

pd.DataFrame(similiarities, columns = users, index = users)

Out[3]:
          U1        U2   U3        U4   U5        U6
U1  1.000000  0.166667  0.0  1.000000  0.4  0.333333
U2  0.166667  1.000000  0.5  0.166667  0.5  0.166667
U3  0.000000  0.500000  1.0  0.000000  0.2  0.400000
U4  1.000000  0.166667  0.0  1.000000  0.4  0.333333
U5  0.400000  0.500000  0.2  0.400000  1.0  0.000000
U6  0.333333  0.166667  0.4  0.333333  0.0  1.000000


Time series classification with convolutional neural network I have to classify a time series signal with a CNN (not a LSTM or some kind of RNN). The input signal is a window with a width of 90 samples (currently, but can be changed) and 59 sensor values. So my signal can be represented by a 90x59 matrix for a single class. I have 28 classes.

Now I´m looking for a good solution to classify this. My first try ends in a very poor result of 24% accuracy and I´m thinking of multi channel models for all 59 channels in my model.

Is there any good way to start for some kind of good prediction results?

I'm not convinced 24% accuracy is very poor, depending on what your dataset looks like. Your baseline accuracy is in the realm of 3.5% with 28 classes. But let's assume you want to do better.

CNNs for images use 2-d convolution filters, because there is an expectation that pixels that are close together should be processed together. I guess you could use a 2-d convolution on the 90x59 matrix, but it isn't clear to me that we should be looking for information across "close" sensors. Perhaps you can use some sort of "tall" 2-d convolution, so that you capture information across all sensors in the same time span.

You may want to try training a model on each channel and ensembling them instead, as you mention. If you do this, I guess you could use a 1-d convolution to process each channel, but I don't really see why you would. Is there a reason you feel the need to use CNNs? There are very likely more appropriate models specific to processing time-series data.


What is the difference between LS-SVM and P-SVM? What is the difference between least-squares SVM (LS-SVM) and proximal SVM (P-SVM)? How does the decision boundary change in case of both of these types of SVM?

Least squares support vector machines were first introduced by Suykens and Vandewalle in 1999. They are slightly prior to proximal SVM proposed by Fung and Mangasarian in 2001. It appears that the latter did not know about LS-SVM and re-discovered the approach in a very similar manner.

Both methods are very close in the aspect that, instead of solving a quadratic programming (quadratic problem optimization with linear constraints), they solve a linear system. The trick to do this is to transform the optimization constraint from an inequation to an equation. Both methods use the same trick.

There appear to be minor differences though. In the conclusion of their paper on P-SVM, Fung and Mangasarian note:

  Least squares are also used in [33] to construct an
  SVM, but with the explicit requirement of Mercer’s positive
  definiteness condition [35], which is not needed here. Furthermore,
  the objective function of the quadratic program
  of [33] is not strongly convex like ours. This important feature
  of PSVM influences its speed as evidenced by the many
  numerical comparisons given here but not in [33].

Where [33] is the original LS-SVM paper by Suykens and Vandewalle.

As of today, a quick internet search shows that LS-SVM are much more common. Whether it is because of anteriority / initial popularity, because they perform better, and/or because their implementation is easier, however, I don't know.


Magnifying or reducing the size of input groups into a neural network Say you've got two inputs (X1 and X2) that you want to use to predict Y.

You're not sure how important X1 and X2 are for predicting Y, but you assume about even.

One-hot encoding is a good strategy for X1 and it yields a vector of size 10,000.

X2 is an unsigned int between 0 and 1, so you can just pass it as-is.

So your network will look something like

10,001 -> {some hls} -> Out-layer

In theory it could "learn" to assign a lot of importance to X2 compared to X1, but in practice I assume this is hard when the difference in dimension varies so much. At least based on some datasets I tested this was certainly true.

The one "simple" solution I can think of to this issue is instead having a network shapes more like:

10,000 -> {some hls} -> 1,000 --
                               |
                               |-> {some hls} -> Out-layer
                               |
1 -> {some hls} ------> 1,000 --

So basically have some encoders/backbones/whatever-you-want-to-call-them, that increase/reduce the size of certain groups of inputs and train them at the same time as the normal network.

My question here is:

a) Is the problem I identified here a "real" one or would it not come up in practice ?
b) Does it have a name and are there already established solutions to it ?
c) Is the solution I proposed here a "good one" ?
d) Do you have any examples of networks where this method is actually being used ? Preferably implemented in pytroch but it doesn't matter really, the implementation per-say seems easy enough.
f) Are there potential pitfalls in terms of performance and or implementation to this solution that might not be obvious ?

The main problem I see here is that OHE is almost never a good idea with that many categories. With neural networks you will usually get better performance by using embeddings. So instead of

X1 -{OHE}-> 10,000 -> {..} -> 1,000 you could go straight to X1 -{embedding}-> 50, where the embedding dimension should probably a lot lower than 1,000.

Either way, more important is that one calculates feature importance in a coherent framework. The easiest one is probably permutation importance (another could be shap values), which basically comes down to (repeatedly) shuffling the input features X1 and X2 and record the impact on Y. This works also for OHE variables if you make sure to shuffle X1 before one-hot encoding (and not shuffling the 10,000 encoded vectors independently). In such a framework X2 doesn't need to be "magnified" for a fair comparison with X1.


Can a Gradient Boosting Regressor be tuned on a subset of the data and achieve the same result? I am working with a large data set (~9M rows with 20+ features). Is it ok to tune via grid search on a fraction of the data (~100k rows) to determine optimal hyperparameters? This is mostly for choosing max_features, min_samples, max_depth. Trees and learning rate come later. Will I get different results tuning the fraction versus the whole data set?

You should never train or do grid search on your entire data set, since it will lead to overfitting and reduce the accuracy of your model on new data. What you have described is actually the ideal approach: do grid search / training on a subset of your data. Yes, your model will get different results vs if you used the entire set of data, but your model will be much stronger because of it.

For more details on why you would want to split up / sample your data, see this quesiton: https://stats.stackexchange.com/questions/19048/what-is-the-difference-between-test-set-and-validation-set


Is there a tool/ library/ algorithm that can learn from the steps I perform in cleaning text and apply them to similar data? I have a dataset of organization names that is quite messy. I used all the popular NER tools on it without much success(I assume it's because they lack context). I resolved to using OpenRefine but I reached a dead end with it's filters not picking up a lot of similar strings. I would like to use the data that I cleaned so far with OpenRefine for a (preferably supervised) machine learning algorithm that can afterwards continue the cleaning.
Are there any resources that could help with this?

Trifacta (https://www.trifacta.com/) supposedly can do that (learn from some examples provided by the user. I have no interest in the company, but it comes from academic research that I'm familiar with (http://vis.stanford.edu/wrangler/). Google had something free online (Google Refine) that can also do some intuitive things very simply, but I don't think it's as well developed.
If you try either product, please let us know about your experience!


Evaluating metrics F1, F2, Mean Average Precision for object detection Up today in the company where I work we are using the F1 Score for evaluating the performance of our model, also our competitor's using the same metric.

I would like to understand what's the difference between F1,F2 and mAP?(Please do not explain me how to calculate them, also I know that F measure gives the same weight to the precision and recall while mAP choose the best precision from all recalls)
Why in competitions (e.g. PASCAL VOC) and articles for object detection I am reading it is always preferred to use mAP instead of F1 or F2 scores ?

Thanks !

AP is more accurate than the F scores because it considers the PR relation globally. Articles adopt mAP on VOC because it is the official metric and they have to do comparison with other methods which also adopt this metric. Other competetitons such as some text detections also adopt P R and F score as the default metrics.


GAN - am I seeing mode collapse? Common fixes not working I have a 2 part question.

Context
I am learning about GANs and writing my own starting from the very simplest example of adversarial learning (1-parameter node), then implementing a very simple 1-dimensional pattern (1010) learning GAN .. and now I am trying to implement an MNIST learning GAN, before I proceed to more realistic photos.

I have some background in machine learning and data mining (masters from a long time ago) and moderately understand how neural networks work.

You can read about my progress on the initial steps here:

GANs - Part 1
GANs - Part 2

I've read lots of the latest blogs, articles, and watched youtube tutorials but still can't get past 2 key problems with implementing MNIST learning GANS.

Q 1. Am I seeing mode collapse?

After some tweaking and iteration I have a GAN which does learn to generate images which look like they might come from the MNIST dataset. Actually they're not digits yet but they are recognisable pen strokes, and certainly not random noise.

You can see a recent iteration of my pytorch code here: github notebook.

When fed random noise, the trained GAN always generates the same, or extremely similar image. Feeding it 1-shot noise (00001000...) also generates a similar image.

Are GANs supposed to generate just one image? Are the different images we see reported from separate training of the GAN? I thought the idea was a single trained GAN could generate many different images from random input noise. Have I misunderstood it?

Q 2. How To Escape Mode-Collapse?

If the answer above is that a train GAN should output many diverse but valid images, then I have mode-collapse.

I've read extensively and tried many approaches to avoiding mode-collapse but none have worked:

with / without batch normalization
with / without maxpooling
with / without dropout
with / without label softening
with / without noise added to both input and target labels, including noise that decays over training time
various widths and depths for the generator, less so the discriminator
increasing training time (but poor compute power only allows me about 6-10 epochs on the full dataset)

What I've observed or noted is:

the discriminator width/depth/architecture is tested first to ensure it has the capacity to learn the multi-class MNIST before it is used in the GAN to avoid the case of a discriminator that can't actually learn to discriminate MNIST
plotting the error as training proceeds (D error, D error on G input) is useful to show that the D error approaches 1/2 as it should, and that D error on G input approaches 1 (well more like 0.8).
plotting errors also shows stability or collapse of some sort which helps tune the learning rates for example
I would have thought adding noise to the input or the target labels might kick the GAN out of any local minima which is mode-collapse but I suspect the theory is more complicated than this
trying different suggestions for the optimiser parameters doesn't help, I have to find my own optimal tuning, and the learning rates are much lower than what others use, eg Adam lr=0.00002 not 0.001 which causes instability
higher training epochs result in high contrast images which don't look like the MNIST data which has softer edged strokes .. I was hoping higher training epochs would improve diversity of outputs

One area where I can't find much guidance is on the actual architecture of the discriminator and generator:

Do they have to be matched but opposite? mine aren't - the discriminator is proven to have the learning capacity and that's it. After that the shallower the better to ensure easier back propagation to the generator.
The use of deconvolution is common in the generator, but some examples online use simple fully connected mapping. Computationally the deconvolutions have fewer learning parameters and intuitively make sense in building an image.

I'd welcome your thoughts and suggestions.

Example output which I think looks good, but is mode-collapsed:

Since this question I have made significant progress, which I have written up as part 3 of the series:

http://makeyourownalgorithmicart.blogspot.com/2019/05/generative-adversarial-networks-part-iii.html

Although my GAN doesn't now have mode collapse - the reasons are still unclear. The post reports visually the impact of combinations of SGD->Adam, none->LayerNorm, Sigmoid->ReLU which I hope others find useful:


Is there any clustering algorithm to find longest continuous subsequences? I have data which contains access duration of some items.

Example:
t0~t5 is the access time duration, 1 means the items was accessed in the time duration, 0 means it wasn't.

ID,t0,t1,t2,t3,t4
0,0,0,1,1,1
1,0,1,1,1,1
2,0,1,1,0,0
3,1,1,0,0,1
4,1,1,0,0,1

In the above example, groups ID=0,1 are what I want.

ID=3,4 aren't because their distance is short but they are not continuous.

I tried KMeans and DBSCAN, they all cluster ID=3,4 as one group and it makes sense. But it doesn't do what I want.

Is there any possible pre-processing of data to reach what I want ?

Or I should use other analytic tool?

What might help is a custom distance computation as input to the clustering algorithm. These algorithms usually take Euclidean distance as a measure of dissimilarity.

You can try DBSCAN (in Python scikit-learn), with metric='precomputed' and 'X' as a custom distance matrix. You can construct this distance matrix to conform to your requirement. Eg: specify that nodes 3 and 4 have a large distance, even though they are equal.


Why is my prediction using ARIMA better if I'm using less historic data? I have a data set containing hourly electricity prices for since 1.01.19 until September. Since the process turned out to be (weakly) stationary, I applied an ARIMA model in Python in order to predict the prices for the next day.

It turned out that the best prediction was made using the last two days as historic data and the worst was the one using almost 6000 values.

What is a possible explanation for this happening?

Electricity prices are essentially the same as stock prices: best modelled by a random walk, where the best prediction for tomorrow is the price today. Therefore I am not really surprised that you get worse results using more historical data.
 Some versions of ARIMA will also include regularisation, which will punish your model for including more and more data - inclusion of new data must be justified by contributing to a lower residual error to be "worth" inclusion.

Other models that tend to be a little more robust use features other than the actual target, here the price. For example, trying to predict the volatility of the price might  prove to be more accurate. For this there are GARCH models (Generalised AutoRegressive Conditional Heteroskedasticity).

Another thing you might consider is to include external data... for example, electricity consumption is heavily influenced by the weather - if it is cold outside, a lot of people heat their homes using electrical heaters, they also drink more hot drinks etc.


I'm building a movie recommendation system based on genre, where the user will enter his choice and will receive recommendations based on his choice: Column Name: "Genres" //This will contain the genres of movies
User genre: "Action Adventure"
I want to perform cosine similarity on this data, compare user genre with genres of different movies, sort and display the names of movies having similar genre. How should I do it?

First of all, your question is very general and missing crucial information - you should highlight which language and dataset you are planning to use. You should also include it in tags of the post.
Anyway, I suppose that you are working with one of the IMDB datasets. I can personally recommend this article and this article. The authors explain the basic techniques how to build a recommender system in Python using scikit-learn library including cosine similarity.


Can xgboost (or any other algorithm) give bad results with some bad features? till now I was under the impression that machine learning algorithms (gbm, random forest, xgboost etc) can handle bad features (variable) present in the data.

In one of my problems, there are around 150 features and with xgboost I am getting a logloss of around 1 if I use all features. But if I remove around 10 bad features (found using some technique) I am observing a logloss of .45. That is huge improvement.

My question is, can bad features really make such big differences?

Yes it can for sure, some algorithms are more robust to this than others but doing proper feature selection is adviced. This is due to the curse of dimensionality. It's not only the algorithm but also has a lot to do with the amount of data points you have compared to the number of features. If you have 10,000,000 data points 150 features is not an issue, if you have 400 data points 150 features is way too much. How can you learn something when there is so little to learn from? Overfitting is a real issue for almost any algorithm, and noisy features make it more easy to learn things that are not actually there.


How to develop Top-N recommendation for evaluating my system I want to evaluate my recommender system with the Top-N recommendation method and I have a problem. In some situations, e.g. N = 5, I don't have 5 items for listing and I cannot do that for evaluating. now, you suppose, I want to increase N from 5 to 10 or 15, I think I will have some problems with some active users.
Would you please give me a suggestion?!!!

Discounted cumulative gain (DCG) is one of the most common metrics for ranking quality. The gain is accumulated from the top of the result list to the bottom, with the gain of each subsequent result logarithmic discounted.

The DCG accumulated at a particular rank position n is defined as

Since search result lists vary in length, normalized DCG is used. It is normalized by the ideal ranking, sorting all relevant documents in the corpus by their relative relevance and producing the maximum possible DCG:


Image clustering with deep learning I want to cluster image, since varibility intra and inter class of images is huge I think reducing dimensions with a convolutional autoencodeur can be a good tools. Then I apply clustering on the feature vector

My question: is there a theorical link between my convolutional autoencodeur loss and the potential for clustering of my extracted feature vector?

To add more detail, can we says if I can minimise loss then it'll be easier to do clustering on feature vector? My intuition tel me answer is NO

But if we can't use loss to select potential best model for clustering, then isn't it almost impossible to do a good clustering using this methodology?

Few clustering algorithms have any kind of loss. That is pretty much only k-means and a few closely related methods that fit this narrow world view of "everything is optimizing a loss".

Now if your features would minimize the same loss, then the result should be close to just doing k-means on the raw pixels - which is known to not work well. You do want features that lose a lot of irrelevant information.


How and where to set weights in case of imbalanced cost sensitive learning in machine learning? I confront with a binary classification machine learning task which is both slightly imbalanced and cost sensitive. I wonder what (and where in the modeling pipeline, say, in sklearn) is the best way to take all these considerations into account.
Class proportionality: positive: 0.25% negative: 0.75%. This could be addressed with sklearn.utils.class_weigh.compute_class_weight:
class_weights = compute_class_weight(y=y, class_weight='balanced')

OK, but this is only for rebalancing proportionalty, I should take misclassification cost into consideration as well. Let's say that this is 10* larger in case of false negatives than false positives, so I guess that I should still increase the weights in class_weights accordingly by upweighting positives further by 10, right?
But there is another point in the pipeline where I could take care of this, namely in the evaluation metrics with F-beta for example with upweighted recall (F2, for instance). Does it have the same effect? Should I pick one method (F-beta for evaluation OR upweighting classes) or both of them simultaneously?.
Additionally, in case I upweight my classes with compute_class_weight(), I assume that no further class distribution should be taken into consideration downstream (so when I use RandomForestClassifier(), class_weight hyperparameter shouldn't be ='balanced', again, because this would further distort the weights proportionality that is already set before. Is this correct?

Additionally, in case I upweight my classes with compute_class_weight(), I assume that no further class distribution should be taken into consideration downstream (so when I use RandomForestClassifier(), class_weight hyperparameter shouldn't be ='balanced', again, because this would further distort the weights proportionality that is already set before. Is this correct?

The util function is just a utility that will return you the dictionary for class weight.
The place where you will use this dictionary is the class_weight parm of the Model e.g. RandomForest.
So, you will have to pass the return of the Util to the model.
Better, just keep the parameter value = "balanced" Or any specific value e.g. [{1:1}, {2:5}, {3:1}, {4:1}]

OK, but this is only for rebalancing proportionalty, I should take misclassification cost into consideration as well. Let's say that this is 10 larger in case of false negatives than false positives, so I guess that I should still increase the weights in class_weights accordingly by upweighting positives further by 10, right?*

Just using the Util and getting the dict of weight will not have any effect. Action will happen only when the Model uses the class_weight parameter.
The Model will add corresponding weight to the misclassification cost of the minority class. Low-level design detail changes with Model e.g. Neural network, Decision tree.

But there is another point in the pipeline where I could take care of this, namely in the evaluation metrics with F-beta for example with upweighted recall (F2, for instance). Does it have the same effect? Should I pick one method (F-beta for evaluation OR upweighting classes) or both of them simultaneously?

Change in Scoring technique will be needed because whatever be the Class weights, the counts are still unbalanced e.g. 100:1. So, 90 correct of the majority will make the overall accuracy ~90 which can be deceiving.


Appropriate Clustering Algorithm I need to find a good clustering for this data using sci-kit.

KNN is not appropriate as it creates blobs although these data are linearly separated.

import hypertools as hyp
hyp.plot(tocluster,'o', n_clusters=10)

I'd rather not rely on clustering (clearly, DBSCAN would be the first method to try).

Instead, I'd look for a projection that removes the correlation (x - y maybe?), then define a grid to separate the data.


Is rolling of biased dice random phenomenon? Random phenomenon is a situation in which we dont know what the out come is going to come.
Rolling of unbiased dice is a random phenomenon since we dont know what number is going to come. We can only say every number has
1/6 probability.
Rolling a biased coin on other hand also we dont know which number is going to come. We only know every number does not have 1/6 probability.
So does that mean rolling of biased dice is a random phenomenon ?

Yes, you are correct. Rolling of dice is (almost always) a random phenomenon. In case of biased v unbiased dice, the difference is solely in the probability distributions. For unbiased the distribution must be uniform, for biased it can be anything we want.
The only case where biased dice stop being random phenomena is when we load them in such a way that they always produce a predetermined number (or a specific sequence).


I need sources of interrogative, exclamatory, and imperative sentences I am working on accumulating a large database of labeled sentences for several projects/experiments. At present I am only using Wikipedia and Project Gutenberg as sources of data. Between these two sources I expect I will be able to extract millions of sentences. Labeling them may be another matter, but I'm less concerned about that right now.

My chief need is to find a broader variety of sentences. Questions, exclamations, invective, conversation, and so on. My project is several deep learning experiments to look at different types of sentences and structures. It would be helpful if the sentences were labeled.

The ultimate goal of my project is to be able to infer the type, structure, and coherence of a sentence (with separate models). Generating gibberish is pretty easy to do, so I'm not as concerned about finding sources of non-sense sentences.

My current concern is that the vast number of sentences are declarative (statements) rather than interrogatives, exclamatory or imperatives. This bias towards declarative sentences means that my sample sizes will be limited unless I can find sources of questions, exclamations and commands.

Alternatively, I would be open to suggestions about how to generate the data I need.

I ultimately settled on Gutenberg Project as a source of massive amounts of data. I realized that books (fiction and non-fiction) cover pretty much every permutation of human communication that can be represented in plain text. I've written a couple of scripts to parse Gutenberg text files and next I'll be working on a support vector machine to classify each passage as regular content or other (such as headers, titles, etc).

https://www.gutenberg.org/wiki/Gutenberg:Information_About_Robot_Access_to_our_Pages


(De-)Scaling/normalizing input and output data inside Keras model as layer I am building a 2-hidden layer MLP using Keras.
I'm using a SciKit learn wrapper to be able to use the GridSearchCV functionality.
My sample-size is limited, forcing me to use K-fold verification as well for trustworthy results.

However, it is to my understanding that for every iteration in the K-fold validation, input data should be scaled (and output descaled) only using the training data. This requires the scaling to be performed inside the Keras model.

In order to have understandable results, the output should than be transformed back (using previously found scaling parameters) in order to calculate the metrics.

Is it possible to

Z-score standardize my input data (X & Y) in a normalization layer (batchnormalization for example)
Transform the output layer back (before calculating metrics), using scaling parameters found in 1.

I've looked at the batchnormalization functionality in Keras, but the documentation mentions: "During training time, BatchNormalization.inverse and BatchNormalization.forward are not guaranteed to be inverses of each other because inverse(y) uses statistics of the current minibatch, while forward(x) uses running-average statistics accumulated from training." Which seems like this prevents the functionality from being used in this manner.

Does anybody have a known solution or function for this?

Let's start from X. As you use the sklearn-style interface, the natural choice is to use StandardScaler. You could use pipeline to integrate it in the grid search.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
pipeline = Pipeline([
   ('scaler', StandardScaler()),
   ('model', your_cool_model)])
param_grid = {...}
search = GridSearchCV(pipeline, param_grid, cv=5).fit(X, y)

That would do the trick.

Now about y. The chances are you don't need to scale target. The good discussion is in this question

If you are certain that targets should be scaled, the sklearn provides TransformedTargetRegressor just for the case

wrapped_model = TransformedTargetRegressor(
    regressor=model,
    transformer=StandardScaler())