Too high performances on a classification problem I have a .json file as dataset of the type:

and I am working on a classification problem in which I have to predict 4 classes, which are rhe semantic. I have worked through the problem, and after splitting the dtataset into training and test sets, I get an accuracy of $1$ .I have an unbalanced dataset, so I have oversampled it:

my code is the following:
dataFrame = pd.read_json('dataset.json',lines = True)
df = dataFrame[["lista_asm", "semantic"]].copy()

from sklearn.feature_extraction.text import TfidfVectorizer
tfidf_vectorizer=TfidfVectorizer()
df_x = df['lista_asm']

X_all = tfidf_vectorizer.fit_transform(df_x)
y_all = df['semantic']

#oversampling
from imblearn.over_sampling import RandomOverSampler

 ros = RandomOverSampler()
 X_ros, y_ros = ros.fit_sample(X_all, y_all)

 print(X_ros.shape[0] - X_all.shape[0], 'new random picked points')

#splitting

X_train, X_test, y_train, y_test = train_test_split(X_ros, y_ros,
    test_size=0.2, random_state=15)

from sklearn import svm
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier

#fitting the model
clf = svm.SVC(kernel='linear', C=10).fit(X_train,y_train)

y_pred = clf.predict(X_test)

#checking accuracy
acc = clf.score(X_test, y_test)
print("Accuracy %.3f" %acc)          #from here I get accuracy 1

I have not written the imports in order to avoid making the code too long here, but if needed I can add them.
So I am not testing on the training set, but the results are too good so for sure there mmust be something wrong.I don't understand what is wrong.
Is this something that happen for any specific reason?
I have tried to change some stuff to make the code worse, in order to see if the accuracy would go down, but it remains to one.
[EDIT]I have tried to use your suggestions, and I am already sorry since for sure I am doing something wrong.
Here is what I have done:
X_all = df['lista_asm']
y_all = df['semantic']

X_train, X_test, y_train, y_test = train_test_split(X_all, y_all,
    test_size=0.2, random_state=15)

tf = TfidfVectorizer(analyzer='word', ngram_range=(1,2), lowercase = True,
max_features = 20000)
tf_transformer = tf.fit(X_train)
xtrain = tf.fit_transform(X_train)

from sklearn import svm
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier

clf = svm.SVC(kernel='linear', C=10).fit(xtrain,y_train)

tf2=TfidfVectorizer(analyzer='word', ngram_range=(1,2), lowercase = True,
max_features = 20000)
tf_transformer = tf2.fit(X_test)
X_test = tf2.fit_transform(X_test)

y_pred = clf.predict(X_test)

acc = clf.score(X_test, y_test)
print("Accuracy %.3f" %acc)       #gives accuracy 0.277

and if I plot the other performance metrics:

with the following message:

UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use zero_division parameter to control this behavior.
_warn_prf(average, modifier, msg_start, len(result))

[EDIT]I have also tried to use the same approch but with a .jsonl file, and in this case it works fine giving a reasonable accuracy. Maybe it is wrong to do so, but could the problem be here, so that I am not handling correctly a .json file?
Can somebody please help me?

As noted by Peter, you should split your data into test and train
sets before fitting the TfidfVectorizer to the training set (not to
all the data).

Pandas read_json() is not reading your data from file the way you
want. You are expecting df["lista_asm"] to be a Series of list
objects (each containing strings). This Series is instead a  Series
of strings (formatted to appear like a list of strings). You must
parse the strings into a list, if that is what you want. But I
believe that the TfidfVectorizer().fit method wants strings, not a
list of strings. (This issue was apparent before your edit)

Similar to the first issue, you should avoid using the
RandomOverSampler() on the test data. If you want to use it, apply it
only to the training data. I suspect that it may be creating
duplicates or near-duplicate records, some of which are later split
into the test set. Try not oversampling at all and evaluate the
performance.


Model selection: large mean and variance vs small mean and variance This question was always in my mind. Imagine you are doing 5-10 fold cross validation and one model gives you mean accuracy of 0.8, but with 0.2 standard deviation and the other one gives 0.7 with 0.05 standard deviation. Which one is better?

This is exactly a question I asked Sebastian Raschka (author of the great Python Machine Learning book), here you can find his answer saying "I also recommend the 1-standard error method, which basically means that you select the best model from k-fold based on pure performance, and then you select the simplest model that is within 1 standard error of that model".
The more extended explanation of his answer can be found on his github link.


CNN backpropagation between layers I have this CNN architecture:

I know how to calculate error for weights based on the output and update weights between output<-->hidden and hidden<-->input layers.

The problem is that I have no idea how to calculate delta for values in input layer based on the error and then use it in convolution backpropagation.

Let's look at the layers before the reshaping stage since everything after that is simply a densely connected neural network.

Backpropagation in Max Pooling

Max pooling takes a window of values and only the maximum value passes through. This means that the error can only be contributed by the maximum values, thus only the weights for these values will be updated.

Backpropagation in the Convolutional Layers

This is the same as for the densely connected layer. You will take the derivative of the cross-correlation function (mathematically accurate name for convolution layer). Then use that layer in the backpropagation algorithm.

An Example

Let's look at this following example

Forward pass

The forward pass of the convolutional layer can be expressed as

$x_{i, j}^l = \sum_m \sum_n w_{m,n}^l o_{i+m, j+n}^{l-1} + b_{i, j}^l$

where in our case $k_1$ and $k_2$ is the size of the kernel, in our case $k_1=k_2=2$. So this says for an output $x_{0,0} = 0.25$ like you found. $m$ and $n$ iterate across the dimensions of the kernel.

Backpropagation

Assuming you are using the mean squared error (MSE) defined as

$E = \frac{1}{2}\sum_p (t_p - y_p)^2$,

we want to determine

$\frac{\partial E}{\partial w^l_{m', n'}}$ in order to update the weights. $m'$ and $n'$ are the indices in the kernel matrix not be confused with its iterators. For example $w^1_{0,0} = -0.13$ in our example. We can also see that for an input image $H$x$K$ the output dimension after the convolutional layer will be

$(H-k_1+1)$x$(W-k_2+1)$.

In our case that would be $4$x$4$ as you showed. Let's calculate the error term. Each term found in the output space has been influenced by the kernel weights. The kernel weight $w^1_{0,0} = -0.13$ contributed to the output $x^1_{0,0} = 0.25$ and every single other output. Thus we express its contribution to the total error as

$\frac{\partial E}{\partial w^l_{m', n'}} = \sum_{i=0}^{H-k_1} \sum_{j=0}^{W-k_2} \frac{\partial E}{\partial x^l_{i, j}} \frac{\partial x^l_{i, j}}{\partial w^l_{m', n'}}$.

This iterates across the entire output space, determines the error that output is contributing and then determines the contribution factor of the kernel weight with respect to that output.

Let us call the contribution to the error from the output space delta for simplicity and to keep track of the backpropagated error,

$\frac{\partial E}{\partial x^l_{i, j}} = \delta^l_{i,j}$.

The contribution from the weights

The convolution is defined as

$x_{i, j}^l = \sum_m \sum_n w_{m,n}^l o_{i+m, j+n}^{l-1} + b_{i, j}^l$,

thus,

$\frac{\partial x^l_{i, j}}{\partial w^l_{m', n'}} = \frac{\partial}{\partial w^l_{m', n'}} (\sum_m \sum_n w_{m,n}^l o_{i+m, j+n}^{l-1} + b_{i, j}^l)$.

By expanding the summation we end up observing that the derivative will only be non-zero when $m=m'$ and $n=n'$. We then get:

$\frac{\partial x^l_{i, j}}{\partial w^l_{m', n'}} = o^{l-1}_{i+m', j+n'}$.

Then back in our error term

$\frac{\partial E}{\partial w^l_{m', n'}} = \sum_{i=0}^{H-k_1} \sum_{j=0}^{W-k_2} \delta_{i,j}^l o^{l-1}_{i+m', j+n'}$.

Stochastic gradient descent

$w^{(t+1)} = w^{(t)} - \eta \frac{\partial E}{\partial w^l_{m', n'}}$

Let's calculate some of them

import numpy as np
from scipy import signal
o = np.array([(0.51, 0.9, 0.88, 0.84, 0.05),
              (0.4, 0.62, 0.22, 0.59, 0.1),
              (0.11, 0.2, 0.74, 0.33, 0.14),
              (0.47, 0.01, 0.85, 0.7, 0.09),
              (0.76, 0.19, 0.72, 0.17, 0.57)])

d = np.array([(0, 0, 0.0686, 0),
              (0, 0.0364, 0, 0),
              (0, 0.0467, 0, 0),
              (0, 0, 0, -0.0681)])

gradient = signal.convolve2d(np.rot90(np.rot90(d)), o, 'valid')

  array([[ 0.044606,  0.094061], [ 0.011262,  0.068288]])

Now you can put that into the SGD equation in place of $\frac{\partial E}{\partial w}$.

Please let me know if there are errors in the derivation.


Neural Network - Adjust number of hidden layers and neurons I am using KNIME MLP Neural Network learner (If you are not familiarized with KNIME think of that like a package which implements Neural Network to a set of data).

The thing is you can tune the number of hidden layers, the number of iterations and neurons per layers, so here comes the first question: How to know a priori values for these parameters? For instance, for Random Forest 500 trees is supposed to be a standard to test its performance in the first stages.

Second, I know in general add more layers and/or neurons does not ensure any enhance in the prediction, sometimes even the opposite happen. Someone could give some theoric insights respect this topic; why by adding more layers the prediction might get deteriorated?

Thanks in advance!

Sergi

Sorry for not knowing exact number for the first question, but my point is to start from the most simple model, rather than any random parameter. Then use a grid search for the parameter by validation.

As for the second question, the error of a model comes from two parts, variance and bias. Bias comes from lack of complexity of your model. For example, there will always be a large bias when using linear model to fit the distance v.s. the shooting angle of a canon. This can be overcome by using a more complex model,  adding layers to the NN for example.

And variances comes from being too sensitive to the data, even to the noise in the data. When there is a high variance due to using a too complex model, it is called over-fitting. You may take a look at this page(over-fitting on wiki). And using more layers/neurons is using a more complex model, which means the model may suffer from high variance. This can be overcome by lower the model complexity(which may increase the bias) or adding training data.

Basically if you got constant number of data, then you have to find somewhere where both bias and variance is not too high.


I got the following error : 'DataFrame' object has no attribute 'data' can you help please I do have the following error: AttributeError: 'DataFrame' object has no attribute 'feature_names'

appreciate your input

from sklearn.tree import DecisionTreeClassifier, export_graphviz
from sklearn import tree
from sklearn.datasets import load_wine
from IPython.display import SVG
from graphviz import Source
from IPython.display import display
from ipywidgets import interactive
import pandas as pd

df1 = pd.read_csv("/Users/dean/Desktop/AI TECNOMATIX EXCEL/cartest1.csv")
#print(df1)
inputs = df1.drop('StatNumOut',axis='columns')
target = df1['StatNumOut']
from sklearn.preprocessing import LabelEncoder
le_company = LabelEncoder()
le_job = LabelEncoder()
le_degree = LabelEncoder()
inputs['name_n'] = le_company.fit_transform(inputs['name'])
inputs['MachineAvailability_n'] = le_company.fit_transform(inputs['MachineAvailability'])
inputs['MachineExitLocked_n'] = le_job.fit_transform(
    inputs['MachineExitLocked'])
inputs['MachineEntranceLocked_n'] = le_degree.fit_transform(
    inputs['MachineEntranceLocked'])
inputs['MachineMTTR_n'] = le_degree.fit_transform(inputs['MachineMTTR'])
inputs['StatNumIn_n'] = le_degree.fit_transform(inputs['StatNumIn'])
inputs_n = inputs.drop(['name',
                        'MachineAvailability',
                        'MachineEntranceLocked',
                        'MachineExitLocked',
                        'MachineMTTR',
                        'StatNumIn'],
                       axis='columns')
inputs_n
target

model.score(inputs_n,target)

labels = df1.feature_names
def plot_tree(crit, split, depth, min_split, min_leaf=0.2):
    estimator = DecisionTreeClassifier(random_state = 0
       , criterion = crit
       , splitter = split
       , max_depth = depth
       , min_samples_split=min_split
       , min_samples_leaf=min_leaf)
    estimator.fit(inputs_n, target)
    graph = Source(tree.export_graphviz(estimator
       , out_file=None
       , feature_names=labels
       , class_names=['0', '1', '2']
       , filled = True))

    display(SVG(graph.pipe(format='svg')))
    return estimator

inter=interactive(plot_tree,
                  crit = ["gini", "entropy"],
                  split = ["best", "random"],
                  depth=[1,2,3,4],
                  min_split=(0.1,1),
                  min_leaf=(0.1,0.5))
display(inter)
```

As pointed out in the error message, a pandas.DataFrame object has no attribute named feature names. You probably meant something like df1.columns.


openai gym - what is an agent I can use with a multi-discrete action space? I have a custom environment with a multi-discrete action space.

The action and observation spaces are as follows:

Action:

MultiDiscrete([  3 121 121 121   3 121 121 121   3 121 121 121   3 121 121 121   3 121
 121 121   3 121 121 121   3 121 121 121   3 121 121 121   3 121 121 121
   3 121 121 121   3 121 121 121   3 121 121 121   3 121 121 121   3 121
 121 121   3 121 121 121   3 121 121 121   3 121 121 121])

Observation:

MultiDiscrete([100   3   2 121   2 121   2 121   2 121   2 121   2 121   2 121   2 121
   2 121   2 121   2 121   2 121   2 121   2 121   2 121   2 121   2 121
   2 121   2 121   2 121   2 121   2 121   2 121   2 121   2 121   2 121
   2 121   2 121   2 121   2 121   2 121   2 121   2 121   2 121   2 121
 121 121 121 121 121 121 121 121 121 121 121 121 121 121 121 121 121 121
 121 121 121 121 121 121 121 121 121 121 121 121 121 121 121])

I am having an extremely tough time finding an agent (for example in keras-rl) that is capable of handling these spaces.

This issue: https://github.com/keras-rl/keras-rl/issues/224 indicates that the keras-rl DDPG agent is capable of handling a multi-discrete action space, but the model has a float output that I cannot use as an action for the step() function, which expects an integer output!

Most other agents seem to use a tanh activation layer, or some layer that produces a binary output. I need an output in the same shape as my action space.

How can this be handled?

OpenAI Baselines - or for me even better, Stable Baselines - has many model options which can handle MultiDicrete Action and/or Observation spaces.  Building a custom gym environment is also quite straightforward.


How to conclude the generality of any classification methods? Suppose a classification task A, and there exist a lot of methods $M_1, M_2, M_3$.
The task $A$ is measured by a consistent measure.
For instance, the task A can be a binary classification. In this case, F-score, ROC curve can be used.
I did a survey on some research are and found that

$M_1$ is evaluated with dataset $D_1$ (open) using pre-processing $P_1$ only (seems the seminal work).
$M_2$ is evaluated with dataset $D_1$ (open), $D_2$ (private) and compared with $M_1$, claiming $M_2$ has more accurate result, but using different data pre-processing $P_2$.
$M_3$ proposes new way with dataset $D_3$ (private) and did not provide any comparison against $M_2$ and $M_1$

I'm trying to work on this area, but there are lots of inconsistency.
None of the methods are validated with validation data. They just used train and test data.
I think some parameters are tuned for test dataset although authors do not claim so.
Since this field is not a data science-oriented and the amount of dataset is few, this may happen.
Which method can we consider as a state-of-the-art?
How can we conclude the generality of each of the method?

You're experiencing an unfortunately common issue with the current state of system/model evaluation. In addition to evaluating on different datasets, authors often leave out important details, such as the procedure for hyperparameter tuning, detailed evaluation metrics (i.e. true positives, false negatives, etc. in addition to F-score), and ablation analyses. In cases like these, we cannot conclude that one method is necessarily better than the others or state-of-the-art.
The best way to estimate the generality of each method when the literature has so far failed to do so is to implement each yourself and do a fair comparative evaluation. You would evaluate all methods on the same dataset with the same pre-processing steps and hyperparameter tuning procedure and, if possible, introduce additional evaluation datasets. It can also be very enlightening to perform an ablation analysis in which you iteratively remove certain components of the methods and re-evaluate to see how much of a performance hit you take.
Doing the above and communicating it (via a publication, blog post, or whatever) will not only help you, but everyone else working in the area.


Using the trainbr function for classification in Matlab I am training a neural network for classification using Matlab, and I don't understand if I can use the trainbr training function (Bayesian Regularization Backpropagation). It uses the MSE performance measure, but I want to use the crossentropy. If I set crossentropy as the performance function, the algorithm sets it back to MSE.

On the other way, I can't use a validation set with this training and I don't find how to change it.

The code is:

x = A';
t = y';

% Choose a Training Function
% For a list of all training functions type: help nntrain
% 'trainlm' is usually fastest.
% 'trainbr' takes longer but may be better for challenging problems.
% 'trainscg' uses less memory. Suitable in low memory situations.
trainFcn = 'trainbr';  % Scaled conjugate gradient backpropagation.

% Create a Pattern Recognition Network
net = patternnet(hiddenLayerSize,trainFcn);

% Choose Input and Output Pre/Post-Processing Functions
% For a list of all processing functions type: help nnprocess
net.input.processFcns = {'removeconstantrows','mapminmax'};
net.output.processFcns = {'removeconstantrows','mapminmax'};

% Setup Division of Data for Training, Validation, Testing
% For a list of all data division functions type: help nndivide
net.divideFcn = 'dividerand';  % Divide data randomly
net.divideMode = 'sample';  % Divide up every sample
net.divideParam.trainRatio = 60/100;
net.divideParam.valRatio = 20/100;
net.divideParam.testRatio = 20/100;

% Choose a Performance Function
% For a list of all performance functions type: help nnperformance
net.performFcn = 'crossentropy';  % Cross-Entropy
net.trainParam.epochs = 5000;

% Choose Plot Functions
% For a list of all plot functions type: help nnplot
net.plotFcns = {'plotperform','plottrainstate','ploterrhist', ...
    'plotconfusion', 'plotroc'};

% Train the Network
[net,tr] = train(net,x,t);

% Test the Network
y = net(x);
e = gsubtract(t,y);
net.performParam.regularization = 0;
performance = perform(net,t,y);

tind = vec2ind(t);
yind = vec2ind(y);
percentErrors = sum(tind ~= yind)/numel(tind);

% Recalculate Training, Validation and Test Performance
trainTargets = t .* tr.trainMask{1};
valTargets = t .* tr.valMask{1};
testTargets = t .* tr.testMask{1};
trainPerformance = perform(net,trainTargets,y);
valPerformance = perform(net,valTargets,y);
testPerformance = perform(net,testTargets,y);

Thanks

The trainbr mode uses the Bayesian regularization backpropagation. This method was presented in 1, which presents a regression problem with the loss function
$$ E_D = \sum_{i=1}^n (t_i - a_i)^2 $$
where $t_i$ is the target and $a_i$ is the network's response. The paper proposes to add a regularization term, leading to a loss function $F$ of the form
$$ F = \beta E_D + \alpha E_W $$
where $E_W$ is the square of the sums of all network weights, i.e. $E_W = \sum_{i,j} \| w_{ij} \|^2$.
The two parameters $\alpha$ and $\beta$ control the weighing of the two parts $E_D$ and $E_W$: For $\alpha \ll \beta$, the network will minimize the loss, without really trying to keep weights low. For $\alpha \gg \beta$, the network will minimize the weights, allowing for some more error. In reality, this means a large $\alpha$ will stop the network from overfitting, which leads to a better generalization at the cost of a larger training error.

The key to find a train a model which generalizes well, but still has a low error rate, is the right setting of $\alpha$ and $\beta$. This is achieved by treating them as random variables and finding an optimal setting, using the Bayesian methods presented in 2. (I won't talk about the details on that here, you can find that in the two linked papers.) Finally, the paper presents an algorithm, which calculates the optimal $\alpha$ and $\beta$ in each training iteration. This makes this algorithm generalize really well, especially in the presence of noisy input signals.

However, as described, the loss function is a weighted sum between the MSE ($E_D$) and the regularization term ($E_W$). So, in short: you can only use it with the MSE, and not with cross-entropy. You'll need a different training algorithm, you can find a list in the MATLAB documentation, here.

References:

  1 F. D. Foresee and M. T. Hagan: "Gauss-Newton Approximation to Bayesian Learning", in Proceedings of the 1997 International Joint Conference on Neural Networks, June 1997. DOI: 10.1109/ICNN.1997.614194. [Link to PDF].

  2 D. J. C. McKay: "Bayesian Interpolation", Neural Computation, May 1992, Vol. 4, No. 3, pp. 415-447. DOI: 10.1162/neco.1992.4.3.415. [Link to PDF].


Who invented the concept of over-fitting? I list the references that I found so far.
Shortly, the first appearance of the term was in 1670, first appearance in in close meaning was in 1827, first appearance in a biological paper was in 1923 and first appearance in statistics was in 1935.
However, the references indicate that there are gaps in this chronology.
Earliest reference I found was The flying pen-man; or, The art of short-writing by William Hopkins (teacher of stenography.) in 1670.
However, it is a table of words with "overfit" as one of them and the context or meaning is not clear.
I check the other references before 1800 and those were transcription errors, "over, fit" or "overfet".
The earliest appearance in context I found was in "The Enigmatical Entertainer and Mathematical Associate ..., Issue 1" in 1827 saying:
"I , though for poetry not overfit , 7 Will boldly dare to lay down rules for wit ;" hence the meaning of overfit here is "fit too well" but in a positive meaning.
It seems that Darwin's "On the Origin of Species" in 1859 made the concept of fitting popular.
Bray, Charles I. "Fitting livestock for show." Bulletin (Colorado Agricultural College. Extension Service); 171A (1923). : "They believe
that these are almost equally valuable, and that there is less
danger of over-fitting the pigs than with whole milk."
The 1934 paper Twenty-Two Years of the Eastern Percheron Futurity mentions the penalty of over-fitting yet regarding stallions.
The Quarterly Review of Biology Sep 1935 Volume 10, Number 3pp. 341 – 377, saying : "Perhaps we are old fashioned but to us a six-variate analysis based on thirteen observations seems rather like overfitting".
So it was well known and maybe a bit old-fashioned during 1935 already.
First statistics reference to over fitting I found appears in "Tests of Fit in Time Series", 1952.
Yet it says: " Such a graduation implies a gross
overfitting, but this can be allowed for."
Using overfitting as a well known concept.
The wikipedia article on overfitting references to Oxford dictionary entry that claims: "Origin 1930s; earliest use found in Quarterly Review of Biology. From over- + fitting."
It is possible that the reference is The Quarterly Review of Biology Sep 1935 Volume 10, Number 3pp. 341 – 377, yet I couldn't verify it.
I couldn't find the paper in Google Scholar.
Looking in Google ngram viewer it seems that the usage started around 1885.
Looking at Google book for that period shows references about sheep.
I'll be happy to learn that sheep are the origin of a central concept in AI but I'd like to see references.
Reference not related to sheep are welcome too.

Yes! It seems like first statistics reference to over fitting appears in The Quarterly Review of Biology  . It says
"Perhaps we are old
fashioned but to us a six-variate analysis
based on thirteen observations seems rather
like overfitting.
I am attaching the screenshot of that particular page for your reference.


GridSearchCV() to fine tune outputs ValueError and FitFailedWarning I want to fine tune some parameters for my linear SVM.
This is the code:
class SVMSentiment(Base):
"""Predict sentiment scores using a linear Support Vector Machine (SVM).
Uses a sklearn pipeline.
"""
def __init__(self, model_file: str=None) -> None:
    super().__init__()
    # pip install sklearn
    from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer, TfidfVectorizer
    from sklearn.linear_model import SGDClassifier
    from sklearn.svm import LinearSVC
    from sklearn.pipeline import Pipeline
    self.pipeline = Pipeline(
        [
            #('vect', CountVectorizer()),
            #('tfidf', TfidfTransformer()),
            ('tfidf', TfidfVectorizer()),
            ('clf', LinearSVC(

            #('clf', SGDClassifier(
              #  loss='hinge',
               # penalty='l2',
                #alpha=1e-4,
                #random_state=42,
                #max_iter=100,
                #learning_rate='optimal',
                #tol=None,

            )),
        ]
    )

def predict(self, train_file: str, test_file: str, lower_case: bool) -> pd.DataFrame:
    "Train model using sklearn pipeline"
    from sklearn.model_selection import GridSearchCV
    from sklearn.svm import SVC
    train_df = self.read_data(train_file, lower_case)
    parameters = {'C': [1, 10],
                  'gamma': [0.001, 0.01, 1]}
    model = SVC()
    grid = GridSearchCV(estimator=model, param_grid=parameters)
    grid.fit(train_df['text'], train_df['truth'])
    print(grid)
    # summarize the results of the grid search
    print(grid.best_params_)
    print(grid.best_estimator_)

    learner = self.pipeline.fit(train_df['text'], train_df['truth'])
    # Fit the learner to the test data
    test_df = self.read_data(test_file, lower_case)

    test_df['pred'] = learner.predict(test_df['text'])
    return test_df

However, I'm having some issues as it raised some ValueErrors: ValueError: could not convert string to floatand I don't know how to deal with this information. And also:  FitFailedWarning: Estimator fit failed. The score on this train-test partition for these parameters will be set to nan. How can I solve these?

The problem here is that the data train_df or test_df where ever the error is coming has a column which has a string(object) data type. you can't fit string data into a model. Only numerical data is allowed.

try printing
print(train_df.info())
print(test_df.info())

to see which column should be float type but is a string.


Using z-score for neural network normalization I've read many people use z-score to normalize their data for presenting to a neural net, and that all data should lie in a range (usually -1 to 1), but z-score can return results beyond those bounds.

So, I guess I'm right in saying the feature data then needs to be scaled (using min-max?), after calculating z-score?

One of the reasons z-scores are useful in neural nets is that they can allow gradient descent to converge faster. I don't think taking these z-scores and scaling them further to be between -1 and 1 will add any extra benefit, as you already have your features expressed on a common scale.


PrefixSpan model input RDD format I am trying to understand the input for the PrefixSpan model (pyspark). I thought in the input RDD, each row should be a list. However, as I looked into the source code, each row is actually a list of list. Why does it need a list of list instead of just list? Thanks!

https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala

 def trainPrefixSpanModel(
      data: JavaRDD[java.util.ArrayList[java.util.ArrayList[Any]]],
      minSupport: Double,
      maxPatternLength: Int,
      localProjDBSize: Int ): PrefixSpanModelWrapper = {...}

You’ve asked your question a while ago, but maybe it helps someone anyway.

In the mllib guide they say to refer to the PrefixSpan paper by Jian Pei and his team, there’s another version fully available here

There, they explain that a sequence is composed of elements which are composed of items:
For example “a sequence (a (abc)(ac)d(cf)) has five elements: a , (abc), (ac) , d and (cf)  where items a and c appear more
than once respectively in different elements.” (taken from the paper)

That's why you need a list of lists.

And note that items within an element shouldn’t follow any particular order (in the paper they are assumed to be sorted alphabetically).  So if you are dealing with time series I suppose that the events arriving at the same time (so no particular order) would form the items of an element.


Proof of the connection between V and Q in Reinforcement Learning I've been studying some basics RL in the SpinningUp materials.
Is there any mathematical proof that $V^\pi(s) = E_{a \sim \pi} [Q^\pi(s, a)|s_0 = s]$ ?

It follows very simply from the definitions of V and Q.
These are the relevant definitions:
$$G_t \space \dot{=} \space \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}$$
$$V^{\pi}(s) \space \dot{=} \space \mathbb{E}_{A \sim \pi}[G_t|S_t=s]$$
$$Q^{\pi}(s, a) \space \dot{=} \space \mathbb{E}_{A \sim \pi}[G_t|S_t=s, A_t=a]$$
You can ignore the definiton of return, $G_t$, I included it for completeness, and to simplify the other definitions.
The only difference between the definitions of V and Q is adding the selection of $a$ in the conditions. If you remove the action selection in the definition of Q, and condition it instead on following the policy, you are back to the definition of V.
You can also expand the expectation in the definition of V using the policy function to show the same thing:
$$V^{\pi}(s) = \mathbb{E}_{A \sim \pi}[G_t|S_t=s]$$
$$V^{\pi}(s) = \sum_{a}\pi(a|s) \mathbb{E}_{A \sim \pi}[G_t|S_t=s, A_t=a]$$
$$V^{\pi}(s) = \sum_{a}\pi(a|s) Q^{\pi}(s,a)$$
Convert this back to an expectation over the policy:
$$V^{\pi}(s) = \mathbb{E}_{A \sim \pi}[Q^{\pi}(S_t,A_t)|S_t=s]$$


Use spark_csv inside Jupyter and using Python My ultimate goal is to use Jupyter together with Python for data analysis using Spark. The current hurdle I face is loading the external spark_csv library. I am using Mac OS and Anaconda as the Python distribution.

In particular, the following:

from pyspark import SparkContext
sc = SparkContext('local', 'pyspark')
sqlContext = SQLContext(sc)
df = sqlContext.read.format('com.databricks.spark.csv').options(header='true').load('file.csv')
df.show()

when invoked from Jupyter yields:

Py4JJavaError: An error occurred while calling o22.load.
: java.lang.ClassNotFoundException: Failed to find data source: com.databricks.spark.csv. Please find packages at http://spark-packages.org

Here are more details:

Setting Spark together with Jupyter

I managed to set up Spark/PySpark in Jupyter/IPython (using Python 3.x).

System initial setting

On my OS X I installed Python using Anaconda. The default version of Python I have currently installed is 3.4.4 (Anaconda 2.4.0). Note, that I also have installed also 2.x version of Python using conda create -n python2 python=2.7.

Installing Spark

This is actually the simplest step; download the latest binaries into ~/Applications or some other directory of your choice. Next, untar the archive tar -xzf spark-X.Y.Z-bin-hadoopX.Y.tgz.
For easy access to Spark create a symbolic link to the Spark:

ln -s ~/Applications/spark-X.Y.Z-bin-hadoopX.Y ~/Applications/spark

Lastly, add the Spark symbolic link to the PATH:

export SPARK_HOME=~/Applications/spark
export PATH=$SPARK_HOME/bin:$PATH

You can now run Spark/PySpark locally: simply invoke spark-shell or pyspark.

Setting Jupyter

In order to use Spark from within a Jupyter notebook, prepand the following to PYTHONPATH:

export PYTHONPATH=$SPARKHOME/python/lib/py4j-0.8.2.1-src.zip:$SPARKHOME/python/:$PYTHONPATH

Further details can be found here.

Assuming the rest of your configuration is correct all you have to do is to make spark-csv jar available to your program. There are a few ways you can achieve this:

manually download required jars including  spark-csv and csv parser (for example org.apache.commons.commons-csv) and put them somewhere on the CLASSPATH.
using --packages option (use Scala version which has been used to build Spark. Pre-built versions use 2.10):

using PYSPARK_SUBMIT_ARGS environmental variable:

export PACKAGES="com.databricks:spark-csv_2.11:1.3.0"
export PYSPARK_SUBMIT_ARGS="--packages ${PACKAGES} pyspark-shell"

adding Gradle string to spark.jars.packages in conf/spark-defaults.conf:

spark.jars.packages    com.databricks:spark-csv_2.11:1.3.0


Why don't different output weights break symmetry? My deep learning lecturer told us that if a hidden node has identical input weights to another, then the weights will remain the same over the training/there will be no separation. This is confusing to me, because my expectation was that you should only need the input or output weights to be different. Even if the input nodes are identical, the output weights would affect the backpropagated gradient at the hidden nodes and create divergence. Why isn't this the case?

It is fine if a hidden node has identical initial weights with nodes in a different layer, which is what I assume you mean by output weights. The problem with weight-symmetry arises when nodes within the same layer that are connected to the same inputs with the same activation function are initialized identically.

To see this, the output of a node $i$ within a hidden layer is given by
$$\alpha_i = \sigma(W_i^{T}x + b) $$
where $\sigma$ is the activation function, $W$ is the weight matrix, $x$ is input, $b$ is bias.

If the weights $W_{i}=W_{j}$ are identical for nodes $i,j$ (note that bias is typically initialized to 0), then $\alpha_i = \alpha_j$ and the backpropagation pass will update both nodes identically.


How to calculate a weighted Hierarchical clustering in Orange I am doing my first cluster analysis with Orange (which I recently discovered and looks promising for this iterative and interactive process).

Apparently, there are several methods of creating clusters based on distance algorithm:

Single linkage (which computes the distance between the closest elements of the two clusters)
Average linkage (which computes the average distance between elements of the two clusters)
Complete linkage (which computes the distance between the clusters’ most distant elements)
Weighted linkage
Ward

Since I have several columns, and some of them are more important than others in terms of defining clusters, it seems to me that using weighted linkage method may be what I am looking for. Unfortunately, I don't know how to do that, since I didn't find a way to assign a weight to each column.

To make things worse, I have only found an explanation about the first three on this Orange's blog post but nothing about Weighted linkage  (nor Ward, which may be a recent addition, since it is not even mentioned on widget's help).

Am I on the right path to achieve what I am looking for? Is there any way to make some columns more or less important/definitory when calculating the distances?

Weighted linkage probably does not mean you get to specify weights of features (build the distance matrix yourself!)

Instead this most likely refers to the well-known weighted group average strategy you will find in most textbooks often called WPGMA. There are two different definitions of "average", so this is likely simply the "other" average linkage.


Effect of NOT changing filter weights of CNN during backprop What is the effect of NOT changing filter weights of a CNN during backpropagation? I changed only the fully connected layer weights while training on the MNIST dataset and still achieved almost 99 percent accuracy.

By not changing the weights of the convolutional layers of a CNN, you are essentially feeding your classifier (the fully connected layer) random features (i.e. not the optimal features for the classification task at hand).

MNIST is an easy enough image classification task that you can pretty much feed the input pixels to a classifier without any feature extraction and it will still score in the high 90s. Besides that, perhaps the pooling layers help a bit...

Try training an MLP (without the conv/pool layers) on the input image and see how it ranks. Here is an example where an MLP (1 hidden & 1 output layer) reached 98+% without any preprocessing/feature extraction.

Edit:

I'd also like to point out to another answer I wrote, which goes into more detail on why MNIST is so easy as an image classification task.


Which Machine Learning methods would be good for a beginner-level Data Science project for described dataset? I want to do my first  beginner-level project regarding Data Science. I picked my data set which is described below. I want to make a Python script which will make some predictions on our data set using training and test data. That could be for example a prediction which countries will see rise in number of customers and which ones will see a decline, but that's just the first idea that popped in my head. I want to implement a few Machine Learning methods to do the same prediction on our data set and compare their results. Finally comes my question - which Machine Learning methods would you suggest for a beginner Data Science project with this data set which is described below? Thank you in advance for any suggestions. I don't want to pick something very hard as the beginner and I also don't want to pick a method that just doesn't "fit" my data set (because it's rather simple).

EDIT: To specify this project will concern business predictions based probably on time-series. I'm looking for recommendations for which ML methods are easy to implement use and understand for the beginner. Project will be written in Python.

Data Set Description:

Number of Attributes : 8

Invoice : Integer

StockCode : String

Description: String

Quantity : Integer

InvoiceDate : Timestamp

Price : Decimal

CustomerID : Integer

Country : String

Number of records: 541910

Date ranges: 01/12/2010 08:26 - 09/12/2011 12:50

Price ranges: -11062.06 - 38970.00

Number of distinct Countries: 38

That doesn't look at all like a simple dataset for a beginner's project:

First if you want to predict future trends you need to look at time series, which is not the easiest type of method. Additionally our data covers only 2 years, it might be too short for good time predictions.
Your dataset is made of various types of attributes so it will probably require some serious preprocessing work (e.g. description string) in order to make the data usable for a particular task.
Start with plotting your data: volume of sales by country, number of customer by country, most common products, this kind of thing.


Predicting a Continuous output in a dataset with categories Let's say this:
I'm working on a machine learning project and I'm working on a dataset with a 4250,13 as shape and it is already grouped in 7 categories!
Note that those categories can't be considered as predictor.
Here is how my data is grouped in the categories feature 'FAC':

FSEG       32.852598 % of the dataset
FSTA       19.151644 % of the dataset
FD       19.003181 % of the dataset
FM      16.076352 % of the dataset
FT      5.132556 % of the dataset
PPSE       4.814422 % of the dataset
FSDC    2.969247 % of the dataset

I have a continuous output that I want to predict 'CGPA', so the task is a regression, my goal is to predict it in each category, and the final decision will be the category where the predicted output is maximized.

My approach to deal with this problem is to sample my dataset into 7 sub-datasets and train the model in all those 7 datasets. And for a new input, predict the output in each category, and the final category will be where the predicted output is maximal.

Now I want to know, is there any way to do it in one dataset and automatically predict the category where my output is max? With a single model instead of 7?

PS: I'm using python and scikit learn

Sound like random-forest but not sure that is it ...
can someone help? Any help will be appreciated.

Here is my dataset with samples collected in each category. My final output is called CGPA and the category label is 'FAC'.

A random forest will work, however standard regression will also work with categorical variables as predictors. You will have to "one-hot" encode your categorical predictors into 6 "dummy" variables (classes-1 = 7-1 = 6). The first dummy variable will encode 0/1 for whether or not the observation is class A, second dummy variable as 0/1 for class B, etc. You only need 6 dummy variables because if all of them are 0 for a given observation, that means the observation is in group 7 (G).

In some languages, such as R, the regression command will automatically do this one-hot conversion for you. For python, the pandas package can do this for you with pd.get_dummies(data).


RGB Image Segmentation using Clustering I want to apply some segmentation on a dataset for preprocessing purposes. I have tried the "otsu thresholding" approach in order to segment the image. It's a good method, however, I think a clustering algorithm such as K-Means can be even more succesful for the basic segmentation. Here is some problem with Otsu Thresholding:

As you can see, in some samples, Otsu thresholding may fail like in the picture 1. I think that, using a pixel-wise positioning approach as well as color thresholding may increase the success even further.

However, I couldn't manage to work with scikit-learn's KMeans function to work with an RGB image, since it is a 3 channel 2-d matrix(basically 3d matrix).

How can I combine both pixel-wise distance approach as well as color clustering with K-Means clustering or alike method?

You will want to implement this yourself.

Encoding the data as 5 dimensional vectors per pixel crates a substantial memory overhead. But if you can try that first, to see if the results are as good as you believe then to be...
Pay attention to scaling!


How the embedding layer is trained in Keras Embedding layer How is the embedding layer trained in Keras Embedding layer?
(say using tensorflow backend, meaning is it similar to word2vec, glove or fasttext)

Assume we do not use a pretrained embedding.

Both the answers are wrong.
An embedding layer is a trainable layer that contains 1 embedding matrix, which is two dimensional, in one axis the number of unique values the categorical input can take (for example 26 in the case of lower case alphabet) and on the other axis the dimensionality of your embedding space. The role of the embedding layer is to map a category into a dense space in a way that is useful for the task at hand, at least in a supervised task. This usually means there is some semantic value in the embedding vectors and categories that are close in this space will be close in meaning for the task.
This is related to one-hot encoding in a sense that it maps a discrete category into a vector feature representation. You could still do this for a neural network but if you use this in a dense layer you would create an enormous amount of weights of which most of them are not used regularly. Putting an embedding layer in between reduces the amount of learnable weights before feeding them to interact with other parts of your input. Another advantage is that the embedding matrix basically works as a lookup table, so you can really use the sparsity of the index of your category to look up what the current value of the embedding is and when applying backpropagating only adapting that entry of the weight matrix.
This blog post explains clearly about How the embedding layer is trained in Keras Embedding layer. Hope this helps.


Basic Time Series Classification Examples I've been using matlab until now to classify a large number of labelled time series I have. This has been relatively successful but I'd like to try using Tensorflow to apply a Deep Learning paradigm instead.

I'm a complete noob at this and so I'm a bit overwhelmed with the literature as I'm struggling to generalise examples such as the 0-9 digits classification examples to my problem.

My current code reads in the 1064 time series (length 3125), reads in the labels, converts the labels to onehot_encoding and extracts training and validation sets.

#Not all code included
def read_data():
    #Get labels from the labels.txt file
    labels = pd.read_csv('labels.txt', header = None)
    labels = labels.values

    data = pd.read_csv('ts.txt',header = None)

    return data, labels

data, labels = read_data()

# Default split is 75% train 25% test
ts_train, ts_test, labels_train, labels_test = train_test_split(data,labels)

onehot_encoder = OneHotEncoder(sparse=False)
labels_train = onehot_encoder.fit_transform(labels_train)
labels_test = onehot_encoder.fit_transform(labels_test)

#Construct the graph
batch_size = 100
seq_len = 3125
learning_rate = 0.0001
epochs = 1000

But now I need to construct the graph and I'm a bit lost. If someone could link me some useful examples I'd be very grateful thank you.

What you need to learn are RNNs and LSTMs. Here are some links I used to learn these:

CS231n RNN+LSTM lecture
Understanding LSTMs

Also I would suggest you to use Keras, a Tensorflow API. In my experience, it makes working with RNNs and LSTMs way easier, if you're a beginner.

I found these articles which seemed related to your problem:

Time series classification project by naveen sai on github
https://aboveintelligent.com/time-series-analysis-using-recurrent-neural-networks-lstm-33817fa4c47a
https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/

Keywords to search: Time series classification tensorflow keras


Data transformations in hierarchical classification I am building a hierarchical text classifier using the Local Classifier Per Parent Node (LCPN) approach with the 'siblings' policy
as described in the A survey of hierarchical classification across different application domains:
E.g. if we have the classes 1.1, 1.2, 2.1, 2.2, 2.3 then in the first level we use all the training set to train a classifier to distinguish between class 1 (1.1,1.2) and 2 (2.1,2.2,2.3),
at the second level we use two multiclass classifier the first one to classify between 1.1 and 1.2 using as training set only the data belonging to these classes and the second classifier for the rest.
Should any data transformation (e.g. scaling, tf-idf) that we do to the data happen at each level of the classifier?
I.e. since at the first level the tf-idf vectors are created by fitting to the whole training set, can we use them at the second level or should we fit to the new training subsets?

It depends on the dataset, but generally fit again

why? If you dont fit again on the second level when classifying 1.1 and 1.2 you are introducing bias that you got from the first level when you classified between classes 1 and 2.

why it depends? if information is intertwined between all of the parent and children classes and you will use these models again in the future, you could be loosing important information when fitting again, in other words you will be only over-fitting on the current train (classify 1.1 1.2)


Keras multi-label time-series classification considering time-series as an input image vector I am trying to build a multi-class classifier using Keras. I am not quite sure I have implemented it correctly.
Data is like this

label time-series variables [0:25728}

index 0  1   2   3   4            25728
  0   1  2.5 3.2 1.6 1.05 ........ 2.54
  1   5  3.2 1.6 1.5 1.49 ........ 1.41
  2   1  2.3 3.2 1.5 1.52 ........ 2.11
  3   3  0.2 3.1 1.5 1.89 ........ 0.81
  4   8  1.2 1.1 0.2 1.19 ........ 3.71
  .   5  .    .   .   .   ........   .
  .   7  .    .   .   .   ........   .
1323  5  .    .   .   .   ........   .

Here is the code.
I split data by 68 % then reshaping 1D array to a 2D array.
as 384*67 = 25728
So forming an image of vector 384 by 67 for one label

def readucr(filename):
data = np.loadtxt(filename, delimiter=',')
Y = data[:, 0]
X = data[:, 1:]
return X, Y

x_train, a = readucr(path+'p2_TRAIN')
x_test, b = readucr(path+'p2_TEST')
df_train_y = pd.read_csv(path+'p2_TRAIN',header=None)
df_test_y = pd.read_csv(path+'p2_TEST',header=None)

x_train = x_train[:,0:25728]
x_test = x_test[:,0:25728]

scaler = MinMaxScaler(feature_range=(0, 1))
x_train = scaler.fit_transform(x_train)
x_test = scaler.fit_transform(x_test)

x_train =x_train.reshape(x_train.shape[0],384,67)
x_test =x_test.reshape(x_test.shape[0],384,67)

train_label_y = df_train_y[0].values
test_label_y = df_test_y[0].values
batch_size = min(x_train.shape[0] / 10, 10)

y_train = np_utils.to_categorical(train_label_y)
y_test = np_utils.to_categorical(test_label_y)

x_train = x_train.reshape(x_train.shape + (1,))
x_test = x_test.reshape(x_test.shape + (1,))

input_shape = x_train.shape[1:]
model = Sequential()

model.add(Conv2D(32, kernel_size=(3, 3), padding='same',
                 input_shape=input_shape))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(64, kernel_size=(3, 3), padding='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(128, kernel_size=(3, 3), padding='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(8, kernel_size=(3, 3), padding='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(GlobalAveragePooling2D())
model.add(Dense(9, activation='softmax'))

optimizer = keras.optimizers.Adam()
model.compile(loss='categorical_crossentropy',
              optimizer=optimizer,
              metrics=['accuracy'])

hist = model.fit(x_train, y_train, batch_size=batch_size, epochs=nb_epochs, verbose=1)
score = model.evaluate(x_test, Y_test)
print("Accuracy: %.2f%%" % (score[1] * 100))

It gives 96.16% accuracy but I don't believe it is true.
I want to predict the labels.

How can I predict labels?
What I am doing wrong?

Please help! Thank you.

model.predict(X)

Will return an array of probabilities across your classes, the effective discrete prediction can be achieved by:

np.argmax(model.predict(X))

Accuracy is only one measure of the fit of a classifier, and can be strongly affected by properties of the evaluation set such as class imbalance. Does one class represent a large proportion of your observations? If so, this accuracy measure could be misleading you as to its performance.

Otherwise, with the predictions gathered as shown above, you can further analyze the performance of your network using confusion matrices, muticlass log loss (cross-entropy) and/or others.


How do I compare coefficients from my glm when I have more than one factor variable in my formula? I am trying to model a binary outcome in R that has many independent variables. 5 of the Ivs are factors with more than two levels. When I try to remove the intercept it only does it for one of the independent variables. I would like to see if any of the coefficients for the levels are significantly different from any of the others within that variable(and it would be nice to see if any of the levels are significantly different from the average). When I do a summary of the glm it only shows the values of the coefficient as it relates to the first level(Except for the first IV).
To recreate it I have the R code for something similar using mtcars:

md <- datasets::mtcars
md <- mutate(md, ncyl = ifelse(cyl==6,"Six Cyl", cyl))
md <- mutate(md, ncyl = ifelse(cyl==4,"Four Cyl",ncyl))
md <- mutate(md, ncyl = ifelse(cyl==8,"Eight Cyl",ncyl))
md <- mutate(md, wgear = ifelse(gear==3,"Three G",gear))
md <- mutate(md, wgear = ifelse(wgear==4,"Four G",wgear))
md <- mutate(md, wgear = ifelse(wgear==5,"Five G",wgear))
md <- mutate(md, wcarb = ifelse(carb==1,"One Carb",carb))
md <- mutate(md, wcarb = ifelse(carb==2,"Two Carb",wcarb))
md <- mutate(md, wcarb = ifelse(carb==3,"Three Carb",wcarb))
md <- mutate(md, wcarb = ifelse(carb==4,"Four Carb",wcarb))
md <- mutate(md, wcarb = ifelse(carb==6,"Six Carb",wcarb))
md <- mutate(md, wcarb = ifelse(carb==8,"Eight Carb",wcarb))
md <- md[,-c(2,10,11)]
model <- glm(mpg~., data = md)
summary(model)
modelint <- glm(mpg~.-1, data = md)
summary(modelint)

So for summary(modelint) I can see the values for the coefficients of all 3 possible values ncyl, and with confint(modelint) I can at least see confidence intervals for all of the possible values. But wgear and wcarb are just compared to their first level alphabetically and the t values are just if it is different from the first level. I know if I just did mpg~wgear-1 I could see their comparisons, but I think this does not take into account all of the other variables. Ideally I would like to see how 1 carb compares to two carb, three, four,...  and 1 carb vs not 1 carb while taking into account the other variables. Is there a better function to use than glm? Any advice would be greatly appreciated!

R chooses the baseline level by itself unless you specify it. You can read here how to specify it:
https://stackoverflow.com/questions/3872070/how-to-force-r-to-use-a-specified-factor-level-as-reference-in-a-regression

Then you can change the reference (baseline) level to get all the possible results. This is easy to do but I am not sure if it is the best way to go, since you need to repeat yourself many times. But I think this might be better if you could apply it on your own data:

https://stats.stackexchange.com/questions/60352/comparing-levels-of-factors-after-a-glm-in-r

This is the link to the R package documentation:
https://cran.r-project.org/web/packages/multcomp/multcomp.pdf


Training a Siamese Neural Network for object similarity assessment I am training a Siamese neural network with pairs of similar and dissimilar objects. The features of the objects are binary data on whether they contain some properties or not (2048 features per object).

I then split my dataset into training, validation and test set (60:20:20). Afterwards, I prepared the dataset myself by pairing up at random the objects accordingly yielding 50% similar and 50% dissimilar pairs and I augment the data in the training set by generating extra pairs by random (resulting in 100,000 different instances, again a balanced dataset (50:50), vs. 1,000 instances for the validation set). I then proceed to train the Siamese network and end up estimating the cosine distance between the two outputs to get a similarity metric which is compared to my label with the binary cross entropy loss function. The learning rate used is low (lr = 0.0001) and I am using the Adam optimiser. I have tried producing really small batches (batch_size = 25), adding dropout and increasing the number of instances to avoid overfitting, but the model does not seem to generalise well regardless (see picture). I was wondering if anyone could give me any hint on what is going on - and also why is it that such bumps can be appreciated during the learning process -.

without going into architecture that looks reasonable but always can be updated: Weights are dominantly updated for the negative pairs. why?

In the train set where you have (approx) 10 000 times more negative samples, you are training on them and letting the network learn almost only negative pairs, but then you test it on validation and you get what you would expect. No improvement on accuracy or loss on validation set. Make sure you are more balanced (down sample train or upsample valid/test)

EDIT: given new ifo here is a couple of suggestions.

Indication is overffiting obviously

Early Stopping as soon as validation dataset reaches minimum STOP training. ITs overkill after that
Covariate-shift make sure that patterns are similiar in all three sets (I assume they are but just in case)
Reduce complexity, how deep is your architecture, maybe you are over doing it. If you took some random Siamese network architecture from online it may be too much (they used it for images for example where you need additional complexity)


Predicting app usage on mobile phone I'm currently building an app that strives to predict how the users uses different apps and give the user a suggestion based on which apps it think the user will currently use (a ranked list based on the user's current conditions). I've been collecting some data over the past week now, and I'm not really sure which approach to take. I've been thinking about using multiple seasonalities (correct me if I'm using the wrong terminology) such as time of day, day of week, week of month, month and quarter. I also want to use location, and other sensor data (such as the user state "walking" or "sitting" later on).

I've summarised the usage over the last week on an hourly level for some apps.
The bars represents each time the app was opened during that time, and the green line is a weighted moving average which has a weight of 0.5 on its closest neighbours.

Now I see several challenges in front of me and would greatly appreciate some input from others, or some good resources to find further information.

Do you think my model is a good one for this problem
How do I account for ageing data?
How do add up the different seasonalities/states/location? Multiply them?
Does it make sense smoothening the curve as I do?

Here some data for the last week on an hourly basis:

So you have collected data that shows which app is being used at any time, binned into hours in the day. And you have several apps. You mention other dimensions, like user state when using the app (walking, not-walking), (active, not-active - it appears to me that you are not collecting much usage 2-6. is that because the usage is from a self-directed ping from an app while the user is truly away?), location (is this going to be all possible values, or are you going to use something like the fact that this location has been seen often before?). Another interesting relationship could be pairing apps, i.e., mining for a relationship between App A being used after using App B or before App B.

Regardless, then you will definitely have many different dimensions upon which to measure the usage characteristics for any particular usage measurement, and so you are definitely going to have a multiple dimensional problem. You might try to visualize this as an N-space problem, with an axis of measurement for each of your characteristics. Each of your previous measurements represents vectors and you are producing a new vector with your next measurement.

From this, you want to predict future behavior based on measuring the input characteristics from your usage space. You could go for something that classifies as nearest neighbor, and you probably want to do this for your first stab at the problem. You might end up wanting to make the predictive model more sophisticated by adding probabilities to the classifier and acting on that. This means getting estimates of class membership probability rather than just simple classifications. But I would build the whole thing incrementally. Start simple and add complexity as you require it. The increased complexity will also have effects on performance, so why not baseline with something.

For the aging of data, are you wanting to reduce the predictive power of characteristics that are too long in the tooth? If so, be explicit with yourself about what that means, quantitively. Do I trust the usage data from last month less than yesterday's data? Perhaps so, but then why? is my usage different because I am different or because last month was special compared to yesterday, or vice-versa? Again, you might benefit from ignoring this at first, but then trying to search for "seasonal" or periodicity characteristics from the data. Once you determine if/how it changes, you can weight that contribution compared to your immediate usage in different ways. Perhaps you want to amplify the contribution of a similar period (same time of day && same location && same previous app usage). Perhaps you want to provide an exponential dampening on historical data because the usage is always adapting and changing, and recent usage seems to be a much better predictor than 3xcurrent.

For all of this, the proper data science perspective is to let the data lead you.


How fit pairwise ranking models in xgBoost? As far as I know, to train learning to rank models, you need to have three things in the dataset:

label or relevance
group or query id
feature vector

For example, the Microsoft Learning to Rank dataset uses this format (label, group id and features).

1 qid:10 1:0.031310 2:0.666667 ...
0 qid:10 1:0.078682 2:0.166667 ...

I am trying out xgBoost that utilizes GBMs to do pairwise ranking. They have an example for a ranking task that uses the C++ program to learn on the Microsoft dataset like above.

However, I am using their Python wrapper and cannot seem to find where I can input the group id (qid above). I can train the model using just the features and relevance scores but I feel like I am missing something.

Here is a sample script.

gbm = XGBRegressor(objective="rank:pairwise")

X =  np.random.normal(0, 1, 1000).reshape(100, 10)
y = np.random.randint(0, 5, 100)

gbm.fit(X, y) ### --- no group id needed???

print gbm.predict(X)

# should be in reverse order of relevance score
print y[gbm.predict_proba(X)[:, 1].argsort()][::-1]

According to the XGBoost documentation, XGboost expects:

the examples of a same group to be consecutive examples,
a list with the size of each group (which you can set with set_group method of DMatrix in Python).


Connect a dense layer to a LSTM architecture I am trying to implement an LSTM structure in plain numpy for didactic reason. I clearly understand how to input the data, but not how to output.
Suppose I give as inputs a tensor of dimension (n, b, d) where:
    •   n is the length of the sequence
    •   b is the batch size (timestamps in my case)
    •   d the number of features for each example
Each example (row) in the dataset is labelled 0-1. However, when I fed the data to the LSTM, I obtain as a result the hidden state h_out which has the same dimension of the hidden size of the network. How can I obtain just a number that can be compared to my labels and properly backpropagated?
I read that someone implements another dense layer on top of the LSTM, but it's not clear to me the dimensions that such layer and its weight matrix should have.

What you are getting as the output is the internal LSTM state.
In order to get value comparable to your labels, add a dense layer on top of it. Output dimension of dense layer would be the number of labels you want result.

If its 0 and 1, only 1 output neuron can work along with sigmoid
If there are 5 label classes, then output dimension of dense layer should also be 5


learning rate in reinforcement learning Does anyone know how to get the learning rate from participant data?

I'm computing all the expected values for all trials (=200)

$$V_t(S) = V_{t-1}(S)+ \alpha \cdot \text{error }_t $$

$$(\text{error }_t = R_t − V_{t-1}(S))$$

and then computing alpha at each trial using

$$\alpha= (V_t(S)  - V_{t-1}(S) )/ \text{error }_t$$

but it seems not the right way!

Suggestions?

If you still have the question I hope my answer helps you. From what I understand you want to fit a RL model to participant's data who did a decision making task. The whole process of fitting RL models to data is described in great detail in Trial by Trial analysis using RL by Nathaniel Daw.

Briefly, what you observe (your data) is just actions and rewards at every time step. If the participant is using a RL model you should expect to update the values of his actions according to:

$Q_t(a_t)=Q_t(a_{t-1})+\alpha\cdot (r_t-Q_t(a_{t-1}))$

and action selection with the Boltzmann function:

$p(a_{t}|s_{t})=\frac{e^{\beta Q(s_{t},a_{t})}}{\sum_{a'}e^{\beta Q(s_{t},a')}}$

We prefer to use a stochastic policy as it simulates better human choices and it is straightforward to create a likelihood function. The likelihood function would be the product of all the probabilities $p(a_t|s_t)$. The model parameters are $[{\alpha,\beta}]$ and these can be inferred by standard Maximum Likelihood techniques.

Finally, our hypothesis (=model) can be summarized as: A subject generates a sequence of (observed) actions. To generate these actions we assumed that uses a RL model with some parameters. We want to fit the model to the data to infer these parameters and test the suitability of the model and the cognitive/psychological links of the values of the parameters.


How should I tackle this real-life hypermarket problem? I registered myself in the payback program of the hypermarket I am going to.

For every 2$ I get 1 point.
I buy the same products every week (Feta 2.19\$, Milk 0.99$, ...).
I visit only in weekdays.

I would like to maximize the amount of points I gather, while I also maximize the times I visit that hypermarket (so buying all the stuff at once is an awful solution).

How should I go about modeling this real-life situation in terms of a Data Science problem in the first place? If you feel like it, suggest an approach of tackling it as well...

Question

I feel that the question needs to be reformulated for clarity to:

  Given a fixed set of purchased products $S=\{s_i\}$ with associated prices $p_i$, find the greatest number of disjoint subsets $S_1, S_2, etc$ such that the number of reward points attained in each subset equals the number of points attained for the whole.

  A reward point is obtained for every whole number of 200 cents divisible into the total price of the set.

In other words you have a set of products that you will definitely buy then the maximal number of points you can acquire from those purchases is $floor(\frac{1}{200}\sum_i p_i)$. Since you cannot acquire more points than this any subdivisions you make, representing different trips to the hypermarket, need to reach the same amount of points.

Answer

The brute force way of considering every possible combination will be an exponential time algorithm. So you probably need to develop an algorithm which uses greedy approaches.

For example, one such approach would be to identify subsets. Say you had four products:
{milk, cheese, bread, oilves} with prices {189c, 200c, 211c, 205}, possible 4 reward points,
then a preliminary scan (over a single item) to detect prices with zero residuals will return two disjoint sets;
{cheese} and {milk, bread, olives}.
Now you can recursively treat the 3-set as the same problem, in this case no single product has a multiple of 200, but milk and bread does sum to 400 so that is a perfect split:
{cheese} and {milk. bread} and {olives}
And you have arrived at the result.

In general you cannot know how many products might be needed in a combination to make a perfect multiple of 200, but you can use some heuristics derived from your initial set. In this case $\sum_i p_i mod 200 = 805 mod 200 \equiv 5$, so you know that you cannot have the sum of residuals of subgroups totalling more than 5 otherwise your solution returns less reward points.

For example if you grouped {cheese, bread} you get 2 reward points with a residual of 11 cents. There is no combination of the remaining {milk, olives} that can collect 2 more reward points. In that case the residual is 194 cents, which actually suggests the two disjoint sets can be combined to produce an additional reward point since 11 + 194 > 200.

PS

I have no idea why you mention weekdays without giving any information about their significance to the problem. Unless you want me to assume that you only visit the hypermarket once per day in which case the total number of disjoint subsets you can have is 5.


Relu with not gradient vanishing function is possible? I'm beginner in ML.
In the ANN, relu has the gradient of 1 in x>0

how ever, i wonder in x=<0 relu has gradient of 0 and may have gradient vanishing problem
in deep neural networks.

if activation function like y=x(for the all the x) has no gradient vanishing problem,
why we dose not use this function in deep neural networks?
Is there any side effect for y=x(for all x)? (maybe, the weight may go infinity in deep neural networks...... however, I think this problem is also being happen in ReLU. so it is not a problem(I think.))

If you are using an activation like y=x, then your model is a simple linear one. Multiple layers with such activation will be equivalent/reduced to only one layer with a linear activation! Thus you can only map linear function satisfactorily with this type of model. To be able to learn complex non-linear functions, you need to use multiple layers with non-linear activation in between to make the whole model non-linear

To prevent the vanishing gradient problem, there is a variant of relu called Leaky ReLU. This activation is same as relu in the positive region of x. For negative region of x, it is a linear function with a small slope (e.g. 0.2). This makes Leaky ReLU a non linear activation at x=0 point.


What pandas function is most similar to the transpose procedure in SAS? I am trying to transpose Amt by Txn_ID across Item_Desc in Python using pandas. Which pandas function(s) is/are most similar to the transpose procedure in SAS?
Below is an example of the input dataframe and the desired output dataframe.
Input DataFrame:
| Txn_ID | Item_Desc | Amt  |
|--------|-----------|------|
| 1      | Apples    | 0.96 |
| 1      | Milk      | 2.10 |
| 2      | Eggs      | 7.00 |
| 2      | Flour     | 2.20 |
| 2      | Salt      | 4.75 |
| 3      | Eggs      | 3.50 |

Output DataFrame:
| Txn_ID | Apples | Eggs | Flour | Milk | Salt |
|--------|--------|------|-------|------|------|
| 1      | 0.96   | NaN  | NaN   | 2.10 | NaN  |
| 2      | NaN    | 7.00 | 2.20  | NaN  | 4.75 |
| 3      | NaN    | 3.50 | NaN   | NaN  | NaN  |

The pivot operator can be used to achieve the desired result. As per your example:
Define original data:
import pandas as pd
data = {'Txn_ID': [1, 1, 2, 2, 2, 3],
        'Item_Desc': ['Apples', 'Milk ', 'Eggs', 'Flour', 'Salt', 'Eggs'],
        'Amt': [0.96, 2.10, 7.00, 2.20, 4.75, 3.50]
}

Create dataframe
df = pd.DataFrame.from_dict(data)

Transpose result using the .pivot operator
df_new = df.pivot(index = 'Txn_ID', columns = 'Item_Desc')

| Txn_ID | Apples | Eggs | Flour | Milk | Salt |
|--------|--------|------|-------|------|------|
| 1      | 0.96   | NaN  | NaN   | 2.10 | NaN  |
| 2      | NaN    | 7.00 | 2.20  | NaN  | 4.75 |
| 3      | NaN    | 3.50 | NaN   | NaN  | NaN  |


What happens when the vocab size of an embedded layer is larger than the text corpus used in training? Full disclosure this question is based on following this tutorial: https://tinyurl.com/vmyj8rf8
I am trying to fully understand embedded layers in Keras. Imagine having a network to try and understand basic sentiment analysis as a binary classifier (1 positive sentiment and 0 negative sentiment). The toy dataset for this is as follows:
# Define 10 restaurant reviews
reviews =[
          'Never coming back!',
          'horrible service',
          'rude waitress',
          'cold food',
          'horrible food!',
          'awesome',
          'awesome services!',
          'rocks',
          'poor work',
          'couldn\'t have done better'
]#Define labels
labels = array([1,1,1,1,1,0,0,0,0,0])

This data can be used to train a really simple network as follows:
Vocab_size = 50
model = Sequential()
embedding_layer = Embedding(input_dim=Vocab_size,output_dim=8,input_length=max_length)
model.add(embedding_layer)
model.add(Flatten())
model.add(Dense(1,activation='sigmoid'))
model.compile(optimizer='adam',loss='binary_crossentropy',metrics=['acc'])
print(model.summary())

In order to feed this data into he network, we can one hot encode it using Keras one_hot as follows:
encoded_reviews = [one_hot(d,Vocab_size) for d in reviews]
print(f'encoded reviews: {encoded_reviews}')

We get the following output:
encoded reviews: [[14, 45, 43], [8, 2], [6, 43], [24, 1], [8, 1], [11], [11, 21], [16], [34, 40], [2, 25, 36, 15]]

I understand that the purpose of setting Vocab_size = 50, even though there are only around 20 unique words in the corpus is to give a large enough hashing space for the hashing algorithm behind one_hot to avoid collisions when the text is encoded.
If I train the model on these words (assume fixed length input and padding) and then get the weights of the embedded layer:
print(embedding_layer.get_weights()[0].shape)

(50, 8)

We can see this it is an array of 50 vectors that look like this as an example:
[ 0.17051394  0.13659576 -0.05245572 -0.12567708  0.06743167  0.05893507
-0.14506021  0.06448647]
My understanding is that each of these vectors corresponds to a word embedding for each word in the corpus. But if there are only 20 unique words in the corpus and Vocab_size is set larger than this then that can't be completely true? If Vocab_size > corpus_vocab_size, then what do these embeddings represent? Any help would be appreciated.

tf.keras.layers.Embedding(..., embeddings_initializer="uniform"*,..., *kwargs)

All the weights are initialized with the init strategy
All learn the optimum values  with the backprop
Weights for which there is no input will have zero output every time, hence no learning.
Hence these extra weights will remain at their initialization value

You may check these extra weights before and after.
weight = model.layers[0].get_weights() # Save before training
history = model.fit(x, y)

# These two should be same
weight[0][-1]  # Last weight - Before
model.layers[0].get_weights()[0][-1] # Last weight - After


Is there any package in python that can identify similarity between alphanumeric alias names of a parameter? For example: for a parameter like input voltage,
     Alias names : V_INPUT, VIN etc.

Now, I want the software to recognize each of the alias names as same. Is there any package/method by which I can achieve this?
Nltk is only allowing for dictionary words.

If you know there are only specific variants, you can obviously make a look-up table yourself (i.e. a Python dictionary).
Otherwise you could try using a fuzzy matching library, like fuzzywuzzy.
This will give you a "closeness" score for your search term, based on your list of parameters (measurements). Here is an example of how you could use it:
In [1]: from fuzzywuzzy import process

In [2]: measurements = ["Voltage", "Current", "Resistance", "Power"]

In [3]: variants = ["VOLT", "voltage_in", "resistnce", "pwr", "amps"] # notice typos etc.

In [4]: for variant in variants:
   ...:     results = process.extract(variant, measurements, limit=2)
   ...:     print(f"{variant:<11} -> {results}")  # See which two were found to be closest
   ...:     best = results[0]                     # Take the best match by score (first in the list)
   ...:     if best[1] < 70:                      # Set a threshold at 70%
   ...:         print(f"Rejected best match for '{variant}': {best}")

VOLT        -> [('Voltage', 90), ('Current', 22)]
voltage_in  -> [('Voltage', 82), ('Resistance', 30)]
resistnce   -> [('Resistance', 95), ('Current', 38)]
pwr         -> [('Power', 75), ('Current', 30)]
amps        -> [('Voltage', 26), ('Resistance', 22)]
Rejected best match for 'amps': ('Voltage', 26)

So most worked out pretty well, including the typo example.
Obviously this does not kind of semantic search, as so amps do not get related to Current in any way.

To go the way of semantic encodings, you might want to look into "word embeddings", which do indeed try to match the real meaning of words, based on their semantic meaning. To start here, you could look into Word2Vec or GloVe` embeddings. Perhaps there is even a tool or library that already offers this capability.
These approaches will not inherently deal with things like typos, so for best results, you could even combine the two approaches.


how to use tensorflow graphs in multithread？valueerror：tensor a must be from the same graph as tensor b！ I am doing instance detect and image retrieval task by Keras and Tensorflow as backend.

I plan to use multi thread to load two model, I load maskrcnn in a thread and load mobile net in another one.

I load the maskrcnn in a thread successfully, but I failed to load mobile net in another thread, and:

  show:ValueError: tensor a must be from the same graph as tensor b.

The code is as below:

Merge.py

from keras import backend as K

g1=tf.Graph()
g2=tf.Graph()
sess1=tf.Session(graph=g1)
sess2=tf.Session(graph=g2)

def intiMaskrcnn():
     with g1.as_default():
           with sess1.as_default():
                   Model1=........

def instanceDetect():
       K.set_session(sess1)
       with g1.as_default():
             Model1.predit()
             ............

def intiMobilenet():
     with g2.as_default():
           with sess2.as_default():
                   Model2=........

def Retrieval():
       K.set_session(sess2)
       with g2.as_default():
             Model2.predit()
             ............

Thread1.py

intiMaskrcnn()
instanceDetect()

Thread2.py

intiMobilenet()
Retrieval()

Where am I going wrong?

TensorFlow stores all operations on an operational graph. This graph defines what functions output to where, and it links it all together so that it can follow the steps you have set up in the graph to produce your final output. If you try to input a Tensor or operation on one graph into a Tensor or operation on another graph it will fail. Everything must be on the same execution graph.

Try to use

tf.reset_default_graph()

to reset the tf graph once it executes.

For keras use K.clear_session().

For more details about the keras and tensorflow interface view Keras documentation blog.https://blog.keras.io/keras-as-a-simplified-interface-to-tensorflow-tutorial.html this helps you a lot.


Projecting data from $S^n$ to $S^2$ I have few points in $S^n$, i.e., the $n$-dimensional unit sphere embedded in $\mathbb{R}^{n+1}$, and I would like to project them down to $S^2$, i.e., the 2-dimensional unit sphere (embedded in $\mathbb{R}^3$) to visualize it with the constraint that neighboring points should be close by. I spent some time playing with t-sne but of course, the points no longer lie on $S^2$. I normalized the projections but that introduces weird distortions, for instance, if the variance of one dataset is very small in $S^n$ as compared to other, I expect the same to hold in their $S^2$ projections; that is not the case upon normalizing t-sne. Any ideas? I would really like something that makes the previous statement hold.

I would view my points as elements of $\mathbb{R}^{n+1}$ and use PCA to project my points to $\mathbb{R}^3$ and then scale them to $S^2$.


Ensemble models - neural network input both original data and predictions of other models? From my understanding in order to improve accuracy with ensemble models you need a wide range of independent ensemble methods. I was wondering whether using the ouput of a random forest model as one of the inputs for a neural network where the other input is the original data and the targets remain the same could improve the model? Why add extra complexity? basically the problem is a multi-dimensional regression problem and although the random forest gets a smaller MSE the neural network is bette at preservering some of the properties of the target labels. Therefore, I was wondering if by putting these two models together I would get a lower MSE while preserving some of the properties. Is it worth a shot? or will it just drastically overfit?

I suggest you test it by cross-validation to prevent overfitting. However, I guess because of a strong relationship between this variable and output your work will not be better. Instead, try to use an ensemble.

I read something about deep learning which in that way they use several models as a sequence and use results of previous models as inputs of the next models. Maybe if you continue this work it will be in some way a deep learning.


Visualize frequency of 5 Boolean variables together I have a data set with 5 variables,
a b c d e
1 0 0 1 0
0 1 0 1 1
0 1 1 0 0
0 0 0 1 0
1 1 1 0 0
0 1 1 0 1
1 0 1 0 0
1 0 0 1 1
0 1 0 1 1
0 0 1 1 0
I am only interested in the percentages of occurrence,
occurrence,
| a | b | c | d | e |
.4 | .5 | .5 | .6 | .4
BUT, I would like to visualize in such a way that I can see the overlap, or not, among all the different groups.
Any idea?

If you have richer data (ie more than 10 rows), you will want an upset plot.  Upset plots are a way to view information in an intuitive way like a Venn diagram, but is more useful for 4+ categories.
Some references which may give you some ideas and implementation in R:

https://cran.r-project.org/web/packages/UpSetR/vignettes/basic.usage.html (attached image from r-project.org).
https://www.littlemissdata.com/blog/set-analysis


EEG data layout for RNN How should one structure an input data matrix (containing EEG data) for an RNN?

Normally, RNNs are presented as language models where you have a one hot vector indicating the presence of a word. So if you input was the sentence "hello how are you", you would have 4 one hot vectors (I think):

[1, 0, 0, 0]
[0, 1, 0, 0]
[0, 0, 1, 0]
[0, 0, 0, 1]

How do these individual vectors get condensed into a single data matrix?

In the case of single channel EEG (with only 1 electrode), 256 samples per second, 1 second long samples, how should this data be structured? Would it be 256 vectors? If so, what does each vector represent/contain? Or should it be 1 vector that is 256 elements long?

Furthermore, how does this extend to multi channel EEG with, say, 64 electrodes over 256 time samples?

I would prefer to use the raw EEG data, rather than trying some dimensionality reduction (calculating means, spectrograms etc)

RNNs are not designed to do language modeling exclusively, they are designed to process time series data, and language happened to be representable as time series.
There is plenty of papers demonstrating how to use RNNs to do classification and regression on time series (awesome list of papers).

One-hot encoding is often used in cases where the input is discrete and not a number that can be directly fed into a model. However, one-hot encoding is actually not always the norm for language-modeling, some research actually map each character (or each word depending on how one wants to model the problem) to a unique numerical identifier (e.g. with $id\in \left\{ 0\ldots n \right\}$; with $n$ the size of the vocabulary) and the model map that identifier to a vector representation. This is particularly useful when the size of the vocabulary is huge and you want to avoid having to deal with big one-encoded vectors. Take a look at word2vec and word2vec Tensorflow for more details about this.

In your case, you want to process sensor data. There is no need for one-hot encoded because your data are continuous and numerical. In other words, you can input the recorded EEG data directly; although it's usually better to clean them up and normalize them beforehand. There is actually plenty of papers about EEG data processing with RNNs.

Regardless of the type of data, the idea to fed them into a RNN remains the same: provide a data sample $x$ for each timestep $t$.

For EEG data recorded for 5 timesteps with 1 electrode $a$, you would have a 1-dimensional vector with one sample per timestep:

$a = [0.12, 0.44, 0.134, 0.39, 0.23]$

$inputvector = [0.12, 0.44, 0.134, 0.39, 0.23]$

For EEG data recorded for 5 timesteps with 3 electrodes $a, b, c$, you would have a 3-dimensional vector with one sample per timestep:

$a = [0.12, 0.44, 0.134, 0.39, 0.23]$

$b = [0.43, 0.92, 0.3, 0.37, 0.4]$

$c = [0.13, 0.1, 0.4, 0.21, 0.14]$

$inputvector = [[0.12, 0.43, 0.13], [0.44, 0.92, 0.1], ...]$

I highly recommend you to read some of the literature I linked above.


ML algorithm with fixed number of inputs and variable number of outputs I am trying to solve the following problem:

Let's say I have a chess position:

I encode each square as one-hot encoded vector of length 13

index 0 for empty square
index 1 for white pawn
index 2 for black pawn
index 3 for white bishop
index 4 for black bishop
etc..

So, at the end I have a matrix ${M}$ with a shape 64x13, which I intend to use as an input of the model. I want to create a model able to map matrix ${M}$ to FEN representation of the position.

Given the example above, FEN representation of the position is: 1b1N4/7k/4b3/8/8/8/3P4/3K4

The length of a FEN-string is not fixed. I failed to find a ML algorithm which accepts fixed number of inputs and produces variable number of outputs. What machine learning algorithm can I use in that case ? Is RNN appropriate for that task ?

P.S. I do understand, that it is trivial to just write a function to map ${M}$ to FEN, but I do not want it, I want the model to learn it from data.

RNN (and LSTM / GRU)can generate variable length output. For example, generation of text.

In such problems, RNN is designed to generate next character (It keeps track of what characters have generated in past).  RNN should generate "End Of Output" character to indicate end of sequence.

For example : https://chunml.github.io/ChunML.github.io/project/Creating-Text-Generator-Using-Recurrent-Neural-Network/

Following method will have to be changed :

def generate_text(model, length):
    ix = [np.random.randint(VOCAB_SIZE)]
    y_char = [ix_to_char[ix[-1]]]
    X = np.zeros((1, length, VOCAB_SIZE))
    for i in range(length):
        X[0, i, :][ix[-1]] = 1
        print(ix_to_char[ix[-1]], end="")
        ix = np.argmax(model.predict(X[:, :i+1, :])[0], 1)
        y_char.append(ix_to_char[ix[-1]])
    return ('').join(y_char)

to :

     stop_characters = set(['.','?'])
..
..
     ix = np.argmax(model.predict(X[:, :i+1, :])[0], 1)
     predicted_char=ix_to_char[ix[-1]]
     if(predicted_char in stop_characters ):
         break
     y_char.append(predicted_char)


sklearn.decomposition.PCA explained_variance_ratio_ attribute does not exist When trying to identify the variance explained by the first two columns of my dataset using the explained_variance_ratio_ attribute of sklearn.decomposition.PCA I recieve the following error:

AttributeError: 'PCA' object has no attribute 'explained_variance_ratio_'

My code (condensed):

import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA

df = pd.read_csv('Input.csv')
df = df.dropna()
df_transform = StandardScaler().fit_transform(df)
pca = PCA(n_components=2).fit_transform(df_transform)
var_exp = pca.explained_variance_ratio_

When the last line is executed, I get the error:

AttributeError: 'PCA' object has no attribute 'explained_variance_ratio_'

I am using sklearn version 0.20.0

Edit

After examining the attributes of sklearn.decomposition.PCA, I see that the attribute does indeed not exist (as shown in image).

This is untested, but I believe the error is occurring because you're calling explained variance on the fit_transform object, as opposed to simply just the results of fit.

Try:

df = pd.read_csv('Input.csv')
df = df.dropna()
df_transform = StandardScaler().fit_transform(df)
pca = PCA(n_components=2).fit(df_transform)
new_df = pca.transform(df_transform)
var_exp = pca.explained_variance_ratio_


Handling dimensions for RGB data with Keras CNN I'm trying to make work the code from Keras' documentation getting started. There is something I do not understand about handling RGB data. I made one work with MNIST data (which is greyscale), but I can't seem to figure out the CIFAR10.
HEIGHT = 200
WIDTH = 200
def build_model():

    inputs = keras.Input(shape=(HEIGHT, WIDTH, 3))

    # making things simple, I don't CenterCrop, just set the input to whatever I'm feeding it
    # I do rescale to 0-1 values
    x = Rescaling(scale=1.0 / 255)(inputs)

    # this basically the doc's architecture
    x = layers.Conv2D(filters=32, kernel_size=(3, 3), activation="relu", )(x)
    x = layers.MaxPooling2D(pool_size=(2, 2))(x)
    x = layers.Conv2D(filters=32, kernel_size=(3, 3), activation="relu",)(x)
    x = layers.MaxPooling2D(pool_size=(2, 2))(x)
    x = layers.Conv2D(filters=32, kernel_size=(3, 3), activation="relu",)(x)
    x = layers.GlobalAveragePooling2D()(x)
    outputs = layers.Dense(10, activation="softmax")(x)
    model = keras.Model(inputs=inputs, outputs=outputs)
    return model

# fake data, easier to play with shapes  than with the actual CIFAR10 data to debug
# shape is (500, 200, 200, 3), cifar10's is (60000, 32, 32, 3)
data = np.random.randint(0, 255, size=(500, HEIGHT, WIDTH, 3)).astype("float32")
# shape is (500,1) cifar10's is (60000,1). Just 10 categories to match the output layer
labels = np.random.randint(0,9, size=(500,1)).astype("int8")

print("got fake data... ")
model = build_model()
print("model built... ")
model.summary()
model.compile(optimizer=keras.optimizers.RMSprop(learning_rate=1e-3),
              loss=keras.losses.CategoricalCrossentropy())
print("model compile...")
model.fit(data, labels)
print("done")

For some reason, I get:
ValueError: Shapes (None, 1) and (None, 10) are incompatible

Strangely, if change the label's fake data to size=(500, 10) then it "works". But obviously that makes no sense, since it would mean I have 10 labels for each sample. The MNIST dataset that works which I refer to can also be found in the linked documentation. The architecture of the network is much simpler (only Dense layers linked together).
What am I missing here? Why would it be so different for a colored image than it is for a greyscale?
I also tried the work with the data from     train, test = cifar10.load_data(), but it yields the same results.

Strangely, if change the label's fake data to size=(500, 10) then it "works". But obviously that makes no sense, since it would mean I have 10 labels for each sample.

It would most certainly not mean that; it simple means that, for the type of softmax classification you are attempting here with CategoricalCrossentropy loss, your labels should be one-hot encoded, and not single digits, as here (hence the error).
You should

Either convert your single-digit labels to one-hot encoded ones using to_categorical (docs)
Or change your loss to sparse categorical cross-entropy (docs)

Changes in your code for the first approach:
labels_ohe = tf.keras.utils.to_categorical(labels, num_classes=10)
# [...]
model.fit(data, labels_ohe)

Changes in your code for the second approach:
model.compile(optimizer=keras.optimizers.RMSprop(learning_rate=1e-3),
              loss=keras.losses.SparseCategoricalCrossentropy())

The MNIST dataset that works which I refer to can also be found in the linked documentation.

The MNIST labels, like the CIFAR10 ones, are also single-difit integers and not one-hot encoded; your MNIST model from the example you refer to you works because of the loss="sparse_categorical_crossentropy" setting.


Anomaly Detection System I need a sanity check. I want to create an anomaly detection system.

The logic which I am planning to use is the following:

Find anomalies in the past using Seasonal Hybrid Extreme Studentized Deviate Test.
Binarise the anomalies (1 the anomalies and 0 the trends).
Run several algorithms (Autoencoders, SVM, Logistic Regression, Naive Bayes, Lasso Regression, etc) with variables that are correlated and validate the models and use it.

Does the binarisation process makes sense?

Yes, your logic and what you are thinking is excellent.

There is only a flaw in your thinking: The variables you run the model with must not necesarily be "correlated" in a linear sense of the word, just don't discard any variable because any of them could explain your binary output, and not have a linear relationship with it.

Is a common solution to binarise an output to detect anomalies, but you will lose the ability to predict "how much" outlier is an outlier, make sure you don't need this information after.


Logistic Regression: Is it viable to use data that is outdated? TLDR: Want to predict who makes the playoffs (1,0), but there are more playoff spots now than there were in the past, is it okay to use that past data?

I want to use binary logistic regression on MLB data to estimate each team's probability of reaching the playoffs this upcoming season.

There is data going back as far as the seasons of the 1870s. However, my issue is that the structure of the playoffs and baseball as a whole has changed often over the years. Specifically, the changes deal with the number of playoff spots, which is in part due to an increase in the number of teams. For example, up until 1969 there were 20 teams, and there was only the championship (World Series), so, technically, only 2 teams made it to the "playoffs". The number of playoff spots has increased gradually to its present state, which is 10, in 2012, and there are now 30 teams.

To me, it makes sense to only use data from 2012 (to 2019) since it reflects the state of the upcoming season. This gives me 240 observations, thus 80 positive outcomes for my playoff (dependent) variable. However, I have about 40 predictors after removing highly correlated ones, which means that I should have way more observations. Though I know that the number of predictors will likely decrease once I fit the model, I still fear my sample size may still be too low. This makes me consider going further back to the previous era beginning in 1994 when there were 8 playoff spots, simply for the sake of more observations.

My question is that would it be viable to use such data in a regression, given that it may not accurately reflect the circumstances of what I'm trying to estimate? Could I maybe even go back to 1969?

I found this article which is pretty much exactly what I'm trying to do, and he uses data back to 1969, but it just seems like an issue to me.

Your thinking is sensible. Indeed, in a perfect world, your training data should be completely representative of the data you'll encounter. However, in practice, you often find that "unrepresentative" data may still have some value.

Ultimately, whatever you do is good if it improves your model, so if using "outdated" data helps, then do it!

Here is what you could experiment with:

Let the data speak

You could compare models using more or fewer data and it might give you an idea of the ideal cutoff point.

Implement "time decay"

Maybe, 1870 data is useful but likely, it's less useful than 1871 and even less useful than last year's. You could weight your training instance based on how old they are so that your recent data points have a bigger impact.

Create time-insensitive features

By this I mean, you could find a way to turn your problem such that the number of playoffs team doesn't matter. Instead of a binary "playoff/no playoff", your problem could be to instead rank your teams, then you can select the playoffs team based on the cutoff at this specific year.

You could also add how many playoff teams there are as a feature so that the learning algorithm is aware of how many spots there are.


Predictive analysis of rare events I'm trying to predict rare events, meaning less than 1% of positive cases. I basically try to predict if a subject will have 0, 1, 2 ... , 6, > 6 failures (there are cases in all those categories).

I've tried several algorithms:

decision trees
random forest
adaboost
grouping using k-means clustering and finding associations with failures (which group has most failure)

In any case, learning either goes to no failure or has too much variance (leading poor reasults on C.V. set).

Do you know any machine learning algorithms which are better suited for rare events?

Or is it surprising that I get those bad results using those algorithms, which means that my features list is not good?

Thanks a lot.

When you have an unbalanced data set, the algorithm is going to weight its success on each data point equally, meaning the majority class comes out as much more important than the minority class. The typical solution is to sample down the majority class until it's the same size as the minority class, and an alternate (similar) solution is to adjust the cost function so that the minority class is weighted appropriately.

See these similar questions for more:

Should I go for a 'balanced' dataset or a 'representative' dataset?
Quick Guide into training highly imbalanced data sets
What are the implications for training a tree ensemble with highly biased datasets?
Skewed multi-class data
Ratio of positive to negative sample in data set for best classification