Layer shape computation in convolutional neural net (pyTorch) How can you know the expected input size (image input size (tensor size)), for example for this network (cf. pyTorch tutorial example ):

import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 3x3 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 3)
        self.conv2 = nn.Conv2d(6, 16, 3)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 6 * 6, 120)  # 6*6 from image dimension
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        # If the size is a square you can only specify a single number
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = x.view(-1, self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

net = Net()
print(net)

Since it is nowhere explicitely stated. Moreover the comment

self.fc1 = nn.Linear(16 * 6 * 6, 120)  # 6*6 from image dimension

is unclear. What does this shape: (16 * 6 * 6, 120) have to do with image size  (e.g. 32x32 as claimed by the authors of the tutorial) ?
I cannot find a way, by looking at the code, to know what input size is expected by the net?

Well, with conv layers in pyTorch, you don't need to specify the input size except the number of channels/depth. However, you need to specify it for fully connected layers.
So, when defining the input dimension of the first linear layer, you have to know what is the size of the images you feed.

You can find information on the output size calculation of conv layers and pooling layers here and here or here

If you feed images of size 32x32, the outputs layer by layer of this model are :

conv1 : $6$ feature maps of size $\left \lfloor{\frac{32 + 2\times0 -1\times(3-1)-1}{1}+1}\right \rfloor   = 30$
max_pool2d: $6$ feature maps of size $\left \lfloor{\frac{30 + 2\times0 -1\times(2-1)-1}{2}+1}\right \rfloor = 15$
conv2 : $16$ feature maps of size $\left \lfloor{\frac{15+ 2\times0 -1\times(3-1)-1}{1}+1}\right \rfloor = 13$
max_pool2d: $16$ feature maps of size $\left \lfloor{\frac{13+ 2\times0 -1\times(2-1)-1}{2}+1}\right \rfloor = 6$

Therefore, for the flattened size of the output before the first linear layer is $16\times6\times6$ :

 self.fc1 = nn.Linear(16 * 6 * 6, 120)

By doing all sizes calculations in reverse, you could have find that the input size must be $32\times32$.


SARIMAX if your "seasonality" is a clear sine/cosine can work OK (as long as you give the sine/cosine wave as the exogenous input).
Also you could just try bunging it all into Facebook's Prophet.


How to obtain the final values of a DecisionTreeRegressor in Scikit-Learn? This page shows the paths in the decision trees in scikit-learn. After reaching the leaf nodes of the decision tree, where do we obtain the final resultant value?

For documentation, check the Scikit-Learn code at Github i.e. line#535 [Link]

value : array of double, shape [node_count, n_outputs, max_n_classes]
Contains the constant prediction value of each node.

You can check the same using this sample code.

Identify the leaf nodes
Slice the value attributes for leaf nodes

from sklearn.datasets import load_diabetes
import numpy as np, matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeRegressor

X, y = load_diabetes(return_X_y=True)
regressor = DecisionTreeRegressor(max_depth=5)
regressor.fit(X, y)

children_left = regressor.tree_.children_left
children_right = regressor.tree_.children_right
leaf_nodes = []
for i in range(n_nodes):
    if children_left[i] == children_right[i]:
        leaf_nodes.append(i)

y_pred =  regressor.predict(X)
all_val = regressor.tree_.value[leaf_nodes,0,0] # Sliced on Leaf nodes

# Check if all the y_pred is from these values
set(y_pred) - set(all_val) # Or, [elem for elem in y_pred if elem not in all_val]


Extract datatable column by indexing a character vector I am asking a similar question to this one except I am trying to index a vector of column names.

If I have a data.table like a:

a <- data.table("NAME" = c("A", "B", "A"),
                "PASS_FAIL" = c("F", "P", "P"),
                "TEST_COUNT" = c(NA, NA, 1))

And a vector of column names:

col_names <- c("NAME", "PASS_FAIL", "TEST_COUNT")

Then try to index with this syntax:

a[col_names[1]]

I get this error:

Error in `[.data.table`(a, col_names[1]) :
  When i is a data.table (or character vector), the columns to join by must be specified
  either using 'on=' argument (see ?data.table) or by keying x (i.e. sorted, and, marked as
  sorted, see ?setkey). Keyed joins might have further speed benefits on very large data
  due to x being sorted in RAM.

I also tried this a$col_names[1] which results in NULL (but since it is a data.table this is the wrong syntax.

Calling just col_names[1] gives a character "NAME" which is what I expected.

Any ideas on what I am doing wrong here? Thanks!

You need [[]] as otherwise you are attempting to index the row.

a[[col_names[1]]]


Training a CNN to convert ellipses into circles My current project has to do with modeling the effects of blurring/convolution of objects in various imaging processes. Right now, I am starting off with a preliminary, artificial model. I am using Keras to accomplish this.

I create artificial ideal data as a set of circles in random locations in a 128 x 128 image. I then have a routine that takes the coordinates and sizes of these circles as input and replaces the circles with ellipses at the same locations.

I am trying to train a convolutional neural network to perform the inverse of this function, i.e. read the images with ellipses as input and replace them with circles. I create training pairs by grouping these two image types together with the images with ellipses as inputs and the images with circles as outputs.

How can I design a neural network which accomplishes this? Right now, the one that I am using merely returns distorted copies of the images of ellipses instead of outputting images with circles.

I have played around with multiple different CNN structures, as you can see in the commented-out code:

model = Sequential()

#model.add(Dense(10, activation = 'relu', input_shape = (80, 128, 128, )))
model.add(Conv2D(nb_classes, kernel_size=3, padding = 'same',
                 activation='relu',
                 input_shape=(128,128,1)))
#model.add(MaxPooling2D(pool_size=(2, 2), padding='same'))
model.add(Conv2D(32, kernel_size = 3, activation='relu', padding = 'same'))
#model.add(MaxPooling2D(pool_size=(2, 2), padding='same'))
#model.add(Dropout(0.25))
#model.add(Flatten())
#model.add(Dense(128, activation='relu'))
#model.add(Dropout(0.5))
#model.add(Conv2D(32, (3, 3), activation='relu', padding='same'))

#model.add(UpSampling2D((4,4)))

model.add(Conv2D(nb_classes, (3, 3), activation = 'relu', padding='same'))
#model.add(Dense(10, activation = 'relu', input_shape = (128, 128,)))
model.add(Activation('softmax'))

Use your generated images to train an encoder-decoder that attempts to rebuild the input image. We'll call these pieces the input encoder and ellipse decoder. In parallel, train a second decoder that takes the input encoder (i.e. embedding representation) as input and attempts to rebuild the original circle image. The input encoder + circle decoder can then be used together for scoring out-of-sample observations.


How big should training data be? I am solving a problem connected with medicine and from each patient I get about 100 features. It's a classification diseases problem, however measurements take a lot of time and also require money.

What is the optimal size of training data? I understand that I should collect as much data as possible, however is there any empirical advice?

While the other answers aren't wrong, they don't touch anything about bioinformatics. I'll go into details.

In bioinformatic, simply asking for the size of the training set makes no sense. You'll need to understand deeper about bioinformatic to answer this question. There is a good reference: An ensemble approach to accurately detect somatic mutations using SomaticSeq for you.

The goal of the project is to build a gradient-boosting machine for classifying somatic mutation given an alignment file. This is very close to what you're doing.

Now, you want to know the training size? You should ask yourself the following:

Sequencing errors
Reads
Sequencing depth
Sensitivity
F1 score
Allele frequency

In an next-generation sequencing, your sequencing machine always give you errors. How reliable is your pipeline to detect those errors?

How good are your reads for your problem? For example, if you're detecting disease related to structural variant, you might need more samples to compsenate for the short reads.

What's your sequencing depth? In the paper I quote, they have like 30x depth. It'd be pointless to talk about the training size unless you know well about your depth.

Does your sequencing cover well your region of the genome you're interested in?

Sensitivity and F1 score. You can have good idea by drawing an F1 curve something like as in the paper:

What's your expected allele frequency range? If you have something like close to 0.5 (ie: maximum heterozygosity), you'll probably be safe with a smaller sample size. However, if you're trying to detect a rare disease, something like <0.1 as in the diagram, you'll need to have more samples and most likely more features.

Empirically, we generally use spike-in standards to measure the sensitivity. Once you draw the sensitivity against training size, you'll know at what training size to use achieve whatever sensitivity you're looking for. A typical tool would be ROC.


Evaluating value functions in RL I'm working my way through the book Reinforcement Learning by Richar S. Sutton and Andrew G. Barto and I am stuck on the following question.

The value of a state depends on the the values of the actions possible in that state and on how likely each action is to be taken under the current policy.
We can think of this in terms of a small backup diagram rooted at the state and considering each possible action:

Give the equation corresponding to this intuition and diagram for the value at the root node

, in terms of the value at the expected leaf node,

, given

. This expectation depends on the policy,

. Then give a second equation in which the expected value is written out explicitly in terms of

such that no expected value notation appears in the equation.

I should mention that...

...

Where...

 = Probability of taking action a from state s

 = Given any state s and a, the probability of each next state s'

 = Expected reward given any state s, next state s, and action a

How can I re-evaluate this value function in the way that is asked?

In a discrete probability space, the expectation of a random variable (RV) is a sum over all possible values multiplied by their individual probabilities. Here, your RV is $q_\pi(s,a)$, with $s$ being fixed. Therefore you need to take the expecation with respect to the actions only:

$v_{\pi}(s) \stackrel{\text{1. task}}{=} \mathbb{E}_{a\sim \pi(s, a)}[q_{\pi}(s, a)] \stackrel{\text{2. task}}{=} \sum_\limits{a}\pi(s, a) q_{\pi}(s, a)$.

$\pi(s, a)$ represents the "individual probabilities".


Predicting which apps users may be interested in I am building a mobile app that can predict what apps users may be interested in downloading from the play store, based on what apps the user has already installed on their device and how much time they have spent on these apps. Also, there is the option to scroll through the top apps in the play store and you can "favourite" any apps that you find interesting/want to download.

Based on this data, I would like to make predictions and notify users of any new popular apps that are released in the future, however since the predictions will be user specific, I am unsure if the dataset will be too small for k-means clustering (Group the apps into genres and find most popular genres). 35 is the average number of apps installed on user's smartphones, and if you include any "favourited apps" the total dataset could be around 50.

Perhaps there is another more suitable technique I could apply to make accurate predictions but I am unsure where to go at the moment.

Broadly speaking, what you want is a recommender system. If you consider only the user-app association data (app installs, likes, etc.), and no user or app metadata, then you should look at collaborative filtering.

More specifically, a simpler technique you can consider is item-similarity based recommendation (users who install app A, also install app B). A slightly more complex method involves factorization of the partial user-app association matrix and then inferring missing points in the matrix. This paper describes one such factorization technique used in the Netflix Prize movie recommendation challenge.


Is there a C library for machine learning algorithms? Are there any machine learning libraries for C.
Specifically interested in unsupervised learning.

Here's a detailed table about different machine learning libraries on different languages:

https://github.com/josephmisiti/awesome-machine-learning

Checkout C version here and C++ version here.

Personally speaking, try OpenCV ! OpenCV provides multiple machine learning implementations including KMeans, kNN, SVM, etc.


How to give importance to recent data in a large dataset I am solving a binary classification problem which is also partly a time-series problem. This means that the recent data is more relevant than the old data. But how can I assign more importance to the recent data as compared to the old one?

Time-series methods like LSTM, Kalman filters and Markov models handle this automatically. So, assigning more importance to more recent data is in their structure.


Timeseries LSTM: does test data need to come after training data? I have one single, very long time series. I want to train an LSTM to distinguish between two behaviours (A or B) at every timestep (sequence-to-sequence).
Because the time series is very long, I plan to extract shorter, partially-overlapping subsequences and use each of them as one training input for the LSTM.
In my train/validation/test split, do I have to use older subsequences for training and newer for validation and test? Or can I treat them as if they were independent samples and just randomly shuffle them, given that anyway the LSTM will start each subsequence with empty memory?
The reason I ask is because I noticed that, due to how the timeseries was collected, the first half contains mostly behaviour A while the second half mostly behaviour B. This would cause training to be mostly on A and testing mostly on B, which does not reflect the fact that, in production, the system will see both periods of predominant A and periods of predominant B.

If I got it correctly, you might have the following scenearios:

labeled samples with tag A or B per date-time index, having as informative attributes the present + some lag values of interest --> (this would be a standard classification approach without time ordering bein necessary)

sliding window of samples (what you mean by subsequences) --> here you should respect the time ordering at least for the validation set, so you make sure you evaluate your LSTM with a real scenario with future never seen sequences.
With this second approach, you can indeed shuffle the training batches (via the keras shuffle(BUFFER_SIZE).batch(BATCH_SIZE) functionality for instance (info here), but leaving the validation set without shuffling, as follows:
  BATCH_SIZE = 256
  BUFFER_SIZE = 10000

  train_data_multi = tf.data.Dataset.from_tensor_slices((x_train_multi,y_train_multi))
  train_data_multi = train_data_multi.cache().shuffle(BUFFER_SIZE).batch(BATCH_SIZE).repeat()

  val_data_multi = tf.data.Dataset.from_tensor_slices((x_val_multi, y_val_multi))
  val_data_multi = val_data_multi.batch(BATCH_SIZE).repeat()

You can find a complete worked-out example here
You can also make use of the time series data preprocessing helper where you can decide whether to shuffle or not.


Does the mean/median of a set sentence embedded vectors represent anything? Please bear with me as I am new to NLP.
I am specifically using tensorflow's universal sentence encoder: https://tfhub.dev/google/universal-sentence-encoder-large/3

I am clustering text based on the cosine similarity of the embedding produced by the model and I want to see what cluster a new text would most likely lie in.  I was going to compare the new text embedding to the mean/median of all the embeddings within a cluster to see which cluster it would most likely lie in.  Would taking the mean/median of the cluster's vectors "represent" the general idea of the cluster or will the vector not represent what I am looking for?

Well, the mean is pretty average for all the words.

These tend to all be quite similar, cluster in the center of the data, and have nearest neighbors to pretty bland, generic words.

The average word vector is not a good representation of what a text is about.


How to determine the correct target for classification probability when the observed samples are probabilities of each class? I have data in which each event's outcome can be described by a probability of a categorical occurrence. For example, if all of the possible class outcomes are A, B, C, or D suppose in one event 7/10 people selected category A, 2/10 selected category B, and 1/10 selected C.
The goal of the model is to predict the probability of each class as close to the observed truth as possible. However, it is unclear how to determine the proper target. Here are the options I've thought about doing:

The target becomes the predominant class selected. That is, in the event described above A=1, B=0,
C=0, D=0.

Additional classes/categories are created and then become bins of all possible observed
probabilities. For example, the event above would become A=1 when Pr(A) >= 0.7.

I THINK option 1 is the the best as option 2 would result in a lot of categories and spread. However, when considering verification of the model output, i.e. Y = {A=0.9, B=0.1, C=0, D=0} for each sample would it not be advantageous to use the actual observed probabilities? Are there other options for the target?

Imho the "cleanest" option would be to train a probabilistic model on the original categorical target, then obtain the predicted probabilities for every category as the final "predictions". By "training on the original target" I mean designing each instance as an event, e.g. in order to represent that 7/10 people select category A there would be 7 instances where the target is category A out of 10 instances in total.

The most simple option is Naive Bayes, but depending on the data it tends to always predict extreme probabilities, which would defeat the purpose.
An ad hoc Bayesian model could give very good results but it's probably more work to design it, depending on the features.


What are some nice algorithms/techniques for optimizing and predicting Click Through Rates (CTR)? Definition of Click Through Rates:

  CTR is the number of clicks that your ad receives divided by the
  number of times your ad is shown expressed as a percentage (clicks ÷
  impressions = CTR).

Logistic regression is very common and efficient.

This will provide you with a probability of click for a given impression. You will likely need additional information for optimization, like keywords or A/B testing.


Feature importance after PCA (or other dimensionality reduction methods) I have text data which I one hot encoded and then used PCA on it (although I'm experimenting with other methods as well, LDA, NMF..). I am using the result of the dimensionality reduction as an input for a supervised classification task.

Now I can use the random forest feature importance or other methods to get feature importance of the input to the supervised cls task. However, naturally these features are meaningless. I would like to know which words are most the most important for this classification. In other words, somehow propagate the feature importance score back through the PCA.

Is there any known method to do it?

Most of the PCA methods return the linear transformation matrix which allows to convert from Component to Variable and viceversa.

What is impossible to do is to assign a feature importance from component to variable (unless you are willing to accept the assumption that "feature importance" could be assigned linearly).

If you accept the idea between parenthesis, you could have a linear equation system $Ax = b$ where $A$ is the component-to-variable matrix (of size components-by-variables), $b$ is the feature importance and $x$ is the result (the variable importance).

This idea I am giving to you is what I think could work, you are likely not going to find it in any research or similar.


In k-fold-cross-validation, why do we compute the mean of the metric of each fold In k-fold-cross-validation, the "correct" scheme seem to compute the metric (say the accuracy) for each fold, and then return the mean as the final metric.

Source : https://scikit-learn.org/stable/modules/cross_validation.html

However, why can't we just compute the metric directly on all the predictions, as we have a estimation for all the data.

Thanks

It's ok to compute the global performance on the concatenation of the predictions for all the K folds after running the cross-validation process, it depends on the goal and on the metric (for instance the mean accuracy over the folds gives the same result as the global accuracy, but that's not true for every evaluation measure).

But very often the goal involves not only measuring performance accurately but also measuring the variance of the performance across the folds, in order to detect instability. This can't be done from the concatenation of the predictions, so it's often more convenient to keep the folds results separated.

(this is my interpretation, there might be other reasons)


How do I show my pandas dataframe that I have in my jupyter notebook so that I can ask a question on stack exchange/stack overflow? I've noticed that people produce such an amazing pandas dataframe which allows people to answer the questions. However, whenever I try to copy my table after doing df.head(), and "paste" it on here so that I am able to ask a question it leaves me with a table that needs to be restructured.

I have searched for similar questions here but have been struggling to find a step-by-step answer on how to ask questions especially when it is related to pandas and that one has to get a sense of the data at hand before being able to provide an answer.

df.head().to_dict()

will give something which looks like

  {'x': {0: 1, 1: 200, 2: 4, 3: 5, 4: 6}, 'y': {0: 4, 1: 5, 2: 10, 3: 24, 4: 4}}

you can then add pd.DataFrame() like so

pd.DataFrame({'x': {0: 1, 1: 200, 2: 4, 3: 5, 4: 6},
              'y': {0: 4, 1: 5, 2: 10, 3: 24, 4: 4}})

This will allow anybody else just to copy and paste into their editor and run your code.

Here is a really detailed discussion on what you are describing
https://stackoverflow.com/a/20159305/6646710


Cosine similarity of averaged random word vectors I am trying to find the cosine similarity (using glove vector) of two random words. As expected, the distribution of the similarity concentrated around 0 since it is reasonable to think that two random words will not be similar to each other.

However, when I try to do a similar thing to 2 random sets of 10 words, that is I take the average vector of the 10 words in both sets and calculate the cosine similarity, the similarity tends to concentrate at 0.8.

It seems to suggest that given 2 random sentences of 10 words, they are very likely to be similar semantically. What could be the explanation of this?

Included python code to reproduce the result.

import spacy
nlp = spacy.load('en')

vocab = nlp.vocab
words = np.array([x.orth_.encode('utf8') for x in vocab])

hist1 = []
n = 1000
num_words = 1
for _ in range(n):
    x,y = choice(words, size=(2,num_words))
    x = nlp(" ".join([u.decode('utf8') for u in x]))
    y = nlp(" ".join([u.decode('utf8') for u in y]))
    s = x.similarity(y)
    hist1.append(s)

hist10 = []
n = 1000
num_words = 10
for _ in range(n):
    x,y = choice(words, size=(2,num_words))
    x = nlp(" ".join([u.decode('utf8') for u in x]))
    y = nlp(" ".join([u.decode('utf8') for u in y]))
    s = x.similarity(y,)
    hist10.append(s)

plt.hist([hist1,hist10], label=[1,10])
plt.legend()

Nothing too surprising here. As you sample more and more words, the sample mean is a better and better estimator of the population mean. This is called the law of large numbers.


Solving multi label image classification using TimeDistributed dense layer I have a multi label image dataset having 5 labels. Each image can have more than one label at the same time. I am using a convolutional neural network to extract features and those extracted features I am giving to RepeatVector layer to create 5 copies of extracted features and after RepeatVector layer I have connected TimeDistributed layer with TimeDistributed(Dense(2)).

y_train is a 3D array and its shape is (1600, 5, 2) and x_train is an array of images.

For example:

>>> x_train.shape
(1600, 3, 100, 100)

>>> y_train.shape
(1600,5,2)

>>> y_train[0] =
array([[0, 1],   # [0,1] = 1 label present and  [1,0] = 0 label abset
       [1, 0],
       [1, 0],
       [1, 0],
       [1, 0]])

Code:

def get_label(y):
  tmp = []
  d = {0:[1,0],1:[0,1]}   # 0 absent 1 present
  for i,value in enumerate(y):
    tmp.append( d[value] )
  return tmp

X,Y= get_data()
Y = Y.tolist()
y = []
for value in Y:
  y.append(get_label(value))

Y = np.array(y,dtype=int)

x_train, x_test, y_train, y_test = train_test_split(X,Y,test_size =0.2,random_state=100)

img_channels = 3
img_rows     = 100
img_cols     = 100
nb_classes   = 5

model = Sequential()

model.add(Convolution2D(32, 3, 3, border_mode='same',input_shape=(img_channels, img_rows, img_cols)))
model.add(Activation('relu'))
model.add(Convolution2D(32, 3, 3))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

model.add(Convolution2D(64, 3, 3, border_mode='same'))
model.add(Activation('relu'))
model.add(Convolution2D(64, 3, 3))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

model.add(Flatten())
model.add(Dense(512))
model.add(Activation('relu'))
model.add(RepeatVector(nb_classes))
model.add(TimeDistributed(Dense(2)))
model.add(Activation('softmax'))

# let's train the model using SGD + momentum (how original).
# opt = RMSprop(lr=0.001, rho=0.9, epsilon=1e-06)
opt = SGD(lr=0.01, momentum=0.0, decay=1e-6, nesterov=False)
model.compile(loss='categorical_crossentropy', optimizer=opt,metrics=['accuracy'])
model.fit(x_train,y_train,nb_epoch=10,batch_size=32,validation_data=(x_test,y_test),shuffle=True)
out = model.predict_classes(x_test)

But after training, I get all zeros for the test set. Is this approach wrong?

There is no need to repeat any features. What you want to do is change your activation function from softmax to sigmoid, otherwise leaving everything else as it would normally be done for binary classification.

...
model = Sequential()

model.add(Convolution2D(32, 3, 3, border_mode='same',input_shape=(img_channels, img_rows, img_cols)))
model.add(Activation('relu'))
model.add(Convolution2D(32, 3, 3))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

model.add(Convolution2D(64, 3, 3, border_mode='same'))
model.add(Activation('relu'))
model.add(Convolution2D(64, 3, 3))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

model.add(Flatten())
model.add(Dense(512))
model.add(Activation('relu'))
model.add(Dense(5)) # whatever the number of classes is
model.add(Activation('sigmoid'))
...

You will then need to simplify your Y tensor from N x 5 x 2 to N x 5, e.g. with Y = Y[:, :, 0].

This may or may not be why you get all zeros, but it is certainly a more standard approach to a multi-label classification than using RepeatVector and TimeDistributed. Something else to look at is how much you are training, 5 epochs with only 1600 examples is not a lot. Similarly, a Dense(512) is very large for your small dataset size. Maybe reduce that down to something like Dense(16)?


how to I replace numeric values with a string in an R dataframe? I want to replace all numeric values in a column in my data frame with a string value. The following doesn't seem to work.

df <- within(df, myCol[is.numeric(myCol)] <- 'NOTMISSING')

Even though the df has some values as NA and others as numbers, all values are being replaced with NOTMISSING.

Also tried

df <- within(df, myCol[is_numeric(myCol)] <- 'NOTMISSING')

Any pointers highly appreciated.

From the documentation of is.numeric:

  The default method for is.numeric returns TRUE if its argument is of mode "numeric" (type > "double" or type "integer") and not a factor, and FALSE otherwise.

So for a vector, is.numeric returns a single TRUE, it doesn't test each element as you might expect.

is.numeric(c(5, 4, 3))
[1] TRUE

is.numeric(c(5, 4, NA))
[1] TRUE

That's why either all or none of the values are changed to NOTMISSING.

@eg-r's fix is correct.  Here's a tidyverse way to accomplish the same.

> df<-tibble(myCol=c(5, 4, NA))
> df
# A tibble: 3 x 1
  myCol
  <dbl>
1     5
2     4
3    NA
> df %>% mutate(myCol = ifelse(is.na(myCol), myCol, "NOTMISSING"))
# A tibble: 3 x 1
       myCol
       <chr>
1 NOTMISSING
2 NOTMISSING
3       <NA>


SRCNN - how to get a color image output training only with luminance I'm quite new to convolutional neural network, applied to super-resolution. I read this review article, itself based on this paper as an attempt to understand it better.

In the review, the author says :

  where c is [the] number of channels of the image [...].
  In this case, c=1 [...]

But the example shows a color image. So, first, I asked myself "How a color image could have only 1 channel ?".

Then I read the following in the paper :

  The majority of SR algorithms [...] focus on gray-scale or single-channel
  image super-resolution. For color images, the aforementioned
  methods first transform the problem to a different
  color space (YCbCr or YUV), and SR is applied
  only on the luminance channel.

Ok, so the network only works with luminance. Therefore how can it generate a color image as an output ? Converting a full color system to only luminance, a part of the information on the color is lost. How does the network get it back in the process ?

The authors use the Y-channel as luminance channel since their images are in the YCbCr color space. The other two channels are upscaled by bicubic interpolation. Later in the paper, they write:

  Specifically, we first transform the color images into the YCbCr space. The SR algorithms are only applied on the Y channel, while the Cb, Cr channels are upscaled by bicubic interpolation.

In their paper, the authors also compare this method to other learning strategies, e.g. training on all three channels or first training the Y-channel and then fine-tune the network with all three channels. The results are on page 11. The section ends with:

  It is also worth noting that the improvement compared with the single-channel network is not that significant (i.e., 0.07 dB). This indicates that the Cb, Cr channels barely help in improving the performance.


How mean and deviation come out with MNIST dataset? I am a novice at the data science, and I notice some repository state the mean value and deviation in MNIST dataset are 0.1307 and 0.3081.

I cannot imagine how these two numbers come from. Based on my understanding, the MNIST dataset has 60,000 pics and each of them has (28 * 28 = 784) features. How do I convert this feature vectors to get the mean and deviation?

Especially, this should classify by the label, right? For example, the number 0 should have its mean and deviation. For number 1 should also have its mean and deviation.

mean : It is the mean of all pixel values in the dataset ( 60000 × 28 × 28 ). This mean is calculated over the whole dataset.
deviation : It is the standard deviation of all pixel values. The dataset is treated as a population rather than a sample.

  What are the uses of these values?

Mean and standard deviation are commonly used to standardize the data in this case the images. Standardized data has mean close to 0 and standard deviation close to 1. You can read more here.

  Why to standardize the data?

Standardization transforms your data in such a manner that it has unit variance.
According to Wikipedia,

  In statistics, the standard score is the signed number of standard
  deviations by which the value of an observation or data point is above
  the mean value of what is being observed or measured


What should be the requirement for training data in order to obtain a good regression model using neural network? I have made a neural network regression model using the theory for the first time and would like to clarify some basic doubts, whose concrete answers I couldn't find yet.

Data:- I have 3000 samples with 7 input features (consisting of 5 normalized continuous variables and 1 binary variable recoded using one-hot encoding). I have 2 outputs per sample.

So I have trained my model and I am getting around 95% accuracy. I feel it's too good to be true. Is it?
I have normalized my training data and used the mean and standard deviation of the training data to normalize the test data. Right approach, right?
I have read that neural network is after all a method to do interpolation, so how would it behave on the data with features with values which are not within the range while it's being trained?

I ask this because when I take a $80\%$ random sample out of the available data, it's possible that for a given feature (continuous variable) $X_1$ out of $X_1, X_2, ..., X_7$, $X_1 \in [a,b]$, where $a,b$ are some scalars. However, the other $20\%$ test data might have $X_1$ values $<a$ or $>b$. Is my trained model expected to give correct results then?

Should I treat each output independently and create models, $i.e.$ weights separately (2 in my case) or should I find weights with the joint output? (The outputs may have different scales, so should I normalize them too in this case?)

How different would be the behaviours of both the models?

I believe the questions are preliminary but it would be really helpful to get my basics clear.

If this is your test accuracy (and you make sure not to use any test samples as part of the training), then I don't see any reason not to trust it. Maybe the distribution of the training data is a very good approximation of the real distribution. You can perform a quick sanity check by trying a different separation between the training and test dataset.
Yes, that is correct.
The short answer: you have no way of knowing how it will behave outside the range of trained samples. However, a case where a considerable amount of the test set is taken out of the range of all the training set with a 80-20 division is not likely.
Don't have any personal experience with this question, but from a quick read of several similar questions, it seems that the optimization process would be driven by the target variable which has the largest scale. So you either need to add some weights to the loss function or normalize the outputs (same-same).
Regarding the multi-output vs. multi-model question, I think that the first answer in the following thread is on point:
What is the difference between using single multi-output NN and multiple single-output NNs


How to filter rows in Python pandas dataframe with duplicate values in the columns to be filtere Overall context:

I have a data frame that contains observations for every five minute starting at 5 AM in the morning and ending at 8 PM in the evening for several days. I need to filter all the observations that start from 9 AM in the morning and end at 5 PM in the evening for every day.

The input data frame looks like this:

Date Time
2019-09-20 05:00:00,..,..
2019-09-20 05:05:00,..,..
...
2019-09-20 09:00:00,..,..
...
2019-09-20 17:00:00,..,..
2019-09-20 17:05:00,..,..
...
2019-09-20 20:00:00,..,..
2019-09-21 05:00:00,..,..
2019-09-21 05:05:00,..,..
...
2019-09-21 09:00:00,..,..
...
2019-09-21 17:00:00,..,..
2019-09-21 17:05:00,..,..
...
2019-09-21 20:00:00,..,..

and the output data frame should look like this:

2019-09-20 09:00:00,..,..
...
2019-09-20 17:00:00,..,..
2019-09-21 09:00:00,..,..
...
2019-09-21 17:00:00,..,..

Steps taken so far

In order to extract the rows between 9 am and 5 pm, I determined the number of seconds since midnight for every row by
extracting the hours, minutes and seconds using  vectorized data operations
so input dataframe will have column like:

Date Time, Number of seconds since midnight
2019-09-20 05:00:00,xxxx,..,..
2019-09-20 05:05:00,yyyy,..,..
...
2019-09-21,05:00:00,xxxx,..,..
2019-09-21, 05:05:00,yyyy,..,..

Note that for the same time on every day, the number of seconds will remain the same
Now I was hoping to  extract alll the rows between 9 am and 5 pm by

df[(df['Number of seconds since midnight'] > (nseconds for 9 am from midnight)) &  ((df['Number of seconds since midnight'] < (nseconds for 5 pm from midnight))

but I get the rows from only the last date between 9am and 5 pm.
TO me, it looks it is ignoring all the duplicate rows with the same time.

Can anyone suggest a possible solution that does not iterate over each row and uses the vectorized operations as the database is very large

I think you have defined midnight as today's 00:00. Therefore, the rows before today are out of your range.

I think this may work for this cases:

# Convert string to datetime format
df['Date Time'] = pd.to_datetime(df['Date Time'])

selected_rows = df[((df['Date Time'].dt.hour * 60 + df['Date Time'].dt.minute) >= 9 * 60) &
                   ((df['Date Time'].dt.hour * 60 + df['Date Time'].dt.minute) <= 17 * 60)]

The filter rules use the time only and ignores the date.


Randomforest parameter tuning R(caret) and Python(scikit learn)? Question: is it possible, or even necessary, to perform a cross-validation check to tune the parameters of a Python randomforest implementation (eg scikit learn) when training a new model, as can be done in R's caret?
Background R: When using R's caret's randomforest library, one can tune the parameters by performing a n-fold cross validation, e.g.
train_control <- trainControl(method = "repeatedcv",
                          number = 10,
                          repeats = 5,
                          verboseIter = TRUE,
                          allowParallel = TRUE,
                          summaryFunction = multiClassSummary)
rf1 <- train(Class ~ .,
         data = train_transformed,
         method = "rf",
         metric = "Accuracy",
         tuneGrid = my_grid1,
         trControl = train_control)

This outputs -
## Aggregating results
## Selecting tuning parameters
## Fitting mtry = 1 on full training set

or whatever mtry is the best for a dataset.
Background Python:  Using scikit learn, one can instantiate a randomforest regressor and can perform a cross-validation check on that regressor
RFRegModel = RandomForestRegressor(random_state=42)
cv = cross_validate(RFRegModel,X,y,cv=5,verbose=1)
print(cv['test_score'])

the difference here is that the cross validation doesn't appear to influence the tuning of the randomforest parameters, as I think it does in R. I think all that is happening here is that we are taking random subsamples of the training data set and testing the implementation of the
Question again: is it possible to train the model in python in a similar manner as is done in caret, i.e. by forcing an n-folds cross-validation
parameter tuning?  or, is this even necessary? Is the caret implementation overconstrained by it's methodolgy?

I guess what you are looking for is sklearn.model_selection.GridSearchCV() or a similar function regarding the type of "search" (grid, random, ...) you would like to conduct. This function does hyperparameter tuning and uses cross validation when doing so and the cv parameter will allow you to specify the number of folds.


Autonomous evalution Do you think it is possible to learn the app, how to autonomous evaluate good or bad parking of the bikes? The thing is you need to take a picture with your phone and app need to decide according to this pictures. The parking will be in the streets.
Deciding will be based on the huge database with the pictures of the bikes which parked good/bad.

It is definitely possible!

You just need thousands of labeled pictures, a team of ML engineers, a few months, and some tens of thousands dollars to invest.

Jokes aside, it is not a simple task. Also, the labeling of the images will be a pain if they are not already labeled. You will need a team for that.


IterativeImputer Evaluation I am having a hard time evaluating my model of imputation.
I used an iterative imputer model to fill in the missing values in all four columns.
For the model on the iterative imputer, I am using a Random forest model, here is my code for imputing:
imp_mean = IterativeImputer(estimator=RandomForestRegressor(), random_state=0)
imp_mean.fit(my_data)
my_data_filled=  pd.DataFrame(imp_mean.transform(my_data))
my_data_filled.head()

My problem is how can I evaluate my model. How can I know if the filled values are right?
I used a describe function before and after filling in the missing values it gives me nearly the same mean and std. Also, the correlation between variables stayed nearly the same with slight changes.

When imputing data, one is looking not to modify the true distribution of your data. So a way to test how good your imputation was is to make a test to contrast the true distribution of every feature that has been imputed vs the true (via KS test for example) distribution of the feature (prior imputing) if you can sate with a level. of confidence that your imputation preserved the distribution that would be a way.
Another way would be in case you have a supervised task, you can compare the performance of your model on each imputation technique. Like in the below's image from Scikit-learn documentation:


MAE estimation for k-fold cross-validation I have code that estimates RMSE for k-fold cross-validation and I think it is correct (from book: Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition by Aurélien Géron)

scores = cross_val_score(forest_reg, a, b, scoring="neg_mean_squared_error", cv=10)
print(pd.Series(np.sqrt(-scores)).describe())

So what about MAE? Should I use (with sqrt):

scores = cross_val_score(forest_reg, a, b, scoring="neg_mean_absolute_error", cv=10)
print(pd.Series(np.sqrt(-scores)).describe())

or this (without sqrt):

scores = cross_val_score(forest_reg, a, b, scoring="neg_mean_absolute_error", cv=10)
print(pd.Series(-scores).describe())

Also for MAE estimation, it should be  -scores or scores?

It’s an issue of units.

Compare what you’re doing in RMSE and in MAE.

RMSE is a way of getting MSE back to the original units, like how we take the square root of variance to get standard deviation. This makes more physical sense. Sure, we can make sense of square meters, but what about square dollars?

When you do MAE, you don’t have that squaring action to give you squared units. Consequently, while MSE in in square units, MAE is in the original units.

You could take the square root of MAE, but then you’d wind up with measurements with units of $\sqrt{\$}$ or the square root of whatever units you’re using. The result is that your measure of dispersion is not in the original units, which a probably what you want.

I do not see any use for taking the square root of MAE. If you do, please do share. That would be very interesting.


Which part should be frozen during transfer learning? I want to use transfer learning and fine tuning and I need to decide which part of
the original model will be used and which part will be frozen. I'm thinking about four possilbe cases:
small/large new dataset and this set is similar/not similar to the original dataset. What should be done to achieve best results in each of the cases?

Ideally I follow the below rules for performing transfer learning, I could do successful object detection using Yolo model by traning only last layer and was able to do leaf classification by training good number of layers.


Algorithm query for bank customer segmentation I've been using k-means clustering for bank customer segmentation up until now and I'm looking to explore other clustering algorithms in the banking domain. Is it a good idea to use affinity propagation algorithm for bank data? It will also be great if you can recommend me other algorithms frequently used for clustering in the banking domain.Thanks in advance.

So the question asks for other clustering algorithms which can be used in customer segmentation.
In a similar way to using Affinity propagation to identify the "optimal" number of customer segments, to gain more control over the number of customer segments, you can also use agglomerative clustering (https://www.datanovia.com/en/lessons/agglomerative-hierarchical-clustering/), which is a bottom-up clustering method. Here, you can see how customer instances cluster (using a dendrogram) and determine an "optimal" number of customer segments from this.


Find Phone Numbers in messy data I hope this isn't too basic of a question, I'm banking on the Data Science site description being true where it says "...and those interested in learning more about the field". I'm not looking for programming help, just validation that machine learning could help me with a problem.

I'm trying to find all customer phone numbers in our databases. One database has a field with free-form comments from our customer service center. Here is an obfuscated snippet:

  multiple #'s 123-456-7890 and 2345678901...current account #2233445566

As you can see, this record contains two phone numbers and also a 10 digit account number. One of the phone numbers has dashes while the 2nd doesn't. Looking for parenthesis helps, but only finds a small set. There are also other 10 digit numbers that could look like a phone number but in fact aren't.

If I run a query to return all records with a 10 digit number formatted with dashes, I get thousands of records. If I check for any 10 digit number, I get tens of thousands. So manually scanning these records to validate accurate matches is not practical.

I'm wondering if I could build a machine learning model that I could train to accurately find phone numbers in this mess. When I say "accurately", I don't mean 100%, just better than standard SQL queries. If I can, I would use this going forward to parse new data that is created in this database.

It seems to me this problem could be a good candidate for machine learning. But I'm new to machine learning, and the research I've done so far talks about different scenarios that don't seem quite the same.

In principle this seems close to a NER task, you could try to annotate a sample and train a sequence labeling model on it. However this would require quite a lot of work to get it right: annotation, then probably a good bit of trial and error to tune the right combination of features.

In a case like this I would rather go for a few carefully chosen regular expressions, they are likely to perform at about the same level without requiring as much work.


Time Series:Outlier Detection I have time series data which looks like the graph mentioned below.
I am familiar with the method of removing outliers based on the standard deviation and median values. Drawback of these methods are that they do not account for the neighboring data points.
For example, in the data show below I do not want to remove the values which are simply maximum or standard deviation away from the mean. I want to remove the points which are circled in the red. The other extreme values are common in the area should not be detected as outlier as these data points have similar nearby data points.
Is there any method to remove these points or is there any python library I can use to remove these points. Normal standard deviation and median filters do not work well for these as they also remove the points which are not circled.

You could compute mean and standard deviations in sliding windows, and use those to remove outliers.

For example, taking windows of, say, length 100, you can compute the mean and std for for these 100 successive observations, and see whether any point falls above the 3 sigma rule. In this case, the circled outlier would still be recognized as such, while the others should not, as they are not so outlying w.r.t. neighboring data (i.e. within the window that includes them and their neighboring observations).


To calculate my confusion matrix with recall and precision, my test set need to be equal(balanced)? In my CNN, I have 200 'negative' images and 50 'positive' images in my test set and I want to make a confusion matrix. My doubt is if I have to equalize the samples in the dataset because if I keep this 200 - 50 my precision falls because I have a lot of 'false positives'.

So, I have to divide the percentage of negatives by 50, or keep 200/50?

This is my results without balance the samples:

                    predicted positive         predicted negative
actual pos.              41                             9
actual neg.              31                            169

recall = 41 / 50 = 82%
precision = 41 / 72 = 57%

I don't think there is any reason to modify the matrix so keep it as it is. Even if you scale it what purpose does it serve? At the end of the day your model does not change even if you modify your confusion matrix.

In my opinion you can use other metrics e.g. f1-score (or f beta score), AUC score, etc to judge your model. Confusion matrix only provides visualization where your model "confused" and I would say it is less useful for binary classification (as you only have False positive or False negative). Metrics above serve as better judge for evaluating your model.

This is a related question which you can probably check.


Intractability in Variational Autoencoders I'm having difficulty understanding when integrals are intractable in variational inference problems.
In a variational autoencoder with observation $x$ and latent variable $z$ we want to maximize data likelihood $p_\theta (x) = \prod_{i=1}^N p_\theta (x_i)$ which is
$p_\theta (x) = \int p_{\theta_1} (z) p_{\theta_2} (x|z) dz$
We know $p_{\theta_1} (z)$ (usually a Gaussian with mean $\mu$ and covariance matrix $\Sigma$), and we also know $p_{\theta_2} (x|z)$ which is usually a Gaussian distribution with mean $\mu_z$ and covariance matrix $\Sigma_z$ modeled as a neural network. We want to maximize the likelihood with respect to $\theta = \{\theta_1, \theta_2\}$. I understand that it is not possible to analytically compute this integral to optimize for $\theta$. It is possible to obtain a sample-based approximation of the integral for a stochastic gradient ascent update. Denote by $z_l$ samples from $p(z)$ and write log likelihood to optimize:
$\nabla_{\theta_1} \log p_\theta (x_i) =  \frac{\int \nabla_{\theta_1} p_{\theta_1}(z) p_{\theta_2}(x_i|z)dz}{\int p_{\theta_1}(z) p_{\theta_2}(x_i|z)dz} \approx \frac{\frac{1}{L} \sum_{l=1}^L \nabla_{\theta_1} \log p_{\theta_1}(z_l)}{\frac{1}{L} \sum_{l=1}^L p_{\theta_2}(x_i|z_l)}$
where we used the following general property: $p_\theta (x) \nabla_\theta \log p_\theta (x) = \nabla_\theta p_\theta (x)$. As I understand, we don't like this approach because it requires sampling $L$ samples $z_l$ for every observation which is inefficient. I'm confused to see that this kind of sampling is still used later.
Now, we assume to have an encoder network $q_\phi (z|x)$ (again typically a Gaussian) and write the following objective:
$E_{z \sim q_\phi(z|x)} \big[ \log p_{\theta_2} (x|z) \big] - D_{KL}(q_\phi(z|x)||p_{\theta_1}(z))$.
In the paper, authors say that it is possible to calculate gradient updates for $\theta$ and they use a reparametrization trick to calculate updates for $\phi$. I tried to derive the updates for $\theta$ below. To optimize with respect to $\theta_1$ we can sample $z_l$ from $q_\phi(z|x_i)$
$\nabla_{\theta_1} \Big(\int q_\phi(z|x_i) [\log p_{\theta_1} (z) + \log p_{\theta_2} (x_i|z)] dz \Big) = \int q_\phi(z|x_i) \frac{\nabla_{\theta_1}p_{\theta_1}(z)} {p_{\theta_1}(z)} \approx \frac{1}{L} \sum_{l=1}^L \frac{\nabla_{\theta_1}p_{\theta_1}(z_l)} {p_{\theta_1}(z_l)}$
and similarly for $\theta_2$
$\nabla_{\theta_2} \Big(\int q_\phi(z|x_i) [\log p_{\theta_1} (z) + \log p_{\theta_2} (x_i|z)] dz \Big) = \int q_\phi(z|x_i) \frac{\nabla_{\theta_2}p_{\theta_2} (x_i|z)} {p_{\theta_2} (x_i|z)} \approx \frac{1}{L} \sum_{l=1}^L \frac{\nabla_{\theta_2}p_{\theta_2} (x_i|z_l)} {p_{\theta_2} (x_i|z_l)}$
I don't understand why the last two updates are ok but the original update is not. I'd appreciate it if you'd point out my mistakes/misinterpretation.

I believe that you got bogged down by this thought:

  I understand that it is not possible to analytically compute this integral to optimize for $\theta$.

From the paper I understood that:

$$
p_\theta (x) = \int p_{\theta} (z) p_{\theta} (x|z) dz
$$

Can be analytically solved if we simplify $p_{\theta} (x|z)$.  i.e. in the case where the Bayes' Rule for its posterior is easy.  This integral is instead intractable if the Bayes Rule for $p_{\theta} (x|z)$ turn out to be a very complex function, e.g. $(z|x)$ is very complex.  In general integrating a simple distribution is easy, conditional distributions can be (very) complex to integrate.

The reparametization trick on the other hand is what makes the paper's point.  They do not try to compute the the integrals as you derived them but they attempt to substitute all $(z|x)$ with a $g_{\phi}$ function (parametrized by $\phi$) which will build $z$ from $x$ and $\epsilon$.

The idea is that we choose $p(\epsilon)$ and instead of estimating $q_{\theta}(z|x)$ we estimate $\phi$ (in $g_{\phi}(z_{parametrized}, x$).  Estimating (SGD) a value for a parametrization of a vector function is not difficult.

I understood (analogically and very lousily) the reparametrization they do in not a very different way than what we do with, say, integrals of sphere or cylinder-like surfaces: we reparametrize the variables and recalculate the interval. Here we want the interval to not have the two sums of the KL (including both distributions) but one sum.


Recurring events - finding in a time series I have an event dataset from which I would like to detect recurring events (i.e: weekly, bi-weekly, monthly). The dataset contains:

Timestamp (date)
Event type which can get any value (e.g: purchase, withdrawal, inquiry, bid, etc...).
Amount in USD (can be bid or price for example)

My goal is to find and label recurring events and their frequency. Most of the time there's one event per day, but sometimes there are two or more events for the same day (can be with the same event type, but doesn't have to be). I should also be able to account for a change in recurrence (from bi-weekly to weekly or vice-versa).

How would you approach this?

Any help would be appreciated!

I'd suggest to use ARIMA (autoregressive iterate moving average) as a way to detect the regularities in your time series. Take a look to the following link, in which you will be introduced to ARIMA by means of series of blog posts:

http://ucanalytics.com/blogs/forecasting-time-series-analysis-manufacturing-case-study-example-part-1/


How to Classify an Image in a Class and a Subclass? I am a beginner in Machine Learning and I have a situation where an Image needs to be classified into first a super class, then a subclass.
For Example,
I have a set of images of fruits, containing images of Mangoes, Banana, Peaches, Apples etc. Then each fruit can either be fresh or rotten. I have training set containing images of rotten and fresh fruits for each Fruit type.
So for a test image I need to classify which fruit it is (Mango, Banana, etc) and then whether it is fresh or rotten.
I want to know how to approach this kind of problem

The simple solution

Have one classifier which can classify all the combinations:

Fresh banana
Rotten banana
Fresh mango
Rotten mango
...

The good solution

Or have one network with multiple outputs. See the master Thesis of Marvin Teichmann for example.
https://arxiv.org/abs/1612.07695

So one path of the network would be:

Fresh vs Rotten

The other path of the network would be

Banana vs Mango vs ...

The difference to the simple solution is that you can give a probability to the cases, e.g. for the ground truth "fresh banana" you might say 99% banana but only 42% fresh.

The obvious solution

This one should not be done: have multiple classifiers


Why does only Adam optimiser perform well in my case? I am trying to classify texts into categories (one text can have multiple categories), to do this I used one-hot encoded labels (with sklearn.preprocessing.MultiLbaleBinarizer()). My texts are also one-hot encoded but with keras.preprocessing.text.one_hot.

This is my code :

csv_data_name = 'learning_base'
df = pd.read_csv('../../datasets/csv/' + csv_data_name + '.csv ')
df['label_list'] = np.array(df.labels.str.split('|'))
df = df.drop(columns=['labels'])

# Remove incomplete lines (lines without labels)
df = df.dropna()

encoder = MultiLabelBinarizer()
labels_binary = encoder.fit_transform(df.label_list)
output_dim = labels_binary.shape[1]

vocab_size = 200000
encoded_docs = [one_hot(d, vocab_size) for d in df.text]
max_length = 200
padded_docs = pad_sequences(encoded_docs, maxlen=max_length, padding='post')

# Data division
training_set_data, testing_set_data_whole, training_set_target, testing_set_target_whole = train_test_split(
    padded_docs,
    labels_binary,
    test_size=0.1,
    random_state=0,
    shuffle=False)

model = Sequential()
model.add(Embedding(vocab_size, 200, input_length=max_length))
model.add(Flatten())
model.add(Dense(64, activation='relu'))
model.add(Dense(output_dim, activation='sigmoid'))

model.compile(optimizer=TFOptimizer(tf.train.AdamOptimizer()), loss='mean_squared_error',
              metrics=['acc'])

model.fit(training_set_data, training_set_target, epochs=50, verbose=True, batch_size=200,
          callbacks=[EarlyStopping(monitor='acc', patience=3, verbose=True), tensorboard])

Everything works fine except when I try another optimizer the learning seems incorrect, only Adam seems to work as intended.

This is my results :

|Optimizer|Time by epoch|Strict classification succes|Precision|Recall|epoch|
|-------- | ----------- |--------------------------- | ------- |------|---|
|GradientDescent                          |00:21|0%|12.5%|0%|3 (ES)|
|GradientDescent (`learning_rate = 0.01`) |00:21|0%|3.9%|16.1%|7 (ES)|
|Adadelta                                 |00:24|0%|3.1%|25.7%|3 (ES)|
|Adagrad                                  |00:26|0%|/|/|13 (ES)|
|Ftrl                                     |00:27|0%|/|/|8 (ES)|
|Momentum (`momentum = 0.01`)             |00:22|0%|/|/|18 (ES)|
|Adam (`learning_rate = 0.01` : default)  |02:37|53%|80%|60.54%|13 (ES)|

And here is the tensorboard for accuracy :

There could be a couple of possible reasons:

One reason could be the Adam optimizer is a combination of several other
optimization techniques (e.g., momentum and running average of
gradient squares). The combination of those techniques works well on
multi-label text classification.
Another reason could be that multi-label text classification is a sparse
problem. The Adam optimizer in TensorFlow has a sparse
implementation.


Why labels are not used in outlier detection algorithms? I read this article from sklearn: https://scikit-learn.org/stable/modules/outlier_detection.html

While these algorithms are very useful for outlier detection, I'm surprised to see that they are not using labels to find outliers.. Is it normal? Would it be not useful to use them?

  Outlier detection is then also known as unsupervised anomaly detection

Are there existing libraries which provide supervised outlier detections algorithms?

More explanations:

How to know if a data point is really an outlier without the label? Let say we want to predict the price of a house. An house having a lot of good features (huge size, big swimming pool, big garden, .) might be an outlier if the price is low, or as a normal point if the price is higher than average.

Outlier detection can be performed in unsupervised fashion if there are no labels, or supervised fashion if there is a training set where outliers have already been marked as such by an "expert".

Unsupervised outlier detection is often (always?) based on density. Algorithms will call observations outliers if they are too far away from most of the other observations. This is useful to detect cases which are likely to be either errors or exceptional cases; and eventually, for instance, to remove the errors from the dataset.

Supervised outlier detection is less common. Indeed, in real world applications, it is rather easy to determine what should be the normal situations (e.g. some equipment mode of operation), but quite complex to define abnormal situations. You could have examples of abnormal situations, but probably not all possible abnormal situations. Then, a supervised learning outlier detection (basically a binary classification problem with normal and abnormal classes) will learn how to detect known abnormal situations, but will likely fail on unknown ones.

This is why there is the semi-supervised outlier detection (called novelty detection in the scikit-learn example). The training dataset contains only normal points, the model is trained to define the boundaries of the normal domain. Then, at classification step, the algorithm can predict if an observation can be deemed normal or not. Semi-supervised outlier detection techniques are often based on the likelihood that observations were generated by the same unknown process which generated the learning dataset (see Bayesian networks or Gaussian mixture models for instance).

Notes:

Unsupervised and semi-supervised techniques may be combined. For instance, you could use a first unsupervised approach to remove supposed outliers, then train a semi-supervised model on the remaining observations. The resulting model will likely be more robust than if outliers had been labelled as such, and a supervised model had been trained on the full dataset.
For semi-supervised or unsupervised outlier detection, the decision boundary position depends on model hyperparameters (expected fraction of outliers, distance to inliers, acceptable likelihood, etc.)


How can you estimate the value of a component part where you only know the value of a whole? I'm not sure how to frame this question, or where to start. I'm new to data analytics, but looking to develop skills and knowledge.

An example of what I'm asking is if you have a retailers sales data where you have the total value of a given transaction, and details of each item related to a given transaction but don't have price data for each individual item, is it possible to estimate the value of each individual item given a large enough transaction data set?

The analogy breaks down a little for the use case I'm actually considering, as under this retailer example an item would likely have a fixed price. However, in my use case each item has a known arbitrary tariff value but an unknown actual value. We would only know an aggregated actual value in conjunction with other items which may be grouped under the same "transaction". Tariff and actual values will likely have a strong linear correlation with some variability, though this isn't known as actual values for individual items are not captured.

Hope this makes sense! Was wondering how define this problem, what approach would you take to a problem like this? And any links to related reading materials would be much appreciated.

Clearly this problem doesn't always have a unique solution, but if you are interested in finding one possible solution you could try a simple genetic algorithm simulation:

Each individual gene represents an item from the list of all possible items.
Each gene/item is assigned a price randomly at first (gene expression)
When a mutation is applied to a gene/item, its price is slightly modified randomly.
A crossover causes a "child gene" to take as value the mean of its two "parents genes".

This setting means that every individual in a population consists of all the items being assigned a particular price. At each generation each individual/assignment is evaluated by applying the prices assignment to the actual data and then measuring the error compared to the actual prices. Finally the top N individuals/assignments which perform the best are selected as parents for the next generation. Eventually the population should converge to realistic prices assignments.

I think this is a perfect case for a genetic algorithm because the evaluation of a potential price assignment is a very simple calculation, so there is no major efficiency issue repeating the process over many generations (as opposed to many problems where evaluation is prohibitively expensive).


Which is better: Cross validation or a validation set for hyperparameter optimization? For hyperparameter optimization I see two approaches:

Splitting the dataset into train, validation and test, and optimize the hyperparameters based on the results of training on the train dataset and evaluating on the validation dataset, leaving the test set untouched for final performance estimation.

Splitting the dataset into train and test, and optimize the hyperparameters using crossvalidation on the train set, leaving the test set untouched for final performance estimation.

So which approach is better?

The $k$-fold cross-validation (CV) process (method 2) actually does the same thing as method 1, but it repeats the steps on the training and validation sets $k$ times. So with CV the performance is averaged across the $k$ runs before selecting the best hyper-parameter values. This makes the performance and value selection more reliable in general, since there is less risk to obtain the best result by chance. However it takes much longer (since repeating $k$ times), so if the training process is long it's not always practical to use CV.


Why is XGBClassifier in Python outputting different feature importance values with the same data across different repetitions? I am fitting an XGBClassifier to a small dataset (32 subjects) and find that if I loop through the code 10 times the feature importances (gain) assigned to the features in the model varies slightly.
I am using the same hyperparameter values between each iteration, and have subsample and colsample set to the default of 1 to prevent any random variation between executions. I am using the scikit learn feature_importance_ function to extract the values from the fitted model.
Any ideas as to why this variation in feature importance could be occurring? Does this mean that some of my features may be correlated and is there a way to ensure the XGBoost outputs the same importance values each time it is called? Note that the prediction and the predicted probabilities are all constant across iterations: it is just the feature importances.
Thanks in advance!

Look at the 'max_features' parameter in the GradientBoosting Classifier in scikit learn, here: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html#sklearn.ensemble.GradientBoostingClassifier
That means that the algorithm is fitting different trees to try and account for the residual information after each sequence of trees, and that the features for each tree can be randomly permuted based on this parameter's selection.
Scikit learn does say that this can be made deterministic by setting random_state...


Order of hyperparamater tunning I'm trying to do hyperparameter tunning for random forest regression model.
My question is- is there any order I should do it? like starting with specific parameter and then move on on the other?  should I check the model each time with one paramter only or for each parameter i'm  checking I should add the ones I have found already?
Exmaple: if I run model with only n_job -1 and min_sample_leaf, and I got the best value for min_sample_leaf and now I want to check the max depth, and after that more parameters, should I put already the best value I found for min)sample_leaf? or I should check it without any other parameter and in the end create model with the best values from each one?

Since the various hyperparameter are related you cannot be sure that - for instance - a tuned value for min_samples_split from a model where all other hyperparameter are set to default values, will be generally optimal. When you have a situation where you limit the depth of trees (by max_depth assuming you use sklearn), min_samples_split may be a non-binding constraint. It may however be relevant when you set max_depth=None, since as stated in the docs:

If None, then nodes are expanded until all leaves are pure or until
all leaves contain less than min_samples_split samples.

From a practical perspective I would start tuning n_estimators (more tend to be better), max_depth, and max_features (square root of the number of features is a good starting point) with a fixed random_state. Contingent on the reults you may start adding additional constraints.


Cost Function for evaluating a Regression Model There are several "classical" ways to quantify the quality of (any!) regression models such as the RMSE, MSE, explained variance, r2, etc...

These metrics however do not take "costs" into account, for example, for me it is worse to under-predict a value (Real: 0.5, Predicted: 0.4) than to over-predict it (Real: 0.5, Predicted: 0.6).

How can I model such costs into an evaluation function? I just need a first idea to start with and will welcome any suggestions.

A loss function and cost function are the same thing.  As you intuit, classical regression treats loss/cost as symmetric, which is not always what you want. In classification tasks, you can make an asymmetric loss matrix.  You can do a similar thing with regression if you solve it with gradient descent, but the ordinary least squares has symmetric loss baked in.

So I would consider either (1) using a numeric optimization library like sklearn or tensorflow to explicitly define the regression parameters you want to estimate, write your own custom loss function, and then do parameter estimation via gradient descent, or (2) finding a software package that allows for asymmetric loss, for example see this discussion.


Gensim word2vec training error on tweets I am trying to train wor2vec embeddings on tweets. I defined the sentence class as follows:

def tokenize_tweets():
    for line in codecs.open('../data/sample_tweets.txt', encoding='utf-8'):
        tweet_text = ' '.join([token for token in tknz.tokenize(line) if token not in stopwords.words('english')])
        try:
            mod_text = tokenize(tweet_text)
            tokens = tknz.tokenize(mod_text)
            if len(tokens) > 0:
                yield tknz.tokenize(mod_text)
            else:
                yield ['NULL']
        except UnicodeEncodeError as e:
            yield ['<NULL>']

Voacb. building from this class runs fine. But when I try running the train method, I am getting the following errors:

ValueError: You must specify either total_examples or total_words, for proper alpha and progress calculations. The usual value is total_examples=model.corpus_count.

Not sure what is wrong with it.

In newer W2V version it is not enough just to write:

model_name.train(sentences)

You have to write inside word_count or total_examples.

For example, I write:

model_name.train(sentences, total_examples = token_count, epochs = model_name.iter )

where token_count = sum([len(sentence) for sentence in sentences]). And here is how I get sentences:

sentences = []
for raw_sentence in raw_sentences:

    if len(raw_sentence) > 0:
        sentences.append(sentence_to_wordlist(raw_sentence))

More documentation here.


How many time a recommender system can recommand the same item to an user? I'm working on an hybrid music recommender system project, my goal is to create recommendation playlists in accordance with users tastes.
I already implemented the first part which use a collaborative filtering algorithm, and I am now working on the content-based filtering part.

In order to make my recommender system accurate I read dozens of research papers talking about evaluating recommender system. There are many variables taken in account in those evaluations (such as coverage etc..)

Some of those papers talk about user confidence in recommender systems, defining it as one of the most important parameters to take in account. I read that for 10 items recommended if the user like something like 3 or 4 of them, it's enough to gain his confidence.

There is a point that i could'nt find in all those papers:
How many times can we recommend the same item to an user ?

To explain my question, during the playlists generation process there is a risk that the same music appear in two different playlists and I'm wondering what will be the impact on the users confidence in my recommender system in this case.

I don't think it matters if you recommend the same song in multiple playlists, just that your recommendations were accurate.

For example, imagine a user has two playlists a rock playlist and a classical playlist. If you recommended Crazy Train - Ozzy Osbourne in both playlists, you could theoretically have a "successful recommendation" in the rock playlist, and an bad recommendation in the country playlist.

The number of times a song appears is a design decision/variable - there is no right answer. You could potentially optimize on in the future using A/B testing (if I limit a song to being recommended x times, do those users then value my recommendations more than the general population?).


What do this Classification evaluation results mean to you? Do they are suspicious or not? I have collected dataset with two class labels and used the SVM Method to classify the dataset, and this is the results. Does this appear suspicious or not?

scikit-learn classifiers with SVM SVC train on 114859 instances, test on 49227 instances
Excution (Training) Time:  9.82799983025
Excution (Testing) Time:  3.75
accuracy: 0.999837487558
Precision-Recall AUC: 1.00
ROC AUC: 1.00
pos precision: 0.999822253822
pos recall: 1.0
pos F-measure: 0.999911119012
neg precision: 1.0
neg recall: 0.998107404779
neg F-measure: 0.999052806062

The statistics here are obviously very good, in fact too good for any practical data set. Your model is almost perfect... Unfortunately, it's practically useless and I'll explain.

In machine learning, if you see something like this you know you are in trouble. That can happen if there are problems with your data workflow. For example, you might have removed all outliers that you shouldn't, or you actually used a subset of your training data for the test set.

It's fine if you're just toying SVM, but you'll never encounter something like this in real life.


Unbalanced classes -- How to minimize false negatives? I have a dataset that has a binary class attribute. There are 623 instances with class +1 (cancer positive) and 101,671 instances with class -1 (cancer negative).

I've tried various algorithms (Naive Bayes, Random Forest, AODE, C4.5) and all of them have unacceptable false negative ratios. Random Forest has the highest overall prediction accuracy (99.5%) and the lowest false negative ratio, but still misses 79% of positive classes (i.e. fails to detect 79% of malignant tumors).

Any ideas how I can improve this situation?

Thanks!

Class imbalance is a very common problem.  You can either oversample the positive class (or undersample the negative) or add class weights.

Another thing to remember in this case is that accuracy is not a very useful metric here.  You might consider AUC or F1 score.

Changing your decision threshold may seem appealing, but will obviously lead to (in this case likely drastically) increased false positives (though perhaps FPs aren't as bad as FNs in the case of medical diagnosis, if tests aren't prohibitively expensive).

A more in-depth analysis of options in the case of class imbalance is provided here.


How to understand Inconsistent and ambiguous dimensions of matrices used in the Attention layer? Attention-scoring mechanism seems to be a commonly-used component in various seq2seq models, and I was reading about the original "Location-based Attention" in Bahadanau well-known paper at https://arxiv.org/pdf/1506.07503.pdf. (it seems this attention is used in various forms of GNMT and text-to-speech sythesizers like tacotron-2 https://github.com/Rayhane-mamah/Tacotron-2).

Even after repeated readings of this paper and other articles about Attention-mechanism, I'm confused about the dimensions of the matrices used, as the paper doesn't seem to describe it. My understanding is:

If I have decoder hidden dim 1024, that means ($s_{i-1}$} vector is 1024 length.
If I have encoder output dim 512, that means $h_{j}$ vector is 512 length.
If the maximum number of inputs to the encoder is 256, then the number of $j$ can be from 1 to 256.
Since $W \times S_{i-1}$ is a matrix multiply, it seems $\text{cols}(W)$ should match $\text{rows}(S_{i-1})$, but $\text{rows}(W)$ still remain undefined. The same seems true for matrices $V$, $U$, $w$, $b$.

This is page-3/4 from the paper above that describes Attention-layer:

I'm unsure how to make sense of this. Am I missing something, or can someone explain this?

What I don't understand is:

What is the dimension of previous alignment (denoted by $\alpha_{i-1}$)? Shouldn't it be total values of $j$ in $h_{j}$ (which is 256 and means total different encoder output states)?
What is the dimension of $f_{i,j}$ and convolution filter $F$? (the paper says F belongs to $k\times r$ shape but doesn't define $r$ anywhere). What is $r$ and what does $k \times r$ mean here?
How are these unknown dimensions for matrices $V$, $U$, $w$, $b$ described above determined in this model?

First to equation (7): $s_{i-1}$ is a vector, not a matrix. When you multiply it with a matrix $W$, you get another vector of which as the length of the intermediate attention projection, let us call it $d_a$. The shape of $W$ is thus $1024 \times d_a$. Similarly, the shape of $V$ is $512 \times d_a$ and bias $b$ is a vector of length $d_a$. The vector $w$ has $d_a$ dimensions and projects the intermediate projection into a scalar number.

You would probably like to compute everything with large matrix multiplications, which is possible only at training time, and which might the reason for your confusion about the dimensions.

If you have all decoder states in a matrix $S$ of shape $n \times 1024$. You can project it using $W$ into the shape $n \times d_a$.
You can also project all encoder states $H$ of shape $m \times 512$  using $V$ and get a matrix of shape $m \times d_a$ for input and output lengths $n$ and $m$.*
Now, you need to get a tensor of size $n \times m \times d_a$ for the values of $e_{i,j}$.

There is a simple way to implement it, you just reshape the first matrix to $n \times 1 \times d_a$, the second one to $1 \times m \times d_a$ and sum them together. The deep learning/linear algebra libraries will broadcast the tensor (i.e., copy along the singleton dimension), so it will give you a tensor of shape $n \times m \times d_a$. When you multiply this with $w$, you get a table of attention energies of shape $n \times m$, after applying softmax on the last dimension, you get $\alpha_{i,j}$ There is no unknown dimension anywhere.

Equation (8) is just a fancy formulation of the weighted average. In $F$, you have $k$ vectors of dimension $r$, the distribution $\alpha_i$ has length $k$:

$$f_i = \sum_{j=0}^k f_{i,j} \cdot \alpha_{i-1,j}$$

Note that $\alpha_{i-1,j}$ is scalar. This is the context vector form the previous decoding step, i.e., a weighted average of $h_j$ based on the probability distribution  $a_{i-1,j}$.

Equation (9) is basically the same as (7).


How to interpret axis of Histogram and distribution curve? After doing statistical analysis of data I got the below figures:

The data contains a line length in meters which is represented with Line(m). The purpose here is to confirm the distribution behavior of line lengths(m). Now, based on above two figures:

What kind of information x-axis and y-axis of both figures convey? Particularly for the distribution curve (pdf)?. From the first figure, it is somehow clear that it shows the percentage of the data fall within that region.
The x-axis of second figure shows the values from [0 - 80]. Does this means line length (m)?

First figure shows Frequency(Y-axis) distribution over varied values of Line data(X-axis). Similar information gets conveyed by your second figure as well, but second one provides a deeper insight to frequency fluctuation over smaller bins of Line data. Additionally in second figure, various types (Lognorm, Exponential, etc.) of distribution gets line traced on the population distribution. This additional line helps to figure out co-ordinates on this grid where there is a spike or dip in Frequency values based on Line data.
Every distribution has a spread or else mathematical computations won't be feasible for any kind of inference/analysis. For Line data, in both your plots, X-axis takes that responsibility. Unit (m specified by you) associated to values shall entirely depend on nature of your Line data feature.

Also, I remember writing a series of articles and creating a GitHub repository few years back on Statistical data visualization with Python and Seaborn which might be of some help. So please feel free to check, if required.


Pipeline heterogeneous data I'm facing an issue I don't know how to solve, but as I'm a beginner probably there is an easy solution I can't find.

I'm playing with the  titanic dataset and I want to work with pipelines (In order to avoid data leakage using cross validation). For that reason I'm using two pipelines (one for numerical, one for categorical) + FeatureUnion().

What's the problem? In the numerical pipeline I fill the NaN values of Age and then I create some buckets for that variable. The result of this pipeline would be a dataframe containing all the numerical features + 1 categorical variable. For encoding categorical variables, I use a pipeline for the categorical variables, and then use FeatureUnion to join both datasets. But the problem is that the new variable I create in the numerical pipeline doesn't go into the categorical pipeline, resulting with a dataframe with one categorical variable that hasn't been encoded. How can I solve this?

CODE:

num_pipeline = Pipeline(steps = [
('selector', DataFrameSelector(numerical_features)),
('imputer', df_imputer(strategy="median")), #Numerical
('new_variables', df_new_variables()) #Numerical
])

cat_pipeline = Pipeline(steps = [
('selector', DataFrameSelector(categorical_features)),
('label_encoder', MultiColumnLabelEncoder()) #Categorical
])

full_pipeline = FeatureUnion(transformer_list=[
("num_pipeline", num_pipeline),
("cat_pipeline", cat_pipeline)
])

Thank you for your time

Best regards

EDIT:

I was thinking about using ColumnTransformer as I think it suits better in my example as I have to apply different transformations for different columns, but the problem is that when working with ColumnTransformer the output would be an array with no columns' names, which I think would be hard to deal if we want to use feature selection. That's why I chose Pipelines rather than ColumnTransformer.

Talking about the option of creating the bucket before going into the pipeline, I can't because it's created based on the variable I'm dealing with missing values.

What would be the best option in this case?

Approach 1: create features before transforming

If you want to create a categorical variable based on a numerical variable and then treat it in cat_pipeline, you need to do create it before the column transformer.

Implement a transformer (called "bucketer" ?) that takes p variables and transforms it into p+1 (if you want to add the categorical representation and keep the initial numerical feature). This transformer is the FIRST step of your pipe.

Then, create a ColumnTransformer (I think it is more suited for your case, but have not enough details to be sure. I suggest you read this to be sure). This second transformation is the second step in your pipe.

Each branch is fed with respect to what it should output because feature creation (bucketing) was done prior to column transformer.

Approach 2: create features in transforming

Otherwise, you can create two main paths :

One branch will output only categorical features, whatever the input type.
The other will output only numerical features, whatever the input type.

You may want to re-use your feature selector in some branches, I just wanted to illustrate the two possible approaches

Hope this helps


AUC-ROC for Multi-Label Classification Hey guys I'm currently reading about AUC-ROC and I have understood the binary case and I think that I understand the multi-classification case. Now I'm a bit confused on how to generalize it to the multi-label case, and I can't find any intuitive explanatory texts on the matter.

I want to clarify if my intuition is correct with an example, let's assume that we have some scenario with three classes (c1, c2, c3).

Let's start with multi-classification:

When we're considering the multi-classification setting we look at each label separately.

So if we're looking at the ROC for label c1, we can bunch together c2 and c3 as "negatives".

I.e, when we have a sample that belongs to c1, we only look at the predictive score of c1, and build a predictive score distribution of the positive samples. Then we look at samples that belongs to either c2 and c3,i.e the bunched together negative samples and we look at their prediction scores and build a distribution of those scores as well. This results in something like the following:

Based on those distributions we can get the TPRs and FPRs based of some thresholds and calculate the ROC for c1. Then we can do the same for c2 and c3, and if we want we can average over the three ROC curves, to get an aggregated score for the problem.

That's my intuition so far anyways.

But what about the multi-labeled scenario?

Here's when things get confusing for me. Do we calculate it in the exact same manner? I understand that we still calculate the ROC for every class individually, but I'm not sure of how to think of it. Let's say that we're looking at it from the perspective of class c1. For every sample that is regarded as c1 (and possibly c2 and c3 as well), we add the model's predictive score for c1 the the distribution. But what if we encounter for example a sample that is regarded as c2 AND c3 (this can't happen in the multi-class scenario), do we think of this as TWO negative samples and add two predictive scores to the distribution?

Am I thinking in the right tracks here?

I am not sure if I understand your thinking, but my understanding is following - maybe it can help you see it from another perspective.

The multi-label classification problem with n possible classes can be seen as n binary classifiers. If so, we can simply calculate AUC ROC for each binary classifier and average it. This is a bit tricky - there are different ways of averaging, especially:

  'macro': Calculate metrics for each label, and find their unweighted
  mean. This does not take label imbalance into account.

  'weighted': Calculate metrics for each label, and find their average,
  weighted by support (the number of true instances for each label).

as explained here.

There is another option (which may be the closest to how you see it):

  'micro':
  Calculate metrics globally by considering each element of the label indicator matrix as a label.

which is implemented here.

There is also the fourth option, which I don't really understand:

  'samples':
  Calculate metrics for each instance, and find their average.

I have been using myself weighted averaging.


Proper way of fighting negative outputs of a regression algorithms where output must be positive all the way Maybe it is a bit general question. I am trying to solve various regression tasks and I try various algorithms for them. For example, multivariate linear regression or an SVR. I know that the output can't be negative and I never have negative output values in my training set, though I could have 0's in it (for example, I predict 'amount of cars on the road' - it can't be negative but can be 0). Rather often I face a problem that I am able to train relatively good algorithm (maybe fit a good regression line to my data) and I have relatively small average squared error on training set. But when I try to run my regression algorithm against new data I sometimes get a negative output. Obviously, I can't accept negative output since it is not a valid value. The question is - what is the proper way of working with such output? Should I think of negative output as a 0 output? Is there any general advice for such cases?

A standard trick is to estimate the logarithm of the desired quantity, then take its exponential, which is always positive. The drawback is that the error is optimized for the log, which treats differences in order of magnitude as equal. Another option is to do your regression as usual then project onto the feasible set (use the positive part of the output; $max(0, \cdot)$)


Sequence prediction with unlimited predictions I have a special kind of prediction problem.

I have observed $M$ sequences $X_m = [x_1, x_2, ..., x_N]$ where the distance $d$ between $x_n$ and $x_{n+1}$ is drawn from the same normal distribution, eg $d \sim N(\mu, \sigma^2)$. I can learn the parameters $\mu, \sigma$.

Now I need to predict/generate a whole sequence at once where a prediction $\hat{x}_n$ is considered correct if it falls within some absolute tolerance of the true data point $x_n$. There is one caveat: I can make as many predictions as I want without penalty but there must always be a minimum distance $\epsilon$ between predictions $\hat{x}_n$ and $\hat{x}_{n+1}$, where we can safely assume $\epsilon << d$. Intuitively, this makes me want to predict a pattern rather than trying to predict each point individually.

To re-iterate: a prediction that is outside the tolerance of a true data point is not penalized. We only need to maximize the number of correct predictions (that fall within the tolerance of a true data point).

Example 1

Prediction:       [10, 20, 30]
True observation: [11, 21, 31]
------------------------------
3 correct predictions if tolerance >= 1, else 0 correct predictions

Example 2

Epsilon = 4 (eg we can predict with a minimum distance 4)

Prediction: [6, 10, 15, 22, 30, 35]
True observation:      [11, 21, 31]
-----------------------------------
tolerance = 1 => 2 correct predictions (10, 30)
tolerance = 2 => 3 correct predictions (10, 22, 30)

What would be a good way to approach this problem? Are there problems that are similar?

Edited for clarity.

Your biggest issue with the evaluation scheme you have - "success" means within tolerance, "failure" means outside tolerance, plus your constraint on model outputs needing to vary per time step - is that it will be hard to extract gradients in order to train the prediction model directly. This rules out many simple and direct regression models, at least if you want to use "maximise number of scores within tolerance" as your objective function. The constraints on sequential predictions and allowing re-tries are also non-differentiable if taken as-is.

I think you have two top level choices:

1. Soften the loss function, and add the hard function as a metric

Use a differentiable loss function that has best score when predictions are accurate and constraints are met. For example your loss function for a single predicted value could be

$$L(\hat{x}_n, \hat{x}_{n+1}, x_{n+1}) = (\hat{x}_{n+1} - x_{n+1})^2 +  \frac{a}{1+e^{s(|\hat{x}_n - \hat{x}_{n+1}| - \epsilon)}}$$

the second constraint part is essentially sigmoid with $a$ controlling the relative weight of meeting constraints with accuracy of the prediction and $s$ controlling the steepness of cutoff around the constraint.

a. The weighting between prediction loss and constraint loss will be a hyper-parameter of the model. So you would need to include $a$ and $s$ amongst parameters to search if you used my suggested loss function.

b. You can use your scoring system, not as an objective function, but as a metric to select the best model on a hyper-parameter search.

c. With this approach you can use many standard sequence learning models, such as LSTM (if you have enough data). Or you could just use a single step prediction model that you feed current prediction plus any other features of the sequence that is allowed to know, and generate sequences from it by calling it repeatedly.

This system should encourage re-tries that get closer to the true value.

2. Use your scoring system directly as a learning goal

This will require some alternative optimising framework to gradient descent around the prediction model (although some frameworks can generate gradients internally). Genetic algorithms or other optimisers could be used to manage parameters of your model, and can attempt to change model parameters to improve results.

For this second case, assuming you have some good reason to want to avoid constructing a differentiable loss function at all, then this problem can be framed as Reinforcement Learning (RL):

State: Current sequence item prediction (or a null entry), as well as any known information such as tolerance, length of sequence, current sequence item value (which may be different from current prediction) $\epsilon$, $d$, $\mu$ or $\sigma$ can be part of the current state.
The action is to select next sequence value prediction, or probably more usefully, the offset for the next sequence item value. Using offsets allows you easily add constraint for minimum $\epsilon$
The reward is +1 for being within tolerance or 0 otherwise.
Time steps match the time steps within a current sequence.

You can use this to build a RL environment and train an agent that will include your prediction/generator model inside it. There are a lot of options within RL for how to manage that. But what RL gives you here is a way to define your goal formally using non-differentiable rewards, whilst internally the model can still be trained using gradient based methods.

The main reason to not use RL here is if the prediction model must be assessed at the end of generating the sequence. In which case the "action" might as well be the whole sequence, and becomes much harder to optimise. It is not 100% clear to me from the question whether this is the case.

Caveat: RL is a large and complex field of study. If you don't already know at least some RL, you can expect to spend several weeks getting to grips with it before starting to make progress on your original problem.

There are alternatives to RL that could equally apply, such as NEAT - deciding which could be best involves knowing far more about the project (e.g. the complexity of the sequences you wish to predict) and practical aspects such as how much time you have available to devote to learning, testing and implementing new techniques.

Have you forgotten something?

If you allow infinite re-tries, then an obvious strategy is to generate a very large sequence moving up and down using different step sizes (all greater than $\epsilon$). This doesn't require any learning model, just a bit of smart coding to cover all integers eventually. Chances are this model is only a few lines of code in most languages.

If this is to be ruled out, then some other rule or constraint is required:

Perhaps only positive increments are allowed in the predicted sequence (so we cannot re-try by subtracting and trying again)? This conflicts with your "unlimited predictions" statement.
Perhaps a sub-goal here is to make the guessing efficient? In which case RL could be useful, as you can add a a discount factor to reward processing in order make the model prefer to get predictions correct sooner.