Approximation of function with neural networks I am looking for the relevant techniques to approximate a function on $[0,1]$ with a neural network when the function has a huge amplitude?

Here is an example with a simple neural network. For test = 1 everything works well. For test = 2 then the algo "converges" towards a constant. Why?

import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt
import matplotlib

sample_size = 5000000

test = 1

if test == 1:
    learning_rate = 1e-2
    alpha = 2.
elif test == 2:
    learning_rate = 5e-10
    alpha = 2000000.

xs = np.random.normal(0.,1.,sample_size).reshape(-1,1)
ys = alpha * xs * xs

xs_test = np.arange(-3.,3.,0.01).reshape(-1,1)
ys_test = alpha * xs_test * xs_test

matplotlib.interactive(True)
fig = plt.figure()
ax = fig.gca()
ax.plot(xs_test, ys_test, 'r')
fig.canvas.draw()
fig.canvas.flush_events()

x = tf.placeholder(tf.float32, shape=(None,1))
y = tf.placeholder(tf.float32, shape=(None,1))

batch_size = 50
nb_batches = int(np.ceil(sample_size/batch_size))

with tf.name_scope('nn'):
    hidden = tf.layers.dense(x, 50, name="hidden", activation=tf.nn.elu)
    outcome = tf.layers.dense(hidden, 1, name="output")
with tf.name_scope('loss'):
    loss = tf.reduce_mean(tf.square(outcome - y))
with tf.name_scope('train'):
    optimizer = tf.train.GradientDescentOptimizer(learning_rate)
    training_op = optimizer.minimize(loss)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    for i in range(nb_batches):
        indices = slice(i*batch_size, (i+1)*batch_size)
        mini_batch_x = xs[indices]
        mini_batch_y = ys[indices]
        if i % 100 == 0:
            print('loop: ', i, 'loss:', loss.eval(feed_dict = {x: xs_test, y: ys_test}))
            ax.clear()
            ax.plot(xs_test, ys_test, 'r')
            ax.plot(xs_test, outcome.eval(feed_dict = {x: xs_test}), 'b.')
            fig.canvas.draw()
            fig.canvas.flush_events()
        sess.run(training_op, feed_dict={x: mini_batch_x, y: mini_batch_y})

Neural networks are function approximators, thus the answer is yes. Neural networks go from a set of inputs and map them to a set of outputs. That is precisely what a function does as well. In theory a sufficiently deep enough network and sufficient data coverage can approximate any function within a finite range.

Let's look at an example

import numpy as np
import matplotlib.pyplot as plt

We will approximate some arbitrary function

$y = x_1^2 + x_2^2 + x_1x_2^2 + \text{tan}(x_1)$

def func(x1, x2):
    return x1**2 + x2**2 + x1*x2**2 + np.tan(x1)

We will bound our inputs from $[-1, 1]$.

n = 10000
x_train =  (np.random.rand(n,2) - 0.5) * 2
y_train = np.zeros((n,))

for i in range(n):
    y_train[i] = func(x_train[i,0], x_train[i,1])

n = 1000
x_test =  (np.random.rand(n,2) - 0.5) * 2
y_test = np.zeros((n,))

for i in range(n):
    y_test[i] = func(x_test[i,0], x_test[i,1])

plt.scatter(x_train[:,0], x_train[:,1], c=y_train)
plt.show()

Let's now build a model.

from __future__ import print_function
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv1D, MaxPooling1D, Reshape
from keras.callbacks import ModelCheckpoint
from keras.models import model_from_json
from keras import backend as K
from keras import optimizers

input_shape = (2,)

model = Sequential()
model.add(Dense(32, activation='linear',
                 input_shape=input_shape))
model.add(Dense(64, activation='tanh'))
model.add(Dense(32, activation='tanh'))
model.add(Dense(1, activation='linear'))

sgd = keras.optimizers.RMSprop(lr=0.001)
model.compile(loss=keras.losses.mean_squared_error,
              optimizer=sgd,
              metrics=['mae'])

We will train the model

epochs = 50
batch_size = 128
# Fit the model weights.
history = model.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test, y_test))

Let's see how well we did. We will compare the test set that we have to the predicted values.

plt.scatter(x_test[:,0], x_test[:,1], c=y_test)
plt.show()

plt.scatter(x_test[:,0], x_test[:,1],
            c=model.predict(x_test).reshape(x_test.shape[0],))
plt.show()

Other functions

def func(x1, x2):
    if x1 >= 0 and x2 >= 0:
        return x1**2 + x2**2
    else:
        return x1*2 + x2*2

Test set

Prediction

def func(x1, x2):
    return np.sinc(x1 + x2)

Test set

Prediction

Try your own function with this network.


In sequence models, is it possible to have training batches with different timesteps each to reduce the required padding per input sequence? I want to train an LSTM model with variable length inputs. Specifically I want to use as little padding as possible while still using minibatches.
As far as I understand each batch requires a fixed number of timesteps for all inputs, necessitating padding. But different batches can have different numbers of timesteps for the inputs, so in each batch inputs only have to be padded to the length of the longest input-sequence in that same batch. This is what i want to implement.
What I need to do:

Dynamically create batches of a given size during training, the inputs within each batch are padded to the longest sequence within that same batch.
The training data is shuffled after each epoch, so that inputs appear in different batches across epochs and are padded differently.

Sadly my googling skills have failed me entirely. I can only find examples and resources on how to pad the entire input set to a fixed length, which is what i had been doing already and want to move away from. Some clues point me towards tensorflow's Dataset API, yet I can't find examples of how and why it would apply to the problem I am facing.
I'd appreciate any pointers to resources and ideally examples and tutorials on what I am trying to accomplish.

The answer to your needs is called "bucketing". It consists of creating batches of sequences with similar length, to minimize the needed padding.
In tensorflow, you can do it with tf.data.experimental.bucket_by_sequence_length. Take into account that previously it was in a different python package (tf.contrib.data.bucket_by_sequence_length), so the examples online may containt the outdated name.
To see some usage examples, you can check this jupyter notebook, or other answers in stackoverflow, or this tutorial.


Potential speedup by applying PCA once on dataset with m rows vs. IncrementalPCA to x batches of size m/x? I've been working on trying to perform dimensionality reduction on high-dimensional, high-volume datasets (with many rows and columns - around 100,000 - 1M rows and ~500 columns).

While the size of my dataset (m being the # of rows and n being the # of columns) doesn't prevent PCA from applying, PCA can take a fair bit of time to train and transform the data.

I was wondering whether splitting up the dataset into x batches (each of equal size m/x rows) and then partially fitting the IncrementalPCA() against each of those x batches one after the other would be faster than trying to fit a traditional PCA() against the entire dataset of size m rows?

I believe the runtime of PCA is O(m * n2) + O(n3), so effectively IncrementalPCA in this case would simply be dividing the inner factor with m/x and multiplying both terms outside by x, therefore leading to a higher overall time complexity and slower performance, with overall x * O(m/x * n2) + x * O(n3) = O(m * n2) + x * O(n3). So, naively it seems that this idea would not work.

However, could someone provide more analysis on whether such a speedup using mini-batches with IncrementalPCA is nonetheless feasible? I suspect my reasoning may be incomplete.

An alternative approach I've been trying out as well is random stratified sampling, which seems more standard, but I'm trying to combine various optimizations for achieving faster runtimes.

Incremental PCA typically only reduces the memory footprint of the algorithm. So it's used when your full dataset doesn't fit into memory. The time complexity remains the same.

If you want to speed up the computation you can look into randomized matrix decompositions. Sklearn, for example, implements it with the keyword svd_solver='randomized' where you only approximate the singular values you will ultimately keep.


how to represent location-code as a feature in machine learning model? I am trying to predict the damage to a buildings after earthquake on a dataset which contains "district number" as feature. I think the feature will have a significant importance in predicting the label but I am not sure how to best represent it.

Any thoughts?

You can get as creative as you want, but here are two general approaches that work for me.

Clustering the data into known geographical divisions and create dummy variables.  For example, in the United States a person can use zip codes.
Find the center for known clusters (ie zipcodes) or by some similar unsupervised cluster and use the longitude and latitude.

How you choose to augment that information depends on what exactly you're trying to predict.


What does SpatialDropout1D() do to output of Embedding() in Keras? Keras model looks like this

inp = Input(shape=(maxlen, ))
x = Embedding(max_features, embed_size, weights=[embedding_matrix], trainable=False)(inp)
x = SpatialDropout1D(dropout)(x)
x = Bidirectional(LSTM(num_filters, return_sequences=True))(x)
max_pool = GlobalMaxPooling1D()(x)
x = concatenate([x_h, max_pool])
outp = Dense(6, activation="sigmoid")(x)

According to the documentation the output shape = input shape i.e. (samples, timesteps, channels)

Questions

What does SpatialDropout1D() really do to the output of Embedding()? I know the output of LSTM Embedding is of dimension (batch_size, steps, features).
Does SpatialDropout1D() just randomly replace some values of word embedding of each word by 0?
How is SpatialDropout1D() different from Dropout() in Keras?

Basically, it removes all the pixel in a row from all channels.
eg: take [[1,1,1], [2,4,5]], there are 3 points with values in 2 channels, by doing SpatialDropout1D it zeros an entire row ie all attributes of a point is set to 0; like [[1,1,0], [2,4,0]]

number of such choices would be 3C0 + 3C1+ 3C2 + 3C3 = 8

The intuition behind this is in many cases for an image the adjacent pixels are correlated, so hiding one of them is not helping much, rather hiding entire row, that's gotta make a difference. reference material


Practical use of oop in R R supports a wide range of OOP designs like s3,s4,RC and others via packages,and it's a bit overwhelming to decide on which to use and a more basic question that I have is when and where do you use OOP  while doing  machine learning or data analytics ,can someone answer this from a data scientist/ML practitioner point of view .

I'm aware of how OOP works in R at a superficial level ,but

  should I invest time in learning OOP in R ?
  how practical is it from a data science point of view  ?

I have used S3, S4 and R6 (you forgot that one in your overview ;)). I would agree with Hadley Wickham that S3 is sufficient for most tasks. However, this is only required if you start building advanced functions that operate on objects. Say for example you build a model with one function and you want to create a summary and print function for the object returned by your model building function. For general Data Science purposes I would say that it helps to know the systems but none of them are very good examples of real OOP. For that I would recommend to work in Python, Ruby or Java. All have been build with OOP in mind.

In regards to knowing OOP, I think it is vital for someone involved in ML. Not when you are prototyping in R or Python but definitely when you start working on production code. I think this Quora thread gives a good run down on when OOP becomes important in ML.

If your focus is more on statistics in R it may be of less importance.


How do you map phrase IDs to sentence IDs in the Stanford Sentiment Analysis dataset? I am trying to reproduce results obtained on the Stanford Sentiment Analysis dataset, which contains sentiment annotations for syntactic components of sentences taken from Rotten Tomatoes reviews.

The data is partitioned into training, development, and test sets by sentence in the datasetSentences.txt and datasetSplit.txt files. Components within these sentences are mapped to sentiment annotations in the dictionary.txt and sentiment_labels.txt files. The README states that the sentence and phrase IDs are different, but I can't find any mapping between them, so I don't know how to split the sentiment annotations up into the partitions used in the experiment I am trying to reproduce.

Of course I would expect the mapping from phrases to sentences to be many-to-one (because the same phrase might appear in multiple sentences) but I'd still expect to be given an explicit mapping instead of having to compare substrings.

Has anybody worked with this data set? Is there something I am overlooking?

Only full sentences are used for testing and validation, though sentences and phrases are used for training. See Yoon Kim 2014, Convolutional Neural Networks for Sentence Classification for an overview of various text classification experiments.

Here is code that creates training, dev, and test .CSV files from the various text files in the dataset download.

"""
Put all the Stanford Sentiment Treebank phrase data into test, training, and dev CSVs.

Socher, R., Perelygin, A., Wu, J. Y., Chuang, J., Manning, C. D., Ng, A. Y., & Potts, C. (2013). Recursive Deep Models
for Semantic Compositionality Over a Sentiment Treebank. Presented at the Conference on Empirical Methods in Natural
Language Processing EMNLP.

https://nlp.stanford.edu/sentiment/
"""

import os
import sys

import pandas

def get_phrase_sentiments(base_directory):
    def group_labels(label):
        if label in ["very negative", "negative"]:
            return "negative"
        elif label in ["positive", "very positive"]:
            return "positive"
        else:
            return "neutral"

    dictionary = pandas.read_csv(os.path.join(base_directory, "dictionary.txt"), sep="|")
    dictionary.columns = ["phrase", "id"]
    dictionary = dictionary.set_index("id")

    sentiment_labels = pandas.read_csv(os.path.join(base_directory, "sentiment_labels.txt"), sep="|")
    sentiment_labels.columns = ["id", "sentiment"]
    sentiment_labels = sentiment_labels.set_index("id")

    phrase_sentiments = dictionary.join(sentiment_labels)

    phrase_sentiments["fine"] = pandas.cut(phrase_sentiments.sentiment, [0, 0.2, 0.4, 0.6, 0.8, 1.0],
                                           include_lowest=True,
                                           labels=["very negative", "negative", "neutral", "positive", "very positive"])
    phrase_sentiments["coarse"] = phrase_sentiments.fine.apply(group_labels)
    return phrase_sentiments

def get_sentence_partitions(base_directory):
    sentences = pandas.read_csv(os.path.join(base_directory, "datasetSentences.txt"), index_col="sentence_index",
                                sep="\t")
    splits = pandas.read_csv(os.path.join(base_directory, "datasetSplit.txt"), index_col="sentence_index")
    return sentences.join(splits).set_index("sentence")

def partition(base_directory):
    phrase_sentiments = get_phrase_sentiments(base_directory)
    sentence_partitions = get_sentence_partitions(base_directory)
    # noinspection PyUnresolvedReferences
    data = phrase_sentiments.join(sentence_partitions, on="phrase")
    data["splitset_label"] = data["splitset_label"].fillna(1).astype(int)
    data["phrase"] = data["phrase"].str.replace(r"\s('s|'d|'re|'ll|'m|'ve|n't)\b", lambda m: m.group(1))
    return data.groupby("splitset_label")

base_directory, output_directory = sys.argv[1:3]
os.makedirs(output_directory, exist_ok=True)
for splitset, partition in partition(base_directory):
    split_name = {1: "train", 2: "test", 3: "dev"}[splitset]
    filename = os.path.join(output_directory, "stanford-sentiment-treebank.%s.csv" % split_name)
    del partition["splitset_label"]
    partition.to_csv(filename)

There is a gist for the code here.


Tensorflow Adjusting Cost Function for Imbalanced Data I have a classification problem with highly imbalanced data. I have read that over and undersampling as well as changing the cost for underrepresented categorical outputs will lead to better fitting. Before this was done tensorflow would categorize each input as the majority group (and gain over 90% accuracy, as meaningless as that is).

I have noticed that the log of the inverse percentage of each group has made the best multiplier that I have tried. Is there a more standard manipulation for the cost function? Is this implemented correctly?

from collections import Counter
counts = Counter(category_train)
weightsArray =[]
for i in range(n_classes):
    weightsArray.append(math.log(category_train.shape[0]/max(counts[i],1))+1)

class_weight = tf.constant(weightsArray)
weighted_logits = tf.mul(pred, class_weight)
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(weighted_logits, y))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

This seems like a good solution for the loss function. I've had success with a similar approach recently, but I think you'd want to reorder where you multiply in the class_weight.

Thinking about it logically, the class_weight will be a constant w.r.t. the output, so it will be carried along and applied to the gradient in the same way it's being applied to the cost function. There is one problem though.

The way you have it, the class_weight would affect the prediction value. But you want it to affect the scale of the gradient. If I'm not wrong I think you'd want to reverse the order of operations:

# Take the cost like normal
error = tf.nn.softmax_cross_entropy_with_logits(pred, y)

# Scale the cost by the class weights
scaled_error = tf.mul(error, class_weight)

# Reduce
cost = tf.reduce_mean(scaled_error)

I'd be very interested to know how this performs in comparison to simply oversampling the underrepresented class, which is more typical. So if you gain some insight there post about it! :)

Interestingly I successfully used a very similar technique in a different problem domain just recently (which brought me to this post):

Multi-task learning, finding a loss function that "ignores" certain samples


Correctly evaluate model with oversampling and cross-validation I'm dealing with a classic case of dataset with binary imbalanced target (event 3%, non event 97%). My idea is to apply some sort of sampling (over/under, SMOTE etc.) to address the issue.

As I see, the correct way of doing this is to sample ONLY the train set, in order to have a test performance that is more similar to reality. Moreover, I want to use CV for hyperparameters tuning. So, the tasks in order are

Divide dataset into train-test
Perform the 5 fold-CV, as...
Sample the "training" portion of the CV
Sample the "validating" portion of the CV
Train the model on the "training"
Validate it on the "validating"
Repeat 3-6 5 times
Evaluate performances on test

My doubt is: how can I compare the CV performances with the test, since the formare are based on sampled data and the latter does not?

An idea is to skip 4 and sample only "training" portion, but in this case how can I compare the "training" with the "validating"?

EDIT: added target ratio.

I believe, the sequence for combination of CV and SMOTE should be as below.

1. Perform the 5 fold-CV ( Loop through for each fold )
2. Training Sample and Testing Sample ( for each fold )
3. Smote Training Samples
4. Train the model on the "training"
5. Prediction ( test samples )
6. Evaluate performances on test
Repeat for next fold


When does it make sense to choose gradient descent for SVM over liblinear? I understand using gradient descent methods with SVM is intractable if you've used the kernel trick. In that case, best to use libsvm as your solver.
But in the case that you are not using a kernel and simply treating it as a linear separation problem, when does it make sense to use gradient descent as your solver?
As I see it, liblinear is $O(N)$ time and doesn't require hyperparameter tuning.
In some past tests, liblinear has converged to a lower error rate at much faster rate than gradient-based methods.
Yet, Sklearn's own tests show it can be faster in many cases.
When is it optimal to use gradient-based methods with the SVM? Is it with a certain sized dataset or data that is highly linear and convex? What heuristics or explanations are available?

As mentioned in the comment, if the question is when does it make sense to use coordinate descent over stochastic gradient descent, then, one advantage with coordinate descent is that, it updates only one parameter at a time. Thus, when the data has a very very large number of features, it might make sense to use Coordinate Descent over stochastic gradient descent as SGD will try to update all model parameters at the same time. Also, for huge amounts of data CD might have lower computational complexity as against SGD. Also, it makes sense to use Coordinate descent for optimization when the function is not differentiable. Although, there are ways to convert non-differentiable functions to differentiable ones, still Coordinate descent might be one of the ways to move forward.For example, coordinate descent is a good technique to optimize a loss with L1 penalty. It is considered state of the art for Lasso regression as well as for linear SVMs.


Are Support Vector Machines still considered "state of the art" in their niche? This question is in response to a comment I saw on another question.
The comment was regarding the Machine Learning course syllabus on Coursera, and along the lines of "SVMs are not used so much nowadays".
I have only just finished the relevant lectures myself, and my understanding of SVMs is that they are a robust and efficient learning algorithm for classification, and that when using a kernel, they have a "niche" covering number of features perhaps 10 to 1000 and number of training samples perhaps 100 to 10,000. The limit on training samples is because the core algorithm revolves around optimising results generated from a square matrix with dimensions based on number of training samples, not number of original features.
So does the comment I saw refer some real change since the course was made, and if so, what is that change: A new algorithm that covers SVM's "sweet spot" just as well, better CPUs meaning SVM's computational advantages are not worth as much? Or is it perhaps opinion or personal experience of the commenter?
I tried a search for e.g. "are support vector machines out of fashion" and found nothing to imply they were being dropped in favour of anything else.
And Wikipedia has this: http://en.wikipedia.org/wiki/Support_vector_machine#Issues . . . the main sticking point appears to be difficulty of interpreting the model. Which makes SVM fine for a black-box predicting engine, but not so good for generating insights. I don't see that as a major issue, just another minor thing to take into account when picking the right tool for the job (along with nature of the training data and learning task etc).

SVM is a powerful classifier. It has some nice advantages (which I guess were responsible for its popularity)... These are:

Efficiency: Only the support vectors play a role in determining the classification boundary. All other points from the training set needn't be stored in memory.
The so-called power of kernels: With appropriate kernels you can transform feature space into a higher dimension so that it becomes linearly separable. The notion of kernels work with arbitrary objects on which you can define some notion of similarity with the help of inner products... and hence SVMs can classify arbitrary objects such as trees, graphs etc.

There are some significant disadvantages as well.

Parameter sensitivity: The performance is highly sensitive to the choice of the regularization parameter C, which allows some variance in the model.
Extra parameter for the Gaussian kernel: The radius of the Gaussian kernel can have a significant impact on classifier accuracy. Typically a grid search has to be conducted to find optimal parameters. LibSVM has a support for grid search.

SVMs generally belong to the class of "Sparse Kernel Machines". The sparse vectors in the case of SVM are the support vectors which are chosen from the maximum margin criterion. Other sparse vector machines such as the Relevance Vector Machine (RVM) perform better than SVM. The following figure shows a comparative performance of the two. In the figure, the x-axis shows one dimensional data from two classes y={0,1}. The mixture model is defined as P(x|y=0)=Unif(0,1) and P(x|y=1)=Unif(.5,1.5) (Unif denotes uniform distribution). 1000 points were sampled from this mixture and an SVM and an RVM were used to estimate the posterior. The problem of SVM is that the predicted values are far off from the true log odds.

A very effective classifier, which is very popular nowadays, is the Random Forest. The main advantages are:

Only one parameter to tune (i.e. the number of trees in the forest)
Not utterly parameter sensitive
Can easily be extended to multiple classes
Is based on probabilistic principles (maximizing mutual information gain with the help of decision trees)


Classification on time series data Context:
I am working on a classification project. where I recommend items to customers based on their past purchase history.

Question:
How will "time leakage" affect training?

Example:
Let's say that I am trying to predict today's purchases for some customer. I train only on the previous history, without any knowledge of what will be purchased today. My features consist of a set of binary variables that for any given day in history could be thought of as either recommending (1) or not recommending (0) an item (essentially acting as "dumb" classifiers themselves). For example, one of these features could be whether or not an item is in the customer's top 10 list, etc.

In this case, is it acceptable to compute top 10 over all of history and based on this list create a feature that is applied to each day in history? Of course, this will cause the features from the first day in history to have knowledge of the future.

Or would having a sliding window where feature creation that can only look into the past be more appropriate?

I don't completely understand your example, but:

if you want to put your model into operation/production environment at some day ... then you should not allow 'looking into the future' at training time.


What is RFormula in simple English? Question
Is R Forumula is to train a linear regression model?
If a response value "cancel probability" is a function of (age, height), then R Formula is to generate a model (w-age, w-height, w-bias)?
cancer_probability = w-age * age + w-loc * height + w-bias

And this is written as "RFormula":
cancer_probability ~ age + height

Is this RFormula about?
Background
Trying to understand what RFormula is. Went through multiple articles but they do not give a simple definition.
RFormula has fit and transform, hence it seems to me it is just a training algorithm, but could not find any article/book which simply says so. Hence not sure what it is for sure.

R Formulas and an Example

Suppose you’re interested in how the temperature varies with the month. Having lived through many Mays through Septembers in one place, you might guess is that the temperature generally increases in this data frame from month to month.
In this example, let’s say that Temperature depends on Month. Another way to say this is that Temperature is the dependent variable and Month is the independent variable.
An R formula incorporates these concepts and serves as the basis for many of R’s statistical functions and graphing functions.
Read the tilde operator (~) as “depends on.” Here’s how you can address the relationship between Temp and Month:
analysis <- lm(Temp ~ Month, data=airquality)

R for Data Science 3.4 Formulas and model families

In R, formulas provide a general way of getting “special behaviour”. Rather than evaluating the values of the variables right away, they capture them so they can be interpreted by the function.
The majority of modelling functions in R use a standard conversion from formulas to functions. You’ve seen one simple conversion already: y ~ x is translated to y = a_1 + a_2 * x.

R manual - Formula

The generic function formula and its specific methods provide a way of extracting formulae which have been included in other objects.

Spark 2.4.4 - RFormula

RFormula selects columns specified by an R model formula. Currently we support a limited subset of the R operators, including ‘~’, ‘.’, ‘:’, ‘+’, and ‘-‘. The basic operators are:
~ separate target and terms
+ concat terms, “+ 0” means removing intercept
- remove a term, “- 1” means removing intercept
: interaction (multiplication for numeric values, or binarized categorical values)
. all columns except target
Suppose a and b are double columns, we use the following simple examples to illustrate the effect of RFormula:

y ~ a + b means model y ~ w0 + w1 * a + w2 * b where w0 is the intercept and w1, w2 are coefficients.
y ~ a + b + a:b - 1 means model y ~ w1 * a + w2 * b + w3 * a * b where w1, w2, w3 are coefficients.

R formulas are just a common grammar for practical model specifications.

When you input a dataset and a formula like : cancer_probability ~ age + height to a model fitting function. The function will know that it has to work on age and height predictors to provide a model to give a cancer probability output.

As mentionned in the Spark link you give at the bottom, this grammar rely on some convention, as introducing an intercept by default.


Time series prediction using ARIMA vs LSTM The problem that I am dealing with is predicting time series values. I am looking at one time series at a time and based on for example 15% of the input data, I would like to predict its future values. So far I have come across two models:

LSTM (long short term memory; a class of recurrent neural networks)
ARIMA

I have tried both and read some articles on them. Now I am trying to get a better sense on how to compare the two. What I have found so far:

LSTM works better if we are dealing with huge amount of data and enough training data is available, while ARIMA is better for smaller datasets (is this correct?)
ARIMA requires a series of parameters (p,q,d) which must be calculated based on data, while LSTM does not require setting such parameters. However, there are some hyperparameters we need to tune for LSTM.
EDIT: One major difference between the two that I noticed while reading a great article here, is that ARIMA could only perform well on stationary time series (where there is no seasonality, trend and etc.) and you need to take care of that if want to use ARIMA

Other than the above-mentioned properties, I could not find any other points or facts which could help me toward selecting the best model. I would be really grateful if someone could help me finding articles, papers or other stuff (had no luck so far, only some general opinions here and there and nothing based on experiments.)

I have to mention that originally I am dealing with streaming data, however for now I am using NAB datasets which includes 50 datasets with the maximum size of 20k data points.

Statement 1 is correct, statement 2 is correct, but requires elaboration, and statement 3 is incorrect for seasonal ARIMA:

The following might point you in the right direction but hopefully you'll get a few more answers with more depth in the arena of LSTM.

You mention that you have tried both algorithms and that you are simply trying to figure out which one is better, which leads me to think you may be having more trouble with the data science process and cross validation than with the specifics of the models.

Time series in general:

Time series, in general, are difficult to forecast. If they were easy to forecast then all data scientists would be wealthy, having accurately forecast the value of all of the stocks. The reality is that hedge funds, on average, do not outperform the market and that time series forecasting is typically very poor and applies only to very short durations.  The main problems are that there is a lot of noise, there are many hidden influences, models are overly simplistic, influencers do not behave as we think they should, the interplay between linearity and nonlinearity is subtle and confusing, ... ad infinitum.

ARIMA

You are incorrect in your assessment that ARIMA requires stationary time series to forecast on.  Non-seasonal ARIMA has three input values to help control for smoothing, stationarity, and forecasting ARIMA(p,d,q), where:

p is the number of autoregressive terms,
d is the number of nonseasonal differences needed for stationarity, and
q is the number of lagged forecast errors in the prediction equation.

By contrast seasonal ARIMA has six input values ARIMA(p,d,q,P,D,Q), where:

P is the number of seasonal autoregressive terms,
D is the number of seasonal differences, and
Q is the number of seasonal moving-average
terms.

Subject to the qualifying statements above, I suggest playing with seasonal ARIMA to get a feel for the intricacies involved in smoothing, de-seasoning, de-trending, de-noiseing, and forecasting.

LSTM

I don't know enough about LSTM to add much here. I will add that red flags tend to be raised when someone begins at data science exercise with deep learning. I suggest learning as much as you can using ARIMA and then applying some of your ARIMA expertise to help you learn LSTM. Neural networks can be a very powerful tool, but they:

can take a long time to run,
often require more data to train than other models, and
have lots of input parameters to tune.

Cross validation and comparing models:

Time series are fun in that all training data can usually be turned into supervised learning training sets.  Once can simply take a time series and roll back time. That is... pick a point in time and pretend that you don't have any additional data, then produce a forecast and see how well you did.  You can march through the time series doing this $n$ times in order to get an assessment of the performance of your model and to compare models while taking the necessary precautions to prevent overfitting.

Hope this helps and good luck!


layers of transfer learning The disadvantage of using transfer learning is that it cannot be layered to reduce the number of parameters.
In that statement what are the layers of transfer learning and the number of parameters?

If you are looking for pre-trained model for your solution, you must already have a dataset which needs to be trained. However to get world class model, you need to build very deep networks, for e.g. Resnet50 has 50 layers, Resnet101 has 101 layers. Think of building those from scratch. It takes ages to train them.

So advantage of pre-trained model is you take existing model with already fitted weights and attach your last layer for new targets. Now, no. of trainable parameters should not be as large as original model. You need set layers unattainable with following code

for layer in resnet50.layers:
     layer.trainable = False

This makes only the last layer trainable. Hope this helps


Why the results from Matlab curve fitting function differ when using custom equation? I'm using Matlab's curve fitting app, When I select Power fitting it returns values which perfectly describe the data but when I use the custom equation and enter ax^b as the equation it returns very bad coefficients. Shouldn't the results be the same?

I need to fit a data to a $ax^{-6} + bx^{-4} + cx^{-2}$ function so I need to use custom equation.

EDIT: Added the screenshots.

I think your value of $y$ is too small. So, when the sum of squares is calculated it is already below the threshold, and the coefficients are accepted as good ones.

And the power function is fitted differently. $log(y) = log(a) + x*log(b)$ is a linear regression that has a direct formula. While when you input it manually it is something close to gradient decent to minimize sum is squares.

I do not know Matlab well, but you can multiply $y$ by $10^6$ and fit the curve. Then the real $a$ coefficient is $a/10^6$. Or try clicking fit options button and search the threshold value there.


How big is big data? Lots of people use the term big data in a rather commercial way, as a means of indicating that large datasets are involved in the computation, and therefore potential solutions must have good performance. Of course, big data always carry associated terms, like scalability and efficiency, but what exactly defines a problem as a big data problem?

Does the computation have to be related to some set of specific purposes, like data mining/information retrieval, or could an algorithm for general graph problems be labeled big data if the dataset was big enough? Also, how big is big enough (if this is possible to define)?

To me (coming from a relational database background), "Big Data" is not primarily about the data size (which is the bulk of what the other answers are so far).

"Big Data" and "Bad Data" are closely related. Relational Databases require 'pristine data'. If the data is in the database, it is accurate, clean, and 100% reliable. Relational Databases require "Great Data"  and a huge amount of time, money, and accountability is put on to making sure the data is well prepared before loading it in to the database. If the data is in the database, it is 'gospel', and it defines the system understanding of reality.

"Big Data" tackles this problem from the other direction. The data is poorly defined, much of it may be inaccurate, and much of it may in fact be missing. The structure and layout of the data is linear as opposed to relational.

Big Data has to have enough volume so that the amount of bad data, or missing data becomes statistically insignificant. When the errors in your data are common enough to cancel each other out, when the missing data is proportionally small enough to be negligible and when your data access requirements and algorithms are functional even with incomplete and inaccurate data, then you have "Big Data".

"Big Data" is not really about the volume, it is about the characteristics of the data.


Coordinate System's influence on $L$ distances (Manhattan and Euclidean) I don't understand this picture, which says if we change the coordinate system, we would have the same result for $L_2$ distance, whereas, our result would differ for $L_1$ distance.
What does it mean by coordinate system? $(0,0)$ if yes, the assertion is not true.

I mean, suppose we have a picture with this matrix A, and another with B, for calculating their L1(Manhattan) and L2(Euclidean) distances, we would have the following code, how is this slide applied to the proposed problem?

import numpy as np
A = [[0,21,2],[3,4,5],[6,7,8]]
B = [[5,6,37],[8,0,10],[11,12,13]]
L1 = np.zeros((3,3))
L2 = np.zeros((3,3))
C = np.zeros((3,3))
for i in range(len(A)):
    for j in range(len(A)):
        L1[i][j] = np.abs(A[i][j] - B[i][j])
        L2[i][j] = np.power((A[i][j] - B[i][j]),2)
sum(sum(L1)),np.sqrt(sum(sum(L2)))

For example, consider the green line. What is its length? In $L_2$, the answer is $1$, in $L_1$, the answer is $1$ as well.

Now, for the same line, let's rotate it $45^\circ$ counterclockwise. What is the length again? In $L_2$, its length remains to be $1$. However, in $L_1$, using Manhattan distance, it's length is now $\frac{1}{\sqrt{2}}+\frac{1}{\sqrt{2}}=\sqrt2$. As we can see, rotation can change $L_1$ distance.

Rather than rotating the line, we could also equivalently rotated our $x$ and $y$-axis instead.


Getting uniform distribution over topics from gensim's LDA? I am trying to learn topics distribution for each document in a corpus.

I have term-document matrix (sparse matrix of dim: num_terms * no_docs) as input to the LDA model (with num_topics=100)  and when I try to infer vectors for each document I am getting uniform distribution over them. This is highly unlikely since documents are of different topics.

The relevant code snippet is:

#input : scipy sparse term-doc matrix (no_terms * no_docs)

corpus = gensim.matutils.Sparse2Corpus(term_doc)

lda = gensim.models.LdaModel(corpus, 100)

vec_gen = lda[corpus]

vecs = [vec for vec in vec_gen]

Now for each vector in vecs I am getting same probability for each topic.

Can anyone point out where I am going wrong?

I solved this issue. There is a parameter for minimum probability in gensim's LDA which is set to 0.01 by default. So topics with prob. < 0.01 are pruned from output.

Once I set min. prob to a very low value the results had all topics and their corresponding probability.


Noisy behavior in deep learning feed-forward net I am a bit unsure about optimizing a neural net with 3 or more layers. The input data is quite noisy and I seem to project the noise into the learning (strong bias in the data, 90% belong to one class out of five).

However, I would like to get some feedback on the interpretation (I use 50/25/8/8/8neurons with dropout (keep_prob=0,9 after first hidden layer):

The overal logarithmic shape of your f1 score graph indicates that learning is effective and cost is heading towards some minimum. That's good. I'm assuming that the noise you're referring to is the instability of the graph after approximately 3k iterations: cost dropping and rising in a zig-zag manner.

This often hints at the learning rate being too large. Back propagation finds the right gradient but you take too big step and end up climbing rather than descending along the edge of the cost function. It's especially evident when a graph seems to oscillate around some middle value. You haven't mentioned what exact value of learning rate you're using but try to reduce it. A good starting point would be 0.01 but it depends on many factors so try to experiment.

Another issue might be a batch size: that is, how many examples contribute to the calculation of a gradient. If it's too large you might end up with an average gradient pointing in the wrong direction. And then even a small step (i.e., low learning rate) won't help. And it might again manifest itself in a zig-zag pattern. If batch size is one of the parameters try to decrease it.

The least likely issue might be the encoding architecture of your network. And especially the modest number of 8 neurons on the last layers. In this case individual neurons might have a considerable impact on the final output. And even little adjustments resulting from a single step of back propagation could potentially flip the sign of that neuron's activation value, impacting the results of other examples. Try increasing the number of neurons on the last layers. I'd personally suggest trying an architecture of 50x50x50.

Hope this helps!


Back propagation through a simple convolutional neural network Hi I am working on a simple convolution neural network (image attached below). The input image is 5x5, the kernel is 2x2 and it undergoes a ReLU activation function. After ReLU it gets max pooled by a 2x2 pool, these then are flattened and headed off into the fully connected layer. Once through the fully connected layer the outputs are converts into Softmax probabilities. I've propagated froward through the network and am now working on back propagation steps. I have taken the derivative of cross entropy and softmax, and calculated the weights in the fully connected layer.
Where I get confused is how to preform back propagation through Max pooling and then ultimately find the derivatives of the weights in the convolution layer.
What I've found online is that you need to find the derivative of the loss with respect to flattened layer, but I am unsure on how you do that.
If I could get some help with an explanation, ideally with equations it would be awesome.
Cross posted in stack overflow (https://stackoverflow.com/questions/63022091/back-propagation-through-a-simple-convolutional-neural-network)

The backpropagation algorithm attributes a penalty per weight in the network. To get the associated gradient for each weight we need to backpropagate the error back to its layer using the derivative chain rule.
Flattening layer
The derivative of a layer depends on the function that is being applied. In the case of the flattening layer it is simply reshaping (a mapping) the values. Thus no additional loss will be added at this layer. All you need to know is how the flattening occurs.
For example if the forward pass flattening is
$flatten\begin{pmatrix}
  a & b\\
  c & d
\end{pmatrix} = \begin{pmatrix}
  a \\
  b \\
  c \\
  d
\end{pmatrix}$,
then you can easily map the associated cost so far back to the $2 \times 2 \times 1$ feature map.
Max pooling layer
In the forward pass the max pooling layer is taking the maximum value in a $3 \times 3$ window that is passed along your image. For example the bold values in the first $3 \times 3$ window would have a maximum of $11$.
$maxpooling \begin{pmatrix}
  \bf{1} & \bf{2} & \bf{3} & 4 \\
  \bf{5} & \bf{6} & \bf{7} & 8 \\
  \bf{9} & \bf{10} & \bf{11} & 12 \\
  13 & 14 & 15 & 16
\end{pmatrix} = \begin{pmatrix}
  \bf{11} & 12\\
  15 & 16
\end{pmatrix}$
Thus the resulting error backpropagation would only pass through the maximum values which were passed down by the forward pass. For all other values the error term would not backpropagate. Thus the current error matrix you had backpropagating until this point would be multiplied by
$\begin{pmatrix}
  0 & 0 & 0 & 0 \\
  0 & 0 & 0 & 0 \\
  0 & 0 & 1 & 1 \\
  0 & 0 & 1 & 1
\end{pmatrix}$
Thus only 4 error terms would continue onto earlier layers.
Convolutional layers
I have gone in detail about how to do backpropagation through convolutions here: CNN backpropagation between layers.


Clustering related areas with k-means I am trying to cluster related areas of knowledge based in publications.For example a researcher has 3 keywords in a paper, in another paper he has 5 keywords, but 3 keywords are the same in the both papers. Then these 3 keywords are similar and It could be an area of knowledge. A sample of my file is:

'Domain Ontology,Semantic Web'
'Linked Data,Domain Ontology,Use Case'
'Domain Ontology,Linked Data,Semantic Annotation'
'GIS,Open GIS,Integrated Geo Systems'
'Open GIS,GIS'

My file contains 48963 rows and 19000 keywords. I have tried grouping words (like in the sample), using StringToWordVector (STWV), but I don't have good results. So I tried different K for my cluster since 3 until 13000. When K is greater my log likelihood decrease. In many models my words are grouped perfectly, but the percentages are not really good for some clusters. For example for K=3000, the 2999 have between 0,1% and 1% of the data, but the 3000 have the 21%. I am using WEKA. Anyone know some work related, some advice or any help is helpful...

Cheers.

I don't think k-means is the correct approach to this problem. You should look at doing topic modelling using latent semantic indexing.


Does auto.arima of the forecast package deal with seasonality and trend automatically I'm reading some code involving auto.arima method from the forecast package in R. What I'm curious is whether there is a necessity for decomposing the time series data into seasonal, trend and stochastic compoents before passing to the auto.arima method, or is it automatically handled by the functionality of the method?
Thanks in advance!

Yes, the aim of auto.arima is for fitting ARIMA models automatically. You do not need to decompose your time series before hand. See how the algorithm works here https://otexts.com/fpp3/arima-r.html.
You may still want to look at arguments available in the auto.arima function, and you may want to change default maximum values for p, q, and d, etc.


Not able to restore attention model properly I am referring this article on building an attention model using tensorflow.

I am trying to train a similar model on my dataset using google colab. Due to the session limit of colab and my large dataset, I need to save the model state and restore it to resume training.

However, I am not able to restore the model upon saving the parameters. I have saved the input and target tokenizers, model checkpoint and even the input and output tensors. However, every time I use checkpoint.restore and resume training the model it resumes training with a high loss(equal to random weights).

I always test my model before saving using the translate function on some test data and it generates a one line summary. However, when I restore the model and run some sample data on the translate function, I only get a single  tag as output (as if it is a newly initialised model).

Here is my code

checkpoint_dir = './training_checkpoints'
checkpoint_prefix = os.path.join(checkpoint_dir, "ckpt")
checkpoint = tf.train.Checkpoint(optimizer=optimizer,
                                 encoder=encoder,
                                 decoder=decoder)

manager = tf.train.CheckpointManager(checkpoint, 'checkpoint_dir', max_to_keep=1)

The training step is

EPOCHS = 50

for epoch in range(EPOCHS):
  start = time.time()

  enc_hidden = encoder.initialize_hidden_state()
  total_loss = 0

  for (batch, (inp, targ)) in tqdm(enumerate(dataset.take(steps_per_epoch))):
    batch_loss = train_step(inp, targ, enc_hidden)
    total_loss += batch_loss

    if batch % 100 == 0:
      print('Epoch {} Batch {} Loss {:.4f}'.format(epoch + 1,
                                                   batch,
                                                   batch_loss.numpy()))
  # saving (checkpoint) the model every 3 epochs
  if (epoch + 1) % 3 == 0:
    manager.save()

  print('Epoch {} Loss {:.4f}'.format(epoch + 1,
                                      total_loss / steps_per_epoch))
  print('Time taken for 1 epoch {} sec\n'.format(time.time() - start))

I restore by doing

checkpoint.restore('ckpt-ckptnumber.index')

I save the tokenizers (both input and output) using pickle

with open('inp_tokenizer.pickle', 'wb') as handle:
      pickle.dump(inp_lang_tokenizer, handle, protocol=pickle.HIGHEST_PROTOCOL)

I save the tensors using numpy.save()

np.save('X.npy', input_tensor)

you have to use entire things like ckpt.restore("./tf_ckpts/ckpt-10"). please check the https://www.tensorflow.org/guide/checkpoint.


Word vectors as input I have a corpus on which I want to perform sentiment analysis using LSTM and word embeddings. I have converted the words in the documents to word vectors using Word2Vec. My question is how to input these word vectors as input to Keras? I don't want to use the embeddings provided by Keras.

You can just skip the Embedding layer and use a normal input layer with n input nodes where n is the dimensions of your word2vec embeddings. The rest is the same as you would with an embedding layer, just pass a sequence of n dimensional vectors as the input, potentially padded or truncated depending on your model.


Why is the "dying ReLU" problem not present in most modern deep learning architectures? The $ReLU(x) = max(0,x)$ function is an often used activation function in neural networks.
However it has been shown that it can suffer from the dying Relu problem (see
also What is the "dying ReLU" problem in neural networks?)
Given this problem with the ReLU function and the often seen suggestion to use a leaky ReLU instead, why is it that to this day ReLU remains the most used activation function in modern deep learning architectures? Is it simply a theoretical problem that does not often occur in practice? And if so, why does it not occur often in practice? Is it because as the width of a network becomes larger the probability of dead ReLUs becomes smaller (see Dying ReLU and Initialization: Theory and Numerical Examples
)?
We moved away from sigmoid and tanh activation functions due to the vanishing gradient problem and avoid RNN's due to exploding gradients but it seems like we haven't moved away from ReLUs and their dead gradient? I would like to get more insights in to why.

In practice, dead ReLUs connections are not a major issue. Most deep learning networks can still learn an adequate representations with only sub-selection of possible connections. This is possible because deep learning networks are highly over-parameterized.
Even with the possible drawbacks backs of the dying ReLUs problem, the computational effectiveness and efficiency of ReLUs still make them one of the best options currently available.


Lightweight binary image classifier I want to build a fast binary classifier that decides if an image belongs to a given class (e.g. if it is a picture of a person). I want to do this by training a network on the RGB of pixels at a predetermined set of coordinates (e.g. 4 points, one near each corner of the image) and I want to achieve at least 75% accuracy.

How many points and what architecture should I use? Or if this is a very bad method what is another way to build a classifier that makes training and classification as fast as possible maybe on the account of lowering accuracy?

If I wanted to build a classifier that takes short time to be trained, I would rather simplify the classifier than the data. In this case, I would rather train a logistic regression model (probably with Ridge or Lasso regularization), than a very complex architecture with few of the data I have.

If I had to simplify the data, I would take the average of the three channels, thus having a black and white image as input. I would not simplify it further (I am very confident that 4 points won't give decent accuracies).

I don't really know your circumstances, but Lasso in logistic regression is a method that might allow you to select some pixels. As the coefficients obtained by Lasso are sparse (see this answer), Lasso will select the pixels that are relevant towards the prediction of the class. You can then train a bigger network with those pixels that allows you to capture nonlinearities and more complexity.


How to use Cross Entropy loss in pytorch for binary prediction? In the pytorch docs, it says for cross entropy loss:

  input has to be a Tensor of size (minibatch, C)

Does this mean that for binary (0,1) prediction, the input must be converted into an (N,2) tensor where the second dimension is equal to (1-p)?

So for instance if I predict 0.75 for a class with target 1 (true), would I have to stack two values (0.75; 0.25) on top of each other as input?

Actually there is no need for that. PyTorch has BCELoss which stands for Binary Cross Entropy Loss. Please check out original documentation here. Here is a quick example:

m = nn.Sigmoid() # initialize sigmoid layer
loss = nn.BCELoss() # initialize loss function
input = torch.randn(3, requires_grad=True) # give some random input
target = torch.empty(3).random_(2) # create some ground truth values
output = loss(m(input), target) # forward pass
output.backward() # backward pass


Plot overlapping time series I'm trying to plot my test set and test set predictions to check the differences and see how my autoencoder reconstructed the data, but since I have a test set 30x10 I have a huge visualization problem:

How can I solve it? This is the code, I'm just showing the first row (X-test[0]), but I have 29 more and I don't how to show all of them properly.
fig = plt.figure()
plt.plot(X_test[0].T, label='y_true')
plt.plot(x_test_pred[0].T, label='yhat_conv')
_ = plt.legend()
plt.show()
plt.close()

It's not going to be pretty, but you just for-loop your code:
ax, fig = plt.subplots()
for i in range(30):
    ax.plot(X_test[i].T, label='y_true_' + i)
    ax.plot(x_test_pred[i].T, label='yhat_conv' + i)
plt.show()

Because you have so many lines, it would be best to remove the legend from the plot as it will be too much information to interpret. You could use additional subplots e.g. ax,fig = plt.subplots(5,6) and plot one row on each subplot e.g. ax[1].plot(first_plot) etc.
It might be more suitable to generate some summary statistics or differences of the lines (as suggested in the comments). One way would also to plot the error bars/distributions of the lines as shaded regions and just the line of the average for particular groups, similar to the plot below:


Fine-tuning NLP models In computer vision, if we don't have a large training set, a common method is to start with a pre-trained model for some related task (e.g., ImageNet) and fine-tune that model to solve our problem.

Can something similar be done with natural language processing problems?  I have a boolean classification problem on sentences and don't have a large enough training set to train a RNN from scratch.  In particular, is there a good way to fine-tune a LSTM or 1D CNN or otherwise do transfer learning?  And, if we want to do classification on sentences, is there a reasonable pre-trained model to start from?

This paper might be useful....

https://arxiv.org/abs/1801.06146


What is the concept of Normalized Mutual Information in the evaluation of Clustering? I know what mutual information basically is but not quite sure about why and how it is used in the context of evaluation of clustering mechanisms ? Can someone please explain the intuition behind it ?
i.e, how it is defined in the case of clustering evaluation ?

This is a method for evaluation of two clusterings in the presence of class labels so it is not proper for real clustering problems in which class labels are not available.

Imagine you have class labels and you want to evaluate a clustering or (compare two clusterings). The most natural idea is to use Purity score. It simply checks labels with clusters and the best case is, of course, when each cluster contains only and only one class label. This score, however seemingly natural, has a drawback. If you consider each cluster having only one data point, then Purity is maximized! So there should be an
awareness about the number of clusters when calculating the purity score.

The next idea is calculating the Mutual Information. Mutual Information considers two splits: (1) split according to clusters and (2) split according to class labels. Then it tells you how these two splittings agree each other (how much information they share about each other or how can you know about one of them if you know the other one). Like purity, MI also gets bigger when the number of clusters is large.

Then comes NMI which is bias-corrected for the phenomenon explained above and also normalizes the score between $0$ and $1$ (MI does not have an upper bound).

NOTE: I think your question was answered in first line. If you want to evaluate clustering you are not looking for external measures where labels are needed. I just explained a bit for sake of completeness of answer.


Convolution v.s. Cross-Correlation I understand that from mathematical point of view, only difference between Convolution and Cross-Correlation is that Convolution is commutative, while Cross-Correlation is now.

In many articles Cross-Correlation is explained as measuring similarity between two signals, and Convolution is explained as calculating relationship between two signals.

My question is this, what is the point of commutative property? How does it make Convolution different from Cross-Correlation in practice? Or is it just there for "better" mathematics and easier proofs? Also, what are, if any, benefits of implementing CNN with Convolution over Cross-Correlation?

Indeed, convolution and cross-correlation are closely related. The former is a bit more natural in some areas of mathematics; most notably, in the convolution theorem for the Fourier transform, which states that the Fourier transform of the convolution of two functions is equal, under certain conditions, to the product of their Fourier transforms:
$$ \mathcal{F}\{f * g\} = \mathcal{F}\{f\} \mathcal{F}\{g\}, $$
thus providing an efficient way to compute the convolution.

In the context of CNN, there difference between the convolution and the cross-correlation is irrelevant. In the discrete two-dimensional case
$$ (f {**} g)[n_1, n_2] = \sum_{m_1 = -\infty}^{+\infty}\sum_{m_2 = -\infty}^{+\infty} f[m_1, m_2] \, g[n_1 - m_1, n_2 - m_2], $$
$$ (f' {\star\!\star} g)[n_1, n_2] = \sum_{m_1 = -\infty}^{+\infty}\sum_{m_2 = -\infty}^{+\infty} \overline{f'[m_1, m_2]} \, g[n_1 + m_1, n_2 + m_2]. $$
If $f$ is the filter and $g$ is the output of the previous layer of the CNN, you can see that the convolution is equivalent to cross-correlation with a filter
$$ f'[m_1, m_2] = \overline{f[-m_1, -m_2]}, $$
which is just a reflection around the secondary diagonal since all values are real.

In short, there are no benefits to using the true convolution in CNN. The only difference would be that the learned filters would be mirrored about the secondary diagonal.


Training a model that has both 2D and 1D features using a CNN I'm looking to pre-train a model for an RL agent but I'm having some trouble figuring some stuff out.

Dataset: Minerl MineRLNavigateDense-v0

The observation space includes :

2D screen input (64,64) (+ 3 channels of color)
1D (scalar) compass angle
1D (scalar) number of dirt blocks
+ this is all over time.

I am also given the reward based on the action the human took.

When training a model using a CNN for time series classification my understanding is that each feature (out of k total features) at a given point in time can be represented as a scalar value (visualized as each box in each row) and then the "time" component can be visualized by each row being a new timestep:

My question is:
How do you set up the training data in such a way where this sort of CNN can be used to train on, particularly because each datapoint has a different dimension, do we just flatten the screen input into 64*64*3=12288 new features? Because that feels wrong.

UPDATE:
My first problem, combining the 2D image and the scalar data points was answered on Discord group I am in. In Keras at least, there's these special layer types called "merge layers" https://keras.io/layers/merge/ A CNN can be applied to the image first, then merged with the scalar data, this could then be passed into a 1D CNN to add the temporal component, however I haven't actually done any of this yet. :)

I do not see how each datapoint would have a different dimension (you have simply 2 extra scalars, always). Could you clarify that?

Two solutions come to my head.

The first one would be to incorporate the scalars into the original picture - augment the picture with it. So that you would have, for example, an extra row for each scalar and end up with a picture of size 66*64. How to do it reasonably is another question - I do not have experience with any problem exactly like this, but using simply one pixel for the value seems pretty marginal. Maybe having the whole row of pixels be the same, representing the value... Or maybe add another dimension and filling all the points in that dimension with the representation of the scalar.

Another solution would be to create an extra branch in the model for the scalar features (with their own classifier) and then merge them with another classifier. I personally have never done it before, but this article https://www.pyimagesearch.com/2019/02/04/keras-multiple-inputs-and-mixed-data/ seems to explain it quite well. At this site, https://stackoverflow.com/questions/47818968/adding-an-additional-value-to-a-convolutional-neural-network-input/47819063 you can also find another brief example of the code (and the guy is basically proposing both ways that I mentioned above :) )

I hope this gives you at least an idea. You will need to play with it a bit to choose the best alternative.


Spyder 4: changed behavior or "run cell" / run selected code I'm a user of spyder. This weekend I updated to spyder 4, which seems to have received many usefull improvements, however I have a problem with running selected code. The logic seems to have changed.

Unfortunately for me it is very important, that I can select code lines and run them ad hoc, without copying them to the shell each time. For older versions of spyder this could be easily done by selecting the code and pressing [Ctrl]+[Enter], but now this seems to execute not just the code I selected. To be honest, I don't even know, what it executes.

The same applies if I select "run cell" from the menu after selecting code.

Can anybody shed some light into this? how can I execute selected code in spyder 4?

For me this functionality is so important, that I really think about downgrading to a lower version, even though I would lose big improvements in the editor.

[Ctrl]+[Enter] is for debugging mode and F9 is what you are looking for.
See also this stackoverflow question.


N-grams in NLP deep learning Is it real to use word's n-grams for Deep Neural Network?

E.g., sentences list contains in X_train dataframe with "sentences" column.
I use tokenizer from Keras in the next manner:

tokenizer = Tokenizer(lower=True, split=' ')
tokenizer.fit_on_texts(X_train.sentences)
X_train_tokenized = tokenizer.texts_to_sequences(X_train.sentences)

And later I use padding:

X_train_sequence = sequence.pad_sequences(X_train_tokenized)

Also I use simple LSTM network:

model = Sequential()
model.add(Embedding(MAX_FEATURES, 128))
model.add(LSTM(32, dropout=0.2, recurrent_dropout=0.2,
               activation='tanh', return_sequences=True))
model.add(LSTM(64, dropout=0.2, recurrent_dropout=0.2, activation='tanh'))
model.add(Dense(number_classes, activation='sigmoid'))
model.compile(loss='categorical_crossentropy', optimizer = 'rmsprop',
              metrics=['accuracy'])

In this case, tokenizer execution.
In Keras docs: https://keras.io/preprocessing/text/
I see character processing only, but it is nt apprepriate for my case.

My main question: Can I use n-grams for NLP tasks with deep learning (not necessary Sentiment Analysis, any abstract NLP task).

Indeed, in many tutorials or books I doesn't see any remainder n-grams for text processing, only embeddings.

For clarification: I'd like to consider not just words, but combination of words - I'd like to try it for my task.

Theoretically, you could use n-grams to model text sequences. But there are some good reasons why it's not often mentioned in textbooks.

Intuitively, modeling sequences is learning that after a, b, and c comes d. So if your sequence is:

"Hello, my name is Bob, what a lovely day today, how are you?"

It technically doesn't matter if a = "Hello", or "H", or "Hello, my name is Bob". To your model, it will just learn that after a thing, comes another thing etc.

So yes, you could use n-grams. But the problem is that you need to make sure that the way the ngrams are split is meaningful (i.e. "Hello, my" is a probably worse split than "my name is", which probably happens more often)

But don't worry, the whole point of using LSTM is that the memory part of the network will mimic using n-grams, as long as it helps with the overall performance. This means that if it needs to take more words into account to predict the next one, it will do it up to the depth of your network.


How do I properly write scipy.stats.binom.cdf() details I need to calculate the probability of my random variable being $\le 0$. It's a binomial distribution, $10000$ trials, probability of success is $\frac{10}{19}$ (roughly $0.53$).

How do I properly use the scipy.stats.binom.cdf() to do that?

I've tried the following:

stats.binom(10000, a).cdf(0)

But it gives me an answer $0$.

I feel like I might be missing something about the formula itself.

>>> from scipy.stats import binom
>>> binom.cdf(0, 10000,10/19)
0.0

Binomial random variable are nonnegative. Hence you want to compute the probability that it is equal to zero which is

$$\left(\frac{10}{19} \right)^{10000}\approx 10^{-2787.54}\approx 0$$


How to increase number of outliers in a dataset? I have a dataset with 1000 rows and 4 columns with 3 outliers .I want to add another 7 outliers related to them for detection by clustering.

     Example TO What I did
         Col1 col2 Col3 col4
     Out1 a1    b1   c1   d1
     Out2 a2    b2   c2   d2
     Out3 a3    b3   c3   d3

I get mean and std for 7 columns of normal data ten calculate

Out4  normal1+mean+stdcol1   norm1mean+stdcol2
 Out5  normal2+mean+stdcol1  norm2mean+stdcol2
 Out6 ...........

I don't know if  what i did is right or a good solution?

I don't  want outliers to be so easy for detection

Thanks

I'm assuming you want to create a point that, each column by itself appears normal, but when looking at all the columns appears as if it's an outlier (thus you'd need some sort of outlier detection). Thus the method of generating an outlier would require looking at all the dimensions in relation to each other. And since we didn't assume normality here, generating is not straightforward.

I would recommend first using some kind of outlier detection method from here on the original dataset, (Somethind like an Isolation Forest would work)

Then you can generate random numbers, (or use the numbers you generated) to test if they are outliers or not. This should be easy to do by hand since you only want 7 points and each point only has 4 dimensions. Also an additional tip would be to test the numbers using one of the methods that returns a score instead of a 0,1 prediction so that you can make sure it's not too obvious of an outlier (since you didn't want that).

Lastly, if you generated points, some sort of sanity check would be to append those points to the dataset, apply PCA to reduce the dimensions down to 2, plot the PCA result with a separate colour for the appended outlier points. And you can check by eye if the outliers are far apart but not too far apart from your dataset.

Hope this helps and gives you some ideas.


image classification with training set with 4 classes and test set with 3 classes I have to do image classificaion with a CNN, and for doing this I have been given a  training set with 4 classes and a test set with 3 classes. I am really confused because I don't know if this is going to influence my prediction. It never happened to me.

How can I deal with this? Thanks in advance.

In principle there's nothing wrong with that, since every instance in the test set is predicted individually. You will have a 4 x 3 confusion matrix, because the model might predict some false positives on the fourth class. Of course you won't be able to know if the model can correctly identify a true instance from the missing class.

It depends what is the goal:

If the model is meant to be able to predict any of the 4 classes, then it should be trained on the 4 classes and it would be preferable to also test it on the 4 classes, but testing it only on 3 should already gives a good indication of its performance.
If the model only needs to predict 3 classes ever, then the instances of the 4th class should be removed from the training set since they just make things more complex.


Why does reducing the n_estimators in RandomForestClassifier improve accuracy? I am taking a course that introduced me to sklearn.ensemble.RandomForestClassifier. At first it uses n_estimators with the default value of 10 and the resulting accuracy turns out to be around 0.28. If I change n_estimators to 15, the accuracy goes to 0.32
Here's some of the code:
pl = Pipeline([
        ('union', FeatureUnion(
            transformer_list = [
                ('numeric_features', Pipeline([
                    ('selector', get_numeric_data),
                    ('imputer', Imputer())
                ])),
                ('text_features', Pipeline([
                    ('selector', get_text_data),
                    ('vectorizer', CountVectorizer())
                ]))
             ]
        )),
        ('clf', RandomForestClassifier())
    ])

I thought that increasing the number of trees (n_estimators) in the RandomForestClassifier would give a better accuracy, but sometimes if I use a value of 100 I can get between 0.30 and 0.32. Could someone please explain? How do you find which is the smallest value for getting the highest possible accuracy?

If you are talking about testing accuracy in this case (ie you are comparing results on data you didn't train with) - it's possible that adding more estimators is overfitting on your training set and is therefore performing poorly on your holdout set. If this is the case I would recommend approaching the problem with a more basic method such as LogisticRegression - as it is less likely to overfit when compared to ensemble methods.
As for finding the best parameters - try sklearn's RandomizedSearchCV to fine-tune your hyperparameters to maximize performance.


Ordered elements of feature vectors for autoencoders? Here is a newbie question; when one trains an autoencoder or a variational autoencoder, does the order of the objects in the training vector $x$ matter?

Suppose I take an MNIST image image $(28\times28)$ and turn it into a feature vector of size $x \in \mathbb{R}^{1\times784}$. Then does it matter if I e.g. flatten the whole image vertically, or horizontally, or some other fancy way? Or if I were to scramble the order of the elements in the feature vector, would that make the VAE or AE mess up?

For a fully-connected network the precise order of features does not matter initially (i.e. before you start to train), as long as it is consistent for each example. This is independent of whether you have an auto-encoder to train or some other fully-connected network. Processing images with pixels as features does not change this.

Some caveats:

To succeed in training, you will need the pixel order to be the same for each example. So it can be randomly shuffled, but only if you keep the same shuffle for each and every example.

As an aside, you will still get some training effect from fully random shuffling the variables, because for example writing an "8" has more filled pixels than writing a "1" on average. But the performance will be very bad, accuracy only a little better than guessing, for most interesting problem domains.

To visualise what the auto-encoder has learned, your output needs to be unscrambled. You can actually input a (same shuffle each example) scrambled image and train the autoencoder to unscramble it - this will in theory get the same accuracy as training to match the scrambled input, showing again that pixel order is not important. You could also train autoencoder to match scrambled input to scrambled output and visualise it by reversing the scrambling effect (again this must be a consistent scramble, same for each example).

In a fully-connected neural network, there is nothing in the model that represents the local differences between pixels, or even that they are somehow related. So the network will learn relations (such as edges) irrespective of how the image is presented. But it will also suffer from being unable to generalise. E.g. just because an edge between pixels 3 and 4 is important, the network will not learn that the same edge between pixels 31 and 32 is similar, unless lots of examples of both occur in the training data.

Addressing poor generalisation due to loss of knowledge about locality in the model is one of the motivations for convolutional neural networks (CNNs). You can have CNN autoencoders, and for those, you intentionally preserve the 2D structure and local relationships between pixels - if you did not then the network would function very poorly or not at all.


Using Pandas to_numeric() in Azure Machine Learning Studio I am facing an issue that Azure Machine Learning Studio fails to find the to_numeric method in pandas.

After reading a .csv in AMLS I try to process it in a python script. The line that is throwing me an error is:

dataframe1['Monthly Debt'] = pd.to_numeric(dataframe1['Monthly Debt'])

pd of course is pandas, dataframe1 is my working dataframe. The error thrown is:

AttributeError: 'module' object has no attribute 'to_numeric'

Of course everything works on my local python. Do you have any idea what AMLS might be talking about?

What’s new in Pandas v0.17.0

  DataFrame.convert_objects has been deprecated in favor of
  type-specific functions pd.to_datetime, pd.to_timestamp and
  pd.to_numeric (new in 0.17.0) (GH11133).

So for Pandas versions < 0.17.0 you can and should use: df.convert_objects(convert_numeric=True)

Demo:

In [213]: x = pd.DataFrame({'a':['11', 'aaa', '0', np.nan, '123']})

In [214]: x
Out[214]:
     a
0   11
1  aaa
2    0
3  NaN
4  123

In [215]: x.dtypes
Out[215]:
a    object
dtype: object

In [216]: x = x.convert_objects(convert_numeric=True)

In [217]: x
Out[217]:
       a
0   11.0
1    NaN
2    0.0
3    NaN
4  123.0

In [218]: x.dtypes
Out[218]:
a    float64
dtype: object


How to delete entire row if values in a column are NaN I'd like to drop all the rows containing a NaN values pertaining to a column. Lets assume I have a dataset like this:

Age  Height  Weight  Gender
12    5'7     NaN     M
NaN   5'8     160     M
32    5'5     165     NaN
21    NaN     155     F
55    5'10    170     NaN

I want to remove all the rows where 'Gender' has NaN values. The output i'd like:

Age  Height  Weight Gender
12   5'7     NaN     M
NaN  5'8     160     M
21   NaN     155     F

Thanks in advance!

Well if the dataset is not too large I would suggest using pandas to clean the data. So you would need to first do

Python2

python2 -m pip install pandas

Python3

python3 -m pip install pandas

If you already have anaconda installed you can skip the above step. Next you could go through an IDE (like jupyter) or through the shell type the following commands

import pandas as pd
df = pd.read_csv("filename", dtype=str)
#or if excel file
#df = pd.read_excel("filename", dtype=str)

df = df[pd.notnull(df['Gender'])]

Then you would want to save your result in a file with

df.to_csv("newfile");


Understanding projection layer for BLSTM In many research papers there are 'projection layers' related to BLSTM layers. For example, from here:

"we trained an 8-layer BLSTM encoder including 320 cells in each layer
and direction, and the linear projection layer with 320 units followed
by each BLSTM layer"

I can't understand what this means and how it works.
Any help on this topic would be appreciated. Thanks in advance!

A "projection" is a simple linear/dense layer, that is, a matrix multiplication and a bias vector addition.
It is called projection because you "project" a representation of dimensionality $M$ into a representation space of dimensionality $N$.
Sometimes, especially for sequences or 2D data, these projections are implemented as a convolution of size 1, which is equivalent to the computations I described above.


The reason behind using a pre-trained model? These last month I have been studying all about word embeddings and the most known pre-trained word embeddings, Word2Vec, GloVe, FastText, etc. I have read many times how important It is to take advantage of pre-trained models when doing a given task however I don't understand how a pre-trained model can adapt to my given corpus. Furthermore, If I have new words not present in the pre-trained model will I be able to use this pre-trained model to learn the embeddings for the new words?

I don't understand how a pre-trained model can adapt to my given corpus

You are correct in thinking this way. It is not a magic wand. It learns the embedding values based on the underlying context of the corpus(e.g. news) which may work in the broad sense but not in a specific case.
Two cities may get the embeddings based on their geographical location but that might not be the embedding we may like if we are comparing cities on crime rate/GDP etc.
One of the best posts on Word Embedding [Blog post by Sebastian Ruder] has mentioned all the limitations in detail.
An excerpt from the post,

"...One of the major downsides of using pre-trained embeddings is that the news data used for training them is often very different from the data on which we would like to use them. In most cases, however, we do not have access to millions of unlabelled documents in our target domain that would allow for pre-training good embeddings from scratch...."

If I have new words not present in the pre-trained model will I be able to use this pre-trained model to learn the embeddings for the new words?

It will get a value from a bucket for all the out-of-vocabulary(OOV) list. Sometimes it can be the same value for all the new words. But there is some known strategy to deal with the scenario. Check the post for relevant papers.
From the same posts,

"....One of the main problems of using pre-trained word embeddings is that they are unable to deal with out-of-vocabulary (OOV) words, i.e. words that have not been seen during training. Typically, such words are set to the UNK token and are assigned the same vector, which is an ineffective choice if the number of OOV words is large...."

But this issue will prevail even if you learn embeddings with your own corpus. So I believe it is not just limited to the pre-trained embeddings.


subsetting R data frame I am subsetting some data frames and am sure there is a better way.

Essentially, I have two data frames.  The first is the actual data.  The second has some meta data and importantly a flag on whether or not the row is in the subset I am interested in.  All I would like to do is pull out the subset and write a file.

For instance, here is my subset data frame:

     head(temp[,c(1,4:8)])
     ensembl_gene_id   FGF Secreted  SigP Receptors    TF
1 ENSMUSG00000000001 FALSE    FALSE FALSE      TRUE FALSE
2 ENSMUSG00000000001 FALSE    FALSE FALSE      TRUE FALSE
3 ENSMUSG00000000001 FALSE    FALSE FALSE      TRUE FALSE
4 ENSMUSG00000000001 FALSE    FALSE FALSE      TRUE FALSE
5 ENSMUSG00000000001 FALSE    FALSE FALSE      TRUE FALSE
6 ENSMUSG00000000001 FALSE    FALSE FALSE      TRUE FALSE

Here is my actual data:

Expt1_C1_EXONS_dxd_res_wDesc[1:5,1:5]
                                   groupID featureID exonBaseMean dispersion         stat
ENSMUSG00000000001:E001 ENSMUSG00000000001      E001    624.80240 0.04271781  1.255734504
ENSMUSG00000000001:E002 ENSMUSG00000000001      E002     30.92281 0.02036015  0.514038911
ENSMUSG00000000001:E003 ENSMUSG00000000001      E003     41.61413 0.01546023 10.105615831
ENSMUSG00000000001:E004 ENSMUSG00000000001      E004    137.47209 0.03975305  0.281105120
ENSMUSG00000000001:E005 ENSMUSG00000000001      E005     85.97116 0.05753662  0.005482149`

What I was doing is:

write.table(Expt1_C1_EXONS_dxd_res_wDesc[temp$FGF,],
            "Expt1_C1_EXONS_dxd_res_wDesc_FGF.tab",col.names=T,row.names=F,quote=F)

This is taking 20+ min per subset just to subset and write the data.  The data frame is 370,000 rows by 27 variables, so large but not huge.  I have about 30 of these to do.  Is there a more efficient way?  Note, the groupID does NOT equal the first column in my subset data frame.  In some instances the groupID contains a concatenated set of ensembl ids.  So I have preprocessed to get the temp data frame to have what I want in the same row order.

Thanks,
Bob

Only reason for such a long subset time is that you're running out of memory and the OS starts to swap. AFAIK base data.frame will create a copy of the data in memory before writing it to file.

What you can do is use either dplyr or data.table packages as they manage some of these steps without copying the data.


Strange Behavior for trying to Predict Tennis Millionaires with Keras (Validation Accuracy) I'm trying to make an NN with Keras to predict the ATP players that will get more than US$1 million in prize money based on their weight and height (from a dataset I mined some weeks ago), but I have been getting a weird behavior especially for the validation accuracy. Sometimes it gets to 84-85%, which is reasonable since SVMs and GaussianNB seem to be able to hit only 83.3% at best (check this post for more info), but sometimes it gets only around 15% accuracy. And the accuracy mostly doesn't change at all if I add capacity or more epochs. I guess one of the weirdest trends in this problem is that the loss is comparably quite small when I get low accuracies:

All of the code is available in this repo (specially in ATP NN.ipynb and the atp_python_2018-08-27_1-1500.json), but the main part of it is:

model = Sequential()
model.add(Dense(20, activation = 'relu', input_shape = (2,)))
model.add(Dense(20, activation = 'relu'))
model.add(Dense(2))
model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])
model_training = model.fit(x = hw_a, y = prize_a_bin, epochs = 4, validation_split = 0.1)

At first, I thought this was happening because the validation split done by Keras would be getting somewhat improper sets that were not very consistent with the rest of the data, since the millionaires are only 10% of the whole dataset, but that doesn't seem to be the case, the weirdness persists. The only pattern I can kind of recognize is that the normal performance and the strange one are complementary, i.e., if one is 85%, the other will be 100% - 85% = 15%. Is that linked to some mistake I've made?

On a sidenote, since I'm a beginner with Keras, should I have specified the last layer's activation function? I assume he is doing either a softmax or a sigmoid by default, but it seems make no difference if I specify either of them. I wonder why with only class (either millionaire or not), you have reshape y into a to_categorical and have two units - instead of two - in the output layer.

I can't find your script hidden in your repo which is your dataset
  which is full of stuff...you should organize your work better if you
  plan to ask on SO.

This would be the reply from a veteran but I am not.

You should use binary_crossentropy as loss, and then I think you would be able to not use to_categorical.

Then the last layer should have activation sigmoid.

More suggestion

Try to use only a neuron in the last layer: Dense(1, activation='sigmoid')

The accuracy problem may be on how you split the data: maybe only samples from one class go into the validation or in training data. I suggest you use train_test_split from sklearn with fixed random_state or something similar.


How to define tidy data if there is repeated measures? Some time ago I read R for Data Sciece and there is the following definition of tidy data:

There are three interrelated rules which make a dataset tidy:
Each variable must have its own column.
Each observation must have its own row.
Each value must have its own cell.

Back then, the idea seemed quite reasonable but now the concept seems not that consistent to me. In chapter 12 it is said that table1 is tidy and this is how it looks like:

I would expect the country to be the observation unit and therefore, countries must not appear in multiple rows. I expect the following to be the tidy data table of table1:

The problem with my solution seems to be that we have values (of the year) as part of variable names. On the other hand, the dataset table1as it is suggested to be tidy has no obvious observation unit in my understanding. We could say that the combination of the two columns country and yearforms the observation unit but there is nowhere a definition of such a rather complex observation unit (not in the book and not in the publication on tidy data).

As you correctly observed, there is no assumption that an observation must be defined by a single key (variable). In this example an observation must be defined by a pair country + year, hence the correct tidy version in table1. It's not a complex case at all, this is very common and sometimes with more than two variables.

In general "tidying" a dataset increases the number of rows and often decreases the number of columns. A way to see that your second table is not tidy is that it would require new columns for every new year added to the dataset. As you noticed another indication is simply that it requires variables values in the columns names, which is a very bad design idea in general.

This being said, tidy data isn't a magical solution to every data design problem: it tends to demultiply the number of rows to an extent which makes it impractical in some cases.


How can I improve the recall of a certain class in a multiclass-classification result I am working on a multiclass classification which is to assign medical related queries of web search to certain departments of hospital.My classifier is based on the fastText.

I found for most conditions, the result is good enough say recall is 0.8 for Nephrology. However, for just one department, Dermatology, the recall is pretty low,like 0.5. Unfortunately, this label has most samples in the test data.

How can I improve the recall of one class while maintain the performance of other classes? Will ensembling method work?

Recall is the ability of a search model to find the correctly labeled items amongst all the items for a given query.

One common method to improve specific query results is to create a custom model. Create a model just for "Dermatology". That model can be tuned to increase recall without impacting other queries that can use the generic search model.


Should you cluster before performing collaborative filtering? So I am building a recommendation model using customer and product information. This will be done via implicit, that is, a customer has a product or not as we don't have rating information about products.
I want the model to recommend a product using collaborative filtering so I am thinking of using a recommender model like Matrix factorisation/ SVD etc. I will perform multiple tests/experiments to determine the best algorithm to go with.
Should I perform clustering first to determine similar customers based on their demographic and geographic information before the recommender? Then subsequently should I train a recommender model for each cluster?
Would there be any benefit in this approach or would the recommender model be able to naturally determine similar customers?

Short Answer: No, you should not perform clustering before doing matrix factorisation.
First, I just want to say SVD is a special case of Matrix Factorisation. Another thing, SVD in recommendations is not traditional SVD, but has a different form.
Detailed Answer for why (in general) you should not perform clustering before doing matrix factorisation
Matrix Factorisation (MF) itself performs clustering. When you perform Matrix Factorisation, you end up with latent vectors for user and items. By running a clustering algorithm, such as K-Means, on these latent vectors, you end up doing clustering. But, if your end goal is just to recommend, then, there is no point performing clustering apriori.
In some corner cases, you may consider performing clustering

Do you want to cluster to understand the data better and build different recommendation models for every different cluster? If yes, then you may consider doing clustering before hand.


Ideas for prospect scoring model I have to think about a model to identify prospects (companies) that have a high chance of being converted into clients, and I'm looking for advice on what kind of model could be of use.

The databases I will have are, as far as I know (I don't have them yet), the list of current clients (in other words, converted prospects) and their features (size, revenue, age, location, stuff like that), and a list of prospects (that I have to score) and their features. However, I don't think I'll have a list of the companies that used to be prospects but for which the conversion to clients failed (if I had, I think I could have opted for a random forest. Of course I could still use a random forest, but I feel it would be a bad idea to run a random forest on the union of my two databases, and treat the clients as converted and the prospects as non-converted...)

So I need to find, in the list of prospects, those who look like the already existing clients. What kind of model can I use to do that ?

(I'm also thinking about things such as "evaluating the value of the clients and apply this to the similar prospects", and "evaluating the chance each prospect has of going out of business" to further refine the value of my scoring, but it's kinda out of the scope of my question).

Thanks

I faced almost exactly the same scenario a year and a half ago -- basically what you have is a variation of the one-class classification (OCC) problem, specifically PU-learning (learning from Positive and Unlabelled data). You have your known, labelled positive dataset (clients) and an un-labelled dataset of prospects (some of which are client-like and some of which are not client-like). Your task is to identify the most client like of the prospects and target them... this hinges on the assumption that prospects that look most like clients are more likely to convert than prospects that look less like clients.

The approach we settled upon used a procedure called the Spy-technique. The basic idea is that you take a sample from your known positive class and inject them into your unlabelled set. You  then train a classifier on this combined data and then run the unlabelled set back through the trained classifier assigning each instance a probability of being a positive class member. The intuition is that the injected positives (so-called spies) should behave similarly to the positive instances (as reflected by their posterior probabilities). By setting a threshold this allows you to extract reliable negative instances from the unlabelled set. Now, having both positive and negative labelled data, and you can build a classifier using any standard classification algorithm you choose. In essence, with the spy technique, you boot-strap your data to provide you with the needed negative instances for proper training.

For starters you should look into the work of Li and Liu who have a number of papers exploring the topic of OCC and PU-learning.


Imbalanced training set vs smaller balanced training set? Say I am using a maximum likelihood approach and my output unit computes a softmax function. My training set is distributed as follows over 6 classes:
class_samples[0]=23, class_samples[1]=5, class_samples[2]=44,
class_samples[3]=14, class_samples[4]=19, class_samples[5]=31

What should I do?

use the training set as given above with a normalizing weight balancing(e.g. using sklearn.utils.class_weight.compute_class_weight).

or should I simply use the minimum number of samples in a class(i.e. 5) to extract a balanced distribution of examples?

Why should I choose one over the other? Intuitively, I would think that using as many training examples as possible is the better option. However, I have tried to do some computations but I fail to show that usage of all examples with a normalizing weight balancing is better.
I have of course tried to do some heavy research but for some reason I cannot find the answer. If you know a good article, I would accept a reference as an answer, just as I would accept a "self-made" answer!

Why should I choose one over the other?

You sould prepare a common validation set out of your dataset and try out each and every method on your dataset.
Following are the methods that I know to handle imbalanced datasets. -

Use weighted cross-entropy loss (as you mentioned)

You can assign weights to your loss such that it will penalize more to the smaller classes and the less to larger classes. Many frameworks have a very easy way to do this.
In Scikit-learn you can look out for class_weight parameter. For eg - random forest
Here is how you can use this in Pytorch
Here is how you can use this in Keras

Use focal loss

Originally proposed for object detection, but we can also use this for any other use case. Read more about it here
Here is how you can use this in Pytorch for multi-class classification
Here is how you can use this in Keras

Over Sampling and Under Sampling

There are so many techniques in this, check out imblearn a dedicated library just to deal with imbalanced datasets.

Create a separate model for small classes

If you have some classes that have very small number of instances, you can consider creating a separate classifier for these small classes (called small_classifier for eg). You can group together these small clases under a single class (called small_class for eg) so that your main classifier will classify small_class with all other big classes in the dataset. And if your main classifier encounters any instance of small_class, it will pass it to small_classifier, which will predict the actual class for the small_class instance. This technique can give you accuracy boosts are now main classifier does not need to deal with very small classes, and insted small_classifier will be looking just at these small classes.