Why not use max(returns) instead of average(returns) in off-policy Monte Carlo control? As I understand it, in reinforcement learning, off-policy Monte Carlo control is when the state-action value function $Q(s,a)$ is estimated as a weighted average of the observed returns.

However, in Q-learning the value of $Q(s, a)$ is estimated as the maximum expected return.

Why is this not used in Monte Carlo control?

Suppose I have a simple 2-dimensional bridge game, where the objective is to get from a to b. I can move left, right, up or down.
Lets say a reward of +1 is given for reaching b and -1 otherwise.

a|x|x|x|x|b

If the agent reach the final x (using epsilon-greedy policy for instance), then goes up to get a reward of -1. Then in the next episode, I reach the final x and go right to get a reward of +1.

Why wouldn't I update all the steps leading to 'b' to +1, as I would do in Q-learning?

I'm assuming the environment is deterministic in this case, so I don't overestimate $Q(s,a)$ based on an unlikely event.

Monte Carlo methods can be incremental in an episode-by-episode sense, but not in a step-by-step (online) sense. The term Monte Carlo is often used more broadly for any estimation method whose operation involves a significant random component. It is used specically for methods based on averaging complete returns.

An obvious way to estimate it from experience, then, is simply to average the
returns observed after visits to that state. As more returns are observed, the average should converge
to the expected value. This idea underlies all Monte Carlo methods.


State-of-the-art Python packages that can evaluate language similarity I am trying to evaluate the likelihood of generating a specific sentence out of a large set of sentences. To do this, I start from a simple approach: training a custom n-gram language model and calculating the perplexity values for a list of sentences.
I found that the package KenLM (https://www.aclweb.org/anthology/W11-2123/) was often used to do this task. However, it's kind of old (published in 2011).
On the other hand, I noticed that the two most famous state-of-the-art NLP packages, BERT and GPT-2, are both about pre-trained models.
I wonder if there is any package newer than KenLM suitable for this kind of likelihood evaluation task.

I suggest you use the Hugging Face implementation which has all the state of the art language models, and fine tune them on your dataset. They have easy to use APIs for finetuning which are same across all the LM models.


Performance issues when merging two dataframe columns into one on millions rows with Pandas I am trying to merge two address columns into one and separate the resulting string with '--'. The dataset has 10 million rows and 33 columns - but the number of rows grows for a million or so a month. This line in Pandas/Python is very slow. Any ideas of how can I make it faster and scalable for future use?

df['address'] = df[['address1', 'address2']].apply(lambda x: '--'.join(x.dropna().astype(str).values), axis=1)

With my solution, you have to parse your column with string type:

First:

df[["address1", "address2"]] = df[["address1", "address2"]].dropna().astype(str)

Then define concatenation fonction to concatenate two strings

def concat_string(a, b):
    return a + '--' +  b

Finally I advise you to work with pandas series, it will be improve your operations.

Here I apply combine (from pandas series) with in param the function define above concat_string. concat_string takes in param element from df['address1'] and combine them with df['address2'].

Combine vectorize the operation, it "replaces" traditional loop.

df['address'] = df['address1'].combine(df['address2'], concat_string)

Benchmark:

df.shape => (10000, 2)

%time df['address'] = df[['address1', 'address2']].apply(lambda x: '--'.join(x.dropna().astype(str).values), axis=1)
CPU times: user 4.6 s, sys: 18.8 ms, total: 4.62 s
Wall time: 4.64 s

%time df['address_Test'] = df['address1'].combine(df['address2'], concat_string)
CPU times: user 302 ms, sys: 27.6 ms, total: 329 ms
Wall time: 321 ms


Setting sparse=True in Scikit Learn OneHotEncoder does not reduce memory usage I have a dataset that consists of 85 feature columns and 13195 rows. Approximately 50 of these features are categorical features which I encoded using OneHotEncoder. I was reading this article about sparse data sets and was intrigued to see how changing the value of the sparse parameter when defining a OneHotEncoder object may reduce memory usage for my dataset.
Before applying OneHotEncoding to categorical features in my dataset, I have a memory usage of 9.394 MB. I found this by running this code:
    BYTES_TO_MB_DIV = 0.000001
    def print_memory_usage_of_data_frame(df):
        mem = round(df.memory_usage().sum() * BYTES_TO_MB_DIV, 3)
        print("Memory usage is " + str(mem) + " MB")

    print_memory_usage_of_data_frame(dataset)

Setting OneHotEncoder spare=True:
    from sklearn.compose import ColumnTransformer
    from sklearn.preprocessing import OneHotEncoder
    from sklearn.impute import SimpleImputer
    from sklearn.pipeline import Pipeline

    numeric_transformer = Pipeline(steps=[
        ('knnImputer', KNNImputer(n_neighbors=2, weights="uniform")),
        ('scaler', StandardScaler())])

    categorical_transformer = Pipeline(steps=[
        ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),
        ('onehot', OneHotEncoder(handle_unknown='ignore', sparse=True))])

    preprocessor = ColumnTransformer(transformers=[
        ('num', numeric_transformer, selector(dtype_exclude="object")),
        ('cat', categorical_transformer, selector(dtype_include="object"))
    ])

    Z = pd.DataFrame(preprocessor.fit_transform(X))
    print_memory_usage_of_data_frame(Z)

Memory usage is 25.755 MB
Then running the same code above but setting spare=False like so:
    OneHotEncoder(handle_unknown='ignore', sparse=False)

Memory usage is 25.755 MB
According to the linked article, which used the sparse option in pandas get_dummies, this should result in reduced memory storage, is this not the same for Scikit Learn's OneHotEncoder?

Based on @BenReiniger's comment, I removed the numeric portion from the ColumnTransformer and ran the following code:

    from sklearn.compose import ColumnTransformer
    from sklearn.preprocessing import OneHotEncoder
    from sklearn.impute import SimpleImputer
    from sklearn.pipeline import Pipeline

    categorical_transformer = Pipeline(steps=[
        ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),
        ('onehot', OneHotEncoder(handle_unknown='ignore', sparse=True))])

    preprocessor = ColumnTransformer(transformers=[
        ('cat', categorical_transformer, selector(dtype_include="object"))
    ])

    X = pd.DataFrame(preprocessor.fit_transform(X))

    print_memory_usage_of_data_frame(X)

The result was Memory usage is 0.106 MB,
Running the same code above but with sparse option set to False:
    OneHotEncoder(handle_unknown='ignore', sparse=False)

resulted in Memory usage is 20.688 MB.
So it is clear that changing the sparse parameter in OneHotEncoder does indeed reduce memory usage.


Do I load all files at once or one at a time? I currently have $1700+$ CSV files. Each of them is in the same format and structure, give or take a row or possibly a column at the end. Each CSV is $\approx 3.8$ MB.
I need to perform a transformation on each file

Extract one data set, perform a summation and column select, then store inside a folder.
Extract an array, no need for column names here, then sore inside a folder.

From an algorithmic POV, is it better to load each file, perform the actions needed and then move on to the next file?
OR
Do I load all files, perform the action on all files and then store to hard drive?
I know how to do the actual process, I am after a 20,000 feet POV of dynamical programming/optimisation.
Thanks.

Since the process can be run independently on every file/batch, I would tend to recommend processing each file one by one for the sake of scalability:

Depending on the details of the task there could be some minor optimizations which can be done only by processing the whole data at once, like things related to I/O and memory usage. I can't think of any significant improvement obtainable this way.
By itself processing the files independently won't bring any performance improvement either, but it has a massive advantage in terms of scalability: the process can be distributed easily by using multiple cores and processing any number of files on each core. This option offers flexibility for processing the current data (if you can afford multiple cores/machines, the process can be faster) and importantly any future version of the data, even if it grows very large.


weighted cross entropy for imbalanced dataset - multiclass classification I am trying to classify images to more then a 100 classes, of different sizes ranged from 300 to 4000 (mean size 1500 with std 600). I am using a pretty standard CNN where the last layer outputs a vector of length number of classes, and using pytorch's loss function CrossEntropyLoss.

I tried to use $weights = \frac{max(sizes)}{sizes}$ for the cross entropy loss which improved the unweighted version, not by much.

I also thought about duplicating images such that all classes ends up to be of the same size as the larges one.

Is there any standard way of handling this sort of imbalance?

If you are looking for just an alternative loss function:

Focal Loss has been shown on imagenet to help with this problem indeed.

Focal loss adds a modulating factor to cross entropy loss ensuring that the negative/majority class/easy decisions not over whelm the loss due to the minority/hard classes.

I would look into using that is it seems to be really promising.

Link To Focal Loss Paper:  https://arxiv.org/pdf/1708.02002.pdf


Algorithms to get intent of an article? I am currently working on a project where my job is to intent analysis of an article. Suppose i am given with the article and i want to classify that what kind of article it is?. Any suggestions, Which kind of model would be best to apply?

I think the easiest way would be to cluster them with wordvectors.
The "20 Newspaper" dataset would be a good test for the algorithm
http://qwone.com/~jason/20Newsgroups/

On the Sklearn Website, you will find some easy example for using wordvectors and classification:
http://scikit-learn.org/stable/datasets/twenty_newsgroups.html

If you go deeper into this space you could start using LSTM networks and neural networks, but this is a little bit harder to develop. So first, try to use the problem statistically. If you want to analyze the text semantically you will need LSTM.

I hope I could help you.


Attributes extraction from unstructured product descriptions I am trying to match new product description with the existing ones. Product description looks like this: ￼Panasonic DMC-FX07EB digital camera silver. These are steps to be performed:

Tokenize description and recognize attributes: Panasonic => Brand, DMC-FX07EB => Model, etc.
Get few candidates with similar features
Get the best candidate.

I am having problem with the first step (1). In order to get 'Panasonic => Brand', DMC-FX07EB => Model, silver => color, I need to have index where each token of the product description correspond to certain attribute name (Brand, model, color, etc.) in the existing database. The problem is that in my database product descriptions are presented as one atomic attribute e.g. 'description' (no separated product attributes).

Basically I don't have training data, so I am trying to build index of all product attributes so I can build training data. So far I have attributes from bestbuy.com and semantics3.com APIs, but both sources lack most of attributes or contain irrelevant ones. Any suggestions for better APIs to get product attributes? Better approach to do this?

P.S. For every product there is a matched product description in the Database, which is as well in a form of one atomic attribute. I have checked this question on SO, it helped me and it seems we have same approach but I am still trying to get training data.

Left you a quick response on SO. The gist is that you can collect a lot of information from electronics shops and manufacturers' web sites, and lots you can annotate manually. If your goal is to only get training data, that's all you need:

My answer form the cross-post:
"Having developed a commercial analyzer of this kind, I can tell you that there is no easy solution for this problem. But there are multiple shortcuts, especially if your domain is limited to cameras/electronics.

Firstly, you should look at more sites. Many have product brand annotated in the page (proper html annotations, bold font, all caps in the beginning of the name). Some sites have entire pages with brand selectors for search purposes. This way you can create a pretty good starter dictionary of brand names. Same with product line names and even with models. Alphanumeric models can be extracted in bulk by regular expressions and filtered pretty quickly.

There are plenty of other tricks, but I'll try to be brief. Just a piece of advice here: there is always a trade-off between manual work and algorithms. Always keep in mind that both approaches can be mixed and both have return-on-invested-time curves, which people tend to forget. If your goal is not to create an automatic algorithm to extract product brands and models, this problem should have limited time budget in your plan. You can realistically create a dictionary of 1000 brands in a day, and for decent performance on known data source of electronic goods (we are not talking Amazon here or are we?) a dictionary of 4000 brands may be all you need for your work. So do the math before you invest weeks into the latest neural network named entity recognizer."


Intuition behind the fact that SVM uses only measure of similarity between examples for classification I have read about SVM and although I did not understand the math behind it completly, I know that it produces decision plane with maximum margin between examples of different classes and role of support vectors in the process.
I also know that SVM is a kind of dual learing algorithm(algorithms that operate only using the dot product between examples). It uses kernel functions to calculate dot product(measure of similarity) between training examples.

What I want to understand in simple terms is that: Suppose I have a similarity matrix of all training examples specifying amount of similary between any(all) two examples in training sample. How Can I make a classifier or cluster based only on this information?

You can try running SVM just on this similarity matrix. But you'll then need to provide the sikikaritirs also for new data points.

Furthermore, SVMs rely on the similarities bring dot products in some vector space. If they aren't you may get inconsistencies.
They may rely on the triangle inequality being satisfied for the distance function d(x,y)=sqrt(2k(x,y)-k(x,x)-k(y,y)). Although I cannot find a clear reference on whether or not this is needed. If k is a scalar product in some vector space, this obviously is satisfied.

Last but not least, SVMs are good for larger amounts of data, where you cannot afford to keep the entire similarity matrix in memory! By reducing the data set to the support vectors only, the resulting classifier will need much less memory and much less time. Much of the challenge of learning a SVM is to manage memory during training.


Naming conventions for dataframes I often find myself writing code like the following (oversimplfied example)

df = read_csv('customer_data_export.csv')
df2 = df.query("date > '2017-01-10'")
data = df_filtered.groupby('transaction_id').sum()
plot_data = pivot_table(data, columns='weekday', rows='n_items')
# Etc etc

Basically the problem is that while it's relatively easy to come up with semantic names for columns (as random variables) it's hard for me to come up with meaningful names for each step of transformed dataframe. Additionally I prefer to have short names to make the code easier to type. (Working in Jupyter notebook, the tab-completion isn't the best).

What are some best practices that people follow with this kind of thing?

Why not give them a name describing their purpose?

df_csv = read_csv('customer_data_export.csv')
df_date_filtered = df.query("date > '2017-01-10'")
df_grouped_by_trans_id = df_date_filtered.groupby('transaction_id').sum()

#cleanup
rm(df_csv, df_date_filtered)

plot_data = pivot_table(df_grouped_by_trans_id, columns='weekday', rows='n_items')


How do I simultaneously select multiple values for k-means in WEKA? I have tried WEKA's Experimenter. However, it's for classification.

I'm looking for a way to apply the k-means algorithm on the same dataset but with multiple 'k' values. Is there any option in WEKA's GUI allow this?

Try writing a program.

It's easy to do a for loop over k.

You will also want to use multiple random seeds.


Problem with overfitting for a CNN I am doing image classification with a CNN and I am having trouble building a network that does not do overfitting. I have in my training set 2000 images of 4 classes, while in my test set I have 3038 of the same 4 classes. My CNN is the following:

def Network(input_shape, num_classes, regl2 = 0.0001, lr=0.0001):

model = Sequential()

# C1 Convolutional Layer
model.add(Conv2D(filters=32, input_shape=input_shape, kernel_size=(3,3),\
                 strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation before passing it to the next layer
model.add(BatchNormalization())

# C2 Convolutional Layer
model.add(Conv2D(filters=64, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Batch Normalisation
model.add(BatchNormalization())

# C3 Convolutional Layer
model.add(Conv2D(filters=128, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Batch Normalisation
model.add(BatchNormalization())

# C4 Convolutional Layer
model.add(Conv2D(filters=256, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
#Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation
model.add(BatchNormalization())

# C5 Convolutional Layer
model.add(Conv2D(filters=256, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Conv2D(filters=256, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation
model.add(BatchNormalization())

 # C6 Convolutional Layer
model.add(Conv2D(filters=512, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Conv2D(filters=512, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation
model.add(BatchNormalization())

 # C7 Convolutional Layer
model.add(Conv2D(filters=512, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Conv2D(filters=512, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation
model.add(BatchNormalization())

# Flatten
model.add(Flatten())

flatten_shape = (input_shape[0]*input_shape[1]*input_shape[2],)

# D1 Dense Layer
model.add(Dense(4096, input_shape=flatten_shape, kernel_regularizer=regularizers.l2(regl2)))
model.add(Activation('relu'))
# Dropout
model.add(Dropout(0.4))
# Batch Normalisation
model.add(BatchNormalization())

# D2 Dense Layer
model.add(Dense(4096, kernel_regularizer=regularizers.l2(regl2)))
model.add(Activation('relu'))
# Dropout
model.add(Dropout(0.4))
# Batch Normalisation
model.add(BatchNormalization())

# D3 Dense Layer
model.add(Dense(1000,kernel_regularizer=regularizers.l2(regl2)))
model.add(Activation('relu'))
# Dropout
model.add(Dropout(0.4))
# Batch Normalisation
model.add(BatchNormalization())

# Output Layer
model.add(Dense(num_classes))
model.add(Activation('softmax'))

# Compile

adam = optimizers.Adam(lr=lr)
model.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['accuracy'])

return model

and everytime I train and test I clearly overfit, because if I test the model I obtain a low accuracy, around 45%, and the curves of accuracy for test and training are really far apart if I plot them.

How could I improve my network in such a way it does not overfits?

Thanks in advance.

If the model is overfitting you can either increase regularization or simplify the model, as already suggested by @Oxbowerce: remove some of the convolutions and/or maybe reduce the dense layers.

Given that you already have several different types of regularizers present, I can suggest another one for convolutional layers: spatial dropout. By using SpatialDropout2D available in Keras, you can drop entire features from convolutional layers. You can try using it after the first convolutions after C5,C6 and C7 for starters.

But given that you have a very small dataset, I am not sure if there is a lot of room for improvement. It would be easier to comment if you also share the training and test accuracy graphs directly. Anyway, the best way to approach image recognition problems with small datasets is via transfer learning. I suggest you look into it if you do not have to build the model from scratch for some other reason.


Batch normalization vs batch size I have noticed that my performance of VGG 16 network gets better if I increase the batch size from $64$ to $256$. I have also observed that, using batch size $64$, the with and without batch normalization results have lot of difference. With batch norm results being poorer. As I increase the batch size the performance of with and without batch normalization gets closer. Something funky going on here.

So I would like to ask is the increase in batch size have effect on batch normalization?

By increasing batch size your steps can be more accurate because your sampling will be closer to the real population. If you increase the size of batch, your batch normalisation can have better results. The reason is exactly like the input layer. The samples will be closer to the population for inner activations.


How to group this dataframe in python? I have this problem:

import pandas as pd

stripline = "----------------------------"

rawData = {
    'order number': ['11xa', '11xa', '11xa', '21xb', '31xc'],
    'working area': ['LLA', 'LLE', 'LLS', 'MLA', 'MLE'],
    'time': ['1', '6', '13', '35', '24']
}

df = pd.DataFrame(rawData)
print("original data:")
print(df.head())

print(stripline)

rawData2 = {
    'order number': ['11xa', '21xb', '31xc'],
    'working area': ['LLS', 'MLA', 'MLE'],
    'time': ['20', '35', '24']
}
df2 = pd.DataFrame(rawData2)

print("expected result:")
print("group after order number, sum all times to that order and choose working field with the biggest time")
print(df2.head())

How can I manipulate my dataframe df to get the df2?

I want to sum up all values in the time column that correspond to an order number. I want to use the working field with the highest time and especially I want to keep the rest of the data. The new data frame has three orders, the old one five.

This line of code should do it for you :

df.groupby(["order number", "working area"])['time'].agg(sum)


ValueError: Input contains NaN, infinity or a value too large for dtype('float64') I am trying to fit my data into my model which takes numpy as input, so I feed the model with the dataframe values

stacked_averaged_models.fit(train.values, y_train1)

I am getting the following error

ValueError                                Traceback (most recent call last)
<ipython-input-145-9ba69af8df05> in <module>()
      1 X_traintrain = train.as_matrix().astype(np.float)
      2 from sklearn.metrics import r2_score
----> 3 stacked_averaged_models.fit(train.values, y_train1)
      4 stacked_train_pred = stacked_averaged_models.predict(train.values)
      5 stacked_pred = np.expm1(stacked_averaged_models.predict(test.values))

<ipython-input-140-dfca4af6e9d1> in fit(self, X, y)
     18                 instance = clone(model)
     19                 self.base_models_[i].append(instance)
---> 20                 instance.fit(X[train_index], y[train_index])
     21                 y_pred = instance.predict(X[holdout_index])
     22                 out_of_fold_predictions[holdout_index, i] = y_pred

~\Anaconda3\envs\deeplearning\lib\site-packages\sklearn\pipeline.py in fit(self, X, y, **fit_params)
    248         Xt, fit_params = self._fit(X, y, **fit_params)
    249         if self._final_estimator is not None:
--> 250             self._final_estimator.fit(Xt, y, **fit_params)
    251         return self
    252

~\Anaconda3\envs\deeplearning\lib\site-packages\sklearn\linear_model\coordinate_descent.py in fit(self, X, y, check_input)
    705                              order='F', dtype=[np.float64, np.float32],
    706                              copy=self.copy_X and self.fit_intercept,
--> 707                              multi_output=True, y_numeric=True)
    708             y = check_array(y, order='F', copy=False, dtype=X.dtype.type,
    709                             ensure_2d=False)

~\Anaconda3\envs\deeplearning\lib\site-packages\sklearn\utils\validation.py in check_X_y(X, y, accept_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, warn_on_dtype, estimator)
    574     if multi_output:
    575         y = check_array(y, 'csr', force_all_finite=True, ensure_2d=False,
--> 576                         dtype=None)
    577     else:
    578         y = column_or_1d(y, warn=True)

~\Anaconda3\envs\deeplearning\lib\site-packages\sklearn\utils\validation.py in check_array(array, accept_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, warn_on_dtype, estimator)
    451                              % (array.ndim, estimator_name))
    452         if force_all_finite:
--> 453             _assert_all_finite(array)
    454
    455     shape_repr = _shape_repr(array.shape)

~\Anaconda3\envs\deeplearning\lib\site-packages\sklearn\utils\validation.py in _assert_all_finite(X)
     42             and not np.isfinite(X).all()):
     43         raise ValueError("Input contains NaN, infinity"
---> 44                          " or a value too large for %r." % X.dtype)
     45
     46

ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

I did a check on NaN and infinity, it did pass the test

X_traintrain = train.as_matrix().astype(np.float)
print(np.any(np.isnan(X_traintrain)))
print(np.all(np.isfinite(X_traintrain)))

Output:

False
True

How else can I solve, or at least, debug this?

X1      X2       X3     X4       X5    X6   X7     X8   Y1      Y2
0.64    784.00  343.00  220.50  3.50    5   0.00    0   10.56   16.67
0.62    808.50  367.50  220.50  3.50    2   0.00    0   8.60    12.07
0.62    808.50  367.50  220.50  3.50    5   0.00    0   8.50    12.04
0.98    514.50  294.00  110.25  7.00    2   0.10    1   24.58   26.47

This is few rows of my dataset

I have tried so many suggested solutions but I found this one solve the problem.

data =data[~data.isin([np.nan, np.inf, -np.inf]).any(1)]


How to interpreting the exponential coefficent in poisson regression with offset? I am trying to find the village level risk factors for malaria. Therefore, I ran a poisson model in r with the prevalence of malaria(y) as dependent variable, altitude(x1) and Forestation(x2) as independent variable and log of Population(x3) as offset.

glmer(y~x1+x2+(1|cluster), family = poisson, offset = log(x3))

From what I have read, I understand that the interpretation of model with offset is different than a non-offset model.

It would be very helpful, If any one can clear the air on how to interpret the coefficients and exponential coefficient in the above-mentioned case.

Thanks

The offset means "exposure", so when you are calculating a regression with offset, instead of calculating absolute values, you are calculating ratios.

You are changing "number of thefts" for "number of thefts for every 10.000 vehicles"

$y = x1+x2+(1|cluster) + log(x_3)$: This is the equation you are calculating.

I think you are missing a log in $y$ because the offset and the endogenous variable have to form a ratio:

$log(y) = x1 + x2+(1|cluster)+log(x_3)$

$log(y)-log(x_3) = x1 + x2+(1|cluster)$

$log(\frac{y}{x_3}) = x1 + x2+(1|cluster)$

$log(ratio) = x1 + x2+(1|cluster)$


Image classification vs medical grading problems For image classification problems like cat vs dogs, the output layer is 2.
Image classification problems like diabetic retinopathy seem to be more of a grading classifier. Although the targets range from 0 to 4, (signifying the severity of the condition), is it better to have 1 as the output layer, or 5, for these kinds of problems?
I have seen Kaggle kernels where both are utilized.

The number of outputs will depend on the task and the number of targets. In the case of medical grading, you can either interpret the problem as:

a classification i.e. each grade is a class (5 outputs, one for each class)
a regression: predict a grade and round the result (1 output)

Make sure to check the loss function for each task. That will usually tell you how the code writer decided to approach the problem. To come back to your example, the first task should have a classification loss (crossentropy for instance) and the second a regression (MSE). Also check for the last layer activation function: usually softmax/sigmoid for classification and None for regression.
It's complicated to say what would work without seeing the data. A recent competition (Janestreet Market predicition) also saw people approach the problem both as a classification and a regression. The highest rated public notebook on the private set so far is a regression-based approach.
I suspect that this happened because the classification (of whether a stock is going to have a positive return in the future) requires to binarize the labels which basically lead to information loss.


find monthly climatology for a particular month I have a 10-year sst monthly data. I want to plot the interannual monthly climatology for the month of October. By using the code I have for python with me I am able to plot the interannual monthly climatology for all the 12 months. If anyone can help me by going over my code and say where I have to put index so that I can have monthly climatology only for the month of October it will be much appreciated. Below is my code:

import xarray as xr
import numpy as np
import matplotlib.pyplot as plt
#time_slice = slice(9,10)
lat_slice = slice(30, 0)
lon_slice = slice(40, 80)
nc = xr.open_dataset('E:/X49.37.14.247.143.9.43.4.nc')
np.disp(nc)
ds = nc.groupby('time.month', squeeze = False).mean('time')
dt = ds.sst.sel(lat = lat_slice,lon =lon_slice)
np.disp(dt)
dt.plot.pcolormesh(x = 'lon', y = 'lat', col = 'month', col_wrap =3)

ds = nc.groupby('time.month', squeeze = False).mean('time')

Here, you may put a "filter" either before or after group by.

Either you can do this-

df = pd.DataFrame({'A' : ['foo', 'bar', 'foo', 'bar',
                          'foo', 'bar'],
                   'B' : [1, 2, 3, 4, 5, 6],
                   'C' : [2.0, 5., 8., 1., 2., 9.]})
grouped = df.groupby('A')
grouped.filter(lambda x: x['B'].mean() > 3.)

Source - https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.core.groupby.DataFrameGroupBy.filter.html

Or something like this-

gdf.apply(lambda g: g[g['team'] == 'A']).reset_index(drop=True).groupby(gdf.grouper.names)

Source- https://stackoverflow.com/questions/27488080/python-pandas-filter-rows-after-groupby


I want to be able to collapse and sum values dependent on the gene name I have a table that looks like this:

I want add together all the values for each gene for each column. For example, for LINC01128, it should read:
             ConN1   ConN2   ConN3   StN1   StN2   StN3
LINC01128 :    22      14      37     34     54     67

My table is very long and this would need to be done for all the genes.

You can use dplyr, where df is your data.frame:

df %>% group_by(gene) %>% summarise_if(is.numeric, sum)

This will sum each column separately (assuming it is numeric), grouped by each gene. Any column that is not included in the group_by or is not numeric will be dropped at this stage.


Publicly Available Datasets One of the common problems in data science is gathering data from various sources in a somehow cleaned (semi-structured) format and combining metrics from various sources for making a higher level analysis. Looking at the other people's effort, especially other questions on this site, it appears that many people in this field are doing somewhat repetitive work. For example analyzing tweets, facebook posts, Wikipedia articles etc. is a part of a lot of big data problems.

Some of these data sets are accessible using public APIs provided by the provider site, but usually, some valuable information or metrics are missing from these APIs and everyone has to do the same analyses again and again. For example, although clustering users may depend on different use cases and selection of features, but having a base clustering of Twitter/Facebook users can be useful in many Big Data applications, which is neither provided by the API nor available publicly in independent data sets.

Is there any index or publicly available data set hosting site containing valuable data sets that can be reused in solving other big data problems? I mean something like GitHub (or a group of sites/public datasets or at least a comprehensive listing) for the data science. If not, what are the reasons for not having such a platform for data science? The commercial value of data, need to frequently update data sets, ...? Can we not have an open-source model for sharing data sets devised for data scientists?

There is, in fact, a very reasonable list of publicly-available datasets, supported by different enterprises/sources.

Some of them are below:

Public Datasets on Amazon WebServices;
Frequent Itemset Mining Implementation Repository;
UCI Machine Learning Repository;
KDnuggets -- a big list of lots of public repositories.

Now, two considerations on your question. First one, regarding policies of database sharing. From personal experience, there are some databases that can't be made publicly available, either for involving privacy restraints (as for some social network information) or for concerning government information (like health system databases).

Another point concerns the usage/application of the dataset. Although some bases can be reprocessed to suit the needs of the application, it would be great to have some nice organization of the datasets by purpose. The taxonomy should involve social graph analysis, itemset mining, classification, and lots of other research areas there may be.


Should I prevent augmented data to leak to the test/cross validation sets I have been working with the cats vs dogs dataset from kaggle which consist on 25000 images of cats and dogs labelled accordingly (btw, great dataset, totally recommended!)

One of the things I did was to augment the data by simply fliping the images 180 degrees, so an image like this

will became this

So far so good, now I have 50000 images, but the question is: should I make SURE that an image that has been augmented remains totally into the training set? I mean, with the previous two images, what would happen if I have one of them in the training set and another one in the test (or cross validation) set? does it mean that I am leaking training data into test data?

I understand that technically they are DIFFERENT images, but still my intuitions seems to resist such idea.

Am I correct to assume that augmented images should not leak to the test set?

Yes!

The accuracy of your algorithm should be tested on images that you expect to receive. This would not include data that has been rotated, shifted, blurred, etc. However, augmenting your data in such ways can and usually will lead to better results.

Split your data first. Then augment only the training data while staying away from the validation data. Only use your testing data when you are confident that your solution leads to good results on your validation set without overfitting it.


Perform k-means clustering over multiple columns I am trying to perform k-means clustering on multiple columns. My data set is composed of 4 numerical columns and 1 categorical column. I already researched previous questions but the answers are not satisfactory.
I know how to perform the algorithm on two columns, but I'm finding it quite difficult to apply the same algorithm on 4 numerical columns.
I am not really interested in visualizing the data for now, but in having the clusters displayed in the table.The picture shows that the first row belongs to cluster number 2, and so on. That is exactly what I need to achieve, but using 4 numerical columns, therefore each row must belong to a certain cluster.
Do you have any idea on how to achieve this? Any idea would be of great help. Thanks in advance! :

There is no difference in methodology between 2 and 4 columns. If you have issues then they are probably due to the contents of your columns. K-Means wants numerical columns, with no null/infinite values and avoid categorical data. Here I do it with 4 numerical features:

import pandas as pd
from sklearn.datasets.samples_generator import make_blobs
from sklearn.cluster import KMeans

X, _ = make_blobs(n_samples=10, centers=3, n_features=4)

df = pd.DataFrame(X, columns=['Feat_1', 'Feat_2', 'Feat_3', 'Feat_4'])

kmeans = KMeans(n_clusters=3)

y = kmeans.fit_predict(df[['Feat_1', 'Feat_2', 'Feat_3', 'Feat_4']])

df['Cluster'] = y

print(df.head())

Which outputs:

     Feat_1    Feat_2    Feat_3    Feat_4  Cluster
0  0.005875  4.387241 -1.093308  8.213623        2
1  8.763603 -2.769244  4.581705  1.355389        1
2 -0.296613  4.120262 -1.635583  7.533157        2
3 -1.576720  4.957406  2.919704  0.155499        0
4  2.470349  4.098629  2.368335  0.043568        0


Why do we use the F1 score instead of mutual information? We often use the classification threshold that maximizes the F1 score, if we don't have a prior cost function of false positives and false negatives.

This balances the desire for precision and recall. If either one is 0 the F1 score is 0; and if we have a perfect classification the F1 score is 1.

On the other hand I'm hard pressed to find a scientific justification to maximize F1 in general, or a business problem where F1 is the thing we need to maximize.

F1 is not symmetric. If we have an 60/40 binary distribution and choose the 40% class as the positive class, and we classify everything as positive, we get 100% recall and 40% precision for F1 score of 0.4. (F1=0 if we classify everything negative). If we choose the 60% class as positive, and classify everything positive, we get F1 score of 0.6.

Why not use mutual information, which minimizes the surprise in the prediction vs. actual?

When we estimate probabilities by minimizing log loss, we are also minimizing K-L divergence or entropy or surprise in the information theory sense. If a 50/50 probability prediction contains 1 bit of entropy or surprise, minimizing the log loss minimizes the number of bits of entropy or surprise in our predictions vs. actual.

And in information theory, if we don't have probabilities and want to measure the information transmitted by a noisy signal, we use mutual information.

And mutual information is symmetric, and 0 for a signal of all 1s or 0s. Predicting all 1s or 0s in some sense gives no information about the ground truth of the response variable.

In a way, a prediction is like a noisy channel from the present to the future, and an information theory concept like mutual information seems like a well-grounded criterion to choose the classification threshold, and F1 seems arbitrary.

Is there any reason why F1 is preferred over mutual information?

The reason why one uses any particular evaluation measure should be based on the semantics of the task. Corollary: there's no unique evaluation measure which is perfect for every task.
Obviously there are also technical constraints to take into account, but imho considering only the technical aspect of performance optimization is the most common mistake made in ML applications.

For instance OP mentions non-symetry of the F1-score: in some tasks it makes complete sense to use a non-symmetric measure, for example any task where the goal is to extract specific elements, e.g. named entity recognition. Moreover the question of symmetry makes sense only for binary classification, and many tasks involve more than 2 possible classes. In such cases it's common to use micro or macro F-score, but again it depends what one wants to measure. It's also important to keep in mind that a single metric is always a simplification of the performance, often there are many other relevant aspects. For instance, mistakes for a particular class might be more costly than for another, like in the case of medical prediction systems where a false negative means that a patient doesn't get a potentially life-saving treatment (note that generic F-score can be weighted to account for that).

The interpretability of the performance measure is also very important. In many applications one needs to estimate some kind of average error rate that a non-expert can understand, otherwise the ML system is just a magic black box to its users and this can cause serious ethical and practical issues.


May the training set and validation set overlap? May the training set and validation set overlap?

Similarly, may the testing set and validation set overlap?

Definitions, so we are on the same page:

Training set: the data points used to train the model.
Validation set: the data points to keep checking the performance of your model in order to know when to stop training.
Testing set: the data points used to check the performance once training is finished.

May training and validation sets overlap?

They should not.

The validation set is used to know when to stop training your model. The idea is that you check your model performance often and when there seems to be no more improvement, you stop.

Take a look at the plot below. It is plotting the loss of the model. If the loss is still decreasing means that you can keep on improving your model, but if the loss stops decreasing you stop training.

Notice how the valid loss stops decreasing before the train loss. That is because the model could keep improving to improve the performance on the training set, but if you do that, you will get overfitting. So, by having an unseen validation set, you will stop training earlier and not overfit your mode, otherwise, if you keep on training, the accuracy on the validation set will start to decrease.

That means that if the two sets overlap, the validation loss will become more similar to the training loss, so you model will keep on training and you will overfit your model.

May validation and testing sets overlap?

They should not.

You already used your validation set to stop training your model. That means that you already know the performance of your model again the validation set.

Now your model is trained and you want to test your model with unseen data points, i.e. with the testing set. If your sets overlap, you are biasing your test results towards the performance which you already know your model has.


Recommended data cleaning techniques for multivariate time series prediction? I have to predict the next step(s) in a multivariate time series with about 30 features and 50.000 samples. I am thinking of using LSTM. Which techniques are usually recommended for cleaning the data when using LSTM?

Does it make sense to transform the data into a stationary time series when using LSTM? Should the data be normally distributed when you are using PCA?

There is also a very large amount of missing timestamps. Does it make sense to impute/fill (by forward filling or something else) big gaps or is it just better in that case to ignore the missing data completely?

3 questions:

Does it make sense to transform the data into a stationary time series when using LSTM?
Always Stationarity is always desired property and data should be transformed (read more)
Should the data be normally distributed when you are using PCA? No
There are multiple assumptions around PCA IF you use some matrix factorization techniques. For example if using SVD you should make sure that your matrix is of full rang.
Does it make sense to impute/fill (by forward filling or something else) big gaps or is it just better in that case to ignore the missing data completely? If for certain features a lot of data is missing you should drop them all together. Dont try to impute it, you will add false information.


Is there an advantage to use decision trees in OneVsAll than using "classical multiclass trees"? I wonder if the is cases where it's better to use OneVsAll decision trees for multiclass classification ?
I think that maybe it could be better for explainability of the model, but I didn't see anybody saying that.

In general OneVsAll is a device useful when you want to transform a binary classifier into a multi-class one. This happens when models cannot handle multiple classes or when it is difficult to adapt it. By doing so, you have to normalize the probability estimates, otherwise they are not too useful, being rather non comparable. That alone is probably a sufficient reason why nobody use this when a multi class version is available.
There are cases when something similar might be useful. For example when you have a lot of classes. One might try to group the target levels, build a classifier for groups and later do subsequent classifiers for each group. But there are not overlaps between predictions.


Differentiating roadmap of a loss function Let's say I'm performing Stochastic Gradient Descent (SGD) on binary cross entropy error while optimizing weight $w_{2}$.

Binary cross entropy error:

$$L(y|p(x_{i}))=-y_{i}*ln(p(x_{i}))-(1-y_{i})*ln(1-p(x_{i}))$$

The derivative of such roadmap would be:

$$\frac{\partial L(y|p(x_{i}))}{\partial w_{1}}$$

We can use a chain rule to solve this:

$$\frac{\partial L(y|p(x_{i}))}{\partial w_{1}}=\frac{\partial L(y|p(x_{i}))}{p_{i}}*\frac{\partial p_i}{w_1}$$

First term is easily understandable, it's just a differentiation of a loss function with respect to the estimated probability:

$$\frac{y_i -p_i}{p_i(1 - p_i)}$$

But the second term seems a little weird, from my understanding $p_i$ is just estimation of Bernoulli distribution (therefore it should be Empirical distribution). Somehow it's derivative is:

$$p_{i}*(1-p_i)*x_i$$

This is definitely somehow associated with the Bernoulli distribution, but how exactly? What is the definition of $p_i$ in this particular case?

Thank you!


Question about bias in Convolutional Networks I am trying to figure out how many weights and biases are needed for CNN.

Say I have a (3, 32, 32)-image and want to apply a (32, 5, 5)-filter.
For each feature map I have 5x5 weights, so I should have 3 x (5x5) x 32 parameters. Now I need to add the bias. I believe I only have (3 x (5x5) + 1) x 32 parameters, so is the bias the same across all colors (RGB)?

Is this correct? Do I keep the same bias for each image across its depth (in this case 3) while I use different weights? Why is that?

Bias operates per virtual neuron, so there is no value in having multiple bias inputs where there is a single output - that would equivalent to just adding up the different bias weights into a single bias.

In the feature maps that are the output of the first hidden layer, the colours are no longer kept separate*. Effectively each feature map is a "channel" in the next layer, although they are usually visualised separately where the input is visualised with channels combined. Another way of thinking about this is that the separate RGB channels in the original image are 3 "feature maps" in the input.

It doesn't matter how many channels or features are in a previous layer, the output to each feature map in the next layer is a single value in that map. One output value corresponds to a single virtual neuron, needing one bias weight.

In a CNN, as you explain in the question, the same weights (including bias weight) are shared at each point in the output feature map. So each feature map has its own bias weight as well as previous_layer_num_features x kernel_width x kernel_height connection weights.

So yes, your example resulting in (3 x (5x5) + 1) x 32 weights total for the first layer is correct for a CNN with first hidden layer processing RGB input into 32 separate feature maps.

* You may be getting confused by seeing visualisation of CNN weights which can be separated into the colour channels that they operate on.


What is the allowable limit of oversampling? Suppose I have 2 classes. One class has 16 samples and the other class has 435 samples. Is it justified to oversample the class with 16 sample to have a 435 number of samples? Or is it better to undersample the class with 435 samples? If so, what should be the number of samples after the undersampling is carried out?

This depends on the nature of your data. If you can effectively simulate 435 samples using any given oversampling methods such as SMOTE or ADASYN for instance, then I would say oversampling would be better. Because it would provide data for various scenarios. But if exact replication is not possible and if replicated there might be an inherent problem with the model then you should choose to undersample.

A good example for undersampling would be that for example if you're conducting a scientific experiment and the data you have are limited to a few scenarios and the other data is yet to be fully observed, you would choose to undersample.

Oversampling would be the reverse case, where you can effectively simulate data for various classes based on some parameter and the generated data can mimic actual data and scenarios, then you should oversample.


How to compare two unsupervised anomaly detection algorithms on the same data-set? I want to solve an anomaly detection problem on an unlabeled data-set. The only information about this problem is that the anomalies population is lower than 0.1%. It should be notice that the size of the feature vector for each sample is 40. Is there any clear way to compare the performance of unsupervised algorithms?

For unlabeled data-sets, unsupervised anomaly detectors can be compared either subjectively or objectively.

Subjective comparison: based on our domain-knowledge and by using some visualizations and statistics, we can compare two detectors and select the one that outputs better anomalies subjectively.

Here is a well-cited survey on unsupervised anomaly detectors that compares the algorithms on labeled data-sets (with known, domain-specific outliers) using AUC, and concludes that local detectors (such as LOF,
COF, INFLO and LoOP) are not good candidates for global anomaly detection:
2016 A Comparative Evaluation of Unsupervised Anomaly Detection Algorithms for Multivariate Data

Objective comparison: possible in theory, impossible in practice.

Requirements for objective comparison:

Anomaly definition: $x$ is an anomaly if $P(x)< t$ for some threshold $t$,
Anomaly detector requirement: $D$ is an anomaly detector if for every detected $x$, $P(x)< t$,
Comparing anomalies: $x_1$ is more anomalous than $x_2$ if $P(x_1)<P(x_2)$ or equivalently $r(x_1, x_2) = P(x_1) / P(x_2) < 1$,
Comparing anomaly detectors: proposal $x_1$ from detector $D_1$ is better than $x_2$ from $D_2$ if $r(x_1, x_2) < 1$,

As you can see, for qualification and comparison of two detectors we need to know the underlying $P(x)$ or at least $r(x_1, x_2)$. But if we know these quantities (which act as a  judge $J$) or at least a close enough estimation of them, we have a better anomaly detector $J$ and can throw $D_1$ and $D_2$ away! We plug any observation $x$ or pair of observations $x_1$ and $x_2$ into $J$ and check which one is an anomaly or which one is more anomalous, done! So it is impossible to compare two anomaly detectors objectively unless we have a better anomaly detector (judge). So we should use a subjective comparison.


One hot encoding large dataset Initially, I have a dataset where for each row there is user_id and product_ids he bought.

In that dataset there are 478191 products bought by different users.

In order to discover frequent items that are bought together, I will use association rules, apriori algorithm.
As apriori algorithm expects to have the features one-hot encoded.   I need the product_ids one-hot encoded.

Scikit-learn one-Hot encoding (sparse matrix=True) resulted in memory error. What other methods can I try? (using Python)

Apparently, you need an implementation of the a priori algorithm (or whatever algorithm you want to use) that supports sparse feature vectors.  If one doesn't already exist, you might need to implement it yourself.

Alternatively, you can use the feature hashing trick.  However, beware that it can cause loss of accuracy and loss of interpretability; nothing comes for free.


Two steps optimization of a credit card limit I have a problem similar to what is on the title but not the same, the problem on the title allows me to explain the dynamics of my need.
I have to determine how much is the optimal value for a variable called QUOTA or LIMIT for a credit card.
The goal of the model is to allow me to minimize the probability of default, given this variable and others which caracterize my costumer. Which is the best way to determine this?
I have thought two possibilities:

Train a model with all the exogenous and the LIMIT as variables for any classification model (with Y=Default/No Default)
Train a model with all the exogenous variables but not a LIMIT (with Y=Default/No Default), then train another model with the resulting probability and LIMIT as the only two variables (with Y=Default/No Default).

Any of this possibilities sound right?
With both models, the application dataset should repeat costumers with two or more possible values for the LIMIT variable and obtain the default probability of every customer repetition. Does this sound correct?

You can model it as a binary logistic regression. The target is the probability of default. One of the features is credit card limit.
Logistic regression allows an odds ratio to be calculated. The odd ratio will give the effect of increasing credit card limit on wether a default is predicted to happen.
Logistic regression allows a threshold to be set to "minimize the probability of default".


Are there libraries or techniques for 'noisifying' text data? Data augmentation techniques for image data and audio data (eg speech recognition) have proven successful and are now common.

Are there libraries or techniques for augmenting text data?

For example:

  in: 'How are you?'
  out: ['how are you?', 'HOW ARE YOU?', 'hwo are y ou?', 'How're you?', 'how r u', ...]

I you want some kind of data-sets like Google spell checking data I suggest you look into the The WikEd Error Corpus dataset. The corpus consists of more than 12 million sentences with a total of 14 million edits of various types, this edits include: spelling error corrections, grammatical error corrections, stylistic changes. All these from the Wikipedia correction history. The owners (authors) of the data-set describe the data mining process in this paper. Also check this question in quora it contains links to various data-sets with spelling errors. Finally this page can also be useful.


Should the times series be made stationary before doing a clustering analysis? For times series analysis and forecasting, we try to make the times series stationary before proceeding with the experiment. I would like to know if such a procedure is necessary if one is working on an unsupervised learning task and needs to do times series clustering? Thanks.

Not sure, but I don't know any task related to time series algorithms and analysis for which it wouldn't be useful to make the time series stationary first. You simply get rid of a lot of unexplainable anomalies this way.
Maybe only if you use several different time serieses of the same period, then one could try to find some usefulness in comparing the unstationary serieses.
The effects of the unstationary series, are not explainable by the same series alone.


Using recurrent neural networks for modeling errors in IMUs Inertial measurement units (IMU), usually composed of accelerometers and gyroscopes; are well known to have inherent errors in their data, originating from bias, random walk noise, temperature dependence etc. creating a highly non-linear dependence. Typically, extended Kalman filters are used to estimate and remove these errors for stable measurement of orientations and angular velocities: but even this is not entirely accurate, as some higher order errors are ignored or approximated, and the fact that the Markov assumption ignores the effect of values older than the immediately previous value while predicting a future value. The highest levels of accuracy in IMUs are usually obtained after a rigorous factory calibration (which, in turn, makes the good ones very expensive).

In a scenario like this, how applicable would recurrent neural networks be in modeling these errors? Assuming my 'training data' consists of the accelerometer and gyroscope values, which can be fused to obtain a noisy orientation estimate, and at the same time, a much more precise orientation estimate coming from another sensor (for example, a very accurate GPS): would it be possible to replace the functionality of the Kalman filter by an RNN for error estimation and sensor output prediction?

The only way to find out is to try it, but I doubt it.  A Kalman makes a specific assumption about the "priors", i.e., about the nature of how the signals are related over time.  This is based on knowledge about the physics of movement and how this affects the sensor values, combined with some assumptions about the likelihood of different kinds of movements.  Assuming the model and assumptions are accurate, this helps provide a more accurate estimate of the orientation.

In contrast, a recurrent neural network has no knowledge built in, and no useful "priors".  It has no model about the physics of movements and no assumptions.  In principle, this makes it more general.  But in practice, it means that, if you have a limited amount of data, it is likely to be less accurate.

In general, a strong prior and useful model can be very helpful.  In the limit, as you have an unlimited amount of data, you might not need a model.  But with a limited amount of data, the model can be helpful.  The Kalman filter provides a pretty reasonable model, and when trying to work with data from IMUs, we typically have only a limited amount of data.

The place where RNNs could be more effective would be if the assumptions made by the Kalman filter's model do not correspond to reality.  Then, you could imagine that RNNs could possibly learn the dynamics, because they don't assume a particular model, whereas Kalman filter is locked into a particular model.  I don't particularly expect this to happen in practice, but it's a possibility.

Overall, from first principles, I would expect/predict RNNs to be less effective than Kalman filtering.  But that's just speculation/guessing.  If you really want to know, the way to find out for sure is to try some experiments.


How many folds on cross validation Cross validation on a dataset with 22680 observations. Want the training set to contain 21420 entries. How many folds can you use for your cross validation?
How do you calculate the folds?

The number of folds is nothing you can really calculate. It is more a parameter you choose by good judgement. Typically used values are between 5 and 10, but you can even go up to the so-called leave-one-out cross validation, in which each fold consists of all but one observation. The arguments that guide your decision are typically the following:
- computational cost: the more folds you use, the more computations you need.
- variance: the more folds you use, the higher the variance of the cross validation result
- bias: the more folds you use, the smaller the bias of the cross validation result.

In your case, depending on the exact context in which you want to apply cross validation, you will probably want to choose between 5 and 10 folds.

For more details, you might want to consult Kohavi's 1995 paper on cross validation.


How different should discriminator be from generator in GAN When training a GAN, the generator $G$ strives to fool the discriminator $D$, while $D$ attempts to catch any output generated $G$ and isolate it from a real data point. They grow together training in turns for each epoche.

Assuming $D$ is already an expert classifier (for example, classifying birds and nonbirds images). What will happen if I freeze the weights of $D$ and only train $G$ (to generate high-resolution bird images from low-resolution ones for example)? Is there a mathematical problem here? Is $D$ so good that the generator will not be able to learn due to a very high initial error? I have obviously simulated it and failed.

Generator cant learn if the discriminator error is too small. But generator should always be "ahead" of generator in order to learn.

There is a paper explaining why the generator's gradient vanishes if the discriminator gets too strong. And how does this proportion look mathemactically optimal

TL;DR Dont make it too strong, but make sure D is ahead to ensure optimal learning. GANS are notoriously unstable (expensive also)


Get how similarity between the training data and the income data? I'am trying to use Clustering and Classification methods as SVM using scikitlearn. I'm also studying some outliers/novelty detections

I want something like a semi-supervised model. I want to predict some class labels, however, I cant training my model with some anomalies/new behaviors that are going to happen eventually.

I'm working with motors vibration, I'll train my model with the vibration when they are "ON" and the vibration when they are "OFF".

With the time the vibration of the "ON" state will start to become slightly different because of some defects, however if I just train with "ON" and "OFF" for sure it'll continue to identify that my motor is "ON" because the vibration with defects are more close to the model "ON".

Therefore I would like to identify how far is my new data from the data that I have already trained my model.

For example, 1 - My predict_prob returns ON because has $98$% of chance to be ON and $2$% to be OFF. So far so good, however my DATA is slightly different from the ON mode in the training data, so I would like to measure this difference, like it has $98$% probability to be ON between ON and OFF and it is $95$% similar to the original model, so with the time this number ll be reduced. For example:

1 day: $98$% to be ON $2$% to be OFF and $95$% similar to the training data

2 day: $98$% to be ON $2$% to be OFF and $93$% similar to the training data

3 day: $99$% to be ON $1$% to be OFF and $90$% similar to the training data

4 day: $95$% to be ON $5$% to be OFF and $85$% similar to the training data

Can someone help me with this part "and $85$% similar to the training data"?

I think I got your difficulty.  You are trying to detect defective motors whilst there is more than one state in which they are non-defective.  I am assuming that since there are several of those motors even when a motor is OFF the sensors still catch "noise" vibrations.  You do have two distinct known patterns: ON and OFF, i.e. OFF is not just the absence of vibration.  I see two options for this classification (below) but first let us make a small detour:

  and 85% similar to the training data

The vast majority of ML techniques are based on distance measures: the training data is used to define a blobs (e.g. distributions) in (possibly very high dimensional) space, one for each class.  The classes of new samples are predicted by comparing their distance to the class blobs - the closest blob is the predicted class.

Now, this means that the classification probability is the similarity to training data.  Probabilities in the ranges (0.9, 0.1) or (0.8, 0.2) are very similar to training data, whilst probabilities in the ranges (0.6, 0.4) or (0.5, 0.5) are the most likely distinct from training data.  This leads us to one solution you may attempt:

Understand probabilities as distances

The probabilities (.predict_proba) of models such as SVMs in sklearn are literally distances, in the case of SVMs the distances within the support vectors.  In other words, we can understand these as distances between the middle points of the two blobs: ON and OFF.  e.g.

This means that assuming two things:

That the classes ON and OFF are well separated (there's reasonable amount of distance between them), and,
Defective motors will produce less vibrations than motors ON, but more vibrations than motors OFF (I'm a little worried about this assumption but bear with me).

We can say that:

Any probability pair akin of (0.8, 0.2) means that we have an ON or OFF motor.  In other words, if we have at least one of the classes with a probability higher than, say, 80%, we are close enough to the blobs of training data.
If the probability of ON and OFF is somewhere around 50% then we very likely have a defective motor.

One can choose the probability threshold to his liking (be it 80%, 70% or 90%), as a way to adjust for false positives or false negatives.

Issues with this solution

The assumption that a defective motor will have vibrations between the ON and OFF classes.  i.e. if a defect makes the motor vibrate more than the normal ON it will be undetectable with this solution.
Assumes linearity and same scale of distance from ON and from OFF for defective motors.  i.e. classes are of the same size and equally distributed.  Such solution will check distances from each class using the same linear scale (the threshold), it may not detect defects that happen close to the smaller class.

Use Novelty Detection

You do have two known classes: ON and OFF.  But what you are trying to find is whether a motor is defective or not, where we understand non-defective as the fact that the motor is very similar to most ON motors or very similar to most OFF motors.

Forget about the ON and OFF classes and dump both sets of samples of non-defective motors as one single (non-defective) class and into a novelty detection algorithm.  In other words, do not differentiate between ON and OFF motors.  Say that both cases (ON and OFF) are OK, just anything that is quite far away from any known cases is not-OK.  This will look as a bunch of blobs in space and anything far away from these blobs is a defective motor.

I'll suggest starting with an One Class SVM because it has a parameter called nu= which is quite easy to interpret.  It varies between 0 and 1 and the closest to zero it is the more the algorithm is permissive of anomalies (defects) away from the blobs.

Note that we have exactly the same points on this picture as in the previous one.  The only difference is that this time all of them are red.

This solution will cope with defects that are anywhere, not only in between the states of ON and OFF.

Issues with this solution

On Class SVM does not have a .predict_proba.  To get an estimate of how far away from a blob an anomaly marked as an anomaly is, one must evaluate the .decision_function of the model and keep track of the distances oneself.
Even if the classes ON and OFF are well separated one will have false positives and false negatives.  A very low nu= parameter still cannot go beyond the blobs.  Using an Isolation Forest may allow for better tuning but the decision function would be even more complex to get distances from.  Another option is to add marginal versions of ON and OFF to the training data but finding those may not be easy.

P.S. If one needs to do both classify ON and OFF AND classify defective and non-defective motors, then one should build two separate models.


Oversampling before Cross-Validation, is it a problem? I have a multi-class classification problem to solve which is highly imbalanced. Obviously I'm doing oversampling, but I'm doing cross-validation with the over-sampled dataset, as a result of which I should be having repetition of data in the train as well as validation set. I'm using lightgbm algorithm, but surprisingly there is not much difference between cross-validation score and the score on the unseen dataset.

However I just want to know whether its fine to do cross-validation after oversampling the dataset, if not why am I getting such close score on the validation set and the unseen test set?

Also if its not correct to do oversampling before the cross-validation, then it becomes to lengthy to split the data into validation and training and then again sample the training set, and again during final prediction if you're looking to use all the data then you've to append the validation and the training data and then again oversample. Is there any shortcut method to solve the problem?

Oversampling the training data may help the classifier to better predict on the originally less represented class. This does not mean that it should be applied to performance metrics, as it changes the original target distribution and thus creates bias in the results.

Imagine the problem of cancer detection, where your original dataset is unbalanced: 10% of the patients have cancer y=1 and the remaining 90% don't y=0. If you train a classifier which is prone to error on unbalanced datasets (such as an Artificial Neural Network), you may end up predicting always the majority class: y=0.

If you oversample to a new distribution, let's say 50/50,  your classifier is expected to increase the performance, specially on the positive class. Nonetheless, to measure the performance on real data, which is by itself skewed, measure on oversampled may not be the best choice.

Thus, if you are optimizing the hyperparameters or choosing from a set of classifiers, cross-validating with oversampled data may provide you with a different perspective on the classifier's ability to predict on both classes with equal importance. Nonetheless, if you are estimating the real-life prediction capability, I would not advise you to oversample such validation data!


Which machine learning algorithms support a feedback ingestion loop? I understand stochastic gradient descent does. any other alorithm which supports incremental training preferably updating on a real time basis?

A few other methods support incremental learning. (Sorry it should be a comment instead but I don't have enough reputation ;)

http://scikit-learn.org/stable/modules/scaling_strategies.html#incremental-learning


MNIST 2-layer NN failing to recognize certain digits Recently I've been taking a look into some basic multilayer neural networks, and I decided to try tackling the MNIST database of handwritten digits using a 2 layer neural network with 100 neurons in the hidden layer.
My network has a hidden layer using ReLU, and the output layer uses the sigmoid function. The cost is calculated by MSE, and weights are updated using SGD. Below is the code written in MATLAB for training the network.
When training this network, I discovered that it has somehow failed to recognize every 8 and 9 in the data set. It has some problems recognizing other numbers, but not nearly as much as 8 or 9.
How can I fix this issue?
Edit: The results when training a network from a random set of weights are quite inconsistent and oftentimes have low accuracy. Is there something wrong with the backpropagation step or is the network structure not the best for digit recognition?
*dsigmoid and dReLU are the derivatives of their respective functions.
function [trainedWeights1, trainedWeights2] = twoLayerSGD(weights1, weights2, inputs, outputs, alpha, shuffle)
% Trains a 2 layer perceptron using stochastic gradient descent and MSE
%
% function [trainedWeights1, trainedWeights2] = twoLayerSGD(weights2, weights1, inputs, outputs, alpha, numHiddenLayers)
% Inputs:
%   weights1 = weights between input layer and hidden layer as a matrix
%              where row i and column j represents the weight between node
%              j of the input layer and node i of the hidden layer
%   weights2 = weights between hidden layer and output layer as a matrix
%              where row i and column j represents the weight between node j of the
%              input layer and node i of the hidden layer
%   inputs = inputs of the training data, where each set of inputs is
%            stored as a row vector
%   outputs = the correct outputs of the training data, where each set of
%             outputs is stored as a row vector
%   alpha = the learning rate, default is 0.005
%   shuffle = boolean value, determines whether the training cases will be
%             randomly ordered, default is false
%
% Outputs:
%   trainedWeights1 = the trained weight matrix for the weights between the
%                     input layer and hidden layer
%   trainedWeights2 = the trained weight matrix for the weights between the
%                     hidden layer and output layer

if ~exist('alpha','var') || isempty(alpha)
    alpha = 0.005;
end

if ~exist('shuffle','var') || isempty(shuffle)
    shuffle = false;
end

% Determining the size of the input data set
N = size(inputs,1);
fprintf('Data set size: %.0f\n', N);

% Shuffle input data
if shuffle
    fprintf('Shuffling data...\n')
    order = randperm(size(inputs, 1));
    inputs = inputs(order, :);
    outputs = outputs(order, :);
    fprintf('Shuffle complete\n')
end

% Train Model
fprintf('Starting training process...\n')
trainedWeights1 = weights1;
trainedWeights2 = weights2;

for dataSet = 1:N
    % Get inputs of current data set
    input = inputs(dataSet,:); % row vector

    % Calculate activations of hidden nodes and the output
    Z1 = trainedWeights1*input'; % column vector
    hiddenActivation = ReLU(Z1); % column vector
    Z2 = trainedWeights2*hiddenActivation; % column vector
    output = sigmoid(Z2)'; % row vector

    % Calculate the error/cost of the current weights
    correctOutput = outputs(dataSet,:); % row vector
    cost = sum((output - correctOutput).^2);

    % Update Weights
    trainedWeights2 = trainedWeights2 - alpha * 2*(output'-correctOutput') .* hiddenActivation' .* dsigmoid(Z2);
    S = sum(2*(output'-correctOutput') .* trainedWeights2 .* dsigmoid(Z2),1);
    trainedWeights1 = trainedWeights1 - alpha * S' .* input .* dReLU(Z1);

    % Display status
    if mod(dataSet, 1000) == 0
        fprintf('Data set %.0fk/%.0fk complete!\n',dataSet/1000,N/1000);
    end
end
end

Both sigmoid activation and MSE loss often don't work so well.  I would start by using standard methods, i.e., softmax and cross-entropy loss.


Average reward reinforcement learning What is the bellman equation update rule for the average reward reinforcement learning? I searched a few articles, but could not find any practical answer.

In general, the average reward setting replaces the discounted setting in continuous tasks. It relies on there being a long term stable distribution of states under any particular policy (this is called ergodicity) - and in this will usually be true for continuous MDPs that don't have absorbing states.

If you see an update rule in the discounted setting that looks like this (for Q learning):

$$Q(s,a) \leftarrow Q(s,a) + \alpha(r + \gamma\text{max}_{a'}Q(s',a') - Q(s,a))$$

Then you replace the discounted TD error by the differential TD error:

$$Q(s,a) \leftarrow Q(s,a) + \alpha(r -\bar{r} + \text{max}_{a'}Q(s',a') - Q(s,a))$$

where $\bar{r}$ is the mean reward per time step under the current policy. You can estimate this simply from the rewards seen so far.

  I searched a few articles, but could not find any practical answer.

See Reinforcement Learning: An Introduction (Second Edition) chapter 10, sections 3 and 4 for a more thorough description and more examples.


Feature extraction from a scatter plot Say I have a scatter plot like this:

Since I have many scatter plots like this I want to do feature transformation i.e. squash (x,y) in a single term to be input to a NN. Which transformation like x/y or (x/y)^2 or any other transformation will work best in this type of graphs i.e. increase the separation more while squashing it to a single term.

As @David Masip mentioned, Principal Component Analysis would be a good method to use here. Essentially PCA is a method by which a mapping is found between a high dimensional space to a lower dimenional space while keeping as much variation in the data as possible - perfect for dimensionality reduction of high dimensional data.

However, you mention that you want to use this reduced data to train a neural network model. It may be best to first train the neural net model and see how well it performs, as neural nets are usually very good at identifying interactions between features as well as other hidden structures in the data. If it doesn't perform well, then one approach to improve performance may be to use PCA - although this is highly dependent on your use case, content / type / amount of data, neural network architecture etc.

p.s PCA is also good to visualize high dimensional data (reduce the dimensionality to 2 or 3 dimensions, then plot it. This is better than plotting only 2 features at a time as you have done above).


Customer Segmentation: Should I use a variable, representing a product, that is unpopular in the dataset for K-Means Clustering? I am working with a data set that, besides customer age and income, tells the balance a customer has in different type of bank accounts: Checking, Shares, Investment, Savings, Deposit, Mortgage, Loan, and Certificates. For accounts other than Checking, 0 represents that the account does not exist for the customers. There are 9800 customer observations with roughly 6000 checking accounts and 4000 savings accounts. For the others, the observations are less than 300.

I have to use K-Means Clustering analysis for the segmentation with the objective to understand how customers use savings and investment offerings and I am using the Elbow Method to predict the number of clusters. I am confused whether to use a variable like Investment with just 250 observations with another like Savings that has 4000. If I do use such variables, then these are heavily positively skewed and I'm not sure if K-Means handles that well. Can someone advise whether to include such variables or not?

I suggest you use an algorithm that accommodates categorical variable since there are missing data. You can one hot encode it so missing data will be relevant. Making it zero will be misleading.

Try algorithms like tSNE and Self Organizing Map and use the Jaccard/Tanimoto distance.


What is meant by 'training patch size'? Currently I read a paper about symmetric skip connections for autoencoder (link). One experiment of them changes the the 'training patch size'.

In my understanding patches are sub-boxes of an image that is used at one time of an convolutional layer. So if you have a 3x3 filter the patch is a part of the image with the size 3x3.

Do they mean by 'training patch size' the size of the input image? (The network is a symmetric autoencoder, so the input size is arbitrary)

Thanks in advance

If you follow the linked literature (down the rabbit hole for a few levels), you end up at a paper from 2005 from Kervann and Boulanger
- at least that's as deep as I got.

In that linked webpage, they define the patch-based image noise reduction methods as such:

  The main idea is to associate with each pixel the weighted sum of data points within an adaptive neighborhood.

So a patch is an area of a single image, like a convolutional kernel, but it doesn't convolve.

They talk about adaptive patches, meaning that you need to (perhaps randomly) select a pixel, then adapt the patch size used in order to include enough surrounding information to reproduce a homogenous patch as the output.

It seems as though training with clean images, to which noise is added (additive Gaussian white noise),  are used to train. This will help with robustness of final models by reducing the variance, but doing so also must introduce a bias to re-create areas where the noise is somehow uniform. The first link above, if you scroll down, shows many examples of typical images to be de-noised. The noise is not always so uniform.

Here is a picture taken from that 2005 paper, where they show patch-regions (marked in yellow). Page 5 gives a nice short description of the general idea. Patch sizes in their work were typically $7 x 7$ or $9 x 9$ in pixel size.


Application of Machine learning or Neural Networks for automatic Time table scheduling I've been trying to come up with an intelligent solution to build a Time table scheduling application with the use of Machine learning or Neural networks. What would be the algorithm or approach to build such application. I'm planing to take data from google calendar API and through the system. The system should purpose best time slots to conduct lectures considering the data taken from lecturers and students.

I found some research documents where they have used some genetic algorithms but can this be done with the help of neural networks?

Scheduling problems might be NP-Complete problems.
It is not clear what are the specific details.
You might get lucky and have specific constraints that are leading to an easy sub problem or just an easy instance.
However, since many variations like On the Complexity of Scheduling University Courses (which might be your case), Job shop scheduling,Multiprocessor scheduling and Open-shop scheduling are NP-complete, you are probably in the same situation.

Usually, it is better to treat such problems like optimization problems and not like classification problems. If you'll try to treat the problem like a classification problem you might have severe problems in building a classified dataset that will represent well the time scheduling problem that you would like to solve.

There are general technique in order to cope with optimization problem. I also found some work related to your case. I'm not familiar with this specific case but it seems that Solving the Course Scheduling Problem Using
Simulated Annealing and the work done here might help you.


Feature Importance from GridSearchCV I created a GridSearchCV for a Random Forest Regressor. Now I want to check the feature importance. I searched around and I found this:
rf_gridsearch.best_estimator_.named_steps.feature_importances_

This already works, but my training data is huge, 669 attributes. Therefore, I need the attribute names. So I found this code:
rf_gridsearch.best_estimator_.named_steps["step_name"].feature_importances_

But I don't know what the "named_steps["step_name"]" are.
I tried something like this:
named_steps = X_train.columns

But this doesn't work. Could somebody explain me what named_steps["step_name"] is?

I think that you just need:
feature_importances = rf_gridsearch.best_estimator_.feature_importances_

This provides the feature importance for all the attributes in your dataset. For more information on this as well as other options, you may also refer to the Scikit-learn official documentation.


Classification with a lot of the classes I’m trying to make model which will classify text into about 500 different classes. I think that I have to customize architecture of the Pooling Classifier which looks now like this:

(1): PoolingLinearClassifier(
(layers): Sequential(
   (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
   (1): Dropout(p=0.2, inplace=False)
   (2): Linear(in_features=1200, out_features=50, bias=True)
   (3): ReLU(inplace=True)
   (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
   (5): Dropout(p=0.1, inplace=False)
   (6): Linear(in_features=50, out_features=498, bias=True)
)

I think that I have to change in (2): Linear layer to have more out_features because in the last (6) Linear layer I predict more out_features than I’ve got in_features. What do you think?

Best regards

Try pooling in between, that will reduce size so that its compactible.
Have a look here


How to decide how many n_neighbors to consider while implementing LocalOutlierFactor? I have a data set with rows: 134000 and columns: 200. I am trying to identify the outliers in data set using LocalOutlierFactor from scikit-learn. Although I understand how the algorithm works, I am unable to decide n_neighbors for my data set.

Kindly suggest.

One normally uses Grid Search for calculating the optimum parameters in these situations:

from sklearn.model_selection import GridSearchCV
from sklearn.neighbors import KNeighborsClassifier
import numpy as np

n = 30 # Max number of neighbours you want to consider
param_grid = {'n_neighbors': np.arange(n)}
grid = GridSearchCV(KNeighborsClassifier(), param_grid)

Then given this grid, you can fit it to your data to compute its optimum values (from those you provided, they may not be global optima (or even local if the returned value is one of the extrema of your input range)):

grid.fit(X_train, y_train)

You can view the optimum parameters from your input by calling:

grid.best_params_
>>> {'n_neighbors': ?}

You can automatically select an estimator with said optimum parameters by calling:

model = grid.best_estimator_
y_pred = model.fit(X_train, y_train).predict(X_test)

Note: you can find the optimum values of other parameters by adding them to the input dictionary param_grid.


How to normalize results of Singular Value Decomposition (SVD) between 0 and 1? I'm building a recommender system and using SVD as one of the preprocessing techniques.
However, I want to normalize all my preprocessed data between 0 and 1 because all of my similarity measures (cosine, pearson, euclidean) depend on that assumption.

After I take the SVD (A = USV^T), is there a standard way to normalize the matrix 'A' between 0 and 1? Thanks!

Edit: I want all of my similarity measurements to give results between 0 and 1 and my normalized euclidean distance in particular fails if the input matrix does not have values between 0 and 1.

Singular Value Decomposition is a linear algebraic technique as a result of which the notion of normalization is hard to define. In principle, you can do this normalization by dividing each element A(i,j) of the matrix by the sum (or max) of the elements in that particular (ith) row, i.e. A(i,j) = A(i, j) / \sum_{k=1}^{n} A(i,k)

However, a more elegant way to achieve this would be to apply a probabilistic dimensionality reduction technique such as PLSA or LDA. These dimensionality reduction techniques ensure that you always end up with probability values strictly between 0 and 1.


Seeking advice on database architecture -- given my problem, what tools should I learn? I'm a fairly experienced R user, but until now I haven't had a good reason to learn to use databases.  Now I have a problem where I am dealing with model output that I need to save to disk, and then query for another process.  If the data were smaller, I'd store everything in a list, with hierarchical elements.  For example, if my object is called output.OLS:

1> summary(output.OLS)
        Length Class  Mode
SEP0307 3      -none- list
SEP0308 3      -none- list
SEP0306 3      -none- list
SEN0308 3      -none- list
<<snip -- hundreds of sub-lists>>
SEN0307 3      -none- list

the second level of hierarchy:

1> summary(output.OLS[[1]])
      Length Class  Mode
c1  11     -none- list
c2  11     -none- list
c2  11     -none- list

the third:

1> summary(output.OLS[[1]][[1]])
         Length Class      Mode
Baseline 1008   data.frame list
CanSEM45 1008   data.frame list
CanSEM85 1008   data.frame list
CCSM45   1008   data.frame list
CCSM85   1008   data.frame list
GISS45   1008   data.frame list
GISS85   1008   data.frame list
HadGEM45 1008   data.frame list
HadGEM85 1008   data.frame list
MIROC45  1008   data.frame list
MIROC85  1008   data.frame list

and we can see that now we're looking at data.frames

1> dim(output.OLS[[1]][[1]][[1]])
[1]   33 1008
1> colnames(output.OLS[[1]][[1]][[1]])[1:10]
 [1] "scen"     "year"     "region"     "size"    "amount"     "mean"
 [7] "semean"   "varres"   "result.1" "result.2"
<<snip>>
[1006] "result.998"  "result.999"  "result.1000"

There are four such output objects, for a total of about 6GB.  They don't all have the columns mean, semean and varres

This list is about 1.5 gigs, which is unwieldy given what I want to do with it.  It'd be much better to read subsets of it from a database.  I can think of two potential architectures:

Architecture 1: Single, huge table
The variable names would be region (the outermost hierarchy), cx (i.e.: c1, c2, c3), model (Baseline, CanSEM, etc), and then all of the results: result.1, result.2, ..., result.1000

This would be most convenient.  The problem is that the table would be both deep and wide.  Too wide for SQLLite, which is the simplest database, as far as I can tell.

Architecture 2:: one keyvalue table and many data tables.  The keyvalue table would have the following columns:
cx, scenario, year, place, keyvalue = coerce_to_integer(unique(scenario, year, place, cx))

The data tables would look like this
keyvalue, result_number, result

This option uses hierarchy and is actually a relational database (relational via the keyvalues, right)?

But it has the drawback that it nearly triples the size of my data:  for each datum, I need to store a keyvalue and an index (result_number).  This seems like it wouldn't make sense.

Are there other options
For example, are there databases similar to R's lists that can be stored in memory, and read from memory?  Is that sort of like what NoSQL databases are?

So, my question

What database architectures would make sense for this problem?
Where should I start learning about these sorts of databases?
How does R interface with them?  Can I create the database from R?  If so, how?

I used R for several years but have since moved to Python and so I have a hard time understanding output.OLS's data hierarchy. Nonetheless, here are my thoughts.

In-Memory Databases: If you're struggling to fit your R object in memory, then I'm guessing the memory requirement is too great for your laptop (i.e., the problem isn't due a 32-bit installation of R). If that's the case, putting an in-memory db, like MongoDB, on your laptop will only transfer the memory problem to a new technology. In other words, 6 GB in R will be 6 GB in Mongo, so if your laptop simply can't handle 6 GB, then it doesn't matter how you put it there. You could set up a MongoDB on AWS, which could hold 100s of GB, but this can be expensive and also difficult if you don't have AWS experience.

On the other hand, MongoDB is good at storing hierarchical data like lists and hash tables / dictionaries (I forgot what R's term for that is) and is fast if you have enough memory.

Relational Databases (RDMS): I would recommend going this route and creating a wide and deep table. You can still get value out of an RDBMS even if your machine has limited memory because everything is efficiently stored on disk. If you create table indexes on the columns you expect to use/filter/select most, you can very quickly query the data that you want even if your data is multiple GB. If you don't create table indexes, your queries will be horribly slow, and you'll probably abandon the RDBMS effort entirely (so make sure to index).

A long time ago my go-to package for accessing MySQL through R was RMySQL. MySQL has been around for many years, so you'll be able to find most answers to your problems on Goggle, but MySQL user interfaces like Sequel Pro (for Mac only) have sort of fallen behind on maintenance, so setting up tables and databases can be a little more challenging than they used to be. You'll now have to go the command-line route.

Postgres has now replaced MySQL as the open-source DB of choice, but it lacks good user-interface support and so you'll probably have to set everything up via the command line. I started using Postgres after I stopped using R, so I never used a Postgres R package, but it looks like there is one here.

Whether you go with MySQL or Postgres, you'll probably have to use the command-line outside of R to set up the database, but after that creating, dropping, joining, or querying tables can all be done through R. RDBMS systems are also great because you can guarantee data quality (no duplicates) by creating primary keys -- I found this feature very helpful for my R modeling. In any case, SQL is a very basic language that every data-oriented person should know so setting up an RDBMS and possibly learning SQL are skills worth learning.


Target and output shape/type for binary classification using PyTorch so I have some annotated images that I want to use to train a binary image classifier but I have been having issues creating the dataset and actually getting a test model to train. Each image is either of a certain class or not so I want to set up a binary classification dataset/model using PyTorch. I had some questions:

should labels be float or long?

what shape should my labels be?

I am using a resnet18 class from torchvision model, should my final softmax layer have one or two outputs?

what shapes should my target be, during training, if my batch size is 200?

what shape should my outputs be?

Thanks in advance
Quote Delete

According to your questions:

Labels should be long and advised.
[num_samples, ]
It should have two outputs.
If your batch_size=200 then target somehow similar to this: [0, 1, 0, 1, 1, 0, ....1]
3rd ^


Need help understanding data leakage I am a newbie to this stuff so I am sorry if my question is stupid~

I need help understanding what data leakage between X_train and X_test is and when exactly it happens.
I am currently working on a dataset where I am using KNN imputer to fill in the missing values. I need to scale the data for knn imputation and I am doing the train-test-split and applying machine learning models after the imputation process. I read that data leakage might happen during scaling so we should scale after splitting, fit_transform the train set, and only transform the test set.
I am unsure about how that would work in my case since I am scaling the data to be able to impute for missing values and I actually reach the train-test-split stage later. Should I be worried about data leakage the way I am doing things?

Here is the code:

Although here I am doing the splitting + applying DT algorithm right after imputation, I have other steps like feature selection left so I won't reach the train-test-split and decision tree part until much later.

Data leakage occurs whan you have in your training set information which you are not supposed to have. This might be, for example, having in the training set information contained in the test set. Generally speaking, data leakage can provide a model which is "too good to be true", but not to be trusted as it was build using "cheatful" information. Find more information here.


How to deal with the difference in the range of Dependent variable in Train and Test data I am training a XGBoost model for predicting number of applications and the minimum number of applications in training data is 40 and the maximum number of applications is 2000, while in test set minimum number of applications in training data is 400 and the maximum number of applications is 800. Hence the model is getting confused while predicting the value.
Any suggestions on how should I solve this issue?

The assumption in supervised ML is that the test set follows the same distribution as the training set. It's clear from your description that this assumption is not satisfied in your data, so it's not very surprising that things don't work well. Beyond the difference in the range of the dependent variable, if the training and test set have been collected independently it's likely that the distributions of the features also differ.
So how to fix this depends on the goal of the task and in particular why the test set and training set don't follow the same distribution:

If the difference is not meaningful, i.e. there is no reason to assume that real production data would be different than the training set, then the experiment should be redone after properly randomizing the split between training and test set.
If the difference is meaningful, i.e. the test set is purposefully different from the training set because the production data will also be different, then the setup should be redesigned to account for that. For example there could be a step of semi-supervised learning on the test set, which would adapt the model to the new data distribution while leveraging the information from the training data.


How to resolve country and nationality entities? I've tried stemming and  lemmatization on this but nothing has quite worked so far.

How can I resolve country name and nationality as a singular entity?

For example:

Canada and Canadian should just be one entity: Canada
Uganda and Ugandan should just be Uganda

It does seem like stemming is one approach here. I just have found it misses a fair number of countries.

I believe that Lemmatisation is the right thing. Anyways another way to do it is to use WordNet. For a word which its POS is Noun you can query if it has a member holonym, this feature will show the country which it belongs. For instance, Canadian -> Canada. Then you have to be careful, cause if you take the member holonym of Canada that will be the British Commonwealth. I guess you could have a Levenshtein distance threshold to ignore these.

You can have a look on the online web app.