using feature selection to improve model performance I have a highly sparse dataset that I am using to predict a continuous variable via a random forest regression. I have achieved an acceptable level of performance following cross-validation, and I am now thinking of potential ways I might further improve accuracy.

Given that my dataset is highly sparse, I was thinking that recursive feature elimination (using the cross validated version in sklearn) might be a good way to go. My understanding is that this will give me the 'optimal' number of features and thus may reduce issues related to over-fitting.

My question is, is it then appropriate to re-run the analysis with these optimized features, or am I in someway biasing the model? I have a test set that has not been used at all in training/validation, so I am assuming that as long as I don't leak info between training and testing, I should be good. But I'm unclear if these assumptions are correct.

Is this a suitable use of RFE, or should i be considering a different path?

For info, my training/validation dataset is 370 rows, with approx. 900 features.

370 rows, with approx. 900 features is not optimal. I would suggest some dimension reduction. PCA, factor analysis, PLS regression are some alternatives.

You could try a lasso- / elasticnet -regression as well.

Here is a good guide. https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html


How to slice an input in keras? I give to keras an input of shape input_shape=(500,).
For some reasons, I would like to decompose the input vector into to vectors of respective shapes input_shape_1=(300,) and input_shape_2=(200,)
I want to do this within the definition of the model, using the Functional API. In a way, I would like to perform slicing on a tf.Tensor object.
Help is welcome!

If it's just the input you like to decompose, you can preprocess the input data and use two input layers:
import tensorflow as tf

inputs_first_half = tf.keras.Input(shape=(300,))
inputs_second_half = tf.keras.Input(shape=(200,))

# do something with it
first_half = tf.keras.layers.Dense(1, activation=tf.nn.relu)(inputs_first_half)
second_half = tf.keras.layers.Dense(1, activation=tf.nn.relu)(inputs_second_half)
outputs = tf.keras.layers.Add()([first_half, second_half])

model = tf.keras.Model(inputs=[inputs_first_half,inputs_second_half],outputs=outputs)

data = np.random.randn(10,500)
out = model.predict([data[:,:300],data[:,300:]])

If you like to split after the input layer you could try reshaping and cropping, e.g,:
inputs = tf.keras.Input(shape=(500,))

# do something
intermediate = tf.keras.layers.Dense(500,activation=tf.nn.relu)(inputs)

# split vector with cropping
intermediate = tf.keras.layers.Reshape((500,1), input_shape=(500,))(intermediate)

first_half = tf.keras.layers.Cropping1D(cropping=(0,200))(intermediate)
first_half = tf.keras.layers.Reshape((300,), input_shape=(300,1))(first_half)

second_half = tf.keras.layers.Cropping1D(cropping=(300,0))(intermediate)
second_half = tf.keras.layers.Reshape((200,), input_shape=(200,1))(second_half)

# do something with decomposed vectors
first_half = tf.keras.layers.Dense(1, activation=tf.nn.relu)(first_half)
second_half = tf.keras.layers.Dense(1, activation=tf.nn.relu)(second_half)
outputs = tf.keras.layers.Add()([first_half, second_half])

model = tf.keras.Model(inputs=inputs, outputs=outputs)

data = np.random.randn(10,500)
out = model.predict(data)

The Cropping1D() function expects a three-dimensional input (batch_size, axis_to_crop, features) and only crops along the first dimension, therefore we need to add "pseudo-dimension" to our vector by reshaping it.


How do you visualize Decision boundaries? My features for a problem are just a single feature, so X_train has just a single dimension. Now, I have used SVM , naive Bayes, and Logistic Regression. So now How do I draw the decision boundaries of these classifiers? Is there is a way to draw a one dimension feature vector with the decision boundary using python? I have done many searches but I couldn't find any results.  The X_train shape is (489, 1) (489 instances each with a single value)

To visualize a decision boundary of a classifier, specifically a binary classifier as in your case, you can instantiate a grid of points that spans the domain of interest that you want to classify. Then use your trained classifier to predict the class of each and every point in the grid. If the grid resolution is fine enough, when you plot the contours of the grid points with color corresponding to its class, you can then visualize the decision boundary. It is helpful to overlay the original training data points so that you can see how your decision boundaries are generated based on various training sets. See the following example


Sales person performance prediction I am trying to predict/forecast salesperson performance weekly, monthly, quarterly, and yearly based on the products that they sold over 3 years. As part of this effort, I grouped their number of units sold by the state, location, stores, product types, and sale dates.

By using this information I grouped their id and month of sales and number of units sold. Example data is shown below.

For ARIMA modeling, does this format work well enough to predict/forecast salesperson performance weekly, monthly, quartely, and yearly?

Id      Month   Units
65381   201703  467.0
65381   201710  3.0
65381   201712  6.0
65381   201803  20.0
65381   201805  2.0
65381   201807  20.0
65381   201812  16.0
65381   201904  2.0

Here is a simple way of fitting a linear model to each salesperson by week, month and quarter. You'll need to extract the week, month and quarter from the date using the lubridate package.

library(lubridate)
library(tidyverse)

#create fake data
n_var <- 50
id <- sample(x=seq(from=1,to=4),size=n_var,replace=TRUE)
dte <- sample(seq(as.Date('2018/01/01'), as.Date('2019/01/01'), by="day"), size=n_var)
sales <- runif(min=0,max=25,n=n_var)

sales_data <- data.frame(id,dte,sales) %>%
  mutate(
    weeknumber=week(dte),
    month=month(dte),
    qtr=quarter(dte)

  )

#fit an OLS model
ols_model <- sales_data %>%
  group_by(id) %>%
  summarise(
    week_model=list(lm(sales~weeknumber)),
    month_model=list(lm(sales~month)),
    qtr_model=list(lm(sales~qtr))
    )

#extract coefficients
ols_model$week_model[[1]]$coefficients
ols_model$month_model[[1]]$coefficients
ols_model$qtr_model[[1]]$coefficients

ols_model$week_model[[2]]$coefficients
ols_model$month_model[[2]]$coefficients
ols_model$qtr_model[[3]]$coefficients

#simple version on one id
df <- sales_data %>%
  filter(id==1)

mod <- lm(sales~qtr,data=df)
mod$coefficients


How to compare the performance of different number of mixing components for EM algorithm? I am reading about the EM (Expectation-Maximization) algorithm in a machine learning book. At the end remark of the chapter, the authors mentioned that we cannot decide the "optimality" of the number of components (# of mixtures of Gaussians distributions) based on each model's log likelihood at the end--since models with more parameters will inevitably describe the data better.

Therefore, my questions are

1) How do we compare the performance of each model using a different number of components?

2) What are the important factors that help us to decide that an EM model is sufficient for modeling the observed data?

1.

The simplest and most common way is to use AIC or BIC. You would pick a model with the minimum AIC/BIC value. AIC/BIC work well here because you have a likelihood function.

Bayesian model selection is another possibility. It's more advances than AIC or BIC, but you get chance to add your own prior distribution. Section 5.3 in

  Machine Learning - A Probabilistic perspective (Kevin P.Murphy)

has the details. There're also lots of papers on Google.

Alternatively, you can use the cross-validated likelihood as a performance measure, although this can be slow, since it requires fitting each model N times, where N is the number of CV holds.

Slide 17 in

  https://www.doc.ic.ac.uk/~dfg/ProbabilisticInference/IDAPISlides13.pdf

is a good reference.

2.

EM algorithm is a technique for estimating the most likely parameters and is not limited to mixed Gaussian. The algorithm is useful when there's no closed form formula for the solving maximum likelihood problem with one or more latent variables.


Visualize n-dimensional bayesian optimization results I am working on a 6-dimensional bayesian optimization problem using (skopt's gp_minimize).
After the optimizer ran for j iterations I would like to somehow visualize the "progress/result" of the optimization. As I am new to Bayesian optimization I would like to ask for input on how and what to visualize. What are good parameters to visualize to show the improvement and maybe even the parameter dependency of the optimized parameters?

One way to do it is to look at the "trajectory" of the best point as a function of the number of functions evaluations. When the objective function is 2D, you can actually look at contours of the objective functions and plot the best point trajectory at the same time, just like I did here:
http://infinity77.net/go_2021/thebenchmarks.html#solver-solvers-example
For higher dimensionalities, you can try and track a one-dimensional evolution of the objective function and its parameters. Example below with N=3:
import numpy
import matplotlib.pyplot as plt

from scipy.optimize import dual_annealing

def rosen(x, function_history, params_history):
    """
    Rosenbrock objective function.

    This class defines the Rosenbrock global optimization problem. This is a
    minimization problem defined as follows:

    .. math::

       f_{\\text{Rosenbrock}}(x) = \\sum_{i=1}^{n-1} [100(x_i^2 - x_{i+1})^2 + (x_i - 1)^2]

    Here, :math:`n` represents the number of dimensions and
    :math:`x_i \\in [-5, 10]` for :math:`i = 1, ..., n`.

    *Global optimum*: :math:`f(x) = 0` for :math:`x_i = 1` for :math:`i = 1, ..., n`

    Jamil, M. & Yang, X.-S. A Literature Survey of Benchmark Functions
    For Global Optimization Problems Int. Journal of Mathematical Modelling
    and Numerical Optimisation, 2013, 4, 150-194.
    """

    f = numpy.sum(100.0 * (x[1:] - x[:-1] ** 2.0) ** 2.0 + (1 - x[:-1]) ** 2.0)

    params_history.append(x)
    function_history.append(f)

    return f

def solve():

    N = 3
    numpy.random.seed(1)
    bounds = list(zip([-5.] * N, [10.0] * N))
    minimizer_kwargs = {'method': 'L-BFGS-B', 'options': {'maxiter': 100, 'disp': False, 'ftol': 1e-6},
                        'bounds': bounds}

    x0 = numpy.random.uniform(-5.0, 10.0, N)
    function_history, params_history = [], []
    res = dual_annealing(rosen, bounds, x0=x0, seed=1, local_search_options=minimizer_kwargs,
                         args=(function_history, params_history))

    fig = plt.figure()
    ax = fig.add_subplot(2, 2, 1)

    start_f = function_history[0]
    nfun = [1]
    f_progression = [start_f]
    params_progression = [params_history[0]]

    for k, f in enumerate(function_history):
        if f < start_f:
            nfun.append(k+1)
            f_progression.append(f)
            params_progression.append(params_history[k])
            start_f = f

    ax.plot(nfun, f_progression, lw=2)
    ax.set_ylabel('$F(X_1, X_2, X_3)$', fontsize=16)
    ax.set_yscale('log')
    params_progression = numpy.array(params_progression)

    for i in range(3):
        ax = fig.add_subplot(2, 2, i+2)
        ax.plot(nfun, params_progression[:, i], lw=2)
        ax.set_ylabel('$X_%d$'%(i+1), fontsize=16)

    for ax in fig.axes:
        ax.grid()
        ax.set_xlabel('Functions Evaluations', fontsize=16)

    plt.show()

if __name__ == '__main__':
    solve()

You get a picture like the one below:

Not perfect, but one way to do it :-) .
Good luck.


Predicting future automobile sales I am new to Machine Learning. Recently I am trying to build a model to predict the sales of a particular automobile make and model of a dealership and to which location. The data given to me has information about the dealer name, state, zip code, owner state, zip code, vehicle make, model, year, vehicle segment, body type, odometer reading, new/used indicator. How do I build a model to do predictive analysis?

Welcome to the forum (and machine learning). Choosing an estimator/model can sometimes be the hardest problems, as such there are many different opinions and ways to do so.

The first thing I notice with your problem statement is that you're wanting to predict the Sales (either in $ or quantity, I would assume). Because you're not trying to predict a category or binary label, we can rule out classification models and look at instead regression style models.

A logical starting point for this would be to explore using a Linear Regression. This assumes that you have some existing data with the price, quantity, or target you're trying to predict.

Linear Regression is a logical starting point as they're simple to implement, yet can get some pretty good results. Scikit-learn has a nice package in Python on linear regressions.

If you don't get satisfactory results using Linear Regressions, you may wish to explore Decision Trees, either scikit-learns implementation or xgboost. This rules out classification models.

Briefly, I think it's important to note that the features to your model are nearly all categorical, eg. make, model, etc. To handle these features, you'll need to encode these features, a guide on that is here.

Take your time, explore your data a lot and start with a simple model. The more you understand your data, the more you'll understand the model. Enjoy and have fun! :)


Help with understanding cross-validation My understanding of cross-validation is that we divide our data set into parts 1-k, then use part 1 as a validation set and parts 2-k as a training set, then use part 2 as a validation set and the remainder as a training set, etc., until we've used each part as a validation set.

What I don't totally understand is:

What is the actual goal here? I get that we take an average of the k rounds of cross-validation that we do, but what is our output? I've read that we discard the result of each round after calculating the validation error, so how do we actually produce a model from this?
What is the difference between a validation and test set, if any? i.e., we normally separate our data into training and test data, but do we then further divide our training data into validation sets, and keep our test data aside during the cross-validation process, or do we run cross-validation on the entire data set?

What is the actual goal here? I get that we take an average of the k rounds of cross-validation that we do, but what is our output? I've read that we discard the result of each round after calculating the validation error, so how do we actually produce a model from this?

The goal is to obtain an accurate estimate of the model's performance. Cross-validation produces unbiased model predictions for every example in the dataset. We use these predictions to compute metrics, like accuracy or log loss, for the training dataset. When we are actually building a model, we will use the entire dataset rather than cross-validation.

  What is the difference between a validation and test set, if any? i.e., we normally separate our data into training and test data, but do we then further divide our training data into validation sets, and keep our test data aside during the cross-validation process, or do we run cross-validation on the entire data set?

A validation set is a set of labeled examples used during the training phase, but not actually for training the model. This can be done in many ways. For example, when building a model through an iterative process (like gradient descent for neural networks), we can use a validation set to decide when to stop iterating. The performance on the training set almost always improves after each training step, but will eventually overfit and produce poor predictions on unseen data. In this scenario, we can test the model on the validation set to detect when overfitting occurs.

A test set, on the other hand, is only used to evaluate the model after training is complete. Optimising our training for performance on the validation set means we are slightly biased towards it, and we should finally test on a truly held-out set of labeled data to evaluate the performance of the model.

Cross-validation is intended to "replace" the need for a test set for providing an unbiased estimate of model performance, so you usually would not bother with having a test set when doing cross-validation. However, if you have a large dataset it is generally recommended to split into train/test rather than to use cross-validation.


Pandas dataframe resample aggregation by mills too slow Given this test data:

import pandas as pd
import numpy as np

data = {'date': ['2014-05-01 18:47:05.069722', '2014-05-01 18:47:05.119994', '2014-05-02 18:47:05.178768', '2014-05-02 18:47:05.230071', '2014-05-02 18:47:05.230071', '2014-05-02 18:47:05.280592', '2014-05-03 18:47:05.332662', '2014-05-03 18:47:05.385109', '2014-05-04 18:47:05.436523', '2014-05-04 18:47:05.486877'],
        'battle_deaths': [34, 25, 26, 15, 15, 14, 26, 25, 62, 41], 'prisioners': [3, 4, 3, 2, 2, 6, 4, 5, 2, 8]}
df = pd.DataFrame(data, columns = ['date', 'battle_deaths', 'prisioners'])

Set index datetime:

df = df.set_index(pd.DatetimeIndex(df['date']), drop=True)
del df['date']

It works when I want to resample to the milliseconds, but it takes too long...

timeit df.resample('1L').sum()

I guess is because is aggregating all the milliseconds with NaN data, but when I drop it ..

timeit df.resample('1L').sum().dropna()

It takes even longer

guessing again that the dropna is done at the end... There is any way which by dropping the NaN samples will accelerate the process?


Ensembling vs clustering in machine learning Following the raise of ensembling (e.g ensembling of xgboost learners) after its recurrent wins in Kaggle competitions, using it has become a tradition in machine learning. However, some argue that clustering is a form of ensembling where we apply clustering on a data set first, then applying a learning algorithm (to do classification or regression) on each of the resulting clusters.

I would like to understand how clustering is different from ensembling?

EDIT

I understand that clustering can be similar to the idea of ensembling if we simply consider "homogeneous ensembling" where we train the same model on each of the clusters. But I am looking for potential limitations of clustering where using heterogeneous ensembles (different learns for all the data) is the "only" way to go about the problem.

Short answer: Ensembling and clustering are completely unrelated techniques.

Ensembling: Combine the strengths of many diverse models. Ensembles generally do not involve training models on separate sets of data -- it's the models themselves that are different. Generally the more diverse the models the better. For example, an ensemble might comprise the following models: a support vector machine (SVM), a random forest, a gradient boosting model (GBM), logistic regression, k-nearest-neighbors, and a neural network. On Kaggle, I have also seen top competitors create ensembles comprising the same type of model (e.g., random forests) but with a variety of tuning parameters (e.g., one with 100 trees, another with 250 trees, etc). Clustering could be one of the models included in an ensemble. Ensembles are very effective (although extremely complex to operationalize) and can generally improve your error about 2-5%.

Clustering: Divide the data into mutually exclusive groups with the hope that your cluster-specific models can reduce error by specializing, except in the case of fuzzy clustering where the groups need not be mutually exclusive. At financial companies that build credit risk models, these cluster-specialized models are sometimes called "sub-models" in the sense that they predict the same target variable and therefore fit under an abstract/figurative parent model. This makes model risk reporting easier in the sense that it's simpler to refer to model 1 then model 1a, 1b, 1c, 1d. Clustering in this context is generally based on business knowledge and not a formal clustering technique such as k-means or other types of centroid-based clustering. For supervised learning tasks, formal clustering is generally considered a pretty poor technique because measuring the quality of the clusters is subjective. Your clusters can also vary dramatically depending on how you initialize the centers and how many clusters you specify. In practice, clustering is used much less frequently than ensembling.


What exactly is A dataset? is a database contain more than 2 tables counted as 1 or 2 datasets? A data set (or dataset) is a collection of data. Most commonly a data set corresponds to the contents of a single database table, or a single statistical data matrix, where every column of the table represents a particular variable, and each row corresponds to a given member of the data set in question.

It seems that the iris data set could be counted as 1 dataset.

Is a database that contains more than 2 tables counted as 1 or 2 datasets?

A dataset might refer to any collection of information about a specific topic. Namely, a file, an excel file, a .txt file, etc.

Usually, we use datasets that are saved in a table format (like .csv files)
a dataset in this form is exactly like a table having columns (fields) for the name of its features and rows that are the records (samples) and an index for each row.
A problem might be solved using multiple datasets (tables) but each dataset is referred to a single table (data frame)

Whatever the format is, for utilizing, we turn the dataset to a table (dataframe)


Understanding why shuffling reduces weirdly the overfit I am a student currently trying to create a classification model, however I am having difficulty understanding a weird overfitting problem.

A dataset of about 30 000 entries, 30 features. The data is sorted by date of entry. I split my data to 80% training & 20% testing. I get a training accuracy score of about 98% and a test accuracy of about 71% using random forest. When I remove 3 specific parameters, the overfitting disappears, and I get to 73% training accuracy and 68% test accuracy. Which means these 3 parameters cause a big deal of overfitting. The weird thing happens when I shuffle the data. With all the 30 parameters, the training accuracy remains 98% and the test accuracy gets up to 92%. Which for me indicates that these 3 features values change unexpectedly during the last month or so of the data (the data was sorted by date before shuffling) and shuffling them gives the model enough examples from this last month to pick up the sudden change. But plotting the mean of their values/day, for the whole spectrum of dates shows that they follow the same seasonality throughout the whole data, and there are no weird changes. Can someone please give me some ideas to explain why shuffling the data helps to reduce the overfit massively?

There is a crucial assumption made by any supervised ML approach: both the training set and the set are samples drawn from the same population. This means that the model expects to see the same distribution of features in the test set as in the training set.

In order to make sure that this assumption is satisfied, it is important to shuffle the instances before splitting between training set and test set. This will avoid any visible or invisible bias due to the order in which the data was collected or assembled. That's the effect that you can observe in your first experiment: the main issue is not so much about overfitting due to the model relying too much on some parameters, it's that neither the training or test set are a random subsets of the population. It might make complete sense for the model to rely on these 3 parameters based on the training set, the problem is that they don't behave the same way in the test set. "behaving in the same way" includes their relation to other features, so the difference might not be visible by looking at these features independently. This bias (which probably affects more than the 3 features) causes a huge discrepancy when evaluating the model on the test set.


Classification affected by a lot of outliers in features? How do you deal with outliers? I  am working on a classification problem and I found my data having  a lot of outliers which has resulted in reduction in my recognition rate. I have tried rescaling, normalization techniques like min max, box cox and even log transformation.
I am considering of eliminating outliers from box plots but I am afraid I might be eliminating useful features/data required to define the model.

Are there any suggestion on how to deal with such cases.
Also further analysis of data revealed that my data constitutes of features belong to dfferent process like web application, apps. I segregated the data based on the processes and I  do see that large variation of process  resulted in different accuracy ranging from 60-95%

Any tips on how to deal with such cases? In the end I want my classifier to classify irrespective of the process type. So with my current issue, does this imply that my features defined are not good enough or is there something else I can do?

If the number of outliers is small and you are concerned that they will destabilize your solution, you could attempt a random forest classifier. The RF fits trees to random selections of data and variables, and collects "votes" from each, thus reducing the impact of outlier valuers.

On the other hand, if the number of outliers is fairly substantital, you might want to create a new class called "outlier". In the training set, apply this label to those values you have deemed to be outliers and then fit the model with the augmented class. Check if the model correctly identifies outliers in the test set.

This is equivalent to removing the outliers, only it creates a repeatable and machine learning way of doing so.


How do I iterate over my images in dataset? I am building an autoencoder with help from this site. There I was trying to build an autoencoder for my own custom data. My images are stored in a folder IMG and have names like 0.jpg, 1.jpg, 2.jpg.....

I tried to develop an iterator to iterate over all my images but the problem arises that when I convert all of my 124 images in a single training_data array the model responds that it expected a single array yet 124 arrays were given to it. Can anyone tell me how I should write the iterator? I tried using the keras flow_from_directory function from the "Machine Learning mastery" website but it shows 0 images from 0 classes.

Here is my code:-->

import tensorflow as tf
from keras.layers import Input, Dense, Conv2D, MaxPooling2D, UpSampling2D
from keras.models import Model
from keras import backend as K
from keras.callbacks import TensorBoard
import numpy as np
from PIL import Image

i = int(0)
images_dir = "/home/awesome_ruler/Documents/Atom projects/Compression_enc/Images/IMG/{}.jpg".format(i)
training_data = []

while i < 125:
    print("working on ", i, 'file')
    image = Image.open(images_dir)
    pic_array = np.asarray(images_dir)
    training_data.append([pic_array])
    i += 1

input_img = Input(shape=(600, 400, 3))  # adapt this if using `channels_first` image data format

x = Conv2D(48, (3, 3), activation='relu', padding='same')(input_img)
x = MaxPooling2D((2, 2), padding='same')(x)
x = Conv2D(24, (3, 3), activation='relu', padding='same')(x)
x = MaxPooling2D((2, 2), padding='same')(x)
x = Conv2D(24, (3, 3), activation='relu', padding='same')(x)
encoded = MaxPooling2D((2, 2), padding='same')(x)

# at this point the representation is (4, 4, 8) i.e. 128-dimensional

x = Conv2D(24, (3, 3), activation='relu', padding='same')(encoded)
x = UpSampling2D((2, 2))(x)
x = Conv2D(24, (3, 3), activation='relu', padding='same')(x)
x = UpSampling2D((2, 2))(x)
x = Conv2D(48, (3, 3), activation='relu')(x)
x = UpSampling2D((2, 2))(x)
decoded = Conv2D(1, (3, 3), activation='sigmoid', padding='same')(x)

autoencoder = Model(input_img, decoded)
autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')

autoencoder.fit(training_data,
                epochs=50,
                batch_size=128,
                shuffle=True,
                callbacks=[TensorBoard(log_dir='/tmp/autoencoder')])

  Also I want the images to retain the 'color' feature so I am using the input shape as (600,400,3) because RGB is on 3 channels. is it correct?
   I would have simply used my iterator but it is my understanding that I need a different function that communicates to the model and gives it images one-by-one while I am just loading them all in a single variable. So can anyone help me with this?

Here is the full TraceBack:-

Traceback (most recent call last):
  File "autoencoder.py", line 46, in <module>
    callbacks=[TensorBoard(log_dir='/tmp/autoencoder')])
  File "/home/awesome_ruler/.local/lib/python3.7/site-packages/keras/engine/training.py", line 1154, in fit
    batch_size=batch_size)
  File "/home/awesome_ruler/.local/lib/python3.7/site-packages/keras/engine/training.py", line 579, in _standardize_user_data
    exception_prefix='input')
  File "/home/awesome_ruler/.local/lib/python3.7/site-packages/keras/engine/training_utils.py", line 109, in standardize_input_data
    str(len(data)) + ' arrays: ' + str(data)[:200] + '...')
ValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 1 array(s), but instead got the following list of 125 arrays: [array([['/home/awesome_ruler/Documents/Atom projects/Compression_enc/Images/IMG/0.jpg']],
      dtype='<U76'), array([['/home/awesome_ruler/Documents/Atom projects/Compression_enc/Images/IMG/0.jpg']]...

Your images_dir actually seems to be the path to a single image... but nevertheless,

I would simply create a single number array with shape: (num_images, height, width, channels) by doing the following:

import os
import numpy as np

# Root directory holding all images (I recommend removing the space in "Atom projects")
images_dir = "/home/awesome_ruler/Documents/Atom projects/Compression_enc/Images/IMG/"

# Number of images you want to load
N= 125

# Get all paths and take the first N
n_image_paths = sorted([f.path for f in os.scandir(data_root)])[:N]

# Load the images using on of the variants of loading images
images = np.array([Image.open(f) for f in n_image_paths])

# Be careful with variants: the order of channels is different for different methods!
# images = np.array([plt.imread(f) for f in n_image_paths])    # Matplotlib
# images = np.array([cv2.imread(f) for f in n_image_paths])    # OpenCV

Now images can be passed directly to your model:

autoencoder.fit(x=images,
                epochs=50, ...)


What are the hidden states in the Transformer-XL? Also, how does the recurrence wiring look like? After exhaustively reading the many blogs and papers on Transformers-XL, I still have some questions before I can say that I understand Transformer-XL (and by extension XLNet). Any help in this regard is hugely appreciated.

when we say hidden states are transferred from one segment to another, what exactly is included in these hidden states? Are the weights of the networks implementing the attention mechanism (i.e. calculating the Q, K and V) included? Are the weights involved in calculating the input word embedding included in the hidden state?
When the hidden states are transferred during recurrence, is this transfer from the encoder of one segment to the encoder of the next segment? Or is it from the decoder of the current segment to the encoder of the next segment? Is the decoder involved at all in the hidden state transfer?
I see images like the following in the following in the papers and blogs. what do the dots represent? encoders? decoders? or an entire unit? I guess the answer to my second question will shed a light on this one too.

Thank you

By hidden states, they mean outputs of the layers, i.e., what you get after the feed-forward sub-layer. For Transformer-XL, it is important that these are also what you use as an input to the self-attention. Therefore, at inference time, if you want to compute the states recursively by segments (presumably because you cannot fit the entire input int he memory), this is the only thing you need to remember from the previous steps to continue the computation.
There is no encoder, you can imagine Transformer-XL as a decoder-only model. Transfering the states just means remembering them, so you can do the self-attention over them, but you can no longer back-propagate through them because you only remember the values and the entire graph telling how you got them.
The dots in the scheme correspond to the hidden states: one state per input subword and per layer. The lines between them are the self-attention links.


Is it possible to use the saved xgboost model (with one-hot encoding features) on unseen data (without one-hot encoding) for prediction? I think the question is self-explanatory. But let's say you have a data with a few features with categorical data, and when building a model for example XGBoost you one-hot encode categorical features. Now you want to do prediction based on test data using the saved model. Obviously the test data needs to be one-hot encoded and need have similar features as training set. The question is whether it is possible to find a way not one-hot encode the test data and directly use it for prediction? Would this be somehow possible?

To me it appears that whatever comes in to my saved model need to be as it was used during training i.e. one-hot encoded features! But this is not neat, especially when building widgets and dashboards!

Any comments/hints are appreciated.

A model is built on a specific set of features, which may include categorical features encoded using one-hot encoding.  If you have new data with additional categories, your model has no idea how to interpret the significance of those categories.  You should either map the new value to none of the 1-hot values identified in training, or to an 'other' value.

For example, say you trained on data that had color=[blue,green].  Your one-hot fields would have color_blue and color_green.  You could also have a field called color=other, that you might use to encode very infrequent values.  That's a data preparation choice. So for 'red', you could encode that as either:

color_green = 0
color_blue = 0

or

color_green = 0
color_blue = 0
color_other = 1

Using either of these techniques will work with xgboost, but as xgboost only accepts numeric inputs, you will have to choose one of these methods as a data pre-processing step.


solving multi-class imbalance classification using smote and OSS I am trying to solve multi-class imbalance classification problem for that i am using SMOTE for oversampling and OSS for under-sampling. But I have a doubt as I am working on multi-class so i have to convert it into binary classification. So we can convert it using OVA/OAA. So how can I use OVA/OAA with both under-sampling and oversampling on the same data-set.

Please solve my problem, It will be a great help.

If you convert your problem to a binary classification task, you do not need to worry about any conflict with re-sampling techniques. You can  then just use the imblearn.combine which combines Oversampling and under-sampling in one algorithm. There are classes available for this but I have found SMOTEENN to produce better results. SMOTEENN


When is Non-Stochastic Global Optimization Preferable or Necessary? Background
I'm specifically referring to non-convex black-box optimization problems of the form:
$ \text{min} f(\vec{x})$

$s.t. \ \ a_i\le x_i \le b_i \ \forall i\in \{1,2,...,n\} \ \ \ \text{and}\ \ \ \vec{a},\vec{b}\in \Bbb{R}^n $
Assume any $n$
Of interest are global optimizers that yield good solutions, but the solutions are not necessarily nor provably the global optimum.
Evolutionary Algorithms
Evolutionary algorithms (EAs) are often the go-to optimizers for these types of problems; such methods include: genetic algorithms, particle swarm optimizers, differential evolution, and all the algorithms based on organismal interactions. Pretty much every EA has stochastic components. Whether it's random initialization, or stochasticism in inter-generational (or intra-generational) subroutines such as selection for crossover or random mutations, stochasticism is pretty ubiquitous in the realm of EAs. Almost everything you'd find in this journal or this one would fall under this category.
Non-Stochastic Optimization vs. Deterministic Global Optimization
I am not interested in deterministic global optimizers. Such methods provide some form of probability/confidence or even guarantee that the found solution is indeed the global optimum. These are more commonly seen in discrete/combinatorial optimization, but occassionally some deterministic optimizers become tangentially related when the user has some form of a priori knowledge/assumptions. The preference/necessity of deterministic optimizers is clear, even when they just give associated confidence with the solutions they find. So again, I am not referring to these.
Non-Stochastic Global Optimizers
I only know of a few non-stochastic global optimizers. Probably the most famous are the many variants of direct search (also called pattern search) algorithms. Conceived by Fermi and Metropolis, then popularized by Hooke and Jeeves, and extended to a generalized pattern search (GPS) making heavy use of positive bases as meshes, the direct search algorithms are similar to the classic Nelder-Mead method in that they use a neighborhood of points with an underlying geometric structure to (deterministically) explore the search space. Of course some non-stochastic variants exist as well, including Luus-Jaakola's sampling of a uniformly distributed neighborhood or the more popular mesh adaptive direct search (MADS) and all of its spin-offs.
There are some other non-stochastic global optimizers hiding out there on the internet, like this one, but I have yet to find one that explains the practical significance of the non-stochasticism.

Question
What are some concrete use-cases for a non-stochastic global optimizer as described in the above-mentioned background?
Are there situations where non-stochastic optimization is necessary? Possibly mission-critical optimization, or where you need repeatability? Maybe something medically-oriented? Or for interpretability?
The only example I can think of (coming from an ML/DL background) is a situation where it would be slightly preferred, but certainly not necessary. In particular, we could train an ML model using a non-stochastic optimization algorithm, which would let us observe the effects of the ML model hyperparameters. In other words, eliminating the stochasticism in the optimizer could help interpret/tune the actual ML model hyperparameters as you'd be able to see causes/effects of modifications, where currently there's uncertainty due to the randomness involved in training.

Non-stochastic global optimization could be practical for the directed search of the solution space. If there is a priori knowledge, that knowledge can direct the optimization to specific regions. Stochastic global optimization might explore inefficiently explore regions of the solution space.
A specific example of non-stochastic global optimization is Bayesian optimization. In Bayesian optimization, samples are not picked at random. Samples are picked based the calculated posterior distribution, often to maximize the expected improvement.
Bayesian optimization has been used in many applications, including machine learning hyperparameter tuning.


Any advantage of sklearn wrappers for xgboost over python API? Are there any advantages of using the XGBoost sklearn wrappers XGBRegressor or XGBClassifier over using the Python API with the train method?
Is one faster than the other or does one use less memory than the other?
I am asking for a generic case where the data comes in some form which I convert to a pandas dataframe.

One advantage of using scikit-learn wrappers is that they are more likely to work within the scikit-learn ecosystem. For example, it might be easier to put the algorithm into pipelines.


Set for building ROC curve and and choosing logistic regression cut-off I'm building a logicsic regression classifier for binary classification. I have trained it and going to choose a cut-off value using ROC curve.

But what set to use for it: training or validation?

Usually you need to generate the ROC curve and choose the threshold within the training data. Then, with the selected threshold, you have the possibility to report accuracy, sensitivity, recall results you reach on your validation set.


How can we create neural net to detect false predictions? I created a convolutional network to recognise certain substrings.

For example, the following phrases would be mapped to the "What" class:

What are you doing?
What you doing?
Whatcha doing?

The following to the "How":

How are you doing?
How you doing?

Now, say I have millions of examples and thousands of classes, my network learns relatively well. However, sometimes a new "phrase" that belongs to a never seen class appears. Of course, the model cannot map it to something it has never seen, however, when I look at the output layer values, I was hoping to see all neurons to have small values. My idea was that, I would use the softmax and then if the max value is less than 0.95 or some high value, I would assume the network is unsure, therefore remove that result for manual observation.

However, in many cases, I have seen my network to be almost sure (a neuron with value close to 1) even when it has never seen that class.

For example, say I have the two classes What and How, I would expect:

Whatcha doing? --> "What"= 0.95, "How"= 0.05
How you doing? --> "What" = 0.02, "How"= 0.98
Where you going?--> "What"= 0.6, "How"=0.4

In the last case, the network is not too sure on either, and I would check the input. Unfortunately, this is not working, and I am getting high values (close to 1) neurons for some new input.

How can I fix this? Please, bear in mind, I am happy to sacrifice some accuracy in order to ensure the network never tells me it is sure about new inputs, in other words, I am happy to go and manually check more inputs (even if seen before) rather that let even one unseen input go unnoticed.

One thing you could try is to change the softmax activation on your final layer to sigmoid. This will give a value between zero and one for each class, the sum of which may not equal one. This means that for a given example there may be more than one output neuron activated, or no output neurons activated, in which case you can treat the example as 'unsure' and manually check it out. In the case that there is exactly one output neuron activated (should be most of the time), that should be the classification given.


Lower training accuracy than testing accuracy (MLP/Dropout) I am working on a problem of multi-class classification by MLP. I have set dropout to each middle layer. Now I observe the training accuracy is around 10% less than the testing accuracy.
My guess is, dropout is active only during training but inactive during testing. So part of the neurons are reset at training (leading to low accuracy), but it is not happening for testing.
My questions:

Is my understanding correct? In other words, if I remove the dropout part, will the training accuracy increase but the testing accuracy will remain the same?
When reporting the MLP accuracy, should I report training acuracy or testing accuracy?

Yes, your understanding is correct: dropout can be the cause. Nevertheless, you can only be sure if you evaluate on the training data with dropout disabled.

This depends on the context, but normally you would report the test accuracy.


Which model is better, one just before overfitting with higher accuracy or one with no overfitting and lower accuracy? I am training a CNN model.
In the first one I got a training accuracy of 87%(0.29 loss) and validation accuracy of 87%(0.30 loss) at 5th epoch, I kept training it for total of 15 epochs and as expected it started overfitting with training accuracy increasing to 97%(0.01 loss) and validation remaining at 87%(0.35 loss).
In the second model, I used Data Augmentation and Dropout layer to deal with overfitting (trained for total 10 epochs). These are the results : 5th Epoch : Train Accuracy 77% (0.45 loss) and Validation accuracy 77% (0.41 loss). 10th Epoch : Train Accuracy 82% (0.38 loss) and Validation accuracy 82% (0.35 loss)
From the loss and accuracy graph which you can see below, it's clear that Model is overfitting in first scenario, but in second it's not overfitting.
Scenario One

Scenario Two

My question is, which model is better in real world based on accuracy ? Model one stopped at epoch 5 with 87% accuracy or model 2 which has not overfit with 82% accuracy (validation)? I understand just based on accuracy model 1 sounds better but it's ultimately overfitting, but if I stop the training using early stopping or something similar, will this be a better model than my second one ?

Right now I would say model 1 is better since it has better validation results (If you take the model at epoch 5, not the overfitted one).
However, I would procceed by retraining model 2 for more epochs and check if the results improve.


What is the difference between horizontal and vertical ensemble? I am looking at different ways to do model ensembling and I came across the terms horizontal and vertical blending/ensembling but it is not well defined.

My questions will be:

What is the difference between horizontal and vertical blending?
In which cases which one is better?
Are there any python implementations for them?

I think the most highly-referenced source for these terms is Horizontal and Vertical Ensemble with Deep Representation for Classification (Xe, Xu, Chuang 2013). That would be the best place to get a technical answer to your first question. For purposes of searching, you could also look into "Stacking" as a synonym for Vertical ensembling. I will provide some intuition here:

Ensembling is, to over-simplify, adding new content to your data based on results from other or intermediate modeling results. In this case, "horizontal" and "vertical" are references to where in your dataset you add this content. Horizontal ensembling takes the results from one model (such as the predicted classification) and appends it as a new column to your data. In the case of a binary classifier result, this would effectively be adding a column that says "A different model classified this row as a 1 or 0 (A or B, red or green, etc)." The final model would use this information in its own process as a new feature. Vertical ensembling adds new predicted values (as rows, if you want to think about a dataframe paradigm) that are generated by a model. To be clear, Horizontal ensembling adds what some external/interediate model thinks is the right classification while vertical ensembling adds what the model predicts as a value.

Consider a problem that is trying to predict whether the image is "cat" or "dog". Based on the features in the row, a model can say "cat", which gets added to that row when the Horizontal ensemble adds the results of a different classifier. Think of that as an extra vote that the image is of a cat. Or a Vertical ensembling can add new information from that external model for the predicted probabilities of the image being "cat" or "dog", maybe as a tuple: (0.73, 0.27).

Give the article a good read to see when you might prefer one over the other, and the statistical issues that arise in both cases. And, as always, keep the No Free Lunch Theorem in mind.
Ensembling can refer to multiple things. Boosting methods can be considered ensembling, since these algorithms are doing intermediate calculations to see what learners are performing better in order to shift weights in the modeling. These methods are built into the algorithms themselves, so I would look into scikit learn and its ensembling methods to get a sense of what's going on. To your specific concern, however, you can look through Github repos for ensemble learning.


Anonymize continuous variable for masking purposes I am about to kick off a large hackathon event.
We have a dataset that is comprised of one continuous variable with high precision, and a number of categorical variables qualifying these data 3-levels deep.
Data provider wants to 'mask' the data such that the original values cannot be reverse-engineered.  I'm not worried about the categorical variables, this is simple.  But the continuous variables are tricky.

a logarithmic transformation is easily reverse engineered
a nonlinear transformation is better, but will mess with the relationship of values between categories
a pure linear transformation would work, but doesn't seem to 'mask' enough.

I need to preserve the relationships between numbers whilst also protecting the actual, true values.
Ideas greatly appreciated.

I think you can use a much more complicated monotonic transformation, like
log(1.234578 + sqrt(x + 7.4142) ** 3)

which will be harder to invert than a simple log. But, as Nikos says, strictly monotonic functions are invertible, so all you can do is make it very hard to compute the inverse by composing many monotonic functions.


What is a good approach for embedding both textual and spatial features for document classification? I am working on a document classifier that can perform the classification based on the document structure as well. My plan is to get the word embedding as well as the word coordinates and somehow combine the two features and pass it through a Graph Convolutional Network (GCN) to generate a graph embedding which I can then use to train a classifier. I was referencing this paper and they do data extraction by first getting the text embedding and image embedding and then combining them using elementwise addition. I was wondering what would be a good solution to combine both text features and positional features (x,y,height and width).

When text is available as scanned image:

Divide your image into small grids.
Assign  each grid a row/column number like (i,j)
Now to your word vector append 2 more cells which are the row and column number of cell to which the word belongs.

When text is available as
Have the document in the form of html. Then have the embeddings for entire html DOM tree of that document, which would include both  tags and the actual text. This way html tags would be giving the spatial/positional information.


error while running lasso.py The following is the error code generated while running lasso.py.
Can anybody help in fixing the same.

Here is the code:

from cvxpy import *
import numpy as np
import cvxopt
from multiprocessing import Pool

# Problem data.
n = 10
m = 5
A = cvxopt.normal(n,m)
b = cvxopt.normal(n)
gamma = Parameter(nonneg=True)

# Construct the problem.
x = Variable(m)
objective = Minimize(sum_squares(A*x - b) + gamma*norm(x, 1))
p = Problem(objective)

# Assign a value to gamma and find the optimal x.
def get_x(gamma_value):
    gamma.value = gamma_value
    result = p.solve()
    return x.value

gammas = np.logspace(-1, 2, num=100)
# Serial computation.
x_values = [get_x(value) for value in gammas]

# Parallel computation.
pool = Pool(processes = 4)
par_x = pool.map(get_x, gammas)

for v1,v2 in zip(x_values, par_x):
    if np.linalg.norm(v1 - v2) > 1e-5:
        print("error")

Error: while running

Traceback (most recent call last):
  File "lasso.py", line 31, in <module>
    objective = Minimize(sum_squares(A*x - b) + gamma*norm(x, 1))
  File "/usr/local/lib/python3.5/dist-packages/cvxpy-1.0.11-py3.5-linux-x86_64.egg/cvxpy/expressions/expression.py", line 45, in cast_op
    return binary_op(self, other)
  File "/usr/local/lib/python3.5/dist-packages/cvxpy-1.0.11-py3.5-linux-x86_64.egg/cvxpy/expressions/expression.py", line 435, in __sub__
    return self + -other
  File "/usr/local/lib/python3.5/dist-packages/cvxpy-1.0.11-py3.5-linux-x86_64.egg/cvxpy/expressions/expression.py", line 45, in cast_op
    return binary_op(self, other)
  File "/usr/local/lib/python3.5/dist-packages/cvxpy-1.0.11-py3.5-linux-x86_64.egg/cvxpy/expressions/expression.py", line 423, in __add__
    return cvxtypes.add_expr()([self, other])
  File "/usr/local/lib/python3.5/dist-packages/cvxpy-1.0.11-py3.5-linux-x86_64.egg/cvxpy/atoms/affine/add_expr.py", line 33, in __init__
    super(AddExpression, self).__init__(*arg_groups)
  File "/usr/local/lib/python3.5/dist-packages/cvxpy-1.0.11-py3.5-linux-x86_64.egg/cvxpy/atoms/atom.py", line 41, in __init__
    self._shape = self.shape_from_args()
  File "/usr/local/lib/python3.5/dist-packages/cvxpy-1.0.11-py3.5-linux-x86_64.egg/cvxpy/atoms/affine/add_expr.py", line 41, in shape_from_args
    return u.shape.sum_shapes([arg.shape for arg in self.args])
  File "/usr/local/lib/python3.5/dist-packages/cvxpy-1.0.11-py3.5-linux-x86_64.egg/cvxpy/utilities/shape.py", line 49, in sum_shapes
    len(shapes)*" %s" % tuple(shapes))
ValueError: Cannot broadcast dimensions  (10,) (10, 1)

As suggested by multiple links, I tried with all possible python and library versions. But I am unable to fix this. Kindly throw some light.

(A*x)  Expression has dimensionality of (10, ) and b has shape of ( 10, 1) - this is why you see this error.

My fix solves the error, but you should double check the results

objective = Minimize(sum_squares(A*x - np.array(b).reshape(10,)) + gamma*norm(x, 1))

Hope this helps!


Perceptron learning rate irrelevant in convergence Via this MIT document i studied the single layer Perceptron convergence proof (= maximum number of steps).

In the convergence proof inside this document , the learning rate is implicitly defined as 1 .
After studying it, by myself i tried to re-do the proof inserting this time a generic learning rate $\eta $.

The result is that the result remains the same:

$k\leq \frac{R^{2}\left \| \theta ^{*} \right \|^{2}}{\gamma ^{2}}$

that is the learning rate $\eta$ cancelled out in the proof.

Is it possible , or i make mistakes in my proof ?

Your conclusion is correct.

Note that the classifier is of the form of

$$f(x; \theta) = \operatorname{sign}(\theta^Tx)$$

while the update rule is

$$\theta^{(k+1)}=\theta^{(k)}+\eta y_tx_t$$ which only occurs when there is a misclassification and we only care about the in the classification but

$$\operatorname{sign}(\theta^Tx)=\operatorname{sign}(\eta \theta^Tx)$$

as long as $\eta$ is positive.

It only scales $\theta^Tx$ but our concern is just the sign, hence the learning rate is irrelevant.

Remark: You might like to note the assumptions being used in the proof though. For instance, there is this assumption that $\theta^*$ exists. It is assumed that the data can be perfectly separated by a hyperplane.


Convergence of vanilla or natural policy gradients (e.g. REINFORCE) I am reading in a lot of places that policy gradients, especially vanilla and natural, are at least guaranteed to converge to a local optimum (see, e.g., pg. 2 of Policy Gradient Methods for Robotics by Peters and Schaal), though the convergence rates are not specified.

I, however, cannot at all find a proof for this, and need to know if and why it is true.  I am wondering the following:

What is the proof of this?  Can someone point me to a reference?
What is the best convergence rate out there that's known (if any are proven)?
If I am reading at least the Peters/Schaal paper correctly, this only holds for discounted reward.  Is there any formulation that works for the mean reward of a rollout?  I am in a situation where I am running many simulations of a robot and most directly want to optimize its mean reward.

What is the proof of this? Can someone point me to a reference?

It is called "The Policy Gradient Theorem", and a good reference would be Sutton & Barto Reinforcement Learning: An Introduction In the second edition, the theory behind REINFORCE and Actor-Critic is examined in Chapter 13.

In brief, the proof is focused on showing that a sample of a particular expression based on reward seen and the parameters of the policy function, is a sample of the gradient of the policy functions with respect to the parameters. It thus follows that a step in the direction of this sampled value, will in expectation be a step in the direction that increases expected sum of discounted reward (or the average expected reward) associated with that policy.

  What is the best convergence rate out there that's known (if any are proven)?

This is not something that can be proven analytically, except in situations where you already know the loss function expressed in terms of the policy parameters (and nothing else that depends on them). Even for simple toy environments this is not really possible. The usual approach is to compare different algorithms experimentally, plotting learning curves. There is a lot of variance in practice, so learning curves are often plotted as averages of multiple training scenarios.

As far as I know, there is no single clear winner in a general sense between different policy gradient methods. However, Actor-Critic and similar methods that combine TD learning with policy gradients are usually preferred over the basic approach of REINFORCE, as the agents that estimate a value function allow "bootstrapped" updates to be made on every step. Although this adds initial bias (as value function estimator parameters initially have no relation to the true value function), it reduces variance and allows for more updates, thus often faster convergence.

  Is there any formulation that works for the mean reward of a rollout?

The base theory of policy gradients applies to episodic, continuous discounted, and continuous average reward formulations with minor changes in the proof.

The algorithm REINFORCE cannot be applied to a continuous environment, as updates are made only at the end of episodes. Actor-Critic approaches can be adapted to average reward formulation - for examples see this survey.


Which functions neural net can't approximate I read somewhere on the StackExchange that a neural network can't approximate the Pi number as a function of circles length and radii. Maybe this is an incorrect example or wrong information. Help me to understand.

What about the sum or multiplication of any arbitrary numbers? Are there any other specific functions neural networks can't approximate or not?

A neural network can approximate any continuous function, provided it has at least one hidden layer and uses non-linear activations there. This has been proven by the universal approximation theorem.

So, there are no exceptions for specific functions. You ask:

  I read somewhere on the StackExchange that a neural network can't approximate the Pi number as a function of circles length and radii.

A neural network to approximate $\pi$ is very easy. Possibly what you read is that a neural network cannot generate new digits of $\pi$ that it has not already been shown. More on that later . . .

  What about the sum or multiplication of any arbitrary numbers?

Yes, a neural network can approximate that.

  Are there any other specific functions neural networks can't approximate or not?

No, there are no specific functions that a neural network cannot approximate.

However, there are some important caveats:

Neural networks do not encode the actual functions, only numeric approximations. This means there are practical limits on the ranges of inputs for which you can achieve a good approximation.
A neural network being able to approximate a function in theory is not the same thing as you or I being able to construct a neural network that approximates that function. There is no known method to construct a neural network by analysis of a function alone (it can be done for specific simple functions such as xor).
The usual way to achieve approximation is to train a neural network by giving example data. The network will approximate to data it has been shown. There is no guarantee that this will generalise to new inputs that it has not been trained on and approximate the correct outputs. In fact for certain types of input/output it cannot possibly do so. For instance, it will not learn how to generate the 4th digit of $pi$ if it has been shown digits 1,2,3,5,6,7,8,9. The best generalisation results occur for functions that have smooth transitions between the training examples.
Neural networks do not extrapolate well to inputs outside of the data they have used for training. They "fit" to the training data (imagine a rubber sheet draped over all the points in the training set).
Neural networks do not learn to copy algorithms, only functions. So if you take a complex algorithm, such as AES encryption, and attempt to train a neural network to perform this given lots of input examples, has no real chance of working. Now, AES encryption can be considered a function e.g. $output = encrypt( input, key )$. So the NN can approximate it. But it will only do so for the specific inputs and outputs it has been shown. In addition AES does not respond well to approximation - a single bit wrong will cause it to be a bad encryption. So you won't see NNs used to encrypt or decrypt in cryptography.
The capability of a neural network to approximate is limited by the number of neurons and connections it has. More complex functions require larger networks. In order to train a larger network on a more complex function takes more time and more training data. You could in theory train a neural network to learn a random number generator function. However, that would take an impossible amount of resources - memory to store the network, and time to train it against the whole output of the RNG.


Tokenize text with both American and English words I need to tokenize a corpus of abstracts from an international conference. The abstracts are usually American English but sometimes British English.

Consequently, I get 2 tokens for “organization” and “organisation” or “color” and “colour”.
Examples : https://en.oxforddictionaries.com/spelling/british-and-spelling

Do you know a (python) library converting “British English” to “American English” (or vis versa) ?

I would be happy to that ... (but I am french and my english is not soo good)

Thanks.

Grouping related tokens is called text normalization.

There is not an established Python package that does this. You could create a custom dictionary or write a function to rewrite the tokens.


Logistic Regression in R models for 1 or 0? According to this link, it says SAS models for 0,

  To model 1s rather than 0s, we use the descending option. We do this because by default, proc logistic models 0s rather than 1s

What happens in case of R's glm function? Does it model for 1 or 0? Is there a way to change it? Does it matter?

I always learned to model for 1. See below for the impact of switching the encoding,

model <- glm(ans ~ x, data = simulated_data, family = binomial)
summary(model)

#
# Call:
# glm(formula = ans ~ x, family = binomial, data = simulated_data)
#
# Deviance Residuals:
#       1        2        3        4        5
#  1.6388  -0.6249  -1.2146  -0.8083   0.7389
#
# Coefficients:
#             Estimate Std. Error z value Pr(>|z|)
# (Intercept)  -2.4753     2.5006   -0.99    0.322
# x             0.5957     0.6543    0.91    0.363
#
# (Dispersion parameter for binomial family taken to be 1)
#
#     Null deviance: 6.7301  on 4  degrees of freedom
# Residual deviance: 5.7505  on 3  degrees of freedom
# AIC: 9.7505
#
# Number of Fisher Scoring iterations: 4
#

simulated_data$ans <- !simulated_data$ans
model_opp <- glm(ans ~ x, data = simulated_data, family = binomial)
summary(model_opp)

#
# Call:
# glm(formula = ans ~ x, family = binomial, data = simulated_data)
#
# Deviance Residuals:
#       1        2        3        4        5
# -1.6388   0.6249   1.2146   0.8083  -0.7389
#
# Coefficients:
#             Estimate Std. Error z value Pr(>|z|)
# (Intercept)   2.4753     2.5006    0.99    0.322
# x            -0.5957     0.6543   -0.91    0.363
#
# (Dispersion parameter for binomial family taken to be 1)
#
#     Null deviance: 6.7301  on 4  degrees of freedom
# Residual deviance: 5.7505  on 3  degrees of freedom
# AIC: 9.7505
#
# Number of Fisher Scoring iterations: 4

Hope this helps.


Appending DataFrames to lists in a dictionary - why does it seem like the list is being referenced by each new DataFrame? I have a DataFrame that pairs one or more labels to a sample group and id, for a given sample stored in a database at SampleGroup/SampleID:

There are ~100 labels. I want to create binary models to do classification on each label, and then run these models in parallel to do multi-class classification. To store these models, I am creating a dictionary of form

{label_1:[df_1, model_object_1],
 label_2:[df_2, model_object_2],
...,
label_n:[df_n, model_object_n]
}

Where each df is a DataFrame of the form above, except that the value of the 'Labels' column is replaced with a 1 or 0, depending on whether dictionary key 'label_i' is in the original label list for that row. Here's the code that (should) do that, that has been giving me some trouble:

models = dict.fromkeys(target_labels, [])

for label in target_labels:
    label_list = []
    for multi_label_list in df['Labels']:
        if label in multi_label_list:
            label_list.append(1)
        else:
            label_list.append(0)

    data = {
        'SampleGroup':df['SampleGroup'].copy(),
        'SampleID':df['SampleID'].copy(),
        'Labels':label_list
    }

    models[label].append(pd.DataFrame(data=data, index=df.index))
    print(len(models[label]))

When I run this, each new binary label_list that is created for a label gets appended to every model in the dictionary, as if I'm creating a reference to the same label_list (similar to how df2 = df would create a reference to df, instead of a copy). The output of the above code tells the story clearly:

[

I managed to hack a fix for this by assigning each new DataFrame to the key instead of appending it to the key's value list:

models[label] = (pd.DataFrame(data=data, index=df.index))

What property of DataFrames (or perhaps native Python) am I invoking that would cause this to work fine, but appending to a list to act strangely?

The problem is when you create the dictionary models using models = dict.fromkeys(target_labels, []), you actaully only created one empty list, and all keys point to that list. Everything you append goes into that same list.

For example,

models = dict.fromkeys('abcd', [])
print(models)
print(models['a'] is models['b'])
models['a'].append(3)
print(models)

will return

{'d': [], 'a': [], 'b': [], 'c': []}
True
{'d': [3], 'a': [3], 'b': [3], 'c': [3]}

Notice that models['a'] is models['b'] is True.

You can create the dictionary instead using

models = dict([(key, []) for key in 'abcd'])
print(models)
print(models['a'] is models['b'])
models['a'].append(3)
print(models)

Now it returns

{'d': [], 'a': [], 'b': [], 'c': []}
False
{'d': [], 'a': [3], 'b': [], 'c': []}

Notice that models['a'] is not models['b'].


Relationship between log-odds and weighted sums in Logistic Regression I've read several articles/tutorials on Logistic Regression and I've come across this idea of log-odds being equal to the weighted sum of features.

i.e. if $p$ is the probability of a sample belonging to positive class(target variable: 1), $(1-p)$ is the probability of it belonging to the negative(target variable: 0). Then, for input features $x_1$, $x_2$, $x_3$ ... $x_n$ and weights $\theta_0, \theta_1, \theta_2... \theta_n$, they write

\begin{equation}
log\bigg(\frac{p}{1-p}\bigg) = x_0 \theta_0 + x_1 \theta_1 + x_2 \theta_2 \dots x_n \theta_n
\end{equation}

where $x_0$ is bias

Now I get what features are, I get what weighted sum of feature is, I know about the odds, I know log, what I cannot comprehend is, how are those two quantities equal? I've gone through plenty of online articles but they always say that these two are equal, but they don't explain how.

Ordinary regression, Logistic regression, Poisson regression are all examples of Generalized Linear Models (GLM).

A GLM can be broken down into 3 components:

A random component: Target Variable $Y_i$ which is assumed to follow a distribution with $E(Y_i)$ and $Var(Y_i)$
A systematic component: A linear combination of the features variables known as the linear predictor; $\gamma_i= \beta_{0} + \beta_1{x_1} \dots + \beta_n{x_n}$
Link Component: The link function, $g(\cdot)$, connects the random component to the systematic component. If $E(Y_i) = u_{i}$, then the link function connects $\mu_{i}$ to $\gamma_{i}$ through some function $g(\cdot)$, and we then have that
$g(\mu_i)= \beta_{0} + \beta_1{x_1} \dots + \beta_n{x_n}$. Moreover, $E(Y_i) = \mu_i = g^{-1}(\beta_{0} + \beta_1{x_1} \dots + \beta_n{x_n})$

The link function for linear regression is the identify link: $g(\mu_i) = \mu_i$

The link function for logistic regression is the logit link: $g(\mu) = log(\frac{\mu}{1-\mu})$. The logit link is used when $\mu$ takes values between 0 and 1 since the function maps to -infinity to infinity.

There are a lot details which I have skimmed over on purpose, GLM is a rich subject. I would recommend reading more about it, here are some references below:

References:

Categorical Data Analysis by Alan Agresti

An Introduction to Categorical Data Analysis

Hope this helps!


Finding the most phonetically similar word from WordNet Besides soundex and other libraries, that take two words and determine whether they are similar, is there any way to find the most similar sounding word from WordNet, for a given word? I tried to use autospell as a well to correct the spelling, but unable to get the desired results.

You could use e-speak to convert your words into a phonetic alphabet. What you can also do is use fuzzy-matching. Here is a nice blog post about this.


Is there a way to rank the Extracted Named Entities based on their importance/occurence in a document? Looking for a way to rank the tens and hundreds of named entities present in any document in order of their importance/relevance in the context.

Any thoughts ?

Thanks in advance!

An easy way would be to use TF-IDF (term frequency–inverse document frequency). It can help you find how much terms stand out in a document (by comparing with your entire corpus) and use it to rank your entities.

TfidfVectorizer from scikit-learn

Just note that the TfidfVectorizer is on a word level. So some processing will be needed if your entities can consist of more than one word.

Alternatively you could use a model that allows you to produce a heatmap of the words. Then you can use that heatmap to look up your NEs in that heatmap. This paper, A Structured Self-Attentive Sentence Embedding, could give you some ideas.


Forecasting/predicting techniques for qualitative data? I have a food alert dataset composed of nominal qualitative variables, such as type of alert, country of origin, action taken, etc. as well as the date on which the alert was recorded.

What techniques are there to predict what type of alert is most likely in a time window when there are only qualitative variables in our dataset?
Or given a variable (for example, country of origin = France) predict the rest of the variables?

I have used machine learning algorithms before but not with datasets composed of nominal qualitative variables.

On the other hand, what are the most appropriate techniques for clustering and classification this kind of datasets?

I know it's a broad question, so I expect broad answers to guide me in a general way

I have about 50000 registered alerts. In the example of the image not all the variables are shown

You can encode the various columns of qualitative data using LabelEncoder(). For example:

label_encoder = LabelEncoder()
risk = label_encoder.fit_transform(df['RISK_DECISION'])
action = label_encoder.fit_transform(df['ACTION_TAKEN])
etc...


How to do vocabulary estimation based on observed writings? Below is a scatter plot of the data set I am dealing with. The X axis is the total number of words per essay for a particular individual, and they Y axis is the number of unique words. In principle, the number of unique words should approach the individuals vocabulary.

I am attempting to find that individual's vocabulary from the data below, but I don't know what kind of a fit would work. A logarithm would have no limit, a quadratic fit doesn't make sense (the gradient should remain non-negative over the entire domain).

In short, I am looking for a decent model to fit the data below, and don't know where to start.

Thank you.

In my opinion, this estimation cannot be achieved merely based on this plot, because:

From 4000 words onward, the unique words are increasing linearly around 250 per 2K words: (4K, 1.25K), (6K, 1.5K), (8K, 1.75K), (10K, 2K), (12K, 2.25K). So there is not enough evidence to hypothesize an upper-bound for this linear growth,
On average, an adult knows 20K-35K unique words, but this plot goes only up to 2K which is far behind the final expected value. The extrapolation from 2K to 20K is unreliable.

Vocabulary of Shakespeare

The estimation of a person's vocabulary is quite complicated. Below is a paper that estimates the vocabulary of Shakespeare. He had used 31K unique words in all of his writtings. The paper estimates that he knew at least 35K more words which he did not use (at least 66K vocabulary). As you see, the estimated vocabulary is only twice the observation, which sheds light on unreliability of going from 2K to 20K and beyond.

1976 Estimating the number of unseen species - How many words did Shakespeare know


Clustering of news combining headline and main article I want to classify German police news articles and do an automated classification/clustering with regards to the kind of crime committed. Thus far I am not getting great results. Often times the headline is pretty telling but sometimes it is not that revealing, so I need to include the main article to do a proper clustering on all articles.

However the headline is a very good indicator and I don't want to throw it away. I could simply combine the words from headline and main article but I think there should be more weight given to the headline. Has somebody an idea how to do this?

Put more weight on the title.

For example if you are using tf-idf, you first compute the tf-idf vectors of headline and body separately. Then you combine them into a single vector by simply adding both vectors and rescaling to unit length. That would put 50% of the weight onto the headline even though it is much shorter.


Integration of Sentiment analysis in CRM What is the process for integrating sentiment analysis in a CRM? What I am searching for is a system which analyzes the customer comments or reviews using the CRM and finds out the customer sentiment on the services provided by the system or company or a product.

I have done a sentiment analyzer which takes text and shows the sentiment of the text. Now I want to integrate the above-mentioned sentiment analyzer to a CRM, how can I do that?

After a lot of searching, I got one paper on the integration of opinion mining in CRM, which helps to get an idea about the process involved.


What is the difference between normalization and re-scaling? This site does not describe the nature of the normalization tag. How does it differ from re-scaling? Many authors use the two terms interchangeably. I can not understand normalization's operational meaning.

Re-scaling means that you multiply your variable by a factor, i.e. $x \to x/a$. A normalization is a specific kind of re-scaling, where the factor $a$ is such that the values of $x$ become of order one. Its form depends on the context and what you are trying to do. Examples are $a= \langle x \rangle$ or $a= \textrm{max}(x)$. Perhaps the most common one is when you do $x \to (x-\langle x \rangle)/\sigma$, because if $x$ follows a Gaussian distribution, the re-scaled variable follows a normal distribution (average 0 and std 1). Note, this last transformation actually involves a re-scaling, but also a translation.


What are some standard ways of computing the distance between individual search queries? I made a similar question asking about distance between "documents" (Wikipedia articles, news stories, etc.).  I made this a separate question because search queries are considerably smaller than documents and are considerably noisier.  I hence don't know (and doubt) if the same distance metrics would be used here.

Either vanilla lexical distance metrics or state-of-the-art semantic distance metrics are preferred, with stronger preference for the latter.

The cosine similarity metric does a good (if not perfect) job of controlling for the document length, so comparing the similarity of 2 documents or 2 queries using the cosine metric and tf idf weights for the words should work well in either case. I would also recommend doing LSA first on tf idf weights, and then computing the cosine distance\similarities.

If you are trying to build a search engine, I would recommend using a free open source search engine like solr or elastic search, or just the raw lucene libraries, as they do most of the work for you, and have good built in methods for handling the query to document similarity problem.


Is it advisable to combine two dataset? I have two datasets on heart rate of subjects that were recorded in two different places (two different continent to be exact). The two research experiments aimed to find the subjects' emotions based on how much their heart rate change over time. I am using machine learning to predict the subjects' emotions and i am getting acceptable result when tested separately on each dataset. However, i get even better result if i merge the two datasets.

I am not however sure if combining the two datasets is acceptable. As I am combining two somehow different datasets, will it create statistical bias? How should i report my finding in a journal paper?

Adding to what @Super_John said, if adding continents as a Feature, then you can also probably have at-least 2 more features as well,

The Latitude
The Longitude

Also add another temporary column to indicate the Source (like $1$ to $1st$ df, $2$ to $2nd$ df etc), So that we can add Colors to the k-means

So now we can have a k-means Cluster to see whether values are overlapping or not...
(Trying to see it in an Unsupervised Way)

(The analogy is equivalent to the  fact that that you can cluster time(24  hours in a day) in a cyclic fashion , like plotting  $sin(x)$, $cos(X)$ and then trying to cluster them)

Have a look at this answer,
Features Selection, Extraction


Exploration vs exploitation tradeoff to find a price that maximizes revenue Is there a practical strategy that can learn to price a product optimally? Right now I have the following arbitrary hill-climbing algorithm:

Run an experiment at starting price P and gather 500 data points (e.x. 20 buy and 480 not buy).
Run a t-test on what confidence level P yields higher revenue per visitor than P * 1.1 and P * 0.9. Then do a 3-way weighted coin-flip and the winner gets to run the next experiment.

There's many problems with this approach. For example, if price is at optimal, it can't price a product at a more optimal pricing e.x. P * 1.03. Another is that if at some price point P = K we happen to get really unlucky and get 1 buy of 500 data points, the algorithm won't converge fast.

The problem gets easy if we take lots of data points but that would reduce long term revenue. Is there a fast algorithm that can converge to the optimal price and then not do anymore exploration?

Without making any underlying assumptions you will not get anywhere. That said, there are multi-arm bandit strategies that try to optimize the rewards, there is a ton of research on this field. It comes down to sampling from a distribution of your options (in your case two) and adapting this distribution based on the rewards.

https://en.wikipedia.org/wiki/Multi-armed_bandit

Once you know that the reward distribution from each bandit comes from a specific distribution, you can deduce optimal sampling strategies. Once you have at least some prior information you can do fairly well although not always optimal. Regardless, most strategies will do better than normal A/B testing if the strategy is not super greedy.


Meaning of this notion in 0-1 loss? I am reading a paper and encountered this notion:

$$1_{\{Y=1\}}$$

To me it seems to be the expression as below, but I am not entirely sure and I don't think the author explictly explained it:

if Y==1:
  return 1
else:
  return 0

Can someone help me to clarify this notion? Much thanks for your time (:

It appears in:

https://papers.nips.cc/paper/5073-learning-with-noisy-labels.pdf

Your understanding is correct.

This is known as the indicator function.

The indicator function of a subset $A$ of a set $X$ is a function

$$1_A(x)= \begin{cases}1, & x \in A \\ 0, & x \notin A \end{cases}$$


Do you apply outlier detection of numerical data in practical applications? In data science we often get raw data to work on. It is the main task to draw conclusions from the training data that can be generalized to future unseen data.

Do you apply outlier detection in your usual routine? By this I mean: among $N$ training data decide $K<<N$ data points to be outliers and discard them.

What are your experiences doing this? Which methods do you apply for numerical data.

You'd have to think about where the outliers come from. Technical malfunction? Then they are probably safe to disregard. But when you write that your main task is

  to draw conclusions from the training data that can be generalized to future unseen data

you have to keep in mind that your future data will contain outliers as well and deal with them appropriately. Models like trees are usually somewhat robust against those while regression can go haywire. There are several well working methods like disregarding everything that is more than 4 standard deviations (or median deviations from the median) out or just disregarding the n-th top and bottom percentiles.

But then there is the case where outliers are not just flukes and can actually tell you something about the problem domain you are working on. If you work in fraud detection, the outliers are the points that interest you. If you work on optimizing network performance, the packets that don't arrive can be the interesting ones.

So my advice is to not blindly throw away data out of principle but always think about what the data represent and why outliers may be present.


pdf to json libraries I am looking for a library which converts pdf to json. Basically in that json the paragraph heading is the and the value is the content of paragraph. Is there any python library for that ? I am already using pdfminer but that just converts to plain text. It cannot persist the structure/organisation of the document. For now it is ok to not read images and table although if there is a library to do that would be great.

It cannot persist the structure/organisation of the document.

To tackle this, you can look at pdftotext (which comes by default in case you are on a linux system). It preserves the exact layout of the pdf page. You will have to specify the layout option.

For now it is ok to not read images and table although if there is a library to do that would be great.

Table extraction is a different(and more complicated) problem. However if the tables are not that complicated, you can use Camelot/Tabula to extract those. I would suggest you to go with Camelot as it also provides a lot of tunable parameters that you change for your case specific tables. (More details :Camelot  )


What can help decrease outliers' influence on non-tree models? I have a feature with all the values between 0 and 1 except few outliers larger than 1. I am trying to collect all the methods that can help to decrease outliers' influence on non-tree models:

StandardScaler
Apply rank transform to the features
Apply np.log1p(x) transform to the data
MinMaxScaler
Winsorization

I wasn't able to imagine any other ... I guess that's all?

Here are couple of other options:

Set a threshold and remove all values larger than the threshold

RobustScaler which removes the median and scales the data according to the quantile range.

QuantileTransformer which transforms the feature to follow a uniform or a normal distribution.


A Question: is it possible for me to create an AI that learns to play games which I did not create? Mainly, I have a question that I could not find an answer for anywhere, about an AI machine learning game.
Is it possible for me to create an AI which can learn a game that I did not myself create, without any source codes or APIs? And I'm talking about simple games such as chrome t-rex, snake, etc. I was told this is possible using image processing but I am still not so sure. I am referring to just open up a game, not a downloaded one, a one available at google, such as the mentioned above, run my code, and it will start playing and learning the game. Just creating 1 algorithm to learn multiple easy 2D google games, is such a thing possible with image processing?

As far as I know you might archieve this with Reinforcement Learning. But there are some limitations in this approach: You have to train your algorithm a long time for even learning to press a button. The second limit is that you have to define the possible actions, that the agent might choose. And remember: Every single thing your system has to learn will affect your training time.

You can dive into this topic by searching for the openai gym, which will give you a good overview about what is done right now in order to build AIs which play games. A good book about this topic is Reinforcement Learning - An Introduction by Barto and Sutton.


what is sentence embeding and how to do sentence embedding for a sentence and how to use word embedding to create a sentence embedding? What is sentence embedding? How would you do sentence embedding for a sentence like: "How old are you?" How do you use word embedding to create a sentence embedding?

It could be just a simple average of word vectors; words that are a member of the sentence. However, there are other methods that could work better. Please see this article:

In the paper Unsupervised Learning of Sentence Embeddings using Compositional N-Gram Features, a new model for sentence embeddings called Sent2Vec is introduced. Sent2Vec presents a simple but efficient unsupervised objective to train distributed representations of sentences. It can be thought of as an extension of FastText and word2vec (CBOW) to sentences. The sentence embedding is defined as the average of the source word embeddings of its constituent words. This model is furthermore augmented by also learning source embeddings for not only unigrams but also n-grams of words present in each sentence, and averaging the n-gram embeddings along with the words.


Training LSTM Recurrent Network in TensorFlow I have trained RNN's before "by hand" using basic tools like Numpy or BLAS, but I am having trouble getting a simple RNN to converge in TF. Full Code

I tried standard things like adjusting the learning rate, the momentum, and adding noise to the gradient, but I am concerned that I don't understand what TF is doing underneath.

In a couple tutorials, it appeared that in TF you train the networks like Echo State Networks where the hidden state is evolved on random interconnections to train up a reservoir, then you use a linear (or non-linear) transformation to map the hidden state onto your labels.

I examined the gradients that Tensorflow uses to train, and it appears that it is training the internal state transition matrices as one would expect if using "Back-Propagation Through Time", which I can't tell if it is trying to use.

Can you help me understand what I am doing/understanding incorrectly with regards to training RNN's in TF?

cell_layer = tf.contrib.rnn.BasicLSTMCell(state_size, state_is_tuple=True)
cell = tf.contrib.rnn.MultiRNNCell([cell_layer]*1)
x, y_ = tf.placeholder(tf.float32, shape=(batch_size,1)), tf.placeholder(tf.float32, shape=(batch_size,1))
outputs, states = tf.nn.dynamic_rnn(cell, tf.expand_dims(x, -1), dtype=tf.float32)# init_state)

W = tf.Variable(tf.random_uniform((state_size, 1), -1.0, 1.0))
b = tf.Variable(tf.random_uniform((1, 1), -1.0, 1.0))
y = tf.matmul(tf.reshape(outputs, (-1, state_size)), W) + b

loss = tf.reduce_mean(tf.square(y - y_))
opt = tf.train.MomentumOptimizer(1e-3, 0.9)
grad = [(g + tf.random_uniform(v.shape, -1.0, 1.0) * tf.reduce_mean(tf.abs(v))*noise, v)
        for g, v in opt.compute_gradients(loss)]
train = opt.apply_gradients(grad)

EDIT: Loss plot

I am less concerned about the loss than I am about the fact that there appear to not be any dynamics in between iterations. It appears to converge to the average. Below are some of the later losses and a part of the sequence to be predicted.

Loss:  11.9126
Actual:  bcdefghijklabcdefghijklabcdefghijklabcde
Pred:    ffffffffffffffffffffffffffffffffffffffff

Loss:  11.9092
Actual:  bcdefghijklabcdefghijklabcdefghijklabcde
Pred:    ffffffffffffffffffffffffffffffffffffffff

Loss:  11.918
Actual:  bcdefghijklabcdefghijklabcdefghijklabcde
Pred:    ffffffffffffffffffffffffffffffffffffffff

Loss:  11.9274
Actual:  bcdefghijklabcdefghijklabcdefghijklabcde
Pred:    ffffffffffffffffffffffffffffffffffffffff

Note

>>> s = 'bcdefghijklabcdefghijklabcdefghijklabcde'
>>> n = map(ord, s)
>>> chr(sum(n)/len(n))
'f'

I removed the gradient noise (which did not seem to help, at least as you did it) and replaced your momentum optimizer with Adam using the default hyperparameters and it just worked. After 10,000 epochs I got a loss of ~8 with bcdefghijklabcdefghijklabcdefghijklabcde (actual) vs ccdeeffgghhiccdeeffgghhiccdeeffgghhiccde (predicted).

The moral is that optimizing neural networks is still an art.

p.s. It seems weird to use a regression loss with a classification problem.


How is the fit function in SimpleImputer working to find the mean in the Salary column as well when just the Age column is given as its argument? The only argument inside the fit function of SimpleImputer is: 'Age'. Yet the returned output worked on the 'Salary' column as well. That is what I am unable to understand.
Here is my code (considering all the necessary libraries imported):
from sklearn.impute import SimpleImputer
imputer = SimpleImputer(missing_values = np.nan)
imputer = imputer.fit(df[['Age']])
X[:, 1:3] = imputer.fit_transform(X[:, 1:3])
print(X)

Dataset:
   Country   Age   Salary Purchased
0   France  44.0  72000.0        No
1    Spain  27.0  48000.0       Yes
2  Germany  30.0  54000.0        No
3    Spain  38.0  61000.0        No
4  Germany  40.0      NaN       Yes
5   France  35.0  58000.0       Yes
6    Spain   NaN  52000.0        No
7   France  48.0  79000.0       Yes
8  Germany  50.0  83000.0        No
9   France  37.0  67000.0       Yes

Output:
[['France' 44.0 72000.0]
 ['Spain' 27.0 48000.0]
 ['Germany' 30.0 54000.0]
 ['Spain' 38.0 61000.0]
 ['Germany' 40.0 63777.77777777778]
 ['France' 35.0 58000.0]
 ['Spain' 38.77777777777778 52000.0]
 ['France' 48.0 79000.0]
 ['Germany' 50.0 83000.0]
 ['France' 37.0 67000.0]]

imputer = imputer.fit(df[['Age']])
X[:, 1:3] = imputer.fit_transform(X[:, 1:3])

You are again applying "fit" with fit_trasnform. Try with "transform" only
You will get a value error

ValueError: X has 3 features per sample, expected 2


What to do after selecting a model with cross-validation? I have been building a neural network for classification.

To select my best model. I have been using 10-Fold cross validation. and selected the network that gives the highest mean accuracy.

Now that I have selected the best model, I want to use all the data I have to train this model because the amount of data I have is limited (I will merge training, dev and test data).

My issue is that, when training with all the data, I don't know when to stop training. Training loss is not an indicator for sure. Usually, I have a development set that I use to monitor training. When the training loss does not improve anymore, I stop training.

Any suggestions on how to supervise a model with only training data? In other words, how to tell when the network needs to stop?

When 'overtraining' is not a problem (as in it will not diverge if you use more time), just use all your data and the empirically found optimal hyper parameters. In case of neural networks this is not the case (although in my experience, a lot of architectures converge to a specific test error, and take a long time to end up diverging again). I see a few options that you could try:

Most obvious one is keeping a (small) validation set around to use as indicator for early stopping (don't think of this as throwing away data, you still use it to train your network better)
Use same weight initalization as one of your folds and run for the same amount of epochs, same initialization should make convergence rate more similar than new random initialization
Keep all the cross validation models and use them in an ensemble instead of retraining the full model


How to apply Normalisation using the MinMaxScaler() to all Columns, but Exclude the Categorical? Below, I have the following datatset:
sample_df.head(2)
ID     S_LENGTH     S_WIDTH     P_LENGTH     P_WIDTH     SPECIES
-------------------------------------------------------------------
1      3.5          2.5          5.6         1.7        VIRGINICA
2      4.5          5.6          3.4         8.7         SETOSA

Therefore, how to I apply normalisation to this dataset using the following code below to all my columns, excluding the ID and SPECIES columns?
I basically want to use the preprocessing.MinMaxScaler() to apply normalisation, so that all the features are in a range of 0 and 1.
This is the code I am using...
min_max = preprocessing.MinMaxScaler()
min_max.fit_transform(sample_df)

...but when I execute it, I get this error:
ValueError: could not convert string to float: 'SETOSA'

Alternatively, if I do this...
min_max = preprocessing.MinMaxScaler()
min_max.fit_transform(sample_df[['S_LENGTH', 'S_WIDTH']])

sample_df.head(2)

...I get this error:
AttributeError: 'numpy.ndarray' object has no attribute 'sample'

Any help on how to accomplish what I want to do is much appreciated!

If you want to apply the result of fit_transform, you must assign to your columns.
columns = ['S_LENGTH', 'S_WIDTH', 'P_LENGTH', 'P_WIDTH']

min_max = preprocessing.MinMaxScaler()
df[columns] = min_max.fit_transform(df[columns])
df.head()

Output
   ID  S_LENGTH  S_WIDTH  P_LENGTH  P_WIDTH    SPECIES
0   1       0.0      0.0       1.0      0.0  VIRGINICA
1   2       1.0      1.0       0.0      1.0     SETOSA


how to calculate recency(RFM) from a given date attribute in R custid  tot_prod       tot_rev          Last_tran   total_trans

1   1002          13       1465.25         11/23/2011       4
2   1003          2         353.25          1/1/2011        1
3   1004          12        2466.5          11/25/2011      12

Above is the Retail dataset of customer. I need to calculate recency(a new attribute when the customer is last seen) from last transaction(last_tran)  in terms of month.i.e. he purchased in last 3 months or last 6 months or may be one year in R. please suggest.

Try this

as.POSIXct(Sys.Date(), format= "%Y-%m-%d", tz="UCT") - as.POSIXct("11/25/2011", format = "%m/%d/%Y",tz="UCT")

More vectorized operation in your situation

as.POSIXct(Sys.Date(), format= "%Y-%m-%d", tz="UCT") - as.POSIXct(Last_tran, format = "%m/%d/%Y",tz="UCT")


Why cant I overfit this dataset with my neural network? I have read that given a model is complex enough and I train for enough epochs, my model should at some point overfit the dataset. However I implemented a simple neural network in keras and my validation set loss seems to never go up:
import tensorflow as tf
from tensorflow import keras
import numpy as np
import random
from sklearn import datasets, preprocessing
import matplotlib.pyplot as plt

# import and scale
dataset = datasets.load_boston()
X = dataset.data
y = dataset.target
X = preprocessing.scale(X)
y = y.reshape((y.shape[0], 1))

# shuffle
shuffle_indices = list(range(X.shape[0]))
random.shuffle(shuffle_indices)
X = X[shuffle_indices]
y = y[shuffle_indices]

# tain-validation split
X_train, y_train  = X[:int(X.shape[0] * 0.7)], y[:int(X.shape[0] * 0.7)]
X_val, y_val = X[int(X.shape[0] * 0.7):], y[int(X.shape[0] * 0.7):]

# define and fit model
model = keras.Sequential([keras.layers.Dense(X.shape[1], use_bias=True, activation="sigmoid"),
                              keras.layers.Dense(128, use_bias=True, activation="sigmoid"),
                              keras.layers.Dense(128, use_bias=True, activation="sigmoid"),
                              keras.layers.Dense(128, use_bias=True, activation="sigmoid"),
                              keras.layers.Dense(128, use_bias=True, activation="sigmoid"),
                              keras.layers.Dense(128, use_bias=True, activation="sigmoid"),
                              keras.layers.Dense(128, use_bias=True, activation="sigmoid"),
                              keras.layers.Dense(y.shape[1])
                         ])
model.compile(optimizer=tf.keras.optimizers.SGD(
    learning_rate=0.0001
), loss='MeanSquaredError')

model.fit(X_train, y_train, validation_data=(X_val, y_val), epochs=500, verbose=1)

# plot data
plt.plot(range(1, len(model.history.history['loss']) + 1), model.history.history['loss'], label='Train Set Cost')
plt.plot(range(1, len(model.history.history['val_loss']) + 1), model.history.history['val_loss'], label='Validation Set Cost')
plt.xlabel("epoch")
plt.ylabel("loss")
plt.legend()
plt.show()

The model is a simple dense neural network with Mean Squared Error as its loss function and gradient descent as it's optimizer. I tried to make the network deeper, but the validation loss only keeps decreasing until it stops at one point.

It is usually good to start with a small model because you can then evaluate the contribution of adding layers, etc.
Also, Boston dataset is a popular dataset so there are several tutorials showing good neural network architectures, like this one.
Concerning your model, here are some notes.

The use of sigmoid activation is likely to worsen results, since sigmoid function compresses values between 0 and 1, while you are trying to predict outputs between 5 and 50.
Instead of sigmoid, you can use ReLU activation, which has better convergence properties for inner layers
You can standardize your target data to reduce the variance of your data and control the mean, this usually improve a lot regression models. In this case, a sigmoid activation on your final layer would be a good choice