Not enough memory for operations with Pandas Wes McKinney, the author of Pandas, writes in his blog that
"... my rule of thumb for pandas is that you should have 5 to 10 times as much RAM as the size of your dataset. So if you have a 10 GB dataset, you should really have about 64, preferably 128 GB of RAM if you want to avoid memory management problems."
I frequently use Pandas with datasets not much smaller than my RAM (16GB). So I wonder, what are   some practical implications of these "memory management problems"? Could anyone provide more insights into this? Does it mean it will store data in virtual memory on disk and therefore be very slow?

When Pandas finds it's maximum RAM limit it will freeze and kill the process, so there is no performance degradation, just a SIGKILL signal that stops the process completely. Speed of processing has more to do with the CPU and RAM speed i.e. DDR3 vs DDR4, latency, SSD vd HDD among other things.
Pandas has a strict memory limit but there are options other than just increasing RAM if you need to process large datasets.
1.- Dask
Here are certain limitations in dask.

Dask cannot parallelize within individual tasks.
As a distributed-computing framework, dask enables remote execution of arbitrary code. So dask workers should be hosted within trusted network only.

A Dask tutorial:
https://medium.com/swlh/parallel-processing-in-python-using-dask-a9a01739902a
2.- Jax
3.- Feather Format
Language agnostic so it's usable in R and Python, can reduce the memory footprint of storage in general.
4.- Decreasing memory consumption natively in Pandas
Reducing the number of bits of memory to encode a column help specially when you use tree-based algorithms to process the data later. This is a script popularized by Kaggle.
import pandas as pd

def reduce_mem_usage(df, verbose=True):
   numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']
   start_mem = df.memory_usage().sum() / 1024**2
   for col in df.columns:
       col_type = df[col].dtypes
       if col_type in numerics:
           c_min = df[col].min()
           c_max = df[col].max()
           if str(col_type)[:3] == 'int':
               if c_min > np.iinfo(np.int8).min and c_max <    np.iinfo(np.int8).max:
                   df[col] = df[col].astype(np.int8)
               elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:
                   df[col] = df[col].astype(np.int16)
               elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:
                   df[col] = df[col].astype(np.int32)
               elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:
                   df[col] = df[col].astype(np.int64)
           else:
               if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:
                   df[col] = df[col].astype(np.float16)
               elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:
                   df[col] = df[col].astype(np.float32)
               else:
                   df[col] = df[col].astype(np.float64)

   end_mem = df.memory_usage().sum() / 1024**2
   print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))
   print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))

   return df


Ngram based Langauge Models learned using an Encoder-Decoder Model I have been going through a Ngram based Langauge Model learned using an Encoder-Decoder Model for Email smart compose.

The program output only 1 prediction for given input.
I want to know how to get multiple predictions out of the same.

Here is the link to the notebook: https://nbviewer.jupyter.org/github/PrithivirajDamodaran/NLP-Experiments/blob/master/Gmail_style_smart_compose_with_char_ngram_based_language_model.ipynb

Here, for input sequence : "hi there" the predicted sequence is: ", how are you today?"
But it maybe possible that I have multiple sentences starting with "hi there" in my training dataset. So how do I get all of those?

For your case, I feel that skip-gram model will be fit for your business problem. Where u can predict a sequence of a word rather than one word. I recommend you to train the skiagram model on your corpus.


Python - Predicting data based on multidimensional array with Keras I've a list of data which is so called 3D array. Each of 10350 rows contains a 2D matrix with size of 150x16 (elements are float) (x_train). Corresponding training data for this huge array a linear array with size of 10350 integer which can be either 0 or 1 (y_train).

I used different methods to estimate the test_data. None of them worked with such type of array.

from matplotlib import pyplot as plt

from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
from sklearn.preprocessing import MinMaxScaler

min_max_scaler = MinMaxScaler()

train_data_set = np.load("files/samples/A08_Block_sample.npy")
test_data_set = np.load("files/samples/A10_BLOCK_sample.npy")

x_train = []
y_train = []

for i in range(len(train_data_set)):
   x_train.append(train_data_set[i].arr_)
   y_train.append(train_data_set[i].flag_)

   num_units = 4
   activation_function = 'sigmoid'
   optimizer = 'adam'
   loss_function = 'mean_squared_error'
   batch_size = 5
   num_epochs = 100

   regressor = Sequential()

   regressor.add(LSTM(units=num_units, activation=activation_function, input_shape=(10403, 16, 150)))

   regressor.add(Dense(units=1))

   regressor.compile(optimizer=optimizer, loss=loss_function)

   regressor.fit(x_train, y_train, batch_size=batch_size, epochs=num_epochs)

   inputs_x = []
   inputs_y = []
   for i in range(len(test_data_set)):
      inputs_x.append(test_data_set[i].arr_)
      inputs_x.append(test_data_set[i].flag_)

   predicted_price = regressor.predict(inputs_x)

The library that I'm using is Keras. I'm new at data science, any suggestion will be welcome for me.

Change first layer with

regressor.add(LSTM(units=num_units, input_shape=(x_train.shape[1],x_train.shape[2])))

You need to have 2 numbers as input shape.

Hope will help you.


Impossible to increase model accuracy I'm building binary classification models on my company's dataset. The problem I'm having is that I haven't been able to increase the accuracy of my models. I have trained, tuned, cross validated models such as logistic regression, knn, neural networks and they all have results with the same accuracy. I feel that I have tried everything. The dataset is about 100 data points with 21 features. I'm aware of the curse of dimensionality so I have tried using only subsets of the features but the accuracy is the same no matter which subsets of features I use.
Is it possible that with this dataset, no further improvements can be done? Should I just tell my boss that this is the maximum accuracy possible given the amount of data we have?

100 data points is a very very low number of points. I am quite surprised that the algo managed to learn something meaningful (well unless you have very obvious relationships between the features and the target). I'll suggest you look at what the logistic regression learned exactly (what coefficients are deemed significant). There is an important risk that you are overfitting on your very very small dataset. And you can't really catch that so your are just left with checking if the rationale of your model make sense.
Yes, you should probably tell your boss that this is not enough data points.


When is a Neural networked considered to be overfitted? When is a neural network considers to be overfitted to a specific task.

I read somewhere that if the number of input parameter equate the number of hyper parameter, that would be an indication of overfitting..

I use CV for training.
first i split the data into train/test => 90/10.
and within the training set ,i do another 90/10 split, which are those data I use for the training.

I then use the model for processing the complete dataset.

CV would state how much i am overfitting (I am getting minor error, deviating with 2%).. But since i am using most of the data for training? am I then creating an overfitted neural network for this dataset.

I am currently having 230043 training samples,  107145 total params, and validates on 25561 samples.  am I overfitting?

A model is said to have overfitted when it performs awesomely on the training set (error's low), but when tested on test set, it fails badly. This might be due to bad sampling distribution over our training and tes divisions, so that model might not have seen this kind of data or due to using many irrelevant features in the samples than we were supposed to etc.

You cannot really say that the cat is dead until you open the box. So until you test, you would never know.

  I read somewhere that if the number of input parameter equate the number of hyper parameter, that would be an indication of overfitting

This statement can be explained in general, not only for neural nets. Assume you know the relationship is linear for your data, but you got some error and you are not happy with this, then say you add an extra term say $ax^2$ to this equation and the model now has an extra parameter to tune and this makes our line into curve. For a neural net we basically start by random weights from a distribution and then we keep reducing them until we feel like our training error is cool.

Assuming that you are using a multi-layer perceptron (fully connected), pruning is one of the techniques to remove weights from the model which might not be so contributing still making sure your model still behaves well with lesser number of parameters. SVD on this weight matrix can be one of the useful techniques to do this. But how many weights to keep depends on how many singular values you keep for reconstructing the matrix. I heard that early stopping can reduce overfitting, but I had never really done this, so can't really give my opinions about this.

Hope this helps.


Machine Learning Best Practices for Big Dataset I am about to graduate from my Master and had learnt about machine learning as well as performed research projects with it. I wonder about the best practices in the industry when performing machine learning tasks with Big Datasets (like 100s GB or TB). Appreciate if fellow data scientists can share their experience. Here are my questions:

Obviously, very large datasets take longer time to train (can be days or weeks). Many times we need to train various models (SVM, Neural Network, etc.) to compare and find better performance model. I suspect, in industry projects, we want the results as quick as possible but produce the best performance. Are there any tips for reducing the training & testing time? If you recommend subsetting the dataset, I will be interested to learn how best to subset the dataset to cover all or majority of scenarios from the dataset.
We know that performing cross validation is better as it may reduce over-fitting. However, cross validation also takes time to train and the model trained with cross validation may not be implemented straight (speaking from python sklearn experience: I need to train the model with dataset again after cross validation testing for it to be implemented). Do you normally do cross validation in your big data projects or getting by with the train-test split?

Appreciate the feedback.

I'll list some practices I've found useful, hope this helps:

Irrespective of whether the data is huge or not, cross validation is a must when building any model. If this takes more time than an end consumer is willing to wait, you may need to reset their expectations, or get faster hardware/software to build the model; but do not skip cross validation. Plotting learning curves and cross-validation are effective steps to help guide us so we recognize and correct mistakes earlier in the process. I've experienced instances when a simple train-test set does not reveal any problems until I run cross-fold validations and find a large variance in the performance of the algorithm on different folds.
Before sizing up a dataset, eliminate the records with missing values of key variables and outliers, columns of highly correlated variables, and near zero variance variables. This will give you a much better estimate of the real usable dataset. Sometimes you may end up with only a fraction of the available dataset that can actually be used to build a model.
When sizing up a dataset for building a model, it is easier to estimate the computing resources if you enumerate the dataset in rows and columns and memory size of the final numeric matrix. Since every machine learning algorithm is ultimately going to convert the dataset into a numeric matrix, enumerating the dataset size in terms of GBs/TBs of raw input data (which may be mostly strings/textual nominal variables/etc.) is often misleading and the dataset may appear to be more daunting and gigantic to work with than it is.
Once you know (or estimate) the final usable size of your dataset, check if you have a suitable machine to be able to load that into memory and train the model. If your dataset size is smaller than memory available/usable by the software, then you need not worry about the size any longer.
If the dataset size is larger than the memory available to train a model, then you could try these approaches (starting from the simplest ones first):

Use a machine with more memory: If you're using a cloud service provider then the simplest approach could be just to provision more memory and continue building the model as usual. For physical machines, try to procure additional RAM, its price continues to reduce and if your dataset is going to remain this big or grow bigger over time, then it is a good investment.
Add nodes to the cluster: For Hadoop and Spark based cluster computing deployments, training on a larger data-set is as easy as adding more machines to the cluster.
Quite often classification tasks require training on data with highly imbalanced classes, the ratio of positive to negative classes could sometimes be as large as 1:1000 or more. A straightforward method to improve accuracy in these cases is to either over-sample the minority class or under-sample the majority class, or do both together. If you have a large dataset, under-sampling the majority class is a very good option which will improve your algorithm's accuracy as well as reduce training time.
Build an ensemble: Split the dataset randomly and train several base learners on each part, then combine these to get the final prediction. This would most effectively make use of the large dataset and produce a more accurate model. But you need to spend more time to carefully build the ensemble and keep clear of the usual pitfalls of ensemble building.
If you're using an ensemble, train many single-thread models in parallel. Almost all ML software provide features to train multiple models on different cores or separate nodes altogether.
Evaluate multiple different algorithms on the time taken to train them for your specific dataset vs. their accuracy. While there is no universal answer, but I've found when using noisy data, SVMs take much longer time to train than carefully built ensemble of regularized regression models, but may be only slightly more accurate in performance; and a well built neural network may take a very long time to train as compared to a CART tree, but perform significantly more accurately that the tree.
To reduce time taken to build the model, try to automate as much of the process as you can. A few hours spent automating a complex error-prone manual task may save your team a hundred hours later in the project.
If available, use those algorithm implementations which use parallel processing, sparse matrices and cache aware computing, these reduce processing time significantly. For example, use xgboost instead of a single-core implementation of GBM.
If nothing else works, train the model on a smaller dataset; as Emre has suggested in his answer, use learning curves to fix the smallest sample size required for training the model, adding more training records than this size does not improve model accuracy noticeably. Here is a good article which explores this situation - largetrain.pdf


What is the significance of underflow during parameter update using stochastic gradient descent? Background

I am using scikit-learn's MLPRegressor to learn a model with the following arguments:

hidden_layer_sizes=(16, 16),
learning_rate_init=1e-2,
solver='sgd',
momentum=0.9,
activation='tanh',
learning_rate='constant'

I use partial_fit() to incrementally fit the model. Occasionally I get a FloatingPointError on this line in the MLPRegressor's code:

updates = [self.momentum * velocity - self.learning_rate * grad
                   for velocity, grad in zip(self.velocities, grads)]

>>> FloatingPointError: underflow encountered in multiply

Problem

I am getting an underflow error when the model is calculating how much to update parameters by multiplying learning rate, momentum, velocity, and error gradients.

Does not an underflow mean that the updates are too small? Does it imply the model has reached some local minimum? Should I take this as a hint that:

I have exhausted my data and training has ended,
I should up my learning rate to keep updates happening
Something else?

Underflow is happening because the result of that calculation is a number of smaller absolute value than your computer can actually represent in memory.
Underflow is not directly related to the performance of the model (e.g., learning rate or local minimum). Underflow is the interplay between computer software and hardware.
I suggest rescaling the data. For example, batch normalization often stops underflow errors.


Monte Carlo for non-episodic tasks In Sutton's textbook (Chapter 5) it says "To ensure that well-defined returns are available, here we define Monte Carlo methods only for episodic tasks". Can someone explain what exactly breaks down for non-episodic tasks for Monte Carlo methods in Reinforcement Learning?

Can someone explain what exactly breaks down for non-episodic tasks for Monte Carlo methods in Reinforcement Learning?

It is this first part of the sentence that you quote:

"To ensure that well-defined returns are available..."

In more detail, the return distribution in an episodic MDP can be defined as
$$G_t = \sum_{k=0}^{T-t-1} \gamma^k R_{t+k+1}$$
where $R_x$ is reward distribution at time step $x$, $\gamma$ is discount factor, and $T$ is the last time step of the episode.
From this you can see that you will need to collect all of $r_{t+1}$ to $r_T$ for an episode in order to calculate a single concrete return value $g_t$. You can only collect $r_T$ at the end of an episode.
In non-episodic tasks, there is no terminal time step $T$, and the return definition becomes:
$$G_t = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}$$
This cannot be resolved directly by collecting data for infinite time steps. Or in other words you can never sample an actual return value as it is defined. To get around this you could do one or more of the following:

You could use an artifical time horizon $h$, and define $T = t + h$. This will result in a biased sample. However, you can make the bias arbitrarily small relative to total return by making $h$ large. If you are using a discount factor, that may allow for smaller value of $h$.

You could use temporal difference (TD) learning to bootstrap return estimates without needing a full sample of return. This will result in a biased sample, but the bias due to bootstrapping should reduce as more samples are collected.

You could use TD with eligibilty traces and a high trace decay parameter $\lambda = 1$, which is a way of getting very similar learning behaviour to Monte Carlo, but still allowing learning to occur on each step in a continuing problem.

These options technically stop the approach being true Monte Carlo. The last two are described in the book under TD methods.


Moving from macbook (without GPU) to linux system with Titan V, only getting a 4x speedup, what am I doing wrong? I was prototyping a network architecture out on the macbook, and after finding something I was somewhat happy with, I wanted to test it out on a big data set on a system with a Titan V as the macbook was very slow for the bigger dataset (12 hours). I was expecting at least a 50x speed up per epoch over the CPU, if not more. Why might the speedup be only 4x (3 hours)?

Err it turned out to be some very tricky driver issues, and by just cleaning out my vm and starting over, was able to fix it.

This line of code in particular was invaluable - some of the other checks on "is tensorflow using the gpu" I found hovering around the web were not adequate to solving this:

import tensorflow as tf
with tf.device('/gpu:0'):
    a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
    b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
    c = tf.matmul(a, b)

with tf.Session() as sess:
    print (sess.run(c))

# Copyright 2015 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
"""A script for testing that TensorFlow is installed correctly on Windows.

The script will attempt to verify your TensorFlow installation, and print
suggestions for how to fix your installation.
"""

import ctypes
import imp
import sys

def main():
  try:
    import tensorflow as tf
    print("TensorFlow successfully installed.")
    if tf.test.is_built_with_cuda():
      print("The installed version of TensorFlow includes GPU support.")
    else:
      print("The installed version of TensorFlow does not include GPU support.")
    sys.exit(0)
  except ImportError:
    print("ERROR: Failed to import the TensorFlow module.")

  candidate_explanation = False

  python_version = sys.version_info.major, sys.version_info.minor
  print("\n- Python version is %d.%d." % python_version)
  if not (python_version == (3, 5) or python_version == (3, 6)):
    candidate_explanation = True
    print("- The official distribution of TensorFlow for Windows requires "
          "Python version 3.5 or 3.6.")

  try:
    _, pathname, _ = imp.find_module("tensorflow")
    print("\n- TensorFlow is installed at: %s" % pathname)
  except ImportError:
    candidate_explanation = False
    print("""
- No module named TensorFlow is installed in this Python environment. You may
  install it using the command `pip install tensorflow`.""")

  try:
    msvcp140 = ctypes.WinDLL("msvcp140.dll")
  except OSError:
    candidate_explanation = True
    print("""
- Could not load 'msvcp140.dll'. TensorFlow requires that this DLL be
  installed in a directory that is named in your %PATH% environment
  variable. You may install this DLL by downloading Microsoft Visual
  C++ 2015 Redistributable Update 3 from this URL:
  https://www.microsoft.com/en-us/download/details.aspx?id=53587""")

  try:
    cudart64_80 = ctypes.WinDLL("cudart64_80.dll")
  except OSError:
    candidate_explanation = True
    print("""
- Could not load 'cudart64_80.dll'. The GPU version of TensorFlow
  requires that this DLL be installed in a directory that is named in
  your %PATH% environment variable. Download and install CUDA 8.0 from
  this URL: https://developer.nvidia.com/cuda-toolkit""")

  try:
    nvcuda = ctypes.WinDLL("nvcuda.dll")
  except OSError:
    candidate_explanation = True
    print("""
- Could not load 'nvcuda.dll'. The GPU version of TensorFlow requires that
  this DLL be installed in a directory that is named in your %PATH%
  environment variable. Typically it is installed in 'C:\Windows\System32'.
  If it is not present, ensure that you have a CUDA-capable GPU with the
  correct driver installed.""")

  cudnn5_found = False
  try:
    cudnn5 = ctypes.WinDLL("cudnn64_5.dll")
    cudnn5_found = True
  except OSError:
    candidate_explanation = True
    print("""
- Could not load 'cudnn64_5.dll'. The GPU version of TensorFlow
  requires that this DLL be installed in a directory that is named in
  your %PATH% environment variable. Note that installing cuDNN is a
  separate step from installing CUDA, and it is often found in a
  different directory from the CUDA DLLs. You may install the
  necessary DLL by downloading cuDNN 5.1 from this URL:
  https://developer.nvidia.com/cudnn""")

  cudnn6_found = False
  try:
    cudnn = ctypes.WinDLL("cudnn64_6.dll")
    cudnn6_found = True
  except OSError:
    candidate_explanation = True

  if not cudnn5_found or not cudnn6_found:
    print()
    if not cudnn5_found and not cudnn6_found:
      print("- Could not find cuDNN.")
    elif not cudnn5_found:
      print("- Could not find cuDNN 5.1.")
    else:
      print("- Could not find cuDNN 6.")
      print("""
  The GPU version of TensorFlow requires that the correct cuDNN DLL be installed
  in a directory that is named in your %PATH% environment variable. Note that
  installing cuDNN is a separate step from installing CUDA, and it is often
  found in a different directory from the CUDA DLLs. The correct version of
  cuDNN depends on your version of TensorFlow:

  * TensorFlow 1.2.1 or earlier requires cuDNN 5.1. ('cudnn64_5.dll')
  * TensorFlow 1.3 or later requires cuDNN 6. ('cudnn64_6.dll')

  You may install the necessary DLL by downloading cuDNN from this URL:
  https://developer.nvidia.com/cudnn""")

  if not candidate_explanation:
    print("""
- All required DLLs appear to be present. Please open an issue on the
  TensorFlow GitHub page: https://github.com/tensorflow/tensorflow/issues""")

  sys.exit(-1)

if __name__ == "__main__":
  main()

Copy paste, run this code. Will tell you if you have tensorflow GPU support.
For example, on my MacBook Pro it says

TensorFlow successfully installed.
The installed version of TensorFlow does not include GPU support.


How does Naive Bayes classifier work for continuous variables? I know that for categorical features we just calculate the prior and likelihood probability assuming conditional independence between the features.

How does it work for continuous variables? How can we calculate likelihood probability for continuous variable?

The difference boils down to "how we define $P(x_i|C_k)$?", where $x_i$ is a single feature, and $C_k$ is a class from a total of $K$ classes.

Discrete

In discrete case,  $P(x_i|C_k)$ is represented by a table as follows:

x_i   P(x_i|C_k)
a     0.5
b     0.2
c     0.3

We have one of these tables for each feature-class pair $(i, k)$.

Lets denote $i$-th feature of data point $n$ as $x_{n,i}$. Each row of this table can be estimated using

$$\hat{P}(x_i=a|C_k) = \frac{\sum_{n:n \in C_k}{\mathbb{1}_{x_{n,i}=a}}}{N_k}$$

which divides the number of samples that have i-th feature equal to $a$ by total number of samples in class $C_k$ (of course from the training set).

Also, check out pseudocount that avoids zero estimations.

Continuous

In continuous case, we either discretize the continuous interval of $x_i$ into bins $\{b_1,..,b_m\}$ and proceed the same as discrete case, or we assume a function like Gaussian (or any other one), as follows:

$$P(x_i|C_k)=\frac{1}{\sqrt{2\pi}\sigma_{i,k}}e^{-(x_i-\mu_{i,k})^2/2\sigma_{i,k}^2}$$

This way, for each feature-class pair $(i, k)$, $P(x_i|C_k)$ is represented with two parameters $\{\mu_{i,k}, \sigma_{i, k}\}$ instead of a table in discrete case. The estimation of the parameters is the same as fitting a Gaussian distribution to one dimensional data, that is:

$$\hat{\mu}_{i,k} = \frac{\sum_{n:n \in C_k}{x_{n,i}}}{N_k}, \hat{\sigma}^2_{i, k} = \frac{\sum_{n:n \in C_k}{(x_{n,i} - \hat{\mu}_{i,k})^2}}{N_k-1}$$

Instead of Gaussian, we can opt for a more complex function, even a neural network. In that case, we should look for a technique to fit the function to data just like what we did with Gaussian.

The rest is independent of feature type

Representing $P(C_k)$ is the same for both discrete and continuous cases and is estimated as $\hat{P}(C_k)=N_k/N$.

Finally, the classifier is $$C(x_n) = \underset{k \in \{1,..,K\}}{\mbox{argmax }}\hat{P}(C_k)\prod_{i}\hat{P}(x_i=x_{n,i}|C_k)$$

Or equivalently using log-probabilities,

$$C(x_n) = \underset{k \in \{1,..,K\}}{\mbox{argmax }}\mbox{log}\hat{P}(C_k)+\sum_{i}\mbox{log}\hat{P}(x_i=x_{n,i}|C_k)$$


Input and output layer activation functions of neurons in Orange What are the activation functions of the neurons in the input and output layer of a neural network model from the Orange machine learning application?

The activation functions are used in hidden layers and output layer. The output layer will usually have sigmoid or softmax activation while hidden layers usually use ReLU. ReLU is available in Orange at least (there is a dropdown which you can explore but ReLU works best mostly, read this). Also, check this link for looking at neural networks in Orange.


Features for blink detection in real-time single channel EEG I am looking to detect blink events in real-time single channel EEG. Classification of a moving window of samples to determine whether a blink artifact exists requires feature extraction (except when using deep learning, I am not experienced enough for this). What features would be useful to extract from a window of approx. 50-200 samples of time series data for detecting a blink event. The blink event can be easily seen in the below picture:

I would try a combination of the following:

A rolling-window variance to detect unusually large deflections
Bollinger bands to detect the spikes
Autocorrelation to detect the 'up down up' pattern


a data point - too little to spend resources during data preparation? I am learning analytics online and have some quick questions.

Usually when we do analysis, why is that we usually ignore the items/data points that are less frequent?

Let's say for ex: we have is drug frequency data and no of patients who consumed that drug in a hospital. As an example, the data looks like as shown below but in real-time, I might even have millions of records

From the above screenshot, we can know that whatever analysis and insights we come up with the above data (including few more columns of data which aren't shown here), we will definitely not consider Drug D.

Meaning we can't base our conclusion/insights that we derive from our data based on Drug D because only 2 out of 5000 patients have it which is less than 0.05% of our data.

Through it seems to intuitively make sense because 0.05% is very less to have any impact on output.

Now my question is what about Drug G. It occurs 1.14% times in our data?

How do I know it's okay to believe that Drug D - 0.05% is very less to have any impact on output and can be ignored whereas Drug G - 1.14% has to be retained?

I am not sure whether my English skills helped you understand what I am trying to convey.

EDIT - UPDATED (Apologies if my question wasn't clear earlier)

What I am trying to do is (not an ML task but a Data Preparation task), manually map the drug names to the terms available in the dictionary (Data Preparation task). As you can see in the screenshot, Drug A is mapped to ABCDE A. Similarly, I have to manually map for all 50K drugs. However, my question is given below

a) I can't spend resources (money/people) to manually (as it cannot be automated) go through all the 50K drugs and map it to dict terms because no one is interested to do this job. Whoever is interested, is not willing and it would be impossible to do all the 50K drugs and it would incur so much money to pay them. So, I have to make sure that manual reviewers focus on important (high frequent) terms first and it's even okay to ignore DRUG D or DRUG G because they contribute very little value to the data (considering the full dataset of million records)? Question is mainly on decision making based on systematic approach/mathematical approach rather than my judgement/visual inspection/subjective..

b) Hence now, I am trying to know whether there is any objective/systematic/mathematical approach that can tell me, we can ignore all drugs below a certain N% etc... Because I can't just say that through visual inspection I feel Drug G and Drug D can be ignored. If you are gonna suggest me Statistical significance test, can you please guide me on how can I set this as a problem? Because I usually see, it is used in hypothesis testing. Can I kindly request you to guide me on this?

Practically everything related to statistics (including Machine Learning) has to do with studying chance, i.e. trying to determine to what extent an observation is due to chance or not.

For example one might want to know whether a drug actually helps with a particular disease or not. If we observe that one patient improves after taking the drug, there's not enough evidence to conclude since many other factors could have caused the improvement. This is why one needs a very strict protocol in order to obtain a statistically meaningful observation (two groups of patients, placebo etc.).
A reasonably high number of observations is needed, otherwise it's impossible to distinguish the effect of "chance" (any other factor) and the real effect of the drug.

  a) Is there any systematic/mathematical/theoretical approach that can tell me anything less than N% is too little to impact/influence the output?

The standard method for knowing whether an observation is due to chance or not is to use the appropriate statistical significance test. There are many of them and they depend on what exactly is being tested.

  b) How do you decide which items are too little to impact output. Do you go with your judgement which is a subjective approach?

In ML it's common to take a more experimental approach, for example trying with/without an observation or feature and then evaluate which versions works better. Of course it helps to have an intuition of what is more likely to work. In general including extremely rare observations is a bad idea because it's likely to cause overfit, i.e. when the model "learns" something which is actually due to chance.

[added following OP's update]

In this case this is a resource allocation problem, I don't think  statistical significance is relevant here. Assuming that you want to optimize the use of manual labor based on how often a drug is used, i.e. the only thing to maximize is the sum of the frequencies of the drugs being labelled, then it's simple: rank all the drugs by their frequency in descending order, then proceed with manual annotation following this order. This way you're sure that the drugs which account for more patients are done first, so whenever manual annotation stops the largest possible amount has been labelled.


Unstable accuracy of CNN - When should I stop training? I'm using caffenet for fine-tuning.
I'm doing cross validation (15 vs all) with a very small data set of about 250 images. I'm testing every 10 iterations (~2 epochs). My batch size is 50. With some sets I'm getting very unstable accuracy - Can jump from 70% to 90% and back to 70% and back and fourth. My question is: Let's say I hit 90% accuracy after 40 iterations (~8 epochs) - Does this mean that the net had reached an optimal state or could it be that it just had a lucky guess on the validation set? My final question is: Should I stop training and save the net? Thanks.

If accuracy regresses something is wrong in either the network, or (more likely here) the meta-parameters (probably learning rate.)

It can be difficult to tell when a model converges. I'd recommend looking at diagnostic graphs (typically training loss, training/validation accuracy, and ratio of weights:updates) over epochs. Typically convergence is considered when loss and accuracy level out and show diminishing returns beyond some threshold (your tolerance for 1.0e-x% improvements.) So, stop training/validation when it's improving less than what you care about.


Do recommendation systems necessarily use machine learning algorithms? I am studying about evaluation of both recommendation systems and machine learning algorithms in recent times, trying to define a scope for my masters research. After some reading time I'm starting to understand several concepts, but one thing was not clear to me:

Do recommendation systems necessarily use machine learning algorithms?

I mean, I know these two can be used combined, but in most of the papers I read about recommender systems evaluation, they do not even mention anything about Machine Learning.

Also, if you can suggest some papers that I can read, I would be very grateful

There's nothing about a recommendation system that absolutely necessitates some kind of machine learning. Indeed, I've seen decision systems in use that were essentially just someone's idea about what the customer's preferences ought to be.

A recommender can be based on anything from a few ad-hoc 'common sense' rules, to a logistic regression someone did on some data a few years ago and whose parameters are hardcoded into the system, to a complicated ensemble of machine-learning algorithms that are regularly and constantly trained on new data.

The use of machine learning for recommender systems is partly driven by necessity, partly by fad (at least from what I have seen). If a simple recommender works well, and accurately predicts what the user wants, there's no need for a machine to learn anything. If there's a huge amount of data, hiding some very deep relationships that humans are unable to pick out, that's where machine learning becomes useful.


Tweedie Loss for Keras We are currently using XGBoost model with Tweedie loss for solving a regression problem which works very good, now I wanted to move our model to Keras and experience with neural networks, do anybody know how I can implement Tweedie loss for Keras? I only care about the instance when p=1.5 which gives us the best result in XGBoost.

Thanks

I end up implementing something like this:

def tweedieloss(y_true, y_pred):
    p=1.5
    dev = 2 * (K.pow(y_true, 2-p)/((1-p) * (2-p)) -
                   y_true * K.pow(y_pred, 1-p)/(1-p) +
                   K.pow(y_pred, 2-p)/(2-p))
    return K.mean(dev)

which I don't know how right it is, for now it seems to be working.


is learning rate schedule a hyperparameter? I believe term ‘learning rate schedule’ is a certain solution for tuning the learning rate. But at the same time, every parameter evaluating the parameter itself can be called a hyperparameter. So can the learning rate schedule be called a hyperparameter?

Hyper-parameters are essentially parameters that you have to tune manually and aren't learned.

The schedule can be considered a hyper-parameter, as well as all the scheduler's parameters (patience, decay, etc.) and even the decision to use or not a scheduler can be considered a hyperparameter.


List of samples that each tree in a random forest is trained on in Scikit-Learn In Scikit-learn's random forest, you can set bootstrap=True and each tree would select a subset of samples to train on. Is there a way to see which samples are used in each tree?
I went through the documentation about the tree estimators and all the attributes of the trees that are made available by Scikit-learn, but none of them seems to provide what I'm looking for.

I don't think it is possible to get it directly but we may utilize the random seed.

random_stateint, RandomState instance or None, default=None
Controls both the randomness of the bootstrapping of the samples used when building trees (if bootstrap=True)

This is from the RF Github code
def _generate_sample_indices(random_state, n_samples, n_samples_bootstrap):
    """
    Private function used to _parallel_build_trees function."""

    random_instance = check_random_state(random_state)
    sample_indices = random_instance.randint(0, n_samples, n_samples_bootstrap)

    return sample_indices

So, we can get these with a custom code if we fixed the seed above e.g. for 2 Tree,
import pandas as pd, numpy as np
num = 20 # max index
np.random.seed(0) # Fix the seed

sample_1 = np.random.randint(0,num,(1,num))
oob_1 = [elem for elem in np.arange(num) if elem not in sample_1 ]
sample_2 = np.random.randint(0,num,(1,num))
oob_2 = [elem for elem in np.arange(num) if elem not in sample_2 ]

Please verify it with a custom code. I have not verified it.


Minimum Neurons in Neural Network I use a brute-force mechanism to determine optimal hidden layers/neurons by incrementing the layers/neurons by 1 up to some maximums and then picking the optimal counts from the best performing model.  My question here is about the starting point for the hidden neuron count of this "brute-force" process.  What are some good mechanisms for determining what neuron count to start at such that I am not likely to miss some minimum optimal count?

For example:

Say I have 8 inputs that are NOT linearly related.  If I take (number of inputs: 8) + 1 (as some post suggest) for my starting point of incrementing the number of neurons within the layers, would I likely be missing the optimal neurons count as it may be less than 9 neurons?

The brute-force mechanism I am using sets a max number of neurons per hidden layer, so if I have 8 inputs, and I started with a range of 9 to 30 neurons per hidden layer, I would have 9 neurons in the first hidden layer which would increment to 30 and then roll back to 9 when starting to test with a second hidden layer:

Cycle 1
Hidden Layers = 1
Hidden Neurons = 9
...
Hidden Layers = 1
Hidden Neurons = 30

Cycle 2
Hidden Layers = 2
Hidden Neurons = 9
...
Hidden Layers = 2
Hidden Neurons = 30

Cycle 3
Hidden Layers = 3
Hidden Neurons = 9
...
Hidden Layers = 3
Hidden Neurons = 30

I admit this has inefficiencies as each layer may not need the same number of neurons, but the point of this post is to try and improve this mechanism by minimizing the number of layers/neurons combinations that need to be tested to determine the "optimal" counts.

EDIT:

So just to clarify the question is about determining what a good minimum starting point would be if using the brute-force mechanism outlined above regardless of data domain (hence the input + 1 example).

There is no hard-and-fast rule for this.

The number of hidden nodes you should have is based on a complex relationship between

Number of input and output nodes
Amount of training data available
Complexity of the function that is trying to be learned
The training algorithm
Too few nodes will lead to high error for your system as the predictive factors might be too complex for a small number of nodes to capture
Too many nodes will overfit to your training data and not generalize well

To conclude,

There is no hard and fast rule. You just have to keep trying with different number of layers to see which one works best...
In neural networks, model architecture is an art that you can master with some experience and domain knowledge.

Check this answer too...


How to validate regex based Resume parser efficiently I am using rule based logic to extract features from resume. Basically I am trying to find if the candidate switched the company in less than 1 year. So I have the code in place to find it using python. However if I want to validate it, I am currently doing it manually for few resumes, meaning I open the resume and find if the candidate switched the company in less than 1 year and compare it with what my regex logic gives. But this is time consuming process to validate.

Is there any better way to validate this piece of python code. ?

Is there any better way to validate this piece of python code. ?

No, because any automatic method would be equivalent to creating another learner, and there would be no way to know if it's good or bad at the job. So unless you find another annotated dataset (i.e. a set of resumes with labels indicating whether or not they switched jobs in the past year), you must create your own annotated set.


minimization with a negative cost function: works in MATLAB, not in Python I'm trying to use a particular cost function (based on doubling rate of wealth) for a classification problem, and the solution works well in MATLAB. See https://github.com/acmyers/compareCostFXs

When I try to do this in Python 2.7.6 I don't get any errors, but it only returns zeros for the theta values.

Here is the cost function and optimization method I've used in Python:

def costFunctionDRW(theta, X, y):

    # Initialize useful values
    m = len(y)
    # Marginal probability of acceptance
    marg_pA = sum(y)/m
    # Marginal probability of rejection
    marg_pR = 1 - marg_pA

    # =============================================================
    pred = sigmoid(np.dot(X,theta))
    final_wealth_individual = (pred/marg_pA)*y + ((1-pred)/marg_pR)*(1-y)
    final_wealth = np.prod(final_wealth_individual)
    final_wealth = -final_wealth

    return final_wealth

result = scipy.optimize.fmin(costFunctionDRW, x0=initial_theta, \
                   args=(X_array, y_array), maxiter=1000, disp=False, full_output=True )

Any advice would be much appreciated!

The problem is with division on python 2.x.  In python 2.x, division involving two integers produces an integer result.  So 1/2==0.  Python 3.x does not have this problem, 1/2==.5.

There are two ways to avoid this.  First, you can always convert one value you a float.  So 1./2==0.5 and 1/2.==0.5.  However, you have to remember to do this everywhere, and if you forget it can lead to hard-to-find errors.

The more reliable method is to always put this at the top of your code: from __future__ import division.  This will switch python 2.x to the python 3 behavior, so 1/2==.5.  In python 3.x it does nothing, so it also makes your code python 3.x compatible in this regard.


Multi-class logistic regression I am trying to understand logistic regression for a multi-class example and I have the following code:

import numpy
import theano
import theano.tensor as T
rng = numpy.random

num_classes = 3
#N = number of examples
N = 100
#feats = number of input neurons
feats = 784
#training rate
tr_rate = 0.1

D = (rng.randn(N, feats), rng.randint(size=N, low=0, high=num_classes))
training_steps = 1000

# Declare Theano symbolic variables
x = T.matrix("x")
y = T.ivector("y")

w = theano.shared(rng.randn(feats), name="w")
b = theano.shared(0.01, name="b")

# Construct Theano expression graph
sigma = T.nnet.softmax(T.dot(x,w) + b)
prediction = T.argmax(sigma, axis=1)          # The class with highest probability

# Cross-entropy loss function
xent = -T.mean(T.log(sigma)[T.arange(y.shape[0]), y])

cost = xent.mean() + 0.01 * (w ** 2).sum()      # Regularisation
gw, gb = T.grad(cost, [w, b])                   # Compute the gradient of the cost

# Compile
train = theano.function(
          inputs=[x,y],
          outputs=[xent],
          updates=((w, w - tr_rate * gw), (b, b - tr_rate * gb)),
          allow_input_downcast=True)
predict = theano.function(inputs=[x], outputs=prediction)

# Train
for i in range(training_steps):
    train(D[0], D[1])

However at this point the code gives me the following error:

ValueError                                Traceback (most recent call last)
<ipython-input-246-c4753ce8ccc7> in <module>()
      1 # Train
      2 for i in range(training_steps):
----> 3     train(D[0], D[1])

/Users/vzocca/gitProjects/Theano/theano/compile/function_module.pyc in __call__(self, *args, **kwargs)
    869                     node=self.fn.nodes[self.fn.position_of_error],
    870                     thunk=thunk,
--> 871                     storage_map=getattr(self.fn, 'storage_map', None))
    872             else:
    873                 # old-style linkers raise their own exceptions

/Users/vzocca/gitProjects/Theano/theano/gof/link.pyc in raise_with_op(node, thunk, exc_info, storage_map)
    312         # extra long error message in that case.
    313         pass
--> 314     reraise(exc_type, exc_value, exc_trace)
    315
    316

/Users/vzocca/gitProjects/Theano/theano/compile/function_module.pyc in __call__(self, *args, **kwargs)
    857         t0_fn = time.time()
    858         try:
--> 859             outputs = self.fn()
    860         except Exception:
    861             if hasattr(self.fn, 'position_of_error'):

ValueError: number of rows in x (1) does not match length of y (100)
Apply node that caused the error: CrossentropySoftmaxArgmax1HotWithBias(Elemwise{Add}[(0, 0)].0, Alloc.0, y)
Toposort index: 12
Inputs types: [TensorType(float64, row), TensorType(float64, vector), TensorType(int32, vector)]
Inputs shapes: [(1, 100), (100,), (100,)]
Inputs strides: [(800, 8), (8,), (4,)]
Inputs values: ['not shown', 'not shown', 'not shown']
Outputs clients: [[Sum{acc_dtype=float64}(CrossentropySoftmaxArgmax1HotWithBias.0)], [], []]

Backtrace when the node is created:
  File "<ipython-input-244-834f04b2fb36>", line 16, in <module>
    xent = -T.mean(T.log(sigma)[T.arange(y.shape[0]), y])

HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.

Can anyone help understand what is wrong?
Thank you!

I found my own answer. I had defined

w = theano.shared(rng.randn(feats), name="w")

and that was wrong.
The correct definition is:

w = theano.shared(rng.randn(feats, num_classes), name="w")

since the weights link 'feats'-number of input neuron to 'num_classes'-number of output neurons.


How to scrape a website with a searchbar How do I scrape a website that basically looks like google with just a giant searchbar in the middle of the screen. From it you can search after various companies and their stats.

I have a list of 1000 companies I want to get information about. I want some bot to search each company from my list in the search bar, open the specific company's info window and extract a certain company code that exist on each page for each company.

Is there any easy and (of course) legal way to do it?

I would suggest reading about http query methods, specifically about
GET and POST. You can pass parameters with query and open directly
company page.

For example:

http://google.com/search?q=GET+and+POST

where (q=GET+and+POST) is a parameter.
Once you have page you can parse it with your favorite library.
(for example beautifulsoup)

EXAMPLE:

Getting number of results from couple of google queries with python 3 and beautifulsoup:

from bs4 import BeautifulSoup
import urllib.request

# List with google queries I want to make
desired_google_queries = ['Word' , 'lifdsst', 'yvou', 'should', 'load', 'from']

for query in desired_google_queries:
    # Constracting http query
    url = 'http://google.com/search?q=' + query
    # For avoid 403-error using User-Agent
    req = urllib.request.Request(url, headers={'User-Agent' : "Magic Browser"})
    response = urllib.request.urlopen( req )
    html = response.read()
    # Parsing response
    soup = BeautifulSoup(html, 'html.parser')
    # Extracting number of results
    resultStats = soup.find(id="resultStats").string
    print(resultStats)


What is the proper order of normalization steps before and after splitting data I use a classification model on time-series data where I normalize the data before splitting the data into train and test. Now, I know that train and test data should be treated separately to prevent data leaking. What could be the proper order of normalization steps here? Should I apply steps 1,2,3 separately to train and test after I split data with the help of a sliding window? I use a sliding window here to compare each hour (test) with its previous 24 hrs data (train). Here is the order that I am currently using in the pipeline.

Moving averages (mean)
Resampling every hour
Standardization
Split data into train and test using a sliding window (of a length 24 hrs (train) and slides every 1 hr (test))
Fit the model using train data
Predict using the test data

Assuming I understand it correctly, I think your process is ok this way but I'm not sure about step 3 "standardization":

Steps 1 and 2 are ok since this cannot leak any kind of information from the test set to the training set.
If the standardization step involves calculating values (e.g. mean and s.d.) over the whole data (including test set) and then standardizing the whole data using these values, then that could be an issue. I'd suggest moving step 3 after the split, so that standardization is based only on the training set and then applied "as is" to the test set.


Q table creation and update for dynamic action space I am trying to implement a Q-learning algorithm for energy optimization. It is a finite MDP with states represented as 6 dimensional vectors of integers. The number of discrete values in each index of the state vector varies from 24 to 90.

The action space varies from state to state and goes up to 300 possible actions in some states, and below 15 possible actions in some states.

If I could make some assumptions (just for the purpose of testing the model), I could reduce the states to about 400 and actions to less than 200.

How can I construct a Q-table for such an environment? I am not sure how to approach this in Python, how to prevent the table containing lots of impossible state/action combinations, or prevent the agent trying to take those unwanted actions.

It is a finite MDP with states represented as 6 dimensional vectors of integers. The number of discrete values in each index of the state vector varies from 24 to 90.
The action space varies from state to state and goes up to 300 possible actions in some states, and below 15 possible actions in some states.

In combination, this could be over a billion state/action combinations. It is likely to be too much for running a tabular method over. The dynamic actions, changing depending on what the state is are not an issue, but the total size of such a table probably will be.

If I could make some assumptions (just for the purpose of testing the model), I could reduce the states to about 400 and actions to less than 200.

This is a much more tractable number for creating a table. Your total number of records will be at most 80,000.
There are a couple of things worth looking at in more detail:
Number of training examples needed
With a tabular method, as well as constructing the table, you need to collect data for each possible state/action combination. If there is some randomness in the environment, then you need to collect data from each possible state/action combination multiple times. For a table with size 80,000 this might be very fast and easy if you have a fast simulation of the environment. However, it might still not be a feasible size if you can only collect information from a single real environment and the time step represents a day . . .
You will need to check your own numbers to figure out whether this is an issue.
Perhaps even the billion plus items for your full description is not an issue if you have a fast computer and lots of memory to store the table.
Dealing with ragged data in a table
You don't have to do this. If you have enough memory spare and want to take the easy route, you could just use a tensor description (i.e. allow all values in each dimension, and enumerate them from 0 to whatever size). See next section for how you should limit action choice.
However, it may be more efficient to do something about the ragged data.
The simplest way to do this is to generate a unique ID for each allowed state vector, and use that to reference a value in a nested hash structure, e.g. a Python dict of dicts.
A very simple ID generator would be to concatenate your state vector and action labels into a list, then convert that into a string using some kind of join function. Example in Python:
# Empty Q table
q = dict()

# Define an example state, action and value
state = [20,12,56,9,76,30]
action = 176
value = -2.4

# Store an example state, action and value

# This state to state_id conversion should be a re-usable function
state_id = '/'.join([str(x) for x in state])
# Example state_id is '20/12/56/9/76/30'

if state_id in q:
  q[state_id][action] = value
else
  q[state_id] = {action: value}

The easiest way to manage the Q table with this approach is to add entries as and when you need them. Don't pre-populate with zeros, only write action values when they are needed by your agent. That will mean that you only create table entries for state, action pairs that exist.
Filtering action values
You will need code, that you consider to be part of the environment, that knows which actions are valid depending on the current state. This code could return the list of all possible actions. For instance you might define a function allowed_actions(state). You then use that list when working with the Q table:

When looking for a maximising action to drive a policy, look up the whole dictionary of action to value (in q[state_id] and pick the max one)

When you reach a new state that you have not seen before, it may be convenient to populate all possible actions from that state with zeros. Prototype Python code for doing that might look like this:

state_id = '/'.join([str(x) for x in state])
if not state_id in q:
  q[state_id] = { action: 0 for action in allowed_actions(state) }


Variable input/output length for Transformer I was reading the paper "Attention is all you need" (https://arxiv.org/pdf/1706.03762.pdf ) and came across this site
http://jalammar.github.io/illustrated-transformer/ which provided a great breakdown of the architecture of the Transformer.

Unfortunately, I was unable to find any explanation of why it works with input/output lengths that are not equal (eg. input: “je suis étudiant” and expected output: “i am a student”).

My main confusion is this. From what I understand, when we are passing the output from the encoder to the decoder (say $3 \times 10$ in this case), we do so via a Multi-Head Attention layer, which takes in 3 inputs:

A Query (from encoder), of dimension $3 \times k_1$
A Key (from encoder), of dimension $3 \times k_1$
A Value (from decoder), of dimension $L_0 \times k_1$, where $L_0$ refers to the number of words in the (masked) output sentence.

Given that the Multi-Head Attention should take in 3 matrices which have the same number of rows (or at least this is what I have understood from its architecture), how do we deal with the problem of varying output lengths?

Your understanding is not correct: in the encoder-decoder attention, the Keys and Values come from the encoder (i.e. source sequence length) while the Query comes from the decoder itself (i.e. target sequence length).

The Query is what determines the output sequence length, therefore we obtain a sequence of the correct length (i.e. target sequence length).

In order to understand how the attention block works maybe this analogy helps: think of the attention block as a Python dictionary, e.g.

keys =   ['a', 'b', 'c']
values = [2, 7, 1]
attention = {keys[0]: values[0], keys[1]: values[1], keys[2]: values[2]}
queries = ['c', 'a']
result = [attention[queries[0]], attention[queries[1]]]

In the code above, result should have value [1, 2].

The attention from the transformer works in a similar way, but instead of having hard matches, it has soft maches: it gives you a combination of the values weighting them according to how similar their associated key is to the query.

While the number of values and keys has to match, the number of queries is independent.


Is it possible to decompose a scalar value to a inter-dependent vector neural network? My data contains a scalar feature $r$, I found this feature is important for training my deep model. My idea is supposing there is a 3-layer MLP $f(x), x \in \mathbb{R}^{n}$, where $n=1$. It outputs a vector with dimension $m$ where each value in $[0, 1]$.

For my data, it inputs $r$ and outputs an m-sized vector.

So here is my decomposition idea make sense?

Yes, you can do that by interchanging the position of decoder and encoder in an autoencoder.

In an autoencoder, you give a long vector as input - the encoder reduces it to a short length vector (compressed) - the decoder now takes this compressed vector as input and upsamples it to the size of the original vector. The autoencoder is trained by taking the Mean Square Error (MSE) of the output of decoder with respect to the input vector. This enforces the compressed vector representation to contain the information of the input vector.

Now coming to your case. You simply need to pass the single scalar value to a decoder that upsamples, say your 3 layer fully connected neural networks. Let this output be denotes as "latent representation". Now pass this "latent representation" to the encoder which uses this "latent representation" to output just a single scalar value. Use MSE objective to enforce the the above single scalar output to match the input scalar value. Once the training is done of the above reverse autoencoder, the "latent representation" will give you the required vector containing the information about the scalar value you wished to represent as a vector.


Good-Turing Smoothing Intuition I'm working through the Coursera NLP course by Jurafsky & Manning, and the lecture on Good-Turing smoothing struck me odd.

The example given was:

  You are fishing (a scenario from Josh Goodman), and caught:
  10 carp, 3 perch, 2 whitefish, 1 trout, 1 salmon, 1 eel = 18 fish
  ...
  How likely is it that the next species is new (i.e. catfish or bass)
  Let's use our estimate of things-we-saw-once to estimate the new things.
  3/18 (because N_1=3)

I get the intuition of using the count of uniquely seen items to estimate the number of unseen item types (N = 3), but the next steps seem counterintuitive.

Why is the denominator left unchanged instead of incremented by the estimate of unseen item types? I.e., I would expect the probabilities to become:

  Carp : 10 / 21
  Perch : 3 / 21
  Whitefish : 2 / 21
  Trout : 1 / 21
  Salmon : 1 / 21
  Eel : 1 / 21
  Something new : 3 / 21

It seems like the Good-Turing count penalizes seen items too much (trout, salmon, & eel are each taken down to 1/27); coupled with the need to adjust the formula for gaps in the counts (e.g., Perch & Carp would be zeroed out otherwise), it just feels like a bad hack.

There are no unseen item types in the given data, by definition. 3 is the count of items seen once, and they are already included in the denominator 18. If the next item were previously unseen, it would become seen once when it appears. Since 3-of-18 examples were seen-once items, this is an estimate of the probability that the next item will be seen-once too on its first appearance.

It is certainly a heuristic. There is no way to know whether there are 0 or 1000 other types out there.


How to convert an array of numbers into probability values? I would like some help with respect to certain numerical computation. I have certain arrays which look like:
Array 1:
[0.81893085, 0.54768653, 0.14973508]

Array 2:
[0.48078357, 0.92219683, 1.02359911]

Each of the three numbers in the array represents distance of a data point from the cluster centroid in k-means algorithm. I want to convert these numbers into probabilities. The element which has a high distance should be converted into a low probability. For example, [0.81893085, 0.54768653, 0.14973508] can be converted into a probability vector like [0.13, 0.22, 0.65]. As it can be seen, the elements which have a high value in the original array have low value in the probability array (and of course the values in the probability array sum to 1).

Is there any mathematical technique that will achieve this result?

What I have tried till now is, I took the inverse of each of the values in the original array:

1/[0.81893085, 0.54768653, 0.14973508] = [1.22110431, 1.82586195, 6.67846172]

And then I input the resulting array to softmax function (softmax function converts an array of numbers to probabilities) - https://en.wikipedia.org/wiki/Softmax_function

This gives a probability vector of [0.00421394, 0.00771491, 0.98807115].
Is this a good approach? Is there any other approach?

Any Survival Function (1 minus the CDF) will have the desired property. Exponential is a potentially good candidate here, as it sometimes can be used to describe distances, but it's hard to say without more information.

$$S(x) = \exp(-ax)$$

The parameter $a$ can be tuned or possibly estimated from the data.

For reference, if $a = 1$ then you get,

$$ [0.44, 0.58, 0.86]$$

$$[0.62, 0.4, 0.36]$$

for the first and second arrays respectively.


Optimum values of all predictors in logistic regression I am trying to figure out in logistic regression, with the help of coefficients of $x$ variables, we could figure out that on unit change in any $x$ variable what will be the change in probability of y variable.

But I want to understand what must be the values of different $x$ variables to be set so that we are most likely to get only desired results.

For example if we are trying to predict defaulters in loan dataset, now by running logistic regression and its coefficients, we would be able to get the idea that by unit change in $x_1$(or $x_2$ or $x_3$) what will be the change in probability of $y$ to be $1$.

Now if I want to understand that what must be the values of $x_1,x_2,x_3, \ldots$ etc to be set so that the chances of $y$ to be $1$ will be maximum?

Theoretically you won't be able to get the value $1$. However, suppose your variable is bounded and you can indepedently set the values of the variables, you can let it be as large as possible.

Look at the coefficients that you obtained. For those coefficients that you obtained, if they are positive, set the value to be as large as possible. For negative coefficients, set the variable to be as small as possible. If the coefficients is zero, any value is fine.


Can we do multivariate time series analysis using holt-winter ( Exponential smoothing) method? Just like we have a method like ARIMAX and SARIMAX where we can provide exog and endog variable for perfroming multivariate analysis. I was hoping is there a way, we can achieve same using ETS as well. Please let me know in case any has worked on this.

Exponential smoothing is usually defined on univariate data with no exogenous variables (see e.g. this book).

However, Rob Hyndman discusses different approaches of including exogenous variables in this blog post and gives some more references.


If There is a case where decision trees are getting overfitted so by using gradient boost method do we solve that problem? I have came across a case where my decision trees are getting overfitting so by using methods like gradient boost can I solve that problem.

If your trees are overfitting you should probably check the hyperparameters of the tree. You shouldn't let the tree have leaves with too few samples, for example, since that would mean that the tree is memorizing every possible outcome of the training set, in other words, it's overfitting.

Maybe if you share some more information of your case we could help a little further, but I think you now at least have something to check.

Hope this helps!


How to feed LSTM with different input array sizes? If I like to write a LSTM network and feed it by different input array sizes, how is it possible?

For example I want to get voice messages or text messages in a different language and translate them. So the first input maybe is "hello" but the second is "how are you doing". How can I design a LSTM that can handle different input array sizes?

I am using Keras implementation of LSTM.

The easiest way is to use Padding and Masking.

There are three general ways to handle variable-length sequences:

Padding and masking (which can be used for (3)),
Batch size = 1, and
Batch size > 1, with equi-length samples in each batch.

Padding and masking

In this approach, we pad the shorter sequences with a special value to be masked (skipped) later. For example, suppose each timestamp has dimension 2, and -10 is the special value, then

X = [

  [[1,    1.1],
   [0.9, 0.95]],  # sequence 1 (2 timestamps)

  [[2,    2.2],
   [1.9, 1.95],
   [1.8, 1.85]],  # sequence 2 (3 timestamps)

]

will be converted to

X2 = [

  [[1,    1.1],
   [0.9, 0.95],
   [-10, -10]], # padded sequence 1 (3 timestamps)

  [[2,    2.2],
   [1.9, 1.95],
   [1.8, 1.85]], # sequence 2 (3 timestamps)
]

This way, all sequences would have the same length. Then, we use a Masking layer that skips those special timestamps like they don't exist. A complete example is given at the end.

For cases (2) and (3) you need to set the seq_len of LSTM to None, e.g.

model.add(LSTM(units, input_shape=(None, dimension)))

this way LSTM accepts batches with different lengths; although samples inside each batch must be the same length. Then, you need to feed a custom batch generator to model.fit_generator (instead of model.fit).

I have provided a complete example for simple case (2) (batch size = 1) at the end. Based on this example and the link, you should be able to build a generator for case (3) (batch size > 1). Specifically, we either (a) return batch_size sequences with the same length, or (b) select sequences with almost the same length, and pad the shorter ones the same as case (1), and use a Masking layer before LSTM layer to ignore the padded timestamps, e.g.

model.add(Masking(mask_value=special_value, input_shape=(None, dimension)))
model.add(LSTM(lstm_units))

where first dimension of input_shape in Masking is again None to allow batches with different lengths.

Here is the code for cases (1) and (2):

from keras import Sequential
from keras.utils import Sequence
from keras.layers import LSTM, Dense, Masking
import numpy as np

class MyBatchGenerator(Sequence):
    'Generates data for Keras'
    def __init__(self, X, y, batch_size=1, shuffle=True):
        'Initialization'
        self.X = X
        self.y = y
        self.batch_size = batch_size
        self.shuffle = shuffle
        self.on_epoch_end()

    def __len__(self):
        'Denotes the number of batches per epoch'
        return int(np.floor(len(self.y)/self.batch_size))

    def __getitem__(self, index):
        return self.__data_generation(index)

    def on_epoch_end(self):
        'Shuffles indexes after each epoch'
        self.indexes = np.arange(len(self.y))
        if self.shuffle == True:
            np.random.shuffle(self.indexes)

    def __data_generation(self, index):
        Xb = np.empty((self.batch_size, *X[index].shape))
        yb = np.empty((self.batch_size, *y[index].shape))
        # naively use the same sample over and over again
        for s in range(0, self.batch_size):
            Xb[s] = X[index]
            yb[s] = y[index]
        return Xb, yb

# Parameters
N = 1000
halfN = int(N/2)
dimension = 2
lstm_units = 3

# Data
np.random.seed(123)  # to generate the same numbers
# create sequence lengths between 1 to 10
seq_lens = np.random.randint(1, 10, halfN)
X_zero = np.array([np.random.normal(0, 1, size=(seq_len, dimension)) for seq_len in seq_lens])
y_zero = np.zeros((halfN, 1))
X_one = np.array([np.random.normal(1, 1, size=(seq_len, dimension)) for seq_len in seq_lens])
y_one = np.ones((halfN, 1))
p = np.random.permutation(N)  # to shuffle zero and one classes
X = np.concatenate((X_zero, X_one))[p]
y = np.concatenate((y_zero, y_one))[p]

# Batch = 1
model = Sequential()
model.add(LSTM(lstm_units, input_shape=(None, dimension)))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
print(model.summary())
model.fit_generator(MyBatchGenerator(X, y, batch_size=1), epochs=2)

# Padding and Masking
special_value = -10.0
max_seq_len = max(seq_lens)
Xpad = np.full((N, max_seq_len, dimension), fill_value=special_value)
for s, x in enumerate(X):
    seq_len = x.shape[0]
    Xpad[s, 0:seq_len, :] = x
model2 = Sequential()
model2.add(Masking(mask_value=special_value, input_shape=(max_seq_len, dimension)))
model2.add(LSTM(lstm_units))
model2.add(Dense(1, activation='sigmoid'))
model2.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
print(model2.summary())
model2.fit(Xpad, y, epochs=50, batch_size=32)

Extra notes

Note that if we pad without masking, padded value will be regarded as actual value, thus, it becomes noise in data. For example, a padded temperature sequence [20, 21, 22, -10, -10] will be the same as a sensor report with two noisy (wrong) measurements at the end. Model may learn to ignore this noise completely or at least partially, but it is reasonable to clean the data first, i.e. use a mask.


Using simulations to train ML algorithms Possibly similar question: Is it ok to collect data using algorithm to train another?

I have a model that accurately describes an underlying physical, complex, system. The model is basically a set of ODEs based on the physics of the system, validated against measurements. When a system perturbation occurs, I can run thousands of simulations to assess if the new system status is secure or not. This is basically a classification procedure (yes/no).

This procedure is very time consuming and has to be performed in real-time (thus, requiring huge computational resources). There are thousands of possible perturbations and infinite number of initial points. The same perturbation from 1 initial point can lead to stable system, while from another, to an unstable.

My question is:

Is it possible to use data generated by a huge number of simulations to train a classification algorithm to perform this detection online? What are the considerations when using simulated data to train an algorithm that will then be used online with real data (expect from the obvious that the simulation needs to be very very accurate)? Any references to such examples?

I apologise if this is a basic question. I am new to data science techniques with a more physics/engineering background.

Is it possible to use data generated by a huge number of simulations to train a classification algorithm to perform this detection online?

Yes, it is always possible to train a classification algorithm when you have labeled i.i.d. training data, and there is no hard reason why you cannot use a simulator to generate that.

Whether or not such a trained model is fit for purpose is hard to say in advance of trying it.

Using a simulation as your data source has some benefits:

Generating more training and test data is straightforward.
You will automatically have high quality ground truth labels (assuming your goal is to match the simulation).
If you find a problem with certain parameter values, you can target them when collecting more training data.

Just as with data taken from real world measurements, you will need to test your results to get a sense of how accurate your model is.

  What are the considerations when using simulated data to train an algorithm that will then be used online with real data (expect from the obvious that the simulation needs to be very very accurate)?

Your model is a function approximator. At best it will match the output of the simulator. In practice it will usually fall short of it by some amount. You will have to measure this difference by testing the model, and decide whether the cost of occasional false negative or false positive is outweighed by the performance improvement.
Statistical machine learning models perform best when interpolating between data points, and often perform badly at extrapolating. So when you say that inputs can vary infinitely, hopefully that is within some constrained parameter space of real values, as opposed to getting inputs that are completely different from anything you have considered before - the simulation would cope with such inputs, but a statistics-based function approximator most likely would not.
If your simulation has areas where the class switches rapidly with small changes in parameter values, then you will need to sample densely in those areas.
If your simulation produces near chaotic behaviour in any region (class value varies a lot and is highly sensitive to small changes in value of one or more parameters), then this is something that is very hard to approximate.
If you have some natural scale factor, dimensionless number or other easy to compute summary of behaviour in your physical system, it may be worth using it as an engineered feature instead of getting the machine learning code to figure that out statistically. For instance, in fluid dynamics, the Reynolds number can characterise flow, and could be useful feature for neural network predicting vortex shedding.

  Any references to such examples?

The examples I have found here are about are in renderings of fluid simulations and other complex physical systems where a full simulation can be approximated and they all use neural networks to achieve a speed improvement over full simulation.

Accelerating Eulerian Fluid Simulation With Convolutional Networks - I saw a video about this on YouTube's Two Minute Papers channel.
Using neural networks in weather prediction ensembles to improve performance
Fast Neural Network Emulation of Dynamical Systems for Computer Animation

However, I don't think any of these are classifiers.


Making Use of the Target Values for Regression Problem:
I have a regression problem and I decided to useg Gradient Boosting Regression Trees to solve it. After all the preprocessing, I end up having around 130 attributes, 70K rows, and my cross-validated R-squared is 0.62.
Work So Far:
To increase my R-squared (and tighten my prediction intervals) I tried using a hint-feature that divides the target into linear groups (I manually decided for 4 groups and placed the target into 4 bins, - why 4? not too small not to big and not cleverly decided.).
I am aware that this is not a usual way to go, but this is doable for my application since eventually, I will be collecting these hints from the user. (I will be asking for the target value for that user and suggest him/her another target value depending on his/her data.)
So it would be possible for me to place each user into such groups since it seems like not all of the users behave the same and they need a separated modeling for the ones that are extremely high/low target values, plus the distribution is extremely right-skewed.
After using this cheating-hint-feature, my cross-validated R-squared became 0.81. However, obviously, dividing the target into linear groups like that, brings the disadvantage of having a step-like model that has a clear separation between those groups while this is not necessarily always the case (a user that gives his target value most probably but not necessarily should belong to that group), the boundary values for such binning should not be manually decided, and this model would definitely need to be smoothed.
Next Steps?
After seeing that such an approach can be helpful (and I could really make use of the user-given values), I am now thinking about smarter ways to separate between the users regarding these hints. I am thinking about applying hierarchical clustering and using the cluster information as a hint-feature but not sure if this is the best way to proceed.
Would you maybe have suggestions for me for such method (and papers would be great) or statistical approach to figure out such boundary values (other than density-based methods)?

Clustering is definitely something that can help. As you describe, the issue is that you can see that not all instances "behave" the same way. So if you can cluster them into groups that do behave more similarly, you would probably improve.

Any clustering technique would work in theory. Although, techniques like K-means for instance do favour clusters of even-size, and so if your problem isn't necessarily "balanced", I would be cautious.

What you could also do to improve your clustering is to look at which features are more discriminative (perhaps look at the information gain) and use a weighted distance to favour these features.

Another way I feel could help you is if you used meta learning/stacking. By that I mean that you could use decision trees to split your data in a supervised way (as opposed to clustering), and at the leaves, use a more elaborate regressor than majority voting.


Linear Regression in Python using gradient descent I am trying to implement a simple multivariate linear regression model without using any inbuilt machine libraries. So far, I have been able to get a root mean squared error for training about $2.93$ and the model from the normal (closed-form) equation is able to produce a training RMSE of $~2.3$. I am looking for ways in which I can improve my implementation of the gradient descent algorithm. Below is my implementation:

My gradient descent method looks like this:
$\theta = \theta - [(\alpha/2N) * X (X\theta - Y)]$ where $\theta$ is the model parameter, $N$ is the number of training elements, $X$ is the input and $Y$ are the target elements. $\alpha$ is the step size.

def gradientDescent(self):
    for i in range(self.iters):
        # T = T - (\alpha/2N) * X*(XT - Y)
        self.theta = self.theta - (self.alpha/len(self.X)) * np.sum(self.X * (self.X @ self.theta.T - self.Y), axis=0)
    return errors

I had set the $\alpha$ as $0.1$ and number of iterations as 1000. The gradient descent reaches convergence at around 700-800 iterations (checked).

My error function is like:

def error_function(self):
        # Error function: (1/2N) * (XT - Y)^2 where T is theta
        error_values = np.power(((self.X @ self.theta.T) - self.Y), 2)
        return np.sum(error_values)/(2 * len(self.X))

I was expecting the training error from the gradient descent and the normal equations would turn out to be similar, but they have a bit of a huge difference. So, I wanted to know whether I am doing anything wrong or not.

PS I have not normalized the data, yet. Normalizing leads to a much lower RMSE (~$0.22$)

That could be due to many different reasons. The most important one is that your cost function might be stuck in local minima. To solve this issue, you can use a different learning rate or change your initialization for the coefficients.
There might be a problem in your code for updating weights or calculating the gradient.
However, I used both methods for a simple linear regression and got the same results as follows:
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets import make_regression

# generate regression dataset
X, y = make_regression(n_samples=100, n_features=1, noise=30)

def cost_MSE(y_true, y_pred):
    '''
    Cost function
    '''
    # Shape of the dataset
    n = y_true.shape[0]

    # Error
    error = y_true - y_pred
    # Cost
    mse = np.dot(error, error) / n
    return mse

def cost_derivative(X, y_true, y_pred):
    '''
    Compute the derivative of the loss function
    '''
    # Shape of the dataset
    n = y_true.shape[0]

    # Error
    error = y_true - y_pred

    # Derivative
    der = -2 / n * np.dot(X, error)

    return der

# Lets run an example

X_new = np.concatenate((np.ones(X.shape), X), axis = 1)
learning_rate = 0.1
X_new_T = X_new.T
n_iters = 100
mse = []

#initialize the weight vector
alpha = np.array([0, np.random.rand()])

for _ in range(n_iters):

    # Compute the predicted y
    y_pred = np.dot(X_new, alpha)

    # Compute the MSE
    mse.append(cost_MSE(y, y_pred))

    # Compute the derivative
    der = cost_derivative(X_new_T, y, y_pred)

    # Update the weight
    alpha  -= learning_rate * der
alpha

for the gradient descent the coefficients were:
array([-3.36575322, 28.06370831])

Here is the code for closed-form solution:
np.dot(np.linalg.inv(np.dot(X_new_T,X_new)), np.dot(X_new_T, y))

And the coefficients for the closed-form solution:
array([-3.36575322, 28.06370831])

As the coefficients are equal, the RMSE,  MSE, R2 are equal.


SciKit-Learn Decision Tree Overfitting I'm pursuing a computer science minor at my university, and one class I'm in is Machine Learning.
We have a project to utilize a few algorithms we have learned so far.
I've been using SciKit-Learn to perform these algorithms, but when it comes to decision trees I keep getting a feeling I am overfitting.

I'm using a dataset about weather, giving characteristics such as city, state, month, year, wind direction, wind speed, etc... where the target variable is the average temperature for the day. Now I know this is hard to classify, as it is pretty much a continuous variable space, but I've simplified it to the predicted being within a range of 5 to the actual.

Here is a link to the csv I'm using

The following is my code:

address2 = 'C:/.../weather.csv'
weather = pd.read_csv(address2)

cityCode= le.fit_transform(weather.iloc[:,2])
windDirection = le.fit_transform(weather.iloc[:,3])
month = le.fit_transform(weather.iloc[:,8])
precip = le.fit_transform(weather.iloc[:,9])
windSpeed = le.fit_transform(weather.iloc[:,10])
state = le.fit_transform(weather.iloc[:,11])
week = le.fit_transform(weather.iloc[:,12])
year = le.fit_transform(weather.iloc[:,13])

Xweather = list(zip(cityCode,windDirection,month,precip,windSpeed,state,week,year))
yweather= weather.iloc[:,0]

yweather_test = train_test_split(Xweather, y, test_size = 0.2, random_state=413)

cWeather = tree.DecisionTreeClassifier()
cWeather.fit(Xweather_train,yweather_train)
accu_train_weather=np.sum(abs(cWeather.predict(Xweather_train)-yweather_train)<=5)/float(yweather_train.size)*100
accu_test_weather=np.sum(abs(cWeather.predict(Xweather_test)-yweather_test)<=5)/float(yweather_test.size)*100
print("Classificaton accuracy on training set", accu_train_weather, "%")
print("Classificaton accuracy on test set", accu_test_weather, "%")

My training set constantly gets 100% training accuracy, but the test set is constantly 57% accurate, which leads me to believe the tree is overfitting to the training set.

I know I'm not doing any pruning, but even when I do, I can get the same test accuracy as unpruned at best.
By pruning I mean setting the tree classifier to have a maximum number of leaves, minimum samples per leaf, and maximum depth.

I'm not an advanced coder (as you can probably tell by my code), but any help would be great.

Predicting average temperature is a regression task, not classification. You should be using DecisionTreeRegressor instead. Temperature is a continuous value and you are treating it as a category by using a classifier.

Tinkering with the hyperparameters (maximum number of leaves, minimum samples per leaf, and maximum depth, etc) is still important since decision trees always are prone to overfitting. If you struggle to find good parameters yourself then you can try some automated methods such as GridSearchCV or RandomizedSearchCV in sklearn.


Which libraries in Python are there in NLP to tokenize the Hindi sentence? For English language there are libraries like NLTK, CoreNLP which are used for Text Normalization, Word Tokenization and Detokenization, Sentence Splitting etc.
Like English, is there any library to do above operation using Hindi Script ?

There is an Indic NLP library developed by iitB for Hindi texts. you can check out the below links

https://www.cse.iitb.ac.in/~anoopk/pages/softwares.html

https://github.com/anoopkunchukuttan/indic_nlp_library


Random forest classifier is predicting only one class even when the dataset is not imbalanced This is a binary classification task, I have 15K 1's and 11K 0's (target)

I have tried the following:

X = feature_cols
y = department_wise[['Threshold']]
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2, random_state=1)
model = RandomForestClassifier()
model.fit(X, y)
predicted_labels = model.predict(X_test)

X_test predicts only 0 and the accuracy comes around 88%. I don't understand why since my data set is not even imbalanced.
Whatever other classifier I am trying, it shows the same result with high accuracy. Please let me know where I am going wrong.

It is not unusual that some method (given some data) predict only one class (meaning probabilities are all larger or smaller than 0.5). I had this case recently with a logistic regression application. The reason usually is that you don‘t have good features (x) to predict y. So first thing: have a look at correlation of X,y.

What you can do to improve your fit is basically to use boosting. Boosting is also tree based and similar to RF it is also an ensemble technique. But boosting gives higher weight to observations which are hard to predict. LightGBM or Catboost are good routines for boosting.

In my case, Logit predicted only one class with an AUC of about 0.3. LightGBM was much better and much more balanced in terms of prediction with an AUC of about 0.7.

You could also try Logit with L1 regulation (Lasso). Maybe some of your features are not very helpful in making predictions. Lasso shrinks these features, which helps to make okay predictions.


about predict the class for a new datapoint I have a new data point and want to classify it into the existing classes.I can calculate pairwise distance for the new point to all existing points(in the existing classes). I know using KNN would be a straightforward to classify this point. Is there a way I could randomly sampling existing classes and then correlated the new point to a potential classes without calculating all pairwise distances?

Since you have your clusters set up already you should be able to calculate cluster centroids and then determine the distance between a each cluster centroid and the new data point. You could follow the same process with random samples of each cluster.


Output of classifier.predict Tensorflow extract probabiltity When I do a prediction with my DNN clasifier I get a dictionary like this.

{'probabilities': array([9.9912649e-01, 8.7345875e-04, 8.5633601e-12], dtype=float32), 'logits': array([ 12.641698,   5.599522, -12.840958], dtype=float32), 'classes': array(['0'], dtype=object), 'class_ids': array([0])}

Can someone explain me the values of probability and logits? Why the three values ?

The docs just states

  Evaluated values of predictions tensors.

And do not refer (the docs) to a struct/explanation of the output

Thanks!

At the probabilities key you will find the probabilities of every label. Tensorflow just chooses the one with the highest. So in order to get the probability of the current outcome, you need to do something like this.

results = classifier.predict(input_fn = lambda: mem_input_fn())

for r in results:
    idx = r["classes"][0] # idx is the predicted label
    print idx, r["probabilities"][int(idx)]


Selecting Transforms with sklearn pipelines So I am currently working on a Data set, and I want to use Pipelines to select the transforms. Here is an example of what I want to do :
# I missed the imports just for brevity
pipeline = make_pipeline(StandardScaler(), PolynomialFeatures(), PCA(), LogisticRegression())

So what I want to do, I want to try both StandardScaler, MinMaxScaler and no scaler when I am using other methods such as XGBOOST. Then I want to try to use polynomial features and then use a PCA, only if I used polynomial features. And finally, I want to try Logistic Regression and XGBOOST with GridSearch (This last part I know how to do).
Anybody did something similar?
Thank you!

The purpose of the pipeline is to assemble several steps that can be
cross-validated together while setting different parameters. For this,
it enables setting parameters of the various steps using their names

Check the docs
For your requirement, you will have to create multiple pipelines, you cannot do it via single pipeline.


Data splitting for a binary classification model I'm trying to build a binary classification model that will tell who's going to buy the product and who's not. I've heard that splitting a dataset into two different subsets is a common way when you prepare an input data.

  [ ================ Training Data 80% ================= ] [ ==== Test Set 20% ==== ]

Is it just mindlessly splitting a chunk of dataset by some amount of proportion like above? Is it that simple?

Imagine I have this simple dataset below.

UserId,UserName,AppId,Purchased
1,Lianne,1,1
1,Lianne,2,1
1,Lianne,3,1
1,Lianne,4,1
1,Lianne,5,1
1,Lianne,6,0
1,Lianne,7,0
1,Lianne,8,0
1,Lianne,9,0
1,Lianne,10,0

As the common recommended way, I splitted it into two groups.

// Training Data Set
UserId,UserName,AppId,Purchased
1,Lianne,1,1
1,Lianne,2,1
1,Lianne,3,1
1,Lianne,4,1
1,Lianne,5,1
1,Lianne,6,0
1,Lianne,7,0
1,Lianne,8,0

// Test Set
UserId,UserName,AppId,Purchased
1,Lianne,9,0
1,Lianne,10,0

Would this work? well it seemed not and it turned out it actually didn't. The model was wrong about predicting on the appId of 6,7,8,9. It thought the user number one would buy them with a slightly high chance. The metrics look like...

TP : 5
FP : 4
FN : 1
Accuracy : 0.5
Auc : NaN
F1Score : NaN
Precision : 0
Negative Precision : 1
Negative Recall : 0.5

To make a proper model, what my test dataset should look like on this sample training data?

My 2 cents:
the number of records in the data set used here is very small. If we have a look into the data set we can see that the target variable split is exactly 50:50 which means the probability is half. Its like flipping a coin to get heads or tail.

The training set contains a known output and the model learns on this data in order to be generalized to other data later on. The dependent variables and the independent variable should be in splatted and  then do a train test fit.

You can use the library from scikit learn as well
from sklearn.model_selection import train_test_split


Naive Bayes: Divide by Zero error OK this is my first time in ML and for starter I am implementing Naive Bayes. I have Cricket(sports) data in which I have to check whether the team will win or lost based on Toss Won|Lost and Bat First|Second. Below is my code:

from sklearn.naive_bayes import GaussianNB
import numpy as np

"""
    Labels : Lost, Draw, Won [-1,0,1]
    Features
    ==========
        Toss(Lost,Won) = [-1,1]
        Bat(First, Second) = [-1,1]
"""
#Based on Existing Data Features are:
features = np.array([[-1, 1],[-1, 1]])
labels = np.array([0,1])
# Create a Gaussian Classifier
model = GaussianNB()

# Train the model using the training sets
model.fit(features, labels)

# Predict Output
predicted = model.predict([[1,0]])
print(predicted)

On running this I get error:

/anaconda3/anaconda/lib/python3.5/site-packages/sklearn/naive_bayes.py:393: RuntimeWarning: divide by zero encountered in log
[0]
  n_ij = - 0.5 * np.sum(np.log(2. * np.pi * self.sigma_[i, :]))
/anaconda3/anaconda/lib/python3.5/site-packages/sklearn/naive_bayes.py:395: RuntimeWarning: divide by zero encountered in true_divide
  (self.sigma_[i, :]), 1)
/anaconda3/anaconda/lib/python3.5/site-packages/sklearn/naive_bayes.py:395: RuntimeWarning: invalid value encountered in subtract
  (self.sigma_[i, :]), 1)

Update

Code given here

Although I haven't verified it, a first glance at the features and training set you used shows an obvious problem. You have just two data samples with the exact same features while you give it different labels.

Although other types of models might not break and just give both samples an equal probability of being 0 or 1, something in the Naive Bayes classifier internal calculations probably requires there to be at least some difference in differently labeled samples (not an unreasonable assumption).

I can't confirm this for myself as I'd have to go deeper into the source code to actually check the scikit implementation so perhaps someone else more familiar with it can check.

EDIT: I just tested this out by changing one line in your code to

features = np.array([[-1, 1],[-1, 0]])

It worked.