Which is faster for imputing: R or Python? I have a fairly large dataset (about 40k rows, 40 columns) with many NA's (up to 40% of each variable). All of my work has been done in R so far, but I know R (which I'm told runs only on a single core) won't be able to impute the data with any type of respectable turn around time, especially on my laptop.

While I'm looking into more powerful cloud computing to help, I'm wondering if Python could offer better imputation times (presumably it can be run on more than one core).

I'm also still deciding on the algorithm -- likely Cart, KNN, or Forest. It's survey data.

I recommend you to use Python. With sklearn, you have many ways to perform algorithm multicore on your data.
Another reason to use Python: If you want to perform distributed computing you have many modules like DASK which are very easy to use and schedule automatically Tasks on different Machines without spending hours in coding multithreading algorithm.
40k rows, 40 columns are not a big problem for python, BUT I think "R" should be able to handle this, it's a very popular language for statistical algorithms. Maybe some settings are false...
Short:

Install Python (or better Anaconda)
on Sklearn-site you will find some demo code
Try it with your Dataset
Maybe you could benchmark it :)


Understanding LSTM input shape for keras I am learning about the LSTM network. The input needs to be 3D. So I have a CSV file which has 9999 data with one feature only. So it is only one file.

So usually it is (9999,1) then I  reshape with time steps 20 steps

timesteps = 20
dim = data.shape[1]
data.reshape(len(data),timesteps,dim)

but I am getting following error

  ValueError: cannot reshape array of size 9999 into shape (9999,20,1)

and the input in LSTM

model.add(LSTM(50,input_shape=(timesteps,dim),return_sequences=True, activation="sigmoid"))

(9999,1) has 9999*1 elements = 9999. However, (9999,20,1) will have 9999*20*1 elements, which are not available. Break your data into a batch/sequence length of say 99.
Then reshape it into (101,99,1)


Combine two DataFrames column wise in Pandas I have 2 Dataframes as shown and I want a new DataFrame where the 1st column is the 1st column of the 1st DataFrame and 2nd column from the 1st column of the 2nd DataFrame. I have tried pivot table etc but have no luck.

    A   B
0   1   2
1   3   67
2   4   54

    C   D
0   0   19
1   23  6
2   55  5

I need a new DataFrame like this.

    A   C   B   D
0   1   0   2   19
1   3   23  67  6
2   4   55  54  5

Use pandas.DataFrame.reindex()

df = pd.concat((df1, df2), axis=1)

#reorder columns
column_names=["A","C","B","D"]
df = df.reindex(columns=column_names)


Suggestions for Matchmaking Algorithm I run a heterosexual matching making service.  I have my male clients and my female clients.  I need to pair each of my clients with their "soul mate" based on several attributes (age, interests, personality types, race, height,horoscope, etc.)

After I create all my pairings, there will be some sort of score to grade the quality of my matches.

I can't match a man with multiple women or vice versa.  I also want to minimize the number of unmatched clients.

What's the best algorithm to come up with a way to match my clients based on their attributes?

Again, this is a toy example.  My actual use case is completely different.

Edit:

The score is computed at the pair level and then summed.  I can calculate how the score changes when I swap partners by looking at the new scores of two pairs.  I do have access to the internals of the metric, but it's complicated.  I don't have any constraints, other than I'd prefer it to be fast and simple for my own sanity.

Although you might find a way to apply machine learning (ML) to this optimisation problem, it does not look necessary, and is probably a distraction. ML might help if the scoring system was complex or if you had incomplete data for most matches, and needed to compute matching score estimates from some more limited set of attributes.

Instead here you seem to have a combinatorial optimisation problem. A well known example of this is the Travelling Salesman Problem.

There are many possible algorithms to attack these kinds of problem. Which to choose may depend on other traits of the data, such as how quickly you can calculate the scores - both for the whole set and for individual changes. If calculating for changes is fast enough, you can use optimisers that work from a complete (but not yet optimal) solution and make changes.

There is a free PDF/book called Clever Algorithms (Nature-Inspired Programming Recipes) covering selection choice amongst all the varied optimisers. This may allow you to find something optimal in terms of speed and reliability of algorithm.

Here's a simple thing you could try though

Create a "greedy" solution

Shuffle one set of items that need to be paired
For each item in turn, pair it with the best scoring partner
Calculate the score for this solution

Refine the solution

Sample some subset of pairs (e.g. 2, 3, 4, 5 pairs). You could do this deterministically for small enough dataset, or stochastically, or use some algorithm to filter to "at least has some chance of improving".
Find the best score amongst all permutations in this small subset (i.e. brute force all 24 pairings amongst 4 couples)
Put the best subset back into the solution
Repeat until no improvement found after some number of tests

This routine can be altered in various ways to take advantage of aspects of your problem. A nice thing in your case, making this easier that the Travelling Salesman Problem, is that you don't have any constraints on valid pairings. In TSP you care about making a single circuit, and not multiple separate loops which limits how you can make changes, whilst in your case each of your pairs is entirely separate.

One example of possible algorithm improvement: You could pre-calculate the scores for the top N matches for each person, and only search amongst those when considering local changes.


Lightweight execution of Spark MLLib models I have some training data which I am using to build a Spark MLLib model which is in a Hive database. I am using simple linear regression models and the PySpark API.
I have a code set up to train this model every day to get the most up-to-date model. (the real-world use case is that I am predicting vehicle unloading times, and my model must always be recently trained since the characteristics of the vehicles and locations change over time.)
However, when I use my model for inference I want to do it from an existing Java codebase. I need fast inference for individual data points, not batch inference. So I need a lightweight low latency way of calculating inference.
One solution that I have found is to export the parameters of my MLLib model into PMML or another representation, and then re-implement the inference code in pure Java without any of the boilerplate that would come with Spark. So I have a function like this:
    public static double predict(double[] parameters) {
        double prediction = bias + weights[0] * parameters[0] +  ....;
        return prediction;
    }

where the array weights are updated every day with values exported from a trained MLLib model.
However, this seems inefficient, since the logic of the model design is now reproduced unnecessarily in the Java code, and also restricts me to the kind of simple models that can be represented this way. For example, I could not do this for random forest regression.
Ideally, I would like a lightweight inference call to Spark MLLib within Java, without any of the overhead of Spark sessions, servers, APIs, URLs, etc.
Is there such a lightweight Spark Java function that I could use which allows the inference of a single instance? I don't imagine it being an uncommon situation that somebody needs the benefit of Hive and parallel processing for training, but just fast and simple inference with minimal overheads.

You can export the model (Weights + Hyper parameters) to a common format from Spark and then execute the same in another Language/Framework.

https://docs.databricks.com/spark/latest/mllib/model-export.html

Above link has examples for Python / Spark, should work for Java as well.


Variable importance in ensemble models I have noticed that when you make a small decision tree model, and then extend the model by creating an ensemble of trees around the same tree settings, the variable importance is diluted in the sense that the least important and most important variables become a lot more closer together. In some cases, there may be almost do distinction in importance.
Are there methods available to either mitigate this effect or to measure it, with a view to defining any tradeoff between understanding variable importance and overall accuracy?

I've been using tree-based enesembling methods such as random forests and gradient boosting for several years now, and I have to say that I've never seen that behavior.

Some packages measure variable importance solely based on trees' final splits rather than candidate/surrogate splits, so if you have two important inputs that are correlated, but one is consistently a tiny bit better than the other, the less important input might never get selected as a final split and thus look as important as some of the less valuable inputs. However, this phenomenon is independent of the number of trees/ensembling, so I don't think it fully explains the behavior you're seeing.


model with features of different sizes I want to train a model (either classification or regression, doesn't matter) with features/inputs of different sizes, but I am not sure how to do it.

For example, for each data-point, feature 1 and 2 are just real numbers and feature 3 is a vector of length 100, so if I have $n$ data-points, then I have a $n \times 1$ vector for feature 1, another $n \times 1$ vector for feature 2, and a $n \times 100$ matrix for feature 3.

The first question is what kind of model I should use or try? Gradient boosting, random forest, or neural network?

The second question is how to feed the features of different sizes to the model? Just concatenate them to form a $102 \times 1$ vector for each data-point?

Thanks.

Several options you can try, all of them have different preconditions you need to consider.

1. Direct Ensemble (with high number of n, should be the first choice)

If n >> 103 you can directly use standard ML procedure different models and parameters with CV and find the best fit. Since most of the time its not the case, you may want to use ensemble tree models since they outperform other learners when number of observations are relatively small compare to number of features.

2. Clustering

If those 100 features are sourced by a common factor, you can cluster those features with optimal K (see silhouette score) and add those cluster labels as 3rd feature on your model. Finally your model becomes n observations, 3 features.

3. Fit Twice

In case of classification, you can first fit 100 features and use their probability score (predict_proba in sklearn) as 3rd feature on your model.

4. PCA + Fit

You can use PCA to 100 features and obtain p features. After that with p+2 features you can construct your model. Again common factor condition applies in here. (In direct approach you can apply PCA to 102 features as well.)

Final Comment:

All of above methods are should be tried and the best one should be picked. However, if you have n >> 102 observations you should directly fit the model with strong learners (xgboost, lgbm, nn's etc.) without clustering/twice fitting since those methods generates error by their nature in addition to final learner error.


Covariance as inner product Why is covariance considered as inner product if there is no projection of one vector onto another?

Right now I perceive this as just a multiplication of $x$ segment of vector($x_i - \bar{x}$) and $y$ segment($y_i - \bar{y}$) of the same vector in order to understand direction of relationship.

Definition

A inner product (AKA dot product and scalar product) can be define on two vectors $\mathbf{x}$ and $\mathbf{y}$ $\in \mathcal{R^n} $ as

$$ \mathbf{x.x^T} = <\mathbf{x},\mathbf{y}>_\mathcal{R^n}=<\mathbf{y},\mathbf{x}>_\mathcal{R^n} = \sum_{i=1}^{n} x_i \times y_i $$

The inner product can be seem as the length of the projection of a vector into another and it is widely used as a similarity measure between two vectors.

Also the inner product have the following properties:

Commutative or symmetric
Distributive (over vector addition)
Bilinear
Positive-definite: i.e $\mathbf{x.x^T}  > 0,\forall \mathbf{x} $

The covariance of two random variables $X$ and $Y$ can be defined as

$$ E[(X-E[X]) \times (Y - E[Y])] $$

the covariance holds the properties of been commutative, bilinear and positive-definite.

These properties imply that the covariance is an Inner Product in a vector space, more specifically the Quotient Space.

Association with the kernel trick

If you are familiar with Support Vector Machines you probably familiar with the Kernel Trick where you implicitly compute the inner product of two vectors into a mapped space, called feature space. Without performing the mapping you can compute the inner product into even a possibly infinite dimensional space given that this mapping.

To perform that inner product, you need to find a function, known as kernel functions, that can perform this inner product without explicitly mapping the vectors.

For a kernel function to exist it needs to have the following atributes:

It needs to be symmetric
It needs to be positive-definite

That is sufficient and necessary to a function $\kappa(\mathbf{x,y})$ to be considered a inner product in an arbitrary vector space $\mathcal{H}$.

As the covariance, comply to this definition it is a Kernel Function and consequentially it is an Inner Product in a Vector Space.


Logistic Regression Maximum Likelihood Is it true that we assume our P(y|x;theta) to follow Bernoulli's distribution given y has binary output in Logistic Regression? Is there any specific reason why we consider Bernoulli's distribution?

If 1) is true., What happens if we consider P(y|x; theta ) as Gaussian distribution? What would be our cost function? I mean if we assume Gaussian, we can have optimal value of mean and variance by taking mean of (y|x) and its variance which is what we are maximizing the likelihood for.

A Bernoulli distribution is a probability distribution where a random variable takes the value 1 with probability p and the value 0 with probability 1-p. This distribution maps very well onto logistic regression because the random variable is the target variable and the model is estimating the probability of 0 or 1 given the specific values of the features.
The Gaussian distribution is not as well suited because it is a continuous probability distribution that is goes from negative infinity to positive infinity. The result would be predictions outside the bounds of the problem space.
The cost function is something different. The cost function is how errors in prediction are penalized.


How did the author got this final result from this Gaussian Distribution formula? How did they got the final result?

Take natural logarithm of the expression. Logarithm of product of two terms can be written as sum of logarithms of each term: log(a.b) = loga + logb. e and natural logarithm cancel each other, there is left first term in the result. Logarithm of a 1/number is equal to -log(number), it gives second term in the result.
Taking logarithms of exponentials is quite common method for preventing numerical overflows and ensuring numerical stability.


Regression problem with Deep Learning I'm working on Housing Price dataset, where target is to predict the housing price.
Price of the house will always be positive and according to me, it's possible that model can predict a negative outcome for some of the samples.

If it's correct, is there any way to control the training such that
model always predict at least the positive value.

As in case of classification case we use Sigmoid/Softmax activation
function to normalized the outcome in probability. Can we have some
activation function for positive value?

Can I use Poisson loss?

The ability of the model to predict negative value for the housing price depends on the data. On the large amount of data, where there are no negative pricing, the model does not predict a negative number. However, in rare case, where the model is not trained well or has not seen such samples, then it is still possible.

The models prediction on the positive value can be still controlled post predictions. Just like using a treshold. y = y if y>0 else 0; Where the housing cost (y) is as it is if it's positive, 0 otherwise.

ReLu function, works in the way you desire. Negative values gets converted to 0 by the activation.

I am not very sure about the Poisson  loss, you may try it.


Tensorflow.Keras: How to get gradient for an output class w.r.t a given input? I have implemented and trained a sequential model using tf.keras. Say I am given an input array of size 8X8 and an output [0,1,0,...(rest all 0)].
How to calculate the gradient of the input w.r.t to the given output?
model = ...
output = np.asarray([0, 1, 0, 0, 0, 0, 0, 0, 0, 0])
input = np.random.randn(1,64)
pred = model.predict(input)

gradient = somefunction(model,input,output,pred)

Is there any way to achieve that?
I am trying to implement a project similar to Google Deep Dreams, so that given a random picture, if I mention a digit, then with each iteration I will update the input with its gradient to make the picture more like the digit that was asked.
I tried to follow other StackOverflow answers and Keras documentation. But as they are not mentioning the input shape, when someone writes input[:,:,something] I am having a hard time translating it to my requirement.
Can you provide a simple generic answer so that given an array as input and an expected output array, one can calculate the gradient of loss w.r.t the input for that exact output.

You can get that from the weight updates, not sure if it is the best approach
-Save the model
-Save the weights of the first layer
-Load the model and Compile the model with SGD w/o momentum
-Set all the weights = that of the previous model
-Train with the input and output i.e. the Array for epoch=1 and batch_size=1
-Get the weights again
-Calculate the respective gradient using two weights of 1st layer
-If Gradient is very small, it might become zero due to matching digits of both the weights
w0 = model.get_weights().copy()
w0_0 = w0[0]

optimizer = tf.keras.optimizers.SGD( learning_rate=0.01, momentum=0.0, nesterov=False, name="SGD")
model.compile(optimizer=optimizer, loss='mse')
model.fit(input, output, epochs=1, verbose=1, batch_size=1)

w0_new = model.get_weights().copy()
w0_0_new = w0_new[0]

'''
w_new = w0 - learning_rate * g
g = (w0 - w_new)/learning_rate
'''
grad = (w0_0 - w0_0_new)/0.01


I do feature engineering on the full dataset, is this wrong? I am aiming to predict the number of days it takes to sell a given property, let's call this variable "DaysForSale" - in short DfS
Using the DfS I created a variable called "median_dfs_grouped_street_name" which returns the median days it takes to sell a property for the different streets available in the dataset. (The street names are all categorized).
After this, I do my train/test split and run my Random Forest method.
Using the feature_imporatances function I see that the new feature is the second most important, which makes me wonder if this is the correct approach?
I have two questions:

Is it wrong to develop features using the target variable?
Is it wrong to do feature engineering on the full dataset?

Is it wrong to develop features using the target variable?

Not necessarily. It is called "target encoding" or "Mean encoding" and can be very useful. In your case you could, for example, use the DfS of your train data to calculate a median value per street. But you need to carefully design the target encoding to avoid overfitting (there are different strategies to do that - see below link). And for the test data you can only use the target encoding based on your train data.
The Coursera course "How to Win a Data Science Competition: Learn from Top Kagglers" has great content on target/mean encoding to be found here.

Is it wrong to do feature engineering on the full dataset?

Not necessarily. As pointed out in Nicolas' answer you need to be careful to not leak data though.
Here's an example where it would be ok: let's assume one of your features is date of enlisting which is the date when the property was published for sale. You could, for example, add a feature to the whole dataset called days since enlisting which simply calculates the days between now and when the property was published for sale. However, your median is an example which results in data leakage since it is not "per row" data engineering but "across rows" data engineering applied to train and test data.
That's why the safer approach is to first split the data, remove the target variable from the val/test data and then do feature engineering. Thereby, you avoid any unintended data leakage.


Keras loaded model output is different from the training model output When I train my model it has a two-dimension output - it is (none, 1) - corresponding to the time series I'm trying to predict. But whenever I load the saved model in order to make predictions, it has a three-dimensional output - (none, 40, 1) - where 40 corresponds to the n_steps required to fit the conv1D network. What is wrong?

Here is the code:

df = np.load('Principal.npy')

    # Conv1D
#model = load_model('ModeloConv1D.h5')
model = autoencoder_conv1D((2, 20, 17), n_steps=40)

model.load_weights('weights_35067.hdf5')

# summarize model.
model.summary()

    # load dataset
df = df

    # split into input (X) and output (Y) variables
X = f.separar_interface(df, n_steps=40)
# THE X INPUT SHAPE (59891, 17) length and attributes, respectively ##

# conv1D input format
X = X.reshape(X.shape[0], 2, 20, X.shape[2])

# Make predictions

test_predictions = model.predict(X)
## test_predictions.shape =  (59891, 40, 1)

test_predictions = model.predict(X).flatten()
##test_predictions.shape = (2395640, 1)

plt.figure(3)
plt.plot(test_predictions)
plt.legend('Prediction')
plt.show()

In the plot below you can see that it is plotting the input format.

Here is the network architecture:

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
time_distributed_70 (TimeDis (None, 1, 31, 24)         4104
_________________________________________________________________
time_distributed_71 (TimeDis (None, 1, 4, 24)          0
_________________________________________________________________
time_distributed_72 (TimeDis (None, 1, 4, 48)          9264
_________________________________________________________________
time_distributed_73 (TimeDis (None, 1, 1, 48)          0
_________________________________________________________________
time_distributed_74 (TimeDis (None, 1, 1, 64)          12352
_________________________________________________________________
time_distributed_75 (TimeDis (None, 1, 1, 64)          0
_________________________________________________________________
time_distributed_76 (TimeDis (None, 1, 64)             0
_________________________________________________________________
lstm_17 (LSTM)               (None, 100)               66000
_________________________________________________________________
repeat_vector_9 (RepeatVecto (None, 40, 100)           0
_________________________________________________________________
lstm_18 (LSTM)               (None, 40, 100)           80400
_________________________________________________________________
time_distributed_77 (TimeDis (None, 40, 1024)          103424
_________________________________________________________________
dropout_9 (Dropout)          (None, 40, 1024)          0
_________________________________________________________________
dense_18 (Dense)             (None, 40, 1)             1025
=================================================================

The problem was due to the output format. Instead of outputting one value per input sequence, in order to produce a complete and unique sequence as output, it was outputting sequences. Though instead of having a unique sequence as result it got a sequence of sequences. The solution was to fix the network output shape and data shape as well.


Filter phrases based on correspondent POS tags I have two lists:
The first one includes sentences and the second one includes parts of speech (POS) tags. I want to use POS tags as a filter and find phrases according to a pattern such as 'JJ JJ NNS' (adjective+adjective+noun)

How can I do that?

import re

key_list= [' techniques from nonlinear analysis and partial differential equations form the basis for these studies .',
 ' in this case , center-manifold theory can be applied to study the ordinary differential equations which replace the nonlinear \
partial differential equations for the wave motion .',
' differential equations are cool .']

value_list = ['NNS IN JJ NN CC JJ JJ NNS VBP DT NN IN DT NNS .',
 ' IN DT NN , JJ NN MD VB VBN TO VB DT JJ NN NNS WDT VB DT JJ JJ JJ NNS IN DT NN NN .',
 ' JJ NNS VBP JJ .']

pattern= re.compile('JJ JJ NNS',re.M|re.DOTALL)

for value in value_list:
    value_output = pattern.findall(value)
    print(value_output)

# I want  partial differential equations and ordinary differential equations

Hey,

Here's how i would solve this.
The problem with regex is that you don't have any index so I find it better to convert your inputs into lists of single words/tags by splitting on spaces. Then it is easier to find the pattern in the tag list and get the matching words.

words = [word for key in key_list for word in key.split()]
tags = [tag for value in value_list for tag in value.split()]

pattern = ['JJ', 'JJ', 'NNS']
matches = []

for i in range(len(tags)):
    if tags[i:i+len(pattern)] == pattern:
        matches.append(words[i:i+len(pattern)])

matches = [' '.join(match) for match in matches]
print(matches)

# result : ['partial differential equations', 'partial differential equations']

Also, if I'm not mistaken your pos-tagger marked "differential" as a noun (NN) in "ordinary differential equations". Nltk tagger does the job and it is really fast to implement but it is not perfect.

You can try using another tagger, I personally use the Stanford one for nlp processing.


Do I need to convert strings before using LSTM? I have a dataset which includes one column with a URL and another column with value 0 or 1 indicating if it is a phishing link.

I want to process this dataset using LSTM. Do I need to first convert the string column into some other data type?

The model I'm hoping to use would be something like below:

# Model building using the Sequential API
model3 = Sequential()
model3.add(Dense(10, activation='relu',
          kernel_initializer='uniform',input_dim=x3.shape[1]))

# Add a LSTM layer with 128 internal units.
model3.add(layers.LSTM(128))

# Add a Dense layer with 10 units.
model3.add(layers.Dense(10))

You can use one-hot encoding to encode your domains as char arrays. Then your training samples should have dimension (samples, longest domain length, all chars used). Here is a code sample:

import pandas as pd
import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, LSTM

# make some fake samples
urls = ['https://datascience.stackexchange.com/',
        'https://github.com/',
        'www.google.com/']

labels = [1,0,1]
df = pd.DataFrame(zip(urls, labels),
                  columns=['domain', 'label'])

# Make it all to a long string
concat_domains = '\n'.join(df['domain']).lower()

# Find all unique characters by using set()
chars = sorted(list(set(concat_domains)))
num_chars = len(chars)

# Build translation dictionaries, 'a' -> 0, 0 -> 'a'
char2idx = dict((c, i) for i, c in enumerate(chars))
idx2char = dict((i, c) for i, c in enumerate(chars))

# Use longest name length as our sequence window
max_sequence_length = max([len(name) for name in df['domain']])

# build dataset with domains as one-hot encoded chars
X = np.zeros((df.shape[0], max_sequence_length, num_chars), dtype=np.bool)
y = df['label'].values

for i, sequence in enumerate(df['domain']):
    for j, char in enumerate(sequence):
        X[i, j, char2idx[char]] = 1

# build a model with input dim: (length of longest domain name, number of unique chars found)
model = Sequential()
model.add(LSTM(64, input_shape=(max_sequence_length, num_chars)))
model.add(Dense(units=1, activation='sigmoid'))

model.compile(loss='binary_crossentropy', optimizer='adam')

# train for 10 epochs
model.fit(X, y, epochs=10)


How can I output tokens from MWE Tokenizer? How to output the tokens produced using MWE Tokenizer?

NLTK's multi-word expression tokenizer (MWETokenizer) provides a method/function add_mwe() that allows the user to enter multiple word expressions prior to using the tokenizer on the text.

Currently, I have a file consisting of phrases / multi-word expression I want to use with the tokenizer.  My concern is that the manner in which I am presenting the phrases to the function correctly and so not resulting in the desired set of tokens to be used in tokenizing the incoming text.

So this leads me to ask if anyone knows how to output the token generated by add_mwe() so that I can verify that I am correctly passing the phrase to the function?

You can check the exact input and output parameters of the add_mwe method in NLTK's documentation for the class here.

This is the expected input:

>>> tokenizer.add_mwe(('in', 'spite', 'of'))

So, each phrase must simply be a tuple with the words in that phrase. If you provide that input, you should get the output you expect (in_spite_of). I've added a full snippet of working code below for convenience, there you can see how to use the class as intended.

Regarding the output of add_mwe, every time you call the method it adds a new word to the dictionary,  and all the words are stored in the class's _mwes attribute. So, given mwe = MWETokenizer(), you can then inspect the contents of mwe (with e.g. print mwe._mwes) to see what the class actually stores.

As stated in the documentation, it is actually a Trie with all the terms, so it won't look exactly as the words you added (it is a more efficient representation thereof). The link I mentioned earlier has more details on that.

Hope this helps!

import nltk

from nltk import (
    sent_tokenize as splitter,
    wordpunct_tokenize as tokenizer
)

from nltk.tokenize.mwe import MWETokenizer

test = """Anyone know how to output the tokens produced using MWE Tokenizer?

For a clearer explanation of what I am asking for those who did not understand my original brief question.

The multi-word expression tokenizer (MWETokenizer) provides a method/function (add_mwe()) that allows the user to enter multiple word expressions prior to using the tokenizer on text. Currently I have a file consisting of phrases / multi-word expression I want to use with the tokenizer. My concern is that the manner in which I am presenting the phrases to the function correctly and so not resulting in the desired set of tokens to be used in tokenizing the incoming text. So this leads me to ask if anyone knows how to output the token generated by this method/function so that I can verify that I am correctly passing the phrase to the function (add_mwe()).?"""

mwe = MWETokenizer()

phrases = [
    ('multi', '-', 'word'),
    ('expression', 'tokenizer'),
    ('word', 'expressions'),
    ('multi', '-', 'word', 'expression')
]

for phrase in phrases:
    mwe.add_mwe(phrase)

for sent in splitter(test):
    tokens = tokenizer(sent)
    print ' '.join(tokens)
    print ' '.join(mwe.tokenize(tokens))
    print '---'

# Expected output:
#
# Anyone know how to output the tokens produced using MWE Tokenizer ?
# Anyone know how to output the tokens produced using MWE Tokenizer ?
# ---
# For a clearer explanation of what I am asking for those who did not understand my original brief question .
# For a clearer explanation of what I am asking for those who did not understand my original brief question .
# ---
# The multi - word expression tokenizer ( MWETokenizer ) provides a method / function ( add_mwe ()) that allows the user to enter multiple word expressions prior to using the tokenizer on text .
# The multi_-_word_expression tokenizer ( MWETokenizer ) provides a method / function ( add_mwe ()) that allows the user to enter multiple word_expressions prior to using the tokenizer on text .
# ---
# Currently I have a file consisting of phrases / multi - word expression I want to use with the tokenizer .
# Currently I have a file consisting of phrases / multi_-_word_expression I want to use with the tokenizer .
# ---
# ...


Text mining of big data using R Server I have a big table (150 Million rows and ~ 70 columns).
In three of the columns in the table I have text input (3-20 words/column), which I need to use for a classification algorithm.

For smaller datasets, I have used the tm R package and created a DocumentTermMatrix, where I used the frequency of word (or word parts) as predictors in a SVM & Decision Forest algorithm.

However, now my dataset is much bigger, which is why I converted it to a xdf file and used RevoScaleR packages for merging, joining etc. so far.

However, I am unsure how to do text mining with very large datasets (e.g., create a document term matrix which I can use for classification). I did not find any pre-build functions. It may be possible to create a document term matrix in a chunk of data, and then somehow sum the frequencies. I am not sure if that is the best way, could you help me with that?

Thank you,
Lisa

Thank you for your reply.
By accident I found out that MicrosoftML (https://msdn.microsoft.com/en-us/microsoft-r/microsoftml-introduction) offers just what I need - by using the featurizeText functions offered within Microsoft R client and R server.
Maybe it helps others - I found their example  and could it translate to my data (featurizeText() help under Microsoft ML).

trainReviews <- data.frame(review = c(
        "This is great",
        "I hate it",
        "Love it",
        "Do not like it",
        "Really like it",
        "I hate it",
        "I like it a lot",
        "I kind of hate it",
        "I do like it",
        "I really hate it",
        "It is very good",
        "I hate it a bunch",
        "I love it a bunch",
        "I hate it",
        "I like it very much",
        "I hate it very much.",
        "I really do love it",
        "I really do hate it",
        "Love it!",
        "Hate it!",
        "I love it",
        "I hate it",
        "I love it",
        "I hate it",
        "I love it"),
     like = c(TRUE, FALSE, TRUE, FALSE, TRUE,
        FALSE, TRUE, FALSE, TRUE, FALSE, TRUE, FALSE, TRUE,
        FALSE, TRUE, FALSE, TRUE, FALSE, TRUE, FALSE, TRUE,
        FALSE, TRUE, FALSE, TRUE), stringsAsFactors = FALSE
    )

    testReviews <- data.frame(review = c(
        "This is great",
        "I hate it",
        "Love it",
        "Really like it",
        "I hate it",
        "I like it a lot",
        "I love it",
        "I do like it",
        "I really hate it",
        "I love it"), stringsAsFactors = FALSE)

outModel <- rxLogisticRegression(like ~ reviewTran, data = trainReviews,
    mlTransforms = list(featurizeText(vars = c(reviewTran = "review"),
    stopwordsRemover = stopwordsDefault(), keepPunctuations = FALSE)))
# 'hate' and 'love' have non-zero weights summary(outModel)


Classification for High dimensional data I am trying to build a classification model with 1000+ features and
I am performing the below steps but I am not sure if it's the correct way to do it

Step 1. Finding Variable Importance using the entire dataset

Step 2. Build a model using the subset of the features returned from Step 1
Once I get my new data I am splitting the data in training and testing and then performing cross validation on training data.

My question is should I use CV while performing variable importance on the entire dataset?

Thanks for your help!

Before answering your question I would ask if you actually need so many variables or you can apply some dimensionality reduction technique (i.e. PCA)? It could boost your training speed and maybe result in a better model efficiency.

Lets suppose for some reason (i.e. model explainability) you need the exact variables. You can still remove some of them without using feature importance (i.e. removing highly correlated ones, but there are other alternatives too). Indeed using only feature importance to remove variables can be dangerous. Imagine there is an important factor that highly influence your model but it shows up in several variables. In this case it can happen that the importance of this factor will be distributed among those variables. You will experience maybe that those variables are less important and remove them all thereby also eliminating an otherwise important factor.

To answer your question. I would before step 1 take away my test set and only use the training set for any kind of modeling activity. It's a good practice to guarantee that your test set has never been seen by the model and it did not influence it in any way.

In this case you can just remove less important features and measure the final results on the test set, it's not necessary to do CV.

On the other hand you will probably face with the challenge how to define your threshold for eliminating variables based on feature importance.You can try to do some gridsearch on it. In this case you should avoid measuring the results of different models on the test set or you risk over-fitting on it. In this case using CV can be a good solution to define your threshold.


Handling outliers and Null values in Decision tree Outliers : As I understand, decision trees are robust to outliers. Can anybody please confirm if my hypothesis is right with an example?
(What if I have a features ranging from 0 to 9 but there is an outlier of which value is 10000?)
Whether it creates a separate leaf for that outlier sample or would it be merged with some other tree's leaves?

NULL Values : Do we need to replace the null values before building the model using Decision tree or it would be taken care automatically by the decision tree technique?

Thank you.

Outliers: In decision tree learning, you do splits based on a metric that depends on the proportions of the classes on the left and right leaves after the split (for instance, Giny Impurity). If there are few outliers (which should be the case: if not, you cannot use any model), then they will not be relevant to these proportions. For this reason, decision trees are robust to outliers.

Null values: You have to replace them (unless the software you use already does that for you, which is not generally the case).

Edit about outliers:

What I have said in outliers is only about classification trees. However, it is certainly not true in regression tress. Regression tree split criterium depends on the averages of the two groups that are splitted, and, as the average is severly affected  by outliers, then the regression tree will suffer from outliers. There are two main approaches to solve this problem: either remove the outliers or build your own decision tree algorithm that makes splits based on the median instead of the average, as the median is not affected by outliers. However, basing the tree algorithm on the median will be very slow, as computing the median is way slower than computing the average.


Partitioning data into features/labels and train/test after reading from csv file I need to read data from csv file and then first partition that data into features and labels and then into training and testing set. However, there are several issues cropping up again and again. Below is the code I tried with error ,

ValueError: could not convert string to float: 'mon'
on line
Y: train_y})

The code for Linear Regression :-

import pandas as pd
from sklearn.model_selection import train_test_split
import tensorflow as tf
import numpy as np

learning_rate = 0.01
training_epochs = 1000
display_step = 50

data = pd.read_csv('forestfires.csv')
y = data.temp
x = data.drop('temp', axis=1)

train_x, test_x, train_y, test_y = train_test_split(x, y,test_size=0.2)
n_samples = train_x.shape[0]
n_features = train_x.shape[1]

X = tf.placeholder('float', [None, n_features])
Y = tf.placeholder('float', [None, 1])

# Model weights.
W = tf.Variable(np.random.randn(n_features, 1), dtype='float32')
b = tf.Variable(np.random.randn(1), dtype='float32')

# Construct linear model.
prediction = tf.matmul(X, W) + b
loss = tf.reduce_sum(tf.pow(prediction - Y, 2))/(2 * n_samples)
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)

# Start training.
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    for epoch in range(training_epochs):
        for (x, y) in zip(train_x, train_y):
            sess.run(optimizer, feed_dict={X: train_x,
                                           Y: train_y})
            # Display logs per epoch step.
            if (epoch + 1) % display_step == 0:
                c = sess.run(loss, feed_dict={X: train_x,
                                              Y: train_y})
                print ('Epoch:', '%04d' % (epoch+1), 'cost=','{:.9f}'.format(c), \
                       'W=', sess.run(W), 'b=', sess.run(b))
    print ('Training Done!')
    training_cost = sess.run(loss, feed_dict={X: train_x,
                                              Y: train_y})
    print ('Training cost=', training_cost, 'W=', sess.run(W), 'b=', sess.run(b), '\n')
    # Graphic display.
    plt.plot(train_x, train_y, 'ro', label='Original data')
    plt.plot(train_x, sess.run(W) * train_x + sess.run(b), label='Fitted line')
    plt.legend()
    plt.show()

Could anyone help me with reading data properly in a rather general way. Snapshot of the data :-

I don't know exactly how your data is but y = data.temp may be a Series containing the string values which should be cast to float values. Try to change it to the following alternative.

  y = data.temp.astype(float)


How to retrain the neural network when new data comes in? I am new to deep learning. Can anybody help me with the online learning implimentation for deep learning models. As per my understanding, i can save a keras/tensorflow model after training and when new data comes in, i can reload the network back and retrain the network using the new data. I haven’t seen anywhere documenting this method. Is my understanding incorrect? If yes, let me know what can be done so that the model keeps on getting retrained when new data comes in?

Yes, there is some methods to do this based on the library. The proper keyword for this functionality is Checkpoint. Hence, you can check functions to save or restore the checkpoints from hard disk to continue learning or prediction in a new session. For example you can see the following in Tensorflow:

  The tf.train.Saver class provides methods for saving and restoring models. The tf.train.Saver constructor adds save and restore ops to the graph for all, or a specified list, of the variables in the graph. The Saver object provides methods to run these ops, specifying paths for the checkpoint files to write to or read from.

For more details about this in Tensorflow, you can follow this link.

To complete the case you can follow this subject in another library such as CNTK in this link.


Infer family type, size from reviews I have a bunch of reviews:

User_id, review
1, "We (a family of 4 adults) chose this and view and loved this place"
1, "My husband and I, with our 2 teen sons, visit this restaurant at least once..."
2,"My partner and I booked table for a short holiday, their wine menu was awesome"
2,"My wife is a fan of jazz and she's expecting, so visited this place "

What techniques/packages are available to, for instance, estimate that:

User Id 1 => family of 4, 2 sons (13-19)
User Id 2 => family of 2, expecting
:
:

I have been googling around, to little help, and other than creating my own labeled dataset, I was hoping there are some NLP techniques that can help bootstrap my training set, which can then be curated by humans.

I can think of two options:

Train a custom supervised tagger for your data, typically with a sequence labeling method such as CRF. This will require a quite large amount of annotated data (and a specific formatting), but if done well it should give you quite accurate results.
Use manually defined patterns based on keywords (such as "My * and I", "family", "friends") directly associated with a predefined category, and match these patterns in the data. You can do several iterations of defining new patterns for cases that are not matched, thus refining progressively. Depending on your data and how far you go with this you should be able to correctly match most cases, possibly reaching as good accuracy as a tagger for much less work.

Btw be careful how you represent your data: the same user id might not always give you the same group of people since they can go to a restaurant one day with their family, the next day with their wife, the day after with their mistress, etc. Also a "family of 2, expecting" usually becomes "a family of 3" after 9 months ;)


Text extraction from large pool of documents of different formats I have a collection of 6 million documents stored on a hard drive (around 500GB of data storage). Those documents contain text, tables, images and come in different formats: pdf, jpg, png, rar, vsd, xlsx, docx, and other Microsoft Office file types.

Some documents contain digital text (90%), other documents are scanned copies (10%). Some documents are in english (5%), some are in russian (5%) and rest of them is mix of english and russian (containing translations with similar meaning). Also some documents are packed into zip or rar archives.

What is the best approach to parse all this documents for text data?

What is the best approach to store extracted data?

How to make this data searchable? E.g. if I want to list top 30 documents that contain text similar to my search paragraph.

PS: I am interested to do this task in the shortest time possible, so I guess possible solution would involve big data techniques and distributed computing.

One option is to use Apache SOLR + Apache TIKA.
Apache TIKA has support for most common file formats, it extracts test content from files. Extracted text can be stored in SOLR.  SOLR supports various kinds of text + aggregation queries.
Tutorials :
https://blog.webnersolutions.com/apache-solr-indexing-all-files-in-a-folder-recursively
https://lucene.apache.org/solr/guide/6_6/introduction-to-solr-indexing.html
https://lucene.apache.org/solr/guide/6_6/uploading-data-with-solr-cell-using-apache-tika.html


What pre processing should I use on data to feed into a CNN? I have a dataset of shape 105 x 501 x 266 where 105 is the number of data and 501 x 266 is the shape of 1 data i.e. The labels_dataset is of shape 105 x 1.

Each value of the 501 x 266 matrix is a complex number.

So it essentially becomes 501 * 266 * 2(real and imaginary part of the number)

And now I have to feed this data to a CNN.
I 'm new to training networks. So need to know whether my data is in best possible form for the CNN or not.

I've printed out max, min, sd, mean of real part, imaginary part and magnitude of the dataset for more info:

max real = 0.186396, min real = -0.204375
max imag = 0.166608, min imag = -0.159017
max abs = 0.219019, min abs = 2.33527e-10
mean real = 4.01718e-10, complex = 6.79294e-15, abs = 8.82916e-05
std dev real = 0.000442753, complex = 0.000400677, abs = 0.000590573

Is this a good form of data for input to a CNN. What are the options to make it more suitable?

You should normalize your input data for different purposes. As you can read from here, Normalizing data is done for accelerating optimization. If you have features with different scales, it will take too much time for your optimizer function to find optimal points. Suppose you have age feature which can change between 0 to 150 (!) and salary which can be changed from 0 to whatever, like 500,000,000 $. your optimization algorithm used in your ML model will take too much time, if possible, to find appropriate weights for each feature. Moreover, if you don't scale your data, your ML algorithm may take too much care to features with large scales.

You have not explicitly specified your task, if its classification, you may need to turn the labels into one-hot encoded versions or if possible you may need to use word embeding for adding logical distance to different classes. If your task is a regression one, you may need to normalize your outputs. In regression problems it is customary to normalize the output too, because the scale of output and input features may differ.


Best way to gather musical data I am looking for a relatively large amount of jazz (and other) solos to use for data science purposes - probably somewhere up to a 100 transctipts. Although that is not that much and many pdf transcripts are available, I want the data to be in a format where I can easily access all the information in Python (which note was played and at what beat). Manually writing all that data would, however, be very tedious and time-consuming.

Thus my question: does anyone know of any software that automatically reads pdf transcripts and puts it in a different file format, or alternatively of a website with transcripts already in a format in which I can access all the data easily from Python?

I think you are looking for musical optical character recognition software. A google search pointed me to Wikipedia where links to two open source applications resides: Audiveris and OpenOMR. I can not vouch for either program.


Removing constant from the regression model I am trying to calibrate two variables $(X,Y)$ of different measuring techniques from two instruments, the result of the linear regression analysis appears as shown in the image.
The result shows the regression constant is not statistically significant but the model is significant. I have tried to remove the regression constant (it is a very small value close to zero) and $R$ of the new model is raised to 90%. Is it correct to remove the regression constant?

When you estimate a linear model without constant, you essentially "force" the estimated function to go through the $(0,0)$ coordinates.
With an intercept, you estimate a linear function like:
$$ y = \beta_0 + \beta_1 x .$$
Without intercept, you estimate a linear function like:
$$ y = 0 + \beta_1 x .$$
So when $x=0$, $y$ will be $0$ as well.
You should not only look at $R^2$ since $R^2$ often will go up when you have no intercept. Think about the structure of your model, how the data look like, and what you want to achieve.
Example in R:
library(ISLR)
auto = ISLR::Auto

ols1 = lm(mpg~horsepower,data=auto)
summary(ols1)
plot(auto$horsepower, auto$mpg)
lines(auto$horsepower, predict(ols1, newdata=auto), type="l", col="red")

ols2 = lm(mpg~horsepower+0,data=auto)
summary(ols2)
plot(auto$horsepower, auto$mpg)
lines(auto$horsepower, predict(ols2, newdata=auto), type="l", col="red")

Results:
Model with intercept:
Coefficients:
             Estimate Std. Error t value Pr(>|t|)
(Intercept) 39.935861   0.717499   55.66   <2e-16 ***
horsepower  -0.157845   0.006446  -24.49   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 4.906 on 390 degrees of freedom
Multiple R-squared:  0.6059,    Adjusted R-squared:  0.6049
F-statistic: 599.7 on 1 and 390 DF,  p-value: < 2.2e-16

Model without intercept:
Coefficients:
           Estimate Std. Error t value Pr(>|t|)
horsepower 0.178840   0.006648    26.9   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 14.65 on 391 degrees of freedom
Multiple R-squared:  0.6492,    Adjusted R-squared:  0.6483
F-statistic: 723.7 on 1 and 391 DF,  p-value: < 2.2e-16

Summary:
In this example, excluding the intercept improved the $R^2$ but by forcing the (estimated) function to go through $(0,0)$, the model results are entirely different. In essence, the model without intercept produces bullshit in this case. So be very careful to exclude the intercept term.


How to handle missing date data? I have a column named GarageYrBlt which just lists the year the garage of that house was built.  I have one nan value for this column.  Does it make sense to fill it with the columns median (This was a common approach for the missing age values in the Titanic competition)?

I suspect you're working with the Ames house price dataset - one of Kaggle's introductory competitions.

Replacing the missing values with the dataset mean / median is very general.

I believe you have a variable which describes which neighbourhood the house is in. A more focussed approach would be to replace that missing value in GarageYrBlt with the mean / median for that neighbourhood (subject to additional sensibility checks like the garage not being built before the house).


Why is ElasticNet performs worse than both Lasso and Ridge? I am using the following codes to build a few models on the same dataset:

X_train, X_test, y_train, y_test = train_test_split(X_in, y, test_size=0.25, random_state=42)

# Lasso regression
lasso = linear_model.Lasso()
lasso.fit(X_train, y_train)
pred_lasso = lasso.predict(X_test)

# Ridge regression
ridge = linear_model.Ridge()
ridge.fit(X_train, y_train)
pred_ridge = ridge.predict(X_test)

# ElasticNet
elastic = linear_model.ElasticNet()
elastic.fit(X_train, y_train)
pred_elastic = elastic.predict(X_test)

# R^2 Evaluation
print('R^2 for Lasso', r2_score(y_test, pred_lasso))
print('R^2 for Ridge', r2_score(y_test, pred_ridge))
print('R^2 for ElasticNet', r2_score(y_test, pred_elastic))
print('\n')

And the r2_score for the 3 models are:

R^2 for Lasso 0.28
R^2 for Ridge 0.14
R^2 for ElasticNet 0.02

This is confusing to me ... shouldn't the ElasticNet result fall somewhere between Lasso and Ridge? Why is ElasticNet result actually worse than the other two? Thanks!

The ElasticNet model is not being tuned. By default in scikit-learn, ElasticNet's l1_ratio parameter, the mixture of L1 and L2 penalty, is set to .5. A .5 l1_ratio represents an even mixture of L1 and L2 penalty and does not fit the data very well. Best practice is a cross-validation grid search for the optimal value of l1_ratio.


How to predict weather? I want to know how does one predict day to day weathers. Like what are the factors that must be known to predict day to day weathers using a NN. Few factors I could think of are:

Humidity
Weather history
Wind speed

What are some factors that a meteorology ignorant data scientist would require to correctly predict the weather?

To ‘correctly’ predict the weather would require a clairvoyance that our best meteorologists have yet to divine.  In other words, they’re often quite wrong.

Weather prediction is very much a data limited problem. If you had access to each and every variable that contributes, you’d do a great job. The problem is one only has access to a subset, and the mechanics are still not fully understood. I’m not a meteorologist but I also suspect the sensors we have are sparse. I’d want temp, humidity, wind speed, and barometric pressure at 100 meter intervals both parallel and perpendicular to the earths surface, plus historical data, rate of change, ocean temp and currents, and the airspeed velocity of every unladen swallow in the area.


Modifying a distribution by adding in samples incrementally I would like to calculate the distribution (e.g., Gaussian) of a set of samples.  However, I would also like to see how the distribution changes as I fit the samples into the distribution incrementally.

One way to do this would be to compute the distribution over all relevant samples every increment (e.g., first increment: calculate distribution of 2 samples, second increment: calculate distribution of 3 samples).  However, this is computationally intensive.

Would I be able to calculate the distribution of 3 samples from solely the 3rd sample and the properties of the distribution of 2 previous samples?

For example, say I have 5 ordered samples.

I start off by calculating the mean and standard deviation of the first 2 samples This is the 1st Gaussian distribution.
I then look at the third sample, and fit it into the first Gaussian distribution (knowing the mean, std, number of samples).  This is the 2nd Gaussian distribution.
I then look at the fourth sample, and fit it into the 2nd Gaussian distribution (knowing the mean, std, number of samples).  This is the 3rd Gaussian distribution.

Calculation of standard deviation on the fly is possible (turn to our brothers at math.stackexchange):

https://math.stackexchange.com/questions/198336/how-to-calculate-standard-deviation-with-streaming-inputs

It is easier to keep track of the variance and only take the square root to calculate the stdev when you really need it.

And the mean is even easier, don't overthink it. If you have 4 samples with a mean of 4 and 3 samples with a mean of 3, this totals up to 25 in 7 samples.

So

$$\mu = \frac{\mu_1 N_1 + \mu_2 N_2 + \mu_3 N_3}{N_1 + N_2 + N_3}$$

and

$$\sigma^2 = \frac{\sigma_1^2 N_1 + \sigma_2^2 N_2 + \sigma_3^2 N_3}{(N_1 + N_2 + N_3)^2}$$


How to combine two CART decision trees learned in same type of data? We have distributed data centers and we build decision trees in each data center. Our problem is to combine our CART decision trees into one CART decision tree. The data in each data center related to the same event (data from light sensor for instance). I know about boosting methods but they don't give result we want to.

Is there any known method to do this ?

You mention two decision trees. Traversing a decision tree is very cheap, so running a feature instance through multiple trees is very fast, you could just take all the decision trees from the data centers and average the outcome, maybe weigh it by the (crossvalidated) strength of the models. Random Forests are powerful models that also combine decision trees in this way, except that they are done on random subsets of the features (and in some cases also random subsets of the data).


Which supervised learning algorithm to classify customers buying items I am very new to machine learning, and it's hard for me to know in what direction I should go. Thanks in advance for your help.

I have a set of data showing what customers bought:

A customer may have bought 1 or more items
Each item is discribed by :

A quality level which is an integer between 1 and 7
Its content from a first point of view (the item has x1% of content 1, x2% of content 2, ...)
Its content from a second point of view (the item has xA% of content A, xB% of content B, ...)

A customer says that he belongs to the group G1, G2 or G3

I would like to be able to guess the group a new customer belongs to, given the items he bought.

What algorithm should I ideally use ?
What algorithm would you recommend if I can only use numpy, without scipy or scikit (it's on a server which only has numpy) ?

Thank you very much

Edit : I'm adding the appropriate words to the title and to the tags

As per the problem statement statement it does look like 3 class classification problem.
But I don't think you can use any ML algorithms using Python without sklearn package in it. Try getting it installed and you can implement the problem using Logistic regression, Decision Tree, there are still more classification algorithms.
However if you are planning to extract a pattern from the dataset and making into cluster, you can use K-means clustering which is a un-supervised algorithm.


Batch normalization variance calculation In batch  normalization the variance calculation during the training phase is done by ($x_i$ are the individual elements in the training batch of size $m$)

$\sigma_B^2 = \frac 1m \sum_{i=1}^{m} (x_i - \mu_B)^2$
where $\mu_B = \frac 1m \sum_{i = 1}^{m}(x_i)$

and during the test time we calculate the population statistics for the same as

$E[x] = E_B[\mu_B]$ and $Var[x] = \frac{m}{m-1}E_B[\sigma_B^2]$ (as you can see the unbiased estimate of population variance is calculated during the test time as we are building the model to predict the population distribution)

Would it not seem more logical that bias correction be done during training phase itself and subsequently its expected value can be used without the $\frac {m}{m-1}$ correction factor. i.e

During training:

$\sigma_B^2 = \frac 1{m-1} \sum_{i=1}^{m} (x_i - \mu_B)^2$
where $\mu_B = \frac 1m \sum_{i = 1}^{m}(x_i)$

and during test time:

$E[x] = E_B[\mu_B]$ and $Var[x] = E_B[\sigma_B^2]$

Well, the only reason, I would say here, is the numerical stability. For instance, if the Batch size $m =1$ during the training, then you may end up with $NaN$ variance!

Update: I doubled check the BN paper again, they clearly mentioned the condition that $m>1$. However, I found an interesting post discussing the cases when biased estimator preferable than the unbiased one. This might explain why the authors used the biased one during the training and the unbiased one inference.


Do smaller neural nets always converge faster than larger ones? In your experience, do smaller CNN models (fewer params) converge faster than larger models?
I would think yes, naturally, because there are fewer parameters to optimize. However, I am training a a custom MobileNetV2-based Unet (with 2.9k parameters) for image segmentation, which is taking longer to converge than a model with greater number of parameters (5k params). If this convergence behavior is unexpected, it probably indicates a bug in the architecture

For most cases, probably. For all cases, no. Especially if you are training on small data with very aggressive regularization in place, you may need a very long time until the desired performance level is achieved.
For instance, for some popular text generation networks called Transformers trained on small datasets, it is necessary to use very aggressive regularization techniques and train during a very very large number of iterations (see this Twitter thread where they describe how to train a Transformer model on the PTB and Wikitext-103 datasets).


Show distribution of users affected by outlier response times My dataset is performance metrics (response time) of a web page over the course of a single day.
The data looks roughly like this:
| Date | User Id | Response time in milliseconds |
|------+---------+-------------------------------|
| ...  | U1      |                           390 |
| ...  | U2      |                          1965 |
| ...  | U2      |                          7789 |
| ...  | U1      |                           479 |
| ...  | U1      |                          9876 |

Charting the percentiles 50th, 75th, 90th, 95th and 99th I could see that the response time for the vast majority of users is below 600 ms - which is seen as acceptable.
But there are extreme spikes in response times: I tried to sketch that in the data example above.
What I would like to do is visualize if it's always the same users affected by these spike or if the behaviour is more erratic than that. In the example above, user U2 would experience frequent spikes, whereas U1 has better performance with one spike (I know the sample size is obviously small in the example but I hope it helps to illustrate what I'm after.)
Goal of this effort is to reveal if it's always the same few users that are affected by spikes, then we could narrow down the problem.

An idea would be to use a bubble chart, which visualizes the following triplet for each user:

Count of how often we saw the user id during the day. (This will be the size of the bubble)
99th percentile for this user (x-axis)
75th percentile for this user (y-axis)

What I thought we could see from such kind of chart:

The bigger the bubble, the more frequent we saw the user.
The closer to 0 on both axes, the better the overall performance.
The further away from 0 on the x-axis, the more extreme the performance peaks.
The further away from 0 on the y-axis, the worse the common case performance.

Assumption is, users that are shown as large bubbles near the upper right corner could be the ones to investigate.
I thought a while about this, but I can't really say I'm totally convinced this is what I'm after.
Does this approach make sense to find out if we have mostly the same users experiencing the worst performance?

One option would be look at conditional probability based on percentiles.
First, find percentiles based on all the data. For example, 99th percentile is 1432 milliseconds.
Then, find percent of a specific user request above that threshold. For example U1 has 50% of requests above 99th percentile.
This could be made into a cross-tabs table for easier exploration. The rows would be each user and the columns would be binned response time percentiles.


F1 score vs accuracy, which metric is more important? I have two multiclass classification models for making predictions (number of classes is three to be precise). One is Keras neural network, other is Gradient Boosted Classifier from Scikit Learn library.

I have noticed that after training on same data GBC has higher accuracy score, while Keras model has higher F1 score. Which model should I use for making predictions on future data? Which metric is more important?

Well it is highly dependent on your use cases and how your data is distributed.

Let me breakdown the pros and cons in practice :

F1-Score

Pros :

Takes into account how the data is distributed. Useful when you have data with imbalance classes.

Cons :

Less interpretable. Precision and recall is more interpretable than
f1-score, since it measures the type-1 error and type-2 error. However, f1-score measures the trade-off between this two.
When positive class is minority class, the score is quite sensitive when there is switching where the ground truth is positive.

Accuracy

Pros :

Easy to understand.

Cons :

It does not take into account how the data is distributed (example case below). This error could be crucial and might lead to incorrect conclusion.

Here is an example depicting cons of accuracy. Without loss of generality consider a binary classification task. Imagine having a data which of 100 samples with 90 negative sample and 10 positive sample. Suppose you have a classifier that predicts all negative. You will have an accuracy of 90%, but let's consider the f1 score, you will actually get 0 because your recall (which is a component of f1 score) is 0.

In practice, for multi-class classification model (which is your use-cases) accuracy is mostly favored. f1 is usually used for multi-label or binary label where the classes are highly unbalanced.


Is my bayes classification right or meaningful? I have this dataset and I am learning about Bayes Classifier. After data cleaning, I have tried to use bayes classifier on it. I used R with this code:

library(klaR)
trainingChoco <- chocolateApriori[1:707,]
testChoco <- chocolateApriori[708:884,]
naivebayesChocolate <- NaiveBayes(RatFactor ~ ., data=trainingChoco)
predictionChoco <- predict(naivebayesChocolate, testChoco)
predictionChoco$posterior
library(caret)
tableChocolate <- table(predictionChoco$class, testChoco$RatFactor)
confusionMatrix(tableChocolate)

Where RatFactor only has 1, 2, 3, 4, 5 as values (I rounded the ratings). This is my result:

  predictionChoco$posterior
                  1            2           3           4            5

1434 5.987980e-14 2.619121e-05 0.080559640 0.919414168 5.987980e-11

1435 4.205489e-10 8.759363e-03 0.022926453 0.968314183 4.205489e-10

1436 4.205489e-10 8.759363e-03 0.022926453 0.968314183 4.205489e-10

1439 1.004950e-10 4.709587e-03 0.006445339 0.988845074 1.004950e-10

1442 9.257687e-10 3.133371e-04 0.004947920 0.994738741 9.257687e-10

1443 3.260598e-13 8.227926e-03 0.171718690 0.820053058 3.260598e-07

[...]

  confusionMatrix(tableChocolate)

Confusion Matrix and Statistics

 1  2  3  4  5

1  0  0  0  0  0

2  0  2  5  7  0

3  0  7 42 28  0

4  0  5 45 36  0

5  0  0  0  0  0

Overall Statistics

           Accuracy : 0.452

             95% CI : (0.3772, 0.5284)

No Information Rate : 0.5198

P-Value [Acc > NIR] : 0.97

              Kappa : 0.0431

Mcnemar's Test P-Value : NA

Statistics by Class:

                 Class: 1 Class: 2 Class: 3 Class: 4 Class: 5

Sensitivity                NA   0.1429   0.4565   0.5070       NA

Specificity                 1   0.9264   0.5882   0.5283        1

Pos Pred Value             NA   0.1429   0.5455   0.4186       NA

Neg Pred Value             NA   0.9264   0.5000   0.6154       NA

Prevalence                  0   0.0791   0.5198   0.4011        0

Detection Rate              0   0.0113   0.2373   0.2034        0

Detection Prevalence        0   0.0791   0.4350   0.4859        0

Balanced Accuracy          NA   0.5346   0.5224   0.5177       NA

Do you think this result is right? Or am I missing something? Can you explain what would you understand by looking at this result please? Or can you post a similar example done step-by-step? Thanks.

Do you think this result is right?

Depends what you mean by "right"... the results seem reasonable, I don't see any obvious sign of mistake in the process.

  Can you explain what would you understand by looking at this result please?

I observe that you don't have any data for classes 1 and 5, so technically it's a 3-classes problem.

First with 3 classes the random baseline accuracy would be 0.33, 0.45 is better so your model does better than this (that's the bare minimum).
However according to the confusion matrix class 3 has 92 instances out of a total of 172, which means that a basic majority class learner always predicting class 3 would get 52% accuracy (if my calculation is correct). So 45% is not very good.


How to place multiple graphs in same coordinate system (pandas,mathplotlib) -> look pic in short how to set graphs in same coordinate system, not separated as in pic each in its own coordinate system in one frame....

Problem is with the dtype of your y values. It's "object" and thus matplotlib thinks two object containing same float value are different.
Use,
y  = a.astype(float).values
y1 = b.astype(float).values
y2 = c.astype(float).values


Object Detection classification I am currently training a classifier for detecting resistors using TensorFlow Object Detection API. For that, I downloaded resistor images from ImageNet and I am currently labeling those who will be used for training.

However, I found two different kinds of resistors in my training images :

I want to know if it's safe to put them under the same label or it would be better to classify them under two different labels so that the training will be more accurate ?

In general, you want to show your model all types of "resistors" it would see in the real world.

Is it feasible for you to manually go through your dataset and label each of these images as resistor A or resistor B? Do you care if your model can make this distinction? If the answer to either of these is no, I would just leave them all as one label and train the model. It's possible you don't have enough training data for your model to learn that these are different types of resistors, but you can evaluate that after you train.


What kind of data is not appropriate using CF to do recommendation? I am currently working on a recommendation system for daily news. At first, I evaluated all the recommender algorithms and their corresponding settings (e.g., similarities, factorizers, ...etc) implemented in Mahout. Since we want to recommend daily news for users, we use the reading behavior of each user collected two days ago as training set, data of the next day as the testing set. The evaluated RMSE is good, the best recommender is SVD+SGD, so we implemented the recommender on our system for several days of trial run.

However, the result, the actually recommended news, seems to be not very attractive for real users ("not attractive" here means, the users feel like "why you recommend this to me?"). So we decided another approach: use the tags and categories and their relationship to do the main job of recommendation, the result from CF is for just supporting.

This makes me wonder if CF if not appropriate for some kind of content. Because I also worked on movie and music recommendation, CF is a good tool. But for news, it seems not the case.

Can anyone explain why this happening, and also give some guideline about how to choose appropriate recommendation methods? Thanks:)

The key is establishing a proper validation metric.

I notice you talk about how you tried different recommendation algorithms, but at the end of the day you evaluated them all with RMSE. But there's no particular reason to believe that minimizing RMSE generates a "subjectively better" recommendation experience for the user - it just happens to be convenient, and happens to work well in some industries, but there is no real reason why it must.

RMSE is measuring how well your recommender algorithm is predicting user behavior. But that's not the same as measuring recommendation quality. Maybe users value something else - familiarity, or serendipity, or some other quality of the item being recommended. Users don't really care about being predicted.

Given your results, if you want to understand your users further, I'd focus my efforts in coming up with a mathematical metric that more closely matches the target you care about - user satisfaction - rather than RMSE. Once you know what metric you're trying to optimize, the algorithm to optimize it is much easier to select!


What kind of algorithms can be used as a stacker in stacked generalization? In stacked generalization, several algorithms (I use some random trees, booster trees, etc.) are first trained and used to make the predictions which are used as input for another algorithm. However, can I use any kind of algorithms, or is there a preference?
P.S.: I often see people using linear models in this case.

There is no preference for stacked generalization. You can use any algorithm whether it be a NN, RF model, XGB model, etc. The only thing that you need to take care of is the fact that the algorithm which you are applying to your model is useful to your data or not. Also, model stacked at level 1 may not be a very good model for level 2. For example, suppose at level 1 you stacked an XGB model with max_depth=12, eta=0.01, colsample_bytree=0.7, but at second level it might happen that the same depth is just an overkill and hence you  should discard using that model at second level. So, in short there is no preference for stacked models, the preference is for the data on which you are going to train your model.


Null predictions for ALS in Pyspark I am trying to read from my dataset which has three coloumns. (User, Repository and Number of Stars)

In[10]

lines = spark.read.text("Dataset.csv").rdd
print(lines.take(10))

Out[10]

[Row(value='0,0,0,290'), Row(value='1,1,1,112'), Row(value='2,2,2,87.8'), Row(value='3,3,3,69.7'), Row(value='4,4,4,65.7'), Row(value='5,5,5,62'), Row(value='6,6,6,61.6'), Row(value='7,7,7,60.7'), Row(value='8,8,8,57.7'), Row(value='9,9,9,56.2')]

In[10]

# Need to convert p[1] from str to int
parts = lines.map(lambda row: row.value.split(","))

print(parts.take(2))

Out[11]

    [['0', '0', '0', '290'], ['1', '1', '1', '112']]

In[12]

# RDD mapped as int and float from Dataset

ratingsRDD = parts.map(lambda p: Row(userId=int(p[1]),repoId=int(p[2]),repoCount=float(p[3])))
ratings = spark.createDataFrame(ratingsRDD)
print(ratings.head(10))

Out[12]

    [Row(repoCount=290.0, repoId=0, userId=0), Row(repoCount=112.0, repoId=1, userId=1), Row(repoCount=87.8, repoId=2, userId=2), Row(repoCount=69.7, repoId=3, userId=3), Row(repoCount=65.7, repoId=4, userId=4), Row(repoCount=62.0, repoId=5, userId=5), Row(repoCount=61.6, repoId=6, userId=6), Row(repoCount=60.7, repoId=7, userId=7), Row(repoCount=57.7, repoId=8, userId=8), Row(repoCount=56.2, repoId=9, userId=9)]

In[13]

(training, test) = ratings.randomSplit([0.8, 0.2])

In[14]:

# Build the recommendation model using ALS on the training data
# Cold start strategy is set to '"drop" to make sure there is no NaN evaluation metrics which would result in error.
als = ALS(maxIter=5, regParam=0.01, userCol="userId", itemCol="repoId", ratingCol="repoCount"
        ,coldStartStrategy="drop") #Cold-start is set to DROP
model = als.fit(training)

In[15]

#Evaluate the model by computing the RMSE on the test data
predictions = model.transform(test)
type(predictions)

predictions.show(3)

Out[15]

    +---------+------+------+----------+
    |repoCount|repoId|userId|prediction|
    +---------+------+------+----------+
    +---------+------+------+----------+

My model is giving NULL values. Is there a problem in my dataset, or wrong assumptions on Training.

Note that my ratingCol in ALS is Number of Stars which is an explicit rating and not implicit rating.

I would need to understand certain aspects of your dataset to give you a better answer but:

The reason you're receiving NULL Values:

Depending on the class distribution, You may receive null predictions due to not having classes("repoId") in both samples of the original data. So classes that exist in test data may not exist in the training data. So when you apply the transform to the test data it has no basis on which to make a prediction on the supplied data. When you use "coldStartStrategy" it just omits those records altogether.

I would recommend first setting "ColdStartStrategy" to False just to see if all your records are just returning Null prediction values.

If that is the case, You would then need to check class distributions in both the training and test samples of your data. This should be the class distribution for "repoId". Then you would have to sample the data in a way that makes sure classes are present in both samples then reapply.

Potential flaws in your process:

Use seeding in "randomSplit" so your samples are always reproducible therefore you can detect issues regardless of when you ran your program
You may have assumed equal class distribution in your dataset
Normalizing your "Repocount" values would benefit the algorithm, but likely has no bearing on the overall results


Keras Conv1D model Input_shape value error I am not sure why I am receiving this value error. Additionally, I haven't found a tutorial that explicitly talks about the appropriateness of size of filters and kernel. I would appreciate some input and some links. I am predicting the next to the last or last column.

Here is my code:

import pandas as pd
import keras
from keras import Input, layers , Model
from sklearn.model_selection import train_test_split
from keras.layers import Dropout

input_tensor=Input(shape=(6,))
x= layers.Conv1D(filters=128,kernel_size=18, padding='same', activation='relu')(input_tensor)
#x=layers.MaxPooling1D(5)(x)
#x = Dropout(0.5)(x)

x= layers.Conv1D(256,5, activation='relu')(x)
#x = Dropout(0.4)(x)
#x= layers.Dense(4, activation='relu')(x)
#x = Dropout(0.2)(x)
x= layers.Conv1D(256,5, activation='relu')(x)
x=layers.MaxPooling1D(5)(x)

x = Dropout(0.3)(x)
#x= layers.Dense(16,1, activation='relu')(x)

callbacks_list=[
        keras.callbacks.EarlyStopping(monitor='acc',
                                      patience=3,),
                                      keras.callbacks.ModelCheckpoint(
                                              filepath= 'C:/Users/vtodorova/results3/APIFunctional.py',
                                              monitor='accuracy',
                                              save_best_only=True),
                                              keras.callbacks.ReduceLROnPlateau(
                                                      factor=0.1, patience=10,)]

model.compile(optimizer='RMSprop', loss='mse', metrics=['accuracy'])
#callbacks=callbacks_list
model.fit(X1_train, y1_train, epochs=3, batch_size=256, verbose=1)
model.fit(X2_train, y2_train, epochs=3, batch_size=256, verbose=1)

score1=model.evaluate(X1_train, y1_train)
score2=model.evaluate(X2_train, y2_train)

output_tensor=layers.Dense(1)(x)

model=Model(input_tensor, output_tensor)

Here is the head of the data. The names of the columns and the columns got a little misplaced when I copied and pasted, but I hope its ok.

AutoLeadID  leadage  leadstatustypeid  hasCob  hasSRE
0   695746319        5                 1           0                 0
1   695746320        5                 1           0                 0
2   695746321        5                 1           0                 0
3   695746322        5                 1           0                 0
4   695746323        5                 1           0                 0

   hasSRC   hasCRE                                                hasCRPC
0                0                      0                         0
1                0                      0                         0
2                0                      0                         0
3                0                      0                         0
4                0                      0                         0

After running your code & the major issue found out is the lack of proper mentioning of input shape.

In your case insisted of passing shape=(6,) try to pass shape=(6,1) and make sure your data should in the following format.

x = [[695746319], [5], [1], [0], [0], [0], [0]]


How to map topic to a document after topic modeling is done with LDA Is there any way I can map generated topic from LDA to the list of documents and identify to which topic it belongs to ?
I am interested in clustering documents using unsupervised learning and segregating it into appropriate cluster.
Any link, code example, paper will greatly be appreciated.

After training your LDA topic model you can input documents into the model and it will classify them into the pre defined number of topics. In gensim (python), this would look something like this:

ques_vec = dictionary.doc2bow(tokenized_document)
topic_vec = ldamodel[ques_vec]

The dictionary is something you should have created for training
ldamodel is the model that you trained.
The topic_vec will contain the classified topic number (class) and the probability that the document belongs to that class.

At this point, you will not know what is the meaning of each topic (class), because it is the result of unsupervised classification. To know what is the meaning of each topic that your lda model clusters your documents into, you have to look into the trained parameters like this:

words = ldamodel.show_topic(topic_number, topn = 200)

If you print that, you'll see the top 200 words that make up that topic number. Based on the meaning of the words in each topic, you name that topic as an appropriate class.


Why use different variations of Softmax in training and validation for neural networks with Pytorch? Specifically, I'm working on a modeling project, and I see someone else's code that looks like

def forward(self, x):
    x = self.fc1(x)
    x = self.activation1(x)
    x = self.fc2(x)
    x = self.activation2(x)
    x = self.fc3(x)
    x = self.activation3(x)
    # use log softmax + NLLLoss in training; softmax to make predictions
    if self.training:
        x = self.log_softmax(x)
    else:
        x = self.softmax(x)
    return x

For context, this is using PyTorch, and it is on a classification problem. The criterion is NLLLoss.
What's the rationale behind using log_softmax  for training but using softmax for actual predictions?

It's more of a pytorch implementation thing. log_softmax() outputs the raw logits and they are passed to NLL Loss in training. During inference you just need the probabilities so softmax will suffice.
You don't use different algorithms for training and testing, the question is quite misleading as currently stated. You are using different implementations of the same algorithm.
You can find more on the matter here:

Pytorch discuss
Pytorch discuss #2
Pytorch docs

P.S: I remember encountering the same question on a Udacity scholarship. If you 're reading code from one of their courses, there is probably an explanation on the solution jupyter.


In a recommender system, how can you normalise the similarity between two arbitrary users? Consider the following problem:

There are 1000 users, 100 items (movies, for example), and 10000 ratings. The probability of a user, $u$, rating a movie, $i$, is $\mathbb{P}(R_{u_i}=\text{yes})=\frac{1}{10}$, and the probability of any two users rating the same movie is $\mathbb{P}(R_{u_i}=R_{v_i}=\text{yes})=\frac{1}{100}$.

If we let a random variable $x_i=1$ if $i$ is rated by both $u$ and $v$, and $0$ otherwise, then we have $$E\Big[\sum_{i=1}^{100}x_i\Big]=100\times\big(1\frac{1}{100} + 0\frac{1}{100}+0\frac{1}{100}+ ... \big)=1.$$ The similarity between $u$ and $v$ is $$PC(u, v)=\frac{\sum_{i\in T_{uv}}(R_{u_i}-\bar{R_u})(R_{v_i}-\bar{R_v})}{\sqrt{\sum_{i\in T_{uv}}(R_{u_i}-\bar{R_u})\sum_{i\in T_{uv}}(R_{v_i}-\bar{R_v})}}=1,$$ where $T_{uv}$ is the set of all items rated by $u$ and $v$, and I used the fact that $R_{u_i}$ and $R_{v_i}$ become irrelevant in comparison to $\bar{R_u}$ and $\bar{R_v}$. This is an obvious problem because it means that in situations like this you will always have a perfect similarity between $u$ and $v$.

I believe that the normalisation solution is something like this:

$$PC(u, v)=\frac{\text{min}(|T_{uv}|, \beta)}{\beta}PC(u, v),$$ where $\beta\in[25, 50]$.

I don't understand the logic behind this normalisation. Could you explain please?

This normalization means if the number of mutual voting of $u$ and $v$ is low and not enough proportional to total number of voting (here from $\frac{25}{10000} = \frac{1}{400}$ to $\frac{50}{10000} = 0.005$) you should apply this fact into the similarity in some way. Because, your observation is not enough and you can't talk about it confidently.

In other words, it can be confident factor which you apply into the similarity measure.


Regression: How to deal with positive skewness in continuous target variable I'm working on a regression problem. My aim is to "learn" the distribution of a continuous target $y$ as good as possible to make predictions. My model looks like:

$$y_i=\beta X_i + u_i.$$

$y$ is right skewed (positive skewness) and consists of positive and negative integer values.
$X$ is a matrix containing columns with float and integer values. There also is a large(er) number of indicators (dummy variables)

These are my results. Here I compare the distribution of the actual $y_i$ (truth, blue) and the predicted $\hat{y_i}$ (predicted, yellow):

As you can see from the figure, I tend to underestimate values in the lower range (around zero) and I overestimate values somewhere between 200 and 500.

I would like to improve my estimation especially in the range somewhere around zero and 600, where most observations are found. I wonder what my options are?

Please note that the whole range of data is relevant to me, so that I cannot simply prune the data/distribution.

So far I did no feature engineering, scaling etc. What I tried was a number of different estimation methods, such as linear regression (OLS), generalised models (GAM) with local regression and regression splines, lasso/ridge, Keras neural net, boosting with LightGBM and Catboost. All of them produced very similar results (as shown in the figure above).

Edit [2019-12-28]:

A brief follow-up to my original question: I have applied a logistic transformation to my $y$ and estimated a GAM model. Unfortunately the results are not much better than with non-transformaed data. MAE decreases slightly, but the general fit is still not too good (see figure below, the "flat" curve are actual values and the pointed curve are predicted values after re-transformation).

I prefer to stick to linear models (OLS, GAM, Lasso) since boosting or neural nets did not deliver much better results than linear models.

Problem in this competition is VERY similiar. Instead of me copy-pasting some interesting ideas just check out winner write-ups in the discussions.

Some takeways:

Add count of values of features as new features.
This is information that LightGBM can’t see
Check unique value counts for all features of train and test set.
Model stacking almost always works better than single model (forget about it in production)
Check for feature independence. For this competition, features were all uncorrelated (naive bayes worked great)
etc...

And finally and independently from this competition general note: Its all about the right data representation and not about the model. If you can represent data in the right way than basic linear regression will score good. I would do some serios data analysis for residual values in range negative until 200 and 500 until infty and see what confuses your model, obviously there are some conflicting features that force your model to predict low when it should be high. Maybe this is not satisfying to you but to really solve this problem you have to find what confuses him so that he predicts high when it should be high, and then engineer feature/data representation so that he can discriminate around this.


Why is finite precision a problem in machine learning? Can you explain what is finite precision? Why is finite precision a problem in machine learning?

Finite precision is decimal representation of a number which has been rounded or truncated. There many cases where this may be necessary or appropriate. For example 1/3 and the transcendental numbers $e$ and $\pi$ all have infinite decimal representations. In the programming language C, a double value is 8 bit and precise to approximately 16 digits. See here.

http://www.learncpp.com/cpp-tutorial/25-floating-point-numbers/

To concretely represent one of these numbers on a (finite) computer there must be some sort of compromise. We could write 1/3 to 9 digits as .333333333 which is less than 1/3.

These compromises are compounded with arithmetic operations. Unstable algorithms are prone to arithmetic errors. This is why SVD is often used to compute PCA (instability of the covariance matrix).

http://www.sandia.gov/~smartin/presentations/SMartin_Stability.pdf

https://en.wikipedia.org/wiki/Numerical_stability

In the naive bayes classifier you will often see multiplication transformed into a sum of logarithms, which is less prone to rounding errors.

https://en.wikipedia.org/wiki/Naive_Bayes_classifier#Multinomial_naive_Bayes


Handling missing values to optimize polynomial features I was playing around with some data to practice my Python and machine learning skills and wanted to create polynomial features from two features that I think are related and have a strong influence on the predicted output.

Unfortunately my data has missing values (np.NaN) and sklearn's PolynomialFeatures() can not handle these values. What is the best way to impute these values?

I've been trying to replace them with 0, 1, mean and median and for my dataset using the median seems to be the best solution. But can this be generalized and what is the intuition behind it?

I was also wondering if filling methods like ffill, bfill or even KNN modelling can be useful in this context.

Thanks a lot!

There is no globally - one could say even locally - ideal way to deal with missing data. This aspect points to incompleteness in the data you're feeding your algorithms, and imputing is simply a technique meant to fill gaps.

Data imputation's motivation is to make feature distribution in your sets the closest possible to whatever real-world distribution it attempts to portray.

The intuition behind why the median is what worked better for your scenario is not something I could be precise about without having access to the data you worked upon, but the positive result is deeply related to your feature distribution, which for the missing points in your data is better represented by the feature's median than all other metrics you've calculated. I'd recommend reading material such as this article that both explains and shows implementation of different imputation techniques - KNN, as you mentioned, being one of them. One of the advantages is seeing how different methods work for a given distribution:

As you have already pointed in the end of your question, methods such as KNN are some straightforward means of imputation that could benefit your case better than mean/median imputation. The biggest difference between those is that KNN better preserves the variance in your data, whereas mean imputation (as you can see from the above image) shifts missing data towards a single value.

Since no method is 100% globally optimal, I'd advise you to try them - KNN, Multiple Interpolation and the likes - and compare. Invest the appropriate amount of time on the techniques that make the most sense for your data.


encoding datetime features .. label or onehot? I am working on a cab booking prediction problem where I need to use datetime aspects like hour,day ,week etc for prediction. As I need to do categorical encoding for the purpose. can anyone help me out on which one to use i.e label encoder or one hot encoder. I am not able to gather much information on the web regarding this. I tried doing label encoding but as onehotencoding is preferred for ordinal data like this. Any help would be appreciated.

One hot encoding removed the order information compared to the label encoder. Which one to use depends on the specific question. If you think the order of categories matters, use the label encoder. If not, use the one-hot encoder. For booking prediction, I will prefer one hot.


Memory error on using data generator in keras I am using the following augmentations on dataset of size 9 GB:

datagen = ImageDataGenerator(
    featurewise_center=False,  # set input mean to 0 over the dataset
    samplewise_center=False,  # set each sample mean to 0
    featurewise_std_normalization=False,  # divide inputs by std of the dataset
    samplewise_std_normalization=False,  # divide each input by its std
    zca_whitening=True,  # apply ZCA whitening
    rotation_range=30,  # randomly rotate images in the range (degrees, 0 to 180)
    width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
    height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
    horizontal_flip=True,  # randomly flip images
    vertical_flip=True)  # randomly flip images
datagen.fit(a)

model.fit_generator(datagen.flow(a,b, batch_size=32),
                    steps_per_epoch=len(a) / 32, epochs=epochs, class_weight = sclass_weight, validation_data = [c, d],callbacks = [MetricsCheckpoint('logs')])

When the code comes to the datagen.fit, I get into memory error ( the code doesn't even go into training)

I have 50 gb ram and am training it on a K80 with a batch size of 32, so don't think that will be a problem.

It works fine when I comment all the augmentations.

Can someone tell me where I am going wrong?

It seems like the root cause for this is that zca_whitening is set to True. According to the answer from the keras-collaborator rragundez in this related Github issue, there is no work-around for it, except disabling ZCA whitening:

  This is known problem with the ZCA method. There is no real immediate way around it. The problem is with the calculation of the sigma matrix which is a dot product.

The other alternative to disabling it would be to use images with a smaller resolution. But I suppose that's rarely a practical solution.


Splitting hold-out sample and training sample only once? I have a question related to evaluating out-of-sample predictions.

For my research I want to tune two parameters related to Support Vector Machines, and use these optimized parameters to predict the hold-out sample as good as possible. To evaluate my model I obviously have to split my data in a training sample (80%) and a hold-out sample (20%). When tuning the parameters I also use 10-fold CV, but this only involves the training sample.

Now I thought that my approach is not super valid, as the 20-80 split of hold-out versus training sample is only done once, and thus might be too subject to randomness. However, I feel that this approach is used quite often, for example in http://www.sciencedirect.com/science/article/pii/S095741740400096X

In my earlier research I have done leave-one-out predictions, where I used n-1 observations and predicted the remaining observation (so that every single observation is predicted once, using the model that is trained based on only the other n-1 observations). However, in my new project this is not possible, because I then also need to tune the parameters n times, which would require too much time and would mean I lose the interpretation/plots relating to these parameter values.

In short: is it a bad thing that I only split my training and hold-out sample once? Does anyone have any comments about my current approach?

In general, it's a good idea to split up your data into three sets:

Training Set (60-80% of your data)
Cross-Validation Set (10-20% of your data)
Test Set (10-20% of your data)

When you select a model using only a train and test set, you are selecting the model which performs the best on the test set after. This seems reasonable at first, but this is usually an overly optimistic estimate of your model's generalization error. This is because you are essentially fitting your model to another parameter $d$, where $d$ is your test set.

In order to address this problem, you can train a model on your training set and then choose the model that performs best on your cross-validation set. Now, this model is no longer fit to the test set and you can safely estimate the generalization error on the test set.

It sounds like you are already following this approach in selecting the model by choosing which one performs best on a 10-fold cross-validation on your training set. In this case, the error on your test set should be a good estimate of generalization error.

You also mentioned that randomness might affect your training due to outliers in your training data:

  ...as the 20-80 split of hold-out versus training sample is only done once, and thus might be too subject to randomness.

In machine learning, it is often assumed that our data is iid (independent and identically distributed) from some unknown distribution. On average, we should get data points that are representative of this distribution when we split up our data. A common practice to better ensure this is to randomly shuffle the entire dataset before splitting up your data.