How to use df.groupby() to select and sum specific columns w/o pandas trimming total number of columns I got Column1, Column2, Column3, Column4, Column5, Column6
I'd like to group Column1 and get the row sum of Column3,4 and 5
When I apply groupby() and get this that is correct but it's leaving out Column6:
    df = df.groupby(['Column1'])[['Column3', 'Column4', 'Column5']].sum

I tried with this but it doesn't group according to Column1 and it doesn't sum anything, but I get all my columns:
    df.sort_values(['Column1']).groupby(['Column3', 'Column4', 'Column5']).sum()

How can I use groupby() correctly in this case?
Thank you!
I add my code:
    df = df.drop(['Position', 'Swap', 'S / L', 'T / P'], axis=1)
    df = df.groupby(['Symbol']).agg({'Profit': ['sum'], 'Volume': ['sum'], 'Commission': ['sum'], 'Time': pd.Series.mode})
    df['Comm. ratio'] = (df['Commission'] / df['Profit'])
    df['Comm. ratio'] = df['Comm. ratio'].round(2)
    df['Net profit'] = (df['Profit'] + df['Commission'])
    df.loc['Total'] = pd.Series([df['Commission'].sum(),df['Profit'].sum(),df['Net profit'].sum()], index = ['Commission','Profit','Net profit'])

The output is:

As you can see it adds "sum" and "mode" rows that I'd like not to have.
Moreover, it ignores the df.loc['Total'] code and it leaves the Total row empty.

I think the answer depends on what you want to do with column 6. Keep in mind that the values for column6 may be different for each groupby on columns 3,4 and 5, so you will need to decide which value to display. Typically, when using a groupby, you need to include all columns that you want to be included in the result, in either the groupby part or the statistics part of the query.
If you don't want to group by that column, you can just display the min or mode value. In general, if you want to calculate statistics on some columns and keep multiple non-grouped columns in your output, you can use the agg function within the groupyby function.
Example with most common value for column6 displayed:
df.groupby('Column1').agg({'Column3': ['sum'], 'Column4': ['sum'], 'Column5': ['sum'], 'Column6': pd.Series.mode})

Full example with code:

If there is a tie for most common, with one Mary and one Jane both being Female Engineers, this will generate an error as mode doesn't reduce to a single answer:

You will need to use another aggregate in that case, such as min, which will choose Jane as an alphanumeric min:

If you don't like the look of the multi-index, you can remove it using as_index=False and replacing the column names with a list(map(...join...))

or remove it using to_flat_index() which gives a slightly different format for the columns:


Modelling data with Machine learning without a target variable I want to understand the required steps that need to be taken into account while handling a dataset that does not have a target variable.
I can do machine learning on top of a labeled dataset having a target variable, but not sure what would be the best way to start with a dataset where is there is no target variable.

I need a step by step guide to achieve an efficient clustering at the end.

Do I need to do the following in order to achieve that?:

Data Cleansing
EDA
Encoding and scaling
Model build
Validation

Or are there any more steps that I need to take care of while dealing with an unsupervised class of data. I am doing this in python

There cannot be a unique answer to your question. There is a discrepancy in your question though -

I am aware that this is a classification problem on which I am working on.
Could you please help me with the right step by step guide that I should follow in order to achieve an efficient clustering at the end?

However, I am assuming that you are trying to do clustering and you want methods that would give you mathematically better clusters.
clustering is an unsupervised learning problem that does not require target variables. The steps that you mentioned are pretty standard and theoretically correct but there are also other steps that you should take care of. I am listing a few :

Selection of input features - Input features that go into a clustering algorithm are of great importance. It should be noted that a variable not containing any relevant information (say, the telephone number of each person) is worse than useless because it will make the clustering less apparent. In general, the selection of “good” variables is a nontrivial task and may involve quite some trial and error
Selection of clustering algorithm - Use of a good clustering algorithm as per your data is an important step. For example, K- Means better work with numerical features, K- Modes with categorical and K- prototypes in case if you have the data which is a  mix of numerical and categorical features.


Isolation forest sklearn contamination param I'm working on an unsupervised anomaly detection task on time series using isolation forest algorithm.
I'm developing in Python, more in detail using sklearn.

I found out a lot of examples on this, but what is not very clear, is how to set the contamination param during the instantion of IsolationForest.

Looking the documentation, contamination is

  the amount of contamination of the data set, i.e. the proportion of outliers in the data set.

Should I use some statistical techniques to identify this percentage?

Thank you

I think the answer is "it depends". The contamination parameter simply controls the threshold for the decision function when a scored data point should be considered an outlier. It has no impact on the model itself.

Depending on your application it could make sense to use some statistical analysis to get a rough estimate of the contamination. (If you could narrow it down more specifically then you wouldn't need the isolation forest...)

Usually, I would say, this is informed by some business expectation. Something like "we expect to see a similar rate to our competitors who have reported a rate of x".

Or you expect a certain number of outliers in your dataset. Then you can use the raw scores to find a threshold that gives you that number and set the contamination parameter retrospectively when applying the model to new data.


Import csv file contents into pyspark dataframes How can I import a .csv file into pyspark dataframes? I even tried to read csv file in Pandas and then convert it to a spark dataframe using createDataFrame, but it is still showing some error. Can someone guide me through this? Also, please tell me how can I import an xlsx file?
I'm trying to import csv content into pandas dataframes and then convert it into spark data frames, but it is showing the error:

"Py4JJavaError" An error occurred while calling o28.applySchemaToPythonRDD. : java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

My code is:

from pyspark import SparkContext
from pyspark.sql import SQLContext
import pandas as pd
sqlc=SQLContext(sc)
df=pd.read_csv(r'D:\BestBuy\train.csv')
sdf=sqlc.createDataFrame(df)

"How can I import a .csv file into pyspark dataframes ?"
-- there are many ways to do this; the simplest would be to start up pyspark with Databrick's spark-csv module. You can do this by starting pyspark with

pyspark --packages com.databricks:spark-csv_2.10:1.4.0

then you can follow the following steps:

from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)

df = sqlContext.read.format('com.databricks.spark.csv').options(header='true', inferschema='true').load('cars.csv')

The other method would be to read in the text file as an rdd using

myrdd = sc.textFile("yourfile.csv").map(lambda line: line.split(","))

Then transform your data so that every item is in the correct format for the schema (i.e. Ints, Strings, Floats, etc.). You'll want to then use

>>> from pyspark.sql import Row
>>> Person = Row('name', 'age')
>>> person = rdd.map(lambda r: Person(*r))
>>> df2 = sqlContext.createDataFrame(person)
>>> df2.collect()
[Row(name=u'Alice', age=1)]
>>> from pyspark.sql.types import *
>>> schema = StructType([
...    StructField("name", StringType(), True),
...    StructField("age", IntegerType(), True)])
>>> df3 = sqlContext.createDataFrame(rdd, schema)
>>> df3.collect()
[Row(name=u'Alice', age=1)]

Reference: http://spark.apache.org/docs/1.6.1/api/python/pyspark.sql.html#pyspark.sql.Row

"Also, please tell me how can I import xlsx file?" -- Excel files are not used in "Big Data"; Spark is meant to be used with large files or databases. If you have an Excel file that is 50GB in size, then you're doing things wrong. Excel wouldn't even be able to open a file that size; from my experience, anything above 20MB and Excel dies.


Looking for advice: data transfer I'm requesting data from a government body, and they asked me what format I want to receive the data in. This will be a table of about 400,000 rows and about 10 columns. My options are:

"comma or tab delimited ASCII, Microsoft Access database, Microsoft Excel file etc"

They also want to know the media by which I want to receive data. My options are:

"encrypted CD, DLT Tape, etc."

I'm used to working with JSON, and doing data analysis via Python and R. These data will have to remain confidential.

Any pointers?

Either format will work with R and Python, though you'll need a library for MS Access, which is the least common option. If any of these formats is the native format of the data, I'd go for that, which will avoid weird transformation artefacts. If neither of the formats is, I'd choose CSV, which would allow me to use a text editor if needed. However, if one of the columns contains user generated texts, it might be worth the trouble to choose Excel, that would limit the chance that the export would get botched.

It is way easier to get your hands on a CD-reader. Your data doesn't sound that impressively large that it warrants using tape.


Please review my sketch of the Machine Learning process It's amazingly difficult to find an outline of the end-to-end machine learning process. As a total beginner, this lack of information is frustrating, so I decided to try scraping together my own process by looking at a lot of tutorials that all do it a slightly different way.

I would like to have a standard process to go by, and once I am comfortable with it, I can choose to deviate. I'd like some input from you pillars of the industry. Is this a good routine for a beginner to follow?

Get Data
Clean Data
Split data into Training and Test Data ~(80/20)
Separately, for training and test sets:

Normalize Data (continuous features):

standardize (divide by std. deviation)
center (subtract mean)

Impute missing values
Feature Engineering
Encode Categorical Variables:

Integer Encoding
One Hot Encoding
Target Encoding
Weight of Evidence

Separate labels from Test set if classification problem. Keep aside.
Choose a few models.
for each model, using k-fold cross-validaton:

Train base model on "training set".
Tune and test hyper parameters on "validation set"
Save best scores and parameters

Compare each model's final scores on the never touched test data
Choose the model with highest scores.

Edit: Thank you for the overwhelming number of responses. Lots of times my questions get a single answer or none at all. I appreciate the time taken to help out a beginner.

I have edited the steps above to reflect the wonderful answers below. I hope that this helps another beginner somewhere else.

This process will result in data leaks. The split needs to happen earlier. Normalizing data before the split means that your training data contains information about your test data. I would put the split at 3. in your flow chart.

A common step I think you have missed is imputation of missing values. I would put that before feature engineering.

Overall I think this is a good rough outline for a beginner to follow. It is overly simplistic and leaves a lot out, but I think you know that and you have to start somewhere.


Split a tensoflow model into two parts I have a pre-trained tensorflow model for image classification. It has has some convolution and maxpooling layers followed by dense layers.

I would like to split it in two pats. The first containing the convolutions/maxpooling part, the second containing the dense layers. Then I would like to feed the first part with an image, store the result into a file/variable and then using it as input for the second part.

The idea is that I could encode on the fly images with the first part, save them to disk and then using the encoded files as input. In this way I can avoid storing the original images.

I am writing an example of a whole model (based on Coursera's/deeplearning.ai "Introduction to TensorFlow for Artificial Intelligence, Machine Learning, and Deep Learning")

import tensorflow as tf
mnist = tf.keras.datasets.fashion_mnist
(training_images, training_labels), (test_images, test_labels) = mnist.load_data()
training_images=training_images.reshape(60000, 28, 28, 1)
training_images=training_images / 255.0
test_images = test_images.reshape(10000, 28, 28, 1)
test_images=test_images/255.0
model = tf.keras.models.Sequential([
  tf.keras.layers.Conv2D(32, (3,3), activation='relu', input_shape=(28, 28, 1)),
  tf.keras.layers.MaxPooling2D(4, 4),
  tf.keras.layers.Conv2D(32, (3,3), activation='relu'),
  tf.keras.layers.MaxPooling2D(2,2),
  tf.keras.layers.Flatten(),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dense(10, activation='softmax')
])
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
model.summary()
model.fit(training_images, training_labels, epochs=5)

In the example above,

first_part would have the first 5 layers
second_part would have the last two (and possibly an input layer)

Thank you.

After several attemps I have the following piece of code. I created two new networks and transferred the weights from the initial network to them.

# first part of initial model
part1_model = tf.keras.models.Sequential([
  tf.keras.layers.Conv2D(32, (3,3), activation='relu', input_shape=(28, 28, 1)),
  tf.keras.layers.MaxPooling2D(4, 4),
  tf.keras.layers.Conv2D(32, (3,3), activation='relu'),
  tf.keras.layers.MaxPooling2D(2,2),
  tf.keras.layers.Flatten()
])
part1_model.layers[0].set_weights(model.layers[0].get_weights())
part1_model.layers[1].set_weights(model.layers[1].get_weights())
part1_model.layers[2].set_weights(model.layers[2].get_weights())
part1_model.layers[3].set_weights(model.layers[3].get_weights())
part1_model.layers[4].set_weights(model.layers[4].get_weights())
part1_model.summary()

# second part of initial model
part2_model = tf.keras.models.Sequential([
    tf.keras.layers.Dense(128, activation='relu',input_shape=(1, 128)),
  tf.keras.layers.Dense(10, activation='softmax')
])
part2_model.summary()
part2_model.layers[0].set_weights(model.layers[5].get_weights())
part2_model.layers[1].set_weights(model.layers[6].get_weights())
# predictions holds prediction for test set from initial model
predictions = model.predict(test_images)

# predictions1 holds output of first part when test set is used as input
predictions1 = part1_model.predict(test_images)

import numpy as np
# tmp is used to transform prediction1 in a format recognizable from part2
tmp=np.zeros((10000,1,128))
for i in range(0,10000):
    tmp[i,:,:]=predictions1[i,:]

# predictions2 holds the output of second part when the result of first part (predictions1) is used as input
predictions2 = part2_model.predict(tmp)

# check that the result of initial model is the same with the result of the two parts
ok=0
for i in range(0,10000):
    if np.argmax(predictions[i])!=np.argmax(predictions2[i]):
        print(i," False")
    else:
        ok=ok+1
print(ok)
# also sample some cases and check probabilities
print("last entry")
print(predictions2[9999])
print(predictions[9999])
print("2nd entry")
print(predictions2[1])
print(predictions[1])


Feasibility: train a model to learn how to extract data from documents I am searching for an approach for solving the following problem:

Given I have a large amount of printed and scanned documents. I am already able to detect text and the corresponding bounding boxes in that document. I have the texts and the coordinates of the bounding boxes. Now, if I want to extract specific data like "insurance contract number" or "recipient" and stuff like that, I would have to manually program that (to some degree).

I was wondering if there is an approach to show the computer what is the correct data extraction. Is it possible to train a model against a large amount of documents and the correctly extracted data as labels. So the features would be the extracted text plus the coordinates and the model should learn which data to extract depending on the training examples with correctly extracted text.

All algorithms I came across so far only "allow" one label. From my understanding this apporach would need like 20 labels for every training example. And also the prediction is also only "correct", if all labels are matching.

Anyone has an idea how to approach that?

Both of the answers from FelixGK and Wargream have merit.  With the information you have posted though, I am not sure you are approaching the problem with the right objective, so the expected solution may seem further away than it probably is.

As FelixGK suggests, you should break the models down into a set of one-vs-the-rest binary classifiers, one for each label.

I am not sure what you mean when you state

  From my understanding this apporach would need like 20 labels for every training example. And also the prediction is also only "correct", if all labels are matching.

Do you expect one model to collect multiple data elements within a document and assign the correct label to the element?  This is probably difficult in a single model, but could be easier using some form of ensemble approach, like a set of binary classifiers.  If you want to try it all in one model, I would look at neural networks and specifically, a network that implements a SoftMax activation function in the final layer. This will allow you to produce a multi-class output with probabilities assigned to each class. When using softmax, you generally want to accept the result with the highest probability.


When to use GRU over LSTM? The key difference between a GRU and an LSTM is that a GRU has two gates (reset and update gates) whereas an LSTM has three gates (namely input, output and forget gates).

Why do we make use of GRU when we clearly have more control on the network through the LSTM model (as we have three gates)? In which scenario GRU is preferred over LSTM?

GRU is related to LSTM as both are utilizing different way if gating information to prevent vanishing gradient problem. Here are some pin-points about GRU vs LSTM-

The GRU controls the flow of information like the LSTM unit, but without having to use a memory unit. It just exposes the full hidden content without any control.
GRU is relatively new,  and from my perspective, the performance is on par with LSTM,  but computationally more efficient (less complex structure as pointed out).  So we are seeing it being used more and more.

For a detailed description, you can explore this Research Paper - Arxiv.org. The paper explains all this brilliantly.

Plus, you can also explore these blogs for a better idea-

WildML
Colah - Github

Hope it helps!


Time series feature extraction from raw sensor data for classification? I have a tabular raw data from sensors with associated label and i want to extract the time series features like mean,max,min and std from the data all the sensor data and form another table or export to csv file so that i can do classification task on that data.

Data table

You can also use an open source python library called 'tsfresh' (https://tsfresh.readthedocs.io/en/latest/) to extract time series features


Bootstrapping or Randomly Dividing Dataset to reduce variance? If I have 10,000 training samples then what should I do:

Bootstrapping and train 10 classifiers on it and then aggregating

Or

randomly divide the dataset into 10 parts and train 10 classifiers on them and then aggregating. Which will be better?

Will the 2nd method reduce variance and will it be better than 1st method

I think the second method will yield less correlated models than the first method. It is particularly true with decision trees which tend to quickly overfitting in the bottom nodes. It will help reduce the variance.

However, by using the second approach, you will end with 10 smaller datasets and so you risk to introduce a variance error due to the too small number of observations. Discussing about decision trees again, it would mean that your trees algorithms will tend to overfitting upper in the tree. And so you will increase your variance error.

In my opinion, for most of datasets, it is still better to use the first approach than the second. I think that very low correlated estimators won't bring a better improvement that the first method.

We can also observe that differences in the two approaches depend also of the number of observations, the number of features, the kind of estimators you are using.. A benchmark would be really interesting !


Adjusted R-squared is too high (=1) in Linear Model I built a Linear model which has an adjusted r-squared value of 1. I understand that this is a near perfect number. Upon further investigation, I found that one of the 96 independent variables in the dataset is highly correlated with the dependent variable. This is also a variable which I would like to keep (and not drop). Are there any additional steps that I should undertake to handle this?

Sample df reproducing the situation above:

df1 <- data.frame("y" =   c(0.0166 , -0.2380 , -0.3192 , -0.2774 ,  9.3148 , 0.3142) ,
          "x1" =  c(0.0103 , -0.2347 ,  -0.3182 , -0.2793 ,  9.4638  , 0.3297) ,
          "x2" = c( -0.1838 , -0.2458 , -0.2581 ,-0.2533 , 6.7566 ,-0.0835) ,
          "x3" = c(0.3426 ,-0.0543 ,-0.4512 ,-0.0543, 10.4637 , 0.3426) ,
          "x4" = c(-0.161 , -0.270 ,-0.318, -0.280 , 8.279 , 0.169))
df1
df1_lm <- lm(y~. , data = df1)
summary(df1_lm)

I'd strongly disagree with Viktor an adjusted r² of 1 is not "luck to have a good model" it is a clear indicator of data leakage and a model that will absolutely overfit on new data!
To mediate this you need to understand your data and domain better (which means we need more infos to help).
But generally try to answer the following questions:

In any case I have not measured Y would I also be unable to measure X (the good variable)?

Is X a derivative value of Y? E.g. if you try to predict height in meters it does not make sense to include height in cm, you'd simply calculate it.

What is my use case? Do you want to explain relations / impact, do you want to predict something on new unseen data, etc.? Depending on the use case you have to deal with it in different ways.

It does not make sense however to keep this variable in your model because it invalidates the need for the model in the first place.
Either you could go ahead simply calculating Y from X and ignore all other variables (in case it is a simply unit conversion or a deterministic classifier like age > 18y --> adult) or you have to remove X to actually learn something about all the other variables (in case you want to explore impact / relation of other variables).
Also keep in mind that unless the linear regression of Y and the single good X is also r² = 1 you might have to remove or cluster two variables!


What to do with large number of collinear variables? I have this time-series dataset that has 63 features, out of which 57 were manually engineered. While checking for collinearity, I get this correlation matrix:

As can be seen there are a number of variables that are correlated/collinear. The ones that are deep red certainly need to be removed, but what about the ones on the bluer range? How do such variables (on the negative range of collinearity) effect regression models?
Also, I ran a recursive feature extraction process from sklearn.feature_extraction module and it has recommended me 39 features to be the best (at default settings). Is RFE the best strategy while dealing with such features?

First, if you're doing regression on a time series, you have to check for autocorrelation or your p-values and significance tests are going to be highly inaccurate. Also, the R^2 value you will get will be seriously misleading if you don't account for autocorrelation.

Secondly, if you are using lagged variables, it's highly likely that many of them are going to be correlated.

Collinearity affects linear regression models by making it difficult for the model to determine which coefficient is causing the effect on the dependent variable. The result is that between highly collinear variables, you will have inaccurate p-values and very small coefficients or even coefficients with the wrong signs.

Multicollinearity can be ignored when you are using some of those variables as control variables or when you don't care very much about interpretation; furthermore, collinearity doesn't matter when the variables of interest are not collinear.

Cross-validated has many good discussions on regression as well.


Forecasting using Boosting methods on Non-stationary Time Series data Theoretical Noob question -
Can we use boosting methods to effectively forecast the future after being trained on a non-stationary time series? Or do you train/fit on the residual of the training set and then add seasonality/trend components while forecasting?
Thanks in advance.

As @10xAI said, a tree-based gradient boosted approach may miss the mark for time series because it cannot forecast a growing trend.  However, we can apply gradient boosting methodology to any algorithm.  You can mess around with some code I wrote that is based on gradient boosting and decomposition: LazyProphet.  The code is badly written and I think the example data pulls break now but the method itself tends to produce some decent results.
Essentially if we do boosting with some piecewise approach we can get new changepoints at each boosting round and update our seasonality + exogenous measures.  I use binary segmentation so it ends up being very similar to a wild binary segmentation approach for change points.  Round 0 the 'trend' is just the mean/median, then you measure seasonality and set your 'y' to the original time series - (trend + seasonality).  Round 1 then finds the optimal point which splits the data into 2 and fits a trend estimator (could be the mean kind of like a tree output) then measures seasonality and adds these measures to what was found in round 0 to find the new residuals to fit for the next round and so on.  Hopefully this image makes it clearer:
I do have a much better written and more generalized approach coming soon!


How to shift rows values as columns in pandas? Input: I have csv file like below as input....

ID, Year,Specialty,AgeRange,PlaceSvc,Count, Group
101,2009,Internal,  20-29,  Office,     0,  PRGNCY
101,2010,Emergency, 20-29,  Urgent Care,0,  GIOBSENT
101,2011,Internal,  20-29,  Office,     0,  GYNEC1
102,2010,Other,     30-39,  Office,     1,  PRGNCY
102,2010,Laboratory,30-39,  Independent,1,  MSC2a3
103,2009,Laboratory,30-39,  Independent,1,  MSC2a3
103,2011,Other,     30-39,  Office,     0,  PRGNCY

Output: I want output like below...

ID,Year,Specialty_Internal,Specialty_Emergency,Specialty_Labrotory,Specialty_Other,Age20_29,Age30_39,PlaceSvc_Urgent,PlaceSvc_Office,PlaceSvc_Independent,Count,GroupPrgncy,GroupGiobsent,GroupGynec1,GroupMsc2a3

101,2009,1,0,0,0,1,0,0,1,0,0,1,0,0,0
101,2010,0,1,0,0,1,0,1,0,0,0,0,1,0,0
101,2011,1,0,0,0,1,0,0,1,0,0,0,0,1,0
102,2010,0,0,1,1,0,1,0,1,1,2,1,0,0,1
103,2009,0,0,1,0,0,1,0,0,1,1,0,0,0,1
103,2011,0,0,0,1,,1,0,1,0,0,1,0,0,0

How can i do this by pandas? or is there any other techinque to do this?

You want pandas.get_dummies.

If you call get_dummies on a categorical column, it will output the binary dummy variables you're looking for. You should then be able to merge this with your original DataFrame on the index, or construct a new DataFrame using only the columns you want.


How to pre-process the name String of a customer? I implement logistic regression to predict if a customer is a business or a non-business customer with the help of TensorFlow in Python. I have several feature candidates like name, street, zip, longitude and latitude. At them moment I am thinking of how to use the name field. The name often has repeating parts like “GmbH” (e.g. “Mustermann GmbH”) which in this context has a similar meaning to Corp. which is an indicator that the customer is a business customer. This information is useless in combination with the other parts of the name because then the name will be unique. So my question is: how should I pre-process this field so that only repeating parts will be used to predict the classification?

You may want to tokenize the strings, e.g., “Mustermann GmbH” tokenizes into "Mustermann" and "GmbH". Allow for spaces and commas certainly, perhaps also hyphens and other punctuation.

You may want to look into Natural Language Processing (NLP) if you're classifying text, but whatever method you choose should have better luck sniffing out business vs. non-business using tokens of the strings.


Mean and Variance of Feature Scaling Many people use the mean and variance of the training set to standardize the test set, instead of calculating the mean and variance of the test set and use these. Isn't risky to do that ? If no, why ?

It doesn't make sense to standardize your test set with the mean and variance computed on the test set. At best the test mean and variance will be close enough to the training mean and variance to not break your model - but why not guarantee they are the same by using the same standardization procedure for both. Typically it is best to choose one of the following:

Compute mean and variance on entire data set (train + test) and use these to standardize each set.
Compute mean and variance on train set and use these to standardize each set.

If you are not worried about leaking information from the test set then choose (1), otherwise choose (2).

If you standardize the test set with different values than the training set you might end up with train and test data from different distributions, which will lead to failure to make accurate predictions.


Best statistic books which have solutions I saw some questions which are asked about books in data science, so I think my question is not out of topic.

I am looking for a book in statistic which has complete answers for exercises.
The solutions helps me to make sure, that i learned well the concepts.

I have review the book open intro statistic, but it is not easy to find solutions for it.

Can you suggest me any book?

Some Statistical Books that I have used are:-

Introduction to Statistical Learning by Trevor Hastie and Rob Tibshirani - Good free book with R and Python examples

Elements of Statistical Learning

Statistical Computing with R


How to force weights to be non-negative in Linear regression I am using a standard linear regression using scikit-learn in python.
However, I would like to force the weights to be all non-negative for every feature. is there any way I can accomplish that? I was looking in the documentation but could not find a way to accomplish that.

What you are looking for, is the Non-negative least square regression.
It is a simple optimization problem in quadratic programming where your constraint is that all the coefficients(a.k.a weights) should be positive.

Having said that, there is no standard implementation of Non-negative least squares in Scikit-Learn. The pull request is still open.

But, looks like Scipy has implemented the same.

PS: I haven't tried the scipy version. I found it solely by googling around.


Creating features from raw accelerometer data I have a dataset containing raw 3-axis accelerometer data collected from a users lower leg and I want to create a classification model (as simple as possible) that detects if the user is sat down or standing. I have around 50,000 events (with more coming in) and these events are labeled with the correct posture and have the raw acceleration for the duration of the posture. I've created a handful of features from the raw data (e.g signal mean, range, frequency content etc) for each event but none of these clearly distinguish the events from each other (when visualising the dataset).
Is there a way of automatically generating useful features from labeled raw data that enables good seperation between outcomes?
If this is not possible, is it best to create a feature set with all the features you can come up with and then try to find the ones that explain the most variance between the outcomes? How do you do this? I've looked at PCA and LDA but they don't appear to 'pick' the best features, they just combine them to new components.
Finally, does anyone have any ideas on features that could explain the differences in standing and sitting lower leg movement. I assume that the lower leg can move in more extreme angles when sitting compared to standing but how do you describe this in features.

This is an active area of research called Human Activity Recognition. There are several public datasets available to cross-validate your methods, and you might want to start here: UCI HAR Dataset. There's a paper that accompanies the dataset that describes their preprocessing methods, so you'll want to have a look at that and see if anything helps in terms of feature engineering.
There are plenty of regression-based models that exist in the exercise science space, but I've had good results using 1D convolutional neural networks on very simply transformed data. We just downsampled the signal (by averaging) to 1 or 2 Hz and reduced the 3 axes to a single time series via euclidean vector magnitude (our subjects were performing higher-intensity activities, which often cause the sensor to rotate on their limb, changing the direction of each axis' input - reducing to EVM removes that potential confounding).
I'm not totally sure what your data looks like - do you have transitions (from sitting to standing and back)? I wouldn't use PCA (there is some argument that LDA can work if you're careful). There's something called tICA (time-series independent components analysis) but I don't think that's what you want because it's reducing the dimensionality at each time step (and each of your time steps is only 3 features). You're probably going to be going with fourier features (see the link above) or simple downsample / EVM.
Another concern here is that it's possible that the data isn't reliably separable because the intensity of each activity is low. How long is each "event" (I'm understanding each event to be a "bout" of activity with some duration)? My models have, in the past, had difficulty separating certain low-intensity activity bouts because if one person isn't moving standing up and another person isn't moving sitting down, the model can't tell 0 from 0 and is basically guessing.  Hopefully there's enough signal in there to get a good result, it just depends on your experimental setup.


How significant the variables like "id", "region code" etc are in the predictive modelling? I participated in one of the hackathon. And there the variables were like id, region, gender, age .etc.  It was a regression problem. I did scaling on the variables. But I am not sure what to do with the variables like, id and region code, numerical and ordinal variables respectively. Are they relevant and significant for modeling?

That depends on the problem and the data. Usually, at hackathons, there are at least several thousand id's and hundreds of region codes in your data and your task is to train on one dataset and predict on another dataset with new customers in existing regions.

In this case, you can exclude the id-column. Other categorical variables like region, age-group, race, etc, are usually relevant. The region code can be used as a categorical variable, but it depends on the number of regions. If there are hundreds of region codes in your data, map these codes to a larger region. For example, assume your data contains the region codes of all American counties. The easiest way to go is to map each county to its respective federal state and use the state instead of the county.


How does the cost function of LSTM works? I am searching to understand how does LSTM network work, but I couldn't find any good sources that explains how it's cost function works?

I mean I know we have a sequence of inputs x<1> to x<t>, and a sequence of y_hat<1> to y_hat<t>, but how do we calculate loss or cost function? Should we calculate  y_label<i> - y_hat<i> for each unit? If so, where do we accumulate them and when we use it? Where? How?

Is LSTM network supervised learning or I am confused?

It is my understanding that there are no differences for loss function optimization in RNN's as compared to any other neural network models.  The loss is calculated at the very end of your network output.

Say you have this network architecture for predicting a categorical variable:

Layer Type, Output Size

Input: (None, 10, 6) = (batch_size,seq len,num_features)

LSTM: (none, 10,100) =(batch_size,sequence_len,hidden_unit_size)

TimeDistributed Dense: (None, 10, 10) = (batch_size,sequence_len,num_categories)

Softmax Activation: (none, 10, 10) = (batch_size, sequence_len, num_categories)

Here, you have 10 time-steps for the sequence and the final softmax output is giving a prediction for the next value of the categorical variable at every timestep (Assuming 1-hot representation of the categorical feature).  The Y value for each time step for training is simply the 1-hot representation of the categorical value at t+1.  The LSTM layer output gives (batch_size, 10, 100).  So, you now have a 100 dimensional representation of every time-step.  You pass this to the TimeDistributed Dense and then to softmax.  So, at the very end, at the softmax output layer, you have a Y vector of size (batch_size, seq_len, 1) that contains the true output for all 10 time-steps, for every sample in the batch.  You then would use this the same exact way as any other NN model using mini-batches to calculate loss and backprop errors to network weights.


Cannot underfit/overfit on the IRIS dataset I am playing with the IRIS dataset and want to see underfitting and overfitting in action. I am using a multilayer perceptron (2 layers).

The problem is that I cannot underfit or overfit the data (see the plot below). I understand why I cannot underfit: it might happen if data is easily separable, but why I cannot overfit? The dataset capacity is 600 (# of samples (150) times # of features (4)), so I should be able to overfit using a network with a capacity bigger than that. I am trying to use a multilayer perceptron with a total # of parameters ranging from 15 to ~32000, but neither under-, nor over-fitting happens. What is going on? Maybe overfitting does not happen for the same reason, because the data is easily separable? Thank you!

IRIS is a very small data set, with not that many data points. I have found that to play with a data set, MNIST does much better job, and you can easily do over fitting and under fitting there.


Keras: Built-In Multi-Layer Shortcut Problem
In Keras for Python, I have to use multiple lines of code for a simple XOR neural network:
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation
from keras.optimizers import SGD
import numpy as np

X = np.array([[0,0],[0,1],[1,0],[1,1]])
y = np.array([[0],[1],[1],[0]])

model = Sequential()
model.add(Dense(8, input_dim=2))
model.add(Activation('tanh'))
model.add(Dense(1))
model.add(Activation('sigmoid'))

sgd = SGD(lr=0.1); model.compile(loss='binary_crossentropy', optimizer=sgd)

model.fit(X, y, show_accuracy=True, batch_size=1, nb_epoch=1000)
print(model.predict_proba(X))

Although the network above is quite small, the current implementation may become frustrating with deeper networks.
Question
Is there a built-in shortcut in Keras that quickly adds multiple layers?
For example:
[2,3,4,5] would be a network with 2 input neurons, 3 neurons in the first hidden layer, 4 neurons in the second hidden layer, and 5 output neurons. I use the term built-in because otherwise I can loop through the example array, and add a layer for each element.

There is no shortcut syntax that goes as far as accepting [2,3,4,5] as a param and create a model. However, it would be very simple for you to create this as a Python function yourself, provided in your case you already have made the decisions about activation functions, the types of layer etc. The need to make those decisions and have them available in the Keras API means that Keras itself does not offer such a short form build function.

You can make the model building slightly less verbose by using a list of layers when you instantiate the model, instead of adding them afterwards:

model = Sequential([
  Dense(8, input_dim=2),
  Activation('tanh'),
  Dense(1),
  Activation('sigmoid')
])

However, if you want to try variations of a model, where the only things you change are the number and size of hidden layers, then you can write a short Python function to encapsulate that requirement. Here's an example (I'm sure you already know how to do this, just included for completeness):

def build_model(hidden_layer_sizes):
  model = Sequential()

  model.add(Dense(hidden_layer_sizes[0], input_dim=2))
  model.add(Activation('tanh'))

  for layer_size in hidden_layer_sizes[1:]:
    model.add(Dense(layer_size))
    model.add(Activation('tanh'))

  model.add(Dense(1))
  model.add(Activation('sigmoid'))

  return model

If you take this approach, you may find you end up parametrising other choices such as input size (as you try some feature engineering), hidden layer activation function, whether to use a Dropout layer etc. Again, it is this need to define all the other choices in a typical network that lead to Keras' design. The best you can do is compress down the choices for your case, with a custom function.

I'd like to address this comment in your question:

  Although the network above is quite small, the current implementation may become frustrating with deeper networks.

In practice, I have not found Keras' design difficult to use for deep networks. I typically write a separate build function, and parametrise a few things, such as input dimensions. However, I typically don't loop through a list of different layer sizes as the main param of the build function. Instead, I find the more verbose approach just fine, even when trying variations of network size/shape (I guess that might change if I wanted to grid search including layer sizes). I think that is because I find the function names very easy to read - even with a screen full of .add() functions, I can see quite quickly what the NN structure is.


How to use fminunc in Matlab on only some specified parameters of a function So I want to use fminunc(fun, X0) in Matlab on my function fun. The problem is that fun is a function of three parameters:(X, y, X0), all of which are vectors. What I want to do is to find the minimum, given the parameters X, y. I.e. the vector X0 which minimises the function for some given values X, y. How would I go about doing this?

What we have here is a function fun(X, y, X0) and fminunc(fun,X0). You know X, and y, lets call them X* and y*, so we could replace fun(...) by fun*(X0)

% function to be minized
fun = @(X, y, X0) (dot(X,y) + X0)^2
X_star = [1,2,3]
y_star = [4,5,6]
$ same function but with X and y filled in
fun_star = @(X0) fun(X_star, y_star, X0)
X0_star = 0
fminunc(fun_star, X0)


Weighted sum of word vectors for document similarity I have trained a word2vec model on a corpus of documents. I then compute the term frequency (the same Tf in TfIDF) of each word in each document, multiply each words Tf by its corresponding word vector (this is the weighted part), and sum each of these weighted vectors together element wise to obtain a single vector for a document.

Is this method valid?

An example, to make it clearer. Take the document:

"The car drives on the road"

The Tf for each word would be {'the':0.3333, 'car':0.1666, 'drives':0.1666, 'on':0.1666, 'road':0.16666} (obtained by taking the word count of a word and dividing by the total number of words in the document). If we have a trained word2vec model, we can do the following

$$0.333*\begin{bmatrix}
    the_0 \\
    the_1 \\
    \vdots \\
    the_n
\end{bmatrix}
+ 0.1666*\begin{bmatrix}
    car_0 \\
    car_1 \\
    \vdots \\
    car_n
\end{bmatrix}
+ ...$$

where each of the column vectors is the word vector for that word. The final result is an $n\times 1$ dimensional vector representing the document.

$$
\begin{bmatrix}
    0.333*the_0 + 0.1666*car_0 + \dots \\
    0.333*the_1 + 0.1666*car_1 + \dots \\
    \vdots \\
    0.333*the_n + 0.1666*car_n + \dots
\end{bmatrix}
$$

I appreciate there are other methods such as doc2vec that aim to do very much the same, but in a much more sophisticated way. But is my method valid / is there anything blatantly wrong here?

I have tested this method and even used some document similarity metrics (yes, after normalizing the vectors of course), and yielded some good results for my industrial application. But I want to know more about the academics of this method.

The nice thing about this way, is that by using the word2vec vectors, similarity queries between documents yield very good results due to semantic similarities (euclidean closeness) between word vectors, even if different words are used across documents; this is something TfIDF cannot do as each word is treated differently.

Thanks in advance!

Yes, your method is valid and it has been studied before it is known as Mean of Word Embeddings (MOWE) or Sum of Word Embeddings (SOWE), although your method is more a weighted mean of vectors. I think that a good starting point for knowing more about the academics of the method is this paper: How Well Sentence Embeddings Capture Meaning. It discusses your method and give some pointers to other papers that also discuss the validity of the method.


Deployed Web service When I deploy my web service in AZURE Machine learning, is it possible to have NULL or empty in my test data set? Or I have to fill all the information to get the results of my prediction?

In training set:

You would better use some methods to "presume" certain values...average / pro-rata......which you could find in various learning sources

Leaving them as blank (not sure of Azure but in a low level programming platform) would cause either inaccuracy or problems.

For testing set:

You are not changing the Learned Parameters, but still that would probably causing some inaccurate measurement but Azure ML may have done something internally.

Reference from MSDN:

  How to deal with missing values?

  To deal with missing values, it is best to first identify the reason for the missing values to better handle the problem. Typical missing value handling methods are:

  Deletion: Remove records with missing values
  Dummy substitution: Replace missing values with a dummy value: e.g, unknown for categorical or 0 for numerical values.
  Mean substitution: If the missing data is numerical, replace the missing values with the mean.
  Frequent substitution: If the missing data is categorical, replace the missing values with the most frequent item
  Regression substitution: Use a regression method to replace missing values with regressed values.


Vector space model cosine tf-idf for finding similar documents Have corpus of over million documents

For a given document want to find similar documents using cosine as in vector space model

$d_1 \cdot d_2 / ( ||d_1|| ||d_2|| )$

All tf have been normalized using augmented frequency, to prevent a bias towards longer documents as in this tf-idf:

$tf(t,d)=0.5+0.5\frac{f(t,d)}{\mathrm{max}\{f(t,d): t\in d\}}$

Have pre-calculated all $||d||$
Have the values for the denominator pre-calculated
So for a given $d1$ need to score over 1 million $d2$
Have a threshold of 0.6 cosine for similarity

I can observe that for a given $||d_1||$ there is a fairly narrow range of $||d_2||$ for cosine $\ge$ 0.6
For example in one search for similar for a cosine of $\ge$ 0.6 and a $||d_1||$ of 7.7631 then $||d_2||$ range from 7.0867 to 8.8339
Where outside the threshold of cosine 0.6 $||d_2||$ range from to 0.7223 to 89.3395
This was with standard tf document normalization
It is looking at a LOT of $||d_2||$ that don't have a chance of being a cosine 0.6 match

Finally the question:
For a give $||d_1||$ and cosine of >= 0.6 how can determine the range of $||d_2||$ that have a chance?
Which $||d_2||$ can I safely eliminate?

I also know the number of terms in $d_1$ and $d_2$ if there is term count range.

Via experimentation
$||d2|| > .8 ||d1||$ and $||d2|| < ||d1|| / .8 $
seems to be safe but hopefully there is range that is proven to be safe

Created some test cases with a very some unique terms, some not so unique, and some common.  Sure enough you can take the most unique term and increase that frequency in the compare.  The numerator will (dot product) go up and so will ||compare|| and will get a cosine very close to 1.

Kind of related and NOT the question.
I am also using the tf-idf to group documents into groups.
The customer base I am selling into are used to near near dup groups.
There I am taking a related approach in I look as the smallest term count and evaluate it against term count up to 3x.  So a term count of 10 looks at 10  thru 30 (4-9 already had their shot at 10).  Here I can afford to miss one have it picked up in another.  I am 10% done and the biggest ratio is 1.8.

Please identify the flaws in this analysis
As pointed out by AN6U5 there is a flaw in this analysis
It is no longer a cosine if the document is normalized on weighted
And as pointed out by Mathew also cannot conclude  d1⋅d2≤d1⋅d1
I am still hoping for something to give me a hard bound but people that seems to know this stuff are telling me no
I don't want to change the question so just ignore this
I will do some analysis and maybe post a separate question on document normalization
For the purpose of this question assume the document is normalized on raw tf
Sorry but I am just not good with what ever markup is used to make the   equations
So in my notation
||d1|| = sqrt(sum(w1 x w1))
d1 dot d2 = sum(w1 X w2)
Assume d1 is the shorter document
The very best d1 dot d2 that can be achieved is d1 dot d1
If d1 is marry 100 paul 20
And d2 is marry 100 paul 20 peter 1
Normalized
d1 is marry 1 paul 1/5
d2 is marry 1 paul 1/5  peter 1/100
Clearly marry and paul have the same idf in both documents
The best possible d1 dot d2 is d1 dot d1
The maximum possible match to d1 is d1
cos = d1 dot d1 / ||d1|| ||d2||
square both sides
cos X cos = (d1 dot d1) X (d1 dot d1) / ( (d1 dot d1) X (d2 dot d2) )
cos X cos = (d1 dot d1) / (d2 dot d2)
take the square root of both side
cos = ||d1|| / ||d2||
is ||d2|| not bounded by the cos?
If I just use ||d2|| >= cos ||d1|| and ||d2|| <= ||d1|| / cos I get the computational speed I need

Unfortunately, the math simplifies to show that you can't rigorously justify restricting the cosine similarity comparison of the vectors based on their lengths.

The key point is that the cosine similarity metric normalizes based on length, so that only the unit vectors are considered.  I know this isn't necessarily the answer that you wanted, but the math clearly shows that the cosine similarity metrics is agnostic to vector length.

Lets look at the math in more detail:

You are applying a cosine similarity metric and requiring that that metric be larger than 0.6:

$$similarity=\cos{(\theta)}=\frac{\mathbf{A}\cdot\mathbf{B}}{||A|| ||B||}\geq0.6$$.

But the scalar lengths on the bottom can be distributed into the cross products above (distributive property):

$$\frac{\mathbf{A}\cdot\mathbf{B}}{||A|| ||B||} = \frac{\mathbf{A}}{||A||}\cdot\frac{\mathbf{B}}{||B||}=\hat{\mathbf{A}}\cdot\hat{\mathbf{B}}$$.

Now $\hat{\mathbf{A}}$ and $\hat{\mathbf{B}}$ are vectors that point in the same direction as $\mathbf{A}$ and $\mathbf{B}$ but they have been normalized to length one.  So the definition of the cosine similarity metric is the take the original vectors, normalize them to length one, and then measure the dot product of the unit vectors.

Therefor:

$$similarity=\cos{(\theta)}=\frac{\mathbf{d1}\cdot\mathbf{d2}}{||d1|| ||d2||}=\hat{\mathbf{d1}}\cdot\hat{\mathbf{d2}}\geq0.6$$

depends only on the orientation of the vectors and not on their magnitude (i.e. length).

Reconciling this with what you are doing:

Despite what the results of the linear algebra show, you might still be seeing a statistically significant result.  Practically speaking you may be finding that the statistics show that length restrictions are valid for your data.  For instance, you might be finding that tweets never share a cosine similarity $\geq0.6$ when compared with Tolstoy's "War and Peace".  If your statistics look good for using $||d2|| > .8 ||d1||$ and $||d2|| < ||d1|| / .8$ then I suggest you go with it as these type of canopy restrictions are very useful in saving computing time.

You can perhaps reconcile what you have been doing with distance metrics by also considering the Euclidean distance.  Where as the cosine similarity only returns a value between -1 and 1 based on the angle between the two vectors, the Euclidean distances will return values that depend on the lengths of the two vectors.  In some sense, you are combining aspects of the Euclidean distance with cosine similarity.

It makes fairly good sense to require that the relative lengths be within 25% of one another in the sense that this combines an aspect of Euclidean distance to create group-by canopies, which cuts computation time, then length agnostic cosine similarity can be used as the final determinant.

Note that 1/.8 =1.25, so d2>=.8d1 is a tighter restriction than d2<=d1/.8.  I suggest using d2>=.75d1 and d2<=1.25d1 as this is symmetric.

Hope this helps!


How do I control sensor data readings against (measured) outside influences? I need some hints in this problem I have.
I have a dataset of electro-dermal activity readings that are, by nature, influenced by the movements of the person that is being surveyed. In the same dataset, however, I also have accelerometer data that I could use to "control" the data. Meaning that, I hope, I could somehow factor in the person's accelerometer readings when looking at EDA readings and filtering them/correcting against posture changes made obvious in the accelerometer values.
As an example, fig. 1 shows EDA (also called galvanic skin response, or gsr) readings with higher peak prominence at the beginning of the experiment:

This second figure shows accelerometer readings in x, y and z directions, appearing to show more activity during these phases of higher peak prominence:

How do I control my EDA values against these accelerometer readings? Any hints to literature or math will be greatly appreciated.
Thanks in advance!

In short, I think the focus of your question is mostly on how to deal with covariates (watch out for the not always clear use of this term and its synonyms - see here and here. What I mean by covariates is "In statistics, a covariate represents a source of variation that has not been controlled in the experiment and is believed to affect the dependent variable.")
If we first take a step back and ask what you really want from your analysis, it is to my understanding to investigate the relationship between two (or more) variables. These would be e.g. EDA values on one hand and some output variable such as e.g. emotional states based on heart beat data on the other (since I don't have much information on your outcome variable(s) at this point, I will simply call them y). Thus, you have EDA values as an independent variable and y as a dependent variable. So far, so good - bringing together dependent and independent variables statistically can be done in many ways, ranging from even just calculating a correlation between the two (given they are both numeric variables) across fitting linear models and up to machine learning techniques etc. Let us for now go with setting up a linear model:
Short-hand notation: y = EDA
Same model written out more detailed: y = intercept + EDA + error
You already know, however, that you have this other outside influence that affects your EDA-values. Luckily, you even tried to measure/quantify this influence via the accelerometer. Thus, besides dependent variables that you are interested in and the independent variables you are interested in, you now have a third type of variable: the independent variables you are not really interested in, but that you expect to have an influence on the variables that you are interested in. This is what I mean by covariates. Taking a covariate into a model can greatly improve the statistical model in terms of its ability to actually analyze the variables we are interested in. In other words: In the optimal case, a covariate explains so much of the unexplained noise in the data, that the variability that is left lets us draw conclusions about the relationship between the other variables that we could not see before. So the model from above may become:
y = EDA + COVARIATE
y = intercept + EDA + COVARIATE + error
Thus, the question comes up on how to decide whether or not to include such a covariate in your model/analysis. One analysis that is very much in this realm is ANCOVA - analysis of covariance.
I will stop here to make sure I understood the problem correctly and would be glad to hear whether this is helping so far.


Calculating an estimate of KL Divergence using the samples drawn from distributions Given two sets of samples drawn from two different distributions, is it computationally possible to get an estimate of KL-Divergence between the two distribution using these samples?
Here I am assuming the dimensionality of the two distributions is high (say d). To compute the estimate, we first need to discretize the entire space and then estimate probabilities based on the frequencies. Let us say, we discretize each dimension into p bins. Then the total number of grids in the space will be $p^d$. So we need to compute the probabilities of the two distributions for $p^d$ grids, which is exponential in time. Hence I assume we cannot compute an estimate of KL Divergence using the samples for any practical problem.
I wanted to check if this explanation is correct or if I am missing anything. Could someone assert if this rationale is correct?

Check this article. They use k-NN to interpolate the values of P(x) and Q(x), so that you can use the KL-divergence formula with 'approximated histograms'.


How do you use KS-test in a data science report? I'm writing a data science report, I want to find an exist distribution to fit the sample. I got a good looking result  , but when I use KS-test to test the model, I got a low p-value,1.2e-4, definitely I should reject the model.
I mean, whatever what distribution/model you use to fit the sample, you cannot expect to have a perfect result, especially working with huge amount of data. So what does KS-test do in a data science report? Does it means only if we got high p-value in KS-test then the model is correct?

In your case, the null hypothesis $H_0$ is that your sample follows your the distribution that your model has learned. The alternative hypothesis $H_1$ is that it follows some other distribution. Assuming you have fixed your significance level $\alpha$ to be $0.05$ (the most common choice for $\alpha$, but up to you if you want to go lower), getting a p-value lower than that means you should reject the null hypothesis.
The p-value can be interpreted as the probability of a type I error, in other words a false positive: the probability that you reject the null hypothesis when it is in fact true. In your case, rejecting the hypothesis means stating that there is statistically significant evidence that the distribution your model has learned is not the underlying distribution of the sample. So yes, you would like as large a p-value as possible.
You are using a Kolmogorov-Smirnov test to compare your sample to a reference distribution, in this case, so it's a one-sample KS test. The way I would put it is that getting a high p-value means that: "it is highly unlikely that your model has learned a wrong distribution". In other words, it is highly likely it has learned a pretty good approximation of the underlying distribution. However, nothing is certain when doing statistical hypothesis testing!
I'm not sure what you're showing on your plots though, since there doesn't seem to be an empirical cumulative distribution function on them (lines look smooth).


Creating more than one worker nodes for local windows machine I am using windows laptop. And I installed apache spark for my laptop. And I try to measure spark performance by changing spark components. because of that I want to create more than one worker nodes for my local windows machine. first of all I want know is possible to add more worker nodes in my windows machine??
Next thing is if it is possible what are the steps I want to follow??
If some one know about this things please tell me because i search lot time with google but I couldn't find any document regarding that(windows machine)??

There is no value in this. local[*] master will already use all cores and you can use as much local memory as you like already. Starting a worker just adds overhead.


Right order for Data preparation in Machine Learning For the below mentioned steps of data preparation

Outlier detection/treatment
Data imputation
Data scaling/standardisation
Class balancing

There are two sub questions

Should each of these steps performed post test/train split?
Should it be done on test data?

Would appreciate explanation for each step individually.

Data processing is a state of the art. You need your gut or intuition to be able make a good model. You have to perform data cleaning first. Garbage IN = Garbage OUT. If you don't clean data your model will be a garbage itself. Data Scientist spend 20%-30% of the time in cleaning data. There is no rule for cleaning data. But you could follow few guidelines for Data cleaning.

Remove Duplicate values:- You need to be an artist to recognise the duplicate data. You could drop the duplicate records and just keep one record out of all duplicate records. pandas.DataFrame.drop_duplicates will perform data cleaning for you.
Data integrity:- you need to ensure whether the data is accurate. If few fields of Age column contains Male or Female that actually don't make any sense. Or say Age contains 200 years for a person who in actual is 20 years old. You could say its outlier detection as well. But it doesn't necessarily be an outlier.
Null Values (Data Imputation):-
You could use sklearn library or pandas library for this. Both comes with its own features.
Tip: Use latest KNNImputer. Imputation for completing missing values using k-Nearest Neighbors. It gives far better results. Reference

PERFORM SPLIT NOW :-

To avoid Data Leaks this has to be done. Standardising data before the split means that your training data contains information about your test data.

Column Standardisation: It is required to scale the data. By doing this, you transform your data such that each column have a mean equal to zero and Variance of value 1. This is implemented by default in many algorithms. This will help your model to execute faster and eliminate problems related to scaling. (Every column is of same scale and model does't give preference to one column which has high values.)
Encode categorical features.

NOTE:- Splitting is performed before Feature Scaling (Standardisation). StandardScaler is fit_transformed on training data and only transformed on test data.


Different significant variables but same Adjusted R-squared value I performed a multiple linear regression on 64 variables with 3 different models:

Performed Multiple Linear Regression on all 64 variables
Perform Feature Selection with Random Forest and then perform multiple linear regression on selected features
Performed Stepwise Linear Regression

I achieved the same adjusted R squared value for all 3 models but different significant variables. How should I make sense of this? Which model should I go with?

Will appreciate any advice! Thank you!

It seems that removing features was not really helpful in bringing up the model fit. Differences in significance of features may be due to exclusion.

One thing you should try is regulation by lasso/ridge regression. In R, this can be easily implemented by the glmnet package. Here is a tutorial. This is the best feature selection method imo since there is a mathematical rational behind (see Introduction to Statistical learning for more background).

Lasso can shrink features to zero (viz drop them). Ridge cannot shrink features to zero. Just give it a try.

Hint: In linear regression with continuous features you can also add polynomials to your regression to increase fit using the poly function. You could also see if regression splines help you to deal with hidden non-linearity. The gam packages is a really good start. Here are the docs.

The book Introduction to Statistical Learning covers these topics in a very good way and comes with useful R examples. Here is the code.


What's the difference between sequence preprocessing and text preprocessing in Keras? In Keras, we mainly have three types of preprocessing, i.e., sequence preprocessing, text preprocessing, and image preprocessing. However, for me, I think the meanings of the word "sequence" and "text" are the same. How to understand the differences between these two preprocessing operations?

In tf.keras.preprocessing.text (docs) you have utilities to process discrete token sequences, normally used to represent text.
In tf.keras.preprocessing.sequence (docs) you have utilities to process both continuous value sequences (normally used to represent time series) like TimeSeriesGenerator, and discrete token sequences (i.e. text), like the skipgrams function.


Training a neural network with TWO possible correct outputs for one input I have a system as a black box that has two correct outputs for a single input sample.
now I want to train a neural network to generate at least one of the correct outputs for that input sample.
what should I do ♂

With Keras, you could use the functional API, to estimate a model with two outputs („multioutput“). Simply train the model on two outputs like:
# Outputs
out1 = Dense(1)(x)
out2 = Dense(1)(x)

# Compile/fit the model
model = Model(inputs=Input_1, outputs=[out1,out2])
model.compile(optimizer = "rmsprop", loss = 'mse')

# Add actual data here in the fit statement
model.fit(train_data, [train_targets,train_targets2], epochs=500, batch_size=4, verbose=0, validation_split=0.2)

Find the full model here: https://github.com/Bixi81/Python-ml/blob/master/keras_multitarget_functionalAPI.py


How to compute the maximum likelihood hypothesis? The Bayes theorem states that:

\begin{equation}
P(h|D) = \frac{P(D|h)P(h)}{P(D)}
\end{equation}
where $D$ is the dataset and $h$ is an hypothesis from the hypothesis space $H$.   Now (I'm not sure so if I'm wrong please correct me) I can consider:

$P(h|D)$ = the probability $h$ has generated the dataset $D$. More specifically, for each $h$ we have a probability that it has generated the dataset $D$.
$P(D|h)$ = the probability that $D$ has been generated by $h$. More specifically, for each possible dataset $D$, a certain hypothesis $h$ (that we have) can have generated it.

And I can represent them visually, for example:

Now, if we know the prior probability $P(h)$ then we can compute the maximum a posteriori hypothesis with the following formula:

\begin{equation}
h_{MAP} = argmax_{h \in H} P(h|D) = argmax_{h \in H} \frac{P(D|h)P(h)}{P(D)}
\end{equation}

Otherwise, we can consider the maximum likelihood hypothesis:

\begin{equation}
h_{ML} = argmax_{h \in H} P(D|h)
\end{equation}

At this step I don't understand how I compute $h_{ML}$ because if I consider $P(D|h)$ represented as in the previous example in the cartesian space we have $D$ in the x-axis, so if I consider the $argmax P(D|h)$ I will find the best $D$ and not the best hypothesis $h$.

What am I doing wrong? Are probabilities $P(h|D)$ and $P(D|h)$ not well interpreted in the cartesian space?

We can define a function $f(D,h) =P(D|h)$, that is it is a function of both $h$ and $D$ and we want to maximize it.

In the event that $D$ is already fixed to be $\hat{D}$, then the goal should be to maximize the function $$g(h) = f(\hat{D}, h) = P(\hat{D}|h).$$

We can plot the function $g$, here $g$ is a function of $h$ rather than $D$.


How to reconstruct a scikit-learn predictor for Gradient Boosting Regressor? I would like to train my datasets in scikit-learn but export the final Gradient Boosting Regressor elsewhere so that I can make predictions directly on another platform.
I am aware that we can obtain the individual decision trees used by the regressor by accessing regressor.estimators[].tree_. What I would like to know is how to fit these decision trees together to make the final regression predictor.

There are two estimators i.e. The initial predictor and the sub-estimators

init_estimator
The estimator that provides the initial predictions. Set via the init argument or loss.init_estimator.
estimators_
ndarray of DecisionTreeRegressor of shape (n_estimators, 1)
The collection of fitted sub-estimators.

Prediction after the first (i.e. init) estimator is controlled by the learning rate.
You can get the prediction as done in the below code -
trees = model.estimators_

x  = x_test.iloc[10,:].values # A sample X to be predicted
y_pred = model.init_.predict(x.reshape(1, -1)) # prediction from init estimator

for tree in trees:
    pred = tree[0].predict(x.reshape(1, -1)) # prediction from sub-estimator

    y_pred = y_pred + model.learning_rate*pred  # Summing with LR
y_pred


How to convert vector values to fit k-means algorithm function? I have a set of user objects that I want to group using a $k$-means function from their quiz answers. Each quiz question had predefined answers with letter values "a", "b", "c", "d". If a user answers the question #1 with letter "b", I put this answer into vector $(0, 1, 0, 0)$.  The $k$-means function I have to use takes a two-dimensional array of numbers as an input vector (in this case array[user][question]), and I can't figure out how to use it, because, instead of a number value representing a user's answer to question, I have a vector input.
How can I convert my vector values to numbers so that I can use my $k$-means function?

You are 95% there, you just have one hangup...

The vectorization that you are doing is alternatively known as binarization or one-hot encoding.  The only thing you need to do now is break apart all of those vectors and think of them as individual features.

So instead of thinking of the question one vector as $(0,0,1,0)$ and the question two vector as $(0,1,0,0)$, you can now think of them as individual features.

So this:

-      q1,        q2
-      (a,b,c,d), (a,b,c,d)
user1  (0,0,1,0), (0,1,0,0)
user2  (1,0,0,0), (0,0,0,1)

Becomes this:

-      q1a,q1b,q1c,q1d,q2a,q2b,q2c,q2d
user1  0   0   1   0   0   1   0   0
user2  1   0   0   0   0   0   0   1

And you can think of each one of those binary features as an orthogonal  dimension in your data that lies in a 8-dimensional space.

Hope this helps!


speed up the fit of model What are the ways to speed up the fit of the model on large files (more than 200 mb)? I tried to divide into pieces (chunks) or use dask but the effect is unsatisfactory. I use 16GB RAM and i7 2.2GHz processor 4 core. It remains to buy a cloud service or use a better computer?
My clasiffier

clf = xgb.XGBClassifier(n_estimators=500,
                        n_jobs=4,
                        max_depth=9,
                        learning_rate=0.05,
                        subsample=0.7,
                        colsample_bytree=0.7,
                        missing=-999,
                        gamma = 0.1)
# Number of variable > 400
# Number of rows > 200000

One thing you could try is to use a "normal" Logit, as it is computationally not very expensive. Using Lasso, Elastic Net, or Ridge can yield good results (often similar to boosting) shown here.

Here is a recent code example for Logit/Lasso.

If you would like to stick with boosting, you may check LightGBM as it tends to be faster. XGBoost tends to be "heavy" in terms of data handling. LightGBM (as the name indicates) aims at resolving this problem and I have had good experiences so far.

Here is the link to the the LightGBM docs.


One-Hot Vector representation vs Label Encoding for Categorical Variables I have one categorical variable of string type in my dataset. I need to convert it to numerical value for further processing. I know standard way to represent categorical data is to use one-hot encoding. But that will convert each entry of the variable to a vector.

LabelEncoder of sklearn converts each entry to a scalar value. I realise this is a very naive and possibly stupid question but which representation is more commonly used and is there a reason for the bias?

The main difference I can think of is that using one-hot encoding will mean that all your strings will be at the same (hamming) distance from each other, while using a scalar value means that distances between the resulting features will be meaningless (it may encode "red" as 1, "blue" as 2 and "green" as 3, but there is no reason why red is more similar to blue than to green).


Predict set elements based on other elements Sorry if this has been answered before but could someone help me with solving the following problem:

Each symbol in a dataset has a set of labels. Given a set of labels how can we predict more labels for that set?

Or to attempt a more formal wording: let a set of sets $S = \{ s_1, s_2, s_3,  ... \} $ where $ s_i \subseteq L $ and $ L = \{ l_1, l_2, l_3,...,l_m\} $ where $L$ is the finite set of labels. Given a set of labels $Y = \{ y_1 ,..., y_n \} $ where $Y \subseteq X $ what is the probability $ P(r = l_i) \forall l_i \in L$ so that $Y \cup \{r\} \in S $.

Maybe you could use some algorithm that solves "Market Basket Analysis". The problem is explained here:

  Market Basket Analysis is a modelling technique based upon the theory
  that if you buy a certain group of items, you are more (or less)
  likely to buy another group of items. For example, if you are in an
  English pub and you buy a pint of beer and don't buy a bar meal, you
  are more likely to buy crisps (US. chips) at the same time than
  somebody who didn't buy beer.

http://albionresearch.com/data_mining/market_basket.php

One example of such an algorithm is: https://en.wikipedia.org/wiki/Association_rule_learning


How to properly store a list of pandas dataframes, so that I can import them in a new python script? I'm trying to make a GUI in PyQt5 to present the results of my research. The results are around 800 pandas dataframes, stored in lists (so that I can call them individually according to the list index).

In this GUI, I want to have Radiobuttons to tell the program which list to use, and an Input to tell the program which one of the dataframes in the corresponding list to show. Now, I have made a crude algorithm (without a GUI) to do this, and it works, however the lists and, thus, the dataframes, are stored as variables in the original Jupyter notebook, which is full of the convoluted and inelegant code I used to process and store these results.
I managed to save them using %store, and was able to use them in a new Jupyter notebook, but for the GUI I will need to use a separate Python script file. My problem is that this command doesn't work when I try to use it in the script file, to restore and use the variables there.

tldr: Is there a proper way to store lists of dataframes, exactly as they are (I don't want the indexing to be messed up), so that I can call them/import them in a separate Python script, and use them in the GUI?

P.S.: As you can understand, I'm new to Python, so I'm sorry if my programming methods seem inelegant and my question trivial.

Use pickle package, it's meant to export and/or load python objects.

You can find information of how is done in this topic of stack overflow: https://stackoverflow.com/questions/4529815/saving-an-object-data-persistence


Why and how BERT can learn different attentions for each head? https://towardsdatascience.com/deconstructing-bert-distilling-6-patterns-from-100-million-parameters-b49113672f77

I read the blog above. It visualizes that different color/head has different attention of words.

Based on my understanding, the code implementation of each head is almost the same.

After re-read the code, I am wrong. Each head is different part of the tensor's last dim.

Each head is to split the last dim of the tensor.

For example, head number is 12 and last dim of the tensor should be 12*N.

Each value of the tensor's the last dim represent a feature for the tensor.

But it seems the training could let different heads be different representations. This is amazing.

Think about CNN. CNN also has a lot of kernels. And the info of kernels may have some redundancy. As the init values of kernels are different, the kernels also learn some different info.

Then watch the figure in the link again, BERT's attentions also have some redundancy.


Training of Region Proposal Network (RPN) There are some interesting literature about RPNs (Region Proposal Network). The most concise and helpful documentation that I found so far is the following: https://www.quora.com/How-does-the-region-proposal-network-RPN-in-Faster-R-CNN-work?share=1.

But there is something that I still don't understand through my various lectures. RPNs are designed to propose several candidate regions. From which, a selection will be done to know which candidates fits our needs.

But, RPNs and neural network in general are deterministic. Thus, once trained, they will always produce the same output for a given input; there is no way to query new candidates given the same input image. As far as I understood, RPNs are trained to produce a fix number of proposal, for each new image. But how the training work then? If the RPN has to produce 300 candidates, what should be the labeled data that we use for the training, knowing that a training image probably won't have more than 5 golden truth bounding boxes?

And then, knowing that the bounding box sizes are not consistent among candidates, how does the CNN behind operates with the different size of the input?

The first answer in your commented link answers one point about how region proposals are selected. It is the Intersection Over Union (more formally the Jaccard Index) metric. So how much of your anchor overlaps the label. There is usually a lower limit set for this metric to then filter out all the useless proposals, and the remaining matches can be sorted, choosing the best.

I recommend reading through this excellently explained version of a proposal network - Mask-R-CNN (Masked Region-based CNN).
If you prefer looking at code, there is the full repo here, implemented in Keras/Tensorflow (there is also a PyTorch implementation linked somewhere).

There is even an explanatory Jupyter notebook, which might help make things click for you.


Can 1D-CNN method apply to real-time time series classification? So I got an EEG dataset with shape (data points, 19), each row's shape (1,19) represent 1 second of EEG.
I read much research on EEG classification that used many Deep Learning method and 1D-CNN is one of that.
My question is as the input of the 1D-CNN must have multi-row data, ex (50,19) for my dataset so it can filter a input matrix. But I want to predict new data row by row ((1x19) shape), can 1D-CNN use this input for predict new data?

The inputs of a CNN must have the same shape during prediction as when it was trained. So if you have a CNN trained on 50 time-steps windows, then you can make predictions on a stream of input by updating the data in this window continuously. Each new time step you push the most recent row onto the end, and drop the earliest row.
Of course it is possible to make a CNN model that take 1x19 as the input also. But such a model would not have any ability to detect patterns over time, which would probably limit its predictive power quite a lot.


What are the use cases for Apache Spark vs Hadoop With Hadoop 2.0 and YARN Hadoop is supposedly no longer tied only map-reduce solutions. With that advancement, what are the use cases for Apache Spark vs Hadoop considering both sit atop of HDFS? I've read through the introduction documentation for Spark, but I'm curious if anyone has encountered a problem that was more efficient and easier to solve with Spark compared to Hadoop.

Hadoop means HDFS, YARN, MapReduce, and a lot of other things. Do you mean Spark vs MapReduce? Because Spark runs on/with Hadoop, which is rather the point.

The primary reason to use Spark is for speed, and this comes from the fact that its execution can keep data in memory between stages rather than always persist back to HDFS after a Map or Reduce. This advantage is very pronounced for iterative computations, which have tens of stages each of which is touching the same data. This is where things might be "100x" faster. For simple, one-pass ETL-like jobs for which MapReduce was designed, it's not in general faster.

Another reason to use Spark is its nicer high-level language compared to MapReduce. It provides a functional programming-like view that mimics Scala, which is far nicer than writing MapReduce code. (Although you have to either use Scala, or adopt the slightly-less-developed Java or Python APIs for Spark). Crunch and Cascading already provide a similar abstraction on top of MapReduce, but this is still an area where Spark is nice.

Finally Spark has as-yet-young but promising subprojects for ML, graph analysis, and streaming, which expose a similar, coherent API. With MapReduce, you would have to turn to several different other projects for this (Mahout, Giraph, Storm). It's nice to have it in one package, albeit not yet 'baked'.

Why would you not use Spark? paraphrasing myself:

Spark is primarily Scala, with ported Java APIs; MapReduce might be friendlier and more native for Java-based developers
There is more MapReduce expertise out there now than Spark
For the data-parallel, one-pass, ETL-like jobs MapReduce was designed for, MapReduce is lighter-weight compared to the Spark equivalent
Spark is fairly mature, and so is YARN now, but Spark-on-YARN is still pretty new. The two may not be optimally integrated yet. For example until recently I don't think Spark could ask YARN for allocations based on number of cores? That is: MapReduce might be easier to understand, manage and tune


How to find similar time series? I've got a collection of yearly data (one value per year per category), and I'd like to find series that are most similar to one another.

Example data is here.

I don't know much about data science, but it seems like cosine similarity might be the way to go? If so, how do I account for the nil values in the datasets?

Since the time-series are annual, the data points you have for each time-series are limited and also quite distant (the values are 1 year apart). So I wouldn't use Dynamic Time Wrapping on your data.

If you are interested in comparing the patterns, a very simple approach would be Pearson's correlation. Keep in mind that this will not compare the actual values but the patterns (i.e. if the values have similar fluctuations with the years, so for example time-series [1 2 3 4] would have higher correlation with [5 6 7 8] than with [1 1 2 2])

If you are interested in both values and pattern, I would use a distance-based metric: Euclidean Distance, Manhattan distance etc. I believe you will find this post interesting, where the mathematical background of similarity is explained. Also, python implementations of several distance metrics in python (including cosine-similarity) can be found in this blog-post.


Why is the numeric column treated as a categorical column in Microsoft Learning? Learning Microsoft's DP-100 and saw this notebook:
The part of its code which I have question with is
# Define preprocessing for categorical features (encode the Age column)
categorical_features = [7]
categorical_transformer = Pipeline(steps=[
    ('onehot', OneHotEncoder(handle_unknown='ignore'))])

here is the link for the entire notebook for you to see the full picture. I cannot copy the full pages of code here.
Here is how the data look like:

My question is: Why is the DiabetesPedigree column(which is the 7th column) treated as a categorical feature?
To me, it looks nothing different from the rest of other columns(except the Diabetic column).
If DiabetesPedigree could be treated as categorical why not Pregnancies?
What are the rules to tell whether a column of data is numeric or categorical?

Have a look at the comment in that notebook:
# Define preprocessing for categorical features (encode the Age column)

It seems that the data you have is different (maybe only in order) to what is used in the notebook as the Age column is the eigth column (index seven) in the notebook. See also the cell where data is selected from the input file:
# Separate features and labels
features = ['Pregnancies','PlasmaGlucose','DiastolicBloodPressure','TricepsThickness','SerumInsulin','BMI','DiabetesPedigree','Age']
label = 'Diabetic'
X, y = diabetes[features].values, diabetes[label].values


Neural Turing Machine - how the shifting works? I am reading the paper on Neural Turing Machine, page 9, and got stuck on just 1 pesky point.
I can't understand how the shift is done:

Each head emits a shift weighting $\vec{s_t}$ that defines a normalized distribution over the allowed integer shifts (Authors mean entries in s will add up to 1). For example, if shifts between -1 and 1 are allowed, $\vec{s_t}$ has three elements corresponding to the degreee to which shifts of -1, 0 and 1 are performed.
If we index the $N$ memory locations from 0 to $N-1$, the rotation applied to $\vec{w_t^g}$ by $\vec{s_t}$ can be expressed as the following circular convolution:
$$w_{new}(i) \leftarrow \sum_{j=0}^{N-1}w^g_t(j)s_t(i-j)$$
where all index arithmetic is computed modulo $N$.

The authors use notation $w_t(i)$ to signify i'th element in the vector.
I cannot understand how $s_t$ is applied to the $w^g_t$ and what makes it possible to actually perform shift, say by +1

For example, assume we are working on the the i'th element that is at index 3 and $w^g_t$ has 5 entries.
Like in the paper, let's allow $s$ to be able to perform a shift of -1, 0 or 1. Thus, $s$ has 3 dimensions. Now, I want to shift every element in $w$ by +1 forward.
I have a feeling I will get out of bounds error, and modulo won't help too much. We can unfold this computation for the 3rd element:
w[3] = wg[0]*s[(3-0)%5] + wg[1]*s[(3-1)%5] + wg[2]*s[(3-2)%5] + wg[3]*s[(3-3)%5] + wg[4]*s[(3-4)%5]

when simplified this means:
w[3] = wg[0]*s[3] + wg[0]*s[2] + wg[0]*s[1] + wg[0]*s[0] + wg[0]*s[4]

As you can see, I got s[3] at the start and s[4] at the end. ...But we've just agreed s will only have 3 entries {-1, 0, 1} and this means out of bounds error.
And, I can't really understand how what we've done would "shift" the entire $w^g$ into $w_{new}$ once we do it for all 5 entries (we've done the above just for 3rd entry). Can someone give me the intuition for it as well?

Ok, it makes sense if we don't think about the formula

Look at the bottom half of the following slide, taken from Page 20, "Neural Turing Machine" side-presentation by Kiho Suh; June 19, 2017


How to cluster/identify points away from a regression line For many vine plots, I have NDVI and Leaf Area values for each vine.
I already know that NDVI and LA has a strong positive correlation as you can see in this picture.

But as you can see too, there are some points away from this pool of points.
How can I cluster or identify these points?
I'm using Python with sklearn, scipy and other libs like them.

As already mentioned in a comment, there are some methods to detect outliers. The best known might be Cook‘s distance in (or after) regression. In essence you look for observations with a high residual.

Here is a short R tutorial. Here is a short Python approach.


How can I identify and remove outliers in R I am performing regression analysis on prices of product that we have purchased, based on size and other attributes.

However there are often buys in odd circumstances which factor into the price, that is not (and cannot be) addressed directly in the features of the analysis.

Each time I run a regression, I will check the 20 with the largest error manually, and 90%+ of the time they will be odd buys like mentioned before, and for my purposes can be completely ignored.

I have been looking into cooks distance to remove these, however I'm not sure how to best set the threshold, or if there is a better method to use.

Cook's distance and the alternative method DFFITS are not strictly speaking methods to detect 'outliers' in the sense of purely anomalous values, rather they detect 'influential' points, where leaving the value in or out of the analysis noticeably changes the result. Hence, Cook's distance measures how much the beta values change when a specific observation is omitted, which is a different idea to detecting an outlier as such, although they will naturally often be the same points.

From the description of your problem, it is possible that detecting influential points is what you are actually trying to do.

To directly answer your question on the Cook's distance threshold, the F-statistic with p and n-p (where p is number of regressors, and n is observations) degrees of freedom is often used.

Outliers per se in regression are more often identified via residual analysis e.g. if the data point's corresponding residual is large in the context of the data set. This is often taken to be when the value lies more than three standard deviations away from the mean of the residuals (though note that in a sufficiently large data set, a well fitted model will still have some observations at this level)


Is there an inherent recency bias in deep learning? When working with very large models within Deep Learning, training often takes long and requires small batch sizes due to memory restrictions.
Usually, we are left with a model checkpoint after training has commenced. I am wondering whether the exact time at which we take that checkpoint significantly factors in to the statistical properties of a model's outputs.
For example:
Within text generation, lets assume that just before we extract the checkpoint, the model learns statistically anomalous batches with longer sentences than the mean.
Would that result in our model generating longer sentences, overrepresenting that recent batch of anomalous texts?
As training batches are often randomly generated from the dataset, such unrepresentative batches may certainly occur, sometimes right before we save the checkpoint.
Has there been any research regarding such, potentially unwanted, recency bias in slower deep learning scenarios?
The only references I could find were intentionally trying to employ such biases, but I have not found any literature on unwanted recency bias.

Your question is very interesting, however I feel you are overlooking a key point in your reasoning:
You usually take a model checkpoint at the point that it performs best on the validation set. This means that the instance of the model you keep is inherently the most robust and generalizable version of the model that you have evaluated, thus suffering the least from recency bias.
Suppose though, you don't checkpoint the model but stop it at a point arbitrarily. Naturally, you'd think that the samples in the final batch influenced the current state of the model much more than the first batches of the epoch. In practice this would show up as regular overfitting, however, instead of recency bias.
Some ways to deal with this:

relatively small learning rate
equivalently regularization as parameter norm penalties (i.e. L1, L2, ...)
ensembling
other more specialized techniques such as SGDA


Small number of estimators in gradient boosting I am tuning a regression gradient boosting-based model to determine the appropriate hyperparameters using 4-folds cross validation. More specifically, I am using XGBoost and lightGBM for the models and Bayesian optimization algorithm for the hyperparameters search (hyperopt)

One of the hyperparameter which is being tuned in the number of estimators used for each model. For the 1st round of testing, I starting with number of estimators in the range of 100-300. The outcome from hyperparameters optimization algorithm was that the best number of estimators is 100.I repeated the same analysis where the range of number of estimators was modified to 50-300, this time the outcome from hyperparameters optimization algorithm was that the best number of estimators is 50. I repeated the same analysis for the 3rd time setting the range of possible number of estimators to 2-300 (Just to check the extreme case) this time the outcome from hyperparameters optimization algorithm was that the best number of estimators is 2. The same outcome was noticed both for the XGBoost and lightGBM models.

What does this say about my model and data? Does this mean that the best model is Random forest instead of gradient boosting? is it smart 'force' the number of estimators by fixing the hyperparameter value (e.g., 150) and tuning all other parameters?

The training dataset has > 0.5M Samples where ~ 90% of the data is zero and the rest is mix of positive and negative values. Should I consider using a different object function rather than MSE?

The range of the tuned parameters for LightGBM are:

'max_depth': 5 ==> 15
'colsample_bytree': .6==> .9
'subsample':.5 ==> .8
'reg_alpha': np.log(1e-4) ==> np.log(1e-1)
'reg_lambda': np.log(1e-4) ==> np.log(1e-1)
'n_estimators': 50 ==> 300
'num_leaves': 10 ==> 150, 2
'min_child_samples': 20 ==> 800
'subsample_for_bin': 20000 ==> 300000
'subsample_freq': 1 ==> 20

Output:
{'subsample_freq': 16, 'num_leaves': 10, 'max_depth': 6, 'colsample_bytree': 0.7577614465604802, 'subsample_for_bin': 80000, 'min_child_samples': 415, 'n_estimators': 56, 'subsample': 0.6531478473538894, 'reg_alpha': 0.025744268683186224, 'reg_lambda': 0.0001729942781329532}

The range of the tuned parameters for XGBoost are:

'colsample_bytree',.6 ==> .9
'max_depth', 4 ==> 9
'n_estimators', 50 ==> 300
'reg_alpha', np.log(1e-4) ==> np.log(1e-2)
'subsample',.5 ==> .8
'gamma', 0 ==> 4

Output:
{'gamma': 2.4257700330471357, 'max_depth': 4, 'n_estimators': 57, 'subsample': 0.5568564232616263, 'reg_alpha': 0.0009876777981446033, 'colsample_bytree': 0.7073279309167877}

First what is n_estimators:

  n_estimatorsinteger, optional (default=10)
   The number of trees in the forest.

Gradient Boosting and Random Forest are decision trees ensembles, meaning that they fit several trees and then they average(ensemble) them.

If you have n_estimators=1, means that you just have one tree, if you have n_estimators=3 means that you have 3 trees and that it predicts the results of each tree and then it "averages" the result to get you the best.

It is not a good idea to force hyperparameters, cross-validation is the way to go in hyperparameter selection.

If you make your search a bit bigger you might be able to find different results.  Per example move wider in the subsample, in the penalties, in the max depth...

More parameters to search will mean more computational time.


Application of Control Theory in Data Science I have recently completed an MSc in Control Systems from a top university. It seems to me that control theory must have an application within data science. I would like to apply my degree within this domain, but I want to be sure that it is relevant to the role of a data scientist.

The topics which I have particular interest and experience in are State Space Control, Systems Identification, Model Predictive Control and Optimal Control. I imagine that effective management of any large dataset must involve modelling of the system in terms of transfer functions/state space models (based on large sets of historical input/output data). These models could then be used to predict the evolution of a market/variable over time, and therefore optimise a given cost function such as profit, risk etc.

If this kind of role exists within data science/ other areas, can you please give me more information/ ideas of job roles/ industries to research.

Have you tried the Internet search? The results should be able to answer most, if not all, of your questions. The topics of your interest sound like rather general or high-level. I'm sure that they can, in one form or another, be applied in the data science context. In my opinion, those topics are more related to operations research (OR), therefore, I would recommend you to perform some research on the Internet on the intersections between control systems (theory) and data science.

Having said that, first thing that comes to my mind is that the most likely candidate for use of control theory concepts and methods in data science context would be distributed systems and algorithms for data analysis, such as MapReduce (Hadoop, etc.), as well as other parallel processing systems. If there exist an intersection between OR's area of optimization and control theory, then it very well could be used for big data algorithms optimization, among other tasks.


How to apply model to training data to identify mislabeled observations? I have a list of people, attributes about those people (height, weight, blood pressure, etc.), and a binary target variable called has_heart_issues. This data represents the full population of data, and I am trying to determine whether anyone who is listed as "No" for has_heart_issues is similar to the people who are listed as "Yes".

To answer this question, I split the data into training (70%) and testing (30%). I trained a random forest model on the training, and I tested it on the testing. The results are good, but I don't know how to apply to the population since I used most of it for training. Is there any way to apply the model to the full dataset (including the training) since I had labels for the full dataset to start with? Essentially, I am trying to determine whether any of the people were mislabeled.

Is it okay to apply the model to the training data to find the "mislabeled" records?

Sure, its called cross validation. Have a look