Properties for building a Multilayer Perceptron Neural Network using Keras? I am trying to build and train a multilayer perceptron neural network that correctly predicts what president won in what county for the first time. I have the following information for training data.

Total population Median age % BachelorsDeg or higher Unemployment rate Per capita income Total households Average household size % Owner occupied housing % Renter occupied housing % Vacant housing Median home value Population growth House hold growth Per capita income growth Winner

That's 14 columns of training data and the 15th column is what the output should be.

I am trying to use Keras to build a multilayer perceptron neural network, but I need some help understanding a few properties and the pros of cons of choosing different options for these properties.

ACTIVATION FUNCTION

I know my first step is to come up with an activation function. I always studied neural networks used sigmoid activation functions. Is a sigmoid activation function the best? How do you know which one to use? Keras additionally gives the options of using a softmax, softplus, relu, tanh, linear, or hard_sigmoid activation function. I'm okay with using whatever, but I just want to be able to understand why and the pros and cons.

PROBABILITY INITIALIZAIONS

I know initializations define the probability distribution used to set the initial random weights of Keras layers. The options Keras gives are uniform
lecun_uniform, normal, identity, orthogonal, zero, glorot_normal, glorot_uniform, he_normal, and he_uniform. How does my selection here impact my end result or model? Shouldn't it not matter because we are "training" whatever random model we start with and come up with a more optimal weighting of the layers anyways?

1) Activation is an architecture choice, which boils down to a hyperparameter choice. You can make a theoretical argument for using any function, but the best way to determine this is to try several and evaluate on a validation set. It's also important to remember you can mix and match activations of various layers.

2) In theory yes, many random initializations would be the same if your data was extremely well behaved and your network ideal. But in practice initializations seek to ensure the gradient starts off reasonable and the signal can be backpropagated correctly. Likely in this case any of those initializations would perform similarly, but the best approach is to try them out, switching if you get undesirable results.


How to model to predict hotel booking abnomality? We are trying to build a model, gathering specific hotels booking data, try to find the pattern how the hotel is booked, stayed, what type of people lived, how many bookings average per day. The bookings might vary from weekdays to weekend, also from winter to summer, normal days and vacation period. All these factors are accountable.

Then, as the time passes by, we want to know if the booking becomes abnormal, e.g. normally, young couple books the hotel quite a lot, all the sudden, a bunch of business men checked in for a couple of days.

Since in this case, we don't have samples/labels for normality and abnormality, we started thinking use unsupervised learning, like clustering for a start. Say, we construct a sample(booking features for every week) going back all the way to the beginning of the year. And then, we try to cluster them, then, every week, we calculate the current week, and see if it belongs to any clusters? or it is an abnormal point stands out required attention.

Is this a reasonable approach or there are some better ways?

Look at the distribution of the features you want to consider for anomalies (e.g., user attributes) conditioned on the date, so you do not trigger a warning for normal seasonality. An anomaly then is when the current conditional distribution is significantly different from the historical average. For more information look into contextual anomaly detection. Welcome to the site and good luck!


Implementation of Association Rules in Javascript I have implemented an interactive visualization with d3.js, javascript to explore the frequency and various combinations of co-occurring item sets. I want to complement the interactive exploration with some automated options.
Does someone know an efficient javascript implementation of the association rules mining ?
My typical scenario will have just up to 30 different items.
There are some good web site with implementations of frequent item set mining (improvements from the initial apriori algorithm): http://www.borgelt.net/apriori.html
Any help is greatly appreciated.

I don't know how efficient they are, but I did find some implementations:

https://github.com/dmarges/apriori
https://github.com/seratch/apriori.js


Clarification on K Means Clustering Algorithm and its Analysis I fail to understand how the circled point is assigned to cluster c2 and not c1. From what I have understood, the points are assigned to the (nearest?) centroid in order to minimise the squared distances i.e. to minimise the assignment cost.

Visually too, it appears to me that the circled point belongs to the first cluster.

Please correct me if I'm wrong.

Source - http://www.cs.yale.edu/homes/el327/papers/OnlineKMeansAlenexEdoLiberty.pdf

K-means is an iterative algorithm. So what you have in the picture may be one of the early iterations and therefore the sum of squared distances (between points and clusters) has not been minimised yet. In further iterations, the circled point would indeed be assigned to cluster centroid C1.

Now, having said that, and this is purely speculation, but if the diagram in your picture is not simply 2 dimensional as we interpret it to be, but is actually a 2-d projection from a higher dimensional feature space, then indeed the circled point may be closer to centroid C2 in that higher dimensional space (although it doesn’t look like it in 2-d).


Classifications as long-term memory and short-term memory in LSTM How is the data classified as long-term memory and short-term memory?
Is there some standards programmers set?

I recommend you to read http://colah.github.io/posts/2015-08-Understanding-LSTMs/

Basically, the long-term memory and short-term memory are reflected in the structure of the LSTM Network. Please take a look at the diagram from the website I've mentioned.

The long-term memory is constructed in the "bus" on the top, where the previous value can be passed, depends on the $\sigma$ value. The $\sigma$ is between $[0,1]$ and $0$ means- forget the last value, $1$ means- get all (exact) the value from the last state. Sigmas for each cell are established during the training phase.


Does K-Means' objective function imply distance metric is Euclidean The objective/loss function of K-Means algorithm is to minimize the sum of squared distances, written in a math form, it looks like this:
$$J(X,Z) = min\ \sum_{z\in Clusters}\sum_{x \in data}||x-z||^2$$

If we have different distance metric, for instance, cosine (I realize there's a conversion between cosine and Euclidean but let's forget it for now), manhattan etc, does it mean we will have a different loss function?

That is, the traditional K-Means based on expectation maximization won't be working right? Because for every iteration we revise centroids, normally, by computing the average. However, for some of the metrics other than Euclidean, average might not be a legit representation for the center.

You're right and you're wrong.

  The objective/loss function of K-Means algorithm is to minimize the sum of squared distances

Yes, absolutely.

  written in a math form, it looks like this:
  $$J(X,Z) = min\ \sum_{z\in Clusters}\sum_{x \in data}||x-z||^2$$

Err... sort of. This is definitely the most popular formulation of kmeans, but a more appropriate formulation would be:

$$J(X,Z) = min\ \sum_{z\in centroids}\sum_{x \in data} distance(x,z)$$

In your formulation, you're concretely defining distance as euclidean distance, i.e. the L2 norm. But you can swap out L2 for any distance kernel and apply kmeans.

The caveat here is that kmeans is a "hill climbing" algorithm, which means each iteration should always be at least as good as the previous iteration, and so it must be the case that this improvement will be true for both the E and M steps. For most common distance metrics (L1, L2, cosine, hamming...) this is the case and you're good to go, but there are infinite possible distance metrics and if we're going to be technical about it, the probability that a random distance metric will satisfy this criterion is almost surely 0.

So, to circle back to your question: does the objective function as you formulated it imply the distance metric is euclidean? Yes. But does kmeans only apply to euclidean space? No, absolutely not. Use whatever distance metric you want and throw the EM algorithm at it and bam: you've got yourself a non-euclidean kmeans.

Generally, when people say "kmeans", they're talking about euclidean kmeans. But Kmeans is super easy to modify with a different distance metric and I think only the most pedantic people would argue that you shouldn't call it "kmeans" after such a modification. Although it's generally always described with the objective you posted (which, yes, does imply euclidean distance), you can really drop in pretty much any useful distance metric, throw EM at it, and it'll work. Some people might call it "___ kmeans" depending on the distance metric, but really it's all kmeans.

I think part of the reason kmeans often isn't described this way isn't formulated like this is because its often compared with gaussian mixture models (GMM). With a euclidean norm, kmeans is equivalent to a GMM with diagonal covariance (and hard cluster assignment), and is consequently often described as an easy way to fit a GMM with spherical clusters. But this comparison fails if we use a different distance metric.

I suggest you check out the CrossValidated discussion on this topic. Full disclosure: the highest voted answers disagree with me, but as you probably guessed I think they're being fairly pedantic.


Why Keras Dense layer is expanding number of tensors in each layer I have following model:

and I wonder why is the number of parameters different for e.g. dense_2477 and dense_2482? Both layers have the same amount of neurons so why do they provide different parameter numbers?

The number of parameters is the number of weights connecting the layer to the previous one, so for the layer $i$ it depends on the number of neurons in the layer $i$ and the previous one $i-1$.

The exact formula for a fully connected neural network is: $$n_i(n_{i-1}+1),$$ where $n_i$ is the number of neurons in the layer $i$, $n_{i+1}$ the number of neurons in the layer $i+1$, and the $+1$ term takes into account the bias.

So for dense_2477 you indeed get $$n_{\mathrm{params}}=8(16+1) = 136,$$ and
for dense_2482, $$n_{\mathrm{params}}=16*(8+1) = 144,$$ as expected.


Machine learning - 'train_test_split' function in scikit-learn: should I repeat it several times? I am a beginner in machine learning, and I hope someone can help me.

In Python's 'scikit-learn' library, the function 'train_test_split' splits the dataset into training and test sets. This is done in a random way (possibly using a seed to obtain the same result in repeated executions). But, with a single random split, how much can we trust the result (classification accuracy) obtained through the 'fit' method? I mean... if we are "unlucky", we may obtain a very bad outcome (or the opposite, if we are "lucky"). Shouldn't we repeat the split/fit procedures multiple times (e.g., 100) and then average the classification accuracies obtained? (possibly after parameter tuning by means of cross validation).

I am asking this because previously I used the Python's libraries of Orange Data Mining, which include a method ('proportion_test') that splits the dataset into training and test sets and then evaluates it according to a specific classifier, repeating the operation a specified number of times (e.g., it performs 100 iterations of 70:30 test). My question is: should I manually do this also with the split/fit functions in scikit-learn? (e.g., 100 iterations using 100 different random seeds). Would results be better?

I am very confused about this...

I want to stress that I know about cross validation, leave one out, etc. But, if I understand it right, these techniques are used for model validation (i.e., model parameter tuning). My question is whether the final evaluation of the model should be based on repeated split/fit operations. For example, in the book 'Introduction to Machine Learning with Python' (by Andreas C. Müller and Sarah Guido, O'Reilly), the suggested operation pipeline is: (1) split the original dataset into training set and test set; (2) perform parameter tuning (i.e., best parameter selection) using cross validation on the training set; (3) re-train using the just found best parameters with the training set; (4) perform the final evaluation (calculating the classification accuracy) using the trained model and the (single) test set. My question is: is this enough? Or, once the classifier has been trained using the best parameters (step 3 above), should I repeat the split/fit procedures multiple times on the whole dataset (original training + test sets) to obtain more reliable results? I did this using the Orange library, but maybe it is not necessary (it is not done in the book quoted above).

THANK YOU VERY MUCH in advance for your help!

You can use KFold cross validation if you want to average the results of the training.

sklearn.model_selection.KFold

This will split your data in a specified number of folds (k) and train your data on all but one folds, then validate on the last fold. This operation is done k times and the results are averaged out.

Normally, how I do is:

Train/Test split
Model selection + Hyperparameters tuning using KFold on training set
Retrain the final model on the whole training set
Evaluate on the test set

Note that if you want to check whether your split was 'lucky' or 'unlucky', you can still change the seed, or not give a seed at all and compare the results with different runs.

[EDIT] As stated in the comments below, the seed is controlled by the random_state argument and is mainly there for reproducibility. If you want a different train/test split at each run, just leave the default as is. It's always good to check at least twice to see whether you've been particularly lucky or not but it never happened to me ! :)


Optimising for Brier objective function directly gives worse Brier score than optimising with custom objective - what does it tell me? I am training an XGBoost model and as I care the most about resulting probabilities, not classification itself I have chosen Brier score as a metric for my model, so that probabilities would be well calibrated. I tuned my hyperparameters using GridSearchCV and brier_score_loss as a metric. Here's an example of a tuning step:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=123)

model = XGBClassifier(learning_rate=0.1, n_estimators=200, gamma=0, subsample=0.8, colsample_bytree=0.8, scale_pos_weight=1, verbosity=1, seed=0)
parameters = {'max_depth': [3, 5, 7],
              'min_child_weight': [1, 3, 5]}
gs = GridSearchCV(model, parameters, scoring='brier_score_loss', n_jobs=1, cv=cv)
gs_results = gs.fit(X_train, y_train)

Finally, I train my main model with chosen hyperparameters on two ways:

optimising for custom objective - brier, using custom brier_error function as a metric

model = XGBClassifier(obj=brier, learning_rate=0.02, n_estimators=2000, max_depth=5,
                      min_child_weight=1, gamma=0.3, reg_lambda=20, subsample=1, colsample_bytree=0.6,
                          scale_pos_weight=1, seed=0, disable_default_eval_metric=1)
model1.fit(X_train, y_train, eval_metric=brier_error, eval_set=[(X_train, y_train), (X_test, y_train)],
          early_stopping_rounds=100)
y_proba1 = model1.predict_proba(X_test)[:, 1]
brier_score_loss(y_test, y_proba1) # 0.005439
roc_auc_score(y_test, y_proba1) # 0.8567

optimising for default binary:logistic and auc as an evaluation metric

model2 = XGBClassifier(learning_rate=0.02, n_estimators=2000, max_depth=5,
                      min_child_weight=1, gamma=0.3, reg_lambda=20, subsample=1, colsample_bytree=0.6,
                          scale_pos_weight=1, seed=0, disable_default_eval_metric=1)
model2.fit(X_train, y_train, eval_metric='auc', eval_set=[(X_train, y_train), (X_test, y_train)],
          early_stopping_rounds=100)
y_proba2 = model2.predict_proba(X_test)[:, 1]
brier_score_loss(y_test, y_proba2) # 0.004914
roc_auc_score(y_test, y_proba2) # 0.8721

I would expect Brier score to be lower in the model1 since we optimise directly for it, but apparently it is not the case (see results above). What does it tell me? Does optimising brier is somehow harder? Should I use more boosting rounds? (Although this was found using grid search with brier_score_loss...) Is it explainable somehow but data distribution? (e.g. such an issue can occur in the event of unbalanced classes or something like that?) I have no idea where does that situation come from, but probably there is a reason behind that.

Brier score has known shortcomings for very rare or very frequent events.

Binary logistic regression objective function is relatively more robust than the Brier score to the rate of rare or frequent events.

It is possible that the difference in performance between the two objective functions is due to the event frequency in the data set.


You need to reset index using df.reset_index().This will make your Dates attribute/column and not an index as it is now.
import pandas as pd
import pandas_datareader.data as web
import datetime as dt

start = dt.datetime(2019, 2, 1)
end = dt.datetime.today().strftime('%Y-%m-%d')
slug = "tsla"
df = web.DataReader(slug, 'yahoo', start, end)
modified = df.reset_index()
print(modified['Date'])


ValueError: Shapes are incompatible when fitting using ImageDataGenerator I got this error ValueError: Shapes (None, 1) and (None, 3) are incompatible when training my Sequential model. I could not figure out which shapes are actually incompatible. This is the first time I do image classification. Here are my codes:

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Conv2D(32, (3, 3), activation='relu', input_shape=(150, 150, 3)))
model.add(tf.keras.layers.MaxPooling2D((2, 2)))
model.add(tf.keras.layers.Conv2D(64, (3, 3), activation='relu'))
model.add(tf.keras.layers.MaxPooling2D((2, 2)))
model.add(tf.keras.layers.Conv2D(64, (3, 3), activation='relu'))
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(64, activation='relu'))
model.add(tf.keras.layers.Dense(3, activation='softmax'))

model.summary()

model.compile(optimizer='rmsprop',
  loss='categorical_crossentropy',
  metrics=['accuracy'])

# this throws value error
model.fit(
  train_generator,
  steps_per_epoch=25,
  epochs=20,
  validation_data=valid_generator,
  validation_steps=5,
  verbose=2
)

I want the last Dense layer to return an array of 3 probability scores (large, medium, small).

Here is how I created the train_generator and valid_generator:

from tensorflow.keras.preprocessing.image import ImageDataGenerator
image_datagen = ImageDataGenerator(
  rescale=1./255,
  rotation_range=20,
  horizontal_flip=True,
  shear_range = 0.2,
  fill_mode = 'nearest')

index = len(df)//10
df_train = df[index:]
df_valid = df[:index]

train_generator = image_datagen.flow_from_dataframe(
  df_train,
  x_col = 'png_image',
  y_col = 'target',
  target_size=(150, 150),
  batch_size=4,
  color_mode = 'rgb',
  class_mode='sparse',
)

valid_generator = image_datagen.flow_from_dataframe(
  df_valid,
  x_col = 'png_image',
  y_col = 'target',
  target_size=(150, 150),
  batch_size=4,
  color_mode = 'rgb',
  class_mode='sparse',
)

The dataframes look like this:

Have you tried setting class_mode='categorical' in your generators?

I believe class_mode='sparse' works when your loss function is defined as 'sparse_categorical_crossentropy' whereas 'categorical_crossentropy' works for class_mode='categorical'.


Logic behind the Statement on Non-Parametric models I am currently reading 'Mastering Machine Learning with scikit-learn', 2E, by Packt. In Lazy Learning and Non-Parametric models topic in Chapter 3- Classification and Regression with k-Nearest Neighbors, there is a paragraph stating-
Non-parametric models can be useful when training data is abundant and you have little prior knowledge about the relationship between the response and the explanatory variables. kNN makes only one assumption: instances that are near each other are likely to have similar values of the response variable. The flexibility provided by the non-parametric models is not always desirable; a model that makes assumptions about the relationship can be useful if training data is scarce or if you already know about the relationship.
My doubt is that I am able to reason out the logic behind the statement- "a model that makes assumptions about the relationship can be useful if training data is scarce or if you already know about the relationship." But, I am not able to see any logic for the first statement- "Non-parametric models can be useful when training data is abundant and you have little prior knowledge about the relationship between the response and the explanatory variables."
Please help me to find out the logic behind it other than for computation. Thanks for your time and consideration.

Non-parametric machine learning algorithms try to make assumptions about the data given the patterns observed from similar instances.By not making assumptions, they are free to learn any functional form from the training data.

Unlike parametric approach, where the number of parameters are fixed,
in non-parametric approaches the number of parameters grow with
training data.

If your data set is too small or otherwise is a set that is not representative of the entire population, then your result will be biased in more ways than possible with parametric methods.
So,to get better results in Non-parametric machine learning algorithms we need large amount of data where relationship between features are not known.
While non-parametric machine learning algorithms are often slower and require large amounts of data, they are rather flexible as they minimize the assumptions they make about the data.


Can I get numeric data from a color map? In my class I often need to work with color map images. I would show the image and try to make inferences/observations about different subjects. Often times I need to actually quantify some aspects, but it is always very approximate and somehow vague because the images are provided "as is" and I do not necessarily know their content a priori.
Let's imagine I'm working with two images (*). Is it possible to indicate the computer "learn" the color scale bar so I can click at any point in the map and get the value of altitude at that point? Image 1 is a discrete color scale bar, while Image 2 has a continuous color mapping. I included both examples because maybe there are differences in the approach of how to do this.
So, is there a software/way to do this? Preferably open source. I tried ImageJ and couldn't come up with a decent solution.
Image 1:

Image 2:

(source: topocreator.com)
As you can see, the color scale is part of the image in all cases.
(*) I do not own any of these images, just examples I found online to illustrate my point.

Maybe You can try some workaround (first manually then programatically):

Convert to grey-scale
Make density map (digital)
Trace=vectorise image with any Inkscape / Illustrator / etc...
Map vector parts with density map values.

You can write js script for Adobe Illustrator (Gimp /  probably also) to automate some things. I'm not suggesting to spend money, just play around with demo until you wiil find Ideal Solution.

Also probably You can try any online API from Big4 or new emergers:

tensorflow
https://cloud.google.com/vision/
https://aws.amazon.com/rekognition/
https://www.ibm.com/watson/services/visual-recognition/
http://opentracing.io/

Also there are some commercial with free options
i.e. imagga - "Sign up for Free: Tag up to 2000 images per month"

You can try DEMO Vector Magic's Desktop edition $295.00 / license One-time charge allows Group shapes by color (unfortunately do not offer special pricing for Academic Institutions or Non-profits)

Also have you tried FreeMat COLORMAP - Image Colormap Function?


Temporal train test split for recommender systems When evaluating a collaborative filtering recommender system, it is practical to split the data temporally. However, by doing so, some users might be present in only either of the train or test set. For example, consider the below example:
user  year
0     2020
0     2020
0     2021
1     2021
1     2021
1     2021
2     2020
2     2021
2     2021

If we decide to split by year such that ratings after 2020 will be in the test set, then:
Train
user  year
0     2020
0     2020
2     2020

Test
user  year
0     2021
1     2021
1     2021
1     2021
2     2021
2     2021

This means that user 1 will not be in the train set at all. When using matrix factorization/latent models, since user 1 is not in the train set, when we multiply the latent factors U and V to get back the predicted rating matrix, user 1 will not be in there at all, and thus we will not be able to predict the ratings for user 1. This applies to items as well, although it is not shown here.
How does one deal with that? Does one simply remove users that are not in the train set from the test set? Wouldn't this lead to a lot of data wastage?

What I have previously done, and what have worked well in my domain (retail), is to first get a sense of the distribution of your user/rating frequencies. More specifically, say you want a test-set consisting of 10% of the observations, then figure out the threshold (n) where, if you take all users with at least n interactions, and their latest interaction(s), you end up with 10%-ish datapoints.
In your case, perhaps some fraction of the users have at least 5 interactions (ratings), then take the latest interaction(s) these users have done, and use that as your test-set (use more or less than 5 to change the size of your test-set). This way, there exists no users in your test-set that are not a part of your training-set, although, the users with very few interactions will not be represented in the test-set. This train/test-skew could also pose a problem, but not being able to provide recommendations for these users, I have found, pose an even larger problem.


High performance R libraries for interactive graphics In the past few years there has been an explosion in the quantity and quality of available data visualization tools in R. I've long ago stopped using plot base graphics in favor of the wonderful ggplot2 package, and I'm considering going for interactivity now, perhaps using one of the JS-integration packages, and outputting HTML.

However, I'm often in the business of creating long HTML report pages with dozens of graphics in there, and I care about a number of factors:

load-time for the graphics generated cannot be too long.
should work decently with Shiny and Rmarkdown (ideally flexdashboard).
should have a non-insane syntax for tweaking options (i.e no 20 line JS code inserts to change legend location).
should be relatively well-documented.
should be actively maintained.

What are your go-to packages for interactive data viz?

My current go-to are some graphics from Rstudio's flexdashboard and a set of relatively well-maintained packages:

  Leaflet, a library for creating dynamic maps that support panning and
  zooming, with various annotations like markers, polygons, and popups.

  dygraphs, which provides rich facilities for charting time-series data
  and includes support for many interactive features including
  series/point highlighting, zooming, and panning.

  Plotly, which via its ggplotly interface allows you to easily
  translate your ggplot2 graphics to an interactive web-based version.

  rbokeh, an interface to Bokeh, a powerful declarative Bokeh framework
  for creating web-based plots.

  Highcharter, a rich R interface to the popular Highcharts JavaScript
  graphics library.

Source: Flexdashboard Components


Is there a package for using SQL to manipulate Pandas dataframes in Python? Rather than learn a new package/language, I'd like to use my existing SQL skills to manipulate pandas dataframes in Python. Does anyone know of a way to do this, or perhaps a package that will allow me to do this?

I found a package called pandasql, which is based on sqldf for R. It seems quite a bit slower than doing the transformations with the pandas package, but it gets the job done. Just put the SQL query into a string like this:

query_string = """
    select * from df
"""

Then use the string in the pandasql.sqldf package, as follows:

new_dataframe = pandasql.sqldf(query_string, globals())

Choose globals() or locals(), depending on the scope you want for your variables.

As I mentioned, it seems a bit slow, but I couldn't find anything else. I may use this from time to time until I become better at Pandas.

Sean


How to find correlation between two junctions of a road I am working on traffic data analytics, where I need to predict the number of vehicles passing through each traffic junctions.

Shape of my dataset is something like this :

Road A-Junction B-Junction   Date       Hour  CarCount
X         a          b      2000-06-09   7       10
Y         b          c      2000-06-09   7       20

Road – this is the road name (for instance M25 or A3).
A-Junction – The road name of the start junction of the link
B-Junction – The road name of the end junction of the link
Hour - time in 24 hour format
CarCount - actual car count

I want to find out the correlation between A-Junction and B-Junction.
Or in simple words , let's say in the given example for road X and Y , can we correlate between a , b and c roads ?

Sorry to be posting as an answer, but the comment section doesn't allow me to place preformatted text. Using your example above, is the graphic representation similar to the one below?

X=== a -10- b ---- c
     |      |      |
     |      |      |
Y=== a ---- b -20- c

So you are wondering if the 10 (Road X, junction a - b) and 20 (Road Y, junction b - c) are correlated?

Update: Thank you for confirming.
Sorry I am totally not in the traffic domain, but I will give my 2 cents given the little I know about ML in general.

I assume that you want to understand the correlation between 10 and 20 to help you build a prediction model. If you unpack the question a bit, I think perhaps we can restructure the question to.

(If you are simply looking at all the available combination of and their correlations, it will probably take forever, and it might not be as actionable unless you normalize the information into your prediction model)

Label  = number of cars

Features = date, Road, junction a, junction b, [Number of cars in nearby intersections] + other related features

If it is the case, perhaps we can try to normalize the data to the Road, junction a, b through feature engineering.

Expanding on the diagram

W z-4--a --6- b --8- c
  |    |     |      |
  3    20    28     9
  |    |     |      |
X-z-1- a -10-b -19- c
       |     |      |
       15    12     7
       |     |      |
Y--10- a -14-b -20- c

For Road X, junction a - b, we have 10 as the Label

Reconsider the nearby roads as:

level 0 -- Immediate connecting segments

level 1 -- Immediate connecting segments to level 0 roads

level 2 -- Immediate connecting segments to level 1 roads (if required)

W z-1--a -1-- b --1- c
  |    |     |      |
  1    0     0      1
  |    |     |      |
X-z-0- a -**-b -0- c
       |     |      |
       0     0      1
       |     |      |
Y--1-- a -1 -b --1- c

You will know best how further away you want your output to be counted. you just create the necessary number of features to accommodate that. Perhaps distance to the junction in question, etc. Whatever makes more sense for your model
You will know best the necessary granularity. To start, perhaps you can count the number of level 0 and 1, recording each with the average in your model, etc.

Example:

Road     Jun A     Jun B    L0 Connection  L0 Avg Car    L1 Connection   L1 Avg
   X         a         b                6        17.5                9    [you get the idea]

Not sure how to handle bidirectional traffic. does the order of a and b imply direction already?
Your model will probably take care of the influence of your level 0 and level 1 (and level 2, etc.) connections.

Sorry for the long-winded response; hopefully it helps brainstorm some ways to tackle your problem.


Create a binary-classification dataset (python: sklearn.datasets.make_classification) I would like to create a dataset, however I need a little help. The dataset is completely fictional - everything is something I just made up.

Since the dataset is for a school project, it should be rather simple and manageable.

I would like a few features could be something like:

x1: temperature

x2: color

x3: moisture

and then I would have to classify with supervised learning whether the cocumber given the input data is eatable or not.

I am having a hard time understanding the documentation as there is a lot of new terms for me.

import sklearn.datasets as d           # Python #

a = d.make_classification(n_samples=100, n_features=3, n_informative=1, n_redundant=1, n_clusters_per_class=1)

print(a)

n_samples: 100 (seems like a good manageable amount)

n_features: 3 (3 is a good small number)

n_informative: 1 (from what I understood this is the covariance, in other words, the noise)

n_redundant: 1 (This is the same as "n_informative" ? - well, 1 seems like a good choice again)

n_clusters_per_class: 1 (forced to set as 1)

Would this be a good dataset that fits my needs? If not, how could I could I improve it?

Let's build some artificial data. There are many ways to do this. I usually always prefer to write my own little script that way I can better tailor the data according to my needs.

Let us first go through some basics about data. A lot of the time in nature you will find Gaussian distributions especially when discussing characteristics such as height, skin tone, weight, etc. Let us take advantage of this fact.

According to this article I found some 'optimum' ranges for cucumbers which we will use for this example dataset.

Temperature: normally distributed, mean 14 and variance 3. If a value falls outside the range $[10,18]$ then it is said to be not edible.
Color: we will set the color to be 80% of the time green (edible). 10% of the time yellow and 10% of the time purple (not edible).
Moisture: normally distributed, mean 96, variance 2. If the moisture is outside the range $[94, 98]$ then the cucumber is not edible.

Building the dataset

We will build the dataset in a few different ways so you can see how the code can be simplified.

A very verbose example

This example will create the desired dataset but the code is very verbose.

import numpy as np

# Number of samples
n = 100

data = []
for i in range(n):
    temp = {}

    # Get a random normally distributed temperature mean=14 and variance=3
    temp.update({'temperature': np.random.normal(14, 3)})

    # Get a color with 80% probability green, 10% probability yellow
    # and 10% probability purple
    color = 'green'
    color_random_value = np.random.randint(0,10)
    if color_random_value == 8:
        color = 'yellow'
    elif color_random_value == 9:
        color = 'purple'
    temp.update({'color': color})

    # Get a random normally distributed moisture mean=96 and variance=2
    temp.update({'moisture': np.random.normal(96, 2)})

    # Verify if the instance is edible (label=0) or not (label=1)
    label = 0
    if temp['temperature'] < 10 or temp['temperature'] > 18:
        label = 1
    elif temp['color'] != 'green':
        label = 1
    elif temp['moisture'] < 94 or temp['moisture'] > 98:
        label = 1
    temp.update({'label': label})

    data.append(temp)

Then we can put this data into a pandas DataFrame as

df = pd.DataFrame(data=data)
df.head()

A cleaner example

import numpy as np

n = 100
data = {'temperature': np.random.normal(14, 3, n),
        'moisture': np.random.normal(96, 2, n),
        'color': np.random.choice(['green', 'yellow', 'purple'],
                                  size=100,
                                  p=[0.8, 0.1, 0.1])}
df = pd.DataFrame(data=data)

Then we will get the labels from our DataFrame

def get_label(color, moisture, temperature):
    if temperature < 10 or temperature > 18:
        return 1
    elif color != 'green':
        return 1
    elif moisture < 94 or moisture > 98:
        return 1
    return 0

df['label'] = df.apply(lambda row: get_label(row['color'],
                                             row['moisture'],
                                             row['temperature']), axis=1)

Getting the data ready for applying a classifier

One of our columns is a categorical value, this needs to be converted to a numerical value to be of use by us.

This can be achieved using

df['color_codes'] =df['color'].astype('category').cat.codes

Now we are ready to try some algorithms out and see what we get.

Visualizing the data

The first important step is to get a feel for your data such that we can try and decide what is the best algorithm based on its structure.

I prefer to work with numpy arrays personally so I will convert them

X = np.asarray(df[['color_codes', 'moisture', 'temperature']])
y = np.asarray(df['label'])

Let's plot the data in 3D

from mpl_toolkits.mplot3d import Axes3D

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

ax.scatter(X[:,0], X[:,1], X[:,2], c=y)

ax.set_xlabel('Color')
ax.set_ylabel('Moisture')
ax.set_zlabel('Temperature')

plt.show()

The blue dots are the edible cucumber and the yellow dots are not edible. We can see that this data is not linearly separable so we should expect any linear classifier to be quite poor here. I would presume that random forests would be the best for this data source.

Let's split the data into a training and testing set

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33)

Let's see the distribution of the two different classes in both the training set and testing set

import matplotlib.pyplot as plt
%matplotlib inline

n_classes = 2

training_counts = [None] * n_classes
testing_counts = [None] * n_classes
for i in range(n_classes):
    training_counts[i] = len(y_train[y_train == i])/len(y_train)
    testing_counts[i] = len(y_test[y_test == i])/len(y_test)

# the histogram of the data
train_bar = plt.bar(np.arange(n_classes)-0.2, training_counts, align='center', color = 'r', alpha=0.75, width = 0.41, label='Training')
test_bar = plt.bar(np.arange(n_classes)+0.2, testing_counts, align='center', color = 'b', alpha=0.75, width = 0.41, label = 'Testing')

plt.xlabel('Labels')
plt.xticks((0,1))
plt.ylabel('Count (%)')
plt.title('Label distribution in the training and test set')
plt.legend(bbox_to_anchor=(1.05, 1), handles=[train_bar, test_bar], loc=2)
plt.grid(True)
plt.show()

Linear classifier

from sklearn import linear_model
clf = linear_model.SGDClassifier(max_iter=1000)
clf.fit(X_train, y_train)
clf.score(X_test, y_test)

  0.54545454545454541

Support Vector Classifier

from sklearn.svm import SVC
clf = SVC()
clf.fit(X_train, y_train)
clf.score(X_test, y_test)

  0.72727272727272729

K-Nearest Neighbors

from sklearn.neighbors import KNeighborsClassifier
neigh = KNeighborsClassifier(n_neighbors=3)
neigh.fit(X_train, y_train)
neigh.score(X_test, y_test)

  0.66666666666666663

Random Forests

from sklearn.ensemble import RandomForestClassifier

forest = RandomForestClassifier(n_estimators = 100)
forest.fit(X_train, y_train)
print('Score: ', forest.score(X_test, y_test))
predictions = forest.predict(X_test)

  Score:  1.0

Well we got a perfect score. As expected this data structure is really best suited for the Random Forests classifier.


Multiply Tensorflow sequential layer by fixed integers I'm trying to make a simple reinforcement learning model that makes one of three decisions, A, B, or C.  However, C will be effectively a "do nothing" decision.  Because C will never result in an action, it should always have a value of 0.

So all I want to do is multiply the 3 output nodes by 1, 1, 0.  That is, I want to do an element-wise multiplication by a [1,1,0] tensor so that the A and B nodes retain their values from the previous layer and the C node always has a value of 0.

I see that there's a Multiply layers class, which I think might work, but I'm seeing examples with the functional API.  Is there another way to do this with the Sequential API?

This turned out to be really simple.  I just used a Lambda layer to multiply the previous output layer by the multiplication vector.

model.add(tf.keras.layers.Dense(3)) # previous output layer
model.add(tf.keras.layers.Lambda(lambda x: x * [1,1,0])) # new output layer

It appears to be working correctly.


I am getting very minimal mse values and not sure if it is correct? Below is the linear regression model I fitted and not sure if I am doing the right way as I am getting neat to 99% accuracy

Fitting Simple Linear Regression to the Training set

from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score

ln_regressor = LinearRegression()
mse = cross_val_score(ln_regressor, X_train, Y_train , scoring = 'neg_mean_squared_error', cv = 5)
mean_mse = np.mean(mse)
print(mean_mse)

ln_regressor.fit(X_train, Y_train)

** MSE SCORE =-6.612466691367042e-06**

Predicting the Test set results

y_pred = ln_regressor.predict(X_test)

Evaluating accuracy of test data

mse2 = cross_val_score(ln_regressor, X_test, y_pred , scoring = 'neg_mean_squared_error', cv = 5)
mean_mse2 = np.mean(mse2)
print(mean_mse2)

**MSE score = -4.645751512870382e-31**

Please Note:
My data is in log scale & transformed to standard scaling later on

R2= cross_val_score(ln_regressor,X_test, y_pred,cv = 10)

R2.mean()

R2 mean is '0.9999030728571852'

So first thing first, accuracy is a classification concept. You can't say you have 99% accuracy for a regression problem.

Your code seems ok. Cross validation is not necessary here since you are not doing any hyper-parameter tuning or model selection. The mse error is indeed low, so I would suggest you go back to normalize your data, since if your target $y$ has a very small span, i.e. low $\sigma$ in Gaussian case, you will get a meaningless low mse guaranteed.


Running huge datasets with R I'm trying to run some analysis with some big datasets (eg 400k rows vs. 400 columns) with R (e.g. using neural networks and recommendation systems).
But, it's taking too long to process the data (with huge matrices, e.g. 400k rows vs. 400k columns).
What are some free/cheap ways to improve R performance?

I'm accepting packages or web services suggestions (other options are welcome).

Although your question is not very specific so I'll try to give you some generic solutions. There are couple of things you can do here:

Check sparseMatrix from Matrix package as mentioned by @Sidhha
Try running your model in parallel using packages like snowfall, Parallel. Check this list of packages on Cran which can help you runnning your model in multicore parallel mode.
You can also try data.table package. It is quite phenomenal in speed.

Good reads:

11 Tips on How to Handle Big Data in R (and 1 Bad Pun)
Why R is slow & how to improve its Performance?


Use a single Gaussian to represent a mixture of Gaussians I want to merge a Gaussian mixture, $\sum_{i=1}^{K} w_i \exp(x; \mu_{i}, \Sigma_{i})$ into one single Gaussian, under the constraints that $w_1 >> w_2 \geq \dots \geq w_K$, i.e. we have a dominant Gaussian. Intuitively, the resulting Gaussian will be near the first Gaussian. I want to use a MLE estimate, seeking the optimal Gaussian $\exp(x; \mu, \Sigma)$. Obviously, formula for the expression of the mean is $$\mu = \sum_{i} w_i \mu_{i}$$
What about the covariance? Can any one help me with this problem?

One option is to re-estimate a single Gaussian distribution directly from the empirical data.


Looking for a data set that shows hospital patients' illnesses and their vital signs, during their stay I'm looking for a data set that shows hospital patients' vital signs (body temperature and/or heart rate, etc.) and their illnesses, over time. Is there such a data set?

For example:
Patient #1, Male, age:35, ..., Diagnosed w/: Heart failure
22:32:10: Temperature:32.1
22:32:20: Temperature:34.2
22:32:30: Temperature:33
22:32:40: Temperature:35

Or:
Patient #2, Female, age:35, ..., Diagnosed w/: Migraine Attack
22:32: Heart Rate:68
22:33: Heart Rate:75
22:34: Heart Rate:69
22:35: Heart Rate:70

I don't have any insistence for the data to come from a hospital. I already have searched some sources but with no luck.

I'm assuming you are looking for medical datasets, not only what you have stated in the question, but also some image recognition datasets which need something like CNN.

Try this one. This dataset contains several medical features including blood sugar, serum cholesterol etc, and wants you to find out the presence of heart disease.

In the meanwhile, there are some medical competitions and datasets on Kaggle, including the famous Data Science Bowl.


How to implement word to word Co-occurence matrix in python To implement co-occurence matrix in sucha a way that number of times word1 occured in context of word2 in neighbourhood of given value, lets say 5. There are 100 words and a list with 1000 sentences. So how can i calculate co-occurence matrix of size (100* 100) using python?

from nltk.tokenize import word_tokenize
from itertools import combinations
from collections import Counter

sentences = ['i go to london', 'you do not go to london','but london goes to you']
vocab = set(word_tokenize(' '.join(sentences)))
print('Vocabulary:\n',vocab,'\n')
token_sent_list = [word_tokenize(sen) for sen in sentences]
print('Each sentence in token form:\n',token_sent_list,'\n')

co_occ = {ii:Counter({jj:0 for jj in vocab if jj!=ii}) for ii in vocab}
k=2

for sen in token_sent_list:
    for ii in range(len(sen)):
        if ii < k:
            c = Counter(sen[0:ii+k+1])
            del c[sen[ii]]
            co_occ[sen[ii]] = co_occ[sen[ii]] + c
        elif ii > len(sen)-(k+1):
            c = Counter(sen[ii-k::])
            del c[sen[ii]]
            co_occ[sen[ii]] = co_occ[sen[ii]] + c
        else:
            c = Counter(sen[ii-k:ii+k+1])
            del c[sen[ii]]
            co_occ[sen[ii]] = co_occ[sen[ii]] + c

# Having final matrix in dict form lets you convert it to different python data structures
co_occ = {ii:dict(co_occ[ii]) for ii in vocab}
display(co_occ)

Output:

Vocabulary:
 {'london', 'but', 'goes', 'i', 'do', 'you', 'go', 'not', 'to'}

Each sentence in token form:
 [['i', 'go', 'to', 'london'], ['you', 'do', 'not', 'go', 'to', 'london'], ['but', 'london', 'goes', 'to', 'you']]

{'london': {'go': 2, 'to': 3, 'but': 1, 'goes': 1},
 'but': {'london': 1, 'goes': 1},
 'goes': {'london': 1, 'but': 1, 'you': 1, 'to': 1},
 'i': {'go': 1, 'to': 1},
 'do': {'you': 1, 'go': 1, 'not': 1},
 'you': {'do': 1, 'not': 1, 'goes': 1, 'to': 1},
 'go': {'london': 2, 'i': 1, 'to': 2, 'do': 1, 'not': 1},
 'not': {'do': 1, 'you': 1, 'go': 1, 'to': 1},
 'to': {'london': 3, 'i': 1, 'go': 2, 'not': 1, 'goes': 1, 'you': 1}}

PS

Do text preprocessing yourself (remove punctuations, lemmatization, stemming, blahblah)
Continue the code for any conversion you want. You have the dict, you can convert it to sparse matrix or pandas datframe


Do Data Miners realise Machine Learning cannot establish Causality? Do people in Machine Learning (or, more generally, in Data Mining) realise that no causal link can be inferred from a correlation?

My impression of the ML/AI community is that most people involved have a Computer Science background. I think it uncontroversial to point out that this is not a Science in the academic sense (think Physics, Chemistry, Psychology). Therefore, most of these people have no formal training in the Scientific Method, in Statistics or even in Mathematics.

"Correlation does not equal causation" is lesson nr. 1 in Statistics. In order to establish causal links, experiments are required (under some very special circumstances, other techniques are available). To me, this seems a fundamental and disastrous limitation to Machine Learning. ML will do fine with describing data (and classifying it), but it will never substitute an experiment. The application of ML is, therefore, limited in scope.

My question is about the atmosphere within the ML community.

Do people within the field recognise this limitation? Or do people think the causal-inference problem is not fundamental and will be solved in the future by better CPU's and better algorithms?
Do they restrict the use of ML to descriptive analyses? Or do they, erroneously, assume ML can help us understand and influence the world?
Is there a lively debate on this topic within the community? And are there articles/blogs related to these issues?

I think you're a bit confused about the role machine learning plays in the order of things, and I find your impression of the ML community to be a bit bizarre.

To begin, ML experts are often not the people who use ML algorithms to solve science problems - those people are usually called "data scientists" these days, and in principle they are supposed to have some sort of background in science, statistics, and/or mathematics.  And yes, (good) data scientists spend a lot of time worrying about causal inference - they have to because there are big companies with lots of money riding on their predictions.  Contrary to your impression ML experts are usually aware of these issues too, but their job is to build and analyze modeling tools rather than actually apply them to real data.  Of course there are lots of ML and data science people which straddle the line between the two fields, but my point is that your question is a bit like asking why mathematicians don't worry about experimental error.

That said, many (most?) ML algorithms are more oriented towards classification problems which are not quite as vulnerable to your concerns.  You still have to worry about sticky statistical problems like overfitting and building good training / validation datasets, but the focus is more on understanding the structure of the data that you have than predicting the future.


Does experience with Keras count as experience with Tensorflow? Many Machine Learning job postings I've seen request experience with Tensorflow. If I have experience with Tensorflow, but only through building neural networks using the Keras API. Does that count?
I have yet to see a tutorial or any code anywhere that uses Tensorflow without the Keras API, so I don't see how one learns Tensorflow strictly. But then why don't employers just request experience with the Keras API?

Currently, Keras is part of Tensorflow, so if you are using the Keras that comes bundled with Tensorflow (tf.keras), technically one is a subset of the other. This was not always like that (see this), so if you are using the old version of Keras that is a separate package, then technically it hides the complexity of Tensorflow, so your experience would be only Keras.
About tutorials not using the Keras API (I understand that you are referring to using model.compile, model.fit, etc), you should understand that tutorials are normally introductory material, which makes using only Keras a sensible decision, avoiding the rougher Tensorflow reality (static graph vs. eager execution, sessions, etc). Bare Tensorflow (no model.fit) is used all over the place for real-world use.
Anyway, in the end, it is up to employers to decide whether your experience fits their needs.


Does Minibatch reduce drawback of SGD? Many expert said "Batch has more local optimal possibility than SGD". But, I don't know the reason..

How SGD could avoid local optimal better than Batch?
(Some people tell me about over shooting as the reason. But, I don't also know there is more over shooting in SGD)

If you explain this reason by using Geometric, Mathmatics or other intuition concept, It's gonna be great!

For gradient descent, batch updates calculate the weights based on all examples. This process continues until convergence which is typically measured by a derivative of zero. The resulting weights might end up in a local minimum or saddle point because both have a derivative of zero.

Stochastic gradient descent takes a random sampling of examples, introducing randomness into the updates. This randomness increases the possibilities of escaping a local minimum or saddle point. The updates might randomly overestimate the area with a derivative of zero and find a new section of the error surface that leads to a better optimum.


How can I use data science / machine learning / statistics / data analytics techniques to find better customers in the future? I work for a business to business company with a data set containing approximately 2700 businesses as customers. For each business customer I have approx number of staff, location, industry, the sales rep managing their account and for the last 7 years how much profit I have made with each of them for each month. What data science/machine-learning/statistics techniques or analysis can I perform to use this data to select which businesses I should try to acquire as customers in the future? And what if any methods might be worth considering to enrich this data set?

You can use linear regression to model the relationship between the profit each customer has given you and the other variables: staff, location, industry, and so on.

With this technique, you will by the end obtain the regression coefficients for each considered variable. These coefficients will tell how one variable affects the profit, and then you'll be able to check things such as: do customers with more staff generate more profit? how much this profit increases for each staff member? and so on.

Also, with these coefficients, you can easily predict, for the potential costumers, how much would be their profits. This prediction is not perfect, as it is an associated error, but it can be very good if you specify the model properly. As there is a loooooot of material about this on the internet, I won't write much more.

I don't know how much you know about math, or intend to dive in this subject, but this article can help as a first introduction: https://towardsdatascience.com/mathematics-for-machine-learning-linear-regression-least-square-regression-de09cf53757c

Linear regression is, for sure, not the only solution for your problem, but it is a simple one, somewhere to start. You may also want to look (later) for regression tree models and models that take in account the longitudinal dependency of your data. Longitudinal means you have repeated measures for the same costumers along the time.


Weka alternative - visual data exploration with a very large dataset I love Weka for its visual data exploration features (it's great at quickly giving you a mile-high aerial view of the distributions of all your variables).

I have a 0.5gb dataset, and Weka is unable to load it (on a 64-bit Windows PC with 16gb of RAM).

For the Weka fans out there, what's the closest 'visual data exploration' alternative for very large datasets?

(FWIW - I've done all the various things that are recommended when Weka fails at loading a very large dataset, e.g., increasing the Java heap size, etc.)

Update: I tried RapidMiner Basic (it's open source and free), and that worked.

I tried Azure Machine Learning (by Microsoft) and it is very nice. It can easily handle large data sets and has similar to weka pipeline for ML tasks and visualisation. They offer it free for 30 days so you can try it. You can google it or follow this link Azure ML


Standarizing the Label to postive intergers in Machine Learning Classification task, why is it recommended? When doing a classification task suppose we have 3 targets with notation -2, -1, 0.
I read somewhere it is a good practice to Standardize the labels to positive integers.. In this case suppose we change it to 2, 1, 0. Why is it recommended?
Theoretically I don't think it should matter since each class is unique. But in practice it matters, why?

Please can anybody be kind enough to explain this to me? When doing a
classification task suppose we have 3 targets with notation -2, -1, 0.
I read somewhere it is a good practice to Standardize the labels to
positive integers. In this case suppose we change it to 2, 1, 0. Why
is it recommended?

Can you please link a reference to this!. Usually in practice you wouldn't use integer labels at all. Instead you would use one-hot encoded labels. Here since you have 3 output classes, the neural network will have 3 output nodes. To be concrete the classes can be represented as follows :

0  == >  [1 0 0]
-1  == >  [0 1 0]
-2  == >  [0 0 1]

where one out of the three bits is turned ON. This is because in practice the output of the neural network may be a sigmoid (output: range[0,1]) or softmax function (output: probabilities)


How to treat column with potentially meaningful NaNs My data set has a column that indicates the time taken (in days) for members on a site - each with an ID - to sign up for an event. This can range between 1 to 300 days, with about half of the rows filled with NaNs for members who have yet to sign up for any events. Another column indicates the channel through which members initially joined the site, for example through direct referrals, a forum, or other websites.

The question is: If I'd like to try and estimate which groups I should target, as well as the best time, to send an e-mail reminder that increases the chances that members sign up for an event, what would be the appropriate approach if I have no labels for whether past e-mail reminders is successful or not? I was thinking of taking the average (median) of days taken to sign up for an event grouped by the channel through which members joined the site, but that would remove half the data which seems like a waste. Is there an alternative method?

What you are asking about is called imputation - https://en.wikipedia.org/wiki/Imputation_(statistics)

Imputation basically means replacing NaNs with other data. There are lots of ways to do it and the method usually depends on the application. In this case you are the domain expert so you would be in the best position to decide what imputation method you should use.

Some common imputation methods are:

Mean (replace all NaNs with the average)

Median (replace all NaNs with the "middle" of the data)

MICE (Multiple Imputation by Chained Equations - probably overkill for your application, but worth looking into)

In your case I would go with mean imputation to start and then see what kind of results you get.


Outlier detection with non normal distribution What are some techniques that I can use for anomaly detection given a non-Normal distribution? I have less than twenty available observations.

I would suggest a nearest neighbors approach. This technique is non-parametric, such that it does not assume your features follow any given distribution. The degree from which a novel instance can be classified as anomalous can set through some p-value estimation. These techniques are computationally expensive however due to your small dataset this may be well suited.

Check out:

Learning Minimum Volume Sets
http://www.stat.rice.edu/~cscott/pubs/minvol06jmlr.pdf

Anomaly Detection with Score functions based on Nearest Neighbor Graphs
https://arxiv.org/abs/0910.5461

New statistic in P-value estimation for anomaly detection
http://ieeexplore.ieee.org/document/6319713/

You can also use more rudimentary anomaly detection techniques such as a generalized likelihood ratio test. But, this is kind of old-school.


Tool vs Python Script for Transforming Data in Mongo We have a bunch of Mongo collections (data collected from APIs, web scraping, etc) that we need to transform to a cleaner data structure (standardized schema) on a monthly basis. Are there any good tools to help us manage this process, or would you recommend writing a Python script instead?

Many ETL tools support MongoDB and are commonly used for normalizing and standardizing data. For example:

http://www.pentaho.com/analytics-for-mongodb#tab-2


Covariates in machine learning classificatoin I would like to classify a number of people as sick and healthy. I have all the body measures as features. I have to control for age and sex because body features are affected by that. How do I adjust for age and sex?
Should I just add all the variables including age and sex as variables and try to classify? Or, should I do something else?

As a classification task, you should add age and sex as features to your classifier in addition to your body measures. This will incorporate to be considered when learning or making a prediction.
So your first option would be reasonable.


How can I replace outliers with maximum non-outlier value? I am doing univariate outlier detection in python. When I detect outliers for a variable, I know that the value should be whatever the highest non-outlier value is (i.e., the max if there were no outliers).

How can I impute this value in python or sklearn? I guess I can remove the values, get the max, replace the outliers and bring them back. But hoping there’s a function for that already.

Second, is this a bad idea? I see others remove the outlier completely or replace with the mean or median. I wonder if my approach is wrong.

My answer to the first question is use numpy's percentile function. And then, with y being the target vector and Tr the percentile level chose,  try something like

import numpy as np
value = np.percentile(y, Tr)
for i in range(len(y)):
   if y[i] > value:
       y[i]= value

For the second question, I guess I would remove them or replace them with the mean if the outliers are an obvious mistake. But your approach seems reasonable otherwise.


Classify big changes in target variable I am using a CNN to predict large changes in my target variable X. I am classifying several "set-up" states visually from my images.

I am only interested in big changes, or maybe no change. So, I can classify in 3 ways:

x is future period absolute change in X

Label BigChange when x > A%. Label Flat when x < A%.
Label BigChange when x > A%. Label Flat when x < B%, where B < A.
Label BigChange when x > A%. Label Flat when x < B%. Label SmallChange when x > B && x < A.

Which of these approaches is likely to give me the best predictive accuracy for BigChange?

  EDIT 1: Additional clarification

Input data is a set of images (60x60) with connection to Target variable (Float) X.

x is the delta between X values, and a big value might be 2-3% change. I am interested when the next x > some A (maybe 2%) i.e. there is a BigChange, but if x < B ( maybe 0.25% ) then I call that Flat. I want to classify my images so that I can predict a big change in x or when change is small. But I am NOT interested in predicting anything else (like changes > 0.25% but < 2%).

So, I am asking whether I should just classify and train on images that I have previously labelled BigChange or Flat. BUT this means when I run model on new data it will received lots of images that represent changes in x that are NOT Big or Flat. Is this problematic? Should I label things I  am not interested in aswell for completeness?

I can only offer comments base on your statements. First of all,set your goal clear and straight. I think it is the maximization of BigChange and Flat Sensitivities .

I think Number 2 is problematic because of your concern of upcoming images. Classification is not Regression. There is a chance that model will never have a clue about the middle ranges. But you can set the classification threshold for flat and big change prediction for maximum sensitivities. The remaining unclassified images will be middle range changes. The problem here is that there could be a lot of mixed-ups from two different thresholds and unusual metric.

So I think 3 wins because 1 doesn't care if it is small or middle range changes. It is the most straightforward thing towards your goal. Note that even if you are classifying with 3 things the metric for optimization can just involve two things. But you should verify it. I think it is all right to compare the two remaining models. Cross validate and choose the winner base on combined sensitivity metric (including only flat and big changes). Just don't be fooled by randomness in selection.

You might just cross validate three models in the end just to be sure and always keep in mind the unusual metric for optimization. In addition, you should always remember classification thresholding (not target thresholding) for all models and there can always be a choice of no classification for some examples after thresholding.

I'll be happy if you share the result of your experimentation and verify my hunch if it is wrong or not.


How will a decision tree cope if it cannot find a suitable feature to choose as root node from which to split further? When I look at decision trees, they start with a root node choosing the most suitable feature from which to split further. What if the decision tree is unable to find the most suitable feature from the data as root node? How does the decision tree cope in that situation?

In scikitlearn at least the code is in this file: https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/tree/_splitter.pyx.

Basically the determination of the root node is the same as the determination of any other node, it's just the first one that's done. What I think it's doing is either random splits (in which case your question is irrelevant) or "best" split, where best is determined by the evaluation of some criterion like decrease of gini impurity for each feature. In the case of a "Best" split it's evaluating the criterion for each feature in turn and keeping track of the best one, only overwriting that where a new feature's criterion evaluation is strictly better than the currently stored best feature. That strictly better evaluation means that, where two features have the same value for the criterion, the model will select the first feature it encounters as the one to split.

In other words, if Feature A and Feature B would result in an equally good split, it will set the root node to be Feature A purely because it evaluated that first.


Replacing missing value by class conditional mean I have two classes, $p(x|y=0)$ and $p(x|y=1)$ with ${{\mu }_{0}}$ and ${{\mu }_{1}}$ as mean and shared covariance matrix $\Sigma $. Now, I have a missing feature ${{x}_{n}}$ for a particular observation. We replace this missing value by the class conditional mean $E({{x}_{n}}|y=0)$ and $E({{x}_{n}}|y=1)$. How do I justify this missing value treatment.

This is actually an exercise. But, something that I noticed is that if $p(x|y=0)$ and $p(x|y=1)$ have means ${{\mu }_{0}}$ and ${{\mu }_{1}}$, then how is their covariance matrix the same?

Assuming their covariance matrix is same, the only advantage I see is that if I take class conditional mean as missing value then in multivariate Gaussian, the effect of this is that the $n^{th}$ row of $\left( x-\mu  \right)$ vector will be $0$ since $\left( {{x}_{n}}-{{\mu }_{n}} \right)$ will be $0$, thus making no impact in calculating $p(y=0|x)$ and $p(y=1|x)$.

Is this explanation correct?

Your intuition about 'no effect' is true in some sense. But this replacement may be not the best use of the information you have.

The choice of missing value treatment depends on your initial problem statement. In all the cases I assume that you have already somehow estimated conditional means $\mu_0$ and $\mu_1$ and the common variance matrix $S$.

Problem 1. For your test observation, you know $y$, but know nothing about $x$, and want to estimate $x_n$. If this is the case, $\mu_{y, n}$ is indeed your best guess of $x_n$.

Problem 2. For your test observation, you know $x_{-n}$ (that is, $x$ except $x_n$) and $y$, and want to predict $x_n$ as best as possible. If it is the case, you should use expectation conditional not only on $y$, but also on $x_{-n}$:
$$
\hat{x_n} = \mathbb{E}(x_n|y, x_{-n}) = \mu_{(y,n)} + S_{(n,-n)}S_{(-n)}^{-1}(x_{-n}-\mu_{(y,-n)})
$$
where $S_{*}$, $\mu_{y,*}$ are corresponding parts of covariance matrix and conditional expectation vector.

Problem 3. You know $x_{-n}$ but don't know $y$, and you want to estimate $x_n$. In this case, conditional distribution of $x_n$ is a mixture of two normal distributions, $p(x_n|y=0, x_{-n})$ and $p(x_n|y=1, x_{-n})$, with weights, proportional to $p(y=0|x_{-n})$ and $p(y=1|x_{-n})$ respectively. The mean of this distribution is just the weighted average of two means conditional on $y$. But this distribution may be bimodal, and its mean may be not a very useful statistic.

Problem 4. You know only $x_{-n}$, and your final goal is to estimate probability of $y=1$. In this case, you should just forget about $x_n$ (you don't know anything of it anyway), and use the distribution of $x_{-n}$ conditional on $y$ for whatever algorithm you apply.


Is there any package in python similar to darch in R? I want to work in Deep Belief Network using Python. Is there any package available in python similar to darch in R.

A simple google with keyword    gives me:

A tutorial about implement DBN from scratch
http://deeplearning.net/tutorial/DBN.html
A python package about the DBN
https://github.com/albertbup/deep-belief-network
Some interesting discussion about the DBN (and little history)
https://github.com/keras-team/keras/issues/748
SO discussion
https://stackoverflow.com/questions/23188131/deep-belief-network-with-scikit


Can tanh be used as an output for a binary classifier? I am creating a binary classifier in Keras and here;s the code

model = Sequential()
    model.add(Dense(30, input_dim=60, activation='relu'))
    model.add(Dense(1, activation='sigmoid'))
    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

I was wondering, if the dataset's range is changed from [0,1] to [-1,1], shouldn't I be able to use tanh as the activation of the output layer? If so, what are the advantages of one activation over the other?

If you label your data using -1 and 1 as classes, then yes you can. However, there are two reasons why data scientists normally prefer Sigmoid activations:

Loss functions, such as cross entropy based, are designed for data in the [0, 1] interval.
Better interpretability: data in [0, 1] can be thought as probabilities of belonging to acertain class, or as a model's confidence about it.

But yeah, you can use Tanh and train useful models with it.


How can a NN guess the velocity of an object from a video source? In the case of a video stream, I'd like to detect the speed (an approximation) of an object, that is moving. What would be the best approach to take?

I am thinking of 3 methodologies to take, though I've not found a lot resources to read about.

Method 1: I could feed Frame images to my CNN and pinpoint the object in the frame. Then calculate the difference of the position boxes.

Frame 1: [  o     ] ---> NN --> 3
Frame 2: [      o ] ---> NN --> 7

7-3 = 4 -> so dX = +4 between 1 frame, so I can estimate the dT (if I know the sampling rate of the frames)

Method 2: Is there any way where the NN can take as a feed-input the context of the previous frame and make the calculation itself?

video lapse 1sec: [  oooo ] ---> NN --> 4m/sec

Method 3: What if I can control the shutter speed of my camera, could I calculate the velocity of an object by the motion blur?

Frame image: [   ---o ] ---> NN --> 4m/$shutterSpeed

Any relevant resource to read would help a lot.

That is not possible without external labeled data (i.e., each video would have to be labeled with the object speed). Without labels, the neural network would be unable to learn the object's velocity because the object might be moving in any direction relative to the camera. For example, an object directly away would be getting smaller without motion blur.


Selecting ML algorithm for music composition I'm a composer and programmer. I'd like to use ML for composing music. There's already research on the general problem of composing by machine in known musical styles. I'm more interested in the specific problem of using a computer to assist myself in composing... to speed up certain repetitive tasks, or to delight me with unexpected ideas.

First, I want to design ML that can make a good/bad categorization on a single short passage. (Not looking to generate music at first.)

It's pretty easier to decompose such a passage into significant features. I can say things like

maybe there are particular "intervals" (harmony) present, and others absent
maybe notes are spread out evenly from low to high, or maybe not
maybe notes are clustered around a certain register rather than spread out
maybe the rhythm is particularly uneven
maybe a particular note occurs a lot

I can easily write algorithms that produce a list of such features for any given passage, and I can even be pretty sure that I'm including all of the relevant features. For an "art form," music is surprisingly amenable to analysis.

The trick is, first of all, there will be hundreds of observations, and second, that no one of these observations is good or bad in itself. Say we have three factors A, B, and C. A is not bad alone, but perhaps when A and B occur together, that's bad. But then maybe if A, B, and C all occur together, it becomes good again! And then maybe a fourth factor D, changes the context so that A, B, and C no longer matter much.

This seems like an ML problem, but I wonder what the best approach is?

I can construct or compose hundreds of examples of music labeled good and bad, and identify hundreds of "features" in each example. Some of the features are "present/not present" categories, while others are numbers that can be scaled from 0 to 1.

Is there a general and simple learning algorithm that even an ML novice such as myself (but I'm an experienced programmer) can implement?

I have to generate my own training examples and would be limited to around 300 to 1000, as I'm training it on simple passages in my style. This may actually be good problem to have... my job is not that complex compared to general musical intelligence.

I will contrast this task with image recognition. An image has millions of pixels and the features are spread out, occurring in endless  permutations. It makes my head hurt to try to figure out what's going on in a deep neural net.

However, in music, there will be at most a few hundred features. The patterns will not occur in endless permutations. In fact, in stripped-down situations my wild guess is that you could make an 80% accurate classifier by looking at 10 features (assuming you have the weights and combinations well-trained). I'm aiming somewhat higher than that (I need more features and a bit more accurate).

To give you some idea of my data set: I'm in the early stages of writing the software to extract the features, but I have a pretty good mental idea of what I'm aiming for (been composing by hand for years and years). To make this more concrete I created a web page showing what the data will look like. This is very simple and just totally made up.

http://theunexpectedpearl.com/dataset.html

This shows four rows. Each row is an example passage of music that I will compose or generate automatically. Each row is classified "good" or "bad" by my own ear. The rest of the row is a long list of features, each of which is either "yes/no" or a single number scaled from 0.0 to 1.0.

I don't think a Naive Bayes classifier will work for this (if I understand correctly) because features are neither good nor bad in and of themselves... it is only combinations of them (context) that matter. In fact, the very same feature can contribute to "goodness" in one context, and be an immediate strong indication of "bad" in another context.

The data could possibly be structured differently. In practice, this spreadsheet would be "sparse" and maybe features should be grouped in some way, or arranged in a tree.

Let us formulate this problem in such a way that it can be understood from a machine learning perspective. You have a set of instances $X$ where each instance $x_i \in \mathbb{R}^m$ where $m$ is the dimensionality of the instance. In other words $m$ is the number of features that describe the instance. Your problem intends to go from a set of features to a class label good or bad. Thus, this is a mapping from $\mathbb{R}^m$ to $y \in \{0, 1\}$.

How to achieve this mapping?

This is when we will use the machine learning algorithms. We will train a model to effectively approximate the function which gives the output label from a set of inputs. It is evident that sparse features (low information entropy) will complicate the mapping function and will thus provide worse results. This is why feature engineering is of upmost importance for machine learning. It is probably the hardest part of the machine learning pipeline, however it is the lead factor in dictating your results.

You can use some feature reduction techniques in order to remove features which are uninformative with respect to the output label. Some techniques that I use frequently are principle component analysis (PCA), linear discriminant analysis (LDA). Alternatively, you can use some projection methods to reduce the dimensionality of the data whilst maintaining separation between the classes. Such techniques are Isomap, MDS, Spectral Embeddings and TSNE. You can check to see which is best suited for your type of data.

How to choose a model?

Firstly, your problem is a supervised classification problem. This already narrows the types of models you can use. Furthermore, model selection is based on some key factors such as: the number of instances you have, the number of features per instance and the number of output nodes. You should also keep in consideration that the separability of the probability distribution between the output classes will impact the performance of the model directly. For example discriminating between cars and oranges is much easier than oranges and clementines.

In your case, you have 1,000 instances and around 13 features. This means that deep learning based techniques are possible but discouraged. You do not have enough data. You can then attempt the following popular classification models

Support Vector Classifier
Naive Bayes
K-Nearest Neighbors
Decision Trees
Random Forests

To evaluate which model performs the best you will use the accuracy attained with a trained model on a test set. This set should be drawn independently from the training set as to catch overfitting. This is when the model cannot generalize to new data.

In code

Assuming matrix $X$ contains the data where rows are the instances and columns are the features, and matrix $Y$ contains the labels.

First we split our data into a training and testing set

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33)

from sklearn.svm import SVC
clf = SVC()
clf.fit(X_train, y_train)
print('Score: ', clf.score(X_test, y_test))

from sklearn.neighbors import KNeighborsClassifier
neigh = KNeighborsClassifier(n_neighbors=3)
neigh.fit(X_train, y_train)
print('Score: ', neigh.score(X_test, y_test))

from sklearn import tree
clf = tree.DecisionTreeClassifier()
clf.fit(X_train, y_train)
print('Score: ', clf.score(X_test, y_test))

from sklearn.ensemble import RandomForestClassifier
forest = RandomForestClassifier(n_estimators = 100)
forest.fit(X_train, y_train)
print('Score: ', forest.score(X_test, y_test))

This should be a starting point. Let us know if you fall into any problems, and let us know what accuracy you are getting we can then look deeper into these models and better suit them to your data source.


Base model in ensemble learning I've been doing some research on ensemble learning and read that for base models, model with high variance are often recommended (can't remember which book I read this from exactly).
But, it seems counter-intuitive because wouldn't having base models with low variance(doing good on test set) be better than having multiple bad base models?

Intuitively speaking, ensembles benefit most from diversity.
Imagine being in a room of people making a decision together. If everyone more or less agrees, you don't benefit from having more people at the table. But if people tend to have different opinions, when they DO agree, it is a stronger message that the decision must be correct.
The same applies to ensembles. Models with high variance are more likely to produce different predictions, which will improve the quality of the prediction. High variance also minimizes the risk that multiple models are all wrong at the same time, based on the assumption that models are right more often than wrong.


Which solutions are there for RL agents when not all actions are always available? I'm working in an RL environment where not all actions are always available. In this case, depending on the state where the environment is at, some of the actions are not available for the agent to choose.
I could find this work that deals with this situation: https://ojs.aaai.org/index.php/AAAI/article/view/5740
Before moving on and starting implementing it, I'd like to ask if you have any other suggestions of solutions to deal with such a scenario.
I'd appreciate any pointer.

You could use an out-of-the-box learning algorithm if the availability of actions is deterministic and depends only on the state the agent is in, i.e. if availability of actions does not violate the Markovian assumption.
You need only substitute the full action space with the 'available action space' in any place where the action space shows up.
As an example, consider the update rule in Q learning, then replace the Q(S,A) with Q(S, A') where S' indicates the available actions for that particular S.
Please note that the referenced paper targets a setting in which availability of actions is not Markovian.


Is it correct to create topic models using both train and test data? I have a dataset of text documents splitted into train and test sets. My task is a binary classification, classifying these documents to either 1 or -1. I have already computed some features using TF-IDF and n-grams and tested my model. Now, I want to add some other features using topic models (LDA and LSA) to see if it can improve the F1-score and performance of my model.

My question is fairly simply: should I create my topic models only using my train set? Or, since topic models are not created based on a label (target or dependent variable,) would it be correct if I use both train and test sets to create the topic model?

You should only use your training set in this context though it may benefit you if you used an additional cross-validation set for feature selection. If you use training data in your feature selection step you will optimistically bias your model and expose it to overfitting because of the data leakage. A similar scenario is also described in this post.


Better input for Doc2Vec I want to perform Doc2Vec on a twitter dataset. As each tweet consists of a nummber of special characters ,numbers, urls, mentions and hashtags, non-english words, what should be my input for Doc2Vec? How should i approach for initial tweet pre-processing?

I saw many tutorials but all of them used plain texts. I am a newbie in data science.

There are plenty of different approaches you can use, and none is the universal best solution. However, in general, preprocessing in twitter data, especially for Doc2Vec, follow:

tokeninzing (nltk tokenizer, custom regex tokenizer) to identify words. Depending on the application, you can work on special cases such as english contractions, negations (sentiment analysis especially) and others
normalizing urls and/or mentions, transforming each of them to the same version to reduce voacbulary size (http://someurl, @mention) or totally removing them.
removing of numbers, punctuation or others. Note that this is highly domain dependent, for example "!" has been shown to express a lot in sentiment analysis.

In fact, doc2vec doesn't expect anything, it uses the vocabulary you give. So you decide what you want to keep. A good practice is to experiment different tokenizers and training phases if you have time to see what works best. Also use what has already been done in your field of application.

Don't forget that the input of Doc2Vec is an iterator over a list of TaggedDocument. See this tutorial for more.

Good luck !


DeepLearning: does it make sense to have more nodes in the initial layer than inputs--for tabular data I was looking at a couple of keras tutorials and something struck me as odd. So in these two tutorials, both on housing data, the number of nodes in the first layer of the network were larger than the number of columns in the dataset itself. NOTE: I am talking about tabular data and not image data or autoencoders. I understand that autoencoders will widen during the decoding phase, and I know that segmentation models will widen during the deconvolution phase. Both of these examples seem to be different than what I am seeing in these tutorials.

The first tutorial used data on Kings County, CA housing prices to create a simple neural network. In this case, the Kings County data only has 19 variables but the network itself has an initial dense layer of size 100 nodes. Here is the code:

def basic_model_1(x_size, y_size):
    t_model = Sequential()
    t_model.add(Dense(100, activation="tanh", input_shape=(x_size,)))
    t_model.add(Dense(50, activation="relu"))
    t_model.add(Dense(y_size))
    print(t_model.summary())
    t_model.compile(loss='mean_squared_error',
        optimizer=Adam(),
        metrics=[metrics.mae])
    return(t_model)

In the second example I was looking at Aurelion Geron's recent book on Keras and Tensorflow. In chapter 10 he uses some different California housing data to estimate a keras neural network. Again in that case he has 8 variables in the dataset, but he starts with a Dense layer of 30.

As a caveat, I did not see any columns get converted over to one-hot encodings or some sparse formulation.

This situation where the network starts wider than the data seemed odd to me, but perhaps that is just me coming from the statistics world. Can anyone explain this? Thanks.

This might seem a little bit confusing, especially for newcomers. But the problem here is basically agreeing on some terminology.

Typical Neural Networks by definition have 3 types of layers, namely Input, Hidden and Output. Because the Input layer has no specific function other that passing the input values to the next layer, many times is not mentioned. This layer does not have any weights, biases or activations and in the vast majority of frameworks (such as Keras) you don't have to write it in you code. You only need to specify the size of that input layer with the argument input_shape=(x_size,) in the first hidden layer.

What you call in your example initial layer of 100 nodes is actually that first hidden layer. Which can be on any size. And your intuition is completely right: the number of nodes in the input layer of course has to match the size of the inputs.

Hope this clarifies.


Weights and bias' relative to preprocessed X I am currently using sklearn scale to preprocess my X data before being put into a perceptron - mean/stddev so as to prevent the data converging to infinity or 0. My perceptron returns the weights + bias after the network has been trained:

X = preprocessing.scale(X)

After processing the X and Y data through my perceptron I am returned with weights. From these weights I can calculate the line of best fit:

ls = cp.linspace(cp.min(X), cp.max(X))
best_fit = w[1]+w[0]*ls

where w[1] is a bias. This best_fit line is accurate but it is relative to the preprocessed X rather than the original X which I would like to plot. What is the technique to make these weights relative to the original X values if it is possible?

If $$y = w_1 + w_0 \left(\frac{x_{}-\mu_1}{\sigma_1} \right),$$

then we have $$y = \left(w_1-\frac{\mu_1}{\sigma_1}\right)+ \left(\frac{w_0}{\sigma_1}\right)x$$


How to grid search feature selection and neural network hyperparameters in the same grid? I'm using the GridSearchCV () class from scikit to perform hyperparameter optimization in a sequential neural network. I've built a pipeline to also find the best number of features by putting a feature selector inside the pipeline. The problem is how to define the input_shape, since this depends on the k parameter from the feature selector. Is it possible to set the value of classifier__input_shape to be the same value (at all times) of feature_selector__feature__selector_k?
I've provided the correspondent piece of code below.
def create_model (learn_rate = 0.01, dropout_rate = 0.0, weight_constraint = 0, input_shape):
  model = Sequential ()
  model.add (Dense (units = 64, activation = 'relu',
                   input_shape = (input_shape, )))
  model.add (Dropout (dropout_rate))
  model.add (Dense (32, activation = 'relu'))
  model.add (Dense (1, activation = 'sigmoid'))
  model.compile (loss = 'binary_crossentropy',
                optimizer = Adam (lr = learn_rate),
                metrics = ['accuracy'])#, metrics.CategoricalAccuracy ()])
  return model

standard_scaler_features = remaining_features
my_scaler = StandardScaler ()
steps = list ()
steps.append (('scaler', my_scaler))
standard_scaler_transformer = Pipeline (steps)

my_feature_selector = SelectKBest ()
steps = list ()
steps.append (('feature_selector', my_feature_selector))
feature_selector_transformer = Pipeline (steps)

clf = KerasClassifier (build_fn = create_model, verbose = 2)
clf = Pipeline (steps = [('scaler', my_scaler),
                         ('feature_selector', feature_selector_transformer),
                         ('classifier', clf)],
                verbose = True)

param_grid = {'feature_selector__feature_selector__score_func' : [f_classif],
              'feature_selector__feature_selector__k' : [7, 9, 15],
              'classifier__input_shape' : [7, 9, 15],
              'classifier__epochs' : [2, 3, 4]}
cv = RepeatedStratifiedKFold (n_splits = 5, n_repeats = 1, random_state = STATE)
grid = GridSearchCV (estimator = clf, param_grid = param_grid, scoring = 'f1',
                     verbose = 1, n_jobs = 1, cv = cv)
grid_result = grid.fit (X_train_df, y_train_df)

And the error:
ValueError: Input 0 of layer sequential_9 is incompatible with the layer: expected axis -1 of input shape to have value 9 but received input with shape [None, 7]

I see two solutions:

either you pass a list of dictionnaries to param_grid avoiding irrelevant combinations
or you use a single variable in your pipeline for feature_selector__feature__selector_k and classifier__input_shape

First solution:  you can generate the right list of combinations using something close to this:
param_grid = [
    {
        'feature_selector__feature_selector__score_func' : [f_classif],
        'feature_selector__feature_selector__k' : [k],
        'classifier__input_shape' : [k],
        'classifier__dropout_rate' : [0.0, 0.5]
    }
    for k in [7, 9, 15]
             ]

Second solution, you can use a specific class that create your model when fitting based on the shape of X. Here is a code sample:
class MyKerasClf():
    def predict(self, X):
        y_pred_nn = self.clf.predict(X)
        return np.array(y_pred_nn).flatten()

    def create_model(self, learn_rate = 0.01, weight_constraint = 0 ):
        model = Sequential ()
        model.add (Dense (units = 64, activation = 'relu',
                       input_shape = (self.input_shape, )))
        model.add (Dropout (self.dropout_rate))
        model.add (Dense (32, activation = 'relu'))
        model.add (Dense (1, activation = 'sigmoid'))
        model.compile (loss = 'binary_crossentropy',
                    optimizer = Adam (lr = learn_rate),
                    metrics = ['accuracy'])
        return model

    def fit(self, X, y, **kwargs):
        self.input_shape = X.shape[1]
        self.clf = KerasClassifier(build_fn = self.create_model, verbose = 2)
        self.clf.fit(X, y, **kwargs)

    def set_params(self, **params):
        if 'dropout_rate' in params:
            self.dropout_rate = params['dropout_rate']
        else:
            self.dropout_rate = 0.0

Then you can use the class in your pipeline
X, y = make_classification(n_features=50, n_redundant=0, n_informative=2,
                           random_state=42, n_clusters_per_class=1)

my_scaler = StandardScaler ()
steps = list ()
steps.append (('scaler', my_scaler))
standard_scaler_transformer = Pipeline (steps)

my_feature_selector = SelectKBest ()
steps = list ()
steps.append (('feature_selector', my_feature_selector))
feature_selector_transformer = Pipeline (steps)

# Create a specific clf
my_clf = MyKerasClf( )
pip_clf = Pipeline (steps = [('scaler', my_scaler),
                         ('feature_selector', feature_selector_transformer),
                         ('classifier', my_clf)],
                verbose = True)

param_grid = {'feature_selector__feature_selector__score_func' : [f_classif],
              'feature_selector__feature_selector__k' : [7, 15],
              'classifier__dropout_rate' : [0.0, 0.5]
             }
cv = RepeatedStratifiedKFold (n_splits = 5, n_repeats = 1, random_state = 42)
grid = GridSearchCV (estimator = pip_clf, param_grid = param_grid, scoring = 'f1',
                     verbose = 1, n_jobs = 1, cv = cv)
grid_result = grid.fit(X, y)

Note: I although added the dropout to be tested in the gridsearch as an example.


Loss function for age classification I am building a CNN model for age classification. Assuming age of a person is between 1-100, my last Linear Layer contains 100 output neuron.
Now i want to find an appropriate loss function for this classification problem.
I dont wish to use Regression
My Observations:

I cannot use MSE or BCE loss because they only works element wise so unsuitable as if actual age is 25 then there will be same loss for predicted age 26 and 50. (Prediction will be 100 element vector as last layer has 100 neurons.)
I found about Hinge Loss and Cosine Proximity Loss. But i dont think they can be used in this type of classifiaction either because they are only finding similarity between two vectors without giving any importance or weight to nearby actual-predicted pairs (ex actual age 25 and predicted age 26 should have a very low loss)

Can anyone suggest me a suitable loss function (Preferrably in Pytorch) for this classification problem?
Edit
Lets say I want a Loss Function (L(predicted, actual)) such that (Assuming for 5 class classification)
let actual = [0,0,1,0,0]

L([0,0,1,0,0], actual) < L([0,1,0,0,0], actual) < L([1,0,0,0,0], actual)
L([0,0,1,0,0], actual) < L([0,0,0,1,0], actual) < L([0,0,0,0,1], actual)

If your network outputs a vector  $x \in \{0,1\}^N$ with $N=100$ and $\sum_{i = 1}^{N} x_{i}= 1$, you could consider weights $\mathbf{W}=(w_{i}) \in \mathbb{R}^{N}$ with $w_{i} :=i$ for $i \in \{1,\ldots,100\}$.
Then, for prediction vector $x$ and ground-truth vector $y$, you could use the loss function $L(x,y):= || \mathbf{W}^{T}x - \mathbf{W}^{T}y || = || \mathbf{W}^{T}(x-y)||$


Prohibitive size of random forest when saved to disk When saved to disk using cPickle: https://stackoverflow.com/questions/20662023/save-python-random-forest-model-to-file, my random forest is 6.57 GB.

with open('rforest.cpickle', 'wb') as f:
    cPickle.dump(rforest, f)

I want to use the forest itself to make predictions via a python API hosted on Heroku -- of course, that file size is unacceptable.

Why is the file size so large? There are 500 trees in the ensemble -- all I want to save are the completed trees themselves, since they will be used as prediction. Is it the actual nodes and edges that compose each of the 500 trees that requires nearly 7 GB of space on disk?

I used scikitlearn's randomforestregressor:

def buildForest(self, X_train, y_train):
    rf = RandomForestRegressor(n_estimators=500, verbose=1)
    rf.fit_transform(X_train, y_train)
    return rf

Also, if there is a better way to make my model accessible via API, that would also be good to know.

Update: I reduced it to 100 trees without losing much predictive power, and so now the saved size is 1.3 GB -- much more manageable, but still not great.

The size of each tree depends very much on its depth. Thus, change the maximal depth (max_depth). Try to set it to finite number (as opposed to the default "None") and then try to reduce this number. In addition (or as alternative) try to increase min_samples_split or min_samples_split.

You can also analyze you features and keep only important ones. The simplest way would be to have a look at the clf.feature_importances_ of your forest. (In general, finding important features is an art and science on itself.) Exclude non-relevant features and rebuild the forest.


Pandas how to fill missing values in one column if the values in another column are equal I have a dataframe where I need to fill in the missing values in one column (paid_date) by using the values from rows with the same value in a different column (id). There is guaranteed to be no more than 1 non-null value in the paid_date column per id value and the non-null value will always come before the null values. For example:

index     id       paid_date
6         25220    2017-01-05 00:00:00
9         30847    None
11        30847    None
14        29369    2017-06-21 00:00:00
17        31232    2017-08-31 00:00:00
20        26196    2017-02-20 00:00:00
21        26196    None
24        28303    2017-05-09 00:00:00
25        28303    None

How can I replace the None values in the paid_date column if there is a row with a paid_date with a matching id?

index     id       paid_date
6         25220    2017-01-05 00:00:00
9         30847    None
11        30847    None
14        29369    2017-06-21 00:00:00
17        31232    2017-08-31 00:00:00
20        26196    2017-02-20 00:00:00
21        26196    2017-02-20 00:00:00
24        28303    2017-05-09 00:00:00
25        28303    2017-05-09 00:00:00

I tried using fillna with a dictionary that mapped ids to paid_dates and I tried using pd.Series.map but neither worked.

paid_dates = df[pd.notnull(df['paid_date'])]
pds = pd.Series(data=paid_dates['paid_date'].values, index=paid_dates['id'])
pds_dict = pds.to_dict()

# doesn't work
df['paid_date'].fillna(value=pds_dict)

# also doesn't work
df['paid_date'].map(pds_dict)