Classifying transactions as malicious I have a big data set of fake transactions for a company. Each row contains the username, credit card number, time, device used, and amount of money in the transaction. I need to classify each transaction as either malicious or not malicious and I am lost for ideas on where to start. Doing it by hand would be silly.

I was thinking possibly checking for how often a credit card is used, if it is consistently used at a certain time, or if it is used from lots of different devices (iOS AND Android, as an example) would be possible starting places. I'm still fairly new to all this and ML. Would there be some ML algorithm optimal for this problem?

Also, side question: what would be a good place to host the 600 or so GB of data for cheaps?

Thanks

This problem is popularly called the "Credit Card Fraud Detection"

There are several classification algorithms, which aim to tackle this problem.

With the knowledge of the dataset you possess, the Decision Trees algorithm can be employed for detecting malicious transactions from the non-malicious ones. This paper is a nice resource to learn and develop the intuition about fraud detection and the usage of basic classification algorithms like the Decision Trees and the SVMs for solving the problem.

There are several other papers which solve this problems employing algorithms like Neural Networks, Logistic Regression, Genetic Algorithms, etc. However, the paper which uses the decision trees algorithm is a nice place to start learning.

what would be a good place to host the 600 or so GB of data for cheaps?

Aws S3 would be a nice, cheap way to do that. It also integrates nicely with Redshift, in case you want to do complex analytics on the data.


Data pre-processing before feeding into a deep learning model Generally speaking, when training a deep learning model, like MLP,  what kind of data pre-processing operation has to be conducted when the input is a numerical sequence.

It depends on the context of your problem.
If it is an sequence of images with not great quality then we need to perform some morphological operation on it e.g.grayscale,binary image etc.
Also if the digits are distorted,then we could translate it up-to certain extent for model to learn easily.


color compression using k-means algorithm I am reading about k means algorithm at this link.

At ln[22] here author mentioned that Input color space is 16 million possible colors. How author came up with 16 million number here. Kindly explain.

Additionally at the end it is mentioned as below

  Some detail is certainly lost in the rightmost panel, but the overall
  image is still easily recognizable. This image on the right achieves a
  compression factor of around 1 million! While this is an interesting
  application of k-means, there are certainly better way to compress
  information in images. But the example shows the power of thinking
  outside of the box with unsupervised methods like k-means.

How compression is done here as we are still using data of shape (273280, 3) after compression. How compression of 1 million is achieved.

Thanks

As already pointed out in a comment, the 16 million is the number of colors in the RGB space, $256^3=16\ 777\ 216$.  (The image itself only contains $96\ 615$ unique colors.)

As for the reduction, the author is generating 16 clusters; while the center points are still given in RGB format, there are only 16 of them.  (If you're following along in Colab or your own clone, run np.unique(new_colors, axis=0); that gives the 16 unique colors remaining in the compression.)  Now, to actually save the color-compressed image with smaller disk usage, you'd probably need to convert those 16 RGB codes with simple label encoding, and be able to reference their colors upon loading.  Anyway, as the author points out this isn't really meant to be an effective way to reduce image size, just an interesting example of clustering.


Confusion in Policy Iteration and Value iteration in Reinforcement learning in Dynamic Programming What I understood for value iteration while coding is that we need to have a policy fixed. According to that policy the value function of each state will be calculated. Right?

But in policy iteration the policy will change from time to time. Am I right?

In policy iteration, you define a starting policy and iterate towards the best one, by estimating the state value associated with the policy, and making changes to action choices. So the policy is explicitly stored and tracked on each major step. After each iteration of the policy, you re-calculate the value function for that policy to within a certain precision. That means you also work with value functions that measure actual policies. If you halted the iteration just after the value estimate, you would have a non-optimal policy and the value function for that policy.

In value iteration, you implicitly solve for the state values under an ideal policy. There is no need to define an actual policy during the iterations, you can derive it at the end from the values that you calculate. You could if you wish, after any iteration, use the state values to determine what "current" policy is predicted. The values will likely not approximate the value function for that predicted policy, although towards the end they will probably be close.


Is there any tool for data visualization and manipulation? I have a time series data set that I need to manually label for supervised learning.
What I am doing now is using excel to the plot, and when I see the pattern that I want, I hover over the data on the plot, read its index, then mark the data accordingly on the data. I think it is not very efficient, for example, I can not zoom or scroll.
I want to ask is there any tool that I can visualize the data, and manipulate the data at the same time, please.

What language are you using? A quick Google search of {lang} visualization can help you with this.

For example, Python has Bokeh, Pandas, and Seaborn. Bokeh is for interactive visualizations which can help with digging deeper into the visualizations. See:

Are there any python based data visualization toolkit?


Because an R xts object stores the date-column in a special column called index, and the programmers who implemented the  xts package have decided  to not print a label such as "Date" in the table header of an xts output.

When the print() method is called with an xts object as argument, the first column of the output is actually printed without any label, like a rowname "column" of a data frame.

Why? I don't know. Perhaps "Date" is not always right. It could just be a different label that your time series needs (e.g. "Microseconds" for some timeseries object of Sensor measurements)

# returned as data frame by Quandl
 R> str(a)

    'data.frame':   1985 obs. of  8 variables:
     $ Date                : Date, format: "2011-01-03" "2011-01-04" "2011-01-05" "2011-01-06" ...
     $ Open                : num  1154 1146 1097 1074 1045 ...
     $ High                : num  1158 1149 1099 1078 1078 ...
     $ Low                 : num  1139 1098 1065 1047 1041 ...
     $ Last                : num  1140 1107 1069 1050 1042 ...
     $ Close               : num  1145 1104 1069 1053 1049 ...
     $ Total Trade Quantity: num  1379646 4175490 5307600 4769781 6178882 ...
     $ Turnover (Lacs)     : num  15846 46520 57075 50448 65700 ...
     - attr(*, "freq")= chr "daily"

# returned as xts timeseries object

    R> str(ax)
    An ‘xts’ object on 2011-01-03/2019-01-04 containing:
      Data: num [1:1985, 1:7] 1154 1146 1097 1074 1045 ...
     - attr(*, "dimnames")=List of 2
      ..$ : NULL
      ..$ : chr [1:7] "Open" "High" "Low" "Last" ...
      Indexed by objects of class: [Date] TZ: UTC
      xts Attributes:
     NULL


Help required to implement the below model using Bi-GRU As you can see in above images I need to model Bi-GRUs stacked as shown in table which takes input (N,1,64) and outputs (N,204). The input data is binary number stream and so is output data. Can anyone please help me get started?

Thank you.

from keras.layers import LSTM
from keras.models import Sequential
from keras.layers import LSTM
from keras.layers import Dense,BatchNormalization
model = Sequential()
model.add(LSTM(800, return_sequences=True,
               input_shape=(1, 64)))  # returns a sequence of vectors of dimension 32
model.add(BatchNormalization())  # returns a sequence of vectors of dimension 32
model.add(LSTM(800))  # return a single vector of dimension 32
model.add(BatchNormalization())
model.add(Dense(204, activation='sigmoid'))

model.compile(loss='binary_crossentropy', optimizer='Adam',metrics=['accuracy'])

model.summary()

I figured it out on my own.


Binary classification as a 2-class classification problem I want to create a dog-classifier, which outputs the probability of an image containing a dog.

I have two approaches in mind -

Binary classifier (1-class), which just outputs the probability of the image containing a dog. This seems reasonable to me.
2-class classifier with two classes denoting "dog" and "not-dog". But my problem with this approach is that the neural network has to learn the "not-dog" class as well, which is impossible since it has no pattern and is different in each training example.

Would the second approach be less effective than the first? Or even work at all?

Thank you.

tl;dr

The two approaches you mention are equally effective.

Why?

First of all, I'll assume you're referring to a neural-network-type model. You should note that Convolutional Neural Networks are discriminative models, meaning that they are trained to find the differences between two classes (e.g. dog and not-dog).

  Would the second approach be less effective than the first?

Just to disprove this claim, consider the following. You have a 2-output classification network; let's name the outputs dog and not-dog. Now imagine, as you say, that the network can only identify dog patterns. All it would have to do was to give a positive weight to those patterns for the dog class and a negative weight for the not-dog class. In this sense the not-dog class would be trained as the opposite of the dog class, which would not make it less effective than a single-output binary classifier.

Even if the not-dog class could not be trained and remained constant, due to the fact that we're using a softmax activation, predictions would be generated just by the relative difference between the dog output and the constant not-dog output. This is exactly like having a single-output binary classifier.


K-Means initialization K-Means initializes the centroids randomly, but there are other methods to initialize. In this paper, http://ilpubs.stanford.edu:8090/778/1/2006-13.pdf, they propose randomly choosing a data point initially then choose the other centroids based on the distance from the initial centroid.

My question is: how does this give you the right result? Say my data clusters naturally into three clusters, a noisy cluster around the (x, y) points (1, 1), (0, 0), and (-1, -1). Say I use the method from the paper and initially choose a data point (1.32, 0.98) and mark it as the center of cluster #1. According to the paper, I choose the next centroid based on distance, so the next point will be around (-1, -1). Say the data point chosen for cluster #2 is (-1.12, -0.89). These first two steps make sense, but now I continue on to cluster #3 and again I chose based on distance so I'll end up putting another cluster center very close to cluster #2's center. What am I missing here? Shouldn't the centers be chosen based on the sum of distances from the already initialized cluster centers?

EDIT:
Initially, I randomly choose a data point to mark as center of cluster #1. I choose the red point. Now I calculate the distances between red point and all other data points and choose the furthest point away as center of cluster #2. This is the green point. My question is: according to the paper, I repeat this and calculate distances from the red point to all remaining points and take the furthest away, but this puts me back near the green point, but I was trying to get to the center cluster.

The nth centroid is chosen from a distribution proportional to $D(x)^2$, but pay careful attention to how $D(x)$ is defined.  From the paper (top of page 3):

  In particular, let $D(x)$ denote the shortest distance from a data point to the closest center we have already chosen.

Notice that $D(x)$ is the distance from $x$ to the nearest centroid.  Compare that to the description of how you are choosing the points:

  I repeat this and calculate distances from the red point to all remaining points and take the furthest away, but this puts me back near the green point, but I was trying to get to the center cluster.

Instead of computing the distance between all remaining points and the red point, you should instead compute the distance between each point and its nearest centroid.

For the points in the cluster around (1, 1) you'll compute the distance from the red point, and all the distances will be relatively small.  Similarly, for the points around (-1, -1), you'll compute the distance from the green point, and all these distances will be pretty small.

However, for points around (0, 0), some of them will be closer to the red centroid and some will be closer to the green centroid, but they are not very close to either one.  This means that $D(x)$ will be large for points in the (0, 0) cluster, and you are likely to choose one of these points as the next centroid.

Does that make sense?


Classifier not predicting real data I'm trying to train a classifier to recognize my own signature. This is how I built my classifier

How did I collect data?

Signed on a piece of paper for 50 times and created 50 images out of it.
for negative test cases, I downloaded IAM Handwriting database. which contains around 600MB of handwriting data. this is to negate other possible matches.

How did I do Feature Engineering?

Step 1 : Read and convert image in gray scale. Perform median blur.

img = cv2.imread(image,0)
img = cv2.medianBlur(img,5)

Step 2 : Perform adaptive threshold followed by morphological opening to reduce noise in the image.

edges = cv2.adaptiveThreshold(img,255,cv2.ADAPTIVE_THRESH_GAUSSIAN_C,cv2.THRESH_BINARY_INV,11,2)
kernel = np.ones((2,2),np.uint8)
dilation = cv2.morphologyEx(edges, cv2.MORPH_OPEN, kernel)

How did I do model training?

Extracted ORB features from all the sample images for training (matrix size
RX32) and used a RandomForestClassifier for training.

The problem

The accuracy of my classifier is a whopping 0.9874066374996978, but it fails to recognize almost all of my new signature samples, signed on same paper under same lighting conditions. I'm new to applied ML. What do you experts think I should check??

You are effectively trying to classify images. Convolutional neural networks are much better at this task. And Keras is an easy to use library for neural networks.

Check tutorials such as: https://machinelearningmastery.com/handwritten-digit-recognition-using-convolutional-neural-networks-python-keras/

and

https://www.pyimagesearch.com/2017/12/11/image-classification-with-keras-and-deep-learning/


Is (manual) feature extraction outdated? I recently attended a PhD thesis defence in which one committee members claimed that "manual feature extraction is outdated. Nowadays, we have [deep] machine learning models doing that job for us automatically."

Is this statement true? If yes, please provide a reference substantiating this claim.

Edit: Apparently, there seem to be different answers depending on the data type. Thus, please let me know about any references substantiating your claims for images, time series, etc... separately.

In the general case, this is by no means true. Let's break down the case for different data scenarios:

For discriminative image models (e.g. image classification/labeling) this is true for some scenarios. You just throw some convnets (even pretrained models) at your data, and that's it. Nevertheless, convnets themselves profit from the "expert knowledge" that information locality is important and so is hierarchical information processing. For some other scenarios, applying domain knowledge (e.g. specific data transformations) may give the edge to reach the needed level of quality in the results.
For many image processing problems, neural networks work best when infused with some kind of inductive bias, e.g. attention.
For Natural Language Processing (NLP) problems, a good amount of craftsmanship is needed nowadays, especially in the data preprocessing stage.
For "typical data science" problems, it is also crucial to do feature extraction. You can have a look at Kaggle competitions to verify this.
For time series problems, it is also normal to rely on expert knowledge to understand which models fit best based on the nature of the data.

However, I think that the trend of the areas where deep learning is applicable (i.e. tons of available data) is to try to devise systems that are trained end-to-end, with the least possible ad hoc processing. Nevertheless, many times this is achieved by infusing the expert knowledge into the network in the form of inductive biases.


Using word embeddings with additional features I have the set of queries for classification task using Gradient Boosting Classifier of scikit learn. I want to enrich the model by feeding additional features along with GloVe. How should I approach scaling in this case? GloVe is already well scaled, however, features are not.
I have tried StandardScaler, but this reduced the performance in comparison with just using GloVe. The problem maybe with the feature itself, however, I need your opinion on scaling starategies in case of glove and dummy variable.

My first comment would be that you have to remember that Tree-based models are not scale-sensitive and therefore scaling should not affect model's performance, so as you well mention it should a problem with the feature itself.
If anyway you want to scale all your features you could use MinMaxScaler with the min and max values, being the min and max fo the Glove Vectors so that all the features are on the same scale


Is it possible to deploy a python trained machine learning model (e.g. a .pkl file) in C language? I would like to train my machine learning using Python and libraries such as tensor flow, keras, and scikit-learn. After trained, I would like to export this trained model to a file, so far I have been using the library pickle. I feel that this is pretty standard in any ML project.

However, and the point of this question, is it possible to use this trained model (e.g. a .pkl or .sav file) in C language? When I say "use", I mean passing values to the model and getting a prediction back.

This is somewhat similar to TensorFlow Lite for Microcontrollers, however, I am not sure if this is the most appropriate and easier approach to this problem.

Library http://www.picklingtools.com/

#include "chooseser.h"

// assume that pickle file is "loaded_pkl";
int main()
{

  Val loaded_pkl;
  LoadValFromFile(loaded_pkl, "my_pkl.pkl", SERIALIZE_P0);
  cout << loaded_pkl << endl;

}


Product classification according to description I have some products, along with their description. I wish to assign USPSC code to each product. I have a really basic doubt here. What exactly is my test file and training file? Eg. Should the training file be entries of product description along with manually entered codes assigned to each product? And the test file only product descriptions?

So the question is what should the training and test sets consist of for classifying products by their product description.
To clarify, for any machine learning task we require data. In the case of supervised machine learning, the dataset will be labelled (in your case with USPSC codes).
The dataset is then split into a training, (validation if possible) and test set.
So the answer to your question would be that both trading and test sets will have the same format in that they contain the input text which will go into the model. The only real difference is that the labels in the training set are observed so that the model can find an underlying function which translates the text into a class. In the test set, the labels are used to evaluate the model.


How is the "base value" of SHAP values calculated? I'm trying to understand how the base value is calculated. So I used an example from SHAP's github notebook, Census income classification with LightGBM.
Right after I trained the lightgbm model, I applied explainer.shap_values() on each row of the test set individually. By using force_plot(), it yields the base value, model output value, and the contributions of features, as shown below:

My understanding is that the base value is derived when the model has no features. But how is it actually calculated in SHAP?

As you say, it's the value of a feature-less model, which generally is the average of the outcome variable in the training set (often in log-odds, if classification).  With force_plot, you actually pass your desired base value as the first parameter; in that notebook's case it is explainer.expected_value[1], the average of the second class.

https://github.com/slundberg/shap/blob/06c9d18f3dd014e9ed037a084f48bfaf1bc8f75a/shap/plots/force.py#L31

https://github.com/slundberg/shap/issues/352#issuecomment-447485624


Is logistic regression actually a regression algorithm? The usual definition of regression (as far as I am aware) is predicting a continuous output variable from a given set of input variables.

Logistic regression is a binary classification algorithm, so it produces a categorical output.

Is it really a regression algorithm? If so, why?

Logistic regression is regression, first and foremost. It becomes a classifier by adding a decision rule. I will give an example that goes backwards. That is, instead of taking data and fitting a model, I'm going to start with the model in order to show how this is truly a regression problem.

In logistic regression, we are modeling the log odds, or logit, that an event occurs, which is a continuous quantity. If the probability that event $A$ occurs is $P(A)$, the odds are:

$$\frac{P(A)}{1 - P(A)}$$

The log odds, then, are:

$$\log \left( \frac{P(A)}{1 - P(A)}\right)$$

As in linear regression, we model this with a linear combination of coefficients and predictors:

$$\operatorname{logit} = b_0 + b_1x_1 + b_2x_2 + \cdots$$

Imagine we are given a model of whether a person has gray hair. Our model uses age as the only predictor. Here, our event A = a person has gray hair:

log odds of gray hair = -10 + 0.25 * age

...Regression! Here is some Python code and a plot:

%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns

x = np.linspace(0, 100, 100)

def log_odds(x):
    return -10 + .25 * x

plt.plot(x, log_odds(x))
plt.xlabel("age")
plt.ylabel("log odds of gray hair")

Now, let's make it a classifier. First, we need to transform the log odds to get out our probability $P(A)$. We can use the sigmoid function:

$$P(A) = \frac1{1 + \exp(-\text{log odds}))}$$

Here's the code:

plt.plot(x, 1 / (1 + np.exp(-log_odds(x))))
plt.xlabel("age")
plt.ylabel("probability of gray hair")

The last thing we need to make this a classifier is to add a decision rule. One very common rule is to classify a success whenever $P(A) > 0.5$. We will adopt that rule, which implies that our classifier will predict gray hair whenever a person is older than 40 and will predict non-gray hair whenever a person is under 40.

Logistic regression works great as a classifier in more realistic examples too, but before it can be a classifier, it must be a regression technique!


Advanced Activation Layers in Deep Neural Networks I'm wondering about the benefits of advanced activation layers such as LeakyReLU, Parametric ReLU, and Exponential Linear Unit (ELU). What are the differences between them and how do they benefit training?

ReLU
Simply rectifies the input, meaning positive inputs are retained but negatives give an output of zero. (Hahnloser et al. 2010)
$$
f(x) = max(0,x)
$$
Pros:

Eliminates the vanishing/exploding gradient problem. (true for all following as well)
Sparse activation. (true for all following as well)
Noise-robust deactivation state (i.e. does not attempt to encode the degree of absence).

Cons:

Dying ReLU problem (many neurons end up in a state where they are inactive for most or all inputs).
Not differentiable. (true for all following as well)
No negative values means mean unit activation is often far from zero. This slows down learning.

Leaky ReLUs
Adds a small coefficient ($<1$) for negative values. (Maas, Hannun, & Ng 2013)
$$
f(x) = \begin{cases}
x & \text{if } x \geq 0 \\
0.1 x & \text{otherwise}
\end{cases}
$$
Pros:

Alleviates dying ReLU problem. (true for all following)
Negative activations push mean unit activation closer to zero and thus speeds up learning. (true for all following)

Cons:

Deactivation state is not noise-robust (i.e. noise in deactivation results in different levels of absence).

PReLUs
Just like Leaky ReLUs but with a learnable coefficient. (Note that in the below equation a different $a$ can be learned for different channels.) (He et al. 2015)
$$
f(x) = \begin{cases}
x & \text{if } x \geq 0 \\
a x & \text{otherwise}
\end{cases}
$$
Pros:

Improved performance (lower error rate on benchmark tasks) compared to Leaky ReLUs.

Cons:

Deactivation state is not noise-robust (i.e. noise in deactivation results in different levels of absence).

ELUs
$$
f(x) = \begin{cases}
x & \text{if } x \geq 0 \\
\alpha(exp(x)-1) & \text{otherwise}
\end{cases}
$$
Replaces the small linear gradient of Leaky ReLUs and PReLUs with a vanishing gradient. (Clevert, Unterthiner, Hochreiter 2016)
Pros:

Improved performance (lower error and faster learning) compared to ReLUs.
Deactivation state is noise-robust.


What does "Find pattern" mean in data science? I am doing certification and i have a project to complete. In project, they have said "Find Patterns". What does it mean? what steps should i carry out?

First, I'm going to directly quote the two inner questions you put in this question:

  "Find Patterns", What does it mean?:

Find patterns is the term given to the process of analyzing data(Data Analysis) or dataset for insights in it that is to say; the classification of input into per-determined categories,recognize particular categories of input by their characteristics.

So when they ask you to find patterns,they assume that you don't necessarily know what is in that data(it can be structured,semi-structured and unstructured) so you can't even make decisions upon that data.therefore you have to find patterns in it. You have to look for them, which can often involve a lot of hunch-following and guess-work combined with a bit of solid testing.you tend to know the patterns already and you search for them based on their features or attributes etc.

  what steps should i carry out?

Ask Question
Do some background research
Construct hypothesis
Test with an experiment
Analyze results and draw effective conclusions.

Concerning Hypothesis

If the hypothesis is true,then proceed and report results
or Else,if it's false or partially true report results.

Hope this can give you a glimpse.


Custom functions and pipelines I'm not really used to working with pipelines, so I'm wondering how can I use custom functions and pipelines.

Situation: I want to fill some missing values with the mean but using groups based on other feature. That's why I'm using this custom function:

def replaceNullFromGroup(From, To, variable, by):

    # 1. Create aggregation from train dataset
    From_grp = From.groupby(by)[variable].median().reset_index()

    # 2. Merge dataframes
    To_merged = To.merge(From_grp, on=by, suffixes=['_test', '_train'], how = "left")

    # 3. Create dictionaries
    to_cols = [col for col in To_merged.columns if 'test' in col]
    from_cols = [col for col in To_merged.columns if 'train' in col]
    dict_cols =dict(zip(to_cols, from_cols))

    # 4. Replace null values
    for to_col, from_col  in dict_cols.items():
        To_merged[to_col] = np.where(To_merged[to_col].isnull(),
                                     To_merged[from_col],
                                     To_merged[to_col])

    # 5. Clean up dataframe
    To_merged.drop(from_col, axis=1, inplace=True)
    To_merged.columns = To_merged.columns.str.replace('_test', '')
    return To_merged

Variables meaning:

From: Dataframe where I'm taking the information (Train dataset)
To: Dataframe where I will fill the missing values (Train and test dataset)
variable: variable with missing values
by: Variables I'm using to make groups

Can I use this function in a pipeline so I can use cross validation avoiding data leakage?

Thank you very much

To include this logic into a pipeline you have to create a custom transformer. You need to ask yourself:

[INIT] Are there any parameters in my logic?

The variable you want to impute and the category you want this imputation to be based on.

[FIT] What part of the logic is related to computing what the transformation will be?

When you compute the median() by groups and store the data somehow for later transform.

[TRANSFORM] What part of the logic transforms the data, given the parameters (in 1) and the setting that was made (in 2)?

When you get the parameters (access specific key in the dictionary) to retrieve what was the mean for that group, and then fill the missing value with this.

Here is an example :

from sklearn.base import BaseEstimator, TransformerMixin

class CustomImputer(BaseEstimator, TransformerMixin) :
     def __init__(self, variable, by) :
          #self.something enables you to include the passed parameters
          #as object attributes and use it in other methods of the class
          self.variable = variable
          self.by = by
          return self

     def fit(self, X, y=None) :
          self.map = X.groupby(self.by)[variable].mean()
          #self.map become an attribute that is, the map of values to
          #impute in function of index (corresponding table, like a dict)
          return self

     def transform(self, X, y=None) :
          X[variable] = X[variable].fillna(value = X[by].map(self.map))
          #Change the variable column. If the value is missing, value should
          #be replaced by the mapping of column "by" according to the map you
          #created in fit method (self.map)
          return X

Now, it can be included in any pipeline :

#Minimal example, you could include this imputer in columns transformer to
#apply it multiple time
pipeline = Pipeline(steps = [('myImputer', CustomImputer('variabletofill',
                                                         'based_on_variable'),
                              ('model', LinearRegression())])

y_pred = pipeline.fit(X_train, y_train).predict(X_test)

As you can see, the mapping is computed only based on train data. Then it is re-used to impute the missing values. It is data-leakage-proof.
Here is a good article that explains how to create a custom transformer.

Hope this helps


What is the difference between multiply and dot functions that is used to merge layer in Keras? I want to merge two CNN deep learning model using Keras and would like to know what is the difference multiply and dot functions that is used to merge layer?

keras.layers.multiply(inputs)

keras.layers.dot(inputs, axes, normalize=False)

The multiply() function performs element-wise multiplication. For example, let us consider 1D CNN for simplicity and you pass two inputs of batch size b with a tensor length of 5, the output will be (b,5) as it's element-wise multiplication.
Let us assume two tensors of length 5 as follows: [1,2,3,4,5] and [6,7,8,9,10], the result shall be [6,14,24,36,50] as it's just element-wise multiplication.

In the case of dot(), it takes the dot product, and the dot product for 1D is mathematically defined as: a.b = sum(a_i * b_i), where i ranges from 0 to n-1; where n is the number of elements in vector a and b. (For 2-D , you can consider it as matrix multiplication). So the result shall be of length (b,1) where b is the  batch size. In case of out example, the results of dot product of [1,2,3,4,5] and [6,7,8,9,10] shall be [130]


Compare Coefficients of Different Regression Models in my project, I am using asuite of shallow and deep learning models in order to see which has the best performance on my data. However, in the pool of shallow machine learning models, I want to be able to compare the coefficients of each regression model between each other. Example: I have

Lasso
Ridge
ElasticNet
AdaBoostRegressor
GradientBoostRegressor

I am aware that I could get the coefficients of Lasso, Ridge, and ElasticNet from model.coef_ and model.intercept_ from sklearn. However, AdaBoostRegressor does not have this, but rather, it has weights assigned to each of the estimators. Similarly, GradientBoostingRegressor of sklearn does not even have weights at all. How am I to compare the coefficients between each of the regressive models I am using??

Weights across different types of models are not always comparable, so
I think that it would make more sense to do this kind of comparison not across different types of model but within a single type of model varying:

the hyper-parameters (if any),
the set of instances (e.g. selecting different subsets randomly),
the set of features.

I'd recommend in particular varying the instances: if the weight of a feature tends to change a lot depending on the training set it's a sign of overfitting.

If the goal is to compare the importance of some specific features for different models, it's probably more reliable to directly evaluate how the model performs with/without the feature(s).


Google Earth Pro Satellite image segmentation using clustering I have downloaded a satellite image from Google Earth Pro software corresponding to a particular date for a selected area around a place. I want to specifically segment the road lanes from the image using Kmeans algorithm. How can this be achieved?

Whoa, are you trying to do clustering based on satellite images?!  Just use the underlying long & lat coordinates to do this.  It will be infinitely easier and so much more accurate too.  Look at the example code below and try to adapt that to your specific case.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans
import seaborn as sns; sns.set()
import csv

df = pd.read_csv('C:\\your_path\\properties_2017.csv')
df.head(10)

df.dropna(axis=0,how='any',subset=['latitude','longitude'],inplace=True)

# Variable with the Longitude and Latitude
X=df.loc[:,['parcelid','latitude','longitude']]
X.head(10)

K_clusters = range(1,10)
kmeans = [KMeans(n_clusters=i)

for i in K_clusters]
Y_axis = df[['latitude']]
X_axis = df[['longitude']]
score = [kmeans[i].fit(Y_axis).score(Y_axis)

for i in range(len(kmeans))] # Visualize
plt.plot(K_clusters, score)
plt.xlabel('Number of Clusters')
plt.ylabel('Score')
plt.title('Elbow Curve')
plt.show()

kmeans = KMeans(n_clusters = 3, init ='k-means++')
kmeans.fit(X[X.columns[1:3]]) # Compute k-means clustering.X['cluster_label'] = kmeans.fit_predict(X[X.columns[1:3]])centers = kmeans.cluster_centers_ # Coordinates of cluster centers.labels = kmeans.predict(X[X.columns[1:3]]) # Labels of each pointX.head(10)

X['cluster_label'] = kmeans.fit_predict(X[X.columns[1:3]])
centers = kmeans.cluster_centers_ # Coordinates of cluster centers.
labels = kmeans.predict(X[X.columns[1:3]]) # Labels of each pointX.head(10)

X.head(5)

X = X[['parcelid','cluster_label']]
X.head(5)

clustered_data = df.merge(X, left_on='parcelid', right_on='parcelid')
clustered_data.head(5)

centers = kmeans.cluster_centers_
print(centers)

# Final Result:
   parcelid  cluster_label
0  10754147              0
1  10759547              0
2  10843547              2
3  10859147              2
4  10879947              2

The data source is here:

https://www.kaggle.com/c/zillow-prize-1/data


How to calculate KL-divergence between matrices Given there are two matrices of dimensionality 100x2 with absolute values ranging from -50 to +50. Is it possible to determine the kl-divergence by applying the entropy algorithm from scipy.stats to the two flattened vectors of size 200?

I think you can. Just normalize both of the vectors to be sure they are distributions. Then you can apply the kl divergence .
Note the following:
- you need to use a very small value when calculating the kl-d to avoid division by zero. In other words , replace any zero value with ver small value
- kl-d is not a metric . Kl(AB) does not equal KL(BA) . If you are interested in it as a metric you have to use the symmetric kl = (Kl(AB) +KL(BA) )/2


Negative value in information gain calculation through gini index I am trying to  determine the root node for the decision tree on given data

annual income target variable has been renamed as low, mid, and high.

I am using gini index to measure the impurity of my nodes.

The process I am following is simple:

1- calculate the Gini index for the dataset(target is annual income)

gini(annual income)=1-((5/20)^2+(12/20)^2+(3/20)^2) = 0.445

2 - for each variable calculate gini and then remainder and information gain

3 - choose variable with the highest information gain

for remainder i am using this

just instead of entropy, I am using gini

when I am trying to calculate information gain if education becomes root note I am getting a negative information gain (which is obviously not possible)

MY CALCULATION:

as you can see I got a gini index of 0.532 for the node if I do

Information gain (0.445-0.532)=-ve value

can you point towards what am I doing wrong

I quickly checked your calculation and you seem to have miscalculated the gini(annual income)

  gini(annual income)=1-((5/20)^2+(12/20)^2+(3/20)^2) = 0.445

When it actually equals 0.555 (you probably forgot the 1-... part) which is larger than 0.532 so you might be fine for the rest of the calculations.


Is it wrong to transform the target variable and test the model without dropping the column that was transformed? What's the disadvantage about it? I have a linear regression model, I have transformed the target variable Item_Outlet_Sales into Item_Outlet_Sales_log on both training and testing dataset. I did not delete the Item_Outlet_Sales.
Here is the snippet:
#treat extreme values in Item_Outlet_Sales
train['Item_Outlet_Sales_log'] = np.log(train.Item_Outlet_Sales)
test['Item_Outlet_Sales_log'] = np.log(test.Item_Outlet_Sales)

sns.distplot(train.Item_Outlet_Sales_log);
sns.distplot(test.Item_Outlet_Sales_log);
#distribution

I dropped the target variable: Item_Outlet_Sales_log and assigned it to y
 #creating dummies for the training dataset
 X = train.drop('Item_Outlet_Sales_log', 1) #drop the log target column
 y = train.Item_Outlet_Sales_log

 X = pd.get_dummies(X)
 train = pd.get_dummies(train)
 test = pd.get_dummies(test)

Is it wrong to to do in terms of giving the model proper training? What is the disadvantage of it? Is it recommended to delete the old target variable or not?

The problem is that, by definition, your target variable is not available at inference time, and that is why you want to predict it. If your target variable was available at inference time, then there is no point in predicting it.
Therefore, if you use the target variable (or a transformation of it) as input to your model, what data are you going to feed to that variable at inference time?


With automated hyperparameter tuning available, do we still need to learn hyperparameter tuning Tools like AWS Sagemaker have capability to do automated hyperparameter tuning, even with complex algos like Neural Networks using Tensorflow. So do we still need to learn how to do hyperparameter tuning, or simply leave it to tools like Sagemaker? Thx

So do we still need to learn how to do hyperparameter tuning

If you're saying this based on the context of acquiring a new skill, then go for it. It's always a good thing to get an idea an idea of how hyper-parameter testing is done for real. In addition to sagemaker you can use tools like weights and biases


How do we interpret the outputs of DBSCAN clustering? I am starting to learn DBSCAN for clustering but the interpretation part of it seems to be tricky to understand.

dataset = np.vstack((quotient_times, quotient)).T

scaler = StandardScaler()
dataset = scaler.fit_transform(dataset)

db_scan = DBSCAN(eps=0.6, min_samples=1)
db_scan.fit(dataset)

colors = [int(i % 23) for i in db_scan.labels_]

plt.figure();
plt.title(fname)
plt.xlabel("time")
plt.ylabel("quotient")
plt.scatter(dataset[:, 0], dataset[:, 1], c=colors)
plt.legend()
plt.show()

This produces the following plot.

When i print the values of db_scan.labels_, I am getting

array([0, 0, 1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1])

I was expective my y-axis values from 0-1 and x-axis values from 1-7. Why do I have the negative scales both on the x-axis and y-axis? and how do we interpret these values and the plot?

The output from db_scan.labels_ is the assigned cluster value for each of the points that you provide as input to the algorithm.

You provided 20 points, so there are 20 labels. As explained in the relevant documentation, you will see:

  labels_ : array, shape = [n_samples]
  Cluster labels for each point in the dataset given to fit(). Noisy samples are given the label -1.

So each of your points was assigned to one of two clusters, and there were no points marked as noise (receiving a value of -1).

Interpreting the clusters

Your plot shows the original data, plotted on the axes using the original values; only the colour encodes the cluster to which each point was assigned.

As with many clustering algorithms, DBSCAN measures the distance between points in some n-dimensional space. It uses parameters you provide to decide if points belong to the same group, based on their distance from one another.

  Why do I have the negative scales both on the x-axis and y-axis?

The distance metrics used do not really care about the sign of the points (where they lie on an axis), simply the distances. In the case of DBSCAN, it is the Euclidean distance. There are other distances that you can pass to the metric parameter of the DBSCAN algorithm; here is the valid list.


Measuring Difference Between Two Sets of Likert Values I asked users to complete a likert survey (1-5) at the start of an activity, and conclusion of the activity. What is the best manner to show the rate of change/difference between these two result sets? I was considering a Pearson correlation, but after thinking about things more I don't believe that it is the most appropriate correlation metric.
My data looks similar to below:

Before   After
1         2
1     &nbsp   2
2     &nbsp   2
3     &nbsp   3
3     &nbsp   4
4     &nbsp   2

Result: XXX?

Thanks all

If you are trying to see whether the survey had made any positive effect on the people, modeling a relationship between before and after with correlation is a fair approach while avoiding some edge cases. Correlation looks whether $X's$ and $Y's$ are linearly correlated or how one variable is influenced by other. But I guess what you might be looking for is just something simpler like average increase in rating after the activity. For example

import numpy as np
from scipy import stats
ratings_before = np.array([1,2,1,3,4,5])
ratings_after = np.array([5,5,5,5,5,5])
print "P.Corr: ",stats.pearsonr(ratings_before, ratings_after)[0]
print "Avg.Increase: ",np.mean(ratings_after-ratings_before)

which gives

Nan
2.333

And assume that your activity went so good that everyone rated you a 5-star. Then correlation turns as $NaN$ as denominator goes to zero as no $Y$ can be explained by $X$.

Hope this helps.


Policy Gradient Methods - ScoreFunction & Log(policy) In Policy Gradient Methods, Lecture 7 (34:15), David describes a Score Function as being the Gradient of the Log of the policy

Question:

If we have a Neural Network that holds parameters defining our policy,
we will have to perform backpropagation to each weight (each parameter). Why would we want to avoid dealing with:

$\nabla_{\theta}\pi_{\theta}(s,a)$

and instead would wish to deal with its re-arranged version:

$\pi_{\theta}(s,a)\nabla_{\theta} log \big( \pi_{\theta}(s,a) \big) $

Don't we still have a $\pi_{\theta}$ that sits inside the log? By the Chain Rule, we will still have to compute $\frac{\partial \pi_{\theta}}{\partial \theta}$ once we arrive to it. Aren't we just postponing inevitable?

My Understanding:

I know that in Neural Networks that perform classification, using a Cross-Entropy function (that deals with logs) right after a Softmax Activation would allow for a really neat derivative during backprop: (result - target) Perhaps that's the intent here as well?

Thus, since we are dealing with the Log(policy), am I implied to always use a Softmax function for the last layer of my Neural Net, when working with Policy Gradients? - at 39:37 in the video, David also describes Gaussian Policy. Are these two selected because they work really well with Log, during backprop?

As I understand, in general, the benefits of wrapping a product in a $log()$  is being able to split it into 2 summands:  $log(ab) = log(a) + log(b)$ but I can't see where it would be useful in Policy Gradients.

My intuition is this might be useful to prevent vanishing gradients.

Maybe it's related to the fact that it's called "Score Function", therefore we are dealing with a Log for some other reason?  In supervised learning we have a Squared Error cost function that provides us a "beginning value", during the backpropagation.  Is Score function = cost function?

You have to see this approach more general and not so much from a neural network point of view. The reason one uses the re-arranged version is really just because in the end the gradient of your objective function ends up being an expectation again. This is also pointed out by David Silver at min 44:06.

For an example of the benefits of using $\log$, see UC Berkeley's DeepRL lecture by S. Levine here. The name of the video is "CS294-112 9/6/17"

At some later point, S. Levine also goes into more details regarding the math of the policy gradient and draws comparisons to the maximum likelihood approach with neural networks.


How to initialize word-embeddings for Out of Vocabulary Word? I am trying to use CoNLL-2003 NER (English) Dataset and I am trying to utilize pretrained embeddings for it. I am using SENNA pretrained embeddings. Now I have around 20k words in my vocabulary and out of this I have embedding available for only 9.5k words.

My current approach is to initialize an array of 20k X embedding_size with zeros and initialize the 9.5k words whose embeddings is known to me and make all the embeddings learn-able.

My question is what is the best way to do this? Any reference to such research will be very helpful ?

I have also asked a similar question later on Stackoverflow, here is the link.

You have a few options here. Of these, I think 1 will be the easiest to implement, as it's a standard language model with an alignment term added to the loss. I'd recommend 2a if you think you have the time, as I imagine its performance might be much better.

1 Use existing corpus to learn embeddings for new words

You can do this as you describe, though I would initialize the unknown embeddings with random noise instead of zeros. This will probably do fine, but it suffers from the problem that the model has no way of knowing that the SENNA embeddings are to be trusted more than the randomly initialized ones. So, it will generally take more (time, examples, etc.) to train this one well.

Another option is to try to capture the difference between known embeddings and randomly initialized ones. My suggestion would be to create your own embeddings [20k x emb_dim] and initialize it however you want. Then add a penalty to your model's loss for deviation from the SENNA embeddings. This is what's done in bilingual embedding models like this one. This will push the known embeddings close to SENNA and allow the unknown ones to freely vary. You could also reduce the coefficient for this alignment penalty as training progresses.

You could either learn these embeddings on a dedicated embedding task (like CBOW) or as part of your primary task.

2 Hybrid model for unknown embeddings

A second approach is to forgo attempting to learn embeddings for your unknown words at all. Typically, mosty word RNN models do this for low-frequency words through something like the <unk> tag. For you, that would be half your vocabulary. Fortunately, there are a lot of tools for filling in these out-of-vocabulary (OOV) words.

2a Subword embeddings

As @Emre pointed out, subword embeddings have been used to great effect to solve out-of-vocabulary problems. Here's an example of a model that uses subword embeddings to achieve an "open vocabulary" model. This means you would combine your pre-trained embeddings (which you would not need to make learnable) with a character-level model to learn when and how to produce OOV words.

2b <unk> replacement

Another approach is to, as a post-processing step, replace the <unk> symbols with words from your corpus. You can build your own language model (even a simple markov chain would be an improvement). Also, some modern models have more sophisticated methods for replacement that work quite well.


Derivative of a custom loss function with the logistic function I have costum loss function with $\mu ,p, o, u, v$ as variables and $\sigma$ is the logistic function.
I need to derive this loss function.
Due to multiple variables in the loss function, I need to use the softmax function which is the generalization of the logistic function?
$L = -\frac{1}{N}\sum_{i,j \in S}^{2}{a_j\{y_{i,j}log[\sigma{(\mu + p_i + o_j + u^{T}_{i}v_{j})]} + (1 - y_{i,j})log[1 - \sigma{(\mu + p_i + o_j + u^{T}_{i}v_{j})]}\}}$
As far I understand, it is a multivariate gradient descent problem so I need to provide 4 different update rules (doing partial derivative for each of the variables as the others are constants) but how I "merge" them into one update rule or maybe there is only one update rule from the beginning, I got pretty confused when it's become multivariable problem.

Each parameter will use the partial derivative of Loss w.r.t the parameter i.e.
$\mu = \mu - {\partial L}/{\partial \mu}$
$p = p - {\partial L}/{\partial p}$
$o = o - {\partial L}/{\partial o}$
$u = u - {\partial L}/{\partial u}$
$v = v - {\partial L}/{\partial v}$
If the direct derivative is not possible, then may need the chain-rule.


How to choose the good number dimension of autoencoder? I'm using Autoencoder for feature extracting. I stuck with how to choose good number of dimension of encoder layer (latent layer). After training dataset, the model gave the latent layer (embedding layer) with some zero value in the vector result.
For example, the embedding layer have 4 dimensions, one of node (unit) in embedding layer has value [0.67 0.0 2.13 0.43]. That I suppose they should 4 values different zero value.
I think my problem that I choose too many dimension for embedding layer which is not necessary and should smaller dimension, such as 3 or 2 dimensions.
So, my question how to choose good size for embedding layer?

Treat it as a hyperparameter, and do a hyperparameter search over it. This requires having a good metric for your model performance. Which usually means having a validation set with labels. Training can still be unsupervised.


Text post-processing I have set of newspaper articles and I use TextRank algorithms to identify their keywords to perform a classification.

Apart from the important informative keywords, I am also receiving garbage keywords as follows.

"viewed", "flutter", "function", "k", "neighbour"

Is there a way to post process and remove these keywords?

Yes, you can do that by add them to the existing NLTK Stop-word dictionary for all such words/Creating a Custom Stop-word dictionary.

for Custom Stop-word dictionary you need to include all the key words in the dictionary before processing, make sure to check if the words are present in the stop-word dictionary. If yes, remove them from the text else nothing needs to be done.

set of code is for updating new words to the NLTK Stop-words dictionary:

#Custom function to remove the stop-words
    def removeStopWords(str):
    #select english stopwords from NLTK
    cachedStopWords = set(stopwords.words("english"))
    #add custom words, mentioned above
    cachedStopWords.update(("viewed", "flutter", "function", "k", "neighbour"))
    #remove stop words
    new_str = ' '.join([word for word in str.split() if word not in cachedStopWords])
    return new_str
set of code is for making a Custom stop-word dictionary:

stopwordList = ["viewed", "flutter", "function", "k", "neighbour"]
StopWordsRemover(inputCol="words", outputCol="filtered" ,stopWords=stopwordList)


Appropriate objective function and evaluation metric when I DO care about outliers? I am reading these two pages:
xgboost documentation
Post on evaluation metrics
I have a dataset where I am trying to predict future spend at the user level. A lot of our spend comes from large spenders, outliers. So, we care about them. I am using XGBoost.
I have tried xgboost with objective reg:squarederror. This tended to underpredict a little. I then tried with reg:squaredlogerror and this resulted in predictions that under predict by much more than just using squarederror.
I have tried tuning with several differing hyper parameter combinations but none made as big a difference as changing the objective. So, I'm dwelling on the objective function and trying to understand if there's another one out there that would be worth a shot?
On the xgboost docs above, some of the out of the other regression objective options are reg:pseudohubererror as well as count:poisson.
There is no option, that I can see, for just MAE. If using an objective function less susceptible to outliers with rmsle took me further away from accuracy whereas rmse took me closer, would using MAE potentially be worth a shot? In this dataset, outliers are more important, but so are regular users.
What would be a good objective and evaluation metric?
Is MAE worth trying? If so, how? Looking at the docs above, I cannot see MAE as an option under regression parameters.

These are several things you can try:

Use quartic error, $(y - \hat{y})^4$, instead of quadratic error. This is going to penalize a lot big errors, way more than MSE. The issue is that this is not implemented in xgboost, and you would need to develop a custom loss.
If your target is always positive, you can use the target as training weights. This will give more weights to the outliers. If it is not always positive, you can use the absolute value of the target as weights. If using the target values directly puts too much weight on the outliers, you might want to transform it (e.g. using the log or square root), and if you have samples whose target value is zero, you might want to add some epsilon to all the weights. Note that xgboost can be easily trained using weights.
Try to predict the quantile of the training distribution, then transform your predictions using the training cumulative probability function.


One way of doing it is by creating an auxiliary function to extract the category from your tag and return it when found or 'unknown' otherwise. Then using .apply() with that function will do the trick:

json_string = '''{"submissionTime":"2019-02-25T09:26:00","b_data":{"bName":"Masato","b_Acc":[{"id":0,"transactions":[{"date":"2019-12-19","text":"PERIODICAL PAYMENT","amount":3397,"type":"","tags":[{"institution":"University of MC"},{"lenderType":"private"},{"category":"birdy"},{"creditDebit":"credit"}]},{"date":"2019-12-03","text":"LINE FEE","amount":-460.21,"type":"Overdrawn Fees","tags":[{"category":"Overdrawn"},{"creditDebit":"debit"}]},{"date":"2019-12-31","text":"INTEREST","amount":-871.62,"type":"Interest Charge","tags":[{"category":"Fees"},{"creditDebit":"debit"}]},{"date":"2019-12-31","text":"LOAN SERVICE FEE","amount":-120,"type":"Loan Related Fees","tags":[{"category":"Fees"},{"creditDebit":"debit"}]},{"date":"2019-12-18","text":"PERIODICAL PAYMENT","amount":3397,"type":"","tags":[{"institution":"University of MC"},{"lenderType":"private"},{"category":"birdy"},{"creditDebit":"credit"}]},{"date":"2019-12-02","text":"LINE FEE","amount":-498.34,"type":"Overdrawn Fees","tags":[{"category":"Overdrawn"},{"creditDebit":"debit"}]},{"date":"2019-11-29","text":"INTEREST","amount":-794.4,"type":"Interest Charge","tags":[{"category":"Fees"},{"creditDebit":"debit"}]},{"date":"2019-11-19","text":"PERIODICAL PAYMENT","amount":3397,"type":"","tags":[{"institution":"University of MC"},{"lenderType":"private"},{"category":"birdy"},{"creditDebit":"credit"}]},{"date":"2019-11-01","text":"LINE FEE","amount":-484.87,"type":"Overdrawn Fees","tags":[{"category":"Overdrawn"},{"creditDebit":"debit"}]},{"date":"2019-10-31","text":"INTEREST","amount":-882.04,"type":"Interest Charge","tags":[{"category":"Fees"},{"creditDebit":"debit"}]},{"date":"2019-10-21","text":"PERIODICAL PAYMENT","amount":3397,"type":"","tags":[{"institution":"University of MC"},{"lenderType":"private"},{"category":"birdy"},{"creditDebit":"credit"}]},{"date":"2019-10-01","text":"LINE FEE","amount":-503.59,"type":"Overdrawn Fees","tags":[{"category":"Overdrawn"},{"creditDebit":"debit"}]},{"date":"2019-09-30","text":"INTEREST","amount":-916.98,"type":"Interest Charge","tags":[{"category":"Fees"},{"creditDebit":"debit"}]},{"date":"2019-09-30","text":"LOAN SERVICE FEE","amount":-120,"type":"Loan Related Fees","tags":[{"category":"Fees"},{"creditDebit":"debit"}]},{"date":"2019-09-19","text":"PERIODICAL PAYMENT","amount":3397,"type":"","tags":[{"institution":"University of MC"},{"lenderType":"private"},{"category":"birdy"},{"creditDebit":"credit"}]},{"date":"2019-09-02","text":"LINE FEE","amount":-489.65,"type":"Overdrawn Fees","tags":[{"category":"Overdrawn"},{"creditDebit":"debit"}]},{"date":"2019-08-30","text":"INTEREST","amount":-892.13,"type":"Interest Charge","tags":[{"category":"Fees"},{"creditDebit":"debit"}]}]}]}}'''
js = json.loads(json_string)
df = pd.DataFrame(js['b_data']['b_Acc'][0]['transactions'])

def extract_category(tag):
    dall = {}
    # we create a new unique dict with all the items in the tag
    for d in tag:
        dall.update(d)
    # if category is in our new dict, return it else return unknown
    if 'category' in dall.keys():
        return dall['category']
    else:
        return 'unknown'

df['category'] = df.tags.apply(lambda x: extract_category(x))


intuitive understanding of (1+parameter)/parameter When looking at a function that contains term (1+parameter)/parameter, should there be an intuitive understanding for why this term is included. For example if a function for fishing mortality includes this term and "parameter" is a a growth parameter for fish, is there something I can say about the function with regard to fish growth? Why is the value 1 added to parameter in the numerator?

I'm not sure, without knowing more, but this is of course equal to $1 + \frac{1}{parameter}$. Out of curiosity I searched for something about fish mortality and growth params and found http://www.fao.org/tempref/docrep/fao/008/a0212e/a0212E08.pdf which sets this up as a log-log regression in terms like a growth param $L_\infty$, which makes sense if there's some power-law distribution.

Maybe the parameter being used is its inverse, and it may be that this is fudged with the +1 just to make sure the value is greater than 1 so that the log is nonnegative. Maybe there's a better domain-specific reason for tweaking it.


How do I identify clusters that match on categorical data? I am seeking some directions for a proper path to research the solve for this problem:

My company made all our employees take a "StrengthFinders" test, which results in every employee being assigned their top five (ordered) "strengths" from a possible list of 34 strengths.  We have 500 employees.  I am supposed to identify all the employees that match each other for the same 5 strengths (order not important), and also for employees that match each other for 4 out of 5 strengths (again, order doesn't matter).  I could potentially have multiple groups matching on different sets of strengths, e.g.:
Group 1: Billy, Sally, Michael have strengths A, H, I, K, Z
Group 2: Bobby and Suzy have strengths A, B, L, S, W

For the case where strengths match for 4 out of 5, I might have the same people from Group 1 above, plus Joe, whose strengths are A, H, M, K, Z; and
Seth, whose strengths are A, H, G, K, Z.  I would expect more groupings for the case of 4 out of 5 than the 5 out of 5 case.

The strengths are categorical in nature, so what I've read so far has largely revolved around clustering of continuous numerical variables.

I am looking for an algorithmic way to identify clusters and the members of those clusters for this situation.  I think I could do this brute force by repeatedly sorting data in Excel, but I'm confident that a better way must exist, and I ask you to point me in that direction.  Thank you.

You have just 500 data points...

Excel of course is the worst possible tool though.

Anyway, build a dictionary. Put everybody in there 6 times: 1 with all five strengths, and 5 times with one strength omitted. Then you can easily identify the largest groups, and you can also perform various completion operations easily: if you have identified a group with strengths A B C D E, you can add all that have ABCD etc. using the dictionary.


How can I create a custom tag in JPMML? I'm trying to create a logistic regression model in jpmml, then write the PMML to a file. The problem I'm having, is that I can't find any way to create a custom tag, such as "shortForm" and "longForm" in the following example:

<MapValues outputColumn="longForm">
  <FieldColumnPair field="gender" column="shortForm"/>
  <InlineTable>
    <row><shortForm>m</shortForm><longForm>male</longForm>
    </row>
    <row><shortForm>f</shortForm><longForm>female</longForm>
    </row>
  </InlineTable>
</MapValues>

Here's what I have so far:

MapValues mv = new MapValues("output")
  .withFieldColumnPairs(
        new FieldColumnPair( new FieldName("gender"), "shortForm" )
  ).withInlineTable(
        new InlineTable().withRows(
                new Row().with???( new ??? )
)))

In short, I am asking for an API call I can use to instantiate the "shortForm" element in the example, and attach it to the "row" object. I've been all through the API, examples, and Google/SO, and can't find a thing.

Thanks for your help!

You can/should use a generic Java Architecture for XML Binding (JAXB) approach.

Simply put, call Row#withContent(Object...) with instances of org.w3c.dom.Element that represent the desired XML content.

For example:

Document document = documentBuilder.newDocument();
Element shortForm = document.createElement("shortForm");
shortForm.setTextContent("m");
Element longForm = document.createElement("longForm");
longForm.setTextContent("male");
row = row.withContent(shortForm, longForm);


Tool to label images for classification Can anyone recommend a tool to quickly label several hundred images as an input for classification?
I have ~500 microscopy images of cells. I would like to assign categories such as 'healthy', 'dead', 'sick' manually for a training set and save those to a csv file.
Basically, the same as described in this question, except I do not have proprietary images, so maybe that opens up additional possibilities?

I just hacked together a very basic helper in python
it requires that all images are stored in a pyton list allImages.

import matplotlib.pyplot as plt
category=[]
plt.ion()

for i,image in enumerate(allImages):
    plt.imshow(image)
    plt.pause(0.05)
    category.append(raw_input('category: '))


The impact of using different scaling strategy with Clustering I'm currently learning about clustering. To practice clustering, I am using this dataset.
After running K-means clustering for multiple values of k and plotting the results, I can see that scaling is affecting the results (within-cluster SSE) and I want to use this post to confirm my intuition as to why this is the case.
I don't believe that this is a meaningful reduction in the Within-Cluster SSE because the numerical distances are sensitive to scale, and I don't think that this has any effect on how accurate the model is. Is that intuition correct?
I just wasn't expecting the reduction to be this drastic between standardizing and normalizing.
Code and the results:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('customers.csv')
X = df.iloc[:, [3, 4]].to_numpy()

from sklearn.preprocessing import StandardScaler, MinMaxScaler
ssc, mmsc = StandardScaler(), MinMaxScaler()
X_ssc = ssc.fit_transform(X)
X_mmsc = mmsc.fit_transform(X)

from sklearn.cluster import KMeans
# Unscaled
k_vals = list(range(2, 21))
WCSSE = []
for k in k_vals:
    kmeans = KMeans(n_clusters=k)
    model = kmeans.fit(X)
    WCSSE.append(model.inertia_)
plt.plot(WCSSE, marker='o', markersize=10)

# Standard Scaler
k_vals = list(range(2, 21))
WCSSE = []
for k in k_vals:
    kmeans = KMeans(n_clusters=k)
    model = kmeans.fit(X_ssc)
    WCSSE.append(model.inertia_)
plt.plot(WCSSE, marker='o', markersize=10)

# MinMax scaler
k_vals = list(range(2, 21))
WCSSE = []
for k in k_vals:
    kmeans = KMeans(n_clusters=k)
    model = kmeans.fit(X_mmsc)
    WCSSE.append(model.inertia_)
plt.plot(WCSSE, marker='o', markersize=10)

Welcome to the community!
Some points which might help:

Clustering, as an unsupervised task, can not be evaluated and usually some external criteria are used to find the best clustering.
According to the point above, better to make those assumptions as direct as possible. Starting with EDA (inspecting histograms, plotting boxplots, etc.) gives you a better initial point for analysis. Trying to understand the underlying structure of data directly from decrease of WSCC seems pretty indirect to me thus more difficult. For example a simple EDA tells you if data needs scaling urgently or not. Specially iin your data, ranges are not "dramatically" different, but if you see histogram, you may see, for instance, a exponential feature which needs a log-transform instead of scaling.
The last but not the least, the larger the number of clusters get, the smaller the WCSS become. You see it in plots. So the question is if these three scaling show a significant difference in dropping point of WCSS, which they don't and boosts this idea that scaling is not the biggest help here. Otherwise your code is pretty right (and well-written ;) )

I hope it could help to some extent.


When does cache get expired for a RDD in pyspark? We use .cache() on RDD for persistent caching of an dataset, My concern is when this cached will be expired?.

dt = sc.parallelize([2, 3, 4, 5, 6])
dt.cache()

It will not expire until Spark is out of memory, at which point it will remove RDDs from cache which are used least often. When you ask for something that has been uncached it will recalculate the pipeline and put it in cache again. If this would be too expensive, unpersist other RDDs, don't cache them in the first place or persist them on your file system.


eta and learning_rate different in xgboost I am creating a classification model using xgboost in python. I am using different eta values to check its effect on the model. My code is-

for eta in np.arange(0.2, 0.51, 0.03):
    xgb_model = xgboost.XGBClassifier(objective = 'multi:softmax', num_class = 5, eta = eta)
    xgb_model.fit(x_train, y_train)
    xgb_out = xgb_model.predict(x_test)
    print("For eta %f, accuracy is %2.3f" %(eta,metrics.accuracy_score(y_test, xgb_out)*100))

I expected different accuracies for some eta values, but to my surprise I got same accuracy for each eta. When I printed the model, I got this-

>>> print(xgb_model)
XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
       colsample_bytree=1, eta=0.5, gamma=0, learning_rate=0.1,
       max_delta_step=0, max_depth=3, min_child_weight=1, missing=None,
       n_estimators=100, n_jobs=1, nthread=None, num_class=5,
       objective='multi:softprob', random_state=0, reg_alpha=0,
       reg_lambda=1, scale_pos_weight=1, seed=None, silent=True,
       subsample=1)

Here you can see eta = 0.5, but learning_rate = 0.1. While in xgboost docs, learning_rate is an alias for eta. So how is this possible that both have different values?

Seems like eta is just a placeholder and not yet implemented, while the default value is still learning_rate, based on the source code. Good catch.
We can see from source code in sklearn.py that there seems to exist a class called 'XGBModel' that inherits properties of BaseModel from sklearn's API.
Tracing this to compat.py, we see there's an import statement:
from sklearn.base import RegressorMixin, ClassifierMixin

These are sklearn's Mixin classes for classifiers/regressors. Basically, they're the shells from which any specific classifier/regressor is generated, to guarantee that the descendant class has a score() method.
Looking carefully at the init function, we see:
__init__($self, /, *args, **kwargs)

From Yasoob's PythonTips (See link for examples of *args/**kwargs):

*args and **kwargs allow you to pass a variable number of arguments to a function. What variable means here is that you do not know beforehand how many arguments can be passed to your function by the user so in this case you use these two keywords...
*args is used to send a non-keyworded variable length argument list to the function.
**kwargs allows you to pass keyworded variable length of arguments to a function. You should use **kwargs if you want to handle named arguments in a function.

Now for a fun experiment to prove this. Try the following in your own notebook/console:
>>> XGBClassifier(eta=5, potato=1234)
XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
   colsample_bytree=1, eta=5, gamma=0, learning_rate=0.1,
   max_delta_step=0, max_depth=3, min_child_weight=1, missing=None,
   n_estimators=100, n_jobs=1, nthread=None,
   objective='binary:logistic', potato=1234, random_state=0,
   reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=None,
   silent=True, subsample=1)

See that our argument was stored in the object:
`...objective='binary:logistic', potato=1234, random_state=0,`

We're pretty sure potato is not a hyperparameter right? :)


Face embedding of unseen images I've read about FaceNet but the main problem is still unclear for me. Does embedding work on the trained images only? Or once trained on a big dataset it will readily cluster unknown faces without re-training? I need an approach to add new faces automatically.

Yes embedding works on unseen images. That is the point of the approach - instead of learning to classify each face in the training set directly, the model learns a vector that is intended to be:

far apart when the face itself is different
similar when the face is the same, but other items in the image - e.g. background, hair style etc - are different

In order to use embeddings to recognise someone, you need one or more sample images of that person, so that you can generate a target embedding. Then you can predict any new image that generates a new embedding close enough to the target embedding(s) to be an image of the same person.

This can still suffer usual problems of overfitting to training set. So it is important to understand what training data was used for the model. Embeddings generated from images that are very different from the training data will not be as reliable as embeddings of images that are similar to the training data.


Which algorithm to use to match two categories with n dimensions Suppose I have examples of two categories, Products, and Aisles.

Each product has variables like price, weight, moisture rating etc and then Aisles has variables like max weight it can handle, moisture near it, etc.

Now I want to run some algorithm which classifies which products should go in which aisles based on product and aisles variables.

examples if product A has weight 10kg, moisture it can handle x, it should go to Aisle which can handle that weight and has that moisture rating.

Provided I have same variables across product and aisles, how can I solve my problem?

There is no generalized algorithm to match two categories of data where the dimension is different and perhaps the attribute could also be different. However you can transform the data into similar dimension. One method I would suggest you is to take PCA of both the data.  After taking PCA you can pick the first k principal component of both the data making them virtually identical data with similar attributes. Then you can use k-nearest neighbour on the transformed data of Aisles to train your model. The model could be easily used to test the transformed data of products.


Do logistic regression and softmax regression do the same thing? If the both do the same thing then which give us better accuracy?

There is a key difference:

Softmax regression provides class probabilities for mutually exclusive classes.
Logistic regression treats class membership for each class separately. Classes do not need to be mutually exclusive.

The two are equivalent for a scenario with two mutually exclusive classes - e.g. a "positive" and "negative" class - where softmax would have two outputs summing to 1, and logistic regression would have one output giving probability of the "positive" class.

I am not certain of the results comparing logistic regression one-vs-all (taking max output) with softmax regression on the same multi-class problem. I would expect the performance to be quite similar. Neither model copes well with non-linear relationships between input and target classes.


Is using samples from the same person in both trainset and testset considers being a data leakage? Suppose a neural network is built for a binary classification problem such as recognize the face as a smiley face or not, by using a dataset of 1000 persons and each person has ten images of his face.
If the dataset randomly spilt into trainset and testset by a ratio of 70:30, in this case, there is a big chance face image of same persons will be used in both the trainset and testset, so is this considered to be data leakage (train-test contamination)?

Yes, this is a form of data leakage. The testing data should not be linked to the training data in any way.
Another way to think of it is, if someone were to try replicating your results with their own test set, would your test set have given you an advantage such that your results are generally better than theirs?


Try this:
If needed, reset the index (this should not be a date time)
datos_frecuencia.groupby(datos_frecuencia.index // 3).agg({"tiempo":lambda x: x.head(3).min(),"state":sum})

Output:


How to set input for proper fit with lstm? My input training and test dataset is the following size:

print(trainX.shape):(53394, 3)
print(testX.shape):(17799, 3)
print(trainY.shape):(53394,)
print(testY.shape):(17799,)

I reshaped it as follows:

trainX.shape:(1, 53394, 3)
testX.shape: (1, 17799, 3)
trainY.shape: (1, 53394)
testY.shape: (1, 17799)

Now, I pass it as the input layer of a LSTM:

model = Sequential()
model.add(LSTM(66, input_shape=(len(trainX),3)))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')
model.fit(trainX, trainY, epochs=100,batch_size=1, verbose=2)

I am getting the error:

Error Message:
ValueError: Error when checking input: expected lstm_6_input to have shape (1, 3) but got array with shape (53394, 3)

Please help me to fit my data properly into a LSTM.

You should be structuring your data as a tuple: Number of samples, timesteps, features. In your case number of samples is 53394, timesteps is 1 and number of features is 3. So the input shape will be (53394, 1, 3).
You can use this snippet of code for the tranformation:

trainX = trainX.reshape((trainX.shape[0], 1, 3))

And set the argument input_shape  as below:

input_shape = (trainX.shape[1], trainX.shape[2])

Hope this helps!


Scaling does not speed up the SVM model I tried to standardize the training data with samples of 629,145 rows and 24 features:

from sklearn import datasets
import pandas as pd
from sklearn import svm
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler
from sklearn.preprocessing import StandardScaler

df = pd.read_csv('mydata.csv', dtype='object')

#manually choosing 24 features
X=df.loc[:, ['Bwd Pkt Len Min','Subflow Fwd Byts','TotLen Fwd Pkts','TotLen Fwd Pkts','Bwd Pkt Len Std','Flow IAT Min',
             'Fwd IAT Min','Flow IAT Mean','Flow Duration','Flow IAT Std','Active Min','Active Mean','Fwd IAT Min',
             'Bwd IAT Mean','Fwd IAT Mean','Init Fwd Win Byts','ACK Flag Cnt','Fwd PSH Flags','SYN Flag Cnt','Fwd Pkts/s',
             'Bwd Pkts/s','Init Bwd Win Byts','PSH Flag Cnt','Pkt Size Avg']]
Y= df['Label']

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.4,random_state=42) # 60% training and 40% test
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

#Create a svm Classifier
clf = svm.SVC(kernel='rbf') # not linear Kernel
clf.fit(X_train, y_train)

Already 6 hours and SVM does not converge. Same data is converged with other algorithms very fast as RF and I know it is quite normal as SVM considered computationally high algorithm compared to KNN and RF.. I read quite a bit of questions/answers and articles.

I wonder how can I track and analyze the problem visually (probably plotting some graphs as cost function or ?).
Can parameter tuning (C parameter) be of the help and speed up this?
What could be your advice?

Thanks very much

I do not program on python. Nevertheless, I would say the key relies in the number of samples (629,145). They are many. The SVM has to test them to pick which are good support vectors for data partition/regression and given the size of the dataset there are a lot of alternatives. That issue plus the number of different C, gamma/sigma and, perhaps, epsilon (not sure you are classifying or regressing) tested during the optimisation of the SVM hinders convergence.
There are people who used clusters instead of the original dataset to train the SVM. For instance:
Barros de Almeida, M., de Padua Braga, A., Braga, J.P., 2000. SVM-KM: speeding SVMs learning with a priori cluster selection and k-means, in: Proceedings. Vol.1. Sixth Brazilian Symposium on Neural Networks. Rio de Janeiro, (Brazil), pp. 162–167.
can be a good starting reference but, I am aware there a others. There is at least one R package based on that idea (LinearizedSVR).
In addition, there are analytical methods to infer the values of the parameters C, gamma/sigma and epsilon (for regression) based on the characteristics of the training dataset. That means no optimisation is necessary, although there is a friend that still tunes gamma/sigma but fixes C fallowing these approaches. I think there is available code (at least for R) somewhere in the net. The refs are:
Cherkassky, V., Ma, Y., 2004. Practical selection of SVM parameters and noise estimation for SVM regression. Neural Networks 17 (1), 113–126. 10.1016/S0893-6080(03)00169-2
for regression and,
Keerthi, S.S., Lin, C.-J., 2003. Asymptotic Behaviors of Support Vector Machines with Gaussian Kernel. Neural Comput. 15 (7), 1667–1689. 10.1162/089976603321891855
for classification SVMs.


Context Based Embeddings vs character based embeddings vs word based embeddings I am working on a problem that uses English alphabets in the text but the language is not English. Its a mixture of English and different language text. But all words are written using English alphabets. Now, word-based pre-trained embedding models will not work here as it gives a random embedding to out of vocabulary words.
Now my question is that how the Context-based pre-trained embeddings deal with "out of vocabulary" words?
Besides, what's the difference between context-based embeddings and character-based embeddings?

Context-based or contextual means that the vector contains information about the use of the word in a context of a sentence (or rarely a document). It thus does not make sense to talk about the word embeddings outside of the context of the sentence.
Models such as BERT use input segmentation into so-called subwords which is basically a statistical heuristic. Frequent words are kept intact, whereas infrequent words get segmented into smaller units (which often resemble stemming or morphological analysis, and often seem pretty random), ultimately keeping some parts of the input segmented into characters (this is typically the case of rare proper names). As a result, you get contextual vectors of subwords rather than words.
Character-based embeddings usually mean word-level embeddings inferred by from character input. For instance, ELMo used character-level inputs to get word embeddings that were further contextualized using a bi-directional LSTM. ELMo embeddings are thus both character-based and contextual.
Both when using sub-words and when using embeddings derived from characters, there are technically no OOV words. With subwords, the input breaks to characters (and all characters are always in the vocabulary). With character-level methods, you always get a vector from the characters. There is of course no guarantee that the characters are processed reasonably, but in most cases they are.
Models that use static word embeddings (such as Universal Sentence Encoder) typically reserve a special token for all unknown words (typically <unk>), so the model is not surprised by a random vector at the inference time. If you limit the vocabulary size in advance, the OOV tokens will naturally occur in the training data.


Any case studies using Bayesian Networks for system design trades? I am exploring using Bayesian Networks to identify the best parameters within a system design, to improve its performance. I'm trying to find any case studies where this has been used successfully or unsuccessfully.

As an example consider an automatic assisted anti-collision system on a car.  It could use some combination of ultrasonic, radar & IR seekers to sense and measure the closing speed with the car in front.  These all have different costs / reliabilities and perform differently in different weather conditions.  I am interested in understanding how effective Bayesian Networks can be in evaluating the trade space and identifying the best solutions to these types of problems.

As I can see you're searching for a case study. Learning from data is useful because you can learn both the conditional probability distributions (CPDs) and the structure of the network. The best resource I've read is a freely-available Microsoft tech report.

Have a look at the paper, maybe it can help you upto some extend. I'll suggest you to start with Chapter 7 of it.

Hope it helps. Cheers!


Multiple solutions with same minima in MLP with same weights I came across an excercise on deep learning from here.
It goes as follows:
Consider a simple MLP with a single hidden layer of $d$ dimensions in the hidden layer and a single output. Show that for any local minimum there are at least $d!$ equivalent solutions that behave identically.
As the network is a MLP with one hidden layer, the equation would be:
$O = W^{(2)}(W^{(1)}x + b_1) + b_2$
Assuming I am correct, where do I need to go from here to get to the solution?

If one permutes the connections of the hidden layer ($d!$ ways to do that), and move and rename connections appropriately, then one effectively has the same MLP with the exact same minima, yet the configuration has changed (in a trivial sense). Thus there are (at least) $d!$ configurations only trivialy different with the exact same minima.
To see it in your notation, effectively the hidden layer output is the following sum:
$$O_{\text{hidden}} = \sum_{i=1}^d w_{\pi_i} \cdot x + b_{\pi_i}$$
Where $\pi_i$ is some order of the connections. For example $\pi_i = i$
But for $d$ items there are $d!$ permutations thus $d!$ order functions $\pi(i) = \pi_i$. Yet the difference is only in re-ordering the configuration and is trivial. The rest follow from that.
This is one reason why Neural Networks are non-convex models.
See also: Explanation of why Neural Networks are non convex


Training an ensemble of small neural networks efficiently in TensorFlow 2 I have a bunch of small neural networks (say, 5 to 50 feed-forward neural networks with only two hidden layers with 10-100 neurons each), which differ only in the weight initialization. I want to train them all on the same, smallish dataset (say, 10K rows), with a batch size of 1. The aim of this is to combine them into an ensemble by averaging the results.

Now, of course I can build the whole ensemble as one neural network in TensorFlow/Keras, like this:

def bagging_ensemble(inputs: int, width: int, weak_learners: int):
    r'''Return a generic dense network model

    inputs: number of columns (features) in the input data set
    width: number of neurons in the hidden layer of each weak learner
    weak_learners: number of weak learners in the ensemble
    '''
    assert width >= 1, 'width is required to be at least 1'
    assert weak_learners >= 1, 'weak_learners is required to be at least 1'

    activation = tf.keras.activations.tanh
    kernel_initializer = tf.initializers.GlorotUniform()

    input_layer = tf.keras.Input(shape=(inputs,))
    layers = input_layer
    hidden = tf.keras.layers.Dense(units=width, activation=activation, kernel_initializer=kernel_initializer)\
                (input_layer)
    hidden = []
    # add hidden layer as a list of weak learners
    for i in range(weak_learners):
        weak_learner = tf.keras.layers.Dense(units=width, activation=activation, kernel_initializer=kernel_initializer)\
                        (input_layer)
        weak_learner = tf.keras.layers.Dense(units=1, activation=tf.keras.activations.sigmoid)(weak_learner)
        hidden.append(weak_learner)

    output_layer = tf.keras.layers.Average()(hidden)  # add an averaging layer at the end

    return tf.keras.Model(input_layer, output_layer)

example_model = bagging_ensemble(inputs=30, width=10, weak_learners=5)
tf.keras.utils.plot_model(example_model)

The resulting model's plot looks like this:

However, training the model is slow, and because of the batch size of 1, a GPU doesn't really speed up the process. How can I make better use of the GPU when training such a model in TensorFlow 2, without using a larger batch size?

[The motivation for using this kind of ensemble is the following: Especially with small datasets, each of the small networks will yield different results because of different random initializations. By bagging as demonstrated here, the quality of the resulting model is greatly enhanced. If you're interested in the thorough neural network modelling approach this technique comes from, look for the works of H. G. Zimmermann.]

I have a problem with this approach, though I'm up for discussion about it and might be wrong.

Idea: You should train every sub-model on the main task, not the ensemble. For that, you can utilise your GPU to actually calculate everything in parallel (Even with a 5-dimensional input, effectively encoding 5 batches at once, further enhancing the independent training of the sub-models). To do this, my quick bet would be to just have five independent outputs, all optimised with a weight of 1/5th on the same loss you use for the averaging function. Then, do the averaging afterwards, without further optimisation.

Motivation: The fact that you have the averaging layer inside of the model as the output layer means, that the loss differentiation takes place there as well. This means, that during training, you allow interdependancies inbetween the single sub-networks. After short consideration it should be clear that this is not really what your motivation suggests:

  [The motivation for using this kind of ensemble is the following: Especially with small datasets, each of the small networks will yield different results because of different random initializations.

The small networks don't yield anything, because you don't measure and optimise for their yield. In short, you just built a bigger model with a slightly more complicated connectivity.


Why not train the final model on the entire data after doing hyper-paramaeter tuning basis test data and model selection basis validation data? By entire data I mean train + test + validation

Once I have fixed my hyperparameter using the validation data, and choose the model using the test data, won't it be better to have a model trained on the entire data so that the parameters are better trained rather than having the model trained on just train data

The question is under a wrong assumption. Many people do what you say they "cannot" do.

In fact, the grid search implementation in the widely deployed sklearn package does just that. Unless refit=False, it will retrain the final model using the entire data.

I think for some hyperparameters this might not be very desirable, because they are relative to the volume of data. For instance, consider the min_samples_leaf pre-pruning tactic for a decision tree. If you have more data, the pre-pruning may not perform as you want.

But again, most people do in fact retrain using the entire data after cross-validation, so that they end up with the best model possible.

Addendum: @NeilSlater says below that some people perform hold-out on top of CV. In other words, they have a train-test split and then perform model selection on the training. According to him, they re-train using the original training set split, but not the testing set. The testing set is then used to perform a final model estimation. Personally, I see three flaws on this: (a) it does not solve the problem I mentioned with some hyperparameters being dependent on the volume of training since you are re-training anyway, (b) when testing many models, I prefer more sophisticated methods such as nested cross validation so that no data goes to waste, and (c) hold-out is an awful method to infer how a model will generalize when you have little data.


Data science Ph.D. program, what do you think? As Yann LeCun mentioned, a number of PhD programs in data science will be popping up in the next few years.

NYU already have one, where Prof.LeCun is at right now.

A statistics or cs PhD in machine learning is probably more rigorous than a data science one.  Is data science PhD for the less mathy people like myself?

Are these cash cow programs?

There is a huge industry demand for big data, but what is the academic value of these programs, as you probably can't be a professor or publish any paper.

It seems to me that the premise of a PhD is to expand knowledge in some little slice of the world. Since a "data scientist" is by nature is somewhat of a jack-of-all-trades it does seem a little odd to me. A masters program seems much more appropriate.

What do you hope to gain from a PhD? If the rigor scares (or bores) you, then what about a more applied area? Signal processing, robotics, applied physics, operations research, etc.


Best way to deploy and Schedule Deep Learning Model I am looking for best way to deploy my Deep Learning application which does the following:

Fetches data from BigQuery
Finds anomaly (based on trained model)
Throws the output in a mysql database.

I want this program to run everyday on a specific time like a cronjob.

Any lead on how can I achieve this deployment, other than using a cronjob because it will be unnecessary to pay for the time and resources when it will not be in use.

Thanks in Advance.

There are two steps necessary, you need a trained model and you need to apply this model regularly to new data.

Step 1: After training your model you need to serialize this in some way. This could be via the tooling in your deep learning framework, or by saving the parameters somewhere and the code somewhere else or by pickling the whole object. After storing this model artifact we can reuse this fitted model in step 2.

Step 2: Schedule some job that loads this model artifact, loads the data from BigQuery (in chunks or the full scope of the rows for the last time period), runs these through your model and saves the results. The schedule only kicks off some script, how to properly run these predictions depends a bit on the scale. If it is a very big job you would likely have to distribute this via something like Spark. If it is managable you could just do this on a single machine, maybe by chunking the data.


How can I improve the accuracy of my confusion matrix to 100%? Is there a possibility of attaining the above? Can someone share with me how to go about doing it if it is?

Bayes error

To answer your question, first I should explain Bayes error. Assuming we know the exact joint distribution of feature vectors ($\mathbf{x}$) and each class (${C_k}$) as
$P(\mathbf{x},{C_k})$, we build a classifier which assigns label $k$ to each feature vector by this criteria $$\mathop {\arg \max }\limits_k P(\left. {{C_k}} \right|\mathbf{x})$$
It can be shown that this is the best possible classifier by calculating the expected classification error on the whole feature space. This expected classification error is called Bayes error and is the minimum achievable classification error for this feature-label space.

  Training error

If you evaluate your model on the training data and calculate confusion matrix using the training samples you may achieve 100% accuracy because your model may overfit your training data. It means your training error is 0 even the Bayes error may not be.

  Generalization error

If you evaluate your model on the test data and calculate confusion matrix using the test samples you can not achieve 100% accuracy because you are evaluating the generalization capability of your model and its error can not be less than Bayes error.


How can I do tree_method ='exact' in XGBoost classifier? I am doing XGBoost classification on a huge data set and its showing:

Tree method is automatically selected to be 'approx' for faster speed. to use old behavior(exact greedy algorithm on single machine), set tree_method to 'exact'

How can I shift it to be exact?

I have tried clf = xgb(tree_method='exact') and it returned error!

Indeed tree_method is a parameter for Tree Booster. There are 4 choices, namely, auto, exact, approx and hist. The default is set to auto which  heuristically chooses a faster algorithm based on the size of your dataset. Approximate algorithm (approx) is chosen in your case since your dataset is huge.

To change the tree construction algorithm, you have to pass tree_method in the train function as follows:

import xgboost as xgb
...
parameters = {'tree_method': 'exact'} #You can add other parameters as well
model = xgb.train(parameters, x_train, num_round) #num_round is the number of rounds for boosting
prediction = model.predict(x_test)
...

I hope this helps.


Regular Expressions in Word I've used optical character recognition (OCR) on a historic directory, and am trying to clean up the text with Microsoft Word. Specifically, I need some help writing a Regular Expression to combine two lines together. For example something that is

John Smith, 87 Bank

Bldg

should actually be

John Smith, 87 Bank Bldg

I've tried several approaches, but haven't been successful at all. Can anyone help me with this?

I have a solution that might not be very standardized but will suffice your need. Copy all of your data into any advanced text editor like, Notepad++ or Sublime Text. Next, Use CTRL+H to toggle the find and replace feature. Find : '\n' and Replace with ''.


Generating rules from data for classification I have a set of unlabeled numeric data and i want to generate rules from it for classification purpose.  Which technique can i use?

If the data is unlabeled you either

manually label the data (if you have not too many samples)
manually label part of the data, train a classification model on it (and validate your model!) and use the model to label the unlabeled part, to speed up the labeling process
use pretrained classification/segmentation models (if your data is images)
use unsupervised learning algorithms to find patterns in the data, then manually inspect the groups obtained to find recognizable patterns and label the groups accordingly. The algorithm you should use depends on the type of features you have. With numerical features you can use distance-based clustering techniques like k-means, or dimensionality reduction techniques like Self-Organizing Maps or t-SNE

Basically, it depends on what kind of data you have and how many samples. I believe that more information on your data is required for providing more specific answers.


How Do I Find the Matthews Correlation Coefficient (MCC) in Python? I would like to find the Matthews Correlation Coefficient (MCC) for predictions made by a binary Sklearn machine learning model. How do I do it?

Taken form the Matthews Correlation Coefficient documentation in sklearn:

  The Matthews correlation coefficient is used in machine learning as a measure of the quality of binary and multiclass classifications. It takes into account true and false positives and negatives and is generally regarded as a balanced measure which can be used even if the classes are of very different sizes. The MCC is in essence a correlation coefficient value between -1 and +1. A coefficient of +1 represents a perfect prediction, 0 an average random prediction and -1 an inverse prediction. The statistic is also known as the phi coefficient.

So it's important to have the actual targets and the prediction values in the range of: -1 and +1. With this in mind you can simply run the following code:

> from sklearn.metrics import matthews_corrcoef
> y_true = [+1, +1, +1, -1]
> y_pred = [+1, -1, +1, +1]
> matthews_corrcoef(y_true, y_pred)

-0.33...

For more details about the formula refer to the Wikipedia page of Matthews Correlation Coefficient


Features selection: should I mix features? I have sets of features of different nature(for example, 300 features from FFT-transform, 1000 categorical features and so on). However there are only 900 samples and I`m trying to select important features using Lasso.
So the question is: should I perform feature selection differentially on subsets of features or better firstly, concatenate features and only then perform feature selection?

Let LASSO pick the best ones. If the features are highly correlated and you want them picked as a group, add some L2 regularization too. This is called Elastic Net regularization, and it is a generalization of L1 and L2 regularization. Other than that, do not feel obliged to artificially group features.