How do I calculate the delta term of a Convolutional Layer, given the delta terms and weights of the previous Convolutional Layer? I am trying to train an artificial neural network with two convolutional layers (c1, c2) and two hidden layers (c1, c2). I am using the standard backpropagation approach. In the backward pass I calculate the error term of a layer (delta) based on the error of the previous layer, the weights of the previous layer and the gradient of the activation in respect to the activation function of the current layer. More specifically the delta of layer l looks like this:

delta(l) = (w(l+1)' * delta(l+1)) * grad_f_a(l)

I am able to compute the gradient of c2, which connects into a regular layer. I just multiply the weights of h1 with it's delta. Then I reshape that matrix into the form of the output of c2, multiply it with the gradient of the activation function and am done.

Now I have a the delta term of c2 - Which is a 4D matrix of size (featureMapSize, featureMapSize, filterNum, patternNum). Furthermore I have the weights of c2, which are a 3D matrix of size (filterSize, filterSize, filterNum).

With these two terms and the gradient of the activation of c1 I want to calculate the delta of c1.

Long story short:

Given the delta term of a previous convolutional layer and the weights of that layer, how do I compute the delta term of a convolutional layer?

I am first deriving the error for a convolutional layer below for simplicity for a one dimensional array (input) which can easily be transferred to a multidimensional then:
We assume here that the $y^{l-1}$ of length $N$ are the inputs of the $l-1$-th conv. layer, $m$ is the kernel-size of weights $w$ denoting each weight by $w_i$ and the output is $x^l$.
Hence we can write (note the summation from zero): $$x_i^l = \sum\limits_{a=0}^{m-1} w_a y_{a+i}^{l-1}$$
where $y_i^l = f(x_i^l)$ and $f$ the activation function (e.g. sigmoidal).
With this at hand we can now consider some error function $E$ and the error function at the convolutional layer (the one of your previous layer) given by $\partial E / \partial y_i^l $. We now want to find out the dependency of the error in one the weights in the previous layer(s):
\begin{equation} \frac{\partial E}{\partial w_a} = \sum\limits_{a=0}^{N-m} \frac{\partial E}{\partial x_i^l} \frac{\partial x_i^l}{\partial w_a} = \sum\limits_{a=0}^{N-m}\frac{\partial E}{\partial w_a} y_{i+a}^{l-1} \end{equation}
where we have the sum over all expression in which $w_a$ occurs, which are $N-m$. Note also that we know the last term arises from the fact that $\frac{\partial x_i^l}{\partial w_a}= y_{i+a}^{l-1}$ which you can see from the first equation.
To compute the gradient we need to know the first term, which can be calculated by:
$$ \frac{\partial E}{\partial x_i^l} = \frac{\partial E}{\partial y_i^l} \frac{\partial y_i^l}{\partial x_i^l} = \frac{\partial E}{\partial y_i^l} \frac{\partial}{\partial x_i^l} f(x_i^{l})$$
where again the first term is the error in the previous layer and $f$ the nonlinear activation function.
Having all necessary entities we are now able to calculate the error and propagate it back efficiently to the precious layer:
$$ \delta^{l-1}_a = \frac{\partial E}{\partial y_i^{l-1} } = \sum\limits_{a=0}^{m-1} \frac{\partial E}{\partial x_{i-a}^l} \frac{\partial x_{i-a}^l}{\partial y_i^{l-1}} = \sum\limits_{a=0}^{m-1} \frac{\partial E}{\partial x^l_{i-a}} w_a^{flipped}$$
Note that the last step can be understood easy when writing down the $x_i^l$-s w.r.t. the $y_i^{l-1}$-s. The $flipped$ refers to a transposed weight maxtrix ($T$).
Therefore you can just calculate the error in the next layer by (now in vector notation):
$$\delta^{l} = (w^{l})^{T} \delta^{l+1} f'(x^{l})$$
which becomes for a convolutional and subsampling layer:
$$\delta^{l} = upsample((w^{l})^{T} \delta^{l+1}) f'(x^{l})$$
where the $upsample$ operation propagates the error through the max pooling layer.
Please feel free to add or correct me!
For references see:

Convolutional Neural Network by Stanford
Convolutional Neural Network by Andrew Gibiansky

and for a C++ implementation this (without requirement to install)


how to determine hashing bit length for multiple categorical features? Say I have $N$ categorical features $f_i$ $i\in(1,N)$ each of which of different alphabet size $n_i$. How can I efficiently optimize the hashing trick on that feature vector? Should I enumerate hash bit length for each feature independently (greedy)? Should i assume no hashing is necessary for features with small alphabet ($n_i$ is small)? What is the best practice strategy?

How can I efficiently optimize the hashing trick on that feature vector?

Use different size of hash, and hash function and see which work best.
One cannot tell in advance.

  Should I enumerate hash bit length for each feature independently (greedy)?

No, this is NEVER a good practice. The hash should be common to all features.

  Should I assume no hashing is necessary for features with small alphabet (ni is small)?

You can always choose not to hash specific features. It implies you think these features are more important, and should avoid collision.


Difference between mathematical and Tensorflow implementation of Softmax Crossentropy with logit Softmax cross entropy with logits can be defined as:

$a_i = \frac{e^{z_i}}{\sum_{\forall j} e^{z_j}}$

$l={\sum_{\forall i}}y_ilog(a_i)$

Where $l$ is the actual loss.

But when you look deep inside into C++ Tensorflow implementation of SoftmaxCrossEntropyWithLogits operation, the exact formula which they use is descibed as:

$l={\sum_{\forall j}}y_j ((z_j-max(z))-log({\sum_{\forall i}}e^{z_i-max(z)}))$

The part: $z-max(z)$ - is perfectly understood - it is just normalization which helps to avoid under/overflow.

BUT:

Where is the actual Softmax in their implementation?
Why from each $z_j$ they subtract $log({\sum_{\forall i}}e^{z_i-max(z)})$ before multiply it by $y_j$?

Note: One may argue that the code I provide is just Tensorflow's implementation of CrossEntropyWithLogits operation, but the actual SoftmaxCrossEntropyWithLogits operation - additionaly checks only dimentions and do not perform any more computation.

As I understand it, the softmax function for $z_i$ is given by $a_i$. Then just taking the loss you've defined you get back exactly the formula that is implemented. The way it is written down however is, as you mentioned, to avoid underflow/overflow.

For instance, suppose you want to compute the following:

$A=\log(\sum_{i=1}^{4}\exp(z_i))$, with $z_i=(-1000.5,-2000.5,-3000.5,-4000.5)$

Clearly, if you just type in the formula directly, you will get an underflow error. Instead if you isolate the main contribution in the exponential by taking the $\max(z_i)$, the same formula can be written as:

$A=\max_i(z_i)+\log(\sum_{i=1}^{4}\exp(z_i-\max_i(z_i)))$

The difference now is that the expression is "numerically stable" and we see that $A\approx -1000.5$.

Thus, let's make the softmax numerically stable:
\begin{align}
\log(a_i)&=z_i-\log(\sum_j e^{z_j})\\
&=z_i-\max_j(z_j)-\log(\sum_je^{z_j-\max_j(z_j)})
\end{align}
which is the expression that is implemented for the loss (just multiply by $y_i$ and sum over $i$).


Best approach for A/B testing two different recommendation systems I have two recommendation systems for musical preference which make a list of predictions for a particular user based on the songs they have saved in their library. The user then rates how good each recommendation was out of 6. I will be evaluating the performance of the recommendation systems based on the average rating given to songs recommended by system A and system B.
Let's use A to denote a song recommended by system A, and B to denote a song recommended by system B. For a particular user, should the recommendations be (AAAAAA or BBBBBB)  or should they be (ABABABAB)? I have implemented the first for now, being (AAAAAA or BBBBBB). Thus, in the current system, each respondent will be randomly assigned A or be, and only get the recommendations from that system. Is this the right approach, or does only recommending 1 system to each respondent bias them against what the other system could have recommended ?
Let's assume that B is far better than A. If we only recommend the same system to each user, and a user listens to songs which are all system A, they would never had heard system B, and the ratings of A would probably have been different (lower) if they had listened to the better system too. Is the ABABAB approach the best one ? Which is the best method to evaluate the performance of each system while reducing bias ?
Thanks

If user rates how good each recommendation then ABABAB approach is the best one. There could be case when system A becomes better than B overtime, then having 2 system does not make sense.


How can I know how to interpret the output coefficients (`coefs_`) from the model sklearn.svm.LinearSVC()? I'm following Introduction to Machine Learning with Python: A Guide for Data Scientists by Andreas C. Müller and Sarah Guido, and in Chapter 2 a demonstration of applying LinearSVC() is given.  The result of classifying three blobs is shown in this screenshot:

The three blobs are obviously correctly classified, as depicted by the colored output.

My question is how are we supposed to know how to interpret the model fit output in order to draw the three lines?  The output parameters are given by

print(LinearSVC().fit(X,y).coef_)
[[-0.17492286  0.23139933]
 [ 0.47621448 -0.06937432]
 [-0.18914355 -0.20399596]]

print(LinearSVC().fit(X,y).intercept_)
[-1.07745571  0.13140557 -0.08604799]

And the authors walk us through how to draw the lines:

from sklearn.svm import LinearSVC
linear_svm = LinearSVC().fit(X,y)
...
line = np.linspace(-15, 15)
for coef, intercept in zip(linear_svm.coef_, linear_svm.intercept_):
    plt.plot(line, -(line * coef[0] + intercept) / coef[1])  #HOW DO WE KNOW
plt.ylim(-10, 15)
plt.xlim(-10, 8)
plt.show()

The line of code with the comment is the one that converts our coefficients into a slope/intercept pair for the line:

y = -(coef_0 / coef_1) x - intercept/coef_1

where the term in front of x is the slope and -intercept/coef_1 is the intercept.  In the documentation on LinearSVC, the coef_ and intercept_ are just called "attributes" but don't point to any indicator that coef_0 is the slope and coef_1 is the negative of some overall scaling.

How can I look up the interpretation of the output coefficients of this model and others similar to it in Scikit-learn without relying on examples in books and StackOverflow?

Here's one (admittedly hard) way.

If you really want to understand the low-level details, you can always work through the source code. For example, we can see that the LinearSVC fit method calls _fit_liblinear. That calls train_wrap in liblinear, which gets everything ready to call into the C++ function train.

So train in linear.cpp is where the heavy lifting begins. Note that the w member of the model struct in the train function gets mapped back to coef_ in Python.

Once you understand exactly what the underlying train function does, it should be clear exactly what coef_ means and why we draw the lines that way.

While this can be a little laborious, once you get used to doing things this way, you will really understand how everything works from top to bottom.


Attention for time-series in neural networks Neural networks in many domains (audio, video, image text/NLP) can achieve great results.
In particular in NLP using a mechanism named attention (transformer, BERT) have achieved astonishing results - without manual preprocessing of the data (text documents).
I am interested in applying neural networks to time-series.
However, in this domain, it looks like most people apply manual feature engineering by either:

transposing the matrix of events to hold columns for each time observation and a row for each thing (device, patient, ...)
manually generating sliding windows and feeding snippets to an RNN/LSMTM.

Am I overlooking something? Why can't I find people using attention? Wouldn't this be much more convenient (automated)?

It is an interesting question. I would not completely agree with you though when you say that most time-series models dont use attention. However there is not as much documentation available on the web as there is for other applications.
LSTNet was one of the first papers that proposed using an LSTM + attention mechanism for multivariate forecasting time series. Temporal Pattern Attention for Multivariate Time Series Forecasting by Shun-Yao Shih et al. focused on applying attention specifically attuned for multivariate data.
Attend and Diagnose leverages self attention on medical time series data. This time series data is multivariate and contains information like a patient’s heart rate, SO2, blood pressure, etc.
A good link to further study this would be: https://towardsdatascience.com/attention-for-time-series-classification-and-forecasting-261723e0006d
Further quoting form the above paper: "self-attention and related architectures have led to improvements in several time series forecasting use cases, however, altogether they have not seen widespread adaptation. This likely revolves around several factors such as the memory bottleneck, difficulty encoding positional information, focus on pointwise values, and lack of research around handling multivariate sequences. Additionally, outside of NLP many researchers are not probably not familiar with self-attention and its potential."
I dont completely agree with the last statement, nonetheless I do agree that the benefits of attention have not yet captured the attention of researchers outside of NLP to the extent that it should have


Generating text using NLP based on parameters I want to generate some text based on the value of certain parameters. For instance, let's say I want to generate descriptions of video games. So, besides real descriptions as training data, I would like that the model takes in account the following parameters (for example) about the game:

Violent: yes
Multiplatform: yes
Drugs: no

So that if the game has drugs content, the output text has some phrase referring to it.

Is this possible? If so, how could I do it in Python? I was going to use LSTM neural networks in Tensorflow.

A direct way would be to encode any binary input features as embedded vectors and add them together as the initial hidden state for the LSTM, and then you train it as a normal language model.

The "little manual introduction" could be supplied to the language model (together with the initial hidden state created from the binary features) at inference time and then use it autoregressively to generate the description.

Given that you are going to deal with proper nouns (i.e. the name of the game), you should use a vocabulary that does not lead to out-of-vocabulary words. I would suggest using a subword-level vocabulary (e.g. byte-pair encoding).

For the implementation, you could do it in Pytorch, using as a starting point their language model example.


Image Classification on non real images I was wondering how image classifier networks perform on images that are not photographs. For example if you were to feed a drawing of a car or a face to an image classifier that was only trained on photos would the network still be able to classify the image correctly?

Furthermore what if you were to feed more and more abstract drawings into the network. As humans we are able to recognize objects even in abstract forms (i.e. modern art) but do current image classifiers generalize well enough in order to do this?

Are there any networks that are also trained on artist renditions of objects and not just photographs?

This question is slightly philosophical, but can be explained in this way - If you model is trained on real photographs it will likely not generalize well to things like drawings unless they are photorealistic or contain the features that the model is using to classify the images. You would likely have to include drawings in your training data to be able to do this effectively.

An interesting project leveraging Machine Learning that may have some relevance to what you are talking about is a recent Google project called "Quick Draw!" (link here) You should check it out. Essentially, it takes input of human drawings and outputs a classification label for what it thinks you drew.


Am I overfitting my random forest model (more information in description)? First off, sorry if this a novice question! Relatively new to all this stuff. Posted this in Stack Overflow and someone sent me here! Hope it's the right place.

Anyway, I'm working with 22 datasets that each have 180 observations of "Oddball" data and 720 observations of "Standard" data. I'm trying to use random forests for classification (i.e., oddball=1, standard=0). I understand there should be approximately equal trials/observations for both factors, but if I use 75% of the oddball data, then I'm barely using over 18% of the standard data. These data are pretty variable, and I think this could be problematic.

If I make four models, still using the same training data for each, am I overfitting my model? There's a lot more I've written, but this is basically what I'm trying to do:

jj = sample(1:180,(180*75),replace = F #Take 75% of all oddball data
kk = sample(181:900,(720*.75),replace = F) #Take 75% of all standard data
jj = sample(jj); kk = sample(kk) #Mix them up
kk = matrix(kk,4) #Divide the standard data so there are 4 sets of equal numbers for jj

samp1 = c(jj,kk[1,])
samp2 = c(jj,kk[2,])
samp3 = c(jj,kk[3,])
samp4 = c(jj,kk[4,])

I would then make four models (all while not touching the out-of-sample data) using each of these sample sets and average all their predictions to give me a "master" probability (i.e., an average of .8 would be deemed an oddball).

Is this overfitting the data? Is it even possible to overfit the data when using random forests? Is there something wrong with this intuition?

Thank you for anyone that helps! Your time and expertise is much appreciated.

This is more like a comment than an answer but my newbie status doesn't allow me to comment yet.

You might be interested in reading this article:
Random forest

4 samples appears to me being low a number of samples,  I would recommend that you try using more samples and compare your results.

Besides, there are some good implementation of the algorithm out there, are you trying to re-implement it for academic purposes?


What does the “numDecimalPlaces” in J48 classifier do in WEKA? Can anyone explain the purpose of "numDecimalPlaces" parameter in J48 WEKA classifier ? Its default value is 2 and its description is given as follows:

  The number of decimal places to be used for the output of numbers in
  the model.

However, I can't understand how it going to affect the results.
For example, if my feature values are something like this (0.00123 or 0.00234 etc). Does it means that the trained model will only consider these values up to 2 decimal points ? The documentation also says:

In the documentation, its mentioned in the summary fields section:                Field Summary

  Fields inherited from class weka.classifiers.AbstractClassifier
  BATCH_SIZE_DEFAULT, NUM_DECIMAL_PLACES_DEFAULT
  The number of decimal places used when printing numbers in the model.

It means that changing this decimal point should change the decimal points in the printed output of the trained classifier in WEKA explorer. However, when I try to change it (1,2,3,4,5 etc decimal points), it doesn't affect the number of decimals in the decision tree conditional statements in the printed output.
A snapshot:

k1 <= 0.12544
|   k4 <= 4.504
|   |   k22 <= 0.24044: Class1 (179.0)

Cheers

The short answer is: nothing. The numDecimalPlaces option, like debug and doNotCheckCapabilities, is part of the base class that all classifiers in WEKA inherit from. However, it is up to the implementation of the actual classifier to use this value to change the way the model is printed. There are no mentions of numDecimalPlaces anywhere in the source code for J48, which suggests to me that it does nothing in this case.

It looks like the only classifier in WEKA that does anything with the value entered into numDecimalPlaces is LinearRegression.


Expected 2D array, got scalar array instead: array=11 import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from sklearn.metrics import r2_score

# veri yukleme
veriler = pd.read_csv(r'C:\Users\k\Desktop\maaslar_yeni.csv')
# x burada bagımsız degısken y ise bagımlı degiskendir.
x = veriler.iloc[:,2:3]
y = veriler.iloc[:,5:]
X=x.values.reshape(-1,1)
Y=y.values.reshape(-1,1)

#linear regression
from sklearn.linear_model import LinearRegression
lin_reg = LinearRegression()
lin_reg.fit(X,Y)

import statsmodels.api as sm
model=sm.OLS(lin_reg.predict(X),X)
print(model.fit().summary())

In a tutorial, the instructor just changed x and by making new X,Y. Then he tries to use LinearRegression as you see in the picture for learning the R value. But when I try this I see the above error. How can I solve this?

Your code has a potential error

x = veriler.iloc[:,2:3]
y = veriler.iloc[:,5:]
X=x.values.reshape(-1,1)
Y=y.values.reshape(-1,1)

As you can see, you x has ONE column (2:3), but your y may have more than one (5: means 6th column to the end). As a result, when you do the reshape, then len(Y) might be two  times (or more) than len(X).

Please post few records of you data to help you more; but generally, X can have multiple columns and Y has one column. However, if you are going to do the regression for 1 column X and Y, the possible fix for your code is as below (remove the : for your y)

y = veriler.iloc[:,5]


How to train a single Artificial Neuron to predict f(x), where f(x)=x Conditions : 1. Network architecture: one single neuron.
2. x=99 (real number)
3. input to the neuron is x, output must be f(x) after trained.
4. use one bias and one weight.
5. use MSE for calculating error.
6. use gradient descent(original version) for back propagation.
7. learning rate: 0.01, no decay, no momentum.
8. must assign random value of weight and bias
will it work for this problem?

This will not work.
because, for this problem x=99, and when finding gradients, the results will be large numbers and updating weight and bias using learning rate 0.01 will take huge steps, and so the bias and weight will cross the solutions.
using learning rate less than 0.001 will solve this problem.
I tried learning rate 0.0001 and it worked.


How to enable GPU on GradientBoostingClassifier? Is there a way to enable GPU on GradientBoostingClassifier?

You are looking for this.
 Gradient Boosting, Decision Trees and XGBoost with CUDA


Why does joint embedding of word and images work? I often see some papers where the authors do point-wise multiplication of word and image embedding (e.g the image below).

Why does this  implementation works? I do not understand.

The model is more complex than point-wise multiplication of word and image embedding. It is a single neural network model thus backpropagation can improve the weights throughout the entire model. The training signal will adjust all layers to do better at the task, in this case the task appears to be question answering.
The figure is only showing the forward pass.


How to combine different kernels for Gaussian process in GPyTorch? I am trying to learn gaussian process by using GPyTorch to fit a Gaussian Process Regression model. However, I can't figure out a way to combine different kernels as shown in sklearn implementation of gaussian process. I am using GPyTorch as it is more flexible and have lot more kernels that one can play with compared to scikit-learn.
Any help, particularly with code snippet would be very useful.

Thank you.

From official documentation: For example, to compose two kernels via addition, you can either add the kernel modules directly:

self.covar_module = ScaleKernel(RBFKernel() + WhiteNoiseKernel())

Or you can add the outputs of the kernel in the forward method:

covar_x = self.rbf_kernel_module(x) + self.white_noise_module(x)

The class for GPRegressionModel would then look like this:

class GPRegressionModel(gpytorch.models.ExactGP):
def __init__(self, train_x, train_y, likelihood):
    super(GPRegressionModel, self).__init__(train_x, train_y, likelihood)

    self.mean_module = gpytorch.means.ConstantMean()
    self.covar_module = ScaleKernel(RBFKernel() + WhiteNoiseKernel())

def forward(self, x):
    mean_x = self.mean_module(x)
    covar_x = self.covar_module(x)
    return gpytorch.distributions.MultivariateNormal(mean_x, covar_x)


scikit-learn classification report's f1 accuracy? When I run scikit-learn classification_report() on my 2-class y and yhat, I get the following:
              precision    recall  f1-score   support

           0       0.84      0.97      0.90    160319
           1       0.67      0.27      0.38     41010

    accuracy                           0.82    201329   <--- WHAT?
   macro avg       0.75      0.62      0.64    201329
weighted avg       0.80      0.82      0.79    201329

I understand F1, macro-averaged F1, etc. But what is the accuracy value of 0.82 under the f1-score column?
I already read other questions about this function:
How to interpret classification report of scikit-learn?
Sklearn classification report is not printing the micro avg score for multi class classification model
macro average and weighted average meaning in classification_report

I give you that this is a weird way of displaying the data, but the accuracy is the only field that don't fit the schema. For example:
              precision    recall  f1-score   support

           0       0.84      0.97      0.90    160319
           1       0.67      0.27      0.38     41010

As explained in How to interpret classification report of scikit-learn?, the precision, recall, f1-score and support are simply those metrics for both classes of your binary classification problem.
The second part of the table:
    accuracy                           0.82    201329   <--- WHAT?
   macro avg       0.75      0.62      0.64    201329
weighted avg       0.80      0.82      0.79    201329

The accuracy is the overall accuracy of the model (note that accuracy is not a measure that is relative to a certain class, but a performance across all classes). The macro average for the precision and recall score is just the harmonic mean of the two classes. ie:
recall macro avg = (recall_class_1 + recall_class_0) / 2

The weighted average is just the average metric of the two classes weighted by the support/size-of-sample. ie:
recall weighted avg = (support_class_0 * recall_class_0 + support_class_1 * recall_class_1) / (support_class_0 + support_class_1)

This is a pretty long-winded way of saying the accuracy is just the overall accuracy. It has nothing to do with the column f1-score that it happens to be under.


difference between empirical risk minimization and structural risk minimization? I understand the meaning of empirical risk minimization as separate topic and was reading about structural risk minimization, it is hard for me to understand the difference between these two.

I read somewhere that perceptron uses Emperical risk minimization where as SVM uses structural.

I know that SVM takes into consideration model complexity so it is somewhat related to structural risk minimization put this is not clear to me whats the difference

SVMs were developed by Vapnik (1995,1998), based on the structural risk minimization principle (Vapnik, 1982) from statistical learning theory.

The complexity of the class of functions performing classification or regression and the algorithm’s generalizability are related. The Vapnik-Chervonenkis (VC) theory provides a general measure of complexity and proves bounds on errors as a function of complexity. Structural risk minimization is the minimization of these bounds, which depend on the empirical risk and the capacity of the function class

More Details with Math & Paper

More on SRM

In brief:

Empirical risk minimization (ERM) is a principle in statistical learning theory that defines a family of learning algorithms and is used to give theoretical bounds on their performance. The core idea is that we cannot know exactly how well an algorithm will work in practice (the true "risk") because we don't know the true distribution of data that the algorithm will work on, but we can instead measure its performance on a known set of training data (the "empirical" risk).

  Note the h here stands for the f — our classifier. This function is called Empirical Risk Minimization (ERM).

Structural risk minimization (SRM) is an inductive principle of use in machine learning. Commonly in machine learning, a generalized model must be selected from a finite data set, with the consequent problem of overfitting – the model becoming too strongly tailored to the particularities of the training set and generalizing poorly to new data. The SRM principle addresses this problem by balancing the model's complexity against its success at fitting the training data.

  which is called Structural Risk Minimization (SRM). J(f) is the complexity of the model, usually can be the bound of vector space. λ≥0 is the coefficient choosing the strength of the penalty term.

More Info at


Trying to train Stylegan, stopping with errors after first tick I'm just getting started with GANs and have prepared a dataset for Stylegan of around 5500 256x256 images to train it on. I've pored through the scant resources outlining the training process and have all of the software set up, using pretty much default settings for the training.py and training_loop.py files aside from specifying GPU number and dataset paths.

When I run the train.py script it runs for a single tick every time before giving me this error about input depths vs filter depths. These don't seem to be settings in Stylegan so I don't really know how to troubleshoot the issue. I've tried changing many different params in the training files and it always fails after one tick. Do these errors look familiar to anyone? My dataset seems like it shouldn't cause any issues.

Traceback (most recent call last):
File "/home/futureprojects/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
return fn(*args)
File "/home/futureprojects/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/futureprojects/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.(0) Invalid argument: input depth must be evenly divisible by filter depth: 1 vs 3
 [[{{node InceptionV3/_Run/InceptionV3/InceptionV3/import/conv/Conv2D}}]]
 [[InceptionV3/_Run/InceptionV3/Reshape/_4879]]
(1) Invalid argument: input depth must be evenly divisible by filter depth: 1 vs 3
 [[{{node InceptionV3/_Run/InceptionV3/InceptionV3/import/conv/Conv2D}}]]
0 successful operations.
0 derived errors ignored.

I haven't been able to make much sense of it from what I've read of Tensorflow's docs yet.

After further trial and error, it seems that commenting out the lines using the "metrics" object in Stylegan's training_loop.py script on lines 207, 264, and 267 have resolved the crashing issue.


Is addition of a random control variable a good idea, to ensure that results are correct? I'm dealing with a complex dataset with many patients who have a condition, and various qualities about these patients.  I'm trying to determine patient outcome based on patient qualities.
I'm using xgboost Classifier to try to determine one of the patient outcomes.
In order to ensure that my results are making sense, I've added a random variable, rand which is a random number within (0,10) that will have no impact on patient outcome.
The rand variable should show last in variable importance.
rand is meant as a control variable.
Is adding a control variable like rand a valid method to ascertain that my model makes sense?

This approach makes some sense but it's not the best approach for several reasons.
First, this control variable might not always be last in importance because it's possible that some other variable also don't have any impact at all on the target variable (outcome).
More importantly, the concept of a control group/control variable is useful in cases where one cannot evaluate objectively the effect of a method. Typically a drug trial needs a control group because of the placebo effect and various other biases. The control group plays the role of a baseline, and the performance of the method is measured relatively to this baseline.
In supervised classification, there is a much more direct way to objectively evaluate the effect of the method: this is exactly what we do when we evaluate the performance on a proper test set (fresh instances). Additionally there are simple ways to compare the performance of a system to a baseline: if there is no previous similar system for the task, one can simply use a random baseline or a majority baseline classifier.


How keras.layers.embedding learn word embeddings? I was trying some tensorflow tutorials and see that in all of them they use layers.embedding to learn these word embeddings, but how are these learned? , with a NN? which arquitecture? , or word2vec?
Thanks

The keras embedding layer is initialized with random weights and will learn an embedding for all of the words in the training dataset. So,the output vectors are not computed from the input using any mathematical operation. Instead, each input integer is used as the index to access a table that contains all possible vectors.
You could also use pre trained word2vec embedding.
Refer this to know more.


Oracle in optimization I have encountered the word oracle in the following context:

Given an $\alpha$-approximate oracle for stochastic optimization we
show how to implement an $\alpha$-approximate solution for robust
optimization under a necessary extension, and illustrate its
effectiveness in applications.

I saw this question, but it doesn't seem to have the same meaning. I was wondering what does oracle mean in this context.
Edits:
I found the following definition in this paper:

A $\rho$-approximate Bayesian optimization oracle is a function
$\mathcal{O}_{\rho}:(\Theta \rightarrow \mathbb{R}) \rightarrow
> \Theta$ for which: $$ f\left(\mathcal{O}_{\rho}(f)\right) \leq \inf
 _{\theta^{*} \in \Theta} f\left(\theta^{*}\right)+\rho $$ for any $f: \Theta \rightarrow \mathbb{R}$ that can be written as a nonnegative
linear combination of the objective and constraint functions $g_0,
g_1, \dots, g_m$.

I would appreciate if someone can shed some light on it.

The definition of $\alpha$-Approximate  Stochastic oracle has been given in the paper that you provide on Page $3$.

Given a distribution $D$ over $\mathcal{L}$, an $\alpha$-approximate stochastic oracle $M(D)$ computes $x^*$ such that such that
$$\mathbb{E}_{L \sim D}[L(x^*)] \le  \alpha \min_{x \in \mathcal{X}} \mathbb{E}_{L \sim D}[L(x)]$$

You  need not know the details of $M$, but if you provide it with a distribution $D$, then it returns you an $x^*$ such that if you evaluate the expected loss it is an most $\alpha$ times the objective value of the minimum of the expected loss.
Notice that different papers use different definitions and you should find the definition in the paper itself, as you can see, the first paper uses a multiplicative definition but the second paper uses an additive definition.


WHY or WHEN to convert numeric data to a categorical data? This is an open ended WHY TO or WHEN TO question rather than a question on HOW TO encode numeric to categorical data.

I am currently working on  Telco Customer Churn dataset from kaggle.

This is classification problem to predict if a customer will Churn or Not.

It has total 7043 rows and 21 columns, a mix of categorical and numerial columns. One of the column is Tenure which has data distributed in the range

> df.tenure.unique()

array([ 1, 34,  2, 45,  8, 22, 10, 28, 62, 13, 16, 58, 49, 25, 69, 52, 71,
       21, 12, 30, 47, 72, 17, 27,  5, 46, 11, 70, 63, 43, 15, 60, 18, 66,
        9,  3, 31, 50, 64, 56,  7, 42, 35, 48, 29, 65, 38, 68, 32, 55, 37,
       36, 41,  6,  4, 33, 67, 23, 57, 61, 14, 20, 53, 40, 59, 24, 44, 19,
       54, 51, 26, 39])

In one of the public kernel I saw the guy changing it into a categorical column.

> pd.cut(df['tenure'], bins=[0,12,24,48, 60, df['tenure'].max()], labels=['Tenure_0-12', 'Tenure_12-24', 'Tenure_24-48','Tenure_48-60','Tenure_gt_60'])
0        Tenure_0-12
1       Tenure_24-48
2        Tenure_0-12
3       Tenure_24-48
4        Tenure_0-12
            ...
7027    Tenure_12-24
7028    Tenure_gt_60
7029     Tenure_0-12
7030     Tenure_0-12
7031    Tenure_gt_60
Name: tenure, Length: 7032, dtype: category

It is a common practice to group/bin some numeric columns (usually with date/time columns like Age, Seasonal). I am not quite sure of the benefits of doing so across various Traditional ML algorithms.

Some Algorithms which are sensitive to dimentionality may perform bad, as the number of dimensions increases by One-Hot encoding of the newly created Categorical column.

Rationally, with the columns grouped into bins it will be very difficult to cluster the target for algo like K-NearestNeighbour or SVM to find a plane.

Please! Lets discuss on the impact of this technique on different ML Algorithms (Not just limited to tradiotnal or classification algorithms).

One reason to convert numerical data to categorical data is to improve the signal-to-noise ratio. Fitting a model to bins reduces the impact that small fluctuates in the data has on the model, often small fluctuates are just noise. Each bin "smooths" out the fluctuates/noises in sections of the data.


What is the difference between NLP and text mining? As discussed with Sean in this Meta post, I thought it would be nice to have a question which can help people who were confused like me, to know about the differences between text mining and NLP!

So, what are the differences between nlp and text-mining?

I have included my understanding as an answer. If possible, please explain your answer with a brief example!

I agree with Sean's answer.
NLP and text mining are usually used for different goals.
Also, there is indeed an overlap and both definitions are vogue.

Other than the difference in goal, there is a difference in methods.
Text mining techniques are usually shallow and do not consider the text structure. Usually, text mining will use bag-of-words, n-grams and possibly stemming over that.

In NLP methods usually involve the text structure. You can find there sentence splitting, part-of-speech tagging and parse tree construction. Also, NLP methods provide several techniques to capture context and meaning from text.

Typical text mining method will consider the following sentences to indicate happiness while typical NLP methods detect that they are not

I am not happy
I will be happy when it will rain
If it will rain, I'll be happy.
She asked whether I am happy
Are you happy?


Problems with Concatenating Embedded Categorical and Numerical variables for LSTM use I am new to here and new to Deep Learning too, so apologies in advance for any ill formatted code or wordings. I have a data set where I track 4 variables with 2 categorical and 3 numerical fields, over 4 time steps.
I was able build a dataframe like this:

cat1 - SckitLearn Label Encoded variable names
cat2 - SckitLearn Label Encoded time steps
num1 : num3 - SckitLearn Normalized variables

I got it turn into a list of numpy arrays like this:

I want to embed and concatenate all these fields before feeding it into a LSTM in Keras, using this function:
def build_concat(df):
  global inputs
  inputs = []
  global embeddings
  embeddings = []
  cat_cols = df.filter(like='cat')
  num_cols = df.filter(like='num')
  for cat_col in cat_cols:
    cat_input = Input(shape=(4,4), name=cat_col)
    unique_cat = cat_cols[cat_col].nunique()
    embedding_size = min(np.ceil((unique_cat)/2), 20)
    embedding_size = int(embedding_size)
    cat_dim = unique_cat + 1
    inputs.append(cat_input)
    embeddings.append(Embedding(cat_dim, embedding_size, input_length = (4,4),
        name=str(cat_col) + "_emb")(cat_input))
  for num_col in num_cols:
    num_input = Input(shape=(4,4), name=num_col)
    inputs.append(num_input)
    embeddings.append(num_input)

But I get this error :

ValueError: Shape must be rank 3 but is rank 2 for '{{node so2_model/concat/concat}} = ConcatV2[N=5, T=DT_FLOAT, Tidx=DT_INT32](so2_model/cat1_emb/embedding_lookup/Identity_1, so2_model/cat2_emb/embedding_lookup/Identity_1, IteratorGetNext:2, IteratorGetNext:3, IteratorGetNext:4, so2_model/concat/concat/axis)' with input shapes: [2,4,2], [2,4,2], [2,4], [2,4], [2,4], [].

This is my embedding list :

[<KerasTensor: shape=(None, 4, 4, 2) dtype=float32 (created by layer 'cat1_emb')>, <KerasTensor: shape=(None, 4, 4, 2) dtype=float32 (created by layer 'cat2_emb')>, <KerasTensor: shape=(None, 4, 4) dtype=float32 (created by layer 'num1')>, <KerasTensor: shape=(None, 4, 4) dtype=float32 (created by layer 'num2')>, <KerasTensor: shape=(None, 4, 4) dtype=float32 (created by layer 'num3')>]

This is my input list, before embedding :

[<KerasTensor: shape=(None, 4, 4) dtype=float32 (created by layer 'cat1')>, <KerasTensor: shape=(None, 4, 4) dtype=float32 (created by layer 'cat2')>, <KerasTensor: shape=(None, 4, 4) dtype=float32 (created by layer 'num1')>, <KerasTensor: shape=(None, 4, 4) dtype=float32 (created by layer 'num2')>, <KerasTensor: shape=(None, 4, 4) dtype=float32 (created by layer 'num3')>]

Thank you for your help!

You should reshape your numerical layers so that they have a shape (None,4,4,1). To concatenate, you need the all but one axis to be equal.
If you use:
tf.keras.layers.Reshape((4,4,1))(num_input)

this should work.


How to train the machine so that it can give 'out of bound/classes' as an output for neural network I know I was not able to word the title of the question properly. So I am trying to explain the problem here: Suppose, I built and trained a CNN to identify numbers from 0 to 9. However, when I deployed the CNN, someone gave "#" as an input (anything but 0-9). What can I do to my neural network during training, so that it's output can say that this is none of the training characters?

I'd like to present a second example: instead of classifying, suppose we want to do denoising using an autoencoder. Again, the CNN autoencoder is trained for 0 to 9. Now, how can we prepare it so that if someone gives "$"symbol (it can be anything that is not 0-9) as an input, it'll be able to identify that this symbol is none of what it was trained on? And the CNN autoencoder will be able to give an output according to that?

I think the best way would be to augment some data and have an additional output class "unknown". However, if that is not possible or the neural net can not be retrained I would compare the distribution of the outputs of a hidden layer.

For the CNN architecture below, calculate the empirical distribution for the outputs of a hidden layer after the flatten layer for the training data (e.g.$n1$). Save this distribution and during deployment, calculate the output of $n1$ for the test instance. If the output of $n1$ corresponds to a highly unlikely value with respect to the empirical distribution of training examples, return "unknown". Else, predict the digit for this instance.

(image source)


Gradient descent formula implementation in python So I recently started with Andrew Ng's ML Course and this is the formula that Andrew lays out for calculating gradient descent on a linear model.

$$ \theta_j = \theta_j - \alpha \frac{1}{m} \sum_{i=1}^m \left( h_\theta(x^{(i)}) - y^{(i)}\right)x_j^{(i)} \qquad \text{simultaneously update } \theta_j \text{ for all } j$$

As we see, the formula asks us to the sum over all the rows in data.

However, the below code doesn't work if I apply np.sum()

def gradientDescent(X, y, theta, alpha, num_iters):

    # Initialize some useful values
    m = y.shape[0]  # number of training examples

    # make a copy of theta, to avoid changing the original array, since numpy arrays
    # are passed by reference to functions
    theta = theta.copy()

    J_history = [] # Use a python list to save cost in every iteration

    for i in range(num_iters):
        temp = np.dot(X, theta) - y
        temp = np.dot(X.T, temp)
        theta = theta - ((alpha / m) * np.sum(temp))
        # save the cost J in every iteration
        J_history.append(computeCost(X, y, theta))

    return theta, J_history

On the other hand, if I get rid of the np.sum(), the formula works perfectly.

def gradientDescent(X, y, theta, alpha, num_iters):

# Initialize some useful values
m = y.shape[0]  # number of training examples

# make a copy of theta, to avoid changing the original array, since numpy arrays
# are passed by reference to functions
theta = theta.copy()

J_history = [] # Use a python list to save cost in every iteration

for i in range(num_iters):
    temp = np.dot(X, theta) - y
    temp = np.dot(X.T, temp)
    theta = theta - ((alpha / m) * temp)
    # save the cost J in every iteration
    J_history.append(computeCost(X, y, theta))

return theta, J_history

Can someone please explain this?

Your goal if to compute the gradients for the whole theta vector of size p (number of variables). Your temp is a vector also of size $p$, which contains the values of gradients of the cost function relative to each of your theta values.

Therefore, you want to substract point-wise the two vectors (with learning rate $\alpha$) to make an update, so no reason to sum the vector.


ggplot (in R)/matplotlib (in Python) with linear model? How can I plot the histogram below, using ggplot (R) and/or matplotlib (Python)?

In base plot I use:

histogram(~ Wrkday | Year+Avg, data=Data, layout=c(3,2))

The data I work on:

Code in R:

Input <- ("
Year       Student  Wrkday
      '1st year'  a        1200
      '1st year'  b        1400
      '1st year'  c        1350
      '1st year'  d         950
      '1st year'  e        1400
      '1st year'  f        1150
      '1st year'  g        1300
      '1st year'  h        1325
      '1st year'  i        1425
      '1st year'  j        1500
      '1st year'  k        1250
      '1st year'  l        1150
      '1st year'  m         950
      '1st year'  n        1150
      '1st year'  o        1600
      '1st year'  p        1300
      '1st year'  q        1050
      '1st year'  r        1300
      '1st year'  s        1700
      '1st year'  t        1300
      '2nd year'  u        1100
      '2nd year'  v        1200
      '2nd year'  w        1250
      '2nd year'  x        1050
      '2nd year'  y        1200
      '2nd year'  z        1250
      '2nd year'  aa       1350
      '2nd year'  ab       1350
      '2nd year'  ac       1325
      '2nd year'  ad       1525
      '2nd year'  ae       1225
      '2nd year'  af       1125
      '2nd year'  ag       1000
      '2nd year'  ah       1125
      '2nd year'  ai       1400
      '2nd year'  aj       1200
      '2nd year'  ak       1150
      '2nd year'  al       1400
      '2nd year'  am       1500
      '2nd year'  an       1200
      '3rd year'  u        1600
      '3rd year'  v        1700
      '3rd year'  w        1450
      '3rd year'  x        1650
      '3rd year'  y        1800
      '3rd year'  z        1550
      '3rd year'  aa       1950
      '3rd year'  ab       1750
      '3rd year'  ac       1925
      '3rd year'  ad       1825
      '3rd year'  ae       1625
      '3rd year'  af       1525
      '3rd year'  ag       1800
      '3rd year'  ah       1725
      '3rd year'  ai       1200
      '3rd year'  aj       1600
      '3rd year'  ak       1950
      '3rd year'  al       1100
      '3rd year'  am       1400
      '3rd year'  an       1600
      ")

Data <- read.table(textConnection(Input),header=TRUE)
los<-rbinom(nrow(Data), size = 1, prob=0.7)
Data$Avg<-ifelse(los==1,"Above 4.0","Below 4.0")

I would like to plot linear model in each box instead of histogram.

I don't know what you mean by "with linear model" in the title, but here's code that generates a toy dataset and replicates your plot.

library(tidyverse)
x<-crossing(year=paste("Year", 1:3), avg=c("Above 4.0", "Below 4.0"))
x$dat<-replicate(6, tibble(wrkday=runif(100, 1000, 2000)))
x %>%
  unnest(dat) %>%
  ggplot(aes(dat)) +
  geom_histogram(aes(y=..ncount..), bins=20) +
  facet_grid(avg~year)

EDIT:

Given your comment I think I understand what you mean by "with linear model."  You want to see how wrkday varies as a function of year, faceted by avg.  Here's some code:

library(tidyverse)

x<-tibble(year=sample(factor(paste("Year", 1:3)), 600, replace=T),
          avg=sample(c("Above 4.0", "Below 4.0"), 600, replace=T)) %>%
  mutate(wrkday=rnorm(600, mean=1000*(as.integer(year)-1)/2, sd=300))

ggplot(x, aes(x=as.integer(year), y=wrkday)) +
  geom_jitter(width=0.2) +
  geom_smooth(method="lm") +
  facet_wrap(~avg)


From pairwise comparisons to ranking - python I have to solve a ranking ML issue. To start with, I have successfully applied the pointwise ranking approach.

Now, I'm playing around with pairwise ranking algorithms. I've created the pairwise probabilities (i.e. probability of item i being above item j) but I'm not sure how I can transform this to rankings.

For the historical data (let's assume these are queries), I have their pairwise probs AND the actual ranking (the ideal one). I want a solution that will provide a ranking for a new query as well (i.e. the ideal ranking is what I'm looking for here).

Any python package that has, at least partially, the functionality I'm looking for?

EDIT: I have the pairwise probs for for all possible pairs of i and j.

Pairwise comparison models such as that of Bradley and Terry can easily be extended to your case, when you have pairwise comparison probabilities instead of binary outcomes.

Let $N$ be the number of items, and let $p_{ij}$ be the probability that query $j$ is better than query $i$.
Then, the log-likelihood of the Bradley-Terry parameters $\lambda_1, \ldots, \lambda_N$ given probabilities $\{ p_{ij} \}$ is

$$
\sum_{i,j} p_{ij} [\log(\lambda_j) - \log(\lambda_i + \lambda_j)]
$$

This can be reparametrized into a convex function, and the maximum-likelihood parameters can be found by one of many convex opimization methods.

Here is a simple Python algorithm that will find the ML estimate, using a minorization-maximization approach.

import numpy as np

def mle(pmat, max_iter=100):
    n = pmat.shape[0]
    wins = np.sum(pmat, axis=0)
    params = np.ones(n, dtype=float)
    for _ in range(max_iter):
        tiled = np.tile(params, (n, 1))
        combined = 1.0 / (tiled + tiled.T)
        np.fill_diagonal(combined, 0)
        nxt = wins / np.sum(combined, axis=0)
        nxt = nxt / np.mean(nxt)
        if np.linalg.norm(nxt - params, ord=np.inf) < 1e-6:
            return nxt
        params = nxt
    raise RuntimeError('did not converge')

Example usage:

import itertools

# Generating pairwise probability matrix.
pmat = np.zeros((10, 10))
for i, j in itertools.permutations(range(10), r=2):
    pmat[i][j] = (j + 1) / (i + j + 2)

# Estimating Bradley-Terry model parameters.
params = mle(pmat)

# Ranking (worst to best).
ranking = np.argsort(params)

Source: I am an author of a Python library for parameter inference in various statistical comparison models, choix.


What exactly is a trained model for machine-learning? I am curious about the deployment phase of a machine-learning model. So, after you run your script using python to train your millions of data and it works, what are some methods of deploying your model to the real world? I know libraries such as pickle which is being used to serialize a model exists, but what exactly is a model?

Is it some sort of blueprint which is then used to craft the predicted answer? If so, what does that mean that having a pickled object in the real world is much faster at churning predictions than re-training your model again?

A machine learning model is an algorithm which learns features from the given data to produce labels which may be continuous or categorical ( regression and classification respectively ). In other words, it tries to relate the given data with its labels, just as the human brain does.

Mathematical functions are used to map the features ( produced as a result of feature extraction ) to their labels.

  Models trained on a high amount of data can generalise themselves
  better. Generalisation is the ability of a model to give generalised
  predictions across varied or diverse data. It is not biased towards
  the data on which it was originally trained.

Production of models has these basic steps involved:

Collection of suitable data.
Preprocessing of data for training the model.
Training the model.
Evaluating the model.
Hosting the model for production.

Deploying a model:

This could be done in many ways. You can serve a model or run it on an IoT device. You basically want to freeze all the trainable parameters so that they are constants. The model's learning capabilities are removed so that it could only make predictions. There are some models which exhibit Online Learning.

Different frameworks like TensorFlow, Keras, PyTorch etc. have their own methods of saving models. Like in Keras, we can save a NN model as a hd5f serialized file. With TensorFlow Lite, we can run a model on an IoT device.

Firebase MLKit hosts the model in the cloud created with TensorFlow.

  The best deployment platform is determined by its usability,
  scalability and developer friendliness.


Which one of these tasks will benefit the most from SPARK? My company processes Data (I am only an intern). We primarily use Hadoop. We're starting to deploy spark in production. Currently we have have two jobs, we will choose just one to begin with spark. The tasks are:

The first job does analysis of a large quantity of text to look for ERROR messages (grep).
The second job does machine learning & calculate models prediction on some data with an iterative way.

My question is: Which one of the two jobs will benefit from SPARK the most?

SPARK relies on memory so I think that it is more suited to machine learning. The quantity of DATA isn't that large compared to the logs JOB. But I'm not sure. Can someone here help me if I neglected some piece of information?

I think second job will benefit more from spark than the first one. The reason is machine learning and predictive models often run multiple iterations on data.

As you have mentioned, spark is able to keep data in memory between two iterations while Hadoop MapReduce has to write and read data to file system.

Here is a good comparison of the two frameworks :

https://www.edureka.co/blog/apache-spark-vs-hadoop-mapreduce


How to get an intution for generating Keras CNN layers for a model I am going through tutorial for handwritten text recognition. And to do hand written digit recognition the author has constructed a Keras model as follows:

# # Creating CNN model

input_shape = (28,28,1)
number_of_classes = 10

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),activation='relu',input_shape=input_shape))

model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPool2D(pool_size=(2, 2)))

model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))

model.add(Dropout(0.5))
model.add(Dense(number_of_classes, activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),metrics=['accuracy'])

model.summary()

history = model.fit(X_train, y_train,epochs=5, shuffle=True,
                    batch_size = 200,validation_data= (X_test, y_test))

model.save('digit_classifier2.h5')

Source (here)

I am very confused that on how has the author choose these layers. I know how Conv2D works by applying filters to an image, I know what is activation function. In short I have a rough understanding of what each term means.

What I am finding it difficult is how do I know what is happening in each step of this code?
For example lets take this python code:

values_List=[11,34,43]
for index, num in enumerate(values_List):
    print(index,num)

I know that line 1 initializes a list named values_List
Line 2 iterates through this list
Line 3 prints output as (index of a number , number)

This python code is easy to understand and debug. But I am confused that if there is any error inside the keras layers. How do I proceed to debug this Keras code ?

Layer Choices : I believe this is an issue that requires domain expertise and experience. Neural network in general is still considered a black-box algorithm. As far as I understand what we try to do is improving model based on previous discovery of what actually works for our cases. Basically as you are more familiar with various model you may understand what works and what doesn't work and you will improve your model based on this. For example, you may want to check a more recent papers such as efficientnet, which is a very light yet powerful image classification model. They combined techniques and ideas that people have proven to work before and we have this amazing model. As for simple models like what you did above you parameter search should suffice. You might also consider a more sophisticated approach is using automl approach such as autokeras.

Debugging : I believe the tools you are looking for is model summary and visualization https://keras.io/visualization/. The first one is simple you simply call model.summary() which I believe you have already used on your code, this will get you a summary of the model and if you already understand how some CNN building block works you should have already get the idea how the shape changes and hence you can judge based on this whether or not it satisfy your expectation. As for your model it is still fairly easy to debug, since it is sequential and I believe the points that I mentioned above are enough.


Why is there such a wide range of prices for whole genome sequencing services? Below is a list of labs around the country that offer genetic sequencing.  What would be the reasons why they each cost a different amount if the end result is that you get a 90GB hard drive with 3+ billion base pairs for your genome?

https://www.scienceexchange.com/services/whole-genome-seq

Quality
Certificates (is "for research use only" ok for you?)
There is no "common" or "average" genome, some of the prices are for bacterial some for human(4,6 x 10 ^6 bp (E.coli) vs. 3.3 x 10^9 bp(human))


Should one normalize the frequency values when feeding it as an input to machine learning model? Consider an unsupervised data. The data is in the form of a csv file( I am using pandas dataframe for this). Its a web page data at different time steps and the way I am converting data to be fed to my model(K-means) is by taking difference of the time steps of the current web-Page ID load to next web_page ID load.

Now, there are some features in the data like "scroll" (which represents a human scrolling on that webpage) which is occurring multiple times for the same web page ID. Since I am only using delta the way I want to encode this "scroll" as a feature is how many times it happened between the delta(time difference). This gives the frequency.

Now the question is should I do some processing on this raw frequency I calculated, or can I directly feed it to my model. In case more processing is needed, what do you suggest?

As generic advice, any algorithm that uses distances might be affected by scaling or normalization. Thus, if you use one of them, you need to think about it twice and consider how this will alter your problem.

I will take the answer from the CrossValidated StackExchange since another user has already explained it there.

  If your variables are of incomparable units (e.g. height in cm and
  weight in kg) then you should standardize variables, of course. Even
  if variables are of the same units but show quite different variances
  it is still a good idea to standardize before K-means. You see,
  K-means clustering is "isotropic" in all directions of space and
  therefore tends to produce more or less round (rather than elongated)
  clusters. In this situation leaving variances unequal is equivalent to
  putting more weight on variables with smaller variance, so clusters
  will tend to be separated along variables with greater variance.

ttnphns's answer


Methods / Algorithms for rank scales based on cumulative scoring Say you have an organization that requires employees to participate in a Q&A site similar to StackOverflow - questions and answers are voted upon, selected answers get extra points, certain behaviors boost your score etc.  What we need to do is assign a rating from 1-100 to these users with even distribution.

The behaviors that add points:

Ask a question [fixed]
Answer a question [fixed]
Receive an upvote on a question [determined by relative ranking]
Receive an upvote on an answer [determined by relative ranking]
Have your answer selected [determined by relative ranking]
Responding to a comment, etc [fixed]

Likewise, there are behaviors that subtract points.

If a user with a high ranking upvotes a question asked by a lower-ranking user, more points should be awarded than the inverse situation.  Likewise if a lower-ranking user downvotes a higher-ranking user's question, the impact should be minimal compared to the inverse.  There should be a limit to this impact though so that a high-ranking user doesn't unintentionally destroy any momentum of a low-ranking user by issuing a powerful downvote.

We have a few challenges here:

How do we determine how many points to assign to each type of behavior, with actor/recipient relative rank taken into account?
I'm thinking we just assign a flat number to each behavior, that number decided relative to the importance of the other behaviors, and then have a variable score that can alter the score if there is a wide variance between the users.  The mechanics of this - does the score double at most? - are unclear.
How to we assign this rank? This one is a little easier - I'm thinking we just order the users according to score and then split the dataset into 100 sections, assigning each "chunk" a number 1-100.
Should we be worried about these numbers getting "very big"?  The scenario described above has been trivialized; actions taken by these users may happen hundreds of times per day so the scores can become very high, very quickly. Is there a way we can keep this under control while avoiding a large number of duplicate scores?
How do we define the "fixed" scores as the total scores become very big?  Over time we may have users with hundreds of thousands of points - but the fixed-score behaviors should still reward them.  They should reward lower-ranking users more than higher-ranking users.

I don't know if there are some standard practices, algorithms, or terminologies that I should be aware of when facing a problem like this - any input would be appreciated.

To solve challenges #3 and #4, let's limit the overall available rank volume. For example, sum of this rank for all the users will be 1 (100%).

From challenge #2 I understood, that you accept 2 different ranks: (1) place from 1 to 100, and (2) simple sum of all earned points (fixed and relative). Did I got it right? I so, there is no need to worry about unlimited growth, or fixed scores inflation. Let's just use percentages, not 1-100 ranks.

These percentage ranks could be calculated based on interaction behaviors (vote/selecting answer/etc), using PageRank-like algorithm. Such algorithm will consider all previous reactions (and ranks of acted users), obtained by an exact user. Unfortunately, you cannot use PageRank algorithm "as is", because it supports only "positive" links, but you can look for it's extensions. For example, look at this paper with PageRank extension  for both positive and negative links (as users can down vote). You can iteratively estimate percentage rank (TrustRank, TR) using this algorithm.

The second task is to calculate reward/penalty rate in points for each single action. Let's determine (predefine) maximal reward/penalty rate (X) for each type of action. And will use coefficient to discount it, based on TrustRanks of acting users (e.g., author and voter). Slightly modified Sigmoid will map this ratio from [-Inf,+Inf] range to [0,1].

Here for peer users you will have ~0.5 of predefined maximal rate. If "voter" has TR twice more than "author", "author" will recieve ~0.75 of predefined value, and so on. You can tune steepness with additional parameter, or try to find any other mapping transformation function.

Anyway, now simply multiply maximal penalty/reward by this coefficient, and you'll get the number of point, you need to deduct or add. The only issue, I see, is a user with zero TR - such user as a voter will "give" nothing, and as an object of voting, will recieve the maximal amount of points regardless voter's rank. To avoid this, you can predefine minimal TR (like 1e-10), and don't let user's TR to fall beyond this value.


Document classification: tf-idf prior to or after feature filtering? I have a document classification project where I am getting site content and then assigning one of numerous labels to the website according to content.

I found out that tf-idf could be very useful for this. However, I was unsure as to when exactly to use it.

Assumming a website that is concerned with a specific topic makes repeated mention of it, this was my current process:

Retrieve site content, parse for plain text
Normalize and stem content
Tokenize into unigrams (maybe bigrams too)
Retrieve a count of each unigram for the given document, filtering low length and low occurrence words
Train a classifier such as NaiveBayes on the resulting set

My question is the following: Where would tf-idf fit in here? Before normalizing/stemming? After normalizing but before tokenizing? After tokenizing?

Any insight would be greatly appreciated.

Edit:

Upon closer inspection, I think I may have run into a misunderstanding at to how TF-IDF operates. At the above step 4 that I describe, would I have to feed the entirety of my data into TF-IDF at once? If, for example, my data is as follows:

[({tokenized_content_site1}, category_string_site1),
 ({tokenized_content_site2}, category_string_site2),
...
 ({tokenized_content_siten}, category_string_siten)}]

Here, the outermost structure is a list, containing tuples, containing a dictionary (or hashmap) and a string.

Would I have to feed the entirety of that data into the TF-IDF calculator at once to achieve the desired effect? Specifically, I have been looking at the scikit-learn TfidfVectorizer to do this, but I am a bit unsure as to its use as examples are pretty sparse.

As you've described it, Step 4 is where you want to use TF-IDF. Essentially, TD-IDF will count each term in each document, and assign a score given the relative frequency across the collection of documents.

There's one big step missing from your process, however: annotating a training set. Before you train your classifier, you'll need to manually annotate a sample of your data with the labels you want to be able to apply automatically using the classifier.

To make all of this easier, you might want to consider using the Stanford Classifier. It will perform the feature extraction and build the classifier model (supporting several different machine learning algorithms), but you'll still need to annotate the training data by hand.


Best framework for recognizing a specific cartoon character's face? I have a supply of images of a specific cartoon character's face. I have hours of video. I would like to automatically find the sections of the video in which this cartoon character appears.

https://github.com/ageitgey/face_recognition doesn't seem to work very well on cartoon characters (it fails to recognize any face in my images).

What is the state of the art on this? Is there an open-source library or framework that is good at this?

You will most likely need to train a neural network to detect your particular cartoon characters. Although some parts of this are tedious, this type of task is well-documented and there are many user-friendly frameworks available. I would recommend reading the tensorflow object detection api tutorial.

The most difficult part will simply be collecting your data. You will need to sample video frames to train the neural network. Depending on how many characters are in your cartoon, the complexity and variability of the inputs, and network model that you choose, I suspect you will need to collect anywhere from 200-500 distinct samples for training.

After you collect your frames, you will need to annotate your data. "Annotation" is the process by which you manually draw bounding boxes around your characters so that the neural network knows what to look for. This process is described in further detail in the above link. Fortunately, you do not need to program the annotation tool yourself; the Tensorflow tutorial instructs you to install LabelImg, which provides a graphical interface for you to label your selected frames.

After creating your dataset, you can proceed to train your network using the Tensorflow instructions.

If you have a lot of video to inference, I would recommend sampling your frames in relatively large intervals such as 5 or even 10 (depending on the fps of the video). You can then inference 5 to 10 times faster with reasonable accuracy.

For example, suppose your network inferences frames $ x_t $ through $ x_{t+10} $ in 5 frame intervals. If both $ x_t $ and $ x_{t+5} $ have at least one bounding box of reasonable size, then frames between $ x_t $ and $ x_{t+5} $ most likely have characters as well. If frame $ x_{t+10} $ does not have a bounding box of reasonable size, then we might assume that the character(s) leave the frames at about $ x_{t+7} $ or $ x_{t+8} $. This method will let you control the inherent speed/accuracy trade-off.


Regularization practice with ANNs I have learnt from some examples the existence of regularization option at ANNs (concretely, at Keras implementation). As far as I know, regularization in general is a kind of "penalty" on parameters to prevent model complexity and overfitting.

Accordingly, W_regularizer and b_regularizer options in Keras are for weight and bias parameter regularization, unless I am mistaken. But what is activity_regularizer for? How is it related to the weight/bias regularization? And more generally: what is a good practice to using all these regularization possibilites (apart from the blind brute force tuning)? Because of ANNs/CNNs are produce very low overfitting measured on the validation set, it seems me that regularization is not a really useful tool with neural nets.

activity_regularizer are used to control the output of a neural network. They tend to make the output smaller. Suppose the loss function is give as :

loss function = DataLoss + regularizationLoss

Then for weight_regularizer, regularizationLoss = f(Weights in a network).
But for activity_regularizer, regularizationLoss = f(Predicted outputs from a network). Activity regularizers are generally used when you are quite aware of the distribution of the test dataset.

For your second argument, I would say you are pretty wrong. ANN as well as CNN both can suffer from overfitting. In order to prevent the model to overfit, we generally use a lot of regularization techniques among which Dropout is quite popular.


why trainable parameters are not considered right? I have tested the "ResNet" block and it works fine,but when i call it in the model class, in some how it does not work properly ?
Is it related to the model definition ?

In your code by default training=False, set training=True it will work right.
Also, since you are already subclassing keras.Model you don't need to again call keras.Model. Remove the model() method, pass the input directly to call() method by setting training=True.


General Machine Learning Workflow Question I'm very new to machine learning and want to understand the general process by which it is carried out. I've worked through the famous 'iris' tutorial and want to ask if the principles in that tutorial are applicable to every future machine learning project I undertake.

I work in biological sciences, and am interested in applying machine learning algorithms to biological sequence data as away of categorizing classes or doing unsupervised clustering. From my understanding, each project starts with:

Me defining the 'scope' or aim of what I want to learn from the raw data.
Generating features/attributes from my OWN code/algorithms that could potentially differentiate between the two classes
This basically generates a huge matrix of x_features by y_entries
Feed the matrix into a machine learning algorithm (I'm sure this is vastly oversimplified).

To give an example, say I have 10,000 protein sequences, and I believe 5000 are 'Class1' and 5000 are 'Class2', but I do not know how to differentiate them by eye. I need to generate x_features (in some informed way) of this sequence using my own custom algorithms, and feed the resulting 10000 entries into an algorithm.

Is this the right approach? I'd be eternally grateful if someone could direct me towards a beginners tutorial that revolves around analyzing biological sequence data.

I agree with most of the answer. However, I think you are missing some points including the cross-validation step. I try below to provide an overview of a common machine learning project. I assume a common project is a supervised machine learning problem (like iris dataset).

1. You defining the 'scope' or aim of the project :

You have to define the purpose of the learning.
When working with business in a company, it is a good idea to correctly express the need, the value of the project and its goal.
You have to define the evaluation metrics (accuracy, recall, F1 score, AUC...). You can also define a minimum result you want to reach (say 80% accuracy for example).
You can also ask yourself about the level of interpretability you need (do you care about model explanation? If no, maybe you could try more blackbox algorithms such as boosting, neural networks...).

2. Explore your data :

Using statistics, visualization and intuition, try to learn your dataset and understand your features and labels.
You can also search for missing data and outliers. Correct these observations refers as data cleaning process.
Understanding your input variables will greatly help you to create and select relevant features.

3. Generating features/attributes from your own code/algorithm :

This phase refers as features engineering. It is about creating features relevant to the learning problem.
In this phase, you can clean your missing data and outliers in order to help the learning.
You can derive new features from input variables relevant to your learning problem (handle categorical variables, rescale your features, apply transformations on input variables).

4. cross-validation :

Cross-validation refers to your algorithm evaluation. In supervised machine learning, it is common, at least, to split dataset into 3 datasets (train, validation and test).
Train dataset (about 60% of data) aims to train the algorithm.
Validation dataset (20% of data) helps to find the best hyperparameters of your model (max depth for a tree, regularization for a linear/logistic regression...).
Finally, test set (20% of data) gives you the true result you get on unseen data. It is the final evaluation.

5. Machine learning, feed the matrix into an algorithm :

In this part, you train machine learning algorithms with regards to the cross-validation process (part 4).
You can test different models. Some yield different performance results. Interpretability is not the same neither.
To help the learning, you can diagnose your algorithm performs on both train and validation sets. This diagnostic is also called learning curves. It can tell you how to improve your learning. The purpose of learning curve is to help handle the underfitting/overfitting tradeoff. Underfitting is when you have a large bias error meaning your algorithm is not complex enough while overfitting means your algorithm is too complex and learns perfectly but is not able anymore to generalize learning on new unseen observations.
You can also look at residuals (errors between predictions and real values) to improve your algorithm.
Make features selection may also improve your algorithm learning.

6. Restitution

Interpret the model and the performance you get.
Create restitutions to business? Run into production?

Improve your machine learning model and performance is mostly about improving the above introduced points. By making new exploration, create new features, try a more powerful algorithm and so on, you can reach best results. Machine learning is a whole pipeline you have to optimize. I also think machine learning projects managements are really suitable with agile approaches.


How to force a NN to ouput the same output given a reverse input? I want to choose an architecture that can deal with an input symmetry.
As input, I have a sequence of zeros and ones, like [1, 1, 1, 0, 1, 0] and at the output layer I have N neurons that outputs a categorical distribution like [0.3, 0.4, 0.3].
How can I force a NN to ouput the same distribution when I feed its reverse copy, i.e [1, 1, 1, 0, 1, 0]?
A simple way just to learn twice:
feed straight [1, 1, 1, 0, 1, 0] -> [0.3, 0.4, 0.3]
feed reverse  [0, 1, 0, 1, 1, 1] -> [0.3, 0.4, 0.3]

Or maybe, there are more "elegant" ways? What type of architecture I should use or maybe I need to play with loss functuions?

The most general and robust way is to double the training set using the "mirror image" of each sample. Also it is sound approach from a statistical learning perspective.
There can be some approaches to symmetry where one hacks the NN architecture (eg by symmetrising the network) but these are usualy very fragile and do not generalise well, while the above approach is robust and lets the network adjust itself as it needs.
A similar but cheaper approach is to sort the samples before feeding them to the network. Eg for your example:
If you have this input [1, 1, 1, 0, 1, 0] then the sorted related mirror input (in lexicographic order) is the [0, 1, 0, 1, 1, 1] (reverse). So you train the NN only the sorted lexicographic inputs and before testing each sample you convert it to the sorted lexicographic related input (by reversing it) if it is not already in that form.
So effectively you normalise the samples (by reversing them if necessary) both before training and before testing. This is similar approach to doubling the train set, only it is cheaper (and in practice exactly accurate, while doubling the training set is only theoreticaly accurate, in practice it may have discrepancies).


How to create a good list of stopwords I am looking for some hints on how to curate a list of stopwords. Does someone know / can someone recommend a good method to extract stopword lists from the dataset itself for preprocessing and filtering?

The Data:

a huge amount of human text input of variable length (searchterms and whole sentences (up to 200 characters) ) over several years. The text contains a lot of spam (like machine input from bots, single words, stupid searches, product searches ... ) and only a few % of seems to be useful. I realised that sometimes (only very rarely) people search my side by asking really cool questions. These questions are so cool, that i think it is worth to have a deeper look into them to see how people search over time and what topics people have been interested in using my website.

My problem:

is that i am really struggling with the preprocessing (i.e. dropping the spam). I already tried some stopword list from the web (NLTK etc.), but these don't really help my needs regarding this dataset.

Thanks for your ideas and discussion folks!

One approach would be to use tf-idf score. The words which occur in most of the queries will be of little help in differentiating the good search queries from bad ones. But ones which occur very frequently (high tf or term-frequency) in only few queries (high idf or inverse document frequency) as likely to be more important in distinguishing the good queries from the bad ones.


How do I correctly build model on given data to predict target parameter? I have some dataset which contains different paramteres and data.head() looks like this

Applied some preprocessing and performed Feature ranking -

dataset = pd.read_csv("ML.csv",header = 0)

#Get dataset breif
print(dataset.shape)
print(dataset.isnull().sum())
#print(dataset.head())

#Data Pre-processing
data = dataset.drop('organization_id',1)
data = data.drop('status',1)
data = data.drop('city',1)

#Find median for features having NaN
median_zip, median_role_id, median_specialty_id, median_latitude, median_longitude = \
    data['zip'].median(),\
    data['role_id'].median(),\
    data['specialty_id'].median(),\
    data['latitude'].median(),\
    data['longitude'].median()

data['zip'].fillna(median_zip, inplace=True)
data['role_id'].fillna(median_role_id, inplace=True)
data['specialty_id'].fillna(median_specialty_id, inplace=True)
data['latitude'].fillna(median_latitude, inplace=True)
data['longitude'].fillna(median_longitude, inplace=True)

#Fill YearOFExp with 0
data['years_of_experience'].fillna(0, inplace=True)
target = dataset.location_id

#Perform Recursive Feature Extraction
svm = LinearSVC()
rfe = RFE(svm, 1)
rfe = rfe.fit(data, target) #IT give convergence Warning - Normally when an optimization algorithm does not converge, it is usually because the problem is not well-conditioned, perhaps due to a poor scaling of the decision variables.

names = list(data)
print("Features sorted by their score:")
print(sorted(zip(map(lambda x: round(x, 4), rfe.ranking_), names)))

Output

Features sorted by their score:

[(1, 'location_id'), (2, 'department_id'), (3, 'latitude'), (4, 'specialty_id'), (5, 'longitude'), (6, 'zip'), (7, 'shift_id'), (8, 'user_id'), (9, 'role_id'), (10, 'open_positions'), (11, 'years_of_experience')]

From this I understand that which parameters have more importance.
Is above processing correct to understand the feature important. How can I use above information for better model training?

When I to model training it gives very high accuracy. How come it gives so high accuracy?

from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

dataset = pd.read_csv("prod_data_for_ML.csv",header = 0)

#Data Pre-processing
data = dataset.drop('location_id',1)
data = data.drop('status',1)
data = data.drop('city',1)

#Find median for features having NaN
median_zip, median_role_id, median_specialty_id, median_latitude, median_longitude =\
    data['zip'].median(),\
    data['role_id'].median(),\
    data['specialty_id'].median(),\
    data['latitude'].median(),\
    data['longitude'].median()

data['zip'].fillna(median_zip, inplace=True)
data['role_id'].fillna(median_role_id, inplace=True)
data['specialty_id'].fillna(median_specialty_id, inplace=True)
data['latitude'].fillna(median_latitude, inplace=True)
data['longitude'].fillna(median_longitude, inplace=True)

#Fill YearOFExp with 0
data['years_of_experience'].fillna(0, inplace=True)

#Start training

labels = dataset.location_id
train1 = data
algo = LinearRegression()
x_train , x_test , y_train , y_test = train_test_split(train1 , labels , test_size = 0.20,random_state =1)

# x_train.to_csv("x_train.csv", sep=',', encoding='utf-8')
# x_test.to_csv("x_test.csv", sep=',', encoding='utf-8')

algo.fit(x_train,y_train)
algo.score(x_test,y_test)

Output: 0.981150074104111

from sklearn import ensemble
clf = ensemble.GradientBoostingRegressor(n_estimators = 400,
                                         max_depth = 5,
                                         min_samples_split = 2,
                                         learning_rate = 0.1,
                                         loss = 'ls')
clf.fit(x_train, y_train)
clf.score(x_test,y_test)

Output: 0.99

What I want to do is, predicting  location-id  correctly.
 Am I doing anything wrong? What ithe s correct way to build model for this sort of situation?

I know there is some way that I can get Precision, recall, f1 for each paramteres. Can anyone give me reference link to perform this.

Train data includes latitude, longitude and zipcode. Output variable is location.

It is trivial to predict location if zip and lat,lon are known.  Try removing these attributes and see if that has any impact on validation score.


What are real world applications of Doc2Vec? I am new to Doc2Vec. As I understand Doc2Vec group similar documents based on the context of their words.

I have a set of newspaper documents and I want to identify what are the main topics of the newspapers (group 'Politics' news documents to one group, 'Sports' news documents to another group etc.) based on their content. Thus, I am interested in knowing;

What variant of Doc2Vec is more suitable for this (dbow, dm)?
What are the real world applications of Doc2Vec algorithm?

I will answer your second question first, doc2vec and word2vec both are primarily good representations of text data that capture the semantics of words and documents. So whenever you are working with text data, you need a representation for it and that is what word2vec and doc2vec provides. Now think of any real world task on text data, like document similarity, using doc2vec you can find cosine similarity between two documents easily, now think of real world applications for it, finding duplicate question on a site like stack overflow, ranking candidate answers for a question answering model, features for text cladsification like sentiment analysis ( word2vec doesnt work well here, the context of good and bad are quite similar so it struggles to differentiate between positive and negative reviews). So these are just representations, you can apply them to any NLP and a lot of IR task.
To answer your first question, a model is not just task dependent but also data dependent. So you can read Mikolov's paper to find out how each model works for the baseline tasks but a good idea is to try both models on your data and extrinsically evaluate which algorithm performs better.


classification performance metric for high risk medical decisions What is the best classification performance metric for risky medical treatments like surgery? for example a patient should NOT suggest a surgery (negative) if he/she can be treated by medicine (positive). Does Negative predictive value (TN/TN+FN) works for this situation?

Check ROC curve, increase the threshold and measure PPV.
Also you can not use only one in isolation, you have to check sensitivity, specificity and PPV in order to understand the complete scenario.


Reduce data length to train effectively I have  customer buying data with each row specifying an item bought by customer. The problem is that even if at the same time customer buys five items then there are five different rows for it and as a result the total number of rows in data have gone too much to train. what can i do to reduce the size of data so that i can train it effectively.Just to give the context of the problem, i want to recommend products to the customers based on their buying data.

Dataset size: (7981262, 16)

Data description:

Variable                                           Description
customerID                                    unique customer ID
DOB                                         date of birth of customer
Gender                                               gender
State                                          customer's state
PinCode                                pincode of area where customer lives
transactionDate                                  date of transaction
store_code                                 unique code of store
store_description                             description of store
till_no                                       counter no. in the store
transaction_number_by_till                 unique transaction number by counter,
transactionDate, store_code
promo_code                      if promotional code (offer) used in the transaction
promo_description                            description of the offer
product_code                               unique code of the product purchased
product_description                       description of the product purchased
sale_price_after_promo                 sale price of the product after applying
promotion
discountUsed                  after promo, customer used this discount(s) on transaction

If your total dataset size is too much, take a random sample of n%. I suggest taking a sample of n% of users, then using all rows for each of those sampled users. If you have timestamps, I further suggest making your train/test split based on time. I.e., for a user with 10 purchased items, train on her first 8 items and try to predict her last 2 items.


Binary classification of partially labeled data One of my friends was asked this question in an interview. A clue/restriction is given: Do NOT use semi-supervised learning techniques.

Suppose you have a binary classification problem: There are ~10K labeled points and ~1M unlabeled points. What are the steps in solving this problem?

You of course cannot use them as training data, but they can still have some potential uses, for example:

-You can make predictions on that data and see if the distribution of the classes are similar to those in the labeled data.

-You can use them for feature engineering – e.g. if you are using something like principal components, or if you are using averages for some categorical variable (e.g. you have a variable like state and another like income, and you augment the state variable by creating a feature that is the average income for that state).

-If you need to impute missing values in the predictors, it’s better to use the whole dataset for that.

-You can see if the distributions of the predictor variables are similar, and if you see that, for example, in your test data you tend to predict poorly for observations with a low value of variable ‘x1’ and the data with no labels has mostly low values of ‘x1’, then you should perhaps change your model.


LightGBM results differently depending on the order of the data I have two datasets A and B which are exactly the same in terms of the number of columns, name of columns and the values. The only difference is the order of those columns. I then train the LightGBM model on each of the two datasets with the following steps

Divide each dataset into training and testing (use the same random seed and ratio for both A and B)
Leave the hyperparameters as pretty much default
Set a random state as a fixed number (for reproduction)
Tune the learning_rate using a Grid Search
Train a LightGBM model on the training set and test it on the
testing set
Learning rate with the best performance on the testing set will be
chosen

The output models on the two datasets are very different, which makes me thinks that the order of columns does affect the performance of the model training using LightGBM.

Do you know why this is the case?

A possible explanation is this:

When the order of the columns differ, there is a little difference in the procedure.

What LightGBM, XGBoost, CatBoost, amongst other do is to select different columns from the features in your dataset in every step in the training.

The selections of these columns is done randomly: Let's say your dataset has 20 columns. The root node selects the features 1st, 3rd and 18th, on both datasets the 1st, 3rd and 18th features are different in both possible datasets. This is repeatedly done and in every step there is randomness affecting your ultimate result.


How to get accuracy, F1, precision and recall, for a keras model? I want to compute the precision, recall and F1-score for my binary KerasClassifier model, but don't find any solution.

Here's my actual code:

# Split dataset in train and test data
X_train, X_test, Y_train, Y_test = train_test_split(normalized_X, Y, test_size=0.3, random_state=seed)

# Build the model
model = Sequential()
model.add(Dense(23, input_dim=45, kernel_initializer='normal', activation='relu'))
model.add(Dense(1, kernel_initializer='normal', activation='sigmoid'))

# Compile model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

tensorboard = TensorBoard(log_dir="logs/{}".format(time.time()))

time_callback = TimeHistory()

# Fit the model
history = model.fit(X_train, Y_train, validation_split=0.3, epochs=200, batch_size=5, verbose=1, callbacks=[tensorboard, time_callback])

And then I am predicting on new test data, and getting the confusion matrix like this:

y_pred = model.predict(X_test)
y_pred =(y_pred>0.5)
list(y_pred)

cm = confusion_matrix(Y_test, y_pred)
print(cm)

But is there any solution to get the accuracy-score, the F1-score, the precision, and the recall? (If not complicated, also the cross-validation-score, but not necessary for this answer)

Thank you for any help!

Metrics have been removed from Keras core. You need to calculate them manually. They removed them on 2.0 version. Those metrics are all global metrics, but Keras works in batches. As a result, it might be more misleading than helpful.

However, if you really need them, you can do it like this

from keras import backend as K

def recall_m(y_true, y_pred):
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
    recall = true_positives / (possible_positives + K.epsilon())
    return recall

def precision_m(y_true, y_pred):
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
    precision = true_positives / (predicted_positives + K.epsilon())
    return precision

def f1_m(y_true, y_pred):
    precision = precision_m(y_true, y_pred)
    recall = recall_m(y_true, y_pred)
    return 2*((precision*recall)/(precision+recall+K.epsilon()))

# compile the model
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['acc',f1_m,precision_m, recall_m])

# fit the model
history = model.fit(Xtrain, ytrain, validation_split=0.3, epochs=10, verbose=0)

# evaluate the model
loss, accuracy, f1_score, precision, recall = model.evaluate(Xtest, ytest, verbose=0)


PCA shows overlapping boundaries, then why SVM performs best I am trying to understand which model might work for a given problem before trying the models, I find this case against my knowledge. Please guide what I am missing. I am new to Data Science.

Here is the graph which I got through PCA :

Now you can see the boundaries are very much overlapping.
The theory for SVM says that this model might work best with overlapping non linear data, which does not seems to be this case.

But still its able to identify all data in test set. So can you provide some clarity on why SVM performing good in this.

So my final results it is below order:

Logistic Regression and SVM are same (Accuracy Score : 1.0)
Random Forest (Accuracy Score : 0.9680851063829787)
KNN (Accuracy Score : 0.925531914893617)

other details :

feature set : 40
sample data : around 500

I assume you applied SVM to your initial data and use PCA only for visualization. I this case:

I guess your projection via the PCA is not showing the real picture.

You should check first how much of your data is explained with the first two principal components of the PCA. Your projection might change to much the structure of your data so that its not seperable anymore. In case the projection to the selected principal components is not changing your data too much, it could be that you maintain the seperability.

Finally, note:
If the projection is linearly seperable, then so is your data.
If the projection is not linearly seperable, you cannot conclude about seperability of your data.


What are key dataset requirements for topic models and word embeddings? I have a dataset of 2000 documents where avg doc size is 300 words. The vocab is dominated by domain-specific words.

My goal is to find similar documents. For this, I tried LDA, LSI, Doc2Vec (topics=100) but results are not great. LSI is better than the others in my dataset. I also tried word2vec (size=100) & word movers distance but again no luck. I am thinking of trying POS tags & then building some ontology model but not sure.

Are my results poor because of a bad dataset? What are some other techniques that I can try?

Similarity of documents can be done with varied approaches.

As your documents are based on domain based words, you could employ a tfidf representation for each document and compute similarity based on this.

Previously, I have used word2vec representations of words and constructed document vectors by taking an average of all the vectors ( as one of the approaches ) and did a cosine similarity between these vectors. Basically an n*n similarity computation.

When you do topic modelling, even though the idea is to find documents which have similar topic distribution, there is a prior to the approach i.e identifying number of topics. Just using randomly 100 topics without looking at the distribution of the words might lead you on a wrong path.

A very interesting approach I have recently come across is combining topic modelling and word vectors, here is the link to a blog by stitchfix : http://multithreaded.stitchfix.com/blog/2016/05/27/lda2vec/