How neural style transfer work in pytorch? I am using this pytorch script to learn and understand neural style transfer. I  understood most part of the code but having some hard time understanding some parts of the code.

In line 15 Its not clear to me how model_activations work. I made a sample style tensor of the shape style.shape -> torch.Size([3, 300, 374]) and tried this sample code first without layers dict.

x = style
x = x.unsqueeze(0)
for name,layer in model._modules.items():
    x = layer(x)
    print(x.shape)

Output:

torch.Size([1, 64, 300, 374])
torch.Size([1, 64, 300, 374])
torch.Size([1, 64, 300, 374])
torch.Size([1, 64, 300, 374])
torch.Size([1, 64, 150, 187])
torch.Size([1, 128, 150, 187])
torch.Size([1, 128, 150, 187])
torch.Size([1, 128, 150, 187])
torch.Size([1, 128, 150, 187])
torch.Size([1, 128, 75, 93])
torch.Size([1, 256, 75, 93])
torch.Size([1, 256, 75, 93])
torch.Size([1, 256, 75, 93])
torch.Size([1, 256, 75, 93])
torch.Size([1, 256, 75, 93])
torch.Size([1, 256, 75, 93])
torch.Size([1, 256, 75, 93])
torch.Size([1, 256, 75, 93])
torch.Size([1, 256, 37, 46])
torch.Size([1, 512, 37, 46])
torch.Size([1, 512, 37, 46])
torch.Size([1, 512, 37, 46])
torch.Size([1, 512, 37, 46])
torch.Size([1, 512, 37, 46])
torch.Size([1, 512, 37, 46])
torch.Size([1, 512, 37, 46])
torch.Size([1, 512, 37, 46])
torch.Size([1, 512, 18, 23])
torch.Size([1, 512, 18, 23])
torch.Size([1, 512, 18, 23])
torch.Size([1, 512, 18, 23])
torch.Size([1, 512, 18, 23])
torch.Size([1, 512, 18, 23])
torch.Size([1, 512, 18, 23])
torch.Size([1, 512, 18, 23])
torch.Size([1, 512, 18, 23])
torch.Size([1, 512, 9, 11])

and then with layers

layers = {
    '0' : 'conv1_1',
    '5' : 'conv2_1',
    '10': 'conv3_1',
    '19': 'conv4_1',
    '21': 'conv4_2',
    '28': 'conv5_1'
    }

x = style
x = x.unsqueeze(0)
for name,layer in model._modules.items():
    if name in layers:
      x = layer(x)
      print(x.shape)

Output:
torch.Size([1, 64, 300, 374])
torch.Size([1, 128, 300, 374])
torch.Size([1, 256, 300, 374])
torch.Size([1, 512, 300, 374])
torch.Size([1, 512, 300, 374])
torch.Size([1, 512, 300, 374])

My question is how the second method maintained the height and widght of style tensor (300, 374)?

The second confusion is in the line 91 where optimizer was set as optimizer = torch.optim.Adam([target],lr=0.007). In most pytorch tutorials I have seen them doing some thing like this optimizer = torch.optim.Adam(model.parameters(), lr = 0.01)

Why is the optimizer initialization in neural style transfer different from other neural network tutorials?
What is reason behind this optimizer = torch.optim.Adam([target],lr=0.007) >

Question 1

  My question is how the second method maintained the height and widght of style tensor (300, 374)?

Applying the second piece of code does not call all layers but only the ones listed in the layers dict. In contrast to that the first method runs through all layers. Now, if you look at the network architecture:

IN: model
OUT:
Sequential(
  (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (1): ReLU(inplace=True)
  (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (3): ReLU(inplace=True)
  (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (6): ReLU(inplace=True)
  (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (8): ReLU(inplace=True)
  (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (11): ReLU(inplace=True)
  (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (13): ReLU(inplace=True)
  (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (15): ReLU(inplace=True)
  (16): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (17): ReLU(inplace=True)
  (18): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (19): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (20): ReLU(inplace=True)
  (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (22): ReLU(inplace=True)
  (23): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (24): ReLU(inplace=True)
  (25): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (26): ReLU(inplace=True)
  (27): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (29): ReLU(inplace=True)
  (30): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (31): ReLU(inplace=True)
  (32): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (33): ReLU(inplace=True)
  (34): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (35): ReLU(inplace=True)
  (36): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
)

You can see that in fact all max pooling layers are being skipped with the second method. Accordingly, your input does not get pooled, i.e. height and width remain unchanged. (the CNN layers in this network do not change height and width but only the channels)

If you slightly change the second piece of code to:

x = style
x = x.unsqueeze(0)
for name,layer in model._modules.items():
    x = layer(x)
    if name in layers:
        print(x.shape)

It runs through all layers again and gives:

torch.Size([1, 64, 300, 374])
torch.Size([1, 128, 150, 187])
torch.Size([1, 256, 75, 93])
torch.Size([1, 512, 37, 46])
torch.Size([1, 512, 37, 46])
torch.Size([1, 512, 18, 23])

Question 2

  Why is the optimizer initialization in neural style transfer different from other neural network tutorials?

In image style transfer learning you do not optimize the weights (model parameters) but the target image. Since you would like the target image to resemble your content and style pictures as close as possible. All weights (model parameters) are being kept constant. Therefore, the optimizer needs to receive the target image as an input and provide updates for it (and not for your model parameters like usually).

Question 3

  What is reason behind this optimizer = torch.optim.Adam([target],lr=0.007) >

See Question 2. The target is initialized in line 69. Typically, you use the content image as a start:

target = content.clone().requires_grad_(True).to(device)

As you can see it also activates the gradient for the target.
And this target image is then being optimized in order to minimize the loss
total_loss = content_wt*content_loss + style_wt*style_loss.

A good read on the general approach is Image Style Transfer Using Convolutional Neural Networks.


How can Time Series Analysis be done with Categorical Variables Most of the time series analysis tutorials/textbooks I've read about, be they for univariate or multivariate time series data, usually deal with continuous numerical variables.

I currently have a problem at hand that deals with multivariate time series data, but the fields are all categorical variables. Hence, I was wondering if there is any way to use the standard time series analysis techniques (such as ARIMA, ARMA etc.)

Specifically, my data is a stream of alert data, where at each time stamp, information such as the alert monitoring system, the location of the problem etc. are stored in the alert. These fields are all categorical variables.

By definition time-series ARIMA models assume that, given a numerical observation at time $t-k$, the value of the numerical variable $X$ at time $t$ can be approximated as
$$
X_t = \sum_{j=1}^p a_j X_{t-j} + \varepsilon_t + c
$$
where $\varepsilon$ is a white noise error term and the $a_j$ are parameters to be determined. The idea is that the numerical variable $X$ at time $t$ only depends on some of its values at previous times; as you can see, by construction the above works for numerical variables only. Then one introduces some more conditions about moving averages and deviations to be verified and is able to prove that, under such conditions, the form of the coefficients $a_j$ can be determined.

The standard way to deal with categorical variables in these cases is to use one-hot encoding, namely you introduce dummy variables for each level of your category and fit against the dummy being 1 or 0, according to whether such category is present or not at time $t-k$. A similar question was asked here and you might want to have a look.

Another thing that you may want to ask yourself is whether time series is really what you are looking for, rather than just any classification model that provides a prediction given a set of categories and a seasonal variables.


Features selection in imbalanced dataset I have some doubts regarding an analysis. I have a dataset with class imbalance. I am trying to investigate some information from that data, e.g., how many urls contain http or https protocols.
My results are as follows:
http in dataset with class 1: 10
http in dataset with class 0: 109
https in dataset with class 1: 180
https in dataset with class 0: 1560

I am trying to build a classifier based on some features and the presence of protocols was supposed to be taken into account.
However, on the basis of the above results, what do you think I should say?
Does it make sense to say that the most websites having class 0 have an https protocol, even if I have a dataset with class imbalance?
For a model, I would consider resampling techniques. Should I work on this analysis (so make this conclusion) after the resampling, or it would make sense to check features importance with other tests (e.g., Pearson correlation, if it is appropriate in this case)?
Any suggestion would be greatly appreciated it.

What this shows is that the protocol is not a very discriminative feature:

the probability of class 1 given http is 10/(109+10)=0.084
the probability of class 1 given https is 180/(180+1560)=0.103

If these conditional probabilities were very different this feature would be more helpful to predict the class, but they differ only slightly. Note that the feature might still be useful, but it doesn't have a huge impact on its own. In case you're interested to know if the difference is significant (i.e. not due to chance), you could do a chi-square test.

Does it make sense to say that the most websites having class 0 have an https protocol, even if I have a dataset with class imbalance?

It is factually correct, but most websites having class 1 also have https so it's not a very useful information (and on its own this information might be confusing for some readers).

For a model, I would consider resampling techniques. Should I work on this after the resampling, or it would make sense to check features importance with other tests (e.g., Pearson correlation, if it his appropriate in this case)?

Feature selection can done either before or after resampling, it doesn't matter. The two things are independent of each other because the level of correlation between a feature and the class is independent from the proportion of the class.
I don't think Pearson correlation is good for categorical variables. I think conditional entropy would be more appropriate here (not 100% sure, there might be other options).


How many normalization methods are there and what are they for? this post lists 5 types of normalization.

Zscore
MinMax
Logistic
LogNormal
TanH

is there any other types of normalization that are most commonly used in machine learning?

StandardScaler

RobustScaler for outliers-heavy dataset(can handle better outliers than other scalers)

to name a few..


LSTM to multivariate sequence classification How can I train multivariate to multiclass sequence using LSTM in keras?

I have 50000 sequences, each in the length of 100 timepoints.
At every time point, I have 3 features (So the width is 3).

I have 4 classes and I want to bulid a classifier to determine class for sequence.
What is the best way to do so?

I saw many guides for univariate sequence classification but none for multivariate, and I don't know how to apply this on the multivariate case

Since you are using LSTMs for classification using the
multivariate time series data, you need to model your time-series data into a supervised learning problem and specify the previous
time steps you need to look before by specifying the time-lag count

You need to look into the to_supervised function and specify the number of outputs your model has. In your case, it is 4.

Split your train and test set from the whole set of data. A 70:30 ratio for the train, test would be a good start.

Also, note that you need to scale your values using the
sklearn.preprocessing.MinMaxScaler() function.

You need to reshape your train and test values into (batch_size/sample_size, time_steps, feature_size) as an LSTM Layer in Keras expects your data to be fed in a 3D array format.

For eg: Your training shape would be train.shape = (batch_size, 100, 3)

And for the model building in Keras
model = Sequential()
model.add(LSTM(number_of_hidden_units, activation='relu', input_shape=(n_timesteps = 100, n_features = 3)))
model.add(Dense(4, activation='softmax')) #since number of output classes is 4
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.summary()
model.fit(X_train, y_train, validation_data=(X_test, y_test), no_of_epochs, batch_size)

Note that I have just given a rough outline of the model building and left out the hyperparameters at your convenience. You can either stack more LSTM layers onto the model or tune the number of hidden_layers in the model Refer Multi-Class Classification for more details on it.


Is it possible to change misclassification cost in caret? 0
down vote
favorite

I have an imbalanced dataset to work with (with about 20-fold positive examples than negative). I know several solutions to deal with this type of data (under/oversampling, optimizing for AUC, etc.) but I would like to try changing the misclassification costs for the two groups. However, I cannot figure out the proper way to do this in caret. Do you have any idea how to do this?

Thanks for any help!

I would have to read more carefully: train has a weights parameter that do this.


Loss of MSE always be 0 when keras for topic predict my input is a 200 dims vector, which is generated by mean of the word2vector of all words of a article, my output is a 50 dims vector,which is generated by the LDA results of a article I want to use mse as the loss function,but the value of the loss always be 0 my code as follows:

model = Sequential()
model.add(Dense(cols*footsize, 400,init = "glorot_uniform"))
model.add(Activation('relu'))
model.add(Dropout(0.2))
model.add(Dense(400, 400,init = "glorot_uniform"))
model.add(Activation('relu'))
model.add(Dropout(0.2))
model.add(Dense(400, 50,init = "glorot_uniform"))
model.add(Activation('softmax'))
model.compile(loss='mse', optimizer='rmsprop')

And regardless of the number of epochs, the loss is always 0

First, is your output a one-hot vector of predicted classes? i.e: class one is [1, 0, 0, ...] and class two is [0, 1, 0, 0, ...].

If so, then using softmax activation at the output layer is acceptable and you are doing a classification problem. If you are doing a classification problem (one-hot output) you cannot use MSE as the loss, use categorical cross-entropy.

Softmax scales the output so that the number given is a predicted probability of a certain class. Wikipedia here: https://en.wikipedia.org/wiki/Softmax_function

If you are expecting the output vector to be real numbers then you need to use linear activation on your output neurons.


Get an equation from the relationship among variables I have an agricultural dataset of crop yield (in kilograms) under environmental conditions such as humidity, rainfall, sunlight over few years. I want to find out relationship between the environmental factors and the crop yield.

A score is assigned based on the crop yield each term.

The closer the environmental conditions to the ideal conditions, higher the crop yield.
The highest crop yield under ideal conditions is scored as 10.
For non-ideal environmental conditions, the score can be between 1-10.

The output I'm expecting is an equation which anyone can plug-in the environmental conditions, and get a score between 1-10 on how ideal the conditions are.

Could you help me on choosing an approach to achieve this?

This seems like a problem that can be fixed with any regression problem. Here are models I would recommend to get started with:

If you need your model to be explainable (i.e. show which factors contributed to the score most):

Decision trees / random forests
Linear regression

If you only care about giving a final score

Essentially any other model, which can include:

Support vector machines
Neural Networks
K-Nearest Neighbours
Gaussian Processes
and more


Parallel active optimization I'm trying to optimize an expensive function for which I can choose sample points. The difficulty is that many function evaluations may be computed in parallel, taking varying amounts of time. I don't know which keywords to search for to find existing literature(/implementations).

So at a time, I might have already computed function values at 18 points, with 15 still being computed, and I want to start evaluating the function another point. Without the running jobs, I could make a model and find which next point might provide the most information. But now, I need to somehow tell the model that there are also 15 points for which I don't have values, but near which I don't want to evaluate the function.

Specifically, I'm looking for something with minimal assumptions and no gradient information. But I'll be happy if I just know the keyword(s) to search for. (I can come up with several hacks to that kind of work, but I wonder if there is any 'real' solution).

For illustration, say every column is a function evaluation point and every row is a timestep in the below ascii-figure. An X means the evaluation is still being computed and a * in the bottom row means it's done. A new evaluation starts as soon as an old one terminates, with others still busy.

XXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXX
 XXXXXXXXXXX XXXXX
 XXXXXXXXXXX XXXXX
 XXXXXXXXXXX XXXXX
 XXXXXXXXXXX X  XXX X
 XXXX XXXXXX X  XXX XX
 XXXX XXXXXX X  XXX  XX
 XXXX XXXXXX X  XXX  XX
 XXXX XXXXXX X  XXX  XX
 XXXX XXXXXX X  XXX  XX
 XXXX X XXXX X  XXX  XXX
 XXXX X XXXX X  X X  XXXX
 XXXX X XXXX X  X X  XXXX
  XXX X XXXX X  X X  XXXXX
  XXX X XXXX X  X X  XXXXX
  XXX X XXXX X  X X  XXXXX
  XXX X XXXX X  X X  XXXXX
  XXX X XX X X  X X  XXXXXX
  XXX   XX X X  X X  XXXXXXX
  XXX   XX X X  X X  XXXXXXX
  XXX   XX X X  X X  XXXXXXX
  XXX   XX   X  X    X XXXXXXXX
  XXX   XX   X  X    X XXXXXXXX
  XXX   XX   X  X    X XXXXXX XX
  XXX   XX   X  X    X XXXXXX XX
  XXX   XX   X       X XXXXXX XXX
  XX    XX   X       X XXXXXX XXXX
**  ****  *** ******* *      *

Each time an evaluation completes, how to choose the next point to evaluate, given several complete and several incomplete evaluations?

SMAC has a parallel version available.

I'm not sure it handles the single runs as you want, but it sounds strange to me if it don't. Give a try!


Discrepancy between training set and real-world data set: domain adaptation? I have read in literature that in some cases the training set is not representative for a real-world dataset. However, I cannot seem to find a proper term describing this phenomenon; what is the proper term to address this problem?

Edit:

So far I have settled for the term domain adaptation, shortly described as a field in machine learning which aims to learn from a certain data distribution in order to predict data coming from a different (but related) target distribution.

You may be looking for sampling bias. Also the other case (where training set does in fact represent the real world data-set well) is generally known as representative sample.

Hope this helps.


What is the layer above/below in a NN? In the lecture notes of CS231n, it says (emphasis mine)

  ... There are three major sources of memory to keep track of:

  From the intermediate volume sizes: These are the raw number of activations at every layer of the ConvNet, and also their gradients (of equal size). Usually, most of the activations are on the earlier layers of a ConvNet (i.e. first Conv Layers). These are kept around because they are needed for backpropagation, but a clever implementation that runs a ConvNet only at test time could in principle reduce this by a huge amount, by only storing the current activations at any layer and discarding the previous activations on layers below.
  ......

Normally we imagine a neural network propagating from left (input) to right (output), but what is the layer above/below the current layer? Is the input the bottom layer or the top layer?

I guess output is the top layer. This is because we only need the gradients of the next (closer to output) layer to compute those of the current layer. The exact activations of the next layer are not required, so we can discard them after the corresponding gradients have been computed. This is possible because we compute gradients backwards (last layers first, first layers last).

Normally we imagine a neural network propagating from left (input) to right (output), but what is the layer above/below the current layer? Is the input the bottom layer or the top layer?

Yes, a "below" layer means an "earlier" layer, i.e. closer to the input. For CNNs it is quite common to speak of lower layers, lower level feature maps etc.

And for RNNs you can even take this literally as they are usually visualized bottom-to-top instead of left-to-right (e.g. see figures here).


Deep Learning Project to Predict Stock Prices So I have a background in computer programming and a little in machine learning in general.  What I would like to do is create a fun project in A.I. with deep learning.

I have a dataset that has a whole bunch of stock prices at a certain date, with a bunch of features for each entry to go with it.  I also have some "experts" who made predictions on whether the stock will go up or down. As my dataset grows I can evolve the game to make selections from multiple stocks...etc

Essentially what I would love to do is create an A.I. app that will be fed the same data that the "experts" had and see if I can create something more accurate and beat them at it.  Is this a viable approach?

Essentially what I would love to do is create an A.I. app that will be fed the same data that the "experts" had and see if I can create something more accurate and beat them at it. Is this a viable approach?

Sure, you can use one or more supervised learning techniques to train a model here. You have features, a target variable and ground truth for that variable.

In addition to applying ML you have learned, all you need to do to test your application fairly is reserve some of the data you have with expert predictions for comparison as test data (i.e. do not train using it).

I would caveat that with some additional thoughts:

You haven't really outlined an "approach" here, other than mentioning use of ML.
Be careful not to leak future data back into the predictive model when building a test version.
Predicting stock and markets is hard, because they react to their own predictability and many professional organisations trade on the slightest advantage they can calculate, with experienced and highly competent staff both gathering and analysing data.

Not directly part of the answer, but to anyone just starting out and discovering machine learning, and finding this Q&A:

Please don't imagine rich rewards from predicting markets using stats at home, it doesn't happen. If you think that this is a route to "beating the market" be aware that you are far from the first to think of doing this, and such a plan can be summarised like this:

Market Data + ML
???
Profit

You can fill in the ??? by learning loads about financial markets - i.e. essentially by becoming one of the experts. ML is not a short-cut, but it might be a useful tool if you are, or plan to be, a market analyst.


Which data set to use to find correlation between Predictor and response variables? Test data set? Training data set? or the entire data set? I have a dataset with 4 predictor variables X1, X2, X3, X4, and one response variable Y. I have been asked to check the correlation between these variables and see how they are related and then use linear model to fit them.

No split of training set:test set is given. I have one data set with 10000 samples. I was planning of splitting this data set in the ratio 80:20 for training and testing respectively.

Now I have a doubt on whether the correlation should be found after the data has been split or is it better to check correlation with the entire dataset? Which is the std way of doing?

P.S. I am going to use the R programming to do the same.

This depends on your research question. Do you want to make predictions? - Then you need to split your data set in training and testsamples.

However, if you are more interested in answering the following question: What impact does X1, X2, X3 and X4 have on Y, respectively? Then you are interested in estimation. For this, you do not need to split your sample, but have to test your data set for underlying model assumptions (e.g. heteroscedasticity, autocorrellation of the residuals etc.) to get unbiased/accurate estimations.

For OLS (linear regression) these model assumptions follow the Gauss-Markov-Theorem.

Most statistical tests for model assumptions are already implemented in R:

Test for autocorrelation - Breusch-Godfrey - bgtest
Test for heteroscedasticity - White's Test - het.test
Test for normality - Jarque Bera Test - jarque.bera.test

However, even the tests make some assumptions about your data - but these tests are the most common ones.

UPDATE - Example for clarification

According to your comment. Imagine this real-life problem. You want to build a model that can predict the chance of cancer (y), using variables like age, blood pressure, weight, cholesterol value etc.. (X1, X2, X3, X4)

To assess the nature of the variables and their relationships you do descriptive analysis (Mean, Variance etc.) + Correlation analysis.

Let's have a look at the following data. (Yellow records would be your test samples - the data you DON'T have in real life, as those patients haven't seen the doctor yet).

As you can see the mean and variance can differ already a lot depending on the fact if you include test samples or not in your calculations.

Now we are coming to the correlation part - the relationship between the independent variables or also called predictors

I highlighted the relationship between weight and cholesterol. Without the test sample, it seems like weight and cholesterol are uncorrelated or have only a slightly positive relationship. If we add the test data it turns out that the correlation turns negative.

Question: If the correlation between your variables would have an impact on your choice of variable selection for your model, would it make sense to include the test-sample in your correlation analysis? Especially, knowing that this data is not available in real life yet.

Remember Model Estimation: if you build a multivariate regression model mean, variance and covariance are used to find the best parameters to estimate your dependent variable (Cancer). So a model that was already trained with all available data is likely to make better predictions as it has already seen the data it should predict.

Summary

Whenever you plan to make predictions with a model you should work as closely to real-life assumptions. So you split your data in train - test data. You ònly use the training data to perform all your tests and checks and you will not touch the test set before making a prediction.


Neural Networks getting stuck at local optima I'm training a NN with 8 features and 8000 training examples with a single output (0, 1) using the scipy.optimise CG algorithm and the results are somewhat inconsistent. The goal is to get the NN to be as 'precise' as possible (recall doesn't really matter too much) so I've set the threshold for y value quite high (0.75). Most of the time it gets a precision of around 80%, however sometimes it fails (using exactly the same parameters, lambda etc..) to generate any outputs which are above the 0.75 threshold, meaning the precision equals 0.

I've successfully trained NNs before without these unusual results (albeit the goal was a somewhat more conventional multi-class classifier with many more features).

I'm wondering if the training NNs with fewer features increases the chances of it getting stuck at a local optima; or getting stuck at local optima has a more significant impact on NNs with fewer features?

Any thoughts on what's going on!?

Ok all sorted - Bit embarrassing but forgot to normalise the data!


How to load a saved model in TensorFlow? This is my code in Python:

from __future__ import absolute_import, division, print_function, unicode_literals
import tensorflow as tf
from matplotlib import pyplot as plt
import numpy as np

I checked if the saved model is there using the following code:

tf.compat.v1.saved_model.contains_saved_model(
    '/Link_to_the_saved_model_directory/'
)

which returns True
and I can use the following code to further make sure the model is saved correctly, as  far as I understood:

tf.saved_model.Asset(
    '/Link_to_the_saved_model_directory/'
)

which returns this:

<tensorflow.python.training.tracking.tracking.Asset at 0x2aad125e5710>

So, everything looks fine. But, when I use the following script to load the model I get an error.

LoadedModel = tf.saved_model.load(
    export_dir='/Link_to_the_saved_model_directory/', tags=None
)

Error:

OSError: Cannot parse file b'/Link_to_the_saved_model_directory/saved_model.pbtxt': 1:1 : Message type "tensorflow.SavedModel" has no field named "node"..

The files in the /Link_to_the_saved_model_directory/ look like this:

['saved_model.pbtxt',
 'saved_model.ckpt-0.data-00000-of-00001',
 'saved_model.ckpt-0.meta',
 'checkpoint',
 'saved_model.ckpt-0.index']

Any suggestion is greatly appreciated on how to load the model so that I could reuse it for transfer learning. It might also be the case that the model is partly written in a previous version of TensorFlow (e.g. TensorFlow 1.x) and this error is thus due to compatibility issue, but I could not find a solution for that yet.

Update: I tried the following method but it does not work (tf was imported using the compatible version import tensorflow.compat.v1. as tf):

with tf.Session() as sess:
    saver = tf.train.import_meta_graph('/dir_to_the_model_files/saved_model.ckpt-0.meta')
    saver.restore(sess, "/dir_to_the_model_files/saved_model.ckpt-0")
    loaded = tf.saved_model.load(sess,tags=None,export_dir="/dir_to_the_model_files",import_scope=None)

It returns the following warnings and errors:

WARNING:tensorflow:The saved meta_graph is possibly from an older release:
'metric_variables' collection should be of type 'byte_list', but instead is of type 'node_list'.
INFO:tensorflow:Restoring parameters from /dir_to_the_model_files/saved_model.ckpt-0
<tensorflow.python.training.saver.Saver object at 0x2aaab4824a50>
WARNING:tensorflow:From <ipython-input-3-b8fd24f6b841>:9: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.

OSError: Cannot parse file b'/dir_to_the_model_files/saved_model.pbtxt': 1:1 : Message type "tensorflow.SavedModel" has no field named "node"..

The TensorFlow documentation for tf.saved_model.load might help:

  SavedModels from tf.estimator.Estimator or 1.x SavedModel APIs have a
  flat graph instead of tf.function objects. These SavedModels will have
  functions corresponding to their signatures in the .signatures
  attribute, but also have a .prune method which allows you to extract
  functions for new subgraphs. This is equivalent to importing the
  SavedModel and naming feeds and fetches in a Session from TensorFlow
  1.x.

You might have to use deprecated v1 api call
https://www.tensorflow.org/api_docs/python/tf/compat/v1/saved_model/load


How is this score function estimator derived? In this paper they have this equation, where they use the score function estimator, to estimate the gradient of an expectation. How did they derive this?

This is simply a special case (where $p_\psi = N(0,1)$) of the general gradient estimator for Natural Evolution Strategies (proved in another reference, look it up):

Outline of derivation based on the general formula for the gradient estimator:
$$\nabla_\psi E_{\theta \sim p_\psi} \left[ F(\theta) \right] = E_{\theta \sim p_\psi} \left[ F(\theta) \nabla_\psi log({p_\psi}(\theta)) \right]$$
If
$$\epsilon \sim \mathbb{N}(0, 1) = \frac{1}{\sqrt{2 \pi}}e^{-\frac{\epsilon^2}{2}}$$
then
$$\psi = \theta + \sigma \epsilon \sim \mathbb{N}(\theta, \sigma) = \frac{1}{\sigma\sqrt{2 \pi}}e^{-\frac{(\psi-\theta)^2}{2\sigma^2}}$$
Thus: $\psi = \theta + \sigma \epsilon \sim \mathbb{N}(\theta, \sigma) \Longleftrightarrow \epsilon = \frac{\psi-\theta}{\sigma} \sim \mathbb{N}(0,1)$
So:
$$\begin{align}
\nabla_\theta E_{\psi \sim N(\theta,\sigma)} \left[ F(\theta + \sigma \epsilon) \right] &= E_{\psi \sim N(\theta,\sigma)} \left[ F(\theta + \sigma \epsilon) \nabla_\theta (-\frac{(\psi-\theta)^2}{2\sigma^2}) \right] \\
&= E_{\epsilon \sim N(0,1)} \left[ F(\theta + \sigma \epsilon) \nabla_\epsilon (-\frac{\epsilon^2}{2}) \frac{d(\frac{\psi-\theta}{\sigma})}{d\theta} \right] \\
&= \frac{1}{\sigma} E_{\epsilon \sim N(0,1)} \left[ F(\theta + \sigma \epsilon) \epsilon \right] \\
&= \nabla_\theta E_{\epsilon \sim N(0,1)} \left[ F(\theta + \sigma \epsilon) \right]
\end{align}$$
note: scalar variables were considered in above steps for simplicity, but easy to extend/derive for vector variables


How to preprocess data for image classification from a .txt file? Basically my issue is that im building an image classification model using AlexNet. I have this pre-split dataset thats already split into training, test, validation. However the issue is that these splits are in .txt files (e.g. trainingsplit.txt) and inside the .txt file is a list of image_001.png files etc. How do I extract this data and pass it through my preprocessor? This is what I would normally do for preprocessing:

train_dir = '/content/gdrive/My Drive/alexNet/Training'
validation_dir = '/content/gdrive/My Drive/alexNet/Validation'

train_datagen = ImageDataGenerator(
      rescale=1./255,
      rotation_range=40,
      width_shift_range=0.2,
      height_shift_range=0.2,
      shear_range=0.2,
      zoom_range=0.2,
      horizontal_flip=True,
      fill_mode='nearest',
      )

test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
        train_dir,  # This is the source directory for training images
        target_size=(224, 224),
        batch_size=20,
        class_mode='binary')

validation_generator = test_datagen.flow_from_directory(
        validation_dir,
        target_size=(224, 224),
        batch_size=20,
        class_mode='binary')

Edit: Content of training.txt

image_001.jpg
image_002.jpg
image_014.jpg
image_017.jpg
etc.

The training data is a .txt file containing several rows of text, which are the names of different images. This txt file is inside the main directory Dataset, which contains class directories like Apples, Bananas, Oranges, etc.

I am assuming from the description inside main directory there are more directories of  Apples, Bananas, Oranges, etc. and inside them you have .txt files containing information about the images.

import shutil

with open(file_path, 'r') as f:
    img_names = f.readlines()
    img_names = [img.strip() for img in img_names]

for i in range(len(img_names)): shutil.move(img_names[i], folder_path)

Once images are moved into there respective folder you can use ImageDataGenerator approach to train model.


Compute utility of a notification I have no idea whether this is the right StackExchange flavor to post this question in, but here goes:

I have an agent (an IRC bot that listens to an event stream) that I would like to add "intelligent notifications" to. Let me give a specific example.

The bot monitors the results from our continuous integration system, so it basically sees a stream of test results associated with changes (and the changes are associated with users.) If I ask it to notify me of interesting events, then for each event it sees, it decides whether to tell me about it (or rather, about the current results so far.)

I would like it to decide based on how interesting something is for me. Something is interesting if (1) it is a result of a change that I pushed to the continuous integration system, (2) it is a test failure, and (3) the information conveyed by that failure is a significant indicator of whether my change was bad.

Tests have a baseline probability of failing for no good reason ("intermittent failures"). If one test fails, it does not necessarily mean my change was bad. It might just be a flaky test. (Any given push tends to result in a couple bogus failures, so this isn't some obscure edge case.) But the baseline probability of an intermittent failure varies according to the test suite running (we have several dozen different test suites that fire off for every change pushed.)

So I don't want to be bothered with reports of failing tests that are probably just noise. And if my push is bad, I don't want to be flooded with notifications for every failed test. (If I break something, it'll probably show up in multiple test suites. So if the agent sees two failures and 20 successes, maybe it won't bother me, but if it then sees another failure or two it should.)

I am imagining I could look at the change between the prior and posterior probabilities and use that to estimate the entropy of that new test result, and perhaps also compute my personal entropy (if that makes any sense -- as in, the bot should assume I don't know what any of the results are until it notifies me, at which point it should assume I am aware of the current full set of results and not bother me again until enough additional results have come in to substantially change the probability estimate of my change being bad.)

Or something like that. I'm really looking for the right mathematical framework to compute things like this. I know next to nothing about machine learning or... well, mathematics in general.

(The above is simplified; I would additionally like to do crazy things like take into account "labeling", where a third party looks at the test failures and decides whether they are real failures or not. But the time between the failure coming in and when that person labels it for me can also be modeled by a distribution, and I'd like to hold off notifications for interesting results if there's a good chance that this other person may tell me that the result is not interesting after all. But only if that's going to happen "soon". Also, we have an automated system for guessing whether something might be an intermittent vs real failure, and it'd be nice to take its opinion into account as well.)

Good ideas there. But start with simple things first and then ramp up to a more complex system once you have something already working and some human feedback on the solution.

Simpler solution is just a velocity check - if there is X number of failures per Y number of seconds, then send the notification. From your post it seems like 3 failures over 20 second time window should be a good start. It takes a fair bit of tinkering to get this kind of system running - work out the windowing functions, etc.

Good luck :)


Optimization of a custom loss function I want to implement a deep learning model in Keras, but I want to use my own loss function, i.e. custom loss. If I implement some loss function and use Keras Functional API for the model, do I need to change the way the optimizer works, because the optimizer will minimize my loss function? If I need to do that, what is the way to do that?

You don't need to change the way optimizer works. You just need to define your loss function in some standard way.

Checkout the answer in this post to have an idea on an example of customizing loss function in Keras.

https://stackoverflow.com/questions/45961428/make-a-custom-loss-function-in-keras


How many epochs does fit method run So I wrote Linear regression from scratch using y = mx+b and ran the algorithm for 50 epochs (times) to minimize the cost and get the best parameters.

When I use Scikit Learn, I just call the Linear Regression method and fit the data-set to it then start predicting. How many epochs does fit method run ? Not only for Linear Regression but also other ML methods in general.

In scikit-learn's linear regression, the parameters that minimise the squared error loss aren't estimated using gradient descent, they are computed exactly.

The minimisation problem for linear least squares is

$$
\hat{\beta} = \underset{\beta}\arg \min || \mathbf{y} - \beta \mathbf{X} ||^2
$$

which has a unique solution (assuming the columns of $\mathbf{X}$ are linearly independent):

$$
\hat{\beta} = (\mathbf{X}^\text{T}\mathbf{X})^{-1}\mathbf{X}^\text{T}\mathbf{y}
$$

For classifiers that are fitted with an iterative optimisation process like gradient descent, e.g., MLPClassifier, there is a parameter called max_iter which sets the maximum number of epochs. If tol is set to 0, the optimisation will run for max_iter epochs.


Can Gaussian Process be fit incrementally? I am using Gaussian Process Regressor to fit data for a Bayesian Optimiser. This is a relevant part of my Python code.

from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import (RBF, Matern, RationalQuadratic,
                                          ExpSineSquared, DotProduct,
                                          ConstantKernel)
kernel = Matern(length_scale=lenSc,nu=2.5)
gp = GaussianProcessRegressor(kernel=kernel)
l2=10
Xi=np.concatenate((nprand.uniform(10.0,30.0,[l2,6]),nprand.uniform(-0.3,0.3,[l2,42]),nprand.uniform(0.0,30.0,[l2,1]),nprand.uniform(-30.0,30.0,[l2,1]),nprand.uniform(0.0,0.3,[l2,6]),nprand.uniform(5.0,20.0,[l2,6])),axis=1)
yi=map(lambda x: f1(x), Xi)
gp.fit(Xi, yi)
while(1):
   #Some code to get the new values- cc and nv
   Xi=numpy.row_stack((Xi,cc))
   yi=np.concatenate((yi,[nv]))
   gp.fit(Xi, yi)

Each iteration of the Bayesian Optimiser, I add a new element to $Xi$ and $yi$ and fit $gp$, the Gaussian Process, again. As the size of $Xi$ and $yi$ reach over 200, the time taken to fit the data becomes noticeable, this decreases the overall efficiency of my Bayesian Optimiser.

The Bayesian Optimiser has to make around 1000 iterations, each iteration needs to fit the data all over again. So, I was wondering if there's a way to incrementally fit the data? If that's not already being done by sklearn's GaussianProcessRegressor.

Thank you!

I think with GPR, the computational cost increases on an order of N^3, so the amount of time with more data increases drastically.

I don't think the current version of sklearn's GPR implements the kind of incremental fit you are expecting. With larger data and iterations, I suggest implementing your own version on a GPU or somethings since GPR implementation is not that difficult.


Restrictions on my skewed validation data I have a severely skewed data sets consisting of 20 something classes where the smallest class contains on the order of 1000 samples and the largest several millions.

Regarding the validation data, I understand that I should make sure that it represent a similar ratio between classes compared to the one in my original raw data. Hence, I shouldn't do any under- or over-sampling on that validation data, but can do it on the training data.

Because I have such greatly skewed data set, is it still viable to add some restriction to the selection of my validation data set? Say I want there to be at least 1000 samples from each class in order to accept it, as I want to have a reasonable accuracy on the metrics of all classes.

Would this ruin my validation as the ratio between the largest and smallest class could then go from ~0.01-0.1% to ~1.0%, or is it still safe as the validation data still is significantly skewed?

I suggest using your whole validation set but providing class-specific metrics. E.g., AUC only considering validation rows with targets in a single class. That will help you see if you're underperforming for a specific class. You can then say "This model earned 0.xxx AUC on the whole validation set and no worse than 0.yyy for any single class."


Concrete example to elaborate on the relationship between `linear function` and `aﬃne function` in the context of machine learning I am learning the deeplearningbook

Section 5.1.4 says:

  so the mapping from parameters to predictions is still a linear
  function but the mapping from features to predictions is now an aﬃne
  function

Can any one give a concrete example to elaborate on the relationship between linear function and aﬃne function in the context of machine learning?

There's a great answer on the Mathematics stack exchange about the distinction between linear and affine functions.

  A linear function fixes the origin, whereas an affine function need not do so. An affine function is the composition of a linear function with a translation, so while the linear part fixes the origin, the translation can map it somewhere else.

In section 5.1.4, the book introduces linear regression:
$$\hat{y} = \pmb{\omega}^{T} \pmb{x} + b$$

In this example, the $\hat{y} = \pmb{\omega}^{T} \pmb{x}$ part is a linear function.  It has to pass through the origin.  The addition of a vertical translation $b$ causes $\hat{y}$ to become an affine transformation, since it no longer has to pass through the origin.

I think the authors point out this distinction because it's technically incorrect from a purely mathematical point-of-view to describe $$\hat{y} = \pmb{\omega}^{T} \pmb{x} + b$$ as a linear function of the parameters $\pmb{x}$.  It's actually an affine function, so "linear regression" may appear to be a misnomer if you have a strong background in linear algebra.

For what it's worth, I don't think this is very helpful information to include in the main text of an introductory chapter on machine learning.  If you still don't understand, then just keep reading! Don't let this block you from progressing through the book.


Experiment click to lead prediction with Azure ML I am experimenting now with the Azure ML Studio and I am trying to predict leads based on the clicks I have.

I am exporting a data set of 60.000 Clicks and 8.000 Leads from these clicks.

My data set:

With the "Edit Metadata" I have transformed all the features, except the hasLead in Categorical Feature.

I am testing as many combinations as I can with the data, but my "Recall" is very bad. Do you have some suggestions about what kind of missing data maybe can increase the Recall?

Here the evaluation of the model:

This looks like a two class boosted decision classifier.  That should perform quite well.  I used it several times about 2-3 months ago and got excellent results.  I'm guessing you are not training/testing the right features.  What is your target variable?  Is it 'hasLead'?  I'm not sure what will be statistically influential on that dependent variable, but you  can run a feature engineering exercise to see what independent variables have the most influence.  Run the code below; change to suit your specific needs (obviously, use your specific data).

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
# matplotlib inline

df = pd.read_csv("https://rodeo-tutorials.s3.amazonaws.com/data/credit-data-trainingset.csv")
df.head()

from sklearn.ensemble import RandomForestClassifier

features = np.array(['revolving_utilization_of_unsecured_lines',
                     'age', 'number_of_time30-59_days_past_due_not_worse',
                     'debt_ratio', 'monthly_income','number_of_open_credit_lines_and_loans',
                     'number_of_times90_days_late', 'number_real_estate_loans_or_lines',
                     'number_of_time60-89_days_past_due_not_worse', 'number_of_dependents'])
clf = RandomForestClassifier()
clf.fit(df[features], df['serious_dlqin2yrs'])

# from the calculated importances, order them from most to least important
# and make a barplot so we can visualize what is/isn't important
importances = clf.feature_importances_
sorted_idx = np.argsort(importances)

padding = np.arange(len(features)) + 0.5
plt.barh(padding, importances[sorted_idx], align='center')
plt.yticks(padding, features[sorted_idx])
plt.xlabel("Relative Importance")
plt.title("Variable Importance")
plt.show()


Why does MAE differ after prediction (Neural Network)? I'm having trouble understanding what's happening in the following code.
I already have defined x_train, y_train, x_val, y_val and x_test which define my training, validation and test sets. I'm using keras library.

model = Sequential()
model.add(Dense(8,input_dim=x_train.shape[1], activation='relu'))
model.add(Dense(1, activation='linear'))
model.summary()

rms = optimizers.RMSprop(lr=0.0003)

model.compile(optimizer=rms,
              loss='mean_squared_error',
              metrics=['mae'])
history = model.fit(x_train, y_train, epochs=1000, validation_data=(x_val, y_val), batch_size = 32)

pred_train = model.predict(x_train)
pred_val = model.predict(x_val)
pred_test = model.predict(x_test)

MAE_train = np.mean(np.absolute(y_train - pred_train ))
print('Mean absolute error for  training is MAE = '+str(MAE_train))
MAPE_train = 100 * np.mean(np.absolute(y_train - pred_train )/ y_train)
print('MAPE for training is MAPE = '+str(MAPE_train))
print(pred_val)
MAE_val= np.mean(np.absolute(y_val - pred_val ))
print('MAE  val is MAE = '+str(MAE_val))
MAPE_val = 100 * np.mean(np.absolute(y_val - pred_val )/ y_val)
print('MAPE for val is MAPE = '+str(MAPE_val))

here is the result of an execution :

What I don't understand is that the val_mean_absolute_error that is shown at the end of the training differs from the one that I compute.
At first I tried to save my best model using Callbacks and then predict with it but the values also differ in this case.

What am i missing ?

Your MAE_val is:

MAE_val= np.mean(np.absolute(y_val - pred_val ))

On the other side, you fit your model on:

history = model.fit(x_train, y_train, ...

So you are calculating them on different objects. Training data on one side, and validation set for a final evaluation.


How to predict the probability of an event? I have a dataset where a set of people donated for charity along with the dates of the donation. I have to find the probability of each donor donating in the next three months.

Data is available from August 2014 - February 2016. I have to predict the probability of each person donating for March-June 2016.

Any help would be appreciated?

Below is a snapshot of the data

id  date    amount
1   13-08-14    2485
1   21-11-14    2105
1   17-09-15    1359
2   13-08-14    2542
2   20-04-15    1276
2   12-10-15    2694
3   20-11-14    3556
4   28-07-15    3383
5   13-08-14    1698
5   11-12-14    1725
5   09-06-15    1376
5   17-09-15    3230

Regards

Please double-check if there's the only data you have got, because all you have is a single predictor date.

If this is indeed your only data source, then you only have a single predictor, and your independent variable is continuous. Now, you should plot date vs amount and fit a single linear regression. Does the fitting look good? Only you can tell because we don't have the full data-set.

If it's not a good fit, look at the plot and ask yourself does this look like a curve? If so, you might want to fit a spline curve or something like that.

You should also check the autocorrelation. This makes sense because your data look like a time series (you'll need to check it yourself). If this is the case, you might want to consider MA and ARCH model.

It's not possible for us to give you accurate advice because we don't know your data.


What is the best option to test the data set of images using weka? I have two hundred and fifty images, and extracted the features from them and put them in an Excel file, how to use the weka program so that the first 200 images for training and the remaining fifty images for testing.

Do I put the data for two hundred images in a separate file and the data for fifty images in another file? Or do I put the data for two hundred and fifty images in the same file?.
Please help me.

Data Preparation

Weka uses the arff file type.

A modified example arff file content from the previous link:

% 1. Title: Iris Plants Database
% 2. Creator: R.A. Fisher, Donor: Michael Marshall (MARSHALL%PLU@io.arc.nasa.gov)
@RELATION iris
@ATTRIBUTE petallength  NUMERIC
@ATTRIBUTE petalwidth   NUMERIC
@ATTRIBUTE class        {Iris-setosa,Iris-versicolor,Iris-virginica}
@DATA
1.4,0.2,Iris-setosa
1.4,0.2,Iris-virginica
1.3,0.2,Iris-setosa
1.5,0.2,Iris-versicolor
1.4,0.2,Iris-setosa
1.7,0.4,Iris-setosa

The above example can be broken down into sections:

 Comments

Lines beginning with "%" are comments and are not read by the WEKA data readers

@RELATION

dataset name (use quotes around the name if there are spaces)

@ATTRIBUTE

each attribute defines a feature of one of the following types:

NUMERIC: real number
NOMINAL: discrete value from a predefined list (see example @ATTRIBUTE class feature)
STRING: a string where quotes must be used if spaces are included
DATE: @ATTRIBUTE `name` date [`date-format`] where the default date-format is "yyyy-MM-dd'T'HH:mm:ss"

@DATA

Delimiter specifying the start of the CSV data

Weka Explorer

After creating your arff file, you have a few options when using the Weka explorer:

Load entire dataset into Weka explorer and use the built-in k-fold cross validation option which randomly partitions your dataset and automatically performs CV for your given a classifier
Load entire dataset into Weka explorer and use the built-in train/test split option with a user-specified percentage to randomly partition your dataset.
Manually partition your arff file into a train file and test file and use the supplied test-set option in the Weka explorer. To manually partition your arff file, simply copy your original arff file and and keep non-intersecting sets of CSV rows (i.e. delete instances from one file and retain those instances in the other).

If this is for educational purposes, it might be easiest to manually partition your dataset so that you know which images are in training and which are testing. If this is for research, of course k-fold CV is preferred.

If you are using the Weka Java library rather than the desktop application, there are plenty of online resources that explain the library architecture: Waikato Github io site


Any difference between adding epochs and duplicating data for neural nets? Let's say I am training a neural net (e.g. convolutional network or LSTM).

Generally, the longer the training (more epochs) leads to better accuracy, albeit at times at the expense of overfitting.

Another approach is to duplicate the data. Again, if I have 1000 examples, I could duplicate them 10 times and randomise them, and the neural net will learn better (but again, it may not be able to generalise well).

Is there any difference between duplicating data and adding epochs when training a neural net in terms of its learning?

The most obvious difference is that by adding epochs, you (still) never get the same observation twice in a same batch, which isn't the case when you duplicate data.

I'm no expert in NNs, but it seems that if you do not introduce any noise in the duplicated data, then you might as well just add epochs.

Of course, if you fit two models over a single epoch,

model 1: using your n=1000 observations

model 2: using your n*10 = 10,000 observations (with duplications)

your second model will most probably return a lower loss value. However, a fairer comparison would be to train model 1 with 10 epochs so that in both cases you expose your net to your observations the same number of times in total.


Should I go for a 'balanced' dataset or a 'representative' dataset? My 'machine learning' task is of separating benign Internet traffic from malicious traffic. In the real world scenario, most (say 90% or more) of Internet traffic is benign. Thus I felt that I should choose a similar data setup for training my models as well. But I came across a research paper or two (in my area of work) which have used a "class balancing" data approach to training the models, implying an equal number of instances of benign and malicious traffic.

In general, if I am building machine learning models, should I go for a dataset which is representative of the real world problem, or is a balanced dataset better suited for building the models (since certain classifiers do not behave well with class imbalance, or due to other reasons not known to me)?

Can someone shed more light on the pros and cons of both the choices and how to decide which one to go choose?

I would say the answer depends on your use case. Based on my experience:

If you're trying to build a representative model -- one that describes the data rather than necessarily predicts -- then I would suggest using a representative sample of your data.
If you want to build a predictive model, particularly one that performs well by measure of AUC or rank-order and plan to use a basic ML framework (i.e. Decision Tree, SVM, Naive Bayes, etc), then I would suggest you feed the framework a balanced dataset. Much of the literature on class imbalance finds that random undersampling (down sampling the majority class to the size of the minority class) can drive performance gains.
If you're building a predictive model, but are using a more advanced framework (i.e. something that determines sampling parameters via wrapper or a modification of a bagging framework that samples to class equivalence), then I would suggest again feeding the representative sample and letting the algorithm take care of balancing the data for training.


How to build a comunication interface between a reinforcement learning and a videogame? I want to try to build a reinforcement learning model in order to play old arcade videogame, I know about projects in order to build AI for playing videogames, but I don't know how could I build an interface between a videogame and a python app in order to train the model while the model is playing.
is there any library in order to train a reinforcement learning model using an emulator or something like that on python? How can I build an app like that?
Thanks

You might wanna check OPEN AI GYM The project has several environments for atari games, robotics and a lot of other useful resources on reinforcement learning.
I think this good place to start. If you need a bridge for some specific game, you need to google, for example there is repo for TorchCraft - A bridge between Torch and StarCraft.


Stacking doesn't improve accuracy I am trying to build a 2 level stacking model in order to tackle a multiclass classification problems with 8 classes.
My base (level 1) models andd their micro f1 scores in the test set are:

Random Forest Classifier (0.51)
XGBoost Classifier (0.54)
LightGBM Classifier (0.54)
Logistic Regression (0.44)
keras neural network (0.57)
keras neural netqork (0.56)

As a level 2 model I use an XGBClassifier which is not tuned.
I use 7 fold cross validation to produce the meta features for level 2 model.
The code I use to produce the meta features for the simple classifiers is:

ntrain = X_train.shape[0]
ntest = X_test.shape[0]
seed = 0
nfolds = 7
kf = StratifiedKFold(nfolds, random_state=seed)
def get_meta(clf, Χ_train, y_train, Χ_test):
meta_train = np.zeros((ntrain,))
meta_test = np.zeros((ntest,))

for i, (train_index, test_index) in enumerate(kf.split(X_train, y_train)):
    Χ_tr = X_train.iloc[train_index]
    y_tr = y_train.iloc[train_index]
    Χ_te = Χ_train.iloc[test_index]

    clf.train(Χ_tr, y_tr)

    meta_train[test_index] = clf.predict(Χ_te)

clf.fit(X_train,y_train)
meta_test = clf.predict(X_test)
return meta_train.reshape(-1, 1), meta_test.reshape(-1, 1)

and for the keras neural networks is:

def get_meta_keras(clf, Χ_train, y_train, Χ_test, epochs = 200, batch_size = 70, class_weight=class_weights):
meta_train = np.zeros((ntrain,))
meta_test = np.zeros((ntest,))

encoder = LabelEncoder()
encoder.fit(y_train)
encoded_Y = encoder.transform(y_train)
# convert integers to dummy variables (i.e. one hot encoded)
dummy_y = np_utils.to_categorical(encoded_Y)

for i, (train_index, test_index) in enumerate(kf.split(X_train, y_train)):
    Χ_tr = X_train.iloc[train_index]
    y_tr = dummy_y[train_index]
    Χ_te = Χ_train.iloc[test_index]

    clf.fit(Χ_tr, y_tr, epochs = epochs, batch_size = batch_size, class_weight=class_weights)

    meta_train[test_index] = clf.predict_classes(Χ_te)

clf.fit(X_train, dummy_y, epochs = epochs, batch_size = batch_size, class_weight=class_weights)
meta_test = clf.predict_classes(X_test)
return meta_train.reshape(-1, 1), meta_test.reshape(-1, 1)

My final micro f1 score is 0.54 which is less than the best of my base models score. My models are uncorrelated (corr<0.55). I tried to add more simple models like knn, naive bayes etc but the score fell even more.
Why doesn't my stacking approach improve the score ?

Although there is no data provided I'll try to share my ideas.
Stacking
We stack models by making several predictions with hold-out data sets and then collecting these predictions to form a new data set, where you can fit a new model on it
Often times the stacked model (also called 2nd-level model) will outperform each of the individual models due its smoothing nature and ability to highlight each base model where it performs best and discredit each base model where it performs poorly
Model diversity
But one of the most important thong in stacking is model diversity, how different a model is to each other. You should consider what is the new information each model brings into the table?
It will find why one of your model is good, and when another one is bad or fairly weak.
You don't need to worry too much to make all the models really strong.
Therefore, what you really need to focus is what information does this model brings, even though it is generally weak? Such models bring in new information that the meta model could leverage.
Normally, you introduce diversity from two forms:

By choosing a different algorithm or by . Which makes sense, certain
algorithms capitalize on different relationships within the data.
For example, a linear model will focus on a linear relationship, a
non-linear model can capture better a non-linear relationships. So
predictions may come a bit different

Running the same model, but you try to run it on different
transformation of input data, either less features or completely
different transformation. For example, in one data set you may treat
categorical features as one-hot encoding. In another, you may just
use label encoding, and the result will probably produce a model
that is very different.

So you need to test different models. As @Aditya said - the improvement isn't guaranteed either ways until and unless you have strong baselines


Validation during fitting in Keras How does Keras' fit function calculate when validation set/validation split are NOT defined (I understand that the default values are None/0.0 respectively, so being not defined is practically default)? One always need a reference set to evaluate model performance...

If you don't define the validation set/ validation split for your model, then it has no way to check for it's performance because you have not provided anything to the model on which it can validate its performance. In this case, the model will run through training examples, will learn a way to minimize the cost function as per your code and that's it. You get a model which has learned a hypothesis from your data but how good  the model is, it can only  be checked by making predictions for the test set.


Problem regarding calculating correlation approach? There are 3 columns say product_id, product_type, and price_drop. product_id is unique while a product can belong to say 100 classes that information is given by product_type. price_drop column value is 1 when a drop in the price of the product occurred else 0. What I have done is, I one-hot encoded the product_type column created the dummy variable for them. Using each dummy variable I calculated their correlation with the price_drop column. I wanted to see the correlation between each product type and a drop in price. Is this approach is correct?

Most probably, you are using Pearson's correlation method. This method is used for two Continuous features.
Here, both the price_drop and the OHE features are Binary Categorical features.
So, you can use these methods -
Phi - Phi is a measure of the degree of association between two binary variables      (two categorical variables, each of which can have only one of two values)
Crammer's V - Cramer’s V is an extension of phi for tables larger than 2×2.
Both are extensions of the Chi-square test of Independence.
Since both the Features have 2 values, both of the above methods will output the same result.
# dataset is your DataFrame
s1 = dataset['Status']
s2 = dataset[product_type_OHE_01]

import pandas as pd
from scipy.stats import chi2_contingency
n = len(s1)
r,c = s1.nunique(), s2.nunique()
matrix = pd.crosstab(s1,s2).values
chi_sq = chi2_contingency(matrix)

phi = np.sqrt(chi_sq[0]/n)
cramm_V = np.sqrt(chi_sq[0]/(n*min(r-1,c-1)))

print(phi, cramm_V)


found \N in my data does not count as missing values in r So scrolling through my columns I find \N embedded. I need to count them, but I get an error.  Would it be considered a missing value, it's new to me.

# Check whether attribute HeadOfState in country has any missing values, and if so, how many.
country$headOfState[country$headOfState==""] <- NA
country$headOfState[country$headOfState==\N] <- NA
sum(is.na(country$headOfState))

# Check whether attribute IndepYear in country has any missing values, and if so, how many.
country$IndepYear[country$indepYear==""] <- NA
country$indepYear[country$indepYear=='\N'] <- NA
sum(is.na(country$indepYear))

Error: unexpected input in "country$headOfState[country$headOfState==\"

  country$headOfState[country$headOfState==\N] <- NA
  Error: unexpected input in "country$headOfState[country$headOfState==\"

The reason its shorws an error, bacause '\' is a part of base regex expressions in R.
As states here:

The metacharacters in extended regular expressions are . \ | ( ) [ { ^ $ * + ?

So the comparisson like this

country$indepYear=='\N'

or this ( Its not a valid comparisson at all)

country$headOfState==\N

will throw an error. You need to "escape" the "\" symbol.
Try something like this instead if you want to replace "\N" with NA:

country$indepYear[country$indepYear=='\\N'] <- NA

If you need just to count them, you can use this approach:

sum(country$indepYear=='\\N')

Hope this helps.


What is the BLEU score used in Google Brain's "Attention Is All You Need" paper? Google Brain's Attention Is All You Need paper on sequence-to-sequence translation reports:

  Our model achieves 28.4 BLEU on the WMT 2014 Englishto-German
  translation task, improving over the existing best results, including
  ensembles, by over 2 BLEU

but the Wikipedia entry on BLEU says:

  BLEU’s output is always a number between 0 and 1

What definition of BLEU is the Google Brain paper using? I could not find a separate definition in the paper itself.

BLEU (Bi Lingual Evaluation Understudy) is an algorithm for evaluating the quality of text which has been machine-translated (MT) from one natural language to another. BLEU is typically measured on a 0 to 1 scale, with 1 as the hypothetical “perfect” translation.

Google uses the same definition of BLEU but multiplies the typical score by 100.


Where are the 60 million params of AlexNet? On the abstract of the AlexNet paper, they claimed to have 60 million parameters:

The neural network, which has 60 million parameters and 650,000 neurons, consists
of five convolutional layers, some of which are followed by max-pooling layers,
and three fully-connected layers with a final 1000-way softmax.

When I implement the model with Keras, I get ~25 million params.
model = tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(96, 11, strides=4, activation="relu", input_shape=[227,227,3]),
    tf.keras.layers.MaxPooling2D(pool_size=(3,3), strides=(2,2)),
    tf.keras.layers.Conv2D(256, 5, activation="relu", padding="SAME"),
    tf.keras.layers.MaxPooling2D(pool_size=(3,3), strides=(2,2)),
    tf.keras.layers.Conv2D(384, 3, activation="relu", padding="SAME"),
    tf.keras.layers.Conv2D(384, 3, activation="relu", padding="SAME"),
    tf.keras.layers.Conv2D(256, 3, activation="relu", padding="SAME"),
    tf.keras.layers.Dense(4096, activation="relu"),
    tf.keras.layers.Dense(4096, activation="relu"),
    tf.keras.layers.Dense(1000, activation="softmax"),
])

Note that I removed the normalization and set an input of 227*227 instead of 224*224. See this question for details.
Here is the summary from Keras:
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d (Conv2D)              (None, 55, 55, 96)        34944
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 27, 27, 96)        0
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 27, 27, 256)       614656
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 13, 13, 256)       0
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 13, 13, 384)       885120
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 13, 13, 384)       1327488
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 13, 13, 256)       884992
_________________________________________________________________
dense (Dense)                (None, 13, 13, 4096)      1052672
_________________________________________________________________
dense_1 (Dense)              (None, 13, 13, 4096)      16781312
_________________________________________________________________
dense_2 (Dense)              (None, 13, 13, 1000)      4097000
=================================================================
Total params: 25,678,184
Trainable params: 25,678,184
Non-trainable params: 0
_________________________________________________________________

I'm really far from 60 million. So, how did they sum 60 million params?
For reference, here is the architecture of the model as described in Sec. 3.5 of the paper:

The first convolutional layer filters the 224x224x3 input image with 96 kernels of size 11x11x3 with a stride of 4 pixels (this is the distance between the receptive field centers of neighboring neurons in a kernel map). The second convolutional layer takes as input the (response-normalized and pooled) output of the first convolutional layer and filters it with 256 kernels of size 5x5x48. The third, fourth, and fifth convolutional layers are connected to one another without any intervening pooling or normalization layers. The third convolutional layer has 384 kernels of size 3x3x256 connected to the (normalized, pooled) outputs of the second convolutional layer. The fourth convolutional layer has 384 kernels of size 3x3x192, and the fifth convolutional layer has 256 kernels of size 3x3x192. The fully-connected layers have 4096 neurons each.


Is a 100% model accuracy on out-of-sample data overfitting? I have just completed the machine learning for R course on cognitiveclass.ai and have begun experimenting with randomforests.

I have made a model by using the "randomForest" library in R. The model classifies by two classes, good, and bad.

I know that when a model is overfit, it performs well on data from its own trainingset but badly on out-of-sample data.

To train and test my model I have shuffled and split the complete dataset into 70% for training and 30% for testing.

My question: I am getting a 100% accuracy out of the prediction done on the testing set. Is this bad? It seems too good to be true.

The objective is waveform recognition on four on each other depending waveforms. The features of the dataset are the cost results of Dynamic Time Warping analysis of waveforms with their target waveform.

High validation scores like accuracy generally mean that you are not overfitting, however it should lead to caution and may indicate something went wrong. It could also mean that the problem is not too difficult and that your model truly performs well. Two things that could go wrong:

You didn't split the data properly and the validation data also occured in your training data, meaning it does indicate overfitting because you are not measuring generalization anymore
You use some feature engineering to create additional features and you might have introduced some target leakage, where your rows are using information from it's current target, not just from others in your training set


Reformatting data- Giving each value in a list it's own row while retaining the list's ID I am looking to reformat some data. It currently looks like this:
<img src="https://i.stack.imgur.com/Ts8Cn.png" alt="Site_ID Section_ID
1, "A100/2020, B100/1001, C130/2000"
2, "A100/2021, ZW00/2002, W300/0999"
3, "A100/2022, TS100/4000, RW100/0000"" />
Using this as an example, the below is the format i'm trying to achieve:

So each element in the list gets it's own row, but the original site_ID is retained.
A solution in Python would be ideal as this is the only language I am currently comfortable with.

You can use split with iterrows:
import pandas as pd
df = pd.DataFrame([{'Site_ID': 1, 'Section_ID': 'a,b,c'},
               {'Site_ID': 2, 'Section_ID': 'd,e,f'}])

df

  Site_ID   Section_ID
0   1   a,b,c
1   2   d,e,f

pd.concat([pd.Series( row['Site_ID'], row['Section_ID'].split(',') ) for _, row in df.iterrows()])

a    1
b    1
c    1
d    2
e    2
f    2

iterrows goes through the series row by row and split will find the separate values in a string that is separated by a comma.
If you want the column names back, you can convert back to a dataframe:
import numpy as np
df1 = pd.DataFrame(np.array(pd.concat([pd.Series( row['Site_ID'], row['Section_ID'].split(',') ) for _, row in df.iterrows()]).reset_index()), columns=['Site_ID','Section_ID'])

df1

 Site_ID Section_ID
0   a   1
1   b   1
2   c   1
3   d   2
4   e   2
5   f   2


ImportError: DLL load failed: The specified module could not be found Getting a error when trying to import tensorflow library for tensorflow gpu help

>>> import tensorflow as tf
Traceback (most recent call last):
  File "C:\Users\Admin\Python_\envs\tensorflow\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "C:\Users\Admin\Python_\envs\tensorflow\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "C:\Users\Admin\Python_\envs\tensorflow\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "C:\Users\Admin\Python_\envs\tensorflow\lib\imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "C:\Users\Admin\Python_\envs\tensorflow\lib\imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: DLL load failed: The specified module could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Admin\Python_\envs\tensorflow\lib\site-packages\tensorflow\__init__.py", line 24, in <module>
    from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import
  File "C:\Users\Admin\Python_\envs\tensorflow\lib\site-packages\tensorflow\python\__init__.py", line 49, in <module>
    from tensorflow.python import pywrap_tensorflow
  File "C:\Users\Admin\Python_\envs\tensorflow\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 74, in <module>
    raise ImportError(msg)
ImportError: Traceback (most recent call last):
  File "C:\Users\Admin\Python_\envs\tensorflow\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "C:\Users\Admin\Python_\envs\tensorflow\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "C:\Users\Admin\Python_\envs\tensorflow\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "C:\Users\Admin\Python_\envs\tensorflow\lib\imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "C:\Users\Admin\Python_\envs\tensorflow\lib\imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: DLL load failed: The specified module could not be found.

Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/errors

for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.

Use the following command to install tensorflow (even if you have already installed it)

For me it worked on win7 64bit and python 3.6.5

  conda install tensorflow keras


How to visualize trending over geographical area? I would like to visualize how a percentage evolves in time over a geographical area.

In particular I have the percentage of products delivered in different regions and I want to show the weekly trend for each of the 21 regions. Currently I show a heatmap with the values of the current week and three line plots showing the trends of the 3 worst regions. I would like to show the full picture. Unluckily I can't use animations as I have to put the visualization on a static powerpoint.

I'm working with d3, sql and excel.
Any idea?

Overly simple answer: "animate" this in PowerPoint by having one slide per time slice then advancing the slides quickly with the keyboard or set them to auto-advance.

If you have a little more flexibility outside PPT, there are free tools (e.g. Microsoft Power BI) where you could do a shaded map and have a time slider to make it more interactive.


Dataframe request with groupBy I'm a beginner in Spark and I want to calculate the average of number per name.

I have a JSON file with this information

df = spark.read.json("myjson.json")
df.select(avg(df["number"]), df["name"]).groupBy("name").show()

But I'm doing it wrong..

How can I solve my problem?

Thanks a lot

You are probably thinking in terms of regular SQL but spark sql is a bit different.

You'll need to group by field before performing your aggregation. Thus the following, you can write your query as followed :

df.groupBy('name').agg({'number': 'mean'}).show

You can also write it using a SQL dialect by registering the DataFrame as a temp table and then query on it use the SQLContext or HiveContext :

df.registerTempTable("people")
df2 = sqlContext.sql("select name, avg(number) as average from people group by name")

I hope that this answers your question.


Reinforcement Learning algorithm for Optimized Trade Execution My question deals with the algorithm described in the paper:
Reinforcement Learning for Optimized Trade Execution

This paper uses reinforcement learning technique to deal with the problem of optimized trade execution. They divide the data into episodes, and then apply (on page 4 in the link) the following update rule (to the cost function) and algorithm to find an optimal policy:

(T is the total time units, I is the volume, L is the possible number of actions, x represents the state, and c represents the cost function, and c_im is the immediate reward at a certain state and a certain action. n is the number of times that the state-action pair were visited)

Here are my questions:

If I understand correctly, the algorithm is basically a dynamic programming, when we move backwards in time.

Why do we need n in the cost function update rule. Aren't we visiting each state exactly once?
If I understand correctly, we should run this algorithm on every episode (in the experiment in the paper they had 45000 episodes). In such case, how do we combine the results from all the episodes? That is, each episode provides an optimal policy. How do we combine all these policies to one final policy?

Why do we need n in the cost function update rule. Aren't we visiting each state exactly once?

The update is assuming a static distribution and estimating the average value. As each estimate is made available, it is weighted less of the total each time. The formula means that the first sample is weighted $1$, second $\frac{1}{2}$, third $\frac{1}{3}$ which is what you need to get the mean value when you apply the changes due to the samples serially whilst maintaining the best estimate of the mean at each step.

This is a little odd in my experience of RL, because it assumes the bootstrap values (the max over next step) come from a final distribution to weight everything equally like this. But I think it is OK due to working back from final step, hence each bootstrap value should be fully estimated before going backwards to previous time step.

  If I understand correctly, we should run this algorithm on every episode (in the experiment in the paper they had 45000 episodes)

This looks like an algorithm that you run on the whole data set, where each episode is the same length $T$. So you run each timestep (starting with the end time step and working backwards since the ultimate reward is established at the end of the episode, so this is more efficient), and sample from every episode at that timestep in the While (not end of data) loop. The values are therefore combined inside the loop at that stage, and there is no need to add anything to the algorithm to combine episodes.


How to extract sub sentences from sentence mentioning a particular subject? I am trying to solve an NLP problem. For a given sentence like :

"The Pasta was delicious, the Pizza was average"

I want to extract the sentiment attached to food items. Having built my own NER model, I am able to extract Pasta and Pizza and hence the sentences containing them. But using a sentiment analyser on the entire sentence would be wrong in this case

EXPECTED OUTPUT:

Pasta - Good score
Pizza - Average score

CURRENT OUTPUT:

Pasta - Kinda Good Score
Pizza - Kinda Good Score

I know I am getting this output because I am considering the same sentence for getting the sentiment attached to both the subjects in hand.

Is there a way to extract the sub sentence like "Pasta was good" and "Pizza was average" to associate sentence with each item instead of whole sentences which I am currently doing ?

If you would know the target(s) (pasta, pizza) and the sentiment features (good, bad, etc), you could try to catch the feature that is closest to the target in a sentence.

But to say more, it really is necessary to see more of your sentences to understand the structure.


What metrics must i use in my data(unstructured) preprocessing research? i am currently working on preprocessing unstructured data (emails,logs,bug reports and irc chats).

i wish to prove preprocessing improves the content quality.

are there metrics available to prove this? that is to compare the raw data and the preprocessed data and show some quality improvement in numbers.

content quality - i mean to say that the preprocessed corpus is more suitable for mining with topic models than the raw one.

I am aware of perplexity. i dont want to use this metric as it only evaluates the topic model performance and not the corpus.Also i wish to evaluate the corpus quality even before the topic model is applied.

Can topic coherence be used as a metric to evaluate the quality of the corpus?
can more coherent topics be generated from preprocessed documents than the raw one?

The corpus quality can be determined using the zipf law ,vocabulary growth curve and the pareto distribution. An interesting thesis which explains this can be found at

http://ilk.uvt.nl/downloads/pub/papers/hait/camp2008.pdf


Unable to use pip package obtained from building Tensorflow 2.3 from source I've managed to build Tensorflow 2.3 from source, following these instructions:
https://towardsdatascience.com/how-to-compile-tensorflow-2-3-with-cuda-11-1-8cbecffcb8d3
But, when I install obtained pip package in new conda environment, and import tensorflow, I get following error: Could not load dynamic library 'libcudart.so.11.1'; dlerror: libcudart.so.11.1: cannot open shared object file: No such file or directory
I've managed to use GPU support with CUDA 11.1 for Tensorflow 2.5 nightly, without creating soft links between libs (I get Successfully opened dynamic library libcudart.so.11.0 message).
Any help appreciated.

It's working with symbolic link:
libcudart.so.11.1 -> libcudart.so.11.0
I guess it's because the nightly build has CUDA 11.1 fully supported.


Re-training regression model on covid data I am trying to re-train a regression model (XGB regressor) which was used in the pre-covid times (Feb 2020).
The dependent variable for the model is the number of bookings done, and due to covid, the bookings have been affected ,hence it will affect the accuracy of my model.
Any suggestions on how I can manage to re-train my model on covid data and still manage to get decent accuracy?

Your previous model won't be useful. In my opinion, you can not use it to make predictions in Covid Pandemic, since all bookings have had a huge decrease.
I would recommend build a new model with only data of the Covid Pandemic time. Then, you will have a more accurate model for these times. The only problem if you have less historical data, but for sure, is better have less data but useful, than huge dataset that is not representative of current times.
Be aware that you can probably add some useful features in this new model, for example, number of deaths for Covid per day of the country. For sure this feature will help you because if you have a huge number of deaths per day, probably you will have less booking, otherwise, if there are less deaths per day, there should be more bookings.


Receipt fraud detection I'm developing an OCR service that scans receipts and assigns points to user, based on amount of money spent. But the problem is that user can forge fake receipts and redeem them for extra points. User takes picture of receipt in app, which sends it to server, so it's not possible for user to modify picture after taking it. But user can print modified receipt (for example with changed receipt id) and take a picture of it. Is there a data science approach to solve this problem?

This is an anomaly detection problem. In your case, we would refer to it as supervised anomaly detection problem as you have the labels of categories.
This typically involves taking a large "normal" dataset, in this case, this would be receipts which are valid. And then using a machine learning method to learn features from this dataset (e.g. the typical words used, normal amounts entered, etc.) to generate a model of the "normal" data.
You could either go down the Computer Vision direction (e.g. for physical features in the photos) or down the Natural Language direction (e.g. for textual features in the receipts), or perhaps any other features available!
Once you have this model of "normal" behaviour, you can then test it against fraudulent receipts by predicting whether the receipt is normal.
But remember, the model will only be as good as your catalogue of "normal" and "abnormal" data


What's an efficient way to compare and group millions of store names? I'm a total amateur as far as data science goes, and I'm trying to figure out a way to do some string comparison on a large dataset.

I've a Google BigQuery table storing merchant transactions, but the store names are all over the board.  For example, there can be 'Wal-Mart Super Center' and 'Wal-Mart SC #1234', or 'McDonalds F2222' and 'McDonalds #321'.

What I need to do is group ALL 'Wal-mart' and 'McDonalds' and whatever else.  My first approach was doing a recursive reg-ex check, but that took forever and eventually timed-out.

What's the best approach for doing that with a table of 20 million+ rows?  I'm open to trying out any technology that would fit this job.

This is an entity resolution aka record linkage aka data matching problem.

I would solve this by removing all of the non-alphabetical characters including numbers, casting into all uppercase and then employing a hierarchical match.  First match up the exact cases and then move to a Levenshtein scoring between the fields.  Make some sort of a decision about how large you will allow the Levenshtein or normalized Levenshtein score to get before you declare something a non-match.

Assign every row an id and when you have a match, reassign the lower of the IDs to both members of the match.

The Levenshtein distance algorithm is simple but brilliant (taken from here):

def levenshtein(a,b):
    "Calculates the Levenshtein distance between a and b."
    n, m = len(a), len(b)
    if n > m:
        # Make sure n <= m, to use O(min(n,m)) space
        a,b = b,a
        n,m = m,n

    current = range(n+1)
    for i in range(1,m+1):
        previous, current = current, [i]+[0]*n
        for j in range(1,n+1):
            add, delete = previous[j]+1, current[j-1]+1
            change = previous[j-1]
            if a[j-1] != b[i-1]:
                change = change + 1
            current[j] = min(add, delete, change)

    return current[n]

This Data Matching book is a good resource and is free for seven days on Amazon.

Nominally, this is an $n^2$ algorithm without exploiting some sorting efficiencies, so I would expect to have to use multiple cores on $2\times10^7$ rows.  But this should run just fine on an 8 core AWS instance.  It will eventually finish on a single core, but might take several hours.

Hope this helps!


How much time do scikit classifiers take to classify? I am planning to use scikit linear support vector machine (SVM) classifier for text classification on a corpus consisting of 1 million labeled documents. What I am planning to do is, when a user enters some keyword, the classifier will first classify it in a category, and then a subsequent information retrieval query will happen in within the documents of that category catagory. I have a few questions:

How do I confirm that classification will not take much time? I don't want users to have to spend time waiting for a classification to finish in order to get better results.
Is using Python's scikit library for websites/web applications suitable for this?
Does anyone know how amazon or flipkart perform classification on user queries, or do they use a completely different logic?

I don't see a huge problem here. So, I would try to answer all of your questions from the production-level point of view:

  How do I confirm that classification will not take much time?

Take a subset of the corpus data you have (you can randomly do it, no need of sampling), and test your algorithm on it, and they approximate/generalize it to the overall dataset.

(SVM is comparatively faster. Nevertheless, do the above process just to be sure.)

And do test it in the development environment before pushing to production.

  Is using Python's scikit library for websites/web applications
  suitable for this?

Yes, it is. It is already being used by a nice chunk of companies out there.

The third question about Amazon and Flipkart cannot be answered by someone outside their teams.

In addition, I would advise you to use the mapreduce techniques for training your models. And as already advised, pickle your models so that you don't need to train them with every request.


Where to upload large (0.5Gb) weights anonymously? I need to upload a number of checkpoints for ConvNets (weights + optimizers, all dicts of pytorch tensors), each about 0.5Gb anonymously. I don't want to use Google Drive. I trained models on the university cluster (if it's relevant).

Where can I load these files anonymously? The files must be publicly available, but my identity must remain anonymous.

file.io or transfer.sh are good choices. transfer.sh has a 14 day lifespan for all files, while you can increase or decrease the lifespan on file.io. Additionally, you can use both from a command line or terminal via curl, and they're anonymous.


Backpropagation in Faster R-CNN I understand how the convolution layers are applied after selective search finds the regions of interest in vanilla R-CNN and so the back-propagation or any weight updating is done in the individual convolution networks. But in Faster R-CNN a convNet is applied first and then RPN works out the regions of interest in a separate convNet. Where exactly are the weights updated? Is it between the initial convNet and the RPN. But weights can only be updated after comparison with the ground truth values. Or is it that after a full forward pass of both networks, a backward pass is done. In this case how are both considered simultaneously? If not, Where and how is this comparison done? Any pointers to any resource will be highly appreciated.

Where exactly are the weights updated? Is it between the initial convNet and the RPN. All updateable weights are updated during backpropagation.

I assume you were trying to ask when are the weights updated?, well according to the original paper there are 3 way to train Faster R-CNN, each has it's own when to update weight time:

Alternating training aka 4-step alternating training, which is the one used by the author in the original faster r-cnn paper. It's explained quite well in the original paper
Approximate joint training, the one you mention in the question, after a full forward pass of both networks, a backward pass is done. I'm not quite sure what you mean by "In this case how are both considered simultaneously?"
Non-approximate joint training, I don't really understand about this one

Back to the Where exactly are the weights updated? question, well, some to be mention are the RPN kernerls, RPN Cls and Reg FC Layer, the detection (Fast R-CNN) Cls and Reg FC layer also the CNN kernels (if you didn't freeze them)

I hope this help since I'm also learning about Faster R-CNN too, also sorry for the bad English.