Neural network regression problem, integer output neuron constraint The problem I'm solving is a regression problem using neural networks, and the "y" value covers a very large range (let's say y represents the number of people, ranging from 0 people to 10000 people), which makes it impossible to be interpreted as a classification problem.

The only difference between the problem I'm solving right now and the ordinary regression problem is that the output neuron should produce a strict integer, not decimal.

Is there any activation function that can handle this constraint?  Or what's the appropriate way to approach to model this constraint?

You are correct to approach this as a regression problem, mostly because you are interested in the order of your outputs. For example if there are 1000 people present and you predict 1005, it's a better prediction than 7005. If you were treating this as a classification problem, both of these would be interpreted as missclassifications.

The most practical way to approach your constraint is to round the output of your neural network, i.e. if it outputs 3456.78, round it up to 3457. You need to have a linear activation function to be able to use the NN for regression.


Improvements to video-game cover CNN classifier (keras) As a personal project, I'm trying to build a classifier which attempts to predict the metacritic score of a game based purely on its cover. I figured it would be a fun project to learn Keras image classification with (2D-Convolution to be more specific).

I'm definitely new to this so if I've made any rookie mistakes, please tell me. Here are some notes before I post the CNN:

1) I've written a Metacritic Scrubber which takes PS2, PS3, PS4, Xbox360 and Xbox One games which have metacritic scores, downloads the artwork and labels it with the score. Duplicate games are removed (which may be a mistake since they sometimes have different scores based on platform). I'm pretty happy with this code.

2) I've rounded scores on a scale of 0-9 instead of 0-100 to make the number of classifications smaller.

3) images are 123x98 with 3 channels. All images have been stretched to this size. I wonder if this may be a source of problems because some covers have been stretched. Values are between 0-255 for each channel.

4) Games with square cover art (DLC, non-retail, etc) have been omitted.

This leaves me with a data set of 3816 game covers. I figured this would probably be enough for an initial investigation.

The model I've built has been based on the work by Iwana et al in this paper:
https://arxiv.org/abs/1610.09204

model = Sequential()
model.add(Conv2D(64, (2, 3),activation='relu', input_shape=(123,98,3)))
model.add(MaxPooling2D(pool_size=(2, 2),padding='valid'))

model.add(Conv2D(128, (2, 3),activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='valid'))

model.add(Conv2D(256, (2, 3),activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='valid'))

model.add(Conv2D(256, (2, 3),activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='valid'))

model.add(Conv2D(256, (2, 3),activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='valid'))

model.add(Flatten())

model.add(Dense(720, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(720, activation='relu'))
model.add(Dense(720, activation='relu'))
model.add(Dense(10, activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer=SGD(lr=0.001,momentum=0.1,decay=0.0005,nesterov=True), metrics=['accuracy'])
model.summary()

#Save output to file
csv_logger = CSVLogger('training.log')
#Save best model when possible
checkpoint = ModelCheckpoint('training_model_best.hdf5', monitor='val_acc', verbose=1, save_best_only=True, mode='max')

model.fit(np.array(images), np.array(labels), epochs=225, batch_size=10,
    validation_split=0.50,verbose=2,shuffle=True,callbacks=[csv_logger,checkpoint])

I'm using quite a harsh validation split of 0.5 because I really wanted to avoid overfitting. But regardless of whether I use between 0.2 - 0.5, I typically reach a validation accuracy of approximately 35%, which is pretty disappointing. A lot of the hyper-parameters I have no idea what to set to however.

Sorry for the long post. Ultimately, my result is a little better than random guessing but I'm hoping through some pointers, I can bring this accuracy up a bit. I wonder if actually, the cover files I've downloaded are too small.

Thanks for reading.

Here are some points to consider:

  1) I've written a Metacritic Scrubber which takes PS2, PS3, PS4, Xbox360 and Xbox One games which have metacritic scores, downloads the artwork and labels it with the score. Duplicate games are removed (which may be a mistake since they sometimes have different scores based on platform). I'm pretty happy with this code.

Good!

  2) I've rounded scores on a scale of 0-9 instead of 0-100 to make the number of classifications smaller.

This also should be fine.

  3) images are 123x98 with 3 channels. All images have been stretched to this size. I wonder if this may be a source of problems because some covers have been stretched. Values are between 0-255 for each channel.

I would normalise the values between 0-1 (norm_values = values/255). Regarding the size of the files I don't think this should be a problem. For example, Cifar-10 are 32x32 or the average image resolution on ImageNet is 469x387 pixels although most approaches resize them to 256x256. I think your size should be fine.

  4) Games with square cover art (DLC, non-retail, etc) have been omitted.

Not much to add here.

  This leaves me with a data set of 3816 game covers. I figured this would probably be enough for an initial investigation.
  The model I've built has been based on the work by Iwana et al in this paper: https://arxiv.org/abs/1610.09204

Here is where I actually think we have some problems. First, your dataset is quite small. 3816 reduced to 1908 when splitting train/test... This is not good especially with such structure. In the paper you mentioned, they are using a network with around 2.5M parameters and they used ~137K samples. Your 1908 dataset seems tiny compared and you are using the same model structure...

In my opinion your model is not able to do better, simply put. It doesn't matter what parameters you choose you need (a lot) more data. You might try reducing the size of your network, also create more data using noise addition, mirror samples, etc... and see if these help somehow.

Finally, apart from the size, you have no way to tell if your data is representative enough so that any model can learn from it. Therefore, 35% accuracy is as good as any other value I am afraid. Anyway, your mission seemed to be able to complete your experiment and you did. And you learnt a lot about limitations of deep models, so I'd say: good work!


Can I save only some VGG19's layers into a .H5 file? I am training a deep-learning style transfer model with the pretrained-VGG19 CNN.

My aim is to use it in my Android app for personal purposes with Google Firebase Machine Learning Kit (which would host my .H5 model to make it usable by my Android app). The maximum .H5 model file's size allowed by Machine Learning Kit  is: 8MB. However when I save the whole VGG19 model, I end with 80MB... So I can't use it.

Since only some layers of the VGG19 network are used in my style transfer program, is it possible to reduce the .H5's size by saving only those layers' weights, or something like that? Is there any other solution to my problem?

To save my VGG19 network as a .H5 file, I use the following Python command:

model.save('./style_transfer/st.h5', include_optimizer=False) , where model = vgg19.VGG19(input_tensor=input_tensor, weights='imagenet', include_top=False).

As you can see, I already don't include the optimizer in order to reduce the saved .H5's size.

after creating the model you can create another model as below ( I created model till 8 layers)

model = Model(vgg19.input, vgg19.layers[8].output)
model.save('./style_transfer/st.h5')

You can also use post-training Quantization techniques to reduce the size of the model to deploy in mobile/IoT devices. please check tensorflow documentation here


Why use Gradient Descent when Gradient just solve the problem? (With Neural Nets) My knowledge is that gradient actually gets to the global minimum and gradient descent try to take steps to the direction that he judges to be the lowest.
I know that calculate the gradient from complex functions is non trivial problem, but let's say that we are a using neural net with a quadratic cost function, would be that hard to calculate the gradient and get actual global minimum?

You might be mistaking the gradient itself with the mathematical approach to find the critical points of a differentiable function.

In the latter approach, you take the derivative of the function with respect to its parameters and find the values of the parameters that make it zero. These points in parameter space are critital points, that is, where the function has either a local maxima, a local minima or a saddle point. In order to know the actual type, you have to take the second derivative.

The problem with applying such an approach to neural networks is not finding the gradient of the loss function with respect to the parameters; currently we have automatic differentiation software that computes the gradients for you. The actual problem is solving the equation where such gradient equals to zero. We don't know how to do that but in the trivial cases. Furthermore, this only gives you local optima, not global ones.

A solution to that problem is using numerical optimization techniques, like the gradient descent family. They basically explore the parameter space in the descending direction of the gradient of the loss function, hoping to reach a minimum. This cannot not assured for non-convex problems. Nevertheless, in practice it works quite well.

The reasons why gradient descent techniques work well in non-convex problems are an active line of research.


Poisson distribution, Standard deviation, fitting line Let's say I have system A, B, C, and D. Each system contains 10,000 numbers generated by Poisson distribution. The difference is the mean is different for different systems. I calculated the std dev for each distribution corresponding to each system. I plot the standard deviation with respect to mean numbers. (I observe std deviation increases with mean). I want to fit the plot with some line, which gives me the general estimate of std dev with respect to the mean number. I need help in finding the fitting line!
Thank you!

This sounds like simple linear regression. Your x-value is the vector of means and the y-value is the vector of corresponding standard deviations:
$$Y = X\beta$$
$\beta$ a vector containing the slope of your regression line and the offset term.
In R for example you can do fit <- lm(Y~X).


Error while using flow_from_generator Getting this error while using flow_from_generator in keras

  63/3851 [..............................] - ETA: 6:41:59 - loss: 12.8586 - acc: 0.1930
  64/3851 [..............................] - ETA: 6:41:40 - loss: 12.8544 - acc: 0.1934Traceback (most recent call last):
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/utils/data_utils.py", line 551, in get
    inputs = self.queue.get(block=True).get()
  File "/tools/anaconda3/envs/py35/lib/python3.5/multiprocessing/pool.py", line 644, in get
    raise self._value
  File "/tools/anaconda3/envs/py35/lib/python3.5/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/utils/data_utils.py", line 391, in get_index
    return _SHARED_SEQUENCES[uid][i]
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/preprocessing/image.py", line 761, in __getitem__
    return self._get_batches_of_transformed_samples(index_array)
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/preprocessing/image.py", line 1106, in _get_batches_of_transformed_samples
    interpolation=self.interpolation)
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/preprocessing/image.py", line 364, in load_img
    img = img.resize(width_height_tuple, resample)
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/PIL/Image.py", line 1743, in resize
    self.load()
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/PIL/ImageFile.py", line 233, in load
    "(%d bytes not processed)" % len(b))
OSError: image file is truncated (42 bytes not processed)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "convolutional.py", line 394, in <module>
    runCNNconfusion()
  File "convolutional.py", line 379, in runCNNconfusion
    epochs=epochs,verbose=1, callbacks = [MetricsCheckpoint('logs')])
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/legacy/interfaces.py", line 87, in wrapper
    return func(*args, **kwargs)
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/models.py", line 1227, in fit_generator
    initial_epoch=initial_epoch)
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/legacy/interfaces.py", line 87, in wrapper
    return func(*args, **kwargs)
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/engine/training.py", line 2115, in fit_generator
    generator_output = next(output_generator)
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/utils/data_utils.py", line 557, in get
    six.raise_from(StopIteration(e), e)
  File "<string>", line 3, in raise_from
StopIteration: image file is truncated (42 bytes not processed)

Can someone please help, totally blank on how to proceed

I guess based on the answer here and the point referred here your problem is not from Keras. Where you can read ... we see that PIL is reading in blocks of the file and that it expects that the blocks are going to be of a certain size. It turns out that you can ask PIL to be tolerant of files that are truncated (missing some file from the block) by changing a setting.

Somewhere before your code block, simply add the following:

from PIL import ImageFile
ImageFile.LOAD_TRUNCATED_IMAGES = True

...and you should be good.

EDIT: It looks like this helps for the version of PIL bundled with Pillow ("pip install pillow"), but may not work for default installations of PIL.


Linear regression, R²? When I do a linear regression, R²: 0.90, but the estimates are not correct, why is this happening?

(Deep Not : Adjusted R-squared:  -0.3872)

You should also check the correlation between the features. The problem you mentioned arises when features are highly correlated.

Also perform Kfold validation in your dataset.


r glm - Error in names(coef) <- xnames only for 2 columns in data I am getting the below error when i run the R code for glm():

> fmla = 'status_id~ratings'
> logisticmodel <- glm(fmla, data = playdata, family=binomial(link="logit"))

Could you please correct me where I have gone wrong, I googled a lot on it and the only answer is generally naming convention, however, this is not case for me since I tried with changing the data by placing all the values as 2's and it worked -  so it is not a naming issue with variables.

Below is the data description:

> str(playdata)

'data.frame':   24160 obs. of  13 variables:
 $ idd         : int  57251659 63385939 51939145 64339389 33725679 47000250 62738883 33725679 53589441 36670488 ...
 $ status_id       : int  1 1 1 1 1 1 1 1 1 1 ...
 $ id              : int  22820543 22953283 22919397 22699949 22658030 22720403 22581860 22915483 22621108 22651736 ...
 $ group_id        : int  2 2 2 2 2 2 2 2 2 2 ...
 $ created_date    : POSIXct, format: "2017-03-31 12:45:10" "2017-04-04 10:50:11" "2017-04-03 16:40:04" "2017-03-28 14:40:48" ...
 $ question_id     : int  20073221 20219031 20185301 19948471 19906458 18651404 19816152 20175471 19865888 19897662 ...
 $ surftoanswertime: int  533 484 98 476 388 0 741 757 2222 381 ...
 $ skiprate        : num  0.981 0.95 0.993 0.875 0.966 ...
 $ ratings         :integer64 0 0 1 2 0 0 1 1 ...
 $ pos             :integer64 0 0 1 2 0 0 1 0 ...
 $ negratings      :integer64 0 0 0 0 0 0 0 1 ...
 $ cf              : num  NA NA 1 1 NA NA 1 0 NA 1 ...
 $ cf1             : num  -1 -1 1 1 -1 -1 1 0 -1 1 ...

Actual formula is as below -  only the ratings and negratings are throwing the above error.

fmla = 'status_id~skiprate+ratings+negratings+surftoanswertime+cf1'

Could you please help me in understanding and correcting the error.

> logisticmodel <- glm(fmla, data = playdata, family=binomial(link="logit"))
Error in names(coef) <- xnames : names() applied to a non-vector
In addition: Warning messages:
1: In glm.fit(x = c(1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,  :
  non-finite coefficients at iteration 1
2: glm.fit: algorithm did not converge

If I use more columns I am getting the below error:

Error in `*tmp*`[fit$pivot] : object of type 'closure' is not subsettable
In addition: Warning messages:
1: In glm.fit(x = c(1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,  :
  non-finite coefficients at iteration 1
2: glm.fit: algorithm did not converge

The problem is with datatype integer64. I don't know why this is the problem, but change it to type integer.

ratings = as.integer(ratings)

Normally, I have not seen anyone using integer64 datatype.


Encoding categorical data with pre-determined dictionary in case feature encoding, if I'd like to encode my values based on my pre-determined dictionary, how do I do that?
For instance, say, I've values as [Red, Green, and Blue] and I want to encode them as [-1,0,1]  -1 for red, 0 for Green, 1 for Blue... I'll apply it to my feature. I believe I can do it by mapping, apply method, not sure. But is there any better way to do that?
Column     expectedEncoding
Red             -1
Red             -1
Blue             1
Green            0
Red             -1
Blue             1

```

Assuming you have a pandas DataFrame and one mapping per column, with all mappings stored in a 2-level dict where the keys of the first level correspond to the columns in the dataframe and the keys of the second level correspond to the categories:
{'fruit': {'banana': -1, 'apple': 1}, 'color': {'yellow': -1, 'red': 1}}

Then, you can do the following:
encoded_data = data.apply(lambda col: col.map(mappings[col.name]))

[EDIT] if have columns for which you don't have a mapping, you can do one of the following:
data.update(data[list(mappings)].apply(lambda col: col.map(mappings[col.name])))

or if you want it in a new dataframe (eg to keep the dataframe with the original values):
encoded_data = data.copy()
encoded_data.update(data[list(mappings)].apply(lambda col: col.map(mappings[col.name])))


How to evaluate the "importance" of a variable in a function Let's say that we have
$$f(x,y,z) = x/k - (y/k) ((z - x/k)/(z - y/k))$$
$$k = constant \in ]0,1[$$
And  I need to show in some way that the variable $x$ is more important in  some metric that I don't know which one could be good. I thought about  analyze the partial derivatives of that functions, but I don't think  that is a good way, because one will only see some restricted path  through the surface.
Another  approach would be do monte carlo simulation in some domain of interest  (it exists, actually is a real world problem) and see that the variance  of the function value increases more varying x fixing the another  variables rather than the same simulation with $x$ fixed.
In some extent one can thinking in this function as a model, and I need to calculate the variable importance, it could be a way as well.
But I'm really  puzzled how to approach this problem.
Any metric or some approach is welcomed!
Thank you very much!

The way to start would be to understand intuitively what makes some variable important.
For example, the output value of the function might depend to a large extend on the value on that variable (so correlation would be quite high between output and that variable).
Another example would be to make that variable zero and measure the correlation between the original function and the new function (eg it would exhibit low correlation).
Still another way would be to compute the rate of change (derivative) of the function with respect to the rates of change (derivatives) of each variable and see how much a small perturbation of that variable affects the output of the function.
Similar other approaches are possible as well. Since you have the formula of the function wrt the variables one of the above approaches is fine.
There is no agreed upon universal metric for variable/feature importance but there are various approaches one can take depending on problem at hand.
Further reading:

Variable importance analysis: A comprehensive review

Assessing Variable Importance for Predictive Models of Arbitrary Type

Variable-importance Measures

Methods to quantify variable importance: implications for the analysis of noisy ecological data


Classification models with multi-class allowed for each record I am training a multi-class classification model. Each record can belong to one or more classes. (actually can I still call it a classification model? or should it be a topic model or something else?)

The training data looks like this (format_A)

record_id, f_1 , f_2, f_3, ... , f_n, output_A, output_B, output_C
        1,  0.1, 0.2, 0.3, ... , 1.2,        1,        0,        1
        2,  0.3, 1.2, 0.5, ... , 2.1,        1,        0,        0
        3,  0.2, 3.2, 1.3, ... , 0.4,        1,        1,        0
        4,  1.1, 0.1, 0.7, ... , 0.5,        0,        0,        1
        5,  2.1, 0.5, 0.8, ... , 1.9,        0,        1,        1
        6,  0.5, 0.4, 0.2, ... , 0.8,        1,        1,        1
                     :
                     :

Right now, I am using the sklearn logistic regression with multi-class. I preprocess the above training data such that there is only one target per record. For record with more than 1 output, I duplicated the records. That is, the above training data becomes below (format_B):

record_id, f_1 , f_2, f_3, ... , f_n,    target
        1,  0.1, 0.2, 0.3, ... , 1.2,  output_A
        1,  0.1, 0.2, 0.3, ... , 1.2,  output_C
        2,  0.3, 1.2, 0.5, ... , 2.1,  output_A
        3,  0.2, 3.2, 1.3, ... , 0.4,  output_A
        3,  0.2, 3.2, 1.3, ... , 0.4,  output_B
        4,  1.1, 0.1, 0.7, ... , 0.5,  output_C
        5,  2.1, 0.5, 0.8, ... , 1.9,  output_B
        5,  2.1, 0.5, 0.8, ... , 1.9,  output_C
        6,  0.5, 0.4, 0.2, ... , 0.8,  output_A
        6,  0.5, 0.4, 0.2, ... , 0.8,  output_B
        6,  0.5, 0.4, 0.2, ... , 0.8,  output_C
                         :
                         :

I use format_B to train a multi-class logistic regression model. The model then predicts the probability of each category, and I set my own threshold to decide the final outputs.

While I have done the above, I am wondering would it possible to use format_A as the input of sklearn logistic regression with multi-class directly?

Or is there any package/library actually takes format_A as input? Or any other kind of models actually more suitable for this type of problem?

Thank you!

The task is called multilabel modelling. Many libraries in sklearn support this -http://scikit-learn.org/stable/modules/multiclass.html.

These algorithms generally use one vs rest strategy, building separate binary classifiers for each of the label (zero for all others) and assigning all labels which clear a probability threshold.


What is the best tutorial to quickly learning machine learning in R I want to learn machine learning methods, preferably high-performance methods, by R in the quickest way possible. I am totally familiar with R but very less experience with ML packages. I know some resources like books but what I intend by opening this topic is to find the quickest way possible. I also need to see some examples in the tutorial and I DO NOT need any theory.

There is the Wikibook Data Mining Algorithms In R  and some useful examples at rdatamining.com


Recommendation model that can recommend already bought item Most recommendation algorithms recommend new products to users.

  If you bought this you might like that

But sometimes the item user is most likely to buy is an item that he bought sometime ago.

Is there any algorithm appropriate for this use?

Usually recommendation algorithms provides the confidence that a user will like an item. Items that the user already bought should get high confidence - the user bought them since he liked them.

In most cases filtering out item the user already bough is done in the application level, not in the algorithm level.

So , you can use regular recommendation algorithms in order to know if the user will like the item.

Please note that you might face a different problem - will the user buy the item for the second/third time.
In order to cope with this problem, using some domain knowledge might be beneficial. If the products are usually bough many times (e.g., milk), just use the recommendation algorithm. If all the products have the same tendency to be bought few times, build a model for that (e.g., probability for buying another time given already buying x times) and combine the models.
If your products are very different from this aspect you might need to get into a lower level. Different domains require different solution but you might split the product by the buying behavior and train few recommendation systems, add the number of buying as a feature, etc.


How to deal with overestimation of small values and underestimation of high values in XGBoost? I'm running XGBoost to predict prices on a cars dataset, I was wondering what alternatives are there for this kind of problem where smaller values are overestimated and higher prices underestimated.

I tried applying log to prices since it has a skewed to the right distribution, but still having this undesirable effect.

Also, as a bonus question, log(price) improved the prediction score, the mean relative error or MRE calculated as mean(ABS(RD)) by 2 percent, if anyone has the intuition onto why this could have happened that would be great.

In the image below RD is the relative difference between predictions and the actual values, and the price bucket is a bucketized variable where the number indicates the price low interval bound over 1000.

I dont think its necessarily related to the type of algorithm performing the regression(XGBoost here) - but to inherent nature of regression algorithms.
Many loss function are aimed to reduce distance between $y$ and $\hat{y}$.
That can lead to model predictions distribution being tighter around $y$'s mean.

Couple of things I would check to verify this:
1) Compare distribution of real $y$ and predicted $\hat{y}$.
2) Verify this error pattern on others regression model.

Things I would try to improve results:
1) over sample low/high y values in training set.
2) Adjust loss function so errors on low/high y values will have more weight.
3) Look for features that emphasize low/high y values and engineer them better.


Why does keras.Model subclass behave differently to the sequential API? I have implemented WGAN based on this blog. I tried to implement define_critic() as a subclass of keras.Model with the goal of having the same result.
class CNNBlock(layers.Layer):

    def __init__(self, channels, kernel_size=(4, 4)):
        super(CNNBlock, self).__init__()

        w_init = initializers.RandomNormal(stddev=0.02)
        const = ClipConstraint(0.01)

        self.conv = layers.Conv2D(channels, kernel_size, strides=(2, 2), padding="same",
                                  kernel_initializer=w_init, kernel_constraint=const)
        self.bn = layers.BatchNormalization()
        self.relu = layers.LeakyReLU(alpha=0.2)

    def call(self, input_tensor, training=False):
        x = self.conv(input_tensor)
        x = self.bn(x)
        return self.relu(x)

class CriticModel(Model):

    def __init__(self, channel):
        super(CriticModel, self).__init__()
        self.cnn1 = CNNBlock(channel[0])
        self.cnn2 = CNNBlock(channel[1])

    def call(self, input_tensor, training=False, mask=None):
        x = self.cnn1(input_tensor, training=training)
        x = self.cnn2(x, training=training)
        x = layers.Flatten()(x)
        x = layers.Dense(1)(x)
        return x

    def get_config(self):
        super(CriticModel, self).get_config()

However, when I run the code, I get an error on the call c_model.train_on_batch(X_real, y_real).
Error
ValueError: tf.function-decorated function tried to create variables on non-first call.

I don't understand why my change leads to this error.
The entire code can be found here

I create two new objects in call(): layers.Flatten and layers.Dense. When I create the objects in the constructor, the model works as expected.
class CriticModel(Model):

    def __init__(self, channel):
        super(CriticModel, self).__init__()
        self.cnn1 = CNNBlock(channel[0])
        self.cnn2 = CNNBlock(channel[1])
        self.flat = layers.Flatten()
        self.dense = layers.Dense(1)

    def call(self, input_tensor, training=False, mask=None):
        x = self.cnn1(input_tensor, training=training)
        x = self.cnn2(x, training=training)
        # x = layers.Flatten()(x) <-- This creates a new obj at eacht forward call
        # x = layers.Dense(1)(x)  <-- This creates a new obj at eacht forward call
        x = self.flat(x)
        x = self.dense(x)
        return x

    def get_config(self):
        super(CriticModel, self).get_config()
```


Conceptual question on CNN and any multi layer neural network (Part 2) I have read a number of tutorials and online lectures (https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/) but none of them mention the rationale for selecting a particular design. How do we decide on the following design aspects?

1) Is there a rule of thumb for deciding on the number of layers?  Or is it purely on the basis of trial and error?

2) Can somebody please explain the intuition and the rationale for designing a CNN architecture for this example -- considering a binary classification problem. For an input RGB image of size 500*500*3, how would you design the architecture -- how many layers, number of filters, size of the filter, how much is the stride, etc.

There's an amazing book called "Deep Learning with Python" by Francois Chollet that you could refer to. To answer your question:

You usually add high number of layers and check where the validation accuracy halts and training loss decreases rapidly. This means your network starts to overfit and it's easy to fine tune your network from there on with the help of regularization.
You determine the layers of your network depending upon the complexity of the problem and the resolution of the images as well. Bigger pictures need larger networks. The filters need to be increasing as the network increases and the size of the feature map should decrease in an ideal network. I don't know much about strides though. Max_Pooling is the most common solution instead of strides.

I could go on about Max_Pooling but that won't answer your question.


What are the practical differences between pooling types at convolution? I could find max-pooling in most -- actually, exactly all -- concrete CNN implementations I've seen. However, I see that there are several other kinds (min, average, global average pooling, etc.) but I cannot see what they differ on and what would be the best criteria to select them at application. Has anyone a clue?
I have learned that pooling is a sort of downsampling where we can preserve the locations of the original image that showed the strongest correlation to the specific features we searching for. So far, so reasonable. But what is the point of finding the weakest correlations with min pooling, for example?

See Analysis and Optimization of Convolutional Neural Network Architectures for the pooling types in general.

  What is the difference between max pooling, average pooling and min pooling?

A pooling function $f: \mathbb{R}^n \rightarrow \mathbb{R}$ is a function which is very symmetric:

$$f(x_1, x_2, \dots, x_n) = f(x_2, x_1, \dots, x_n) = f(x_n, x_1, \dots, x_2) = \dots$$

So the function does not care about the order of the arguments. This is the case for $\min$, $\max$ and average.

In the context of CNNs, you apply the pooling function to each feature map independantly. The output of a pooling layer has exactly as many feature maps as the input. For each feature map, you take a $p \times p$ section and apply the pooling function $f$ to it. Usually, you apply it with a stride $s > 1$ so that the output is much smaller than the input (only $\frac{1}{s^2}$).

  I have learnt that pooling is a sort of downsampling where we can preserve the locations of the original image that showed the strongest correlation to the specific features we searching for.

This is only true if you talk about max-pooling with $s > 1$. Which is usually the case. Although I would not say "correlation" but "activation".

  But what is the point of find weakest correlations with min pooling, for example?

I don't know. I have never seen min pooling even be mentioned. Where have you seen it?


Preprocessing for Text Classification in Transformer Models (BERT variants) This might be silly to ask, but I am wondering if one should carry out the conventional text preprocessing steps for training one of the transformer models?
I remember for training a Word2Vec or Glove, we needed to perform an extensive text cleaning like: tokenize, remove stopwords, remove punctuations, stemming or lemmatization and more. However, during last few days I have had a quick jump into transformer models (fascinated btw), and what I have noticed that most of these models have a built-in tokenizer (cool), but none of the demos, examples, or tutorials are performing any of the these text preprocessing steps. You may take fast-bert for instance, there are no text preprocessing involved for the demos (maybe it is just a demo), but at inference the whole sentences are passed without any cleaning:
texts = ['I really love the Netflix original movies',
         'this movie is not worth watching']
predictions = learner.predict_batch(texts)

The same is true for the original transformer by HuggingFace. Or many tutorials that I have looked at (take this or another one). I can imagine that depending on the task this might not be required, e.g. next work prediction or machine translation and more. More importantly I think this is part of the contextual-based approach that these models offer (that is the innovation so to say) that are meant to keep most of the text and we may obtain a minimum but still good representation of the each token (out of vocabulary word). Borrowed from medium article by HuggingFace:

Tokenisation
BERT-Base, uncased uses a vocabulary of 30,522 words. The
processes of tokenisation involves splitting the input text into list
of tokens that are available in the vocabulary. In order to deal with
the words not available in the vocabulary, BERT uses a technique
called BPE based WordPiece tokenisation. In this approach an out of
vocabulary word is progressively split into subwords and the word is
then represented by a group of subwords. Since the subwords are part
of the vocabulary, we have learned representations an context for
these subwords and the context of the word is simply the combination
of the context of the subwords.

But does that hold true for tasks like multi-label text classification? In my use case the text is full of not useful stopwords, punctuation, characters and abbreviations and it is multi-label text classification as mentioned earlier. And in fact the prediction accuracy is not good (after a few rounds of training using fast-bert). What do I miss here?

A quick experiment you can do is to once do the preprocessing steps that you usually do and then feed it to the model and get the results. And once feed the dataset as it is to the model to compare the difference.
In my experience doing the preprocessing won't make any difference, based on the dataset it gave me 1 more or less percent difference in accuracy (not a considerable change).
When these models are trained, no preprocessing is done, as they want to learn the context of all sorts of sentences.
A reason that your results are not good enough might because of your label distribution. Most of the time the datasets are populated with only one or two labels and other labels are only a small portion of the dataset. If that's the case you might want to look into oversampling solutions.


Problem reconstructing Keras's IMDB reviews Keras's sample IMDB reviews are represented as a sequence of word indexes. However,  a dictionary is provided that matches the index to the word I'm trying to use this to reconstruct the text of a review, but I'm getting gibberish. Here's my code:

from keras.datasets import imdb

(x_train, y_train), (x_test, y_test) = imdb.load_data()
ind = imdb.get_word_index()

def find_word(val):
    for key, value in ind.items():
        if value == val:
            return key

rev = x_train[0] #which review to look at
rev_recon = [] #hold the reconstructed review

for i in range(len(rev)):
    word = find_word(rev[i])
    rev_recon.append(word)

print(' '.join(rev_recon))
>>>the as you with out themselves powerful lets loves their
becomes reaching had journalist of lot from anyone to have
after out atmosphere never more room titillate it so heart
shows to years of every never going villaronga help moments

I thought the sequence of word indexes was supposed to represent the sequence in which the words originally appeared in the review. Am I missing something? Here's my code:

Nevermind, it wasn't very clear to me that the defaults for load_data() actually shift the sequence indices by 3, and so to obtain the proper indices for the dictionary you either need to offset those indices during lookup or shift the dictionary values, e.g.:

ind = imdb.get_word_index()
ind = {k:(v+3) for k,v in ind.items()}

And also every review is padded with a 1 at word [0], and so reconstructing the review means starting at [1] or passing start_char=None in load_data()


Data preparation for Regression Model Hi I'm currently trying to predict if an item will be successful in my store, this means (How much is going to sale in USD)
My training dataset contains many features:

Item name
Item weight
Item category
Item country of origin
Item sales overall
Item sales per store
Item rating
Item price

Etc....
Since I will be introducing a new item for sale, I know very little about this new item:

Item name
Item weight
Item category
Item country of origin
Item price

Not all the features present in training data/test data will be present when I will be making predictions. Is this normal in ML ? What is the rule of thumb when doing feature engineering for this type of cases.

Usually you can only predict with the variables you have trained on. But in a case like this, I would suggest you check the Multicollinearity of these missing variables with the ones you will have. May be they are already highly correlated with the features you already have. In that case, you can just model using the available features.

If that is not the case and the missing variable looks like a significant variable in your regression, then you might need to pick some other variables which can help you indirectly derive the missing features. For example may be category helps you derive the rating with some accuracy.


Computing Image Similarity based on Color Distribution Image Similarity based on Color Palette Distribution

I am trying to compute similarity between two images based on their color palette distribution, let's say I have two sets of key value pairs as follows,

Img1: {'Brown': 14, 'White': 13, 'Black': 40, 'Gray': 31}

Img2: {'Pink': 82, 'Brown': 8, 'White': 7}

Where the numbers denote the % of that color present in the image. What would be the best way to compute similarity on a scale of 0-100 between the two images?

Try using Earth Mover's Distance. It measures how much it "costs" to optimally reshape one histogram into another, where an elementary transform (using the terminology of edit distance-like distance measures) is moving a unit of mass from a bin to a bin. If you have an implementation of EMD, then your distance is dist = EMD(H1, H2, D), where
 H1 = [ 14, 13, 40, 31 ]; -- histogram #1 (first image)
 H2 = [ 82, 8, 7 ]; -- histogram #2 (second image)
 D  = [ D11, D12, D13; ... ; D41, D42, D43 ]; -- (cross-bin) ground distance

Dij is a distance from the i'th bin of the first histogram to the j'th bin of the second histogram. For example, D21 is a distance between 'White' and 'Pink'. It may be defined differently, based on which color space you want to use, and further, based on how you define what it means for two colors to be similar. If I use RGB, and I do not care much as to how human eye perceives colors, then D21 can be the \ell_2 norm of the difference [1, 1, 1] - [1, 0.05, 0.7]. (As far as I remember, the paper I mentioned uses LAB instead of RGB.)
P.S. Besides color, you may include spatial information into comparison. EMD can handle it well, as long as you properly combine similarity of colors with spatial similarity in the definition of ground distance D.


How to apply object detection? I am struggling on object detection in an image, but not sure which model to use. I tried to design a model from scratch, CNN, but it could not meet my requirements. I need a right direction or step by step approach to help me achieve it.

There are multifarious studies about object detection. The method you are going to use depends on your data. For instance, if you have centre of the objects alongside their height and width, you can simply use YOLO method which is one of the pioneer approaches for detection. As another example, If you have data which is classified pixel-wise, you can employ RCNN methods or Fully Convolutional methods like FCN itself or SegNet. There are open source implementations with appropriate trained parameters for these models on GitHub.

Apart from the methods, you have to consider the restrictions they may exist for your conditions. For instance, there may be situations where your task is online and you have limited computational power. In such cases, I don't think you can employ very deep architectures like SegNet.


Can we implement random forest using fitctree in matlab? There is a function call TreeBagger that can implement random forest. However, if we use this function, we have no control on each individual tree. Can we use the MATLAB function fitctree, which build a decision tree, to implement random forest? Thanks a lot.

I would highly recommend doing some research into the architecture of random forests.  There are many sites that provide in depth tutorials on RFs (Implementation in Python).

Quick  explanation: take your dataset, bootstrap the samples and apply a decision tree. Within your trees, you  want to randomly sample the features at each split. You should not have to build your own RF using fitctree however.  You don't want to control each individual tree in the forest. This introduces bias, and the point of the RF is that by bagging many trees, you remove the risk of overfitting.

Define your hyperparameters and let the algorithm do it's thing.  Carefully cross-validate to ensure you are not under-fitting.


Derive features from test-set? I have a dataset of choices (between A,B and C) done by certain users, and I want to train a neural network to predict the choices. I divide in train and test sets.
An instance is formed by: [UserId, some features, choice]. Choice is the target variable.
Same user can have multiple instances (with different features and choice).
My assumption is that the same user makes often the same choice, so I want a feature describing the "frequency" of a choice (e.g. User 1 choose A 60% of times).
From which dataset can I compute this frequency?

I cannot use the training set, otherwise is overfitting (with a frequency=0, the model is sure it was never chosen in the training-set).
I do not have an additional dataset of choices in a different period (this would be ideal).
Can I compute the frequency from the test-set? In that way, I'm actually training a model (on training set), based on the frequencies in a different period (the test set), which is what I want. Does it overfit the performances in the test-set? I assume not, since the model is trained on the training set. My results confirm similar performances in the two sets.

I think that the issue depends on what you'd expect the model to learn:

If the model is supposed to "know" the users it has seen during training, i.e. exploit the user id in order to infer particular choices for a specific user, then I don't see the point in adding this kind of frequency feature: the model already "knows" what choices this user tends to make, and it's supposed to know it only from the training set. Disadvantage: for any new/unique user not seen during training, the model probably has to fall back to a generic prediction.
If the user id is not used (removed from the features), then I think it makes sense to add the frequency features. I would see this mostly as a matter of feature engineering: let's assume that there are 3 features which represent how many times so far this user has chosen option A, B and C. The counting is made from any previous data available for this user. Of course, this is assuming that the same features can be calculated at testing time (equivalently, in a production environment).

So in the second option the features exist before any question about what is the training set and what is the test set. However they introduce a potential risk of data leakage, so what I would do is to separate the training set and test set based on the users, i.e. make sure that users in the training set and test set don't overlap.


Having a reward structure which gives high positive rewards compared to the negative rewards I am training an RL agent using PPO algorithm for a control problem. The objective of the agent is to maintain temperature in a room. It is an episodic task with episode length of 9 hrs and step size(action being taken) for every 15 mins.During the training of an agent, from a given state, agent takes an action.Then I check the temperature of the room after 15 mins(step size) and if this temperature is within limits, I give the action a very high positive reward, and if the temperature is not in he limits, I give a negative reward.Episode ends after 36 actions(9hrs * 4 actions/hour)(step size is 15 mins)

My formulation of reward structure.

zone_temperature = output[4]  # temperature of the zone 15 mins after the action is taken

thermal_coefficient = -10

if zone_temperature < self.temp_limit_min:
    temp_penalty = self.temp_limit_min - zone_temperature
elif zone_temperature > self.temp_limit_max:
    temp_penalty = zone_temperature - self.temp_limit_max
else :
    temp_penalty = -100

        reward = thermal_coefficient * temp_penalty

value of zone_temperature varies from limits in a range of 0 to 5 deg.  So the reward for when the actions are bad(temperature not in limits) varies from 0 to -50 , but when the actions are good(temperature is in limits) reward is +1000 . I had such a formulation so that the agent can easily understand which is a good action and which one is bad. Is my understanding correct and is it recommended to have such a reward structure for my use case ?

Thanks

Is my understanding correct and is it recommended to have such a reward structure for my use case ?

Your understanding is not correct, and setting extremely high rewards for the goal state in this case can backfire.

Probably the most important way it could backfire in your case, is that your scaling of bad results becomes irrelevant. The difference between 0 and -50 is not significant compared to the +1000 result. In turn that means the agent will not really care by how much it fails when it does, except as a matter of fine tuning once it is already close to an optimal solution.

If the environment is stochastic, then the agent will prioritise a small chance of being at the target temperatures, over a large chance of ending up at an extreme bad temperature.

If you are using a discount factor, $\gamma$, then the agent will prioritise being at the target temperatures immediately, maybe overshooting and ending up with an unwanted temperature within a few timesteps.

Working in your favour, your environment is one where the goal is some managed stability, like the "cartpole" environment, with a negative feedback loop (the correction to the measured quantities is always to force in the opposite direction). Agents for these are often quite robust to changes in hyperparameters, so you may still find your agent learns successfully.

However, I would advise sticking with a simple and relatively small scale for the reward function. Experimenting with it, after you are certain that it expresses your goals for the agent, is unlikely to lead to better solutions. Instead you should focus your efforts on how the agent is performing, and what changes you can make to the learning algorithm.

What I would do (without knowing more about your environment):

Reward +1 per time step when temperature is in acceptable range
Reward between -0.1 * temperature difference per time step when temperature is outside acceptable range. It doesn't really matter if you measure that in Fahrenheit or Celsius.
No discounting (set discount factor $\gamma =1$ if you are using a formula that includes discounting)

The maximum total reward possible is then +36, and you probably don't expect a worse episode than around -100 or so. This will plot neatly on a graph and be easy to interpret (every unit below 36 is roughly equivalent to performance of an agent spending 15 mins per day just outside acceptable temperatures). More importantly, these lower numbers should not cause massive error values whilst the agent is learning, which will help when training a neural network to predict future reward.

As an aside (as you didn't ask), if you are using a value-based method, like DQN, then you will need to include the current timestep (or timesteps remaining) in the state features. That is because the total remaining reward - as represented by action value Q - depends on the remaining time that the agent has to act. It also doesn't matter to the agent what happens after the last time step, so it is OK for it to choose actions just before then that would make the system go outside acceptable temperatures at that point.


Questions for data analysis of a class room I have Wireless Sensor Network setup in a classroom. I'm sensing data like temp, humidity, light, occupancy, etc. Classroom can accommodate 150 students and has 6 AC with light. From one side the classroom has big windows and that side is towards west and other 3 sides have thick walls and doors always closed. I have been looking for some questions that can be answered by analyzing data collected. Some of the questions I can think of are:

Predicting optimum time to turn on AC before class starts so that class is attains required temp with least energy consumed.
As of now I have 9 sensors on all walls and ceiling too. So to decide best position for sensor such that I can co-relate and predict other sensor values from only one or two sensor values.
Decide optimum amount of light needed by occupants to be comfortable in classroom.

Please suggest to me more questions that can asked about this situation. Also suggest if I can use somehow machine learning in WSN I have deployed.

Very difficult question for someone to answer as ultimately your examples and many questions contain certain constraints you won't get from your sensors.
For example:
Your first question is predicting "optimum" time to turn on AC.  who defines optimum?  Optimum for the classroom at full capcity, half capacity?
Your second question looks like it one, requires a map of the classroom and where the sensors are, two you won't be able to predict 7 sensor values from 2 sensor values, I don't know why you would want to do this.
Your third questions is a social sciences problem.  Define "light needed to be comfortable".  I imagine you would get 150 answers if you had 150 students.

Your scenario lacks context/comparison.  Let's say you had sensors outside the classroom for your various data points.  Then you could do things like, can I predict the temp inside the room based on my other room and external data points.

Finally, yes machine learning can be used with your wireless sensors.


Ranked tag recommendation for university courses Our system allows an admin to manage a database of university courses. These courses have multiple fields, like the department, a title, and a description.

I am adding the ability to add learning objectives to a course. To simplify the problem, let's say that learning objectives are just tags. Courses can have more than one learning objective associated with them. So a course like CHEM 101 might have "chemistry", "technology", "science", and several others.

Assuming I can reduce a course to a set of features, (using keywords/stemming/nlp, I suppose?), what kind of problem is this and what algorithm would you suggest? It seems very similar to a classification problem, but I want to provide a sorted list of suggestions with the most relevant at the top.

This will most likely turn out to be multiple binary classification problems.

Instead of building just one classifier, you would have to build as many classifiers as there are tags. Each classifier's task would be to predict whether a certain tag exists or not. A tag specific probability threshold can then be learnt by tuning over the precision-curve where course wise precision recall can be calculated.

The percentile of the predicted probability for each tag can be used to obtain a ordered (sorted) list of tag suggestions.

This way of converting a multi-class problem to multiple binary classification problems is known as binary relevance. While it is very efficient, it assumes that the labels themselves are independent. However, picking the probability threshold for each tag in order to optimize the course wise precision recall curve can lead to modeling dependence in the tags.


Kernel selections in SVM I want to understand the kernel selection rationale in SVM.
Some basic things that I understand is if data is linear, then we must go for linear kernel and if it is non-linear, then others.
But the question is how to understand that the given data is linear or not, especially when it has many features.
I know that by cross validation I can try and feed different kernels and see the output whichever performs best to be selected, but I'm looking for something anyway to have some early indications.

Start with linear kernel and see if your data is linearly seperable or not. Performing that is simpler than looking for early indications.

Linear kernels are suggested when the number of features is larger than the number of observations in the dataset (otherwise RBF will be a better choice).

However, once you conclude that you have non-linear data, you can try to visualize and map the non-linear separable data into a higher dimensional space and see if it makes the data linearly separable. That is what a kernel does anyway and so you visualizing will give you insights and indications about the type of kernel to be used.

This link might help.


tensorflow issue with image reading I am following this example to read an image (from my disk) with tensorflow:

img_raw = tf.io.read_file(image_path)
img_tensor = tf.image.decode_image(img_raw)

print(repr(img_raw))
print(img_tensor.shape)
print(img_tensor.dtype)

but the shape is coming as unknown:

<tf.Tensor 'ReadFile_4:0' shape=() dtype=string>
<unknown>
<dtype: 'uint8'>

Any idea what I am doing wrong?

For tensorflow 1.x we need to execute

tf.enable_eager_execution()

to see results before creating a tf session.

For tf 2.x eager execution is enabled by default.


Output a word instead of a vector after word embedding? I'm trying to play around with a toy implementation of translation or text-summarization. I understand now that most people use an embedding layer before whatever model they use, which produces something like 300-dimensional vector. But what does the model output? Like for a encoder-decoder model, it's inputs are sequences of those vectors. Then what does the final layer of decoder come out with? Not like event extraction or something like that, which we classify to a small number of classes.

So my core question is: Is the output also a 300-dimensional vector, which then I have to generate the word based on the most similar word vector? Or is it the index of a word from the original word space? In the second case, the network is classifying to more than 100000 classes?

And also, is there any existing package that support this "reverse embedding to word"?

I have not seen any question alike on the site. Please mark it as duplicate if it is.

Any help would be appreciated.

It depends on the architecture of the model.

A lot of work uses the text data to classify some specific category (e.g., the IMDB movie reviews or sentiment analysis on Tweets).

Other work can actually take the words of a sentence and predict the last word. Or break it into each word predicting the subsequent word, which is really what the RNN/embedding dimension is doing.

Other's have made these even at the character level. Ultimately, it depends on how you process the data and specify your outcome. Below are some useful tutorials that should add a little more clarity.

http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-2-implementing-a-language-model-rnn-with-python-numpy-and-theano/
http://karpathy.github.io/2015/05/21/rnn-effectiveness/
http://deeplearning.net/tutorial/lstm.html
http://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/
https://github.com/fchollet/keras/blob/master/examples/imdb_lstm.py


Why is cross-validation score so low? I am using Scikit-Learn for this classification problem. The dataset has 3 features and 600 data points with labels.

First I used Nearest Neighbor classifier. Instead of using cross-validation, I manually run the fit 5 times and everytime resplit the dataset (80-20) to training set  and test set. The average score turns out to be 0.61

clf = KNeighborsClassifier(4)
score = 0
for i in range(5):
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
        clf.fit(X_train, y_train)
        score += clf.score(X_test, y_test)
print(scores / 5.0)

However when I ran cross-validation, the average score is merely 0.45.

clf =  KNeighborsClassifier(4)
scores = cross_val_score(clf, X, y, cv=5)
scores.mean()

Why does cross-validation produce significantly lower score than manual resampling?

I also tried Random Forest classifier. This time using Grid Search to tune the parameters:

param_grid = {
    'bootstrap': [True],
    'max_depth': [8, 10, 12],
    'min_samples_leaf': [3, 4, 5],
    'min_samples_split': [8, 10, 12],
    'n_estimators': [62, 64, 66, 68, 70]
}
clf_ = RandomForestClassifier()
grid_search = GridSearchCV(estimator = clf, param_grid = param_grid,
                          cv = 5, n_jobs = -1, verbose = 2)
grid_search.fit(X, y)
grid_search.best_params_, grid_search.best_score_

The best score turned out to be 0.508 with the following parameters

({'bootstrap': True,
  'max_depth': 10,
  'min_samples_leaf': 4,
  'min_samples_split': 10,
  'n_estimators': 64},
 0.5081967213114754)

I went ahead make prediction on the whole 600 data points and accuracy is quite high 0.7688.

best_grid = grid_search.best_estimator_
y_pred = best_grid.predict(X)
accuracy_score(y, y_pred)

I know .best_score_ is the 'Mean cross-validated score of the best_estimator.` But I don't understand why it seems so much lower than the prediction accuracy on the whole set.

In your random forest, this is due to the fact that your final model is overfitting. Sklearn's GridSearchCV has a default argument refit = True, that takes the model with the best performance based on cross-validation and retrains it in the whole dataset. Your accuracy score is very high due to the fact that it is only measured on your training data, and the best_score is a measure that incorporates how your model performs in models that it has not seen.

To wrap up, in your random forest you are overfitting very badly, as there is a big gap between your validation and training error. Try to use refit = False and you will not see this gap anymore (but you still have a problem since you are still overfitting your training set with this model).


When using Absolute Error in Gradient Descent, how to calculate the derivative? What is the derivative of the Loss Function (Absolute Error) with respect to the feature weights that is used to update the weights?

Couldn't find anything specific about it anywhere.

The gradient of MAE is not continuous in $y_{pred} = y_{true}$ and therefore there is no defined (bounded, direction independent) derivative at that point.
Elsewhere you have -1, where  $y_{pred} > y_{true}$ and +1 where  $y_{pred} < y_{true}$
Usually frameworks like TensorFlow, Keras, etc... use an approximate derivative for that point.


Keras model evaluation accuracy vs. observation I am a newbie here and trying to make sense out of the scores from model.evaluate from what I am actually seeing in model.predict

I have a created a CNN model for the Google Audio Set data and achieved a 99%+ accuracy on training.

Here is how I do the prediction

model = load_model('model_audioset.h5')
for x, y in unbal_generator:
  score = model.evaluate(x, y, verbose=0)
  pred_y = model.predict(normalized_x)

Here is what I am seeing for one specific iteration of x and y from model.evaluate

model.metrics_names = {list: 2} ['loss', 'acc']
 0 = {str} 'loss'
 1 = {str} 'acc'

score = {list: 2} [0.03851451724767685, 0.9905123114585876]
 0 = {float64} 0.03851451724767685
 1 = {float64} 0.9905123114585876

Here is a readable output from model.predict and comparing it to y

Wondering how Keras came up with an accuracy score of 99.05 for this output? Clearly, the predicted classes are not the same as expected

I am using a binary_crossentropy loss function and sigmoid activation in the predictions layer as classes are NOT mutually exclusive

Actually, I think accuracy is not a good metric for this case as y_actual and y_expected have most values 0 and the length of y_actual is quite big too.

So the accuracy calculation using an equation like K.mean(K.equal(K.argmax(y_true, axis=-1), K.argmax(y_pred, axis=-1))) (Reference) is not a good metric on model performance


How to determine a sufficient number of training examples for a linear regression classifier? How do we determine a sufficient number of training examples for a linear regression classifier?
What kind of behaviors might we expect if we use too few training examples?

For every 15-25 samples, you can have 1 additional feature. In rarest cases it might be low as 10 samples for each feature, however, it is not recommended unless you have good knowledge of your data and can safely be sure or test that your model is not showing spurious results. For more, please read the source.
Using a few samples or breaking above mentioned rule of using too many features for a few samples (e.g. 80 samples 15 features) may cause overfitting. In this case, it will not be able to generalize your data instead will learn your input data in detail. However, those details will probably be redundant information or randomness of your data so, your model will be useless in production. Another possibility is that it may give a rise to spurious results. In other words, since your samples are few, your model can somehow find a way to imitate your target variable with no logical relationship but by chance.
Those, if you have few samples try to preserve the above-mentioned rule. If you have many features do a bivariate analysis to find how good are these features to explain your target variable alone. One other way is to use algorithms that are more robust against the overfitting and can handle many features of a few samples. For example, SVM is accepted as better than Linear Regression in such cases.


Any "rules of thumb" on number of features versus number of instances? (small data sets) I am wondering, if there are any heuristics on number of features versus number of observations. Obviously, if a number of features is equal to the number of observations, the model will overfit. By using sparse methods (LASSO, elastic net) we can remove several features to reduce the model.

My question is (theoretically): before we use metrics to assess the model selection are there any empirical observations which relate the optimal number of features to the number of observations?

For example: for a binary classification problem with 20 instances in each class, is there any upper limit on the number of features to use?

Multiple papers have opined that

only in rare cases is there a known distribution of the error as a function of the number of features and sample size.
The error surface for a given set of instances, and features, is a function of the correlation (or lack of) between features.

This paper suggests the following:

For uncorrelated features, the optimal feature size is $ N-1 $ (where $ N $ is sample size)
As feature correlation increases, and the optimal feature size becomes proportional to $ \sqrt N $ for highly correlated features.

Another (empirical) approach that could be taken, is to draw the learning curves for different sample sizes from the same dataset, and use that to predict classifier performance at different sample sizes. Here's the link to the paper.


Decision tree not using all features from training dataset I have built CART model using sklearn. I'm having total 6 features in training dataset and passing all of them in fit function. I've tested both criteria Gini and entropy. But whenever I plot tree using graphviz, the tree uses only 3 features in case of Gini and 4 features in case of entropy. I've also implemented CART from scratch for cross check purpose and still, Gini uses 3 features and entropy uses 4 features.

Everything else is working fine. I'm getting the result as expected in test dataset with accuracy 97%. I just want to confirm that, is this normal? Does cart chooses best features only and ignore other features to avoid the wrong classification? Also, my dataset is quite small in size, only 220 records.

When a feature is not that informative of your target, the algorithm can choose not to use it. This can be for two reasons:

All the information is already in the other features, meaning it adds nothing to include it
There is some regularization going on, meaning your algorithm punishes complexity if it doesn't add enough

It can also be that the regularization is very explicit, meaning there is a maximum depth set as hyperparameter. In this case the other features are more informative and useful for your task.


Is it recommended to compute the average of metrics for classification? For computing the performance metrics of the models used for classification. Is it recommended to repeat the experiments many times (for example 10 times) and compute the average? In my case, I used 116745 samples for training and 125927 for testing.

Yes, you can try k-fold validation, where indicidual models are generated from various subsets of the data and performance measured on a validation set.
You can the compute the average and variance of model accuracy, other performance metrics to understand how stable your model is.


XGBoost most important features appear in multiple trees multiple times I am fitting xgboost model (scala-spark) to my dataset of transactions. I have about 2 millions of transactions in my training set which is highly unbalanced with a ratio of positive/negative<0.001 classes. I have now about 300 features in the model.

Then I take an output model and count for each feature

In how many trees a feature was present
In how many splits a feature was present

Then I sort the features from the ones with most trees to least trees.
I get some results that I am not sure of. Features at the top of the list with most trees and splits basically appear in each single xgboost tree multiple times. For example in xgboost with 100 rounds and colsample_bytree=1.0 and max_depth=6 I would see a feature A appear in 100 trees and in ~400 splits. Feature B appears in 98 trees and 350 splits etc... Basically it seems that all my trees are based on all the same same top features in different configurations.

Question:

Does it mean:

It normal
Do those features overfit my model
I tried to force model to take other features by decreasing colsample_bytree or colsample_bylevel and it helps but somewhat, but model performance does not dramattically improve.

Any other suggestions?

Update Observations (Feb 2019)

The features are continuous not categorical
When I sort all features based on their total gain (sum of gain in all nodes that split on the feature), the features that appear in all trees multiple times can have highest total gain or quite low total gain. There is not rule here.

To your questions:

It can be normal.
Features itself are not responsible for overfitting. It can be noise from them or just correlation without causation. I.e. Feature correlates in the training set, but not in the test set. Overfitting is related to models.
Maybe you still don't have a feature that would improve your model significantly. It's hard to say with such limited information. You need to try some experiments with feature selection.


Value function and Q-value I am new to Reinforcement Learning, and I am having trouble understanding the difference between Value function and Q-value

Here is my current understanding: Q-value is just a value for a particular action taken, and the Value function of a state sums over the Q-value of every action possible in that state.

Would this description be accurate?

Thanks!

Here is my current understanding: Q-value is just a value for a particular action taken, and the Value function of a state sums over the Q-value of every action possible in that state.
Would this description be accurate?

Not quite.
First the Q value is a type of value function, it is often called the action value. Both $Q(s,a)$ and the state value function $V(s)$ calculate the expected future return given their parameters, a known environment and a known policy $\pi$ that describes how an agent will select actions in that environment.
In the case of action value $Q(s,a)$, the expected future return is based on the agent taking the action $a$ in state $s$ first, and afterwards following $\pi$. Whilst the state value $V(s)$, the expected future return will depend on what action $\pi$ chooses in state $s$.
There a few ways to write the relationship between $V(s)$ and $Q(s,a)$. If the policy function is deterministic, action in state $s$ given by $\pi(s)$ then:
$$V(s) = Q(s, \pi(s))$$
If the policy function is stochastic, with probability of selecting action $a$ in state $s$ given by $\pi(a|s)$:
$$V(s) = \sum_a \pi(a|s)Q(s, a)$$
This is a sum, as you suggest, but weighted by the probability of taking each action.


Why is this ANN not working? I have created a Tf.Sequential model to when given an input of a number, get either a 1 if that number is bigger than 5 or 0 if it is not.

const model = tf.sequential();

      model.add(tf.layers.dense({ units: 5, activation: 'sigmoid', inputShape: [1]}));
      model.add(tf.layers.dense({ units: 1, activation: 'sigmoid'}));
      model.compile({loss: 'meanSquaredError', optimizer: 'sgd'});
      const xs = tf.tensor2d([[1], [2], [3], [4], [5], [6], [7], [8], [9]]);
      const ys = tf.tensor2d([[0], [0], [0], [0], [0], [1], [1], [1], [1]]);
      model.fit(xs, ys);
      model.predict(xs).print();

With 5 hidden neurons, not even the right trend is detected. Sometimes all the number are too low, or the outputs decrease even if the inputs increase, or the outputs are too high...

I later thought that the best way to do this is to have 2 neurons, where 1 is for the input and the other applies a sigmoid function to the input with a bias that would point towards 5 being the "decisive" number. I thought that Tensorflow.js automatically assigns biases to weights, so I simply wrote this code:

const model = tf.sequential();

      model.add(tf.layers.dense({ units: 1, activation: 'sigmoid', inputShape: [1]}));
      model.compile({loss: 'meanSquaredError', optimizer: 'sgd'});
      const xs = tf.tensor2d([[1], [2], [3], [4], [6], [7], [8], [9]]);
      const ys = tf.tensor2d([[0], [0], [0], [0], [1], [1], [1], [1]]);
      model.fit(xs, ys);
      model.predict(xs).print();

Usually, this ANN does detect the right trend (the higher the input, the higher the output), but still, the results are never correct and are usually simply too high, always providing an output too close to 1.

How do I make my ANN work, and why is what I did incorrect?

You're using the SGD optimizer, but you're only making one pass over the training data.  I think the default learning rate for SGD is something like 0.01 or 0.001, and, with such a low learning rate, your model won't be able to learn the pattern with a single pass over the data.

To fix this, you need to do one or both of the following:

raise the learning rate
train the model for more epochs (an epoch is one pass over the training data)

To raise the learning rate, you'll need to create a custom optimizer:

. . .
const learningRate = 0.5;
const sgdOptimizer = tf.train.sgd(learningRate);
model.compile({loss: 'meanSquaredError', optimizer: sgdOptimizer});
. . .

To train for more epochs, you just need to provide some additional arguments to the fit() function:

. . .
model.fit(xs, ys, {
    epochs: 25,
    batchSize: 8
});
. . .

Raising the learning rate will increase the magnitude of each weight update, allowing your model to learn more quickly, but if the learning rate is too high then the model may not converge.  Raising the number of epochs allows for more weight updates, but will increase training time.


Difference between cluster centers and means The following is the output of the cluster centers I got from a cluster model (kmeans - 6 clusters)

  3.371069, 3.920354, 3.629747, 3.700000, 3.988506, 3.740385

However, after segmenting the data into the 6 clusters and taking the average of the data for each of the 6 clusters, I get a different set of numbers than the above.

  3.7, 3.6, 3.6, 3.8, 4.2, 3.3

My question is are cluster centers different than means?

Why am I getting different values?

They are the same.

When you run K-Means, the cluster center changes every iteration. In each iteration, cluster center or mean is given for that specific cluster. In the next iteration, a new cluster may be formed. The cluster centers need to be calculated again.

When the algorithm converges, you get the absolute means of the clusters formed. For most intents and purposes, they are the same.


Removing strings after a certain character in a given text I have a dataset like the one below. I would like to remove all characters after the character ©. How can I do that in R?
data_clean_phrase <- c("Copyright © The Society of Geomagnetism and Earth",
"© 2013 Chinese National Committee ")

data_clean_df <- as.data.frame(data_clean_phrase)

For instance:

 rs<-c("copyright @ The Society of mo","I want you to meet me @ the coffeshop")
 s<-gsub("@.*","",rs)
 s
 [1] "copyright "             "I want you to meet me "

Or, if you want to keep the @ character:

 s<-gsub("(@).*","\\1",rs)
 s
 [1] "copyright @"             "I want you to meet me @"

EDIT: If what you want is to remove everything from the last @ on you just have to follow this previous example with the appropriate regex. Example:

rs<-c("copyright @ The Society of mo located @ my house","I want you to meet me @ the coffeshop")
s<-gsub("(.*)@.*","\\1",rs)
s
[1] "copyright @ The Society of mo located " "I want you to meet me "

Given the matching we are looking for, both sub and gsub will give you the same answer.


How can this CNN for the portfolio management problem be implemented in keras? In the paper "A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem" the CNN in the picture below is applied on the portfolio management problem.

I am trying to understand how this network works and can be implemented in keras.

What I don't understand is, why do we have 20+1 feature maps after the second convolution? And how I am supposed to include the portfolio w from last period in keras?

My current implementation is as follows:

def network(self):
    input1 = Input(shape=(3, 50, 4))
    conv1 = Conv2D(2, (1, 3), activation='relu',)(input1)
    conv2 = Conv2D(10, (1, 48), activation='relu')(conv1)
    conv3 = Conv2D(1, (1, 1), activation='relu')(conv2)
    flat1 = Flatten()(conv3)
    preds = Activation('softmax')(flat1)
    model = Model(input1, preds)
    model.compile(optimizer='rmsprop', loss='mse') # todo loss function
    return model

Thanks for any help.

You can concatenate your tensors in Keras. I have written the model below

from keras.models import Sequential, Model
from keras.layers import Concatenate, Dense, LSTM, Input, concatenate, Flatten
from keras.optimizers import Adagrad

price_history = Input(shape=(11, 50, 3))
feature_maps = Conv2D(2, (1, 3), activation='relu')(price_history)
feature_maps = Conv2D(20, (1, 48), activation='relu')(feature_maps)

# Add the w from the last period
w_last = Input(shape=(11, 1, 1))
feature_maps = concatenate([feature_maps, w_last], axis = -1)
feature_map = Conv2D(1, (1, 1), activation='relu')(feature_maps)
feature_map = Flatten()(feature_map)

# Add the cash bias
cash_bias = Input(shape=(1,))
feature_map = concatenate([feature_map, cash_bias], axis = -1)
w = Activation('softmax')(feature_map)

model = Model(inputs=[price_history, w_last, cash_bias], outputs=w)
print(model.summary())

model.compile(optimizer=ada_grad, loss='binary_crossentropy',
               metrics=['accuracy'])

I am not sure about the loss function, I have not read the paper in detail. But this model describes what they have in the paper.


Gradient Descent in ReLU Neural Network I’m new to machine learning and recently facing a problem on back propagation of training a neural network using ReLU activation function shown in the figure. My problem is to update the weights matrices in the hidden and output layers.

The cost function is given as:

$J(\Theta) = \sum\limits_{i=1}^2 \frac{1}{2} \left(a_i^{(3)} - y_i\right)^2$

where $y_i$ is the $i$-th output from output layer.

Using the gradient descent algorithm, the weights matrices can be updated by:

$\Theta_{jk}^{(2)} := \Theta_{jk}^{(2)} - \alpha\frac{\partial J(\Theta)}{\partial \Theta_{jk}^{(2)}}$

$\Theta_{ij}^{(3)} := \Theta_{ij}^{(3)} - \alpha\frac{\partial J(\Theta)}{\partial \Theta_{ij}^{(3)}}$

I understand how to update the weight matrix at output layer $\Theta_{ij}^{(3)}$, however I don’t know how to update that from the input layer to hidden layer $\Theta_{jk}^{(2)}$ involving the ReLU activation units, i.e. not understanding how to get $\frac{\partial J(\Theta)}{\partial \Theta_{jk}^{(2)}}$.

Can anyone help me understand how to derive the gradient on the cost function...?

Have a look at this post. I found it quite useful when starting out with neural networks.

http://neuralnetworksanddeeplearning.com/chap2.html


What is the difference between word-based and char-based text generation RNNs? While reading about text generation with Recurrent Neural Networks I noticed that some examples were implemented to generate text word by word and others character by character without actually stating why.

So, what is the difference between RNN models that predict text per-word basis and the ones that predict text per-char basis? Do word-based RNN require a bigger corpus size? Do char-based RNN generalize better? Maybe the only difference is input representation (one-hot encoding, word embeddings)? Which ones to choose for text generation?

Here is what I learnt recently.

Obviously, when talking about text generation RNNs we are talking about RNN language models. When asking about word/char-based text generation RNNs, we are asking about word/char-based RNN language models (LM).

Word-based LMs display higher accuracy and lower computational cost than char-based LMs.

  This drop of performance is unlikely due to the difficulty for character level model to capture longer short term memory, since also the Longer Short Term Memory (LSTM) recurrent networks work better with word-based input.

This is because char-based RNN LMs require much bigger hidden layer to successfully model long-term dependencies which means higher computational costs.

Therefore, we can say that

  one of the fundamental differences between the word level and character level models is in the number of parameters the RNN has to access during the training and test. The smaller is the input and output layer of RNN, the larger needs to be the fully connected hidden layer, which makes the training of the model expensive.

However, char-based RNN LMs better model languages with a rich morphology such as Finish, Turkish, Russian etc. Using word-based RNN LMs to model such languages is difficult if possible at all and is not advised.

The above analysis makes sense especially when you look at the output text, generated by char-based RNNs:

  The surprised in investors weren’t going to raise money. I’m not the company with the time there are all interesting quickly, don’t have to get off the same programmers.

While simple char-based Maximum Likelihood LM with a 13-character window delivers this:

  And when she made many solid bricks. He stacked them in piles and stomped her
  feet. The doctor diagnosed him with a bat. The girl and her boyfriend asked
  her out.

Of course I cherry-picked the example (actually most ML LM examples looked better than any RNN generated text I've read so far) and this tiny ML LM was trained on a simpler corpus but you get the idea: straightforward conditional probability generates better texts than far more complex char-based RNN.

Char-based RNN LMs can mimic grammatically correct sequences for a wide range of languages, require bigger hidden layer and computationally more expensive while word-based RNN LMs train faster and generate more coherent texts and yet even these generated texts are far from making actual sense.


How to put the variable names of pandas data frame on Diagonal of Seaborn PairGrid? The given below code is not working now(earlier it was working) to put the column names on the diagonal of Seaborn PairGrid.

import matplotlib.pyplot as plt
import seaborn as sns
iris = sns.load_dataset('iris')

def diagfunc(x, **kws):
  ax = plt.gca()
  ax.annotate(x.name, xy=(0.05, 0.9), xycoords=ax.transAxes)

sns.PairGrid(iris).map_diag(diagfunc)

At present it showing this error!

<ipython-input-137-f4d9b71087cb> in diagfunc(x, **kws)
      5 def diagfunc(x, **kws):
      6     ax = plt.gca()
----> 7     ax.annotate(x.name, xy=(0.05, 0.9), xycoords=ax.transAxes)
      8
      9 sns.PairGrid(iris).map_diag(diagfunc)

AttributeError: 'numpy.ndarray' object has no attribute 'name'

Can anyone help me how to put the column name in diagonal?

With minimal changes, I got this to work (python3.7.3, SeaBorn 0.9.0):

import matplotlib.pyplot as plt
import seaborn as sns
iris = sns.load_dataset('iris')

# Add this before your call to map_diag
next_iris_label = iter(iris).__next__

def diagfunc(x, **kws):
  ax = plt.gca()
  #  replace x.name with `next_iris_label` call
  ax.annotate(next_iris_label(), xy=(0.05, 0.9), xycoords=ax.transAxes)

sns.PairGrid(iris).map_diag(diagfunc)

Output:

I dug into it and understood that the reason this fails is because once the IRIS dataset has been passed to SeaBorn.PairGrid, the original Names of observation columns are not retained. map_diag ends up calling diagfunc with diagonal components of the PairGrid, which do not have name property, let alone name anywhere else in the arguments passed.
Here is what is passed to the first time diagfunc is called:

{
    #  this is what you passed as positional parameter
    #  As you'll notice `x` is just an array, and
    #   does not have any additional properties like a label
    'x': array([5.1, 4.9, , ... 5.9]),
    #  these get passed in as **kws
    'label': '_nolegend_',
    'color': (0.12156862745098039, 0.4666666666666667, 0.7058823529411765)
}

Now, if we could somehow iterate over all labels in IRIS dataset, we could put each one against the diagnoals of the PairGrid. Since diagfunc is called each time and has a new scope, I found that having an iterator that returned a new label each time it was called solves that.

next_iris_label defined as

next_iris_label = iter(iris).__next__

solves that problem. This only has one constraint that I dont like, you will need to reset this by creating a new instance of the iterator by the binding statement next_iris_label = iter(iris).__next__ when you want to do such labelling again(otherwise it would throw a StopIteration error.)


Custom Metrics with Keras How to define a custom performance metric in Keras?

I am trying to use it but I can not see the metrics values on each epoch.

clf51.compile(optimizer=sgd51, loss='binary_crossentropy', metrics=["accuracy"])

clf51.fit(X_train, Y_train, batch_size=384, epochs=5, callbacks=[metrics], validation_split=0.30, verbose=2)

As you can see:

Train on 139554 samples, validate on 59810 samples

Epoch 1/5 - 4s - loss: 0.3576 - acc: 0.9885 - val_loss: 0.0531 - val_acc: 0.9989

Epoch 2/5 - 3s - loss: 0.0261 - acc: 0.9987 - val_loss: 0.0135 - val_acc: 0.9987

Am I doing something wrong? I would like to make a earlystop using f1s.

The callback you are using isn't for displaying the desired metrics, just recording them. For example if you want to access the F1-score you need to type: metrics.f1s. This is useful, let's say, if you want to make a graph on how the F1-score reduced during training.

To use the EarlyStopping callback, however, f1-score needs to be a metric not a callback like you have it!

You need to write (or find) a function that calculates the F1-score through keras' backend functions. You might want to check if this works for you.


How to plot time series data from SQLite database I am trying to plot time series data by accessing the SQLite database. I successfully connect the database by coding the below script,however, i couldn't figure out how to plot the time series data from .sqlite file.

import sqlite3
from sqlite3 import Error

def create_connection(db_file):
    """ create a database connection to the SQLite database
        specified by the db_file
    :param db_file: database file
    :return: Connection object or None
    """
    conn = None
    try:
        conn = sqlite3.connect(db_file)
    except Error as e:
        print(e)

    return conn

def main():
    database = r"E:\Data\SINCAL_Models\AusNetData\20180128_PQV\pq.sqlite"

    # create a database connection
    conn = create_connection(database)
    with conn:
        print("Database connected:")

if __name__ == '__main__':
    main()

The database file is shown in attached picture. I am trying to plot the yellow highlighted variables to get the plot like here

Since you are already using python this will be very easy for you :).
I presume the table name as it is not clear to me and a db file can have several.

You can use pandas to access the data using your connections.

import pandas as pd
def fetch_table_data_into_df(table_name, conn):
    return pd.read_sql_query("select * from " + table_name, conn)

This will give you a pandas Dataframe- they are very well suited for plotting.

I usually use pylab directly but I think this should also work:

df = fetch_table_data_into_df("NameOfTable", conn)
df.plot("READ_TIME", "VALUE")

Let me know if you need more information :).

Here is the entire code (using yours and mine) as well:

import sqlite3
from sqlite3 import Error
import pandas as pd
import pylab

def fetch_table_data_into_df(table_name, conn):
    return pd.read_sql_query("select * from " + table_name, conn)

def create_connection(db_file):
    """ create a database connection to the SQLite database
        specified by the db_file
    :param db_file: database file
    :return: Connection object or None
    """
    conn = None
    try:
        conn = sqlite3.connect(db_file)
    except Error as e:
        print(e)

    return conn

def main():
    database = r"E:\Data\SINCAL_Models\AusNetData\20180128_PQV\pq.sqlite"

    TABLE_NAME = "pq_data"
    # create a database connection
    conn = create_connection(database)
    with conn:
        print("Database connected:")
        df = fetch_table_data_into_df(TABLE_NAME, conn)
        # reading what you want to do, I presumed you want
        # to plot per measuremnt
        for measurement in df.MEASUREMENT.unique():
            df[df.MEASUREMENT == measurement].plot("READ_TIME", "VALUE")
            pylab.savefig(f"{measurement}.png")
            pylab.clf()

if __name__ == '__main__':
    main()


Calculating optimal path(s) to increase probability of a classification I have trained several classifiers using Python's scikit-learn which are fairly accurate when applied on a test set at identifying different classes with a standardized set of input features. These different classifiers provide a certain probability for the classification.

The input features are controllable physical parameters that I am measuring (e.g. temperature, volume) which intricately influence an output which can essentially be either 1 or 0 (and others in mutli-class cases). I can already do basic identification, but what I am curious about is: given an initial feature vector starting in class 0, are there known methods to find the optimal ways to change my input features so as to increase my probability of going into class 1? The input feature space has a high number of dimensions, and there may be certain constraints on the inputs (e.g. temperature cannot exceed a certain value if volume is kept at a particular value).

This is a multivariate optimisation problem. You have a function f(X) that returns an output that you want to maximise - Maximising the probability of belonging to class. That function f(X) is just your model.

You also mention "constraints", and again this is standard constrained optimisation territory.

The problem itself may be hard to solve, but the framework seems to be that.

Have a look at scipy optimise here


High Recall but too low Precision result in imbalanced data I was training a model using XGBoost Classifier on heavy imbalanced data base with 232:1 of binary class. Because my training data contains 750k rows and 320 features (after doing many feature engineering, feature correlation filtering and low variance filtering), I prefer to use scale_pos_weight to dealing with imbalanced rather than oversampling data. After parameter tuning using Bayesian optimization to optimize PR AUC with 5 fold cross validation, I got the best cross validation score as below:
PR AUC = 4.87%, ROC AUC = 78.5%, Precision = 1.49%, and Recall = 80.4%
and when I tried to implement the result to a testing dataset the result is below:
accuracy: 0.562
roc_auc: 0.776293
pr_auc: 0.032544
log_loss: 0.706263
F1: 0.713779
Confusion Matrix:
[[9946 7804]
 [  18   84]]
          precision    recall  f1-score   support

       0       1.00      0.56      0.72     17750
       1       0.01      0.82      0.02       102

    accuracy                           0.56     17852
   macro avg       0.50      0.69      0.37     17852
weighted avg       0.99      0.56      0.71     17852

My parameter range to be optimize (consume 2-3 days with 100 iteration) is:
{'learning_rate':(0.001,0.2),'min_split_loss':(0,20),'max_depth':(3,10),'min_child_weight':(0,50),'max_delta_step':(0,10),'subsample':(0.5,1),'colsample_bytree':(0.5,1),'colsample_bynode':(0.5,1),'colsample_bylevel':(0.5,1),'reg_lambda':(1e-5,100),'reg_alpha':(0,1), 'objective':'binary:logistic','booster':'gbtree','scale_pos_weight':232,'n_estimators':200}

According to business request, we have more consideration to high recall (to save those in positive class), however I am frustrated by too low precision result (this is impact to the cost to save positive class). Is there any solution to increase the precision at least to be 10% without hurting the Recall?

Given that both the f1-score and PR AUC are very low even for the prevalence of ~0.45%, it can not be deduced if the limitations are imposed by the nature of the data or the model (features plus the algorithm used).
In order to build a better understanding and to resolve the issue, I would suggest to break the problem into two parts:

Build a model that works for the selected features. For this purpose, you may try creating a slightly balanced dataset 80-20? both for training and testing. Once you are satisfied with the performance of your approach, move to 2 below
Use your original imbalanced dataset and see if the situation is any better. If not, it is now clearer that the issue is with the imbalanced nature of the data, and you should try all standard techniques for dealing with imbalanced classes. I hope this helps you, because otherwise only options at hand are synthesizing or collecting more data for the minority class.


Time Series Classification with multiple rows per date I have a time series data set with the lifecycle of 9000 different B2B sales leads. What I call lifecycle consists of a dataset with one registry per day for every different sales Lead identifier with 4 predictive variables (DAYS_SINCE_START, LEAD_ID, CUSTOMER_INTEREST, MARKET, TYPE_SERVICE) and one response variable (OUTCOME). The response variable outcome can have 2 different values: Won (1) or Lost (0).

A mock example of the data frame would be the following:

As it can be seen, some leads “die” before others, this is because we receive the final outcome of the customer at that day of its lifecycle (we won or lost the lead), so that lead identifier drops from the dataset.

My mission is to create one single model that could be able to define the outcome of a new sales lead that is entering to the data set, and the prediction would be done 30 days after current time (Why 30 days after? The first 30 days after the resources of the company have already been assigned). How would I model this?

This problem can be modeled with a standard binary classification since you labeled data. Time can be considered a feature which can be encoded relative to outcome date. Random forest would be a good algorithm to try.
You need to rearrange the data to be tidy data where each LEAD_ID is in a single with row with all features for that id in the same row.


Which classification algorithms can handle 24000 features Which classification algorithms can handle 24000 features? What are their pros and cons?

Deep Learning Algorithms and Graphical model algorithms can handle that scale of features.

For example a typical parsing algorithm using CRF++ computes millions of features.
In case of Deep Learning, A typical image of 256*256*3 has to deal with 196608 number of features where each pixel in image is a feature.


xgboost performance with predicted values as input I have predicted the probability of loss using different features. Now when I used this with a non-important feature to predict the probability of loss. The first one is very close. logloss was close to 0.11. However, I have few more other features, I wanted to know if the features are important or not. So, I used the new features with this predicted probability. I found volatile behavior. Not only did the performance (logloss) drop to 0.14, but the model didn't pick the predicted probability as an important feature.

My Primary Questions:

What is the reason behind this outcome?
Should I dump every single
feature into one model, then see which feature is important ?

You could try   the next thing:,   get some   top  k predictions made with   xgboost, and use  only those as features  and   feed them  into  an LogisticRegression model.
I am currious if that will help  in case  you want  to  give a  feedback   on my proposed solution


Keras create custom network with certain unconnected neurons I am trying to create an autoencoder where a few inputs can affect the other inputs, but not the other way around. An example of this is given by the following picture. I looked into defining my own Keras layers, but don't know how to proceed.

I believe you can do this using the functional API and playing around with your input. Suppose you have the sets of input features (can be singleton sets) $a$,$b$,and $c$ for some $x$ in $X$. If you split your input into $X_a$, $X_b$, and $X_c$, where all elements in $x$ is transformed to include only features in $a$, you can theoretically build this model.

Here's a code snippet demonstrating this idea, given $X_a$, $X_b$, and $X_c$, and their corresponding shapes. You could widen this model by expanding the fully-connected layers (by increasing the variable set to 1 in the Dense calls). No promises on the code working off the bat: I've brushed over some details but this should give you a start on your problem.

input_A = Input(shape=shape_A)
input_B = Input(shape=shape_B)
input_C = Input(shape=shape_C)
A = Dense(1)(input_A)
A = Dense(1)(A)
B = Merge([input_A, input_B, input_C], mode='concat')
B = Dense(1)(B)
C = Merge([A,B],mode='concat)
C = Dense(1)(C)
B = Merge([A,B],mode='concat)
B = Dense(1)(B)
A = Dense(1)(A)

From here, if you want one output, you can call merge on $A$, $B$, and $C$ and then tack on a Dense (Fully-Connected) layer with, for example, a Softmax layer for multi-label classification.