Why positive-unlabeled learning? Machine learning can be divided into several areas: supervised learning, unsupervised learning, semi-supervised learning, learning to rank, recommendation systems, etc, etc.

One such area is PU Learning, where only Positive and Unlabeled instances are available.

There are many publications about this, usually involving a lot of mathematics...

When looking at the literature, I was expecting to see methods similar to self-training (from semi-supervised learning), where labels are adjusted gradually according to the classifier margins.

I don't think these is what practitioners from the area do, and I was unable to navigate the mathematics or to find a survey on PU learning.

Could someone from the area perhaps clarify what said practitioners do? Why can they not just use a binary classifier where the negative class=unlabeled? Can negative labels exist among the unlabeled data? What is the goal and what metrics exist to evaluate said goal?

A couple of points I have since found myself:

I was right in suspecting that self-training could be used for PU learning. In fact, I found the original paper on PU Learning, and indeed the paper is a variation on self-training. (Oddly enough, the original authors had Positive, Unlabeled and Negative examples!)
The authors of this survey identify three families of methods: (i) two-step strategy (identify reliable negative examples in the unlabeled data and then use supervised learning), (ii) weight the positive and unlabeled
examples, and estimate the conditional probability of positive label given an example (I believe this is akin to semi-supervised self-training), and (iii) just treat the unlabeled data as highly noisy negative data.
There are also some interesting loss functions to be used with neural networks (and I imagine could be adapted for gradient boosting) described in Table 1 of this paper.


Dealing with NaN for predictive models I have data set that has data for patients:

Arrival_Date : is when the patient has arrived

Seen_By_Nurse : is number of minutes patient take to be seen by nurse since arrival
when value is NaN it means patient is still waiting and has not been seen by Nurse yet

Seen_By_Dr: is number of minutes patient take to be seen by dr since arrival
when value is NaN it means patient is still waiting and has not been seen by Dr yet

Upset_Patient : Is this patient upset or not 1 or 0

The data look like this:

Arrival_Date   Seen_By_Nurse   Seen_By_Dr   ... many other fields ...   Upset_Patient
14/8/19 14:15  10              25                                       1
14/8/19 15:32  13              NaN                                      0
14/8/19 15:35  16              NaN                                      1
14/8/19 15:39  NaN             NaN                                      0
14/8/19 15:44  NaN             NaN                                      1

My question is: I know that working with NaN failing many predictive models. Therefore, I have to either fill it with other values, such as mean. Which is going to be wrong in my case. Or delete it which also going to be wrong in my case as the majority has not been seen by Dr or Nurse yet.

Should I fill it with big number such as 9999? Are there any other approaches?

What is the best way to deal with NaN in this case?

Going according to your problem, you should set the NaN values to 0. It follows logically as the number of minutes spent with the nurse or doctor will be zero if the patient hasn't visited them yet. And yes as you said filling with mean is not correct as it would not represent the problem correctly.

Also, if you fill the NaN values with big number such as 9999 they might be treated as outliers which is not the case. Hence, purely thinking from a logical point of view you should replace them with 0. But I also suggest you to run trial experiments with some other values such as large values. Machine Learning is a lot about hyperparameter tuning and in this case how you replace your NaN values behaves like a hyperparameter!


How to test dev set on Time Series data via forecasting I'm implementing $3$ Bayesian Deep Learning models (links below) for my masters. I'm supposed to test them on a civil engineering time series data.

My models should take a time series covariate vector ($X_t$ = {$x1_t$, $x2_t$, ...}) as input and predict single values $y_t$. I will use past values of $X$, (e.g. $X_{t-1}$, $X_{t-2}$) on each $y_t$ but the models won't be feed with past values of $y_t$ because these will not be available on a real situation. The data was generated daily, so every t is a new day. Summing up, the model shall take windows of X values (e.g. $X_{t-1}$, $X_{t-2}$ ...) with a fixed window size, and them predicting a single y value. We will have multiple pairs $((X_{t},X_{t-1}),y_t)$ as our training data, for multiple values of $t$.

I have several years of data, and I've separated the data on some arbitrary point T so that the first years are used as training, and the last months of the last year of data are used as test/dev data.

So, training examples are generated until $T (X_T,y_T)$, and them $T+1, T+2...$ are used to test the models.

The only value that matters on this forecasting task is the next day value ($y_{T+1}$).

Initially, I was training the model until day T, and then using that model to predict the annotations $y_{T+n}$ until the dev data stops. But then the people that gave me the data asked about retraining the model every new day of the dev set, and then predicting the next day:

For example, on the first training, I would train until day T and then forecast T+1. Then I would add the real value of $y_{T+1}$ to the train set, retrain the models, and forecast T+2 and so on.

Which approach would be more proper? The accuracy of the models drop when I try to retrain the models every day to predict just a single day, I can't understand why.

I'm using DeepAR, DeepFactors, Enc-Dec-Forecaster (by Uber) and an ARIMA model just to compare. I'd really appreciate some help!

EDIT: For clarity

I feel your frustration. I am not sure if this is a complete answer, but I’ll try to add some value here.

Disclaimer: I am not familiar with the neural nets you mentioned but have used other times series nns. So my answers will be general.

First, there is nothing specifically wrong about retraining every day unless you are doing so at the expense of the number of training epochs. For example, you could conceivably need to train for 20k epochs to get the best predictive power. If this takes 20 hours, it may not feasible. So you may be tempted to train for 5000 epochs. This is probably a reason not to retrain every day. My recommendation would be to invest in more epochs and retrain less frequently. Hopefully the resulting model is more generalized and powerful in the long run.

If you DO still want to train every day, I would load the weights and biases from the previous training session and continue training for a smaller number number of epochs. Start with as many epochs as you can to produce the first model. Then add the additional row and continue training for a manageable number of epochs. TBH, one extra row should not really change your model weights and biases significantly unless the new y is pretty extreme. In fact, I doubt it would even be obvious.

Now, why does your accuracy drop with the new data? Several ideas:

Just stating the obvious. In case it’s not. You will need many training epochs. Each application is different, but sometimes people only train for a few epochs and don’t understand why the model sucks. You might need 200 epochs, you might need  2000000.
You are overfitting and the addition of the new data is generalizing the model a little more each day. Doubtful, but possible.
You are reinitializing your model with random weights and biases and do not have a seed set. So your model could be finding different local minimums each time you train. I would be surprised if it was always in the negative, but it may be that you have just flipped heads 25 times in a row. I would set a seed when initialize to take this out of the picture.

Hopefully there’s something of use in here.


Normal equation for linear regression I am going through the derivation of normal equation for multivariate linear regression. The equation is given by :

$\theta = (X^{T}X)^{-1}X^{T}Y$

The cost function is given by:

$J(\theta) = \frac{1}{2m}(X\theta-Y)^{T}(X\theta-Y)$

Simplifying,

$J(\theta) = \frac{1}{2m}(\theta^{T}X^{T}X\theta - 2(X\theta)^{T}Y + Y^{T}Y)$

Differentiating w.r.t $\theta$ and equating to zero

$\frac{dJ(\theta)}{d\theta} = \frac{d}{d\theta}(\theta^{T}X^{T}X\theta)-\frac{d}{d\theta}(2(X\theta)^{T}Y) = 0$

I want to specifically understand the differentiation of the left term:

$\frac{d}{d\theta}(\theta^{T}X^{T}X\theta) = X^{T}X\frac{d}{d\theta}(\theta^{T}\theta) $

$\frac{d}{d\theta}(\theta^{T}\theta) = [\frac{d}{d\theta_1}(\theta_1^{2}+\theta_2^{2}+...\theta_n^{2}),\frac{d}{d\theta_2}(\theta_1^{2}+\theta_2^{2}+...\theta_n^{2}) ,...., \frac{d}{d\theta_n}(\theta_1^{2}+\theta_2^{2}+...\theta_n^{2})]$

$\frac{d}{d\theta}(\theta^{T}\theta) = [2\theta_1,2\theta_2,...,2\theta_n] $

$\frac{d}{d\theta}(\theta^{T}\theta) = 2\theta^{T} $

But the final equation is obtained by using $\frac{d}{d\theta}(\theta^{T}\theta) = 2\theta$

How is $\frac{d}{d\theta}(\theta^{T}\theta) = 2\theta$ and not $2\theta^{T}$

It is basically a matter of convention, which becomes a bit more clear if you write the whole thing in terms of elements, rather than vectors. Consider

$$ \theta^T \theta = \sum_{n=1}^N \theta_n \theta_n = \sum_{n=1}^N \theta_n^2 $$

What is usually meant if you write $\frac{df}{d\theta}$ is that you take the gradient of a scalar $f$, i.e. you get a vector $\frac{df}{d\theta}$ where each element $i$ is the derivative with respect to the respective coordinate $\theta_i$:

$$ \left( \frac{df}{d\theta} \right)_i = \frac{df}{d\theta_i} $$

Let's apply this to $\theta^T\theta$:

$$ \left( \frac{d}{d\theta} \theta^T\theta \right)_i = \frac{d}{d\theta_i} \sum_{n=1}^N \theta_n^2 = 2 \sum_{n=1}^N \theta_n \frac{d\theta_n}{d\theta_i}  $$

and since $\frac{d\theta_n}{d\theta_i} = \delta_{i,n}$:

$$ \left( \frac{d}{d\theta} \theta^T\theta \right)_i = 2 \theta_i $$

If you interpret this as columns or rows is pretty much up to you. Commonly we would take $\theta_i$ as the elements of a column vector $\theta$ and for practicality we usually want the gradient to live in the same vector space as our coordinates, so $\left( \frac{df}{d\theta} \right)_i$ would also be a column vector. Hence

$$ \frac{d}{d\theta} \theta^T\theta = 2 \theta $$


K means visualisation after reducing dimensionality with PCA In clustering ($K$ means, for example) when I have $N$ features and after creating the model (with this $N$ features) to visualize this model I need to reduce this $N$ dimensions into $2$ or $3$ dimensions, let's  say I will use (PCA) fro example.

My question is how I can analyze the results (grape with principal component)?

this is simple example :

Data without reducing :
AGE,GENDER,SPENT,SALARY,CAR,.....

Data after reducing dimension:

Principal Component $1$, Principal Component $2$, Principal Component 3

what do  PC1,PC2,PC3 means

K-means minimizes sum of squared errors, and PCA finds the projection with maximum sum of squares. So they are a quite natural fit.

Just run k-means and project it to 2d for visualization with PCA. They you are largely seeing the data the same way as k-means (if you only use the rotation, not the scaling!)

What I'd be more concerned about is the input data, as it is not particularly well suited for neither k-means nor PCA. So I wouldn't be surprised if the results are barely interpretable. Both methods make most sense if the input variables are continuous and of the same scale.


How to calculate mAP for detection task for the PASCAL VOC Challenge? How to calculate the mAP (mean Average Precision) for the detection task for the Pascal VOC leaderboards?
There said - at page 11:

Average Precision (AP). For the VOC2007 challenge, the interpolated
average precision (Salton and Mcgill 1986) was used to evaluate both
classification and detection. For a given task and class, the
precision/recall curve is computed from a method’s ranked output.
Recall is defined as the proportion of all positive examples ranked
above a given rank. Precision is the proportion of all examples above
that rank which are from the positive class. The AP summarises the
shape of the precision/recall curve, and is defined as the mean
precision at a set of eleven equally spaced recall levels
[0,0.1,...,1]:
AP = 1/11 ∑ r∈{0,0.1,...,1} pinterp(r)
The precision at each recall level r is interpolated by taking the
maximum precision measured for a method for which the corresponding
recall exceeds r: pinterp(r) = max p(r˜), where p(r˜) is the measured
precision at recall ˜r

About mAP

So does it mean that:

We calculate Precision and Recall:

A) For many different IoU > {0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1} we calculate True/False Positive/Negative values
Where True positive = Number_of_detection with IoU > {0, 0.1,..., 1}, as said here and then we calculate:
Precision = True positive / (True positive + False positive)
Recall = True positive / (True positive + False negative)

B) Or for many different thresholds of detection algorithms we calculate:
Precision = True positive / (True positive + False positive)
Recall = True positive / (True positive + False negative)
Where True positive = Number_of_detection with IoU > 0.5 as said here

C) Or for many different thresholds of detection algorithms we calculate:
Precision = Intersect / Detected_box
Recall = Intersect / Object
As shown here?

Then we build Precision-Recall curve, as shown here:

Then we calculate AP (average precision) as average of 11 values of Precision at the points where Recall = {0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1}, i.e. AP = 1/11 ∑ recall∈{0,0.1,...,1} Precision(Recall)

(In general for each point, for example 0.3, we get MAX of Precision for Recall <= 0.3, instead of value of Precision at this point Recall=0.3)

And when we calculate AP only for 1 something object class on all images - then we get AP (average precision) for this class, for example, only for air.

So AP is a  integral (area under the curve)
But when we calculate AP for all object classes on all images - then we get mAP (mean average precision) for all images dataset.

Questions:

Is it right, and if it isn't, then how to calculate mAP for Pascal VOC Challenge?
And which of the 3 formulas (A, B or C) is correct for calculating Precision and Recall, in paragraph 1?

Short answer:

mAP = AVG(AP for each object class)
AP = AVG(Precision for each of 11 Recalls {precision = 0, 0.1, ..., 1})
PR-curve = Precision and Recall (for each Threshold that is in the Predictions bound-boxes)
Precision = TP / (TP + FP)
Recall = TP / (TP + FN)
TP = number of detections with IoU>0.5
FP = number of detections with IoU<=0.5 or detected more than once
FN = number of objects that not detected or detected with IoU<=0.5

To answer your questions:

Yes your approach is right
Of A, B and C the right answer is B.

The explanation is the following: In order to calculate Mean Average Precision (mAP) in the context of Object Detection you must compute the Average Precision (AP) for each class, and then compute the mean across all classes. The key here is to compute the AP for each class, in general for computing Precision (P) and Recall (R) you must define what are: True Positives (TP), False Positives (FP), True Negative (TN) and False Negative (FN). In the setting of Object Detection of the Pascal VOC Challenge are the following:

TP: are the Bounding Boxes (BB) that the intersection over union (IoU) with the ground truth (GT) is above 0.5
FP: two cases (a) BB that the IoU with GT is below 0.5 (b) the BB that have IoU with a GT that has already been detected.
TN: there are not true negative, the image are expected to contain at least one object
FN: those ground truthes for which the method failed to produce a BB

Now each predicted BB have a confidence value for the given class. So the scoring method sort the predictions for decreasing order of confidence and compute the P = TP / (TP + FP) and R = TP / (TP + FN) for each possible rank k = 1 up to the number of predictions. So now you have a (P, R) for each rank those P and R are the "raw" Precision-Recall curve. To compute the interpolated P-R curve foreach value of R you select the maximum P that has a corresponding R' >= R.
There are two different ways to sample P-R curve points according to voc devkit doc.
For VOC Challenge before 2010, we select the maximum P obtained for any R' >= R, which R belongs to 0, 0.1, ..., 1 (eleven points). The AP is then the average precision at each of the Recall thresholds. For VOC Challenge 2010 and after, we still select the maximum P for any R' >= R, while R belongs to all unique recall values (include 0 and 1). The AP is then the area size under P-R curve. Notice that in the case that you don't have a value of P with Recall above some of the thresholds the Precision value is 0.
For instance consider the following output of a method given the class "Aeroplane":
BB  | confidence | GT
----------------------
BB1 |  0.9       | 1
----------------------
BB2 |  0.9       | 1
----------------------
BB3 |  0.7       | 0
----------------------
BB4 |  0.7       | 0
----------------------
BB5 |  0.7       | 1
----------------------
BB6 |  0.7       | 0
----------------------
BB7 |  0.7       | 0
----------------------
BB8 |  0.7       | 1
----------------------
BB9 |  0.7       | 1
----------------------

Besides it not detected bounding boxes in two images, so we have FN = 2. The previous table is the ordered rank by confidence value of the predictions of the method GT = 1 means is a TP and GT = 0 FP. So TP=5 (BB1, BB2, BB5, BB8 and BB9), FP=5. For the case of rank=3 the precision drops because BB1 was already detected, so even if the object is indeed present it counts as a FP. .
rank=1  precision=1.00 and recall=0.14
----------
rank=2  precision=1.00 and recall=0.29
----------
rank=3  precision=0.66 and recall=0.29
----------
rank=4  precision=0.50 and recall=0.29
----------
rank=5  precision=0.40 and recall=0.29
----------
rank=6  precision=0.50 and recall=0.43
----------
rank=7  precision=0.43 and recall=0.43
----------
rank=8  precision=0.38 and recall=0.43
----------
rank=9  precision=0.44 and recall=0.57
----------
rank=10 precision=0.50 and recall=0.71
----------

Given the previous results:
If we used the way before voc2010, the interpolated Precision values are 1, 1, 1, 0.5, 0.5, 0.5, 0.5, 0.5, 0, 0, 0. Then AP = 5.5 / 11 = 0.5 for the class of "Aeroplanes".
Else if we used the way since voc2010, the interpolated Precision values are 1, 1, 1, 0.5, 0.5, 0.5, 0 for seven unique recalls that are 0, 0.14, 0.29, 0.43, 0.57, 0.71, 1.Then AP = (0.14-0)*1 + (0.29-0.14)*1 + (0.43-0.29)*0.5 + (0.57-0.43)*0.5 + (0.71-0.57)*0.5 + (1-0.71)*0 = 0.5 for the class of "Aeroplanes".
Repeat for each class and then you have the (mAP).
More information can be found in the following links 1, 2. Also you should check the paper: The PASCAL Visual Object Classes Challenge: A Retrospective for a more detailed explanation.


Intuition / Importance of intermediate supervision in deep learning These days, I have seen many papers using intermediate supervision.

Single Network

When using a single neural network, multiple neurons output predictions, perhaps by processing data in different ways. Then, the loss function sums up the individual losses computed over each prediction.

For example, consider a part of the FlowNet architecture below:

In this network, all the Convolution# layers (Convolution1, Convolution2, ...) output predictions at different stages of the network. Then, the loss function is computed by applying a mean-squared-error over all these predictions separately and adding them all.

Multiple Networks

When using multiple networks, like in Stacked Hourglass Networks:

Each individual network outputs a prediction, and the overall loss function is computed by computing a mean-squared error over each network's prediction and summing them all up.

My question is: what is the intuition behind doing this? I thought that this will force the first network to learn to predict the task well, while the remaining networks just perform identity transformations. Why is this not observed in practice?

Also, I have only seen this applied in CNNs, but I could be wrong.

Unfortunately there is no mathematical proof whether or not the intermediate supervision improve the performance of the network.

However stack hourglass article shows that according to ablation study: using more encoder – decoder units with intermediate supervision improve the performance – please have a look at the figure below.

In that case – the remaining network doesn't perform identity transformation (although there is a residual connection between the remaining networks).

My intuition is the every encoder – decoder serves as a PCA,  that keep the most dominant feature maps.  A series of encoder – decoder let the network reassess the feature maps until it get its most discriminate feature maps at the end.


Visitor's probability to purchase on eCommerce site, based on aggregate historic data On an eCommerce website we want to create some personalization for visitors who are more likely to make a purchase.

Let's assume we only have one single item for sale.

The likelihood should be based on data that we have, for example:

Price of the item when the customer visited (it may vary)
Average customer reviews rating of the item
If a discount coupon was offered or not
If certain words were present in the item title or description
etc.

Our plan is to create a DB that will contain each visitor's visit data (the above data), and indication if eventually he made a purchase or not.

Once we have at least a few purchases, we need to somehow look for a pattern among all these cases, and based on this pattern we can later forecast the likelihood.

For example, if at least in 50% of the cases that the item was purchased, its price was no more than $100, than we will make personalization only on cases where the item is being sold at a max price of $100.

Does this make sense?

How should we do it technically? From both DB structure concerns, and website speed concern -

Should we collect each variable in its own DB column, or is it better to create some kind of long string in the browser session like p:99-r:4,  (which means: price $99, rating: 4) and then later break this data down using another script?

Is there some kind of library that can analyze this data and show it maybe in graph, like some fancy fronted, or does this project need to be made from scratch?

What kind of technical skills does the person who do it need to have? Is this related to machine-learning?

This is a big topic known as Recommender Systems or Recommendation Engines.

The most common method is use what is known as collaborative filtering. Most methods on the internet try to guess what the user will rank an item, but I think this can be extended to if the user will make a purchase or not, given that if a user were to rank something highly they would also be more likely to buy that item.

The idea is that each item has a n-dimensional vector, and each user has an n-dimensional vector. You can predict a user's predicted rating by some operation between the two vectors. In this example lets use the dot product between the two vectors as a means to predict the user's rating for an item they haven't yet seen.

While taking the 2 vectors and getting a prediction is not a machine learning problem, finding suitable values for the vectors is most definitely a machine learning problem. I won't get too deep into how you can get these vectors because many different algorithms apply and it seems that you are asking more of a system design question.

Once you have trained 1 vector for each user and 1 vector for each item, you have 2 more problems to solve:
1. Estimate the top N candidate items to recommend
2. Take the dot product between each of these N items and recommend the few with the highest score.

Number 1 will likely involve some use of spatial indexing in your database (example PostGIS for PSQL) so you can find some vectors that would give reasonably good dot-products. You may want to consider using a ball tree data structure to search for a good set of candidate items.

Number 2 will be pretty simple by just dotting the vectors and picking the best few.

It will be a big challenge for you to undertake but I'm certain you'll learn a lot. Since it was such a big system you want to create I wasn't able to go into detail but hopefully, I pointed you in the right direction.


Identifying potential customers based on their Rank and Value I have a dataset which has demographic data available for a list of new customers.
the data does'nt include transaction data of the customers.
I want to identify the top 100 potential customers among these customers. Im aware that we can make use of clustering to segment these customers.However, I have two more variables in my data which are Rank and Value.
What approach should be taken when rank and value of customers are given.How do we interpret the clusters given these 2 variables.
Need some guidance on this

Do you have some data on past clients ? For example, do you have a dataset with past clients Rank and Value, and a variable indicating if the client was good or not (binary variable 1 if the client is good and 0 if it's bad, or a variable to use as indicator, like profit done on the client for example).
If you have such a thing, your problem is a basic classification problem, where your threshold is not 0.5 (the probability you want to be "Good Client" is not 0.5, as it is on default cases), but where you just select the 100 customers with the best probability (closer than 1).
If you don't have such dataset, and such variable indicating if the client was good or bad, based on what you know from the past, then you can't create a model since you don't know how to train it.


Aggregate Categorical Data I have a scenario in which I'm required to run my analysis at the Account level. One of the features that I'd like to look at is the no. of subscriptions against an account. There can be multiple subscriptions against one account. I wonder how I can "aggregate" these multiple subscriptions and roll them up at the Account level, such that I have a single row for each account.

I could think of binary encoding but I have 5000 products and that would require creating these many features.

You can use pandas groupby function to group your rows according to  accounts and then perform your desired operation on them.


Implementing U-Net segmentation model without padding I'm trying to implement the U-Net CNN as per the published paper here.
I've followed the paper architecture as closely as possible but I'm hitting an error when trying to carry out the first concatenation:

From the diagram, it appears the 8th Conv2D should be merged with result of the 1st UpSampling2D operation, however the Concatenate() operation throws an exception that the shapes don't match:
def model(image_size = (572, 572) + (1,)):

    # Input / Output layers
    input_layer = Input(shape=(image_size), 32)

    """ Begin Downsampling """

    # Block 1
    conv_1 = Conv2D(64, 3, activation = 'relu')(input_layer)
    conv_2 = Conv2D(64, 3, activation = 'relu')(conv_1)

    max_pool_1 = MaxPool2D(strides=2)(conv_2)

    # Block 2
    conv_3 = Conv2D(128, 3, activation = 'relu')(max_pool_1)
    conv_4 = Conv2D(128, 3, activation = 'relu')(conv_3)

    max_pool_2 = MaxPool2D(strides=2)(conv_4)

    # Block 3
    conv_5 = Conv2D(256, 3, activation = 'relu')(max_pool_2)
    conv_6 = Conv2D(256, 3, activation = 'relu')(conv_5)

    max_pool_3 = MaxPool2D(strides=2)(conv_6)

    # Block 4
    conv_7 = Conv2D(512, 3, activation = 'relu')(max_pool_3)
    conv_8 = Conv2D(512, 3, activation = 'relu')(conv_7)

    max_pool_4 = MaxPool2D(strides=2)(conv_8)

    """ Begin Upsampling """

    # Block 5
    conv_9 = Conv2D(1024, 3, activation = 'relu')(max_pool_4)
    conv_10 = Conv2D(1024, 3, activation = 'relu')(conv_9)

    upsample_1 = UpSampling2D()(conv_10)

    # Connect layers
    merge_1 = Concatenate()([conv_8, upsample_1])

Error:
Exception has occurred: ValueError
A `Concatenate` layer requires inputs with matching shapes except for the concat axis. Got inputs shapes: [(32, 64, 64, 512), (32, 56, 56, 1024)]

Note that the values 64 and 56 correctly line up with the architecture.

I don't understand how to implement the model as it is in the paper. If I change my code to accept an image of shape (256, 256) and add padding='same' to the Conv2D layers, the code works as the sizes are aligned.
This seems to go against what the authors specifically state in their implementation:

Could somebody point me in the right direction on the correct implementation of this model?

$\hspace{3cm}$
If we follow the definition of each arrow.
Gray => Copy and Crop

Every step in the expansive path consists of an upsampling of the
feature map followed by a 2x2 convolution (“up-convolution”) that halves the
number of feature channels, a concatenation with the correspondingly cropped
feature map from the contracting path, and two 3x3 convolutions, each followed by a ReLU. The cropping is necessary due to the loss of border pixels in
every convolution. Paper

So, believe(I have added 3 coloured circles)

Blue - 28x28 is upsampled and become 56x56, 1024 is halved to 512
Red - 64x64 is cropped to 56x56. Then Concatenated along FM axis.
Black - 3x3 convolutions, followed by a ReLU


How to Include Features that Apply to Specific Classes I'm predicting hours that will be worked for building tasks. Due to the overall low sample size, I've stacked multiple related tasks together into a single model. (There may be 100 total samples in a single model, each task having 10 to 20 samples individually) An example would be - how long will it take a worker to complete each task associated with installing 2 different sizes of pipe in a hospital.
There are many tasks associated with installing a pipe -

Cutting the pipe
Welding the pipe
Bending the pipe
Riveting the pipe

We know from experience that the more bends a pipe has - the more difficult it is weld. But the difficulty of cutting and riveting are completely unrelated to the number of bends. Additionally there are multiple sizes of pipe in a single model, and the above tasks are completely unrelated between different sizes of pipe. An example of the data is:

| Task       | Pipe Size | Amount | Ratio of Bends to Welds | Predicted Hours |
|------------|-----------|--------|-------------------------|-----------------|
| Cut Pipe   | 3 inches  | 5      | NULL                    | 2               |
| Weld Pipe  | 3 inches  | 10     | 2                       | 4               |
| Bend Pipe  | 3 inches  | 20     | NULL                    | 8               |
| Rivet Pipe | 3 inches  | 10     | NULL                    | 2               |
|            |           |        |                         |                 |
| Cut Pipe   | 10 inches | 1      | NULL                    | 1               |
| Weld Pipe  | 10 inches | 2      | 5                       | 2               |
| Bend Pipe  | 10 inches | 10     | NULL                    | 15              |
| Rivet Pipe | 10 inches | 1      | NULL                    | 0.5             |

There are many different types of these "ratio" features within a single model, my current plan is to include them and null out the feature in all other tasks where it isn't relevant. It's the first time I've stacked this many classes together in a single model, and also the first time I've encountered features which are only applicable to some rows and not others. I'm currently using a random forest model. Is there anything conceptually wrong with doing this?

If I understood correctly, you said that  Pipe Size does not have a correlation with the Predicted Hours. If you are sure that one or more variables do not possess the information about the target then drop them.
Other than that, if a feature (or some of your features) has a relation only with a specific task, then replace the null values with 0.
But in your case, I think you should also try the simpler algorithms like Linear Regression. But be careful about the sample size vs a number of features balance. Since you have few samples, use at most 1 variable for every 15 samples (In rare cases, 10 can be used too, but it is not recommended). As this source mentions:

In summarizing the findings by Schmidt, Green suggested that the minimum number of SPV ranges from 15 to 25.

Also, you need to evaluate your model's performance, but since you may have an unbalanced number of samples for each Task Type, be careful while splitting your data to train and test, make sure your test data has enough samples from each Task Type.
Some other important things:

Try to do bivariate (between the target variable and an independent variable) analysis before modeling, so you can eliminate unnecessary variables.
In the case of using Linear Regression, be careful about multicollinearity if you will use the coefficients to interpret your results. Because if your model has multicollinearity, then your coefficients will be incorrect although your predictions will note be spurious i.e. they will still be trustable.


how many rows have values from the same columns pandas I have a df with many columns that represent the market cap of companies that compose an index. The index of the dataframe is dates. Before the company enters the index or after it leaves it, the market cap of the company in the df is zero.

I want to know the mean number of days in which the index doesn't change.

for example:

[in]:
df = pd.DataFrame(np.array([[1, 1,np.nan], [np.nan,2, 10], [1,3, 100],[4,np.nan, 100]]),
                           columns=['a', 'b','c'])

df
[out]:
        a   b   c
    0   1.0 1.0 NaN
    1   NaN 2.0 10.0
    2   1.0 3.0 100.0
    3   4.0 NaN 100.0

what I want to know is how many rows have entries from the same column.

For example, row 0 has entries from columns a and b. row 1 from columns b and c. row 2 from columns a,b and c. an row 3 from columns a and c. therefore there are 4 rows with unique column combinations and 4 changes. The mean is then 1.

By building a tuple of whether a value is nan or not, you can then construct each unique like:

Code:

def get_average_run_length(a_df):
    unique_run_lengths = set(
        tuple(not np.isnan(x) for x in row[1]) for row in df.iterrows())
    return len(a_df) / len(unique_run_lengths)

Test Code:

df = pd.DataFrame(
    [[1, 1, np.nan], [np.nan, 2, 10], [1, 3, 100], [4, np.nan, 100]],
    columns=list('abc'))

print(df)
print(get_average_run_length(df))

# double the last row
df = pd.concat([df, df.tail(1)])
print(df)
print(get_average_run_length(df))

Results:

     a    b      c
0  1.0  1.0    NaN
1  NaN  2.0   10.0
2  1.0  3.0  100.0
3  4.0  NaN  100.0

1.0

     a    b      c
0  1.0  1.0    NaN
1  NaN  2.0   10.0
2  1.0  3.0  100.0
3  4.0  NaN  100.0
3  4.0  NaN  100.0

1.25


How to fit an odd relationship with a function? Let's say there is a function $f$ such that $y = f(x)$. However, if $f$ is a piecewise function such that:

$$y = \begin{cases} 0 \quad x \leq 0 \\ 1 \quad x >0\end{cases} $$

How do I fit $f$ in that case?

Many thanks, guys.

The definition you gave is the definition of the function. This is called the Heaviside Step Function. There is not a simple analytic way to express it (like as a ratio, product, or composition  of trigonometric functions, exponentials, or polynomials). Note that it is neither continuous nor differentiable at x = 0.

There are a couple of cool ways to represent it. The coolest and most intuitive way is as an integral of a Dirac Delta Function:

$$
H(x) = \int_{-\infty}^x { \delta(s)} \, \mathrm{d}s
$$

Note, though, that a Dirac Delta Function is itself not an "official" function, since it is not well-defined at x = 0. Check out Distribution Theory for some cool info on weird "functions" like this.

Now, I think you may be trying to approximate this function, because you asked how to "fit" it. Taken straight from Wikipedia:

For a smooth approximation to the step function, one can use the logistic   function

$$
H(x) \approx \frac{1}{2} + \frac{1}{2}\tanh(kx) = \frac{1}{1+\mathrm{e}^{-2kx}},
$$

where a larger k corresponds to a sharper transition at x = 0.


Bad Input Shape -- How to interpret and Diagnose; Also side ML question I apologize I am a ML novice, but I am trying to learn. I am making a classifier based on this dataset to predict mental health disorders based on features. I wanted to run a very simple NB classifer model but I keep getting a bad input shape error (I want to feed in features such as age, ethnicity and gender to yield potential diagnoses). Unfortunately, I am having trouble diagnosing where my error is coming from and troubleshooting. Any guidance? (ignore the multiple input stuff at the top; I was trying different things but I am assuming that there is a problem with how I am inputting the data parameters)
Namely, for these labels (diagnoses) I want an output that will show the presence/lack thereof each [0 or 1] based on the features (which are numeric)
Feature Names
['YEAR', 'AGE', 'EDUC', 'ETHNIC', 'RACE']
Values
[ 9, -9,  4 , 2]
Labels:
['ADHDFLG', 'CONDUCTFLG', 'DELIRDEMFLG', 'BIPOLARFLG', 'DEPRESSFLG', 'ODDFLG', 'PDDFLG', 'PERSONFLG', 'SCHIZOFLG', 'ALCSUBFLG']
Corresponding Label values
[0, 1, 0, 0, 0, 1, 0, 0, 0, 0]
Also, side question -- does anyone have any recommendations for other Maching Learning tasks I can try with this? I am doing this for a class and am trying to push myself to learn new topics.
Thanks in advance!
import pandas as pd
import numpy as np
from sklearn.preprocessing import LabelEncoder
import random
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.naive_bayes import GaussianNB

import scipy

from sklearn.model_selection import train_test_split

df = pd.read_csv("https://csprojectdatavisualizationsample50k.s3.us-east-2.amazonaws.com/sample_df.csv")
df_columns = df.columns
df_feature_names = (df_columns[1:6]).to_list()
df_features = df.iloc[:,2:6].values
df_label_names = (df_columns[26:36]).to_list()
df_labels = df.iloc[:, 26:36].values
#Input
print(df_label_names)

# Split our data
train, test, train_labels, test_labels = train_test_split(df_features,
                                                          df_labels,
                                                          test_size=0.50,
                                                          random_state=42)

print(train.shape)
print(test.shape)

# Initialize our classifier
gnb = GaussianNB()

# Train our classifier
model = gnb.fit(train, train_labels)

# Make predictions
preds = gnb.predict(test)
print(preds)

According to doc GaussianNB can handle multiple classes, but its fit function still accepts a one dimensional array. This array is allowed to contain multiple classes like: [0, 1, 2, 3,...]
When I changed your train labels as below fit function worked:
train_labels = np.random.randint(0, 9, 25000)

So you can create an array for labels. If each sample only belongs to one class then you can label this sample with corresponding class, but if there are some samples that have multiple classes at the same time then you should use one of them, or create groups that consist of these classes, and label sample if it belongs to this group or not.


Scaling continuous data to discrete range Edit: the context is as follows: I've trained some ML model that predicts some feature vector. Thats a. But I know that a can take some values from discrete range, but the model outputs continuous values. After applying predict I want to enforce the outputs to take the discrete values they are allowed to be in. I've also edited my example to be more informative.
I have this continuous data:
a = [0.003 0.994 1.334 3.2 1.464 2.454 2.941 999.999] (outputs)
How can I easily convert it to the following discrete array:
b = [0 1 2 3 3] (allowed values)
so that every element of a will be mapped to its closest counterpart in b:
a_scaled = [0 1 1 3 1 2 3 3] (what I need)
I know how to implement this, but I want to stick with "don't invent the wheel". So is there any nice function from some library that does this?

Update Accordingly: Your question was not clear before, therefore, sorry for the irrelevant solution.
To achieve that, I don't think there is a public library function, however, you can build your solution using some beautiful "ready" functions.
Two solutions come into my mind:

First one's time complexity is O(N*M)

N is your prediction (list a in your case) size, and M is your dictionary (list b in your case) size.
import numpy as np

def findClosest(dictionary, value):
    idx = (np.abs(dictionary - value)).argmin()
    return dictionary[idx]

#[findClosest(b, elem) for elem in a]

print([findClosest(b, elem) for elem in a])

This just subtracts your prediction value from the values in your dictionary and takes the absolute value of them. Then in the resulting array, it looks for the location of value that is smallest.

Second one's time complexity is O(N*log(M))

N and M denote the same thing as the first solution.
from bisect import bisect_left
def findClosestBinary(myList, myNumber):
    pos = bisect_left(myList, myNumber)
    if pos == 0:
        return myList[0]
    if pos == len(myList):
        return myList[-1]
    before = myList[pos - 1]
    after = myList[pos]
    if after - myNumber < myNumber - before:
        return after
    else:
        return before

#[findClosestBinary(b, elem) for elem in a]

print([findClosestBinary(b, elem) for elem in a])

Note: To save time I didn't implement myself but took the findClosestBinary() function from here.
This one is a better algorithmic approach in terms of time complexity. This does the same thing but uses a binary search to efficiently find the closest value in the dictionary.  However, it assumes your dictionary (list b) is sorted. Since your dictionary is a predefined list, you can improve the performance by providing a sorted array.
However, if your dictionary that you will map predictions to is not a very big one, then you can just use the first one. In the case of the dictionary being small, these two functions will behave the same in terms of time.


Ranking tumors from most malignant to most benign I have a classification problem classifying tumors as benign or malignant. However, I want to go a step further to provide a ranking of these tumors as most malignant to most benign. Are there any good algorithms out there to help with this ranking? Any suggestions?

The features of the dataset are the radius of the tumor, perimeter of the tumor, concavity, smoothness, etc.

Most classification algorithms actually provide continuous scores that are compared to a given threshold to give a binary output. Using that score directly give you a ranking. But unless you give us a specific algorithm, it would be difficult to help you further.

You usually can find that trough the performance metrics. AUC for exemple, has a general definition obtained by varying the above mentionned threshold. However, it has a more natural interpretation (and way faster approximation) when you use a definition based on individual scores.

Some models can even be calibrated in probabilities. That means, under some conditions, the continuous score can be intepreted as a probability to be of a given class.


prediction using LSTM i have training data from 2015-2017 and testing data of 2018. i have multiple variables my data is multivariate time series data.i want to predict 2019 data by using test data of 2018.is it possible? i am confused about Long short term memory neural networks working what is actually it will do.does my problem come under multivariate multi step forecasting? or multivariate single step forecasting?

You should post your code, or no one here can see what you have tried so far.  Anyway, I'll throw this out there for you, and hopefully it will clarify things.

from pandas_datareader import data as wb
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.pylab import rcParams
from sklearn.preprocessing import MinMaxScaler

start = '2019-02-20'
end = '2020-02-20'

tickers = ['AAPL']

thelen = len(tickers)

price_data = []
for ticker in tickers:
    prices = wb.DataReader(ticker, start = start, end = end, data_source='yahoo')[['Open','Adj Close']]
    price_data.append(prices.assign(ticker=ticker)[['ticker', 'Open', 'Adj Close']])

#names = np.reshape(price_data, (len(price_data), 1))

df = pd.concat(price_data)
df.reset_index(inplace=True)

for col in df.columns:
    print(col)

#used for setting the output figure size
rcParams['figure.figsize'] = 20,10
#to normalize the given input data
scaler = MinMaxScaler(feature_range=(0, 1))
#to read input data set (place the file name inside  ' ') as shown below
df.head()

df['Date'] = pd.to_datetime(df.Date,format='%Y-%m-%d')
#df.index = names['Date']
plt.figure(figsize=(16,8))
plt.plot(df['Adj Close'], label='Closing Price')

ntrain = 80
df_train = df.head(int(len(df)*(ntrain/100)))
ntest = -80
df_test = df.tail(int(len(df)*(ntest/100)))

#importing the packages
from sklearn.preprocessing import MinMaxScaler
from keras.models import Sequential
from keras.layers import Dense, Dropout, LSTM

#dataframe creation
seriesdata = df.sort_index(ascending=True, axis=0)
new_seriesdata = pd.DataFrame(index=range(0,len(df)),columns=['Date','Adj Close'])
length_of_data=len(seriesdata)
for i in range(0,length_of_data):
    new_seriesdata['Date'][i] = seriesdata['Date'][i]
    new_seriesdata['Adj Close'][i] = seriesdata['Adj Close'][i]
#setting the index again
new_seriesdata.index = new_seriesdata.Date
new_seriesdata.drop('Date', axis=1, inplace=True)
#creating train and test sets this comprises the entire data’s present in the dataset
myseriesdataset = new_seriesdata.values
totrain = myseriesdataset[0:255,:]
tovalid = myseriesdataset[255:,:]
#converting dataset into x_train and y_train
scalerdata = MinMaxScaler(feature_range=(0, 1))
scale_data = scalerdata.fit_transform(myseriesdataset)
x_totrain, y_totrain = [], []
length_of_totrain=len(totrain)
for i in range(60,length_of_totrain):
    x_totrain.append(scale_data[i-60:i,0])
    y_totrain.append(scale_data[i,0])
x_totrain, y_totrain = np.array(x_totrain), np.array(y_totrain)
x_totrain = np.reshape(x_totrain, (x_totrain.shape[0],x_totrain.shape[1],1))

#LSTM neural network
lstm_model = Sequential()
lstm_model.add(LSTM(units=50, return_sequences=True, input_shape=(x_totrain.shape[1],1)))
lstm_model.add(LSTM(units=50))
lstm_model.add(Dense(1))
lstm_model.compile(loss='mean_squared_error', optimizer='adadelta')
lstm_model.fit(x_totrain, y_totrain, epochs=3, batch_size=1, verbose=2)
#predicting next data stock price
myinputs = new_seriesdata[len(new_seriesdata) - (len(tovalid)+1) - 60:].values
myinputs = myinputs.reshape(-1,1)
myinputs  = scalerdata.transform(myinputs)
tostore_test_result = []
for i in range(60,myinputs.shape[0]):
    tostore_test_result.append(myinputs[i-60:i,0])
tostore_test_result = np.array(tostore_test_result)
tostore_test_result = np.reshape(tostore_test_result,(tostore_test_result.shape[0],tostore_test_result.shape[1],1))
myclosing_priceresult = lstm_model.predict(tostore_test_result)
myclosing_priceresult = scalerdata.inverse_transform(myclosing_priceresult)

Epoch 1/3
 - 7s - loss: 0.0163
Epoch 2/3
 - 6s - loss: 0.0058
Epoch 3/3
 - 6s - loss: 0.0047

totrain = df_train
tovalid = df_test

#predicting next data stock price
myinputs = new_seriesdata[len(new_seriesdata) - (len(tovalid)+1) - 60:].values

#  Printing the next day’s predicted stock price.
print(len(tostore_test_result));
print(myclosing_priceresult);

# next day's predicted closing price
[[329.42258]]

So, on 2020-02-20, we are predicting what AAPL will close at, on 2020-02-21.  The model said it would be 329.42 and the actual close was 313.05.  Less than 5% difference.  Not bad, but I would have expected a little better accuracy.  Oh well, we illustrated the point, and that was the goal of this exercise.

See the link below for more info.

https://www.codespeedy.com/predicting-stock-price-using-lstm-python-ml/


Dropping missing rows in two dataframes I have two files :
Test_data  - contains the features of a dataset to find predictions for
Submission_data - contains two columns : The index column for test data and another column for its corresponding predicted value
So , I have to make predictions on the test data and store the predicted values in the submission file.
During preprocessing of the test data , I am dropping rows that do not contain values (NaN) for atleast 50% of the features(columns) :
test_data = test_data.dropna(thresh=math.ceil(test_data.shape[1]/2))

Now , How do I remove the corresponding rows in the submissions dataframe ? Because , If I drop some rows in the test data , I cannot make a prediction for the corresponding row in the submissions dataframe/file.
The problem is , there is an Index column that does NOT HAVE UNIQUE values (In both test data and submissions data)
So , How do I drop the rows in Submissions data that were also dropped in Test data ?
I am new to ML challenges and I find this challenging .

When you read the two csv files and store the data in two dataframes, you could then combine it into one dataframe, do the dropna and then split it back. I will give an example using pandas
import pandas as pd
df1 = pd.read_csv('test_data.csv')
df2 = pd.read_csv('submission_data.csv')
df3 = pd.concat([df1, df2], axis=1) # this will combine the two dfs.
reduced_data = df3.dropna(thresh=math.ceil(test_data.shape[1]/2))
predictions = reduced_data.loc[:,['predictions']]
reduced_data.drop(columns=['predictions'], inplace=True)
#instead of 'predictions', use whatever column name you have for the predictions in submission_data.csv file.
Hope this helps.


Filling missing data with other than mean values What are all the options available for filling in missing data?

One obvious choice is the mean, but if the percentage of missing data is large, it will decrease the accuracy.

So how do we deal with missing values if they are are lot of them?

There are of course other choices to fill in for missing data. The median was already mentioned, and it may work better in certain cases.

There may even be much better alternatives, which may be very specific to your problem. To find out whether this is the case, you must find out more about the nature of your missing data. When you understand in detail why data is missing, the probability of coming up with a good solution will be much higher.

You might want to start your investigation of missing data by finding out whether you have informative or non-informative missings. The first category is produced by random data loss; in this case, the observations with missing values are no different from the ones with complete data. As for informative missing data, this one tells you something about your observation. A simple example is a customer record with a missing contract cancellation date meaning that this customer's contract has not been cancelled so far. You usually don't want to fill in informative missings with a mean or a median, but you may want to generate a separate feature from them.

You may also find out that there are several kinds of missing data, being produced by different mechanisms. In this case, you might want to produce default values in different ways.


Regression turned into classiffication Are there any reasons to turn regression into classification - binning continuous targets into classes and then learning model on those instead? (Let's say the algorithm isn't a problem.)

You actually convert the output of your algorithm from continuous to categorical.

I see many reasons of why you would want to do that. A simple case for this would be when you have very long time-series data that take a lot of space to be saved. In this case, it is convenient to convert the timeseries in a histogram representation, which actually is a number of predefined bins where all the values fall into. This will greatly reduce the memory requirements of your algorithm because you just increase the count of a bin (integer number) when a measurement is taken that falls in it, instead of storing the measurement itself as a float number.

This of course has pitfalls, such as losing the "time" dependencies between the measurements as well as reduction in resolution of your dataset. BUT, it can be really useful when forecasting measurements because instead of having the continuous R space as output of a predictor, you actually have (e.g.) 10 bins (classes) that the next measurement is predicted to fall into, making it easier for your model to be trained.

FYI, this is exactly what is done in the automotive industry.

EDIT - (Added sources)

Sources:

https://people.dsv.su.se/~tony/papers/dmin_2015.pdf (check the algorithm)

https://www.phmsociety.org/node/2284 (the previous algorithm is applied here for trucks)

https://link.springer.com/article/10.1007/s10618-017-0538-6 (another algorithm based on histograms of truck data)


How to create a custom loss function from sklearn metrics in Keras? I'd like to use the mutual information metric from sklearn as a loss function for a neural network in Keras, but I'm not sure how to do it. I'd like to try this because relationships in my dataset are non-linear and I believe it would help to use this metric instead.

Posts about creating custom loss function are more than a year old so I was hoping there was maybe some easier way to do this as some of it goes above my head.

Any idea about how to go about this?

Thanks a lot

Use the cross-entropy loss function.  That is already built into Keras, and while you haven't defined exactly what you mean by "mutual information metric", if it's what I suspect you mean, it is equivalent to the cross-entropy loss.


Plotting grafic result I am learning how to plot data. I am trying to obtain this grafic result on my dataset. The thing is, I can't do it. The cool plot is from a kaggle's kernel, which code is:

ggplot(cocoa,aes(x=pct, y=rating)) +geom_point(aes(colour=factor(location))) +
  theme( legend.position="bottom",legend.key.width=unit(0.2,"cm"),legend.key.height=unit(0.2,"cm")) +
  xlab("Cocoa Percent(%) ") + ylab("Chocolate Bar Rating") +
  ggtitle("Scattor plot of Cocoa Percent vs Chocolate Bar Rating")

When I try to install "ggplot" on RStudio, it says Error in install.packages : object"ggplot" not found. So I tryied copying the text in plot() function. Obviously, it gives me errors on the arguments.
The only thing I managed to do is this:

plot(data=chocolate, Rating~CocoaPerc, xlab="Rating", ylab="Cocoa Percentage", col="brown",
     main="Valutazione e Percentuale di cacao")

Can anybody help me achieving the cool result? Or can you advise me any other "tricks" or guides about data plotting and grafic study of datasets? Thanks

You probably did:

install.packages(ggplot)

This is a common mistake: when installing a package you need quotes, otherwise R interprets the name of the package as a string and since the string isn't defined you get an "object not found" error. The correct version is:

install.packages("ggplot")

(with single or double quotes, it doesn't matter)

Note: I would normally use the ggplot2 package, but maybe ggplot redirects to the same package:

install.packages("ggplot2")


How to know the model has started overfitting? I hope the following excerpts will provide an insight into what my question is going to be. These are from here.

The learning then gradually slows down. Finally, at around epoch 280 the classification accuracy pretty much stops improving. Later epochs merely see small stochastic fluctuations near the value of the accuracy at epoch 280. Contrast this with the earlier graph, where the cost associated to the training data continues to smoothly drop. If we just look at that cost, it appears that our model is still getting "better". But the test accuracy results show the improvement is an illusion. Just like the model that Fermi disliked, what our network learns after epoch 280 no longer generalizes to the test data. And so it's not useful learning. We say the network is overfitting or overtraining beyond epoch 280.

We are training a neural network and the cost (on training data) is dropping till epoch 400 but the classification accuracy is becoming static (barring a few stochastic fluctuations) after epoch 280 so we conclude that model is overfitting on training data post epoch 280.

We can see that the cost on the test data improves until around epoch 15, but after that it actually starts to get worse, even though the cost on the training data is continuing to get better. This is another sign that our model is overfitting. It poses a puzzle, though, which is whether we should regard epoch 15 or epoch 280 as the point at which overfitting is coming to dominate learning? From a practical point of view, what we really care about is improving classification accuracy on the test data, while the cost on the test data is no more than a proxy for classification accuracy. And so it makes most sense to regard epoch 280 as the point beyond which overfitting is dominating learning in our neural network.

As opposed to classification accuracy on test data compared with training cost previously we are now placing cost on test data against training cost.
Then the book goes on to explain why 280 is the right epoch where the overfitting has started. That is what I have an issue with. I can't wrap my head around this.
We are asking model to minimize the cost and thus cost is the metric it uses as a measure of its own strength to classify correctly. If we consider 280 as the right epoch where the overfitting has started, have we not in a way created a biased model that though is a  better classifier on the particular test data but nonetheless is making decisions with low confidence and hence is more prone to deviate from the results shown on the test data?

Let's say we want to predict if a student will land a job interview based on her resume.

Now, assume we train a model from a dataset of 10,000 resumes and their outcomes.

Next, we try the model out on the original dataset, and it predicts outcomes with 99% accuracy… wow!

But now comes the bad news.

When we run the model on a new (“unseen”) dataset of resumes, we only get 50% accuracy… uh-oh!

Our model doesn’t generalize well from our training data to unseen data.

This is known as overfitting, and it’s a common problem in machine learning and data science.

Overfitting V/s Underfitting

We can understand overfitting better by looking at the opposite problem, underfitting.

Underfitting occurs when a model is too simple – informed by too few features or regularized too much – which makes it inflexible in learning from the dataset.

Simple learners tend to have less variance in their predictions but more bias towards wrong outcomes (see: The Bias-Variance Tradeoff).

On the other hand, complex learners tend to have more variance in their predictions.

Both bias and variance are forms of prediction error in machine learning.

Typically, we can reduce error from bias but might increase error from variance as a result, or vice versa.

This trade-off between too simple (high bias) vs. too complex (high variance) is a key concept in statistics and machine learning, and one that affects all supervised learning algorithms.


Keras LSTM with 1D time series I'm learning how to use Keras and I've had reasonable success with my labelled dataset using the examples on Chollet's Deep Learning for Python. The data set is ~1000 Time Series with length 3125 with 3 potential classes.
I'd like to go beyond the basic Dense layers which give me about 70% prediction rate and the book goes on to discuss LSTM and RNN layers.
All the examples seem to use datasets with multiple features for each timeseries and I'm struggling to work out how to implement my data as a result.
If for example, I have 1000x3125 Time Series, how do I feed that into something like the SimpleRNN or LSTM layer? Am I missing some fundamental knowledge of what these layers do?
Current code:
import pandas as pd
import numpy as np
import os
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM, Dropout, SimpleRNN, Embedding, Reshape
from keras.utils import to_categorical
from keras import regularizers
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

def readData():
    # Get labels from the labels.txt file
    labels = pd.read_csv('labels.txt', header = None)
    labels = labels.values
    labels = labels-1
    print('One Hot Encoding Data...')
    labels = to_categorical(labels)

    data = pd.read_csv('ts.txt', header = None)

    return data, labels

print('Reading data...')
data, labels = readData()

print('Splitting Data')
data_train, data_test, labels_train, labels_test = train_test_split(data, labels)

print('Building Model...')
#Create model
model = Sequential()
## LSTM / RNN goes here ##
model.add(Dense(3, activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

print('Training NN...')
history = model.fit(data_train, labels_train, epochs=1000, batch_size=50,
    validation_split=0.25,verbose=2)

results = model.evaluate(data_test, labels_test)

predictions = model.predict(data_test)

print(predictions[0].shape)
print(np.sum(predictions[0]))
print(np.argmax(predictions[0]))

print(results)

acc = history.history['acc']
val_acc = history.history['val_acc']
epochs = range(1, len(acc) + 1)

plt.plot(epochs, acc, 'bo', label='Training acc')
plt.plot(epochs, val_acc, 'b', label='Validation acc')
plt.title('Training and Validation Accuracy')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend()
plt.show()

LSTM layers require data of a different shape.

From your description, I understand the starting dataset to have 3125 rows and 1000 columns, where each row is one time-step. The target variable should then have 3125 rows and 1 column, where each value can be one of three possible values. So it sounds like you're doing a classification problem. To check this in code, I would do:

>>> X.shape
(3125, 1000)

>>> y.shape
(1000,)

The LSTM class requires each single sample to consist of a 'block' of time. Let's say you want to have a block of 100 time-steps. This means X[0:100] is a single input sample, which corresponds to the target variable at y[100]. this means your window size (a.k.a number of time-steps or number of lags) is equal to 100. As stated above, you have 3125 samples, so N = 3125. To form the first block, we unfortunately have to discard the first 100 samples of y, as we cannot form an entire block of 100 from the available data (we would end up needing the data points before X[0]).

Given all this, an LSTM requires you to deliver batches of shape (N - window_size, window_size, num_features), which translates into (3125 - 100, 100, 1000) == (3025, 100, 1000).

Creating these time-blocks is a bit of a hassle, but create a good function once, then save it  :)

There is more work to be done, perhaps look at more in depth examples of my explanation above here... or have a read of the LSTM documentation, (or better still, the source code!).

The final model would then be simple enough (based on your code):

#Create model
model = Sequential()
model.add(LSTM(units=32, activation='relu',
               input_shape=(100, 1000))    # the batch size is neglected!
model.add(Dense(3, activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adam',
              metrics=['accuracy'])

Have a look at the documentation regarding input shape for the Sequential model. It basically says that we don't need to specify the number of batches within input_shape. It can be done with e.g. batch_size=50, if you require it to be a fixed number.

I know the input_shape argument is not in the documentation for LSTM, but the class itself inherits from RNN, which in turn inherits from Layer - so it will be able to use the info you provide.

One last tip: if you plan on adding several LSTM layers ('stacking' them), then you shall need to add one more argument to all but the last LSTM, namely, the return_sequences=True.


difference in l1 and l2 regularization I have seen at different places saying that: l1 regularization penalizes weights more than l2.

But the derivative of l1 norm is $\lambda$ and l2 norm is 2$\lambda$w. So l1 regularization subtracts smaller value than l2. Then why is it called that l1 penalizes weights more than l2. Or is it incorrect to say it like this?

That is generally not true, to be more accurate we can say that L1 promotes sparsity. if a weight is larger than 1 then L2 cares more about it than L1 while if a weight is less than 1 then L1 cares more about it than L2.

For a quick example imagine two weights, $w_1 = 15$ and $w_2 = 0.02$, let's imagine that the model considers reducing both of those weights by a small number $\epsilon=0.001$ (for the sake of simplicity reducing either weight by $\epsilon$ increases the model-error part of the loss at the same rate). Now, let's calculate how much the loss changes for both weights under both regularization terms.

For $w_1$ and L1: $\Delta L=|15|-|14.999| = 0.001 $ (Loss decreased by 0.001)

For $w_1$ and L2: $\Delta L=|15|^2-|14.999|^2 = 0.029999 $ (Loss decreased by 0.029999)

We can see that for $w_1$ the loss decreases about 28 times more for L2 compared to L1. So L2 regularizes the weights more in this case (i.e. L2 is willing to sacrifice more model complexity to regularize this weight more).

For $w_2$ and L1: $\Delta L=|0.02|-|0.019| = 0.001 $ (Loss decreased by 0.001)

For $w_2$ and L2: $\Delta L=|0.02|^2-|0.019|^2 = 0.000039 $ (Loss decreased by 0.000039)

We can see that for $w_2$ the loss decreases about 24 times more for L1! So for smaller weights L1 is willing to sacrifice more model expressiveness just to reduce the weights while L2 barely pays any attention (compared to L1) to weights close to 0.

It's possible that in practice you perceive L1 regularizes weights more than L2 becauses when you perform L1 regularization you notice a lot of the weights tend to be exactly 0 while in L2 almost no weights are exactly 0 so you might tend to think that L1 is "stronger", but that just comes from the point we notices above where L2 barely pays any attention to weights close to 0 while L1 still sees a benefit of $\epsilon$ no matter how small the weight was before the $\epsilon$ change and thus it promotes sparsity.

There are a lot of practical and theoretical differences between L1 and L2 regularization, too many to list here. For example one practical difference is that L1 can be a form of feature elimination in linear regression. A theoretical difference is how L2 regularization comes from the MAP of a Normal Distributed prior while the L1 comes from a Laplacean prior.

EDIT: I just reread your post and yes, looking at the derivatives you should also get the same insight. For $w>1 \Rightarrow w\lambda > \lambda$ thus L2 regularizes large weights more while for $w<1 \Rightarrow w\lambda < \lambda$ thus L1 regularizes small weights more.


R - Interpreting neural networks plot I know there are similar question on stats.SE, but I didn't find one that fulfills my request; please, before mark the question as a duplicate, ping me in the comment.

I run a neural network based on neuralnet to forecast SP500 index time series and I want to understand how I can interpret the plot posted below:

Particularly, I'm interested to understand what is the interpretation of the hidden layer weight and the input weight; could someone explain me how to interpret that number, please?

Any hint will be appreciated.

As David states in the comments if you want to interpret a model you likely want to explore something besides neural nets.  That said it you want to intuitively understand the network plot it is best to think of it with respect to images (something neural networks are very good at).

The left-most nodes (i.e. input nodes) are your raw data variables.
The arrows in black (and associated numbers) are the weights which you can think of as how much that variable contributes to the next node.  The blue lines are the bias weights.  You can find the purpose of these weights in the excellent answer here.
The middle nodes (i.e. anything between the input and output nodes) are your hidden nodes.  This is where the image analogy helps.  Each of these nodes constitute a component that the network is learning to recognize.  For example a nose, mouth, or eye.  This is not easily determined and is far more abstract when you are dealing with non-image data.
The far-right (output node(s)) node is the final output of your neural network.

Note that this all is omitting the activation function that would be applied at each layer of the network as well.


Would a convolution network make sense for policy based tic-tac-toe approach? This is inspired from my previous question, comments to which made me realize that a CNN was unsuitable for the problem The CNN required over 700k training datasets while a MLP did it in in less than 50k.

Now, I'm trying to solve the next problem and need to figure out if a CNN makes sense.

CNN detail -

Input - The board as an array of 9 elements that represents the board (0=empty, 1='X', 2='O')

Output - Recommended move as a one-hot encoded array of 9 elements. The index of 1 is the recommended move (for example, in [0,0,1,0,0,0,0,0,0] the recommended move is 2)

So, basically the CNN will be trained with a dataset that consists of boards and the move that the winner of the game has made for each of the boards. Then during evaluation, it'll try to predict the best move for a given board.

Does a convolution neural network make sense for this problem?

Note: The convnet that I was going to use for this problem is the same as in my previous question

For anyone interested, I'm posting my results of trying out both CNNs and MLPs -

A 3x3 kernel 4 feature maps CNN consistently performed better than a 9 neuron MLP. I trained both on completely random tic-tac-toe training data size of 10kx20 and the CNN learned to to achieve ~80% win/draw against a random opponent while the MLP achieved around ~60% for the same data. A 2x2 CNN performed much worse than 3x3.

Training data - Inputs and outputs are same as mentioned in the question with the exception that the AI is trained only if it loses or draws the game and the dataset is chosen from randomized player for drawn games.

Note: It is possible to achieve much better accuracy with less data but in this case my goal was to just use this game to learn machine learning programming so the code was not particularly optimized for anything. The comparison of MLPs and CNNs is also incidental and needs to be done with more rigor for any serious purposes.


Can be possible to solve Rubik's cube using DQN? I'm trying to solve Rubik's cube using deep learning and I came across with DQN, so I decided to give it a try. I developed all the code and started training bu I got this results:

Loss goes up and test never get better results. I have tried to change learning rate, epsilon greedy decay, reducing scramble moves to one but it still can't solve it with just one move. That's why I would like to know if it just impossible to do it this way or I could miscalculating some values.

I think yes you can since rubric group can be represented as grid word environment
see this link from GitHub they attempted solving using dqn


What is valued more in the data science job market, statistical analysis or data processing? Not sure if this question is OT here.  If it is, perhaps move it to meta?  If not:

I'm trained in classical statistics, work in research, and have learned much (and taught) machine learning.  I've got a doctorate, but not in statistics or computer science.  I'm not a software engineer and am only really fluent in R.  Maybe I know a bit of C++ and can write a SQL query, but I wouldn't be the guy you'd hire to set up a data pipeline.

There are plenty of engineers who know maybe a thing or two about statistics and machine learning, but who can build apps in Java, manage a spark cluster, set up a distributed database, etc.  But they wouldn't be the person you'd hire to do any sort of analysis that isn't already canned in python.

Browsing jobs in the private sector, I see a lot of demand for the second type of data scientist.  Is there much demand for the first type?  What is the pay differential in the private sector between the two types?  Did you start as the first type and become more like the second type "on the job?"

Given your current situation, a data analyst or scientist position makes more sense, but with some practice you can acquire the software skills to become a data- or machine learning engineer. There is a glut of people from academia similar to you, and software companies want people who can actually build software, so it is not surprising you see demand for engineers. But that does not mean you have to become one. Do you want to build software products?

My recommendation would be to make sure you are familiar enough with common applications of statistics in industry (e.g., A/B testing, and time series forecasting) to get a job in analysis, then learn computer science on the side. There is a wide range of salaries, depending on seniority, company and location. You can find specific figures on sites like LinkedIn, Glassdoor, Paysa, Payscale.


How to deal with version control of large amounts of (binary) data I am a PhD student of Geophysics and work with large amounts of image data  (hundreds of GB, tens of thousands of files). I know svn and git fairly well and come to value a project history, combined with the ability to easily work together and have protection against disk corruption. I find git also extremely helpful for having consistent backups but I know that git cannot handle large amounts of binary data efficiently.

In my masters studies I worked on data sets of similar size (also images) and had a lot of problems keeping track of different version on different servers/devices. Diffing 100GB over the network really isn't fun, and cost me a lot of time and effort.

I know that others in science seem to have similar problems, yet I couldn't find a good solution.

I want to use the storage facilities of my institute, so I need something that can use a "dumb" server. I also would like to have an additional backup on a portable hard disk, because I would like to avoid transferring hundreds of GB over the network wherever possible. So, I need a tool that can handle more than one remote location.

Lastly, I really need something that other researcher can use, so it does not need to be super simple, but should be learnable in a few hours.

I have evaluated a lot of different solutions, but none seem to fit the bill:

svn is somewhat inefficient and needs a smart server
hg bigfile/largefile can only use one remote
git bigfile/media can also use only one remote, but is also not very efficient
attic doesn't seem to have a log, or diffing capabilities
bup looks really good, but needs a "smart" server to work

I've tried git-annex, which does everything I need it to do (and much more), but it is very difficult to use and not well documented. I've used it for several days and couldn't get my head around it, so I doubt any other coworker would be interested.

How do researchers deal with large datasets, and what are other research groups using?

To be clear, I am primarily interested in how other researchers deal with this situation, not just this specific dataset. It seems to me that almost everyone should have this problem, yet I don't know anyone who has solved it. Should I just keep a backup of the original data and forget all this version control stuff? Is that what everyone else is doing?

What I am ending up using is a sort of hybrid solution:

backup of the raw data
git of the workflow
manual snapshots of workflow + processed data, that are of relevance, e.g.:

standard preprocessing
really time-consuming
for publication

I believe it is seldom sensible to have a full revision history of large amount of binary data, because the time required to review the changes will eventually be so overwhelming that it will not pay off in the long run.
Maybe a semi-automatic snapshot procedure (eventually to save some disk-space, by not replicating the unchanged data across different snapshots) would be of help.


Why don't get the expected result using a SVM training model? I want to learn a model for recognizing facial emotions. . I used a dataset with 213 samples. I extract firstly features using the Gabor filter. Then, I reduce the data dimensionality with the PCA and the genetic algorithm. Finally, I test performance using cross validation with a test size of 25%, and I got 98% acuraccy. The problem is when I test the SVM model on other images, I don't get the expected result.

Without seeing the data and your code it's hard to examine and tell what's the problem. Below are what I would think as a first step.

98% accuracy is a high level, so I would initially check for over-fitting.
The ideal solution to over-fitting is using more data. If you can't, then increasing the portion of the training dataset (maybe to 50%) and then going through cv & testing might help.

Another common reason is over/under-represented class. What's the ratio between the classes?


Train/Test Split after perform SMOTE I am dealing with a highly unbalanced data, so I used the SMOTE algorithm to resample the dataset.
After SMOTE resampling, I splitted  the resampled dataset to training/testing sets, using the training set to build a model and testing set to evaluate the model.
However, I am worried about that some data points in the testing sets might actually jittered from data points in the training set (i.e. the information is leaking from the training set to testing set), so the testing set is not really a clean set for testing.
Does anyone have similar experience? Does the information really leak from training to testing? Or the SMOTE algorithm actually took care of it and we don't have to worry about it?

When use any sampling technique ( specifically synthetic) you divide your data first and then apply synthetic sampling on the training data only. After you train you use the testing set ( which contains only original samples) to evaluate.
The risk if you use your strategy is to have the original sample in training ( testing) and the synthetic sample ( that was created based on this original sample) in the testing ( training) set.


Document Categorization Problem I'm very new to data science in general, and have been tasked with a big challenge.

My organization has a lot of documents that are all sorted on document type (not binary format, but a subjectively assigned type based on content, e.g. "Contract", "Receipt", "Statement", etc...).

Generally speaking assignment of these types is done upon receipt of the documents, and is not a challenge, though we would like to remove the human element of this categorization.  Similarly, there are times when there are special attributes that we would like to identify, like "Statement showing use."  Thus far, this is entirely done by human intervention.

I am a python programmer, and have been looking at tools to extract the text from these docs (all PDFs, all OCR'ed and searchable) and run analysis.  Research has led me to look at standard libraries like NLTK, scikit-learn and gensim.  But I'm struggling to identify what would be the best methodology for categorizing newly received documents.

My research is leading me down a few paths...one is creating a Tf-iDf vector model based on a sampling of current corpa and then creating a model for an incoming document's corpus and doing a naive bayes analysis against existing models to discern which category the incoming document belongs to based on highest probability.  Question 1: is this right?  If so question 2 becomes what is the right programmatic methodology for accomplishing this?

The reason I bring this up at all is because most tutorials I find seem to lean toward a binary discernment of text corpa (positive vs negative, spam vs ham).  I did see scikit-learn has information on multi-label classification, but I'm not sure I'm going down the right road with it.  The word "classification" seems to have different meaning in document analysis than what I would want it to mean.

If this question is too vague let me know and I can edit it to be more specific.

Except for the OCR part, the right bundle would be pandas and sklearn.

You can check this ipython notebook which uses TfidfVectorizer and SVC Classifier.

This classifier can make one-vs-one or one-vs-the-rest multiclass predictions, and if you use the predict_proba method instead of predict, you would have the confidence level of each category.

If you're looking for performances and you don't need prediction confidence levels, you should use LinearSVC which is way faster.

Sklearn is very well documented and you will find everything you need for text classification.


Should I divide my dataset in multiple prediction models? Let's say I have an expertise in the domain knowledge of the dataset I am working on. I know that part of my dataset acts 100% differently than the rest.

Also, it is straightforward to check if a row belongs to one part of the dataset or the other with a couple of simple if-else.

Should I split my dataset upfront and create two different prediction models for the two parts of the dataset? Or should I keep one model and try to improve it?

Then when I want to use the model for predictions, I check the input values and see in which category it belongs and I call the appropriate model for it.

Yes, you can divide your dataset and use different prediction models for different part of your dataset. Take advantage of your domain knowledge to build a good model.

After all, you can actually count the following as a single model
$$f(x) = \begin{cases} f_1(x) & ,x \in A \\ f_2(x) & ,x \in A^c \end{cases}$$


How do I arrange my data to predict 6 weeks of daily sales I have a data.table base that has many variables to use them to forecasting sales for the next 6 weeks of daily sales. In fact, all the database is arranged by date as you can see here.Note that here I just show you some of variables.

> Data_train[order(Date)]
         Store DayOfWeek       Date Sales Customers Open Promo StateHoliday SchoolHoliday
      1:     1         2 2013-01-01     0         0    0     0            a             1
      2:     2         2 2013-01-01     0         0    0     0            a             1
      3:     3         2 2013-01-01     0         0    0     0            a             1
      4:     4         2 2013-01-01     0         0    0     0            a             1
      5:     5         2 2013-01-01     0         0    0     0            a             1
     ---
1017205:  1111         5 2015-07-31  5723       422    1     1            0             1
1017206:  1112         5 2015-07-31  9626       767    1     1            0             1
1017207:  1113         5 2015-07-31  7289       720    1     1            0             1
1017208:  1114         5 2015-07-31 27508      3745    1     1            0             1
1017209:  1115         5 2015-07-31  8680       538    1     1            0             1 .

My question is about the arrangement of the data according to the target goal. My problem is really about the Date variable.
In fact, I suggest this path:

I sum all sales by each date (because I have many types of stores).
I order my database according to the Date 's ascending order.
I didn't need duplicated rows in the date variables so I delete them.

Just to show you the new base for considered variables.

> Data_train[,SumSaleseachDay:=sum(Sales),by=c('Date')][order(Date)][!duplicated(Date)][,-c('Sales','Customers'),with=FALSE]
     Store DayOfWeek       Date Open Promo StateHoliday SchoolHoliday SumSaleseachDay
  1:     1         2 2013-01-01    0     0            a             1           97235
  2:     1         3 2013-01-02    1     0            0             1         6949829
  3:     1         4 2013-01-03    1     0            0             1         6347820
  4:     1         5 2013-01-04    1     0            0             1         6638954
  5:     1         6 2013-01-05    1     0            0             1         5951593
 ---
938:     1         1 2015-07-27    1     1            0             1        10707292
939:     1         2 2015-07-28    1     1            0             1         9115073
940:     1         3 2015-07-29    1     1            0             1         8499962
941:     1         4 2015-07-30    1     1            0             1         8798854
942:     1         5 2015-07-31    1     1            0             1        10109742

ADDED INFORMATION:
I have a database of 1017209 rows. And for each Store, I have its historic of  Sales between 2013-01-01 and 2015-07-31. And I have also 17 variables included to build the model.

The steps above just lead to forecast by day.

If I want to forecast for each Store and by day, what should I do?
thank you in advance!

I guess it depends on exactly what you want - do you want a forecast per store? If so you would need to leave the data set aggregated only to a store level. If not, and you want the overall sales by day, then you are OK to leave it aggregated.

Another consideration is whether there are many store-level features e.g. location, weather, local demographics. These features may be useful for the model to learn from and would lose resolution if store level data is aggregated, therefore leaving it un-aggregated would be better. The counter point to this is whether you have enough data for each store to learn representative trends! So it does get kind of circular and depends on the size of your data set.


Grouping company information I have 3 different datasets with company information, in all of them I have company name, but is not perfect: For example:

Dataset A: Company name: Facebook
Dataset B: Company name: Facebook, Inc
Dataset C: Company name: facebook

Some other signals like company url exists, but in terms of name matching wondering if text similarity is a good approach for this grouping problem.

You can consider string similarity based on edit distance. eg: Levenshtein distance

However, you also need to compute this with respect to prefixes of the strings, and not only the full strings. (eg: to compare "Facebook" with "Facebook Inc."). Also normalize the capitalization of the strings before comparison (make them all lowercase, etc.), and maybe remove punctuation as well.

Experiment with a few of the above, see if the results make sense, and fine tune the method accordingly. Also using other metadata will help. URL and context/topic words/entities occurring around the company name will be associated with the company, and can help find the right groupings. This will be more involved though, so you may want to attempt this later on, if the simpler approaches do not suffice.


Which feature selection technique to pickup(Boruta vs RFE vs step selection) I have data with 103 columns. I would like to understand which algorithm is best for feature selection and what may be the logic to call any feature as best.

I run below feature selection algorithms and below is the output:

    1) Boruta(given 11 variables as important)
    2) RFE(given 7 variables as important)
    3) Backward Step Selection(5 variables)
    4) Both Step Selection(5 variables)

I not able to decide which one to pick up; with domain knowledge it appears I must pick up results from Boruta (as it is giving most number of variables and all seems important).

However I don't find any concrete reason to pickup the best combination.

There is a tradeoff between selecting features and precision. Fewer features probably have less precision (predictive power).

Select the features that make sense for your problem taking into consideration the trade-off of information vs performance.  The fewer features the model sees the predictive power that it has.


oversampling data with subclass Oversampling of under-represented data is a way to combat class imbalance. For example, if we have a training data set with 100 data points of class A and 1000 data points of class B, we can over sample the 100 A data (may be with some sophisticated oversampling methods) to generate 1000 A data to mitigate the data imbalance.

Now, let's say we have 1100 data points of class B, and class A has 2 subclasses, A1 and A2, which have 100 and 10 data points, respectively. And we are still interested in binary classification.

In this case, how should I over sample data of class A to address class imbalance? Should I over sample A1 to 1000 and A2 to 100, or over sample both A1 and A2 to 550?

Besides running an experiment, is there any theoretical analysis of this kind of class imbalance problem?

It totally depends on you and your problem. In fact, if the distribution of the data must be unchanged, oversampling is not a suitable solution.
If you can change the Loss function of the algorithm, It will be very helpful. There are many useful metrics which were introduced for evaluating the performance of classification methods for imbalanced data-sets. Some of them are Kappa, CEN, MCEN, MCC, and DP.

Disclaimer:

If you use python, PyCM module can help you to find out these metrics.

Here is a simple code to get the recommended parameters from this module:

>>> from pycm import *

>>> cm = ConfusionMatrix(matrix={"Class1": {"Class1": 1, "Class2":2}, "Class2": {"Class1": 0, "Class2": 5}})

>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]

After that, each of these parameters you want to use as the loss function can be used as follows:

>>> y_pred = model.predict      #the prediction of the implemented model

>>> y_actu = data.target        #data labels

>>> cm = ConfusionMatrix(y_actu, y_pred)

>>> loss = cm.Kappa             #or any other parameter (Example: cm.SOA1)


Is there an AI service that could be used to classify 30,000 different tools and parts? I'm trying to build an image classifier where people can take a picture of a tool or part and have the image classified. Much like bixby or amazon's tool to do something similar, but with only 30,000 different items to classify instead of a huge shopping catalog.

I was hoping there would be an AI service that takes a lot of the work out as I am trying to avoid training my own CNN because the manpower necessary to take the amount of pictures I would need is not achievable for my project. I can definitely get the manpower to take some pictures, to the tune of maybe 300,000 to 1 million, but I don't believe that would be enough while training a brand-new CNN.

Use pre-trained models and transfer learning.

take keras as example:

https://keras.io/applications/

you just need prepare dozens of pictures for each category and can have a pretty good performance.


GPU computing: how much VRAM do I need for mini batch gradient descent? I want to do some GPU computing with an NVIDIA card, and am deciding between having a GTX 960 with a 2GB or 4GB ram. Which one should I take? How much difference would these make in terms of the batch size I can use for mini batch gradient descent? Would this difference be significant?

Thank you for answers.

Generally memory cost increases linearly while mini-batch size increases. If batch size 32 costs you 2GB memory, then batch size 64 will cost you 4GB memory.

In practice, deep learning framework(for instance, keras) will load the entire full-batch into your memory, thus you cannot observe apparent memory increment or decrement during mini-batch tuning.

References

https://github.com/fchollet/keras/issues/146


correcting conditional and marginal distribution in transfer learning I understand that in case of transfer learning, we can have the target and the source data having different domain distributions. In such cases, authors in many papers suggest bringing the marginal and conditional distributions of the target and the source closer, i.e, minimize the difference between the marginal and conditional distributions. Can someone please help me understand this by giving an intuitive explanation for this? I am unable to understand what exactly the author means when he says by bringing the distributions closer? Explanations using visual representations would be helpful.

Bringing the distributions closer means that we are trying to modify the source data usually by performing weights in the instances or in the features (or in both in hybrid algorithms) in order to make the weighted data more similar to the targer data. If we achieve that then we will be able to train a model considering also the source data, which usually has a larger size or it is well annotated in comparison with the target.


Is it true more CPU core is better for deep learning? I just started to learn the deep learning in my free time. I was hoping to buy a laptop where I want to implement some small(alexnet) to medium(GoogleNet) networks maybe something bigger. I searched for the GPU. Everyone is suggesting minimum RTX 2060. However, it's vague about the CPU. I heard people saying that it's better to have more number of core in CPU. All I understand CPU only do the data pre-processing. What I want to know is it feasible to have a 4 core CPU(e.g. i5 9300H) with RTX 2060 having a tight budget? Will this configuration serve my purpose?
Another things is new 10th Gen Intel Core i7-10750H processor with up to 5.0 GHz3 have a 6 cores. While AMD Ryzen 7 4800HS have 8 cores. Which will be best for deep learning. Is the number of cores really matters?

It is better to go with a good GPU than CPU. Most of the time you will train your model on a GPU which gives you around 100x speed boost compared to CPU. So i guess your first config is good enough for normal DL task.


Get the probabilities of Tensorflow Hi I am studying tensorflow for cifar-10 image classification using the code here

I want to ask you that how to predict the probabilities of each class in test images.

At line 27 in the train.py you have the following code:

correct_prediction = tf.equal(y_pred_cls, tf.argmax(y, axis=1))

It tries to find whether the predicted values are the same as the real ones. You can run y_pred_cls to see the probability of each class for your desired input.

  I want to use the code to predict the probabilities of new data's labels, how to save and load the model we have trained which used the train data.

for saving your model and its weights you can take a look at here. As you can see from there, you have to make a saver object:

import tensorflow as tf

#Prepare to feed input, i.e. feed_dict and placeholders
w1 = tf.placeholder("float", name="w1")
w2 = tf.placeholder("float", name="w2")
b1= tf.Variable(2.0,name="bias")
feed_dict ={w1:4,w2:8}

#Define a test operation that we will restore
w3 = tf.add(w1,w2)
w4 = tf.multiply(w3,b1,name="op_to_restore")
sess = tf.Session()
sess.run(tf.global_variables_initializer())

#Create a saver object which will save all the variables
saver = tf.train.Saver()

#Run the operation by feeding input
print sess.run(w4,feed_dict)
#Prints 24 which is sum of (w1+w2)*b1

#Now, save the graph
saver.save(sess, 'my_test_model',global_step=1000)

And for loading that you have to restore it:

import tensorflow as tf

sess=tf.Session()
#First let's load meta graph and restore weights
saver = tf.train.import_meta_graph('my_test_model-1000.meta')
saver.restore(sess,tf.train.latest_checkpoint('./'))

# Access saved Variables directly
print(sess.run('bias:0'))
# This will print 2, which is the value of bias that we saved

# Now, let's access and create placeholders variables and
# create feed-dict to feed new data

graph = tf.get_default_graph()
w1 = graph.get_tensor_by_name("w1:0")
w2 = graph.get_tensor_by_name("w2:0")
feed_dict ={w1:13.0,w2:17.0}

#Now, access the op that you want to run.
op_to_restore = graph.get_tensor_by_name("op_to_restore:0")

print sess.run(op_to_restore,feed_dict)
#This will print 60 which is calculated

Edit: Actually the code is a bit strange, anyway. The following part:

tf.nn.softmax_cross_entropy_with_logits(logits=output, labels=y)

can help you. It outputs the probabilities for each class.


Cleaning time series data I have a time series data about daily usage of a computer program, here is an example

2017-11-10: 0
2017-11-09: 14
2017-11-08: 0
2017-11-07: 6
2017-11-06: 102
2017-11-05: 0
2017-11-04: 0

As you can see 11-06 has a spike at 102. Due to our way of gathering this data, we know that data is probably erroneous and we are sure that 102 is not correct according other values.

So we need to clean these dirty values.

Is there a mathematical way to do this?
Is there a python lib to help us?

I think you have a few options:

If you have a pre-set rule to exclude outliers, such as a hard-threshold at 100 which you know the data shouldn't exceed, then something as simple as x = [e for e in x if e < 100] will do.
If you have a parametric belief, such as any observation that falls beyond so many standard deviations from mean, or quartiles, are outliers; then you can implement the other answers that have been mentioned.
Else, you can go for a clustering approach. Here I believe your first shot should be a k-means clustering. This is super easy to build and interpret. See my code below.

x = [0,14,0,6,102,0,0]
from sklearn.cluster import KMeans
kmeans = KMeans(n_clusters=2).fit(np.array(x).reshape(-1, 1))

#First cluster:
np.array(x)[np.where(kmeans.labels_ == 0)]

#Second cluster (outliers):
np.array(x)[np.where(kmeans.labels_ == 1)]
K-means is known to be sensitive to outliers, hence a more robust method such as MeanShift, which you tried, is a good rival to k-means. I would run both, and stick with the result that makes better sense to me.

Hope this helps!