How to forecast time series analysis for more then one dependent variables? I have three datasets:

Dataset_1= column name as ID, Date, counter_id

Dataset_2= column name as ID, no. of tickets sold

Dataset_3= column names as ID, country_code.

Here, country_code ranges from 1 to 30 as codes for different countries and counter_id as counter codes ranging from 1 to 100. I merged the datasets based on common column ID.

I have to estimate how many tickets will be sold during the next 20 days in a given country and through a specific counter.

What I have done is predict the number of tickets sold for next 10 days using time series forecasting. But, I'm not able to decide how I can predict the counter_id and country_code along with tickets sold? Please give me some useful feedback. I am a beginner in data science. Thanks!

I think you do not need to "predict" counter_id and country_code as variables, what you need is to produce results to every counter and country (and sometimes these results will be zero).

The technique you are looking for is called VAR (Vector Autoregression)

VAR

The vector autoregression (VAR) model is one of the most successful, flexible, and easy to use models for the analysis of multivariate time series. It is a natural extension of the univariate autoregressive model to dynamic multivariate time series.

The equation you propose is solvable by VAR:

$Y_t = C + A_1 Y_{t-1} + A_2 Y_{t-2} + ... + A_n Y_{t-n}$

Where $Y_t = \begin{bmatrix} Y_{i=1,j=1,t} \\ Y_{i=1,j=2,t} \\
\vdots \\
Y_{i=1,j=m,t} \\
\vdots \\
Y_{i=n,j=m,t} \end{bmatrix}$

Where, $i$ is every country, $j$ is every counter.

The usage is very similar to univariate time series analysis, which you are probably familiar with.


How to create A/B test segements for highly variable data I have a data in which there is a high degree of variability.

My Objective is to do an AB test to check the behavior change due to new changes.

All samples has shown historically high and low performances. This means if I take any 2 cohorts randomly, they show vast historical comparison difference Following is the example for weekly comparison. Same behavior holds true for monthly and daily too.

W1: -10.04%

W2: 3.9%

W3: -4.2%

W4: -3.7%

W5: 5.4%

W6: -2.6%

W8: 0.8%

W9: -2.3%

W10: 1.92%
...
..
.

No matter I tried to segment users by their aggregate behavior and see the difference, its still this level of variability is observed.
I'm looking for clearly difference to be consistent no matter in which ever direction. Either its positive or negative. And, with small range of magnitude.

Can someone please suggest the alternative methods to minimize this difference?

Your ability to detect a difference depends on the relative standard deviation of your data.  Sometimes the noise is due to seasonal effects, which an AB test does an excellent job in correcting.  However if the metric itself is intrinsically high variance, you will need a large sample size, or adjust your significance and power parameters.
Run a check of your standard deviation and do a power calculation to evaluate your situation.
Keep in mind that there are some statistical distributions which have variances approaching infinite.


Transfer Learning on Resnets/VGGs -- Validation accuracy can never be over 75% I am trying to classify skin cancer images into two categories -- malignant and benign. Literatures suggest that using pre-trained resnet/vgg network achieves more than 90% accuracy. However, with my dataset, whatever I try, the validation accuracy can never be greater than 75%.
I am using a well-balanced dataset where there are 500 malignant and 500 benign images in the training dataset. The number of images is on a smaller end; however, the fact that I am using transfer learning and other literatures showing that with only ~1000 images they could achieve more than 90% accuracy, I believe there's something wrong with the transfer learning logic than the data itself.
I additionally have tried the followings:

Used different number of batches (8/16/32/64/128) and different learning rates (from 1e-7 to 0.1)
"unfreezed (change require_grad=False to True)" the pretrained resnet/vgg after the validation error platues and updated the weights for smaller epochs (a.k.a. fine tuning the pretrained network after setting require_grad=True after validation accuracy platues)
Decayed the learning rate using scheduler
used other nets: inception v3, GoogleNet
rather than having one fully connected layer, used dropout-FC-relu-FC

This is the core logic:
model_ft = models.resnet18(pretrained=True) # some other place in my code, I set all of resnet18 param to require_grad=False
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 2)

model_ft = model_ft.to(device)

criterion = nn.CrossEntropyLoss()

optimizer_ft = optim.SGD(model_ft.parameters(), lr=0.001, momentum=0.9)

and this is how I normalize my data (I have tried cropping, rotating, and horizontal-flipping; none of these boosted the performance. For now, in order to preserve some relevant clinical features, I decided not to touch the data):
data_transforms = {
    'train': transforms.Compose([
        transforms.Resize(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

Can someone please let me know what I could do to improve the model performance? Without too much of a data pre-processing, we think the model should achieve well-above 85% accuracy in the validation set.

There was a Contest in Kaggle - SIIM-ISIC Melanoma Classification.  In this contest also tasks for participants to classify two class: malignant & benign.
But in the dataset, there were the same images in both the class. And this produces the Dataleakage.
Dataleakage,  also can produce in the same class images both in the training and in the validation set.
And to solve this problem Chris Deotte(Kaggle Grandmaster), Applied RAPIDS cuML kNN using CNN embeddings for finding the similar images and then remove those image.
Here are link of the Chris Deotte Kaggle Notebook.
This is the one solution to detect similar images for classes if the model does not work well. There are obviously other factors to improving model accuracy. Best of luck.


BERT training on two tasks: what is the order of tasks? I read that BERT has been trained on two tasks: Masked Language Modeling and Next Sentence Prediction. I want to gain clarity how exactly it was done.

Was it initially trained on  Masked Language Modeling (where we predict masked token) and later on Sentence Prediction (where we predict isnext or not)? It seems like these two tasks would require different architecture. Does BERT has some common architecture for these two tasks with interchanging layer for individual tasks? Still question of ordering remains.

BERT is trained on both task at the same time.

MLM is done using the token representations.
NSP is done using the CLS representation.


Balanced Train set to predict Imbalanced Prediction set One of the methods to address a classification predictive analysis on an imbalanced set consist on undersample the majority class (others approaches consist on: undersample the majority class, synthesize new minority classes...).

So assume hereafter we use any of those solutions and then we train an algorithm with the new generated data set. Will this trained algorithm be useful to predict further data from this system which is in general imbalanced?

Or to make it more concrete, is it possible in general to train a model with a balanced training set so that we can effectively predict an imbalanced prediction set? Or both should be generarly either balanced or imbalanced?

Thanks!

Sergi

It's possible. Consider KNN. When you train a model, it essentially "remembers" the training set. Now, when you use the model to predict an unbalanced data set, the model simply read off from the memory, so there's no problem.

For example, imagine you train a model with an uniformly distributed age group from 10 to 80. Now you have a test set where everybody is 70-80 years old. KNN simply find out the neighbors for your old people (say 60-80 old people in your training data) and then compute a weighted average.

Using a balance training set to predict a imbalanced test set is not super challenging. The other way around is much more challenging.


Is 10,000 images for one class of spectrograms good enough for music classification model? I'm debating on using a DNN or CNN for this classification, but I have 10,000 spectrograms of songs. I was wondering if I need more because the images have a low variants.

Agreed with Emre. One thing that can be helpful when asking the question is to look at comparable datasets since there's usually some around. For spectrograms for example, there are 150,000 samples w/ 12 labels here, 2890 samples here, and 1 million samples with various labels here.

Find a few examples and gauge a rough order of magnitude of samples per class given how close the tasks are to yours. That should give you at-least a starting point. Then, if lets say one set is very large, usually someone will have written a paper using that set, and that can help start both the architecture and size jumping off point (and maybe a baseline for transfer learning) of your task.


Training multi-label classifier with low quality training set So I'm creating a topics classifier where a document may be tagged for several different topics, let's say - A, B while actually the document belongs to A, B and C.
In the training stage I want the classifier to learn that the document belongs to A and B but I'm not sure about class C so I don't want it to learn that the document doesn't\does belong to class C.
Any ideas on how to implement such thing?

I thought about adding weights to the output labels (low weight means that there's no way the document belongs to the aspect, high weight means that the document belongs to the aspect for sure and mid weight means I'm not sure (so the penalty in this case will be lower).

You almost solved the problem in the last paragraph. Expressed more formally, your cost function could be

$$\frac{1}{N} \sum_{i,j} c_{i,j} y_{i,j} \log x_{i,j}$$

where $i$ runs over items/documents, and $j$ runs over classes, $x$ is your prediction, $y$ is the binary label (1 if item $i$ has class $j$), and $0 < c < 1$ is your confidence. This is a simple modification of the cross entropy. When the confidence $c$ is low, the value of the prediction matters less.


while re-training a pre-trained model, I'm facing this issue RuntimeError: You must compile your model before using it model summary:

RuntimeError: You must compile your model before using it.

It says that the model needs to be compiled. But as far i know, if i compile a model, all the previous trained data will be lost, and the model will be trained from scratch. I don't want this to happen because i want to use this model for further re-training purposes. Can you help me in this.

Can I re-train the model without compiling it and ,save it and use this model for future training ?

There 2 techniques while using the pre-train model

1. Transfer learning: In the case of transfer learning, we use the pre-train model to train on new kind of dataset so generally, remove the last layer and add a new layer into the network. At this point, as you are adding a new layer the model should be compiled. But don't worry if you are compiling it then weights will not be lost.

2. Retraining (I guess this is your case): As you are not modifying the architecture but just training it on additional dataset then also it is not going to loose weights.


Should I initiate size before loop and append in Python I am going to build a 1,000,000 * 2 dataframe in Python and the input I have is in the form of 1*2. So obviously the most straightforward method is to use loop and append a 1*2 pair into the existing dataframe for 1,000,000 times... However, it really takes forever and as a Matlab user, I am almost forbidden in Matlab to do this and always told to initiate the final size of the matrix (or dataframe in Python) first and put elements into the matrix... Is this the same in Python? Thanks in advance!

Yes it will be just as slow as in matlab. You’re going to be copying the whole data frame every time you append something.


How to get data back into two separate audio files after successfully applying kmeans clustering on an audio file? Okay so I wrote a very simple python code to read a wav file, get the mfcc features and then use kmeans clustering on the features. The hello.wav file has two different people saying hello at the same time, I wanted to output two separate voices

from python_speech_features import mfcc
from scipy.io import wavfile
from sklearn.cluster import KMeans
sampfreq, data = wavfile.read('hello.wav')

mfcc_feat = mfcc(data,sampfreq)

kmeans = KMeans(n_clusters=2)
kmeans.fit(mfcc_feat)
print(kmeans.cluster_centers_)
print(kmeans.labels_)

Now I got the labels and cluster centers but how do I get back my two separate sound samples using this?

What you are trying to do is Blind Source Separation, and unfortunately it does not works as easy as that. When you are extracting MFCC from an audio file with a mixture of voices, MFCC clusters won't represent two people. For the separation, you should end up with two sound signals with the same number of samples. The MFCC are used as the feature's space for speakaer recognition tasks, for example, but not for separation in general. For more mathematical background about BSS you can visit: https://arxiv.org/pdf/1603.03089.pdf

The most simple case is the so-called Cocktail Party Problem, in where you have at least as many microphones as persons and the mixture has to be instantaneous (no delay between signals, only attenuations due to path-losses). This very simple case can be solved applying Independent Component Analysis, with the FastICA algorithm: http://scikit-learn.org/stable/auto_examples/decomposition/plot_ica_blind_source_separation.html

For delayed mixtures you can try the DUET algorithm: https://pdfs.semanticscholar.org/1413/746141f2871e0f45056a7696e019b8f8a100.pdf

For convolved mixtures and undetermined problems (more speakers than microphones) this PARAFAC-based algorithm performs quite well: http://people.ece.umn.edu/~nikos/05233821.pdf


Image classification with CNN Python I'm working on image classification using CNN, my dataset contains more than 50 classes (50 folders) which represent the types of car parts, and in each folder we have vehicle brands, each vehicle folder contains the images of the part taken in different sides: front, back, left , right, top, bottom. For example: a folder 'door' wil contain different brand of cars, in each car we will have images of the door tooken by different sides.
I have to create a model to recognize the type of the car part and also the side that the picture was taken.
Please, can anyone suggest how can I use all these folders for my training dataset and how to recognize those two things ( type and side).

You can try and use the tf.keras.preprocessing.image.ImageDataGenerator's flow-from_directory() method. Here are a few links for reference:

tf.data: Build TensorFlow input pipelines

Retraining an Image Classifier

Notebook on Image Captioning


Forecasting with a Machine Learning Algorithm Im sorry if it is a too general question, but i am stuck somewhere between perfect and adequate in my model. So, i wanted to ask here. If it is not a suitable question, your negative feedbacks are all welcome, sorry. But theoretically subject is inside a gray area and all comments are valuable i think. It is not a widespread approach to use ML for forecasting, as far as I know.

Our sales team publishes a forecasting report at the beginning of each year. But it is mostly far from reality, not very scientific. It includes subjective assesments also.

We decided to construct a ML model to make a forecasting. Not a classical forecasting because there are many input variables that can effect the output. So, it is an experimental project actually.

So far, i have developed an adequate model, which is already far more better than sales one. But still i think it is not a good one, only adequate.

Data includes Customer-FromCountry-ToCountry based sales. So there are many combinations. By looking at similar applications at MachineLearningMastery.com, i have developed my model like that:

customer-from-to-M9-M8.....M1 - month - quarter - movingavg3 - mov6 - mov9 - dptcountryholidaycount - arvcountryholidaycount

m9 to m1 are previous 9 month sales of predicted month M(output variable) M's vary from 0 to 800 mostly. Holidays are from 0 to 6 for example. I Scaled all of these numerical variables between 0-1

Customer-From-To-Month-Quarter variables are categorical and i used OneHotEncoding for these.

I shuffled all the data, so there is not a time ordered dependency inside my dataset.

I have divided the data into 3 segments.
1: above 100 average for last 3 months;
2:10-100 average
3: below 0 average for 3 months.
I run different models for each 3 segments.

There are many 0 values, so i am using a LGBClassifier first to determine if the sales is 0 or not. Then if it is not 0, i am using a LGBRegressor to determine the sales value for that input row. (SGDRegressor for segment 1, it performs better)

I tried to add MonthSeasonality index and QuarterSeasinality index for each combination into dataset as variables. But it didn't perform well.

What can i try to get better score, im not asking about algorithms etc. My main purpose here is to get insights about construction of dataset, preprocessing. Your precious ideas about preprocessing tricks before applying a ML algorithm.

Thanks

I'm afraid your data is probably too complex and specific for somebody else to understand exactly what's going on.

The only idea I can suggest is to try to analyze manually the errors that your model makes:

Are there any patterns, like a kind of errors which happens quite often? For instance a country which tends to be overestimated, a month which tends to be under-estimated, this kind of thing.
If yes try to investigate why: for instance is there some additional feature that could help the model make better predictions for these cases?


Binary Search Tree with categorical and numeric values in python I coded bst in python and it is working for both categorical and numeric values. Is it draws tree with ASCI value.? So in Isolation Forest algorithm how it is helping to get better accuracy than considering only numeric features and removing categorical features?

Thank you.

If PlayTennis is your target classification label, you could map the numeric values to intervalls. Finding appropriate intervalls can be done by first sorting the numeric values and then using Intervalls at which there is a change in your target label. In this case we have a change between 48 and 60, and between 80 and 90. (48+60)/2 = 54 and (80+90)/2 = 85 are good candidates for label-intervalls. (T>54) and (T>85) would be the labels.


Trying to understand the result provided by np.linalg.norm function in numpy (normalisation) I'm new to data science with a moderate math background. I'm playing around with numpy and can across the following:

So after reading np.linalg.norm, to my understanding it computes the 2-norm of the matrix. Wanting to see if I understood properly, I decided to compute it by hand using the 2 norm formula I found here:

Following computing the dot product, the characteristic equation, applying the formula for quadratic equation and taking square root of the max, I end up with a different result, namely:

So here is my question. What went wrong? Did I use the right formula? Also I couldn't find a conclusive way to get the 5. The doc he doc that says:

  If this is set to True, the axes which are normed over are left in the result as dimensions with size one. With this option the result will broadcast correctly against the original x.

But I can't wrap my head around it. What does it represent and how you compute it?

I hope the formatting and the question are clear enough.

The formula you cited is not the formula scipy is using. According to the documentation it calculates the "Frobenius norm" which is defined to be the root of the squared sum of the elements of each row (row since you specified dim=1)

$ O_i = { \biggl(\sum_{n=1}^N \mathbf{A}_{i, n}^2\biggr) }^{1/2} $

Thus, $(3^2+4^2)^{1/2}=5$ and $(2^2+6^2+4^2)^{1/2}=56^{1/2}$

EDIT: Also, the keepdims argument just says whether to collapse the dimension that you calculated or not, a simple example is to look at the output shape of the matrix that you get with and without the argument.

a = np.array([[0, 3, 4], [2, 6, 4]])

np.linalg.norm(a, axis=1, keepdims=True).shape # output is (2, 1)
np.linalg.norm(a, axis=1, keepdims=False).shape # output is (2,) because the second dimension was collapsed


sklearn.neighbors.NearestNeighbors - knn for unsupervised learning? From basic theory I know that knn is a supervised algorithm while for example k-means is an unsupervised algorithm.

However, at Sklearn there are is an implementation of KNN for unsupervised learning (http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.NearestNeighbors.html#sklearn.neighbors.NearestNeighbors).

What is exactly this unsupervised version of knn at SkLearn?

Is this a knn algorithm?

If yes how it is unsupervised since by definition knn is supervised?
If no what is it then?

The unsupervised version simply implements different algorithms to find the nearest neighbor(s) for each sample.

The kNN algorithm consists of two steps:

Compute and store the k nearest neighbors for each sample in the training set ("training")
For an unlabeled sample, retrieve the k nearest neighbors from dataset and predict label through majority vote / interpolation (or similar) among k nearest neighbors ("prediction/querying")

The unsupervised version is basically only step 1, the training phase of the kNN algorithm.

(This is useful because if your dataset is large, a pairwise comparison for all samples (algorithm='brute') is often infeasible. Therefore two alternative algorithms for the training stage are implemented that make use of previous comparisons to reduce the number of distance calculations. See the documentation here.)


How to convert longitude and latitude in time series data from daily to weekly? I have time series data like this:

date
longitude
latitude

01/01/2010
-5.42766
107.5784

02/01/2010
-6.42728
104.5245

07/01/2010
-7.42702
105.5816

14/01/2010
-4.42728
99.57834

17/01/2010
-6.41523
103.5562

...
...
...

31/12/2013
-4.42728
99.57834

This is location data (longitude and latitude).
I want to change the data from daily to weekly, something like this:

week
longitude
latitude

week 1
...
...

week 2
...
...

week 3
...
...

week 4
...
...

how to transform the data? since calculating the mean of the data doesn't make sense because it is location data.

I don't know exactly whether it is useful for your case, but you can use last day of the week, so coordinates of the object at end of the week.
I copied your example data to a text file, read it with Pandas, and resampled data from daily to weekly by getting coordinates of the last day of each week.
df = pd.read_csv('untitled.txt',
                 infer_datetime_format=True,
                 parse_dates=['date'], dayfirst=True)
df.set_index('date', inplace=True)
df.resample('W').last()


How to decrease the gap between training and validation error? I have been executing an open-source Text-to-speech system Ossian. It uses feed forward DNNs for it's acoustic modeling. The error graph I've got after running the acoustic model looks like this:

Here are some relevant information:

Size of Data: 7 hours of speech data (4000 sentences)
Some hyper-parameters:

batch_size       : 128
training_epochs  : 15
L2_regularization: 0.003

Can anyone point me to the directions to improve this model? I'm assuming it is suffering from over-fitting problem? What should I do to avoid this? Increasing data? Or changing batch-size/epochs/regularization parameters? Thanks in advance.

As Learning is a mess points out, more information would help make useful suggestions. What is the size of the model (in terms of the trainable parameters)? Also, how much training data do you have?
However, you can try these:
a. Dropouts - this a regularization method, it is a kind of proxy for ensemble learning. Not sure if you are using this.
b. Smaller Batches: 8/16/32 - Empirically it has been found that smaller batches find better minimums in the sense of generalization on the test/validation sets. See https://arxiv.org/pdf/1609.04836.pdf.
c. Holding the number of parameters the same, deeper models (more layers) have been found to generalize better. See https://arxiv.org/abs/1706.10239.


Understanding AC_errorRate loss function I'm reading an article about Rolling Window Regression: a Simple Approach for Time Series Next value Predictions.

He explains about 5 different loss functions. I managed to understand the first four, but I don't understand the fifth one:

  Almost correct Predictions Error rate (AC_errorRate) — the percentage of predictions that is within %p percentage of the true
  value

From my understanding, this loss type means:

you define a threshold percentage error (let’s say 2%)
for each true value y, the desired prediction should be between y + 0.02*y and y - 0.02*y
the percentage of predicted values fulfilling the rule above contribute to the “inliers” predictions, I.e., good predictions

This idea reminds me to what RANSAC does


How to detect a no good person in data? I am struggling to figure a way to determine if a person is 'no good' or 'good'.

A little bit about what I am trying to accomplish, I have a data set of a payment for a violation with a check or a bounced check on a given day, if there is a bounced check it means there is no such account.

If they have a pattern of payments and bounces in chronological order then they are doing something fishy such as gaming the system. But, if they have payments, bounces, payments, and little to no bounces after that then the initial bounces could have been human entry errors.

I want to use the data set to determine who are the 'no good' people that have a pattern, ex. paying, bouncing, paying, bouncing.

Here is an example of my data set:

person     state date                           status  bounce_code
A           NY  23-DEC-15 06.27.08.000000000 PM Paid
A           NY  23-DEC-15 06.12.58.000000000 PM Paid
A           NY  18-DEC-15 10.14.39.000000000 AM Return  R03
A           NY  15-DEC-15 04.16.58.000000000 PM Return  R03

The original data set contains thousands of people and their history of payments and bounces.

One idea is to analyze each person to see if they fit a "mold" (payment, bounce, payment, bounce) and if they do i will label them as a 'no good' person.

Please let me know if this is confusing and I am open to suggestions on how to approach this problem.

I would start by looking at the distribution of bounce rates for each individual. That distribution would provide some details around what could be considered a "normal" bounce rate, and where you would draw a line for an extreme bounce rate that implies a 'bad'. Additionally, you can also look at average bounce rates (divide bounce rate by the tenure), and review these averages by tenure. This may reveal that different threshold values might be appropriate for customers with different tenure. For instance, if a customer is new (first month of activity) then even a moderate bounce rate could imply a 'bad'.

Additional metrics -- like the ratio of bounce to valid payments, time between bounce -- could be useful to look at as well.

Once a set of initial thresholds are identified, I would closely review some of the customers who are identified as 'bad' based on those thresholds to ensure that there are not too many false positives. The business point of view (from those who would be using this information to make decisions) would be great to incorporate into this review.


Explaining the similarities between instances in a cluster with KMeans If I create clusters using the KMeans clustering algorithm in Python, is there any way I can find which attributes were used to group those instances in clusters?
Example: I have a dataset of cars with columns 1 - 10. I get 3 clusters, how can I know which attributes were used to make them? Is it because all instances in cluster 1 have the same value in columns 2 and 3?

First things first - You got back 3 clusters from K-Means because you asked it to find the 3 best clusters (and not the other way around). Google up the "elbow-method" to help you determine the best number of clusters for your dataset.
Clustering would make use of the all 10 columns in your dataset.
In order to figure out which fields matter to a cluster - Find the mean and standard deviation of every field within the context of a cluster. The fields with the lowest standard deviation probably tell you why these instances were grouped together in a single cluster.
Another thing you could try out is - use CLusters as Labels - and run your dataset by a classification model such as Random Forests or xgboost. Random Forests for instance gives you a Feature Importance - which will provide you insights on the most impactful features in your dataset.


Evaluating the performance of a random forest classifier I'm using a random forest classifier (in R) to impute missing data in a dataset. Basically, I have a bunch of objects (companies) and I want to guess an attribute (size) from other attributes (capital, owning_group and state). The dependent attribute is a categorical variable (size) with 3 possible values (small|medium|large). A random forest (R package randomForest) on a set of 3 variables provide this output:

ff = size ~ capital + owning_group + state

Call:
 randomForest(formula = ff, data = df, importance = T, ntree = ntree, na.action = na.omit)
               Type of random forest: classification
                     Number of trees: 2000
No. of variables tried at each split: 1

        OOB estimate of  error rate: 32.41%
Confusion matrix:
       large medium small class.error
large    238     17   237  0.51626016
medium    80     25   322  0.94145199
small     73     30  1320  0.07238229

  Overall Statistics

               Accuracy : 0.7297
                 95% CI : (0.7112, 0.7476)
    No Information Rate : 0.8049
    P-Value [Acc > NIR] : 1

                  Kappa : 0.426
 Mcnemar's Test P-Value : <2e-16

Statistics by Class:

                     Class: large Class: medium Class: small
Sensitivity                0.7087       0.84211       0.7294
Specificity                0.8868       0.83981       0.8950
Pos Pred Value             0.5488       0.14988       0.9663
Neg Pred Value             0.9400       0.99373       0.4450
Prevalence                 0.1627       0.03245       0.8049
Detection Rate             0.1153       0.02733       0.5871
Detection Prevalence       0.2101       0.18232       0.6076
Balanced Accuracy          0.7977       0.84096       0.8122

I interpret this output as saying that the model has a 73% accuracy, and that the classifier makes a lot of mistakes for medium and large, but gets small mostly right. Does the P-value indicate that the model is not significant?

Assuming that this precision is OK for my context, how can I validate this model beyond these simple observations?

First of all, if you are trying to impute missing values with a RF model then take a look at the rfImpute() function.

Second, your data in unbalanced, which is why your classification is not good, your model is biased towards the majority class (small) and so it classifies a lot of you cases into the majority class. The issue of imbalance needs to be addressed.

Validating is done with a test set, as the results you have obtained from the model are already done using Cross-Validation (known as OOB scores).


Clustering the data if some values in each vector is not applicable I have four different categories which the number of variables in each category is 7, 3, 8, and 9 (in total 27). The value of each variable changes from 1 to 5 but some of the variables were not applicable for some observations. In total, there are 20 observations which their data is available for 27 variables in the mentioned context (from 1 to 5 plus not applicable). How can I cluster these four categories for 20 observations to get three clusters for each category?

edit: There are 4 categories (A, B, C, and D). Each category has a different vector including 7, 3, 8, and 9 values (from 1 to 5 plus NA). Now, it's 18 observations for each category of A, B, C, and D. I need to make three groups for each category. How can I find the centroid of each cluster for each category? It means I need three centroid in each category, totally 12(=4 category* 3 groups). Any feedback would be appreciated. Thank you.

First, are your categories independent of each other? If this is really four parallel problems, solve one at a time and it'll simplify things a bit.

Second, the heart of clustering is the distance function between any two observations. Because your data has missing values, you need to explicitly decide how to handle them. A common approach is to ignore the problem by throwing away any rows that have any null values, which won't work at all for your data.

A simple approach would be to ignore any NA columns, but keep the rest of the columns--for category C, the difference between rows 1 and 2 would be 4 (columns 11, 14, 15, and 16 are all off by one). There's an issue with this approach that I'll get to later, but let's implement it for now.

For each category, we need three rows of centroid positions (we might as well start off with randomly selected rows that have no missing values in them). We then calculate distance to each centroid using a bunch of summed ifs: if(matrix cell != NA, matrix cell - centroid cell, 0) across all the columns. (If you're doing this computation in a spreadsheet, I'd use intermediate cells to calculate the pieces instead of dealing with one huge formula.)

We then optimize the centroids with a standard iterative procedure: each row gets a group membership corresponding to the closest centroid, then each centroid becomes the mean of its group members (and it looks like your mean function automatically excludes NAs), then we repeat until satisfied.

Why did I specify that we had to seed with a row without NAs? If you allow NA measurements in the centroids, this will fail because a row of all NAs is as central as it gets! If our update step never lets us move to NAs, the NA column will be transient, but what happens when we take the mean of a missing column? (In your sample data, consider what happens if rows 4-12 are a group in the B category.)

So if we want to allow NAs in the centroids, we need to modify our distance function to something like the following:

If both cells are numbers, the distance is the difference
If both cells are NA, the distance is 2
If only one cell is NA, the distance is 3.

Those numbers are chosen such that for any full column of numbers, a numerical value in the centroid position will be closer, but if the column has sufficiently many NA rows, a NA value in the centroid will be closer. You might want to adjust the values so that the NA threshold looks like what you want it to.

In order to find the centroid of a set of rows using our distance function, we should use an optimization procedure, but we can cheat: if the number of NAs in the group is above the threshold, we set it to NA, and otherwise we set it to the mean.


Convert Lat /Lon of User input to Lat/Lon of Open Data I have data from a public data set in gridded form 2.5 degree x 2.5 degree(lat,lon).
Latitude goes from 90 N to -90 S. Longitude goes from 0 to 357.5. It is stored every 2.5 degrees and there are no intermediate values.

My user will want to download data from this data set. However they are not restricted from entering only the values that the public data set possesses. My goal then is to convert the data they enter to the "nearest" latitude and longitude of the public data set. For example while data may exist at 5 N and 60 E(and the next pont is 7.5 N and 62.5E) they may enter 6 N and 61.3 E.

How do I map the user input to the nearest latitude and longitude ? Note - user input will be in the form of a rectangle - lat_min,lat_max, lon_min and lon_max.

I took a shot at this and I hope I can find a better algorithm through DS SE.

lat_min = floor(lat_min/2.5) * 2.5

lat_max = ceiling(lat_max/2.5) * 2.5

lon_min = floor(lon_min/2.5) * 2.5

lon_max = ceiling(lon_max/2.5) * 2.5

What's wrong with what you've suggested?  It seems fine.

You've supplied four of possible points, essentially:

(latmin, lonmin)
(latmin, lonmax)
(latmax, lonmin)
(latmax, lonmax)

Clearly one of them is the closest grid point to the user input. Then just use whatever distance calculation you like (e.g. Haversine distance, or better measurements if you have access to them) and pick which of the four points is closest to what the user specified.  It'll be accurate and performant.


An error coming out when fitting a model with keras I am trying to build a model using kerras but for some reason I receive a ValueError: Error when checking target: expected activation_5 to have 4 dimensions, but got array with shape (50000, 10) error. I have recently changed the model parameters, so I think it has something to do with that, but I cannot understand what exactly. Before I changed the model everything worked

Here is the code

from keras.models import Sequential, load_model
from keras.layers import Dense, Dropout, Flatten, BatchNormalization, Activation
from keras.layers.convolutional import Conv2D, MaxPooling2D
from keras.constraints import maxnorm
from keras.utils import np_utils
from keras.datasets import cifar10
from tensorflow.python.keras.callbacks import TensorBoard
from time import time
seed = 21
tensorboard = TensorBoard(log_dir="logs/{}".format(time()))

#loading and preprocessing training and testing data
(train_data, train_labels), (test_data, test_labels) = cifar10.load_data()
train_data = train_data.astype('float32')
test_data = test_data.astype('float32')
train_data = train_data / 255.0
test_data = test_data / 255.0
train_labels = np_utils.to_categorical(train_labels)
test_labels = np_utils.to_categorical(test_labels)
class_num = test_labels.shape[1]

#building model
model = Sequential()
model.add(Conv2D(32, (5,5), input_shape=(32,32,3), padding ='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(3,3), strides=(2,2), padding='same'))
model.add(Conv2D(32, (5,5), padding ='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(3,3), strides=(2,2), padding='same'))
model.add(Conv2D(64, (5,5), padding ='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(3,3), strides=(2,2), padding='same'))
model.add(Dense(64))
model.add(Activation('relu'))
model.add(Dense(10))
model.add(Activation('softmax'))

epochs = 50
optimizer = 'adam'
model.summary()
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])

numpy.random.seed(seed)
model.fit(train_data, train_labels, validation_data=(test_data, test_labels), epochs=epochs, batch_size=64, callbacks=[tensorboard])
scores = model.evaluate(test_data, test_labels, verbose=0)
print("Accuracy: %.2f%%" % (scores[1]*100))
```

You are missing a flatten() layer between your last MaxPooling2D and Dense layer.


Implicit Training Models in Spark MLlib? I have something of a ground floor question I’d like to ask.  I’m looking at various options for recommendation engines using Spark.  I feel that I have a decent grasp of the basics of collaborative filtering and have been able to replicate the simple examples provided by the Spark docs and AMPlab.  However, I am seeking to investigate how to create an effective Product Recommendation Algorithm using Implicit Feedback where there is an abundance of user level data available.  Let’s say that I have a large dataset with individual user attributes such as a unique user ID, Product ID, user demographic data, user behavioral data, etc.  What this data set does not contain is an explicit rating that the user gave the product.  In the absence of that explicit feedback, is it still possible to train a Recommendation model using:

ALS.trainImplicit(ratings, rank, numIterations, alpha=0.01)

Furthermore, I found this statement from the Spark website to be a bit nebulous: “If the rating matrix is derived from other source of information (i.e., it is inferred from other signals), you can use the trainImplicit method to get better results.”  Does this mean that the large table from my hypothetical example would need to be compacted or reduced to a singular value for the ‘rank’ argument in the trainImplicit function? I have looked everywhere I can think of, but cannot find an accessible example describing how this would be accomplished.  If anyone knows of a good resource, please let me know.  Lastly, if anyone provides a code example, please do so in Python since that is my preferred Spark API.

Thanks in Advance.

Implicit feedback simply means that given a ratings-dataset, the zeroes are treated as "nothing was rated" as opposed to "the user rated this as a zero". The algorithm they use is based on the one in this paper. The code is pretty difficult to read but available here.

That being said, if you do have binary product choice variables (i.e., user A chose product B, but not the actual ratings), the implicit training could also work as is, even if was not originally designed for it.


WGAN is too slow what are some ways to tweak for speed ups? I have implemented a vanilla GAN which gave good results very fast but it had a lot of mode collapse issue, because of this I learned about WGAN which suppose to fix this, in fact they claim they have never encountered mode collapse problem which is great. My main issue is how slow WGAN takes to converge to good results. With a vanilla GAN by epoch 10 I was getting good looking generations, with WGAN I am still seeing noise images on epoch 50! Besides bumping up the learning rate is there anything else I can do? The network is a modified smaller DCGAN that has 1 less output layer than the original. So instead of the extra 1024 output layer mine stops at 512. I knew WGAN was going to be slow from all the research I have done but I didn't except to be this slow. Can some one point me to the right direction on how to better optimize WGAN's speed?

I have googled around but most results are just vague comment chains or agreements that there is a speed problem with no further leads on what you should do next to counter this issue.

If you are using WGAN with Gradient Penalty, I think the Framework you are using is the limited factor since computing all the gradients will take time.

If you are using WGAN with Gradient Penalty one way to get faster results is to omit the gradient penalty and just do weight clipping as mentioned in original WGAN Paper. But be careful in Improved WGAN (with gradient penalty) they showed, weight clipping can lead to bias in the discriminator.

But for my experience with GANs WGAN GP gave the overall best results and I will spend the time training for it.

But also your training routine could be part of the problem since lot of reference implementations try to get an rate of which the generator updated, e.g. 1 generator update for 10-100 discriminator updates (which in some scenarios totally makes sense). So there are several factors for slow training but most important is that gradient penalty is computational expensive.


Identify the parameter causing the anomaly in a multivariate dataset I have a payment transaction dataset with a large number of predictor variables. I am trying to build a model for anomaly detection and I have evaluated various algorithms/approaches for the same like Isolation Forest, kNN, Autoencoders, and One-class SVM.

I am able to identify if a payment record is an anomaly or not but I am not able to pin-point the predictor variable that is causing the anomaly.

e.g.:

Account  ||  Currency  ||  Beneficiary  ||  Amount ||  isAnomaly(target)

I want to identify if, for an anomalous record, Currency variable is causing the anomaly or Amount variable is causing the anomaly.

I have gone through the below sources amongst many others but couldn't find anything helpful.

Anomaly Detection in Database

Anomaly Detection in multiple parameters

I have recently started my journey in data science and would be glad if someone could help me with this issue.

Usually it will not be a single feature value that is responsible for the decision of a ML model. Neural Networks, Random Forests, SVM, etc. intend to transform the input in a more beneficial feature space, where making decisions is more easier for them.

As a draw back, this makes interpratability for humans more intricate. Explainability of ML methods is a whole research field.

You could check out some explainability approaches. For autoencoders for example you could use Layerwise Relevancy Propagation (LRP). https://arxiv.org/pdf/1708.08296.pdf


Training network with variable frame rate? I would like to train a temporal network, but the video data available are in different frame rates(ex 7,12,15,30). How should I train this network, without down-sampling higher frame rate videos.

I tried up-sampling everything, but there is some artifacts generated.

What is the suitable approach?

I don't believe there is a well-known method to deal with this.

Simple pre-processing

While I haven't done this with images/videos, I know from general time-series analysis that you basically have to interpolate the lower frequencies or you need to down-sample the higher frequencies. If you think about, what else is there to do...?

Modelling solution

Nvidia released a research paper with an accompanying video showing how they were able to train a model, which could estimate the frames between frames - effectively interpolating video and increasing its frame rate. This would essentially be the equivalent of interpolation between frames and allow you to scale up your lower frequency videos to match the higher frequency ones. The paper is named:

  Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation

... sounds like something worth reading.

There are older algorithms that try to do the same thing (e.g. "twixtor"), but I read they have problems with things such as rotating objects. Another thing to keep in mind is the usual GIGO: garbage in garbage out. There are still some artefacts of interpolation in the Nvidia video, but that likely comes from blurry input images used during training when e.g. objects were moving faster than the recording frame rate could handle.

It seems that they train two models: the first encodes the optical flow between frames and the second model uses that, along with the base images to perform the interpolation. Please read the paper for more details. It also outlines how they train the model (learning rates, number of epochs, augmentation steps, etc.).

Here is the sketch of their model for flow computation/interpolation:

We can see that it is an encoder/decoder-looking model, introducing a bottleneck that condenses the information, before upsampling again. This is based on the U-net model architecture: an encoder/decoder that also introduces skip connections between layers of different scales.


What is Bit Per Character? What is Bits per Character (bpc) metric which has been used to measure the model accuracy with reference to text8 and enwiki8 datasets. I encountered the term bpc in transformer -XL paper here. How different is it from the perplexity as a metric?

Perplexity is a measurement of how well a probability model predicts a sample.
Intuitively, perplexity can be understood as a measure of uncertainty. Say the real thing you want to predict is the sequence of numbers from one to six. If you predict each number in turn with a six-sided die, you will be right about one-sixth of the time. The die's perplexity with this data (and really with any data it could possibly predict - dice are dumb) is six. The perplexity of whatever you're evaluating, on the data you're evaluating it on, sort of tells you "this thing is right about as often as an x-sided die would be."
Wikipedia defines perplexity as:

“a measurement of how well a probability distribution or probability model predicts a sample."

What is perplexity? from stats.exchange.com by Henry

It gives the perplexity of a discrete distribution as
$$2^{-\sum_x p(x)\log_2 p(x)}$$
which could also be written as
$$\exp\left({\sum_x p(x)\log_e \frac{1}{p(x)}}\right)$$
i.e. as a weighted geometric average of the inverses of the probabilities. For a continuous distribution, the sum would turn into an integral.
The article also gives a way of estimating perplexity for a model using $N$ pieces of test data
$$2^{-\sum_{i=1}^N \frac{1}{N} \log_2 q(x_i)}$$
which could also be written
$$\exp\left(\frac{{\sum_{i=1}^N \log_e \left(\dfrac{1}{q(x_i)}\right)}}{N}\right) \text{ or } \sqrt[N]{\prod_{i=1}^N  \frac{1}{q(x_i)}}$$
or in a variety of other ways, and this should make it even clearer where "log-average inverse probability" comes from.

Bits-per-character and bits-per-word
Bits-per-character (BPC) is another metric often reported for recent language models. It measures exactly the quantity that it is named after the average number of bits needed to encode on character. This leads to revisiting Shannon’s explanation of entropy of a language:
“if the language is translated into binary digits (0 or 1) in the most efficient way, the entropy is the average number of binary digits required per letter of the original language."
By this definition, entropy is the average number of BPC. The reason that some language models report both cross-entropy loss and BPC is purely technical.

In the case of Alex Graves' papers, the aim of the model is to approximate the probability distribution of the next character given past characters. At each time step $t$, let's call this (approximate) distribution $\hat{P}_t$ and let $P_t$ be the true distribution. These discrete probability distributions can be represented by a vector of size $n$, where n is the number of possible characters in your alphabet.
So the BPC or average cross-entropy can be calculated as follows:
\begin{align}
bpc(string) = \frac{1}{T}\sum_{t=1}^T H(P_t, \hat{P}_t) &= -\frac{1}{T}\sum_{t=1}^T \sum_{c=1}^n P_t(c) \log_2 \hat{P}_t(c), \\
& = -\frac{1}{T}\sum_{t=1}^T \log_2 \hat{P}_t(x_t).
\end{align}
Where $T$ is the length of your input string.
The equality in the second line comes from the fact that the true distribution $P_t$ is zero everywhere except at the index corresponding to the true character $x_t$ in the input string at location $t$.

How to compute bits per character (BPC)?, from stats.exchange.com by user6952886


Crop all written letters from image to form a website Above image has handwritten hindi / deavanagari letters all on a single

image.
Task is to Prepare a dataset from the given assignment.jpg image using image

processing tools
Idon't have clarity whether i can use photoshop editing tools like adobe photoshop etc.
If image processing tools mean opencv , is it really possible to create like a emnist like dataset from single image.
I am not able to come up with any way to seperate the letters from eachother

Finally i had done it by combining adaptive thresholding,bluring,and morphological, It is helpful to check out find contours for two part letters by @Salman and @Jeru Luke , it really helped me.

This can be done with OpenCV.

Code needs to :

Prep-process image to enhance contrast
Use OpenCV to identify 'bounding box' for each character
Save each character's image as a new file

Code example : http://cvisiondemy.com/extract-roi-from-image-with-python-and-opencv/


How to identify sentiment of a given word from a sentence Use case is like this:

Suppose I have a sentence(review data): The staffs were very kind and helpful. The room is ok for its price. There did not seem to be a heater in the room. So, a bit for our January trip.

Now if I want to know the sentiment of a word for example room then how should I proceed.

I have designed sentiment analysis for a given sentence using bag of words model, but determining the sentiment for the given word from the sentence (review data) is kind of new to me. What approach should I take here ?

Providing some approach or any link is enough.

One of the ways to do it is to build bag of words and apply a linear model for sentimental analysis. Then you can look at model's coefficients - positive coefficient means word is positive, negative coefficient - negative word.


Analysis of sales performance This is my first question on this stackexchange - i hope i'm in the right place and i'm asking the right question.

I work for a business, we sell certain products.
We are running a trial currently on a a subset of these products where we are trying to impact on the margin we make on these products by altering (on purpose) which sales channel we sell them on. However, this may not work in all cases, e.g. we may stop selling the item because it is not sellable on the new sales channels.

I'm trying to do the following:

I have a list of productids where we applied this mechanism
I know how much we sold before and after the trial started, i know the margin made on each sale
I know how much we have received into stock from the item before and after the trial
I know how much we had on stock when we started the trial

I need advice on the following:
* Am i right to assume that the ratio of margin of total sales vs. the units we have receied into stock is the right answer? So if the margin is 10 before the trial and 20 after, but we received 2 items before and 5 after, we are worse off?

How should i involve the stock amount we had on sale when we started the trial in my calculations?
Is there any available reading i should look into to educate myself?
Last question: provided i get answers on the above questions, i'll be able to flag each of my products with a "SuccessfulTrial" flag - true or false. Based on that, i can build some regression to find which product specifics have an impact on the success of this mechanism. Is my thinking right?

Thanks for any answers,

Welcome to the forum. I don‘t know how well you understand statistical models. However, as you want to evaluate some (exogenous and controlled) change, you could start off with „difference in difference“ https://en.wikipedia.org/wiki/Difference_in_differences.

This is the standard method for ex post evaluation and it is very flexible in controlling other influential factors.


Backpropagation - softmax derivative I have a question on the backpropagation in a simple neural network (I am trying to derive the derivative for the backpropagation).

Suppose that the network is simple like so (forward pass):

$$\begin{aligned}z_1 &= xW_1 + b_1  & (1) \\
  a_1 &= \tanh{(z_1)} & (2) \\
z_2 & = a_1W_2 + b_2 & (3)\\
a_2 & = \text{softmax}(z_2)=\hat{y}_n & (4) \\
l_n & = \log{(\hat{y}_n)} & (5) \\
PL_n & = l_n\dot{y}_n^T & (6) \\
L & = -\frac{1}{N} \sum_{n \in N} PL_n & (7)
\end{aligned}
$$

So only one hidden layer. Let the dimensions be the following:

$x$ is $1 \times 2$, $W_1$ is $2 \times 500$, $b_1$ is $1 \times 500$, $a_1$ is $1 \times 500$, $W_2$ is $500 \times 2$, $b_2$ is $1 \times 2$, then $\log{(\hat{y_n})}$ is $1 \times 2$, then $\dot{y}_n^T$ is $2 \times 1$.

Then the backpropagation algorithm (for a single sample), for this staged computation would proceed as follows (equipped with the knowledge that the derivative of $X$ whose dimensions are $N \times M$ will preserve those dimensions):

$$\begin{aligned}
dPL_n & = -1/N & (7) \\
d\dot{y}^T_n & = l_n^T  dPL_n & (6) \\
dl_n & = \dot{y}^T_n dPL_n & (6) \\
d\hat{y}_n & = 1/\hat{y}_n & (5)
\end{aligned}
$$

Backpropagation equation $(5)$ above is a bit of an abuse of notation, but what I am trying to say that it is a vector, whose values are $[1/\hat{y}_n^{(1)}, 1/\hat{y}_n^{(2)}]$. But I am stuck at this step. Because I am not sure about the softmax. I have found this. Which tells me that if $a_2$ is the resulting softmax vector, then I can take following four derivatives: $\frac{\partial a_2^{(1)}}{\partial a_2^{(1)}}$,$\frac{\partial a_2^{(1)}}{\partial a_2^{(2)}}$,$\frac{\partial a_2^{(2)}}{\partial a_2^{(1)}}$,$\frac{\partial a_2^{(2)}}{\partial a_2^{(2)}}$. However, the dimensions of $z_2$ are $1 \times 2$, thus there should only be two partial derivatives. It makes conceptual sense to me, that $dz_2$ should equal to $\left[\frac{\partial a_2^{(1)}}{\partial a_2^{(2)}}, \frac{\partial a_2^{(2)}}{\partial a_2^{(1)}} \right]$, but I cannot motivate my intuition. Also, if I generalize this. Suppose $z_2$ is $1 \times 10$, what symbolic expression for derivative: $dz_2$ would we have then?

(I was motivated to implement my neural network from scratch after reading this post by Stanford; hence this question).

Another resource I am using. I am inclined to believe that $dz_2 = \hat{y}_n - \dot{y}_n$, but why, I do not know ;(

I am also missing $dl_n$ in backprop in $(5)$, the only way to end up at $1 \times 2$, is if we do: $d\hat{y}_n = 1/\hat{y}_n \circ dl_n$.

I have just looked at the porblem in the following way, which did not help me much. We need to find $dz_2  = \frac{\partial L}{ \partial z_2}$, we can rewrite this using backpropagation chain rule $$\begin{align} \frac{\partial L}{ \partial z_2} &= \frac{\partial l_n}{\partial z_2} \frac{\partial L}{\partial l_n} \\ &= \frac{\partial l_n}{\partial z_2} dl_n \\ &= \frac{\partial}{\partial z_2} \log{(\text{softmax}(z_2))} dl_n \\ &=\frac{\partial}{\partial u} \log{u} \frac{\partial u}{\partial z_2} dl_n \\ &= \frac{1}{\text{softmax}(z_2)} \frac{\partial}{\partial z_2} \text{softmax}(z_2) dl_n \end{align}$$

Also, same question on Maths.SE.

The partial derivative of the cost function with respect to the neuron $i$ ,in  the layer $Z_2$ defined in your questions above, is $\frac{\partial L}{\partial z_i}$, where $L$ is the cost function.

please note that the derivation will be on the neuron level not at the layer level, where $z_i$  is the neuron number $i$ in the output layer/softmax layer/ $Z_2$ layer. In other words $Z_2$ layer consists of 2 elements $z_0$ and $z_1$ and the derivation will be w.r.t the neurons/elements.

I re-arrange your equations to keep things as simple as possible: combining equations 5,6,7 yields:

$L = -\ \sum_{n} \dot{y}_n$ x $ log(y_n)$

Here is a brief derivation

$$\frac{\partial L}{\partial z_i}=-\sum_n\dot{y}_n\frac{\partial \log y_n}{\partial z_i}=-\sum_n\dot{y}_n\frac{1}{y_n}\frac{\partial y_n}{\partial z_i}$$

The softmax function $y_n$ is defined as
$$y_n = \frac{e^{z_n}}{\sum_ke^{z_k}}$$

when n=i
$$\frac{\partial y_n}{\partial z_i} = \color{blue}{y_i(1-y_i)}$$
when n <> i
$$
\frac{\partial y_n}{\partial z_i} = \color{red}{-y_ny_i}$$

now let's substitute $\frac{\partial y_n}{\partial z_i}$

$$\frac{\partial L}{\partial z_i}=-\dot{y}_i\frac{1}{y_i}\color{blue}{y_i(1-y_i)}-\sum_{n\neq i}\dot{y}_n\frac{1}{y_n}\color{red}{({{-y_ny_i}})}$$
$$
\\=-\dot{y}_i(1-y_i)+\sum_{n\neq i}\dot{y}_n({{y_i}})\\=-\dot{y}_i+{\dot{y}_iy_i+\sum_{n\neq i}\dot{y}_n({y_i})}
\\={y_i\left(\sum_n\dot{y}_n\right)}-\dot{y}_i=y_i-\dot{y}_i$$ where $\sum_n\dot{y}_n = 1$

For detailed derivation and explanations please see
https://deepnotes.io/softmax-crossentropy


What are the differences between SVC, NuSVC, and LinearSVC? What are the differences between SVC, NuSVC, and LinearSVC?

Please shed some light.

In scikit SVC and nuSVC are mathematically equivalent with both methods based on the library libsvm. The main difference is that SVC uses the parameter C while nuSVC uses the parameter nu.

LinearSVC is based on the library liblinear. As the documentation says, LinearSVC is similar to SVC with parameter kernel='linear', but liblinear offers more penalties and loss functions in order to scale better with large numbers of samples. Please check out this question and this question for more details.


How do I control for some patients providing multiple samples in my training data? I have a dataset with 50 patients. The patients are tracked over many years with a set of a few thousand features measured at somewhat random intervals. I am trying to predict a certain outcome (this is a regression problem) which is measured either once, twice, or three times per patient over the entire time period the patient is tracked. So far, I have been assuming that each outcome feature is a data point in my Y vector, and using the closest time point where the features are measured for the patient. I want my final model to be able to predict the outcome variable from only one measure of features, which is why I am OK ignoring the temporal aspect of the data. However, I am not sure how to control for the fact that some patients contribute 2 or even 3 samples to my training data. Wouldn't this affect the performance of my model since there is some in-patient variation? How do I account for this?

Like all models, the final outcome will depend heavily on the data you use to train.

Given you don't have many samples for each patient and that the samples are temporally irregular, I imagine you might be lumping all recorded data (from all patients) into a single training set... ? In that case, it will be important to either perform random sampling or something like stratified sampling - both without replacement - to make sure patients don't appear in both train and test sets. Stratification simply means to create training/test splits, which do their best to preserve class balance. So if you select one feature as most important, stratification will make sure that training and test sets each have a proportional number of samples exhibiting that feature.

If you are using R, look into the Caret package - here is an example of usage. That is a very powerful package in general and worth investigating (if you use R or not!).
If you are using Python, have a look at the Splitter Classes, which include: Stratified K-folds and Stratified Shuffle Split, among others.

If temporal ordering does play a big role, taking the most recent sample is usually considered best practice. For example, measurements driven by general information flow, e.g. finance, would also consider the most recent measurement as the most valueable for prediction.

This is the case in many applications, but perhaps not with your measurements - in medical data, behavioural/environmental conditions of the patient around the time of measuring may outweigh the recentness of the data.

One more observation, in the you are trying to predict some feature that is very specific to a single patient (and you say you may only have one observation from each patient!), then you might have a hard time making accurate predictions.


Designing CNN that does one column convolution across the x-axis I am currently working on designing a certain number of CNN for extracting features from images.

The images are spectogram, and each have a shape being (276,x,3).
X is here the number of column, which incidently also the length of the feature vector that should  be created.

So the cnn somehow has to use a kernel that has to be 276 rows and 1 column wide, but is it possible in keras to make a 2d kernel and perform 1d convolution. The most important factor is the 1d convolution and the shape of the 2d kernel, as it is being used to alter the importance of some of the entries in the spectogram.

Designing CNN that does one column convolution across the x-axis
So the cnn somehow has to use a kernel that has to be 276 rows and 1 column wide, but is it possible in keras to make a 2d kernel and perform 1d convolution.

I'm not sure I understand your question correctly, but if you have input images that are 276 high, X=500 wide and have 3 color channels, then the following performs column-wise convolutions across the X-axis:
rows, cols, chans = 276, 500, 3

...
model.add(Convolution2D(64, rows, 1, input_shape=(rows, cols, chans)))
...

The output shape of this layer will be (1, 500, 64).
I inserted "64" as the number of filters.. you can set that to whatever you need / whatever works best.
Note: shapes are specified in tensorflow mode (i.e. the order is (rows, cols, channels)). Depending on your configuration you might have to change those shapes to theano mode: (channels, rows, cols).

So the cnn somehow has to use a kernel that has to be 276 rows and 1 column wide,

It actually has to use a kernel that is 276 rows, 1 column and 3 channels wide since your input images have 3 channels.. it would be 276 rows and 1 column wide if you used grayscale images.. but that shouldn't matter.


How to favour a particular class during classification using XGBoost? I am using a simple XGBoost model to classify 2 classes (0 and 1) in a binary context. In case of the original data, the 0 is the majority class and 1 the minority class. The thing which is happening is that in case of classification, most 0s are being classified correctly, with many going into 1s, but most 1s are being misclassified into 0s.

I am fairly new to this, and having looked at various documentations and questions on SE, am really confused as to how I can specify my XGBoost model to favour class 1 (to be precise, if most 0s are misclassified into 1s, that is not a problem, but I want that most 1s are correctly classified as 1s (to increase the true positives, if there are false positives that is something which isn't much of a problem). The segment of code I am presently using to train and test the XGBoost are as follows (afterwards I use the confusion matrix in which the true positives (1s) are highly misclassified into 0s).

from xgboost import XGBClassifier

# fit model on training data
model = XGBClassifier()
model.fit(X_train, labels) # where labels are either 1s or 0s

# make predictions for test data
y_pred = model.predict(X_test)
y_pred = y_pred > 0.70 # account for > 0.70 probability
y_pred = y_pred.astype(int)

print(y_pred)

I just want to know if there is a simple way to specify to the XGBoost model any parameter in my code, so that the true positive rate can be increased? I can compromise of false positives being high, but I want the number of 1s to be correctly classified as 1s, instead of most of them going into 0s. Any help in this regard is appreciated.

UPDATE:

I have now tried to use scale_pos_weight in the XGBoost, with its value set to 0.70 (a random figure), but it is still landing most samples to 0, instead of 1.

XGBoost has the scale_pos_weight parameter to help with this, depending on how you want to evaluate it (see tuning notes). It should be the ratio of negative count to positive count (or inverse based on how you indexed your classes).

An example in Python is here.


Assigning values to missing target vector values in scikit-learn I have a dataset containing data on temperature, precipitation, and soybean yields for a farm for 10 years (2005 - 2014). I would like to predict yields for 2015 based on this data.

Please note that the dataset has DAILY values for temperature and precipitation, but only 1 value per year for the yield (since harvesting of crop happens at end of growing season of crop).

I would like to build a regression or some other machine learning based model to predict 2015 yields, based on a regression/some other model derived by studying the relation between yields and temperature and precipitation in previous years.

As per, Building a machine learning model to predict crop yields based on environmental data, I am using sklearn.cross_validation.LabelKFold to assign each year the same label.

The question is that since I have a single target value per year, do I need to interpolate to fill in target values for all the other days of the year? Should I just use the same target value for each day of the year?

The model likely won't have much predictive power if the input is a single day. No weather patterns longer than one day can be captured that way.

Instead you should aggregate the days together. You can come up with different features that describe your larger, aggregated unit of time (months, year). For example mean precipitation is a very simple one. Binning the data and using counts within those bins would also work.

More advanced options would roll the time all the way up to a full year and learn a feature set at that level.


How would I model hysteresis? I have the task of modeling the current to torque mapping for a given motor. I have an experimental set up where I can retrieve current, torque pairs.

Now my initial approach was to model the relationship with a regression curve, but I realized that the motor certainly shows some kind of hysteresis.

How would I be able to compute a current to torque mapping that also includes hysteresis from the data I obtained

If your goal is solely predicting torque for a given engine, you can ignore hysteresis. A regression model will implicitly model hysteresis in the learned parameters of the model.

If your goal is building a model hysteresis for a given motor, then you should systematically measure performance with different recent histories. Then apply a time series model to model the effect of recent history.


Why is predicted rainfall by LSTM coming negative for some data points? I have used supervised learning with LSTM network using tanh activation function and 0.1 dropout for time series prediction.my loss='mean_squared_error', optimizer='adam'. The predicted time series is shown below where x axis shows future months and y axis shows rainfall in mm.Orange line is predicted and blue line is actual. Some points are being predicted below 0 (negative) even though training dataset has all points having values >=0. My training dataset is of 64 years (all positive data)  and I am predicting for 12 years (144 data points shown)

Model code using ReLU activation:

model = Sequential()
# LSTM model
model.add(LSTM(128, batch_input_shape=(batch_size, look_back, 1),activation='relu', stateful=True, return_sequences=False))
model.add(Dropout(0.1)) #for better regularization
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')

Currently your model is not restricted from guessing any number. In this case, you may want to use something like a ReLU activation function to restrict your output domain.

The ReLU activation needs to be added to the last layer of your model. If you add it to an intermediate layer but not the last layer then your model can still output negative numbers, e.g. in the case of the last layer having a negative weight.

Note this is not really a new idea. There are strong similarities between logistic regression and what we’re doing here. In both cases we want to limit the possible guesses, in your case to non-negative numbers and in logistic regression‘s case to [0, 1].


Does CART algorithm takes into account in the order of the set of attributes? when using matlab command 'fitctree' for classification purpose,  and I change the order of the attributes I do not find the same Tree and thus the same classificaiton error? why? CART algorithm does take account on the attributes firstly introduced ?

A possible answer.

Check to see if you have nominal variables with more than a reasonable number of levels. If that is the case then a heuristic is applied to select the best split. One such heuristic is to group levels on left to right, and that might affect the order. Anyway, it is reasonable to expect that the order of levels should not affect the output.

See this link for detalied explanation on how this is implemented on matlab
matlab documentation.


Split time series by python or by keras? In Python you can use TimeSeriessplit() to split a time series properly for training but you can also do the same(?) in Keras by TimeseriesGenerator.
Which one is recommendable? And/or what are certain differences?

Assuming they can both do what you want them for (can perform the correct splits for your needs), I would recommend simply using the one that fits best with your pipeline. E.g. the data is being fed straight into a Keras model, just do as much as possible in Keras -> use the TimeseriesGenerator. Otherwise, stick to Sci-kit Learn.

The both generate the splits, meaning they only actually create each split as you loop through the data. This saves memory.

Looking at the source code of the Keras variant, it doesn't seem to support running on the GPU if you combine e.g. with your Tensorflow graph - meaning there really isn't a big difference between the two, just the APIs/functionality.


How to get the probability of belonging to clusters for k-means? I need to get the probability for each point in my data set. The idea is to compute distance matrix (first column contsins distances to first cluster, second column conteins distances to second cluster and etc). The closest point has probability = 1, the most distant has probability = 0.
The problem is linear function (like MinMaxScaller) have output where almost all points have almost the same probability.

How to choose nonlinearity for this task? How to automatizate this process on python? For example for the most closest point p=1, for the most distant point that belongs to cluster p=0.5, for the most distant point p is almols 0.

Or you can propose another methods for computing this probability.

Let us briefly talk about a probabilistic generalisation of k-means: the Gaussian Mixture Model (GMM).

In k-means, you carry out the following procedure:
- specify k centroids, initialising their coordinates randomly
- calculate the distance of each data point to each centroid
- assign each data point to its nearest centroid
- update the coordinates of the centroid to the mean of all points   assigned to it
- iterate until convergence.

In a GMM, you carry out the following procedure:
- specify k multivariate Gaussians (termed components), initialising their mean and variance randomly
- calculate the probability of each data point being produced by each component (sometimes termed the responsibility each component takes for the data point)
- assign each data point to the component it belongs to with the highest probability
- update the mean and variance of the component to the mean and variance of all data points assigned to it
- iterate until convergence

You may notice the similarity between these two procedures. In fact, k-means is a GMM with fixed-variance components. Under a GMM, the probabilities (I think) you're looking for are the responsibilities each component takes for each data point.

There is a scikit-learn implementation of GMM available if you wanted to look into that, but I'm guessing you just want a quick way to amend your existing code, in which case, if you're happy to assume your clusters are fixed-variance Gaussians, you could transform your distance matrix element-wise as $y = e^{-x}$ (giving you an exponential fall-off), and then calculating the softmax over your columns (normalising your distribution so $P(Y=1) + P(Y=2) + ... + P(Y=k) = 1$).

It's worth pointing out that the assumption your clusters are fixed-variance Gaussians isn't necessarily valid. If your dimensions have wildly different scales, this may produce strange results, as dimensions with smaller-magnitude units will appear more "probable". Standardising your data before running your clustering procedure should remedy this.


How to predict the duration of burst given several series I've got sets of time series data collected from Weibo. It contains the number of posts under certain topics over a year. It can be found that from time to time there're bursts of discussion on certain topic. What's interesting is some burst die down quickly while others sustain on the agenda for quite a long time and die down gradually. How to predict and model the decays given a time series data. Also, I've got data of some exogenous data including time series data of links, pictures, and retweeting and original data in the community. Can I predict the duration of decay from these variables?

I guess, a nice place to start is seminal work of John Kleinberg:  https://www.cs.cornell.edu/home/kleinber/bhs.pdf
Yes, this paper is outdated, but anyway it's still very influential and simple. Also, digging articles, citing this paper, could be very helpful. By the way, these guys solve very similar task ( and they even use Weibo logs as well!): http://www.csce.uark.edu/~xintaowu/publ/pakdd15.pdf


Keras intuition/guidelines for setting epochs and batch size I'm using Python with Keras to make a convolutional neural network (CNN) for an image classifier. I took about 50 images of documents and 150 images of non-documents for training. I shrunk the resolution, using scipy.misc.imresize(small, (32,32)) to standardize the images, when looking through these pixelated images I thought that I could still tell the difference between photos and documents, so I figured that the ML algorithm should be able to as well.

My question is in regards to the number of epochs and batch size. I'm currently using model.fit(X_train_temp, y_train_temp, epochs=N_epochs, batch_size=batch_size, verbose=verbose) . I've seen this question regarding the topic here:

How to set batch_size, steps_per epoch and validation steps

But the answer was very definition-based; I'm looking for intuition. I would like to know if there are general guidelines as to what values to set the number of epochs and batch size to for a given problem. I performed a crude parameter sweep across the number of epochs and batch size. Here is the CNN model:

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3), input_shape=input_shape, activation='sigmoid'))
model.add(Conv2D(32, kernel_size=(3, 3), input_shape=input_shape, activation='sigmoid'))
model.add(Flatten())
model.add(Dense(num_classes, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

And here are the results:

Here are some observations that I've made so far. The circled region seems to be pretty good for training since high accuracy is achieved relatively early on and it does not seem to oscillate much as further epochs pass. It seems that batch_size = 1 is generally a bad idea since training does not seem to improve the model. Additionally, it seems that batch_size >~ N_epochs seems to be desirable.

Question 1: is batch_size = 1 generally a bad? Or is this just for my particular parameter sweep?

Question 2: Is it generally the case that N_epochs >~ batch_size is recommended?

Question 3: Can anyone speak to the robustness of the generalizations in questions 2?

Thank you.

Actually it's controversial and it can differ from one problem to another. the best thing to do is to search the hyperparamteres space for an optimal result.

For Question 1

Here I quot the father of CNN (Yann Lecun) :
"
Training with large minibatches is bad for your health.
More importantly, it's bad for your test error.
Friends dont let friends use minibatches larger than 32.

Let's face it: the only people have switched to minibatch sizes larger than one since 2012 is because GPUs are inefficient for batch sizes smaller than 32. That's a terrible reason. It just means our hardware sucks.

What's worse is that the easiest way to parallelize training is to make the minibatch even larger and distribute it across multiple GPUs and multiple nodes.
Minibatch sizes over 1024 aren't just bad for your health. They cause brain tumors.
They learn quickly, but the wrong thing.
...

Update2: I'm pointing to this particular paper as a way to make a general point. But I'm not necessarily endorsing the particular results nor the methodology in this paper (some folks have pointed out a few flaws in the methodology). Still, the main point is that the only reason to use a batch size larger than 1 is the limitations of the limitations of the hardware at our disposal."

In this paper a value for batches between 2 and 32 is recommended

For Questions 2 & 3:

Usually an early stopping technique is used by setting the number of epochs to a very large number and when the generalization error gets worse we just stop. please see

For more details and explanations please see

At the end I would recommend using "ReLu" activation function for the hidden layers rather than the "sigmoid". it learns much faster


Is LASSO regression implemented in Statsmodels? I would love to use a linear LASSO regression within statsmodels, so to be able to use the 'formula' notation for writing the model, that would save me quite some coding time when working with many categorical variables, and their interactions. However, it seems like it is not implemented yet in stats models?

The question-asker resorted to scikit-learn until now, but statsmodels has just come out with its implementation of Lasso regression. The docs here are pretty self-explanatory and concise.


Example of a problem with structured output labels I'm studying SSVM (Structured SVMs).
On my book is stated that Structured SVM is an extension of the SVM, in which

  Each sample is assigned to a structured output label z ∈ K, e.g. partitions,
  trees, lists, etc.

It's not clear to me what a structured output label is.

Could you provide me with an explanation of this term along with some tangible examples?

Structured Learning is basically learning prediction functions that is used to map input data to complex output space.
The regular SVMs are used for univariate classification where input is mapped to atomic labels or regression where input is mapped to vectors or scalar numbers.
Structured Output learning extends this very process to more complex output spaces like:
- Multi Class Classification : where output is a "set" of class labels.
- Object localization Problem : where output space defines bounding boxes.
- Sequence alignment problems in protein homology detection,
- Parts of Speech tagging in NLP , etc.

Hope this clears it up for you.


NLP: Compare tags semantically with machine learning? (finding synonyms) Let's say I have multiple tags that I need to compare semantically. For example:

tags = ['Python', 'Football', 'Programming', 'Handball', 'Chess', 'Cheese', 'board game']

I would like compare these tags (and many more) semantically to find a similarity value between 0 and 1.
For example, I want to get values like these:

f('Chess','Cheese') = 0.0  # tags look similar, but means very different things
f('Chess', 'board game') = 0.9 # because chess is a board game
f('Football', 'Handball') = 0.3 # because both are sports with a ball
f('Python', 'Programming') = 0.9 # because Python is a programming language

So what is the state of the art approach to get a function f like this? I know that machine learning might be doing this, but this area is huge and overwhelming for me (on the first glance it looks like NLP focuses on other problems).
So what would be the best approach for this specific problem?

It's a complex problem, the standard method is to train a model which represents this semantic similarity function using a large text corpus (i.e. not only the tags themselves), by considering that the meaning of a word is provided by its context words.

A more direct approach is to use a pre-trained word embeddings vectors that can be compared directly to find their similarity.

A similar idea is to use WordNet, which is a database of words with their semantic similarity to each other.


Is it a good idea to tune the number of folds for cross validation when tuning hyperparameters of RF I'm new to data science. I'm trying to get the best model for Random Forest. Unfortunately, I'm not sure if my idea can produce a good generalized model.

1) I have split data to TrainingSet (70%) and TestSet (30%)

2) Then randomly selected hyperparameters for RandomForest and a number of folds for CrossValidation between (2-15)

3) Then I fetch the TraingSet data to the RandomForest learner

4) Then do CrossValidation of the model
- from CrossValidation I'm getting array with predictions

5) Measure Accuracy of prediction from CrossValidation against targets from the TrainSet

6) Repeat all steps and try minimize the AccuracyError

Is this a good way to get best generalized model?

Do I need to split data into TrainSet and TestSet?

OR I should I search for the optimal hyperparameters and number of folds with all data? I have feeling I don't need to split data when using k-fold CrossValidation during Hyperparameters tunning.

Do I need to split data to TrainSet and TestSet?

It depends:

It is acceptable to cross-validate on the whole dataset to obtain the performance of a particular model, since in this case the model is always tested on unseen data.
In your case, the selection of the best generalized model/hyperparameters is part of the training stage, so yes it makes sense to cross-validate only on your training set and then measure the performance of your final model on some fresh data. This is because the selection of the best hyperparameters (especially from a large set of possibilities) can still be partly due to chance despite CV (especially if the dataset is small), so testing on unseen data avoids the risk of overestimating the performance.

  Then randomly select hyperparameters for RandomForest and a number of folds for CrossValidation between (2-15)

Usually the number of folds k for k-fold CV is not really part of the hyper-parameters, so it's a quite unusual to do this (as far as I know). The risk is that your final model might be selected partly because a particular k happens to produce a higher performance on the training set. To see this intuitively: a larger training set is more likely to produce a better model, and a higher k makes CV use a larger training set for every fold. So a higher k is likely to artificially improve the performance, just because the training set is larger. This is why it's safer to use the same k for comparing different hyper-parameters, possibly repeating the whole process for different values of k.  Also for a particular value of k you can randomly reshuffle the partitions to minimize the effect of chance in the CV partitioning.


Orange Data Mining load saved models I am planning to use the Orange Data Mining Tool for easy data exploration and model generation. What is still unclear to me is: after finding a good model, what can I do with it, how can I use or deploy it in production?
I already found out that there is no Orange server which can run the Orange workflow. But is it at least possible to load the model in Python and use it there which was generated and saved via the Orange UI?

Yes, this is what the Save Model widget does under the Model tab. Create your model, then click save model and your model will be saved to a pickle file. Then just load your pickle file in Python.

Here is a like to a similar question on Stack Overflow:

https://stackoverflow.com/questions/45598627/orange-save-model-in-python-script


How many features do I select when doing feature selection for regression algorithms? Is R2 and RMSE good measures of success for overfitting? Context: I'm currently crafting and comparing machine learning models to predict housing data. I have around 32000 data points, 42 features, and I'm predicting housing price. I'm comparing Random Forest Regressor, Decision Tree Regressor, and Linear Regression. I can tell there is some overfitting going on, as my initial values vs cross validated values are as follows:
RF: 10 Fold R Squared = 0.758, neg RMSE = -540.2 vs unvalidated R Squared of 0.877, RMSE of 505.6
DT: 10 Fold R Squared = 0.711, neg RMSE = -576.4 vs unvalidated R squared of 0.829 and RMSE of 595.8.
LR: 10 Fold R squared = 0.695, neg RMSE = -596.5 vs unvalidated R squared of 0.823 and RMSE of 603.7
I have already tuned the hyperparameters for RF and DT, so I was thinking about doing feature selection as a next step to cut down on some of this overfitting (especially since I know my feature importances/coefs).I want to do feature selection now with a filter method (i.e. pearsons) as I want to keep the features going into each model consistent.
Question: How would I decide on a number of features to choose using feature selection? Is it arbitrary? Or do I basically just remove all of them that don't have much correlation with the data? Is there a way to spit out an optimized set of features without doing grid search or random search?
Follow up question: Are the R2 and RMSE cross validated values good measures of success for overfitting comparison?

You have overfitting when your model corresponds too closely to the training data and may therefore fail to fit additional data or predict future observations reliably. Basically, when the performance on training (or validation) set are much more better than on test set.
You have the opposite case: performance on test set much better than validation set.
This can happen if the two sets do not come from the same distribution (how did you divide train/validation/test set?). In this case, the data of the test set could be much easier to predict.
Another possibility is that the size of the test set is too small.
My suggestion is: shuffle your dataset; divide in 70% training and 30% test set. Do cross validation on training set. Compute the R2 on both sets.


How to find out what portions of an image is helping CNN to classify it I am working on an image classification problem using Transfer learning. Right now, I am getting an accuracy of 75% on train data and 67% in test data. Now I want to understand what portions/parts of the image are being utilized for classification graphically.
Are there any packages that can return a copy of one of the input images with markings on the most useful pixels or parts of the image?

Yes, the typical approach is to obtain the saliency map of the input, which are "heatmaps" of the contribution of each pixel to the final classification.
In this free online book about Explainable ML, you can find the most relevant approaches to obtain saliency maps, like vanilla gradients, together with other pixel attribution techniques.
Here you can find an end-to-end tutorial on how to implement saliency maps with Keras.


Size of Output vector from AvgW2V Vectorizer is less than Size of Input data Hi,
I have been seeing this problem for quite some time. Whenever I tried vectorizing input text data though avgw2v vectorization technique. The size of vectorized data is less than the size of the input data. Is there any statistical reason behind this? In my case 100K is the size of the input and it gives 999,98 sized output
I'm wondering what is causing this problem. Thanks in advance

Code:

listofsentences=[]
for sent in x_train:
    listofsentences.append(sent.split())

training_model = Word2Vec(sentences=listofsentences, workers=-1,min_count=5)
modelwords = list(training_model.wv.vocab)

std_avgw2v_x_train = []
for everysentence in tqdm(listofsentences):
    count = 0
    sentence = np.zeros(100)
    for everyword in everysentence:
        if everyword in modelwords:
            w2v = training_model.wv[everyword]
            count += 1
            sentence += w2v

    if count != 0:
        sentence/=count
        std_avgw2v_x_train.append(sentence)

len(std_avgw2v_x_train)
>99998

len(x_train)
>100000

EDIT1: I'd like to mention that I Just started learning ML, Its been 55 days since I started. Also, the same code gives our 100K output samples While I vectorize with TFIDFW2V

I have attached the image of the same. Kindly look into it

I think the issue can be one of the two :

A . You have missing value in x_train

B . One of the values in x_train has no word that is there in modelwords.

In both the cases ,

if everyword in modelwords:
     w2v = training_model.wv[everyword]
     count += 1
     sentence += w2v

condition doesn't get satisfied and you end up not addding any new value to sentence


Predict next month's loan balance from historical data I work for a bank. Every month I get a list of 10,000 customers of the bank with the outstanding loan balance for each customer. I add them up and report the total outstanding balance every month. I have data for past 24 months.

I am asked to give a rough prediction of the next month's total outstanding balance. I am not a statistician, hence need some help. So far ,I have below ideas, can someone comment on pro/cons of each?

Compute the average monthly % growth over last 6 months on Total balance; Multiply current balance by % growth  to predict next month balance
Do the same at an individual customer level, that gives us 10,000 predicted balances for next month. Add them up to get total balance
Are there any time series techniques I can use here?
Can I find the answer by some simulation techniques?

You have quick and easy but rough predictions:

The "meteo method": next month will be the same as last month. No as bad as you may think, and you can even replace last month by an average of the last 6 month.
Linear regression: the meteo method will fail if there is a constant growth. Make a linear regression in function of time $x(t) = a \cdot t + b$. Your forecast will be $x(today+1)$.
Exponential regression: use a regression on a growth in percentage (exponential growth). You can find the best $a$, $b$ fitting $x(t) = a \exp(b \cdot t)$, or the best $c$, $d$ to fit $log(x(t)) = c \cdot t + d$. These are the same formula in disguise and you can use a linear regression for the last one (fit $y(t) = log(x(t))$ instead of $x(t)$).

Note that this a "scientific" version of you method 1.
Exponential smoothing: It is an average giving more weight to the last values. There is a technical trick that makes computation specially easy if you have to forecast every month. In reference with the wikipedia page, take $\alpha = 1/6 = 0.1667$ for a 6 month history.

In the future, you will ba able to add a trend (Double exponential smoothing) and a seasonality (Holt-Winter model).

I would avoid method 2, consisting in adding forecast of individual account. This method works when every thing is linear so that errors cancel each others. When the the process is not linear, you'll have a systematic bias which will add together.

I also mention that your main challenge will more probably be the seasonality than the main trend.


Creating a posterior distribution for classic coin flipping in python using grid search I'm reading the book "Bayesian Analysis with Python" and the author provides some python code designed to show the grid search method of obtaining an approximate posterior distribution for the classic coin flipping example. In this example we set a prior on the probability of obtaining a head, then given some data our likelihood is the binomial distribution.

So from my understanding using grid search we will break the interval $[0,1]$ into chunks, in our case 100. We have a prior probability $P(\theta=\theta_{0})$ for each of the points in the discretised grid we multiply this by $P(D|\theta=\theta_{0})$ and we multiply these together to get an un-normalised estimate at each location.

The code is the following

def posterior_grid_approx(grid_points=100, heads=6, tosses=9):
    grid = np.linspace(0, 1, grid_points)
    prior=np.repeat(5,grid_points)
    likelihood=stats.binom.pmf(heads, tosses, grid)
    unstd_posterior = likelihood * prior
    posterior = unstd_posterior / unstd_posterior.sum()
    return grid, posterior

I don't understand why the prior is set to a list of $5$'s isn't this meant to be a probability value? From my understanding they've broken the grid into 100 different points obtained a binomial pmf for each of these locations and then multiplied the number by 5. How is this a valid prior?

I'd really appreciate if someone could clearly explain the reasoning behind this as i like to understand the code as clearly as the concept. Thanks!

In short

The problem assumes a uniform prior distribution function. All possible $P(H)$ are equally likely. Because they are standardizing the probability distribution function at the end it does not matter what value is placed in

prior=np.repeat(1,grid_points)

Likelihood function

The likelihood function answers the question, how probable is the prior which we assumes.

For example, when flipping a fair coin we can assume that the probability of getting a heads is $P(H) = 0.5$. Now, if we do an experiment and in 3 flips we observe 3 heads then $P(HHH|P(H)=0.5) = 0.5^3$. This tells us that their is only a 0.125 chance that we selected the correct prior.

$Posterior \propto Likelihood \times Prior$

The code

As you observed it is strange that they set their entire list with the value $5$. However this will have no effect on the results because the before last line standardizes the results. Try changing the $5$ to a different value and you will see that your results remain unchanged.

Let's go through the code

grid = np.linspace(0, 1, grid_points)

This creates an array between $0$ and $1$. This is the different probabilities we will assume to calculate our likelihood. For example if we use 5 grid_points we have $[0, 0.25, 0.5, 0.75, 1]$. So we will assume that $P(H)$ is equal to these values in turn and get the resulting likelihood for each value.

likelihood=stats.binom.pmf(heads, tosses, grid)

We can now set out prior however we wish. We can assume that the coin is fair, or we can have no clue and just assume that it's a uniform distribution. This is essentially what they are doing here. The value they chose makes no difference in the results. However, I think for clarity making it equal to 1 is more obvious.

prior=np.repeat(1,grid_points)

To calculate the posterior we multiply the prior by the likelihood

unstd_posterior = likelihood * prior

And then to get a probability distribution function we standardize the function.

posterior = unstd_posterior / unstd_posterior.sum()