What's the proper way to do back propagation in Deep Fully Connected Neural Network for binary classification I tried to implement a Deep fully connected neural network for binary classification using python and numpy and used Gradient Descent as optimization algorithm.

Turns out my model is heavily under fitting, even after 1000 epochs. My loss never improves beyond 0.69321, i tried checking my weight derivatives and instantly realized they're very small ( as small as 1e-7 ), such small gradients are causing my model to never have bigger gradient descent updates and never reach the global minima. I will detail out the math/pseudo code for forward and backward propagation's, please let me know if I'm on the right track. I will follow the naming convention used in DeepLearning.ai By Andrew Ng.

Say we have 4 layer neural network with only one node at the  output layer to classify between 0/1.

X -> Z1 - > A1 - > Z2 - > A2 - > Z3 - > A3 - > Z4 - > A4

Forward propagation

Z1 = W1 dot_product X + B1
A1 = tanh_activation(Z1)

Z2 = W2 dot_product A1 + B2
A2 = tanh_activation(Z2)

Z3 = W3 dot_product A2 + B3
A3 = tanh_activation(Z3)

Z4 = W4 dot_product A3 + B4
A4 = sigmoid_activation(Z4)

Backward Propagation

DA4 = -( Y / A4 + (1 - Y /  1 - A4 ) ) ( derivative of output activations or logits w.r.t to loss function )

DZ4 = DA4 * derivative_tanh(Z4) ( derivative of tanh activation, which I assume is ( 1 - (Z4 ) ^ 2 ) )
Dw4 = ( dZ4 dot_produt A3.T ) / total_number_of_samples
Db4 = np.sum(DZ4, axis = 1, keepdims = True ... ) / total_number_of_samples
DA3 = W4.T dot_product(DZ4)

DZ3 = DA3 * derivative_tanh( Z3 )
DW3 = ( DZ3 dot_product A2.T ) / total_number_of_samples
DB3 = np.sum( DZ3, .. ) / total_number_of_samples
DA2 = W3.T dot_product(DZ3)

DZ2 = DA2 * derivative_tanh( Z2 )
DW2 = ( DZ2 dot_product A1.T ) / total_number_of_samples
DB2 = np.sum( DZ2, .. ) / total_number_of_samples
DA1 = W2.T dot_product(DZ2)

DZ1 = DA1 * derivative_tanh( Z1 )
DW1 = ( DZ1 dot_product X.T ) / total_number_of_samples
DB1 = np.sum( DZ1, .. ) / total_number_of_samples

After the above back propagation steps I updated the weights and biases using gradient descent with their respective derivatives. But, no matter how many times I run the algorithm, the model never improves it's loss beyond 0.69 and the derivatives of output weights ( in my case dW4 ) is pretty low 1e-7. I'm assuming that either my derivative_tanh function  or my calculations of dZ is really off, which is causing bad loss values to propagate back to the network. Please share your thoughts whether my implementation of backprop is valid or not. TIA. I went through
back propagation gradient descent calculus

and

how to optimize weights of neural network .. and many other blogs, but couldn't find for what I was looking for.

The problem faced with back propagation with activation functions that has a max limit used in a Deep Neural Network (i.e. more then 2 layer network) is that the deeper the network is the quicker your back propagation will degrade.

Take for example your tanh activation function

If say you backpropagate from the output layer to the hidden layer and your tanh derivative is 0.25 then the next layer will only be limited by 0.25 from your tanh function your derived. Since you compute the next layer's derivative which in this case can be 0.5 of the total input and the previous layer's derivative totals to 0.25 then your derivative would be a total of 0.125. By the time the back propagation reaches the input layer, the weights would change by such a small fraction that it does not even matter.

This is why the ReLu function was such a big breakthrough as it does not have a max limit. There should be methods to counter this effect but ReLu is way more effective.


Does the transformer decoder reuse previous tokens' intermediate states like GPT2? I recently read Jay Alammar's blogpost about GPT-2 (http://jalammar.github.io/illustrated-gpt2/) which I found quite clear appart from one point :
He explains that the decoder of GPT-2 processes input tokens one at a time, only actively processing the last input token, the past tokens being saved in memory already and "passively" reused without reevaluation.

From my understanding of the transformer architecture, I had the impression that the decoder reevaluates every token generated at each generation. Is this then a difference between the decoder from GPT-2 or does the decoder from a "classical' transformer also work this way ?

Intuitively I would think that it would make more sense the reevaluate everything at each iteration since new dependencies between words can appear that weren't there at the beginning and would then not be taken into account if past processed words are passively reused.

I hope I am making sense, can someone with knowledge about the GPT2 architecture help me clarify this ?

The past token internal states are reused both in GPT-2 and any other Transformer decoder.

For example, in fairseq's implementation of the transformer, these previous states are received in TransformerDecoder.forward in parameter incremental_state(see the source code).

Remember that there is a mask in the self-attention blocks in the decoder that prevents the predictions and intermediate states to attend to positions equal to or greater than the current one, which means that the internal state won't change even if you recomputed them at every decoding step.

Update: of course it is technically possible to recompute past tokens attending future tokens, but then, what do you do with the future tokens after you re-compute the past ones? Do you recompute them? This is a totally different beast, which has been studied to some degree and is referred to as "iterative refinement". An example can be found in article "Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement". AFAIK, this kind of approach has not studied in autoregressive models, only in non-autoregressive ones.


Why do we Softmax at all? Why take softmax at all at the final layer for multi-class classification problems? For example softmax of the vector [1, .5]

Is [.621, .379]

I mean if we just took the straight ratio, it'd give me
[.667, .333] instead

Does that really make a difference?

Is it cause the vector can have negative numbers that we softmax things? What benefit do we get from making an odder way to give a ratio/probability to certain numbers as opposed to just taking a ratio of the numbers?

The ratio doesn't take into account the fact that the last layer may have negatives results, in which case the ratio doesn't work, while the softmax does. Another corner case is if the denominator of the ratio is zero.

Also, softmax has two interesting properties:

It will accentuate the differences between the input and the output, by squashing the lower values and increasing the highest value, which makes the output of the network closert to a max instead of a softmax (note that max is not differentiable and hence cannot be used).
In a classification problem, where a softmax layer is used, the loss function is the cross entropy, which has the form - y * log(y_hat). Turns out that log of softmax is very easy to compute, as log(a/b) is log(a) - log(b) and log(exp(x)) is simply x. That makes softmax more desired.


How to estimate a user's gender based on what apps the user download? I have a dataset of what apps users downloaded and I try to estimate these users' gender based on what apps they downloaded, using machine learning algorithm. However, what kind of features of the apps should I focus on? As far as I know, app category plays a big role. Do Google Play and Apple App Store estimate users' gender based on what apps they download?

If you have data on how many downloads have been done for each category, you can use that data to perform logistic regression, where your target variable would be 0/1 classification of male/female. This will form your benchmark result and then you can improve upon the results. Especially look for feature selection, because there might be categories of apps which have nothing to do with target. You should also try bi variate profiling of variables with target to see if there is good correlation with target.
Try using some advanced techniques once you have done some data exploration.
KNN, Random Forest and Neural network for classification might give good results. Also cart might work fantastically here.


Product classification in e-commerce using attribute keywords I am working on a product classification problem (E-Commerce) in which I have to identify product category based on keywords.

Say for example, if input is given as 'Samsung Galaxy On Nxt 3 GB RAM 16 GB ROM Expandable Upto 256 GB 5.5 inch Full HD Display' , It should correctly identify it as 'Mobile'.

The problem here is that I have certain key-value pairs on which I have to train the model. Previously, I was doing the same problem by training model on product level using LinearSVC and it was giving satisfactory results.

Details about data:

I have 39 classes/ categories currently, which might increase in future.
I am using a csv file which is of around 10 MB and it has around 7000 rows.(Refer below structure)
I am using LinearSVC from sklearn.svm

I have the following structure for training now:

--------------------------------------------------------------
| Attribute_Names   |  Attribute_Values        |   Category  |
--------------------------------------------------------------
| Brand             |  Samsung, Nokia, OnePlus | Mobile      |
| RAM, Memory       | 2 GB, 4 GB, 3 GB, 6GB    | Mobile      |
| Color,Colour      | Black, Golden, White     | Mobile      |
--------------------------------------------------------------
| Fabric, Material  | Cloth, Synthetic, Silk   | Ethnic Wear |
| Pattern, Design   | Digital, floral print    | Ethnic Wear |
--------------------------------------------------------------

I have the following queries:

1 - Which Model to use for this problem ?

2 - How would you handle such problem ?

3 - Any similar problem you have seen.

Any help is much appreciated.

EDIT:

step 1) - merge column(attribute_names, attribute_values)

step 2) - Cleaned the data(removing stopwords,special characters, steming)

step 3) - Feature extraction using TfidfVectorizer(stop_words='english',lowecase=True)

step 4) - OneVsRestClassifier(LinearSVC(loss='hinge',penalty='l2'))

This are the steps we have performed till now,and the results are not good enough (miss-classification still exist)

format/structure of data is same as we mentioned above.

The problem type you're dealing with is referred to as multiclass classification. Not all algorithms are suited to handle it, but tree based methods and neural networks are popular choices. If you need it to run quickly and probability calibration isn't too important, Naive Bayes also works quite well for some data sets. To see an example of a dataset of this type, check of the Kaggle Spooky Author Identification competition. The published kernels give some good examples of feature engineering and modeling choices.

(I'm assuming the category label is unique. If there can be more than one label per record, it's called multilabel classification, which I would handle by building a separate binary model for each of the 39 labels in your data set. For an example dataset, check out the Kaggle Toxic Comments competition.)

As for modeling with your data in specific, the structure you have it in now seems a bit odd to feed to a model. A training data set should have each row represent one record, and each column represent a feature with the value in the column describing the record, whereas in your format, any cell phone should have information regarding each value in the corresponding mobile Attribute_Names. When you get the raw input 'Samsung Galaxy On Nxt 3 GB RAM 16 GB ROM Expandable Upto 256 GB 5.5 inch Full HD Display', how is that transformed into a format that can be fed into the model?

Also, where are the Attribute_Names, Attribute_Values coming from, are they manually specified? If so, that is limiting the performance potential from a model since there could be additional words in the data the model could detect if left to generate features on it's own. For a good modeling flow, the training data should contain the inputted raw text, then process the text to generate features, then feed into the actual model to output a label. So one row of raw training data set would be:

------------------------------------------------------------------------
| Description                                            |   Category  |
------------------------------------------------------------------------
| 'Samsung Galaxy On Nxt 3 GB RAM 16 GB ROM Expandable   | Mobile      |
| Upto 256 GB 5.5 inch Full HD Display'                  |             |
------------------------------------------------------------------------

You would then turn the text into features using some sklearn pre-processors. Count Vectorizer or TF-IDF Vectorizer are popular choices. You can also create your own features from keywords from the Attribute key and values you already have, such as creating a Brand indicator column if the words Samsung, Nokia or OnePlus appear in the text, but I would never use only manually specified features when modeling with text features.


Structuring extensive medical histories + demographic information for prediciting future medical outcomes I'm looking for advice structuring extensive medical histories for predicting future outcomes, specifically hospital admissions.
Let's say I want to predict the whether or not someone will be admitted to the hospital in the next 60 days from today and for each person I have medical histories with tons of data like blood pressure readings, weight, hospital admissions, illnesses, etc. going back 5 years. Of course, stats like blood pressure readings and weight and other stats are are not all happening on the same day, which is where I think I'm getting tripped up.
I'm thinking of this following scenario -- Let's say blood pressure is a good predictor of having a future hospital admission.

Patient One had blood pressure readings that were rising 5 years ago and then had a hospital admission 14 days later.
Patient Two had blood pressure readings that were rising 3 years ago and then had a hospital admission 11 days later.
Patient Four had blood pressure readings that were rising 2.5 years ago and then had a hospital admission 17 days later, but in the last year there were many instances of rising blood pressure that did not lead to hospital admission.
Patient Four has shown rising blood pressure measurements in the past but has never been admitted to the hospital.

How could I structure this data so that if all patients were having rising blood pressure readings now, it might lead to a prediction that

Patients One and Two might have a hospital admission in the next 60 days
Patient Three and Four would likely not be predicted to have a hospital admission in the next 60 days.

I guess I'm having difficulty wrapping my heard around how to use extensive past histories that do not line up for any given patients, with respect to time and data availability (For example, I might not ever have blood pressure readings for a given patient, but because they do not know how to navigate the medical system so they take the ambulance to the ER every single month, therefore, they would likely be predicted to have a hospital admission in the next 90 days on any given day), while also taking into account recency.
I was initially thinking of having a row for every day for every patient, but this does not let me fully utilize a patients entire history which I think will be very important.  Here is another question I've been asking myself... let's say every row is a patient and i have a column called blood_pressure_reading.  Is there a way to represent all of a patients blood pressure readings over the last 5 years inside one cell?  Could I put an array of NULLs and actual readings going back the last 5 years here?

There are many options and many different problems here, I think you should start simple and then try to improve on it.
Imho the most basic option is to restructure the dataset so that there's one instance for every patient at any given time $t$, for instance every month. The binary label represents whether the patient was admitted to hospital in the period $t$ to $t$+60 days. The features should represent whatever relevant indicators can help the model based on past data (before $t$), for instance mean/standard deviation of the blood pressure in the past N months, possibly with various values of $N$ and/or a sliding window. This way both short-term and long-term information can be integrated. Occasionally you might have missing values, it's important to take this into account.
In this design there will be several instances for every patient, and this could cause a bias in the evaluation. There are two options: either there is one independent model for every patient, but this is probably too complex. Or the datasets is made of all the patients, and in this case it's important to avoid having the same patient in the training set and test set.
The problem of patients who would take an ambulance to the ER on a regular basis is a preprocessing problem: if possible, these instances should either be discarded or re-labelled as if it was a regular consultation, for example.


caret dummyVars on unseen data I created my dummy variables, trained my model and tested it as below:

dummy <- dummyVars(formula = CLASS_INV ~ ., data = campaign_spending_final_imputed)
campaign_spending_final_dummy <- predict(dummy, newdata = campaign_spending_final_imputed) %>% as.data.frame() %>%
  mutate(CLASS_INV = campaign_spending_final$CLASS_INV)

I now want to test my process on an unseen real world data. How can I do it? How can I encode the category class to its respective dummy encoding?

The predict function will work on the new data.  It has some options for how to handle entries that don't appear in the original data's labels.

https://rdrr.io/cran/caret/man/dummyVars.html


Data Science for Social Good I'm interested in doing a mapping or inventory of all data science for social good and technology for social good organizations/initivaives. Is anyone aware of any similar efforts or places to start?

Sources that I've found so far;

datakind
Data4Democracy
Data Science for Social Good
eScience DSSG projects

Other ideas: find a set of data that interests you depending on where you live for example and contribute to your community. Things like data.gov in US and open-data in Canada.


Adding variables to Random Forest decreases performance Let's say you have 6 variables.

A Random-Forest regression using the first 5 variables has an R^2 of 0.1. Another regression using just the 6th variable yields R^2 of 0.3. All of the first 5 variables are uncorrelated with variable 6 (<0.1 correlation in absolute value).

Why would a regression with all 6 variables have an R^2 of 0.31 or 0.29, i.e. adding first 5 variables to the 6th variable only negligibly increases performance by 0.01 or even decreases it?

Notice that all models are tuned via random search cross-validation tuning depth, number of features, splits and number of trees.

Even if your variables are linearly uncorrelated, they may still be non linearly correlated. Just using the correlation coefficient to say the data is unrelated is not enough for non linear regression.

Still, adding variables to a model may not improve the end result if the variables cannot explain anything new from the target data. This seems to be such a case.


Predicting the Evolution of a Vector with a Neural Network (RNN?) I want to predict the evolution of a 2D sequence (third dimension being time) and use the predicted sequence (along with several previously predicted sequences and some scalar values changing with time) to predict the next 2D sequence, and ultimately use that prediction to predict the following sequence in time.

Example of a 2D time-slice of the data:

I want to train my neural network to predict a new 2D "profile" (a time-slice of a 3D profile) based on the last ~3 profiles and a time-varying scalar parameter. What neural network architecture(s) would be capable of predicting the next 2D profile in time? I suspect RNNs, but which sub-models would work for this (e.g. seq2seq, NMT, etc.)?

Help would be greatly appreciated!

Ahoy, all! Figured out the answer.

Firstly, this is a time-series forecasting problem. You're looking to forecast (predict the temporal-evolution) of these 2D profiles. In a classification problem (most popular application of neural networks), the target value vector can be a one-hot encoded vector, where the "1" value is the correct classification; the other values (classes) are "0". You give this to your network and it will "learn" patterns in the data that correspond to each particular class. It will do this by minimizing the cost function until the predicted class (the one closest to 1) matches the correct class, and the other predictions are very close to zero. The network is essentially getting closer and closer to the actual target values.

This EXACT idea can be applied to this forecasting problem, where you feed in a vector of target values, but this time the vector isn't one-hot encoded (or, as in other cases, a probability distribution of the classes). It is just the known (target) values at each point along the profile. Yes, RNNs are good, but MLPs can also do the job. Specifically, for RNNs, you can use LSTM units and cells. No need for seqtoseq or NMT architectures for this particular problem.

One important difference is that forecasting problems do not use softmax functions. Instead, they use linear activation functions such as ReLU, and MSE/MAD for the loss.

Setting up this type of problem in your model (in the pre-processing step) may be slightly different than in classification problems. This tutorial here is very good overall, but also has invaluable pre-processing code that you can use in many forecasting applications (particularly the series_to_supervised() function.

Hope this helps!


Are Word2Vec and Doc2Vec both distributional representation or distributed representation? I have read that distributional representation is based on distributional hypothesis that words occurring in similar context tends to have similar meanings.

Word2Vec and Doc2Vec both are modeled according to this hypothesis. But, in the original paper, even they are titled as Distributed representation of words and phrases and Distributed representation of sentences and documents. So, are these algorithms based on distributional representation or distributed representation.

How about other models such as LDA and LSA.

Effectively, Word2Vec/Doc2Vec is based on distributional hypothesis where the context for each word is its nearby words. Similarly, LSA takes the entire document as the context. Both techniques solve the word embedding problem - embed words into a continuous vector space while keeping semantically related words close together.

On the other hand, LDA isn't made to solve the same problem. They deal with a different problem called topic modeling, which is finding latent topics in a set of documents.


In "Attention Is All You Need", why are the FFNs in (2) the same as two convolutions with kernel size 1? In addition, why do we need a FFN in each layer when we already have attention?

Here's a screenshot of the relevant section from Vaswani et al. (2017):

Convolution is actually applying a sliding window projection over a tensor. With the kernel size of 1, the convolution reduces a to a for loop over the vectors, i.e., each of the vectors gets multiplied by a filter matrix. But this is actually a definition of matrix multiplication which is how feed-forward layers are implemented.


Which one I should choose for random forest? I fit a dataset with a binary target class by the random forest. In python, I can do it either by randomforestclassifier or randomforestregressor.

I can get the classification directly from randomforestclassifier or I could run randomforestregressor first and get back a set of estimated probabilities. Then I can find a cutoff value to derive the predicted classes out of the set of probabilities. Both methods can achieve the same goal (i.e. predict the classes for the test data).

Also I can observe that

randomforestclassifier.predict_proba(X_test)[:,1])

is different from

randomforestregressor.predict(X_test)

So I just wanna confirm that both methods are valid and then which one is better in random forest application?

If you are doing a classification task, use random forest classifier. If you want probabilities use predict_proba or you can use predict directly to get classes. You might be getting correct results but understand that the random forest regressor works on a different cost function, and is not constrained to give outputs between 0 and 1, so what you are getting out of the regressor are not really probabilities. If you run it on enough datasets it might start giving you outputs greater than 1. Hence, bottom line stick to classifier. Also I will recommend reading up a bit on the differences of regression and classification tasks.


Does variance and standard deviation both measure how spread out the numbers are? I've heard from different sources that, standard deviation measures how spread out the numbers are. But I've also heard the same for variance.

Is it technically correct to say this statement for both std and var?

Yes it is. Standard deviation is a square root of variance. Square root is a monotonic transformation, meaning that it preserves the order, e.g, if a > b then sqrt(a) > sqrt(b), assuming a and b are non-negative and they always are for variance.

Standard deviation is easier to interpret and is more commonly used, when we calculate variance we square every term, taking a square root in standard deviation, we undo that.


I have this data set for crimes of a 12 month time period, over 250k rows. I want to predict future crimes by date and location I have this 250k data set with these features

    date_time       FullAddress             call_type priority   lat       long
0   6/14/17 21:54   10 14TH ST\, San Diego\, CA 1151    2.0 32.705449   -117.151870
1   3/29/17 22:24   10 14TH ST\, San Diego\, CA 1016    2.0 32.705449   -117.151870
2   6/3/17 18:04    10 14TH ST\, San Diego\, CA 1016    2.0 32.705449   -117.151870
3   3/17/17 10:57   10 14TH ST\, San Diego\, CA 1151    2.0 32.705449   -117.151870
4   3/3/17 23:45    10 15TH ST\, San Diego\, CA 911P    2.0 32.705722   -117.15035

Date and time , full address , lat and long , and call type , and level of the seriousness of the crime.
I want to predict the time when Future crimes will happen or predict the location it will happen again. How can I make that happen, will I use regression or classification? I already predicted the priority, but how can I predict the time it will happen or the location?

I predicted the priority but doesn't really give me anything. I want to predict time and location or either or.

this is some code i have for my priority prediction

from sklearn.ensemble import RandomForestClassifier
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
my_RandomForest = RandomForestClassifier(n_estimators=100, random_state=0)

my_RandomForest.fit(X_train, y_train)
y_predict_fr = my_RandomForest.predict(X_test)
from sklearn.metrics import accuracy_score
print(y_predict_fr)
accuracy_fr = accuracy_score(y_test, y_predict_fr)
print(accuracy_fr)

[4. 3. 2. ... 3. 1. 2.]
0.95100761598545

Also I would say think about the implications of your model (even if it is just some toy data).

If your model has overtly data from one location and not the other you might end up introducing some bias in your data - some crimes are not always known to the authorities. Crime is a very complex topics are your features relevent to the problem? Maybe some other features should be in the dataset and they are not?

Models applied in "high stake scenarios" often discussed on topics such as the fairness of AI should be very carefully validated. Once again that is just some toy data, but I thought it was noteworthy.


How does ML algorithms treat unseen data, a conceptual discussion I want to predict the occurrence of certain events, but these events only occur say 5% of the time in my data, hence in 95% of the data there is nothing to learn.

In order to teach the ML algo something I have learned to single out the 5% and drop the rest of the data. Let us say that I want to predict if a picture is of a dog or a cat. In my data 2.5% of the pictures are of dogs and 2.5% of cats, the rest are just random pictures. So, I single out the cat and dog pictures and label them so that the ML algo can learn from that. Am I broadly right so far?

So, if I train my algo on only cat and dog pictures and get a satisfactory accuracy, what will then happen in live usage when 95% of the pictures are not of cats or dogs? I.e. I show my model a picture of a house, what does it predict? Will my algo always predict either cat or dog, or will it somehow tell me that it has no clue what this picture is?

Any thoughts?

Define two flag variables: flag_is_cat and flag_is_dog, which take on values of 1 if the picture shows a cat or dog, respectively, and 0 otherwise. Define another flag that takes on the value of 1 if the picture contains either a cat or a dog. In a word, label the data.

If you train the model using all of the pictures, even those with neither a cat nor a dog, then the model outputs a probability that the picture contains a cat, a probability that it contains a dog, and another that it contains either. This is the approach mentioned by @marco_gorelli . Dividing the probability that the picture has a cat by the probability that it has either a cat or a dog gives the probability that the picture has a cat conditional on the picture having at least one of them.

Alternatively, if you train a model using only those pictures that contain either a cat or a dog, then the model would output the probability that a cat is contained in the picture and that a dog is contained in the picture conditional on at least one of them being in the picture.


Storing masks in jpg format I've created masks(numpy array with 0,1 as values) and tried exporting this array to jpg using matplolib but it's not exporting the values as it is. I'm getting a range of pixel values in resulting image. Is there any way in which I can add the array values to pixel values as it is. For example numpy array is 40X40 and resulting jpg image resolution is 40X40 pixels.

When you simply use say plt.savefig() to jpg, matplotlib by default does not save it to a binary image as you require. JPG by default the definition is values ranging from 0-255.
Suggest using PIL instead of matplotlib then you can save as BW binary image.


Pandas change value of a column based another column condition I have values in column1, I have columns in column2.
What I want to achieve: Condition: where column2 == 2 leave to be 2 if
 column1 < 30 elsif change to 3 if column1 > 90.

Here is what i did so far, the problem is 2 does not change to 3 where column1 > 90.

filter1 = data['column1']
for x in filter1:
    if x < 30:
        data['column2'] = data['column2'].replace([2], [2])
    else:
        data['column2'] = data['Output'].replace([2], [3])

What I want to achieve: Condition: where column2 == 2 leave to be 2 if column1 < 30 elsif change to 3 if column1 > 90

This can be simplified into where (column2 == 2 and column1 > 90) set column2 to 3. The column1 < 30 part is redundant, since the value of column2 is only going to change from 2 to 3 if column1 > 90.

In the code that you provide, you are using pandas function replace, which operates on the entire Series, as stated in the reference:

  Values of the Series are replaced with other values dynamically. This differs from updating with .loc or .iloc, which require you to specify a location to update with some value.

This means that for each iteration of for x in filter1 your code performs global replacement, which is not what you want to do - you want to update the specific row of column2 that corresponds to x from column1 (which you are iterating over).

  the problem is 2 does not change to 3 where column1 > 90

This is truly strange. I would expect the code you provided to have changed every instance of 2 in column2 to 3 as soon as it encountered an x >= 30, as dictated by your code conditional statement (the execution of the else branch). This discrepancy may stem from the fact that you are assigning to column2 the result of global replacement performed on the column Output (the contents of which are unknown). In any case, if you want your program to do something under a specific condition, such as x > 90, it should be explicitly stated in the code. You should also note that the statement data['column2'] = data['column2'].replace([2], [2]) achieves nothing, since 2 is being replaced with 2 and the same column is both the source and the destination.

What you could use to solve this particular task is a boolean mask (or the query method). Both are explained in an excellent manner in this question.

Using a boolean mask would be the easiest approach in your case:

mask = (data['column2'] == 2) & (data['column1'] > 90)
data['column2'][mask] = 3

The first line builds a Series of booleans (True/False) that indicate whether the supplied condition is satisfied.
The second line assigns the value 3 to those rows of column2 where the mask is True.


which classification model allows user to choose importance of data inputs? I am working on a match analytics project where I have to deal with the situation in which I am having some inputs like skills, experience, certifications etc. and my output is candidate selected Yes or No, my problem is, Is there any algorithm which will allow users to give weightage to one Input than the others? For example skills matters more for someone whereas for other user experience matters . I have tried neural network and Naive bayes algorithm but unable to choose importance of variable . Any help will be highly appreciable.

The answer is that  you don't need to . Say you decide to go with some classification model, for instance a simple decision tree, the model in question will implicitly learn which are the importances (or weights) of the features of your training data, and will use these features as main nodes of the tree.

To better understand this idea, lets take as an example the ID3 algorithm, which is one of the several existing algorithms to generate decision trees from a dataset. This algorithm will build a tree by iteratively setting as decision nodes those features that maximise the information gain at each step, or in other words, those that are the best predictors.

So it will implicitly be giving more importance to the attributes that are better predictors, hence there is no need to assign weights to the features of the dataset.

So my suggestion is that you try using some classifier from scikit-learn such as RandomForestClassifier for instance.


Time Series Classification using LSTM I have multiple .csv's files, each of them represent a product.
I am using LSTM to classify these products as good or bad.
All .csv's have been clubbed together in form of a 3d matrix of (#files, time_steps, #features).
I am passing the matrix in batch_size=128.

````below is the psuedo code.
model = Sequential()
model.add(LSTM(#cells(64), input_shape=(time_steps, #features)))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimiser='adam',
metrics=['accuracy'])
model.fit(train_X, train_y, batch_size=128, epochs=100, validation_split=0.1)

The problem is that I am getting NaN's in loss function and accuracy too is 0.000.

Can anyone suggest me where to look. Thanks in advance.

There might be many reasons for NAN's-

https://stats.stackexchange.com/questions/325451/cost-function-turning-into-nan-after-a-certain-number-of-iterations

https://stackoverflow.com/questions/33962226/common-causes-of-nans-during-training

basically LSTM is not able to train. try changing parameters esp activation function.

I haven't worked on python but what is the use of metrics=['accuracy']) in ur code, when u have already given cross entropy as loss function.


Best model for Machine Learning I am a beginner in the field of machine learning, I have small doubts for which I did not find any suitable answer. How to select the best model for prediction of unknown data. I have learned two ways I am not able to figure out which one is correct one.

Stating with, train your model by splitting the data into training and test data, then fit the model to predict the test output and error, shuffle the data and average the error over some 100 or more cycles. This will give us an average error (rmse for test) over 100 cycles (python way random state). Now to predict the unknown data (validate the model), which model should I consider the model for prediction.

1-Which have rmse closer to average rmse: pick a model which reports error approximately equal to the average error achieved over 100 cycles to predict the unknown data and called that as the model for prediction, or

2-Best performance: Chose a model out of 100 which is best (lowest rmse for test) for prediction of unknown data?

Another thing I am also struggling is that if considered the lowest error model as the model for unknown data prediction, by keeping X_train and y_train same of the best model.And If I chose my X_test and y_test from the same database (10% only) over 400 different times and predict the error, will it be an overfitted prediction?

Thanks in advance

If you use the test set for model selection (it does not matter whether you use some random state, or do hyperparameter tuning) then it is dirty and can no longer be used to predict the prediction performance.

The general workflow you should follow is this:

Split into train and test. Lock "test" in a safe.
Use cross-validation on the train set only to tune parameters.
Fix the hyperparameter to the best setting found. Write them in boldface as 'final parameters' into a stone plate on your wall.
Train a new classifier on the entire training set using exactly these parameters
Get the test set from the safe, and run the classifier on the test set exactly once to get a prediction of how well your classifier is going to work in the future
If the results are substantially worse than those from 2.: panic. It's not going to work.
Optional: If you are confident that hyperparameters do not depend on the data size but that you need every little bit of training data, you could train a classifier on train+test with the in-stone parameters. This is risky though, a you don't have a test set anymore.
Deploy, but monitor that it really performs as good as predicted.

The golden rule is: if you used the test set, you must not change the model anymore. You've got one shot to score.

That is why you split the train set again in CV.


Population Stability Index(PSI) Can we calculate PSI value if there is no training data?
if we can please suggest me the steps how  can we do it.

Yes. PSI uses actual and expected datasets. These can be defined however you want. Expected could be training, hold-out, last quarter, last week, artificial, etc. Think of expected as the benchmark that the current (actual) is being compared to. Defining that is up to you and the problem you are trying to solve.


confused about parameter updates and forward/backward pass according to batches and epochs in CNN? I am working on a CNN model, the code written in tensorflow, I did some googling about parameter updates such as weights ana biases when method is optimized and the loss is computed, two things made me confuse:

1- after output layer, the data goes to loss, the loss compute and then the model is begin optimization or in reverse of that?

2- Is parameters updated after each mini-batch fed to network (i.e. the forward and backward pass is done for every batch) or only updates when one epoch is completed? why some tutorials said that each epoch is ba forward/backward pass?

anyone can clarify it please? if with a reference its better for me.

I understand your confusion, and the real cause of all this mismatch between different tutorials is because there is many equivalent ways to train a neural network when dealing with batches, epochs. However, I think it is best to stick with the most common terminology, i.e. that used by the deep learning libraries.

Backpropagation

For question 1, you are correct. We feed the data through the inputs, the data goes through a forward pass and then we obtain an output. With the output we can calculate a loss. Then we will use backpropagation to attribute some fault to each model parameter for the resulting error (loss).  Then we will use gradient descent to update the model parameters accordingly. You can see how this works here.

Epochs and batches

For question 2. First let's define some terms. One epoch ends when all the training data available has been consumed. The second epoch goes through all the data again. In a simple neural network with not much data, you will pass all the training instances through the network successively and get the loss for each output. Then we will get an average of these losses to estimate the total loss for all instances. This results in one backpropagation per epoch.

However, most of the time it is not possible to fit all the data into memory so we must use batches, this means we will only feed-forward some training instances at a time. Then we will calculate the loss resulting from these instances and tune the parameters using backpropagation. Say we have 1000 training instances, then we can use a batch size of 100, we will thus do back-propagation 10 times per epoch.

Pros and cons

The pros of using batches is you can use larger datasets to train your model, however the smaller the batch size the less accurate the loss function estimate.


Evaluate prediction from multiple classification model Given that I have data containing images of oranges, apples and pineapples and I want to classify depending on a set of features.

Expected that I have completed the model and ready for prediction.

My questions are:

How can I output each score for every category the model predicted like this one?

Image 1:
     60% apple, 20% orange, 1% pineapple
     It implies that the image is an apple.

Image 2:
     40% apple, 60% orange, 10% pineapple
     It implies that the image is an orange.

Is there any libraries I can use with?

Does it depend on the model I am using? If yes, what are these models that implement this evaluation?

Based on the description of your question, it seems that you want the probability of outcome of each class (in multiclass-classification).

I would suggest you to use XGBoost to get output based on your requirement. By setting the value of objective parameter to multi:softprob, you can get probability of prediction of each and every class. If you set the value of objective parameter to multi:softmax, then you will only get the class with maximum probability among other classes.

Here, I am writing a example for your reference and to explain this description in a better way. You can get output by printing y_test_preds.

import xgboost as xgb

xgb_class = xgb.XGBClassifier(**params)
bst = xgb.train(params, dtrain, num_rounds)
y_test_preds = bst.predict(dtest)

By the following way, you can set the parameters for XGBoost. I would strongly suggest you to modify these parameters (except objective) based on your data and requirements.

params = {
     'objective' : 'multi:softprob',
     'max_depth' : 6,
     'silent' : 1,
     'eta' : 0.4,
     'num_class' : 3,
     'n_estimators' : 500,
     'learning_rate' : 0.1,
     'num_rounds' : 15
}

Note: In XGBoost, you have to use DMatrix instead of DataFrame. You can also get the DMatrix from DataFrame by this way.

dtrain = xgb.DMatrix(X_train.values, label = y_train.values)
dtest = xgb.DMatrix(X_test.values, label = y_test.values)

If you are new to XGBoost, then I would recommend you to go through this link once. https://xgboost.readthedocs.io/en/latest/get_started.html


Clarifying my understanding of on-policy RL (online SARSA) I want to clarify I have understood how SARSA works in nuances. Consider an original definition taken from ON-LINE Q-LEARNING USING CONNECTIONIST SYSTEMS. G. A. Rummery & M. Niranjan. CUED/F-INFENG/TR 166. September 1994 (which is the first publication where SARSA wss mentioned, according to a Wikipedia article).

The authors proposed an update rule which "... differs from normal Q-learning in the use of the Qt+1 associated with the action selected, rather than the greedy max(Qt+1 | a) used in Q-learning." (A citation from page 6.)

Please note that this is essential to on-policy nature of the algorithm. The term SARSA mentioned in the footnote for this definition.

And a later pseudo-code of SARSA update used in many readings:

begin

  initialize Q[S,A] arbitrarily
  observe current state s
  select action using a policy based on Q

repeat forever:

carry out an action a
observe reward r and state s'
select action a' using a poicy based on Q
Q[s,a] <- Q[s,a] + alpha * (r + gamma * Q[s', a'] - Q[s, a])
s <- s'
a <- a'

end-repeat

end

source: http://www.cse.unsw.edu.au/~cs9417ml/RL1/algorithms.html

I want to understand if I must use exactly the same Q function (and policy) to get A and A'. If I update Q function in each iteration, it follows that the next action in a subsequent iteration will be derived using the latest Q updated, while the previous action was obtained using the former Q.\
On the other hand, I really can make A and A' with exactly the same Q, and only after that update the Q. So I will always consider A and A' derived using the same function.

Which is more orthodox / correct?

I want to understand if I must use exactly the same Q function (and policy) to get A and A'. If I update Q function in each iteration, it follows that the next action in a subsequent iteration will be derived using the latest Q updated, while the previous action was obtained using the former Q.\ On the other hand, I really can make A and A' with exactly the same Q, and only after that update the Q. So I will always consider A and A' derived using the same function.
Which is more orthodox / correct?

The algorithm pseudocode given is more orthodox, since in order to revise the value of $A$ you would have to "roll back" the environment and see where the newly-sampled $A$ would take you from state $S$. To make this clearer, you can see that:
select action a' using a policy based on Q

could be re-phrased:
select action a' by sampling epsilon-greedy function over Q(s',*)

. . . you cannot do that unless you have the value of $S'$, and you may only have that value if you have already taken action $A$ when in state $S$. Changing $A$ at that stage therefore means going back in time . . .
In practice it doesn't matter much, even if you have capability to roll back (in a simulator, or in a planning algorithm). If your policy is based on e.g. $\epsilon$-greedy over the current Q values, then you are performing SARSA for optimal control (as opposed to prediction). In that case, changing Q means changing the policy. "On-policy" in SARSA for control must allow for the non-stationarity of the policy. Occasionally that means that the $A'$ value you just chose would have been chosen with a lower probability in a more optimal policy. But you chose it anyway this time, and the agent should choose it less often in future. The learning-rate based updates will remove estimation bias due to earlier poor/too-frequently-sampled choices over time.
Revising a single step "mistake" is possible, but not common practice in a purely online algorithm. I have not seen it in planning look-ahead or offline algorithms either that I have studied. I don't know for certain, but I suspect the occasional boost to learning you might get from revising the immediate part of the trajectory is too small to be worth the loss of generality of the algorithm. You may find it does help sometimes though, and worth an experiment to review whether it is helpful, provided you are working with a simulator/planner where rolling back state is relatively easy.

Note that the way you are thinking does turn up again when using function approximators (e.g. linear functions or neural networks) in semi-gradient versus "true gradient" methods, where instead of this being an issue with which Q values to use, it is an issue with calculating gradients due to the TD error, when your TD target is based on the same parameters that you are are taking the gradient for. In semi-gradient methods, this issue is ignored, and the methods still work OK. However, the "true gradient" methods are more theoretically correct.


Using Gini index, how to calculate the probability of correctly classifying a new randomly selected case to the highest probability class? I have the following binary Decision Tree:

Can you please explain how can I report this tree to a person who only understands probabilities?

If ca=1 and cp_4.0=1, what’s the probability of Yes HD?

While you can calculate the underlying class probabilities from the Gini index (for binary classification), it'll be more straightforward to calculate it from the "value" line in each box. This line simply represents the number of samples that belong in each node, split by the target variable, so you can use this count to calculate a probability.
In the bottom-most node, for example, you have value = [4, 55], indicating that among 59 samples which belong in that node (samples that have ca > 0.5 and cp_4.0 > 0.5), 4 are No HD and 55 are Yes HD. Therefore, the probability of Yes HD in that node is 55/59.
The Gini index is a measure of how "pure" a node is - as this number gets closer to 0, probability values will become more extreme (closer to 0 or 1), indicating that the decision tree is doing a better job of discriminating the target variable.


Use of Correlation score How do we use a correlation score between two variables for analysing data?

I have a set of 20 features and need to predict 21st feature. Now is it necessary that correlation between any two features should be close to 1 ? If I have 2 features with corr score close to -1, then does this mean that they are contradicting and thereby decreasing the accuracy ?

So how do we use a correlation score in analysis ?

Correlation should be as less as possible between different features, because correlated features mean that those features are giving out same kind of information/trend for the predictor to learn. Thus only one of them is actually useful for prediction.

Keeping more number of uninformative features (correlated features) would result in degraded accuracy if your sample size is similar to you feature set size. Feature selection using Recursive Feature elimination or PCA etc. can help you reduce your feature set to optimal size.

We calculate correlation score in predictive analysis between features and Target variable. When using linear regression to model a data set, we first see if the plot between different features and target variable values follow an upward (+ve correlation) or downward trend (-ve correlation) and not scattered randomly. If such a relationship exists then regression modelling on the data would work well.


Domain scoring based on ranking I am a computer science student working on a small information retrieval project. I have a dictionary with a domain as a key and it's ranking as value.

Based on that ranking, I need to score every domains. I was thinking to do 1/ranking but the disparity is too high. For example the first domain will have a score of 1 (1/1) and the domain ranked 10th will have a score of 0.1 which does not make sense for this.

I have 1000 domains in total and the last one should be close to 0.10 and the first one close to 1

You are essentially asking how Google calculates the score of the domains. The correspondence of the score with ranking is different for every search. Sometimes it is closer to x(i) = alpha*x(i-1) (exponential weights). Sometimes it could be linear x(i) = (n-i)*2/((n+1)*n). I've seen some papers that just use the number of the ranking for regression (linear), but this was not about internet domains, but some other topic.

I will try to answer the question about your weights and you weighting system:

You are using 1/1, 1/2, 1/3 ... 1/100 ... 1/1000

Instead you could use 1/1, 1/1.01, 1/1.02, 1/1.03 ... 1/2 ... 1/10

So your weights are $$weights = \frac{1}{1+ranking*0.01*\frac{9}{9.99}}$$
where ranking starts with 0 and ends with 999


how to make moving moving graphs that vary in time like a video? like this, like shown in this video.

You can find a list of great library here.
I would recommend this one but you have to write some css to have that same effect.


Neural network is not giving the expected output after training in Python My neural network is not giving the expected output after training in Python. Is there any error in the code? Is there any way to reduce the mean squared error (MSE)?

I tried to train (Run the program) the network repeatedly but it is not learning, instead it is giving the same MSE and output.

Here is the Data I used:

https://drive.google.com/open?id=1GLm87-5E_6YhUIPZ_CtQLV9F9wcGaTj2

Here is my code:

#load and evaluate a saved model
from numpy import loadtxt
from tensorflow.keras.models import load_model

# load model
model = load_model('ANNnew.h5')
# summarize model.
model.summary()
#Model starts
import numpy as np
import pandas as pd
from tensorflow.keras.layers import Dense, Activation
from tensorflow.keras.models import Sequential
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
# Importing the dataset
X = pd.read_excel(r"C:\filelocation\Data.xlsx","Sheet1").values
y = pd.read_excel(r"C:\filelocation\Data.xlsx","Sheet2").values

# Splitting the dataset into the Training set and Test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.08, random_state = 0)

# Feature Scaling
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

# Initialising the ANN
model = Sequential()

# Adding the input layer and the first hidden layer
model.add(Dense(32, activation = 'tanh', input_dim = 4))

# Adding the second hidden layer
model.add(Dense(units = 18, activation = 'tanh'))

# Adding the third hidden layer
model.add(Dense(units = 32, activation = 'tanh'))

#model.add(Dense(1))
model.add(Dense(units = 1))

# Compiling the ANN
model.compile(optimizer = 'adam', loss = 'mean_squared_error')

# Fitting the ANN to the Training set
model.fit(X_train, y_train, batch_size = 100, epochs = 1000)

y_pred = model.predict(X_test)
for i in range(5):
    print('%s => %d (expected %s)' % (X[i].tolist(), y_pred[i], y[i].tolist()))

plt.plot(y_test, color = 'red', label = 'Test data')
plt.plot(y_pred, color = 'blue', label = 'Predicted data')
plt.title('Prediction')
plt.legend()
plt.show()

# save model and architecture to single file
model.save("ANNnew.h5")
print("Saved model to disk")

It looks like your loss is improving (the loss is decreasing every epoch), so I wouldn't say the network isn't training. Try looking at a training plot (which would have epoch on the x-axis and loss on the y-axis). See if there are any jumps or strange patterns in the training. If you don't see a roughly decreasing training loss, then you have a problem.

It's also possible that this model is just completely overfitting your data, leading to poor prediction on the test set. How many inputs do you have? Why are you using three hidden layers? Try reducing the size of your model and see if your prediction improves.


How to make my Neural Netwok run on GPU instead of CPU I have installed Anaconda3 and have installed latest versions of Keras and Tensorflow.

Running this command :

from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())

I find the Notebook is running in CPU:

[name: "/device:CPU:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 2888773992010593937
]

This is my Nvidia version:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:04_Central_Daylight_Time_2018
Cuda compilation tools, release 10.0, V10.0.130

running nvidia-smi, I'm getting this result:

I want to make the neural network to train on GPU. Please help me in switching from CPU to GPU.

First some stupid sanity-check questions: do you have a GPU in your local machine? (you didn't mention that explicitly). I ask because it will not work e.g. on an integrated Intel graphics card found in some laptops.

Second, you installed Keras and Tensorflow, but did you install the GPU version of Tensorflow? Using Anaconda, this would be done with the command:

conda install -c anaconda tensorflow-gpu

Other useful things to know:

what operating system are you using? (I assume Linux e.g. Ubuntu)
what GPU do you expect to be shown as available?
Can you run the command: nvidia-smi in a terminal and update your question with the output?

If you have installed the correct package (the above method is one of a few possible ways of doing it), and if you have an Nvidia-GPU available, Tensorflow would usually by default reserve all available memory of the GPU as soon as it starts building the static graph.

If you were not already, it is probably a good idea to use a conda environment, which keeps the requirements of your model separate from whatever your system might already have installed. Have a look at this nice little walkthrough on getting started - this will likely be a good test to see if your system is able to run models on a GPU as it removes all other possible problems created components unrelated to your script. In short, create and activate a new envrinment that includes the GPU version of Tensowflow like this:

conda create --name gpu_test tensorflow-gpu    # creates the env and installs tf
conda activate gpu_test                        # activate the env
python test_gpu_script.py                      # run the script given below

UPDATE

I would suggest running a small script to execute a few operations in Tensorflow on a CPU and on a GPU. This will rule out the problem that you might not have enough memory for the RNN you're trying to train.

I made a script to do that, so it tests the same operations on a CPU and GPU and prints a summary. You should be able to just copy-paste the code and run it:

import numpy as np
import tensorflow as tf
from datetime import datetime

# Choose which device you want to test on: either 'cpu' or 'gpu'
devices = ['cpu', 'gpu']

# Choose size of the matrix to be used.
# Make it bigger to see bigger benefits of parallel computation
shapes = [(50, 50), (100, 100), (500, 500), (1000, 1000)]

def compute_operations(device, shape):
    """Run a simple set of operations on a matrix of given shape on given device

    Parameters
    ----------
    device : the type of device to use, either 'cpu' or 'gpu'
    shape : a tuple for the shape of a 2d tensor, e.g. (10, 10)

    Returns
    -------
    out : results of the operations as the time taken
    """

    # Define operations to be computed on selected device
    with tf.device(device):
        random_matrix = tf.random_uniform(shape=shape, minval=0, maxval=1)
        dot_operation = tf.matmul(random_matrix, tf.transpose(random_matrix))
        sum_operation = tf.reduce_sum(dot_operation)

    # Time the actual runtime of the operations
    start_time = datetime.now()
    with tf.Session(config=tf.ConfigProto(log_device_placement=True)) as session:
            result = session.run(sum_operation)
    elapsed_time = datetime.now() - start_time

    return result, elapsed_time

if __name__ == '__main__':

    # Run the computations and print summary of each run
    for device in devices:
        print("--" * 20)

        for shape in shapes:
            _, time_taken = compute_operations(device, shape)

            # Print the result and also the time taken on the selected device
            print("Input shape:", shape, "using Device:", device, "took: {:.2f}".format(time_taken.seconds + time_taken.microseconds/1e6))
            #print("Computation on shape:", shape, "using Device:", device, "took:")

    print("--" * 20)

Results from running on CPU:

Computation on shape: (50, 50),       using Device: 'cpu' took: 0.04s
Computation on shape: (500, 500),     using Device: 'cpu' took: 0.05s
Computation on shape: (1000, 1000),   using Device: 'cpu' took: 0.09s
Computation on shape: (10000, 10000), using Device: 'cpu' took: 32.81s

Results from running on GPU:

Computation on shape: (50, 50),       using Device: 'gpu' took: 0.03s
Computation on shape: (500, 500),     using Device: 'gpu' took: 0.04s
Computation on shape: (1000, 1000),   using Device: 'gpu' took: 0.04s
Computation on shape: (10000, 10000), using Device: 'gpu' took: 0.05s


EM parameter estimation for conditional Gaussians Let $$X_1\sim N(\mu_{X_1},\sigma_{X_2}^2)$$ $$X_2\sim N(\mu_{X_2},
\sigma_{X_2}^2)$$
where $\mu_{X_2}=c+aX_1$. Also, I have data $D$ (with missing values on $X_1,X_2$).

How can I update/estimate the parameters $\mu_{X_1},\sigma_{X_1},\mu_{X_2},a,c,\sigma_{X_2}$ using EM?i.e. what is the formula for updating $\sigma_{X_2}$?

My model is a conditional Gaussian, which is a conditional form of a bivariate Gaussian $(X_1,X_2)$ with mean vector $(\mu_1,\mu_2)^\top$ and covariance matrix
$$\left( \begin{matrix}
   \Sigma _{11} & \Sigma _{12}  \\
   \Sigma _{21} & \Sigma _{22}  \\
\end{matrix} \right)$$

Here is a reference to convert bivariate Gaussian to conditional Gaussian: $$\mu_{2|1}=\mu_{2}+\Sigma_{21}\Sigma_{11}^{-1}(X_1-\mu_1)\quad,\quad \Sigma_{22|1}=\Sigma_{22}-\Sigma_{21}\Sigma_{11}^{-1}\Sigma_{12}$$ which yields my model.

It seems that when $X_1$ has different observations and $X_2$ is unobserved, the variance of $X_2$ remains unchanged.So how to update the $\sigma_{x_2}$?

Initial setting for the model
$$X_1\sim N(5,7)\quad,\quad X_2\sim N(0.5X_1,8)$$

Data:

$$X_1:\operatorname{9\,\,4\,\,NA}$$

$$\quad X_2:\operatorname{NA\,\,NA\,\,3}$$

When $X_1$ is unobserved, at iteration $k=1$ of EM, the posterior mean value (when $X_2=3$) is $5.18$ by using an inference algorithm, i.e. Junction tree/kalman filter.
Then the sufficient statistics for $X_1$ is:

$s_1=\Sigma_{i=1}^nx_{i1}=9+4+5.18$ and $s_{11}=\Sigma_{i=1}^nx_{i1}^2=9^2+4^2+(5.18^2+\sigma_{11.2})$

where $\sigma_{11.2}$ is the posterior conditional variance of $X_1$ given $X_2$, when $X_1$ is unobserved.

$s_2=\Sigma_{i=1}^nx_{i2}$ and $s_{22}=\Sigma_{i=1}^nx_{i2}^2$. When $X_2$ is unobserved a term $\sigma_{22.1}$ is also added to $x_{i2}^2$.

So parameters $\mu_{X_1}$ and $\sigma_{X_1}$ are obtained. Similarly, posterior mean of unobserved $X_2$ is also obtained by using i.e. Junction tree inference.

The sufficient statistics for $s_{12}=\Sigma_{i=1}^nx_{i1}x_{i2}$, and $s_{21}=\Sigma_{i=1}^nx_{i2}x_{i1}$.

Mean vector is then: $\mu^{(k)}=(s_1/n, s_2/n)$.

Covariance matrix is: $\Sigma^{(k)}=$$\left( \begin{matrix}
   \Sigma _{11} & \Sigma _{12}  \\
   \Sigma _{21} & \Sigma _{22}  \\
\end{matrix} \right)$$=$$\left( \begin{matrix}
   s_{11}/n-(s_1/n)^2 & s_{21}/n-(s_2/n)(s_1/n)  \\
   s_{12}/n-(s_1/n)(s_2/n) & s_{22}/n-(s_2/n)^2  \\
\end{matrix} \right)$$
 $

Other parameters in the question are obtained. Conditional mean and conditional variance of $X_2|X_1$ is calculated by the formula in the reference of the question.


K-Means clustering for mixed numeric and categorical data My data set contains a number of numeric attributes and one categorical.
Say, NumericAttr1, NumericAttr2, ..., NumericAttrN, CategoricalAttr,
where CategoricalAttr takes one of three possible values: CategoricalAttrValue1, CategoricalAttrValue2 or CategoricalAttrValue3.
I'm using default k-means clustering algorithm implementation for Octave.
It works with numeric data only.
So my question: is it correct to split the categorical attribute CategoricalAttr into three numeric (binary) variables, like IsCategoricalAttrValue1, IsCategoricalAttrValue2, IsCategoricalAttrValue3 ?

The standard k-means algorithm isn't directly applicable to categorical data, for various reasons.  The sample space for categorical data is discrete, and doesn't have a natural origin.  A Euclidean distance function on such a space isn't really meaningful.  As someone put it, "The fact a snake possesses neither wheels nor legs allows us to say nothing about the relative value of wheels and legs." (from here)

There's a variation of k-means known as k-modes, introduced in this paper by Zhexue Huang, which is suitable for categorical data.   Note that the solutions you get are sensitive to initial conditions, as discussed here (PDF), for instance.

Huang's paper (linked above) also has a section on "k-prototypes" which applies to data with a mix of categorical and numeric features.  It uses a distance measure which mixes the Hamming distance for categorical features and the Euclidean distance for numeric features.

A Google search for "k-means mix of categorical data" turns up quite a few more recent papers on various algorithms for k-means-like clustering with a mix of categorical and numeric data.  (I haven't yet read them, so I can't comment on their merits.)

Actually, what you suggest (converting categorical attributes to binary values, and then doing k-means as if these were numeric values) is another approach that has been tried before (predating k-modes).  (See Ralambondrainy, H. 1995. A conceptual version of the k-means algorithm. Pattern Recognition Letters, 16:1147–1157.)  But I believe the k-modes approach is preferred for the reasons I indicated above.


Techniques for trend extraction from unbalanced panel data My data set is formatted like this:

User-id | Threat_score
aaa       45
bbb       32
ccc       20

The list contains the top 100 users with the highest threat scores. I generate such a list monthly and store each month's list in its own file.

There are three things I would like to get from this data:

1. Users who are consistently showing up in this list

2. Users who are consistently showing up in this list with high threat scores

3. Users whose threat scores are increasing very quickly

I am thinking a visual summary would be nice; each month (somehow) decide which users I want to plot on a graph of historic threat scores.

Are there any known visualization techniques that deal with similar requirements?

How should I be transforming my current data to achieve what I am looking for?

I would add a third column called month and then concatenate each list. So if you have a top 100 list for 5 months you will create one big table with 500 entries:

User-id | Threat_score | month
aaa       45             1
bbb       32             1
ccc       20             1
...       ...            ...
bbb       64             2
ccc       29             2
...       ...            ...

Then, to answer your first question, you could simply count the occurrences of each user-id. For example, if user bbb is in your concatenated table five times, then you know that person made your list all five months.

To answer you second question, you could do a group by operation to compute some aggregate function of the users. A group by operation with an average function is a little crude and sensitive to outliers, but it would probably get you close to what you are looking for.

One possibility for the third question is to compute the difference in threat score between month n-1 and month n. That is, for each month (not including the first month) you subtract the user's previous threat score from the current threat score. You can make this a new column so your table would now look like:

User-id | Threat_score | month | difference
aaa       45             1       null
bbb       32             1       null
ccc       20             1       null
...       ...            ...     ...
bbb       64             2       32
ccc       29             2       9
...       ...            ...

With this table, you could again do a group by operation to find people who consistently have a higher threat score than the previous month or you could simply find people with a large difference between the current month and the previous month.

As you suggest, visualizing this data is a really good idea. If you care about these threat scores over time (which I think you do), I strongly recommend a simple line chart, with month on the x-axis and threat score on the y-axis. It's not fancy, but it's extremely easy to interpret and should give you useful information about the trends.

Most of this stuff (not the visualization) can be done in SQL and all of it can be done in R or Python (and many other languages). Good luck!


How to add a CNN layer on top of BERT? I am just playing with bert (Bidirectional Encoder Representation from Transformer)
Research Paper

Suppose I want to add any other model or layers like Convolutional Neural Network layers (CNN), Non Linear (NL) layers on top of BERT model. How can I do this?

I am not able to figure out where should I change in code of BERT. I am using the pytorch implementation of bert
from huggingface.

This is what I want to do:

Please show steps to implement this using sudo code which will help me in implemention of cnn on top of BERT.

You could use HuggingFace's BertModel (transformers) as the base layer for your model and just like how you would build a neural network in Pytorch, you can build on top of it. HuggingFace's other BertModels are built in the same way. For reference you can take a look at their TokenClassification code over here. This hasn't been mentioned in the documentation much and I think it should.

For Tensorflow however, you would have convert the Bert Model into a Keras layer. I haven't really gone through it much or tried it, but I think this blog post does a pretty good job of explaining it.


anomaly detection alert system I need to design and implement my 3rd year project which is a real-time alert system for parents to keep their kids safe with their smartphones.

My data is time-based and unsupervised, which means I don't have score for any events, it contains events from :

mobile browsing ( http requests )
gps coordinates
calls ( outgoing & incoming )

Should I choose cluster based / density based approach ?
How do I find rare & suspicious events that parents should be notified for with the help of machine learning ?

Any suggestions on how to present the results visually ?

Is a week of data enough for training ?

Are there any good open sources in Java to detect anomaly ?

Should I choose cluster based / density based approach?

Yes, but depends on how good the algorithm is, and how effectively is the algorithm identifying important clusters. There is a lot of literature where people have taken the clustering approach.

If I were to go with the clustering approach, I would go with the hierarchical clustering technique, cause it would help me re-check my clusters and build hierarchies in them so that the seriousness of the fraud can also be explored.

  How do I find rare & suspicious events that parents should be notified
  for with the help of machine learning ?

I would point out an answer which I have written for a very similar question, which also explores the aspect of fraud detection and anomaly analysis. And as I have said in the answer, the genetic algorithm still goes strong when it comes to fraud detection, owing to it's neat method of fitness calculation of the data points.

  Any suggestions on how to present the results visually ?

Generally such charts are presented through pie charts and network graphs. you can also use infographics if you have a nice idea about design and aesthetics.

  Is a week of data enough for training ?

Generally, data science teams at Paypal, Palantir, etc churn years of data for sensible insights. As you say it's a class project, data of several months should do good. You wouldn't really get anything suspicious or sensible from a week of data.

  Are there any good open sources in Java to detect anomaly ?

Yes, there are some:

Genetic algorithms library
A Distributed Online Machine Learning Framework in java


Text comparison using word embeddings Currently I have a list of Books which I need to compare to a different list.
For example in my local list I have:

 - Harry Potter and the Philosopher's Stone | J.K. Rowling
 - Harry Potter and the Deathly Hallows | J.K. Rowling
 - Harry Potter and the Goblet of Fire | J.K. Rowling
 - Harry Potter and the Prisoner of Azkaban | J.K. Rowling

When I compare I end up having a simple operation:

Book Title + Author (local list) vs [Book Title + Author] (remote list)

I'm currently using Levenshtein distance, based on that I can find out if Book is in remote system or not.

In this case if Book1 vs Book2 result score exceeds X, I assume is the same Book.

levenshtein(Book1 Title + Author, Book2 Title + Author)

I read about Word Embeddings and before I start coding want to check if converting my text information to embeddings may improve my results.

You do not need to use Word Embedding to solve this problem.

You are performing set membership check. In this problem, each book title and author combination is an element. You can create a hash for each element and perform constant time lookup. This assumes you have consistent normalization for each data source.


Passing a custom kernel with more than two arguments into `svm.SVC` in scikit-learn I'm trying to use a custom kernel that accepts 3 arguments, with the SVM in sk-learn:

def k_gaussian(_x1, _x2, _sigma):
    normsq = np.square(np.linalg.norm(_x1-_x2))
    return np.exp(- normsq/(2 * np.square(sigma)))

According to the documentation, a custom kernel must only have two arguments, which the svm.SVC class will handle automatically with the given input data. We are told to pass the custom kernel in a form like:

clf = svm.SVC(kernel=my_kernel)

However, I'm working on an assignment which requires us to run experiments on SVM performance with varying values of _sigma.

How can I achieve that in this case? Can I pass in something like?:

clf = svm.SVC(kernel=k_gaussian(_sigma=2))

Would things like decorators help me here?

That can be done with a closure like:

Code:

def build_k_gaussian(sigma):

    def k_gaussian(_x1, _x2):
        diff = _x1[:, np.newaxis] - _x2
        normsq = np.square(np.linalg.norm(diff, axis = 2))
        return np.exp(- normsq / (2 * np.square(sigma)))

    return k_gaussian

clf = svm.SVC(kernel=build_k_gaussian(sigma=2))

How does this work?

The function k_gaussian is defined when build_k_gaussian() is called.  k_gaussian will be able to access the value of sigma from when the function was created.  This is known as a closure.

So in the end, build_k_gaussian returns a function that takes two parameters, which is what the kernel parameter required.

According to the Using Python functions as kernels on scikit-learn:

  Your kernel must take as arguments two matrices of shape (n_samples_1, n_features), (n_samples_2, n_features) and return a kernel matrix of shape (n_samples_1, n_samples_2).

So you need to apply the kernel function on all pairs of samples, therefore
diff broadcasts the _x1 matrix and subtracts all samples in _x2 from all samples in _x1. You have to calculate the norm along axis=2.


CNN unbalanced and small dataset I would like to use CNN to make classification with 5 classes, but 4 of these classes only have between 16 and 60 images, while the last one has more than 1300. I know 16 or 60 images are not enough, so I want to use transfer learning, fine tuning and data augmentation. However, I have several questions.
As the data augmentation must only be used for the training data, I will have very few images from the 4 classes for the validation set, would it be a problem?
Is it needed to split it in training/validation/test, or is training/validation enough?
Another problem is the unbalanced data: with such a difference between the number of images in each class, would oversampling or undersampling be a good solution?
For transfer learning and fine tuning, should I freeze all the convolutional layers or should I only train a FC layer?

As the data augmentation must only be used for the training data, I will have very few images from the 4 classes for the validation set, would it be a problem?

You may first perform data augmentation on the image dataset and then split it into train/test or train/test/validation. Just make sure that the images present in the test split are not included in the training split or vice-versa. See this.

Is it needed to split it in training/validation/test, or is training/validation enough ?

The validation dataset is particularly to fine-tune the model's hyperparameters i.e for hyperparameter optimization. If you want to simply skip that procedure, just make two splits, train, and test. See this.
See this answer to know the difference between test and validation dataset.

Another problem is the unbalanced data with such a difference
between the number of images in each class, would oversampling or
undersampling be a good solution ?

I would encourage you to augment the images for a specific class until you are left with a near-balanced dataset.

For transfer learning and fine tuning, should I freeze all the convolutional layers or should I only train a FC layer?

As a common practice, you may try unfreezing some initial layers ( layers that usually have lesser filters ). Refer to this blog. Deeper layers have more filters and extract high-level features.


Impulse Response Function - Negative Shocks on R I have two questions on how to produce impulse responses using R

(1) Impulse responses to a negative shock in the independent variable (money supply)

(2) Impulse responses at 2 standard deviations

The code I used to generate the impulse responses to a positive shock at 1 standard deviation is the following:

m1 <- read.csv("m1.csv", header=T)

m1

varm1 <- VAR(m1, p=8, type="cons")

irfm1 <- irf(varm1, impulse="m1", response= c("gdp"), boot = FALSE)

plot(irfm1)

irfm1

Here is a simple example that should work:

library(vars)
data("Canada")
var.2c=VAR(Canada,p=2,type="const")
# 1 SD impulse reponse function
irf.rw.e=irf(var.2c,impulse="rw",response=c("e"))
# gamma is the number of standard deviations for the irf
gamma=-0.25
irf.rw.e_gamma = irf.rw.e
n=length(irf.rw.e_gamma$irf$rw)
for(i in 1:n){
    irf.rw.e_gamma$irf$rw[i]   = irf.rw.e_gamma$irf$rw[i]*gamma
    irf.rw.e_gamma$Lower$rw[i] = irf.rw.e_gamma$Lower$rw[i]*gamma
    irf.rw.e_gamma$Upper$rw[i] = irf.rw.e_gamma$Upper$rw[i]*gamma
}
plot(irf.rw.e)
plot(irf.rw.e_gamma)

Source


Update machine learning model while retaining the prediction results exact for old data New to ML here. In our industry, we are looking for a type ML method/model that can be updated to accommodate new data points while keeping the prediction value of the historical data exactly the same. Of course a simple hash table would do, but are there other better alternatives.
First question is what is this type of method/model called in the ML world. Where could I find some good examples/explanations of such techniques?
Thanks

The safest way to keep the same predictions for your older data is versioning your trained models. It is, keeping the generated model artifact (in python could be pickle file, h5 file, etc) to make sure that you can use it getting the same results as you say, and generating new models (so new artifacts) via retraining when new data come in.
The usual approach is deciding an evaluation metric for your ML model, and based on it, when new data come in, you retrain your models (the frequency for this is something to decide on your team) and when the metric value is better (enough) than the latest model version, you can promote this new model for use in production. Theere are quite goos frameworks for models versioning, as MLflow.
Keep in mind that transfer learning does not guarantee predictions on older data are going to be same, since a new model is trained based on another model, but a different model is eventually created.


How to think about prediction error that is not convex in hyperparameter, or over the course of training Take the following case of a hyperparameter and prediction error:

Imagine that the hyperparameter is a L2 penalty or a dropout rate -- something that we think that should have a single sweet spot -- too high and you're underfit and too low and you're overfit.

I keep getting nonconvex plots like the one above when doing cross-validation.

I guess this just points to a lot of noise during training -- I've got a lot of variables for a modest sample size, and I need to regularize heavily to get a good model.  But still I'm a little bit unsure whether this sort of thing might point to a bug in some aspect of my implementation.

Has anyone come across this sort of thing before?  And did you just shrug off the nonconvexity and go with the model that minimized the prediction error?

If so, that begs the question: why not just compute a prediction error at each update during training, saving any set of weights that minimizes prediction error -- even if the model is nowhere near converged.  Basically letting the noise work in your favor.  This seems appealing, because sometimes I get really low prediction errors early on, only to have them evaporate as the loss function declines.

This seems horribly unprincipled, but I ask myself "why should I care if it is"?  And "is it unprincipled anyway?"

The underlying true performance is likely convex or at least likely only has one minimum, but you don't know the true underlying performance, you only get a stochastic sample with a tiny sample size because you apply cross validation. The performance is a random variable where you are interested in the expected value, but you only get k samples where k is the number of folds. I think if you would increas k your curve would look more smooth. That said, even if your hypothesis counts in general, for this specific set there might be some weird data points that make random settings perform worse than expected still.

The problem with your suggested approach is that you are treating this noise as signal. If you have a normally distributed random variable, you sample 1000 times from it and take the max value, this will be much higher than the true expected value. An alternative explanation could be that it is not noise, but that you are actually overfitting. Then your approach makes more sense, and if the samples surrounding it during training are also relatively low then this is actually a good strategy, it is called early stopping, a different form of regularization and commonly used in Neural Networks.

EDIT: I think you are right that I didn't explain that analogy very well, we are trying to optimize a function g(x) where x is our hyperparameters and g(x) is a random variable function, let's say it had a true underlying value f(x) and additive noise $\epsilon$. If $\epsilon$ is Gaussian distributed with a high variance compared to the underlying value, if we take the max without looking at anything else we could take one that had lucky noise. I think what we are seeing in your graph is relatively high variance due to low sample size of your data (which means small validation sets in your folds) whih will lead to non-convex behaviour. That said, your explanation of early on seeing lower validation losses does imply overfitting which early stopping can help with.


How much training data does Word2Vec need? I'd like to compare the difference among the same word mentioned in different sources. That is, how authors differ in their usage of ill-defined words, such as "democracy".

A brief plan was

Take the books mentioning the term "democracy" as plain text
In each book, replace democracy with democracy_%AuthorName%
Train a word2vec model on these books
Calculate the distance between democracy_AuthorA, democracy_AuthorB, and other relabeled mentions of "democracy"

So each author's "democracy" gets its own vector, which is used for comparison.

But it seems that word2vec requires much more than several books (each relabeled word occurs only in a subset of books) to train reliable vectors. The official page recommends datasets including billions of words.

I just wanted to ask how large should be the subset of one author's books to make such inference with word2vec or alternative tools, if available?

It sounds like Doc2Vec (or paragraph/context vectors) might be the right fit for this problem.
In a nutshell, in addition to the word vectors, you add a "context vector" (in your case, an embedding for the author) that is used to predict the center or context words.
This means that you would benefit from all the data about "democracy" but also extract an embedding for that author, which combined should allow you to analyze the bias of each author with limited data about each author.
You can use gensim's implementation. The doc includes links to the source papers.


How do SOMs make visualization of high-dimension data possible? Self Organizing Map is credited to be a very effective tool for exploratory data analysis as it aids in visualization of high-dimensional data.

However, the part where the dimensionality reduction is happening is not clear to me. From this article, I understand that the information is essentially embedded in the weight vectors of neurons which are of the same dimensions as of the original patterns. The same weight vectors are supposed to be used for visualization (if I follow from the highly popular colors example).

If this is true, I do not understand how SOM is performing dimensionality reduction and how it is making visualization of high-dimensional data simpler.

Could someone explain (if possible with an example of >3 dimensional data) how SOM performs dimension reduction and makes visualization possible?

Consider a data set of 1000 patterns, each with 100 features. It is impossible to plot this data. When this data is run through SOM, the network learns the weights such that each of the neurons summarizes a subset of the vectors. After the training is complete, similar patterns get mapped to neighborhood neurons and farther apart neurons represent dissimilar patterns. Visualizing this mapping tells which patterns lie where, in a 2D space.

For example, in the world poverty map[1][2], each pattern is a vector of length 39 and represents one country. When this data set is run through SOM, the countries(patterns) are mapped to nearest neurons and it can be seen that the countries are arranged together based on their economic wellness.

After SOM is run until the stopping criterion is met, each pattern/input vector is mapped to the best matching unit. Hence, for example, Belgium (BEL) is closest to the weight vector of the first node in the map (top-right corner) and hence, mapped to it.

Each node is colored based on the average distance of its weight vector with the weight vectors of its neighbors (U-matrix).

However, it is still not clear how dimensionality reduction is achieved.


CUDA 8.0 is compatible with my GeForce GTX 670M Wikipedia says, but TensorFlow rises an error: GTX 670M's Compute Capability is < 3.0 According to Wikipedia, the GeForce GTX 670M has a Compute Capability of 2.1 (and a Fermi micro-architecture), which is confirmed by TensorFlow (I can read "2.1" in the error it rises).
Wikipedia says that CUDA 8.0 supports compute capabilities from 2.0 to 5.x (Fermi micro-architecture included). It even says that it's the "last version with support for compute capability 2.x (Fermi)". However, the error rised by TensorFlow says that my being-used CUDA version support at least compute capability of... 3.0... And thus my GeForce GTX 670M is ignored and my CPU is used to compute :-( . It's a big problem.
Since several CUDA versions are installed in my computer, I wanted to be sure it was CUDA 8.0 that was being used. So I typed, in the PyCharm's terminal: nvcc --version which outputs:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Mon_Jan__9_17:32:33_CST_2017
Cuda compilation tools, release 8.0, V8.0.60

Thus, CUDA 8.0 is actually being used.
The error is the following:

2019-08-01 22:04:28.366003: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX
2019-08-01 22:04:28.561338: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1030] Found device 0 with properties:
name: GeForce GTX 670M major: 2 minor: 1 memoryClockRate(GHz): 1.24
pciBusID: 0000:01:00.0
totalMemory: 3.00GiB freeMemory: 2.48GiB
2019-08-01 22:04:28.561814: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1093] Ignoring visible gpu device (device: 0, name: GeForce GTX 670M, pci bus id: 0000:01:00.0, compute capability: 2.1) with Cuda compute capability 2.1. The minimum required Cuda capability is 3.0.

My setup
I've described the problem. Now I'm going to give you more informations about my setup.

I'm working on Windows 10 with PyCharm and Keras (so TensorFlow)
Since I thought I had to use CUDA 8.0, I checked in the TensorFlow Website which version I should use. I've found that: for CUDA 8.0, I should use TensorFlow GPU version 1.4.0 with CUDNN version 6.
In reality, I'm using: CUDA V8.0.60, CUDNN V6.0 (found on the CUDNN Website for CUDA 8.0) and TensorFlow-GPU V1.4.0
Here is a screenshot of my Windows 10's %PATH%'s value. Note that the being-used CUDA version ("CUDA_PATH") is actually 8.0, as you can see it on the screenshot.

Final question
Could you please tell me what version of CUDA I should use for being able to use my GeForce GTX 670M as a compute unit for my networks training? Wikipedia seems to be wrong...

CUDNN and Tensorflow require a GPU which has a compute capacity of 3.0 at least: not only the CUDA version must take account of this CC, but also these both programs.

Indeed: https://stackoverflow.com/questions/38542763/how-can-i-make-tensorflow-run-on-a-gpu-with-capability-2-0/38543201#38543201


Area under the ROC curve approximation I wanted to compute the Area under the ROC curve for a logistic regression model in the context of binary classification. For that I computed, for a list of thresholds, say 0.1 0.2 ... 0.9, the TPR and FPR. I have thus
 (TPR_threshold1, ... TPR_thresholdN)
 (FPR_threshold1, ... FPR_thresholdN)

Can I just do
TPR_threshold1*FPR_threshold1 + ... + TPR_thresholdN*FPR_thresholdN

in order to compute the area under the roc curve or do I need some more elaborate mathematical modeling?
Thanks a lot!

The thresholds don't matter; what matters are the (FPR, TRP) values at those thresholds, as they are points on the curve. Sort them by FPR ascending. For this to work out, you'll want to include the points (0,0) and (1,1) in your list, corresponding to thresholds 1 and 0.
You can use a trapezoidal approximation, as each successive pair of points defines a trapezoid of area under the curve. You'll just add up the areas.
Let's take two successive points (FPR1, TPR1) and (FPR2, TPR2). The area is (FPR2 - FPR1) * (TPR1 + TPR2) / 2. Just sum that over all successive pairs of points.
Of course, most libraries can compute this for you from this input, like scikit-learn.


R langauge how to create xgb.DMatrix object from data frame (newbe) In R, how does one create an xgb.DMatrix object from an R data frame?

Welcome to the site!

Assume that y is your response, and x is your data set of predictors (where categorical variables have been appropriately converted to numeric). Your data does not necessarily need to be sparse, although sparse data will improve computation speed.

Then dtrain <- xgb.DMatrix(label = y, data = as.matrix(x)).

As you get more proficient with XGBoost you can start exploring the weight and base_margin parameters.


How to optimize XGBoost performance accuracy? I have dataset to predict customers dropout(yes,no), with 5 numerical features and 2 categorical features. I have applied a scaler to the numerical data and transformed the categorical features into dummies variables, creating 29 features. My dataset has shape of 6552 rows and 34 features. What is the recommend approach to tune the parameters of XGBClassifier, since I created the model using default values, i.e., model=XGBClassifier()? Should I use a brute-force looping the values in some parameters until I find a optimal prediction value? In this case what is recommended?

There are three main techniques to tune up hyperparameters of any ML model, included XGBoost:

1) Grid search: you let your model run with different sets of hyperparameter, and select the best one between them. Packages like SKlearn have routines already implemented. But also in this case you have to pre-select the nodes of your grid search, i.e. which values have to be tried by the routine

2) Random search: similar to Grid Search, but you basically only choose the parameters boundaries, and the routine randomly try different sets of hyperparameters.

more informations about method 1 and 2 are here.

3) Bayesian optimization algorithms; this is the way I prefer. Basically this algorithms guesses the next set hyperparameter to try based on the results of the trials it already executed. An easy to use and powerful is SMAC.


Pytorch : Loss function for binary classification Fairly newbie to Pytorch & neural nets world.Below is a code snippet from a binary classification being done using a simple 3 layer network :

n_input_dim = X_train.shape[1]
n_hidden = 100  # Number of hidden nodes
n_output = 1   # Number of output nodes = for binary classifier
# Build the network
model = nn.Sequential(
    nn.Linear(n_input_dim, n_hidden),
    nn.ELU(),
    nn.Linear(n_hidden, n_output),
    nn.Sigmoid())

x_tensor =  torch.from_numpy(X_train.values).float()
tensor([[ -1.0000,  -1.0000,  -1.0000,  ..., -99.0000, -99.0000, -99.0000],
       [ -1.0000,  -1.0000,  -1.0000,  ...,   0.1538,   5.0000,   0.1538],
       [ -1.0000,  -1.0000,  -1.0000,  ..., -99.0000,   6.0000,   0.2381],
       ...,
       [ -1.0000,  -1.0000,  -1.0000,  ..., -99.0000, -99.0000, -99.0000],
       [ -1.0000,  -1.0000,  -1.0000,  ..., -99.0000, -99.0000, -99.0000],
       [ -1.0000,  -1.0000,  -1.0000,  ..., -99.0000, -99.0000, -99.0000]])
y_tensor =  torch.from_numpy(Y_train).float()
tensor([0., 0., 1.,  ..., 0., 0., 0.])
#Loss Computation
loss_func = nn.BCELoss()
#Optimizer
learning_rate = 0.0001
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

train_loss = []
iters = 500
for i in range(iters):
    y_pred = model(x_tensor)
    loss = loss_func(y_pred, y_tensor)
    print " Loss in iteration :"
    print (i, loss.item())

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    train_loss.append(loss.item())

In the above case , what i'm not sure about is loss is being computed on y_pred which is a set of probabilities ,computed from the model on the training data with y_tensor (which is binary 0/1).
Is this way of loss computation fine in Classification problem in pytorch? Shouldn't loss be computed between two probabilities set ideally ? If this is fine , then does loss function , BCELoss over here , scales the input in some manner ?

Any insights towards this will be highly appreciated

You are right about the fact that cross entropy is computed between 2 distributions, however, in the case of the y_tensor values, we know for sure which class the example should actually belong to which is the ground truth. So, you can think of the binary values as probability distributions over possible classes in which case the loss function is absolutely correct and the way to go for the problem. Hope that helps.


Cross-entropy via softmax: Mathematical derivation I am trying to understand the derivation of cross entropy loss in the context of softmax. However, some steps are still not clear to me. Hence, I would appreciate if someone could please explain.
y = one hot encoding (representing true label) = selector
$y ̂= $value predicted by NN and the softmax.
Through the NN:
$X\xrightarrow{\smash{NN, θ}} Z \xrightarrow{\smash{Soft-max}} \hat{y}$
To derive the loss function, we start out from the likelihood function that a given set of parameters θ of the model can result in prediction of the correct class of each input sample. The maximization of this likelihood can be written as:     $$ arg max _θ L(θ|y ̂,Z) $$
The likelihood $ L(θ|y ̂,z) $ an be rewritten as the joint probability of generating t and z given the parameters θ, which can be written as a conditional distribution:
$$ L(θ│\hat{y} ,Z)=P(\hat{y} , Z│θ)=P(y ̂│Z,θ)P(Z│θ) $$
Since we are not interested in the probability of z we can reduce this (assumption 1):
$$ L(θ│y ̂,Z)∝P(y ̂│Z,θ) $$
Which can be written as $P(y ̂|z) $ for fixed θ (assumption 2).
Since each yc is dependent on the full z and only 1 class can be activated in the t we can write
$$ P(y ̂│Z)= ∏_{(c=1)}^C(P(y ̂_c |Z))^{y_c} =∏_{(c=1)}^C((SoftMax(Z))^{y_c }=∏_{(c=1)}^C(y ̂_c )^{y_c} $$
Assumptions 1 and 2 are not clear to me. Some explanation would be appreciated.

Hints:

$P(Z|\theta)$ is independent of $\hat{y}$ so it can be left out as merely a multiplicative factor as we are interested in $\hat{y}$.
$\theta$ is a parameter and for each model is indeed fixed, it is not a variable


Jupyter notebook running very slow I am running a RandomForestClassifier on my data but my jupyter notebook is very slow. It took almost 2 hours to run the below code:

rf = RandomForestClassifier()
rf_random = RandomizedSearchCV(estimator = rf, param_distributions =
   random_grid, n_iter = 100, cv = 3, verbose=2, random_state=42, n_jobs = -1)
rf_random.fit(X_train, y_train)

My dataset has 30K rows and 300 features.

I am not sure if something is wrong with my code or jupyter notebook configuration.
I am using a remote desktop windows machine.

I would really appreciate any help!
Thanks in adnvace

In this case you are running a RandomizedSearchCV which is running 100 iterations.
If you consider the fact that for every run of your 30K rows worth of data with youe 300 features (which is a fair amount), you would be looking at an average run time of ~ 1.2 minutes per run.

You could however speed this if you were running thia via GPUs instead of CPUs as you could do more rapid calculations.

So to answer your question the issue is not with your machine or your Jupyter Notebook.
Rather it is with how many iterations you have had with your RF Randomized Search Algorithm. If you reduce the iterations, you will also see a reduction in the run time.