Confusion Matrix vs AUC Confusion with Random Forest I'm currently training a random forest classifier with 1000 trees and am receiving the following confusion matrix. I've split the training set 75/25 against the test set but am not sure why I'm seeingf such a low True Negative score?  This is a binary classification problem and my dataset contains 88% of Class 0 whereas Class 1 represents 12% of the total content.  I've tried experimenting with class_weights but I have not been able to get the True Negative score above 6%. What am I missing as it seems to pass most tests including Cross-Validation > .90 etc.

From your confusion matrix it actually looks like your class imbalance is over-favoring the positive class (90.04% to 9.96%). Because you have such an extreme class imbalance in your data your random forest is likely just classifying almost everything as positive class and calling it a day.
To solve this issue you need to either down-sample your larger class or up-sample your lesser class to reach closer to 50/50 split. Otherwise you will likely just end up with a classifier that grossly over-predicts the positive class.
When you are splitting your data you can also try stratifying by your label (so that the same proportion of positive to negative class exists in your training and testing sets).


Why does my model produce too good to be true output? I am trying to run a binary classification problem on people with diabetes and non-diabetes.
For labeling my datasets, I followed a simple rule. If a person has T2DM in his medical records, we label him as positive cases (diabetes) and if he doesn't have T2DM, we label him as Non-T2DM.
Since there are a lot of data points for each subject, meaning he has a lot of lab measurements, a lot of drugs taken, a lot of diagnoses recorded, etc, I end up with 1370 features for each patient.
In my training, I have 2475 patients and in my testing, I have 2475 patients. (I already tried 70:30. Now am trying 50:50 still the same result (as 70:30))
My results are too good to be true as shown below

Should I reduce the number of features? Is it overfitting? Should I retain only the top features like top 20 features, top 10 features etc? can help me understand why is this happening?
Detailed Update
We check for the presence of T2DM by a set of diagnosis codes (like icd9,10 codes for T2DM and its complications etc). For ex: let's say if a patient has an icd9 code of 250 in his diagnosis records, we know that he was diagnosed with Type 2 diabetes. Let's not worry about the accuracy of this labeling approach at this point. Similarly, we label all other patients as T2DM and Non-T2DM.
But when we extract features, all his medical records are treated as features. The frequency of the drug/condition/lab tests will be used as a feature value. So, basically, the same diagnosis code (250) will be an input feature as well. Does it mean I should drop the diagnosis codes which were used to label a dataset from being used as features? But those are really very good features that can help me find out whether a patient is diagnosed with T2DM or not (when I apply my model on a totally different dataset). My objective is not to find out whether a patient will develop diabetes in the future or not but my objective is only to find out whether a patient is diabetic or not (from his records). So, I label my dataset with an imperfect heuristic as stated above and build a model. Once I build this model, I would like to validate this model at another site and find out how good is the model built using this imperfect heuristic in identifying whether a patient is diabetic or not. Hope this helps

Assuming that these results are obtained on a valid test set with no data leakage, these results don't show overfitting because overfitting would cause great performance on the training set but significantly lower perfomance on the test set.
Make sure that your instances between the training and test set are truly distinct: there might be some data leakage, for example if there are duplicate records for the same patient.
Another potential flaw is the gold standard labeling: if the patient has this T2DM  in their medical record, it means that they are already diagnosed right? And since the features are also based on the medical record, it's likely that this medical record contains direct indications about this diagnosis (for example in the drugs prescribed to the patient). There are two interpretations about this:

either the task is purposefully defined by this T2DM label, and in this case you can just enjoy the great performance but it's not technically about detecting diabetes in general.
or the goal is to detect patients with diabetes including the ones who are not diagnosed yet, but then it's likely that your gold standard is incorrect for this task.

[edit following updated question]
Your update clarifies which exact task you're targeting, and it corresponds exactly to my first interpretation above: given that your goal is to predict which instances satisfy this T2DM criterion and that the features contain direct indications about it:

I think you're right to keep these features, if a very useful information is available it would be absurd not to exploit it (assuming it's also available in the same form in any future dataset you plan to use, of course)
The very high performance you obtain makes perfect sense for this task, it's not a bug. It simply happens that the task is easy, so the system is able to predict the label very well.

However this also means that you could do even better without any ML: currently the ML method gives you around 99% F-score because it doesn't perfectly represent the criterion used for the gold. But since the gold standard label is based entirely on the features, the most direct way to "predict" the label is to apply the criterion. There's no point using a complex ML method to predict an information that you can obtain from the same data more accurately with a deterministic method.


Predict customer action from previous buying history I'm trying to predict what service a customer wants when he comes to our office from his previous transactions history. I have 7 years transactions data(3 crore txns) and good amount of customers are frequent ones.Each service is personal to each customers.

sample data

[
    {
        "customerId":"1xxxx",
        "txns":[
            {
                "serviceId":"12ds23",
                "date":"2016-08-03T08:43:33Z"
            },
            {
                "serviceId":"1dsd89",
                "date":"2016-09-03T08:43:33Z"
            },
            {
                "serviceId":"1dbbb89",
                "date":"2016-10-03T09:43:33Z"
            }
        ]

    },{
        "customerId":"2xxxx",
        "txns":[
            {
                "serviceId":"dds2dfsd",
                "date":"2016-08-03T08:43:33Z"
            },
            {
                "serviceId":"dsdsdsdf",
                "date":"2016-09-03T08:43:33Z"
            },
            {
                "serviceId":"sdfbb9",
                "date":"2017-10-03T09:43:33Z"
            },
            ...
            ...
        ]

    },
    ...
    ..
    ..
]

Can someone please advice which Machine learning technique or statistical approach would be best in this case.

I can think of a Decision tree classification/Logistic regression model taking date(month,day,day of week) as features for predicting the service he wants as Class labels

You can have a variety of solutions, starting from very simple to a more complex and beneficial ones.

I suggest that you'll start with the simple solutions, jain much of the benefit and continue by need.

Note that though the problem can be treated as a supervised learning problem, it is a multi label problem (a customer can be interested in many services). Therefore, decision tree or logistic regression is not suitable (unless you have a handful of services and you try to predict if the user would like a service, for each service).

The classical method to provide such recommendations are "People that like X, also liked Y". Though very simple, at time 50% of Amazon's revenues where due to that.

You should make the recommandation more sepcific by considering the lift - Harri Potter is very popular and many people like it. However, it isn't the most sutabile recommandation for someone reading browsing books.

You can take such a item-to-item recommendation and use all user history by aggregating using Naive Bayes or similar methods.
You can take care of temporality and dates by giving more eight to recent data.

The next step is moving to recommendation system. You can find on the web some implementation.
The winner in the Netflix challenge used matrix factorization algorithm, so this direction is very popular for recommendation systems.


What is the vector value of [CLS] [SEP] tokens in BERT In BERT, They replace separator and start of sentence with special token labels. What are there corresponding values in embedding_matrix. Are they 0-vector?

I wanted to replace the proper nouns like names, buildings, locations with similar approach. How should i go about masking the same?

First a clarification: there is no masking at all in the [CLS] and [SEP] tokens. These are artificial tokens that are respectively inserted before the first sequence of tokens and between the first and second sequences.

About the value of the embedded vectors of [CLS] and [SEP]: they are not filled with 0's but contain numerical representations like any of the other vectors in the embedding table. From the point of view of the model, there is nothing special about their embedded vectors, so they are trained normally like the others.

About masking proper names: you should take into account that BERT is a subword-based model. This means that words get split into subwords that are part of the vocabulary. Sometimes a word maps to only one token, but other times a single word maps to a sequence of several tokens. Infrequent words tend to be sliced almost at the character level. Depending on the data the model is trained with, the proper nouns you are referring to may be split into multiple subwords. You should take this into account when designing your masking procedure by either:

Before tokenizing, you replace the proper nouns with a text sequence that is included in the vocabulary and that is exclusive to them.
After tokenizing the original sentence, you check if the text contained a proper noun and if so, you locate the sequence of subwords associated to them and replace each subsequence with a special proper noun token ID.

Take into account that this is not similar to BERT's inherent masking, where the input token is masked but the loss forces the model to predict the original token. In your case, the original proper noun would be masked to never return.


Does finetuning BERT involving updating all of the parameters or just the final classification layer? Currently learning and reading about transformer models, I get that during the pretraining stage the BERT model is trained on a large corpus via MLM and NSP.  But during finetuning, for example trying to classify sentiment based on another text, are all of the BERT parameters (110M+ parameters + final classification layer) updated or just only final classification layers?  Couldn't find a concrete answer to this in the resources I've been looking at.
Thank you in advance.

Both approaches are reasonable. Updating the BERT weights will train for longer period of time, but should give more accurate results.


ModuleNotFoundError: No module named 'deeplab' I am trying to run Tensorflows DeepLab using my own dataset, while following: heaversm tutorial on GitHub/Youtube.
P.S I am running Windows.
I want to run the model_test.py script (on my windows terminal - CMD) but it just returns the typical:
Traceback (most recent call last):
  File "model_test.py", line 21, in <module>
    from deeplab import common
ModuleNotFoundError: No module named 'deeplab'

I have installed all dependencies including tensorflow:
Name: tensorflow
Version: 2.3.1

I have also included (permenantly in the "Path" environment) the directories:
C:\Users\Name\Desktop\models-master\models-master\research\deeplab\
C:\Users\Name\Desktop\models-master\models-master\research\slim\
C:\Users\Name\Desktop\models-master\models-master\research\

But nothing worked when I decided to re-run model_test.py.
So I then decided to temporarily set a environment variable named PYTHONPATH with the following (as the tutorial mentions):
# Making sure I am in the models\research directory (as per the tutorial)
# In my case it is models-master\research directory
export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim  # --> macOS version
set PYTHONPATH=%PYTHONPATH%;%cd%;%cd%/slim  # --> Windows version

Again nothing worked... I am running out of options and hope.
Am i doing anything wrong here?? Am i missing a certain folder maybe?
Any help would be appreciated.

You need to start you program from the models/research repository (cd ./models/research before executing)


LeCun paper on deeplearning (Nature, 2015) As I was reading Y. LeCun's paper on Deep Learning (Nature, vol. 521, 2015), I came across a figure (the 1st one in the paper) which was associated to the backward pass during backpropagation through a convolutional network. I understood everything apart from the very last sentence of the legend that accompanied the figure: "Once the $\frac{∂E}{∂zk}$
is known, the error-derivative for the weight $w_{jk}$ on the connection from unit j in the layer below is just $y_j\cdot\frac{∂E}{∂zk}$".

It must be purely mathematical but I can't figure out why the error-derivative mentioned above equal $y_j$ times $\frac{∂E}{∂zk}$. Can someone explain to me this result?

Thx.

PS : link to the figure mentioned above is https://www.researchgate.net/figure/Multilayer-neural-networks-and-backpropagation-a-A-multi-layer-neural-network-shown-by_fig4_277411157

It would be helpful:
https://www.youtube.com/watch?v=d14TUNcbn1k

As far as I understand, in backpropagation, we need to multiply the local gradient of each node by the gradient from the upper level and this is the value that we send back


Use output of K-Mean for Logistics regression I've created a binary classifier using K Mean, which predicts fraud and legitimate accounts, 0 and 1. This uses two features, let's say, A and B.

Now, I want to use other features like C and D, to predict fraud and legitimate accounts. Can I use output class predicted in Kmeans, and use it as a label for logistics regression using features C and D?

Sorry, if this sounds like dummy questions, I'm just starting into Data Science, so please let me know if I missed something fundamental.

It is strange to use k-means in addition to logistic regression. Usually k-means is reserved for unsupervised learning problems, this is when you do not have labelled data. Unsupervised learning algorithms are not as powerful and it seems here you have labelled data, thus you should stick to supervised learning techniques.

If you have additional features for your instances then you should include them in your logistic regression.


Naive Bayes Denominator clarification I came across an earlier post that was resolved and had a follow up to it but I couldn't comment because my reputation is under 50. Essentially I am interested in calculating the denominator in Naive Bayes.
Now I understand that the features in Naive Bayes are assumed to be independent so could we calculate $p(x) = p(x_{1})p(x_{2})...p(x_{n})$ or would we have to use this formula $$p(\mathbf{x}) = \sum_k p(C_k) \ p(\mathbf{x} \mid C_k)$$ with the conditional independence assumption that$$ p(\mathbf{x} \mid C_k) = \Pi_{i} \, p(x_i \mid C_k) $$
My question is would both ways of calculating give the same p(x)?
Link to the original question : https://datascience.stackexchange.com/posts/69699/edi
Edit** : Sorry I believe the features have  conditional independence, rather than complete independence. Therefore it is incorrect to use $p(x) = p(x_{1})p(x_{2})...p(x_{n})$?
Lastly, I understand we don't actually need the denominator to find our probabilities but am asking out of curiosity.

The way to calculate $p(x)$ is indeed:
$$p(x) = \sum_k p(C_k) \ p(x| C_k)$$
Since in general one needs to calculate $p(C_k,x)$ (numerator) for every $k$, it's simple enough to sum all these $k$ values. It would be incorrect to use the product, indeed.

Lastly, I understand we don't actually need the denominator to find our probabilities but am asking out of curiosity.

Calculating the marginal $p(x)$ is not needed in order to find the most likely class $C_k$ because:
$$argmax_k(\{ p(C_k|x)  \}) = argmax_k(\{ p(C_k,x)  \})$$
However it's actually needed to find the posterior probability $p(C_k | x)$, that's why it's often useful to calculate the denominator $p(x)$ in order to obtain $p(C_k | x)$, especially if one wants to output the actual probabilities.


What happens when you have highly correlated columns in a dataset? I am doing a regression model. And I was wondering what would be the consequence if we have two or more Highly correlated columns in the dataset? Is that something that can decrease the accuracy of the model?
Answering this question would help decide how to deal with it. Would PCA be the best option here?

Having highly correlated features is a type of redundancy in features. And yes, it effects a regression model if you are having highly correlated features. A very nice explanation is given here.

PCA is a nice choice when it comes to dimensionality reduction.


Variable time units LSTM I am working with data that requires classifying if a patient will develop cancer or not in the future, based on medical tests done over time. The tests have a sequential relationship. A, then B, then C, etc. For example:

| Patient ID | Test ID | RBC Count | WBC Count | Label
|     1      |    A    |     4.2   |    7000   |   0
|     1      |    B    |     5.3   |    12000  |   0
|     1      |    C    |     2.4   |    15000  |   1
|     2      |    A    |     7.6   |    8000   |   0
|     2      |    B    |     7.4   |    7500   |   0

Each point is not taken at a regular time interval, so this may not be considered a time-series data. How can I apply Keras Sequential model to this? Especially since each patient can have a different number of tests, up to 5, eg. Patient ID 1 has 3 tests, and Patient ID 2 has 2 tests.

[Wow.A Great problem]

Possible solution:(Personal opinion)

First Opinion is: Use SVM or KNN instead of a neural net as your data is trivial like 2D vector.

Second Opinion is:

As you aren't trying to predict the next text possibility of a particular patient so in this case RNN(LSTM,GRU) might not help you. So a simple solution is Autoencoder to compress the data and then run a simple logistic regression or svm or may be a neural net.

How?

1.As your Patient numbers are not same in all cases so you should represent his/her all medical test data into a single vector.That is where Autoencoder comes into play.example:

Suppose we have 3 patient: First-3 test , second-2 test ,three-5 test. Our goal is to compress each patient data into 2 dimension vector(as your features are RBC and WBC , 2 features , so it will learn well if we compress it into 2D vector).

(inside asterik sign(*) your data goes)

First Patient=[[*],[*],[*]] # three test
Second Patient=[[*],[*]] # two test
Third Patient=[[*],[*],[*],[*],[*]] # five test

Now you compress the data via Variational autoencoder.

scale your data as wbc>>rbc

then use variational autoencoder to get the latent space vector

use dimesion=2 in latent space via Dense(2)

now when you get the vectors just add the vectors of test for each patient

Like : For First patient you got three vector in latent space like-

[2.33,4.2]+[5.11,9.22]+[0.21,6.32]=result..and You got the representaion for First patient.

after you got all of them a trivial svm/polynomial regression would be just fine.

Let me know how you solved it.


Linear Regression and scaling of data The following plot shows coefficients obtained with linear regression (with mpg as the target variable and all others as predictors).

For mtcars dataset (here and here) both with and without scaling the data:

How do I interpret these results? The variables hp and disp are significant only if the data is scaled. Are am and qsec equally important or is am more important than qsec? Which variable should one say are important determinants of mpg?

Thanks for your insight.

You can't really talk about significance in this case without standard errors; they scale with the variables and coefficients. Further, each coefficient is conditional on the other variables in the model, and collinearity actually seems to be inflating the importance of hp and disp.

Rescaling variables should not change the significance of results at all. Indeed, when I reran the regression (with the variables as is, and normalized by subtracting off the mean and dividing by the standard errors), each coefficient estimate (except the constant) had exactly the same t-stat as before scaling, and the F-test of overall significance remained exactly the same.

That is, even when all of the variables have been scaled to have a mean of zero and a variance of 1, there is no one size of standard error for each of the regression coefficients, so just looking at the magnitude of each coefficient in the standardized regression is still misleading about significance.

As David Masip explained, the apparent size of the coefficients has an inverse relationship with the magnitude of the data points. But even when the coefficients on disp and hp are huge, they are still not significantly different from zero.

In fact, hp and disp are highly correlated with each other, r=.79, so the standard errors on those coefficients are especially high relative to the coefficient magnitude because they are so collinear. In this regression, they're doing a weird counterbalancing, which is why one has a positive coefficient and one has a negative coefficient; it seems like a case of overfitting and does not seem meaningful.

A good way to see which variables explain the most variation in mpg is the (adjusted) R-squared. It is literally the percentage of the variation in y that is explained by the variation in the x variables. (Adjusted R-squared includes a slight penalty for each additional x variable in the equation, to counterbalance overfitting.)

A good way to see what's important--in light of the other variables--is to look at the change in adjusted R-squared when you leave out that variable from the regression. That change is the percentage of variance in the dependent variable that that factor explains, after holding constant the other variables. (Formally, you can test whether the left-out variables matter with an F-test; this is how stepwise regressions for variable selection work.)

To illustrate this, I ran single linear regressions for each of the variables separately, predicting mpg. The variable wt alone explains 75.3% of the variation in mpg, and no single variable explains more. However, many of other variables are correlated with wt and explain some of that same variation. (I used robust standard errors, which might lead to slight differences in standard error and significance calculations but will not affect coefficients or R-squared.)

+------+-----------+---------+----------+---------+----------+-------+
|      |   coeff   |   se    | constant |   se    | adj R-sq | R-sq  |
+------+-----------+---------+----------+---------+----------+-------+
| cyl  | -0.852*** | [0.110] |        0 | [0.094] |    0.717 | 0.726 |
| disp | -0.848*** | [0.105] |        0 | [0.095] |    0.709 | 0.718 |
| hp   | -0.776*** | [0.154] |        0 | [0.113] |    0.589 | 0.602 |
| drat |  0.681*** | [0.123] |        0 | [0.132] |    0.446 | 0.464 |
| wt   | -0.868*** | [0.106] |        0 | [0.089] |    0.745 | 0.753 |
| qsec |  0.419**  | [0.136] |        0 | [0.163] |    0.148 | 0.175 |
| vs   |  0.664*** | [0.142] |        0 | [0.134] |    0.422 | 0.441 |
| am   |  0.600*** | [0.158] |        0 | [0.144] |    0.338 | 0.360 |
| gear |  0.480*   | [0.178] |        0 | [0.158] |    0.205 | 0.231 |
| carb | -0.551**  | [0.168] |        0 | [0.150] |    0.280 | 0.304 |
+------+-----------+---------+----------+---------+----------+-------+

When all the variables are in there together, the R-squared is 0.869, and the adjusted R-squared is 0.807. So, throwing in 9 more variables to join wt just explains another 11% of the variation (or merely 5% more, if we correct for overfitting). (Many of the variables explained some of the same variation in mpg that wt does.) And in that full model, the only coefficient with a p-value under 20% is wt, at p= 0.089.


Feature Importance Scores Python I have a dataset having 7 attributes viz., time, C1, ... C7 pertaining to earth quake reports where each column/attribute represents a certain aspect of damage viz., power, sewer_and_water, shake_intensity, etc. Each of these attributes have a rating from 0 to 10, where 0 represents no damage and 10 represents maximum damage.

Since, some of the ratings in each attribute might not be trustworthy, hence I am trying to compute a weighted score as follows-
(C1 * W1) + (C2 * W2) + ... + (C7 * W7); where Ci is ith attribute and Wi is the feature importance score for it.

In order to compute a weighted score of each row/data point, I am trying to train a classifier such as Random Forest, LightGBM or XGBoost which will give me a feature importance score for each attribute. However, since I don't know which attribute is supposed to be target, I used a brute force approach where I chose each attribute as target and trained a classifier to see which gave me the highest accuracy.

However, this approach has the problem that out of n attributes, I will only get feature importance score for (n - 1) attributes as the nth attribute is the target variable.

Can you suggest a way in which I can get the feature importance scores for each attribute in dataset.

Thanks!

Not all the algorithms provide you feature_importance, some of them give you coeff(weights W1..W7) values.
So according to me, the way you are extracting feature_importance is correct.

  However, this approach has the problem that out of n attributes, I
  will only get feature importance score for (n - 1) attributes as the
  nth attribute is the target variable.

Model gets trained based upon the features and target value. So you will get the feature_importance only for features and not for the target.

Using statistical approach you can also analyse the relationship between the various features and the target value.

As a side note:

Make sure you normalized the whole features on the same scale for
better and faster training.


NER with Unsupervised Learning? If we treated NER as a classification/prediction problem, how would we handle name entities that weren't in training corpus?

For example, "James was born in England." James was labeled as a PERSON and England as a LOCATION. But we type another completely strange sentence like "Fyonair is from Fuabalada land." We as human can understand Fyonair is a person (or maybe princess from a fairy tale) and Fuabalada is the land where she comes from.

How would our model identify it if it wasn't included in billions of corpus and tokens? Can unsupervised learning achieve this task?

First, when we as humans interpret "Fyonair is from Fuabalada land" we use our knowledge of "is from" and "X land" to infer that Fyonair is probably a person and Fuabalada land probably a location. Therefore our process is not (at least not completely) unsupervised: we have seen this sentence structure before ("training") and we use our "model" to "predict" the two entities.

A good (supervised) NER is trained to recognize this kind of patterns as well. The example that you mention might be a bit too hard, but with something like "Dr Fyonair is the new CEO of Fubalada" a decent NER should be able to recognize that "Dr Fyonair" is a person (thanks to the "Dr") and that "Fubalada" is a company (thanks to the "CEO of") even though it has never seen these particular proper names.

So a standard (supervised) NER is supposed to recognize entities it has never seen before, provided it has clues in the sentence about them. Technically it's not really a state of the art NER if it recognizes only the entities that it has seen during training, it's just a string matching program. It's true that NERs make a lot of errors with entities which haven't been seen previously, but that's simply because they are harder to catch in general.


Feature importance with scikit-learn Random Forest shows very high Standard Deviation I am using scikit-learn Random Forest Classifier and I want to plot the feature importance such as in this example.

However my result is completely different, in the sense that feature importance standard deviation is almost always bigger than feature importance itself (see attached image).

Is it possible to have such kind of behaviour, or am I doing some mistakes when plotting it?

My code is the following:

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier

clf = RandomForestClassifier()
clf.fit(predictors.values, outcome.values.ravel())

importance = clf.feature_importances_
importance = pd.DataFrame(importance, index=predictors.columns,
                          columns=["Importance"])

importance["Std"] = np.std([tree.feature_importances_
                            for tree in clf.estimators_], axis=0)

x = range(importance.shape[0])
y = importance.ix[:, 0]
yerr = importance.ix[:, 1]

plt.bar(x, y, yerr=yerr, align="center")

plt.show()

You are using RandomForest with the default number of trees, which is 10.
For around 30 features this is too few. Therefore standard deviation is large. Try at least 100 or even 1000 trees, like

clf = RandomForestClassifier(n_estimators=1000)

For a more refined analysis you can also check how large the correlation between your features is.


Using several documents with word2vec I want to train word embeddings using word2vec. My corpus is split into several documents (it's a large set of patient notes). Should I just concatenate all documents into one before running word2vec on it, or is there a better way?

There are a number of implementations of Word2Vec, but most assume the basic unit to be 'sentences' - though they don't care what those sentences look like. If you are using something like gensim you will need each sentence in its own list, and each sentence will be a list of tokens. If you are using another package you may be reading from the disk in which case you likely need all documents concatenated with each newline representing a new sentence.

The fundamental consideration in deciding on the size of a 'sentence' (whether it's the entire note, all a patient's notes, or a single sentence) is what surrounding words should be used to train the model. The W2V model will consider all words within a certain distance of the target word when tuning its vector representation (regardless of if it's cbow or skipgram), but will not look beyond a 'sentence' boundary. So if your fundamental unit is a document not a sentence, associations will bleed across sentences (this may or may not be what you want).

As an aside, consider the nature of the documents first. Patient notes can be messy and full of automatically generated text, so it can be important to strip out or replace certain strings. Similarly, they often use whitespace and newlines rather than punctuation. You might want to consider treating existing whitespace as the end of a sentence and you might want to use a parser to replace dates, names, etc with default tokens to increase generalizability.


How to interpret results of a Clustering Heart Failure Dataset? I am doing an analysis about this dataset: click
In this dataset there are 13 features, 12 of input and 1 is the target variable, called "DEATH_EVENT". I tried to predict the survival of the patients in this dataset, using the features.
Hoewever, now I was trying to do a cluster analysis to see if the patients are grouped in clusters.
This is the code I have written.
from sklearn.cluster import KMeans
Features = ['ejection_fraction','serum_creatinine'] #the highest correlated features with death_event
X = heart_data[Features]

wcss = []
for i in range(1, 11):
    kmeans = KMeans(n_clusters=i, init='k-means++', max_iter=300, n_init=10, random_state=0)
    kmeans.fit(X)
    wcss.append(kmeans.inertia_)
plt.plot(range(1, 11), wcss)
plt.title('Elbow Method')
plt.xlabel('Number of clusters')
plt.ylabel('WCSS')
plt.show()

From this graph I can observe that there are 2 clusters. Now
kmeans = KMeans(n_clusters=2, init='k-means++', max_iter=300, n_init=10, random_state=0)
pred_y = kmeans.fit_predict(X)
plt.scatter(X["ejection_fraction"], X["serum_creatinine"])
plt.scatter(kmeans.cluster_centers_[:, 0], kmeans.cluster_centers_[:, 1], s=300, c='red')
plt.show()

And I obtained this chart:

Now, what can I say from this chart? I think that it is unuseful, right?
I used only the features ejection_fraction and serum_creatinine because these are the only I used for the prediction. Or I have to use all the variables except from DEATH_EVENT? In this way:
X = heart_data.iloc[:, :11]

But in this case I obtain this:

I am not able to understand these charts, I think that I am doing something wrong, but what? Where are the clusters? How to interpret these results?
UPDATE: I am not able to use Umap_learn, my Mac can not install it, I received a lot of errors. However, I did something related to your advices.
Here there is all the code: https://pastebin.com/RdJb0ydu
The first 2 parts are the code that you have written here.
In the 3rd part I use kmeans(n_clusters=2) because from the silhouette I saw that the best was with 2 clusters. Then I did the prediction and concatenated the results to the original dataset and I printed out the column of DEATH_EVENT and the column with the results of clustering.
From this column, what can I say? How can I understand if the 0 of the prediction refers to the survived patients or to the died patients?

I would use all the features and see how the separateness of my clusters behave according to some metric, for example, silhouette score
Additionally, it is very important to scale your data prior to clustering since kmeans is a distance-based algorithm.
heart_data = pd.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/00519/heart_failure_clinical_records_dataset.csv")

from sklearn.cluster import KMeans
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import silhouette_score

Features = heart_data.drop(["DEATH_EVENT"], axis = 1).columns

X = heart_data[Features]

sc = []
for i in range(2, 25):
    kmeans = Pipeline([("scaling",StandardScaler()),("clustering",KMeans(n_clusters=i, init='k-means++', max_iter=300, n_init=10, random_state=0))]).fit(X)
    score = silhouette_score(X, kmeans["clustering"].labels_)
    sc.append(score)
plt.plot(range(2, 25), sc, marker = "o")
plt.title('Silhouette')
plt.xlabel('Number of clusters')
plt.ylabel('Score')
plt.show()

You could also try different combinations of features so that score is maximum
For visualization purposes you can use a decomposition technique
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
plt.style.use("seaborn-whitegrid")

pca = Pipeline([("scaling",StandardScaler()),("decompositioning",PCA(n_components = 2))]).fit(X)

X2D = pca.transform(X)

plt.scatter(X2D[:,0],X2D[:,1], c = kmeans["clustering"].labels_, cmap = "RdYlBu")
plt.colorbar();

Last but not least, I recommend to use a manifold projection such as UMAP in your data, It might help on your task by generating "well-defined" clusters (might but not necessarily, nonetheless it is worthy to try)
Look, by using UMAP the results seems to improve:
code:
# pip install umap-learn

heart_data = pd.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/00519/heart_failure_clinical_records_dataset.csv")

from sklearn.cluster import KMeans, DBSCAN
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import silhouette_score
from umap import UMAP

Features = heart_data.drop(["DEATH_EVENT"], axis = 1).columns

X = heart_data[Features]

sc = []
for i in range(2, 25):
    kmeans = Pipeline([("scaling",StandardScaler()),("umap",UMAP()),("clustering",KMeans(n_clusters=i, init='k-means++',random_state=0))]).fit(X)
    score = silhouette_score(X, kmeans["clustering"].labels_)
    sc.append(score)
plt.plot(range(2, 25), sc, marker = "o")
plt.title('Silhouette')
plt.xlabel('Number of clusters')
plt.ylabel('Score')
plt.show()

from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
plt.style.use("seaborn-whitegrid")

kmeans = Pipeline([("scaling",StandardScaler()),("umap",UMAP()),("clustering",KMeans(n_clusters=3, init='k-means++',random_state=0))]).fit(X)

pca = Pipeline([("scaling",StandardScaler()),("umap",UMAP()),("decompositioning",PCA(n_components = 2))]).fit(X)

X2D = pca.transform(X)

plt.scatter(X2D[:,0],X2D[:,1], c = kmeans["clustering"].labels_, cmap = "RdYlBu")
plt.colorbar();

Plot show first and second principal components of umap projection (It is simply a projection of how  all the features would look in 2D space)

Colours are the cluster id. i.e. for every colour we see in which cluster the algorithm (k-means) assigned each observation to.


Online machine learning tutorial Does anyone know some good tutorials on online machine learning technics?
I.e. how it can be used in real-time environments, what are key differences compared to normal machine learning methods etc.

UPD: Thank you everyone for answers, by "online" I mean methods which can be trained in a real-time mode, based on a new inputs one by one.

There is a very nice library of online machine learning algorithms from a group at NTU, called LIBOL. This would be a very good place to start experimenting with the algorithms.
The accompanying user guide, and associated JMLR publication are very nice introductions to the basic algorithms in this field.
Avrim Blum has an older and more technical survey paper on online learning algorithms.


Is R2 score a reasonable regression measure on huge datasets? I'm running a regression model on a pretty large data set and getting a fairly woeful $R^2$ score of ~0.2 (see plot below), despite the plot looking like the model is generally pointing in the right direction.

My question is, when you have over a million data points, how high can you realistically expect the $R^2$ to go in real world data with a decent amount of noise?

What prompts by scepticism of such traditional measures are articles such as this that discuss how quantity of data can degrade statistical tests.

Let me know what you think and any regression examples using $R^2$ score as a quality metric.

The coefficient of determination $r^2$ is defined in terms of variance: it is the proportion of variance in the dependent variable that is explained by the independent variable. Variance is a property of normal distributed data. Hence, the coefficient of determination can only be used when you assume that  both the dependent and independent variables are normally distributed.

Just like other properties of normal distributed data, the estimation of $r^2$ improves when the amount of data increases. With very little data a coincidental colinearity might be the case, but this is not possible with large amounts of data.

Back to your example. Your data is cleary not normal distributed, it has right-skew and has large outliers. For this reason it is not advisable to use   $r^2$. Imagine for example that in the lower left corner (where the majority of the data is) you would observe a negative trend, but in general there is a positive trend. The regression line would be the same and the $r^2$ would be in the same range. This is known as the Simpsons paradox.

In short, if your data is normal distributed you can use $r^2$ for any size of dataset. If it is not normal distributed you cannot use $r^2$.


Sequence to carry out data analysis? I have a dataset with 4700 records and it's a classification problem. Proportion of classes is 33 and 67%

few questions

1) does this proportion qualify dataset as imbalanced ?

2) should I do cross validation and then apply (over/under or SMOTE sampling)  or I should first balance my sample through these sampling techniques and then do cross validation?

3) Why is propensity score matching used only in healthcare related studies and not much in other applications?

4) How is Propensity score matching different from other ML classification algorithms?

You should fit preprocessing transformers, i.e. imputation, scalers, encoders, resampling, only to train set and apply them to both train and test respectively. Your dataset is imbalanced and you may expect some improvement using resampling techniques, though you should always confirm it conducting cross validation tests.


laptops which are suitable for heavy Image and video processing I am a graduate student and My thesis is based on deep learning, vision, video and Image processing. I am going to do heavy computing and processing. I am looking for a compatible laptop which has a fast GPU-enabled laptop. CUDA enabled on an NVIDIA GPU with good TFOPS would be a perfect choice. Also, it must have SSD. Anything by hp, Dell, Microsoft, Acer. do you have any recommendations?  my budget is $CAN 1800.
which GPU-Nvidia is suitable for this kind of programming?

The answer to such a question is opinion based and the question itself is very broad.

I have used an HP Envy in the past, with a 4GB Nvidia 950M GPU, which worked well with Linux installed.

In general:

The higher the compute capability of the GPU the better (look here at the list for Nvidia GPUs under GeForce products for notebooks).
Another option worth considering would be to build compute - you will lose portability, but get a vast improvement of power for the same money.
Finally, you could buy a cheap laptop and then use the rest of your budget on cloud services, such as Google Cloud, Amazon, etc. This might be the best option to get started (with some free credit they usually offer for new accounts) - also if you think you might only be using the GPU for your thesis, and not training huge models for days at a time after that period. If you know you'll be training large models over long periods of time, it's generally more cost effective to buy the hardware.


Ideas on speeding up code cols = ['permit_creation_date', 'current_status_date','filed_date', 'issued_date', 'completed_date',
           'first_construction_document_date', 'permit_expiration_date']

df_raw[cols] = df_raw[cols].apply(pd.to_datetime)

I wrote this code to convert some of the columns in the dataframe to convert the datatype to a datetime. When I run the cell in jupyter notebook it takes about 1 min in runtime. Any thoughts on how I could speed up the runtime.

Pass the panda series directly to pandas.to_datetime() function. It works faster than using apply function and passing pandas.to_datetime as a parameter.

In[20]: %timeit -n 1000 pd.to_datetime(mdp)
96.1 µs ± 12.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In[21]: %timeit -n 1000 mdp.apply(pd.to_datetime)
5.59 ms ± 91.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Incase of dataframe, looping through the series works fine as well.

In[45]: %timeit [pd.to_datetime(df[x]) for x in df]
230 µs ± 20.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)`

In[46]: %timeit df.apply(pd.to_datetime)
1.22 ms ± 10.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)


What is the fully-convolutional model? What is the fully-convolutional model?

Is fully-convolutional model a model that has only convolutional layers (with Batch-norm and Activation) and has not any: max-pool, fully-connected, and other layers?

If it is true, then why Yolo v2 neural network for object detection is named Fully-Convolutional model if it uses Max-pool layers: https://arxiv.org/pdf/1612.08242v1.pdf

  Table 2: The path from YOLO to YOLOv2. Most of the listed design
  decisions lead to significant increases in mAP. Two  exceptions are
  switching to a fully convolutional network with anchor boxes and using the new network.

Also why Fully Convolutional Networks for Semantic Segmentation is named Fully Convolutional if it has Max-pool layers: https://people.eecs.berkeley.edu/~jonlong/long_shelhamer_fcn.pdf

In general, a network with CNN with no Fully connected layers is termed as Fully Convolutional Network(FCN). It can include any type of pooling layers, batch norm, dropouts etc.


in NLP academic paper, would it be okay to refer the "token embeddings" as "tokens"? I am writing a paper in Natural Language Processing (NLP), and I just have a quick question about terminology.

In language models like Transformers, "token" refers to individual word in a text sequence, whereas there is a special term "token embedding" to refer to the embedding that results after token gets passed through the initial embedding layer.

Would it be problematic if I just refer a "token embedding" as a "token"?

(e.g. "interaction between hidden embeddings and token embeddings" ---> "interaction between hidden embeddings and tokens")

I am trying to accommodate the different terminologies, but my sentences are getting really wordy...

Thank you,

At least to me, it would sound strange, as I would understand tokens as the discrete textual units they are, not their assigned vectors.

I would suggest that you don't try to force nonstandard simplifications. Just say what you want, in a technically accurate way, preferably using short sentences to avoid making it difficult for the reader to follow your discourse.


What exactly is the "hyperbolic" tanh function used in the context of activation functions? I know the plot of $\tanh$ activation function looks like. I also know that its output has a range of $[-1, 1]$. Furthermore, I also know the it is defined as follows

$$
\tanh(x) = \frac{\sinh(x)}{cosh(x)} = \frac{e^x - e^{-x}}{e^x + e^{-x}}
$$

What exactly is the meaning of term "hyperbolic"?
What is the interpretation of the word hyperbolic in this context? How is the equation above derived?

The hyperbolic trig functions follow the equation for a Rectangular hyperbola, which is something you should be familiar from analytical geometry.  The recentgular hyperbola is defined by $x^2 - y^2 = 1$ and if you let $x = cosh(t)$ and $y = sinh(t)$ and plug it into the rectangular hyperbola equation you can verify this fact quite easily.

From this idea you can derive the rest of the hyperbolic trig functions.  (Also they reason they are given trig names is because they resemble the properties trig functions have).


Is my dataset unlearnable, or is my LSTM model not smart enough? I have time-series data obtained from a video. The data is composed of bitrate and corresponding label pairs for each timestamp:

The distribution over the first 30 seconds is as follows:

I have built an LSTM model for this dataset to be able to classify the labels based on the bitrate. However, it seems that my model is not able to learn. Validation accuracy starts from approximately 0.3 (makes sense, since I have 2 classes (log2 = 0.3)) and it does not improve.
Do you have any idea about this? Is it normal considering this sample data distribution, or is something might be wrong with my model? Thanks!

An LSTM uses the preceding sequence of bitrates to produce a label. If the label can be predicted from those bitrates, then your problem is learnable with an LSTM.
From your comment: The label is the "stage of the video": Running, walking, etc. The question is if this can be derived from the bitrate? Walking with a busy background might yield a higher bitrate than running. Judging your example by eye, I suspect it cannot. You need to get more features from the video than just the bitrate, I think.


Curse of dimensionality: Data set with 1 million rows, max number of features can I have approximately? As the question entails, I would like to know as a rule of thumb, an upper limit on the number of features I could have for a data set with 1,000,000 rows/observations, before hitting the curse of dimensionality. If not a million observations, an answer with a rough estimate for the number of features for a data set with a specified size would be great too.

Thanks

The curse of dimensionality means that your intuition fails at a certain number of features. See Average Distance of Random Points in a Unit Hypercube for some examples. It does not matter how many points you have.

For example, at around 100 dimensions every two points randomly sampled points are really close.


How to classify a data as not matching any label I'm creating a supervised convnet which recognizes a phrase in a regional language.

For example:

  I have a dataset which contains 100 sounds for labels A and another 100 sounds for labels B. Now normal operation of the network is to differentiate a sound between label A and label B.

But let's say that I give a new sound to the network to classify, which is actually neither A nor B. I want the neural net to tell me that it doesn't qualify any of the labels.

How can I achieve this?

Let's phrase it another way, decomposing into two problems:

given a sound, we want to know if it's of class A
given a sound, we want to know if it's of class B

This way of putting it is valuable to some techniques, notably "one-class classification" and "PU learning" (learning from positive and unlabeled examples). These techniques are very relevant when you want to know "does my item follow a given distribution D?", which is exactly what you're looking for.

Still, if you have (or can collect) a lot of data that is neither A nor B, you may simply label it as a garbage class C and use a common classifier - however, it may actually be harmful, because you're likely assuming wrong things about class C [Li, Liu and Ng, "Negative Training Data can be Harmful to Text Classification", 2010]. It could work and is much simpler to develop, may be good enough depending on your case.


Identifying similar data points I have a data set where customers who are non-delinquent with us have defaulted with others. So from the total data set, 30% are those who defaulted with others but 70% are those who are good with us and other others as well.
With the help of 30% customers who defaulted with others (being good/non-delinquent with us), can we identify the similar customers (matching in profile) from the remaining 70% customers who are currently good with us as well as with others, and also to any new customer?
This is required so that we can have an early indication that who are the customers (from remaining 70% & any new addition) most likely turn bad in near future?
I am not sure how to apply any machine learning algorithm (as we just have 1 class defaulted with others) to identify similar customers.

It's generally treated as a binary classification problem, often called credit scoring. You are trying to know whether a specific entity will default with you or not. The fact that the entity has defaulted with another company can be used as feature of your problem. As mentionned by others, there are other approaches, but I honestly think they are in theoretical infency, and can't seem to find any literrature on their practical implementation to credit scoring.

Then you need to build an historical database to learn on, with an associated label. There are two main approaches : either you construct a table based on periodic (yearly) data and set an horizon for the default. It allows you to build a label by observing what happened in the past. Say you have yearly data and consider a 1 year horizon. You will have multiple line for a company : one for each year, say 2010 to 2018, the label will be built on their status the following year, 1 if they defaulted respectively in 2011, ..., 2019, 0 otherwise. Another approach is to consider life-time default, it is better in theory, and probably better match what you want to do, but it is unlikely you will be able to do that, notably because it require complete data and you usually don't keep those that defaulted.

You can learn on that database, but this is a complex process including variable selection, model selection, model training, model validation. Pretty much standard data science, there is a lot of introductory material you should look at, starting with : an introduction to statistical learning, which is free). Then you can go look at ressources oriented towards credit scoring like Credit Scoring and its application, then to supervised learning applied to credit scoring like deep learning for mortgage risk: https://arxiv.org/abs/1607.02470). This will help you measure the risk associated with your current population. More so if you calibrate your model in probability such that the output of your model reflect the probability of default.

However if you want to use that model for making decisions about lending, then the problem will be made more complex, because :

1) Your initial decision will have an impact on the credit situation of said entity. So you actually would need to assess the impact of both legs of your decision on the caracteristic of the entity before putting it in your prediction model.

2) You need to take costs into account. The costs are usually asymetrical and hard to predict. For loans for exemple, not doing the loan will cost you interests on the nominal (it loops to (1) as the % interest you may ask will modify the contract and hence the situation of the entity), but also associated returns (a bit of money for maintaining their account for exemple), while a default would cost you a proportion of the outstanding amount, which is another order of magnitude, and that proportion is often rather difficult to predict (given insurances for exemple).

I am not really aware for advanced literature for points 1) and 2), despite working in the field, I even suspect there is a complete black-out on those topics as they are the one that provide competitive advantages to a given credit lender.

3) You want to avoid social discrimination, mostly for ethic and legal purpose, which is counterproductive to statistical discrimination. There is not much literature on the topic in general, and fewer in credit scoring, except maybe : https://arxiv.org/abs/1610.02413. Depending on your jursidiction this may give you some headaches.


problem of entry format for a simple model in Keras I apologize if this question is too elementary for this site. I am new in learning Keras and Tensorflow and I have the following type/shape problem on which I have already wasted too much time.

I entered this code (found on the web) to construct a keras model using sequential()

from keras.models import Sequential
from keras.layers import Dense, Activation

model = Sequential()
model.add(Dense(32, activation='relu', input_dim=100))
model.add(Dense(1, activation='sigmoid'))
model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy'])

I then want to try the function model.evaluate(). But I can't find in the documentation nor in my trials and errors under what format the entry of evaluate should be. Among many other things, I have tried:

import numpy as np
model.evaluate(np.random.random((100,)))

but I get a long error message ending in

ValueError: Error when checking input: expected dense_1_input to have shape (100,) but got array with shape (1,)

Anyone has an idea what is happening here? Just a simple line of code
creating a dummy entry that my model could evaluate() would unstuck me, I think.

model.evaluate requires both input and output, for example

evaluation = model.evaluate(np.random.random((1, 100)), np.random.random((1, 1)))

I think a step-by-step example would be more beneficial. Here is a working example:

from keras.models import Sequential
from keras.layers import Dense
import numpy as np

N = 1000
dimension = 100

# create some random input features (x) and output (y)
np.random.seed(0)
x = np.random.random((N, dimension))
y = np.random.random((N,))

# split the data into train and test sets
split = int(0.8 * N)
x_train = x[:split]
y_train = y[:split]
x_test = x[split:]
y_test = y[split:]

# build the model architecture
model = Sequential()
model.add(Dense(32, activation='relu', input_dim=dimension))
model.add(Dense(1, activation='sigmoid'))
model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy'])

# train the model
model.fit(x_train, y_train, epochs=100)

# evaluate the model on train and test sets
train_loss = model.evaluate(x_train, y_train)[0]
test_loss = model.evaluate(x_test, y_test)[0]

print('train loss:', train_loss, ', test loss:', test_loss)

# predict (y) for a random input (x)
y_predict = model.predict(np.random.random((1, dimension)))
print('prediction:', y_predict)

which outputs binary_crossentropy loss:

train loss: 0.5500347983837127 , test loss: 0.7403841614723206
prediction: [[0.38731796]]

If you skip the training, i.e. commenting out

# train the model
model.fit(x_train, y_train, epochs=100)

the output will be

train loss: 0.7098221921920777 , test loss: 0.7191445398330688
prediction: [[0.32682237]]


Word embedding vectors for keyphrase extraction I am just interested to know if we can use the generated word embedding vectors to extract keywords from a document or not.

If yes, how?

Word embeddings are just a way to represent tokens (often words, but could be characters) in a way that it inherently carries semantic meaning (i.e. as opposed to simply one-hot encoding), but it is not a keyphrase extraction technique.

Word embeddings can help you extract key phrases better because they will make your input more meaningful, but unless you have a technique to extract such phrases, they won't help you.


Can I use regression to solve a multiple equation problem I'm working on a problem which is a multiple equation. I have a group of people and each person in the group is working on different tasks (e.g. n tasks in total). Each person in this group is working on multiple tasks and complete them. I'd like to find an estimation for the time each type of task takes.

I have equations like below:

#of days person i worked = time(task1) * #task of type 1 completed + time(task2) * #task of type 2 completed +...+ time(task n) * #task of type n completed

The only unknown in these equations are time (task i).

I have one equation per person in the group (there are 100k people). Does it make sense to use regression to find time (task i)?

Yes, linear regression is perfect for this.

If you knew that all the quantities were measured with perfect accuracy (no errors), you could solve using linear algebra; the system amounts to solving $Ax=y$, where $A$ is a matrix where each row corresponds to the number of each task for one person, and $x$ is a vector that reflects the time of each task, and $y$ is a vector that records the number of days each person worked.

If you suspect there might be small errors in the measurement of $y$, then using linear regression is appropriate.

You'll need that the number of people exceeds the number of tasks.

If you have a different error model, the optimal procedure might be something other than linear regression.  For instance, if you have errors in both the measurements of $y$ and $x$, PCA (also called orthogonal regression) might be more suitable.  If you have a more complex error model, you might want to ask on Stats.SE for the most appropriate inference procedure, or consider using stochastic gradient descent to minimize an appropriate loss function chosen based on your error model.


problems during training a MLP type of network I trained a neural network model, a MLP type of network, where the first several layers are 1-D convolution for processing sequence type of input.

However, the training process looks like as follows, where the orange line represents the validation loss and the blue line represents the training loss. The validation loss is large compared to the training loss and the training loss also  stops decreasing after the first several iterations. Are there any generic guidance to improve the performance? I have about 1 million training traces, and the number of parameters of the network is about 140K.

When the training loss is lower than the validation loss, the model is said to overfit the training data, i.e. it has learned so much from the training data that it only adjusts well to it and it can't generalize to new data. This phenomena is regarded as the variance of the model. The bias of the model is the difference between the training loss and and the loss you've previoussly selected as the minimum loss reachable, or the desired one.

However, this analysts is usually done over other well known metrics, such as precision and recall. You first calculate these metrics on your training data, and then, on the evaluation data. Then you perform the analysis taking the same considerations.

In order to reduce the variance/overfitting, there are common techniques:

Increse the training data by adding more instances, if available
If no more instances are available, perform data augmentation to increase the training dataset
Use regularization, for example, dropout.
Shorten the network. If there are lots of layers, the network may be learning too specific features from the training data

As mentioned before, I'd perform the analysis using other metrics rather than the loss.


A/B testing with non-Gaussian distributions I have two sets of samples (A, B) with a relatively high number (~10,000) and I want to see if a factor has affected sample B or not. Naturally, I should use A/B testing. The problem is, the distributions are not normal and I'm interested in the maximum change, not the mean values! So if all you know is how CLT is gonna make everything Gaussian, this is a good point to stop and move on to the next question.
The data are distances, so there's a minimum of 0, but there's no max and no guarantee what the distribution is going to look like. As an example, the histograms look like this:

My gut feeling tells me that the maximum of orange sample is just randomly higher than the blue one, but gut feelings are usually wrong. So I want to have some method of testing. I would appreciate any input.

PS: Welch's t-test tells me that with 100.000% confidence, these two distributions are different, but are they?

Welch's t-test assumes normal distribution. I'd assume your sample size is big enough to see that these two distributions are different, based on mean, variance and range differences


Comparsion between DCGAN and WGAN What is the main architectural difference between DCGAN and WGAN?  For which problems each models can be more useful than the other one?

DCGAN is more about network architecture alterations, while WGAN is an change to the loss function. There's nothing stopping you from using the DCGAN architecture with the WGAN objective function: all this means is minimizing an approximate Wasserstein loss, rather than a Jensen-Shannon divergence, using a particular network architecture. The WGAN (or its followups, e.g. WGAN-GP) objective is agnostic to the architecture. The only thing (that I can think of) that you need to watch for is the use of batch norm: DCGAN recommends putting it everywhere, but (at least for WGAN-GP) it messes with statistics of the critic regularization.


How can I make big confusion matrices easier to read? I have recently published a dataset (link) with 369 classes. I ran a couple of experiments on them to get a feeling for how difficult the classification task is. Usually, I like it if there are confusion matrices to see the type of error being made. However, a $369 \times 369$ matrix is not practical.

Is there a way to give the important information of big confusion matrices? For example, usually there are a lot of 0s which are not so interesting. Is it possible to sort the classes so that most non-zero entries are around the diagonal in order to allow showing multiple matrices which are part of the complete confusion matrix?

Here is an example for a big confusion matrix.

Examples in the Wild

Figure 6 of EMNIST looks nice:

It is easy to see where many cases are. However, those are only $26$ classes. If the whole page was used instead of only one column this could probably be 3x as many, but that would still only be $3 \cdot 26 = 78$ classes. Not even close to 369 classes of HASY or 1000 of ImageNet.

See also

My similar question on CS.stackexchange

You can apply a technique I described in my masters thesis (page 48ff) and called Confusion Matrix Ordering (CMO):

Order the columns/rows in such a way, that most errors are along the diagonal.
Split the confusion matrix into multiple blocks such that the single blocks can easily printed / viewed - and such that you can remove some of the blocks because there are to few data points.

Nice side effect: This method also automatically clusters similar classes together. Figure 5.12 of my masters thesis shows that:

You can apply confusion matrix ordering with clana


difference in between CNN and Inception v3 What is the difference in between the inception v3 and Convolutional neural network?

The Inception models are types on Convolutional Neural Networks designed by google mainly for image classification. Each new version (v1, v2, v3, etc.) marks improvements they make upon the previous architecture.

The main difference between the Inception models and regular CNNs are the inception blocks. These involve convolving the same input tensor with multiple filters and concatenating their results. Such a block is depicted in the image below.

On the contrast, regular CNNs performs a single convolution operation on each tensor.

Inception-v3 is Deep Neural Network architecture that uses inception blocks like the one I described above. It's architecture is illustrated in the figure below.

The parts where the layers "branch off" and then are merged together again are the inception blocks described previously.

You can read more about Inception-v3 here.


Interpreting Machine Learning Classification Metrics I'm trying to understand the results of a classifier I used to predict two possible classes. Here is what I get:

     Precision     Recall     F1-score     Support

 0      0.97        1.00        0.98        341091
 1      0.60        0.18        0.27        12629

 Avg    0.96        0.97        0.96        353720

If I'm trying to predict Class 1, does this model have any value? Can I improve the results of Class 1 prediction?

I assume the results you show have been evaluated according to a train-validation-test split approach. With the information you have provided, it is possible to figure your confusion matrix. It has to be something like :
label=0 and pred=0 :  330.000
label=0 and pred=1 :  4800
label=1 and pred=0 : 11.000
label=1 and pred=1 : 7200
According to this, your model has some value but and is better that what random choices could give you. Thinking of a random choice between the two classes given the distribution of the classes, you would get something close to :
label=0 and pred=0 :  328.000
label=0 and pred=1 :  12.600
label=1 and pred=0 : 12.600
label=1 and pred=1 : 484
So, I think you model has some value. Although, I think you have to confront this performance with the use-case you are working on, the features and model you are using and obviously the level of performance you think acceptable.
To increase your model performance, it exists many approaches you can try. Here are some of them :

Try other classification algorithms (Naive Bayes, Logistic regression, SVM, tree-based technics...).
Clean your input variables if there are dirty and extract insights from it. This is called features engineering.
Diagnose your results and search if you are in an underfitting or an overftitting situation. Adapt your algorithm if needed.
Apply technics relative to unbalanced classes. See more here.


Which convolution should I use? Conv2d or Conv1d I've dataset which contains dlib landmark points of the faces. I'm using keras to train a model.
The dataset shape is (length_of_dataset,68,2). I know that I've two options.

The first is using conv1d with input_shape = (68,2).
The second is using conv2d with input_shape = (1,68,2).

My question is which one is better? And why?

When using Conv2D , the input_shape does not have to be (1,68,2). The number of samples does not have anything to do with the convolution, one sample is given to the layer at each time anyway.

What changes is the number of spatial dimensions of your input that is convolved:

With Conv1D, one dimension only is used, so the convolution operates on the first axis (size 68).
With Conv2D, two dimensions are used, so the convolution operates on the two axis defining the data (size (68,2))

Therefore you have to carefully chose the filter size. For instance, if you chose a Conv2D with a filter size (4,2), it will produce the same results as a Conv1D with size (4) as it will operate fully on the second axis of data.

Finally, there is no response to what is the best method. Generally, Conv2D work well on images and Conv1D on text. Given the size of your second dimension of data, Conv2D does not seem to make a lot of sense hence Conv1D should work well.


Creating statements from a raw text Background
I have a massive text block of poorly written statements, some facts, some opinions. It's all scraped from various public internet resources and all in english.
I am now using nltk in python to try to work out some basic statements from this massive text block.
My goal is not to capture all statements, false positives are not too bad either. I would simply like to get a few probably true and simple assertions from this text block.
Examples

Bob is married to Lisa since 1983.
Lisa's cat Snowball died two years ago.
Bob and Lisa has a 30 year old son named George.

From these simple sentences, I would like to parse out some simple facts, like:
"Bob is married", "Lisa is married", "George is Lisa's son", "George is Bobs son", "Snowball is a cat"
It does not need to be exactly this.
Actual question
What is this kind of processing called? (Except for that it is a subcategory of Natural Language Processing.)
What are the simpler and the harder approaches to this problem?
How well has this problem been solved by public and proprietary algorithms?

You may want to read about the task of Relation Extraction. The most common scenario is to extract triplets (entity1, relation, entity2). Usually entity1 and entity2 are given and the list of possible relations is known. Common methods use a classifier, others can be rule-based.

With your example from the sentence "Bob is married to Lisa since 1983." the system can extract (Bob, is_married, Lisa)

A variant is Open Information Extraction which tries to extract relations but with no information specified.

There exist multiple projects on the subject such as DeepDive, IEPY or openie-standalone.

This this presentation or this review give an overview of the task.


Evaluation metrics for Decision Tree regressor and KNN regressor I have started working on the Decision Tree Regressor and KNN Regressor.

I have built the model and not sure what are the metrics needs to be considered for evaluation. As of now I have considered Root mean squared error.

Can we use $R^2$ values for Decision Tree and KNN or is it applicable only to Linear Regression Model

I have also pasted the $R^2$ and MSE values.
As per my understanding Linear regression model is stable, Decision tree is over fitting and KNN is having less error. Which model needs to be considered here?

R2 Score for train - KNN regression 0.8215942683102192
R2 Score for test - KNN regression 0.7160388084850589
Mean squared error for train - KNN regression 49.92162362176166
Mean squared error for test - KNN regression 78.30907381395349

R2 Score for train - linear regression 0.6141419744748021
R2 Score for test - linear regression 0.6117893766210736
Mean squared error for train - linear regression 107.97107771851036
Mean squared error for test - linear regression 107.0583420197463

R2 Score for train - Decision Tree regression 0.9962039204515297
R2 Score for test - Decision Tree regression 0.7866182225490949
Mean squared error for train - Decision tree regression 1.0622217832469776
Mean squared error for test  - Decision tree regression 58.84511637596899

Updating the AIC and BIC values

AIC value - Test - KNN       :   -3.8272461328797505
BIC value - Test - KNN       :   1169.4748549110836
AIC value - Test - Linear Reg:   -4.452667046616746
BIC value - Test - Linear Reg:   1250.154152783156
AIC value - Test - Decision T:   -3.2766787253336602
BIC value - Test - Decision T:   1098.4516593376377

Thank you.

Generally when ever we are trying to compare between models and to choose the best one, we go for other metrics like AIC, BIC, AUC(this is not applicable as it is used for classification algorithm) etc along with $R^2$.
Now why are they important criteria because AIC tries to select the model that most adequately describes an unknown, high dimensional reality. This means that reality is never in the set of candidate models that are being considered. On the contrary, BIC tries to find the TRUE model among the set of candidates. I find it quite odd the assumption that reality is instantiated in one of the model that the researchers built along the way. This is a real issue for BIC.
Generally we use both AIC and BIC together.
you can got through this Link to understand better on AIC, BIC and which is better but conclusion is both are important.
Now when we compare between $R^2$ and AIC value:
$R^2$ and AIC are answering two different questions. I want to keep this breezy and non-mathematical, so my statements are non-mathematical.  $R^2$ is saying something to the effect of how well your model explains the observed data.  If the model is regression and non-adjusted $R^2$ is used, then this is correct on the nose.
AIC, on the other hand, is trying to explain how well the model will predict on new data. That is, AIC is a measure of how well the  model will fit new data, not the existing data.  Lower AIC means that a model should have improved prediction.
Frequently, adding more variables decreases predictive accuracy and in that case the model with higher $R^2$ will have a higher (worse) AIC. A nice example of this is in "Introduction to Statistical Learning with R" in the chapter on regression models including 'best subset' and regularization. They do a pretty thorough analysis of the 'hitters' data set.  One can also do a thought experiment.  Imagine one is trying to predict output on the basis of some known variables.   Adding noise variables to the fit will increase $R^2$, but it will also decrease predictive power of the model.  Thus the model with noise variables will have higher $R^2$ and higher AIC.
To understand more with respect to the above explanation, you can go through this Link

I have also pasted the R2 and MSE values. As per my understanding Linear regression model is stable, Decision tree is over fitting and KNN is having less error. Which model needs to be considered here?

To answer this I think you need to derive AIC and BIC values and finally you can decide on which model to choose.
To derive the AIC value for Linear Regression can be done by using OLS, as the value of AIC is directly available.
If you want to derive, you can go through this Link
AIC:
$AIC= 2k - 2ln(sse)$
where k= number of variables
BIC:
$BIC = n * ln(sse/n) + k * ln(n)$
Do let me know if have any additional questions.


The connection between optimization and generalization Optimization algorithms such as gradient descent or particle swarm can find a minima in a function.

On the other hand, learning methods such as back-prop define learning as an optimization problem and are used to learn weights in deep neural networks and etc.

We know that DL models can learn/memorize basically anything in the training data [2], even random noise, which brings us to the question of then how we can achieve generalization (as defined in [5]) *.

In [7, 1, 3] authors tried to show the connection between generalization and the loss landscape (sharp minima/flat minima) plus its effect from the batchsize.

The effect of training data on generalization studied in [4] shows that models that have high influence from more data points generalize better than models that rely/influenced on less data points.

On the other hand [6] sheds some light on the role of the model's parameters from a neuroscience point of view in terms of effect of neurons in generalization (Networks that generalize better are harder to break with neuron deletion.).

Although all of these papers investigated generalization up to a certain point, the effect of optimization used in the learning algorithm on the generalization is not yet clear.

Is there any study done in this direction? What are the possible related work on this? Or do you have an answer to this question?

*:
Let me point out that although many believe explicit regularization is crucial for generalization, [2] already explains explicit regularization (l1/l2/dropout) does not play a big role in generalization. Many tricks that were known as a generalizer, are shown to be a myth. They also show that interestingly, SGD works as an implicit regularizer which might be a connection to the effect of optimization alg. to generalization.

[1]: Li, Hao, et al. "Visualizing the Loss Landscape of Neural Nets." arXiv preprint arXiv:1712.09913 (2017).

[2]: Zhang, Chiyuan, et al. "Understanding deep learning requires rethinking generalization." arXiv preprint arXiv:1611.03530 (2016).

[3]: Keskar, Nitish Shirish, et al. "On large-batch training for deep learning: Generalization gap and sharp minima." arXiv preprint arXiv:1609.04836 (2016).

[4]: Koh, Pang Wei, and Percy Liang. "Understanding black-box predictions via influence functions." arXiv preprint arXiv:1703.04730 (2017).

[5]: Kawaguchi, Kenji, Leslie Pack Kaelbling, and Yoshua Bengio. "Generalization in deep learning." arXiv preprint arXiv:1710.05468 (2017).

[6]: Ari S. Morcos, David G.T. Barrett, Neil C. Rabinowitz, Matthew Botvinick, "On the importance of single directions for generalization." arXiv preprint arXiv:1803.06959 (2018). https://deepmind.com/blog/understanding-deep-learning-through-neuron-deletion/

[7]: Dinh, Laurent, et al. "Sharp minima can generalize for deep nets." arXiv preprint arXiv:1703.04933 (2017).

What part the optimization alg. (Grad. Descent) plays in generalization of the learning algorithm?

Actually for generalizing you have to find a model that does not overfits the training data. For doing so, there are numerous approaches, like L1/L2 regularization which adds noise to the cost function and somehow weights to prohibit the network from having large weights which may lead to overfitting. Take a look at here. Other techniques are drop-out which adds noise to the activations to help the network not to depend on special nodes. These techniques add some noise to different parts of the network which lead to a cost function with a high error value. After finding the error back prop techniques try to set the parameters/weights to go downhill of the cost function. These techniques help the algorithm not to overfit which means the constructed model will be able to generalize, although you have to test it using cross-validation data and test data. Consequently, optimization techniques by themselves always try to reduce the amount of error and they somehow always lead to overfitting because they try to reduce the cost and this is regularization techniques and other variants that have to be used to construct a cost function which its optimum points do not lead to overfitting. This link also may help you.

  Or the influence of the data [4] plays a more important role? Although [6] sheds the light on the role of the model parameters from a neuroscience point of view in terms of the effect of neurons in generalization, the question on how to learn such models remains unsolved.

Data is the most important part of the learning process. Your data have to be representative enough. Deep learning problems are considered those problems which may have better results if their training data increases. If the model should be able to generalize, it is vital to use data from the real distribution. This is just for training. You can use some techniques in order not to overfit the data using your training data-set. It is customary to add small noises to the input to let the model generalize well. Transforming data by translation, rotation and even distorting image inputs are examples of adding noise to the input to avoid overfitting. Although changing the data is dangerous because it may change the distribution of the real population. So, you can do something with your input data to avoid overfitting and let your model to generalize well.

  So how can we really better understand the generalization considering different factors as mentioned above?

Generalization techniques are used to make the network generalize well. The customary approaches are:

Drop Out
L1/L2 Regularization
Early Stop
Adding Noise and data-augmentation

It should be considered that adding noise to the input should be without changing the distribution. The ratio of signal to noise also should not be small because the information may be lost compeletely. Depending on your problem each of them make work well but I guess there is no consensus which one is the best but in deep-learning the first one is so helpful.

  What is the connection between optimization and generalization (as defined in [5])?

What I'm saying is based on my experience. Optimization itself always lead to overfitting. You have to use generalization techniques to avoid that. To help you figure out the problem, suppose that deep-learning algorithms are able to learn all functions. If you provide them with relatively small number of data, they will memorize a hypothesis, and they do not learn the problem.


Best practices to store Python machine learning models What are the best practices to save, store, and share machine learning models?

In Python, we generally store the binary representation of the model, using pickle or joblib. Models, in my case, can be ~100Mo large. Also, joblib can save one model to multiple files unless you set compress=1 (https://stackoverflow.com/questions/33497314/sklearn-dumping-model-using-joblib-dumps-multiple-files-which-one-is-the-corre).

But then, if you want to control access rights to models, and be able to use models from different machines, what's the best way to store them?

I have a few choices:

Store them as files, and then put them in a repository using Git LFS
Store them in an SQL database as binary files:

For instance in Postgresql https://wiki.postgresql.org/wiki/BinaryFilesInDB
This is also the method recommended by the SQL Server team:

https://docs.microsoft.com/en-us/sql/advanced-analytics/tutorials/walkthrough-build-and-save-the-model
https://microsoft.github.io/sql-ml-tutorials/python/rentalprediction/step/3.html
https://blogs.technet.microsoft.com/dataplatforminsider/2016/10/17/sql-server-as-a-machine-learning-model-management-system

HDFS

You may have a look at nexus or dvc or datmo.

There was recently a presentation at at meetup in berlin, zalandos AI data engineering meetup.


Import image into Tensorflow model I trained a simple Handwriting model with Tensorflow and MNIST:
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
x_train = x_train.astype('float32')/255
x_test =  x_test.astype('float32')/255
model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28,28)),
    keras.layers.Dense(100,activation = 'relu'),
    keras.layers.Dense(100,activation = 'sigmoid'),
    keras.layers.Dense(10,activation = 'sigmoid'),
])

As you can see, I flatted my first layer into 784 px One-dimensional array.
Now on paper I wrote a number :

After changing it's scale into 28*28 with image editor(GIMP), I loaded my image into my code:
img_width, img_height = 28, 28
img = image.load_img('rgb_seven.jpeg', target_size=(img_width, img_height))
x = image.img_to_array(img)

This is x result:
array([[[167., 170., 179.],
        [168., 171., 180.],
        [168., 171., 180.],
        ...,
        [174., 175., 180.],
        [174., 175., 180.],
        [167., 170., 175.]],

       [[173., 176., 185.],
        [172., 175., 184.],
        [166., 169., 178.],

and :
images = np.vstack(x)
images

And it's images result is :
array([[167., 170., 179.],
       [168., 171., 180.],
       [168., 171., 180.],
       ...,
       [166., 173., 179.],
       [166., 173., 179.],
       [166., 173., 179.]], dtype=float32)

Before to predict I have to do flat my images so I do this:
x_images_flattened = images.reshape(len(images),28*28)

But I got error:
ValueError                                Traceback (most recent call last)
<ipython-input-30-db6c299f19c6> in <module>
      3 images = np.vstack(x)
      4 # images
----> 5 x_images_flattened = images.reshape(len(images),28*28)

ValueError: cannot reshape array of size 2352 into shape (784,784)

Why I got cannot reshape array of size 2352 into shape (784,784) my image has 28*28 size.
And how can I predict that?

By default, the image is loaded as a color Image i.e. 784*3 = 2352

Load image as grayscale i.e. use parameter color_mode="grayscale"
No need to of np.vstack(), simply reshape to (-1,784)


Python library for segmented regression (a.k.a. piecewise regression) I am looking for a Python library that can perform segmented regression (a.k.a. piecewise regression).

Example:

numpy.piecewise can do this.

  piecewise(x, condlist, funclist, *args, **kw)

  Evaluate a piecewise-defined function.

  Given a set of conditions and corresponding functions, evaluate each
      function on the input data wherever its condition is true.

An example is given on SO here. For completeness, here is an example:

from scipy import optimize
import matplotlib.pyplot as plt
import numpy as np

x = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ,11, 12, 13, 14, 15], dtype=float)
y = np.array([5, 7, 9, 11, 13, 15, 28.92, 42.81, 56.7, 70.59, 84.47, 98.36, 112.25, 126.14, 140.03])

def piecewise_linear(x, x0, y0, k1, k2):
    return np.piecewise(x, [x < x0, x >= x0], [lambda x:k1*x + y0-k1*x0, lambda x:k2*x + y0-k2*x0])

p , e = optimize.curve_fit(piecewise_linear, x, y)
xd = np.linspace(0, 15, 100)
plt.plot(x, y, "o")
plt.plot(xd, piecewise_linear(xd, *p))


dealing with dropped event data I have a set of game play event data. One of our goals is to provide a picture of how much time the person has spent playing. This sounds simple but for the fact that events are often dropped and result in vast amounts of time between events.

We are in field trials right now and don't have a huge pool of data to use as reference.

I don't have much experience in statics, though I'm a competent programmer. My question is: What might I do to account for the missing data? How do I ensure that these missing events don't skew my data?

What you have described is that you want to consider the distribution of observations in a way that models the population to serve a better purpose.

I would recommend reading into how different types of distributions can be transformed so that your model might achieve a higher score.

In terms of your plots during analysis of your dataset, I would look at resampling at different frequencies, this will change how your plots will look and allow you to identify issues with seasonality, such as breaks in the account activity.

If the observations are missing because the person simply wasn't active, then this is a good representation of reality, the question is maybe how would you increase the users engagement?


Décision tree, How to see under/over fitting with just looking at the leafs? My question is: how with just looking at the leafs of a decision tree could you tell if the model is under/over-fitting?

Any sort of advice will be helpful.

You can look at number of leaves. If number of leaves are very high compare to number of classes available for classification then tree might be overfitting or vice versa.

You can also check number/percentage of training examples correctly classify by each and every leaf. You can put some threshold on top of that so that every leaf has to classify threshold of examples correctly.

But again, cross validation and holdout method will give you better idea for overfitting or underfitting.


EGAN Paper With Confusing Notation I am reading a research paper that tries to fix some issues with GAN,  but for one of the equations, the paper does not fully explain where it comes from and why it works. Although the overall algorithm for EGAN is straight forward, I wish to understand it a little better.

The paper uses an evolutionary approach to fix stuff like instability and mode collapsing. A fitness function for evaluating the different generators is give as:

Where f_d is:

and f_q is:

What I'm focusing on is f_d = -log||delta_d - E_x[log(D(x))] - E_z[log(D(G(z)))]||.

In f_d, what does delta_d represent. It was not defined in the paper so I figured maybe it was very typical notation for something so I looked around but could not find anything. The only things I know is from what it says in the paper, and the paper that the writers referenced to create the equation (proofs >.<)

What I know right now is the entirety of f_d represents what the paper calls the diversity fitness score. I also know that the second part of the equation is the same as the loss function for a discriminator so it could be re written as f_d = -log(delta_d + loss_d). But what is delta_d? and how does this equation properly evaluate the diversity of the generated samples?

$\nabla_D$ means the gradient w.r.t. the discriminator parameters. The authors explained below as "the log gradient value of updating D".

BTW it's not standard to write something like $\nabla-a-b$ as this paper, which confuses readers as "$\nabla$ minus $a$ minus $b$". It's much better to write $-\nabla(a+b)$.


Are Undergraduate Statistics Concepts Used in Practice? I'm curious for more experienced Data Scientist, have you ever used t - test, ANOVA, Wilcoxon, etc?

Basically my question is, do you perform inference task, or purely prediction tasks? (Machine Learning)

t-test and ANOVA are used pretty often, more in statistical data analysis which is a "must know" for a data scientist but not necessarily their everyday work. More you go toward medical/bio statistics or social sciences, you see them more used.

In everyday life of a data scientist, the problem of feature selection, for example, is one of situations where ANOVA helps. For instance imagine numerical features and discrete classes in a classification problem. A way to select good features is to compare the distributions of features values for each class and see if those distributions differ significantly.

I also have used Wilcoxon test once in a low-sample size project where I was trying to recognize patterns in Complex Networks, created from time-series of two control groups. The story was to see which features of those networks (centrality measures, clustering coefficient, average path length, etc.) makes the significant difference (so can be used for recognition i.e. classification problem).

This difference was the difference between distribution of feature values among those two groups where those distributions were not necessarily Gaussian. Wilcoxon helped a lot there, as some bold difference between two groups for certain features were actually statistically not significant and without such test the analysis could go the wrong way.

Hope it helped!


Non-Compact Domain/Range Machine Learning What are some machine learning methods that can be applied to non-compact input domain and output range?

Neural Networks is clearly not one of them as they well approximate in an interval.

I have a problem at hand where I need to identify a function with domain the entire real line. The good thing is I know the asymptotic behavior, which is a finite point.

Unfortunately common ML methods diverge immediately outside of training bounds and cannot be hoped to have any asymptotic stability. Polynomials were one idea that came to my mind, but they also diverge asymptotically.

Notice that the limitation of finite domains and ranges stems from the finite nature of hardware and thus finite representation of numbers. Therefore, a more promising approach is remodeling the domain, rather than using an "infinite" algorithm.

You can apply logarithmic scaling to your domain and range. This will get you off from the open bounds while still preserving order and continuity, which is important fur numerical ML algorithms.

The log function is the basic building block that you can use for your transformation. You can scale it to fit your ranges and this would solve your positive part of the range. You can do the same for the negative, even using  different coefficients if this suits you.


How should I construct a binary classifier for small set of positive data and million of unlabeled data? Does anyone have suggestions for specific algorithm or implementation for labeled data of only one class and unlabeled data that can be from either classes? And I'm unsure what is the proportion of Class A to B that exists within the unlabeled data and also my labeled data is not randomly chosen.

This is called PU Learning, and it can be used when using a probabilistic classifier and certain assumptions are met about how the data is labeled.
If the assumptions are met, you

Label positive, already labeled instances as positive
Labeled unlabeled instances as negative
Train a probabilistic classifier.

This produces the same ranking of class probabilities as a classifier would if trained on a dataset labeled with true positive/negative labels.
This video covers the assumptions pretty well and the Elkan paper is pretty accessible.


Training an AI to play Starcraft 2 with superhuman level of performance? I'm interested in working on challenging AI problems, and after reading this article (https://deepmind.com/blog/deepmind-and-blizzard-open-starcraft-ii-ai-research-environment/) by DeepMind and Blizzard, I think that developing a robust AI capable of learning to play Starcraft 2 with superhuman level of performance (without prior knowledge or human hard-coded heuristics) would imply a huge breakthrough in AI research.

Sure I know this is an extremely challenging problem, and by no means I pretend to be the one solving it, but I think it's a challenge worth taking on nonetheless because the complexity of the decision making required is much closer to the real world and so this forces you to come up with much more robust, generalizable AI algorithms that could potentially be applied to other domains.

For instance, an AI that plays Starcraft 2 would have to be able to watch the screen, identify objects, positions, identify units moving and their trajectories, update its current knowledge of the world, make predictions, make decisions, have short term and long term goals, listen to sounds (because the game includes sounds), understand natural language (to read and understand text descriptions appearing in the screen as well), it should probably be endowed also with some sort of attention mechanism to be able to pay attention to certain regions of interest of the screen, etc. So it becomes obvious that at least one would need to know about Computer Vision, Object Recognition, Knowledge Bases, Short Term / Long Term Planning, Audio Recognition, Natural Language Processing, Visual Attention Models, etc. And obviously it would not be enough to just study each area independently, it would also be necessary to come up with ways to integrate everything into a single system.

So, does anybody know good resources with content relevant to this problem? I would appreciate any suggestions of papers, books, blogs, whatever useful resource out there (ideally state-of-the-art) which would be helpful for somebody interested in this problem.

Thanks in advance.

You pose an interesting question. The problem that I see is that even if you develop all the items you mentioned (and I'm not sure you need all of them) you wouldn't have the computing power needed for this.

You see, most of these items are based on reinforcement learning. Which means that the models are given a relatively small set of data (roughly speaking, the rules of the game) and then are set off to play millions & millions of games. Read the whitepaper that was written by Google on beating Go. They basically admit that there whole point was to write something very, very simple that could learn on it's own and then just set it free on Google's massive power. It's nothing for them to say, "let's have this robot play 10 million games". So the level of sophistication from your initial model is pretty low, you just need to make it an exceptional learner.

So you make an exceptional learner of Starcraft - then what? How are you going to have the power to make your algorithm to play millions of games?


Assigning vector of weights to Neural Network I am using nnet package in R for training Artificial Neural
Network. Some tell that weights are assigned to each case in training set,
and some say that no. of weights are equal to the no. of connections in
structure of neural network. I didn't understand this. Can anybody help me
out.

I looked at the R code of nnet i.e.

What is the difference between weights & Wts in the nnet code below?

nnet(x, y, weights, size, Wts, mask, linout = FALSE, entropy = FALSE, softmax = FALSE, censored = FALSE, skip = FALSE, rang = 0.7, decay = 0, maxit = 100, Hess = FALSE, trace = TRUE, MaxNWts = 1000, abstol = 1.0e-4, reltol = 1.0e-8, ...)

From the documentation we can find out that:

weights: (case) weights for each example – if missing defaults to 1.
Wts: initial parameter vector. If missing chosen at random.

Though I'm no R expert, my understanding is:

weights is something like sample_weight in python, which controls the weight of each sample. It's useful when dealing with imbalanced class issue.
Wts is the initial weights of neural network. Random weights will be initiated if this parameter is not passed.