The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all() using panda python enter preformatted text hereHere I have data to import from a CSV file.I wrote an equation inside the class and to solve the equation data will import from the CSV file.
When I run my code I got an error like "tuple indices must be integers or slices, not str" using panda python.
Can anyone help me to solve this problem?
I upload my code and CSV file here.

def ph_convert(time,we,h,a,w):
while time <= 30:
    level = 1.1
    level = float(level)
    if w == 1:
        ph= ((((6*we)+(1*h))/level -(4*a)))/time
    else:
        ph= ((6+((1*we)+(3*h))/level -(6 *a)))/time
    break
while time <=60:
    level = 1.25
    level = float(level)
    if w == 1:
        ph= ((((6*we)+(1*h))/level -(4*a)))/time
    else:
        ph= ((6+((1*we)+(3*h))/level -(6 *a)))/time
    break
print(ph)

data = pd.read_csv('data1.csv')
data['time'] = data['time'].apply(time_convert)
we = data['we'].astype(float)
h = data['h'].astype(float)
a = data['a'].astype(float)
w = data['w'].astype(float)
time = data['time'].astype(float)
print(ph_convert(time,we,h,a,w))

Subset of my CSV file:

we  h   a   w   time
48.1    150 53  1   6:15:00
48.1    150 53  1   9:00:00
48.1    150 53  1   9:25:00
48.1    150 53  1   9:30:00
48.1    150 53  1   11:00:00

error:

[![enter image description here][2]][2]

while time <= 30:

You are trying to evaluate the vector time, but in python, boolean evaluations with that expression can only be done over scalars. The thing is, as time has multiple values, the Python interpreter can't figure out if the condition must hold over all of them, just one, etc.

If you want to evaluate if any of the values of time, or all the values in time you can use methods like time.any() or time.all()

You can always walk through all the items in time and perform the evaluation on them:

for t in time:
    if t <= 30:
        ...


How to classify a document by image? I need an opens source solution to classify a document. I do not want to use NLP i need only to check the look and feel.

I tried OpenCV. I have a template and i need to match it.

import cv2

template = cv2.imread(template_file,0)
template = cv2.normalize(template, None, alpha=0, beta=1, norm_type=cv2.NORM_MINMAX, dtype=cv2.CV_32F)

 method = ['cv2.TM_CCOEFF_NORMED']

 img = img2.copy()
 method = eval(meth)
 # Apply template Matching
 res = cv2.matchTemplate(img,template,method)

 min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(res)
 if (max_val > threshold):
    print('match OK')

this method doesn't seems so robust and I get a lot of false positive.

You might want to look at Siamese CNNs depending on the size of your dataset. A good introduction can be found here.


What is the meaning of colors in activation maps? I am trying to understand how the neural network works. I plotted the intermediate layer activations and I got the following image.

I used the matplotlib library and selected cmap='jet'. I would like to know what is the meaning of these colors? Does the area with color red mean the filter is activated in that region? Then, what about the green, yellow and blue color means?

In heat maps, generally reddish regions represents higher values where as bluish ones represent lower.


Effect of a few wrongly scaled feature values on logistic regression model I was given a situation to predict the validity of the logistic regression model when it was found that certain values of a heavily weighted feature were found to be erroneously multiplied by 1000. For example, the feature has values 20.000, 78.000, 56.000, 10.000 and so on. However, few values were found to be missing the decimal point and were therefore scaled up by 1000.
Will the results of the previously fit model be valid in such a situation and why ? Also, how will the model be affected by the wrongly scaled up values.

In my opinion, the model will still be valid because even if the range of the predictor variable is high in the case of Logistic Regression, the algorithm itself will improvise itself by reducing the value of the "coefficient" corresponding to this variable.
You can read more about this here and here
Dhanyawad.


Normal distribution and QQ plot I have data that I plotted with a normal distribution and a QQ plot. I was wondering especially in QQ plot it seems that 95.4% of the data is normally distributed.

My question is what does numbers meanings that above 2 sigmas in QQ plot? Should remove them or I need to transform this variable and make it more normally distributed?

The normal distribution is a theoretical model of data. Empirical data can be distributed more similarily or more dissimilarly to a normal distribution.

That empirical data has a couple of notable divergences from a theoretical normal distribution:

Presence of outliers
Not symmetric
Relative "peakedness"

Depending on your goal, you can pick a better model or transform the data to fit the normal distribution model. The power-law distribution might be a better model for the data. If you want to transform the data to better fit to a normal distribution, you can drop the outliers and then apply a log transformation.


How many LSTM cells should I use? Are there any rules of thumb (or actual rules) pertaining to the minimum, maximum and "reasonable" amount of LSTM cells I should use? Specifically I am relating to BasicLSTMCell from TensorFlow and num_units property.

Please assume that I have a classification problem defined by:

t - number of time steps
n - length of input vector in each time step
m - length of output vector (number of classes)
i - number of training examples

Is it true, for example, that the number of training examples should be larger than:

4*((n+1)*m + m*m)*c

where c is number of cells? I based this on this: How to calculate the number of parameters of an LSTM network? As I understand, this should give the total number of parameters, which should be less than number of training examples.

The minimum number of training examples is what you have up there:

$$4(nm + n^2)$$

For more information refer to this article:
Refer to this link if you needed some visual help:
Number of parameters in an LSTM model

The number of units in each layer of the stack can vary. For example in translate.py from Tensorflow it can be configured to 1024, 512 or virtually any number. The best range can be found via cross validation. But I have seen both 1000 and 500 number of units in each layer of the stack. I personally have tested with smaller numbers as well.


How do tech-companies employ Random Forest on large data sets? The algorithm takes quite a long time to train on large data sets with a moderate number of parameters:

https://stats.stackexchange.com/questions/37370/random-forest-computing-time-in-r

https://stackoverflow.com/questions/34997134/random-forest-tuning-tree-depth-and-number-of-trees

https://stackoverflow.com/questions/31278688/how-can-you-reduce-the-default-ntree-500-parameter-passed-to-rf-from-caret

I've been trying to run it on a ~25,000 row data set with 36 predictors and it has been using 6GB of RAM for over 2 hours.

Are there instances where this algorithm is used in production or is being run daily? If so, how does one approach re-training it or optimising it for large data sets?

For the Random Forests algorithm, the time complexity for building a complete un-pruned tree is $O(m.n\log(n))$, where $n$ is the number of records/instances and $m$ is the number of variables. The algorithm is embarrassingly parallel so in many cases companies with available resources will simply use sufficient compute nodes to enable the model to run in a time that they consider reasonable.


RL Sutton book, initial estimate of q*(a) for 10 arm testbed The Sutton book does not mention what the initial estimate is for q*(a) before the first reward is received.  In this code repo that seems to go along with the book:  Sutton code repo

They have initialized it with 0 per snippet below:

def __init__(self, kArm=10, epsilon=0., initial=0., stepSize=0.1, sampleAverages=False, UCBParam=None,
                 gradient=False, gradientBaseline=False, trueReward=0.):

But the explanation for Figure 2.1 that shows the distribution of rewards for the 10 arms of the bandit says,

  Figure 2.1: An example bandit problem from the 10-armed testbed. The
  true value q ⇤ (a) of each of the ten actions was selected according
  to a normal distribution with mean zero and unit variance, and then
  the actual rewards were selected according to a mean q ⇤ (a) unit
  variance normal distribution, as suggested by these gray
  distributions.

So should I initialize instead with np.random.randn()?

Edit:  The distribution

The description you quote explains how the true values will be set in the test when setting up a test run. This is necessary to fully state how the test works.

Initialisation of your estimates is a different issue. If you know something about the distributions of the true action values, then it would make sense to use that. For instance you could set all action values to the mean expected true value. Which is $0$. However, you may also use $0$ if you have no idea about true values, as it is a simple arbitrary value.

Setting the estimates to something using the same distribution is not unreasonable, as it that itself is an unbiased estimate of the same mean. However, it does not really serve you well here, because it adds variance to the initial estimates (as well as possibilty of them being closer to true values, they are equally likely to be worse) and on average slowing down some agent types slightly.


How to train my model efficiently? I am new to ML and have been reading online about training bottlenecks when there are frequent updates to data.

Let's say I have a built a model based on a dataset of 10M records.

Now, in another 2 months, I might receive another 1M records which we would like to feed into our model as well.

Similarly this goes on for every 2 months. We would like to update/train our model with latest data as and when it's available

1) But for example, let's say the training takes 1 week for every new data update

2) Any suggestions on how we can minimize the training time (when we train every 2 months?)?

3) Should we select a representative sample from 1 Million datapoints? Is that good enough

4) I understand it's all about tradeoff but I am curious to know whether I am missing any known approaches to save training time? I am thinking representative sample can reduce the sample size and help us fasten the training process

Can you share your suggestions on this?

From what I understand - your problem is "sample selection bias" problem. Any kind of pattern to select a subset out of large data may lead to bias. This raises two question.

How to choose? Random/stratified random (If you have multiple classes) under sampling to obtain a smaller subset.
How big to choose? we can set percentage of undersampling.

Reducing training time : We can perform preprocessing : dimensionality reduction techniques to remove correlated data by aplying techniques like- PCA. Or Sparse reconstruction methods - Transform data to sparse data and then process.


SKLearn - Understanding Discrepancy Between LogisticRegressionCV classification_report and scores_ Cross-posting from Stack Overflow:
I'm running into a weird situation where my sklearn LogisticRegressionCV model is apparently getting 100% accuracy (the lack of shuffling is intentional). However, when I ask the model to report its accuracy scores, the accuracy scores per inverse regularization are much lower than 100%. What am I doing wrong?
The model:

    decoder = LogisticRegressionCV(
        penalty='l1',                       # want sparse parameters
        cv=KFold(n_splits=5, shuffle=False),
        Cs=np.logspace(-1, 2, 3),
        random_state=0,                     # reproducibility
        dual=False,                         # Prefer dual=False when n_samples > n_features.
        solver='liblinear',                 # for l1 penalty, must use liblinear or saga
        max_iter=1e5,                       # give plenty of time to get good performance
        fit_intercept=False,
        class_weight='balanced',
    ).fit(X, y)

    y_hat = decoder.predict(X)

    y_hat_probs = decoder.predict_proba(X)

    print(classification_report(y, y_hat))
    print(confusion_matrix(y, y_hat))

outputs
              precision    recall  f1-score   support

         0.0       1.00      1.00      1.00       374
         1.0       1.00      1.00      1.00       105

    accuracy                           1.00       479
   macro avg       1.00      1.00      1.00       479
weighted avg       1.00      1.00      1.00       479

[[374   0]
 [  0 105]]

Asking for decoder.scores_ yields values nowhere near 100%:
{1.0: array([[0.35416667, 0.60416667, 0.61458333],
        [0.64583333, 0.64583333, 0.65625   ],
        [0.59375   , 0.65625   , 0.64583333],
        [0.54166667, 0.57291667, 0.59375   ],
        [0.17894737, 0.61052632, 0.65263158]])}

Why is there this discrepancy?

You are training and evaluate on the same data.
You should split data to train set(to train) and test set(to evaluate) to get proper results.
Kinda:

decoder.fit(X_train, y_train)
y_hat = decoder.predict(X_test)

print(classification_report(y_test, y_hat))
print(confusion_matrix(y_test, y_hat))


How imagenet mean and std derived? To use pre-trained models it is a preferred practice to normalize the input images with imagenet standards.
mean=[0.485, 0.456, 0.406] and std=[0.229, 0.224, 0.225].
How are these parameters derived?

These are calculated based on millions of images of ImageNet.

Ref - SO
Ref - MachinelearningMastery


Good performance metrics for multiclass classification problem besides accuracy? I am trying to solve a multiclass classification problem. The dataset is balanced. I have been using accuracy as a performace metric till now. Are there any other good performance metrics for this task?

I already know about precision and recall but as far as I know they are used when the dataset is imbalanced.

If you can use python I suggest PyCM module. A vast variety of performance evaluation parameters is in access by this module and also you can use its documentation if you want to implement it by yourself.

There is an example of it:

>>> from pycm import *
>>> y_actu = [2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2] # or y_actu = numpy.array([2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2])
>>> y_pred = [0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2] # or y_pred = numpy.array([0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2])
>>> cm = ConfusionMatrix(actual_vector=y_actu, predict_vector=y_pred) # Create CM From Data
>>> cm.classes
[0, 1, 2]
>>> cm.table
{0: {0: 3, 1: 0, 2: 0}, 1: {0: 0, 1: 1, 2: 2}, 2: {0: 2, 1: 1, 2: 3}}
>>> print(cm)
Predict          0        1        2
Actual
0                3        0        0
1                0        1        2
2                2        1        3

Overall Statistics :

95% CI                                                           (0.30439,0.86228)
Bennett_S                                                        0.375
Chi-Squared                                                      6.6
Chi-Squared DF                                                   4
Conditional Entropy                                              0.95915
Cramer_V                                                         0.5244
Cross Entropy                                                    1.59352
Gwet_AC1                                                         0.38931
Joint Entropy                                                    2.45915
KL Divergence                                                    0.09352
Kappa                                                            0.35484
Kappa 95% CI                                                     (-0.07708,0.78675)
Kappa No Prevalence                                              0.16667
Kappa Standard Error                                             0.22036
Kappa Unbiased                                                   0.34426
Lambda A                                                         0.16667
Lambda B                                                         0.42857
Mutual Information                                               0.52421
Overall_ACC                                                      0.58333
Overall_RACC                                                     0.35417
Overall_RACCU                                                    0.36458
PPV_Macro                                                        0.56667
PPV_Micro                                                        0.58333
Phi-Squared                                                      0.55
Reference Entropy                                                1.5
Response Entropy                                                 1.48336
Scott_PI                                                         0.34426
Standard Error                                                   0.14232
Strength_Of_Agreement(Altman)                                    Fair
Strength_Of_Agreement(Cicchetti)                                 Poor
Strength_Of_Agreement(Fleiss)                                    Poor
Strength_Of_Agreement(Landis and Koch)                           Fair
TPR_Macro                                                        0.61111
TPR_Micro                                                        0.58333

Class Statistics :

Classes                                                          0                       1                       2
ACC(Accuracy)                                                    0.83333                 0.75                    0.58333
BM(Informedness or bookmaker informedness)                       0.77778                 0.22222                 0.16667
DOR(Diagnostic odds ratio)                                       None                    4.0                     2.0
ERR(Error rate)                                                  0.16667                 0.25                    0.41667
F0.5(F0.5 score)                                                 0.65217                 0.45455                 0.57692
F1(F1 score - harmonic mean of precision and sensitivity)        0.75                    0.4                     0.54545
F2(F2 score)                                                     0.88235                 0.35714                 0.51724
FDR(False discovery rate)                                        0.4                     0.5                     0.4
FN(False negative/miss/type 2 error)                             0                       2                       3
FNR(Miss rate or false negative rate)                            0.0                     0.66667                 0.5
FOR(False omission rate)                                         0.0                     0.2                     0.42857
FP(False positive/type 1 error/false alarm)                      2                       1                       2
FPR(Fall-out or false positive rate)                             0.22222                 0.11111                 0.33333
G(G-measure geometric mean of precision and sensitivity)         0.7746                  0.40825                 0.54772
LR+(Positive likelihood ratio)                                   4.5                     3.0                     1.5
LR-(Negative likelihood ratio)                                   0.0                     0.75                    0.75
MCC(Matthews correlation coefficient)                            0.68313                 0.2582                  0.16903
MK(Markedness)                                                   0.6                     0.3                     0.17143
N(Condition negative)                                            9                       9                       6
NPV(Negative predictive value)                                   1.0                     0.8                     0.57143
P(Condition positive)                                            3                       3                       6
POP(Population)                                                  12                      12                      12
PPV(Precision or positive predictive value)                      0.6                     0.5                     0.6
PRE(Prevalence)                                                  0.25                    0.25                    0.5
RACC(Random accuracy)                                            0.10417                 0.04167                 0.20833
RACCU(Random accuracy unbiased)                                  0.11111                 0.0434                  0.21007
TN(True negative/correct rejection)                              7                       8                       4
TNR(Specificity or true negative rate)                           0.77778                 0.88889                 0.66667
TON(Test outcome negative)                                       7                       10                      7
TOP(Test outcome positive)                                       5                       2                       5
TP(True positive/hit)                                            3                       1                       3
TPR(Sensitivity, recall, hit rate, or true positive rate)        1.0                     0.33333                 0.5

>>> cm.matrix()
Predict          0        1        2
Actual
0                3        0        0
1                0        1        2
2                2        1        3

>>> cm.normalized_matrix()
Predict          0              1              2
Actual
0                1.0            0.0            0.0
1                0.0            0.33333        0.66667
2                0.33333        0.16667        0.5


How can I arrange data in different columns (for different constituents) from one? I have a data file that has all the values for various constituents at different dates in same column. I want different constituents in different columns. Following is the example data format. I want Aluminium, Berrylium and other variables in that column to be in different columns. The data file is attached.

SITE_NAME   SAMP_DATE_TIME      SAMPLE      Value   Units   METHOD_CATEGORY
SITE-7-1    10/04/1988 00:00:00 Aluminum    150     ug/L    INORGANIC
SITE-7-1    10/04/1988 00:00:00 Aluminum    150     ug/L    INORGANIC
SITE-7-1    10/04/1988 00:00:00 Beryllium   5       ug/L    INORGANIC
SITE-7-1    10/04/1988 00:00:00 Beryllium   5       ug/L    INORGANIC

Data : https://drive.google.com/open?id=1SUhYd21U5iQmilABNwoaIXLfsc7Mo6dW

The output I am trying to get is:

SITE_NAME   SAMP_DATE_TIME  Aluminum    Beryllium   Bromide Bromomethane    Chloride    Hexachlorobenzene   Isosafrole  Zinc
SITE-7-1    10/4/1988 0:00   150        5           10      7250            10          10                  75
SITE-7-1    12/29/1988 0:00  150        5           10      8100            10          10                  16
SITE-7-1    5/10/1989 0:00   150        5           1000    10              8100        10                  10          16

It is passing 6 days from your question and I didn't want to waste your chance of getting an appropriate response by answering. So, my solution might be naive.

Step 1: Changing the categorical data of column Sample to numerical

dictionary = {'SITE_NAME':['SITE_7_1', 'SITE_7_1','SITE_7_1'],
'SAMP_DATE_TIME':['10/04/1988 00:00:00', '10/04/1988 00:00:00', ' 10/04/1988 00:00:00'],
 'SAMPLE':['Aluminium','Aluminium','Beryllium'],
 'Value':[150, 150,5]}
import pandas as pd
dataset = pd.DataFrame(dictionary)
X = dataset.iloc[:,:].values

from sklearn.preprocessing import LabelEncoder
labelencoder_X = LabelEncoder() #Creating an object from LabelEncoder class
X[:,2] =labelencoder_X.fit_transform(X[:,2])

Result so far:

array([['SITE_7_1', '10/04/1988 00:00:00', 0, 150],
       ['SITE_7_1', '10/04/1988 00:00:00', 0, 150],
       ['SITE_7_1', ' 10/04/1988 00:00:00', 1, 5]], dtype=object)

Step 2 Adding the dummy variables as columns

dummies = pd.get_dummies(dataset['SAMPLE'])
df = pd.concat([dataset, dummies], axis=1)  #concating the dataset and the dummies

The result so far:

  SITE_NAME        SAMP_DATE_TIME     SAMPLE  Value  Aluminium  Beryllium
0  SITE_7_1   10/04/1988 00:00:00  Aluminium    150          1          0
1  SITE_7_1   10/04/1988 00:00:00  Aluminium    150          1          0
2  SITE_7_1   10/04/1988 00:00:00  Beryllium      5          0          1

Step 3: filling the dataframe

The logic for filling the values in their corresponding columns is as follows :

for i in range(len(df)):
    if df.iloc[i]['SAMPLE'] == 'Aluminium':
         df.at[i, 'Aluminium'] = df.iloc[i]['Value']

If we want to apply this to the whole dataset, we can change that Aluminium each time for different samples
After this, you can groupby your dataframe to have the rows based on SITE_NAME or whatever you want.
feel free to ask if there is any vague point here.

Filling in the whole table

samples = list(df['SAMPLE'].unique()) #storing all the unique values of the SAMPLE column

Creating a dictionary of samples as keys and their values as values:

dictionary = {}
df_new = df.groupby(['SAMPLE'], as_index=False).mean() #dataframe in which `SAMPLE` is grouped.
for i in range(len(df_new)):
    dictionary[df_new.iloc[i]['SAMPLE']] = df_new.iloc[i]['Value']

now dictionary equals:
{'Aluminium': 150, 'Beryllium': 5}
Now we can fill in the table:

for sample in samples:
    for i in range(len(df)):
        if df.iloc[i][str(sample)] == 1:
            df.at[i, str(sample)] = dictionary[str(sample)]


Why does the votingClassifier in sklearn.ensemble gives higher accuracy than each of the classifiers that make it? and should i depend on it? I am running an ML classifier on my data.
I used SVM, RF and KNN. I used GScv for each of them and then used votingclassifier.The accuracy i got in each classifier independently was low, but from the hard and soft vote of the voting classifier is much higher! Why is that?
Here is my code
Fitting Kernel SVM to training set
from sklearn.svm import SVC
clf = SVC(C=1, cache_size=200, class_weight=None, coef0=0.0,
  decision_function_shape='ovr', degree=3, gamma=0.005, kernel='rbf',
  max_iter=-1, probability=True, random_state=1, shrinking=True,
  tol=0.001, verbose=False)

Checking accurancy, Best score of GV
print('Checking accurancy, Best score of GV')
best_accuracy =  grid_search.best_score_
print (best_accuracy)

[out] Checking accurancy, Best score of GV
0.5169491525423728

Applying RandomForest (RF) Classification
from sklearn.ensemble import RandomForestClassifier

clf_rf = RandomForestClassifier(random_state=42)
clf_rf.fit(X_train, y_train)
clf_rf.score(X_test, y_test)
print ("Mean accuracy is", (clf_rf.score(X_test, y_test)))

y_pred2 = clf_rf.predict (X_test)

from sklearn.metrics import accuracy_score
accuracy_score(y_test, y_pred2)

[out] accuracy 0.21

Applying KNN
from sklearn.neighbors import KNeighborsClassifier

neigh = KNeighborsClassifier(n_neighbors=10)
neigh.fit(X_train, y_train)

print (neigh.score(X_train, y_train))

[out] accuracy 0.58

Predicting the test set results
y_pred3 = neigh.predict (X_test)

from sklearn.ensemble import VotingClassifier

eclf1 = VotingClassifier (estimators= [('svm', clf), ('rf', clf_rf), ('KNN', neigh)], voting = 'hard')

eclf1 = eclf1.fit(X_train, y_train)

print 'Hard vote accuracy =', eclf1.score(X_train, y_train)

eclf2 = VotingClassifier (estimators= [('svm', clf), ('rf', clf_rf), ('KNN', neigh)], voting = 'soft')

eclf2 = eclf2.fit(X_train, y_train)

print 'Soft vote accuracy =', eclf2.score(X_train, y_train)

[out]
Hard vote accuracy = 0.9830508474576272
Soft vote accuracy = 0.7711864406779662

Basically I would say that your individual classifiers might be overfitting the training data. Ensemble voting is known to avoid overfitting, so it does not seem unlikely that the ensemble classifier behaves better than each classifier individually.


Extracting name, date and total from a set of heterogeneous receipts So, this is how the problem goes: I am trying to extract information from scanned receipts like this,

What I have been told is that I would get the textual data from a OCR software, so in short I will be working with a textual version of the image directly.

Problem:

The problem in hand here is that I have to extract certain information here, namely,

The Location. (for eg: New York, United States)
The complete Total (after all the discounts, tips, etc. have been involved) (Eg: 1033.42)
The Currency. ($, £, €, etc)
The Date. (Easier to guess)

The reason I want to extract the Location information is that, if for example, the currency is not explicitly mentioned here, then I can infer from the location where the receipt is generated.

Challenges:

The challenge here is that, information like Total Due could be anything like Grand Total or Total (only), or something semantically similar to Total, as I am not going to get the same receipt from the same restaurant only. (The restaurant could be anywhere in the world, but the problem is limited to only English speaking countries for now.)

The other challenge is to actually get the total information. It's very easy for us to see that 1033.42 is the total in the above receipt. But how do I get the software to know that? The way I see it, is that 1033.42 is near to the total(proximity). But there could be other numbers near to it as well.

Where I have tried and failed:

I have been told to start from NLTK(NER) but NER doesn't work for everything here. I can get the date information through it but the problem isn't just only to identify what the named entities are, imo.

What I think would work

The way I see it, I think I need to use a machine/deep learning model where the machine would be able to understand the proximity match between anything which semantically says Total and the number near to it (most probably on the right side).

Any help regarding which model would work best in terms of speed(first and foremost) and accuracy would be greatly appreciated.
I would also appreciate any help regarding where I could find any dataset or existing model which I could use for Transfer Learning.

There is already an ML engine that does these extractions, here is a general process layout:

Hre is the orginal paper describing the architecture, features, approaches etc. read it up, instead of me copying it here. cloudscan


Converting non-numeric data values into equivalent rank scores Consider a data-frame similar to  the one shown (the actual data-frame is much larger)

ID EDUCATION   OCCUPATION      BINARY_VAR
1  Undergrad   Student              1
2  Grad        Business Owner       1
3  Undergrad   Unemployed           0
4  PhD         Other                1

The final objective is to apply PRIDIT scoring on individual profiles (ID) based on discrete "rank" scores of the individual values in the cell. These ranks can be thought of as indicator variables which will be used to collectively rate any $ID_i$

So for example, the ranks could signify the possibility of some $ID_i$ committing fraud:

1 : Low
2 : Medium
3 : High

The variable, BINARY_VAR is something like a "training variable" or rather, a "predictor variable", such that

$Var = 0:$ Fraud

$Var = 1:$ Non-fraud

By this reasoning, an unemployed Undergraduate would be a Rank 3 profile.

In order to apply PRIDIT, I must first convert the non-numeric variables into scores or levels.

The way it is currently being done is by applying correspondence analysis on each column against BINARY_VAR and then calculating the distance of the column contribution scores from row contribution score for non-fraud.

Row and column scores look something like this (respectively):

            CONTR
0           1.654
1           98.346
------------------------------
                  CONTR
Undergraduate     2.803602e-04
Graduate          3.147824e+00
PhD               9.176451e+00
Other             1.179664e+01

The obtained distance (supposedly) gives the required score for the level, which is written back to the data-frame as a rank (higher value resulting in a higher rank).

My main concerns about this technique are:

The data-frame is really large, and resources are limited - it is a computationally expensive method.
It involves a lot of steps, and the result of the scoring can not really be verified (can it?).

My questions are:

Does this technique seem viable?
What are betters way to assign "ranks" to non-numeric variables?

To me the approach looks overcomplicated. If you're not limited to that one algorithm use one-hot encoding and try out various classifiers. Many of them can predict probability which you can use to calculate the ranks.


How do i convert a Named num [1:4] to a data frame in R? newbie to r, taking The R Programming Environment from coursera. one of the assignments is to select some columns from a data frame and find the means.

the code below seems to get the correct answer, but the answer should be a data frame.

wc_2 <- worldcup %>%
  select(Time, Passes, Tackles, Saves) %>%
    colMeans()

How do i convert this to a data frame? i tried: wc_2<-as.data.frame(wc_2) but that gets it column wise. i do not see any way to pass the rows and columns or do a transpose in what i assume is a data frame constructor.

perhaps this can be done as part of the pipe?

thanks

edit: i got it to work:

means <- worldcup %>%
  select(Time, Passes, Tackles, Saves) %>%
    colMeans()
wc_2<-data.frame(t(matrix(means)))
colnames(wc_2)<-names(means)

My preferred way to transpose a data.frame (or data.table) is to use the transpose function found in the data.table package.

It means you might have to install it: install.packages("data.table").
This give you a function that will do what you want. Here is a demo how to use it:

library(data.table)    # makes the transpose function available

col_names <- colnames(worldcup)      # keep track of original column names
wc_2 <- colMeans(worldcup)           # compute the means
wc_2 <- transpose(as.data.frame(wc_2))              # this gives you generic column names
colnames(wc_2) <- col_names          # reapply the column names

Or combining it with your example (not fully tested):

library(magrittr)    # to import the pipe operator: %>%

wc_2 <- worldcup %>%
  select(Time, Passes, Tackles, Saves) %>%
    colMeans() %>% as.data.frame() %>% transpose()       # you might need to put a dot (`.`) in the empty brackets to pass the argument before the pipe operator

then maybe add the names you would like:

colnames(wc_2) <- col_names

If you like the sound of the package, I recommend going through the mini intro, built in to the package:

install.packages("data.table")        # install it
library(data.table)                   # load it
example(data.table)                   # run the examples section of ?


TF-IDF not a strong measure in this senario? I am dealing with a data where I have only two documents and there are some words which are present in both. Now Term Frequencies (tf) of these words are very high for respective single document than the other. For e.g.

Word1 is present in Documents D1 and D2, and
tf(Word1,D1) = 1000
tf(Word1,D2) = 3
But since Word1 is present in both the documents
IDF(Word1) = 0
TF-IDF(Word1,d) = 0 for all d belonging to {D1,D2}

So inspite having a very powerful presence in a single document, TF-IDF score will always be 0. One solution I could think is to take word1 as absent if tf(Word1) < threshold. However I still don't feel this is good enough as the Word present in only one single document is given IDF score of only 0.5. I feel like TF-IDF is not a good measurement when number of documents are very low. Any suggestions here?

Weighting scheme 2 in table Recommended TF-IDF weighting schemes in tf–idf | Wikipedia should solve your problem.


Are there any rules regarding correlation between the variables used in a CNN? I was wondering if there are any rules regarding correlation between the predictor variables or between the predictor and outcomes variables of a CNN. What should be the value of correlation? Does it matter when working with CNN?
I have the case of using two categorical predictor variables and

In theory, convolutional and pooling layers should be able to account for this kind of collinearity. Convolutional filters will learn to extract the relevant information from your pixel data, and pooling layers significantly reduce their dimensionality, reducing multicollinearity as a side effect.

However, if you think multicollinearity is still a problem, you can apply some dimensionality reduction techniques. They work on image data as well!


LSTM future steps prediction with shifted y_train relatively to X_train I'm trying to predict simple one feature time series data with shifted train data. The source looks like this:
   DATE              PRICE
0  1987-05-20        18.63
1  1987-05-21        18.45
2  1987-05-22        18.55
3  1987-05-25        18.60
4  1987-05-26        18.63

Actual code with data link download gisted here: gist
So the main problem is that it actually can't predict next steps. Roughly speaking: y_train "shifted" relatively to X_train by timesteps defined in parameters. So we getting for X_train and y_train something like this:
timesteps = 5
data = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20]
# After manipulations which you can find in gist we getting this:
X_train =
    [[ 1  2  3  4  5]
     [ 2  3  4  5  6]
     [ 3  4  5  6  7]
     [ 4  5  6  7  8]
     [ 5  6  7  8  9]
     [ 6  7  8  9 10]
     [ 7  8  9 10 11]
     [ 8  9 10 11 12]
     [ 9 10 11 12 13]
     [10 11 12 13 14]]
y_train =
    [[ 6  7  8  9 10]
     [ 7  8  9 10 11]
     [ 8  9 10 11 12]
     [ 9 10 11 12 13]
     [10 11 12 13 14]
     [11 12 13 14 15]
     [12 13 14 15 16]
     [13 14 15 16 17]
     [14 15 16 17 18]
     [15 16 17 18 19]]

So it is fair to assume that after training LSTM model with X_train (as input) and y_train (as output) we getting model which able to forecast n timesteps ahead. BUT I encountered a problem that trained model not predicting anything - only "duplicates" X_test data. For the convenience
I rebuild X_test data and plot it with y_test data which returns from model.predict():

So this is result which also contains 'Dataset prices' (pure data from dataset[upper_train + timesteps:]) for clarity.
I can not find where I made a mistake (or maybe this approach is bad?) so I will be grateful for any help!

Unfortunately it is more likely that this approach itself is bad. It's not the fault of your LSTM or neural netowrk.

You may be able to find a lot of online tutorials using RNN/LSTM to predict stock price/crude oil price/bitcoin price or whatever price, based on its price history, and their results are almost surely meaningless.

Note that I am not saying analyzing a market price is meaningless, but predicting future prices based only on historical prices only is usually meaningless, given that the market is well established and most relevant information influencing its price is readily available; or in other words, the market is efficient.

Outside of the domain of data science, the assumptions for the efficient-market hypothesis are not entirely true for most markets, but its consequence is close to the truth for large markets: the asset price follows a random walk. In terms of prediction into the future, the price behaves as a martingale, i.e. the best prediction for tomorrow's price is today's price.


How to choose solution - Neural Neworks or Scikit-Learn/Numpy/Pandas? I am trying to solve a problem - categorising and routing service desk emails to concerned teams for resolution.
Created and tested a model using Scikit-Learn, Numpy and Pandas. - Tokenized the email subject and body, used Bag-of-Words/TfIdf, and applied ML algos like - SVMs, Random Forest, Linear Classification, etc.

Now, as I read more came across NLP and Neural Networks, Keras, PyTorch, Tensorflow, etc.

May I ask how to choose the correct tool or solution for my particular problem? Please advise.

There is no magic solution. You will have to try. The rule of thumb I would say is to try firstly classifiers which are less costly to train and deploy (i.e. Scikit-learn) then give it a try with NNs.

Anyways, for your problem, you may be interested to try the Genism library for topic modelling to extract what kind of topics are mentioned in an email.

Then spaCy is a great tool for NLP production tasks, there should be some text classification walkthroughs using it.

These are only a couple of ideas I am sure there are tons more. I believe the key thing is to understand which key indicators, features, words are responsible for an email to be labelled of a particular class.


Semantic networks: word2vec? I have some doubts on how to represent the relationships between words in texts.
Let’s suppose I have two sentences like these:
Angela Merkel is a German politician who has been Chancellor of Germany since 2005.

What I would expect is a connection between name Angela and Merkel (Angela is the name, Merkel the surname) who is German and politician and Chancellor.
I read about the use of word2vec to determine the semantic structure of a sentence.
My question is therefore if this model can allow me to determine such semantic structure or if another method would be better.

There are a few models that are trained to analyse a sentence and classify each token (or recognise dependencies between words).

Part of speech tagging (POS) models assign to each word its function (noun, verb, ...) - have a look at this link

Dependency parsing (DP) models will recognize which words go together (in this case Angela and Merkel for instance) - check this out

Named entity recognition (NER) models will for instance say that "Angela Merkel" is a person, "Germany" is a country ... - another link


What did Geoffrey Hinton mean when he said this? I was going through some lectures from the Deep Learning course that Geoffrey Hinton taught on Coursera and I came across this statement:

"RNNs could potentially learn to implement lots of small programs that each capture a nugget of knowledge and run in parallel, interacting to produce very complicated effects."

I have no idea what he meant nor seen/figured out any examples of this idea (and I'm sure other people are wondering too.) So if somebody would care to demonstrate, it would be much appreciated.

Creating a model is having the computer write a small program. If you have a multi-layer network, then multiple nuggets might be interacting to produce complicated results.

That its pretty much how things like face detection work - you start with a series of really dumb "detectors" that look like basic rectangles and they combine in ways so that at the top, you're recognizing Brad Pitt.


Stacked Bar Chart in R I wondered if someone could help with a re-labelling 'fill' variables ' and the x-axis on a stacked bar chart please?
My ambition is to:

Express the fill variable(s) as of six domains in the health survey, titled ' MR, Mob, SC, Act, Pain, Anx')
And have the x-variable representing the 'health states' (0,1, 2, 3,4, 5), rather than X0, X1....X5

library(ggplot2)
library(dplyr)
library(reshape2)
library(viridis)
library(hrbthemes)
library(readxl)

data <- read.table(text = "0    1   2   3   4   5
MR  155 211 64  14  1   1
Mob 0   393 51  2   0   0
SC  0   427 12  7   0   0
Act 0   386 45  15  0   0
Pain    0   379 62  5   0   0
Anx 0   355 73  18  0   0", header = TRUE)

data$row <- seq_len(nrow(data))
data2 <- melt(data, id.vars ="row")

ggplot(data2, aes(x = variable, y = value, fill = row)) +
  geom_bar(stat = "identity") +
  xlab("\nHealthState") +
  ylab("Count\n") +
  theme_bw()

Would appreciate all feedback.

Thank you for providing a good reproducible code, I can't resist answering the question :)
Hope the following code helps:
data <- read.table(text = "0    1   2   3   4   5
MR  155 211 64  14  1   1
Mob 0   393 51  2   0   0
SC  0   427 12  7   0   0
Act 0   386 45  15  0   0
Pain    0   379 62  5   0   0
Anx 0   355 73  18  0   0", header = TRUE)

#data$row <- seq_len(nrow(data))
data$row <- row.names(data)
data2 <- melt(data, id.vars ="row")
data2$variable <- sub("X",as.character(data2$variable),replacement="",fixed=TRUE)

ggplot(data2, aes(x = variable, y = value, fill = row)) +
  geom_bar(stat = "identity") +
  xlab("HealthState") +
  ylab("Count") +
  theme_bw()


Time-series classification using convolutional neural networks I am working with time-series data. How can we use convolutional neural networks (CNN) for time-series classificaiton?

For instance, using 1-D convolutions:
https://blog.goodaudience.com/introduction-to-1d-convolutional-neural-networks-in-keras-for-time-sequences-3a7ff801a2cf


How Do I Classify Dirty/Dusty Camera Images? I would like to test (predict) whether camera images are dusty/dirty or not.; a classification task indeed.

The question is about finding the most appropriate Machine Learning approach, perhaps any suggestions for possible Deep Learning methods specifically tailored for this particular problem?

What kind of distortion does dirt/dust produce? If it is somewhat regular, for example more or less regularly sized speckles (greyish circles) it might be worthwhile to try to classify the logarithm of the real part of the fourier transform of the picture, instead of the original picture.

Also, knowing the physical properties of the distortion might guide you towards devising simple heuristics (for example based on some frequencies in the fuorier spectrum) that are as good as a ml-based approach.


How to make a neural network output a specific number out of a certain range? I have a neural network with an Input layer, 2 hidden Dense layers and an Output layer.
I would like for each neuron in the Output layer to give me a number between 0 and 2 (either 0, 1 or 2), like so:

If I use a neuron for each possibility (a neuron for 0, a neuron for 1 and another for 2) and then pick the one with the best prediction, the output layer length would be far too much.
Is there a way to implement this ? (I am fairly new to neural networks and the like)

There are two ways to do that:

Scale your output data from [0, 2] to [0, 1] and apply Sigmoid activation at the end.

Make your own custom activation function that output everything in [0, 2]

I strongly suggest you no. 1, it's way faster to implement.


How to use correlation matrix when the dataset contains multiple columns with text data? How to use it with Amazon fine food reviews dataset?

The problem is that the correlation matrix has to be done with numerical values. So what you have to do is to transform the texts into numerical vectors. There are several ways of doing this, there are libraries like gensim that can make implementation easier.

Bag-of-words
A vector of the most frequently used words across all texts is created. Then for each text sample the occurrence of each word in that sample is counted.

"John likes to watch movies Mary likes movies too"

Would give

{"John":1,"likes":2,"to":1,"watch":1,"movies":2,"Mary":1,"too":1, ... other words in corpus}

And then the values of this could be used to calculate the correlation matrix

[1,2,1,1,2,1,1, ...]

This method can also be improved by using TF-IDF.

Word embeddings and averaging
Word embeddings are when a word is mapped to vectors that try to encode the information in a word. These can trained from scratch for the task at hand, but there are also pretrained word embeddings that can input a word and output a embedding vector. To get a single vector from an entire text the average word embedding can be used.

Doc2Vec
Doc2Vec expands on word embeddings and is speficially used for embedding entire chunks of texts.


Computing Jaccard Similarity between two documents Data Mining:

Compute the Jaccard similarity of D1 and D2 on 2-shingles is. Sim(D1,D2) =

D1 = the quick brown fox jumps over the lazy dog

D2 = jeﬀ typed the quick brown dog jumps over the lazy fox by mistake

Could you explain how to reach the conclusion, step-by-step? Thanks.

You forgot a few 2-shingles (bigrams but without duplicates) in the second set but  you got the idea right:

$S_1$ = { "the quick", "quick brown", "brown fox", "fox jumps", "jumps over", "over the", "the lazy", "lazy dog" }

$S_2$ = { "jeff typed", "typed the", "the quick", "quick brown", "brown dog", "dog jumps", "jumps over", "over the", "the lazy", "lazy fox", "fox by", "by mistake" }

Remark: For this particular example, in each of these two sets every sequence of 2 words appears only once, so there's no need to remove duplicates to obtain the set. In the general case this might be necessary (see the Wikipedia example).

To calculate Jaccard similarity we need to count:

The intersection $S_1 \cap S_2$, i.e. the 2-shingles in common:  | { "the quick", "quick brown", "jumps over", "over the", "the lazy" } | = 5
The union $S_1 \cup S_2$, i.e. all the distinct 2-shingles: | { "the quick", "quick brown", "brown fox", "fox jumps", "jumps over", "over the", "the lazy", "lazy dog", "jeff typed", "typed the", "brown dog", "dog jumps", "lazy fox", "fox by", "by mistake" } = 15

Jaccard similarity:

$$\frac{S_1 \cap S_2}{S_1 \cup S_2} = \frac{5}{15} = 0.33$$


How can I combine two single-column datasets into a single Pandas data frame? I'd like to import the Rotten Tomatoes Movie Review dataset into a single data frame. How can I combine two datasets that are 1-column strings into a text, label shape?

Here's where I'm at so far (you can duplicate in Google Colab) :

import os
import pandas as pd

# Reset
!rm -rf "rt-polarity.csv"

def fetch_rt_polarity_data():
# Fetch Data
if not os.path.isfile("rt-polaritydata.tar.gz"):
    !wget -q http://www.cs.cornell.edu/people/pabo/movie-review-data/rt-polaritydata.tar.gz
    !tar -xzf rt-polaritydata.tar.gz
    !mv rt-polaritydata/rt-polarity.pos rt-polarity.pos
    !mv rt-polaritydata/rt-polarity.neg rt-polarity.neg
    !rm -rf rt-*

# Format Data
df_pos = pd.read_csv("rt-polarity.pos", encoding='latin-1', sep="\t", lineterminator="\n")
df_pos = df_pos.reset_index(drop=True)
df_pos.columns = ['text']
df_pos['label'] = 1

df_neg = pd.read_csv("rt-polarity.neg", encoding='latin-1', sep="\t", lineterminator="\n")
df_neg = df_neg.reset_index(drop=True)
df_neg.columns = ['text']
df_neg['label'] = 0

# Combine
df = pd.concat([df_pos, df_neg], ignore_index=True)
df.head(10)

df.to_csv("rt-polarity.csv")

df = pd.read_csv("rt-polarity.csv", encoding='latin-1', sep="\t", lineterminator="\n")
return df

df = fetch_rt_polarity_data();
df.head(5)

I would import the datasets in pandas separately, mold them as you please, and then you can use the pd.concat function. This will assume that the instances are aligned by the automatically assigned index in pandas. If there is more data in one list than the other, the missing values will be NaN.

df1 = pd.DataFrame(data=[1,2,3])
df2 = pd.DataFrame(data=['a','b','c','d'])
dfs = pd.concat([df1, df2], axis=1)

If you have an index to link the text to the labels, then you can use the pd.merge function.

df1 = pd.DataFrame(data=[1,2,3])
df2 = pd.DataFrame(data=['a','b','c','d'])
dfs = df1.merge(df2, on='index')


How do I test a difference between two proportions representing fatality rate for Covid 19 in Philippines and World (except Philippines)? I'm trying to analyse if the fatality rate from my country (A third world country) vary significantly from the world's fatality rate.

So I'd basically have two samples, labeled (Philippines) and (World excluding the Philippines) then i can compute the fatality rate for the 2 groups.

Does Mcnemar's test apply here for me to check if fatality rate in the Philippines is higher, or do you have any suggestions? Thanks

It is not a case of paired nominal data. Hence, Mc Nemar's test can not be applied to check whether there is a higher fatality rate   in Philippines ?. THE fatality rate is given for Philippines and world ( excluding Philippines ). As defined,  it is expressed as proportion. Therefore, t-test/z-test shall be appropriate given that you meet other conditions such as sample-size.


Need some tips regarding starting out with the field's specific programming languages, with a heavy focus on data visualization I've got rather expansive development experience, but I'm new to data science. I have been trying to familiarize myself with the main concepts and deciding between R and Python to put my time into.

I know you can write Python in R and vice versa, but since data visualization in a beautiful manner would be a very high priority for me in the future and both Python and R use additional packages and libraries for data representation, I'm not entirely sure whether it's possible to use the additional libraries when integrating code into one another.

I would really appreciate it if someone could give me some advice regarding this and also provide me with some general tips about the best approaches, practices and tools when it comes to data visualization.

R is a more compact, target oriented, package. Good if you want to focus on very specific tasks (generally scientific). Python, on the other hand, is a general purpose language.

That being said, and obviously this is a matter of opinion, if you are an experienced developer go for Python. You'll have far more choices in libraries and a far bigger potential to build big software.

Some examples of 2D scientific plotting libraries:

Matplotlib
Bokeh (targeted D3.js)
Chaco
ggplot
Seaborn
pyQtGraph (some significant 3D features)

Some examples of 3D scientific plotting libraries

Vispy
Mayavi
VTK
Glumpy

Some examples of libraries typically used in Data Science in Python:

Pandas
Numpy
Scipy
Scikit-learn
Scikit-image

Also check the list for other relevant Scikit packages.

As for starting software I would advise you to use any of the already prepared Python distributions that already come with a bunch of scientific libraries inside as well as software such as IDEs. Some examples are:

WinPython
Python XY
Anaconda
Canopy

Personally I'm a user of WinPython due to being portable (former user of Python XY, both are great). In any case these distributions will greatly simplify the task of having your scientific Python environment (so to speak) prepared. You just need to code. One IDE known to be specially good for scientists is Spyder. Yet these ones also will work:

PyDev
PyCharm
WingIDE
Komodo
Python Tools for Visual Studio

As for data visualization tips you'll see that the most common functions in the libraries mentioned above are also the most widely used. For instance a library like Pandas let's you call plots directly from the object so there is already an intuitive approach to data visualization. A library like scikit-learn (check the site) already shows examples followed by data visualization of the results. I wouldn't be too concerned about this point. You'll learn just by roaming a bit on the libraries documentation (example).


How do I go from number data to string data (NN)? I am quite confident with using number data with a neural network, but I want to use string data. My question is where do I begin? Obviously you cant times weights and strings together (x * w) because they are of different data types, so what should I do to the string data to turn it into numbers? If you're interested, here is my dataset. It's by no means the final version but here it is just to give you an example. It's based on spelling:

enormous..........| Difficulty: 2
rhythm..............| Difficulty: 3
hamster.............| Difficulty: 2
walk...................| Difficulty: 1
accommodate...| Difficulty: 3
blue....................| Difficulty: 1
projector.............| Difficulty: 1
regression..........| Difficulty: 2
go.......................| Difficulty: 1
playwright........| Difficulty: 3
weird.................| Difficulty: 3
conscience.......| Difficulty: 2

...so after training, when I will input a word, the network should return  either 1, 2 or 3, depending on how hard the network thinks the word is to spell. To sum up, my question is: I am used to using number data with my projects instead of strings, so what steps should I take when creating a network based around strings for the first time? I use numpy only

There are various ways to handle string inputs to neural networks, but since you are trying to predict spelling difficulty, I suggests representing your words as a sequence of characters. This will preserve information about the particular spelling of the words including the order of the letters.

To represent a sequence of characters you can one-hot encode each character, so each word will be represented as a sequence of one-hot length 26 vectors.

To handle this kind of input I suggest either a 1D Convolutional Neural Network or some flavor of a Recurrent Neural Network.

If you choose a 1D CNN you will have to feed it fixed sized inputs. To do this choose a max word length, k, and either cut off or pad with zeros each input word to fit into a k*26 input matrix.

Here is an example of a CNN architecture you might use in Keras:

model = Sequential()
model.add(Convolution1D(nb_filter=32, filter_length=5, activation='relu', input_shape=(k, 26))
model.add(MaxPooling1D())
model.add(Convolution1D(nb_filter=32, filter_length=5, activation='relu')
model.add(MaxPooling1D())
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(64, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(3, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])


How the term "R-squared" in VIF(variance inflation factor) is different from normal R-squared calculation? In normal calculation of R2 , more the value of R2 , it indicates variable represents more variance across the dataset.
But in the calculation of VIF (variance inflation factor), higher the value of R2 , more will be the multi-collinearity present in the variables, hence more unstable the dataset is.
My question is whether R2 is the same on both cases or will it be different calculation? If different, how? If its the same, why R2 value treated as good and bad in both cases? Can someone explain this for me?
Thank you!

It is calculated in the same way. Instead of the response variable $y$ of the original model with numeric regressors $x_1, \dots, x_p$, the response of the VIF-model is $x_i$ and the regressors all other $x_j, j \ne i$.


Find all potential similar documents out of a list of documents using clustering I'm working with the quora question pairs csv file which I loaded into a pd dataframe and isolated the qid and question so my questions are in this form :

0        What is the step by step guide to invest in sh...
1        What is the step by step guide to invest in sh...
2        What is the story of Kohinoor (Koh-i-Noor) Dia...
3        What would happen if the Indian government sto...
.....
19408    What are the steps to solve this equation: [ma...
19409                           Is IMS noida good for BCA?
19410              How good is IMS Noida for studying BCA?

My dataset is actually bigger (500k questions) but I will use these questions to showcase my problem.

I want to identify pairs of questions that have a high probability of asking the same thing. I thought about the naive way, which is to turn each sentence into a vector using doc2vec and then for each sentence calculate the cosine similarity with every other sentence. Then, keep the one with the highest similarity and in the end print all those that have a high enough cosine similarity. The problem is this would take ages to finish so I need another approach.

Then I found an answer in another question that suggests to use clustering to solve a similar problem. So following is the code I implemented based on that answer.

"Load and transform the dataframe to a new one with only question ids and questions"
train_df = pd.read_csv("test.csv", encoding='utf-8')

questions_df=pd.wide_to_long(train_df,['qid','question'],i=['id'],j='drop')
questions_df=questions_df.drop_duplicates(['qid','question'])[['qid','question']]
questions_df.sort_values("qid", inplace=True)
questions_df=questions_df.reset_index(drop=True)

print(questions_df['question'])

# vectorization of the texts
vectorizer = TfidfVectorizer(stop_words="english")
X = vectorizer.fit_transform(questions_df['question'].values.astype('U'))
# used words (axis in our multi-dimensional space)
words = vectorizer.get_feature_names()
print("words", words)

n_clusters=30
number_of_seeds_to_try=10
max_iter = 300
number_of_process=2 # seads are distributed
model = KMeans(n_clusters=n_clusters, max_iter=max_iter, n_init=number_of_seeds_to_try, n_jobs=number_of_process).fit(X)

labels = model.labels_
# indices of preferable words in each cluster
ordered_words = model.cluster_centers_.argsort()[:, ::-1]

print("centers:", model.cluster_centers_)
print("labels", labels)
print("intertia:", model.inertia_)

texts_per_cluster = numpy.zeros(n_clusters)
for i_cluster in range(n_clusters):
    for label in labels:
        if label==i_cluster:
            texts_per_cluster[i_cluster] +=1

print("Top words per cluster:")
for i_cluster in range(n_clusters):
    print("Cluster:", i_cluster, "texts:", int(texts_per_cluster[i_cluster])),
    for term in ordered_words[i_cluster, :10]:
        print("\t"+words[term])

print("\n")
print("Prediction")

text_to_predict = "Why did Donald Trump win the elections?"
Y = vectorizer.transform([text_to_predict])
predicted_cluster = model.predict(Y)[0]
texts_per_cluster[predicted_cluster]+=1

print(text_to_predict)
print("Cluster:", predicted_cluster, "texts:", int(texts_per_cluster[predicted_cluster])),
for term in ordered_words[predicted_cluster, :10]:
    print("\t"+words[term])

I thought that this way I could find for each sentence the cluster that it most likely belongs in and then calculate the cosine similarity between all other questions of that cluster. This way instead of doing it on all the dataset I will be doing it on far fewer documents. However using the code for an example sentence "Why did Donald Trump win the elections?" I have the following results.

Prediction
Why did Donald Trump win the elections?
Cluster: 25 texts: 244
    trump
    donald
    clinton
    hillary
    president
    vote
    win
    election
    did
    think

I know that my sentence belongs to cluster 25 and I can see the top words for that cluster. However how could I access the sentences that are in this cluster. Is there any way to do it?

As much as i understood from your description, you can just assign your labels as a new column to questions_df. There is no shuffling involved, so the following will do the trick:

questions_df["cluster"] = labels


Route to picking up Deep learning I would like to pick up on the topic of deep learning. Should I begin from the topic of AI before working my way into Deep learning?

Start with neural nets. They are the basic building blocks of Deep Learning.

Neural Nets for Newbies video is a well-explained introduction. CS-449 from Willamette University is an free and approachable class in neural networks.

That should lay the foundation for more advanced Deep Learning material. Deep Learning by Bengio, Goodfellow, and Courville is a quality book on the subject.


Using Mean Squared Error in Gradient Descent I've recently been writing linear regression algorithms from scratch to gain an understanding of how the maths behind it works (something that was a bit of a black box beforehand), and so I got around to differentiating the cost function. Without realising it I used the Squared Error for the cost function - the MSE but without dividing by the dataset length. Is there any benefit (faster approach of the minimum or other) to using the Mean Squared Error over just summing the squares of the error?

No, it is exactly the same. Optimizing a function and the same function divided by a constant is equivalent, both in the analytical and the numerical sense. You will get exactly the same optimal parameters.


Algorithmic approach to model blending Model blending -- by which I mean creating multiple sets of predictions from models that have the same dependent variable and the same or similar independent variable candidates, as opposed to model stacking -- is a popular way of creating ensembles of Machine Learning models.

For example:

Y = regression_predictions * .5 + tree_predictions * .5

While this approach is useful in a wide variety of use cases, a good example is Kaggle competitions. In these competitions you have:

Labeled training data
Unlabeled prediction (test) data
A certain number of allowed submissions per day, often 5, wherein you submit your predictions and receive instant scoring based on a specified evaluation metric like RMSE

My question is if there is an statistical or logical methodological framework that can guide the testing of various blends of your models?

That is, can some algorithm or methodology be applied to the process of deciding how to alter the blend of predictions?

Ideally, such a framework would also guide the decision of when to add or remove a set predictions, though even a framework which assumes a given number of models would be highly useful.

This is normally called "Ensemble learning": wiki link

Bayesian Model Combination is a popular modern technique which usually delivers better results than Bayesian Model averaging, on which it was based. The latter had a tendency to give almost all the weight to one model.


NLP grouping word categories Suppose I have a dictionary:

{apple:large apple, apple:red apple, apple:aple, orange:mandarin, orange:orang, orange:blood orange}

and so on...

And then I want to replace a large document of entries with the keys. However, occasionally a new value will come up, i.e. {apple:green apple}

Is there a method where I can replace all values with the corresponding key, but then also replace 'close' values like the one given if they appear?

Example document:

var1
_____
aple
apple
orange
Apple
Red apple
gren Apple
blood Orange
orang

var1_replaced
______________
apple
apple
orange
apple
apple
apple
orange
orange

Well ... the simplest approach is using Fuzzy String Matching and it will work. Just go through the examples in python implementation of it (fuzzywuzzy) and you will understand how it works. You need to find a threshold by practice to determine if two strings are similar enough to be considered as same concept.

If it didn't work please drop a line in the comments so I can propose more sophisticated algorithms.

Good luck!


Efficient way to create matrix that shows if data exits per day So I have a dataset containing different ID's and the time the data was created.
    ID              Date
0   123123          2021-03-24 12:43:13.494000+00:00
1   123412          2021-03-24 12:43:13.494000+00:00
2   123123          2021-03-24 12:43:15.935000+00:00
3   234234          2021-03-24 12:43:15.935000+00:00
4   432424          2021-03-24 12:43:13.494000+00:00

The goal should be to validate that there is at least one data row for every id for every given day.
What I did so far is converting the timestamps to dates like this:
0         2021-03-24
1         2021-03-24
2         2021-03-24
3         2021-03-24
4         2021-03-24

Now I missing a solution to create a matrix that tells me if for the given ID on the given date a data row of this data exists or not:
The data frame creation looks like this:
df = pd.DataFrame(index=df['id'].unique(),columns=df['date'].sort_values().unique())

which creates this matrix:
    2021-03-19  2021-03-20  2021-03-21  2021-03-22  2021-03-23  2021-03-24  2021-03-25
12341   NaN NaN NaN NaN NaN NaN NaN
12312   NaN NaN NaN NaN NaN NaN NaN
12324   NaN NaN NaN NaN NaN NaN NaN
12345   NaN NaN NaN NaN NaN NaN NaN
12345   NaN NaN NaN NaN NaN NaN NaN
... ... ... ... ... ... ... ...
12399   NaN NaN NaN NaN NaN NaN NaN
12394   NaN NaN NaN NaN NaN NaN NaN
34567   NaN NaN NaN NaN NaN NaN NaN
98764   NaN NaN NaN NaN NaN NaN NaN
10023   NaN NaN NaN NaN NaN NaN NaN

sure I could just use loops to now fill the values, however this would be a a really inefficient way to do it.. I think there is a better way and I am super sure somebody can tell me how so I can learn it for the future.
Thank you very much!

You should be able to just use a pandas pivot table for this purpose:
import pandas as pd

pd.pivot_table(df, index="ID", columns="Date", aggfunc="count")


Rendered Image Denoising I am learning about "Image Denoising using Autoencoders". So, now I want to build and train a model. Hence, when I read into how Nvidia generated the dataset, I came across: We used about 1000 different scenes and created a series of 16 progressive images for each scene. To train the denoiser, images were rendered from the scene data at 1 sample per pixel, then 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536, and 131072 samples per pixel.

I was trying to understand-

1) what is meant by rendering images at n samples per pixel?

2) How to do this in python to generate the dataset?

I have read some articles regarding this but could not form a confident opinion from a Data Science perspective.

https://area.autodesk.com/tutorials/what-is-sampling/

Any leads would be much appreciated! Thanks

Your link is to paid course :) In ray-tracing too few samples will generate something like at the top  In fact the link with the picture answers your question https://chunky.llbit.se/path_tracing.html

2) Ray-tracing is hard... but not impossible, google for "python ray tracing module"... But something looking close - easily https://stackoverflow.com/questions/22937589/how-to-add-noise-gaussian-salt-and-pepper-etc-to-image-in-python-with-opencv Although actually on the ray-traced images the noise can change because of slope and environment.

If you still want ray-traced noisy images, better to find tutorials for 3D modelling programs, like "ray tracing in 3D Studio MAX tutorial"


Is machine learning the right tool for this job? I would like to create a troubleshooting wizard.

The user will go through the wizard and choose different options, what options they choose will determine what is displayed next in the wizard.

Eventually the user will solve their problem (or not), at the end they will choose 'yes' or 'no' if there problem was solved.

The wizard would learn what most common resolutions to a problem was based on past input.

Before I go exploring machine learning further, does this sound like the right tool for the job?

Yes and no,
it sounds ok, but depends on the complexity of the wizard very much. It is possible that a simple rule-based algorithm will be super useful and easier to implement.

If you really want to try it (for example for learning purposes), the bottleneck could be the data which needs to be available for initially training the model. You should have enough user queries (or at least simulated queries) even before your ML wizard exists so that the model can learn.

If it is something you need for work, I would consider consulting somebody with ML/Data Science experience with the specific details of what you need.

Hope, this helps at least a bit.

Best regards,
Jakub


Evaluation of linear regression model I want to evaluate the performance of my linear regression model. I have the true values of y (y-true). I am thinking of two way for evaluation but not sure which one is correct.

Let's assume that we have 2 samples and each sample has two outputs as following:

y_true = [[0.5, 0.5],[0.6, 0.3]]
y_pred = [[0.3, 0.7],[0.9, 0.1]]

- Approach#1 :

One way to calculate the sum of the difference between the actual and predicted for each vector and then average all, as follows:

sum_diff_Vector(1) = abs( 0.5 - 0.3 ) + abs( 0.5 - 0.7 ) =  0.4

sum_diff_Vector(2) = abs( 0.6 - 0.9 ) + abs( 0.3 - 0.1 ) = 0.5

Then avg ( sum_diff_Vector(1) , sum_diff_Vector(2) ) = 0.45

- Approach#2 :

Another way to use the mean absolute error provided by sklearn.metrics in python.  The thing with this metric, as opposed to the previous method, it calculates the mean absolute error for each output over all samples independently and then average all of them, as follows:

MAE_OUTPUT(1) = abs(( 0.5 - 0.3 ) + ( 0.6 - 0.9 )) / 2 = 0.25

MAE_OUTPUT(1) = abs(( 0.5 - 0.7 ) + ( 0.3 - 0.1 )) /2 = 0.2

Then avg ( MAE_OUTPUT(1) , MAE_OUTPUT(1) ) =  0.225

Which way is correct and I should use ? please advise?

The only difference is in your example is that you divide by an additional two, because you take the mean per vector instead of the sum. Correctness does not play here because for comparison between different models the only difference is a constant factor and for interpretability it depends on the problem you are solving.

The mean absolute error punishes mistakes linearly while the mean squared error punishes larger mistakes more heavily. This means this depends a bit on what you want to measure, based on the problem you are solving. Next to proper evaluation you could use this same measure to change the KPI you are optimizing directly with a different loss function.


What kind of classification should I use? I'm very new to machine learning (and statistics) and I'm struggling with some basics (I'm using R as my primary environment).

I have a lexicon with words and all of their forms (in different genders, persons - for a Slavic language). So each word with its word forms is one training sample. My training attributes are last character, last two characters, last three characters of each word form.  So each training sample contains from around 100 categorical attributes. I've got 10000 samples. One attribute has around 1700 levels, one around 1200, 5 around 500, others have less than 100 levels.

I have used DBSCAN (and gower distance) to cluster similars words to the same cluster. This step was pretty successful. I get (depending whether is a noun, verb or adjective) from 40-100 clusters.

Now here comes the part where I don't know how to continue. If I get a new (unknown) word I want to predict to which cluster it belongs. Sometimes for this new word I only know its base form, sometimes I also part of its word forms and sometimes I know all of its word forms.

So this should be some kind of classification with multiple classes. Question: which algorithm should I use? I've tried to use multinom (in R) but I get an error of too many weights. After I increased MaxNWts I get an error:

weights:  641676 (633954 variable)
Error: cannot allocate vector of size 396 Kb

Where should I continue? Clean the data? Use a different algorithm?

Also, around 15 attributes have only 1 level or NA. How does this impact to clustering (and classification)? Is it safe to remove those attributes?

Many thanks in advance!

Using your input data as-is (many categorical attributes), you should start with random forests.  The levels of your features probably aren't ordinal in any sense, which would explain why multinom (a linear model) didn't work.  A decision tree (component of a random forest) won't have trouble with categorical features that aren't ordinal.

If your features are correlated (I'm guessing they are highly correlated), you might also try first reducing the dimensionality of your input dataset.  A nice way to do this is to view your categorical features as "words" and map each input sample (a word in all its forms) to a sparse vector (training a TF-IDF model).  Then, reduce the dimensionality of your sparse vectors using LSI or LDA (dense vectors).  Train a classifier (again, random forest is a good starting point) using these dense vectors as input data.


Passing Dependency/Constituency trees to a Neural Machine Translator I am working on a project on Neural Machine Translation in the English-Irish domain. I am not an  expert and have researched entirely on my own for a technology exhibition so apologies if my question is simple.
I am trying to parse all of my English corpus to constituency trees. Of course, the format of a sentence when using the Stanford Parser is something like:

(ROOT (S (NP (VBG cohabiting) (NNS partners)) (VP (MD can) (VP (VB make) (NP (NP (NNS wills)) (SBAR (WHNP (WDT that)) (S (VP (VBP favour) (NP (DT each) (JJ other)))))))) (. .)))

Of course, when dealing with simple sequences, each word is used, it's not symbolism like NP or NNS in constituency trees.
Right now, I'm working with PyTorch and Fairseq to produce all my models and have gotten a working seq2seq model. But, can I simply just pass my English input like shown above to a model and expect it to train? Do I need to make a new model from scratch that can deal with tree structures? I've tried very hard to research this, reading papers, books and playing around with tools but since I'm not in a class for this and since it's not really documented, I'm finding it hard to find this on my own.
Any help would be greatly appreciated

First of all, I would like you to discourage you from using structured input in NMT. In most cases, the best thing you can do is just do some subword input and output segmentation and learn simple sequence-to-sequence conversion.
You can certainly pass the parsed input in the format that you showed above and the worse thing you can expect the model will to ignore the mark-up. Some modest improvements from using syntax in this way are shown in this WTM19 paper from Edinburgh.
People were more successful when using graph convolutional networks for processing the structured input, AFAI used for the first time in 2017. In this case, you would need to add a custom encoder into FairSeq and specify a model with this encoder. There are several PyTorch implementations of graph convolutional networks that you can reuse in FairSeq, e.g.,

gcn-over-pruned-trees

pygcn

AllenNLP also has some tools for structure input.

Note, however, that once you change model architecture, you need to be careful about the learning rate schedule that is otherwise tuned for the standard Transformers.


Right ML mode and metric to minimize FN and FP on imbalanced dataset So I have a dataset in which I have to predict class binary label (1 or 0), the problem, out of 120k data points, only 200 have the label '1'.
the aim is to minimize FN and FP.

Which  ML model should I use? Gradient boost or XGBoost or logistic regression. How do I calculate the class weights?
Which accuracy metric captures the minimization of FP and FN?

If you use class weights, I don't think it would make much of a difference which model you use (regarding the imbalance). If you are familiar with python, I would suggest this sklearn function that could help you out in computing the class weights. However, I have found that oversampling (or undersampling) often works better than class weights. Consider using SMOTE for this.
While there are metrics and loss functions better suited at handling class imbalance (GMS, quadratic kappa, etc.) I don't think you would have an issue with using any loss
if you considered oversampling your data.


Does a Convolutional Layer in a Neural Network learn the correlation between its input signals via its kernel? I am interested in the theory behing what a convolutional neural network learns with its convolutional operations. I think it learns (useful) kernels which measure the correlation between its input signals. Moreover, on a technical perspective: the convolution operation is implemented as "cross-correlation". So, is my assumption right?

In a convolutional neural network (CNN), a convolutional layer has several channels, each of which has one convolution kernel, often written down as a matrix. This convolution kernel is nothing more than a collection of weights used to compute a linear combination of elements of the input.
While both "traditional" dense layers and convolutional layers compute a linear combination of their inputs, convolutional layers have the added structure of preserving spatial information. Additionally, the output of a convolutional layer, called a feature map, can be understood as the result of sliding the kernel along the input.
For instance, take a simple kernel $K = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ 7 & 8 & 9 \end{bmatrix}$ and a small input "image" $I = \begin{bmatrix} 9 & 8 & 7 & 6 \\ 5 & 4 & 3 & 2 \\ 1 & 2 & 3 & 4 \\ 5 & 6 & 7 & 8 \end{bmatrix}$. We take an input with only one channel for convenience: if it's the first layer of the CNN, we can assume the values of $I$ represent a grayscale image. Let us look at how the convolution is performed if no padding is added to the image, in which case the output will be a 2 by 2 matrix.
Starting at the top left corner, we convolve $K = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ 7 & 8 & 9 \end{bmatrix}$ with $\begin{bmatrix} 9 & 8 & 7 \\ 5 & 4 & 3 \\ 1 & 2 & 3 \end{bmatrix}$. To do so, we multiply the two matrices element-wise, then sum up all the elements of the resulting matrix: $1 \cdot 9 + 2 \cdot 8 + 3 \cdot 7 + 4 \cdot 5 + 5 \cdot 4 + 6 \cdot 3 + 7 \cdot 1 + 8 \cdot 1 + 9 \cdot 3 = 146$.
Doing the same three more times yields the following feature map: $\begin{bmatrix} 146 & 157 \\ 200 & 233 \end{bmatrix}$.
Depending on the kernel of a layer, the feature map will look different, and the information extracted from the input has a different "meaning". The idea behind a CNN is to make the coefficients of the kernel learnable parameters. Often, multiple "meanings" must be extracted from a single input, so a convolutional layer has multiple channels.
However, CNNs didn't invent convolution, they merely made the kernels learnable. Indeed, such methods have been used in "traditional" computer vision (CV) for a long time. In traditional CV, the kernels are hand-crafted to fill specific roles, extract specific types of information.
For edge detection or line detection, you might use a Prewitt filter which is the discrete equivalent of a derivative operator or a slightly more complicated Sobel filter.
In fact, it is not uncommon for a CNN to learn approximations of well-known hand-crafted convolution kernels from traditional CV on the first layer. Also, by visualizing feature maps on deeper layers, as the training of a CNN progresses, gives you good insight into what the specific convolution channels are learning to detect.


Visualize/analyze data before or after imputing missing values? My understanding is that we impute missing values in order to preserve those training examples so our Machine Learning algorithms have as many training examples as possible.

To me, it would make intuitive sense to visualize/analyze the data before imputing the missing values, as imputation will skew distributions and may lead to false assumptions about the real data before imputation. I can see a case for doing both before and after, but that adds time to the feature analysis. On the other hand, I can see a case for not imputing values if the % of missing values is high enough to affect the data's distribution.

In Python I imagine using something along the lines of pd.Series.dropna() to isolate the existing values.

TL;DR: Should one impute missing values before or after visualizing the data and pulling insights from it?

Why not do both? Like you mention, it might be worth first computing the percentage of all values that are values. Generally you might also have a percentage in mind that is acceptable, like up to 10% missing values, if they are scattered at random throughout your dataset.

There are libraries built specifically for visualising missing data, such as missingno, which offers quite a few ideas. Here is an example heatmap of missing variables across features:

"missing" normally implies you have a sequential dataset, for example time-series data. If you had discrete observations e.g. of people's height versus shoe size, there is no sequential causality (autocorrelation: dependency on previous values). In this case, imputing makes little sense.

So assuming you do have sequential data, whether or not to impute or drop time steps with missing values will really depend on your use case. Also perhaps the frequency of the data. If all the missing vaiues appear in one chunk at either end of the time series, it is cokmmon to simply leave out that chunk.

For example, if you have minute frequency data and you wish to predict a value once per day, then missing a few minutes here and there might be tolerable, and imputation of some kind (e.g. fill-forwad) wouldn't have a huge impact overall, but could help the model optimisation work more effectively. Some models cannot handle missing values, so imputation is necessary.

In any case, it would always visualise the data before and after imputation. You can usually run the same visualisation anyway. Sure it costs a few extra minutes, but you might catch important issues. This can save a lot of time compared to only finding the issues later on while debugging your trained model.


How to normalize a boolean feature for neural nets? I have a feature that is boolean and I would like to feed it to a neural net as one of the inputs. I think in theory the best is to encode as false->0 and true->1 because 0 as an input will deactivate weights of a neuron. Is this correct?

Actually, it is not clear what you mean by deactivating but if it means the output of neuron would be zero, it is not correct due to having bias term, also known as intercept. Furthermore, we usually use normalisation for features which are of different scales. Your boolean values do not have a large range. You don't need to scale them. If I want to be more precise, you may need depending on the other features' range, because they may change slightly among different input patterns and vary less than let say 1e-5 for different samples, but most of the time, booleans are not needed to be scaled.


Could someone please offer me some guidance on some kind of particular, SPECIFIC project that I could attemp, to "get my feet wet, so to speak" I am COMPLETELY new to the field of Data Science, mainly because every employer I have worked for, simply COULDN'T sell any customers anything that would use techniques learned in this field.

Of particular interest to me is machine learning/Predictive Analysis.

I have attempted many "test projects" myself, but I seem to NEED some sort of outside "catalyst" to tell me a specific goal, and a specific set of guidelines, when I am trying to learn something.
Otherwise, I tend to lose focus, and jump from one interesting topic to the next, without ever gaining any experience.

Thank you!!

I would suggest Kaggle learning projects - http://www.kaggle.com/competitions

Look for the ones in the 101 section that offer knowledge. There's many pre-made solutions ready, which you can ingest and try variations of.

Also, I have bookmarked a Comprehensive learning path – Data Science in Python, which among other things gives a few answers to your specific question.


Generate predictions that are orthogonal (uncorrelated) to a given variable I have an X matrix, a y variable, and another variable ORTHO_VAR.  I need to predict the y variable using X, however, the predictions from that model need to be orthogonal to ORTHO_VAR while being as correlated with y as possible.

I would prefer that the predictions are generated with a non-parametric method such as xgboost.XGBRegressor but I could use a linear method if absolutely necessary.

This code:

import numpy as np
import pandas as pd
from sklearn.datasets import make_regression
from xgboost import XGBRegressor

ORTHO_VAR = 'ortho_var'
TARGET = 'target'
PRED = 'yhat'

# Create regression dataset with two correlated targets
X, y = make_regression(n_features=20, random_state=245, n_targets=2)
indep_vars = ['var{}'.format(i) for i in range(X.shape[1])]

# Pull into dataframe
df = pd.DataFrame(X, columns=indep_vars)
df[TARGET] = y[:, 0]
df[ORTHO_VAR] = y[:, 1]

# Fit a model to predict TARGET
xgb = XGBRegressor(n_estimators=10)
xgb.fit(df[indep_vars], df[TARGET])
df[PRED] = xgb.predict(df[indep_vars])

# Correlation should be low or preferably zero
pred_corr_w_ortho = df.corr().abs()[PRED][ORTHO_VAR]
assert pred_corr_w_ortho < 0.01, "Correlation score: {0} is superior to the given threshold.".format(pred_corr_w_ortho)

Returns this:

---------------------------------------------------------------------------
AssertionError
      1 pred_corr_w_ortho = df.corr().abs()[PRED][ORTHO_VAR]
----> 2 assert pred_corr_w_ortho < 0.01, "Correlation score: {0} is superior to the given threshold.".format(pred_corr_w_ortho)

AssertionError: Correlation score: 0.5895885756753665 is superior to the given threshold.

...and I would like something that maintains as much predictive accuracy as possible while remaining orthogonal to ORTHO_VAR

This requirement can be satisfied by adding sufficient noise to predictions $\hat{y}$ to decorrelate them from orthogonal values $v$. Ideally, if $\hat{y}$ is already decorrelated from $v$, no noise would be added to $\hat{y}$, thus  $\hat{y}$ would be maximally correlated with $y$.

Mathematically, we want to create $\hat{y}'=\hat{y}+\epsilon$ from $\epsilon \sim \mathcal{N}(0, \sigma_{\epsilon})$, to satisfy $$r_{\hat{y}'v} = \frac{\sigma_{\hat{y}'v}}{\sigma_{\hat{y}'}\sigma_{v}}  < \delta$$ for arbitrary threshold $\delta$. Now, lets expand this inequality to find a lower-bound for std of noise $\epsilon$, i.e. $\sigma_{\epsilon}$.
$$\begin{align*}
\sigma_{\hat{y}'}^2&=\sigma_{\hat{y}}^2 + \sigma_{\epsilon}^2,\\
\sigma_{\hat{y}'v}&={\Bbb E}\left[(\hat{y}+\epsilon - \mu_{\hat{y}} - \overbrace{\mu_{\epsilon}}^{=0})(v-\mu_{v})\right]\\
&={\Bbb E}\left[(\hat{y} - \mu_{\hat{y}})(v-\mu_{v})\right]+\overbrace{{\Bbb E}\left[\epsilon(v-\mu_{v})\right]}^{=0}&\\
&=\sigma_{\hat{y}v},\\
r_{\hat{y}'v} &= \frac{\sigma_{\hat{y}'v}}{\sigma_{\hat{y}'}\sigma_{v}} =\frac{\sigma_{\hat{y}v}}{\sigma_{v} \sqrt{\sigma_{\hat{y}}^2+\sigma_{\epsilon}^2}} < \delta\\
&\Rightarrow \sigma_{\hat{y}}\sqrt{\left(\frac{r_{\hat{y}v}}{\delta}\right)^2 - 1} < \sigma_{\epsilon}
\end{align*}$$

Since all the variables in the left side of inequality can be calculated, we can sample noises from $\mathcal{N}(0, \sigma_{\epsilon})$ and add them to $\hat{y}$ to satisfy the original inequality.

Here is a code that does the exact same thing:

import numpy as np
import pandas as pd
from sklearn.datasets import make_regression
from xgboost import XGBRegressor

ORTHO_VAR = 'ortho_var'
IND_VARNM = 'indep_var'
TARGET = 'target'
CORRECTED_VARNM = 'indep_var_fixed'

seed = 245
# Create regression dataset with two correlated targets
X, y = make_regression(n_samples=10000, n_features=20, random_state=seed, n_targets=2)
indep_vars = ['var{}'.format(i) for i in range(X.shape[1])]

# Pull into dataframe
df = pd.DataFrame(X, columns=indep_vars)
df[TARGET] = y[:, 0]
df[ORTHO_VAR] = y[:, 1]

# Fit a model to predict TARGET
xgb = XGBRegressor(n_estimators=10)
xgb.fit(df[indep_vars], df[TARGET])
df['yhat'] = xgb.predict(df[indep_vars])

delta = 0.01

# std of noise required to be added to y_hat to bring the correlation
# of y_hat with ORTHO_VAR below delta
std_y_hat = np.std(df['yhat'], ddof=1)
corr_y_hat_ortho_var = np.corrcoef(df['yhat'], df[ORTHO_VAR])[1, 0]
corr_y_hat_target = np.corrcoef(df['yhat'], df[TARGET])[1, 0]
std_noise_lower_bound = std_y_hat * np.sqrt((corr_y_hat_ortho_var / delta)**2 - 1.0)
std_noise = max(0, std_noise_lower_bound) + 1
print('delta: ', delta)
print('std_y_hat: ', std_y_hat)
print('corr_y_hat_target: ', corr_y_hat_target)
print('corr_y_hat_ortho_var: ', corr_y_hat_ortho_var)
print('std_noise_lower_bound: ', std_noise_lower_bound)
print('std_noise: ', std_noise)

# add noise
np.random.seed(seed)
noises = np.random.normal(0, std_noise, len(df['yhat']))
noises -= np.mean(noises)  # remove slight deviations from zero mean
print('noise_samples: mean:', np.mean(noises), ', std: ', np.std(noises))
df['yhat'] = df['yhat'] + noises

# measure new correlation
corr_y_hat_ortho_var = np.corrcoef(df['yhat'], df[ORTHO_VAR])[1, 0]
corr_y_hat_target = np.corrcoef(df['yhat'], df[TARGET])[1, 0]
print('new corr_y_hat_target: ', corr_y_hat_target)
print('new corr_y_hat_ortho_var: ', corr_y_hat_ortho_var)
# Correlation should be low or preferably zero
assert corr_y_hat_ortho_var < delta, corr_y_hat_ortho_var
assert -delta < corr_y_hat_ortho_var, corr_y_hat_ortho_var

which outputs:

delta:  0.01
std_y_hat:  69.48568725585938
corr_y_hat_target:  0.8207672834857673
corr_y_hat_ortho_var:  0.7663936356880843
std_noise_lower_bound:  5324.885500165032
std_noise:  5325.885500165032
noise_samples: mean: 1.1059455573558807e-13 , std:  5373.914830034988
new corr_y_hat_target:  -0.004125016071865934
new corr_y_hat_ortho_var:  -0.000541131379457552

You can experiment with other deltas. By comparing std_y_hat with std_noise_lower_bound, you can see that a huge noise must be added to $\hat{y}$ to decorrelate it from $v$ bellow $0.01$, which dramatically decolerates $\hat{y}$ from $y$ too.

Note: Assertion might fail for too small thresholds $\delta$ due to insufficient sample count.


How to estimate probabilities of different classes for a Text Suppose I have a piece of writing and I want to assign probabilities to different genres (classes) based on its contents. For example

  Text #1 : Comedy 10%, Drama 50%, Fiction 20%, Romance 1%, Mythology 5%, Adventure 10%

  Text #2 : Comedy 40%, Drama 3%, Fiction 2%, Romance 30%, Mythology 5%, Adventure 10%

We have given keywords possibly ngrams in each class through which we make a comparison

  Class 1 Comedy : k11, k12, ..., k1m

  Class 2 Drama : k21, k22, ..., k2n

  Class 3 Fiction : k31, k32, ..., k3o

  Class 4 Romance : k41, k42, ..., k4p

  Class 5 Mythology : k51, k52, ..., k5q

  Class 6 Adventure: k61, k62, ..., k6r

What can be the best probabilistic model that we can use for this task

If I understand correctly, we are interested in soft multilabel classification, where a single text can have multiple correct genres.

According to your comment, we don't have any training data, just a list of keywords associated with each genre.

We can try computing the similarity between each document and each keyword list:

Normalize the document (convert to lowercase, remove punctuation, diacritics, non-alphanums, etc)
Remove stopwords
Convert the document to tf-idf vector over our genre keyword vocabulary: Each document gets an n-length vector where each entry is the frequency of the ith genre keyword in the document. Normalize this vector to magnitude 1.
Convert each genre keyword list to a tf-idf vector in the same way (again over the keyword vocabulary for all genres).
Compute the cosine similarity between the document vector and each genre vector.

For each document, this will give us a number in the range [0,1] for each genre. For example:

         Comedy Drama Fiction Romance Mythology Adventure
Text #1: 0.15   0.11  0.03    0.00    0.00      0.07

If we were doing single label classification we could normalize each row to add up to 1 and we might have a working model. However there is no such trick for multilabel classification here. We don't have a good way to calibrate these values into probability estimates.

At this point the only solution I see is to build a small training set so we can fit our model to actual data.

After gathering some training examples, we can run a multilabel regression with sigmoid activation and binary crossentropy loss with the cosine similarities as input features to get a probability estimate for each class.

Using this method our list of genre keywords will at least save us having to build a large training set to solve the problem directly with bag-of-words or similar approaches.


How do neural networks account for outliers? How do neural networks account for outliers and overfitting?

Here's a math answer for you.

Neural network is an approximation function $f(\theta)$ of the joint distribution $p(X, Y)$ of input data $X$ and labels $Y$. The learning process is the process of tweaking parameters $\theta$ to make $f$ as close as possible to $p$

$$f(\theta) \approx p(X, Y)$$

Side note: usually $f$ is considered to approximate the conditional $p(Y|X)$, but it can be viewed more generally.

So, in this terms, outliers are the values $(x, y)$ clearly outside of the distribution $p(X, Y)$. Hence, a neural network can not account for them, this has to be done separately. One possibility is to gather more data to make these outliers look ordinary, i.e., from $p(X, Y)$.

As for overfitting, it's completely different term, which relates to inability to generalize. This happens because in practice the true distribution $p(X, Y)$ is never fully known. Instead, the researches have the sample of it -- $(\hat{X}, \hat{Y})$, a.k.a. the training data. If the dimensionality of $\theta$ is large, $f$ can approximate $p(\hat{X}, \hat{Y})$ so well that it actually learns the noise and fails to capture $p(X, Y)$. There are ways to deal with overfitting, most importantly regularization, which is equivalent to adding noise to $(\hat{X}, \hat{Y})$. So the answer to this question is: it can be extended to account for it, by using special techniques. But if the researcher does nothing, the neural network won't learn to deal with overfitting by itself.


Why an increasing validation loss and validation accuracy signifies overfitting? When I train a neural network, I observe an increasing validation loss, while at the same time, the validation accuracy is also increased.
I have read explanations related to the phenomenon, and it seems an increasing validation loss and validation accuracy signifies an overfitted model.
However, I have not really grokked the reasons why an increasing validation loss and validation accuracy signifies an overfitting.
Could you please give the explanations behind this phenomenon?

Overfitting is when your model is memorizing the training data, and therefore it is not able to generalize to unseen data, like the validation set.
This way, you see overfitting when the training loss decreases but the validation loss increases.
The fact that, while the validation loss increases, the validation accuracy also increases is a separate effect. To understand it, we should take into account that the accuracy is computed comparing the highest output of the softmax with the correct label, and it does not vary depending on the actual value of the softmax output, which is what the validation loss takes into account.