Find-out abnormal behavior over the time I want to detect abnormal behaviour in a oil pipe where the oil is flowing with some constant pressure. I have a sensor which monitors and pressure over the time and push it to my cloud server.

I got the dataset . My requirement is to do data analytics in python and find-out the abnormal pattern in the dataset over the time. And I need to suggest abnormal behaviors present in the dataset ., say for example., from 1 pm to 3:30 pm today , the pressure raises and falls may be due to some leakage in the pipe.

can we do it by simple statistical model? or machine learning is required?

Can please suggest the best suitable machine learning algorithm for this scenario?

Also please mention the web links over here.

Thanks

Your problem definition

You have time series data which is used to measure the pressure using your sensor. You wish to identify when the pressure recordings are abnormal. This problem would be best solved using anomaly detection algorithms. But, there are so many ways that you can approach this problem.

I would use a sliding window approach and use that as your feature space to detect the distribution of your recordings. The window length you will select $m$ is the first of your hyper-parameters that you will need to tune.

Using a statistical model

The retained time series $X \in \mathbb{R}^m$, you can treat this signal as a queue, first in first out. When a new recording is defined then discard the oldest datapoint. For a given set $X$ of samples, get the mean and the standard deviation $\sigma$. If the new point exceeds a multiple of $\sigma$, usually this is set to $3\sigma$, but it depends on the expected variation of your sensor, then we flag a state change.

You can further extend this by using the generalized likelihood ratio test (GLRT) to determine when a new sample causes the point to fall significantly outside the distribution of the null hypothesis. In which case this indicates a state change (the new point came from a different distribution than the normal flow through the pipe).

Using machine learning

You can collect multiple instances of your time series and annotate them based on your experience as being nominal or anomalous. Then you will have a supervised 2-class classification problem. First you should attempt some feature extraction using PCA, LDA or similar techniques.. Then you can attempt to use all the fun algorithms available to you through scikit-learn (SVM, Random Forests, K-NN, etc.).

If there is a significantly higher number of nominal instances than anomalous ones, this will introduce bias to your model. Anomaly detection algorithms are better suited for these types of problems. These algorithms learn the distribution within which your nominal set should belong. Then for novel instances it evaluates the probability of it being contained in the learned distribution. If the probability is small, then the algorithm will flag the instance as anomalous.

For more information on anomaly detection for time series refer to:

Using time series data from a sensor for ML

How to train model to predict events 30 minutes prior, from multi-dimensionnal timeseries


What representation of data should I choose, pandas, numpy or tensors to train neural network model I am having a pandas dataframe divided to X_train, X_test, y_train, y_test by train_test_split and I am using it to train my neural network model for binary classification. It is taking some time and I was wondering if it would be faster if I changed it from pandas.dataframe to numpy.array or tensor.

It's hard to say for sure without knowing more about the size of your dataset, the architecture of your model, and the libraries you are using.
Changing from pandas.Dataframe to numpy.array is very unlikely to make your training faster.  Pandas dataframes are already backed by numpy arrays.
Depending on what neural network library you are using and what hardware you have available, changing from Dataframes to tensors may help.  Libraries like Tensorflow and PyTorch allow you to move computation to a GPU/TPU, which may speed up training.
The benefit of using tensor data structures from DL libraries is that they can be moved to a GPU/TPU.  They aren't inherently faster.  So if you don't have a GPU, or if you are using scikit-learn to train the model, then there's no point switching from pandas to tensors


Cross Validation and training set In RapidMiner I want to create a k-NN model in order to create a classifier. To generate the test sets and the training sets I use the the cross-validation.

If I choose 10 as the number of folds, the cross-validation algorithm will do 10 iterations with different test sets and training sets.

At the end, what training set will be used to create my k-NN model ?

Apart from having an average accuracy of the trained model, also you can use cross-validation to approximate the optimal value for the k parameter and the optimal metric.


How to fill df with data from another df - Python I am trying to fill df with values form a different one.
The first df is:

The second df is:

I want to add the column to first df having values from the second one to receive output:

number
trans
date
value

HF01
one
sty
1

HF01
one
lut
2

-
-
-
-

HF05
five
paz
50

I came up with code:
found = []

for index, row in one.iterrows():
    for ind, rw in two.iterrows():
        if ind not in found:
            if (one.loc[index, 'number'] == two.loc[ind, 'number']) & \
               (one.loc[index, 'trans'] == two.loc[ind, 'trans']) & \
                    one.loc[index,'value'] =  two.loc[ind, one.loc[index, 'date']]
                    found.append(int(ind))
                    break

It works but it is extremely slow. How it should be done properly?

First answers is close, the only thing you need to do is merge both data frames using two fields. You do not need both data frames to have the same length at all
Try:
pd.merge(df1, df2, on = ["number","trans"], how = "left")


Tensorflow CNN not predicting correctly for a well trained model So I have built a CNN and trained it to around 95% accuracy on training data, and 90% on testing data. The issue is, when I save this model, load it in again, and predict, it always predicts 1 or close to 1. I even used training data.

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import cv2
import os
import random
from keras.models import Sequential
from keras.layers import Conv2D,Dense,Flatten,Dropout,MaxPooling2D
from keras.preprocessing.image import ImageDataGenerator
from keras.callbacks import ReduceLROnPlateau

image_height = 150
image_width = 150
batch_size = 10
no_of_epochs  = 10

model = Sequential()
model.add(Conv2D(32,(3,3),input_shape=(image_height,image_width,3),activation='relu'))
model.add(Conv2D(32,(3,3),activation='relu'))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.2))
model.add(Conv2D(64,(3,3),activation='relu'))
model.add(Conv2D(64,(3,3),activation='relu'))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.2))
model.add(Conv2D(128,(3,3),activation='relu'))
model.add(Conv2D(128,(3,3),activation='relu'))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.2))
model.add(Flatten())
model.add(Dense(units=128,activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(units=1,activation='sigmoid'))
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

train_datagen = ImageDataGenerator(rescale=1./255,
                                   rotation_range=15,
                                   shear_range=0.2,
                                   zoom_range=0.2
                                   )

test_datagen = ImageDataGenerator(rescale=1./255)

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import cv2
import os
import random
from keras.models import Sequential
from keras.layers import Conv2D,Dense,Flatten,Dropout,MaxPooling2D
from keras.preprocessing.image import ImageDataGenerator
from keras.callbacks import ReduceLROnPlateau
from keras import *
from keras.preprocessing.image import ImageDataGenerator
from keras.preprocessing import image

training_set = train_datagen.flow_from_directory('/content/hackmed2020/train',
                                                 target_size=(image_width, image_height),
                                                 batch_size=batch_size,
                                                 class_mode='binary')

test_set = test_datagen.flow_from_directory('/content/hackmed2020/test',
                                            target_size=(image_width, image_height),
                                            batch_size=batch_size,
                                            class_mode='binary')

# Updated part --->
val_set = test_datagen.flow_from_directory('/content/hackmed2020/val',
                                            target_size=(image_width, image_height),
                                            batch_size=1,
                                            shuffle=False,
                                            class_mode='binary')

reduce_learning_rate = ReduceLROnPlateau(monitor='loss',
                                         factor=0.1,
                                         patience=2,
                                         cooldown=2,
                                         min_lr=0.00001,
                                         verbose=1)

callbacks = [reduce_learning_rate]

history = model.fit_generator(training_set,
                    steps_per_epoch=5216//batch_size,
                    epochs=no_of_epochs,
                    validation_data=test_set,
                    validation_steps=624//batch_size,
                    callbacks=callbacks
                   )

model.save('path_to_my_model.h5')

model = models.load_model('path_to_my_model85.h5')
path_to_file = "/content/Pneumonia1.jpeg"

img = image.load_img(path_to_file, target_size=(150,150))
img = image.img_to_array(img)
img = np.expand_dims(img,axis=0)
prediction = model.predict(img)

I dont think you are rescaling the image after reading it. Because you have rescaled it during training.

img = image.load_img(path_to_file, target_size=(150,150))
img = image.img_to_array(img)

img = img/255 # this must be done.


training neural nets for OCR we are trying to build an in house OCR system to extract alphanumeric strings from images (kindly note, most of our clients cant afford to send their data onto the cloud so it rules out any API from the googles and microsofts..and also tesseract is way too slow for our purpose ..we are talking 10's of users simultaneously uploading images .. tesseract, i guess was never designed to be used as such). We have already started using a CTC based approach to train on our data sets but there's a fundamental/conceptual issue. By definition, CTC finds a happy path to the most probable unit (in this case, a word) but alphanumeric strings by definition DO NOT have a pattern ! So i have a feeling CTC will not work for us. But juxtaposed to that is the multiclass classification approach (where we just predict every individual char) which has this big problem about how to split any image into specific characters before passing it to a CNN layer. We are a little clueless about the approach to be taken here. Any pointers will be heartily appreciated :)

CTC can also work with totally random text (and thus without any "word pattern").
I know that because I trained an OCR network with CTC loss on random text which was then able to read non random text (fortunately).
The CTC just sums the probabilities of the all the paths which are "compatible" with the target text (random or not).
The higher the sum of probas of "compatible" paths, the better the prediction.
If not done yet, you should read this very good article explaining the CTC loss.

Edit to clarify and answer questions asked in comments :

do u mean specifically alphanumeric strings ?
Yes I mean random string with any character, alpha or numeric, it does not matter (even Chinese or Arabic alphabet if you want).
it needs to have seen a lot of data with patterns to generalize weights for transition from C->A->T, correct ?
No, when you train on random text, the network is not using context of previous chars to predict current time step but it is still forced to predict each time step (slice of text) correctly because of the CTC loss. The CTC loss pushes the CNN to compute good features and the RNN to give good probabilities by character for all time steps.
In other words, even with random text, the CTC loss still pushes the network to predict each char correctly independently of the other chars of the text.
And actually a system trained on random text generalizes better on unseen words than a system trained on real words because the latter is biased toward words it has seen during training.
For example a system trained on random text with Latin alphabet could read English, French and German but a system trained only on English words won't work well to read another language with same alphabet.
how can we ever generalize A623BBCEF ??
Just because a network trained on random text learnt to read each char independently of other chars.
the output of such a RNN is a matrix containing character probabilities for each time-step" ..how do we calculate character probabilities ?
The RNN takes the feature map for the current time step and compute a "score vector" from it. Sigmoid activation is used to map the scores to probabilities (between 0 and 1).
In the end it is just a fully connected layer with as many units as characters with a sigmoid activation on each unit.


Is cross-entropy a good cost function if I'm interested in the probabilities of a sample belonging to a certain class? I'm training a neural network that, for each of six classes, tries to predict the probability that a sample belongs to it. After that, I want to use these probabilities as fractions of the sample belonging to that class. My network gives softmax output and is trained using cross-entropy costs (in fact linear output which is then transformed to softmax by tf.nn.softmax_cross_entropy_with_logits)

Is this the right cost function to use when I want to train the network in getting all 6 probabilities correct instead of just classifying each sample as one of the 6 classes? I started hesitating because the tensorflow documentation says about tf.nn.softmax_cross_entropy_with_logits:

  Measures the probability error in discrete classification tasks in which the classes are mutually exclusive (each entry is in exactly one class).

UPDATE
Even though it sound strange, the cross-entropy cost function seems to work best here. In my understanding this is because the cross-entropy for discrete probability distributions (wikipedia) is

  $H(p, q) = -\sum_x p(x)\, \log q(x).$

This function is minimal when $p(x) = q(x)$ for all $x$. This explains why minimizing the cross-entropy error forces the output distribution $q$ to the target distribution $p$, even though I'm abusing tf.nn.softmax_cross_entropy_with_logits by instead of feeding it labels I feed it a discrete probability distribution.

Why it works better than MSE probably is because the magnitudes of the 6 class probabilities are very different. Class 1 may have a probability of 80% while class 2 has a probability of 0.5%, so MSE pays more attention to getting the 80% class right.

Does this mean that CE is the best option still, or is there a way to scale the outputs or weigh them such that the network pays attention to getting each class correct?

Cross entropy is indeed appropriate for multiclass classification.

When the tensorflow documentation states about cross_entropy that:

  "Measures the probability error in discrete classification tasks in which the classes are mutually exclusive (each entry is in exactly one class)."

it is referring to the fact that, at a conceptual level, classes are not expected to overlap. An example of non-overlapping classes are "cat" and "dog". An example of overlapping classes are "Elephant" and "African Elephant"; this type of problem is called multilabel classification, because instead of classes, you have labels and each individual can be assigned many labels.

Update: with the new information about the problem, we can tell that the problem being faced is not a classification one, as the desired output are specific probability values. This implies that the problem is a regression one. As the outputs are probabilities, it is appropriate to use softmax to ensure they add up to 1. As it is regression, it would be appropriate to use MSE as loss function.


Need a Work-around for OneHotEncoder Issue in SKLearn Preprocessing So, it seems that OneHotEncoder won't work with the np.int64 datatype (only np.int32)! Here's a sample of code:

import numpy as np
import pandas as pd
from sklearn.preprocessing import OneHotEncoder
a = np.array([[56748683,8511896545,51001984320],[18643548615,28614357465,56748683],[8511896545,51001984320,40084357915]])
b = pd.DataFrame(a, dtype=np.int64)
ohe = OneHotEncoder()
c = ohe.fit_transform(b).toarray()

When I run this I get the following error: "ValueError: X needs to contain only non-negative integers."
As you can see, X DOES contain only non-negative integers! When I trim a few of the digits and change the datatype to int32 it works fine:

a = np.array([[56748,8511896,51001984],[18643548,28614357,56748],[8511896,51001984,40084357]])
b = pd.DataFrame(a, dtype=np.int32)
ohe = OneHotEncoder()
c = ohe.fit_transform(b).toarray()

Unfortunately, the data I need to encode has 11 digits (which can't be represented by int32). So, any suggestions would be helpful...
Also, I should mention, I don't necessarily need a one hot encoding, just need to create dummy variables. Thanks!


Is it possible to identify different queries/questions in sentence? I want to identifies different queries in sentences.

Like - Who is Bill Gates and where he was born? or Who is Bill Gates, where he was born? contains two queries

Who is Bill Gates?
Where Bill Gates was born

I worked on Coreference resolution, so I can identify that he points to Bill Gates so resolved sentence is "Who is Bill Gates, where Bill Gates was born"

Like wise

MGandhi is good guys, Where he was born?
single query
who is MGandhi and where was he born?
2 queries
who is MGandhi, where he was born and died?
3 quries
India won world cup against Australia, when?
1 query (when India won WC against Auz)

I can perform Coreference resolution (Identifying and converting he to Gandhi) but not getting how can I distinguish queries in it.

How to do this?

I checked various sentence parser, but as this is pure nlp stuff, sentence parser does not identify it.

I tried to find "Sentence disambiguation" like "word sense disambiguation", but nothing exist like that.

Any help or suggestion would be much appreciable.

The basic thing, you can do in that situation, is to split your query into N simple sentences each of which should be processed in order to receive YES/NO answer considering if the sentence is a query. That way you will receive following results:

Input: Gandhi is good guys, Where he was born?
->
Gandhi is good guys - not query
Where he was born?  - query
===
1 query

Input: who is MGandhi and where was he born?
->
who is MGandhi     - query
where was he born? - query
===
2 queries

This approach will require anaphora resolution (in order to convert he into Gandhi in first example) and a parser to correctly divide complex sentence into simple ones.


Reinforcement Learning different patients I have historical data with labels and features about medical data about treating cancer. The labels represents:

How happy is the patient after the treatment (other patients did a survey in the past). From scale 1 till 7.

There features are numerical:
Via facial recognition we extracted six types emotions on a scale from zero till one.

This is performed for patients who are not able to talk and are in electric wheelchair with a tablet.

I trained a random Forrest Regressor. That predicts how happy will be patient on a scale from 1 till 7.

After the patient is done with the treatment the patient fills in a survey behind a screen. Each patient has a unique user ID. How would you apply personalization via reinforcement learning in order to know happy the patient will be during the next treatment.

The happiness of the patients it's face should be updated if this is not correct for this user.

For example:
I want the agent to update the model for this patient it's belief (the old model that we trained) Let's say we predicted 6, but actually it was 4 next time this same patient arrives, we want to estimate the patients happiness with the new updated model.

My model can be correct for patient A, and incorrect for patient B. Because for patient B some emotion was more or less important compared to patient A. How would you solve this problem and it should be scalable for 2000+ patients. Patients with cancer are coming back to the hospital for treatment (on average 10 or 20 treatments). And each patient has an Unique ID.

The model is making the prediction. Based on the predictions the doctors "receive" an happiness estimation.

The model makes: the wrong prediction for patient B but the correct one for patient A. Because patient B indicated after the treatment that the estimation was incorrect based on the scale scores. But Patient A indicated that his prediction was correct.

Let's first begin with a primer to reinforcement learning. Reinforcement Learning is a method of learning that predicts the action to perform at a given state or predicts the value associated with a given state and action pair.

So, Neil is correct that this problem is not explicitly a reinforcement learning problem. This begs the question, how does one define a reinforcement learning problem. It is defined as a 5-tuple containing the following elements. $S$ is the set of states, which must be finite. $A$ is the set of actions available at a given state, it is also sometimes written as $A_s$. $P_a(s, s') = \Pr(s_{t+1}=s' \mid s_t = s, a_t=a)$ is the probability that action $a$ at state $s$ will lead to state $s'$. $R_a(s, s')$ is the reward received when using action $a$ to transition from state $s$ to $s'$. $\gamma$ which is the discount factor which is a value between $0, 1$ inclusive to discount rewards at certain steps.

Now, we have to frame the question in the form of a reinforcement learning problem. You need to establish a state, in your case this would be the information about the patient that you believe has a high correlation with their happiness (or is a causal factor for their emotions). Now, you have a state of your patient with these features you've engineered. Now you must decide how to frame your question appropriately. Here you have two choices, either, you can choose what their emotion is and update your weights (your model weights, or the weights you want to output) as necessary (based on the true labels) or you can use your model to choose an action (in this case it would be a class of 1-7). These approaches are labelled as $Q$ learning or $\pi$ Policy methods. These are the primary methods present in RL, one chooses the value to assign to a given state and action $Q(a, s)$ and the other takes in a state $\pi(s)$ and maps it to an action to perform. You can decide which one you'd like to use based on convenience/experience.

Once this is done, you'd have to decide on an architecture you'd like to use to update the gradients, this is totally up to you. You can do a literature review to see what's going on in this domain and use what you think is appropriate. Just as an example, let's say you'd want to use a Policy method that is updated by a vanilla CNN. You would pass in the state to your CNN in some form that would be interpretable by the CNN and then choose an action based on the class labels. Now, you would have a reward function which would choose how far you are from the true class (you can do this heuristically, like Happiness is 10 units from Sadness). The reward function would be propagated to the network and it would update the weights as required.

PERSONAL COMMENTS: I think this is completely unnecessary for the current task. In my opinion a simple vision classifier should do perfectly well. There are an abundance of datasets that are available which takes in facial features and classifies sentiment/happiness. Also, a lot of statisticians believe that numeric values for happiness/sadness is a bad metric because questions like, "How different is very happy from happy" are hard questions to answer in the context of associating a reward with them. Do take Neil's comments into consideration.


How to make a classification problem into a regression problem? I have data describing genes which each get 1 of 4 labels, I use this to train models to predict/label other unlabelled genes. I have a huge class imbalance with 10k genes in 1 label and 50-100 genes in the other 3 labels. Due to this imbalance I'm trying to change my labels into numeric values for a model to predict a score rather than a label and reduce bias.

Currently from my 4 labels (of most likely, likely, possible, and least likely to affect a disease) I convert the 4 labels into scores between 0-1: most likely: 0.9, likely: 0.7, possible: 0.4, and least likely: 0.1 (decided based on how similar the previous label definitions were in their data). I've been using scatter plots with a linear model to try to understand which model would best fit my data and reducing overfitting, but not sure if there's more I can infer from this except that the data has homoskedasticity (I think? I have a biology background so learning as I go):

I'm not sure if there is a more official way I should be approaching this or if this regression conversion is problematic in ways I don't realise? Should I be trying to develop more scores in my training data or is there something else I should be doing?

Edit for more information: The current 4 labels I have I create based on drug knowledge of the genes and the drug data I currently have for each gene, I could incorporate other biological knowledge I have to make further labels I think.  For example, currently the 'most likely' labelled genes are labelled as such because they are drug targets for my disease, 'likely' label because they are genes which interact with drugs to cause a side effect which leads to the disease, and the other 2 labels go down in relevance until there are least likely genes with no drug related or statistical evidence to cause the disease.

So, the direct answer here is clearly NO.

The answer comes from the definitions of classification and regression. In a classification task what a model predicts is the probability of an instance to belong to a class (e.g. 'image with clouds' vs 'image without clouds' ), in regression you are trying to predict continuous values (e.g. the level of 'cloudiness' of an image).

Sometimes you can turn a regression problem into a classification one. For example if I have a dataset of images with labels of their cloudiness level from 0 to 5 I could define a threshold, i.e. 2.5 and use it to turn the continuous values into discrete one, and use those discrete values as classes (cloudiness level < 2.5 equal image without clouds) but the opposite is definitely not possible.

Here's also a link to a similar question Could I turn a classification problem into regression problem by encoding the classes?

To solve the problem of imbalanced classes there are many ways, not just oversampling, you can generate artificial data, add class weights in the loss function, use active learning to gather new data, use models that returns uncertainty score for each prediction (like bayesian networks), I'm sure here is plenty of answers and strategy you can try.


Fast and/or memory efficient classification solutions I'm working on a classification problem with a very large dataset (a little under 1 billion obs) and around 25 predictors.

I'm doing this analysis in R on a VM with 128GB of memory, but am still hitting memory issues when training certain types of models, not to mention having to wait for quite a while for runs to complete. I'm primarily using logistic regression and random forests, and keeping my training dataset to 10% of the overall sample.

What are some solutions (packages, platforms, techniques) I could use to address these memory and/or speed issues? I'm fairly comfortable with Python, so solutions do not have to be R-specific.

LightGBM is the answer you are looking for. It uses less memory than xgboost, catboost and the fastest GBM library. Also it has a RF mode which is faster than sklearn implementation. It works on both R and Python. Also it has some parameters to control memory management.
You can check this benchmark repo.


Neural Nets: Backpropogation from Conjugate Gradient Descent In neural nets, the back-propagation algorithm is 'based on' gradient descent, optimizing over a cost function $C(w,b)$ of the weights and biases of the network.

Conjugate gradient descent has way faster convergence than standard gradient descent (and stochastic gradient descent). Is there a similar back-propagation algorithm based off of conjugate gradient descent?

Yes,

see the article Optimization in an error backpropagation neural network environment with a performance test on a spectral pattern classification problem


How is GPT able to handle large vocabularies? From what I understand, GPT and GPT-2 are trained to predict the $N^{th}$ word in a sentence given the previous $N-1$ words. When the vocabulary size is very large (100k+ words) how is it able to generate any meaningful prediction? Shouldn't it become extremely difficult to correctly label the next word given that there are 100k possible labels to choose from? Even a large-scale classification problem like ImageNet has only 1k classes to choose from.

GPT-2 does not use a word-level vocabulary but a subword-level vocabulary, specifically byte-pair encoding (BPE). This means that it does not predict the next word but the next subword token.
BPE tries to find the pieces of words that are most reusable. BPE also keeps character subwords (e.g. "a", "W").
The subword vocabulary used by GPT-2 is 50k tokens big. You can take a look at the vocabulary here. There, you can see that token #12175 is "retro" and token #11147 is "fit", so when tokenizing word "retrofit", you would probably get the 2 subword tokens #68 and #12541. When tokenizing a low-frequency word (i.e. that did not appear in the BPE training data), you probably end up with small subwords tokenization, e.g. "Kilimanjaro" --> "Kil", "iman", "jar", "o".
Using a word-level vocabulary is very troublesome, because:

the amount of existing word surface forms exceeds the size that it is manageable for a neural net, even for a morphologically simple language like English. For morphologically-rich fusional languages and, especially, agglutinative languages, using word-level vocabularies is much more inconvenient.
language is "open", in the sense that new words can be created, either by stitching together existing words (e.g. manspreading), or by making up new words altogether (e.g. bazinga).
the less frequently a token appears in the training data, the less our net will learn to use it. If we have many many different tokens, the frequency of appearance of each token in the training data will be very low (i.e. data scarcity), so our net will not learn properly.

Therefore, subword vocabularies are the norm nowadays.


For what condition boosting work better than bagging in Ensemble Learner? In boosting process we get a better accuracy for training data, but there is a lot of chance to over fit. For bagging ensemble method over fitting chance is lower than boosting.
Why we use boosting and for what condition ??

Bagging (and features sampling) aim to reduce variance by providing low-correlated trees. Estimators can then be aggregated together to reduce variance. Reason is simple decision trees tend to quickly overfit in the bottom nodes. Bagging and features sampling only adress high-variance (overfitting) problems.

On the other hand, boosting is a meta-algorithm that aims to reduce both bias and variance by training estimators in a sequential way. Each predictor tries to reduce residuals from previous estimators.

In my opinion, the two meta-algorithms can't be easily compared that way. They will give you different results depending of data distributions, potential outliers, usecase... Potentially, boosting is more sensitive to outliers than bagging for example. I think trying the 2 methods is your best chance and you don't have to restrict to only one model.


Which model may be best for outcome of a surgery? New to data science and am trying to be a self-starter and implement advanced data analytics in my subspecialty of surgery. Below is a description of my data set. I know that I will have to explore multiple methods, but wanted to get your take on which you think may be best. I will most likely be using R to achieve this analysis.

Have a data set with about 200 patients (rows)
Each patient has about 10-15 variables (preoperative and intraoperative)
Each patient has undergone either nonoperative or operative management
Success in nonoperative or operative management is determined by a questionnaire that patients fill out 1 year after they are seen. This questionnaire gives a binary outcome on whether they (1) Benefited or (2) Did not Benefit from the surgery.

My questions for the study are as follows:

In the surgery group, I am trying to find out which variables lead to patients (1) Benefit vs (2) Do not benefit from surgery, and create a model which can better help predict which patients we can operate on (I have left out some details such as patient population, type of surgery, etc).
In the second study, I would like to determine which patients we should operate on. In other words, I would like to find out which preoperative characteristics make some patients more likely to benefit from (1) Operative treatment vs (2) Nonoperative treatment and in this case the outcome will also be binary from the questionnaire.

I have tried linear and logistic regressions for this which have not been very good, hence why I am trying to learn more advanced models.

Models which are easier to comprehend by clinicians are more valuable which is why I haven't delved into neural nets. I appreciate any and all advice that can be provided. In addition, if I expand this data set to 600 people, would you use another model? I don't have access to large servers so most of this will be done on my laptop though I can use online resources if necessary (Azure etc).

Thank you all for your help and input.

Question 1: I would propose Decision Trees as a first thing you should look into, as they are easy to implement and the results are straightforward to interpret.
You will end up with a tree-structure, where the nodes hold intervals/values of your variables. At each node, the tree tries to split your data most efficiently into your binary classifications $C_1$ and $C_2$. Thus, in the end, you will be able to extract most important features in your variables at the top of the tree. A typical measure of "efficiency" when selecting attributes is Entropy.

Question 2: I think it is also solved with the above method. For each node of the tree you know the amount of $C_1$ and $C_2$ classifications on the left and on the right. Assume your attribute in the node is "Has diabetes". Your data (and thus the tree) tells you, 100 out of 110 patients "failed" the operation if the person had diabetes. Thus, you can give an estimate about how certain attributed contribute to success of your operations, and conclude it would not make much sense to operate the given person.

Nonetheless, always be careful about how you interpret the results. You should not try to interpret attributes that split small subgroups, since this can easily lead to false assumptions about real-world behaviors.


Why doesn't neural networks use the concept of degree of freedom? In most (if not all) NMIST neural network tutorials you will see that the last two layers reduce to a multi-layer perceptron (MLP) and the number of labels is 0-9 for a total of 10 labels. It is well known in statistics that when you have 10 labels, you can set the score of one of them to be 0 and let the other 9 vary. This is the idea of degrees of freedom where 9 variables are allowed to "run free" but one can stay fixed leading to 9 degrees of freedom. More intuitively, when you calculate the probability that an image is a particular number you only need to specify the probabilities of it being one of the 9 numbers, because the probabilities for all ten labels must sum to 1 so the remaining label's probability must be 1 - sum(of the other 9).

The question is: why doesn't neural networks make use of this degree of freedom idea and instead estimate a score for all 10 labels instead of fixing the score of one of them to 0?

Actually neural networks do take this information into account.

When we do a hard classification with NN (i.e. only one class for the output), we (almost always) use a softmax output layer, then taking the max of the softmax layer to get the class and its probability.

Then why do we still have 10 outputs before the softmax and why 10 after? Well, you need to have the 10 inputs to the softmax to indicate the weight the network gives to each class, as this is still not scaled back to a probability. Then to ease the usage of the output layer, we want to keep having 10 outputs to be able to use the values without reconstructing the missing value (which would need to be computed anyway due to the way softmax works).


Use Cases of Neo4J and Spark GraphX I have used Neo4J to implement a content recommendation engine. I like Cypher, and find graph databases to be intuitive.

Looking at scaling to a larger data set, I am not confident No4J + Cypher will be performant. Spark has the GraphX project, which I have not used in the past.

Has anybody switched from Neo4J to Spark GraphX? Do the use cases overlap, aside from scalability? Or, does GraphX address a completely different problem set than Neo4J?

Neo4j and Spark GraphX are meant for solving problem at different level and they are complimentary to each other.

They can be connected by Neo4j's Mazerunner extension:

  Mazerunner is a Neo4j unmanaged extension and distributed graph
  processing platform that extends Neo4j to do big data graph processing
  jobs while persisting the results back to Neo4j.

  Mazerunner uses a message broker to distribute graph processing jobs
  to Apache Spark's GraphX module. When an agent job is dispatched, a
  subgraph is exported from Neo4j and written to Apache Hadoop HDFS.

  After Neo4j exports a subgraph to HDFS, a separate Mazerunner service
  for Spark is notified to begin processing that data. The Mazerunner
  service will then start a distributed graph processing algorithm using
  Scala and Spark's GraphX module. The GraphX algorithm is serialized
  and dispatched to Apache Spark for processing.

  Once the Apache Spark job completes, the results are written back to
  HDFS as a Key-Value list of property updates to be applied back to
  Neo4j.

  Neo4j is then notified that a property update list is available from
  Apache Spark on HDFS. Neo4j batch imports the results and applies the
  updates back to the original graph.

Check out this tutorial to get an idea on how to combine the two:
http://www.kennybastani.com/2014/11/using-apache-spark-and-neo4j-for-big.html


How to obtain the word vectors optimally I have a list of strings as shown

sent_list = ["Carrefour is in France", "Apple pie is delicious", "Amazon has just delivered", ...]

My code to get word embeddings below

import spacy

nlp = spacy.load("en_trf_bertbaseuncased_lg")

for sent in sent_list:
    print(nlp(sent).vector)

This takes considerable time when the list is of large size (>10000). I tried disabling sentencizer within the nlp pipe but with not much improvement. How can this be optimized for shorter run time?

Use nlp.pipe() to process texts in larger batches, which is much faster, especially for a lot of short texts:

for doc in nlp.pipe(sent_list):
    # averaged doc vector
    print(doc.vector)
    # token vectors
    print([token.vector for token in doc])


data splitting into 3 sets based on years let's suppose we have a customer data from the year 2015 to 2019, I want to train_test_split() my data such that my data gets divided into three sets, set-1 is from 2015 to 2017 (3 years) on which i will train my model, set-2 i.e. 2018(1 year) on which i will validate my model , set 3 is 2019(1 year) on which I will test my model. I want a code to divide data into 3 sets based on time(years).

Seems to me the best (or at least quickest) way to do this would be have all the data in a Pandas dataframe, then create masks based on year and create new dataframes for each group. Ex:

train_df = data[data['year'].isin(['2015', '2016', '2017'])
validate_df = data[data['year'] == '2018']
test_df = data[data['year'] == '2019']

Hope this is what you're looking for. If not, let me know and we can work out another solution.


Should dimensionality reduction be done before k-means clustering if there are many features? My data contains over 200 features and over 500 observations. I want to place the observations into a number of clusters based on the features that make them different.

There are numerous ideas I have and I'm not sure which one is apt:

1) Conduct principal component analysis (PCA) to reduce the features to two dimensions. I've already done this so that I could visualize them on a 2D plot. It would now be quite easy to use k-means clustering with these two dimensions to create the clusters, but I wonder if this isn't a good idea because of all the components that are being lost. But then again, if they're being lost they're probably not that important? Not sure

2) Conduct principal component analysis (PCA) to determine which features are worth including and then conduct k-means clustering on those features. So I probably wouldn't be reducing the dimensions to two, but they would be reduced and then the k-means clustering would be done. This seems like the best idea intuitively to me, but I'm not sure.

3) Forget the PCA and just conduct k-means clustering on all the features I have at the beginning. This feels like it's probably the worst idea because some of the features could be useless but could still be factored into the distance calculations for the clustering, but I'm just including everything I've thought of.

For the first idea about PCA, you can not simply just use 2 components. You need to take a look at the explained variance by your principal components and based on that you should select the required number of components. If, for example, you found that the first two components explain a significant amount of variance (e.g., more than 95%), then, you can use them to perform k-means clustering. In that case, it is expected (but not for sure and for all cases) that you get the same results as when you perform k-means using all features.

My suggestion is to use all of your features if you need to consider a large number of components. My reason is that your dataset is too small and it is not computationally demanding to perform k-means clustering using your dataset.

As a side note, also you can try all of your options because your dataset is small and then you will find out what's going on in your data.


How to visualize (make plot) of regression output against categorical input variable? I am doing linear regression with multiple variables. In my data I have n = 143 features and m = 13000 training examples. Some of my features are continuous (ordinal) variables (area, year, number of rooms). But I also have categorical variables (district, color, type). For now I visualized some of my feautures against predicted price. For example here is the plot of area against predicted price:

Since area is continuous ordinal variable I had no troubles visualizing the data. But now I wanted to somehow visualize dependency of my categorical variables (such as district) on predicted price.
For categorical variables I used one-hot (dummy) encoding.
For example that kind of data:

turned to this format:

If I were using ordinal encoding for districts this way:

DistrictA - 1
DistrictB - 2
DistrictC - 3
DistrictD - 4
DistrictE - 5

I would plot this values against predicted price pretty easy by putting 1-5 to X axis and price to Y axis.

But I used dummy coding and now I do not know how can I show (visualize) dependency between price and categorical variable 'District' represented as series of zeros and ones.

How can I make a plot showing a regression line of districts against predicted price in case of using dummy coding?

One possible first step is to convert the data back to the original coding.
This is called in SQL unpivot, in R melt.

Here an R example

> my.df <- read.table(
+ text = "DistrictA     DistrictB    DistrictC    DistrictD     DistrictE     Price
+         1             0            0            0             0             10000
+         0             1            0            0             0             20000
+         0             0            1            0             0             30000
+         0             0            0            1             0             40000
+         0             0            0            0             1             50000"
+      , header = TRUE)
> my.df
  DistrictA DistrictB DistrictC DistrictD DistrictE Price
1         1         0         0         0         0 10000
2         0         1         0         0         0 20000
3         0         0         1         0         0 30000
4         0         0         0         1         0 40000
5         0         0         0         0         1 50000

> library(reshape)
> subset(melt(my.df, id="Price", variable = "District"),value == 1)[,c(1,2)]
   Price  District
1  10000 DistrictA
7  20000 DistrictB
13 30000 DistrictC
19 40000 DistrictD
25 50000 DistrictE

After that you plot the Price dependent on a factor variable. You may additionally consider to order the factor based on the predicted price.

I provide no details, as you don't tagged your tool, but I would recommend additional to a scatter plot to consider a box plot and/or density plot - always combined with the prediction value from the model for each factor level.


Best approach for detecting human in a picture I am trying to detect if an image contains human or not. I will use CNN for this. I am planning to make a dataset like following.

Train
     human (contains 2000 or more human image)
     other object (contains 2000 or more non human image)

Test
     human (contains 500)
     other object (contains 500 )

I am planning to use keras and I will train data from directory .

Is there any alternative approach ? Because I am having trouble with collecting human and non human object image.

I had a similar problem with skin disease classification. You can see the question and its answers too here.  Gathering a dataset of human pictures could be easy but that of non-human could be nearly impossible ( non-human objects are a huge set of objects )

Solution:

We can use a Siamese Convolutional Neural Network. I have written a blog on it. See here. They perform one-shot learning and are best when we have smaller datasets.

  A standard CNN tries to classify whether the pictures belongs to any one of the classes -> (HUMAN, NON_HUMAN ). The Siamese CNN outputs a similarity score rather than class probabilities. A standard CNN learns to classify and on the other hand, a Siamese CNN learns to differentiate images.

How does it work? ( In the context of your task )

Suppose I have an input image $x$. This is our input for the Siamese CNN. I have two more images, say $f$ and $g$. Image $f$ belongs to class 1 ( HUMAN ) whereas $g$ belongs to CLASS 2 ( NON-HUMAN ). The Siamese CNN takes in 2 images as input and outputs how similar they are.

$ Siamese( x , f ) = 0.678$

$ Siamese( x , g ) = 0.987$

$Clearly, \space Siamese( x , f ) < Siamese( x , g )$

Therefore, the image belongs to CLASS 2. You can find its other advantages on the blog. Hope this helps.


Are all DNN models supervised learning (or reinforcement learning)? Are there any type of unsupervised learning models in DNN?

There are many types of unsupervised learning models in deep neural networks. One common type is an autoencoder, which learns a compressed representation of the data.


Why do my Latent Dirichlet Allocation Topics mix words that never co-occurred? I have one corpus of documents on diabetes, another on Leonardo da Vinci, and another on animation and computer graphics.  I combined all of these documents into a LDA and got a topic like the one below.  I'm listing the top 30 terms, in descending order.

TOPIC 83: ['produced', 'company', 'john', 'weight', 'motion', 'information', 'painting', 'original', 'training', 'people', 'health', 'life', 'jesus', 'feature', 'body', 'lucretia', 'computer', 'graphics', 'time', 'madonna', 'story', 'florence', 'type', 'animated', 'paul', 'diabetes', 'animation', 'exercise', 'peter', 'film']

Many of these words do not co-occur with each other in any documents.  For instance, 'lucretia' does not co-occur with 'exercise'.  Yet how are these terms put together in the same topic, in the top 30 words? (30 out of 20K words or so)

Perhaps it has to do with my particular implementation?  I'm using the gensim library for Python.  Or is it a flaw of LDA generally?

It could be partially due to the number of topics you selected but the fact that two words rank high for a given topic doesn't necessarily mean that the two words will frequently occur in the same document. Consider a topic that generally corresponds to "medicine". You could have a number of medical documents associated with heart disease and a number of medical documents associated with injuries. They may all be linked by common terms such as "diagnose", "treatment", "patient", etc. But you wouldn't necessarily expect that words like "heart" and "aorta" would appear in documents containing words such as "sprain" and "fracture", even though they are all associated with the general topic of medicine. If one were to choose a larger number of topics, those words might be separated into more specific topics (e.g., "heart disease" and "sports injuries").


How can I get an objective measure of song similarity? I was browsing through ML project ideas and found an interesting one (just the problem statement ) which was: detecting if two songs are similar using lyrics. I found it to be an interesting idea but Im not quite sure how I'd go about getting a score of similarity for the songs. For my dataset, I have features of genre, artist and lyrics. What is a potential method of scoring the similarity considering there is no such 'training data' to begin with.

I have come across word embeddings and stuff but their working isnt completely clear to me. Moreover, I think they dont take the song-like features that are available into account: things like type-token ratio, sentiment rating, word density(average number of words per sentence)etc.

Could an approach where first the songs were first clustered based on "high level features" such as type-token ratio, sentiment etc followed by a semantic similarity measure i.e. something like a cosine similarity metric on the word embeddings of songs in the same cluster make sense? How would I validate the usefulness of such an approach?

Two songs are two separated documents which have characteristics that make them similar or non-similar. There are plenty of techniques to determine similarness between documents:

How to compute the similarity between two text documents


How is the validation set processed in PyTorch? Say, one uses the MNIST dataset and splits the provided training data of size 60,000 into a training set (50,000) and a validation set (10,000). The provided test data of size 10,000 is used as the test set. The ML algorithm is a neural network.
The training set is processed (in minibatches) by the code below. First, one sets the gradients to zero. Then, the model makes a prediction, and the loss is calculated. Next, the gradients are computed, and the weights are updated via backpropagation.
def train(data, label):
    model.zero_grad()
    prediction = model(data)
    loss = loss_function(prediction, label)
    loss.backward()
    optimizer.step()
    return loss

As I understand, the validation set is used for hyperparameter tuning, whereas the test set is used for evaluation of the final model (as a reference to compare performance to other models). The accuracy on the test set is measured after "freezing" the model, like in the code below.
for parameter in model.parameters():
    parameter.requires_grad = False
model.eval()

So, my questions are:

When processing the validation set, is it correct to use the code of
train() or must one omit the backpropagation?

If one assumes that a neural network applies dropout, is dropout
enabled or disabled while processing the validation set?

With regards to the backpropagation, since the network is not trained on the validation set and parameters are not updated we do not have to use backpropagation. The calculation of gradients can be disabled by using the torch.no_grad() context manager from pytorch. By not calculation gradients the model is able to process data faster. Dropout is turned off when setting the model in evaluation mode using model.eval() and is often used in combination with torch.no_grad() to turn off the computation of gradients, see also this stackoverflow answer and the pytorch documentation.


What could be some Classification techniques to classify a tree of webpages given the category of each webpage I want to perform a website classification task where I have modeled a website as a tree of webpages. I already have a model which can assign categories to the nodes in the tree (webpages). I need guidance on how can I combine those node categories to get the overall category of the tree. What classification techniques could be used ?
As of now I know about usage of markov chains for this task from this research paper, Web Site Mining. I would be grateful if i could get some more ideas about how to perform the task.

There are of course a number of ways this can be done, such as majority voting or some other rule-based algorithm, however it can also be done through supervised learning since you have some labels for the trees.

I would make the input space of my model the normalized frequency of the categories for a tree. This means you will need a dictionary of possible categories for the nodes, usually obtained from your training set. Then you can tabulate the frequency of instances.

For example if we have a website with the following node classes:

News: 5
Opinions: 9
About: 1

Then we can formulate our input vector as $[0.33, 0.6, 0.067]$.

You can then train this model using your already labeled trees. The model will then be capable of classifying future trees in this same way.

To determine the top $K$ classes for a tree you will need a model which can do this (most can). If you use K-NN (different K) then you can pick the $K$ closest neighbourhoods. With Random Forests or Naive Bayes you can pick the $K$ classes with the highest probabilities.

To consider the fact that you have a list of $K$ categories for each node you can add a weighting when calculating the normalized frequencies. For example let's say we have 3 classifications and the following webpages (nodes).

Page 1: News, Opinion, Commentary
Page 2: News, Advertisement, Opinion
Page 3: Commentary, News, Adverisement
Page 4: News, Opinion, Advertisement

Then the input vector can be calculated by awarding 3 points to the first category, 2 to the next and 1 to the last. This results in $[0.49, 0.21,0.17, 0.17]$. Alternatively, if you have a probability for these classifications you can use that as the weighting factor.


Stochastic gradient descent based on vector operations? let's assume that I want to train a stochastic gradient descent regression algorithm using a dataset that has N samples. Since the size of the dataset is fixed, I will reuse the data T times. At each iteration or "epoch", I use each training sample exactly once after randomly reordering the whole training set.

My implementation is based on Python and Numpy. Therefore, using vector operations can remarkably decrease computation time. Coming up with a vectorized implementation of batch gradient descent is quite straightforward. However, in the case of stochastic gradient descent I can not figure out how to avoid the outer loop that iterates through all the samples at each epoch.

Does anybody know any vectorized implementation of stochastic gradient descent?

EDIT: I've been asked why would I like to use online gradient descent if the size of my dataset is fixed.

From [1], one can see that online gradient descent converges slower than batch gradient descent to the minimum of the empirical cost. However, it converges faster to the minimum of the expected cost, which measures generalization performance. I'd like to test the impact of these theoretical results in my particular problem, by means of cross validation. Without a vectorized implementation, my online gradient descent code is much slower than the batch gradient descent one. That remarkably increases the time it takes to the cross validation process to be completed.

EDIT: I include here the pseudocode of my on-line gradient descent implementation, as requested by ffriend. I am solving a regression problem.

Method: on-line gradient descent (regression)
Input: X (nxp matrix; each line contains a training sample, represented as a length-p vector), Y (length-n vector; output of the training samples)
Output: A (length-p+1 vector of coefficients)

Initialize coefficients (assign value 0 to all coefficients)
Calculate outputs F
prev_error = inf
error = sum((F-Y)^2)/n
it = 0
while abs(error - prev_error)>ERROR_THRESHOLD and it<=MAX_ITERATIONS:
    Randomly shuffle training samples
    for each training sample i:
        Compute error for training sample i
        Update coefficients based on the error above
    prev_error = error
    Calculate outputs F
    error = sum((F-Y)^2)/n
    it = it + 1

[1] "Large Scale Online Learning", L. Bottou, Y. Le Cunn, NIPS 2003.

First of all, word "sample" is normally used to describe subset of population, so I will refer to the same thing as "example".

Your SGD implementation is slow because of this line:

for each training example i:

Here you explicitly use exactly one example for each update of model parameters. By definition, vectorization is a technique for converting operations on one element into operations on a vector of such elements. Thus, no, you cannot process examples one by one and still use vectorization.

You can, however, approximate true SGD by using mini-batches. Mini-batch is a small subset of original dataset (say, 100 examples). You calculate error and parameter updates based on mini-batches, but you still iterate over many of them without global optimization, making the process stochastic. So, to make your implementation much faster it's enough to change previous line to:

batches = split dataset into mini-batches
for batch in batches:

and calculate error from batch, not from a single example.

Though pretty obvious, I should also mention vectorization on per-example level. That is, instead of something like this:

theta = np.array([...])  # parameter vector
x = np.array([...])      # example
y = 0                    # predicted response
for i in range(len(example)):
    y += x[i] * theta[i]
error = (true_y - y) ** 2  # true_y - true value of response

you should definitely do something like this:

error = (true_y - sum(np.dot(x, theta))) ** 2

which, again, easy to generalize for mini-batches:

true_y = np.array([...])     # vector of response values
X = np.array([[...], [...]]) # mini-batch
errors = true_y - sum(np.dot(X, theta), 1)
error = sum(e ** 2 for e in errors)


Channels in convolutional layer I usually see convolutions performed over all the channels of the input. For example a $3x3$ kernel is really a $3x3xN$ kernel for a an input with $N$ channels, thus resulting in a single output channel per filter.

What would happen, if you were to do a convolution on individual channels or pairs of channels instead, with independent filters per channel? I am wondering if when certain channels are not correlated could you prevent some information loss by keeping the channels separate for some layers?

The technique you have mentioned is called "Depth-wise Separable Convolutions" and it relies on the idea that spatial and depth-wise information can be decoupled. Its main advantage is that compared with standard convolution, since it doesn't need to perform convolution across all channels, it has considerably less connections, (so less parameters) which results in a lighter model.

There is an architecture named "Xception" ("Extreme version of Inception") by Google that uses a version of such convolutional layers and produces close to state-of-the-art results. There is a really good review of it at Towards Data Science.

Personally I don't like the idea of decoupling the channels, because I think it gets us a less "sophisticated" model, but it seems to work so who am I to judge?


how to create word2vec for phrases and then calculate cosine similarity I have just started using word2vec and I have no idea how to create vectors (using word2vec) of two lists, each containing set of words and phrases and then how to calculate cosine similarity between these 2 lists.

For example :

list1 =['blogs', 'vmware', 'server', 'virtual', 'oracle update', 'virtualization', 'application','infrastructure', 'management']
list2 = ['microsoft visual studio','desktop virtualization',
'microsoft exchange server','cloud computing','windows server 2008']

Any help would be appreciated.

You cannot apply word2vec on multiple words. You should use something like doc2vec, which gives a vector for each phrase:

phrase = model.infer_vector(['microsoft', 'visual', 'studio'])

You can also average or sum the vectors of words (from word2vec) in each phrase, e.g.

phrase = w2v('microsoft') + w2v('visual') + w2v('studio')

This way, a phrase vector would be the same length as a word vector for comparison. But still, methods like doc2vec are better than a simple average or sum. Finally, you could proceed to compare each word in the first list to every phrase in the second list, and find the closest phrase.

Note that a phrase like "cloud computing" has a completely different meaning than the word "cloud". Therefore, these phrases, specially if frequent, better to be treated as a single word, e.g.

phrase = w2v('cloud_computing')

Extra directions:

Here is an answer by Astariul on stackoverflow that uses a function from word2vec package to calculate similarity between two sets of words.
Take a look at fastText that works better when there is a lot of misspelled, or out-of-vocabulary words.


How to annotate text documents with meta-data? Having a lot of text documents (in natural language, unstructured), what are the possible ways of annotating them with some semantic meta-data? For example, consider a short document:

I saw the company's manager last day.

To be able to extract information from it, it must be annotated with additional data to be less ambiguous. The process of finding such meta-data is not in question, so assume it is done manually. The question is how to store these data in a way that further analysis on it can be done more conveniently/efficiently?

A possible approach is to use XML tags (see below), but it seems too verbose, and maybe there are better approaches/guidelines for storing such meta-data on text documents.

<Person name="John">I</Person> saw the <Organization name="ACME">company</Organization>'s
manager <Time value="2014-5-29">last day</Time>.

Personally I would advocate using something that is both not-specific to the NLP field, and something that is sufficiently general that it can still be used as a tool even when you've started moving beyond this level of metadata. I would especially pick a format that can be used regardless of development environment and one that can keep some basic structure if that becomes relevant (like tokenization)

It might seem strange, but I would honestly suggest JSON. It's extremely well supported, supports a lot of structure, and is flexible enough that you shouldn't have to move from it for not being powerful enough. For your example, something like this:

{'text': 'I saw the company's manager last day.", {'Person': [{'name': 'John'}, {'indices': [0:1]}, etc...]}

The one big advantage you've got over any NLP-specific formats here is that JSON can be parsed in any environment, and since you'll probably have to edit your format anyway, JSON lends itself to very simple edits that give you a short distance to other formats.

You can also implicitly store tokenization information if you want:

{"text": ["I", "saw", "the", "company's", "manager", "last", "day."]}

EDIT: To clarify the mapping of metadata is pretty open, but here's an example:

{'body': '<some_text>',
 'metadata':
  {'<entity>':
    {'<attribute>': '<value>',
     'location': [<start_index>, <end_index>]
    }
  }
}

Hope that helps, let me know if you've got any more questions.


Is there any conceptual relationship between 'kernel' in SVM and 'kernel' in convolution neural net? In SVM, we have kernel function that maps an input raw data space into a higher dimensional feature space

In CNN, we also have a 'kernel' mask that travels the input raw data space (image as a matrix) and map it to another space.

Given the fact that both these to methods are called 'kernel', I am wondering what is the connection between them from a mathematical perspective.

My guess is that it might have something to do with functional analysis.

There is no direct relationship between these two concepts. However we can find some indirect ones.

According to Merriam Webster,

  kernel means a central or essential part

which hints why they are called "kernel". Specifically, deciding "how to measure point-point similarity (a.k.a. kernel function)" is the central part of kernel methods, and deciding "what array, matrix, or tensor (a.k.a. kernel matrix) to be convoluted with a data point" is the central part of convolutional neural networks.

A kernel function receives two data points, implicitly maps them into a higher (possibly infinite) dimension, then calculates their inner product.

A kernel matrix (or array, or tensor) is convoluted with one data point to map the data point explicitly into an often lower dimension. Here, we are ignoring a subtle difference between filter and kernel (a filter is composed of one kernel per channel).

Therefore, these two concepts are indirectly related based on mapping to a new representation. However,

Kernel functions map implicitly, but kernel matrices map explicitly,
Kernel functions cannot be stacked over each other (shallow representation), but kernel matrices can be since the input and output (explicit representations) has the same structure (deep representation),
The non-linearity of map is integrated into kernel functions, but for kernel matrices, we should apply a non-linear activation function after the (input, kernel) convolution to reach a similar non-linearity,
Implicit representations cannot be learned for kernel functions, a specific function implies a specific representation. However, for kernel matrices, representations can be learned by adjusting (learning) the weights of kernels, and can also be enriched by stacking kernels over each other.


How to create a feature vector given final set of feature maps? I've got a faster-rcnn (resnet-101 backbone) for object detection, and am extracting feature tensors for each detected object, which is a 7x7x2048 tensor (basically 2048 feature maps, each 7x7). For object tracking, I want to turn this into a Nx1 vector. What is the standard way to do this? I have a few ideas that all seem reasonable:

Flatten each feature map, and then concatenate all these together (so each feature vector would be 49*2048 x 1)?
Do the same after doing some max pooling operation to decrease dimensionality.
Take the mean or max of each feature map, and end up with a 2048x1 feature vector.

After asking around about this, it seems the third option is the standard: take the mean of each feature map, and create a 2048-element feature vector. The search term for this is global pooling. That's what we are talking about that is the terminology I was missing.

Global average pooling is good b/c it reduces the dimensionality before classification. Also, by the time you have reached this level of abstraction in your feature extractor, you are probably not that interested in the finer-grained spatial aspects of the features because you are already representing a very abstract "object" (depending on how deep your feature extractor is) after a great deal of pooling of your original image.

That said, if you are worried about losing spatial information you can always try one of the other two options. I'd be curious if anyone has tried this if they have experiences to share about how they went about it.

Also, for a really nice summary from a paper that addresses this topic, see this answer from crossvalidated:
https://stats.stackexchange.com/a/308218/17624

IN particular:

  Conventional convolutional neural networks perform convolution in the
  lower layers of the network. For classification, the feature maps of
  the last convolutional layer are vectorized and fed into fully
  connected layers followed by a softmax logistic regression layer. This
  structure bridges the convolutional structure with traditional neural
  network classifiers. It treats the convolutional layers as feature
  extractors, and the resulting feature is classified in a traditional
  way.

  However, the fully connected layers are prone to overfitting, thus
  hampering the generalization ability of the overall network. Dropout
  is proposed by Hinton et al as a regularizer which randomly sets half
  of the activations to the fully connected layers to zero during
  training. It has improved the generalization ability and largely
  prevents overfitting.

  In this paper, we propose another strategy called global average
  pooling to replace the traditional fully connected layers in CNN. The
  idea is to generate one feature map for each corresponding category of
  the classification task in the last mlpconv layer. Instead of adding
  fully connected layers on top of the feature maps, we take the average
  of each feature map, and the resulting vector is fed directly into the
  softmax layer. One advantage of global average pooling over the fully
  connected layers is that it is more native to the convolution
  structure by enforcing correspondences between feature maps and
  categories. Thus the feature maps can be easily interpreted as
  categories confidence maps. Another advantage is that there is no
  parameter to optimize in the global average pooling thus overfitting
  is avoided at this layer. Futhermore, global average pooling sums out
  the spatial information, thus it is more robust to spatial
  translations of the input. We can see global average pooling as a
  structural regularizer that explicitly enforces feature maps to be
  confidence maps of concepts (categories). This is made possible by the
  mlpconv layers, as they makes better approximation to the confidence
  maps than GLMs.

Which is from this paper: https://arxiv.org/pdf/1312.4400.pdf


In batch normalization, shouldn't using DropConnect harm test accuracy? In my understanding of batch normalization, mean and variance are calculated over the entire batch and then added to the population average. This average is then applied to the test set to estimate the mean and variance of the overall test set.
However, DropConnect removes connections during training as a form of network regularization. The model calculates batch statistics through a network that is missing usually half its connections. At test time, all connections are used. Shouldn't this affect the mean and variance of the population and/or the test set, throwing the estimated mean and variance off, or will the network stabilize itself over time?

TL;DR

This is an interesting idea and probably best to be tested with your specific problem; however it is generally understood that you will get better results by not using both Dropout (including DropConnect) and Batch-Norm together, given their overall effects during training. Recent evidence/tests. Having said that, I think it would balance out in the end anyway, as you propose.

More considerations

Looking at the explanation of the base implementation, you could be correct in saying that batch normalisation computations of mean and variance could be influenced - but surely it depends in which order you compute your batch statistics?

There is an ongoing debate (see e.g. this question) as to which order to apply layers such as batch-norm, dropout and activations themselves to the weights. There is the argument (pointed out above) that using both together isn't a good idea. There are some more great points made here.

If you have a look at the three implementations summarised here, you can see that there isn't a big difference between Dropout and DropConnect... it is just about scaling the values correctly, so that the (expected) sum remains consistent.

To each their own!

It almost gets philosphical! I believe it depends on your own personal perspective on why we use each of these layers. If you see Dropout as a means of restricting the network and preventing the co-adaptation of neurons, we are trying to block flow of information through certain paths of the network, so it might make sense to also remove that information from the batch-norm computations.
If you see batch-norm as being a surgically precise method to man-handle the input distribution to a layer, you might want to compute batch-norm stats before the DropConnect sets any weights to zero and use those on the entire outbound batch.


How can I use a class variable with many possible values in logistic regression? I am attempting to build a logistic regression model that determines the probability of an outcome based on a set of independent variables.

For context, the data is based on a project in which sales representatives and branch managers for a builders merchant were given price recommendations for their customers' deals, and were given the option of saying 'Yes' or 'No' to these price recommendations.  The Yes or No answer is my dependent variable; I need to determine which variables can predict with the highest probability whether the respondent will say 'no' to a price recommendation.

Most of the independent variables work fine in this model, except for one which I am currently unable to test; individual differences between the respondents.

My hypothesis is that the propensity to say 'no' will be stronger in some individual respondents than others, be it for psychological or geographical reasons, and that these individual differences will be a stronger determiner for the dependent variable than any other class variable.

There are about 800 respondents, so simply shoving them in as an independent variable does not produce the desired results.

Is there a method of doing this in logistic regression?  Should I use another analysis technique for this?

I am using Statistical Analysis Software (SAS) to carry out the logistic regression.

This may not be the answer you are looking for, but I think this is a telling part of your challenge

  My hypothesis is that the propensity to say 'no' will be stronger in some individual respondents than others, be it for psychological or geographical reasons, and that these individual differences will be a stronger determiner for the dependent variable than any other class variable.

Unless I misunderstood you, it seems you want to have an input parameter that indicates how reflexively a respondent will be to just respond with 'No'.  The problem is, as I see it, you don't know the answer to this.  You have no way to measure this directly.  No?

If you have more data about the respondents, maybe you can back into this.  Huge assumption here, but if you could get a count of the number of recommendations made vs the number of times they have said 'No' you could come up with some factor:
$$f_{(No)} = {{Number\ of\ "No"\ Responses}\over{Number\ of\ Recommendations}}$$

This may get you what you are looking for, but it assumes you can tie back the data to the respondent and that the respondent is, in fact, the Branch Manger or Sales Representative, not the customer.  The problem with this approach is that it assumes that all the recommendations were equally competitive.

HTH


Latent loss in variational autoencoder drowns generative loss I'm trying to run a variational auto-encoder on the CIFAR-10 dataset, for which I've put together a simple network in TensorFlow with 4 layers in the encoder and decoder each, an encoded vector size of 256. For calculating the latent loss, I'm forcing the encoder part of my network to output log variances instead of standard deviations, so the latent loss function looks like:

latent_loss = -0.5 * tf.reduce_sum(1 + log_var_vector - tf.square(mean_vector) - tf.exp(log_var_vector), axis=1)

I found this formulation to be more stable than directly using the logarithms in the KL-divergence formula since the latter often results in infinite loss value. I'm applying a sigmoid activation function on the last layer of the decoder, and the generative loss is computed using mean-squared error. The combined loss is simple a sum of both latent and generative losses. I train the network in batches of 40 using Adam Optimizer with a learning rate of 0.001.

The problem is that my network doesn't train. The latent loss immediately drops to zero, and the generative loss doesn't go down. However when I only optimize only for the generative loss, the loss does reduce as expected. Under this setting, the value of the latent loss quickly jumps to very large values (order of 10e4 - 10e6).

I have a hunch that the culprit is the extreme mismatch between the magnitudes of both losses. The KL-divergence is unbounded, whereas the mean-squared error always remains <1, so when optimizing for both, the generative loss basically becomes irrelevant.

Any suggestions to solve the problem are welcome.

I don't like the reduce_sum version of the kl-loss because it depends on the size of your latent vector. My advise is to use the mean instead.
Moreover it is a notorious fact that training a VAE with the kl loss is difficult. You may need to progressively increase the contribution of the kl loss in your total loss. Add a weight w_kl that will control the contribution :
Loss = recons_loss + w_kl * kl_loss
You start with w_kl=0 and progressively increase it every epoch (or batch) to 1. This is a classic trick.
Your learning rate seems good, maybe you can try a little higher (4e-4).
If you don't like the tricks, the Wasserstein auto-encoder may be your friend.


How to add bias consideration into logistic regression code? I am a complete beginner in coding and machine learning, and I've been tasked with learning what's under the hood of logistic regression (so I have pieced together the python code below) but I've been asked to figure out how to add bias into this code. I'm completely stuck on at what point it would need to be added into, I think at the point I am defining the hypothesis function - but if anyone would be able to point me in the right direction to figure this out I would really appreciate it.

If it helps, this logistic regression is being used to classify if a tumour is benign of malignant with the wisconsin breast cancer dataset (https://www.kaggle.com/uciml/breast-cancer-wisconsin-data)

X_train,X_test,Y_train,Y_test = train_test_split(X,Y,test_size=0.3)

X = data["diagnosis"].map(lambda x: float(x))

X = data[['texture_mean','perimeter_mean','smoothness_mean','compactness_mean','symmetry_mean', 'diagnosis']]
X = np.array(X)
X = min_max_scaler.fit_transform(X)
Y = data["diagnosis"].map(lambda x: float(x))
Y = np.array(Y)

def Sigmoid(z):
    if z < 0:
        return 1 - 1/(1 + math.exp(z))
    else:
        return 1/(1 + math.exp(-z))
def Hypothesis(theta, x):
    z = 0
    for i in range(len(theta)):
    z += x[i]*theta[i]
return Sigmoid(z)enter preformatted text here

def Cost_Function(X,Y,theta,m):
sumOfErrors = 0
for i in range(m):
    xi = X[i]
    hi = Hypothesis(theta,xi)
    error = Y[i] * math.log(hi if  hi >0 else 1)
    if Y[i] == 1:
        error = Y[i] * math.log(hi if  hi >0 else 1)
    elif Y[i] == 0:
        error = (1-Y[i]) * math.log(1-hi  if  1-hi >0 else 1)
    sumOfErrors += error

const = -1/m
J = const * sumOfErrors
print ('cost is: ', J )
return J

def Cost_Function_Derivative(X,Y,theta,j,m,alpha):
sumErrors = 0
for i in range(m):
    xi = X[i]
    xij = xi[j]
    hi = Hypothesis(theta,X[i])
    error = (hi - Y[i])*xij
    sumErrors += error
m = len(Y)
constant = float(alpha)/float(m)
J = constant * sumErrors
return J

def Gradient_Descent(X,Y,theta,m,alpha):
new_theta = []
constant = alpha/m
for j in range(len(theta)):
    CFDerivative = Cost_Function_Derivative(X,Y,theta,j,m,alpha)
    new_theta_value = theta[j] - CFDerivative
    new_theta.append(new_theta_value)
return new_theta

initial_theta = [0,1]
alpha = 0.01
iterations = 1000
Logistic_Regression(X,Y,alpha,initial_theta,iterations)

If I understood correctly, by bias you mean the intercept term in your model, that is, $\alpha$ in the equation
$$
p(y=1|x) = \frac{1}{1+e^{-(\alpha + \beta_1 x_1 + \beta_2 x_2 + ... + \beta_k x_x)}}
$$

If it is the case, you can easily incorporate intercept by adding a colum of ones into your X:

X = np.hstack([np.ones([X.shape[0],1]), X])

So the first colum of $X$ is always one. Then, your model will look like

$$
p(y=1|x) = \frac{1}{1+e^{-(\beta_0 x_0 + \beta_1 x_1 + \beta_2 x_2 + ... + \beta_k x_x)}}
$$
where $\beta_0$ is your bias term and $x_0$ always equals $1$. Thus, your model would be equivalent to the one with explicit intercept.


Algorithm selection for controlling a model vehicle I've built an autonomous sailing robot (https://github.com/kolosy/ArduSailor). Turns out, the problem of piloting it is fairly complex, and my procedural approach to solving it hasn't worked well (or at all). I think that an ML-based approach may be better, and I'm trying to figure out the right algorithm to use.

I'm viewing it as an optimization problem of sorts - I've got a small set of parameters:

Position (lat, lon)
Orientation (in 9 DOF)
Speed
Wind speed & direction
Distance to waypoint
Heading to waypoint
Sail position (basically winch orientation, a single value between 0 and 180)
Rudder position (same as above)

If I'm thinking about this right, I need to vary my rudder and winch over time in response to my current position, orientation and wind direction to minimize the difference between my orientation and the heading to the waypoint, and minimize the distance to the waypoint.

My approach right now, is to train an ANN (using this code) by recording a manual run through my course. Is this the right approach and algorithm? Is there a better / more suitable way of thinking about this?

Please, do not use ANN, because what you are looking for actually is SLAM method. It is a set of algorithms working together to localize your ship in an environment and mapping it simultaneously.

You can find an introduction on the Wikipedia:

https://en.wikipedia.org/wiki/Simultaneous_localization_and_mapping

This is what autonomous cars are using.


Cable angle measurement (rotation) I need to detect the rotation of a cable (degree) in the x-axis with high precision [0.2 (or more) degree detection] from its original state.
Detailed description:

I have a cable that is set in its original state.
The system has rotated the cable in the x-axis.
I want to know the degree the cable has been rotated from its original state.

Example:
There're following images for a specific cable in different rotation (angle) [0, 0.4, 0.6, 0.8]:
1)
2)
3)
4)
[First image shows the initial state of the cable; the following three images show 0.4, 0.6, 0.8 rotation from its initial state respectively)].
Important facts:

It is not possible to make pictures in better resolution.
It is not possible to make pictures in better contrast.
The cables may vary (different positions of the light strips on the cable).

Can this problem be solved using computer vision, neural networks, or other methods/techniques?

Answer:
Since the difference can't be seen by the human eye, you may need to try teaching an algorithm the difference.
You might try two approaches:

Train 1 model per cable defining the initial state implicitly

Train a model with a triplet loss to define an anchor in the initial state.

Observation:
I strongly recommend to keep the images as raw files or use compression without loss for this kind of image treatment as that will avoid adding artifacts that will reduce your precision. If your data acquisition system cannot sustain the load of transferring raw image files, I would recommend using PNG or jpeg2000(used by NASA to compress images, way more efficient and the transformation used as base is used as inspiration for multiple object detection algorithms)


Neural network example not working with sigmoid activation function I'm running the Neural Network example written in in BogoToBogo

The program worked fine:

(array([0, 0]), array([  2.55772644e-08]))
(array([0, 1]), array([ 0.99649732]))
(array([1, 0]), array([ 0.99677086]))
(array([1, 1]), array([-0.00028738]))

The neural network learned XOR, using tanh as activation function by default. However, after I changed the activation function to "sigmoid"

nn = NeuralNetwork([2,2,1], 'sigmoid')

Now the program outputs:

epochs: 0
...
epochs: 90000
(array([0, 0]), array([ 0.45784467]))
(array([0, 1]), array([ 0.48245772]))
(array([1, 0]), array([ 0.47365194]))
(array([1, 1]), array([ 0.48966856]))

The output for the 4 inputs are all near 0.5. The result shows that neural network (with the sigmoid function) didn't learn XOR.

I was expecting the program would output:

~0 for (0, 0) and (1, 1)
~1 for (0, 1) and (1, 0)

Can somebody explain why this example with sigmoid doesn't work with XOR?

I found the answer by myself. The reason of the difference is that the definition of prime of tanh in BogoToBogo (tanh_prime) takes arguments that's already applied with activation function:

def tanh_prime(x):
    return 1.0 - x**2

while sigmoid_prime is not. It calls sigmoid in it:

def sigmoid_prime(x):
    return sigmoid(x)*(1.0-sigmoid(x))

So the definition of sigmoid_prime seems more accurate than tanh_prime. Then why not sigmoid is working? It's because their parameters are already applied with the activation function.

Background

The derivatives of sigmoid ($\sigma$) and tanh share the same attribute in which these derivatives can be expressed in terms of sigmoid and tanh functions themselves.

$$
\frac{d\tanh (x)}{d(x)} = 1 - \tanh (x)^2
$$
$$
\frac{d\sigma (x)}{d(x)} = \sigma(x) (1 - \sigma(x))
$$

When performing backpropagation to adjust their weights, neural networks apply the derivative ($g^{'}$) to the values that's before applied with activation function. In BogoToBogo's explanation, that's variable $ z^{(2)} $ in

$$
\delta^{(2)} = (\Theta^{(2)})^T \delta^{(3)} \cdot g^{'}(z^{(2)}).
$$

In its source code, the variable dot_value holds such values. The Python implementation, however, calls the derivative with the vector stored in variable a. The vector is after applied with activation function. Why?

I interpret this as optimization to leverage the fact that derivatives of sigmoid and tanh use their parameters only to apply the original function. As the neural network already holds the value after activation function (as a), it can skip unnecessary calculation of calling sigmoid or tanh when calculating the derivatives. That's why the definition of tanh_prime in BogoToBogo does NOT call original tanh within it. However, the definition of sigmoid_prime, on the other hand, calls sigmoid function unexpectedly, resulting in miscalculation of derivative function.

Solution

Once I define sigmoid_prime in such a way that it assumes the parameter is already applied with sigmoid, then it works fine.

def sigmoid_prime(x):
    return x*(1.0-x)

Then calling the implementation with

nn = NeuralNetwork([2,2,1], 'sigmoid', 500000)

successfully outputs:

(array([0, 0]), array([ 0.00597638]))
(array([0, 1]), array([ 0.99216467]))
(array([1, 0]), array([ 0.99332048]))
(array([1, 1]), array([ 0.00717885]))


Why does LSTM performs better when the source target is reversed? (Seq2seq) I'm currently reading this paper : https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf

Here, the author mentioned that LSTM learns better when the order of the source target is reversed. "Mapping (c,a'), (b,b'), (a,a') (a',b',c' is a translation of a,b,c each) is better." but I don't know why reversing the sentence makes it easy for SGD to map input n output sequence better.

Also the paper said, we do not have a complete explanation to this phenomenon.

But here is kinda explanation about that.

  While we do not have a complete explanation to this phenomenon, we
  believe that it is caused by the introduction of many short term
  dependencies to the dataset.

  Normally, when we concatenate a source sentence with a target
  sentence, each word in the source sentence is far from its
  corresponding word in the target sentence.

  As a result, the problem has a large “minimal time lag” [17].

  By reversing the words in the source sentence, the average distance
  between corresponding words in the source and target language is
  unchanged.

  However, the first few words in the source language are now very close
  to the first few words in the target language, so the problem’s
  minimal time lag is greatly reduced.

  Thus, backpropagation has an easier time “establishing communication”
  between the source sentence and the target sentence, which in turn
  results in substantially improved overall performance.


transform a supervised neural network to reinforcement learning? I have a functional LSTM model that works with an acceptable performance. How can I now convert this supervised model to a reinforcement learning model for improving the performane? Is there any example about how to transform a supervised model to a reinforcement learning model?

Details: I have a multi-input multi-output system (since I can not share the actual problem lets assume weather forcast as example) where I need to predict the output (e.g temperature, wind speed, etc) in real time. I have a large dataset and I trianed a supervised learning model that can do the predictions pretty well in real time.

The problem is that sometimes there is big deviation between predictions and actual values. This means that probably there was a new trend in the inputs that has never been in the data sets. For such cases, I would like to gradually enhance my model predictions.

Does it mean that I have to train again my model with the old dataset and the new data or I can simply have a RL mechansim to gradually improve the model?

Interesting question!

I think re-using your model in a DQN setting could be interesting. Even more so if you retrain your supervised model every now and then to update the DQN model (although in that case you'd have to figure out how to re-use what your model learned from DQN vs what it learned supervised).

I think to get you started you would have to define your context that you are using RL on. Basic elements that are needed for RL are:

an Agent (whoever is making a decision on what to do)
a State (how can you describe a snapshot of your current environment that has been influenced by an agent's action)
an Action (so something your Agent can actively "do" to induce another state in its environment)
a Reward (something you can "give" your agent to learn if he has chosen something good or bad)

Assuming you want to use something like DQN you have to define those things to be able to run the algorithm. Your current supervised model would then be the starting status of your NN that is used to choose your Agent's decisions.

The keyword for what you are looking for is "Transferlearning", which describes how to use trained models for other cases. In your case even other learning methods.


Validation fluctuating on keras training My model is showing a very large variation in validation.
In generating the model used, 25% of the data as a test and 75% as a validation. The data was also divided.

Is there a way to reduce / eliminate this variation?

Maybe the distribution of the validation data is not similar to the training data, and therefore the training signal does not lead the model to perform well on the validation data.

The key point here is therefore: how did you split the data into training and validation?


Elimination of features based on high covariance without affecting performance? I ran into a question where the answer ran me into a big doubt.
Suppose we have a dataset $A=${$x1,x2,y$} in which $x1$ and $x2$ are our features and $y$ is the label.
Also, suppose that the covariance matrix between these three random variables are as follows:
$$\begin{array}{|c|c|c|} \hline
    &x1&x2&y\\  \hline
x1&a&d&e\\
   \hline
x2&d&b&f\\\hline
y&e&f&c\\\hline
\end{array}$$
When $|d| \gg 0$, someone may ignore either of the two features, without losing any performance.

The claim was marked as a true question. Is there any idea why the
mentioned proposition is True and performance was not affected?

$|d| \gg 0$ means there is a very strong correlation between $x_1$ and $x_2$. This means one can be expressed (almost completely) in terms of the other, thus one of two is almost redundant.
A simple example:
Consider that $x_2$ is simply a copy of $x_1$, ie $x_2=x_1$. Does $x_2$ offer any new information about the the label $y$ apart from the information $x_1$ provides? No, it is clear.
Now consider that $x_2$ is simply a shifted version of $x_1$, ie $x_2=c_0+x_1$. Does it now offer any new information apart from what $x_1$ provides? Again no, by simply changing the reference point $c_0$, which is equivalent to changing the baseline of the measuring system, does not  by itself offer new information.
Thirdly consider that $x_2$ is a re-scaled version of $x_1$, ie $x_2 = s
 \cdot x_1$. Does it offer now different information than what $x_1$ provides? Again, no since scaling is equivalent to changing the measuring unit of $x_1$, but this is simply a re-naming of units, no new information is added.
One can see that any transformation of the form $x_2 = c_0 + s \cdot x_1$ does not offer any new information from what $x_1$ already provides, by the arguments above.
If one wants to be more realistic one can add some noise, ie $x_2 = c_0 + s \cdot x_1 + \epsilon$, where $\epsilon$ is some random variable with zero mean and very small variance.
A correlation value $|d|$ which is quite high actualy means that $x_2$ can be expressed as such a transformation of $x_1$, ie $x_2 = c_0 + s \cdot x_1 + \epsilon$ (or vice-versa) (there are some non-trivial exceptions to this). So $x_2$ offers no new information apart from what $x_1$ offers and is redundant.
So if such a high correlation exists between variables/features, one of them can be always eliminated without diminishing performance as being redundant (in the sense that, as far as second-order statistics are concerned, the two variables are identical).
This is one of the main rationales of feature selection and dimensionality reduction in machine learning.


Building a prediction model for dynamic coordinates and later categorize as binary classification Project Summary

I have a website (academic project) that records mouse movements such as click, mouse up, mouse down, etc. It records the coordinates for each event on a given web page from a visitor.

What am I trying to accomplish?

I would like to do a dynamic polling per 100 msec from a website and compare if those points/graphs/velocities etc of user movement already existed in ML Model or not. Each time it polls, it will check against the ML model and will give me the confidence of the matching values. So, this will be an online ML model which will poll against multiple models that I have.

What I do not know and where to start?

In the first approach, I was brainstorming: Would it be better to have each 100 msec during polling of the coordinates to draw a graph and compare it against the models that I have? The idea that I got was image similarity matching. Each 100 msec will draw a dynamic graph and compare on models that I have allocated against each polling.
The second approach that I was brainstorming: Have velocities of coordinate changes into the time-series graph and compare the slice of the graph for the given time series (T-current). Here, the graph will be sliced and compared against the time series.

Expertise seeking summary

I would like to identify, with the dynamic mouse movement pattern, that this pattern was already in the ML model or not in an online mode. This will be handled prefetching the information from the already trained model. It would be great if you could point me in the following suggestions:

What kind of models or ways to handle this example would be best suited? Example of links that support this.
Real-time modeling interpretation/classification using the already built modeling

References - These are the tutorials that looked so far

https://www.vis.wiwi.uni-due.de/uploads/tx_itochairt3/publications/cgf-dynamicgraphs.pdf
https://ieeexplore.ieee.org/abstract/document/4734024
https://www.sciencedirect.com/science/article/pii/S2468502X20300140
https://ieeexplore.ieee.org/document/5190876
https://scholarworks.umass.edu/cgi/viewcontent.cgi?article=1906&context=dissertations_2
https://stackoverflow.com/questions/5730631/image-similarity-comparison
https://towardsdatascience.com/prediction-and-analysis-of-time-series-data-using-tensorflow-2136ef633018
https://towardsdatascience.com/how-not-to-use-machine-learning-for-time-series-forecasting-avoiding-the-pitfalls-19f9d7adf424
https://medium.com/@akarshzingade/image-similarity-using-deep-ranking-c1bd83855978
https://medium.com/dataseries/easily-deploy-deep-learning-models-in-production-13db48071578

I was able to find a solution.

Thanks to this article which uses LSTM with binary classification modeling: https://www.analyticsvidhya.com/blog/2019/01/introduction-time-series-classification/