How to identify the appropriate predictors in regression model I have a big data and want to perform some of prediction models(such as Regression,Decision Tree,..) on it in R but I have a problem. As there are too many variables, Is there any way to identify which variables should i use as predictors in the model to predict a specific variable?
Any help would be greatly appreciated.

I assume by too many variables you mean the case where number of variables is greater than the number of observations ($p>n$) or close to that. You will need to either reduce the number of variables before feeding them in, use some kind of automatic feature selection methods as @M.Mashayekhi has pointed out or utilize learning models that are robust to high feature dimension. Keep in mind that feature selection/creation, in my opinion, is the most important part of model building. There is no point in using a complex machine learning algorithm if all your input variables are useless.

It is always a good idea to first reduce the number of input variables by business intuition or context. By that I mean answering the questions: "Do I really need all of these variables to be in the model?", "What variables just wouldn't make sense to include in the model as a predictor?" "Are there variables that has to be in the model even if it is not the strongest predictor, simply for business reasons? If these questions are not easy to answer, or you have already reduced the number of variables but are still in the case of too many variables, then you can try the following.

Principal Component Analysis (PCA)

One suggestion would be to use PCA to reduce your feature space then run learning algorithms like regression, decision trees on the reduced feature space. You can do that in R using pca. Pick the first however many principal components where the next PC has a decline in marginal variance explained (Since each addition principal component always increases variance explained).

Regression with Lasso ($\mathcal{L1}$) Regularization

An alternative would be to let the model do the feature selection for you. A good starter would be a regression with the lasso ($\mathcal{L1}$) penalty, this shrinks the estimated coefficients toward zero. It does that by shrinking large effect coefficients more slowly toward zero than small effect ones. This effectively removes small effect coefficients from the model by setting them equal to zero, hence, auto feature selection. The challenge is to pick a good shrinkage parameter $\lambda$, which governs how much you are shrinking the coefficients. You can do that with cross validation. glmnet in R let's you run lasso regressions, and glmnet.cv let's you pick $\lambda$ using CV. Also note that by shrinking coefficients, lasso regression reduces prediction errors by balancing between variance reduction and bias increase. You will have to decide which is more important, inference or making good predictions, since unlike OLS regression, estimates by lasso regression are no longer unbiased.

Random Forest

Random Forest is an ensemble tree learning algorithm (implemented in randomForest in R) which is robust to many useless variables. Particularly, it works well in the case of $p>n$. RF is very popular because of its robustness and simple tuning. The downside is this is what's called a Black Box algorithm. Meaning that there is no good way to interpret the relationship between the input and dependent variables. You can however use something like the "variable importance plot" to give you a rough idea of what variables were most useful during the splits on average. This is fine if you are only concerned about making good predictions.

Other techniques

There are certainly other techniques that goes well with high dimensional data like Support Vector Machines and Boosted Regression Trees (Difference between RF and Boosted Trees), but they might usually require a lot more tuning than the ones mentioned above.


What is the State of the art method for full body gesture recognition in images I am working on gesture recognition in images and the best way that I am aware of, is whether using end to end approaches with deep neural networks or extracting body joint positions in an image and then doing some pre-processing on these joints positions and finally, feeding them to a classifier. I wanted to now what is the state of the art method for full body gesture recognition in images and if there are some good papers or a blog post reviewing different methods, as I have problem finding one.

If you are planing to train a model which is using images taken by a single camera, I encourage you to take a look at this paper. They use a camera to track a person and recognize gestures involving arm motion. In this paper, two alternative methods for gesture recognition are compared: a template based approach and a neural network approach.

However, another way which I suggest is using depth cameras such as Intel Realsense or Microsoft Kinect cameras. This paper uses depth cameras for full body gesture recognition. I think using these cameras take your project a step ahead and more accurate in initial states.


Custom POS tags with SpaCy for NER Quite new to NLP and especially NER. I'm trying to train a NER model on a custom dataset. This is a dataset of houses for sale. As part of the entities I'm training the model to extract are reference numbers. These are of variable length (but usually between 4-9) and look like G55L7 or LPP01Z1-32.

How can I give these entities a new "POS tag", as from what I'm aware of, I can't find any in SpaCy's default list that would match these?

Ideally, I'd like to train this alongside a pre-existing NER model so that I can also extract ORGs which SpaCy already has support for.

For your first question, I would try to use Regex to identify the reference numbers since they seem unique comparing to normal words. I assume they are:

All capital
Start with one or more letters
After follow numbers
Any other pattern may follow (i.e. letters, numbers, hyphens)
Do not include spaces

Could be something like this: \b[A-Z]+\d+([A-Z]|\d|-)*\b

I can imagine you could do it better...


Data sets for evaluating text retrieval quality I'm searching for data sets for evaluating text retrieval quality.

TF-IDF is a popular similarity measure, but is it the best choice? And which variant is the best choice? Lucenes Scoring for example uses IDF^2, and IDF defined as 1+log(numdocs/(docFreq+1)). TF in lucene is defined as sqrt(frequency)...

Many more variants exist, including Okapi BM25, which is used by the Xapian search engine...

I'd like to study the different variants, and I'm looking for evaluation data sets. Thanks!

TF IDF will give you the degree of measure of how relevant a document is to your query. owever, to evaluate your IR system you need to use metrics such as - Precision, Recall and F score.

Precision: Out of all the documents that your system retrieves, which ones are relevant? This measures how much noise there is in the output of your IR system.

Recall: Out of all the documents that are relevant, which ones did your system retrieve? This measures how much coverage does your IR system have?

It is possible to get 100% recall all the time by basically retrieving ALL documents from a collection for any query. However, the precision in this case will be very low.

It is possible to get a very high precision by hand modeling an IR system ti produce very accurate results. However, it would produce a very bad recall as there will not be coverage over all the documents.

So we need to measure F score- which is the harmonic mean between Precision and Recall
Check out Chapter 8 of the Stanford IR book.

If you are looking for datasets only here are a few that are relevant:

http://zitnik.si/mediawiki/index.php?title=Datasets
http://www.daviddlewis.com/resources/testcollections/
http://boston.lti.cs.cmu.edu/callan/Data/


Find optimal P(X|Y) given I have a model that has good performance when trained on P(Y|X) Input Data:

$X$ -> features of t shirt (colour,logo,etc)

$Y$ -> profit margin

I have trained a random forest on the above $X$ and $Y$ and have achieved reasonable accuracy on a test data.
So, I have

$P(Y|X)$.

Now, I would like to find $P(X|Y)$ i.e probability distribution of $X$ features given I am expecting this much profit margin.

How do I do that with a random forest(or any other discriminative model)?

One suggestion for me could be to start with a generative model rather than a discriminative model. But, my understanding is generative model generally require a lot of data to train unless that make some very restrictive assumptions such as conditional independence of $X$'s in case of Naive Bayes?

Other suggestion could be to just switch $X$ and $Y$ and train a discriminative model. Now $X$ will be profit margin and $Y$ will be features of a t shirt. $P(Y|X)$ will directly give me the probability distribution of t shirt features, given a target profit margin. But this approach doesn't seem right to me, as I have always though of $X$ as casual variables and $Y$ to be effect.

Also, from what I have heard, similar question has been posed for drug discovery and algorithms have been designed which come up with candidate new drugs that have high degree of success. Can someone point me to research literature in this domain?

Update:

I have come across this and this which talks about GANs being used for drug discovery. Generative adversial networks seem like a good fit for my problem statement so I have been reading about them. But one thing I understood is GAN generate samples in an unsupervised way. They try to produce sample which is like first capturing the underlying distribution of X and then sampling from that distribution. But I am interested in X|Y. X and Y are defined above. Should I explore something other than GANs? Any pointers please?

Follow up Question:

Imagine I have a GAN trained that has learned how to make t shirts(output sample Xs). How can I get the top 5 shirts for given Y?

This response has been significantly modified from its original form. The flaws of my original response will be discussed below, but if you would like to see roughly what this response looked like before I made the big edit, take a look at the following notebook: https://nbviewer.jupyter.org/github/dmarx/data_generation_demo/blob/54be78fb5b68218971d2568f1680b4f783c0a79a/demo.ipynb

TL;DR: Use a KDE (or the procedure of your choice) to approximate $P(X)$, then use MCMC to draw samples from $P(X|Y) \propto P(Y|X)P(X)$, where $P(Y|X)$ is given by your model. From these samples, you can estimate the "optimal" $X$ by fitting a second KDE to the samples you generated and selecting the observation that maximizes the KDE as your maximum a posteriori (MAP) estimate.

Maximum Likelihood Estimation

... and why it doesn't work here

In my original response, the technique I suggested was to use MCMC to perform maximum likelihood estimation. Generally, MLE is a good approach to finding the "optimal" solutions to conditional probabilities, but we have a problem here: because we're using a discriminative model (a random forest in this case) our probabilities are being calculated relative to decision boundaries. It doesn't actually make sense to talk about an "optimal" solution to a model like this because once we get far enough away from the class boundary, the model will just predict ones for everything. If we have enough classes some of them might be completely "surrounded" in which case this won't be a problem, but classes on the boundary of our data will be "maximized" by values that aren't necessarily feasible.

To demonstrate, I'm going to leverage some convenience code you can find here, which provides the GenerativeSampler class which wraps code from my original response, some additional code for this better solution, and some additional features I was playing around with (some which work, some which don't) which I probably won't get into here.

np.random.seed(123)
sampler = GenerativeSampler(model=RFC, X=X, y=y,
                            target_class=2,
                            prior=None,
                            class_err_prob=0.05, # <-- the score we use for candidates that aren't predicted as the target class
                            rw_std=.05,          # <-- controls the step size of the random walk proposal
                            verbose=True,
                            use_empirical=False)
samples, _ = sampler.run_chain(n=5000)

burn = 1000
thin = 20
X_s = pca.transform(samples[burn::thin,:])

# Plot the iris data
col=['r','b','g']
for i in range(3):
    plt.scatter(*X_r[y==i,:].T, c=col[i], marker='x')
plt.plot(*X_s.T, 'k')
plt.scatter(*X_s.T, c=np.arange(X_s.shape[0]))
plt.colorbar()
plt.show()

In this visualization, the x's are the real data, and the class we're interested in is green. The line-connected dots are the samples we drew, and their color corresponds to the order in which they were sampled, with their "thinned" sequence position given by the color bar label on the right.

As you can see, the sampler diverged from the data fairly quickly and then just basically hangs out pretty far away from values of the feature space that correspond to any real observations. Clearly this is a problem.

One way we can cheat is to change our proposal function to only allow features to take values that we actually observed in the data. Let's try that and see how that changes the behavior of our result.

np.random.seed(123)
sampler = GenerativeSampler(model=RFC, X=X, y=y,
                            target_class=2,
                            prior=None,
                            class_err_prob=0.05,
                            verbose=True,
                            use_empirical=True) # <-- magic happening under the hood
samples, _ = sampler.run_chain(n=5000)

X_s = pca.transform(samples[burn::thin,:])

# Constrain attention to just the target class this time
i=2
plt.scatter(*X_r[y==i,:].T, c='k', marker='x')
plt.scatter(*X_s.T, c='g', alpha=0.3)
#plt.colorbar()
plt.show()

sns.kdeplot(X_s, cmap=sns.dark_palette('green', as_cmap=True))
plt.scatter(*X_r[y==i,:].T, c='k', marker='x')
plt.show()

This is definitely a significant improvement and the mode of our distribution corresponds roughly to what we're looking for, but it's clear we're still generating a lot of observations that don't correspond to feasible values of $X$ so we shouldn't really trust this distribution either.

The obvious solution here is to incorporate $P(X)$ somehow to anchor our sampling process to regions of the feature space that the data is actually likely to take. So let's instead sample from the joint probability of the likelihood given by the model, $P(Y|X)$, and a numerical estimate for $P(X)$ given by a KDE fit on the entire dataset. So now we're... sampling from... $P(Y|X)P(X)$....

Enter Bayes Rule

After you hounded me to be less hand-wavey with the math here, I played around with this a fair amount (hence me building the GenerativeSampler thing), and I encountered the problems I laid out above. I felt really, really stupid when I made this realization, but obviously what you are asking for calls for an application of Bayes rule and I apologize for being dismissive earlier.

If you're not familiar with bayes rule, it looks like this:

$$P(B|A) = \frac{P(A|B)P(B)}{P(A)}$$

In many applications the denominator is a constant which acts as a scaling term to ensure that the numerator integrates to 1, so the rule is often restated thusly:

$$P(B|A) \propto P(A|B)P(B)$$

Or in plain English: "the posterior is proportional to the prior times the likelihood".

Look familiar? How about now:

$$P(X|Y) \propto P(Y|X)P(X)$$

Yeah, this is exactly what we worked up to earlier by constructing an estimate for the MLE that is anchored to the observed distribution of the data. I've never thought about Bayes rule this way, but it makes sense so thank you for giving me the opportunity to discover this new perspective.

To backtrack a tiny bit, MCMC is one of those applications of bayes rule where we can ignore the denominator. When we calculate the acceptanc ratio, $P(Y)$ will take the same value in both the numerator and denominator, canceling out, and allowing us to draw samples from unnormalized probability distributions.

So, having made this insight that we need to incorporate a prior for the data, let's do that by fitting a standard KDE and see how that changes our result.

np.random.seed(123)
sampler = GenerativeSampler(model=RFC, X=X, y=y,
                            target_class=2,
                            prior='kde',         # <-- the new hotness
                            class_err_prob=0.05,
                            rw_std=.05,          # <-- back to the random walk proposal
                            verbose=True,
                            use_empirical=False)
samples, _ = sampler.run_chain(n=5000)

burn = 1000
thin = 20
X_s = pca.transform(samples[burn::thin,:])

# Plot the iris data
col=['r','b','g']
for i in range(3):
    plt.scatter(*X_r[y==i,:].T, c=col[i], marker='x')
plt.plot(*X_s.T, 'k--')
plt.scatter(*X_s.T, c=np.arange(X_s.shape[0]), alpha=0.2)
plt.colorbar()
plt.show()

Much better! Now, we can estimate your "optimal" $X$ value using what's called the "maximum a posteriori" estimate, which is a fancy way of saying we fit a second KDE -- but to our samples this time -- and find the value that maximizes the KDE, i.e. the value corresponding to the mode of $P(X|Y)$.

# MAP estimation

from sklearn.neighbors import KernelDensity
from sklearn.model_selection import GridSearchCV
from scipy.optimize import minimize

grid = GridSearchCV(KernelDensity(), {'bandwidth': np.linspace(0.1, 1.0, 30)}, cv=10, refit=True)
kde = grid.fit(samples[burn::thin,:]).best_estimator_

def map_objective(x):
    try:
        score = kde.score_samples(x)
    except ValueError:
        score = kde.score_samples(x.reshape(1,-1))
    return -score

x_map = minimize(map_objective, samples[-1,:].reshape(1,-1)).x

print(x_map)

x_map_r = pca.transform(x_map.reshape(1,-1))[0]
col=['r','b','g']
for i in range(3):
    plt.scatter(*X_r[y==i,:].T, c=col[i], marker='x')
sns.kdeplot(*X_s.T, cmap=sns.dark_palette('green', as_cmap=True))
plt.scatter(x_map_r[0], x_map_r[1], c='k', marker='x', s=150)
plt.show()

And there you have it: the large black 'X' is our MAP estimate (those contours are the KDE of the posterior).


Elastic Regression fitting good mean and bad variance So, I'm kinda new to machine learning and I was trying to predict the monthly sales of a business using a set of features and using a sliding window of the past sales of 12 months.

I used some algorithms to do it, including linear/polynomial regression, lasso/elastic and SVR. I got the best results with elastic regression resulting in the following result:

As it shows, the model fit the mean of the curve somewhat well, but I would like it to fit the variance as well. So, I've been searching what technique or feature to use could better fit my data, but I still found nothing precise.

Would someone knows what could I do to to take the variance of the system into account?

Thanks in advance!

Your model actually looks pretty good. What it sounds like you are asking to do is to overfit your model. I would not recommend that you do that. You can do that by finding more variables that you can input into the model, fitting extra polynomial terms or anything else like that, fitting a neural network will potentially do it for you too. However, you generally want to smooth your predictions out, like what you have.

One thing that you could try is to add an autocorrelation term. That might cause your model to behave as you intend. With negative autocorrelation your predicted values will have a tendency to bounce back and forth around the mean. But I wouldn't recommend doing that, your performance will probably suffer, just judging by the graph that you provided.


Is there any way to define custom entities in Spacy 1) I have just started working on NLP the basic Idea is to extract meaningful information from text. For this I am using "Spacy".

As far as I have studied Spacy has following entities.

ORG
PERSON
DATE
MONEY
CARDINAL

etc. But I want to add custom entities like:

Nokia-3310 should be labeled as Mobile
and XBOX should be labeled as Games

2) Can I find some already trained models in Spacy to work on ?

For pretrained models, spaCy has a few in different languages. You can find them in their official documentation https://spacy.io/models

The available models are:

English
German
French
Spanish
Portuguese
Italian
Dutch
Greek
Multi-language

If you want support for extra labels in NER, you could train a model in your own dataset. Again, this is possible in spaCy and from their official documentation https://spacy.io/usage/training#ner, here is an example

LABEL = "ANIMAL"

TRAIN_DATA = [
    (
        "Horses are too tall and they pretend to care about your feelings",
        {"entities": [(0, 6, LABEL)]},
    ),
    ("Do they bite?", {"entities": []}),
    (
        "horses are too tall and they pretend to care about your feelings",
        {"entities": [(0, 6, LABEL)]},
    ),
    ("horses pretend to care about your feelings", {"entities": [(0, 6, LABEL)]}),
    (
        "they pretend to care about your feelings, those horses",
        {"entities": [(48, 54, LABEL)]},
    ),
    ("horses?", {"entities": [(0, 6, LABEL)]}),
]

nlp = spacy.blank("en")  # create blank Language class
ner = nlp.create_pipe("ner")
nlp.add_pipe(ner)

ner.add_label(LABEL)  # add new entity label to entity recognizer

optimizer = nlp.begin_training()

move_names = list(ner.move_names)
# get names of other pipes to disable them during training
other_pipes = [pipe for pipe in nlp.pipe_names if pipe != "ner"]

with nlp.disable_pipes(*other_pipes):  # only train NER
    sizes = compounding(1.0, 4.0, 1.001)
    # batch up the examples using spaCy's minibatch
    for itn in range(n_iter):
        random.shuffle(TRAIN_DATA)
        batches = minibatch(TRAIN_DATA, size=sizes)
        losses = {}
        for batch in batches:
            texts, annotations = zip(*batch)
            nlp.update(texts, annotations, sgd=optimizer, drop=0.35, losses=losses)
        print("Losses", losses)

If you want to use an existing model and also add a new custom Label, you can read the linked article in their documentation where they describe the process in details. Actually, it is quite similar to the code above.


Training Objective of language model for GPT3 On page 34 of OpenAI's GPT-3, there is a sentence demonstrating the limitation of objective function:

Our current objective weights every token equally and lacks a notion of what is most important to predict and what is less important.

I am not sure if I understand this correctly. In my understanding, the objective function is to maximize the log-likelihood of the token to predict given the current context, i.e., $\max L \sim \sum_{i} \log P(x_{i} | x_{<i})$. Although we aim to predict every token that appears in the training sentence, the tokens have a certain distribution based on the appearance in human litterature, and therefore we do not actually assign equal weight to every token in loss optimization.
And what should be an example for a model to get the notion of "what is important and what is not". What is the importance refer to in here? For example, does it mean that "the" is less important compared to a less common noun, or does it mean that "the current task we are interested in is more important than the scenario we are not interested in ?"
Any idea how to understand the sentence by OpenAI?

This may be best understood with a bit more of context from the article:

A more fundamental limitation of the general approach described in this paper – scaling up any LM-like model, whether
autoregressive or bidirectional – is that it may eventually run into (or could already be running into) the limits of the
pretraining objective. Our current objective weights every token equally and lacks a notion of what is most important to
predict and what is less important. [RRS20] demonstrate benefits of customizing prediction to entities of interest.

I think that the relevant part of the reference [RRS20] is this paragraph:

Recently, Guu et al.(2020) found that a “salient span masking” (SSM)
pre-training objective produced substantially better results in open-domain question answering.
This approach first uses BERT (Devlin et al., 2018) to mine sentences that contain salient spans
(named entities and dates) from Wikipedia. The
question answering model is then pre-trained to reconstruct masked-out spans from these sentences,
which Guu et al. (2020) hypothesize helps the
model “focus on problems that require world
knowledge”. We experimented with using the
same SSM data and objective to continue pretraining the T5 checkpoints for 100,000 additional
steps before fine-tuning for question answering.

With that context in mind, I understand that the sentence in the GPT-3 papers means that in normal language models, the predictions of every token has the same importance weight toward the computation of the loss, as the individual token losses are added together in an unweighted manner. This as opposed to the salient span masking approach, which finds tokens that are important to predict by means of a BERT-based preprocessing.


What can be the cause of a sudden explosion in the loss when training a CNN (Deeplab) I am training the following deeplab CNN:
https://github.com/tensorflow/models/tree/master/research/deeplab

During training I see the following loss:

The first 50k steps of the training the loss is quite stable and low, and suddenly it starts to exponentially explode. I wonder how this can happen. Of course there are many reasons a loss can increase, such as a too high learning rate. But what I do not understand is the following:

I use a batch size of 16 and I have 24k images, so 24k/16=1500 steps are used for a full pass on the train data
Only after 50k steps the loss starts exploding, before that it is remarkably stable.
So around the 34th iteration through my train set the loss starts to increase all of a sudden. Why only now? How can it be stable for so long and suddenly increase sharply?

One possible cause is a high learning rate. High values of this hyperparameter usually cause updates that are too drastic, and therefore divergence from the optimum.

Please keep in mind this is only a suggestion, your problem might be due to completely different reasons. Try different learning rates and schedules, in order to understand if that's the case. You can check this very good post on how to set the learning rate.

Also, consider using early stopping to save some training time.


Application of ideas from graph theory in machine learning I work with neural networks (ConvNNs, DeepNNs, RNNs/LSTMs) for image segmentation and recognition and Genetic Algorithms for some optimization problems. Recently I started to learn some deep graph theory ideas (random graphs, chromatic numbers, graph coloring). I'm familiar with combinatorics at undergrad level. Are there any existing interesting applications and areas of research of graph theory and combinatorics in ML?

Graphs are a very flexible form of data representation, and therefore have been applied to machine learning in many different ways in the past. You can take a look to the papers that are submitted to specialized conferences like S+SSPR (The joint IAPR International Workshops on Structural and Syntactic Pattern Recognition and Statistical Techniques in Pattern Recognition) and GBR (Workshop on Graph-based Representations in Pattern Recognition) to start getting a good idea of potential applications. Some examples:

Within the Computer Vision field, graphs have been used to extract structure information that can later on be used on several applications, like for instance object recognition and detection, image segmentation and so on.
Spectral clustering is an example of clustering method based on graph theory. It makes use of the eigenvalues of the similarity matrix to combine clustering and dimensionality reduction.
Random walks may be used to predict and recommend links in social networks or to rank webpages by relevance.


How to replace NA values with another value in factors in R? I have a factor variable in my data frame with  values where in the original CSV "NA" was intended to mean simply "None", not missing data. Hence I want replace every  value in the given column with "None" factor value. I tried this:

DF$col[is.na(DF$col)] <- "None"

but this throws the following error:

Warning message:
In `[<-.factor`(`*tmp*`, is.na(DF$col), value = c(NA, NA,  :
  invalid factor level, NA generated

I guess this is because originally there is no "None" factor level in the column, but is it the true reason? If so, how could I add a new "None" level to the factor?

(In case you would ask why didn't I convert NAs into "None" in the read.csv phase: in other columns NA really does mean missing data).

You can use this function :

forcats::fct_explicit_na

library(forcats)
fct_explicit_na(DF$col, na_level = "None")

Usage

It can be used within the mutate function and piped to edit DF directly:

library(tidyverse) # for tidy data packages, automatically loads dplyr
library(magrittr) # for piping
DF %<>% mutate(cols = fct_explicit_na(col, na_level = "None"))

Note that "col" needs to be a factor for this to work.


What is the best approach for classifying non-English text What would be the best approach for classifying non-English (Sinhala / Tamil) text? Currently I use Fasttext. Are there any better options?

I want to classify user questions into chatbot intents. Therefore, there may be many target classes.

As far as I know, the best way would be to use pretrained embedder. Embedder encodes your text into language-agnostic latent space. You input your text and you get fixed-length numerical vector as an output. You can use latent space encodings as a feature vector, which you can use to train discriminative models. They also suit well for resampling like SMOTE or ADASYN.

Some time ago Facebook released a model called LASER. You can read about it here. It supports Sinhala and Tamil as well. Here is a github repository. There is also unofficial distribution on pypi. It substitutes internal tools for tokenization and BPE encoding. For sake of convenience I've been working mostly with this distribution and I can confirm it works just fine. Here is a repository.

I'd also suggest to consider embedder abuse. It covers a lot of languages, which means you can train your models on e.g. English and prediction will work for Tamil out of the box!

Natural language is hyper-dimensional space and most seminal models are using encoders. To my knowledge it is the go-to approach for any language.


How to handle Tokenized text content which is given in number? i have one data set of customer review, but the text data is given is tokenized text number. I am unable to proceed thinking about how to proceed?
As I am encountering such data set the first time, so just need guide how to proceed.

As you can see text field is given in number, so how to proceed please guide?. it will predict the 0/1 +ve or -ve category.

Well, in general case, machines do not understand the text, but they understand the numbers. Thus, we always tokenize the text followed by converting them to some form of numbers. We build a vocabulary of words from the given document, where each word can be assumed as a number corresponding to its index in the vocabulary. Further, this number is converted to one-hot vector representations.
Since, you already have sequence of numbers instead of words, you can make an assumption that you can convert them directly to it's one-hot vectors. Or in a advanced way, you can use a embedding layer to learn the embeddings (especially if you are using the neural networks.)


Any Machine Learning algorithm to know a yes/no answer? Any Machine Learning algorithm to know the meaning of sentences? Specifically, I have sentences like, "we do not allow managers to trade derivatives", "We do not have policy on hedging", "our policy do not permit trading on stock", "trading is not allowed"... The actual sentence is longer. Any algorithm I can know whether trading on stock is allowed or not by programming? Thank you for your help!!!

I sounds like your need to consider Language Understanding services instead. There are a couple of providers of this type of service. The one I have used before is called LUIS from Microsoft:

https://www.luis.ai/home/index

It is not the algorithm you are looking for per se, but to solve your problem by implementing an algorithm, you need to develop a language understanding system using models based on deep learning neural nets, which is not a trivial exercise.


How to classify parametric curves? I am working on a project which aims at determining whether a patient has cervical issues or not, based on a certain movement (for instance, turning the head from left to right and so on).

For each patient, I have one or more sets of coordinates acquired with a VR headset. One problem is, all the patients are assumed healthy and I cannot compare their data with patients who actually have cervical issues.

I am currently working with two coordinates at a time, not the three, and I am considering two approaches: the first is to use approximation (splines...), the second is to use concave hulls. I am a bit more inclined to use hulls and I thought that I could calculate the distance between two hulls A and B as follows: area(A\B)+area(B\A). Note that the curves are parametric.

I have two questions:

Is it possible to "classify" patients using a data set that only consists of healthy patients? Or to find a "descriptor" for them ?
If so, or assuming I can get data on unhealthy patients, what tools can I use to classify the curves? I did not find anybody who worked on a similar problem.

Whenever you have skewed dataset, it means that you know some classes better than some others. In such cases it means that the data is your knowledge and there are learning algorithms for such occasions.

Consider an important fact here. Suppose that you have feature vectors of conditions of a nuclear company and they describe whether the company is in danger of nuclear radiation or not. In such case it is clear that it does not happen a lot that you have infected companies so most of your data have label of healthy condition. You have so much knowledge about the healthy class but you don't know much about the infected class because you don't have much data consequently you don't know its distribution and you can not estimate it well. Whenever your data is skewed, it means that e.g. you have 1 million feature vectors of negative class and 5 feature vectors of positive class. Now suppose that you change the feature vectors. In such cases you have imbalanced data-set or you just have the data samples of some classes without some other, you can use anomaly detection.

  In data mining, anomaly detection (also outlier detection) is the identification of items, events or observations which do not conform to an expected pattern or other items in a dataset. Typically the anomalous items will translate to some kind of problem such as bank fraud, a structural defect, medical problems or errors in a text. Anomalies are also referred to as outliers, novelties, noise, deviations and exceptions.


Machine learning libraries for Ruby Are there any machine learning libraries for Ruby that are relatively complete (including a wide variety of algorithms for supervised and unsupervised learning), robustly tested, and well-documented? I love Python's scikit-learn for its incredible documentation, but a client would prefer to write the code in Ruby since that's what they're familiar with.

Ideally I am looking for a library or set of libraries which, like scikit and numpy, can implement a wide variety of data structures like sparse matrices, as well as learners.

Some examples of things we'll need to do are binary classification using SVMs, and  implementing bag of words models which we hope to concatenate with arbitrary numeric data, as described in this StackOverflow post.

I'll go ahead and post an answer for now; if someone has something better I'll accept theirs.

At this point the most powerful option appears to be accessing WEKA using jRuby. We spent yesterday scouring the 'net, and this combination was even used by a talk at RailsConf 2012, so I would guess if there were a comparable pure ruby package, they would have used it.

Note that if you know exactly what you need, there are plenty of individual libraries that either wrap standalone packages like libsvm or re-implement some individual algorithms like Naive Bayes in pure Ruby and will spare you from using jRuby.

But for a general-purpose library, WEKA and jRuby seem to be the best bet at this time.


Forward and backward process in pyTorch When I write a network, do I have to write the whole forward property in nn.Module.forward()? I mean if I do some operations outside the net, does grad correctly flow?

For example, I have two networks, in which the output of one is the input of the other(net1 -> midresults -> net2).

If I do some operations on midresults(net1 -> midresults -> operations -> net2), can (net1+net2) be trained end to end?

As long as your operations are all compatible with pytorch tensors and Autograd then yes your network will be trained end-to-end.

A good rule of thumb is to ask yourself : I'm I using tensors and pytorch operators end-to-end? If yes (and unless stated otherwise in the docs), you should be pretty safe.


Do RL agents learn the optimal "degree" of an action to take? I have a game environment I want to train an RL model on. This environment has 2 fundamental actions that the agent can take; "Left" or "Right" (say, 0 or 1).

However, the actions "Left" or "Right" can be taken in a discrete number of "degrees". For example, I can take action "Left" with degree 70% , or take action "Right" with degree 16%.

Assuming a discrete action space between 0-100% for each "Left" or "Right", making the total action space a discrete size of 201 (0-200 in increments of 1), does an agent learn the optimal degree to take either "Left" or "Right" in any given state?

If you are using a value-based method, like Q-learning in Deep Q Networks (DQN), then the "degree" concept has little meaning to the agent, and you are effectively training an agent to learn the best discrete action out of 200 actions.

Yes this could take a lot longer than learning a simple 2-action scenario, as the agent will not easily learn that the value of "Right 49%" is correlated to the value of "Right 52%" in any particular state.

This is a similar problem you might face in supervised learning if you wanted to move from classifying a dog vs cat image classifier to add bounding boxes for where the creature is. You don't add the bounding box as a large number of classifications - instead you add it as a regression, mapping form the image directly to some vector of real numbers.

There might be ways to address this using value-based methods, but the usual approach here would be to move to a continuous action space (maybe with -1 for 100% Right and +1 for 100% Left), and use a Policy Gradient method. There are a few different choices of Policy Gradient, but they all share the basic principle of modelling a policy function - which might be a probability distribution over action space which you need to sample from, or a specific action - converting from state to action.

The most basic Policy Gradient approach, which you could use as an introduction for your problem if it is simple enough, is REINFORCE. Algorithms like A3C, DDPG, PPO are based over this idea and can cope with more challenging environments.


Customer Segmentation and Category association I have to solve two questions on the following dataset:
1. arrange customers into mutually exclusive groups.explain the clusters.
2.identify 1-1 product category association rules for each cluster, i.e. if a customer bought from this category, they are likely to buy from this category too.

Question 1.:

Encode the columns, (Labelencoding, tf-idf, feature generation etc...)<-> quantify the columns, than iterate fastly with different algos and evalute and compare the results, for example with silhuette score.

Question 2. Once you identified the best clustering algorithm, look for the categories that (uniquely or very likely) determine certain cluster/group. It wont be 1-1 but in probabilities determined.


Train error vs. Test error in linear regression by samples analysis I have run a multivariate linear regression model on a small set of about 3500 samples. While the model's error is as large as expected, I also ran a bias vs. variance analysis by comparing the train set error vs. the test set error using different sample sizes. I was expecting something like this:

But instead I found that the train error doesn't plateau at any point.

The code that generates this graph is the following:
def loss_by_sample(X, Y, testX, testY, learning_rate):
    samples_list = list()
    train_loss_list = list()
    test_loss_list = list()
    m = X.shape[0]
    for i in range(50, len(X), 50):
        samples_list.append(i)
        weights, loss = gradient_descent(learning_rate, X[:i], Y[:i])

        X2 = tf.concat([tf.ones([X.shape[0], 1]), X], 1)
        train_loss = (1/(2 * m) * tf.tensordot(tf.transpose(h(X2[:i], weights) - Y[:i]), (h(X2[:i], weights) - Y[:i]), axes=1))[0][0]
        train_loss_list.append(train_loss)

        X2 = tf.concat([tf.ones([testX.shape[0], 1]), testX], 1)
        test_loss = (1/(2 * m) * tf.tensordot(tf.transpose(h(X2, weights) - testY), (h(X2, weights) - testY), axes=1))[0][0]
        test_loss_list.append(test_loss)
    // plot train_loss, test_loss

I have tried different learning rates and the one I picked is the one that minimizes the loss using mean squared error:

(values to the right of the x axis after the last datapoint are inf.
Is there something I can conclude from this? Any reason why the train error can surpass the test error?

It occurs because your data is noisy. Suppose that your model is
$$ y = X\beta + noise. $$
Suppose that $\beta$ is recovered exactly. Since the noise is present, our model will NOT predict accurately, it will be always some noise. In other words, think that any time we predict the values, there is some random perturbation. The amount of perturbation is of the same order each time. When we compute the error, we square these perturbations, sum them up, and then take the square root, i.e., compute the norm
In your example, the size of the training set is growing, and, therefore, the test error WITHOUT averaging is growing as well -- you sum up squares of the perturbation. The size of the test set is steady, so the norm of error is about the same there.
Solution: use appropriate averaging to both train and test set.
train_loss = (1/np.sqrt(i) * tf.tensordot(tf.transpose(h(X2[:i], weights) - Y[:i]), (h(X2[:i], weights) - Y[:i]), axes=1))[0][0]
test_loss = (1/(2 * X2.shape[0]) * tf.tensordot(tf.transpose(h(X2, weights) - testY), (h(X2, weights) - testY), axes=1))[0][0]

P.S. X2 in my formulas are as defined in your cell, accordingly. I would suggest choosing a different name for the variable.


FastText Model Explained I was reading the FastText paper and I have a few questions about the model used for classification. Since I am not from NLP background, some I am unfamiliar with the jargon.
In the figure, what exactly is are the $x_i$? I am not sure what $N$ ngram features mean. If my document has total $L$ words, then how can I represent the entire document using $N$ variables ($x_1$,..,$x_n$)? What exactly is $N$?

$$-\frac{1}{N}\sum_{n=1}^Ny_n\log(f(BAx_n)) $$
If $y_n$ is the label, then what sense does it make to multiply it with the output vector after softmax (lables would be like 0,1,2,3,.. )? Does the author mean we take the $y_n$-th component of the output vector in loss calculation?

The formula would make sense if $y_n$ is a row vector representing the one-hot encoded label of the classes, and the multiplication is with the single column matrix $log(f(B A x_n))$ representing the log likelihood over all the clases given by the softmax function $f$.

As for $x_n$, it of course must be a vector as well, representing the $N$-grams in the $n$-th document.


Pandas - Avoid boolean result when using groupby() I have this script:
    sectors = df.groupby(['company_sector']).mean()['investment_in_millions']

Output:

I wanted to keep the same groupy() but having a result in "investment_in_millions" column filtered as mean > 10 or another value.
If apply this:
     sectors = df.groupby(['company_sector']).mean()['investment_in_millions']>10

I keep the groupby() but it returns a boolean into the investment column.
If I use:
     filtered = df[df['investment_in_millions']>10]

I get the filtered values mean>10 but the groupby() is not there anymore and I get all the other columns in the excel.
How can I get the groupby() together with the mean>10 without getting a boolean result?
Thanks!

First filter your results:
filtered_df = df[df['investment_in_millions']>10]

And then group it by company_sector
import numpy as np

sectors = filtered_df.groupby(['company_sector']).agg({"investment_in_millions":np.mean})

You can do it in one line:
sectors = df[df['investment_in_millions']>10].groupby(['company_sector']).agg({"investment_in_millions":np.mean})


Is knn similar to this version of k-means? If we use k-means in a dataset where k is equal to the number of points in the dataset, and each cluster is made out of only a point. Considering that we have given a distance method, we can classify a test instance associating it with the cluster whose centroid is nearest to that given point. The centroids are not changing in the test phase.
With what automatic classification method is the above strategy similar to?
Yes, this is a homework. My answer was knn, is this correct? But I do not know how to argue this properly.

Your case is where K=number of points in dataset :
K-means: Lets suppose, there are 10 data points and k=10, so you have 10 clusters. the new test point will be matched with the cluster nearest to it
KNN: If K=1, then the new test point classified will be same as in K-means.
So, if there is one data-point per cluster, then your given answer ie. knn is technically correct, if we take k=1.


Adaptive learning of user's IoT setting preferences We are working on a project to Predict the settings of the IoT devices(fan/light/AC) the user is using, based on his: location, outside temperature, humidity, time of the day etc...

Training data is not available and the model should start building/adapting itself every time the user uses the IoT device, to predict his preferred settings in any new environment he goes. This model will be unique for each user. Possibly we can start off with a demo model but it should adapt itself as per each user's preferences.

Which (machine?)learning algorithm can be useful here? Any links to the same and some tips on implementation would be appreciated.

This sounds like a fun problem but it very open ended!  I will provide some links from the Scikit-Learn User Guide.  There are many more reading options, but the SKL Userguide has lots of examples and usually links to academic publications for more in-depth reading.  Another great resource is: Introduction to Statistical Learning, and if you are good at math: Elements of Statistical Learning

Model Selection

The first big question is whether the settings you are trying to predict are ordinal values or continuous values?  Continuous value will allow you to use regression methods, where as ordinal values will give you a choice between regression methods and classification schemes.  I would suggest employing a couple of different models to start and then selecting the best one moving forward.  Some possible candidates are:

Linear regression (with nonlinear features as appropriate) because linear models are usually insightful,
Support Vector Classification/Regression (SVC/SVR) as these are often very accurate classifiers. and either
Naive Bayes or Random Forests as these often give good results where other models fail.

Initial Training Data

You will need to prime your solution with some deterministic data or data from another dataset that you have adapted to fit your model.  I suggest priming the system with common sense values and then employing some sort of algorithm to forget those values as the system becomes well-trained.

Moving Forward

You will have to decide whether each user's model will use other user's data or not.  This is sometimes solved via clustering methods like k-means or DBSCAN), where by users are clustered into groups each cluster has a different model associated with it.  I would suggest retraining (fitting) the model on regular time frames initially, and then investigating the possibility of moving to an online learning system.

I absolutely suggest using a well known library for your problem. Scikit-Learn is great for the reasons mentioned above and for prototyping, but doesn't scale particularly well. H20 and Mahout are high quality scalable libraries that I would recommend for a big data production system.  Mahout in Action is a great book to learn from, also.

Hope this helps!


cross validation on whole data set or training data? I am always having cross validation score smaller then the training score and I am performing cross validation on just training data is that normal thing ? Kfold = 5

Yes, it's called overfitting. Your model is beginning to memorize the training set, but not performing well on any validation or test set. If your question is why is this happening, I'd like to refer you to another answer I wrote explaining this phenomenon in more detail.

One interesting question that could be made is why is the performance on the cross-validation folds worse than on the test set?

This is a bit more tough to answer, because I don't have all the details. Some possible explanations could be that the since the training set is larger than each fold, the model was trained better, or that simply the test set examples happened to be easier.


Determining Important Atrributes with Feature Selection I have a labeled dataset of two classes. Let’s say sick and healthy patients. My features are patient data as well as diagnostic data such as blood-test results. It’s categorical as well as continuous data. In total several thousand features.

I want to identify the features that best differentiate the sick patients from the healthy ones.

It is not really an anomaly detection problem, because the sickness is not directly indicated by anything in my features. Meaning that for none of my features the sick patients would be indicated by outliers. For example a result could be that the risk of sickness is higher for patients over the age of 50. But being over 50 is not generally uncommon.

The obvious approach would be to use a dedicated feature selection technique. For example a chi-square test, variable importance in decision trees or backward elimination.

The problem about all of these approaches is that they are usually preprocessing steps before another learning algorithm is deployed. I on the other hand don’t want to use another learning algorithm. The entire task is to figure out the most important features.

Are feature selection algorithms really the best way to proceed, or are there any better approaches? Maybe some techniques especially designed for my kind of problems?

Many thanks in advance!

Sounds like pretty orthodox feature importance analysis. Easy option:

FeaturePlot from caret

It basically creates a correlation matrix for you to see which seem to separate most.

Slightly fancier:

VarImpPlot from randomForest

This uses the random forest ML technique to directly tell you which features were most important. It sounds very similar to what you're looking for.

Edit:
Here is a more detailed discussion of VarImpPlot.


Can't interpret the text information and ratings matrix imported to NN I have a Recommender system which uses a Collaborative bayesian approach using pSDAE for recommending scientific articles from the Citeulike Dataset

The text information (as input to pSDAE) is in the file mult.dat and the rating matrix (as input for the MF part) is in the file cf-train-1-users.dat and is loaded using the following code:

def get_mult():
    X = read_mult('mult.dat',8000).astype(np.float32)
    return X

def read_user(f_in='cf-train-1-users.dat',num_u=5551,num_v=16980):
    fp = open(f_in)
    R = np.mat(np.zeros((num_u,num_v)))
    for i,line in enumerate(fp):
        segs = line.strip().split(' ')[1:]
        for seg in segs:
            R[i,int(seg)] = 1
    return R

The raw data is in proper Excel format with citations as doc-id, title, citeulike-id, raw-title, raw-abstract.

The mult.dat file containing hte text information looks like:

63 1:2 1666:1 132:1 901:1 1537:2 8:1 9:1 912:1

The trainusers.dat file looks like:

10 1631 3591 10272 14851 4662 13172 12684 5324 3595 3404

Here is the link to th ipynb for the whole Recommender system:

https://github.com/js05212/MXNet-for-CDL/blob/master/collaborative-dl.ipynb

I am the author of the CDL paper.

For the mult.data file, in

63 1:2 1666:1 132:1 901:1 1537:2 8:1 9:1 912:1

63 is the number of words for this documents, 1:2 means word 1 appears twice in the document, 1666:1 means word 1666 appears once in the document, etc.

For the trainuser.dat, in

10 1631 3591 10272 14851 4662 13172 12684 5324 3595 3404

10 is the number of positive samples for this user, the rest is a list of 10 items that are related to (liked by) this user.

You can check the README file in the Datasets collection for more details on the datasets.


Matching Algorithms in Python We run a online marketplace for Commercial Real Estate industry and are looking to write matching algorithms to reduce the cost of search and transaction for the property owners/tenants.

We have two groups of users - owners and tenants and would like to implement matching algorithms based on their characteristics. Without any prior transactions data to work with, how would you approach this problems? More specifically, what matching algorithm techniques would you implement?

Code is in python.

Example:

Property

"property": [
{
       "amenities": ["kitchen",
                     "conference room",
                     .....,
                     .....]

        "location": "Munich",
        "feature2: "xyz"
        ...
}

Tenants

"Tenants": [
{
       "amenities": ["kitchen",
                     "conference room",
                     .....,
                     .....]

        "location": "Munich",
        "Type": 'Retail',
        "feature3": 'xyz'
        ...
}

According to your description you can only use similarities between descriptions, and since there's no labelled data it has to be unsupervised.

Option 1: heuristic (i.e. ad-hoc unsupervised method). Based on your knowledge of the specifics of the data, implement a function which returns a score representing how similar two descriptions are. For example a basic matching could simply count the number of common items between the two descriptions divided by the number of items in the longest description. For a query description return the N top similar matches after comparing it to every potential match. Naturally this can be improved in many ways.
Option 2: represent each description as a vector and use any generic similarity measure over pairs of vectors (e.g. cosine).  The representation can be some kind of direct representation, e.g. some kind of TF-IDF vector, or a more sophisticated  representation as a tree or graph embedding.


Machine learning method to predict event date Let's say I have a big dataset consisting of variables including but not limited to the start/end date of loans, their notional amount, a loan prepayment indicator etc.

My goal is to create a model that will be trained on past data in order to predict the prepayment date of current loans and I was wondering which ML method would be most suitable for this case. My first thought was to handle this as a classification problem, using interval dates to predict a prepayment date interval, but I believe that there should be a more robust & sophisticated approach to this.

You can both utilize regression and classification models in here.
Also I suggest log transform your financial features (eg: debt, last credit risk, average loan amount etc) when using algorithms except decision tree based models (since they are not affected by monotonic transformations).

Classification

You can construct a model such that predicts if a customer would pay "x" days after payment day or not. In this model you need to find best "x" by comparing results of different x day models. Also you need to find optimal confidence range in each model in comparison.

Regression

In this setting, you will predict how much day a customer will pay after payment date. Again you should obtain confidence intervals such that you calculate 3 days for a customer which means she will pay 3 days after (it can be earlier for negative predictions). However you know that your model is powerful on +2 -2 day interval. Then you should say this particular customer will pay in 1 or 5 days after payment date.

Algorithms

I suggest xgboost, lightgbm and RNNs. But its not so clear maybe you have a linear space and it converges fast with SVMs, line fits etc. Yet they are pretty good, kaggle winner learners.

I hope it helps. I thinks it will be a good starting point.


COUNT on External Table in HIVE I have been trying around the EXTERNAL table concepts in HIVE

CREATE EXTERNAL TABLE IF NOT EXISTS MovieData
(id INT, title STRING,releasedate date, videodate date,
URL STRING,unknown TINYINT, Action TINYINT, Adventure TINYINT,
Animation TINYINT,Children TINYINT, Comedy TINYINT, Crime TINYINT,
Documentary TINYINT, Drama TINYINT, Fantasy TINYINT,
Film-Noir TINYINT, Horror TINYINT, Musical TINYINT,
Mystery TINYINT, Romance TINYINT, Sci-Fi TINYINT,
Thriller TINYINT, War TINYINT, Western TINYINT)
COMMENT 'This is a list of movies and its genre'
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '|'
LINES TERMINATED BY '\n'
STORED AS TEXTFILE;

Created a table using the above statement and then used the LOAD statement to get the data populated.

LOAD DATA LOCAL INPATH '/home/ubuntu/MovieLens.txt' INTO TABLE MovieData;

Next time I DROP the table in HIVE and recreate it again and LOAD the data... But when I do a COUNT operation on the table I get double the values that's present in the file that I loaded.

I read through few articles that EXTERNAL table does not delete the data but the schema alone from the HIVE metastore... External Table

Can you please advise why does HIVE behave this way...

In Hadoop framework, there are multiple way to analyze the data. This depends on your use case, expertise and preference. Hive EXTERNAL tables are designed in a such way that other programmer can also share the same data location from other data processing model like Pig, MapReduce Programming, Spark and other without affecting each other work.

In case of external table you will not loose the data if you have accidentally dropped your table  as you already know it only drops the meta data and delete the schema and data remain untouched.

This method is useful if there is already legacy data in HDFS on which the user wants to put some metadata so that the data can be queried and manipulated using Hive. Or you are loading data on HDFS from other ETL tools.

Since EXTERNAL table doesn't delete the data and you are loading file again you are getting the count difference. if you are on your own to do all operation like load, analysis, drop etc, Hive support the INTERNAL table as well. If you want to delete the data when you drop table you can use Hive INTERNAL table. To create internal table you just need to remove the EXTERNAL keyword from your query and when you will drop this table it will delete the data as well.


Why is shuffling timeseries a bad thing? I'm trying to understand precisely why it is a bad idea to shuffle time-series when splitting train and test data.
Like, what is false about shuffling time-series? How does it tamper with the model?

Because the different observations in a timeseries by definition have an order, i.e. Jan 1st comes before Jan 2nd. If you then shuffle your observations this inherent order will be lost and you might be leaking data, meaning that your model will see data that is actually in the future since Jan 31st might suddenly be before Jan 1st.


Help training Back-propagation Neural Network With 150k training pairs I am currently trying to train my backpropagation to classify 150k training pairs. Each training pair is a vector of 18 Bipolar numbers and it runs through 2 hidden layers with a final output of 1 number(18-18-18-1).

When I feed my neural network with only a couple thousand training pairs it can classify them 100% but when I try to feed more it makes many errors. Does anyone have any recommendations for what can help a back prop net handle more training pairs? Should I do batch updating of weights or update them for every training pair(what I do now)? Also whats the best way to test for convergence with so many training pairs(when summing up error it's ~40k).Any help is much appreciated.

(FYI: each training pair is a loan account I am trying to evaluate with data on balance, age, how long it's been open, average income of zipcode and the target is whether or not it was paid)

Typical things to look for when training neural networks:

Learning Curve: a plot of the error on the validation set X size of training set. The error must decrease as the network is trained, so if you see fluctuations or an increase something is wrong.
Learning Rate: the learning rate influences the convergence of gradient descent. A small learning rate will make the model overfit the data while a big one may make it underfit.
Precision: of the classified examples in the training set, how many did the network got right, or more precisely, how many did it classify as expected?
Recall: of all the positive examples, in your case the paid ones, how many did it classify as paid?

If you notice the performance of the network is bad you may:

Add other features that make the model fit the data better - for example add average family income to the features
Get more training examples
Some papers suggest deeper networks are able to catch subtleties about the model they are trying to fit, so experimenting with the network architecture is an option.


Formula to calculate confidence value in Adaboost I am coding an AdaBoostClassifier with the two class variant of SAMME algorithm. Here is the code.

def I(flag):
    return 1 if flag else 0

def sign(x):
    return abs(x)/x if x!=0 else 1

AdaBoost Class

class AdaBoost:

    def __init__(self,n_estimators=50):
        self.n_estimators = n_estimators
        self.models = [None]*n_estimators

    def fit(self,X,y):

        X = np.float64(X)
        N = len(y)
        w = np.array([1/N for i in range(N)])

        for m in range(self.n_estimators):

            Gm = DecisionTreeClassifier(max_depth=1)\
                        .fit(X,y,sample_weight=w).predict

            errM = sum([w[i]*I(y[i]!=Gm(X[i].reshape(1,-1))) \
                        for i in range(N)])/sum(w)

            '''Confidence Value'''
            #BetaM = (1/2)*(np.log((1-errM)/errM))
            BetaM = np.log((1-errM)/errM)

            w = [w[i]*np.exp(BetaM*I(y[i]!=Gm(X[i].reshape(1,-1))))\
                     for i in range(N)]

            self.models[m] = (BetaM,Gm)

    def predict(self,X):

        y = 0
        for m in range(self.n_estimators):
            BetaM,Gm = self.models[m]
            y += BetaM*Gm(X)
        signA = np.vectorize(sign)
        y = np.where(signA(y)==-1,-1,1)
        return y

The much I know the formula for confidence is

The much I read, the actual minima occurs when c=1/2 but for any value of c the classifier should produce the same result. But when I am coding the class the output for c = 1 and c = (1/2) are coming different. Moreover if I am not multiplying anything ie. c=1 then the output of my classifier is better and produces identical results with the sklearn implementation of AdaBoost Classifier.

So why multiplying 1/2 is giving bad results?

Actually this equation is not quite right.

The actual minima occurs at

In the derivation this BetaM is the solution to the equation

Notice  how in the function we are scaling the weights of wrong predictions up by exp(BetaM) and scaling down the weights of correct predictions by multiplying exp(-BetaM).

But in the code that I have written (also in the sklearn implementation) only the wrong predictions are being scaled up. So to get the relative scaling correct we have to scale the wrong predictions by exp(2BetaM)
i.e.

And as of the hypothesis function has the form

Any scalar multiple of BetaM will work fine as all the predictions will be scaled by the same amount. So for convenience in coding BetaM is written as just

Note: You can perfectly use BetaM = 0.5 * log((1/errM)/errM) . But then you have to scale down the correct predictions too. If you do that then the code will give correct results.


How to interpret the original time series approximated using principal component analysis? I've read some posts about PCA applied on time series, but still a bit confused and I have the following questions(Suppose I am working with a time series of the return of 50 industries and I want to use a clustering algorithm to divide them into several group):

Say I have calculated the eigenvalue and eigenvector from the correlation matrix, and found that the first twenty eigenvalue account for 85% of the total, and I then use these twenty eigenvalue to approximated the original time series. I know if I choose all eigenvalues then I can get the identical original time series, but what information did I lose if I choose twenty of them specifically? What is the purpose to do so?

I found some post said we can always drop the first principle component(means we don't use it), why can we do that?

Can I interpret each eigenvalue as a trend in the market, for example the first principle component, can I derive whether the corresponding industry is in the same direction or different from the market trend based on the sign of its corresponding eigenvector, and if so, can I apply k-means to all industries by using the eigenvectors of the first few principle component to group them, is this make sense?

Welcome for any kind of hint or ideas, thanks.

Usually, the norm to asking any question on Stack Overflow or any other sister websites is that one is supposed to ask one question only unless they are very similar which doesn't seem to be in this case.
To answer your questions, $X^TX$ is called Sample Covariance (or Correlation) Matrix where $X$ is the data matrix of dimensionality $(m,d)$. So, the resultant matrix has a dimensionality of $(d,d)$ where $d$ is the dimensionality of the feature space.
And as you said, this matrix is made to go an eigendecomposition to get $w\cap w^{-1}$ where $\cap$ is a diagonal matrix of eigenvalues, arranged in decreasing order and $w$ is the normalized eigenvectors stacked according to the corresponding eigenvalues. The reason one might want to choose some $k$ number of dimensions is to reduce dimensionality.
Reduced dimensionality provides multiple benefits - reducing space complexity, faster computations etc. The problem arises when you specifically talk about time series. PCA, ICA does not take into account temporal dependence which might cause data to have suboptimal forecast. There are different ways to tackle this issue, one might want to use Forecastable Component Analysis, autoencoders etc to ensure they are reaping the benefits of not only gaining the benefits that PCA provides but also ensuring that the problems with PCA's are avoided.
To answer the second question, I'm NOT sure of the reason but one may want to drop the first principal because it is in the direction of maximum variance, i.e. it varies most in this direction.


Deduplicating e-commerce products based on title/description/images I am trying to clean a set of e-commerce data.
Data consists of products which has multiple images, a title and a description.
Some of these products has duplicates and I need to de-duplicate the set.
I have worked on NN's and Genetic Algorithms back in college but that's it.

What I need is a starting point to dive into. What methods/techniques should I use, what should I learn first?

As a starting point chapter 3 of Mining Massive Datasets (MMDS) offers a very good introduction on document similarity. You can also look here. For near-duplicate image detection look at this question on StackOverflow.

For the text data, first you have to convert your strings to a vector of features, this features could be tf-idf vectors of all the tokens (words) that appear in the database, could also be a vector of n-grams. For a discussion of semantic and syntactic features you can look here.

Given the vectors for each product on your database you would have to compute the similarity among all pair of products. Then the most common similarities measures are cosine similarity and Jaccard similarity.

The brute force approach is O(n^2), where is the number of products in the database so this could be unfeasible. A common technique to deal with this issue is to use fingerprinting (hashing) together with Locality Sensitive Hashing (LSH), both techniques are discussed in chapter 3 of MMDS.


How can we show that one model might have higher accuracy than another model but at the same time lower AUC? Assume that we have two classiﬁcation models M1 and M2 that are evaluated on ﬁve test instances. How to show with an example that M1 can have a higher accuracy than M2, while at the same time M2 has a higher area under the ROC curve (AUC) than M1?

Normally this can happen when there's imbalance in your classes. Imagine that you want to predict these 5 values:

y_test = [1,0,0,0,0]

and your models M1 and M2 predict the following:

M1_pred = [0,0,0,0,0]
M2_pred = [1,0,0,1,1]

Clearly we can see that M1 is biased towards class 0, getting 4/5 results correct (80% accuracy!). While the second one, only gets 3/5 values right (60% accuracy).

However, because AUC ROC is defined by both true positive rate against the false positive, combining those the M2 classifier gets a 75% auc, while M1 only a 50% because it has failed the only one positive, thus penalising the final score.

Try running this toy example:

from sklearn.metrics import accuracy_score
from sklearn.metrics import roc_curve, auc
import matplotlib.pyplot as plt

y_test = [1,0,0,0,0] #ground truth

M1_pred = [0,0,0,0,0] #predictions M1
M2_pred = [1,0,0,1,1] #predictions M2

#Calculate both accuracy scores
M1_acc = accuracy_score(y_test, M1_pred)
M2_acc = accuracy_score(y_test, M2_pred)

#Calculate both ROC AUC
fpr1, tpr1, _ = roc_curve(y_test, M1_pred)
roc_auc1 = auc(fpr1, tpr1)
fpr2, tpr2, _ = roc_curve(y_test, M2_pred)
roc_auc2 = auc(fpr2, tpr2)

#Plot both graphs
plt.figure(figsize = (12,6))

#Graph M1
plt.subplot(121)
lw = 2
plt.plot(fpr1, tpr1, color='darkorange',
         lw=lw, label='ROC curve (area = %0.2f)\n ACC Score = %0.2f' % (roc_auc1, M1_acc))
plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic ROC')
plt.legend(loc="lower right")

#Graph M2
plt.subplot(122)
plt.plot(fpr2, tpr2, color='darkorange',
         lw=lw, label='ROC curve (area = %0.2f)\n ACC Score = %0.2f' % (roc_auc2, M2_acc))
plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic ROC')
plt.legend(loc="lower right")

plt.show()

and you will get something like this:


How to implement accurate counts or sums when you have different numbers of days of the week? I have a dataset of transaction data for a retail outlet. I am using pandas and want to analyse revenue by day of the week, but there are unequal days in the dataset (i.e. an extra weekend). I have used df.dt.dayofweek to create an integer value for the day of the week, and grouped the data by that integer value using df.groupby(["Day_int",]).sum()

So I have a 'total' column at the end that I am interested in, but I want to create a a new column, something like 'adj_total' that applies a division operation of /3 to the days Monday to Friday and /4 to the days Saturday and Sunday. Is the best way to loop through the dataframe or is it better

Here is the df that I am working with (most of the column values are nonsensical, only total is of interest). Day_int is the group_by variable and should appear lower than the other columns.

Day_int Section Prod_name Cashier Date Time Receipt Total
0 ..................................................91341
1 ..................................................82262
2 ..................................................84145
3 ..................................................90115
4 ..................................................115497
5 ..................................................151971
6 ..................................................109210

I would add a column that is a 3 if it's a weekday and a 4 if it's not using an apply, something like this:

df['divide_by'] = df.apply(lambda x: 3 if x['Day_int']<5 else 4, axis=1)

Assuming days are Monday 0 to Sunday 6. Then you can add the column as follows:

df['adj_total'] = df.apply(lambda x: x['Total']/x['divide_by'], axis=1)

Then you can remove the divide_by column and you have the result


How to interpret ANOVA results? I am trying to identify what attributes are not relevant in my dataset to remove them before fitting a classifier.

The target is a categorical variable with three different values.

I also have a lot of numerical attributes.

For ANOVA, I used the following code:

grouped_test2=df[['room_type', 'price']].groupby(['room_type'])
f_val, p_val = stats.f_oneway(grouped_test2.get_group('Entire home/apt')['price'], grouped_test2.get_group('Private room')['price'], grouped_test2.get_group('Shared room')['price'])

The independent variable is room_type, and the explanatory variable is price.

In this case, the f_val is equal to 1061.64 and p_val is equal to 0.

I read that 0 or values near 0 imply a relationship between the two variables but I am not sure about that?

What mean f_val is near enough to 0 to can say that the two variables are related?

f_val is the F Statistic value. Mathematically it is

$ F = \frac{MS_{Between}}{MS_{Within}}$

The null hypothesis for your ANOVA is

$H_0: \mu_{Entire home/apt} = \mu_{Private room} = \mu_{Shared room} $ which means all means ($\mu_i$ s) are equal and there is no need of grouping using explanatory variable

vs

$H_A:$ At least one $\mu_i$ is different. There is a need for grouping

The p-value for this test was very very low, hence python returned 0. Anything less than 0.05 is considered low enough to reject the null hypothesis.

Independent variables are also called explanatory variables. I believe price is a dependent variable.


From analysing their previos transactions how can I predict for what type of product is the customer more likely to take take an EMI So basically I need a kind of product/ category affinity for EMI for all customers  eg - Customer A is more likely to take an EMI on her insurance premium.
One approach I had thought was to broadly categorize the the transactions into 3-4 categories and predict the amount( linear regression)  that a customer is going to spend on each of the category in that particular month.eg It is estimated that customer A will spend huge amount on category 1 in month of September, then this customer A will be targetted for a marketing campaign for taking EMI for products in category 1 in September
Another approach I had thought was to do a multi class logistic regression to classify each customer into 3-4 category and marketing can target customers for EMI of specific products
Or I could do clustering and check the conversion for EMI of different types of categories in each cluster
Please let me know if I'm going in the right direction or if there is any other approach I could take or any algorithm I can use.
Thank-you

This problem could also be formulated as a recommendation system for your marketing department.
Given the user's past purchases and the product information and payment methods, you could build a recommendation model to recommend products for your marketing department to push forward for the specific customer.


GPA prediction of college student I have a dataset consist of 8 columns and 15600 rows with the following columns:-

1.Entry_academic_year which have 5 discrete value (2558,2559,2560,2561,2562)
2.Faculty (It is the faculty that student has taken like engineering)
3.branch  (It is the branch  that student has taken like software engineering)
4.Admission type (how the student enter the college)
5.Graduated_high_school (it is the high school where student got graduated)
6.province_of_school
7.GPA_high_school(It is the GPA of student in high school)
8.GPA_college(It is the GPA of the student during college)

I am trying to predict the GPA of the student at the college by dividing the GPA into 4 quartiles with respect to percentile (25,50,75), The problem I faced is that the Graduated_high_school columns have around 1732 unique value with some school contain only one row which makes the prediction around 30-35 % accuracy
Any idea on how to fix it?

Perhaps you can see if Graduated_high_school is correlated in any way to GPA_college? If there is no correlation, you can try to fit a model by dropping the Graduated_high_school column.
Else, you can try to drop rows belonging to under-represented high schools. However, one problem I foresee is that future predictions might have Graduated_high_school that are unseen in the training dataset, leading to problems (e.g. schools that weren't mentioned in the dataset, or if someone  decides to use your model, on a dataset from another country). So, if the Graduated_high_school is not important, I would consider dropping it altogether.
Or, maybe you can change Graduated_high_school to something else that is related, such as number of teachers in high school, teacher-student ratio etc.


How exactly does DropOut work with convolutional layers? Dropout (paper, explanation) sets the output of some neurons to zero. So for a MLP, you could have the following architecture for the Iris flower dataset:

4 : 50 (tanh) : dropout (0.5) : 20 (tanh) : 3 (softmax)

It would work like this:

$$softmax(W_3 \cdot \tanh(W_2 \cdot \text{mask}(D, \tanh(W_1 \cdot input\_vector)))$$

with $input\_vector \in \mathbb{R}^{4 \times 1}$, $W_1 \in \mathbb{R}^{50 \times 4}$, $D \in \{0, 1\}^{50 \times 1}$, $W_2 \in \mathbb{R}^{20 \times 50}$, $W_3 \in \mathbb{R}^{20 \times 3}$ (ignoring biases for the sake of simplictiy).

With $D = (d)_{ij}$ and

$$d_{ij} \sim B(1, p=0.5)$$

where the $\text{mask}(D, M)$ operation multiplies $D$ point-wise with $M$ (see Hadamard product).

Hence we just sample the matrix $D$ each time and thus the dropout becomes a multiplication of a node with 0.

But for CNNs, it is not clear to me what exactly is dropped out. I can see three possibilities:

Dropping complete feature maps (hence a kernel)
Dropping one element of a kernel (replacing an element of a kernel by 0)
Dropping one element of a feature map

Please add a reference / quote to your answer.

My thoughts

I think Lasagne does (3) (see code). This might be the simplest to implement. However, closer to the original idea might be (1).

Seems to be similar for Caffe (see code). For tensorflow, the user has to decide (code - I'm not sure what happens when noise_shape=None is passed).

How it should be

(2) and (3) don't make much sense as it would cause the network to add invariance to spacial positions, which is probably not desired. Hence (1) is the only variant which makes sense. But I'm not sure what happens if you use the default implementation.

As you mentioned, the mask matrix is sampled and multiplied with the activations in the feature map at layer $l$ to produce dropped out modified activations which are then convolved with the filter at the next layer $W^{(l+1)}$. (3)

For more details, I think section 3 in this paper might help you out: Max-pooling & Convolutional dropout. Specifically 3.2.

When you test you use all nodes of the network but with the filter’s weights scaled by the retaining probability, as explained in the paper.

Please feel free to refine or correct my answer.

Hope this helps at least a little.


Value error in an embedding layer I am new to deep learning and I am trying to build a book recommender system using embedding layers. I use one layer for the book and one for the user.

I am having trouble with fitting the model. More specifically, when I try to feed the first layer with the books' ISBN list I get back the " ValueError : could not convert string to float: '087584877X' ".

Note that a book's ISBN ( as '087584877X' for example )  is either a sequence of numbers or a sequence of letters and numbers. The second category seems to be the problematic one.

I dont get why it should convert the string to float. I though that what the network does is to assign each string to a new integer regardless of the structure of the string.

Here is my code for the books embedding layer :

book_input = Input(shape=(1,))

book_embedding = Embedding(input_dim = n_books+1, output_dim = 4)(book_input)

It looks like you're using Keras, so this answer is going to refer to the Keras embedding layer.

The error message is pretty clear - Keras embedding layers don't accept strings as inputs. The embedding layer requires an array of positive integers and can only be used as the first layer of a model. The embedding layer requires that you encode each word as an integer so you might have to use the Keras tokenizer.

Source: Embedding Layers


How much of data wrangling is a data scientist's job? I'm currently working as a data scientist at a large company (my first job as a DS, so this question may be a result of my lack of experience). They have a huge backlog of really important data science projects that would have a great positive impact if implemented. But.

Data pipelines are non-existent within the company, the standard procedure is for them to hand me gigabytes of TXT files whenever I need some information. Think of these files as tabular logs of transactions stored in arcane notation and structure. No whole piece of information is contained in one single data source, and they can't grant me access to their ERP database for "security reasons".

Initial data analysis for the simplest project requires brutal, excruciating data wrangling. More than 80% of a project's time spent is me trying to parse these files and cross data sources in order to build viable datasets. This is not a problem of simply handling missing data or preprocessing it, it's about the work it takes to build data that can be handled in the first place (solvable by dba or data engineering, not data science?).

1) Feels like most of the work is not related to data science at all. Is this accurate?

2) I know this is not a data-driven company with a high-level data engineering department, but it is my opinion that in order to build for a sustainable future of data science projects, minimum levels of data accessibility are required. Am I wrong?

3) Is this type of setup common for a company with serious data science needs?

This is a situation that many blogs, companies and papers acknowledge as something real in many cases.
In this paper Data Wrangling for Big Data: Challenges and Opportunities, there is a quote about it

data scientists spend from 50 percent to 80 percent of their time
collecting and preparing unruly digital data.

Also, you can read the source of that quote in this article from The New York Times, For Big-Data Scientists, ‘Janitor Work’ Is Key Hurdle to Insights
Unfortunately, the real world is not like Kaggle. You don't get a CSV or Excel file that you can just start the Data Exploration with a little bit of cleaning. You need to find the data in a format that is not suitable for your needs.
What you can do is make use of the old data as much as you can and try to adapt the storing of new data in a process that will be easier for you (or a future colleague) to work with.


How to generate ratings without training data? I am working on generating restaurant ratings automatically and I have various feature values like delivery time, cost estimate, etc. I want to generate a rating for each restaurant between 0 to 5. But I don't have any training data or ground truth to validate. This rating might vary with user. Most of the related work, mostly related to the Yelp data challenge, have some relevance score as training data. I though of using reinforcement learning to learn the rating with user feedback, but not sure how to do that. Can anyone please suggest a relevant technique or algorithm for this problem?

If you model your system by means of reinforcement learning, you will make your system learn from the user's feedback. The system will provide a rating based on the input features (it may be just a random rating during the first stages, since you will not have any prior information), and then the user will tell the system how well it predicted such rating. Based on the difference between the suggested and the actual user's ratings, the reinforcement learning algorithm will refine the recommendation system's model in order to provide more accurate ratings in the future.

Sutton's book on reinforcement learning (http://incompleteideas.net/book/the-book-2nd.html) is a good introduction to the reinforcement learning field.


Triplet loss function for face recognition? In the Andrew-NG coursera course on Convnets he talked about triplet loss function for one shot face recognition.

The formula given in the video is,
$$\to \small \small \small ||f(A)-f(P)||^2 \;+\;\alpha \leq\;||f(A)-f(N)||^2$$
$$\to \small \small \small D(A, P) + \alpha \leq D(A, N)$$
$$L(A,P,N) = max(||f(A)-f(P)||^2 - ||f(A)-f(N)||^2 + \alpha,  0)$$
Here,
$$f(A) - \small \text{ Person }A$$
$$f(P) - \small \text{Different Picture of Person }A$$
$$F(N) - \small \text{Another Person}$$

I couldn't understand why did we use $\alpha$ in the formula. I understood that the ideal loss function is to decrease $\small \small D(A, P)$ and increase $\small \small D(A, N)$ but if we add $\alpha$ to $\small \small D(A,P)$ it will increase it which is not we require right?

$\alpha$ is known as the margin.

Not only that we want to minimize $D(A,P)$ and maximize $D(A,N)$, that is we want  $D(A,P)-D(A,N)$ to be small. Not only that we want it to be non-positive, we want it to be sufficiently negative.

That is not only that we want  $$D(A,P)-D(A,N) \le 0$$

We want $$D(A,P)-D(A,N) \le -\alpha.$$


How do I calculate for each person in the network how many people agree with their opinion? I have a signed bipartite graph in which the nodes are (1) students and (2) topics. An edge is drawn between a student and topic node if the student mentions their opinion about the topic in a short answer (i.e., some students have an opinion about one topic but not another). The valence of the edge indicates whether the opinion is positive or negative.

My question is: how do I find out how many other students agree with a particular student? In terms of not just which topics they have an opinion on, but also what that opinion is (positive/negative).

EDIT: Based on a comment below

1) What exactly do you mean by agree? Should all existing opinions align or only those on a specific topic? What if one student has additional opinions about a topic?

All existing opinions (including topics and the valence) should align. If a student gives an same opinion about the same topics as another, but happens to talk about another topic as well, that would not count as full agreement. Perhaps there is a way to calculate partial agreement as well?

2) What exactly is your problem? Defining the characteristics is straightforward: Just count. Do you perhaps struggle with an algorithmic implementation?

Algorithmic implementation is perhaps what I am going for. Since I have 100 students, it would be difficult to hand-count the number of peers that agree with them. So if there is a way to calculate a value for each student, that would be helpful.

As a fast answer, you can represent each student as a vector with $K$ elements (where $K$ is the number of topics) and values $\{+1, 0, -1\}$, denoting positive/non-existent/negative opinion about this topic.

Then, a simple measure of agreement between two students is the element-wise product between two student-vectors. That is the product will be:
$similarity = \sum_{i=1}^{K}st_1[i]*st_2[i]$, where $st_1,st_2$ are the student-vectors. Obviously, only the topics where both students have aligned opinions will boost the total [e.g. $1*1=1$ and $(-1)*(-1)=1]$, while misaligned opinions will decrease the sum. If any of the two students haven't expressed an opinion about a topic, then this topic won't matter in the sum.

In that sense, you can find the most like-minded students to a specific student, as the ones with the highest $similarity$. If what you really need is a number of agreeing students for each unique student, then a threshold on the $similarity$ score can be set. The value of the threshold can be decided empirically from your data.

This is easily implemented and if you are comfortable with coding, I could post a sample script in python. One thing to consider though, is in what format is the bipartite graph (a .csv, a graph file of some kind etc.).

EDIT: MINOR EXAMPLE. Fetch example .csv file used from here.

import pandas as pd
import numpy as np

# Change location of file according to your needs
with open('students_example.csv', 'r') as f:
    df = pd.read_csv(f)
# Print for visualization
print(df.head())
print("~"*25)

# Delete column containing the student_id
del df['Student_ID']
# Parse the pandas DataFrame as matrix
student_vectors = df.as_matrix()
# The number of students at hand, let it be N.
N_students = student_vectors.shape[0]
# Initialize empty matrix of similarity between students
# Its size will be NxN (each student with each other)
similarity_scores = np.zeros((N_students, N_students))
# Iterate over each student vector and calculate the
# similarity with all students
for i, student in enumerate(student_vectors):
    # Reshaping and transposing to get the dot product between each student
    # And all the student vectors
    similarity_scores[i,:] = np.dot(student.reshape(1,-1), student_vectors.T)
# Fill the diagonal (that is the similarity of each student with him/herself)
# with low similarity scores so as not to confuse them with other possibly
# agreeing students
np.fill_diagonal(similarity_scores, -1000)

# Random wanted student for example purposes
wanted_id = 3
# Print Students Opinion
print("Wanted Students Opinion:")
print(df.loc[wanted_id].to_string())
print("~"*25)
print("Most similar:(Student ID = %d)"% np.argsort(similarity_scores[wanted_id,:])[::-1][0])
print df.loc[np.argsort(similarity_scores[wanted_id,:])[::-1][0]].to_string()
print("~"*25)
print("Second most similar:(Student ID = %d)"% np.argsort(similarity_scores[wanted_id,:])[::-1][1])
print df.loc[np.argsort(similarity_scores[wanted_id,:])[::-1][1]].to_string()
print("~"*25)

If you follow the example, the output for the wanted student (with $student_{ID}=3$) with opinions: {Trump -1, Net Neutrality   -1,Vaccination 1, Obamacare -1}

will give you two other students with the same opinions and their ids.

You can modify the script to fit your needs accordingly.

P.S.: Sorry for the messy code, it was written rather hastily. Also, i tried it with Python 2.7.


Guidance to create a machine learning alg to find multiple overlapping date over multiple period i have a dataset of work period from a lot of people during multiple years and i would like to find the overlapping date between these periods, the data would look something like this:

Name;Start;End;Valid
John Doe;2015-01-01;2017-01-01;Yes
Jonh Doe;2016-02-03;2016-06-04;No
Jane Doe;1980-01-01;2018-01-01;Yes

As you can see i can have multiple lines for the same person and are mapped as valid or not, there are a few more rules like if the person is a professor he can have overlapping work periods but the core is to find these wrong periods created by human error.

Since i am a beginner in machine learning i would to try to implement this as a learning project although i intend to use and expand in the future if all goes well, at least to my eyes seems like a somewhat easy task to do.

I would like your opinion in what kind of algorithm i should use, or what framework i should or not use, or even if this is not very cost effective to implement on ML.

I thought of somekind of supervised learning using tensorflow, maybe is too much?

Also any tips, tricks or tutorials would be much appreciate it.

EDIT: More Information
At first i am providing the answer, which would be the "Yes" or "No" on Valid, but when i provide something like:

Row;ID;Name;Start;End;
1;1;John Doe;2015-01-01;2017-01-01;
2;1;Jonh Doe;2016-02-03;2016-06-04;
3;1;Jonh Doe;2017-06-02;2018-04-01;
4;1;Jonh Doe;1990-01-01;2017-07-01;

So i would like it to point out, hey, John Doe has 2 invalid periods, row 2 and 4, why should it give me that? because 2016-02-03 to 2016-06-04 is already covered by row 1 and row 4 overlap with row 1,2,3

At first i thought on making this with regular programming to check for everything, but there are some cases where human error make this quite a difficult task to do, thats why ml got in to my mind, the algorithm does not need to have really high accuracy, this is mostly a learning project.

This is quite easy and you don't need machine learning at all.

Order by Person and Start date. Then simply iterate over each data line for a person and see if the start dates are earlier than the end dates for the previous entry. Add rules as necessary.


How to split the data as per the label name count using sklearn Before training a model, I would like to split the data in a 80/20 ratio.

For example, in my dataset of 3000 rows, I have different labels. Out of which A is one label name, has 100 records in the dataset. How can I divide the dataset so that 80% of the label A data is in training and 20% is in the test set.

Do we have any pre defined functions for this?

Use a stratified test split.

import numpy as np
from sklearn.model_selection import StratifiedShuffleSplit
X = np.array([[1, 3], [3, 7], [2, 4], [4, 8]])
y = np.array([0, 1, 0, 1])
stratSplit = StratifiedShuffleSplit(y, n_iter=1, test_size=0.5, random_state=42)
for train_idx, test_idx in stratSplit:
    X_train=X[train_idx]
    y_train=y[train_idx]

print(X_train)
# [[3 7]
#  [2 4]]
print(y_train)
# [1 0]

Source: https://stackoverflow.com/a/35473089/9524424


Unsupervised Sentimental Analysis in R How would you evaluate unsupervised sentimental analysis?
I am reading on evaluating sentimental analysis and learning that much of the classification models that are being used, the data has target/label variables. How would you evaluate a survey that is only text with no target/label variables? Right now, I am just giving the responses a sentimental score (positive/negative/neutral) but I am having trouble approaching validation with 1000+ responses. Any ideas/suggestions?

In a truly unsupervised setting, the only possibility is to cluster the documents. Typically this would be done with topic modelling. Then the result could be evaluated by inspecting the words most associated with every topic, and assign a sentiment class to the whole topic/cluster, i.e. all the documents labelled with it.


training and Predicting probabilities using logistic regression model I have these game data where the output variable is continous which indicates probability of winning. How do I train my classification model using this probabilites and predict output probabilites for test data

Since your output variable is continuous it can't be used directly in classification. There are some steps through which you can achieve inference.

Based on the probability, convert your 'y' (output) variable into a class label.

for example, if your class A & class B has a probability of 0.6, 0.4 respectively. convert it into a single variable y and it should have the value that represent the respective classes.

Now you have the features mapped to their respective class label. Now we can train the model with the available data.
After training the model, the prediction will give you class probabilities on your test data.

Note : Use softmax in case of neural networks to squash the output values to probabilities.


What statistical method should i use to find Correlation between number of days and AmountEarned I am new to Data Science and I have a python data frame with Number of days, CountofJobs, and AmountEarned what statistical method should I use to find a correlation between Days and AmountEarned.
NumberofDays     CountofJobs    AmountEarned
20                3                 50000
22                18                10000
35                10                80000

Whether to use pearson's correlation coefficient or Spearman's correlation coefficient depends on what you want to measure ?
Pearson's correlation coefficient measures the strength of a linear relationship between two variables. The variables should be on an interval or ratio scale. Ordinal variables cannot be used in pearson's test. The variables should satisfy certain assumption before the pearson's product moment correlation is applied. They are:

The variables should be approximately normally distributed
There should be no significant outliers in the data
The variables should have a linear relationship

In case, the variables do not satisfy these assumptions they can either be transformed to meet them or we can use the spearman's rank correlation coefficient to measure correlation.
The Spearman's method is primarily used when the data can be ranked, therefore correlation between ordinal, interval and ratio variables can be measured using this method. Also, this method measures the strength of a monotonic relationship between two variables.