How label smoothing and label flipping increases the performance of a machine learning model I have seen posts and research papers mentioning these techniques for improving the performance of a machine learning model.

These techniques certainly make some sense in the case when we are not sure about how correct is the labelling our dataset.

However, I am wondering if these two techniques are proven to be beneficial for a ML model in the case when the labelling is correct?

Label flipping is a training technique where one selectively manipulates the labels in order to make the model more robust against label noise and associated attacks - the specifics depend a lot on the nature of the noise. Label flipping bears no benefit only under the assumption that all labels are (and will always be) correct and that no adversaries exist. In cases where noise tolerance is desirable, training with label flipping is beneficial.

Label smoothing is a regularization technique (and then some) aimed at improving model performance. Its effect takes place irrespective of label correctness.

From 1 Adversarial Perturbations of Deep Neural Networks, 2016:

  Without label smoothing, a softmax classifier is trained to make infinitely confident  predictions on the training set. This encourages the model to learn large weights and strong responses. When values are pushed outside the areas where training data concentrates, the model makes even more extreme predictions when extrapolating linearly. Label smoothing penalizes the model for making overly confident predictions on the training set, forcing it to learn either a more non-linear function or a linear function with smaller slope. Extrapolations by the label-smoothed model are consequently less extreme.

From Regularizing Neural Networks by Penalizing Confident Output Distributions, 2017:

  Confident predictions correspond to output distributions that have low entropy. A network is over-confident when it places all probability on a single class in the training set, which is often a symptom of overfitting. The confidence penalty constitutes a regularization term that prevents these peaked distributions, leading to better generalization.

As a result of label smoothing, the model becomes more robust in general. Its increased ability to deal with incorrect labels is just part of the overall improvement. However, one cannot claim that the effects of label smoothing are purely beneficial.

From When Does Label Smoothing Help?, 2019:

  Despite having a positive effect on generalization and calibration, label smoothing can hurt distillation. We explain this effect in terms of erasure of information. With label smoothing, the model is encouraged to treat each incorrect class as equally probable. With hard targets, less structure is enforced in later representations, enabling more logit variation across predicted class and/or across examples. This can be quantified by estimating mutual information between input example and output logit and, as we have shown, label smoothing reduces mutual information.


Why underfitting is called high bias and overfitting is called high variance? I have been using terms like underfitting/overfitting and bias-variance tradeoff for quite some while in data science discussions and I understand that underfitting is associated with high bias and over fitting is associated with high variance. But what is the reason of such association or in terms of a model what is high bias and high variance, How can one understand it intuitively?

How can one understand it intuitively?

Underfitting is called "Simplifying assumption" (Model is HIGHLY BIASED towards its assumption). your model will think linear hyperplane is good enough to classify your data which may not be true. consider you are shown a picture of cat 1000 times, Now you are blindfolded, No matter Whatever you are shown the 1001th time, probability that you will say cat is very high(You are HIGHLY BIASED that the next picture is also gonna be a cat). Its because you believe its gonna be a cat anyway. Here you are simplifying assumptions

In statistics, Variance informally means how far your data is spread out.
Overfitting is you memorise 10 qns for your exam and on the next day exam, only one question has been asked in the question paper from that 10 you read. Now you will answer that one qn correctly just like in the book, but you have no idea what the remaining questions are(Question are HIGHLY VARIED from what you read). In overfitting, model will memorise the entire train data such that it will give high accuracy on train but will suck in test. Hope its helps


How to visualize multivariate regression results Are there commonly accepted ways to visualize the results of a multivariate regression for a non-quantitative audience? In particular, I'm asking how one should present data on coefficients and T statistics (or p-values) for a regression with around 5 independent variables.

I personally like dotcharts of standardized regression coefficients, possibly with standard error bars to denote uncertainty. Make sure to standardize coefficients (and SEs!) appropriately so they "mean" something to your non-quantitative audience: "As you see, an increase of 1 unit in Z is associated with an increase of 0.3 units in X."

In R (without standardization):

set.seed(1)
foo <- data.frame(X=rnorm(30),Y=rnorm(30),Z=rnorm(30))
model <- lm(X~Y+Z,foo)

coefs <- coefficients(model)
std.errs <- summary(model)$coefficients[,2]

dotchart(coefs,pch=19,xlim=range(c(coefs+std.errs,coefs-std.errs)))
lines(rbind(coefs+std.errs,coefs-std.errs,NA),rbind(1:3,1:3,NA))
abline(v=0,lty=2)


How to find False discovery rate of a ML model? Situation explained below Lets say:- ML Model denied 1000 transactions
FDR= FP/(FP+TP)
I randomly picked 400 accounts and checked if the model denied good user or bad user.
Good user denied- 110
Bad user denied - 290
How will I calculate False Discover rate of the model. Can it be based on the sample that I checked manually and then extrapolate ?

In your case, you can estimate the FDR by
$$110 / 400 = 27.5\%$$
the approach of sampling some data and estimating the empirical FDR in there is valid, it's the same as estimating the mean of a population using a sample, it makes sense, and the more data you use the more confident you're going to be on your estimate, as long as you're sampling randomly.
400 seems a pretty decent number to estimate FDR, so your estimate is going to be pretty good.


Solution for in Time/Space Complexity challenge in Recommendation System? I have a book Recommendation System project and have a huge data set of feature vectors. What is the best solution for in memory computation? I mean, the program should:

calculate the cosine Similarity among Database Books
be REAL TIME, because it's an interactive business site.

I've already thought about big data analysis tools and management such as Spark, Hadoop, ... . However, I am really new to Spark Techs and therefore not sure whether it's practical or not. I am really mixed up how can it be helpful? I have studied spark's documentation in java but it confused me more about how spark can be helpful?

Apache Spark is a great solution for such problems.
But, first let's be clear about the definition of real time processing. It's the type of processing that must guarantee response within specified time which on an interactive business site is actually very low. You can read about those kind of specifications in this answer.
Spark doesn't provide such luxury in predicting under 0.1 sec and I'm citing

Excerpt from Chapter 5 in my book Usability Engineering, from 1993:

0.1 second is about the limit for having the user feel that the system is reacting instantaneously, meaning that no special feedback is necessary except to display the result.

Having an interactive business site on which you'd want to display predictions doesn't mean that your predictions has to be in real time.
So the obvious is, actually, the following :
Q: Now that I have computed recommendations for my users, what should I do ?
A: Let's define a serving layer for our systems fast enough to query when recommendations are needed.
It can be anything fast enough to answer your calls e.g Elasticsearch, Solr, HBase, Redis. Whatever flavor suits you.
On other hands, well
Q: I don't want my system to be static, I need to recompute my predictions every T hours/days/etc
A: Spark can do a scheduled job perfectly here. (a simple cron would do)
Q: But when do I retrain my recommender system ?
A: I would say it actually depends on so many stuff a bit too broad to discuss here. You can read about the topic here if you wish.
Ok, so we defined now our batch layer.
Q: And what about data coming in real time, through Kafka, Rabbit, etc. ?
A: This is actually when it can get more complicated, because the method that you'll use to compute distances, approximations, new recommendations will depends on what type of recommender systems you are building and what technologies you are using.
Spark streaming can fit very well to apply "simple" computations on "window" based micro batches. This can be our speed layer.
To conclude, all of the above defines what is called a lambda architecture. And one of the best framework that follows this design is Oryx (personal opinion).
It's quite interesting, you ought taking a look at it.
I also believe that it's quite possible to have a RT set-up for a recommendation system without the speed layer.
I hope that this answers your question.


How should labeled data from multiple annotators be prepared for ML text classification? My specific question is how NLP data from multiple human annotators should be aggregated - though general advice related to the question title is appreciated. One critical step that I've seen in research is to assess inter-annotator agreement by Cohen's kappa or some other suitable metric; I've also found research reporting values for various datasets (e.g. here), which is helpful for baselining.

How many annotators should work on each data point depends on time, personnel, and data size requirements/constraints, among other factors (I may ask a followup question for how to find optimal n). However, once n annotators have finished a dataset, how should those n datasets be unified for a "ground truth"? A couple approaches that I have seen used or seem reasonable to me:

Take averages over all annotators. Classification problems are sometimes hard to restate as graduated ones, although that seems necessary if an average is to be taken.
Express some level of uncertainty in the data for controversial labels, or even omit them from training and evaluation.
Add an arbitration step to unify or discard controversial labels. I am not sure this would be worth the annotators' time.
Choose some "principal annotator(s)" (possibly determined by IAA scores) who get the final word in conflicts.

Guidance/references for the above and any other steps I can take to make a high quality dataset are much appreciated. I am mostly interested in efficiently removing individual annotator bias even when n is low.

Most machine learning algorithms are designed with complete trust in the labels. There is no standard way to model uncertainty in data labels. Thus, create a project-specific threshold for uncertainty to omit data or labelers. For example, a trusted classification data label would require n of m ensemble voting.

One major issue is re-labeling. Systems tend to evolve over time and the definition of labels is refined. Mature data labeling system have a notion of data lineage  - "Who labeled what data when with what criteria".

The book "Human-in-the-Loop Machine Learning" by Robert Munro goes into greater detail.


Negative Rewards and Activation Functions I have a question regarding appropriate activation functions with environments that have both positive and negative rewards.

In reinforcement learning, our output, I believe, should be the expected reward for all possible actions. Since some options have a negative reward, we would want an output range that includes negative numbers.

This would lead me to believe that the only appropriate activation functions would either be linear or tanh. However, I see any many RL papers the use of Relu.

So two questions:

If you do want to have both negative and positive outputs, are you limited to just tanh and linear?

Is it a better strategy (if possible) to scale rewards up so that they are all in the positive domain (i.e. instead of [-1,0,1], [0, 1, 2]) in order for the model to leverage alternative activation functions?

This would lead me to believe that the only appropriate activation functions would either be linear or tanh. However, I see any many RL papers the use of Relu.

Generally you want a linear output, unless you can guarantee scaling total possible reward to within a limited range such as $[-1,1]$ for $\text{tanh}$. Reminder this is not for estimating individual rewards but for total expected reward when following the policy you want to predict (typically the optimal policy eventually, but you will want the function to be able to estimate returns for other policies visited during optimisation).

Check carefully in the papers you mention, whether the activation function is applied in the output. If all rewards are positive, there should be no problem using ReLU for regression, and it may in fact help stabilise the network in one direction if the output is capped at a realistic minimum. However, you should not find in the literature a network with ReLU on output layer that needs to predict a negative return.

  If you do want to have both negative and positive outputs, are you limited to just tanh and linear?

There are likely others, but linear will be far the most common.

  Is it a better strategy (if possible) to scale rewards up so that they are all in the positive domain (i.e. instead of [-1,0,1], [0, 1, 2]) in order for the model to leverage alternative activation functions?

It may sometimes be worth considering scaling rewards by a factor, or normalising them, to limit gradients, so that learning is stable. This was used in the Atari-games-playing DQN network to help the same algorithm tackle multiple games with different ranges of scoring.

In continuous problems, the absolute value of reward is usually flexible, you are generally interested in getting the best mean reward per time step. So in that case you could scale so that minimum reward is 0, and use ReLU or other range limited transform in output - as above that might help with numeric stability.

In episodic problems without a fixed length, you typically don't have a such a free choice, because the agent is encouraged to end the episode quickly when rewards are negative. This is something you might want for instance if the goal is to complete a task as quickly or energy-efficiently as possible. A good example of this is "Mountain Car" - granting only positive rewards in that scenario would be counter-productive, although you might still get acceptable results with positive reward only at the end and discounting.

The general case is that rewards can be arbitrarily scaled and centred for continuous problems without changing the agent's goal meaningfully, but only arbitrarily scaled for episodic problems.


What is used for Machine Translation besides RNN I am doing a university report and it seems that encode-decode RNN are optimal for Machine Translation. I need something else to compare it to but I can't seem to make a proper google search for it.

Can you recommend what else can be used for Machine Translation that I can compare RNN to.

Recursive Neural Networks assume you have a relation between the current input and the previous input, if you organize data correctly CNNs can be used in place, some CNNs models have score higher than RNNs in BLEU metric. There is an article in Medium about it Why Convolutional Neural Networks are a Great Architecture for Machine Translation

Some implementation examples are Facebook's architecture proposed in the paper A novel approach to neural machine translation and this one from a Stanford's professor.

Also CNN's have been used by Google for language understanding and even in some fast-compact models or their new AI that talks on the phone.


How do I get started with machine learning and image recognition? I'd like to get started with machine learning, specifically image recognition. I know that Python is the most popular language for ML since it's easy to pick up and there's tons of libraries for it. I have a few questions:

What are the advantages of using a pre-trained model vs training my own?
Where would I find pre-trained models? At what point would it be worth it to train my own models, and how can I do so?
Which library should I be using as a beginner? TensorFlow, scikit-learn, etc.

What are the advantages of using a pre-trained model vs training my own?

The main advantage of pre-trained models is that it greatly reduces required data for your model. You can see it as a better way to initialize your model.

  Where would I find pre-trained models? At what point would it be worth it to train my own models, and how can I do so?

Keras has a variety of pretrained models for image classification. If your data consists of rare objects or uncommon situations, it might be better to train your own model, say for very specific task. Pretrained models only work if patterns in your data share similarities to patterns learned by the pretrained model.

  Which library should I be using as a beginner? TensorFlow, scikit-learn, etc.

Scikit-learn and Keras are pretty straightforward. They both have toy datasets for experimenting. Deep Learning with Python by François Chollet has a nice chapter on computer vision, which has a section on using pretrained models.


Applying a robust scaler to column 0 to 6 of my data frame? I have a dataframe X and would like to scale columns 0 to 6 using robust scaler, how do i achieve this please? the following code only returns the scaled section of the data frame without the remaining columns, but i want the whole columns returned with 0-6 scaled only.
from sklearn.preprocessing import RobustScaler

scaler=RobustScaler()
X=pd.DataFrame(scaler.fit_transform(X),columns([['Administrative',
'Informational','ProductRelated','BounceRates','PageValues','SpecialDay']]))

X

Check out the documentation for sklearn's columnTransformer. This allows you to apply transformations to specific column indices in your dataframe.
Note the 'passthrough' option for the transformer parameter - this will be needed for the columns that you do not wish to scale/modify.
Example taken from the documentation:
>>> import numpy as np
>>> from sklearn.compose import ColumnTransformer
>>> from sklearn.preprocessing import Normalizer
>>> ct = ColumnTransformer(
...     [("norm1", Normalizer(norm='l1'), [0, 1]),
...      ("norm2", Normalizer(norm='l1'), slice(2, 4))])
>>> X = np.array([[0., 1., 2., 2.],
...               [1., 1., 0., 1.]])
>>> # Normalizer scales each row of X to unit norm. A separate scaling
>>> # is applied for the two first and two last elements of each
>>> # row independently.
>>> ct.fit_transform(X)
array([[0. , 1. , 0.5, 0.5],
       [0.5, 0.5, 0. , 1. ]])


How to find the mean of a column relative to another column? I am working on the Boston house price prediction.  I have a column named GarageYrBlt that holds the year the garage was built for a specific house.  My assumption is that the garage would most likely be built at the same time as the house so I want to fill the missing value with the median of the column GarageYrBlt relative to the column YearBuilt.

To explain my idea further:  While I was working on the Titanic problem, I filled the missing Age column with the median relative to the Sex column.  So all female passengers with the missing age would get the value that is the median age of all female passengers.

This is what I did:

train['GarageYrBlt'] = train['GarageYrBlt']
     .fillna(train.groupby('YearBuilt')['GarageYrBlt']
     .transform("median"), inplace=True)

And when I do print(train['GarageYrBlt']), this is my output:

0       None
1       None
2       None
3       None
4       None
5       None
6       None
7       None
8       None
9       None
10      None
11      None
12      None
13      None
14      None
15      None
16      None
17      None
18      None
19      None
20      None
21      None
22      None
23      None
24      None
25      None
26      None
27      None
28      None
29      None
        ...
1430    None
1431    None
1432    None
1433    None
1434    None
1435    None
1436    None
1437    None
1438    None
1439    None
1440    None
1441    None
1442    None
1443    None
1444    None
1445    None
1446    None
1447    None
1448    None
1449    None
1450    None
1451    None
1452    None
1453    None
1454    None
1455    None
1456    None
1457    None
1458    None
1459    None
Name: GarageYrBlt, Length: 1460, dtype: object

You need to change inplace to False (which is the default).

Setting it to True changes the dataframe in place, so you don't have to assign the column again. Also, setting inplace=True returns None. So, either:

train['GarageYrBlt'] = train['GarageYrBlt']
     .fillna(train.groupby('YearBuilt')['GarageYrBlt']
     .transform("median"), inplace=False)

or

train['GarageYrBlt']
     .fillna(train.groupby('YearBuilt')['GarageYrBlt']
     .transform("median"), inplace=True)


Predicting equipment failure with time series alarm data I am trying to predict machine failures based on alarm data.
The situation:
There is approximately 4000 machine failures per year. These are labelled poorly (it is entered manually and can have multiple names for the same failure). This dataset consists of the timestamp, failure name and failure description in the following way.
timestamp, name, description
01/01/2020 - 08:10, Motor Failure, Motor failed due to overheating

Also there is approximately 1 million alarms per year showing possible issues (There are 8000 possible alarms in the system but only ~1200 makeup the 1 million that activate). Alarms fire at the point of failure, but also in a large percentage of cases there are multiple related alarms activating for weeks before hand that are unable to be acted upon due to the shear number. This dataset consists of the timestamp, alarm id, and alarm description like the following.
timestamp, alarm id, description
01/01/2020 - 08:10, MFHeatHiHi, Motor temperature critical

Please excuse my ignorance as I am new to data science. I am trying to work out the best way to clean up/modify the data first and then in which directions to go for creating the prediction. Please let me know if I am going about it the wrong way or have the incorrect idea on where to start.

Update the 4000 failures with the same name for same events (I estimate there are approximately 600 discrete failures within this). Or I call them all the same name "failure" and test the alarms against only one variable and use the alarm description to give me the issue that may cause the failure.

Break the alarms in to time windows proceeding the failure (possibly 30 days or less). Then use the alarm id - description as columns with the number of occurrences as the value.

For example:
Failure timestamp, MFHeatHiHi - Motor temperature critical, FanHeatHiHi - Fan temperature critical,
01/01/2020 - 08:10,                4,                                        3,

Then I was thinking of using random forest with R.
Is this the right way to go about it, should I be making failure just one variable, is removing the time component of the alarm and transforming to number of alarms the correct way to go about it and would that mean I need 8000 columns for every possible alarm.
I am pretty sure I have the wrong method so would really appreciate some guidance so I am heading in the right direction.
Thanks for your help,

I think what you are describing would be called anomaly detection. I suggest trying a different approach. There are several standard solutions to deal with this topic, here are a few.
Main problems to address are:

Setting a good threshold to balance false alarms with missed
events. Selecting the model will influence this setting (see below for 2 typical models)

Correct Labels/responses: If you have enough examples of alarms that are labelled correctly, you might look at clustering methods to determine the correct labels for un-labelled or mis-labelled examples. This will give you a larger dataset to use for training.

Selecting the model:
Model 1: Manual - Multivariate using the Covariance Matrix
Alarm settings to decrease false alarms:
You can use Euclidean (for spherical distributions), or better yet, Mahalanobis (for ellipsoidal distributions) distance and a n Sigma threshold from the centroid to determine what is the normal reading for the machine and then setting a cutoff for what is normal vs what is an anomaly.
Model 2: Neural Network - Autoencoder
Train on normal data (filter out alarm data) and test on new data looking at the probability distribution in the re-construction error in the model output (high MAE), and selecting a threshold based on the output

Online reference (may need account to access):
https://towardsdatascience.com/how-to-use-machine-learning-for-anomaly-detection-and-condition-monitoring-6742f82900d7


Lower-than-random ROC If I have an ROC for a single classifier [y(x) in the range 0...1] that is 'worse than random', namely the AUC of the ROC is less than 0.5,  would a classifier that reversed the class predictions [y'(x)=1-y(x)] be better than random to the same degree?

$y'(x)$ means you will work with $ROC' = ROC^{-1}$ (inverse of $ROC$), as all true positive will be falsely negative and vice versa. Therefore, $AUC' = 1 - AUC$ (As ROC is an increasing function and inside a unit square), and your answer is yes.


How to tune an weighted voting ensemble method? I am working on kidney cancer patients' data with 5 unbalanced labels. These codes are contained of Normalization, Oversampling on Feature Engineering part. A list of 9 ordinary  Machine Learning methods is provided which are used for the classification task. Then, I take advantage of two kinds of ensemble methods of hard voting and weighted voting methods. 10-fold CV has is exploited to validate results.
methods = ['Support Vector Machine', 'Logistic Regression', 'K Neighbors Classifier', 'Random Forest', \
          'Gaussian Naive Bayes', 'Linear Discriminant Analysis', 'Decision Tree', 'Gradient Boosting',\
          'soft_VotingClassifier','hard_VotingClassifier']

I would like to know how to tune weights for the soft voting method one?
Here, are the code and results I have right now:
from sklearn import preprocessing
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.ensemble import VotingClassifier

if method == 'soft_VotingClassifier':
    cl1 = LogisticRegression()
    cl2 = KNeighborsClassifier(n_neighbors=10)
    cl3 = RandomForestClassifier(max_depth=35)
    cl4 = GaussianNB()
    cl5 = LinearDiscriminantAnalysis()
    cl6 = DecisionTreeClassifier()
    cl7 = SVC(C=0.1, gamma=0.0001, kernel='poly')
    cl8 = GradientBoostingClassifier()
    estimator = [(method[0],cl1), (method[1],cl2), (method[2],cl3), (method[3],cl4),\
                 (method[4],cl5), (method[5],cl6), (method[6],cl7), (method[7],cl8)]
    eclf = VotingClassifier(estimators=estimator,
    voting='soft', weights=[5, 5, 10, 5, 6, 8, 4, 10])

if method == 'hard_VotingClassifier':
    cl1 = LogisticRegression()
    cl2 = KNeighborsClassifier(n_neighbors=10)
    cl3 = RandomForestClassifier(max_depth=35)
    cl4 = GaussianNB()
    cl5 = LinearDiscriminantAnalysis()
    cl6 = DecisionTreeClassifier()
    cl7 = SVC(kernel='linear',gamma='scale')
    cl8 = GradientBoostingClassifier()
    estimator = [(method[0],cl1), (method[1],cl2), (method[2],cl3), (method[3],cl4),\
                 (method[4],cl5), (method[5],cl6), (method[6],cl7), (method[7],cl8)]
    eclf = VotingClassifier(estimators=estimator, voting='hard')

Confusion Matrix results on test data:
[50  5  1  4  2]
[ 0 13  1  0  3]
[ 0  1  2  1  1]
[ 4  0  0  2  0]
[ 0  3  0  0  2]

Accuracy result on test data:

For those kind people who want to check my code or even run them, I gonna put the repository link.

The question is a classical optimization problem.
To make it more simple, lets say that you have 2 models (m1 and m2). And you want to do an ensemble.
First idea, is doing the mean of both
(m1.predict(X) + m2.predict(X))/2

Then you want to optimize the best combination of both models.
for w in [0,0.25,0.5,0.75,1]:
     (w * m1.predict(X) + (1-w) * predict(X) ) / 2
     print(evaluate results/MSE/Accuracy...)

Your problem is exactly the same as this one but using the voting classifier.
You just need to optimize the array. In a for loop.
weights=[5, 5, 10, 5, 6, 8, 4, 10]

In this case, your array is not normalized and it has more models. So it will take longer. Also voting classifier fits every time the model, so I will take you a long time. Is easier to code it yourself.


Most Efficient Post Processing with Python and Pandas This question is about best practices for working in Pandas dataframes.  Speed, ease of use, and memory consumed could all impact any answers you might have.  I start by pulling a data set into a dataframe like this:

Date    Location    Value
3/4/2018    1        4795
3/5/2018    1        4795
3/4/2018    2        5022
3/5/2018    2        5088
3/4/2018    3        100
3/5/2018    3        100
3/4/2018    4        117154
3/5/2018    4        117154

I would like to sum the Value based on some other criteria.  For this example, lets use two states, SD and ND.  Location 1,2,4 are in ND, and Location 3 is in SD.  As I see it, I have two options:

Have Pandas post process the location numbers.  IE Pythonicly: ND = Sum(Loc 1,2,4), SD = Sum(Loc 4).  Then build/pivot the time series based on state
Build a lookup table and have Pandas append the state to each row in the dataframe.  Then filter/group by state for the time series.  Lookup table would look like so:

Location    State
1           ND
1           ND
2           ND
2           ND
3           SD
3           SD
4           ND
4           ND

In option one, would Pandas add a new row to the dataframe for each day of the timeseries with the state totals?  Or would the output be a pivot table like structure with only the state totals.

If option two, what would be the best type of way to host the lookup table?  CSV?  JSON?  Table in SQL DB?

I'm concerned in option one changes would need to be made directly to the code.  Whereas in option two an addition to the lookup table could add the information required to add location to the correct group.

While I know this is open ended, I hope someone is willing to provide thoughts on efficient structure for this type of data flow.

I'd do it this way:

helper dictionary:

In [79]: d = {'SD':[3], 'ND':[1,2,4]}

let's convert it to a Pandas Series:

In [80]: lkp = pd.Series({el:key for key,lst in d.items() for el in lst})

In [81]: lkp
Out[81]:
1    ND
2    ND
3    SD
4    ND
dtype: object

now we can map Location into State and group by it:

In [82]: df.assign(Location=df['Location'].map(lkp)).groupby('Location')['Value'].sum()
Out[82]:
Location
ND    254008
SD       200
Name: Value, dtype: int64


Click Through Rate calculation (CTR) calculation problem So I'm doing a use case for a company interview and one of the questions is to calculate the CTR for a sorting algorithm.
My question would be: Should I remove the operations where there were no products shown (even though some of them there were clicks on products)?Example (should discard this columns?):
Number of impressions | Number of clicks
            0         |      1
            0         |      3
            0         |      0

CTR = #products clicked/#products shown
My other doubt is wether should I sum the clicks and the products shown and then divide one on other OR should I do the CTR for each operation:
Pseudo Python code (imagine that I have a dataframe with several operations as rows):
CTR = pd_df["no_clicks"].sum() / pd_df["no_impressions"].sum()

OR
pd_df["CTR"] = pd_df["no_clicks"] / pd_df["no_impressions"]
CTR = pd_df["CTR"].mean()

I have no knowledge at all about this kind of problem, but logically I would say:

Should I remove the operations where there were no products shown (even though some of them there were clicks on products)?Example (should discard this columns?):

Yes, because otherwise you could in theory end up with a rate higher than 1, this wouldn't make sense. However this raises the question of what exactly is represented: since some clicks can be counted when there is no product, it looks like the number of clicks is over-estimated in the data since the clicks are supposed to be on products.
Also note that it's the rows which would be removed in this case, not the columns.

My other doubt is wether should I sum the clicks and the products shown and then divide one on other OR should I do the CTR for each operation:

This issue means that either the data doesn't have the proper format to apply the definition (strictly speaking the data should have one row for every product shown, and the number of clicks should be either 0 or 1), or that the instructions for applying the CTR formula with this specific data are under-specified. I would say that the first option is closer to the original formula since it doesn't require to do a mean over individual rates.


Conditionally replace dataframe cells with value from another cell I have a couple pandas dataframe questions. I would like to replace the values in only certain cells (based on a boolean condition) with a value identified from another cell. I have defined the dataframe from an imported text file, which returns a dataframe with column headers 'P' and 'F' and values in all of the cells. I want to replace only those cells in column 'F' where the corresponding 'P' entry in the same row is less than a defined variable, Plithos. The new value for all of the replaced cells is defined as Fmax, which is the value of 'F' when 'P' in the same row == Plithos:

Plithos = 5.0

Fmax = df.loc[(df['P']==Plithos),'F']

The above part seems to work. The Fmax value returned is the correct one from the table. But when I try to replace the appropriate values using the code below, the right cells are identified but the new entries are all NaN:

df.loc[(df['P'] < Plithos),'F'] = Fmax

I'm not sure why this is happening, or what I should change in the syntax to fix it? Any help is appreciated!

Second question: Ideally, I would also like to define the condition for Fmax to be the value found in the row where P is equal to or less than Plithos, but the closest possible (so, the Price is Right match, I guess?). That way if the user input spreadsheet doesn't have a perfect matching value for Plithos in their table, the code still works. I assume pandas doesn't have a Price is Right boolean built in, though...

Aha! For some reason, when I named a variable that pointed to a value in a dataframe, and then tried to use the variable to put it back into the table replacing other cells, it changed the dtype. That's why I was getting NaNs. I just had to convert it to a usable type using tolist.

As for the nearest-value identification, this seems to have worked:

Pfinal = df.iloc[(df['P']-Plithos).abs().idxmin()]

F_max = Pfinal[1]

F_max = F_max.tolist()

df.loc[(df['P'] < Plithos),['F']] = F_max


Replace a value in pytorch tensor t=tensor([0.1 0.2 0.3 0.4 0.5 0.6])

now i need to modify this existing tensor t as follows:
t=tensor([0.1 0.2 0.7 0.4 0.8 0.6])

I tried as follows:
t=tensor([0.1 0.2 0.3 0.4 0.5 0.6])
a=tensor([0.1 0.2 0.7 0.4 0.8 0.6])
index=range(len(a))
t.index_copy_(0,index,a)

But still it is not updating
how can i modify the tensor in pytorch?

Try this:
t = tensor([0.1, 0.2, 0.3, 0.4, 0.5, 0.6])
t.index_copy_(0, tensor([2, 4]), tensor([0.7, 0.8]))

Reference:
torch.Tensor.index_copy_


Any mini-batch implementation of Random Forest? I'm using the scikit-learn's Random Forest to perform some classification task but run out of memory because of the amount of data.

Is there any mini-batch implementation of the Random Forest algorithm (or similar decision trees-based method)?

Check out the CART algorithm. This is essentially a bootstrapping method with subsampling that I am sure you can generalize for batch processing.


Where can I find dataset for word analogy task? In the paper of Word2Vec by Thomas Mikolov and others, there is a accuracy report on the full Semantic-Syntactic data set. Where I can find this dataset or a related dataset for word analogy task?

Here is the paper link and I'm referring to table 5.
https://arxiv.org/abs/1301.3781

If i understand correctly you want a long text file of some sort that can easily be analysed. I would suggest that you use project Gutenberg which publishes thousands of free to use books in plain UTF-8 text format. For example here is Pride and Prejudice by Jane Austin.


Calculate Average Error Rate in KNN I have been given different error rates for k=1,2,3.

How to calculate the average value of error,and determine whether it is a good error rate or not?

Based on the explanation you gave I think you are talking about k-fold Cross validation.

Assuming it as k-fold cross validation then you need to take mean of all the error rates. And the mean outcome would be your final error rate.


Running an R script programmatically I have an R script that generates a report based on the current contents of a database. This database is constantly in flux with records being added/deleted many times each day. How can I ask my computer to run this every night at 4 am so that I have an up to date report waiting for me in the morning? Or perhaps I want it to re-run once a certain number of new records have been added to the database. How might I go about automating this? I should mention I'm on Windows, but I could easily put this script on my Linux machine if that would simplify the process.

For windows, use the task scheduler to set the task to run for example daily at 4:00 AM

It gives you many other options regarding frequency etc.
http://en.wikipedia.org/wiki/Windows_Task_Scheduler


Is "adding the predictions to the real data for new training and prediction" a good idea for LSTM? Considering we have trained our model with a lot of data for "many-to-one" prediction. Then we like to forecast the future data of next 10 days. So we use last 60 of existent data and predict the single next day. From here there are 2 approaches:

We can put our model.predict() function in a for loop for 10 times and do predictions like this(adding our predictions to end of our real data).
We can put all of our model(consisting training part, not just predict part), in a for loop and this means we train our model 10 times whenever we do a new prediction and adding it to our real data.

EDIT:
Thinking you have X_train = (100,60,1) array that means 100 examples, 60 time-steps(hidden units) and 1 feature for each example. Also you have y_train array of size (100,1,1) that means 100 labels with time-steps = 1 and 1 feature. Then you train your network to read 60 of inputs and predict the next single output. Also you create a X_test array like this: X_test = X_train[len(X_train - 60):] that means you use last 60 numbers of your series to predict the next number.  So you use the new_number = model.predict(X_test) for that and you predict the time-step 61 that is not a real number. It's your prediction. Then you want to continue your predictions. So what do you do is adding the 61'th predicted number to the last of your X_test = np.append(X_test, new_number) and do new number = model.predict(X_test) again. But the difference is that the last number in your new X_test is your previous prediction. And you keep this way for 10 times to predict 10 next numbers. (This was the first approach).

The other approach(2) has a difference. After doingnew_number = model.predict(X_test) for the first time, you add the predicted number to x_train instead of X_test, like this X_train = np.append(X_train, new_number) and train your model again model.fit(X_train , y_train) with the new predicted number. Then you use new number = model.predict(X_test) and again adding predicted number into the X_train, then train your model again(this time, with 2 new predicted numbers that you have added to the end of your X_train) and so on for 10 times!

Are you saying you would like to predict 10 days ahead in this instance?

If this is the case, the LSTM model is able to do this and iterating in the way you are suggesting is unnecessary and could give you unreliable results.

For instance, consider a dataset whereby we are attempting to predict one-step ahead:

# Training and Test data partition
train_size = int(len(dataset) * 0.8)
test_size = len(dataset) - train_size
train, test = dataset[0:train_size,:], dataset[train_size:len(dataset),:]

# reshape into X=t and Y=t+1
previous = 1
X_train, Y_train = create_dataset(train, previous)
X_test, Y_test = create_dataset(test, previous)

# reshape input to be [samples, time steps, features]
X_train = np.reshape(X_train, (X_train.shape[0], 1, X_train.shape[1]))
X_test = np.reshape(X_test, (X_test.shape[0], 1, X_test.shape[1]))

# Generate LSTM network
model = Sequential()
model.add(LSTM(4, input_shape=(1, previous)))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')
model.fit(X_train, Y_train, epochs=100, batch_size=1, verbose=2)

# Generate predictions
trainpred = model.predict(X_train)
testpred = model.predict(X_test)

# Convert predictions back to normal values
trainpred = scaler.inverse_transform(trainpred)
Y_train = scaler.inverse_transform([Y_train])
testpred = scaler.inverse_transform(testpred)
Y_test = scaler.inverse_transform([Y_test])

# calculate RMSE
trainScore = math.sqrt(mean_squared_error(Y_train[0], trainpred[:,0]))
print('Train Score: %.2f RMSE' % (trainScore))
testScore = math.sqrt(mean_squared_error(Y_test[0], testpred[:,0]))
print('Test Score: %.2f RMSE' % (testScore))

In this batch of code, you can see that we have set the previous parameter equal to 1, meaning that the time step being considered by the model is t-1.

In this particular instance, here are the training and test predictions compared to the actual series:

Now, the same model is run, but this time the previous parameter is set to 10. In other words, the previous 10 days are being considered as one time step, and the model is forecasting for time t+10 in this instance. Here is another sample prediction comparing the test set with the actual. A fuller example of this is provided here:

In this regard, my advice would be to define the time series you wish to forecast and then work off that basis. Using iterations only complicates the situation, and could even cause issues with prediction.


Creating a Feature to determine popularity I am building a recommendation system where I have multiple categories. I would like to  Know how popular a product is in each category. For that, I am considering probability as one factor. For example, I have 3 categories ($C1$,$C2$,$C3$) so, I am calculating:
(Number of times a particular item in C1 is purchased)/total number of items purchased from C1)
This gives me the probability of each item within a category. Apart from that, I am also trying to consider:
(Total number of times an item has been purchased/total number of times it has been viewed)
However, I do not understand how to use the purchase/view ratio with probability. Apart from that, what another factors can I conclude?

You need to put your shoes into business people and communicate with people from that department on regular basis, trying to understand what the business's needs are. In e-commerce your second metric is Conversion Rate and is defined as:

CR = Orders/Views*

Please note there that there are two variations of Views. One that comes from Product List Page (aka PLP) that is mostly know as Impression, or the one comes from Product Detail Page(aka PDP) is where the user actually lands on the product detail page. The latter is more common in the CR calculation.

Your first one i.e. the popularity, that somehow I liked, can be used. Only be careful that it is going to be affected by number of items available in that category.

In general, I find it useful to think about edge cases when coming up with new measures and to see whether the resultant is desirable. What do I mean by edge cases, let's take your popularity score, by your definition, for example. Imagine the following two scenarios:

popularity item 1 = 1/2 (very limited items in this category where item 1 is)
popularity item 2 = 1000/2000 (large number of items in this category where item 2 is)

What you see here looks quite normal by the original definition from mathematics perspective. However, what it means from recommendation/business perspective that you have similar scores from two items from two different categories. Here the business may jump in and say looks we have a lot of more assortments in the category where item 2 is and often items in this category are in the warehouse or have faster delivery etc. etc., then we prefer our products from there to be recommended first. This is the point you realize that while results seem reasonable, at least according to the popularity definition, falls short on certain aspects.

Other ideas: I noticed you are leaning towards using multiple scores. That is a great idea. Having multiple factors may catches these corners and address multiple business needs at the same time. Like having both CR and popularity. Or may think of each item's profit contribution to the business and take that into consideration. Imagine you have a score in place that gives equal chances of recommending an iPhone or Wireless Mouse, however iPhone contribute way more to the business (it is more profitable), then you may want to push for that factor too. You may also like to look at Impression I mentioned if it is available from your tracking. Item's rating, delivery, Add_To_Cart, Wish_List, or Time_Spent_on_Item are other things to think about. Check this post to get some other ideas how things are roughly done in Amazon. Hope these help.


Chossing between regression models This is the first time I attempt to use machine learning with Keras. In contrast to others I need to use one of the disadvantage of such algorithms.
I need a function that accepts an angle and distance to an object and output a new angle and power (imagine aiming for an object with a bow for example and the algorithm tells me how far up my arm should go and the bow's power).
There's nothing predictive in this configuration. I will generate a large set of 4D (input,output) data with every possible case. I want the AI to "evaluate" some inputs and return the corresponding outputs for that set of inputs, in other words to remember the data and output the same numbers.
I need an AI for this task because I need smooth values between inputs values it has never seen (limited interpolation)
I have used two models:
model = Sequential()
model.add(Dense(12, input_dim=2, activation='relu'))
model.add(Dense(24, activation='relu'))
model.add(Dense(24, activation='relu'))
model.add(Dense(24, activation='sigmoid'))
model.add(Dense(2, activation='linear'))

Which now I know is incorrect because Sigmoid is used for binary classification. Still, it works! I end up with a mse of 4. I did not manage the get the same loss with all-ReLu layers with the same # of epochs.
model = Sequential()
model.add(Dense(12, input_dim=2, activation='relu'))

model.add(Dense(24, activation='linear'))
model.add(LeakyReLU(alpha=0.1))

model.add(Dense(24, activation='linear'))
model.add(LeakyReLU(alpha=0.1))

model.add(Dense(2, activation='linear'))

This models has a loss of 5.43 after 500 epochs and seems to stabillize here.
Note:

I am forced to retrain the model because I generate the training data very fast. I need to stay with a model that will keep going reduce its loss.
Normalization is worse than anything I have seen.
Because my model should be sensitive to different very close inputs, I tested with batch values from 5 to 2.
My dataset is currently 1100 of lines
This model will be trained very closely to the data I give it. It should not matter how many lines I feed it because I don't want prediction or generalization. I want the AI to output what it has seen for a corresponding set of inputs. That would mean overfitting the AI to the maximum until it reaches a very low loss. Then I can test it for exactly those values it was trained on.

Should I continue with the first model? Does it make sense to use the Sigmoid layer? How can the second model be improved?
Sample of my data:
theta[-90,90], distance [0,40],  theta_output[-90,90] power[0,1,2]
0.0,8.696802,0.25688815116882324,1
-1.990075945854187,5.455038,11.56562614440918,1
-56.81309127807617,3.1364963,-53.07550048828125,1
-38.21211242675781,4.718147,-32.30286407470703,1
-33.828956604003906,5.163292,-35.61191940307617,0
-27.64937973022461,6.182574,-25.107540130615234,1
2.8613548278808594,13.922726,-2.3708770275115967,2
-8.812483787536621,14.951225,-3.919188976287842,2
0.0,21.448895,-3.9320743083953857,2

The suggestion in the comment is appropriate.
Still, if you want to try NN, you may try these suggestions -
 - None of the models seems best as per general guidlines
 - Keep ReLu as all the hidden layer, linear for last layer(Regression)
 - Standardization/Normalization must be done before training
 - Add Batch normalization layer
Can also try
 Since your output seems like Classes [0,1,2]. Try a Classification model for power and a Regression model for output angle
 For classification - last activation - Softmax, Loss - categorical_cross_entropy, Label - one-hot encoded


Unsupervised Anomaly Detection on system metrics like memory, cpu, io, net, etc In all the examples that I can see online, people have used a labelled dataset. I however am stuck trying to construct a model to perform anomaly detection on unlabelled dataset (unsupervised anomaly detection).

  Problem Statement: To separate the anomalous points from the normal ones. The features available with us are mem_usage, mem_total, cpu_usage_idle, cpu_usage_iowait, etc

Using the anomaly detection, I plan to label the dataset saying whether anomalous or not. A step further would be aggregate each row based on the server the data belongs to and find if the server is healthy, problematic. Even further would be predicting when a server might face a similar problem and thus suggest ways to mitigate the problem.

Any kind of resources (papers, code blogs, videos) are really welcome. I am just starting out and am open to trying out new stuff as well.

Do you have sequence data about these metrics? E.g. do they evolve over time and you want to locate anomalies in these sequences?

If so, you can have a look at the tools described here:

  https://github.com/rob-med/awesome-TS-anomaly-detection

They are all unsupervised methods, so you don't need anomaly-free data for training.

If you are not dealing with time-series, there are also many unsupervised approaches you can take. This paper offers a good survey on the topic

  https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0152173


Hyper parameters and ValidationSet Please correct me if I am wrong. "Training Set is used for calculating parameters of a machine learning model, Validation data is used for calculating hyperparameters of the same model (we use same weights with different hyperparameters), and Test set is used for evaluating our model". If true, can someone explain the whole process in a little more detail. TIA.

Not completely true. In validation set, we find the best hyperparameters, but not with the same parameters of the model. That is, for every value of the hyperparameters we run the training process and obtain the loss for that set of hyperparameters, and we select the hyperparameters values with the lowest validation loss.

Edit

Ridge regression example: Ridge regression depends on one parameter, $\lambda$, that penalizes your model weights. Ridge regression has a single hyperparameter. As ridge regression is a linear model, it has also some weights $\theta_0, \dots, \theta_n$ (parameters) to train. The way to proceed to choose $\lambda$ and $\theta_0, \dots, \theta_n$ is the following:

For each value of $\lambda$ in a grid, we train by gradient descent the ridge regression model (thus optimizing the training loss) and obtain a set of parameters $\theta_0^*, \dots, \theta_n^*$. We measure the loss of the trained ridge model in the validation set.
We choose the $\lambda$ that minimizmes the loss in the validation set and the parameters $\theta_0^*, \dots, \theta_n^*$. If we wish, and we don't have much data, we can retrain the ridge model with the optimal $\lambda$ in the concatenation of the training and validation sets, obtaining some parameters $\hat\theta_0, \dots, \hat\theta_n$.
We compute the test error with the parameters obtained in the last step and report that as the most realistic value of the error of the model.


RL Invertible Value Function approach - why it prevents rewards from exploding? Authors of "Recurrent Experience Replay in Destributed RL", page 3, use the function $h$ to prevent rewards from exploding:

$$h(x) = \operatorname{sign}(x)(\sqrt{|x|+1} -1) + \epsilon x$$
where $\epsilon$ is a very small number, for example $0.0001$ and $x$ is the score (aka reward). h re-scales the rewards the network receives.

This is different to a more common "reward-clipping" technique.

Question 1 (most important):

The Q-vaue formula looks like this:

$$y = h \bigg( nstepReward + \gamma^n \cdot h^{-1}(Q(s_{t+n}, a^*)) \bigg)$$

Why do authors pass the Q-scores through the inverse (exponential) function and then through the squashing function $h$? The effect of this is that Q-target-scores remain unchanged, because composition of a function and its inverse is identity. So to me, the $Q$ is still a big source of threat of "exploding rewards", because it's left untouched. Why do we keep it untouched?

It looks like the whole formula merely squashes $nStepReward$, but why not just squash everything, including Q to prevent exploding rewards? In other words, why not to have:

$$y = h \bigg( nstepReward + \gamma^n Q(s_{t+n}, a^*) \bigg)$$

Question 2:

Why to use $\epsilon x$ in the formula for $h$

The answer of your first question can be answered by considering when you do not step forward and $n = 0$. So what happend in your case? $y = h(Q(S_t, a^*))$. However, $y= Q(S_t, a^*)$ could be more reasonable. Hence, we need that $h^{-1}$ to have a more explainable reward in the edge case.

One answer to the second question could back to the differentiation of the $h$ function. If the $\epsilon x$ is removed, the $\frac{dh(x)}{dx}$ goes to zero when $x$ grows. But, by the presence of $\epsilon$, we will have $\frac{dh(x)}{dx} \to \epsilon$ for big enough values of $x$. Hence, the reward function will not cap for the big values of $x$ and grows increasingly.


Modelling promotions for demand forecasting I am trying to develop a model to predict future demand for a product. Now, there are always some promotional events that affect the sales. I am trying to solve this problem using dummy variables. Here's how:

Supposing that the firm runs 7 promotional events on a particular product. So, I construct 7 dummy variables, that are boolean. For supposing that a for a particular week, promotion 3 was running. So, my training data-point becomes [0,0,1,0,0,0,0] and the corresponding sales. I construct a linear regression model for promotions in this way.

Now, here is my problem. When we model seasonality using this method, we construct a base linear model, after deseasonalising the data, and  then use the two models to predict the final output. In case of promotions, how do I 'depromotionalise' the data?

Any tips in solving the problem are appreciated. Thanks!

To create dummy variables for days promotion holidays, you might find this example useful.

If you are sure you do not have an out-of-stock problem anytime in the history you could use an autoregressive model to predict future sales (demand) for any product that has historical data. Depending on your data you could choose a model. Following code is an example which combines four different models by giving different weight to different models. This type of models capture seasonality and trends of your data. For more details about the models please check Rob Hyndman's forecast package documentation.

choose_model<-function(x,h,reg,new_reg,end_train,start_test){
      library(forecast)
      library(tidyverse)

  #train data

  x_train <- window(x, end = end_train )

  x_test <- window(x, start = start_test)

  #train and test for regressors

  reg_train <- window(reg, end = end_train )

  reg_test <- window(reg, start = start_test)

  h1=length(x_test)

  #model1

  stlf(x_train , method="arima",s.window= nrow(x_train),xreg = reg_train, newxreg = reg_test, h=h1)-> fc_stlf_xreg

  #model2
  auto.arima(x_train, stepwise = FALSE, approximation = FALSE,xreg=reg_train)%>%forecast(h=h1,xreg=reg_test) -> fc_arima_xreg

  #model3
  set.seed(12345)#for nnetar model
  nnetar(x_train, MaxNWts=nrow(x), xreg=reg_train)%>%forecast(h=h1, xreg=reg_test) -> fc_nnetar_xreg

  #model4
  stlf(x_train , method= "ets",s.window= 12, h=h1)-> fc_stlf_ets

  #Combination

  mod1 <- lm(x_test ~ 0 + fc_stlf_xreg$mean + fc_arima_xreg$mean + fc_nnetar_xreg$mean + fc_stlf_ets$mean)
  mod2 <- lm(x_test/I(sum(coef(mod1))) ~ 0 + fc_stlf_xreg$mean + fc_arima_xreg$mean + fc_nnetar_xreg$mean + fc_stlf_ets$mean)

  #model1

  stlf(x, method="arima",s.window= 12,xreg=reg, newxreg=new_reg, h=h)-> fc_stlf

  #model2
  auto.arima(x, stepwise = FALSE, approximation = FALSE,xreg=reg)%>%forecast(h=h,xreg=new_reg) -> fc_arima

  #model3
  set.seed(12345)#for nnetar model
  nnetar(x, MaxNWts=nrow(x), xreg=reg)%>%forecast(h=h, xreg=new_reg) -> fc_nnetar

  #model4
  stlf(x , method= "ets",s.window= 12, h=h)-> fc_stlf_e

  #Combination

  Combi <- (mod2$coefficients[[1]]*fc_stlf$mean + mod2$coefficients[[2]]*fc_arima$mean +
              mod2$coefficients[[3]]*fc_nnetar$mean + mod2$coefficients[[4]]*fc_stlf_e$mean)

  return(Combi)
}

The usage of the function:

coose_model(x,h,reg,new_reg,c(2018,02),c(2018,3))

$x$ is a time series

$h$ is time horizon to predict

$reg$ is the historical promotions, dummy date variables, holidays...

$new_ reg$ is the promotions, dummy date variables, holidays that are that you know it is going to happen

If you know that there is out-of-stock problem then take a look to this paper.


Machine learning techniques for estimating users' age based on Facebook sites they like I have a database from my Facebook application and I am trying to use machine learning to estimate users' age based on what Facebook sites they like.

There are three crucial characteristics of my database:

the age distribution in my training set (12k of users in sum) is skewed towards younger users (i.e. I have 1157 users aged 27, and 23 users aged 65);
many sites have no more than 5 likers (I filtered out the FB sites with less than 5 likers).
there's many more features than samples.

So, my questions are: what strategy would you suggest to prepare the data for further analysis? Should I perform some sort of dimensionality reduction? Which ML method would be most appropriate to use in this case?

I mainly use Python, so Python-specific hints would be greatly appreciated.

One thing to start off with would be k-NN.  The idea here is that you have a user/item matrix and for some of the users you have a reported age.  The age for a person in the user item matrix might be well determined by something like the mean or median age of some nearest neighbors in the item space.

So you have each user expressed as a vector in item space, find the k nearest neighbors and assign the vector in question some summary stat of the nearest neighbor ages.  You can choose k on a distance cutoff or more realistically by iteratively assigning ages to a train hold out and choosing the k that minimizes the error in that assignment.

If the dimensionality is a problem you can easily perform reduction in this setup by single value decomposition choosing the m vectors that capture the most variance across the group.

In all cases since each feature is binary it seems that cosine similarity would be your go to distance metric.

I need to think a bit more about other approaches (regression, rf, etc...) given the narrow focus of your feature space (all variants of the same action, liking) I think the user/item approach might be the best.

One note of caution, if the ages you have for train are self reported you might need to correct some of them.  People on facebook tend to report ages in the decade they were born.  Plot a histogram of the birth dates (derived from ages) and see if you have spikes at decades like 70s, 80s, 90s.


Fitting glm without explicit declaration of each covariate When I fit a linear model with many predictor variables, I can avoid writing all of them by using . as follows:

model = lm(target_deathrate~., data = full_data)

But for models with higher complexity, I cannot make this work:

x = glm(target_deathrate~poly(., i),data = full_data)

In these cases I have to write all variables.

How to avoid writing all variable names and include all variables in my model?

This has already been answered in stackoverflow here and here.

The main idea is to create the formula from a string with as.formula:

xnam <- paste("x", 1:25, sep="")
fmla <- as.formula(paste("y ~ ", paste(xnam, collapse= "+")))


Need help in selecting a model for a classification problem with binary data I have a clinical data set. The goal is to develop a model that predicts race based on binary data, whether or not a gene is present.

I am struggling to find a classification model that works without continuous data. One model that might work is k-modes. Are there any other models that I should consider?

Logistic regression models can certainly be used with dichotomous features. It also provides coefficient estimates for each feature so that relationships between features and the target labels can be tested and interpreted. Predictions made from logistic regression models are probabilities rather than binary decisions which can be helpful if you have targets for Type II Error rates, False Omission rates, etc. which is often the case in clinical data.

Distance-based methods can also be used such as the k-nearest neighbor algorithm. With all binary features, it would make sense to use distance measures designed for dichotomous data, such as the Russell Rao distance metric. These models will make predictions based on the class labels of the k-nearest observations in the feature space.


On simple 1D dataset, LogisticRegressionCV selects terrible hyperparameters, resulting scores are nonsensical I am trying to use LogisticRegressionCV to fit a logistic regression model to a simple 1D dataset. Very oddly, when given a choice, it seems to select a tiny C value, which forces my model to select a tiny theta resulting in a useless model.
I tried looking at the scores_ provided by the model, but they make no sense. For example, when I tell it to do 3-fold cross validation with 5 choices of C values, it gives me:
{1: array([[0.47058824, 0.47058824, 0.47058824, 0.47058824, 0.47058824],
        [1.        , 1.        , 1.        , 1.        , 1.        ],
        [0.63636364, 0.63636364, 0.63636364, 0.63636364, 0.63636364]])}

The dataset is not linearly separable, and yet it claims to be getting 100% accuracy regardless of which C values I've given it to try.
Example code below:
import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegressionCV
from sklearn.linear_model import LogisticRegression

def gen_y(x):
    p1 = np.clip(x + 0.5, 0, 1)
    v = np.random.uniform(0, 1)
    if v < p1:
        return 1
    return 0

np.random.seed(6)
x_data = np.sort(np.random.normal(0, 0.3, 100))
y_data = np.array([gen_y(x) for x in x_data])

regularized_logistic_regression_model = LogisticRegressionCV(Cs = np.array([10**-8, 10**-4, 1, 10**4, 10**8]), fit_intercept = False, cv = 3)

regularized_logistic_regression_model.fit(x_data.reshape(-1, 1), y_data)

print(regularized_logistic_regression_model.C_) # yields 10^-8
print(regularized_logistic_regression_model.coef_) # yields incredibly tiny value
print(regularized_logistic_regression_model.scores_) # yields nonsensical scores

First I did a visualization of what your code is doing (see code at the bottom)

The model seems completely fine. The coefficient of the linear regression is close to 0, where it should be given how you have created the data.

You are misunderstanding

  regularized_logistic_regression_model.scores_

    {1: array([[0.47058824, 0.47058824, 0.47058824, 0.47058824, 0.47058824],
            [1.        , 1.        , 1.        , 1.        , 1.        ],
            [0.63636364, 0.63636364, 0.63636364, 0.63636364, 0.63636364]])}

I quote from sklearn documentation:

  scores_dict: dict with classes as the keys, and the values as the grid of scores obtained during cross-validating each fold

This is the prediction of one of the folds (note that if you increase the cv parameter it changes length)

The data created is sorted! If you plot the values of X vs the Index you can see

You are getting those results because the data is sorted! Not just because of luck. If you shuffle the data you won´t have perfect predictions.

Your result is around 0.7 that is actually what makes sense by looking at the image I attached.

I shuffled the data, not in the most elegant way but you get now different results.

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
from sklearn.linear_model import LogisticRegressionCV
from sklearn.linear_model import LogisticRegression
plt.style.use('seaborn-whitegrid')

def gen_y(x):
    p1 = np.clip(x + 0.5, 0, 1)
    v = np.random.uniform(0, 1)
    if v < p1:
        return 1
    return 0

np.random.seed(6)

x_data = np.sort(np.random.normal(0, 0.3, 100))
y_data = np.array([gen_y(x) for x in x_data])

df = pd.DataFrame(x_data.reshape(-1, 1),columns=['X'])
df['y']= y_data

plt.figure()
plt.title('Values of X vs Index')
df.X.plot()
plt.savefig('x')
plt.show()

df = df.sample(frac=1)

regularized_logistic_regression_model = LogisticRegressionCV( fit_intercept = False, cv = 3)

regularized_logistic_regression_model.fit(df['X'].values.reshape(-1, 1) , df['y'])

plt.figure()

df['results'] =regularized_logistic_regression_model.predict(df['X'].values.reshape(-1, 1))
plt.scatter(df[df['results']==0].y,df[df['results']==0].X,label='pred=0')
plt.scatter(df[df['results']==1].y,df[df['results']==1].X,label='pred=1')
plt.legend()
plt.hlines(y=regularized_logistic_regression_model.coef_.squeeze(),xmin=-0.1,xmax=1.1)
plt.savefig('ex')
plt.show()

regularized_logistic_regression_model.scores_


What is the best approach for specified optical character recognition? I have a quite understandable request of extracting information (invoice number, invoice data, due date, total etc.) from scanned invoices (the digital format is image, not PDF), preferably in Python. The good thing is that the necessary information is more or less certain to exist on the page, and the (regexp-like) textual format of these is also tend to be consequent. The downside on the other hand is that the layout of the invoices are very diverse.

I have played with the following possible approaches:

Use character recognition to extract pure text and later try to
puzzle with the fragments. This method has some considerable
problems: the quality of the OCR (at least of the implemented one in
tesseract library) are quite mediocre and the output is hopelessly
unstructured (practically a big pile of words), it is very difficult
to come out any regexp or other rule even for regular phrases.
My other approach would be to apply some kind of deep learning either to
the raw image itself or the text pile where we leave the heavy
lifting to the network, but in this case I'm not sure what is
supposed to be the output? Is it some kind of a sequence to sequence
mapping?

Very unusual task, indeed.

Optical character recognition is a well-studied problem with many possible solutions (ressources). CNNs have proven to work extremely well even for hand-written character recognition. Take a look at this two papers:

Backpropagation Applied to Handwritten zip code
Comparaison of Classifier Methods: A Case Study in Handwritten Digit Recognition

Here is a beginner tutorial to do just that with Tensorflow.
If you need extra data to train your model, take a look at the MNIST dataset.


Definition of "inside" in K-means? After conducting a cluster analysis using K-means, I have new data coming online that I need to detect anomalies with.  Anomalies are assumed to not be within the clusters.

So, how is one to define "inside a cluster" in K-means?

Strictly speaking, the k-means algorithm does not have a definition for "inside the cluster" and is therefore not a great candidate for anomaly detection.  In k-means, every point is assigned to one of k clusters and then a new cluster centroid is calculated.

But as previous uses have pointed out, you could construct some sort of ad-hoc system where you process a set of data and then define new data as anomalous when its extends beyond 2 standard deviations of the centroid location. DON'T DO THIS!

K-Means will not work well for this ad-hoc method.  If k is poorly chosen, then the distribution within a cluster will not be normally distributed.  You very, very frequently see natural distributions of points which are split between two clusters.  For instance, take a look at the ad-hoc segmentation of points in this location data for a cell phone user's location over a month:

I suggest you use another clustering method.  The first option that comes to mind is DBSCAN.  This allows one to set a threshold for noise and the cluster numbers are not set a-priori.  DBSCAN is therefor much more likely to return normal distributions within a cluster.  Here is a single DBSCAN cluster of the same data:

Finally, I'll point out that the method you are proposing is not as good as other novelty and anomaly detection methods.  You should consider doing novelty detection using a single (or possibly even multiple) class support vector machine (SVM) with a nonlinear kernel. The nonlinear kernel will allow you to recover multiple "clusters" while the SVM will do much better at predicting which points are inside the class.


stem() output order in R In R, stem() produces the graph with "stems" in ascending order. Is there a code for obtaining them in descending order?

Example:

stem (cars$dist)

The decimal point is 1 digit(s) to the right of the |:

   0 | 14004678
   2 | 002466668822244466
   4 | 002668024466
   6 | 046806
   8 | 04523
  10 |
  12 | 0

The desired output:

  12 | 0
  10 |
   8 | 04523
   6 | 046806
   4 | 002668024466
   2 | 002466668822244466
   0 | 14004678

Try to capture output, and modify it in a desired way:

rev.stem <- function(data){
  out <- capture.output(stem(data))
  out <- c(out[1:3], rev(out[4:length(out)]))
  cat(out, sep="\n")
}

rev.stem(cars$dist)


Can we use both validation loss and cross validation at the on CNN? I have seen that validation loss is used for avoiding overfitting of the training set and cross-validation is used for generalized the models' results.

Are they use for similar purposes or results? If not, then how can I use both validation loss and cross-validation combined on CNN?

Yes, you are right validation loss is used to show when an ML model is overfitting it's training data. However this only occurs when your training data doesn't fit your model, especially when it's quite low. cross validation is iterating your validation data over your training data because there is no sufficient data to account for both training, validation and test data.
Thus its possible to used both of them to evaluate and improve your models performance although they are different concepts.


Determine document novelty/similarity with the aid of Latent Dirichlet allocation (LDA) or Named Entities Given an index or database with a lot of (short) documents (~ 1 million), I am trying to do some kind of novelty detection for each newly incoming document.

I know that I have to compute the similarity of the new document with each document in the index. If the similarity is below a certain threshold, one can consider this document as novel. One common approach - which I want to do - is to use a Vector Space Model and compute the cosine similarity (e.g. by using Apache Lucene).

But this approach has two shortcomings: 1) it is computationally expensive and 2) it does not incorporate the semantics of documents and words respectively.

In order to overcome these shortcomings, my idea was to either use an LDA topic distribution or named entities to augment the Lucene index and the query (i.e. the document collection and each new document) with semantics.

Now, I am completely lost regarding the concrete implementation. I have already trained an LDA topic model using Mallet and I am also able to do Named Entity Recognition on the corpus. But I do not know how to use these topics and named entities in order to realise novelty detection. More specifically, I do not know how to use these features for index and query creation.

For example, is it already sufficient to store all named entities of one document as a separate field in the index, add certain weights (i.e. boost them) and use a MultiFieldQuery? I do not think that this already adds some kind of semantics to the similarity detection. The same applies to LDA topics: is it sufficient to add the topic probability of each term as a Payload and implement a new similarity score?

I would be very happy if you could provide some hints or even code snippets on how to incorporate LDA topics or named entities in Lucene for some kind of novelty detection or semantic similarity measure.

Thank you in advance.

Short answer: Topic Models and LDA aren't suited at all for novelty detection. I did some experiments with data that show a succession of topics in temporal order, and at the boundaries of the time interval one topic gets really high percentages in the document-topic matrix, i.e., the upcoming new trend (or the outrunning old trend) is not seen by LDA.

Topic Models are also not really suited for "online" processing—you read the whole corpus of documents once and than infer the topics.


How to evaluate the quality of speech-to-text data without access to the true labels? I am dealing with a data set of transcribed call center data, where customers are being recorded when interacting with the agent. This is then automatically transcribed by an external transcription system. I want to automatically assess the quality of these transcriptions.
Sadly, the quality seems to be disastrous. In some cases it's little more than gibberish, often due to different dialects the machine is not able to handle. We have no access to the original recordings (data privacy), so there is no way whatsoever to get or create the true labels. The system cannot be replaced as we are committed to it.
Again to the question: is there any way to automatically assess the quality of transcriptions with NLP methods? We want to quantify and compare transcription quality to filter out the best samples for semantic inference of our customers' input in a downstream task. I am thinking about something like a coherence measure in order to find the sentences which make the most sense, grammatically or semantically. Sadly, things as BLEU, WER or Rouge do not work in this case.
I'd be grateful for anything pointing in the right direction. Most importantly again, we have no labels and it needs to be scalable.
Thanks a lot!

There is at least one way:

Create/Acquire a grammar model for the language spoken (there are several such models for various languages used in NLP)
Test the transcripts for beign grammaticaly/syntacticaly correct.
This assesment will at least rule out gibberish and most of transcripts that do not correspond to valid sentences of the language spoken


How can I create convolutions or linear layers that operate on vectors rather than scalars in pytorch? Consider an nn.Linear(2,3) layer transform like the one below.  It uses a 2x3 matrix of scalar weights to create a weighted sum for each scalar element in the output.

Now suppose that instead of scalars I want to operate on 1d vectors (e.g. word embeddings).  So in this case a and b would be vectors and the weights w would each be square matrices.

Can one of the pre-built layer types in nn accomplish this (for linear or convolutions)?  Or is there some way to transform my data to accomplish the equivalent of this with the standard layer types?

thanks!

EDIT: I'm aware that for a given layer like the example I could just do all of the permutations of matrix operations and store the weights for them myself.  What I'm wondering is if there is a better way.

EDIT: Ok, I think what I want does not exist but let me try to clarify on the hopes that maybe there is some similar operation...

Ignoring the bias vector nn.Linear(2,3) is doing the following:

What I'm looking for is the analog of this where a, b, c, ... are vectors and the weighting is a linear transformation of those vectors.  I apologize if my notation is off here but, written as row vectors:

This means that we'd need a 3 dimensional set of weights, not a 2d as in nn.Linear. Again, I guess this is just a different operation entirely but I'm wondering if this kind of thing has a name or if there is some reason it  isn't normally needed (e.g. some transformation of the data that really is equivalent to this).

If I understand your question1, you'd like an input tensor of shape (2, 3), where each element is actually a vector. If that is the case, you are essentially talking about a 3d tensor (instead of 2d).

For example, if each vector is 5 elements long, your input would have the shape (2, 3, 5).

To compute convolutions over this, there are layers that take the dimensions as parameters - have a look at the Convolutional layers like Conv3d.

To compute a linear layer on this input, you still just need to flatten or reshape the tensor to be a single vector.

If you vector lengths can vary, you will have to consider making them all match by either truncating them (making some shorter), or padding (adding e.g. zeros to make shorter ones longer) - or a mixture of both techniques.

1 If I haven't fully understood, please add more details to the example in your question


Neural network recommendations if only few features Can there be some general recommendations for architecture of neural network if there are only a few features, say 2-5 features?

What should be the number of hidden fully connected layers here? How many neurons may be there in each layer? Do number neurons in different layers need to stay constant (e.g. 16,16,16) or rise (e.g. 8,16,32) or fall (32,16,8) or rise and fall (e.g. 16,32,16) or fall and rise (e.g. 32,16,32)?

There's all sorts of rules of thumb for the structure of a neural net (if n features use n+1 or 2n or 1.5n nodes, etc). I wouldn't take any of them as gospel.

The size and structure of your neural network chiefly depends on the complexity of the data you're trying to learn. If your data is already pretty well represented by, say, linear regression, then you don't really need much additional size to your neural network to represent it well. On the other hand, if your five inputs were, say, the move and shoot commands of a recording of a game of Doom, you're gonna need a large neural network to accommodate all the deep and meaningful interactions between the individual commands.

Each successive layer of nodes of a neural network deals with increasingly complicated shapes. The first hidden layer is all sigmoid or ReLU functions, the second layer is  combinations of the first layer, the third is combinations of the second layer, etc.

The general structure of the hidden layers is cone-shaped, tapering towards the right. There are lots of nodes in the first layer, and fewer in each successive layer. You can have constant numbers of nodes per layer, though in my playing around it doesn't seem to help much, and many of the layer nodes go relatively unused. The opposite direction (having few first layer nodes and many last layer nodes) is actively harmful, because the later nodes can only ever be combinations of the earlier nodes, and if the first nodes don't create enough shapes for the later nodes to combine, it won't be able to fit anything.

I've provided a couple of interactive examples through tensorflow's playground.

Normal structure (Decreasing nodes per layer)

Same no. of nodes per layer

Rising no. of nodes per layer

The first two should converge relatively easily, whereas the third shouldn't converge at all.


Unsupervised Learning upon a column of dataset (graph shown) I’m new into Machine Learning so here I am asking for a sanity check, if the question I am asking is even reasonable.

I have a Dataset of columns, so I want to call one of the columns from the csv using pandas.

Take one of the number from that column of numbers, do some unsupervised learning to determine if this value is an anomaly in this column and it belongs there or not.

The graph below shows, how I would see this process going. I am not sure on what I would do regarding what unsupervised method would be best for this case.

A few observations/questions/considerations:

Wikipedia: Unsupervised learning is a type of self-organized Hebbian learning that helps find previously unknown patterns in data set without pre-existing labels.
Being an anomaly is a label.
Why not just Z-score it? https://en.wikipedia.org/wiki/Standard_score#Z-test . Calculate distribution: $\mu$ and $\sigma$, and check if value is
within $\mu$ +/- 1.96 $\sigma$, $95\%$ chance it is from the normal distribution of the numbers in the column.
Can the Value column be something else than numbers? (E.g. color names)
Or could it be a time series with drift?

Edit:

Under the hood, Machine Learning is basically a set of smart statistics and decisions on subsets of columns and rows. But with only one column and no time series, there is nothing smart to subset or decide. Then essentially, it is an exceptional Z-score for the elements. You have to decide on how many $\sigma$ you want.


What does the phrase 'underlying mapping' mean? In a paper by KaimingHe entitled Deep Residual Learning for Image Recognition, there is a phrase 'underlying layer'. What does this mean?

Functions map domains to ranges.  Neural networks learn such functions, so you can think of a neural network as a mapping of input spaces to output spaces.  Deep neural networks are stacked with many layers of course, and each of those can be viewed as sub-functions of the network with their own underlying mappings.  For example, each layer in a convolution network consists of some convolution layers + some other helper layers such as normalization and pooling.
The paper is about the discovery that residual connections in 'very' deep networks help them converge to an accuracy at least as good or better than a shallower version of the same network.
Basically, it was found that deep convolutional networks actually underperformed compared to shallower versions, but that was not due to overfitting.  Intuitively, this is confusing because you would expect to be able to construct a deeper network from a shallower version and have it at least perform as well.  How?  You take a network of say 5 convolution layers, and add 5 new ones.  If the 5 new layers learned an underlying mapping that was the Identity function (i.e. input to output in a one-to-one manner), then the networks should perform similarly.  This is not what happens in reality and the deeper network has a difficult time learning Identity mappings.
If we assume an underlying mapping function of a sublayer (one layer of convolutions), $H(x)$, that can learn a complex function, we find that it has difficulty learning mappings that help the network using SGD.  Their insight was to reformulate the underlying mapping as $H(x) = F(x) - x$. That is, some mapping that is a new function with the original input subtracted.  This naturally leads to 'wiring' the network with a residual and the underlying mapping becomes: $F(x) = H(x) + x$.  The Identity mapping is now trivial and the $H(x)$ mapping can learn to add new complexity.


Seaborn subplots massive whitespace I'm trying to plot three heatmaps in a vertical column using Seaborns subplot method.

import seaborn as sns
initalCorr = inputX.corr()
secondaryCorr = inputX_corr.corr()
finalCorr = inputX[selected_columns_pvalue].corr()

fig, axs = plt.subplots(3, figsize = (15,45))
fig.suptitle('Heatmaps of correlation between features selected for training',
fontsize = 20)

sns.heatmap(data = initalCorr, center = 0, square = True, ax = axs[0])
axs[0].set_title('Correlation between all features', fontsize = 20)

sns.heatmap(secondaryCorr, center = 0, square = True, annot = True, linewidths = .5, fmt = '.2f', ax = axs[1])
axs[1].set_title('Correlation between features slected for P-Value analysis', fontsize = 20)

sns.heatmap(finalCorr, center = 0, square = True, annot = True, linewidths = .5, fmt = '.2f', ax = axs[2])
axs[2].set_title('Correlation between finally selected features', fontsize = 20)

The graphs themselves plot fine but the title appears out of place relative to the plots.

How do I make the title directly above the heatmaps?

Additional:
seaborn version 0.9.0
matplotlib version 3.0.3

You can move the title closer to the first figure. Just add the following two lines at the end of the code

fig.tight_layout()
fig.subplots_adjust(top=0.95)


Abbreviation Classification using machine learning I would like to classify abbreviations using machine learning. For example:
I have watermel. and I ask for user what is watermel.(my application context is about food). Then He classify as watermelon.
In other time, If Other user insert waterme. Is It exist a way to infer that waterme is the same as watermelon using machine learning techniques?

Look into this package for Python:
https://pypi.python.org/pypi/Distance/

You can use this to generate a numeric value representing the similarity between word.

Here is a similar post that should help:
https://stats.stackexchange.com/questions/123060/clustering-a-long-list-of-strings-words-into-similarity-groups

Additionally, a level up in complexity would be to use t-SNE on an array generated using word2vec (this is word embedding).  Examples and resources for this are:
https://www.codeproject.com/tips/788739/visualization-of-high-dimensional-data-using-t-sne
http://sebastianruder.com/word-embeddings-1/


Necessity of balancing positive/negative examples in binary classification machine learning? When using machine learning models like gradient boosted trees and CNN, is it required (or considered as an always-do good practice) to balance the amount of positive/negative examples when learning for binary classification?

Given P positive examples and N negative examples, where P << N, I can think of several choices: (Let's forget about validation set and test set)

Choice A) No balancing at all, put all examples (totally P+N) into the training set without weighting w.r.t. their ratio.

Choice B) Put all examples (totally P+N) into the training set, but weight all positive examples 1/2P and all negative examples 1/2N, so that total weight of positive examples and negative example equal.

Choice C) Take all P positive examples, then sample P negative examples (out of N), and train with these 2P examples with uniform weighting.

What are the pros/cons for each of the approach and which one(s) do we usually go with?

Let's start by answering your first question. Is it required to balance the dataset?

Absolutely, the reason is simple in failing to do so you end up with algorithmic bias. This means that if you train your classifier without balancing the classifier has a high chance of favoring one of the classes with the most examples. This is especially the case with boosted trees. Even normal decision trees, in general, have the same effect. So it is always important to balance the dataset

Now let's discuss the three different scenarios placed.

Choice A): This would be what I explained all along. I'm not saying necessarily you will have a bias. It depends on the dataset itself. If the nature of the dataset has a very fine distinction with the boundaries then the chance of misclassification is reduced, you might get a decent result but it's still not recommended. Also if the data does not have good boundaries then the rate of misclassification rises a lot.

Choice B): Since you are placing weights for each sample you are trying to overcome the bias with a penalty. This is also called as an Asymmetric method. Normally these methods increase the accuracy of a model by a slight margin but that mostly depends on the machine learning algorithm you are using. In examples like Adaboost such a model the effectivity of the model increases. This method is also called Asymmetric Adaboost. But this might not necessarily work with all algorithms.

Choice C): Assuming you have weighted the samples accordingly it should do the same as either choice A or choice B. I'll leave this for you to extrapolate based on my previous explanations.


Examples on kohonen self organizing maps Is there a simple example to start with for using kohonen 1.1.2 or is it only the test file that will be the reference?

There are alternative solutions for self organizing maps.

Best of them I found pymvpa where the example is easy to read and understand. It is also maintained quite activately as you can see from their Github.

I tried to run the kohonen 1.1.2 test file, but it did not run after two days of trying. So, let's have a try for the another solution.

To run pyMVPA example som.py, you have to do the following (at least):

1 install some pre-requisites: at least numpy, scipy, nibabel and swig to be able to run setup.py
2 run the commands:
   python setup.py build
   python setup.py install
3 modify example under doc/examples (som.py) to have
 import matplotlib.pyplot as pl
4 run command:
   python som.py

Note that this works only in python 2, not python3 can be used as command.


Do we need to increase training data size when increasing dropouts? I am using a fully connected feed forward neural network built using keras for text classification. It consists of 3 hidden layer. I am planning to add a dropout layer after each hidden layer to prevent overfitting. While tuning the dropout rate, I am increasing the value from 0.2 -> 0.3 -> 0.4 -> 0.5.

I want to know if I should increase the training data size to have a more accurate comparison. What I mean is suppose I am having training data of size 1 million for a dropout rate 0.2. Should I increase the training data size to 1.5 million for dropout rate 0.3?

Calculation:

1000000 * 0.8 * 0.8 * 0.8 = 512000 (for dropout ratio 0.2)

1500000 * 0.7 * 0.7 * 0.7 = 514500 (for dropout ratio 0.3)

No, you should only tune one hyperparameter at a time. If you change two hyperparameters and the performance increases, how do you know which of the two parameters is responsible?

If you have the time, do a full grid search on the dropout rate and the training data size.


Understanding text conversion into SVM input In Support Vector Machines, when used for sentiment analysis, text gets converted into a set of data points. How does this happen, usually?

Text can be converted to data via the use of concept clusters (after stemming and stopping), or to count (frequencies) via use of n-grams.  N-grams are basically tabulations of the 1-gram count (frequency) of alphabet characters (a though z) in each document, and counts of 2-grams (aa to zz), 3-grams (aaa through zzz), up to about 5-grams (aaaaa through zzzzz).  Beyond 5-grams, the data will be sparse and less informative.  Thus, a dataset can be constructed for which rows represent documents, and columns represent n-grams. The data values themselves are the total number of occurrences of each gram found in each document.

FYI - n-grams have proven to be the best technique for identifying different languages based on characters.

Regarding SVMs, focus on the SVM literature.


How to train neural networks with large sized data sets? I have a dataset size of ~500000 with input dimension 46.
I am trying to use Pybrain to train the network but the training is extremely slow for the whole dataset. Using batches of 50000 data points, each batch takes more than 2 hours for training.

What are caveats of optimizing the network design so that the training is faster?

Here are some of the things that influence your training speed:

Number of weights in your network
Speed of your CPU
Package you are using (mostly engine it is working on, in PyLearn this is Theano)
If all your data fits in memory or you are reading from disk in between batches

With regards to network design the only thing you can really do is make the network more shallow to reduce the number of weights. To reduce the number of epochs there might be other options like adding residual connections but that will not decrease the training time of 1 epoch.

Without more information it is unclear where the bottleneck is, but 20 hours for one epoch seems a bit high. The easiest and biggest improvement you will be able to get is to use a good GPU, which should be possible using pylearn since it is built on top of Theano.


How to handle multiple datasets of sequential data New to data science and I am working on a personal project in sports analytics.
I have data in the form of one dataset for each year/season.

Each dataset has the team's wins, losses, and various offensive and defensive stats.
These will be pandas.DataFrames indexed by team name.

My goal is to predict wins and losses based on this data for each year, and then create a more generalized model to predict next season using all of the previous years data.

I am having difficulty deciding how best to combine the data. I think that it would be very inefficient to combine all of the datasets into 1 and have variables for each stat in the year prior, 2 years prior, etc.
How is this type of sequential data usually handled?

(For clarification, I plan on using gradient boosting via xgboost package and an LSTM via the Keras package)

Generally, to perform machine learning all data needs to be in a single dataframe. Team name, better yet team id, should be the primary key. Then modeling each year requires selecting the relevant columns (e.g., wins_year_1, loses_year_1, …). Finally, the general model would select all the columns.

It is best practice to transform all data before modeling. This avoids making transformation errors on some of the data which can lead to errors in modeling.

Pandas support this type of combining through the merge command.

Typically, this data is not called sequential data. Sequential data implies a notion of state. The state changes values over time. In most sports, recent performance is not a strong indicator of how likely a team will the next game. This is often referred to as the Hot Hand Fallacy. Thus, current "win/lose" state of a team is not a strong feature in modeling.


LSTM classification with different sizes I'm relatively new to the world of recurrent neural nets and I'm trying to build a classifier using an LSTM model to predict HIV activity from a given molecule (the original dataset can be found here ).
I have sequences of different lengths (from few dozen to almost 400 characters) but I'm not sure how to proceed. Let's say that I have a dataset structured like so:
import pandas as pd
import random
import string

random.seed(42)
seqs = [[random.choice(string.ascii_letters) for i in range(random.randint(1,10))] for i in range(5)]
classes = [random.randint(0,1) for i in range(5)]
df = pd.DataFrame({
    "seq": seqs,
    "class": classes
})
print(df)

    seq                             class
0   [b, V]                          0
1   [p, o, i, V, g]                 0
2   [f, L, B, c, b, f, n, o, G]     1
3   [b, J, m, T, P, S, I, A, o, C]  0
4   [r, Z, a, W, Z, k, S, B, v, r]  0

I know I should:

one hot encode the elements,
perform masking and padding

But I don't know how to perform it in Keras/TF2 and I can't find any resources online that explain how to code something similar.

masking and padding are indeed popular preprocessing choices when working with LSTM. The easiest way is to pad all the short sequences to the same length as the longest sequence.
Here is an example of masking and padding in Keras
https://www.tensorflow.org/guide/keras/masking_and_padding
Is that what you're looking for?
raw_inputs = [
    [711, 632, 71],
    [73, 8, 3215, 55, 927],
    [83, 91, 1, 645, 1253, 927],
]

# By default, this will pad using 0s; it is configurable via the
# "value" parameter.
# Note that you could "pre" padding (at the beginning) or
# "post" padding (at the end).
# We recommend using "post" padding when working with RNN layers
# (in order to be able to use the
# CuDNN implementation of the layers).
padded_inputs = tf.keras.preprocessing.sequence.pad_sequences(
    raw_inputs, padding="post"
)
print(padded_inputs)

Output:
[[ 711  632   71    0    0    0]
 [  73    8 3215   55  927    0]
 [  83   91    1  645 1253  927]]

Besides, you can also add start and end tokens to your sequences, like this example (https://www.tensorflow.org/tutorials/text/nmt_with_attention)
def preprocess_sentence(w):
  w = unicode_to_ascii(w.lower().strip())

  # creating a space between a word and the punctuation following it
  # eg: "he is a boy." => "he is a boy ."
  # Reference:- https://stackoverflow.com/questions/3645931/python-padding-punctuation-with-white-spaces-keeping-punctuation
  w = re.sub(r"([?.!,¿])", r" \1 ", w)
  w = re.sub(r'[" "]+', " ", w)

  # replacing everything with space except (a-z, A-Z, ".", "?", "!", ",")
  w = re.sub(r"[^a-zA-Z?.!,¿]+", " ", w)

  w = w.strip()

  # adding a start and an end token to the sentence
  # so that the model know when to start and stop predicting.
  w = '<start> ' + w + ' <end>'
  return w

```