How to automatically test for the best parameters for transformed independent variable in linear model Let's assume that I have a linear model with $k$ variables:

$y = \beta_0 + \beta_1\cdot x_1 + \dots + \beta_k \cdot x_k$.

Now, I want to add variable $x_{k+1}$, but, according to domain knowledge, the dependency of $y$ onto $x_{k+1}$ is not linear, but rather "S-shaped". To capture this dependency, a well-established method is to use parametrised arcus tangens function: $D \cdot \arctan{\frac{x_{k+1} - A}{B}} + C$. To include it in the linear model we can safely ignore $D$ and $C$ parameters (as they will contribute to $\beta_{k+1}$ and $\beta_0$ respectively), so eventually I'd end up with the model:

$y = \beta_0 + \beta_1\cdot x_1 + \dots + \beta_k \cdot x_k + \beta_{k+1}\cdot \arctan{\frac{x_{k+1} - A}{B}}$.

What I would like to do is to find a way to test different transformations (with different $A$ and $B$ parameters automatically. Firstly, I went for searching the grid of different $A$ and $B$ values, i.e. fitting the model to different transformations of $x_{k+1}$ and returning list of models, sorted by $R^2$, $RMSE$ or $AIC$. However, this usually favours very sharp curves:

This doesn't make much sense in the eye test though. I know that eye-test may be biased but to me, more propable dependency would be captured by a shape as below:

One could already noticed that there are plenty of observations where $x_{k+1}$ is or is near to $0$, which is actually a characteristic of those variables that will be approximated by $arctan$. That may probably affect the fitting, but still something clearly doesn't work here.

Another approach that I considered is to fit the model using gradient descent search rather than using standard software (like lm from R). The problem is that with this $arctan$ transformation, we don't have a convex objective function anymore.

How can I then automatically test for the best parameters of the $arctan$ transformation? Am I on the right track with either idea or can I approach this problem differently?r

One approach would be to use an algorithm designed for non-convex problems like Bayesian optimization. However, if you have already evaluated a fine grid of parameters this is unlikely to offer significant improvement. Here is an example of how you could implement Bayesian optimization for this problem.

First, we need some data. Just for fun let’s extract the data from the images you posted (in brief since this is off topic).

In Mathematica:

img = Import[NotebookDirectory[]<>"LHrXQ.png"]
img2 = ImageResize[ImageTake[img, {40, 450}, {50, 1000}], 200]
pixels = {(#[[1]]-10)*3.8,#[[2]]*9+100}&/@PixelValuePositions[img2,Black, 0.4];

ListPlot[pixels, Frame->True, ImageSize->500]

Now in order to use Bayesian optimization we need to define the objective as a function of parameters A and B. Here we will maximize the R^2 value for the fitted model.

In python:

import pandas as pd
import numpy as np
import random
from sklearn.linear_model import LinearRegression

data = pd.read_csv('mathematica_data.csv')

def objective(params):
    """Whatever you want to do for your regression."""

    # So it works with GpyOpt
    A = params[0][0]
    B = params[0][1]

    temp = data.copy()

    # Transform variable
    xt = [np.arctan((x - A)/B) for x in data['x1'].tolist()]
    temp['x1'] = xt

    # Fit a linear model
    reg = LinearRegression().fit(temp.drop('y', axis=1), temp['y'])

    # Compute scores of interest
    r2 = reg.score(temp.drop('y', axis=1), temp['y'])

    # GPyOpt will minimize so we want -f
    return - r2

Now use GPyOpt to optimize the objective.

import GPyOpt

domain = [{'name': 'A', 'type': 'continuous', 'domain': (-300.0,300.0)},
          {'name': 'B', 'type': 'continuous', 'domain': (0.1,300.0)}]

bo = GPyOpt.methods.BayesianOptimization(f=objective,
                                         domain=domain,
                                         model_type='GP',
                                         acquisition_type='EI',
                                         initial_design_numdata=10,
                                         initial_design_type='random',
                                         acquisition_jitter=0.01,
                                         num_cores=-1,
                                         de_duplication=True,
                                         exact_feval=True)

# Run optimization
bo.run_optimization(max_iter=100)

We can plot the optimizers convergence:

bo.plot_convergence()

And information about how it sampled the parameters:

bo.plot_acquisition()

Of course using only the independent variable extracted from your plot even the best R^2 values suggests there is little to no relationship between arctan((x-A)/B).


How to implement conditional probability distribution on set-valued Random variables I'm trying to implement conditional probability distribution when the events of two RVs are sets. If I try to extrapolate concepts from real or categorical variables to sets things become confusing for me. In particular, for some pair of discrete RVs, $X,Y$ we can compute the conditional probability of two events by using the Bayes theorem:
$$P(Y=y|X=x)=\frac{P(X=x|Y=y)P(Y=y)}{\sum_{y'\in\mathcal{Y}}P(X=x|Y=y')P(Y=y')}=\frac{f(x,y)}{\sum_{y'\in\mathcal{Y}}f(x,y')},$$
where $f(x,y)=\sum_{(x',y')\in (\mathcal{X},\mathcal{Y})}\chi(x'=x,y'=y)$ is an accumulator function giving the total number of times the specific joint event $x,y$ is observed in the sample space (i.e. the indicator function $\chi(x'=x,y'=y)$ equals one when $x'=x,y'=y$ holds). I'd like to compute such probability when events drawn by $X,Y$ are sets $X_i,Y_j$, respectively. So, how to define the corresponding $f(X_i,Y_j)$?
I've tried with:
$$f(X_i,Y_i)=\sum_{j}|(X_i\cap Y_i)\cap(X_j\cap Y_j)|.$$
The motivation behind is that $X_i\cap Y_i$ provides a measure of how the attributes of both sets should be observed jointly (such as any intersection event). In turn intersecting this intersection event against other intersection events $X_j\cap Y_j$ should provide the measure of how much the joint attributes are observed through the sample space. However, I'm not sure whether this analogy is correct with respect to the usual notion of observing joint events in a sample space. Please, some one can help me to clarify/fix/prove this?
Thank you in advance

Keeping it to discrete space, the joint probability of an event can be interpreted as the fraction of the size of this subset compared to the whole state space.
In case of events, being sets themselves, the definition remains the same.
$$P(X_i,Y_j) = \frac{|\{(x,y) \in  (X_i,Y_j)|\}} {|\Omega|} $$
$|\Omega|$ is just the set of all combinations of single events (x,y).
Let us now check your formula with the following example:
$$X_0 = Y_1 = \{0\}, ~~ X_1 = Y_0 = \{1\}$$
Then the total space space $X \times Y$,
$$ \Omega  = \{(0,0),(0,1),(1,0),(1,1)\} $$
And for example:
$$ P(X_0,Y_0) = P( (0,0) ) = \frac{1}{4} $$
Your formula however implies,
$$ | (X_i \cap Y_i) \cap  (X_j \cap Y_j) | = 0 $$
as $X_i \cap Y_i$ is empty per contruction in the example.


Understanding Node Embeddings I have only just started to look into graph neural networks and I am a little confused on the node embedding process. Here is my understanding, please let me know if i misunderstood:
Given unlabelled data, we try to construct a graph $G=(V,E)$ and generate a mask that contains important features of each node such as shape, size, intensity etc. Then let's say the aim is to try and classify the nodes. So, this is where we use node embedding, it seems that it takes all of those aforementioned features and tries to condense them into a embedding matrix. And the reason we do this is because this removes all unneccessary information from the adjacency matrix and only contains all the important features of the nodes. Then afterwards we use this matrix with graph convolutions to ultimately classify unseen images. Is this right?
So let's take an example, I've been looking into GraphSage, as this seems one of the best methods (is it?). It seems it can learn embeddings from a local neighbourhood, but i want to know how does that work?

First, let me give you an intuitive explanation. When you drop a pebble in water, you see the ripples being formed. Imagine that in reverse. All those ripples comes together at the point from which they started.
Node embeddings are like that. You take the information of neighbourhood nodes and combine it with the information in the original node. The art lies in how you combine the neighbourhood nodes and until which k-hop neighbours you do it. 1-hop means direct neighbours, 2-hop means neighbour of neighbours and so on.
In context of GraphSAGE, the combination of information is done using a simple feedforward neural network and the neighbourhood is incorporated iteratively. See link for further understanding.


Suggestions for good data science certified online courses I am wondering if someone here can recommend some good data science certified online courses.

A simple google research brings hundreds of online academy and I am lost.

It is been one year since I am working in this field and I want to validate my knowledge and acquire more.

There are many but some of the resources are awesome. One such resource is Udacity. This is probably the best resource I have encountered yet.


Weighted clustering of lat lon coordinates I have millions of lat long points that have been grouped into squares. Some squares have thousands of points, others have a couple of points. The idea is that we have one set of lat long for the square with a weighting related to the square based on number of items it has instead of having millions of rows of data to cluster.

I was originally using the leadercluster algorithim which allowed me to specify the distance each of my clusters should cover. This is ideal for my use case but now i would like to use it to cluster squares factoring in the weight of the squares. This is basically a learning experience for me and any help would be greatly appreciated

I have seen this which could be useful but I'm not really able to make headway with it

Below is some sample data in R about airports and arrivals

df <- read.csv('https://raw.githubusercontent.com/plotly/datasets/master/2011_february_us_airport_traffic.csv')

ggplot(df, aes(x = long, y = lat)) +
  geom_point()

sample <- df %>%
  select(long, lat, arrivals = cnt)

Leader clustering is so simple, the weight does not make a difference.

It assigns points to a cluster if the distance is less than a threshold. It does not matter whether the point has 1 weight or is a "square" of 100.


This is the so-called "Class Imbalance Problem". Fortunately, there are a number of possible approaches.

Probably easiest is going to be to use the class_weight option present in a number of scikit-learn's classifiers. For example, LogisticRegression.

You can either set this option to "balanced" (or "auto" if you're using an older version of sklearn), in which case it automatically adjusts weights inversely proportional to class frequency (i.e. it tries to compensate for the class imbalance); or you can manually set the weights for each class label using a dict, e.g. {0:1, 1: 20}.

For a worked example using an SVM, see here.

See here for more information: https://stats.stackexchange.com/questions/131255/class-imbalance-in-supervised-machine-learning


Why does the non autoregresive transfomer model in fairseq require the prev_output_tokens input? fairseq includes an implementation of a non autoregressive transformer - which (as much as I understand) means that the whole output sequence is generated in a single forward run (in contrast to autoregresive models where each forward run predicts the next token from the input and the previous predicted tokens)
However, from the code it appears that the models still expects the previous tokens as input:
def forward(self, src_tokens, src_lengths, prev_output_tokens, tgt_tokens, **kwargs)

https://github.com/pytorch/fairseq/blob/master/fairseq/models/nat/nonautoregressive_transformer.py#L78

It is there to maintain consistency with the signature of the forward method of the base class TransformerModel and therefore allow to use it in place of any other autoregressive transformer, but it is actually not used. The same happens in the model decoder.


Creating similarity metric with Doc2Vec and additional features I have a dataset which contains many features. Each record is company that has many features.

For example...

Company A:

Keywords - data, big data, tableau, dashboards, etc.
Industry - Information Technology
Sub-Industry - Data Visualization
Total Funding - $150,000,000

I want to create a similarity metric between multiple companies, incorporating both doc2vec embeddings trained on the keyword lists as well as the additional features listed. I had a hard time searching/finding papers that did something like this. Any ideas?

You could think of your similarity measure as a search problem if you consider one record a query, and the "near" records as search results.

I've had some good results following this paper: https://arxiv.org/pdf/1602.01137.pdf

As I understand it, the document vectors used in the paper were only good for improving the relevance of search results for results that were already decent (the top N results).

That to me suggests you might try developing a similarity score that works with your other attributes first, and then do something like a weighted average, where the significance of the doc2vec score decays quickly based on the first metric.


Learning the uncertainty of a ML algorithm I have a regression GAM (General Additive Model) and I want to learn its epistemic uncertainty( the variance of my residuals or predictions as a function of my input).
I have already used a bayesian approach to turn my GAM into a gaussian process so I can construct a covariance matrix but this approach is not scalable due to the high dimension of my problem.
I am trying to use an approach that uses the current model as a black-box and observe only the input and the residual, the closest thing that I found is quantile regression but I was wondering if there is any deep learning approach that learn the variance from the input.
Most of deep learning approach that I found estimate the mean and the variance simultaneously (Deep bayes, MVE, MC dropout ...)
A naive approach that I am implementing currently is a neural network that learn the variance as function of my input by minimizing the likelihood of my residuals as a centred gaussian but I didn't find any paper or ressources on this approach.
Do you have any idea on the problem, any possible ressources or an opinion on my current approach ?
Thanks

Take a look on Google trust score - https://github.com/google/TrustScore
Based on the following paper - https://papers.nips.cc/paper/9547-can-you-trust-your-models-uncertainty-evaluating-predictive-uncertainty-under-dataset-shift.pdf
A relatively naive but useful approach


seaborn regplot does not show the line Question

Why lmplot show the line but regplot does not?

lmplot

import seaborn as sns
import matplotlib.pyplot as plt

sns.lmplot(x="Value", y="dollar_price", data=merged_df, height=8, aspect=1.5)

regplot

# seaborn.regplot() returns matplotlib.Axes object
plt.rcParams['figure.figsize'] = (15,10)
ax = sns.regplot(x="Value", y="dollar_price", data=merged_df, fit_reg=False)
ax.set_xlabel("GDP per capita (constant 2000 US$) 2017")
ax.set_ylabel("BigMac index (US$)")

# Label the country code for those who demonstrate extreme BigMac index
for row in merged_df.itertuples():
    ax.text(row.Value,row.dollar_price+0.1,row.country)

Documentation

seaborn.regplot documentation page shows the line. So I suppose there is something missing?

>>> import seaborn as sns; sns.set(color_codes=True)
>>> tips = sns.load_dataset("tips")
>>> ax = sns.regplot(x="total_bill", y="tip", data=tips)

ax = sns.regplot(x="Value", y="dollar_price", data=merged_df, fit_reg=False)

You have the fit_reg set to False here. Just set it True, you should see the regression line.


How can I read input node and output node from .h5 file in keras model? I trained my own dataset using MaskRCNN. My trained file is in .h5 file format. Now I want to know/read input node names and output node names.

How can I read?

I assume that you have the Keras .h5 file ready. Now, we need to load the model from that file using,

model = keras.models.load_model( 'model.h5' )

model represents the Keras model. Now, model.input will correspond to the input tensor.

input = model.input
output = model.output

Also, you get the output in a similar way. The input and output tensors will have their names. This can be observed by printing the tensor using Python's print.


Will my neural network get lazy if I give it an "easy" feature? Let's say I start with a standard conv net architecture capable of 99% accuracy on MNIST such as this one, but let's say I merge in an "easy" feature to the fully-connecter layer such as a vector of length 10 that encodes the correct output digit 95% of the time and a random digit otherwise.

Will my network ever reach 99% accuracy
Will it take longer to get there?

I think the answer the question 1 is yes, because we will should always be able to find a downhill in the error before the conv net path has reached its full potential. In fact I think it may surpass the accuracy of the original architecture since we are leaking information about the correct output labels.

However I am not so sure about question 2. Does this make the shape of error function unfavorable in any way? I can't decide whether reaching the same accuracy of the original architecture will be faster, slower, or exactly the same.

Hopefully there is an easy answer, otherwise I will run an experiment and report back!

Interesting theoretical question. While I cannot answer this with 100% certainty, my own intuition and experience say:

1) I would be very surprised if this is not the case. If we look at different problems, where we have one very important feature that holds most of the information but some other features with some signal in there the model generally still improves by adding these lower value features. That is exactly what you are doing here, you have the MNIST pixels that individually hold little value but combined tell you quite a bit. The value of these pixels goes down quite a bit by adding such a strong feature, but the value is absolutely not reduced to 0. I think in theory this should lead to a strictly better model.

2) I think it will take around the same time to converge on average. Fitting to the one-hot encoded feature will be very fast which means most of the time will be spent on fitting to the features, which is a similar problem as without the feature. However, instead of going from 10% to 99% accuracy, we are now kind of going from 95% accuracy to 99.7% accuracy.

I'm less certain about the second question but I think this is what you would find if you do the (interesting) experiment. Let us know if you do implement it!


Text Classification on a very small data set with a lot of classes I have a data set consisting of 455 rows spread over 21 different classes. The data set is imbalanced as well as you can see below.

job_alerts                    45
howto_apply                   40
application_status            31
salary                        30
job_close_date                30
multiple_role                 27
feedback                      26
assessment_campatilibility    24
interview_reschedule          23
disability                    22
reinstate_application         19
job_account_issue             19
assessment_link_problem       16
age_limit                     16
assessment_timebox            16
cv_past_experience            15
late_for_interview            13
interview_response_time       13
work_experience               11
assessment_validity           10
special_needs_at_work          9

I'm trying to perform classification on this. Firstly, when preprocessing the data I convert the text to lowercase, remove stopwords and punctuation and convert words to their lemmas using spaCy.

So far, I have tried the TextCategorizer in spaCy which uses a simple CNN for classification and I'm getting around 70% accuracy on an 80-20 split of the data. I have also tried various Sklearn algorithms such as SVC, LogisticRegression, RandomForestClassifier and MultinomialDB with the  TfIDVectorizer. I have used GridSearchCV to tune the parameters. The best results I get are with SVC at around 74-75% accuracy.

I want to know what else I can try to improve my accuracy. I am a beginner to NLP and I haven't worked on something like this before. Right now the ideas I have are to improve the TfIdVectorizer through some parameter tuning and to try a One vs all classifier for SVC. What else can you suggest to me?

Accuracy is not the best measure for imbalanced data. Prefer precision and recall.
Do undersampling/oversampling to get equal samples for each class and try XGBoost.
Or else you can use SVC with class weights, give lower class weight to classes with more samples and vice versa.


Is my model overfitting? The validation loss keeps on fluctuating I have trained a 4 layer neural network

model = Sequential()

#get number of columns in training data
n_cols = X_train.shape[1]

#add model layers
model.add(Dense(8, activation='relu', input_shape=(n_cols,)))
model.add(Dense(8, activation='relu'))
model.add(Dropout(rate = 0.05))
model.add(Dense(8, activation='relu'))

model.add(Dense(1))
#adam = optimizers.Adam(lr=0.0001, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False)
model.compile(optimizer='adam', loss='mae')
history = model.fit(X_train, y_train, epochs= 200, validation_split=0.2, batch_size=128)

When I plot the graph between train and validation loss the graph seems to be like
The validation loss is fluctuating. Am I doing it right?

Overfitting is a situation when model starts to perform more better on training set than on validation (example of such behaviour: loss curves are moving to different ways). According to your plot the model hasn't overfitted. Validation loss seems to fluctuating more than train, because you have more points in training dataset and errors on test have higher influence while loss is calculated.


How can calculate Efficiency for predictive models based on accuracy or error over time? I was wondering if I could express the efficiency of prognostic models according to their accuracy(error, e.g. MAPE or MSE) over time [sec]. So let's imagine I have the following results for different predictive models:

models MSE   MAE     MAPE       predicting Time[sec]
LSTM   0.12  0.13   15.67%          456789
GRU    0.06  0.05   5.89%           688741
RNN    0.45  0.51   25.33%          55555

What is the best way to illustrate the efficiency of predictive models over predicting time? Is the following equation right? how about its unit when we use MSE or MAE instead of MAPE which is expressed by %?

\begin{equation}
\text{Efficiency} = \frac{Accuracy}{predictingtime }= \frac{error (e.g.MAPE)}{predictingtime }= \frac{percentage}{sec}
\end{equation}
which graph demosntraete efficiency scientifically for prognestic models?

I read this article and the introduce following criteria under name of prediction efficiency PE. is it possible to use such that?

Given that computation time is system dependent maybe you could consider MAE/number of parameters (in your case it would be weights since they are all NNs)

If you want to use time, your are right cant compare apples with oranges, so just drop the percantage and make sure they are normalized values (all future metrics)


Precision vs. Recall Can I make the following statement about a binary classification, please?

Precision 1: 0.10
Recall 1: 0.83

Statement: "We can expect 90% false alarms (1 - 0.10). But for the remaining 10%, we can be around 83% certain (Recall 1: 0.83), that we caught a label 1."

Thanks in advance!

I would phrase it like so:

"Of all records that were labelled 1 by the model, 10% were actually 1 (90% incorrect predictions). Of all records that were truly labelled 1 we predicted 83% correctly."

While this is out of context of your question, if support refers to the number of records then it would be beneficial to get a more balanced dataset. The reason your precision is so poor for label 1 is because there are many more "negatives" (0's) than "positives" (1) increasing the chance for false positives to occur, affecting your precision.

EDIT:
This on Cross Validated will help provide more explanation.


how to perform Step calcuations in R I have a rate-step plan for consumption with which I'm trying to total up costs. I have a rate plant that looks like this:

First 20 Kwh is Free;
Second 5 Kwh is $.1
    Next 100 Kwh is $.053
Next 875 Kwh is $.042
    Over 1000Kwh is $.039

Can someone provide a worked example for me, e.g., with $\text{consumtpion}=625$? If possible, with R code for calculating the total cost.

For my own approach, I was thinking of using a loop, but if it's a lot of accounts the loop will take too long.  Can anyone think of a vector-based approach for calculating the result?

Try a simple equation like this:
\begin{eqnarray}
&&\max(x-20,0)*0.1 + \max(0,x-25)*(-.047) + \\
&&\max(0, x-125)*(-.011) + \max(0,x-1000)*(-.003)
\end{eqnarray}

This should take the usage $(x)$ and walk it through steps, initially charging 0.1 for each Kwh over 20: $\max(x-20,0)*0.1$

Then, if $x$ is above 20, it takes away 0.047 (0.1-0.053) for every Kwh between 20 and 25: $\max(0, x-25)*(-0.047)$

Then, if $x$ is above 125, it takes away 0.011 (0.053-0.042) for every Kwh between 25 and 125: $\max(0, x-125)*(-0.011)$


Machine learning algorithm for a dynamic set of attributes I'm new to machine learning programming and am working on an application which will calculate weights of the attributes to calculate the ETAT (Estimated Turn Around Time) of some tasks in workflow. The set of attributes is not fixed.

For example: ETAT will depend on-

Language
No of properties
Experience level of resources and many
more.

For each workflow entity there can by a set of these attributes, i.e. a single workflow may be like:

Language - English
No. of properties - 1200
Experience level of
resources - Expert

and

Language - Russian
No. of properties - 21000
Experience level of resources - Basic

Now for the above set of attributes, how can I design my machine learning algorithm? Any pointers will be highly useful.

I would train a gamma-distributed GLM; one for each language. Now, I assume the set of parameters is fixed within each language. You can incorporate ordinal variables (basic -> intermediate -> expert) using dummy coding (i.e., treat them as categorical variables). If there is no intermediate level one bit will do (0 = basic, 1 = expert). And so on for the other properties.

You can find free software to implement GLMs in various language, including python and R.


H2O deep learning model performance I am discovering H2O deeplearning and I would like to have your point of view about the performance that's performed my model on classification problem. Do you think my model is overfitting?

dl_fit2 <- h2o.deeplearning(x = predictors, y = response,
                           training_frame = train,
                           validation_frame = valid,
                           epochs = 200,
                           score_validation_samples=10000,
                           score_duty_cycle=0.025,
                           activation = "RectifierWithDropout",
                           hidden = c(80, 10, 80),
                           hidden_dropout_ratios = c(0.2, 0.2, 0.2),
                           loss = "CrossEntropy",
                           rate=0.01,
                           rate_annealing=2e-6,
                           adaptive_rate = FALSE,
                           momentum_start = 0.2,
                           momentum_ramp = 1e7,
                           momentum_stable = 0.4,
                           nesterov_accelerated_gradient = TRUE,
                           l1 = 1e-5,
                           l2 = 1e-5,
                           max_w2=10
)

MSE:  0.009757329
RMSE:  0.09877919
LogLoss:  0.03527449
Mean Per-Class Error:  0.01219048
AUC:  0.9991871
pr_auc:  0.4974259
Gini:  0.9983743

Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:
          NO   YES    Error        Rate
NO     10022   140 0.013777  =140/10162
YES      109 10170 0.010604  =109/10279
Totals 10131 10310 0.012181  =249/20441

Maximum Metrics: Maximum metrics at their respective thresholds
                        metric threshold    value idx
1                       max f1  0.367137 0.987906 212
2                       max f2  0.178219 0.990603 270
3                 max f0point5  0.765904 0.990982 122
4                 max accuracy  0.371864 0.987819 210
5                max precision  0.999979 1.000000   0
6                   max recall  0.001865 1.000000 387
7              max specificity  0.999979 1.000000   0
8             max absolute_mcc  0.367137 0.975640 212
9   max min_per_class_accuracy  0.385816 0.987256 204
10 max mean_per_class_accuracy  0.371864 0.987812 210

In general we would need more info on your data and what you are trying to achieve.

However, with 0.999 AUC, it is very likely you are overfitting (or you have a very simple problem).


How to structure time series for binary classification in Keras? I'm trying to classify churn (1 or 0) for a user based on day-to-day time series and activity levels, something like this:

User     Date      Activity Churn
1    09-01-2018       35      0
1    09-02-2018       17      0
2    09-01-2018        0      1
2    09-02-2018       13      1
etc...

Could someone give a few pointers on how this would be setup in Keras to predict the churn value?

you want to treat the date as a cyclical feature to capture patterns in the time stamp.

http://blog.davidkaleko.com/feature-engineering-cyclical-features.html

Keras + SK learn work nicely together. here's an example in my GitHub to extract patterns from the time stamp (just adjust to your needs and replace my Decision Tree with Keras code).

https://github.com/FrancoSwiss/Fraud_Detection/blob/master/Kaggle%20Fraud%20(1).ipynb]1


Ensembling expressions I have two models, $m_1$ and $m_2$, and I want to ensemble them into a final model. I want to be able weight one or the other more according to a grid search. There are two main ideas that come to my mind when doing so:

Define a family of models $m_1 \cdot a + m_2 \cdot (1 - a)$, where $0 < a < 1$, find the $a$ that gives the best score.
Define a family of models $m_1^a \cdot m_2^{1 - a}$, where $0 < a < 1$, find the $a$ that gives the best score.

However, in certain cases, I've seen top models in Kaggle competitions doing fairly different things, like having a final model of the form $m_1^a + m_2^b$.
My question is, what are the advantages and disadvantages of every solution? When do they work better and when do they work worse? When is the third kind of ensemble suitable, and is there any heuristic to tune $a$ and $b$?

That is an empirical question. The answer will change for different models and different datasets.
The best approach would use cross validation to see which ensembling technique has the best score on the evaluation metric for the given data.


Why is my training accuracy 0.0? The Sizes of both the true label and predicted label are same still, the training accuracy is 0.0

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.metrics import accuracy_score

Data Preprocessing

train=pd.read_csv(r'C:\Users\yashd\Downloads\Datasets\titanic\train.csv')
train=train.dropna()
y_train=np.array(train['Survived'])
train=train.drop('Survived',axis=1)  #removing the label from the data

train=train.drop('PassengerId',axis=1) #removing irrelevant features from the training data
train=train.drop('Cabin',axis=1)
train=train.drop('Name',axis=1)
train=train.drop('Ticket',axis=1)

train['Sex']=np.where(train['Sex']=='male',1,0) #assigning a value of 1 to male and 0 to female
train['Embarked']=np.where(train['Embarked']=='S',1,np.where(train['Embarked']=='Q',2,3))
train['Fare']=(train['Fare']-train['Fare'].mean())/train['Fare'].var()
train['Age']=(train['Age']-train['Age'].mean())/train['Age'].var()

x_train=np.array(train)
x_train=x_train.T
y_train=y_train.reshape(1,-1)

Neural Network with 2 hidden layers ,128 neuorns in the first hidden layers and 64 in the second hidden layer. The output layers consists of single sigmoid Neuron

class FNN:
    def __init__(self):
        self.W1=None
        self.b1=None
        self.W2=None
        self.b2=None
        self.W3=None
        self.b3=None
    def sigmoid(self,x):
        return 1/(1+np.exp(-x))
    def forward_prop(self,x):
        self.Z1=np.dot(self.W1,x)+self.b1
        self.A1=np.tanh(self.Z1)
        self.Z2=np.dot(self.W2,self.A1)+self.b2
        self.A2=np.tanh(self.Z2)
        self.Z3=np.dot(self.W3,self.A2)+self.b3
        self.A3=self.sigmoid(self.Z3)
        return self.A3
    def back_prop(self,x,y):
        self.forward_prop(x)
        m=x.shape[1]
        self.dZ3=self.A3-y
        self.dW3=np.dot(self.dZ3,self.A2.T)/m
        self.db3=np.sum(self.dZ3,axis=1,keepdims=True)/m
        self.dZ2=np.dot(self.W3.T,self.dZ3)*(1-self.A2**2)
        self.dW2=np.dot(self.dZ2,self.A1.T)/m
        self.db2=np.sum(self.dZ2,axis=1,keepdims=True)/m
        self.dZ1=np.dot(self.W2.T,self.dZ2)*(1-self.A1**2)
        self.dW1=np.dot(self.dZ1,x.T)/m
        self.db1=np.sum(self.dZ1,keepdims=True)/m
    def fit(self,x,y,epochs=100,learning_rate=0.01,plot=True,disp_loss=False):
        np.random.seed(4)
        self.W1=np.random.rand(128,x.shape[0])
        self.b1=np.zeros((128,1))
        self.W2=np.random.randn(64,128)
        self.b2=np.zeros((64,1))
        self.W3=np.random.randn(1,64)
        self.b3=np.zeros((1,1))
        m=x.shape[1]
        loss=[]
        for i in range(epochs):
            self.back_prop(x,y)
            self.W1-=learning_rate*self.dW1
            self.b1-=learning_rate*self.db1
            self.W2-=learning_rate*self.dW2
            self.b2-=learning_rate*self.db2
            self.W3-=learning_rate*self.dW3
            self.b3-=learning_rate*self.db3
            logprobs=y*np.log(self.A3)+(1-y)*np.log(1-self.A3)
            cost=-(np.sum(logprobs))/m
            loss.append(cost)
        e=np.arange(1,epochs+1)
        if plot:
            plt.plot(e,loss)
            plt.show()
        if disp_loss:
            print(loss)
    def predict(self,x):
        y=np.where(self.forward_prop(x)>=0.5,1,0)
        return y

F=FNN()
F.fit(x_train,y_train)
y_pred=F.predict(x_train)
print('Predicted Label:',y_pred)
print('True Label:',y_train)
acc=accuracy_score(y_train,y_pred)
print(acc)

Output
Loss Plot

Based on your screenshot, it's quite clear that the accuracy isn't 0.0 since the first two predictions match the true labels. So something must be wrong with how the accuracy is calculated.

If you go to sklearn's documentation, you'll see that accuracy_score requires 1-d arrays while it seems that you are feeding it 2-d arrays. My guess is that right now, it compares the elements of your arrays and checks if they are identical. Because you feed a 2-d array, it checks whether all predictions match, which unless you are perfectly correct, will always yield you 0.0.

Doing the following should fix your issue:

acc=accuracy_score(y_train[0], y_pred[0])


Prediction interval for very small dataset Does it make sense to calculate prediction interval for very small dataset (about 60 samples)? I know it's easy for linear regression. But it seems like for tree-based methods or non-linear methods it doesn't make any sense. Am I wrong?

In the case of quantile modeling, it makes sense as long as the quantile model makes sense. If you are able to build a good quantile model with 60 observations, then your estimates of intervals between quantiles should also be good. Keep in mind how errors will propagate when building intervals:

The bias in the width of the interval will be the sum of the biases in the quantile estimates; they might balance out, or they might make things worse.
The variance of the width of the interval will also be the sum of the variances of the quantile estimates. This could be a pretty big number, knowing that with 60 observations the "true" model variance will be large and hard to estimate.

Outside of that, it all depends on the model. Common statistical models tend to have explicit expressions for prediction intervals. In MCMC-based inference they can be inferred from the sampled values. In these cases, 60 is as good as 6,000 as long as the model works. As above, keep in mind that error propagation is not always easy to control. If these intervals are needed for serious applications, it's a good idea to generate fake data and run a sensitivity analysis.

The most generic solution is to obtaining a prediction interval is resampling. With just 60 observations, you can run full Leave-One-Out cross-validation (LOOCV). Or you can do bootstrap resampling. It might be a good idea, or the variance of the procedure will be through the roof. Again, simulation-based sensitivity analysis will help you decide.


Majority voting in scikit-learn Random forest My main concern is that i need to understand that how does the random forest do majority voting in scikit learn source code. I did not find that specific code in source code of RandomForest. if anybody knows, then kindly explain that. Thanks

To quote from the documentation:

  The predicted class probabilities of an input sample are computed as
          the mean predicted class probabilities of the trees in the forest. The
          class probability of a single tree is the fraction of samples of the same
          class in a leaf.

The predictions are then simply the class with the highest average class probability.

You can find the source code here.

Note that this is different from classical majority voting which is usually understood to be the most common class prediction among trees whereas here the voting happens on the class probability level.

The source code for regression is here:

  The predicted regression target of an input sample is computed as the
          mean predicted regression targets of the trees in the forest.


Create a new column in Pandas Dataframe based on the 'NaN' values in another column What is the most efficient way to create a new column based off of nan values in a separate column (considering the dataframe is very large)

1 2 3
4 5 NaN
7 8 9
3 2 NaN
5 6 NaN

Should give

1 2 3    0
4 5 NaN  1
7 8 9    0
3 2 NaN  1
5 6 NaN  1

EDIT:

What if it were based on 2 columns? Like:

1 2   3    0
4 NaN 1    1
7 8   9    0
3 2   NaN  1
5 NaN 2    1

Try this -

Column is your column with NaN values and column B is the new column

df['B'] = df['A'].isnull()*1


Computing confidence interval of average output from random forest in scikit learn I am computing confidence intervals for random forests using the package available here: http://contrib.scikit-learn.org/forest-confidence-interval/auto_examples/plot_mpg.html#sphx-glr-auto-examples-plot-mpg-py

I want to plot the average MPG and associated confidence interval. I can compute the average MPG but how do I compute the associated confidence interval?

In the link that you post there is a portion that describes how to compute and plot errorbars. These bars are the confidence intervals. If you prefer to have this plotted as bands you could follow this example


How should one deal with null values in time stamp data types? I have a data set where certain features are time stamps for certain important events like time stamp of when a user logged in etc. How should I deal with null values in such cases. Should one just replace null values with current or most recent time stamp. Are there any standard techniques followed ?

Leave those time stamps null in order to indicate that you do not possess the time of those events. Everything else depends on the logic of your application.


How to determine the weights of features when calculating the weighted average? I have a dataset of Tweets along with their Retweet Counts and Favorite Counts. After extracting features from the Tweets, I am training a model to predict the Retweet Counts and Favorite Counts. I want to calculate a value called "Popularity" by taking the weighted average of Retweet Count and Favorite Count for each Tweet. For this, I need score values (or weights) for Retweet Count and Favorite Count. How do I determine these weights?
I am writing a research paper and need to justify why I assigned those weights for Retweet Count and Favorite Count.

Retweet and Favourite are totally different features and although they are correlated with the popularity of the tweet, the exact correlation will be subjective to the training dataset you are using.

One way to come up with the weights is: Information Gain

This paper tries to come up with a weighted average using the user and tweet features for the prediction of retweet counts for a given tweet: http://www.nlpr.ia.ac.cn/2012papers/gjhy/gh154.pdf

Steps by step procedure:
1) Give every tweet a popularity score between 0 to 5 (You have to label your training data)
2) Calculate information gain for different weights of Retweet and Favourite count (refer: https://homes.cs.washington.edu/~shapiro/EE596/notes/InfoGain.pdf)


What is the best way to treat datetime in the preprocessing step of machine learning I have two datetime columns in my dataset. What I have done so far I have extracted year, month, dayofweek and hourofday from these columns.
So as you expect they will be something like this:
2015-5-5 08:21:20   ----> 2015   5   5   8

So my question is that what is the best way to normalize these numbers. Because I think that year or other numbers will dominate my machine learning model.
I have not found any article regarding this, all explaining till this point that we convert them to year month ...
Thanks.

You already have a good beginning. Transforming data into 4 columns. Year, month, day and hour. Now These 4 are all categoricals, you can than just apply one hot Encoding. Than no Domination will happen.


Initializing a recurrent neural network (RNN) in Keras When we create an RNN in Keras, does it learn an initial hidden layer (ie. $h_0$, like a bias term)? I'm having trouble figuring this out, but I have looked here.

Can anyone highlight a part in the code that shows an initial hidden layer is being learned? Or is it just randomly initializing the hidden layer?

EDIT: The code is here , I'm just having trouble identifying where the hidden layer is initialized and what it is initialized to.

In line 809, you can find that SimpleRNNCell takes a bias_initializer as an argument to the constructor function and it defaults to zeros.


loss/val_loss are decreasing but accuracies are the same in LSTM! I am trying to train a LSTM model, but the problem is that the loss and val_loss are decreasing from 12 and 5 to less than 0.01, but the training set acc = 0.024 and validation set acc = 0.0000e+00 and they remain constant during the training. Why this happening and how can I fix it?

I think this is because your targets y are continuous instead of binary. Therefore, either ignore the accuracy report, or binarize your targets if applicable.

I assumed you are using Keras.
When you use metrics=['accuracy'], this is what happens under the hood:

if metric in ('accuracy', 'acc'):
   metric_fn = metrics_module.binary_accuracy

where

def binary_accuracy(y_true, y_pred):
    return K.mean(K.equal(y_true, K.round(y_pred)), axis=-1)

In the case of continuous targets, only those y_true that are exactly 0 or exactly 1 will be equal to model prediction K.round(y_pred)). Therefore, accuracy cannot be used for continuous targets.

Here is a code that demonstrates this issue:

from keras import Sequential
from keras.layers import LSTM, Dense
import numpy as np

# Parameters
N = 1000
halfN = int(N/2)
seq_len = 10
dimension = 2
lstm_units = 3

# Data
np.random.seed(123)  # to generate the same numbers

X_zero = np.random.normal(0, 1, size=(halfN, seq_len, dimension))
y_binary_zero = np.zeros((halfN, 1))  # output is only 0
y_continuous_zero = np.random.randint(0, 50, (halfN, 1)) / 100  # output is in [0, 0.5]

X_one = np.random.normal(1, 1, size=(halfN, seq_len, dimension))
y_binary_one = np.ones((halfN, 1))  # output is only 1
y_continuous_one = 0.5 + np.random.randint(0, 50, (halfN, 1)) / 100  # output is in [0.5, 1.0]

p = np.random.permutation(N)  # to shuffle zero and one classes
X = np.concatenate((X_zero, X_one))[p]
y_binary = np.concatenate((y_binary_zero, y_binary_one))[p]
y_continuous = np.concatenate((y_continuous_zero, y_continuous_one))[p]

# Build model
model = Sequential()
model.add(LSTM(lstm_units, input_shape=(None, dimension)))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
print(model.summary())

# Fit model
# fit using binary outputs
print('-----------------------Binary---------------------')
model.fit(X, y_binary, batch_size=32, epochs=10)
# fit using continuous outputs
print('-----------------------Continuous---------------------')
model.fit(X, y_continuous, batch_size=32, epochs=10)

which outputs

...
-----------------------Binary---------------------
...
1000/1000 [==============================] - 0s 122us/step - loss: 0.3989 - acc: 0.9500
-----------------------Continuous---------------------
...
1000/1000 [==============================] - 0s 135us/step - loss: 0.5759 - acc: 0.0050


How to visualize categorical data over time? I'm visualizing results from lab tests. Most of these results are numerical and therefore easily communicated using line graphs, where the x-axis is time and the y-axis is determined by the range of possible values, like [0, 100].

EDIT: To be clear, I'm not measuring frequency. I'm displaying an individual's lab test results for one specific test (like cholesterol) over time. Since most people don't get more than 1 or 2 lab tests a year, most people's graphs will have less than a dozen (x,y) values.

I'm less sure how to visualize this information when the y-axis values are categorical:

Sometimes the possible values are themselves ranges. A test value will be something like 2-6, meaning that the value falls between 2 and 6 — and that's all we need to know. Would it make sense to graph these values with something like error bars? Or would it make more sense to graph the average of the range (in this case, 4) as a regular line chart?
The other types involve categories which are non-numerical. Sometimes these are ordinal. In the ordinal case the categories might be something like [none, few, moderate, many]. Does it make sense to just assign an index to each [0, 1, 2, 3] and use a standard line graph? If not, what would make more sense?
Lastly, non-numerical, nominal types. These are categories with no clear rank hierarchy, like colors: [yellow, orange, red]. In this case, I think it might be misleading to use a standard line graph, because it would imply that certain colors are "higher" or "better" than others. What would be appropriate?

I'll give it a go, but a lot depends on how you want to communicate your results.

try to keep the buckets, and add counts to them, so you end up with something like:

date bucket count
2015-09-11 2-6 6
2015-09-12 2-6 9
2015-09-13 2-6 2
2015-09-11 10-16 9
2015-09-12 10-16 15
2015-09-13 10-16 4

This you can communicate either via a line chart, but I would go with side by side bar charts per period, maybe aggregated to week or month depending your timeline. If you have many ranges, splitting them in different graphs.

Don't index the ordinal values. The reader of the graph should not look somewhere else to figure out what the numbers mean. Just use counts with the ordinals and use line charts or bar charts, summary tables or a cross(table)
Count the occurences and use side-by-side or stacked bar charts and color the bars according to the colors you have. But human perception will always give a higher bar / line / area more weight than a smaller / lower one. Or just represent it as a table. That way you don't give any opinion except that some colors might occur more frequently than other.


Prediction problem across a wide space of Clash Royale card games I have assembled a database of Clash Royale games in an attempt to understand the outcomes of various match-ups.  The game is composed of an 8 card deck drawn from 102 cards.  As you can see from the Cnr, it's a very wide space with many possibilities.  Similarly, this deck will be opposed by a wide and diverse space.
I have been using essentially a 206 element wide table.  One of the variables relates to the relative strength of the players; one hundred and two variables are indicators for the "home deck"; one hundred and two variables are indicators for the "away deck"; and the final element is an outcome variable that is the binary response variable of interest.
I have utilized an xgboost model to predict the outcome of the opposing decks, but I understand that it is not that useful for extrapolation where card combinations have not actually been played in the data.  It seems to me that a logistic approach would be better, but the challenge i am faced with is that the interactions are highly critical, but also explode into too many features to manage.
As additional information, I have used segmentation to "group" the decks into commonly utilized "metas" with good success.  Per my note above, however, this does not help in the identification of "new metas" that might be worth exploring.
I am interested in suggested approaches to solving this problem.  And if you care to see some of my Clash Royale data, I host a website for my clan at The6ixclan.ca

Given the domain is a digital card game, deck value and game play strategy can be simulated. The wide space of options can filled in with synthetic data.
Classic machine learning algorithms, such as logistic regression and boosted trees, will have limited success given the sequential nature of the problem. It would be more useful to frame it as reinforcement learning problem which has built-in modeling concepts of sequential decision and "exploration" of different strategies.


nltk's stopwords returns "TypeError: argument of type 'LazyCorpusLoader' is not iterable" While trying to remove stopwords using the nltk package, the following error occurred:

from tqdm import tqdm
import nltk
from nltk.corpus import stopwords

preprocessed_reviews = []
for sentance in tqdm(final["Text"].values):
    sentance = ' '.join(e.lower() for e in sentance.split() if e.lower() not in stopwords)

TypeError
  Traceback (most recent call last)
<ipython-input-136-ac5c19fafd9c> in <module>()
---> 7     sentance = ' '.join(e.lower() for e in sentance.split() if e.lower() not in stopwords)
     8     preprocessed_reviews.append(sentance.strip())
     9

TypeError: argument of type 'LazyCorpusLoader' is not iterable

There are a couple of items that could be improved in your code:

nltk.corpus.stopwords is a nltk.corpus.util.LazyCorpusLoader. You might want stopwords.words('english'), a list of stop English words.
It can cause bugs to update a variable as you iterate through it, for example sentance in your code.
In your code preprocessed_reviews is not being updated.
You might want to tokenize instead of str.split().

Here is a revision version:

import nltk
from nltk.corpus import stopwords
from tqdm import tqdm

reviews_raw = ('The quick brown fox jumps over the lazy dog', "I have been doing what I should not been doing")
reviews_processed = []
for sentence in tqdm(reviews_raw):
    reviews_processed.append(' '.join(token.lower() for token in nltk.word_tokenize(sentence) if token.lower() not in stopwords.words('english')))

assert reviews_processed == ['quick brown fox jumps lazy dog', '']


In a neural network with tensor input X, it seems there are times when it will never learn... Why? import numpy as np
import keras.models as km
import keras.layers as kl
import keras.optimizers as ko
import keras.losses as kloss

# This will cause no learning
np.random.seed(1692585618)

def f(x):
    a = x[0]* 3.141 + x[1]
    return a;

# Create a sample dataset
# Input is (*, 2)
# Output is (*, )
x_train=np.array([[1,2], [3,4], [5,6]])
y_train=np.array([f(x) for x in x_train])

# These are required by the shape of x_train and y_train
in_dim = x_train.shape[1]
out_dim = 1

model = km.Sequential()
model.add(kl.Dense(units=3, activation='relu', input_shape=(in_dim,)))
model.add(kl.Dense(units=out_dim, activation='relu'))
model.compile(
  loss=kloss.mean_squared_error
, optimizer=ko.Adam(lr=0.1)
)

model.fit(x_train, y_train, epochs=500, batch_size=1, verbose=True)

Output:

Epoch 1/50
3/3 [==============================] - 1s 221ms/step - loss: 225.9046
Epoch 2/50
3/3 [==============================] - 0s 3ms/step - loss: 225.9046
Epoch 3/50
3/3 [==============================] - 0s 3ms/step - loss: 225.9046
Epoch 4/50
3/3 [==============================] - 0s 2ms/step - loss: 225.9046

...and so on (hundreds more)

In short

You are waaaaay undertraining. Increase the number of times you show the network your data. I am guessing training may take longer than you expect because typically networks train best with 0-mean data, which yours is not.
ReLU seems to cause problems with such a shallow network. Try increasing depth or using elu activation instead.
I confirmed having an activation in the last layer doesn't cause huge problems, but it is still a good idea of getting in the habit of knowing when you should and should not have an activation in the last layer.

Undertraining

I handled this by increasing the number of training examples by 10,000 times (you could increase the number of epochs instead but this results in better printing):

x_train=np.array([[1,2], [3,4], [5,6]] * 10000)
y_train=np.array([f(x) for x in x_train])[:, np.newaxis]

Problems with ReLU

The problems with ReLU can be handled in one of two ways, increase the number of layers when using ReLU, or use a different activation such as elu. Both trained just fine for me:

model = km.Sequential()
model.add(kl.Dense(units=3, activation='relu', input_shape=(in_dim,)))
model.add(kl.Dense(units=3, activation='relu'))
model.add(kl.Dense(units=out_dim))
model.compile(
  loss=kloss.mean_squared_error, optimizer=ko.Adam(lr=0.1)
)

or

model = km.Sequential()
model.add(kl.Dense(units=3, activation='elu', input_shape=(in_dim,)))
model.add(kl.Dense(units=out_dim))
model.compile(
  loss=kloss.mean_squared_error, optimizer=ko.Adam(lr=0.1)
)

Full working code

Below shows the code with elu, you can swap the block out for the ReLU version (shown above) instead and it prints very similar values.

import numpy as np
import keras.models as km
import keras.layers as kl
import keras.optimizers as ko
import keras.losses as kloss

# This will cause no learning
np.random.seed(1692585618)

def f(x):
    a = x[0]* 3.141 + x[1]
    return a;

# Create a sample dataset
# Input is (*, 2)
# Output is (*, )
x_train=np.array([[1,2], [3,4], [5,6]] * 10000)
y_train=np.array([f(x) for x in x_train])[:, np.newaxis]

# These are required by the shape of x_train and y_train
in_dim = x_train.shape[1]
out_dim = 1

model = km.Sequential()
model.add(kl.Dense(units=3, activation='elu', input_shape=(in_dim,)))
model.add(kl.Dense(units=out_dim))
model.compile(
  loss=kloss.mean_squared_error, optimizer=ko.Adam(lr=0.1)
)

model.fit(x_train, y_train, epochs=3, verbose=True)
print('predicted: {}'.format(model.predict(x_train)[:3, 0]))
print('actual   : {}'.format(y_train[:3, 0]))

prints

Epoch 1/3
30000/30000 [==============================] - 1s 29us/step - loss: 3.0553
Epoch 2/3
30000/30000 [==============================] - 1s 20us/step - loss: 5.0199e-06
Epoch 3/3
30000/30000 [==============================] - 1s 20us/step - loss: 5.4414e-06
predicted: [ 5.1426897 13.420667  21.707573 ]
actual   : [ 5.141 13.423 21.705]


Dataset for Named Entity Recognition on Informal Text I'm currently searching for labeled datasets to train a model to extract named entities from informal text (something similar to tweets). Because capitalization and grammar are often lacking in the documents in my dataset, I'm looking for out of domain data that's a bit more "informal" than the news article and journal entries that many of today's state of the art named entity recognition systems are trained on.

Any recommendations? So far I've only been able to locate 50k tokens from twitter published here.

As I understand it, these are the properties that you're seeking in a sample dataset:

Text data
It should be informal, i.e. have typos, slang, and basically something not professionally edited
Something other than Twitter (I don't blame you, Twitter is a useful yet way overused example datasource in text mining)

Here are some recommendations:

Emails from the SpamAssassin corpus -- note that both "ham" (non-spam) and spam datasets are available
microblogPCU data set from UCI, which is data scraped from the microblogs of Sina Weibo users -- note, the raw text data is a mix of Chinese and English (you could perform machine translation of the Chinese, filter to only English, or use it as-is)
Amazon Commerce reviews dataset from UCI
Within the bag-o-words dataset, try using the Enron emails
The Twenty Newsgroups dataset
This nice collection of SMS spam
You can always scrape (extract) your own text data from the Internet; I'm not sure which language or statistical package you're using, but XPath-based packages are available in R (rvest, scrapeR, etc) and Python to accomplish this


Quality check for preprocessing of Text data I have developed a pipeline for text data preprocessing with different clean up techniques like Stemming , Lemmatization, Stop words removal etc. But now the ask from the business team is to quantify the quality of the preprocessing steps (or, the text data it produced). How can we develop some metrics to evaluate the preprocessing quality of text data?

Evaluating any task consists in defining the task formally so that there is a way to define what is a correct output as objectively as possible. For example a good Machine Translation system produces a good translation if it has the same meaning as the input sentence and is grammatically correct in the target language.
Assuming that this task of preprocessing is formally defined, then the evaluation should measure how "correctly preprocessed" is the output:

are the stem and lemma always correct?
are the stop words and only the stop words removed?
Etc.

Usually one would build a test set, manually add the correct output and then compare the system output against this gold standard.
However "preprocessing" is generally not considered a task by itself, because by definition it's a step for another task. Importantly, the steps of preprocessing depend on the other task, they are not always the same. For example stop words removal makes sense only for tasks based on distributional semantics, i.e. related to the topic. Preprocessing may also include steps which depend on the volume of data.


What is the difference between fit() and fit_generator() in Keras? What is the difference between fit() and fit_generator() in Keras?

When should I use fit() vs fit_generator()?

In keras, fit() is much similar to sklearn's fit method, where you pass array of features as x values and target as y values. You pass your whole dataset at once in fit method. Also, use it if you can load whole data into your memory (small dataset).

In fit_generator(), you don't pass the x and y directly, instead they come from a generator. As it is written in keras documentation, generator is used when you want to avoid duplicate data when using multiprocessing. This is for practical purpose, when you have large dataset.

Here is a link to understand more about this-

A thing you should know about Keras if you plan to train a deep learning model on a large dataset

For reference you can check this book- https://github.com/hktxt/bookshelf/blob/master/Computer%20Science/Deep%20Learning%20with%20Python%2C%20Fran%C3%A7ois%20Chollet.pdf


Can I do the below problem with ML? My dataset has 3 fields ColumnX, ColumnA, Column B as below.

Column X, Column_A ,      Column_B

AA, 1     ,        KEYWORD1 KEYWORD2 KEYWORD3

AA, 2    ,         KEYWORD4 KEYWORD5 KEYWORD6

XX, 3 , KEYWORD3 KEYWORD7

XX, 4, KEYWORD6 KEYWORD8

YY, 5, KEYWORD9

I removed the stop words in Column_B and kept the keywords.

So here Can I train a model which can classify the data with Column_X and Column_B and given an input from user which matches any keyword in Column B , returns Column_A

Sample output:

user gives an input : KEYWORD9.

my output should show ColumnX: YY and ColumnA: 5

I know it can be done with basic python but I want to use ML here

Try an LSTM Encoder-decoder model. The encoder takes Column_B as input and Decoder gives Column X, Column_A as output. Or create a two machine learning models one for each column prediction.


What is the difference between dynamic programming and Q-learning? What is the difference between the DP-based algorithm and Q-learning?

Both Q learning and Value Iteration (a DP technique) use similar update rules based on Bellman optimality equations:

$$v_*(s) = \text{max}_{a}\sum_{s',r} p(s',r|s,a)(r + \gamma v_*(s'))$$

$$q_*(s,a) = \sum_{s',r} p(s',r|s,a)(r + \gamma\text{max}_{a'}q_*(s',a'))$$

The main difference is that DP uses an explicit model. DP requires that you know $p(s',r|s,a)$. The update rule for DP is literally the first equation turned into an update rule:

$$v_{k+1}(s) = \text{max}_{a}\sum_{s',r} p(s',r|s,a)(r + \gamma v_{k}(s'))$$

In comparison, Q learning does not require knowing $p(s',r|s,a)$, as it is based on sampling from experience. The update rule is modified to be based on samples of observed data, which have the same values in expectation, as if you had used $p(s',r|s,a)$, but without knowing it:

$$Q_{k+1}(S_t,A_t) = Q_{k}(S_t,A_t) + \alpha(R_{t+1} + \gamma\text{max}_{a'}Q_k(S_{t+1},a') - Q_{k}(S_t,A_t))$$

This is still an important difference even when both systems are run on an internal model/simulation. DP does not need to simulate anything, it iterates over the model directly. Whilst Q learning needs to work with sampled transitions - they might be simulated, but this is not the same as iterating over all states as in DP. It can often be the case that it is easier to simulate the environment than to calculate $p(s',r|s,a)$ for the full model.

Which should you choose:

Choose Dynamic Programming when you have access to the full state transition and reward model in a simple form (i.e. you have $p(s',r|s,a)$ or equivalent), and the state space is not too large - ideally the number of states is small enough to fit in memory. However, there are ways to use DP when you have a larger state space, by modifying which states it processes. So you still can use DP on larger problems if you really want to.
Choose Q learning when you don't have a model, or when the state space is too large to iterate over in full.


How to apply dataset balancing techniques whilst using Pipeline in Sklearn? I am new to Machine Learning and trying to construct machine learning models that adhere to good practice and not susceptible to biases. I have decided to use Sklearn's Pipeline class to ensure that my model is not prone to data leakage. I am building an ML model that attempts to predict the trend Buy, Hold, Sell for the next hour.

However, my multi-class classification dataset is extremely imbalanced. Whilst it is not necessarily a concern that the test set is imbalanced, it is important that the train set is balanced. However, I have researched properly but I cannot find an answer as to where this dataset rebalancing step should be conducted. Should it be done before scaling or after? Should it be done train/test split or after?

I cannot figure out where this crucial step should be done. For simplicity's sake, I will not be using SMOTE, but rather random minority upsampling. Any answer would be greatly appreciated.

My code is as follows:

#All necessary packages have already been imported

x = df['MACD', 'MFI', 'ROC', 'RSI', 'Ultimate Oscillator', 'Williams %R', 'Awesome Oscillator', 'KAMA',
    'Stochastic Oscillator', 'TSI', 'Volume Accumulator', 'ADI', 'CMF', 'EoM', 'FI', 'VPT','ADX','ADX Negative',
    'ADX Positive', 'EMA', 'CRA']

y = df['Label']

X_train, X_test, y_train, y_test = train_test_split(x, y, test_size = 0.3, random_state = 0)

pipe = Pipeline([('sc', StandardScaler()),
                 ('svc', SVC(decision_function_shape = 'ovr'))])

candidate_parameters = [{'C': [0.0001, 0.001, 0.01, 0.1, 1, 2, 3],
                        'gamma': [0.0001, 0.001, 0.01, 0.1, 1, 2, 3], 'kernel': ['poly']
                        }]

clf = GridSearchCV(estimator = pipe, param_grid = candidate_parameters, cv = 5, n_jobs = -1)

clf.fit(X_train, y_train)```

Should it be done before scaling or after?

According to this post, you should scale the data first :

  My thought would be to standardize (normalizing is typically using the min and max values not mean and standard deviation) the data first and then over-sample if that is what you are thinking in terms of balancing. I say this because you will want to use that same mean/std dev of the original set when you standardize new data so that it mirrors the training set that was used.

Should it be done train/test split or after?

According to this post :

  Sampling should always be done on train dataset. If you are using python, scikit-learn has some really cool packages to help you with this. Random sampling is a very bad option for splitting. Try stratified sampling. This splits your class proportionally between training and test set.

You should also set the class_weight parameter of sklearn.svm.SVC :

  Set the parameter C of class i to class_weight[i]*C for SVC. If not given, all classes are supposed to have weight one. The “balanced” mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as n_samples / (n_classes * np.bincount(y)).

Hope this helps.


How to grow a list of related words based on initial keywords? I recently saw a cool feature that was once available in Google Sheets: you start by writing a few related keywords in consecutive cells, say: "blue", "green", "yellow", and it automatically generates similar keywords (in this case, other colors). See more examples in this YouTube video.

I would like to reproduce this in my own program. I'm thinking of using Freebase, and it would work like this intuitively:

Retrieve the list of given words in Freebase;
Find their "common denominator(s)" and construct a distance metric based on this;
Rank other concepts based on their "distance" to the original keywords;
Display the next closest concepts.

As I'm not familiar with this area, my questions are:

Is there a better way to do this?
What tools are available for each step?

The word2vec algorithm may be a good way to retrieve more elements for a list of similar words. It is an unsupervised "deep learning" algorithm that has previously been demonstrated with Wikipedia-based training data (helper scripts are provided on the Google code page).

There are currently C and Python implementations. This tutorial by Radim Řehůřek, the author of the Gensim topic modelling library, is an excellent place to start.

The "single topic" demonstration on the tutorial is a good example of retreiving similar words to a single term (try searching on 'red' or 'yellow'). It should be possible to extend this technique to find the words that have the greatest overall similarity to a set of input words.


Difference between class_weight and loss_weights in Keras Keras has parameters class_weight used in fit() function and loss_weights used in compile() function. From the explanation(Docs) and what I understand, it seems that both are identical, as they are used to indicate the importance of each class. However, what is the difference between the two? And what are the scenarios in which they can be appropriately used?

From Keras Team at GitHub:

  loss_weights parameter on compile is used to define how much each of
  your model output loss contributes to the final loss value ie. it
  weighs the model output losses. You could have a model with 2 outputs
  where one is the primary output and the other auxiliary. eg. 1. *
  primary + 0.3 * auxiliary. The default values for loss weights is 1.

  class_weight parameter on fit is used to weigh the importance of each
  sample based on the class they belong to, during training. This is
  typically used when you have an uneven distribution of samples per
  class.


How can we scale up the number of classes for deep learning after training a model? I have trained a deep learning model for a face recognition application. The model has been trained on the existing subjects (or call it classes) in the gallery. Now consider the real world problem, where the gallery size is dynamic as new subjects are to be enrolled, how can deep learning solve those type of problems? I have tried transfer learning, are there any other techniques?

Face recognition from images is still an open area of research. Many different techniques have been tried.

One approach is not to directly learn the identities of the people as classes, but to learn a multi-dimensional embedding for images of faces (e.g. 128 dimension vector) and train the network to make this embedding value closer for two different images of the same person than for two images of different people*.

You can use the network to generate feature vectors for each person who is registered, and store those values. Later when you want to identify someone, find the closest one (according to norm of the difference) and if the distance is less than a certain cutoff you can consider it to be a match.

To perform the training for the embeddings with this goal in mind, you can use a loss function called triplet loss, which is trained on 3 forward passes at a time. The network calculates feature vectors for an anchor image, a positive match (i.e. another image of the same face, so you need at least 2 for each person, and they should be taken from different contexts) and a negative match. Then it measures whether the distance between features from same person is less than distance between different people. The loss function to express that looks like:

$$\text{max}(0, |A - P|^2 - |A - N|^2 + \alpha)$$

. . . where $A$ is feature vector for the anchor image, $P$ for positive match image, and $N$ for negative match image. The parameter $\alpha$ is a hyper-parameter which creates a "margin" that you reward the model for maintaining between different identities. The idea being that if an image of different person is clearly further away in the feature space than a cutoff, the loss on that training example is zero. The model does not get rewarded for very large differences, just having "consistently enough distance".

The tricky thing with triplet loss is picking which samples to use for training. Purely random samples could make the network's task too easy. You need it to focus a bit on the more difficult cases to prevent mistaken identity. One way of doing this is to pick preferentially from closer faces, e.g. using k-means clusters and picking negative cases within own cluster.

The model can be trained on large numbers of faces, not necessarily the ones you will later want to classify. The point is to learn a feature vector that emphasises important differences between faces (and ideally ignores other variations such as lighting, background, pose). It's possible to start with any image classifier and adapt it for this task, including your original face classifier (just remove the final classifier layer and work with one of the later fully-connected layers as your output).

  How can we scale up the number of classes for deep learning after training a model?

This approach may not work well in all cases covered by your question title. I have recommended it for face recognition based on a Coursera course on CNNs where a few lectures and coding assignment are about face recognition specifically.

In some cases you may be better off re-building and re-training the model with new training set data, augmented by examples of your new class - you might start by fine-tuning only the last few layers.

* In fact this will be the case in many CNN classifiers, but is more likely to be reliably linked to a person's identity if you have trained the network explicitly for distinguishing between identities in general (as opposed to just classify specific identities). The triplet loss approach is one way to train for useful representations, and makes the goal of disambiguating identity explicit.


Apply Labeled LDA on large data I'm using a dataset contains about 1.5M document. Each document comes with some keywords describing the topics of this document(Thus multi-labelled). Each document belongs to some authors(not just one author for a document). I wanted to find out the topics interested by each author by looking at documents they write. I'm currently looking an LDA variation (labeled-LDA proposed by D Ramaga: https://www.aclweb.org/anthology/D/D09/D09-1026.pdf .). I'm using all the documents in my dataset to train a model and using the model to predict the labels of an author(here I use the aggregation of all documents belonging to an author to represent him, treat authors like documents).

I'm using python and I found some implementation available on Github:
https://github.com/JoeZJH/Labeled-LDA-Python
https://github.com/shuyo/iir/blob/master/lda/llda.py

I tried the first one and tested it on a small data. It worked but when it came to a big data, the memory required grew a lot (For initialisation, this algorithm generalizes a zero matrix of size #labels times #terms/#vocabulary). I think I need to find a way to reduce the dimension of my corpus. I thought about directly reducing the size of my vocabulary, but it will reduce the length of my texts. And LDA does not work for short text. I’m afraid that if I do this, my texts will not be enough and some information will be lost in the process.

So here are my questions:

Is my method theoretically correct? (I will appreciate it if someone can propose other models suitable for this task)
If there is any method to reduce the dimension while keeping as much information as possible. For instance, is TF-IDF an option for the reduction?

I do appreciate it if you can give some advice on those two questions.

You matrix will be sparse, so the memory usage will be inefficient. Try to look on LSH - locality sensitive hashing for clustering.

https://en.wikipedia.org/wiki/Locality-sensitive_hashing


Can we define a data partitioning in K clusters, by cutting the branches of the tree at some levels in the tree below the root node? Assume we have a dendogram (hierarchical clusterisation tree), can we define a data partitioning  in K clusters, by cutting the branches of the tree at some levels in the tree below the root node?

Yes absolutely, in fact that's exactly how one obtains actual clusters with hierarchical clustering.
With hierarchical clustering it's possible to decide the number of clusters (i.e. the level of "precision") after obtaining the tree (dendogram).


Predict output sequence one at a time with feedback I would like to solve the following classification problem: Given an input sequence and zero or more initial terms of the true output, predict the next term in the output sequence.

For example, my training set is a large corpus of question and answer pairs. Now I would like to predict the response to "How are you?" one word at a time.

Some reasonable responses might be "I'm fine.", "OK.", "Very well, thanks." etc. So the predicted distribution over possible first words would include the first words in those examples, among others.

But now if I see the first word is "Very", I would like my prediction for the second word to reflect that "OK" is now less likely, because "Very OK" is not a common response.

And I would like to repeat this process of predicting the next word given the input sentence and every word so far in the output sentence.

One approach might be to train a stateful RNN on examples like How are you<END> I am fine<END>. But if I understand correctly this would also learn to predict words in the "How are you" part, which I think might detract from the goal.

Maybe separate layers for the input and partial output that are then merged into a decoder layer?

I assume, that your data looks somehow like this (a pair of question and answer)

Question,Answer
How are you?,I'm fine.
How are you?,OK.
How are you?,Very well, thanks.

I'd suggest to transform it in three features: Question  (remains the same),
InputSeq is the prefix of the answer generated in all possible lengths between zero and the number of words minus one.

Finally the NextWord is the expected result.

So for your initial data you ends with the following setup.

Question, InputSeq, NextWord
How are you,,I'm
How are you,I'm,fine
How are you,,OK
How are you,,Very
How are you,Very,well
How are you,Very well,thanks

The model takes Question and InputSeq   as features and predicts the NextWord.


LSTM Timeseries recursive prediction converge to same value I'm working on Timeseries sequence prediction using LSTM.
My goal is to use window of 25 past values in order to generate a prediction for the next 25 values. I'm doing that recursively:
I use 25 known values to predict the next value. Append that value as know value then shift the 25 values and predict the next one again until i have 25 new generated values (or more)

I'm using "Keras" to implement the RNN
Architecture:

regressor = Sequential()
regressor.add(LSTM(units = 50, return_sequences = True, input_shape = (X_train.shape[1], 1)))
regressor.add(Dropout(0.1))

regressor.add(LSTM(units = 50, return_sequences = True))
regressor.add(Dropout(0.1))

regressor.add(LSTM(units = 50))
regressor.add(Dropout(0.1))

regressor.add(Dense(units = 1))

regressor.compile(optimizer = 'rmsprop', loss = 'mean_squared_error')

regressor.fit(X_train, y_train, epochs = 10, batch_size = 32)

Problem:
Recursive prediction always converge to the some value no matter what sequence comes before.

For sure this is not what I want, I was expecting that the generated sequence will be different depending on what I have before and I'm wondering if someone have an idea about this behavior and how to avoid it. Maybe I'm doing something wrong ...

I tried different epochs number and didn't help much, actually more epochs made it worse. Changing Batch Size, Number of Units , Number of Layers , and window size didn't help too in avoiding this issue.

I'm using MinMaxScaler for the data.

If you want to use an LSTM for time series prediction, you have to try every possible parameter to find the best possible result. In order to check your model performance, you should also check your train/validation loss (this will probably show that 10 epochs is too few). Check your network architecture. Why do you use three layers? More layers won't mean better results. Try different optimizers with different learning rates. Finally, ask yourself why you are even using an LSTM for prediction? Have you tried other methods? LSTMs aren't magic, they won't have perfect prediction without any pattern to draw from, and you have pretty small samples for training.


Which is the "most properly working" Bert-Ner repository I am trying to find a repository in Github to get a Pytorch-reimplementation of the Bert model for NER task. So far, I found the following repos:

https://github.com/kamalkraj/BERT-NER
https://github.com/Louis-udm/NER_BERT_CRF
https://github.com/sberbank-ai/ner-bert

They are not exactly the same models (for example some use CRF as additionally), but for me the correctness is more important. When I check the issues, I saw that people were complaining about either mistakes in the evaluation or training procedures. So, my question, has anyone tried any of these repositories for Ner tasks before? If you know any other options, I'm open to hear.

While not NER specific, the go-to PyTorch implementation of BERT (and many other transformer-based language models) is HuggingFace's PyTorch Transformers. There is plenty of documentation to get you started.

They also have models which can directly be used for NER, such as BertForTokenClassification.

I am not sure if you were looking for something higher-level, but as the question doesn't specify, this is the answer I would give to someone looking to use BERT in PyTorch.


What's the best way to approach a regression problem with a lot of features (over 300), many of which are categorical? I have $20,000$ training examples of various candidate attributes (highest level of education, country, year of formal training completion, etc).

My output (prediction) will be for job_performance, which is measured between $700$ and $4,200$ (in my training data). Initially, I was thinking about putting together a fully connected neural network, but I attempted a similar problem in a Kaggle competition, but that didn't produce great results. What method(s) would you use to start?

Try doing a Principal component analysis to get the reduced feature set and then apply regression techniques in loop to see which one predicts better. SVM may take sometime to execute so better exclude it. Random Forest should be a good fit in this case.


How to group a unique column and find the mean value in a separate column in R My table of data looks like this image (but with many more lines):

I want to be able to find the mean age of each region, so the mean age for region SSC20790 and so on. I believe I can use the dplyr package?

Something like this for instance:

library(plyr)
ddply(data,'region',function(x) {mean(x$age)})


Deploying an LSTM Model I have trained and validated my LSTM and I would like to deploy it. So, I know that we can save and load the Sequential object of Keras (I am working with Keras as you can guess). I thus implemented a code using these functions.
But, I would like to know whether I must train my model with all the available data (training + test) or just on the training set as I did during my study.
Many tutorials talk about how to train a model but not so many are clear about how to deploy a model.
I would like to know what is common to do, knowing that I am doing Time Series Forecasting which is a specific problem.

Personally, I would side with deploy it as is. I.e. do not retrain on all the data.

Once you train on new data, then you have a new model. You have no idea how this model is going to react to unseen test data coming in from the big wide world. You can’t validate or test it, because you’ve used all that data up in training.

Who knows what might happen!

As you receive more data and get feedback on the model you can create new train/test sets, fine tune the model, then redeploy it as required.


Normalization set data There are several methods to normalize data, among them are:

min-max, z-score and scale decimal.

Can I use any one or with what criteria should I choose one of them?

No specific answer to your question, it all depends on which algorithm you are using or in other words how you will use the normalized data. Based on my experience I found that the zscore normalization performs the best, especially if you are using svm or nn.


Translating a Business Metric into an ML Accuracy Metric Let's say that I'm building a binary classifier for giving loans to consumers. But apart from the standard accuracy metrics I can use, I have a requirement from the business, which wants to be able to "auto-approve" 80% of the loans.

My gut instinct is that this should be 1 - False Positive Rate, but I'm not quite sure.

Any ideas?

The purpose of this algorithm appears to be about reducing cost. So a metric which includes the financial risk associated with defaults should be preferred over one that simply measures number of defaults.

A weighted false_positive * loss_on_default metric would seem to be ideal, where you set the probability cutoff of positive class (auto-approve) so that 80% of loans were approved regardless of absolute probability of the prediction. Conceptually this is similar to picking a point on the ROC curve and assessing the algorithm at that point.

This metric could not be used directly as an objective function, you'd probably use want something like logloss weighted per sample by the loss_on_default in order to train a model.

I did a quick search for similar metrics, and could not find anything (it is also not something I have done before, so do please test the idea carefully, it is mostly conjecture on my part). There are plenty of metrics in e.g. SciKit learn however that take an array sample_weight that does something similar - e.g. Sklearn's Zero-One loss is pretty close except it will include false negatives as well.


Anomaly detection in nominal big data I have to apply an anomaly detection algorithm on big data, the values of each column on my dataframe are nominal and vary over 10000 times, the algorithms I've found only accept numeric values, is there any way to transform this nominal values into numeric ones in a way that the algorithm will work?

I've used preprocessing.LabeledEncoder(), but then when I apply the algorithm it finds anomalies - the values that most differ from the mean it seems.

Are there any examples of an algorithm or another way to transform the data?

You would have to post more info about your data and maybe even some samples, but have you considered using tokenization for the data? It's from the NLP world and would allow you to assign a numeric value for each bit of you "nominal" data.


Eliminating rows in a dataframe based on specific conditions I have two columns in a dataframe namely 'origin' and 'destination'  which contains the names of different places. I need to remove the rows that don't contain 'PHX', 'JFK', 'NTU' in either of these columns. If atleast one among the 2 columns has one of these 3 places, the row can stay, else the row must be dropped altogether. Can you please help me to code this part ?

This snippet should do the work:

city = ['PHX', 'JFK', 'NTU']
colomn_to_exclude = df.apply(lambda row: (row['origin'] not in city) and (row['destination'] not in city))
new_df = df[~colomn_to_exclude]

the second line checks the lines where your exclusion condition is verified and the line after subset the dataframe accordingly.

You could also write something like

colomn_to_exclude = df.apply(lambda row: (row['origin'] not in city) and   (row['destination'] not in city))
df.drop(~colomn_to_exclude, axis=1, inplace=True)

and you won't have to make a copy of your dataframe


testing new data in model I have ensembled 3 algorithms as below,

estimators = []
model1 = MultinomialNB().fit(X,y)
estimators.append(('Naive Bayes', model1))
model2 = LinearSVC(random_state=0).fit(X,y)
estimators.append(('SVM', model2))
model3 = RandomForestClassifier(bootstrap=True).fit(X,y)
estimators.append(('Random Forest', model3))

# create the ensemble model
ensemble = VotingClassifier(estimators)
results = model_selection.cross_val_score(ensemble, X, y)
print(results.mean())

while passing new sentence to test the ensembling I have used,

predicted = results.predict(X_new_tfidf)

But I am getting error:

  AttributeError: 'numpy.ndarray' object has no attribute 'predict'

How can I fix/debug this?

Your line results = model_selection.cross_val_score(ensemble, X, y) just returns an array of scores for each run of the cross validation, so the error is telling you that it's just an array of numbers, not a model itself and has no defined method for fitting things.

To use predict, you need to call it on a model object, which I'm assuming you're going for the VotingClassifier. Your ensemble variable has a fit method, so you want to call

predicted = ensemble.predict(X_new_tfidf)


Choice of replacing missing values based on the data distribution I am building a classification model based on a relatively small dataset. I have some missing values on the different attributes that I have. I cannot afford deleting any of the record that has missing values so I want to replace them.

I made some general calculation to get some understand of the disruption of the data and to help me choose the value which would replace the missing values,

Assume that I have attribute A with the following:
mean = 121.68676278
std = 30.51562426
median = 117
mode =
min = 44
max = 199
[in all the calculations, I ignored the missing values]

If I were to choose between mean, median, or mode which one would be most suitable?

and there is something else which was very confusing for me, the std is very large and when I asked about it I was told that this could be normal based on the range of my data but I did not understand what that means?

I would not definitely recommend substituting missing values by mean or by median or mode. If you want to go through some techniques and get a glance at them, I would recommend going through this link and for imputation techniques this wiki page gives you a brief .
Do you think that there is a way to predict missing values from the other cells. If yes, apply a regression model on those variables and estimate the missing value. But remember this lacks variability as values fall on the regression line itself. There are methods like regression imputation which can add this variability component to the estimated value.
If you are unable to go anywhere from the previous step, then see how the values are distributed for the missing variable, substitute them according to that distribution by using a random function.
And if your unable to perform any of the above mentioned ones and want to go by mean, median. I cant really give my opinion as in this case they are nearer to each other. See what gives you the best predictable and decide between them.
Coming to your final question, Std. deviation just shows how far your values are falling away from mean. If your data has large range with good enough number of  points distributed at extremes, you would be expected to have high std. deviation.


Data Imbalance in Regression Tasks Having read a lot about class imbalance in classification tasks, I'd like to know what is the methodology for data imbalance in regression tasks.

Particularly,
- What is the procedure to check for it (for example in classification, we'd check if all classes have the same training examples)
- What are possible ways to solve it?

In a regression framework one would usually not speak about (class) imbalance. In this case it is about the distribution (mostly of the dependent variable). A distribution has several moments. The expected value (1st), variance (2nd), skewness (3rd) and kurtosis (4th). They all together define how a distribution looks like.

Generally, a normal distribution works well with regression. You can also test normality. As mentionned by others, you can plot the (empirical) probability density function. Another useful thing is a boxplot. And you can look at the moments.

R example (skewed distribution):

library(sn)
library(moments)

set.seed(12)
d1=rsn(n=10000, xi=1, omega=5, alpha=100, tau=0.5, dp=NULL)
plot(density(d1))
boxplot(d1)
skewness(d1) # 0 for normal
kurtosis(d1) # 3 for normal

This yields:

> skewness(d1) # 0 for normal
[1] 0.8130714
> kurtosis(d1) # 3 for normal
[1] 3.412369

The probability density function shows a "tail" to the right (positive skewness / right-skewed distribution).

Also the box plot indicates that there are some "unusually high" values in the distribution.

R example (normal distribution):

Same procedure for normal...

d2=rnorm(10000, 3, 2)
plot(density(d2))
boxplot(d2)
skewness(d2) # 0 for normal
kurtosis(d2) # 3 for normal

Density function and boxplot look much more "harmonic" in this case.

> skewness(d2) # 0 for normal
[1] 0.002732919
> kurtosis(d2) # 3 for normal
[1] 2.947005

What can you do with non-normal distributions?

With skewed/non-normal distributions you often get bad predictions from a regression model, especially where "unusually low/high" values are found. See this post for example. The solution often is to do some kind of linear transformation to the data. This is one reason why it often is a good idea to standardize data in predictive modelling. One can also try other linear transformations, e.g. taking logs. However, there is no "silver bullet" to adress the problem.

Another consequence of skewed data in linear regression can be heteroscedasticity, which leads to unreliable standard errors, p-values, and confidence bands. See this post for a discussion.


Predicting a final exam score from arbitrary set of practice exams I posed this question to math.stackexchange without response, and feel it is better suited here. I am writing an application to predict a final exam score given at least one tuple, where the tuple represents the results of a practice exam:

(score, exam_name, date_taken[optional])

Users input practice exam scores and, later, their official exam score (date required); not all users take all available practice exams. The dataset can be represented like this:

$$
\begin{bmatrix}
a & b & c & d & e & f\\
g\\
h & i & & j\\
k & l & m &&n
\end{bmatrix}
\begin{bmatrix}
(actual,date)\\
(actual,date)\\
(actual,date)\\
(null,date)
\end{bmatrix}
$$

Each row is a user. Each column in the first matrix is a particular practice exam; the second matrix is the score of the final exam (actual) and date taken. Each element in the first matrix is a tuple of the form described above.

Users complete an arbitrary combination of practice exams. The user represented by the 4th row has inputted four events (four practice exams), the date he plans to take the official exam, but not his actual score; as such, we want to provide him with a prediction.

What model(s) might be suited for this problem?

I have a hunch that naïve Bayes might be best because of the "incompleteness" of the matrix, and because I want to provide a prediction to the user after any input of data (i.e., user inputting a single tuple with no date, (512, mcat_24), should still receive a prediction). The feature of the problem most difficult to me is the fact that not all users take all practice exams; I'm not sure how to formalize that mathematically. I am working my way through this paper1, which may help.

[1] https://arxiv.org/pdf/1508.03865.pdf

I think that the most natural way to design this problem is indeed with missing values, and of course that means using a learning algorithm which can deal with these. Additionally to NB I think decision trees (at least some implementations) can handle missing value as well.


Datasets for String Classification I would like to test an experimental algorithm for string classification. More precisely, the dataset should be split into a set GOOD of good strigs and a set BAD  of bad strings. The algorithm should learn a model that is consistent with the dataset, in the sense that the output model always answers Yes for strings in GOOD, No for strings in BAD. The algorithm can behave arbitrarily on strings that are neither in GOOD nor in BAD.
Note that I'm not interested in prediction for strings not in the dataset. The only important thing is to obtain a model that is consistency with the given dataset. What makes the problem interesting is the fact that the model can be much smaller than the dataset. For instance, if the model is an automaton, then in some cases one can find automata that are exponentially smaller than the number of bits necessary to represent the dataset.
I would like to test our heuristics against known algorithms/datasets.

What are standard datasets used for testing new experimental heuristics in this area?
What are the standard tools used for such classification tasks on strings.

Note: The goal is to classify strings, not large texts. In particular, a typical size for the strings being considered is 1000 characters (at most). Additionally, one can assume these strings have the same length.

Your description corresponds to the general problem of grammatical inference (or grammar induction): as opposed to statistical models, the goal is to learn the exact language represented as a formal grammar. Importantly this is symbolic learning, i.e. there is no concept of minimizing error because errors are not acceptable at all.
As far as I know the field is rather theoretical, as opposed to statistical ML: one formally proves that some classes of languages can or cannot be learnt under specific constraints. A type of learning framework which used to be studied in this context (not sure if it's still active) is language identification in the limit. A quite different framework is Probably approximately correct learning (PAC), but in this is more general and not strictly about grammatical inference (to the best of my knowledge).
So I'm not aware of any real dataset: in general arbitrary datasets don't have the kind of formal constraints which are relevant for this kind of problem. But it's possible that some parts of the literature contain at least artificial datasets.


Accuracy of the model I'm using this dataset and i'm trying to do logistic regression

heart_data = pd.read_csv('../input/heart-disease-uci/heart.csv')
X = heart_data.iloc[:,:-1]
y = heart_data.iloc[:,-1]
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.20,random_state=5,shuffle=True)

from sklearn.preprocessing import MinMaxScaler

nr = MinMaxScaler()
X_train = nr.fit_transform(X_train)
X_test = nr.transform(X_test)

from sklearn.metrics import accuracy_score

def cal(method,c=1):
    lr = LogisticRegression(C=c,solver=method)
    lr.fit(X_train,y_train)
    pre_test = lr.predict(X_test)
    pre_train = lr.predict(X_train)
    train_score = accuracy_score(y_train,pre_train)
    test_score = accuracy_score(y_test,pre_test)
    return train_score,test_score
for i in method:
    print(i,'->>>>>>>>',cal(i))

First is training accuracy and second one is testing accuracy.
why am i getting more testing accuracy over training?

And Is there another way to increase the both accuracy?
I'm using min-max scaling so are there any other normalization to increase the accuracy more or this is the best accuracy we can get using logistic?

Good that you raise the question because there's most likely a bug.

I've done over 100 ML models and never seen test accuracy being higher than train accuracy.

Potential bug? Most likely there are much more class labels in your training than in your test set. Solution? Try in class_weight = "balanced". This most likely fixes it.

Another potential problems? Did you fix the random seed? You did not set C at a high level? E.g. 100, 1000..

PS: Nice! I see you normalize first and then split train and test! Well done. By doing so, you avoid data leakage.