How to launch a Machine Learning model? First of all thank you for taking your time to read my question. I have done a Machine Learning model with a dataset (The famous one about Cancer) and I want to know how can I do to predict the results for new variables. I think that I have to keep training the data (often) to have more accured data to use in my prediction but for predicting new data, ¿Is as simple as changing the test data (y variable) to the new data? Thank you so much for taking your time and any help would be appreciate it.

Personally , at the first time, I used this link to do predictions in deep learning:

https://machinelearningmastery.com/how-to-make-classification-and-regression-predictions-for-deep-learning-models-in-keras/

if you're using another algorithm (not neural networks), which one?


Regression model to predict probability of rare event I have a dataset with around 900.000 records, around 1000 of which are marked as positive (the studied event occurred).

The probability of the event occurring is always low (i.e. < 0.1), and I would like to create a regression model to predict the probability of the event occurring.

My first thought was to use logistic regression, but I am not sure if I could directly interpret the output as the probability of the event happening. The same doubt arises when using other models, such as SVM or RF.

Another doubt would be whether usual evaluation metrics (e.g. RMSE) would work well on such a model, since even a predictor that always outputs 0 would have a very good score.

The output of logistic regression is exactly that - the probability of an event happening. If your covariates are informative then your model will do better than just saying "P=1000/900000" everytime, because it might say "P=10000/900000" for a positive event, or even "P=0.9" of a positive event given certain covariates.

If there's no predictive power in the covariates (ie they don't correlate with the positive events) then yes, the model will say P=1000/900000 but that's the way it is.

If you want a binary outcome then you have to decide what threshold of P you choose, based on your fear of false positives/negatives or value of true positives/true negatives. For example, if one false positive means people die, you set your threshold such that you don't get false positives. These thresholds are generally not statistical decisions but based on application logic. ROC curves can help here.


Influence of trend on (supposedly) correlated time series TL;DR: What is the impact of a linear trend on the correlation between time series that are (most likely) not spuriously correlated?

I'm currently trying to reconstruct/cross-validate an analysis delivered by one of my companies contractors.

The data is based on time series of sensor data (approx. 3.5m timestamps). Goal was to find the signals with the highest correlation with one specific signal.

Despite not being an expert in data science I was able to reproduce their data cleaning (drop columns with zero variance, interpolate linearly over smaller gaps, drop remaining columns containing NaN-values). But after that I'm not sure if I can confirm their findings.

Seemingly they did a simple pearson-correlation like

corr = df.corrwith(df['DesiredSignal'])

Yet looking at the data the signals seem definitely trended.

When I then apply a detrend-function like

from scipy import signal

df_d = signal.detrend(df[column])
df_n = pd.DataFrame(data=df_d)

and apply the corrwith-function to this new dataframe I get totally different results (e.g. a significant higher amount of highly negativ correlations).

My Question now is: Can I trust the findings of the contractor or are they rendered invaild by not considering the influence of trends on correlation or am I getting something completly wrong?

Q1: What impact does a linear trend have on the correlation between non-spurious time series?
The 4 main measures of correlation are Pearson, Kendall rank, Spearman and Point-biserial (the latter of which is not applicable for this type of problem). For simplicity, I'll only explain how it affects measuring Pearson correlation.
Let's assume $X$ represents a sinusoidal time series without trend: $x_t = sin(t)$, $Y$ represents $X$ with an upwards linear trend: $y_t = t + sin(t)$ and $Z$ represents $X$ with a downwards linear trend: $z_t = -(t + sin(t))$. All series have identical timestamps and have the same unit of measurement (for ease of plotting):

One of the assumption to measure Pearson correlation between two time series is called linearity, that when both series are plotted against one another on a scatter graph, there is a linear relationship:

As you can see, $X$ and $Y$ do not satisfy this condition and so Pearson correlation is the incorrect statistical measure to use, whereas for $Y$ and $Z$ it is. Why though?
Pearson correlation measures the degree to which values deviate from the linear line of best fit between the two series provided. If the relationship is not linear, this relationship will not be measured accurately. This can be shown by plotting the Pearson correlation coefficient as $t$ increases:

Notably, $X$ and $Y$ will also violate the assumption of a monotonic relationship for both Spearman and Kendall rank, and so you cannot measure correlation with any of these methods for $X$ and $Y$ unless transformations of the data are performed to satisfy the underlying assumptions - as you do in the question post.
Linear trends, therefore, don't have strictly positive or negative impacts on measuring correlation. You will just have to react accordingly to the underlying assumptions of the correlation measure you need to use.
Q2: Can I trust the findings of a contractor who doesn't take these type of trends into account?
To paraphrase Hanlon's razor:

It is better to assume ignorance than malicious intent.

If you provide your feedback, the analyst will have an opportunity to discuss through why they chose to pursue a certain route, give them a chance to realise what they did was incorrect, or that they misunderstood the requirements and/or limitations of the project.
Hopefully, this leads to a more positive outcome, given you want the best results and the analyst wants to provide the best service.


Keras Loss Function for Multidimensional Regression Problem I am new to DL and Keras. I am trying to solve a regression problem with multivariate outputs (y shape (?, 2)) using Keras (tensorflow backend). I am having my confusion about how the loss is calculated. I use mean absolute error as the loss function. However, since my target data has 2 dimensions, is the loss value calculated as the reduced mean on all dimensions (a scalar as the result)? I checked the Keras source code, it uses K.mean(..., axis=-1) for MAE calculation. If K.mean is the same to numpy.mean, "axis=-1" should do the column mean (for my case, it should return a tensor with shape (?,2) but not a scalar).  If this is the case, how could the loss value be a single number (as outputed in the training process log)?

If the MAE return is indeed a scalar (reduced mean), this gives me another problem. The data from each dimension of my target is not in a same range. A reduced mean would be biased towards the high value dimension. Shall I change my model to a multi-task learning model then?

Thanks a lot for your help on this.

L.

To answer:

1) When calculating the loss function for multivariate outputs, keras calculates it as a mean across all your outputs: https://github.com/keras-team/keras/blob/2.0.4/keras/losses.py#L12

Hence, if one output is doing really badly and others not, it could influence your loss result.

2) In the source code there are no mentioning about scaling the outputs for the calculation of loss function and, thus, I would conclude that the loss function will depend highly on the boundaries of each of your Y features.

To avoid this, what you could do is to write a custom loss function that scales your Y outputs before outputting loss value:

scaler = MinMaxScaler()
data_trans = scaler.fit_transform(y_true)

def custom_mean_absolute_error(y_true, y_pred):
    y_true = (y_true - K.constant(scaler.min_)) / K.constant(scaler.scale_)
    y_pred = (y_pred - K.constant(scaler.min_)) / K.constant(scaler.scale_)
    return K.mean(K.abs(y_pred - y_true), axis=-1)


Keras Early Stopping: Monitor 'loss' or 'val_loss'? I often use "early stopping" when I train neural nets, e.g. in Keras:

from keras.callbacks import EarlyStopping

# Define early stopping as callback
early_stopping = EarlyStopping(monitor='loss', patience=5, mode='auto', restore_best_weights=True)

# ...THE MODEL HERE...

# Call early stopping in .fit
history = model.fit_generator(..., callbacks=[early_stopping])

Question: I often wonder if it is better to monitor the loss (monitor='loss') or the validation loss (monitor='val_loss'). Are there some takeaways from the literature? What is best practice?

My intuition would be that monitoring the validation loss gives a more direct feedback of the learning process, since in case the validation loss does not increase any further after some epochs, there seems to be little more to learn.

On the other hand - when I use dropout for instance - the loss will often "lag behind" the validation loss (loss $>$ val_loss). In this case, I believe that there still is something to learn for the model, even if this does not immediately translate into improvements in the validation loss.

Validation loss. Overall training loss should keep decreasing so monitoring it isn't as meaningful. If your issue is noise in the validation loss, increase patience.


PCA ? after the transformed data, are they still same with original data, (if maintain same dimensional) after the step of pca, I try to plot them (see pic)

example in the original (A) 2 features => pca 2 features
if I do not reduce any dimensional data, are they still have the same meaning of these data? (the picture look a bit different from them)
it is a circle example for classification data.
any seem similar example after the step of pca.

As you can see from your pics there is a change.
From an algebra point of view, PCA is a base change. The interest of the algo is how the target base is designed: it is designed as to explain variance in the dataset. Dimension reduction is achieved trough removing some base vector with the least associated variance. In other words : PCA is a base change to an interesting base in term of variance explained, which might be followed by a projection on a sub-base.
When you remove the dimension reduction component you still have a base change. Which in the general case amount to some rotation, translastion, scaling, but no significant change in the underlying data. It seems to be what happen in your second graph : a rotation by 45° plus some scaling.


Predicting the variable's score using predictive analytics I am new to predictive analytics but I am good at programming (like Spark, R).

I have 10 variables and I know their score's for over a long period of time. I want to know which variable will have high score in the future?

If it is a linear plot (plotted between score and time for a variable), we can extrapolate the values and find the max(future_score) among the 10 variables.

But the plot is non-linear (although continuous).

Any suggestions?

Thanks for your help?

Can you show us some plots of them ? it really depends on what they look. There is a lot of temporal series forecast.

It will depends on :

Is there any seasonality ?
Is there a trend ?
Is there a high volatility ?
On how long do you want do to your forecast ?


How to Manipulate data for multiple visits per person? I have a query to solve. I have data regarding customers and number of visits done to them. These are in two tables. So I want to join two table and create different features so that I can find better/lagging customers from the data based on scoring.

customer_dataset:

id(unique id for customer)
name
assigned_user_id(user assigned)
company_id(company_which user works for)
expected value(revenue expected)
status(whether a lead/customer([0,1]categorical))

visit_dataset

customer_id(unique_id)

user_id
company_id
date(visit date done)
schedule_date(actually scheduled)
next_action_date(if any next action)
status(visit done or not)

So I need to score lead/customer on the basis of these data and find a list of top user. Also from the number of visits done, I need to figure out how many average/sum of visits done per customer so that I can find next forecast of visit(In case he doesnt go or forget about lead).

I'm new in manipulating data with pandas. How can I;

find total visit done per customer and other features from dates?
any resource that I can follow for how to work with dates data in pandas?
I'm trying to use featuretools to get the features.(any other advise)?
also I'm trying to automate this feature. So I'll feed in last 30 days visits as data and then I have to figure out the next visit date(prediction)Important (I so don't know how will i do that but trying ;P)

Sorry If i missed any info, please give me feedback if any issue.

Thanks a lot.

So, here you have a couple of questions:

Find total visits per customer by date

To do this bit, you have simply do a pandas Dataframe groupby, like so:

customer_id.groupby(['user_id', 'date']).size()

Resources for working with dates in pandas

https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html
This is official documentation from pandas, so should hopefully give you all the tools you need.

Unfortunately cannot answer the question on feature tools, so I look forward to seeing answers on this.
How to model the next customer visit date given their previous visits.

One way to do this would be that if customer visit frequently, then one idea would be to model this data with a Recurrent Neural Network (RNN) or LSTM (A video on the principles behind RNNs: https://www.youtube.com/watch?v=UNmqTiOnRfg), where it takes the sequence of inputs (dates when particular customer has visited) and return an output date after observing the sequence.
Before this though, might be useful to do customer segmentation and treat observe visiting patterns over customer groups than as individuals. One way to do this any clustering algorithm like agglomerative clustering.


Create multiple matrices from 2 bigger ones in R I have 2 matrices, A (1000x21) and B (1000x7). Matrix A has individuals(=1000) in the rows and their consumption in 21 days at the columns. Matrix B has the SAME individuals(=1000) in the rows and some weights for each day of the week(=7) in the columns.

What I would like to have in the end is 1000 (with the dimension of 2x21) matrices  (one for each individual), lets call them $X_{i}$. In the first row of each $X_{i}$ I would like to have the consumption of the individual $i$ each of the 21 days (this will come from matrix A), and at the second row of the $X_{i}$ I would like to have the respective weight of that day (this will come from matrix B).

So matrix A looks like $[cons_{1,1} \ cons_{1,2} \ ... \ cons_{1,21} \\
                        \ cons_{2,1} \ cons_{2,2} \ ... \ cons_{2,21} \\
                                     . \\
                        . \\
                         . \\
                       \ cons_{1000,1} \ cons_{1000,2} \ ... \ cons_{1000,21}] $

Matrix B looks like $[weight_{1,1} \ weight_{1,2} \ ... \ weight_{1,7} \\ \ weight_{2,1} \ weight_{2,2} \ ... \ weight_{2,7} \\ . \\ . \\ . \\ \ weight_{1000,1} \ weight_{1000,2} \ ... \ weight_{1000,7}]$

And I would like the matrix $X_{i}$ to be like $[cons_{i,1} \ cons_{i,2} \ ... \ cons_{i,21} \\ \ weight_{i,1} \ weight_{i,2} \ ... weight_{i,k}]$

Any ideas how to do this in R in a loop ?

This will extract the rows from matrix A and B.

matrix(c(A[x,],rep(B[x,],times=3)),nrow=2,byrow=T)

If you want to get them into a list (recommend)

single<-lapply(c(1:1000), function(x) matrix(c(A[x,],rep(B[x,],times=3)),nrow=2,byrow=T))

To put them all in diagonal matrix with adiag from magic package

d<-adiag(single[[1]])

for(i in 2:1000){
  d<-adiag(d,single[[i]])
}

I could make it work without the loop (anyone any suggestions?)


Is the decision tree the right choice to classify for this dataset? I have a bunch of tuples like this; [SourceIP, DestinationIP, Port, TimeStamp]

If a destination IP recorded 21, 22, 23 and 80 port (set of 5 tuples) then I will decide something, if it has set of 4 tuples I will decide something if less than 4 I will decide another thing...

I have already handled it with SQL rules but I need to use the decision tree algorithm.

Any idea?

An important thing that I forgot to write is a single tuple has no meaning, I should check timestamps and ports that group of tuple became meaningful...

etc: A set of tuples that is in 5 min and includes ports that 21,22,23,25

If you know what you will decide based on the length of the tuple, then you can probably just hard-code your rules! Did the SQL version not work for you? Or are you just experimenting with decision trees?

You can of course use a decision tree. You data can be split based on the features you provide, so you can look at more than the length of the tuple e.g. look at the values of the DestinationIP and your other variables in the tuple.

It could look something like this:

from sklearn import tree
from sklearn.model_selection import train_test_split

Create a train-test split from your data, assuming you have some input tuples and the response that you want to predict:

X = your_tuple_data
y = corresponding_responses_to_predict

# Split dataset into training set and test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=1) # 70% training and 30% test

Create Decision Tree classifer object

clf = DecisionTreeClassifier()
# Train Decision Tree Classifer
clf = clf.fit(X_train,y_train)

Predict the response for test dataset

y_pred = clf.predict(X_test)

Have a look at the SciKit-Learn documentation for more information. Here is a tutorial with more ideas behind why this could work.


how do I find out if my random forresrt is overfitting? I used 100 trees on time series data to predict and the r square score is 0.9249886998972637. Is this over fitting? How do I know? Also, how do I determine the forest size? My data size is 1400 and variable 27 $(1400*27 =37800)$

The standard method to detect overfitting is to evaluate the model on both the training set and the test set. If the performance obtained on the training set is very high but much lower on the test set, then it's likely that the model is overfit.
Naturally this method works only with a proper test set: if there has been some data leakage from the test set to the training set, then the evaluation on the test is biased (the performance value is unreliable).


Census dataset alternatives I'm looking to replace some of my code samples which use the Census dataset. Criticism is that there is some bias and ML fairness issues around this popular dataset. What other public datasets for binary classification which include both categorical and numerical features as well as a fair amount of examples can be relevant?

Candidates are:

KDD Cup 1999

Chicago Taxi dataset is another option with large number of records and multiple feature types: categorical, numerical and buckets


Breaking down a column in Pandas into a separate CSV for display in Tableau My data is coming from a CSV, which should be visualized in Tableau.

However, the data contains the column category_list, which consists of values separated by a vertical bar (|).

Since Tableau can't handle arrays inside of attributes, I used Python (Pandas) to load the CSV and manipulate the data:

import pandas as pd
companies = pd.read_csv("companies.csv")

I assume that the category_list column needs to be broken down and stored into another CSV (containing the permalink (unique ID) and category pairs).

Something like this:

permalink,category
/organization/-qounter,Application Platforms
/organization/-qounter,Real Time
/organization/-qounter,Social Network Media
/organization/-the-one-of-them-inc-,Apps
/organization/-the-one-of-them-inc-,Games
/organization/-the-one-of-them-inc-,Mobile
/organization/1-4-all,Entertainment
/organization/1-4-all,Games
/organization/1-4-all,Software
/organization/1-800-publicrelations-inc-,Internet
/organization/1-800-publicrelations-inc-,Marketing
/organization/1-800-publicrelations-inc-,Media
/organization/1-800-publicrelations-inc-,Public Relations
/organization/1-mainstream,Apps
/organization/1-mainstream,Cable
/organization/1-mainstream,Distribution
/organization/1-mainstream,Software
...

How to achieve it?

Excerpt of the original CSV:

permalink,category_list,...
/organization/-qounter,Application Platforms|Real Time|Social Network Media,...
/organization/-the-one-of-them-inc-,Apps|Games|Mobile,...
/organization/1-4-all,Entertainment|Games|Software,...
/organization/1-800-publicrelations-inc-,Internet|Marketing|Media|Public Relations,...
/organization/1-mainstream,Apps|Cable|Distribution|Software,...
...

If you don't need other columns, here is a solution. It splits the column, stacks in vertically and combines with "permalink" column

df.set_index('permalink').category_list.str.split('|', expand=True).stack().reset_index('permalink').rename(columns={0:'category'})

permalink   category
0   /organization/-qounter  Application Platforms
1   /organization/-qounter  Real Time
2   /organization/-qounter  Social Network Media
0   /organization/-the-one-of-them-inc- Apps
1   /organization/-the-one-of-them-inc- Games
2   /organization/-the-one-of-them-inc- Mobile
0   /organization/1-4-all   Entertainment
1   /organization/1-4-all   Games
2   /organization/1-4-all   Software
0   /organization/1-800-publicrelations-inc-    Internet
1   /organization/1-800-publicrelations-inc-    Marketing
2   /organization/1-800-publicrelations-inc-    Media
3   /organization/1-800-publicrelations-inc-    Public Relations
0   /organization/1-mainstream  Apps
1   /organization/1-mainstream  Cable
2   /organization/1-mainstream  Distribution
3   /organization/1-mainstream  Software

Then you can save it to csv.


Interpretation of accuracy score on subset of data points I have a multi-class problem that I am building a classifier for. I have N total data points I would like to predict on. If I instead predict on n < N data points and get an accuracy score, is there a way I can say (with some degree of confidence) what I think the same model's accuracy score would be on the remaining data points?

Can somebody point me to an article discussing this, or suggest a formula to research?

Usually when working with classification problems, one tries to have 3 subsets of data:

A training set: this subset is usually the biggest one and can take up to ~80% of the available data. It is used to train the chosen algorithm, using the known labels of each data sample.
A validation set: this subset is much smaller. It will typically be ~5-10% of the available data. It is used to evaluate the performance of the algorithm trained on the training set. Typically, one will tune the parameters of the algorithm in order to reach the best performance on the validation set.
A testing set: this subset is of the same size order as the validation set or bigger. Very important: it should NEVER be used for training purpose! Once the model is trained and tuned using the training and validation sets, the testing set allows one to get the accuracy (or any other performance metric) on unseen data. If the model generalizes well, the score will be close to the ones seen on the validation set, often times a tiny bit worse.

In order for this to work properly, it is important that all the subsets are representative of the available data. For example, that the proportion of each class is approximately the same across the subsets.

In the light of such broadly used process, we can see that most algorithms are tuned and tested on a fraction of the available data. As long as the used test set is balanced in a way that is similar to the training and validation sets and that it has not been used at all during the training/tuning of the model, there is no reason why the performance scores would not generalize well to N > n samples used in the test set.


Measuring performance of different classifiers with different sample sizes I'm currently using several different classifiers on various entities extracted from text, and using precision/recall as a summary of how well each separate classifier performs across a given dataset.

I'm wondering if there's a meaningful way of comparing the performance of these classifiers in a similar way, but which also takes into account the total numbers of each entity in the test data that's being classified?

Currently, I'm using precision/recall as a measure of performance, so might have something like:

                    Precision Recall
Person classifier   65%       40%
Company classifier  98%       90%
Cheese classifier   10%       50%
Egg classifier      100%      100%

However, the dataset I'm running these on might contain 100k people, 5k companies, 500 cheeses, and 1 egg.

So is there a summary statistic I can add to the above table which also takes into account the total number of each item? Or is there some way of measuring the fact that e.g. 100% prec/rec on the Egg classifier might not be meaningful with only 1 data item?

Let's say we had hundreds of such classifiers, I guess I'm looking for a good way to answer questions like "Which classifiers are underperforming? Which classifiers lack sufficient test data to tell whether they're underperforming?".

You need to look at the confidence interval of the statistic.  This helps measure how much uncertainty in the statistic, which is largely a function of sample size.


Incorrect visualisation using Plotly kmeans = KMeans(n_clusters=4)

model = kmeans.fit(europe_july)
pred = model.labels_
europe_july['cluster'] = pred

pca = PCA(n_components=2)
pca_model = pca.fit_transform(europe_july)
data_transform = pd.DataFrame(data = pca_model, columns = ['PCA1', 'PCA2'])
data_transform['Cluster'] = pred

plt.figure(figsize=(8,8))
g = sns.scatterplot(data=data_transform, x='PCA1', y='PCA2',\
                    palette=sns.color_palette()[:4], hue='Cluster')
title = plt.title('World countries clusters with PCA')

But when I run this code it does not seem to take into account this model.
europe_july['country'] = countries
europe_july['iso_alpha'] = iso_alpha

fig = px.choropleth(data_frame = europe_july,
                    locations= "iso_alpha",
                    scope= 'world',
                    title='2020-11-07 (World)',
                    color= "cluster",
                    hover_name= "country",
                    color_continuous_scale= 'earth',
                    )

fig.show()

Since this is the output that I get, as you can see there is clearly a cluster with only three countries, when there is no such cluster predicted by the model.
This is the output of the predictions for clusters and it matches the visualizations by PCA:
array([2, 3, 1, 0, 2, 0, 0, 3, 1, 3, 3, 3, 1, 3, 3, 1, 1, 3, 3, 1, 2, 0,
       1, 1, 0, 1, 2, 3, 0, 2, 3, 2, 2, 1, 2, 3, 2, 0, 2, 2, 3, 3, 1, 2,
       2, 1, 2, 3, 1, 3, 3, 3, 2, 3, 2, 0, 1, 1, 1, 1, 2, 2, 3, 2, 0, 0,
       2, 3, 3, 0, 2, 2, 3, 3, 2, 0, 3, 0, 2, 3, 1, 0, 2, 2, 1, 2, 1, 3,
       3, 3, 1, 1, 3, 1, 3, 0, 3, 3, 1, 3, 0, 2, 1, 2, 0, 3, 1, 2, 3, 3,
       2, 2, 2, 0, 3, 3, 3, 2, 2, 3, 1, 2, 3, 2, 3, 1, 1, 0, 1, 3, 0, 2,
       2, 1, 2, 1, 3, 0, 3, 0, 2, 2, 0, 3, 1, 1, 2, 3, 2, 1, 3, 1, 3, 3,
       3, 3, 3, 3, 2, 0, 1, 0, 0, 2, 3, 2, 1, 3, 2, 3, 0, 3, 3, 2, 1, 3,
       2, 3, 3, 2, 1, 2, 3, 3, 2, 3, 1, 2, 1, 2, 2, 1, 1, 3, 0, 2, 3, 3,
       3, 3, 3, 3, 2, 0, 2, 1, 0, 2, 2, 2, 1, 0], dtype=int32)

Could someone please guide on why my visualisation is wrong?

The visualisation is in fact correct, the only issue is that the Plotly map simply does not have those countries marked on its map at all.


How to extract a subset from a data frame? Let's say I have a dataframe that contains several columns including a column called A (containing integers).

I would only like to consider the rows of table that correspond to $A \ge 3$.

What would be the syntax for this?

There are several ways

df[df['A']>=3]

Maybe using query

df.query('A >=3')

Or using the .loc

df.loc[df['A'] >= 3]

This question is posted on several places.


Dealing with an apparently inseparable dataset I'm attempting to build a model/suite of models to predict a binary target. The exact details of the models aren't important, but suffice to say that I've tried with half a dozen different types of models, with comparable results from all of them.

On looking at the predictions on various subsets of the training data, it appears that a certain subset of features is important for around 30% of the data, while a different subset is important for the remaining 70%. With the training data/holdout set. This separation is fairly easy to detect when the target is known (run a model with subset1, another with subset2, find the subset where one model does better than the other). Obviously, this is not possible with the test data, since the target is not known there.

There are clearly (at least) two regions in the data that are substantially different from each other, since a model trained on the whole dataset does worse than the combination of separate models trained on each of these regions.

However, the dataset seems spectacularly resistant to separation, with no major difference between principal components (linear and RBF kernel) and no clear/stable clusters after applying several different clustering algorithms (KMeans, agglomerative, mean shift).

Are there any popular methods that allow separation/clustering of data that are resistant to "normal" methods? The end goal here is to find out which model to use on which rows when target is unknown.

You might have a domain adaptation problem.
The samples are actually taken from two different sources and have different rules governing their behavior.

I suggest that you'll try the following method.

Split the data into two data set using the target concept.
Build a model for each data set. In case your performance doesn't improve it is likely that the domain adaptation assumption is wrong.
Otherwise, we gain an improvement. Now we should find out which model to apply. Use the full dataset and use the target in order to create a new concept indicating to which dataset the sample belongs. Since you said that given the target you can deduce that I assume the two are different.
Build a model for the new dataset. If it performs well, the problem is solved. Run this model first and apply the proper model for each sample. Otherwise, it means that the two data set are indistinguishable (at least with respect to the methods you applied), weakening the assumption the the cause of the problem is domain adaptation.


PPO, A2C for continuous action spaces, math and code Edit:
Question has been edited to better reflect what I learned after asking the original question.

I implemented the clipped objective PPO-clip as explained here: https://spinningup.openai.com/en/latest/algorithms/ppo.html

Basically I used a dummy actor network to find the new action probability without training the local actor network.

"""use temp_actor to get new prob so we don't update the actual actor until
        we do the clip op"""
        curr_weights = self.actor.get_weights()
        self.temp_actor.set_weights(curr_weights)
        self.temp_actor.fit(state, advantages, epochs=1, verbose=0)
        new_policy = self.temp_actor.predict(state, batch_size=1).flatten()
        new_aprob = new_policy[action]

Then I worked out the ratio of action probabilities and implemented the PPO clipping parts of the algorithm:

ratio = new_aprob / old_aprob
        # scale = min(ratio * advantages, K.clip(ratio, 1 - self.epsilon, 1 + self.epsilon) * advantages)
        no_clip = ratio * advantages
        clipped = np.clip(ratio, 1 - self.epsilon, 1 + self.epsilon) * advantages

        self.actor.fit(state, np.minimum(no_clip, clipped), epochs=1, verbose=0)

The full code is here (please excuse some coarse language in the comments):    https://github.com/nyck33/openai_my_implements/blob/master/cartpole/my_ppo_cartpole.py

How can I adapt it for a continuous action space problem such as Pendulum v0.
Update:  I just read that the distribution of actions is a normal or Gaussian distribution here:  Reddit normal distribution

That seems strange since I imagined the curve of actions to be skewed one way or another depending on whether a particular state would get better results with certain action tendencies.

Update:  I found this on the Stable baselines site for PPO:

From this URI:  ppo explanation

I also saw that section 13.7 in Sutton's RL book seems to be a must-read for this type of problem as the section is titled:  Policy parameterization for Continuous Actions.

Furthermore I also learned in the process of searching for a readable implementation that GAE (advantage) is normalized so that probably makes an implementation more robust against wild fluctuations.  I also think my implementation is incomplete, ie. missing other components compared to the Pytorch "solution" I linked to in the answer.

Packt Publishing's "Deep Reinforcement Learning Hands-On" has an entire chapter on continuous action spaces.  Here is the math in the book:

and the code accompanying the book:  code repo for book

The code for the math above is in:

  The complete source code is in Chapter14/02_train_a2c.py ,
  Chapter14/lib/model.py and Chapter14/lib/common.py . Most of the code
  will already be familiar to you, so the following includes only the
  parts that differ. Let's start with the model class defined in
  Chapter14/lib/model.py .


Compare multiple values from a DataFrame against a single row from another I'm trying to compare address values for inaccuracies, for example, given multiple records like:

Reference
Apartment
Address
PostCode

AS097
NaN
00 Name Road
BH1 4HB

AS097
Flat 1 Building Name
00 Name Road
BH1 4HB

AS097
Flat 2 Building Name
00 Name Rd
BH1 4HB

AS097
Flat 3 Building Name
00 Name Road
BH1 4HB

AS097
Flat 4 Building Name
00 Name Road
BH1 4HB

AS097
Flat 5 Building Name
00 Name Road
BH1 4HX

HO056
NaN
23 Street Road
XG9 9GX

I've a dataframe where I store all the "main" addresses by checking if the Apartment column is empty like so:
main_address = df['Apartment'].isnull()

df_st = pd.DataFrame({'Reference':df[main_address].Reference, 'Address':df[main_address].Address, 'PostCode':df[main_address].PostCode})

df_st will look like this:

Reference
Apartment
Address
PostCode

AS097
NaN
00 Name Road
BH1 4HB

HO056
NaN
23 Street Road
XG9 9GX

df has over 1K records, but df_st containing the "main" address ends up with approx. 200 records.
I'm trying to create a new DataFrame where I can identify where the records don't match by domparing df against df_st.
THE PROBLEM
I try the below:
# Clean the Reference values
refs_list = df['Reference'].str.split('/').str[0]
df['Reference'] = refs_list

# Create a new column titled issues and flag if the references match
df['issues'] = np.where(df['Reference'] == df_st['Reference'], 'True', 'False')

I want the above for the Address and PostCode unfortunately this does not work since df and df_st don't have the same shape.
I'm struggling to find a way to achieve a comparison between the two DataFrames df against df_st.
I want to compare all matching Reference row values from df against it's matching from df_st and if one of them don't match create a new column title Issues and store the conflicting column there.
MY DESIRED OUTCOME
Given the data above, after comparing df data against df_st results in a new DataFrame, like below

Reference
Apartment
Address
PostCode
Issues

AS097
NaN
00 Name Road
BH1 4HB
None

AS097
Flat 1 Building Name
00 Name Road
BH1 4HB
None

AS097
Flat 2 Building Name
00 Name Rd
BH1 4HB
Address

AS097
Flat 3 Building Name
00 Name Road
BH1 4HB
None

AS097
Flat 4 Building Name
00 Name Road
BH1 4HB
None

AS097
Flat 5 Building Name
00 Name Road
BH1 4HX
PostCode

HO056
NaN
23 Street Road
XG9 9GX
None

Where Address appears as an issue in the column Issues since the address don't match against df_st, same for PostCode since it differs from df_st
IN A NUT SHELL
All I want to to know how to compare matching rows by Reference from a DataFrame against another and compare the other values Address and PostCode.
Hope that makes sense.

You can join df and df_st on Reference:
df_merged = pd.merge(df, df_st, on="Reference", how="left")

Note: The how="left" would really depend on what you want in the joined table. You can then compare the values in the joined columns:
df_merged['Address_df'] == df_merged['Address_df_st']


Does resizing images during training affect the bounding box annotations? I am using tensorflow object detection API to train my own custom dataset and I am preparing annotations for the same. I see from the config file of my pre trained ssd inception net, the size of the image is reduced to 300 x 300 during training. My doubt is whether the resize would now change the position of my object according to annotation? I mean now the xmin, ymin width and height of the bouding box would be different since it resized. should i annotate on the resized images(resize them myslef before training?) or original one that i give to training?

My doubt is whether the resize would now change the position of my object according to annotation?

Yes, it will.

  should i annotate on the resized images(resize them myslef before
  training?)

No, you should annotate at the original size.

You solve this by applying the corresponding transformations on your bounding boxas well. So if you resize your image, you rescale your bounding box. This allows you to expand to different image augmentations without redoing annotations for all of them.

I recommend you to chose a library that has to ability to apply transformations on both images and their bounding boxes. I use albumentations but there are others such as imaug.


Image data augmentation with this function Following function is from http://scikit-learn.org/stable/auto_examples/neural_networks/plot_rbm_logistic_classification.html#sphx-glr-auto-examples-neural-networks-plot-rbm-logistic-classification-py where it is used to augment digits data:

Its documentation says:

    This produces a dataset 5 times
    bigger than the original one,
    by moving the 8x8 images in X
    around by 1px to left, right,
    down, up

def nudge_dataset(X, Y):
    direction_vectors = [
        [[0, 1, 0],
         [0, 0, 0],
         [0, 0, 0]],

        [[0, 0, 0],
         [1, 0, 0],
         [0, 0, 0]],

        [[0, 0, 0],
         [0, 0, 1],
         [0, 0, 0]],

        [[0, 0, 0],
         [0, 0, 0],
         [0, 1, 0]]]

    shift = lambda x, w: convolve(x.reshape((8, 8)), mode='constant',
                                  weights=w).ravel()
    X = np.concatenate([X] +
                       [np.apply_along_axis(shift, 1, X, vector)
                        for vector in direction_vectors])
    Y = np.concatenate([Y for _ in range(5)], axis=0)
    return X, Y

Can this function be used for any kind of image dataset?

Data augmentation is a technique for increasing the dataset size by performing certain operations over images such as translation, rotation, hue shift etc and increase the training dataset size. It could be applied to any kind of image dataset.

The above mentioned function seems to be applying shift with images of shape 8x8 and as far as the shape of the image is of this dimension, it will definitely work.

Refer to this paper for further reference.


BERT word embedings for finding word definition Can BERT, GPT or other contextualised embedings be used for finding word definitions? What would be the most effective and not complicated approach for tackling a sample task as described below.

Map the meaning of the word 'bank' in the sentence "I was walking along the bank of the river" with one of the definitions listed in the WortNet database (or other word-sense lookup table).

BERT generates contextualized word embeddings, which means that BERTprovides the most accurate embeddings when a word is in a sentence(context).
For each of the words within the sentence, BERT will generate a vector of numbers. In your case, you will have a good representation of the word "bank".
So if you have a sentence for all the other words that you are trying to match with "bank", all you need to do is to extract the embeddings for them, then compare them with a similarity matric (i.e. cosine similarity). this way you are ranking them to see which ones are most correlated to "bank".


Can Reinforcement learning be applied for time series forecasting? Can Reinforcement learning be applied for time series forecasting?

Yes, but in general it is not a good tool for the task, unless there is significant feedback between predictions and ongoing behaviour of the system.

To construct a reinforcement learning (RL) problem where it is worth using an RL prediction or control algorithm, then you need to identify some components:

An environment that be in one of many states that can be measured/observed in a sequence.
An agent that can observe current state and take actions in the same sequence.
The evolution of state in the sequence should depend on some combination of the current state and the action taken, and may also be stochastic.
There should be a reward signal that the RL agent can observe or  measure. The value of reward should depend on the same factors as the evolution of the state, but can depend on them in a different way.

The general case of time series forecasting can be made to fit with this by treating the prediction as the action, having the state evolution depend on only the current state (plus randomness) and the reward based on state and action. This will allow RL to be applied, but causality only flows one way - from the environment into your predictive model. As such, the best you can do for rewards for instance is to use some metric about the correctness of the predictions. Consequences for good or bad predictions do not affect the original environment. Essentially you will end up wrapping some predictive model for the sequence (such as a neural network) in a RL layer which could easily be replaced by basic data set handling for a supervised learning problem.

One way you could meaningfully extend series forecasting problems into RL problems is to increase the scope of the environment to include the decisions made based on the predictions, and the state of the systems that are affected by those decisions. For instance, if you are predicting stock prices, then include your portfolio and funds in the state. Likewise the actions stop being the predictions, becoming buy and sell commands. This will not improve the price prediction component (and you are likely better off treating that as a separate problem, using more appropriate tools - e.g. LSTM), but it will frame the issue overall as a RL problem.


How to calculate an average cliff date? Please let me know if this question belongs elsewhere.
In my simple data set focused on sales pursuit opportunities, I have the following columns available.

Pursuit Name
Status(Open, Won, Lost)
Date Entered
Date Closed

I need to calculate the an average "cliff" date value of when a pursuit will likely either lose or never close. So if another opportunity has been open for x number of days, we can say (based on past opportunities) that this will either never close or will be lost.
My assumption is that this should be based on the difference between Date Entered and Date closed, but I'm unsure of the calculations needed to produce an average "Cliff date".
Is there another term aside from cliff date I should be using? Are there any ideas of how this can be calculated?
Thank you for your time.

You would indeed need to calculate the difference in days between the Date Entered and the Date Closed column. You can then either (1) group directly by the number of days and look at the percentage of records where the Status is equal to Open or Lost or (2) first create buckets and then perform the same step as in (1). You are then free to choose the percentage which you feel equals "likely" (50%, 80%, 99%?) and then use the difference in days as the threshold.


Why does the same algorithm give very different metrics on similar datasets? I used Random Forest and hypertuned the parameters for a binary classification problem on a dataset (dataset A). I got a F1 score of 0.78. I then used a second dataset (dataset B). It was very similar to dataset A (same variables and the distribution of classes in the target variable). I again built and trained a different Random Forest algorithm for dataset B. I expected the f1 score to be around 0.78, but the f1 score for dataset B was 0.50.

Why could there be such a large difference between the f1 scores of the 2 datasets?

Both datasets (A & B) are very similar to each other and I trained separate models on both of them.

Simple answer: the datasets are different. Apparently, your approach worked well for A but not for B. From this you might infer, that B and A are not so similar after all. I suspect that B has a more complicated relation between the X and targets y. So maybe try boosting on B?!


Kaggle Titanic submission score is higher than local accuracy score This is the starter challenge, Titanic. The original question I posted on Kaggle is here. However, nobody really gives any insightful advice so I am turning to the powerful Stackoverflow community.

Based on this Notebook, we can download the ground truth for this challenge and get a perfect score.

I tested it and it does give me 100% on LB for the purpose of confirming it is the ground truth as it claims. (side question here: how do I remove this perfect submission because now it shows I have 100% on this challenge but I want to show my real score, which is roughly 80% and I will keep improving)

Sometimes submission on Kaggle takes several minutes to get back the score so I used the ground truth locally to test my different models to save time. However, they always give me different results. See the following:

These are the code I use, what's wrong? You can use my code to try your submission and do you also have the same problem?

def mark(pred):
    solution = os.path.join(dirname, './output/solution.csv')
    submission = os.path.join(dirname, './output/'+pred)
    solution = pd.read_csv(solution)
    submission = pd.read_csv(submission)

    solution.columns = ['PassengerId', 'Sol']
    submission.columns = ['PassengerId', 'Pred']

    df = pd.concat([solution[['Sol']], submission[['Pred']]], axis=1)
    num_row = df.shape[0]
    print(pred[:-4], '==', (df[(df['Sol'] == df['Pred'])]).shape[0] / num_row)

if __name__== "__main__":
    mark('achieve_99_dtree_rfe.csv')
    mark('advanced_feature_with_stacking_5_fold.csv')

First question: on certain competitions on kaggle you can select your submission when you go to the submissions window. There you may not be able to on titanic one so you are stuck with 100 percent.

Regarding your second question and why do you have different results locally, there is a couple of explanations, first its randomization did you set all the seeds, second all of the modules/library versions you have locally sam as the ones on kaggle? Third different hardware could also be the reason


How to train model on specific classes from dataset for object detection? I want to use darkflow to perform object detection. I have 5 classes of objects I want to detect, say ,

  chair
  bottle
  door
  phone
  computer

I want to specifically train my model again, not use pre-trained weights. I plan to train it on images which are annotated with bounding boxes. But all available datasets like imagenet, COCO have images that have objects of other classes also annotated in them. So how can I use these datasets to  train specifically on my 5 classes? I don't want my model training on all the other objects in the images.

You need to change xml or csv files of bounding boxes and from there you need to remove the all classes other then your 5 classes.


CNN for subsets of a dataset - how to tune hyperparameters I have a dataset and would like to train CNNs on subsets of different size of the dataset.
I already have a CNN, which classifies very well if I use the entire dataset.
Now the question arises if I should really try to additionally optimize the parameters of the CNN for the subsets, regardless of whether I do Data Augmentation or not?
Does it really make sense if I try to change the CNN model for the subsets by using RandomizedSearchCV or GridSearchCV to optimize the number of convolutional layers, different learning rates, etc....?

In other words, suppose I found the perfect CNN model for a dataset. Is this model also the perfect model for subsets of this dataset?

I hope someone can give me a hint. For any help, I thank you in advance.

It depends on how you choose the subset of the dataset. One required assumption about data for any Machine Learning model to generalize well on unseen data is that the data must come from the same statistical distribution.

So, if you select the subset at (uniformly) random, it's quite sure that your model will continue to perform well. Otherwise, I think it's better to fine-tune the big model on the subset for better performance.


Struggling to integrate sklearn and pandas in simple Kaggle task I'm trying to use the sklearn_pandas module to extend the work I do in pandas and dip a toe into machine learning but I'm struggling with an error I don't really understand how to fix.

I was working through the following dataset on Kaggle.

It's essentially an unheadered table (1000 rows, 40 features) with floating point values.

import pandas as pdfrom sklearn import neighbors
from sklearn_pandas import DataFrameMapper, cross_val_score
path_train ="../kaggle/scikitlearn/train.csv"
path_labels ="../kaggle/scikitlearn/trainLabels.csv"
path_test = "../kaggle/scikitlearn/test.csv"

train = pd.read_csv(path_train, header=None)
labels = pd.read_csv(path_labels, header=None)
test = pd.read_csv(path_test, header=None)
mapper_train = DataFrameMapper([(list(train.columns),neighbors.KNeighborsClassifier(n_neighbors=3))])
mapper_train

Output:

DataFrameMapper(features=[([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39], KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
       n_neighbors=3, p=2, weights='uniform'))])

So far so good. But then I try the fit

mapper_train.fit_transform(train, labels)

Output:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-6-e3897d6db1b5> in <module>()
----> 1 mapper_train.fit_transform(train, labels)

//anaconda/lib/python2.7/site-packages/sklearn/base.pyc in fit_transform(self, X, y,     **fit_params)
    409         else:
    410             # fit method of arity 2 (supervised transformation)
--> 411             return self.fit(X, y, **fit_params).transform(X)
    412
    413

//anaconda/lib/python2.7/site-packages/sklearn_pandas/__init__.pyc in fit(self, X, y)
    116         for columns, transformer in self.features:
    117             if transformer is not None:
--> 118                 transformer.fit(self._get_col_subset(X, columns))
    119         return self
    120

TypeError: fit() takes exactly 3 arguments (2 given)`

What am I doing wrong? While the data in this case is all the same, I'm planning to work up a workflow for mixtures categorical, nominal and floating point features and sklearn_pandas seemed to be a logical fit.

I have never used sklearn_pandas, but from reading their source code, it looks like this is a bug on their side. If you look for the function that is throwing the exception, you can notice that they are discarding the y argument (it does not even survive until the docstring), and the inner fit function expects one argument more, which is probably y:

def fit(self, X, y=None):
    '''
    Fit a transformation from the pipeline

    X       the data to fit
    '''
    for columns, transformer in self.features:
        if transformer is not None:
            transformer.fit(self._get_col_subset(X, columns))
    return self

I would recommend that you open an issue in their bug tracker.

UPDATE:

You can test this if you run your code from IPython. To summarize, if you use the %pdb on magic right before you run the problematic call, the exception is captured by the Python debugger, so you can play around a bit and see that calling the fit function with the label values y[0] does work  -- see the last line with the pdb> prompt. (The CSV files are downloaded from Kaggle, except for the largest one which is just a part of the real file).

In [1]: import pandas as pd

In [2]: from sklearn import neighbors

In [3]: from sklearn_pandas import DataFrameMapper, cross_val_score

In [4]: path_train ="train.csv"

In [5]: path_labels ="trainLabels.csv"

In [6]: path_test = "test.csv"

In [7]: train = pd.read_csv(path_train, header=None)

In [8]: labels = pd.read_csv(path_labels, header=None)

In [9]: test = pd.read_csv(path_test, header=None)

In [10]: mapper_train = DataFrameMapper([(list(train.columns),neighbors.KNeighborsClassifier(n_neighbors=3))])

In [13]: %pdb on

In [14]: mapper_train.fit_transform(train, labels)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-14-e3897d6db1b5> in <module>()
----> 1 mapper_train.fit_transform(train, labels)

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sklearn/base.pyc in fit_transform(self, X, y, **fit_params)
    409         else:
    410             # fit method of arity 2 (supervised transformation)
--> 411             return self.fit(X, y, **fit_params).transform(X)
    412
    413

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sklearn_pandas/__init__.pyc in fit(self, X, y)
    116         for columns, transformer in self.features:
    117             if transformer is not None:
--> 118                 transformer.fit(self._get_col_subset(X, columns))
    119         return self
    120

TypeError: fit() takes exactly 3 arguments (2 given)
> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sklearn_pandas/__init__.py(118)fit()
    117             if transformer is not None:
--> 118                 transformer.fit(self._get_col_subset(X, columns))
    119         return self

ipdb> l
    113
    114         X       the data to fit
    115         '''
    116         for columns, transformer in self.features:
    117             if transformer is not None:
--> 118                 transformer.fit(self._get_col_subset(X, columns))
    119         return self
    120
    121
    122     def transform(self, X):
    123         '''
ipdb> transformer.fit(self._get_col_subset(X, columns), y[0])
KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
           n_neighbors=3, p=2, weights='uniform')


Backpropagation with multiple different activation functions How does back-propagation handle multiple different activation functions?

For example in a neural network of 3 hidden layers, each with a separate activation function such as tanh, sigmoid and ReLU, the derivatives of each of these functions would be different, so do we just compute the output error of each of these layers with the derivative of the activation function of that layer?

Also will the gradient of the cost function used to compute the output error of the last layer change at all, or is the activation function of that layer used to compute the gradient of the cost function?

In short, all the activation functions in the backpropagation algorithm are evaluated independently through the chain rule, thus, you can mix and match to your hearts content.

What are we optimizing in backpropagation?
Backpropagation allows you to update your weights as a gradient function of the resulting loss. This will tend towards the optimal loss (the highest accuracy). After each forward pass of your training stage, you get an output at the last layer. You then calculate the resulting loss $E$.
The consequence of each of your weights on your final loss is computed using its partial derivative. In other words, this is how much loss is attributed to each weight. How much error can be attributed to that value. The larger this value is the more the weight will change to correct itself (training).
$\frac{\partial E}{\partial w^k_{i, j}}$
How can we compute such a random partial derivative? Using the chain rule of derivatives, and putting together everything that led to our output during the forward pass. Let's look at what led to our output before getting into the backpropagation.
The forward pass
In the final layer of a 3-layer neural network ($k = 3$), the output ($o$), is a function ($\phi$) of the outputs of the previous layer ($o^2$) and the weights connecting the two layers ($w^2$).
$y_0 = o^3_1 = \phi(a^3_1) = \phi(\sum_{l=1}^n w^2_{l,1}o^2_l)$
The function $\phi$ is the activation function for the current layer. Typically chosen to be something with an easy to calculate derivative.
You can then see that the previous layers' outputs are calculated in the same way.
$o^2_1 = \phi(a^2_1) = \phi(\sum_{l=1}^n w^1_{l,1}o^1_l)$
So the outputs of the third layer can also be written as a function of the outputs of layer 1 by substituting the outputs of layer 2. This point becomes important for how the backpropagation propagates the error along the network.
Backpropagation
The partial derivative of the error in terms of the weights is broken down using the chain rule into
$\frac{\partial E}{\partial w^k_{i, j}}$ = $\frac{\partial E}{\partial o^k_{j}} \frac{\partial o^k_{j}}{\partial a^k_{j}} \frac{\partial a^k_{j}}{\partial w^k_{i,j}}$.
Let us look at each of these terms separately.
1. $\frac{\partial E}{\partial o^k_{j}}$
is the error caused by the output of the previous layer. For the last layer, using R2 loss, the error of the first output node is
$\frac{\partial E}{\partial o^3_{1}} = \frac{\partial E}{\partial y_{1}} = \frac{\partial }{\partial y_{1}} 1/2(\hat{y}_1-y_1)^2 = y_1 - \hat{y}_1$
In words, this is how far our result, $y_1$, from the actual target $\hat{y}_1$.
This is the same for all previous layers, where we need to substitute in the errors propagating through the network, this is written as
$\frac{\partial E}{\partial o^k_{j}} = \sum_{l \in L} (\frac{\partial E}{\partial o^{k+1}_{l}} \frac{\partial o^{k+1}_{l}}{\partial a^{k+1}_{l}} w^{k}_{j,l}) $
where L is the set of all neurons in the next layer $k+1$.
2. $\frac{\partial o^k_{j}}{\partial a^k_{j}}$
This is where the current layer's activation function will make a difference. Because we are taking the derivative of the output as a function of its input. And the output is related to the input through the activation function, $\phi$.
$\frac{\partial o^k_{j}}{\partial a^k_{j}} = \frac{\partial \phi(a^k_{j})}{\partial a^k_{j}}$
So just take the derivative of the activation function. For logistic function this is easy and its
$\frac{\partial o^k_{j}}{\partial a^k_{j}} = \frac{\partial \phi(a^k_{j})}{\partial a^k_{j}} = \phi(a_j)(1-\phi(a_j))$
3. $\frac{\partial a^k_{j}}{\partial w^k_{i,j}}$
a is simply a linear combination of w and the subsequent layers outputs. Thus,
$\frac{\partial a^k_{j}}{\partial w^k_{i,j}} = o_i$
Finally
You can see that the activation functions of your layers are evaluated separately in the backpropagation algorithm. They will just be added onto your ever growing back-chain as independent terms within your chain rule.


Categorical and ordinal feature data representation in regression analysis? I am trying to fully understand difference between categorical and ordinal data when doing regression analysis. For now, what is clear:

Categorical feature and data example:
Color: red, white, black
Why categorical: red < white < black is logically incorrect

Ordinal feature and data example:
Condition: old, renovated, new
Why ordinal: old < renovated < new is logically correct

Categorical-to-numeric and ordinal-to-numeric encoding methods:
One-Hot encoding for categorical data
Arbitrary numbers for ordinal data

Categorical data to numeric:

data = {'color': ['blue', 'green', 'green', 'red']}

Numeric format after One-Hot encoding:

   color_blue  color_green  color_red
0           1            0          0
1           0            1          0
2           0            1          0
3           0            0          1

Ordinal data to numeric:

data = {'con': ['old', 'new', 'new', 'renovated']}

Numeric format after using mapping: Old < renovated < new → 0, 1, 2

0    0
1    2
2    2
3    1

In my data I have 'color' feature. As color changes from white to black price increases. From above mentioned rules I probably have to use one-hot encoding for categorical 'color' data. But why I cannot use ordinal representation. Below I provided my observations from where my question arised.

Let me start with introducing formula for linear regression:

Let have a look at data representations for color:

Let's predict price for 1-st and 2-nd item using formula for both data representations:
One-hot encoding:
In this case different thetas for different colors will exist. I assume that thetas already derived from regression (20, 50 and 100). Prediction will be:

Price (1 item) = 0 + 20*1 + 50*0 + 100*0 = 20$  (thetas are assumed for example)
Price (2 item) = 0 + 20*0 + 50*1 + 100*0 = 50$

Ordinal encoding for color:
In this case all colors will have 1 common theta but  my assigned multipliers (10, 20, 30) differ:

Price (1 item) = 0 + 20*10 = 200$  (theta assumed for example)
Price (2 item) = 0 + 20*20 = 400$  (theta assumed for example)

In my model White < Red < Black in prices. Seem to be that correlation works correctly and it is logical predictions in both cases. For ordinal and categorical representations. So I can use any encoding for my regression regardless of the data type (categorical or ordinal)? This division in data representations is just a matter of conventions and software-oriented representations rather than a matter of regression logic?

The distinction between ordinal and categorical does matter. If in truth the difference between white and red was drastically different from red and black, your (10,20,30) ordinal model would not have performed well.

One hot encoding can learn the relationship between the ordinal values more finely, but throws out the information that the variables are related. Similarly, with insufficient data it is more likely to overfit.

Ordinal variables lessen those  problems but at the cost of forcing you to define the interval. There are a number of methods for defining the values of your ordinal variables, like rologit.


Which outlier detection can detect these outliers? I have a vector and want to detect outliers in it.

The following figure shows the distribution of the vector. Red points are outliers. Blue points are normal points. Yellow points are also normal.

I need an outlier detection method (a non-parametric method) which can just detect red points as outliers. I tested some methods like IQR, standard deviation but they detect yellow points as outliers too.

I know it is hard to detect just the red point but I think there should be a way (even combination of methods) to solve this problem.

Points are readings of a sensor for a day. But the values of the sensor change because of system reconfiguration (the environment is not static). The times of the reconfigurations are unknown. Blue points are for the period before reconfiguration. Yellow points are for after the reconfiguration which causes deviation in the distribution of the readings (but are normal). Red points are results of illegal modifying of the yellow points. In other words, they are anomalies which should be detected.

I'm wondering whether the Kernel smoothing function estimate ('pdf','survivor','cdf',etc.) could help or not. Would anyone help about their main functionality (or other smoothing methods) and justification to use in a context for solving a problem?

You may view your data as a time series where an ordinary measurement produce a value very close to the previous value and a re-calibration produce a value with a large difference to the predecessor.

Here are simulated sample data based on normal distribution with three different means similar to your example.

By calculating the difference to the previous value (a sort of derivation) you get the following data:

My interpretation of your description is, that you tolerates re-calibration (i.e. points on greater distance from zero, red in the diagram), but they must interchange between positive and negative values (i.e. corresponding to shift from blue state to the yellow one and back).

This means you can set up an alarm seeing a second one red point either on the negative or positive side.


Which direction of the normal vector that defines a Perceptron hyperplane is default? This slide here defines a hyperplane by using a normal vector $w$.

And claims that:

  "The hyperplane passes through origin"

Is it a general default parameter? w = [-1,1]?

I see a lot of figures with different directions.

Where $w$ and $b$ play in the opposite direction.

This question is for general hyperplane.

The normal direction is the direction that is perpendicular (also known as normal) to the plane.

Take a piece of paper (treat it as a plane), hold a pencil perpendicular to the paper. The pencil is parallel to the normal direction.

There is no such thing as the default direction as you see, a piece of paper that is oriented in a different manner would have a different normal direction.


Choosing sample from large dataset? How to choose sample from a large dataset such that each unique row from the dataset is selected at least once in the sample? Is there a way of doing this in python?

Let's say you have a dataframe with 10,000 rows, and  you have only 1,000 unique ones.

You can do:

df_unique = df.drop_duplicates()
df_sample = df.sample(n)

df_final = pd.concat([df_unique, df_sample], axis=0)

In the above code, n is the amount of sample you want.
In this way you can assure that every unique row is in your dataset and you have more samples on it.


How many features to sample using Random Forests The Wikipedia page which quotes "The Elements of Statistical Learning" says:

  Typically, for a classification problem with $p$ features, $\lfloor \sqrt{p}\rfloor$  features are used in each split.

I understand that this is a fairly good educated guess and it was probably confirmed by empirical evidence, but are there other reasons why one would pick the square root? Is there a statistical phenomenon happening there?

Does this somehow help decrease the variance of the errors?

Is this the same for regression and classification?

I think in the original paper they suggest using $\log_2(N +1$), but either way the idea is the following:

The number of randomly selected features can influence the generalization error in two ways: selecting many features increases the strength of the individual trees whereas reducing the number of features leads to a lower correlation among the trees increasing the strength of the forest as a whole.

What's interesting is that the authors of Random Forests (pdf) find an empirical difference between classification and regression:

  An interesting difference between regression and classification is that the
  correlation increases quite slowly as the number of features used increases.

Therefore, for regression often $N/3$ is recommended, which gives larger values than $\sqrt N$.

In general, there is no clear justification for $\sqrt N$ or $\log N$ for classification problems other than that it has shown that lower correlation among trees can decrease generalization error enough to more than offset the decrease in strength of individual trees. In particular, the authors note that the range where this trade-off can decrease the generalization error is quite large:

  The in-between
  range is usually large. In this range, as the number of features goes up, the
  correlation increases, but PE*(tree) compensates by decreasing.

(PE* being the generalization error)

As they say in Elements of Statistical Learning:

  In practice the best values for these parameters will depend on the problem,
  and they should be treated as tuning parameters.

One thing your problem can depend on is the number of categorical variables. If you have many categorical variables that are encoded as dummy-variables it usually makes sense to increase the parameter. Again, from the Random Forests paper:

  When many of the variables are categorical, using a
  low [number of features] results in low correlation, but also low strength. [The number of features] must be
  increased to about two-three times $int(log_2M+1)$ to get enough strength to
  provide good test set accuracy.


Approach for training multilingual NER I am working on multilingual (English, Arabic, Chinese) NER and I met a problem: how to tokenize data?
My train data provides sentence and list of spans for each named entity.
e.g.

[('The', 'DT'),
 ('company', 'NN'),
 ('said', 'VBD'),
 ('it', 'PRP'),
 ('believes', 'VBZ'),
 ('the', 'DT'),
 ('suit', 'NN'),
 ('is', 'VBZ'),
 ('without', 'IN'),
 ('merit', 'NN'),
 ('.', '.')]

[('你', 'PN'),
 ('有', '.'),
 ('没', 'AD'),
 ('有', '.'),
 ('用', 'VV'),
 ('过', '.'),
 ('其它', 'DT'),
 ('药品', 'NN'),
 ('？', 'PU')]

What the best way to tokenize input data? There are main different alternatives I consider: word level, wordpiece level, BPE.
BPE does not work with Chinese and Arabic because of unicode thing. I doubt about word level because I am not sure what is a word in Chinese.
What can you recommend me?

First, some clarification: BPE does work with Chinese and Arabic.
The only problem with Chinese is that there are no blanks between words, and therefore there is no explicit word boundary. In order to address that problem, normally you would segment words before applying BPE. For that, the typical approach is to use Jieba or any of its multiple ports to other programming languages. Other languages without blanks, like Japanese, may have their own tools to perform word segmentation.
Now, the answer:
Subword vocabularies are the norm nowadays. The norm also consists of finetuning one of the many pre-trained neural models available (BERT, XLNet, RoBERTa, etc), and the tokenization is imposed by the model you choose.
BPE is, in general, a popular choice for tokenization, no matter the language. Lately, the unigram tokenization is becoming popular also.
I suggest you take a look at the recent tner library, which builds on top of Huggingface Transformers to make NER very easy.


How do I deal with changing values in a categorical variable when I am aggregating customer records My requirement is to build a model to predict if a new customer will return to their website. I need to analyze what drives customer repeat for both new and returning customers. The only information given is the dates in which the customer has performed the transaction, the marketing channel through which they came, the deal which they used, their age and income, a flag which says whether they are a new customer or not. The data is for the last 6 years. A customer is regarded as a new customer if they don’t return within 24 months.

My approach is to aggregate the customer transaction records and derive the variables such as the Average number of visits, the average amount of transaction, create a flag variable which records did they return since their last visit.  My problem here is there are a lot of categorical variables as you can see (marketing channel they came through, the interface they used, their income, the deal program they used, etc) which keeps on changing for the same user in different transactions. How do I deal with those variables?

Any small hint at least is appreciated as I am stuck here completely.
Thanks in advance

The standard option would be to create one feature for each possible pair (variable, category) and use the frequency of this particular pair as value for this particular customer. If the number of times doesn't matter, it could be transformed into a binary feature, i.e. just indicating whether this customer has ever been seen with this category.

In case some categories appear very rarely, it's usually counter-productive to include them so you could have a minimum frequency in order to filter out rare pairs.


Is there a flexible way to get the original data indices from each cell of a confusion matrix? Let's say I have model A and model B, that I want to compare the performance of by inspecting a confusion matrix. They both produce a list of predictions, pred_A and pred_B, corresponding to a set list of ground truth values ground_truths. I can use something like sklearn.metrics.confusion_matrix in order to generate a confusion matrix for each of these models:
cm_A = confusion_matrix(ground_truths, pred_A) # e.g. [[5,1], [2,3]]
cm_B = confusion_matrix(ground_truths, pred_B) # e.g. [[6,0], [3,2]]

What I want to figure out is how to get the specific indices from each cell of the confusion matrix, corresponding to the ground truth class. This will let me investigate which specific data points are true-positives, false-positives, false negatives and true negatives, and let me search for patterns based on their input features. In the example above, are two of the false negatives the same false negatives in cm_A and cm_B? Or did all of the true positives switch with all of the false negatives between the two model predictions? If I knew their indices in the ground_truths list I could answer the question easily.
I can think of stupid/inefficient ways to do this, but I would love to find a solution that could scale to an arbitrary number of labels.

You can absolutely get this information, but not from the confusion matrices. You want to be comparing the prediction vectors themselves, not the confusion matrices, because as you've rightly identified, the confusion matrix dumps all the false postives / negatives into the same buckets (so that we can see how full each bucket is, also useful information).
You can do all sorts of comparisons of the prediction vectors to get the info you want (assuming you're using numpy arrays):
# Which examples did classifier A get wrong?
A_mistakes = np.invert(pred_A == ground_truths)

# Which examples did classifier B get wrong?
B_mistakes = np.invert(pred_B == ground_truths)

# Where did the classifiers make the same mistakes?
common_mistakes = A_mistakes == B_mistakes

# Where did the classifiers make a wrong prediction that the other didn't?
unique_mistakes = np.logical_xor(A_mistakes, B_mistakes)

You can then compare unique mistakes with A_mistakes or B_mistakes to trace them back to a classifier (or anything else you're interested in). You can also just np.sum the binary vectors to count the number of unique mistakes, common mistakes, etc.


Error: ValueError('%r cannot be used to seed a numpy.random.RandomState') I am getting this error message while trying to fit a model for the isolationForest algorithm.

raise ValueError('%r cannot be used to seed a numpy.random.RandomState'

Below is my code:

 import numpy as np
 import matplotlib.pyplot as plt
 from sklearn.ensemble import IsolationForest
 import pandas as pd

np.random.RandomState(1234)
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
df = pd.read_csv('E://Market_dat.csv',names=['EVENT_DT', 'MARKET_NAME', 'Duration', 'TOTAL_COUNTS'],skiprows=1,index_col=0)
for column in df.columns:
if df[column].dtype == type(object):
    le = LabelEncoder()
    df[column] = le.fit_transform(df[column])
    np.random.get_state()

X_train, X_test = train_test_split(df, test_size=0.3)
print(X_test)
print(X_train)
X_outliers = rng.uniform(low=-4, high=4, size=(20, 2))
clf = IsolationForest(behaviour='new', max_samples=100,
                      random_state=df, contamination='auto')
clf.fit(X_train)

Can anyone give any insight as to why I might be getting this error?

Because you are creating a IsolationForest instance with random_state getting initialised to pandas.DataFrame. The code docs for random_state explain it as

  random_state : int, RandomState instance or None, optional (default=None). If int, random_state is the seed used by the random number generator; If RandomState instance, random_state is the random number generator; If None, the random number generator is the RandomState instance used by np.random.


My Keras bidirectional LSTM model is giving terrible predictions I am trying to predict velocity (dynamics) values for notes that make up a piece of music using a bi-directional LSTM, following this blog post pretty closely: http://imanmalik.com/cs/2017/06/05/neural-style.html

I think I have set up everything correctly, but when predicting my model seems to produce only gibberish. Here's a plot of a prediction done on a validation sample (y-axis is always time and x-axis is, for I., a note activation array; for II. the predicted velocities and for III. the true, expected velocities). As you can see, the predicted velocity values (in the middle graph) are identical for all time steps (save for a few in the very beginning and at the very end, but that's hard to see in the plot):

For this particular prediction I had trained my model with 44 batches of 4 samples each and for 1 epoch, but I have also tried training it for 20 epochs and with different batch sizes and it doesn't seem to give better results.

Using TensorFlow backend.
Loading numpy data ...
173 train sequences
10 test sequences
x_train shape: (173,)
y_train shape: (173,)
Setting up model ...
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
bidirectional_1 (Bidirection (4, None, 128)            123392
_________________________________________________________________
bidirectional_2 (Bidirection (4, None, 128)            98816
_________________________________________________________________
bidirectional_3 (Bidirection (4, None, 128)            98816
_________________________________________________________________
time_distributed_1 (TimeDist (4, None, 88)             11352
=================================================================
Total params: 332,376
Trainable params: 332,376
Non-trainable params: 0
_________________________________________________________________
None
Training model ...
Epoch 1/1
2018-03-10 14:47:27.650226: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.2 AVX
44/44 [==============================] - 2145s 49s/step - loss: 0.0783 - mean_squared_error: 0.0783
Saving model ...
Loss and metrics: [0.002714238129556179, 0.0027141105383634567]

So, in other words, I end up with a validation MSE of 0.0027 after 1 epoch. Here is the code for my Keras model:

number_of_notes = 88
input_size = number_of_notes * 2 # 88 notes * 2 states (pressed, sustained).
output_size = number_of_notes # 88 notes.

units = 64
dropout = 0.2 # Drop 20% of units for linear transformation of inputs.

model = Sequential()
model.add(Bidirectional(LSTM(units, return_sequences=True, dropout=dropout),
                        input_shape=(None,input_size),
                        batch_input_shape=(args.batch_size,None,input_size)))
model.add(Bidirectional(LSTM(units, return_sequences=True, dropout=dropout)))
model.add(Bidirectional(LSTM(units, return_sequences=True, dropout=dropout)))
model.add(TimeDistributed(Dense(output_size, activation='sigmoid')))
model.compile(loss='mse', optimizer=Adam(lr=0.001, clipnorm=10), metrics=['mse'])

print('Training model ...')

number_of_train_batches = np.ceil(len(x_train)/float(args.batch_size))
model.fit_generator(batch_generator(x_train, y_train),
                    steps_per_epoch=number_of_train_batches,
                    epochs=args.epochs)

Here is how I predict:

prediction_data = np.load(path)

# Copy prediction input N times to create a batch of the right size.
tiled = np.tile(prediction_data, [args.batch_size,1,1])

raw_prediction = model.predict(tiled, batch_size=args.batch_size)[0]
prediction = (raw_prediction * 127).astype(int) # Float -> MIDI velocity.

So what I am asking is, where am I likely to be going wrong with my model? Which parameters should I try changing in order to get better results? Or how can I change my model or training set-up to get better results? Etc. Any and all help is appreciated!

I think you cannot use a bi-directional LSTM for prediction, because of the time dimension of the music. I mean the backwards layer has to predict the latest value first and only after predicting it sees the sequence which gives the context- This is like you watch a reversed movie and yo have to guess how the first frame looks like without knowing the rest of it.

Also, check out stylenet of the imanmalik model. The interpretation lstm which does the prediction is unidirectional. Only the genre net units are bidirectional, but they get the predicted value already as input.


How to classify neutral sentiments using BERT We can do text classification as positive and negative as mentioned in below notebook. But is there any way to classify neutral sentiment also?

https://colab.research.google.com/github/google-research/bert/blob/master/predicting_movie_reviews_with_bert_on_tf_hub.ipynb

Actually, I want to know like what kind of changes do we need to make in the above notebook so that it can classify neutral sentiments also besides positive and negative.

Thanks in advance.

Welcome to DS.SE, @prashanth

Yes! In general, it is possible to classify the documents to more than two classes of positive and negative sentiments as long as you have such labels in your training set.

Please take a look at this for some more general information about the sentiment analysis, tools, and applications.


Will cross validation performance be an accurate indication for predicting the true performance on an independent data set? I feel that this question is related to the theory behind cross-validation. I present my empirical finding here and wrote a question related to the theory of cross-validation at there.

I have two models M1 and M2, I use the same data set to train them and perform cross validation using that same data set to find the optimal parameters for each model. Say eventually I found that M1 under its optimal parameter, performs better than M2 under its optimal parameter in terms of the 10-fold cross validation score. Now if I have another independent test data set with both predictors and labels and this test data set is generated from the same distribution of my training data set, then before I apply these 2 well-tuned model on that new test data set, can I claim or should I expect to see that M1 will still perform better than M2 over that new test data set?

I was playing Kaggle Titanic example. I have 2 xgboost model, M1 is well-tuned and M2 is less well-tuned in the sense that M1 has a better 10 fold cross validation performs on the training data set. But then when I submit both, I found that the less well-tuned model actually has a better scores on the test data set. How could that be? And if it is true, then what should we looking for when we fit the data to different models and tune the model parameters?

Here are my specific submission results: I did a random grid search

params_fixed = {'silent': 1,'base_score': 0.5,'reg_lambda': 1,
'max_delta_step': 0,'scale_pos_weight':1,'nthread': 4,
'objective': 'binary:logistic'}
params_grid = {'max_depth': list(np.arange(1,10)),
'gamma': [0,0.05,0.1,0.3, 0.5,0.7,0.9],
'n_estimators':[1,2,5,7,10,15,19,25,30,50],
'learning_rate': [0.01,0.03,0.05,0.1,0.3,0.5,0.7,0.9,1],
'subsample': [0.5,0.7,0.9], 'colsample_bytree': [0.5,0.7,0.9],
'min_child_weight': [1,2,3,5], 'reg_alpha': [1e-5, 1e-2, 0.1, 0.5,1,10]
}
rs_grid = RandomizedSearchCV(
          estimator=XGBClassifier(**params_fixed, seed=seed),
          param_distributions=params_grid,
          n_iter=5000,
          cv=10,
          scoring='accuracy',
          random_state=seed
)

Each time I change the variable n_iter. First, I set n_iter=10, it gives me a set of values of those hyper parameters, let's call this vector $\alpha_1$ and the cv score (accuracy rate) is 0.83389, then I use $\alpha_1$ to train my model and generate prediction on the independent test data set, and when I submit to Kaggle it generates true accuracy on the test data set 0.79426

Second, I set n_iter=100, it gives me $\alpha_2$ and the cv score is 0.83614, i.e., higher than the first one, makes sense, but when I submit to Kaggle, 0.78469, lower than the first one.

Third, I set n_iter = 1000, it gives me $\alpha_3$ and the cv score is 0.83951, i.e., higher than the second one, makes sense, but when I submit to Kaggle, 0.77990, lower than the second one.

Fourth, I set n_iter = 5000, it gives me $\alpha_4$ and the cv score is 0.84512, i.e., higher than the third one, makes sense, but when I submit to Kaggle, 0.72249, lower than the third one.

This is really frustrated. The model is getting better and better on the cross-validation score but when performed on an actual independent data set, its performance is getting worse and worse. Did I interpret the CV scores in the exactly opposite way? I see some paper mentioned that the CV score can be too optimistic for inferring the true test score. However, even if that is true, then I think the CV scores for all of my 4 models should be all optimistic about their own true test score, i.e., the order should preserve. But when applying on the real test data set, the order reversed.

The only reason I can imagine would be, that test data set has a different distribution than the training data set. However, if it is indeed the case, then I believe there is no method under then sun that can cure this problem.

First off, a pragmatic answer: don't discount the possibility that the test set comes from a somewhat different distribution than the data set you're using for training and cross-validation.  You might think that shouldn't happen, but in practice it does seem to occur.

That said, let's go with your hypothetical and assume that the test set comes from exactly the same distribution as the rest of your data.  In that case, it is possible for cross-validation to lead you astray about which model is better, if you're using cross-validation to select hyper-parameters.

You can use cross-validation to either (a) select hyper-parameters, or (b) estimate the accuracy of your model -- but not both at the same time.

It appears you're using cross-validation to select the optimal hyper-parameters: you try many different choices for the hyper-parameters, for each choice estimate accuracy of that choice using cross-validation, and select the best choice.  When you do that, there's no guarantee that the resulting accuracy (with the best parameter) will be predictive of performance on the test set -- it might be an overestimate (due to overfitting).  If it's more of an overestimate for M1 than M2, then you might see what you saw.

If you want to both select hyper-parameters and estimate accuracy, I suggest that you have a separate held-out validation set for estimating accuracy, or use nested cross-validation.  See https://stats.stackexchange.com/q/65128/2921 and http://scikit-learn.org/stable/auto_examples/model_selection/plot_nested_cross_validation_iris.html.


converting sql to width bins in r I want to create a histogram out of the range of a column named surfacearea with country data frame. I made an sql file

# In R
range <- max(country$surface) - min(country$surfacearea)

#in SQL
# part B Divide the range into 5 equal width bins
#SELECT CASE
#WHEN surfacearea  <  3415080 THEN "LOW"
#WHEN surfacearea <= 6830160 and surfacearea > 3415081 THEN "MEDIUM-LOW"
#WHEN surfacearea <= 10245240 and surfacearea > 6830161 THEN "MEDIUM"
#WHEN surfacearea <= 13660320 and surfacearea > 10245240 THEN "MEDIUM-HIGH"
#ELSE "HIGH" END AS `type`, COUNT(surfacearea)
#GROUP BY `type`
#ORDER BY `type` ASC;

I'm working on putting mySQL into r

You could use the dplyr package and its case_when function. Note the spacing is just for readability:

library(dplyr)

country %>%
  mutate(
    type = case_when(
      surfacearea <  3415080                           ~ "LOW",
      surfacearea >  3415081 & surfacearea <=  6830160 ~ "MEDIUM-LOW",
      surfacearea >  6830160 & surfacearea <= 10245240 ~ "MEDIUM",
      surfacearea > 10245240 & surfacearea <= 13660320 ~ "MEDIUM-HIGH",
      TRUE                                             ~ "HIGH"
    )
  )

The last line is the catch-all case. For more see ?case_when. To complete your pipeline, you could do:

country %>%
  group_by(type) %>%
  mutate(
    type = case_when(
      surfacearea <  3415080                           ~ "LOW",
      surfacearea >  3415081 & surfacearea <=  6830160 ~ "MEDIUM-LOW",
      surfacearea >  6830160 & surfacearea <= 10245240 ~ "MEDIUM",
      surfacearea > 10245240 & surfacearea <= 13660320 ~ "MEDIUM-HIGH",
      TRUE                                             ~ "HIGH"
    )
  ) %>%
  arrange(type)

There is also the between function which could further help the readability, but that function is a shortcut for x >= left & x <= right so it might not work since your first bucket is strictly less-than.


How to decide for the contamination value (proportion of the outliers) in my dataset? I should decide on the contamination value while using the Isolation Forests algorithm (I am using the sklearn implementation). Otherwise, sklearn's default is 0.1.

I am worried if I decide for this parameter with experimenting on my current dataset, it will differ by time as I would be receiving new data points very soon.

How can I know that this parameter better than any other, just by experimenting it with outlier removal?

Thanks.

If you want to experiment something like that I think you can add some outliers into your data for that respective field and see what are its effects.

If you find that these are prone to outliers(most of of the features are), then you can handle situation by removing such outliers before appending to the existing data.

What is your target variable?
Did you do any kind of Correlation analysis/ Predictor Importance?


Orange file import method for raw binaries I tried to add a  binary import method to
    class FileFormat(metaclass=FileFormatMeta)
in io.py, but gave up trying to adapt a copy of the CSV method since so much is specific to CSV issues and the comments are sparse...

Basically, given an Ncols x Mrows raw file of known type and meta data, one should be able to supply the shape, sample rate etc as defaults, and then one "only" needs to satisfy the parameters for Table.from_numpy(). It is the creation of Orange-specific objects that have me stuck, and mainly the domain object:

def data_table_from_bin(cls, data, headers=None):
    ...
    domain = Domain(attrs, clses, metas)

    table = Table.from_numpy(domain, X, Y, M, W)
    return table

Has anyone else tried this, or have suggestions?

What is the problem exactly?

When you create Domain you have to specify each Feature eg:

from Orange.data import Domain, Table, ContinuousVariable, DiscreteVariable

d = Domain([ContinuousVariable.make('Attribute 1'), DiscreteVariable.make('Attribute 2')])

Here is a link to doc, regarding Domain (yes, it's not the best)


How to identify patterns in dataset I have a dataset that pertains to calls received to a hospital emergency helpline. My task is to identify patterns in the data.
How do I approach examining the dataset to identify patterns?
Extract from dataset.

Try general linear model to see how the features interact.  Decide if the problem is classification or trend or anomaly then post back your discovery then I can help .
I applied K-Means, TSNE dimension reduction, and PCA and saw maybe two clusters.
it is not conclusive clustering.  There were a few observations discovered by severity, age, and gender of the heart events.
from sklearn.decomposition import PCA
from scipy.cluster.vq import vq, whiten
from scipy.cluster.hierarchy import linkage, fcluster,dendrogram
from sklearn.preprocessing import LabelEncoder
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
from sklearn.cluster import KMeans
from sklearn.manifold import TSNE

calltype=['Emergency','Emergency','Emergency','Hosp to Hosp','Emergency','Emergency','Emergency','Emergency','Hosp to Hosp','Hosp to Hosp','Emergency','Hosp to Hosp','Emergency','Emergency','Emergency','Emergency','Emergency','Emergency','Emergency']
district=['District 1','District 2','District 3', 'District 4','District 5','District 1','District 2','District 3','District 4','District 5','District 1','District 2','District 3','District 4','District 5', 'District 1','District 2','District 3','District 4']
gender=['Male','Male','Male','Male','Female','Male','Male','Female','Female','Male','Male','Male','Male','Male','Male','Female','Male','Male','Female']
complaint=['Chest Pain','Heart Problem','Cardiac Arrest','Heart Problem','Cardiac Arrest','Chest Pain','Cardiac Arrest','Heart Problem','Heart Problem','Heart Problem','Heart Problem','Chest Pain','Heart Problem','Chest Pain','Heart Problem','Heart Problem','Heart Problem','Chest Pain','Heart Problem']
 age=[28,50,76,45,52,34,44,55,86,65,60,36,45,65,60,35,80,46,70]
 area=['Rural','Rural','Rural','Urban','Urban','Urban','Urban','Rural','Rural','Rural','Urban','Urban','Urban','Rural','Rural','Urban','Rural','Urban','Urban']

 df=pd.DataFrame({'calltype': calltype,'district':district,'gender':gender,'complaint':complaint,'age':age,'area':area})
 df['severity']=df['complaint'].apply(lambda row: 3 if row=='Chest Pain' else 2 if row=='Heart Problem' else 1 if row=='Cardiac Arrest' else 0)

 encoder=LabelEncoder()
 df['district']=encoder.fit(df['district']).transform(df['district'])
 df['gender']=encoder.fit(df['gender']).transform(df['gender'])
 df['area']=encoder.fit(df['area']).transform(df['area'])
 df['calltype']=encoder.fit(df['calltype']).transform(df['calltype'])
 print(df)

 sns.countplot(x='gender',hue='severity',data=df)
 plt.legend(['Cardiac Arrest','Heart Problem','Chest Pain'])
 plt.show()

 print('Gender 0=Female 1=Male')
 fp=df.pivot_table(index=['severity'],columns=['gender'],values=['age'],aggfunc='mean')
 print(fp)

 print("\n More males had severe heart problems")

 fig,ax=plt.subplots(figsize=(10,8))
 x=np.linspace(0,len(df),len(df))
 ax.bar(x,df['severity'],alpha=0.3,color='green')
 twin_ax=ax.twinx()
 df[['age']].plot(ax=twin_ax, c='red')
 plt.show()

 print('\nAges 30-40 had high frequencies of cardiac arrest')

 print("\nK-Means\n")
 labels=df['gender']
 xs=df['age']
 ys=df['severity']
 points=np.column_stack((xs,ys))
 model = KMeans(n_clusters=3)
 model.fit(points)
 print(model.inertia_)
 new_points=np.random.permutation(points)
 new_labels = model.predict(new_points)
 print(new_labels)
 xs = new_points[:,0]
 ys = new_points[:,1]
 plt.clf()
 # Make a scatter plot of xs and ys, using labels to define the colors
 _=plt.scatter(xs,ys,c=labels, alpha=0.5)
 # Assign the cluster centers: centroids
 centroids = model.cluster_centers_
 # Assign the columns of centroids: centroids_x, centroids_y
 centroids_x = centroids[:,0]
 centroids_y = centroids[:,1]
 # Make a scatter plot of centroids_x and centroids_y
 _=plt.scatter(centroids_x,centroids_y, marker='D',s=50)
 plt.show()

 print("\nTSNE\n")

 model=TSNE(learning_rate=100)
 X=df[['calltype','district','gender','age','area','severity']]
 transformed=model.fit_transform(X)

 xs=transformed[:,0]
 ys=transformed[:,1]

 plt.scatter(xs,ys,c=labels)
 plt.show()

 print("\nPCA\n")
 pca=PCA(n_components=3)
 pca.fit(X)

 transformed=pca.transform(X)
 print(transformed.shape)

 xs=transformed[:,0]
 ys=transformed[:,1]
 plt.scatter(xs,ys,c=labels)
 plt.show()


Datamodel for cluster analysis terms & segmentation Hey fellow data lovers!

I have a data modelling problem for a clustering analysis that I can't wrap my head around. Perhaps I'm thinking to advanced and I should simplify my analysis.

I have 2 data sets:

Persons with their properties:

PersonId
Gender
Age
Country
Etc.

Conversations and their properties:

ConversationId
Sentences of the conversations
Keyword of a sentence + it's frequency
Polarity of the sentence (pos, neg or neutral)

I would like to cluster terms that have the same polarity for a group of people. Eg. male persons between 20 and 35 speak positively about economy and neg about privacy
Eg. male persons from BE, NL and DE speak neg about climat changes.

Problem is that I need to cluster multiple terms for an undefined group of people. For one term (eg. privacy) it would be 'easy' to determine the properties of the different segments as this is a classification problem. As I would like to cluster multiple terms together, I'm strugling to model my data as this means I have multiple records per person (they can talk about multiple terms).

Assumption: you can assume I have one record per person per term.

clustering algo would take any data type as long as it is measurable, ideally shouldn't be string (which can be translated into vectors using topics).
If you tell me what application you are using i can guide you on how to go about it step by step