Dealing with NaN (missing) values for Logistic Regression- Best practices? I am working with a data-set of patient information and trying to calculate the Propensity Score from the data using MATLAB. After removing features with many missing values, I am still left with several missing (NaN) values. I get errors due to these missing values, as the values of my cost-function and gradient vector become NaN, when I try to perform logistic regression using the following Matlab code (from Andrew Ng's Coursera Machine Learning class) :

[m, n] = size(X);
X = [ones(m, 1) X];
initial_theta = ones(n+1, 1);
[cost, grad] = costFunction(initial_theta, X, y);
options = optimset('GradObj', 'on', 'MaxIter', 400);

[theta, cost] = ...
    fminunc(@(t)(costFunction(t, X, y)), initial_theta, options);

Note: sigmoid and costfunction are working functions I created for overall ease of use.

The calculations can be performed smoothly if I replace all NaN values with 1 or 0. However I am not sure if that is the best way to deal with this issue, and I was also wondering what replacement value I should pick (in general) to get the best results for performing logistic regression with missing data. Are there any benefits/drawbacks to using a particular number (0 or 1 or something else) for replacing the said missing values in my data?

Note: I have also normalized all feature values to be in the range of 0-1.

Any insight on this issue will be highly appreciated. Thank you

Use caution when removing features with missing values. Sometimes the fact that a feature has missing values is valuable data in and of itself.

What you are asking about is called imputation. A google search will give you a TON of literature about various imputation methods. Here are some of the most common:

-Mean imputation: replace all missing values with the mean of other values

-Mode imputation: same as above, with the mode

-Median imputation: get the idea? Replace all with the median

-Multiple Imputation by Chained Equations (MICE): basically do linear regression to predict the missing values based on other variables. This is a high-variance solution so some domain knowledge may be necessary. This can also take a long time. There are packages available to do this in R and Python.

-Replace missing values with 0 or 1 (looks like you have already tried this)

Play around with the different methods and see which ones give you the best result!


Question on Scipy - Minimize. Adding additional constraints I am trying to using scipy minimize function for the following optimization:

V = np.matrix(pd.read_csv('V.csv'))`

R = np.matrix(pd.read_csv('R.csv', index_col = 'Ticker'))`

w0= list()
for i in range(0, 84):
    w0.append(1/84)

def calculate_portfolio_var(w,V):
    w = np.matrix(w)
    return (w*V*w.T)[0,0]

cons = ({'type': 'eq', 'fun': lambda x:  np.sum(x)-1.0})
myBound = [(0, 1) for i in range(0, 84)]
res= minimize(calculate_portfolio_var, w0, args=V, method='SLSQP',constraints=cons, bounds = myBound)

where V is the variance-covariance matrix, R is the series of annualized return of stocks.

In addition to the 2 constraints (cons and myBound), I want an additional constraint that the result portfolio return, which is the weighted average of the result weights and stock returns, be equal to certain number and the number of stocks to be less than equal to certain number..

For example, it should look like:

cons = ({'type': 'eq', 'fun': lambda x:  np.sum(x)-1.0},
        {'type': 'eq', PortfolioReturn = 10%,
        {'type': 'ineq', number of result stocks <= 40)

I am not so familiar with the Scipy minimize, and I would appreciate if someone can help me.

I assume problem is in accessing your columns, elements of the V matrix, since you say:
"it should look like:"

Well to actually write down these constraints, you just need to access your columns in the functions your defined in the cons dictionary.

Lets say you have your data V

> array([[ 0.00749589,  0.01255155,  0.02396251,  0.04750988, 0.09495377]
>        [ 0.01255155,  0.02510441,  0.04794055,  0.09502834,  0.18996269],
>        [ 0.02396251,  0.04794055,  0.09631614,  0.19092151,  0.38165151],
>        [ 0.04750988,  0.09502834,  0.19092151,  0.38341252,  0.7664427 ],
>        [ 0.09495377,  0.18996269,  0.38165151,  0.7664427,   1.53713523]])

You should define your constraints dictionary in the following way (for example):

> cons = ({'type': 'eq', 'fun': lambda x:  x[0] - 2 * x[1] + 2},
> {'type': 'ineq', 'fun': lambda x: -x[0] - 2 * x[1] + 6},
> {'type': 'ineq', 'fun': lambda x: -x[0] + 2 * x[1] + 2})

While indexing desired columns inside of the lambda function.


Why my svm.SVC.fit( ) (linear kernal) run so long time? I am using sklearn.svm.SVC( ) to train & test my dataset. 80% are used for training, 20% are used for testing.

Here is my Python code:

data = pd.read_csv(trainPath, header=0)

X = data.iloc[:, 5:17].values
y = data.iloc[:, 17:18].values

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

print(X_train.dtype, y_train.dtype)  # float64 int64

clf = svm.SVC(kernel='linear').fit(X_train, y_train.ravel())
print('done')

y_pred = clf.predict(X_test)

print("Accuracy:", metrics.accuracy_score(y_test, y_pred))
print("Precision:", metrics.precision_score(y_test, y_pred))
print("Recall:", metrics.recall_score(y_test, y_pred))
tn, fp, fn, tp = confusion_matrix(y_test, y_pred).ravel()
print(tn, fp, fn, tp)

For data.shape = 30,000 x 13, it runs around 15 mins.

For data.shape = 130,000 x 13, it runs more than 1 hour.

Why it runs so long time, I don't think it is normal.

i5, 2.8GHz, 16.0 GB memory

From scikit-learn documentation:

  The implementation is based on libsvm. The fit time scales at least quadratically with the number of samples and may be impractical beyond tens of thousands of samples. For large datasets consider using sklearn.linear_model.LinearSVC or sklearn.linear_model.SGDClassifier instead, possibly after a sklearn.kernel_approximation.Nystroem transformer.

Yo can change

clf = svm.SVC(kernel='linear').fit(X_train, y_train.ravel())

by

from sklearn.svm import LinearSVC
clf = LinearSVC(random_state=0, tol=1e-5)
clf.fit(X_train, y_train.ravel())


Does importance of SVM parameters vary for subsample of data? I'm training a support vector machine classifier (SVM) on 100.000 observations. I'd like to try different parameter combinations (including kernel types) using cross-validation. However, it is computationally expensive to optimize over these different parameter values. Therefore, I'd like to test them on a subset of 5.000 observations on a validation set and then based on those results choose the parameters for the final model that is trained on 100.000 observations.

My question is then whether there is something intrinsic in these parameters that will make this approach problematic. More specifically, is there a parameter, where training on the full data set will require a different value for optimal performance?

I'm using the following kernels with their parameters in parentheses:

rbf (gamma, C)
poly (degree, C)
sigmoid (gamma, C)
linear (gamma).

In theory, if you have a large enough random sample of your data set, it should be representative of the characteristics in the larger population of data that will affect the relationship between parameter values and performance. It seems to me that a $5\%$ sample of your data might be too small for what you're trying to do. When I'm performing cross-validation on a test/dev data set, I usually set aside somewhere in the neighborhood of $10\%$ for the hold-out test/evaluation data, and use $90\%$ for my development work, but, as you pointed out, this may be too big given the computational constraints you're up against. Depending on the type of application you're working on, you could look at feature distribution in your sampled data and compare it to that in the larger population. If the two sets are comparable, then you might be ok to do your dev. work on the smaller data set. I say it depends on the type of application you're developing, because many statisticians would consider this cheating, in that it's generally not good model evaluation practice to look at hold-out data at all. Here's the procedure I'd recommend:

Separate out $10\%$ of your data for hold-out evaluation.
Divide the remaining $90\%$ into $5\%-10\%$ parameter optimization, and the remainder for final parameter eval.
Compare the feature distributions of the parameter optimization and final parameter eval data sets, if they're not comparable, redraw the samples (within the context of the $90\%$ development sample)
When you have a good subsample, run your parameter optimization experiments on the small parameter optimization data set.
With your final parameter settings, evaluate with cross-validation on the the $90\%$ combined parameter optimization and final parameter eval data sets.
Perform final model analysis by training on the $90\%$ data set and classifying on the $10\%$ hold-out evaluation set.

It might also be worth looking into optimizing your analytics pipeline. For example, is file I/O, feature generation and feature extraction part of the the workflow you're using, or have you done all of that off-line and are just concerned with the SVM evaluation part?


Metrics/Methods for deciding duration of video retention for on-demand websites This might be a general question but I thought this might be the best place to brainstorm. If I had a video website that only wants to retain videos on-demand for a certain number of days before getting rid of them to maximize operating dollars, what sort of data measures/methods would I want to consider or use in general to decide the time period?

Just like a typical website, the video can be tracked for views, followers etc. and the trade off could be on views/operating cost

Clustering can be an ideal choice here. From the question, seems the data will most probably be in continuous format. Essentially, clustering is a method of finding groups of similar objects. The similarity between the objects is determined by the type of distance measure being used.
With this background, you can start by finding the videos that are most liked or disliked by people. Measurements like mutual information gain can be helpful.


Data normalization in nonstationary data classification with Learn++.NSE based on MLP I need to predict technical aggregate condition using vibration monitoring data. We consider this data to be nonstationary i.e. distribution parameters and descriptive statistics are not constant. I found that one of the best algorithms for such tasks in Learn++.NSE and we us it with MLP as a base classifier.

As I know, it's necessary no normalize data for operations with ANN. We decided to normalize using mean, stdev and sigmoidal function. We train networks of ensemble with sets with different values distribution parameters.

So, my questions are the following

How to normalize new trainig set during previous networks evaluation? Problem is in description statistics change
How to normalize input data while ensemble usage? Current statistics differ from the previous ones.

Batch normalization is critical technique for fast learning
speed and generalization [8]. In this paper, batch temporal
normalization layer is proposed for stationarity of input
time series.


Can k-means clustering get shells as clusters? Imagine you have $k$ classes. Every class $i$ has points which are follow a probability distribution, such that their distance to 0 is $i$ in mean, but this distance follows a normal distribution. The direction is uniformly distributed. So all classes are in shells around the origin 0.

Can $k$-means get these shells when you choose the "right" distance metric?
(Obviously it can't find it if you take the euclidean metric, but I wonder if there is any metric at all or if this problem is inherently unsolvable by $k$-means, even if you know the number of clusters $k$)

You cannot just use arbitrary distance functions with k-means.

because the algorithm is not based on metric properties but on variance.

https://stats.stackexchange.com/q/81481/7828

Fact is that k-means minimizes the sum of squares. This does not even give you the "smallest distances" but only the smallest squared distances. This is not the same (see: difference between median and mean) - if you want to minimize Euclidean distances, use k-medians or if you want other distances PAM (k-medoids).

You can generalize k-means to use a few more distances known as "Bergman divergences" and you can do some variant of the kernel trick. But that is not very powerful, because you don't have labels for optimizing the kernel parameters! Still, that could be what this exercise question is up to... If your "shells" are indeed centered at 0, then you can transform your data (read: kernel trick done wrong) to angle+distance from origin, and k-means may be able to cluster the projected data (dependent on the not well defined scaling of the axes).
Or the textbook did not realize that a kernel k-means has been proposed long ago. Then the argument is probably this: the mean of each shell is 0, and thus the shells cannot be distinguished. This clearly holds for unmodified k-means.


How to present the final model (e.g. random forest)? I've run random forest on my dataset (imbalanced, binary target class) and used cross validation to tune the parameter and use recursive feature elimination with cross-validation to get the subset of features.

Then I can present the optimal parameter setting as well as the set of features for model fitting.

But how about the probabilistic threshold?

How can I conclude a threshold as the important part of the final model?

Also using CV?

If this threshold is not stable among folds of CV, what to do next?

The probability threshold can be changed and you would obtain different results for accuracy / recall / precision / F1 score (the objective). You could see this as an aspect of the parameters you have tuned to obtain the best objective.

Similarly for CV, the out-of-training fold objective would be reported together with the other parameters, such as number of trees and predictor subset size. One reason I could think of regarding the instability of threshold across CV folds is that the CV folds were not split up using stratified sampling, resulting in different proportions of positives observed across the folds.


Taking average of multiple neural networks? I'm fitting a neural network using a very small data set,
so try splitting the data into training and validation sets.
(there is a separate test set)
If I split training/validation randomly multiple times,
construct a neural network for each training/validation split,
and take average of predicted values of the neural networks on the test set,
can it be called a ensemble model?
Or is there a specific name for such a technique?

Edit: I just found that a similar technique is called 'repeated random sub-sampling validation,' but the RRSSV splits the data into training and test set (though it is called 'validation data' according to Wikipedia, it's actually test data). My method splits the given data into training and validation set, and use separate test data.
I think my method can also be called RRSSV.

I think even this method is also called Ensemble Method.

How could I conclude that?

You might have heard about this algorithm named Random Forest,
what does it do? It take data randomly at row level and column level
builds different trees and takes an average of it. It is also
considered as one of the best algorithm for Prediction and
Classification. Can go through this explanation for better
understanding. Random Forest is called Ensemble model(of trees).

A suggestion, as you have mentioned in the question that you have very less data, at that time the models cannot generalize well and you cannot achieve good results. If you have any way to increase the dataset size by collecting more data, it could aid you in achieving better accuracy. This is also explained the link attached(explanation). Do go through and let me know if you have any additional questions.


Create new column in Orange using Python Scripting Widget This is a fairly basic question. How do you create a new column in Orange - based upon an existing column?

For example:

NewCol = 2 * ExistingColumn

or

NewCol = len(ExistingColumn)

I would like to use the Python Script Widget - not the feature widget.

You'd think it would be basic. So far I'm not finding it to be so. Here's one example that worked for me. I'm sure it can be much simpler.:
from Orange.data import ContinuousVariable, DiscreteVariable, Domain

out_data = in_data.copy()
var1 = ContinuousVariable("NewCol")
domain = out_data.domain
new_domain = Domain(attributes=domain.attributes + (var1,), metas=domain.metas, class_vars=domain.class_vars)
out_data = out_data.transform(new_domain)

i2 = out_data.domain.index(out_data.domain["NewCol"])
i1 = out_data.domain.index(out_data.domain["ExistingColumn"])
out_data.X[:, i2] = out_data.X[:,i1] * 2


What is the purpose of reward threshold in OpenAI Gym? I've seen that OpenAI Gym environments can be registered with an optional reward threshold (reward_threshold) which represents:

  The reward threshold before the task is considered solved

How does this value affect the learning process? Or does one have to manually compare the reward obtained in each episode with the reward_threshold and stop the learning process if it surpasses it?

I did some digging in the gym codebase, and at least as of v.0.18.0, gym itself doesn't appear to be using reward_threshold at all (as opposed to max_episode_steps, which is used to compute the Done signal when stepping in the environment).
So one would have to manually access this field from the env if they wanted to use it.


Learning Resources for Recommendation system Beginner here:

Could you please suggest some of the learning resources (books/youtube/articles) for beginners who want to build a recommendation system for their organization.

Have no clue about it except for Amazon and Netflix example.

If you have a technical background, I would certainly recommend the chapter 9 from Mining Massive Datasets by Jure Leskovec. Video lecture of the topic is also available there. Look for chapter 9. This blog could also give you some overall insight.

Please keep in mind that, ML-wise, recommendation is a more creativity-related topic than other classic questions like Classification or Regression and indeed all those classic problem could be a step in building a reasonable recommender engine.


When to split data into multiple regression models instead of one model? I'm playing with regression models in scikit-learn. The goal is to predict how much inventory we should purchase for the next 90 days. My data set has hundred of product categories. Each category has many unique features that do not apply to every category.

For Example: Shirt category could have "size" and "color" features where as the category Razors could have a "number of blades" feature.

Should I split my data up by category and make a different model for each? Or is it suffient to have one model in which I keep the products category as one of the features?

You should split them by category since their features do not apply to each category.

Under certain circumstances that perhaps you manage to group some categories together based on some business logic, then perhaps you can build less models.


Why are parameter updates downscaled by uncentered variance (instead of centered variance) in Adam optimizer? In Adam optimizer algorithm, parameter updates are computed as follows:

$\theta_t \leftarrow \theta_{t-1} - \alpha \frac{\hat{m}_t}{\sqrt{\hat{v}_t}+\epsilon}$

Where $\hat{m}_t$ is a bias-corrected moving average of gradient means:

$\hat{m}_t \propto \beta_1 m_{t-1} + (1 - \beta_1) g_t$

And $\hat{v}_t$ is a bias-corrected moving average of uncentered variance:

$\hat{v}_t \propto \beta_2 m_{t-1} + (1 - \beta_2) g_t^2$

I understand the idea that, if the mean is large compared to the variance, gradients can be trusted more and you should take larger steps. But it this case, wouldn't it be more logical to use centered variance instead of uncentered variance in the above formula?

I looked for the answer and found this thread but I still wasn't convinced by the answer. Essentially the explanation was that uncentered variance is roughly equal to centered variance because the mean gradient is approximately zero. But if the mean gradient is zero, it means that you are equally likely to move in any direction, so you aren't optimizing anything...

So, why is Adam optimizer using uncentered variance? Why not centered variance which would seem a lot more logical?

If anything, you don't have access to the mean. When we talk about the variance, we talk about the variance of a distribution. The distribution in this case is the total collection of gradients at each step of the optimization process, which you don't know until you're done.


How to get summary statistics in Orange? I just started using Orange, and am having trouble finding how to get basic summary statistics, like the n (count), average, and standard deviation.

Is there a widget that does this and I'm simply overlooking it?

I found 3 ways to do it :

you have to use the Data Info widget who gives the number of rows of your dataset and the Box Plot widget who prints the average and the standard deviation of each feature in your dataset. Pro: easy to use. Con: you are unable to use this data.
you can create a Python script who will compute the average and the standard deviation using the statistics package. Pro: you can reuse your data. Con: you have to write a script and format the output data correctly.
it exists an add-on call "Orange3-Timeseries" with a Moving Transform widget who can compute the average and the standard deviation of your series. Pro: easy to use. Cons: the package is still a beta-version and have some bugs, the widget only works with timeseries.


How to add Error Bar to Matplotlib line plot? I have the following dataset which I use to plot a line plot. The plot is obtained as the mean of values obtained from the data. I want to add error bars to this plot which shall show the standard deviation. I have looked up to different answers but in most of them they had defined x and y explicitly, but here I calculate the plot directly from the dataframe. How to add error bar to this plot?
Dataframe df
UserId     |   date                 |-7|-6|-5|-4|-3|-2|-1|0 |1 |2 |3 |4 |5 |6 |7
     1      2009-10-17 17:38:32.590 |0 |0 |0 |0 |0 |0 |1 |0 |1 |0 |0 |0 |0 |0 |0
     2      2009-10-19 00:37:23.067 |0 |0 |0 |0 |0 |1 |1 |0 |1 |0 |0 |0 |0 |0 |0
     3      2009-10-20 08:37:14.143 |0 |0 |0 |0 |0 |0 |1 |0 |0 |0 |0 |0 |0 |0 |0
     4      2009-10-21 18:07:51.247 |0 |0 |0 |0 |0 |0 |1 |0 |0 |0 |0 |0 |0 |0 |0
     5      2009-10-22 21:25:24.483 |0 |0 |0 |0 |0 |0 |1 |0 |0 |0 |0 |0 |0 |0 |0

Code
badges = ["A", "B", "C"]

for badge in badges:
  res.iloc[:,2:].mean().plot(kind='line', label = badge)

Output

Why not compute the x values explicitly? In the dataframe above,
x = [c for c in res.columns() if isinstance(int, c)]

Then, the code can be as follows.
badges = ["A", "B", "C"]
fig, ax = plt.subplots()
for badge in badges:
    means = res.iloc[:,2:].mean()
    std = res.iloc[:, 2:].std()
    means.plot(kind='line', label = badge)

    ax.plot(means)
    ax.fill_between(x, means.sub(std), means.add(std), color='b', alpha=.1)

Note: In the code above, you use badges only in the plot. Maybe computing means and std should be done outside the cycle? I am not sure because your sample data frame above does not have badges.


Probability distribution in input-output pairs This question might sound silly. But I have been wondering why do we assume that there is
a hidden probability distribution between input-output pairs in machine learning setup ?

For example, if we want to learn a function $f: \mathcal{X} \rightarrow \mathcal{Y}$, we generally tend to assume a probability distribution $\rho(x,y)$ on $Z=\mathcal{X} \times \mathcal{Y} $ and try to minimize the error
$$
\mathcal{E}(f) = \int (f(x)-y)^2 \ d\rho(x,y)
$$

Is the probability distribtution $\rho$ inherent to the very nature of $Z$ or depends on $f$ ?

Can anyone please provide a good intuitive explanation for this ?

Some notes:

In the case of supervised learning you assume there is a function $f:\mathcal{X} \rightarrow \mathcal{Y}$, which means there is a connection somehow between inputs and outputs and you want to use it in order to predict. But how this function might look? Without considering the model itself, usually what happens is that there is some noise. And this noise can be in $Y$ and also can be in $X$. This noise gives use the probabilistic setup of the problem, because without it we would have only to solve some equations.
Now is important to understand that the noise defines the distribution. So a random variable can be imagined as a function having something fixed and well defined and something not fixed, but taking values according with a distribution. If the variable part does not exist, that you would not have a random variable, right, it would be a simple formula. But its not. Now the $P(X)$ incorporates what happens in $X$ alone, and $P(Y)$ what is in $Y$ alone. When you predict the decision theory says that you are interested in saying which is the most probable value $y_i$ given some input value $x_i$. So you are interested in finding $P(Y|X)$.
A joint probability is not always completely described by marginal probabilities. In fact, is completely described only when marginal probabilities are independent. Which means for r.v. $X, Y$ knowing $P(X)$ and $P(Y)$ does not put you in position to know the joint density $P(X, Y)$ (thought for independence you have  $P(X,Y)=P(X)P(Y)$).
From here you can go directly and try to estimate $P(Y|X)$. In fact if your only interest in only in prediction, this might be a fair bet. A lot of supervised learning algorithms tries to estimate directly this probability. They are called discriminant classifiers. The reason is because they are used to classify something to  the class with maximal conditional probability, you discriminate (choose) the maximum value.
Now arriving at your question. Notice the obvious $P(X,Y) = P(Y|X)P(X)$ than you see that by trying learning the joint probability, you also learn the conditional (what you need for prediction). This kind of approach is called generative, because knowing the joint probability you can not only predict, but you can generate new data for your problem. More than that, knowing the join probability can give you more insights related with how what are you model works. You can find such kind of information which is not contained only in marginal distributions.

Some final notes:

Technically you do not minimize the error function, but it's expectation. The error function remains as it is.
$\mathcal{Z}$ is only a domain it's impossible to describe a probability only by it's domain.

It's late, I hope I was not completely incoherent.


Cohort analysis over 3 months I am trying to look at the customer retention & churn by using cohorts for an e-commerce usecase. From a business perspective, a client is defined as churned if it hasn't performed any transactions in the following 3 months after the month of its first purchase.

Now, I've done some R&D but all studies that I've found look at the churn on a monthly basis and thus all the coding is done this way.

Now, I am trying to adapt my code in Python so that the aggregation is done on a 3-months level rather than at monthly level.

grouped = txn_new_cif.groupby(['CohortGroup', 'Txn_date'])
cohorts = grouped.agg({'CIF': pd.Series.nunique,
                     'Merchant': pd.Series.nunique,
                     })[![enter image description here][1]][1]

What I would want is that for 2018-05 cohort, the 06-2018 to look at any transactions performed in 2018-06, 2018-07 and 2018-09. If there is any transactions performed by a user across this 3 months interval, then he would be considered active ( For 2018-07 projection, look for transactions in 2018-07, 2018-08 and 2018-09).

To demonstrate how this can be done, I created some sample/dummy records. See my answer below with comments explaining each step:

(1) Prepare data and create a column to capture all ID's of active customers

import pandas as pd
import numpy as np
from dateutil.relativedelta import relativedelta

# Create a dataframe with five customers from three cohorts
df = pd.DataFrame({'txn_new_cif': [1, 1, 2, 3, 4, 5],
                   'CohortGroup': [np.datetime64('2018-01-01'), np.datetime64('2018-01-01'),
                                   np.datetime64('2018-02-01'), np.datetime64('2018-02-01'),
                                   np.datetime64('2018-02-01'), np.datetime64('2018-03-01')],
                   'Txn_date': [np.datetime64('2018-01-30'), np.datetime64('2018-07-01'),
                                np.datetime64('2018-05-01'), np.datetime64('2018-08-01'),
                                np.datetime64('2018-06-01'), np.datetime64('2018-12-30')]})

# Count the number of days between each transaction date and the cohort date
days_diff = df['Txn_date'] - df['CohortGroup']

# Convert into integer
df['Diff_days'] = days_diff.dt.days

# If the transaction occured during the first 120 days (30 days of cohort month + 90 days)..
# Then those customers are considered as 'active'
# Capture the ID of those active customers
df['Active_cif'] = df.loc[df['Diff_days'] <= 120]['txn_new_cif']

# Find the date of each of the following three months after the end of the cohort month
df['First_month_after'] = [x+ relativedelta(months=2) for x in df['CohortGroup']]
df['Second_month_after'] = [x+ relativedelta(months=3) for x in df['CohortGroup']]
df['Third_month_after'] = [x+ relativedelta(months=4) for x in df['CohortGroup']]

# Calculate the cut-off date to determine whether the txn occured with the 1st/2nd/3rd month after
days_diff_1 = df['First_month_after'] - df['CohortGroup']
days_diff_2 = df['Second_month_after'] - df['CohortGroup']
days_diff_3 = df['Third_month_after'] - df['CohortGroup']

df['First_month_days'] = days_diff_1.dt.days
df['Second_month_days'] = days_diff_2.dt.days
df['Third_month_days'] = days_diff_3.dt.days

# Based on these cut-offs, determine if the activity took place within the 1st/2nd/3rd month after
df['Active_month'] = np.where(df['Diff_days'] <= df['First_month_days'], df['First_month_after'],
                              np.where(df['Diff_days'] <= df['Second_month_days'], df['Second_month_after'],
                                      np.where(df['Diff_days'] <= df['Third_month_days'], df['Third_month_after'], None)))

df['Active_month'] = pd.to_datetime(df['Active_month'])

df.head()

(2) Aggregate data by cohort

# Count the total number of customers in each cohort
cohort_cust_ct = pd.DataFrame(df.groupby(['CohortGroup', 'Active_month'])['txn_new_cif'].nunique()).reset_index()

# Count the total number of customers who were active in each cohort
cohort_act_ct = pd.DataFrame(df.groupby(['CohortGroup', 'Active_month'])['Active_cif'].nunique()).reset_index()

# Combine both results
df_grp = cohort_cust_ct.merge(cohort_act_ct, left_on=['CohortGroup', 'Active_month'], right_on=['CohortGroup', 'Active_month'],
                             how='outer')

# Calculate churn rate for each cohort
df_grp['Churn_rate'] = 1 - (df_grp['Active_cif'] / df_grp['txn_new_cif'])

# View results
df_grp.head()


Measuring document cluster cohesion I have a set of clusters which each cluster contains a list of short documents. I want to compute how coherent and cohesive each cluster is and filter out the incoherent and in-cohesive ones.

I am aware of intra-cluster distance and within-cluster dispersion, which are part of computing clustering evaluation metrics Silhouette Coefficient and Calinski-Harabaz Index, respectively. My question is, is there any other metrics or ways to compute such intra-cluster coherence or cohesion? Is there any standard ways for this that I am not aware of? Thanks.

The Silhouette Coefficient is a good option. The Calinski-Harabaz Index has been shown to be unstable for evaluating noisy datsets, so I would recommend the Silhouette Coefficient over Calinski-Harabaz Index.

The Clustering Validation index based on Nearest Neighbors (CVNN) index has been demonstrated to work well for many different datasets containing high noise, arbitrarily shaped clusters, and skewed distributions.

A few other validation measures you can try using are the SD validity index, the S_Dbw validitiy index, and the Xie-Beni index.

The following is a good manuscript for interal clustering validation metrics manuscript.


Training an LSTM to track sine waves I'm experimenting (read: playing around) with LSTMs on Keras.
I want to train an LSTM network so it would "track" sine waves, that is, given sine waves with different wave length, phases and lengths, it would output the rest of wave.
In a sense, a many-to-many problem.

I decided that the network would need to "track" 5 sine waves simultaneously.
I would generate random sine waves and use the 50 last data points as the required output of the NN.

My code is a follows:

from keras.models import Sequential
from keras.layers import LSTM, Dense, TimeDistributed,Lambda, Dropout
import numpy as np

model = Sequential()

model.add(LSTM(32, return_sequences=True, input_shape=(None, 5)))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(5, return_sequences=True))
model.add(Lambda(lambda x: x[:, -50:, :])) # Grab only the last 50 values

print(model.summary())

model.compile(loss='mean_squared_error',
              optimizer='adam')

def make_line(length):
    shift= np.random.random()
    wavelength = 5+10*np.random.random()
    a=np.arange(length)
    answer=np.sin(a/wavelength+shift)
    return answer

def make_data(seq_num,seq_len,dim):
    data=np.array([]).reshape(0,seq_len,dim)
    for i in range (seq_num):
        mini_data=np.array([]).reshape(0,seq_len)
        for j in range (dim):
            line = make_line(seq_len)
            line=line.reshape(1,seq_len)
            mini_data=np.append(mini_data,line,axis=0)
        mini_data=np.swapaxes(mini_data,1,0)
        mini_data=mini_data.reshape(1,seq_len,dim)
        data=np.append(data,mini_data,axis=0)
    return (data)

def train_generator():
    while True:
        sequence_length = np.random.randint(50, 150)+50
        data=make_data(1000,sequence_length,5)
        x_train = data[:,:-50,:] # all but last 50
        y_train = (data[:, -50:, :]) # last 50
        yield x_train, y_train

model.fit_generator(train_generator(), steps_per_epoch=30, epochs=100, verbose=1,validation_data=train_generator(),validation_steps=30)
model.save('vi2h_model.h5')

I then try to generate data and use the model to predict the continuation of a sine wave:

def data_generator():
    while True:
        sequence_length = np.random.randint(50, 150)+50
        data=make_data(1000,sequence_length,5)
        x_train = data[:,:-50,:]
        y_train = (data[:, -50:, :])
        return x_train, y_train

x,y= data_generator()

model = load_model('vi2h_model.h5')
pred=model.predict (x)
np.save ('pred.npy',pred)
np.save ('y.npy',y)
np.save ('x.npy',x)

When inspecting the ability of the network to predict the continuation of a sine wave (comparing y[0,:,0] to pred[0,:,0] for example, in order to compare the first sine wave of the first generated datum), see that my network performed very badly:

(Edit: legend: The predicted vs actual 50 last points in a sample sine wave (blue and orange, respectively), placed after the part of the wave used as input for the NN)

What did I do wrong? And how should I my code to correctly track sine waves?
Many thanks!

I have added a dense layer to the network:

model = Sequential()

model.add(LSTM(32, return_sequences=True, input_shape=(None, 5)))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(TimeDistributed(Dense(5)))  # <--- This is new :)
model.add(Activation('linear'))
model.add(Lambda(lambda x: x[:, -50:, :])) # Grab only the last 50 values

This has provided some improvement :

These are two sine waves and the predictions of the last 50 data points of them.
As can be seen, the results have improved, with some of them being okay approximations of the sine waves. Other times... not so much.

Still a lot better than the previous results.

I then tried another approach: Instead of having one (multilayered) LSTM, in which I ignore all but the 50 inputs (in a sense trying to predict from each one of the 50 last values the value located 50 places after it... which might not be an ideal approach) I split the network into 2 LSTMs, with one receiving the input and "injecting" its results via a dense layer into the second LTSM layer which generates the output.

features_num=5

model.add(LSTM(40, return_sequences=True, input_shape=(None, 5)))
model.add(LSTM(40, return_sequences=False))

model.add(Dense(80))
model.add(Activation('tanh'))
model.add(RepeatVector(50)) # 50 output points

model.add(LSTM(40, return_sequences=True))
model.add(LSTM(40, return_sequences=True))

model.add(TimeDistributed(Dense(features_num)))
model.add(Activation('linear'))

Results are, for the most part, better:


Lemmatization Vs Stemming I have been reading about both these techniques to find the root of the word, but how do we prefer one to the other?

Is "Lemmatization" always better than "Stemming"?

I would say that lemmatization is generally the preferred way of reducing related words to a common base.

This Quora question is a good resource on the subject: Is it advisable to choose lemmatization over stemming in NLP? The top answer quotes another good resource that motivates why lemmatization is usually better, Stemming and lemmatization, from Stanford NLP:

Why lemmatization is better

  Stemming usually refers to a crude heuristic process that chops off
  the ends of words in the hope of achieving this goal correctly most of
  the time, and often includes the removal of derivational affixes.

  Lemmatization usually refers to doing things properly with the use of
  a vocabulary and morphological analysis of words, normally aiming to
  remove inflectional endings only and to return the base or dictionary
  form of a word, which is known as the lemma.

But that is just generally, it is not always better. Stemming still has some advantages and it will depend on the use case. Some reasons you would use stemming over lemmatization could be:

Some possible exceptions when stemming can be better

Simplicity
Speed
Memory constraints


Is it possible to measure the object using deep learning Is there a way we can measure the length, width and the depth of an object in the picture using deep learning?

Not without some kind of reference object. I also don’t see a reason to use deep learning for such a task.
Sure - you could create a dataset with all kinds of objects and their typical dimensions. But the results for inference would probably be pretty bad and the effort to create the dataset ridiculously huge.


Sequence labeling with partially known labels I am working on a sequence labeling task where, based on experience, many of the labels of a given input sequence can be reliably extracted with a simple rule-based approach. For example, considering the following input sequence:

[1, 2, 3, 1, 2, 1, 1] (ground truth labels of input sequence)
[1, X, 3, 1, 2, 1, Y] (labels as extracted by the rule-based approach)

In the above example, the rule-based system was able to extract all but two labels from the input. That is, only X and Y still need to be determined by a machine learning approach.

Are there known ways or algorithms to incorporate the labels extracted by a rule-based system into the machine learning process instead of running a sequence labeling algorithm from scratch? Intuitively, the additional information from the rule-based system should make it "easier" for the machine learning algorithm to fill the remaining slots.

You can train a regular sequence labeling model (typically CRF) where one of the features is the rule-based predicted label: its value is the actual label when known or a special unknown value otherwise. Given that the model can take into account dependencies between labels (as specified in the parameters) and that the rule-based feature always gives the label except if unknown, the model should be able learn:

in which cases it should "trust" the rule-based feature (when value is not unknown)
to predict the missing labels by exploiting both the features and the known labels.

Note that in some rare cases the model might still predict a wrong label even if the rule-based label is provided. That would happen if the probability of the sequence is maximized in this way, but it's unlikely to happen if the training data is representative of the distribution.


A tool like Matlab for NLP? Matlab is a great tool for some mathematical experiments, Neural Networks, Image Processing ...

I would like to know if there is such a comprehensive and strong tool for data manipulation and NLP tasks? such as tokenization, POS tagging, parsing, training, testing ....

However I am new to NLP and I need a tool which let me experiment, get familiar and progress

I have just finished my Ph.D. and have used some NLP in it. My university didn't offer any NLP courses. So I ended up teaching myself NLP. I used this book.

Which serves as a great introduction to NLP using NLTK (Natural Language Tool Kit). It gives a good introduction into programming with Python. So handy if you've never programmed in Python before too.

I would highly suggest using nltk from nltk.org (sorry can't post more than two links)

The book I used is now out of date as NLTK is now on version 3.0, the book mentioned previously is for NLTK 2.x. But the Authors are working on a new version of the book for NLTK 3.X, you can view the unfinished book here.

I would highly suggest using NLTK and if you're new to natural language processing. I would highly suggest you try and get yourself a copy of the following book:

Foundations of Statistical Natural Language Processing by Manning and Shutze

Even though it doesn't contain any code, it servers a great introduction to natural language processing.


How to implement a YOLO model using TensorFlow? I want to implement a face detection model using TensorFlow. I have a dataset downloaded from Google's OID. I have prepared the annotation file as required in the YOLO. But I don't understand how to convert the annotations to the required output tensors.
Is learning Darknet a simpler option?

I personally prefer to use darknet rather than Tensorflow. However, use these codes to build YOLOV3 in TensorFlow.

Also, take a look at this link


Which is better for a beginner: R or Python? From what I can tell, R and Python are the two most popular languages for data science.

My question is which one would you recommend for someone just starting out in data science? Does any one have any clear advantages on the other? Is any one easier to learn or does any one have more potential?

Thanks a lot!

It's a perennial debate. Python is more readable and quicker to learn, no question. It's also a bit more general-purpose language than R. R, on the other hand, has special statistical packages for just about anything you could even dream of doing. There are some stats it can do that Python doesn't have a library for (though I suspect people are working on that).

What's more important in the early stages of learning data science is the more fundamental theory: mathematics, linear algebra, calculus, and statistics. A firm grasp of those areas is a considerably larger share of the learning path than a particular language.

Having said that, I do happen to prefer Python because of the readability. The fact is, that very often other people will need to come in after you and read what you wrote. For that matter, you sometimes have to read your own code! Readability is more important than the ability to do fantastic things with one-liners. As for the advanced stats, that comes into play most often in the medical fields; there you definitely find a preponderance of R over Python.


Test data for statistical t-test in Python first of all sorry if this is not the proper place to ask but i have been trying to create some dummy variables in order to run a students t-test as well as a welch t-test and then run a monte-carlo simulation.Problem is, I am only given the sample size and standard deviation of the 2 populations. How can I go about creating some sort of representation for this data in order for me to run these tests? I wish to run these tests in either python or R. Thanks in advance.

EDIT : both populations come from a normal distribution

In Python, to generate random numbers from a certain distribution you would pick the corresponding distribution from np.random (documentation) and pass the corresponding parameters. Thus to draw from a normal distribution you would do

import numpy as np

# for reproducible results, seed the number generator
np.random.seed(42)

n = 100
mu_1, std_1 = 0, 1
mu_2, std_2 = 0.2, 1.5

dataset1 = np.random.normal(loc=mu_1, scale=std_1, size=n)
dataset2 = np.random.normal(loc=mu_2, scale=std_2, size=n)

And the output

print('dataset 1:')
print(f'mean: {dataset1.mean():.2f}')
print(f'std: {dataset1.std():.2f}')
print(f'shape: {dataset1.shape}')
print('--------------')
print('dataset 2:')
print(f'mean: {dataset2.mean():.2f}')
print(f'std: {dataset2.std():.2f}')
print(f'shape: {dataset2.shape}')

--------------
dataset 1:
mean: -0.10
std: 0.90
shape: (100,)
--------------
dataset 2:
mean: 0.23
std: 1.42
shape: (100,)

PS: You don't have to use np.random.seed that's just to make the random generator consistent with the output every time the code is run.

EDIT: Also, if you want to use a t-test on python you can use scipy.stats, thus if you want to calculate the T-test for the means of two independent samples use scipy.stats.ttest_ind, or if you want to calculate the t-test on two related samples use scipy.stats.ttest_rel


h in LSTM increasing in size? So I was reading about the LSTM architecture and I was having trouble understanding a certain aspect of it. This article mentions the step in question near the bottom of the page. Here is the image given:
The issue I have is this: If ot is the result of an operation on the concatenation of ht-1 and xt, then ot will be larger than ht-1. Then, ht is calculated by an operation on the now larger ot. ht is then 'passed' on to the next timestep. So wouldn't the size of h continually increase over time?
I'm definitely misunderstanding some part of this, so thanks in advance for any help. :)

o(t) is not the result of concatenation of h(t-1) and x(t), but a simple matrix multiplication.
See wikipedia for further details:

https://en.wikipedia.org/wiki/Long_short-term_memory


Training a sound localization neural network I am trying to train a neural network, to estimate the location (in degrees from 0 to 180) a sound is coming from.

I am using tensorflow keras in python to train the model.

The input data are two binaural cues, specifically the ILD (Interaural Level Difference) and the ITD (Interaural Time Difference), each vector, consisting of the two above described features, is of dimensions [1,71276]. I have a total of 2639 measurments, 10% of which is used as validation data, and another 10% as test data.

The output should be an angle in the range [0,180].

I have normalized the data in the range [-1, 1] and the best loss I've been able to achieve is MSE = 16.

The model that achieved the highest MSE is the following:

model = tf.keras.Sequential(([
    tf.keras.layers.Input(shape=(71276,), name='input'),

    tf.keras.layers.Dense(units=900,activation='relu', name='dense_1'),
    tf.keras.layers.Dense(units=360,activation='relu', name='dense_2'),
    tf.keras.layers.Dense(units=180,activation='relu', name='dense_3'),

    tf.keras.layers.Dense(units=1,activation='linear', name='output')
]))

model.compile(loss='mse',
              optimizer=tf.keras.optimizers.Adam(lr=0.001),
              metrics=['mae'])

EPOCHS = 500
BATCH_SIZE = 32

callbacks = [
    tf.keras.callbacks.EarlyStopping(monitor='val_loss', mode='min', min_delta=0.5, patience=100, verbose=1),
    tf.keras.callbacks.ModelCheckpoint('best_model.h5', monitor='val_loss', mode='min', save_best_only=True),
    tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=50, verbose=1, mode='min', min_delta=2, cooldown=0, min_lr=0.000001)
]

history = model.fit(X_train, y_train, validation_data=(X_val,y_val), shuffle=True,
                    batch_size=BATCH_SIZE,
                    epochs=EPOCHS, verbose=1,
                    callbacks=callbacks)

Since this is the first neural network I've trained using my own data, I'm wondering whether there is anything obvious I've missed that could reduce the loss function, and if not, any suggestion is welcome!

I should note that I'm using google colaboratory and I've already tried adding another hidden layer but I got a ran out of memory error. I've also tried increasing/reducing the number of neurons in each layer but I haven't gotten better results and I tried using a CNN architecture as well, with little success as it didn't even converge after 300 epochs.

Thanks in advance!

Change your activation layers, use sigmoid or Tanh for your final layer.

I would try CNN again but with different strides, filter sizes, and number of filters. The thing about CNN is that because you have fewer features per layer you will be able to have more layers.

Here is an example of a Convolutional layer used for audio:

import keras
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPooling2D
from keras.optimizers import SGD
from keras.layers.advanced_activations import LeakyReLU
model = Sequential()
model.add(Conv2D(16, (3,3), padding='same', input_shape=(513, 25, 1)))
model.add(LeakyReLU())
model.add(Conv2D(16, (3,3), padding='same'))
model.add(LeakyReLU())
model.add(MaxPooling2D(pool_size=(3,3)))
model.add(Dropout(0.25))
model.add(Conv2D(16, (3,3), padding='same'))
model.add(LeakyReLU())
model.add(Conv2D(16, (3,3), padding='same'))
model.add(LeakyReLU())
model.add(MaxPooling2D(pool_size=(3,3)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(64))
model.add(LeakyReLU())
model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))

If you have time, try training an LSTM layer.


Pytorch: Starting with a high loss value, but the loss converged at the end. I dont know if the model could start with a loss > 100. Help! I have been trying to attempt plant disease detection using transfer learning methods. I chose ResNet50 first. I also performed a baseline model which is a CNN model. In resnet50, I used cross entropy loss and trained the model for 30 epochs. I did batch normalization too. Initially epoch 1 loss was 112.5250 and the training loss was 87.512. But, for the last epoch it was 2.1660 and Validation loss was 1.8905 with Validation accuracy as 0.995. The overall accuracy of the model was 98.8% and the model doesn't seem to overfit too.
Before training the model, I performed hyper parameter optimization where I optimized Learning rate and momentum using Bayesian optimization. I optimized weight decay using Cross validation. I performed batch normalization. Before that, I trained the model without any optimization. I just assumed the learning rate to be 0.001, same loss function and trained for 30 epochs. In this case, the model started with a loss value of 114.2 and validation loss was 130.46. It converged to 9.9038 with accuracy of 97%.
So, my question is, it is possible for the model to start with such a high loss value despite giving a good accuracy at the end? Does the magnitude of the loss have nothing to do with the correctness of the model nor the accuracy? If the model started with a huge loss, but gave a good accuracy, is my model bad?
First 5 epochs
EPOCH: 1

Epoch time taken: 28.61398434638977 seconds. Epoch 1 loss: 118.5250 Validation loss: 87.5214 Validation accuracy: 0.753

EPOCH: 2

Epoch time taken: 28.737553358078003 seconds. Epoch 2 loss: 68.8141 Validation loss: 50.3452 Validation accuracy: 0.858

EPOCH: 3

Epoch time taken: 28.45273518562317 seconds. Epoch 3 loss: 41.5181 Validation loss: 31.1520 Validation accuracy: 0.928

EPOCH: 4

Epoch time taken: 28.434630155563354 seconds. Epoch 4 loss: 26.8844 Validation loss: 21.1203 Validation accuracy: 0.948

EPOCH: 5

Epoch time taken: 28.762181282043457 seconds. Epoch 5 loss: 19.3646 Validation loss: 15.4843 Validation accuracy: 0.955

Last 5 epochs
EPOCH: 26

Epoch time taken: 28.35645818710327 seconds. Epoch 26 loss: 2.8168 Validation loss: 2.2771 Validation accuracy: 0.990

EPOCH: 27

Epoch time taken: 28.51978588104248 seconds. Epoch 27 loss: 3.3986 Validation loss: 2.2274 Validation accuracy: 0.992

EPOCH: 28

Epoch time taken: 28.51390767097473 seconds. Epoch 28 loss: 2.2707 Validation loss: 1.9976 Validation accuracy: 0.992

EPOCH: 29

Epoch time taken: 28.456573247909546 seconds. Epoch 29 loss: 2.5344 Validation loss: 1.9272 Validation accuracy: 0.990

EPOCH: 30

Epoch time taken: 28.486974239349365 seconds. Epoch 30 loss: 2.1660 Validation loss: 1.8905 Validation accuracy: 0.995
```

The value of the loss in the first epochs is irrelevant. The network weights are initialized at random, so it is perfectly expected that the behavior of the network differs from the desired one.
However, this does not mean that you should only look at the last value of the training and validation loss. Instead, you should plot the whole curves, to understand the training dynamics. From the curves, you can see whether your network overfitted or underfitted, if there's still room for improvement if you train the model for longer, etc.
There are many questions in this site asking about how to interpret training/validation curves (e.g. this and this). You may take a look at them to understand how to interpret yours. From the values you posted, it seems there's still room for improvement if you train for more epochs.


Attention network without hidden state? I was wondering how useful the encoder's hidden state is for an attention network. When I looked into the structure of an attention model, this is what I found a model generally looks like:

x: Input.
h: Encoder's hidden state which feeds forward to the next encoder's hidden state.
s: Decoder's hidden state which has a weighted sum of all the encoder's hidden states as input and feeds forward to the next decoder's hidden state.
y: Output.

With a process like translation, why is it important for encoder's hidden states to feed-forward or exist in the first place? We already know what the next x is going to be. Thereby, the order of the input isn't necessarily important for the order of the output, neither is what has been memorized from the previous input as the attention model looks at all inputs simultaneously. Couldn't you just use attention directly on the embedding of x?

Though translations are not on a word-word basis, nonetheless there is a strong merit in retaining the sequence of the words at the encoder side. There is a huge penalty to pay for this as this causes serialization, but in spite of it LSTMs and GRU's became so popular so one can imagine that sequence order matters. After the encoder is done processing in sequence, the final state that is generated is sort of a sentence embedding and contains the essence of the sentence. This is a good starting point for the decoder to pick and use. Unlike what you have assumed, the model does not look 'only' at the context generated by the attention layer to make predictions. It also uses the prev LSTM state along with the context (and the last translated word to make the next prediction). If you trace back the prev LSTM state right to the beginning of the decoder, you can see it has its origins in the final LSTM state of the encoder.
Having said that, your question still is very pertinent. The concept of Attention is so powerful that with self attention and multi-head attention, it is now possible to do away with the RNN at the encoder end altogether and just use the representation generated purely by the process of 'Attention'. But even here the authors of the landmark paper - Attention is all you need - add in a small hack to retain the order of the sequence of the words of the input sentence. This seems to improve the model in better prediction.


Call routing using AI I'm trying to find out how AI can help with efficient customer service, in fact call routing to the right agent. My usecase is given context of a query from a customer and agents' expertise, how can we do the matching?

Generally, how is this problem solved? What sub-topic within AI is suitable for this problems? Classification, recommender systems, ...? Any pointers to open-source projects would be very helpful.

I would recommend to have a look at intent classification. Once intent of the conversation is known, it could be routed to specific departments.

Basically in chat bot development, we try to do intent classification and entity extraction.

Intent classification is simple text classification and you would need training data.


Detect malicious GIFs I was reading this article talking about a form of targeted internet bullying which involves sending flashing images via Twitter to people with epilepsy.

I was wondering whether there is a way to detect these malicious GIFs? I first though of it as a classification problem but in reality the content can be anything - what we would need to detect is the flashing lights which have a particular frequency.

Any idea on how to approach this? Which technique can be used?

GIFs are just a sequence of images

Given that you just need to find right quantification/pooling of these images to display a GIF. When you displayed a single GIF it will be a collection of different images, where flashing images will have sudden changes in pixel intensity, exactly that is what you want to catch in set of images=GIFs that are malicious.  You just need binary classification from there on.

have a look at here


Logistic Regression Manual Update For the logistic regression below, how can I manually update the coefficients a and b manually?
EDIT
y = 1.0 / (1.0 + exp(-ax - b))

after observing the following data examples (x, y), the parameters a and b need to be updated using gradient descent with a learning rate = 1.
1.
input: x = 1, y = 1, a = 1, b = 1
expected output: a = 0.881, b = 0.881
2.
input: x = 2.2, y = 0, a = 5.1, b = 5.7
expected output: a = 7.3, b = 6.7
UPDATE
def update(x, y, a, b):
    t = 1 + np.exp(-a * x - b)
    da = (y - 1 / t) * x * (t - 1) * t**-2
    db = (y - 1 / t) * (t - 1) * t**-2
    new_a = a - da
    new_b = b - db
    return new_a, new_b

This got me
update(1, 1, 1, 1)
(0.9874844578263232, 0.9874844578263232)

Did I miss something here?
FINAL
def manual_update(x, y, a, b):
  p = 1 / (1 + np.exp(-a*x-b))
  pdpda = x*np.exp(-a*x-b) / (1+np.exp(-a*x-b))**2
  pdpdb = np.exp(-a*x-b) / (1+np.exp(-a*x-b))**2
  pdLda = -y*pdpda/p+(1-y)*pdpda/(1-p)
  pdLdb = -y*pdpdb/p+(1-y)*pdpdb/(1-p)
  return round(a + pdLda, 3), round(b + pdLdb, 3)

this worked perfectly.

The update rules are ;
$a := a - \alpha.\frac{\partial L}{\partial a}$
$b := b - \alpha.\frac{\partial L}{\partial b}$
Here L which is the Loss [Squared Error in this case ] is given by;
$L  = \frac{1}{2}.(\hat{y} - \frac{1.0}{(1.0 + exp^{-ax + b})})^{2}$
So ;
$\frac{\partial L}{\partial a} = \frac{1}{2}.2(\hat{y} - \frac{1.0}{(1.0 + exp^{-ax + b})}).\frac{-1.0}{(1.0 + exp^{-ax + b})^{2}}.(-x).e^{-ax+b}  $
$\frac{\partial L}{\partial b} = \frac{1}{2}.2(\hat{y} - \frac{1.0}{(1.0 + exp^{-ax + b})}).\frac{-1.0}{(1.0 + exp^{-ax + b})^{2}}.e^{-ax+b} $
So now guess an intial value for a and b.
You can plug in the an x and y pair for and update the weights simultaneously. The thing to be careful is that when updating for a and b in an iteration use the previous a and b values. That is if you have updated your a prior to b dont use the updated value of a to update b.


Convolutional Neural Networks in R I don't see a package for doing Convolutional Neural Networks in R. Has anyone implemented this kind of algorithm in R?

I guess there is no package for cnn but you can write your own convolutional layer. mxnet or h2o will be useful for it.

check this out:

http://dmlc.ml/rstats/2015/11/03/training-deep-net-with-R.html


How to visualize change in a distribution with a few outliers that account for a very large percent of the total? I'm working on an edtech product where some of our traffic lands on webpages about textbooks.

Textbooks belong to subjects like Algebra, Calculus and Spanish. In each of our subjects, we have "whales" - individual books that account for a large percent (~20%) of total subject traffic.

Year over year, these whales grow or shrink (sometimes books are replaced in a school, or schools drop textbooks altogether). This change in whale traffic contributes to a big change in overall subject traffic.

I'm trying to figure out how to visualize this change, given a dataset that looks something like the table attached below (it shows only 2 months traffic per book, but I have access to all the months).

I've tried overlapping histograms (and boxplots), where each histogram is a month. But this visualization doesn't indicate how huge my whales (outliers) are, and how much influence they have.

Any help on chart types or how to otherwise tell this story would be much appreciated.

To visualize change in the size of multiple entities that are contributing to a total through time, e.g. total (t) = book_1 (t) + book_2 (t) + ..., we can use Stacked Area Plot. This plot can be used for normalized and un-normalized (absolute) values.

Preprocessing

For large number of entities, to avoid cognitive load, we can keep only
those entities that become significant (whale) at some point in the plot, and group all the others under an "ordinary" entity. This way, cognitive load is minimized and only those entities that matter at some point are distinguished. For example, distinguishing books that have more than 10% of traffic at some point in the plotted time span.
If total fluctuation is very high, logarithm of values can be plugged into the plot.


How to calculate temporary/periodic similarities of an increasing series in real time? Considering there are two series over time and new data is added in the series over a gap of n second . The series might have periodic similarity/dis-similarity within themselves. How to calculate correlation among the series value in real time?

well, This is an interesting question! Let's formulate it again to be precise in answer:

  I have two real-time streams of values. There are temporary similarities (transient similarities) between two. How to capture it?

As you are talking about similarity, I did not limit the scope to correlation. One may define another measure of similarity. I would say you should define your similarity function first (you may choose correlation. I do as well for my example) and a period of inspection (it's simply a window of length $m$ ending at current timestamp).

Using these two and starting from $m+1$th sample, you can calculate your similarity measure (here correlation) as a time-series. This time-series starts after $m$th sample of input series so keep it in mind to keep similarity measure and original timestamps synchronized. The interval between two consecutive points in output (similarity series) is defined based on the resolution you need. after $m+1$th sample you can shift your window on step at time and calculate the similarity accordingly. Then the intervals in output will be as same as input.

Of course it keeps you more away from real-time as the frequency and number of calculations is maximum. You can define a margine for yourself (let's say $d$ sample) and calculate similarity every $d$ time. Then you lose some info (discarding the samples in between) but you get closer to real-time (you will have $d*n$ seconds to perform computation).

Please let me know if it need clarification. Welcome to the community BTW :)


In sklearn Pipeline, why are all parameters fit_transform(), but the last one can be just fit()? I'm reading through Hands-On Machine Learning with Scikit-Learn & TensorFlow. We're going over sklearn Transformation Pipelines for preparing data for the ML algorithms.

Here's the code (housing_num is a dataframe of the numerical attributes of the main dataframe housing):

num_pipeline = Pipeline([
    ('imputer', Imputer(strategy='median')),
    ('attribs_adder', CombinedAttributesAdder()),
    ('std_scaler', StandardScaler())
])

housing_num_tr = num_pipeline.fit_transform(housing_num)

The text explains, "The Pipeline constructor takes a list of name/estimator pairs defining a sequence of steps. All but the last estimator must be transformers (i.e., they must have a fit_transform() method)."

1) Why do we fit_transform() some, but only fit() without transforming others? I'm guessing this has to do with the difference between estimators and transformers, which I'm confused on in this context.

2) So when using Pipeline, you can only include one estimator that won't be transformed, and it has to be the last input? ("all but the last estimator must be transformers")

Thanks,

Greg

To center the data (make it have zero mean and unit standard error), you subtract the mean and then divide the result by the standard deviation.

$$z' = \frac{x-\mu}{\sigma}$$

or

$$z = ln(x)$$

You do that on the training set of data. But then you have to apply the same transformation to your testing set (e.g. in cross-validation), or to newly obtained examples before forecast. But you have to use the same two parameters $\mu$ and $\sigma$ (values) that you used for centering the training set.

Hence, every sklearn's transform's fit() just calculates the parameters (e.g. $\mu$ and $\sigma$ in case of StandardScaler) and saves them as an internal objects state. Afterwards, you can call its transform() method to apply the transformation to a particular set of examples.

fit_transform() joins these two steps and is used for the initial fitting of parameters on the training set $x$, but it also returns a transformed $x'$. Internally, it just calls first fit() and then transform() on the same data.


Post training classifier configuration I have a behaviours vector representing some identity. I need to binary classify [malicious or benign] each instance [ideally with a normalised severity score].

For that I can use a variety of linear classifiers/kernelized SVM/Random Forest etc...

The issue is that once the classifier has been trained I'd like to allow the user the ability to configure which behaviours are more (or less) critical.

For example, one behaviour might be encryption done by some process and a user fearing ransomware might want to make this behaviour more significant.

Given linear classifier (which I'd like to avoid) simply multiplying the given W with the given configuration would do the trick. What can be done in kernelized SVM/Random Forest/DNN etc for equivalent result?

From the description of your problem, it sounds like applying a weight to the input prior to sending the data to the classifier might do the trick and would be easy to implement.  The weight could be positive or negative.


Morphological Analyzer for Hindi and English in Python I am unable to find morphological analyzers for English or Hindi. I am looking for a tool that can split the word into morphemeses e.g.
Independently = In + dependent + ly

One option is the ploygot package which can perform morphological analysis in English and Hindi.
from polyglot.text import Word

word = Word("Independently", language="en")
print(word, w.morphemes)


From regression neural network to generative one I made a neural network with word embedding, RNN and some dense layers to predict the score of computer game reviews based on the title of the game. I used Keras in Python. I was wondering if it would be somehow possible to transform my model into a generative one that would produce titles with high (or low) predicted score. It could be possible to do brute forcing and run all combinations through my network but maybe there is a smarter solution.

There's more than one type of generative network. However, I am not aware of a generic approach that can take a trained RNN-based network and essentially run it backwards to sample an input that is expected to produce a given output. So I am suggesting a couple of generative approaches that I have seen working, but that will require that you construct and train a new network.

You can bring in some knowledge about the typical size of network that learns the regression model, but you cannot AFAIK directly re-use the regression model and somehow reverse it.

A caveat: Although I have played briefly with both types of generative network, I have never constructed one conditioned on desired goal like the one you want to work with.

With a purely RNN-based approach, you might do OK by making your network predict the next letter/word in the title - a classifier - whilst taking the title so far (or X characters/words of it) and the rating (normalised) as inputs.

Then, once trained, you can sample from the RNN randomly to generate new strings. This is the technique used by e.g. Karpathy in his now famous blog "The Unreasonable Effectiveness of Recurrent Neural Networks". There are many examples of such sequence sampling generators available to study.

You could also take the output of such a model and see if it matches your regression model from earlier. But I don't think there is much you can do if it does not, except perhaps filter out generated titles if they don't meet expectations - e.g. generate many titles with intended rating of 10, and only display one when your initial model also agrees with a close to 10 rating.

The most relevant Keras example for this I could find is lstm_text_generation.py

A RNN-based generative adversarial network (GAN) might also be able to achieve what you want. However, please note that GANs are notoriously fiddly to train.

A GAN is actually 2 networks. You create a discriminator and a generator and train them in parallel. The generator takes a small completely random vector (e.g. 10 numbers sampled from N(0,1)), plus the rating you want to achieve. Then it generates a text sequence output. The discriminator would take a text input, and a rating, and outputs 1 if it is real, or 0 if it is fake. You present either real training data or output form the generator to the discriminator, and use these to train it. You train the generator based on whether it fools the discriminator.

The tricky part is to maintain balance between the two components - if either becomes too good relative to the other, training will stall.

However, if you can get it to work, you will have a true generative model, which samples from a population space (the noise vector that you input) plus is conditioned on the rating.

The most relevant Keras example for this I could find is mnist_acgan.py which generates an image, not text sequence, but hopefully should give a start.


Is there a good alternative to XGBoost for learn to rank? My problem with XGBoost is that when I load the train dataset into the XGBoost DMatrix, there is a memory spike that is unavoidable, and I can't get my dataset loaded into RAM without crashing first.
I am trying to predict rankings over time, similar to a search engine query problem. XGBoost has produced good results using the pairwise objective but I have reached my limits due to the memory spike issue.

You asked for alternatives - also consider: LightGBM, CatBoost or RankLib.
How big is your dataset?  If you are hitting memory problems just creating the DMatrix its possible the other libraries will have similar problems.
What are you loading your dataset from?  Perhaps creating the DMatrix from a numpy array is trying to fit two copies of your data in memory at the same time?  Perhaps you could divide your dataset into smaller chunks and append them to the DMatrix to avoid having them all in memory at once?


Would it be okay to stop training my neural network? When the validation error of my Neural Network that I am trying to train is slowly decreasing but not by much, is it okay to stop train the network at that point, or do I need to increase the training time until the minimum validation error is reached?

For instance, in the last 5 epochs my validations errors are shown below:

| end of epoch   1 | time: 3782.50s | valid loss 6.7914 | valid ppl 890.1194
| end of epoch   2 | time: 3802.14s | valid loss 6.6084 | valid ppl 741.2616
| end of epoch   3 | time: 3791.33s | valid loss 6.5249 | valid ppl 681.8797
| end of epoch   4 | time: 3792.55s | valid loss 6.4513 | valid ppl 633.5318
| end of epoch   5 | time: 3804.15s | valid loss 6.3884 | valid ppl 594.8927

so like between the 4th epoch and the 5th epoch, the loss decreased by ~0.975% (= (6.4513-6.3884)/6.4513 * 100)? would it be okay to stop training the network at this point?

Thank you,

You should keep training.

In many scenarios ~1% decrease in validation loss a bid deal in itself. However, looking at the trend it looks like you validation loss is set to decrease by more than ~1%, if you let it train form, say, 20 more epochs. The decreases will get smaller and smaller, but they will accumulate.

Generally, you should continue training if your validation loss is decreasing.


Can AI (NLP) convert user questions (text) into database SQL queries? I have been reading about NLP but got confused and not able to figure out - if it is feasible for NLP to convert questions in natural language to transform into SQL queries (so that it can execute on concerned database and fetch the output).

Ex. If the user raises a question to the AI engine (application) -

  How many new customers have registered on my website this month?

The AI should parse this using NLP techniques and convert it into SQL and execute on database table say "User_management" -

select count(1) from User_management
where joining_month = 'Oct' and joining_year = 2019

Please share your thoughts and advice.

Update 1) Got this SO link and this NLTK chapter as starting point

Yes it is possible. You can train end to end an RNN over the training data. It means input the user query as an input and set the output for the sql query.
Definitely the most challenging part is preparing the training set as RNN needs a lot of data to be learned.

Also, for the training set, it might you can use a GAN to generate more queries for the users' query dataset that you have. Notice that, as you want the query generator for the specific schema of the database, you need your own training data and it could not be generalized over all kinda schemas.


Pandas/Python - comparing two columns for matches not in the same row I have this data:

I wanted to compare A and B for matches not by row but rather search A0 if it is in column B and so on. Moreover, I wanted to ignore the .AX in column A because it would not find any matches in column B anyway.
I used this, but it matches values row by row and it returns False or True. I would like to print the matches in a new Column C:
    df3['match'] = df3.A == df3.B

Thank you.

To clarify, this question is about comparing two columns to check if the 3-letter combinations match.
So, I would approach this in the following manner:
# Extract the 3-letter combinations from column a
df3["a normalised"] = df3["a"].str[:3]

# Then check if what is in `a normalised` is in column b

b_matches = list(df3[df3[“b”].isin(list(df3[“a normalised”]))][“b”].unique())
df3.loc[:, "match"] = False

b_match_idx = df3[df3["a normalised"].isin(b_matches)].index

df3.at[np.array(b_match_idx),"match"] = True

EDIT: The parentheses have now been resolved. Also the .loc warning can now be mitigated.


Identify credit card shape using machine learning I have to approach this task: identify credit card from an image. I am attaching example image below:

I have to identify and localize the credit card from this image. The real challenge is that the card can be placed on any background and the color of the card can change based on the company it belongs to.
In order to solve this task I tried using the tensor flow object detection api. The downside of this api is that it fails to recognize cards which are not in its training data set.
My problem here is I am not concerned with the color of card or what information the card has. I am only concerned about finding the outline of card in an image and isolating the outline of card from rest of the image.
Is there a way using ML/CNN to do this. I tried OpenCV approaches to detect contours but even this approach fails when there is lot of text or other noise in the cards background.

I am only concerned about finding the outline of card in an image and isolating the outline of card from rest of the image.

This can be efficiently solved by semantic segmentation (aka dense prediction) - problem in which every pixel must be labeled with class.
In your case, you will have 2 classes: credit card and background. And you will need to have annotated dataset in such way: for every image, every pixel should have a class label. If you will be annotating it manually, I guess that credit cards (because of their simple shape) can be annotated easily.
Here is a good solution in Keras for semantic segmentation models. It offers a lot of different architectures and backbones and it will be straightforward to apply it to your problem.
There are also implementations in other frameworks on the web.


Finding Criminal Name in news? We have news URLS, which we want to classify into crimes or non-crimes and further identify criminals by using NERs.
For creating a model that identifies criminals, we tried SPacy which gave all the names like lawyers name , president ,criminal etc..
can Anyone help on how to get only Criminal name, Not all these irrelevant names.
I am just a beginner trying things, any help is appreciated
Thanks in Advance

By downloading the Free Wolfram Engine for Developers you may use Wolfram Language.  If you are using Python then use the Wolfram Client Library for Python to access the engine in Python.

You may use FindTextualAnswer.

FindTextualAnswer["Person Jane Doe found guilty in case, Judge John Dude to give final verdict on it tomorrow.",
 "Who was found guilty?",
 TargetDevice -> "GPU"]

"Jane Doe"

FindTextualAnswer["Person Jane Doe found guilty in case, Judge John Dude to give final verdict on it tomorrow.",
 "Who was the judge?",
 TargetDevice -> "GPU"]

"John Dude"

Hope this helps.


Metrics for presenting RNN/LSTM result I am working on a two different architecture based on LSTM model to predict the users next action based on the previous actions. I am wondering, what is the best way to present the result? Is it okay to present only the prediction accuracy? Or Should I use other metrics? I found a paper using top_K_accuracy where on a different paper I found AUC of ROC. Overall, I want to know what is the state of art of presenting prediction accuracy based on LSTM model.

There really is nothing special about LSTMs when it comes to classification and metrics. So your question should be what metrics are good for multi-class classification. Both accuracy and auc works. Another metric that is good here is Matthew's Correlation Coefficient (MCC) which is similar to F1 score for a binary problem.


How to pad real-valued sequences I have several sequences of univariate real-valued time-series data. The sequences are of different lengths and right now I cannot batch them and feed them to a network.
What is the correct procedure to pad these sequences? Is it even possible in this case since I can't use any number as a special symbol?

UPDATE 1

I'm working with arbitrary univariate time-series data (not related to one specific domain, unbounded range). To give example of one such a series consider standardized stock dataset (only first 10 elements shown):

d = array([-0.37807043, 0.14321786, -0.37807043, 0.13478392, 0.18733381,
   1.19576774, 0.25675156, 0.26064414, 0.30930144, 0.38650436])

How you pad it (and even whether you do so) would depend on what you expect of the data. This imposes boundary conditions on the data which will induce artifacts in any transform you make. How bad this effect depends on how well geared your data is to accepting a particular padding method.

Padding methods include zero padding or a periodic bound.

Padding doesn't have to be done in the time domain. Eg interpolating in the frequency domain and back transforming allows you to extrapolate.

If your analytics has a finite history (eg FIR filters) then you can isolate time regions where padding is unnecessary and draw comparisons therefrom.


What to use pretrained models (Glove) or train my own model? I have been using pre-trained models such as google news or Glove 6B model but many words in my text data does not have their vectors representation in those pre trained model. So I was thinking of maybe train my own model with the data I have.

Is there any disadvantages in training our own model for two class classification model? or Should I keep using pre-trained model. What is the difference between training our own model or using pre-trained model.

# This is how I am thinking to train the model
from gensim.models import Word2Vec
w2v_model=Word2Vec(list_of_sentance_train,min_count=5,size=50, workers=4)

Usually, I would test my models in both scenarios: with pre-trained word embeddings (GloVe, Word2Vec, etc.) and with my own text, giving that I have a reasonably large data set.

The difference will be that the pre-trained models are more general in context. They are trained with really large data sets, thus capturing information and relationships (semantic, syntax) in a more generic way. The famous example of Mikolov's paper where linear combinations of two words such as city and Paris gives a result near to France in a high dimensional vector. In your own data set, you might capture some different relationships, as an example I could think in a situation where Paris is a character name (I am giving an extreme example for sake of clarity, hope it workds) and then it will be closer to other characters in the embeddings, etc.

To conclude, I have read some papers where word embeddings from specific events (in this case was disaster management) did not work better than the general ones for topic classification. I have tried myself with the same task in a large Portuguese data set and the results were better with my own embeddings (Actually, SVM beats all Deep Learning algorithms in this case) and worked better. Maybe someone with more expertise can give a more detailed answer, but my advice is: try with both.


How to perform topic modelling on query search results How can I model topics in the results returned by a search engine with higher weightage to documents ranked higher in the result set?

The use case that I am looking at involves extracting the most significant topics returned in the search results.

Eg. If the user searches for a query q1 which returns documents D1...Dn with scores S1...Sn (in descending order) then I propose the notion that the theme of such a set of documents is represented better by documents scored higher in the result set.

Is it possible to incorporate this information in to topic modelling algorithms like LDA?

If you assume that the documents returned for a query already share a common "topic", then what's the point of using topic modeling? You already have a kind of "topical subset" of the documents.

However if the goal is to clean up the results so that documents which are "topically similar" don't appear together at the top, i.e. to favor "topical diversity" in the top resulting documents, then this could be done in the following way:

train a topic model for all the documents offline. You obtain a vector of posterior probabilities p(T|D) for every topic T and document D.
for every query, modify the scoring method so that there is for instance a minimal topic distance between documents ranked at the top (this part might be tricky to get right but it's feasible).

Incidentally, this approach is also more realistic because it would be quite inefficient to train a full topic model for every particular query.


Debugging a simple 1-D CNN for solving a simple classification problem I have a rather simple classification problem that I am trying to solve. Each instance in my problem is a list of 1024 bytes (each byte is represented by a digit between 0 and 255). There are 2 classes (say, Class 'A' and 'B'). In class 'A', all instances have a common feature. There exists a particular 2-byte pattern say: "200 180" that is present in high frequency among all instances in Class 'A'. However, in class 'B', this is not the case. In class 'B', the presence of the byte string '200 180' is more of an artefact of randomness rather than anything else.
I would like my CNN to differentiate between A and B classes without explicitly feeding it the byte pattern. I am using the following code to try to do that:
model = models.Sequential()
model.add(layers.Conv1D(filters=1, kernel_size=2, activation='relu', input_shape=(1024, 1)))
model.add(layers.MaxPooling1D(pool_size=64))
model.add(layers.Flatten())
model.add(layers.Dense(1, activation='sigmoid'))
model.compile(optimizer='adam',
          loss='binary_crossentropy',
          metrics=['accuracy'])

When I use some simple heuristics to gauge the "distinguishability" between the two classes, I can see that it's possible to easily get more than 90% accuracy rates by simply looking for the frequency of the byte pattern. However, my above CNN fails miserably at doing this. For example, here is a 5-epoch training attempt result for my CNN:
Epoch 1/5
1563/1563 [==============================] - 65s 41ms/step - loss: 0.5297 - accuracy: 0.8468
Epoch 2/5
1563/1563 [==============================] - 61s 39ms/step - loss: 0.7534 - accuracy: 0.5000
Epoch 3/5
1563/1563 [==============================] - 61s 39ms/step - loss: 0.7100 - accuracy: 0.5029
Epoch 4/5
1563/1563 [==============================] - 61s 39ms/step - loss: 0.6977 - accuracy: 0.5216
Epoch 5/5
1563/1563 [==============================] - 62s 40ms/step - loss: 0.6934 - accuracy: 0.5303

As you can see, the result is more like a random guess than anything else. I wanted to know if you have any advice on how you think this task can be handled better.

you can add dropout to the model It will change your model accuracy.
from keras.layers import Dropout

model = models.Sequential()
model.add(layers.Conv1D(filters=1, kernel_size=2, activation='relu', input_shape=(1024, 1)))
model.add(layers.MaxPooling1D(pool_size=64))
model.add(layers.Flatten())
model.add(Dropout(0.5))
model.add(layers.Dense(1, activation='sigmoid'))
model.compile(optimizer='adam',
      loss='binary_crossentropy',
      metrics=['accuracy'])


How to use Scikit-learn's affinity propagation clustering with my own datasets? I am trying to cluster my datasets using affinity propagation. I followed this and this links to grasp the basics of affinity propagation clustering. The sample code available at sklearn is as follows:
from sklearn.cluster import AffinityPropagation
from sklearn import metrics
from sklearn.datasets.samples_generator import make_blobs
# Generate sample data
centers = [[1, 1], [-1, -1], [1, -1]]
X, labels_true = make_blobs(n_samples=300, centers=centers, cluster_std=0.5,
                            random_state=0)
# Compute Affinity Propagation
af = AffinityPropagation(preference=-50).fit(X)
cluster_centers_indices = af.cluster_centers_indices_
labels = af.labels_

While running it, it works as stated on the website. However, I could not understand it completely. I want to modify this code to use with my datasets. My datasets consist of values from different sensors on a 2-D surface. I want to cluster the values with similar sensors readings at various points on the 2-D surface. How can I do it?
Thank you.

Clustering algorithms usually assume that you have positions of objects, and want to find dense groups of observations.

If I understand you correctly, you have a 2d grid of sensor readings, and you want to segment them into regions. That is a slightly different problem. If you'd just put your sensor readings into a clustering, then the clusters will not be spatially coherent: clustering assumes there is no particular order to the points.

So you'll need to look into segmentation.

A naive way would be to use (sensor.x, sensor.y, sensor.value) tuples. Including the sensor positions will cause the results to be somewhat spatially coherent. But that makes it very sensitive to scaling, and there is no "correct" way of scaling this. There is a trade-off between spatial coherence and measurement coherence.


How Many Features Are Used in Random Forest When max_features is not an integer? In the RandomForestClassifier function in sklearn.ensemble, one of the parameters is max_features, which controls the size of the random subset of features that are considered at each node splitting. I would think this needs to be an integer, since of course the size of a subset needs to be an integer.
But one of the options here is "log2", which sets max_features to log_2(n_features).
I was wondering how that works, considering log_2(n_features) will not be an integer unless n_features is a power of 2.
Thanks.

The argument for the number of features gets passed down to the BaseDecisionTree class. Looking at the code, you can see that the value that is used is calculated by first taking log with base two and then converting it to an integer (i.e. rounding it).
if self.max_features == "auto":
    if is_classification:
        max_features = max(1, int(np.sqrt(self.n_features_)))
    else:
        max_features = self.n_features_
elif self.max_features == "sqrt":
    max_features = max(1, int(np.sqrt(self.n_features_)))
elif self.max_features == "log2":
    max_features = max(1, int(np.log2(self.n_features_)))
else:
    raise ValueError("Invalid value for max_features. "
                     "Allowed string values are 'auto', "
                     "'sqrt' or 'log2'.")


Generating a set of different scenarios based on some initial observations I have a in my hands 3 different time series which model 3 different scenarios (base, downside, upside). Every of this time-series depends on a set of 11 different attributes, which take values for different time intervals. Most of the different features of the input are highly correlated. There is also a (cdf) probability function which defines how probably every scenario is (every quintile), for every point in time.
In my case, I want to create more input data based on the current observations and create different time-series/ simulations. For example, let’s take the case of the base scenario. The first idea I had was to calculate the Covariance matrix and the mean value of the different data points through time and then just draw samples from a multivariate normal distribution. But this obviously is not correct because even if the correlation between the different attributes is preserved, some of the properties of the time series are not. If I just draw random points and assign them at the different time intervals, some of these attributes can become wiggle. And for example, if one attribute is kind of similar with let’s say ‘GDP’ then it does not make sense to fluctuate for short-periods of time.

At this point I have think couple of ways how to deal with this problem but I have not come up with a complete analytical solution. Someone else had attempted to take a look before. He just basically defined normal distributions for each point in time using the values of the attributes for the 3 scenarios and the cdf. For example if the value of an attribute is [1.25, 1.5, 2] for every scenario respectively and for some values of the cdf, these points can be used in order to create a distribution and then sample different points. But, with this way all the new times series have the form of the base one (same fluctuation)  and they are concentrated close to this series.

At this point I mostly interested in a simple approach on this problem that will give some reasonable results. It would be great if you could give some advice or any kind of reference to look for.

You could do block bootstrapping. For example GDP was 100, 150, 300, 100, 130. In percentage it is +50%, +100%, -67%, +30%. You choose blocks of two. So you have two blocks (+50%, +100%) and (-67%, +30%). So you take random blocks for your scenario. Possible variants:

+50%, +100%, -67%, +30%
+50%, +100%, +50%, +100%
-67%, +30%, -67%, +30%
-67%, +30%, +50%, +100%

The blocks allow you to hold autocorrelation in the data and if one block includes all variables then all correlations will also remain the same.

An explanation on how to do block bootstrapping on Cross Validated StackExchange site:
https://stats.stackexchange.com/questions/25706/how-do-you-do-bootstrapping-with-time-series-data


Multicollinearity(Variance Inflation Factor). Variables to remove before doing a model I am doing an exercise of a Machine Learning System module in python that takes a dataset of cars (cylinders, year, consumption....) and asks for a model, being the variable to predict the consumption of gasoline. As it has three categorical variables, I have generated the dummies.

In the exercise I need to eliminate the variables with multicollinearity, so I used the method showed on my course notes:

from sklearn.linear_model import LinearRegression

def calculateVIF(data):
    features = list(data.columns)
    num_features = len(features)

    model = LinearRegression()

    result = pd.DataFrame(index = ['VIF'], columns = features)
    result = result.fillna(0)

    for ite in range(num_features):
        x_features = features[:]
        y_featue = features[ite]
        x_features.remove(y_featue)

        x = data[x_features]
        y = data[y_featue]

        model.fit(data[x_features], data[y_featue])

        result[y_featue] = 1/(1 - model.score(data[x_features], data[y_featue]))

    return result

Then if I launch the method it calculates a coefficient for each variable:

In my course notes it is said:

$VIF>5$ is a high value.
$VIF>10$ is a very high value

What should I do? I need to remove the variables that have a $VIF>10$ before executing the model?

The problem I see, for my categorical variable cylinders, is only cylinders_5 has a VIF under 10 so should I remove the others and leave cyclinders_5?

Here is a code I have written to handle Multicollinearity in a dataset. This code snippet is able to handle the following listed items:

Multicollinearity using Variable Inflation Factor (VIF), set to a default threshold of 5.0
You just need to pass the dataframe, containing just those columns on which you want to test multicollinearity.
This function will drop those columns which contains just 1 value. For a bit more further details on this point, please have a look my answer on How to run a multicollinearity test on a pandas dataframe?.
The calculation of VIF is parallelized over multiple cores.

    from joblib import Parallel, delayed
    from statsmodels.stats.outliers_influence import variance_inflation_factor

    def removeMultiColl(data, vif_threshold = 5.0):
        for i in data.columns:
            if data[i].nunique() == 1:
                print(f"Dropping {i} due to just 1 unique value")
                data.drop(columns = i, inplace = True)
        drop = True
        col_list = list(data.columns)
        while drop == True:
            drop = False
            vif_list = Parallel(n_jobs = -1, verbose = 5)(delayed(variance_inflation_factor)(data[col_list].values, i) for i in range(data[col_list].shape[1]))
            max_index = vif_list.index(max(vif_list))
            if vif_list[max_index] > vif_threshold:
                print(f"Dropping column : {col_list[max_index]} at index - {max_index}")
                del col_list[max_index]
                drop = True
        print("Remaining columns :\n", list(data[col_list].columns))
        return data[col_list]

Good luck !