How would I be able to improve my CNN model (Keras)? Recently I read a research paper on age detection using facial images. So right now because of that I was trying to  see how far I could get by applying a CNN to a dataset of facial images (with their respective ages) in order to predict their ages which would be in bins (ex. 0-10, 11-20, 21-30...).

For training and testing

training.shape (50000, 28, 28)
testing.shape (2938, 28, 28)

I tried to keep the images small as they would be able to run faster as well as using grayscale.
And for the actual layers themselves I tried to keep it simple, for now,

model = Sequential()
model.add(Conv2D(64, kernel_size=3, activation='relu', input_shape=(28,28,1)))
model.add(Conv2D(32, kernel_size=3, activation='relu'))
model.add(Flatten())
model.add(Dense(10, activation='softmax'))

Layer (type)                 Output Shape              Param #
=================================================================
conv2d_1 (Conv2D)            (None, 26, 26, 64)        640
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 24, 24, 32)        18464
_________________________________________________________________
flatten_1 (Flatten)          (None, 18432)             0
_________________________________________________________________
dense_1 (Dense)              (None, 10)                184330
=================================================================
Total params: 203,434
Trainable params: 203,434
Non-trainable params: 0
_________________________________________________________________

Compiled with the following

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

So far the best accuracy after running for 100 epochs has been 37.16. Which isn't great but recently I've gotten access to one of my schools gpu's so I wanted to fix anything that I'm doing wrong and improve my model. Is there anything you could recommend when it comes to improving the model, theres probably a lot this is more my first time trying to do this.

The first thing you should notice is that you've almost ruined your input signal. Take a look at a $28\times28$ image of a face? what can you see? is there any difference between a teenager and a middle-aged person? The point is that the network should be trained using data that does not have high Bayes error which means you as an expert can distinguish between inputs and label them correctly. Increase the size of your inputs. By doing so, if you use the current regime, you may have lots of trainable parameters between dense layers and convolutional layers. Consequently, try to employ more convolutional layers and some pooling layers among them. Also, try to add more dense layers with more neurons in each.


Supervised Learning on Sequential data I am doing credit risk modelling on costumer transaction data a part of which looks like this :

str(x)
'data.frame':   412516 obs. of  26 variables:
 $ Tenure           : num  1.26 1.25 1.26 1.31 1.32 ...
 $ Product          : Factor w/ 24 levels "BACKHOE LOADER",..: 4 4 4 9 9 9 9 9 9 9 ...
 $ Net.Exposure     : num  333339 528049 327335 350000 460000 ...
 $ OD.On.31.01.2017 : num  0 90386 0 0 1099692 ...
 $ LM.Bucket        : Ord.factor w/ 11 levels "0"<"1 TO  30"<..: 1 1 1 1 11 11 11 11 11 11 ...
 $ Bucket           : Ord.factor w/ 11 levels "0"<"1 TO  30"<..: 1 3 1 1 11 11 11 11 11 11 ...
 $ Billing          : num  65380 0 8800 6339 8331 ...
 $ Fin.IRR          : num  13.5 14.6 14.6 18.1 23.3 ...
 $ NPA.Flag         : Factor w/ 2 levels "No","Yes": 1 1 1 1 2 2 2 2 2 2 ...
 $ Inst.Due         : num  0 0.85 0 0 3 3 3 3 3 3 ...
 $ FR.On.31.01.2017 : num  65380 0 38940 35043 499860 ...
 $ POS.On.31.01.2017: num  56453 0 32920 33368 293943 ...
 $ Del.String       : int  2 1 1 1 53720 53720 53720 53720 53720 53720 ...
 $ Territory        : Factor w/ 43 levels "AGRA","AHMEDABAD",..: 41 41 41 41 41 41 41 41 41 41 ...

The variables like OD(Overdue) and LM.Bucket( How many months he has been due on his loan payment till last month) change every month .I have 2 tasks :Predict Bucket and NPA Flag(Non performing asset)

I built a model for this based only on the Jan data(x). But my question is since these variables change every month, should i treat this as a sequential data and build a deep learning model(HMM/NN) on it? If i should what should I do with the static variables like Product type etc.?

I asked my boss regarding the same and he said it shouldn't be done because external economy factor change with time. Is that a reason for concern?

This basically asks for a recurrent network, like the LSTM. But if you only have 2 properties that are dynamic, I don't think you will have as much luck because they might be affected from external factors as your boss said. However, this will happen regardless of the model you're using.

You should not throw away static properties, unless they are the same for every test case. E.g. farmer/politician/baker category should always be included, but you call this 'static' but it's not completely static as it is not the same for every sample in your test cases.


How can I know if my conv1D model is overfitted or underfitted from loss curve? I am working on classification of time series multivariate data. By doing PCA, I converted multivariate to uni-variate and fed it into a conv1d in keras.

However, I am getting a very high accuracy and low loss both in validation and in training. How can I justify this?

I have tried cross validation, but the results are not much different.
I am using adam optimizer (learning rate:0.0001). With 0.001, my model fails to converge.

I have made sure that I am not mixing the training and validation datasets.
I have shuffle both datasets independent of each other. I trained on 3728 samples and validated on 610 samples.

Can we expect such a high accuracy with binary classification?

From the curves you are showing yes. Over-fitting would mean that your validation accuracy would be lower than your training accuracy, which is not the case here. Since you say that your training and validation sets are completely independent (i.e. no training samples are present in the validation set) you can consider the results reliable.

However accuracy might not be the best indicator of the model's performance. Make sure your dataset is balanced (i.e. the number of samples in both classes are equal to one another). If not try another metric that better represents the performance of your model.


X_train, y_train from ImageDataGenerator (Keras) Can I have X_train, y_train, X_test, y_test from data_generator?
Here is my code:

data_generator = ImageDataGenerator(
    rescale = 1. / 255,
    shear_range = 0.2,
    zoom_range = 0.2,
    horizontal_flip = True,
    vertical_flip = True,
    rotation_range = 180,
    width_shift_range = 0.2,
    height_shift_range = 0.2,
    validation_split = 0.2)

train_generator = data_generator.flow_from_directory(
    train_data_dir,
    target_size =(img_width, img_height),
    batch_size = batch_size,
    shuffle = True,
    class_mode = 'categorical',
    seed = 42,
    subset='training')

validation_generator = data_generator.flow_from_directory(
    train_data_dir,
    target_size =(img_width, img_height),
    batch_size = batch_size,
    shuffle = True,
    class_mode = 'categorical',
    seed = 42,
    subset='validation')

in python 2:

X_train, y_train = train_generator.next()
 X_test, y_test = validation_generator.next()

in python 3:

X_train, y_train = next(train_generator)
X_test, y_test = next(validation_generator)


Off-policy n-step learning with DQN I'm reviewing the Rainbow paper and I'm not sure I understand how they can use DQN with multi-step learning, without doing any correction to account for off-policiness.

So. I understand how you can use 1-step update off-policy: the reward for a single transition doesn’t depend on the current policy, so you can reuse this experience in the future.

I also understand the point of n-step updates: it’s a trade-off between having high biais with 1-step update (as you get only limited information from a single transition) and having high variance with n-step update (as in that case both the policy and the environment can be stochastic, so you end up adding n random variables together).

What I do not get is how you can use n-step return off-policy, which is what the Rainbow DQN seems to do. With n-step returns you are considering trajectories, and you can’t assume that these trajectories would have been taken if the agent was using the current policy.

If I understood correctly, in the case of policy gradient this is dealt with using importance sampling, which will reduce the impact of policies which are further away from the current one.

But I don’t see the equivalent of this for multi-step DQN?

In the Rainbow approach, theoretical correctness of the off-policy return values is completely ignored, and it just uses:

$$G_{t:t+n} = \gamma^n\text{max}_{a'}[Q(S_{t+n},a')] + \sum_{k=0}^{n-1} \gamma^{k}R_{t+k+1}$$

It still works and improves results over using single-step returns.

They rely on a few things for this to work:

$n$ is not large, compared to amount of variability in the environment that occurs over those steps. So the return is usually correct even if there was an exploratory action taken. Many game events (and also real-world physics) have some fuzziness - e.g. as long as in Pong you hit the ball with a part of the paddle, it will be a reasonable result, and only some (literal) edge cases will cause a major difference in rewards.
Exploration is low ($\epsilon$ set to $0.01$ or $0.001$)
Rewards are sparse, so the n-step return very often only depends on state progression. As long at $S_{t+n}$ is as likely to be reached starting from $S_t$ under both the exploratory and target policies, then the simple returns are approximately correct.
The policy changes slowly, maybe taking a million frames to converge to optimal, which is far larger than the replay memory.

From a theoretical basis in Q learning, it would be better to use "up to n-step" returns that truncate at the time step when the action taken diverges from the  current learned policy. However, that is more complex, as you have to assess the learned policy at each of the n steps in order to decide when to truncate - I suspect given the researchers know how $Q(\lambda)$ works, that they may have tried this and just found the extra correctness was not worth the extra complexity and CPU cost. In other environments it may be worth the effort to implement this.


What version of spark in latest Cloudera QuickStart VirtualBox? I am thinking to install the quick start virtual box of Cloudera, search on Cloudera download page, but unable to find the spark version in the latest Cloudera QuickStart virtual box, can any have knowledge of this, please let me know.

QuickStart VM latest 5.7 has spark version 1.6.0 by default.


How does an individual feature affect predictions in neural network classification problem? In the literature, I've come across statements like People with higher income and with long working hours are more likely to be diagnosed with chronic diseases such as stroke. The above-mentioned study (Page:8), explores the association between Behavioral Habits and Chronic Diseases using ANN.

As I'm new to ML,

I am unable to figure out how to make such conclusions with feature
study in neural networks or other machine learning techniques.
Is there a way to quantify the likelihood in ANN similar to logistic
regression wherein regression coefficients give the change in the
log odds of the outcome for a one unit increase in the predictor
variable?

Currently using Azure ML studio

Is there a way to quantify the likelihood in ANN similar to logistic regression wherein regression coefficients give the change in the log odds of the outcome for a one unit increase in the predictor variable?

Good question.

Yes, there is a way. The approach that can help you is called partial dependence plot (PDP), see the links below for further details and examples.

The approach is model agnostic, i.e. works well for any predictive model, powerful yet simple.

The main steps for one-dimensional partial dependence plot are as follows

Fit your model as usual
Select the predictor of interest and a set of values to be investigated (e.g. income as in the article you refer to and values of say 50k, 70k, 80k, ..., 120k)
For all observations in your dataset replace the values of your predictor with the first value from the set above (50k).
Calculate the model output for the modified dataset from the previous step and calculate the average over all observations.
Repeat steps 3-4 for the remaining values (70k, 80k, ...) and plot the values of your predictor along X axis and the corresponding averaged model predictions along Y axis.

With one-dimensional PDP described above you can easily see the marginal impact of a predictor being analysed on the model output. Furthermore, one can use similar technique to perform multi-dimensional analysis, e.g. to investigate the impact of interactions.

partial dependence plots- scikit-learn documentation

partial dependence plot - tutorial by Dans Becker on Kaggle


How to choose an appropriate Machine Learning algorithm? I am currently new in Data Science field especially in ML domain. There are various approaches to a problem ie. Supervised, Unsupervised learning. I want to know what is the approach and how to determine using exactly which algorithms in real life test cases.

Even if a particular algorithm is selected, how do we know other algorithm might have better accuracy and precision than this one

Try as many models as you have time for and pick the best one, otherwise use your intuition to narrow the candidates. If you have no intuition, start with a simple model and move to more complex models as performance requirements dictate. Often there are reasons beyond pure performance metrics to choose a model in the real world, such as simplicity, interpretability, and speed.

You will not, however, have to pick between supervised and unsupervised learning, for these are drastically different things. If I ask you to group all similar items, how are you going to apply supervised learning?

You might like this map about choosing the right estimator.


What does the alpha and beta hyperparameters contribute to in Latent Dirichlet allocation? LDA has two hyperparameters, tuning them changes the induced topics.

What does the alpha and beta hyperparameters contribute to LDA?

How does the topic change if one or the other hyperparameters increase or decrease?

Why are they hyperparamters and not just parameters?

The Dirichlet distribution is a multivariate distribution. We can denote the parameters of the Dirichlet as a vector of size K of the form ~$\frac{1}{B(a)} \cdot \prod\limits_{i} x_i^{a_{i-1}}$, where $a$ is the vector of size $K$ of the parameters, and $\sum x_i = 1$.

Now the LDA uses some constructs like:

a document can have multiple topics (because of this multiplicity, we need the Dirichlet distribution); and there is a Dirichlet distribution which models this relation
words can also belong to multiple topics, when you consider them outside of a document; so here we need another Dirichlet to model this

The previous two are distributions which you do not really see from data, this is why is called latent, or hidden.

Now, in Bayesian inference you use the Bayes rule to infer the posterior probability. For simplicity, let's say you have data $x$ and you have a model for this data governed by some parameters $\theta$. In order to infer values for this parameters, in full Bayesian inference you will infer the posterior probability of these parameters using Bayes' rule with $$p(\theta|x) = \frac{p(x|\theta)p(\theta|\alpha)}{p(x|\alpha)} \iff \text{posterior probability} = \frac{\text{likelihood}\times \text{prior probability}}{\text{marginal likelihood}}$$ Note that here comes an $\alpha$. This is your initial belief about this distribution, and is the parameter of the prior distribution. Usually this is chosen in such a way that will have a conjugate prior (so the distribution of the posterior is the same with the distribution of the prior) and often to encode some knowledge if you have one or to have maximum entropy if you know nothing.

The parameters of the prior are called hyperparameters. So, in LDA, both topic distributions, over documents and over words have also correspondent priors, which are denoted usually with alpha and beta, and because are the parameters of the prior distributions are called hyperparameters.

Now about choosing priors. If you plot some Dirichlet distributions you will note that if the individual parameters $\alpha_k$ have the same value, the pdf is symmetric in the simplex defined by the $x$ values, which is the minimum or maximum for pdf is at the center.

If all the $\alpha_k$ have values lower than unit the maximum is found at corners

or can if all values $\alpha_k$ are the same and greater than 1 the maximum will be found in center like

It is easy to see that if values for $\alpha_k$ are not equal the symmetry is broken and the maximum will be found near bigger values.

Additional, please note that values for priors parameters produce smooth pdfs of the distribution as the values of the parameters are near 1. So if you have great confidence that something is clearly distributed in a way you know, with a high degree of confidence, than values far from 1 in absolute value are to be used, if you do not have such kind of knowledge than values near 1 would be encode this lack of knowledge. It is easy to see why 1 plays such a role in Dirichlet distribution from the formula of the distribution itself.

Another way to understand this is to see that prior encode prior-knowledge. In the same time you might think that prior encode some prior seen data. This data was not saw by the algorithm itself, it was saw by you, you learned something, and you can model prior according to what you know (learned). So in the prior parameters (hyperparameters) you encode also how big this data set you apriori saw, because the sum of $\alpha_k$ can be that also as the size of this more or less imaginary data set. So the bigger the prior data set, the bigger is the confidence, the bigger the values of $\alpha_k$ you can choose, the sharper the surface near maximum value, which means also less doubts.

Hope it helped.


Can I consider this pattern of data as a linear and use parametric multiple linear regression? In the data, there are 355 observations including one continuous dependent variable (Y: ranges from 15-55) and 12 independent variables (continuous, categorical, and ordinal). The X1 (2 levels) and X6 (3 levels) are considered as categorical variables. Here are some questions that I have:

Can I assume that all the coefficients (except X1 and X6 which are categorical) are linear with respect to Y?
Can I consider X5 as continuous variable; however, it is ordinal and ranges from (1-7)?
Can I get the X7 (year) as continuous variable; however, it’s ordinal and rages from 2002-2006 (In fact, year of data per se does not improve the response; it is the other factors occurring in the same time period which result in improvements and we don’t know those factors), does this approach seem logical?.
In general if I use different transformations on independent variables such as log, squared, square root, and inverse, do I need to standardize the data also?

Here is the scatter plot:

Any feedback and insights would be highly appreciated. Thank you

I don't think "can" is the right question to ask; it's not going to give you a syntax error. The right question is "what could go wrong?". Any modeling technique will have assumptions that may be broken, and knowing how those assumptions impact the results will help you know what to look for (and how much to care when those assumptions are broken).

The best test of whether or not linearity is appropriate is whether the residuals are white or structured. For example, it looks like X9 might have a nonlinear relationship with Y. But that might be an artifact of the interaction between X9 and other variables, especially categorical variables. Fit your full model, then plot the residuals against X9 and see what it looks like.
Treating it as continuous won't cause serious problems, but you might want to think about what this implies. Is the relationship between 1 and 2 in the same direction and half the strength as the relationship between 2 and 4? If not, you might want to transform this to a scale where you do think the differences are linear.
Same as 2, except it's even more reasonable to see time as linear.
Standardization is not necessary for most linear regression techniques, as they contain their own standardization. The primary exception is techniques that use regularization, where the scale of the parameters is relevant.

It's also worth pointing out that multivariate linear relationships, while they can capture general trends well, are very poor at capturing logical trends. For example, looking at X3 and X4, it could very well be that there are rules like Y>X3 and Y>X4 in place, which is hinted at but not captured by linear regression.


k fold cross validation i want to use KFold from mode_selection instead of cross_validation ut it didn't work for the pobject Kfold

 from sklearn.model_selection import KFold
    import xgboost as xgb
    # Some useful parameters which will come in handy later on
    ntrain = X_train.shape[0]
    ntest = X_test.shape[0]
    SEED = 123 # for reproducibility
    NFOLDS = 10 # set folds for out-of-fold prediction
    kf = KFold(shuffle=False, n_splits= NFOLDS, random_state=SEED)
    def get_oof(clf, x_train, y_train, x_test):
        oof_train = np.zeros((ntrain,))
        oof_test = np.zeros((ntest,))
        oof_test_skf = np.empty((NFOLDS, ntest))

        for i, (train_index, test_index) in enumerate(kf):
            x_tr = x_train[train_index]
            y_tr = y_train[train_index]
            x_te = x_train[test_index]

            clf.train(x_tr, y_tr)

            oof_train[test_index] = clf.predict(x_te)
            oof_test_skf[i, :] = clf.predict(x_test)

        oof_test[:] = oof_test_skf.mean(axis=0)
        return oof_train.reshape(-1, 1), oof_test.reshape(-1, 1)
    xgb_oof_train, xgb_oof_test = get_oof(xgb,x_train, y_train, x_test)

i got this error TypeError: 'KFold' object is not iterable

Try to use the split method as enumerate argument, instead of kf (e.g.:for i, (train_index, test_index) in enumerate(kf.split(X)):

Hope it helps!


Comparing multi-class vs. binary classifiers in predicting a single class I've pretty much read the majority of similar questions, but I haven't yet found the answer to my question.
Let's say we have n samples of four different labels/classes namely A, B, C, and D. We train two classifiers:

First classifier: we train a multi-class classifier to classify a sample in data to one of four classes. Let's say the accuracy of the model is %x.
Second classifier: now let's say all we care about is that if a
sample is A or not A. And we train a binary classifier for classifying samples to either A or non-A. Let's say the accuracy of this models
is %y.

My question is, can we compare x and y as a way to measure the performance of classifiers on classifying A? In other words, does a high performance in a multi-class classifier mean that the classifier is capable of recognizing the single classes with high performance as well?
The real-world example of this is that I've read papers that trained multi-class classifiers on a dataset that contains four different types of text. They achieved pretty high performance. But all I care about is for a model to be able to correctly classify one specific type of text. I trained a binary classifier that achieves a lower accuracy. Does this show that my model is working poorly on that type of text and the multi-class classifier is doing better? Or shouldn't I compare these two?

In general we can't compare the performance of a multiclass classifier with the performance of a binary classifier since the former expresses how good the classifier is at classifying any instance of any class. So if there are $n_A$ samples labelled A, there's only a proportion of $n_A/n$ of the global accuracy of the multiclass classifier which is about A. In particular a multiclass classifier usually tends to favor the largest classes, so if class A happens to be a small proportion of the data then the global performance will not reflect how good it is at classifying A: for example it might have 90% accuracy simply because class B is 90% of the data, this doesn't prove anything about class A. By contrast the performance of the binary classifier is by definition solely about class A.
However if one has access to the detailed evaluation of the multiclass classifier, typically the confusion matrix, then it becomes possible to calculate the performance of the classifier for a single class, say class A. Actually by merging all the B,C,D rows together and all the B,C,D columns together in the confusion matrix one obtains exactly a binary classification confusion matrix, and from that one can calculate a performance which can be compared against another binary classifier. But in this setting the multiclass classifier is at a disadvantage for the reason mentioned above: it also has to deal with the other classes and this can cause it to "sacrifice" a class, whereas the binary classifier doesn't have this issue.


Neural network activation function im fairly new to neural networks.
I want to ask what exactly does the activation output, is it the probability the combined summation of inputs and weights lead to a match for the next neuron?
Thanks

The goal of activation functions is to introduce non-linearity in the neural network, so it can learn beyond linear relations between input and outputs.
As stated in your image, each neuron output is a linear combination of inputs + a bias, and then all this is sent into a non-linear function (activation function). This allow the network to learn complex relations between input ans outputs.
Most used activation function is ReLU, which is not a probability as it can return a value greater than 1.


different results with MEKA vs Scikit-learn! I have a multilabel data which I've trained using different classifiers with MEKA (multilabel version of WEKA) and the evaluation results (e.g. accuracy) that MEKA gives me are different from those I get from the same classifiers of scikit-learn.

I am using Binary Relevance method (which is both in MEKA and scikit-multilearn), because of my data being multilabel. And for instance, I am using SVM which is SMO in MEKA (and WEKA) and linearSVC in sklearn.

Now the accuracy I get from 10-fold cross-validation with BR>SMO in MEKA is about 50% where I get something around 20% accuracy from BinaryRelevanvce>LinearSVC. Other evaluation measures (e.g. f1-score, hamming loss etc.) don't match as well.

I tried to give all the parameters I could find in the input of LinearSVC, the same value as those of SMO but there's no luck. My Python code would be like this:

from skmultilearn.problem_transform import BinaryRelevance
from sklearn.svm import LinearSVC

my_classifier = BinaryRelevance(
                classifier=LinearSVC(C=1, random_state=1, tol=0.001, max_iter=10000, dual=False),
                require_dense=[True, True])

I must add that using BR>SMO in MEKA to train and evaluate the model would take almost an hour on my data, but sklearn's linearSVC (with skmultilern's BinaryRelevance) takes less than 10 minutes on the same data (hence the lower accuracy).

And by the way, I'm using accuracy_score method in sklearn.metrics for accuracy.

So my question is what am I missing? How can I get the same accuracy from SVM in MEKA and scikit-learn?

According to this question

  LinearSVC is not linear SVM, do not use it if do not have to.

and furthermore

  LinearSVC [is] one of the mistakes of sklearn developers.

In this picture from the question you can observe how it is inaccurate, even compared to SVC with kernel=linear.

Further differences may be explained with differences how SMO and SVC with linear kernel are implemented.

This post compare the two, and the answer is SMO should be the fastest one, otherwise your dataset maybe is not suitable for it.

I would test SVC with linear kernel and see if things change.


Correlation with missing values. Is least squares an acceptable option? I have been tasked with finding a correlation matrix for a lot of variables. Many of them have missing values. I read here that pairwise deletion may not be the best way of dealing with this situation, but if I use listwise deletion I am afraid I'll end up with too little data.

I am wondering if I could estimate the missing values using least squares, and calculate the correlation using the now complete data set.

Yes you "can" use least squares to do that. However, you would probably need some parametric assumptions to do it, which in turn might require some knowledge about the correlation matrix...

To go back to your original problem, I would actually argue against your reference about pairwise deletion. The weird behavior he identifies mostly happens because the estimator is computed with a dataset that is too small. The empirical estimator of Pearson's correlation is NOT guaranteed to be positive-definite. Also, there is some sort of pattern in its missing data: entries for the $X_3$ are missing only when $X_1 \neq X_2$. Let us instead assume that entries are missing purely randomly.

The estimator you compute using $\hat\Sigma$ := cor(X, use="pairwise.complete.obs"), for a data matrix $X$ still makes a lot of sense. The biggest difference with the non-missing data case is that you now get some entries of $\hat\Sigma$ that are computed on less data; but they still estimate the quantity you wish to estimate.

What is less clear is how to use $\hat\Sigma$ in the desired application: the fact that some entries were computed on less data means that some entries might be less reliable than others. However, notice that even in the classical case, some entries are less reliable than others anyway (this depends on the underlying distribution).

To understand the impact of using all available pairwise data to compute $\hat\Sigma$ in the presence of missing data, consider the two following situations:

Situation 1: You have a $n \times d$ data matrix $X$ and compute the empirical correlation with $\hat\Sigma^{(1)}$ := cor(X).

Situation 2: You have a $[nd(d-1)/2] \times d$ data matrix $X$ in which you have $n$ pairwise observations for all $d(d-1)/2$ pairs of variables and NA everywhere else. in this case you compute the empirical correlation with $\hat\Sigma^{(2)}$ := cor(X, use="pairwise.complete.obs").

In both situations, you have the same number of non NA pairs (although more non NA entries in the second case). However, you have, for $i_1,j_1 \neq i_2,j_2$
\begin{align*}
\mathrm{cor}(\hat\Sigma^{(1)}_{i_1,j_1},\hat\Sigma^{(1)}_{i_2,j_2}) \neq 0
\end{align*}
assuming the your variables are indeed related, and
\begin{align*}
\mathrm{cor}(\hat\Sigma^{(2)}_{i_1,j_1},\hat\Sigma^{(2)}_{i_2,j_2}) = 0.
\end{align*}
Somehow, I tend to think the zero correlation between the estimates might be a good thing more that a bad one...

All this to say, pairwise deletion is not necessarily a bad way to proceed.


Do Random Forest overfit? I have been reading around about Random Forests but I cannot really find a definitive answer about the problem of overfitting. According to the original paper of Breiman, they should not overfit when increasing the number of trees in the forest, but it seems that there is not consensus about this. This is creating me quite some confusion about the issue.

Maybe someone more expert than me can give me a more concrete answer or point me in the right direction to better understand the problem.

Every ML algorithm with high complexity can overfit. However, the OP is asking whether an RF will not overfit when increasing the number of trees in the forest.

In general, ensemble methods reduces the prediction variance to almost nothing, improving the accuracy of the ensemble. If we define the variance of the expected generalization error of an individual randomized model as:

From here, the variance of the expected generalization error of an ensemble corresponds to:

where p(x) is the Pearson’s correlation coefficient between the predictions of two randomized models trained on the same data from two independent seeds. If we increase the number of DT's in the RF, larger M, the variance of the ensemble decreases when ρ(x)<1. Therefore, the variance of an ensemble is strictly smaller than the variance of an individual model.

In a nutshell, increasing the number of individual randomized models in an ensemble will never increase the generalization error.


TypeError: '<' not supported between instances of 'int' and 'str' I have the following code
rf = RandomForestClassifier()
rf.fit(X_train, Y_train)
print("Features sorted by their score:")
print(sorted(zip(map(lambda x: round(x, 2), rf.feature_importances_), X_train), reverse=True))

and I get the following error:
> TypeError Traceback (most recent call last)
>
> ipython-input-109-c48c3ffd74e2> in <module>()
>
> 2 rf.fit(X_train, Y_train)
>
> 3 print ("Features sorted by their score:")
>
> ----> 4 print (sorted(zip(map(lambda x: round(x, 2),
> rf.feature_importances_), X_train), reverse=True))
>
> TypeError: '<' not supported between instances of 'int' and 'str'

I am not sure what I am doing wrong. I only have int and float in my dataframe.

Based on our discussion, omit reverse = True and use list instead of sorted and print it. I guess you will see what you want.


How to handle non-numeric user id's in a recommendation model Hi i'm working on a practice project on amazon movie reviews and i did everything that was asked except building a model that will recommend movies to users which have not been watched nor been rated. but when i'm trying to train and split the model I have a problem on assigning x/features and y/target as user_id's are not numeric e.g 'A3R5OBKS7OM2IR'. (The data contains 207 columns with user_id column which contains 4.9k users and 206 movies(movie 1 to movie 206 with ratings corresponding with the respected user_id)

I'm not sure why you would include user ID as a predictive variable, unless there is specifically some part of the user ID that gives you information. For example, maybe the first three letters tell you that the user is a premium user and then you could feature engineer a category from that.


Tools for project management Data Science I'm in charge of a small data science team (3 data scientists, me included). We do our projects with at least one business person (PM) per project ( we have 5 of these).
We managed everything with meetings and emails, but as the number of projects and people keeps increasing, I find it necessary to have a proper management tool.
I would like to have something were we could, per project, add business needs (requirements). These requirements could translate into coding actions. One could estimate the time needed for each action.
It should be possible to then assign these tasks to DSs, and organise the priority to be given to each task. This way the whole team could see what need to be done in each project, what each DS is going to do after, and when each task is going to be completed.
I'm looking for a free or very cheap solution. I looked at Jira, but I'm not sure if it's a good solution.

Data science project planning is carried out through any of the scrum application such as Jira, Asana or even using Github projects. Gitlab also offers efficient tools for the same. Once you are done with planning, you can use open-source projects such as Git-lfs and DVC to version control your datasets. The notebooks that you prepare could also be version controlled in Github or Gitlab. You can easily find courses and tutorials to dynamically set up your boards and workflows in Github as well as Gitlab.


Classification of Conversations in Text I am trying to pick a technique for classifying conversational text.  I am concerned about treating the problem at a level of fidelity of each individual message because people often say things like, "ok" or short responses that have no inferable meaning.   How does conversation classification typical handle these types of problems?

To elaborate, a conversation might be:

P1: Hi I want to buy a car?
P2: Ok. Great!
P1: What cars do you have?
P2: A large variety!

The topic is cars, but this can not be inferred by anything P2 says, nor should it be.  So would you break a conversation into blocks of time, or is there a technique for partitioning?

There are multiple techniques that help with the problem you sketch, the applicability of which usually depends on the classification technique and the corpus. But I get the idea you would be helped by some practical examples. So let's go over some of them. Feel free to comment if I tread over familiar grounds, or you want me to elaborate on some of them. Or where to start experimenting with them

Stopping A simple technique is applying a stoplist: A list of common words that should be removed. There are pre-packaged lists, but most packages allow you to provide your own.
TF/IDF A technique that transforms your features by weighing words by their term frequency (how often do they occur in the document) divided by the document frequency (how often does the word occur in other documents. This way frequent words are made less relevant to the document
POS Many packages offer you a part-of-sentence-tagger, that will tag the words by their grammatical function (Verb for instance). You can leverage that in the tokenization step to filter out words (usually you'll look for verbs and nouns). Some vectorizers can do this straight out. (This could also be done with NER)
Stemming transforms inflections of words (eg: train/trains) to a stem. This might make some of your words a little more relevant by upping the chance a pair of them collides
Restricting your vectorizer: Most packages sport a vectorizer that you can instruct to either look for a minimum/maximum document count (ignore words that either occur in to many different documents, or to little different documents), or to cap the amount of features (words). Capping the amount of words usually selects for most frequently used words.
Encoding word/token based features to more semantic features: Word2Vec, but also older techniques such as LDA/LSI.
Picking the classification algorithm: Some algorithms are very capable of handeling large feature spaces (Naive Bayes for instance), some algorithms learn to transform the feature space to find better ways to weigh the features.

Packages such as Sklearn, NLTK and Gensim offer most of these techniques.

Let me know if this was helpful


Minimizing error on unseen data The classifier aims to minimize the loss function (($F(x)$ - $\hat{F}(x)$)2), where $F(x)$ is unknown function and $\hat{F}(x)$ is the predicted function. If $F(x)$ is not known for unseen data, how do we compute this loss? Why is the training error used to estimate the error for unseen data?

if we don't know $F(x)$ for unseen data, how does a decision tree minimize this error?

Every supervised ML method relies on the assumption that the test data (any unseen data) follows the same distribution as the training data (note that this is not specific to Decision Trees). In fact both the training data and the test data are assumed to be sampled from the true population data. As a consequence $F(x)$ is assumed to be the same for the training data and the test (unseen) data.
If one uses a trained model on some unseen data which is not distributed like the training data, the results are simply unpredictable and the performance is likely to drop.

Why do we estimate the error for unseen data using the error observed in the training data?

You seem to suggest to use the "unseen data" in the training process. You would indeed get better results on the "unseen data" if you optimized on it, but then you would lose the point of having a portion of data set apart. "Unseed data" is necessary to estimate how good your model will perform on data never seen before. If you don't keep some data set apart you may have a better model but you have no way of estimating how good it will be when put into production.


Why is Amazon P2 instance cost much more than G3? I'm currently looking for the correct number of 2018 budget for the AWS instances in preparation of more deep neural net training, but now a bit confused because of this chart below:

p2.16xlarge costs 3 times more than g3.16xlarge with the same number of processors but slightly less memory. Is this diff just because of the memory difference? or is there any other key difference between p2 and g3 instances? If I'm missing anything here, please share your wisdom!

Thanks,

p2.16xlarge has 16 Tesla K80 GPUs
g3.16xlarge has 4 Tesla M60 GPUs
p3.16xlarge has 8 Tesla V100 GPUs.

I believe the number is a reference to the vCPUs (they all have 64).  Comparing the 1 GPU instance of each type, p3 is the most expensive, then g3, then p2, as expected.


supervised learning and labels In this wiki page, I came across with the following phrase.

  When data is not labeled, a supervised learning is not possible, and
  an unsupervised learning is required

I cannot figure out why supervised learning is not possible?

Appreciate any help to resolve this ambiguity.

The main difference between supervised and unsupervised learning is the following:

In supervised learning you have a set of labelled data, meaning that you have the values of the inputs and the outputs. What you try to achieve with machine learning is to find the true relationship between them, what we usually call the model in math. There are many different algorithms in machine learning that allow you to obtain a model of the data. The objective that you seek, and how you can use machine learning, is to predict the output given a new input, once you know the model.

In unsupervised learning you don't have the data labelled. You can say that you have the inputs but not the outputs. And the objective is to find some kind of pattern in your data. You can find groups or clusters that you think that belong to the same group or output. Here you also have to obtain a model. And again, the objective you seek is to be able to predict the output given a new input.

Finally, going back to your question, if you don't have labels you can not use supervised learning, you have to use unsupervised learning.


Python implementation of cost function in logistic regression: why dot multiplication in one expression but element-wise multiplication in another I have a very basic question which relates to Python, numpy and multiplication of matrices in the setting of logistic regression.
First, let me apologise for not using math notation.
I am confused about the use of matrix dot multiplication versus element wise pultiplication. The cost function is given by:
$J = - {1\over m} \sum_{i=1}^m y^{(i)}log(a^{(i)})+(1 - y^{(i)})log(1-a^{(i)})$
And in python I have written this as
    cost = -1/m * np.sum(Y * np.log(A) + (1-Y) * (np.log(1-A)))

But for example this expression (the first one - the derivative of J with respect to w)
${\partial J \over{\partial w}} = {1 \over{m}} X(A-Y)^T$
${\partial J\over{\partial b}} = {1\over{m}} \sum \limits_{i = 1}^m (a^{(i)}-y^{(i)})$
is
   dw = 1/m * np.dot(X, dz.T)

I don't understand why it is correct to use dot multiplication in the above, but use element wise multiplication in the cost function i.e why not:
   cost = -1/m * np.sum(np.dot(Y,np.log(A)) + np.dot(1-Y, np.log(1-A)))

I fully get that this is not elaborately explained but I am guessing that the question is so simple that anyone with even basic logistic regression experience will understand my problem.

In this case, the two math formulae show you the correct type of multiplication:

$y_i$ and $\text{log}(a_i)$ in the cost function are scalar values. Composing the scalar values into a given sum over each example does not change this, and you never combine one example's values with another in this sum. So each element of $y$ only interacts with its matching element in $a$, which is basically the definition of element-wise.
The terms in the gradient calculation are matrices, and if you see two matrices $A$ and $B$ multiplied using notation like $C = AB$, then you can write this out as a more complex sum: $C_{ik} = \sum_j A_{ij}B_{jk}$. It is this inner sum across multiple terms that np.dot is performing.

In part your confusion stems from the vectorisation that has been applied to equations in the course materials, which are looking forward to more complex scenarios. You could in fact use cost = -1/m * np.sum( np.multiply(np.log(A), Y) + np.multiply(np.log(1-A), (1-Y))) or cost = -1/m * np.sum( np.dot(np.log(A), Y.T) + np.dot(np.log(1-A), (1-Y.T))) whilst Y and A have shape (m,1) and it should give the same result. NB the np.sum is just flattening a single value in that, so you could drop it and instead have [0,0] on the end. However, this does not generalise to other output shapes (m,n_outputs) so the course does not use it.


Cut-Off using Frequent Pattern Mining - Spark Mllib I am using the Association Rules algorithm using this:

http://spark.apache.org/docs/latest/mllib-frequent-pattern-mining.html

I'm have 83945 transactions in my dataset. And I want to filter some products that only appears X times in my dataset. Basically, I want to set my cut-off. My question is: How can I define my cut-off, this is how to define the minimum number of occurences that my products need to have?

Many thanks!

Like @SeanOwen pointed out its called support.

spark.mllib’s FP-growth implementation takes it as a hyper-parameter under minSupport.

It is the minimum support for an itemset to be identified as frequent, e.g : if an item appears 4 out of 5 transactions, it has a support of 4/5=0.8.

Usage:

import org.apache.spark.mllib.fpm.FPGrowth

val transactions: RDD[Array[String]] = ???

val fpg = new FPGrowth().setMinSupport(0.2)

val model = fpg.run(transactions)

I hope this helps.


How to apply Gradient Descent to a summ of function My target is to find a center of a circle that approximate a set of dots

i want to find minimum of a function:
$$\sum_{i=0}^N (\sqrt{(x_i - a)^2 + (y_i - b)^2} - R)^2$$

this function represent an error of a my approximation of a set a dots on a plane with a circle.

I did a bit of a googling and find that Gradient Descent is a decent method for a numerical searching for a minimum of function.

But i have a trouble with understanding how i should correcnt my A,B,R with partial derrivative, when i have function of summ.

picture related for better explanation of my problem:
http://imgur.com/gallery/ecUdXnB

I came up with the simple answer after I just rewrite this idea on paper:

$$\frac{\delta \sum \Big[f(x,y)\Big]}{\delta x}  = \sum \frac{\delta f(x,y)}{\delta x}$$

so it's just partial derivative for each Experimental Dot summed.
Algo works quite well.


Python stemmer for Georgian I am currently working with Georgian texts processing. Does anybody know any stemmers/lemmatizers (or other NLP tools) for Georgian that I could use with Python.
Thanks in advance!

I don't know any Georgian stemmer or lemmatizer. I think, however, that you have another option: to use unsupervised approaches to segment words into morphemes, and use your linguistic knowledge of Georgian to devise some heuristic rules to identify the stem among them.
This kind of approach consists of a model trained to identify morphemes without any labels (i.e. unsupervisedly). The most relevant Python package for this is Morfessor. You can find its theoretical foundations in these publications: Unsupervised discovery of morphemes; Semi-supervised learning of concatenative morphology.
Also, there is a Python package called Polyglot that offers pre-trained Morfessor models, including one for Georgian. Therefore, my recommendation is for you to use Polyglot's Georgian model to segment words into morphemes and then write some rules by hand to pick the stem among them.
You should be able to evaluate the feasibility of this idea by adapting this example from Polyglot's documentation from English to Georgian (by changing the language code en and the list of words):
from polyglot.text import Text, Word

words = ["preprocessing", "processor", "invaluable", "thankful", "crossed"]
for w in words:
  w = Word(w, language="en")
  print("{:<20}{}".format(w, w.morphemes))


NLTK Sklearn Genism Text to Topic I aint no data scientist/machine learner.

What Im Lookin for

text = "Donald Trump became the president of America"
#some data science works
print(topics)
#prints ["politics"]

text = "Rihanna is starring in the new movie The Inception 2"
#some data science works
print(topics)
#prints ["movie","music"]

What I can do

I can extract words like Donald Trump,America,Rihanna using POS

I can get huge paragraphs/lists of articles/words on politics,movies etc and save them in a text file.

What i can't do

Make meaningful topics out of  those words like sports,politics,movies

What i want you do to

Point me in the right specific direction towards solving this problem aka enlighten me

I am staying quite generic since you asked for enlightenment, just mentioning some possible directions that you can explore.

You have basically two possibilities:

Classification of the text (Supervised learning).
Supervised means
that you need first to externally apply labels (for example manually by humans) to examples of texts
(labels could be "politics" or "show") and then use one of the
classification algorithms.  You have extracted words from the text,
so you could use a "bag of words" approach for classifying.  There exist adaptations
of classification algorithms (multi-label classification) in order to
provide multiple labels (such as one text is labelled both with
"music" and "movie").
You can find text corpora already pre-labelled, to train the algorithm and partly avoid the manual effort.
Clustering the text, topic modelling (unsupervised learning).
In this case you do not need to provide examples with labels but the algorithm will cluster the texts according to parameters like the similarity of two texts or on keywords/topics extracted from the texts.
Although this method do not require labeled examples, the clusters you get as output will require fine tuning, e.g. number of clusters to produce, names of the clusters, etc.

Since you mentioned SKlearn, you can find some directions on their web site: Text Feature Extraction.

Here is a comprehensive (bit older) summary: Machine Learning in Automated Text Categorization by Fabrizio Sebastiani


Predicting missing data. Looking for good data predicting technique I am analysing data for Countries Trade GDP. Some of the countries have missing GDP value for given a year. However, I have Grand Total for the entire region for that year. Is there a good data science technique to fill the missing Trade in GDP for the below example.

there are several ways how to interpolate the missing data depending on how long history record do you have available:

you can compute ration of GDPs r_i (where all r_i sums to 1) in year Y-1 of each state with missing values in year Y; then you know the amount of GDP in year Y which should be spread among missing (GDP_total - GDP_known) and multiply by r_i
a more sophisticated way would be computing the gradient from the timeline Y-1, Y-2, Y-3, ... and you predict a value for year Y for each missing state; then you would again compute ration r_i among missing and distribute remaining GDP accordingly. This regression can be done also backward if you know Y+1, Y+2, Y+3, ... or interpolation from Y-1 and Y+1.


What is a good way to start Data Analysis of unknown dataset (JSON data) I am working with an organization to analyse their data residing in Mongodb and to look for any trends/patterns in the data. I am quite new to the professional field of Data Analysis but have a good background of Statistics and Data Mining (University coursework). I will be doing a proof of concept on the data to understand if the data the organization is gathering is good for Analytics and if no what enhancements should they include in their datasets to make it better for Analytics. I do have some predefined questions that I am planning to answer but looking strategically what would be a good way to approach this kind of problem.

I have previously worked with some of the Kaggle datasets and projects in University where datasets were fixed and questions to be answered were given.

@Emre makes a great point. You should probably ask the business what they're trying to accomplish because only then will you know which data matters. You don't want to waste time cleaning up data that has minimal importance to the business.

Regardless of the business objective though, it's worth pointing out that Mongo DB is a pretty poor choice for storing analytical data. There is no enforced schema, so each record might be in a completely different and unexpected format. You might have entire columns missing, maybe no columns at all, wrong data types, duplicates, and no way to get easy summary statistics like you would in a relational database. You're pretty much going to have to do a database dump of your MongoDB data and probably comb through your records semi-manually, looking for commonly occurring schemas and going from there.

Summary: MongoDB is great for operational data when all you want to do a quick lookup. It's a poor choice when you want to do any sort of in depth analysis on it though. Relational databases are better for analytical queries because the data is structured and also because you can more easily enforce data quality.


how to use multiple generator in keras fit_generator() I want to train the multi-input model on a set of images. I use ImageDataGenerator.flow_from_directory() and fit_generator in keras.

The problem is I don't know how to use multiple generators.

This is my inputs & output generator and seeds are same:

input1 = imageDataGenerator.flow_from_directory(directory=base_data_directory + 'img/' + mode+'/', **img_generator_config)
input2 = imageDataGenerator.flow_from_directory(directory=base_data_directory + 'edge/' + mode,**edge_generator_config)
output = imageDataGenerator.flow_from_directory(directory=base_data_directory + 'label/' + mode, **label_generator_config)

How should I use fit_generator with this data?

In my opinion, you should build the custom data generator or just use the fit function of Keras and make your input data as follows:

x = [[image1, feature1], [image2, feature1], [image3, feature1]]

y = [class1, class2, class1]


Fitting an arimax model on out of sample dataset I have built an arimax model where we have sales data across time as the response variable and price is one of the external variables. I used the below code to build a simple arimax model. I had data points from 1 to 24, I have kept only 1 to 20 data points in the training dataset

library(stats)
fit=arima(window(tssales, end=20), order = c(0,1,1), xreg = window(tsprice, end=20))
summary(fit)
fcast=forecast(fit, h=5, xreg = window(tsprice, end=20))
plot(fcast)

Now when I try to fit the model from the training dataset in out of sample dataset (last 4 data points) I use the below code

library(stats)
out_of_sample=arima(window(tssales, start=21), xreg = window(tsprice, start=21), model=fit)

I am getting the following error

  Error in arima(window(tssales, start = 21), xreg = window(tsprice, start = 21),  :
    unused argument (model = fit)

the arima function from the stats library does not take an argument called model, which is why you are receiving an error. Here is the function signature:

arima(x, order = c(0L, 0L, 0L),
      seasonal = list(order = c(0L, 0L, 0L), period = NA),
      xreg = NULL, include.mean = TRUE,
      transform.pars = TRUE,
      fixed = NULL, init = NULL,
      method = c("CSS-ML", "ML", "CSS"), n.cond,
      SSinit = c("Gardner1980", "Rossignol2011"),
      optim.method = "BFGS",
      optim.control = list(), kappa = 1e6)

The returned object, does contain the model. Read the related documentation for more details.

I think your workflow is perhaps a little confused. You fit a model first on 20 data points, which is fine (more data would be nice!). You make some forecasts and plot it, which is also good - you can see if the model learned much and if there is perhaps some systematic error, e.g. simply predicting the previous time-step and not a more intelligent trend.

The final step, however, should be to once again make predictions on your hold-out data; the last 4 data points. You should not fit another model to the hold-out data! Just predict what you model would say for that data, which it has never before seen.

The reason we work like this, is so we can assess the model's performance independently from the data that was used to train it. We want to know what will happen in the future, when you get your 25th datapoint.

Have a look at this nice tutorial, which explains the mains concepts of ARIMA and has a working example. Here is a very similar tutorial, but it is a video.


Contributions of each feature in classification? I have some features and I am using Weka to classify my instances.

For example I have:

Number of adj   number of adverb    number of punctuation

in my feature set. However, I would like to know the contribution of each feature in the feature set. So what metrics or parameters are helpful to get the contribution of features?

This is called feature ranking, which is closely related to feature selection.

feature ranking = determining the importance of any individual feature
feature selection = selecting a subset of relevant features for use in model construction.

So if you are able to ranked features, you can use it to select features, and if you can select a subset of useful features, you've done at least a partial ranking by removing the useless ones.

This Wikipedia page and this Quora post should give some ideas. The distinction filter methods vs. wrapper based methods vs. embedded methods is the most common one.

One straightforward approximate way is to use feature importance with forests of trees:

Other common ways:

recursive feature elimination.
stepwise regression (or LARS Lasso).

If you use scikit-learn, check out module-sklearn.feature_selection. I'd guess Weka has some similar functions.


How to apply ensemble clustering method? I need to use ensemble clustering method by using python in my data set. I already applied k-means clustering by using scikit learn library. I also applied different classification method also find ensemble classification method in scikit-learn. Now I am confused is there any library exist in scikit learn for ensemble clustering or how I can apply ensemble clustering method on my data set?

You might find the following libraries helpful:

Python: Cluster Ensembles
R: diceR


XG Boost result interpretation for unbalanced datasets (Accuracy & AUCROC) My dataset is of shape – 5621*8 (binary classification)

Label/target : Success (4324, 77 %) & Not success (1297, 23 %)
(success and Not success were been equally important for my prediction i.e, TP & TN)

I split my data into 3 (Train, Validate, test)

For train & Validate i perform 10 fold CV.

Test is the seperate data, which I evaluate for each folds

I tune my scale_pos_weight ranging between 5 to 80, and

Finally I fixed my values as 75 since I got average higher accuracy rate for my Test set (79 %) for those 10 folds
But, If i check my average auc_roc metrics it is very poor, i.e only 50 % for all 10 folds.

If i did not tune scale_pos_weight my avg.accuracy drops to 50% & my avg auc_roc increases to 70 %.
How can I interpret from the above results between AUCROC & Accuracy in this situation?
What might be the problem in my case?

With Success already being the larger class, you probably shouldn't be using a scale_pos_weight larger than one: you want to scale the positive class's contribution to the loss function down rather than up.
I suspect that's what's happening in the first case.  With scale_pos_weight=75, the model ends up basically only caring about the positive class, predicts everyone is in the positive class, and so your accuracy is just a little better than the 77% baseline you'd expect with that strategy.  With that motivation, it's not too surprising the AUC is poor, although I wouldn't have expected a drop all the way to the 50% baseline...
If you don't use scale_pos_weight (you said "if I did not tune", but does that mean you left it at the default 1?), then the model performs better in rank-ordering (AUC=70%), but not so well in the hard classification.  You might want to tweak the prediction threshold here; there's probably a different threshold that will perform better for accuracy score.  You could also try scale_pos_weight=0.25 or so; that should make the default cutoff better, hopefully with little effect on AUC?


Multi-label classification using output quantization Problem statement

It's a fact that in order to train the network for multilabel-dataset, a one-hot-vector output is usually used.

Example:

  dog         [1 0 0]

  cat         [0 1 0]

  rabbit      [0 0 1]

Consequently, we're increasing size of the weight matrix as well as the required training time.

Question: Is there an approach that we can use one output and quantize it for different classes?

Example:

  0 <= output < =1

  dog [0.0 - 0.33]

  cat [0.33 - 0.66]

  rabbit [0.66 - 1]

TL;DR

Bad idea, don't do that.

Explanation

1)
You are introducing an ordering in the classes that doesn't exist. Basically, with your example, you are saying that: "dog is close to cat", "cat is close to rabbit", but "dog is far from rabbit", which is an extra learning feature your network needs to learn.

For example, if the network outputs 0.35, then it is class cat but it is also close to dog and far from rabbit.

2)
Another problem, that some results are not possible to be represented.

For example, what if you changed the range to:

  0 <= output < =1

  cat [0.0 - 0.33]

  rabbit [0.33 - 0.66]

  dog [0.66 - 1]

How will you represent that the same example from before ("it is class cat but it is also close to dog and far from rabbit.")? It become impossible.

With the previous representation you could have:

output = [0.5, 0.0, 0.5] # cat, rabbit, dog

how would you map that to the new range where cat = [0.0-0.33] and dog = [0.66-1.0]? Can we pick the middle (0.5)? No, because that is class rabbit.

Conclusion

By doing that, you reduce the number of weights in the final layers (good), however, you introduce dependencies which needs to be learned and some results are not representable or interpretable.


What exactly is a dummy trap? Is dropping one dummy feature really a good practice? So I'm going through a Machine Learning course, and this course explains that to avoid the dummy trap, a common practice is to drop one column. It also explains that since the info on the dropped column can be inferred from the other columns, we don't really lose anything by doing that.
This course does not explain what the dummy trap exactly is, however. Neither it gives any examples on how the trap manifests itself. At first I assumed that dummy trap simply makes the model performance less accurate due to multicollinearity. But then I read this article. It does not mention dummy trap explicitly, but it does discuss how an attempt to use OHE with OLS results in an error (since the model attempts to invert a singular matrix). Then it shows how the practice of dropping one dummy feature fixes this. But then it goes on to demonstrate that this measure is unnecessary in practical cases, as apparently regularization fixes this issue just as well, and algorithms that are iterative (as opposed to closed-form solution) don't have this issue in the first place.
So I'm confused right now in regards to what exactly stands behind the term "dummy trap". Does it refer specifically to this matrix inversion error? Or is it just an effect that allows the model to get trained but makes its performance worse, and the issue described in that article is totally unrelated? I tried training an sklearn LinearRegression model on a OHE-encoded dataset (I used pd.get_dummies() with the drop_first=False parameter) to try to reproduce the dummy trap, and the latter seems to be the case: the model got trained successfully, but its performance was noticeably worse compared to the identical model trained on the set with drop_first=True. But I'm still confused about why my model got successfully trained at all, since if the article is to be believed, the inversion error should have prevented it from being successfully trained.

There are two main problems -

You have one Feature which is correlated (multi-collinearity) to all the others.

If you are trying to solve using "closed-form solution", the following will happen

$y = w_0 + w_1X_1 + w_2X_2 + w_3X_3$
$w_0$ is the $y$ intercept and to complete the matrix form 1=$X_0$. Hence,
$y = w_0X_0 + w_1X_1 + w_2X_2 + w_3X_3$
Solution for $w$ is $(X^{T}X)^{-1}X^{T}y$
So, X must be an Invertible Matrix. But,
If the model contains dummy variables for all values, then the encoded columns would add up (row-wise) to the intercept ($X_0$ here)(See below table) and this linear combination would prevent the matrix inverse from being computed (as it is singular).
\begin{array} {|r|r|}
\hline
X_0 & X1 &X2 &X3 \\
\hline
1 &1 &0 &0 \\
\hline
1 &0 &1 &0 \\
\hline
1 &0 &0 &1 \\
\hline
\end{array}

why my model got successfully trained at all since if the article is to be believed, the transposition error should have prevented it from being successfully trained.

Valid question! Aurelien Geron(Author of "Hands-On Machine Learning" has answered Here.
- The LinearRegression(Scikit-Learn) class actually performs SVD decomposition, it does not directly try to compute the inverse of X.T.dot(X). The singular values of X are available in the singular_ instance variable, and the rank of X is available as rank_
On Performance

In a practically large dataset, a closed-form solution is not preferred. May use
an Iterative approach algorithm i.e. Gradient-Descent
Multi-collinearity too, will not impact the performance but Interpretability of Features.
Coeff changes depending upon which dummy is removed - This is obvious as each dummy is now a Feature with a different level of contribution (based on data). The only thing that is sure is their effect together and one-less is the same.
The inconsistent result you are getting should be due to some other issue.

Dummy-variable Trap

I have never heard of this term except "Udemy course A-Z ML". So I don't think that there is any special meaning of the word "trap" if you understand the points(i.e. Singularity, Multi-collinearity, and Interpretability) separately
References -
www.feat.engineering - Sec#5.1
Sebastian Raschka
stats.stackexchange


Dropout backpropagation implementation details Just to summarize
Understanding dropout and gradient descent
and
https://stats.stackexchange.com/questions/207481/dropout-backpropagation-implementation

Suppose I need to implement inverted dropout in my CNN. All the neuron outputs in dropout layer during feedforward phase are multiplied by mask/p, where mask is 0 or 1, p is retain rate. But should I apply the same operation (include division by p) at the backpropagation phase? I suppose positive answer (see the second link above), but I need to know for sure.

As given in the links, the answer is yes! note that you divide the mask by p so that you won't need to multiply by p in the test time and since this is a coefficient for the new activation, it will come out of the derivative in chain rule in backprop.


How do we choose which feature(column) should we normalize or scale I am a newcomer to machine learning, I was going through scaling and normalization. When I tried to explore most of the documentation on web, I found people explaining that the dataset needs to be scaled and normalized.
But wanted to understand when to scale or normalize a column? And how to choose which feature/column in a dataset for scaling and normalization.
And how to decide on which normalization algorithm to choose for a column?

When to scale or normalize a column?

When you are using an algorithm that assumes your features have a similar range, you should use feature scaling.

The main example is distance algorithms like Euclidean distance.
Thus, the algorithms that are distance-based algorithms like K-means, SVM, etc also require feature scaling.
Gradient Descent Based Algorithms also require your data to be scaled to safely converge. From the article:

The difference in ranges of features will cause different step sizes for each feature. To ensure that the gradient descent moves smoothly towards the minima and that the steps for gradient descent are updated at the same rate for all the features, we scale the data before feeding it to the model.

For more, check this.

How to choose which feature/column in a dataset for scaling and normalization?

If the ranges of your features differ much then you should use feature scaling. If the range does not vary a lot like one of them is between 0 and 2 and the other one is between -1 and 0.5 then you can leave them as it's. However, you should use feature scaling if the ranges are, for example, between -2 and 2 and between -100 and 100.
Check this explanation for a better and detailed explanation.

And how to decide on which normalization algorithm to choose for a column?

I assume here you are asking which feature scaling method to use. The answer to this question also may vary due to context, however, the general answer is:

Use Standardization when your data follows Gaussian distribution.
Use Normalization when your data does not follow Gaussian distribution.

However, some algorithms do not require feature scaling and they can handle it. For example, since Decision Tree builds the model by splitting the features, it is invariant to features' scale.


Weighted Linear Combination of Classifiers I am trying to build an ensemble of classifiers whereby I want my algorithm to learn a set of weights such that it can weight the outputs of different classifiers for a set of data points.

I am wondering, how would I go about learning these weights? I tried using automatic differentiation but the weights are not moving at all (no gradient information).

Does anyone know how I can fix this?

I don't know how to fix your automatic differentiation, but I can show you what I did (and I have seen others do too) when I wanted to achieve the same thing. You can fit a linear meta-classifier on the outputs of your classifiers that you want to ensemble. Here is the implementation from my scikit toolbox:

'''
-------------------------------------------------------
    Stack Classifier - extrakit-learn

    Author: Simon Larsson <larssonsimon0@gmail.com>

    License: MIT
-------------------------------------------------------
'''

from sklearn.base import BaseEstimator, ClassifierMixin
from sklearn.utils.validation import check_X_y, check_array, check_is_fitted
import numpy as np

class StackClassifier(BaseEstimator, ClassifierMixin):
    ''' Stack Classifier

    Ensemble classifier that uses one meta classifiers and several sub-classifiers.
    The sub-classifiers give their output to to the meta classifier which will use
    them as input features.

    Parameters
    ----------
    clfs : Classifiers who's output will assist the meta_clf, list classifier

    meta_clf : Ensemble classifier that makes the final output, classifier

    drop_first : Drop first class probability to avoid multi-collinearity, bool

    keep_features : If original input features should be used by meta_clf, bool

    refit : If sub-classifiers should be refit, bool
    '''

    def __init__(self, clfs, meta_clf, drop_first=True, keep_features=False, refit=True):
        self.clfs = clfs
        self.meta_clf = meta_clf
        self.drop_first = drop_first
        self.keep_features = keep_features
        self.refit = refit

    def fit(self, X, y):
        ''' Fitting of the classifier

        Parameters
        ----------
        X : array-like, shape (n_samples, n_features)
            The training input samples.

        y : array-like, shape (n_samples,)
            The target values. An array of int.

        Returns
        -------
        self : object
            Returns self.
        '''

        X, y = check_X_y(X, y, accept_sparse=True)

        # Refit of classifier ensemble
        if self.refit:
            for clf in self.clfs:
                clf.fit(X, y)

        # Build new tier-2 features
        X_meta = build_meta_X(self.clfs, X, self.keep_features)

        # Fit meta classifer, Stack the ensemble
        self.meta_clf.fit(X_meta, y)

        # set attributes
        self.n_features_ = X.shape[1]
        self.n_meta_features_ = X_meta.shape[1]
        self.n_clfs_ = len(self.clfs)

        return self

    def predict_proba(self, X):
        ''' Probability prediction

        Parameters
        ----------
        X : {array-like, sparse matrix}, shape (n_samples, n_features)
            The prediction input samples.

        Returns
        -------
        y : ndarray, shape (n_samples,)
            Returns an array of probabilities, floats.
        '''

        X = check_array(X, accept_sparse=True)
        check_is_fitted(self, 'n_features_')

        # Build new tier-2 features
        X_meta = build_meta_X(self.clfs, X, self.keep_features)

        return self.meta_clf.predict_proba(X_meta)

    def predict(self, X):
        ''' Classification

        Parameters
        ----------
        X : {array-like, sparse matrix}, shape (n_samples, n_features)
            The prediction input samples.

        Returns
        -------
        y : ndarray, shape (n_samples,)
            Returns an array of classifications, bools.
        '''

        X = check_array(X, accept_sparse=True)
        check_is_fitted(self, 'n_features_')

        # Build new tier-2 features
        X_meta = build_meta_X(self.clfs, X, self.keep_features)

        return self.meta_clf.predict(X_meta)

def build_meta_X(clfs, X=None, drop_first=True, keep_features=False):
    ''' Build features that includes outputs of the sub-classifiers

    Parameters
    ----------
    clfs : Classifiers that who's output will assist the meta_clf, list classifier

    X : {array-like, sparse matrix}, shape (n_samples, n_features)
        The prediction input samples.

    drop_first : Drop first proba to avoid multi-collinearity, bool

    keep_features : If original input features should be used by meta_clf, bool

    Returns
    -------
    X_meta : {array-like, sparse matrix}, shape (n_samples, n_features + n_clfs*classes)
                 The prediction input samples for the meta clf.
    '''

    if keep_features:
        X_meta = X
    else:
        X_meta = None

    for clf in clfs:

        if X_meta is None:
            if drop_first:
                X_meta = clf.predict_proba(X)
            else:
                X_meta = clf.predict_proba(X)[:, 1:]
        else:
            if drop_first:
                y_ = clf.predict_proba(X)
            else:
                y_ = clf.predict_proba(X)[:, 1:]
            X_meta = np.hstack([X_meta, y_])

    return X_meta

This would allow you to use any meta-classifier, but with linear models like ridge/lasso/logistic regression it will acts as learned linear weights of your ensemble classifiers. Like this:

from sklearn.tree import DecisionTreeClassifier
from sklearn.linear_model import LogisticRegression
from xklearn.models import StackClassifier

X, y = make_classification(n_classes=2, n_features=4, n_samples=1000)

meta_clf = LogisticRegression(solver='lbfgs')
ensemble = [DecisionTreeClassifier(max_depth=1),
            DecisionTreeClassifier(max_depth=5),
            DecisionTreeClassifier(max_depth=10)]

stack_clf = StackClassifier(clfs=ensemble, meta_clf=meta_clf)
stack_clf.fit(X, y)

print('Weights:', stack_clf.meta_clf.coef_[0],' Bias: ', stack_clf.meta_clf.intercept_)

output:

Weights: [0.50017775 2.2626092  6.30510687]  Bias:  [-4.82988374]


ValueError: Input contains NaN, infinity or a value too large for dtype('float32') I got ValueError when predicting test data using a RandomForest model.

My code:

clf = RandomForestClassifier(n_estimators=10, max_depth=6, n_jobs=1, verbose=2)
clf.fit(X_fit, y_fit)

df_test.fillna(df_test.mean())
X_test = df_test.values
y_pred = clf.predict(X_test)

The error:

ValueError: Input contains NaN, infinity or a value too large for dtype('float32').

How do I find the bad values in the test dataset? Also, I do not want to drop these records, can I just replace them with the mean or median?

Thanks.

With np.isnan(X) you get a boolean mask back with True for positions containing NaNs.

With np.where(np.isnan(X)) you get back a tuple with i, j coordinates of NaNs.

Finally, with np.nan_to_num(X) you "replace nan with zero and inf with finite numbers".

Alternatively, you can use:

sklearn.impute.SimpleImputer for mean / median imputation of missing values, or
pandas' pd.DataFrame(X).fillna(), if you need something other than filling it with zeros.


Are there readily available models that can handle conditional correlation? I've been working my way through the features of the Kaggle House Prices dataset (Note: this is a non-ranking entry, so this is just for exercises), and I've found a couple situations where there is a positive correlation between the feature and the house sale price, but only if the data exists. In one case (shown below) over 10% of the dataset had null (means "does not apply", not "missing", so I can't fill it in with imputation), but of the non-null values, a scatter plot showed a positive correlation. This looks like a conditionally useful feature and I'd like to keep it, but the nulls are tripping me up.
I experimented and replaced the null values with 0, and when I looked at the scatter plot again, I found that the values that the new 0s spanned the a good chunk of the price range and altered the trend.
Before replacing null values (blue trend line estimated by hand):

After replacing null values with 0 (~13% of the dataset):

Is there a readily available model in sklearn or some other python library that will perform a regression fit only if the data is non-null? If not, would it be best to just drop the column?
Note: This is a small dataset (<1500 entries). This is too small for neural network techniques.

It is totally ok to drop null values here (in your case all the null values of LotFrontage), because this data isn't real. This is a standard part of data preprocessing called data cleaning. If the reason for the null values is known you could do additional steps like imputing or filling, but without this information just dropping them would be fine.
I would be careful changing null to zero however, as this is now creating data (i.e. apartments with zero size front lots).
Finally, you can see from your scatter plot that the relationship is not completely linear, so the linear model may have a large degree of error and/or low R squared values. So it may be more suitable to pick another algorithm for this dataset (e.g. 2D gaussian model).


Classification of obfuscated text data I am relatively new to data science and have an exercise task. This consists of the classifications of excerpts of texts. However, the texts are obfuscated such that one cannot read the words, spaces etc. But the "patterns" are preserved. I have a training set of the following form. One .txt file with the text excerpts looking like

shdbcjhbjhbefbhbwhbkbehbwbwbfhwb /
wbhbwtjnwkjbrfbqlenk /
wjnfkjebrkbrghkbibgibib /
tberbtewtwbkwtjbrkbwkbtwrbt /
.
.
.

and one .txt file with the labels of the texts looking like

1 /
4 /
11 /
0 /
.
.
.

This means that each string of symbols belongs to a certain text and one knows to which one.

The task is now to set up a classifier, to train it on above training data and to test it on test data which also consists of text excerpts.

My basic idea was to interpret the data as images and to set up a deep learning NN with e.g. TensorFlow. However, I am not sure about the shape of the data which is needed to feed the data to the NN.

Do I have to create label-folders containing all texts with a certain class-label or is there a more direct way (so far I created vectors in R consisting of the texts and the labels)? Must the excerpts all have the same length? How do I have to manipulate the data until I can feed it to the NN?

The text is obfuscated for you but to the computer, this is as incomprehensible as English for instance. When classifying text, you always first vectorize your input, meaning that your text becomes a vector. This vector typically represents the frequency of the tokens in your vocabulary in that particular string (bag-of-words representation).

What you can do is represent each document/string as a vector representing the frequencies of characters, character n-grams, or even words. One you have your input, you can deal with it as any classification task.

Naive Bayes has shown to be a good lower bound for these types of tasks.


Why use mean revenue in a split test? I asked a data science question regarding how to decide on the best variation of a split test on the Statistics section of StackExchange. I hope I will have better luck here. The question is basically, "Why is mean revenue per user the best metric to make your decision on in a split test?"

The original question is here: https://stats.stackexchange.com/questions/107599/better-estimator-of-expected-sum-than-mean

Since it was not well received/understood I simplified the problem to a discrete set of purchases and phrased it as a classical probability problem. That question is here: https://stats.stackexchange.com/questions/107848/drawing-numbered-balls-from-an-urn

The mean may be the best metric for such a decision but I am not convinced. We often have a lot of prior information so a Bayesian method would likely improve our estimates. I realize that this is a difficult question but Data Scientists are doing such split tests everyday.

If you've got prior information then you should certainly not use simple mean in a split test. I assume you're trying to just predict which group will produce the greatest amount of revenue overall, by trying to emulate the underlying distribution.

Firstly, it's worth noting that any metrics you choose will actually reduce to mean in a pretty trivial way. Eventually mean will necessarily work out, though using a standard bayesian method to estimate the mean is probably your best bet.

If you've got a prior then using a standard bayesian approach to update the prior on your mean revenue is probably the best way to do it. Basically, just take the individual results you get and update a multinomial distribution representing your prior in each case.

If you want some more full background on multinomial distributions as bayesian priors are pretty well, this Microsoft paper does a pretty good job of outlining it. In general, I wouldn't care so much about the fact that your distribution is technically discrete, as a multinomial distribution will effectively interpolate across your solution space, giving you a continuous distribution that is a very good approximation of your discrete space.


Implementation of kmeans clustering using R I have implemented kmeans clustering on iris dataset (inbuilt dataset) in R. The code is given below:

X=as.matrix(iris[-5]);

K=3;

prevCentroids=matrix(0,K,dim(X)[2]);
centroids=X[sample(1:dim(X)[1],K),];

dot=numeric(3);
C=numeric(150);

while(!isTRUE(all.equal(centroids,prevCentroids)))
{
  for(i in 1:dim(X)[1])
   {
    for(j in 1:dim(centroids)[1])
     {
       dot[j]=(X[i,]-centroids[j,])%*%(X[i,]-centroids[j,]);
     }
     C[i]=which.min(dot);
   }

  prevCentroids=centroids;

  for(k in 1:K)
   {
     centroids[k,]=colMeans(X[which(C==k),]);
   }
}

print(cbind(iris,C));

Sometimes, with this code, I get 85% clustering correct. But sometimes, it is just 37% as correct, if I compare it with the already clustered inbuilt iris dataset.

Could anyone please tell me where I am going wrong?

Actually, this is the expected behavior for running a k-means algorithm on the iris dataset. The iris dataset contains only two distinct clusters. So if you randomly set up three centroids, the third centroid will either end up on the right or on the wrong cluster, causing the algorithm to split that cluster into two (see the left picture).

Your outcome with 37% accuracy will most probably look somewhat similar to the left picture where the left cluster has been split into two clusters. The 85% accuracy will most probably come from those times in which the right cluster has been split into two clusters (you cannot be sure though without printing the results).

For testing purposes, you can simply use set.seed(123) before running the algorithm, so that you always get the same results. However, this doesn't solve the problem that the iris dataset isn't ideal for clustering purposes. I would suggest you check out other datasets or you give the R-package cluster.datasets a try.


Water supplier ontology I am new in this world, I am going to start working on my final degree project about data mining. I need to look for an ontology related with water supplier companies.

I mean, I have words like: SuppliedWater, RegisteredWater, LengthSupplyingNetwork...

I am looking at lod-cloud, but I don't know where to start looking for this.

Any help would be appreciate.

Thanks for your help.

I finally used this website, since I found it easier to found any ontology.
For example I found this one, very similar to what I was looking for.


how to balance the data set with different number of observations I am pretty new to ds field and recently I was working on a self project of the clustering model. My goal is to create clusters and see in each cluster what the common features are between customers. The data set contains customer's information, when and what product they order and more. Below is a sample of the data:

id   gender   age   order_dt   ship_dt  product
1     male     23   1/2/2018   1/9/2018    a
1     male     23   1/5/2018   1/6/2018    b
2    female    45   1/10/2018  1/20/2018   c
3    female    30   1/1/2018   1/2/2018    a
3    female    30   1/15/2018  1/20/2018   c
3    female    30   1/21/2018  1/21/2018   b
3    female    30   1/29/2018  2/1/2018    a

However, each id could contribute to a different number of records in the data set. Some could have a lot of records because they order many times while some only order once. I have googled around for unbalanced data, but most of them are talking about unbalancing in one category(feature maybe?) instead of the unbalanced number of observation.
Should I aggregate on each id so that each id would only have 1 record in the data set or if there is any technique to handle data like this?

Thanks in advance,

Clustering the data depends on what your objective is

In this case I am assuming you would want to see similarity among customers.

This would mean that the data should be unique at a customer level. This will help find similarities in some customer attributes.

For ex
Number of transactions
Gender
Age group
Weekend or weekday txns etc

Hence you would need to first aggregate your data at a user level and create features of interest like the examples above and the try to cluster the customers.


Multi-country model or single model I am working on a ML model to be deployed in a product operating in many countries.
The issue that I am having is the following: should I

train one model and serve it for all countries?
train a model per country and serve each model in its country?

I've faced this problem several times, and to me, there's a trade-off in the learning: in the first case, the model has more data to learn, and it'll be more robust (also, the solution is simpler). In the second case, I'll have a more tailored model to each country, and will be able to see effects that are specific to that country.
I'm very interested in knowing if there's an intermediate solution - a general model with some country-specific fine-tuning that can see all the data but also specialize in each specific country. If I were to use Neural Networks, this fine-tuning is natural - you train some epochs with all the data, then the last epochs with each specific country. I am wondering if something similar can be done in Linear regression models and Xgboost, which are the models I generally use.
Is there any literature on this? I think it is kind of a generic topic and there should be some.

In the paper of Hinton - Distilling the knowledge of Neural Networks, the following is mentioned (Section 5) when defining specialist models:

When the number of classes is very large, it makes sense for the cumbersome model to be an ensemble that contains one generalist model trained on all the data and many “specialist” models, each of which is trained on data that is highly enriched in examples from a very confusable subset of the
classes (like different types of mushroom).

What they do is they use a general model first and then a specialist model each to focus on a different subset of the classes.
You could consider your problem something similar, instead of a specialist to classes, and specialist in countries. This way you could build a country(cluster of countries) specific ensemble of models.


How do I find a repeating pattern of unknown length and start within a string Background:

I am trying to do some analysis on our customer's around identifying early markers for debt delinquency.  We bill customers on a weekly basis and payment patterns will differ between accounts.  Good customers will normally pay Weekly, Fortnightly, 4 weekly or monthly on a 4-4-5 pattern.  Other customers will have an erratic payment pattern.

Even where the payment pattern is consistent, they may be overpaying or not be paying the balance in full so there may be an underlying rising or falling debt.

Work So Far:

To this end I have created two strings to represent:

Whether the weekly balance has decreased (v), increased (^) or not moved (-) (infer a payment has been made if balance is down or unchanged) and ...
The level of indebtedness on the account based on the weekly charge ( - for in credit, . for balance cleared, 0-9 for balance less than 10 weeks, X for balance over 10 weeks)

So I now have a number of strings that look like this (good accounts)

^v^v^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^^^^    231212-012-012-012-012-0120112-012-0-0120012-012-012-0120112-012-012-012-012-0123
^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^^^^    12-012-012-012-012-012-012-012-012-0-012-012-012-012-012-012-012-012-012-012-0123

and like this for bad/erratic accounts

^v^^^vv^^v^^^v^^^v^v^v^^^v^^^v^^^v^v^^vv^^^v^^vv^^vv^^vv^^vv^^vv^^vv^^vv^^vv^^v^    341234112312231234233412341234123423123312341233123301220122-011-011-011--00--001
v^v^v^^v^vv^vv^vv^v^^^^v^^^v^^^^v^v^^^v^^^v^^^^v^^^v^^^^v^^^v^^^v^^^^v^^^v^^^v^ XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX9XXX9XXXX9XXX9XXXX9XXX89XX89XXX89XX789X78

Next Steps:

What I want to do now is identify accounts with specific payment patterns:

Fortnightly = ^v^v
4weekly = ^^^v^^^v^^^v
Monthly = ^^^v^^^v^^^^v (note 4-4-5)

I also want to identify where those patterns have changed. Bearing in mind that on a 4 weekly or monthly pattern we may not be at the start of a pattern I also need to try and find the 'seed' week for each pattern

I will regard a pattern as consistent if I have seen it repeat at least 3 times so 12 weeks for the fortnightly pattern, 36 weeks for the 4 weekly and 36 weeks for the Monthly [monthly could be 445, 454 or 544 but must repeat] so I then want to flag up accounts that were consistent that have become erratic or changed their pattern.  I have data going back up to 5 years.

I have the data in Microsoft SQL and currently the only analysis tools I have to hand are Microsoft SQL and Excel.  I'm not looking for code, just what the next practical steps should be in my strategy to analyse the data.

I'm thinking find the first down step in the pattern and remove this and the preceding up ticks (and associated characters from the multiplier) then run the remaining string through a number of REPLACE() functions but this doesn't feel very 'smart'

Is there a better strategy to adopt as it feels like I am MUNGing the data?

From your description, it sounds like autocorrelation could be a good initial step to identify the frequency. Autocorrelation is basically the product of the sequence with itself shifted at 0, 1, 2, 3, and so forth. If a client tends to pay monthly, you will see the first positive bump around 4, because that is where each payment aligns with the payment after it. Note that this will work irrespective of the start of the sequence, and is even resilient to some noise. Even Excel seems to have an autocorrelation function.

Of course, you could refine this further with other steps, and/or try this on different windows (in case the patter changes over time).


Methods to detect this kind of outliers Background
I don't know much about (or to say anything) about data science or machine learning. But I'm interested in learning and thought this problem can be solved with machine learning. That's why I'm posting this here, don't even know if this is appropriate or not.
Problem
I have some 2D data in pairs of (x,y). I only have the data and don't know anything about what kind of functional feature the data have. Now this data randomly have some wrong values or (should I say) outliers. And I have to fix them before working with that data. Lets show an example:

As you can see the data have a nice uniform feature except for the 3rd point, and obviously that's a wrong point. I have fix this kind of points. You can see If I can just detect that point then I can just delete and fit it with surrounding (e.g with a spline).  Now I have lots of data just like this, and plotting every time to check and then manually detect and smooth those points is really tiresome.
Now I don't have any idea about how to proceed with this problem. I tried to search but was lost in the huge world of machine learning. So, what kind of methods and technique do I have to learn, so that I can aromatically detect those points. You can show me a sample code (in python preferably) to do this kind of jobs. Thanks.

I recently had a similar problem (removing abnormal peaks from a time series). That's what I suggest you:

Get the smoothed trend. There are several techniques you can employ, such as various forms of exponential smoothing.
Find the difference between actual trend observations and smoothed ones.
Normalize this distribution of distances (using Z-score, i.e. sklearn's StandardScaler)
Substitute the observations that lie k standard deviations away from the mean (that is 0). The choice of x can be arbitrary or data-driven; in my case I chose k = 3 (i.e. a very conservative anomaly removal). You can use the smoothed values as a substitute. In your case, the interpolation could be perhaps a good choice (that's up to your preference).

This will automatically remove abnormal peaks like the one you displayed.


Why would it be bad to fit a regression model to a binary classification problem? Let's say that we have a binary classification problem.

Why would it be bad to fit a linear regression and then classify given a threshold?

The output would be continuos and it could be out of range, but we could convert it to binary with the threshold.

It would work, afterall ML is a lot about engineering and hacking the things together. However, it would perform not so well, as for example logistic regression. If you compare the a linear line and logistic regression you will notice that the gradients of their respective loss functions for points near the decision boundary ("threshold") and far away from it are quite different. While for logistic regression points, that lie close to the decision boundary or are wrongly classified the absolute value of the gradient is larger. As a consequence the optimization, if a gradient approach is chosen, will be impacted by these points the most. Thus, the curve will be fitted to improve the classification.

Whereas in linear regression the distance of each point to the linear line is to be minimized, so independently of if the point is already classified correctly, it will impact the optimization. This leads to the optimization to be influenced a great deal by points that are already classified correctly and don't really need to influence the adjustement of the linear function. They suppress the importance of the points that are actually wrongly classified and that should impact the optimization at large.