Noramlization Time Series to Predict Stocks exact Price I am trying to made a model for neural network that tries to predict prices of the stocks point by point using a LSTM (yeah I know that probably did not get anything and I should predict up/down or something realted boolean decision but I like to try this first)

I wil use as features HIGH, LOW, OPEN of last 60 days and try to predict ADJ_CLOSE of next day.

The probem come when I need to normalize the data. I am using data from AAPL from 1980 to 2019 the main problem is that in 80's stocks prices are in pennys and in 10's they are in houndreds of dollars, so I decided to do window normalization using MinMaxScaler from Sklearn.
There is no problem with the train data, I have 7887 data points so I scaled them by windows of 500 and last remaing separetly. The code is like this

x_train = train.values

window_size = 500
for i in range(0,7500,window_size):
    x_train[i:i+window_size] = min_max_scaler.fit_transform(x_train[i:i+window_size])

x_train[i+window_size:] = min_max_scaler.fit_transform(x_train[i+window_size:])

The problems comes when testing I do not know that doing something like

x_test = test.values
x_test = min_max_scaler.fit_transform(x_test)

is kind of chating because they are fiting by themselves. But I can not do

x_test = test.values
x_test = min_max_scaler.transform(x_test)

after window normalization of train data(using last fitted scaler) because the range between min and max on last window of train data difers a lot from the range in test data and in that case the normalization would not be well done and the test data would not fit between [0,1].

Any help?

You should always fit the scaler with the train data. This is one of the typical validation problems, it is hard to replicate train test conditions.

Have you considered doing rolling/rollout window split?

This way the range of values will be closer between train and test.

THe problem of this is that you loose  some of the training information. On the other hand information about 40 years ago of stock has not much to do when forecastin the present


Does multi-head attention remove the need for self-attention? The title may be confusing but suppose I were to build Transformer Neural Network with a masking network that utilizes multi-head attention (like that in SepFormer), would adding self-attention in the encoder and decoder still be necessary?

Self-attention means X pays attention to X, as opposed to "normal" attention where X pays attention to Y.
Multi-head attention is as opposed to single-head attention. You can choose to use multi- or single-head attention equally for self-attention and for normal-attention.
Masking X and/or Y is a third independent aspect of a design.
In a Transformer encoder there is only self-attention and feed-forward networks (FFNs). Without the self-attention aspect the layers are reduced to just being FFNs. The FFNs operate only on one item, so without the self-attention they would never discover relationships between items. In a text application that means it would never discover the relationships between words. In a time series analysis it would never discover relationships across time.
The decoder uses both self-attention and normal attention. If there is no relationship across words/time in the output of the model, then maybe you could get rid of the self-attention part. But, if the transformer is a useful model for your data, that is unlikely to be the case.
So if you wanted a clear-cut yes/no answer (to if it removes the need for self-attention): in the case of the encoder, "no", and in the case of the decoder, "probably not".
UPDATE: Having now read the papers (SepFormer, and the two in comments, TSTNN, and DPTNet) I think the above still applies. Those models only use the encoder part of the transformer.
DPTNet and TSTNN change the first part of the FFN with a RNN or GRU respectively, so I suppose theoretically it can learn about other tokens from that; but the motivation for this is as an alternative to needing positional encodings, and I doubt it is worth the trade-off of no longer being able to process the data in parallel.


Multivariate linear regression accounting for threshold / data cleaning I am trying to make a linear regression model for the sale price of a house based on many variables (based on the data from this Kaggle challenge https://www.kaggle.com/c/house-prices-advanced-regression-techniques)

The distribution above is the 2nd-floor size in square feet and the y-axis is the sale price. It shows a clear linearity, except for the fact that homes without a 2nd floor clearly are not for sale for 0 dollars.

I have many variables like this that have some threshold either upper or lower that has a large distribution of response for a single value. If I simply exclude these values then the intercept for this curve will be through the origin. Should I let that be the case and assume that the $0 price tag will be corrected by the other variables in my regression?

What is the best way to treat/fit data such as this?

Thanks!

You could try including "Does not have a 2nd floor" as a separate categorical variable, which you could encode as 1 or 0 in your linear model.  Something like:

price ~ no_second_floor + area_of_second_floor + [Other Variables]

Event without the other variables, it's already going to be able to fit the data better than price ~ area_of_second_floor because, instead of forcing the price of the single-floor houses to $0, it would be able to fit it to the average price of all the single-floor houses in your data set - which is the best you can do for those houses until you add other variables besides area_of_second_floor.

(Actually, it wouldn't be forced to $0 if you were to include a constant term in your model, but the constant term that best lets you fit the linear portion of the data still probably isn't what you want for the single-floor houses, some of which - as you see - are quite expensive).


Is Siamese network rotation invariant? Is Siamese network rotation invariant which means if I train my siamese network on the different rotated versions of the same image so will it treat each image as different image or same.

Also if I pass for example a image of cat and a rotated image of same cat as a pair to siamese network will it give a similarity score of 0 or 1.

That depends on the latent representation the model has learned. Typically it won't be rotational invariant because there are no inbuilt mechanism in vanilla CNN or constraints enforcing that!

If you want to learn rotational invariant representation then either you can use a Rotation Invariant CNN or you may set up the training objective of the Siamese network such that when you pass random rotated versions of the same image of the same input, you classify as positive or else negative. The 2nd method is enforcing the constraint that your representation must be rotational invariant.


Is it reasonable that a problem "solved" by a traditional ANN can also be solved by a CNN? There is a data science numerical problem, which me and my team were able to get an ANN model that predicts down to a 1% MAPE error (with roughly 70000+ trainable parameters).

Given the nature of the dataset, this data can also be processed as images, which has led me to trying a CNN based approach to solve the same. As of now, I'm able to converge to error figures of 14% with around 2000 parameters. Is it reasonable to assume that I'd be able to bring down the error even further with a CNN based architecture (without a significant increase in the parameters)?

EDIT: Basically, what I mean by image is, in the ANN case, the data is being processed as a 1D vector, but as a 2D vector in the CNN case.

In theory yes in practice maybe.

What do I mean by that. There is universal approximation theorem for smooth functions regarding ANN, there is also theory backing this up for CNN variant. Another thing that backs this claim up is that you can replicate any ANN with CNN architecture, hence effectively backtracking to the original ANN.

Practice, you say this data can also be processed as images it can be processed but does it have the same amount of informativness as other data representation? Could be but it does not have to, think about doing regression with images of time series data, sure you can catch some general trends but to really catch all of it it will get expensive, really fast.


Why does a machine learning algorithm need a bias? The first line of section 2.7.3 in Mitchell's Machine Learning is:

"A Learner that makes no prior assumptions regarding the identity of the target concept has no rational basis for classifying any unseen instances."

Why is it that a machine learning algorithm needs a bias? Can someone please help me understand this, with examples perhaps?

Inductive bias means all assumptions your learning approach assumes for generalising to unseen data. What we do in machine learning is induction which means we don't have rules. By seeing data, we make rules. For example, in linear regression you may consider that the output is linear with respect to inputs, or it is polynomial. Another example can be this that in women gesture recognition, you don't care about moustaches because you know they do not have that. You can also take a look at here.


Restricting a weight vector (optimization parameter) to be in a certain domain using python ML library linear regression model Sorry if the title is a bit long, but basically I'm trying to predict values $$ \hat{y}_i \in [-1,1]$$ using a simple model i.e. something like OLS or ridge regression, I'd like to know if anyone knows of a library that has models with features that all you restrict your weight vector to be in a certain domain.

There are many, I'd recommend you the following: cvxpy (in turn interfaces some other solver like scs or cvxopt), cvxopt, casadi, even tensorflow (some optimizes have options for bounds).


Tensorflow model works for classification but not for regression (all predictions equal the output layer bias) I'm trying to build a model for FX prediction.  It's giving some promising results for classifying each period as buy/sell/neutral.  When used as a classifier, actual returns are converted to 0, 1, or 2 to represent the possible actions (sell=0, neutral=1, buy=2) based on whether the future price change exceeds a threshold to the high or low side.

However, when trying to predict the future change as a linear regression problem, it's making nonsensical predictions.  Every time I train the model, it quickly converges to a different fixed value prediction for every period.

Here's a boiled down version of the model build/compilation:

def build(self, is_evolution=False):
    self.model = models.Sequential()
    self.model.add(layers.Dense(self.num_layers, activation="relu", input_shape=(self.x_train.shape[1],)))
    self.model.add(layers.Dense(self.num_layers, activation="relu"))
    if self.is_classification:
        class_counts = np.array(list(map(lambda x: self.y_train[self.y_train == x].shape[0], self.classes)))
        total_count = self.y_train.shape[0]
        bias_initializer = tf.constant_initializer(tuple(class_counts / total_count))
        self.model.add(layers.Dense(len(self.classes), activation="sigmoid", bias_initializer=bias_initializer))
    else:
        self.model.add(layers.Dense(1))

    loss = "sparse_categorical_crossentropy" if self.is_classification else tf.keras.losses.MeanSquaredError()
    self.model.compile(optimizer=tf.keras.optimizers.Adam(), loss=loss, metrics=["accuracy"])

Here's an example of the predictions when is_classification is set to False.  Not only is this nonsensical because all of the predictions are the same, but the numbers in this case are also very big (i.e., the actual mean is very close to zero), although the prediction numbers can vary quite a bit with different trainings.

[[0.00081437]
 [0.00081437]
 [0.00081437]
 ...
 [0.00081437]
 [0.00081437]
 [0.00081437]]

Based on suggestions here: Tensorflow regression model giving same prediction every time, I've tried changing the learning rate to no avail.  I'm also making sure to scale the train and test set, although I'm not sure that should matter for linear regression.

At first, I thought this was just because the dataset was very noisy, but thinking about it more, I think a model that can classify a buy or sell should also be able to make regression predictions when the output layer is modified as it is above.

Any thoughts are appreciated.

Update: I was inspecting individual weights and I see that the linear regression predictions are equal to the output layer bias, however, the other weights appear to be "normal" (i.e., various positive and negative weights).

Surprisingly, changing the optimizer from adam to tf.keras.optimizers.SGD(lr=0.01, momentum=0.9) eliminated the identical predictions, but it looks like more tinkering is required to get good results like with the classification model.

The only question is why does this change make that much of a difference?  I'd be curious to know the answer if anyone has one.


What is the best way Reinforcement learning, RNN or others to predict the best action we have to take to maximize sales? I have a dataset composed of few features :

customerId, actionDay1, SalesDay1, actionDay20, SalesDay20, actionDay30, SalesDay30

action can be :

call
email
face 2 face
nothing

sales : amount of sales in $

my goal here is to predict the best action we have to take in any day (1, 20 or 30) to sale more.

some customer prefers email, some prefer to be called every time, some others prefer to not be contacted to buy.

the pattern is different for each customer. my first thought was to apply reinforcement learning for this problem, but I found some difficulty to do the right environment, but also the reward function. a reinforcement learning would work without data and the model will be different for each customer. I would like to have a model that can approximate the behavior of all customers using the dataset I have.

does anyone have an idea on how I can approach this problem using reinforcement learning or something different like Recurrent Neural Net?

I don't know if reinforcement learning is right for this problem.  A reinforcement learning model needs a simulated "game" environment where it can interact and select actions initially at random.  But real customer reactions are hard to simulate, and you don't want the model randomly sending out spam e-mails before it's trained.

There are other ways to train a model to recommend an action based on existing data.  If you want to keep it simple, you can perform a regression to estimate each action's impact on likelihood of a sale.  If you have more data and want to get more complex, perhaps look into recommender systems (Netflix, Amazon, etc.)


How to choose a classifier after cross-validation? When we do k-fold cross validation, should we just use the classifier that has the highest test accuracy? What is generally the best approach in getting a classifier from cross validation?

You do cross-validation when you want to do any of these two things:

Model Selection
Error Estimation of a Model

Model selection can come in different scenarios:

Selecting one algorithm vs others for a particular problem/dataset
Selecting hyper-parameters of a particular algorithm for a particular problem/dataset

(please notice that if you are both selecting an algorithm - better to call it model - and also doing hyper-parameters search, you need to do Nested Cross Validation . Is Nested-CV really necessary?)

Cross-validation ensures up to some degree that the error estimate is the closest possible as generalization error for that model (although this is very hard to approximate). When observing the average error among folds you can have a good projection of the expected error for a model built on the full dataset. Also is importance to observe the variance of the prediction, this is, how much the error varies from fold to fold. If the variation is too high (considerably different values) then the model will tend to be unstable. Bootstrapping is the other method providing good approximation in this sense. I suggest to read carefully the section 7 on "Elements of Statistical Learning" Book, freely available at: ELS-Standford

As it has been mentioned before you must not take the built model in none of the folds. Instead, you have to rebuild the model with the full dataset (the one that was split into folds). If you have a separated test set, you can use it to try this final model, obtaining a similar (and must surely higher) error than the one obtained by CV. You should, however, rely on the estimated error given by the CV procedure.

After performing CV with different models (algorithm combination, etc) chose the one that performed better regarding error and its variance among folds. You will need to rebuild the model with the whole dataset. Here comes a common confusion in terms: we commongly refer to model selection, thinking that the model is the ready-to-predict model built on data, but in this case it refers to the combination of algorithm+preprocesing procedures you apply. So, to obtain the actual model you need for making predictions/classification you need to build it using the winner combination on the whole dataset.

Last thing to note is that if you are applying any kind of preprocessing the uses the class information (feature selection, LDA dimensionality reduction, etc) this must be performed in every fold, and not previously on data. This is a critical aspect. Should do the same thing if you are applying preprocessing methods that involve direct information of data (PCA, normalization, standardization, etc). You can, however, apply preprocessing that is not depend from data (deleting a variable following expert opinion, but this is kinda obvious). This video can help you in that direction: CV the right and the wrong way

Here, a final nice explanation regarding the subject: CV and model selection


Time series forecast using SVM? I have a pandas data frame like this:

(index) 0           sie

0       1997-01-01  11.2
1        1997-01-03  12.3
2        1997-01-04  11.5
...
12454    2017-02-01  13.2

I would like to use SVM to predict the future values of the sie. How can I implement python code to predict these values?

I am doing something like this:

model = svm.SVR().fit(df[0],df['sie'])

But it is giving me this error:

ValueError: Found input variables with inconsistent numbers of samples: [1, 12455]

Although both df[0]anddf['sie'] have same shape of (12455,)

Note: I don't have continuous data (some dates, in between, are missing), also values in 0 are datetime.date() objects.

Here, a very good article: http://machinelearningmastery.com/time-series-forecasting-supervised-learning/

In a few words, define a window of size n and that is the size of your feature vector. Reshape the dataset and play.


Why would you split your train data to compute a value on half of the data to then fill the Nan values on the other half? I was checking a kernel written in python from the Bosch kaggle competition (kaggle link to python kernel) and I came across with a weird (at least to me) way to fill Nan values.

The train data is split into two halves and then some kind of average is computed in one half by using the non-Nan values of a field along with the target value and then fill Nan values on the other half with these computed values.

Then, when training the model after filling Nan values, the model is only trained on the half data where the Nan values have been replaced.

The question is, why would you split the data into two halves to compute on one and then fill the other?
Are you introducing some kind of leakage when mean values are related to the target value and that's the reason why just the half part with filled values (the one in which you haven't computed anything, just filled Nans) is used to train?
Is this procedure prone to overfitting if you perform this operation on all the train data?

Thanks in advanced.

You basically answered your own question:

  Are you introducing some kind of leakage when mean values are related to the target value and that's the reason why just the half part with filled values (the one in which you haven't computed anything, just filled Nans) is used to train? Is this procedure prone to overfitting if you perform this operation on all the train data?

Yes. You have already used the target values and the first half of the training set. In other words, the filled-out NaN values have information about the target values in them, so if you were to perform any cross-validation for performance estimation, you would end up with an over-estimation of your performance metric. It's best practice to leave that used part out of any further training steps to avoid information leakage.


Looking for a causality to effect dataset I am looking for a causality dataset that would look like this:

animal + car -> accident
speeding -> traffic ticket
tomatoes + lettuce -> salad
virus -> illness
tooth decay -> dentist

Does this already exists?

I doubt anything so general exists. The closest I can think of would be to exploit resources which exist in the medical domain, in particular https://semrep.nlm.nih.gov/

Also try asking on https://opendata.stackexchange.com/


Keras custom loss function as True Negatives by (True Negatives plus False Positives) I have some model for which I can construct the confusion matrix, although I need a custom loss function which will be as:

true negatives (TN): We predicted no, and it is no.

false positives (FP): We predicted yes, but it is no.

My loss function should be like:

           TN
  1 -  _________
        TN + FP

How do I implement the same in Keras:
There is an explanation for the custom loss function here: StackOverFlow

Although, I am not sure how the differentiation part for the same would be like.

Image source: http://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/

The Above Solution does not work when you use it during training it throws an error because of this line:

TN = np.logical_and(K.eval(y_true) == 0, K.eval(y_pred) == 0)

Maybe this could be a better solution to this problem you can calculate specificity after every epoch and also plot!

def specificity(y_pred, y_true):
    """
    param:
    y_pred - Predicted labels
    y_true - True labels
    Returns:
    Specificity score
    """
    neg_y_true = 1 - y_true
    neg_y_pred = 1 - y_pred
    fp = K.sum(neg_y_true * y_pred)
    tn = K.sum(neg_y_true * neg_y_pred)
    specificity = tn / (tn + fp + K.epsilon())
    return specificity


Is it good to remove outliers from the dataset? Suppose you have some training dataset that you want to use to train some ML models, where targets are comprised between let's say 1 and 100. However, from the 4000 samples, there are few of them (less than 10) which have values out of the previous range, much higher than 100, let's say 300. Is it reasonable to ignore these samples and remove them from the data set, or should they be kept ? I saw people react differently, some of them say that they may hurt the model, while others say no as these samples give additional information to the model.

It mostly depends on what you are trying to achieve with your model.
Sometimes the information carried by outliers is indeed negligible if not of interest (say, for example, that those high values are caused my data collection/input errors), and may affect your model performance. In some other cases though, outliers carry a lot meaning and you might want your model to be aware of their existence/possibility. In other scenarios, the outliers are what you actually care about (see Anomaly/Novelty detection e.g.).

Long story short, if these outliers are really such (i.e. they appear with a very low frequency and very likely are bad/random/corrupted measurements) and they do not correspond to potential events/failures that your model should be aware of, you can safely remove them. In all other cases you should evaluate case by case what those outliers represent.


k-means and perceptron generalization ability I've implemented simple Widrow-Hopf perceptron and k-means clustering and compare results against MNIST data set. I didn't expect great results because of linear nature of these algorithms.  WH perceptron ended up with ~70% accuracy and k-means with ~50% (I have unit tests and compare with pure random chance 1/10 I suppose they do work somehow correct).

Also I gave "a hint" to k-means by set up initial centroids around different digits, so it can converge faster.

I suppose that k-means showed this result because digits are pretty much similar (6 is like 8 and 8 is like 9). Then I assumed that generalizations ability of WH perceptron is better than k-means. Did anyone see any article/book about this topic? I want to understand results from a rigor mathematical perspective.

Comparing K-means and perceptrons doesn't make sense, they are different types of algorithms.

K-means doesn't have a generalization ability at all, since it is an unsupervised algorithm, and generalization is a property of supervised learning algorithms.

If you are changing it to behave like a supervised algorithm, than it will have poor generalization ability compared to the perceptron, because K-means will have a tendency to overfit the data since it is forced to assign a cluster to each of its data points.

If you want to compare perceptron's performance to other algorithms, you should compare it to k-NN (K nearest neighbors), Random Forests, or Support Vector Machines.


Checking regression coefficients' stability? I have a model with 13 independent variables (2 of them are categorical variables) and 678 observations. All of the variables are significant.I'm planning to check the coefficients stability. My approach is to start from a model with one variable and find that coefficients. Then I’ll add another variable; however, I'm keeping the previous variable also. I’ll continue until that all variables are in the final model. I'm using the following R code:

add.APV<-lm(HPV~log(APV),data=Mydata)
add1(add.APV,scope =~ AWD + Variety.bodyandchassis + Flexibility + Model.types + Year + log(CAC)+ Outsourcing + Platformstrategy + Hourly.total + Newownership + Vehicle.Launch + Segment.MD., test="F", k=2, trace=TRUE)

And I'm getting the following error:

Error in factor.scope(attr(terms1, "factors"), list(add = attr(terms2,  :
  upper scope has term ‘NA’ not included in model

I was wondering what the problem is. Any feedback would be appreciated.

I had the same problem, and my data is all numeric and free from NAs.  The fix is really trivial:  instead of scope = ~ var1 + var2 + ..., you should use
scope = ~ . + var1 + var2 + ....  .  Seems like this ought to be covered in the add1 examples on its help page.


How to use TFIDF vectors with multinomial naive bayes? Say we have used the TFIDF transform to encode documents into continuous-valued features.

How would we now use this as input to a Naive Bayes classifier?

Bernoulli naive-bayes is out, because our features aren't binary anymore.
Seems like we can't use Multinomial naive-bayes either, because the values are continuous rather than categorical.

As an alternative, would it be appropriate to use gaussian naive bayes instead? Are TFIDF vectors likely to hold up well under the gaussian-distribution assumption?

The sci-kit learn documentation for MultionomialNB suggests the following:

  The multinomial Naive Bayes classifier is suitable for classification
  with discrete features (e.g., word counts for text classification).
  The multinomial distribution normally requires integer feature counts.
  However, in practice, fractional counts such as tf-idf may also work.

Isn't it fundamentally impossible to use fractional values for MultinomialNB?
As I understand it, the likelihood function itself assumes that we are dealing with discrete-counts:

(From Wikipedia):

${\displaystyle p(\mathbf {x} \mid C_{k})={\frac {(\sum _{i}x_{i})!}{\prod _{i}x_{i}!}}\prod _{i}{p_{ki}}^{x_{i}}}$

How would TFIDF values even work with this formula, since the $x_i$ values are all required to be discrete counts?

When predicting the most likely class, the factorial terms are present in the probability calculation for every class, so they can essentially be ignored in the calculation. This leaves word probabilities which are raised to fractional powers, which are easily calculated. See this paper for a more detailed explanation: http://www.cs.waikato.ac.nz/ml/publications/2004/kibriya_et_al_cr.pdf.


Python Pandas agg error I am trying to generate descriptive statistics using agg function in Pandas.  I am having trouble with one line with a lambda function.  They work when I run them as separate lines of code, but when I put them as a single line I get errors.

Any guidance is much appreciated.

The following two lines of codes work when I run them individually.

First line of code:

bh_df.groupby('CAT.MEDV').agg(
  avg_Nox=('NOX', 'mean'))

Second line with lambda function.

bh_df.groupby('CAT.MEDV').agg(
   rng=("NOX", lambda x: (max(x) - min(x))))

However, when I combine them into a single line of code as:

bh_df.groupby('CAT.MEDV').agg(
   avg_Nox=('NOX', 'mean'),
   rng=("NOX", lambda x: (max(x) - min(x))))

I get a whole bunch of errors:

  File "", line 4, in

  rng=("NOX", lambda x: (max(x) - min(x))))

  File "C:\Users\pdile\Anaconda3\lib\site-packages\pandas\core\groupby\generic.py", line 1455, in aggregate
      return super().aggregate(arg, *args, **kwargs)

  File "C:\Users\pdile\Anaconda3\lib\site-packages\pandas\core\groupby\generic.py", line 264, in aggregate
      result = result[order]

  File "C:\Users\pdile\Anaconda3\lib\site-packages\pandas\core\frame.py", line 2986, in getitem
      indexer = self.loc._convert_to_indexer(key, axis=1, raise_missing=True)

  File "C:\Users\pdile\Anaconda3\lib\site-packages\pandas\core\indexing.py", line 1285, in _convert_to_indexer
      return self._get_listlike_indexer(obj, axis, **kwargs)[1]

  File "C:\Users\pdile\Anaconda3\lib\site-packages\pandas\core\indexing.py", line 1092, in _get_listlike_indexer
      keyarr, indexer, o._get_axis_number(axis), raise_missing=raise_missing

  File "C:\Users\pdile\Anaconda3\lib\site-packages\pandas\core\indexing.py", line 1185, in _validate_read_indexer

Final error:

  raise KeyError("{} not in index".format(not_found))

  KeyError: "[('NOX', '')] not in index"

Because there is no more column "NOX"

you changed it in the previous mean aggregation. Try "avg_Nox" or whatever column  name you get after applying your first aggregation function

bh_df.groupby('CAT.MEDV').agg(
  avg_Nox=('NOX', 'mean'))


Risk prediction vs classification model I am working on a binary classification model.  Currently, when I use scikit logistic regression, it outputs binary values like 0s and 1s. However, I understand, from online reading, that it outputs probability, and based on threshold of 0.5, converts them into two classes.

1) Does building a risk prediction model mean just stopping our project as soon as we get the probability output and not apply this threshold? Is that what is called as Risk prediction model?  If yes,  how do I do that using scikit logistic regression?

2) Does scikit logistic allow us modify the threshold?

3) Can all classification algorithms like SVM, RF, XGBOOST, etc. be used to build a risk prediction model without going for the threshold cutoff?

I will try to answer your question as shortly as possible.

Yes, if you define probability as a risk, then the probabilities are risk scores. But, there's a catch in these scenarios, you will have to include the prevalence of a class to calibrate them. If Person A has a risk score of 0.9 but you have observed that the positive class is only 20% of the data, then the actual risk is much lower than the probability it self. You can use clf.predict_proba() function to get these probabilities.
Once you have these probabilities, you should be able to set your custom threshold.
Yes, most of these models in different libraries have predict_proba() functions.

Let me know if you have more questions.


Confusion matrix to check results I am a new user in StackExchange and a new learner of Data Science. I am working on better understanding how to estimate the results collected, specifically fake users extracted from a dataset running some analysis.
Using a specific algorithm, I found some users
User_Alg

user1
user2
user3
user28
user76
user67

and I would like to estimate accuracy of my algorithm comparing with the dataset which contains all the fake users manually labelled:
User_Dat
user1
user5
user28
user76
user67
user2
user29

As you can see, there are some users, in my extracted list (User_Alg), who are missing, i.e. not included in the list manually labelled (all the fake users in the dataset; User_Dat).
I have thought to use a confusion matrix to check the accuracy, but I would like to know from people with more experience in statistics and machine learning than me, if such method can be ok and how it looks like, or if you recommend another approach.
Thanks for your attention and your time.

A confusion matrix is indeed a very useful way to analyze the results of your experiment. It provides the exact number (or percentage) of instances with true class X predicted as class Y for all the possible classes. As such it gives a detailed picture of what the system classifies correctly or not.
But a confusion matrix is a bit too detailed if one wants to summarize the performance of the classifier as just one single value. This is useful especially when one wants to compare two different classifiers, since there's no general way to compare two confusion matrices. That's why people often use evaluation measures: for binary classification, the most common ones are:

Accuracy, which is simply the number of correct predictions divided by the total number of instances.
F-score, which itself is the harmonic mean of precision and recall. Quite ironically, F-score gives a more accurate picture of performance than accuracy because it takes into account the different types of possible errors.


Normalization in production I am currently writing a machine learning pipeline for my time series application. At the end of each month, I get the data gathered, normalize it ([0, 1]), retrain the ML model with the new observation only and predict future values.

Question

Should I be reading the entire dataset each time I get a new Observation, normalize the entire dataset, create the ML model, then predict?

How I got stuck:

Let's say I have 1 feature and at t-1 all of the values have min/max = [0, 1000]
At t, a new observation comes in with value = 1001
How should I normalize the new value given that the ML model has been trained with different min/max?

Thank you

Really depends

Why? updating everything in production (pre-processing, fitting etc) can get extremely expensive. If you have some complex architecture it is not worth it.

Alternatives

Approximate covariate shift if you know distribution of your future data you can adjust all your, for example normalisation parameters, in advance.
Save your you future data every time you make prediction, it could be cheaper to quickly save your data in DB and depending on your system do updates weekly,monthly


Definition of the Q* function in reinforcement learning I'm making my way through Sutton's Introduction to Reinforcement Learning. He gives the definition of the $q_*$ function as follows
$$
q_*(a) = \mathbf{E}[R_t | A_t = a]
$$
where $A_t$ is the action taken at time t and $R_t$ is the reward associated with taking $A_t$. From my understanding, $q_*$ represents the true value of taking action $a$, which is the mean reward when $a$ is selected.
But I'm confused about why $t$ is included in this equation at all. Should $q_*(a)$ really be $q_*(a, t)$? Or are we to understand $q_*$ as taking the expected reward across all $t$?

The reward of action $a$  is defined as a stationary probability distribution with mean $q_*(a)$. This is independent of time $t$.
However the estimate of $q_*(a)$ at time $t$, denoted by $Q_t(a)$, is dependent on time $t$

Or are we to understand q∗ as taking the expected reward across all t?

The expectation is not over time, but over a probability distribution with mean $q_*(a)$.
For eg., in the 10-armed bandit problem, the reward for each of the 10 actions comes from a Normal distribution with mean $q_*(a), a= 1,...,10$ and variance 1.


How are treemaps hierarchical? According to various sources on the internet, treemaps are a way to visualize hierarchical, tree-like data structures, for example the tree generated by hierarchical clustering.
I looked at a lot of treemaps and I just cannot seem to understand where/how the sense of hierarchy/nestedness is conveyed. It just seems to be tiles arranged next to each other. How do I know which tile is a parent/child of which other tile(s)?

Treemaps are not very intuitive when trying to understand hierarchy levels. Sunburn chats are better.

First hierarchy level is either shown by borders or different colors.
Subsequent hierarchy level is represented by smaller rectangle which is part of bigger rectangle.

Refer this article


Multiclass Classification that includes a Geospatial Element I am attempting to train a classifier to predict different prices for an item in different suburbs. I have several features, two of which are a latitude and longitude for the centroid of the suburb.

I am attempting to train the model to classify the price of an item in a bin of $10 size. The geospatial element will definitely affect the price of the item, however the training data I have will have gaps in it (i.e. I don't have prices for all suburbs).

What is the best way to engineer a feature that will include this geospatial information and be able to fill in the gaps in the training/test data?

So far I have tried creating new features for the bearing and distance from the capital city which seemed to work okay, as well as binning the latitude and longitude which performs worse than the bearing/distance. I did consider using a geohash, however I think that this will be too complex a feature for a classifier to understand.

If this is a U.S. model you can try County (FIPS) codes, which are coarser-grained areas than ZIP codes (there can be many ZIP codes in a county).  If bearing and distance worked for you, this should also work.  Your classifier would need to handle categorical data.  In R, this is pretty easy to do if you already have a ZIP code:

library(noncensus)
data(zip_codes)
my_dataframe['fips']<-zip_codes$fips[match(my_dataframe$zip, zip_codes$zip)]

This assumes you already have a dataframe called my_dataframe with ZIP codes as a column named 'zip'.

As a bonus you can use the choroplethr package to create a nice visualization of some scalar value, for example, your item price:

library(choroplethr)
df_to_plot<-data.frame()
df_to_plot['region']<-my_dataframe$fips
df_to_plot['value']<-my_dataframe$price

county_choropleth(df_to_plot, title='price by location')


Assigning new word by creating a new column when finding match I have a survey df and I want to assign a new value of either existing customer
    or new customer based on their answers. if someone has 3 answers but one of
    them matches "coca cola" I want to give them the value of existing customer
     here is the data frame

ID       Question                                                Answer
101005   what brands did you purchase the past 5 months   Coca-Cola or Pepsi or vitamin water
026458   what brands did you purchase the past 5 months           None
045987   what brands  did you purchase the past 5 months        Coca-Cola

Here is the table I would like to have

ID        Question                                          Answer                      Buyer_Type
101005   what brands did you purchase the past 5 months  Coca-Cola,Pepsi,fanta          Existing Users
026458   what brands did you purchase the past 5 months  None                           New Buyer
045987   what brands did you purchase the past 5 months  Coca-Cola                      Existing Users

Here is what I tried

deux['Buyer_Type'] = deux['answer'].apply(lambda x: 'existing buyer' if x == 'Coca-Cola' else 'new buyer')

For some reasons it is not recognizing 101005 as an existing user.

You are not getting the results you're expecting because you're explicitly comparing the value in the Answer column to 'Coca-Cola' (i.e x == 'Coca-Cola') but the value in that column is (I'm assuming) a list or a string. In either case comparing it to 'Coca-Cola' would result in False.
What you need to do is change your condition to check in 'Coca-Cola' appears anywhere in the list or string.
Here's one way of achieving this (I'm assuming the Answer column contains lists):

import pandas as pd

data = {
          "ID": ["101005", "026458", "045987"],
          "Question": ["what brands did you purchase the past 5 months",
                       "what brands did you purchase the past 5 months",
                       "what brands did you purchase the past 5 months "],
          "Answer": [["Coca-Cola" ,"Pepsi" ,"fanta"], [None], ["Coca-Cola"]]
}
deux = pd.DataFrame(data)
deux['Buyer_Type'] = deux['Answer'].apply(lambda x: 'existing buyer' if 'Coca-Cola' in x else 'new buyer')


data pre-processing before image classification I'm working on a machine learning project, Images classification (shape: 100 x 100)-> (vector of 10000), I did some pre-processing before applying decision trees algorithm , I got an accuracy of 55 % I tried to change parameters but the accuracy did not increase, Can someone suggest something to do in pre-processing exept what I did :

Denoising Images: Images are extremely noised, so I only kept relevant information in each image.(it works well)
Centring Images: I translated the relevant values for all images to the top left to keep relevent information in same part of my vectors
Resizing Images: I resized each image to (40,40) -> (1600 vector) to make the inputs smaller

Thank you

Why are you convinced that the issue is in pre-processing? It could be in many other spots or a combination of steps in your process.

If I were you, I would revisit WHY you're using a decision tree. There are many other approaches to image processing and neural networks are far superior - in most cases - to decision trees.


Interpreting Learning Curves I'm curious on how to interpret the following learning curves, which can be considered an actual good learning curve of the three learning curves. I'm not sure when to consider if there is a high variance or bias, likewise the presence of underfitting or overfitting respectively. Thank you in advanced.
1.

2.

3.

Bias and variance and their effect on overfitting and underfitting summarized in one illustration

Therefore, I think you a fit model, with reasonable variance and bias.


Is an Arma model equivalent to a 1-layer Recurrent Neural Network without activation function? Given a time series $f(t)$ to forecast, let us consider an Arma model of the form:
$$
f(t) = c + \sum_{i=1}^p a_i f(t-i) + e(t) + \sum_{j=1}^q b_j e(t-j)
$$

where $e(t)$ are the forecast errors.

On the train set, if $f(t)$ is the ground truth, then we define its estimate obtained with this model as $\widetilde{f}(t) = f(t) + e(t)$.

Let $m = min(p,q)$, we can rewrite the first equation as:
$$
\widetilde{f}(t) = c + \sum_{i=1}^m (a_i + b_i) f(t-i) + \sum_{i=m+1}^p a_i f(t-i) - \sum_{j=1}^q b_j \widetilde{f}(t-j)
$$
Then after reparametrization can be rewritten as:
$$
\widetilde{f}(t) = c + \sum_{i=1}^k c_i f(t-i) - \sum_{j=1}^q b_j \widetilde{f}(t-j)
$$
Which is the equation of a 1-layer recurrent neural network (RNN) without activation function.

So, are Arma models a subset of RNNs or is there a flaw in this reasoning ?

It's correct. The reason it sounds so weird is that a 1-layer-NN without activation function is simply a linear map, so it's equivalent to any linear model, the only difference being the inputs having some interpretation.

This even holds true for any NN, no matter the number of layers, without activation functions. The reason: A k-layer-NN is just k matrix multiplications (=linear maps) with activation functions in between. If you remove the latter you are left with a long chain of matrices that you can simply multiply to get one matrix that now defines your linear map.

Many deep learning researches concluded that giving the networks the possibility to create non-linear maps is what makes deep learning so powerful, even when using seemingly unsophisticated activation functions such as the ReLU.


Validation loss and validation accuracy stay the same in NN model I am trying to train a keras NN regression model for music emotion prediction from audio features. (I am a beginner in NN and I am doing this as study project.)
I have 193 features for training/prediction and it should predict valence and arousal values.
I have prepared a NN model with 5 layers:
model = Sequential()
model.add(Dense(100, activation='elu', input_dim=193))
model.add(Dense(200, activation='elu'))
model.add(Dense(200, activation='elu'))
model.add(Dense(100, activation='elu'))
model.add(Dense(  2, activation='elu'))

And this is my loss and optimizer metrics:
model.compile( loss = "mean_squared_error", optimizer = 'RMSprop', metrics=['accuracy'] )

When I try to train this model, I get this graph for loss and validation:
So the model is trained and reaches accuracy of >0.9 on training data, but on test data accuracy wont fall, but it stays on ~0.5.

I don't know how to interpret this graph. I don't think this is overfitting, because validation accuracy wont fall, but it stays the same.
How can I try fix this?
Update:
I tried to add dropout and regularization and it worked in a way that now I clearly see that I have a problem with over-fitting. But now I am stuck again.
I can not make my model to decrease validation loss. It always stops at about 0.3 validation loss.
I tried changing my model architecture, data preprocessing, optimizer function, and nothing helped.

You say it's a regression task, predicting valence and arousal values, although you use accuracy as a performance metric. This does not make much sense, so your accuracy graph doesn't really say much. MSE is a valid performance metric for regression tasks in general, so your loss graph is more descriptive of what is going on. The loss-graph most definitely displays the characteristic of over-fitting, so I would recommend you to add regularization to your model.
This can be done by for example incorporating dropout and L1/L2-regularisation.


Should use sklearn or tensorflow for neural networks? I have just started learning Neural Networks for deep learning from cs231. I am trying to implement Neural Network in Python. I am looking at using Tensorflow or scikit-learn.  What are some pros and cons of these libraries for this application?

Sklearn doesn't have much support for Deep Neural Networks. Among the two, since you are interested in deep learning, pick tensorflow.

However, I would suggest going with keras, which uses tensorflow as a backend, but offers an easier interface.


How is BCELoss counted in PyTorch? [different result comparing to mathematical implementation] I am trying to understand how Binary Cross Entropy is counted in PyTorch.
I've tried the same code from the PyTorch documentation here, but I get a different result comparing to mathematical implementation of this function.
Code (I made little changes comparing to documentation):
import torch
import torch.nn as nn
m = nn.Sigmoid()
loss = nn.BCELoss()
input = torch.randn(1, requires_grad=True)
print(input)
print(m(input))
target = torch.empty(1).random_(2)
print(target)
output = loss(m(input), target)
print(output)
output.backward()
print(output)

Output for the code above is:

The mathematical formula for BCELoss is:
output = -[target*log(our_value) + (1-target)*log(1-our_value)]

From the mathematical formula above I should get 'output'=0.3215 for 'our_value'=0.4770 and 'target'=1. But PyTorch shows that the 'output'=0.7403.
I've also found a C code here of this function but the formula is almost the same (only very small epsila should be added and it gives no difference in output). So, it's still unclear for me.
How does it come to? Any hints are welcome.

Take a look at the documentation for BCELoss

Our solution is that BCELoss clamps its log function outputs to be
greater than or equal to -100. This way, we can always have a finite
loss value and a linear backward method.

If you amend the definition to include that it should yield the correct results:
def binary_cross_entropy(pred, y):
    log_pred = torch.clamp(torch.log(pred), -100, 100)
    log_pred_rev = torch.clamp(torch.log(1-pred), -100, 100)
    return -(log_pred*y + (1-y)*log_pred_rev).sum()

a = torch.linspace(0, 1, 10)

w = np.array([F.binary_cross_entropy(x,torch.tensor(0.2)).item() for x in a])
q = np.array([binary_cross_entropy(x,torch.tensor(0.2)).item() for x in a])

print(w)
print(q)

Gives
[20.          0.53367138  0.501867    0.54409462  0.63241535  0.76630157 0.95998287  1.2535249   1.78133631 80.        ]
[20.          0.53367138  0.501867    0.54409462  0.63241535  0.76630157 0.95998287  1.2535249   1.78133631 80.        ]


Neural network training with known erroneous data I am trying to make a neural network to predict some values but I know my training data contains lots of "bad" expected outputs. That is I know some of the data would train it in the wrong direction.

Would it make sense to train the network once, then remove data that the network produces a big error for (supposing that that's the "bad" data) and then retrain it with the rest? Is there some other better way to approach this?

I'm sorry I'm only starting out dealing with NNs and I don't know much of the related jargon to make the question clearer.

If you want to identify and remove problematic data, then this is much better done before training.

You might have some luck with your approach, but you have no guarantees that the neural network will help you isolate the problem entries based on error values, if it has been trained on the "good" and "bad" entries at the same time. It depends on the nature of the problem data. You take a significant risk that the model will fit to the problem data well enough to cause you to reject good data.

You should instead try to think of a way to identify the bad data more directly, and prior to training for the predictive task.

Here's one idea: Are you able to identify enough good and bad data manually - enough to train and test a classifier? Then label some of the data "good" or "bad", train a classifier and test it - the test should be on some held out labeled values, to help you assess accuracy. If your classifier has a good accuracy, you can have some confidence to use it to filter the remainder of your data, and only use the "good" classified data for your original training goal.


Word embedding of a new word which was not in training Let's say I trained a Skip-Gram model (Word2Vec) for my vocabulary of size 10,000.
The representation allows me to reduce the dimension from 10,000 (one-hot-encoding) to 100 (size of hidden layer of the neural network).

Now suppose I have a word in my test set which was not in my training vocabulary. What is a reasonable representation of the word in the space of dimension 100?
For me, it seems that I cannot use the neural network I trained to come up with the word embeddings.

Generating word embeddings for "OOV"(out of vocabulary) words is one of the major limitations of many standard embeddings like Glove and word2vec. However, fastText circumvents this problem to some extent.

Instead of the traditional approaches which have distinct vectors for each word, they take a character n-grams level representation.
For  instance,  a  word  with n=  3,  will  be  represented  by  the  character  n-grams:

<wh, whe, her, ere, re>

and  the  special  sequence:

< where >

Here,<>are part of the n-grams.

$$
s(w,c) = \sum_{g\varepsilon G_{_{w}}} z_{g}^{T} v_{c}
$$
Here, $G$ represents the size of a dictionary of n-grams, and given a word $w$, then $G_{w}\subset \left \{ 1, ..., G \right \}$ represents the set of n-grams appearing in
$w$. They associate a vector representation $z_{g}$ to each
n-gram $g$ and represent a word by the sum of the
vector representations of its n-grams.

This helps them tackle OOV words via knowing some representations of a subword.
For an instance, an OOV word:
sechero

The 3-gram:

<se, sec, ech, che, her, ero, ro>

since, these 3-grams may have been encountered during learning, through other known words, like:

<se - section
che - cheer
ro> - hero

Hence, it can form at least some sensible embedding, instead of returning a useless <UNK>

Fastext in fact is an extension to word2vec, with majorly the feature explained above.


comparing data from first quarter in two years How can I compare data from two quarters? What software can I use to do it parallelly side by side so I can compare them. What kind of factors can I take into consideration, I'm posting a sample data set pic

I am not sure what output you are looking for.

There are many ways to do it. You can do it by creating a measure in Power BI or in PowerPivot or just using Powerquery - below is an example.

PowerPivot and PowerQuery can both put the output into a pivot table in Excel if that's something you prefer.

Pick up city    Q1       Q2
X              Amount1   Amount 2
Y              ...       ...


Linear combination of weak estimators over fuzzy classifiers? Having:

a set of soft fuzzy classifiers (classification onto overlapping sets) $C_i(x) \to [0,1]$;
a corresponding set of weak estimators $R_i(z)$ of the form $R_i(z) = \mathit{EX}(y\mid z)$.

The estimators $R_i$ are just some kind of regression, Kalman or particle filters. The classifiers $C_i$ are fixed and static. How to make a strong estimator out of a weighted combination of the form:

$$L(x, z) = \sum_{i}C_i(x)R_i(z)Q_i$$

In other words how to choose the weights $Q_i$? Is there some kind of online approach to this problem?

Here is brief description of a practical application. When an event $E$ is registered, multiple measurements are made. Based on these measurements, the classifiers $C_i$ make a soft assignment of the event to multiple overlapping categories. What we get is fit ratios for the soft clusters.

Now there is some chance that event $E$ may trigger a subsequent event $D$, depending on another variable $z$ -- independent from the event $E$. We know that all the soft cluster "memberships" may influence the probability of event $D$ being triggered.

We want to estimate the probability that $E$ triggers $D$, given the $C_i$ fitness ratios and value of $z$.

You might find a solution for this by checking out Viola & Jones face detection algorithm (and object detection in general) http://www.cs.ubc.ca/~lowe/425/slides/13-ViolaJones.pdf. Particularly the AdaBoost algorithm for building a strong classifier from weak classifiers. https://www.cs.princeton.edu/~schapire/papers/explaining-adaboost.pdf

In this algorithm this is used particularly for feature selection, where it is wanted to select just the features that grouped together classify better, discarding the other as noisy features.

The approach to obtain this strong classifier is having a set of examples X, a set of weights W and a set of expected results (classifications) Y. For a two-class example (e.g. face o no face image) the first weak classifier is selected and then the classification error is found between this classifier and the expected output. For the samples that were misclassified their weights are incremented, so the next weak classifier to find is more biased to neglect this wrong classification.

The algorithm (AdaBoost) converges when the sign of the sum of classifications for each sample outputs the correct classification.

For example:

Y = +1 -1 +1 +1

WC1=+1 +1 +1 +1
WC2=-1 -1 -1 -1
WC3=+1 -1 +1 +1

So the strong classifier is WC1+WC2+WC3:

SC=+1 -1 +1 +1 == Y

Hope this solves your question.


Pandas: Group by Single Column Entries So have this table above. I'm trying to aggregate the occupations such that the table results in:

I've tried using df.groupby(['Occupation']) but I get an error.
All I know is that my final step would be to set the index to "Occupation". But I still don't know how to group via entries in the single Occupation column here.
Also, what type of table would the final table be name/called?
I know it's not called a mutiindex table because there is only one index that the results are being grouped by.

Your issue might occur if you didn't tell that the second row in the table should also be considered a header line.
To address this, try to reset the header at the beginning.
import pandas as pd

df = pd.read_csv('YOUR/FILE/DIRECOTRY.csv', skiprows=1) // ignore the 2nd row (0-indexed)
df.rename(columns={0:'Index'}, inplace=True) // optional
df.groupby(['Occupation'])


How can I dynamically distinguish between categorical data and numerical data? I know someone who is working on a project that involves ingesting files of data without regard to the columns or data types. The task is to take a file with any number of columns and various data types and output summary statistics on the numerical data.

However, he is unsure of how to go about dynamically assigning data types for certain number-based data. For example:

    CITY
    Albuquerque
    Boston
    Chicago

This is obviously not numerical data and will be stored as text. However,

    ZIP
    80221
    60653
    25525

are not clearly marked as categorical. His software would assign the ZIP code as numerical and output summary statistics for it, which does not make sense for that sort of data.

A couple ideas we had were:

If a column is all integers, label it as categorical. This clearly wouldn't work, but it was an idea.
If a column has fewer than n unique values and is numeric, label it categorical. This might be closer, but there could still be issues with numerical data falling through.
Maintain a list of common numeric data that should actually be categorical and compare the column headers to this list for matches. For example, anything with "ZIP" in it would be categorical.

My gut tells me that there is no way to accurately assign numeric data as categorical or numerical, but was hoping for a suggestion. Any insight you have is greatly appreciated.

I'm not aware of a foolproof way to do this. Here's one idea off the top of my head:

Treat values as categorical by default.
Check for various attributes of the data that would imply it is actually continuous. Weight these attributes based on how likely they are to correlate with continuous data. Here are some possible examples:

Values are integers: +.7
Values are floats: +.8
Values are normally distributed: +.3
Values contain a relatively small number of unique values: +.3
Values aren't all the same number of characters: +.1
Values don't contain leading zeros: +.1

Treat any columns that sum to greater than 1 as being numerical. Adjust the factors and weights based on testing against different data sets to suit your needs. You could even build and train a separate machine learning algorithm just to do this.


R package clogitL1 no longer available? When I try to install clogitL1 on my work server I get

Warning message:
package ‘clogitL1’ is not available (for R version 3.5.1)

I get the same warning message when I try to install it using previous versions of R.

At cran I see

Package ‘clogitL1’ was removed from the CRAN repository.
Formerly available versions can be obtained from the archive.
Archived on 2018-06-17 as check problems were not corrected despite reminders.

Questions:
What's going on here?
It seems that this package can no longer be installed?
Is there still a way to install and use it?

I use this package extensively professionally.  We just had our servers upgraded and I would like to keep using this package...

The solution is to install an archived version of the package.


What is the num_initial_points argument for Bayesian Optimization with Keras Tuner? I've implemented the following code to run Keras-Tuner with Bayesian Optimization:
def model_builder(hp):

    NormLayer = Normalization()
    NormLayer.adapt(X_train)

    model = Sequential()

    model.add(Input(shape=X_train.shape[1:]))

    model.add(NormLayer)

    for i in range(hp.Int('conv_layers',2,4)):
        model.add(Conv1D(hp.Choice(f'kernel_{i}_nr',values=[16,32,64]), hp.Choice(f'kernel_{i}_size',values=[3,6,12]), strides=hp.Choice(f'kernel_{i}_strides',values=[1,2,3]), padding="same"))
        model.add(BatchNormalization(renorm=True))
        model.add(Activation('relu'))
        model.add(MaxPooling1D(2,strides=2, padding="valid"))

    model.add(Flatten())
    model.add(Dropout(hp.Choice('dropout_flatten',values=[0.0,0.25,0.5])))

    for i in range(hp.Int('dense_layers',1,2)):
        model.add(Dense(hp.Choice(f'dense_{i}_size',values=[500,1000])))
        model.add(Activation('relu'))
        model.add(Dropout(hp.Choice(f'dropout_{i}_others',values=[0.0,0.25,0.5])))

    model.add(Dense(hp.Choice('dense_size_last',values=[100,200])))
    model.add(Activation('relu'))

    model.add(Dense(2))
    model.add(Activation('softmax'))

    opt = Adam(learning_rate=lrn_rate_init)

    earlystop = EarlyStopping(monitor='val_loss',patience=8,restore_best_weights=True)

    model.compile(loss='categorical_crossentropy',optimizer=opt,metrics=['accuracy'])

    return model

tuner = BayesianOptimization(model_builder,objective='val_loss',num_initial_points=??,max_trials=tuner_trials,directory='BayesianOptimization/',project_name='BayesianOptimization')

What do the num_initial_points argument does exactly and what should I set it to in my case?
Reading the documentation I see the description

The number of randomly generated samples as initial training data for Bayesian optimization

but not being an expert I don't exactly get what it means and how it will impact the optimization process.

The Bayesian optimization algorithm selects points to test based on a balance between exploring uncertain regions and exploiting high-performing regions.  But before you've tested very many points, there's not much information to go on.  So, in this implementation you can specify a number of completely-at-random points to evaluate to start, and after that the actual Bayesian exploration begins.
Setting a high number of random points gives you guaranteed "exploration" points; indeed, in the documentation for the package bayesian-optimization, we find:

init_points: How many steps of random exploration you want to perform. Random exploration can help by diversifying the exploration space.

(The default is 5 in that package, and 3 times the dimension in keras-tuner.)
That said, you can also make the algorithm focus more or less on exploration/exploitation directly, using the beta parameter (kappa in bayesian-optimization, see this example notebook).


Are real analysis and measure theory essential to learn for data science? Some people say data scientists don't necessarily need to know real analysis and measure theory, but for others, real analysis and measure theory are very important for the undersdanding of kernel methods, stochastic processes etc. Should I learn real analysis and measure theory? I don't want to be too academic

Neither of real analysis or measure theory are necessary for data science. Most of the mathematics you need to know is at the undergraduate level and these courses should suffice:

Calculus
Linear Algebra
Probability and Statistics
Optimization

Most of machine learning involved in data science (that deals with application vs creating novel machine learning algorithms), requires a linear combination of concepts learnt in the above mathematics courses. For example, you can understand enough kernel methods well enough if you understood linear algebra (inner products, matrices etc.).

Also, data science has to do with practice and experimentation. A lot of key concepts become clearer only through experimentation and if you need to understand a new method, the best way is to read up specifically about it and then experiment with it. In this respect, I feel studying an entire branch of involved mathematics is not worth it if your goal is to understand a machine learning method -- you can always ask a Guru in such situations to explain what you're missing at a conceptual level :-)


R: Rates of change from an initial value I have a collection of csvs and must produce yearly rates of change per group within each csv, as well as a rate of change compared to the initial value.

I am using the function below to calculate yearly rates of change, and it works fine through my loop.

func <- function(x, n=1) {
  c(rep(NA, n), diff(x, n) / head(x, -1*n)*100)
}
 df$RateOfChange <- ave(df$Data, factor(df$Group), FUN=func)

Year    Group   Data    RateOfChange
2010    1a      3.5     NA
2011    1a      4       14.29
2012    1a      3.8     -5.00
2013    1a      4.1     7.89
2014    1a      4       -2.44
2010    1b      2       NA
2011    1b      2.3     15.00
2012    1b      2.4     4.35
2013    1b      3       25.00
2014    1b      1.8     -40.00
2009    2       10      NA
2010    2       12      20.00
2011    2       13      8.33
2012    2       15      15.38
2013    2       14      -6.67
2014    2       11      -21.43

However, I also would like to calculate the percentage change in each year compared to the initial value, like so:

Year    Group   Data    BaselineChange
2010    1a      3.5     NA
2011    1a      4       14.29
2012    1a      3.8     8.57
2013    1a      4.1     17.14
2014    1a      4       14.29
2010    1b      2       NA
2011    1b      2.3     15.00
2012    1b      2.4     20.00
2013    1b      3       50.00
2014    1b      1.8     -10.00
2009    2       10      NA
2010    2       12      20.00
2011    2       13      30.00
2012    2       15      50.00
2013    2       14      40.00
2014    2       11      10.00

Surely there is a way to do this. It'd be easy enough in Excel for a single csv, but I have several hundred, each with multiple groups. I have tried dplyr::diff and dplyr::lag, but I can't get the pipeline to abide by the group variables in the way that other questions posted on this site do.

You can reproduce both of your desired outputs using dplyr. I don't know how you were trying to use lag, but one solution is to have mutate create dummy columns that you will remove at the end.

df %>%                                            # Set up the pipe
arrange(Group, Year) %>%                         # Ensures the data are in ascending order
group_by(Group) %>%                              # Group by 'Group'
mutate(lag = lag(Data),                          # Create a dummy column 'lag'
     RateOfChange = (Data - lag) * 100 / lag,    # Calculate RateOfChange
     First = head(Data, 1),                      # Dummy column for the frist yearly value by group
     BaselineChange =                            # Calculate BaselineChange
       case_when(Data != First ~ (Data - First) * 100 / First,
                 TRUE ~ 1 * NA)) %>%             # This line provides NA value for first year
select(Year, Group, Data,                        # Select only columns you want
       RateOfChange, BaselineChange) %>%
head(5)                                          # Print the first five rows to validate

Here is the output:

# A tibble: 5 x 5
# Groups:   Group [1]
  Year Group  Data RateOfChange BaselineChange
  <int> <fct> <dbl>        <dbl>          <dbl>
1  2010 1a      3.5        NA             NA
2  2011 1a      4          14.3           14.3
3  2012 1a      3.8        -5.             8.57
4  2013 1a      4.1         7.89          17.1
5  2014 1a      4          -2.44          14.3


What is an autoencoder? I am a student and I am studying machine learning. I am focusing on deep generative models, and in particular to autoencoders and variational autoencoders (VAE).
I am trying to understand the concept, but I am having some problems.
So far, I have understood that an autoencoder takes an input, for example an image, and wants to reduce this image into a latent space, which should contain the underlying features of the dataset, with an operation of encoding, then, with an operation of decoding, it reconstrunct the image which has lost some information due to the encoding part.
After this, with a loss function, it reconstruct the latent space and so get the latent features.

about the VAE, it uses a probabilistic approch, so we have to learn the meand anv covariance of a gaussian.
So far this is what I have understood.
What I have really unclear is what are we trying to learn with autoencoders and VAE?
I have seen examples where an image goes feom a non smiling to a smiling face, or to a black and white image to a colored image.
But I don't understand the main concept, which is: what does an autoencoder do?
I add here some sources of where I studied so that who needs can see them:

https://hackernoon.com/autoencoders-deep-learning-bits-1-11731e200694
https://www.youtube.com/watch?v=yFBFl1cLYx8
https://www.youtube.com/watch?v=9zKuYvjFFS8

what does an auto-encoder do?

The simplest auto-encoder takes a high dimensional image (say, 100K pixels) down to a low-dimensional representation (say, a vector of length 10) and then uses only those 10 features to try to reconstruct the original image. You can imagine an analogy with humans: I look at someone, describe them ("tall, dark-haired, ...") then after I've forgotten what they look like, I try to sketch them using only my notes.

what are we trying to learn?

In other words, why bother? A few reasons:

dimensionality reduction: 10 features are a lot more convenient than 100K pixels. For example, I can perform classification by clustering in the 10-dimensional space (while clustering in the 100K-dimensional space would be intractable).
semantic meaning: if all goes well, each of the 10 features will have some obvious "explanation" -- e.g., tweaking one value will make the subject look older (though it's normally not so simple). As opposed to pixel values, which are impacted by translation, rotation, etc.
Exception recognition: if I train my auto-encoder on dogs, it should normally do a good job encoding and decoding pictures of dogs. But if I put a cat in, it will probably do a terrible job -- which I can tell because the output looks nothing like the input. So, looking for places where an auto-encoder does a bad job is a common way to look for anomalies.

I have seen examples where an image goes from a non smiling to a smiling face, or to a black and white image to a colored image.

There are many different types of auto-encoders. What I described above is the simplest kind. Another common type is a "denoising" auto-encoder -- instead of reconstructing the original image, the goal is to construct an image that is related to the original image, but different.
The classic example of this is denoising (hence the name): you can take a clean image, add a bunch of noise, run it through an auto-encoder, and then reward the auto-encoder for producing the clean image. So, the input (noisy image) is actually different from the desired output (clean image). The examples you give are similar.
The challenge in designing these types of auto-encoders is normally the loss -- you need some mechanism to tell the auto-encoder whether it did the right thing or not.

about the VAE, it uses a probabilistic approch, so we have to learn the mean and covariance of a gaussian.

A VAE is a third type of auto-encoder. It's a bit special because it is well-grounded mathematically; no ad-hoc metrics needed. The math is too complicated to go through here, but the key ideas are that:

We want the latent space to be continuous. Rather than assigning each class to its own corner of the latent space, we want the latent space to have a well-defined, continuous shape (i.e., a Gaussian). This is nice because it forces the latent space to be semantically meaningful.
The mapping between pictures and latent spaces should be probabilistic rather than deterministic. This is because the same subject can produce multiple images.

So, the workflow is this:

You start with your image as before
As before, your encoder determines a vector (say, length 200).
But that vector is not a latent space. Instead, you use that vector as the parameters to define a latent space. For example, maybe you choose your latent space to be a 100-dimensional Gaussian. A 100-dimensional Gaussian will require a mean and a standard deviation in each dimension -- this is what you use your length-200 vector for.
Now you have a probability distribution. You sample one point from this distribution. This is your image's representation in the latent space.
As before, your decoder will turn this vector into a new "output" (say, a vector of length 200K).
But, this "output" is not your output image. Instead, you use these 200K parameters to define a 100K-dimensional Gaussian. Then you sample one point from this distribution -- that's your output image.

Of course, there's nothing special about a Gaussian, you could just as easily use some other parametric distribution. In practice, people usually use Gaussians.
This sometimes gives better results than other auto-encoders. Further, you sometimes get interesting results when you look between the classes in your latent space. An image's distance in the latent space from the cluster center is sometimes related to uncertainty.
Moreover, there is the nice property that these high-dimensional Gaussians are probability distributions in a rigorous mathematical sense. They approximate the probability that a given image belongs to a given class. So, there is some thought that VAEs will be able to overcome the "hand waving" of deep learning and put everything back on a firm Bayesian probabilistic grounding. But of course, it is only an approximation, and the approximation involves a lot of deep neural networks, so there is still plenty of hand waving at the moment.
By the way, I like to use this question during interviews -- an astonishing number of people claim to have experience with VAEs but in fact do not realize that VAEs are different than "regular" AEs.


How to represent genre or artist name in a neural network I am writing a music recommendation system using machine learning. I'm attempting to make sense of ensemble networks to allow the system to learn from both the content-based features, as well as the global 'meta' features, such as the genre, year, artist etc.
However, I do not know how I should represent the genre and artist information. Making them integers would imply some sort of order. If Kanye West is 24, what should 25 be? 26?
Or is there something fundamentally wrong with my approach?

You can represent this kind of discrete information by means of embeddings.
An embedding is simply a table of vectors. It is defined by the number of vectors in the table and the length of each vector (i.e. dimensionality). You need to define a priori all the elements you want to support, e.g. for genres, you should list all the genres you will support, and list them. The index of each item is how you represent it.
You can have an embedding for each of the discrete features you need (genre, artist, etc).
In Keras, there is a layer for Embeddings.


Can the dependency between variables be deduced from data? And if so, how? I have a data set $X$ that consists of $m$ vectors $\vec{x}$ of $n$ real-valued components. Each vector component lies within a corresponding predefined interval of valid values, which is the same for all vectors in $X$. The assumption is that there exists a dependency graph between the components of each vector, which is also the same for all vectors; for example, the value of the component $x_k$ (maybe) depends on the values of both components $x_p$ and $x_q$ for all $\vec{x} \in X$ and $k \neq p \neq q$. However, we do not know the exact structure of this graph. In other words, we suppose that there exists a dependency between the variables of each data point, but we don't know how it is structured. So the problem is to deduce this dependency graph using only the data available in $X$.
My question is: is there a method or algorithm that allows me to solve this problem? And if so, can somebody point me to a source where I can learn more about it?

Yes, We can do that. We can measure the linear relation between two variables by correlation metric to know more see this. If you want to identify the linear relationship between more than two variables we can do it by VIF scores. If a VIF score of a variable is >5 we can say this variable can be expressed as linear combination of other variables to know more see this. If you want to implement in python consider this.


Does keras' model.fit() remember learning rate when called multiple times? Let's say I'm using the Adam optimizer, and calling fit() on my model multiple times. What parameters does the fit function remember?

From what I've observed, the loss function/metrics seem to resume from the same value (because the same saved weights are iterated on), but does the value of the learning rate reset? Does it resume from the same value it ended on the last fit() call?

Provided that you are in the same scope, will remember not only the learning rate but the current state of all tensor, hyper parameters, gradients and so on.
In fact you can call fit many times instead of setting epochs and will work mostly the same.


Comparing accuracy of models in ordinal regression / classification I am looking into creating a model to predict whether an item is "Very Good", "Good", "Bad" or "Very Bad".

After I fit the training data to the models, comparing the accuracy of the models during test stump me: should it matter if a model misclassified a G to VG while the other G to VB? What about a model that has two misclassifications of one level away versus another model with only one misclassification but three levels away (eg VG to VB)?

Any guideline on what is the common approach? Also, my thinking at the moment is that this should be a regression problem, but I'm happy to be corrected if I should approach this labeling of datasets more as a classification problem.

Your classes express a certain order. You can classify apples to, say, "green", "red" or "yellow", and then every disagreement with a reference set is equal. After all, colours express no order. So as you already suggested, I would certainly use regression. Assume that the classes could be distributed as something like this:

Very bad = 0 - 0.25
Bad = 0.25 - 0.50
Good = 0.50 - 0.75
Very good = 0.75 - 1.00

Now, the mismatch of Very good vs. Bad is at least 0.25, where is must be at least 0.50 with Very good vs. Very bad, which gives a better and more honest impression of the performance of your model.


Which output of a BiLSTM layer should be used for classification I am trying to implement a BiLSTM layer for a text classification problem and using PyTorch for this.
self.bilstm = nn.LSTM(embedding_dim, lstm_hidden_dim//2, batch_first=True, bidirectional=True)
lstm_out, (ht, ct) = self.bilstm(embeddings)

Now, in some examples (I came across in internet) people are passing ht through Linear layer and generating output. Some people are using lstm_out for the same. Now I have two questions -

For a text classification which one should be used?
If I use ht, for bidirectional=True it is of the shape (2, m, n) - but we need (1, m, n) - what is the best way to convert it?

Nice question! I'm looking at the PyTorch documentation: https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html
If i get that right, lstm_out gives you the output features of the LSTM's last layer, for all the tokens in the sequence. This might mean that if your LSTM has two layers and 10 words, assuming batch size of 1, you'll get an output tensor of (10,1, h) assuming uni-directionality and sequence-first orientation (also see the docs).
For the same LSTM, the size of the ht will be (2*1,1,h) because this output only relates to the features of the last word in the sentence, and is collated from all the layers in the LSTM, unlike the lstm_out which only look at the top layer.
In a nutshell, i think you want to use lstm_out for sequence classification tasks because you have features for all the tokens. Then you want to use ht for tasks that apply at sentence level like classification, as i think it is assumed that the features of the last token are the most rich in an LSTM and representative of the sentence because it contains LSTM-like linkages to all the other tokens (assuming uni-directionality, i think for bi-directionality PyTorch will slice the backward states across the layers for the last sequence too)..
if you want ht to be (1,m,n) instead of (2,m,n) , then you need only 1 layer and 1 direction. This is closely linked to the design of your architecture though, you would need to think through whether you really want to get rid of bidirectionality and keep a shallow LSTM.


How does SHAP values help us to determine importance of a feature for a model trained by gradient boost? I've read http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf and https://medium.com/@gabrieltseng/interpreting-complex-models-with-shap-values-1c187db6ec83 which is like a summary of the first link.

In general I didn't understand anything about how does SHAP values helps and how it helps us determine importance of features from the first paper. At the second article he has a very simple decision tree and calculates shapely value for a feature for a specific training example. It doesn't say that at the end which value determines its importance(i.e mean of a feature for every training example. I don't know). Or why this works.

And there is a confusion between them. First the first article uses SHAP values which defined as "Shapley values of a conditional expectation function of the original model" at the second article it just uses shapley values.

I read several academic papers and website articles but I couldn't address to my question. Most of websites deal with its framework application anyways. If you can explain or give a useful resource I would be appreciated.

I think that the decision tree that appears in the second article is just illustrating the xgboost model that the shap is applied on.

I would like to suggest you to read Christoph Molnar tutorial book on explainability, especially the chapters about Shapley and about the shap algorithm.

The first term (shapely) helps to decompose the effect of each feature on the predicted output by trying all the combinations ("coalitions") of features with the input values (vector). By that, the method allows deconstructing how the output of the model been constructed by each of the input feature values. This approach looks on the features as players that their different strategies affect the final prediction, and it originated in the game theory domain.

The second term, SHAP, is basically calculating the same shapely values with some extensions for missing values that are not handled well in the first and improvements for the algorithm that estimates the shapely values themselves both by speed (the Kernel function that approximates them) and quality of the approximations (the weighting method that combines the combinations).

Besides does algorithmic differences, I think that the shap method is also the approach of using the Shapley algorithm for the usage of interpreting ML (tree-based) models, mainly, by exploiting their additivity property to describe the prediction as a decomposition of the sum of features contributions.


Difference between rbfnn and svr with gaussian kernel I try to understand the difference between radial basis neuron network and support vector regression with Gaussian Kernel. I watched Andrew Ng presentation about kernels in SVN, I have read also an introduction to RBFNN in Bishop's book and Wikipedia's article about SVR.

As I understand it correctly in both cases I try to find a function from features space to R by computation of gaussians of a distances between data from training set to some centers distributed in feature space (activation of rbfs). Next I try to find the best set of linear combination of coefficients, to approximate my data as a linear combination of activated RBFs.

However in some papers these methods are mentioned as two different, competitive methods.

So my question is what exactly is the difference?

Your description of the model functioning is correct. Structurally, both models are representatives of the so called kernel methods. As such they are very similar, the same in many cases.

What is completely different between the two methods is the way the kernel centers and linear coefficients are derived. My experience with SVM is limited, so I cannot go into the details of the training method, but in summary is consists of solving the constrained optimization problem of finding the best approximation subject to maximum coefficients. This is achieved by linear programming or through a method like Sequential Minimal Optimization (SMO).

Training RBFNs, on the other hand, is a different story. Identifying the kernels is done in one of two ways usually; either by using a clustering algorithm or by using Orthogonal Least Squares (OLS). In either case, the linear coefficients are identified as a second step through least squares (LS).

The differences in training mean that, even through in structure the resulting models may be the same, in functioning they will most probably be completely different, due to the different fitting procedures.

Some references:

J. C. Platt, “Fast Training of Support Vector Machines Using Sequential Minimal Optimization,” Adv. kernel methods, pp. 185–208, 1998.

J. Moody and C. J. Darken, “Fast Learning in Networks of Locally-Tuned Processing Units,” Neural Comput., vol. 1, no. 2, pp. 281–294, Jun. 1989.

S. Chen, C. F. N. Cowan, and P. M. Grant, “Orthogonal least squares learning algorithm for radial basis function networks,” IEEE Trans. Neural Networks, vol. 2, no. 2, pp. 302–309, Mar. 1991.


Generating random numbers from best probability distributions? I have done some statistical distributions on real-world data. The distribution fitting gives me the below results (lognormal fits the best on data based on chi-squared test):

Best Distribution fits with parameters:
---------------------------------------

Distribution: lognorm
Parameters: (0.5921510072108613, -0.006454418407435666, 0.021090953536473916)

Distribution: expon
Parameters: (2e-06, 0.018877666177431276)

Distribution: pareto
Parameters: (11.057961210885452, -0.20440106940633995, 0.20436985884797232)

Here, i want to achieve two things:

Using the best parameters,how to generate the new synthetic data that represent our original data?.
Does the rvs method in python is useful?
Compare the actual and generated data distribution fitting as shown in below figure.

Good question. Lots of options.

Most would recommend you use CDFs instead of histograms. So convert your observed distribution to empirical CDF (in R it’s ecdf() - I dint know python). Then, plot another line that is the theoretical CDF using your best-fit parameters over the span of a to b, where a to b gives you at least the same coverage as your original data set limits.

Alternatively you could go for a chi-square option that’ll give you observed vs predicted and can be plotted as histogram but I don’t find that better in most cases, though it might make quantitative comparison more accessible


Kaggle notebook Vs Google Colab What are the major differences between Kaggle notebook and Google Colab notebook?
To work on a dataset my first step is to start a Kaggle notebook but then I cant help thinking what could be the advantage of using Colab notebook instead.
I know few differences, correct me if I'm mistaken about any:

Kaggle has a console and Colab doesn't (but I still don't know what to do with the console).
Kaggle notebook allows collaboration with other users on Kaggle's site while Colab allows collaboration with anyone using the notebook's link.
Kaggle doesn't have "Stackoverflow" instant search like Colab does.
Kaggle requires uploading the data onto the Kaggle's site while Colab allows using data from Drive.
Kaggle doesn't have the feature of uploading the notebook directly to GitHub like Colab does.
Kaggle's notebook can be made public for all the users in Kaggle to view,vote,fork and discuss but Colab's notebook remains in your Drive until you decide to share with someone or upload somewhere.

Anything to add that is technically advantageous?

There are several rundowns between the two services like this one.
Key for me:

Only Kaggle supports R, only Colab supports SWIFT
Colab is a Google product and is therefore optimized for Tensorflow over Pytorch
Colab is a bit faster and has more execution time (9h vs 12h)
Yes Colab has Drive integration but with a horrid interface, forcing you to sign on every notebook restart

Kaggle has a better UI and is simpler to use but Colab is faster and offers more time.


How to shuffle input data using stochastic gradient decent? Let's say that I have input with size $n\times n$ and after that I operated shuffled operations and get the output with the same dimension. But I don't know if we can do that or not. The idea is to permute all the configuration and find the minimum configuration when given loss. But the operation became very expensive with the growth of $O((n^2)!)$, and with each iteration we need to compute loss separately. So is there an optimization for such problem or it just not feasible?

In other words, is there a gradient correlation between different configuration that will lead to the local minima of the loss function?

If I understand your question correctly, I would advise you not to spend time on this.

If you have data that has a structural order to it, for example time-series data where each sample follows from the previous, then shuffling would be throwing that information away.

If your data is not sequential, on the other hand, then computing all possible permutations and training the model on each of them is really just a waste of time. We shuffle the data e.g. to prevent a powerful model from trying to learn some sequence from the data, which doesn't exist.

Training a model on all permutations might be a way to uncover the correct order of the data, is you know it has an order, but it was already shuffled. Otherwise, this experiment would be as useful as training the same model $(n^2)!$ times, just using a different random seem each time.

Have a look here for some more points on shuffling data and why we might do it. Here is more discussion on shuffling input samples.


Joint spatial clustering: How to force clusters to minimally contain datapoints from all datasets I want to jointly cluster datapoints coming from different datasets (50 datasets with around 2000 points each). I would like to then extract information associated to the datapoints belonging to the different clusters to compare aspects of the datasets.

Now my problem is that if I just place the datapoints originating from the different datasets into the same feature space, too few clusters comprise datapoints coming from all datasets simultaneously, so that after filtering I'm left with too few clusters (I tried with kmeans and similar methods).

My question is: Which is the best way to jointly cluster my points while imposing the condition that a given cluster should contain points from all datasets? The ideal solution would also allow some outliers to not fulfill this condition. The first thing I could think about is to define distances between points and clusters which are updated depending on whether a point belonging to the same dataset is already present in the cluster? Seems too far fetched though.

I would appreciate any ideas, thanks a lot in advance!

Edit: Three of my six features are spatial coordinates and ideally I would also want the clusters to be connected within a given dataset.

I don't understand the purpose of imposing a condition that requires any cluster to contain at least some (let's say even 1) point from every dataset but at the same time find a solution that also allows that condition to be broken by outliers (I assume these are specific cluster outliers or data set outliers?).

Have you considered implementing an overlapping cluster algorithm?  You don't get the full distinctness of a traditional single membership but your might better fit your data to an algorithm that achieves the desired condition.  It will probably require some exploration of the data and testing for cluster membership after the fact.


what to do if the missing data in one column is based on some value/condition in another column in r? I have a dataset with 20,000 observations and 19 variables. To start off with I have a gender column which has three levels namely 'M', 'F' and 'U' where U can be taken as not disclosed. Whenever there is a 'U' in the gender column, there is an NA in two of the other columns namely Age and Tenure. This could basically be interpreted as a person who is not ready to disclose their Gender is not ready to disclose their age and tenure. How do I deal with such a situation? Apart from these three columns there are other 16 columns in the dataset that have got meaningful data in them. Would the normal imputation techniques out there like a KNN Imputation help me out in such a case?

Here is my reproducible example that I have tried my best with:

x<-data.frame(gender=c('M','M','F','F','U','F','M','U'),age=c(21,24,20,34,NA,40,56,NA),tenure=c(7,4,5,3,NA,2,4,NA),job=c('Doctor','IT','Banking','Truck Driver','Finance','Agriculture','Electrician','Teacher'),country=c('Australia','America','New Zealand','Sweden','England','France','Denmark','Norway'))

The Dataframe:

 gender age  tenure      job      country
1      M   21     7       Doctor    Australia
2      M   24     4        IT       America
3      F   20     5      Banking    New Zealand
4      F   34     3    Truck Driver Sweden
5      U   NA     NA      Finance    England
6      F   40      2    Agriculture  France
7      M   56      4    Electrician  Denmark
8      U   NA     NA      Teacher    Norway

As you can see from the example above whenever the gender is undefined, there are missing values in both age and tenure and this is the case overall in the entire dataset. What would be the best way to deal with such a situation? And this what is called a Missing at Random data, is that right? Any suggestions would be extremely helpful. Thanks a lot.

If you plan to use LightGBM or XGBoost, the advise is "Do not do anything". These methods treat NA in a specific way, different in each decision tree and the results obtained are usually much better than using imputing.


Binary Classification I wanted to start off by saying this is not an exact duplicate of the other question. I checked it and it didn't have what I urgently need.
So here is the problem. I have a dataset with 30 features and 100000 rows of train data. I want to make a Binary classification model which determines whether the person is eligible or not for membership at our club. I am a rookie data scientist, and binary classification is a first for me. So please help me and tell me which model would be the most accurate for this purpose. Also, the time taken by the model to train doesn't matter... Thank you so much

update:
Ok, I have used logistic regression and instead of giving some members as accepted(1) it is showing me al members do not get the membership...So I thought there might be a mistake in choosing the Model, so I am looking for other models...

Looks like to me this is a classic imbalance binary classification problem (see comments above). What loss are you using ? It looks like your model is predicting the non-membership class because it’s minimising it’s averaging loss. Here are some techniques you might wanna try to solve this issue:

use regularization
over sampling the membership class
under sampling the non-membership class
select variables, linked to regularization above with l1 and l2 penalties
feature engineering: you specified you have 30 features but are all of them useful ? How do you preprocess them to feed the model ? Are they numerical or categorical ?

I hope this gives you some ideas about how tackling the problem. Changing the model won’t miraculously solve your issue.