Difficulty in choosing Hyperparameters for my CNN My task is to estimate a person's age based on a face image of that person.
To that end I'm using a CNN and at first stage I was based on the following article: DeepExpectation which uses a VGG16 architecture to predict a person apparent age (the age that other people would vote). I'm using ResNet 50 architecture (and I'm using this implementation of it: ResNet Tensorflow in tensorflow). The dataset I use for learning is taken from the same article above and can be downloaded from here: WIKI-IMDB dataset. It is composed of 523,051 face images with tagged ages but I found that most of them are garbage (doesn't have real age as the label or has more than one face in it or no face at all). after filtering this dataset (by throwing images that have more than one face or no face at all or the label that represents the age doesn't make sense) I'm left with approximately 150k images in my dataset. in the process of making this dataset I centered all faces in the image and cropped it to be 224*224*3. To increase the size of my dataset I flipped horizontally each image in my dataset to get a total of 300k images in my dataset. I then split the dataset to 240k train, 30k validation and 30k test images.

I'm loading a pre-trained model (trained on ImageNet) and I hoped to get the results they got in the article above (something close to 3.2 years with MAE as the evaluation metric). I also should mention that I'm using cross-entropy as the loss and an l2 loss for regularization and I have 101 classes as the logits (ages 0-100). The batch size I'm using is 64 (The maximum my GPU can handle with). I tried several combinations of hyperparameters but until now the best MAE I got on the validation set was 5.8 years.

Examples of loss graphs I got and their hyperparameters:
weight decay = 5e-4,
momentum optimizer with momentum=0.9,
first learning rate = 0.01 and reducing it by a factor of 10 each 2 epochs until reaching to 1e-6,
the above LR is the LR assinged to the last FC layer. for the layer before it I used 0.5 of that LR. for the middle layers I used 1e-2 of that LR and for the first layer I used 1e-3 of that LR.
I got the following graphs: (orange = train , blue = validation)
  loss:
  LR:
  MAE:

example #2:
same hyperparameters as in the example above only different ratios between first layers and last FC layer. in this example I used 0.5 of the LR in the last FC layer for the layer before it and 0.1 of the LR in the last FC layer for the other layers.
I got the following graphs: (orange = train , blue = validation)
  loss:
  LR:
  MAE:

As you can see from the last 2 examples, I'm reaching to ~6 pretty fast but it seems as the optimization on the train loss gets stuck and also that the validation loss doesn't approach the train loss from some point.
Do you have any suggestions?

Also, I'm having some thoughts on whether it is good to apply some of the following:

Initalize weights from the pre-trained network until some layer
Initialize all weights from the pre-trained network beside the batch-norm
Freeze some of the layers after loading them and train only the last layers
Apply different LR to different layers (as I did in the examples above)
Should I use Momentum optimizer or maybe Adam Optimizer
If I'm using Momentum optimizer, what should be the learning rate schedule (learning rate decay)?
If I'm using Adam optimizer, does it make sense to use different learning rates to different layers?
What should be the weight decay hyperparameter?

Any help would be much appreciated.

I didn't know which stackexchange site is more appropriate for this question so I also posted this question on CrossValidated: Difficulty in choosing Hyperparameters for my CNN

I am going to comment on your ideas:

This is generally recommended. There is no proven reason not to do it but many reasons to do it, and you should initialize as much weights as possible. Typically it is done until the last layer.
Initialize everything that you can from pre-trained network.
I would freeze the first layers in the first epochs, and after that unfreeze them. If possible, I would rather use differential learning rates, as this post shows.
Totally, as shown above.
Adam is more widely used, I would use that.
Cosine scheduling with reestarts has been shown to be very effective, but any learning rate with reestarts should be similar.
It does make sense.
I wouldn't worry about that until you are overfitting, and you are not overfitting yet. If you overfit, I would recommend dropout instead of weight decay.


Knime : Scatter Plot I have a sample dataset containing around 77k of rows and I have clustered them by using k mean clustering and colored the clusters using color manager. Now i would like to view them, However I could not set the No. of rows to display as 77k in Scatter Plot and Is there any way to visualize the clusters in Knime?

Can't help if you're fixed on KNIME, but if you can use Orange Data Mining, I've had success clustering (took a minute or so) and previewing a scatter plot of a dataset with 32k rows.


Is sampling a valid way to reduce complexity? I'm facing an issue where I have a massive amount of data that I need to cluster. As we know, clustering algorithms can have a very high O complexity, and I'm looking for ways to reduce the time my algorithm is running.
I want to try a few different approaches, like pre-clustering (canopy clustering) or subspace clustering, correlation clustering etc.
However, something that I haven't heard about, and I wonder why - Is it viable to simply get a representative sample from my dataset, run the clustering on that, and generalize this model to the whole dataset? Why/why not is this a viable approach? Thank you!

I would get a sufficiently large random/representative sample and cluster that.
To see what is such a sample, you will have to get two such samples and cluster them to get cluster solutions c1 and c2. If the matching clusters of c1 and c2 have the same model parameters, then you probably have representative samples.
You can match the clusters by looking at how c1 and c2 assign drawn data to clusters.


Cross-validation strategy I have a regression problem and I am in doubt about how I can calculate RMSE in my life-cycle.

I deal with time-series and for every prediction, I want to look N points in the future. It is apparent how to calculate RMSE for a single iteration. My question is how to calculate RMSE for N predictions of N points to get a meaningful prediction performance metric.

I guess, I can average RMSE of all iterations though as I said I am not sure at all if this would reflect actual performance.

The natural choice would be the total squared error across the N predicted values, averaged across all examples. This is the simple extension of mean squared error from the univariate case. If you're using multivariate linear regression, this is in fact what you want to optimize in order to get the maximum likelihood estimate of the parameters as well.


Could we use image hasing techniques for images classification tasks? I have read some articles about image hashing, and I would like to know if we could apply this technique for general purpose images classification tasks.

Especially I would like to know which could be the drawbacks in using image hashing for this kind of tasks.

You could, but it won't be very effective.

Image hashing is aimed at detecting two instances of almost the same image.  So, if your training set contains an image of a dog, and the test set contains an almost-identical image, then using image hashing you could use that to learn the label of the test set image.  But in practice that doesn't provide much generalization power.  In practice, we want to take hundreds of different images of dogs, and use that to learn to recognize new images of docs, so that if in the test set we are given a totally new image of a dog, we can still classify it as a dog.  Image hashing won't help with that.

In other words, image hashing isn't designed for this sort of thing and will work poorly.


Linear regression using math or machine learning? Why even use machine learning for this? I have studied statistical math and is now taking a course in machine learning. The first example the teacher talked about is how to find a linear trend line using machine learning. Why would anybody do that? In my statistical courses we calculated linear trend-lines on datasets using far less computer power than a machine learning approach would use according to my understanding. Is linear regression using machine learning only a nice example to learn machine learning basics, or are there any real world reasons for why anyone would use machine learning to find such a simple trend line?

Linear Regression is Linear Regression regardless of how you calculate/estimate the parameters.
The question becomes significant in case of a large multi-variate dataset where it is not easy/fast/possible to compute the parameters using algebraic equations (aka fitting simple line). In such cases Machine Learning techniques such as Stochastic Gradient Descent (SGD) can be helpful.


Determinate K in K-Means Clustering I have salary data of several user (Python list).
Now I am using KMeans to cluster them.
Given this data, Is there a way to figure out the best value for 'K' automatically through program?
I tried silhouette analysis
take k with highest silhouette_avg score. Is this the best way?

Wang, Kaijun, Baijie Wang, and Liuqing Peng. "CVAP: Validation for cluster analyses." Data Science Journal 0 (2009): 0904220071.:

To measure the quality of clustering results, there are two kinds of
validity indices: external indices and internal indices.
An external
index is a measure of agreement between two partitions where the first
partition is the a priori known clustering structure, and the second
results from the clustering procedure (Dudoit et al., 2002).
Internal
indices are used to measure the goodness of a clustering structure
without external information (Tseng et al., 2005).
For external indices, we evaluate the results of a clustering algorithm based on a known cluster structure of a data set (or cluster labels).
For internal indices, we evaluate the results using quantities and features inherent in the data set. The optimal number of clusters is usually determined based on an internal validity index.

(Dudoit et al., 2002): Dudoit, S. & Fridlyand, J. (2002) A prediction-based resampling method for estimating the number of clusters in a dataset. Genome Biology, 3(7): 0036.1-21.
(Tseng et al., 2005): Thalamuthu, A, Mukhopadhyay, I, Zheng, X, & Tseng, G. C. (2006) Evaluation and comparison of gene clustering methods in microarray analysis. Bioinformatics, 22(19):2405-12.

In your case, you need some internal indices since you have no a priori clustering structure. There exist tens of internal indices, like:

Silhouette index (implementation in MATLAB)
Davies-Bouldin
Calinski-Harabasz
Dunn index (implementation in MATLAB)
R-squared index
Hubert-Levin (C-index)
Krzanowski-Lai index
Hartigan index
Root-mean-square standard deviation (RMSSTD) index
Semi-partial R-squared (SPR) index
Distance between two clusters (CD) index
weighted inter-intra index
Homogeneity index
Separation index

Each of them have pros and cons, but at least they'll give you a more formal basis for your comparison. The MATLAB toolbox CVAP might be handy as it contains many internal validity indices.

More information:

Determining the number of clusters in a data set
How to determine which method is the most reasonable clustering results?
How can I detect if my dataset is clustered or unclustered (i.e. forming one single cluster)?
How can we choose a "good" K for K-means clustering?
Cluster analysis in R: determine the optimal number of clusters


Skewed binomial data for small p Say you have a binomial distribution with $p$ very small ($\approx 0.001$).

You are asked to predict the conditional success rate $SR=S/T$ with $S$ successes out of $T$ trials given a set of conditions $X$.

One would expect (correct me if I'm wrong, though I ran simulations and am quite confident) $SR$ to have a downward with increasing $T$ when $T$ is still small, and to approach $p$ when $T$ is large.

The distribution of trials and successes is not uniform across $X$, so the train set tends to give higher $SR$ to conditions with smaller $T$ values.

How would you treat this data skew when constructing a model?

When fitting a GLM (at least in R), I know there is a optional weight vector that you can include.  This weight is not to give more importance to an observation, but to rather weight observations based on $T$ for example.

The R documentation says:

 For a binomial GLM prior weights are used to give the number of trials
 when the response is the proportion of successes

So I believe this would help adjust for your non-uniformity.  How to choose those weights can be tricky I suppose and can vary a lot with your data, but its worth looking into if it looks like it will help your problem.

Do you have enough data where you can play with dropping data, or maybe if you simulate how much the $SR$ is inflated at low $T$ you can try to adjust for it somehow.  It looks like after about 1000 trials the high success rate issue begins to mellow out, and after 3000 or so you start getting more reliable measures.


50% accuracy on multiclass classification I am trying to do a multiclass classification on a significant amount of output labels (1000).

I built a model using KNN. The accuracy given by accuracy = knn.score(X_test, y_test) is 0.5.

Does this mean that given an input, the model is able to predict 50% of the time which label the data belongs to? If yes, I would intuitively say that this is good, since randomly choosing a label would have a probability of 0.1%.

Assuming it really is a one form many class problem and not a multi-label problem.

An accuracy of .5 would mean that half of the instances were classified correctly. That would also mean that the model is able to generate the correct class half of the time. For your test data, results can still vary for new data.

If that is a good score depends on more than the number of classes. Quite often one class is represented more than others. The ZeroR-score is a good baseline for that.

Suppose you have 10 classes and Class A makes up 30% of the population. A random proces would achieve an accuracy of .1, a proces that simple says "Class A" all the time, would post a .3 accuracy.


In what stage should you plot validation and learning curve So lets assume I have a binary classification problem and I started out with a logistic regression model. I quickly evaluate the models accuracy (lets assume we don't have a skewed dataset). After this I start to implement other models to compare, I begin with checking the accuracy score and plot a ROC curve to evaluate AOC score.
Lets say that I carry on with my logistic regression model. I wan't to fine-tune it and use a grid search algorithm and from there carry on choosing the best hyperparamters w.r.t the accuracy score.
When do I actually plot the learning and validation curve? I pressume that I should plot it next to see how a larger set of paramters affect the score and then to see if my model is underfitting or overfitting.
But shouldn't I plot the learning curve much earlier for example in the first step where I choose to start with a logistic reg. model?
To sum it up: In what stage is it appropiate to plot a learning curve?

I understand that with "learning curve" you are referring to the plot of the loss function over the training data (or subsets of it) when optimizing with iterative methods, like gradient descent. This is sometimes referred to as the training loss plot.
You use a training loss plot during training, to evaluate the convergence of your training algorithm and model. Usually, together with the training loss plot, you visualize the validation loss, that is, the loss function computed on a held-out dataset, to evaluate whether your model is overfitting your training data.
Normally, the training loss gets a new point after each optimization step. For instance, in stochastic gradient descent, the training loss points are the loss evaluated at each minibatch. On the other hand, the validation loss curve gets a new point after each epoch, or at least after a few optimization steps have been performed.
This is different from a ROC curve, which is evaluated once the model is fully trained, to understand the behavior of the trained model.


References for longitudinal data analysis So my goal is to study longitudinal data (data in time series) by applying some data mining techniques. Ultimately I want to be able to "predict" outcomes. For example, a study of patients along the years/months (repeated measures) can be used to predict some diseases using things like logistic regression, neural networks, etc..

I'm using R. However I'm having a hard time finding good references on this subject. I have a mathematic background, even though I've applied some data mining techniques before, I've never applied those to time series.

If someone can recommend me some good theoretical references or hands-on examples in R, that would be greatly appreciated. Also if you know a relevant article on time series analysis, that would be great too. Thanks in advance.

In data science, people normally refer to “longitudinal data” as one or more time series. So you can try searching for that. A classic is to use ARIMA (available in many languages including R). Personally I also like Facebook’s Prophet library as a starting point, because it’s easier to use and works surprisingly well. Also available in R.


Why you shouldn't upsample before cross validation I have an imbalanced dataset and I am trying different methods to address the data imbalance. I found this article that explains the correct way to cross validate when oversampling data using SMOTE technique.
I have created a model using AdaBoost algorithm and set the following parametres to be used in Grid Search:
    ada = AdaBoostClassifier(n_estimators=100, random_state=42)
    params = {
        'n_estimators': [50, 100, 200],
        'random_state': [42]
    }

According to the article, this is the wrong way to oversample:
    X_train_upsample, y_train_upsample = SMOTE(random_state=42).fit_sample(X_train, y_train)

    # cross-validate using grid search

    grid_naive_up = GridSearchCV(ada, param_grid=params, cv=kf,
                                 scoring='recall').fit(X_train_upsample,
                                                       y_train_upsample)
    grid_naive_up.best_score_

0.6715940782827282
    # test set
    recall_score(y_test, grid_naive_up.predict(X_test))

0.2824858757062147
Whereas the correct way to oversample is like so:
    from imblearn.pipeline import Pipeline, make_pipeline

    imba_pipeline = make_pipeline(SMOTE(random_state=42),
                                  AdaBoostClassifier(n_estimators=100, random_state=42))
    cross_val_score(imba_pipeline, X_train, y_train, scoring='recall', cv=kf)
    new_params = {'adaboostclassifier__' + key: params[key] for key in params}
    grid_imba = GridSearchCV(imba_pipeline, param_grid=new_params, cv=kf, scoring='recall',
                            return_train_score=True)
    grid_imba.fit(X_train, y_train);

    # How well do we do on our validation set?
    grid_imba.best_score_

0.29015614186873506
    # compare this to the test set:
    y_test_predict = grid_imba.predict(X_test)

0.2824858757062147
So, according to the article, the first method is wrong because when upsampling before cross validation, the validation recall isn't a good measure of the test recall (28.2%).
However when using the imblearn pipeline for upsampling as part of the cross validation, the validation set recall (29%) was a good estimate of the test set recall (28.3%). According to the article, the reason for this is:

When upsampling before cross validation, you will be picking the most
oversampled model, because the oversampling is allowing data to leak
from the validation folds into the training folds.

Can anyone explain to me simply how the oversampling allows data to leak into the validation and causes the overfitting? And why does this problem not occur in the imblearn pipeline?

To see clearly why the procedure of upsampling before CV is mistaken and it leads to data leakage and other undesired consequences, it is useful to imagine first the simpler "baseline" case, where we simply upsample (i.e. create duplicate samples) without SMOTE.
The first reason why such a procedure is invalid is that, this way, some of the duplicates due to upsampling will end up both to the training and the validation splits (CV folds); the result being that the algorithm is validated with some samples that have already been seen during training, which invalidates the very fundamental requirement of a validation set (fold) and it is actually the very definition of data leakage. For more details, see own answer in the SO thread Process for oversampling data for imbalanced binary classification; quoting from there:

I once witnessed a case where the modeller was struggling to understand why he was getting a ~ 100% test accuracy, much higher than his training one; turned out his initial dataset was full of duplicates -no class imbalance here, but the idea is similar- and several of these duplicates naturally ended up in his test set after the split, without of course being new or unseen data...

But there is also a second reason: this procedure shows biased performance measures in our validation folds that are no longer representative of reality: remember, we want our validation folds to be representative of the real unseen data, which of course will be imbalanced. Performing CV after upsampling results also to artificially balancing our validation folds; doing so, and claiming that we get X% accuracy when a great part of this accuracy will be due to the artificially upsampled minority class makes no sense, and gives misleading impressions. For details, see own answer in the SO thread Balance classes in cross validation. Notice that the author of the post you have linked to says (rather cryptically, and only in a parenthesis):

(we are smart enough not to oversample the test data)

For more corroboration, here is Max Kuhn, creator of the caret R package and co-author of the (highly recommended) Applied Predictive Modelling textbook, in Chapter 11: Subsampling For Class Imbalances of the caret ebook:

You would never want to artificially balance the test set; its class frequencies should be in-line with what one would see “in the wild”.

Now, it's true that the above hold for the case of balancing through simple upsampling of the minority class; but SMOTE does not do that - it uses interpolation to create synthetic samples that are "close" enough to the real minority ones. How does this change the situation?
Not much.

The second reason stated above (biased performance measures in the validation folds) is still fully applicable - in fact, it holds independently of the exact nature of the upsampling (duplicate samples or synthetic ones).

Given that the synthetic samples generated by SMOTE are indeed highly correlated with the real ones, the problems due to the first reason mentioned above are still largely present, although somewhat ameliorated.

In contrast, the pipeline approach does not suffer from these issues, because it first splits into training and validation folds, and applies SMOTE subsequently only to the training ones.


Is this an over-fitting case? I'm a new programmer and this is my first ever neural network for real world application.

Here is the deal, I'm using a top-less pre-trained VGG-16 with some dense layers on top of it.(for image classification problem)

But no matter what hyper parameters I change I always get plots similar to these.

So my question is at what loss (val_loss) can we consider the model trained?

And is there a problem if the images that I feed the VGG-16 are RGB/255?

And if someone could explain how high validation loss (even if validation accuracy is high) could impact the model?

Thanks for taking time to answer me, and sorry if the questions seems stupid but I didn't find answers when searching else where.

Your model is indeed overfitting. There can be a lot of reasons why it could be happening - How many images are you training with?, Aren't you using Regularization, data augmentation using random crops or flips, Dropout to prevent overfitting?

  So my question is at what loss (val_loss) can we consider the model
  trained?

We can generally stop training when Validation loss doesn't decrease anymore. This is generally done by defining a variable eg: n = 5 and checking if the validation loss has decreased after running n epochs. If you're solving a well-known problem(MNIST, Fashion-MNIST, CIFAR-10, etc), you can check the best score/loss achievable by the model and tweak your model(add or remove layers, prevent overfitting, etc) to achieve the score.

  And is there a problem if the images that I feed the VGG-16 are
  RGB/255?

You should preprocess the images for faster training and also subtract  the mean RGB pixel values of images used to train pretrained network(if the pretrained net is trained using imagenet data,then subtract imagenet dataset's mean RGB pixel values from your input images) from your input images. Here you can read keras preprocessing_input
method.

  And if someone could explain how high validation loss (even if validation accuracy is high) could impact the model?

Validation loss measures how generalizable your model is to unseen data. If your training loss and validation loss don't improve during training, then you are underfitting. If you're training loss is low but your validation loss is stagnant or increasing then you're overfitting, which means that your model is learning unwanted noise or patterns which help it learn or memorize training data but are not generalizable on an unseen validation set.

Here is a link explaining data preprocessing, augmentation, transfer learning using pretrained net.


Is it better to have one model with more categories or less with two for multi-label classification? For classifying text into three classes question, complain and  complements
where each sample can have multi-labels (question and complains,  question and complements):

is it better to have one model for all three targets?
or two models, the first for (question or not) and the second one for (complains, complements or else)?

which approach is better when the data are labeled, unlabeled and unbalanced?

Generally there are two main options for multi-label classifications:

A binary model for every class, in this case the labels are supposed to be independent of each other.
A joint model which learns all the classes together, so at training stage the class represents the three values for the labels, for instance:

class "NNN" means no label at all
class "NNY" means no for question, no for complain and yes for complements
class "NYN" means no for question, yes for complain and no for complements
...
class "YYY" means yes for question, yes for complain and yes for complements

In theory there are $2^3$ possible classes for 3 labels, but in practice there might be less combinations in the data (as suggested in the question). If the labels depend on each other (for instance if it's unlikely to have a document which is both complain and complements), a joint model is more appropriate. However a joint model might need more instances to be trained properly, since it has more work to do than a binary model.


Offline evaluation of recommender systems Let's say I want to compare whether one recommender system (A) is better than the other (B).

One approach is to let people rate recommendations returned by both systems.

However, there situations when I would like to evaluate my recommender system offline. One approach I considered is to collect user ratings for recommendations from system A and turn them into a test dataset. However, if the system B returns recommendations that do not appear in test dataset, that does not mean they are bad. It just means I have no rating about them.

Some alternatives:

try to increase test dataset volume as much as possible to increase overlap between recommenders.
limit the size of items space, let recommender select items from 500 instead of 5*10^6 items during tests.

What other approaches would you recommend suggest?

There is always a bias in offline testing in recommender systems. Real evaluation happens in A/B testing. But, that should not discourage one from doing offline testing.

There is ongoing research on this topic using multi-armed bandits. I recommend reading offline testing procedures using reco-gym.

Following workshop talks about removing these biases and developing efficient offline estimators for recommender systems. They also conducted an online course on efficiently doing an offline evaluation of recommender systems.

You can also follow a video tutorial on this website on counterfactual evaluation


the probability distribution of dependent variables There are three variables, X3 is a function of X1 and X2, X2 also depends on X1.The dependency relationship is shown as the following graph.

In specific, X3= f(X1, x2) and X2=g(X1). Therefore, X3=f(X1, g(X1)). If the probabilistic distribution of X1 is known, how to get the probabilistic distribution of X3.

You need to tell us what the distribution of x1 is and what f and g function are exactly? When f and g are simple, we can usually solve this analytically. But when the function is complex, this kind of problems are typically approximated by Monte Carlo simulation.


Join two dataframes - Spark Mllib I've two dataframes. The first have the some details from all the students, and the second have only the students that haved positive grade.

How can I return only the details of the student that have positive grade (make the join) but not using SQL Context.

I've this code:

val all_students = sc.textFile("/user/cloudera/Data");

case class Students(Customer_ID:String,Name:String,Age:String);

def MyClass(line: String) = {
val split = line.split(',');
Students(split(0),split(1),split(2))
}

val df = all_students.map(MyClass).toDF("Customer_ID","Name", "Age").select("Customer_ID","Name", "Age");

val students_positive_grande = sc.textFile("/user/cloudera/Data");

How can I make the join between this datasets? I want to join the "Customer_ID" with the first column of the second column...

Use this syntax:

val joinedDF = students_positive_grande.as('a).join(
                          df.as('b),
                          $"a.Customer_ID" === $"b.Customer_ID")

joinedDF.select($"a.Customer_ID", $"b.Customer_ID")


Should I use coordinates as features? I am building a regression model. Each sample/object in my dataset has some numerical and categorical features. Let's call those $f_1, f_2, f_3$ for now. The output that I'm trying to learn is a $2D$ grid of numerical values. Kind of like an $3\times 3$ image with values in it. The output variables are correlated.

For my first attempt at the problem, I thought of using $f_1, f_2, f_3$ as input features and then flatten my image into 9 output features $out_1,...,out_9$. There are multiple techniques that can optimize for those variables independantly/jointly so this is fine.

The other option that I thought of was to use the output grid coordinates as input features. So I would have $5$ input features: $f_1, f_2, f_3$ (as before) and $f_4, f_5$ are my $x, y$ coordinates for the target variable. In this case I would have more samples and my output would consist of only $1$ output target variable.

My problem is that I'm not sure whether any of the options is fundamentally wrong or just bad practice. Is there anything I should watch out for in training and testing if I proceed with either approach?

There is a couple of separate questions I can identify here: one is whether it is appropriate to include coordinate features and the other is whether it is appropriate to convert a multivariate regression (more than one outcome variable - 9 in your example) into a regression with single dependent variable.

Coordinate features

If you think that the dependent variable value in a particular cell of your grid is somehow affected by the location of the cell within the grid (in other words, that values in the same cell are correlated), then you should probably include coordinate features (which can be simple cell identifiers or something more complex, like a more specific identification of where the cell is located within the grid relative to a particular point).  It is possible to even use target-based encoding (i.e., use the mean of dependent variable values in the same cell as a new feature), though that gets into issues like data leakage and overfitting (which can be addressed separately).  You may also possibly want to create interaction terms, allowing for the effects of non-coordinate features to vary by level of a coordinate feature.

This is different than modeling correlation between values across different cells.  If you expect that there is some spatial autocorrelation between cells in this 2-d grid, e.g., a value in cell (1,1) is correlated with values in cells near it: (1,2), (2,1), (2,2), you should probably model that relationship by including some set of spatial features (not just coordinate identifiers), like spatially clustered covariates, or by using a spatially lagged dependent variable model.

If you are not sure if spatial correlation is present, you can estimate a model without any spatial features and then test for spatial autocorrelation (spatial dependence between residuals of regression) using Moran's I (a positive Moran’s I hints at data is clustered, a negative Moran’s I implies data is dispersed.)

Multiple vs single dependent variable

If you go with option 2 as you described it, with one output variable, you will be modeling a completely different variable than if you were modeling 9 (in your example) separate variables.  Each of the original 9 variables likely has its own statistical properties, but if you combine them into one, you will have a new variable that has its own properties (some combination of the 9).  So if you consider the grid to be 9 separate variables, you should stick with modeling 9 separate variables.


Understanding GMM-MMI While studying Gaussian mixture models and the expectation maximization algorithm, I also came across a number of studies that used 'Discriminative training' for the models that addressed some limitations of the EM algorithm. In particular, the use of maximum mutual information is quite common.

I understand the basic idea behind maximizing mutual information; however, I am unable to find any source that gives the equations involved in estimating the means, variances, mixing parameters and the predicted labels.

Can you please state them along with an explanation?

Multiclass Discriminative Training of i-vector Language Recognition,
this article contains update formulas for mean and covariance (diagonal).

First you should obtain parameter estimations using ML (so calculate mean and covariance from class data) and then iteratively update them according to formulas (16-20). I'm not sure about the meaning of C0.


Suitable sample data set to test machine learning algorithms I'm new to Machine Learning and I just came across the sci-kit package. On this interesting page there are many toy data sets used to test different clustering algorithms. Each data set has a unique pattern and some algorithms perform better than others depending on the data sets.
I want to ask why these data sets are chosen as tests for the algorithms? What are the properties for them to be suitable for use in testing? Are there any other data sets with common attributes that are used for the same purpose? Do they have certain names that I can read more about?
Thank you.

The toy examples or the common datasets you are talking about are so because they are simple to visualise it and work with. Their simplicity helps the beginner to train simple models which don't require much compute. The simplicity in the structure of the dataset allows visualisation of the data on lower dimensions.

Reason for using them as test datasets is that they provide us a quick sanity check to see if the algorithm performs or not. The link you provided is specifically for clustering problems. So, datasets which can be easily visualised on 2D plane would be a simple dataset to check the performance of the algorithm via inspection. Had it been a complex datasets like a dataset of human faces, it would be difficult to evaluate the performance of the model through visualisation and inspection.

Some examples for such datasets:

MNIST dataset - collection of handwritten digits used to train classification network to identify the class of digit during test time.

Cifar-10 : collection of RGB images of 10 classes of objects in real world (e.g cars and birds).

Cifar-100: upgrade of Cifar-10. Contains images from 100 classes


Difference between business and financial and data analyst Can anyone please tell me whats the difference between

Business analyst
Financial analyst
Data analyst

And what should one need to learn in order to achieve either of these profile.

Business Analyst
A business analyst is one who understands the specific domain of the project (ex. retail, merchandising to be specific, supply chain etc.). His/her role is to understand the business problem, analyze the current state and capture requirements using various tools like surveys, interviews, group discussions and then provide recommendations and create a Requirements document for sign off.
Business Analyst Skillset-

Communication,  Analytical thinking, Domain Knowledge, Generic
technical Knowledge, Problem solving skills,Decision solving skills,
Managerial Skills,Negotiation and Persuasion Skills.

Financial Analyst
Financial analysts use financial data to spot trends and extrapolate into the future, helping their employers and clients make the best investing decisions. Businesses rely on financial analysts to determine when it is an auspicious time to buy or sell specific securities and, in some cases, companies use reports put together by financial analysts to determine if the entire business should be sold.
Financial Analyst Skillset-

Financial analysts should be great problem-solvers, excel at the use
of logic and possess strong skills in quantitative analysis. In
addition, successful financial analysts have an in-depth understanding
of various financial markets and investment products.

Data Analyst
A data analyst's role is one that works with lots of data to derive meaningful insights to either address business problems or discover hidden trends and patterns that can be leveraged to meet the business objectives.
Data Analyst Skillset-

Data analyst also needs similar skills with some additional skills
like to analyze data like SQL, DATA mining, OLAP, Reports etc.

Note: Strong people skills, leadership ability, and teamwork are beneficial for each type career.
For more info, you can visit these resources-

Career Advice: Financial Analyst Vs. Data Analyst
What is the difference between data analyst and business analyst?
The Differences Between a Business Analyst & a Data Analyst

Hope it helps, cheers! :)


Why decision tree needs categorical variable to be encoded? As per my intuition, decision trees should work better with categorical variables than with continuous variables. If this is the case, why is encoding needed on categorical variables? Can someone give me the intuition behind this?

...why is encoding needed on categorical variables?

That isn't true; decision trees can be built on both continuous and categorical features.  (Why don't tree ensembles require one-hot-encoding? )  Some implementations, however, do not support categorical variables (notably sklearn (for now, update) and xgboost (their politics)).

Now, there is a question of efficiency: the number of bipartitions of the set of categories is exponential in the number of categories, so a complete search of the possible splits is only practical for categorical variables with few categories.
There turns out to be a (surprising?) simplification though: if the underlying problem is a regression with MSE, or a binary classification with cross-entropy or Gini index, then the optimal split can be found by ordering the categories according to their average response value and treating it now as an ordinal variable split.  (That said, still having many categories, especially small ones, might lead to heavy overfitting.)  See Elements of Statistical Learning, section 9.2.4.

Some implementations perform the exhaustive bipartition search but cap the number of categories allowed.  LightGBM and rpart perform the ordered search.  (some R discussion, LightGBM).


Should I oversample my validation data to get better F1 score and PRC? I am currently working with a dataset that is imbalanced, about 30k rows * 14 features (just for you know), and 99.5% of the data is labeled 0. Since the model is strongly imbalanced I decided to use precision/recall/f1 score to decide the performance of model.
I used SMOTE to oversample my training data (after splitting the validation set out). Now my model is trained with oversampled training data, and I am going to test it with validation set. If I just validate it on original validation data, I get a F1 score around 0.05, and the classification report is followed:
          precision    recall  f1-score   support

 Class 0       1.00      0.86      0.93      7606
 Class 1       0.03      0.75      0.05        36

If I oversample my validation data, I get a F1 score around 0.85:
          precision    recall  f1-score   support

 Class 0       0.84      0.86      0.85      7606
 Class 1       0.86      0.83      0.85      7606

My question is:

Should I use an oversampled validation set? (because the result is much prettier but I think the model is the same anyway)

Why do I have such bad metrics on the original validation data? Is it because the data size is not big enough?

what you encounter are real-world problems rarely taught in classes.

For training, I would test SKLearn's class_weight = "balanced" or class_weight={0:0.995, 1:0.005}. It's a very robust technique.
For testing, you can't fiddle with the class_weight. It's meant to simulate the real-world data.
Make sure you don't overfit. E.G. Decision Tree max_leaf_node 5-10 OR (not both) max_depth 3-5.

Your results aren't that bad. A precision 1 of 0.03 is "low", but you only had 36 labels. Roughly speaking, the model labeled 1,000 as 1 (100/0.03 * 36). i.e. 97% false alarm. But with a good recall, you got most of the labels.
Putting it differently: in a dataset of around 8,000 you got 1,000 labeled as 1. That's around 12%. Out of those 1,000 there are about 30 real 1's. You get most of them (around 75%).
Imagine this was Churn Prediction. You don't want to lose customers. Without ML, you might send all your 8,000 customers a discount of 10%. That's expensive. With ML, you would only send 1,000 customers a discount and get most of them wanting to leave. That's a strong improvement over no model.
The same argument applies to many other cases such as Predictive Maintenance.


Big data and data mining for CRM? We are currently developing a customer relationship management software for SME's. What I'd like to structure for our future CRM is developing CRM with a social-based approach (Social CRM). Therefore we will provide our users (SME's) to integrate their CRM into their social network accounts. Also CRM will be enhance intercorporate communication of owner company.
All these processes I've just indicated above will certainly generate lots of unstructured data.
I am wondering how can we integrate big data and data-mining contepts for our project; especially for the datas generated by social network? I am not the expert of these topics but I really want to start from somewhere.
Basic capabilities of CRM (Modules)
-Contacts: People who you have a business relationship.
-Accounts: Clients who you've done a business before.
-Leads: Accounts who are your potential customers.
-Oppurtunites: Any business opportunity for an account or a lead.
-Sales Orders
-Calendar
-Tasks
What kind of unstructured data or the ways (ideas) could be useful for the modules I've just wrote above? If you need more specific information please write in comments.

The two modules where you can really harness data mining and big data techniques are probably Leads and Opportunities. The reason is that, as you've written yourself, both contain 'potential' information that you can harness (through predictive algorithms) to get more customers. Taking Leads as an example, you can use a variety of machine learning algorithms to assign a probability to each account, based on that account's potential for becoming your customer in the near future. Since you already have an Accounts module which gives you information about your current customers, you can use this information to train your machine learning algorithms. This is all at a very high level but hopefully, you're getting the gist of what I'm saying.


Calculating KL Divergence in Python I am rather new to this and can't say I have a complete understanding of the theoretical concepts behind this. I am trying to calculate the KL Divergence between several lists of points in Python. I am using this to try and do this. The problem that I'm running into is that the value returned is the same for any 2 lists of numbers (its 1.3862943611198906). I have a feeling that I'm making some sort of theoretical mistake here but can't spot it.
values1 = [1.346112,1.337432,1.246655]
values2 = [1.033836,1.082015,1.117323]
metrics.mutual_info_score(values1,values2)

That is an example of what I'm running - just that I'm getting the same output for any 2 input. Any advice/help would be appreciated!

First of all, sklearn.metrics.mutual_info_score implements mutual information for evaluating clustering results, not pure Kullback-Leibler divergence!

  This is equal to the Kullback-Leibler divergence of the joint distribution with the product distribution of the marginals.

KL divergence (and any other such measure) expects the input data to have a sum of 1. Otherwise, they are not proper probability distributions. If your data does not have a sum of 1, most likely it is usually not proper to use KL divergence! (In some cases, it may be admissible to have a sum of less than 1, e.g. in the case of missing data.)

Also note that it is common to use base 2 logarithms. This only yields a constant scaling factor in difference, but base 2 logarithms are easier to interpret and have a more intuitive scale (0 to 1 instead of 0 to log2=0.69314..., measuring the information in bits instead of nats).

> sklearn.metrics.mutual_info_score([0,1],[1,0])
0.69314718055994529

as we can clearly see, the MI result of sklearn is scaled using natural logarithms instead of log2. This is an unfortunate choice, as explained above.

Kullback-Leibler divergence is fragile, unfortunately. On above example it is not well-defined: KL([0,1],[1,0]) causes a division by zero, and tends to infinity. It is also asymmetric.


What features are extracted from pre-trained model of CNN Keras? I would like to use the CNN pre-trained model in feature extraction but I don't know what features are extracted from that.
Please let me know!

Here is a good post about how features are extracted using CNN: https://towardsdatascience.com/cnn-application-on-structured-data-automated-feature-extraction-8f2cd28d9a7e

"I don't know what features are extracted from that."
It depends on what dataset was that CNN pre-trained on. E.g. if the model was pre-trained on a dataset consisting of dogs and cats, the features should contain useful information about dogs and cats.

In practice it turned out that models pre-trained on ImageNet dataset extract features which are a good starting point for training on other custom datasets.


Gradient boosting algorithm example I'm trying to fully understand the gradient boosting (GB) method. I've read some wiki pages and papers about it, but it would really help me to see a full simple example carried out step-by-step. Can anyone provide one for me, or give me a link to such an example? Straightforward source code without tricky optimizations will also meet my needs.

I tried to construct the following simple example (mostly for my self-understanding) which I hope could be useful for you. If someone else notices any mistake please let me know. This is somehow based on the following nice explanation of gradient boosting http://blog.kaggle.com/2017/01/23/a-kaggle-master-explains-gradient-boosting/

The example aims to predict salary per month (in dollars) based on whether or not the observation has own house, own car and own family/children. Suppose we have a dataset of three observations where the first variable is 'have own house', the second is 'have own car' and the third variable is 'have family/children', and target is 'salary per month'. The observations are

1.- (Yes,Yes, Yes, 10000)

2.-(No, No, No, 25)

3.-(Yes,No,No,5000)

Choose a number $M$ of boosting stages, say $M=1$. The first step of gradient boosting algorithm is to start with an initial model $F_{0}$. This model is a constant defined by $\mathrm{arg min}_{\gamma}\sum_{i=1}^3L(y_{i},\gamma)$ in our case, where $L$ is the loss function. Suppose that we are working with the usual loss function $L(y_{i},\gamma)=\frac{1}{2}(y_{i}-\gamma)^{2}$. When this is the case, this constant is equal to the mean of the outputs $y_{i}$, so in our case $\frac{10000+25+5000}{3}=5008.3$. So our initial model is $F_{0}(x)=5008.3$ (which maps every observation $x$ (e.g. (No,Yes,No)) to 5008.3.

Next we should create a new dataset, which is the previous dataset but instead of $y_{i}$ we take the residuals $r_{i0}=-\frac{\partial{L(y_{i},F_{0}(x_{i}))}}{\partial{F_{0}(x_{i})}}$. In our case, we have $r_{i0}=y_{i}-F_{0}(x_{i})=y_{i}-5008.3$. So our dataset becomes

1.- (Yes,Yes, Yes, 4991.6)

2.-(No, No, No, -4983.3)

3.-(Yes,No,No,-8.3)

The next step is to fit a base learner $h$ to this new dataset. Usually the base learner is a decision tree, so we use this.

Now assume that we constructed the following decision tree $h$. I constructed this tree using entropy and information gain formulas but probably I made some mistake, however for our purposes we can assume it's correct. For a more detailed example, please check

https://www.saedsayad.com/decision_tree.htm

The constructed tree is:

Let's call this decision tree $h_{0}$. The next step is to find a constant $\lambda_{0}=\mathrm{arg\;min}_{\lambda}\sum_{i=1}^{3}L(y_{i},F_{0}(x_{i})+\lambda{h_{0}(x_{i})})$. Therefore, we want a constant $\lambda$ minimizing

$C=\frac{1}{2}(10000-(5008.3+\lambda*{4991.6}))^{2}+\frac{1}{2}(25-(5008.3+\lambda(-4983.3)))^{2}+\frac{1}{2}(5000-(5008.3+\lambda(-8.3)))^{2}$.

This is where gradient descent comes in handy.

Suppose that we start at $P_{0}=0$. Choose the learning rate equal to $\eta=0.01$. We have

$\frac{\partial{C}}{\partial{\lambda}}=(10000-(5008.3+\lambda*4991.6))(-4991.6)+(25-(5008.3+\lambda(-4983.3)))*4983.3+(5000-(5008.3+\lambda(-8.3)))*8.3$.

Then our next value $P_{1}$ is given by $P_{1}=0-\eta{\frac{\partial{C}}{\partial{\lambda}}(0)}=0-.01(-4991.6*4991.7+4983.4*(-4983.3)+(-8.3)*8.3)$.

Repeat this step $N$ times, and suppose that the last value is $P_{N}$. If $N$ is sufficiently large and $\eta$ is sufficiently small then $\lambda:=P_{N}$ should be the value where $\sum_{i=1}^{3}L(y_{i},F_{0}(x_{i})+\lambda{h_{0}(x_{i})})$ is minimized. If this is the case, then our $\lambda_{0}$ will be equal to $P_{N}$. Just for the sake of it, suppose that $P_{N}=0.5$ (so that $\sum_{i=1}^{3}L(y_{i},F_{0}(x_{i})+\lambda{h_{0}(x_{i})})$ is minimized at $\lambda:=0.5$). Therefore, $\lambda_{0}=0.5$.

The next step is to update our initial model $F_{0}$ by $F_{1}(x):=F_{0}(x)+\lambda_{0}h_{0}(x)$. Since our number of boosting stages is just one, then this is our final model $F_{1}$.

Now suppose that I want to predict a new observation $x=$(Yes,Yes,No) (so this person does have own house and own car but no children). What is the salary per month of this person? We just compute $F_{1}(x)=F_{0}(x)+\lambda_{0}h_{0}(x)=5008.3+0.5*4991.6=7504.1$. So this person earns $7504.1 per month according to our model.


How to split train/test data 50% by class and grouping by Object ID in R? I get pixel values from it using reference polygons. Extracted pixel values are in data frame, but one row represent extracted values for single pixel. In the classification I need to split the dataset into test (50%) and training (50%) by class (tree, meadow e.t.c)

I know how to split a set according to classes. However, I want values extracted for one polygon to be assigned to one of the sets (training OR test ) and they were not mixed

For this purpose I want to use the polygon ID (Object Identification).
I would like to do this using the createDataPartition function. These are just two sample classes (there are many more)

Here is part of table with extracted values:

"band_1"    "band_2"    "band_3"    "CLASS" "Id"
110 134 119 "tree"  1
112 133 118 "tree"  1
105 125 110 "tree"  2
112 132 117 "tree"  2
109 125 115 "meadow"    6
93  110 101 "meadow"    6
86  106 95  "meadow"    7
105 136 116 "meadow"    7
102 128 111 "meadow"    8
108 129 115 "meadow"    8
113 134 119 "meadow"    8

Here is code:

trainIndeks <- caret::createDataPartition(EXTRACTED$CLASS, p = 0.5, list=FALSE, times = 1)

    dataTrain <- EXTRACTED[trainIndeks,]
    dataTest <- EXTRACTED[-trainIndeks,]

Instead of:

dataTrain <- EXTRACTED[trainIndeks,]
dataTest <- EXTRACTED[-trainIndeks,]

try:

dataTrain <- EXTRACTED[ID %% 2 == 1,]
dataTest  <- EXTRACTED[ID %% 2 == 0,]

dataTrain will contain only odd ID, dataTest only even ones.

Using this method you may lose the balance of CLASS distribution between dataTrain and dataTest, but you may improve it by replication or removing of a few records, or by checking the distribution in other splitting, e.g.

dataTrain <- EXTRACTED[ID %% 4 < 2,]
dataTest  <- EXTRACTED[ID %% 4 > 1,]

EDIT:

For random changing the split, you can use:

set.seed(123)
N <- 10
#N <- round(max(EXTRACTED$ID)/10) # for more random grouping
p <- 0.5 # train/(train+test) ratio
idx <- sample(0:(N-1),round(N*p))
dataTrain <- EXTRACTED[ID %% N %in% idx,]
dataTest  <- EXTRACTED[!(ID %% N %in% idx),]


Conv1D layer input and output Consider the following code for Conv1D layer
# The inputs are 128-length vectors with 10 timesteps, and the batch size
# is 4.
input_shape = (4, 10, 128)
x = tf.random.normal(input_shape)
y = tf.keras.layers.Conv1D(32, 3, activation='relu',input_shape=input_shape[1:])(x)
print(y.shape)
(4, 8, 32)

It has been given that there are 10 vectors, with each of length 128. Then how does the output will be of shape (8, 32)?
If we apply a filter of size 3, we will then get a vector of length 126, if stride is 1. But, I cannot see 126 anywhere in the output.
How to understand the shapes of input and output?

As described on the linked page, 128 is the dimensionality of each vector (i.e. the number of input channels), and 10 is the number of timesteps.
8 is the resulting number of timesteps after applying the filter of size 3 to the initial 10 timesteps.
In order to make it clearer, let's visualize the 1D convolution (source):

The missing parts in the picture are the batch size (4) and the output time length (8).


group the similar words array(['Ruby on Rails', 'Ruby', 'AWS DynamoDB', 'Python', 'MySQL',
         'Swift', 'Android', 'iOS', 'JavaScript', 'React Native', 'ReactJS',
         'TypeScript', 'Vue.js', 'Webpack', 'Amazon Web Services(AWS)',
         'Kubernetes', 'PHP', 'CI/CD', 'Java', 'C#', 'C', 'Node.js',
         'REST API', 'Go', 'Redux-Saga', 'Redux.js', 'Babel', 'GraphQL',
         'Tensorflow', 'PyTorch', 'Jenkins', 'Spring', 'Django', 'Git',
         'AWS EC2', 'CSS', 'HTML', 'MongoDB', 'Docker', 'Scala', 'SQL',
         'Embedded System', 'NLP', 'Apache', 'Kotlin', 'Angular', 'jQuery',
         'C++', 'RxJS', 'AngularJS', 'Redis', 'Next.js', 'NoSQL',
         'GCP(Google Cloud Platform)', 'Elasticsearch', 'OpenStack',
         'JPA(Java Persistent API)', 'TCP/IP', 'Objective-C', 'Realm',
         'Firebase', 'Ajax', 'Linux', 'PostgreSQL', 'ES6', 'AWS Lambda',
         'HTML5', 'AWS S3', 'GitHub', 'RxSwift', 'Terraform', 'AWS EKS',
         'AWS RDS', 'Microsoft Azure', 'Sass(SCSS)', 'CodeIgniter', 'Flask',
         'Nuxt.js', 'Ansible', 'Spring Boot', 'Linux kernel',
         'Apache Kafka', 'Deep Learning', 'Nginx', 'ActionScript', 'OOP',
         'Shell', 'gulp', 'Celery', 'SQLAlchemy', 'ExpressJS', 'RxJava',
         'Apache Spark', 'WebGL', 'OpenGL', 'Machine Learning',
         'MSSQL(Microsoft SQL Server)', 'Database', 'styled-components',
         'MVC', 'Retrofit', 'Machine Vision', 'Oracle', 'web3.js', 'R',
         'AWS ElasticBeanstalk', 'Elastic Stack', 'Laravel', 'ASP.NET',
         'Aurora DB', 'Redux-Observable', '.NET', 'AWS Backup',
         'AWS CloudWatch', 'Kibana', 'Fluentd', 'Logstash', 'JSP',
         'Bootstrap', 'Datadog', 'Rust', 'Azure', 'Apache Hadoop',
         'AWS X-Ray', 'Memcached', 'Jest', 'Mocha',
         'DRF(Django REST framework)', 'Spring Cloud', 'Data Analysys',
         'Big Data', 'GitLab', 'Gradle', 'SQLite', 'Microsoft IIS', 'Unity',
         'Electron', 'MariaDB', 'mSQL', 'gensim', 'Scikit-Learn',
         'AWS Simple Queue Service(AWS SQS)', 'gRPC',
         'Naver Cloud Platform', 'Ubuntu', 'Microservice Architecture',
         'Apache ActiveMQ', 'Oracle Database', 'Apache Subversion(SVN)',
         'Apache Tomcat', 'Red Hat Ceph Storage', 'Puppeteer', 'OpenLayers',
         'Vuex', 'Less.js', 'JIRA', 'Keras', 'NCP(Naver Cloud Platform)',
         'NestJS', 'PKI(Public key infrastructure)', 'AWS ECS', 'Hibernate',
         'UML', 'BitBucket', 'Arduino', 'Raspberry Pi', 'RabbitMQ',
         'Capistrano', 'Bamboo', 'MVP', 'OkHttp', 'Cocos2d', 'Ethereum',
         'Blockchain', 'DSP(Digital Signal Processing)', 'D3.js', 'Cocoa',
         'Axios', 'Ionic', 'WPF', 'AWS IAM', 'Shell Script',
         'Responsive Web', 'Canvas', 'ThreeJS', 'Apache ZooKeeper',
         'Pandas', 'Spring Batch', 'JUnit', 'Spring Data JPA', 'ASP',
         'Grunt', 'WordPress', 'MyBatis', 'AWS ElastiCache',
         'Apache HTTP Server', 'AWS Security Hub', 'Google API', 'Qt',
         'CAD', 'GatsbyJS', 'PostCSS', 'Socket.IO', 'Backbone.js',
         'Azure Linux Virtual Machines', 'Heroku', 'CUDA', 'IOCP', 'Unix',
         'CocoaPods', 'MVVM(Model-View-ViewModel)',
         'Google Firebase Crashlytics', 'Google Cloud Platform',
         'Windows kernel', 'OpenCV', 'Unreal Engine', 'Google Cloud SDK',
         'RxAndroid', 'Windows Embedded', 'Entity Framework', 'Packer',
         'Nexus', 'Consul', 'Selenium', 'Jekyll', 'XML',
         'Dependency Lookup', 'RxKotlin', 'Expo', 'Sketch', 'InVision',
         'Azure Text Analytics', 'Google Dialogflow',
         'Google Cloud Natural Language'], dtype=object)

let say I have this array list
if I want to group them by similar things
is there any pretrained language model to do this job easier?

for example
pytorch and tensorflow should be in one group because
most of the deep learning people are using pytorch or tensorflow

There will not be any pre-trained models to cluster these words.

In fact, in order to build your own clustering model you will need more metadata about each observation/word in your array.

At the moment any model would only be able to "see" the name of the package/software in your array. So the best you could hope for is a model that clusters these words based on their spellings.

Now let's pretend you find a brief description of each software, now you could do a bit more. With this longer text you could use supervised or unsupervised methods to cluster these softwares into groups (see topic models, k-means etc) based upon similar words in the descriptions.

Long story short, there is not a pre-built model to do this, and to build one yourself you're going to need more information about each observation.


How to tell if a problem should use regression or classification model? Once we know that the problem needs to be solved using supervised learning, how do we know if we have to use regression analysis or classification algorithm to solve it? Is there some thumb rules that can be used?

Most of the resources online give the standard example of prices of house for regression model and malignant/benign cancer plot for classification model. This isn't helping me much.

Is there come conceptual method to analyze the given problem?

A good rule of thumb is to look at the level of measurement of the target/response variable. If the response is measured on a nominal scale, the problem is a classification problem. Values on a nominal scale are for example labels of a categories where the categories have no natural order, like political parties in political science, species in biology, or parts-of-speech in grammar.

If the response is measured on a ratio or interval scale, you have a regression problem. Values on an interval scale are values where you can compare the degree of difference between values, but not the ratio between them, for instance temperature (on Farenheit or Celsius scales, but not Kelvin), or date values in a calendar. Values on ratio scales can be compared both with regards to degree of difference and ratio, like most physical quantities like mass, velocity or temperature on the Kelvin scale.

Ordinal scales are more difficult to place in either corner. I would generally say that you have a ranking problem with an ordinal response. However, the ranking problem can be approached using both classification, for instance using comparators, and and regression, like ordinal regression. Values on ordinal scales are ordered, or ranked, but you can't say anything meaningful about the degree of difference between any two values, for instance the ranking of racing drivers in a race.


Using Discretization from Training Set on Test Set in R I am currently discretizing my training set in R with discretize from the bnlearn package.

library(bnlearn)
discretize(train, method = "quantile", breaks = 2)

The lower bound of one interval and the upper bound of the other interval are the minimum and the maximum of the respective column.

Executing the same command on the test set

discretize(test, method = "quantile", breaks = 2)

will result in a different discretization, as the minimum and the maximum will likely be different on the test set. Is it possible in R (with another library or command) to transfer a discretization from a training set to a test set?

Apparantly this is easy to do in Weka and Orange, however, I would prefer to do this in R (not using RWeka).

You need to save the breaks/endpoints when you perform the bucketing/discretization on the training set as a (named) vector. The same breaks/endpoints can be then re-used on test set.

The I am giving below sample code using cut function in base R to keep the answer more generally applicable (rather than giving answer specific to bnlearn package). Note the way output of quantile function is saved in vector breaks_to_use in line 3 below and re-used while applying cut on test data. You can (hopefully) do the same using the breaks argument in discretize function.

training_data <- runif(100)
test_data <- runif(100)
breaks_to_use <- quantile(training_data, seq(0, 1, 0.25))
discretized_training_data <- cut(training_data, breaks = breaks_to_use)
discretized_test_data <- cut(test_data, breaks = breaks_to_use)


Significance of Number of Calls and Reset Call in Ball Tree Why does the Scikit Implementation has functions to reset and get number of calls?
How are these parameter important in Trees?
https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.BallTree.html#sklearn.neighbors.BallTree.reset_n_calls

Indeed, they don't seem to be used for anything.  Probably they were included for some initial debugging?
An issue (and a linked PR) discussing these methods:
https://github.com/scikit-learn/scikit-learn/issues/13330
(At time of posting,) searching for n_calls only finds results in _binary_tree.pxi, and none of them seem actually used for anything other than tracking the number of distances computed:
https://github.com/scikit-learn/scikit-learn/search?q=n_calls


Deep neural net modelling strategy are there any resources out there (book, blog, your own answer post etc.) that gives advise on modelling strategy of deep neural net?

I know how to fit a neural net, I know how to change settings like number of hidden layers, activation function, dropout etc. I know how to use cross-validation to validate models.

But what I need is advise on the actual modelling process. i.e. given a dataset (which has been cleaned and explored), where do you start? What type of neural net do you train first? How do you then tune it?

I cannot recommend highly enough this online book on neural networks.

The tricky part about neural networks is the stuff you mentioned -- what value to use for the learning rate, what topology to use for the network, etc etc -- we call these things hyperparameters to distinguish from parameters which are estimated by the optimization process.

This big section of the aforementioned book covers exactly that. I don't know if you'll find what you want though. The reason is because there is no silver bullet.


MCMC for finding Bayesian Neural Network Is someone familiar with such an approach:

Suppose I want to build a bayesian neural network, with distributions over my parameters instead of point estimates. First I train my network with standard backprop. After training I start some MCMC sampling over my network weights in order to estimate the distributions.

Somehow this approach seems very natural to me and probably there are already some papers or drawbacks, which I do not see. I am aware that sampling can be a slow process, but with a good starting point it should run quite fast. Also sampling could be run in batchmode in order to speed up.

There are in fact a bunch of papers on this topic. I'd recommend this one and this one. Not sure what you want to do with your Bayesian Neural Network, but maybe this one is useful too.


Best parameters to try while hyperparameter tuning in Decision Trees I want to post prune my decision tree as it is overfitting, I can do this using cost complexity pruning by adjusting ccp_alphas parameters however this does not seem very intuitive to me.
From my understanding there are some hyperparameters such as min_samples_split, max_depth, min_impurity_split, min_impurity_decrease that will prune my tree to reduce overfitting.
Since I am working with a larger dataset it takes a long time to train therefore don't want to just do trial-error.
What are some possible combinations of above mentioned hyperparameters that will prune my tree? + reasoning behind choosing particular combination will be helpful.
Thanks in advance!

There are no combinations that work for all cases, hyperparameter tuning is still something that is mostly done by trial and error. Things like Gridsearch and Randomsearch exist though.
A good start is always the default setting. An idea if performance is an issue is to tune on a small percentage of the training set to later switch to the full set.


Estimating the accuracy of a model? Given that I have a machine learning model.

I evaluated the model over several labeled datasets and acquired the accuracy (or any other metrics) for these datasets.

Now I receive a new dataset without labels. I run the model and got the prediction.

Is there any way to estimate the accuracy of my model on the new dataset?

I was thinking of measuring the similarity of the datasets (in the feature space), then based on the similarity to predict the accuracy of the new dataset. For instance, if the new dataset is not far from the old dataset, the accuracies might be similar.

Is there any way to estimate the accuracy of my model on the new dataset?

No.

If you want to evaluate the classification accuracy of your model, you need to know the ground truth to compare, simple as that. Estimating the similarity of the datasets is totally irrelevant, unless there is prior knowledge that feature similarity  leads to label similarity.


Cutting numbers into fixed buckets I am trying to put numeric data into fixed number of buckets using Python/R.

I have data in key:value format {1 : 12.3, 2 : 4.7, 3 : 7.4, 4 : 15.9, ......, 50 : 24.1}, which is device_id:data_usages I need to bucket based on value into nine buckets (1,5,25,50,150,250,1000,5000,10000), So later I can see which data points are in which bucket.

What function can do this in Python OR R?

You don't really need to implement an algorithm to achieve this. There are a few tools that will do this for you.

You can get the data assigned to buckets for further processing using Pandas, or simply count how many values fall into each bucket using NumPy.

Assign to buckets

You just need to create a Pandas DataFrame with your data and then call the handy cut function, which will put each value into a bucket/bin of your definition. From the documentation:

  Use cut when you need to segment and sort data values into bins.

In [1]: import pandas as pd
In [2]: import numpy as np    # to create dummy data

Create some dummy data, put it in a dataframe and define the bins:

In [3]: data = np.random.randint(low=1, high=10001, size=1000)
In [4]: df = pd.DataFrame(data=data, columns=["data"])
In [5]: bins = np.array([1,5,25,50,150,250,1000,5000,10000])

Pass the data, along with the bin definitions to the cut function and assign it back as a new column in the dataframe:

In [6]: df["bucket"] = pd.cut(df.data, bins)

You can then inspect the first few rows to see that the values have now been labelled with the relevant bucket:

In [7]: df.head()
Out[7]:
   data         bucket
0  8754  (5000, 10000]
1  2970   (1000, 5000]
2  6778  (5000, 10000]
3  2550   (1000, 5000]
4  5226  (5000, 10000]

Counting how many in each bucket

Here is an example using NumPy, to get an idea of the distribution, as a histogram.

Using the data and bins as defined above, we pass them to the numpy histogram function, which will count how many data points fall into each bin:

In [8]: np.histogram(data, bins)
Out[8]:
(array([  0,   2,   1,   8,   6,  61, 417, 505]),
 array([    1,     5,    25,    50,   150,   250,  1000,  5000, 10000]))

Where the first row tells you how many values fell into each bin, and the second row confirms the bins used.

You can get your dictionary of data into the same form as my dummy data above (into a numpy array) by doing this:

data = np.array([v for v in your_dict.values()])


No loss decay while learning Neural Network for X-OR operation in torch I have implemented a simple 1 hidden layer feed forward neural network in torch to learn X-OR operation. Below is my code:

require 'torch'
require 'nn'

m = nn.Sequential()
m:add(nn.Linear(2,2))
m:add(nn.Linear(2,1))
m:add(nn.Sigmoid())

torch.manualSeed(1)

m.modules[1].weights = torch.rand(2,2)
m.modules[2].weights = torch.rand(2,1)

--print(parax_m)

criterion = nn.BCECriterion()

inputs = torch.Tensor(4,2)
inputs[1][1] = 0
inputs[1][2] = 0

inputs[2][1] = 0
inputs[2][2] = 1

inputs[3][1] = 1
inputs[3][2] = 0

inputs[4][1] = 1
inputs[4][2] = 1

targets = torch.Tensor(4,1)
targets[1][1] = 0
targets[2][1] = 1
targets[3][1] = 1
targets[4][1] = 0

function trainEpoch(m,criterion,inputs,targets)
    for i=1,inputs:size(1) do
        local input = inputs[i]
        local target = targets[i]
        local output = m:forward(input)
        --print(output)
        local loss = criterion:forward(output,target)
        print(loss)

            -- backward
        local gradOutput = criterion:backward(output,target)
        m:zeroGradParameters()
        local gradInput = m:backward(input,gradOutput)
        --update
        --module:updateGradParameters(0.9) -- momentum (require dpnn)
        m:updateParameters(0.01) -- W = W -0.1*dL/dW
    end
end

for i=1,10000 do
    trainEpoch(m,criterion,inputs,targets)
end

-- prediciton
testinput = torch.Tensor(4,2)
testinput[1][1] = 0
testinput[1][2] = 0

testinput[2][1] = 0
testinput[2][2] = 1

testinput[3][1] = 1
testinput[3][2] = 0

testinput[4][1] = 1
testinput[4][2] = 1

for i=1,testinput:size(1) do
    local output = m:forward(testinput[i])
    print(output)
end

When I run the above code, there is no decay in loss(almost same in all iterations) so it does not predict the correct output. Can anyone help me to find the mistakes what I am doing wrong here?

I have also tried with different manual seed value, different initialisation of weights but still loss remains same in all iterations.

Finally I found the error in my Network.
1. I haven't added the non-linear layer after the first linear layer.
2. No randomisation while running stochastic gradient descent.

By updating these two things, now it is working fine.
Updated Code:

require 'torch'
require 'nn'

m = nn.Sequential()
m:add(nn.Linear(2,2))
m:add(nn.Tanh())
m:add(nn.Linear(2,1))
m:add(nn.Sigmoid())

--print(parax_m)

criterion = nn.BCECriterion()

inputs = torch.Tensor(4,2)
inputs[1][1] = 0
inputs[1][2] = 0

inputs[2][1] = 0
inputs[2][2] = 1

inputs[3][1] = 1
inputs[3][2] = 0

inputs[4][1] = 1
inputs[4][2] = 1

targets = torch.Tensor(4,1)
targets[1][1] = 0
targets[2][1] = 1
targets[3][1] = 1
targets[4][1] = 0

function trainEpoch(m,criterion,inputs,targets)
    for i=1,inputs:size(1) do
        local idx = math.random(1,4)
        local input = inputs[idx]
        local target = targets[idx]
        local output = m:forward(input)
        --print(output)
        local loss = criterion:forward(output,target)
        print(loss)

            -- backward
        local gradOutput = criterion:backward(output,target)
        m:zeroGradParameters()
        local gradInput = m:backward(input,gradOutput)
        --update
        --module:updateGradParameters(0.9) -- momentum (require dpnn)
        m:updateParameters(0.01) -- W = W -0.1*dL/dW
    end
end

for i=1,10000 do
    trainEpoch(m,criterion,inputs,targets)
end

-- prediciton
testinput = torch.Tensor(4,2)
testinput[1][1] = 0
testinput[1][2] = 0

testinput[2][1] = 0
testinput[2][2] = 1

testinput[3][1] = 1
testinput[3][2] = 0

testinput[4][1] = 1
testinput[4][2] = 1

for i=1,testinput:size(1) do
    local output = m:forward(testinput[i])
    print(output)
end


What are bias and variance in machine learning? I am studying machine learning, and I have encountered the concept of bias and variance. I am a university student and in the slides of my professor, the bias is defined as:
$bias = E[error_s(h)]-error_d(h)$
where $h$ is the hypotesis and $error_s(h)$ is the sample error and $error_d(h)$ is the true error. In particular, it says that we have bias when the training set and the test set are not independent.
After reading this, I was try to get a little deepr in the concept, so I searched on internet and found this video , where it defines the bias as the impossibility to capture the true relationship by a machine learning momdel.
I don't understand, are the two definition equal or the two type of bias are different?
together with this, I am also studying the concept of variance, and in the slides of my professor it is said that if I consider two different samples from the sample error may vary even if the model is unbiased, but in the video I posted it says that the variance is the difference in fits between training set and test set.
Also in this case the definitions are different, why?

What are Bias and Variance?
Let's start with some basic definitions:

Bias: it's the difference between average predictions and true values.
Variance: it's the variability of our predictions, i.e. how spread out your model predictions are.

They can be understood from this image:

(source)
What to do about bias and variance?
If your model suffers from a bias problem you should increase its power. For example, if the prediction of your neural network is not good enough, add more parameters, add a new layer making it deeper, etc.
If your model suffers from a variance problem instead, the best possible solution is coming from ensembling. Ensembles of Machine Learning models can significantly reduce the variance in your predictions.
The Bias-Variance tradeoff
If your model is underfitting, you have a bias problem, and you should make it more powerful. Once you made it more powerful though, it will likely start overfitting, a phenomenon associated with high variance. For that reason, you must always find the right tradeoff between fighting the bias and the variance of your Machine Learning models.

(source)
Learning how to do that is more an art than a science!


How to handle noisy data? I have X values such as

[0.2, 0.1, 0.3, 0.5, 20, -0.2, -0.1, -0.6, -0.8, -30]

Now I know that if the abs value is below a certain threshold, let's say 2, the y-value (-1 or 1) is pretty much random. But when the x-value is above that threshold, there is a good linear/non-linear relationship between X and y (so high positive numbers such as 20 would more often map to y-value of 1 and negative numbers such as -30 in this case would more often map to -1).

What is best practice to account for that? I'm using Keras/Tensorflow, but I'm rather looking for a general answer.

I see two options:

Find out that threshold somehow and train/test the model only on values above that threshold (but this means I lose data)
Generate probabilities in the prediction (e.g. for x-values that are not significant to the output, the output layer generates values close to 0.5, but for more significant values, the output would be closer towards 0 or 1)

What is the best practice to approach this?

Noisy data is meaningless data.

• It includes any data that cannot be understood and interpreted correctly by machines, such as unstructured text.

• Noisy data unnecessarily increases the amount of storage space required and can also adversely affect the results of any data mining analysis.

• Noisy data can be caused by faulty data collection instruments, human or computer errors occurring at data entry, data transmission errors, limited buffer size for coordinating synchronized data transfer, inconsistencies in naming conventions or data codes used and inconsistent formats for input fields( eg:date).

Noisy data can be handled by following the given procedures:

Binning:
• Binning methods smooth a sorted data value by consulting the values around it.

• The sorted values are distributed into a number of “buckets,” or bins.

• Because binning methods consult the values around it, they perform local smoothing.

• Similarly, smoothing by bin medianscan be employed, in which each bin value is replaced by the bin median.

• In smoothing by bin boundaries, the minimum and maximum values in a given bin are identified as the bin boundaries.

• Each bin value is then replaced by the closest boundary value.

• In general, the larger the width, the greater the effect of the smoothing.

• Alternatively, bins may be equal-width, where the interval range of values in each bin is constant.

• Binning is also used as a discretization technique.

Regression:
• Here data can be smoothed by fitting the data to a function.

• Linear regression involves finding the “best” line to fit two attributes, so that one attribute can be used to predict the other.

• Multiple linear regressionis an extension of linear regression, where more than two attributes are involved and the data are fit to a multidimensional surface.

Clustering:
• Outliers may be detected by clustering, where similar values are organized into groups, or “clusters.”

• Similarly, values that fall outside of the set of clusters may also be considered outliers.


Suggest DataSet for Sentiment Analysis on Dutch languages Can anybody provide any links for any training dataset for sentiment analysis on Dutch languages? I googled trying to find maybe movie review on Dutch but didn't find website with persons review like in IMDB


Back Propagation Using MATLAB I am new to neural networks. I tried coding the backpropogation alogrithm and tried running it on a test set which gave wrong results.
I have used the following knowledge to code it,

For the forward pass,
$$z^l = w^la^{l-1} + b^l$$
$$a^l = g^l (z^l)$$

For the backward pass, (Here $\circ\text{ - Element wise Product}$)

For the last layer,
$$\delta ^L = (a^l - y) \circ g'^L(z^L)$$
$$\frac{\partial L}{\partial w^L} = \delta^L (a^{L-1})^T$$
$$\frac{\partial L}{\partial b^L} = \delta^L$$

For the other layers,
$$\delta ^l = (w^{l+1})^T(\delta^{l+1}) \circ g'^l(z^l)$$
$$\frac{\partial L}{\partial w^l} = \delta^l (a^{l-1})^T$$
$$\frac{\partial L}{\partial b^l} = \delta^l$$

For the Update,
$$W : = W - \alpha \frac{\partial L}{\partial w^l}  $$
$$b : = b - \alpha\frac{\partial L}{\partial b^l} $$

The following is the code which I have written,

    X = load('iris.csv');
    W1 = rand(3,4);
    W2 = rand(1,3);
    b1 = rand(3,1);
    b2 = rand(1,1);
    y = [ones(1,50) 2*ones(1,50) 3*ones(1,50)]';

    for j = 1:100    %for epoch
    for i = 1:150    %for iteration through dataset
    x1 = X(i,:)';
    z1 = W1*x1 + b1;
    a1 = tanh(z1);
    z2 = W2*a1 + b2;
    a2 = relu(z2);

    dz2 = (a2 - y(i)).*reluGradient(a2);  %this is delta L
    dw2 = dz2*transpose(a1);              %The derivative term of Lth
    db2 = dz2;                            %for the bias

    g1 = 1 - a1.^2;                       %Derivative
    dz1 = transpose(W2)*dz2 .* g1;        %FOr delta l
    dw1 = dz1*transpose(x1);              %derivative term for lth layer
    db1 = dz1;                            %bias update

    W1 = W1 - 0.1*dw1;
    b1 = b1 - 0.1*db1;

    W2 = W2 - 0.1*dw2;
    b2 = b2 - 0.1*db2;

    end
    end

I am trying to train the net for the iris data set (150 X 4 - dataset Size).
I have considered 4 input units, 1 hidden layer with 3 hidden units and 1 output unit. Hence the dimension of the weight matrix for first layer is 3 X 4 and for the last layer is 1 X 3. When I try to test the network I always get the input classified to class 3. I tried changing the hyper parameters, but it seems there is something wrong with the code.

In the code , I first load the CSV file, and then initialize the weight matrices accordingly. I have run two for loops one for the epoch and other for the iteration. I do a forward pass first using the above equations then a backward pass. I have made functions for the RELU and RELU_GRADIENT.

For relu:

function g = relu(z)
g = max(0,z);
end

For relu gradient:

function z = reluGradient(z)
z(z>=0) = 1;
z(z<0) = 0;
end

I would be obliged if someone can direct to me to the solution to this problem

About IRIS.csv:

Has four input features and 150 data samples.
50 of each class. Hence there are 3 classes.

Updates:

The cost function here I have used is the quadratic cost function:

$$C = \Sigma (a^L - y)^2 $$

Hence all the derivatives have been calculated with respect to that cost function. For example for the last layer,

$$\frac{\partial C}{\partial z} = (a^L - y) \circ g'^L(z^L)$$ which is $\delta^L$

Also,

I have tried using the sigmoidal transfer function instead of the relu transfer function with all the proper changes at all derivatives,weight and output vectors. (The output vector was then of (3,1) dim)
As 3 classes hence - $[1,0,0]^T$ , $[0,1,0]^T$ and $[0,0,1]^T$

Thanks

Your problem is a classification problem with 3 classes. The appropriate function for it is softmax function, whereas you are using Relu. Another thing is that you need a cost function. The function that you optimize and the one you do calculate derivatives of with respect to parameters.

The cost function and last layer activation function must match each other. So first you choose appropriate "last" activation funtion for your problem and then cost function.

For iris you can use softmax function with cross entropy function. The derivatives of these you can easily find. :)

Another thing are the dimensions. The result of softmax function is the vector of the size (number of classes, 1) for each observation whereas you have (1,3) dimensions.

Regarding the backpropagation algorithm for the other layers it is looks ok, but the last layer equation is wrong and should be like the one below:

where C is the cost function and we calculate derivative of C with respect to a (activation of last layer) and multiply element-wise by derivative of a (here it should be softmax function with respect to z). dw and db for last layers have correct equation.

I hope this hints will help you come up with final neural network implementation for iris classification problem.


How to use LSTM for time series data? I've an ECG data spread over time. The duration for each data is around 3 minutes (approx 180 seconds).

Each second around 200 recordings were taken. So total length for each sample is approx 36000.

Now how to use this data for lstm. My purpose is to make an encoder-decoder network to encode this data into small size. I'm planning to use reconstruction error for loss function.

If you are going to use the ECG time series as the input for the LSTM-network you will need to reshape your data

data = data.reshape(signal length,time step pr sample, ECG channel)

So if your data is 36000 samples long and you only use 1 channel you will do like this:

data = data.reshape(36000,1,1)

Then you can make a simple LSTM model like this:

model = Sequential()
model.add(InputLayer(input_shape=(1, 1)))
model.add(LSTM(32,activation='tanh', return_sequences=True))
model.add(Dense(1, activation='sigmoid'))

I think the same principals will apply to your encoder/decoder


How do i interpret this correlation Does this mean that as long as the student has good gpa and good gre even though his Alma Mater's prestige is low - he will get admitted in a college

Any additional things i can interpret from below ?

No. From this correlation matrix you cannot draw the conclusion that

  as long as the student has good gpa and good gre even though his Alma Mater's prestige is low - he will get admitted in a college

The reason is that correlation is a measure of association between single pairs of variables. The conclusion you draw above - on the contrary - is based on a combination of three different variables plus the outcome variable.

If you want to get an estimation of the probability that a student will be admitted to college based on her gpa, gre and prestige the right way is to create a logistic regression model. Here's an example in R (provided that admit is a binary variable, with admit=1 indicating that the student is admitted and admit=0 that she is not admitted)

model <- glm(admit ~.,family=binomial(link='logit'),data=data)

With this fitted model you can then compute the probability that a student is admitted given her particular combination of gpa, gre and prestige.


Supervised learning on sources of information with different importance I am trying to classify customer support sessions using supervised machine learning.

In each customer support session I have 3 bags of information.
1. The title of the customer's complaint
2. Information about the device the customer was using
3. Text of the chat session with the customer support agent

In each customer support session, there are 6 different classes. Is it better to:
1. Train a classifier on each bag of information and have them vote on which class the session belongs to?
2. Put all of the information into a single set of features, and train a single classifier to determine which class the session belongs to?
3. Other?

Its better to do supervised classification then unsupervised clustering since you have the right labels for the response.

Now in supervised classification approach 1 is better than approach 2 because in approach 2 you will the reducing the already available information by combining features.

Approach 3 is advised where you will create more features from the given 3 features by use of NLP processing, sentiments, tags, chunking, time taken, text length etc and then selecting the features those provide more information gain using feature selection approaches.


How to include user features in a recommender system? I'm novice in that matter but I was thinking about the formulation of a recommender system. Let's take the example of a movie recommendation system. We have a column dedicated to movies ID (or names), a matrix related to the rates the users gave to each movie, and a matrix with movie features (romance, drama, etc..)- joined a photo that shows this formulation. What about I would like to use users features to improve my recommendation? If I had information like age, profession, revenue of each user I would like to use it in my formulation. But if I include user features, this formulation is not Content-‐based neither Collaborative filtering anymore. Anyone knows what kind of formulation it can assume?

This is referred to as side information. This is used to enhance the recommender system.
A good library for collaborative filtering (and beginner friendly) is turicreate. Have a look at this link. o summarise, the traditional, basic matrix factorisation will encode user i and items j respectively as vectors $u_i$ and $v_j$ so that the predicted score that a user would give to the unseen item is:
$$
score(i,j) = u_i^T  v_j
$$
but you can have a more complex model that will also take into account idiosyncratic characteristics of both items and users:
$$
score(i,j) = u_i^T v_j + a^T x_i + b^T y_j
$$
which increases the capacity of the model.


What is the best references / books to get familiar with machine learning algorithms? I'm just starting off as a data scientist and i need to understand how regression, random forest and much more.. algorithms function under the hood, the maths behind them and when to use them. I started with some MOOC courses but not a fan, it's just not my style of learning as i need practical examples or books i can dive into with detailed examples i can just apply myself after understanding how each algorithm works and some 20 mins vids with scarce maths explanation is not what i aim for, i've done some research and ended up with these books:

-python for data analysis

-Python Crash Course

-oreilly hands on machine learning with scikit learn and tensorflow

-automate the boring stuff with python

kindly note that i have some python experience(less than a year) and  unfamiliar with tensorflow , keras, scikit-learn etc..

Kindly also note that i didn't mention any maths books to explain how the machine learning algorithms work so i'm open to suggestions  .

If you want to get more into Machine Learning and the mathematical/theoritical aspects that it includes, the book mentioned below is an option:

Machine Learning by Tom Mitchell, McGraw Hill, 1997.


Can absolute or relative contributions from X be calculated for a multiplicative model? $\log{ y}$ ~ $\log {x_1} + \log{x_2}$ (How) can absolute or relative contributions be calculated for a multiplicative (log-log) model?
Relative contributions from a linear (additive) model
E.g., there are 3 contributors to $y$ (given by the three additive terms):
$$y = \beta_1 x_{1} + \beta_2 x_{2} + \alpha$$
In this case, I would interpret the absolute contribution of $x_1$ to $y$ to be $\beta_1 x_{1}$, and the relative contribution of $x_1$ to $y$ to be:
$$\frac{\beta_1 x_{1}}{y}$$
(assuming everything is positive)
Relative contributions from a log-log (multiplicative) model
In log-space, a model could take the following form:
$$\log{y} = \beta_1 \log{x_{1}} + \beta_2 \log{x_{2}} + \alpha$$
which in the 'real-world', assumes the following form:
$$y =  e^\alpha x_{1}^{\beta_1}x_{2}^{\beta_2}$$
But (how) can absolute or relative contributions be calculated from such a multiplicative (log-log) model? i.e., (how) can we calculate how much $x_1$ contributes to $y$? For example if $e^\alpha=10$, and $x_1^{\beta_1} = 100$ and $x_2^{\beta_2} = 1000$, then $y = 10^6$, but what portion of that 1,000,000 was contributed to by $x_1$?

One way I can think of is the following (beware that relative importance can mean different things and be counted in different ways):
The relative contribution of log-log model, seen as a linear model, is found by setting everything else to zero and taking the ratio:
Sp:
$$1/r_{x_1} = \frac{\beta_1 \log x_{1}}{\log y}$$
or:
$$\log y \approx r_{x_1} \beta_1 \log x_1$$
so the relative contribution of the multiplicative model can be derived from:
$$y \approx  x_1^{r_{x_1} \beta_1}$$
$$m_{x_1} = x_1^{r_{x_1} \beta_1}/y$$
One can even use the original variable and its exponent as:
$$m_{x_1} = x_1^{\beta_1}/y$$
Still another way to quantify importance for multiplicative models (inspired by geometric means) is:
Given $y = ax_1^{\beta_1}x_2^{\beta_2} \cdots x_n^{\beta_n}$, then:
$$m_{x_k} = \frac{x_k}{\sqrt[\beta_1 + \beta_2 \cdots + \beta_n]{y}}$$
So if $y = x^2$ then $m_x = 1$ since $x$ has $100$% importance on $y$.

Is there a standard way to make the relative contributions add to
unity? Or alternatively, to make the absolute contributions to add to
y?

When one wants to determine the percent a certain variable affects another variable, one uses a ratio between the two variables. Only for linear additive models do the percents add up to the original value. For multiplicative models these two properties of relative importance diverge and require different approaches (what I explicitly mentioned in the beginning).
So a percent of $x_k$ over $y$ is always some ratio like $\frac{x_k}{y}$ on the other hand for multiplicative models these ratios no longer can be added and don't add up. So one will have to modify their requirements, for example instead of adding up to original value, maybe the product equals the original value.
In this sense one can do a variation of the above proposals (eg the geometric mean inspired formula) and use:
$$y = x_1^{\beta_1} \cdots x_n^{\beta_n}$$
$$m_{x_k} = \frac{x_k^{\beta_k}}{y^{\frac{\beta_k}{\beta_1+\cdots+\beta_n}}}$$
With this definition of relative importance one still uses a ratio (so percent of importance of some value on some other value) while at the same time the product of all $m_{x_k}$ equals $1$, or:
$$m_{x_1} \cdots m_{x_n} = 1$$
Some references which follow another route:

Looking for relative contribution of log and standard variables

How do you find contribution of an independent variable in the given log-log regression model?

Measures of relative importance:

Quantifying the Relative Importance of Predictors in Multiple Linear Regression Analyses for Public Health Studies
Quantifying relative importance: computing standardized effects in models with binary outcomes
Relative Importance Analysis: A Better Way to Communicate Multiple Regression Results
Investigating the Measures of Relative Importance in Marketing Research


what is correct way to perform normalization on data in Auto encoder? working on anomaly detection problem. i'm using auto-encoder to denoise given input. I trained network with normal data(anomaly free). so model predict normal state of given input. Normalization of input is essential for my dataset.

problem with normalization is that when noise value is very high compare to entire dataset. then prediction follows noise. for example if I add  noise (delta=300) to 80% of the data and perform normalization on the dataset which mean value is 250 and standard deviation is 79. here noisy data points(80% of the total dataset) are greater than 300. after normalization, I feed this dataset to the model, prediction follows the noise and giving wrong output. this happens because of feature scaling. when I add noise to most of data points, model consider this points as normal data points and rest of as anomalous data points.

In inverse scaling process, I can not use min-max values of my input to perform inverse scaling on prediction otherwise its follow noise in dataset.

so what is right way to perform feature scaling in denoise kind of problem?

Min-Max-Scaling will not perform well for your problem, as you already said. For noisy data, scaling according to the quantile range should perform better.

After scaling, you could also try to clip your data to $[-1,1]$, as it it is often done in adversarial learning methods.


Machine Learning based Multivariate Time Series Prediction - How to create supervised data format Q1:

I have a multivariate time series dataset. For each timestep, there are 11 features and 1 output. I am going to use supervised ML to predect the output. I understand that in univariate cases, if I am going to use the past 3 days to predict the t-th day, the dataset will be formatted as

x(t-3) | x(t-2) | x(t-1) | x(t)

, where x(t) is the output to predict.

How should I format the dataset when it is a multivariate problem?

I saw that in some kernels, the problem is formatted as

x12(t-3) | x12(t-2) | x12(t-1) | x1(t), x2(t), ..., x12(t)

, where x12(t) is the output to predict.

In this case, variables x1 to x11 for the past 3 days are ignored.

However, these variables may be important in my case. Can I format the problem into

x1(t-3),...,x12(t-3) | x1(t-2), ..., x12(t-2) | x1(t-1),..., x12(t-1) | x1(t), x2(t), ..., x12(t)

?

(some of the features are just day, month, day of week, etc. created from the datetime index)

Q2:

With only 11 features, is it necessary to conduct feature selection?

Regarding how to handle multivariate time series problem, I believe that GitHub link Timeseries multivariate will be helpful.

You have to change n_inputs and n_outputs as 12 and 1 respectively.


Normal equation result simplification The derivation of the normal equation can be noted
$\theta  = (X^TX)^{-1}(X^T)y$, where $X^{-1}$ is the inverse of $X$ and can also be written $inv(X)$.

But why can't we write $inv(X^T X)$ as $inv(X)inv(X^T)$, and also use the definition of inverse $inv(X^T)(X^T) = I$?

Then $\theta  = inv(X^T X)(X^T)y = inv(X)inv(X^T)(X^T)y = inv(X)Iy = inv(X)y$

Why isn't the final result of normal equation in regression in machine learning written as  $\theta = inv(X) y$ after simplifying?

Only square matrices have an inverse, and neither $X$ nor $X^T$ here are (necessarily) square. I guess that's the most direct answer. $X$ may have a left-inverse $A$, such that $AX = I$. In that case, if $X\theta = y$ then $AX\theta = Ay$ so $\theta = Ay$.

The direct way to construct that $A$ is just where you started though; it's $(X^TX)^{-1}X^T$. The Gramian matrix $X^TX$ is (square, and) invertible if the features (columns) of $X$ are linearly independent, and, $X^TX$ is more importantly relatively quite small and so it's feasible to invert.

You normally wouldn't compute the inverse anyway, but try to solve the system $X^TX\theta = X^Ty$ with QR decomposition or something, but that's not what you're asking.