Loss function that penalizes under-predictions more than over-predictions? I have a regression problem where I am predicting a continuous variable. Loss functions used most often in these cases (RMSE, MAE, etc.) don't treat over- or under- predictions differently.

I am in a scenario where under-predicting would be a much worse outcome than over-predicting.

What type of loss function would appropriately capture this?

Pick an asymmetric loss function. One option is quantile regression (linear but with different slopes for positive and negative errors).


Policy gradient: why does this converge with Adam and not SGD? I am looking into policy gradient methods. I stumbled into this implementation:
https://gist.github.com/calclavia/cfcd41ad4e47d7b9b6ab8af15410747a
It uses a Nesterov Adam optimizer.

If I run it, it converges and gets good scores on OpenAI Gym's CartPole-v0.

However, if I change the optimizer from Adam to stochastic gradient descent (SGD), it never converges and seems to act randomly. Why is this? Is there something about policy gradient methods which make SGD a poor choice?

NOTE: there is a bug in that code which only runs the episode for 100 time steps. The episode can run for up to 200 time steps. I fixed this when running it.

Stochastic gradient descent (SGD) can get stuck in saddle points which results in the model not converging. A saddle points gradient is where zero in many directions but not all directions.
Adam adaptively changes the learning rate based on past gradients. This additional information can allow the model to escape saddle points, thus the model is more likely to converge.


Getting parameters of the best model with crossvalidation in with SparkMLLib I am having trouble accessing the parameters of estimators of model in SparkMLlib. More precisely my problem is:
I have a logistic regression model for which I want to find the best regularization parameters (regParam and elasticNetParam). To do that, I use the CrossValidator which works and finds me a model better than all the other one I tried. The issue is that I do not know how to access the actual value of the parameters that were found by the cross validator. Below is the code I use to fit my cross validator:

from pyspark.ml.evaluation import BinaryClassificationEvaluator
from pyspark.ml.tuning import CrossValidator, ParamGridBuilder
from pyspark.ml.classification import LogisticRegression

lr_predictor = LogisticRegression(featuresCol='polyFeatures', labelCol='label', maxIter=10)
paramGrid = ParamGridBuilder() \
    .addGrid(lr_predictor.elasticNetParam, [0., 0.5, 1]) \
    .addGrid(lr_predictor.regParam, [0.1, 0.01]) \
    .build()
crossval = CrossValidator(estimator=LogRegPipeline,
                          estimatorParamMaps=paramGrid,
                          evaluator=BinaryClassificationEvaluator(),
                          numFolds=2)
cvModel = crossval.fit(train_set)
bestModel = cvModel.bestModel
# How to get the best parameters fitted by cvModel

It's not the most elegant solution but you can use the following to at least zip together evaluation metrics and hyper parameters

zip(cvModel.avgMetrics, paramGrid)


cost-complexity-pruning-path with pipeline I'm using Kaggle's titanic set. I'm using pieplines and I'm trying to prune my decision tree and for that I want the cost_complexity_pruning_path. The last line of code produces the error:
   ValueError: could not convert string to float: 'male'
Do you know what I'm doing wrong? I have looked at Sklearn: applying cost complexity pruning along with pipeline but that doesn't seem to be helping in my case
cat_vars = ['Sex','Embarked']
num_vars = ['Age']

num_pipe = Pipeline([('imputer', SimpleImputer(strategy='mean')),('std_scaler', StandardScaler())])
cat_pipe = Pipeline([('imputer', SimpleImputer(strategy='most_frequent')),('ohe', OneHotEncoder())])

col_trans = ColumnTransformer([('numerical', num_pipe, num_vars),('categorical', cat_pipe, cat_vars)] ,remainder='passthrough')

final_pipe = Pipeline([('column_trans', col_trans), ('tree', DecisionTreeClassifier(random_state=42))])
final_pipe.fit(X_train, y_train)

path = final_pipe.steps[1][1].cost_complexity_pruning_path(X_train, y_train)

Because cost_complexity_pruning_path refits the tree model on the data you provide before doing the pruning (source), you need to preprocess the data first.  So this should do it:
X_preproc = final_pipe[:-1].transform(X_train)
path = final_pipe.steps[-1][1].cost_complexity_pruning_path(X_preproc, y_train)


Principal Component Analysis and abnormal data I know that PCA is good in differentiating between anomalies and normal data and it helps to differentiate between them when it tries to transfer the data to another dimension. I mean it can somehow maximize the separation between the regular and irregular data points. I also have seen this in my codes. However, someone has asked me to prove it! by a simple detailed example (e.g, a simple 2*2 matrix)? Can anyone help?

Thanks in advance.

The first eigenvector of the PCA have the direction of the maximum variance of the data. Then the second follows with the direction of the next highest variance and so it continues....

The easiest way to think about this, by my experience, is with 2D data and transform it to 1D. Visualization can be found here: PCA Visualization

It does not maximize the separation between the regular and irregular data points. It does dimension reduction, by trying to explain as much of the variance as possible by creating a new basis, eigenvectors(directions).
However, it is not very robust to outliers. Imagine all your datapoints are basically on a line, now suddenly you receive an abnormal value. As this abnormal value will have an impact on the variance, the eigenvalue directions will change, and with that the values of the eigenvectors!

For anomaly detection:
You can use the PCA eigenvectors of "normal data" against thos of new data. Simply do distance calculations between the eigenvectors. If you have a big distance => Anomaly

There are methods such as robust PCA that better cope with noisy data and can find the outliers.

Hope this helps.


How much of a background in programming is necessary to become a data scientist? I once knew some Java, but that was close to 10 years ago.  Assuming I can learn a language to get into data analytics.... what language do you recommend?

First of all, the fact that you have known some Java, even ten years ago, already means that you don't "know nothing about programming" (I suggest you update the title of your question to reflect that - change "nothing" to "a little"). I'd like to make several points, which I hope will be useful to you.

In terms of the level of programming proficiency, which is expected (needed) for a data scientist, the following popular definition says it all:

  A data scientist is someone who is better at statistics than any
  software engineer and better at software engineering than any
  statistician.

Another perspective on the role of programming abilities in a data scientist's skill set can be found in a popular visual representation of data science, using Venn diagrams. The original data science Venn diagram was presented by data scientist Drew Conway (see this blog post):

Since its original introduction, the original diagram was modified by various people for various reasons. The two interesting adaptations are for data science in the social sciences domain (http://www.datascienceassn.org/content/fourth-bubble-data-science-venn-diagram-social-sciences), as well as data science Venn diagram V2.0, where data science is represented not as an intersection of knowledge domains, but as their union (http://www.anlytcs.com/2014/01/data-science-venn-diagram-v20.html). Another very interesting and useful visual perspective of data science skill set, also based on Venn diagram, is the following Gartner's diagram, mapping specific skills to business intelligence (BI) or business analytics knowledge domains:

An alternative perspective for a data scientist's skill set and domain knowledge is a taxonomy of data scientists, such as this taxonomy, which classifies data scientists, according to their focus (or the strongest skill set): mathematics, data engineering, machine learning, business, software engineering, visualization, spacial data (GIS) or others.
If you're curious about the meaning of the "Danger Zone" in the original data science Venn diagram, this Quora discussion, containing, among other nice answers, also an answer by the original diagram's author, can be very helpful.
If you're interested in learning about a range of skills and knowledge domains, useful for a data scientist, check this open source curriculum for learning data science: http://datasciencemasters.org, or on GitHub: https://github.com/datasciencemasters/go. Of course, popular and research papers, lectures on YouTube, MOOC courses, online and offline bootcamps as well as a wealth of other resources is only an Internet search away.
Finally, a note on programming languages for data science. I think that it is important to understand that this aspect is really of secondary importance. The focus should be on two words, which the term "data science" consists of: data and science. Focus on data means that it is important to think about data science (or BI, or analytics) tasks in terms of the corresponding domain knowledge as well as to pay attention to data quality and representativeness. Focus on science means adhering to scientific approaches to data collection and analysis, of which reproducibility plays an important role. A programming language for data science is just a tool and, therefore, should be chosen to match the task at hand. Python and R represent very good and the most popular programming languages and environments for a data scientist, however, you should be aware of other options (tool set).


In image classification/recognition how are images coded? I am trying to understand how images are coded for classification/recognition.

Suppose that I have images that contain animals (e.g. dogs, cats, birds) -- does each image have a label for that animal or does the label only apply to the portion of the image with the animal in it? If the entire images are coded does it take a classifier longer to learn?

This will depend on the type of task you want to perform:
Object recognition is a wide task than can be approached in different ways such as:

[Multi-part OR][1]
[Multi-scale OR][2]
Multi-class OR
[Multi-label OR][4]
[Multi-modal OR]

Each approach (and its combinations) uses different representations for feature vectors extracted from the images. Most typical representations try to be invariant to certain conditions on the images depending on it purpose, such as scale, rotation and illumination changes.
Some successful representations are:

Bag of Visual Words [5]
Histogram of visual words [8]
Haar-like features [2]
Statistical-based structures (Active shape models, active appearance models) [9][10]

With the rise of deep networks, surprising results are being obtained feeding the images just as the raw pixels that represent them [6]. However, you will likely need hundreds of thousands of images to proper train a network that can abstract features to further recognize objects. You can actually take a look to the highly successful VGG Convolutional Network implementation at [Find actual links here][2]

The previous approaches will typically required that you have each example labeled with target value (cat, dog, etc). In case of deep learning you are being able to learn first representation of images with unsupervised data (not labeled) and the being able to train a classifier on top of the network in a supervised way (with labeled data)

You may find most of the references below at: [Google Scholar][3]

[1]: [Xin Chen and Shen(2009)] Xiaohua Hu Xin Chen and Xiajiong Shen.
Spatial weighting for bag-of-visual-words and its application in
content-based image retrieval. Advances in Knowledge Discovery
and Data Mining, 2009

[2]: [Viola and Jones(2004)] Paul Viola and Michael J. Jones. Robust realtime
face detection. Int. J. Comput. Vision, 57(2):137–154, May 2004.
ISSN 0920-5691.

[4]: [Jia Deng and Fei-Fei(2014)] Jonathan Krause Michael Bernstein
Alex Berg Jia Deng, Olga Russakovsky and Li Fei-Fei. Scalable
multi-label annotation. ACM Conference on Human Factors in
Computing Systems, 2014.

[5]: [Xin Chen and Shen(2009)] Xiaohua Hu Xin Chen and Xiajiong Shen.
Spatial weighting for bag-of-visual-words and its application in
content-based image retrieval. Advances in Knowledge Discovery
and Data Mining, 2009.

[7]: [Krizhevsky et al.(2012)Krizhevsky, Sutskever, and Hinton] Alex
Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet
classification with deep convolutional neural networks. In
P. Bartlett, F.c.n. Pereira, C.j.c. Burges, L. Bottou, and K.q.
Weinberger, editors, Advances in Neural Information Processing
Systems 25, pages 1106–1114. 2012.

[8]: [Lowe(2004)] David G. Lowe. Distinctive image features from scaleinvariant
keypoints. Int. J. Comput. Vision, 60(2):91–110, November
2004.

[9]: [van Ginneken(2002)] A.F. Staal J.J. ter Haar Romeny B.M. van Ginneken,
B.Frangi. Active shape model segmentation with optimal
features. Medical Imaging, IEEE Transactions on. Volume 21, 2002.

[10]: [Matthews and Baker(2004)] Iain Matthews and Simon Baker. Active
appearance models revisited. International Journal of Computer
Vision. Volume 60., 2004.


Techniques to manage high dynamic range in neural network regression? I have a problem where I take a noisy sequence in (in one domain) and generate a clean sequence out, in another domain.  The issue I'm having is that when the input data is relatively clean, the output can be generated with very high numerically accuracy, approximately 1e-6.  At some points in some sequences though, the input data is too noisy to do very well and the best possible error may be on the order of 1.0 to 10.0.
My problem is that while training using typical loss functions like MSE, the network is understandably "obsessed" with the high errors, with the result being that at no time does it achieve the very high accuracy obtainable in most cases.
Are there special loss functions or other techniques that can encourage the network to  go for highly accurate results when possible without completely punting on the hard cases?  I've played around with loss functions that only score the best X% of a batch, but so far it hasn't really helped much, and I still would prefer it to do the best it can on the hard cases.

You may try cycle GANs.
In normal Generative Adversarial Networks (GANs), a generative network -the generator- is trained to generate images with an auxiliary network, the discriminator. The discriminator learns to tell apart data generated by the generator (i.e. fake data) from real data, while the generator learns to generate data that fools the discriminator. GANs are known to generate good quality data where MSE and the likes generate "blurry" data that lack detail.
Cycle GANs are two coupled GANs, where the generators learn to "convert" the input data from domain A to domain B, and vice versa, plus an extra loss term to ensure cyclic consistency (i.e. generator1(generator2(a)) = a).
The results are very impressive. Here you can see cycle GANs trained to convert images, changing horses into zebras and vice versa, winter landscape into summer, etc:

If your data is continuous and your architecture is fully convolutional, then cycleGANs may actually work in your case.


Is an $F_1$ score of 0.1 always bad? I'm currently building a model to predict early mortgage delinquency (60+ days delinquent within 2 years of origination) for loans originating in 2018Q1. I will eventually train out-of-time (on loans originating in 2015Q4), but for now I'm just doing in-time training (training & testing on 2018Q1) -- and even this I've found challenging. The dataset contains ~400k observations, of which ~99% are non-delinquent and ~1% are delinquent. My idea so far has been to use precision, recall, and $F_1$ as performance metrics.
I am working in Python. Things I've tried:

Models: logistic regression & random forest.
Model selection: GridSearchCV to tune hyperparameters with $F_1$ scoring (results were not significantly different when optimizing for log-loss, ROC-AUC, Cohen's Kappa).
Handing imbalanced data: I tried random undersampling with various ratios and settled on a ratio of ~0.2. I also tried messing with the class weights parameter.

Unfortunately, my validation & testing $F_1$ scores are only around 0.1, (precision & recall are usually both close to 0.1). This seems very poor, since with many problems you can achieve $F_1$ scores of 0.9+. At the same time I've heard there's no such thing as a "good $F_1$" range, i.e. it is task-dependent. Indeed, a dummy classifier which predicts proportional to the class frequencies only achieves precision, recall, and $F_1$ of 0.01.
I've tried to find references on what a "good" score for this type of task is, but I can't seem to find much. Others' often report ROC-AUC or Brier Score, but I think these are hard to interpret in terms of business value added. Some report $F_1$ but see overly optimistic results due to data leakage or reporting testing performance on undersampled data. Finally, I've seen some people weight confusion matrix results by expected business costs as opposed to reporting $F_1$, which seems like it may be a better route.
My questions are: (1) is an $F_1$ score of 0.1 always bad?, (2) does it even make sense to optimize for $F_1$ or should I used another metric?, (3) if $F_1$ is appropriate and a score of 0.1 is bad, how might I improve my performance?

From a credit scoring point of view : a $F_1$ score of $0.1$ seems pretty bad but not impossible with an unbalanced data-set. It might be enough for your needs (once you weight your errors by the cost). And it might not be possible to go higher (not enough data to predict an event that appears random). In credit scoring there is always a 'random' part in the target (sudden death, divorce ...) depending on the population and the goal of the loans.

You might want to investigate your features and your target. Basically : statistically, on an univariate approach, do you have features that appears predictive of the target ? (Age of the person ? revenue ? purpose of the loan ?). You might also need to investigate the target : do you have some questionnaire that would allow to get an insight on why the person defaulted ? (If the majority of default come from random event, you might not be able to modelise it).

The main problem with $F_1$ score in credit scoring is not data imbalance, but cost imbalance. Type I and Type II errors have far differents consequences. Given that you already gave the loans I am not even sure there is a cost associated with false positive (saying someone will default when it won't). It might be interesting to weight precision and recall (i.e. use $F_\beta$ as defined here). Another problem is that it is usually good for a binary decision. Depending on what you want to use the model for (measuring risk of already granted loans ? granting new loans ? pricing new loans ?) there might be alternatives that better capture model discrimination (AUC - see its statistical interpretation) or individual % chance of default (Brier Score).

Assuming that there is no specific problem with your current modelling (Feature engineering, imbalance treatment, 'power' of your model). There are some credit-scoring specific things you can do. Work on your target definition (what if you do 90+ days delinquant in the 5 years after origination ?). Try to collect more data about your clients and their behavior (purpose of the loan, others products they use at your bank... etc.).


Terminology - regression with one output and multiple output variables I am trying to predict the response when the input is represented by Fourier transform. These form the features and are typically represented as a vector, $x_1,x_2,...,x_d$ where $d$ is the length of the fourier transform. Based on my understanding each such $d$ dimensional vector can be an input to a regression model. The output is $y_1$ which is a scalar real-valued number and there is another output $y_2$ denoting another scalar real-valued number. These are the dependent variables. I have $N$ number of $d$ dimensional examples of fourier transform each labelled by $y_1$ and $y_2$.

Question 1) When the task is of predicting only one output response using the input fourier transform then is the problem termed univariate regression? Is univariate associated by the input's dimension(which is d>1) or the output's dimension (which is 1)

Question 2) When the task is of jointly predicting the two response variables - $y_1$ and $y_2$ then is the problem termed multivariate regression?

Question 1. Both. If you think in opposite to multivariate terms, than in univariate regression both input and output variables should be 1-d

Question 2. Multivariate regression where more than one independent variable (predictors) and more than one dependent variable (responses), are linearly related. So input needs to be more than 2 also.


Upper bound on size of sample set for decision trees Say I have an instance space with 4 features and I know that a decision tree with 8 nodes can represent the target function I want to learn. I want to give an upper bound on the size of the sample set needed in order to achieve a true error of at most x%.

I found this theorem in a text book.

  Let $\mathcal{H}$ be an hypothesis class and let $\epsilon, \delta > 0$. If a
  training set $S$ of size

  $$n\geq\frac{1}{\epsilon}\operatorname{ln}\left(\frac{|\mathcal{H}|}{\delta}\right)$$

  is drawn from distribution $\mathcal{D}$, then with probability greater than or equal to $1-\delta$ every $h \in \mathcal{H}$ with true error $err_D(h) \geq \epsilon$ has training error $err_S(h) > 0$. Equivalently, with probability greater than or equal to $1-\delta$, every $h \in \mathcal{H}$ with training error zero has true error less than $\epsilon$.

My question is, can I use this theorem to give such an upper bound? If yes, what would the size of my hypothesis space $|\mathcal{H}|$ be in this case? If not, how can I give this upper bound?

As far as I understand this theorem (not much), it doesn't really say anything about the relation between $n$ (the size of the training set) and the error rate. The main relation that it claims is between the error on the training set and the true error, assuming a minimum size on the size of the training set and that the training set is drawn from the true distribution.

I don't think it's possible to give an absolute upper bound on the number of samples needed in the training set for at least the following reasons:

The success of the learning depends on which instances are in the training set, i.e. one needs a sample representative enough so that the model correctly captures the distribution. For instance, in theory it could happen by chance that all the $n$ instances have the same value for feature $x$, so the model would never know that feature $x$ can have another value. In general there's no way to be sure that the training set contains all the diversity required for a successful model in the instances.
There's also the question of what the learning algorithm does exactly: if one wants to prove that a particular DT learning algorithm (say C4.5) can learn a particular model with less than $n$ instances (assuming these $n$ instances are sufficient), the proof must involve the particular properties of the algorithm itself. Otherwise it's obvious that the claim is false: if one chooses an algorithm which randomly builds a tree, it's clear that this cannot lead to the right model.

I think it's more feasible and more common to provide an experimental upper bound: run many cross-validation experiments, varying the training set size and possibly other parameters. Then based on this one can estimate the probability of reaching the correct model given the size and other parameters.


feature selection using genetic algorithm in Python? I have a dataset of 4712 records and 60+ features working on a binary classification problem. I already tried out all the feature selection approaches like filter, embedded and wrapper but am just curious to learn and try genetic algorithm for feature selection.

The reason for choosing genetic algorithm is because I guess it will just provide me the best model fit based on best features.

1) I understand it might take time but would you people help me know how can I do this in Python?

2) In addition, is genetic algorithm any different or better than all other feature selection approaches discussed above? What are its disadvantages?

Is there any python packages and tutorial available on how to use this?

I see tutorials but they all about the theory of genetic algorithm

Can you help me by sharing any tutorial or package for genetic algorithm?

post update

Feature selection is a combinatorial optimization problem. And genetic algorithms is an optimization technique.

So there really isn't anything special, you just need to formulate your problem as an optimization one, and understand how do genetic algorithms optimize. There are enough tutorials on this.

Whether it's better or worse you already know the answer. It depends. On the dataset, constraints etc. What I can tell you from experience is that

You can not expect it to blow your mind but they do work pretty well
They are a great ensembler, meaning results are pretty different (yet accurate) from tree-based methods, NN etc...

Finally regarding implementation, here is completely (maybe too much) automated library based on genetic programming. (notice the word programming here referring to optimization not writing code) Also, it covers feature selection.


Make a random forest estimator the exact same of a decision tree The idea is to make one of the trees of a Random Forest, to be built exactly equal to a Decision Tree.

First, we load all libraries, fit a decision tree and plot it.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
plt.style.use('ggplot')
%matplotlib inline
import random
from pprint import pprint
import pdb
random.seed(0)
np.random.seed(0)
from sklearn.tree import DecisionTreeClassifier
from sklearn import tree
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier

data = load_iris()

dtc = DecisionTreeClassifier(random_state=0)
dtc.fit(data['data'].squeeze(),data.target)

tree.plot_tree(dtc)

We then do the same thing with the random forest

rf  = RandomForestClassifier(n_estimators=1,max_features=None,random_state=0)
rf.fit(data['data'].squeeze(),data.target)
tree.plot_tree(rf.estimators_[0])

My question:

Is it possible to make the exact same the first tree of the random forest and a decision tree?

You need to set bootstrap=False in the random forest to disable the subsampling.  (I originally commented because I expected there to be more impediments [in addition to your already-coded random_states and max_features=None], but I guess there aren't any!)

You probably don't want to do this in general; by stripping out all the randomness so that the first tree is the same as the DecisionTreeClassifier, you'll end up with all the trees being the same, and the random forest loses its usefulness.


How to handle the CEO expectations from a company that's new to data science? I'm new here. I'm about to have a final interview for a data scientist position for a company (it's in the e-commerce field) that is new for data science. It's a pretty new position for the company, and from the interviews I had so far, I noticed that they don't fully understand what they want from a data scientist. They barely know what data science is. I explained them the "standard" data science workflow (Ask A Question, Get the Data, Explore the Data, Model the data and Communicate the data). But I don't think they got it. I want to set the proper expectations, so the company and I can agree on my job description, and what they can expect from me in one month, tree months, six and twelve months. So, how do you handle the expectations(specially the CEO) from a company that's new to data science?

I have not worked as a data scientist.  But managing expectations is something we've all done. Considering you're referring to the company CEO, tact is especially required.  There are three practices that have helped me a lot over the years:

Never say "it can't be done" right away.  Many, many, many times I have said that (as a programmer), only to think of a solution later on.  When I'm skeptical that something can't be done, I've learned to say, "Can I think about it and get back to you?"  That approach has kept egg off my face many-a-time.  It also gives me time to do some research.
Even when something can't be done, focus on what you can provide. If the requested solution is unrealistic for some reason, try to think of what is realistic and offer that.
When discussing what can and cannot be realistically done, clients often don't care about the technical details (even though that's what excites you, and what you want to talk about).  Be ready to go into the technical details if asked, but start off with a very high level explanation.  Rather than focus on the negative details of why something cannot be done, describe the positive details of what it would require to accomplish the task. That shows you have thought it through, and that you're trying to be team player rather than a road block.


How to print x-axes labels in pandas.Series.plot()? I am trying to visualise my data to understand the data skewness. For that purpose, I use the below and get desired output -

df.groupby('owner_team').inc_subj.count().plot.bar(ylim=0)

Output -

My concern is the x-axes labels are shown as numbers, which is the exactly values present. But, my desire is to see the names (string values) corresponding to those numbers.

To give a little bit of background, initially they were string values which I converted to integer values using factorize():

df['owner_team'], mapp = df['owner_team'].factorize()

I am referencing this Pandas doc but couldn't find the exact parameter to set.
Tried labels but didn't help.

Any pointers please.

ps. Using Pandas v0.23.4 and Python v3.6

Having a look at the Pandas plot method (on the DataFrame object), we can see that it returns a matplotlib Axes object.

Try something like this:

ax = df.groupby('owner_team').inc_subj.count().plot.bar(ylim=0)

ax.set_xticklabels(df.owner_team)     # if they are still present as strings

If you removed that column, go back to your original processing and keep a copy of it somewhere then use that column above, instead.

Matplotlib will also generally be able to link to the current/latest plot (figure) that has been created. So using the Pandas plot method, you would need to intercept that.You can then try using standard matplotlib methods (e.g. plt.xlabels and so on).

Ther emight be a nice way using the pandas API directly, but I haven't come across that.


Dataset of extremely low-dimensional images for PCA I am looking for a public data-set of images that differ from each other only slightly, so that after applying PCA they can be reconstructed with a small error from very few PCA coefficients. It can be any type of images, the purpose is only to demonstrate an extreme example of PCA.

Actually in your case I guess the pure images are not that important. The features that you extract from them are important because if your feature space is constructed base on intensity of images at different picture elements, pixels, then you will need so many coefficients. As an easy solution, use MNIST digits and use shape features to extract features from the images of numbers. You can use plausible number of features and then use PCA for the data that is in the new feature space that you have just constructed. In this case smaller number of coefficients will be needed if the features are fine.


Hadoop for grid computing Background:
I run a product that compares sets of data (data matching and data reconciliation).
To get the result we need to compare each row in a data set with every N rows on the opposing data set
Now however we get sets of up to 300 000 rows of data in each set to compare and are getting 90 Billion computations to handle.

So my question is this:
Even though we dont have the data volumes to use Hadoop, we have the computational need for something distributed. Is Hadoop a good choice for us?

Your job seems like a map-reduce job and hence might be good for Hadoop. Hadoop has a zoo of an ecosystem though.

Hadoop is a distributed file system. It distributes data on a cluster and because this data is split up it can be analysed in parallel. Out of the box, Hadoop allows you to write map reduce jobs on the platform and this is why it might help with your problem.

The following technologies work on Hadoop:

If the data can be represented in a table format you might want to check out technologies like hive and impala. Impala uses all the distributed memory across a cluster and is very performant while it allows you to still work with a table structure.
A more new, but promising alternative might also be spark which allows for more iterative procedures to be run on the cluster.

Don't underestimate the amount of time setting up and the amount of time needed to understand Hadoop.


Should I use regex or machine learning? I am thinking of two use cases:

You submit a resume in PDF format to a web site and it extracts your contact information, job titles, etc.
You receive an email from a friend that says, "let's have lunch next Tuesday" and your email program detects it and asks if you want to save a new calendar entry for "lunch on Tuesday".

I can see how this can be done with both ML and with (very sophisticated) regex. What approaches are typically used for these scenarios? I'm assuming that the same approach can be applied to both of the scenarios above?

I would say depends on the requirements and how much effort you want to give the task.

Using regex is definitely easier but in the some time you will not be able to cover everything specially if the text you have is not structured as you expect all the time. So in this case you will be updating you regex patterns each time you miss something ...
If you go for machine learning approach then you will need data to train your models on it ( a lot of this data). Time for training, and enhancing the quality of your model. Hopefully you will get something good.

As a conclusion, I think, if you can cover the requirement with regex go for it. If regex will not be a good solution then start thinking in machine learning solutions.


How is Bayes theorem being applied in expanding the formula of Binary Independence Model? The BIM formula makes use of Bayes theorem. Can anyone please explain:

How to read the probability of the form P(R=1|x,q)? Is it P((R=1|x),q) or P(R=1|(x,q))? Does , stand for AND here?
How exactly Bayes theorem being applied to expand it?

It can be read as the probability of R=1, given (x and y) as your second formula.
Anything after | stand for the given part.

Since:

$P(x | r=1 \wedge q) = \frac{P(x \wedge R=1 \wedge q)}{P(R=1 \wedge q)}$

$P(R=1|q) = \frac{P(R=1 \wedge q)}{P(q)}$

$P(x|q) = \frac{P(x \wedge q)}{p(q)}$

Therefore:

$$P(R=1|x,q) = \frac{P(x | R=1 \wedge q)*P(R=1|q)}{P(x|q)} =$$

$$\frac{P(x \wedge R=1 \wedge q)}{P(R=1 \wedge q)} * \frac{P(R=1 \wedge q)}{P(q)}* \frac{P(q)}{P(x \wedge q)} =$$
$$\frac{P(x \wedge R=1 \wedge q)}{P(x \wedge q)}$$


order of features for model tuning vs model fitting Assuming that the same columns (i.e., features) are used for hyperparameter tuning and model fitting, and ensemble models are used for modeling (e.g., Random forest or XGboost), then does the order of columns used during the hyperparameter tuning process should be identical to the order of columns used when fitting the model based on the best hyperparameters?

I am using sklearn's make_column_transformer functions in my CV pipeline for the hyperparameter tuning. Unfortunately, this function modifies the order of provided columns when setting the remainder argument to 'passthrough'. Should I ensure that when fitting the model the same order of columns is preserved, or the order does not matter as long as I am using the same features.

Ah, I was too quick, and misinterpreted your question!  At the bottom of this post I'll leave my old answer, answering why the test set needs to have the same column order.

As for column order of the data in hyperparameter selection vs. training a final model, no, I suppose there's no real reason these need to be the same.  In a tree model with column subsampling, you're right (in your comment) that the columns will be selected randomly anyway, so the original order doesn't matter.
 Even if you don't use column subsampling, and even for other models: a model generally won't be using the column order as informative; if anything, it's used as a fallback tiebreaker.  (Time series are an obvious exception, but in that case maybe the data isn't tabular in the same way.)

That said, it's still perhaps best practice to use the same pipeline, so that the column order will be the same anyway.  sklearn's hyperparameter tuners make this easy, with refit=True by default just refitting the model pipeline on the best hyperparameters found.

Since sklearn operates on numpy arrays and not pandas dataframes (one of the first things in most sklearn steps is conversion to arrays), you need to make sure columns arrive in the same order as the training data.  Otherwise the model will mistake values of some features as being different features!  Hopefully this will actually break things (wrong feature type, e.g.), but perhaps it will silently make very bad predictions!

This shouldn't be hard if you use pipelines.  The make_column_transformer (and all other steps) will be applied to the testing data in the same way as your training data, so the array after these steps will have columns in the right order.  (Alas, if you want to dig into the results, attaching names to the columns after the preprocessing parts of the pipeline may be a hassle.)


How do I add a column to dataframe on condition that certain observations in my dataframe contain a target word in Python? Here's what my data frame look like:

Number  Age      Famous_for
1       35       "businessman chairman of IBM (1973–1981)"
2       42       "musician (House of Freaks Gutterball)"
3       87       "baseball player (Oakland Athletics)"

I want to create additional column that will contain a dummy variable whether or not person was involved with entertaining business. Something like that:

Number..Age..................Famous_for.....................Entertaining
1.......35.......businessman chairman of IBM (1973–1981)........0
2.......42.......musician (House of Freaks Gutterball)..........1
3.......87.......baseball player (Oakland Athletics)............0

How do I create a column based on the certain words in the famous for column (such as "musician", "club", "actor" and etc)?
I tried the following:

df['entertaining'] = np.where(df['famous_for']>="musician", 1, 0)

but it wouldn't work.
How can I do this in Python?

You can use str.contains on the DataFrame:

df = pd.DataFrame({'Age' : pd.Series([35,42,87], index=[1,2,3]),
        'Famous_for': pd.Series(['businessman chairman of IBM (1973–1981)',
                                 'musician (House of Freaks Gutterball)',
                                 'baseball player (Oakland Athletics)'],
        index=[1,2,3])
        })

df['entertaining'] = df['Famous_for'].str.contains('musician')
print (df)

   Age                               Famous_for entertaining
1   35  businessman chairman of IBM (1973–1981)        False
2   42    musician (House of Freaks Gutterball)         True
3   87      baseball player (Oakland Athletics)        False

Note that str.contains also accepts a regular expression, so if you wish to search for various words you might use

df['entertaining'] = df['Famous_for'].str.contains('musician|businessman')


Error in Neural Network To design a neural network based predictor for f(x) = sin(x) I designed the following network. Input ----> Dense(num_neurons=1) ---> Relu() ----> Dense(num_neurons=100) ---> Relu() ---> Dense(num_neurons=1) ----> Relu()--> Output.

The choice of loss was L2 and optimizer was standard gradient descent. The predictor was trained on X_train = numpy.arange(0.0, 314.1, 0.1) and Y_train = numpy.sin(X_train). It is subsequently tested on X_test= numpy.arange(-10.0, 10.0, 0.001) and Y_test = numpy.sin(X_test) The predictor however performs badly on the test data. What could have gone wrong?

The training data is too small to train the network. Adding more data especially when X < 0

Relu activations used in the network are wrong. Changing the activation in the output node from Relu to Linear should help network learn the function better.

Network is not deep enough. Adding more hidden units/layers to the network will help the network generalize

L2 loss is not a good metric to measure loss when regressing for sin(x)

Gradient descent is not the best optimizer for this problem. Adam optimizer would be a better option.

This type of neural network architecture can fundamentally not learn a function like sin(x).

Since we know that sin(x) can have -ve values as well,
Relu will kill all those mercilessly..

Also we can't have relu just before the output layer as it doesn't makes any sense..(Atleast I have never seen this architecture before..)

Also you can make that 100 drop to around 40

Check this link also


[under/over]-sampling teaches model the wrong distribution? TLDR: Will under/oversampling during the training phase teach the model the wrong distribution and adversely affect accuracy?
Let us assume you want to train a classifier to differentiate between class A and class B. Unfortunately, the population distribution of A and B is unbalanced at a ratio of [1:100]. As such, you utilize under-sampling or over-sampling such that the training and validation sets effectively achieve a [1:1] ratio between A and B. You do nothing to the test set. The distributions of the sets and and the training results are in the below table:
           Train  Val    Test
A Dist.    0.5    0.5    0.99
B Dist.    0.5    0.5    0.01
Accuracy   1.0    0.999  0.85

You have now trained a model which performs worse on the population than a "classify all as A" approach. Does over-sampling or under-sampling teach a model the wrong distribution - causing it to over-confidently predict minority classes? If not - what could be happening in this example?

Yes, teaching the model to expect a balanced distribution definitely will impact the results on the test set. Oversampling the minority class to balance the distribution will make the classifier more likely to predict that a given example is the minority class. Each iteration of gradient descent will push the model to a location where approximately half of the loss was contributed by examples from a minority class, which is not the situation when you're testing the model. There are a few things that I've seen that try to overcome this issue; focal loss is one example, where easily classified examples (predictions that the model has a lot of confidence in) contribute less to the loss than examples that are difficult to classify. Another approach that I've seen used is training a model on an artificially balanced training set, and then freezing all of the weights except the last layer (or last few layers), and finetuning the model on the true distribution. This usually increases the precision of the model.
Accuracy is a misleading metric in this situation. As you noted, the model trained with a balanced training set performs worse when compared to a model that predicts everything as class A. Despite the fact that the accuracy of the second model is higher, the predictions don't make sense, and don't reflect the nature of the data. I would encourage you to examine the precision and recall of the two models to get an idea of where the model is doing poorly.
Also, I think your validation split should have the same statistics as the test set. When the validation data doesn't reflect the test data, you're essentially optimizing your model to hit a different target than you actually want.


Evaluating the test set Please find attached a part of the code which explains what I'm trying to do. Essentially I'm trying to predict the sales of supermarket stores. Im using RandomForestRegressor for this and have predicted the results on the test set. The cross validation is done on the training set with a mean accuracy of 0.52 and I have tried to calculate RMSE which comes to 1180. Now, I want to know how do I make sense of all this and know how well it performed on the test set and how to evaluate my model. It'll be extremely helpful if you could make me understand. Thanks

Ignore the np.exp(rmse) part

Best,

Pranveer

welcome to the forum, I will try to clarify some things.

First of all, when talking about regression you do not calculate accuracy. Accuracy is a metric used in classification tasks, it basically measures how much labels your model got right divided by the total labels.

You are working in a regression task, that is, you are trying to predict not labels, but continuous values as your target. By default, scikit-learn estimators calculate as regression scores the R_squared metric. Intuitively, R_squared measures how much of variance your model explains. It is calculated by the below equation:

(Image taken from scikit-learn r2_score page: https://scikit-learn.org/stable/modules/model_evaluation.html#r2-score

Basically, I like to think that R2 is testing your model against the simplest possible one in a regression task, that is, just predicting the mean for your target variable. It makes sense talking about accuracy as an analogy, however I wanted to make clear that they are different things.

Second, talking about results, RMSE and how can you measure the effectiveness of your model, in linear regression context it will be highly dependent of the problem your are trying to solve. Maybe you are a social scientist trying to predict with some data the number of voters in certain candidate, an R2 of 0.4-0.5 would be amazing (?). Maybe you are sure that this is not sufficient because your variable are strongly related to your target variable (some problems and physics) and a R2 of 0.80 is unacceptable. Another thing is RMSE, it tries to measure the variance of your model that is, how far off you usually are when predicting your target. I had a problem where I was prediciting the number of days of a certain event to happen, and my variance was 15 days off, and this was not helpful at all.

A highly recommend you to read the Chapter 3 of introduction to Statistical Learning Book that will help you a lot (Actually, the whole book). They have a free version here: http://www-bcf.usc.edu/~gareth/ISL/

I hope this helps. Also, any problem to understand what I wrote, please let a comment. And if anyone spots something wrong in my answer, tell me please!


Fraud detection use text mining I would like to find different patterns recognition algorithm to detect different type of fraud. I have 1 million unstructured text documents about the clients' information with metadata about the client name, viewers, location in the cloud.
Here are the patterns I was thinking of and the related fraud that i would like to detect:

Numeric Patterns - fictitious invoice numbers, fictitiously-generated transaction amounts.
Time Patterns - transactions occurring too regularly, activity at unusual times or dates.
Name Patterns - similar and alerted name and addresses.
Geographic Patterns - Proximity relationships between apparently unrelated entities.

What technique can I use? any keywords?

I have worked on building a fraud detection solution using text mining, so I understand the scenario that has led to this question. I'll talk about the approach/techniques that is to be followed to build the fraud detection solution.I'll divide it into 4 sections

Build line of business fraud dictionary
Fraud detection & scoring
Tools
Additional notes

Building Fraud Dictionary:

As a first step you'll have to identify fraud concepts with help from subject matter expert for the line of business in concern. Fraud concept is a person, characteristic, entity, or event that represents a suspicious scenario and similar concepts can be grouped together to form a concept category. Each concept is further represented by words, phrases, entities etc. The presence of these words/phrases in the document implies occurrence of fraud concept in that transaction. The end result of this exercise will be a fraud dictionary that is a repository of concepts and suspicious key words.

For example considering the data set mentioned in the question, the numeric patterns become the concept category, and fictitious invoice numbers and fictitiously-generated transaction amounts are 2 different concepts. There will be a keywords/phrases that is associated to this concept which should be captured onto the fraud dictionary.

You can use NLP technique to build the dictionary.

Steps:

1) Convert the sentence to lowercase

2) Remove stopwords, these are common words found in a language. Words like for, very, and, of, are, etc, are common stop words

3) Extract n-gram i.e., a contiguous sequence of n items from a given sequence of text. simply increasing n, model can be used to store more context

4) Assign a syntactic label (noun, verb etc.)

5) Knowledge extraction from text through semantic/syntactic analysis approach i.e., try to retain words that hold higher weight in a sentence like Noun/Verb

Fraud detection & scoring:

Phrases/Keywords to be semantically matched with Phrases/Keywords from Fraud Dictionary and Fraud concepts in transactions can be identified. At this point, care needs to be taken to maintain the context (negation, positive sense) in which a keyword/phrase has been used to avoid false positives.

For each identified occurrence of Fraud concept in the transaction, a weight of 1 can be assigned. This is done for calculating a suspicious score for the transaction. Higher the score high suspecious.

Note that using a combination of larger data set with high textual content, and extensive fraud dictionary would result in higher number potentially identified.

Tools:

You can consider using OpenNLP / StanfordNLP for Part of Speech tagging. Most of the programming language have supporting library for OpenNLP/StanfordNLP. You can choose the language based on your comfort.

Additional notes:

You can refer here to get an idea of extracting concepts from sentences
Fruther read my blog on Text mining 101 to learn more about

TM process overview
Calculate term weight (TF-IDF)
Similarity distance measure (Cosine)
Overview of key text mining techniques

Hope this helps.


LSTM - How to prepare train from a dataset which contains multiple observations for different events I m using LSTM in a project related to MobiFall dataset which contains falls and daily activitives - such as walking, sitting etc - sensed by accelerometer, gyroscope and orientation sensors in x,y,z axes.

So I need to modify LSTM into multi-variate form. How could it be done?

And after this problem is solved, I have to deal with another, there are multiple time-series events in different files which were done by different people. For example, I have got ADL_1_walking_1_.txt, ADL_1_walking_2.txt, ADL_2_jumping_3.txt (Naming convention of text files is ActivityType_SubjectNo_ActivityName_SampleNo.txt) which contains acc-x, acc-y,acc-z, gyro-x, gyro-y, gyro-z, ori-x, ori-y, ori-z data where acc means accelerometer, gyro means gyroscope and ori means orientation.

So how should I train my model? Should I combine all of those files? But doing this will cause model to suppose they are all from the same observation right? So it may lead wrong calculations. Should prepare a training and test test for each file and train the model over and over again? So my question is what are the general solutions for this kind of problem. Can you tell me some of these solutions?

It's just several time-series with same data. You should make preprocessing (if it's neccessary) and can provide as an input just for one RNN layes.
for example

model.add(GRU(units=512,return_sequence=True,input_shape=(None,number_of_inputs))


Undersampling improvement F1-score Im doing a 2-class classification project for an imbalanced data set.
The imbalance is about 18%/82%. Im noticing a huge improvement in F1-score when I under-sample; from 16% without under-sampling to 49% with under-sampling. Im wondering if this is possible or that I am doing something wrong? Before under-sampling I split the data set in a train- and test set, apply outlier detection and normalisation (normalisation first on train set, then on test using the scales obtained from train set). Does anybody have any idea, if this improvement is too good to be true, what could be going wrong?? Many thanks in advance!

Improvement in the evaluation metric by rebalancing depends on the type of classifier algorithm. Certain algorithms are very sensitive to rate imbalances, thus adjusting the respective levels of support will change the algorithm's performance.

One way to empirically verify that only under-sampling is driving the improvement is to run an experiment - gradually increase the level of under-sampling and plot the result on the F1 score.


Classification problem: estimate the number of people in a household through account transactions I want to estimate how many people are in a household by looking at account transactions. It would be also interesting to understand if in the household there are children.

I thought that a possible way to proceed could be to compare the living expenses with that of the average individual according to the national statistics. Moreover I could also perform text analytics to retrieve informations about what products people bought (in order to understand if they spent money to buy children products).

As model I could maybe use a random forest for classification (but please correct me if I'm wrong).

Are there better way to solve this problem?

I think regression makes more sense here than classification; you are trying to predict a number. In particular I would recommend Poisson regression since you are modeling count data. Software is readily available for various languages.


Transfer learning on new image size Transfer learning: Take a trained neural network and use it for a new classification task.

When we want to use transfer learning with a convolutional neural network, we don't have to use the same image size as input than the image size used for training. But if we change the input size we will have to re-train fully connected layers. See this post on Stackoverflow.

I don't undertand why changing input shape will not affect convolutional layer weight and why it will affect fully connected layer weights.

Tell me if my question is not understanble.

In convolution layers, weights are the filter values.
Weights of a CNN network is determined by the number and size of filters. Weights doesn't depend on shape of input image.
Filters of pre-trained network are capable of detecting different features in an image. Consider sobel filter which detects edges in an image. A 3x3 sobel filter can be
used to detect edges in an image of any size. Similarly, those filters can be reused instead of again learning from scratch.

That's why, weights of CNN network can be transferred to another CNN network with different input shape.
However, number of layers, no. of filters, filter sizes and number of image channels must be same.

For transfer learning, best practices would be to use pre-trained model for similar task and don't change the input shape to very small or large.

On the other hand, weights of Fully Connected(Dense) layer can't be transferred. Because, those weights depend on image size. Dense layers at the end of CNN network should be cut off and recreated to adopt to change in image size.
It would be more clear if you get to know how the trainable parameters are calculated in CNN and Dense layers. It would be very long if I try to explain in this answer. You can google search on that.


Does this data work with Convolutional Neural Network I have a data set of 900k+ row with each row consisting of a pattern (e.g. 1421143) and a result (flawed, good).

Can I use the patterns in a CNN model to predict the defects occurrence ?

What distinguishes CNNs from other deep learning neural networks is the use of a convolving filter which utilizes the structure of the data set.

CNNs are used for image classification because grayscale images are 2D arrays where the location of each pixel is relevant (if you rearrange all the pixels randomly, no one can tell what the image is). The convolving filter which is almost always a pixel of squares with weights is used to reduce the dimensionality of the image.

Likewise, 1D CNNs are used for time-series data because the location in time of each set of features is relevant (imagine a convolving filter moving across this 1D array in the same way the 2D filter moves over an image).

Does the pattern you mention (e.g. 1421143) have any relationship between different rows? Do the individual digits have a relationship with each other? If so then a CNN going in either axis (row-wise or column-wise, perhaps both) makes sense, but if, for example, scrambling all of the rows wouldn't make a difference in your analysis, then I am not sure if a CNN is the right direction over a shallow neural network given the small number of features.


What is the difference between semantic segmentation, object detection and instance segmentation? I'm fairly new at computer vision and I've read an explanation at a medium post, however it still isn't clear for me how they truly differ.

Object Detection : is the technology that is related to computer vision and image processing. It's aim? detect objects in an image.
Semantic Segmentation : is a technique that detects , for each pixel , the object category it belongs to , all object categories ( labels ) must be known to the model.
Instance Segmentation : same as Semantic Segmentation, but dives a bit deeper, it identifies , for each pixel, the object instance it belongs to. The main difference is that differentiates two objects with the same labels in comparison to semantic segmentation.

Here's an example of the main difference.

In the second image where Semantic Segmentation is applied, the category ( chair ) is one of the outputs, all chairs are colored the same. In the third image, the Instance Segmentation, goes a step further and separates the instances ( the chairs ) from one another apart from identifying the category ( chair ) in the first step.
Hope this clears it up for you a bit.


Agent always takes a same action in DQN - Reinforcement Learning I have trained an RL agent using DQN algorithm. After 20000 episodes my rewards are converged. Now when I test this agent, the agent is always taking the same action , irrespective of state. I find this very weird. Can someone help me with this. Is there a reason, anyone can think of why is the agent behaving this way?

Reward plot

When I test the agent

state = env.reset()
print('State: ', state)

state_encod = np.reshape(state, [1, state_size])
q_values = model.predict(state_encod)
action_key = np.argmax(q_values)
print(action_key)
print(index_to_action_mapping[action_key])
print(q_values[0][0])
print(q_values[0][action_key])

q_values_plotting = []
for i in range(0,action_size):
    q_values_plotting.append(q_values[0][i])

plt.plot(np.arange(0,action_size),q_values_plotting)

Every time it gives the same q_values plot, even though state initialized is different every time.Below is the q_Value plot.

Testing:

code

test_rewards = []
for episode in range(1000):
    terminal_state = False
    state = env.reset()
    episode_reward = 0
    while terminal_state == False:
        print('State: ', state)
        state_encod = np.reshape(state, [1, state_size])
        q_values = model.predict(state_encod)
        action_key = np.argmax(q_values)
        action = index_to_action_mapping[action_key]
        print('Action: ', action)
        next_state, reward, terminal_state = env.step(state, action)
        print('Next_state: ', next_state)
        print('Reward: ', reward)
        print('Terminal_state: ', terminal_state, '\n')
        print('----------------------------')
        episode_reward += reward
        state = deepcopy(next_state)
    print('Episode Reward' + str(episode_reward))
    test_rewards.append(episode_reward)

plt.plot(test_rewards)

Thanks.

This may seem obvious, but have you tried using a Boltzmann distribution for action selection instead of argmax? This is known to encourage exploration and can be done by setting the action policy to

$$p(a|s) = \frac{\exp(\beta Q(a,s)}{\sum_{a'} \exp(\beta Q(a',s))},$$

where $\beta$ is the temperature parameter and governs the exploration-exploitation trade-off. This is also known as the softmax distribution.

Put into code, this would be something like this:

beta = 1.0
p_a_s = np.exp(beta * q_values)/np.sum(np.exp(beta * q_values))
action_key = np.random.choice(a=num_act, p=p_as)

This can lead to numerical instabilities because of the exponential, but that can be handled e.g. by first subtracting the highest q value:

q_values = q_values - np.max(q_vaues)


Similarity of XGBoost models? Is xgboost with n_estimators = 100 and learning_rate = 0.1, same as xgboost with n_estimators = 50 and learning_rate = 0.2 ?

No, they won't have neither the same performance nor the same architecture if you were to try to visualize it. An XGBoost with 100 n_estimators and a learning rate of 0.1 is a 100 trees grown sequentially, and each tree's output is multiplied by 0.1. The latter is a model of 50 trees grown the same way and the outputs are multiplied by 0.2, so your second model ( given the usual size of datasets nowadays ) will probably underfit the data, a low number of trees to learn from the data and a high learning rate which will make it somewhat innaccurate.

Unless you're looking for another kind of similarity, this should explain it enough. I hope. If not, feel free to comment to clarify your question more.


How to compare two dataframes and put the counted unique values in a first dataframe's column? I have two different (geo)dataframes, one has 690 and the other has 1826 rows. The first one is a grouped based on the nearness (spatial near) of the second dataframe. Thus, they bound with FID_1 and NEAR_FID columns. Every FID_1 in the first dataframe corresponds to at least 2 NEAR_FIDs in the second dataframe. My challenge is to count the number of NEAR_FIDs in the second dataframe and put as a row in the first one. I am using python 3.5+. Thanks for any suggestion.

What you want can be achieved using themerge function in pandas. Here is the code:

df1 = pd.DataFrame({'FID_1':[1,2,3]})
df2 = pd.DataFrame({'NEAR_FID':[1,1,1,1,2,2,3,3]})

df_freq = df2.NEAR_FID.value_counts().reset_index()
df = pd.merge(df_freq, df1, how='inner', left_on=['index'], right_on=['FID_1'])
df
       index  NEAR_FID  FID_1
0      1         4      1
1      3         2      3
2      2         2      2

You can remove any of the column index or FID_1 to get your desired output.

For more on merge function look at the official documentation
 and for a very good explanation of its working look here.

EDIT 1:
Merge function only accepts Dataframes and not series. So, if your data is in series format, convert it to dataframe first.
You can check the type of your data by:

print(type(df2))

And that should output pandas.core.series.Series

Use this to change it to a dataframe:

df = df.to_frame().reset_index()


How to implement keras LSTM time series I am learning how to implement Keras LSTM on a simple time series data.
The dataset I'm using has $12$ columns and $300k$ rows. Each group of $200$ rows represents one time-series cycle. Then, the time starts again from zero and run for the next $200$ rows. I want to make predictions for sixof the columns.
My questions are, how many (multiple of 200?) rows do I choose for train/test sets? Is my batch size $200$? What is the basis for choosing number of neurons?

You've not provided the dataset but I will try to answer based on your descriptions.

  how many (multiple of 200?) rows do I choose for train/test sets?

Actually, the train/validation/test splits do not really depend on the length of the time series. It is more dependent on the number of examples, time series here. $300k / 200 = 1.5k$ which means the size of your dataset is small and you can use the customary percentages for your splits. Something like 60/20/20 for train/validation/test.

  Is my batch size 200?

$200$ is not your batch size, it is the length of your signal. In LSTM networks, you usually deal with temporal data. For each sample, you have 12 features for each time step. $200$ means each sample has $200$ steps. In vectorise implementations of LSTMs it is customary to define batch size. That is, you stack $m = batch size$ samples for acceleration in training, it has details that I skip.

  What is the basis for choosing number of neurons?

Take a look at here. Although the details of MLPs are described, they can be generalised to LSTMs too. Consider RNNs like simple MLPs which take the inputs of the current time step and the outputs of the previous step.


Calculating saliency maps for text classification I'm following the text classification with movie reviews TensorFlow tutorial, and wanted to extend the project by looking, for a certain input, which words influenced the classification the most.

I understand this is called a saliency map, but I'm having trouble calculating it. I believe that I need to calculate the gradients of the output with respect to the input. I tried to implement code similar to the code in this answer to no avail. A confounding issue is that the model uses an embedding layer, which doesn't propagate the gradient, so I think one needs to calculate the gradients with the input being the output of the embedding layer.

It's probably wrong for all sorts of reasons, but this is the closest I've gotten with the Python code:

# Create the saliency function
input_tensors = [model.layers[1].input, keras.backend.learning_phase()]
model_input = model.layers[1].input
model_output = model.output
gradients = model.optimizer.get_gradients(model_output, model_input)
compute_gradients = keras.backend.function(inputs=input_tensors, outputs=gradients)

# Word encoding
idx = 0 # Calculate the saliency for the first training example
embeddings = model.layers[0].get_weights()[0]
embedded_training_data = embeddings[train_data[idx]]
matrix = compute_gradients([embedded_training_data.reshape(sum([(1,), embedded_training_data.shape], ())), train_labels[idx]])

But the final matrix is the same row repeated and I'm not sure how to interpret it. Any help would be greatly appreciated. Thankfully, as this is extending a tutorial, there is a complete working example of the code!

I've been working on this for the last few days, and I think I've answered my own question. Calculating individual word saliency is not possible with the model structure as is. This is because of the GlobalAveragePooling layer of the model. This averages the embedding matrix in the 'word' dimension, removing the ability to distinguish the effect of an individual word on the classification. This is the code I used to convince myself of what was happening, left here in the hope that the next soul to try this finds this answer.

outputTensor = model.output

embeddingTensor = model.layers[1].input
gradientsEmbedding = tf.gradients(outputTensor, embeddingTensor)

globalAverageTensor = model.layers[2].input
gradientsAverage = tf.gradients(outputTensor, globalAverageTensor)

idx = 1

sess = keras.backend.get_session()

embedding = sess.run(embeddingTensor, feed_dict={model.input:train_data[(idx-1):idx,:]})
globalAverage = sess.run(model.layers[2].input, feed_dict={model.input:train_data[(idx-1):idx,:]})

print(np.mean(embedding, 1))
print(globalAverage)

gradientMatrixEmbedding = sess.run(gradientsEmbedding, feed_dict={embeddingTensor:embedding})
gradientMatrixAverage = sess.run(gradientsAverage, feed_dict={globalAverageTensor:globalAverage})

print(np.sum(gradientMatrixEmbedding, 2))
print(gradientMatrixAverage)


Finding variable values to make equations behave like boolean gate I came across following problem:

Consider the following simple perceptron, for which the inputs are the always $1$ feature and two binary features $x_1 ∈ \{0, 1\}$ and $x_2 ∈ \{0, 1\}$. The output $y ∈ \{0, 1\}$.
$$y=\begin{cases}
1,  & \text{if $w_0+w_1.x_1+w_2.x_2>0$ } \\
0, & \text{otherwise}
\end{cases}$$
Choose all the correct option below that the weights $w_0,w_1,w_2$ should satisfy such that $y=(x_1 \text{ AND } x_2 )$.
(A) $w_0>0$      (b) $w_0≤0$   (c) $w_1>0$        (d) $w_2>0$

I tried to solve the problem and came up with following:

(Eqn# stands for Equation# in above table)
Eqn#4 $= w_0≤0$  (Option B)
Eqn#2 $→w_0≤-w_1$    (Eqn#5)
Eqn#3 $→w_0≤-w_2$    (Eqn#6)
Eqn#1 $→ 0>-w_1-w_2-w_0$
$→ 0>-w_1+(non-ve)$    … from Eqn#3
$→ -(non-ve)>-w_1$
$→ -ve>-w_1$
$→ w_1>0$ (Option C)
Similarly from Equation 4 and 2, it can be proven that $w_2>0$ (Option D)

But I feel something is wrong with above. I feel option B contradicts with both option C and option D. For example, option C and Eqn#5 makes $w_0=0$ impossible which is allowed by option B. Where did I made mistake? I feel its incorrect to directly conclude $w_0\leq 0$ from Eqn#4 and "stop". What I mean is that I have to continuously keep checking if all facts that I conclude lead to desired effect "together". For example, once I got that option (c) is correct, I have to combine Option C fact with Eqn#4 fact to conclude that $w_0<0$. So finding solution to such problems is kind of continuous process till we get all equations which satisfy each other and target goal (AND gate here). Q1. Am I correct with this? Or I am unnecessarily making things difficult and there is simpler and more importantly "straight forward" way (not involving continuously checking if all equations satisfy each other) to find such solutions?
Q2. This does also mean that Option B should have $<$ instead of $\leq$. Am I correct with this too?
PS:
After getting confused like me, some of my friends have started saying it should be $\geq$ instead of $\gt$:
$$y=\begin{cases}
1,  & \text{if $w_0+w_1.x_1+w_2.x_2\color{red}{\geq}0$ } \\
0, & \text{otherwise}
\end{cases}$$
But I dont feel so as I can still find correct answer with given problem with conclusion that option B should have $<$ instead of $\leq$. Also I dont feel its correct for a function to interpret $0$ as $True$. Q3. Am I correct with this too? (problem doesnt need correction, but option B)

You can only have $w_0 < 0$. If $w_0 = 0$, then eq 1,2,3 imply that
$$ 0 < w_{1,2} \leq 0 $$
which is a contradiction. Hence option b) is incorrect.


Simple Markov Chains Memoryless Property Question I have a sequential data from time T1 to T6. The rows contain the sequence of states for 50 customers. There are only 3 states in my data. For example, it looks like this:

        T1  T2  T3  T4  T5  T6
Cust1   C   B   C   A   A   C

My transition matrix X looks like this:

    A   B   C
A   0.3 0.6 0.1
B   0.5 0.2 0.3
C   0.7 0.1 0.2

Now, we see that at time T6 the state is at C which corresponds to c=[0 0 1] vector.
I am now predicting T7 by doing the matrix multiplication: c * X which gives me [0.7 0.1 0.2]. Based on this, I decide that the state at T7 would be A (highest prob. value).

For T8, I use the result of the probability vector I got above and do [0.7 0.1 0.2]*X = [0.4 0.44 0.14]  and decide that the state is B.

My question is: Am I doing something wrong? Am I contradicting the memoryless property of the Markov Chains?

I think you are not doing anything wrong, the markov property is satisfied when the prediction can be solely based on the present state. I do not think you are contradicting this "memoryless" condition (which applies to the whole chain) when you predict what could probaly happen after a last state that is already just predicted.


How to Write Multiple Data Frames in an Excel Sheet I have multiple data frames with same column names. I want to write them together to an excel sheet stacked vertically on top of each other. And between each, there will be a text occupying a row. This is what I have in mind.

I tried the pandas.ExcelWriter() method, but each dataframe overwrites the previous frame in the sheet, instead of appending.

Note that, I still need multiple sheets for different dataframe, but also multiple dataframes on each sheet. Is it possible? Or any other python library which can dynamically generate the excel sheet from pandas dataframes?

an example to write in same sheet:

import pandas as pd

data1 = """
class    precision   recall
<18      0.0125         12
18-24    0.0250         16
25-34    0.00350         4
"""
data2 = """
class    precision   recall
<18      0               0
18-24    0.25            6
25-34    0.35            5
"""

#create 2 df for sample
df1 = pd.read_csv(pd.compat.StringIO(data1), sep='\s+')
df1.name = "Dataframe1"
df2 = pd.read_csv(pd.compat.StringIO(data2), sep='\s+')
df2.name = "Dataframe2"
print(df1);print(df2)

writer = pd.ExcelWriter('e:\\test.xlsx',engine='xlsxwriter')
workbook=writer.book
worksheet=workbook.add_worksheet('Result')
writer.sheets['Result'] = worksheet
worksheet.write_string(0, 0, df1.name)

df1.to_excel(writer,sheet_name='Result',startrow=1 , startcol=0)
worksheet.write_string(df1.shape[0] + 4, 0, df2.name)
df2.to_excel(writer,sheet_name='Result',startrow=df1.shape[0] + 5, startcol=0)

output:

if you want to write in different sheets:

import pandas as pd

# Create a Pandas Excel writer using XlsxWriter as the engine.
writer = pd.ExcelWriter('e:\\test.xlsx', engine='xlsxwriter')

# Write each dataframe to a different worksheet. you could write different string like above if you want
df1.to_excel(writer, sheet_name='Sheet1')
df2.to_excel(writer, sheet_name='Sheet2')

# Close the Pandas Excel writer and output the Excel file.
writer.save()


How Does Weighted KNN Work? I am reading notes on using weights for KNN and I came across an example that I don't really understand.

Suppose we have K = 7 and we obtain the following:

Decision set = {A, A, A, A, B, B, B}

If this was the standard KNN algorithm we would pick A, however the notes give an example of using weights:

By class distribution (weight inversely proportional to class frequency)

class A: 95 %, class B 5 %

This results in a class of B.

I can't seem to figure out the math that was left out to obtain B as the answer.

We can view nearest neighbor as a voting process where we consult our $k$ nearest neighbor.

We give the $i$-th data point a voting weight $w_i$.

In your example, each data point in class $A$ has weight $\frac1{0.95}$ and each data point in class $B$ has weight $\frac1{0.05}$. There are $4$ votes from class $A$ and $3$ votes from class $B$. We give class $A$ a score of $\frac{4}{0.95}\approx 4.21$ and class $B$ a score of $\frac{3}{0.05}=60$. Class $B$ has a higher score, hence we assign it to class $B$.


How to train LSTM with daily timeseries? I have for each day sensor timeseries data. I just ask myself how to train with that a LSTM eg. for classification? Since I would like to have the LSTM train on all examples and not just one?

I just see examples where LSTM are trained on one timeseries

Do you mean you mave multiple sensors? Ir one single sensor that record one measurement every day?

In short, you will need to simply match the dimensions of your data to the LSTM. To do that, you design your LSTM to consume tensors of the correct shape.

Let's make the following assumptions:

You have 2 sensors
Each sensor records one measurement per hour (so 24 per day)
You want to consider blocks of 5 consecutive hours to predict the 6th hour

Your LSTM shape would have the following input_shape shape:

(5, 2)

You can pass any number of these sample blocks as a batch, using a batch_size parameter.

So it summarises to this format:

(batch_size, num_time_steps, num_features)

This is a brief introduction that hopefully generalises the idea for you. I would suggest looking at more examples to start getting a feeling for the required dimensions and the terminology that people generally use.

Have a look at this answer, where I explained relevant points for a similar question.
If you have multiple sensors, you need a multivariate LSTM - check out this answer
Another Great Source that explains LSTM use in TimeSeries Jason Brownlee - LSTM TimeSeries Prediction in Keras


Prediction in Machine Learning When we use a regression algorithm in out dataset it's because we assume that there is a relation between our input data and some quantitative value. This is expressed as :

$y = f(x)+\varepsilon $, where $x$ is an input vector and $\varepsilon$ is the random error term.

In the case of a Prediction algorithm, this is the equation :

$\hat{y}=\hat{f}(x)$ where $\hat{f}$ represents our estimate for $f$, and $\hat{y}$ represents the resulting prediction for $y$.

What I don't understand is : why n this setting, $\hat{f}$ is often treated as a black box, in the sense that one is not typically concerned with the exact form of $\hat{f}$, provided that it yields accurate predictions for Y.

Not sure if I can get the direction of your "why", but here's a try:

If you were to use some sort of interpolation, you would have an explicit model of this function in the mathematical sense. This would be a white box, as you actually have the formula/algorithm to do the prediction. If you use a conventional decision tree or linear regression you are still in the same white-box category in the sense that your trained model can be expressed by a manageable set of formulas or rules.

However, in typical machine learning scenarios there are at least two factors that limit the transparency of this box:

We care more of the result, and less of the explanation. Thus, we might use different approaches. As long as they fit in the same "interface" as you define it with your function, we don't really care which one is being used. This allows for A/B testing of different approaches and putting in place the one that currently yields best results. After some time, based on new training data or on new advances in machine learning, we can decide to use another algorithm, but the general setup remains the same.
Contemporary algorithms are extremely complex, both because the dimensions of the input and output are much higher than our intuition can cope with, and (inherently) because they use many more iterations. Examples of these iterations could be the layers of deep networks, but also the range of algorithms used in ensemble methods like random forests.


What kind of learning is needed for anomaly detection? Supervised learning, semi-supervised learning or unsupervised learning? I am doing anomaly detection recently, one of the methods is using AEs model to learn the pattern of normal samples.

Determine it as an abnormal sample if it doesn’t match the pattern of normal samples.

I train AE without labels but we need to use ‘label’ to determine which sample is normal or abnormal.

I am wondering what kind of this training is supervised learning,semi-supervised learning or unsupervised learning?

Supervised learning is when your model learns from a well-labeled dataset. By well-labeled dataset, I mean every row is categorized into a class. The model learns patterns from this dataset and makes a prediction. One flaw of supervised learning is that it only makes a judgement based on the patterns it has seen in the training dataset.

Unsupervised learning is the opposite of supervised. You have a dataset but you don't know anything about the classes of the dataset. In this case, you try to identify the patterns among the dataset and try to cluster the same patterns together and a cluster forms a class. The number of clusters depends on the data scientist and the data.

Semi-supervised learning algorithms are trained on a combination of labeled and unlabeled data. This is useful for a few reasons. First, the process of labeling massive amounts of data for supervised learning is often prohibitively time-consuming and expensive. Also, if the data is labelled by humans, it can contain bias from our judgment.

Anomaly detection falls under the bucket of unsupervised and semi-supervised because it is impossible to have all the anomalies labeled in your training dataset. There are several methods to achieve this, ranging from statistics to machine learning to deep learning. To have a detailed idea on these things, refer to the following link  https://www.datascience.com/blog/python-anomaly-detection


Correlation/distance between sparse vectors I am looking for a metric for comparing gene count tables. These are long columns of data (a few millions genes by a few dozen samples), with all non-negative entries, about 90% of which are zeros. The goal is to compare the performance of several tools/algorithms that these tables originate from, by comparing the resulting tables among themselves or with the expected counts (in a case of sumulates data). In principle, one compares on a sample-by-sample basis, but comparing different samples might be also of interest, e.g., to filter out spurious correlations.
What I am using now is Spearman rank coefficient, taking account for the fact that some entries have identical ranks (certainly the zeros). I am looking for an approach more adapted to this setting (and preferably robust to outliers) and will appreciate suggestions.

The first idea that comes to mind is a similarity measure such as cosine. It's often used with sparse vectors (text represented as vectors over the vocabulary).
There are many options for distance/similarity measures:

Basic set measures like overlap coefficient or Jaccard
Entropy-based measures such as KL divergence.
...


Can You Purposely Bias A Clustering Model? We have a large amount (Billions) of high cardinality, mixed nominal & numerical data, and are performing some clustering on it as an experiment.  There is a small subset of these data, however, that we know should belong in its own cluster.

Is it possible to purposely make sure these data end up in their own cluster, while still clustering the remaining data?  If so, how does one approach doing so?

In constrained clustering you can provide examples of objects that should, or that must not, be in the same cluster.

This can be used, e.g., for model selection: run several times and return the result with the fewest violations. Or to guide cluster extraction from a hierarchical result.


NLP problem : Choosing the optimal parser for extracting quantitative values from text I have a clinical NLP problem for which I would need some help to establish a proper framework.

I am trying to extract different elements from echocardiography reports. Those elements are both quantitative and qualitative.

For example, quantitative elements are in the form :

  "LVEF : 40%", "LVIM : 2 mm/s" , "the tricuspid regurgitation is 4mm/s"

and qualitative elements are in the form :

  "The ventricular function is depressed", "We noticed reduced diastolic function"

Ultimately my objective is to obtain the following table for one note :

ID LVEF LVIM TR Diastolic Dysfunction
1  40   2    4  Reduced

My current framework to do so is to use the Quanteda package and Spacyr.

Tokenization of the text

  This is pretty straight forward.

Parsing

  This is where I am a little bit hesitant.
  I believe that it is key for this project to capture the sequence of the word as retrieving what number is associated with which parameter would be difficult otherwise (i.e. for example in a bag-of-word scenario, it would be impossible to know if 4 refers to the value of LVEF or LVIM for example).

  I do not know what would be the best way to parse the text to retrieve that information.

Dictionary use

  A lot of the echocardiogram parameters are registered in different ways. For example, 'LVEF' is documented as 'Left Ventricular Function', 'Heart Function', 'Left Function'.
  I constructed a dictionary mapping all the variations possible for a given concept.

  At what point of the NLP pipeline should the dictionary be used and how does Quanteda work with custom dictionaries ?

Thank you all!

This is an NER problem. Rather than you splitting your sentence to words and finding the right word from dict, I suggest you use an NER (may be spacy NER mentioned by @jindrich).

This NER will point out right block of information from the text your sentences.

Once you get an Entity then you can parse its value. If it is quantitative then it is easy to parse (with simple preprocessing) and if it is qualitative then you may have to convert string to numbers like one to 1 . There are free libraries for that.


Perceptron Learning Rule I am new to Machine Learning and Data Science. By spending some time online, I was able to understand the perceptron learning rule fairly well. But I am still clueless about how to apply it to a set of data. For example we may have the following values of $x_1$, $x_2$ and $d$ respectively:-

\begin{align}&(0.6 , 0.9 , 0)\\
&(-0.9 , 1.7 , 1)\\
&(0.1 , 1.4 , 1)\\
&(1.2 , 0.9 , 0)\end{align}

I can't think of how to begin.

I think we need to follow these rules.

$$W_i = W_i + \Delta W_i$$
$$\Delta W_i = \eta(d_i - y_i)$$
$$\text{ If} y_i = \sum w_ix_i \ge 0, y = 1$ \text{ else} y=0$$
$$x_0 (\text{Bias}) = 0 $$

Where $d_i$ is the target value, $y_i$ is the output value
$\eta$ is the learning rate and $x_i$ is the input value

Any help is appreciated. Thanks!

You don't want to follow those rules.
You have your matrix of observations and you multiply by a matrix of weights, in which, initially al the weights are assigned randomly.
A matrix of weights is composed in this way: each column represents a neuron and the rows the weight to which you multiply your features.
If you have k layers you'll have k matrices of weights.
At each iteration, what the DL framework allows you to do (You don't want to it manually), compares the real target to your predicted and computes a cost function(How far is the predicted from real?). Derivative is then computed respect to the weights, it is demonstrated that the direction of the steepest descent is obtained by updating the weights by a value of $W_i = W_i + \Delta W_i$ where: $\Delta W_i = n(d_i - y_i)$. It goes in this fashion by a number of iterations you defined.
I hope I gave you a hint to understand how it works.


R mice doesn't give a 'valid' sollution EDITED: See below for additional information..

TL;DR: How can I add missing data in a dataset like the sample in a way that it doesn't deviate much from the original dataset.

ORIGINAL:

I have a question about the mice package.

I am looking at a dataset like the example code:

set.seed(1337)
df <- data.frame(x = c(seq(100, 10, -4), seq(100, 20, -3), seq(100, 60, -3),
seq(80, 40, -3), seq(100, 10, -3))) %>% data.table()
df[sample(seq_len(nrow(df)), size = nrow(df) * 0.3), x := NA]
df$y <- seq(1,nrow(df),1)

Tmp <- mice(df, m = 1, maxit = 30, seed = 1337, print = FALSE)
Completed <- mice::complete(Tmp, 1)

plot(df$y, df$x, col = "blue")
points(Completed[is.na(df$x), ]$y, Completed[is.na(df$x), ]$x, col = "red")
legend(70, 30, legend = c("actual", "pred"), col = c("blue", "red"), lty = 1, cex = 0.5)

I am trying to fit the missing data points, but as you can see in the image below, the fit (red) isn't like I want it to be, because I would expect it to appear in between the blue line. Would anyone know how to 'fix' this?

For clearance: The data on the x-axis is interpreted as a datetime object while the value on the right is the current battery level (in percentage) of ONE device. The points where the blue dotted line stops and starts at a higher number is when the battery is charged.

UPDATE:
I have looked at the DMwR-package and especially at knnImputation. I was able to fit it to the sample set without any issue, see image below.

However it gave an issue when I tried to implement it with the real data set, because it threw an error:

Error in scale.default(xcomplete, dm[i, ], FALSE) : length of 'center' must equal the number of columns of 'x'

See sessionInfo below:

> sessionInfo()
R version 3.4.4 (2018-03-15)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: CentOS release 6.9 (Final)

attached base packages:
[1] grid      stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
 [1] lubridate_1.3.3     zoo_1.7-12          tidyr_0.6.0         pROC_1.13.0         randomForest_4.6-14 DMwR_0.4.1          lattice_0.20-35     stringr_1.0.0
 [9] openxlsx_3.0.0      tictoc_1.0          data.table_1.10.4-3 wlhive_1.1.9003     ggplot2_1.0.1       dplyr_0.5.0

After a lucky google search, I found the function na.interpolation from the imputeTS-package. After changing the dataset to a time series (XTS), I was able to impute the right data.

library(xts)
library(imputeTS)

set.seed(1337)
df <- data.frame(x = c(seq(100, 10, -4), seq(100, 20, -3), seq(100, 60, -3),
                       seq(80, 40, -3), seq(100, 10, -3))) %>% data.table()
df[sample(seq_len(nrow(df)), size = nrow(df) * 0.3), x := NA]
df$y <- seq(1,nrow(df),1)

ts <- xts(df[, -c("y")], as.Date(df$y))
head(ts)
#>              x
#> 1970-01-02 100
#> 1970-01-03  96
#> 1970-01-04  NA
#> 1970-01-05  88
#> 1970-01-06  NA
#> 1970-01-07  80

full <- na.interpolation(ts)
plot(full, type = "p")


Unsupervised encoding of categorical features I have multiple log records with discrete categorical features. Shape of my dataset is (100k, 24) My aim is to look for anomalies in these records. I am planning to cluster the data after encoding.

Before going forward with any analysis, the categorical features need to be encoded; I'd have ideally gone with LabelEncoder or OneHotEncoder but the issue is some features have > 40 possible values.

As of now, I've frequency encoded the variables but is there something I am missing and is there a better way of going about this?

High cardinality is a common issue that we run into when doing encoding for categorical variables. One hot encoding and label encoding are the most common approaches however label encoding maynot be ideal if the variable is not nominal (not inherently hierarchical) and one hot encoding will lead to curse of dimensionality when no of categories are high. REplacing with ratio or looking at spectral encoding is a good approach to circumvent this

Details of spectral encoding are :-
https://towardsdatascience.com/spectral-encoding-of-categorical-features-b4faebdf4a


Compute backpropagation I have the question which is mentioned in the above picture. It is trying to find the derivative of f with respect to weight matrix W1. Can any one help me how I can accomplish that?

I solved in this way :

You are missing one little step here. At the output node the backpropagation algorithm starts with the loss function $L$. You want to find out how much the weight $w_1$ impacts our loss function thus
$\frac{\partial L}{\partial w_1}$.
We can break this down with the chain rule as
$\frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial f} \frac{\partial f}{\partial w_1} = \frac{\partial L}{\partial f} \frac{\partial f}{\partial \sigma} \frac{\partial \sigma}{\partial w_1}$.
Why do we do this?
We do this to simplify the math. Because the $L$ is a function of $f$ and $f$ is a function of $\sigma$ and $\sigma$ is a function of the weights. But if we ignore the loss function for now.
How do we calculate $\frac{\partial f}{\partial w^2_{i, j}}$?
$\frac{\partial f}{\partial w^2_{i,j}} = \frac{\partial f}{\partial h^2_i} \frac{\partial h^2_i}{\partial w^2_1}$
$f = h_1^2 w_{1,1}^3 + h_2^2 w_{2,1}^3$
$\frac{\partial f}{\partial h^2_i} = w_{i, 1}^3$
$\frac{\partial h^2_i}{\partial w^2_1} = \sigma(W^2h^1) (1 - \sigma(W^2h^1))$
Backpropagation
Based on this answer we can see that $h_i^2$ is a function of the inputs from the previous layer and the activation at the neuron. Thus, we can substitute this function into our equation and continue taking the derivatives and we can see how the error propagates backwards through the network.
$\frac{\partial f}{\partial w^1_{i,j}} = \frac{\partial f}{\partial h^2_i} \frac{\partial h^2_i}{\partial h^1_i} \frac{\partial h^1_i}{\partial w^1_{i,j}}$

(Update)
The answer
This may not be correct, the work needs to be checked.
$f = \begin{bmatrix} w_{11}^3 & w_{21}^3 \end{bmatrix} \begin{bmatrix} h_{1}^2 \\ h_{2}^2 \end{bmatrix} = \textbf{W}^3\textbf{h}^2$
$\textbf{h}^2 = \begin{bmatrix} h_{1}^2 \\ h_{2}^2 \end{bmatrix} = \sigma(\begin{bmatrix} w_{11}^2 & w_{21}^2 \\ w_{21}^2 & w_{22}^2\end{bmatrix} \begin{bmatrix} h_{1}^1 \\ h_{2}^1 \end{bmatrix}) = \sigma(\textbf{W}^2\textbf{h}^1)$
$\textbf{h}^1 = \begin{bmatrix} h_{1}^1 \\ h_{2}^1 \end{bmatrix} = \sigma(\begin{bmatrix} w_{11}^1 & w_{21}^1 \\ w_{21}^1 & w_{22}^1\end{bmatrix} \begin{bmatrix} x_{1} \\ x_{2} \end{bmatrix}) = \sigma(\textbf{W}^1\textbf{X})$
Then using the chain rule
$\frac{\partial f}{\partial \textbf{W}^1} = \frac{\partial f}{\partial \textbf{h}^2} \frac{\partial \textbf{h}^2}{\partial \textbf{h}^1} \frac{\partial \textbf{h}^1}{\partial \textbf{W}^1} = \textbf{W}^3 * \sigma(\textbf{W}^2\textbf{h}^1)(1-\sigma(\textbf{W}^2\textbf{h}^1)) * \sigma(\textbf{W}^1\textbf{X})(1-\sigma(\textbf{W}^1\textbf{X}))$


Does it matter whether we put regularization parameter ($C$) with error or weight term in Kernel ridge regression? Kernel ridge regression associate a regularization parameter $C$ with weight term ($\beta$):

$\text{Minimize}: {KRR}=C\frac{1}{2} \left \|\beta\right\|^{2} + \frac{1}{2}\sum_{i=1}^{\mathcal{N}}\left\|e_i  \right \|_2^{2}  \\
 \text{Subject to}:\ {\beta^T\phi_i}=y_i - e_i, \text{  }i=1,2,...,\mathcal{N}$

If we associate $C$ with an error term as follows:

$\text{Minimize}: {KRR}=\frac{1}{2} \left \|\beta\right\|^{2} + C\frac{1}{2}\sum_{i=1}^{\mathcal{N}}\left\|e_i  \right \|_2^{2}  \\
 \text{Subject to}:\ {\beta^T\phi_i}=y_i - e_i, \text{  }i=1,2,...,\mathcal{N}$

then how this second formulation is different from the first one?

or

Can we associate $C$ either with weight term or error term in Kernel ridge regression?

Both formulations lead to the same solution if you correctly choose $C$ for both cost functions and if $C>0$.

If we have the regularized loss

$$J_1=\dfrac{1}{2}\sum_{n=1}^Ne_n^2+\dfrac{1}{2}C\sum_{k=0}^pw_k^2$$

we will have strong regularization for larger $C$ and small regularization for small positive $C$.

If we devide the loss $J_1$ by the positive $C$ we obtain the loss

$$J_2= \dfrac{1}{2C}\sum_{n=1}^Ne_n^2+\dfrac{1}{2}\sum_{k=0}^pw_k^2.$$

As $C>0$ we just scaled our loss funtion $J_1$, hence the minimum will not change. But the interpretation of $C$ will change if we replace it with its inverse as proposed in your question to obtain the loss

$$J_3= \dfrac{1}{2}\tilde{C}\sum_{n=1}^Ne_n^2+\dfrac{1}{2}\sum_{k=0}^pw_k^2.$$

With $\tilde{C}$ very small we will have strong regularization. And for large $\tilde{C}$ we will have small regularization. That is the reason why we sometimes call $\tilde{C}$ the inverse regularization parameter (e.g. see support vector machines).


Should we keep all channels when doing image classification? I am discovering the world of image recognition and now trying to build an image classifier. The set of images I have have the shape (101,101,3) which means that it has 3 channels.

If I'm not mistaken the channels corresponds to the Red, Green and Blue channels (Please correct me if I am wrong).

In which classification tasks should we keep the three channels ? I've seen people converting the Image to a grayscale so that the images shape will be (101,101,1). What difference does it make when you do so ?

I am aware that when you keep the 3 channels the computation needs grow with it, here I am more looking for an answer about the intuition behind keeping the 3 channels, move to grayscale or add new custom channels.

Well, it depends on your images and how you want to classify them.

If we take a really basic example, if you want to classify images by their color, you obviously should keep all 3 channels.

In the other hand, if you want to classify images by a shape. For example squares and circle, you don't necessarily need to use all 3 channels.

But changing the number of channel isn't the only preprocessing you can do on your images before classifying them.

You can also change the color space for example (the colorspace LAB can be usefull in some case) , or even use Morphological Image Processing methods to improve your image by, for exemple, bringing out the contrasts of your images or cleaning impurities.

All of this depends of your images, and what you want to do with them.


When is it necessary to use StandardScaler/MinMaxScaler on y_train and y_test? I have been through various kernels where scaling is done on y_train and y_test and many where there isn't. Is there any specific rule which should be followed when to or when not to do this?

As far as I know, scaling target values is not the common practice, though I do not expect it to change results much. Here is a more detailed discussion of this issue:

https://stats.stackexchange.com/questions/111467/is-it-necessary-to-scale-the-target-value-in-addition-to-scaling-features-for-re