Which definition of Likelihood function is correct? In the online version of the Deep Learning book on chapter 5 the estimator for likelihood function is defined as:

That is the product of individual probabilities. After taking the log it arrives at the log-likelihood funciton (Eq.1):

It then rescales the above by dividing it by m to obtain a version expressed as the expected value (Eq.2):

OK. Here is my first question:
The expected value is defined as

And so I think the expected value of log will be the below expression which is not the same as Eq.1 in the book.

The only way for this to be true(that is to divide the Eq.1 by m and arrive at Eq.2 as claimed by the book author) is for the probabilities of p(data) to be uniform. But this is also not a valid assumption.
So I don't see what I am missing here?

Next, the book argues that maximizing the above log-likelihood function (Eq.2) is same as minimizing the KL divergence:

Or more simply just minimizing the second term. And so the author says that either way we arrive at the same function as Eq.2.

On the other hand, from the Wikipedia page the cross entropy of two probability is defined as :

I can understand this definition as the expected value of log(q) which is same as the expression in the Eq.2. but not Eq.1
From the same Wiki page the likelihood definition is given as below which is different than the likelihood function definition from the book(above). Here the probability of q (model) has been raised to the number of occurrences; which then on taking log it is understandable to see it as the expected value.

So I am confused that first of all which definition of likelihood function is the correct one?
Given the definition from the Wikipedia I can understand that maximizing the log-likelihood function is same as minimizing the cross-entropy function. However I cannot arrive at the same conclusion from the definition of the likelihood and the log-likelihood function given in the book, for the reasons I explained above.

1 - Equivalence between $\mathbb{E}_{x\sim \hat{p}_{data}}\log(p_{model}(x;\theta))$ and $\frac{1}{m}\sum_{i=1}^m \log(p_{model}(x;\theta))$:
Yes, as you mentioned in the question, the key issue here is to realize that the following expectation is with respect to the empirical distribution defined by the training data ($\hat{p}_{data}$):
$$ \theta_{ML} = \arg\max_{\theta}\mathbb{E}_{x\sim \hat{p}_{data}} \log(p_{model}(x;\theta)) $$
By definition, an empirical distribution assigns a probability of $1/m$ on each of the $m$ points $\mathbb{X} = \{x^{(1)},x^{(2)},...,x^{(m)}\}$ $\to$ Thereby the value of $\hat{p}_{data}(x_i)=\frac{1}{m}f(x_i)$ where $f(x_i)$ represents the ocurrence frequency of a value $x_i$ in our dataset.
Note that a value $x_i$ may appear more than once in our dataset $\mathbb{X}$ $\Rightarrow f(x_i)$ can be $> 1$. This is why we can express the previous expectation as:
$$\begin{align}
\mathbb{E}_{x\sim \hat{p}_{data}} \log(p_{model}(x;\theta)) &= \sum_{i=1}^m \frac{1}{m}\log(p_{model}(x^{(i)};\theta))\\
\end{align}$$
Hence, the maximization of both expressions lead to the same value of $\theta_{ML}$

2 - Equivalence between the minimization of $D_{KL}(\hat{p}_{data} \Vert p_{model})$ and $H(\hat{p}_{data}, p_{model})$
As said in the question, the KL divergence of the probability distributions $\hat{p}_{data}$ (empirical distribution of $\mathbb{X}$) and $p_{model}$ (our statistical model that we are using to fit $\mathbb{X}$) is given by the next expression:
$$D_{\text{KL}}(\hat{p}_{data} \parallel p_{model}) = \mathbb{E}_{x\sim \hat{p}_{data}} [
 \log(\hat{p}_{data}(x)) - \log(p_{model}(x;\theta))]$$
Here, we can see that $\mathbb{E}_{x\sim \hat{p}_{data}}
 \log(\hat{p}_{data}(x))$ does not depend on $\theta$ (it only depends on the data generating process), so it can be trated as a constant. Hence we can adress the same problem of finding the value of $\theta$ that minimizes the cross entropy by minimizing this KL divergence, because this is the same as minimizing:
$$ \mathbb{E}_{x\sim \hat{p}_{data}} [- \log(p_{model}(x;\theta))] = - \sum_{i=1}^m \frac{1}{m}\log(p_{model}(x^{(i)};\theta))$$
Which is the definition of the cross entropy $H(\hat{p}_{data}, p_{model})$.

3 - Equivalence between the minimization of $H(\hat{p}_{data}, p_{model})$ and the maximization of $\sum_{i=1}^m \log(p_{model}(x;\theta))$
Here, we have to proof that
$$\arg\max_{\theta}\sum_{i=1}^m \log(p_{model}(x;\theta))$$
gives the same value of $\theta_{ML}$ than:
$$\begin{align}
\arg\min_{\theta}H(\hat{p}_{data}, p_{model}) &= \arg\min_{\theta}-\frac{1}{m}\sum_{i=1}^m \log(p_{model}(x^{(i)};\theta))\\
&= \arg\max_{\theta} \frac{1}{m}\sum_{i=1}^m \log(p_{model}(x^{(i)};\theta))
\end{align}$$
Both expressions are equivalent (they give the same value of $\theta_{ML}$) because a scale factor (such as $1/m$) does not affect in a minimization/ maximization problem.

4 - Equivalence between $\prod_{i=1}^m p_{model}(x^{(i)};\theta)$ and $\prod_{j} p_{model}(x_j;\theta)^{\#\text{ ocurrences of }j}$
This is because $i$ and $j$ represent different things:

$i$ represents each individual datapoint of a dataset $\mathbb{X}\Rightarrow x^{(i)}$ is a datapoint.
$j$ represents a certain value present in our dataset $\mathbb{X}\Rightarrow x_j$ is a certain value.

Given this, if a datapoint $x^{(i)}$ has the same value ($x_j$) of other $n$ datapoints, then we can express the likelihood of these datapoints in two equivalent ways:
$$ \prod_{i=1}^n p_{model}(x^{(i)};\theta) = p_{model}(x_j;\theta)^n$$
So, if we extend this reasoning to all the possible values ($x_j$) that our dataset can have, we reach the equivalency given by wikipedia:
$$\prod_{i=1}^m p_{model}(x^{(i)};\theta) = \prod_{j} p_{model}(x_j;\theta)^{\#\text{ ocurrences of }j}$$


What is meant by outliers in text data set. How to detect them? I know that outliers are present in data but their behaviour varies a lot from remaining data points. But today while learning about naive-Bayes  they mentioned that naive-Bayes can affected by the outliers. But which points in data set are termed as outliers and how do we identify them?

I define an outlier in the following ways.

It can be a wrong data entry (Eg. human typing error)
It can be a data that has values that are not relevant (Eg. an entry of total which is calculated as the sum of the above columns. This data can be misleading at times so it should be removed)
It can be a data entry that is all or most fields blank (Eg. a row in the data where all fields are blank. This row maybe not contributing anything to the analysis)
It may be extreme values which fall way out of the range of the other data (Eg. when we are calculating the age of Humans, anyone with age (say) above 120 years is an extreme case and that can be ignored depending upon our analysis goal)


Calculating Feature Importance of Time Series Data I am new to time-series modeling, and I was wondering what the standard way of quantifying feature importances are in a time-series setting? What types of models allow for the greatest interpretation of the feature space?

I am looking for something, which does not necessarily function like the Random Forest Regressor's feature importance call, but provides a similar insight.

For time series data,

Sensitivity analysis can help with overall Importance of a feature. For example, is  "Day of the week" a good feature for stock price forecasting. LIME is one approach that can help. Details :  https://arxiv.org/abs/1606.05386 . One simple way is to mask each feature and check the impact on model's performance.
Auto-corelation and Seasonality removal  (Details in tutorial at end of the answer)
SHAP : (SHapley Additive exPlanations) is good at identifying features that impact output with lag (https://medium.com/datadriveninvestor/time-step-wise-feature-importance-in-deep-learning-using-shap-e1c46a655455)

End to end example :
https://machinelearningmastery.com/feature-selection-time-series-forecasting-python/


.split() function doubt Here is a line of code

str(x).split(' ')[-1][1:]

The split function within its parenthesis has ' '.
Is it necessary, or we can also leave the split function with no arguments like

str(x).split()[-1][1:]

I think both works in the same way.

If the characters in x are separated by spaces, they work the same way. If there are multiple whitespaces between the chacacters, they work differently as str(x).split(' ') splits specifically by each space and str(x).split() splits by consecutive whitespace.

Example:

test_cases = ["a b c", "a   b   c"]
for x in test_cases:
    splitted_by_space = x.split(" ")
    splitted_by_consecutive_ws = x.split()
    print("x =", '"' + x + '"')
    print('x.split(" ") -->',  splitted_by_space)
    print('x.split()    -->',  splitted_by_consecutive_ws)
    print("-" * 50)

Output:

x = "a b c"
x.split(" ") --> ['a', 'b', 'c']
x.split()    --> ['a', 'b', 'c']
--------------------------------------------------
x = "a   b   c"
x.split(" ") --> ['a', '', '', 'b', '', '', 'c']
x.split()    --> ['a', 'b', 'c']
--------------------------------------------------

From the official documentation:

  str.split(sep=None, maxsplit=-1)

  If sep is not specified or is None, a different splitting algorithm is applied: runs of consecutive whitespace are regarded as a single separator, and the result will contain no empty strings at the start or end if the string has leading or trailing whitespace. Consequently, splitting an empty string or a string consisting of just whitespace with a None separator returns [].


how can we feed both data time series and non time series data together in machine learning classification model I have a dataset(IoT wearable dataset) composed of time-series and integer data; the objective of my task is to use the dataset for classification. Whilst current libraries in sktime accept multivariate time-series data as input, I do not believe they accept integer data. What would be the best practice in this case? I am currently considering converting the integer data time-series by repeating the integer value for the same number of increments as the other multivariate inputs, then applying random noise to the manufactured integer data that I converted to Timeseries- Can anyone comment on whether this is an appropriate method or if there are other more appropriate ways to approach the issue? Thanks in advance.'

You need two backbones: for time series, use LSTM/GRU and for non-time series, use 1D CNN or Linear layers. Once you get the final embeddings from both of them, concatenate the embeddings and finally feed to the classification layer.


Union grouping in bipartite graphs? I'm trying to figure out a good (and fast) solution to the following problem:

I have two models I'm working with, let's call them players and teams. A player can be on multiple teams and a team can have multiple players). I'm working on creating a UI element on a form that allows a user to select multiple teams (checkboxes). As the user is selecting (or deselecting) teams, I'd like to display the teams grouped by the players.

So for examples:

If the selected teams have no players that intersect, each team would have its own section.
If the user selects two teams and they have the same players, there would be one section containing the names of the two teams and all the players.
If TEAM_A has players [1, 2, 4, 5] and TEAM_B has players [1, 3, 5, 6]. There would be the following sections: SECTION_X = [TEAM_A, TEAM_B, 1, 5], SECTION_Y = [TEAM_A, 2, 3], SECTION _Z = [TEAM_B, 3, 5]

I hope that's clear. Essentially, I want to find the teams that players have in common and group by that. I was thinking maybe there is a way to do this by navigating a bipartite graph? Not exactly sure how though and I might be overthinking it. I was hoping to do this by creating some type of data structure on the server and using it on the client. I would love to hear your suggestions and I appreciate any help you can give!

Have you heard about the intersection graph? You can try to draw players as points, connections (team mates) as edges and teams as transparent coloured blobs on top.

As for your original question, I cannot understand your goal. I think your formulation is not well defined / incomplete. Suppose you have teams A [1,2] B [2,3] and C [1,3]. What do you want to display? Do you want to list the parts of the Venn diagram? I think that for more than 3 sets this can become more cumbersome than the bipartite graph itself = simple listing of team compositions.


Getting kinds on features that has high confidence on classfication Is there any way get a constraint on features (e.g., ear longer than 100cm and body weight>20kg and ...) or (...)) where it could say 90% guarantee it is a dog or cat? Say if we want to classify dog/cat.

Yes you can use a decision tree for example. These trees can have a stop criterion. For example if the leaf is too small in sample size or, and this is interesting for you, is above a pre-defined purity threshold. By discarding all the final leafs that were created due to not purity related conditions you should end up with a set of rules like you described.


From developper to data scientist I code a lot for web, games and some basic ML scripts. Now I would like to learn about data science.

This post is a good starting point but I would like some readings. I would like advices on books for a beginner (maths, tools, whatever).

I've found these ones from O'Reilly :

Machine Learning for Hackers, By Drew Conway, John Myles White
Agile Data Science, By Russell Jurney
R Cookbook, By Paul Teetor

(Sorry, my reputation is too low to let me post more than 1 link haha)

Do you recommand these readings ? Are there other must-reading books ?

Thanks.

I suggest this guide for a good introduction to the resources available to you to learn to do data science. You are starting at a good time, there are loads of excellent (and mostly free) resources to teach you this interesting field.


Can I not call my project a "Machine Learning" project if it doesn't involve a prediction at the end? I'm doing an analysis on some data involved with the NBA right now, and I'm nearing the end of the project but I need to decide if I want to include an official prediction for future data or not.

From what I've seen online, the steps to a Machine Learning project is Gathering Data, Data Prep, Observe Data, Train Model, Evaluate Model, then Predict (on unknown data).

The first 5 steps seem to all the same as a data analysis but a "data analysis project" would focus a lot more on the observe data part and derive conclusions from that. What I did was follow all the steps, derive conclusions from the data and graphs I've observed, I ran some models on the data as well to prove some correlation between the data, and to prove that a trained model can predict data.

However, at the end I don't think a prediction of the future is necessary for my project. If I don't include this "predicting" part of my analysis, can I not technically call that a "Machine Learning" project? Because all I did was look at and analyze data and graphs and derive a conclusion from that rather than concluding with a "with our trained model we can predict these trends"?

This is a big vague so let me know if I can clarify anything, thanks!

Could you talk a bit more about your model training and evaluation?

If you are viewing correlations, it sounds like exploratory data analysis to me.

Typically, I believe a machine learning model will be trying to predict a data point/variable. This could be in the form of classification or regression (i.e. did you do one of these?)

Just curious, why is it important to label as a machine learning project? If a class project, maybe you would be interested in your model predicting points, points per game, or win/loss ratio. Or maybe based on your model, you would want to be clustering players into different segments.

Good luck!


Reward negative derivative on linear regression I'm actually new to Data Science and I'm trying to make a simple linear regression with only one feature X ( which I added the feature log(X) before adding a polynomial features) on a motley dataset using Python an all the Data Science stack that comes with it (numpy, pandas, sci-kit learn, ...)

Here you can find a piece of code of my regression using scikitlearn:

def add_log(x):
    return np.concatenate((x, np.log(x)), axis=1)

 # Fetch the training set
_X = np.array(X).reshape(-1, 1) # X = [1, 26, 45, ..., 100, ..., 8000 ]
_Y = np.array(Y).reshape(-1, 1) # Y = [1206.78, 412.4, 20.8, ..., 1.34, ..., 0.034]
Y_train = _Y
X_train = add_log(_X) if use_log else _X

# Create the pipeline
steps = [
    ('scalar', StandardScaler()),
    ('poly', PolynomialFeatures(6)),
    ('model', Lasso(alpha=alpha, fit_intercept=True))
]

pipeline = Pipeline(steps)
pipeline.fit(X_train, Y_train)

My feature X can go between 1 to ~80 000 and Y can go between 0 and ~2M

There is one thing I know about the curve I should obtain is that it should always decrease so the derivative should be always negative

I make a little schema to explain what I expect vs what I have:

Therefore I would like to reward prediction where derivative is always negative even if my data suggest the opposite.

Is there a way to do that with sci-kit learn?
Or maybe I'm suggesting a bad solution to my problem and there is another way to obtain what I want ?

Thank you

It's a classic outlier. You could, for example, remove him or replace him with a new value(by interpolation).
You have many ways to work around this problem.

Links for you:

https://towardsdatascience.com/ways-to-detect-and-remove-the-outliers-404d16608dba

With some code:
https://towardsdatascience.com/5-ways-to-detect-outliers-that-every-data-scientist-should-know-python-code-70a54335a623


Train a model using multiple data sources I have to train a classification model to predict if a customer will buy a product or not. I have multiple (eg. 3 or 4) data sources. The variable distributions among the different data sources is quite different (eg. in the first one I have a vast majority of young people, while in the second one there are many adults). When the model will be used in production, the test dataset will be made of records from only one data sources (I don't know in advance which one).
My question is: which is the best way to combine these different sources? The data are not so much, so I cannot train an independent model for each data source. Can I just concatenate the data frame or do I have to perform some other kind of step?
Thank you!

Add all data together, and make sure you have features representing all possible insights. In your case one feature with age/maturity (young, adults...). Lets say you fit a decision tree (or Random Forest, gradient boosting...) the model will decide whether to do a split or not on this feature if it contains meaningful information.
If you combine you should be able to have more data and the models will work better.
In case you have different columns in data1 than in data2, consider leaving them as NaN, and then treating them properly.


Updating 1st dataframe columns from 2nd data frame coulmns I have two data frames df1 and df2 which look something like this.

    Name    Age    Sex
0   Sam      25     M
1   John     22     M
2   Larry    34     M
3   Mark     32     M

    Name    Age    Sex
0   Kelvin   51     M
1   Bob      26     M
2   Larry    37     M
3   Jerry    17     M
6   Nick     68     M
8   Sam      98     M
9   Jax      78     M
10  John     23     M
11  Mark     47     M
12  Jerry    68     M

My output should ideally be this:

    Name    Age    Sex
0   Sam      98     M
1   John     23     M
2   Larry    37     M
3   Mark     47     M

I have tried join and merge but the columns doesn't line up

Try the below code, it should do the trick for you

df1['Age'] = df1['Name'].replace(df2.set_index('Name')['Age'].dropna())
df1['Sex'] = df1['Name'].replace(df2.set_index('Name')['Sex'].dropna())
print(df1)


How do you effectively predict the top 20% most likely customers to churn from a dataset? I am looking to work out that if I have a dataset with 100,000 existing customers who didn't churn and 20,000 previous customers that churned in the past and the business objective is to target the 20% of customers most likely to churn within the business, how would that be done?
For example we would have to take this dataset and split it into a training and test set. Lets say the split is an 80/20 ratio for the training and test set respectively. That means that when we build our model on the 80% of the data, we can no longer use this data to see if any of the existing customers are likely to churn as we cannot evaluate a model on the data we have used to train it as it. We can only use the remaining 20% of the test set in order to evaluate our model and which customers are more likely to churn or not by considering the probabilities of each customer within the test set. What if there are existing customers who have a high chance of churning that we miss because they were in the training set?
Is what I have said above correct? Its very different from predicting, for example, if someone will default on their loan as you can train on all past data and then use this data to predict on new customers coming into the bank etc but with churn you want to predict on the customers who exist in the bank at the moment to avoid them leaving.
Any answers are greatly appreciated.

If you use k-fold cross validation, then every customer in your dataset will lie in the test set of one of the k folds. That might help alleviate the issue at hand. Finally you can train your model against your entire train set and run inferences against net new customers.


Papers on tree based and linear models I am looking for papers which could give a greater understanding on differences between Tree based and linear based models.

Anybody can gove me some pointers?

I suggest chapters „3 Linear Regression“ and „8 Tree-Based Methods“ in An introduction to statistical learning


How to extract crucial features to create an image Imagine, you have a dataset containing pictures of (example only, just to explain the task) cats and dogs. The data set is labeled, so we can train using supervised learning algorithms.

My goal is to make a cat from a dog. How to do it? For now, I have a couple of ideas which I can share:

Use a convolutional autoencoder and train on cats, then give a
picture of a dog and see the result (I suppose, it will show the
most "similar" cat, so the goal is reached)
Use an algorithm like GAN to transfer "style". I have no idea
whether it is possible or not, but looks like a working idea

Which approaches could you recommend to try out?

This is a bit similar, merge an image with style of another https://github.com/jcjohnson/neural-style

Maybe with the GAN you can choose cases where the discriminator predicts 50/50 to be cat or dog


How to measure variable contribution to an observation in a non-linear model? Based on my model, if I decline someone due to their score, it should be able to provide some reasoning as to which variables mainly contributed to the decision to decline.

Typically in Logistic Regression models, this is a simple exercise where you calculate (Beta * X) for each variable and pick 1 or 2 variables which caused the biggest score drop.

However, this isn't very straightforward for non-linear models. I would appreciate any ideas on handling something like this. Thanks.

Try LIME. In other words, compute local representation of your global highly nonlinear model for a given observation. This local proxy should be simple enough to be explainable as well. Decision trees or simple linear regressions are good choices.


Which Regression Machine Learning algorithm to use in Azure for statistical prediction/forecasting I've a golf website that pulls a load of statistical data and makes forecasts for upcoming events.

I'm trying to improve the algorithm and have been looking into Azure's ML Studio. I've lots of data about previous events and stats the competing golfers had and the time of the event so I'm confident I need to be using some sort of regression based algorithm but I'm not sure which one, there's a few that look appropriate?

Any advice would be greatly appreciated!
 Thanks

If I had to choose, it would be one of the last three as they are more sophisticated. If you have some time, try them all out, sweep the hyperparameters, and compare the results. This free EdX course has a segment where you do that comparison of different models in Azure ML:

https://www.edx.org/course/principles-machine-learning-microsoft-dat203-2x-4


Failure tolerant factor coding There are a lot of ml-algorithms which cannot directly deal with categorical variables.
A very common solution is to apply binary (dummy-) coding to still properly handle the categorical nature of the data.

Very often e.g. in sk-learn or apache-spark the actual dummy-coder can only handle numeric values. So label-encoding needs to be performed beforehand.

In a real live ml-scenario, the fitted model will encounter new and formerly not known data. Usually, such a label-encoder (string-indexer) for spark has the option to either skip (ignore) a row of data which contains any unknown value or to throw an error. If multiple values require coding this can lead to a big loss of "new" data.

Are there any approaches which "tolerate" up to x new values per row and still properly evaluate the fitted pipeline?

An example for spark string-indexing + dummy-coding is shown below.

val df = spark.createDataFrame(Seq(
  (0, "a"),
  (1, "b"),
  (2, "c"),
  (3, "a"),
  (4, "a"),
  (5, "c")
)).toDF("id", "category")

val indexer = new StringIndexer()
  .setInputCol("category")
  .setOutputCol("categoryIndex")
  .fit(df)
val indexed = indexer.transform(df)

val encoder = new OneHotEncoder()
  .setInputCol("categoryIndex")
  .setOutputCol("categoryVec")
val encoded = encoder.transform(indexed)
encoded.select("id", "categoryVec").show()

  http://spark.apache.org/docs/latest/ml-features.html#onehotencoder

For a categorical variable, if the fitted model encounters a previously "unseen" category, i.e. it did not exist in the training set when the model was trained; then, you should skip that record. You could also opt to throw an error so that you're notified of the existence of new categories and can re-train the model based on that trigger.

If you skip the records with new categories, you would still be able to evaluate the pipeline successfully. That may be the preferred option for a fully automated production setup.

The one-hot encoding creates a new "column" of data for a new category and if this hasn't been used for training, the ML algorithm has no way of knowing how to use the new dummy variable.

In your sample code, each of the categories are encoded into new variables e.g. is_category_a = 0/1, is_category_b = 0/1 , is_category_c = 0/1, etc. If such a model is sent data with a new category "d", then it would be encoded in another column called is_category_d = 0/1 , but the model would ignore this column (or throw an error) since it doesn't expect its input matrix to contain is_category_d.

Lets assume you've fit a linear regression model with the coefficients as:
$$ y = 1 + 2.is\_category\_a + 3.is\_category\_b + 4.is\_category\_c  $$
Now, when you try to evaluate a record with new category = "d", then, the model is not able to use the new dummy variable since it doesn't have a coefficient for is_category_d .

Hence, so you should not "tolerate" new categories and process such records.


Does PCA decrease the feature on my Data set or just decrease the dimension? I'm new in AI and sorry if my question is simple. I have a data set and want to use PCA to decrease the feature but after some research on the internet I'm confused about decreasing dimensions and features.

As an example I have a data set with 50 rows and 10 columns, if I use PCA it will reduce a data set with 50x5 (as an example) or 50x10 and just removed some dimensions?

I want to do it in MATLAB and want to use PCA function and don't want to write PCA function by myself.

What is the PCA parameters in MATLAB to decrease the feature? It's a lot of parameters and confused me.

From the documentation:

  coeff = pca(X) returns the principal component coefficients, also known as loadings, for the $n$-by-$p$ data matrix X. Rows of X correspond to observations and columns correspond to variables. The coefficient matrix is $p$-by-$p$. Each column of coeff contains coefficients for one principal component, and the columns are in descending order of component variance. By default, pca centers the data and uses the singular value decomposition (SVD) algorithm.

The values in coef represent the transformation from the original features (rows of coef) to the principal components (columns of coef).  You'll want to keep only the first $k$ columns, then multiply your data matrix by this matrix.

Your features are the dimensions that your data lives in, so number of features and dimension are the same.  Very roughly speaking, PCA rotates the the feature axes to align to the most significant directions rather than the original feature directions, then selects only the most significant directions to keep, thus reducing the dimensionality of your problem.  But it also means your new columns won't represent pure features anymore, but linear combinations of them.

If you keep all of the new columns, you won't reduce dimensionality; but your new features will be ordered in such a way that the first ones capture the most variance of your data.


How to know the state-of-the-art recommended approaches for data science? Data science, AI, NLP, and visualization are changing so fast. I wonder if there is a way/blog that shares the latest updates and recommended using certain techniques or avoid using others. For example, many NLP books are old and they would provide examples using TF-IDF. However, nowadays there are much better approaches but they are also changing fast. I am hoping to find some source that would say use these new techniques and avoid using those old techniques. Searching the web can help, but will bring back a lot of noise.

A few comments:

There is no scientific domain of "data science", instead there are multiple fields which are related to data science: statistics, ML, NLP, computer vision, signal processing... and a lot of other fields which overlap and/or focus on specific applications, for instance bioinformatics. All of these domains are highly active and specialized, so it would just be impossible to monitor every possible advance.
There is no unique recommended way: first, people disagree all the time about the best way to do X. Second, it's very rare that a method would become completely obsolete. For example TFIDF still makes sense in many use cases, with low-resource languages or when there are efficiency constraints for instance.
In order to comprehensively follow the state of the art, one would have to follow the research publications. At best it's doable for a specific domain, for example one can more or less get an idea of what happens in NLP by browsing through the main conferences. A more realistic option is to wait for the advances to reach the mainstream professionals, for example by browsing regularly through DataScienceSE and/or CrossValidated.
Final suggestion: old books are very useful to fully understand why/how things are done a certain way. We often see errors here on DataScienceSE which are due to people trying to apply methods without understanding them.


How to reduce position changes after dimensionality reduction? Disclaimer: I'm a machine learning beginner.

I'm working on visualizing high dimensional data (text as tdidf vectors) into the 2D-space. My goal is to label/modify those data points and recomputing their positions after the modification and updating the 2D-plot. The logic already works, but each iterative visualization is very different from the previous one even though only 1 out of 28.000 features in 1 data point changed.

Some details about the project:

~1000 text documents/data points
~28.000 tfidf vector features each
must compute pretty quickly (let's say < 3s) due to its interactive nature

Here are 2 images to illustrate the problem:

Step 1:

Step 2:

I have tried several dimensionality reduction algorithms including MDS, PCA, tsne, UMAP, LSI and Autoencoder. The best results regarding computing time and visual representation I got with UMAP, so I sticked with it for the most part.

Skimming some research papers I found this one with a similar problem (small change in high dimension resulting in big change in 2D):
https://ieeexplore.ieee.org/document/7539329
In summary, they use t-sne to initialize each iterative step with the result of the first step.

First: How would I go about achieving this in actual code? Is this related to tsne's random_state?

Second: Is it possible to apply that strategy to other algorithms like UMAP? tsne takes way longer and wouldn't really fit into the interactive use case.

Or is there some better solution I haven't thought of for this problem?

You can initialize a UMAP embedding with a custom set of initial positions, so potentially you can initialise step 2 with the embedding from step 1 (with random positions for the new points).


Understanding experiments in Continual Learning Via paper Continual Learning Through Synaptic Intelligence, I see this figure for Split MNIST benchmark, but there is a point I can get.

Here there are 5 tasks, and finally we summarize the average accuracy over the 5 tasks.

Here, how the tasks are performed. Does they perform sequentially when first we learn how to categorize 0 and 1, then in the next task we expect that the model can also categorize 2 and 3, 4 and 5 and so on.

And here another question is, in the horizontal axes of each graph, there are 5 tasks, why we perform, for example Task 1 (0 and 1) on 5 tasks. Could someone clarify this point for me?

In the figure, the class-accuracy for each of the two classes per task is shown (y-axis) as the number of different tasks learned is increased (x-axis).
So in the first graph, we see that as the model learns subsequent tasks (task 1: distinguishing 0-1, task 2: distinguishing 2-3, etc), the classification accuracy of the model on '0's remains near 100%, whereas the classification accuracy of the model on '1's decreases.


Convert graph database relationships into node properties I have a neo4j database that stores forum posts. Replies to specific posts are encoded using a replyto relationship. I am trying to retrieve all the information from the database, convert the relationships into node properties, then output all the nodes with the new replyto property in JSON format. I think the replyto property in the JSON object should be an array of all the post IDs that are replies to the given post, but please suggest something better if you think of it.

What cypher query should I use to retrieve this data and how should I convert the relationships into node properties using javascript?

The cypher query that ended up working for me was MATCH (n) OPTIONAL MATCH (n)-[r]->(m) RETURN n,r,m

function makeAllBlocksQuery(cb){
  db.cypher({
    query: "MATCH (n) OPTIONAL MATCH (n)-[r]->(m) RETURN n,r,m",
  }, function(err,results){
    var processedResults = handleAllBlocks(err, results);
    cb(processedResults);
  });
}

function handleAllBlocks(err, results) {
    if (err) throw err;
    if (!results) {
      console.log('No blocks found.');
      return [];
    } else {
      //first, form [_fromId, _toId] relationshipArray
      var relationshipArray = new Array();
      for (count = 0; count < Object.keys(results).length; count++) {
        if(results[count].r !== null) {
          var relationship = [results[count].r._fromId, results[count].r._toId];
          relationshipArray.push(relationship);
        }
      }

      var resultArray = new Array();
      var repliedIdsArray = new Array();
      //add nodes found in results to resultArray
      //and add id property to properties for later
      for (count = 0; count < Object.keys(results).length; count++) {
        results[count].n.properties.id = results[count].n._id;

        //iterate through the relationshipArray for each result
        //and add all the replied Ids to the repliedIdsArray
        for (relationshipcount = 0; relationshipcount < relationshipArray.length; relationshipcount++){
          if (relationshipArray[relationshipcount][1] == results[count].n.properties.id) {
            repliedIdsArray.push(relationshipArray[relationshipcount][0]);
          }
        }
        //add the repliedIdsArray to the result's properties
        results[count].n.properties.hasreplyto = repliedIdsArray;
        repliedIdsArray = [];
        //stringify so it can be sent to client more easily
        resultArray.push(JSON.stringify(results[count].n.properties));
      }
      return resultArray;
    }
}


Idf values of English words I'm working on keyword/phrase extraction from a single document. I started by doing term frequency analysis, but this returns words like "new" which aren't very helpful. So I want to penalize the common words and phrases, for which we normally use idf (inverse document frequency). But since it's for a single document, I'm not sure how to do idf analysis.

Is it possible to use tf-idf method with pre-calculated idf values for (all?) words?
And are such values available somewhere?

I don't believe that there are any precalculated idf values out there. Inverse Document Frequency (idf) is the inverse of the number of documents in which a particular word appears in your corpus. If you only have one document, I'm afraid that value is simply 1.

However, if you are looking to get rid of words such as the, as, and it which don't carry much meaning, nltk in Python has some useful tools to remove these "stop words" from your document and might help you.

Here is a helpful example.


Resources to learn Tensorflow and Keras What all should I learn in sequence to be expert in Tensorflow and Google Keras provided I know Python and Numpy, Pandas, Pyplot basics? Please provide the resources too as most online tutorials are chargeable

Take a look at professor Andrew Ng's course about deep learning. Its homework is written in Keras and TensorFlow. It is a very good point to begin.

I guess it is not fair to mention non-free lectures because the owners of other non-free lectures may get disappointed but take a look at complete guide to TensorFlow for deep learning with python, which is very useful.

I have also found these lectures from Stanford university very useful. Furthermore, you can take a look at the official paper of google research team about TensorFlow where you can read TensorFlow is an interface for expressing machine learning
algorithms, and an implementation for executing such algorithms.
A computation expressed using TensorFlow can be
executed with little or no change on a wide variety of heterogeneous
systems, ranging from mobile devices such as phones
and tablets up to large-scale distributed systems of hundreds
of machines and thousands of computational devices such as
GPU cards. The system is flexible and can be used to express
a wide variety of algorithms, including training and inference
algorithms for deep neural network models, and it has been
used for conducting research and for deploying machine learning
systems into production across more than a dozen areas of
computer science and other fields, including speech recognition,
computer vision, robotics, information retrieval, natural
language processing, geographic information extraction, and
computational drug discovery. This paper describes the TensorFlow
interface and an implementation of that interface that
we have built at Google. The TensorFlow API and a reference
implementation were released as an open-source package under
the Apache 2.0 license in November, 2015 and are available at
www.tensorflow.org..


Why is pandas corr() deleting columns? I'm doing a basic correlation analysis but for some reason pandas corr() is deleting columns, not sure why.

import pandas as pd
data = pd.read_csv("data.csv")
print(len(data.columns))
print(len(data.corr().columns))

Output:

100
64

Pearson's correlation is the default correlation used with Pandas corr method.

Categorical features ( not numerical ) are ignored during this process due to their nature of not being continuous. It makes no sense to say if categorical_var1 is increased by one , categorical_var2 also increases by X ( X's value depends on the correlation between the 2 variables ).

That's why you only see numerical variables! There are other statistical tests you can apply to categorical variables to better understand them.

Note : some columns may appear as numerical at first glance, but a string may be there due to an input mistake, or simply when the formatting of the file was done, that column type was set to 'Object'. Make sure to test the values in your supposedly numerical columns and apply astype to set them back to int or float


Non-prediction Applications of Machine learning Prediction seems to be the dominant theme of machine learning. Most algorithms have fit and predict functions so that model can be created which can predict outcomes or other parameters of interest from new set of features.

What are non-prediction applications of machine learning?

The most obvious applications are indeed the supervised learning approaches (surrogate models, prediction). But there is much more than that! Other usual applications include:

Clustering: This can be seen as a different kind of prediction, but not in the classic supervised learning fashion. For instance, I have been using a clustering algorithm on a 3D geometry (CAD file) to make almost adjacent elements become actually aligned.
Anomaly detection
Study of extreme values (how to learn the behaviour of extreme values without actually observing such values)
Feature importance (determine which sets of features impacts the result the most) or, more generally, data mining
Inference (use a dataset to improve the prior model prediction)

Of course, most of these applications are more or less close to prediction, but not trivial prediction.


How to calculate customer purchase interval and predict next purchase in python? Suppose we have a data set consists of columns

  TransactionId, CardNo, TransactionDate

then how can we calculate the customer purchase interval (means if customer A purchased on Jan 1st and after 10 days he again purchased, and then he again purchased after 15 days.) and how to predict the next visit of customer A by analysing the purchasing intervals of customer A.

Any help will be appreciated.

With Lifetimes it's very straight forward to implement.


Mathematics major for data science So I'm a recent transfer 2nd year student from Computer Science major to Mathematics major. Though I do have a bit of an issue here. I can choose between the applied mathematics, pure mathematics and statistics concentrations.

Along with this major, I'm doing a minor in Data Science with courses focused on Economics and Statistics.

In the future, I'm interested in doing a master's degree in Data Science and a career in data analysis for businesses.

Although I know any degree can be used to get into graduate school, I still want to which would be most beneficial for education in the future as well as opportunities for internships, research, and jobs.

Thank you!

If you know that you want to become a data scientist, you can pretty much rule out pure mathematics.  Note, I'm not saying that pure mathematicians cannot become data scientists, but it's not the most natural transition.  Between the other two branches, stats is probably the most natural path.  Both will having you think about applying math to answer real world problems, but stats is very specifically geared towards focusing on larger scale data analysis.

EDIT:
@rocinante mentions the need for software skills, and suggests CS over econ as a minor.  I would say this really depends; if you are in a bigger data science team, you'll probably work alongside dedicated programers, who will be able to implement your analytics more efficiently than you would be expected to as an analyst.  If you know that you want to apply D.S. techniques to either finance or business, domain specific knowledge is helpful.  Additionally, if you know that it is your desire to end up in data science, you'll be able to look for ways to use programing along the way, and keep those skills sharp.


What are the ways to identify a good attribute test while constructing a decision tree? I'm working through a decision tree by hand to learn it. From my research, I have found the following three ways of determining which variables to split on:

Minimum remaining values - The variable with the fewest legal values is chosen
Degree heuristic - The variable with the most constraints on remaining values
Least Constraining value - The variable that rules out the fewest remaining values in the remaining variables

Do I have these right? What are some other ways of determining the splits?

You might have mixed up CSP (Constrained Satisfaction Problem) search trees and decision trees:

CSP search trees

'Minimum remaining values', 'Degree Heuristic' and 'Least Constraining Value' are used to solve CSPs and not in decision trees (i.e. we are talking symbolic AI here and not a sub-symbolic AI method like decision trees).

However, as far as I know the 'least constraining value' approach is not used to select a variable but after having selected one to choose the order to inspect its values.

So it could go rather like this:

1. Choose a variable applying 'minimum remaining values'
2. If step 1 fails select a variable applying the 'degree heuristic'
3. For the selected variable select its values ordered according to 'least constraining value'

Besides the ones you mentioned you could go the really simple route: just pick 'the next' variable or pick randomly.

Also see section 6.3.1 in 'Artificial Intelligence: A Modern Approach' by Russel and Norvig.

Decision trees

In contrast decision trees are a machine learning method. Here you usually choose the best splits for the tree using a greedy algorithm providing the biggest ad-hoc gain (see below list for how to define 'gain'). Typical criteria for the greedy split are:

Classification

Information gain (based on entropy)
Gini score
Classification error (usually not used when building the tree)

Regression

Squared error

Also see sections 9.2.2 and 9.2.3 in 'The Elements of statistical learning' by Hastie, Tibshirani and Friedman.


Residual plot is tight to zero, but low R2 score Hi I am a beginner at data science, and currently trying to use Gradient Boost Regressor to predict car price based on several attribute such as machine capacity, car brand, car type etc.

I am following this tutorial: https://github.com/PaacMaan/cars-price-predictor/blob/master/cars_price_predictor.ipynb

I got a low R2 score and plotted the residual vs predicted value, what i am confused with is even though my residual value is close to zero (as showed on the graph) my r2_score is low. Can anyone kindly explain this to me?

here is my r2 score

r2_score: 0.38

and the code i am using

car_df_encoded = pd.get_dummies(car_df_encoded, columns=['Store_Loc','Brand','Type','Transmission'], prefix = ['Store_Loc','Brand','Type','Transmission'])
price= car_df_encoded.pop('Price')

x_train, x_test, y_train, y_test = train_test_split(car_df_encoded, harga, test_size=0.25)

gbr = GradientBoostingRegressor(loss='huber', max_depth=5, max_features=None,min_samples_leaf=8, min_samples_split=6, n_estimators=150,random_state=42)
gbr.fit (x_train, y_train)
predicted = gbr.predict(x_test)
residual = y_test - predicted

Tried using cross validation as well

scores = cross_val_score(gbr, car_df_encoded, harga, cv=4, scoring='r2')
print('\nMean Score:')
print(scores.mean())

and my mean r2 score is

Cross Validation Scores:
[0.76621245 0.41771035 0.35744057 0.73955563]

Mean Score:
0.5702297501214788

According to the plot there is a huge outlier in your residual. Plus taking into account that your cross-validation sometimes shows quite good results (0.77, 0.74) and sometimes not so great (0.42, 0.36) I would first try to remove that outlier from your data and see what you get.


Standardization and Normalization Which and all Machine Learning algorithms needs the data to be standardised/normalised before feeding into the model. How do we determine whether the particular model/data needs to be  standardised/normalised.

Thank you.

Whenever you have features that they have different scale and it is significant for some features, you should standardize your feature. Take a look at here.


Including identifier in machine learning model as feature vs separate model for every identifier I am new to machine learning and i am building a model to predict number of customers for the model branch at specific  hour/season/other feature.

I know it will be bad idea to pit id(branch_id in my case) into model but customer count in this case hugely depend on which branch it is so i cannot exclude it.

I can think of two solutions, i am not sure which one is right and what is the best practice.

Create dummy variable(one hot encoding to avoid wieghing one id more than other) for all branch ids,but since i have 600 unique branch ids my features will go up-to 600+rest_of_features.
Learn a separate model for each of the branch(600 models), i am not sure if it is right approach and also i am not very familiar with this approach and it will be very time consuming.

Looking for the suggestion

Example of the data is below

    +-----------+------+-----------+-----------+-------------------+
    | branch_id | hour | feature_2 | feature_3 | Count of customer |
    +-----------+------+-----------+-----------+-------------------+
    |         1 |   12 |        .. |        .. |                19 |
    |         1 |   01 |        .. |        .. |                25 |
    |         2 |   23 |        .. |        .. |                14 |
    |         2 |   01 |        .. |        .. |                 5 |
    +-----------+------+-----------+-----------+-------------------+

In my opinion including id as a feature will not make sense at all, because the model will treat the id as a numeric value which will decrease the model performance, because there should be no connection how big the id is and how many customers there are for that id.

Option 2 can make sense if you have enough data for every branch.

My suggestion will be to look deep into your features and try to find a feature which will replace the branch id. Let's say the number of supporting desks in a branch or the location of a branch as a categorical value. If you find enough features that can describe the specifics of branches, then no need to include ids or to do it separately.


How to make smaller categories with factor character variables I have this data set with consist of ISO3166 Alpha-2 codes for countries.
Example: DE, AD, AE etc
They are coded as factor variables in R and there are about 173 observations.

Now because there are too many and this would just overwhelm a boxplot, I want to make a contingency table with other variables by condensing the codes and create shorter categories (also coded as factors) with the codes, for example, having

DE, RE, ED, FR->  Europe

CA, US->          North America

VF, HG, HY, TY->  South America

HG, TY, UT,FT ->    Africa

How can I do that because I tried a few things that did not work.

Thank you!

P.S  These codes are made up codes but it is just to illustrate.

While you are trying this in R, I have a solution in Pyhton. You could use a similar logic in R:

# Say this is your list:
countries = ['IN', 'DE', 'US', 'UK']

# Create a list of countries in different continents like this:
asia = ['IN', 'NP']
america = ['US', 'CD']
europe = ['DE', 'UK', 'FR']

# You can then map your list with these values using a definition:
for i in range(len(countries)):
    if countries[i] in asia:
        countries[i] = 'Asia'
    elif countries[i] in america:
        countries[i] = 'America'
    elif countries[i] in europe:
        countries[i] = 'Europe'
    else:
        countries[i] = 'Others'

# Now check your list
countries
['Asia', 'Europe', 'America', 'Europe']

Once this list is ready, you could create your boxplot.

Hope this helps.


How to create dictionaries out of pandas dataframes? I import three csv-files with characteristics of countries (rows) and years (columns):

country_data_m = 'country_data_m.csv'
m_year = pd.read_csv(country_data_m, nrows=161, index_col=0, header=0, sep=';', na_values=[""])

country_data_e = 'country_data_e.csv'
e_year = pd.read_csv(country_data_e, nrows=161, index_col=0, header=0, sep=';', na_values=[""])

country_data_i = 'country_data_i.csv'
i_year = pd.read_csv(country_data_i, nrows=161, index_col=0, header=0, sep=';', na_values=[""])

The datasets look like this:

                                       1995          1996          1997  \
Afghanistan                              NaN           NaN           NaN
Albania                                  NaN           NaN           NaN
Angola                          5.538749e+09  7.526447e+09  7.648377e+09
Antigua and Barbuda             5.772807e+08  6.337306e+08  6.806171e+08

                                  1995    1996    1997    1998    1999  \
Afghanistan                        NaN     NaN     NaN     NaN     NaN
Albania                            NaN     NaN     NaN     NaN     NaN
Angola                          0.8565  0.8369  0.8173  0.7976  0.7777
Antigua and Barbuda             0.6957  0.6352  0.6513  0.6401  0.6171

                                  1995    1996    1997    1998    1999  \
Afghanistan                        NaN     NaN     NaN     NaN     NaN
Albania                            NaN     NaN     NaN     NaN     NaN
Angola                          0.0612  0.0626  0.0641  0.0655  0.0670
Antigua and Barbuda             0.1852  0.2264  0.2147  0.2147  0.2030

For each country, I need a dictionary, where the key is the year, and where the values are the variables from the three different datasets. So far, I tried this code:

afghanistan = {m_year.loc["Afghanistan", (year)],e_year.loc["Afghanistan", (year)], i_year.loc["Afghanistan", (year)]): year for year in range(1995, 2017)}

albania = {m_year.loc["Albania", (year)],e_year.loc["Albania", (year)], i_year.loc["Albania", (year)]): year for year in range(1995, 2017)}
...
zimbabwe = {m_year.loc["Zimbabwe", (year)],e_year.loc["Zimbabwe", (year)], i_year.loc["Zimbabwe", (year)]): year for year in range(1995, 2017)}

However, the code cannot find the year in the dataframes and gives me the following error:

TypeError: cannot do label indexing on <class 'pandas.core.indexes.base.Index'> with these indexers [1995] of <class 'int'>

Any help would be very much appreciated!

I have one solution to your problem. please be careful with the names of the columns for every step. You may have different ones:

Concat all transposed data frames

z=pd.concat([m_year.T,i_year.T,e_year.T])

Then melt them:

z=z.reset_index().melt(id_vars='index')
print(z.head())

  index      country value
0  1995  Afghanistan   nan
1  1996  Afghanistan   nan
2  1997  Afghanistan   nan
3  1995  Afghanistan   nan
4  1996  Afghanistan   nan

then do this:

z=pd.DataFrame(z.groupby(['country','index'])['value'].apply(lambda x: [i for i in x])).reset_index()

print(z.head)

       country  index            value
0  Afghanistan   1995  [nan, nan, nan]
1  Afghanistan   1996  [nan, nan, nan]
2  Afghanistan   1997  [nan, nan, nan]
3  Afghanistan   1998       [nan, nan]
4  Afghanistan   1999       [nan, nan]

Fianly, a for loop:

final_dict={}
for i in z.country.unique():

final_dict[i]=z[z['country']==i].drop('country',axis=1).set_index('index').T.to_dict()

it will return this:

{'Afghanistan': {1995: {'value': [nan, nan, nan]},
  1996: {'value': [nan, nan, nan]},
  1997: {'value': [nan, nan, nan]},
  1998: {'value': [nan, nan]},
  1999: {'value': [nan, nan]}},
 'Albania': {1995: {'value': [nan, nan, nan]},
  1996: {'value': [nan, nan, nan]},
  1997: {'value': [nan, nan, nan]},
  1998: {'value': [nan, nan]},
  1999: {'value': [nan, nan]}},
 'Angola': {1995: {'value': [5538749000.0, 0.0612, 0.8565]},
  1996: {'value': [7526447000.0, 0.0626, 0.8369]},
  1997: {'value': [7648377000.0, 0.0641, 0.8173]},
  1998: {'value': [0.0655, 0.7976]},
  1999: {'value': [0.067, 0.7777]}}}

And then with list comprehetion we can transform it into your final dictionary :

final={i:{i:j['value'] for i,j in final_dict[i].items()} for i in final_dict.keys()}

final

{'Afghanistan': {1995: [nan, nan, nan],
  1996: [nan, nan, nan],
  1997: [nan, nan, nan],
  1998: [nan, nan],
  1999: [nan, nan]},
 'Albania': {1995: [nan, nan, nan],
  1996: [nan, nan, nan],
  1997: [nan, nan, nan],
  1998: [nan, nan],
  1999: [nan, nan]},
 'Angola': {1995: [5538749000.0, 0.0612, 0.8565],
  1996: [7526447000.0, 0.0626, 0.8369],
  1997: [7648377000.0, 0.0641, 0.8173],
  1998: [0.0655, 0.7976],
  1999: [0.067, 0.7777]}}


Random Forest vs K Nearest Neighbor as non linear classifier When classes are non-linearly separable, which of the following methods performs better?
Choose correct one :-

Logistic Regression
Random Forest
K Nearest Neighbor Classification
Linear Regression over most important features

Trail : I think  Random Forest and K Nearest Neighbor Classification are the only non-linear classifiers here. But which one performed better? how to judge here?

It is unusual to have such a bad question as an assignement. Maybe it's just misguided phrasing by the question author, but I'm pretty sure that the vast majority of experts in ML consider the idea of "best ML algorithm in general" as a mistake.
First let me explain why this is a bad question: in ML there is no way to guarantee that a particular algorithm/method performs better than any other on any dataset in general. There can be exceptions under very specific constraints, but there is no algorithm which is universally the best for non-linearly separable classes. There is actually a theoretical result called the No Free Lunch Theorem which is often interpreted as a proof that there cannot be a universally best classification/regression algorithm.
Now that the context of this bad question is clearly established, as a subjective personal opinion my answer would be based on interpreting the question as: which algorithm is the least sensitive to linear separability? In this context I would say kNN because this method is not concerned at all about linear separability: a new instance is classified based on its closest instances in the training data, so it does not rely on separating the data points into groups at all.


Does the "laplacian" kernel used in the SVM context come from a Hilbert space inner product? The "kernel" $k(x,y) = e^{-\|x-y\|} $ is used in the context of SVM. Here $x,y \in \mathbb R^n$ and $\|.\|$ is the Euclidean norm.

Is there a Hilbert space H and a map $\varphi:\mathbb R^n \rightarrow H$ such that $k(x,y) = <\varphi(x),\varphi(y)> $ for all $x,y \in \mathbb R^n$?

Yes, this is guaranteed by the Moore–Aronszajn theorem.

$K(\mathbf{x},\mathbf{y}) = e^{-\|\mathbf{x} - \mathbf{y}\|}$ is a positive definite kernel.  This means it is a symmetric function satisfying $$\sum_{i,j=1}^{n} c_i c_j K(\mathbf{x_i},\mathbf{x_j}) \ge 0$$ for all $n \in \mathbb{N}$, all $\mathbf{x_1},\dotsb ,\mathbf{x_n} \in \mathbb{R}^n$, and all $c_1, \dotsb , c_n \in \mathbb{R}$.

The Moore–Aronszajn theorem says that for each such function there exists a unique reproducing kernel Hilbert space with reproducing kernel $K(\mathbf{x},\mathbf{y})$.

That is, there must be a unique Hilbert space $H$ of functions $f: \mathbb{R}^n \to \mathbb{R}$ with a mapping $\phi: \mathbb{R}^n \to H$ such that $K(\mathbf{x},\mathbf{y}) = \langle \phi(\mathbf{x}), \phi(\mathbf{y}) \rangle$ for all $\mathbf{x}, \mathbf{y} \in \mathbb{R}^n$, and such that $K(\mathbf{x},\mathbf{y})$ satisfies the reproducing property: $$\langle f, K(\cdot, \mathbf{x}) \rangle = f(\mathbf{x}) \qquad \forall \mathbf{x} \in \mathbb{R}^n, \enspace \forall f \in H$$

While the reproducing kernel Hilbert space is unique for a given positive definite kernel $K(\mathbf{x}, \mathbf{y})$, there can be other mappings $\phi$ to other Hilbert spaces which also satisfy $K(\mathbf{x},\mathbf{y}) = \langle \phi(\mathbf{x}), \phi(\mathbf{y}) \rangle$ but without the reproducing property.


Ratio between embedded vector dimensions and vocabulary size Using Embedding layer in Keras on a fairly small vocabulary (~300), I am looking at how to choose the output of this layer (dense vector) when given a 300 dimension vector.  I think that the embedded vector need to have a minimum length to be able to map a given vocabulary.

The ratio of vocabulary vs embedding length to determine the size of other layers in a neural network doesn't really matter. Word embeddings are always around 100 and 300 in length, longer embedding vectors don't add enough information and smaller ones don't represent the semantics well enough. What matters more is the network architecture, the algorithm(s) and the dataset size.

A simple way to understand this concept is that a bidirectional LSTM model with 50 neurons (nodes) followed by a fully connected layer of 70 neurons will outperform a simple MLP of 1000 neurons (nodes) connected to a embedding layer simply due to its architecture. Adding dropout will improve performance as well.

In addition, even if the vocabulary is just 300 words, using pre-trained embeddings will probably yield better results than training the embeddings directly on the dataset. The same applies to data size, a dataset with more samples will make a better classifier than a dataset with just a couple thousand samples.

In summary, it is preferable to try many architectures and cross-validate them (and/or ensemble them depending if you have a large enough dataset) with the smallest number of neurons possible and then start building up in size, depending on what computational resources you have and the speed of development you need. Large models slow down development speed whereas small models speed it up. This goes whether your vocabulary is the size of common crawl or just 300. As usual, try feature engineering (sentence length, special characters, etc.) and increase the dataset size as doing so often helps in whatever task you're trying to predict.


How to store subset of columns from a csv file? I need to create a table in hive (or Impala) by reading from a csv file (named file.csv), the problem is that this csv file could have a different number of columns each time I read it. The only thing I am sure of is that it will always have three columns called A, B, and C.

For example, the first csv I get could be (the first row is the header):

 ------------------------
|  X | Y | A | Z | B | C |
 ------------------------
|  1 | 2 | 3 | 4 | 5 | 6 |

and the second:

 ------------
|  C | A | B |
 -------------
|  1 | 2 | 3 |

And I need to store this in a table, maybe an external table. Something like this:

CREATE EXTERNAL TABLE file (A STRING, B STRING, C STRING)
AS
SELECT A, B, C
USING HEADER
LOCATION 'input/loading/';

That obviously does not work. Any ideas?

Hive's RegexSerDe which is often used to process logs could be used in your use case. You could use the regex to extract the first 3 columns from a line

CREATE EXTERNAL TABLE file (A STRING, B STRING, C STRING)
AS
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe'
    WITH SERDEPROPERTIES
      ('input.regex'='^(\\w+)\\t(\\w+)\\t(\\w+)(.*)')
LOCATION 'input/loading/'

The regex included is using pattern groups (\w+) to extract the first 3 columns and the last group (.*) is anything else which may be in the line

See more details here Community Article and Official Documentation


How can I detect patterns and/or keywords or phrases? I am collecting data in a database via php from apache.

I am interested in detecting patterns in each column for now.

For example manual examination of the data shows the pattern phpmyadmin is various forms and capitalization and at different positions in the text.  Also to detect any other patterns.

How would I detect that programmatically using the computer instead of my brain?

I am going to need a detailed explanation as I am brand new to doing this kind of thing.

A pattern is any sequence of characters that repeat together. Like phpmyadmin is a sequence of 9 letters that repeat together. I detected "phpmyadmin" manually, I would like to detect it programmatically. Excluding single digit matches of common characters which is ascii 32-127

Depends on what you want to do and what you define as a "pattern". If you are interested in frequent terms, then tokenize and count the words. If you want to compare various forms of the same terms, I suggest you build two term matrices, one where you take the input as is and one where you take a version of the input that has been transformed to lower case.

BTW, a term frequency matrix is simply a matrix where the rows are your examples (I guess the columns in your database) and the columns are the discovered tokens (ie, the words)

For example, in the phrase 'the cat sat on the mat', the corresponding row vector of word counts would be:

the 2
cat 1
sat 1
on  1
mat 1

To get frequencies, you just divide the resulting vectors by the total count of words.

[EDIT] After reading the comments, I would recommend that you explore solutions for anomaly detection. Here is a good paper that is tackling a similar problem as yours. The authors encode documents as bag of words using term frequencies (see above) and build a classifier to flag  a whole document as an outlier.

In your case, it is a little trickier because you want to flag specific terms as good or bad, and you don't have a labelled dataset. So either find a labelled dataset (no need for a fully labelled dataset, semi-supervised solutions exist but at least a few labels would be nice) or explore unsupervised solutions to outlier detection. As for pattern flagging, your best bet is to use recurrent neural networks, which are designed to process sequential data such as yours. That being said, I doubt you will achieve that with PHP, so maybe consider something else. Tensorflow / PyTorch / Neo4J are very good alternative for building such models and many other exist


How to generate a random sample and distribute values based in an probability distribution? I want to generate a random sample based on this probability distribution:

The line is the KDE of the histogram.
My random sample will have n values, the value is a number of points. Each of the n values generates an amount of points p that must be distributed among the population. So I must distribute the total of n * p points. The distribution of points must follow the probability distribution above.
How should I generate a random sample that follow this probability distribution?
Probably this is a usual problem, so I welcome any help to better formulate my question.

Create some random data
df <- data.frame(
  cat_cols = c(rep("A", 200), rep("B",150)),
  cont_vals = c(rnorm(200, 20, 5), rnorm(150,25,10)))
# Set desired binwidth and number of non-missing obs
bw = 2
n_obs = sum(!is.na(df$cont_vals))

Now plot it
library(ggplot2)
ggplot(df, aes(cont_vals))  +
  geom_histogram(aes(y = ..density..), binwidth = bw, colour = "black") +
  stat_function(fun = dnorm, args = list(mean = mean(df$cont_vals), sd = sd(df$cont_vals)))


Clustering algorithm prior to model building? I would like to understand, how a clustering algorithm can be used (if possible) to identify naturally occurring groups within a data set, prior to building predictive models/model, and to hence improve accuracy of models/model

In clustering the outcome variable or the response is unknown, this is why it's called clustering. Irrespective, of the fact the data being labeled or unlabelled, clustering can be applied as a data preprocessing algorithm.
Essentially, you must proceed by employing the initial data preprocessing tasks (like missing value treatment, collinearity, skewness etc). Once, the data is "statistically clean", then you can apply any clustering technique. However, remember clustering requires data to be "grouped" such that data points within a group are related to each other and unrelated to other data points belonging to another group. This can be achieved only if you have a "statistically clean" data. The next important point to consider is, "how to determine the possible number of clusters". Because any clustering algorithm will divide the data points into groups oblivious of the fact whether the groups exist or not. Therefore, you will have to prove mathematically/statistically the occurrence of groups in the dataset. In literature, there exist several methods like the "Principal Component Analysis (PCA)" or the "elbow method".
Once, you have determined such groups, you can then label the groups and perform predictive analytics.


Finding a vector that minimize the MSE of its linear combination I have been doing a COVID-19 related project. Here is the question:

N = vector of daily new infected cases
D = vector of daily deaths
E[D] = estimation of daily deaths

N is a n-dimensional vector, n is around 60. E[D] is another n-dimensional vector. Under certain assumptions, each entry of E[D] can be calculated as a linear combination of the entries of N.

We want to find the vector N such that the E[D] derived from N has least mean squared error when compared to actual D data. I think a gradient descent algorithm is needed here. However, I am not very familiar with gradient descent.

This seems to be a basic data science problem, but I am kind of lost right now. Does anyone has any idea about which algorithm should I dig into?

If inputs are the same and you expect multiple outputs, I would recommend you to look at multioutput models which can be of two types :

native multioutput algorithms
multiple single output regressors wrapped together. Sklearn’s multioutput does that.


How to calculate convolution for 2nd conv Layer in CNN, Do we need to average across all feature maps? I understand that for the first layer (assuming we have a grayscale image) we calculate the convolution of 3*3 receptive field as a weighted sum of receptive weights with pixels
$ x1 · w1 + x2 · w2 + x3·w3 + ... + xn · wn$
But for the second layer, where we already have $N$ feature maps in the last convolution layer, how would we calculate the convolution(for a particular pixel/cell)? should we take an average of the $N$ weighted sums we have from feature maps?
If the question isn't clear, I have tried to highlighted it from the image taken from famous 3d visualization.

In the image, for the pixel in question(hovered and squared) we have inputs coming from 4 feature maps. I think they are four integers (weighted sums of the receptive field of the bottom right corner from each feature map).
How is the value (convolution)  for this particular pixel/cell calculated? showed we take average like below?
(I can add more details if the question is not clear)
weighted_sum_fmap1 + weighted_sum_fmap1 + weighted_sum_fmap1 + weighted_sum_fmap1 / 4

No, you don't average across all feature maps.
When the input has multiple channels, you need your convolution filter to have the same number of channels. Therefore, the filter "covers" the full depth of the input. Then, you simply perform the element-wise multiplication of the filter with the overlapping region in the input and add all the resulting elements together.
Therefore, if the input to a convolutional layer is a grayscale image, a filter is of dimensions [3,3,1], while if the input is a color image (with 3 channels), your filter is [3, 3, 3], like this (source):

The same happens with feature maps with N channels: we need a filter with N channels.


How to extract all information by using id I need to apply classifier algorithm after clustering. Now after clustering I find the id numbers that which id belongs which cluster. I clustered them into 2 cluster.

Now I need to collect those data by using those id. But I don't know how I can collect all information by using those id.

As I use jupyeter notebook and in main data I have no attribute named id and those id assigned jupyter notebook when I load data from main data file.

This is my main data

Here is my code to find which data belongs which cluster.

x = 0.10
i=0
C_i = np.where(labels == i)[0].tolist()
n_i = len(C_i) # number of points in cluster i

# (2) indices of the points from X to be sampled from cluster i
sample_i = np.random.choice(C_i, int(x * n_i))
print (i, sample_i)

and after clustering I find these ids

New addition:

suppose my loading file name is train.
now using train.loc[26] command I get the info of that specific id.

But I need to collect all info into a new data frame like as train dataframe

Define a new column, then use these ids to select the relevant rows and set that column to the appropriate id:

from pandas import DataFrame,Series
from numpy.random import rand
df = DataFrame(rand(10,10)).assign(cluster=0)
clusters = Series([[1,3,5],[0,2,4,6,7,8,9]])
for cluster,rows in clusters.iteritems():
    df.loc[rows]["cluster"] = cluster

Now you can do your groupbys on column "cluster".


Optimising Kernel parameters using training data in GaussianProcessRegressor of Scikit-learn I want to optimize the Kernel parameters or hyper-parameters using my training data in GaussianProcessRegressor of Scikit-learn.Following is my query:

My training datasets are:

X: 2-D Cartesian coordinate as input data

y: radio signal strength (RSS) at the 2-D coordinates points as observed output

What I've done so far:

I've installed python and Scikit-learn software. I've successfully tested the sample codes. I'm able to predict RSS at test points using training data. I use squared exponential Kernel.

What I want to do:

I want to train the Kernel parameters (hyper-parameter) with different optimizing algorithms like gradient descent, swarm intelligence, and trust-region-reflective algorithms.

What I learned and What help I am asking for:

I've learned that, in the GaussianProcessRegressor class of scikit, the optimizer is an argument where I can use my own optimizing algorithm. Since it is callable, I need to write my own function/method for it. Can I use any inbuilt library (library of optimization algorithm) in GaussianProcessRegressor class? Are there such libraries available for python? Could anybody provide any sample code for using kernel parameter optimization algorithm in GaussianProcessRegressor? I've learned that we use the training datasets for optimizing the hyper-parameters. Could anybody provide any insight about relating the training datasets with the optimization algorithm, please?

Thank you!

Given scikit-learn's API, you create a separate instance for each optimizer and compare the results to see which optimizer makes better predictions. It would looks something like:

from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import RBF

gp1 = GaussianProcessRegressor(kernel=RBF, optimizer=optimizer1)
gp1.fit(X, y)
y_pred1 = gp1.predict(x)

gp2 = GaussianProcessRegressor(kernel=RBF, optimizer=optimizer2)
gp2.fit(X, y)
y_pred2 = gp2.predict(x)

This blog post gives a general overview of fitting Gaussian Processes, including scikit-learn's GaussianProcessRegressor.


Predicting hardware failures with limited data I am exploring using machine learning to predict if a particular hardware component would fail within a timeframe, say 3 months. The ultimate goal is to minimize physical human inspection so that maintenance crew would always performance maintenance/servicing just in time before hardware failure.

However, servicing are carried out at a fixed interval regardless of the condition of the hardware and it has practically never failed. The dataset we have is of this nature:

equipment: pump1
location: outdoor (or indoor)
provider: providerA
operation: -
   temperature (as a function of time)
   moisture (as a function of time)
   team (as a discreet function of time, ie teamA, teamB, teamC)

Basically, we know some features of the equipment (here pump1), and we know some operating parameters (here temperature, moisture) as a function over time, and we also know which team did the servicing over time. I understand that survival analysis would be most suitable for a predictive maintenance problem but each operating parameters vary over time of study. Nothing is constant.

While regression does not take into account time. As such how do we actually model it to predict its failure?

Note: The above parameters are adapted from https://www.kaggle.com/ludobenistant/predictive-maintenance

Can you use a proxy for hardware failure? Something like a state in which failure becomes likely? Then you could try to predict this state which should be more common and use a fudge factor to get to your failures.


My model accuracy doesn't change after first epoch I've created a model to predict housing prices in LA, and what should be a simple regression problem, is giving me headache because the loss is just too big and my accuracy wont change.

I've already tried normalizing, changing the architecture (decreasing layers, hidden units), adding dropout, changed the loss function, batch size, epochs and my accuracy is still only 0.022

input_shape = X_train_2[0].shape

model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=input_shape),
    tf.keras.layers.Dense(units=300, activation=tf.nn.relu),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Dropout(0.1),
    tf.keras.layers.Dense(units=300, activation=tf.nn.relu),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Dropout(0.1),
    tf.keras.layers.Dense(units = 1, kernel_initializer = 'lecun_normal', activation='linear')
    ])

model.compile(optimizer='adam',loss='mean_squared_error', metrics=['accuracy'])
model.fit(X_train, y_train, epochs=5, batch_size=32)
model.summary()
model.evaluate(X_test_, y_test)

Training Log

Epoch 1/5 32444/32444 [==============================] - 1s
    38us/sample - loss: 90230324650039.5469 - acc: 0.0012
Epoch 2/5
    32444/32444 [==============================] - 1s 28us/sample -
    loss: 90230315396180.2031 - acc: 0.0022
Epoch 3/5 32444/32444
    [==============================] - 1s 27us/sample - loss:
    90230293267377.3438 - acc: 0.0022
Epoch 4/5 32444/32444 [==============================] - 1s 27us/sample - loss:
    90230260607518.6250 - acc: 0.0022
Epoch 5/5 32444/32444 [==============================] - 1s 28us/sample - loss:
    90230216684525.4375 - acc: 0.0022

Your metric is accuracy although you are working on a regression problem, this doesn't make sense. You should use instead:

metrics = ['mean squared error']


Non Scaled New Actual Data I am new to Machine Learning and I have a conceptual question.

I have a scaled dataset (scikit-learn and pandas).

After training/testing my algo, I will make new predictions using new actual data which will not be scaled or normalized.

Will this discrepancy be a problem, if so, how should I resolve it?

Best,

You should save the scaler params used to fit the training set and use the same ones to transform all other data used with the model from then on - whether CV, test or new unseen data.

  After training/testing my algo, I will make new predictions using new actual data which will not be scaled or normalized.

No that won't work. Once you add scaling/normalisation to the training pipeline, the exact same scaling (as in same scaling params, not re-calculated) should be applied to all input features.

The scikit-learn scalers like e.g. StandardScaler have two key methods:

fit should be applied to your training data
transform should be applied after fit, and should be used on every data set to normalise model inputs.

fit_transform can be used on the training data only to do both in a single step.

If you need to do the training and predictions in different processes (maybe live predictions are on different devices for instance), then you need to save and restore the scaling params. One basic, simple way to do this is using pickle e.g. pickle.dump( min_max_scaler, open( "scaler.p", "wb" ) ) to save to a file and min_max_scaler = pickle.load( open( "scaler.p", "rb" ) ) to load it back.


Representation of state space, action space and reward system for RL porblem I am trying to solve the problem of an agent dynamically discovering(start with no information about the environment) the environment and to explore as much of the environment as possible without crashing into obstacles
I have the following environment:

where the environment is a matrix. In this the obstacles are represented by 0's and the free spaces are represented with 1s. The position of the agent is given by a label such as 0.8 in the matrix.

The initial internal representation of the environment of the agent will look something like this with the agent position in it .

Every time it explores the environment it keeps updating its own map:

The single state representation is just the matrix containing-

0 for obstacles
1 for unexplored regions
0.8 for position of the agent
0.5 for the places it has visited once
0.2 for the places it has visited more than once

I want the agent to not hit the obstacles and to go around them.

The agent should also not be stuck in one position and try to finish the exploration as quickly as possible.

This is what I plan to do:

In order to prevent the agent from getting stuck in a single place, I want to punish the agent if it visits a single place multiple times. I want to mark the place the agent has visited once as 0.5 and if it has visited it more than once that place will be labelled 0.2

The reason I am marking a place it has visited only once as 0.5 is because if there is a scenario where in the environment there is only one way to go into a region and one way to come out of that region, I don't want to punish this harshly.

Given this problem, I am thinking of using the following reward system-

+1 for every time it takes an action that leads to an unexplored region
-1 for when it takes an action that crashes into an obstacle
0 if it visits the place twice(i.e 0.5 scenario)
-0.75  is it visits a place more than twice

The action space is just-

up
down
left
right

Am i right in approaching the problem this way? Is Reinforcement Learning the solution for this problem?

Is my representation of the state , action, reward system correct?

I am thinking that DQN is not the right way to go because the definition of a terminal state is hard in this problem, what method should I use to solve this problem?

I would suggest you to use Deep Q or A2C (I personally use A2C). As a terminal state you can consider the state in which every tile has been visited once, except if you want to your agent wonder forever. I create OpenAI gym gridworlds so I can use some of their very useful wrappers (example TimeLimit wrapper in which an episode terminates when a certain number of steps has been reached).

For your state representation I would suggest you to follow  well established practices. I assume that you want to use CNNs as a perceptual module and not MLP. You could either use an image of your gridworld with different colors for every entity or use what we call feature-image layers as inputs. These are simply a volume of images and each image has the same dimensions as your grid map and is everywhere 0 except the location of an entity which will have a value of 1. This is in other words a one-hot encoding but in the spatial dimensions of your map.

Actions seem fine to me and your reward function is worth giving a try but I wouldn't penalize that heavily (maybe -0.1 instead of -0.75). The issue will rise initially when your agent explores. If it is heavily penalized it will try to maintain its reward and deliberately crash.

Do not get frustrated if initially it doesn't learn. Designing a proper reward function is quite a task besides the proper feature representation (I suggest images).


What needs to be done to make n_jobs work properly on sklearn? in particular on ElasticNetCV? The constructor of sklearn.linear_model.ElasticNetCV takesn_jobs as an argument.
Quoting the documentation here

n_jobs: int, default=None
Number of CPUs to use during the cross validation. None means 1 unless in a joblib.parallel_backend context. -1 means using all processors. See Glossary for more details.

However, running the below simple program on my 4 core machine (spec details below) shows performance is best when n_jobs = None, progressively deteriorating as you increase n_jobs all the way to n_jobs = -1 (supposedly requesting all cores)
    import numpy as np
    from sklearn.linear_model import ElasticNetCV

    # ------------- Setup  X, y -------------
    IMPORTANT_K = 10
    TOTAL_K = 30
    SAMPLE_N = 1000
    ERROR_VOL = 1

    np.random.seed(0)
    important_X = np.random.rand(SAMPLE_N, IMPORTANT_K)
    other_X = np.random.rand(SAMPLE_N, TOTAL_K - IMPORTANT_K)
    actual_coefs = np.linspace(0.1, 1, IMPORTANT_K)
    noise = np.random.rand(SAMPLE_N)
    y = important_X @ actual_coefs + noise * ERROR_VOL
    total_X = np.concatenate((important_X, other_X), axis=1)

    # ---------------- Setup ElasticNetCV -----------------
    LASSO_RATIOS = np.linspace(0.01, 1, 10)
    CV = 10

    def enet_fit(X, y, n_jobs: int):
        enet_cv = ElasticNetCV(l1_ratio=LASSO_RATIOS, fit_intercept=True,
                               cv=CV, n_jobs=n_jobs)
        enet_cv.fit(X=X, y=y)
        return enet_cv

    # ------------------- n_jobs test --------------
    N_JOBS = [None, 1, 2, 3, 4, -1]
    import time
    for n_jobs in N_JOBS:
        start = time.perf_counter()
        enet_cv = enet_fit(X=total_X, y=y, n_jobs=n_jobs)
        print('n_jobs = {}, perf_counter = {}'.format(n_jobs, time.perf_counter() - start))

What needs to be done to make this work as expected?
Some people seem to think this is broken in Windows as stated here. Indeed I am running Windows 10 on an Intel i7-7700 4.2Ghz machine, but unfortunately the question the above link points to does not have any commentary or answers, nor unfortunately do I have access to a Unix machine to try if this works on Unix.
Some people here say running from an interactive session is a problem. I observe the above program perform in the same way whether run on Jupyter Lab or as a script from the terminal.
I will also add that I have run this on several version combinations of sklearn and joblib, but have not been able to resolve the problem in that way.
One of those combinations is below, put together by conda env create when only specifying the python version python==3.6.10 thereby allowing conda to install the versins it deems are most appropriate.
I also include the numpy, mkl and scipy dependencies that were installed.
intel-openmp              2020.1                      216
joblib                    0.14.1                     py_0
mkl                       2020.1                      216
mkl-service               2.3.0            py36hb782905_0
mkl_fft                   1.0.15           py36h14836fe_0
mkl_random                1.1.0            py36h675688f_0
numpy                     1.18.1           py36h93ca92e_0
numpy-base                1.18.1           py36hc3f5095_1
python                    3.6.10               h9f7ef89_2
scikit-learn              0.22.1           py36h6288b17_0
scipy                     1.4.1            py36h9439919_0

The questions
Does the simple program above give you better performance with increasing n_jobs when you run it?
On what OS / setup?
Is there something that must be tweaked for this to work properly?
All help very much appreciated

When I ran your script, I got the same impression, that n_jobs was hurting you performance. However, you have to consider that parallelizing the cross-validation would only benefit if you have more data samples. With few data, the communication overhead indeed is more expensive than the processing cost involved on the task.

I tried your script with more samples SAMPLE_N = 100000 and got the following results. Setup: macos i5 8gb.

n_jobs = None, perf_counter = 21.605680438
n_jobs = 1, perf_counter = 22.555127251
n_jobs = 2, perf_counter = 15.129894496000006
n_jobs = 3, perf_counter = 11.280528144999998
n_jobs = 4, perf_counter = 13.050180971000003
n_jobs = -1, perf_counter = 20.031103849000004


Cross-entropy loss colloquially I have just made an MNIST classifier from a tutorial. It has 1 layer, no hidden layers. It uses PyTorch's nn.CrossEntropyLoss as a loss function. Plotting my losses after training, it starts out at around 2.25 and goes down to 0.5. What does this number mean, in a colloquial sense? Does a loss of 0.5 mean that it's right 50% of the time?

The cross entropy is equivalent (up to a constant) to the Kullback Leibler Divergence  which has an interpretation based on information theory: (very crudely) it is the amount of information "lost" by representing the true labels via the distribution of their predicted values (measured in "nats" (e) or "bits" (2) depending on the base of the logarithm). I don't find it very useful as a "colloquial" interpretation or to think in units of bits.

Typically, you would calculate more interpretable metrics like accuracy and inspect them along with the loss to get a picture of the performance.

However, you can use the loss and compare it to a simple baseline (e.g. the model only predicting the average of your labels or only the majority class) and thus get a relative performance estimate.


Preparation for Career in Data Analysis Without College I dropped out of college but am interested in a career in data analysis. Now I am self-studying approximately 10 hours per day.  Browsing through job postings on Linkedin has allowed me to compose a rough curriculum.  It would be of great help to me if you would either add a subject I have omitted or eliminate a subject that is not necessary for success in the market place.

Curriculum (in 3-subject groupings):

Group 1

Single-variable calculus
Intro to python
SQL

Group 2

Multi-variable calculus/linear algebra
Discrete math
Data structures and algorithms

Group 3

Calculus-based statistics and probability
Hadoop stack
Differential equations

Group 4

Statistical learning/predictive modelling
Python data analysis techniques/Statistical programming in R
Fundamentals of machine learning

All the while I plan to practice using any data sets I can find online.  Will this be sufficient to land a job in data analysis?  Of course I plan to learn far more than just this, but is this foundation solid enough to land an entry level data engineering/science position?

At least based on what I and other data analysts/scientists do in my company, your technical topics list seems sufficient. But I would also add:

Visualization (ggplot2 in R, matplotlib in Python, d3.js for really cool stuff)
Design of experiments

Communication skills are also quite important.

For more inspiration, here's a good "curriculum" represented as a metro map: http://nirvacana.com/thoughts/becoming-a-data-scientist/

Let me also recommend building up a portfolio of data science projects. This could consist of your analyses of data sets you find online (e.g. on UCI), Kaggle competitions, or class projects (e.g. via Udacity or Coursera). That way, you can give direct proof of your technical skills, your communication in the form of reports or graphics, and your ability to extract insight.


How are selected the features for a decision tree in CART? Suppose I want to use CART as classification tree (I want a categorical response). I have the training set, and I split it using observation labels.

Now, to build the decision tree (classification tree) how are selected the features to decide which label apply to testing observations?

Supposing we are working on gene expression matrix, in which each element is a real number, is that done using features that are more distant between classes?

At each split point, CART will choose the feature which "best" splits the observations. What qualifies as best varies, but generally the split is done so that the subsequent nodes are more homogenous/pure with respect to the target. There are different ways of measuring homogeneity, for example Gini, Entropy, Chi-square. If you are using software, it may allow you to choose the measure of homogenity that the tree algorithm will use.

Distance is not a factor with trees - what matters is whether the value is greater than or less than the split point, not the distance from the split point.


Linear Regression in R Mapreduce(RHadoop) I m new to RHadoop and also to RMR...
I had an requirement to write a Mapreduce Job in R Mapreduce. I have Tried writing but While executing this it gives an Error.
Tring to read the file from hdfs

Error:

Error in mr(map = map, reduce = reduce, combine = combine, vectorized.reduce,  :
   hadoop streaming failed with error code 1

Code :

Sys.setenv(HADOOP_HOME="/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop")
Sys.setenv(HADOOP_CMD="/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/bin/hadoop")

Sys.setenv(HADOOP_STREAMING="/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop-0.20-mapreduce/contrib/streaming/hadoop-streaming-2.0.0-mr1-cdh4.7.0.jar")
library(rmr2)
library(rhdfs)
hdfs.init()
day_file = hdfs.file("/hdfs/bikes_LR/day.csv","r")
day_read = hdfs.read(day_file)
c = rawToChar(day_read)

XtX =
  values(from.dfs(
    mapreduce(
      input = "/hdfs/bikes_LR/day.csv",
      map=
        function(.,Xi){
         yi =c[Xi[,1],]
         Xi = Xi[,-1]
         keyval(1,list(t(Xi)%*%Xi))
       },
  reduce = function(k,v )
  {
    vals =as.numeric(v)
    keyval(k,sum(vals))
  } ,
  combine = TRUE)))[[1]]

XtY =
 values(from.dfs(
    mapreduce(
     input = "/hdfs/bikes_LR/day.csv",
     map=
       function(.,Xi){
         yi =c[Xi[,1],]
         Xi = Xi[,-1]
        keyval(1,list(t(Xi)%*%yi))
       },
     reduce = TRUE ,
     combine = TRUE)))[[1]]
solve(XtX,XtY)

Input:
------------

instant,dteday,season,yr,mnth,holiday,weekday,workingday,weathersit,temp,atemp,hum,windspeed,casual,registered,cnt
1,2011-01-01,1,0,1,0,6,0,2,0.344167,0.363625,0.805833,0.160446,331,654,985
2,2011-01-02,1,0,1,0,0,0,2,0.363478,0.353739,0.696087,0.248539,131,670,801
3,2011-01-03,1,0,1,0,1,1,1,0.196364,0.189405,0.437273,0.248309,120,1229,1349
4,2011-01-04,1,0,1,0,2,1,1,0.2,0.212122,0.590435,0.160296,108,1454,1562
5,2011-01-05,1,0,1,0,3,1,1,0.226957,0.22927,0.436957,0.1869,82,1518,1600
6,2011-01-06,1,0,1,0,4,1,1,0.204348,0.233209,0.518261,0.0895652,88,1518,1606
7,2011-01-07,1,0,1,0,5,1,2,0.196522,0.208839,0.498696,0.168726,148,1362,1510
8,2011-01-08,1,0,1,0,6,0,2,0.165,0.162254,0.535833,0.266804,68,891,959
9,2011-01-09,1,0,1,0,0,0,1,0.138333,0.116175,0.434167,0.36195,54,768,822
10,2011-01-10,1,0,1,0,1,1,1,0.150833,0.150888,0.482917,0.223267,41,1280,1321

 Please Suggest me any mistakes.

I suggest using machine learning libraries with already functional linear regression
Spark MLlib or hivemall.


Accuracy goes straight for about 200 epochs then start increasing Can anyone explain the following observation?
Why did the accuracies keep to be a straight line with a very smooth decrease of loss?
By the way, why is the loss lines so beautifully smooth for the first 400 epochs?
Is this because of the learning rate or other reasons?

The accuracy depends on a threshold, whereas the loss doesn't. ML software tends to assume a threshold of 0.5, which is not a good fit in cases where there's some class imbalance.
I believe that, until epoch 500, your model is learning (loss is going down), but the default threshold doesn't allow you to see it in terms of accuracy.
If you pick another threshold you might see different results.
Regarding the loss going from smooth to noisy, it might be that it is learning the "easy cases" first, and then decreasing very easily and smoothly, and after epoch 400-500 it starts to overfit some of "hard cases", thus the loss becoming noisier.