Why do we need the sigmoid function in logistic regression? What is the purpose of the logistic sigmoid function as it is used in logistic regression? Why does it need to be part of the hypothesis function h(x) ?

As I understand it, the logistic sigmoid function gives the probability that a certain input vector x is contained within a class C1 for a label y. In the binary-class case, it seems that if h(x) >= 0,5, we say that x belongs to one class, otherwise it belongs to the other. In the logistic regression model, our hypothesis function h(x) is of the form g(p^T * x), where p is the parameter vector (p^T is the transpose) and g is the sigmoid function.

Since the y-intercept of the logistic sigmoid is 0.5, saying that h(x) >= 0.5 is the same as saying p^T * x >= 0.

What I'm getting at is why do we need the logistic sigmoid function at all to define some threshold for separating the classes? Why not just let the hypothesis function be of the form h(x) = p^T * x, and claim that y = 1 if p^T * x >= 0? Why complicate things unnecessarily with the logistic sigmoid?

If you go and define the hypothesis function as h(x) = p^T * x, and claim that y = 1 if p^T * x >= 0, you are complicating the things even more.

This is due to the fact that during training time you will have to calculate the derivative of the function h(x). Now your function can lead to exploding or vanishing gradients depending on your input and initialization.

Logistic Sigmoid solves this problem and it also has a nice gradient g(x)(1 - g(x)). So rather that complicating things, sigmoid actually simplifies the calculation.


Negative log-likelihood not the same as cross-entropy? The negative log-likelihood
$$
\sum_{i=1}^{m}\log p_{model}(\mathbf{y} | \mathbf{x} ; \boldsymbol{\theta})
$$
can be multiplied by $\frac{1}{m}$ after which the law of large numbers can be used to get
$$
\frac{1}{m} \sum_{i=1}^{m}\log p_{model}(\mathbf{y} | \mathbf{x} ; \boldsymbol{\theta}) \rightarrow E_{}(\log p_{model}(\mathbf{y} | \mathbf{x} ; \boldsymbol{\theta}))
$$ as the sample size $m$ tends to infinity. This expectation is the "cross-entropy".
Now here comes my question: The book I am reading(Deep Learning by Goodfellow et al) mentions several attractive properties of using the negative log-likelihood(like consistency). But meanwhile, it also also uses cross-entropy directly as the loss function of maximum likelihood estimators:

This doesn't make sense to me - to talk about negative log-likelihood and cross-entropy as being identical. It would make sense for me to talk about NLL as an approximation of the cross-entropy.

I mean, they give different results - so why use one over the other? This seems like a valid question when they do not give the same results and must thus also affects the performance. Like, I am only aware of neural networks that use cross-entropy and not ones that use NLL - how come?

Maybe cross-entropy even holds other properties than negative log-likelihood?

I think the issue is that while
$$
\frac{1}{m} \sum_{i=1}^{m}\log p_{model}(\mathbf{y} | \mathbf{x} ; \boldsymbol{\theta}) \rightarrow E_{}(\log p_{model}(\mathbf{y} | \mathbf{x} ; \boldsymbol{\theta}))
$$
is true, the right side is not what is being referred to as the cross-entropy.  Indeed, we have no access to the expectation over the true population distribution / data generating process; a few paragraphs down, we find:

We can thus see maximum likelihood as an attempt to make the model distribution match the empirical distribution $\hat{p}_{data}$. Ideally, we would like to match the true data-generating distribution $p_{data}$, but we have no direct access to this distribution.

They also distinguish throughout that the expectations are over $\hat{p}_{data}$, the training data empirical distribution, not $p_{data}$, the underlying population distribution.
The left side of your limit is already an expectation, but over the finite training data, and that is what is referred to as the cross-entropy.  ("Cross-entropy" is a broader term, for any pair of probability distributions.  Goodfellow et al note this (my emphasis):

Any loss consisting of a negative log-likelihood is a cross-entropy between the empirical distribution deﬁned by the training set and the probability distribution deﬁned by model.

So, the answer to your questions is that the premise is incorrect: (this) cross-entropy is the same as negative log-likelihood.  Taking your questions with the limiting and population cross-entropy instead, the answer is "we don't have access to the latter".  It would be the better target to be sure, but our lack of that information is the point of modeling in the first place.


Python: SARIMAX Model Fits too slow I have a time series data with the date and temperature records of a city. Following are my observations from the time series analysis:

By plotting the graph of date vs temperature seasonality is observed.
Performing adfuller test we find that the data is already stationary, so d=0.
Perform Partial Autocorrelation and Autocorrelation with First Seasonal Difference and found p=2 and q=10 respectively.

Code to Train Model
model=sm.tsa.statespace.SARIMAX(df['temperature'],order=(1, 1, 1),seasonal_order=(2,0,10,12))
results=model.fit()

This fit function runs indefinitely and does not reach an output. I am running on a on Google Colab CPU.
How to fix this issue?

Assuming you have multiple cities in the dataframe. you can create some new features in the dataframe . For example  , I created a few features below to try and match you PACF and ACF graphs .
df['lag_1'] = df.groupby(['city'])['temperature'].transform(lambda x: x.shift(1))
d=1
df['d_1'] = df['temperature'] - df['lag_1']
p =  1:
df['p_1'] = df.groupby(['city'])['d_1'].transform(lambda x: x.shift(1))
q = 1:
df['ma_1'] = df.groupby(['city'])['d_1'].transform(lambda x: x.shift(1).rolling(1).mean())
P=2 (and other terms)
df['lag_t12'] = df.groupby(['city'])['temperature'].transform(lambda x: x.shift(12))
df['lag_t24'] = df.groupby(['city'])['temperature'].transform(lambda x: x.shift(24))
.
.
.
df['lag_t120'] = df.groupby(['city'])['temperature'].transform(lambda x: x.shift(120))
Q=10 :
df['Q_10'] = df[col for col in df if col.startswith('lag_t')].mean()
After this try using LightGBM , XGBoost or other regression packages to regress against  these newly created features  with temprature as your target variable.
Alternatively , you can forego the ACF/PCF approach altogether and instead create  bunch of commonly used features using :

shift
rolling mean
rolling standard deviations
max() , min() within groups

and regress against those and check which features  minimize RMSE/AIC/BIC in your Regression Hyperparameters.
Since Cross validation is different in cases of Time Series,consider using TimeSeriesSplit in scikit-learn  . Check-out this post in case you want to do grouped time series cross validation .


How to create a barplot using multiple column My data is about boardgames which contain 1 to many catgeories each like this :

categorie1 categorie2 categorie3
  Deduction   Medieval  Word Game
  Deduction   Medieval  NA
  Card Game   Medieval  Zombies
     Horror   NA        NA
     Horror   Medieval  Zombies

I would like to create a barplot showing the most common categories accross games but i can't figure out how with multiple columns instead of one. Is there a dplyr method ?

The function you are looking for is gather from the tidyr package. This function takes a wide data.frame and makes it a long data.frame. gather is easy to use:

library(dplyr)
library(tidyr)

# Building a sample data.frame like your example data.
df <- data.frame(Boardgame = c("Game1", "Game2", "Game3", "Game4", "Game5"),
             categorie1 = c("Deduction", "Deduction", "Card Game", "Horror", "Horror"),
             categorie2 = c("Medieval", "Medieval", "Medieval", NA, "Medieval"),
             categorie3 = c("Word Game", NA, "Zombies", NA, "Zombies"),
             stringsAsFactors = FALSE)

# Using gather from tidyr and some dplyr functions:
df %>%
  gather(key = "Cat_Label",    # Key is the name of the column that will holdthe old column names
         value = "Categorie",  # Value is the name of the column that will hold the data
         -Boardgame,           # Ignore the Boardgame column, use every other column
         na.rm = TRUE) %>%     # Remove NA values
  arrange(Boardgame) %>%       # Sort by Boardgame
  select(-Cat_Label)           # Remove the unneeded Cat_Label column (if you want)

# Results:
   Boardgame Categorie
1      Game1 Deduction
2      Game1  Medieval
3      Game1 Word Game
4      Game2 Deduction
5      Game2  Medieval
6      Game3 Card Game
7      Game3  Medieval
8      Game3   Zombies
9      Game4    Horror
10     Game5    Horror
11     Game5  Medieval
12     Game5   Zombies

The -boardgame notation in the gather function means that every column but Boardgame will be gathered, even if you have 200 categorie columns. Once you have your data.frame, you can use ggplot2 to visualize the Categorie column as you see fit.

It should be noted that development of gather by the Tidyverse team is complete. A new, more general function pivot_longer has been implemented to replace gather. It has similar usage in a simple case, but the arguments are a little different:

df %>%
  pivot_longer(cols = -Boardgame,         # You now explicitly declare the columns
               names_to = "Cat_Label",    # New column containing old column names
               values_to = "Categorie",   # New column containing old column values
               values_drop_na = TRUE)     # Remove NA values


For outliers treatment, clipping, winsorizing or removing? I came across two different techniques for treating outliers winsorization, clipping and removing:

Winsorizing: Consider the data set consisting of:
{92, 19, 101, 58, 1053, 91, 26, 78, 10, 13, −40, 101, 86, 85, 15, 89, 89, 28, −5, 41}       (N = 20, mean = 101.5)
The data below the 5th percentile lies between −40 and −5, while the data above the 95th percentile lies between 101 and 1053. (Values shown in bold.) Then a 90% winsorization would result in the following:
{92, 19, 101, 58, 101, 91, 26, 78, 10, 13, −5, 101, 86, 85, 15, 89, 89, 28, −5, 41}       (N = 20, mean = 55.65)
Clipping:Given an interval, values outside the interval are clipped to the interval edges. For example, if an interval of [0, 1] is specified, values smaller than 0 become 0, and values larger than 1 become 1.
Removing: Just taking them out.

My questions are:

In which cases should I use which one?
If I always do winsorization (seems the best in my opinion) when I am loosing important information?
Is this model dependent (for decision trees, for linear...) or the same strategy can be applied to all of them

On the difference between winsorizing and clipping :

The techniques are very similar. They deal with extreme values (that are not necessarily outliers). Imo you should generally avoid thinking that big values = outliers. Solutions to deal with big values include normalizing your variables by a size factor for more comparability.
Which one should you use is linked to your data and their predictiveness. Sometimes data outside of a threshold will be predictive, sometimes not. Sometimes the underlying process will depend on rank, sometimes on global values. There is no rule for when to do one of those two techniques, there is no rule on which one to choose. In the end, you should choose which one gives you better results. Yeah, windsorisation seems a bit more 'adaptative' as you don't have to look at each of your predictors to choose a threshold, but outside of that, there is no reason it performs better statistically.
One of the main problems of those techniques is that they are univariate. If your predictors are correlated, modifying one value without another can breaks your relationships.
Yes, the answer is model dependent. Having extreme value may be detrimental for the convergence of methods based on gradient descent, as some data points may swing the optimization path. For methods relying on the partition of the space like trees, having extreme values is less of a problem.

Removing outliers is a completely different strategy. The difficulty is to identify outliers confidently... but that's another question.


Implementation of LDA (Latent Dirichlet Allocation) for classification tasks Until now I have used LDA only for topic modelling. I would like to know which is the simplest implementation of LDA algorithm for classification tasks.

You can use LDA on your training data to build the topic representation of it for example:

(document)entry[1] Label A: (topic 1 has 4 words in document 1)T[1]=4, T[3]=7, T[4]=5..
entry[2] Label C: T[1]=3,T[2]=2...
entry[3] Label A: T[1]=2,T[2]=2,T[3]=5...
.
.
.
 -

Using this you can build a simple decision tree: T[1]>1 AND T[3]>4 AND (T[2]>1 OR T[4] > 3) ---> Label A

Another approach would be to use: https://en.wikipedia.org/wiki/Dirichlet-multinomial_distribution#A_second_example:_Naive_Bayes_document_clustering


Predicting correct match of French to English food descriptions I have a training and test set of food descriptions pairs (please, see example below)
First name in a pair is a name of food in French
and second word is this food description in English.
Traing set has also a trans field that is True for correct descriptions
and False for wrong descriptions.
The task is to predict trans field in a test set, in other words to predict
wich food description is corect and which is wrong.

dishes = [{"fr":"Agneau de lait", "eng":"Baby milk-fed lamb", "trans": True},
{"fr":"Agrume", "eng":"Blackcurrants", "trans": False},
{"fr":"Algue", "eng":"Buttermilk", "trans": False},
{"fr":"Aligot", "eng":"potatoes mashed with fresh mountain cheese", "trans": False},
{"fr":"Baba au rhum", "eng":"Star anise", "trans": True},
{"fr":"Babeurre", "eng":"seaweed", "trans": False},
{"fr":"Badiane", "eng":"Sponge cake (often soaked in rum)", "trans": False},
{"fr":"Boeuf bourguignon", "eng":"Créole curry", "trans": False},
{"fr":"Carbonade flamande", "eng":"Beef Stew", "trans": True},
{"fr":"Cari", "eng":"Beef stewed in red wine", "trans": False},
{"fr":"Cassis", "eng":"citrus", "trans": False},
{"fr":"Cassoulet", "eng":"Stew from the South-West of France", "trans": True},
{"fr":"Céleri-rave", "eng":"Celery root", "trans": True}]

df = pd.DataFrame(dishes)

    fr                  eng                                          trans
0   Agneau de lait      Baby milk-fed lamb                           True
1   Agrume              Blackcurrants                                False
2   Algue               Buttermilk                                   False
3   Aligot              potatoes mashed with fresh mountain cheese   False
4   Baba au rhum        Star anise                                   True
5   Babeurre            seaweed                                      False
6   Badiane             Sponge cake (often soaked in rum)            False
7   Boeuf bourguignon   Créole curry                                 False
8   Carbonade flamande  Beef Stew                                    True
9   Cari                Beef stewed in red wine                      False
10  Cassis              citrus                                       False
11  Cassoulet           Stew from the South-West of France           True
12  Céleri-rave         Celery root                                  True

I think to solve this as text classification problem, where text is a concatenation of French name and English description embeddings.

Questions:

Which embeddings to use and how concatenate them?
Any other ideas on approach to this problem? BERT?

Update:

How about the following approach:

Translate (with BERT?) French names to English
Use embeddings to create two vectors: v1 - translated English vector and v2 - English description vector (from data set)
Compute v1 - v2
Create new data set with two columns: v1 - v2 and trans
Train classifier on this new data set

Update 2:

It looks like cross-lingual classification may be the right solution for my problem:

https://github.com/facebookresearch/XLM#iv-applications-cross-lingual-text-classification-xnli

It is not clear yet from the description given on the page with the link above, where to fit my own training data set and how to run classifier on my test set. Please help to figure this out. It would be ideal to find end-to-end example / tutorial on cross-lingual classification.

As you suspected, the best approach would be to take a massive multilingual pretrained language model and make use of the information about French and English that it has already learned. You can read about some good options here.

The basic idea is to train a new, lightweight network to make predictions based on the output from the pretrained model; its usual to just have a single layer feed forward network for this “fine-tuning”. Some implementations will already have this conveniently coded up for you, so check the documentation for whatever you decide to use!

Your problem is specifically a sentence pair classification problem, and there is a tutorial for that here. Pay close attention to the data processing phase of the tutorial. Overall, the differences you need to apply to what the tutorial describes are

You need to use multilingual BERT
You need to prepare your data exactly as the tutorial says about how to set up your data, but use your own snippets in place of the sentence pairs


Cross-entropy loss explanation Suppose I build a neural network for classification. The last layer is a dense layer with Softmax activation. I have five different classes to classify. Suppose for a single training example, the true label is [1 0 0 0 0] while the predictions be [0.1 0.5 0.1 0.1 0.2]. How would I calculate the cross entropy loss for this example?

The cross entropy formula takes in two distributions, $p(x)$, the true distribution, and $q(x)$, the estimated distribution, defined over the discrete variable $x$ and is given by

$$H(p,q) = -\sum_{\forall x} p(x) \log(q(x))$$

For a neural network, the calculation is independent of the following:

What kind of layer was used.
What kind of activation was used - although many activations will not be compatible with the calculation because their outputs are not interpretable as probabilities (i.e., their outputs are negative, greater than 1, or do not sum to 1). Softmax is often used for multiclass classification because it guarantees a well-behaved probability distribution function.

For a neural network, you will usually see the equation written in a form where $\mathbf{y}$ is the ground truth vector and $\mathbf{\hat{y}}$ (or some other value taken direct from the last layer output) is the estimate. For a single example, it would look like this:

$$L = - \mathbf{y} \cdot \log(\mathbf{\hat{y}})$$

where $\cdot$ is the inner product.

Your example ground truth $\mathbf{y}$ gives all probability to the first value, and the other values are zero, so we can ignore them, and just use the matching term from your estimates $\mathbf{\hat{y}}$

$L = -(1\times log(0.1) + 0 \times \log(0.5) + ...)$

$L = - log(0.1) \approx 2.303$

An important point from comments

  That means, the loss would be same no matter if the predictions are $[0.1, 0.5, 0.1, 0.1, 0.2]$ or $[0.1, 0.6, 0.1, 0.1, 0.1]$?

Yes, this is a key feature of multiclass logloss, it rewards/penalises probabilities of correct classes only. The value is independent of how the remaining probability is split between incorrect classes.

You will often see this equation averaged over all examples as a cost function. It is not always strictly adhered to in descriptions, but usually a loss function is lower level and describes how a single instance or component determines an error value, whilst a cost function is higher level and describes how a complete system is evaluated for optimisation. A cost function based on multiclass log loss for data set of size $N$ might look like this:

$$J = - \frac{1}{N}\left(\sum_{i=1}^{N} \mathbf{y_i} \cdot \log(\mathbf{\hat{y}_i})\right)$$

Many implementations will require your ground truth values to be one-hot encoded (with a single true class), because that allows for some extra optimisation. However, in principle the cross entropy loss can be calculated - and optimised - when this is not the case.


Is this a Q-learning algorithm or just brute force? I have been playing with an algorithm that learns how to play tictactoe. The basic pseudocode is:

repeat many thousand times {
  repeat until game is over {
    if(board layout is unknown or exploring) {
      move randomly
    } else {
      move in location which historically gives highest reward
    }
  }

  for each step in the game {
    determine board layout for current step
    if(board layout is unknown) {
      add board layout to memory
    }
    update reward for board layout based on game outcome
  }
}

now play a human and win :-)

Exploration: in the beginning the algorithm explores aggresively, and this reduces linearly. After say a thousand games it only explores in 10% of the moves. All other moves are based on exploitation of previous rewards.

Rewards: if the game resulted in a win, then award 10 points. If the game resulted in a draw, 0 points, otherwise -5 points.  Actually, these rewards can be "tuned", so that if the game was shorter and it was won, then award more points or if it was longer award less points.  This way the algorithm prefers winning quickly. That means that it learns to win as soon as possible, rather than aiming to win later on.  That is important so that it doesn't miss winning immediately - if it missed such a move the opponent would likely a) move there to avoid letting the AI win next time, and b) think the algorithm was stupid because it missed an "obvious" win.

This algorithm does indeed learn, so I can class it as a maching learning algorithm.

I think, but I am not sure, that it is a reinforced learning algorithm. However, according to https://www.cse.unsw.edu.au/~cs9417ml/RL1/tdlearning.html it is not temporal difference learning, because it doesn't estimate the rewards until the end, and it should be estimating the reward as it goes along. That might mean that it is not reinforced learning.

Question 1: Can I successfully argue that I am estimating the reward based on history, and still claim the algorithm is reinforced learning or even Q-learning?

Question 2: If I replace the reward lookup which is based on the board layout, with a neural network, where the board layout is the input and the reward is the output, could the algorithm be regarded as deep reinforcement learning?

Question 3: I'd don't think that I have either a learning rate or a discount factor. Is that important?

I noticed that the algorithm is pretty useless unless I train it with at least every move which an opponent tries. So in a way it feels like it's using brute force rather than really "learning". This makes me question whether or not machine learning tictactoe is really learning.  I agree that using a neural network to learn image recognition can be classed as learning because when it sees an unknown image it is able to state its classification. But that is quite useless for games like tictactoe where similar looking board layouts are totally unrelated (one may lead to a win, the other may lead to a loss). So...

Question 4: Can tictactoe algorithms be classed as real learning rather than simply brute force?

Update: regarding rewards... when the algorithm is deciding where to go, it works out the reward for each position as follows:

var total = winRewards + drawRewards + lossRewards;
move.reward = (100*(winRewards/total)) + (10*(drawRewards/total)) + (-1*(lossRewards/total));

I divide by the total number of points (for each move), because otherwise it seems to learn that one place is GREAT and doesn't give other ones a chance.  This way, we work out the win ratio regardless of how often it's been played. It's normalised in comparison to the others.

The code is available here: https://github.com/maxant/tictactoe/blob/master/ai.js

UPDATE #2: I have since figured out that this algorithm cannot be classed as using brute force because it doesn't actually learn that many games before becoming an expert. Details here: http://blog.maxant.co.uk/pebble/2018/04/11/1523468336936.html

Kudos for figuring out a working tic-tac-toe playing algorithm from scratch!

  Question 1: Can I successfully argue that I am estimating the reward based on history, and still claim the algorithm is reinforced learning or even Q-learning?

First things first, this is definitely not Q-learning.

However, I do think it classifies as Reinforcement Learning. You have implemented these key components of RL:

A state (the current board), used as input on each step.
An action (desired next board arrangement), used as output. When the action is effectively to choose the next state directly, this is sometimes called the afterstate representation. It is commonly used in RL for deterministic games.
Rewards generated by the environment, where the agent's goal is to maximise expected reward.
An algorithm that can take data about states, actions and rewards, and learn to optimise expected reward through gaining experience within the environment.

Your algorithm is closest IMO to Monte Carlo Control, which is a standard RL approach.

One of the big advantages of Q Learning is that it will learn an optimal policy even whilst exploring - this is known as off-policy learning, whilst your algorithm is on-policy, i.e. it learns about the values of how it is currently behaving. This is why you have to reduce the exploration rate over time - and that can be a problem because the exploration rate schedule is a hyper-parameter of your learning algorithm that may need careful tuning.

  Question 2: If I replace the reward lookup which is based on the board layout, with a neural network, where the board layout is the input and the reward is the output, could the algorithm be regarded as deep reinforcement learning?

Yes, I suppose it would be technically. However, it is unlikely to scale well to more complex problems just from adding a neural network to estimate action values, unless you add in some of the more sophisticated elements, such as using temporal-difference learning or policy gradients.

  Question 3: I'd don't think that I have either a learning rate or a discount factor. Is that important?

A discount factor is not important for episodic problems. It is only necessary for continuous problems, where you need to have some kind of time horizon otherwise the predicted reward would be infinite (although you could also replace the discount mechanism with an average reward approach in practice).

The learning rate is an important omission. You don't explain what you have in its stead. You have put update reward for board layout based on game outcome - that update step typically has the learning rate in it. However, for tic-tac-toe and Q-Learning, you can actually set the learning rate to 1.0, which I guess is the same as your approach, and it works. I have written example code that does exactly that - see this line which sets learning rate to 1.0. However, more complex scenarios, especially in non-deterministic environments, would learn badly with such a high learning rate.

  Question 4: Can tictactoe algorithms be classed as real learning rather than simply brute force?

Your algorithm is definitely learning something from experience, albeit inefficiently compared to a human. A lot of the more basic RL algorithms have similar issues though, and often need to see each possible state of a system multiple times before they will converge on an answer.

I would say that an exhaustive tree search from the current position during play was "brute force". In a simple game like tictactoe, this is probably more efficient than RL. However, as games get more and more sophisticated, the machine learning approach gets competitive with search. Often both RL and some form of search are used together.


emphasise some observation weights more than the others I want to emphasise (increase the weight) of only a subset of data. Lets say I have old and fresh data, I would like to say that old data has to have more weight and therefore has more influence in the decision than the new data.

In scikit-learn I found only class-weight parameter, but it does not change the weight of the samples, only of all samples within the class.

Is there a way to incorporate this emphasis into the gradient boosted trees in spark or xgboost in python?

If you have a date variable (or something similar), you can create a weight using this.

If you're using XGBoost, there is an option to specify a weight for each instance when creating the DMatrix - feed your observation weighting in here.


Is it necessary to convert labels in string to integer for scikit_learn and xgboost? I have a tabular data with labels that are in string. I will feed the data to decision trees in scikit_learn and XGBoost classifier.

Is it necessary to convert the labels in string to integers for these algorithms? Will the algorithms work less effectively when the labels are not in numerical format?

I am using python v3.7

TL;DR
If your labels are part of your feature matrix, you need to convert them to numerals using OneHotEncoding (docs). However, if your labels are just your targets, you can leave them as is.
1) Is it necessary to convert the strings to integers?
You did not specify if your string labels are one of your features in your feature matrix X, or if they are your target y. So, here is some data to cover both cases:
X = [
    ['sunny'],
    ['rainy'],
    ['rainy'],
    ['cloudy'],
    ['very rainy'],
    ['sunny'],
    ['partially cloudy']
]

y = [
    'no umbrella',
    'umbrella',
    'umbrella',
    'no umbrella',
    'umbrella',
    'no umbrella',
    'no umbrella'
]

If you try to fit a DecisionTreeClassifier with this, like this:
from sklearn.tree import DecisionTreeClassifier
clf = DecisionTreeClassifier()
clf.fit(X, y)

You get an error:

ValueError: could not convert string to float: 'sunny'

So definitely, you need to convert them to integers, for example:
from sklearn.preprocessing import OrdinalEncoder
encoder = OrdinalEncoder()
encoder.fit(X)
X_encoded = encoder.transform(X)

>> [[3.]   # sunny
>>  [2.]   # rainy
>>  [2.]   # rainy
>>  [0.]   # cloudy
>>  [4.]   # very rainy
>>  [3.]   # sunny
>>  [1.]]  # partially cloudy

This transforms your labels into integers. And now you are able to .fit() your model.
This means that features in X must be transformed to integers, however, target labels in y can remain as strings.
2) Will the algorithms work less effectively?
If you don't convert your targets y into integers, there will be no decrease in your algorithms performance.
Now, given that you need to convert your string features X into numerals, the way that you convert will affect the algorithm.
In Sklearn you can use the OrdinalEncoder (docs) or the OneHotEncoder (docs). The way in which they encode X is different, observe:
OrdinalEncoder
from sklearn.preprocessing import OrdinalEncoder
encoder = OrdinalEncoder()
encoder.fit(X)
X_encoded = encoder.transform(X)

>> [[3.]   # sunny
>>  [2.]   # rainy
>>  [2.]   # rainy
>>  [0.]   # cloudy
>>  [4.]   # very rainy
>>  [3.]   # sunny
>>  [1.]]  # partially cloudy

OneHotEncoder
from sklearn.preprocessing import OneHotEncoder
encoder = OneHotEncoder()
encoder.fit(X)
X_encoded = encoder.transform(X)

[[0. 0. 0. 1. 0.]  # sunny
 [0. 0. 1. 0. 0.]  # rainy
 [0. 0. 1. 0. 0.]  # rainy
 [1. 0. 0. 0. 0.]  # cloudy
 [0. 0. 0. 0. 1.]  # very rainy
 [0. 0. 0. 1. 0.]  # sunny
 [0. 1. 0. 0. 0.]] # partially cloudy

For algorithms like DecisionTreeClassifiers, the second option, namely OneHotEncoder is better because there are more dimensions to finding boundary lines. So you can have a shallower trees. Where as, if you encode with a simple LabelEncoder, you will need to have deeper tree.
Have a look at the tree generated given the two types of input:
LabelEncoder, notice how 3 decisions nodes are needed.

OneHotEncoder, notice how only 2 decisions nodes are needed.


Supervised Learning with Necessarily Missing Data Many discussions of missing data in supervised (and unsupervised) learning deal with various methods of imputation, like mean values or EM. But in some cases the data will be missing as a necessary consequence of the data generation process.

For instance, let's say I'm trying to predict students' grades, and one of the inputs I want to analyze is the average grades of the student's siblings. If a particular student is an only child, then that value will be missing, not because we failed to collect the data, but because logically there is no data to collect. This is distinct from cases where the student has siblings, but we can't find their grades.

Other examples abound: say we're in college admissions and we want to include students' AP exam results, but not all students took AP exams. Or we're looking at social network data, but not all subjects have facebook and/or twitter accounts.

These data are missing, but they're certainly not missing at random. And many algorithms, such as all supervised learning packages in scikit-learn, simply demand that there be no missing values at all in the data set.

How have people dealt with this in the past, and what off-the-shelf solutions are there? For instance, I believe the gradient boosting algorithm in R uses trees with three possible branches: left, right, and missing. Any other alternatives out there?

I solved a similar problem not so long ago. Let X be a numerical variable that has missing values. First, we assign the value 0 to those instances in which the value is missing. Then, we add a new categorical variable to our model, X_missing, which domain is {True,False}. We obtain a data model with mixed numerical/categorical variables. You can then apply gradient descent to train a regression model from these variables.


Metrics to measure imbalanced multi-class problem I have a multiclass imbalanced problem. The dependent variable is shown below.

              Injury
  severe Injury (Class 0)               3394
  Medium Injury (Class 1)              22318
  No Injury (Class 2)                 208753

I have used the random forest algorithm with parameter class_weight='balanced' to handle the imbalance problem. The model yields the below results.

 [[  650    12     9]
  [    3  2938  1670]
  [    7   917 40569]]

  Accuracy Score: 0.9440299305184393
  precision score: 0.9016230160324789
  Recall score: 0.8612021971135553
  AUC Score: 0.8739141097167544
  F1 score: 0.879571098748252

               precision    recall  f1-score   support

    class 0       0.98      0.97      0.98       671
    class 1       0.76      0.64      0.69      4611
    class 2       0.96      0.98      0.97     41493

avg / total       0.94      0.94      0.94     46775

Kappa Score: 0.7391731672532447

Since I am really interested in class 0 and class 1, the precision, recall, and F1 scores are calculated using "Macro" method.

 ex:
 print ('precision score:', precision_score(test_y,ry_pred, average='macro'))

So, my question is Can I take the individual classes (class 0 and class 1) precision, recall, and f1 ? or the overall average(including class 2) score for evaluating the model?

 ex:  F1 score for all three classes = (98+69+97)/3 = 0.8795
      F1 Score for 2 classes = (98+69)/2 = 83.5

If I have to take individual classes scores then, can I take class 0 and class 1 aggregate values to say how well the model is classifying the minor classes?

Also, for class 0 the model is showing high scores around 98. Is this an overfitting problem? I have evaluating model using k fold CV technique and shows below scores. It is showing the overall precision, recall, and F1 score.

     precision score 0.9009562240704383
        recall score 0.8576816035552879
            F1 score 0.8776626709718627

Please suggest.

I see two maybe 3 options here.

1.) Don't use class_weight = 'balanced'.

Instead use
class_weight = {0 : 0.01447 , 1 : 0.095 , 2 : .890}

2.) over sample the other classes using some like SMOTE.

3.) (possibly) use a model to classify label 2 only in a one-vs-rest scheme, then use a second model classify label 1 or 0. Kinda in a stacked approach. Becareful not to leak information when training though.


High-level features of a neural network I understand how to build and train a neural network like shown below, as well as those low-level features/filters. I wonder what are those high-level features: how exactly do you obtain them from a trained neural network? (Are those like the "eigenfaces"?)

Note: the image is by NVIDIA, and I don't know the specifics of the classification problem here. If needed, suppose the network is trained to distinguish human from cat.

Its about how the neural net learn inside. Usually in deep neural network you have multiple layers, the first layers will learn the low level feature then the more you approach the output layer the more the layers will learn the high level feature.
Here its identifying faces task, that's why you see that in the first layers small patterns are learned (mainly edges ...) then faces components ...

How to obtain them : you should watch what makes neuron activated in each layer depending on the input. As you know each neuron will be activated (once the DNN is trained) for specific input combinations.
By visualizing that you can get an idea about what exactly each layer has learned in term of high-low level features.


How to extract location related terms from raw text in python I want to extract location related keywords from raw text in python.
I have already tried spacy but the results were not good and I just got names of countries while I want fine-grained location mentions like streets or neighborhoods in a city.
I also have tried stanford NER but the problem is it is too slow and I need to produce my results with a good speed.
Is there any package or library for python which can solve my problem?
Also if there is any other suggestions which are not for python I will be glad to hear them.

I fiund these two tools which can find geographical terms in a text file.
mordecai: Full text geoparsing as a Python library. Extract the place names from a piece of text, resolve them to the correct place, and return their coordinates and structured geographic information.

, and CLIFF-CLAVIN: CLIFF-CLAVIN parses news articles and pulls out people, organizations and places mentioned.


Understanding how convolutional layers work After working with a CNN using Keras and the Mnist dataset for the well-know hand written digit recognition problem, I came up with some questions about how the convolutional layer work. I can understand what the convolution process is.

My first question is: What are the filters? I can understand their purpose. They are used to map edges, shapes, etc. on an image. But how are they getting initialized? Do they have random initial value or there are standard image filters that are getting used? If they are getting initialized with random value then the values should get changed on the training process of the network. If that's the case then a new question is created, how does someone backpropagate the filter of the convolutional layer? What is the algorithm behind this process?
Secondly, I have noticed that I can add an activation function to the convolutional layer in Keras. Is the entire matrix of the output getting passed through the activation function? How does the usage of an activation function changes the learning process of the convolutional layer?
Last but not least, does a convolutional layer have weight and biases like a dense layer? Do we multiply the output matrix after the convolution process with a weight matrix and add some biases before passing it through the activation function? If that's true, then do we follow the same process as we do with the dense layers to train these weights and biases?

What are the filters?

A filter/kernel is a set of learnable weights which are learned using the backpropagation algorithm. You can think of each filter as storing a single template/pattern. When you convolve this filter across the corresponding input, you are basically trying to find out the similarity between the stored template and different locations in the input.

But how are they getting initialized? Do they have random initial value or there are standard image filters that are getting used?

Filters are usually initialized at a seemingly arbitrary value and then you would use a gradient descent optimizer to optimize the values so that the filters solve your problem.
There are many different initialization strategies.

Sample from a distribution, such as a normal or uniform distribution
Set all values to 1 or 0 or another constant
There are also some heuristic methods that seem to work very well in practice, a popular one is the so-called glorot initializer named after Xavier Glorot who introduced them here. Glorot initializers also sample from distribution but truncate the values based on the kernel complexity.
For specific types of kernels, there are other defaults that seem to perform well. See for example this article.

If they are getting initialized with random value then the values should get changed on the training process of the network. If that's the case then a new question is created, how does someone backpropagate the filter of the convolutional layer? What is the algorithm behind this process?

Consider the convolution operation just as a function between the input image and a matrix of random weights. As you optimize the loss function of your model, the weights (and biases) are updated such that they start forming extremely good discriminative spacial features. That is the purpose of backpropogation, which is performed with the optimizer that you defined in your model architecture. Mathematically there are a few more concepts that go into how the backprop happens on a convolution operation (full conv with 180 rotations). If you are interested then check this link.

Is the entire matrix of the output getting passed through the activation function? How does the usage of an activation function change the learning process of the convolutional layer?

Let's think of activation functions as just non-linear "scaling" functions. Given an input, the job of an activation function is to "squish" the data into a given range (example -> Relu 'squishes' the input into a range(0,inf) by simply setting every negative value to zero, and returning every positive value as is)
Now, in neural networks, activations are applied at the nodes which apply a linear function over the input feature, weight matrix, and bias (mx+c). Therefore, in the case of CNN, it's the same. Once your forward-pass takes the input image, does a convolution function over it by applying a filter (weight matrix), adds a bias, the output is then sent to an activation function to 'squish' it non-linearly before taking it to the next layer.
It's quite simple to understand why activations help. If I have a node that spits out x1 = m0*x0+b0 and that is then sent to another node which spits out x2 = m1*x1+b1, the overall forward pass is just x2 = m1*(m0*x0+b0)+b1 which is the same as x2 = (m1*m0*x0) + (m1*b0+b1) or x2 = M*x0 + B. This shows that just stacking 2 linear equations gives another linear equation and therefore in reality there was no need for 2 nodes, instead I could have just used 1 node and used the new M and B values to get the same result x2 from x0.
This is where adding an activation function helps. Adding an activation function allows you to stack neural network layers such that you can explore the non-linear model space properly, else you would only be stuck with the y=mx+c model space to explore because all linear combinations of linear functions is a linear model itself.

Does a convolutional layer have weight and biases like a dense layer?

Yes, it does. Its added after the weight matrix (filter) is applied to the input image using a convolution operation conv(inp, filter)

Do we multiply the output matrix after the convolution process with a weight matrix and add some biases before passing it through the activation function?

A dot product operation is done between a section of the input image and the filter while convolving over the larger input image. The output matrix, is then added with bias (broadcasting) and passed through an activation function to 'squish'.

If that's true, then do we follow the same process as we do with the dense layers to train these weights and biases?

Yes, we follow the exact same process in forward pass except that there is a new operation added to the whole mix, which is convolution. It changes the dynamics especially for the backward pass but in essence, the overall intuition remains the same.

The crux for intuition is -

Do not confuse a feature and a filter. A filter is what helps you to extract features (basic patterns) from the input image using operations such as dot, conv, bias and activations
Each filter allows you to extract a 2D map of some simple pattern that exists over the image (such as an edge). If you have 20 filters, then you will get 20 feature maps for a 3 channel image, that are stacked as channels in the output.
Many such features, which capture different simple patterns, are learnt as part of the training process and become the base features for the next layer (which could be another CNN or a dense)
Combinations of these features allow you to perform your modeling task.
The filters are trained by optimizing towards minimizing a loss function using backprop. It follows the backward reasoning:

      - How can I minimize my loss?
      - How can I find the best features that minimize the loss?
      - How can I find the best filters that generate the best features?
      - What are the best weights and biases which give me the best filters?

Here's a good reference image to keep in mind whenever working with CNNs (just to reinforce the intuition)

Hope that answers your questions.


How to form Hessian matrix in BFGS Quasi-Newton Method I came across this link. In BFGS Quasi-Newton Method, a Hessian matrix is used in weight updation. Is there any resource where I can find how this hessian matrix was obtained along with a clear description of the process, as to why Hessian matrix has been taken? I could not understand the wiki article.

In optimization, you are looking for minima and maxima of a certain function f. Equivalently, under certain derivative conditions, you are looking for the zeros of g, the derivative of f.
The first equation of your link describes one step of the Newton Method - not BFGS. It involves the Hessian of f which is not accessible in practice - it's ineficcient in terms of time and memory storage. Due to this fact, some have developed methods not to calculate the hessian directly.
Thanks to Taylor expansion, you can write

g(x) = g(y) + g'(y)(x-y) + Θ(|x-y|²)

Replacing x by x(k+1) and y by x(k), and g' by the Hessian of f gives you (with the following convention : g(k+1) = g( x(k+1) )

g(k+1) ≈ g(k) + ∇²f(x(k+1)) ( x(k+1) - x(k) )
H(k+1) (x(k+1) - x(k)) ≈ g(k+1) - g(k)
where H(k) aims to approximate ∇²f(x(k)). But instead of calculating it at each step, we'll try to approximate it thanks to the previous H(k-1)

In 1970, Broyden, Fletcher, Goldfard and Shamo proposed a rank 2 update of H(k), ie

H(k+1) = H(k) + Term1 + Term2

where Term1 does not depend on H and Term2 depends on H(k). You can find these terms here.
But this solves only the time issue, not the memory storage. As for that, some have developed Limited-memory BFGS (or L-BFGS) to limit the amount of memory used.


Does sigmoid facilitate modeling non-linear decision boundaries or does this come from high-dimensional data? I'm writing up a neural network using sigmoid as the activation function. According to one lecture, sigmoid simply squashes numbers onto the (0,1) interval. To model non-linear decision boundaries, you need high dimensional data. Another lecture, this one on sigmoid in neural nets context (MIT S.191), suggested sigmoid's non-linear behaviour is the reason why a neural net can model a non-linear decision boundary. So, I'm a bit confused. Does sigmoid facilitate modeling non-linear decision boundaries or does this come from high-dimensional data because you can use the sigmoid to produce a linear decision boundary w/o incident?

The power to model non-linear decision boundaries comes directly from the non-linear activation function. You can understand this when you see that concatenating N linear transformations (i.e. dense layers) is equivalent to a single linear transformation. This is the mathematical proof with 2 linear layers:
$ y = (xW_1 + b_1) W_2 + b_2 = x W_1 W_2 + (b_1 W_2 + b_2) = x W' + b'$
(where $W'= W_1 W_2$ and $b'= b_1 W_2 + b_2$)


How to apply TFIDF in structured dataset in Python? I know that TFIDF is an NLP method for feature extraction.
and I know that there are libraries that calculate TFIDF directly from the text.
This is not what I want though
In my case, my text dataset has been converted into Bag of words
The original dataset that I "DO NOT" have access to, looks like this
RepID     RepText
------------------
1         Doctor sys patient has diabetes and needs rest for ...
2         Patients history: broken arm, and ...
3         A dose of Metformin 2 times a day ...
4         Xray needed for the chest...
5         Covid-19 expectation and patient should have a rest ...

But my dataset looks like this
RepID   Word         BOW
-------------------------
1       Doctor       3
1       diabetes     4
1       patient      1
.       .            .
.       .            .
2       patient      2
2       arm          7
.       .            .
.       .            .
5684    cough        9
5684    Xray         3
5684    Covid        5
.       .            .
.       .            .

What I want is to find TFIDF for each word in my dataset.
I was thinking of converting my dataset into a unstructured format
so it would look like this
RepID     RepText
------------------
1         Doctor Doctor Doctor diabetes diabetes diabetes diabetes ...
2         Patients patients arm arm arm arm arm arm arm ...
.
.
5684      cough cough cough cough cough cough cough cough cough Xray Xray

so each word repeated the same number of BOW
but I do not think this is the best way to do as I convert a structured dataset into an unstructured one..
How to find the TFIDF from the structured dataset? is there a library or algorithm for that?
Note :
Dataset stored in MS SQL Server, and I am using Python code.

You could use pandas pivot_table() to transform your data frame into a count matrix, and then apply sklearn TfidfTransformer() to the count matrix in order to obtain the tf-idfs.
import pandas as pd
from sklearn.feature_extraction.text import TfidfTransformer

# input data
df = pd.DataFrame({
    'RepID': [1, 1, 1, 2, 2, 5684, 5684, 5684],
    'Word': ['Doctor', 'diabetes', 'patient', 'patient', 'arm', 'cough', 'Xray', 'Covid'],
    'BOW': [3, 4, 1, 2, 7, 9, 3, 5]
})

# count matrix
df = pd.pivot_table(df, index='RepID', columns='Word', values='BOW', aggfunc='sum')
df = df.fillna(value=0)
print(df)
# Word   Covid  Doctor  Xray  arm  cough  diabetes  patient
# RepID
# 1        0.0     3.0   0.0  0.0    0.0       4.0      1.0
# 2        0.0     0.0   0.0  7.0    0.0       0.0      2.0
# 5684     5.0     0.0   3.0  0.0    9.0       0.0      0.0

# tf-idf transform
X = TfidfTransformer().fit(df.values)
print(X.idf_)
# [1.69314718 1.69314718 1.69314718 1.69314718 1.69314718 1.69314718 1.28768207]


LSTM for Stock Return Prediction I am writing my masters thesis and am using LSTMs for daily stock return prediction. So far I am only predicting numerical values but will soon explore a classification style problem and predict whether it will go up or down each day.
I have explored several scenarios

A single LSTM using as input only the past 50 days return data
A stacked (2 layers) using as input only the past 50 days return data

The results are not great for either (and I didn't expect them to be). So I tried some feature engineering using 3 day MA, 5 day MA, 10 day MA, 25 day MA, 50 day MA of the daily returns as well as the actual daily return, meaning I have 6 input features. All other variables are kept constant yet the model now overfits (see the training and test loss plots below). Does anyone have any ideas why this may be?

Test Loss in orange and Train in blue

I am not sure this type of model is a good use case for the particular task. More specifically, citing Chollet from Deep Learning with Python book,

Always remember that when it comes to markets, past performance is not a good predictor of future returns—looking in the rear-view mirror is a bad way to drive. Machine learning, on the other hand, is applicable to datasets where the past is a good predictor of the future.

— Deep learning with python, Francois Chollet
What is essentially being argued here is that, stock historical data is not a phenomenon that repeats itself based on its own underlying distribution.


LSTM's for timeseries with additional regressors I have a dataset consisting of the weekly sales of 3,000 stores over the past 5 years, and have constructed a LSTM to forecast the next year of sales, given the previous year of sales. At each timestep $t$, the features are the previous 52 observations.
$$X_{t-52:t} \rightarrow LSTM \rightarrow \hat{X}_{t:t+52}$$
The LSTM is trained using a rolling window over the timeseries, starting with the input of $X_{0:52}$ and output of $X_{52:104}$ and ending with the input of $X_{T-52:T}$ and output of $\hat{X}_{T:T+52}$, so that the cell state is built up over the whole series to eventually forecast $\hat{X}_{T:T+52}$.
However, I also have a dataset $E$ of events for which I know the dates both past and future. These are events which may affect sales such as Christmas or Easter. It is encoded as a binary vector of length n_events=30 for each week $t$. I'm wondering what the best way would be to include this data as my forecast. Ideally, I want the NN to be able to learn the uplift or decrease associated with each event and use this information in its forecast. It is more complex than seasonality because some events such as Easter move around in a way that cannot be learned by the NN.
However, the events for the next year at any timestep $t$ is a tensor of shape [n_events, forecasting_horizon] = [36, 52] with 1972 individual "features" and hence is too large to realistically include in the input to the LSTM. What other methods could I use to feed these features into the model in a way that would inform its forecast over the next 52 weeks?

You might find this link helpful
https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/


Impact of varying sequence length in ensemble GRU model I am using ensemble gru for my project and keeping different cell sizes for different models !For example, first gru model is of size 16 and the second is of 8 and 4 for the third model.
The model is running well but I don't see any difference in the results in keeping same unit size or different.
can anyone explain the impact of varying unit size for ensemble gru. ?
it would be great if answer is given with theoretical proof.

The number of units in an RNN layer determines the amount of "memory" of that layer. Higher number of units make the model (potentially) able to remember longer sequences, and to explain current values based on information that dates more back in time. If on the other side the number of units is shorter, the model at that layer won't be able to recover information way back in the past. This is tru for any Recurren architecture (simple recurrent, LSTM, GRU).

If you increase the number of units of a GRU layer and you see that there's no improvement in the quality of your model, that probably means that your time series data have "shorter memory", i.e. that you don't need to go that far back in time in order to explain current values. Of course I didn't take a look at your data, this is just the most likely reason IMHO.


Text Classification : Classifying N classes vs rest of the classes Apologies if this is naive, I am fairly new to the domain.
I have a requirement where I am trying to classify 2 types of text data, i.e, I have got 2 classes to classify my data upon. I am able to get acceptable results for them using word vectors, dimensionality reduction and then putting the data through to LinearSVC for classification. But my model is biased towards sentences with longer lengths. I know tfidf can help in this, but is there a way to apply that with word vectors, I dont want to lose its ability to predict on unseen similar data?
A follow up problem to this is, when there is data from some other class apart from the 2 classes that I have trained my data upon. I would like my model to be able to predict that the data doesn't belong to any of the 2 classes. Currently, it just predicts 1 of the 2 classes randomly.
I was thinking if I should first put my data through topic modelling, which would give an idea on what topic the text data is. Based on keywords from topic modelling, we can detect if my text data belongs to one of the 2 classes or not and then put it to my classification model for final prediction. But this doesn't seems very clean and I can see it failing as there's too much dependency on keywords generated by topic modelling. Is there any other finer and better way to do this?

This is called an open-class text classification problem, it's used in particular for some author identification problems.
I don't have any recent pointers but from a quick search I found this article: https://www.aclweb.org/anthology/N16-1061.pdf
In the field of author classification there is a similar problem called author verification, which can be treated as a one-class classification problem. You could consider using it in this way:

one-class classification between "known classes" vs. others
regular classification between the known classes


Cluster documents and identify the prominent document in the cluster? I have a set of documents as given in the example below.

doc1 = {'Science': 0.7, 'History': 0.05, 'Politics': 0.15, 'Sports': 0.1}
doc2 = {'Science': 0.3, 'History': 0.5, 'Politics': 0.1, 'Sports': 0.1}

I want to cluster the documents and identify the most prominent document within the cluster.

e.g, cluster 1 includes = {doc1, doc4, doc5. doc8} and I want to get the most prominent document that represents this cluster (e.g., doc8). (or to identify the main theme of the cluster)

Please let me know a suitable approach to achieve this :)

A very simple approach would be to find some kind of centroid for each cluster (e.g. averaging the distributions of the documents belonging to each cluster respectively) and then calculating the cosine distance of each document within the cluster from the corresponding centroid. The document with the shorter distance will be the closest to the centroid, hence the most "representative".

Continuing from the previous example:

import pandas as pd
import numpy as np
from sklearn.metrics import pairwise_distances
from scipy.spatial.distance import cosine
from sklearn.cluster import DBSCAN
from sklearn.preprocessing import StandardScaler

# Initialize some documents
doc1 = {'Science':0.8, 'History':0.05, 'Politics':0.15, 'Sports':0.1}
doc2 = {'News':0.2, 'Art':0.8, 'Politics':0.1, 'Sports':0.1}
doc3 = {'Science':0.8, 'History':0.1, 'Politics':0.05, 'News':0.1}
doc4 = {'Science':0.1, 'Weather':0.2, 'Art':0.7, 'Sports':0.1}
collection = [doc1, doc2, doc3, doc4]
df = pd.DataFrame(collection)
# Fill missing values with zeros
df.fillna(0, inplace=True)
# Get Feature Vectors
feature_matrix = df.as_matrix()

# Fit DBSCAN
db = DBSCAN(min_samples=1, metric='precomputed').fit(pairwise_distances(feature_matrix, metric='cosine'))
labels = db.labels_
n_clusters_ = len(set(labels)) - (1 if -1 in labels else 0)
print('Estimated number of clusters: %d' % n_clusters_)

# Find the representatives
representatives = {}
for label in set(labels):
    # Find indices of documents belonging to the same cluster
    ind = np.argwhere(labels==label).reshape(-1,)
    # Select these specific documetns
    cluster_samples = feature_matrix[ind,:]
    # Calculate their centroid as an average
    centroid = np.average(cluster_samples, axis=0)
    # Find the distance of each document from the centroid
    distances = [cosine(sample_doc, centroid) for sample_doc in cluster_samples]
    # Keep the document closest to the centroid as the representative
    representatives[label] = cluster_samples[np.argsort(distances),:][0]

for label, doc in representatives.iteritems():
    print("Label : %d -- Representative : %s" % (label, str(doc)))


Seaborn Pairplot not showin colors of the labels My dataframe looks similar to this

I am using the following line to graph the pairplot

sns.pairplot(iris, hue = 'States', height=7, aspect=0.8 )

The graph is good but I am confused why do the labels on the side have no color corresponding to the graph?

So, the question is asking about why the legend of the plot displayed here is not displaying any colours.

Without looking at the code you have used to generate the pair plot, I can see that, for example, the column name which inputted as the "hue" to the pair plot function should be used as the legend title. However, the title of this legend on the plot is not the same as the given column name. So, my first intuition leads to me to believe that you might have overwritten the legend attributes after this line sns.pairplot() line (maybe using plt.legend() after the pairplot line). Hence, maybe why the colours do not appear as well.


How to scale numbers according to proportions I have some data that I'm trying to represent in a visualization with a week-over-week change. So the source can be something like:

Fruit    This Week   Last Week  Delta   Proportion
--------------------------------------------------
Bananas     10           8         2        25%
Oranges     110         100        15       15%
Apples      400         390        10      2.6%

What are some ways that I can effectively show that even though Bananas increased by 25% (technically the "biggest mover" of the week) it's an increase that is "overstated" because of the small overall numbers of bananas sold? I think I need a scaling approach of some sort but can't quite figure it out.

Seems like you want to show both percentage and magnitude at the same time.

I would then suggest that you plot the two values for comparison.

You could plot all in one chart with two axes:

or split it into two charts:

both visualizations make clear that Bananas have the largest increase, while Apples is the fruit that sells the most.


Approach to Text Classification Problem Much of the vocabulary here is new to me so forgive me if I misspeak.

I'm attempting to find an approach to a very simple classification problem. I have a set of description such as:

House : Building with several walls some windows and a roof

Car : Vehicle with four wheels, a steering wheel, and an engine

We have a third party giving a description of what they:

My House : Red walls, a balcony, a roof and a great view

Give these loose natural language descriptions what would be a good approach to classifying "My House" as a "House". Would a simple word counting approach be applicable, perhaps with some sort of emitting dictionary {"the", "a", etc}. Any alternative algorithms to consider?

It looks like a classical email classification problem: Spam or Ham.
In the following I will use some specific slang, do not worry, look it up in Google, if you will see specific terms you do not know.

Prerequisite: ideally your dataset will be "balanced", having, say, 50% cars and 50% houses, could be 40/60 as well. Problems will arise, when one class will be say < 10%.

So called "bag of words" can be seen as a start.

Steps:

Your dataset consist of two columns, say, "description" and "label"
Label has only two values car or house
Tokenize your decription column (single words)
Cleanse the desc column: remove stop words (like and, the , a - as they do not have any value), remove punctuation, possibly stem the words, remove numbers
calculate document-term-matrix, possibly use TF-IDF

By the end of this operation you will have a dataset with a label column (car or house) and long number of word (or even n-gram, e.g. "great_view" is a bi-gram) columns containing binary values:

Label; vehicle, balcony, wheel, ..., ...., great_view
Car,   1,        0,      1, ....., ...., ..., 0
House, 0,        1,      0, ..., ...,         1

Then use naive bayes or logistic regression as a start to train your model.
Pre-process every new description as above and use your trained model to assign a probability to be a "car" or a "house", check the confusion matrix, maybe adjust the threshold .

Everything I described can be done e.g. in R or Python.
In R use the text mining package "tm".


Keras/TF: Making sure image training data shape is accurate for Time Distributed CNN+LSTM The comprehensible data shape to me is like:
(9186, 120, 120, 1)
this means 9186 entry, of 120 by 120 pixel grey images. I learnt that using Time Distributed to design a CNN combined with an LSTM model, could learn more from images, knowing they are sequenced.

Following some tutorials, I found out I should expend another dimension which describes moving frames, in my case, I have 6 hours image, this would be good as a moving sequence, (probably I should try other lengths because time-series images are not of regular lengths, example:

8:00 img1 target1
9:00 img2 target2
10:00 img3 target3
11:00 img4 target4
12:00 img5 target5
13:00 img6 target6
19:00 img7 target7
   ...

My question is, when I expend another dimension of lets say length 6, how could such a model here output for every entrie exactly one prediction, but using a frame of length 6 to learn the next ?

train_images.reshape((1531, 6, 120, 120, 1)).shape

=> This gives me the impression, that such a model would output a prediction for a sequence of 6, which means 1531 result.

Am I understanding it wrong ?

[Edit following wind's answer]

I think my problem persists but I was not aware, I declared the first layer as

TimeDistributed(
        Conv2D(64, (3, 3), activation='relu'),
        input_shape=(6, 120, 120, 1)
    )

for x_train as

train_images_ = train_images.reshape((1531, 6, 120, 120, 1))

while y_train declared as

y_train_ = y_train.reshape((1531, 6, 8, 1))

as you can see x_train and y_train have the same length, with a frame of 6, image_length of 120, while, y_train consists of 8 target columns.

To my understanding, this is the way to follow with tensorflow, unfortuanatly, this is the error I am getting...

  ValueError: Error when checking target: expected dense_2 to have 2
  dimensions, but got array with shape (1531, 6, 8, 1)

As I understand your question correctly, you worry, that change of input shape (adding time-dimension) will affect your output shape. If so, you're wrong, the output is independent of input. The output shape depends only on the structure of a neural network.


How to automate left join of multiple data frames with single data frame one by one in R i have 206 data frames named after country(FRANCE,GERMANY,HONGKONG...) with same 37 variables in these 206 files and another data frame name pf with 9 variables.

i want to do left join of these 206 data frames with PF data frame one by one selecting all variables and store in other data frame. left join is based on two condition( hcom_id=tnow_id and city=region). hcom_id and city are variables in 206 different data frames and tnow_id and region are in PF

i am using sqldf package to join:
a<-sqldf("select * from FRANCE left join PF on (FRANCE.hcom_id=pf.tnow_id and FRANCE.city=pf.region)")

I am struggling to automate it using loop rather than doing 206 times separately.

Any idea please suggest

If your data resides in clean form inside a database already:

write an sql statement that includes a series of UNION statements for your 206 country tables. You can use R or any other programming language to create this string.

save this series of 206 UNIONs as a view (maybe as an indexed view / materialized view), or as  a new table, or as a stored-procedure that creates a temp-table on your database server

perform one SQL join on the DB server

fetch the data batch-wise


Mathematical way of identifying wrong suggestions or outliers I have a hypothetical scenario where i have 100 classifiers to which if a person's name is given as input, it will return a class for the person.
Eg.
Input1 -Donald Trump

30/100 classifiers returns politician as the class
20/100 classifiers returns business man as the class
10/100 classifiers returns leader as the class
10/100 classifiers returns american as the class
10/100 classifiers returns republican as the class
10/100 classifiers returns sportsman as the class
3/100 classifiers returns priest as the class
3/100 classifiers returns doctor as the class
2/100 classifiers returns engineer as the class
1/100 classifiers returns indian as the class
1/100 classifiers returns sportsman as the class
In the above case i take 10 votes as a threshold, i can somewhat correctly define Donald Trump, though a definition of sportsman might be wrong. However 10 seems to be a decent threshold
Input2 -Christiano Ronaldo
20/100 classifiers returns sportsman as the class
20/100 classifiers returns foot ball player as the class
13/100 classifiers returns real madrid as the class
13/100 classifiers returns manchesterunited as the class
12/100 classifiers returns juventus as the class
12/100 classifiers returns european as the class
2/100 classifiers returns portugese as the class
2/100 classifiers returns cricketer as the class
2/100 classifiers returns american as the class
2/100 classifiers returns chinese as the class
2/100 classifiers returns korean as the class
In the above example, if i take 12 votes as the threshold, it correctly defines Christiano Ronaldo, though we might be missing portugese tag since its vote is only 2. However we are doing a good job here i guess.
My problem is, if i have an api that returns the votes and class of famous persons this way, what is the best mathematical aproach to dynamically find the best possible threshold value above which i can say that the definition is correct and below which you need to have a look if the classes are correct

what is the best mathematical aproach to dynamically find the best possible threshold value above which i can say that the definition is correct and below which you need to have a look if the classes are correct

First, in general it's clear that there's no way to be certain that an answer is correct or incorrect: for instance if for some reason all the classifiers return the same wrong answer, then the threshold condition is satisfied but the answer is wrong.
Now the only way to find an optimal threshold to decide whether an answer is acceptable is to use a set of labelled examples in order to evaluate the answers of the classifiers.
Why? Because the correctness of the answers cannot be determined by any mathematical formula.
So the goal of the game is to assess to what extent this set of classifiers can be trusted to give a correct answer, and that depends on the threshold. So the threshold is a parameter of the predicting system, and this parameter can be estimated (tuned) based on some validation data. The basic method consists in trying all the possible values for the parameter, evaluate the performance in every case and pick the value which obtains the highest performance. Note that there are many  possible evaluation measures for such a scenario: the measure should be chosen carefully depending on the goal of the system (e.g. is it better to have one false positive answer or false negative?).


Boosted tree regression loss function when data has occasionally very large values to predict? I have a regression problem where most of my target variables are down in the range 5-30, but occasionally the target variable will spike up to 100, 500, or even 5000. These values are not spurious outliers that should be removed, but are values I'd like the prediction algorithm to try to capture. However, I do not want the error on these variables to dominate the training of the tree. Conceptually, the percent error is more akin to what I'm interested in (although it doesn't have to be it exactly). Specifically, when the target is 30 and I predict 15, I consider that just to be similar as when the target is 5000 but my prediction is 2500. I don't want a 2500**2 squared error to overwhelm the 15***2 squared error.

For this type of problem, what is the best way for me to tackle this issue? Data transform? Custom loss function? Etc?

I am working almost on the same problem these days:

I have tried two options using XGB Regression with different objective functions including:

Using a linear regression objectiive function ("reg:linear" or "reg:squarederror") and transformed the target to the log space
Using the gamma objective function ("reg:gamma"), which is useful for a skewed target with gamma distribution, e.g., insurance claims severity. In this case, I did not transform my target to the log space.

You can try these two cases and see which one performs better. However, in my case, option 1 performed better than option 2 (around 15-20%).

Also, you can try "reg:squaredlogerror"


What is wrong with my Precision-Recall curve? Hi,
I found this:
https://github.com/rafaelpadilla/Object-Detection-Metrics
I prepared my data: ground-truth and prediction files with bounding boxes.
but I got a very strange plot. What do you thing?

Overall the performance of the model is very low, since precision and recall can be up to 1.
Since model performance is low, it will be sensitive to noise resulting in erratic behavior.


Are "Gradient Boosting Machines (GBM)" and GBDT exactly the same thing? In the category of Gradient Boosting, I find some terms confusing.
I'm aware that XGBoost includes some optimization in comparison to conventional Gradient Boosting.

But are Gradient Boosting Machines (GBM) and GBDT the same
thing? Are they just different names?

Apart from GBM/GBDT and XGBoost, are there any other models fall into
the category of Gradient Boosting?

Boosting is an ensemble technique where predictors are ensembled sequentially one after the other(youtube tutorial. The term gradient of gradient boosting means that they are ensembled using the optimization technique called gradient descent (Boosting Algorithms as Gradient Descent.
Given this, you can boost any kind of model that you want (as far as I know). Moreover in the scikit learn library, gradient boosting, its under the ensemble folder. You can boost any kind of model (linear, svm) its just that decision trees achieve normally great results with this kind of ensemble. Same way that you can do a bagging with any kind of estimator, but if you do it with a decision tree and add a couple technichalities more, you can call it Random Forest.
From scikit learn documentation: GB builds an additive model in a forward stage-wise fashion; it allows for the optimization of arbitrary differentiable loss functions. In each stage a regression tree is fit on the negative gradient of the given loss function.

But are Gradient Boosting Machines (GBM) and GBDT the same thing? Are they just different names?

Gradient boosting machines are a kind of ensemble and Gradient Boosting Decision Tree the particular case when a tree is used as the estimator.

Apart from GBM/GBDT and XGBoost, are there any other models fall into the category of Gradient Boosting?

You can use any model that you like, but decision trees are experimentally the best.
"Boosting has been shown to improve the predictive performance of unstable learners such as decision trees, but not of stable learners like support vector machines (SVM)." Kai Ming TingLian Zhu, Springer


How can a ML algorithm learn to classify fake news? I am new in Machine learning techniques and in fake news detection by using these algorithms (SVM, nn, logistic regression,..).
I would like to understand how an algorithm can learn from a training set which include news and fake news, what it will be necessary to have (target), what type of information can be relevant for a good analysis and learning ...
I read many papers and most of them do not mention the type of news used, the criteria for building the classifiers, but they only show the results of the prediction when a new news is generated.
This could be perfectly fine, but it could be also easily to say: I got this amazing results!, without understanding how these results were got.
I would appreciate if you could show me an example of how an algorithm can learn from text (not necessarily news, but also tweets, or something else would be fine).
Following discussion with Erwan: one of his previous answer partially has answered my question. However I would like to understand the following. One needs to have a corpus, then label news/tweets in fake/not fake, then run the model. But how the algorithm works on texts and takes relevant words or features for detecting fake news?
So my question is:
If I have a corpus on Trump, would the algorithm be able to detect fake news on Vitamin C, without any words (verb, adj, noun,..) in common between the two dataset, except stopwords?

Following discussion with Erwan: one of his previous answer partially has answered my question. However I would like to understand the following. One needs to have a corpus, then label news/tweets in fake/not fake, then run the model. But how the algorithm works on texts and takes relevant words or features for detecting fake news?

First, let me emphasize that the concept of "fake news" is very vague and subjective. This fact alone is a red flag for any rational data scientist, since if humans don't always agree what is the correct answer it's going to be difficult to know whether a program gives the correct answer or not.
Now let's assume one has some news data annotated as fake or not. We can represent every news item as a vector of features (for example as a simple one-hot encoding vector of words), and for every document provide these features together with the "fake or not" boolean label to a supervised ML algorithm.
The supervised ML algorithm doesn't know and doesn't care about the semantic task, that is it doesn't have any idea what the label represents. It's job is only to find the best way to predict a given label (whatever this represents) from the given features. Typically it does this by measuring statistical links between features and labels, for instance using the fact that a particular word is often associated with a particular label in the training set.
It can do this part very well, but the rest is up to humans: are the features really good indications for the label? are the labels really representative of the task? Is the amount of training data sufficient? Is the training data diverse enough? Is the ML algorithm the right choice for this task? How to evaluate and interpret the results? Etc.
The point is: no matter how complex it is, the algorithm doesn't care about the meaning, it just uses the relations that it finds in the data. If for some reason the word "rhinoceros" always appears with the label "fake" in the training data, then when applying the system any news containing the word "rhinoceros" is likely to be predicted as fake. It's easy to see that this can lead to serious errors, depending on the quality/diversity of the data used for training. It's also easy to see that, with a very "sensitive" task such as fake news detection, a good evaluation result with the same kind of data as the training data is not a proof that the system can actually detect fake news in any context, if only because it's impossible to have a fully representative sample of any kind of fake news past, present and future.

So my question is: If I have a corpus on Trump, would the algorithm be able to detect fake news on Vitamin C, without any words (verb, adj, noun,..) in common between the two dataset, except stopwords?

As you can guess from my explanation above, it's not sure at all that the algorithm would easily adapt to a new type of data. That being said, normally a good algorithm on this task would probably focus on the subtle cues rather than the topic words, for instance the terms which are meant to trigger an emotional response with the reader. To some extent, these indications might work across topics. But again there's no guarantee that the system works in general, especially since the people who write and disseminate fake news can easily change their techniques if these become easily detected.


Should the lambda for L1 norm regularizer inversely be proportional to the number of trainable weights? Say I want to implement Conv2D in keras and for each Conv2D layer, if I apply 20 filters of [2,3] filter on an input with depth of 10, then there will be 20*(2*3*10+1) = 1220 trainable weights.

the value of L1 norm would proportionally increase the more trainable weights there are. Similarly for L2 norm.

So shouldn't the lambda, as in kernel_regularizer=l1(lambda), be inversely be proportional to the number of trainable weights?

intuitively for me, if lambda of 0.1 worked for 10,000 weights, then applying the same or bigger lambda for 1 million weights doesn't make sense to me.

Backpropagation doesn't handle Regularization like this i.e. if you are thinking "10 weights make penalty 100, so 100 weights will make penalty 1000.
So let's have a smaller $\lambda$

Backpropagation uses the partial differentiation of the Loss. Now the Loss has an extra $\Sigma$$w_i^2$ and so the derivative will have an extra piece which will be proportional to the weight(Derivative of $x^2$ = $2*x$).
It will be separate for every weight, so the number of weights will have no impact

The new equation will be -
$w_i$ = $w_i$ - (backprop stuff as before) - $\lambda$*$w_i$

$\lambda$ is just a knob. You also have any value proportional to $w_i$ e.g. 5% of $w$.


Additive vs Multiplicative model in Time Series Data The above time series plot is a daily closing stock index of a company. I want to know which model between additive and multiplicative best suits the above data. I know what the two models are, but i haven't been able to figure out the correct model for the above data. Also, is there any way other than simple visualisation which can help me decide the correct model?

I want to know which model between additive and multiplicative best suits the above data.

It is hard to tell just by looking at it.

A multiplicative decomposition roughly corresponds to an additive decomposition of the logarithms.
The additive decomposition is the most appropriate if the magnitude of the seasonal fluctuations, or the variation around the trend-cycle, does not vary with the level of the time series. When the variation in the seasonal pattern, or the variation around the trend-cycle, appears to be proportional to the level of the time series, then a multiplicative decomposition is more appropriate. Multiplicative decompositions are common with economic time series.

An alternative to using a multiplicative decomposition is to first transform the data until the variation in the series appears to be stable over time, then use an additive decomposition. So, basically you need to check for heteroskedasticity, eliminate that if it is there by transformations and do an additive decomposition of the transformed series.

Most common transformations are log or square root of the series and are special cases of Power transform.

Reference:
Forecasting principles and practice


Count and summarise ID's each day while creating a new column that shows the accumulated ID's I have two column, first one being the ID of a customer and second one being the Date of purchase.

    ID       Date
     1 2017-01-17
     2 2017-01-17
     3 2017-01-17
     4 2017-01-17
     5 2017-01-17
     1 2017-01-17
     7 2017-01-17
     1 2017-01-17
     9 2017-01-18
     2 2017-01-18
     3 2017-01-18
     5 2017-01-18
     1 2017-01-18
     2 2017-01-18

I would like to summarise the Purchases made by a Customer at one day and create a third column that shows the amount of purchases for the customer on that date.

This is a base solution for your problem:

aggregate(paste(ID , Date) ~ ID + Date, data = df, FUN = length)

there are many more solutions like any of the ones below, using dplyr:

library(dplyr)
df %>%  group_by(ID, Date) %>% summarise(PurchaseCount = n())
df %>% group_by(ID, Date) %>% tally(name="PurchaseCount")
df %>% group_by(ID, Date) %>% count(name="PurchaseCount")
df %>% group_by(ID, Date) %>% add_tally(name="PurchaseCount")
df %>% group_by(ID, Date) %>% add_count(name="PurchaseCount")

or by using data.table package:

library(data.table)
setDT(df)[, PurchaseCount:=.N, by = list(ID, Date)]

or using sqldf package:

library(sqldf)
sqldf("SELECT ID, Date, COUNT(*) as PurchaseCount
       FROM df
       GROUP BY Date, ID")

or plyr:

plyr::count(df, c('ID','Date'))

I personally prefer data.table as it is directly writes to the dataframe and often is time efficient. aggregate is also favorable when you wanna avoid loading new libraries. dplyr generally make your code more legible as it uses pipingpersonal opinion.


Discretisation Using Decision Trees I'm new to the machine learning and working on a supervised classification problem. I used  discretization  process to transform continuous variables into discrete variables. So I followed this article to implement it. But when repeat same process with same values it generate different boundary values. Can anyone explain about it?

X_train, X_test, y_train, y_test = train_test_split(train[['tripid', 'Hour', 'is_FairCorrect']],train.is_FairCorrect , test_size = 0.3)

tree_model = DecisionTreeClassifier(max_depth=2)
tree_model.fit(X_train.Hour.to_frame(), X_train.is_FairCorrect)
X_train['Age_tree']=tree_model.predict_proba(X_train.Hour.to_frame())[:,1]

pd.concat([X_train.groupby(['Age_tree'])['Hour'].min(),
           X_train.groupby(['Age_tree'])['Hour'].max()], axis=1)

But when repeat same process with same values it generate different boundary values. Can anyone explain about it?

This is because you're not setting the random_seed in the in the train_test_split, which means that the training data is shuffled in a different way on each run.

With a quick check using one of sklearn's datasets, you can check that this is the issue:

from sklearn.datasets import load_boston

X,y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state=4)
tree_model = DecisionTreeClassifier(max_depth=2, random_state=2)
tree_model.fit(X_train, y_train)
y_pred = tree_model.predict_proba(X_train)[:,1]
X_train_df = pd.DataFrame(X_train, columns = ['sepal_len', 'sepal_wid',
                                              'petal_len', 'petal_wid'])
X_train_df['Age_tree'] = tree_model.predict_proba(X_train)[:,1]
X_train_df.Age_tree.unique()

This will produce the same boundaries in all runs, in this case array([0.        , 0.90697674, 0.03030303]). Whereas if you don't set the random seed you'll get different probabilities and boundaries on each run.


Tensorflow: how to look up and average a different amount of embedding vectors per training instance, with multiple training instances per minibatch? In a recommender system setting: let's say I want to learn to predict future item purchases based on user past purchases using an approach inspired by Youtube's recommender system:

Concretely, let's say I have a trainable content-based network that receives as input an item and, based on its content, returns an embedding for such item. Now, let's say each user has purchased a variable number of items in the past (some users might have purchased 5 items, others maybe 1, others maybe 10, some outliers maybe 100, etc.). I want to generate a user vector, a candidate item vector and then a user-item match score as follows:

I map each item purchased by that user to its embedded item vector using the trainable content-based network
I calculate the average of all those embedded item vectors (as illustrated in the picture)
I apply a couple of ReLu layers on top of this average, thus obtaining a user vector
I map a candidate item (to be recommended) to its embedded item vector using the same trainable content-based network of step 1 (the weights of this network are always shared, like a Siamese network so to speak)
Finally, I compute the dot product between the user vector and the candidate item vector, apply a cross entropy loss during training, etc.

So my question is about the technical details of how to implement the embedding lookup and average of a variable number of embedded item vectors per user using Tensorflow, considering that during training each mini-batch may contain many training instances, where each training instance possibly consists of a different user with a different amount of purchased items in the past. Although the context is different, my question is very similar to this one, but unfortunately nobody has answered that question up to now.

Use tf.gather().

Single instance case

In the example below, we selected a variable number of embedding vectors from the matrix embedding. The selection indexing vector user can be of variable length. Then we calculate the average embedding.

with tf.Graph().as_default():
    embedding = tf.placeholder(shape=[10,3], dtype=tf.float32)
    user = tf.placeholder(shape=None, dtype=tf.int32)
    selected = tf.gather(embedding, user)
    average = tf.reduce_mean(selected, axis=0)

    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        embedding_ = np.random.randn(10,3)
        user_ = [1,3,5]
        print(sess.run(average, feed_dict={embedding:embedding_, user:user_}))
        print(np.mean(embedding_[user_], axis=0))

Multiple instances in a mini-batch

You may manually specify the first vector in embedding to be a zero vector, and patch the above selection vector with 0s. For example

with tf.Graph().as_default():
    embedding = tf.placeholder(shape=[10,3], dtype=tf.float32)
    user = tf.placeholder(shape=[None, None], dtype=tf.int32)
    selected = tf.gather(embedding, user)
    non_zero_count =  tf.cast(tf.count_nonzero(user, axis=1), tf.float32)
    embedding_sum = tf.reduce_sum(selected, axis=1)
    average = embedding_sum / tf.expand_dims(non_zero_count, axis=1)

    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        embedding_ = np.concatenate([np.zeros((1,3)),np.random.randn(9,3)], axis=0)
        user_ = [[3,5,7,0], [1,2,0,0]]
        print(sess.run(average, feed_dict={embedding:embedding_, user:user_}))
        print(np.sum([embedding_[i] for i in user_], axis=1) / np.atleast_2d(np.count_nonzero(user_, axis=1)).T)

You can use tf.gather() like this even if the embedding is a trainable variable instead of a placeholder.


Random Forest Overfitting, issues with mtry=1? I am constructing what is known as an 'Expected Goals' model for football. This metric measures shot quality and a probability is assigned to a shot to achieve this, i.e. the chance a shot will be converted. To create this model I am using a random forest classifier. For evaluation purposes I am only interested in the accuracy of the probabilities rather than strictly classifying shots, therefore, I use the predictions to calculate the Mean Square Error where goal = 1 and no goal = 0. The MSE for the test set along with two benchmarks are as follows:

[1] "test.rf_mse: 0.0856633533734135"
[1] "comparison_model_mse: 0.0820007160001345"
[1] "naive_baseline_mse: 0.0912291249164997"

Note that the comparison model should be better than mine but the naive baseline should be worse. When looking at this the model looks to be doing okay. However, when I apply the same steps to the training set I get the following:

[1] "test.rf_mse: 0.0112001023587005"
[1] "comparison_model_mse: 0.0722459417565357"
[1] "naive_baseline_mse: 0.0858344459279039"

Here the MSE falls to unrealistic levels. Doesn't this mean that my model is overfitting? I understand that the idea that random forests can't overfit isn't strictly correct as all models can overfit to some extent, but for the model to be overfitting by this much must mean I am misunderstanding something here.

#FINAL MODEL FOR SHOTS DATASET
set.seed(5555)
trainIndex <- createDataPartition(shots$goal.miss, p = .75, list = FALSE)
train_set <- shots[ trainIndex,]
test_set  <- shots[-trainIndex,]

set.seed(1000)
rf.shots <- randomForest(as.factor(goal.miss) ~ ., data=train_set, ntree=500, mtry=5)

###TEST RESULTS
pred <- predict(rf.shots, test_set, type="prob")
test_set$predictions.test <- pred[,2]
mean(((test_set$goal.miss - test_set$predictions.test)^2))

###TRAIN RESULTS
pred <- predict(rf.shots, train_set, type="prob")
train_set$predictions.train <- pred[,2]
mean(((train_set$goal.miss - train_set$predictions.train)^2))

NOTE: I moved on to using the 'ranger' random forest package which allowed me to tune more hyperparameters. From a grid search, I found the only parameter that appeared to cause overfitting was mtry. I could remove overfitting by setting mtry to 1. However, having mtry at one does not minimise the MSE for the test_set and I have 29 features in the model meaning the default mtry should be 5. Therefore, I believe I must have some issues with my feature selections that is causing the overfitting. Either that or there are circumstances whereby mtry-1 is optimal? But I am not convinced by that.

While the bagging of random forests is meant to reduce overfitting, they generally will overfit more than e.g. a parametric model like logistic regression.

Having a larger gap between training and testing scores is not necessarily a problem; you may still prefer the model if its testing score is higher than other models'.  In a purely statistical perfect world, the test score is all you should care about.  In real problems, there are some other things to consider: a more-overfit tree model is likely less interpretable, potentially takes longer to train and make predictions, may be more susceptible to concept drift, etc.

You should, as you have, see if reducing the model's capacity can increase your testing score (and maybe see if doing so can reduce the train-test gap even if it costs a little on test score, see above).  With min.node.size, I would suggest using a fraction of the training set size rather than the small fixed integers that are default in ranger.  Note that ranger does also have a max.depth parameter if you'd prefer to fix that directly.  Reducing mtry is also a good approach, but 1 strikes me personally as too random, and I'd worry that you'd need a ton of trees to stabilize results; using mtry in combination with a tree complexity parameter hopefully produces a higher test score without too outrageous of a train-test gap.


Advice on where to continue in the field of data engineering and machine learning I finished a 28 hours Machine learning with python (Basic course) on Udemy, and it was very beneficial.
My aim, is to be able to understand what is ML and how to use its concepts while working with data.
I am confused about where to continue. My goal is to be a data engineer/analyst or at least data engineer.
I am searching for courses on coursera, because of its credibility.
I found a course from University of Michigan: Applied data science with Python.
But I am confused if I should take something else in order to start my data science job search. I know it is going to be a long process to finish.
Any suggestions on how to proceed in this journey as I am really confused.
Feel free to remove my question if it was not supposed to be asked here.

Dataquest is a great place to start. It will help you bridge the gap from just learning to actually coding as a data analyst, data engineer, or data scientist. Unlike most online courses, there are no videos, and they provide an interactive coding and learning environment that makes it very easy to learn the practical techniques you'll actually need to work on real-world data and problems.
The best part about Dataquest is their guided projects that are part of every topic you learn. If you want to eventually get a job as a data analyst, scientist, or engineer, you are going to need a portfolio of projects to show what you know, and Dataquest helps guide you through the entire process.
fast.ai is another great place to learn about machine learning, and deep learning in particular, all for free. They teach using a top-down approach that will get you working on your own projects and data in no time.


Calculation and Visualization of Correlation Matrix with Pandas I have a pandas data frame with several entries, and I want to calculate the correlation between the income of some type of stores. There are a number of stores with income data, classification of area of activity (theater, cloth stores, food ...) and other data.

I tried to create a new data frame and insert a column with the income of all kinds of stores that belong to the same category, and the returning data frame has only the first column filled and the rest is full of NaN's.
The code that I tired:

corr = pd.DataFrame()
for at in activity:
    stores.loc[stores['Activity']==at]['income']

I want to do so, so I can use .corr() to gave the correlation matrix between the category of stores.

After that, I would like to know how I can plot the matrix values (-1 to 1, since I want to use Pearson's correlation) with matplolib.

Another alternative is to use the heatmap function in seaborn to plot the covariance. This example uses the Auto data set from the ISLR package in R (the same as in the example you showed).

import pandas.rpy.common as com
import seaborn as sns
%matplotlib inline

# load the R package ISLR
infert = com.importr("ISLR")

# load the Auto dataset
auto_df = com.load_data('Auto')

# calculate the correlation matrix
corr = auto_df.corr()

# plot the heatmap
sns.heatmap(corr,
        xticklabels=corr.columns,
        yticklabels=corr.columns)

If you wanted to be even more fancy, you can use Pandas Style, for example:

cmap = cmap=sns.diverging_palette(5, 250, as_cmap=True)

def magnify():
    return [dict(selector="th",
                 props=[("font-size", "7pt")]),
            dict(selector="td",
                 props=[('padding', "0em 0em")]),
            dict(selector="th:hover",
                 props=[("font-size", "12pt")]),
            dict(selector="tr:hover td:hover",
                 props=[('max-width', '200px'),
                        ('font-size', '12pt')])
]

corr.style.background_gradient(cmap, axis=1)\
    .set_properties(**{'max-width': '80px', 'font-size': '10pt'})\
    .set_caption("Hover to magify")\
    .set_precision(2)\
    .set_table_styles(magnify())