Implementing the Dependency Sensitive CNN (DSCNN ) in Keras I'm currently trying to implement the Dependency Sensitive Convolutional Neural Network for Modeling Documents by Rui Zhang in Keras.
For me this is the first network to implement in Keras, so I came up with some questions.

The network looks as follows:

So I'm starting with the input and my thought was that I have to input sequence of sequences, e.g. a list of sentences, where a sentence is a list of words that are word embeddings.
The sentences should be padded, so that all sentences have equal length. Also the documents should be padded, so that each document as equal number of sentences.

We will have a batch of the following type (only symbolic -> according to `fit()´):

s_i = np.array(shape=(max_sentence_length, word_embedding_dimensions))
doc_i = [s_1, s_2, s_3, ..., s_(max_sentences_per_doc)]
batch = [doc_1, doc_2, doc_3, ...]

where we_idx_ij is the word embedding index for the j-th word in the i-th sentence of a certain document.

Question 1:
Does this make sense?

I still have to think about how I will exactly include the w2v word embeddings, but with the model in Keras I would proceed as follows:

Includes:

Include the needed methods/layers and initialize symbolic constants

import numpy as np
from keras.layers import Activation, Input, Embedding, LSTM, concatenate, AveragePooling1D, MaxPool1D, Conv1D, TimeDistributed

# constants (for testing)
max_sentences_per_document = 10
max_words_per_sentence = 100
w2v_dimensions = 300
vocab_size = 20000
batch_size = 60

Model:

# preparing some shared layers
shared_embedding = Embedding(input_dim=w2v_dimensions, output_dim=vocab_size, weights=[W])  # optional,cause no training
shared_sentence_lstm = TimeDistributed(
    LSTM(input_dim=max_words_per_sentence, return_sequences=True, activation='tanh'),
    input_shape=(max_words_per_sentence, w2v_dimensions)
)
shared_sentence_lstm_2 = LSTM(activation='tanh')

# sentence modeling
sentence_inputs = [Input(shape=(batch_size, max_words_per_sentence, )) for i in range(max_sentences_per_document)]
sentence_modeling = [shared_embedding(sentence_inputs[i]) for i in range(max_sentences_per_document)]
sentence_modeling = [shared_sentence_lstm(sentence_modeling[i]) for i in range(max_sentences_per_document)]
sentence_modeling = [AveragePooling1D()(sentence_modeling[i]) for i in range(max_sentences_per_document)]
sentence_modeling = [shared_sentence_lstm_2(sentence_modeling[i]) for i in range(max_sentences_per_document)]

# document modeling
doc_modeling = concatenate(sentence_modeling)
doc_modeling = Conv1D(filters=100, kernel_size=[3, 4, 5], activation='relu')(doc_modeling)
doc_modeling = MaxPool1D()(doc_modeling)
doc_modeling = Activation('softmax')
doc_modeling.compile(loss='hinge', optimizer='sgd', metrics=['accuracy'])

Question 2:

Does it make sense to have the word embedding already in the batch-data or should I provide the s_i only with an index per word that relates to an index inside the word-embedding matrix? (I guess the latter would make more sense regarding the memory)

Question 3:

If I provide the word embeddings in the batch-data, I wouldn't need the embedding-layer, correct?

Question 4:

Would this network work?

Question 5:

Does this code implement the network as it is proposed in the paper?

Question 6:

Do you have any suggestions to improve the performance?

I hope I'm not too far off the track with this implementation and looking forward to your answers :-)
Thanks in advance!

Intermediately I finished the network and it is working great :-) Anyone who is also looking for this network can download it on my Github page.
To refer directly to my questions:
Question 1:
The proposed code actually makes partially sense:

One should use the shared-embedding layer to save memory usage, so this part makes sense
In the initially proposed code I used TimeDistributed layer, this is the wrong way for the purpose of the network. We actually have to use a simple shared LSTM layer. Time distributed layers are needed for many-to-many networks. For more information about this, I refer to karpathy's blog and Brownlees blog.
I applied the second LSTM layer before the concatenation, with this I applied two LSTMs to capture the joint meaning of the sentences. That is not the intention of the DSCNN, so we have to apply the one LSTM layer after the concatenation

So far the code looks like:
sentence_inputs = [Input(shape=(max_sentence_len, embedding_dim,), name="input_" + str(i))
                   for i in range(max_sentences_per_doc)]

# LSTMs and Average Pooling (sentence-level)
shared_sentence_lstm = LSTM(units=embedding_dim, return_sequences=True, activation='tanh')
shared_average_pooling = AveragePooling1D(pool_size=max_sentence_len)
sentence_modeling = [shared_sentence_lstm(sentence_inputs[i]) for i in range(max_sentences_per_doc)]
sentence_modeling = [shared_average_pooling(sentence_modeling[i]) for i in range(max_sentences_per_doc)]

doc_modeling = Concatenate(axis=1)(sentence_modeling)
doc_modeling = LSTM(units=embedding_dim, activation='tanh', return_sequences=True)(doc_modeling)

The convolutional layer with multiple filter sizes has to be applied "manually", therefore we run the convolutions in a loop and concatenate the resulting layers after we flattened them (you could also apply GlobalMaxPooling, then you would not have to flatten the layer, but with this you would lose many information of your feature vectors).
conv_blocks = []
for sz in kernel_sizes:
    conv = Convolution1D(filters=filters,
                         kernel_size=sz,
                         padding="valid",
                         activation="relu",
                         strides=1)(doc_modeling)
    conv = MaxPooling1D(pool_size=2)(conv)
    conv = Flatten()(conv)
    conv_blocks.append(conv)
doc_modeling = Concatenate()(conv_blocks) if len(conv_blocks) > 1 else conv_blocks[0]

After this we can normally apply an activation layer (e.g. softmax or sigmoid, with or without dropout) and compile the model

Question 2 and 3: We need either an embedding directly in the batch-data or the embedding layer.
If you implement your network to have the embedding already in the batch data, you have to generate it only once. It works with small datasets and small embedding sizes (small depends on your memory size etc.), but with large datasets (e.g. 2000 words per document and an embedding dimension of 300) you rapidly exceed like 30gb of memory. Therefore it definitely makes sense to use the embedding layer, since it stores each word vector only once. The downside with this embedding layer is, that it has to perform the word index to word vector replacement after each batch, this costs a little bit of runtime, but since it is only a lookup operation this is not very expensive. So be encouraged to use the embedding layer.
Question 4:
It would not. Firstly, because of some parameter issues (e.g. Conv1D accepts only one integer for the kernel_size), secondly, it implements another architecture, that makes no sense (cf. question 1, TimeDistributed layer).
Question 5: No :-), see question 4


Custom output names for keras model I have a model like this with multiple outputs and i want to change it's output names

class MyModel(Model):
    def __init__(self):
        super(MyModel, self).__init__()
        self.layer_one = Dense(1, name='output_name_one')
        self.layer_two = Dense(1, name='output_name_two')
    def call(self, inputs):
        output_name_one = self.layer_one(inputs)
        output_name_two = self.layer_two(inputs)
        return output_name_one, output_name_two

keras automatically set output names to output_1, output_2, ... how can i change the output names to my desired names?

I also have this issue, but didn't have the answer with a customized Model.
There is a workaround as follows:
 model = Model(inputs=inputs,
                  outputs={'ctr_output': ctr_pred, 'ctcvr_pred': ctcvr_pred, 'cvr_output': cvr_pred})


How to Connect Convolutional layer to Fully Connected layer in Pytorch while Implementing SRGAN I was implementing the SRGAN in PyTorch but while implementing the discriminator I was confused about how to add a fully connected layer of 1024 units after the final convolutional layer
My input data shape:(1,3,256,256)
After passing this data through the conv layers I get a data shape: torch.Size([1, 512, 16, 16])
Code:
class Discriminator(nn.Module):

  def __init__(self):
    super(Discriminator,self).__init__()
    self.sm = nn.Sigmoid()
    self.net = nn.Sequential(
        nn.Conv2d(3,64,3,padding=1),
        nn.BatchNorm2d(64),
        nn.LeakyReLU(0.2),

        nn.Conv2d(64,64,3,2,padding=1),
        nn.BatchNorm2d(64),
        nn.LeakyReLU(0.2),

        nn.Conv2d(64,128,3,padding=1),
        nn.BatchNorm2d(128),
        nn.LeakyReLU(0.2),

        nn.Conv2d(128,128,3,2,padding=1),
        nn.BatchNorm2d(128),
        nn.LeakyReLU(0.2),

        nn.Conv2d(128,256,3,padding=1),
        nn.BatchNorm2d(256),
        nn.LeakyReLU(0.2),

        nn.Conv2d(256,256,3,2,padding=1),
        nn.BatchNorm2d(256),
        nn.LeakyReLU(0.2),

        nn.Conv2d(256,512,3,padding=1),
        nn.BatchNorm2d(512),
        nn.LeakyReLU(0.2),

        nn.Conv2d(512,512,3,2,padding=1),
        nn.BatchNorm2d(512),
        nn.LeakyReLU(0.2),

        nn.Linear(<ADD AN INPUT SHAPE HERE>,1024),
        nn.LeakyReLU(0.2),
        nn.Linear(1024,1)
    )

  def forward(self,x):
    x = self.sm(self.net(x))
    x = (x)
    return x

You simply reshape the tensor to (batch_size, n_nodes) using tensor.view(). In your specific case this would be x.view(x.size()[0], -1).


How can I predict traffic based on previous time series data? If I have a retail store and have a way to measure how many people enter my store every minute, and timestamp that data, how can I predict future foot traffic?

I have looked into machine learning algorithms, but I'm not sure which one to use. In my test data, a year over year trend is more accurate compared to other things I've tried, like KNN (with what I think are sensible parameters and distance function).

It almost seems like this could be similar to financial modeling, where you deal with time series data. Any ideas?

The problem with models like KNN is that they do not take into account seasonality (time-dependent variations in trend).  To take those into account, you should use Time Series analysis.

For count data, such as yours, you can use generalized linear auto-regressive moving average models (GLARMA).  Fortunately, there is an R package that implements them (glarma).

The vignette is a good resource for the theory behind the tool.


Calculating correlation for categorical variables I am struggling to find out a suitable way to calculate correlation coefficient for categorical variables. Pearson's coefficient is not supported for categorical features. I want to find out features with most highest influence on the target variable. My objectives are:

Correlation between categorical and categorical variables. e.g. For a binary target (like Titanic dataset), I want to find out what is the influence of a category on the target (like, influence of gender on survival (0/1))
Capture some non linear dependencies. e.g. For supermarket sales data, the sales is usually higher during weekends as people might visit such store more during holidays. So we expect to see spikes at an interval of roughly 7 days. Is there any way to capture this non-linearity/seasonality by a correlation coefficient?

According to The Search for Categorical Correlation post on TowardsDataScience, one can use a variation of correlation called Cramer's association.

Going categorical
What we need is something that will look like correlation, but will
work with categorical values — or more formally, we’re looking for a
measure of association between two categorical features. Introducing:
Cramér’s V. It is based on a nominal variation of Pearson’s Chi-Square
Test, and comes built-in with some great benefits:

Similarly to correlation, the output is in the range of [0,1], where 0 means no association and 1 is full association. (Unlike
correlation, there are no negative values, as there’s no such thing as
a negative association. Either there is, or there isn’t)

Like correlation, Cramer’s V is symmetrical — it is insensitive to swapping x and y

def cramers_v(x, y):
    confusion_matrix = pd.crosstab(x,y)
    chi2 = ss.chi2_contingency(confusion_matrix)[0]
    n = confusion_matrix.sum().sum()
    phi2 = chi2/n
    r,k = confusion_matrix.shape
    phi2corr = max(0, phi2-((k-1)*(r-1))/(n-1))
    rcorr = r-((r-1)**2)/(n-1)
    kcorr = k-((k-1)**2)/(n-1)
    return np.sqrt(phi2corr/min((kcorr-1),(rcorr-1)))


How does a random forest algorithm deal with a few irrelevant input variables I have a list of variables from which I would like to train a Random Forest Algorithm. I suspect that a few of my input variables, which have noisy distributions, won't be able to predict much. Can I use them anyway, knowing the algorithm will eliminate them in the process, or should I beware that these variables may biais my model?

Random Forest tends to be not too sensitive to features with low predictive power. The reason is that RF looks for a "best split" given a subset of features (columns) and observations (rows). So "weak" features will likely be ignored in most cases.
However, removing the $x$ percent weakest features may increase the model's performance. In case you use sklearn, there are convenience functions to do this, e.g. SelectFromModel(). See the docs for more details.
>>> from sklearn.ensemble import ExtraTreesClassifier
>>> from sklearn.datasets import load_iris
>>> from sklearn.feature_selection import SelectFromModel
>>> X, y = load_iris(return_X_y=True)
>>> X.shape
(150, 4)
>>> clf = ExtraTreesClassifier(n_estimators=50)
>>> clf = clf.fit(X, y)
>>> clf.feature_importances_
array([ 0.04...,  0.05...,  0.4...,  0.4...])
>>> model = SelectFromModel(clf, prefit=True)
>>> X_new = model.transform(X)
>>> X_new.shape
(150, 2)


How to place XGBoost in a full stack for ML? Is XGBoost complete by itself for prod-strength machine learning?  If not, with which other tools or libs is it typically combined, and how?

(I recently read a description of a stack that included ca 5 pieces, including XGBoost and Keras.)

Yes, it is a full-strength Machine Learning paradigm.

XGBoost is basically Extreme Gradient Boosting.

It only takes in numeric matrix data. So, you might want to convert your data such that it is compatible with XGBoost.

The wide range of parameters of the xgboost paradigm is what makes it so diverse. Boosting can be done on trees and linear models, and then more parameters can be defined depending on the model you have selected.

So, yes it is a complete paradigm in itself. But, when you want more than the limitations of xgboost like linear and tree models, then you can use the concept of ensembling.

In the case of ensembles, the tools/libraries which can be used depends on the data scientist who is conducting the experiment. It can be Keras or Theano or TensorFlow, or anything which he/she is comfortable with. (opinion-based)


Cosine Distance > 1 in scipy I am working on a recommendation engine, and I have chosen to use SciPy's cosine distance as a way of comparing items.

I have two vectors:

a = [2.7654870801855078, 0.35995355443076027, 0.016221679989074141, -0.012664358453398751, 0.0036888812311235068]

and

b = [-6.2588482809118942, -0.88952297609194686, 0.017336984676103874, -0.0054928004763216964, 0.011122959185936367]

Running the following code will produce an output of ~1.999:

from scipy.spatial import distance
print(distance.cosine(a,b))

Is there something wrong with my input values? Anyone know why I am getting a result of >1?

The cosine distance formula is:

And the formula used by the cosine function of the spatial class of scipy is:

So, the actual cosine similarity metric is: -0.9998.

So, it signifies complete dissimilarity.


scipy.spatial.distance.mahalanobis return null values for some vectors in python I am using scipy.spatial.distance.mahalanobis to calculate distance between two vectors but i'm getting null values for some vector I don't know why? null value is possible?

It is important to include your test cases when you encounter a potential bug.

Without further information, looking at the formula of Mahanologis distance.
$$\sqrt{(u-v)^TV^{-1}(u-v)}$$
It is possible to obtain an error if $V$ is not positive definite as the input for the square root might be negative.


How to model a Machine learning problem considering links between features Context:

To predict employee turnover ( will an employee leave? ), I have used one of the classification algorithms (LDA) to train my dataset, and then make predictions.

The dataset is quite small (500 lines), some 20 features, the following are some examples:

Age
Years_Spent: Years for an employee has spent in the company.
Department (IT, Commercial, Management...)
Salary
Salary_Increase
Sale_Bonus
...

However, HR Experience tells us that:

For employees whose Years_Spent < 1.5, Salary_Increase is a feature that does not have any impact on turnover (Because of Salary_Increase > 0 only when Years_Spent >1.5).
Sale_Bonus will not have any impact on those who are not commercials. (Because IT guys will never recept sale bonus)

Here comes the problem:
If I set Salary_Increase = 0 for employees whose Years_Spent <1.5 and Sale_Bonus = 0 for those who are not commercials, the classification algorithm will take 0 as a very small value, so a possible conclusion could be drawn by an algorithm: "employeeA will leave because he never receives sale_bonus",  (However, in reality, employeeA is from IT department, employeeA receive never sale_bonus and employeeA will not leave because of that), as we see, the constructed model is not correct.

My question is: How to handle this kind of problem so that HR experience can be understood by classification algorithms?

Thank you for your patient reading and kindly welcome all sorts of discussions!

Welcome to the site!

What you are describing above is known as an interaction.

You should consider the algorithm you wish to use and whether it allows for interactions between predictors. Some techniques - like generalised linear models - will require interactions to be stated explicitly, while tree-based algorithms will capture interactions automatically.


Paper proving neural networks with more data are better in NLP than traditional methods I'm looking for a paper that does some comparisons between neural networks (deep learning) and traditional methods in order to prove that DL usually performs better with enough data.
I know this is "known" fact but I'm struggling to find a good paper doing some research in this area.

Thanks!

Here's a blog post providing a view to the history of the process with several popular references in it. In particular, consider the Natural Language Processing section and its conclusion:

  On NLP tasks that have a long enough history to graph, there seems to be no clear indication that deep learning performs above trendline.

Also, worth noting from the general conclusion of the article:

  Deep learning provides discontinuous jumps relative to previous machine learning or AI performance trendlines in image recognition and speech recognition; it doesn’t in strategy games or natural language processing, and machine translation and arcade games are ambiguous (machine translation because metrics differ; arcade games because there is no pre-deep-learning comparison.)


What are the advantages of contrastive divergence vs the gradient of the quadratic difference between the original data and the reconstructed data? In this example I have a RBM with a visible layer and a hidden layer. The original data is "data", the values of the hidden neurons is "hid", the values of the visible neurons calculated from "hid" is "vis", and then the value for the hidden neurons calculated from "vis"is "hid2".

I calculate the positive_association and the negative_association by:

pos_associations = T.dot(data.T, hid)
neg_associations = T.dot(vis.T, hid2)

where T.dot represents the usual matrix calculation. I then update the weights using contrastive divergence (CD)

w = w + (training_rate)*(pos_associations - neg_associations)

like described in this post

I do get a good reconstruction (vis looks like data) and I can see features representing the weights. However, it takes a while and many epochs.

If instead of using CD I simply calculate the quadratic differnce between data and vis

cost = T.sum((data - vis)**2).mean()

and I update the weights using the gradient of cost, I get a faster approximation of the data. So my question is, what is the advantage of using CD?

It is natural that RBM requires some time to converge, but if you see that the reconstruction is decent, then your algorithm works well.

I don't understand the idea of optimizing the model by your objective function:
cost = T.sum((data - vis)**2).mean().
I think you just cannot train RBM (or any other stochastic model) that way because vis is a nonlinear function of the data. If you omit sampling of the hidden vector you might use this formula and end up having model similar to the feed forward network that learns an identity transformation. It fact it can be used as a feature extractor.

Also notice that your second network no longer will be a proper generative model, because its learning rule doesn't follow the gradient of RBM's cost function. Hence it won't properly model either an energy function or a joint probability P(v, h) of your visible and hidden vector.

You may also be interested in reading this:

  Although it is convenient, the reconstruction error is actually a very poor measure of the progress of learning.  It is not the function that CD learning is approximately optimizing, [...] and it systematically confounds two different quantities that are changing during the learning.  The
  frst  is  the  difference  between  the  empirical  distribution  of  the  training  data  and  the  equilibrium
  distribution of the RBM. The second is the mixing rate of the alternating Gibbs Markov chain.  If
  the mixing rate is very low, the reconstruction error will be very small even when the distributions of
  the data and the model are very different.  As the weights increase the mixing rate falls, so decreases
  in reconstruction error do not necessarily mean that the model is improving and, conversely, small
  increases do not necessarily mean the model is getting worse.

So what is the cost function that RBM tries to optimize? good description can be found here
$$-\frac{\partial log(p(x))}{\partial \theta }=\frac{\partial F(x)}{\partial \theta}-\sum_{\tilde{x}}p(\tilde{x})\frac{\partial F(\tilde{x})}{\partial \theta}$$
On the left hand side we have minus log probability of the data which we must minimize in order to increase the probability of the data. On the right hand side we have two terms. The function $F(x)$ coresponds to the free energy of the network given the data $x$ and current weights $\theta$ so minimization of this function increases the probability of our data. On the other hand we must take into account the normalizing factor $Z$- its role is to give correct probability distribution of the input vectors and this sum in the equation corresponds to the normalizing factor. In other words we must maximize the free energy (decrease probability) over all set of inputs proportionally to its current probability $p(\tilde{x})$. If we omit to maximize this sum our model won't converge to the optimal solution. Contrastive Divergence is only crude approximation of this objective funtion but it works very well in practice. I recommend that you watch this film by Geoffrey Hinton


Structure of a multilayered LSTM neural network? I implemented a LSTM neural network model in Keras. However, how the codes worked under the hood was not quite clear. I want to know if it worked the way I guessed how it worked?

For example:
Say there's a 2-layer LSTM network with 10 units in each layer.
The inputs are a sequence data $X_1, X_2, X_3, X_4, X_5$.

So when the inputs are entered into the network, $X_1$ will be thrown into the network first and be connected to every unit in the first layer. And it will generate 10 hidden states/10 memory cell values/10 outputs. Then the 10 hidden states, 10 memory cell values and $X_2$ will be connected to the 10 units again, and generate another 10 hidden states/10 memory cell values/10 outputs and so on?

After all 5 $X_i$'s are entered into the network, the 10 outputs from $X_5$ from the first layer are then used as the inputs for the second layer. The other outputs from $X_1$ to $X_4$ are not used. And the the 10 outputs will be entered into the second layer one by one again. So the first from the 10 will be connected to every unit in the second layer and generate 10 hidden states/10 memory cell values/10 outputs. The 10 memory cell values/10 hidden states and then the second value from the 10 will be connected and so forth?

After all these are done, only the final 10 outputs from the layer 2 will be used? Is this how the LSTM network works? Thanks!

For layer 1, After all 5 Xi's are entered into the network, the 10 outputs from X5 from the first layer are then used as the inputs for the second layer. The other outputs from X1 to X4 are not used.

No, the outputs from all time steps 1,2,...,5 are used as the input of the second layer.

For layer 2, After all these are done, only the final 10 outputs from the layer 2 will be used?

It depends on you. Some people choose to use only the final size=10 output as an encoding of all the input. Some people choose to average or concat all the outputs of second layers. When using attention, all the outputs are used.

Here is a illustration of a 3-layer LSTM. Notice that all the outputs from the lower layer, not only the final output, are fed into the higher layer.

.


How to detect influence on behavior From a behavioral study data was extracted. The study was about how people change their eating behavior, following visual cues. There were to groups of people: One was shown visual cues and then it was recorded what they chose to eat and the other group was just shown random stuff or nothing and then it was recorded was they chose to eat. This experiment was repeated with the same people a large number of occasions, at different times of day.
I have a dataset that precisely recordes personal information about each individual as well as their eating behaviour during each time the experiment was carried out.

The goal is to predict whether the visual cue will make someone eat the stuff presented in the cue or not. The problem is that it is unclear to me how to separate the influence of the treatment from the behavior that they might have shown anyway. I.e., suppose a person sees an image of a cake and then, when he is presented with a variety of different foods, eats cake. How can we know that it was actually the image that influenced him and that he did not wanted to eat cake anyway, so the image change actually nothing?

Thus I can't directly treat it as a binary classification and define a categorical feature, where I assign "1" if someone ate what was in the picture and "0" if he didn't, since by that way I may also identify those who wanted to eat what was in the cue before it was shown to them. How can I solve this problem?

I think you can approach it as a logistic regression problem, where you will have a on/off (1/0) feature called "exposure_to_image". And your goal will be to detect if that coefficient is statistically significantly different than 0. If yes, then the probability of eating the item was influenced by the exposure.

As for the assumption that they would have eaten it anyway:
That's the whole point of having the control group. So your dataset will have people where exposure_to_image=1, and people for who exposure_to_image=0. All of them will have some baseline probability of wanting to eat the cake. But that coefficient tells you by how much that baseline is affected.


100 items 100 baskets divisor association analysis problem I have the following exercise question:

Suppose there are 100 items, numbered 1 to 100, and also 100 baskets, also numbered 1 to 100. Item i is in basket b if and only if i divides b with no remainder. Thus, item 1 is in all the baskets, item 2 is in all fifty of the even-numbered baskets, etc. for example Basket 12 consists of items {1, 2, 3, 4, 6, 12}

Given this, I'm trying to solve the following 3 questions:

(a) If the support threshold is 5, which items are frequent?
(b) what is the confidence of the following association rules?
(1) {5, 7} → 2.
(2) {2, 3, 4}→ 5.

which way I should be approaching these types of questions?

a) The items that are frequent are $\{1,2,3,4,5,\dots,20\}$ because these all appear in at least 5 baskets.

b)

(1) $\{5, 7\} → 2$. Then $Confidence=\frac{1}{2}$

hence 5 and 7 will appear together in basket no. 35 and 70 and 2 will appear along with 7 and 5 in basket no. 70 so:

$$ Confidence = \frac{support(\{5,7\}\cup\{2\})}{support(\{5,7\})}= \frac{1}{2} $$

(2) $\{2, 3, 4\}→ 5$. Then $Confidence=\frac{1}{8} $

$\{2,3,4\}$ appear in baskets having basket number multiple of 12 ($LCM\{2,3,4\}$) i.e in basket no $\{12,24,...,60,.....,96\}$ and $\{2,3,4,5\}$ appear together only in basket no 60.

$→ Confidence=\frac{1}{8}$


Maximum likelihood estimation vs calculating distribution parameters "manually" I'm sorry for asking probably elementary question, but I cannot understand how estimating probability distribution parameters using maximum likelihood estimation method differs from calculating these parameters from observed data manually.
For MLE we need to know the type of probability distribution anyway so why don't we just use the known formulas for calculating the corresponding parameters from observed data?
I believe that MLE is somehow more general method but I cannot see what is the real advantage of MLE compared to getting these parameters "manually".

Thanks for explanation.

Tomas

I think you are asking about how estimates for these parameters from your data sample (obtained by calculating moments) would be different from maximum likelihood estimation. If so, check out:
https://stats.stackexchange.com/questions/252936/what-is-the-method-of-moments-and-how-is-it-different-from-mle
http://gradquant.ucr.edu/wp-content/uploads/2013/11/Maximum-Likelihood-and-Method-of-Moments-Estimation.pdf


May I use the same data in several time series intervals? I am playing with RNNs / LTSMs for a classification task in predicting financial data.

I have a time-series going many years back, and are planning to divide it into a number of shorter time-intervals that will be used to obtain a classification.

For example divide the data into random length intervals from 30 minutes to 24 hours, with a binary response 'higher' and 'lower' that answers if the value goes higher or lower for the next hours after the interval.

My question is if I can "reuse" the data, as in for example for a certain 24 hour period create the following intervals: 24 hours, 12 hours, 6 hours, 3 hours and 1 hour and train a model on all of these examples. If I do so, obviously the same data will be reused with different responses.

Or is it better to create a number of models; for example a separate model for 24 hour intervals than the 3 hour intervals?

I dont see a problem in „reusing“ observations. Its just new/different info coming in. The question is more if the model will train/perform well with a short lookback or timelag. Why dont use a long(er) sequence of data to predict (eg) the movement in 3h steps. I wonder why you dont use all the data you have to train the LSTM and update the data stepwise. More data usually is better!


Derivation of the cross-entropy equation in Michael Nielsen's book I am reading the book http://neuralnetworksanddeeplearning.com/chap3.html by Michael Nielsen.
So this is a question mostly for the people familiar with the book and understanding the material.

In the equations (71-75) we are trying to find a cost-function $C$ satisfying:

$$\frac{\partial{C}}{\partial{\omega_j}}=x_j (a-y),$$
$$\frac{\partial{C}}{\partial{b}}=(a-y),$$

where $w_j$ and $b$ are weights and bias of a neuron, $a$ is the output of the sigmoid function

$$a = \sigma(\sum{\omega_{j} x_j + b)}=\sigma(z).$$

We apply the chain rule (equation 73):

$$\frac{\partial{C}}{\partial{b}}=\frac{\partial{C}}{\partial{a}}\frac{\partial{a}}{\partial{b}}=\frac{\partial{C}}{\partial{a}}\sigma\prime(z).$$

And in the next line the author writes

Using $\sigma\prime(z)=\sigma(z)(1-\sigma(z))=a(1-a)$ the last equation becomes...

Where does this expression come from? $\sigma(z)(1-\sigma(z))$

The author has taken activation function as sigmoid in this case. The derivative of this function can be re-written like

$$\sigma(z) = \frac{1}{1+e^{-x}}$$

whose derivative $$\sigma'(z) = \frac{e^{-x}}{(1+e^{-x})^2}$$ which can rewritten as $$\frac{1+e^{-x}-1}{(1+e^{-x})^2} \rightarrow \sigma(z) * (1 - \sigma(z))$$. This is subtly mentioned in eq.3 of chap1.


Conditional Entropy and Mutual Information - Clustering evaluation First of all, I am doing clustering and I have the true labels for my data.
For evaluation, I am using the weighted average of the entropy values for each predicted cluster. I also came across with Mutual Information as a similar approach while going over the alternatives. On my data, they seem to give similar results.

However there is one issue that puzzles me.

Given the predicted cluster set $U$ and true clusters $V$, mutual information was defined as:
$$ I(U,V) = H(U) - H(U|V) $$
or,
$$ I(U,V) = H(V) - H(V|U)  $$
If my math is correct, the average entropy that I'm using corresponds to conditional entropy term $H(V|U)$ and trying to minimize this aligns with maximizing the mutual information.

What I cannot see is how weigthed average entropy would differ from mutual information and why we would need the entropy terms $H(U)$ or $H(V)$. It feels like minimising one of the conditional entropies should suffice.

To put it another way, as far as I understood from the equations, having high entropy for true or predicted clusters in itself also results in higher mutual information. Does this mean that mutual information favors equally-sized clusters?

Thanks in advance.

Mutual information does favor many small clusters. Nectar these tend to be "pure". That is why variations wish as normalized mutual information and adjusted mutual information (AMI) are used instead.

  Vinh, N. X.; Epps, J.; Bailey, J. (2009). "Information theoretic measures for clusterings comparison". Proceedings of the 26th Annual International Conference on Machine Learning - ICML '09. p. 1. doi:10.1145/1553374.1553511.


Applying the same changes to the test set I'm busy working through Aurélien Géron's book. (Hands-On Machine Learning with Scikit-Learn, Keras, and Tensorflow)
The idea is to split the data into train and test set as early as possible in order to avoid data snooping bias. Afterwards changes are made to the data.
My question is that since changes were made to the training set, I assume the same changes(dropping columns, filling NA rows, converting categorical to numerical, etc) should be made to the test set before training and evaluating? If that is the case, what is the correct way to perform this? Write everything as a function and run it on both, which seems a bit counter intuitive to working with notebooks? Is there a built-in function that I'm not aware of?

Most transformations are available as built in classes in scikit learn. You can assemble the classes into a scikit pipeline and have your train data pass through it as part of the pipeline's fit operation. When you are ready to evaluate your test data you can simply run the pipeline's predict operation. This ensures that the test data passes through the exact same transformation workflow as your train data.


About the time differences in the Bellman equation I am trying to grasp fundamental mathematics behind the Reinforcement Learning and so far I have unterstood how the Value Iteration and Policy algorithms do converge (contractions, etc.)

I have still some problems about understanding the Bellman Equality. The Value function for a state $s$ under a policy $\pi$ is the expected discounted cumulative reward:

$$ V^\pi(s_0=s) = \mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^t R(s_t,\pi(s_t)) |s_0=s\right]$$.

During the derivation of the Bellman Equations, when the expected cumulative rewards are calculated on an infinite horizon, meaning $t \to \infty$, shouldn't they be equal for different time steps; like $V^{\pi}(s_i=s) = V^{\pi}(s_{i+\Delta i}=s), \forall s$?
Since the summation is an infinite series, the discount factor $\gamma$ always starts with $0$ in the exponent and the state and reward distributions are stationary, as long as the the starting state is the same, the cumulative reward must be also the same:

$$ V^\pi(s_i=s) = \mathbb{E}\left[\sum_{t=i}^{\infty}\gamma^{t-i} R(s_t,\pi(s_t)) |s_i=s\right]= \mathbb{E}\left[R(s_i=s,\pi(s_i=s)) + \gamma R(s_{i+1},\pi(s_{i+1})) + \gamma^2 R(s_{i+2},\pi(s_{i+2})) + \dots\right]$$

$$ V^\pi(s_{i+\Delta i}=s) = \mathbb{E}\left[\sum_{t=i+\Delta i}^{\infty}\gamma^{t-i-\Delta} R(s_t,\pi(s_t)) |s_{i+\Delta i}=s\right]=
\mathbb{E}\left[R(s_{i+\Delta i}=s,\pi(s_{i+\Delta i}=s)) + \gamma R(s_{{i+\Delta i}+1},\pi(s_{{i+\Delta i}+1})) + \gamma^2 R(s_{{i+\Delta i}+2},\pi(s_{{i+\Delta i}+2})) + \dots\right]$$

Is this true for the Bellman Equation?

During the derivation of the Bellman Equations, when the expected cumulative rewards are calculated on an infinite horizon, meaning $t \to \infty$, shouldn't they be equal for different time steps; like $V^{\pi}(s_i=s) = V^{\pi}(s_{i+\Delta i}=s), \forall s$?

Yes, provided the state $s$ has the Markov property, which means it contains all relevant data about immediate state progression and reward, then this is true. I would read your equations as over-specified in fact, and often see the equations written differently:

$$v_{\pi}(s) = \mathbb{E}[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t = s, A_t \sim \pi]$$

although there are several roughly equivalent ways to describe value functions in RL (for instance, and trivially, you could write $\mathbb{E}_{\pi}$ for the expectation which already implies the condition $A_t \sim \pi$). The one in this answer makes it clear that the value of the state $s$ is independent of the time step on which it is measured.

It is worth noting that although the expectation is always the same and independent of $t$, that actual returns seen can be different, and if you know from some other context that you are seeing the first, or last visit to a state - or from the time step whether this is more or less likely given $\pi$ - then that may change the expectation when loops are possible.


How to learn 3D orientations reliably? I am working on neural network models for 3D skeletal character animation, where I learn joint positions and orientations. The problem comes with the orientations. There are several ways I can choose to represent a 3D rotation, but all of them have some form of discontinuity that is making my model unable to produce stable correct rotations in all cases. The alternatives I have considered are:

Quaternions. Quaternions (or unit quaternions) are very commonly used for 3D orientation, and are generally fine except a unit quaternion and its opposite represent the same orientation. It is not hard to come up with a loss formula the accounts for this (1 - abs(sum(componentwise_product(q1, q2))) works fine), but the problem is that in some cases the network somehow learns to produce the same quaternions with both signs, and kind of randomly "flips" from one sign to the other at times, with some frames in between, resulting in a sort of "flickering". One possibility would be to take only "half" of the quaternions, e.g. by fixing one component to be positive (then learn by squared differences for example). It doesn't always work, though. For example, suppose I fix the first component to be positive and I have a quaternion like [0.018, 0.743, 0.557, 0.371], and another one like [-0.017,  0.870,  0.348,  0.348]; these are rather similar quaternions, but if I make the first component positive then the second one will be [0.017,  -0.870,  -0.348,  -0.348], which is numerically very different, and the result is that sometimes the network cannot learn the value properly and comes up with some numbers in between (which are incorrect).
Rotation vectors. A rotation vector has the direction of the axis of rotation and the size of the angle of the rotation. They are used for example in the original paper of phase-functioned neural networks, but they still have bad cases. Suppose vector sizes go from 0 to π (greater angles would be taken along the opposite axis). If you have a rotation that is just around the value of π, the rotation vector will be "swinging" from one direction to the opposite one, again making it hard for the network to learn it.
Euler angles. These are just three rotation angles that are applied along specific axes (X, Y or Z) in specific order. They are generally discouraged because they are not very stable and suffer from the infamous gimbal lock, but geometry aside, they still have the same problems. If my angles go from -180º to 180º (or -π to π in radians), values in the "frontier" will always cause instability.
Extensive rotation encodings. One stable way of encoding rotations is giving both sine and cosine values. So I could for example have both the sine and cosine of each Euler angle, or just the nine values of a 3D rotation matrix (which are not sines and cosines but are in a way derived from these, and also a stable representation). However this obviously increases the number of values to learn significantly, and I would be trying to learn independent values that actually have a relationship among them.

I haven't found relevant literature addressing this problem specifically, although maybe I am not searching with the right terms. Has anyone faced this issue before? Or has some idea or alternative that I could consider?

I found an intermediate solution between minimal representation and full rotation matrix, which is using two unit vectors indicating two orthogonal directions (e.g. forward and right) with respect to the orientation. These can be learned simply through squared differences and are simple to use (in my case the animation runtime already had functions to compute an orientation given two directions, taking into account that they may not be perfectly orthogonal). This representation takes 6 values per orientation which is not as good as 3 or 4 but not too terrible either, and in my experiments it seems to work just as fine, only without the unstable cases.


Converting string_id to number_id I have column with movie ids like this:

tt0984332
tt0984332
tt0847742
ttnanana1

I would like to convert them into numbers that can be inserted into neural network as features, like this:

0
0
1
2

How can I do that?

What you are looking for is performed exactly by sklearn.LabelEncoder, an example of which can be found here.

In your case:

from sklearn.preprocessing import LabelEncoder

encoder = LabelEncoder()
encoder.fit(["tt0984332", "tt0984332", "tt0847742", "ttnanana1"])

# Show the unique classes
print(encoder.classes_)
# out: array(['tt0847742', 'tt0984332', 'ttnanana1'], dtype='<U9')

# Convert labels to integers
encoder.transform(["tt0984332", "tt0984332", "tt0847742", "ttnanana1"])
# out: array([1, 1, 0, 2])


Nested features with one to many relationships I have a dataset that has (among others) a categorical variable with many levels and further attributes associated with each level.

For example, consider predicting machine failure based on its last repair report.

A machine has many (many) different parts.

Every part could have been damaged or not.

If damaged, it could have been inspected, repaired or replaced etc.

These operations have associated costs and the part itself has a money value and other attributes (like the time it took to repair it...).

If I want to make predictions on the level of a machine, I need to aggregate all this information. I could one-hot encode all the parts indicating whether they have been damaged or not. But this still leaves me with the other attributes on the lower part level, like price and performed operation. I could probably further expand them into individual columns by considering all the combinations like part_X-repaired-..-price, part_X-replaced-..-price, ... part_Z-replaced-..-price but this seems to get out of hand.

Is there a better way to handle this type of data? I was thinking maybe some clustering technique but when I try to set it up I run into the same problem.

It is somewhat a reversed hierarchical model structure (in hierarchical (linear) models the outcome variable is on the lowest level if I'm not mistaking).

This is called "bilevel optimization" where one problem is embedded (nested) within another. If there are more than one objectives at one or both levels, it is then called "multi-objective bilevel optimization". There is a rich literature on this problem, including the book Multilevel Optimization: Algorithms and Applications.

Finding solutions to minimize the objectives often does not require machine learning, just linear programming.


Find outliers in time-series data I want to find outliers in power consumption in real-time, at hourly rate, i.e., at the end of the hour, I should say whether power consumption in current hour was outlier/anomalous or not.

Approach:  Till now, I am done with following steps

Say I want to find whether power usage between 9 AM to 10 AM was anomalous? For this, I first find the usage of past n days during the same time interval, then I find the mean/median of all the previous usages
Now, I have usage of the current day and the mean/median usage of previous n days. Which statistical measure should I use to declare whether current day usage was anomalous or not?

Using above approach, for 24 hours of a specific (test) day and using past 10 days consumption, I have obtained results as:

Figure interpretation: Black line represents usage of current hour of current day; red and blue lines represent mean and median of past 10 days for same time interval

From the visual inspection, I can say that the usage between 07:10  - 08:00  and between 22:10 - 23:00 is anomalous as there is big difference between actual and previous mean/median usage. I don't know which statistical measure should I use to point out such anomalous instances automatically, using the discussed approach.

I have been using one heuristic to detect outlier values that is very simple. Start by calculating an interval centered on the mean and with three standard deviations in radius. Next, calculate a second interval using Tukey's box and whiskers measures, placing the interval limits at the whiskers' limits. Finally, calculate the union of both intervals and use this new interval to detect your outliers: any observation outside this interval is a potential outlier. Please note that you may have to tweak both intervals to calibrate for your desired precision. Also, you may have to take into account possible trends and seasonalities in these intervals, depending on how the historic data behaves.


Will oversampling help with generalization (small imbalanced dataset)? I have an imbalanced dataset (2:1 ratio) with about 60 patients and 80 features.
I performed Recursive Feature Elimination (RFE) and stratified cross validation to reduce the features to 15 and I get an AUC of 0.9 with Logistic regression and/or SVM. I don't fully trust the AUC I got because I think it will not generalize correctly because of such a small positive class.
So, I was thinking on oversampling (K-means + PCA) the minority class and re-run the RFE approach, would this help?
Thanks.
My question is more or less the same as this one: Why will the accuracy of a highly unbalanced dataset reduce after oversampling?
but I do use AUC.

The bigger issue might be the small n. With 60 samples and 2:1 ratio, you only have 20 samples in the minority class. Generalization, no matter what machine learning technique is used, will be limited with just 20 samples.


Detecting redundancy with Pearson correlation in continuous features I have a set of variables that I want to use for a regression or a classification problem.
Having computed the correlation matrix of these variables, I discovered that some of them has an inter-variables Pearson correlation
values as high as 1.

Does this mean that these variables hold redundant information for the learner?
Is it safe to remove one of them without risking information loss? if yes, how to chose the one to remove?

If the correlation between two features $x_1$ and $x_2$ is 1 that means that you can write $x_1 = c\cdot x_2 + a$. The only knew knowledge there is are those two constants, the individual values can be retrieved knowing this. I highly doubt there is anything a machine learning algorithm can learn from this and it is a fact that for some having this kind of correlation between features can hurt your performance quite a bit, so I would test it a bit but I would say it's very likely you can remove one of the two, and which one is not going to matter.


Data science related funny quotes It has been customary for the users of different communities to quote funny things about their fields. It may be fun to share your funny things about Machine Learning, Deep Learning, Data Science and the things that you face every day!

Q: How many machine learning specialists does it take to change a light bulb?

A: Just one, but they require a million light bulbs to train properly.

Q: How many machine learning specialists does it take to change a fluorescent light bulb?

A: That wasn't in the training data!


How is bias added in a convolutional layer In a typical neural network , bias is usally added like this

 v = activation ( w1*x1+ ... + Wb*b)

However, I am not really sure how it is done in convolutional layer. My one thought is that it is added with each convoltional operation for a neuron. Is that correct?

There are two ways bias is usually added to a convolutional layer:

Tied bias: where you share one bias per kernel
Untied bias: where you use use one bias per kernel and output

Check this blog post comparing the two.

Other than that, bias isn't really necessary in CNNs. There are many popular implementations which ignore it altogether (e.g. ResNet-152).


Comparing different classification results with different trainig and test data I have different  samples with different sizes. The instances of each sample have different features in comparison to the instances from the other samples.  For each sample i train my model and tested it with 30% of the data as an unseen data. After obtaining the performance result, i would like  to choose the best one that can predict properly ( in other words, the used features in that sample are more predictive than the other). Now, the problem of comparing their performance is that the size of test sets are not the same. I don't know if the calculation of the 95% confidance intervals can lead to a conclusion by chossing the maximum the min ranges of the confidance intervals.

Thanks in advance.

Your question is unclear. If you're saying that you have datasets with different attributes then this answer is not relevant.

If you're saying that you've grouped your dataset by some similarity measure into different groups and then split every group into test and train subgroups then yes, you may have overfitting.

However, if you made a TEST dataset from the COMPLETE dataset which has mixed elements from the original groups and proportional to them, and then tested your model against this test dataset then your results should represent your complete dataset properly.


Regression vs Classification to rank by profit Say that I have a population of 10k customers, for which I expect 100 responders in my next campaign and have budget to send a letter to 5k.

I have past data of how much those who respond spent, and I'm interested in getting the most $ out of my 5k letters (i.e. ideally I find the 100 responders, but if not, I want to find those who leave more money).

I have created a regression model and a classification model (1 if they spent money -i.e. reponded in the past - and 0 otherwise). After running bot models, I find that in a test sample of 10k, if I select the first 5k ranked by probability (for the classification model) and by the expected amount in the regression, the regression model performs better (not evaluating how many responders they found, but how much they found out of those selected).

Does it make sense based on the numbers of my context? I had expected the classification model to perform better, but can't figure out why the regression model is coming on top in my example.

Thank you very much!!

This actually isn't too surprising given the context in my opinion. Classifying an observation as "1" = spent money in past and "0" = did not spend money in past causes you to lose information that in your case is important; that is, the amount of money that each person gave.

Imagine these two scenarios, for example. In one scenario, you send your letter to ten people and all ten people send back 100 dollars each each. Using a classification scheme, these ten people are all "1"'s. In the second scenario, you send your letter to ten people again, but now only one of ten respond. However, this one person sends back a cheque of 10,000 dollars. From a pure profit point of view, clearly the 2nd option is superior.

Ultimately, both your classification and regression models answer two separate questions. Your classification model simply predicts how likely someone is to spend any amount of money, (by how you have phrased the question, anyway) whether that is 10 dollars or 10,000,000 dollars there exists no distinguishing between the two with a classification model (as you phrased it). The regression model predicts the exact amount that each user is expected to spend which if I am reading your question correctly, is what you care about.

If you want to make the problems closer in scope, perhaps try defining in your classification model as "1" if person spends over some profitable amount of money, and "0" if they don't. While I don't agree very much with doing this (you lose information this way by "bucketing" a continuous variable, and also, why make the problem harder?) you might get "closer" results.


Creating Strings corresponding to Location Co-ordinates What are some Python libraries which can convert a (X,Y) tuple to strings?

(1.23,4.56) yields strings “1_4”, “12_45”, “123_456”.

Your question has nothing to do with NLP or text-mining (as you claim in the attached tags), or data science in general. It's a pure programming question best suitable for StackOverflow.

Moreover, you don't really need any libraries to do, what you want to do. A simple function will do. NOTE: I am using map and reduce functions on purpose to include at least a little bit of data science-related stuff (WINK):

coord = (1.23, 4.56)

def get_coord(coord):
   coord_str = map(lambda x: str(x).replace('.', ''), coord)
   for level in range(1, len(coord_str[0])+1):
      yield reduce(lambda x, y: (x[:level]+'_'+y[:level]), coord_str)

print list(get_coord(coord))

Running this code will result in printing:

['1_4', '12_45', '123_456']

You're not discussing any corner cases in your question, so I assume there are none.


Training with a subset of data: relationship between subset size and training metric? I have a not-quite linear regression problem which I am investigating. The data set is fairly large, with ~6000 samples and ~2100 features.

By performing 5-fold cross-validation on different sized subsets of the data set, I have found a strong relationship between the fraction of the data set used and the value of both the R-squared metric and the RMSE metric.

EDIT for clarity: for a fraction of 0.01, I am taking 1/100 of the samples (~60) and then performing repeated 5-fold cross-validation as if this were the whole data set. The values shown are means.

It appears that RMSE varies linearly with the reciprocal of the fraction used (1/Frac), and the R-squared regresses very well against a second-order polynomial of 1/Frac.

The raw results are as follows:

Frac   1/Frac  Test R2      Test RMSE
0.01   100     -1.65628     1.85292
0.013  75      -0.71208     1.71476
0.02   50      -0.00786883  1.38874
0.05   20       0.535839    1.06872
0.10   10       0.626532    1.00598
0.20   5        0.702421    0.95058
0.30   3.33     0.745277    0.860082
0.50   2        0.772211    0.86548

Relationships:

My questions are:

Is this a well-known relationship?
Does it have a name (so I can research further)?
Is there a theoretical basis for it?

Do I understand correctly from your test RMSE, that the error is lower as you increase the size of your fraction used in training?

If you do 5-fold cross validation using a fraction of 0.01, that implies here that you are only a total fraction of only 0.05 of your data? So 0.04 for training and 0.01 for testing.

It should not come as a surprise then, that the performance improves with increased amounts of data used for training. Your RMSE values increase with fraction size - I would normally expect such a correlation (assuming your data is fairly coherent).

EDIT:

To answer your questions,

I would argue it is a well-known relationship, even in the simplest of regression problems. Have a read here (including comments on accepted answer) for more discussion on minimum required data points for valid models. One reason highlighted is that very few data points makes any estimate of variance less meaningful
I don't think there is a name given to this relationship - at least I don't know of one (and neither do people in the link above). I would just describe it as being inherent to problem that the dataset is trying to represent. For example, if we have a high-dimensional complex parameter space, and then only a handful of data-points sampled from that space, it is very unlikely that we can fully describe the original space. The more samples we have, the closer we can approximate it. This is extremely evident in data-intensive modelling techniques, such as image-based tasks (i.e. convolutional neural networks, with millions of parameters.) Other methodologies, such as Bayesian optimisation using Gaussian processes. These are computationally more intense, but perhaps offer higher data efficiency.
There might be some convergence theories, stating how approximations to ground truth reach a threshold given a certain amount of data (that kind of framework would make sense to me) - but alas, I have not seen any formal analysis in this direction. Regarding number of samples, in the meantime, I just follow the mantra: the more the merrier!


Basic encoder-decoder architecture I read a couple of posts (like this one) about the encoder-decoder architecture and their implementation. But I still don't understand a couple of things.

What is the difference between a basic CNN or RNN and encoder decoder ?
Are there some properties that the encoder and decoder need to satisfy ? As far as I understood the encoder encodes the input in another dimension and creates a context vector. Later this context vector is decoded by the decoder.

I know that their are different types of encoders and decoders, what is the most simple architecture to implement ?

Should I use predefined word embeddings like word2vect in the encoder(if i have text as an input) ?

Encoder and decoder are highly overloaded terms. As a generic definition, an encoder-decoder neural architecture has a part of the network called "encoder" that receives an input and generates a code (i.e. expresses the input in a different representation space) and another part called "decoder" that takes a given code and converts it to the output representation space. Normally, the dimensionality of the code is much less than that of the input/output representation spaces.

In principle, the architecture of the encoder and the decoder is arbitrary, i.e. they can be CNN's, RNN's, multilayer perceptrons or whatever other thing. This highly depends on the task. This should answer question (1).

There are two contexts where encoder-decoders network organization is usual:

image autoencoders: normally, the encoder is a CNN and the decoder is the analogous deconvolution.
sequence-to-sequence tasks: normally, both encoder and decoder are GRUs or LSTMs.

As you mention word embeddings and a blog post about text summarization, I guess your problem relates to NLP, so you may start with the Keras blog seq2seq tutorial. This should answer question (2).

The appropriateness of using pre-trained word embeddings depends on the task and the amount of training data you have. When you don't have a lot of data, using pre-trained embeddings is a form of data augmentation to get better results. For translation, it is not frequent to see pre-trained word embeddings. This should answer question (3).


Multiclass Classification on live sensor data I want to use an accelerometer to detect which way a train is heading.
You could do this by just setting thresholds manually to detect the direction of acceleration on the x-axis (which would be aligned to the trains driving axis) after a standstill.

I still want to solve this problem using machine learning, because setting the thresholds takes quite some time.
The sensor data has to be in a context, otherwise its useless. The driving direction can only be estimated if you know the n sensor readings before.

So now I could probably use a RNN to create this context.
My ideas was it to not use a RNN (since it adds complexity) and just put several measurments in one sample. It would look like this:
accelX,accelY,accelZ,accelX1,accelY2,AccelZ2....AccelXn,AccelYn,AccelZn, label

I also measure the Y- and Z-axis because they are important to tell if you are standing still or just driving very smoothly without much acceleration (you have peaks in the Z-axis because you drive over bumps and in curves you have peaks on the Y-axis)

These samples would be taken one after another. This would create the context the net needs to judge it the sample was either standstill,accelerating positive on the X-Axis, accelerating negative on the X-Axis or if the train is just driving normally.

After my net would be trained I could just measure, pack and feed these samples in my net and in the end get a classification. If now one sample is classified as standstill and the next following one as accelerating+, I'd now that we are now driving forward. If a driving then standstill then accelerating- follows I then know, that we are now driving backward.

My dataset will be quite unbalanced since the train is mostly driving and only accelerating once after every station. In addition, my dataset won't be too big since I can only record data when I'm in the train.

I would like to use keras for building my model since its easy to understand.

Now my questions:

Do you think my approach will work?
What model architecture do you suggest me?
Should I manually classify my samples or use unsupervised learning?
Should I also include other features (variance of the data ..)

Thanks for the help

I would suggest you use a simpler method first, just to have a baseline to compare from.

In your case you can adapt this jupyter notebook to fit your purpose. They train a k-NN model to compare time-series using similarity measures.

After you're comfortable with that, you can move on to using LSTMs for classification

LSTMs for Human Activity Recognition
Multilabel time series classification with LSTM

The last one also works if you are trying to predict multiple labels per sample.


Meltdown patch impact on data processing speeds The patch for the Meltdown vulnerability disables speculative execution, which will impact all processing activities. The degree of impact is highly dependent on the type of processing being done. Is there hard data or experience of how machine learning and data processing will be impacted in measurable terms?

It looks like the Tech Web Sites that reported Meltdown, may have overblown the negative impacts.

According to Google, in their cloud infrastructure, there seems to be little to worry about:

  There has been speculation that the deployment of KPTI causes significant performance slowdowns. Performance can vary, as the impact of the KPTI mitigations depends on the rate of system calls made by an application. On most of our workloads, including our cloud infrastructure, we see negligible impact on performance.

Techspot  ran some benchmarks on Windows 10, and they concluded...

  Desktop users have little to worry about in terms of performance loss, particularly gamers. We've yet to test older CPUs, but given the type of workloads we’re seeing impacted by the patch, I don’t think there’s going to be an issue with any desktop hardware, but we’ll certainly report back if there is.


What is the advantage of using log softmax instead of softmax i am wondering if there are any advantages of log softmax over softmax. And also, when i should use softmax or log-softmax. is there any specific reason for choosing one over another?

There are a number of advantages of using log softmax over softmax including practical reasons like improved numerical performance and gradient optimization. These advantages can be extremely important for implementation especially when training a model can be computationally challenging and expensive. At the heart of using log-softmax over softmax is the use of log probabilities over probabilities, which has nice information theoretic interpretations.

When used for classifiers the log-softmax has the effect of heavily penalizing the model when it fails to predict a correct class. Whether or not that penalization works well for solving your problem is open to your testing, so both log-softmax and softmax are worth using.


How to run KNN (or other) on nested features? Image metadata I have looked everywhere and I can't find a straight solution. I have a set of metadata from images and their elements (name, heigh, width, position{top, right}). I would like to create categories of images based on relative positions of several items in the image.

Example:

img1 has

logo in top x1, right y1, and is height h1 and width w1

text in top x2, right y2, and is height h2 and width w2

from img1 to n

I thought about running a KNN classifier to find images that are more similar to others but I am having trouble thinking about how to organize the data.

Should I group them by image? In which case: how would KNN or other algorithms work on grouped data? Should I do OHE to know if there is a specific element? Would it be better to use a DecisionTree?

Its my first data science job, and there is nobody with seniority here so I am having trouble organizing this dataset.

Thanks in advance!

You could try to apply a dimension reduction technique to map all of the variables into 2, such as t-SNE or UMAP (I recommend this one), and then apply a clustering algorithm based on densities, such as HDBSCAN.

Be mind that each run of the HDBSCAN might return different results.
Scale all the variables before starting to apply the algorithms (scale)

Links to these algorithms in R:
UMAP:
HDBSCAN


Sklearn and PCA. Why is max n_row == max n_components? I posted my question on stack overflow, but there someone suggested that I should try it here. What I'm doing now :)

OK, first to my data. I have a word-bi-gram frequency matrix (1100 x 100658, dtype=int), where the first 5 columns contain information about the document. So every row is a document and every column a word-bi-gram like (of-the, on-the, and-that,...). I want to visualize the data, but before I do that, I want to reduce the dimension. So I thought I do that with PCA from sklearn. First I set the column labels with

myPandaDataFrame.columns = word-bi-grams

then I deleted some doc-columns, because I want to see what kind of information I can get if I only look at the proficiency.

del existing_df['SUBSET']
del existing_df['PROMPT']
del existing_df['L1']
del existing_df['ESSAYID']

then I set the proficiency column to be the index with

myPandaDataFrame.columns.set_index(['PROFICIENCY'], inplace=True, drop=True)

and then I did this

from sklearn.decomposition import PCA
x = 500
pcax = PCA(n_components=x)
pcax.fit(myPandaDataFrame)
PCA(copy=True, n_components=x, whiten=False)
existing_2dx = pcax.transform(myPandaDataFrame)
existing_df_2dx = pandas.DataFrame(existing_2dx)
existing_df_2dx.index = myPandaDataFrame.index
existing_df_2dx.columns = ['PC{0}'.format(i) for i in range(x)]

But with this implementation I can only set 1100 n_components as a maximum. This is the number of documents (rows). This makes me suspicious. I tried a couple of examples / tutorials, but I can't get it right. So I hope someone can help me find out what I'm doing wrong? If would also be very happy about a good example / tutorial that can help me with my problem. Thank you.

With best regards.

Given m rows of n columns, I think it's natural to think of the data as n-dimensional. However the inherent dimension d of the data may be lower; d <= n. d is the rank of the m x n matrix you could form from the data. The dimensionality of the data can be reduced to d with no loss of information, even. The same actually goes for rows, which is less intuitive but true; d <= m. So, it always makes sense to reduce dimensionality to something <= d since there's no loss; we typically reduce much further. This is why it won't let you reduce to more than the number of rows.


Uncertainty in connection to explainability When I write "uncertainty" in this post I mean:
If I have a classifier into $a_1,..,a_n$ categories and for an observation $x$ I classify $x$ to $a_i$ with probability $p_i$, then the uncertainty of this decision is $1-p_i$.
I'd like to inquire about connections of this notion and that of accuracy and explainability.
For example, if I have a classifier that is "very certain" (on mean/median on the test/training set) how often is this property correlated to achieving real-time accurate predictions? What about vice-versa?
Moreover, if my classifier is "certain" how does this affect my ability to explain its decision in any sense?
I couldn't find good resources for this notion of uncertainty and these questions so I will really appreciate some references as well!

There is a bit of confusion, I'm afraid:

The definition you propose for uncertainty doesn't really represent the concept of uncertainty: if $p_i$ is the probability that $x$ belongs to category $a_i$, then $1-p_i$ is just the probability that $x$ doesn't belong to category $a_i$.
Yes, if the classifier assigns a very high $p_i$, say 0.99, it is supposed to mean that the classifier is very confident in its prediction. But it's also the case for a very low probability: if $p_i=0.01$, the classifier is very confident that $x$ doesn't belong to $a_i$. According to your definition, the uncertainty in this case would be very high (0.99) even though the classifier is very confident, so your definition is inconsistent.

Now the main problem: whatever you call a confidence measure based on the probability predicted by the classifier, it's not reliable. A prediction is at best an informed decision of the classifier given the data it has seen in the training set and the features of instance. But it could be a random classifier, or a majority-class classifier: in these cases the probability it "predicts" is arbitrary. Imagine you are a teacher and one of your students says "the answer of x=2+2 is x=5", I'm 100% sure". The fact that the student is "100% sure" doesn't make them right, same thing for the classifier. In other words, any reliable measure of uncertainty involves the gold-standard answer, so it's usually part of the evaluation process. That's not to say that the predicted probability is useless, but in general it has no direct link to accuracy, and it would be a mistake to interpret it in this way.
Interpretability (or explainability) is a completely different matter: the general idea is to know whether the answer predicted by a classifier can be understood by a human. Typically traditional models like Naive Bayes or Decision Tree models are more directly interpretable (at least with not too many features) than deep NN models.


Word embedding for a single word I want to tackle email similarity with a word embedding approach, not an expert in embedding text but it is possible to embed emails where similar emails have similar vectors? is that okay?. I know that I can do text similarity approaches like Jacard or Levenhenstein but that would be too expensive for the job that I want to do because I will need to compare email from user i against all other users emails and do this same thing for all user ( i try already this and is too expensive ). I thought maybe i was able to embed the emails somehow ( this is my question - which way would be the best way to do that ? ). My final goal would say that vector from the embed of mereo923@gmail.com would be very similar to mereo329@gmail.com. There is any approach which could help me with the creation of this embedding?

There's a bit of confusion about your task: first, apparently by "emails" you mean email addresses, not the full content of the emails, right?
What you're doing looks like record linkage. The complexity of comparing every pair of records (here email addresses) can be decreased using the technique called blocking, where roughly similar pairs are grouped together in order to minimize the number of real comparisons (see Wikipedia page).
Now you're talking about using embeddings, and in this case what you'd need is character embeddings. However it's very unlikely that embeddings will help with the efficiency issue, quite the opposite actually.


R stem and leaf values How to make a stem and leaf plot with the following data in "R":

1717 1719 1645 3739 3024 3664 3830 2991 2430 2730 3469 5086 2119 3021 3292 2844 3426 2067 3215 2767 3124 2573 2840 2449 2584 1505 1390 1645 2497 3466 3228 3192

The only way to get a reasonable amount of stems is to make the stem the first digit in each number and the leaves will all be 3 digit numbers. I am not able to split the data in this way, R represents the stem with 1 digit but round my numbers  so 2584 is shown as 2| 5 (3 decimals to the right) which would be 2500. How do I adjust my stem and leaf numbers?

The stem function doesn't allow several digits as leaves, as far as I can tell. So you're left with implementing it from scratch. It's not too difficult but OMG it took me such a long time to get it right... I always get confused with R weird types conversions. Anyway, here it is:

d<-strtoi(unlist(strsplit('1717 1719 1645 3739 3024 3664 3830 2991 2430 2730 3469 5086 2119 3021 3292 2844 3426 2067 3215 2767 3124 2573 2840 2449 2584 1505 1390 1645 2497 3466 3228 3192',split=' ')))
dd<-ldply(d,function(x) {data.frame(stem=x %/% 1000,leaf=sprintf("%03d",x %% 1000),stringsAsFactors=FALSE)})
for (i in 0:9) { cat(paste(i,"|","")); l<-dd[dd$stem==i,]$leaf; if (length(l)>0) {cat(sort(l))}; cat("\n") }

Result:

0 |
1 | 390 505 645 645 717 719
2 | 067 119 430 449 497 573 584 730 767 840 844 991
3 | 021 024 124 192 215 228 292 426 466 469 664 739 830
4 |
5 | 086
6 |
7 |
8 |
9 |

Remark: there's no way to do this with the stem function simply because the purpose of a stem and leaves plot is not to display every exact value: the plot is only intended to give a "big picture" view of the distribution of the values. What matters for that is how many leaves there are for every stem, and it's enough to give a single digit since precision doesn't matter. From this point of view the stem function does the job:

stem(d)

Or variant:

stem(d,scale=0.5)


Sneakers representation learning I am trying to make a model which would take an image of shoes as an input and output a meaningful N-dimensional embedding of the shoes, so that they could be searchable/comparable/clustered and used in a recommender system.
My first guess was to employ a siamese CNN (Densesnet + 1 extra fully connected layer for the 32-dimensional embedding generation) with online hard mining triplet loss. So the idea was to train the network on making prediction if shoes on images belong to the same shoe model based on the euclidean distance between the outputs.
However, it failed to generalize well and has shown poor result on the test data.
My current dataset is ~4k images of ~500 different sneaker models.
What are my options in this situation?

Best is to start from pre-trained encoding. Please check TFHub (https://tfhub.dev/google/collections/image/1).
I have worked myself with encoding of shoes for a sport fashion companies. Remember that there are many different ways images might be seen as being similar, e.g. shape, color(!) etc... You might need to make a pre-processing choice first to drop some aspects, such as working with gray images, blurring some patterns on the shoes etc...


Learn the evaluation matrix for hire a job candidate Let us assume that we have a certain number of features that are weighted with some parameters . The features could be the different skills that belongs to a job candidate  applying for a job vacancies .  The decision to give a job to the single candidate is based on a weighted sum of a different features .So we have a neural network model on which we have to learn the matrix that associate the  variable x = features  that a candidate has to y = ( hire/not hire)
I was wondering what is the name in literature of this problem and where i can find suggestion to solve it.
Many thanks

its a simple DNN problem when you have x features and y labels you can have some data for NN training and then NN can be used to classify other candidates. I have a program already made for this plz comment if you need one .


How to change the names of the layers of deep learning in Keras? I am using vgg16 to create a deep learning model. I want to know how to change the names of the layers of deep learning in Keras?

I tried this

for layer in vgg_model.layers:
    layer.name = layer.name + str("_")

But when I change the names of the layers, the model accuracy become low.


Remove Outliers - Market Basket Analysis I'm having some thoughts on whether I should remove the outliers. I'm trying to find the tags that are commonly used together. Imagine that I have the following dataset. The first column is the Tag_ID and the second column is the Number of People that used that Tag.

1   3472034
2   1277918
3   1249839
4   1010770
5   915099
6   898292
7   636792
8   604352
9   555673
10  298495
11  291511
12  211074
13  200868
...

(This was copied from my actual dataset).

My question is: Should I remove a Tag instance when it is much more frequent than the other? Is that regarded as a good practice?

Many thanks!

Since I cannot comment to ask for clarification, I am asking it here. What is your reason to think about removing the most frequent value in your dataset? If the second column actually represent frequency of usage, you probably should not remove it and I think it would be illogical to throw away that piece of information. Having said that, you may consider removing that tag if it is a "less meaningful" word (e.g. a, an etc).

Can you give a little bit more context on what you are trying to achieve?

In general, one way to find outlier is to look at points that lie beyond 1.5 times of inter-quartile range of the distribution i.e. for the frequency count in your data.

Just a quick thought, did you try clustering for finding similar tags? What are the ways you are considering to find similar tags?


What does dimension represent in GloVe pre-trained word vectors? I'm using GloVe pre-trained word vectors (glove.6b.50d.txt, glove.6b.300d.txt) as word embedding.
I have a conceptual question:

What is the difference between these files?
On the other hand, what does the dimension represent in the GloVe pre-trained word vectors?

Glove creates word vectors that capture meaning in vector space by taking global count statistics. The training objective of GloVe is to learn word vectors such that their dot product equals the logarithm of the words probability of co-occurrence. while optimizing this, you can use any number of hidden representations for word vector. In the original paper, they trained with 25, 50, 100, 200, 300. These dimensions are not interpretable. after training, we are getting a vector with 'd' dim that captures many properties of that word. If the dimension is increasing, the vector can capture much more information but computational complexity will also increase. please go through this blog.


statsmodels ols does not include all categorical values I am doing an ordinary least squares regression (in python with statsmodels) using a categorical variable as a predictor. There are 5 values that the categorical variable can have. However, after running the regression, the output only includes 4 of them.

Here is what I am running:

>>> from statsmodels.formula.api import ols
>>> model = ols("normalized_score ~ C(general_subreddit)", data=df_feature)
>>> results = model.fit()
>>> results.summary()

The output of the last command includes the following rows in the table:

I can check the count of each of the categorical variables as follows:

>>> from collections import Counter
>>> Counter(df_feature["general_subreddit"])
Counter({nan: 20,
     'community': 4159,
     'ending_addiction': 3819,
     'mental_health': 4650,
     'other': 6920,
     'relationships': 4318})

Ignoring the NaNs, why does the categorical value of "community" not appear in the model summary?

When a logistic model is built using a categorical variable with N levels, it only considers N-1 levels, as the remaining level is used as a reference by the model.

What this means to your model as a whole is that, each level (when remaining variables remain same) is compared to the reference level.

In your example, "community" level is used as the reference. So, ending_addiction contributes 0.0749 units more than "community" to the dependent variable.

Hit this up if it's not clear -->
https://community.alteryx.com/t5/Alteryx-Designer-Discussions/In-the-Logistic-Regression-Report-Factor-Missing/m-p/10100/highlight/true#M5169


What loss function should I use for auto encoders for text I'm trying to implement an autoencoder for text. But I don't know which loss function I should use ? I tried using the mse but I get a huge loss 1063442. I'm trying to build a very simple autoencoder using only LSTM layers in Keras. On the keras-blog they use binary-crossentropy but I think the reason for this is because they use black and white images.

Thanks

The cause of your huge loss is probably a lack of normalization. As an example, in the Keras-blog post you linked, they conduct such normalization by dividing the input by 255, which is the maximum value an element in the input can be of.

As for which loss function to use, if you have a look at this keras-blog post, they construct a LSTM-based auto encoder for text which uses categorical crossentropy as a loss function.


Should the type of Boolean categorical features be numerical or categorical after encoding? There are categorical features which have two different value in my dataframe next to numerical features. I've converted these categorical values to 0 or 1.

I will apply correalation elimination on features after calculating correlation coefficients. Depending on type of features, methods are given below:

Numeric - Numeric: Pearson

Numeric - Categoric: Cramer_V

Categoric - Categoric: Correlation Ratio

That's why I could not be sure what should be type of converted categorical features? Numerical or categorical ?

Another reason to I asked this question is that when I create dummy features for the categorical features which have only two different values, it creates features contains 0 and 1 like how I did manually. So after this process it's taking these features as numerical. But still each value from the feature represents a class and I think feature type should not be numerical.

In mathematics and mathematical logic, Boolean algebra is the branch of algebra in which the values of the variables are the truth values true and false, usually denoted 1 and 0 respectively. Generally, gender can not  be considered  - true or false . Classification is based on sex - Male or female  . Categories differ and can be assigned particular  numerical values (dummy variable) 1, 2 or 2, 3 etc. Binary variable is different. If you want to find correlation  between  the scores obtained by Male students and female students,  you will have to have matched pairs. Categories measure the levels for grouping. Binary classification can be undertaken for a dependent  variable in regression analysis or point biserial correlation. The standard softwares for statistical analysis do not invoke 0,1 as variable. The probability based solutions may use this classification for grouping.


does index of my data which is of type "Date time index" plays a part in reggression? I'm new to data science and I'm working on a regression problem. My question is the index of my data which is of type "Date time index"  plays a part in regression? I mean is it Okay if i drop the index ?

You can take the date into account, but you should convert it into numeric value.
Suggestions:
1) Consider "datetime" package in python to change it to numeric, or some other techniques such as indexing with date-time, and converting the index value to numeric, float, etc. I don't have your exact case, so cannot say specifically.
or
2) If you have time series data, then in many cases ARMAX, ARIMAX, Seasonal ARIMAX models can do a really good job.


How to merge all the data to have a final dataset I am working with a problem that has different tables, end goal to predict if a customer will end up subscribing based on purchases.
Mother table containing user_id,register_reason
|user_id|reason_reg|source|
|-------|----------|------|
|   1   |     2    |  3   |
|   2   |     3    |  1   |

I have then the purchase data where a customer can have more than one entry
|user_id|product_id|
|-------|----------|
|   1   |     A    |
|   1   |     B    |
|   1   |     C    |
|   1   |     D    |
|   2   |     A    |
|   2   |     E    |

Ideally I want to have one dataset where the uniqueidentifier is the user_id and there are no duplicated rows based on this value:
The final dataset (in my head), could look like
|user_id|reason_reg|source|product_id_A|product_id_B|product_id_C|product_id_D|product_id_E|
|------------------------------------------------------------------------------------------|
|   1   |     2    |   3  |      1     |      1     |      1     |      1     |      0     |
|   2   |     3    |   1  |      1     |      0     |      0     |      0     |      1     |

My questions are:

Is the approach correct?

Is there a dataframe or library that does that automatically? or I do that myself by panda before feeding it to a algorithm.

In your opinion, is there a better way to approach the problem(I could add also add aditional columns like total_products with the sum of how many products the user bought)

A slightly hacky way to get there maybe but you can do this to get what you want from the second table;
df2['count'] = 1
pivot = df.pivot_table(df, index='userid', columns='productid', values = 'count').reset_index()
pivot = pivot.fillna(0)

You would then want to merge this to the first dataset like this;
finaldf = pd.merge(df1, pivot, left_on='userid', right_on='userid')

another great thing to use for generating the dummies for categorical variables is
pd.get_dummies()

The approach seems ok to me and making some more features would also not be a bad idea.


Best common metric for comparing classic time series forecasting methods (ARIMA/Prophet) with ML approach? I am new to time series forecasting and looking to compare the performance of ARIMA/Prophet with an XGBoost model in predicting future stock market values based on historical stock market data and social media sentiment scores.
I am more familiar with ML, so would usually use an evaluation metric like R^2 to assess model performance for this sort of problem.
Are there any common evaluation metrics which forecasting methods like ARIMA/Prophet use to assess their accuracy, so I can do a like-for-like comparison with the XGBoost's prediction accuracy?

In general the most common metrics mentioned in several articles (e.g. like this one) are the same we also commonly use for non-time series prediction:

MAE Mean absolute error
MSE Mean squared error
RMSE Root mean squared error

Outside of linear regressions I have not seen R² used that often to validate prediction models. In fact it isn't even one of the out-of-the-box metrics of xgboost for interval scale prediction tasks. So you should be fine using RMSE for everything.
Edit:
Prophet at least in the R package covers RMSE, MAE and MAPE so you should be fine.


What is the right algorithm to detect segmentations of a line chart? To be concrete, given 2D numerical data as is shown as line plots below. There are peaks on a background average movement (with small vibrations). We want to find the values of pairs (x1, x2) if those peaks drops down to average; or (x1) only if the line doesn't back to the average.

There are thousands of such 2D data.

What is the right statistic or machine learning algorithm to find x1 and x2 above without plotting?

One way to do what you're talking about is called "change point analysis." There is an R package for this called changepoint that you might want to check out.

In Python, you could try changefinder.


SHAP value can explain right? I face a problem with using SHAP value to interpret the Tree-based model.
(https://github.com/slundberg/shapsd)
First, I have input around 30 features and I have 2 features that have high positive correlation between them.
After that, I train the XGBoost model(python) and look at SHAP values of 2 features the SHAP values have negative correlation.

Could you all explain to me, why the output SHAP values between 2 features don't have the correlation the same as input correlation? and I can trust that output of SHAP or not?

=========================

The correlation between input: 0.91788
The correlation between SHAP values: -0.661088

2 features are
1) Pupulation in province and
2) Number of family in province.

Model Performance
Train AUC: 0.73
Test AUC: 0.71

Scatter plot
Input scatter plot (x: Number of family in province, y: Pupulation in province)
SHAP values output scatter plot (x: Number of family in province, y: Pupulation in province)

I guess what you meant by correlation between SHAP values is "SHAP Interaction Value".

SHAP value is a measure how feature values are contributing a target variable in observation level. Likewise SHAP interaction value considers target values while correlation between features (Pearson, Spearman etc) does not involve target values therefore they might have different magnitudes and directions.

The features may grow together but their contribution to target variable in different intervals may reverse.

You may want to check docs and this beautiful work.


Logistic regression if 3 categories in outcome variable Logistic regression is generally performed if there are 2 categories in outcome variables. I just tried it for iris dataset with species as y variable which has 3 categories. I used following code:

import pandas as pd
import matplotlib.pylab as plt

from sklearn.linear_model import LogisticRegression
clf = LogisticRegression()

from sklearn import datasets
iris = datasets.load_iris()

clf.fit(iris.data, iris.target)
logcoefdf = pd.DataFrame(data=clf.coef_,
     columns=["SL", "SW", "PL", "PW"],
     index=['setosa','versicolor','virginica'])
print(logcoefdf)
logcoefdf.plot.bar()
plt.show()

The printout and plot of coefficients is as follows:

                  SL        SW        PL        PW
setosa      0.414988  1.461297 -2.262141 -1.029095
versicolor  0.416640 -1.600833  0.577658 -1.385538
virginica  -1.707525 -1.534268  2.470972  2.555382

(I have labelled rows by names of species but I am not sure if this is correct).

From above output I get following plot:

What is the interpretation of these results? Does it mean that petal length (PL) is lowest in setosa and highest in virginica group? And both sepal width and petal width are less in versicolor species? Thanks for your insight.

Edit:
If I use only 2 categories of iris dataset, I get only one set of coefficients:

clf.fit(iris.data[0:100,:], iris.target[0:100])
print(clf.coef_)

Output:

[[-0.40731745 -1.46092371  2.24004724  1.00841492]]

Is it that Logistic regression is being performed for all possible combinations of categories, i.e. setosa vs versicolor, versicolor vs virginica and virginica vs setosa?

Logistic regression can work on multi-class. Frankly it is not big different from binary classifier. The question is how does it find the coefficients for each class which you had displayed. By default to find the coefficients for a single class, it goes with one vs rest combination. So setosa vs remaining all classes. It gets coefficients iteratively by minimizing cross entropy. So ultimately you will end up with a matrix having a size of n_classes vs features.

Coming to the interpretation part of the question. To interpret the importance of a single feature, you can do it as - Given the coefficients of every other feature to be same (in comparing a class with another) including intercept, the class that has the feature with highest coefficient has greater chance of engulfing the new point. This being said the features having high petal length and width and low sepal length and width contribute good for a data point to belong to class virginica. But it does not make much sense to interpret each coefficient individually. Hope I made some sense.


What does the descision boundary of a relu look like? A single non activated neuron is just a linear combination of its inputs.

Thresholding this neuron's output as-is against 0 would create a hyperplane binary separator, whose parameters can be learned.

What would be the shape of this separator (decision boundary) in case we also take a relu on the output and only then threshold?

I keep thinking it has to be non linear, otherwise NNs wouldn't work, but if something was positive before, it would remain positive, and if it were negative or zero, it would stay, and so the relu actually did nothing for the decision boundary, which makes no sense to me.

If someone could make order of this for me I would be glad.

As a follow up, I would like to understand multi level descision boundaries  - say on a 2 level network. What would the boundary look like with 2 neurons per layer, on a 2 layer network, for, say, a XOR-like dataset?

What would be the shape of this separator (decision boundary) in case we also take a relu on the output and only then threshold?

For just a single neuron, indeed the decision boundary will just be a plane.

I keep thinking it has to be non linear, otherwise NNs wouldn't work...

Single-layer neural networks don't demonstrate the same nonlinearity as multi-layer networks, only as much as an activation function.  (With sigmoid activation, you're just doing logistic regression.)

As a follow up, I would like to understand multi level descision boundaries...

With multiple layers, you gain the ability to combine boundary planes into polytopal regions.  A ReLU-only neural network represents a piecewise-linear function, with finitely many polytopal (i.e., flat-edged) pieces.  That's clear for every neuron in the first layer. In each subsequent layer, you take a linear combination of such functions, which is again such a function; applying a ReLU to that then can cut each region in two, but the result is still such a function.  (You may also be interested in this paper, and this expository blog post.)
You can see some of this in the tensorflow playground.  In a version of that link, I've grabbed the weights and put the resulting function into Wolfram|Alpha.  You can see how the boundary lines of each first-layer neuron shows up as a kink, and each of the resulting regions supports a linear function.  Then slicing at a fixed height gives you the hexagonal region that works well enough to classify the circle dataset.  Increasing the number of layers or neurons per layer allows you to cut the space into more regions.
Then, even if the output neuron applies a sigmoid as in your tensorflow playground, the decision boundary is just a level set of a piecewise-linear function with all linear boundaries, which is itself linear.


How to understand what each layer is learning in a Deep learning neural network? In a recent answer I read on Stack Exchange, I read about a possible way to understand more clearly what happens in each hidden layer of a neural network.

Here's the excerpt-

  You should watch what makes neuron activated in each layer depending
  on the input. As you know each neuron will be activated (once the DNN
  is trained) for specific input combinations. By visualizing that you
  can get an idea about what exactly each layer has learned in term of
  high-low level features.

Source - High-level features of a neural network

I wanted to know if there are any papers who have tried doing this (links would be really helpful). Meanwhile are there any other ways to understand what happens in each hidden layers?

I wanted to know if there are any papers who have tried doing this (links would be really helpful).

The best known paper on this topic is Visualizing and Understanding Convolutional Networks by Matthew Zeiler and Rob Fergus. But you can find many more papers and online articles that do the same.

  Meanwhile are there any other ways to understand what happens in each hidden layers?

Unfortunately, deep neural networks are like black boxes; we have little understanding of their inner workings. Visualizing the activations of the layers is probably the best way to tell what features they extract.


How best to show the best model over multiple labels? I have 4 models I trained and I want to display their prediction success over 45 different labels I tested them on.

I get a very messy plot when I naively try to place them one on top of the other.

Here is the bar of the 4 models over 45 labels:

And here is the line of the 4 models over 45 labels:

I thought to maybe show them in parts, meaning create like 9 graph, each graph show only 5 labels.

What I really would like to is maybe show the winner model in each label and so, the winning model's color will pop right in front of you.

How do you suggest to do that?

You can use some index, such as (possibly weighted) average.

Another way would be to use 3 dimensional graph, with one of the dimensions being the task.