Does it make sense to do train test split when trainning GANS? For normal supervised learning the dataset is split in train and test (let's keep it simple).

Generative Adversarial Networks are unsupervised learning but there is a supervised loss function in the discriminator.

Does it make sense to split the data into train and test when training GANs?

My first opinion will be no, but I am not 100% sure. Is there any reason why having a test set will help the Generator?

Training GANs is only a partially unsupervised task, IMHO. It's certainly unsupervised for the Generator, but it's supervised for the Adversarial Network. So it might be useful to test the Disciminator's ability to distinguish fake and true cases on new data it has never seen before.

In other words, it makes sense to split your dataset in train(-validation)-test if you want to understand the Discriminator's ability to generalize its task on data it has never seen before. In case this is not of your interest, I guess you don't need to do it.


MSE vs Cross Entropy for training with facial landmark (pose) heatmaps I am trying to reimplement the excellent paper https://github.com/1adrianb/face-alignment-training in tensorflow.  I have successfully defined the network and downloaded the LSD3D-W dataset.  I am able to train the model however I am running into a serious issue.

Ground Truth and Loss

For training, I generate ground truths by converting the x,y landmark coordinates into gaussian heatmaps where x,y is the mean of the gaussian.
I first trained with MSE loss as given in the original implementation.  After some iterations, the loss becomes extremely small but the output is completely white!

loss = tf.losses.mean_squared_error(
    predictions=heatmaps, labels=labels_tensor
)

When I tried with cross entropy, I am getting better results.  But they are not sharper

loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=heatmaps, labels=labels_tensor), name= 'cross_entropy_loss')

gtmap means groundtruth map.

I experienced a similar problem while doing pose estimation. Your problem may be due to the huge amount of "background" pixels, where the gaussian heatmap has very small values. So the network simply learns the trivial solution and sets the whole heatmap to zero, which results in a small mse error.

Solution to my problem was:

1) weigh the heatmaps: weigh pixels in the loss function, which are "foreground" more. For example you could count the percentage of "foreground" pixels and the percentage of "background" pixels for each heatmap and weigh foreground pixels with (1-percentage of foreground pixels) and background pixels with (1-percentage of background pixels)

2) Increase the maximum of the gaussians to ~12


In Neural Nets, why Use Gradient Methods as Opposed to Other Metaheuristics? In training deep and shallow neural networks, why are gradient methods (e.g. gradient descent,  Nesterov, Newton-Raphson) commonly used, as opposed to other metaheuristics?

By metaheuristics I mean methods such as simulated annealing, ant colony optimization, etc., which were developed to avoid getting stuck in a local minima.

It would be a waste of information; the gradient is available, so use it and save time.
There is reason to believe that the local optima are good; see, for example, Choromanska et al. (notes).
Over-optimizing for the training set leads to worse generalization, so sometimes we deliberately don't even try by stopping early. Probably the best free lunch in machine learning.


ValueError: y should be a 1d array, got an array of shape (285, 30) instead I am using this data set below and I am trying to find the support vector machine of the data set. Also
I have my code and error below as well.
http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html#sklearn.datasets.load_breast_cancer

import numpy as np
from sklearn import svm,datasets
breastcancer = datasets.load_breast_cancer()
#print(breastcancer)
everydata = breastcancer.data
#print(everydata)
everytarget = breastcancer.target
traindata = []
traintarget = []
testdata = []
testclasses = []
#Class 0 data separtion
for i in range(0,140):
    traindata.append(everydata[i])
    traintarget.append(everytarget[i])
for i in range(140,212):
    testdata.append(everydata[i])
#Class 1 data separation
for i in range(212,357):
    traindata.append(everydata[i])
    traintarget.append(everytarget[i])
for i in range(357,569):
    testdata.append(everydata[i])
traindata = np.concatenate((everydata[:140, :],everydata[212:357, :]),axis=0)
traintarget = np.concatenate((everydata[:140],everydata[212:357]),axis=0)
testdata = np.concatenate((everydata[140:212, :],everydata[357:569, :]),axis=0)
print(len(traindata))
print(traintarget)
print(testdata)
dd = svm.SVC(kernel='linear')
dd.fit(traindata,traintarget)
decide = dd.predict(testdata)
print(decide)

Why am I getting this error is my concatenation incorrect. The results are supposed to output 0s and 1s at the end.

The problem is that you are using the everydaydata to build the traintarget dataset, but you should use the labels in everytarget. That is why is complaining abut the shape, because the labels should be one dimensional.
Try replacing this:
traintarget = np.concatenate((everydata[:140],everydata[212:357]),axis=0)

with this:
traintarget = np.concatenate((everytarget[:140],everytarget[212:357]),axis=0)


Unable to understand which features to choose I am a newbie here, but I am trying to work with a dataset which gives the attempt at the goal by a footballer,which will predict one of 2 possible outputs - whether or not they could score the goal or not.
I have done some basic cleaning but I am still getting only 60% accuracy on whatever classifier I use from sklearn.
I have removed a few features which I thought would not contribute to the 'y' value, and encoded a few values, but still it doesn't bump up the accuracy.
I am not sure how to attach a csv, but this is what the data looks like -- data
For the rows that 'is_goal' doesn't have a value, the aim is to find out if a goal was scored or not.
Should I attach the notebook as well to show what work I've done till now?

Create a correlation plot and remove all highly correlated columns. Remove columns that are irrelevant to the prediction, such as home/away

If you're interested in feature selection you can use the methods in mlxtend library. It's likely that selecting the right features won't improve your results drastically. There can be other things that can be done to do so.

Here are some suggestions to improve your results:

Make sure you're treating the columns with continuous and categorical columns in appropriate ways - encoding only the categorical columns
Normalize all columns that have continuous variables, perhaps using StandardScaler
Experiment with using feature extraction techniques like PCA
Try using gradient boosting (LightGBM or CatBoost)
There seems to be missing data. Make sure to deal with them correctly. If they are very less you can drop those rows.


What is "Missing" in output of plot_tree API of XGBoost What this "Missing" term means here at each node after split in Image? and also what is at leaf, is this means prediction value? I converted Output variable to 1 and 0.
I tried searching on google, but didn't get the answer. Please help

The "missing" at the different nodes are the observations for which the feature on which the split is made is missing. E.g. if the value for TotalSubmissionGuidelines is below 0.5 or missing, then the observation follows the left path of the decision tree. XGBoost supports missing values by default and learns the branch direction during training, see the documentation.


How to get rid of the expectation in Monte Carlo Policy Gradient method? Taken from Policy Gradient lecture notes, slide 16 onward:

Here in the David Silver Lecture series I perfectly understand how he took the expectation value in the theorem part by combining the following equation .

But how he removed the expectation part when talking stochastic gradient decent ?

But how he removed the expectation part when talking stochastic gradient decent ?

A result that shows what happens in expectation can be estimated by sampling it, and that is precisely what stochastic gradient descent (or ascent in this case) methods do - they operate on individual samples on the assumption that this will on average produce reasonable direction for optimising parameters.

So there is no need to "get rid of" the expectation. In fact the sequence of equations is working deliberately towards it, because in most situations we do not know the full characteristics of the MDP and cannot calculate $d(s)$ (and maybe not even $\mathcal{R}_{s,a}$) - the expectation is used to remove the need for knowing those terms.

Importantly, $d(s)$ is the probability density of the state = the likelihood on a random sample of all states visited under a certain policy of finding the agent/environment in that state. This is often hard to calculate directly, even if you know the MDP and the policy, and it is intractable if you do not have a model of the MDP. However, when you take many samples then just by the act of sampling and using the value of $s$ that is observed, then you will approximate the true distribution of states in the long run.


Get a portion of a long field in Pandas? I have a Pandas dataframe that has some fields that contain very verbose text. I want to be able to iterate through the DF but only display a limited set of words. I have code similar to:

for index, row in df.iterrows() :
    print(row['A'], row['B'])

How can I make sure that I only print the first 300 characters from 'A'?

Fairly simply, for 'A' only:

max_chars = 300
for index, row in df.iterrows() :
    print(row['A'][:max_chars], row['B'])


How do you detect seasonality(multiplicative or additive) in a time series data? Trying to apply seasonal_decompose on timeseries data. It looks something like this:

            modal_price
Period
2014-11-01  1469
2015-01-01  1258
2015-03-01  1112
2015-04-01  1373
2015-06-01  1370
2015-07-01  1406
2015-08-01  1520
2015-09-01  1860
2015-10-01  1436
2015-11-01  1455

Using freq=11 and seasonalcomponents are all Nan. What should i do in order to detect the seasonality type(multiplicative/additive) of this data?

Once divide the time-series by its Moving Average and once subtract the moving average from it.

If the seasonality is additive, then the result of subtract will have almost similar fluctuations in magnitude.

If it is multiplicative, then the division result has such a property.

Like the figures below from my course in Feature Engineering:

So the seasonality was multiplicative as the division has similar magnitudes of fluctuations.

Please note that more linear the dynamic of the time-series is, better this naive approach works.

Hope it helps!


Kur vs Keras - pros and cons I just stumbled across Kur. At first look, it seems to be making deep learning easy in almost similar lines like Keras on a high level. What are the pros and cons of each and what is suggested to be used for beginners?

(Disclaimer: I am the core maintainer of Kur.)

Kur and Keras are both deep learning frameworks, but they serve different purposes:

Keras builds on Theano and TensorFlow to provide a high-level API to the underlying tensor libraries. This means that you use Keras as part of your Python program, and you programmatically create your model like this (this is Keras' functional API):

import keras.layers as L
import keras.models as M

my_input = L.Input(shape=(100,))
intermediate = L.Dense(10, activation='relu')(my_input)
my_output = L.Dense(1, activation='softmax')(intermediate)

model = M.Model(input=my_input, output=my_output)

This is great when you are developing deep learning networks that you want to "embed" in some larger Python program, or when you are developing a single model that you intend to use. Keras is also quite mature and has a great community supporting it. Keras currently offers the superior API for models that you will create and use entirely within Python.
Kur builds on top of Keras to provide a descriptive framework for designing and running deep learning models; you don't write programs when using Kur. Instead of programming, you write "Kurfiles" which describe your model:

model:
  - input: my_input
  - dense: 10
  - activation: relu
  - dense: 1
  - activation: softmax
  - output: my_output

That's the same model as I just wrote in Keras. Kur's descriptive language makes quickly prototyping and testing your models. The advantage of the Kurfile is that you can tweak large or small pieces of your model without having to touch your codebase. And when Kurhub is released, you'll have a Docker-like repository for sharing models.

Kur also takes care of a lot of details that you need to consider when writing deep learning systems. For example, Kur knows about several different datatypes (CSV, Python pickle, etc.), and it can load the data and infer shapes for you. Without Kur, you'd have to write your own data processing pipeline (which may be easy or hard, depending on your data). Once you write the Kurfile (in YAML or JSON), you can train with kur train Kurfile or use it for inference with kur evaluate Kurfile. This makes it very easy to use as a tool, rather than as a Python API.

Both Kur and Keras can use Theano or TensorFlow, both offer Python APIs, and both are easy to write models in. It really depends on how to plan to develop and deploy your system.


Multiple confusion matrix for multiple training instances. Which one to take? I am using Matlab Neural Network toolbox for a classification problem. Now  considering a single set of data, if the inbuilt neural network is trained and classified with same data multiple number of time, different accuracy and different confusion matrix is obtained. Now which result should I take? Should I take all the vales obtained in all the training instances and average them fix on one particular result?

I can't check at the moment (no Matlab at hand), but I suppose the differences come from the different random seeds used to initialize the neural networks (at least this is the only part which i can think of that has a random component). I would suggest predicting class probabilities, averaging those and then viewing the resulting confusion matrix of the "averaged" prediction. This way you - to a degree - mitigating the effect of randomness resulting from different initializations of the weights.


Can I arbitrarily eliminate 20% of my training data if doing so significantly improves model accuracy? My dataset contains 2000 records with 125 meaningful fields 5 of which are distributed along highly skewed lognormal behavior.

I've found that if I eliminate all records below some threshold of this lognormal behavior (by combining the fields together then filtering for Nth percentile), my model improves in accuracy from ~78% to ~86%, using a highly tuned random forests classifier. This filter is only done after splitting my data into train, test (which is done after SMOTE).

What makes this particularly odd is that that filter improves results across multiple sampling methods.

Is this filtering acceptable behavior? Why might it be resulting in better predictions?

One flaw in your procedure is the use of SMOTE before splitting in train/test. This should be avoided as you may have synthetic examples in the test data which generation depends on training data and that will be highly close to this data in your feature space (as SMOTE uses Euclidean distance).

Moreover, if most of the minority data belongs to the not-skewed region of your specific variables, these points will be also over sampled and so this reduction in the variables space will produce an overly optimistic performance which does not reflect the real distribution of the data.


Handling Concat and Shift Feature in Pandas for Data Science I am trying to USE Lag features AND Concat() and Shift() function,

    seies = Series.from_csv('sugar_price_2.csv', header=0, sep=';')

    In [25]: prices = DataFrame(series.values)

    In [26]: dataframe = concat([prices.shift(3), prices.shift(2), prices.shift(1), prices], axis=1)

    In [27]: dataframe.coloumns = ['t-2', 't-1', 't', 't+1']

    In [28]: print(dataframe.head(20))

       0       0       0     0
0      NaN     NaN     NaN  2800
1      NaN     NaN  2800.0  2800
2      NaN  2800.0  2800.0  2800
3   2800.0  2800.0  2800.0  2800
4   2800.0  2800.0  2800.0  2800
5   2800.0  2800.0  2800.0  2800

But The 't-2', 't-1', 't' Coloumn names aren't showing up .

Can anyone say what's wrong in my code...

As was pointed out by @Stephan Rauch in his comment, the names of the columns are stored in dataframe.columns - the OP had a typo.

Below is a working example with dummy data, getting the same output as the user - using instead a little loop to compute the shifted values.

from pandas import DataFrame

prices = dict(
    col1=[0, 1, 2, 3, 4, 5, 6],
    col2=[2, 3, 4, 5, 6, 7, 8],
    col3=[5, 6, 7, 8, 9, 10, 11],
    col4=[12, 13, 14, 15, 16, 17, 18])

dataframe = DataFrame.from_dict(prices)
print(dataframe)
new_col_names = ['t-2', 't-1', 't', 't+1']
dataframe.columns = new_col_names
print(dataframe)

# Number of columns we have
N = len(dataframe.columns)

for n, col in enumerate(dataframe.columns):
    shift_by = N - n - 1  # don't shift the final column
    dataframe[col] = dataframe[col].shift(periods=shift_by, axis=0)

print(dataframe)

# If desired, remove the new NaNs that appear in the first
final_dataframe = dataframe.drop(labels=dataframe.index[:N - 1], axis='index')
print(final_dataframe)


How to read the predicted label of a Neural Netowork with Cross Entropy Loss? Pytorch I am using a neural network to predict the quality of the Red Wine dataset, available on UCI machine Learning, using Pytorch, and Cross Entropy Loss as loss function.
This is my code:
input_size = len(input_columns)
hidden_size = 12
output_size = 6 #because there are 6 classes

#Loss function
loss_fn = F.cross_entropy

class WineQuality(nn.Module):
    def __init__(self):
        super().__init__()
        # input to hidden layer
        self.linear1 = nn.Linear(input_size, hidden_size)
        # hidden layer and output
        self.linear2 = nn.Linear(hidden_size, output_size)

    def forward(self, xb):
        out = self.linear1(xb)
        out = F.relu(out)
        out = self.linear2(out)
        return out

    def training_step(self, batch):
        inputs, targets = batch
        # Generate predictions
        out = self(inputs)
        # Calcuate loss
        loss = loss_fn(out,torch.argmax(targets, dim=1))
        return loss

    def validation_step(self, batch):
        inputs, targets = batch
        # Generate predictions
        out = self(inputs)
        # Calculate loss
        loss = loss_fn(out, torch.argmax(targets, dim=1))
        return {'val_loss': loss.detach()}

    def validation_epoch_end(self, outputs):
        batch_losses = [x['val_loss'] for x in outputs]
        epoch_loss = torch.stack(batch_losses).mean()   # Combine losses
        return {'val_loss': epoch_loss.item()}

    def epoch_end(self, epoch, result, num_epochs):
        # Print result every 100th epoch
        if (epoch+1) % 100 == 0 or epoch == num_epochs-1:
            print("Epoch [{}], val_loss: {:.4f}".format(epoch+1, result['val_loss']))

model = WineQuality()

def evaluate(model, val_loader):
    outputs = [model.validation_step(batch) for batch in val_loader]
    return model.validation_epoch_end(outputs)

def fit(epochs, lr, model, train_loader, val_loader, opt_func=torch.optim.SGD):
    history = []
    optimizer = opt_func(model.parameters(), lr)
    for epoch in range(epochs):
        # Training Phase
        for batch in train_loader:
            loss = model.training_step(batch)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
        # Validation phase
        result = evaluate(model, val_loader)
        model.epoch_end(epoch, result, epochs)
        history.append(result)
    return history

loss_value = evaluate(model, valid_dl)

#model=WineQuality()
epochs = 1000
lr = 1e-5
history = fit(epochs, lr, model, train_loader, val_loader)

I can see that the model is good and that the loss decreases. The problem is when I have to do a prediction on an example:
def predict_single(input, target, model):
    inputs = input.unsqueeze(0)
    predictions = model(inputs)
    prediction = predictions[0].detach()
    print("Input:", input)
    print("Target:", target)
    print("Prediction:", prediction)
    return prediction

input, target = val_df[1]
prediction = predict_single(input, target, model)

This returns:
Input: tensor([0.8705, 0.3900, 2.1000, 0.0650, 4.1206, 3.3000, 0.5300, 0.2610])
Target: tensor([6.])
Prediction: tensor([ 3.6465,  0.2800, -0.4561, -1.6733, -0.6519, -0.1650])

I want to see to what are associated these logits, in the sense that I know that the highest logit is associated to the predicted class, but I want to see that class. I also applied softmax to rescale these values in a probability:
prediction = F.softmax(prediction)
print(prediction)
output = model(input.unsqueeze(0))
_,pred = output.max(1)
print(pred)

And the output is the following:
tensor([0.3296, 0.1361, 0.1339, 0.1324, 0.1335, 0.1346])
tensor([0])

I don't know what is that tensor([0]). I expect my predicted label, a value like 6.1 if the target is 6. But I am not able to obtain this.

If you're dealing with classification problem, then model.predict is supposed to give you logits.
outputs = net(images)
_, predicted = torch.max(outputs, 1)

for i in range(num_input):
    print(classes[predicted[i]])

if you have only one input then the predicted class would be as following:
classes[predicted[0]]
In your case:
prediction = F.softmax(prediction)
print(prediction)
output = model(input.unsqueeze(0))
_,pred = output.max(1)
print(pred)

# print class name
print(classes[pred[0]])  # <- should be added


You need to get used to so called wide and long table format, from there you should get the trick rapidly. By then, I would use unstack in order to have three columns and do something like this:

unstacked_data = describedWidth.unstack()
unstacked_data = unstacked_data.reset_index()
unstacked_data.columns = ['metric', 'run', 'y']
sns.barplot(data=unstacked_data, x="metric", y="y", hue="run" )

Which should give a table that has the following format:

And final result:


Clarify recurrent neural networks I'm in the beginning to learn and understand recurrent neural networks. As far as I can imagine, its multiple feed-forward neural networks with one neuron at each layer put next to each other, and connected from left to right, where each neuron is connected not just with the neuron below it, but the one at the left from the previous time. Not sure if it's a right way to think about it, but so far it's my first impression.
Some things are unclear though.

As far as I understood the final output of each timestep is supposed to predict the input of the next timestep. Is this true? What if I would just like to show the network two images of for example a horse, and depending on them, predict what distance did it go, and in which direction? Is this possible?

In the illustration above there's $A_0$. From where? I would assume at least two timesteps are needed to make a prediction, so in my understanding an $x_0$ is missing from the left side of the diagram. Am I right?

I've been reading through an article which says "Lets train a 2-layer LSTM with 512 hidden nodes". Does it mean two layer of activations, and 512 timesteps?

As far as I can imagine, its multiple feed-forward neural networks with one neuron at each layer put next to each other, and connected from left to right, where each neuron is connected not just with the neuron below it, but the one at the left from the previous time.

Not really. Each cyan box in your image represents the exact same cell. Now this cell can be a lot of things just take a look at LSTM cell (h and c represents your A) but it can also be a network which takes $A_i$ and $X_{i+1}$ as input and returns $A_{i+1}$ and $Y_{i+1}$ as output.

It may be true if the RNN tries to predict e.g. a time series. To train such a net you would provide the series as a training input and the same time series but in the next step as an output (so it would try to predict $X_{i+1}$ based on $\forall_{j \in [1;i)} X_{j}$. But in general it's not true. The output may be in a completely different format and represent completely different thing exactly like in your example. In your example your $X_i$ is an encoded i-th frame and $Y_i$ is what the network thinks the distance the horse has traveled is up until i-th frame.
$A_0$ is the starting state of the RNN. What it is exactly depends on the exact architecture used but it's common to just set it all to zeros. We need this starting state because as I mentioned the same cell is used at each recurrent step so there has to be something to provide as network state in the beginning. There is no $X_0$ missing. Also there is nothing stopping you from making a prediction based on a sequence of length 1. It's just that it's not useful to use a RNN in such a situation.
The number of timesteps taken depend on the data - not the network. You can use the same network on sequence of length 512, 2 and 2 million. That's why they're commonly used to solve problems with varying length like speech recognition. LSTMs have weights just like a normal neural network does. You can think of these 512 hidden nodes as the size of a hidden layer in the cell. Using two layers of LSTM means using two LSTMs with 512  nodes, and using output of the first as the input of the second one. The output of the second LSTM is the output of 2-layer LSTM.


How to measure Covid impact by analysing credit card transaction of customer I Want to know how can I identify that is the customer is in financial distress due to the COVID situation using its credit card transactions.

I have a daily transaction of customers till current date,
Any thoughts or ideas would be really helpful.

For example, my thoughts that if sudden increment in credit card utilization then it can be an indicator that person in financial distress as could be a flag to potential lenders or creditors that  having trouble managing your finances.

Do you have some sort of labeled data?

Otherwise I'd hypothesize that this task is close to impossible since any sort of unsupervised algorithm using anomaly detection would likely have an incredibly low precision since almost all data would be classified as an anomaly during this time.

However, if you were to obtain some labeled data (labeling customers as 'in financial distress' and 'not in financial distress') then you could use some supervised learning algorithm to try to predict whether the unlabeled customers fall in category A or category B.

Unless you acquire some sort of domain expertise or labeled data, this would be a very hard task.


How do we make data Obfuscate or "De-identificate" to make it anonymous and share it publicly? Right now, I am working on preparing a small dataset for release to the public by getting rid of sensitive information.  While working on it, I wondered... what are the best practices of dealing private or sensitive polynomial attributes in a dataset?(*) I have heard to create anonymity or to permute is achieved by De-identification, obscurification,  anonymization.

However, I would like to learn more about this topic in data science/analysis.   I am particularly interested in the packages and concepts that can be use in R.

(*)Well besides the obvious of solutions of completely removing the sensitive attribute or using hashcode encryption.  I am a little bit familiar with how this problem can be complicated by the ability to correlate attributes

I have had a couple of talks about this once and I think you should know about three different concepts.

Different attributes

Attributes can be divided into three different categories. Hard identifiers refer to a specific person, e.g. full name or passport number. A combination of soft identifiers can also be used to identify a single person, because the combination is unique. Sensitive attributes are the ones that infringe privacy and that should be protected.

Measuring privacy

There are several measures for how anonymous your dataset is. These measures help you to focus on the attributes that need to be altered. In order of increasingly stronger requirements you have:

A dataset is $k$-anonymous if every combination of soft identifiers occurs at least $k$ times.
A dataset is $i$-diverse if for every combination of soft identifiers there are at least $i$ different values of the sensitive attributes.
A dataset is $t$-closed if for every combination of soft identifiers, the distribution of sensitive attributes is not more than $t$ (i.e. KL-divergence) different from the global distribution of sensitive attributes.

Privacy protecting modifications

Hard identifiers can be treated with masking or pseudomization. Masking refers to deleting the hard identifier. A less drastic approach is  pseudomization where each records hard identifiers gets a pseudonym (e.g. hash) that is only known by the dataset creator.

Soft identifiers can be treated with suppression, generalisation or randomisation. Suppressing values of soft-identifiers refers to replacing it by a default value (e.g. '*' or the mode). Another strategy, called generalisation, is to create groups of variables that are alike (e.g. 'age 10-20' instead of age 19). A third way is to add randomness to quantitative variables by adding zero-mean noise.

Sensitive attributes can be treated with permutation or randomisation. Permutation refers to permuting the sensitive attributes within every combination of soft identifiers (within a dataset that is at least $2$-anonymous). Like soft identifiers, quantitative sensitive attributes can also be randomised by adding noise.

Ps. as an R-package you could consider sdcMicro, and this guideline.


Matrix properties and machine learning/data mining I'm doing some data analysis in a Statistical Pattern Recognition course using PRML. We analyzed a lot of matrix properties, like eigenvalues, column independence, positive semi-definite matrix, etc. When we are doing, for example, linear regression, we need to calculate some of those properties, and fit them into the equation.

So my question is, my question is about the intuition behind these matrix properties, and their implications in the ML/DM literature.

If anyone could answer, can you teach me what is the importance of eigenvalue, positive semi-definite matrix, and column independence for ML/DM. And possibly, other important matrix properties you think important in study the dataset, and why.

I'd be really appreciated if someone can answer this question.

The importance of a concept in mathematics depends on the circumstances of its application. Sometimes, its importance relies on the fact that it allows you to carry on with what you are doing.

For example, you usually need column independence (independent variables between predictors) because multiple regression will behave badly with highly correlated variables. Even worst, when some of your columns (or rows) are dependent, your matrix is not invertible. Why? Because matrix inversion A^-1 involves the determinant 1/|A|, which is 0 when columns or rows are linearly dependent.

Eigenvalues is a common occurrence in calculations related to maximization/minimization in machine learning. Let's say you are interested in principal component analysis. A very important idea there is dimensional reduction (you have a dataset with many variables and want to reduce the number of variables without losing too much explanatory power.) One solution is to project your data onto a lower dimensional space (e.g. taking your data with 50 variables and reducing them to 5 variables.) Turns out a good projection to use is one that includes as much variation as possible and maximization of this variation results in the eigenvalue equation S u  = λ u.

In other cases, you explicitly include the eigenvalue equation of some quantity of interest because in doing so, you're changing the coordinate system in which you represent the variables. Take the case of a (multivariate) Gaussian distribution in which the argument of the exponent is given by Δ = (x-μ)^T Σ (x-μ). If you consider the eigenvalue equation of Σ, the exponent can be written as Δ = y_1^2 / λ_1 + y_2^2 / λ_2 (in two dimensions) This is the equation of an ellipse only if λ_1 and λ_2 are positive. Therefore, you obtain the following graphical interpretation (Bishop, PRML, p.81):

Positive semi-definite matrices are used as a matter of convenience. They are well-behaved and well-understood. For instance, their eigenvalues are non-negative, and if you remember the previous paragraph, the argument Δ required positive eigenvalues. By now, you can see why some concepts are very popular: You need them for your calculations or they need each other.

I can recommend a couple of books:

Linear Algebra: A Modern Introduction by David Poole
Understanding Complex Datasets: Data Mining with Matrix Decompositions  by David Skillicorn.

The second recommendation is more specialized and requires a decent understanding of the basics, but it is of great help to understand matrix decompositions.


Face Recognition Using HOG+SVM This is my approach

svm = cv2.ml.SVM_create()
svm.setType(cv2.ml.SVM_C_SVC)
svm.setKernel(cv2.ml.SVM_RBF)
svm.trainAuto(data["encodings"], cv2.ml.ROW_SAMPLE,data["names"])

where I get encodings using HOG and names from:

for (i, imagePath) in enumerate(imagePaths):
    # extract the person name from the image path
    print("[INFO] processing image {}/{}".format(i + 1, len(imagePaths)))
    name = imagePath.split(os.path.sep)[-2]

but when I train

  'samples is not a numpy array, neither a scalar'

this error appears in the line:

svm.trainAuto(data["encodings"], cv2.ml.ROW_SAMPLE,data["names"])

I think I need to convert the names to a numpy array.

How can I do that?

It's hard to tell exactly what you're asking, and exactly what your data looks like. Nonetheless, I think you're asking how to convert a list to a numpy.array.

Try this:

np.asarray(data["names"])


Find most representative image I have 1,000 photoshoot-quality images of pianos on white background with very little noise (people in the background, etc). How would I go about finding the piano image that looks most like the others? Alternatively, how would I find five pianos that look most like the "average" piano. I'm particularly looking for python libraries that may be helpful (I'm currently looking at skimage).

Find a low-dimensional emdedding of the images using a nonlinear dimensionality reduction method like autoencoding, then select the image closest to the centroid in the embedding space.


How to calculate accuracy on keras model with multiple outputs? I have a keras model that takes in an image with (up to) 5 MNIST digits and outputs a length and then (up to) 5 digits. I see that model.evaluate() reports accuracies for each of the outputs but how do I determine how good the model is at predicting the numbers? Do I need to write that myself?

It's going to take a bit of engineering - since you have a variable size output, you need to encode the length into the output in order to evaluate the accuracy of the model overall. If instead of outputting "up to 5 digits", you output an array of 5 predictions, where some non-digit (such as -1) operates as indicating that there is no digit present, you can better evaluate your network. If you retrain your network as such (where $X$ is the array of images and $Y$ is an array containing arrays of form $[1,4,3,-1,-1]$, for example), then model.evaluate($X_{test}$,$Y_{test}$) will work as expected.

If you don't want to re-train your network, you can write a simple function to take the output from model.predict($X_{test}$) and encode it into the corresponding format. This encode function will simply go from $[1,4,3]$ to $[1,4,3,-1,-1]$. You can then calculate the  accuracy by sklearn.metrics.accuracy_score($encode$(model.predict($X_{test}$)),$Y_{test}$), where $encode$ is the aforementioned function.


Machine learning dataframe dimension concept vs NumPy dimension From Machine Learning for Absolute Beginners: A Plain English Introduction:

  Contained in each column is a feature. A feature is also known as variable, a dimension or an attribute - but they all mean the same thing.

From here (the supplement file for this book):

In NumPy, each dimension is called an axis.
The number of axes is called the rank.

For example, the above 3x4 matrix is an array of rank 2 (it is 2-dimensional).
The first axis has length 3, the second has length 4.

An array's list of axis lengths is called the shape of the array.

For example, the above matrix's shape is (3, 4).
The rank is equal to the shape's length.

The size of an array is the total number of elements, which is the product of all axis lengths (eg. 3*4=12)

Question: Is the dataframe dimension completely different not related to the NumPy dimension (just same word but describing different concept)?

I am learning Python and Machine learning but familial with R and R dataframe from statistical perspective

The dataframe case refers to the linear algebraic notion of a dimension.  In the NumPy context, it just means the number of axes or rank.


MLP on Iris Data not working but it does fine on MNIST - Keras So I'm a little bit baffled. I have just started working with the Keras framework for Python (which is awesome by the way!).

However just trying a few simple test of neural networks has got me a bit confused. I initially tried to classify the Iris data as it was a small, quick and simple dataset. However when I constructed a neural network for it (4 input dimensions, 8 node hidden layer, 3 node output layer for binary classification of the 3 classes). This however didn't produce any predictive powers at all (100 samples in the training set, a separate 50 in the test set. Both had been shuffled so as to include a good distribution of classes in each).

Now I thought I was doing something wrong, but I thought I'd give the network a quick test on the MNIST dataset just in case. So I used basically the exact same network (other than changing the input dimensions to 784, hidden nodes to 30, and output nodes to 10 for the binary encoded 0-9 output). And this worked perfectly! With a 97% accuracy rate and a 5% loss.

So now I'm not sure why the Iris dataset isn't playing ball, does anyone have any clues? I've tried changing the number of hidden layer nodes and also normalized the X input.

Here's an output I've produced when I've manually run some of the test set through the trained Iris model. As you can see it's basically producing a uniform random guess.

Target: [ 1.  0.  0.]  | Predicted: [[ 0.44635904  0.43874186  0.45729554]]
Target: [ 0.  0.  1.]  | Predicted: [[ 0.44618103  0.43869928  0.45735642]]
Target: [ 0.  1.  0.]  | Predicted: [[ 0.44612524  0.43863046  0.45729461]]
Target: [ 0.  0.  1.]  | Predicted: [[ 0.44617626  0.43870446  0.45736298]]
Target: [ 0.  0.  1.]  | Predicted: [[ 0.44613886  0.43865535  0.45731983]]

And here's my full code for the Iris MLP (the MNIST one is essentially the same)

import numpy as np
import random
from keras.models import Sequential
from keras.layers import Dense, Activation
from sklearn.datasets import load_iris
from sklearn import preprocessing

# Model Layers Defnition
# Input layer (12 neurons), Output layer (1 neuron)
model = Sequential()
model.add(Dense(8, input_dim=4, init='uniform', activation='sigmoid'))
model.add(Dense(3, init='uniform', activation='sigmoid'))

model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy'])

# Iris has 150 samples, each sample has 4 dimensions
iris = load_iris()
iris_xy = zip(iris.data, iris.target)
random.shuffle(iris_xy) # Iris data is sequential in it's labels
iris_x, iris_y = zip(*iris_xy)
iris_x = preprocessing.normalize(np.array(iris_x))
iris_y = np.array(iris_y)

# Encode decimal numbers to array
iris_y_enc = np.zeros(shape=(len(iris_y),3))
for i, y in enumerate(iris_y):
    iris_y_enc[i][y] = 1

train_data = np.array(iris_x[:100]) # 100 samples for training
test_data  = np.array(iris_x[100:]) # 50 samples for testing
train_targets = np.array(iris_y_enc[:100])
test_targets  = np.array(iris_y_enc[100:])

model.fit(train_data, train_targets, nb_epoch=10)

#score = model.evaluate(test_data, test_targets)

for test in zip(test_data, test_targets):
    prediction = model.predict(np.array(test[0:1]))
    print "Target:", test[1], " | Predicted:", prediction

By default sklearn.preprocessing.normalize normalizes samples, not features. Replace sklearn.preprocessing.normalize with sklearn.preprocessing.scale. This will center and scale (to unit variance) every feature.

Also give it more than 10 epochs. Here are learning curves (log loss) for 5000 epochs:

This should end up with an accuracy of about 96%.


Removing percentage signs with R I am learning R and have a tutorial that wants me to remove the percent sign.  The code they want me to use is:

 chocolateData$Cocoa_Percent <- sapply(chocolateData$Cocoa_Percent, function(x) gsub("%", "", x))

My question pertains to what I may be typing wrong, or if there is a different way to do this with R?

The following code worked, thanks for the assistance.

chocolateData$Cocoa.Percent = as.numeric(gsub("[\\%,]", "", chocolateData$Cocoa.Percent))


Clustering with geolocation (lat/long pairs) attributes I am trying to cluster customer behavior based on where they shop given by lat/long pairs. I also have other numeric attributes such as volume, average amount spent, etc. I am considering using HDBSCAN to create clusters. However, I'm not sure whether to feed the dataframe directly to the clustering algorithm or whether I would need to scale/normalize the data.

Is it wise to scale the geolocation pairs? Or would important location information be lost?

Any help would be much appreciated.

https://stats.stackexchange.com/questions/89809/is-it-important-to-scale-data-before-clustering

This page explains a lot. However, in the answer by @Anony-Mousse, he mentions not to scale lat/long pairs. That's good but what about other continuous variables?

Don't treat clustering algorithms as black boxes. If you don't understand the question, don't expect to understand the answer.

So before dumping the data and hoping that magically a desired results comes out, understand what you are doing...

Standardizing latitude/longitude is a horrible idea. These values are angles on a sphere. Linearly scaling these values breaks everything that these values mean. There are many valid transformations - even rotations can be good to get a desirable Mercator protection, for example. But standardizing them, I cannot imagine what this would be good for.
Mixing variables with different meaning rarely works well. It's not just the problem of scale. Scaling often helps as a heuristic to prevent one variable dominating another. It also has the nice property that it doesn't matter if your data were feet or yards. But the need to do so usually means that there is something wrong with your approach at a deeper level: that you apparently are trying really hard to compare apples and oranges... You'll get out some result. It's probably even interesting. But once you try to explain or act on it, you're back to square one: what does it mean if you scale your data this way, and why is that better than the infinitely many alternative ways, infinitely many of which lead to other results?


R: Pam function under cluster package not accepting data set having more than 65536 entries I am running pam function for kmedoid under cluster package in R. My dataset has 1 million entries.
It's throwing error  saying

  Have 1 million observation but not more than 65536 observation allowed

Please suggest the solution

Thanks in advance :)

If you read the package News section you will find:

  pam() now signals an error when there are more than 65536 observational units (whereas it could segfault previously), thanks to a patch from Mikko Korpela, Helsinki.

So it seems there isn't a solution.


Classification of skills based on job ads I have around 1,000 job ads in the filed of IT (in excel file). I want to find the skills which are mentioned in each of ads. and then find the similar jobs based on skills.

My method: I created 12 categories Such as programming skills,  testing skills,  communication skills, network skills, ... . Each advertisement may belong to 3-4 categories. In this case, some said multi-variate classification or Multi label classification is useful. But I don't know how to do this kind of classification in RapidMiner.

1- Does anyone know how to do multi-variate classification or Multi label classification in RapidMiner? or is there another way?

2- Do you recommend "classification" in order to analysis required job skills? or another technique?

3- Is there any better way to classify the skills which are stated in job ads?

I'm new in the field of text mining. Please let me know if you have any idea. Thanks

I don't know about RapidMiner, but for a beginner, Orange seems an awesome open-source tool for the job.

Particularly if you want to do unsupervised learning, i.e. clustering. You'd probably want to preprocess your textual data in some bag of words model and save it as CSV, I guess.

I also hear Orange has a text mining add-on, but I haven't tried it yet.


Is it reliable to use TensorFlow (ML in general) to classify baggage bag tags based on the presence of a green stripe? The images are identical except for the presence of the stripe on the side.
I am trying to use a classify the images into 2 classes: greenStripe, noGreenStripe.

I tried to use tensorflow retrain with a small dataset (~40 pictures in each class and batch size of 8) but the results where really bad. I am afraid to commiting to training using more data as it is time consuming.

What do you suggest? Is there a better approach or does the problem lie in the small training dataset?

The scientific answer would be, it depends.

In case you are using any kind of Deep net, then 40 images is far too little.
It might be helpful to describe your problem setting a little bit more in depth.
Are the bags always in the same place, or do they need to be localized first? These kind of details could help other users in their recommendations.

As a first approach, before you try a deep net or any kind of ML I would try a simple baseline first. Do you know what the exact pixel value of your green stripe is? You could then simply check whether this colour is present at all. This is rather coarse, but I would see how far this gets you and it is good to see whether your ML methods can beat this simple baseline.
Subsequently you could also think of trying to localize the bagtags (in whatever way you like) then cropping it and checking for the presence of this green stripe.


How Calculate Effect (percentage) label of the input variables on the output variable by BernoulliNB a description problem below. I have 10 words like X1 , X2 , X3 , ... , X10
and three Label like short , long , hold.
My problem is that how calculate Effect (percentage) label of the input variables
by BernoulliNB Algorithm.
NB = BernoulliNB()
NB.fit(X_train, y_train)

and how calculate Effect (percentage) label of the input variables

A Naive Bayes model consists of the probabilities $P(X_i|Class)$ for every feature $X_i$ and every label $Class$. So by looking at the parameters of the model one can see how important a particular feature is for a particular class. the opposite could be calculated as well: $P(Class|X_i)$ represents the distribution of the classes given a feature.
Now at the level of individual instances it's not so clear what would be the "effect" of a particular feature: For every class the posterior probability is:
$$P(Class| X_1,..,X_n) = \frac{P(Class)\prod_i P(X_i|Class)}{P(X_1,..,X_n)}$$
You can easily order the features by how much they contribute to the prediction, i.e. the class which obtains the maximum posterior probability (for instance obtain the top 3 features). However you cannot quantify precisely the effect of each feature, because the prediction is not a linear combination of the features.

[Details added following comments]
Due to the NB assumption that features are independent, we have:
$P(Class|X_1,..,X_n) = \prod_i P(X_i|Class)$
$P(Class|X_1,..,X_n) = P(X_1|Class) * P(X_2|Class) * .. * P(X_n|Class)$
From the conditional definition:
$P(Class|X_1,..,X_n) = P(Class,X_1,..,X_n) / P(X_1,..,X_n)$
which gives:
$P(Class,X_1,..,X_n) = P(Class) * P(Class|X_1,..,X_n)$
$P(Class,X_1,..,X_n) = P(Class) * P(X_1|Class) * P(X_2|Class) * .. * P(X_n|Class)$
Now we use the marginal to calculate $P(X_1,..,X_n)$:
$P(X_1,..,X_n) = \sum_j P(Class_j,X_1,..,X_n)$
$P(X_1,..,X_n)  = P(Class_1,X_1,..,X_n) + .. + P(Class_n,X_1,..,X_n)$
So at the end we have $P(Class,X_1,..,X_n)$ and $P(X_1,..,X_n)$, so we can calculate:
$P(Class|X_1,..,X_n) = P(Class,X_1,..,X_n) / P(X_1,..,X_n)$
Note that if you do all these steps you should obtain the same probability for $P(Class|X_1,..,X_n)$ as the one returned by the function predict_proba.
Caution: the functions  feature_log_prob_ and class_log_prior_ don't give you the probability directly, they give you the logarithm of the prob. So you need to apply exponential in order to get back the probability.


Accuracy of word and sent tokenize versus custom tokenizers in nltk The Natural Language Processing with Python book is a really good resource to understand basics of NLP. One of the chapters introduces training 'sentence segmentation' using Naive Bayes Classifer and provides a method to perform sentence segmentation on unseen corpus.

NLTK provides word_tokenize and sent_tokenize. Creating our own tokenizers can help us understand how one works, but in a production environment why would we want a custom tokenizer?  And if I built a custom tokenizer, how could I measure if it was better that NLTK's tokenizer?

Why would we want a custom tokenizer?

Segementation is a very large topic, and as thus there is no perfect Natural Language Tokenizer.  Any toolkit needs to be flexible, and the ability to change the tokenizer, both so that someone can experiment, and so that it can be replaced if requirements are different, or better ways are found for specific problems, is useful and important.

  How could I measure if it was better that NLTK's tokenizer?

Anytime you are trying to quantify performance (ie: better) you will need to first define what is meant by better.  Once this is done, then typically you would perform the using the various methods under measurement, and then compare the results against your definition of better.  A couple of links which discuss these topics:

Performance of Different NLP Toolkits in Formal and Social Text
Optimizing to Arbitrary NLP Metrics using Ensemble Selection


will increasing threshold always increase precision? here precision at threshold 0.85 > precision at threshold 0.90. shouldnt it be the other way round? increasing threshold will reduce False positive and precision will be greater than before?

here precision at threshold 0.85 > precision at threshold 0.90. shouldnt it be the other way round? increasing threshold will reduce False positive and precision will be greater than before?

Precision is $\frac{\text{TP}}{\text{TP}+\text{FP}}$

Both $\text{TP}$ and $\text{FP}$ are reduced when you increase the threshold. If both decrease in proportion to the current precision (i.e. they are spread evenly at each confidence value), then precision will remain the same. Most models on most datasets will tend to increase precision as the threshold increases, at least initially (e.g. moving from 0.5 to 0.6) as false positives may commonly be found as uncertain edge cases with low confidence, i.e. false positives tend to occur more frequently at low confidence, so increasing threshold will exclude a higher ratio of false positives than true positives than the current precision. However, there is no guarantee of that.

The value of precision will vary in practice depending on what the model predicted for each example. If you have a cluster of highly confident false positives, they can cause precision to drop as threshold grows, until they get excluded. The most extreme example would be where the most confident classification is incorrect, in which case the highest possible threshold will score zero precision.


Finding point of interest in time series data I am working on a project where I need to figure out the point of interest in time series data.

From the picture you can probably understand a bit more what I mean.
Basically, imagine this is the electricity consumption of a washing machine. From this, I want to identify these points where the consumption changes drastically. In this way I will be able to identify more or less what the machine is doing.

I only have raw data so I can't use any supervised learning algorithm, I was wondering if there are ways to do it mathematically without spending too much computation time.

Regards and thanks in advance.

Try derivative after either a low-pass filter or smoothing (probably exponential smoothing) to cut down on the noise. Big changes result in a big derivative (up or down).


What are helpful annotation tools (if any) I'm looking for tools that would help me and my team annotate training sets. I work in an environment with large sets of data, some of which are un- or semi-structured. In many cases there are registration that help in finding a grounded truth. In many cases however a curated set is needed, even if it just were for evaluation. A complicating factor is that some of the data can not leave the premise.

We are looking to annotate an object detection task, but I anticipate an image segmentation task, a text classification task and a sentiment detection task in the near future.

What I'm looking for is a system that can help a group make an annotation, preferably in a way that motivates the annotators by showing group progress, relative individual progress and perhaps personal inter annotator agreement.

Doccano is an open source simpler alternative to Prodigy. Its native python via Django. I found it suitable for simple implementations.


Machine Learning for user modelling I have a dataset where each row is a interaction of a user with a content.
I have user's features to represent the user (each user is uniquely represented through user.id):

user.id, user.nationality, user.company, user.role

and content's feature:

content.id, content.type, content.activity..

My goal is to use ML techniques to predict a content (given by a user.id).
My main problem is that, for a new prediction, I have the features of the user, but I do not have the features of the new content.

What you have is a classification problem. Given a user and their features, pick the best content for him.

There are many classifiers that are for this task. I would say that your main problem is that most of your features are categorical and not numerical.

Data preparation

With Sklearn for example, you can use LabelEncoder to transform categorical to numerical.

Below is an example of how to do this. You generate X, which is your features matrix. Each row is a user and the columns are the features.

Here are some dummy users I created. You data structure might be different, so you can try to make it like this, or do some more work on it.

users = [
    {
        'id': 1,
        'nationality': 'american',
        'company': 'xyz',
        'role': 'ceo',
        'content.id': 1001
    },
    {
        'id': 2,
        'nationality': 'american',
        'company': 'abc',
        'role': 'cto',
        'content.id': 1002
    },
    {
        'id': 3,
        'nationality': 'canadian',
        'company': 'fgh',
        'role': 'cto',
        'content.id': 1001
    }
]

Now, you can encode your categories. This means that for every word in your data, you assign a number to it. The LabelEncoder can do this for you. With this, you generate X, which is your feature matrix, where every row represents a user.

from sklearn.preprocessing import LabelEncoder

nationality_encoder = LabelEncoder().fit([user['nationality'] for user in users])
company_encoder = LabelEncoder().fit([user['company'] for user in users])
role_encoder = LabelEncoder().fit([user['role'] for user in users])

X = []
for user in users:
    X.append([
        nationality_encoder.transform([user['nationality']])[0],
        company_encoder.transform([user['company']])[0],
        role_encoder.transform([user['role']])[0]
    ])
print(X)
>> [[0, 2, 0],
>>  [0, 0, 1],
>>  [1, 1, 1]]

Then you can generate y, which are your labels. The content.id of each user. Here I assume you have the user-content pairs like you said you did. This vector has the same length as X. Each row of X is associate to the ordered values in y. So the user in X[2] has the content.id in y[2].

y = [user['content.id'] for user in users]
print(y)
>> [1001, 1002, 1001]

Classifier

Now you have your X and y, your features and labels.

Not all classifier will perform well with categorical data, but I would suggest you start with a DecisionTreeClassifier.

Here is an example:

from sklearn.tree import DecisionTreeClassifier

decision_tree = DecisionTreeClassifier()
decision_tree.fit(X, y)

New predictions

Now that you have a trained tree, you can predict the content.id for users that do not have it.

Suppose you have this new user:

new_user = {
        'id': 999,
        'nationality': 'american',
        'company': 'fgh',
        'role': 'ceo'
    }

You can apply the same transformation to get the X for him. However, now you do not know y, which is what you are trying to find out.

new_user_X = [[
    nationality_encoder.transform([new_user['nationality']])[0],
    company_encoder.transform([new_user['company']])[0],
    role_encoder.transform([new_user['role']])[0]
]]

Use the decision_tree you created to make a prediction. In this dummy example, the new_user should have content.id == 1.

new_user_content = decision_tree.predict(new_user_X)
print(new_user_content)
>> [1]


Calculating entropies of attributes Can you please show the step by step calculation of Entropy(Ssun)?  I do not understand how 0.918 is arrived at.

I tried but I get the values as 0.521089678, 0.528771238, 0.521089678 for Sunny, Windy, Rainy.

I was able to calculate the target entropy (Decision) correctly as = -(6/10)*log(6/10) + -(2/10)log(2/10) + -(1/10)log(1/10) + -(1/10)log(1/10) = 1.570950594

I am totally stuck at the next step.  Request your help.

Reference: http://www.doc.ic.ac.uk/~sgc/teaching/pre2012/v231/lecture11.html
Please search for "The first thing we need to do is work out which attribute will be put into the node at the top of our tree:" to reach the line I am referring to.

Consider the formula for Entropy:

$E(S) = \sum_\limits{i=1}^{n}-p_{i}\log_2p_{i}$

Expanding that summation for the four concept decision attributes for your problem gives

$E(S) = -p_{cinema}\log_2\left(p_{cinema}\right) - p_{tennis}\log_2\left(p_{tennis}\right) - p_{stayin}\log_2\left(p_{stayin}\right) - p_{shopping}\log_2\left(p_{shopping}\right)$

There are three observations where Weather=Sunny. Of those three, one has Decision=Cinema and two have Decision=Tennis. So for Weather=Sunny you have $p_{cinema}=\frac{1}{3}$, $p_{tennis}=\frac{2}{3}$, and other decision probabilities are zero. Plugging those values into the equation above gives

$
\begin{align}
E(S_{Sun}) &= -\frac{1}{3}\log_2\left(\frac{1}{3}\right)-\frac{2}{3}\log_2\left(\frac{2}{3}\right) - \left(0\right)\cdot\log_2\left(0\right) - \left(0\right)\cdot\log_2\left(0\right) \\
           &= 0.38998 + 0.52832 + 0 + 0 \\
           &= 0.918
\end{align}$


Extracting structured data from semi structured data I want to use machine learning and NLP to convert semi-structured data in text files to structured data by predicting the patterns in the files and splitting the fields for example if I have a text file that looks like this :
Input :
2021565267MALL1ETAGE ZARA1st FLOOR 2345561
2022565267MALL2ETAGE ZARA1st FLOOR 2345561
2022565267ANFAPLACE2ETAGECOFEESHOP2345561
20225652634ANFAPLACE2ETAGE 2345561

Desired Output :
2021565267,MALL1ETAGE ZARA1st FLOOR,2345561
2022565267,MALL2ETAGE ZARA1st FLOOR,2345561
2022565267,ANFAPLACE2ETAGECOFEESHOP,2345561
20225652634,ANFAPLACE2ETAGE,2345561

The semi-structured files are not fixed-width so we can not just add col specification in pandas like this ( it can work for the first line for example ) :
col_specification =[(1, 10),.... ]

One of the approaches that I found online is to make a dictionary based on the occurrences of the words in the semi-structured file will that work in this situation if so how can I implement something like that?

I am making the following assumptions:

The first column of numbers is the first few digits/numbers in each line
The last column of numbers is the last few digits/numbers in each line
So, in the second column, texts like MALL1ETAGE ZARA1st FLOOR should not have numbers in the front or end. So, 20215652671MALL1ETAGE ZARA1st FLOOR 2345561 will be interpreted as 20215652671, MALL1ETAGE ZARA1st FLOOR, 2345561 and NOT 2021565267, 1MALL1ETAGE ZARA1st FLOOR, 2345561

To this end, you can use the following that makes use of regular expression to capture the first and last group of numbers:
import re

re.findall("^([0-9]+)(.*?)([0-9]+)$", "2021565267MALL1ETAGE ZARA1st FLOOR 2345561")
# [('2021565267', 'MALL1ETAGE ZARA1st FLOOR ', '2345561')]

Applying strip() to each of the result removes the trailing whitespace to get the output you desire.


Covariance matrix in linear regression I have read about linear regression and interpreting OLS results i.e coefficients, t-value, p-value. But unable to find any material related to covariance matrix in linear regression.

I was reading about assumptions in linear regression, came across the term heteroscedasticity and was researching about its consequences. So while going through one of the answer on Stack Exchange about consequences of Heteroscedasticity I came across the term covariance matrix. Here's a link to that StackExchange

What is covariance matrix and how should one interpret it?

One of the OLS assumptions is the zero conditional mean assumption, which states that $E[u|X]=0$, so that errors average out to 0.

Another assumption is homoskedasticity, which means that there is no (auto)correlation in the residuals $E[u u'|X]=\sigma I$. So the covariance matrix (sometimes also called variance-covariance matrix) is:

$$ \sigma I = \sigma \left[ \begin{array}{rrrr}
1 & 0 & \cdots & 0\\
0 & 1 & \cdots & 0 \\
 \vdots & \vdots & \vdots & \vdots \\
0 & 0 & \cdots & 1 \\
\end{array}\right] = \Omega .$$

The important thing is that all elements in the matrix (apart of the diagonal elements from top left to bottom right) are zero, which means "no correlation between residuals". Also there is a constant variance equal to $\sigma$.

In case $ \Omega \neq \sigma I$, you face heteroscedasticity and you would need to "model" $\Omega$, e.g. by "Feasible Generalized Least Squares" (FGLS).

Say you have a linear model  $y=X\beta+u$ with $E(u^2)=exp(Z\gamma)$. Here $exp(Z\gamma)$ is an example for a scedastic function which describes the conditional variance. In order to obtain consistent estimates for $\gamma$, one need to obtain consistent estimates for $u$ in the first place. This can be done by using OLS to obtain $\hat{\beta}$ and $\hat{u}$.

Based on this, one can run an auxiliary linear regression $ \log \hat{u}^2 = Z \gamma + v $ to get an estimate for $\hat{\gamma}$. These estimates can be used to compute  $\hat{\omega} = (\exp(Z\hat{\gamma}))^{1/2}$. Finally, FGLS estimates of $\beta$ are obtained by OLS with weights $\hat{\omega}$ which is called feasible weighted least squares.

See Davidson/MacKinnon: "Econometric Theory and Methods", Ch. 7.4.


Will the MAE of testing data always be higher than MAE of training data? On the Kaggle Course Page the chart below shows that MAE of testing data is always higher than MAE of training data. Why is this the case? Is it only limited to DecisionTreeRegressor model? Or the graph is wrong and in practice the MAE of testing can be lower than MAE of training?

Test MAE of a model can be lower than Train MAE when it falls under below cases:

It is possible when you have not sampled the data or split the test train data perfectly.
It is possible when your test data is small and its not a good representative of train data, then there may or may not be a case when for that test data it behaves good and gives low error.
There can be a case when you use regularization and your train data move away from overfitting i.e. increased train error but generalizing good on test data i.e. low train error.


Need some info regarding string matching algorithms? Let me explain a scenario to better explain my question,

Assume I am working in a credit-card related company in which people uploads their receipts every month, I want to check if that person bought fruits or not. Let's assume we used OCR to extract only the names of items bought and stored in a list.

First thing I did is web scrape all the names of fruits found everywhere and I stored the names of each of them in a big text file.

Now I want to know how can I match/lookup and make a decision that person bought fruits.

1) Any search/match algorithms that works on huge amounts of data.

I'm just looking for advice on what to implement in this type of scenario. Thanks in advance.

Try the simplest approach first - deterministic check looking for intersection overlap between the set of fruit names and the set of items bought.

Set comparisons are scalable because the look-up time for each item is constant.

If scaling is an issue with regular set membership check, bloom filter is an option.


What type of machine learning are able to return feature importance? I've been using random forest to calculate feature importance however I've asked myself why random forest? so my question became what other machine learning models can calculate feature importance? and is there pros and cons between each models?
I've did some research and here are some that I've found

Statistical correlation score -> calculating each X variable with Y. (statistical method)
coefficients in linear models -> Logistic, linear regression, regularizations(L1, L2, elastic net)
Tree based algorithms  -> ex: decision tree, RF, boosting algorithms
permutation importance score

I would like to know if there are any other methods.
Thanks in advance!

You can calculate a feature importance for any classifier. Take a look at lime or shap
Shap unifies seven different methods (cited from the shap GitHub page):

1 LIME: Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. "Why should i trust you?: Explaining the predictions of any classifier." Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2016.

2  Shapley sampling values: Strumbelj, Erik, and Igor Kononenko. "Explaining prediction models and individual predictions with feature contributions." Knowledge and information systems 41.3 (2014): 647-665.

3  DeepLIFT: Shrikumar, Avanti, Peyton Greenside, and Anshul Kundaje. "Learning important features through propagating activation differences." arXiv preprint arXiv:1704.02685 (2017).
4 QII: Datta, Anupam, Shayak Sen, and Yair Zick. "Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems." Security and Privacy (SP), 2016 IEEE Symposium on. IEEE, 2016.

5  Layer-wise relevance propagation: Bach, Sebastian, et al. "On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation." PloS one 10.7 (2015): e0130140.

6  Shapley regression values: Lipovetsky, Stan, and Michael Conklin. "Analysis of regression in game theory approach." Applied Stochastic Models in Business and Industry 17.4 (2001): 319-330.

7  Tree interpreter: Saabas, Ando. Interpreting random forests. http://blog.datadive.net/interpreting-random-forests/

Depending on the classification algorithm you could calculate the feature importance by using Gini or gain based methods, too ..


How to transform entire pandas data frame in one hot representation? I want all the columns one hot encoded without the need of listing out the columns or apply one hot encode one by one. I know how to do it one column then another.

You can use:: pandas.get_dummies

get_dummies will only convert string columns and will keep numerical columns as it is. You can first convert categorical columns into string type and then apply get_dummies.

concated_dataset['1stFlrSF'] = concated_dataset['1stFlrSF'].astype("string")
pd.get_dummies(cacated_dataset)


Depth of the first pooling layer outcome in tensorflow documentation Let's say that we have a CNN with two convolutional layers (https://www.tensorflow.org/tutorials/layers). My question regards the dimension of the tensor, which is the output of the pooling layer 1.

In the first convolutional layer, we apply $32$ filters to the input image (let's say that the output will be $28\times 28 \times 32)$, so as far as I can understand we will get $32$ separate feature maps, because of the number of filters.

In the next step, we can apply an activation function, which does not change the dimensionality.

The $\max$ pooling layer takes as input a tensor of $28\times 28\times 32$ and the output is going to be a $14 \times 14 \times \color{red}{1}$ tensor (according to the link above).

I cannot understand the unit as the depth, since we apply $32$ filters and we apply the $\max$ pooling layer to every feature map. So, why is the output tensor $14 \times 14 \times 1$?

According to my understanding the input in the second convolutional layer should be a $14 \times 14 \times 32$ tensor. Probably, I am missing something here.

I kept spinning my head around this question because I seem to come along the same conclusion as you do. However, it appears to be a mistake in the documentation.

https://stackoverflow.com/questions/43453712/what-is-output-tensor-of-max-pooling-2d-layer-in-tensorflow


Multilabel classifcation in sklearn with soft (fuzzy) labels I have a model which is trained in sklearn on a 5-way classification problem, which performs relatively well (there are kNN and SVM versions, and both reproduce a test set with high accuracy).

When the model is applied in "real life", it is highly likely that many samples will contain linear combinations of multiple classes.  So a sample may be 70% class A and 30% class B.

Much of what I have read about multilabel classification in sklearn relates to problems which don't fit this paradigm well, most of them are "tagging" type problems such as movie genre classification.  Is there a way to apply my SVM/kNN models to this type of problem?  I would prefer to only train on single-class examples but can modify the training set to create some multi-class samples too.

It seems I could work this by simply doing an indivdiual binary classifier for each class.  However, this wouldn't give me the relative strength of each label, i.e. the linear coefficient.  Is that possible?

For example, for a 3-class classification, we want to train with a label like $A$, which is one-hot encoded as $(1, 0, 0)$, and also with a fuzzy label like $(0.8, 0.2, 0)$. In that case, kNN and SVM of sklearn does not support fuzzy labels.

However, we can use sklearn's MultiOutputRegressor that extends a one-output Regressor such as Support Vector Regression (SVR) to multiple outputs. It is worth noting that neural networks are a natural fit for this type of label since they readily work with numerical vectors as labels.

Here is a code that goes through different types of labels for kNN, SVC (multi-class SVM), and MultiRegression SVR:

import sklearn
import pandas as pd
from sklearn.svm import SVC, SVR
from sklearn.model_selection import KFold, cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.multioutput import MultiOutputRegressor
import numpy as np

N = 1000
split = int(0.8 * N)
folds = 5
seed = 1234

# Data
np.random.seed(seed)
feature_1 = np.random.normal(0, 2, N)
feature_2 = np.random.normal(5, 6, N)
X = np.vstack([feature_1, feature_2]).T

Y_label = np.random.choice(['A', 'B', 'C'], N)

Y_one_hot = pd.get_dummies(Y_label).values

smooth_filter = np.array([0.01, 0.98, 0.01])
Y_fuzzy = np.apply_along_axis(
    lambda m: np.convolve(m, smooth_filter, mode='same'), axis=1, arr=Y_one_hot
)

kfold = KFold(n_splits=folds, random_state=seed)

kNN = KNeighborsClassifier(n_neighbors=3)
svc = SVC()
svr = SVR()
multi_svr = MultiOutputRegressor(estimator=SVR())

knn_label = np.average(cross_val_score(kNN, X, Y_label, cv=kfold))
knn_one_hot = np.average(cross_val_score(kNN, X, Y_one_hot, cv=kfold))
try:
    knn_fuzzy = np.average(cross_val_score(kNN, X, Y_fuzzy, cv=kfold))
except ValueError:
    print('kNN: fuzzy classes are not supported')
svc_label = np.average(cross_val_score(svc, X, Y_label, cv=kfold))
try:
    svc_one_hot = np.average(cross_val_score(svc, X, Y_one_hot, cv=kfold))
except ValueError:
    print('SVC: vector is not supported')
try:
    svr_one_hot = np.average(cross_val_score(svr, X, Y_one_hot, cv=kfold))
except ValueError:
    print('SVR: vector is not supported')
multi_svr_one_hot = np.average(cross_val_score(multi_svr, X, Y_one_hot, cv=kfold, scoring='neg_mean_absolute_error'))
multi_svr_fuzzy = np.average(cross_val_score(multi_svr, X, Y_fuzzy, cv=kfold, scoring='neg_mean_absolute_error'))

print('sklearn version', sklearn.__version__)
print('Y example: ',
      "label: ", Y_label[0],
      ", one hot: ", Y_one_hot[0, :],
      ", fuzzy: ", Y_fuzzy[0, :])
print('kNN label: ', knn_label)
print('kNN one hot: ', knn_one_hot)
print('SVC label: ', svc_label)
print('MultiSVR one hot: ', multi_svr_one_hot)
print('MultiSVR fuzzy: ', multi_svr_fuzzy)

Output:

kNN: fuzzy classes are not supported
SVC: vector is not supported
SVR: vector is not supported
sklearn version 0.19.1
Y example:  label:  B , one hot:  [0 1 0] , fuzzy:  [0.01 0.98 0.01]
kNN label:  0.321
kNN one hot:  0.254
SVC label:  0.332
MultiSVR one hot:  -0.4066160996805417
MultiSVR fuzzy:  -0.3970780923514713

Although kNN does not throw an exception for one-hot encoded labels, accuracy 0.254 shows that it does not work correctly with the vector.

Also, Negative Mean Absolute Error is reported for MultiSVR since the task is understood as regression. Score accuracy can only be used after changing the fuzzy labels and predictions back to a label.


Output model from GLM in R I had generate in Ra logistic model using glm using binomial as family, but each session that I started in R the variable that I used to store the glm output gives me another output.
Why this happen? Why R didn’t save/store the output from glm in the variable that I created for.
I’ve been googling for an answer, but I didn’t find anything, so maybe you could help me to understand.
Thanks in advance!

Your question doesn't describe exactly what you did so it's impossible to tell you what happens, but anyway relying on R saving the workspace is not the best option to save your model. It would be much safer to actually save the model to a file, as described in this question for instance:

> set.seed(345)
> df <- data.frame(x = rnorm(20))
> df <- transform(df, y = 5 + (2.3 * x) + rnorm(20))
> ## model
> m1 <- lm(y ~ x, data = df)
> ## save this model
> save(m1, file = "my_model1.rda")
>
> ## a month later, new observations are available:
> newdf <- data.frame(x = rnorm(20))
> ## load the model
> load("my_model1.rda")
> ## predict for the new `x`s in `newdf`
> predict(m1, newdata = newdf)
        1         2         3         4         5         6
6.1370366 6.5631503 2.9808845 5.2464261 4.6651015 3.4475255
        7         8         9        10        11        12
6.7961764 5.3592901 3.3691800 9.2506653 4.7562096 3.9067537
       13        14        15        16        17        18
2.0423691 2.4764664 3.7308918 6.9999064 2.0081902 0.3256407
       19        20
5.4247548 2.6906722


Alternatives to Logistic Regression I have age, gender, height, weight and some other similar parameters of 15000 subjects. I also have one column showing if they had a medical condition (present in about 20% subjects). I now want to analyze the factors which are most contributing to the medical condition. The usual test for this is Logistic regression analysis. However, that assumes that the relationships are linear. What other machine learning tests can I apply for this purpose with this kind of data? Thanks for your insight.

Any classifier that naturally provides feature importance can be applied here. For example, take a look at Decision Trees and Random Forrest.

For other algorithms that do not provide such information (eg KNN and Neural Nets), you may take a (forward/backward/bidirectional) feature selection. For example, chose a subset of features, train a model and evaluate it with any preferred metrics (better to use bootstrapping or cross-validation). Choose another subset if features and try again. After several trials, you can find the subset with the best prediction power. This is the set of the mist contributing features.

Note that different techniques/measures may give you different results.


How do I fit a curve into non linear data? I did an experiment in my Uni and I collected data $(ω,υ(ω))$ modeled by the equation:

$$ v(ω)=\frac{C}{\sqrt{(ω^2-ω_0^2 )^2 +γ^2 ω^2}} $$

where $ω_0$ is known. Do you know how can I fit a curve to my data $(ω,υ(ω))$ ? and how to find the parameter $ γ $ through this process ?

Here is a simple solution that you can compute yourself.
It is trivial to solve your equation for $γ$ as a function
of the other values.  We get
$$
γ = \sqrt{(\biggr( \frac{C}{v(ω)} \biggl) ^2 - {(ω^2-ω_0^2)^2}) / {ω^2}}
$$

It is almost true that you can substitute in your values and get
several estimates for $γ$, which you can simply average to get a good estimate.
I say almost true because as $ω$ approaches zero, both the numerator and denominator
go to zero, so the calculation becomes unstable. As long as you stay away from
$ω=0$, this should give good results.   @Esmailian provided some nice data in his
answer. Using his data I get the following estimates of $γ$.

6.012568  5.999664  5.994524  5.991173  6.009153  5.996748  6.019433
5.998072  6.001754  5.993656  6.002409  5.985938  5.996226  5.998363
6.474674  6.525602 14.140690  5.999124  6.000368  5.998427

Three of his $ω$'s are less than 1. These are the ones that give the estimates of $γ$

6.474674  6.525602 14.140690

The mean of the remaining estimates is 5.999859. As long as you have enough
$ω$'s that are not near zero,  this should provide a good estimate of $γ$.


Coreference Resolution for German Texts Does anyone know a libarary for performing coreference resolution on German texts?

As far as I know, OpenNLP and Stanford NLP are not able to perform coreference resolution for German Texts.

The only tool that I know is CorZu which is a python library.

Here is a couple of tools that may be worth a look:

Bart, an open source tool that have been used for several languages, including German. Available from the website
Sucre is a tool developed at the University of Stuttgart. I don't know if it's available easily. You can see this paper about it.


Accuracy of KFold Cross Validation for Neural Network I have a neural network that Im evaluating using 10 -Fold cross validation. The validation accuracy for a fold changes alot during training in the range of -+10%

So for example the validation accuracy of a fold would range between 80% and 70%.

My question is which number should I consider to be this fold's accuracy.

Should I just take the maximum validation accuracy reached while training or should I just run the training for a certain number of epochs and take the last number (The second approach's result will depend on luck)?

Thanks

For some context, the entire point of running K-Fold cross-validation to compute an estimate of the performance. If the obtained values vary a lot, it just means that your estimate is less precise (i.e. has a larger standard deviation)

In a neural network setting, your network evolves, so obviously, averaging all folds might give you a less accurate estimate if the network learned a lot in the last few iterations for instance.

Typically, you would always take the fold at the end of training, since this is the performance you currently have.

However, it is good to note that analyzing the variance of your performance within a short window can help you understand whether or not your model has converged or whether it is still "exploring".


How can I detect anomalies/outliers in my online streaming data on a real-time basis? Say, I've a huge set of data(infinite in size) consisting of alternating sine wave and step pulses one after the other. What I want from my model is to parse the data sequence wise or point wise and the first time it parses a sine wave and starts facing the step pulses raise an alert as an outlier but as it goes on parsing the data it must recognise the alternating sine and step pulses and treat them as normal pattern. But then if it faces something out of this trend it must treat them as outlier, however if that new pattern repeats constantly it must treat them as normal again. In other words, my model must "remember" what it saw in the past to some extent to predict what is "normal" in the near future and on the basis of that detect anomalies in my constantly streaming data.

I've tried implementing the conventional stateless LSTM to achieve my requirements but LSTM being a supervised learning process needs an initial training and always predicts based on that initially given data. So what happens is if the pattern it recognised while training initially deviates in the test phase it always treats the pattern in the test phase to be an outlier irrespective of how many times it is repeating. Simply put, it fails to update itself with time.

I've gone through relevant papers on 'Anomaly Detection of online streaming data' and found HTM implemented by Numenta and tested on NAB benchmark is the best solution in this respect but I am looking for something open source and absolutely free to use.

Being a newbie in this field, any existing open source implementation will be highly appreciated as writing something from scratch is not preferred but if required that'll be my last option.

There are two well known algorithms called Isolation Forest and One-Class SVM for outlier detection. You will find implementations of these in Sckikit learn.

Doing a search for "Anomaly Detection" on github, there seem to be entries to the NAB competition available publicly eg. nareshkumar66675/Numenta. This one has a Jupyter notebook which mainly uses Scikit learn and some custom, but simple, feature engineering. They may serve your purpose. Although the author of this one has not included Licensing information, it seems simple enough to re-implement.

However, as I understand it, the NAB datasets are more "time series" detection, i.e. a signal is an anomaly if it is very different from previous / recent values. It does not have any notion of patterns in the data, as sine pulses after step pulses, and does not include learning larger patterns as the dataset grows in size.

I'm not aware of algorithms solving your specific problem, though they might well exist in the literature. The key issue in your problem is that you cannot predict if a long sequence is an anomaly until you've seen enough data. It may suffer from combinatorial explosion.

The sines and pulses of your problem can be replaced with 0s and 1s, so your problem is one of detecting patterns in strings. Genomics is concerned with patterns in DNA, so that  body of work may have what you need. (Note that is very different from Genetic algorithms)

There is an older set of algorithms called variously, Market Basket Analysis, the Apriori algorithm or Association Set Mining which has the flavor of increasing set size, but not anomaly detection. See this video explaining it. Apriori creates sets of items commonly bought together. When you have small amounts of data, you can reliably create only small patterns. As the amount of data increases you can create larger patterns.


Resnet 50 as a backbone of Unet I want to use a pre trained Resnet 50 as a backbone for Unet model.
But the issue is resnet 50 is expecting the size of image as 197 x 197 3D channel but the image of mine is 128 X 128 x 1D channel. Is there any keras or tensorflow implementation on those.

When you import ResNet50 import in this way:

image_input=Input(shape=(512, 512, 3))
model = ResNet50(input_tensor=image_input, include_top=False)

After that, you can work with your desired image_size


Difference between Non linear regression vs Polynomial regression I have been reading a couple of articles regarding polynomial regression vs non-linear regression, but they say that both are a different concept. I mean when you say polynomial regression, it, in fact, implies that its Nonlinear right. They why there is a difference in the Data Science world regarding both the concepts?

the difference is probably easily seen with an example. Linear regression assumes a form $$f(x, \beta) = \beta_0 + \beta_1 x_1 + \cdots + \beta_n x_n$$ with some covariates $x_i$ and some parameters to estimate $\beta_i$. An example of non-linear regression would be something like $$f(x,\beta) = \frac{\beta_1 x_1 + \beta_2 x_2}{\beta_3 x_3 + \cdots + \beta_n x_n}.$$ Essentially you are assuming your model to be of a nonlinear form. Polynomial regression on the other hand is a fixed type of regression where the model follows a fixed form $$f(x, \beta) = \beta_0 + \beta_1 x + \beta_2 x^2 + \cdots + \beta_n x^n$$ which is a nonlinear function, however it is still linear in the parameters $\beta$ you are trying to estimate.

That is to say,

Polynomial regression is non-linear in the way that $x$ is not linearly correlated with $f(x, \beta)$; the equation itself is still linear.

In the other hand, non-linear regression is both non-linear in equation and $x$ not linearly correlated with $f(x, \beta)$.


How to combine GridSearchCV with Early Stopping? I'm a beginner in machine learning and want to train a CNN (for image recognition) with optimized hyperparameter like dropout rate, learning rate and number of epochs.

The optimal hyperparameter I try to find via GridSearchCV from Scikit-learn.
I have often read that GridSearchCV can be used in combination with early stopping, but I can not find a sample code in which this is demonstrated.

With EarlyStopping I would try to find the optimal number of epochs, but I don't know how I can combine EarlyStopping with GridSearchCV or at least with cross validation.

Can anyone give me a hint on how to do that, it would be a great help?

My current code looks like this:

def create_model(dropout_rate_1=0.0, dropout_rate_2=0.0, learn_rate=0.001):
    model = Sequential()
    model.add(Conv2D(32, kernel_size=(3,3), input_shape=(28,28,1), activation='relu', padding='same')
    model.add(Conv2D(32, kernel_size=(3,3), activation='relu', padding='same')
    model.add(MaxPooling2D(pool_size=(2,2)))
    model.add(Dropout(dropout_rate_1))
    model.add(Dense(128, activation='relu'))
    model.add(Dropout(dropout_rate_2))
    model.add(Dense(10, activation='softmax'))
    optimizer=Adam(lr=learn_rate)
    model.compile(loss='categorical_crossentropy', optimizer=optimizer,
                             metrics=['accuracy'])
    return model

model = KerasClassifier(build_fn=create_model, epochs=50, batch_size=10, verbose=0)
epochs = [30, 40, 50, 60]
dropout_rate_1 = [0.0, 0.2, 0.4, 0.6]
dropout_rate_2 = [0.0, 0.2, 0.4, 0.6]
learn_rate = [0.0001, 0.001, 0.01]
param_grid = dict(dropout_rate_1=dropout_rate_1, dropout_rate_2=dropout_rate_2,
                        learn_rate=learn_rate, epochs=epochs)
grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=-1, cv=5)
grid_result = grid.fit(X, y)

Just to add to others here. I guess you simply need to include a early stopping callback in your fit().

Something like:

from keras.callbacks import EarlyStopping
# Define early stopping
early_stopping = EarlyStopping(monitor='val_loss', patience=epochs_to_wait_for_improve)
# Add ES into fit
history = model.fit(..., callbacks=[early_stopping])


A the end of a big DS project, should I make trained models available on GitHub? I almost completed two big Data Science personal projects based on Deep Learning. They are the fanciest models I've implemented up to now, and I'm pushing all my code on GitHub.

Do you advice to upload trained models too? Or should I let other users run the code and get their own?

What do you do? What are pros and cons?

From what I've seen on Github while looking for open source projects is that people usually do both.

You can have a section where one loads the models and runs the inference, and another section where you let the user train the models from scratch using your code.

I recommend doing this since some people do not want to retrain the model, especially if it's memory consuming (such as computer vision problems), where they only need to see the same results that you had. People who need to train models usually have a different need than what you had in your results, therefore you should let them tailor the results for their needs.


What are the differences between multivariate data visualizations and multidimensional visualizations? I believe this should be the correct place for this topic since it didn't seem to fit naturally in the math stackexchange.

I am writing an essay about data visualization, but I am having trouble nailing down some terminology. I used data from the OECD to investigate trends in the data about OECD member countries. I created a graph in Tableau using data from the OECD that depicts the GDP per capita, average # of years spent in education system, satisfaction score as reported by the member country citizens, and a "Feel Safe" score as reported by the member country citizens (a percentage of the surveyed population who said they would feel safe walking home alone at night). Click the following link to download the full OECD data set I used.

I am confused about whether to label my graph as a multivariate visualization or a multidimensional visualization.

The core of my confusion comes from someone claiming that my graph is only two-dimensional since it can be displayed on a page. However, "dimensions" usually refers to the number of independent variables in a data set and the data displayed in my graph contains four independent variables.

Could someone please tell me what my graphic should be categorized as, and possibly explain some of the differences between multivariate and multidimensional data/visualizations (if any)?

I don't think there is a well-defined (or well enough defined) terminology for that.

Multi-variate means that it involves multiple (dependent or independent) variables. Which is pretty clear.

Multi-dimensional in mathematical sense means that the data points are from a multi-dimensional space (typically vector space). This is exactly the same as multi-variate. In this sense, your visualization is both multi-variate and multi-dimensional. And both the size of the markers and the color represent individual dimensions or variables, which is then equivalent. Also, time would be an individual dimension. Personally, I would go with this interpretation of the term multi-dimensional.

If you want to restrict the term multi-dimensional to spatial dimensions of the plot, then size and color of markers would not be dimensions in that sense. I personally find this interpretation confusing. Time then also would not be a dimension in this sense, unless you represent it by a 3rd axis in the graph. This, again, is confusing as many people actually often put time into the same category as spatial dimensions.

So, all in all I would stick to the mathematical interpretation of dimensions here so that multi-variate and multi-dimensional will be equivalent.


A situation that validation accuracy is higher than test accuracy at its minimum loss region This case would be an example.
Around epoch = 800, we have the validation loss(orange line) reaches its minimum region.

Should we record the accuracy of the model as an average value from epoch 780 to 800? Or shall we take the accuracy after it became an almost straight line (after epoch = 1200)?

If we record the accuracy at the minium point of validation loss. The accuracy may not be the highest. For example, at epoch = 800, val_loss = 0.627,val_accuracy = 0.783. At epoch=909, val_loss = 0.624, val_accuracy = 0.761

what is the best way to do it for such a situation?
Thanks

Assuming the goal is to give the true performance of the model, then it should be the performance obtained by applying the final model on the test set. The fact that the performance on the validation set can be observed during training is irrelevant.
So the question becomes: which model should you select based on observing the performance on the validation set during training? You can select the model which gives the maximum accuracy, but the real performance is obtained when you evaluate it on a fresh test set.
The second question depends on which performance measure you want to optimize: if you want to optimize accuracy, then you should pick the model which obtains the highest accuracy.


Calculate regression coefficients for individuals (low sample size regression)? Is there a way to calculate the regression coefficients for individuals instead of just a group resp. calculating regression coefficients for a very small sample size?

Background

My goal is to test different ways to cluster/segment a group of people. In the domain in question the best variables to use for segmentation would be variables that best express the importance an individual places on certain attributes.

However I do not have direct measurement of that importance. What I do have is an indirect measurement for the whole sample based on a shapley value regression of attribute ratings and final respondent choice.

Clearly put, I asked a couple of people how they would rate the attributes of a given brand and which brand they would choose in the end.

Now I wonder whether it is possible to obtain this indirect importance measurement also on an individual level which would mean either:

Extracting individual coefficient estimators from the overall regression model ( I don't believe that is possible)
Calculating a regression for each individual (Don't know how to do that hence the question)

Problem and what I tried

I work in R and I do know how to fit a grouped linear model. As all my respondents rated two brands I do have n = 2 per group.

However fitting a normal grouped linear model results in unusable results where almost all coefficients are NA and one or two equal 1. Additionally I would prefer to fit a relative importance or shapley value regression but the relaimpo package throws an error complaining about too few observations.

What other avenues could I pursue?

Update:

I have been using this code so far:

df %>%
  group_by(i_TAN) %>%
  do(model = lm(formula = Consideration ~ ., data = .))

@Ben Norris found out that the relaimpo packages has a hard minimum number of observations, so if I wanted to pursue this path I have to up my sample size.

As I only have the data that I have, I pursued a "hacky" solution which I am going to describe for completionists sake. The steps were as follows:

Assign each individual to one of k groups randomly, so that n/k > 4 (with n being the total sample size)
Repeat this step i couple of times,so that each individual is assigned to igroups
Split the total data set into a list of  k dfs along the groups
Train a regression model with relaimpo for each data set
Average the resulting coefficients over all groups an individual was a part of to get approximate "individual coefficients"

This is a rather unscientific process but seemed better to me than simply "duplicating" individuals answers until the minimum sample size was met.

Here is the code I used:

library(dplyr)
library(magrittr)
library(relaimpo)

#create groups
df %>%
  add_count(i_TAN) %>%
group_by(i_TAN) %>%
  mutate(
    g1 = as.integer(runif(n, 1, 51)),
    g2 = as.integer(runif(n, 1, 51)),
    g3 = as.integer(runif(n, 1, 51)),
    g4 = as.integer(runif(n, 1, 51))
  ) %>%
  {.} -> df

# Create all dfs
df %>%
  select(i_TAN,g1,g2,g3,g4) %>%
  gather(nam,group,-i_TAN) %>%
  distinct() %>%
  select(-nam) %>%
  left_join(df, by = "i_TAN") %>%
  select(1:35) %$%
  split(.,group) %>%
  {.} -> list_of_df

lapply(list_of_df, function(x) { x["group"] <- NULL; x }) %>%
  lapply(function(x) { x["i_TAN"] <- NULL; x }) %>%
  {.} -> list_of_modelDF

# Fit all models
lapply(list_of_modelDF,function(x){lm(Consideration~.,data = x)}) -> list_of_reg

lapply(list_of_reg,function(x){relaimpo::calc.relimp(object = x, type = c("lmg"), rela = TRUE)}) -> list_of_relaimpo
```


How do we solve this logistic regression question? Can an exact answer for this question be found? By intuition, I think the answer is 0. But can someone explain the steps on how to solve this question?

It can be found, assuming a proper learning rate, a suitable threshold, and binary cross-entropy cost, since it translates this into a convex problem, in which we have one global optimum. We don't have closed form solution for logistic regression, but through gradient descent we can get to this optimum arbitrarily close. I'd suggest running a logreg from scikit-learn or a familiar library.


On-line random forests by adding more single Decisions Trees A Random Forest (RF) is created by an ensemble of Decision Trees's (DT). By using bagging, each DT is trained in a different data subset. Hence, is there any way of implementing an on-line random forest by adding more decision tress on new data?

For example, we have 10K samples and train 10 DT's. Then we get 1K samples, and instead of training again the full RF, we add a new DT. The prediction is done now by the Bayesian average of 10+1 DT's.

In addition, if we keep all the previous data, the new DT's can be trained mainly in the new data, where the probability of picking a sample is weighted depending how many times have been already picked.

There's a recent paper on this subject (On-line Random Forests), coming from computer vision. Here's an implementation, and a presentation: Online random forests in 10 minutes