Correlating activity between entities using Python I have some data that contains IDs along with an associated column containing times. What I want to do is to be able to determine which IDs are most similar, based upon the times e.g. Which IDs have correlated times for performing an action

The data looks something like this;

ID      Time(secs)
AAAA    1
AAAA    6
AAAA    5
AAAA    2
AAAA    4
BBBB    2
BBBB    4
BBBB    6
BBBB    3
CCCC    3
CCCC    4
CCCC    1
CCCC    6
DDDD    7
DDDD    4
DDDD    5
DDDD    3

Naively I initially thought this would be a simple case of plotting the values and then calculating the Correlation Coefficient but I soon realised this isn't possible;

ID             CorrCoef
AAAA>BBBB      ????
AAAA>CCCC
AAAA>DDDD
BBBB>CCCC
BBBB>DDDD
CCCC>DDDD

I am now thinking I need to be able to compare 2 populations using something like a t-statistic or perhaps by using something like an AutoRegression. It is fair to say I am now feeling a little bit out of my depth with something that seemed quite basic initially.

Does anyone have any pointers as to the best way of doing this?

Thanks in advance!

Edit

Based upon a suggestion below, it seems a KS test could be useful here. What I need to do therefore is subset the data so that it can be fed into the below line of code

scipy.stats.ks_2samp(data1, data2)

You may compute the $id$ co-occurrences frequencies in a given time-window.
Suppose (without loss of generality) your criteria for co-occurrence is that both $id$s must occur on the second t, than using maximum likelihood estimate $P(id_{i}|id_{j})$ is:

$P(id_{i}|id_{j}) = \frac{count(id_{i}, id_{j})}{count(id_{j})}$

and the maximum likelihood estimate for the joint probability is:

$P(id_{i}, id_{j}) = \frac{count(id_{i}, id_{j})}{\sum{k \in IDs}{}
 \ count(id_{k})}$

Where $id_j, id_i \in IDs\ $ and $IDs$ is the set containing all $id$s (AAAA,BBB,CCCC,...),

You can then calculate the pointwise mutual information between each $id$ pair, that is, how ofter two $id$s co-occur, compared with what we would expect if they were independent:

$I(id_i, id_j) = \log_{2}{\frac{P(id_i, id_j)}{P(id_i)P(id_j)}}$

Given you a estimation of how strong is the association between $id_i$ and $id_j$.

The same strategy may be used to find similarities, you may think about each $id_i$ being a $|IDs|$-dimensional vector with the co-occurrences frequencies being the values. You can then apply cosine-similarity or pearson-correlation to find the most similar vectors ($id$s).

EDIT

Complementing my answer follows python demonstrating the ideas above for the sample dataset given in the question.

First we create our dataframe from the data in the question

import pandas as pd
import numpy as np
import collections

d = {'ID': ['AAAA', 'AAAA', 'AAAA', 'AAAA', 'AAAA', 'BBBB', 'BBBB', 'BBBB', 'BBBB', 'CCCC', 'CCCC', 'CCCC', 'CCCC','DDDD', 'DDDD', 'DDDD', 'DDDD'],
     'Time': [1, 6, 5, 2, 4, 2, 4, 6, 3, 3, 4, 1, 6, 7, 4, 5, 3]}

df = pd.DataFrame(d)

Computes the co-occurrences

dfm = df.merge(df, on='Time')
dfm = dfm[dfm.ID_x != dfm.ID_y]  # ID_x and ID_y are created by the merge
df_M = pd.get_dummies(dfm.ID_x).groupby(dfm.ID_y).apply(sum)
print(df_M)

The dataframe df_M represents the co-occurrence matrix

      AAAA  BBBB  CCCC  DDDD
ID_y
AAAA     0     3     3     2
BBBB     3     0     3     2
CCCC     3     3     0     2
DDDD     2     2     2     0

Pairwise Mutual Information

Let N be the total amount of co-occurrences, than I can compute every joint probability simply dividing the co-occurrences in the df_M matrix  by N.

N = df_M.sum().sum()
df_joint_P = df_M/N  # Computes every joint probability P(id_i, id_j)

The probability of each $id$ is the sum of all its joint probabilities

df_ID_P = df_joint_P.sum(axis=0) # Marginalizes to produces P(id_i)

Now that we have all we need to compute a PMI dataframe

idx = [(r, c) for r in list(df_M) for c in list(df_M)]
pmi_dict = collections.defaultdict(dict)
for r,c in idx:
    pmi_dict[r][c] = np.log2(2*df_joint_P[r][c]/(df_ID_P[r] * df_ID_P[c])) if df_joint_P[r][c] > 0 else 0
pmi_df = pd.DataFrame(pmi_dict)
print(pmi_df)

The pairwise mutual information between every pair of different $id$s is different than zero and like bellow

          AAAA      BBBB      CCCC      DDDD
AAAA  0.000000  1.491853  1.491853  1.321928
BBBB  1.491853  0.000000  1.491853  1.321928
CCCC  1.491853  1.491853  0.000000  1.321928
DDDD  1.321928  1.321928  1.321928  0.000000

We can see that every $id$ seems to be almost equally associated with each other.

Cosine Similarity

Now we can compute the cosine similarity, remembering that it is the $l_2$-normalized dot-product

df_NM = df_M.div(df_M.pow(2).sum(axis=1).pow(0.5), axis=0)
df_cos = df_NM.dot(df_NM.T)
print(df_cos)

The cosines are

          AAAA      BBBB      CCCC      DDDD
AAAA  1.000000  0.590909  0.590909  0.738549
BBBB  0.590909  1.000000  0.590909  0.738549
CCCC  0.590909  0.590909  1.000000  0.738549
DDDD  0.738549  0.738549  0.738549  1.000000

Obviously every $id$ is (trivially) most similar to itself but from the data above we see that every $id$ is similar to all the others (confirming the result we found using PMI) but we may notice that $id$ DDDD is more similar to all the other $id$s (at least for this tiny example).


Is the usage of the "momentum" significiantly superior to the conventional weight update The "momentum" adds a little of the history of the last weight updates to the actual update, with diminishing weight history (older momentum shares get smaller).
Is it significiantly superior?
Weightupdate:
$$
w_{i+1} = w_i + m_i
$$
With momentum:
$$
m_0 = 0
\\
m_1 = \Delta w_{1} + \beta m_0 = \Delta w_1
\\
m_2 = \Delta w_{2} + \beta m_1 = \Delta w_2 + \beta\Delta w_1
$$
So the momentum already contains the actual weightupdate and the momentumhistory.
beta is like alpha a number between 0 and 1 (beta diminishes older momentum items).
Is there a common consent, that the usage of the momentum approach improves the learning quality, in terms of stability and speed?

Using momentum is a noise reduction (noisy gradients) and signal amplification strategy.
Imagine a large hill with a rough terrain with lots of ups-and-downs. We are trying to navigate to the bottom of the hill by using purely local information. A bad strategy is course correct frequently every time we see a potential new direction with steeper descent.
The momentum term makes it difficult to course correct frequently by adding weight to the past directions chosen. It dampens (pun intended) the idea of enthusiastically switching directions.
The challenge with common consent: the exact nature of the terrain is going to change from problem to problem (dataset). So your mileage by choosing momentum based updates may vary. But in general, for non-toy problem using momentum based weight updates will be a good first strategy.


How to predict future sales based upon last year sales data? I have past 1 year of sales data and now i want to predict sales for next coming year.
How can i predict it.

You can not predict sales for the whole next year by training your model with only one year data.

But still one year data is better than having nothing. And you can analyze one year data to find trends and patterns. On top of those trends and patterns, you can apply domain knowledge and make some very some useful strategic decisions.

Problem you are trying to solve is called Time Series Forecasting as temporal order of input matters while training.

Here are reference points you can your start with :
1) https://machinelearningmastery.com/how-to-get-started-with-deep-learning-for-time-series-forecasting-7-day-mini-course/
2) https://www.analyticsvidhya.com/blog/2018/02/time-series-forecasting-methods/
3) https://towardsdatascience.com/an-end-to-end-project-on-time-series-analysis-and-forecasting-with-python-4835e6bf050b


Best way to fuse clusters smaller than a certain size in a distributed way? I have X clusters of size (n1,n2...,nX) and I want to fuse small clusters with less than T members with the nearest adjacent cluster i.e. the nearest neighboring cluster.

The problem is when I've got two small neighbor clusters, A->B and B->A which create an infinite loop.

I try to avoid that by removing one of those which have this property [(A,B),(B,A)].

But is there a better way to accomplish this task in sequential way ?

Is it possible to do the trick in distributed way ( Spark ) ?

My suggestion is just a greedy brute force approach. I am sure that more efficient solutions may exist, but it may be a first step towards your final solution.

Store in a table, for each cluster which size is lesser than T, the distance to its closest neighbouring cluster and the ID of that closest cluster.

Now, sort the table entries in ascending distance order. Merge the cluster in the first entry with its closest neighbouring one. If the closest cluster also had an entry in the table, and its size got larger than T, remove its corresponding entry. Finally, update all the entries of the table, and sort them in ascending distance order again.

Repeat the steps above until there are not more entries in the table.


Checking for stationarity in LSTM An author in his blog checked for stationarity and removed them in a forecasting problem for using LSTM.I asked others and they said no need to check for it in LSTM.I read some articles and it looked like they need to be checked in ARIMA model.Do we have to check for it in LSTM??

In principle no you do not need to check for stationarity nor correct for it when you are using an LSTM.

The thing about stationarity is that it makes prediction tasks much more efficient, and stable. Think about stationarity in terms of a target. When what you are trying to predict is not stationary, it is like trying to shoot a moving target. It isn't impossible to do, but there is a better way to do things.

If you are trying to predict a stationary variable, then what you are trying to do is hit a non-moving, for lack of a better word, stationary target. It is staying in the same spot relative to you.

Correcting for non-stationarities in an ARIMA model is necessary because it is one of the fundamental assumptions that you make about that particular model. LSTMs do not make that assumption. So you don't have to do it with an LSTM. Seems like the answer to your question is the same as what the people you asked originally. Case closed, right?

Well... you remember shooting at moving targets is much more difficult than shooting at a stationary target right? You may want to force stationarity on a problem that you are going to run against an LSTM, simply because you may be able to eek out a little bit more performance by making it easier for the neural network to learn. So although you don't need to do it, it may still be a good idea and give you a boost in performance.


How to get a feature from sound/audio data learning using machine learning supervised classification? I'm a bit confused about how data in audio files can be processed for a ML classification model. I have several .wav files that contain dogs barking and cats "meowing". The pipeline is the following:

Load the data
Convert the data to FFT for desired window
Apply MFCC filtering
Back-transform using DCT
Creating a "spectrogram" for the window
Training a model?

What I don't understand is:

if I have .wav files of different sizes, lets say 1 second and 0.8 seconds, I will end up with a different number of windows, if window size is 0.1 seconds then the first file will have 10 windows and the second will have 8 windows, so how can I feed this information to the learning algorithm consistently.
Does the algorithm learn from the entire .wav file or window by window?
If the algorithm learns from each window, will each window have a different prediction value?

thank you.

Your pipeline is roughly right, but clarifying it a bit below:

1. Load the audio
2. Convert the audio to a `spectrogram` using STFT
3. Apply mel-scale filtering to get `mel-spectrogram`
4. Transform using DCT to get `MFCC`

You now have a time sequence of MFCC frames. It is typical for the classifier to operate on a time window of such frames. The window length can be relatively small (say 4-10 frames), or be the entire length of an audio clip. If they are smaller, the windows are often processed with some overlap.

Learning and outputting predictions for the entire file is the easiest, a standard classification problem.

If labels are only available on the entire file ("weak labeling") but one wants to output a prediction per window, this is usually done with Multi-Instance Learning.


As RELU is not differentiable when it touches the x-axis, doesn't it effect training? When I read about activation functions , I read that the reason we don't use step function is because, it is non differentiable which leads to problem in gradient descent.
I am a beginner in deep learning , as Relu is almost a linear function and also non differentiable where it touches x-axis , why it performs so much better than tanh or sigmoid functions. And why is it so widely used in Deep learning.
As it is non differentiable doesn't it affect in training?

A step function is discontinuous and its first derivative is a Dirac delta function. The discontinuity causes the issue for gradient descent. Further the zero slope everywhere leads to issues when attempting to minimize the function. The function is essentially saturated for values greater than and less than zero.
By contrast RELU is continuous and only its first derivative is a discontinuous step function. Since the RELU function is continuous and well defined, gradient descent is well behaved and leads to a well behaved minimization. Further, RELU does not saturate for large values greater than zero. This is in contrast to sigmoids or tanh, which tend to saturate for large value. RELU maintains a nice linear slope as x moves toward infinity.
The issue with saturation is that gradient descent methods take a long time to find the minimum for a saturated function.
Summarizing:

Step function: discontinuous and saturated at +/- large numbers.
Tanh: Continuous and well defined, but saturated at +/- large numbers.
Sigmoid: Continuous and well defined, but saturated at +/- large numbers.
Relu: Continuous and well defined. Does not saturate at + large number.

Hope this helps!


Update of mean and variance of weights I'm trying to understand the Bayes by Backprop algorithm from the paper Weight Uncertainty in Neural Networks, the idea is to make a NN in which each weight has it's own probability distribution. I get the theory, but I don't undertsand how to update the mean and variance in the learning part. I found a code in Pytorch which simply does:
class BayesianLinear(nn.Module):
def __init__(self, in_features, out_features):
    (...)
    # Weight parameters
    self.weight_mu = nn.Parameter(torch.Tensor(out_features, in_features).uniform_(-0.2, 0.2))
    self.weight_rho = nn.Parameter(torch.Tensor(out_features, in_features).uniform_(-5,-4))
    self.weight = Gaussian(self.weight_mu, self.weight_rho)

How does the optimizer knows how to update mu (the mean) and rho (the variance)? It is the Parameter function which allows it? Or it is happening somewhere in the code? (it looks like a normal neural net from there!).

The PyTorch remembers a computation graph. It means that for each value you get after the forward pass it remembers which operation you have applied and for all basic operations, it knows how to calculate the gradients. So, that's why in PyTorch you get the optimization for free -- at least, for common neural nets. The nn.Parameter method specifically tells to track gradients and change them during optimization.
The approach works well in the particular case for the most part, but there are two not-so-common steps in bayes by backprop:

For each neuron we sample weights. Technically, we start with sampling from $N(0, 1)$ and then we apply the trainable params. The specific values we get from $N(0, 1)$ are kind of extra inputs and for some operations, they change the gradients.
Loss usually depends only on outputs and true labels, but for the KL term, you need to calculate the closed-form value based on weights themselves. You would probably need to explicitly collect the KL loss term from each layer.

Everything else should be handled fine by pytorch


How to calculate pedigree function in diabetes prediction? I am developing a model for diabetes prediction using this dataset using Logistic Regression.

I have completed the model and my input variables are -

Pregnancies, Glucose, blood pressure, BMI, DiabetesPedigreeFunction etc.

The model gives an accuracy of 78% which is quite good for me.

Now when I predict new values which are not in the dataset I have all the required inputs except Pedigree function.

I searched for it and I understand what it is but how can I calculate that?

I had much better results with the decision forest. The diabetes pedigree function requires good diabetes diagnostic history of the persons relatives. Based on this, it might be neglectable.


Comparing two Corpora using Topic Model I want to compare two corpora (two different collections of texts) using Topic Modeling. I trained the model separately on the two collections and manually matched similar topics based on their frequent words.

I was wondering if there is a systematic way of comparing the topics across two corpora and measuring their similarity.

In my eyes, this is not a valid approach.

Note, that there is not one unique topic model (given some parameters like the number of topics and the algorithm for topic modelling) for a corpus. Different runs with different random seeds will give you different topic models for the same corpus.

So, any comparison comes down to a comparison of specific topic models, but not to a comparison of the corpora.

An approach with some better validity is to combine both corpora into one super-corpus, create a topic model of it, and than investigate the distribution of the topics with respect to the sub-corpora formed by the original corpora 1 and 2.


Effect of adding gaussian noise to the input layer in a CNN I often come across Keras code that adds GaussianNoise to the input, however its not clear to me what advantages does it offer to the learning.

input_img = layers.Input(t_x.shape[1:], name = 'RGB_Input')
pp_in_layer = layers.GaussianNoise(GAUSSIAN_NOISE)(input_img)

Adding noise in the input data is equivalent to adding a regularization term to the objective function.

See Training with Noise is Equivalent to Tikhonov Regularization and Adding noise to the input of a model trained with a regularized objective.


Autoencoder and Neural Network Overfitting in terms of parameter number? I have 1100 sequences for 2 classes. Of them 400 are from one class 1 and 700 are from class 2. I have used an one hidden layer auto-encoder of 2 neurons to capture my features. My initial features are tri-grams for each sequences. So, for each sequence I have 6860 tri-grams. As a result most of my input vectors are sparse vectors.

Now, if I calculate the parameters for this network, I have

6860 * 2 = 13720 paramters (1st layer)
2 * 6860 = 13720 parameters (2nd layer)
-----------------------------------------
           27440 parameters (in total)

Now, that is way too many parameters in comparison to my number of data points. Hence, I have used a dropout value of 0.98, on layer 1->hidden layer as well as hidden layer->output layer which makes the number of parameters 13720 * 0.02 =  274 on each layer and in total 548 parameters.

Now, after training, I tried the encoder on my test data of 500 sequences, and extracted the hidden layer of 2 dimensional data. Then I use that data on another 5 neuron single hidden layer neural network to classify. My results are really good in that I am getting around 90% accuracy on my test data which is around the same ball park of the training data accuracy.

My question is am I overfitting in my autoencoder? Am I overfitting by using another neural network? I am worried about my low number of data points. Does my use of dropout seem sensible?

I am worried that I am calculating the parameter numbers wrong, and my network has more parameters than number of data samples even after using Dropout. Does my calculation seem correct?

Additional drop out layers do not reduce parameters. Drop out is applied after each sample independently. This is done to Limit overfitting. So you still have the same parameters.

From over 6000 dimensions to only 2 seems a bit too much. But it is hard to see. You could run a PCA Analysis and use that as a starting point for the number of hidden neurons, e.g. get the dimensionality that Accounts for 75% of the Variation in your data and go from there. Also try a second layer and add some form of regularization.


Binary encoding and its interpretation in Python I have a column named Street that has 2 values: Paved and Gravel.  Here is what print(train[binary_columns[0]].unique().tolist()) gives me:

['Pave', 'Grvl']

I want to encode these values in binary like this:

df['Street'] = df['Street'].replace(['Pave', 'Grvl'], [1, 0])

But I wonder if this is a good idea.  Wouldn't the computer interpret this as Pave > Grvl?  How does the computer differentiate between binary and integer encoding?

Your categorical variable has two levels, so there is no actual difference between dummy-coding vs. simply entering the variable into the analysis. That is, to dummy code you would create one new variable with two values but your original variable is already one variable with two values. Dummy-coding is important for variables with more than two possible values. So, in this case the computer won't consider Pave > Grvl.

But if you have more than two variables then you should use dummy variables.

For your data, you can use pandas.get_dummies() or sklearn's one hot encoder to achieve your result.


Oversampling possible improvement I am currently solving a classification problem for an imbalanced data set (approximately 17% of the minority class). I split the data using a stratified k-fold split from sklearn (Stratified shuffle split) after which I oversample the train data, using ADASYN, and fit the oversampled train data (approximately 250k+ instances after oversampling) to gradient boosting classifier. Oversampling has a huge effect on the performance, recall measure improves from 7% to 75%. Is this possible? If not, any ideas what could be going wrong?

My main question about this is could this improvement be possible?

The evaluation of an imbalanced dataset should be done with more than one single metric since you need to evaluate the performance on both the majority and the minority classes.

In the case that is mentioned the performance is evaluated only with recall (the true-positive rate). It could be that the model is now focusing on the oversampled minority class (especially if it's the new majority class) and having a weaker performance on the original majority class, and I assume that the precision or the false-negative rate were decreased.

There are several evaluation metrics that are considering the performance of both classes that are not sensitive to the classes imbalance - F1 score, Geometric mean, and the most common the ROC/PR AUC


N - fold cross validation in mahout Is there a method/class available in Apache Mahout to perform n-fold cross validation?
If yes how it can be done?

Unfortunately, there isn't any.  However, the AbstractDifferenceRecommenderEvaluator can be extended for the purpose. The class would look something like this:

public abstract class AbstractKFoldRecommenderEvaluator extends AbstractDifferenceRecommenderEvaluator {

private final Random random;
public double noEstimateCounterAverage = 0.0;
public double totalEstimateCount = 0.0;
public double totalEstimateCountAverage = 0.0;

private static final Logger log = LoggerFactory
        .getLogger(AbstractKFoldRecommenderEvaluator.class);

public AbstractKFoldRecommenderEvaluator() {
    super();
    random = RandomUtils.getRandom();
}

public double getNoEstimateCounterAverage(){
    return noEstimateCounterAverage;
}

public double getTotalEstimateCount(){
    return totalEstimateCount;
}

public double getTotalEstimateCountAverage(){
    return totalEstimateCountAverage;
}

/**
 * We use the same evaluate function from the RecommenderEvaluator interface
 * the trainingPercentage is used as the number of folds, so it can have
 * values bigger than 0 to the number of folds.
 */
@Override
public double evaluate(RecommenderBuilder recommenderBuilder,
        DataModelBuilder dataModelBuilder, DataModel dataModel,
        double trainingPercentage, double evaluationPercentage)
        throws TasteException {
    Preconditions.checkNotNull(recommenderBuilder);
    Preconditions.checkNotNull(dataModel);
    Preconditions.checkArgument(trainingPercentage >= 0.0,
            "Invalid trainingPercentage: " + trainingPercentage);

    Preconditions.checkArgument(evaluationPercentage >= 0.0
            && evaluationPercentage <= 1.0,
            "Invalid evaluationPercentage: " + evaluationPercentage);

    log.info("Beginning evaluation using {} of {}", trainingPercentage,
            dataModel);

    int numUsers = dataModel.getNumUsers();

    // Get the number of folds
    int noFolds = (int) trainingPercentage;

    // Initialize buckets for the number of folds
    List<FastByIDMap<PreferenceArray>> folds = new ArrayList<FastByIDMap<PreferenceArray>>();
    for (int i = 0; i < noFolds; i++) {
        folds.add(new FastByIDMap<PreferenceArray>(
                1 + (int) (i / noFolds * numUsers)));
    }

    // Split the dataModel into K folds per user
    LongPrimitiveIterator it = dataModel.getUserIDs();
    while (it.hasNext()) {
        long userID = it.nextLong();
        if (random.nextDouble() < evaluationPercentage) {
            splitOneUsersPrefs2(noFolds, folds, userID, dataModel);
        }
    }

    double result = Double.NaN;
    List<Double> intermediateResults = new ArrayList<>();
    List<Integer> unableToRecoomend = new ArrayList<>();
    List<Integer> averageEstimateCounterIntermediate = new ArrayList<>();

    noEstimateCounterAverage = 0.0;
    totalEstimateCount = 0.0;
    totalEstimateCountAverage = 0.0;
    int totalEstimateCounter = 0;

    // Rotate the folds. Each time only one is used for testing and the rest
    // k-1 folds are used for training
    for (int k = 0; k < noFolds; k++) {
        FastByIDMap<PreferenceArray> trainingPrefs = new FastByIDMap<PreferenceArray>(
                1 + (int) (evaluationPercentage * numUsers));
        FastByIDMap<PreferenceArray> testPrefs = new FastByIDMap<PreferenceArray>(
                1 + (int) (evaluationPercentage * numUsers));

        for (int i = 0; i < folds.size(); i++) {

            // The testing fold
            testPrefs = folds.get(k);

            // Build the training set from the remaining folds
            if (i != k) {
                for (Map.Entry<Long, PreferenceArray> entry : folds.get(i)
                        .entrySet()) {
                    if (!trainingPrefs.containsKey(entry.getKey())) {
                        trainingPrefs.put(entry.getKey(), entry.getValue());
                    } else {
                        List<Preference> userPreferences = new ArrayList<Preference>();
                        PreferenceArray existingPrefs = trainingPrefs
                                .get(entry.getKey());
                        for (int j = 0; j < existingPrefs.length(); j++) {
                            userPreferences.add(existingPrefs.get(j));
                        }

                        PreferenceArray newPrefs = entry.getValue();
                        for (int j = 0; j < newPrefs.length(); j++) {
                            userPreferences.add(newPrefs.get(j));
                        }
                        trainingPrefs.remove(entry.getKey());
                        trainingPrefs.put(entry.getKey(),
                                new GenericUserPreferenceArray(
                                        userPreferences));

                    }
                }
            }
        }

        DataModel trainingModel = dataModelBuilder == null ? new GenericDataModel(
                trainingPrefs) : dataModelBuilder
                .buildDataModel(trainingPrefs);

        Recommender recommender = recommenderBuilder
                .buildRecommender(trainingModel);

        Double[] retVal = getEvaluation(testPrefs, recommender);
        double intermediate = retVal[0];
        int noEstimateCounter = ((Double)retVal[1]).intValue();
        totalEstimateCounter += ((Double)retVal[2]).intValue();
        averageEstimateCounterIntermediate.add(((Double)retVal[2]).intValue());

        log.info("Evaluation result from fold {} : {}", k, intermediate);
        log.info("Average Unable to recommend  for fold {} in: {} cases out of {}", k, noEstimateCounter, ((Double)retVal[2]).intValue());
        intermediateResults.add(intermediate);
        unableToRecoomend.add(noEstimateCounter);

    }

    double sum = 0;
    double noEstimateSum = 0;
    double totalEstimateSum = 0;
    // Sum the results in each fold
    for (int i = 0; i < intermediateResults.size(); i++) {
        if (!Double.isNaN(intermediateResults.get(i))) {
            sum += intermediateResults.get(i);
            noEstimateSum+=unableToRecoomend.get(i);
            totalEstimateSum+=averageEstimateCounterIntermediate.get(i);
        }
    }

    if (sum > 0) {
        // Get an average for the folds
        result = sum / intermediateResults.size();
    }

    double noEstimateCount = 0;
    if(noEstimateSum>0){
        noEstimateCount = noEstimateSum / unableToRecoomend.size();
    }

    double avgEstimateCount = 0;
    if(totalEstimateSum>0){
        avgEstimateCount = totalEstimateSum / averageEstimateCounterIntermediate.size();
    }

    log.info("Average Evaluation result: {} ", result);
    log.info("Average Unable to recommend in: {} cases out of avg. {} cases or total {} ", noEstimateCount, avgEstimateCount, totalEstimateCounter);

    noEstimateCounterAverage = noEstimateCount;
    totalEstimateCount = totalEstimateCounter;
    totalEstimateCountAverage = avgEstimateCount;
    return result;
}

/**
 * Split the preference values for one user into K folds, randomly
 * Generate random number until is not the same as the previously generated on
 * in order to make sure that at least two buckets are populated.
 *
 * @param k
 * @param folds
 * @param userID
 * @param dataModel
 * @throws TasteException
 */
private void splitOneUsersPrefs(int k,
        List<FastByIDMap<PreferenceArray>> folds, long userID,
        DataModel dataModel) throws TasteException {

    List<List<Preference>> oneUserPrefs = Lists
            .newArrayListWithCapacity(k + 1);
    for (int i = 0; i < k; i++) {
        oneUserPrefs.add(null);
    }

    PreferenceArray prefs = dataModel.getPreferencesFromUser(userID);

    int size = prefs.length();
    int previousBucket = -1;
    Double rand = -2.0;
    for (int i = 0; i < size; i++) {
        Preference newPref = new GenericPreference(userID,
                prefs.getItemID(i), prefs.getValue(i));
        do {
            rand = random.nextDouble() * k * 10;

            rand = (double) Math.floor(rand / 10);
            // System.out.println("inside Rand "+rand);

        } while (rand.intValue() == previousBucket);
        // System.out.println("outside rand "+rand);
        if (oneUserPrefs.get(rand.intValue()) == null) {
            oneUserPrefs.set(rand.intValue(), new ArrayList<Preference>());
        }
        oneUserPrefs.get(rand.intValue()).add(newPref);

        previousBucket = rand.intValue();

    }

    for (int i = 0; i < k; i++) {
        if (oneUserPrefs.get(i) != null) {
            folds.get(i).put(userID,
                    new GenericUserPreferenceArray(oneUserPrefs.get(i)));
        }
    }

}

/**
 * Split the preference values for one user into K folds, by shuffling.
 * First Shuffle the Preference array for the user. Then distribute the item-preference pairs
 * starting from the first buckets to the k-th bucket, and then start from the beggining.
 *
 * @param k
 * @param folds
 * @param userID
 * @param dataModel
 * @throws TasteException
 */
private void splitOneUsersPrefs2(int k, List<FastByIDMap<PreferenceArray>> folds, long userID, DataModel dataModel) throws TasteException {

    List<List<Preference>> oneUserPrefs = Lists.newArrayListWithCapacity(k + 1);
    for (int i = 0; i < k; i++) {
        oneUserPrefs.add(null);
    }

    PreferenceArray prefs = dataModel.getPreferencesFromUser(userID);
    int size = prefs.length();

    List<Preference> userPrefs = new ArrayList<>();
    Iterator<Preference> it = prefs.iterator();
    while (it.hasNext()) {
        userPrefs.add(it.next());
    }

    // Shuffle the items
    Collections.shuffle(userPrefs);

    int currentBucket = 0;
    for (int i = 0; i < size; i++) {
        if (currentBucket == k) {
            currentBucket = 0;
        }

        Preference newPref = new GenericPreference(userID, userPrefs.get(i).getItemID(), userPrefs.get(i).getValue());

        if (oneUserPrefs.get(currentBucket) == null) {
            oneUserPrefs.set(currentBucket, new ArrayList<Preference>());
        }
        oneUserPrefs.get(currentBucket).add(newPref);
        currentBucket++;
    }

    for (int i = 0; i < k; i++) {
        if (oneUserPrefs.get(i) != null) {
            folds.get(i).put(userID, new GenericUserPreferenceArray(oneUserPrefs.get(i)));
        }
    }

}

private Double[] getEvaluation(FastByIDMap<PreferenceArray> testPrefs, Recommender recommender) throws TasteException {
    reset();
    Collection<Callable<Void>> estimateCallables = Lists.newArrayList();
    AtomicInteger noEstimateCounter = new AtomicInteger();
    AtomicInteger totalEstimateCounter = new AtomicInteger();
    for (Map.Entry<Long, PreferenceArray> entry : testPrefs.entrySet()) {
        estimateCallables.add(new PreferenceEstimateCallable(recommender, entry.getKey(), entry.getValue(), noEstimateCounter, totalEstimateCounter));
    }
    log.info("Beginning evaluation of {} users", estimateCallables.size());
    RunningAverageAndStdDev timing = new FullRunningAverageAndStdDev();
    execute(estimateCallables, noEstimateCounter, timing);

    Double[] retVal = new Double[3];
    retVal[0] = computeFinalEvaluation();
    retVal[1] = (double) noEstimateCounter.get();
    retVal[2] = (double) totalEstimateCounter.get();
    //retVal.put(computeFinalEvaluation(), noEstimateCounter.get());
    //return computeFinalEvaluation();
    return retVal;
}}

The implementation can be something like this:

public class RMSRecommenderEvaluatorModified extends AbstractKFoldRecommenderEvaluator {

private RunningAverage average;

@Override
protected void reset() {
    average = new FullRunningAverage();
}

@Override
protected void processOneEstimate(float estimatedPreference, Preference realPref) {
    double diff = realPref.getValue() - estimatedPreference;
    average.addDatum(diff * diff);
}

@Override
protected double computeFinalEvaluation() {
    return Math.sqrt(average.getAverage());
}

@Override
public String toString() {
    return "RMSRecommenderEvaluator";
}}

Source


How to use auxiliary target variables only present in train data Imagining in my train data I have 3 target variables y1, y2 and y3, all binary.
My main goal though, is to predict the final variable Y = y1 * y2 * y3.

What should be the approach of a model when dealing with this kind of target variable, that I can decompose into different ones?

Should I try to predict y1, y2, y3, then calculate Y and pass it to a loss function
Should I try to predict Y standalone and ignore y1, y2 and y3?
Should I try to predict the 4 of them and penalize my model when Y != y1 * y2 * y3 ?

It is not possible to predict outcomes of ideas like this with certainty. Too much depends on the specifics of your problem and dataset. You will have to do the experiments yourself. With a decent auto-differentiation engine like TensorFlow or PyTorch, you could try all your different ideas really quickly, with a few variations.

Here is what I think will happen from an intuitive guess:

  Should I try to predict y1, y2, y3, then calculate Y and pass it to a loss function

I think this has potential to be the strongest model, as it incorporates knowledge that you have of the problem into the structure of your model, whilst not adding more hyperparameters. This may also allow you to use less parameters to achieve the same accuracy, plus may work as a form of regularisation.

  Should I try to predict Y standalone and ignore y1, y2 and y3?

This should be considered the default/benchmark model, and you will want to run that anyway. If you only make one model, make this one.

If you run comparisons, you will want to compare best model of this type with best model(s) of other types. Don't just try the same network with an altered top layer to change the prediction mechanism - that will not explore your idea well enough to give you an answer.

  Should I try to predict the 4 of them and penalize my model when Y != y1 * y2 * y3 ?

I expect this would be the hardest to train. Although it may still add some regularisation effect, your problem is that the "best" model in this category may rate consistency over accuracy - you will need to play with different weightings of the penalty metric to find a compromise loss function.

Remember to use cross-validation and test data in order to search parameters and get fair unbiased measures of performance of the best model from each of the approaches at the end.


What does 'energy' in image processing mean? I have been going through this paper: Seam Carving for Content-Aware Image Resizing which talks about resizing images by seam carving depending on the image energy or the energy function.

Some related quotes from the paper are:

  A seam is a connected path of low energy pixels in an image

and

  A seam is an optimal 8-connected path of pixels on a single image from
  top to bottom, or left to right, where optimality is defined by an
  image energy function

So, what does the 'energy' in an image mean, in this particular context?

PS: I have gone through this question, but the answers were very broad and vague because of this reason.

I quote from here which our friend has provided.

  Energy in this context is not, strictly speaking, the same as the conventional notion of energy in physics and the other sciences. The two concepts are, however, closely related, and it is possible to convert from one to the other:

  where Z represents the magnitude, in appropriate units of measure, of the load driven by the signal.

Also for figuring out the interpretation of energy in this context you can see here.

After a while, I found a bit better explanation for energy as you can read the answers from here.

  Energy is used to describe a measure of "information" when formulating an operation under a probability framework such as MAP (maximum a priori) estimation in conjunction with Markov Random Fields. Sometimes the energy can be a negative measure to be minimised and sometimes it is a positive measure to be maximized.

or you can read:

  Energy is a fairly loose term used to describe any user defined function (in the image domain).


Why KL Divergence instead of Cross-entropy in VAE I understand how KL divergence provides us with a measure of how one probability distribution is different from a second, reference probability distribution. But  why are they particularly used (instead of cross-entropy) in VAE (which is generative)?

Answering with some theoretical understanding  of Variational auto-encoders.
In the general architecture of encoders and decoders, the encoder encodes the input a latent-space, and the decoder reconstructs the input from the encoded latent space.
However, the Variational auto-encoders (VAE), the input is encoded to a latent-distribution instead of a point in a latent space. This latent distribution is considered to be Normal Gaussian distribution (Which can be expressed in terms of mean and variance). Further, decoders samples a point in this distribution and reconstructs the input. Since, VAE encoder encodes to a distribution than a point in a latent space, and KL divergence is use to measure the difference between the distribution, it is used as a regularization term in the loss function.


How to replace a part string value of a column using another column How to replace a part string value of a column using another column.

My DataSet here is :

ID          Product Name                            Size ID    Size Name
1   24 Mantra Ancient Grains Foxtail Millet 500 gm      1       500 gm
2   24 Mantra Ancient Grains Little Millet 500 gm       2       500 gm
3   24 Mantra Naturals Almonds 100 gm                   3       100 gm
4   24 Mantra Naturals Kismis 100 gm                    4       100 gm
5   24 Mantra Organic Ajwain 100 gm                     5       100 gm
6   24 Mantra Organic Apple Blast Drink 250 ml          6       250 ml
7   24 Mantra Organic Apple Juice 1 Ltr Tetra Pack      7       1000 ml
8   24 Mantra Organic Apple Juice 200 ml                8       200 ml
9   24 Mantra Organic Assam Tea 100 gm                  9       100 gm

Requirement here is the Product Name column value is 24 Mantra Ancient Grains Foxtail Millet 500 gm and the Size Name column has 500 Gm.  In this case my output will be 24 Mantra Ancient Grains Foxtail Millet.
If the Size Name contains in the Product Name string remove the size name word ignoring the case else no need to take any action.

data['Product Name'] = data['Product Name'].str.replace('\d+','')

This should get rid of the number if that's what you are looking for. I am not sure what you mean by 'chomped.'


How to deal with big output values after classification layer during training? In AI libraries such as, Tensorflow, Keras etc., how the big output numbers are dealt during training process? For example, for classification layer, it will have some outputs such as [23.4, -21254.3, 32123.4]. In next step, these numbers will go into softmax function which will take power of base e with each output numbers. However, this may result in extreme big numbers in case of extremely large positive number or extremely small negative number. So, how are extreme cases dealt during training process?

As you stated in your question, those numbers go into a softmax function. Another name for softmax is normalized exponential function. Softmax normalizes numbers where the sum is constrained to be 1 and each value becomes the probability of categorical membership.
In the specific case of [23.4, -21254.3, 32123.4], applying the softmax transforms them to [0, 0, 1]. The numbers become either 0 or 1 because of numerical underflow.


Python vs R for machine learning I'm just starting to develop a machine learning application for academic purposes. I'm currently using R and training myself in it. However, in a lot of places, I have seen people using Python.

What are people using in academia and industry, and what is the recommendation?

Some real important differences to consider when you are choosing R or Python over one another:

Machine Learning has 2 phases. Model Building and Prediction phase. Typically, model building is performed as a batch process and predictions are done realtime. The model building process is a compute intensive process while the prediction happens in a jiffy. Therefore, performance of an algorithm in Python or R doesn't really affect the turn-around time of the user. Python 1, R 1.
Production: The real difference between Python and R comes in being production ready. Python, as such is a full fledged programming language and many organisations use it in their production systems. R is a statistical programming software favoured by many academia and due to the rise in data science and availability of libraries and being open source, the industry has started using R. Many of these organisations have their production systems either in Java, C++, C#, Python etc. So, ideally they would like to have the prediction system in the same language to reduce the latency and maintenance issues.
Python 2, R 1.
Libraries: Both the languages have enormous and reliable libraries. R has over 5000 libraries catering to many domains while Python has some incredible packages like Pandas, NumPy, SciPy, Scikit Learn, Matplotlib. Python 3, R 2.
Development: Both the language are interpreted languages. Many say that python is easy to learn, it's almost like reading english (to put it on a lighter note) but R requires more initial studying effort. Also, both of them have good IDEs (Spyder etc for Python and RStudio for R). Python 4, R 2.
Speed: R software initially had problems with large computations (say, like nxn matrix multiplications). But, this issue is addressed with the introduction of R by Revolution Analytics. They have re-written computation intensive operations in C which is blazingly fast. Python being a high level language is relatively slow. Python 4, R 3.
Visualizations: In data science, we frequently tend to plot data to showcase patterns to users. Therefore, visualisations become an important criteria in choosing a software and R completely kills Python in this regard. Thanks to Hadley Wickham for an incredible ggplot2 package. R wins hands down. Python 4, R 4.
Dealing with Big Data: One of the constraints of R is it stores the data in system memory (RAM). So, RAM capacity becomes a constraint when you are handling Big Data. Python does well, but I would say, as both R and Python have HDFS connectors, leveraging Hadoop infrastructure would give substantial performance improvement. So, Python 5, R 5.

So, both the languages are equally good. Therefore, depending upon your domain and the place you work, you have to smartly choose the right language. The technology world usually prefers using a single language. Business users (marketing analytics, retail analytics) usually go with statistical programming languages like R, since they frequently do quick prototyping and build visualisations (which is faster done in R than Python).


Simple Linear Regression-----How to make my model more efficient?? I am working on a simple linear regression model,

  This is my Python code :

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

dataset=pd.read_csv('sample.csv')
X=dataset.iloc[:,:-1].values
Y=dataset.iloc[:,1].values

from sklearn.model_selection import train_test_split
X_train,X_test,Y_train,Y_test=train_test_split(X,Y,test_size=1/3)

from sklearn.linear_model import LinearRegression
regressor=LinearRegression()
regressor.fit(X_train,Y_train)

plt.scatter(X_train,Y_train,color='red')
plt.plot(X_train,regressor.predict(X_train),color='blue')
plt.title('X vs Y(Training Set)')
plt.xlabel('X')
plt.ylabel('Y')
plt.show()

plt.scatter(X_test,Y_test,color='red')
plt.plot(X_train,regressor.predict(X_train),color='blue')
plt.title('X vs Y(Test Set)')
plt.xlabel('X')[enter image description here][1]
plt.ylabel('Y')
plt.show()`

  This is my plot of Training Set Training Set
  This is my plot of Test Set Test Set

How can I increase the efficiency of my ML model???
This is my first ML model, so all suggestions are welcome.
Thanks in Advance

You cannot really do much: the fit of the regressor is optimum, therefore it is the best that the algorithm can do given these points. What you can do, is change the weights of individual points of the dataset using the sample weight parameter of

LinearRegression.fit(X, y, sample_weight)

method, to "attract" the line towards them and see how this affects the accuracy.

I would not expect notable difference because your data do not follow a linear pattern.


KerasRegressor serialize/save a model as a .h5df I am been using a script from machinelearningmastery on Keras regression and I would like to save model as a .h5 file.

Machinelearningmastery also has another tutorial for saving models/pickles but the scripts are written in a model.fit() method in Keras… But the script I am using I am defining the model thru calling a function.

Can someone give me a tip on how I can save this model as a .h5df?

import numpy as np
import pandas as pd
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import matplotlib.pyplot as plt
import math
from sklearn.preprocessing import MinMaxScaler

# load dataset
dataset = pd.read_csv("joinedRuntime2.csv", index_col='Date', parse_dates=True)

print(dataset.shape)
print(dataset.dtypes)
print(dataset.columns)

# shuffle dataset
df = dataset.sample(frac=1.0)

# split into input (X) and output (Y) variables
X = np.array(df.drop(['kWh'],1))
Y = np.array(df['kWh'])

def wider_model():
    # create model
    model = Sequential()
    model.add(Dense(20, input_dim=7, kernel_initializer='normal', activation='relu'))
    #model.add(Dense(28, kernel_initializer='normal', activation='relu'))
    #model.add(Dense(21, kernel_initializer='normal', activation='relu'))
    #model.add(Dense(14, kernel_initializer='normal', activation='relu'))
    model.add(Dense(10, kernel_initializer='normal', activation='relu'))
    model.add(Dense(1, kernel_initializer='normal'))
    # Compile model
    model.compile(loss='mean_squared_error', optimizer='adam')
    return model

# fix random seed for reproducibility
seed = 7
np.random.seed(seed)
estimators = []
estimators.append(('standardize', StandardScaler()))
estimators.append(('mlp', KerasRegressor(build_fn=wider_model, epochs=200, batch_size=5, verbose=0)))
pipeline = Pipeline(estimators)
kfold = KFold(n_splits=10, random_state=seed)
results = cross_val_score(pipeline, X, Y, cv=kfold)
print("Wider: %.2f (%.2f) MSE" % (results.mean(), results.std()))
print("RMSE", math.sqrt(results.std()))

Citing Keras' official page:

  It is not recommended to use pickle or cPickle to save a Keras model.

  You can use model.save(filepath) to save a Keras model into a single
  HDF5 file which will contain:

  the architecture of the model, allowing to re-create the model
  the weights of the model
  the training configuration (loss, optimizer)
  the state of the optimizer, allowing to resume training exactly where you left off.


Which algorithm should I choose and why? My friend was reading a textbook and had this question:
Suppose that you observe $(X_1,Y_1),...,(X_{100}Y_{100})$, which you assume to be i.i.d. copies of a random pair $(X,Y)$ taking values in $\mathbb{R}^2 \times \{1,2\}$. Your plot the data and see the following:
where black circles represent those $X_i$ with $Y_i=1$ and the red triangles represent those $X_i$ with $Y_i=2$. A practitioner tells you that their misclassification costs are equal, $c_1 = c_2 = 1$, and
would like advice on which algorithm to use for prediction. Given the options:

Linear discriminant analysis;
K-Nearest neighbours with $K=5$
K-Nearest neighbours with $K=90$.

What would be the best algorithm for this? I think it should be $5$, as the bigger the $K$, the worse the accuracy gets? What would be your choice and why?

You can choose the optimal method using cross-validation. If your sample size is relatively small, use leave-one-out cross-validation... I would not be surprised if $K = 5$ worked well. Linear discriminant analysis (LDA) will not work here because it implies linear decision boundaries. Unless you enlarge the set of predictors with non-linear transformations.
Also, the picture above is a classic case where support vector machines (SVM) with a Gaussian kernel could be of use. R has a friendly implementation of SVM in the "kernlab" package.


What is the best way to find minima in Logistic regression? In the Andrew NG's tutorial on Machine Learning, he takes the first derivative of the error function then takes small steps in direction of the derivative to find minima. (Gradient Descent basically)

In the elements of statistical leaning book, it found the first derivative of the error function, equated it to zero then used numerical methods to find out the root (In this case Newton Raphson)

In paper both the methods should yield similar results. But numerically are they different or one method is better than the other ?

Being a second-order method, Newton–Raphson algorithm is more efficient than the gradient descent if the inverse of the Hessian matrix of the cost function is known. However, inverting the Hessian, which is an $\mathcal O(n^3)$ operation, rapidly becomes impractical as the dimensionality of the problem grows. This issue is addressed in quasi-Newton methods, such as BFGS, but they don't handle mini-batch updates well and hence require the full dataset to be loaded in memory; see also this answer for a discussion. Later in his course Andrew Ng will discuss neural networks. They can easily contain millions of free parameters and be trained on huge datasets, and because of this variants of the gradient descent are usually used with them.

In short, the Newton–Raphson method can be faster for relatively small problems, but the gradient descent scales better with complexity of problem.


Anomaly detection for transaction data I have transaction details for credit data (bank transfers, peer to peer transfers, etc). Currently, I have one year worth of data which I cannot properly classify.

I'm looking for input and suggestions about these two questions:

How do I detect anomalies in the last hour with previous credit transaction data that I have?
How can I detect anomalies in real-time for the current transaction?

There are a few factors to consider in anomaly detection.  A simple method would be to plot a boxplot of the data and calculate outliers this way (boxplot description).

If the time series is seasonal, you could take an approach similar to Twitter's anomaly detection algorithm (Twitter Anomaly Detection).  This uses an outlier detection method (ESD) applied to the seasonal decomposition of the time series.

You could use clustering and SVMs as well.

There is also change-point detection which I understand less but can be very effective.

Really, your answer depends on how complex your data is.  Sometimes, a simple heuristic algorithm works the best.

Here are some similar questions:

Anomaly detection in Time Series Data - Help Required

How to classify and cluster this time series data


How can I solve it? : def preprocess_features(X):
    output = pd.DataFrame(index = X.index)
    for col , col_data in X.iteritems():
        if col_data.dtype == object :
            col_data = pd.get_dummies(col_data , prefix = col)
        output = output.join(col_data)
    return output
X_a = preprocess_features(X_all)

X_train , X_test , Y_train , Y_test = train_test_split(X_a , Y_all , test_size = 0.15, random_state = 2)
#print(X_train.shape,X_test.shape,Y_train.shape,Y_test.shape)
clf = LogisticRegression()
clf.fit(X_train , Y_train)

When I run this code, I get the following error:
How do I resolve it?

  ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

Check the columns inside your dataframe.

You can try:

X.count()

to see how many data entries each column has.

From there, you can try:

.fillna()

or go back to your data and fill the necessary information or use sklearn.preprocessing.Imputer encode mean / median imputation for missing values.

Make sure that each column doesn't have any string or letter in it.
Check also each column and get their max value, maybe you can convert them in much more smaller number (e.g. Normalizing)

Also, try this:

np.isnan(X)

to get a boolean form of your dataframe to check if there's any blanks


Syntax error in pandas dataframe slice in my sklearn data prep? I'm new to sklearn and having trouble formatting the data to predict and evaluate a confusion matrix.  I'm using this Random Forest tutorial.

Here is my code

 from sklearn.ensemble import RandomForestClassifier
 import numpy as np
 import pandas as pd
 dataframe = pd.read_csv('output.txt', sep='\t')
 df = pd.DataFrame(dataframe)
 df['is_train'] = np.random.uniform(0, 1, len(df)) <= .75

 train, test = df[df['is_train']==True], df[df['is_train']==False]
 features = df.columns[1:5]
 clf = RandomForestClassifier(n_jobs=2)
 y, _ = pd.factorize(train['event_count'])
 clf.fit(train[features], y)

It's this line for my predictions which is giving me the error:

 preds = df[:6][clf.predict(test[features])]
 IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices

It is not possible to write numpy style slices in dataframe, like :

observations = data[:6]

Try DataFrame.ix

df.ix[:,6:]

Cf panda indexing : Panda indexing


Why for logistic regression the error is given by [y ln(sigma(x)) + (1 − y) ln(1 − sigma(x)] Why for logistic regression, with target values 0 or 1, it will not work to take the sum of the squares of the difference between target value and prediction, but rather:
$$
error({\bf w}) = -1/m * \sum_{i=1}^{m} [ y_i \ln (\sigma({x_i})) + (1-y_i) \ln (1 - \sigma({x_i} ) ]
$$

This is the log-likelihood:

$\log P(x; w) \equiv \log \prod_i P(x_i | w) = \sum_i \log P(x_i | w)$, where $P(x_i | w) \equiv \left\{ \begin{array}{rl}\sigma(x_i), & y_i =1 \\ 1 - \sigma(x_i), &y_i = 0\end{array} \right.$

Why the log-likelihood? When you have a probabilistic model, such as logistic regression, it's one way (the MLE) of finding the parameters that fit best. Recall that in logistic regression we are, contrary to the name, trying to classify rather than regress, and the MSE is a regression loss; it seeks to minimize the distance from a point, while we wish to penalize being in the wrong subspace (the parts that don't correspond to the correct class). If you squint a bit, you can see that the negative log-likelihood minimizes the cross entropy.


What can be done so that 'teacher' and 'teaches' are treated similar? Who teaches English?

Now, after tokenizing, stemming..
    it gives me

Who, teach, English

In my list of word, I have a word called

teacher

Lemmatizing, stemming teacher gives teacher and lemmatizing, stemming teaches gives teach

Even, calculating edit_distance will not solve this.. As, edit_distance comes out to be 2.

Now, What do I do to have  teacher and teach treated as similar?
Similarly, there may be other cases with extra 's' at the end. Is there some stemmer that solves this problem? Is there any solution?

Other similar example can be: instructor and instructs

Use an aggressive stemmer. The Lancaster Stemmer is one the most aggressive and popular stemmers around.
Here is the Python code:
from nltk.stem.lancaster import LancasterStemmer

lancaster_stemmer = LancasterStemmer()
assert 'teach' == lancaster_stemmer.stem('teacher') == lancaster_stemmer.stem('teaches')


Training textblob with 16k rows of labeled data won't work (only few are working) I've got labeled data in a csv which looks like:

title,type
Women Jacket A,Clothes
Mens Running Shoes B,Shoes
Children backpack,Bags

and a script:

from textblob.classifiers import NaiveBayesClassifier

with open('train_small.csv', 'r') as fp:
    cl = NaiveBayesClassifier(fp, format="csv")

print ( cl.classify("Womens running shoe") )

If I use train_small.csv with 3 categories and 32 rows, I get the printed result, correct and immediate.

If I use my full training set (train.csv, 16k rows, 10 categories) I won't get a result. Today, I ran my laptop for 6 hours and then I had to shut it down by pressing the power button since it was at a load of 60 and irresponsive.

System is:

Operating System: Kubuntu 19.04
Kernel Version: 5.0.0-15-generic
OS Type: 64-bit
Processors: 4 × Intel® Core™ i5-7200U CPU @ 2.50GHz
Memory: 7.7 GiB of RAM

Not great, but completely insufficient ..?
Do I need to use another library / algorithm or just different hardware?

Oh Wow, textblob's default setting can't deal with even a reasonable amount of data.

TL;DR: You do need better control over what the library does.

The class NaiveBayesClassifier, or more exactly it's superclass NLTKClassifier uses a default for of feature_extractor.  In this case it uses basic_extractor, which, apart from stemming the words in some way (not relevant for us here) it does the following:

features = dict(((u'contains({0})'.format(word), (word in tokens))
    for word in word_features))

Once per each document (sample).

Ouch!  That will create a python dictionary for every single document.  And store in these dictionaries the term frequecy for each document.  On 16k rows (documents/samples) that is very likely to be way too much memory.

On Ubuntu you likely have swap space, which is extra disk space used as memory, your OS figuring out it is out of memory starts swapping pages of memory onto this swap space.  Yet, swap space if super slow compared to actual memory, therefore the computation takes forever.  And, in the end, it is this swap space and this slowness that keeps the algorithm running forever, instead of overheating your CPU or making the machine completely frozen and out of memory.

Better approach to the problem

ML techniques do not understand words or strings in general, they only understand numbers.  textblob hides this fact from you because it performs the feature extraction behind the scenes.  That works for toy problems but will likely impair the solution of any more complex problem (be it because of problem complexity or amount of data, or both).

You need to take control of the feature extraction yourself.  A Naive Bayes classifier is a rather simple algorithm and will work well with a plethora of extraction techniques.  Term Frequency (simply counting the number of each distinct word in every document), will work well enough as long as you do not use expensive python dictionaries to hold the frequencies in memory.  But once you get control of your feature extraction there are many more techniques to use: TF-IDF, n-grams.  I find the sklearn's explanation of text features very enlightening.  But I digress.

Unfortunately nltk (the library under texblob) does not make things any easier.  It requires its training sets to have a dictionary interface.  In other words, without a good deal of hacks nltk cannot train on big amounts of data.

Option 1: Hook into nltk directly

First lets assume that your file is named data.csv and we will try to extract features reading the file line-by-line.  First let me generate a data file:

import nltk
nltk.download('gutenberg')  # just in case
from nltk.corpus import gutenberg
corpus = gutenberg.words('melville-moby_dick.txt')
with open('data.csv', 'w') as f:
    for i in range(10**4):  # 10k
        f.write(' '.join([w for w in corpus[i*3:i*3+3] if w != ','])
                + ',' + str(i%2) + '\n')
f.close()

This was only generation of some data, nothing too relevant to the problem at hand.  I do not have your file so I just generate a random 10k rows CSV using the words from Moby Dick text at random.

Onto the actual code.  We will still use texblob's Naive Bayes but will not allow it to construct its huge list of dictionaries.  Instead, we will hook into it directly:

from textblob.classifiers import NaiveBayesClassifier
import csv

cls = NaiveBayesClassifier('')

class FeatureDict(object):
    def __iter__(self):
        with open('data.csv') as f:
            for row in csv.reader(f):
                yield {w: w for w in row[0].split()}, row[1]

cls.train_features = FeatureDict()  # this is passed to nltk
cls.classify('bang Boom!')

It is a hack but runs in constant memory.

Option 2: switch to sklearn

sklearn's vectorizers will perform the operation within NumPy (or within
the NumPy arrays inside pandas' data frames).  Which are memory efficient - especially if compared with thousands of python dictionaries.

With sklearn one will need to do this in two steps: one for data vectorization and another for the prediction.  make_pipeline joins the steps.  Assuming the same data prepared in data.csv we can do:

from sklearn.feature_extraction.text import tfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline
import pandas as pd

data = pd.read_csv('data.csv', names=['doc', 'label'])
# some words are badly formatted, sklearn have issues with them
data = data.dropna()
model = make_pipeline(TfidfVectorizer(), MultinomialNB())
model.fit(data['doc'], data['label'])
model.predict(['buggy ho!'])

The sklearn solution is slightly more elegant but may blow memory at some point (500k samples perhaps).  In that case, one would need to hack sklearn itself to do things in constant memory.


Practical attention models Attention is all you need is a nice paper that suggests using
positional encodings as an alternative to RNNs in their Transformer architecture.
GPT-2 and GPT-3 are examples of using this architecture which
are trained on input data of a massive scale.
Is there a paper and a model that uses positional encodings
and outcompetes RNN/LSTM based models for small scale datasets (MBs of text data, not terabytes)?
If there are many, which ones are the leading ones in production applications?

Is there a paper and a model that uses positional encodings and outcompetes RNN/LSTM based models for small scale datasets (MBs of text data, not terabytes)?

Yes, there are several. Similar to GPT, they still pre-train on terabytes of data. But the embedding they learn generalize well. Then you can fine-tune on a much smaller dataset. It works much in the same way as transfer learning on a CNN where a model first is trained on ImageNet and then trained on a specific task. It tends to give better results than RNN/LSTMs.

If there are many, which ones are the leading ones in production applications?

The one that sees most use is definitely BERT. Here is a really nice explanation of how it works. This transformers library from Huggingface makes it really easy to work with BERT and other transformers that have already been pre-trained.


Handle Unbalanced data I have a data-set with 2 target classes.
In training dataset, the ratio of the 2 classes are 1:93

With my neural network, the current accuracy is 63%.
I tried undersampling, oversampling, equal sampling but not improvements.

You have not specified that what neural network you are using but as comments, you should try to fit your data first. You have to try to find a model that learns your training data. For this purpose you don't have to increase the number of data, at least not at this stage. You should try to find a good model which suits your data. For this purpose you have to change the hyper-parameters of your neural network, e.g. number of layers or number of neurons in layers. You can take a look at here and here which the former can help you and the latter helps you understand the features learned by CNNs in case you are using them.

For using F1 score in Keras I've not seen but you can implement it and pass it to compile method, take a look at here.


What does it mean when someone says "Most of the data science algorithms are optimization problems" I was trying to understand the Gradient Descent algorithm from this article and the author says

  Most of the data science algorithms are optimization problems

I come from software engineering background trying to get a basic understanding of data science. Can someone help me explain what this means?

Most algorithms try to minimizes some objecive functions.

For example, in linear regresssion, given $(x_i, y_i)$, we try to find $\hat{y}_i= \alpha_0 + \sum_{j=1}^d \alpha_j x_{i,j}$ and we we want it to be close to $y$. We try to minimize the mean square error in our estimation.

That is our objective function is
$$\min _\alpha\frac1n \sum_{i=1}^n (y_i-\alpha_o- \sum_{j=1}^d \alpha_j x_{i,j})^2.$$

We might have a model with unknown parameters and we can use maximum likelihood to find out our model, in that case, we maximize the likelihood function. Again, we get another optimization problem.

In general supervised classication error, we are trying to minimize the error which is typically formulated as a minimization of a loss function.  In SVM, we are trying to find a boundary that maximizes the margin between the two classes, again, we are trying to maximize an objective function.


LSTM - divide gradients by number of timesteps IMMEDIATELY or in the end? From this answer I know that

the gradient of an average of many functions, is equal to the average of the gradients of those functions taken separately.
The error gradient that you want to calculate for gradient descent is $\nabla_{\theta} C(X, \theta)$, which you can therefore write as:
$$\nabla_{\theta} C(X, \theta) = \nabla_{\theta}(\frac{1}{|X|}\sum_{x \in X} L(x, \theta))$$
The derivative of the sum of any two functions is the sum of the derivatives, i.e.
$$\frac{d}{dx}(y+z) = \frac{dy}{dx} + \frac{dz}{dx}$$
In addition, any fixed multiplier that doesn't depend on the parameters you are taking the gradient with (in this case, the size of the dataset) can just be treated as an external factor:
$$\nabla_{\theta} C(X, \theta) = \frac{1}{|X|}\sum_{x \in X} \nabla_{\theta} L(x, \theta)$$

Question:
When working with LSTM specifically, am I indeed allowed to apply this $\frac{1}{|X|}$ at the end of the backprop once I've summed-up gradients of weights across all timesteps?
Or should I always apply $\frac{1}{|X|}$ straight away, to any gradient flowing into the top layers of my network?

This bothers me because LSTM is not merely multiplying things like a basic RNN would. Instead, LSTM has summation, which then gets multiplied with other things as we descend to earlier timesteps.
For example, this happens when we have to add some gradient flowing from all the 4 gates the LSTM into grad_wrt_resultAtPreviosTimesep (It will be required to compute grad during previous timestep once we get there)
Using an overly-simplified algebra example:
I feel that multiplying immediately by $\frac{1}{X}$ would represent the left side of this expression:  $(\frac{1}{X}a + 40)*100 \neq \frac{1}{X}
(a+40)*100$. On the other hand, multiplying by $\frac{1}{X}$ in the end of backprop would represent the right side.
Edit:
Just as a reminder, here is what LSTM looks like at each timestep, taken from this blog:

Answer to your question is it doesn't matter. The gradient is just a product of Jacobians (because of the chain rule), so it doesn't matter if you multiply the result or the intermediate multiplicand, these give the same result. Or, better yet, is to multiply the function you're differentiating, since $\tfrac{d(\alpha f(x))}{dx} = \alpha f'(x)$. So instead of manipulating the gradients you should just average your loss over whatever you want it to be averaged over.

Note that I didn't assume anything about the function above. It's because these are fairly general basic rules, and they apply to all kinds of functions, RNNs and LSTMs included.

That said, you might not want to actually average your loss (remembers, it's the same as averaging the gradients) over the timesteps. Average over inputs x – yeah, sure. Over timesteps? Not necessarily a good idea.

If you do average over timesteps, then you essentially decrease importance of longer sequences, and thus your model will care less of making more mistakes in them, whereas if you do not average over timesteps (or if they are all of the same length), then every timestep-loss has the same weight, and the model cares about all of them equally.


Build Deep Belief Autoencoder for Dimensionality Reduction I'm working with a large dataset (about 50K observations x 11K features) and I'd like to reduce the dimensionality. This will eventually be used for multi-class classification, so I'd like to extract features that are useful for separating the data. Thus far, I've tried PCA (performed OK with an overall accuracy in Linear SVM of about 70%), LDA (performed with very high training accuracy of about 96% but testing accuracy was about 61%), and an autoencoder (3 layer dense encoder with 13000 - 1000 - 136 units, respectively, which performed about the same as PCA). I've been asked to try a Deep Belief Network (stack of Restricted Boltzmann Machines) in this problem.
Thus far, I foresee two challenges. First, I have access to a GPU that can be used, but I don't see many implementation of DBNs from the major players in the neural net community (e.g., TensorFlow/Keras, PyTorch), which means that this will need to be implemented on a CPU, bringing up challenge number two. Second, implementing this will require significant memory and will be pretty slow. This brings up my question: Are there any implementations of DBN autoencoder in Python (or R) that are trusted and, optimally, utilize GPU? If not, what is the preferred method of constructing a DBN in Python? Should I use sklearn?

Unlike Autoencoders, Botzmann Machines (restricted or not) do not have an output layer and thus classified as deep generative models.
There is a variety of implementations in Pytorch. This one is GPU compatible (https://github.com/GabrielBianconi/pytorch-rbm) and I have found it particularly helpful in the past.
RBMs can come quite handy in a variety of tasks such as

Dimensionality reduction
Collaborative filtering for recommender systems
Feature learning
and others.

This was an interesting read in case you want to find out more about RBMs. https://heartbeat.fritz.ai/guide-to-restricted-boltzmann-machines-using-pytorch-ee50d1ed21a8


Why increasing the number of units or layers does not increase the accuracy and decrease the loss? I have an LSTM neural network; when I increase the number of units, layers,  epochs or add dropout, it seems it has no effect and still I have persistent errors and accuracies like the following:

  loss: 3.5071 - acc: 0.0981 - val_loss: 6.7042 - val_acc: 0.0122

Why this happens and how can I fix it?

The title of your question and the question itself are somehow different but I'll try to answer the question, the meaning of decreasing loss without changes in accuracy.

The reason is simply due to using probabilities. For instance, for classification task if you have an output $0.7$ for input and the last layer is a softmax, then you classify the input as that class which has $0.7$. Imagine you train more and that output changes to something like $0.95$. Consequently, the accuracy does not change because you already classify it as what it really is but the loss lessens.

To answer the question which is in the body of your post, there can be numerous reasons that I'll try to refer to them.

One of the possibilities is that your data of different classes have overlap in the current feature space. This may lead to high Bayes error. For instance, suppose you have two same inputs and the label of them are contradictory. In this situation, your performance cannot be improved. To check it whether you've got this problem or not, take a look at the histogram of your data.

Another problem can be the weakness of LSTM networks which cannot memorise numerous things. LSTM models are very good at things like considering the gender of a subject or the plural or singular form of subjects but in cases where they should consider many things simultaneously, they have difficulties.

Another reason can be the incorrect way of using dropout. At first, do not use it and let your network overfits the training data to find a good model. After fitting your data, try to use dropout.

You can also test Stacked LSTMs which are powerful models.


Does anyone have memory utilization benchmark for random forest and xgboost? I want to compare which technique has higher memory utilization while training on the same dataset

Try to use memory_profiler.
All what you need to do is to decorate your training function:

@profile
def train_xgb():
   gb = xgb.XGBClassifier(learning_rate=0.1, n_estimators=100, subsample=0.8, max_depth=6)
   gb.fit(new_trainX,new_trainY)


How to merge columns, value count them and then plot the results? How do I get from a dataframe with multiple columns that have similar values and need to be merged:

df1 = pd.DataFrame({'firstcolumn':['ab', 'ca', 'da', 'ta','la'],
'secondcolumn':['ab', 'ca', 'ta', 'da', 'sa'], 'index':[2011,2012,2011,2012,2012]})

To a crosstab that tells me for each year how many values were collected?

Index ab ca da ta sa la
2011 2  0  1  1  0  0
2012 0  2  1  1  1  1

Also, how could then plot the table?

It can be done like:

import pandas as pd
melted = pd.melt(df1, id_vars=["index"], var_name="Var", value_name="Score").dropna()
table=pd.crosstab(index=melted['index'], columns=melted['Score'])
%matplotlib inline
table.plot.bar() #for simple axes subplots


What are the true error and the sample error? I am a student and I am studying machine learning. I am focusing on the concept of evaluation of an hypotesis.
What I have seen is that there are two types of error: true error and sample error.
The true error of an hypotesis $h$ with respect to a target function $f$ and a distribution $D$ is the probability that an hypotesis $h$ misclassifies an instance $x$ drawn according to $D$, and it is computed as:
$error_D(h)=Pr_{x\in D}[f(x)\neq h(x)]$
while the sample error of an hypotesis $h$ with respect to a target function $f$ and data sample $S$ is the proportion of examples that $h$ misclassifies:
$error_S(h)=\frac{1}{n}\sum _{x\in S}\delta (f(x)\neq h(x))$
where
$\delta (f(x)\neq h(x))=1$ if $f(x)\neq h(x)$ and $0$ otherwise.
I ask this question because I have not clear what these errors are.
Moreover, I have seen that the true error cannot be computed, while we can compute only the sample error. I don't understand why.
Can somebody please help me understand?

The true error represents the probability that a randomly drawn instance from the entire distribution is misclassified while the sample error is the fraction of sample which is misclassified.
As true error represents entire population it becomes difficult to calculate hence we use sample to check our hypothesis and use evaluation methods to check it's confidence level. A sample might not be a true representation of population so the difference in results are sample error. We try different sampling methods so that there is no bias in choosing a sample like randomised and stratified sampling.
For more you can refer this


Train an LSTM neural network with time series containing seasonal and trend I am working on a project for predicting the number of DNS queries from the site:
DNS queries statistics. The data I use is minutely data, which means the number of DNS queries of every minute.

If you look at the number of DNS queries from South Korea or any other countries, it has the seasonality and trend characteristics: increase then decrease day by day.

The requirement for me is: given the number of DNS queries of every minute, then for the given data of 20 previous minutes, predict the number of DNS queries in the next 20 minutes.

My problem is: my trained LSTM could not detect these factors, it always predicts my data to decrease everytime.

I have employed some hand-defined features like the minute of the day, period of the day like morning/afternoon... But it still keeps the same problem.

So I want to ask if there is any possible improvement to make my LSTM to understand when it should decrease and when it should increase with the seasonal data?

I don’t know if the seasonality will be achievable in your predictions, due simply to the timeframes you we using. If you see a daily up—down movement, but only provide 20 minutes for a prediction, how will the Model know whether or not it is at a turning point? You would perhaps need to include other features that contain that information - perhaps even the time stamp would suffice.

It is odd, that the model always predicts a downward movement — I would have expected it to simply continue on the current path (up or down), assuming you have both directoins in your training data...?

Perhaps you could look into some ideas used in common timeseries analysis methods, like separating the seasonality, trend and noise and feeding them separately to the model. Search for terms SARIMA, ARIMA, seasonality and cycles. (S)ARIMA stands for “seasonal autoregressive integrated moving average”, and represents a common way to look at data over time using previous values (autoregressivej, the differences of current value to previous values (integrated) and a moving average of past time steps (moving average).

The terms generated would of course catch the phases where the value changes direction and so be able to model seasonality fairly well.


Neural Network Golf: smallest network for a certain level of performance I am interested in any data, publications, etc about what is the smallest neural network that can achieve a certain level of classification performance.  By small I mean few parameters, not few arithmetic operations (=fast).

I am interested primarily in convolutional neural networks for vision applications, using something simple like CIFAR-10 without augmentation as the benchmark.  Top-performing networks on CIFAR in recent years have had anywhere between 100 million and 0.7 million parameters (!!), so clearly small size is not (always) a bad thing.

Small networks are also in general faster to train and overfit less.  Moreover, recent work on Knowledge Distillation, FitNets, etc show ways of making smaller networks from large networks while preserving most of the performance.

Another question is, what is the best performance achievable with a network no larger than a fixed size?

Examples of especially small networks that get good performance (100k parameters with 10% on CIFAR, anyone?) or systematic studies of the size vs performance tradeoff would be appreciated.

You might be interested in this paper that explores a few of the questions you are asking: http://arxiv.org/pdf/1312.6184.pdf. It is aptly titled: "Do Deep Networks Really Need to Be Deep?"

The crux of the matter is that deep networks allow for a LOT of non-linearity in the data that is being described. For CIFAR-10, I suspect that something similar to what works well for MNIST will work well. Yan LeCunn's page about MNIST results gives a lot of clues about models with few parameters: http://yann.lecun.com/exdb/mnist/

The double-problem of image classification is that in addition to the classification you need to extract features somehow. Often the parameters in deep networks go towards feature extraction rather than the classification itself, which is usually a softmax multi-class logistic regression classifier.


when speed is not a major concern, what is the best ML algorithm for high dimensional data? we are trying to build a predicting model using machine learning algorithms.

I have a use case where the input data have a very high dimension. Each sample point has 20000 features.

we have a decent training sample set with around 1 million training samples, if necessary, we can acquire more, say, 2 or 3 million.

We are not very speed sensitive, it's not like a recommendation system that needs to respond within a second. The application allows us to spend up to a couple of minutes for one prediction. Nevertheless, we hope the algorithm can be run in parallel mode in the future.

given above description, what kind of algorithm would you suggest?

Our biggest concern is over fitting, with so many features, it seems that we are doomed to over-fit.

We were trying to do something alone the line of nearest neighbor, but with these many features, calculating distance sounds like a mission impossible. Maybe we should do PCA to do dimension reduction first?

Any comment is welcomed!

There is no one-size-fits-all

Also, data dimensionality is not the same everywhere. Text data, which is inherently sparse, has a very different intrinsic dimensionality than e.g. random Gaussians.

For text data, linear SVMs are known to work very well.

RBF kernels do not work well with high-dimensional data, because they are distance-based at the core, and choosing the sigma parameter becomes next to impossible.

If you can "fold" dimensions, you also get very different behavior. Im image recognition, you usually have thousands of pixels. However, you never look at all of them at once. Instead, you use convolutional kernels that move over the data space, and they may have only say 32x32 pixels. That is still 1024 dimensions, but not millions anymore.


matplotlib subplots_adjust - meaning of parameters What are the meaning of values in subplots_adjust ?

  left  = 0.125  # the left side of the subplots of the figure

The documentation has number 0.125, etc but there is no explanation.

These values represent the distance of your subplot from the boundary of the figure. Its value is between 0 & 1. For example, if you set top=1 the upper boundary of your subplot will coincide with the upper boundary of your figure and if you set top=0 the upper boundary of your subplot will coincide with the lower boundary of your figure and the plot will be entirely squashed down.

It is the percent of figure height and figure width.

A better way to see how this works is to plot a graph and the use the option Configure Subplots, in the window that pops up, to adjust these values. Try this and you will able to appreciate the use of these values in a much better way.


Why use two LSTM layers one after another? In the example on the Keras site, seq2seq_translate.py on line 189, there is a LSTM layer after another (the first with return_sequences=True), but another example lstm_seq2seq.py which does the same thing but letter-by-letter uses only one LSTM in the encoder. My code looks like:

encoder = LSTM(latent_dim, return_sequences=True)(encoder_inputs)
encoder_outputs, state_h, state_c = LSTM(latent_dim, return_state=True)(encoder)

My question is why does the word-by-word version use two LSTM layers? And why is the return_sequences used?

About your first question:
It is because word-by-word NLP model is more complicated than letter-by-letter one, so it needs a more complex network (more hidden units) to be modeled suitably.

About your second question:
When you want to use two-staged LSTMs, the hidden sequence of first LSTM must be used as input of the second LSTM and the return_sequences option is used to do this.


Binary classificaiton for weather data if its class 1 or class 0 alert I am working on weather data and it has few features that are independent variables such as severity, severity_id, urgency_id etc ... Based on these values, I would like to classify alerts into class 0 or 1. For example, below is row item from data source

Alert | Severity | Sev_Id | Urg_Id | Event      | Sys_Rec(Target Variable)
--------------------------------------------------------------------------
dummy | Extreme  |   1    |    1   |   STORM    |      1
dummy | Minor    |   3    |    5   |   RIPTIDE  |      0
dummy | Extreme  |   1    |    1   |   HURRICANE|      1

For severity_id 1 it should be class 1 (Yes) and for others its class 0 (No).The objective is to build a general binary classifier using decision trees. So I started with DTClassifier, but later I realized it could also be done with logistic regression. I am confused which would be a better fit for this kind of data for classification.

Please advice and give me some starting points.

Which model you choose in the end depends on your data. We cannot really answer this for you. Only practice and trials and errors will actually help you doing this.

For binary classification, you can choose among a very diverse range of models, from Logistic Regression to SVMs to Random Forests to Neural Networks. As a rule of thumb, you can use Occam's Razor => for two models giving the same performance, always choose the simplest one.

So my advice is: try out a few model, compare their performance on a testing set and select whichever works best for you...


Reduce drug spend using medical claims analytics I have access to medical claims data from a US health insurance company.

I believe there's an opportunity to find some cost savings by switching the site of service from high cost outpatient facilities to possibly home infusion or physician for some procedures.

An example would be that Botox costs(in dollars) the company 10000 per claim at the outpatient facilities compared to 3000 at the physicians office.

I am interested in learning about how a data scientist would approach this problem of reducing costs from medical claims.

Could I run a cluster analysis on the population of claims to see who is most similar to the existing Botox patients who go to their physician and target these to move them to the lower cost sites of service?

Things depend upon the dataset we have on hand here. See what observations you can grab from them using combination of different features.

If i were to approach this problem, first i would find what kind of patients go to their physicians (Age, area of living, insurance plan, previous diseases, previous admissions, income zone etc). Then would analyse which outpatients facilities serves in which area, their average charges & other features you can find.

After the above analysis i would go with prediction algorithm so to make a theory that an particular set of groups will visit that facility. And using that deduction along with clustering (what i have in mind is 2 clusters 1 going to physicians & other with outpatient facilities) i will select the overlapping clients (I mean the ones who have features similar to lower cost going site but prefer higher cost sites) of insurance company to move towards lower cost site.

Reason behind using predictive model is because it will help you in predicting the future customers, just normal clustering will just work with the current data. So to make it effective for future i would go with predictive model -> clustering model.


Feature importance decision algorithms I have a dataset with 100+ feature columns.
My client asked me to choose "the top 10 most important features" from the 100+.

From this post, I learnt that Random Forest can help me ranking the importance of features.

My Question is, is there any other algorithms can do this job? (also if someone can direct me to some tensorflow examples on this.)

P.S. To explain, the dataset is about eCommerce, with 100+ features and one column depicting "how much the user spent on the website". By "important features", I mean the features that can best predict the user-spent column.

Do you have some kind of outcome associated with the dataset? By 'top 10 most important features', do you mean most important in achieving a specific goal? If you have some kind of outcome column (a score or pass/fail) you can solve this through a good old fashioned regression technique. If you don't, you might have to redefine what importance means and use a technique like PCA to find the most variable features.

A worked example: What is most important for students to pass a test?

For this made up example, we are tracking 30 students and seeing what is more important for passing a test, intelligence (IQ) or study time (in hours).
The code below is in R, but I'm just using it to illustrate the process.

# Generating some sample data
set.seed(300)
IQ <- as.integer(rnorm(30,mean=100,sd=10))
study_hours <- as.integer(rlnorm(30,mean=1.5,sd=0.75))
score <- as.integer(100*tanh( 0.0012*(6*IQ + 20*study_hours
+rnorm(30,mean=150,sd=10))))

    IQ study_hours score
1  113           3    78
2  108           5    80
3  104           2    75
4  107           3    76
5   99           8    80
6  115           5    80
7  108           2    77
8  103           4    76
9  112           3    78
10 103           1    73
11 122           8    84
12  99           2    72
13  86           3    69
14 100           6    77
15 105          12    84
16 100          12    83
17 114           9    84
18  87           7    74
19  97           6    76
20 102           5    77
21  96           9    79
22 106          15    86
23  96           1    71
24 101           4    76
25  88           4    71
26  96           4    75
27 102           2    74
28 113           4    79
29  99           2    73
30 110           2    77

Both IQ and study time seem to have a positive effect on exam score. But how do you tell which one is more important?
Linear regression will give you an equation that fits a straight line to the data, and the slope of the line in each direction gives the relative importance of that feature.

But first you have to standardise the numbers, otherwise the bigger set of numbers (IQ) is going to throw things out of whack.

# Standardising the variables.
# (You take the number, subtract the mean and divide by the standard deviation)

standardisedIQ <- (IQ-mean(IQ))/sd(IQ)
standardised_study_time <- (study_hours-mean(study_hours))/sd(study_hours)

# Perform linear regression. Output variable is score and input variables are
standardised IQ and standardised study time

L <- lm(score ~ standardisedIQ + standardised_study_time)
L

The output is:

Coefficients:
            (Intercept)           standardisedIQ  standardised_study_time
                 77.133                    2.507                    3.182

which can be read as: Everything else being equal, the average score is 77.1, and IQ has an 'effect' of 2.5 and study time an 'effect' of 3.2. Since the study time number is larger, study time has more of an effect than IQ in this particular exam.

How does this translate to your problem?
In your case, you have 100 variables. Just standardise them, set up the appropriate regression on those 100 variables and the outcome variable, and choose the top ten variables with the highest (absolute value) scores.

The specific type of you use will depend on what outcome variable you have. If it's a pass/fail rather than a score, you want to use logistic regression.

As far as I can tell, doing things this way isn't completely statistically sound. However, if your client is non-mathematical and only wants 'an idea' of what the different features are doing, that shouldn't matter.

If you do not have any usable outcome variables, using PCA (principal component analysis) may help. It will highlight the 'variability' or 'salience' of the various features, but not necessary their importance towards any goal. Keep in mind that there are a bunch of caveats with using PCA, and doing some sort of data standardisation is necessary if you're using units of measurement (i.e. one feature is measured in metres, another in seconds, etc.).


K-nearest neighbors complexity Why does the complexity of KNearest Neighbors increase with lower value of k? And when does the plot for k-nearest neighbor have smooth or complex decision boundary? Please explain in detail.

And also , given a data instance to classify, does K-NN compute the probability of each possible class using a statistical model of the input features or just gets the class with the most number of points in favour of it?

The complexity in this instance is discussing the smoothness of the boundary between the different classes. One way of understanding this smoothness complexity is by asking how likely you are to be classified differently if you were to move slightly. If that likelihood is high then you have a complex decision boundary.

For the $k$-NN algorithm the decision boundary is based on the chosen value for $k$, as that is how we will determine the class of a novel instance. As you decrease the value of $k$ you will end up making more granulated decisions thus the boundary between different classes will become more complex.

You should note that this decision boundary is also highly dependent of the distribution of your classes.

Let's see how the decision boundaries change when changing the value of $k$ below. We can see that nice boundaries are achieved for $k=20$ whereas $k=1$ has blue and red pockets in the other region, this is said to be more highly complex of a decision boundary than one which is smooth.

First let's make some artificial data with 100 instances and 3 classes.

from sklearn.datasets.samples_generator import make_blobs
X, y = make_blobs(n_samples=100, centers=3, n_features=2, cluster_std=5)

Let's plot this data to see what we are up against

Now let's see how the boundary looks like for different values of $k$. I'll post the code I used for this below for your reference.

$k$ = 1

$k$ = 5

$k$ = 10

$k$ = 20

The code

The code used for these experiments is as follows taken from here

from sklearn import neighbors

k = 1
clf = neighbors.KNeighborsClassifier(20)
clf.fit(X, y)

from matplotlib.colors import ListedColormap
import matplotlib.pyplot as plt

cmap_light = ListedColormap(['#FFAAAA', '#AAFFAA', '#AAAAFF'])
cmap_bold = ListedColormap(['#FF0000', '#00FF00', '#0000FF'])

# Plot the decision boundary. For that, we will assign a color to each
# point in the mesh [x_min, x_max]x[y_min, y_max].
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                         np.arange(y_min, y_max, h))
Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])

# Put the result into a color plot
Z = Z.reshape(xx.shape)
plt.figure()
plt.pcolormesh(xx, yy, Z, cmap=cmap_light)

# Plot also the training points
plt.scatter(X[:, 0], X[:, 1], c=y, cmap=cmap_bold)
plt.xlim(xx.min(), xx.max())
plt.ylim(yy.min(), yy.max())
plt.show()


define number of features I try to understand my time series data for my ANN for a regression problem. I have temperature values measured with a sampling rate of 10.000 values / 1 second. So I measure in 1 second 10.000 different temperature values. This I repeat with certain time intervals. Do I have than 10.000 different features or only one feature as input dimension for my lstm neural network ? I want to shape my data for a lstm encoder decoder.

How I understand your data is a single feature of temperature. LSTM works with RNN time series like (batch_size, timesteps, feature). In your case it is (none, 10000, 1). Forget about the first one, it's technical.


The reason for this error is that you are trying to flatten an already flat layer. The output of your model is (batch_size, 4), which cannot be flattened further. To simply fix the error remove the flatten layer from your code.

However, when fine-tuning a pretrained model, you should first remove the top layers of that model before adding your own. The reason is that these layers are trained for classification on a task different than yours.

If I were you I'd drop the last two layers of your pretrained model:

# code same as before ...

x = model.layers[-3].output  # Flatten layer output

# don't add flatten again

for fc in fc_layers:
    x = Dense(fc, activation='relu')(x)
    x = Dropout(dropout)(x)

predictions = Dense(num_classes, activation='softmax')(x)

finetune_model = Model(inputs=model.input, outputs=predictions)