Convolutional neural network overfitting. Dropout not helping I am playing a little with convnets. Specifically, I am using the kaggle cats-vs-dogs dataset which consists on 25000 images labeled as either cat or dog (12500 each).

I've managed to achieve around 85% classification accuracy on my test set, however I set a goal of achieving 90% accuracy.

My main problem is overfitting. Somehow it always ends up happening (normally after epoch 8-10). The architecture of my network is loosely inspired by VGG-16, more specifically my images are resized to $128x128x3$, and then I run:

Convolution 1 128x128x32 (kernel size is 3, strides is 1)
Convolution 2 128x128x32 (kernel size is 3, strides is 1)
Max pool    1 64x64x32   (kernel size is 2, strides is 2)
Convolution 3 64x64x64   (kernel size is 3, strides is 1)
Convolution 4 64x64x64   (kernel size is 3, strides is 1)
Max pool    2 32x32x64   (kernel size is 2, strides is 2)
Convolution 5 16x16x128  (kernel size is 3, strides is 1)
Convolution 6 16x16x128  (kernel size is 3, strides is 1)
Max pool    3 8x8x128    (kernel size is 2, strides is 2)
Convolution 7 8x8x256    (kernel size is 3, strides is 1)
Max pool    4 4x4x256    (kernel size is 2, strides is 2)
Convolution 8 4x4x512    (kernel size is 3, strides is 1)
Fully connected layer 1024 (dropout 0.5)
Fully connected layer 1024 (dropout 0.5)

All the layers except the last one have relus as activation functions.

Note that I have tried different combinations of convolutions (I started with simpler convolutions).

Also, I have augmented the dataset by mirroring the images, so that in total I have 50000 images.

Also, I am normalizing the images using min max normalization, where X is the image

$X = X - 0 / 255 - 0$

The code is written in tensorflow and the batch sizes are 128.

The mini-batches of training data end up overfitting and having an accuracy of 100% while the validation data seems to stop learning at around 84-85%.

I have also tried to increase/decrease the dropout rate.

The optimizer being used is AdamOptimizer with a learning rate of 0.0001

At the moment I have been playing with this problem for the last 3 weeks and 85% seems to have set a barrier in front of me.

For the record, I know I could use transfer learning to achieve much higher results, but I am interesting on building this network as a self-learning experience.

Update:

I am running the SAME network with a different batch size, in this case I am using a much smaller batch size (16 instead of 128) so far I am achieving 87.5% accuracy (instead of 85%). That said, the network ends up overfitting anyway. Still I do not understand how a dropout of 50% of the units is not helping... obviously I am doing something wrong here. Any ideas?

Update 2:

Seems like the problem had to do with the batch size, as with a smaller size (16 instead of 128) I am achieving now 92.8% accuracy on my test set, with the smaller batch size the network still overfits (the mini batches end up with an accuracy of 100%) however, the loss (error) keeps decreasing and it is in general more stable. The cons are a MUCH slower running time, but it is totally worth the wait.

Ok, so after a lot of experimentation I have managed to get some results/insights.
In the first place, everything being equal, smaller batches in the training set help a lot in order to increase the general performance of the network, as a negative side, the training process is muuuuuch slower.
Second point, data is important, nothing new here but as I learned while fighting this problem, more data always seems to help a bit.
Third point, dropout is useful in large networks with lots of data and lots of iterations, in my network I applied dropout on the final fully connected layers only, convolution layers did not get dropout applied.
Fourth point (and this is something I am learning over and over): neural networds take A LOT to train, even on good GPUs (I trained this network on  floydhub, which uses quite expensive NVIDIA cards), so PATIENCE is key.
Final conclusion: Batch sizes are more important that one might think, apparently it is easier to hit a local minimum when batches are larger.
The code I wrote is available as a python notebook, I think it is decently documented.


Best practice for developing using Spark I am looking for any tips and best practice on how to develop applications using Spark. I currently have access to a cluster, with data as well as a version of Spark 2.1.0 on an edge node and IntelliJ on my local machine.

I am wondering what the best way would be to go about development of an application to run on the data that is in the cluster. Things on my mind for consideration are the following;

IntelliJ is a nice IDE but I can't store the volume of data I want to look at on my local machine
I can run spark-shell on the edge node but its not great for data exploration and development

How is it best to bridge this gap between data and a nice development environment?

What you want to do is develop your code in IntelliJ, and then package your code and dependencies into an executable jar file using SBT or Maven.

When you have your jar stored locally, you can use spark-submit to transfer the jar to your cluster (along with some other parameters) for execution.

You might also want to take a sample of your data and store it locally so you can run spark locally and test/debug your code in IntelliJ. This can speed up development considerably, and having access to a debugger is a huge help.


Unsure of how to implement an equation in PyTorch I am trying to implement the SummaRuNNer architecture ( Nallapati et al).

The equation I am stuck at in question is:
$$d = tanh(W_{d}\frac{1}{N_{d}}\sum_{j=1}^{N^{d}}[h^{f}_{j},h^{b}_{j}] + b)$$
Where,
$N_{d} = $ Num of sentences in the document, $h^{f}_{j}$ and $h^{b}_{j}$ are the forward and backward passes of the $j^{th}$ sentence, $[]$ is the concatenation operation, $b$ is a bias term and $W_{d}$ is a learnable parameter.

My question is, what is the best way to implement this in PyTorch.

Also, I am aware that there is already a version of this particular architecture implemented in PyTorch here, however, I am not sure if their implementation of this equation is quite right.

Check this GitHub repo for implementing SummaRuNNer with Pytorch.

The PyTorch Implementation Of SummaRuNNer


Is this scheme correct for logistic regression with stochastic gradient descent I am implementing logistic regression with stochastic gradient descent, but it is not working as expected. I've tried many epochs and different learning rates $\alpha$ but the probability of belonging to the correct class oscillates around $0.5$. I have got two classes denoted as $0,1$ and two features. Is this scheme correct?:

Initialize weights $w = [w_0, w_1, w_2]$ to some random number (I choose 1) and loop:

Choose random training sample $x_i = [1, x_1, x_2]$.
Calculate the probability of it belonging to class $1,$
$$P(\hat{y} = 1) = \frac{1}{1+e^{-w \cdot x_i}}$$
Update the weights according to
$$w = w + \alpha x_i(y_i - P(\hat{y} = 1))$$

This scheme makes sense to me since the update to the weights would be big if the probability of belonging to class $1$ was low, if the correct class was $1$ but I can't verify it more than that.

Hard to say without more detail, but isn't your update wrong? you need to subtract rather than add the gradient. Unless alpha is negative, this is wrong.


What is the purpose of Logit function? At what stage of model building process this logit function is used? We have two prominent functions (or we can say equations) in logistic regression algorithm: 1. Logistic regression function. 2. Logit function.

I would like to know: a. Which of these equation(s) is/are used in the logistic regression model building process? b. At what stage of model building process which of these equation(s) is/are used?
I know that logit function is used to transform probability values (which range b/w 0 and 1) to real number values (which range b/w -Inf to +Inf). I would like to know the real purpose of logit function in logistic regression modeling process.

Here are few queries which are directly related to the purpose of logit function in Logistic regression modeling:

Has Logit function (i.e. Logit equation LN(P/1-P)) being derived from Logistic Regression equation or its the other way around?
What is the purpose of Logit equation in logistic regression equation? How logit function is used in Logistic regression algorithm? Reason for asking this question will get clear after going through point no. 3 & 4.
Upon building a logistic regression model, we get model coefficients. When we substitute these model coefficients and respective predictor values into the logistic regression equation, we get probability value of being default class (same as the values returned by predict()). a) Does this mean that estimated model coefficient values are determined based on the probability values (computed using logistic regression equation not logit equation) which will be inputed to the likelihood function to determine if it maximizes it or not? If this understanding is correct then, where the logit function is used in the entire process of model building.
Assume that - "Neither logit function is used during model building not during predicting the values". If this is the case then why do we give importance to logit function which is used to map probability values to real number values (ranging between -Inf to +Inf). a) Where exactly the logit function is used in the entire logistic regression model buidling process? Is it while estimating the model coefficients?
The model coefficient estimates that we see upon running summary(lr_model) are determined using linear form of logistic regression equation (logit equation) or the actual logistic regression equation?

It would be great if someone clarifies my doubts.

Has Logit function (i.e. Logit equation LN(P/1-P)) being derived
  from Logistic Regression equation or its the other way around?

Neither, really. It comes from the probability distribution that you assume in a logistic regression model. First, logistic regression as you probably know is a specific case of a generalized linear model (often abbreviated as "GLM"). A GLM is just that; a generalization of the famous "standard" linear regression model. The "standard" model assumes the conditional distribution of your response, given your predictors (denote this as $Y|X$) is normally distributed. However, in a classification problem we clearly do not have a target variable that can take on any value in the interval $[-\infty, \infty]$, but rather, only two possible outcomes (0 or 1).

A GLM requires three components; a specified probability distribution of $Y|X$ from the exponential family of distributions (important in the context of your question), a linear predictor (i.e. $B_0 + B_1X_1 + B_2X_2 + ...$, denote this as $XB$ in matrix notation), and finally, a specified link function that "links" the linear predictor to the mean (denote this as $\mu$) of the specified probability distribution.

First, let's discuss the probability distribution as this answers the question in quotes. In any logistic regression, you are assuming that $Y|X$ follows a Bernoulli distribution. Recall that a Bernoulli distributed random variable takes on 1 with probability p and 0, with probability 1-p. The mean/expected value of a Bernoulli random variable is therefore $1 *p + 0 * (1-p) = p$. This is clearly much more appropriate to our binary classification problem then say, a normal.

Also recall that I mentioned the importance of the exponential family of distributions. Well, it turns out that all members of this family of distributions have density functions that can be further factored into a very specific form with isolated terms in an exponential. One of these terms is a function that is only an argument of the scale parameter of the distribution. For the Bernoulli distribution, it turns out that this function of the scale parameter is exactly the logit link function, $ln(\frac{p}{1-p})$. Therefore, using the logit link function allows us to map our linear predictor to the exact form of the density function of the Bernoulli distribution.

There are theoretical reasons as to why we in general prefer to use the canonical link function (this is concerned with sufficiency) but there are also many reasons as to why we don't use canonical link functions. For example, the Gamma distribution has support on $[0, \infty]$ but its canonical link function is the negative inverse which runs on $[-\infty, \infty]$. Most prefer to use the log link instead for the Gamma distribution for this reason.

There are other legitimate link functions for classification problems as well. The probit link function, for example, or $ln[-ln(\mu)]$.

  What is the purpose of Logit equation in logistic regression equation? How logit function is used in Logistic regression algorithm? Reason for asking this
  question will get clear after going through point no. 3 & 4.

I basically explained this above. The logit equation, like you correctly state, is used to map the linear predictor $XB$ which runs on $[-\infty, \infty]$ to an interval on $[0,1]$, a probability. That's basically the main purpose of the function but again, there are others you can use that also have supports that run on $[0,1]$, but may have less desirable statistical properties when compared to the logit.

  Upon building a logistic regression model, we get model coefficients. When we substitute these model coefficients and respective predictor values into the logistic regression equation, we get probability value of being default class (same as the values returned by predict()). a) Does this mean that estimated model coefficient values are determined based on the probability values (computed using logistic regression equation not logit equation) which will be inputed to the likelihood function to determine if it maximizes it or not? If this understanding is correct then, where the logit function is used in the entire process of model building.

The estimated model coefficient values ARE determined based on what link function you choose. In the fitting process, the link function is what provides the map between the linear predictor with fitted model coefficients and the mean of the fitted distribution (in our case for logistic regression, the parameter $p$ as explained above). The objective function = log likelihood will end up looking like this:

$$\sum_{i=1}^{n} (y_i * ln(p_i) + (1-y_i) * ln(1-p_i))$$

where $y_i =$ 0 or 1, $p_i =$ predicted probability of class 1 for the ith observation, and $n =$ number of observations.

Notice how there is no linear predictor and no fitted coefficients in this expression. But this is where the link function comes into play; namely,

$X_iB = ln(\frac{p_i}{1-p_i}) \rightarrow p_i = \frac{exp(X_iB)}{1 + exp(X_iB)}$. So the fitted coefficients in the linear predictor are influenced by your choice of link function. There is no way to relate the linear predictor and its coefficients to the loglikelihood function otherwise. We find the most optimal $B$ that maximize the expression above.

  Assume that - "Neither logit function is used during model building not during predicting the values". If this is the case then why do we give importance to logit function which is used to map probability values to real number values (ranging between -Inf to +Inf). a) Where exactly the logit function is used in the entire logistic regression model buidling process? Is it while estimating the model coefficients?

See the above to (hopefully) be convinced that the logit link function is used during model fitting and is not arbitrary. This isn't a very rigorous explanation but I hope it clears up some of your doubts.

  The model coefficient estimates that we see upon running summary(lr_model) are determined using linear form of logistic regression equation (logit equation) or the actual logistic regression equation?

In general, you can typically specify whether you want just the linear predictor (i.e. $X_iB$) or the predicted probabilities (i.e. $p_i = \frac{exp(X_iB)}{1+exp(X_iB)}$). In R this is done via. an argument to the predict function, not sure how it is done in Python but I assume it is similar.


How does Wav2Vec 2.0 feed output from Convolutional Feature Encoder as input to the Transformer Context Network I was reading the Wav2Vec 2.0 paper and trying to understand the model architecture, but I have trouble understanding how audio raw inputs of variable lengths can be fed through the model, especially from the Convolutional Feature Encoder to the Transformer Context Network.
During fine-tuning (from what I have read), even though audio raw inputs within a batch will be padded to the length of the longest input in the batch, the length of inputs can differ across batches. Therefore this implies that the output from the Convolutional Feature Encoder will have varying lengths across batches.
However, the Transformer Context Network has a fixed input dimension; the BASE Wav2Vec 2.0 model uses a transformer of model dimension 768. This means that the output from the Convolutional Feature Encoder must somehow be "manipulated" to become dimension 768 in order to feed into the Transformer.
How is this "manipulation" done? The HuggingFace's Wav2Vec model (see below) showed that there is a Wav2Vec2FeatureProjection layer between the Convolutional Feature Encoder (a.k.a., Wav2Vec2FeatureExtractor) and Transformer Context Network (a.k.a., Wav2Vec2Encoder). The Wav2Vec2FeatureProjection contains a linear layer that takes input with dimension 512 and output dimension 768. How is the input dimension 512 determined when the raw inputs can have varying lengths across batches?
Wav2Vec2ForCTC(
  (wav2vec2): Wav2Vec2Model(
    (feature_extractor): Wav2Vec2FeatureExtractor(
      (conv_layers): ModuleList(
        ...
        (6): Wav2Vec2NoLayerNormConvLayer(
          (conv): Conv1d(512, 512, kernel_size=(2,), stride=(2,), bias=False)
        )
      )
    )
    (feature_projection): Wav2Vec2FeatureProjection(
      (layer_norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
      (projection): Linear(in_features=512, out_features=768, bias=True)
      (dropout): Dropout(p=0.1, inplace=False)
    )
    (encoder): Wav2Vec2Encoder(
      (pos_conv_embed): Wav2Vec2PositionalConvEmbedding(
        (conv): Conv1d(768, 768, kernel_size=(128,), stride=(1,), padding=(64,), groups=16)
        (padding): Wav2Vec2SamePadLayer()
      )
    ...

They key is that the 768 dimensional vector of the transformer is the size of a single input... let me explain.

You start with a variable length audio input
This is passed through a temporal CNN network, which will give you outputs, called $z_1$ to $z_T$ by the paper - where T is variable across the batch and is the number of timesteps in a particular audio input (different for different audio inputs).
Each of these $z$ are passed as 768 dimensional vectors to the
transformer, i.e. there are $T$ 768-dim vectors being passed to your
transformer

As you can imagine, practically you would need to pass a mask to the transformer too, so it knows what the variable $T$ is for the different inputs in your batch.


Predict the outcome of a new time series (mailing) We survey our members on a regular basis on various topics, but with always the same question (basically, do you like it? yes or no), and we classify them based on what is replied, and how many participated in the survey.

When we send the email, we would like to be able to know at what point in time we can estimate that the final result (how many people will fill, what % of yes) is likely to be the one we have now.

We did process the previous surveys, and we can see a pattern where after 6 or 12 hours, the result is similar to the final one (ie when we wait more than a week after the mailing). Obviously, a survey sent friday at 8pm takes longer to reach a conclusive stage than the ones sent on tue at 9am. They are other factors too, for instance if our mail provider experienced delay delivering, then of course we need to wait longer.

Is there a way to know how long to wait (3, 6, 12... hours?) before we get a solid prediction of the result?

Is there an algorithm that can predict the result at any point in time and give an estimate on how likely the estimate is correct (p value like)?

Think about this problem in two parts.

Part 1)
How many samples do you need in order to have the desired confidence in the Null hypothesis?

Part 2) How long does it take to get that many samples?

Part 1 can be answered with A-priori power analysis.  Use this technique to determine the sample size needed to get the p-value desired.

Part 2 can be addressed with an algorithm applied to the data you have collected thus far.  Your predictors in this case would be:

Column 1: the probability that your statistic of interest has leveled off (is not changing significantly with additional input).

Column 2: date/day/time start. You probably will need to codify this.

Column 3: number of samples at finish.  These should be the number which you determined from your prior analysis.

Your response/output in this case would be:

Column 4: the date/day/time finish.

Of course would need to do some data analysis to determine the appropriate algorithm to use on this data set.  The end product should be a function that takes as input a target probability, date/day/time start and a target sample size, while outputting a date/day/time finish.

I hope this at least gives you some ideas on how to proceed.


Varying results when calculating scatter matrices for LDA I'm following a Linear Discriminant Analysis tutorial from here for dimensionality reduction. After working through the tutorial (did the PCA part, too), I shortened the code using sklearn modules where applicable and verified it on the Iris data set (same code, same result), a synthetic data set (with make_classification) and the sklearn-digits dataset.

However, then I tried the exact same code on a complete different (unfortunately non-public) data set that contains spectra recordings of two classes. The LDA crashes at the eigenvector verification part, where the $\lambda \mathbf{v}$ is supposed to be almost equal to $S_W^{-1} S_B \mathbf{v}$ (with $\lambda$ being the eigenvalue and $\mathbf{v}$ the corresponding eigenvector; $S_W$ and $S_B$ are the in/between-class scatter matrices). The first vector to be wrong seems at random positions, meaning each run it's a different vector that's causing this error.

I suspect it's related to rounding during calculations, since I get complex eigenvectors. For the PCA I just discarded the complex part (I think I read it somewhere in this forum), but this approach does not seem to work with LDA. Has anybody encountered similar problems or knows what's wrong?

Following is my code for the analysis, which is more or less the same as in the tutorial. I'm using the manual approach, since I'm interested in how many linear discriminants are needed to describe my data. (I'm not sure how to do this with sklearn's LDA.)

def LDAnalysis_manual(X, y):
    n_features = X.shape[1]
    n_classes = len(np.unique(y))

    print("Mean vectors...")
    mean_vectors = []
    for cl in range(n_classes):
        mean_vectors.append(np.mean(X[y == cl], axis=0))
        # print("Mean vector class {}: {}".format(cl, mean_vectors[cl - 1]))

    print("In-class scatter matrix...")
    S_W = np.zeros((n_features, n_features))
    for cl, mv in zip(range(1, n_classes), mean_vectors):
        class_sc_mat = np.zeros((n_features, n_features))  # each class' scatter matrix
        for row in X[y == cl]:
            row, mv = row.reshape(n_features, 1), mv.reshape(n_features, 1)  # column vectors
            class_sc_mat += (row - mv).dot((row - mv).T)
        S_W += class_sc_mat  # sum class scatter matrices

    overall_mean = np.mean(X, axis=0)
    print("Between-class scatter matrix...")
    S_B = np.zeros((n_features, n_features))

    for i, mean_vec in enumerate(mean_vectors):
        n = X[y == i + 1].shape[0]
        mean_vec = mean_vec.reshape(n_features, 1)  # make column vector
        overall_mean = overall_mean.reshape(n_features, 1)
        S_B += n * (mean_vec - overall_mean).dot((mean_vec - overall_mean).T)

    eig_vals, eig_vecs = np.linalg.eig(np.linalg.inv(S_W).dot(S_B))

    print("Eigenvector test")
    for i in range(len(eig_vals)):
        print("\r{:3}".format(i), end=" ")
        sys.stdout.flush()

        eigv = eig_vecs[:, i].reshape(n_features, 1)
        np.testing.assert_array_almost_equal(np.linalg.inv(S_W).dot(S_B).dot(eigv).real,
                                             (eig_vals[i] * eigv).real,
                                             decimal=6, err_msg='', verbose=True)
    __log.debug("\nAll values ok.")

    eig_pairs = [(np.abs(eig_vals[i]), eig_vecs[:, i]) for i in
                 range(len(eig_vals))]  # make list of value & vector tuples
    eig_pairs = sorted(eig_pairs, key=lambda k: k[0], reverse=True)  # Sort tuple-list from high to low

    __log.info("\nEigenvalues (decending):")
    for i in eig_pairs:
        __log.info(i[0])

    tot = sum(eig_vals)
    var_exp = [(i / tot) for i in sorted(eig_vals, reverse=True)]
    cum_var_exp = np.cumsum(var_exp)
    cum_var_exp = cum_var_exp.real
    plot(len(var_exp), var_exp, cum_var_exp)

    idx_98 = next(idx for idx, val in enumerate(cum_var_exp) if val > .98)
    return idx_98 + 1

The LDA crashes for the exact reason you suspected.  You have
complex eigenvalues.  If you use np.linalg.eigh, which was
designed to decompose Hermetian matrices, you will always get real
eigenvalues.  np.linalg.eig can decompose nonsymetric square
matrices, but, as you've suspected, it can produce complex
eigenvalues.  In short, np.linalg.eigh is more stable, and I would suggest using it for both PCA and LDA.
Dropping the complex part of the eigenvalues may have been acceptable in your specific example, but in practice it should be avoided.  Depending on the size of the complex part of the number, it can significantly change the result.  For example think of the multiplication of two complex conjugates.  $(3+ .1i)*(3-.1i)=9-.01=9.01$ comared to $9$ when droping the complex part is a relatively safe, but $(3-2i)*(3+2i)=13$ compared to $9$ is a significant miscalculation.  Using the above method for the eigen-decomposition will prevent this situation from arising.
Remember that one of the assumptions of LDA is that the features are normally distributed and independent of each other.  Try running print('Class label distribution: %s' % np.bincount(y_train)[1:]).  If the counts are not close to being equal, you've violated the first asumption of LDA, and the within-class scatter matrix must be scaled, in short divide each count by the number of class samples $N_i$.  By doing this it should be obvious that computing the normalized scatter matrix is the same as computing the covariance matrix $\Sigma_i$.  $$\Sigma_i=\frac{1}{N_i}S_W=\frac{1}{N_i}(x-m_i)(x-m_i)^T $$
Make sure your scaling your features before you do your PCA/LDA.
If the above doesn't fix your eigenvector verification step I suspect the problem is that the eigenvectors are scalled differently.  Remember from your linear algebra class that a single eigenvalue, $\lambda_i$, has infinitely many eigenvectors, each being a scalar multiple of the others.  $v_i=[1,2,3]$ and $v_i=[2,4,6]$ can both be the eigenvector of $\lambda_i$.  So while you may get different values when calculating values at any given step after the decomposition, the end result should be the same.

Below is a template I use for LDA data compression.  It assumes that you've split your data into a training and test set, the feature space has been properly scaled, and there are three classes in your label vector (you can adjust accordingly).  It plots the individual and cumulative "discriminability" of each linear discriminant and then relies on the lda package in sklearn to transform the feature space using the number of discriminants you intend on using (here I chose to use the first 2).  It also scales the within class scatter matrices by default.

LINEAR DISCRIMINANT ANALYSIS

calculate mean vectors

mean_vecs = []
for label in range(1, 4):
    mean_vecs.append(np.mean(X_train_std[y_train==label], axis=0))
    print('MV %s: %s\n' %(label, mean_vecs[label-1]))

calculate within-class scatter matrix

d = X_train_std.shape[1]
S_W = np.zeros((d, d))
for label, mv in zip(range(1, 4), mean_vecs):
    class_scatter = np.cov(X_train_std[y_train==label].T)
    S_W += class_scatter
print('Scaled within-class scatter matrix: %sx%s' % (S_W.shape[0], S_W.shape[1]))

calculate between-class scatter matrix

mean_overall = np.mean(x_train_std, axis=0)
S_B = np.zeros((d, d))
for i, mean_vec in enumerate(mean_vec):
    n = X_train_std[y_train==i+1, :].shape[0]
    mean_vec = mean_vec.reshape(d, 1)
    mean_overall = mean_overall.reshape(d, 1)
    S_B += n * (mean_vec - mean_overall).dot((mean_vec - mean_overall).T)
print('Between-class scatter matrix: %sx%s' % (S_B.shape[0], S_B.shape[1]))

eigen decomposition

eigen_vals, eigen_vecs = np.linalg.eigh(np.linalg.inv(S_W).dot(S_B))
eigen_pairs = [(np.abs(eigen_vals[i]), eigen_vecs[:,i]) for i in range(len(eigen_vals))]
eigen_pairs = sorted(eigen_pairs, key=lambda k: k[0], reverse=True)
print('Eigendecomposition: \nEigenvalues in decreasing order:\n')
for eigen_val in eigen_pairs:
    print(eigen_val[0])

plot discriminablithy and select number of linear discriminants

tot = sum(eigen_vals.real)
discr = [(i / tot) for i in sorted(eigen_vals.real, reverse=True)]
cum_discr = np.cumsum(discr)

plt.bar(range(1, 14), discr, alpha=0.5, align='center',
        label='individual "discriminability"')
plt.step(range(1, 14), cum_discr, where='mid',
         label='cumulative "discriminability"')
plt.ylabel('"discriminability" ratio')
plt.xlabel('Linear Discriminants')
plt.ylim([-0.1, 1.1])
plt.legend(loc='best')
plt.tight_layout()
plt.show()

from sklearn.lda import LDA

lda = LDA(n_components=2)
x_train_lda = lda.fit_transform(X_train_std)
x_test_lda = lda.transform(X_test_std)
print('Features projected onto %d-dimensional LD subspace' % lda.n_components)


What's a good machine learning model for an univariate data set? Here's my problem scenario:- I have to come up with a power equation as a function of frequency. The plot fits well with a higher order polynomial (4th or 6th) :-

$$Power = \theta_0 + \theta_1 fr^1 + \ldots + \theta_6*fr^6$$

(This is from MS Excel's trend line for a scattered plot) Frequency (x axis) ranges within fixed limits $f_1$ to $f_2$ and I have data from $100$ different devices for this kind of frequency sweep.

With this limited data set what would be a good ML model to train and generalize the coefficients that would work for any unseen device?

Thanks in advance!

Edit:-

While I can't share the exact data-set here but let me share some info about the data- $fr$ ranges from $4060(f_1)$ to $4165(f_2)$; I have an option to go granular i.e step size of $+1$ .. currently I am going $+5$

Where $Pij$ is the power value for the $i$th device and $j$th frequency sample
Question: Shall I treat each device as an example and each frequency as a feature? In that case the problem becomes multi-variate and I don't want that. I want the equation to remain exactly as stated above. The more desired option is to treat each frequency as an example, then how do I treat each of the $100$ devices? definitely not as features.. How to model the problem space into a feature vector $X$ and ans vector $Y$ and param vector $\theta$?

Your question is not very specific. I am assuming that you are looking for the full solution (Language, package, function, template).  So with that in mind I'll point you to the following question:
Polynomial regression using scikit-learn

Half way down the answers is a solution by Salvador Dali for:

Language: Python
Package:scikit-learn
Function: linear_model.LinearRegression()


Graphically Speaking - how weight vector is perpendicular to hyperplane why-is-weight-vector-orthogonal-to-decision-plane-in-neural-networks

vector-orthogonal-to-hyperplane

I wanted to visualize this by drawing a 3d plane where and the weight vector. By definition of the plane, all the points on the plane have the same dot product with the weight vector.

Question - Why is the weight vector not perpendicular to the plane? Am I mathematically/code wise wrong here or is it that Matlab viewing angle is not appropriate to see the orthogonality of the weight vector (bold arrow) to the plane? Thanks.

Matlab code below:

w = [0.8 0.3 0.24]  ;%weight vector
%get locus of point whose w'x is same (CONTOUR); x is a vector - (x1 x2 x3)
%randomly choose points from grid and get x3 VALUE

[x1 x2] = meshgrid(-5:0.25:5, -5:0.25:5);
coords = [x1(:) x2(:)];
%say we want to get contour for w'x = 3
coords(:,3) = (3 - w(1)*coords(:,1) - w(2)*coords(:,2))/w(3)
%plot the plane
scatter3(coords(:,1), coords(:,2),coords(:,3),'MarkerEdgeColor','k','MarkerFaceColor',[0 .75 .75])
view(-10,35);
xlabel('coords(:,1)')
ylabel('coords(:,2)')
zlabel('coords(:,3)')
mArrow3([0 0 0],[8 3 2.4])   %overlay weight vector
p1 = [0 0 0]
p2 = [8 3 2.4]
text(p1(1),p1(2), p1(3), sprintf('(%.0f,%.0f,%.0f)',p1)) %show the origin
%show the weight vector direction
text(p2(1),p2(2), p2(3), sprintf('(%.0f,%.0f, %.0f)',p2))

Another view

I think if you play with the scales of the axes and the viewing angle, you may find it is perpendicular.

Especially, try to rotate the view that you look sideways to the plane. You will see the perpendicular vector (regardless of scale of the other axes).

Any random view angle perpendicular to the weight vector should work.


Can a neural network compute $y = x^2$? In spirit of the famous Tensorflow Fizz Buzz joke and XOr problem I started to think, if it's possible to design a neural network that implements $y = x^2$ function?

Given some representation of a number (e.g. as a vector in binary form, so that number 5 is represented as [1,0,1,0,0,0,0,...]), the neural network should learn to return its square - 25 in this case.

If I could implement $y=x^2$, I could probably implement $y=x^3$ and generally any polynomial of x, and then with Taylor series I could approximate $y=\sin(x)$, which would solve the Fizz Buzz problem - a neural network that can find remainder of the division.

Clearly, just the linear part of NNs won't be able to perform this task, so if we could do the multiplication, it would be happening thanks to activation function.

Can you suggest any ideas or reading on subject?

I think the answer of @ShubhamPanchal is a little bit misleading. Yes, it is true that by Cybenko's universal approximation theorem we can approximate $f(x)=x^2$ with a single hidden layer containing a finite number of neurons can approximate continuous functions on compact subsets of $\mathbb{R}^n$, under mild assumptions on the activation function.

  But the main problem is that the theorem has a very important
  limitation. The function needs to be defined on a compact subsets of
  $\mathbb{R}^n$ (compact subset = bounded + closed subset). But why
  is this problematic? When training the function approximator you
  will always have a finite data set. Hence, you will approximate the
  function inside a compact subset of $\mathbb{R}^n$. But we can always
  find a point $x$ for which the approximation will probably fail. That
  being said. If you only want to approximate $f(x)=x^2$ on a compact
  subset of $\mathbb{R}$ then we can answer your question with yes.
  But if you want to approximate $f(x)=x^2$ for all $x\in \mathbb{R}$
  then the answer is no (I exclude the trivial case in which you use
  a quadratic activation function).

Side remark on Taylor approximation: You always have to keep in mind that a Taylor approximation is only a local approximation. If you only want to approximate a function in a predefined region then you should be able to use Taylor series. But approximating $\sin(x)$ by the Taylor series evaluated at $x=0$ will give you horrible results for $x\to 10000$ if you don't use enough terms in your Taylor expansion.


Encoding logic rules in machine learning For some problems you have certain rules that you know for a fact are true. A very simplistic example would be -

"A grade below 40% is a failing grade"

There is no ambiguity in the statement and for that system it will always be correct. Encoding such rules within any machine learning model should allow it to avoid simplistic mistakes and theoretically improve accuracy.

I'm aware there are several niche libraries for this and an entire subfield as well. However how would I combine such logic rules within a typical model (CNN,LSTM ,etc) in a platform like TensorFlow, PyTorch ?

Edit : What I'm looking forward is to add explicit label rules (similar to what Snorkel [ https://github.com/snorkel-team/snorkel ] uses to generate noisy labels) to help the model learn. These relations while present in the data might be hard for the model to learn or require large amounts of training data/time. In some scenarios a combination of deterministic rules and ML would be better than using either alone.

I would suggest you apply these rules to the output of your model after it has predicted the results.  In your example, if you are building a DNN classifier that predicts pass or fail, then before you return the prediction back to the calling application, simply check the value and update the model results based on the criteria you have established.

I have performed this in various models where the model may return negative results which are impossible.  In those cases, I would perform a simple update to the results data frame such as:

data.loc[data["y-hat"] < 0, "y-hat"] = 0

You can have as many of these types of 'hard-coded' rules as you need. You can even build a dictionary for them and map the results rather than having a series of statements. That obviously depends on the complexity of the rules.
Hope that helps.


Repeated training and testing in Weka? I am using one file for training (e.g train.arff) and another for testing (e.g test.atff) with the 70-30 ratio in Weka. I want to ask how can I use the repeated training/testing in Weka when I have separate train and test data files and the second part of the question is what is the advantage if we use repeated and what if we dont use it? Thanks in advance

In general the advantage of repeated training/testing is to measure to what extent the performance is due to chance. The most common source of chance comes from which instances are selected as training/testing data. One can use k-fold cross-validation in order to mitigate the effect of chance in this case. Weka performs 10-fold CV by default, as far as I remember, but this is not compatible with providing a specific training/test set.

[edit based on OP's comments]

In the video mentioned by OP, the author loads a dataset and sets the "percentage split" at 90%. This means that the full dataset will be split between training and test set by Weka itself. Weka randomly selects which instances are used for training, this is why chance is involved in the process and this is why the author proceeds to repeat the experiment with different values for the random seed: every time Weka will selects a different subset of instances as training set, resulting in a different accuracy. In other words, the purpose of repeating the experiment is to change how the dataset is split between training and test set. In this case (J48 with default options) there would be no point repeating the experiment with a fixed training set, because there's no chance involved in the process so there's no variation in the result.

It's worth noticing that this lesson by the author of the video seems to be used as an introduction to the more general concept of k-fold cross-validation, presented a couple of lessons later in the course.


Evaluating a Multi-Label Classification model I currently have a multi-label classification problem, for which I am using keras to build a neural network as follows:
n_cols = dataset.shape[1]
print(n_cols)

model = Sequential()
model.add(Dense(128, activation='relu', input_shape=(n_cols,)))
model.add(Dense(64, activation='relu'))
model.add(Dense(26, activation='sigmoid')) # Sigmoid for multi-label classification

sgd = SGD(lr=0.1, decay=1e-6, momentum=0.5, nesterov=True)
#RMSprop
model.compile(loss='binary_crossentropy', optimizer='RMSprop', metrics=['accuracy'])

model.summary()

## Fit the model ##
early_stopping_monitor = EarlyStopping(patience=20)
history = model.fit(dataset, labels, validation_split=0.33, epochs=30, callbacks=[early_stopping_monitor])

plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('Model accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'], loc='upper left')
plt.show()

I was informed that for multi-label classification, we use binary_crossentropy for the loss while having sigmoid for activation in the final layer (output layer). However, with this I am getting a resulting accuracy and val_accuracy of ~0.0931 and ~0.0937 respectively.
For the multi-label classification, is using the accuracy metric the best fit? I've looked around and some suggest that other metrics such as binary_accuracy may be better..
So the question is, how can one best evaluate the multi-label classification?
EDIT: For reference, I have 26 label columns in my target "classes" and the dataset consist of 21 columns. The entire dataset the model is trained on has ~82k samples.

You are correct for using sigmoid+binary CE for multi-label classification problem. On the other hand, try to think about how accuracy would be defined for this problem? I would use categorical accuracy!


Generalization of RNN/LSTM/GRU... model Given a time-series prediction with a Recurrent Neural Network (doesn't matter if LSTM/GRU/...), a forecast might look like this:

to_predict (orange) was fed to the model, predicted (purple) is the forecast resulting from the RNN-model and correct (dashed blue) is how it should have benn forecasted correctly.

As can be seen, to_predict (as well as all the training-data) is quite "spiky", while the forecast is much smoother. The smoothness is presumably the result of the models architecture etc.; anyhow, my question aims somewhere else (even though connected to this):

Is a smooth forecast that more or less middles the zigzag of peaks and valleys of the correct data a hint for...

... a generalizing model, ...
... or is the absence of such zigzags a sign that its forecast-quality is low ...
... or neither (respectively no conclusion can be drawn at all)?

I intentionally did not mention any criteria like MAE, MAPE etc. since I am only concerned with this graphical interpretation.

Maybe a little bit late to the party, but this might help you in the future:
Neural networks in general are heavily overparameterised functions that are fitted to match the training data. This is true for simple MLPs or more complex RNNs. These functions never encapsulate the "true" function, but only approximate it.
In you data, the spikes are simply outliers (meaning that they cannot be explained by the given, underlying input data). This means that to approximate those outliers, a function of your given inputs is not enough, since you are "averaging out" or ignoring these spkies as outliers during training (I assume you have a large dataset to train on).
See it that way: the given input data to your approximated function is not enough to explain the full "world" that your time series exists in.


Expected value in Bellman equation I am reading "reinforcement learning - An introduction" by Sutton and Barto.

At pag. 59, there is the Bellman equation for the state-value function

$\begin{array}{ll}
v_{\pi}(s) &=
\mathbb{E}_{\pi}[G_t|S_t=s] \\&= \mathbb{E}_{\pi}[R_{t+1} + \gamma G_{t+1}|S_t=s] \\&= \sum\limits_{a} \pi(a|s) \sum\limits_{s'} \sum\limits_{r} p(s^{'},r|s,a) \left[ r + \gamma \mathbb{E}_{\pi}[G_{t+1}|S_{t+1}=s'] \right]
\end{array}$

I didn't understand why the expected value survived in the last expression. The definition of the expected value is $\mathbb{E}[X] = \sum\limits x \cdot p(x)$, not $\mathbb{E}[X] = \sum\limits \mathbb{E}[x] \cdot p(x)$

I don't know whether my question is clear. In the last equation of the defition of $v_{\pi}(s)$, I would not have put the expected value inside

With expected values you have a fair bit of freedom to expand/resolve or not.

For instance, assuming the distributions $X$ and $Y$ are independently resolved (i.e. the values are not correlated):

$$\mathbb{E}[X + Y] = (\sum_x xp(x))+ \mathbb{E}[Y]$$

$$\mathbb{E}[XY] = \sum_x xp(x)\mathbb{E}[Y]$$

Each time step of a MDP is independent in this way, so you can use this when handling sums and products within expectations in the Bellman equations (provided you separate terms by time step).

For the Bellman equation, the goal is to relate $v_\pi(s_t)$ to $v_\pi(s_{t+1})$, and the definition of value is given as an expectation, so it makes sense to preserve the second expectation rather than expand it.

Something has to change though, as within the second sum a time step is effectively taken from $s$ to $s'$, so the new expectation has to include that. It has in some sense been expanded, just not fully broken down into the full product of every following policy decision and state transition.

You could try to write out the full expansion from expected to products of sums over distributions using some container like $\Pi_{n=t+1}^{T}$ - showing how to calculate the expected value over full tree of all possibilities - and the maths would still work. But it would be a very longhand way of showing the same relationship.


Are there any algorithms that learn to learn a function mapping? Typical algorithms involve learning and applying a single mapping e.g. $ f: X \mapsto Y$

Are there any algorithms that learn multiple mappings given an extra variable e.g. $ f(Z): X \mapsto Y$
(This feels like an abuse of notation.).

Here we learn a mapping, given $Z$, that is then applied to $X$ to produce $Y$

One solution (although intractable) would be to train an algorithm (e.g. a neural network) whose output is then used as the weights of a second network.

Any suggestions would be much appreciated and feel free to ask for more details. Thanks.

This is equivalent to thinking of this as learning a function of two inputs, i.e.,  $g : (X,Z) \mapsto Y$ (i.e., $g(X,Z)=Y$).  Any machine learning method for learning functions can be used for that; you just treat the feature vector (the input) as the concatenation of the vector $X$ and the vector $Z$, and then use standard ML techniques on the resulting instances.  So, you don't need anything special or fancy to handle this -- you can use any existing machine learning method that you are comfortable with.


Benchmark datasets for collaborative filtering I'd like to test a new algorithm for collaborative filtering. A typical use case is to recommend movies based on the preferences of users similar to the specific user.

What are some common benchmark datasets that researchers often use to test their algorithms? I know that within Computer Vision people often use MNIST or CIFAR, but I haven't found similar datasets for collaborative filtering.

The obvious answer would be the Netflix prize dataset, there is a lot of research into it and most CF algorithms have known scores in it.

There are other available datasets that are usually used as benchmarks:

Movie lens Dataset: a 20 million ratings dataset used for benchmarking CF algorithms;
Jester Dataset: a joke recommendation dataset with more than 6 million ratings;
You can find many more datasets in this link


Is Annoy a machine learning algorithm to find nearest neighbor ? and is it similar to K nearest neighbor algorithm? I was researching about  Google universal sentence encoding and i saw that it uses simple neighbor/Annoy to find the nearest vector for semantic-similarity search engine. This is the first time i'm hearing about it.  Is simple neighbor/Annoy another kind of  machine learning algorithm to find nearest neighbor ? and is it similar to K nearest neighbor  algorithm ?

Don't have enough reputation to comment to a resource, so answering this myself.

About Annoy

Annoy is a library being used here for finding approximate nearest neighbours, approximate being the key word here.

Understanding K-NN and Approximate NN

Now, let's see what is the difference with example of a problem.

Say you have 10 entities (words / sentences / objects / anything) for which you have vectors.

1 vector for each entity.

Let's consider that the metric we're interested in finding for finding neighbours is cosine similarity / cosine distance

How many pairs would this form?
Roughly (10 * 9) / 2 = 45.

So, you can find cosine similarity between the 45 pairs and then find K-nearest neighbours you want. Pretty simple. This is your K-Nearest Neighbours algorithm.

Also given you are storing these 45 cosine similarities somewhere, you basically have to deal with storing 45 float numbers (cosine similarities can be between -1 and 1).

Now, let's scale up the problem. What if instead of having 10 entities, you have 10,000,000, i.e. 10M entities. Pretty easy to have that number of entities exist in real world.

Now, you'll need to find cosine similarity between (10M * (10M-1) / 2) =  4,99,99,99,50,00,000 pairs. Which is an enormous number! Basically, the complexity grows with O(n^2), where n is number of entities.

Now, think about finding K-nearest neighbours for any given entity. You'll also be running sorting algorithm for 10M similarities for it. Also, let's just think about storage again. Say 1 float number takes 64 bits (or 8bytes). How much storage do you need to store those many pairs?

4,99,99,99,50,00,000 * 8 bytes ~ 364 TeraBytes!

Getting a memory (RAM) of that size is near to impossible.

So, you understand the problem we're having with regular K-NN approach. We don't wish to calculate all pairs, and it's very difficult to store distances for those many pairs and calculate true K-nearest neighbours at runtime for any entity.

This is where Approximate Nearest Neighbours comes into picture. These say, that given your need of the problem is almost always going to be to get only the K Nearest neighbours, why bother calculating similarities for any given entity to those entities that have zero chances to appear in neighbourhood?

Think about a space where you have clusters of points. Say you need to only find the 3 most nearest neighbours for every point for your problem, then you would never need to find similarities between points of different clusters. Right?

This is the idea that Approximate Nearest Neighbour algorithm runs on.

  For any given entity, instead of calculating similarities/distances to each of the other point, it chooses X other points and only calculates distances to those.

This reduces the complexity from O(n2) to just O(n*k). Storage becomes easier, Calculation becomes faster.

A hint of Annoy's implementation

While I'm myself not familiar with all implementation details of Annoy's ANN approach. Usually, most ANN approaches require building a Cluster Index first -- which basically decides the X leaders with which you'll find similarities. Then, the cluster index is used to find Approximate K-nearest neighbours for all given points.

The reason why this is called 'approximate' is that the cluster building process is not perfect. If it were, it'll take days or years of computation and defeat the purpose of reducing complexity. It is approximate cluster indices. Which means, sometimes you can miss out on one of the most similar entity appearing in neighbourhood of given entity. Usually, however this doesn't happen so much that it affects practically. Spotify, Facebook, Google and many large companies are using approximate nearest neighbours approaches.

Annoy is one of the most widely used library for doing approximate nearest neighbours.


Testing Multi-Arm Bandits on Historical Data Suppose I want to test a multi-arm bandit algorithm in the contextual setting on a set of historical data. For simplicity, let's assume there are only two arms A and B and suppose the rewards are binary. Furthermore, suppose I have a data set where users were shown one of the two arms and I have a record of the rewards. What would be the best approach to simulating the scenario of running the algorithm online?

I was thinking of doing the following: If the algorithm outputs A then I want to record a reward so the algorithm can learn, to do this we sample uniformly at random from the set of data where users were shown the output to obtain this reward.

I'd like to know if this approach is suitable and if anyone knows of any better ways to simulate online learning algorithms with historical data, in particular the multi-arm bandit problem?

The problem you have is that the users were originally shown A or B under a different policy to that which the optimiser is learning. You probably don't know the probabilities for A or B from that policy (if you do, that would allow for a bit more subtlety).

You will need to reject samples that do not match what the optimiser does at any point. This may leave you with far less historical data to mine, and the possibility of getting a biased result.

  to do this we sample uniformly at random from the set of data where users were shown the output to obtain this reward.

If I understand this correctly, you are planning to:

Pick an event (including user data) for input context
Generate the choice of A or B using your learning agent
Find another user that was presented the same output in the historical data, by sampling randomly from "all users shown A" or "all users shown B"
Count the reward generated in the sampled historical record

Sadly this will not work, because in step 3 you divorce the context from the agent's decision. So you will not be measuring the agent's likely reward, but the hybrid reward of an agent that uses a different user population and gains rewards according to the historic policies (which you may not even know).

A less biased approach might be:

Pick an event (including user data) randomly from historic data
Generate the choice of A or B using your learning agent
If the agent's choice and historic choice for that event match, then count the reward, else discard the event from the test

You may get sampling bias issues here if the agent's policy is different to historic policy for large number of users. The problem is that you are systematically removing users where old policies said one thing and new policies another, due to lack of data about the reward. So unless the original policy was completely random, the samples you actually train/evaluate on will not be chosen with correct ratios across the user population as it varies. Importance sampling could help mitigate that, but only if you know the probabilities of making A|B selection in both historic and learning agent policies.

Another possibility is to train a supervised model that predicts reward given user data and action from your historic data, and use that to drive a simulator of the online environment. If you are also training your agent during this test, you may need to simulate variance - e.g. turn probability of a click-through event into either a click or no click event by sampling $x < p(click)$ - because that is the data you need your online algorithm to learn from. This is clearly limited by the accuracy of the supervised model from the training data, but could reduce variance, because you will get some kind of indication of reward for events that did not happen historically (with some bounds on accuracy that you can estimate). That is due to function approximation in the supervised model - it is pretty similar idea to bootstrapping the Critic evaluation in an Actor-Critic reinforcement learning model.


How can I deal with data that is on the format "Image + single number"? Let say I have a data set where every sample is an image of a landscape and a temperature associated with the landscape. How do I incorporate the temperature into my convolutional neural network for classifying if the data is e.g a winter or summer landscape? Can I simply add the temperature as a feature after the feature learning part of the network? I can not find similar questions but maybe this has a name that I am not aware of?

After the convolutional part you will need to add a normal, dense layer. Concatenate it to this layer and add some more layers if necessary, to add more interactions between the temperature and the image. This wouldn't necessarily need to be too deep because it can learn to represent the image features in a way that it will combine nicely with the temperature already, hopefully.


Facing this issue while predicting "CountVectorizer - Vocabulary wasn't fitted" #these are classifier and  vectorizer

vectorizer = CountVectorizer(tokenizer = spacy_tokenizer, ngram_range=(1,1))
classifier = LinearSVC()

I have created a Pipeline as shown below

# Create the  pipeline to clean, tokenize, vectorize, and classify
pipe = Pipeline([("cleaner", predictors()),
                 ('vectorizer', vectorizer),
                  ('classifier', classifier)])

while predicting the test dataset, im facing this issue.

sample_prediction = pipe.predict(X_test)

ERROR : sklearn.exceptions.NotFittedError: CountVectorizer - Vocabulary wasn't fitted.

Can anyone help ?

You have to give your training set to the model to be trained

_= pipe.fit(triningSet.data, triningSet.target)

I don't see any training dataset here. you have to fit the CountVectoriser to your data set.


Autoencoder implementation using ImageDataGenerator I'm using the concept demonstrated in this paper. Their training data consists of "GOOD" images and "BAD" images. They train the AE using "BAD" images (X) to make it produce "GOOD" image. "BAD" images in this case maybe very similar to "GOOD" image but with small dent or scratch.
I've had success using ImageDataGenerator on mnist number data but in such case it is trained by using X -> X or if not using ImageDataGenerator the model fitting code will be something like model.fit(x_train, x_train).
However, in this AE problem, we want to train using X_good_and_defect -> X_good or  model.fit(x_good_and_defect_train, x_good_train). Not sure how to achieve this using ImageDataGenerator.

I'm using keras's image data generator to load the images.
train_dir = r'chunks/training'
train_datagen = ImageDataGenerator(rescale=1 / 255)
train_generator = train_datagen.flow_from_directory(train_dir, target_size=(256,256),
                   color_mode='grayscale', class_mode='input', batch_size=256)
...
autoencoder.fit(train_generator,
                epochs=5,
                batch_size=128,
                shuffle=True,
                validation_data=(test_generator, train_generator),
                callbacks=[])

As the following images from the paper shows, during training time you create patches from larger images. These patches have no defects and can therefore be seen as 'good' images. To get the accompanying 'bad' images with defects you synthetically generate these defects on the 'good' images.

Your data generator should therefore follow the following steps:

Read in full size 'good' images
Create patches from the full size images
Synthetically add defects to the patches from step 2
Return both the 'good' patches from step 2 and 'bad' patches from step 3

You then train your model using the 'bad' patches as the input and your 'good' patches as the output.


Is pattern recognition the same as unsupervised learning? Is machine learning the same as supervised learning? Firstly, here is the definition of a well-posed learning problem:

  A computer program is said to learn from experience E with
  respect to some task T and some performance measure P, if its
  performance on T, as measured by P, improves with experience
  E.

My machine learning professor claims that supervised learning is machine learning and unsupervised learning is pattern recognition. Here are his definitions of machine learning and pattern recognition:

  Machine Learning is learning from experience. It’s also called
  supervised learning. E consists of features and labels, and P
  and T are well-defined.

And

  Pattern Recognition is finding patterns without experience. It’s
  also called unsupervised learning. E consists of only features,
  and P and T are defined in much broader terms of finding
  ‘interesting patterns'.

If I cross reference this with other sources on the internet, I get different definitions. They say that machine learning involves both supervised and unsupervised learning. Likewise, pattern recognition involves both supervised and unsupervised learning. Some say that pattern recognition is machine learning. Some even say the opposite, that machine learning is a form of pattern recognition.

So is my professor right, or is the internet right, or is the answer somewhere in between? Can we answer this question definitively?

Nice topic. I think it depends on how the "learning" process by a machine is defined, so if we assume that a machine can "autonomously" learn from experience (i.e. new data as it becomes available every X time), why not to include also semi-supervised and unsupervised learning styles?
The more data the machine has about the system being modeled, the more info to find more precise patterns which correctly describe the system, for all kind of learning styles.

As an example I made to explain how the one-class support vector machine works, we could see that, although an unsupervised learning process does not have a ground truth (i.e. labels) or a semi-supervised learning has only the "normality" label, the machine is able to better describe the system as it receives more data and as some hyperparameters are adjusted, if you test your model with known points wich should be outside the learnt cluster (where an anomaly score could be also a metric in this case).


Most important part of feature standardization and how is standardization affected by sparsity? I am thinking of preprocessing techniques for the input data to a convolutional neural network (CNN) using sparse datasets and trained with SGD. In Andrew Ng's coursera course, Machine Learning, he states that it is important to preprocess the data so it fits into the interval $ \left[ 3, 3 \right] $ when using SGD. However, the most common preprocessing technique is to standardize each feature so $ \mu = 0 $ and $ \sigma = 1 $. When standardizing a highly sparse dataset many of the values will not end up in the interval.

I am therefore curious - would it be better to aim for e.g. $ \mu = 0 $ and $ \sigma = 0.5 $ in order for the values be closer to the interval $ \left[ 3, 3 \right] $? Could anyone argue based on a knowledge of SGD on whether it is most important to aim for $ \mu = 0 $ and $ \sigma = 1 $ or $ \left[ 3, 3 \right] $?

No, you are misinterpreting his comments.  If you have data that has some outliers in it then the outliers will extend beyond 3 standard deviations.  Then if you standardize the data some will extend beyond the [-3,3] region.

He is simply saying that you need to remove your outliers so the outliers don't reap havoc on your stochastic gradient descent algorithm. He is NOT saying that you need to use some weird scaling algorithm.

You should standardize your data by subtracting the mean and dividing by the standard deviation, and then remove any points that extend beyond [-3,3], which are the outliers.

In stochastic gradient descent, the presence of outliers could increase the instability of the minimization and make it thrash around excessively, so its best to remove them.

If the sparseness of the data prevents removal then... Do you need to use stochastic gradient descent, or can you just use gradient descent?  Gradient descent (GD) might help to alleviate some of the problems relating to convergence.  Finally, if GD is having trouble converging, you could always do an direct solve (e.g. direct matrix inversion) rather than an iterative solve.

Hope this helps!


What are the differences between MLP and DNN? So I have been reading about the topic for a while, but i did not find a clear answer why MLP and DNN are being used interchangeably even though there are some differences between them.
So far I have filtered some informations:
"The field of artificial neural networks is often just called neural networks or multi-layer perceptrons after perhaps the most useful type of neural network.
A perceptron is a single neuron(input, output, weights, activation) model that was a precursor to larger neural networks.
MLP is a subset of DNN.
While DNN can have loops and MLP are always feed-forward(a type of Neural Network architecture where the connections are "fed forward", do not form cycles (like in recurrent nets).
Multilayer Perceptron is a finite acyclic graph, not like RNN and it's subsets which are cyclic in nature.
MLP uses back propagation for training the network."
So what makes MLP different from DNN ?

You explained it already quite well. An MLP is a type of neural network, the same way CNNs, RNNs, and other types exist. DNN is an umbrella term for all types of neural networks.
The reason some literature may be using these terms interchangeably is likely due to the fact that MLPs are some of the oldest forms of neural networks and therefore, at the time the literature was written, MLPs and DNNs were ubiquitous.


How to get model attributes in scikit learn (not hyper parameters) How to get model attributes list (not hyper parameters passed to Estimator's class)?
For ex:
kmeans = KMeans(n_clusters=5)
kmeans.fit(X)
kmeans.labels_

how to get list of the attributes like labels_ from model object (ending with _)?

I believe you are trying to access "labels_" before fitting the data.
from sklearn.cluster import KMeans
import numpy as np
X = np.array([[1, 2], [1, 4], [1, 0],
               [10, 2], [10, 4], [10, 0]])
kmeans = KMeans(n_clusters=2, random_state=0).fit(X)

def get_properies(model):
  return [i for i in model.__dict__ if i.endswith(‘_’)]

get_properies(kmeans)

['n_clusters',
'init',
'max_iter',
'tol',
'precompute_distances',
'n_init',
'verbose',
'random_state',
'copy_x',
'n_jobs',
'algorithm',
'cluster_centers_',
'labels_',
'inertia_',
'n_iter_']


Hyperparameter optimization performance comparison I have used Bayesian optimization for hyperparameter tuning in a machine learning model. What is the best way to compare the performance of network with and without Bayesian optimization? I found some useful material here, where a comparison was drawn between Random search(grey) and Bayesian Optimization(green) as also shown in below figure. This is exactly what I want to achieve.

Script for hyperparameter tuning [MATLAB]:

% Make some data

Daten = rand(100, 3);
Daten(:,3) = Daten(:,1) + Daten(:,2) + .1*randn(100, 1);  % Minimum asymptotic error is .1
[m,n] = size(Daten) ;
% Split into train and test
P = 0.7 ;
Training = Daten(1:round(P*m),:) ;
Testing = Daten(round(P*m)+1:end,:);
XTrain = Training(:,1:n-1);
YTrain = Training(:,n);
XTest = Testing(:,1:n-1);
YTest = Testing(:,n);
% Define a train/validation split to use inside the objective function
cv = cvpartition(numel(YTrain), 'Holdout', 1/3);
% Define hyperparameters to optimize
vars = [optimizableVariable('hiddenLayerSize', [1,20], 'Type', 'integer');
        optimizableVariable('lr', [1e-3 1], 'Transform', 'log')];
% Optimize
minfn = @(T)kfoldLoss(XTrain', YTrain', cv, T.hiddenLayerSize, T.lr);
results = bayesopt(minfn, vars,'IsObjectiveDeterministic', false,...
    'AcquisitionFunctionName', 'expected-improvement-plus');
T = bestPoint(results)

% Train final model on full training set using the best hyperparameters
net = feedforwardnet(T.hiddenLayerSize, 'traingd');
net.trainParam.lr = T.lr;
net = train(net, XTrain', YTrain');
% Evaluate on test set and compute final rmse
ypred = net(XTest');
finalrmse = sqrt(mean((ypred - YTest').^2))

function rmse = kfoldLoss(x, y, cv, numHid, lr)
% Train net.
net = feedforwardnet(numHid, 'traingd');
net.trainParam.lr = lr;
net = train(net, x(:,cv.training), y(:,cv.training));
% Evaluate on validation set and compute rmse
ypred = net(x(:, cv.test));
rmse = sqrt(mean((ypred - y(cv.test)).^2));
end

Just measure the out-of-fold-value of your choice, you can add time complexity to the table and than just extract percantage difference.


Training and testing AdaBoost for low probability classification I have a dataset that I want to classify as fraud/not fraud and I have many weak learners. My concern is that there is much more fraud than not fraud, so my weak learners perform better than average, but none perform better than 50% accuracy in the complete set.

My question is whether I should set up testing and training sets that are half fraud and half not fraud or if I should just use a representative sample.

Is it possible that adding generated data to your data set will decrease the fraud/non fraud ration and make your dataset more representative / usable ?

At GenieLog, we are producing test data for designing and testing fraud detection tools. We our generator GEDIS Studio we can define regular profiles and fraudster profiles, instantiating each category to a customizable ratio (for ex. 2 % of customer will have fraudulent usage of generated events.)

We did it successfully for telecom CDR (http://www.gedis-studio.com/online-call-detail-records-cdr-generator.html) and Credit Card usages. There's a freely available access to the online generator on http://www.data-generator.com

I'm pretty sure that even if the tool is not matching your needs at least the approach can be valuable. Otherwise I would be interested to read any objection :)

Regards


What is meant by Average Content Distance in Videos generated by GANs? I'm reading a research paper on generating/synthesizing videos:
MoCoGAN: Decomposing Motion and Content for Video Generation

To evaluate the generated videos, they have used a metric called 'Average Content Distance'. I couldn't find any material on google related to this. Can anyone please explain what Average Content Distance means?

Here is the snippet from the paper

  we first computed
  the average color of the generated shape in each
  frame. Each frame was then represented by a 3-dimensional
  vector. The ACD is then given by the average pairwise L2
  distance of the per-frame average color vectors.

What I understood from this is as follows:
For each frame, convert rgb to gray (average of color). Then for successive frame, calculate the l2 distance.
$$\frac{1}{MN} \sum_{x=1}^{M}\sum_{y=1}^{N}{(Frame_i(x,y) - Frame_{i+1}(x,y))^2}$$
This gives ACD. Have I understood it correctly?

Also, how does this metric represents quality of a video? How can this be used to compare qualities of different generated videos? You can also point me towards some references.

Thanks!

This article presents 2 ways to implement ACD metric. You talk about ACD-I (using the article's notation). As I understood,

you first average colors, i.e. sum over pixels in the image plane, for each color in all frames: $\mathrm{avg}_i = \frac{1}{MN}\sum_{x,y}\mathrm{Frame}_i $.
then, in the resulting 3D vectors for every two consecutive ones, you calculate L2 distance: $d_i = \sqrt{\sum_{l=1}^3(\mathrm{avg}_{i,l} - \mathrm{avg}_{i+1,l})^2}$. In general, you might want to use other distances, which is not prohibited.
sum it up (since we take into account all frames we have, not just one pair) and divide by the number of frames (because the metric should not depend on it [at least, I suppose so], otherwise longer videos will have larger metric values): $\mathrm{ACD} = \frac{1}{K-1}\sum_i d_i$ (if there are K frames).

ACD-C is obtained in the same way but instead frames you use feature vectors extracted from images (frames) with 'encoding-like' network. OpenFace probably is a good choice when dealing with facial expressions.

Your formula might work differently from the authors' intention. Imagine a white spot on a black screen. From frame to frame that spot is gradually moving from one side to the other. Your metric shows that the content in this video is changing. And the faster the spot moves, the greater changes happen (ok, in case of a spot and a black screen at a certain speed the saturation point will be reached and your metric will stop changing [when in one frame the spot moves a distance equal to its diameter], but that's another story). However, in fact, there are no changes in the content. You still have the spot and the black screen. This is why you need averaging (not summing 'changes' and making it independent from the image size - this is what your formula does)

Here you can see the implementation of ACD metric(s). I cannot vouch that is 100% correct. So, let me know if there are any kind of uncertainties.


Purpose of test data in binary classification I have a highly biased training dataset where AppId 6,7,8,9,10 are almost never purchased. I made this up just to see how good my comprehension of calculating the classification metrics is such as Accuracy, F1Score, and etc.
UserId,UserName,AppId,Label
1,Lin,1,1
1,Lin,2,1
1,Lin,3,1
1,Lin,4,1
1,Lin,5,1
1,Lin,6,0
1,Lin,7,0
1,Lin,8,0
1,Lin,9,0
1,Lin,10,0
2,Tim,1,1
2,Tim,2,1
2,Tim,3,1
2,Tim,4,1
2,Tim,5,1
2,Tim,6,0
2,Tim,7,0
2,Tim,8,0
2,Tim,9,0
2,Tim,10,0
3,Kirstin,1,1
3,Kirstin,2,1
3,Kirstin,3,1
3,Kirstin,4,1
3,Kirstin,5,1
3,Kirstin,6,0
3,Kirstin,7,0
3,Kirstin,8,0
3,Kirstin,9,0
3,Kirstin,10,0
4,Michael,1,1
4,Michael,2,1
4,Michael,3,1
4,Michael,4,1
4,Michael,5,1
4,Michael,6,0
4,Michael,7,0
4,Michael,8,0
4,Michael,9,0
4,Michael,10,1
5,Timmy,1,0
5,Timmy,2,0
5,Timmy,3,0
5,Timmy,4,1
5,Timmy,5,1
5,Timmy,6,0
5,Timmy,7,1
5,Timmy,8,1
5,Timmy,9,1
5,Timmy,10,0
6,Lick,1,1
6,Lick,2,0
6,Lick,3,0
6,Lick,4,0
6,Lick,5,1
6,Lick,6,1
6,Lick,7,0
6,Lick,8,0
6,Lick,9,1
6,Lick,10,0
7,Shawn,1,1
7,Shawn,2,0
7,Shawn,3,1
7,Shawn,4,0
7,Shawn,5,1
7,Shawn,6,0
7,Shawn,7,0
7,Shawn,8,0
7,Shawn,9,1
7,Shawn,10,1

To test the trained model, I used this test set.
UserId,UserName,AppId,Label
8,Lianne,1,1
8,Lianne,2,0
8,Lianne,3,1
8,Lianne,4,0
8,Lianne,5,0
8,Lianne,6,0
8,Lianne,7,0
8,Lianne,8,0
8,Lianne,9,1
8,Lianne,10,0

Following is the prediction results.
8,Lianne,1,1  => Predicted true
8,Lianne,2,0  => Predicted false
8,Lianne,3,1  => Predicted true
8,Lianne,4,0  => Predicted true
8,Lianne,5,0  => Predicted true
8,Lianne,6,0  => Predicted false
8,Lianne,7,0  => Predicted false
8,Lianne,8,0  => Predicted false
8,Lianne,9,1  => Predicted false
8,Lianne,10,0 => Predicted false

// Sum this up...
TP = 2
TN = 5
FP = 2
FN = 1

Accuracy = 2 + 5 / 2 + 5 + 2 + 1 => 7 / 10 => 0.7
Recall = 2 / 2 + 1 => 0.66
Precision = 2 / 2 + 2 => 0.5
F1Score = 0.5 * 0.66 / 0.5 + 0.66 * 2 =  0.33 / 1.16 * 2 = 0.5689

The calculation was proven alright and as I expected, the model is right on almost all cases except the app id 2. While I'm doing this, this idea comes into my mind.

" What test data is for? "
" Is it for checking out the desired model metrics like above? "
" Why bother with some proportion of data to perform a test? when you just need a certain, small group of dataset to evalute the model? "
" Even if I have a thousands of million rows of training dataset, wouldn't it be enough to have a complete small, unique set of dataset that covers the whole variations? do I still need some proportion of testdata from the training set? "

Well, if I already know that the app id range is from 1 to 10, isn't it fairly enough to just prepare only 10 rows of test set as long as it has app id 1 to 10? even when the training data set is gigantic?
Why do I need to care about N% of test data set from the training data set?

One of the well known problems of machine learning is overfitting. The main reason for having a test data set is to

Obtain a realistic measure of how good your model is doing.
Prevent overfiting the validation set.

Now think about the second point, quite often it might occur that you tune up the hyper parameters so that your validation set provides a good measure, but this means you might be overfitting your validation set and when you deploy your shiny model in the real world, boom! poor results are returned.

As per the amount of data to reserve for testing, it depends on how big is your data set, normally the rule of 60% for training, 20% for validation and 20% for test has been used, but if your data set has 5 millions of points, you probably just need 1% for testing (as that will provide you with 5000 examples for testing anyway).

So, yes, your last question makes sense, you will probably be fine taking a small portion of the original data set (again, say 1%) in order to get good results.


reduction of model accuracy while using PCA for a regression problem I am trying to build a prection problem to predict the fare of flights. My data set has several catergorical variables like class,hour,day of week, day of month, month of year etc. I am using multiple algorithms like xgboost, ANN to fit the model

Intially I have one hot encoded these variables, which led to total of 90 variables, when I tried to fit a model for this data, training r2_score was high around .90 and test score was relatively very low(0.6).

I have used sine and cosine transformation for temporal variables, this led to a total of only 27 variables. With this training accuracy has dropped to .83 but test score is increased to .70

I was thinking that my variables are sparse and tried doing a PCA, but this drastically reduced the performance both on train set and test set.

So I have few questions regarding the same.

Why is PCA not helping and inturn reducing the performance of my model so badly
Any suggestions on how to improve my model performance?

code

from xgboost import XGBRegressor
import pandas as pd
import matplotlib.pyplot as plt

dataset = pd.read_excel('Airline Dataset1.xlsx',sheet_name='Airline Dataset1')

dataset = dataset.drop(columns = ['SL. No.'])
dataset['time'] = dataset['time'] - 24

import numpy as np
dataset['time'] = np.where(dataset['time']==24,0,dataset['time'])

cat_cols = ['demand', 'from_ind', 'to_ind']

cyc_cols = ['time','weekday','month','monthday']

def cyclic_encode(data,col,col_max):
    data[col + '_sin'] = np.sin(2*np.pi*data[col]/col_max)
    data[col + '_cos'] = np.cos(2*np.pi*data[col]/col_max)
    return data

cyclic_encode(dataset,'time',23)
cyclic_encode(dataset,'weekday',6)
cyclic_encode(dataset,'month',11)
cyclic_encode(dataset,'monthday',31)

dataset = dataset.drop(columns=cyc_cols)

ohe_dataset = pd.get_dummies(dataset,columns = cat_cols , drop_first=True)
X = ohe_dataset.iloc[:,:-1]
y = ohe_dataset.iloc[:,27:28]

# Splitting the dataset into the Training set and Test set
from sklearn.model_selection import train_test_split
X_train_us, X_test_us, y_train_us, y_test_us = train_test_split(X, y, test_size = 0.2, random_state = 0)

# Feature Scaling
from sklearn.preprocessing import StandardScaler
sc_X = StandardScaler()
sc_Y = StandardScaler()
X_train = sc_X.fit_transform(X_train_us)
X_test = sc_X.transform(X_test_us)

y_train = sc_Y.fit_transform(y_train_us)
y_test = sc_Y.transform(y_test_us)

#Applying PCA
from sklearn.decomposition import PCA
pca = PCA(n_components = 2)

X_train = pca.fit_transform(X_train,y_train)
X_test = pca.transform(X_test)
explained_variance = pca.explained_variance_ratio_

regressor = XGBRegressor()
model = regressor.fit(X_train,y_train)

# Predicting the Test & Train set with regressor built
y_pred = regressor.predict(X_test)
y_pred = sc_Y.inverse_transform(y_pred)
y_pred_train = regressor.predict(X_train)
y_pred_train = sc_Y.inverse_transform(y_pred_train)
y_train = sc_Y.inverse_transform(y_train)
y_test = sc_Y.inverse_transform(y_test)

#calculate r2_score
from sklearn.metrics import r2_score
score_train = r2_score(y_train,y_pred_train)
score_test = r2_score(y_test,y_pred)

Thanks

When you do PCA to project your problem two a 'n' dimensional space, in your case 2 dimensions. So you are fitting a model with just two features.

If you print your variable 'explained_variance' you will see the percentage of variance respect to the total that you have.

Ideas to improve your model: search for hyperparameters, do target encoding instead of OHE, dont do PCA...

By the way Decision trees model don't need the data to be standarized.


autoEncoder as LSTM input, any benefit? When training my LSTM, is there any incentive to pre-pass its inputs through an auto-encoder, or should I always supply as raw data as possible?

I have is a small amount of training data, and it contains only a few very rare significant events (spikes, spaced out in time)

The purpose of the autencoder would be similar to doing an embedding of text to pass latent space features of the input to the LSTM, as opposed to raw inputs. In the case of text, this is usually done to reduce the very large and sparse dictionary (word subword, whatever) space to a   space which is less sparse, significantly smaller, and therefore more computationally efficient.

In your case, it looks like the data is not high dimensional to start with (I am assuming this because of the significant events over time), so you would not get the benefit of mapping the input to a latent space. If your trend over time is high dimensional, then it may be worth it, as the embedding/autencoding may pick up and reduce correlations between trends in a few dimensions.


I have limited samples for one class, unlimited samples for the other class. Need to balance? I want my machine learning algorithm to learn the difference between two classes, actually picture of X or picture of something else.

My sample data is:

500 pictures of X (I know it is low, unfortunately I can't do much about it)
1,000,000 pictures of something else

Question: Should I do the training with all of the 1,000,000 pictures of something else?
Or would such an imbalance have negative effects? For instance would it kind of "drown" the other data?

Notes:

Computing power and time is not an issue.
In the real world, pictures of X make up 5% to 10% of the data, so I don't think I have a class imbalance problem.
I think the classification is an easy one, machine learning will probably be able to understand quickly.
I am OK with a reasonable amount of misclassifications.
In case that matters, I am planning to use Keras and Tensorflow with Flatten/Dense relu/Dense softmax/AdamOptimizer/sparse_categorical_crossentropy.

Yes, train with all the data.

To adjust for the imbalanced data size, you want your minority class X to be more important than the majority class. You can either (1) use weighted cross-entropy loss, e.g. tf.nn.weighted_cross_entropy_with_logits, or (2) when adding up the loss from different traning examples, weight them inversely proportional to the size of that label, or use both.

This is a good explanation here.


saving the images to a folder with custom filenames I'm new to Python and need assistance. I am performing segmentation with my own medical data set. The test images in my folder are named like "1.0.2.34.56_1.png". I would like to remove the actual image extension and append the image names with "_mask.png". The current code throws an error.

def labelVisualize(num_class,color_dict,img):
    img = img[:,:,0] if len(img.shape) == 3 else img
    img_out = np.zeros(img.shape + (3,))
    for i in range(num_class):
        img_out[img == i,:] = color_dict[i]
    return img_out / 255

def saveResult(save_path,npyfile,flag_multi_class = False,num_class = 2):
    for i,item in enumerate(npyfile):
        img = labelVisualize(num_class,COLOR_DICT,item) if flag_multi_class else item[:,:,0]
        io.imsave(os.path.join(save_path,"%d_mask.png"%i),img)
saveResult("data/membrane/test/",results)

Most of the code you provided doesn't do much to help your actual problem of renaming the files. I will just focus on tha tpart by showing you how to get the filenames you want.

You can rename the files, as you describe them, like this - assuming you have a list of filenames that match your description.

filenames = ['1.0.2.34.57_1.png', '1.0.2.34.58_1.png', '1.0.2.34.59_1.png']

for file in filenames:

    # Do what you want with the file...
    # Image processing and modelling, etc.

    # Replace the extension with your custom ending
    final_filename = file.replace('.png', '_mask.png')

Another way to do it, breaking down the removal and addition into two steps

for file in filenames:
    file_without_ext = file.replace('.png', '')

    # Add you custom ending
    final_filename = file_without_extension + '_mask.png'

    # Another variant - joining the extension to the file_without_ext using "_"
    final_filename = '_'.join(file_without_ext, 'mask.png')

The three variants of final_filename will be identical.

Now you can save the output of your image processing using the final_filename.


Interpolation in nlp - definition of O term Reading definition of interpolation below how are the O terms defined? Is this a value that is set manually?

Example

P( Sam | I am ) = count( Sam I am ) / count(I am) = 1 / 2

Interpolation using N-grams
We can combine knowledge from each of our n-grams by using interpolation.
E.g. assuming we have calculated unigram, bigram, and trigram probabilities, we can do:
P ( Sam | I am ) = Θ1 x P( Sam ) + Θ2 x P( Sam | am ) + Θ3 x P( Sam | I am )

(original problem statement image)

Terminology point: those symbols aren't O, but thetas Θ.

Confusingly, these values labelled theta are normally referred to as lambdas, as in the page you quote. They are weights used in interpolation (as opposed to backoff) that sum to 1, and can be calculated from the corpus itself via a variety of methods:

  How are these λ values set? Both the simple interpolation and conditional interpolation λs are learned from a held-out corpus. A held-out corpus is an additional
  training corpus that we use to set hyperparameters like these λ values, by choosing
  the λ values that maximize the likelihood of the held-out corpus. That is, we fix
  the N-gram probabilities and then search for the λ values that when plugged into
  Eq. 4.24 give us the highest probability of the held-out set. There are various ways
  to find this optimal set of λs. One way is to use the EM algorithm defined in Chapter 7, which is an iterative learning algorithm that converges on locally optimal λs
  (Jelinek and Mercer, 1980).

  Speech and Language Processing. Chapter 4: N-Grams (4.4.3 Backoff and Interpolation) (p.15)


java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream Im traying to integrate Hadoop and R, I was install the pachages rJava and Rhipe in R, I do this steps to start Hadoop and R:
-starting Hadoop services.,
-loading rJava and Rhipe packages by library function.
-Calling rhinit() to initialize Rhipe.
the problem here is when I call rhinit() funtion, it show this error:

    Initializing Rhipe v0.73
    Error in .jnew("org/godhuli/rhipe/PersonalServer") :
     java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream'

please some helps to fixe this problem.

This issue is caused because the following environment variables are not set correctly
PKG_CONFIG_PATH and LD_LIBRARY_PATH
for my configuration, i should execute this:

export PKG_CONFIG_PATH = /usr/local/lib
export LD_LIBRARY_PATH = /usr/local/lib

also set this commands on R console:

Sys.setenv(HADOOP_HOME="/usr/local/hadoop/")
Sys.setenv(HADOOP_BIN="/usr/local/hadoop/bin")
Sys.setenv(HADOOP_CONF_DIR="/usr/local/hadoop/conf")


When memory errors occur with model.fit(), is it due to GPU RAM or normal RAM? With respect to this question,

https://stackoverflow.com/questions/51895278/how-to-know-when-to-use-fit-generator-in-keras-when-training-data-gets-too-big

when memory errors are reported due to excessively large training data, are the memory errors caused by lack of normal PC RAM or lack of GPU RAM? I would like to know which one so that I can buy the right kind of RAM.

Keras was configured to use GPU for training.

Yes. The error is for GPU memory. You should look at training in batches options if you haven't already. This thread has short explanation on possible benefits and implications.

Quick batch size tutorial


how can i implement a loss function with percentile cutting? I want a loss function which only care about the pecentile head and tail.
for example:
I have a rmse target, but i only care about rmse which y_pred > y_pred.percentile(90) and y_pred < y_pred.percetile(10), in the middle of y_pred, i dont want to consider them in my loss.
here is my code:
class myloss(keras.losses.Loss):
  def __init__(self, coef, name='myloss'):
    super().__init__(name=name)
    self.coef = coef

  def call(self, y, y_pred):
    ## TODO: i want to sum over y_pred > y_pred.percentile(90) and y_pred< y_pred.percentile(10)!!!
    # return tf.math.reduce_mean(tf.square(y * self.coef- y_pred), axis=1)

from util import *
from tensorflow import keras
from tensorflow.keras import layers
import tensorflow as tf
if __name__ == '__main__':
  df = read_cache_data('cache')
  df = df.dropna()
  print(df.shape)
  fv_cols = df.columns[21:-3]
  res_col = 'res_10'
  #model = keras.Sequential([layers.Dense(50, input_shape=(len(fv_cols), ), activation='relu'), layers.Dense(30, activation='relu'), layers.Dense(1)])
  model = keras.Sequential([layers.Dense(1, input_shape=(len(fv_cols), ))])#, activation='relu')])
  model.compile(optimizer=keras.optimizers.Adam(), loss = myloss('loss'))#keras.losses.mean_squared_error)
  model.summary()
  model.fit(df[fv_cols].values, df[res_col], batch_size=288)

What you want to do is weird, and i'm not sure i understand it correctly, but i assume you could use the answer to this post.


How to design a many-to-many LSTM? I have an input array of shape (1000,20, 4) and output(labels) of shape (1000,25,1). But don't know how to use Keras LSTM library to build a sequential model for this!

Can someone help me design a simple LSTM for doing that? (I tried to use RepeatVector() and TimeDistributed(Dense()) in several ways but I get errors like the following:

model = Sequential()
model.add(LSTM(units = un , input_shape = (20, 4), return_sequences = False))
model.add(RepeatVector(25))
model.add(LSTM(un , return_sequences=True))
model.add(TimeDistributed(Dense(20)))

  ValueError: Error when checking target: expected lstm_419 to have
  shape (25, 20) but got array with shape (25, 1)

The shape of the labels array was ( 25 , 1 ). The shape of your previous model was ( 25 , 20 ). The dimension 20 was because of this line.

model.add(TimeDistributed(Dense(20)))

The Dense layer had an output dimension in the form of units= argument  as 20. 25 was the length of the sequence.

  So, on each of the 25 timesteps, a Dense layer operation was done and
  each produced a array of shape ( 20 , ). This process happened for 25
  timesteps which resulted in a final output of shape ( 25, 20 ).

RepeatVector : It repeats the input vector given to it n number of times.

Timedistributed: Executes the layer operation which is wrapped in it ( in your case it was Dense layer ) for every timestep in the given input sequence.

To know more about the above layers, refer to this answer of mine.

Hence, In the last layer changing Dense( 20 ) to Dense( 1 ) will correct the vector dimension.


Error: Input 0 is incompatible with layer lstm_1: expected ndim=3, found ndim=2 I am getting the error:
Input 0 is incompatible with layer lstm_1: expected ndim=3, found ndim=2

Using the model:
X = np.reshape(x_train_tfidf.shape[0], 1, x_train_tfidf.shape[1])
print(X.shape)
model.add(LSTM(30, return_sequences=True,
               input_shape=X))

model.add(Flatten())
model.add(Dropout(0.25))
model.add(Dense(100,activation = 'relu'))
model.add(Dropout(0.25))
model.add(Dense(1,activation='sigmoid'))

It's probably this way..(not sure , give it a try)

Iputs needs to be reshaped to be [samples, time steps, features]

so

TrainX= np.reshape(TrainX,(TrainX.shape[0], 1, TrainX.shape[1]))

You need to passin the actual vector also?


call Python script from JavaEE i create model using sklearn library and i want to run this model in JavaEE application i have been trying Jython, but it's impossible to import some important library like pandas and numpy, so how I can do to call a python script for JavaEE application.

You should probably ask programming questions on StackOverflow as more people will be able to help you with this kind of problems. As suggested here, you can serialize scikit-learn model to PMML and then execute it using jpmml-evaluator. There are different suggestions in that post, so you should take a look at it and see what works best for your use case.


Process mining with ML I have a little more general question. My dataset consists of N sequences of events. Example of one sequence could be [A,B,C,D,X,Y] and another [A,B,Z], where letters represent different events. The sequences are at most 80 steps long.

The idea is to predict next letter or next step from known previous events. For very simple example maybe after A will always come B. Next step would be measuring time of each event and the ultimate goal is to predict how long until process reaches specific event.

I tried N-gram, MLP neural network and lastly LSTM network, which had around 80% accuracy.

That would not be bad if the events were balanced in the dataset. To account for that I used weighted loss function in training of the LSTM and then the overall accuracy is around 66%. However the less frequent classes have much much higher accuracy (still not perfect, but higher). How can I create model that will have the best of both? That will learn the less frequent AND the most frequent at the same time.

Also I have read that tree base methods perform very good on unbalanced dataset. However all examples always consider one big timeseries data. My data are many short timeseries. Is it possible to train RandomForest on such data? How?

If you know about different algorithm/method that could be applied to such data please post it :)

Thank you.

I suspect that the problem has more to do with your data than with your algorithms. My recommendation is to spend some time studying your data and ensuring that it is a robust representation of the kinds of problems you're expecting to solve. If possible, come up with a way to generate extra data. Given the fact that you already have many permutations, you could perhaps write a script to create additional permutations by modifying existing samples with rules that you know.


Web Scraping: Multiple small files or one large file? I plan to scrape some forums (Reddit, 4chan) for a research project. We will scrape the newest posts, every 10 minutes for around 3 months. I am wondering how best to store the JSON data from each scrape, so that pre-processing (via Python) later would be as simple as possible. My options are the following:

Dump data from each scrape into a fresh file (timestamp as filename). Resulting in 12,960 files of approx. 150kb each OR
Maintain 1 single large file, whereby the scraper simply appends the new output to the end of the file. Resulting in 1 file of size approx 1.9Gb after 3 months

Does anyone have any recommendations or warnings from their experience about either approach and how this affected pre-processing? I am cautions that having a pre-processing script work on a larger file might take longer to process but then again, opening and closing thousands of files will also be time-consuming.

The danger of one large file is that you may not be able to read the whole thing into memory at once.  Since you already know that 3 months of data will be ~2GB, this should not be a problem, so I would recommend option 2.

Perhaps an even better option is to save the data as several medium-sized files.  For example, you could combine all the observations for a week into a single file.  This way, you eliminate nearly all of the file I/O overhead (~13,000 files down to 12-14) but also free yourself from potential memory problems.
This is certainly what I would do if I were collecting data for a very long time (years instead of months).


Regression model with variable number of parameters in dataset? I work in physics. We have lots of experimental runs, with each run yielding a result, y and some parameters that should predict the result, x. Over time, we have found more and more parameters to record. So our data looks like the following:

Year 1 data: (2000 runs)
    parameters: x1,x2,x3                target: y
Year 2 data: (2000 runs)
    parameters: x1,x2,x3,x4,x5          target: y
Year 3 data: (2000 runs)
    parameters: x1,x2,x3,x4,x5,x6,x7    target: y

How does one build a regression model that incorporates the additional information we recorded, without throwing away what it "learned" about the older parameters?

Should I:

just set x4, x5, etc to 0 or -1 when I'm not using them?
completely ignore x4,x5,x6,x7 and only use x1,x2,x3?
add another parameter that is simply the number of parameters?
train separate models for each year, and combine them somehow?
"weight" the parameters, so as to ignore them if I set the weight to 0?
make three different models, using x1,x2,x3, x4,x5, and x6,x7 parameters, and then interpolate somehow?
make a custom "imputer" to guestimate the missing parameters (using available parameters)

I have tried imputation using mean and median, but neither works very well because the parameters are not independent, but rather fairly correlated.

One simple idea, no imputation needed: build a model using the parameters have always existed, then each time a new set of parameters gets added, use them to model the residual of the previous model. Then you can sum the contributions of all the models that apply to the data you happen to have. (If effects tend to multiply rather than add, you could do this in log space.)