Creating new column in dataframe based on conditions in 2 other columns I would like to create a new column in my dataframe based on values from both the gender and experimental_grouping columns.

As I have it written below, the column df['group_gender'] has 'control_m' in every row.

Any ideas on why the if statement isn't working/a better way to accomplish this goal?

def group_and_gender(group):
    for i in group:
        if i == 'Controls':
            for i in df['gender']:
                if i == 'M':
                    df['group_gender'] = 'control_m'
                else:
                    df['group_gender'] = 'control_f'
        else:
            for i in df['gender']:
                if i == 'M':
                    df['group_gender'] = 'dp_m'
                else:
                    df['group_gender'] = 'dp_f'

group(df['experimental_grouping'])

-the problem with an inaccurate filling of column group_gender is that in df['group_gender'] = 'dp_m' in the following code, if i == 'M' you are filling the whole column with dp_m, instead you should use methods like iloc but it is not really an efficient way specifically when having a large dataset. In following, I have provided a better way.

for i in df['gender']:
    if i == 'M':
        df['group_gender'] = 'dp_m'
            else:
        df['group_gender'] = 'dp_f'

The more appropriate way

#creating an instant dataframe
import pandas as pd
d = {'gender':['M','M','F', 'M', 'F','M']}
df = pd.DataFrame(d)

#filling the 'group_gender' column
df.loc[df['gender'] == 'M', 'group_gender'] = 'dp_m'
df.loc[df['gender'] == 'F', 'group_gender'] = 'dp_f'

-You are not filling in experimental_gropuping and returning the column:

group(df['experimental_grouping'])

-What is this group? (Can you provide more details about the first part of your code?)


What Kind of Network should I be considering? was hoping someone can guide me in the right direction with this problem.

I have a dataset that contains the history of an experiments that lasts 40 minutes; taking a measurement every minute. I have historical data for about 600 runs. So I have a total of 24000 rows in my dataset.

I would like to set up a model in where I can train it on the different multiple runs where all 40 minutes are available; and each minute has same feature count of 11.

My plan then was to feed the model anywhere from 2-35 rows of live data, and I would like it to predict one of the values at the 40 minute mark.

Any help would be greatly appreciated.

You want to use a Recurrent Neural Network.  More specifically, look into the LSTM variant and understand it's use cases over a "regular" RNN, and go from there.  RNN's are good for capturing sequential data patterns and this is essentially what you want to do.


What are the standard parameters values for SMOTE technique? I'm working on an imbalanced class data set (200 samples) with 2 classes, first class has 50 sample and second has 150 sample.

My questions:

When I use SMOTE technique on my data set my total dataset samples
will be greater than 200 or smaller?
Is there standard parameters value about SMOTE technique or not?

First of all, if all your variables are numerical then you can you SMOTE; otherwise you should use SMOTENC.

Answers:
1- Your classes are 150-50, then SMOTE gives you 150-150. So, YES; your total dataset sample will be 300.

2- You can use the default parameters; it works well in most of the problems. However, you can play with random_state and k_neighbors. All parameters have been explained here: imblearn.over_sampling.SMOTE


Your BPE vocabulary is quite small given how the strings you want to segment look like. The biggest problem here is that BPE expects tokenized sentences with tokens separated by spaces. The BPE model is not aware of JS syntax and you feed it with JS code, so there is no wonder that it does not learn anything syntactically plausible.

You can interpret the BPE file a log of what subwords got merged and you can see that the longest string you get is something like:

m').valueOf
H:mm').valueOf

I would recommend some preprocessing, perhaps some JS lexer, so it respects the syntax and perhaps bigger vocabulary.


Removing junk sentences I have transcripts of phone calls with customers and agents. I'm trying to find promises which were made by an agent to a customer.

I already did punctuation restoration. But there are a lot of sentences that don't have any sense. I would like to remove them from the transcript. Most of them are just a set of not connected words.
I wonder what approach is the best for this task?

My ideas are:

• Use tf idf and word2vec to create vectors from all sentences. After that we can do some kind of anomaly detection e.g. look for and delete vectors that are highly deviated from most other vectors.

• Spam filters. Maybe is it possible to apply spam filters for this task?

• Crate some pattern of part of speech tags that proper sentence must include. For example, any good sentence must include noun + verb. Or we can use for example dependency tokens from spacy.

update

Example of a sentence that I want to keep:

  There's no charge once sent that you'll get a ups tracking number.

Example of a junk sentence:

  Kinder pr just have to type it in again, clock drives bethel.

Another junk sentence:

  Just so you have it on and said this is regarding that.

I would suggest training your own custom Dialog Act Classification Model. Detecting a dialog act is a Natural Language Understanding (NLU) problem. Using a dialog act classification model you can detect if a sentence is used for "greeting", "question", "opinion", "promise", etc.

Detailed approach:

Use a pre-trained language model (for example BERT), to train your own Dialog act classification model. You can create your own custom Dialog act class based on your requirement (as you want to remove specific types of sentences). You can class them as junk or use "greeting", "question", "opinion", etc as dialog class.

After you are done with the training you can loop through the list of sentences and filter them based on their predicted classes.

  You might have to research if there are similar corpus which you can
  use or have to manually label and then train the model. It will be a more reliable approach as compared to the word2vec or rule-based POS approach.

For more details on state-of-the-art approaches and dialogue act corpus follow this link. If you have no idea how to use a language model you can use the transformers library from huggingface. I hope you find this helpful.


How does neural network solve XOR problem I'm reading a wonderful tutorial about neural network. This is the best tutorial I've ever seen but I can't understand one thing as below:

In the link above, it is talking about how the neural work solves the XOR problem.

It says that we need two lines to separate the four points. But I don't know the second table.

XOR:

input1    input2    output
  0         0         0
  0         1         1
  1         0         1
  1         1         0

First table:

input1    input2    output
  0         0         0
  0         1         1
  1         0         1
  1         1         1

In my opinion, the first table is OK because it includes the XOR, which means that what the second table need to do is to remove the forth input. So I think the second table should be as below:

input1    input2    output
  0         0         1
  0         1         1
  1         0         1
  1         1         0

How in the link it says the second table is like this:

input1    input2    output
  0         0         0
  0         1         0
  1         0         0
  1         1         1

In a word, I can understand why the single layer neural network can't solve the XOR problem but I can't understand how the two layers neural network work to solve it.

Notice that the first table (orange line) is performing an OR operation and the second table (blue line) is performing an AND operation. XOR can be defined as (x OR y) AND NOT (x AND y) or $(x \lor y) \land \lnot (x \land y)$, so in other words: orange should fire and blue shouldn't fire. If we now look at this figure from your tutorial:

we can see that the weight of blue in the second layer is negative and small enough (more negative) such that the output can never fire if blue fires, i.e. the output can't fire if both inputs are firing.

$0 + 0 \ngtr 1 : \emptyset $ shouldn't fire

$-2 + 0 \ngtr 1 : (x \land y) = T$ shouldn't fire

$0 + 1.1 \gt 1 : (x \lor y) \land \lnot (x \land y) = T$ should fire

$-2 + 1.1 \ngtr 1 : (x \land y) \land (x \lor y) = T$ shouldn't fire


Association rules for classification I'm working on a classification project. I have many rows, containing many binary attributes, some of which are often appearing together, exactly like what we can encounter in the Market Basket problem (in which you can, for example identify, that if you buy 'Milk' to a supermarket, you also have a more than random chance to buy 'Eggs').
My idea is then to take my target as an attribute, extract best Item-set containing my target (so having Target=1, exactly like my previous example if I wanted to make a model to predict whether the client will buy eggs or not).
Then, looking at the fit of each item-set, I'll have the info of which groups of products have best chances to make my target be 1.
Even though the method looks good, I have a problem. I'm wondering if there's a way to only extract item-sets with target on them.
All articles I found about making classification based on association rules explain that one of the big downside of the method is time taken, because it has to find all item-sets, and after removing all the ones that are not including target.
Is there a way, using item-set properties, to specify the algorithm (Apriori, FP-Growth, etc.) to only calculate Item-sets including Target, and not waste time calculating all item-sets?

If I understand correctly you don't have your target and want to create it using associacion rules.
Fp-growth algorithm already reduces itemset checks e.g. based on the fact that if a certain itemset doesn't match selected threshold any other superset of this itemset won't match it either.
For this reason it is several times faster than apriori.
If you want to create your target using associacion rules and then train a model to predict this target you need let the algorithm go over each example and let it discard each itemset not matching your threshold.
What you can do to speed up the computation is:

Use FP-Growth on a distributed system.
Use even more restrictive condititons to discard itesets faster.
Learn association rules only on a subset of items for example top 1000 bestsellers.


Keras input for multivariate classification with LSTM using current features and previous timesteps features and y values I am working on a multivariate binary classification problem. What I want to do is to predict a binary classification given the features at the current timestep and the data (features+real classification) from past timesteps

Keras seems to have a problem with the shape of my inputs so I want to know what I am doing wrong:

X_train = (nb_samples, nb_timesteps, nb_features)
y_train = (nb_samples, nb_timesteps, binary_result)

model = Sequential()
model.add(LSTM(nb_units,
               input_shape = X_train.shape[1:]
              ))
Dense(1,activation='softmax')
model.compile(loss='categorical_crossentropy',optimizer='adam',metrics=['accuracy'])

history = model.fit(X_train, y_train, epochs=ep, validation_data=(X_train, y_train), verbose=2, shuffle=False)

Since you want to get a classification/output per each time step, you should set return_sequences=True in you LSTM layer.

Read more about return_sequences here in LSTM documentation.


Why is taking the gradient of the average error in SGD not correct, but rather the average of the gradients of single errors? I am a little confused about taking averages in cost functions and SGD. So far I always thought in SGD you would compute the average error for a batch and then backpropagate it. But then I was told in a comment on this question that that was wrong. You need to backpropagate the error of every item in the batch individually, then average the gradients you computed through the backpropagation and then update your parameters with the scaled average gradient.

Okay, but why is that not actually the same thing? Isn't the gradient of the average of some points the average of the gradient at these points?

The idea behind SVD is to find the minimum of a cost function $J(\theta)$ of a subset of training items. The cost function is usually defined as the average of some function $J_t(\theta)$ of the errors of the individual predictions and targets for a training item. Let's take MSE as an example. So if we have a batch of $N$ items, we have

$$J(\theta)  : X, Y \mapsto \frac{1}{N} \sum_{i=1}^N (y_i -f(x_i))^2$$

And we want to minimize $J(\theta)$. So we need to find its gradient:

$$\nabla \frac{1}{n}\sum_{i=1}^{n} (y_i - f(x_i))^2$$

But the derivative is linear, so

$$\nabla \frac{1}{n}\sum_{i=1}^{n} (y_i - f(x_i))^2 = \frac{1}{n}\sum_{i=1}^{n} \nabla (y_i - f(x_i))^2$$

What am I doing wrong here?

Another example. Say we do linear regression with an $m x + b$ line fit. Then the partial derivatives taken for $m$ and $b$ are

\begin{align*}
\frac{\partial J(\theta)}{\partial m}   &= \frac{1}{N} \frac{\partial}{\partial m} \sum_{i=1}^N (y_i -f(x_i))^2 & \texttt{factor rule}\\
                                &= \frac{1}{N} \sum_{i=1}^N \frac{\partial}{\partial m} (y_i -f(x_i))^2 & \texttt{sum rule}\\
                                &= \frac{1}{N} \sum_{i=1}^N  2(y_i -f(x_i)) \frac{\partial}{\partial m} y_i -f(x_i)  & \texttt{chain rule}\\
                                &= \frac{1}{N} \sum_{i=1}^N  2(y_i -f(x_i)) \frac{\partial}{\partial m} y_i - (mx_i + b)  & \texttt{definition } f\\
                                &= \frac{1}{N} \sum_{i=1}^N  2(y_i -f(x_i)) (-x_i)  & \texttt{}\\
                                &= -\frac{2}{N} \sum_{i=1}^N  x_i(y_i -f(x_i))  & \texttt{comm., distr.}\\
\end{align*}

\begin{align*}
\frac{\partial J(\theta)}{\partial b}   &= \frac{1}{N} \frac{\partial}{\partial b} \sum_{i=1}^N (y_i -f(x_i))^2 & \texttt{factor rule}\\
                                &= \frac{1}{N} \sum_{i=1}^N \frac{\partial}{\partial b} (y_i -f(x_i))^2 & \texttt{sum rule}\\
                                &= \frac{1}{N} \sum_{i=1}^N  2(y_i -f(x_i)) \frac{\partial}{\partial b} y_i -f(x_i)  & \texttt{chain rule}\\
                                &= \frac{1}{N} \sum_{i=1}^N  2(y_i -f(x_i)) \frac{\partial}{\partial b} y_i - (mx_i + b)  & \texttt{definition } f\\
                                &= \frac{1}{N} \sum_{i=1}^N  2(y_i -f(x_i)) (-1)  & \texttt{}\\
                                &= -\frac{2}{N} \sum_{i=1}^N  (y_i -f(x_i))  & \texttt{comm., distr.}\\
\end{align*}

I don't see an error here and the gradient descent also works with these partial derivatives (tested through implementation). So... what am I missing?

The gradient of the average error doesn't always equal to the average gradient of errors.
The source for the difference between them lies in the non-linear layers of the model.

Example:

You can easily see it in the following example with the gradient of the sigmoid function:

The sigmoid function is defined as:

It has a very convenient derivative:

We now take 2 inputs  and calculate the mean of the sigmoid's gradient with respect to them:

We now calculate the sigmoid's gradient with respect to their mean:

These 2 results are clearly not the same. If you want further proof, just calculate the numerical results for:

You will get that the mean gradient is ~0.2233, and the gradient of means is ~0.235.


What are some situations when normalizing input data to zero mean, unit variance is not appropriate or not beneficial? I have seen normalization of input data to zero mean, unit variance many times in machine learning. Is this a good practice to be done all the time or are there times when it is not appropriate or not beneficial?

A detailed answer to the question can be found here.

  [...]are there times when it is not appropriate or not beneficial?

Short answer: Yes and No. Yes in the terms, that it can significantly change your output of e.g. clustering algorithms. No, on the other hand, if these changes are what you want to achieve. Or to put it in the words of the author of the mentioned source:

  Scaling features for clustering algorithms can substantially change the outcome. Imagine four clusters around the origin, each one in a different quadrant, all nicely scaled. Now, imagine the y-axis being stretched to ten times the length of the the x-axis. instead of four little quadrant-clusters, you're going to get the long squashed baguette of data chopped into four pieces along its length! (And, the important part is, you might prefer either of these!)

The take-home-message of this is: always think carefully about what you want to achieve and what kind of data your algorithms prefer - it does matter!


Audio Analysis : Segment audio based on speaker recognition I have audio clips of people being interviewed and am trying to split the audio clips using python such that all speech segments of the interviewee are outputted in one audio file (eg .wav format) & that of the interviewer in another audio file. Speaker recognition needs to be performed using unsupervised learning.

So far I have found a few libraries that perform the speaker diarization task (pyAudioAnalysis, aalto-speech) but none that combine the different speaker segments and output it in separate audio files. How do I segment the audio files & combine them based on the speaker?

Initially I did using silence detection but later moved to pyAudioAnalsis which is better.

Check "Speaker Diarization" section in Segmentation in pyAudioAnalysis


Machine Learning model to find items that are frequently bought together using Hadoop Spark I need to do my master thesis under Big Data Analytics - Machine Learning subject. I was challenged to create a Market Basket Analysis project on retail industry.

My dataset contains transactions of a supermarket over a year and I will have to predict which products are associated.

The requirements for my project are:
- We must visualize the evolution of the model (Learnable Evolution Model)
- Must be done using Hadoop tool (+ Spark)

I already look for multiple source codes/tutoriais to learn how can I do this but I can't find a machine learning implementation project for find patterns using Spark...

Anyone have an idea how can I find some good tutorials to learn this skills for me to do my master thesis algorithm?

I don't know about learnable evolution models, but Spark ML has a module for collaborative filtering. This can be used to find objects that are similar/often bought together etc. Amazon uses it to show you the "People who bought this also bought these items" feature on their website.


Using EM (Expectation Maximization) algorithm for Training Logistic Regression Is it possible to learn the weights for a logistic regression classifier using EM (Expectation Maximization)algorithm? Is there any instance reference?

Use Algorithm 1 from this paper. The rest of paper is more about the analysis of algorithm and theoretical background which is very nice but not necessary for your purpose. Implementation of the algorithm in Python or R is pretty straightforward.

Hope it helps :)


Solving Bayes Theorem equation -> I can't calculate proper result I am solving questions for an edx course on Machine Learning.
One particular question is giving me a problem:

Assume a patient comes into the doctor’s office to test whether they have a particular disease.
The test is positive 85% of the time when tested on a patient with the disease (high sensitivity):
P(test+|disease)=0.85
The test is negative 90% of the time when tested on a healthy patient (high specificity):
P(test−|heathy)=0.90
The disease is prevalent in about 2% of the community:
P(disease)=0.02
Using Bayes' theorem, calculate the probability that you have the disease if the test is positive.

My solution:

I have created a table

   sick   |   healthy
     2%   |   98%
+   90%   |   15%
-   10%   |   85%

From this I calculated bayes theorem like this:

                      (0,02*0,9)
P(A|B) = -----------------------------------------------------
               (0,02*0,9) + (0,15*0,98)

I get P(A|B)=0,109
however this answer is wrong,
where did I do mistake?

Bayes theorem: $P(A|B) = \frac{P(B|A) P(A)}{P(B)}$

In our case: $P(disease|test_+) = \frac{P(test_+|disease) P(disease)}{P(test_+)}$

Given:

$P(test_+|disease) = 0.85 \\
P(test_-|healthy) = 0.9 \\
P(disease) = 0.02$

Since a test can be positive while the patient has a disease and it also can be positive while the patient is healthy, the overall probability for a positive test is calculated as follows:

$P(test_+) = P(test_+|disease) P(disease) + P(test_+|healthy) P(healthy) \\
P(test_+|healthy) = 1 - P(test_-|healthy) \\
P(healthy) = 1 - P(disease)$

So:

$
\begin{align}
P(disease|test_+)
&= \frac{P(test_+|disease) P(disease)}{P(test_+|disease) P(disease) + P(test_+|healthy) P(healthy)} \\
&= \frac{P(test_+|disease) P(disease)}{P(test_+|disease) P(disease) + (1 - P(test_-|healthy)) (1 - P(disease))} \\
&= \frac{0.85 \times 0.02}{0.85 \times 0.02 + (1 - 0.9) (1 - 0.02)} \\
&= \frac{0.017}{0.017 + 0.1 \times 0.98} = \frac{0.017}{0.115} \approx 0.148
\end{align}
$

So, the probability that you have the disease given that you were tested positively is around 14,8% in this case.


How do I generate a laplacian matrix for a graph dataset? If I have a dataset in a csv that looks like the one shown below.

How do I convert this into a laplacian matrix using Python?

Well the Laplacian matrix is achieved by:

$degree (v_i) $ for $\space$ i=j

$-1$ for $\space$ if $v_j$ and $v_i$ are not adjacent to each other

$0$ otherwise

First, you need to store your file to a 2d-array
Then you need to define another 2d-array matrix the same size of your first matrix. Then loop over the elements to fill the Laplacian matrix

import pandas as pd
data = pd.read_csv('data.csv')
df = pd.Dataframe(data)
M = df.as_matrix()
L = np.zeros(df.shape[0], df.shape[1]) #shape[0] and shape[1] should be equal

Then for each element $A_{i,j}$ we calculate their corresponding value in L

for i in range(len(df.shape[0])):
    for j in range(len(df.shape[0])): # or shape[1]
        if M[i][j] == 0 and M[j][i]== 0:
            L[i][j] = -1
        if i == j:
            L[i][j] = sum(M[i][:])
        else:
            L[i][j] = 0

I haven't tried the code so consider it much like a pseudo-code.


neural network xor gate classification I've written a simple neural network that can predict XOR gate function. I think I've used the math correctly, but the loss doesn't go down and remains near 0.6. Can anyone help me find the reason why?

import numpy as np
import matplotlib as plt

train_X = np.array([[0,0],[0,1],[1,0],[1,1]]).T
train_Y = np.array([[0,1,1,0]])
test_X = np.array([[0,0],[0,1],[1,0],[1,1]]).T
test_Y = np.array([[0,1,1,0]])

learning_rate = 0.1
S = 5

def sigmoid(z):
    return 1/(1+np.exp(-z))

def sigmoid_derivative(z):
    return sigmoid(z)*(1-sigmoid(z))

S0, S1, S2 = 2, 5, 1
m = 4

w1 = np.random.randn(S1, S0) * 0.01
b1 = np.zeros((S1, 1))
w2 = np.random.randn(S2, S1) * 0.01
b2 = np.zeros((S2, 1))

for i in range(1000000):
    Z1 = np.dot(w1, train_X) + b1
    A1 = sigmoid(Z1)
    Z2 = np.dot(w2, A1) + b2
    A2 = sigmoid(Z2)

    J = np.sum(-train_Y * np.log(A2) + (train_Y-1) * np.log(1-A2)) / m

    dZ2 = A2 - train_Y
    dW2 = np.dot(dZ2, A1.T) / m
    dB2 = np.sum(dZ2, axis = 1, keepdims = True) / m
    dZ1 = np.dot(w2.T, dZ2) * sigmoid_derivative(Z1)
    dW1 = np.dot(dZ1, train_X.T) / m
    dB1 = np.sum(dZ1, axis = 1, keepdims = True) / m

    w1 = w1 - dW1 * 0.03
    w2 = w2 - dW2 * 0.03
    b1 = b1 - dB1 * 0.03
    b2 = b2 - dB2 * 0.03

    print(J)

I've solved the problem with three changes:

The weights are too small at the beginning, remove the scaling
I think the biases should also be randomly initialized:

w1 = np.random.randn(S1, S0)
b1 = np.random.randn(S1, 1)
w2 = np.random.randn(S2, S1)
b2 = np.random.randn(S2, 1)

Your are using a fixed learning rate of 0.03, change it to use the learning rate, and you can also increase it:

    learning_rate = 0.1 ...

    w1 = w1 - dW1 * learning_rate
    w2 = w2 - dW2 * learning_rate
    b1 = b1 - dB1 * learning_rate
    b2 = b2 - dB2 * learning_rate

The image below shpws what I get for the progression of J:

Just for curiosity, if we change the sigmoid function with the ReLU, take a look to what happens to J (in the code I didn't change the name of the defs...). The learning with ReLU is much faster.

:

    def sigmoid(z):
    return np.maximum(0,z)

    def sigmoid_derivative(x):
    x[x < = 0] = 0
    x[x > 0]  = 1
    return x


Is Nvidia Jetson product family also suitable for machine learing model training? I recently came accross these products (Nvidia Jetson) and they are all tagged as "edge", so i think they are designed only for machine learning inference and not model training.
They are quite interesting for their low power consumpion and price (eg: Jetson Nano) so i hope they are suitable also for model trainig.
So i would ask if someone may clarify this aspect about the focus of the product.

Training a neural network involves a lot more computation than inferencing from a pre-trained one. During training, a neural network must go through both forward propagation and back propagation steps. The back propagation step is computationally expensive, since it requires many gradient computations and updates to weight variables.

During inference, the only computation needed is forward propagation, where the input is multiplied to the weights to produce a prediction. Boards like the NVIDIA Jetson are optimized for fast matrix multiplication, hence why they are effective during inference but not as much for training.

Another reason for the Jetson's popularity is its small form factor. Architects choose it because it can fit within the size and price requirements of the product while also offering decent performance. If you had unlimited budget and can fit a GTX Titan in your product for inferencing, you would do that, right? I worked in the autonomous vehicle sector for a while and that there is no way we could fit a machine with any full-size GPU in a car. There are no size limitations for training models, so naturally you would go after the machines with more powerful and efficient GPUs.


Machine learning to predict apps (recomendation) I am currently collecting mobile device data about user location and the time at which an app is being used for a cohort of users and apps.  I am trying to predict which apps are likely to be used at a given time by a given user.

Which model should i use in order to predict the apps that are likely to be used by a particular user? For example, should I collect data for 5 days a week then use the 3 days for training and then the other 2 days for testing? Given location, time of day, and day, what model should I use to predict which app is likely to be used?

Much depends on the data available to you.  Perhaps you can be more specific about the scale and scope.

Modelling time is the straightforward bit.  To understand how to conceptualise time as useful features, see this excellent answer on Machine learning - features engineering from date/time data

Modelling the user is more complicated.  You will likely not have enough data on each user, but you can build some user models.  (Too few, then the system will make similar predictions for all users, without nuance.  Too many and there will be sparsity, overfitting, and generally the same problems as having no profile models at all, ie one model per actual user.)

This can be done supervised or unsupervised, finding representative clusters.
(Search for user profile categorisation, user models, user model clustering)


Transposed convolution as umpsampling in DCGAN I read several papers and articles where it is suggested that transposed convolution with 2 strides is better than upsampling then convolution.

However implementing such model with the transposed convolution resulted in heavy checkboard effect, where the whole generated image is just a pattern of squares and no learning takes place. How to properly implement it without totally messing up the generation? With the upsampling+convolution I got okay result but I want to improve my model. I am trying to generate images based on the CelebA dataset.

I use keras with tf and I used the following code:

model.add(Conv2DTranspose(256, 5, 2, padding='same'))

model.add(LeakyReLU(alpha=0.2))

model.add(BatchNormalization(momentum=0.9))

model.add(Conv2DTranspose(128, 5, 2, padding='same'))

model.add(LeakyReLU(alpha=0.2))

model.add(BatchNormalization(momentum=0.9))

model.add(Conv2DTranspose(64, 5, 2, padding='same'))

model.add(LeakyReLU(alpha=0.2))

model.add(BatchNormalization(momentum=0.9))

Here I try to turn a 4x4 image into a 32x32. Later it will be turned into a 64x64 image with 1 or 3 channels depending on the image. However I get the following pattern always. Some tweaking usually leads to some other pattern but it does not really change:

Checkboard effect

Thank you for your answers in advance

This checkerboard effect arises due to taking sum of the overlapping regions during the transpose-convolution operation. Post convolution, in the output, the overlapping pixels have a higher magnitude than the surrounding pixels.
Try using a filter size of 4x4 along with a stride 2 transpose convolution. It may help to alleviate this problem.

For more information, watch this lecture from 30th minute.


Improve CoreNLP POS tagger and NER tagger? The CoreNLP parts of speech tagger and name entity recognition tagger are pretty good out of the box, but I'd like to improve the accuracy further so that the overall program runs better. To explain more about accuracy -- there are situations in which the POS/NER is wrongly tagged. For instance:

"Oversaw car manufacturing" gets tagged as NNP-NN-NN

Rather than VB* or something similar, since it's a verb-like phrase (I'm not a linguist, so take this with a grain of salt).

So what's the best way to accomplish accuracy improvement?

Are there better models out there for POS/NER that can be incorporated into CoreNLP?
Should I switch to other NLP tools?
Or create training models with exception rules?

Your best best is to train your own models on the kind of data you're going to be working with.


How to identify if there is a relationship between 5 categorical independent variables to a binary dependent variable? My dataset has 5 independent variables, each with a value of either Large, Medium or None and a binary dependent variable. The dataset has 67 rows with a split of 17:50.
I would like to identify if there is a relationship between the independent variables and the dependent variable and to explore which variables/set of variables have the largest impact on the result.
Most of the solutions I have checked so far (Nominal Variable Association and Cramér's V) don't support more than one independent variable.

One simple, tricky but probably best approach would be using a Decision Tree. After building a model, you can analyze the structure of the tree to identify the relation. Afterward, you can list the feature importance to see the effect of an independent variable on the dependent variable.
In python, you can use sklearn to achieve it. feature_importances_ will give you Gini's importance of the features and the plot_tree() function to see the structure of the tree. This article might also help to see how to do that.


Are there any python based data visualization toolkits? Data visualization is an important sub-field of data science and python programmers need to have available toolkits for them.

Is there a Python API to Tableau?

Are there any Python based data visualization toolkits?

Bokeh is an excellent data visualization library for python.
NodeBox is another that comes to mind.


How to create column for my csv file in python i have csv file having these values (without column):-

I:30n
J:0n
J:0n
U:1000n
C:0n
I:12n
I:10n
I:10n
I:10n
I:10n

I want to add column name for these rows values.

  Suppose all I (i.e: I:30n, I:12n, etc)value record should be in
  one column and likewise all J (i.e J:0n, J:0n) should be in one
  column and vise-versa using python.

Can anyone help?

You can use Pandas for this, your file format isn't exactly comma separated values file. But still you can us pandas read_csv() method.
Suppose you're file name is test_file

import pandas as pd
df = pd.read_csv('test_file', sep=':', header=None)
>>> df
   0      1
0  I    30n
1  J     0n
2  J     0n
3  U  1000n
4  C     0n
5  I    12n
6  I    10n
7  I    10n
8  I    10n
9  I    10n

Then you can use the pivot() function:

>>> df.pivot(columns=0)
\     1
0    C    I    J      U
0  NaN  30n  NaN    NaN
1  NaN  NaN   0n    NaN
2  NaN  NaN   0n    NaN
3  NaN  NaN  NaN  1000n
4   0n  NaN  NaN    NaN
5  NaN  12n  NaN    NaN
6  NaN  10n  NaN    NaN
7  NaN  10n  NaN    NaN
8  NaN  10n  NaN    NaN
9  NaN  10n  NaN    NaN

If your intention is to write it back to a file you can use to_csv() method.

# this row eliminates the level headers of the columns at level 0
>>> df.columns=df.columns.get_level_values(1)
>>> df
0   C    I   J      U
0      30n
1           0n
2           0n
3               1000n
4  0n
5      12n
6      10n
7      10n
8      10n
9      10n
>>> df.to_csv('new_test_file', index=False)

OR

If you wish to make it less sparse, you can first turn it into a dict and then back to DataFrame:

>>> _dict = df.groupby(0)[1].apply(list).to_dict()
>>> _dict
{'C': ['0n'], 'I': ['30n', '12n', '10n', '10n', '10n', '10n'], 'J': ['0n', '0n'], 'U': ['1000n']}
>>> pd.DataFrame.from_dict(_dict, orient='index')
       0     1     2     3     4     5
C     0n  None  None  None  None  None
I    30n   12n   10n   10n   10n   10n
J     0n    0n  None  None  None  None
U  1000n  None  None  None  None  None

>>> pd.DataFrame.from_dict(_dict, orient='index').T
      C    I     J      U
0    0n  30n    0n  1000n
1  None  12n    0n   None
2  None  10n  None   None
3  None  10n  None   None
4  None  10n  None   None
5  None  10n  None   None

pd.Series.to_dict()
pd.DataFrame.from_dict()
pd.DataFrame.T


Machine Learning methods suited for CPU I have a large number of x86_64 cores available to me, but no GPUs or TPUs. Which Machine Learning techniques are suited for execution on a CPU? I would imagine more "statistical learning" techniques rather than "deep learning".

The reason I ask is that I'm interesting in studying some state-of-the art techniques, but I'm in the situation noted above.

With 12 or more CPU cores, you wan work with pretty much any deep learning model.  Only thing that will not work is training with 100k+ images or more than TBs of test.  For such scenarios; you can seed the model with transfer learning.

For example, training a Cat/Dog image classification model (from scratch) takes 2 hours on 6 core i5 (as compared to 30 minutes on GTX 1080ti).


In Keras how to get the `class_indices` or prediction labels for an existing model I know that keras provides a .class_indicies dictionary containing the mapping from class names to class as part of .flow_from_directory() from its ImageDataGenerator class  (https://keras.io/preprocessing/image/).

HOWEVER, is there a way to access the corresponding class labels from an existing saved model (a model saved in as an .h5 file)?  This seems important when putting my model into production and serving predictions since the classes are not known upfront and therefore the images are not separated in pre-labeled directories.

Similarly, are there any good examples of how a keras model should be deployed into production (especially for mobile)?  Thanks!

Don't know if you solved already.
I faced this problem yesterday, and the only solution that came to my mind is saving a dictionary to a numpy file after a train_generator is set; something like:

np.save('FILENAME', self.train_generator.class_indices)

when you finish your training.
Then, when you do your single prediction:

single_pred = np.squeeze(self.model.predict(image))

if isfile('FILENAME.npy'):
    class_indices = np.load(self.indices_file + '.npy').item()

results_dict = dict(zip(class_indices.keys(), single_pred))


TypeError: Expected int32, got None of type 'NoneType' instead I want my model batch size to be a dynamic shape, and I've assigned none as batch size, but that's causing an error.
Here, in the first line, I specified batch size as None:
inp=L.Input(shape=(28,28,1),batch_size=None)
conv1=L.Conv2D(filters=256,kernel_size=(2,2),activation='relu',padding='valid')(inp)
maxpool1=L.MaxPooling2D(pool_size=(1,1))(conv1)
conv2=L.Conv2D(filters=128,kernel_size=(9,9),activation='relu',padding='valid')(maxpool1)
conv2=L.Conv2D(filters=8*16,kernel_size=(9,9),strides=2,padding='valid',activation=None)(conv2)
# Adding the squash activation
reshape2=L.Reshape([-1,8])(conv2)
squashed_output=L.Lambda(squash)(reshape2)

CapsuleLayer code:
class CapsuleLayer(L.Layer):
    def __init__(self,num_capsule,dim_capsule,routing=3,kernel_initializer='glorot_uniform',**kwargs):
        super(CapsuleLayer,self).__init__(**kwargs)
        self.num_capsule=num_capsule
        self.dim_capsule=dim_capsule
        self.routing=routing
        self.kernel_initializer=kernel_initializer
    def build(self,input_shape):
        assert len(input_shape) >= 3
        self.input_num_capsule=input_shape[1]
        self.input_dim_capsule=input_shape[2]

        #transforming the matrix
        self.W= self.add_weight(shape=[self.num_capsule,self.input_num_capsule,self.dim_capsule,self.input_dim_capsule],initializer=self.kernel_initializer,name='w')
        self.built=True
    def call(self,inputs,training=None):
        input_expand=tf.expand_dims(tf.expand_dims(inputs,1),-1)
        inputs_tiled=K.tile(input_expand,[1,self.num_capsule,1,1,1])
        input_hat=tf.squeeze(tf.map_fn(lambda x: tf.matmul(self.W,x),elems=inputs_tiled))
        b=tf.zeros(shape=[inputs.shape[0] ,self.num_capsule,1,self.input_num_capsule])
        assert self.routing > 0
        for i in range(self.routing):
            c=tf.nn.softmax(b,axis=1)
            output=squash(tf.matmul(c,input_hat))
            if i<self.routing-1:
                b+=tf.matmul(output,input_hat,transpose_b=True)
        return tf.squeeze(output)
    def compute_output_shape(self,input_shape):
        return tuple([None,self.num_capsule,self.dim_capsule])
    def get_config(self):
        config = {
            'num_capsule': self.num_capsule,
            'dim_capsule': self.dim_capsule,
            'routings': self.routings
        }
        base_config = super(CapsuleLayer, self).get_config()
        return dict(list(base_config.items()) + list(config.items()))

The following line when CapsuleLayer function called, the error raised:
digitcaps = CapsuleLayer(num_capsule=3, dim_capsule=16, routing=3, name='digitcaps')(squashed_output)

The error:

TypeError: Expected int32, got None of type 'NoneType' instead.

When I assign a number to the batch size, no error occurred.

The error is raised by tf.zero() because the shape parameter should be a list of ints.
Indeed, as you build the computation graph, batch size is not (yet) know, unless you explicitly specify if.
Sometimes you need to specify shape information manually when working with Tensorflow Keras API.
One possible fix is to use tf.shape() function instead of shape attribute:
   b=tf.zeros(shape=[tf.shape(inputs)[0] ,self.num_capsule,1,self.input_num_capsule])

This line will run without error, but now the batch dimension of b is defined by a computation. It is dynamic and Keras doesn't know anymore, that it equals the batch size.
Maybe because of this softmax(b, axis=1) does not work anymore. Interestingly softmax() with the default axis setting works. So as a workaround we can juggle dimensions back and forth:
    c=tf.transpose(tf.nn.softmax(tf.transpose(b,perm=[0,2,3,1])),perm=[0,3,1,2])

To make the whole code work, you also need to specify shape information more precisely at the beginning of the code:
    reshape2=L.Reshape([6*6*128//8,8])(conv2)

Also you might notice, that all shape information is lost after the CapsuleLayer. You can check it with digitcaps.shape. But we can re-specify it:
    digitcaps = L.Reshape([3,16])(digitcaps)


Neural Network Architecture for Identifying Image Copies I have a large image collection and wish to identify the images within that collection that appear to copy other images from the collection.

To give you a sense of the kinds of image pairs that I wish to classify as matches, please consider these examples:

I have hand classified roughly .25M pairs of matching images, and now wish to use those hand labelled matches to train a neural network model. I am just not sure which architecture would be ideally suited for this task.

I originally thought a Siamese Network might be appropriate, as they have been used for similar tasks, but the output from those classifiers seems more ideally suited to finding different figurations of the same object (which is not what I want), rather than different printings of the same figuration (which is what I want).

If anyone can help recommend papers or architectures ideally suited to identifying images given the training data I have prepared, I would be tremendously grateful for any insights you can offer.

U need to read about triplet loss function. Triplet loss function gets result embeddings from a network, that process 3 images by a network (2 similar and 1 non-similar) for one step:

After that loss computing as:

For more details read the paper from triplet loss authors.

Also may help PSNR, but this is not Deep Learning.


How to do template matching without opencv? How to do template matching without opencv ?

I have a order invoice of documents belonging to Amazon ,ebay,flipkart,snapdeal and i want to extract few information from the order invoice .
Since the fields like order number,customer name ,order details will be present at different position in these 4 templates ,

i need to first classify to which of these 4 templates the input image will belong to and after identifying the template i can do my next work of text extraction using tesseract and regex buy writing code for specific templates .

when i searched i found only openCV had this feature i was not able to find any convolution neural network model .Are there any such neural network model available for template matching and classification ?

i cannot use the standard model available for classifying dogs and cat because here im dealing with image of an invoice templates which has only text with rows and columns in some format .

import numpy as np
import matplotlib.image as img
import matplotlib.pyplot as plt
from skimage.metrics import structural_similarity as ssim

def rgb2gray(rgb):
    return np.dot(rgb[...,:3], [0.299, 0.587, 0.144])

full_image = rgb2gray(img.imread("img/full_image.png"))
sub_image = rgb2gray(img.imread("img/sub_image.png"))

full_w,full_h = full_image.shape[:2]
sub_w,sub_h = sub_image.shape[:2]

print(full_w,full_h)
print(sub_w,sub_h)

winW = 0
found = False
while winW < full_w - sub_w and found == False:
    winH = 0
    while winH < full_h - sub_h:
        window = full_image[winW:winW+sub_w, winH:winH+sub_h]
        if ssim(sub_image, window) > 0.80:
            found = True
            print("found", ssim(sub_image, window))
            plt.imshow(window)
            break
        winH += 2
    winW += 2


K-Nearest Neighbor Classifier Best K Value I created a KNeighborsClassifier for my dataset adjusting the k hyper-parameter (the number of neighbors) in a for loop. The k value was between 1 and 20. The result was the graph below:

How do I interpret this graph? Which would be my best k value?

There's several ways that you can choose your k value for kNN -

You can use the common formula k = sqrt(n) where n is the number of data points in your training set or you can try choosing k where there is a good balance between computation expense vs noise.

Consider your what fits your problem:
Do you care about runtime? The higher the k, the more expensive computationally it is to run.

It looks like around  k= 6-10 you get some diminishing returns - you could set it there to get a good balance between noise and computation cost, but ultimately it is a very arbitrary selection, so pick what suits your use case best.


Attention Mechanism: Why use context vector instead of attention weights? In attention, the context vector ($c$) is derived from the sum of the attention weights ($\alpha$) multiplied by the encoder hidden states ($h$), where the weights are obtained by multiplying the decoder hidden state and the encoder states.

$c_i = \Sigma_j^{T_x} \alpha_{ij} h_j$

My question is, why calculate this context vector and simply not forward the attention weights, as these could indicate how much to focus on each of the encoder states.

Could somebody explain the intuition behind this?

The attention weights themselves do not carry any information about the encoded sentence. It only tells you something like: the $n$-th and $m$-th word carry important information for generating the next word, but not what the words actually are. Moreover, it is a variable-length vector (of the same length as the input size), it would be hard to do anything with it in the decoder.


Single vs Multiple deep learning networks for multi-label classification? Given a machine reaches a broken state, there are potentially fixes that can to be applied to get the machine to run again. We'd like to know if, for the problem further defined below, with millions of datapoints, hundreds of features, and tens of labels, should we be looking at using a single deep neural network with multiple outputs, or create an ensemble of binary networks with human-selected features presumed better for their relevance? Is there a standard approach?

The state of the machine is captured in potentially hundreds of features, a good mix of continuous values and categorical data. We have millions of documented cases of machine states, and can identify when a machine had broke and what fixes were applied to get it running again. There are less than 40 fixes we are interested in, and we are considering a bucket for "other" fixes, and a bucket for "unfixable".

We are treating this as a multi-label classification, because it may take multiple discrete fixes (Fix-1, Fix-2...Fix-N) to get the machine up and running again. Not all features would be relevant to each fix, so the question we're wondering is whether each fix should have its own binary classification (each outputing a single value representing Fix-i or not Fix-i) network with what we think are the relevant features, or should we create one giant deep neural network with multiple labels, sigmoid over each fix (Fix-1, Fix-2...Fix-N). With the latter approach, should combinations of fixes be represented as their own label (Fix-1, Fix-2, Fix-1&2, Fix-3, Fix-1&3, Fix-2&3, Fix-1&2&3 ... etc).

This kind of setup is normally addressed with a single neural network that has an output sigmoid layer of $N$ units (where $N$ is the number of possible fixes).

Having combinations of fixes represented with their own label does not make much sense; instead, let the different labels take the expected value: if 2 fixes are fine, let both expected outputs be 1.

Having $N$ networks each one generating a single fix-nofix is equivalent but does not benefit from the potential gains of the multitask learning obtained when training everything together and obtaining a shared internal representation.


Lasso regression not getting better without random features First of all, I'm new to lasso regression, so sorry if this feels stupid.
I'm trying to build a regression model and wanted to use lasso regression for feature selection as I have quite a few features to start with.
I started by standardising all features and plotting the weights of each feature as I changed my regularisation strength to see which ones are most important. I also plotted the RMSE on holdout set to find a U-shaped plot, i.e. as I increased the regularisation strength, my RMSE would decrease, then after a certain point it start to increase. I couldn't. My RMSE plot was non-decreasing.
Then, I decided to throw in a random feature to see how the model would perform and compare my features vs. the random one. My expectation was again to see a U shape. However, that wasn't the case as I got another non-decreasing RMSE plot again which I pasted below. It also looked like two of my features, feat1 and feat3 were just as good as my random feature.

So here are my questions.

Why did could my RMSE have kept increasing as I increased my regularisation strength?
Does the fact that the coefficients of feat1 and feat3 were pushed to 0 at the same time as the random feature mean that they're not good and I should remove them from the model? (It felt strange that they hit 0 at the same time. I was expecting some difference.)

In answer to your first question:
The reason that your RMSE proceeded to increase as you increased the strength of your regularization (the value of $\lambda$) can be explained by reviewing the intuition behind what is happening when you increase the regularization of your model.

Why did could my RMSE have kept increasing as I increased my regularization strength?

When you have no penalty (i.e. $\lambda = 0$) the error for your model (RMSE) is inflated significantly. As you penalize the model this error begins to fall. At some point in your modeling, by increasing the value of $\lambda$ too much, you are adding too much bias to the model. At this point, the bias will become too large and your model will begin to under-fit the data. This will result in an increase in the RMSE.
For your model above, this seems to occur somewhere between $\lambda = 10^{-3}$ and $\lambda = 10^{-2}$.
In answer to your second question:
Lasso Regression can be used as a form of feature selection since it shrinks some of the values for the coefficients down to zero. If you increase the value of $\lambda$ significantly, many of the features will be shrunk to zero.

Does the fact that the coefficients of feat1 and feat3 were pushed to
0 at the same time as the random feature mean that they're not good
and I should remove them from the model? (It felt strange that they
hit 0 at the same time. I was expecting some difference.)

To decide whether you should remove any of these features, you should consider a few things. What is the degree of collinearity that exists between your features. You can check this easily since it looks like your have a small number of features. Also, it is important to consider the number of features you have before regularization. If you have a small number, perhaps you do not want to drop a large amount of them.
Most importantly, you should examine the change in your RMSE error associated with each increase in $\lambda$ and choose your $\lambda$ that minimizes the RMSE. If certain of the features are pushed to zero based on the optimal choice of $\lambda$ then maybe they should be removed. The rate at which they move towards zero (both at the same time) does not necessarily suggest that they should be removed, only that they might be candidate features to remove.


K-Means vs hierarchical clustering When hierarchical clustering is preferred over k means clustering?

I would say hierarchical clustering is usually preferable, as it is both more flexible and has fewer hidden assumptions about the distribution of the underlying data.

With k-Means clustering, you need to have a sense ahead-of-time what your desired number of clusters is (this is the 'k' value).  Also, k-means will often give unintuitive results if (a) your data is not well-separated into sphere-like clusters, (b) you pick a 'k' not well-suited to the shape of your data, i.e. you pick a value too high or too low, or (c) you have weird initial values for your cluster centroids (one strategy is to run a bunch of k-means algorithms with random starting centroids and take some common clustering result as the final result).

In contrast, hierarchical clustering has fewer assumptions about the distribution of your data - the only requirement (which k-means also shares) is that a distance can be calculated each pair of data points.  Hierarchical clustering typically 'joins' nearby points into a cluster, and then successively adds nearby points to the nearest group.  You end up with a 'dendrogram', or a sort of connectivity plot. You can use that plot to decide after the fact of how many clusters your data has, by cutting the dendrogram at different heights.  Of course, if you need to pre-decide how many clusters you want (based on some sort of business need) you can do that too.  Hierarchical clustering can be more computationally expensive but usually produces more intuitive results.


why does ID3 Decision tree algorithm not give the best decision tree? I was going through ID3 algorithm, and what I believe is it incorporates Greedy Search rule to get come up with the decision tree. If it gives the best split possible at every stage, how does it not give the best decision tree possible?

If you want the best decision tree, you need the "optimal decision tree", which you can get nowadays. Just have a look into the vast literature.


Unstable results in test mode with fractional max pooling in PyTorch I make some variants of ResNet, originally found in TorchVision, modify them, train them and so on. What I have found is that even in .eval() mode, even if I load state right before evaluation, I receive different results. The code looks like

...
imageData = imageDataset.getImage(imageNum)
imageData = np.expand_dims(imageData, 0)
pytImageData = torch.from_numpy(imageData).cuda()
...
self.loadState(self.curEpochNum)
model.eval()
self.pytOptimizer.zero_grad()
with torch.set_grad_enabled(False):
    activations = model.forward(pytImageData).cpu().numpy()

    self.loadState(self.curEpochNum)
    activations2 = model.forward(pytImageData).cpu().numpy()
    diff = activations2 - activations
    print('diff', diff.min(), diff.max())

Difference is quite high. I have found this during investigation of occlusion heatmaps, they were quite noisy (and I would even say gave strange results, but maybe this is just one more mystery of neural nets, with different nature).

ImageNet-1000 dataset,
torch version 1.5.0.dev20200128,
torchvision version 0.6.0.dev20200128

Another question appears given that this is caused by fractional max pooling: how to make it stable? As I suppose, only random mode is implemented in PyTorch, so random positions are selected for each convolution at each run. But the authour also invented pseudorandom mode. Is it possible for example to turn it on somehow?

The answer (maybe one of) is FractionalMaxPool2d layer I use. If I use say ResNet-34, train it a bit (say 1/6 of epoch), everything is ok. If I add one FractionalMaxPool2d layer:

...
self.layer1 = self._make_layer(block, 2, 64, layers[0])
self.maxpool2 = nn.FractionalMaxPool2d(kernel_size=3, output_size=38)
self.layer2 = self._make_layer(block, 3, 128, layers[1], stride=2,
...

(and run it in the forward method of course)

and repeat, the difference appears.


Mine webshop history for clusters I've no experience in data science so this will be one of those questions...

I have data from >100k purchases made via a webshop regarding a catalogue of around >100 items. The history of purchases flattened out looks like

Item1 Item2 ... ItemN Sex State
    5     0         0   M    NY
   25    15         0   F    IL
    0     1         1   ?    NY

By playing around with the data, I can deduce simple facts like "90% of all purchases include at least 3 Item1", "If there are at least 4 of Item2, it is likely that Item3 is 0" or "60% of all customers from NY are male, but only 40% of those from IL are". Given the amount of combinations and data there is, the most obvious question: How can I approach wringing out more information from a data set like the above? I'm mostly interested in how one item does or does not entails inclusion of another...

Frequent Item-Set Mining is what you are looking for. You can see the tree structure of your frequent itemsets and the association rules afterwards.

For your data I'd suggest to look at the whole for a while to get a sense on what you have in hand. Playing with concepts like Probability Distributions, Entropy, etc would be really helpful in case you can reduce the size of your features.

PCA also gives you the opportunity of projecting your data into a low-dimenstional space and you can see also plots showing first several PCs in 2-D or 3-D and get an impression about your data.

Before all above I strongly suggest to see if you have Missing Values and if yes try to cope with them.


Statistical inference on a very small datasets I have been working with machine learning for about a year now, but mostly with large datasets. However, I am currently working on a problem with a very small dataset. Here is my problem: I am creating a rocket fuel with 4 ingredients, x1, x2, x3, x4, and I want to maximize reaction strength, y. I have already mixed them in the arrangements below to get the corresponding values.

(0.9)x1 + (0.0)x2 + (0.1)x3 + (0)x4 = 16.5
(0.0)x1 + (0.9)x2 + (0.1)x3 + (0)x4 = 8.6
(.45)x1 + (.45)x2 + (0.0)x3 + (0.1)x4 = 12.6
(0.6)x1 + (0.3)x2 + (0.05)x3 + (.05)x4 = 18.9
(0.3)x1 + (0.9)x2 + (0.05)x3 + (.05)x4 = 9.8

My next question is, how should I design my next few mixtures to maximize the reaction strength?  Can you suggest any algorithms or statistical frameworks to get me started? Much appreciated.

There are two separate issues:

Sampling - Picking the optional ingredient level for next experiment to run. Given you have only have 4 explanatory variables, just plot them. Either all pairwise or a couple of 3d charts. With the outcome variable on the y or z axis. You'll then see the trend in the data. You can decide to get more data for interpolation (between the data points you already have) or extrapolation (data outside of the current range). There are frameworks, such as Bayesian Optimization, but that is too much work given the small dimensionality.
Inference - Predicting performance for new data. Given the data you have seen thus far (sample data), estimate parameters. In your example that would the estimating the contribution of each of the 4 ingredients, either individually or interaction. Those parameters could be scalar coefficients or distributions.


SVM model classifying into one class only, after standardization I'm trying to use SVM in R (e1071 package) to classify samples as normal or tumor. I have two separate data sets - Training (~50 samples, 100 features) and Test (~60 samples). These data sets are microarray expression values from two different microarray platforms performed by two different research groups and so their value ranges are very different. I am new to SVM and R, and so at first, I did a Z-score standardization on my Training set and then I did a Z-score standardization on my Test set separately. i.e.

(x_Train - mean_Train) / (stdev_Train)

(x_Test - mean_Test) / (stdev_Test)

As I explored more forum posts on this topic, I see that it is suggested to use the parameters from Training set on my Test set

(x_Test - mean_Train) / (stdev_Train)

So that my SVM model is more generalizable and can be applied to new samples.

Here is the problem,

When I standardized the sets separately, my best model (after 10-fold cross validation using the tune function) had around 75% accuracy classifying the test set. However, after standardizing as suggested with my Training parameters the accuracy of my model to classify the Test set dropped drastically to 25% using the same features.

I tried using different combinations of features to see if it makes a difference and I saw that no matter what I had changed, all samples from the Test set were classified into one class (all in normal, and zero in tumor), while my previous model with separate standardization classified samples into both classes.

Here is an example of one of the features

Training set (raw)
         Feature 1
Sample 1 25.82977
Sample 2 42.62437
Sample 3 28.91158
Sample 4 53.5708
Sample 5 30.92296
Sample 6 99.16994
Sample 7 40.75973

Test set (raw)
         Feature 1
Sample 1 2.885865028
Sample 2 2.572860413
Sample 3 2.809136715
Sample 4 2.259630716
Sample 5 2.797155715
Sample 6 2.439700763
Sample 7 2.197087754

Training set (standardized)
         Feature 1
Sample 1 -0.795137358
Sample 2 -0.132081654
Sample 3 -0.673466601
Sample 4  0.300086594
Sample 5 -0.594056732
Sample 6  2.100353934
Sample 7 -0.205698184

Test set (standardized)
         Feature 1
Sample 1 -1.700969415
Sample 2 -1.713326928
Sample 3 -1.703998671
Sample 4 -1.725693328
Sample 5 -1.704471684
Sample 6 -1.71858411
Sample 7 -1.728162542

I know SVM uses a threshold to classify samples into either normal or tumor. From what I showed above, their ranges are completely different, is that what is affecting its decision?

What is the problem here and how can I tackle it, thank you in advance. I am so lost and need every bit of advice!

*Edit: As you can see the values of Training set have a completely different range from the values of the Test set, can I still use the mean and standard deviation from my Training set to scale the corresponding features in the Test set?

I can't add a comment so I'll post this here.

My main question is why are your feature values so wildly different different between you your test and training sets?

For example the raw values of feature 1.

Training set: All values > 20, looks like they average about 35-40.
Test set: All values around 2, looks like they all fall between 2-3.

What is the reason for this. Also have you shuffled you data before performing a train  test split? The you should plot a distribution of the feature values for both your training and test sets sets, and make sure they are somewhat similar.

It looks like you you have a bad covariate shift between training and test sets. Don't worry about this yet as I think it's a data preprocessing problem. First answer the question if the distribution of values of each feature are similar between training and test sets.


Sequence to sequence RNN model, maximum number of training size So when running this example script from Keras repo (https://github.com/keras-team/keras/blob/master/examples/lstm_seq2seq.py), I found that we can easily run into out of memory for the input or output one-hot encoding in this code:

encoder_input_data = np.zeros(
    (len(input_texts), max_encoder_seq_length, num_encoder_tokens),
    dtype='float32')
decoder_input_data = np.zeros(
    (len(input_texts), max_decoder_seq_length, num_decoder_tokens),
    dtype='float32')
decoder_target_data = np.zeros(
    (len(input_texts), max_decoder_seq_length, num_decoder_tokens),
dtype='float32')

When I have a huge size of training samples, this one-hot does not fit into memory. Is there way to handle this issue?

UPDATE:
I changed float32 to uint, but thats about the smallest one hot array can get.

so I found a solution by dividing input training data into small batches and it did the trick. Here is the code:

divideby=1000
for j in range(divideby):
    start=j*len(input_texts)/divideby
    end=(j+1)*len(input_texts)/divideby if j<divideby+1 else -1

    encoder_input_data = np.zeros(
        (end-start, max_encoder_seq_length, num_encoder_tokens),
        dtype='uint8')  # size_of_training_samples, max_length_word_measured_in_characters,number_of_unique_chars,
    decoder_input_data = np.zeros(
        (end-start, max_decoder_seq_length, num_decoder_tokens),
        dtype='uint8')
    decoder_target_data = np.zeros(
        (end-start, max_decoder_seq_length, num_decoder_tokens),
        dtype='uint8')

    for i, (input_text, target_text) in enumerate(zip(input_texts[start:end], target_texts[start:end])):
        for t, char in enumerate(input_text.split(splitby)):
            encoder_input_data[i, t, input_token_index[char]] = 1.
        for t, char in enumerate(target_text.split(splitby)):
            # decoder_target_data is ahead of decoder_input_data by one timestep
            decoder_input_data[i, t, target_token_index[char]] = 1.
            if t > 0:
                # decoder_target_data will be ahead by one timestep
                # and will not include the start character.
                decoder_target_data[i, t - 1, target_token_index[char]] = 1.

    model.fit([encoder_input_data, decoder_input_data], decoder_target_data,
              batch_size=batch_size,
              epochs=epochs,
              validation_split=0.2)
```


Is color information only extracted in the first input layer of a convolutional neural network? In a convolutional neural network (CNN), since the RGB values get multiplied in the first convolutional layer, does this mean that color is essentially only extracted in the very first layer?

Snippets from Stanford CS231n Chapter on CNN:

  [...] One dangerous pitfall that can be easily noticed with this
  visualization is that some activation maps may be all zero for many
  different inputs, which can indicate dead filters, and can be a
  symptom of high learning rates [...] Typical-looking activations on
  the first CONV layer (left), and the 5th CONV layer (right) of a
  trained AlexNet looking at a picture of a cat. Every box shows an
  activation map corresponding to some filter. Notice that the
  activations are sparse (most values are zero, in this visualization
  shown in black) and mostly local.

The quote that you have brought is speaking about something else. Zero means that most of your linear part of neurons have a negative value and after passing through the ReLU non-linearity, they become zero. That's the explanation of that part. About your question, not really. I don't know if you've ever visualized a convolutional network or not but you've done that you may have seen that there may be layers that get activated for special colors even in deep layers. The reason is that the convolutional layer just tries to do cross-correlation operation on the input which has three channels. Next layers of convolution also do this considering e.g. the input has 64 channels. To wrap up, there may be other deep-layers that use the information of colors, Although these operations really do not care about the colors, they care about the local patterns that may happen in the input. The reason for learning colors is that similar colors have similar patterns in their RGB channels. These patterns are what convolutional networks try to learn. Take a look at the tool box that have been discussed here.


sklearn.naive_bayes VS categorical variables In a binary classification, how can I use sklearn.naive_bayes python module to predict the class of inputs with 5 categorical variables (not binary)?

Hot encode the categorical variables and use Bernoulli naive Bayes. Hot encoding is usually the trick one uses in representing categorical variables.


Siamese Neural Network for massive class variation I have case to check whether the person has been registered into a database,and if the images has high similiarity with one of the image in the db, i want to retrieve the image information (name, id, etc) to email or other party. The problem is the dataset are very limited (I only get 1 image per person) so i augmented the image and get about 30 variation of augmented faces. At first i have thinking using Siamese NN since it has good performance with limited data source, but realise its need pair of image (the gaithered and one from the dataset), loop through big dataset (need to loop through 3k of image), SNN is not too efficient. I afraid using VGG or Facenet since the augmented dataset are not too big enough to gain certain accuracy.
Is there any suggestion based on my case? The one idea i have to tackle it is to use NN as feature and use similarity calculation based from stored NN.

I have some discussion about this problem with my friend and found a good solution. The answer is to using CNN not as a classifier but as feature generator (embedded) and store it as vector. The next is to using vector similarities to check whether the person is same or not or who is in the picture.


Cannot see what the "notation abuse" is, mentioned by author of book From Sutton and Barto, Reinforcement Learning: An Introduction (second edition draft), in equation 3.4 of page 38.

  The probabilities given by the four-argument function p completely characterize the dynamics of a
  finite MDP. From it, one can compute anything else one might want to know about the environment,
  such as the state-transition probabilities (which we denote, with a slight abuse of notation, as a threeargument function

  $p(s^{'} | s, a) \dot{=}Pr\{S_t=s^{'} | S_{t-1} = s, A_{t-1}=a\} =
\sum_{r\in{R}}{p(s^{'},r|s,a)}$

The author mentioned, with a slight abuse of notation.
where is the abuse in the notation please? I didn't see anything that is not proper.

Thank you.

The mathematical expression is completely legit. The abuse is in the fact that the function $p$, which is defined first time in equation 3.2, which:

  The function $p: S$ x $R$ x $S$ x $A \rightarrow [0,1]$. is an ordinary
  deterministic function of four arguments...

is re-defined slightly differently just two lines after this definition (equation 3.4), as a three-argument function $p: S$ x $S$ x $A \rightarrow [0,1]$.

If they used $p$ to represent the regular probability measure, there would be no abuse. In the authors' notation, $p$ is a deterministic function, while the regular probability function is denoted as $Pr$; and keeping the same name for slightly different functions, is where the "innocent" notation abuse comes from.


Overfitting in machine learning The graph above shows how accuracy stops increasing after reaching a certain number of features. There are also sudden drops in accuracy at some points. Can this be attrrubuted to overfitting? I am training a decision tree by the way.

I can tell from your screenshot that you are plotting the validation accuracy. When you overfit your training accuracy should be very high, but your validation accuracy should get lower and lower. Or if you think in terms of error rather than accuracy you should see the following plot in case of overfitting. In the figure below the x-axis contains the training progress, i.e. the number of training iterations. The training error (blue) keeps decreasing, while the validation error (red) starts increasing at the point where you start overfitting.

This picture is from the wikipedia article on overfitting by the way: https://en.wikipedia.org/wiki/Overfitting Have a look.

So to answer your question: No, I don't think you are overfitting. If increasing the number of features would make the overfitting more and more significant the validation accuracy should be falling, not stay constant. In your case it seems that more features are simply no longer adding additional benefit for the classification.


Choose CNN architecture first, then optimize parameters - validation vs test performance to pick architecture? I am doing a few experiments on medical data. I am about to transfer learn the pretrained networks for my problem.

Firstly, I have to pick a network architecture.
Secondly, I would like to optimize it's parameters/parameters of optimizer, to get better performance.

I would like to pick the network architecture based on 10-fold cross validation of several architectures. I will perform cross validation in a way that I have data split to train:test in a 80:20 manner, then train is split into 10 splits. Test set shouldn't ever change. Based on the cross validation, I would like to pick a model I would optimize further on validation set.

Is it okay to test the best architecture on test data too, to see whatever I am moving in a right direction, before optimizing the parameters or am I cheating?

The dataset is imbalanced with relatively high class variance, so I am not even sure whatever the 2 folds for testing will trully represent the dataset.

Thanks

You shouldn't choose the best model based on the performance on test set. You should run cross-validation and pick the best model from there and then assess performance on a test set. Otherwise you would just getting the best model and overfitting your test set.

Regarding imbalanced data sets, you should be aware of metrics used to assess your model in this case, such as recall, precision, etc., and not only accuracy.
In python, you can do StratifiedKFold, where you will split your data getting the same proportion for train and test set regarding target variable.


How to correctly pass Word2Vec vectors as input to an LSTM I am trying to build a text classifier using lstm which, in its first layer, has weights get by a Word2Vecmodel.

In order to build a matrix containing the indexes of each word for each sentence, I have tried: (as mentioned here)

X_tr_word2vec = np.array(X_tr_word2vec)
y_tr_word2vec = np.array(y_tr_word2vec)

train_x = np.zeros([X_tr_word2vec.shape[0], max_sentence_length], dtype=np.int32)
train_y = np.zeros([y_tr_word2vec.shape[0]], dtype=np.int32)

for i, sentence in enumerate(X_tr_word2vec):
    for j, word in enumerate(sentence[:-1]):
        train_x[i,j] = model_word2vec.wv.vocab[word].index

but, when I run the code, I get this error:KeyError: 'enquiringly', what does it mean?
I suppose that it is about a wrong train_xdimension.

Update:

I have trained Word2Vec model before, with the entire training set:

model_word2vec = models.Word2Vec(X_tr_word2vec, size=150, window=9)

That means your word 'enquiringly' is not in your word embedding vocabulary vocab.

For out-of-vocabulary(OOV) words, there is usually a embedding vector dedicated to them. Try to find that special symbol in the vocab and use that corresponding embedding vector.


Mask R-CNN Background Subtraction Implementation I am currently attempting to reimplement a paper on fall detection (https://ieeexplore.ieee.org/abstract/document/9186597). It requires a background subtraction algorithm called Mask R-CNN. Are there any current implementations of this algorithm for background subtraction?

I found this, https://github.com/tony92151/maskrcnn-background-remove
but I didnt use it. I will try using it this week and post more updates here.


Architecture for multivariate multi-time-series model where some features are TS specific and some features are global I'm looking to build a time series model (using a TCN or a LSTM)  with $N$ different series, each of which has $P$ series-specific features $\mathbf{X}$.  My input array is of dimension $N \times t \times P$, where $t$ is the number of time steps.

I've also got features $G$, which are constant across all time series.  For concreteness, imagine I'm predicting city-level ice cream sales with weather data, and I also want to use GDP growth as a predictor.  GDP growth is national.  A simple approach could be to augment $\mathbf{X}$ with $G$, adding 1 to the dimension of $P$.  Then my forecast output for the next period would be $N \times 1 \times P+1$, which is no good because there is a GDP forecast for each city, when in reality GDP growth is common across cities (when measured nationally).  I suppose that I want two outputs -- one of shape $N \times 1 \times P$, and the other of shape $1 \times 1 \times 1$, a scalar (if $G$ is of dimension $t \times 1$).

Here's a dummy example in which time is a global variable, but it is constant across all series.  (Let's just assume for the moment that time isn't exogenous, but rather something to include in a multivariate forecast).

import numpy as np
import matplotlib.pyplot as plt
from keras.models import Model
from keras.layers import Input, Conv1D, Dense
from keras.optimizers import Adam

time = np.array(range(100))
brk = np.array((time>40) & (time < 60)).reshape(100,1)
B = np.array([5, -5]).reshape(1,2)
np.dot(brk, B)
y = np.c_[np.sin(time), np.sin(time)] + np.random.normal(scale = .2, size=(100,2))+ np.dot(brk, B)

plt.plot(time, y[:,0])
plt.plot(time, y[:,1])

# Temporal convolutional network
n_filters = 2
filter_width = 3
dilation_rates = [2**i for i in range(5)]
inp = Input(shape=(None, 2))
x = inp
for dilation_rate in dilation_rates:
    x = Conv1D(filters=n_filters,
               kernel_size=filter_width,
               padding='causal',
               activation = "relu",
               dilation_rate=dilation_rate)(x)
x = Dense(2)(x)

model = Model(inputs = inp, outputs = x)
model.compile(optimizer = Adam(), loss='mean_squared_error')
model.summary()

def shift5(arr, num, fill_value=np.nan):
    result = np.empty_like(arr)
    if num > 0:
        result[:num] = fill_value
        result[num:] = arr[:-num]
    elif num < 0:
        result[num:] = fill_value
        result[:num] = arr[-num:]
    else:
        result = arr
    return result

X = y.reshape(2,100,1)
X = np.concatenate([X, np.concatenate([time.reshape(100,1),time.reshape(100,1)], axis = 1).reshape(2,100, 1)],
                    axis = 2)
X_tr = X[:,:95,:]
X_te = X[:,5:,:]

history = model.fit(X_tr, X_te,
                batch_size=2,
                epochs=10,
                verbose = 1)

How would I modify this architecture to have two inputs and two outputs, with both input and output having local and global components?

Stacked LSTM is one option in this scenario

This assumes that First two LSTMs have different frequencies and City has static features (Like lat/long, one-hot-encoded value etc). If City is also time-series like series of population , mean income; it will be an LSTM as well.
Code example for stacked LSTM.


Good introductory reference for Bayesian Non-parametric (Dirichlet Process / Chinese Restaurant Process) I am looking for a recommendation for basic introductory material on Bayesian Non-parametric methods, specifically Dirichlet Process / Chinese Restaurant Process. I am looking for material which covers the modeling part as well as the inference part from ground-up.

Most of the material I found on the internet has slightly advanced material and they skip the inference part, which is usually harder to grasp.

I would recommend Tamara Broderick's 3-part tutorial series from MLSS. She explains both modeling + inference using these methods from the ground up and focuses on the intuition.

Links to part 1, part 2, and part 3.


What kind of algorithm should I choose for this music classification system? I have in mind a program for analyzing short fragments of music, categorizing them as "good" or "bad". This would be part of a larger program that searches for larger good phrases and whole pieces.

The idea now is to take a fragment of music, X, and compare it to known good sample fragments G_1, G_2, ... etc. and get a rank of similarity to each one. Then compare it to known bad fragments B_1, B_2, B_3, .. etc.

"Good" music is subjective of course, but this program would work with G and B samples that I hand-optimized according to my own tastes.

Good music is then music that resembles at least one of the G's, while resembling none of the B's. A fragment that has strong similarity to both G's and B's is probably bad: The B's have veto power.

So, how to determine similarity? Musical fragments can be represented by image-like matrices of pixels. The vertical axis is pitch and the horizontal axis is time. If a note of pitch P_1 occurs between times T_beg and T_end, then that's like drawing a little line between (T_beg, P_1) and (T_end, P_1).

An sample X to be classified can be convolved, in a sense, with a known sample K. It can be transposed up or down (vertical shifting) or moved left or right in time (or stretched in time, or other alterations) and each transposition would be superimposed on the G or B sample. I'm not too familiar with convolution but I think that overlapping pixels are multiplied and the sum of all is taken. The transposition with the brightest result pixel is a good indication of how similar X is to the K sample: it's magnitude becomes the measure of similarity.

Dark pixels don't matter much. A preponderance of dark pixels doesn't make music bad. It just means the real pattern isn't found in those locations. A bright match to a known bad fragment is what makes music bad.

I'd like to perform these computations with NumPy or a similar language optimized for matrix or image computations.

Can I get some idea whether there is a name for this kind of operation, and where to look for efficient implementations of it? Boosting speed with a GPU would be a bonus.

There are two high level approaches (Approach 2 was a better fit for a music-classification problem that I worked on) :

Signal processing + CNN  : Output of signal processing is saved as image. Models use the image as input.

This paper is a good intro to the approach : https://arxiv.org/ftp/arxiv/papers/1712/1712.02898.pdf

Couple of articles on this : https://www.codementor.io/vishnu_ks/audio-classification-using-image-classification-techniques-hx63anbx1 , https://medium.com/datadriveninvestor/audio-and-image-features-used-for-cnn-4f307defcc2f

Raw audio  + recurrent networks : https://deepmind.com/blog/wavenet-generative-model-raw-audio/  ,  https://arxiv.org/pdf/1606.04930.pdf , https://arxiv.org/pdf/1612.04928.pdf , https://gist.github.com/naotokui/12df40fa0ea315de53391ddc3e9dc0b9

GPU will make the project easier, but is not a requirement.


Linear regression with white Gaussian noise I am new to machine learning , so this question may sound fundamental. My task is to estimate the parameter vector of the equation with the least squares method:

$y = \theta_0 + \theta_1x + \theta_2x^2 + η$

Where η corresponds to white Gaussian noise with mean 0 and variance 0.1

Also , I have been given the prior values of the parameter vector , say [-1,0.3,0.6] . I have to generate the N points of the training set .

Should regresor be like this :

[1 0.1 (o.1)^2] (for x = 0.1)

And should I calculate a priori results by adding noise

η is an error term with distribution (0, sigma). The error or residual is how the true x differ from the estimated function.

Do you mean the theta when you say „parameter vector“.

In this case, the estimated function is:
y_hat = -1 + 0.3x + 0.6x^2.

So if you want to infer the training set, you need to generate x points, evenly scattered around the quadratic function (mean of error = 0) with a variance of 0.1.


What is Happening in the training process when we are fitting a model to the data In any prediction task, the process of “fitting” a model to the data observed in the training process can be best described as...

Assessing all observations available and then backsolving for the dimensionality of best fit.
Making use of all observations available to create a model, then using distance or other measures to assess how well the model characterizes or "fits" its data.
Starting with a linear model and then expanding the dimensionality of that model until it fits the data
Starting with a linear model and then regressing the shape of the line until it fits the data.

These are the four points I am confused about. Which one/ones is/are true?

Options 2 and 4 are correct.
During fitting a model the training process Makes use of all observations available to create a model, then using distance or other measures to assess how well the model characterizes or "fits" its data. Also, Starting with a linear model and then regressing the shape of the line until it fits the data.


How to detect overfitting of a stock screener The project I am working on allows users to create Stock Screeners based on both technical and fundamental criteria.  Stock Screeners are then "backtested" by simulating the results of applying in over the last 10 years using Point-in-Time data. I get back the list of trades and overall graph of performance.
(If that is unclear, I have an overview here and there with more details).

Now a common problem is that users create overfitted stock screeners. I would love to give them a warning when the screen is likely to be over-fitted.

Fields I have to work with

All trades made by the Stock Screener

Stock, Start Date, Start Price, End Date, End Price

S&P 500 performance for the same time frame
Market Cap, Sector, and Industry of each Stock

Learning curves or bias-variance decomposition are the gold standard for detecting high variance, aka: overfitting.  Separate your data (in your case the "back data") into 60% training data and 40% testing data.  Fit the model on the training data as you usually would and see how well it is working with the test data.

Finally, when you think you have the model that you want, split each of the training and test sets into 10-100 subsets and retrain and test with incrementally larger sets.  Apply your favorite performance metric and plot the results of performance vs. the number of cases used for testing and training.

The curves will never come together if the model is overfit (high variance).  The curves will come together but the performance will be lower than desired if the model is underfit (high bias) and the lines will come together at an acceptable performance for a well performing model that is not overfit.

Here is an example of overfitting and underfitting with root mean square error as the performance metric:

Here is a pretty good link on the process and here is another one.  Hope this helps!


Context capturing in a Structured PDF? I'm trying to extract resume (PDF) data. resumes always tend to follow a structure. so if you see some numbers in a cv; according to the context, we could tell whether its a telephone number, a birthday, or a date period. if I can classify/identify one entity then that would increase my ability to classify an entity near to it.

I'm still a newbie and appreciate if anyone could give me any thoughts on approaching this problem. what kind of machine learning models should i focus on?

I'm wondering if this is really a machine learning problem.  If you can get to text out of the PDF then you should be able to do a lot with a rules-based text analysis algorithm.


What is the difference between classification and regression? I understand classification....a discrete response or category, like animal is dog or cat.

The author says..."Regression techniques predict continuous changes such as the change in temperature, power demand, or stock market prices."

I can't wrap my head around what he means.

Thanks.

For the sake of illustration, let's imagine that you're trying to predict the amount of gas in the tank of your car.

A classification problem statement of this question would be whether you have gas in your car (yes or no).

A regression statement of this problem would predict the level of gas in your car (anywhere between completely full or completely empty) and could take any value.

The output of a classification model can be one of n options, where n is the number of classes (and/or the probability associated with each class).

The output of a regression model is a (possibly bounded) continuous value.


Preprocessing and dropout in Autoencoders? I am working with autoencoders and have few confusions, I am trying different autoencoders like :

fully_connected autoencoder
convolutional autoencoder
denoising autoencoder

I have two dataset , One is numerical dataset which have float and int values , Second is text dataset which have text and date values :

Numerical dataset looks like:

date ,        id ,             check_in , check_out , coke_per , permanent_values , temp
13/9/2017     142453390001    134.2       43.1        13         87                 21
14/9/2017     142453390005    132.2       46.1        19         32                 41
15/9/2017     142453390002    120.2       42.1        33         99                 54
16/9/2017     142453390004    100.2       41.1        17         39

           89

Any my text dataset looks like :

data              text
13/9/2017         i totally understand this conversation about farmer market and the organic products, a nice conversation ’cause prices are cheaper than traditional
14/9/2017         The conversation was really great. But I think I need much more practice. I need to improve my listening a lot. Now I’m very worried because I thought that I’d understand more. Although, I understood but I had to repeat and repeat. See you!!!

So my questions are:

Should i normalize my numerical data values before feeding to any type of autoencoder? if they are int and float values still i have to normalize?

Which activation function should i use in autoencoder? Some article and research paper says , "sigmoid" and some says "relu" ?

Should i use dropout in each layer ? like if my artichare for autoencoder looks like

encoder (1000 --> 500 -- > 256 ----> 128 ) --> decoder (128 --> 256 --> 500--> 784)

something like this?

encoder(dropout(1000,500) --> dropout( 500,256) --> dropout (256,128) )----> decoder(dropout(128,256),dropout(256,500),dropout(500,784))

For text dataset , If i am using word2vec or any embedding to convert text into vector then i would have float values for each word , should i normalize that data too ?

text ( Hello How are you  ) -- > word2vec(text) ----> ([1854.92002 , 54112.89774 ,5432.9923 ,5323.98393])

should i normalize this values or directly use in autoencoder ?

I'll go through your questions one by one:

  Should i normalize my numerical data values before feeding to any type of autoencoder? if they are int and float values still i have to normalize?

This is strongly suggested, for two reasons. First, if different variables are on different scales, weights distributions will be unequal. Larger scales will dominate smaller scales during application of gradient descent, and this would lead to many parameters to be undertrained, leading to sub-optimal results. Second, your layers have activation functions that are meant to "learn" non-linear patterns in your data. All the commonly used activation functions (Sigmoid, Tanh, all the ReLU family, you name it) tend to be non-linear only around zero. Normalizing your data helps Neural Networks to learn most from them.

  Which activation function should i use in autoencoder? Some article and research paper says , "sigmoid" and some says "relu" ?

This is more an art than a science, however all the activations form the ReLU family have been proved superior to other counterparts. I'd suggest you to go for some kind of ReLU basically always. Some are more fancier but computationally expensive, usually the rank is:   ELU > Leaky ReLU > ReLU

  Should i use dropout in each layer ? like if my artichare for autoencoder looks like

Use some amount of dropout, but not too much. Dropout is a regularization technique that helps you preventing overfitting. This technique "turns off" some neurons during training sampled randomly, in order to make all neurons specialize during training (it turns your Neural Network in an ensemble of Neural Networks). However, keep in mind that dropout also represents an information loss! For example, if you set a Dropout() layer with dropout probability of 0.5, you'd loose half of the information at that layer at each iteration! I suggest you to use it, but not at every layer, and be parsimonious with it.

  For text dataset , If i am using word2vec or any embedding to convert text into vector then i would have float values for each word , should i normalize that data too ?

No, those are learned automatically by the model, you don't have to worry about their internal values.


Can Gaussian dist. use median/MAD instead of mean/SD? Can Gaussian density distributions be modified using median and median absolute deviation (MAD) as opposed to mean and standard deviation (SD) (since the former are more robust)?

I am not 100% sure if I got your question, but I will try my best.

By looking at the gaussian probability density function and inspecting the parameters you will see, that there is actually no mean at all. There is only the expected value $\mu$, which is not necessarily the mean of your data, and $\sigma$, which depends on $\mu$.

When you have data and want to fit it to a gaussian, you will see by applying maximum likelihood, that using the mean is quite the best thing you can do to fit a gaussian.

Ideally your mean and median should be close. If not, there are two options. First your data is simply not gaussian. Second you have outliers or errors in your data. Then you can think about cleaning your data first or as a lazy and fast solution take the median.
Otherwise I see no justification to take the median.


Difference between Blocking and Clustering? What is the difference between blocking and clustering?

As far as I know clustering originates from the Machine Learning branch and refers to methods of grouping data points together in a "We and them" way: Points in the same cluster are related, points of different clusters are not.

Blocking, however, seems to come from the statistics and picked up later in data analysis. I encountered Blocking/Indexing etc. primarily in Record Linkage, where it is used to partition data into blocks before they are analyzed.

One could even have both blocking and clustering methods used in Record Linkage. However, I find, I don't understand their difference too well yet and thus would like to have a different perspective on that matter.

My interpretation thus far: The goal of blocking and clustering is different:

Blocking is used to throw potential unnecessary data out - distinguishing between important and non-important data.

Clustering however does make that distinction, all data points are important. is used for classification purposed - identifying all data.

As you mentioned their goals are different. In clustering, we try to group data such that they have the same variability. For example, clustering customers of a company into different clusters, somehow, members of each cluster have the same behavior in their buying.

On the other side, in blocking we try to reduce the variability, to record linkage, as an instance.


Which Graph Properties are Useful for Predictive Analytics? Let's assume I'm building a content recommendation engine for online content. I have web log data which I can import into a graph, containing a user ID, the page they viewed, and a timestamp for when they viewed it. Essentially, it's a history of which pages each user has viewed.

I've used Neo4J and Cypher to write a simple traversal algorithm. For each page (node) I want to build recommendations for, I find which pages are most popular amongst other users who have also visited this page. That seems to give decent results. But, I'd like to explore alternatives to see which method gives the most relevant recommendations.

In addition to my simple traversal, I'm curious if there are graph-level properties I can utilize to build another set of recommendations with this data set. I've looked at SNAP, it has a good library for algorithms like Page Rank, Clauset-Newman-Moore community detection, Girvan-Newman community detection, betweenness centrality, K core, and so on.

There are many algorithms to choose from. Which algorithms have you had success with? Which would you consider trying?

Using Community detection you can build a recommendation system. The most commonly used algorithm in this field is Blondel Algorithm which u have probably seen in SNAP. Blondel is almost the fastest Community Detection algorithm among widely accepted ones and its result is pretty acceptable (at least according to modularity score). As a side comment, you may also want to have a look at NetworkX and igraph libraries for more graph algorithms and visualizations.

Hope it helps, good luck.


Non-parametric approach to healthcare dataset? I have a Healthcare dataset. I have been told to look at non-parametric approach to solve certain questions related to the dataset. I am little bit confused about non-parametric approach.

Do they mean density plot based approach (such as looking at the histogram)?
I know this is a vague question to ask here. However, I don't have access to anybody else whom I can ask and hence I am asking for some input from others in this forum.

Any response/thought would be appreciated.

Thanks and regards.

They are not specifically referring to a plot based approach.  They are referring to a class of methods that must be employed when the data is not normal enough or not well-powered enough to use regular statistics.

Parametric and nonparametric are two broad classifications of statistical procedures with loose definitions separating them:

Parametric tests usually assume that the data are approximately normally distributed.
Nonparametric tests do not rely on a normally distributed data assumption.
Using parametric statistics on non-normal data could lead to incorrect results.
If you are not sure that your data is normal enough or that your sample size is big enough (n < 30), use nonparametric procedures rather than parametric procedures.
Nonparametric procedures generally have less power for the same sample
size than the corresponding parametric procedure if the data truly are normal.

Take a look at some examples of parametric and analogous nonparametric tests from Tanya Hoskin's Demystifying Summary:

Here are some summary references:

Another general table with some different information
Nonparametric Statistics
All of Nonparametric Statistics, by Larry Wasserman
R tutorial
Nonparametric Econometrics with Python


Trying to figure out which the training set is Can someone help me on this one? As can be seen in the screenshot, it says the loss is 1/2. Where's that 1/2 coming from? How can I replace the values in the h(s) function?

Source PDF

To clarify the proposed classifier (because I think it depends on some nonstandard notation): classify any point that is in the (finite) training set according to its true label.  (Clearly this gives you an error rate of 0 on the training set.)  For any point not in the training set, just predict label 0.
Now, consider a random new point in the gray square: it's the same as a point from the training set with probability 0, so assume it's not.  Then our classifier predicts label 0.  But the point has probability 1/2 (area of blue square over area of gray square) of being label 1.  So the error rate of our classifier, on test data, is 1/2.