What are the individual models within a machine learning ensemble called? I am aware that an ensemble machine learning model is a stack of two or more machine learning models. Is there a word to refer to those individual models that go into the ensemble model? (i.e. a specific term/jargon?)

I wish to refer to them and am not aware of any specific terminology to describe them. I've considered:

"sub-models"
"component models"
even "pancake models" (since people
generally understand what's being referring to when they hear 'one of the
pancakes in the model')

I am not aware of a specific definition. Wikipedia does not mention such a term either. I would prefer "components", "individual/constituent models", or something like that.

If you definitely want to find a "correct" term, one way to discover it (if it exists) is to look into an ensemble learning early paper.
To arrive at one, a good way is to search for a survey paper (e.g. here) and go on a journey to the past to find that original study.

From a quick skim of the above, you might want to look at Exploratory Data Analysis (Tukey 77), if you manage to find the pdf.


Can Precision-Recall be improved for imbalanced sample? I tried out a few models on a highly imbalanced sample (~2:100) where I can get decent AUC from ROC (test sample). But when I plot precision-recall (test sample), it looks horrible. Kind of like the worse PR curve in box (d).

This article contains the picture from below and describes that ROC is better suited since it is invariant to class distribution.

My question is if there's anything than can be done to improve precision-recall?

I am not sure if you are asking how to improve the power of the models or how to better represent the effectiveness of the models on the minority class.  These are information retrieval measures that are sometimes adequate in other circumstances.  If you are working on an information retrieval process, this is probably fine.  If it is more of a statistical analysis you are working on, perhaps sensitivity (which is the same as recall) and specificity would be better metrics to describe your results.  Maybe your model is useful in that it can rule out the negative. These metrics can also be evaluated relative to the prior probability of the minor class in the data.

Sensitivity and specificity have an additional benefit if you are trying to communicate your results with an audience that reviews a lot of medical test results as these measures are used most frequently in that field.

HTH


Should I use pandas get_dummies and create additional columns or use my own encoding code that keeps 1 column? I am running the Kaggle Video Games sales dataset through an XGboost algo.

I want to encode the categorical column of "Game Rating" from E, M, etc. to 0-5

when I use:
    data = pd.get_dummies(data=data, columns=['Game_Rating])

pandas adds a column to my dataframe for each unique observation in Game_Rating

Rating_E  Rating_E10+  Rating_EC  Rating_K-A  Rating_M  Rating_RP  Rating_T
1            0          0           0         0          0         0
0            0          0           0         0          0         0
1            0          0           0         0          0         0
1            0          0           0         0          0         0
0            0          0           0         0          0         0

My question is two-fold:
Does Pandas have the ability to replace categorical values of a dataframe inline, preserving the original column structure

if not:

I have a (slower) method that does the encoding inline and preserves the column structure of the dataframe.  I am trying to keep the dataframe as similiar to the original dataset as possible for future processing. What are the risks I may not be seeing of using my own (inline) encoding method?

credit for this answer goes to @timleathart in the comments above:

You can get the same effect very efficiently in pandas by using df['your column name'].astype('category').cat.codes. Be careful when doing this, because it implies an ordering on the columns. Your learning algorithm might overfit to spurious relationships that appear in the training data


Why the accuracy of my bagging model heavily affected by random state? The accuracy of my bagging decision tree model reach up to 97% when I set the random seed=5 but the accuracy reduce to only 92%  when I set random seed=0. Can someone explain why the huge gap and should I just use the accuracy with highest value in my research paper or takes the average with random seed=None?

Can someone explain why the huge gap

It simply means that there's a quite high variance depending which random set of instances is picked. How many times do you re-sample the instances in the bagging process? Probably increasing the number of runs will decrease the variance. As mentioned in a comment, the most common reason for variance in performance is a sample which is too small (and/or a number of features/classes which is too high). It's likely to cause your models to overfit.

and should I just use the accuracy with highest value in my research paper or takes the average with random seed=None?

Never ever use the highest performance across random runs, this is cherry-picking and it doesn't reflect the true performance. The possibility to select a random seed is for reproducibility purposes, and selecting the one which gives the best results is the opposite of the principle of reproducibility.
Since you're using bagging, you should decrease the variance (that's the whole point) by increasing the number of runs. If you can't do that for any reason, then don't use bagging: simply repeat the regular process splitting-training-evaluating $N$ times (with a fixed proportion training/testing data) or use cross-validation, and report the average performance (preferably report the variance as well, e.g. standard deviation).


Perceptron learning rate Today I've seen many Perceptron implementations with learning rates. According to Wikipedia:

  there is no need for a learning rate in the perceptron algorithm.
  This is because multiplying the update by any constant simply rescales
  the weights but never changes the sign of the prediction

Is there some benefit to implementing a learning rate with Perceptron? If there is not, why do so many implementations have it?

I agree with Dawny33, choosing learning rate only scales w.

While training of Perceptron we are trying to determine minima and choosing of learning rate helps us determine how fast we can reach that minima. If we choose larger value of learning rate then we might overshoot that minima and smaller values of learning rate might take long time for convergence.

It is okay in case of Perceptron to neglect learning rate because Perceptron algorithm guarantees to find a solution (if one exists) in an upperbound number of steps, in other implementations it is not the case so learning rate becomes a necessity in them.

It might be useful in Perceptron algorithm to have learning rate but it's not a necessity.


Using ARIMA parameters when transforming time series to Supervised Learning When forecasting time series one can change the problem from a classical time series (ARIMA type of models) to supervised learning (by adding lag features).

When the time series is long and you convert it to supervised learning you end up having a lot of features.

In case that we had done previous modeling with ARIMA and found the AR, I, MA, parameters.

Could we use this knowledge/parameters to select a subset of lag features?

Yes. It could be use. if you find the lag pattern by using acf, pacf , you can generate the rolling mean or mid value time series as part of feature engineering. And use the combined feature to do supervised learn.

However, one of key important thing need note: time series based cross validate in the begining to avoid data leaking, other wise you metric will be unreasonable good.


Get the prediction probability using prediction function I'm new to SVM models. I took custom SVM classifier from the github. In there standard predict function was overwritten by custom predict function.
def predict(self, instances):
   predictions = []
   for instance in instances:
       predictions.append((int)(np.sign(self.weight_vector.dot(instance.T)[0] + self.bias)))#class is determined based on the sign -1,1

I want to get prediction probability of each class. [-1 probability, 1 probability]
I know from standard SVM predict_proba function I can get the probability. Since this is a customized SVM  how do I get that? Can somebody help me.
Thank you

I have followed scikit-learn https://github.com/scikit-learn/scikit-learn/blob/fd237278e/sklearn/linear_model/_base.py#L293 link and was able to get predicted probabilities.


What is "spatial feature encoding"? Can anyone give a concrete example? This book "Deep Learning and Convolutional Neural Networks for Medical Image Computing" mentioned a term spatial feature encoding

  On the other hand, CNN models have been proved to have much higher
  modeling capacity, compared to the previous image recognition
  mainstream pipelines, e.g., HAAR, SIFT, HOG image features followed by
  spatial feature encoding, then random forest or support vector
  classifiers. Given millions of parameters to fit during model training
  (much more than previous pipelines), CNN representation empowers and
  enables computerized image recognition models, with a good possibility
  to be able to handle more challenging imaging problems. The primary
  risk is overfitting since model capacity is generally high in deep
  learning but often very limited datasets are available (that are with
  good quality of labels to facilitate supervised training). The core
  topics of this book are represented by examples on how to address this
  task-critical overfitting issue with deep learning model selection,
  dataset resampling and balancing, and the proper quantitative
  evaluation protocols or setups.

I googled a lot and didn't find explanation or definition of "spatial feature encoding".

What is "spatial feature encoding"? Can anyone give a concrete example?

Chatfield et al. provides a good survey of feature encoding methods.

From the introduction to that paper (emphasis my own):

  The typical object recognition pipeline is composed of the following three steps: (i) extraction of local image features (e.g., SIFT descriptors), (ii) encoding of the local features in an image descriptor (e.g., a histogram of the quantized local features), and (iii) classification of the image descriptor (e.g., by a support vector machine).

I believe that "spatial feature encoding" refers to step (ii) in the typical pipeline. In this respect, I believe that the simplest possible example (and a standard one) would be the histogram encoding. If "spatial feature encoding" is meant to imply the preservation of some geometric information, than spatial binning is another technique to look at.


Find best (most efficient) combination of Ingredients Question:
Suggest me an approach to that kind of problem. Is it pure optimization problem?
What might Machine Learning techniques can be helpful here?
Case Study
My reduced case study is as follows:

I have 3 types of ingredients (A, B, C). In each experiment, some
combination from the ingredients is chosen and applied on something.
Result can be 1:poor, 2:middle, 3:high.

Example:
| Input      | Output      |
|:----------:|:-----------:|
| A          |       2     |
| AB         |       1     |
| ABC        |       2     |
| BC         |       3     |
| C          |       2     |
|...         |      ...    |

I need to find out what is the winning combination of ingredients according to the results.
Thanks

This is an optimization problem and optimization is all what Machine Learning does in classic sense (mostly not unsupervised approaches).
Your problem can be designed to be solved by Genetic Algorithms. For this you need to make a sequences of ingrediants which is called chromosome (genetic representation of solution) and an objective function which is to be minimized/maximized according to the chromosomes (it's usually called fitness function and is to be maximized then).
You have genetic operators to be applied randomly on your chromosomes:

Reproduction: 2 parent chromosomes result in two children.
Mutation: One single bit (one of ingrediants) is randomly chosen to
be switched with another one (like in a genetically normal family, a
baby is born with a genetic phenomenon).
Natural Selection: Uncle Darwin believed that nature "keeps the best
and  discards the rest". But who is the best? Well, you already have a fitness function!

Continue till you reach the optimum and select the best chromosome.
Of course it was the intuitive explanation of this beautiful idea. See more here and if you needed implementation details please drop a comment here.
Good Luck.
Update: More Details
I have an optimization problem which is too complex or large to be solved analytically so I need a heuristic method to approximate the optimal solution. For instance each subset of $28$ ingredients outputs a result $y$ where $y \in [0,100]$ (it's much bigger than your sample problem just to use optimization properly).

Initialization: Randomly produce $n$ combinations of material as initial population. Choose probabilities $P_{crossover}=0.25$, $P_{mutation}=0.01$ and $n_{selection}=250$ (values are just examples)
Evaluation of Fitness: Evaluate the output for each of them

| input         | output
| ------------- |:-------------:|
| ABCDEFGH      | 23.5          |
| BDCGTRFJ      | 45.6          |
| ZHTRFGOIUPOIZ | 18.3          |
| ...           | ...           |

Choose top $n_{selection}$ and ignore the rest. If you found an optimal solution (in your case you said output=3 is desired) stop, if not:
According to $P_{crossover}$ choose 25% of survived samples and reproduce children by combining them through a breaking point (image from here).

For instance if $ABCDEFGH$ and $BCDEFGHJ$ are survived, then choose a breaking point and exchange their parts (the image is 2-point crossover. You may apply 1-point crossover. It's all about the variations of the same concept). Technically the crossover point should be also determined probabilistically but I just skipped it. Break them from wherever you want! Here we choose $3^{rd}$ gene for this so $ABCDEFGH$ and $BCDEFGHJ$ will become $ABDEFGHJ$ and $BCCDEFGH$. Two children out of two parents. Apply it on all chosen survived samples.

According to $P_{mutation}$ choose 1% of survived samples and randomly(again, technically with a predefined probability) change one material in it (technically the samples are usually binary so you flip between $0$ and $1$. Here as you have nominal variables you need to change the variable). Let's say in survived sample $FVBGTZHU$ I randomly choose $V$ and exchange it with $P$ so I get $FPBGTZHU$.
Now the new generation is born (keep it as the same size of the original population for better structure). Go back to 2 and continue till you find your answer.

For different variations of GA (Elitism, Island Model, etc.) see the literature for Evolutionary Algorithms e.g. this one.
Hope it helps!


Transform large pd.Series into a DataFrame of n columns I have a pd.Series with, for instance, n lines. I would like to transform this series in a pd.DataFrame as follows:
Ex:
Input: pd.Series([10,11,12,13,14,15]) and a variable chunk_size = 2 that will be the number of columns.
Target:
    0 | 1
    _   _
    10  11
    12  13
    14  15

The target DataFrame will have a shape of (n / chunk_size) rows by chunk_size columns.
Thanks in advance.

Here is a quick solution that does not do it in-place but takes up extra space:
def transform_series(x, chunk_size):
    df = pd.DataFrame()
    for i in range(chunk_size):
        df[f'column_{i+1}'] = x[i::chunk_size].reset_index(drop=True)
    return df

input_series = pd.Series([10,11,12,13,14,15])
transformed_df = transform_series(input_series, chunk_size=2)

Output:
print(transformed_df)

   column_1  column_2
0        10        11
1        12        13
2        14        15


Extract data from PDFs I would like to do an experiment. I would like to get the following data: advisor education rank (Ing., Bc. etc), number of pages in the thesis, number of citations etc for each student about thesis works from my school, but unfortunately all of this data is in 3 different PDFs.

Is there any way I could (ideally not too slowly, since I would like to use a large data set) gather this information from PDF? Or perhaps look for this data elsewhere?

On the website the metadata is only basic, such as name of the advisor, student and the title.

I've edited my question to clarify that not all of them are in the same place. I guess I would have to use some sort of web crawler?

EDIT 2: I have looked into PyPDF as recommended and the problem that I see is that the data from the PDF that I want is for example the grade which is just some number somewhere in the PDF, or the professor title which will be different for every file.

I suggest to first parse the pdfs to raw text and then retrieve the information in the next step. I assume you use Python.

As commented, you can use PyPDF2 to obtain the raw text and the number of pages of the documents. Alternatively you can use cloud services such as AWS Textract to parse multiple pdfs to raw text.

You may then apply regular expressions, for example re.findall() with specific matching patterns to the parsed pdf documents to match the terms you seek to find.


How to obtain with a recurrent neural network the Xor function using keras? I'm trying to implement a model  of recurrent neural network to solve the XOR problem, but I am not still able to do that. Any hints?

I think following this link helps you, I have gone trough those tutorials previously.
Also take a look at this tutorial .


Stacked barchart, bottom parameter triggers Error: Shape mismatch: objects cannot be broadcast to a single shape I am working in python3 and I want to obtain a stacked barchart plot, showing three different variables on 5 different columns.
My code works fine if I do not add the 'bottom parameter' in plt.bar (but I need to, in order for the stacks to appear in the correct order):

import numpy as np
import matplotlib as plt

columns=['a','b','c','d','e']
pos = np.arange(5)
var_one=[40348,53544,144895,34778,14322,53546,33623,76290,53546]
var_two=[15790,20409,87224,22085,6940,27099,17575,41862,27099]
var_three=[692,3254,6645,1237,469,872,569,3172,872]

plt.bar(pos,var_one,color='green',edgecolor='green')
plt.bar(pos,var_two,color='purple',edgecolor='purple')
plt.bar(pos,var_three,color='yellow',edgecolor='yellow')
plt.xticks(pos, columns)
plt.show()

However, once I add the bottom parameter in bar.plot (as shown below):

import numpy as np
import matplotlib as plt

columns=['a','b','c','d','e']
pos = np.arange(5)
var_one=[40348,53544,144895,34778,14322,53546,33623,76290,53546]
var_two=[15790,20409,87224,22085,6940,27099,17575,41862,27099]
var_three=[692,3254,6645,1237,469,872,569,3172,872]

plt.bar(pos,var_one,color='green',edgecolor='green')
plt.bar(pos,var_two,color='purple',edgecolor='purple',bottom=var_one)
plt.bar(pos,var_three,color='yellow',edgecolor='yellow',bottom=var_one+var_two)
plt.xticks(pos, columns)
plt.show()

the code triggers the error

ValueError: shape mismatch: objects cannot be broadcast to a single shape

How could I fix it?

if you change your code to the following:

import numpy as np
import matplotlib.pyplot as plt

columns = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'j']
pos = np.arange(9)
var_one = np.array([40348, 53544, 144895, 34778, 14322, 53546, 33623, 76290, 53546])
var_two = np.array([15790, 20409, 87224, 22085, 6940, 27099, 17575, 41862, 27099])
var_three = np.array([692, 3254, 6645, 1237, 469, 872, 569, 3172, 872])

plt.bar(pos, np.add(np.add(var_three, var_two), var_one), color='yellow', edgecolor='yellow')
plt.bar(pos, np.add(var_two, var_one), color='purple', edgecolor='purple')
plt.bar(pos, var_one, color='green', edgecolor='green')
plt.xticks(pos, columns)
plt.show()

The result will be like this:


logistic regression algorithm fails to work I'm trying to code my own logistic regression algorithm using Andrew NG's machine learning using Octave. lectures. So what I did was make a csv file, the first row being some parameter and the second one being the result:

121,1
124,0
97,0
104,0
110,0
...

Overall there are only 24 examples, but I've chosen points such that some pattern can be followed.

Here is my code:

data = load('data.dat');
x = data(:, 1);

y = data(:, 2)
m = length(y);

#plot(x, y, 'rx', 'MarkerSize', 10);
#xlabel('IQ');
#ylabel('Pass/Fail');
#title('Logistic Regression');

x = [ones(size(x, 1), 1) x];
alpha = 0.00001;
i = 15000;

g = inline("1 ./ (1 + exp(-z))")

theta = zeros(size(x(1, :)))';
j = zeros(i, 1);

for num = 1:i
  z = x * theta;
  h = g(z);
  j = (1./m) * ( -y' * log( h ) - ( 1 - y' ) * log ( 1 - h))
  grad = 1./m * x' * (h - y);
  theta = theta - alpha * grad;
end

However the output of the sigmoid function shows every value below 0.5... surely this has to be wrong. I've also tried with different learning rates and iterations, but to no avail. What is wrong with the code, or data?

Help would be appreciated.

Consider that you are doing vector operation, change your cost function to the following:

(1 / m) * sum(((-y) .* (log(h)) - ((1 - y) .* log((1-h)))));

and your gradient to the following:

grad = (1./m) * (x' * (h - y))

Although the latter is just for precedence reassuring.

Based on the discussion in the chat, although the code calculates the cost in a wrong way, the reason the cost does not decrease is that the data is not linearly separable. Logistic regression is a simple algorithm which classifies successfully linearly separable data. Take a look at here.


What methods can be used to detect duplicacy in image dataset? I want to remove duplicate images from a dataset of 50Million images. What is the best method to detect all the duplicates?

Do you think one shot learning is good for this?

I think the dhash technique might help. It essentially creates a signature for each image, then you could isolate the duplicated images. 50M could take a while, so perhaps you can try that with a smaller subset and see how well it works.


Image Matching to solve captcha I am building a bot with python and I need some system to solve captchas like these:

I think I need a deep learning algorithm, but coding one is a pain in the ass. Is there any easy solution to this?

I can code the part that screenshots the images and clicks on the correct answer, image 4 or 5 in the example. The images get rotated and change every time.

Thank you!

Here's one simple yet effective solution without using the deep learning algorithm.

Divide the problem into 2 parts:

Segmentation: You'll need to do some image processing like edge detection and segment all the objects in the given captcha and save the region of interest (OpenCV can help). Assign labels to the objects.
Similarity Measurement: Use some scale and rotational invariant feature descriptor algorithm to extract features from the objects, and a matcher to compare the objects for a similarity measure (check SIFT/SURF features).

Based on the similarity scores, make a decision.


number of parameters for convolution layers In this highly cited paper, authors give the following discussion on the number of weight parameters. I am not very clear why it has $49C^2$ parameters. I think it should be $49C$ since each of $C$ input channels shares the same filter, which has $49$ parameters.

Actually it's $49C*C$, the first $C$ is the number of input channels, and the second $C$ is the number of filters.

Quote from CS231n:

  To summarize, the Conv Layer:

  Accepts a volume of size $W_1 \times H_1 \times D_1$
  Requires four hyperparameters:

  Number of filters $K$,
  their spatial extent $F$,
  the stride $S$,
  the amount of zero padding $P$.

  Produces a volume of size $W_2 \times H_2 \times D_2$ where:

  $W_2 = (W_1 - F + 2P)/S + 1$
  $H_2 = (H_1 - F + 2P)/S + 1$ (i.e. width and height are computed equally by symmetry)
  $D_2 = K$

  With parameter sharing, it introduces $F \cdot F \cdot D_1$ weights per filter, for a total of $(F \cdot F \cdot D_1) \cdot K$ weights and $K$ biases.
  In the output volume, the $d$-th depth slice (of size $W_2 \times H_2$) is the result of performing a valid convolution of the $d$-th filter over the input volume with a stride of $S$, and then offset by $d$-th bias.

  A common setting of the hyperparameters is $F = 3, S = 1, P = 1$. However, there are common conventions and rules of thumb that motivate these hyperparameters. See the ConvNet architectures section below.


Scikit Learn Logistic Regression Memory Leak I'm curious if anyone else has run into this. I have a data set with about 350k samples, each with 4k sparse features. The sparse fill rate is about 0.5%. The data is stored in a scipy.sparse.csr.csr_matrix object, with dtype='numpy.float64'.

I'm using this as an input to sklearn's Logistic Regression classifier. The documentation indicates that sparse CSR matrices are acceptable inputs to this classifier. However, when I train the classifier, I get extremely bad memory performance; the memory usage of my process explodes from ~150 MB to fill all the available memory and then everything grinds to a halt as memory swapping to disk takes over.

Does anyone know why this classifier might expand the sparse matrix to a dense matrix? I'm using the default parameters for the classifier at the moment, within an updated anacoda distribution. Thanks!

scipy.__version__ = '0.14.0'
sklearn.__version__ = '0.15.2'

Ok, this ended up being an RTFM situation, although in this case it was RTF error message.

While running this, I kept getting the following error:

DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().

I assumed that, since this had to do with the target vector, and since it was a warning only, that it would just silently change my target vector to 1-D.

However, when I explicitly converted my target vector to 1-D, my memory problems went away. Apparently having the target vector in an incorrect form caused it to convert my input vectors into dense vectors from sparse vectors.

Lesson learned: follow the recommendations when sklearn 'suggests' you do something.


where to start in natural language processing for a language My native language is a regional language and few people speak it. I have some assignements in a machine learning course and i was thinking about doing some natural languge processing on my native language but i don't know where to start since there is almost no research about this language ( no corpus , no research papers , ... ) and i'm new to machine learning.

I want to start doing everything from bottom and i want to do things right , can you please guide me to steps i should follow?

I also want to build my own corpus ,since it's a very tiring work , is there a way to build a single corpus that can be used on several NLP applications ( at least for translation and speach recognition)  ?

@Amar, you can start with NLTK, Spacy, Flair and Polyglot to start with your NLP learning. They are easier to start and understand. You can create your own model for your language and contribute if you have the corpus available with you. The nouns, verbs are easier to capture in English when compared to Chinese, Korean and Arabic. Every language has it own beautiful representation of context. You would a good library that can read the language that you are developing. You can also try IBM Watson and AWS SageMaker on trial basis with check for Language detection.


Standard Deviation for Z-scores I have a set of data that I'm trying to generate a z-score with. I know I need standard deviation as part of my calculations. I am using the formula of:
$\sigma = \sqrt{p * n * (1-p)}$

My data is binary - the value can either go up or down. However, through historical data, I know that it is more likely to go up than down.

So, which value for $ p $ should I use above? Do you use the binary value of $0.50$ or should I use the historical value of my data going up?

In your example, you should calculate the probability that the value goes up, and use that as your $p$.

That said, I think the calculation should be $\sigma = \sqrt{p(1-p)}$ otherwise, your standard deviation is inflated by a factor tied to $n$.  By using probabilities, you are essentially taking $n$ into account because:
$$p = {Number\ of\ "Up"\ instances \over n}$$
Where $n$ is the total number of observations.

Given the nature of the data, that is, it is binary, we would expect the $\sigma$ to be between 0 and 1 and not particularly sensitive to the number of observations.

That said, if we look at a few calculations, the value of $n$ in your formula significantly impacts the $\sigma$.

Assume for simplicity that $p$ is 0.5, that is an equal chance of going up or down.  If you have three different values of $n$ you get three vastly different values for $\sigma$.

Example 1: $n$ = 1 (Or not included)
$$\sigma = \sqrt{0.5 * (1-0.5)},\ \therefore \sigma = 0.5$$

Example 2: $n$ = 100
$$\sigma = \sqrt{0.5 * 100 * (1-0.5)},\ \therefore \sigma = 5$$

Example 3: $n$ = 1000
$$\sigma = \sqrt{0.5 * 1000 * (1-0.5)},\ \therefore \sigma = 15.8$$

Of the three, only example 1 really makes any sense intuitively. The mean of all instances would be 0.5 and you would expect the observed values to be 0.5 away from the mean.


How is loss computed for multiclass CNN with an output layer larger than the number of classes? I have built a CNN in pytorch for classifying the Fashion-MNIST dataset (10 classes). The images are 28x28.

I have constructed the final layer in my model as an output of 50. (i.e.         $nn.Linear(100, 50)$). Also I am using cross entropy loss.

I am confused about how loss is calculated for these data sizes. From what I had known about backpropagation and loss function, the output of the neural net is compared with the expected result.

For example, using mean square error, the loss function is $(output - expected)^2$. So if I had a binary classifier, say the class labels are $({0,1})$ then the output of the neural network would need to be one dimension to compute the loss.

Now if I had three classes, how would you calculate loss? How many outputs would you need? Since the expected class label is still just a single digit, I don't see how loss can be calculated if the output of the neural network is more than one dimension.

For example, if the output is $[ x1, x2, x3]$ and the expected class label is $y$, I don't see how loss could be calculated since the dimensions don't agree.

So how is loss computed against a class label when the output of a neural network isn't a single digit?

There are many great tutorials covering how exactly cross entropy loss works. The key thing is : the classifier outputs probability for each class and not a single label.
Taking your example, if there are 3 classes then the network will output something like [0.5, 0.25, 0.25] giving probabilities for each class. You can treat expected output as [1, 0, 0] if the output is, say, label 0. Now the dimensions agree and you can calculate the loss.


What does from_logits=True do in SparseCategoricalcrossEntropy loss function? In the documentation it has been mentioned that y_pred needs to be in the range of [-inf to inf] when from_logits=True. I truly didn't understand what this means, since the probabilities need to be in the range of 0 to 1! Can someone please explain in simple words the effect of using from_logits=True?

model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])

The from_logits=True attribute inform the loss function that the output values generated by the model are not normalized, a.k.a. logits. In other words, the softmax function has not been applied on them to produce a probability distribution. Therefore, the output layer in this case does not have a softmax activation function:
out = tf.keras.layers.Dense(n_units)  # <-- linear activation function

The softmax function would be automatically applied on the output values by the loss function. Therefore, this does not make a difference with the scenario when you use from_logits=False (default) and a softmax activation function on last layer; however, in some cases, this might help with numerical stability during training of the model. You may also find this and this answers relevant and useful about the numerical stability when from_logits=True.


Tools and protocol for reproducible data science using Python I am working on a data science project using Python.
The project has several stages.
Each stage comprises of taking a data set, using Python scripts, auxiliary data, configuration and parameters, and creating another data set.
I store the code in git, so that part is covered.
I would like to hear about:

Tools for data version control.
Tools enabling to reproduce stages and experiments.
Protocol and suggested directory structure for such a project.
Automated build/run tools.

The topic of reproducible research (RR) is very popular today and, consequently, is huge, but I hope that my answer will be comprehensive enough as an answer and will provide enough information for further research, should you decide to do so.

While Python-specific tools for RR certainly exist out there, I think it makes more sense to focus on more universal tools (you never know for sure what programming languages and computing environments you will be working with in the future). Having said that, let's take a look what tools are available per your list.

1) Tools for data version control. Unless you plan to work with (very) big data, I guess, it would make sense to use the same git, which you use for source code version control. The infrastructure is already there. Even if your files are binary and big, this advice might be helpful: https://stackoverflow.com/questions/540535/managing-large-binary-files-with-git.

2) Tools for managing RR workflows and experiments. Here's a list of most popular tools in this category, to the best of my knowledge (in the descending order of popularity):

Taverna Workflow Management System (http://www.taverna.org.uk) - very solid, if a little too complex, set of tools. The major tool is a Java-based desktop software. However, it is compatible with online workflow repository portal myExperiment (http://www.myexperiment.org), where user can store and share their RR workflows. Web-based RR portal, fully compatible with Taverna is called Taverna Online, but it is being developed and maintained by totally different organization in Russia (referred there to as OnlineHPC: http://onlinehpc.com).
The Kepler Project (https://kepler-project.org)
VisTrails (http://vistrails.org)
Madagascar (http://www.reproducibility.org)

EXAMPLE. Here's an interesting article on scientific workflows with an example of the real workflow design and data analysis, based on using Kepler and myExperiment projects: http://f1000research.com/articles/3-110/v1.

There are many RR tools that implement literate programming paradigm, exemplified by LaTeX software family. Tools that help in report generation and presentation is also a large category, where Sweave and knitr are probably the most well-known ones. Sweave is a tool, focused on R, but it can be integrated with Python-based projects, albeit with some additional effort (https://stackoverflow.com/questions/2161152/sweave-for-python). I think that knitr might be a better option, as it's modern, has extensive support by popular tools (such as RStudio) and is language-neutral (http://yihui.name/knitr/demo/engines).

3) Protocol and suggested directory structure. If I understood correctly what you implied by using term protocol (workflow), generally I think that standard RR data analysis workflow consists of the following sequential phases: data collection => data preparation (cleaning, transformation, merging, sampling) => data analysis => presentation of results (generating reports and/or presentations). Nevertheless, every workflow is project-specific and, thus, some specific tasks might require adding additional steps.

For sample directory structure, you may take a look at documentation for R package ProjectTemplate (http://projecttemplate.net), as an attempt to automate data analysis workflows and projects:

4) Automated build/run tools. Since my answer is focused on universal (language-neutral) RR tools, the most popular tools is make. Read the following article for some reasons to use make as the preferred RR workflow automation tool: http://bost.ocks.org/mike/make. Certainly, there are other similar tools, which either improve some aspects of make, or add some additional features. For example: ant (officially, Apache Ant: http://ant.apache.org), Maven ("next generation ant": http://maven.apache.org), rake (https://github.com/ruby/rake), Makepp (http://makepp.sourceforge.net). For a comprehensive list of such tools, see Wikipedia: http://en.wikipedia.org/wiki/List_of_build_automation_software.


Prediction for Current month based on last month's Labels I have monthly data of loan installment repayment. The data contains basic features like salary, age, gender, credit score, etc. Along with the above features, I have the data for the last 6 installment failure/success. Now based on this, I want to predict which customers are going to default next month.
The problem I would like to highlight, to apply any machine learning algorithm, I need labels for the same month, i.e., for Feb 20 predictions, I need some rows with labels success/failure in Feb 20 (so that I can train model on this) and then can predict on the remainder of data from Feb 20.
But here I don't have any labels for Feb 20 (I have data for past failures instead).

This is a common problem. In a perfect world, training data is perfectly representative of the data that will need to be predicted. However, we don't live in a perfect world and we need to work with approximations. You don't have data about Feb-20 but you have data about Jan-20, Dec-19, etc. You can still use this data! Nothing is stopping you.


the feasibility of image processing techniques for physics based images When building deep learning models for image analytics-related applications, we sometimes apply various types of operations to enhance the image, such as an image denoising operation.

In my study, we have images generated by physical simulations. In other words, the physical simulations generate matrices of dimensions such as 256*256, which can be visualized as an image as well.

I am trying to apply a deep learning model to perform some analysis over these physics-based images. In the pre-processing steps, I can always apply those image analytics related techniques to pre-process my images, but I am not sure whether it makes sense. For example, denoising or some other contrast enhancement operations can be used to improve the quality of images, like photos. But would it make sense to use them process the images generated by physics simulation?

I think your intuition is correct, that it doesn't make sense to "clean" the matrices, if they are generated perfectly.  I agree with @SyedAliHamza that noise should be eliminated as much as possible in the simulation.

On the other hand, depending on what analysis you are trying to do, you may want your model to generalize better. In that case, it would be good to apply some data augmentation, meaning adding some noise or other types of augmentation to your matrices.


Do records with the same key in two RDDs repartitioned by key reside in the same node in spark? I have two RDDs named "data" and "model", they are repartitioned by key described as below :

Does the tuple records with the same key reside in the same node in my cluster ?

Should it save IO cost in shuffle operation, such as "data.cogroup(model)" , if it comes true ?

The tuple of one partition is always on the same node because a partition itself is impartible. So if you do a groupBy or write your own partitioner which partitions by key, all records with the same key/partition number will be shuffled to the same node.

Otherwise, transformations like mapPartition which pass an iterator to a user defined function wouldn't work.


TF tells me to use Keras optimizer, tells me the opposite when I change it? I have this piece of code:

model = Sequential()
model.add(Dense(32, input_dim=9, activation='relu'))
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(1, activation='relu'))

learning_rate = 0.001
(1) optimizer = tf.train.AdamOptimizer(learning_rate)
(2) optimizer = tf.keras.optimizers.Adam(lr=learning_rate, decay=1e-6)

model.compile(loss='mean_squared_error', optimizer=optimizer, metrics=['mse'])

For the optimizer, at first I had line (1). It worked, but I got the following warning when making ModelCheckpoints:

  WARNING:tensorflow:TensorFlow optimizers do not make it possible to access optimizer attributes or optimizer state after instantiation. As a result, we cannot save the optimizer as part of the model save file.You will have to compile your model again after loading it. Prefer using a Keras optimizer instead (see keras.io/optimizers).

Apparently, you have to use the Keras optimizer if you want to use this. Ok, no problem, so I commented (1) out, and inserted (2).

Now it doesn't even want to compile anymore, and gives me this error instead:

  ValueError: optimizer must be an instance of tf.train.Optimizer, not a class 'tensorflow.python.keras.optimizers.Adam'

This sounds a bit contradictory to me? What am I doing wrong?

The error is telling you to use the actual keras optimizer, not the tensorflow.keras class. Import Adam directly from keras library and it runs ok:

from keras.models import Sequential
from keras.layers import Dense, Dropout
from keras.optimizers import Adam

model = Sequential()
model.add(Dense(32, input_dim=9, activation='relu'))
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(1, activation='relu'))

learning_rate = 0.001
optimizer = Adam(lr=learning_rate, decay=1e-6)

model.compile(loss='mean_squared_error', optimizer=optimizer, metrics=['mse'])


Compute Baseline/Representative of Time-Series Data I have time-series data of 10-days for the same time interval as shown in below figure. Here it shows one-hour power consumption for 10 days. Data is sampled at 10 minutes rate.

I need to show this 10-day usage with a single baseline/representative curve. I can calculate baseline curve simply by taking mean/median of these 10 days data, but before that I need to answer following questions:

The baseline should not represent the outlier (abnormal) usage days. Here in figure, we see that day 1 and 2 are following an unusual pattern from the rest of days. I think the usage of these days should not be used in baseline calculation. How should I exclude these days from baseline calculation automatically?
How should I find the most similar usages out of these 10-day usages for my baseline calculation? I think the most/maximum similar usage days represent the days used for baseline calculation.

Ad 1. Assuming the measurements at any given time are normally distributed (they shape approximately a bell curve), you could use simple standard deviation to detect outliers. Specifically, for any given time, you can calculate the mean and standard error. Then you calculate the mean sans outliers by taking into account only the measurements that fall at most some pre-set distance from the mean (e.g. given normal distribution, 68% of measurements fall within one standard deviation from the mean).

Pseudo-code example:

# Measurements at time t0 for all 10 days
t0 = np.array([0.1, 0.1, 1.4, .9, 1.25, 1.25, 1.5, 0.1, 0.3, 1.75])

# Get mean and standard error
mean0, std0 = t0.mean(), t0.std()

# Inliers are within one sigma from the mean
inliers = np.logical_and(mean0 - std0 < t0,
                         mean0 + std0 > t0)
# ==> [0, 0, 1, 1, 1, 1, 0, 0, 1, 0]

# And the baseline mean at time t0 is
baseline0 = t0[inliers].mean()
# ==> 1.02

Ad 2. You can find the most similar days to the baseline by using any appropriate distance measure (i.e. for time series: Euclidean or dynamic time warping). The result, then, consists of those days where distance is the least.


How understand scatter like this way? I am trying to learn some EDA skills and I get a scatter like this way:

I don't see that there are some model relationships or group-related patterns between these two data. I mean what I thought was right? I am not pretty sure about it.

If not mind, could anyone help me and give me some advice on multivariate plotting.

Thanks in advance.

Simply put, your data shows that for a given value of $x$, you can have a wide range of $y$ values and that the ranges depend on the value of $x$.

This usually happens when you have a partial view of the global picture, meaning that there are other variables in your data that could help explain the relationship.

To leverage your plots, you can consider changing the color of the markers depending on an additional variable. You could also change the size of the marker depending on another one. There are a couple of options there, depending on what you look for.

EDA is usually not the final step per se, you may rather ask yourself some questions related to the problem you are trying to solve, and see if the data accounts for it.

Does this help?


Why is it hard to grant efficiency while using libraries? Any small database processing can be easily tackled by Python/Perl/... scripts, that uses libraries and/or even utilities from the language itself. However, when it comes to performance, people tend to reach out for C/C++/low-level languages. The possibility of tailoring the code to the needs seems to be what makes these languages so appealing for BigData -- be it concerning memory management, parallelism, disk access, or even low-level optimizations (via assembly constructs at C/C++ level).

Of course such set of benefits would not come without a cost: writing the code, and sometimes even reinventing the wheel, can be quite expensive/tiresome. Although there are lots of libraries available, people are inclined to write the code by themselves whenever they need to grant performance. What disables performance assertions from using libraries while processing large databases?

For example, consider an entreprise that continuously crawls webpages and parses the data collected. For each sliding-window, different data mining algorithms are run upon the data extracted. Why would the developers ditch off using available libraries/frameworks (be it for crawling, text processing, and data mining)? Using stuff already implemented would not only ease the burden of coding the whole process, but also would save a lot of time.

In a single shot:

what makes writing the code by oneself a guarantee of performance?
why is it risky to rely on a frameworks/libraries when you must assure high performance?

I don't think that everyone reaches for C/C++ when performance is an issue.

The advantage to writing low-level code is using fewer CPU cycles, or sometimes, less memory. But I'd note that higher-level languages can call down to lower-level languages, and do, to get some of this value. Python and JVM languages can do this.

The data scientist using, for example, scikit-learn on her desktop is already calling heavily optimized native routines to do the number crunching. There is no point in writing new code for speed.

In the distributed "big data" context, you are more typically bottleneck on data movement: network transfer and I/O. Native code does not help. What helps is not writing the same code to run faster, but writing smarter code.

Higher-level languages are going to let you implement more sophisticated distributed algorithms in a given amount of developer time than C/C++. At scale, the smarter algorithm with better data movement will beat dumb native code.

It's also usually true that developer time, and bugs, cost loads more than new hardware. A year of a senior developer's time might be $200K fully loaded; over a year that also rents hundreds of servers worth of computation time. It may just not make sense in most cases to bother optimizing over throwing more hardware at it.

I don't understand the follow up about "grant" and "disable" and "assert"?


Distinguish real building pictures from rendered ones I need to train an algorithm to identify rendered building images from real pictures. Here's an example:

A rendered picture

A real picture

The real pictures will vary from Good quality to Poor quality, (Like the one above) and will have many different aspect ratios. The same goes for the rendered pictures.

So what would you suggest me to use to do that, does anyone knows something that already do something similar?

To apply machine learning, you need to find thousands of labeled examples of both real and rendered images.
After you have labeled data, most likely a Convolutional Neural Network (CNN) will learn the differences between the labels.


Gathering the number of Google results from a large amount of searches. I am trying to build a simple dataset using Google, mainly because it seems like the best option for what I want.

I want to measure fame for a large group of scientists. The quick method is to measure the amount of Google results when searching their name. I do not care about the results, only the number of them. That method has it's flaws I know, so I am not opposed to an alternative.

My scientist data is composed of thousands of entries. Which is causing issues. I tried to programmatically search Google, but less then 1000 searches later they blocked the program. I also looked into their Search API, but that is limited to 100 searches a day unless I pay for more, but since I am a poor college student that isn't an option.

I was hoping someone here may be able to offer suggestions on build a dataset with some way of measuring fame.

With any search engine you will be limited by number of requests and any way of outcoming those limits will be a gray zone of violation of end user agreement (and, eventually, you will get banned for some time, of course). You should be looking into Search APIs of known search engines, for example, Bing gives you 5000 searches per month for free which - for a proof of concept research - might be enough. Also, 5k/month will give you some 20-30k until summer, so your data set will become bigger while you will be polishing your idea.

Also, Google's free tier search is limited to 100 requests per day. Which gives you completely legal 3k per month as well. Combined (given, you treat Google and Bing results as equal) you get 8k per month.


Combining convolution operations Reading an article about 1x1 convolution, I found this:

  It should be noted that a two step convolution operation can always be combined into one, but in this case [GoogLeNet] and in most other deep learning networks, convolutions are followed by non-linear activation and hence convolutions are no longer linear operators and cannot be combined.

What do they mean saying "two step convolution operation can always be combined into one"?

I think the comment is true for any kind of network where the neuron has a linear transformation function and there is no activation. Convolution is just a special case of linear transformation.

Basically, if your first layer outputs linear combinations of your features, and the second layer outputs linear combinations of the first layer outputs, then the second layers outputs is a linear combination of the initial features.

In the case of convolution, the interesting thing is that the convolution product is associative. So that if you apply two kernels consecutively $K_2 * (K_1*X) $ it is equivalent to $(K_2 * K_1)*X $ so that you can calculate the combined operation $ K_1 * K_2 $ as a unique kernel easily.


multilayer perceptron do not converge I have been coding my own multi layer perceptron in MATLAB and it can be compiled without error. My training data features,x, has values from 1 to 360, and training data output, y, has the value of sin(x).

The thing is my MLP only decreases the cost for the first few iterations and will get stuck at 0.5. I have tried including momentum but it doesn't help and increasing the layers or increasing the neurons do not help at all. I am not sure why this is happening ....

I have uploaded the files for your reference here.

The summary of my code is
1) I normalize my input data either using min-max or zscore

2) Initialize random weights and bias within the range of -1 to 1

for i = 1:length(nodesateachlayer)-1
    weights{i} = 2*rand(nodesateachlayer(i),nodesateachlayer(i+1))-1;
    bias{i} = 2*rand(nodesateachlayer(i+1),1)-1;
end

3) then I do a forward pass where input is multiplied by weights and added with bias then activated by a transfer function (sigmoid)

for i = 2:length(nodesateachlayer)
        stored{i} = nactivate(bsxfun(@plus,(weights{i-1}'*stored{i-1}),bias{i-1}),activation);
    end

4) then calculate the error then do a backward pass

dedp = 1/length(normy)*error;
    for i = length(stored)-1:-1:1
        dpds = derivative(stored{i+1},activation);
        deds = dpds'.*dedp;
        dedw = stored{i}*deds;
        dedb = ones(1,rowno)*deds;
        dedp = (weights{i}*deds')';
        weights{i}=weights{i}-rate.*dedw;
        bias{i}=bsxfun(@minus,bias{i},rate.*dedb');
    end

5) I have the cost plotted out at every iteration to see the descent

I assume there is something wrong with the code so where could the error possibly lies in

My view on your question, is that tiny networks seldom work. The above method uses a Neural Network to learn the function $y=\sin(x)$. Although this problems seems simple, it cannot be expected to be solved by a really tiny network (the above model uses a 5-layer MLP with hidden size [5,6,7], which is small).
Even if back-propagation is implemented correctly, would the model learn anything? No. I suppose Tensorflow implemented back-propagation correctly, here is the result using Tensorflow:

You see, it learns almost nothing. In fact, the MSE loss is very close to 0.5 as stated above.
My suggestion is to try a 3 layer MLP with hidden size 256. Here is the result:

You can see it's much better. MSE<0.1 now.
------------------code---------------------

x_ =np.atleast_2d(np.arange(0,360,1)).T
y_ = np.atleast_2d(np.sin(x_/180*np.pi))
g = tf.Graph()
with g.as_default():
    with tf.variable_scope("mlp"):
        input_x = tf.placeholder(shape=[None, 1], dtype=tf.float32)
        input_y = tf.placeholder(shape=[None,1], dtype=tf.float32)
        layer1 = tf.layers.dense(inputs=input_x, units=256, activation=tf.nn.sigmoid)
        #layer2 = tf.layers.dense(inputs=input_x, units=6, activation=tf.nn.sigmoid)
        #layer3 = tf.layers.dense(inputs=input_x, units=7, activation=tf.nn.sigmoid)
        output_y = tf.layers.dense(inputs=layer1, units=1) # inputs=layer1
        loss = tf.losses.mean_squared_error(input_y, output_y)
        train_op = tf.train.AdagradOptimizer(0.01).minimize(loss)
    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        for epoch in range(300):
            _, loss_ = sess.run((train_op, loss), feed_dict={input_x:x_, input_y:y_})
        y_hat_ = sess.run(output_y, feed_dict={input_x:x_, input_y:y_})
        print(loss_, end='\t')
plt.plot(x_,y_, 'g', x_,y_hat_,'b')
plt.legend(['ground truth', 'predicted'])


Keras: do `sample weights` take part in the derivatives According to Keras documentation, sample_weight can be used in order to give any sample in the training data a different importance in the loss.

I have googled around but have not found the answer to my question as follows:

Do sample weights take part in the derivatives? In other words, does Keras use these weights to train the model or they just give rise to a different loss value? i.e., the derivatives are "still" computed w.r.t. to the "unweighted" loss?

Because the loss function is not actually defined based on the sample weights, its rather passed in (as an argument) to the fit function in Keras.

sample_weights affect the total_loss, which is used to calculate the gradients.

  Because the loss function is not actually defined based on the sample weights, its rather passed in (as an argument) to the fit function in Keras

I think that what you're missing is that the loss function actually returns a list of loss values (one per sample in the batch). This is precisely because during fit, it is multiplied by the sample_weight of that batch.

Please note that although "class based" losses support reduction (over the batch), the Loss base class implements the weighting during reduction.


XOR problem with neural network, cost function I am having a problem understanding the cost function in a neural network. I have read many books and blog posts, but all of them describe that point in neural networks is to minimize the cost function (like sum squared error):

I tried to look at code for solving a problem with a multi layer neural network and back propagation. My question is: where in the code can I find the cost function? How can I plot the error surface?

import numpy as np
X_XOR = np.array([[0,0,1], [0,1,1], [1,0,1],[1,1,1]])
y_truth = np.array([[0],[1],[1],[0]])

def sigmoid(x):
    return 1 / (1 + np.exp(-x))
def sigmoid_der(output):
    return output * (1 - output)

np.random.seed(1)
syn_0 = 2*np.random.random((3,4)) - 1
syn_1 = 2*np.random.random((4,1)) - 1

for i in range(60000):
    layer_1 = sigmoid(X_XOR.dot(syn_0))
    layer_2 = sigmoid(layer_1.dot(syn_1))
    error = 0.5 * ((layer_2 - y_truth) ** 2)
    layer_2_delta = error * sigmoid_der(layer_2)
    layer_1_error = layer_2_delta.dot(syn_1.T)
    layer_1_delta = layer_1_error * sigmoid_der(layer_1)
    syn_1 -= layer_1.T.dot(layer_2_delta)
    syn_0 -= X_XOR.T.dot(layer_1_delta)
    if i % 10000 == 1:
        print(layer_2)

print(layer_2)

The cost function can be found in the delta rule, meaning the way you calculate your deltas. This delta is nothing more than the derivative of your error function after the weights: $\frac{\partial E}{\partial w_{ij}}$. So, if you are just interested in where the cost is encoded, this is the answer you are looking for.

If you, on the other hand want to know why this formula works, I can suggest you to read the derivation on wikipedia. The maths behind it is quite uncomplicated, you only compute the derivative in each layer and propagate this derivative through the layers. This is, by the way, how backpropagation got its name.


How do I perform weighted loss in multiple outputs on a same model in Tensorflow? How do I perform weighted loss in multiple outputs on a same model in Tensorflow? This means I am using a model that is intended to have 3 outputs. I did this because I would like the network to learn the relationships of the input variables. More specifically, this is multi-output regression.

The output model shape is like the following:

tf.Tensor: id=606, shape=(3, 3), dtype=float32, numpy=
array([[ 0.01483282,  2.9515972 , -0.07268244],
       [-0.8814691 ,  2.543654  ,  0.08576971],
       [-0.6933001 ,  1.3419302 , -0.25192362]], dtype=float32)

The model.summary() is below:

Layer (type)                 Output Shape              Param #
=================================================================
simple_rnn (SimpleRNN)       (3, 32)                   1152
_________________________________________________________________
dense (Dense)                (3, 3)                    99
=================================================================
Total params: 1,251
Trainable params: 1,251
Non-trainable params: 0
_________________________________________________________________

The shape is (3,3) with 1st index being the batch size (the network is an RNN). I need to set weighted loss for each of the outputs. I tried the argument weighted_losses on tf.keras.models.Model but the error returned:

ValueError: When passing a list as loss_weights, it should have one entry per model output. The model has 1 outputs

What is the correct method to have weighted loss using a single model?

As I understand it, you want to have a model trained on multiple "tasks". If so, this is what it could look like:

input_data = <your input sequence input>
output_data = <array of size (N, 3)>

out1_weight, out2_weight, out3_weight = <your weights for each output to adjust loss contribution>

input = Input()
simple_rnn = SimpleRNN()(input)

# Define the three outputs
out1 = Dense(1)(simple_rnn)
out2 = Dense(1)(simple_rnn)
out3 = Dense(1)(simple_rnn)

model = Model(inputs=[input], outputs=[out1, out2, out3], loss_weights=[out1_weight, out2_weight, out3_weight])

model.compile(<whatever params>)
model.fit(input_data, [output_data[:, 0], output_data[:, 1], output_data[:, 2]])


how I get different array size in calculating tpr? while learning about ROC curves I got confused by the how these are made.

I am considering here the Iris flower classification problem.
To calculate TPR we can use $$TPR=\frac{True \ positive}{True\ positive + \ False \ Negative }$$
Now in this case first we feed in the test data and find out the TPR and FPR by using the formulas above.But in this case how we get different array size.

>>>fpr = dict()
>>>tpr = dict()
>>>roc_auc = dict()
>>>for i in range(n_classes):
       fpr[i], tpr[i], _ = roc_curve(y_test[:, i], y_score[:, i])
       roc_auc[i] = auc(fpr[i], tpr[i])

tpr(true positive rate)
0: array([ 0.04761905,  1.        ,  1.        ]),  1: array([
0.03333333,  0.03333333,  0.1       ,  0.1       ,  0.2       ,
         0.2       ,  0.23333333,  0.23333333,  0.36666667,  0.36666667,
         0.4       ,  0.4       ,  0.7       ,  0.7       ,  0.73333333,
         0.73333333,  0.76666667,  0.76666667,  0.8       ,  0.8       ,
         0.83333333,  0.83333333,  0.86666667,  0.86666667,  0.9       ,
         0.9       ,  0.93333333,  0.93333333,  0.96666667,  0.96666667,
         1.        ,  1.        ]),  2: array([ 0.04166667,  0.79166667,  0.79166667,  0.95833333,  0.95833333,
         1.        ,  1.        ]),

You get different array sizes for your different classes because the number of points in your ROC curve depends on the number of unique predictions.

If your model only predicts 0, .5, and 1 your ROC curve will have three points. If your model predicts 100 different probabilities, your ROC curve will have ~100 points. (In practice, the length of the ROC curve may have a couple extra points where the points (0,0) and (1,1) are appended.

If you look at np.unique(y_score[:,i]).size for different i you should see the differences.


Would a decision tree classifier be applicable here? File from a situation where it is required to predict today’s stockprice from the stock prices of the previous three days:

Could you use a decision tree classifier for this task? Why or why not?

As explained in here, decision trees have their disadvantages of a high chance of overfitting, instability with little variance, and low bias. Moreover, considering the fact that your data appears to be numerical, and continuous, there's a high chance that the decision tree might calculate the entropy of your features while only considering scenarios in your training data.

Decision trees tend to perform better in the scenarios where you have repetitive scenarios, and where your data is pretty much correlated, and most importantly, where you have discontinuous, categorical data in your features, which certainly isn't the case for the data which you mentioned.


Correct order of operations involved into Dropout Suppose we have CNN with any hidden layer with activation followed by dropout layer. What is the correct precedence of activation and dropout operation if dropout implementation is inverted dropout and CNN mode is training mode? Do I need to compute activation in the first layer and then apply dropout with division by retain probability p, or I need to apply activation to the result of the division?

Say we have the following keras code

model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))

If I understand correctly, dropout with division by p will be applied to the activated result:

result = [survive_mask] * relu(output)/p

Is this correct? Wouldn't it be more natural to have

result = [survive_mask] * relu(output/p)

because otherwise dropout operation breaks activation value normalization (i.e. to [0, 1]) ?

The usual processing for your suggested layers:

model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))

would be (reading left to right)

dense output -> relu -> apply dropout mask -> apply "inverse dropout" divide by p

The precise combination may vary depending upon optimisations, and can in theory be changed a little without affecting the result (it doesn't matter to the end result numerically if we scale then mask or mask then scale for instance). However when dealing with vectorised optimisations (like those found in TensorFlow and Theano), it is normal to accept a percentage of "wasted" processing, and just have that naive left-to-right processing happen. It is often harder to parallelise decision branches than to simply process all items, even repeated multiplying by and adding zeroes for a significant fraction of each array.

There is no "normalization" implied by activation functions, so this is not a concern. From your comments, it seems you are worried that dividing by p could mean that the output of a neuron that would be between 0 and 1 (because you were using sigmoid activation for example) would now be between 0 and 1/p - i.e. larger. That is true. Is this a problem? No it is not, and in fact it is required. The impact of the larger values is fully compensated for by the weights learned in the connections between layers. If you used "vanilla" dropout then the weights would be correspondingly larger, but you would need to scale the outputs down (new range would be 0 to p) during testing/prediction.


Gradient Checking LSTM - how to get change in Cost across timesteps? I am performing gradient check for my LSTM which has 4 timesteps. The LSTM looks as follows:

   01       01       01       01
   ^        ^        ^        ^
  LSTM --> LSTM --> LSTM --> LSTM
   ^        ^        ^        ^
   11       11       11       11

So, at every timestep we are feeding in vector {1,1} and expect {0,1} at the output.

Assume I perturb the weight inside LSTM, then perform 4 forward props - one for each timestep - how do I now get delta of the cost function that this single perturbation has caused?

Am I allowed to simply add-up the change in Cost from all 4 timesteps to treat it as derivative estimate?

Also, should I perform it as follows for LSTM:

perturb a single weight upwards
forward prop 4 timesteps
perturb the weight downwards
forward prop 4 timesteps
get 4 deltas
sum the 4 deltas to get a total change in Cost

or

Set N=0
perturb the weight upwards
foward prop at a particular timestep N
perturb the weight downwards
forward prop at a particular timestep N
get single delta, store it away
increment N
until N not equal 4 return to step 2)
sum the 4 deltas to get a total change in Cost

The second approach somehow seems more correct, because LSTM will have a hidden state ..Is this correct intuition or it won't matter?

Interesting question.

  Like standard backpropagation, [backpropagation through time] consists of a repeated application of the chain rule. The subtlety is that, for recurrent networks, the loss function depends on the activation of the hidden layer not only through its influence on the output layer, but also through its influence on the hidden layer at the next timestep.

It looks like both approaches would have similar results but at a different granularity (adding noise at different levels) this is because the backprop is not really disturbed by this addition as it is still chained from last time step to the first. Hence it boils down to why you actually want to add noise in the first place, as mentioned in this paper , sections IIIC and IV.


Tensorflow with Python code I want to integrate simple python code with in the tensorflow graph.
I'm not sure if it's feasible. If feasible, please suggest how to integrate it.
Use case is, I want feed the output of intermediate tensor as an input to my python code and then python code output to another tensor.
Does @tf.function can be use here?
Thanks.

Yes, theoretically that's what tf.function does. This uses a tensorflow module called AutoGraph to basically convert your python/numpy operations to tensorflow ops.
However not all python operations can be converted. Things like printing, appending to lists and mutating global variables won't work in graph mode!
I suggest you take a look at the official guide to see how it is used and if it is applicable in your situation.


feature normalisation problem I am very new to ML and have limited knowledge about it. I am having issue in feature normalization process. I have understood from the post that we need to normalize the training features and scale the test/validation features with the training data. I am facing issue in the implementation as in my case my training samples have fixed dimension but the dimension of validation and test data is variable. So, I can apply zero mean unit variance for training data but I am not sure how can I normalize the validation/test data samples as the sample dimension/length is variable/not fixed.

That is a common case on image and audio processing, you need to find a way in which dimensions stay the same, such as normalizing per channel.
If you have a 1D vector of features, taking mean and variance of all variables will end up normalizing it in a way, it works in Computer Vision like a charm. It is also a way to reduce the space cost of your normalizing algorithm.


XGBoost becomes unstable when predicting more than ~300 classes I'm using the Python implementation of XGBoosts (version 0.80) XGBoostClassifier to predict one of a large number of classes.

My feature data consists of a sparse boolean matrix of ~10M rows, ~5k columns, with a density of 0.003. (so about 15 values per row).

My targets consist of 2000+ different classes in a long-tail skewed distribution (~300 classes occur more than 10k times).

For now I'm grouping all of my uncommon classes (<10k) into 'other' so they at least count as negative examples. But ideally, I'd like to be able to use a lot more of the different classes as actual targets.

However, whenever I use, say, 800 different targets (each occurs ~5k+ times), the model becomes unstable: training takes a lot longer, the training loss doesn't seem to converge (I haven't saved the output of this unfortunately) and the predictive quality becomes increasingly poor (predict_proba returning 0.99+ for the wrong classes).

Should I just change some parameter I've overlooked? Am I running out of float precision for my loss? Is it due to sampling in the approx method? Does anyone have any ideas?

My specific classifier:

clf = XGBClassifier(
    n_jobs=8,
    max_depth=7, #arbitrary
    min_child_weight=5, #arbitrary
    subsample=0.5, #arbitrary
    colsample_bytree=0.9, #arbitrary
    tree_method='approx',
    objective='multi:softprob',
    eval_metric='mlogloss',
    silent=False
)

I am affraid that it may not be a problem about tuning the XGB classifier, but about your dataset.

Even with 10M rows, a density of 0.003 over 5000 rows seems far from enough to get interesting results about 300+ classes.

I would suggest to start with a way simpler approach to confirm that the modelling approach is okay. That would mean :

Group your variables significantly more (<100), identify and keep the one that appears significant predictors
Group your classes significantly more (<10)
Start with a simpler model (random forest maybe)

Once you do that and get baseline performance metrics, you can try to add more classes / variables / complexity to your model and see if it actually improve.

A one step calibration process, where you dump all your data into a very complex model is rarely the way to go.


How to compare a sentence with a paragraph and get its probability in terms of correctness? This is my first post on stackoverflow network and I am dealing with a machine learning.

Lets say I have a paragraph describing a rabbit and tortoise story. The story concludes that tortoise is a winner.

Now I want to train ML engine based on this paragraph. If I provide a statement as "Rabbit won the race", how can it show me the correctness of my statement in terms of probability of correctness?

Need help.

This is a complex problem related to Natural Language Understanding (NLU). The key part in such a system is certainly textual entailment, but it could also use techniques such as Question Answering and summarization.

I'm not aware of any direct model or tool to carry out this task exactly, so I think you will have to study the literature in these fields.


Good practices for manual modifications of data More often than not, data I am working with is not 100% clean. Even if it is reasonably clean, still there are portions that need to be fixed.

When a fraction of data needs it, I write a script and incorporate it in data processing.

But what to do if only a few entries needs to be fixed (e.g. misspelled city or zip code)? Let's focus on "small data",  such as that in CSV files or a relational database.

The practical problems I encountered:

Writing a general script trying solve all similar errors may give unintended consequences (e.g. matching cities that are different but happen to have similar names).
Copying and modifying data may make a mess, as:

Generating it again will destroy all fixes.
When there are more errors of different kinds, too many copies of the same file result, and it is hard to keep track of them all.

Writing a script to modify particular entries seems the best, but there is overhead in comparison to opening CSV and fixing it (but still, seems to be the best); and either we need to create more copies of data (as in the previous point) or run the script every time we load data.

What are the best practices in such a case as this?

EDIT: The question is on the workflow, not whether to use it or not.

(In my particular case I don't want the end-user to see misspelled cities and, even worse, see two points of data, for the same city but with different spelling; the data is small, ~500 different cities, so manual corrections do make sense.)

1.  Do not modify the original data

Having the original data source intact is important.  You may find that updates you make to the data are not valid.  You may also find a more efficient way to make updates and you will want to regression test those updates.

Always work with a copy of the data, and add columns/properties/metadata that includes any processed corrections.

Example, if your data is a .csv file that includes a city name that contains several misspellings:

1. Copy the .csv to a new file
2. Add a column for containing "adjusted_city_name"
3. Copy data from the city_name column to the
   adjusted_city_name column and make corrections in that column.

2. Document proposed changes

Any changes you want to make to data should be documented so that they can be replicated moving forward.

Version control and timestamp the document every time you change it.  That will help in troubleshooting at a later date.

Be explicit.  Do not simply state "correct capitalization problems", state "ensure that the first letter of each city name begins with a capital letter and the remaining letters are lower-case."

Update the document with references to any automation routines that have been built to manage data cleansing.

3. Decide on a standard data cleansing technology

Whether you use perl, python, java, a particular utility, a manual process or something else is not the issue.  The issue is that in the future you want to hand the data cleansing process to somebody else.  If they have to know 12 different data cleansing technologies, delegating the cleansing procedure will be very difficult.

4. Standardize the workflow

There should be a standard way to handle new data.  Ideally, it will be as simple as dropping a file in a specific location and a predictable automated process cleanses it and hands off a cleansed set of data to the next processing step.

5. Make as few changes as is absolutely necessary

It's always better to have a fault tolerant analysis than one that makes assumptions about the data.

6. Avoid manual updates

It's always tempting, but people are error-prone and again it makes delegation difficult.

Notes on manual processing

To more completely address the original question as to whether there's a "good" way to do manual processing, I would say no, there is not.  My answer is based on experience and is not one that I make arbitrarily.

I have had more than one project lose days of time due to a client insisting that a manual data cleansing process was just fine and could be handled internally.  You do not want your projects to be dependent on a single individual accomplishing a judgement based task of varying scale.

It's much better to have that individual build and document a rule set based on what they would do than to have them manually cleanse data.  (And then automating that rule set)

If automation fails you in the end or is simply not possible, the ability to delegate that rule set to others without domain specific knowledge is vital.

In the end, routines to do something like correct city names can be applied to other data sets.


ValueError: Error when checking input: expected conv2d_1_input to have 4 dimensions, but got array with shape (142, 1) I'm trying to pass all my images (71 for each class) from folder 'train' to model.fit.
The method ReadImages get these images and resize them (because are too big 4288x2848)....

But when i run my code throws this error:

ValueError: Error when checking input: expected conv2d_1_input to have 4 dimensions, but got array with shape (142, 1)

This is my code:

def ReadImages(Path):
    LabelList = list()
    ImageCV = list()
    classes = ["nonPdr", "pdr"]

    # Get all subdirectories
    FolderList = os.listdir(Path)

    # Loop over each directory
    for File in FolderList:
        if(os.path.isdir(os.path.join(Path, File))):
           for index, Image in enumerate(os.listdir(os.path.join(Path, File))):
                # Convert the path into a file
                ImageCV.append(cv2.imread(os.path.join(Path, File) + os.path.sep + Image))
                ImageCV[index] = cv2.resize(ImageCV[index], (700, 600))

                # Add a label for each image and remove the file extension
                LabelList.append(classes.index(os.path.splitext(File)[0]))
        else:
            ImageCV.append(cv2.imread(os.path.join(Path, File) + os.path.sep + Image))
            # Add a label for each image and remove the file extension
            LabelList.append(classes.index(os.path.splitext(File)[0]))
    return ImageCV, LabelList

model = Sequential()
model.add(Conv2D(64, kernel_size=(3,3), padding="same",activation="relu", input_shape=(605,700,3)))
model.add(MaxPooling2D((2, 2)))
model.add(Conv2D(128,  kernel_size=(4,4), padding="same",activation="relu"))
model.add(MaxPooling2D((2, 2)))
model.add(Flatten())
model.add(Dense(1, activation='sigmoid'))
model.compile(optimizer='RMSprop', loss='binary_crossentropy', metrics=['accuracy'])

data, labels = ReadImages(TRAIN_DIR)
print(data[0])
model.fit(np.array(data), np.array(labels), epochs=10, batch_size=20)

model.save('model.h5')

detail: when I print (data[0]) it return:

  [0 0 0]
  [0 0 0]
  ...
  [1 0 0]
  [2 2 2]
  [0 0 0]]

 [[3 0 0]
  [3 0 0]
  [3 0 0]
  ...
  [2 0 0]
  [2 2 2]
  [0 0 0]]

 [[8 0 0]
  [8 0 0]
  [8 0 0]
  ...
  [2 0 0]
  [0 0 0]
  [0 0 0]]

 ...

 [[0 0 0]
  [0 0 0]
  [0 0 0]
  ...
  [0 0 0]
  [0 0 0]
  [0 0 0]]

 [[0 0 0]
  [0 0 0]
  [0 0 0]
  ...
  [0 0 0]
  [0 0 0]
  [0 0 0]]

 [[0 0 0]
  [0 0 0]
  [0 0 0]
  ...
  [0 0 0]
  [0 0 0]
  [0 0 0]]]

What should I do to fix it? I appreciate any help

def ReadImages(Path):
    LabelList = list()
    ImageCV = list()
    classes = ["nonPdr", "pdr"]

   # Get all subdirectories
   FolderList = [f for f in os.listdir(Path) if not f.startswith('.')]
   print(FolderList)

   # Loop over each directory
   for File in FolderList:
      for index, Image in enumerate(os.listdir(os.path.join(Path, File))):
          # Convert the path into a file
          ImageCV.append(cv2.resize(cv2.imread(os.path.join(Path, File) + os.path.sep + Image), (600,700)))
          LabelList.append(classes.index(os.path.splitext(File)[0]))

   return ImageCV, LabelList

A bit of change to your code after discussion in the chat room.


Deep Learning for Video Classification Which Deep Learning architecture is best for classifying short videos of variable length? I would like to classify videos that last from 1 up to 3 seconds.

My suggestion is to use Convolutional and Recurrent layers in the same Neural Network.

You'd have to capture a given number of frames of a video (let's say one each 0.5 seconds), and feed arrays of screenshots into the model. Its structure would be:

Conv (and MaxPool) layers to process pixel data - they will extract and process relevant information from each screenshot.
LSTM layers - that will process their sequence, extracting meaning from their flow.
Dense layers at the end to perform classification, with softmax activation at the output layer.

That's how I would do. It's going to be computationally expensive, if you don't have a GPU it won't be easy.


How to plot multiple columns with ggplot in R? I do have a data frame with different categorical and numerical columns with the following schema:

Id | num_col_1 | num_col_2 | num_col_3 | cat_col_1 | cat_col_2

Now I want to draw a combined plot with ggplot where I (box)plot certain numerical columns  (num_col_2, num_col_2) with boxplot groups according cat_col_1 factor levels per numerical columns. Along y axis is the spread of the respective selected columns (not other column). So far I couldn' solve this combined task.

Thank you.

If I understand right your question, you are looking to plot selected numerical columns against a selected categorical column of your dataset, am I right ?

If so, you can have the use of dplyr, tidyr and ggplot2 packages to achieve this.

Starting with this dataframe:

  id        num1      num2      num3 cat cat2
1  C -0.48892284  1.417909 2.8884577   a    f
2  C -0.62795166  1.472390 1.6625688   c    f
3  B -0.04691673  2.731553 0.9692889   c    e
4  B  0.16261812 -1.152528 2.4308332   a    d
5  C  1.29230591 -1.609465 2.2089074   a    f
6  E -0.46355650 -1.070132 0.4517597   b    f

Basically, you are selecting first your columns of interest (here num1, num2 and cat), then, you reshape data into a longer format using pivot_longer function to obtain something like that:

library(tidyr)
library(dplyr)
df %>% select(id, num1, num2, cat) %>%
  pivot_longer(., cols = c(num1,num2), names_to = "Var", values_to = "Val")

# A tibble: 200 x 4
   id    cat   Var       Val
   <fct> <fct> <chr>   <dbl>
 1 C     a     num1  -0.489
 2 C     a     num2   1.42
 3 C     c     num1  -0.628
 4 C     c     num2   1.47
 5 B     c     num1  -0.0469
 6 B     c     num2   2.73
 7 B     a     num1   0.163
 8 B     a     num2  -1.15
 9 C     a     num1   1.29
10 C     a     num2  -1.61
# … with 190 more rows

Finally, you can add to this pipe sequence the plotting part by calling ggplot and geom_boxplot :

library(tidyr)
library(dplyr)
library(ggplot2)
df %>% select(id, num1, num2, cat) %>%
  pivot_longer(., cols = c(num1,num2), names_to = "Var", values_to = "Val") %>%
  ggplot(aes(x = Var, y = Val, fill = cat)) +
  geom_boxplot()

Is it what you are looking for ?

Data

set.seed(123)
id <- sample(LETTERS[1:5],100, replace = TRUE)
num1 <- rnorm(100)
num2 <- rnorm(100)*2
num3 <- rnorm(100)+2
cat <- sample(letters[1:3],100, replace = TRUE)
cat2 <- sample(letters[4:6],100, replace = TRUE)
df <- data.frame(id, num1, num2,num3, cat,cat2)


What is the first tool to learn start your your data science projects? What is the first tool  to learn start your your data science projects?

This question was probably asked million times. Please try to find answers berfore you ask "where to learn start".


Early stopping on validation loss or on accuracy? I am currently training a neural network and I cannot decide which to use to implement my Early Stopping criteria: validation loss or a metrics like accuracy/f1score/auc/whatever calculated on the validation set.

In my research, I came upon articles defending both standpoints. Keras seems to default to the validation loss but I have also come across convincing answers for the opposite approach (e.g. here).

Anyone has directions on when to use preferably the validation loss and when to use a specific metric?

TLDR; Monitor the loss rather than the accuracy

I will answer my own question since I think that the answers received missed the point and someone might have the same problem one day.

First, let me quickly clarify that using early stopping is perfectly normal when training neural networks (see the relevant sections in Goodfellow et al's Deep Learning book, most DL papers, and the documentation for keras' EarlyStopping callback).

Now, regarding the quantity to monitor: prefer the loss to the accuracy. Why?
The loss quantify how certain the model is about a prediction (basically having a value close to 1 in the right class and close to 0 in the other classes). The accuracy merely account for the number of correct predictions. Similarly, any metrics using hard predictions rather than probabilities have the same problem.

Obviously, whatever metrics you end up choosing, it has to be calculated on a validation set and not a training set (otherwise, you are completely missing the point of using EarlyStopping in the first place)


What Non-linearities are best in Denoising RNN Autoencoders and where should the go? I’m employing a denoising RNN autoencoder for a project relating to motion capture data. This is my first time using auto encoder architectures and I was just wondering what non-linearities should be placed in these models and where they should go. This is my model as it stands:

class EncoderRNN(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers):
        super(EncoderRNN, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.num_layers = num_layers

        self.rnn_enc = nn.RNN(input_size=input_size, hidden_size=hidden_size, num_layers=num_layers, batch_first=True)
        self.relu_enc = nn.ReLU()

    def forward(self, x):
        pred, hidden = self.rnn_enc(x, None)
        pred = self.relu_enc(pred)
        return pred

class DecoderRNN(nn.Module):
    def __init__(self, hidden_size, output_size, num_layers):
        super(DecoderRNN, self).__init__()
        self.hidden_size = hidden_size
        self.output_size = output_size
        self.num_layers = num_layers

        self.rnn_dec = nn.RNN(input_size=hidden_size, hidden_size=output_size, num_layers=num_layers, batch_first=True)
        self.relu_dec = nn.ReLU()

    def forward(self, x):
        pred, hidden = self.rnn_dec(x, None)
        pred = self.relu_dec(pred)
        return pred

class RNNAE(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers):
        super(RNNAE, self).__init__()
        self.encoder = EncoderRNN(input_size, hidden_size, num_layers)
        self.decoder = DecoderRNN(hidden_size, input_size, num_layers)

    def forward(self, x):
        encoded_input = self.encoder(x)
        decoded_output = self.decoder(encoded_input)
        return decoded_output

As you can see I have a ReLU non-linearity in each of the encoder and decoder networks but am not sure whether this is the correct implementation for these architectures.
The model learns on the data OK but the MSE loss doesn’t really improve after the first few epochs and I have a suspicion it’s because of these ReLU functions.

Any advice on how to improve this/ whether this is basically correct would be very helpful.

Cheers!

I have some familiarity with Recurrent encoder-decoder archtectures, especially with seq2seq models. This is a standard implementation:

from tensorflow.keras import Sequential

from tensorflow.keras.layers import Bidirectional, LSTM
from tensorflow.keras.layers import RepeatVector, TimeDistributed
from tensorflow.keras.layers import Dense

from tensorflow.keras.activations import elu, relu

seq2seq = Sequential([

    Bidirectional(LSTM(len_input), input_shape = (len_input, no_variables)),

    RepeatVector(len_input),

    Bidirectional(LSTM(len_input, return_sequences = True)),

    TimeDistributed(Dense(100, activation = elu)),

    TimeDistributed(Dense(1, activation = relu))

])

where len_input is the length of the input trends, and np_vars the number of variables that compose each observations (if it's a univariate series, set it to 1). The choice of Bidirectional() wrappers is optional.

The model works like this: The input LSTM layer encodes the sequential information to a vector containing a latent representation. This vector is replicated by the RepeatVector() layer, its signal is then calculated using Dense() layers, and distributed for each timestep prediction sequentially (by the TimeDistributed() wrapper). You don't worry about non-linearities at this point, since this is learned by internal LSTM states and by Dense() vectors. Fancier implementation would require stacked Recurrent layers, or deeper feed-forward blocks.

If your goal is to build a Denoising Autoencoder you should pair each input series to its "clean" counterpart, so that the Network learns how to reduce noise.


Guidance needed with dimension reduction for clustering - some numerical, lots of categorical data I've my data in a Pandas df with 25.000 rows and 1.500 columns without any NaNs. Of the columns about 30 contain numerical data which I standardized with StandardScaler(). The rest are cols with binary values which originated from cols with categorical data. (used pd.get_dummies() for this)

Now I'd like to reduce the dimensions. I'm already running

from sklearn.decomposition import PCA

pca = PCA(n_components=2)
pca.fit(df)

for three hours and I asked my self if my approach was correct. I also saw two variants of PCA, one for sparse data. Does it mean that it doesn't make sense to run PCA in such a mixed scenario?

As I was up to now busy with cleaning and transforming my data, I'd like to understand what a good strategy would be to eliminate irrelevant columns.

I'd appreciate some hints to move forward.

There are many ways to get rid of redundant dimensions. The choice wheter to do it or not depends on what kind of problem you want so solve and what kind of algorithm you plan to choose.


What should I study to find optimal value of best feature combinations in machine learning? I would like to do production optimization with machine learning and/or optimization problem.

My goal is not to find minimizing loss in loss function only to give the best y value. My ultimate goal is to find optimal values of the best feature combination. In other words, 'I would like to find the best x values (feature values) that yields the best y (minimizing loss). For example, 30 (temp), 100 (Pressure), 50 (weight) gave the best product y. The actual best set of x feature values.

Supervised machine learning techniques do not provide the best-predicted x values to me. but I would like to find the x. Therefore, my question is as below:

What is the field for finding optimal x values for optimal y values called? Is continuous and discrete optimizations are ones finding the x values for y?
I found many studies but I am not sure if it is really about finding the x values (the optimal values of the best combination of features). Can someone please tell me what field I must study for this?
What kind of techniques should I use to find what I want?

Welcome to DS.SE!

As you already mentioned, machine learning is not about finding the best values of features regarding a function. Also, loss functions usually measure the errors of predictions and not the quality of the feature values (e.g. " the best product y").

What you are looking for is called optimization or search algorithms but not learning or prediction. If the search space (i.e. the combination of all possible feature values) is not that large, you can just perform an exhaustive search (trying all combination). If the search space is large, then smarter approaches might be useful. These techniques need a function (usually called objective function) to optimize (finding the combination of input values that results in a min/max outcome).

In general, there are two approaches: mathematical modeling (i.e. linear programming) and metaheuristics (e.g., evolutionary algorithms and swarm intelligence). Mathematical models can be fast and guarantee to find the optimum when the objective function is well-defined and the underlying assumption are met (you may need to simplify your constraints sometimes). If these conditions are not met, then metaheuristics usually can find a set of approximate solutions.


How do I download a Google Colab notebook? I have to download code written in a Google Colab notebook, because I need to send it via email. How do I do this?

You do not need to download Google Colab to send a file through email.
To send directly:

On Upper right-hand side there is a share button, Click on it.
A pop-up appears, Enter email addresses or usernames in the field 'Add People and groups'.
(Optional) Go to the setting gear button upper right corner and adjust permissions.
Click on 'Done'.

Download then send to email:

Click on the "File" menu in the upper left corner.
A popup appears, find 'Download .ipynb' and click on it.


Correlation matrix with a dataset with many missing value I am a data-science rookie and I would like to use Python/ R to create a correlation matrix (something like this: http://www.marketcalls.in/python/quick-start-guide-compute-correlation-matrix-using-nsepy-pandas-python.html) and build a machine learning model. However, I have some questions and would really appreciate some guidance.

Question 1: Although the data file is pretty big and have more than 350,000 entries, some columns missed many values (i.e., 60%/ 70% of the values are missing). I am wondering should I abandon those columns/ delete those rows/ any other great recommendations? And what is a good threshold, is it okay to proceed with columns that 20%? 30%? 40% values are missing.

Thank you very much. Greatly appreciated your help!!

It is impossible to say whether or not you should trim your data, without doing some Exploratory Data Analysis first. Those missing values may be important to your understanding of the dataset, so you have to look at your data comparing variable to variable to determine if the missing values hold information.
For example, if your data is about vehicles, there could be a variable representing number of doors.  If your missing data is structural an observation of a motorcycle may have a missing value for that variable.  In which case, the observation for that variable could be discarded as meaningless.  Or, you could impute the missing value to something meaningful like the number 0. It depends on what insight you are ultimately trying extract from you data.  Handling missing values is quite a complex subject.

I would suggest that you "Tidy the data" first.  From the Wickham paper

  Tidy datasets are easy to manipulate, model and visualize, and have a specific structure:
  each variable is a column, each observation is a row, and each type of observational unit
  is a table.

Initially this can help you do some Exploratory Analysis to determine what is important and what is not.  Once you have made those distinction you can find and implement strategies to handle the missing values appropriately.


Words to numbers faster lookup I'm training an LSTM for sentiment analysis on a review dataset downloaded from here. The music review dataset contains about 150K data points (reviews of varying length labelled pos or neg). After creating a dictionary, I'm running a script in Python to replace strings (words) with numbers that keras/theano will embed later.

The problem is that such a large dataset requires a lot of time for lookup. I would appreciate if anyone had suggestion on a tool for faster lookup or similar. Currently I just loop through every word in the corpus and replace it with the corresponding number from the dictionary (1-hot encoding essentially)

EDIT:

I'm doing roughly the following: each Python list is a sentence (before tokenization here):

['noble', 'interesting_superlatives',...,'the_idea']

which I want to conver to a list of integers, like:

[143599, 12387,...,7582]

I referred to it (probably incorrectly) as one-hot encoding because for each word there is exactly one number in the dictionary.

You're doing something wrong. I can query a 100K word dict in nanoseconds

word_list = open('/usr/share/dict/words').read().split()
len(word_list)

> 99171

word_dict = {word: hash(word) for word in word_list}
%timeit word_dict['blazing']

> 10000000 loops, best of 3: 33.8 ns per loop