When/how should I use the validation set for hyper-parameter sweeps for neural networks? I know similar questions have been asked so many times, but I couldn't find the answer to this one particularly, at least not in a way that satisfied me.

I am very confused about how to use validation sets. I know they are used to perform hyper parameter sweeps but I'm not quite sure in what way. For example, suppose I am trying to decide between a neural network that has 1 hidden layer and one which has 2, and I have a 50/25/25 train/validation/test split of my data. I will not be performing k-fold CV for reasons which are inherent to my problem.

Should I be using the validation set during training (to prevent overfitting) and the same validation set after the fact (once when the models are completely trained) to determine which model is better? When do I use the validation set to compare models?

And then once I get to the test set, I know I am supposed to train on the training data and validation set together. But should I actually keep the validation set separate from the training set during training of the finalized model to prevent overfitting? Why would I use the separate validation set during training of the individual models to prevent overfitting, but not use it for that purpose (and make it more training data) on the final model?

Let's take a step back and look at why we make these splits:

  Model selection: estimating the performance of different models in order
  to choose the best one.

  Model assessment: having chosen a final model, estimating its predic-
  tion error (generalization error) on new data.

(Source: "The Elements of Statistical Learning - Data Mining, Inference, and Prediction", Hastie et al)

For model selection you use the validation set and for model assessment you use the test set.

Accordingly, a straight forward approach would be this:

Split data into train/valid/test sets
Train models on train dataset
Compare models on valid dataset
Repeat steps 2 and 3 until you meet your personal stopping criterion (e.g. performance sufficiently)
Select your final model and retrain it on the train and valid dataset
Evaluate the performance of your selected and retrained model on the test dataset

Here is an example with an SVM taken from "Introduction to Machine Learning with Python" by Mueller and Guido:

from sklearn.svm import SVC

# split data into train+validation set and test set
X_trainval, X_test, y_trainval, y_test = train_test_split(
iris.data, iris.target, random_state=0)

# split train+validation set into training and validation sets
X_train, X_valid, y_train, y_valid = train_test_split(
X_trainval, y_trainval, random_state=1)
print("Size of training set: {} size of validation set: {} size of test set:"
" {}\n".format(X_train.shape[0], X_valid.shape[0], X_test.shape[0]))
best_score = 0
for gamma in [0.001, 0.01, 0.1, 1, 10, 100]:
for C in [0.001, 0.01, 0.1, 1, 10, 100]:

# for each combination of parameters, train an SVC
svm = SVC(gamma=gamma, C=C)
svm.fit(X_train, y_train)

# evaluate the SVC on the test set
score = svm.score(X_valid, y_valid)

# if we got a better score, store the score and parameters
if score > best_score:
best_score = score
best_parameters = {'C': C, 'gamma': gamma}

# rebuild a model on the combined training and validation set,
# and evaluate it on the test set
svm = SVC(**best_parameters)
svm.fit(X_trainval, y_trainval)
test_score = svm.score(X_test, y_test)
print("Best score on validation set: {:.2f}".format(best_score))
print("Best parameters: ", best_parameters)
print("Test set score with best parameters: {:.2f}".format(test_score))


How to use unigram and bigram as an feature on SVM or logistic regression How to use unigram and bigram as an feature to build an Natural Language Inference model on SVM or logistic regression?on my dataset i have premise, hypotesis and label column. I'm planning to use the unigram and bigram of the premis or hipotesis or both as one of the features on my training.
for example :
 premise                                      |hipotesis                         |hypothesis bigram
===============================================================================================
I am planning to use the unigram and bigram   |I am planning to use the unigram  |[(i, am), (am, planning), (planning, to), (to, use), (use, the), (the, unigram)]

the hypothesis bigram is a list of bigram(word), so i cant use it as input to my svm or logistic. can i convert the hypothesis bigram into vector?

You need to create a vocabulary of the n-grams, i.e., a numbered inventory of bigrams that you are going to use as features. Typically, these are the most frequent ones. When you create the feature vector, you start with a zero vector and put one (or add one) if the n-gram with the corresponding index appears is in your sentence.
Machine learning libraries typically have functions that do that. For instance, in scikit-learn, you can use CountVectorizer to do the job. The fit method has an ngram_range argument that controls the length of n-grams that are considered in the feature vectors.


Using Information from the rest of a Sequence to Predict the Label for any one Item I have a dictionary of variable-length sequences:

[(file_name[-10:], len(tag_is_header_list)) for file_name,
 tag_is_header_list in HEADER_PATTERN_DICT.items()]

[('37bd1.html', 25),
 ('0bcce.html', 40),
 ('90364.html', 28),
 ('8f9c7.html', 24),
 ('d12d4.html', 73),
 ('46837.html', 37),
 ('adb92.html', 53),
 ('0a1e7.html', 69),
 ('da077.html', 43),
 ('9366a.html', 21),
 ('6ae4d.html', 37),
 ('f62ee.html', 19),
 ('73aee.html', 33),
 ('e090a.html', 35),
 ('8b093.html', 44)]

These contain a label for each item as to whether or not they are a subject heading:

HEADER_PATTERN_DICT[sorted([(file_name, len(tag_is_header_list)) for file_name,
                            tag_is_header_list in HEADER_PATTERN_DICT.items()],
                           key=lambda x: x[1])[0][0]]

[(None, True),
 ('<div', False),
 ('<div', False),
 (None, True),
 (None, False),
 ('<li', False),
 ('<li', False),
 ('<li', False),
 (None, False),
 (None, False),
 ('<li', False),
 ('<li', False),
 ('<li', False),
 (None, True),
 (None, True),
 ('<li', False),
 ('<li', False),
 ('<li', False),
 ('<div', False)]

Every item in the sequence is an instance for which the label should be predicted. So, what is the best way to use some variable-length sequence vectorization to train a model to predict the label?

The fastest way to train a model to predict each item's label is using Conditional Random Fields (CRT) like in this example. h/t @erwin


Calculating the entropy of a neural network I am looking to calculate the information contained in a neural network. I am also looking to calculate the maximum information contained by any neural network in a certain amount of bits. These two measures should be comparable (as in I can compare whether my current neural network has reached the max or is less than the max and by how much).
Information is relative, so I define it relative to the real a priori distribution of the data that the neural network is trying to estimate.
I have come across Von Neumann entropy which can be applied to a matrix, but because it is not additive I can't apply it to a series of weight matrices (assuming the weight matrices encode all the information of a neural network).
I found three other papers Entropy-Constrained raining of Deep Neural Networks , Entropy and mutual information in models of deep neural networks and Deep Learning and the Information Bottleneck Principle. The second contains a link to  this github repo, but this method requires the activation functions and weight matrices to be known which is not the case for finding the max entropy of any neural network in n bits.
How can I calculate the amount of information contain in/entropy of a neural network? And how can I calculate the same measure for any neural network in n bits?

To start, see Information Theory, Inference, and Learning Algorithms by David J.C. MacKay, starting with chapter 40 for information capacity of a single neuron (two bits per weight) through to at least chapter 42 for Hopfield Networks (fully connected feedback). The classic reference for information of a Hopfield Network is Information Capacity of the Hopfield Model by Abu-Mostafa & St. Jacques, but the textbook should have enough.


How much data is created daily (source of IBM estimation) I've stumbled upon this IBM text.

It states that:

  Every day, we create 2.5 quintillion bytes of data

It's very interesting statistic, but I couldn't find any source for this.

Does anyone know how they calculated this? Or maybe someone knows some papers showing similar/different results?

This paper from CSA has the same statistic, but again without any source.

Note: The quote from IBM's text refers to data created by humanity, not by IBM alone.

The McKinsey Global Institute published an interesting report on the topic of Big Data.
You can find it here.

I don’t know where IBM go the numbers from, but this report is a good source.


How would I map categories between two similar but different sets of categories Having multiple sets of categories from different listings sites (e.g. Yelp, yellowpages.com,  Google My Business...).  I want to figure out what category X on one site is on another site.

We have hundreds of thousands of businesses and the categories on all the sites they are on, so we could see that "Galbi Foo Restaurant" is in category "Restaurants > Korean" on one site and "Restaurants" on the other.

Some examples category mappings that will have to happen:

Nail Salons = Manicure & Pedicure
Eyelash Service = Visagist
Tanning = Sunbed Salon
Specialty Food = Grocery (Specialty Food child node doesn't exist)
Diagnostic Imaging = Radiologist

Where would I start to solve this?  It seems like a classification (logistic regression) problem.  But this ML stuff hasn't clicked with me yet, so I'm likely to drastically over or under complicate these things :).

This sounds like a pretty standard supervised learning problem. In this case, your records would be businesses on site X and their actual category on site Z. Your predictors would be tags/categories for a particular business on site X, and your target variable, y (i.e., what you're trying to predict), would be the category on the other website. As far as the code goes, you have a variety of options depending on your preferred language. You could use the caret package in R, the scikit-learn library in Python, or the Weka library (maybe even Spark's ML lib because of its simplicity) in Java/Scala.

Side note, in your question I think you meant to say "logistic regression" instead of "logical regression". You don't need to use logistic regression (although it wouldn't hurt). You could also try algorithms like Random Forests or Naive Bayes.

Also worth noting: your target variable will have many classes (ie every possible category for the site you're trying to predict), so don't get alarmed if it seems like there are a lot of classes. That's normal for a problem like the one you've described.


Determining if a time series is random An example time series would be the stock market, which is sometimes described as a random walk.  Over time, this is clearly not the case as it has essentially gone in one direction (up) with only occasional setbacks.

I'm coming up with several characteristics that could be used to measure orderliness of a specific segment of a time series:

Net movement from start to finish
Linearity of movement
Length of time

In short, the larger the absolute value of the movement relative to the standard deviation of the series over similar lengths of time and the more linear the move, the more significant that segment.  These characteristics could also form a kind of signature of each given segment.

Maybe the answer to this is simply that a time series whose segments don't fit a normal distribution is a non-random time series, but I'm wondering if there are methods that capture the significance of a time segment relative to #1-3 above, i.e., "how rare is this segment"?

You can run a statistical test like Augmented Dickey Fuller to check if the variable follows a random walk.
Use the adfuller function from statsmodels:

from statsmodels.tsa.stattools import adfuller

results= adfuller(df['<your_col>'])
print('p value:', results[1])
print(results)


Help with type of ML problem: when training data is spread across different subgroups/categories I've been searching around for a while without any luck - hopefully someone more knowledgable can give me some advice about the following ML problem that I've been thinking about:
Say you are trying to predict the Rotten Tomato "Tomatometer" score of a film before it's released. Typically you might approach this by compiling a list of features and labels for some existing films and input this into a supervised ML algorithm.
In this example, the feature list will be standard metrics that describe the film such as, budget, filming duration, number of actors, etc., while the label is the Tomatometer score of the film that is given as a value between 0 to 100. Every film can be expressed using this score, but individually they are spread across many genres, country of production, etc. meaning that there are natural subsets within the training data.
Let's say our training data contains only films belonging to five genres (e.g. Action, Thriller, Horror, Fantasy and Documentary), while we want our algorithm to be applicable to films outside of this genre (e.g. Sci-Fi or Animation), but for the sake of question, we do not have access to these entire categories. In this example, were also assuming that some features will be more important to certain genres than others, for example, a large cast size may correlate more with the score for Action films than for Animations.
What is the general way to transform the data to make it invariant to the subgroup (genre), or what ML algorithm can be used here (if any)? Is there a common name for this situation (some keywords I can search?)

I'll try to rephrase your question: How does one use the information contained in some categorical feature x to predict y in the presence of unseen category values in the test set?
Assuming the train set is representative of the test set distribution, you'd expect the large categories to also be present in the test set.
We're thus mostly concerned with the small categories which might be present in the train set but absent from the test set and/or present in the test set but absent from the train set.
One way of dealing with such situation is merging small categories (e.g. under 2% of all observations) into a single category. That way you treat any new category level as part of the merged category.
Below I share how I implement the above in python, in a way that can be combined in scikit learn pipelines:
from collections import Counter
import pandas as pd

class mergeSmallCategoryLevels():
    def __init__(self, min_frac):
        self.min_frac = min_frac

    def fit(self, X, y=None, **fit_params):
        category_counts = pd.DataFrame.from_dict(Counter(X), orient = "index", columns = ["count"])
        min_category_count = len(X)*self.min_frac
        large_categories = category_counts[category_counts["count"] >= min_category_count]
        self.large_categories = list(large_categories.index.values)
        return self

    def transform(self, X, **transform_params):
        ans = [val if val in self.large_categories else ".merged" for val in X]
        ans = pd.DataFrame({"category_feature":ans})
        return ans.to_numpy().reshape(-1, 1)


Multi-dimentional and multivariate Time-Series forecast (RNN/LSTM) Keras I have been trying to understand how to represent and shape data to make a multidimentional and multivariate time series forecast using Keras (or TensorFlow) but I am still very unclear after reading many blog posts/tutorials/documentation about how to present the data in the correct shape (most examples being of slightly less

My Dataset:

several cities
for which I have info about say temperature, car traffic, humidity
for say the last 2 years (one record for each day)

What I want to do:
I'd like to forecast for each city the temperatures I can expect for the next year using a possibly lagged version of temperature, car traffic and humidity (of course there would be several more features but this is just an example for thought).

What I am confused about:
If I have 2 cities, for which I recorded 3 features for 365 days. How should I shape my input so that the model can output a forecast for 365 days for these two cities (i.e. 2 time series of temperatures for 365 days)?

Intuitively the tensor shape would be (?, 365, 3) for 365 days and 3 features. But I'm not sure what to stick into the first dimension and, most importantly, I would be surprised if it had to be for the number of cities. But at the same time, I have no idea how to specify into the model that it has to understand the dimensions properly.

Any pointers will be helpful. I'm pretty familiar with the rest of the problem (i.e. how you build a network in Keras etc since I have done this for other neural networks but more specifically how best to encode the sequence for the desired input.)

Oh and also, I guess I could train and predict for each city independently, but I'm sure everyone will agree there are probably things to be learned that are not particular to any city but that can only be seen if considering several of them, hence why I think it is important to encode it in the model.

The input shape for an LSTM must be (num_samples, num_time_steps, num_features). In your example case, combining both cities as input, num_features will be 2x3=6.

If you lump all your 365 time steps into one sample, then the first dimension will be 1 - one single sample! You can also do sanity check by using the total number of data points. You have 2 cities, each with 365 time-steps and 3 features: 2x365x3= 2190 . This is obviously the same as 1x365x6 (as I said above) - so it would be a possibility (Keras will run) - but it obviously won't learn to generalise at all, only giving it one sample.

Have a look at this relevant question, which I recently answered. There I speak a little about using a rolling window (check the comments of the answer for more info). That will buy you more samples if you need them.

If you want to train a single model with data for both cities as input, then making predictions for both cities at each time-step is as simple as defining a final Dense layer, which outputs 2 units. Your validation/test data must then of course contain a tuple of (city1, city2).

A perhaps more sophisticated way to approach this would be to create datasets on a single-city basis, then train several sub-models on each city individually (say for 5 layers), then Merge/Concatenate them and put several further layers on top. This will mean you are combining the learnt features of each city, which are in turn being combined to a higher level of abstraction. Here is the first image I got from a search engine, which sketches the idea.


predicted labels or predicted probabilities in a ROC plot? What should be taken as a parameter to plot ROC curves , for example in a classification model, I can get predicted labels and predicted probabilities.
I have softmax layer in the output layer.

Should I use predicted labels or predicted probabilities to plot ROC curve in a classification problem.

Use the predicted labels.
Explanation - The ROC curve show possible classification performance at different setups.
Where 'classification performance' is a factor of True-positive-rate and False-positive-rate. And 'different setups' are thresholds used to determine a predicted label(class) based on predicted probability.
The latter is the reason why you should use predicted labels and not predicted probability.

Some additional context:
In my experience, ROC curve can be useful for three things:
1) Determining optimal setup(probability threshold) for a given model. Based on possible setups and problem needs.
2) Evaluating overall model performance and different models comparison. This can be achieved using the AUC.
3) Understanding classification model behavior and identifying possible abnormalities. For example, if the curve has a step like pattern its possible your model is too simple.

Here is a short nice explanation on the ROC curve.


Which features can help to differentiate these two density? I'm wondering that is there any features that can help in differentiating the following two images. I mean differentiating in related numbers.

Regarding two probability distributions $A,B$ on the line, that have densities, the following are equivalent:

$ A = B$ (i.e.: given any Borel set $\mathscr S$, the probability assign to $\mathscr S$ by A equals the probability assign to $\mathscr S$ by B)
$A(-\infty,a] = B(-\infty,a] $, for any $a \in \mathbb R$, i.e.: $A,B$ have the same cumulative distribution function
$A,B$ have the same densities.
In the case of your question, the two densities are clearly different, so the two probabilities are different.

From your question, it seems you want something you can measure (perhaps on sample from these distributions). There are infinitely many quantities that you can measure that when apply to $A$ and $B$ produce different results. The statement above gives some hints for a possible solution.

From your graphs, calling $A$ the first pdf, and $B$ the second:
$$ F_A(0.35) = A(-\infty, 0.35] = \text{ probability that an observation from the first distribution is < 0.35 } = 0 $$
since the density is 0 to the left of 0.35. On the other hand,
$$ F_B(0.35) = B(-\infty, 0.35] = \text{ probability that an observation from the second distribution is < 0.35 } > 0 $$
since the density is positive from -0.2 to 0.35.

We can a little to the right of 0.35 and find a number $a_0$ such that $0 < F_A(a_0) < F_B(a_0) $.

Being less than $a_0$ is something one can measure on samples.

Let $A_1, .., A_n$ be an iid sample from the first distribution, and let $B_1,...,B_m$ be an iid sample from the second distribution. Then $x_j = I(A_j \le a_0)$ is a measure (Note: I am using the notation $I(S)$ = indicator function of a set $S$, or, in more typical notation in the context of probabilitie, $I(S)$ = indicator function of event $S$ = 1 if $S$ is true, 0 if $S$ is false).

Likewise, one can measure $y_j = I(B_j \le a_0).$

$\{x_j\}$ are iid Bernoulli, with $P(x_j=1) = P(A_j \le a_0) = F_A(a_0).$
$ \frac{1}{n} \sum_1^n x_j $ is an estimator of $F_A(a_0)$

$\{y_j\}$ are iid Bernoulli, with $P(y_j=1) = P(B_j \le a_0) = F_B(a_0).$
$ \frac{1}{m} \sum_1^m y_j $ is an estimator of $F_B(a_0)$

Now one can run a rest of the hyposesis that the tw means are the same.

Before proceeding, does this construction answer your question?


What is "Laplacian image space"? I have been working through Beyond Face Rotation: Global and Local Perception GAN for Photorealistic and Identity Preserving Frontal View Synthesis where the authors make the following claim (subjection 2.2.1):

...the Laplacian space is more robust to illumination changes and more indicative for face structure.

what is Laplacian space in this context? How is it robust to illumination changes?

They should really be more clear about what they mean, but I expect they're using a Laplacian pyramid.  As more evidence, they cite: "Denton et al "Deep generative image models using a laplacian pyramid of adversarial networks."

The idea is, store a very low resolution copy of your image, and a series of "difference" images.  Each difference image tells you what to add to the lower resolution copy to get the next higher resolution version of the image.  You can imagine that lots of values of the difference will be close to zero.

The "mean intensity" and therefore "illumination" is only really stored at that lowest resolution copy, and doesn't really (usually, they hope) affect the gradient of the image, which is what the laplacian pyramid stores.  That's why those authors say it's not sensitive to illumination changes.  Does that make sense?


How is the Gaussian noise given to this BLSTM based GAN? In a conditional GAN, we give a random noise along with a label to the generator as input. In this paper, I don't understand why in one section of the paper, they say they are giving the random noise as input and the in another section of the paper they are saying it is concatenated to the output.
page 2

page 2 footnote

page 3 model setup section

little overview of the paper:  Code switching is a phenomenon in spoken language where we switch between two different languages. Mixed language models improve the accuracy of automatic speech recognition to higher degree but the problem is less availability of mixed language written sentences. Thus, as a data augmentation technique, a conditional GAN is developed to synthesize English, Mandarin mixed sentences from a pure Mandarin sentence. The trained generator acts as an agent telling which words in the Mandarin sentence have to be translated. It outputs a binary array (of length equal to input Mandarin sentence length). Both generator and discriminator are BLSTM networks.
#####EDIT: The author accepted that it is a typo, noise should be concatenated after the embedding layer not to the output of BLSTM
Author's reply:
It is a typo in page 3.
The noise is concatenated with the output of the embedding layer.
Thanks for your correction.
#####

As stated in 3.2 Model setup

The generator G is made up of embedding layer, one bidirectional  long  short-term  memory  (BLSTM)  [21]  layer,  one fully connected (FC) layer.

And

Gaussian noise is 10-dim vector concatenated with the output of BLSTM

So the noise is concatenated to the embeddings computed by the BLSTM for each time step. I think they concatenate the same 10-dim vector to each embedding output of the BLSTM but this is not clear.
The obtained vector (embedding concatenated with noise) is fed to the Fully-connected layer with softmax activation to compute the probability that the word should be translated or not.
Concerning page 2 footnote about "ignoring the noise"
I think they simply ignore it in their explanations of the model, but it is effectively used in the model by simply appending a gaussian noise vector to the embeddings output of the BLSTM.
As you see in the following figure they do not show the noise, but it is actually concatenated to the output embeddings of the BLSTM in the Generator.


What are the advantages of Skip-Gram methods? Concerning the notion of word embeddings, Skip-Gram methods aim for computing the probability of a word given its neighborhood. I do not understand the rationale behind it, since it is possible to infer this information by looking directly into the co-occurrence matrix.

In general, I cannot understand those methods aiming to capture as much relevant information from the original co-occurrence matrix as possible. Isn't it easier to work on the co-occurrence matrix directly?

Thank you very much in advance.

Basically, skip-gram aim for computing the probability of a context given a word (it is CBOW that does the opposite). BUT, it does not simply learn a co-occurrence matrix, it compresses the information in a low dimensional space (for example 300 for a 100000 original dimension). By doing this, it learns continuous low dimensional representation for the words.

It is proved that word2vec actually factorizes a word-context pointwise mutual information matrix (close to the co-occurrence matrix). I think this article might give you a better understanding of the underlying processes.


How to make sense of confusion matrix Consider a binary classification problem with 0 labels denoting normal and 1 abnormal or rare. The number of instances with 0 classes are more in comparison to 1. In general,

1) Does 0 always refer to positive or a negative depending on what we define as a positive and negative? What if the labels are reversed?

2) Is there a particular order that the confusion matrix if displayed? If the confusion matrix is given as:

1   4
0   5

I got this confusion matrix in Matlab. How do I know that the first row is for class 0 or for class 1?

Adding to the answer above,

The labeling totally depends on how you define it. You can define 0 as negative or as positive. However, for the sake of understanding and ease of readability, keep it meaningful.
The instances that are correctly predicted are given by the diagonal.
Here, '1' is True Negative or for the class labelled as 0 and '5' is True Positive or for the class labelled as 1.

If you find it difficult to interpret by a simple confusion matrix, you could plot it.
Check out plotConfusion by MathWorks.


Clustering, Mixed Data Set with Ordinal and Nominal Scale Data After reading a bit how categorical data can be considered in clustering, I came to the conclusion that most of the post do not make distinction between nominal scale data e.g. colour: red, green, blue, and ordinal scale data e.g. size: small, medium, large. However, the distances between the items make sense at the ordinal scale even if they are not necessarily the same between all items.

My questions:

Can I simply convert ordinal scale data to numeric scale without
causing much trouble in clustering? I think yes, for the above reasons, but I would be pleased if you could confirm.
For nominal scale data, where the distances between the items make
no sense, would be harder to capture. The easiest way I have found,
if there are not too many items on the scale, was to break down the
scale and add a variable for each item. E.g. originally, we have
colour: red, green, blue, and we make variable colour_red,
clour_green, and colour_blue, where each of them can take a value: 0
or 1. See the post form Jordan A on
K-Means clustering for mixed numeric and categorical data.
It seems to me a valid numeric scale of type ratio as it has a
non-arbitrary zero value expressing the complete absence of
something and 1 expressing the presence of something. Do you have
experience with this in clustering? Is this a valid approach?

Otherwise I know I should use e.g. kproto (Kproto) for mixed data sets or kmodes (Kmodes) for plain, nominal data sets. Thank you for you responses.

Yes, it causes almost no trouble. The only caution you must have is the possibility of having a regular ordinality where the ordinality is actually irregular. For example, having shirt size 1, 2 and 3 but the size "3" is not three times bigger than size "1".
The dummy variable creation is a very valid approach when having categorical variables in your dataset, not only for clustering but for almost every model you could construct.


Why is video classification still not that accurate? I was wondering, with the advent of deep learning, many tasks related to images have been solved to near human accuracy such as classification, object detection etc., however in videos, traditional methods still compare to deep learning methods and both are not as good. I wanted to understand what and how is it that with videos that understanding them makes it much harder.

I've used py-faster-rcnn on video feeds and found it to be pretty usable in a sunny environment. Although on a GTX660, I could only process one image every 600ms, which may highlight a limitation. It still takes a lot of time to compute an inference on a deep net, so for a minimally fast framerate, of 10fps, you're still limited to a speed of 100ms.

There might also be other artifacts due to camera dynamics such as image blurring, sensor saturation, or low light levels that might be more present in video feeds of natural scenes as opposed to the fairly well exposed images in static image datasets.


How to know the probability of correctness of a test data in a Binary Classifier I have written a sequential classifier script using Keras, Tensorflow. Its a binary image classifier that predicts the class, given the directory path of a sample image.

I want to implement a probability feature in my script, where including the class, the correctness probability of my classifier's prediction is given too. for eg:

INPUT

img = image.load_img('dataset/test/sample/fff.jpeg', target_size = (img_width, img_height))
img = image.img_to_array(img)
img = np.expand_dims(img, axis = 0)

prediction = model.predict(img)

OUTPUT (something like)

Prediction: 80% seems like it being Class A

In other words, I want to display my classifiers accuracy for that particular image.

P.S I am sorry if the question seems a little abrupt. I have tried asking it elsewhere too but either It gets closed due to lack of information, Or its misunderstood as "Classifier training accuracy" by google as its hard for me to frame it.

Accuracy is a statistic that you can compute on a dataset if you know the true labels. For a single image, the accuracy is either 0% or 100% based on if you get it right or wrong.

In the newer versions of Keras, the predict method returns the probabilities of the classes, what you want to print is (if I guess correctly), the probability score for the best scoring class.

Not that this is not an unbiased estimate of the class probability. Neural classifiers tend to very confident even if their prediction is incorrect.


How to understand the weights and biases for beginners? I am newbie to deep learning, I was building my first model using MNIST dataset, I understood the full  model, but one thing is a bit confusing to me. How can we get the weights and bias? Is it that, weights are our input data or is it going to get assigned some random values and also bias or some of the weights.

It works based on this formula

Z=Wh∗x+bh

Y = Wx + b

Here, I am assuming that the h is in subscript :

Y is the value that we want to predict
x is the input
b is the bias
W is the weight

Now, the question that you asked If I understood clearly :

  You want to know, How can you get the weight and Bias of your model ? And are the W input to your machine learning model or is it the neural networks task to find the weight for you ?

Weight W is the coefficient of the input x which when combined with bias b returns the predicted value Y. Note that weight W is the coefficient of the feature input x.

The sole aim to run a machine / deep learning algorithm is to find the best set of weights corresponding to each feature and the bias. The bias does not correspond to every weight w, It corresponds to each layer in the network.


Understanding Spikeslab Output I'm using spikeslab for the first time, but don't understand what the output means. It was suggested to me that I use it to tell which variables my dependent variable is most correlated to, in a ranked order.
Particuarly, what is "bma" bma.scale" "gnet" and "gnet.scale"? I also don't understand how to read the corresponding plot to the model.Thanks for any help!

For example, this is one of the models I created using spikeslab, with its output:

model2_ss <-spikeslab(Risk_Pct ~ Race
+                    +hispanic
+                    +Born_In_US
+                    +Highest_Education
+                    +Marital_Status
+                    , na.rm = TRUE, data = LabeledData)
> model2_ss
-------------------------------------------------------------------
Variable selection method     : AIC
Big p small n                 : FALSE
Screen variables              : FALSE
Fast processing               : TRUE
Sample size                   : 26
No. predictors                : 5
No. burn-in values            : 500
No. sampled values            : 500
Estimated mse                 : 0.4299
Model size                    : 3

---> Top variables:
                bma      gnet     bma.scale     gnet.scale
Marital_Status  0.516     0.516     0.319         0.319
Born_In_US      -0.469    -0.447    -0.440       -0.419
Race            0.458     0.421     0.926         0.851

BMA is "Bayesian Model Averaged". GNET is "Generalized Elastic Net".

Have you tried reading the Ishwaran and Rao papers as mentioned in the documentation for spikeslab?  There's an article in the R Journal as well that might be worth reading too: https://journal.r-project.org/archive/2010-2/RJournal_2010-2_Ishwaran~et~al.pdf - no sense duplicating it all here.


xgboost Predictions from R and Python don't match Thought maybe somebody here could help us solve a mystery (https://github.com/dmlc/xgboost/issues/1623):

We are trying to build a xgboost prediction function in R for a model that was trained in Python and the results don't match. See below for an example of how to reproduce.

Fairly new to xgboost, particularly using it across languages, so may be missing something obvious.

Steps to reproduce:

(1) Download this model file: http://ml.stat.purdue.edu/hafen/WTKG.model

(2) Run this script in R:

library(xgboost)
mod <- xgb.load("WTKG.model")
x <- c(91, 9, 9, NA, NA, 273, 20, 170, NA, NA, 14, 14, 0,
  2, 0.94289404091, 0.94289404091, 0.93087973569, 0.0120143052199997, 0.95490834613,
  0.95490834613, 1, 90, 0.95490834613, 1, 90,
  0.93087973569, 357, -266, 0.93087973569, 357, -266,
  0.95490834613, NA, 0.93087973569, NA, NA, NA)
d <- xgb.DMatrix(matrix(x, nrow = 1), missing = NA)
predict(mod, d)
# [1] 0.6483372

(3) Run this script in Python:

import numpy as np
import xgboost as xgb

bst = xgb.Booster({'nthread': 4})
bst.load_model('WTKG.model')
x = [91, 9, 9, np.nan, np.nan, 273, 20, 170, np.nan, np.nan, 14, 14, 0,
  2, 0.94289404091, 0.94289404091, 0.93087973569, 0.0120143052199997, 0.95490834613,
  0.95490834613, 1, 90, 0.95490834613, 1, 90,
  0.93087973569, 357, -266, 0.93087973569, 357, -266,
  0.95490834613, np.nan, 0.93087973569, np.nan, np.nan, np.nan]
d = xgb.DMatrix(data=[x], missing=np.nan)
bst.predict(d)[0]
# 1.3775804

Solution for posterity: In Python, xgboost requires np.array as input. So this line:

d = xgb.DMatrix(data=[x], missing=np.nan)

should be:

d = xgb.DMatrix(data=np.array([x]), missing=np.nan)


Why such a big difference in number between training error and validation error? Question

Why such a big difference between my 'Train loss' and 'Validation loss' as shown in the picture below? Is it a signal that my codes are wrong and my trained network is wrong as well?

Some of my codes are as follows:

DATA_SPLIT_PCT = 0.2

timesteps =  5
n_features =  20

epochs = 100
batch = 32
lr = 0.0001

lstm_autoencoder = Sequential([
    # Encoder
    LSTM(8, activation='relu', input_shape=(timesteps, n_features), return_sequences=True),
    LSTM(4, activation='relu', return_sequences=False),
    RepeatVector(timesteps),

    # Decoder
    LSTM(4, activation='relu', return_sequences=True),
    LSTM(8, activation='relu', return_sequences=True)
    TimeDistributed(Dense(n_features)),
])

adam = optimizers.Adam(lr)
lstm_autoencoder.compile(loss='mse', optimizer=adam)

for stock in stock_list:  # 500 stocks in stock_list
    lstm_autoencoder_history = lstm_autoencoder.fit(X_train_dict[ticker], X_train_dict[ticker],
                                                    epochs=epochs,
                                                    batch_size=batch,
                                                    validation_data=(X_valid_dict[ticker], X_valid_dict[ticker]),
                                                    verbose=False).history

plt.plot(lstm_autoencoder_history['loss'], linewidth=2, label='Train')
plt.plot(lstm_autoencoder_history['val_loss'], linewidth=2, label='Valid')
plt.show()

I used the for loop to feed my data into lstm_autoencoder network. In the dictionary variable stock_list, there are 500 stock names such as 'AAPL'.
I plotted lstm_autoencoder_history['loss'] and lstm_autoencoder_history['val_loss'] and it is weird because usually validation loss is higher than train loss.
I am curious to know why my plot has smaller amount of validation loss. For your information, I used Keras as my deep learning framework. And since I used Keras, I thought this library would handle the different proportion of training set size and validation set size by averaging the errors.

I think this is because your training set is much larger than your validation set. What this means is that for training; losses are being accumulated for comparatively larger number of examples than validation. Hence your training error is larger than your validation error.

I don't think its a problem with your code. If you see closely the trend line has almost same slope for both training loss and validation loss; this suggests that you are okay as far as code is concerned.


Opening a 20GB file for analysis with pandas I am currently trying to open a file with pandas and python for machine learning purposes it would be ideal for me to have them all in a DataFrame. Now The file is 18GB large and my RAM is 32 GB but I keep getting memory errors.

From your experience is it possible? If not do you know of a better way to go around this? (hive table? increase the size of my RAM to 64? create a database and access it from python)

If it's a csv file and you do not need to access all of the data at once when training your algorithm, you can read it in chunks. The pandas.read_csv method allows you to read a file in chunks like this:

import pandas as pd
for chunk in pd.read_csv(<filepath>, chunksize=<your_chunksize_here>)
    do_processing()
    train_algorithm()

Here is the method's documentation


When is it appropriate to split a dataset on a categorical value and generate $n$ models instead? When doing regression or classification when faced with a categorical attribute with $n$ possible values there are two options:

Feed this attribute directly into your model.
Partition your data into $n$ pieces based on the categorical attribute and train a model for each separately. During inference choose the model appropriately based on the same attribute.

One of the advantages of approach #2 is that it allows you to do more specific feature engineering. E.g. if you are modeling property prices and you decided to make separate models for residential/industrial properties you can choose separate features that are relevant for each.
Another advantage of approach #2 I can think of is that it can linearize otherwise non-linear relations. E.g. for a residential property having a railroad track nearby almost always heavily reduces property value while for an industrial property it could be a massive value booster.
In general, what factors go into deciding between approach #1 and #2?

I've tried 2 several times but it has never proved better than 1.
I think the reason is, the more data you feed to a model, the better. The disadvantage of 2 is that the models that are trained use less data than the model in 1.
In addition, some features might be independent of the group. For instance, when modelling property prices, being in the city centre always increases the price, both for residential and industrial.
Let me discuss two of the main models used for tabular data:

Tree based models will already do the feature engineering that you described in your first point. The model will already do a split residential/industrial if it contributes to the gain and then it will keep doing particular splits for each group.
Linear models: a generalization of general linear models are mixed models, that kind of does what you mentioned on the second point, but keeping some structure that allows it to acknowledge that the city center is more expensive.

That being said, if you have very different categories, it might be worth splitting the dataset, it's just a matter of trying.


Error with pandas dataframe (needs to be 1-dimensional) I am trying to determine the conformal predictions for my model with my data. But it gives me following error that occurs at  icp.calibrate(X_cal, y_cal) :
Exception: Data must be 1-dimensional

Below you can find the most recent traceback error about this. Unfortunately I am not sure on what this actually infers based on the code from above. I am using a pandas dataframe for this.
#Code Snippet
from sklearn.tree import DecisionTreeRegressor
from nonconformist.cp import IcpRegressor
from nonconformist.base import RegressorAdapter
from nonconformist.nc import RegressorNc, AbsErrorErrFunc, RegressorNormalizer, NcFactory
from sklearn.model_selection import train_test_split
import numpy as np
import pandas as pd

# -----------------------------------------------------------------------------
# Setup training, calibration and test data
# -----------------------------------------------------------------------------
df = pd.read_csv ("prepared_data.csv")

# Initial split into train/test data
train = df.loc[df['split']== 'train']
valid = df.loc[df['split']== 'valid']

# Proper Validation Set (Split the Validation set into features and target)
X_valid = valid.drop(['expression'], axis = 1)
y_valid = valid.drop(columns = ['new_host', 'split', 'sequence'])

# Create Training Set (Split the Training set into features and target)
X_train = valid.drop(['expression'], axis = 1)
y_train = valid.drop(columns = ['new_host', 'split', 'sequence'])

# Split Training set into further training set and calibration set
X_train, X_cal, y_train, y_cal = train_test_split(X_train, y_train, test_size =0.2)

# -----------------------------------------------------------------------------
# Train and calibrate underlying model
# -----------------------------------------------------------------------------
underlying_model = RegressorAdapter(DecisionTreeRegressor(min_samples_leaf=5))
print("Underlying model loaded")
model = RegressorAdapter(underlying_model)
nc = RegressorNc(model, AbsErrorErrFunc())

print("Nonconformity Function Applied")
icp = IcpRegressor(nc)  # Create an inductive conformal Regressor
print("ICP Regressor Created")

#Dataset Review
print('{} instances, {} features, {} classes'.format(y_train.size,
                                                   X_train.shape[1],
                                                   np.unique(y_train).size))

icp.fit(X_train, y_train)
icp.calibrate(X_cal, y_cal)

#Example Dataframe
new_host  split     sequence    expression
FALSE     train     AQVPYGVS    0.039267878
FALSE     train     ASVPYGVSI   0.039267878
FALSE     train     STNLYGSGR   0.261456561
FALSE     valid     NLYGSGLVR   0.265188519
FALSE     valid     SLGPSNLYG   0.419680588
FALSE     valid     ATSLGTTNG   0.145710993

I've tried splitting the dataset in various ways but I am continuing to have trouble with this. In this case I want to split the data into train and test sets according to an observation's Data Split value. After which, I will split the train set into train and calibration in a second step. Where myfeatures, X_train and my target, y_train

It seems to me that this question is better off on stackoverflow.
Nevertheless, X_cal gets generated from X_train and X_train from valid. But this is an atleast 2-dimensional dataframe with new_host and sequence. Like the error says you should only input data that is 1-dimensional.


How can you include information not present in an image for neural networks? I am training a CNN to identify objects in images (one label per image). However, I have additional information about these images that cannot be retrieved by looking at the image itself. In more detail, I'm talking about the physical location of this object. This information proved to be important when classifying these objects.

However, I can't think of a good solution to include this information in a image recognition model, since the CNN is classifying the object based on the pixel values and not on ordered feature data.

One possible solution I was thinking of was to have an additional simple ML model on tabular data (including mainly the location data), such as an SVM, to give give a certain additional weight to the output of the CNN. Would this be a good strategy? I can't seem to find anything in the literature about this.

Thanks in advance!

edit: Someone asked what I meant by 'location'. With the location I meant the physical location of where the image was taken, in context of a large 2d space. I don't want to go too deep into the domain, but it's basically an (x,y) vector on a surface area, and obviously this meta-data cannot be extracted by looking at the pixel values.

edit2: I want to propose an additional way I found that was useful, but was not mentioned in any answer.
Instead of using the neural network to predict the classes, I use the neural network to produce features.

I removed the final layer, which resulted in an output of shape 1024x1. This obviously depends on the design of your network. Then, I can use these features together with the meta-data (in my case location data) in an additional model to make predictions, such an SVM or another NN.

Other answers suggest to put an additional channel, I disagree. I think it's a very computationally intensive, time consuming process. Moreover, it forces non-pixel data to be processed by Conv filters, which doesn't make much sense IMHO.

I suggest you to establish a multi-input model. It would be composed by three parts:

A Convolutional part, to process pixel data,
A Feed-forward part to process non-image data,
Another Feed-forward part that elaborates the prediction based on the concatenation of the two outputs above.

You will need to instantiate them separately, then combine together in a Keras Model(). You will also need Concatenate() layers to combine the two different sources of data.

You can read more about the implementation of multi-input Neural Networks here.


How to visualize two small samples with different sizes? As part of a psychology experiment, I surveyed 15 users. I separated them in two groups, those who have become smoke free, and those who are still smoking. These groups have different sizes (9 and 6).

I want to show graphically that the smoke free group reported higher values of the survey index. This is my current best attempt, with Google Sheets:

However, I think that a correct comparison should have the same number of data points. I guess I need some form of interpolation, but not sure how to proceed.

An alternative approach is using a bubble chart:

My questions:
1. Would you recommend a different chart?
2. Should I have the same number of data points? If yes, how to achieve that?

By downloading the Free Wolfram Engine for Developers you may use Wolfram Language.  If you are using Python then use the Wolfram Client Library for Python to access the engine in Python.

You may use BoxWhiskerChart with Labeled to show the distribution of responses and label how may responses in each set.

I'll generate some responses for each of the two groups.

SeedRandom[123]
survey = MapThread[RandomVariate, {UniformDistribution /@ {{3, 5}, {2.5, 4}}, {6, 9}}]

{{3.91144, 4.95565, 4.88643, 4.92443, 3.6047, 3.93342},
 {2.59246, 3.07847, 3.14476, 3.66812, 2.57289, 3.4424, 2.91698, 2.63533, 3.81488}}

Then plot with a number of observations label located at the "UpperFence" of each set.

BoxWhiskerChart[
 Labeled[#, "Obs: " <> ToString@Length@#, "UpperFence"] & /@ survey,
 ChartLabels -> {"Smoke free", "Smoking"},
 ChartStyle -> {Orange, Purple},
 GridLines -> {None, Automatic}]

Hope this helps.


what does smooth/soft probablity mean? I was recently reading the Knowledge Distillation paper, and encountered the term smooth probabilities. The term was used to denote when the logits were divided a temperature.

  Neural networks typically produce class probabilities by using a
  softmax output layer that converts the logit, zi, computed for
  each class into a probability, qi, by comparing zi with the other
  logits where T is a temperature that is normally set to 1. Using a
  higher value for T produces a softer probability distribution over
  classes.

What does that mean intuitively?

When $T$ gets larger, i.e. $$T \rightarrow \infty $$ the probability distribution resembles the uniform distribution, so $q_i = \frac{1}{n}$. You can find the proof here, as well as some other interesting properties.

Given the previously mentioned fact, I guess he is referring to smoothing  in the sense that the noise if left out and what remains are the important patterns of the data, that is the distribution is made less 'spiky' as @Emre mentioned.


How to get high accuracy in CNNs? ML and Data Science world.

I am a newbie to CNNs, but do possess a basic understanding of ML and Neural Networks.

I wanted to create my own CNN that works on the Cats and Dogs Dataset. I preprocessed the data and built my network, but when I fit the model with the data, I am not able to get more than 55% accuracy, which means the model isn't learning anything.

Can anybody explain what I am doing wrong here?

model = Sequential()
model.add(Conv2D(filters=32, kernel_size=(3,3),padding='SAME', input_shape=X[0].shape))

model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2), dim_ordering='th'))

model.add(Conv2D(filters=64, kernel_size=(3,3), padding='SAME'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2), dim_ordering='th'))
model.add(Dropout(rate=0.4))

model.add(Conv2D(filters=128, kernel_size=(3,3), padding='SAME'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2), dim_ordering='th'))
model.add(Dropout(rate=0.35))

#model.add(Conv2D(filters=64, kernel_size=(3,3), padding='SAME'))
#model.add(Activation('relu'))
#model.add(MaxPooling2D(pool_size=(2,2), dim_ordering='th'))

model.add(Flatten())
model.add(Dense(1024))
model.add(Activation('relu'))
model.add(Dropout(rate=0.3))

model.add(Dense(2))
model.add(Activation('softmax'))

And this is the optimizer part:

opt = keras.optimizers.SGD(lr=0.0001, decay=0.0)
model.compile(optimizer=opt, loss='binary_crossentropy', metrics['accuracy'])
print(model.summary())

model.fit(X, np.array(Y), validation_data=(test_x, np.array(test_y)), epochs=30, verbose=2)

I've been stuck on this for the past 2 days.
Any and all help is appreciated.

Thanks.

I have a few suggestions to improve your accuracy (these are hardly original with me):

Use an ImageDataGenerator from keras.preprocessing.image to augment your data.
Use k-fold validation to use your training data more thoroughly.
Try the keras.optimizers.Adam() optimizer.


replace values based on Number of duplicate rows are occured I have a dataframe ,that looks like this
       site  Active
0     deals  Active
1     deals  Active
2     deals  Active
3  discount  Active
4  discount  Active

i don't want to drop the duplicate items, but i want to change the Active columns value based on Site column,for example Active  has to change inactive based on  duplicate item in site column,last duplicate item has to Active, other than that Inactive
Expected
       site    Active
0     deals  InActive
1     deals  InActive
2     deals    Active
3  discount  InActive
4  discount    Active

I would do this manually. First, let us create the index set of entries whose state must remain active. To do this, I iterate over all rows and record active instances. Note that the later occurrence overrides earlier ones, so we keep only the last one occurrence of active event.
last_active = dict()
for i, row in df.iterrows():
    if row['Active'] == 'Active':
        last_active[row['site']] = i
keep_active = last_active.values()

Now I assign the state 'Active' to those entries whose index is in keep_active and InActive otherwise.
df['refined_active'] = df.apply(lambda x: 'Active' if x.name in keep_active else 'InActive', axis=1)


Why we need to have the test set remains consistent across multiple runs? In the book "hands-on machine learning with scikit-learn and tensorflow: concepts, tools, and techniques to build intelligent systems" , more specifically in chapter 2 , the writer is teaching us how to create a test set. He mention that we need to keep the test set consistent across multiple runs . To do so he mentioned the following :

A common solution is to use each instance’s identifier to decide
whether or not it should go in the test set (assuming instances have a
unique and immutable identifier). For example, you could compute a
hash of each instance’s identifier and put that instance in the test
set if the hash is lower or equal to 20% of the maximum hash value.
This ensures that the test set will remain consistent across multiple
runs, even if you refresh the dataset. The new test set will contain
20% of the new instances, but it will not contain any instance that
was previously in the training set.
My question is why it is important to do so ? why we don't like to generate different test set for each run ?

You will judge the performance of the trained model based on certain performance metrics.
If you keep updating your test set, then you will not know whether one run is better than the other.
For e.g. you are trying to predict the value of house. And you have only one data point in your test set and you are using mean absolute error (MAE) as your performance criteria.
Scenario 1: the actual value of house is 100K, your prediction with first model is 101K. MAE = 1000
Scenario 2: the the actual value of house is 25K, your prediction with second model is 26K. MAE = 1000
In reality, prediction is better with model1 as compared to model2. But because you changed your test set, you will not be able to make that conclusion. Had you kept your test set same in scenario2, your prediction would have been much higher making MAE much higher, clearly indicating that model1 is better than model2.
Hope this helps.


How can we prove that an autoassociator network will continue to perform if we zero the diagonal elements of a weight matrix? How can we prove that an auto-associator network will continue to perform if we zero the diagonal elements of a weight matrix that has been determined by the Hebb rule? In other words, suppose that the weight matrix is determined from $W = PP^T- QI$, where $Q$ is the number of prototype vectors.

I have been given a hint: show that the prototype vectors continue to be eigenvectors of the new weight matrix.

This is a question from Neural Network Design (2nd Edition) book by
Martin T. Hagan, Howard B. Demuth, Mark H. Beale, Orlando De Jesus .

Resource : E7.5 p 224-225

As we have an autoassociative network prototype vectors are both input and output vectors. So , we have that :
\begin{equation}
    T = P
\end{equation}

\begin{equation}
    W = PP^T - QI = TP^T - QI = \sum_{q=1}^Q p_q p_q^T - QI
\end{equation}

Applying a prototype vector as input :

\begin{equation}
    \alpha = W \cdot p_k = \sum_{q=1}^Q p_k p_q p_q^T - QIp_k
\end{equation}

Because they are orthogonal we have :
\begin{equation}
    \alpha = p_k(p_k^T\cdot p_k) - Q \cdot I \cdot p_k = p_k (p_k^T\cdot p_k - Q \cdot I ) = p_k (\textbf{R - Q $\cdot$ I})
\end{equation}

where $R-Q\cdot I$ is the length of vectors

So since ,

\begin{equation}
    W \cdot p_k= (R - Q \cdot I) \cdot p _k
\end{equation}
prototype vectors
continue to be eigenvectors of the new weight matrix .

It is often the case that for auto-associative nets, the diagonal weights (those which connect an input component to the corresponding output component) are set to 0. There are papers that say this helps learning. Setting these weights to zero may improve the net's
ability to generalize or may increase the biological
plausibility of the net. In addition, it is necessary if we use iterations (iterative nets) or the delta rule is used


How to Normalise features for small datasets? I am working with a small dataset ( N  = 50 ).
I would like to normalise  my input features.
I am facing the following issues:

Because of the small size of the dataset the range of training input features would differ from the range of testing input features.
The input features do not have a theoretical upper bound.

Can you suggest normalisation techniques suitable for this task? Any paper suggestions would also be appreciated.

You can use a MinMaxScaler on your train set, which will normalize your features inside [0, 1]. The same scaler can transform the test set and if there are values greater than the ones found in the train set, the scaler handles that by returning values greater than 1. Essentially, your test set will be normalized.


Methodologies for predicting missing data I have the following problem: I'm searching for methods to predict randomly missing data in a given dataset.

For example: I have a dataset which contains information about a person. This can be gender, weight, age, height etc. Let's assume there are information missing of age and height for a specific person. How can this information be predicted based on the information I have in my dataset?

I have read about PCA-methods, but I would like to get an overview of pros and cons of methodologies and also an overview of recent research (good working algorithms for the given dataset/where to start reading for developing an algorithm and solving the given problem).

As told by @smci, this technique is called Data Imputation. There are several techniques which can be used to deal with the missing data. Some of these are:

Mean/ Mode/ Median Imputation: Imputation is a method to fill in the missing values with estimated ones. The objective is to employ known relationships that can be identified in the valid values of the data set to assist in estimating the missing values. Mean / Mode / Median imputation is one of the most frequently used methods. It consists of replacing the missing data for a given attribute by the mean or median (quantitative attribute) or mode (qualitative attribute) of all known values of that variable. This can further be classified as generalized and similar case imputation.
Prediction Model:  Prediction model is one of the sophisticated method for handling missing data. Here, we create a predictive model to estimate values that will substitute the missing data.  In this case, we divide our data set into two sets: One set with no missing values for the variable and another one with missing values. First data set become training data set of the model while second data set with missing values is test data set and variable with missing values is treated as target variable. Next, we create a model to predict target variable based on other attributes of the training data set and populate missing values of test data set.
KNN(k-nearest neighbor) Imputation: In this method of imputation, the missing values of an attribute are imputed using the given number of attributes that are most similar to the attribute whose values are missing. The similarity of two attributes is determined using a distance function.


The larger an encoding dimension in NLP the better? Is the longer encoding array (word2vec or any other kind) always more precise than the shorter (not regarding the computational cost, only the performance)?

No, it isn't that way. From my personal experience, I have built custom word2vec's to some datasets and observed that at times, shorter vectors did better than longer ones. I think you can try shorters length vectors for smaller datasets. I cannot tell any metric in specific for comparison of dataset size and word2vec length though.


Numpy arithmetic operation between two columns Below is the numpy array. I need to perform two operations on this array.

Add one column with value [column 1] - [column 3].
Add another column with value [column 1] - [previous value of column1].

I can do this using normal list operations, but is it possible to use numpy or pandas? If so, how can it be done?

Input data:

[['78' '3412' '98' '3441']
 ['106' '3412' '127' '3434']
 ['139' '3411' '160' '3434']
 ['170' '3411' '191' '3442']
]

These types of operations can easily be done using both numpy or pandas. However, in this case I would recommend pandas since it is more intuitive. Using the example array we can create a pandas dataframe:

arr = np.array([[78, 3412, 98, 3441], [106, 3412, 127, 3434], [139, 3411, 160, 3434], [170, 3411, 191, 3442]])
df = pd.DataFrame(arr, columns=['a', 'b', 'c', 'd'])

The two new columns can now be added as follows:

df['e'] = df['a'] - df['c']
df['f'] = df['a'].diff(1)

Directly using numpy, one possible way would be to do:

arr = np.c_[ arr, arr[:,0] - arr[:,2], np.append(np.NaN, arr[1:, 0] - arr[:-1, 0]) ]


How to identify outliers from a small list of numbers? I want to identify outliers from a very small group of numbers. How shall I do that?

For example, for the group of numbers: -0.4, 0.4, 52.1, actually 52.1 is an outlier.

I've tried using interquartile range to identify the outliers, but it won't identify 52.1 as the outlier.

An "outlier" is an observation that is so unexpected that we suspect it wasn't valid -- corrupted by noise or something. But what is unexpected? An observation that is highly improbable? But then how do we know what's probable?

Unless you're able or willing to make some assumptions about the distribution that generated these numbers, you can't really declare things outliers. For example, quartiles don't help, since it will only help you find the x% largest or smallest values. But every data set has these -- indeed every data set has a min and max, and being the min or max doesn't mean being an outlier.

In practice, I sense there's an assumption lurking here, that the numbers are probably normally distributed about some mean with roughly some standard deviation. If you know what the mean and/or stdev is supposed to be, you can use it directly to decide how unlikely an observation is and discard it as an outlier if it exceeds a threshold you choose.

You can take the mean and stdev of this sample as a surrogate for that, and for a large enough sample, the sample mean and stdev could be close enough to the real population mean and stdev to work. Here, with such small sets, the outliers influence stats like the mean so much that they throw off attempts to evaluate them in terms of the stats they influence. It's kind of circular.


Would a neural network trained on extracted features have the same accuracy as a full network with frozen layers? Let's say that I train two neural networks on the exact same dataset.

The first network is a VGG19 model with frozen convolutional layers so only the top dense layers are getting optimized.
The second network is composed only of dense layers, with exactly the same architecture and hyper-parameters as the first networks fully connected layers.

Basically, I pass all my data through the convolutional layers of VGG19, saving outputs of the last convolutional layer to disk. Then, I load the data into my second model which is only composed of FC blocks and I train it to be a classifier.
Would combining the two models after training be the same as training a whole VGG19 model (with frozen conv layers) with my own FC layers?
If I understand correctly, the outputs of a trained network are entirely deterministic so in theory it should work. In practice, I have no idea if optimization algorithm takes frozen layers into account when training only the last FC layers.

When layers are "frozen" it generally means that their weights are not updated when backpropagation happens. So, technically, there is no difference between:

Using a "frozen" VGG16 and training some fully connected layers and
Using the VGG16 embeddings and training some fully connected layers

In practice, if you did both and compared the results, you may see differences if you haven't accounted for the fully connected layers' random initialisation.


Handling variable output units in MLP? I am using Keras to build my architecture.

The regression problem I am trying to solve has outputs different for different training samples.

Suppose that I have very first two rows of y_train = [[16, 3], [6], ... ] corresponding to my very first two rows of X_train (input data). I would like to assign the output of the dense layer's unit to length of these y_train rows.

For example, for 1st training sample, I would like model.add(dense(2)) as y_train[0] has length 2, for 2nd training sample, I would like model.add(dense(1)) as y_train[0] has length 1, and so on.

I also think that I can call Lambda Layer instead of output dense layer and in that Lambda Layer I can wrap length of every row of y_train and assign it as output units by using output_shape argument of Lambda Layer. But, I don't know how can I access my y_train at training time of my model so that I can use y_train inside lambda layer to make this scenario happen either, nonetheless.

Can anybody help me on this variable length output problem?

If you want to use arbitrary length input and outputs, there are two common options: train a single model for each length or use a recurrent neural network (RNN) architecture. A non-RNN neural network cannot handle arbitrary length because the shape of the weights depend on the length of the input. From your example, it seems like you want to keep different copies of weights based on the length. You could do that, but in theory, it is the same as training separate models. The weights from one length will only be updated by inputs of the same length, so why not save yourself the hassle and have separate models?

Although RNNs are typically used for natural language processing and understanding, it does not mean they cannot be used to solve other deep learning problems. Specifically for your problem, you should use a many-to-many architecture.

Source: http://karpathy.github.io/2015/05/21/rnn-effectiveness/

In Keras, you can use the LSTM layer and set the parameter return_sequences to True, which will return an output the same length as your input sequence. Documentation here: https://keras.io/layers/recurrent/#lstm.


How to merge two CNN deep learning model using weighted sum and weighted product in Keras? I am using Keras to create a deep learning model and I want to merge two CNNs by using weighted sum or weighted product.

How can I merge two CNNs using weighted sum and weighted product?

I think the most elegant way is to write a layer that does this. For example for the case of the weighted sum:

class WeightedSum(Layer):

    def __init__(self, a, **kwargs):
        self.a = a  # "weight" of the weighted sum
        super(WeightedSum, self).__init__(**kwargs)

    def call(self, model_outputs):
        return self.a * model_outputs[0] + (1 - self.a) * model_outputs[1]

    def compute_output_shape(self, input_shape):
        return input_shape[0]

Suppose you have two models model1 and model2 with outputs out1 and out2 respectively. This layer simply performs the operation:

$$
out = a \cdot out_1 + (1-a) \cdot out_2
$$

You can compute a weighted product the same way, just change the call method.

Example

from keras.layers import Layer, Input, Dense
from keras.models import Model
import keras.backend as K
import tensorflow as tf

# Define the custom layer
class WeightedSum(Layer):
    def __init__(self, a, **kwargs):
        self.a = a
        super(WeightedSum, self).__init__(**kwargs)
    def call(self, model_outputs):
        return self.a * model_outputs[0] + (1 - self.a) * model_outputs[1]
    def compute_output_shape(self, input_shape):
        return input_shape[0]

# Create model1
inp1 = Input((5,))
d1 = Dense(100)(inp1)
out1 = Dense(10)(d1)
model1 = Model(inp1, out1)

# Create model2
inp2 = Input((7,))
d2 = Dense(70)(inp2)
out2 = Dense(10)(d2)
model2 = Model(inp2, out2)

# Weighed sum of the two models' outputs with a = 0.1
out = WeightedSum(0.1)([model1.output, model2.output])

# Create the merged model
model = Model(inputs=[inp1, inp2], outputs=[out])

Let's check the summary:

>>> model.summary()
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to
==================================================================================================
input_5 (InputLayer)            (None, 5)            0
__________________________________________________________________________________________________
input_6 (InputLayer)            (None, 7)            0
__________________________________________________________________________________________________
dense_9 (Dense)                 (None, 100)          600         input_5[0][0]
__________________________________________________________________________________________________
dense_11 (Dense)                (None, 70)           560         input_6[0][0]
__________________________________________________________________________________________________
dense_10 (Dense)                (None, 10)           1010        dense_9[0][0]
__________________________________________________________________________________________________
dense_12 (Dense)                (None, 10)           710         dense_11[0][0]
__________________________________________________________________________________________________
weighted_sum_10 (WeightedSum)   (None, 10)           0           dense_10[0][0]
                                                                 dense_12[0][0]
==================================================================================================
Total params: 2,880
Trainable params: 2,880
Non-trainable params: 0
__________________________________________________________________________________________________

Let's see if it works:

import numpy as np

a = np.random.random(size=(32, 5))  # input for model1 (batch size 32)
b = np.random.random(size=(32, 7))  # input for model2 (batch size 32)

pred = model.predict([a, b])

Let's see if it has the right shape:

>>> model.shape
(32, 10)

Let's see if it's the correct thing:

# Generate model outputs manually:
o1 = model1.predict(a)  # model1's output for array a
o2 = model2.predict(b)  # model2's output for array b

# Compute their weighted sum manually:
o = 0.1 * o1 + 0.9 * o2

Now if we're correct, o should be equal to pred:

>>> np.array_equal(o, pred)
True


sklearn CountVectorizer token_pattern -- skip token if pattern match I apologize if this question is misplaced -- I'm not sure if this is more of a re question or a CountVectorizer question. I'm trying to exclude any would be token that has one or more numbers in it.

>>> from sklearn.feature_extraction.text import CountVectorizer
>>> import pandas as pd
>>> docs = ['this is some text', '0000th', 'aaa more 0stuff0', 'blahblah923']
>>> vec = CountVectorizer()
>>> X = vec.fit_transform(docs)
>>> pd.DataFrame(X.toarray(), columns=vec.get_feature_names())
   0000th  0stuff0  aaa  blahblah923  is  more  some  text  this
0       0        0    0            0   1     0     1     1     1
1       1        0    0            0   0     0     0     0     0
2       0        1    1            0   0     1     0     0     0
3       0        0    0            1   0     0     0     0     0

What I want instead is this:

   aaa  is  more  some  text  this
0    0   1     0     1     1     1
1    0   0     0     0     0     0
2    1   0     1     0     0     0
3    0   0     0     0     0     0

My thought was to use CountVectorizer's token_pattern argument to supply a regex string that will match anything except one or more numbers:

>>> vec = CountVectorizer(token_pattern=r'[^0-9]+')

but the result includes the surrounding text matched by the negated class:

   aaa more   blahblah  stuff  th  this is some text
0          0         0      0   0                  1
1          0         0      0   1                  0
2          1         0      1   0                  0
3          0         1      0   0                  0

Also, replacing the default pattern (?u)\b\w\w+\b obviously messes with the tokenizer's normal function which I want to preserver.

What I really want is to use the normal token_pattern, but apply a secondary screening of those tokens to only include those that have strictly letters in them. How can this be done?

Found this SO post which says to use the following regex:

\b[^\d\W]+\b/g

yielding the following:

>>> vec = CountVectorizer(token_pattern=r'\b[^\d\W]+\b')
>>> X = vec.fit_transform(docs)
>>> pd.DataFrame(X.toarray(), columns=vec.get_feature_names())
   aaa  is  more  some  text  this
0    0   1     0     1     1     1
1    0   0     0     0     0     0
2    1   0     1     0     0     0
3    0   0     0     0     0     0

What I needed in my regex were the \b word boundary characters of which I was not aware of. That does make this a misplaced question as it has nothing to do with data science or that discipline's tools (sklearn).


WGAN-GP slow critic training time I am implementing WGAN-GP using Tensorflow 2.0, but each training iteration of the critic is very slow (about 4 secs on my CPU, and somehow 9 secs on Colab GPU).

Is WGAN-GP usually this slow or there is a flaw in my code?

Here's my code for training the critic:

def train_critic(self, X_real, batch_size, gp_loss_factor, optimizer):
    y_real = np.ones((batch_size, 1))

    # Get batch of generated images
    noise = np.random.normal(0, 1, (batch_size, self.z_dim))
    X_fake = self.gen.predict(noise)
    y_fake = -np.ones((batch_size, 1))

    X = np.vstack((X_real, X_fake))
    y = np.concatenate((y_real, y_fake))

    # Interpolate images
    alpha = np.random.uniform(size=(batch_size, 1, 1, 1))
    X_interpolated = alpha * X_real + (1 - alpha) * X_fake
    X_interpolated = tf.constant(X_interpolated, dtype=tf.float32)

    # Perform weight update
    with tf.GradientTape() as outer_tape:
        # Calculate gradient penalty loss
        with tf.GradientTape() as inner_tape:
            inner_tape.watch(X_interpolated)
            y_interpolated = self.critic(X_interpolated)
        gradients = inner_tape.gradient(y_interpolated, X_interpolated)
        norm = tf.sqrt(
            1e-8 + tf.reduce_sum(tf.square(gradients), axis=[1, 2, 3]))
        gp_loss = gp_loss_factor * tf.reduce_mean((norm - 1.) ** 2)

        # Calculate Wasserstein loss
        y_pred = self.critic(X)
        wasserstein_loss = wasserstein(y, y_pred)

        # Add two losses
        loss = tf.add_n([wasserstein_loss, gp_loss] + self.critic.losses)
    gradients = outer_tape.gradient(loss, self.critic.trainable_variables)

    optimizer.apply_gradients(zip(gradients, self.critic.trainable_variables))

    return wasserstein_loss, gp_loss

def wasserstein(y_true, y_pred):
    return tf.reduce_mean(y_true * y_pred)

This part calculates a second derivative. it should be slow and time consuming.

# Perform weight update
with tf.GradientTape() as outer_tape:
    # Calculate gradient penalty loss
    with tf.GradientTape() as inner_tape:

Try

# Perform weight update
with tf.GradientTape() as outer_tape, tf.GradientTape() as inner_tape:


Is Pearson coefficient a good indicator of dependency between variables? Once I have been asked how would I calculate correlation between two time series. Since I am new to data science I answered: "I would just calculate the Pearson correlation coefficient". That wasn't a good answer since as demonstrated in the figure below, dependency between two variables may not be linear and the Pearson coefficient can be close to $0$ (parabola, circle). I have seen people on Kaggle always starting with correlation matrix and discarding data that are not correlated.

My question is: is Pearson coefficient always a good measure of correlation between variables and we should always rely on it? Can you give practical example when some variable in your problem was important, but Pearson indicated there is hardly any correlation?

As you show, Pearson coefficient is clearly not a good measure of how variables depend on each other. A better measure is distance correlation. A nice property of distance correlation is that distance correlation 0 implies independence. An easy example that can happen in real life is that a variable is the square of another variable, as you show in your first picture. In this case, the Pearson correlation will be 0, but the distance correlation will be around 0.5. I think it is a mistake for Kagglers to choose variables in terms of Pearson correlations.


From Labels to Graph: what machine learning approaches to use? Imagine a world where we have places (e.g., cities, restaurants, national parks, etc.) but no roads connecting them.
Our objective is to build roads connecting any two places while going through some given places. For instance, connect Seattle to San Francisco and make sure the connection goes through restaurants X and Y and goes through the Red Wood national park. Note, there is not necessarily one possible road, hence multiple options as output are also acceptable.
We have some roads as examples to train the system. For instance, we have a road from Washington DC to Boston that goes through Baltimore.
So, the input to a machine learning system will be a list of strings (e.g., city names and places to go through) and the expected output is a tree/graph that connects the given points.
(Note, this cannot be a graph traversing problem, as a path the connects the points does not exist.)
I want a machine learning model that can produce/generate a novel graph in the output given a set of labels.
Any thoughts on what ML machine learning methods I can use and where to start?

Update 1
Note that I have neither of the following:

A graph of route(s) from Seatle to San Francisco;
A larger graph that contains a route from Seatle to San Francisco.

Instead, I have graphs for routes from Los Angles to Texas (with stops for eating and seeing landscape), from Milan to Rome, and etc.
Therefore, I think this is not a graph searching problem.
Update 2
I take that the initial question was not clear, so I reworded it.

Your graph is characterized by a set of nodes and edges. It means given a start point x, show me the next node z1, then from z1 show me z2 and so on. This looks like the definition of graph-searching algorithm.
However, maybe by definining a 'cost function' that you want to optimize (minimize or maximize) you can convert it to a supervised learning task. Basically, given a graph composed of weighted nodes and edges, you want to choose a subset (E,V) that minimzes the cost function.
I am not sure this problem fits a generative problem (as you emphasize on the word generate) whose goal is to learn a distribution.


Python : How to use Multinomial Logistic Regression using SKlearn I have a test dataset and train dataset as below. I have provided sample data with min records, but my data has more than 1000's of record. Here if you see E is my target variable which I need to predict using an algorithm. It has only four categories like 1, 2, 3, 4. It can take only any of these values.

Training Dataset:

A    B    C    D    E
1    20   30   1    1
2    22   12   33   2
3    45   65   77   3
12   43   55   65   4
11   25   30   1    1
22   23   19   31   2
31   41   11   70   3
1    48   23   60   4

Test Dataset:

A    B    C    D    E
11   21   12   11
1    2    3    4
5    6    7    8
99   87   65   34
11   21   24   12

Since E has only 4 categories, I thought of predicting this using Multinomial Logistic Regression (1 vs Rest Logic). I am trying to implement it using Python.

I know the logic that we need to set these targets in a variable and use an algorithm to predict any of these values:

output = [1,2,3,4]

But I am stuck at a point on how to use it using python (sklearn) to loop through these values and what algorithm should I use to predict the output values? Any help would be greatly appreciated.

Put the training data into two numpy arrays:

import numpy as np

# data from columns A - D
Xtrain = np.array([[1,    20,   30,   1],
                   [2,    22,   12,   33],
                   [3,    45,   65,   77],
                   [12,   43,   55,   65],
                   [11,   25,   30,   1],
                   [22,   23,   19,   31],
                   [31,   41,   11,   70],
                   [1,    48,   23,   60]])

# data from column E
ytrain = np.array([1, 2, 3, 4, 1, 2, 3, 4])

Then train a logistic regression model:

from sklearn.linear_model import LogisticRegression

lr = LogisticRegression().fit(Xtrain, ytrain)

Make predictions (on the training data):

yhat = lr.predict(Xtrain)

=> results in "1, 4, 3, 4, 1, 2, 3, 4".. so it's got 7 right and 1 wrong.

Calculate accuracy:

from sklearn.metrics import accuracy_score

accuracy_score(ytrain, yhat)

=> results in 87.5% accuracy

To make predictions for new data, just create another numpy array containing your test data and call lr.predict on it.

You might also want to look into parameter tuning to improve your score. For example the LogisticRegression class has some parameters that control regularization - tuning them with methods found in sklearn.grid_search might improve your score.


Mean estimation for nested location data I want to estimate the average income for a location. I have nested data in the following way: A block is inside a neighborhood, which is inside a zipcode, which is inside a district, which is inside a region, which is inside a state.

I want to estimate the average income at a block level, and the issue is that I don't have much data at that level. I have much more data at a state level, but it is not such a good approximation.

How would you deal with this problem? Are there any ways to incorporate the uncertainty of not having many data points at a block level? Are there any Bayesian frameworks that allow us to incorporate data of all levels? Is it possible that mixed models are able to do so?

If you explain any method, if you can provide a python package where that method is built, it'll be great!

Thanks!

I don't know if that is the case, but if some kind of continuity assumptions are realistic, you could try to move away from categorical variables (block) to continuous variables (longitude and latitude). Then, if you have information on two neighboring blocks, you could interpolate those values with say a spline.

Of course, this can also be fitted into a machine learning model with predictors such as average income of blocks with distance < x.
And if you don't have data of nearby blocks, then your state average might be the next best approximation.

Your state level data can serve as a predictor and also as validation.

Also, plotting your data always helps get some kind of intuition.


Does SQL Server support the Poisson distribution? I am trying to do some work that is primarily based in SQL Server. I cannot seem to find a native function to SQL that supports the Poisson distribution. Has anyone had success with applying Poisson natively within SQL?

Yes, SQL Server does support it, as a part of Data Analysis Expressions (DAX).

The documentation for the function for getting Poisson distribution data is here.


Stability of clusters in a unsupervised machine learning I am new to Unsupervised learning.  I am working on a customer segmentation data (with no labels). I have done K-Means and also calculated the silhouette score for the model. Now I want to study, if the model is good or not (similarity or stability for clusters) using Adjusted Rand Index. Can anyone give an idea how to do it, as I have have only the cluster labels which I believe are 'labels'. In the formula of RAND score, we need the "labels_true, labels" what should be the first one. I have checked on the internet, but not much is available. By the way, I am using Python. Thank you.

From scikit-learn document Link-

Given the knowledge of the ground truth class assignments labels_true and our clustering algorithm assignments of the same samples labels_pred, the (adjusted or unadjusted) Rand index is a function that measures the similarity of the two assignments

labels_true = [0, 0, 0, 1, 1, 1]
labels_pred = [0, 0, 1, 1, 2, 2]
metrics.rand_score(labels_true, labels_pred)

2.3.10.1.2. Drawbacks

Contrary to inertia, the (adjusted or unadjusted) Rand index requires knowledge of the ground truth classes which is almost never available in practice or requires manual assignment by human annotators (as in the supervised learning setting).


loss function of autoencoders I am trying to understand the loss function of autoencoders.
I think the loss function of the decoder is to compare the input image with the decoded image, tell me if I am correct.
but I don't understand what is being compared to find the encoder error.
Someone explains to me that it compares to find the encoder error. and decoder.

Yes, the loss of a normal autoencoder is simply the difference between the input image and the decoded image.
While encoder and decoder have different names, they are effectively part of the same neural network, so the prediction error is backpropagated through the decoder up to the encoder.
In more sophisticated autoencoders, like variational autoencoders, the loss may present extra terms, e.g. to enforce some structure in the output of the encoder.


Could the Input shape of the LSTM layer not be a constant? I am trying a vanilla LSTM on my dataset.
According to this course; the LSTM layer should be built as follows:
model.add(LSTM(50, activation='relu', input_shape=(n_steps, n_features)))

Key in the definition is the shape of the input; that is what the model expects as input for each sample in terms of the number of time steps and the number of features.

In my case, it's a univariate regression also, with "multi-step" though...
Instead of some constant n_steps, I have a different "step" for each Input.
eg:
...
X                                                                                y
[2006-10-11 10:00:00  0.295046,2006-10-21 10:00:00  0.325147,2006-11-11 10:00:00  0.361293] [0,666506515]
[2006-12-01 10:00:00  0.249307,2006-12-11 10:00:00  0.444077] [1,885262136]
[2009-05-11 14:00:00  0.570017,2009-05-21 14:00:00  0.525082,2009-06-01 14:00:00  0.475404,2009-06-11 14:00:00  0.390616] [1,998273253]

Let's assume that fsplit is the list containing the different steps:
What I've tried so far is doing this:
model.add(LSTM(50, activation='relu', input_shape=([i for i in fsplit], 1)))
but it doesn't work of couse
Thanks in advance :)

That line doesn't work because the architecture of ANNs must be kept constant, in order to train them. If you vary the number of input cells, you change the architecture of the ANN itself.

If your input sequences are not of constant length, I suggest you to use zero padding. This is the most common and used padding technique. In other words, you "complete" the input series with zeros at the end, so that they all have the same length. A properly defined Neural Network would then learn to ignore the zeros and capture only the relvant information.

You can read more about padding here.


Replacing the feature variables with encoded variables I have several nominal variables which I've encoded using the LabelEncoder() function. Now I want to replace the encoded values in the place of the raw datas of the features in the dataframe. I tried df.replace() but no luck. Below is the code snippet.

nominal_values = [
    'HouseStyle','ExterQual','BsmtQual','BsmtExposure','BsmtFinType1',
    'KitchenQual','GarageType','GarageFinish','GarageQual','GarageCond']

One of the column "KitchenQual" has values as follows:

{kitchenqual :['Ex','Gd','TA','Fa','Po']} representing "Excellent","Good","Typical/Average","Fair","Poor" respectively.

Another Feature "BsmtFinType1" has values as follows:

{BsmtFinType1: ['GLQ','ALQ','BLQ','Rec','LwQ','Unf','NA']} represeting

"Good Living Quarters","Average Liv. Quat.","Below Avg.Liv.Quat,","Avg. rec room", "Low Quality", "Unfinished","No Basement" respectively.

from sklearn.preprocessing import LabelEncoder

lbe = LabelEncoder()
for noms in nominal_values:
    encode = lbe.fit_transform(cat_var[noms])
    cat_var.replace(to_replace=cat_var[noms],
                    value=pd.Series(encode),inplace=True)

I used pd.Series(encode) since the replace function will support only one of the following data structures

Scalar
dict
Series

I don't know what is wrong with your code but this worked fine for me-

cat_var.replace(to_replace=cat_var[noms].tolist(),value=encode,inplace=True)

Instead of converting encoded values to series, convert series to list using tolist().


I would suggest using a Pandas DataFrame (you didn't mention it in your question).

Then it would be as simple as the following

Create the dataframe from you list x, calling the single column x:

In [1]: import pandas as pd
In [2]: df = pd.DataFrame(x, columns=["x"])   # x is defined in your question

Add a new column (I call it action), which holds your result. As it will be either -1 or +1, I fill it all with +1 to begin with, then only change the values to -1 where your criteria is met:

In [3]: df["action"] = 1

In [4]: df.head()
Out[4]:
         x  action
0  2352.60     1
1  2410.26     1
2  2443.31     1
3  2525.78     1
4  2506.58     1

Use the diff() method to find the difference between each row of x. If the difference is less that zero, we can make action equal to -1.

In [5]: df.action[df.x.diff() < 0] = -1

In [6]: df
Out[6]:
          x  action
0   2352.60       1
1   2410.26       1
2   2443.31       1
3   2525.78       1
4   2506.58      -1
5   2530.69       1
6   2530.49      -1

If you want to shift the difference up one row, because of your rule:

  Action [ i ] = (-1) if x[ i+1 ] < x[ i ]

Then you simply include that in the assignment part, filling the action column from the start index to the penultimate element:

In [7]: df.action[:-1] = df.action[1:]

In [8]: df.head()
Out[8]:
          x  action
0   2352.60     1.0
1   2410.26     1.0
2   2443.31     1.0
3   2525.78    -1.0
4   2506.58     1.0
5   2530.69    -1.0

Note that the last row will contain a value that could be wrong (you cannot know, without knowing the next value of x).

After edit from OP

If you want to do it in normal lists, you can do this:

In [9]: action = []

In [10]: for i, val in enumerate(x):
     ...:    if i == (len(x) - 1): break
     ...:    if x[i+1] < val:
     ...:        action.append(-1)
     ...:    else:
     ...:        action.append(1)

In [11]: action[:10]         # Look at the first ten elements
Out[11]: [1, 1, 1, -1, 1, -1, 1, 1, -1, -1]

enumerate iterates through the iterable you give it (here, x) and also returns the index of the current element.  We need the check that we do not exceed the length of x, because we check i + 1 each time, so could try to check a value one element too far in the list x.


Training accuracy is ~97% but validation accuracy is stuck at ~40%. What does it imply? Training accuracy is ~97% but validation accuracy is stuck at ~40%.

I can not understand the meaning of two concepts and their relationship.

A little more context about the data set and type of model would help, but most likely your model is overfitting to the training data. This means that your model is picking up noise from the training data and has basically "memorized" the data it has seen.

Therefore, the model is not generalizable, and this results in relatively poor performance on a data set it hasn't seen, explaining the 40% validation accuracy.


Using different timesteps for training data and target value I would like to know whether it's wrong; when working with time series data; to use daily prices as features and the price after 3 days as target.

Is this correct or should I use the next-day price as target and after training; predict 3 times; each time for one more day ahead(using the predicted value as a new feature)

Will these 2 approaches give similar results?

is it ok to use daily prices?

You can use the daily prices as time series to forecast an arbitrary horizon.

  LSTM vs NN

In short, vanilla NN have no memory of previous inputs whereas LSTM NN have. This is particularly useful in your problem because LSTM can learn the temporal (sequential) characteristics of your dataset and therefore forecast several steps ahead. LSTM RNN can learn the dynamics of your problem (vanilla NN cannot). A useful Python/Keras tutorial for your problem can be found here.

  Regression vs Classification

In short, classification is the problem of discriminating a sample into discrete categories. Regression refers to using past values of a signal (or more) to estimate future values of the same (or other) signals. In your case, regression is what you ask for.


I have 3 graphs of a binary Logistic Regression that I want to understand better what is happening and learn of a strategy to make the model better My problem is the following: I have a binary Logistic Regression model with a very imbalanced dataset that outputs the percentage of the prediction. As can be seen in the images, as the threshold is increased there's a certain point it stops predicting.
I am researching calibration techniques to try and make it work better but I thought maybe I could get some direction here.

I've tried giving weights to the classes but it didn't seem to get much better.

Is it a probability calibration problem?

The three graphs below are shown in no particular order.

Thanks in advance.

Given a confusion matrix:

            predicted
            (+)   (-)
            ---------
       (+) | TP | FN |
actual      ---------
       (-) | FP | TN |
            ---------

we know that:

Precision = TP / (TP + FP)
Recall = TP / (TP + FN)

  As can be seen in the images, as the threshold is increased there's a
  certain point it stops predicting

Thats not true, it stops predicting accurately only one class. What is udnerstandable because you moved threshold to make all your predictions on the other class.

Dont optimise this yourself, for example random forest will do this for you imiplicitly, (determining cut-off level) or just do hyperparameter optimisation youself.


Computing weights in batch gradient descent I have a reasonably large set of images that I want to classify using a neural network. I can't fil them all into memory at once, so I decided to process them in batches of 200. I'm using an cross-entropy cost function with a minimization algorithm from numpy.

My question is: is it correct to pass learned weights between batches and use them as a starting point of the minimization? Would this eventually cause my hypothesis to fit all the data, or will each iteration simply re-fit the weights for itself? What is the general approach to such a problem?

What you need to use is mini-batch gradient descent or stochastic gradient descent. You will need to shuffle your samples and make draws from it of the batch size you are aiming for, also you will have to make sure all of the samples in your data are included which will constitute of one epoch. Train for a few epochs depending on the data. Here is a good blog post on mini-batch blog post.