Keyword localization in audio file I want to build a model that can localize occurrences of a particular word in an audio file. For example, I want to find the word "pizza" in a ~5min recording. The program should return an array with (start, stop) objects describing the start and stop boundaries of that word in the file.

Can I just use classic object localization with some kind of CNN, where the object is the wanted word in the spectrogram? If I could: how would I need to  prepare the data for training—records with the word "pizza" and the same number of other words or more other words?

Is there perhaps a better method for word-searching in recordings?

The problem you are describing is known as wake word detection or trigger word detection.

I'm sure you could use a CNN to classify a chunked Mel-spectrogram of your audio (see also librosa). As training labels you would simply use 0 for timestamps with no wake word (no "pizza") and 1 for timestamps with the wake word. Alternatively to classifying all timestamps of one chunk, you could also train for just the the center frame of each spectrogram chunk (makes things easier). In any case, you will have to make sure that your dataset is at least mildly balanced, i.e., you'll have to have enough wake word and non-wake word instances. One way to achieve this, is to overlay recordings of background noise with recordings of wake and non-wake words. There are some tutorials that detail how to do this, e.g. this YouTube video, this article or this GitHub repo. Note that all these approaches use RNNs for the task. However, it has been argued by Bia et al. that a temporal convolutional network (TCN) architecture (in essence a CNN skip connections and dilation) may work equally well or better for such tasks as the one you describe and is probably easier to train.

Hopefully this answer will give you some points to start from.


How to implement multi class classifier for a set of sentences? I'm currently involved in an aspect level sentiment analysis project and using Stanford CoreNLP toolkit to implement the system. My knowledge on these concepts is very limited and I'm seeking your help to clarify some things related to both machine learning and classification.

I have a set of sentences, which are travel reviews that needs to be categorized under different labels (AMBIANCE, SCENERY, COST, ENTERTAINMENT etc). This is done by checking if certain aspect terms related to the aforementioned categories are found in the sentences.

Now, I want to train a classifier (I'm using a Stanford classifier), to classify these sentences to respective categories and I have a train data set containing around 3000+ sentences.

My problem is, a sentence may contain aspect terms that belong to not just one, but multiple aspect categories. In that case, I want the classifier to classify them all at once.

For example:

Review Sentence:

  On the upside it was very **calm** there and good for **swimming**

Categories

  AMBIANCE, ENTERTAINMENT

I have tried to pre-process the training dataset in this manner, and trained a classifier. But when I tried to make it classify a sentence containing aspect terms belonging to two different categories, it only identified one category.

Training dataset was of the following format

sentence1    [tab]   category1
sentence2    [tab]   category1, category2
sentence3    [tab]   category2, category3

Can someone please tell me if my approach is wrong? How can I achieve the desired output? I would greatly appreciate any help on this matter, as I'm currently stuck at this phase of my project.

I was thinking if Apriori might be more suited for your purpose.

For your consideration:
1) Tokenize the training sentences into bag of words:
Review Sentence | Upside | Calm | Swimming | Cat

2) Tag the correct consequents for bag of words.

3) Apriori should produce 1 rule for ambiance and 1 rule for Entertainment.

Hope this helps.


Normal distribution and Random Forest I have big table in dataframe (600k rows) which has y column (the variable I want to predict) and other 4 other columns that are the X.
I have run RF regressor  and  I got score of 0.87 when I run it on the train and test.
However, when I tried to predict another set of data (which is very similar, with 1M rows) I got score of 0.65.
So I assumed that is overfitting.
when I tried to understand why it hapenns, I went back to the distribution of the y column, which looks like this:

my question is, can it be that because that my data does not have normal distribution (or very skewd...) my model preformance is bad? Do all variables need to have normal distribution? how does the score of the random forest regrssion is calculated? id value is 0.25 and  predict is 0.26 does it count as correct prediction?

If you use tree-based algorithms like random forests the data distribution should not be an issue. Linear algorithms are more dependent on the distribution of your variables. To check if you overfit can try to predict your training data and compare the result with test data. The score depends on your evaluation metric. If you use scikit-learn you get R^2 as your metric.

The coefficient R^2 is defined as (1 - u/v), where u is the residual
sum of squares ((y_true - y_pred) ** 2).sum() and v is the total sum
of squares ((y_true - y_true.mean()) ** 2).sum().


Gradient Boosted Trees or Neural Networks Using Model Averaging? I am working on a certain insurance claims related data-set to classify newly acquired customers as either claim or non-claim.

The basic problem with the training set is the extremely large imbalance in claim and non-claim profiles, with the claims amounting to just ~ 0.26% of the training set. Also, most claims are concentrated largely towards the final few years (data is sorted by date).

On applying Logistic Regression or even Random Forests, to train on 70% of the data, the test results were well below satisfactory.

I've been looking at alternate models and I came across this blog post. A particular line that got my attention is:

  GBM is better than rf_t. In the paper, the best classifier for two-class data sets was avNNet_t, with 83.0% accuracy

Although, no real clarification was given as to why that was. Can someone help me open this "blackbox"? Which model really works (in the case described above) and why?

I believe in your case, predicting claim is more important than no claim. As you said you have you got 70% Accuracy on the training data, most of the time you might be doing wrong predictions in claim case because of less records, comparatively, what I would suggest is to make the data set balance or select a random balanced data set (20% each of claim and non-clan) and train a model using previous techniques you have applied and test it on the remaining data. If possible use different error measures with respect to your business case such as giving weights to the outcomes. If the accuracy is not improved, you can implement GBM techniques on this data. Most of the times GBM makes better predictions because it increases the randomness (white noise) in residuals by decreasing the similarity among residuals. You can apply many different models on this data and check if the accuracy is improved, eventually we should be able to understand the model to explain someone why they should use this model. Moreover, if you use feature engineered data with different models, there is a high probability that you will do better than the previous models. However, this depends on your business understanding.


Sampling labeled data for anomaly detection I'm currently working on a project that requires the use of unsupervised anomaly detection, but I'm unable to find a relevant data set, so I'm considering the following option:

Assuming I have a data set $X$ of $m$ examples labeled using $K$ classes. Let $X(k)$ be the subset of $X$ where all examples are labeled as $k$, and $k_{max}$ be the larget class.
Can I use $X(k_{max})$ as a training set for an anomaly detector, whose task is to flag elements who weren't labeled as $k_{max}$, as an anomaly?
Using $p << [m - size(X(k))]$ of the remaining examples in $X$ for cv and test sets as the anomalous examples.

I guess it depends on how you define an anomaly. If you already know and are sure that everything other than k_max can be defined as an anomaly then sure what you mention makes sense, assuming the fact that there is not a significant overlap between k_max and other classes in your feature space. I would train an auto encoder and learn an error function, classify anything that doesn't fit the error bounds as anomalous.

If you don't know what should be considered an anomaly, I would suggest looking at the Isolation Forrest Algorithm (https://cs.nju.edu.cn/zhouzh/zhouzh.files/publication/icdm08b.pdf) which is a tree based method used for outlier detection. This would allow you to use an unsupervised (untagged with classes) dataset and figure out what should be considered as a 'clean/healthy' example for your anomaly detection algorithm and the outliers can be used as anomalous examples.

Here is another article that you might find useful, trying to do a similar thing I am trying to suggest here: https://towardsdatascience.com/anomaly-detection-for-dummies-15f148e559c1


How to develop small workable embeddings for debugging Processing language data with deep learning models often involves a lookup of a pre-trained embedding model. In the model development phase, it's very annoying that every time the entire embeddings are loaded, as embeddings could be very big (e.g., Glove) and would consume a lot of timing loading it.

Is there a way to build/find a smaller and workable embedding just for debugging?

Right now what I've done is shrink the dimensions of Glove, for example, reduce the dimension of embeddings from 300d to 30d by taking the first 30 dims, but that might potentially induce a risk that some of words will share the same embedding.

The embeddings are floating point vectors, it's very unlikely that they are exactly the same. And even if they were, it will only minorly impact the performance of your model, which is not what you are after anyway, because you are debugging. That said, if you want to use it for model selection you have a few alternatives. The easiest one is to apply PCA one time to your whole embedding space and take the first n dimensions. This way you reduce your dimensionality without throwing away too much information. A better but more difficult approach would be to train your own Glove vectors, either on a public corpus or a private one, and set a low dimensionality during the training procedure. It might be difficult to use your findings on these embeddings for model selection because more expressiveness could lead to much better performance with deeper models or could lead to more overfitting.


Subtraction of Positive and Negative Frequencies in Sentiment Analysis In the Positive Negative Sentiment Analysis,  Would it make sense mathematically to instead of keeping a score of the positive frequencies and negative frequencies of a word, calculate the difference between them? That way each word would have a positivity 'heat' in which a very high value would indicate a very positive word and vice-versa. How this approach would change the model performance?

Let's take an example, consider two words A & B. A's positive/negative values are +1/0 and B's are +0.5/-0.5. Their difference would appear equivalent (diffAB = 1). When in fact they are quite different sentiments.
If you wanted to compute a single metric, you could do something like "polarity" i.e. take the squared sum of the values which would give you a positive value showing how "polar" the word is. In the case above, the word B would have a lower score because it's positive/negative values are less extreme.
When designing metrics, test a few scenarios to see if it matches your needs.


What is 'parameter convergence'? I'm trying to teach myself data science, with my particular interest being decision trees. A few steps in, I've come across a term, 'parameter convergence' that I can't find a definition for (because, after all, I'm learning on my own and have no access to teachers or peers):

  However, even in studies with much lower numbers of predictor
  variables, the combination of all main and interaction effects of
  interest – especially in the case of categorical predictor variables –
  may well lead to cell counts too sparse for parameter convergence. (from Strobl et al., 2009)

A web search isn't overly helpful because convergence is such a common term, and I'm not sure which results specifically apply in the context of decision trees. And also, the results don't provide an entry-level definition.

So, while a definition or explanation of parameter convergence (in the context of recursive partitioning) would be great, it would also be handy to be directed to a resource (academic or otherwise) that might have a 'glossary' of this and similar terms...

A naive definition of Parameter convergence is when the weights or the values of the parameters reach a point asymptotically. What I mean is that, when your model training is not altering the parameter values(maybe less than epsilon-small values) it might be a good fit. For decision trees, I found this paper which explains rate of convergence and more. It might be a good read if you want to get more details.


Classification - Divide the interval (0 - 1] to lets say 100 classes and use each class to make a calculation class-1 represents 0.01, class-i represents 0.01*i, class-100 represents 1.00.

Thus, when the classifier predicts the class-y and it should have predicted class-(y+1) there is a small error so we can accept class-y.

Is there a way to express this behaviour in a neural network? Maybe with a distribution or something?

PS: Not interested in regression.

Correct me if I am wrong but If I understand your question correctly, what you want is a classifier such that classes close to each other (say class 2 and class 3) are preferable to those far away (class 2 and class 99). If this is the case, this problem is called "Ordinal Categorical Classification".

I was working on a similar problem a while ago, and I found this loss function during my research. I ended up not using it so I don't really know how good it works but anyways, hope that helps.


Using TSNE to Visualize Clusters in Python I'm using TSNE to visualize my clusters but the output seems a bit strange. There are supposed to be 3 clusters but instead, there are 4 lines. Is there something wrong with how I'm visualizing them or is it the kmeans method itself?

import pandas as pd
import numpy as np
import ast
from sklearn import metrics
from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
from sklearn.manifold import TSNE
from sklearn.decomposition import TruncatedSVD

colNames = ['unixTime', 'sampleAmount','Time','samplingRate', 'Data']

data = pd.read_csv("project_fan.csv",  sep = ';', error_bad_lines = False, names = colNames)

# changing data into list
data['Data'] = data.Data.transform(ast.literal_eval)

# Selecting the average value from the list and replacing the list with it
data['Data'] = data.Data.apply(np.mean)

kmeanModel = KMeans(n_clusters = 3)
kmeanModel.fit(data)

y = kmeanModel.labels_

X_train, X_test, y_train, y_test = train_test_split(data, y, test_size = 0.2, random_state = 1)

k = 3
tfs_reduced = TruncatedSVD(n_components=k, random_state=0).fit_transform(data)
tfs_embedded = TSNE(n_components=2, perplexity=40, verbose=2).fit_transform(tfs_reduced)
fig = plt.figure(figsize = (10, 10))
ax = plt.axes()
plt.scatter(tfs_embedded[:, 0], tfs_embedded[:, 1], marker = "x", c = km.labels_)
plt.show()

Sample Dataset:

       unixTime  sampleAmount  Time  samplingRate         Data
0  1.556891e+09         16384   340  48188.235294  1620.242170
1  1.556891e+09         16384   341  48046.920821  1620.237716
2  1.556891e+09         16384   340  48188.235294  1620.236340
3  1.556891e+09         16384   340  48188.235294  1620.229289
4  1.556891e+09         16384   340  48188.235294  1620.227541

Output:

As far as i see, you have your temporal columns involved. They are sequential variables and create such curves in visualization.

Try to exclude them from your data and visualize again. If you still the curves, update the post with new plot and we discuss further.


YOLO algorithm - understanding training data I am taking "Convolutional Neural Networks" on Coursera and it is taught by Andrew Ng. I am in week 3 and confused about YOLO algorithm. I checked the course forums on coursera but I am still not clear and it seems that many people are confused about the same

what is the training data? all the images and the location of
objects in it? so for an image, if there is one car and one man in
it then the input would be vector representation of the image,
location of the centers (x,y), height and width of 2 objects and
object code of what is contained in each box. So in total vector
representation of the image + 4 coordinates of the 2 centers + 4
height and width of 2 boxes + 2 object codes - one for man and one
for car. Right?
what is the test data? vector representation of test images only? The course talks about splitting image into ss grid.
I initially thought that a small square from ss grid is fed
as an input. But i dont think that is the case. The entire image
must be fed. is that correct?
How a square from s*s grid is used?
does size of training image and test image has to be the same? what
they mean by "The car detection dataset has 720x1280 images, which
we've pre-processed into 608x608 images." in week 3 assignment?

update 1------------------------------------------

if a train image has 2 objects and maximum classes is 3 then  then
would input be vector representation of the image + input vector of
8*2?
the length of the input vector would change based upon objects in an individual image?
i am still not clear how ss grid is used. if there is a big car in
the middle of the image and YOLO looks at a small square from ss
grid from the center of the image then it is impossible to detect
that there is a car. We have to provide bigger square from center to
YOLO so that it understands there is a car. So what is the use of
feeding a small square?
How do we feed a bigger square?
is the process of feeding square from s*s grid occurs after multiple
convolution layers (so that basically a smaller square represent a
much larger area)?
do you have an example of an image and its shrink version? is their
some data science around shrinking?

--------------------update 2

i read the answer and looked at videos again and still not clear

2.the length of the input vector would change based upon objects in an individual image? for example if image1 has 5 objects then length of the input is going to be much longer than an image which has only 1 object. How do we feed such kind of data where input is not of fix width. Do we find an image with with maximum objects and decided the length of the input and for rest of images we just pad 0s (to make input vector of the same length)?

I am still  confused about ss grid. I initially asked
How a square from s*s grid is used?

after reading the below answer, my updated question is
i am still not clear how ss grid is used. if there is a big car in the middle of the image and YOLO looks at a small square from ss grid from the center of the image then it is impossible to detect that there is a car. We have to provide bigger square from center to YOLO so that it understands there is a car. So what is the use of feeding a small square?

the answer (in the comment of the orignal answer) says that
3: A grid cell doesn't contain a whole bounding box, but only the mid point of a bounding box.

my confusion is :
as per earlier discussion, we dont feed a grid cell individually. We feed the entire image once. so what is the point of creating the ss grid. If the image is looked only once then how the algorithm detects say two objects - one big car and one small car in a single go? We create the grid and the grid is used only to find midpoint of the object. But then the entire object is identified. I am still not clear with this part.

i feel that i am not the only one who is having hard time understanding YOLO. I saw multiple threads in the comment section of the course asking similar questions and I would appreciate patience and guidance

For each bounding box you need

p_c: any object / no object (background)
b_x, b_y, b_w, b_h: x, y, width and height of the bounding box
c_i: object i / no object i

For e.g. 2 bounding boxes and 3 classes (e.g. car, person, traffic light) your input vector would look as follows (the superscript in brackets denote the index of the bounding boxes)

\begin{bmatrix}
p_{c}^{(1)}\\
b_{x}^{(1)}\\
b_{y}^{(1)}\\
b_{w}^{(1)}\\
b_{h}^{(1)}\\
c_{1}^{(1)}\\
c_{2}^{(1)}\\
c_{3}^{(1)}\\
p_{c}^{(2)}\\
b_{x}^{(2)}\\
b_{y}^{(2)}\\
b_{w}^{(2)}\\
b_{h}^{(2)}\\
c_{1}^{(2)}\\
c_{2}^{(2)}\\
c_{3}^{(2)}\\
\end{bmatrix}

The whole image is fed into the model. That is essentially why YOLO is so fast. It looks at the whole image only once.
This is done by the CNN. Basically each portion of a convolution corresponds to a grid cell. For example the upper right cell in an image would correspond to the upper right part of the filters in each layer. This is visualized at the left of this image:

Yes they have to have the same size. That's what most CNNs expect. All images in the training set must have equal sizes and so do the images of the test set.
The image gets shrinked and deformed into a square of size 608x608


Minimize absolute values of errors instead of squares Calculating absolute values is much more efficient than calculating squares. Is there any advantage, then, to using the latter as a cost function over the former? Squares are easier to treat analytically, but in practice that doesn't matter.

Both the square and the absolute value should work okay for gradient descent but the square will work better.  For calculus based methods, the absolute value method may be intractable.

If you are using gradient descent methods then the square works very nicely in that it tends to form a U shape which takes big steps away from the minimum and smaller steps close to the minimum. Conversely, absolute values tend to form V shapes which take roughly the same step size. The result is that squares have better convergence properties.

Now, calculus based methods tend to take the derivative of a function and set it equal to zero.  Notice that U shapes tend to have well defined derivatives equal to zero at the minima, which works perfectly. By contrast, the V shape has an undefined derivative at its minima which tend to make conjugate gradient methods un-invertible.

Hope this helps!


How does joint probability generate new data in generative model Learning about different Machine Learning concepts, I came across Generative and Discriminative model. To infer from what I have studied, generative model is based on P(x,y)(Joint probability distribution) whereas discriminative model is based on P(y|x) which is conditional probability. But the question is why; infact HOW? How does joint probability give rise to new data meanwhile conditional probability just works on current dataset?
Can anyone explain me with an intuitive example or maybe some link which would elucidate the concept ?

The difference between discriminative and generative models is a common question which has some very good answers on Cross Validated, in particular here and here.

  But the question is why; infact HOW? How does joint probability give rise to new data meanwhile conditional probability just works on current dataset?

The word "generative" in generative models doesn't mean that the model generates actual new data additionally to the dataset. It refers to the nature of the theoretical model, in the sense that the generative approach assumes that any sample of data is generated from some distribution, and it tries to estimate this distribution. Once the distribution is estimated the model could be used to actually generate instances following this distribution, but:

usually that's not the goal, the goal typically being to predict the probability of a new instance using the model (inference)
normally this data should not be reused for estimation (i.e. training), since it is artificial data. Additionally it would be pointless since the best that can be achieved is to re-estimate the same model.
naturally the model obtained from estimation is only as good as the assumptions made for its design, and there are usually many such assumptions.


Isn't (steps_per_epoch = total training data/batch size)? Suppose i have 1000 dog images and my batch size is 10.
It will take 1000/10=100 steps to complete 1 epoch.
So doesn't it mean steps_per_epoch=100 ?
Then why do we have to specify it separately in keras while applying .fit().

As clearly mentioned in the documentation:
Steps_per_epoch is total number of steps (batches of samples) before declaring one epoch finished and starting the next epoch. The default None is equal to the number of samples in your dataset divided by the batch size, or 1 if that cannot be determined. It is a optional parameter and is useful when passing an infinitely repeating dataset.


what could it be a good descriptor for simple greyscale symbols? I need to find a good method to extract key points from drawing symbols like elements in CAD drawings (eg in 2D objects for the bathroom that can be placed on a floor plan).

I've already tried all type available in openCV/accord (FAST, SURF, Harris, ..) but all of them failed to give me any point.

The images are blank and white (with some sort of grey scale) and I'd like to find the position of a limited set of symbols inside bigger images containing other unknown objects/shapes (that I'm not interested in anyway).

The basic idea is to get the descriptors for my symbols and check if the target image contains similar descriptors.

If I have a good method for descriptors then I can train a classifier to check all the descriptors from the input images and so I can also know the position related to the recognized object.

Framework can be any with preference on OPENCV, Accord.net, tensorflow

If you can take a quick look with photoshop and get information on the shade of gray and zoom into the objects and count a pixel pattern.

You would be able to build an algorithm around the color 0-255 in 8-bit grayscale and create a simple matrix for each symbol based on the number of pixels you counted.

 110, 0,   0,   0,   0
 110, 0,   0,   0,   0
 0,  110, 110, 110,  0
 0,   0,   110, 110, 0
 0,   0,   110, 110, 0

You can see a shape emerge from the matrix...that is what you train on, but you do so by having the script read pixel by pixel and create a matrix.

This would be a gray (110) on black (0)  shape. If you only have a few, you could go into photoshop and cut the symbols out at the exact shape and feed the shape into your model and train it to look for that specific color and shape.

You might need to build an equation to scale it up or down so that it could account for using the identifier at different sizes or on files with different pixel depth.

you could also do this in R or python by reading a file in and localizing that data, but it would be easier to bring in exactly the shapes as separate files.
You might also need to give the model some leeway as to the darkness like from 100-120 depending on how the images enter the system.

If they are digital and unmodified straight out of CAD this is likely unnecessary, but if they are scanned or grabbed online, you will not have consistent exact numerical color-depth and need to account for the variation in your algorithm.


How to gauge overfit with MLPClassifier and cross_val_score? I'm learning sklearn.

When using MLPClassifier.fit() and MLPClassifier.predict() I would do a manual validation (looking for overfit) by running the training set again through the prediction and accuracy (as follows)...

from sklearn.neural_network import MLPClassifier
from sklearn.metrics import accuracy_score
from sklearn.model_selection import cross_val_score

target = ["target"]
features = [c for c in shuf_df.columns if c not in target]
neurons =[8, 16, 32, 64, 128, 256]
for activation in ["logistic", "relu"]:
    for neuron in neurons:
        mlp = MLPClassifier(hidden_layer_sizes=(neuron,), activation=activation)
        start = datetime.now()
        mlp.fit(train_df[features], train_df[target])
        nn_predictions = mlp.predict(test_df[features])
        accuracy_test = accuracy_score(test_df["target"], nn_predictions)
        nn_predictions = mlp.predict(train_df[features])
        accuracy_train = accuracy_score(train_df["target"], nn_predictions)
        print("({}) [{}] neuron:{}, accuracy_test:{}, accuracy_train:{}".format(datetime.now() - start,
                                                                                activation,
                                                                                neuron,
                                                                                accuracy_test,
                                                                                accuracy_train
                                                                                )
        )

...which results in...

(0:00:00.750631) [logistic] neuron:8, accuracy_test:0.8611111111111112, accuracy_train:0.9700765483646486 # Probably OK
(0:00:00.860471) [logistic] neuron:16, accuracy_test:0.8916666666666667, accuracy_train:0.9930410577592206 # Approaching overfit
(0:00:01.491433) [logistic] neuron:32, accuracy_test:0.8972222222222223, accuracy_train:0.9993041057759221 # Probably overfit
(0:00:01.951523) [logistic] neuron:64, accuracy_test:0.9166666666666666, accuracy_train:1.0 # overfit
(0:00:02.449780) [logistic] neuron:128, accuracy_test:0.925, accuracy_train:1.0 # overfit
(0:00:03.304685) [logistic] neuron:256, accuracy_test:0.925, accuracy_train:1.0 # overfit
(0:00:00.846773) [relu] neuron:8, accuracy_test:0.8583333333333333, accuracy_train:0.9436325678496869 # Probably OK
(0:00:00.905262) [relu] neuron:16, accuracy_test:0.8777777777777778, accuracy_train:0.9986082115518441 # Probably overfit
(0:00:01.531930) [relu] neuron:32, accuracy_test:0.8972222222222223, accuracy_train:1.0 # overfit
(0:00:01.695193) [relu] neuron:64, accuracy_test:0.9083333333333333, accuracy_train:1.0 # overfit
(0:00:01.503808) [relu] neuron:128, accuracy_test:0.9027777777777778, accuracy_train:1.0 # overfit
(0:00:02.060312) [relu] neuron:256, accuracy_test:0.9194444444444444, accuracy_train:1.0 # overfit

How would I determine overfit when using cross_val_score? (as follows)...

target = ["target"]
features = [c for c in shuf_df.columns if c not in target]
neurons =[8, 16, 32, 64, 128, 256]
for activation in ["logistic", "relu"]:
    for neuron in neurons:
        mlp = MLPClassifier(hidden_layer_sizes=(neuron,), activation=activation)
        cv_scores = cross_val_score(mlp, shuf_df[features], shuf_df[target], cv=4)
        print("[{}] neuron:{}, cv_scores:{}".format(activation,
                                                    neuron,
                                                    cv_scores,
                                                    )
        )

If I understand your question correctly, you would like the algorithm to detect overfitting by itself, without you manual inspection.

For this you need another split of data. So first you split your entire dataset into training and test, and then you split again your training into two subsets: the actual training and validation. You use these two subsets internally during the learning process to estimate accuracy and detect overfitting (and you may use cross_val_score() for this purpose if you want, just do not touch the holdout test part).

On a side note, looking at your two loops (for activation and neuron), you may want to use Grid Search for this.


When is precision more important over recall? Can anyone give me some examples where precision is important and some examples where recall is important?

For rare cancer data modeling, anything that doesn't account for false-negatives is a crime. Recall is a better measure than precision.
For YouTube recommendations, false-negatives is less of a concern. Precision is better here.


CrossValidation using glmnet and very high values of Lambda? I am trying to run crossvalidation (folds=10) using glmnet library on my dataset. My outcome of interest is BMI and predictors include a set of clinical variables. My final goal is to use elastic-net regression to select features and also predict BMI. For crossvalidation, I am using a range of alpha from 0 to 1 using a 0.1 increment. I am using min CVM to decide the values of lambda, with my current code I get extremely high estimates for lambda. Are higher values of lambda acceptable or I am failing to see here? Below is my code snippet.  I appreciate all your help and comments.

size <- floor(nrow(DataFile) * 0.7)
Train_rows <- sample(rownames(DataFile),size=size,replace=FALSE)
Train_Data <- DataFile[Train_rows,]
Train_bmi <- phenoFile[Train_rows,]$BMI

####### Cross Validation Alpha and Lambda #####

myAlpha <-  seq(0,1,by=0.1)

findAlpha_lambda <- function(iAlpha){
   Train_Data <- as.matrix(Train_Data)
   crossModel <- cv.glmnet(Train_Data,Train_bmi,alpha=iAlpha)
   myLambda <- crossModel$lambda.min
   myCVM <- min(crossModel$cvm)
   title <- paste(iAlpha,myLambda,sep="_")
   return(c(iAlpha,myLambda,myCVM))
}

myFrame <- as.data.frame(do.call(rbind,lapply(myAlpha,findAlpha_lambda)))
colnames(myFrame) <- c('Alpha','Lamda','CVM')
myFrame <- myFrame[order(myFrame$CVM),]
print(myFrame)

     Alpha      Lamda      CVM
1    0.0 50.9839208 54.25337
2    0.1  1.7901432 54.37151
3    0.2  3.1427680 54.75240
4    0.3  2.1949422 57.68935
5    0.4  1.8927376 61.68384
9    0.8  1.2510439 63.69622
6    0.5  1.0933677 64.68333
8    0.7  0.2441112 64.73192
7    0.6  2.2050751 65.01727
11   1.0  1.3860429 65.17181
10   0.9  0.5042962 65.70732

I don‘t think a large lambda is a problem per se. It just means that a lot of regularization is going on (under Ridge). See here:

https://stats.stackexchange.com/questions/212056/ridge-lasso-lambda-greater-than-1

Here is a good tutorial from the authors of glmnet. I suggest you check your approach, i.e by looking at various figures as shown in the tutorial, especially plot(cvfit) might be instructive. Also when you go through the tutorial, you see quite „large“ values of lambda (note that log of lambda is plotted).

https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html


Skew and Kurtosis are so similar? I've been taking the histogram of the optical flow in videos and plotting the kurtosis and skewness of each frame. At the end of the video, I noticed that the skewness and kurtosis follow each other - ie, when the skewness goes up so does the kurtosis and same when it goes down. In fact, the kurtosis almost looks just like a scaled version of the skewness.
I know skewness and kurtosis are supposed to be different concepts entirely, since they are different moments (the graphs of x^3 and x^4 do not look similar at all ) but it just makes me wonder why take both when they both seem to be similar? How are they useful separately? Also would it be due to my distributions? For what distributions would this kind of behaviour occur?
For reference the functions I am using are scipy.stats.skew and scipy.stats.kurtosis so I'm assuming that they are using sample skew and kurtosis

A simple and direct answer is that skewness and kurtosis are both defined in terms of the $Z-$values, $Z = (X-\mu)/\sigma$ : Skewness = $E(Z^3)$ and Kurtosis = $E(Z^4)$. When talking about a data set, you can just replace the expectation operator "$E$" with the ordinary average.

Since raising a number to either the third or fourth power amplifies values that are greater than 1 (in absolute value) and diminishes values that are less than 1 (in absolute value), both skewness and kurtosis are primarily measures of the heaviness of the tails of the distribution of $X$.

Note that the $Z$-values than are less than one in absolute value correspond to $X$-values that are less than one standard deviation from the mean; greater than one correspond to values that are more than one standard deviation from the mean. Hence, the averages of the $Z^3$ and $Z^4$ values are highly influenced by the $X$-values that are many standard deviations from the mean; i.e., by the tails or outliers.

Consider skewness first.  By definition of skewness, and by definition of center of gravity, the graph of the probability distribution of $Z^3$ balances (i.e., has a center of gravity) at the skewness of $X$. Hence, if the tails are heavier on the left side of the distribution than on the right, then skewness is negative. Conversely,  if the tails are heavier on the right side of the distribution than on the left, then skewness is positive.

Now consider kurtosis. The graph of the probability distribution of $Z^4$ balances (i.e., has a center of gravity) at the kurtosis of $X$. Note that the kurtosis of the normal distribution is exactly 3.0. Hence, if the tails are heavier on either or both sides of the distribution than the tails of the normal distribution, then the kurtosis is greater than 3.0, and the probability distribution of $Z^4$ falls to the right when a fulcrum is placed at 3.0. Conversely, if the tails are lighter on either or both sides of the distribution than the tails of the normal distribution, then the kurtosis is less than 3.0, and the probability distribution of $Z^4$ falls to the left when a fulcrum is placed at 3.0

Hence, skewness and kurtosis really do not measure things that differently - they are both measures of tail heaviness. The biggest difference is that skewness considers relative heaviness in one tail versus the other. The Wikipedia entry on kurtosis displays both lower and upper bounds for kurtosis in terms of skewness.

There is a mistaken but remarkably persistent meme out there that kurtosis measures "peakedness or flatness" that might be a source of confusion.  Your work shows (yet again) that this characterization is false. See  here for a clear explanation of why the "peakedness/flatness" characterization is wrong.


Normalize Test and Control, or as a whole? I have the following data:
\begin{matrix}
 &Test Sample\ 1 & Test Sample\  2 \dots & Test Sample \ 6& Control Sample\ 1 & Control Sample\ 2 \dots & Test Sample \ 6\\
measurements& 100 & 90 & & 30 & 20\\
\end{matrix}

If I want to normalize and compare between the test and control, should I normalize test and control samples separately or the whole matrix ?

An experimental sample is a group that receives the variable being
  tested in an experiment. The control group is the group in an
  experiment that does not receive the variable you are testing using
  the same experimental procedure. Therefore the control group,
  receiving no intervention, is used as a baseline to compare groups and assess the effect of that intervention.

I gave this as an object lesson to show that test and control must be linked by a similar experimental procedure. So yes, if you were to normalize then use the entire matrix.

Additional point (bonus round)

The purpose of data transformations is to make data easier to model and understand. One purpose of transforming data is to find outliers, reduce the amount of skew, etc.

Looking at a small part of your data, it seems like the values are between 0-100,  Suggesting that normalization is not entirely necessary, IMHO.


how much text data is required for a meaningful use of word2vec how much data does word2vec require?  Are there any public data sets that are useful?

For example, could it be that 1000 newspaper articles are enough to use word2vec?

Here is a word2vec tutorial from Kaggle that uses 50,000 movie reviews.  I am trying to understand the scale of the word2vec input.

As word2vec is a neural network, it benefits from very large datasets.  The Kaggle dataset is 50,000 reviews * ~5 sentences per review, so about a quarter million sentences.  As they note, they get approximately the same results using bag of words and word2vec.  One thing which is of note, since the review data comes from the internet, the sentences are much more loosely structured than what you would encounter in a corpus of newspapers, which typically go through grammatical review.  A great dataset for training word2vec on structured language is the wikipedia dataset:
https://en.wikipedia.org/wiki/Wikipedia:Database_download


Strange patterns from GAN I'm doing experiments with GAN. I've successfully trained GAN on 28x28px MNIST dataset (samples scaled to (-1,1) interval)

My next experiment is to train GAN on bigger images. My dataset consists of grayscale 128x128px images. I've added one more convolutional layer to both generator and discriminator and updated dense layers accordingly.

Sample from MNIST dataset and sample from my custom dataset:

However, when I trained the model, all results from generator looks like this:

What am I doing wrong? Why does the generator generate only this noise pattern instead of samples from my dataset distribution?

Generator:

H = Sequential()

H.add(Dense(1024, input_shape=(noise_dim,)))
H.add(BatchNormalization())
H.add(Activation('relu'))

H.add(Dense(128*16*16))
H.add(BatchNormalization())
H.add(Activation('relu'))
H.add(Reshape([16, 16, 128]))

H.add(Conv2DTranspose(128, (3, 3), strides=(2, 2), padding='same'))
H.add(BatchNormalization())
H.add(Activation('relu'))

H.add(Conv2DTranspose(64, (3, 3), strides=(2, 2), padding='same'))
H.add(BatchNormalization())
H.add(Activation('relu'))

H.add(Conv2DTranspose(32, (3, 3), strides=(2, 2), padding='same'))
H.add(BatchNormalization())
H.add(Activation('relu'))

H.add(Conv2DTranspose(1, (3, 3), padding='same'))
H.add(Activation('tanh', name='out_image'))

gen_in = Input(shape=(noise_dim,), name='in_noise')
gen_out = H(gen_in)

Discriminator:

H = Sequential()

H.add(Conv2D(32, (3, 3), strides=(2, 2), padding='same', activation='relu', input_shape=(128, 128, 1)))
H.add(LeakyReLU(0.2))

H.add(Conv2D(64, (3, 3), strides=(2, 2), padding='same', activation='relu'))
H.add(BatchNormalization())
H.add(LeakyReLU(0.2))

H.add(Conv2D(128, (3, 3), strides=(2, 2), padding='same', activation='relu'))
H.add(BatchNormalization())
H.add(LeakyReLU(0.2))
H.add(Flatten())

H.add(Dense(1024))
H.add(BatchNormalization())
H.add(LeakyReLU(0.2))

H.add(Dense(1, activation='sigmoid', name='out_binary'))

disc_in = Input(shape=(128, 128, 1), name='in_image')
disc_out = H(disc_in)

Training GANs becomes much more difficult with increasing input dimensionality. I encountered that sometimes even small changes to your network can decide about convergence/divergence of your network (like initialization of your variables).

I recommend following the tipps from https://github.com/soumith/ganhacks.

Another thing I personally recognized is that you should not use as many Conv2DTranspose layers as you do in your current network. Try instead increasing the latent space.


How to interpret the Mean squared error value in a regression model? I'm working on a simple linear regression model to predict 'Label' based on 'feature'. The two variables seems to be highly correlate corr=0.99. After splitting the data sample for to training and testing sets. I make predictions and evaluate the model.
metrics.mean_squared_error(Label_test,Label_Predicted) = 99.17777494521019
metrics.r2_score(Label_test,Label_Predicted) = 0.9909449021176512

Based on the r2_score my model is performing perfectly. 1 being the highest possible value. But when it comes to the mean squared error, I don't know if it shows that my model is performing well or not.

How can I interpret MSE here ?

If I had multiple algorithms and the same data sets, after computing MSE or RMSE for all models, how can I tell which one is better in describing the data ?

R2
score is 0.99, is this suspicious ?  Or expected since the label and
feature are highly correlated?
      Feature        Label
0   56171.757812    56180.234375
1   56352.500000    56363.476562
2   56312.539062    56310.859375
3   56432.539062    56437.460938
4   56190.859375    56199.882812
...     ...     ...
 24897  56476.484375    56470.742188
 24898  56432.148438    56432.968750
 24899  56410.312500    56428.437500
 24900  56541.093750    56541.015625
 24901  56491.289062    56499.843750

Whether you model is performing well or not depends on your business case, you might hive tiny RMSE or great looking score on whatever metric you are using, but it just not enough to solve the business problem, in that case the model is not performing well.

MSE is just that Mean Squared Error

Both MSE and RMSE measure by how much the predicted result deviates from actual, because of the squared term more weight is given to larger errors, and because of square root in RMSE, it is in the same units as dependent variable. MAE, Mean Absolute Error is another useful metric to look at when you are evaluating a regression model; it is also easier to interpret.

Given your data, R-squared seems fine to me.


How to create dictionary with multiple keys from dataframe in python? I have a pandas dataframe as follows, I want to convert it to a dictionary format with 2 keys as shown:

    id              name                        energy             fibre
0   11005   4-Grain Flakes                          1404            11.5
1   35146   4-Grain Flakes, Gluten Free             1569             6.1
2   32570   4-Grain Flakes, Riihikosken Vehnämylly  1443            11.2

I am expecting the result to be of

 nutritionValues = {
  ('4-Grain Flakes', 'id'): 11005,
  ('4-Grain Flakes', 'energy'): 1404,
  ('4-Grain Flakes', 'fibre'):  11.5,
  ('4-Grain Flakes, Gluten Free', 'id'): 11005,
  ('4-Grain Flakes, Gluten Free', 'energy'): 1569,
  ('4-Grain Flakes, Gluten Free', 'fibre'):  6.1,
  ('4-Grain Flakes, Riihikosken Vehnämylly', 'id'): 32570,
  ('4-Grain Flakes, Riihikosken Vehnämylly', 'energy'): 1443,
  ('4-Grain Flakes, Riihikosken Vehnämylly', 'fibre'):  11.2}

foods, fiber = multidict({
  '4-Grain Flakes': 11.5,
  '4-Grain Flakes, Gluten Free':   6.1,
  '4-Grain Flakes, Riihikosken Vehnämylly':   11.2})

How can I achieve this?

In order to be able to create a dictionary from your dataframe, such that the keys are tuples of combinations (according to your example output), my idea would be to use a Pandas MultiIndex. This will then generate a dictionary of the form you want.

First I just recreate your example dataframe (would be nice if you provide this code in the future!):

import pandas as pd

# Create the example dataframe
df = pd.DataFrame(["4-Grain Flakes", "4-Grain Flakes, Gluten Free", "4-Grain Flakes, Riihikosken Vehnämylly"])
df["id"] = [11005, 35146, 32570]
df["energy"] = [1404, 1569, 1443]
df["fibre"] = [11.5, 6.1, 11.2]
df.columns = ["name"] + list(df.columns[1:])

print(df)
                                     name     id  energy  fibre
0                          4-Grain Flakes  11005    1404   11.5
1             4-Grain Flakes, Gluten Free  35146    1569    6.1
2  4-Grain Flakes, Riihikosken Vehnämylly  32570    1443   11.2

Now we can create the combinations of each value in "name" with each of the other column names. I will use lists, within a list comprehension, where I bundle up the values together into tuples. We end with a list of tuples:

names = df.name.tolist()
others = list(df.columns)
others.remove("name")         # We don't want "name" to be included

index_tuples = [(name, other) for name in names for other in others]

We can create the MultiIndex from this list of tuples as follows:

multi_ix = pd.MultiIndex.from_tuples(index_tuples)

Now we can create a new dataframe using out multi_ix. To populate this dataframe, notice that we simple need to row-wise values from columns ["id", "energy", "fibre"]. We can do this easily by extracting as an n * 3 NumPy array (using the values attribute of the dataframe) and then flattening the matrix, using NumPy's ravel method:

df1 = pd.DataFrame(df[others].values.ravel(), index=multi_ix, columns=["data"])

print(df1)

                                                  data
4-Grain Flakes                         id      11005.0
                                       energy   1404.0
                                       fibre      11.5
4-Grain Flakes, Gluten Free            id      35146.0
                                       energy   1569.0
                                       fibre       6.1
4-Grain Flakes, Riihikosken Vehnämylly id      32570.0
                                       energy   1443.0
                                       fibre      11.2

Now we can simply use to to_dict() method of the datframe to create the dictionary you are looking for:

nutritionValues = df1.to_dict()["data"]

print(nutritionValues)

{('4-Grain Flakes', 'energy'): 1404.0,
 ('4-Grain Flakes', 'fibre'): 11.5,
 ('4-Grain Flakes', 'id'): 11005.0,
 ('4-Grain Flakes, Gluten Free', 'energy'): 1569.0,
 ('4-Grain Flakes, Gluten Free', 'fibre'): 6.1,
 ('4-Grain Flakes, Gluten Free', 'id'): 35146.0,
 ('4-Grain Flakes, Riihikosken Vehnämylly', 'energy'): 1443.0,
 ('4-Grain Flakes, Riihikosken Vehnämylly', 'fibre'): 11.2,
 ('4-Grain Flakes, Riihikosken Vehnämylly', 'id'): 32570.0}

It is also possible to get your final example of a multidict, directly from the multi-indexed dataframe. You need to just use multi-index slicing:

fibre_df = final_df.loc[(slice(None), ["fibre"]), :]
print(fibre_df)

                                                 0
4-Grain Flakes                         fibre  11.5
4-Grain Flakes, Gluten Free            fibre   6.1
4-Grain Flakes, Riihikosken Vehnämylly fibre  11.2

You can then generate a dictionary as before:

d = final_df.loc[(slice(None), ["fibre"]), :].to_dict()[0]
print(d)

{('4-Grain Flakes', 'fibre'): 11.5,
 ('4-Grain Flakes, Gluten Free', 'fibre'): 6.1,
 ('4-Grain Flakes, Riihikosken Vehnämylly', 'fibre'): 11.2}

And you can drop the "fibre" value from the tuple-keys with a simple dictionary comprehension:

final_dict = {k[0]: v for k, v in d.items()}
print(final_dict)

{'4-Grain Flakes': 11.5,
 '4-Grain Flakes, Gluten Free': 6.1,
 '4-Grain Flakes, Riihikosken Vehnämylly': 11.2}


How to Make Meaningful Conclusions here? I recently appeared for an Interview for my college and I was asked the following question. The Interviewer said that this question was a Data Science question.
The question-
Suppose 7.5% of the population has a certain Bone Disease. During COVID pandemic you go to a hospital and see the records. 25% of the COVID Infected patients also had the Bone Disease. Can we say for sure if the Bone Disease is a symptom of COVID-19?
My Reponse-
I said No, and explained it as it's not necessary that COVID-19 is causing these symptoms, it could very well be possible that the 7.5% of the country's population which already had the disease is more susceptible to the virus due to lowered immunity. Hence making conclusions is not possible.
Then the interviewer asked me How can we be sure if it is a symptom or not?
I replied saying we can go to more Hospitals, collect more data and see if it correlates everywhere.
The Interviewer then said If we have the same results everywhere will you conclude it's a symptom?
I had no good answer but I replied that Just correlation of data is not sufficient, we also need to check if the people who have COVID-19 had the bone disease prior to getting infected or not. See if that percentage also correlates and stuff.
Here he stopped questioning however I couldn't judge If I was right or wrong.
I am in Grade-12 so I have no experience in Data Science as such. I do know a fair bit of statistics however I have never solved such questions. Can someone provide me insights on how to solve such questions and make meaningful conclusions?

It is very difficult (arguably impossible, if you want to get philosophical about it) to be absolutely, 100%, for sure about anything. For this reason, we talk in terms of probability/significance/confidence sets. A refresher on statistical hypothesis testing might help.
So to answer this type of question, people would usually try to attain a well-agreed-upon p-value for their problem, below which we can reject the null hypothesis, and above which we accept the null hypothesis. The null hypothesis in this case is that the bone disease is not a symptom of COVID, and the alternative would be that it is a symptom.

Edit for demonstration as requested in the comments:
Firstly, these methods are purely for association analysis, not for proving whether bone disease is a symptom of COVID—again, correlation $\ne$ causation!
Given that we are dealing with binary variables, you could use the Phi coefficient to measure the association of bone disease with COVID.
consider this contingency matrix:
|----------|---------|---------|-----------|
|          | Bone =0 | Bone =1 | total     |
| COVID =0 |    A    |  B      | I=A+B     |
| COVID =1 |    C    |  D      | J=C+D     |
|------------------------------|-----------|
|  total   |  K=A+C  |  L=B+D  | E=I+J+K+L |
|----------|---------|---------|-----------|

Which we can represent visually via a Venn diagram:

then you can calculate
$ \phi=\frac{AD-BC}{\sqrt{IJKL}}=\frac{ED-IK}{\sqrt{IK(E-I)(E-K)}} $
This is related to the Chi-squared test: $ \phi= \sqrt{\frac{\chi^2}{n}} $ So you can easily retrieve the p-value given that you know the degrees of freedom (in this case, it is just 1).
And you interpret it similar to the Pearson correlation coefficient (both from the same statistician—Pearson).

So given $7.5\%$ of the total has the bone disease, and $25\%$ of COVID patients have it, we can construct our contingency table in terms of $N$ (the number of samples). Where $Q$ is the percent of people with COVID:
$ I=N(1-Q)= $ number of people without COVID
$ J=NQ= $ number of people with COVID
$ K=N(1-0.075)=$ number of people without bone disease
$ L=N(0.075)=$ number of people with bone disease
We know that 25% of people with COVID also have the bone disease, so $D=J(0.25)\Rightarrow$
$\phi=\frac{E(0.25\cdot J)-IK}{\sqrt{IK(E-I)(E-K)}}$
Finally, we can calculate:
$\phi=\frac{0.25\cdot EQ-N(1-Q)(1-0.075)}{\sqrt{(1-Q)(1-0.075)(E-N(1-Q))(E-N(1-0.075))}}$
From here, we can find the associated p-value easily by looking it up in a Chi-Square p-value table, such as this one: http://chisquaretable.net/. Then you can accept/reject the null hypothesis given your predefined $\alpha$ threshold.


kaggle Titanic what is GP? sometimes I see the kernel with GP programming.
but without explanation, they put some random number with unknown equation
Here is some part of GP programming

gp <-function(data)
{
  p<-0.200000*tanh(((((31.006277) * ((((((data$Embarked) + (data$Cabin))/2.0)) + (((((sin((tanh((data$Parch))))) * (3.0))) - (data$Pclass))))))) * 2.0)) +
    0.200000*tanh(((31.006277) * (((((data$Age) * (data$Cabin))) + (((((0.318310) - (data$Pclass))) + (pmin((2.0), (((data$Parch) * 2.0)))))))))) +
    0.200000*tanh(ifelse(rep(1.,NROW(data))*(ifelse(rep(1.,NROW(data))*(data$SibSp>0.),data$Cabin,sin((data$Parch)))>0.),(7.90205097198486328),(((((((data$Cabin) + (data$Fare))/2.0)) - (9.869604))) - (31.006277)))) +
    0.200000*tanh(((((((((((tanh(((((0.636620) < (data$Parch))*1.)))) * 2.0)) - (data$Pclass))) + (((data$Embarked) + (sin((data$Pclass))))))) * 2.0)) * 2.0)) +

full code is here https://www.kaggle.com/scirpus/my-first-gp-in-r

can someone explain what is this GP mean and
how this number and equation come up with?

My guess is same as Neural network right?

GP is also known as genetic programming. It is an algorithm which has been inspired by natural selection (survival of the fittest) to find an ideal algorithm to perform some task. These algorithms create individuals at each generation which have an encoded behaviour as a set of genes. The individuals which best perform the task at each generation are selected to be permutated slightly and progress into the next generation. This causes the solution space in later generations to narrow around some local minimum.

Example

Let's look at a simple example to see how this can be used for finding the minimum of a 2D function from -100 to 100.

This consists in 4 crucial steps: initialization, evaluation, selection and combination.

Initialization

Each individual in the population is encoded by some genes. In our case the genes represent our $[x, y]$ values. We will then set our search range to [0, 1000] for this specific problem. Usually you will know what is naturally possible based on your problem. For example, you should know the range of possible soil densities in nature. We will create 100 individuals in our population.

Evaluation of the fitness

This step simply asks you to put the $[x,y]$ values into your function and get its result. Pretty standard stuff.

Selection

There are many ways with which you can select parents. I will always keep the alpha male. The best individual in the population, he will be cloned to the next. Then I will use tournament selection. We will repeat the following until the next generation population is full. Pick four parents at random, take the best individual from the first two and the best from the last two. These will be the two parents which will gives us our next offspring.

Combination

From the two parents we will build the new genome for the child using the binary values of the $[x,y]$ values of the parents. The resulting binary value for each codon in the genome of the child is selected from the two parent genes by uniform random.

import numpy as np

class Genetic(object):

    def __init__(self, f, pop_size = 1000, n_variables = 2):
        self.f = f
        self.minim = -100
        self.maxim = 100
        self.pop_size = pop_size
        self.n_variables = n_variables
        self.population = self.initializePopulation()
        self.evaluatePopulation()

    def initializePopulation(self):
        return [np.random.randint(self.minim, self.maxim, size=(self.n_variables))
                           for i in range(self.pop_size)]

    def evaluatePopulation(self):
        return [self.f(i[0], i[1]) for i in self.population]

    def nextGen(self):
        results = self.evaluatePopulation()
        children = [self.population[np.argmin(results)]]

        while len(children) < self.pop_size:
            # Tournament selection
            randA, randB = np.random.randint(0, self.pop_size), \
                           np.random.randint(0, self.pop_size)
            if results[randA] < results[randB]: p1 = self.population[randA]
            else: p1 = self.population[randB]

            randA, randB = np.random.randint(0, self.pop_size), \
                           np.random.randint(0, self.pop_size)
            if results[randA] < results[randB]: p2 = self.population[randA]
            else: p2 = self.population[randB]

            signs = []
            for i in zip(p1, p2):
                if i[0] < 0 and i[1] < 0: signs.append(-1)
                elif i[0] >= 0 and i[1] >= 0: signs.append(1)
                else: signs.append(np.random.choice([-1,1]))

            # Convert values to binary
            p1 = [format(abs(i), '010b') for i in p1]
            p2 = [format(abs(i), '010b') for i in p2]

            # Recombination
            child = []
            for i, j in zip(p1, p2):
                for k, l in zip(i, j):
                    if k == l: child.append(k)
                    else: child.append(str(np.random.randint(min(k, l),
                                                             max(k,l))))

            child = ''.join(child)
            g1 = child[0:len(child)//2]
            g2 = child[len(child)//2:len(child)]
            children.append(np.asarray([signs[0]*int(g1, 2),
                                        signs[1]*int(g2, 2)]))
        self.population = children

    def run(self):
        ix = 0
        while ix < 1000:
            ix += 1
            self.nextGen()
        return self.population[0]

To use this algorithm you can define some functions and then run the following.

f = lambda x, y: (x-7)**2 + y**2
gen = Genetic(f)
minim = gen.run()

print('Minimum found      X =', minim[0], ', Y =', minim[1])


How to predict the sentiment of the entities form the tweet? I have a JSON file (tweets.json) that contains tweets (sentences) along with the name of
the author.
Objective 1: Get the most frequent entities from the tweets.
Objective 2: Find out the sentiment/polarity of each author towards each of the entities.
Sample Input:
Assume we have only 3 tweets:
Tweet1 by Author1: Pink Pearl Apples are tasty but Empire Apples are not.
Tweet2 by Author2: Empire Apples are very tasty.
Tweet3 by Author3: Pink Pearl Apples are not tasty.

Sample output:
Entities in the topics extracted: Share a dictionary with extracted entities as keys and the number of
tweets an entity has mentioned as values.
Example dictionary for the above example: {“Pink Pearl Apples” : 2,
“Empire Apples” : 2}

Now, objective 2 -

Now using Count Vectorizer I have completed objective 1, now how to get the sentiment based on the entities(Objective 2)
Thanks in Advance!!!

If you have a fixed number of entities and a labelled dataset, then you can build a CNN multilabel classification model. The final layer of model has Nx3 binary outputs, where N is the number of entities. Each entity has 3 outputs, representing positive, negative and a NA sentiment. The NA is important, an entity may not be represented in a tweet.
The CNN is recommended over a standard dense network so that the model can take advantage of signal words in the neighborhood of entities to estimate the sentiment.


Logistic Regression - ROC curve plots Sensitivity vs Specificity instead of (1-Specificity) I am new to Machine Learning and have been doing some practice on Logistic Regression. To evaluate the models, I've been trying to create some ROC plots.
The package that i used is pROC.

The model name is - 'model' dataset is 'data'.

The code I used in R is:

library(pROC)
predictionData <- predict(model, newdata = data, type = "response")
rocModel <- roc(data$y ~ as.numeric(predictionData>0.5))
plot(rocModel)

Ideally, from what i have learned, ROC should plot Sensitivity or TPR (True Positive Rate) vs 1 - Specificity. But as shown in the picture below, it shows Sensitivity vs Specificity.

Am I missing some obvious trick here or is something wrong with what I have done ?

As you can see, the specificity in the x-axis goes from 1.0 to 0.0 (backwards).

So the effect is the same, you are seeing 1-Specificity


How to play with .cat.codes and get_dummies I may be lost here and this might be a stupid question.
I know that get_dummies returns multiple columns depending upon the number of variables and .cat.code returns a single column with unique integers per unique value(Please correct me if I'm wrong).
I know fro .cat.codes is for categorical data and get_dummies is for object.

Can I use astype("Category") on an object and use .cat.codes on it?
Anything more that you know of to make analysis(and visualization) easier when it comes to these funtions?

You can convert object dtype to category using astype("Category") and then do label encoding on it using .cat.codes but it's better to use sklearn.preprocessing.LabelEncoder for it.
For basic analysis you should look into seaborn


Anomaly detection using RNN LSTM I'm trying to detect anomalies in an univariate time series.
I trained a RNN LSTM and currently I get one-step-ahead predictions.

Could someone explain if it's possible to output a confidence interval (or maybe a prediction interval) with RNN LSTM instead of just a predicted value.

When I want to flag observations as anomulous on real-time data I think I need more information than just the predicted and observed value for point x (f.e. I need a predicted confidence interval).

Please point me into the right direction.
Currently I'm using an Adam optimizer, MSE as the loss function and a Dense output layer with a linear activation function.

Disclaimer, I have not tried any of these ideas.

Predict the CI directly

Note: This method will require very large batch sizes, which may not be possible due to memory constraints.

Set the batch size to something large. You'll also need to decide what you want your confidence interval to be, 95% matches a lot of expectations but may be less numerically stable than something like 90% of 80%. We'll call this value $0 < c < 1$.

In addition to outputting a prediction $\hat{y_i}$ of the true value $y$, have your model also output a lower bound $l_i(c)$ and an upper bound $u_i(c)$ for each individual estimate.

Now, define
$$
z =
\begin{align}
& \left(\frac{1-c}{2} - \frac{\sum_{i=1}^n (1\ \text{ if }\ (y_i < l_i(c))\ \text{ else }\ 0)}{n} \right)^2 \\
+ & \left(\frac{1-c}{2} - \frac{\sum_{i=1}^n (1\ \text{ if }\ (y_i > u_i(c))\ \text{ else }\ 0)}{n} \right)^2
\end{align}
$$

In essence, $z$ is the (squared) measure of how many times the predicted lower/upper bounds are exceeded vs. how many times we expect them to be exceeded. In practice, some tweaks to exactly how $z$ is computed may be necessary, but the main idea of penalizing the model for having too lax or too restricted in its estimations of the lower/upper bounds should be preserved.

Add $z$ to your MSE loss function, you'll likely need to balance these two losses, i.e. add an additional hyperparameter $\alpha$ to your model such that $\text{TOTAL LOSS} = \text{MSE} + \alpha z$.

As an example, if you set your batch size/time-step length so that there are 500
distinct estimates (e.g. $n=500$), and set $c$ to 0.9, in theory this training procedure should encourage the model to estimate $l_i(c)$ and $u_i(c)$ such that exactly 25 actual $y_i$ values are below $l_i(c)$ and exactly 25 $y_i$ are above $u_i(c)$.

Predict the error directly

In addition to getting a prediction $\hat{y}$ of the true value $y$ and training with the MSE loss function, add another output of your model which attempts to predict the squared error $\epsilon = (\hat{y} - y)^2$ directly.

You can create the MSE of the estimate $\hat{\epsilon}$ of $\epsilon$ and combine that with your existing MSE to get the total loss for training.

You can then use this value to normalize your errors to try and get an anomaly score that is usable across different sub-populations:

$$
\text{anomaly score} = \frac{(y - \hat{y})^2}{\hat{\epsilon}}
$$
where higher scores indicate that the true error is larger than expected.


Keras Attention Guided CNN problem I am working on a CNN for XRay image classification and I can't seem to be able to properly train it. I am trying to implement the following paper in Keras: https://arxiv.org/pdf/1801.09927.pdf

In short, the paper describes a 3 network architecture.

The Global Branch is a ResNet or DenseNet (I used a DenseNet) pretrained on imagenet, then trained on the CheXNet dataset. After the training is done, the activations from the last convolutional layer are used to create heatmaps, which are then used to crop the images in the original dataset and create a new dataset.

The Local Branch is a ResNet or DenseNet (I used a DenseNet) with no pretraining. It is trained on the cropped dataset.

Finally, the fusion branch has as inputs the 2 global pooling layers of the Global and Local Branches.

I have managed to train the Global Branch (AUC: 0.77), generate the crops and train the Local Branch (AUC: 0.67). But when I try to train the Fusion Branch, the val_loss doesn't decrease:

This is my code:

def load_model_from_json(models_folder):
    print(" --- Reading model from ", models_folder)
    with open(models_folder + 'model.json', 'r') as f:
        json = f.read()
    print("Read: ", models_folder + 'model.json')
    model = model_from_json(json)

    model.load_weights(models_folder + "model_weights.h5")
    print("Read: ", models_folder + "model_weights.h5")

    return model

global_branch_model = load_model_from_json(self.global_branch_path)

local_branch_model = load_model_from_json(self.local_branch_path)

for l in global_branch_model.layers:
    l.trainable = False
    l.name = 'global_'+l.name
for l in local_branch_model.layers:
    l.name = 'local_'+l.name
    l.trainable = False

global_pooling = global_branch_model.get_layer('global_global_average_pooling2d_1')

local_pooling = local_branch_model.get_layer('local_global_average_pooling2d_1')

merged = concatenate([global_pooling.output, local_pooling.output])
dense = Dense(512, activation='relu')(merged)
dropout = Dropout(self.hyperparameters.dropout)(dense)
out = Dense(1, activation='sigmoid')(dropout)

fusion_model = Model(inputs=[global_branch_model.input, local_branch_model.input], outputs=out)
loss_function = unweighted_binary_crossentropy

optimizer = AdamW(lr=5e-5)

fusion_model.compile(optimizer=optimizer, loss=loss_function)

fusion_model.fit_generator(
            generator=FusionDataGenSequence(self.labels, self.partition['train'],
                                      current_state='train',
                                      batch_size=self.batch_size,
                                      hyperparameters=self.hyperparameters,
                                      num_classes=self.number_of_classes),
            epochs=self.hyperparameters.epochs,
            verbose=1,
            callbacks=callbacks,
            workers=self.num_workers,
            # max_queue_size=32,
            # shuffle=False,
            validation_data=FusionDataGenSequence(self.labels,
                                            self.partition['valid'],
                                            current_state='validation',
                                            batch_size=self.batch_size,
                                            hyperparameters=self.hyperparameters,
                                            num_classes=self.number_of_classes)
            # validation_steps=1
        )

Can you please help me figure out what I did wrong? Thank you!

Maybe i'm missing something, but where is the connection between the global and local branches?

During the training of the local branch you used a dataset that was constructed from masked images. But now that branch should take as input the output of the 7x7x2048 conv layer from the global branch.

Maybe your training set constructed from coupled inputs for the 2 branches? I think you should add the implementation of your "FusionDataGenSequence".

What about your accuracy during the training?
Did you try different learning rates?


Is it ok to use photo collages as dataset instead of single images for training the object detection ssd model? Is it normal/better to use photo collages (multiple photos in one image) as a dataset instead of single images for training the object detection SSD model?
I am using Tensorflow Object Detection API to make an SSD model to identify and detect different leave diseases. My dataset consists of 256x256 leaves (plant village dataset), I want my model to be trained for detecting multiple objects as well with different sizes (most of the leaves are almost as big as the image itself - which gives bad accuracy for small objects) so I thought combining those pictures into the collages to include several leaves in each picture and creating a dataset with the following way, for each label I will use single leaf images, 4 leaves images (2x2 collage) and 9 images (3x3 collage). Do you think that it will give better results for multiple object detection or Should stick with single images and it won't change anything? If it works what would you recommend to set for the image resize option? Should it be still low like 300x300 or it will be better to have 800x800 since having 9 images in some part?

1.You can use Progressive Resizing in your dataset in order to improve your results.
Link here
This method can be helpful when the images only depict parts of the sample.
2.Also, in order for your model to be good at predicting smaller leaves that do not take the entire images, you should make your validation set balanced ie. a number of smaller leave images and leaves that take the whole image should be balanced in the validation set.
3.I don't know much about the collage-based approach, but you can check this
Hope it helps. Please upvote if it helped.


Setting input shape for an NLP task in R(Rstudio) using keras 1D convolution layer, when it expects 3 dimensional input (a tensor) I am using R programming language and using Keras API to build a functional 1D CNN.

I have a matrix of my dataset of the following shape rows*features (6000*1024).

The input layer is set using the following code:

input_layer = layer_input(shape = 1024, batch_shape = c(nrow(train_matrix),1024), dtype = 'float64')

and then I am building a 1d conv layer as follows:

conv1 = input_layer %>% layer_conv_1d(filters = 32, kernel_size = 50, strides = 10, input_shape = 1024, batch_input_shape = list(NULL, 1024) ,dtype = 'float64', activation = 'relu' )

But I get the following error:

  Error in py_call_impl(callable, dots$args, dots$keywords) :
    ValueError: Input 0 is incompatible with layer conv1d: expected ndim=3, found ndim=2

I believe it is due to the fact that 1D cnn layer expects the input in the following form

  Input shape: 3D tensor with shape: (batch_size, steps, input_dim)

I understand that I have to reshape my data as (NULL, nrow(train_matrix), 1; as this has been suggested in various answer for the same issue arising for keras when used in Python.

If I am right,

what values should I provide to input layer
how should i reshape my training data?
does that mean I have to reshape the test data as well?

also if my understanding is wrong what should be done otherwise ?

The input layer was producing a 2d tensor, whereas 1d convolutional layer expects a 3d tensor as input. A good explanation of as to why keras expects it and how should you reshape your 2d input to 3d can be found in this answer.

I used Keras R API function k_reshape( 2d_tensor, (list_of_new_dims)) --> k_reshape(input_layer, list(nrow(train_matrix), num_of_feature_vectors, 1).

nrow(train_matrix) - the total number of rows in my matrix (no of samples)

num_of_feature_vectors - total number of columns in matrix (total number of features)

1 - i want a 3d tensor with only 1 element at z axis, therefore z axis is initialised to 1


Loss found not good enough I'm trying to use a Neural Network in order to predict two different values from an input that contains 24 different features. The results I've gotten so far are not good enough, so any suggestions would be appreciated since I've been stuck for some time. This is what I've done so far:
Input data:
I have an input that contains 24 different features (the total dataset has around 150,000 instances). So I've tried to standarize my input, normalize it, log transform it, and use PCA in order to reduce the dimensionality of the problem. Out of this, PCA has proven to be the best solution (using the first 5 principal components).
To make sure that input is significant, I have done a quick fit using a Random Forest Regressor and a Extra Tress Regressor to calculate the importance of each feature (after having performed PCA). And compared to a random feature, all the features that are being used seem to be significant enough for the model.
Neural Network
For the neural network, I have tried a lot of things arriving to the following architecture:
initializer_sm = tf.keras.initializers.GlorotNormal()

model_nn = keras.Sequential([
    keras.layers.Dense(128,input_dim = xnn_train.shape[1], activation='selu', kernel_initializer=initializer),
    keras.layers.Dense(256, activation='selu', kernel_initializer=initializer),
    keras.layers.Dense(256, activation='selu', kernel_initializer=initializer),

    keras.layers.Dense(2, activation = 'softplus', kernel_initializer=initializer_sm)
    ])

def custom_loss(y_true,y_pred):

    return (K.abs(y_true - y_pred)/y_true)*100

epochs = 100
opt = keras.optimizers.Nadam()
model_nn.compile(optimizer = opt, loss = custom_loss)
history_nn = model_nn.fit(xnn_train, ynn_train, epochs = epochs, batch_size = 1024, validation_data = (xnn_val,ynn_val))

With the following results:

And these are the histogram for each output predicted w.r.t the real values:

I have also tried to implement 1D convolutial layers before the architecture shown before, but the only effect that they have is that the loss function converges faster.

Your network is too large for this much data. Reduce the number of units in the layers, go for simple 'relu' in the layers except the last one where you should use softmax. Consider reducing number of layers.
I would recommend using decision trees libraries like XGBoost, CatBoost, LightGBM etc. for this problem. You will get the best result.


PyTorch vs. Tensorflow Fold Both PyTorch and Tensorflow Fold are deep learning frameworks meant to deal with situations where the input data has non-uniform length or dimensions (that is, situations where dynamic graphs are useful or needed).

I would like to know how they compare, in the sense of paradigms they rely on (e.g. dynamic batching) and their implications, things that can/cannot be implemented in each one, weaknesses/strengths, etc.

I intend to use this info to choose one of them to start exploring dynamic computation graphs, but I have no specific task in mind.

Note 1: other dynamic computation graph frameworks like DyNet or Chainer are also welcome in the comparison, but I'd like to focus on PyTorch and Tensorflow Fold because I think they are/will be the most used ones.

Note 2: I have found this hackernews thread on PyTorch with some sparse info, but  not much.

Note 3: Another relevant hackernews thread, about Tensorflow Fold, that contains some info about how they compare.

Note 4: relevant Reddit thread.

Note 5: relevant bug in Tensorflow Fold's github that identifies an important limitation: impossibility to do conditional branching during evaluation.

Note 6: discussion on pytorch forum about variable length inputs in relation to the algorithms used (e.g. dynamic batching).

There are a couple of good threads on Reddit right now (here and here).

I haven't used either of these frameworks, but from reading around and talking to users I gather that support for dynamic graphs in PyTorch is a 'top down design principle', whereas TensorFlow Fold is bolted on to the original Tensorflow framework, so if you're doing anything reasonably complicated with Tensorflow Fold you're probably going to end up doing a lot more hacking around than if you're using PyTorch.


Merging Training and Validation Sets for better accuracy I am training an Image Classification Model and my train-test set distribution is 80-20.

After Training my train-test loss curve looks like this

As the model is converged after around 20-30 epochs and is not overfitting.
Can I also try to train the same model by merging my training and testing set hoping it would become more accurate with increased size of training data ?

Is it a good idea to merge train, test set in this case ?
What are the disadvantages if I do so ?

After you have finished with the model building process (in which it is assumed that you have used your test set once and only once for assessing the performance of your final model on unseen data), and before deploying your model, both common sense and standard practice say that you should re-train it on all the available data, including the portion that, until then, had been put aside as test. Leaving out available data is a luxury which normally we cannot afford; and, provided that there are no issues with your model building process, and your test set is qualitatively similar to your training one (an assumption implicitly always present), there is nothing to worry about.
Qualitatively speaking, this approach is similar with what we do with cross validation, where afterwards we routinely re-train the model using all the available data.
The following Cross Validated threads might be useful; although they address the cross-validation issue, the rationale is similar -  at the end, use all the data:

Training on the full dataset after cross-validation?
How to choose a predictive model after k-fold cross-validation?


How to perform SMOTE-N when there is no majority vote? In the SMOTE paper, the authors present the logic of creating synthetic examples when all features are nominal (section 6.2, SMOTE-N):

  To generate new minority class feature vectors, we can create new set
  feature values by taking the majority vote of the feature vector in
  consideration and its k nearest neighbors

Along with this example:

  Let F1 = A B C D E be the feature vector under consideration and let
  its 2 nearest neighbors be

  F2 = A F C G N

  F3 = H B C D N

  The
  application of SMOTE-N would create the following feature vector: FS =
  A B C D N

How would FS be chosen in the case that F3 = H B C I N? How does Value Difference Metric by Cost and Salzberg described in the paper assist in this case?

I don't know what "VDM" stands for, but a simple solution for tie breaking is to randomly pick one of the tied options.


How to force pytorch model to predict only positive values I am doing a prediction on a data set where labels have positive values (time values). After training a simple Linear pytorch model I get negative values for time despite being 0 negative values in the training set.
Is there some way to punish the model hard for making negative predictions so that the model would understand to make positive predictions (because taking ReLU does not feel correct to me)?

If you know that your output are positive, I think it makes more sense to enforce the positivity in your neural network by applying relu function or softplus $\ln(1. + \exp(x))$. You could also have a look at Generalized models which extend linear regresssion to cases where the variable to predict is only positive (Gamma regression) or between 0 and 1 (logistic regression). If you are predicting a categorical variable, you could also perform one hot encoding and transform your regression problem in a classification. Last but not least, as suggested in the last question it might be interesting to normalise your output between 0 and 1 and have a logistic regression in the last layer. Hope this help


Neural Network Initialization - Every layer? Does every layer of a Neural Network require weight initialization or just the first? Does the first layer feed into the next layer and initialize itself? My intuition is that every layer needs its own initialization but I'm finding it hard to see that said explicitly. Thanks!

Yes, every layer that contains weights needs to be initialised.

Every weight is just a number (usually a decimal, floating point number). It has to be initialised with some value so that the algorithm (e.g. backpropagation) has some values to work with.

Any learned parameters need to be initialised, as they will be adapted during training. This means you can inialised e.g. the parameters of momentum, and it is changed over time.

There are "layers" that don't need initialisation. I mean layers that perform some operations, but don't have dynamic weights; weights that are altered during training.

An example would be a standard dropout layer (with no learned parameters). This is called a layer, but doesn't require initialisation as it doesn't have any weights that are changed during training.


What does the huge number on top mean? I'm trying to visualize the neural network architecture.

from keras.utils import plot_model
plot_model(model, to_file='model.png', show_shapes=True)

What does the huge number on top mean?

It does not represent anything of significance and is probably a bug (see the links below). It is caused due to the use of Sequential API as it omits the input layer and directly takes the embedding layer as input. It can be removed by the use of functional API or by commenting out this in keras/engine/sequential.py :

@property
def layers(self):
    # Historically, `sequential.layers` only returns layers that were added
    # via `add`, and omits the auto-generated `InputLayer`
    # that comes at the bottom of the stack.
    if self._layers and isinstance(self._layers[0], InputLayer):
        return self._layers[1:]
    return self._layers

You can take a look at issues raised on github regarding this here and here.


How to interpret the loading values of a pca? Imagine I've the following matrix, which gives the grades of students in the subjects German, Philosophy, Math and Physics:

ger = c(2,4,1,3,2,4,4,1,2,3)
phi = c(3,4,1,2,2,3,3,2,2,2)
mat = c(1,3,2,4,1,2,2,4,3,1)
phy = c(2,2,2,5,2,2,3,4,3,3)
A = cbind(ger,phi,mat,phy)

I combine everything to a matrix and scale the data:

As = scale(A)

Now, I perform a summary  on the PCA:

summary(princomp(As), loadings = TRUE)

Which returns the following output:

Importance of components:
                       Comp.1    Comp.2     Comp.3     Comp.4
Standard deviation     1.3257523 1.1657791 0.59600603 0.35793402
Proportion of Variance 0.4882275 0.3775114 0.09867311 0.03558799
Cumulative Proportion  0.4882275 0.8657389 0.96441201 1.00000000

Loadings:
     Comp.1 Comp.2 Comp.3 Comp.4
ger  0.496 -0.502  0.519  0.482
phi  0.548 -0.443 -0.423 -0.570
mat  -0.430 -0.572 -0.546  0.435
phy  -0.518 -0.474  0.503 -0.503

I have a few hints for the first component (based on the loadings):

There is a high positive correlation between german and philosophy and there is also a high positive correlation between math and physics.
Who is good in language (german and philosophy) is often worse in MINT (math and physics) and the other way around.

And an idea about the second one, which I cannot interpret:

It's a weighted arithmetic mean over all four variables.

But I have no idea how to interpret the Comp. 2, Comp. 3 and Comp. 4 based on the loadings. Especially because all values of Comp. 2 are all negative, or have the same orientation. Can someone help me? Thanks in advance!

The columns of your loadings matrix are a basis of orthonormal eigenvectors.  This is an important concept from linear algebra, and well worth learning about in detail if you're not familiar.  But for the purposes of this answer it can be understood as defining a system of coordinates.

For each student, we can define a point in a four-dimensional space (specifically, in $\mathbb{R}^4$) which represents their grades (after centering and normalization).  Or to put it another way, you can imagine the set of all students' grades as a scatterplot in four dimensions, with four perpendicular axes.  We can orient these axes in various directions (just as we can in two or three dimensions).  The most obvious choice is to have one axis for each subject, so the axis which is collinear with the unit vector pointing from the origin to the point $(1,0,0,0)$ represents their grade in German, and likewise the axis which is collinear with the vector $(0,1,0,0)$ represents their grade in Philosophy, the axis which is collinear with the vector $(0,0,1,0)$ represents their grade in Math, and the axis which is collinear with the vector $(0,0,0,1)$ represents their grade in Physics.

However, there's no reason to expect that the direction in which our scatterplot is most spread out (the direction of greatest variance in the data) will align with one of these axes.  PCA picks out a new set of axes so that one axis aligns with the direction of greatest variance, and another aligns with the direction of the greatest remaining variance after the first direction is projected out, and so forth.  The unit vectors (expressed in the original coordinate system) which point along these new axes are the columns in your loadings matrix.

In the case of this particular example, the loading vector for the first principal component is along an axis that basically expresses whether they're better at Math and Physics, or better at German and Philosophy.  The loading vector for the second principal component is along an axis that basically expresses how good or bad a student they are over all (hence all the components of the vector have the same sign and similar magnitude).  You wondered about the negative sign on all four components - if you're familiar with eigenvectors you'll know that changing all components of the vector by an overall sign is irrelevant.  Basically, it's the same as just swapping which end of the axis we call positive and which we call negative.

So in this case the first two loading vectors are fairly close to what many of us might have expected to see.  But even in this fairly intuitive example, you shouldn't be surprised that the loading vectors for the later principal components don't seem as obvious to you.  That's because these are only addressing the variance that remains after we project out the variance that's explained by the first two factors.  We all probably know that students who are good at Physics tend to be good at Math, but how many of us know (for example) if, after controlling for how good they are at Physics, the ones who are also better at Philosophy than German will be better at Math?  These subtler effects will be less obvious to a causal observer than the dominant effects.

Once you get to the loading vector for the fourth principal component (out of four), you really don't need to wonder at all about why it has the particular value that it has.  In fact, this vector was entirely determined by the previous three (up to the irrelevant overall sign) .  This can be understood by remembering that PCA picked out four perpendicular axes in a four-dimensional space - once the first three are specified, there's only one remaining possible choice that's perpendicular to all of them.


Is normalizing the validation set of time series a kind of look ahead bias? Here's the data normalization process of a time series in a paper about stock prediction using LSTM:

Split train and test set based on time (e.g. training set: 2001-2010, test set:2011-2012). This looks fine to me.
Normalize the training set by subtracting the mean and dividing them by the standard deviation of the training set.
Train the data, using 20% of samples as the validation set. Keras's model, by default, uses the last 20% for validation.

So, in the training phase, the model knows a little about the validation set through the mean and standard deviation in step 2.

On one hand, the model is training using some of the future information. On the other hand, no information of the test set is leaked to the model. Is this a kind of look ahead bias?

And what is the best practices in this case?

To block the data leakage from the validation set to the training set in step (2),

We should first split the data to training and validation sets, then
Calculate the mean and standard deviation only using training set, and finally
Use this mean and std to normalize both the training and validation sets.

This way no information is leaked from validation set to training set, although validation set would be almost normalized. This deviation denotes an informative difference between distribution of validation and training sets which later [rightfully] leads to a validation error larger than training error.

We may think of normalizing the sets separately, however this way we may lose a potential discrepancy between the distribution of validation and training sets.

For example, in a 1D time-series of price, if the average price in training set is $1.0\$$, we expect that a validation set with average $2.0\$$ causes problem for a model that is trained on prices around $1.0\$$. However, by normalizing the validation set separately to $1.0\$$, we wrongly bring the validation set to a range that the trained model already expects, which leads to underestimation of validation error.

Edit

  in the training data itself - wouldn't every example be biased by the future? - Alon Gouldman

Correct. If training set is $[P_{t_1}, ..., P_{t_N}]$, then for $n \in [1, N)$ normalized point $\hat{P}_{t_n}$ would be biased toward (contains information of) points $[P_{t_{n+1}}, ..., P_{t_N}]$, but this is not a leakage since the entire training set is what we know, thus we can do whatever we want with it. From another point of view, this seemingly cheaty transformation would turn out OK since (1) the shortcoming of the normalization will be revealed in the validation error (more cheat, more validation error) thus we would not be misguided, (2) normalization could benefit us by lowering the validation error (otherwise, we do not normalize)


Non Deterministc Dimensionality reduction could you please suggest me a nondeterministic algorithm for dimensionality reduction except t-SNE.

Autoencoders are non-deterministic, since they rely on a random weight initialization.


How much neural network theory required to design one? So I have looked at some of the literature on neural networks and read some chapters, but the learning curve is so steep that I have had trouble even getting started on designing the neural network to solve my problem.

From what I understand, the architecture (or arrangement of neurons and their connections) should be designed according to the nature of the problem to be solved. Other parameters to be set according to the nature of the problem include the loss index (how the error will be calculated and if there should be a regularization term), whether or not there should be any scaling/unscaling, bounding, or conditions, and the training algorithm (such as the quasi-Newton method).

The particular type of problem I am interested in is using neural networks to figure out unknown functions (with unknown complexity) that input and output integers (as opposed to continuous values), given a large collection of inputs and outputs.

  An example function takes 4 byte inputs and returns 2 byte outputs. This is done by first taking the first 2 bytes of input and XORing them with the last 2 bytes, to produce an intermediate result. This two byte value is then XORed with a copy of itself that is shifted left by 5 bits. This result is then XORed with a copy of itself shifted right by 7 bits. Then this result is then XORed with a copy of itself shifted left by 2 bits, and this value is outputted by the function, giving the final 2-byte result. Note: More than one unique input can produce the same output.

So given a large set of inputs and outputs of an unknown function, the neural network should then optimize itself to reproduce this function given new inputs. I am not sure how to get started designing this neural network, and I am not sure fully reading through neural networks textbooks is the optimal way to get started. I am using a software library meant for designing neural networks, and I can simply set the network architecture and the parameters described above. How much theory do I need to know in order to get started solving my problem? Where should I start with learning how to design this neural network?

EDIT:
The main goal of all this is to use an existing tool to simplify producing an output (a 2-byte code, in the example above) given a new input, in a situation where the function is unknown. Neural networks seem to match the function-finding-through-trial-and-error characteristics that I need. This tool should be able to try all sorts of possible functions of increasing complexities in order to mimic the working of the actual unknown function.

Here is a good way to get going:

C++ or c#
spend a week learning vectors and matrices.
Be able to code a "class vector" and "class matrix"
Get intuition on how matrices represent weights between 2 layers
Learn the chain rule and the "tree diagram" when computing composite (nested) functions
Spend a week learning backprop. Be able to compute derivatives for all layers, using pen an paper a toy example with 4 layers, 3 neurons each.
be able to make code that reads a .txt file character by character.
Learn about one-hot, to describe your character to the network (it only understands vectors)  - or see how your byte-problem might be used instead of characters.
Learn about hard-max softmax and their derivatives,  tanh and sigmoid
code fwd prop
code bkprop
test if it can predict the next char
debug
debug
DEBUG
Gradient checking


Auto-Encoder to condense (pre-process) large one-hot input vectors? In my 3D game there are 300 categories to which a creature can belong.
I would like to teach my RL agent to make decisions based on its 10 closest monsters

So far, my Neural Network input vector is a concatenation of ten 300-dimensional one-hot encoded vectors. It would be awesome if I had, say ten 40-dimensional vectors instead.

So, is it possible to pass a 300-dimensional one-hot encoded vector through an auto-encoder, to snatch its compressed version (embedding) from the middle of the auto-encoder?

This would allow me to concatenate several of such compressed embeddings as a "total" input (ten concatenated 40D vectors), without bloating this total input vector.

In my game, each one-hot is supposed to represents a distinct category. My categories are not correlating in any way.

1) Will it be valid, or will it introduce unwanted assumptions, like Label-Encoding does?

2) How to train this Auto-Encoder? Can I simply pass my one-hots through it, and fine tune until I get an exact one-hot on the other end of the Auto-encoder?

3) Is there a better way to describe several monsters?

It might be useful to think of this in terms of orthogonality. You state that "categories are not correlating in any way", which effectively means each category should be completely orthogonal to all the others -- otherwise one would be representing some degree of correlation (or anti-correlation) between at least two categories (to the degree that those categories are not purely orthogonal). Now, the only way to get 300 orthogonal vectors is in 300 dimensional space. There is no way to compress that without losing information.

Alternatively one can think in terms of using simpler dimension reduction techniques than autoencoders. Let's consider using PCA (which is, after all, just a kind of linear autoencoder with identity activation function). What does the input data look like? Presumably we have one observation for each creature, so 300 observations. We have 300 features. The result is a 300x300 identity matrix. The singular values of that are all identically 1, so PCA to anything less than 300 dimensions is just going to fail. Adding non-linear activations and extra layers is not going to fix that, it will just make it noisier via stochastic optimization so you may get a result -- it just isn't a meaningful one.

What you need if you want to compress things is some correlation structure amongst your creatures. If you can come up with a set of properties/features that creatures might have and then score all your creatures accordingly you may be able to do some dimension reduction on that -- but dimension reduction on completely orthogonal vectors just isn't a meaningful thing.