Training with multi-series of different length with stateful LSTM I'm training a stateful LSTM.  My data is stored in a series of files, each file relates to a certain city. For each city I might have different amount of data, so City A I might have 4000 days, but City B I might only have 500. I have 20 features for each day.

Say I set the sequence size to 200; for City A I'd have a batch of 20 complete sequences, but for City B I'd only have 2.

With stateful LSTM I need to specify the batch_input_shape in the form [batch size, sequence size, features], which I think will cause me some problems because my batch size changes for each city I process.

As I see it, the only option available to me is to define the batch size for batch_input_shape as 1, and then just effectively issue a single 200-length sequence for each iteration. So batch_input_shape would be [1, 200, 20]

Obviously this isn't ideal because it's such a small amount of data that's being presented to the NN for each iteration (it's certainly mini-batch!) so it will be quite slow to train the network.

But am I right that due to the differing sizes of the data, I don't really have another option?  Besides slow train time, are there any inherent problems training with batch size of 1?

To begin with, you can think of the batch size as a way to control the smoothness of the learning curve. With a huge batch size, you are taking the average of many errors for each update, and this average loss (on average), doesn't have great variance.

Using a batch size of 1, your cost on each iteration is solely dependent on the single sample that you fed the network. Each sample is hopefully a little different from the others, which will lead to a very noisy loss curve.

Assuming model parameters are well-suited and the model converges irrespective of batch size, you should reach similar results. However, there are works that analyse batch size and show other trade-off, which are usually training-times, memory consumption and the like.

This recent paper (Masters, Luschi) analyses the trade-offs of batch size on a standard dataset (CIFAR10). Here is Figure 15 from the paper that shows this pretty succinctly:

Warning: all models and datasets could behave differently, however, so don't take any results like these as fact!

I know that it is possible to train a model that has varying input shapes by using a technique called Adaptive Pooling (or adaptive average pooling, in PyTorch, but you would likely have to come up with your own function that is able to do such a thing within the constraints of a stateful LSTM.

So as the shape of your dataset must be divisible by the batch size, there are a few ways to make that a reality:

using the highest common multiple of your datasets: so with your example datasets having 4000 days (set A) and 500 days (set B), you would compute the

This of course limits the possible choices of sequence length, but for the two examples of 4000 and 500, you could choose from these: 1, 2, 4, 5, 10, 20, 25, 50, 100

Trimming the different datasets so that you get a nice sequence length that works for both

This means possible leaving out some of your data, which is undesireable, but it might not be much.

Which of those two possiblities you might go for will depend on your specific dataset sizes. It might be optimal to use both - so trimming a few datasets to allow the computation of a nice Highest Common Factor.

You can compute the HCF in Python using something like this:

def hcf(a, b):
    while b:
        a, b = b, b % a
    return a

One final idea from my comment below:

... in order to change the batch size between datasets, you could copy them model weights from a trained model (from your first dataset), then compile a new model with the batch size required for the next dataset, but set the weights equal to those from the first model. This is almost like a manual way of implementing stateful behaviour. Have a look here for a simple example.


plotting time series data using matplotlib python I am trying to visualize Time series data is as follows
following is my code to plot the data
plt.plot(data['date'], data['c_16_avg_a'])
plt.xticks(rotation='vertical')
the date is getting truncated here. and the plot looks wired.
Any suggestions will greatly be appreciated

This is happening because the dates in your column are not sorted. To fix this either sort the values by the date column or, even better, plot using the timestamps instead of their dates. Matplotlib inherently handles datetime format better than it does datest represented as strings!
The best option however, is to add the timestamps as the dataframe's index and use pd.Series.plot() instead of matplotlib.
data.index = data['time_stamp']
data['c_16_avg_a'].plot()


Imbalanced Data how to use random forest to select important variables? I am trying to use random forest to select important variables out of 15K features and fit them into logistic regression. My evaluation is based on F1 score. Dataset 2 classes ratio are around: 99.5:0.5.

Here is where 15K features come from:
Initally I have 2000 features, after dumminize them (only taking top 100 categories from categoritical variable) it became 16K. After removing zero variance it became 15K. I didn't want to remove near zero variance because of my class imbalance ratio is also very small. I tried removing near zero variance features before and it significantly reduces the number of features, however the logistic regression result was also not good.

However, after using grid search, random forest still has very low f1 score for cross validation. (less than 0.01)

Also I tried undersampling for the training data to make it 1:1, random forest still has bad F1 score for cross validation. :(

So I was thinking about not selecting the important variables and just fit all features to logistic regression.

Due to memory issue, I am not able to fit all 15K features into logistic regression directly but if I select important variables from random forest, they do not generalize for unseen test data.

Any ideas on how to solve the problem?
I know one alternative could be using hashing so I could fit all 15K features.

15K features is a lot of features. I'd bet your 0.01 F1 score is a direct result of having so many features. Without knowing more details about your data it is hard to suggest a more sensible alternative. I'd want it to be closer to 100 features rather than 15,000 though. Are these features extracted from text? N-grams?
Regarding your feature selection problem, the first step is to eliminate any features with no variance (all samples the same), then one can utilis various statistical methods for selecting features to retain those with high variance (sklearn has great libraries for this: Feature selection).
One could also utilize Ranked Guide Iterative Feature Elimination (RGIFE) for this purpose, which uses random forests: RGIFE - Ranked Guided Iterative Feature Elimination
P.S. Your class ratio is very skewed as well. Once you're all settled with a more reasonable dataset in feature space I'd consider either undersampling the larger class, oversampling the smaller class or a combination of both, and keep an eye on the resulting confusion matrices when performing your evaluation.


How to split data in R using dplyr if we want to have rows of the same group to belong to the same split? In my current pipeline, I have sensed that there is data leakage. This is because the same person, though with slightly different values, is in both training and testing set. As a result, my model is overfitting.
For eg my data looks like this:
PID       Var_1   Var_2
Person A     0      1
Person B     0      1
Person C     0      0
Person A     1      3
Person B     1      2
Person D     0      1
Person C     0      1

I want to split this data such that the rows of the same person will be in either training or testing set, i.e I want the split to look like this:
Training:
PID       Var_1   Var_2
Person A     0      1
Person B     0      1
Person A     1      3
Person B     1      2

Testing:
PID       Var_1   Var_2
Person C     0      0
Person D     0      1
Person C     0      1

Figured out an easy way to do this.

First we will just select PID from the real data.
Then we will just sample 0.75 % of these PID and save these point as training PID and the rest as testing PID.
We will thne find the intersection between this list and the real data using PID.


5 digit number mis-reads analysis Nothing to do with number recognition in the classical 'hand-written' sense

Disclaimer above to avoid this being counted as a repeat.

I have a selection of 96 serial numbers, and a separate selection of >220 serial numbers. Within the larger set typically resides the smaller set (not always though), but also ~ 120 incorrect numbers.

See below for an example - for the record I have matched things up as best as I can... the correct number is first, the 'possibles' are in parenthesis at right:

21490 (21490, 21400, 21498, 21499, 21480, 21488)

21491 (21401, 21481, 1401)

21492 (21492, 21402)

This set gives a good example of the type of thing i'm seeing:

Number being misread the same way (0--> 9 and 8)
Sometimes a number is being missed entirely
Sometimes the right number isn't read at all...

It's not limited to 0, 8s and 9s, but these are the worst, so I'd like to try and understand which numeric characters are problematic (give them all a score), and build a model which takes a number, and knows a list of numbers it CAN be, and give me what number it should be, ideally with a confidence metric.

Anyone done this before and have any ideas?

The first step would be to find how similar a candidate number is against any number in the reference list. I think this is a perfect case for a character-based string similarity measure, typically the Levenshtein edit distance.

In case it's possible to have several matches, there could be a second step which would predict the most likely match, maybe based on the frequencies of the number.


Can I force DecisionTreeClassifier to use integer conditions when the variable is integer? I'm trying to visualize a decision tree in python for the purpose of explainability. I noticed that a condition like "NumGoals >= 1.23" could be quite vague for the user and I would much rather to see something like this: "NumGoals > 1".

So, I tried to switch the normalization off, and I even suspected that it might be a side effect the input matrix's dtype.

This is a simple code to reproduce the results.

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier, plot_tree
import matplotlib.pyplot as plt
import numpy as np

iris = load_iris()

clf = DecisionTreeClassifier()
clf = clf.fit(np.array(iris.data, dtype=int), iris.target)
plot_tree(clf)
plt.show()

So, is it even possible to tell the tree to treat integers as integers?

Just for visualization you could use the precision parameter of plot_tree():

precision: int, optional (default=3)
Number of digits of precision for floating point in the values of impurity, threshold and value attributes of each node.

(see here)
However, since it also applies precision to impurity you might want to turn that off. Here is an example:
tree.plot_tree(clf) gives

tree.plot_tree(clf, precision=0, impurity=false) gives

(If gini importance was shown, all gini values would be 0 here)
Since it uses $\leq$ and all split thresholds are between the closest $X$ values the split criteria should still be correct. Even if your tree does not use integers. But I'd always double check and compare both plots.


Seeking Career guidance I am a recent graduate.Currently I work in a late stage startup (product company) as a Software Engineer where my responsibilities are to interact with customers, understand their environment and write code so that they can use our product. I work on C#, web services and relational databases. I also work rarely on Big Data space to migrate some small components of our product to big data platform. I've got an opportunity to work in a large firm as a big data engineer to work on apache projects like yarn and hadoop and solve customer issues. And also to contribute to open source community. In curreny company, there is no one to review my work and mentor me.
Will it be a wise decision to leave a company where I've an opportunity to see product from end to end and go to a company where I'll not the same visibility of product ?

Is my next opportunity a good one if I want to make my career in big data space ?

Please share your inputs to help me take my decision ?

I think your question is off-topic, but I'll still try to answer because it might help you. Getting a job in data science isn't something everybody can do, it's a competitive field. Well done, for landing yourself in data science. However, data science is a big area, so you'll need to think what you want to do and plan ahead.

Do you want to get yourself into machine learning? Do you want to get yourself into statistics? Right now, you're more like a software engineer in the field, still not quite being a data scientist. Do you want to become a real data scientist? Or are you happy being a software engineer in the field? If you're planning to become a data scientist, you should train yourself hard in statistics, machine learning etc. Being a Hadoop user is simply not enough to step yourself fully into the field.

I give you my own experience. I was a software engineer myself. It wasn't easy to get a job in data science. I had to do a postgraduate degree in mathematics. Trained myself as much as possible. In the end, I landed myself a job as a bioinformatician.

Recommendation

Don't leave just because nobody mentors you. Learn as much as possible and prepare yourself. Once you're comfortable, move on.


How to get windspeed when temperature is maximum for each city? Here is the sample dataset, I have Weather_Data,
I want to calculate
"What is the windspeed when temperature is maximum for each city"
I have tried following code:

df = pd.read_csv('weather_by_cities.csv')
g = df.groupby('city')
g.max()

But Instead of getting corresponding Wind Speed, when temperature is maximum, it's giving maximum wind speed along with maximum temperature, which is little annoying.

First, you should find maximum temperature for each city, then merge it with the main data to find the relative windspeed

city_and_max_temp = data.groupby('city').agg('max')[['temperature']]
data.merge( right=city_and_max_temp ,how='inner')[['city','windspeed']]


Classic sport match prediction So, I am currently learning machine learning and data analysis. I have created for my self a problem that is:
Who will win a match of soccer
Now, I have narrowed it down as being a Binary Classification problem as I only want to figure out who will be the winner of a match.
For this I have some data containing the following data points:

Team One
Team Two
Duration
Goal (Team One)
Goal (Team Two)
Winner

Now, this is where the trouble begins (for me at least) I am unsure which features to choose for my model and also kind of off as to how to manage the data.
Say that I have two teams playing (Liverpool vs Chelsea) now I do have around 5000 data points for all matches played however only around 82 points where it is Liverpool against Chelsea.
Which dataset should I use?
Also sometimes their position in the dataset changes meaning that sometimes Liverpool is Team One and sometimes Chelsea is Team One does this matter or should I process the dataset to always match one team at a certain position?
In general, what is the best way to train my model? Should I use the small dataset containing only the matches between these two teams or should I go for all matches in general?
I am sorry for the beginner question I really hope someone can help me out :)

When making decisions about which data to use in a model you have to be aware of several pitfalls. One of them is information leakage i.e. including data that contains information that you shouldn't have at the time of prediction.
Both Duration and Goals are data points that you do not have at the time of prediction (that is before a match) and therefore should not be used in a prediction model.
This leaves us with only one information point: the teams involved. You could still make a ML model but it would only tell you something trivial: which of these teams won more head 2 head games in the past because that is the only information it would use to determine the likely winner (therefore the model would also always predict the same outcome because the input does not change).
This means you need much more data to actually make a model. Modern models for predicting the outcomes of sport events use data such as recent performance of the teams, time (e.g. time of day, weekday, season, etc.), recent performance of all involved players, context (e.g. weather, etc.).
It does not seem like your dataset contains this information so I am afraid you will be unable to create a model.
I would recommend to start with a different problem that has a better dataset available and is easier to crack for a beginner. The IRIS dataset is a wellknown beginner ML problem and teaches you how to model a classification algorithm.


Merging two pandas dataframes I have two csv files,and if I do:

df_compiler = pd.read_csv('sub_compiler.csv')

I obtain:

and for the other csv file I do:

df_opt = pd.read_csv('sub_opt.csv')

I obtain:

now what I would like to do is merge these two csv files and obtain something like:

can somebody please help me doing this? Thank's in advance.

I would normally suggest something like

merged_df = pd.merge(df_compiler, df_opt, on="instructions")

But the third image you posted does not really look like the inner merge of the other two. Can you describe what exactly you are trying to achieve in a little more detail? The third one looks like ten times longer than the other two.


Python library that can compute the confusion matrix for multi-label classification I'm looking for a Python library that can compute the confusion matrix for multi-label classification.

FYI:

scikit-learn doesn't support multi-label for confusion matrix)
What is the difference between Multiclass and Multilabel Problem

Also take a look at scikit-multilearn. It is a very good library that extends sklearn for multi-label learning. However, I'm not sure how the confusion matrix works for multi-label problems...

This guy claims he has solved it.


How to interpret this Plot of Model Loss from a BiLSTM model? Hi everyone,
the above graph is produced by a BiLSTM model i just trained and tested. I can't seem to interpret it while it is very different from the references that i acquired by googling online.
The graph above has a plateau appearing at the very beginning of the value loss. Shall I set my epochs to smaller than 20?
My model is like this:
prepared_model = model.fit(X_train,y_train,batch_size=32,epochs=100,validation_data=(X_test,y_test), shuffle=False)

and how do you interpret it?
thank you guys.

It looks like your train/val loss curves have a very large generalisation gap, which suggests that your model is overfitting. THis simply means it does a great job making predictions for the training set but a terrible one for your validation set. This appears to be the case even in early epochs, since valid loss appears to never improve.
I see you have shuffle set to False. Is that related to shuffling datapoints in the batches? The unfortunate behaviour in training may as well trace back to the train and validation sets being very different. I suggest

stratified train/val split
QA your train and val sets (e.g. class ratio in each set)
shuffle datapoints in your batches


Should I eliminate all ID columns and similar columns from training data? This is a basic question so bear my ignorance. I feel like they contribute collectively in no way to the target. This is for performance and accuracy.
The target is polar (0,1).

It depends.

If your data samples are IID (independent and identically distributed) then you can remove the sample ID, given that all samples come from or refer to the same source/object and they don't somehow identify the sample class.

But, if your data is sequential, such as time series, it would be a big mistake to neglect the ID, as it identifies the time order between the samples. The time dependencies are a great source of information for problems such as time series forecasting, regression and classification. If you treat sequential data (time series) as IID (by removing the ID), you will have inferior performance, because you neglect one source of information, i.e. temporal dependencies. You should not include the ID as input to your model, but you should respect the order of the samples, which is shown by the ID.


Why does my manual derivative of Layer Normalization imply no gradient flow? I recently tried computing the derivative of the layer norm function (https://arxiv.org/abs/1607.06450), an essential component of transformers, but the result suggests that no gradient flows through the operation, which can't be true.
Here's my calculations:
$\textrm{Given a vector of real numbers $X$ of length $N$, indexed as $x_i$,}\\
\textrm{we define the following operations:}\\
\mu =\frac{\sum_{k=1}^{N}{x_k}}{N}\\
\sigma = \sqrt{\frac{\sum_{k=1}^{N}{(x_k-\mu)^2}}{N}}\\
y_i=\frac{(x_i-\mu)}{\sigma}\\
\textrm{We seek to calculate the derivative of $y_i$ w.r.t $X$. That is,}\\
\frac{dy_i}{dX} = \sum^{N}_{k=1}\frac{dy_i}{dx_k}\\
\textrm{By the quotient rule:}\\
\frac{dy_i}{dx_j}=\frac{(x_i-\mu)'\sigma-(x_i-\mu)\sigma'}{\sigma^2}\\
(x_i-\mu)'=\delta_{ij}-\mu'\\
\mu'=\frac{1}{N}\\
\implies(x_i-\mu)' = \delta_{ij}-\frac{1}{N}\\
\sigma'=\frac{1}{2}(\frac{\sum_{k=1}^{N}{(x_k-\mu)^2}}{N})^{-\frac{1}{2}}*[\frac{\sum_{k=1}^{N}{(x_k-\mu)^2}}{N}]'\\
[\frac{\sum_{k=1}^{N}{(x_k-\mu)^2}}{N}]'=\frac{1}{N}\sum_{k=1}^{N}2*(x_k-\mu)(\delta_{kj}-\frac{1}{N})\\
\qquad =\frac{2}{N}\sum_{k=1}^{N}(x_k-\mu)\delta_{ij}-(x_k-\mu)\frac{1}{N}\\
\textrm{Note that $\delta_{kj}$ is only 1 when when $k=j$ and 0 otherwise, so we can further reduce:}\\
\qquad =\frac{2}{N}((x_j-\mu)-\sum_{k=1}^{N}(x_k-\mu)\frac{1}{N})\\
\qquad =\frac{2}{N}((x_j-\mu)-\frac{1}{N}\sum_{k=1}^{N}(x_k)+\frac{1}{N}\sum_{k=1}^{N}\mu)\\
\qquad =\frac{2}{N}((x_j-\mu)-\mu-\frac{1}{N}N\mu)\\
\qquad =\frac{2}{N}(x_j-\mu)\\
\textrm{Thus plugging that back into $\sigma'$ we get:}\\
\sigma'=\frac{1}{2}(\frac{\sum_{k=1}^{N}{(x_k-\mu)^2}}{N})^{-\frac{1}{2}}*\frac{2}{N}(x_j-\mu)\\
\quad=\frac{1}{N}(\frac{1}{\sigma})*(x_j-\mu)\\
\quad=\frac{(x_j-\mu)}{N\sigma}\\
\textrm{Now that we have all the components we can return to the derivative $\frac{dy_i}{dx_j}$:}\\
\frac{dy_i}{dx_j}=\frac{(x_i-\mu)'\sigma-(x_i)\sigma'}{\sigma^2}\\
\qquad=\frac{(x_i-\mu)'\sigma}{\sigma^2}-\frac{(x_i-\mu)\sigma'}{\sigma^2}\\
\qquad=\frac{\delta_{ij}-\frac{1}{N}}{\sigma}-\frac{(x_i-\mu)\frac{(x_j-\mu)}{N\sigma}}{\sigma^2}\\
\qquad=\frac{\delta_{ij}-\frac{1}{N}}{\sigma}-\frac{(x_i-\mu)(x_j-\mu)}{N\sigma^3}\\
\qquad=\frac{1}{N\sigma}(N\delta_{ij}-1-\frac{(x_i-\mu)(x_j-\mu)}{\sigma^2})\\
\qquad=\frac{1}{N\sigma}(N\delta_{ij}-1-\frac{(x_i-\mu)}{\sigma}\frac{(x_j-\mu)}{\sigma})\\
\qquad=\frac{1}{N\sigma}(N\delta_{ij}-1-y_iy_j)\\
\textrm{Finally, returning to $\frac{dy_i}{dX}$:}\\
\frac{dy_i}{dX}=\sum^{N}_{j=1}\frac{1}{N\sigma}(N\delta_{ij}-1-y_iy_j)\\
\textrm{Note that we are adding $N$ once (when $i=j$) and $(-1)$ $N$ times, so we can simplify to:}\\
\frac{dy_i}{dX}=\frac{1}{N\sigma}(N+(-1)N-\sum^{N}_{j=1}y_iy_j)\\
\quad=\frac{1}{N\sigma}(-\sum^{N}_{j=1}y_iy_j)\\
\quad=\frac{1}{N\sigma}(-y_i\sum^{N}_{j=1}y_j)\\
\quad=\frac{-y_i}{\sigma}\frac{(\sum^{N}_{j=1}y_j)}{N}\\
\quad=\frac{-y_i}{\sigma}\mu_y\\
\textrm{BUT by properties of data following a standard normal distribution $\mu_y=0$, so}\\
\frac{dy_i}{dX}=\frac{-y_i}{\sigma}0\\
\quad=0\\
\textrm{Which means no gradient flows through a layer normalization}\\\\$
I'm almost certain I've simply made a mistake somewhere, so if someone could point it out I'd greatly appreciate it. Thanks!

If you're computing the deriviative of the layer norm for the purpose of using it in backprop, then you need to compute the derivative with respect to the parameters of the layer, not it's inputs.
i.e., fix $x$ and compute $\frac{\partial y }{\partial \mu }$ and $\frac{\partial y }{\partial \sigma }$
Regarding your derivation specifically
I did not look at it closely, partly because I suspect that its not actually the derivative you care about as I said above.  But it looks like you're over-complicating things in the sense that you should treat $\mu$ and $\sigma$ as fixed variables and not as functions of $x$.
If they're functions of $x$, it may be that you're running into something like this:
You define $y = \frac {x - f(x)}{g(x)}$ and you compute $\frac{\partial y }{\partial x }$
... quotient rule ... computation... re-arrange...whatever
then you plug $f(x) = x$ in to the result and correctly observe that $\frac{\partial y }{\partial x } = 0$. Which is kind of obvious if you plug in at the start, but has been obscured because of notation. Now, where you started is not as simple as what I've written, but this same thing could happen.


Mixture Density Network: determine the parameters of each Gaussian component I am reading Bishop's Mixture Density Network paper at:
https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/bishop-ncrg-94-004.pdf

This is a good paper, but I am still confused about a few small details. I am wondering if anyone could give me some help:

Basically the mixing coefficient alpha_i can be computed through a softmax function in eq (25) below. However, in eq (25), what's the upper alpha for each $(z_i)^{alpha}$? Is it a free parameter to be fitted?
Similarly, in eq (26), what's the upper sigma in $(z_i)^{sigma}$? Is it a free parameter to be fitted as well? Thanks!

Actually, the upper alpha and the upper sigma are not free parameters to be set, they are just used to represent the output activations corresponding to the mixture coefficients and the variances, respectively. They are used to distinguish derivatives with respect to the alpha and sigma. I say it from page 275 of the book “pattern recognition and machine learning” by Christopher Bishop”:


Data Structure For Multilevel Analysis I am little confused about how to structure my specific data for multilevel analysis.

I have 10 categories and each category has some items in them.

The dataset is available for 117 weeks. There is quantity share and price index available on all of them.

My usual regression takes place with predicting the quantity mix by using the price indices for items within a category. This gives me regression coefficients for items but by treating every category separately.

I aim to use the multi level modeling approach to see the regression coefficients effects between items and categories.

This makes the model quite complicated and I’m not sure how to structure the data.

Take a look at mixed effects models. These give you a natural way to deal with heiractical and nested data.


How to create custom Keras metric using multiple functions with numpy arrays and matrices? I'm training a model for predicting position and orientation using regression.
I want to implement custom metric using last 3 values of model output to calculate and minimize angular error defined by function like this:
def angular_error(data_true, data_pred):

  # Axes orientation in radians (X,Y,Z)
  axes_orientation_gt = np.transpose(data_true[2:5])
  axes_orientation_pred = np.transpose(data_pred[2:5])

  axes_size = 0.1  # Axes_size in meters

  # Convert euler angles to rotation matrix (XYZ)
  _, Rx_gt, Ry_gt, Rz_gt = eul2rotmat(axes_orientation_gt)

  # Rotation matrix XYZ
  R_gt = Rz_gt * Ry_gt * Rx_gt

  # Y-axis end
  y_a_gt = R_gt * np.transpose(np.matrix([0, 1, 0])) * axes_size

  # Predictions
  # Convert euler angles to rotation matrix (XYZ)
  _, Rx_dnn, Ry_dnn, Rz_dnn = eul2rotmat(axes_orientation_pred)

  # Rotation matrix XYZ
  R_dnn = Rz_dnn * Ry_dnn * Rx_dnn

  ....

  return gaze_ang_err

Is there a way to avoid using Keras/TF functions instead of every numpy function appearance/usage? This function also calls two more functions with numpy and math operations so it seems like an overkill.
Any input appreciated...

Solved it using tf.py_function() as a wrapper function around the angular_error() function and added it to callbacks.


Gini coefficient vs Gini impurity - decision trees The problem refers to decision trees building. According to Wikipedia 'Gini coefficient' should not be confused with 'Gini impurity'. However both measures can be used when building a decision tree - these can support our choices when splitting the set of items.

1) 'Gini impurity' - it is a standard decision-tree splitting metric (see in the link above);

2) 'Gini coefficient' - each splitting can be assessed based on the AUC criterion. For each splitting scenario we can build a ROC curve and compute AUC metric. According to Wikipedia AUC=(GiniCoeff+1)/2;

Question is: are both these measures equivalent? On the one hand, I am informed that Gini coefficient should not be confused with Gini impurity. On the other hand, both these measures can be used in doing the same thing - assessing the quality of a decision tree split.

No, despite their names they are not equivalent or even that similar.

Gini impurity is a measure of misclassification, which applies in a multiclass classifier context.
Gini coefficient applies to binary classification and requires a classifier that can in some way rank examples according to the likelihood of being in a positive class.

Both could be applied in some cases, but they are different measures for different things. Impurity is what is commonly used in decision trees.


What is feature channels mentioned in U-Net? I was reading the U-Net paper for medical image segmentation. I had a doubt in the architecture. The authors mention that the max pooling layers in contraction path double the number feature channels while Downsampling.
Can anyone explain what are feature channels and how are they doubled while max pooling? In simple language please.

The feature channels simply mean the number of channels at a given point in the network. In the U-Net architecture, the number of channels doubles after the max pooling layers, see for example the first max pooling layer. The input is of size 568 by 568 pixels with 64 (feature) channels, after max pooling this becomes an array of size 284 by 284 pixels with the same 64 channels. The next convolutional layer then keeps the same size, so 284 by 284 pixels, but doubles the number of channels to 128.


How to handle Memory issues in training Word Embeddings on Large Datasets? I want to train a word predictability task to generate word embeddings. The document collection contains 243k documents. The code implementation is in torch. I am struggling with the huge size of the dataset and need ideas on how to train word embeddings on such a large dataset which is a collection of 243 thousand full article documents. The research computing resource is timed so get short access to GPU node and so opted for Incremental model training:

Incremental Model training: One way to train on entire dataset is to use incremental model training that is train the model on one chunk of the data and save it. Later on pick up the same pre-trained model and start training on it next chunk of data. The problem that I am facing in this approach is that how do I maintain the vocabulary/dictionary of words. In word embedding methods dictionary/vocab plays an important role. We sweep over all documents and create vocab of words that have count greater than a minimum set frequency. Now, actually this vocab is a hash map which has index associated to each word and in training samples we replace words with their indices in the vocab for the simplicity in the model. In the case of incremental training how do I create dictionary incrementally? Do I have to create vocab/dictionary on entire documents initially and then train incrementally? Or is the a way to extend vocab also in incremental training?
Another problem is memory limit on the size of the vocab data structure. I am implementing my model in Torch which is LUA based. So, LUA puts a limit on tables size, I cannot load vocab for entire documents in a single table. How to overcome such memory issues?
Taking inspiration from Glove vectors. In their paper they say that they “We trained our model on five corpora of varying sizes: a 2010 Wikipedia dump with 1 billion tokens; a 2014 Wikipedia dump with 1.6 billion to- kens; Gigaword 5 which has 4.3 billion tokens; the combination Gigaword5 + Wikipedia2014, which has 6 billion tokens; and on 42 billion tokens of web data, from Common Crawl5. We tokenize and lowercase each corpus with the Stanford tokenizer, build a vocabulary of the 400,000 most frequent words6, and then construct a matrix of co- occurrence counts X” . Any idea on how Glove vectors trained on such a big corpus and big vocabulary and how memory restrictions in their case might have got handled?  Paper reference - http://nlp.stanford.edu/pubs/glove.pdf
Any ideas on how to limit the size of the dataset for generating word embeddings? How would it affect the performance or coverage of word embeddings with the increase or decrease in number of documents? Is it a good idea to use sampling techniques to sample documents from dataset? If yes, then please suggest some of the sampling techniques.

I'm not familiar with Torch, but since basically word2vec and doc2vec are considered, These models learn from each sentences and so there is no need to have all the sentences in the memory. You could iterate via each sentence in the corpora and let the model learn from each of the sentence. And that is probably how people train on huge corpora with or without high computation machines.

A short example in python:

class SentenceIterator(object):
   def __iter__(file_name):
      for line in open(file_name)
         yield line

sentences = SentenceIterator("path/to/file")
for line in sentences:
    model.train(line)

In this way, The memory is loaded with only one sentence at a time and when it is done the memory loads the next one.
For building the vocabulary, you can do the whole iterating via all the documents to build the vocab first and then train the data, depending upon the word-embedding functions implemented.


Can I create a good Speech Recognition Engine while having millions of recorded conversations? I have at my disposal millions of wav files containing recorded conversations between employees and clients, i'm doing some research on the possibility of creating a good speech recognition engine. I've tested Google's Speech-To-Text and it's great. Is it possible to create something similar? ( Of course, no one can beat the quality and quantity of the data Google has but how close one can get?). And of course, what are the technical limits ( like hardware needed for this kind of learning ) and how much time should it take to achieve it?

Note : i'm a beginner in ML, so far , i've done some binary and multiclass classification,I have an idea on Neural Networks but no work done on that. The simpler the answer the easier for me to understand , Thanks!

Yes, having lots of recorded conversations is great for building a speech recognition system. You will still have to create training samples (Each sample will be parts of Wave file --> text), but you will need lesser number of samples.

High level steps are :

Train a GAN on raw audio
Train a Language model on raw text data (it needs not be from these conversations, but has to be from same domain). For example, if conversations are related to medical, train language model on medical text.
Merge these models and train on labeled samples

For step 1, Google WaveNet is a good example (it is eventually used for Text-to-Speech, it is a component in Speech-to-Text as well)

https://deepmind.com/blog/wavenet-generative-model-raw-audio/

Papers that cover design and overall approach :

https://arxiv.org/abs/1711.01567
https://arxiv.org/abs/1803.10132


How can I increase my accuracy avoiding overfitting? CNN-Keras-VGG16 As I asked in this question: Why are my predictions bad, if my accuracy in train is roughly 100% (Keras CNN) , my problem was Overfitting, so, I reduce the number of layers, and now I have this model:

vgg16_model = VGG16(weights="imagenet", include_top=True)

# (2) remove the top layer
base_model = Model(input=vgg16_model.input,
                   output=vgg16_model.get_layer("block5_pool").output)

# (3) attach a new top layer
base_out = base_model.output
base_out = Reshape((25088,))(base_out)
# output layer: (None, 5)
top_preds = Dense(1, activation="sigmoid")(base_out)

# (4) freeze weights until the last but one convolution layer (block4_pool)
for layer in base_model.layers[0:14]:
    layer.trainable = False

# (5) create new hybrid model
model = Model(input=base_model.input, output=top_preds)

# (6) compile and train the model
sgd = SGD(lr=1e-4, momentum=0.9)
model.compile(optimizer=sgd, loss="binary_crossentropy", metrics=["accuracy"])

But, when I predict some images, the class 0 accuracy is roughly 96%, but the accuracy of class 1 is roughly 58%. So how can I increase the accuracy without overfitting.

I've trained my model with 700 images each class and to test I have 50 images each class.

Keras has support for image preprocessing. You can do lateral translations and shears on your images which will alter them but since the content is still the same your have more labeled data to train on. This is known as data augmentation.

Another thing you can do is introduce dropout in your network. This drops nodes in your network based on the probability you provide. This is done to prevent dependency of neurons on each other in your network.

Both of these where introduced in this paper. You can go through the reduce overfitting section to get a better idea and understanding.


Siamese Network in Keras I‘m looking for a minimal applied example for the implementation of a (one shot) Siamese Network, preferably in Keras.

I‘m well aware of the various data science online pages and the respective examples and exercises that can be found there. However, so far I did not found an instructive source there.

I would be thankful if someone could point me to some github source or if someone could share some code or other sources, which provide a sound example on how to implement such an architecture in Python or in R.

Here is one in keras thats not that basic, but still interesting.


k-fold cross validation with RNNs is it a good idea to use k-fold cross-validation in the recurrent neural network (RNN) to alleviate overfitting?

A potential solution could be L2 / Dropout Regularization but it might kill RNN performance as discussed here. This solution can affect the ability of RNNs to learn and retain information for longer time.
My dataset is strictly based on time series i.e auto-correlated with time and depends on the order of events. With standard k-fold cross-validation, it leaves out some part of the data, trains the model on the rest while deteriorating the time-series order. What can be an alternate solution?

TL;DR
Use Stacked Cross-Validation instead of traditional K-Fold Cross-Validation.
Stacked Cross-Validation
In Sckit-learn, this is called TimeSeriesSplit (docs).
The ideas that instead of randomly shuffling all your data points and losing their order, like you suggested, you split them in order (or in batches).
Belows is a picture of traditional K-Fold vs Stacked K-Fold. They both have K=4 (four iterations). But in the traditional K-Fold all the data is used all the time, whereas in the Stacked K-Fold only the past data is used for training and the now data for testing.

Strategy

Split your data in a Stacked K-Fold fashion.
At every iteration, store your model and measure its performance against the test set.
After all iterations are over, pick the stored model with the highest performance (might not even be the last one).

Note: This is a common approach in training neural nets (batching with validation sets).


Neural Network unable to track training data I am new to ML and this is my first Tensorflow project. I am doing regression with Neural Networks on a dataset with 17 features and 1 outcome. But for some reason my network is unable to follow the training data. I am getting massive errors in results, as can be seen from the plots below. I have also tried experimenting with different parameters (learning rate, nodes per layer, number of layers etc) but nothing seems to work.
I have pasted the Tensorflow code here. I have also provided my cost and training plots as well as a link to my datasets. I'd be grateful if someone could please help me figure out what I am doing wrong.
Thank you!

Links to the datasets-
Features arranged in columns-
https://drive.google.com/file/d/1U182Lhf67WygeSbv6BNEx5LHyL7Ba13O/view?usp=sharing
Output column -
https://drive.google.com/file/d/10XWo1d5mhIsxccQBgAyGDWDAVgu2BjAA/view?usp=sharing

import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf

# importing features and observations data for training and validation
training_filename_X = "training_set_X.csv"
training_filename_Y = "training_set_Y.csv"
validation_filename_X = "validation_set_X.csv"
test_filename_X = "test_set_X.csv"
test_filename_Y = "test_set_Y.csv"
validation_filename_Y = "validation_set_Y.csv"
training_features = np.loadtxt(training_filename_X, delimiter=',')
training_observations = np.loadtxt(training_filename_Y, delimiter=',')
validation_features = np.loadtxt(validation_filename_X, delimiter=',')
validation_observations = np.loadtxt(validation_filename_Y, delimiter=',')
test_features = np.loadtxt(test_filename_X, delimiter=',')
test_observations = np.loadtxt(test_filename_Y, delimiter=',')

# normalizing training data
training_features_stddev_arr = np.std(training_features, axis=0)
training_features_mean_arr = np.mean(training_features, axis=0)
normalized_training_features = (training_features-training_features_mean_arr)/training_features_stddev_arr

# layer parameters
n_nodes_hl1 = 5
n_nodes_hl2 = 5
n_nodes_hl3 = 3
no_features = 17
learning_rate = 0.001
epochs = 2000

cost_history = []

X = tf.placeholder(tf.float32)
Y = tf.placeholder(tf.float32)

# defining weights for each layer taken from a normal distribution with variance 2/n
hl1_weight = tf.Variable(tf.random_normal([no_features, n_nodes_hl1], stddev=np.sqrt(2/no_features)))
hl2_weight = tf.Variable(tf.random_normal([n_nodes_hl1, n_nodes_hl2], stddev=np.sqrt(2/n_nodes_hl1)))
hl3_weight = tf.Variable(tf.random_normal([n_nodes_hl2, n_nodes_hl3], stddev=np.sqrt(2/n_nodes_hl2)))
output_weight = tf.Variable(tf.random_normal([n_nodes_hl3, 1], stddev=np.sqrt(2/n_nodes_hl3)))

# defining biases for each layer
hl1_bias = tf.Variable(tf.random_uniform([n_nodes_hl1], -1.0, 1.0))
hl2_bias = tf.Variable(tf.random_uniform([n_nodes_hl2], -1.0, 1.0))
hl3_bias = tf.Variable(tf.random_uniform([n_nodes_hl3], -1.0, 1.0))
output_bias = tf.Variable(tf.random_uniform([1], -1.0, 1.0))

# defining activation functions for each layer
hl1 = tf.sigmoid(tf.matmul(X, hl1_weight) + hl1_bias)
hl2 = tf.sigmoid(tf.matmul(hl1, hl2_weight) + hl2_bias)
hl3 = tf.sigmoid(tf.matmul(hl2, hl3_weight) + hl3_bias)
output = tf.matmul(hl3, output_weight) + output_bias

# using mean squared error cost function
cost  = tf.reduce_mean(tf.square(output - Y))

# using Gradient Descent algorithm
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)

init = tf.global_variables_initializer()

# running the network
with tf.Session() as sess:
    sess.run(init)

    for step in np.arange(epochs):
        sess.run(optimizer, feed_dict={X:normalized_training_features, Y:training_observations})
#        print (sess.run(cost, feed_dict={X:normalized_training_features, Y:training_observations}))
        cost_history.append(sess.run(cost,feed_dict={X:normalized_training_features, Y:training_observations}))

    pred_y = sess.run(output, feed_dict={X:normalized_training_features})

    plt.plot(range(len(cost_history)), cost_history)

Ok I have found out atleast one thing why it's happening that way after asking my friends,

We can't use sigmoid as the last layer since it will always output something between 0 and 1 and hence that's why your predictions are not varying at all...
i.e we can't at all use sigmoid as the last layer until it's a classification of let's say images and all...


How do I convert a pandas dataframe to a 1d array? I'm trying to create a contour map from two variables which store some temperature values and a third variable which is the time stamp.
I used this notebook as a tutorial

  https://plot.ly/pandas/contour-plots/

I'm not able to convert the pandas dataframe created, into a 1d array. And the kde_scipy doesn't work with a nd-array. I tried converting the dataframe into a 1d array using .as_matrix() but this is the error I am receiving.

Degrees of freedom <= 0 for slice

How can I convert this CSV file (with 3 columns of data) imported as a dataframe into individual columns of data? Or can I directly import each column of data into a 1d array and use it in the function kde_scipy?

You can try this

import pandas as pd
import numpy as np
filename = 'data.csv'
df1 = pd.read_csv(filename)
#convert dataframe to matrix
conv_arr= df1.values

#split matrix into 3 columns each into 1d array

arr1 = np.delete(conv_arr,[1,2],axis=1)
arr2 = np.delete(conv_arr,[0,2],axis=1)
arr3 = np.delete(conv_arr,[0,1],axis=1)

#converting into 1D array
arr1 = arr1.ravel()
arr2 = arr2.ravel()
arr3 = arr3.ravel()

This should mostly do the job. Use the arr1 ,arr2,arr3 in the function you mentioned. They are the 1d array of the columns you split


Difference between n-dimensional of length m and m-dimensional of length n? With respect to classifying the MNIST digits I've seen the input representation referred to as both a one-dimensional array of 748 elements and a 784-dimensional vector.

Are these perspectives always interchangeable, or is there a material difference between the two?

They are just transposed version.

Often input shape wont matter for informational (performance purposes yes) what will matter is the information inside these vectors and wether its discriminative enough.


Word2Vec and Tf-idf how to combine them I'm currently working in text mining ptoject I'd like to know once I'm on vectorisation. With method is better.

Is it Word2Vec or Tf-Idf ?
Here I see we can combine them why that? Does it improve quality of data?
What about GloVe?

Thanks

Word2Vec algorithms (Skip Gram and CBOW) treat each word equally,
because their goal to compute word embeddings. The distinction
becomes important when one needs to work with sentences or document
embeddings; not all words equally represent the meaning of a
particular sentence. And here different weighting strategies are
applied, TF-IDF is one of those successful strategies.
At times, it does improve quality of inference, so combination is
worth a shot.
Glove is a Stanford baby, which has often proved to perform better. Can
read more about Glove against Word2Vec here, among many other
resources available online.


Which of the following linear regression model is better? Consider 2 regression models on the same data set.
Model 1 : $R^2 = 90$% , $R^2(adjusted) = 80 $%, $R^2(pred) = 70$%
Model 2 :  $R^2 = 60$% , $R^2(adjusted) = 59 $%, $R^2(pred) = 58$%
In the first model , all  values are very high but the difference between them is also very high.
In model 2, the values are low but difference between them is less.
Which of the above 2 models is better and why?

Which one of the two models is better?

It depends what you care about.

  Why?

If you just want the highest $R^2$, $R_{adj.}$ or $R_{pred}$, obviously Model 1 is better. In fact, in your example, all performance metrics would tell you to go for Model 1! But there are other aspects to picking a model...

Many people care about the density of explanability (I made that term up); think of it as the explanatory power normalised by number of degrees of freedom (also referred to as the elegance of the model). If that was your preference, then Model 2 look more attractive, as its metrics drop by only 1% and 1% again, compared to Model 1's 10% and 10% drop.

Some relevant background

There is the term Minimum Description Length, which formalises Ockham's Razor, stating:

  the best hypothesis (a model and its parameters) for a given set of data is the one that leads to the best compression of the data.

This can help decide which of two models is best, by assessing how dense the explanability of each model. In simple terms: assume models A and B both produce 80% accuracy, but A uses an extra parameter, then B is the more elegant model as it uses its input more efficiently. In other words, its compression of the information is higher.

Here is a really nice blog post (published today!) that explains how Ockham's Razor, Bayes' and Shannon Entropy all come together to equally define how thie density of information in a model is a really important criterion. Spoiler: all three boil down to the same guiding principal.

Another way people try to decide which regression model is best, is to use another metric such as the Akaike Information Criterion (AIC) or the Bayesian Information Criterion (BIC), both of which try to find a compromise between the performance of a model and the number of degrees of freedom (i.e. number of parameters) that were used. We pick the model with the lowest score. For example, the AIC is defined as:

$$
AIC = 2k-2 ln({\hat {L}}),
$$

where $k$ is the number of parameters and $\hat{L}$ is the performance metric (here the Maximum Likelihood is implied). So in plain English, the AIC gets larger and alrger as we include more parameters and out model becomes more complex. However, if the model fits the data more accuractely with each parameter, that is subtracted and so reduces the AIC value. The BIC essentially works the same way. I think it is intuitive what we are trying to do here.

Summary

Coming back to my first answer: "It depends what you care about" - it hopefully now clear that the selection is subjective. If you can live with high model complexity, you might just go for the best $R_{pred.}$ and so Model 1. On the other hand, if you enjoy an elegant model with high information density, the deltas of 1% between your three metrics will lead you to select Model 2.

EDIT

A term I was searching for came back to me: parsimony. A parsimonious model is one that reaches accuracy expectation and does so with a few parameters as possible. It is all about maximum information density and explanatory efficiency. Check out this thread that is relevant to your questions.


How to Ascertain Sine Wave and Harmonics I have some data that I've noticed conforms to a sine wave. Details of the data is unknown. My task is to approximate it as closely as possible.

From some experimentation in Excel, I noticed the data follows a sine wave, so attempted to plot in the base sine wave (orange) and am now trying to approximate the blue line. I believe this is a simple job of harmonics being added to the base sine wave, however; the quantity and their respective amplitude and harmonic number (2nd harmonic being 2 * f, 3rd harmonic being 3 * f etc) is what I now need to calculate. I'm also guessing the amplitude of the resulting wave is modified by possibly a sine wave too.

Is there software which could automate this and approximate it through some analysis (Fourier analysis maybe) or possibly brute force, or is it something I have to write myself?

You want to look at the spectral density to get an idea of the distribution of the frequency components. Check out the 'psd' package in r or the signal processing toolbox if you have access to Matlab.


Method for finding top-k cosine similarity based closest item on large dataset I have a dataset with 40 million item, where each item is 400-dimension double vector. What I want to do is to find top-k (small k, about 3~10) most similar items to an arbitrary given input vector. Similarity measure is cosine similarity, since this dataset is based on word2vec representation.

However, this data is so huge, so it couldn't fit at main memory (I'm now working in single machine). The goal I want to achieve is to find top-k similar items as fast as possible, with small memory(~5G) that could fit in RAM.

Any suggestions for this problem? I've already tried PCA, but I observed that this data projected in low-dimension wasn't working very well..

Locality sensitive hashing is a great tool for this problem.
Pick n random 400-dimensional vectors. (Be careful or not all directions will be chosen with equal probability; pick each dimension as a standard Gaussian.) Each really defines a hyperplane through the origin cutting your space in half. The sign of the dot product of any of these vectors with some new vector tells you which side of the hyperplane it's on. So computing n dot products gives n 0/1 bits, which make an n-bit hash.
Any new vector hashing to the same value must be in the same small sliver of space from the origin. And those are exactly the vectors that have a high cosine similarity with each other since their mutual angles are very small. Likewise anything hashing to almost the same value -- differing in a few bits -- is likely to be nearby. So you can restrict your search for most-similar vectors to things within one or more buckets of hashed candidate vectors.
It doesn't help directly with memory since you might need any particular bucket to satisfy a request. You also lose some accuracy since there is not a guarantee the most similar vectors lie in the buckets you examine (though it's likely, the more you examine). It lets you trade speed for accuracy mostly. However you may find you can get away with some caching scheme, where some buckets are rarely if ever accessed and so don't stay in memory.
You can see an implementation of this in Oryx, which I think is pretty straightforward.
Most of the complexity comes because it lets you specify a target percentage of vectors to evaluate, and works out the optimal hash size based on that and your machine's number of cores.


Reshaping Numpy arrays I am having a problem trying to reshape my numpy array. i have a 2 dimensional array and i want to make it a 3D array. I have looked everywhere but even if i find some answering my question they just make it complicated. I just want to know is there a mathematical way to calculate reshaping a numpy.

I'm not sure if I fully understand what you mean, but numpy.expand_dims can be used to expand an array at the given index, allowing you to convert a 2 dimensional array to a 3 dimensional array.

import numpy as np

array = np.array([[1,2],[3,4]])
print(array.shape)
# (2, 2)
array = np.expand_dims(array, 0)
print(array.shape)
# (1, 2, 2)


Can we use k fold Cross Validation without any extra (excluded) Test Set? I have seen this in two Papers:

The authors use 10 fold cross validation, and then present the results from this validation or even odder the results from the best Fold as their modelling Result.

there has been no testing data put aside to validate the final model. These are peer reviewed papers i am talking about. Is this right to do ? Can one even cite such sources?

The key question is

  Is K-fold cross validation is used to select the final model (or algorithm)?

If yes, as you said, then the final model should be tested on an extra set that has no overlap with the data used in K-fold CV (i.e. a test set).
If no, the average score reported from K-fold CV is the final test score, no extra set is required.

Therefore, if authors only picked the best model from all K models, it should have been tested on an extra set. The score of model on the validation fold (from K-fold CV) is not acceptable.

Also, here is a related post on this site (my answer) that goes into more detail about test and validation sets (scores).

EDIT:

I have found a similar question on stats.stackexchange.com. Also, this comment by amoeba suggestions "nested CV" instead of "CV + test set", which I think it is worth fleshing out here.

K-fold CV evaluation

1. For k = [1,..,K]
  1. tr = (K-1)/K of data, ts = 1/K of data
  2. m[k] = model trained using tr (can be further split into tr2 + v)
  3. score[k] = score of m[k] on ts
3. Test score = average of score[1],...,score[K]

K-fold CV selection and evaluation

1. tr = 80% of data, ts = 20% of data (or some other ratio)
2. For k = [1,..,K]
  1. tr2 = (K-1)/K of tr, v = 1/K of tr
  2. m[k] = model trained using tr2 (can be further split into tr3 + v2)
  3. score[k] = score of m[k] on v
3. M = best of m[1],...,m[K]
4. Test score = score of M on ts

Nested k-fold CV selection and evaluation

1. For k = [1,..,K]
  1. tr = (K-1)/K of data, ts = 1/K of data
  2. For k2 = [1,..,K2]
     1. tr2 = (K2-1)/K2 of tr, v = 1/K2 of tr
     2. m[k2] = model trained using tr2 (can be further split into tr3 + v2)
     3. score[k2] = score of m[k2] on v
  3. M = best of m[1],...,m[K2]
  4. score[k] = score of M on ts
2. Test score = average of score[1],...,score[K]

Note that in these algorithms, model trained includes parameter learning, hyper-parameter tuning, and model selection (of course, except for the outermost model selection). For example, for selecting between hyper-parameters 8 and 12, we need a deeper loop inside model trained.


Are there more layer types like convolution layers and fully connected layers? I've been learning about neural networks and I'm curious: Are there any other layer types like convolution and fully connected layers? I know there are things like max pooling or various regularization layers, but I'm mainly interested in more variations on fully connected layers (like how convolution layers do the same thing as fully connected layers but take the shape of data into account). It just seems strange that we wouldn't need anything except straight matrix multiplication or matrix multiplication on smaller chunks.

Do these exist? If not, why not?

Jan van der Vegt did a good job of mentioning some of the main irregular layer types, but I'll toss in a couple of others for the sake of completeness. I know from experience that layers in Teuvo Kohonen's Self-Organizing Maps can be fully connected, without requiring a square or rectangular matrix. I've gotten excellent results by creating topologies that match the underlying problems, which may feature jagged and possibly multidimensional shapes, then operating on them with sets instead of matrices. That's one reason I prefer using set-based languages like SQL to matrix math when coding neural nets. Another interesting neural net type worth exploring is Neural Gas, which expands across the space of data points to fit the natural topology of a problem, whatever that may be. It is easier to express in terms of manifolds than ordinary space. In a sense, it can be viewed as a generalization and abstraction of Kohonen nets.

Resources on Kohonen nets are easy to find, but the literature on Neural Gas is still scarce. Here are some worthwhile links I've read to Neural Gas and assorted variants:

• Martinetz and Schultern's original article on Neural Gas.

• The Wikipedia article on Neural Gas.

• A Master's Thesis on Growing Neural Gas, which dynamically creates the topology

• A Growing Neural Gas Network Learns Topologies, a research paper by Bernd Fritzke


Is it a good idea to normalize the outputs of a Neural Network for Regression, when the different outputs vary in magnitude? I understood that it is not necessary to scale the output of a neural Network when I predict a single value via regression.

Is it necessary do normalize the Outputs of my neural Network if I have multiple outputs that vary in magnitudes between 10^-2 and 10^4?
My intuition would tell me that the loss function might ignore the smaller values and only focus on the values of a bigger magnitude.

Yes, if proper weights were not introduced in cost function or target variables were not normalized, optimization process would be driven by the target variable which had the largest scale.

Therefore it's a good practice to normalize outputs if they have huge differences in terms of scale.


building a 2-layer LSTM for time series prediction using tensorflow From Tensorflow tutorials i am experimenting time series with LSTM

In the section 'multi-step prediction' using LSTM tutorial says

  Since the task here is a bit more complicated than the previous task, the model now consists of two LSTM layers. Finally, since 72 predictions are made, the dense layer outputs 72 predictions.

where previous task was prediction over a single point.

How do we know how many layers a problem requires (here, 2) ?

Then, from implementation point of view, using Python Tensorflow library,

multi_step_model = tf.keras.models.Sequential()
multi_step_model.add(tf.keras.layers.LSTM(32,
                                          return_sequences=True,
                                          input_shape=x_train_multi.shape[-2:]))
multi_step_model.add(tf.keras.layers.LSTM(16, activation='relu'))
multi_step_model.add(tf.keras.layers.Dense(72))

why is there a need for the adding a Dense(72) layer ? what is the function Dense() doing ? (reading the doc doesn't help really)

First question: How many layers?

This is architectural question and one of them most important when constructing NN. Generally the more complex the task the more layers you should use to approximate (until a certain point than there is overkill, motivation for ResNet)
If you are looking for some guidelines there are some good posts, but the research and general trend nowadays is that we are over-doing it in the first place, and we can achieve some good results with some smart tricks without making it too deep.

TL;DR it depends on the problem, but not as deep as we think for 98% of problems.

Second (third?) question: why is there a need for the adding a Dense(72) layer ? what is the function Dense() doing ? Well you said that Finally, since 72 predictions are made, the dense layer outputs 72 predictions. For the question what is dense function doing in short its producing an (output) vector. In long dense layer represents a matrix vector multiplication. Values in the matrix are actually the trainable parameters (weights) which get updated during backpropagation, and if you seen mathematically representation of NN with matrizes (which all of them are thats how you utilise power of GPU-s bla bla) thats exactly what this dense layer represents.


Data simulation using make_classification in Python I have a question about data simulation in Python. I deal with the classification of imbalanced data and want to test the effectiveness of different methods on simulated data. I have seen in various articles and books that the make_classification function is used to generate data. Then the data is generated from a normal distribution, so the data is continuous and not discrete. Are such data correct for classification (SVM, Decision Trees) research?

There is no obstacle to doing this. For example you can create data by make_classification, and compare different algorithms by building model on it. You can also pass a random_state value to obtain same data each time you call the function. Both SVM, and Decision Trees can work with continuous data.


Question of pretraining text-generation task, it seems that pretraining is not work for a small model? My task is to generate keywords from sentences.
I pretrain a text-generation model. I mask the sentences' tokens and predict the whole sentences' tokens.
Pretraining batch_size = 8 and step = 1000000
I haven't observed improvement from pretraining. BLEU score is 10.5 for not pretraining, BLEU score is 9.5 for pretraining.
Code
I take the python code from
https://github.com/google-research/pegasus/blob/master/pegasus/models/transformer.py#L38
hidden_size = 512
num_encoder_layers = 3
num_decoder_layers = 3
Discussion
The task is to generate keyword from sentences.
The keyword may not appear in the sentences.
So input masked sentences to predict whole sentences, it is not benefit the keywords generation task.
Input masked sentences to predict whole sentences, it do not have relation to the keywords generation task.
Am I right? Is it the reason that pretraining do not improve the BLEU score?
Thank you very much.

1, I pad some zeros in the input tokens for multi sentences. The output positions of output tokens should be exactly same to the input tokens, which means I should keep the padding zeros in the output tokens.
2, The pretraining time should be longer.


Applying neural network for simple x^2 function for demonstration purpose I have tried to train a neural network for a simple x^2 function

I developed training data in excel. First column (X) is =RANDBETWEEN(-5,5) i.e random integer between -5 and 5
Second column simply squares first column
And third column, my output 'y' column is 0 or 1. 0 if second column is less than 12.5 else 1

I made 850 training examples and used the first column as 'X' and third column as 'y'

However I am only able to get a training accuracy of 63%!

Where could I have gone wrong? I changed input_layer to 1 and tried hidden units between 5 and 35. Tried regularization lambda 0 to 2 but still only 63% accuracy! Where could I have gone wrong?

My predict function is p = 1 if h2(i)>0.5 else 0.

Any help will be much appreciated! :-)

I also noticed that my neural network's output is 0.3XXX for all training examples...how is this possible??

I re-implemented your set-up in python using keras. I used a hidden layer size of 25, and all my activations were sigmoid's. I got to an accuracy of 99.88%. Try running your algorithm for a greater amount of epochs. Use binary cross entropy as the loss function and try decreasing the learning rate of your gradient descent algorithm. This should help increase your accuracy. My only explanation for the poor performance would be that you are getting stuck at a local minimum, if that is the case different initiations of your weights should fix that problem.


How many times is backprop used in epoch? As I understand for the algorithms that use gradient descent we have to pass data to the algorithms multiple times so that the optimum is found.

So one epoch means that the forward-backprop (and updating weights with gradient descent) is done only once and in order to find the minimum we need to run the algorithm more times?

Or is forward-backprop combination done multiple times (like for every input of data that is passed to the input layer) in one epoch until the minimum for that epoch is found?

It depends on the type of gradient descent or respectively your batch size: One epoch means that your neural net (NN) has applied the forward pass on all examples of your training data, i.e. it has "seen" all training data. Now to do so you have at least two options (let $n$ be the number of samples in your training data):

You can either run backprop after each forward pass of an example and update your weights accordingly. This is usually implemented as stochastic gradient descent (SGD), has a batch size of 1 and means you run backprop $n$ times per epoch.
Alternatively, you can first run the forward pass for all your training data and then run backprop and update your weights accordingly. This is called batch gradient descent, has a batch size of $n$ and means you run backprop once per epoch.

Usually a mix of these two is applied so that $1 < \text{batch size} < n$, which is called Minibatch Gradient Descent. Following the above explanations, it means you'll run backprop $\frac{n}{\text{batch size}}$ times per epoch.

This blog post provides a good explanation as well.


Where to find statistically relevant documentation of common Python packages? I am trying to entice my lab to transition from Matlab and R to Python. The main objection at this point seems to be that Python's analytical libraries are not sufficiently well documented. Given how prolific Python is, I suspect that sufficiently detailed documentation exists and we simply cannot find it.

Example

Recently, I needed to upsample a signal (1D vector) and found a Python function called resample. In contrast to Matlab and R, pandas documentation doesn't tell me what kind of interpolation resample uses (linear, cubic splines, pchip?).

Is there any place where I could find such information about this and other libraries, preferably with equations or references to papers? I understand that I could analyze source code but this isn't most time efficient. If you know R, I am basically looking for a Python equivalent of CRAN (PDF warning).

Thanks!

I think the documentation, at least for your example, is not too obscure. It doesn't tell you what kind of interpolation it uses because it doesn't use any: resample is a deferred operation. In order for it to work you have to call it in conjunction with a function that performs the interpolation, e.g. series.resample(frequency).mean() or series.resample(frequency).interpolation('cubic') (as can be seen from the examples in the documentation). Sometimes, it can be helpful to also look into the release notes for additional information.

Scipy and Numpy are sometimes not very detailed, but they contain at least a decent number of references with more information. (There is also a section on Matlab versus Numpy here. If you are familiar with dplyr in R , then the pandas cheat sheet can help as a quick comparison between R and pandas for typical data wrangling operations.)

I'm not aware of a "second more detailed documentation" that encompasses the entire libraries. You probably have to fall back to books (which, of course, aren't complete or always up to date).


R, xgboost: eval_metric for count:poisson I wonder what are the recommended eval_metrics for count:poisson as objective in xgboost in R?

This is from XGBOOST documents:
“poisson-nloglik”: negative log-likelihood for Poisson regression.

You might want to go and read more about poisson n loglikelihood.

Here is a link for you to get started:
https://onlinecourses.science.psu.edu/stat504/node/31

Hope this can be helpful


How to apply supervised machine learning when the target label depends on multiple input rows? The problem is a multi-label classification problem.

Now, I know how to train and classify using single row with several attributes. For example, if the dataset looks like the first table from the attached file. Here, each row is associated with a single label. Thus, I can train and test after separating the dataset into training and testing sets.
But the problem occurs when classification label / target label depends on multiple rows such as the second table from the attached file. Consecutive N rows makes one category.
Can you please guide me towards a solution?

Is it possible to fit this problem in any existing tool? For example, WEKA or Neural network using Keras.
Or do I have to change the algorithm in order to fit the problem! Is there any existing solution?
Or do I need to modify the rows in such a way that it transforms into one?

What about doing a concatenation of your rows (i.e. Attr1 -> Attr12) , such that you now have 3*4 features (because 4 rows of 3 features) as an input to a multiclass classification model?

For instance, first sample would be described by :

X = [1.1, 1.4, 2.5, 2.3, 2.5, 2.7, 1.1, 1.6, 1.9, 1.5, 1.6, 1.7]
y = "A"

Otherwise, there is no issue in giving 2D or 3D inputs towards a classifier. Take for example convolutional neural networks that do operations on images!


Is the percepetron algorithm's convergence dependent on the linearity of the data? Does the fact that I have linearly separable data or not impact the convergence of the perceptron algorithm?
Is it always gonna converge if the data is linearly separable and not if it is not ? Is there a general rule ?

Yes, the perceptron learning algorithm is a linear classifier. If your data is separable by a hyperplane, then the perceptron will always converge. It will never converge if the data is not linearly separable.

In practice, the perceptron learning algorithm can be used on data that is not linearly separable, but some extra parameter must be defined in order to determine under what conditions the algorithm should stop 'trying' to fit the data.

For example, you could set a maximum number of iterations (or epochs) for the algorithm to run, or you could set a threshold for the maximum number of allowed misclassifications.


Large mean squared error in sklearn regressors I'm a beginner in machine learning and I want to build a model to predict the price of houses. I prepared a dataset by crawling a local housing website and it consists 1000 samples and only 4 features (latitude, longitude, area and number of rooms).

I tried RandomForestRegressorand LinearSVR models in sklearn, but I can't train the model properly and the MSE is super high.

MSE almost equals 90,000,000 (the true values of prices' range are between 5,000,000 - 900,000,000)

Here is my code:

import numpy as np
from sklearn.svm import LinearSVR
import pandas as pd
import csv
from sklearn.preprocessing import normalize
from sklearn.model_selection import train_test_split

df = pd.read_csv('dataset.csv', index_col=False)
X = df.drop('price', axis=1)

X_data = X.values
Y_data = df.price.values
X_train, X_test, Y_train, Y_test = train_test_split(X_data, Y_data, test_size=0.2, random_state=5)

rgr = RandomForestRegressor(n_estimators=100)
svr = LinearSVR()

rgr.fit(X_train, Y_train)
svr.fit(X_train, Y_train)

MSEs = cross_val_score(estimator=rgr,
                         X=X_train,
                         y=Y_train,
                         scoring='mean_squared_error',
                         cv=5)

MSEsSVR = cross_val_score(estimator=svr,
                         X=X_train,
                         y=Y_train,
                         scoring='mean_squared_error',
                         cv=5)

MSEs *= -1
RMSEs = np.sqrt(MSEs)

print("Root mean squared error with 95% confidence interval:")
print("{:.3f} (+/- {:.3f})".format(RMSEs.mean(), RMSEs.std()*2))
print("")

Is the problem with my dataset and count of features? How can I build a prediction model with this type of dataset?

that's possibly due to poor parameter tuning.
Try reducing C for SVR and increasing n_estimators for RFR.

A nice approach is to gridsearch through the parameter, and plot the metric result.

Another thing that might help is to normalize the parameters (sklearn.preprocessing.StandardScaler) and to remove the skew from the target (usually log transform or 1/target transform works better)


In U-Net, is there a non-linearity (relu) in up-convolution layer? I am doing semantic segmentatio using U-Net. I was wondering whether to include 'relu' activation or not in the up-convolution layer?
x = Conv2DTranspose(filters, kernel_size) (x)

OR

x = Conv2DTranspose(filters, kernel_size, activation='relu') (x)

```

I'd not use activation function in these layers because that's how I saw it done first time, but it's probably worth trying since I can't come to a reason why we woudn't use activation function in this layer, it probably shoudn't change the results much.
U-Net anyway has an architecture with convolutions between each upsampling, which use activation functions, so not using activation function in upsampling layers is not a big deal.


Why cant RNN learn long term dependencies=? In Colah's blog, he explain this.

  In theory, RNNs are absolutely capable of handling such “long-term
  dependencies.” A human could carefully pick parameters for them to
  solve toy problems of this form. Sadly, in practice, RNNs don’t seem
  to be able to learn them. The problem was explored in depth by
  Hochreiter (1991) [German] and Bengio, et al. (1994), who found some
  pretty fundamental reasons why it might be difficult.

In quick explanation what is the fundamental reasons why it might be difficult?

There is no explicit notion of memory (like gates in lstm and gru)

Gates are a way to optionally let information through, ommiting this functionality we will just be updating weights that will in process fade away hence longer memory is hard to learn.


How do you know when you are using a multi gpu? I have 2 gpus on my local machines, but i'm not sure that the model I am training is using both of them (the speed has not changed much).

My code:

def get_model():
    base_model = ResNet50(weights='imagenet', input_shape=(image_size,image_size,3), include_top=False)
    #base_model.trainable = False

    model = models.Sequential()
    model.add(base_model)
    model.add(layers.GlobalAveragePooling2D())
    model.add(layers.Dense(1024, activation='relu'))
    model.add(layers.Dropout(0.5))
    model.add(layers.Dense(196, activation='softmax'))
    model.summary()

    model = multi_gpu_model(model,gpus=2)

    #optimizer = optimizers.SGD(lr=1e-4, decay=1e-6, momentum=0.9, nesterov=True)
    optimizer = optimizers.RMSprop(lr=0.0001)
    model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['acc'])

    return model

I just added the multi_gpu_model setting, but am not sure that this is enough. I checked nvidia-smi every 0.5 seconds, but seems like only one gpu is working. How do I make sure that it uses the full 2 gpus?

If you are using Linux and Nvidia GPUs, you can do the following in a terminal

nvidia-smi

Which will show you some stats about the GPUs available on your system.

You can run it automatically every 2 seconds like this to see how power/memory usage changes during training:

watch -n 2 nvidia-smi

If you see nothing or the nvidia-smi command fails, you likely don't have the correct drivers installed.

If theya re showing, but Keras/Tensorflow is not finding them, have a look at this thread for more checks for Tensorflow backend.


How does Neural Network denoise an image? I understand the mathematical formalism behind how neural networks work as a classifier or perform regression analysis. But I face difficulty to realize how they are such a great denoising instrument. Even a simple fully connected single hidden layer network can denoise appreciable amount of noise from a relatively simple enough signal (with proper training of course). Do neural networks mimic one or more mathematical operations of filter (such as convolution) and that's why they are so effective in denoising application?

in general CNN can generate images, so we can feed them noised images and images without noise at output and it can learn how to figure out pixel color based on his neighbours, actually you can't reconstruct perfectly because noise cause removal of some information from image and all you can do it to find "as best as possible" values to restore original image

for more info this clip is very informative
https://www.youtube.com/watch?v=z4vl3Z6NFW8


What are some method for pre-processing data in OCR? I have a dataset for a supervised learning task.

Each row is a vector with a value of pixmap value in a range [0,255] of gray colormap, each vector is labeled with a character. I have to assign each vector with a character.

My Question:

What are some methods that I can try to pre-process the data to gain better accuracy?

You can do image tresholding, normalize the data first by dividing each elements of the vector by 255, then do thresholding, for each $x_i \in \mathbb{x}$ $x_i=1$ if $threshold\le x_i$ where $\mathbb{x}$ is your data vector, and $x_i$ is the element of vector, you can choose the threshold based on experiment or personaly I choose 0.5. By doing image thresholding we reduce the noise of the image, hence its increase the accuracy


What Base Should Be Used For Negative Log Likelihood? When calculating the negative log likelihood loss, what base of log are we supposed to use?

The change in base is equivalent to multiplying the function by a constant. It does not affect the computation.

$
log_b(x) = \dfrac{1}{log_e(b)}.log_e(x)
$


Detect if word is «common English» word or slang word I have a huge list of short phrases, for example:

sql server data analysis # SQL is not a common word
bodybuilding # common word
export opml # opml is not a common word
best ocr mac # ocr and mac are not common words

I want to detect if word is not a common word and should not be processes further.

I've tried to do this with NLTK, but it gives strange results:

result = word in nltk.corpus.words.words()

sql = false
iso = true
mac = true

Is there a better way to do this?

It all depends on your definition of what a common word is in your domain. You are using an NLTK corpus which likely doesn't fit your domain very well. Either you have a corpus containing the domain you want and you do a simple lookup. Or you don't know in advance and you need to compute these common words from your documents (your short phrases). In that case, the more sentences you have the better.

An easy way to do this is using pure Python is to use a counter

from collections import Counter

documents = [] # here add your list of documents/phrases

counter = Counter()

for doc in documents:

    words = doc.split() # assuming that words can be split on whitespaces

    counter.update(words)

counter.most_common() # this will return words ranked by their frequency

Then it's up to you to apply a threshold to define what words are common and which aren't. A more advanced approach could be using their TFIDF weights but in that case, you are not necessarily keeping common words, you are keeping "important" ones.


Location Invariance not achieved in CNN in spite of 99% test accuracy I have been trying to decode a captcha using CNN. The number of training samples is 1,82,000. So far, I could achieve 99% accuracy on training as well as test set.
The issue however is that LOCATION INVARIANCE is not achieved while training. The captcha text if slightly shifted horizontally gives bad prediction.Below image is one of the samples on which network was trained and gives good prediction accuracy for similar images (Accuracy~99%) .

But if following image is given , it gives some random prediction.

The only difference between above two images is location of captcha text (apart from the actual captcha value.)
As far as I know, the usp of CNN is the fact that location of a pattern doesnt matter. So what could be the possible reasons for this ?

As far as I know, the usp of CNN is the fact that location of a pattern doesnt matter.

This is not true. Some options used in CNN architecture, such as max pooling, or strided convolutions, can add a moderate amount of translation invariance. However this will not cover larger translations - anything that is a significant percentage of the image width/height.

CNNs also support translation equivariance, which is where image textures and motifs that appear repeatedly but in different locations (lines, corners, curves) are learned efficiently and appear in the feature maps. This is probably closer to the USP of CNNs, that they learn the "representation language" of a signal, such as photograph, where it is consistent across the dimensions that are being convolved.

The cause of your problem is therefore very likely that your training set does not include enough images with the more central translation, or enough variations with the separate images.

To cover large translations you could look at one or more of:

Image pre-processing. In your supplied examples it looks very feasible to centre the digits.
Data augmentation. Generate translated versions of your training data to cover expected range of input translations in test.
Network architecture. A RNN or RNN/CNN hybrid could probably consume the images using smaller overlapped tiles of the images in sequence, and be trained to output the captcha string at the end. You might also be able to do this without a RNN, but that would require labelling each tile for the separate digits.


Missing data imputation with KNN -1
down vote
favorite
I have a dataset including missing data for most of the variables. Assume the dataset is as follows:

Obs.  var1  var2  var3  var4  var5  var6
1     x11   x12   x13   x14    Nan  Nan
2     x21   x22   x23   Nan    x25  x26
3     x31   x32   x33   x34    x35  x36
...
n     xn1   xn2   xn3   xn4    Nan  xn6

I have split the dataset to d1 where we have complete data for all variables and d2 where all records have at least one missing variable.

I made different models using KNN: To predict the values of var5 and var6 for the first observation, I used d1 (dataset without missing value) and modeled on var1, var2, var3 and var4.

To predict the value of var5 for the last observation, I used d1 and modeled on var1, var2, var3, var4, and var5.

Does my approach make sense?! Any suggestion are welcome. Thank you.

There are various approaches for dealing with missing values. Suppose we've got 4 instances in a dataset:

x1 = [1 2 3]
x2 = [1 ? 3]
x3 = [2 4 2]
x4 = [1 3 3.5]

one simple approach (specially popular in medical datasets) is finding values with regard to most similar instance; in the above case, missing value of x2 would be then: 2 (since x1 is the most similar)
a more sophisticated approach is weighted averaging through k most similar instances (of course only applicable if the missing value is numeric or at least ordinal) in this case, you should calculate: x2(2) = (2*(1-(0/3)) + 3*(1-(0.5/3)) / (2-(0/3)-(0.5/3)) (x4 has also been counted as a similar case
another approach, is voting among k most similar instances (applicable both for categorical and ordinal values).

cases 1 and 3 are what you have implemented (case1 = 1 Nearest Neighbor / case2 = k-nearest neighbors)

there are other approaches for handling a missing value, but it depends on what you're going to do with your dataset. For example in very large datasets, sometimes it is efficient to simply ignore every instance that contains at least one missing value, or ignore only the missing value (not the entire instance) in further processes (e.g. in VFI algorithm)


How to give names/labels to topics in LDA I want to give labels to different topics created using LDA. I don't want to do it manually. I saw some papers on automatic labeling but I am still confused.

How can I use the information produced by LDA to automatically generate names to the topics generated?

I suppose it is late to answer, but it may turn out to be helpful for others.

LDA presents all topics as a mixture of keywords with their weights
For example, in gensim implementation it looks like this:

topic #0: 0.009*river + 0.008*lake + 0.006*island + 0.005*mountain + 0.004*area + 0.004*park + 0.004*antarctic + 0.004*south + 0.004*mountains + 0.004*dam

So you may name a topic using the keywords with the largest weights (like River+Lake).
Or you may use something like WordNet to find the most common hypernym for them.
For example, in python you can do it like this.

from nltk.corpus import wordnet as wn
wn.synset('river.n.01').lowest_common_hypernyms(wn.synset('lake.n.01'))


Can Shapley/Lime values be used for unsupervised learning? One thing that is really useful when trying to understand what a machine learning model does, is seeing why some instances got predicted. For that Shapley Values and Lime are really usefull. But can they be used with unsupervised learning?
Let's say we are doing anomaly detection with tabular data and we run some algorithm like Isolation Forest (or any other).
Is it conceptually right to use Shapley or Lime to try to give a local explanation of the results when using unsupervised learning?

I don't think so, not directly. SHAP is trying to explain each feature's effect on the prediction, but you have no label here. It might be better to ask therefore, what are you trying to explain?
In the case of an isolation forest, you can find the short path through the trees to any anomaly. That path tells you why the trees separated it, based on what features. It may not be super interpretable, but can be read off directly.


Parsing and storing a large amount of HTML data I have a data chunk (~30k) in which I have htmls pages and pngs saved in a folder for websites. These folders are titled based on some randomly generated hashes. My supervisor wants me to crunch through this data chunk and extract some attributes out of each HTML page and store it in a DB for future use. Attributes to be extracted comprises of page titles and copyright section from the HTML.
As per my understanding this data is unstructured because there is no relation per say in the folder data for now. Moreover, there is a somewhat inherent structure which is of HTML but essentially each page mutually disjoint with the rest which qualifies for unstructured. Please correct me if I am wrong here.

Manager wishes to have the data stored in an ELK stack. By storing, he is quite unclear at this point in time but so far he wants to have the whole HTML file, title and copyright for each single HTML file extracted and stored. Here comes my first concern which I need help with.

Is it a good idea to store whole HTML file into DB? I am of the
opinion that we place HTML files in a centralized storage on some
kind of FS and store the absolute paths of those files against each
entry in the DB (we are already doing the same thing for PNGs btw).

I haven't worked with ELK stack and I thought it would be a good learning opportunity. While going through online tutorials I have learned that it is essentially for logs parsing from different applications servers and storing and visualizing them in a presentable and searchable manner.

If anyone can comment about ELK, if it will work in my case, that
would be very helpful.

So far the end objective is to crunch through this data and store the attributes and when required search through the attributes and use them as per future need. For example, if there is a specific copy right text that is coming up very frequently, then get that copyright text and use it for classifying certain pattern which takes to my third and last question.

Will it help to store it in a non relational database and then query
accordingly? In my opinion RDBMS like mysql is a better contender
because it will be easy to search through the tables for a specific
type of title and then use it accordingly. End goal is not
visualization, but to have data at hand to use whenever required.

The terms "structured data" or "unstructured data" are not defined in such a way that a given dataset is always either one or the other. There are gray areas and I think this is one example. Since you cannot rely on the structure in your data, I would categorize this as unstructured.

To understand if it's a good idea to store the whole HTML in the DB (and same question for the PNGs), you need to weigh the pro's and con's. Pro storing everything in the DB is the simplicity: You don't have separate places where data is stored, so if you take a snapshot from the DB at some point in time and restore it, you restore the entire state as it was at that time. You do not need to worry about your disk storage separately, to restore that to a given state. Against storing everything in the DB is the amount of data. Can the DB handle it, or does performance suffer too much? Think about retrieving the data, searching/querying it, storing data, making back-ups. This will depend on your choice of database.

The same question for ELK and MySQL: What are the pros and cons of each? MySQL is simpler to install, that's always good. MySQL gives you a relational datamodel (tables can be related using foreign keys). Is that an advantage? MySQL gives you transactions. Is that helpful? ELK mainly gives you scalability, meaning that probably it would allow everything to be stored in the DB and still meet your performance needs.

If you can't store everything in MySQL (HTML and PNGs), then before choosing to store part of the data somewhere else, my first option would be to change DB technology to something that can store everything, rather than to start storing things in different places. So in that case ELK might be a good option, but store the PNGs there, too.


Fasttext exception error I'm trying to run language detection using Facebook's fastText through a Python script but I get this error when I load the model :

  Exception: fastText: Cannot load lid.176.bin due to C++ extension failed to allocate the memory

Any ideas? I've already installed all prerequisites and still get this error.

You are probably trying to load word vectors that are shared in the fasttext.cc website or are trying to load using models that are trained with the latest one on master. There is an issue in github with a lot of comments. One solution is to try to load from the text files the code for which is given here. I was able to work around this using java but had to train it in java only. Take a look at the examples in this package.

https://github.com/vinhkhuc/JFastText/tree/master/examples/api


Can mini-batch gradient descent outperform batch gradient descent? As I was reading and going through the second course of Andrew Ng's deep learning course, I came across a sentence that said,

With a well-turned mini-batch size, usually it outperforms either
gradient descent or stochastic gradient descent (particularly when the
training set is large).

But how is it possible? can mini-batch gradient descend really give us a better set of weights and biases even though it's not updating them based on the whole dataset? I can only think that it's maybe not overfitting and in that way, it can give better results.

First of all though SGD gives frequent updates which help improve performance of the model but it can result in noisy result and frequent updates is computationally expensive for large dataset.
Batch gradient has less update frequency which result is stable error gradient but as it stores whole training data the learning process is slow(when data is large).

Mini-batch gradient descent seeks to find a balance between the
robustness of stochastic gradient descent and the efficiency of batch
gradient descent.

Mini batch GD allows to not have all training data in memory and can be performed in distributed manner.
For more info you can check chapter 8 of Deep Learning book by Ian Goodfellow.


Using Sklearn's predefined split I am working on a binary classification task using SVM. The dataset is quite large so I don't want to use k-fold CV for parameter tuning, but instead a simple train-validation-test split. I have done the following:
X_train, X_test, y_train, y_test = train_test_split(
    X, y, stratify = y, test_size=0.2, random_state=1)

X_train, X_val, y_train, y_val = train_test_split(
    X_train, y_train, stratify = y_train, test_size=0.25, random_state=1)

So I have a 60-20-20 training-validation-test split. Since my validation set is predefined I want to use Sklearn's predefinedsplit. So I need to get the indices of my training and validation samples and set the validation indices to 0 and training indices to -1 so I tried the bottom answer of this question:
split_index = [0 if x in X_val.index else -1 for x in X_train.index]
But this returns a list of only -1's. I am unsure where I am going wrong. All my X and y _train, _val, _test are dataframes so .index can be applied. I have printed out X_train.index and X_val.index and both return Int64Index arrays of different lengths.
I also tried using Hypopt as one of the answers in the above link mentioned but it seems to be broken at the moment. Any suggestions as to how I should proceed?

The result that all values are equal to -1 is to be expected, since you're checking if the indices from your training set occur in your validation set, which by definition is not the case since your train and validation sets should not have the same observations. If you want to add the 0/1 indicator to you original dataset you have to use X instead of X_train:
split_index = [0 if x in X_val.index else -1 for x in X.index]

This will set the index to 0 if it occurs in X_val and -1 for all other cases.


How does Keras calculate accuracy? How does Keras calculate accuracy from the classwise probabilities? Say, for example we have 100 samples in the test set which can belong to one of two classes. We also have a list of the classwise probabilites. What threshold does Keras use to assign a sample to either of the two classes?

For binary classification, the code for accuracy metric is:
K.mean(K.equal(y_true, K.round(y_pred)))

which suggests that 0.5 is the threshold to distinguish between classes. y_true should of course be 1-hots in this case.
It's a bit different for categorical classification:
K.mean(K.equal(K.argmax(y_true, axis=-1), K.argmax(y_pred, axis=-1)))

which means "how often predictions have maximum in the same spot as true values"
There is also an option for top-k categorical accuracy, which is similar to one above, but calculates how often target class is within the top-k predictions.


Logit function in Logistic Regression I have studied the Perceptron algorithm and now I'm trying to understand logistic regression. I know what odd ratio and Logit functions are but I don't understand why we create the following equation:

$logit(\frac{p}{1-p}) = \bar{\omega}^{T}\bar{x}$

where w is the weight vector. What does it represent, and why?

Can anyone show me, how does that work?

W is just the coefficients that we estimate to determine the contribution of each variable to the model. We attempt to estimate the log odds ratio by multiplying coefficients by the input variables that we have.

If we didn't have W, by default it would just be the 1 matrix multiplied by X, or each variable output would be assumed to affect the response equally (ignoring the scale differences between predictors).