Can the vanishing gradient problem be solved by multiplying the input of tanh with a coefficient? To my understanding, the vanishing gradient problem occurs when training neural networks when the gradient of each activation function is less than 1 such that when corrections are back-propagated through many layers, the product of these gradients becomes very small.

I know there are other solutions like a rectifier activation function, but my question is why we could not simply use a variation of the often used tanh function.

If the activation function was of the form $\tanh(n x)$ then the maximum possible gradient is $n$. Thus if $n > 1$ we no longer have a case where the product of gradients necessarily goes to 0.

Is there some reason why such an activation function would otherwise fail?

You are correct. For $n > 1$, the multiplication of derivatives does not necessarily go to zero,  because each derivative could be potentially larger than one (up to $n$).

However, for practical purposes, we should ask ourselves how easy it is to maintain this situation (keeping the multiplication of derivatives away from zero)? Which turns out to be quite hard compared to ReLU, which gives derivative = 1, specially now, when there is also a chance of gradient explosion.

Introduction

Suppose we have $K$ derivatives (standing for depth $K$) multiplied together as follows
$$g=\left.\frac{\partial f(x)}{\partial x} \right|_{x=x_1}\cdots \left.\frac{\partial f(x)}{\partial x} \right|_{x=x_K}$$
each evaluated at different values $x_1$ to $x_K$. In a neural network, each $x_i$ is a weighted sum of outputs $\boldsymbol{h}$ from previous layer, e.g. $x = \boldsymbol{w}^t\boldsymbol{h}$.

As $K$ increases, we want to know what it takes to prevent the vanishing of $g$. For example, for the case of$$f(x)=tanh(x)$$we cannot prevent that because each derivative is smaller than one, except for $x=0$, i.e.
$$\frac{\partial f(x)}{\partial x}=\frac{\partial tanh(x)}{\partial x}=1-tanh^2(x) < 1 \text{  for } x \neq 0$$
However, there is a new hope based on your proposal. For $f(x)=tanh(nx)$, derivative could go up to $n > 1$, i.e.
$$\frac{\partial f(x)}{\partial x}=\frac{\partial tanh(nx)}{\partial x}=n\left(1-tanh^2(nx)\right) < n \text{ for } x \neq 0.$$

When are forces balanced?

Now, here is the core of my analysis:

  How far $x$ needs to move away from $0$ to have a derivative smaller than
  $\frac{1}{n}$ to cancel out $n$ which is the maximum possible derivative?

The farther $x$ needs to move away from $0$, the harder it is to produce a derivative below $\frac{1}{n}$, thus, the easier it is to prevent the multiplication from vanishing. This question tries to analyze the tension between good $x$'s close to zero, and bad $x$'s far from zero. For example, when good and bad $x$'s are balanced, they would create a situation like
$$g=n \times n \times \frac{1}{n} \times n \times \frac{1}{n} \times \frac{1}{n} = 1.$$
For now, I try to be optimistic by not considering arbitrarily large $x_i$'s, since even one of them can bring $g$ arbitrarily close to zero.

For the special case of $n=1$, any $|x| > 0$ results in a derivative $< 1/1 = 1$, therefore, it is almost impossible to keep the balance (prevent $g$ from vanishing) as depth $K$ increases, e.g.
$$g=0.99 \times 0.9 \times 0.1 \times 0.995 \cdots \rightarrow 0.$$

For the general case of $n > 1$, we proceed as follows
$$\begin{align*}
&\frac{\partial tanh(nx)}{\partial x} < \frac{1}{n}\\
&\Rightarrow n\left(1-tanh^2(nx)\right) < \frac{1}{n}\\
&\Rightarrow 1 - \frac{1}{n^2} < tanh^2(nx) \\
&\Rightarrow \sqrt{1 - \frac{1}{n^2}} < |tanh(nx)| \\
&\Rightarrow x > t_1(n):=\frac{1}{n}tanh^{-1}\left(\sqrt{1 - \frac{1}{n^2}}\right) \\
&\text{or } x < t_2(n):=-t_1(n)=\frac{1}{n}tanh^{-1}\left(-\sqrt{1 - \frac{1}{n^2}}\right)
\end{align*}$$
So for $|x| > t_1(n)$, the derivative will be smaller than $\frac{1}{n}$. Therefore, in terms of being smaller than one, multiplication of two derivatives at $x_1 \in {\Bbb R}$ and $|x_2| > t_1(n)$ for $n > 1$ is equivalent to an arbitrary derivative for $n=1$, i.e.
$$\left(\left.\frac{\partial tanh(nx)}{\partial x}\right|_{x=x_1\in {\Bbb R}} \times \left.\frac{\partial tanh(nx)}{\partial x}\right|_{x = x_2,|x_2| > t_1(n)}\right) \equiv \left.\frac{\partial tanh(x)}{\partial x}\right|_{x = z}, z \in {\Bbb R}\setminus\{0\}.$$
In other words,

  $K$ mentioned pairs of derivatives for $n > 1$ is as problematic as $K$
  derivatives for $n=1$.

Now, to see how easy (or hard) it is to have $|x| > t_1(n)$, let's plot $t_1(n)$ and $t_2(n)$ (thresholds are plotted for a continuous $n$).

As you can see, to have a derivative $\geq 1/n$, the largest interval is achieved at $n=2$, which is still narrow! This interval is $[-0.658, 0.658]$, meaning for $|x| > 0.658$, the derivative will be smaller than $1/2$. Note: a slightly larger interval is achievable if $n$ is allowed to be continuous.

Based on this analysis, we can now reach a conclusion:

  To prevent $g$ from vanishing, around half or more of $x_i$'s need to be
  inside an interval like $[-0.658, 0.658]$

so when their derivatives are paired with the other half, multiplication of each pair would be above one at best (required that no $x$ is far into large values), i.e.
$$\left(\left.\frac{\partial f(x)}{\partial x} \right|_{x=x_1 \in {\Bbb R}}\left. \times \frac{\partial f(x)}{\partial x} \right|_{x=x_2\in [-0.658, 0.658]}\right) > 1$$
However, in practice, it is likely to have more than half of $x$'s outside of $[-0.658, 0.658]$ or a couple of $x$'s with large values, causing $g$ to vanish to zero. Also, there is a problem with too many $x$'s close to zero which is

  For $n > 1$, too many $x$'s close to zero could lead to a large gradient $g \gg 1$ (potentially up to $n^K$) which moves (explodes) the
  weights into larger values ($w_{t+1} = w_{t} + \lambda g$), which
  further moves the $x$'s into larger values ($x_{t+1} = \boldsymbol{w}^t_{t+1}\boldsymbol{h}_{t+1}$) converting the good $x$'s into (very) bad ones.

How large is too large?

Here, I carry out a similar analysis to see

  How far $x$ needs to move away from $0$ to have a derivative smaller than
  $\frac{1}{n^{K-1}}$ to cancel out the other $K-1$  $x$'s assuming they are very close to zero and acquired the maximum possible gradient?

To answer this question, we derive the below inequality

$$\begin{align*}
&\frac{\partial tanh(nx)}{\partial x} < \frac{1}{n^{K-1}} \Rightarrow |x| > \frac{1}{n}tanh^{-1}\left(\sqrt{1 - \frac{1}{n^K}}\right)\\
\end{align*}$$

that shows, for example, for depth $K = 50$ and $n=2$, a value outside $[-9.0, 9.0]$ produces a derivative $< 1/2^{49}$. This result gives an intuition about how easy it is for a couple of $x$'s around 5-10 to cancel out the majority of good $x$'s.

One way street analogy

Based on the previous analyses, I could provide a qualitative analogy using a Markov Chain of two states $[g \gg 0]$ and $[g \sim 0]$ that crudely models the dynamical behavior of gradient $g$ as follows

When system goes to state $[g \sim 0]$, there is not much gradient to bring (change) the values back to state $[g \gg 0]$. This is similar to  a one way street that will be passed eventually if we give it enough time (large enough epochs) given convergence of training does not happen (otherwise, we have found a solution before experiencing a vanishing gradient).

A more advanced analysis of dynamical behavior of gradient would be possible by carrying out a simulation on actual neural networks (which potentially depends on many parameters such as loss function, width and depth of network, and data distribution) and come up with

A probabilistic model that tells how often vanishing happens based on a distribution of gradient $g$ or a joint distribution ($\boldsymbol{x}$, $g$) or ($\boldsymbol{w}$, $g$), or
A deterministic model (map) that tells which initial points (initial values of weights) lead to gradient vanishing; possibly accompanied by trajectories from initial to final values.

Exploding gradient problem

We have covered the "vanishing gradient" aspect of $tanh(nx)$. On the contrary, for the "exploding gradient" aspect, we should be worried about having too many $x$'s close to zero, which could potentially produce a gradient around $n^K$, causing numerical instability. For this case, a similar analysis based on inequality
$$\begin{align*}
&\frac{\partial tanh(nx)}{\partial x} > 1 \Rightarrow |x| < \frac{1}{n}tanh^{-1}\left(\sqrt{1 - \frac{1}{n}}\right)\\
\end{align*}$$
shows that for $n=2$, around half or more of $x_i$'s should be outside of $[-0.441, 0.441]$ to have $g$ around $O(1)$ away from $O(n^K)$. This leaves an even smaller region on ${\Bbb R}^K$ in which $K$ $tanh(nx)$ functions would work well together (neither vanished, nor exploded); reminding that $tanh(x)$ does not have the exploding gradient problem.


Feature selection in R too large dataset I'm doing credit risk modelling and the data have large number of features.I am using boruta package for feature selection. The package is too computationally expensive, I cannot run it on the complete training dataset. What i'm trying to do is take a subset of the training data(let's say about 20-30%) and run the boruta package on that subsetted data and get the important features. But when i use random forest to train the data I have too use the full dataset. My question is, Is it right to select features only on a part of train data but then build the model on whole of training data?

Technically, it will be alright to use a sample of the data. One important assumption is that most of the features don't have outlier. In such case your sampling might miss out on those.

But, I don't see a necessity of sampling with the size of data that you have given. 150mb is not large data at all, given that you have 8 GB ram at disposal. Rather than sampling the data, first check if there is alternate way of selecting the features. Since there are only 18 columns, look at the summaries for each columns, how do the features correlate with your outcome variable, which variables add very little information. If you have not done more exploratory analysis of the 18 variables, I'll suggest you do that first.

Secondly, why not build your random forest directly with 18 features? If you had, say, 180 features, it would have made sense to do automated feature selection first using Boruta etc. If there are features that do not add information, their variable importance in forest model will be low. You can drop them in subsequent analysis


Will my validation loss eventually go down? I'm currently training a binary classifier that takes in 2 inputs, and outputs which object it thinks is "better."
I have an absolutely massive dataset, about 2 trillion records, and I'm feeding these records into my network about 300k records at a time. Overfitting isn't really a concern as I'm only running one epoch, so the network is only really seeing new data every cycle.
So far, the training loss is decreasing steadily, and my training accuracy is rising.
My validation loss is decreasing slightly, but fluctuating a lot. My validation accuracy is rising at about the same rate as my training accuracy is.
Will I eventually see a large drop in validation loss as the network gets more "confident?" I know that's a very hard question to answer without knowing more details, but have you seen this type of behavior before in your models? I'm just a little spooked since these experiments take up so much time to run.

When training neural network, you need to train many epochs in order for the model to learn enough to generalize. What is happening now the model is changing weights to fit the data that it observes (lower training loss) but does not generalize (as evidenced by high variance on validation data).
There are a couple of options:

Downsampling your data. Take only the most meaningful data to train.
Scale up training. Either use graphics processing units (GPUs) or distributed training.
Run training for longer.


Finding Episodes in event sequence In the paper "Discovery of frequent episodes in event sequences" by Mannila et al., a method for finding frequent episodes in an event sequence if a class of episodes and a sequence of events is given.

It is not clear to me how to find episodes in the first place. I would like to know if there are any tools, algorithms, or published work that can be used to this end, or which describes a method to find episodes in a given stream.

The paper that you cited references two other papers that explain the method you are looking for here and here.  The second one is behind a pay wall.

But... these look like deterministic methods to me, and I suggest you use a machine learning method instead.  You basically need to separate events from each other using some sort of clustering algorithm.  I suggest you use DBSCAN and adjust the parameters based on whether you want every event to be assigned to an episode or not.

Another algorithm that might work well is simple k-means clustering with the addition of an elbow method or silhouette score to determine the optimal number of clusters.


How does Doc2Vec treat numerical data which is a part of text data? I have data containing both numbers and raw text related differently like:

The power of diesel generator is 15kva. I need a single phase generator.
Three phase generator required of 140 kva.
Need 70g/m^2 GSM a4 size copier paper.

I removed the numerical data and trained a classifier on vector obtained from Doc2Vec and got around 80% accuracy.

Is there a way to take in account the numbers (here 15,140,70) involved in text while creating a doc2vec vector? or if there is any way to take in account these numbers in order to improve on the accuracy?

If you are passing in the numbers as a string type Doc2Vec should just accept these as part of the vocabulary without an issue. Have you tried running the Doc2Vec training with the numbers still included?

One way to solve the issue if you can't just pass the values in as is is to give them a unique identifier (e.g. < NUM >). It shouldn't matter what the value is as Doc2Vec should in theory not be able to draw any information (and really shouldn't) from that. Instead if you teach it when a number occurs in a document (by using the identifier) that might improve accuracy.


How to find splits in data so that each split has equal weighting according to function f I have a weight function f that outputs a numeric weighting for a sample s. I also have an ordered set of samples S where the weight of each sample s in set S varies greatly.

How can I create n splitting points so that each split is weighted approximately the same? What kind of methodology, algorithms or models could I use to achieve this?

A classic optimization problem! You can use Linear Programming/Optimization to find a good split. Every of the n samples s $\in S$ has weight f(s) and we want to divide them into m folds. You can use a trick to linearize the L1 objective or you can use Quadratic Programming for a L2 objective function. The Quadratic Programming model is easier to define in this case. Let's define $x_{ij}$ as a binary decision to put sample i in fold j and $\mu$ is the ideal, mean weight per fold. Then this is our objective function:

min $\sum_{j=1}^m(\sum_{i=1}^nx_{ij}f(i)-\mu)^2$

Under the following constraints:

$\sum_{j=1}^mx_{ij}=1$ for all $i\in \{1..n\}$ to ensure exactly one assignment per sample

$x_{ij} \in \{0, 1\}$ to turn it into binary decision varibales

Depending on the size of your dataset and the solver you use this can be a heavy optimization, but there are a lot of greedy heuristics that will get you close fairly fast.


Clustering on imbalanced data that has high correlation I am clustering images of two categories, but for the purposes of the experiment, I do not know the labels i.e. this is an unsupervised problem. Via correlation heatmaps and other experiments, I am confident that my images are highly correlated, at least via a Pearson correlation coefficient. However, I face very large imbalanced datasets in my problem, and as a result, my hierarchical agglomerative clustering fails when the imbalance passes a ratio of around $10:1$.

I have no idea of alternative solutions. I have tried reducing dimensionality via PCA, but this does not help.

When you have high correlation problems you should go for dimensionality reduction. Multicollinear features can be "summarized" and controlled for by techniques such as PCA or Autoencoders for dimensionality reduction. I don't recomment PCA, since it can only extract factors that linearly associated with your data. In my opinion:

Autoencoders  >  PCA

There are many other dimensionality reduction techniques, but these are the most common.

Once you have reduced the dimensionality of your data you can run your cluster analysis on the reduced dataset, and observe how different observations (and their categories) will be distributed with respect to each other.

Hope this helps, otherwise let me know.


How do we decide between XGBoost, RandomForest and Decision tree? What do we take into consideration while deciding which technique should be used when dealing with a particular dataset? I understand that there isn't any hard and fast rule to this. Do we use XGBoost only when there are a lot of features in the dataset and RandomForest for otherwise cases? Or are we suppose to hit and trial and find whichever gets us better results everytime?

Decision Tree is very useful if you want to be able to explain where your result comes from you can often print the tree and see how your model came to this answer.
Random Forest can also provide such information, but you'll have to browse all trees and make some "stats" into them, which is not as easy. But Random Forest often give better results than Decision Tree (except on easy and small datasets).
Finally, XGBoost could give a better result than Random Forest, if well-tuned, but you can't explain it easily. If you don't mind about results-explanation, I'd suggest you to try both XGBoost and RandomForest, with a bit on tuning, to see which one is best fitting on your dataset.


Different number of features after using OneHotEncoder I have train and test data in two separate files.

OneHotEncoder gives different number of features for Train and Test Data based on the different values they have. But the classifier requires that the number of features for test and train data should be equal, how can I solve this problem?

One hot encoding is a way of converting output label for 3 categories like 2 into [0, 1, 0] or 3 into [0, 0, 1].

If you are using scikit learn to convert the value into one hot encoder then in training time you should use

enc = OneHotEncoder()
enc.fit(x_train)

If you are using scikit learn to convert the value into one hot encoder then in testing time you should use
enc.transform(x_test)

The reason we are using transform function in case of testing is we have to consider the label values on the basis of which we have converted data in training time. Because in testing time we not get all labels for that column


Lasso Regression for Feature Importance saying almost every feature is unimportant? I have a metric (RevenueSoFar) that is a great predictor of my target FinalRevenue as you'd expect - it is a metric where we tend to get 90-95% of revenue so far on day 1 and then it can increase over the next 6 days. Therefore i'm also using DaysData (1,...,6) as a feature in the training set to attempt to get the model to learn about how confidence and revenue grows as we get closer to the final day.
In my pre-build analysis I have noticed correlations with several other metrics such as visits to our website, DOW of day1 etc. although not as strong as the correlation with RevenueSoFar. I've therefore ran Lasso Regression for feature importance on the features however my model has reduced every feature's coefficient to zero apart from RevenueSoFar (even the DaysData feature).

Does this mean i should remove all of these from my model and only
use the RevenueSoFar as the predictor?
In the training data set there are a large proportion of products that have
zero revenue and zero revenue so far (however the vast majority of these are
for countries with low demand which is separated out as a feature in the
training set). Could the zeros be skewing the model to think that RevenueSoFar
and FinalRevenue mostly matches perfectly and therefore there is no need for
other features?

Below is my code incase anybody is interested.
import numpy as np
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.linear_model import Lasso
from sklearn.feature_selection import SelectFromModel

#import data

DF = pd.read_csv("LearningDataSet.csv")

DF_categorical = DF[["DOW","Market",”Country”]]
DF_numerical = DF[["DaysData","D0Visits","D0MobVisits","D0DesVisits","D0Carts","D0MobCarts", "D0DesCarts", "RevenueSoFar"]]
DF_target = DF[["RevenueFinal"]]

#Scale numerical
scaler = StandardScaler()
DF_numerical_scaled = pd.DataFrame(scaler.fit_transform(DF_numerical))

#Dummy Categorical
DF_categorical_dummy = pd.get_dummies(data=DF_categorical, drop_first=True)

#Merge categorical and numerical
DF_final = DF_categorical_dummy.merge(right=DF_numerical_scaled, how='left', left_index=True, right_index=True)
#Split into training and test
X_train, X_test, y_train, y_test = train_test_split(DF_final, DF_target, test_size=0.25, random_state=0)

selection = SelectFromModel(estimator=Lasso()).fit(X_train,y_train)

print(selection.get_support())

selected_feat = X_train.columns[(selection.get_support())]
print(selected_feat)
```

Change (search over) the penalty parameter of lasso.  FinalRevenue = RevenueSoFar is a good baseline "model," but hopefully your other features can improve on that.  You might also consider just modeling the target FinalRevenue - RevenueSoFar.


Privacy through fake data? With companies and governments hungry for all the data about people, I was wondering if it was possible to gain some privacy by drowning relevant information in a sea of random data. For example a browser extension which keeps searching for random words and expressions in the background. Maybe sending generated emails to generated addresses. Or producing made up location data on my phone.

Would these measures help at all or the data miner AIs would just see right through them?

This was the idea behind paranoid linux which began as fiction and became a real project that did not reach fruition:

"Paranoid Linux is an operating system that assumes that its operator is under assault from the government (it was intended for use by Chinese and Syrian dissidents), and it does everything it can to keep your communications and documents a secret. It even throws up a bunch of "chaff" communications that are supposed to disguise the fact that you're doing anything covert. So while you're receiving a political message one character at a time, ParanoidLinux is pretending to surf the Web and fill in questionnaires and flirt in chat-rooms. Meanwhile, one in every five hundred characters you receive is your real message, a needle buried in a huge haystack.
~Cory Doctorow (Little Brother, 2008)

When those words were written, ParanoidLinux was just a fiction. It is our goal to make this a reality. The project officially started on May 14th, and has been growing ever since. We welcome your ideas, contributions, designs, or code. You can find us on freenode's irc server in the #paranoidlinux channel. Hope to see you there!"

I think the most modern equivalent is Pirate Linux along with the TOR Project.

BTW, Cory Doctorow's book "Little Brother" is kind of a fun read if you are a geek.  If you are reading this then you probably ARE a geek :-)

The reality is that it is very difficult to hide one's true intent when high quality clustering, classification and anomaly/novelty detection is used.  I work with guys who have done this type of behavior detection for some very high profile three letter agencies and it is very very easy to detect nefarious behavior if you have enough data.  So much so that undercover good guys sometimes clearly show up in the data sets.


Unsupervised binary classification on small dataset I have a small unlabeled textual dataset and I would like to classify all document in 2 categories.

I understand and have used supervised classification. I have searched a lot but still cannot understand how unsupervised binary classification works. Can you give me an example or a simple explanation ? Or some paper, document or code that explains it well ?

Thank you.

Correct me if I'm wrong, but your unsupervised classification is not much different than clustering. This is because since it is unsupervised, you do not actually have a user-provided indication of what your classes should be. Thus an unsupervised machine learning system would guess its own classes, and an straightforward way would be to make these classes correspond to the clusters.

In your case you could represent each document as bag-of-words. Then, if necessary, you can do feature reduction (e.g. using PCA). Finally, you could use k-means (in fact 2-means, since you want binary classification). Assigning a new element to any of the two clusters is your classifier.


What could be a dataset in which the presence of an outlier dramatically affects the performance of Ordinary Least Squares (OLS) regression? I am tasked with giving an example of a dataset in which the presence of an outlier dramatically affects the performance of
Ordinary Least Squares (OLS) regression. I've searched and searched the web and I understand that OLS has a hard time dealing with outliers, but I'm having a hard time figuring out why, and finding a dataset to prove this.

If you are looking for a real-world data set here is one on Harvard's Dataverse that examines state social politics research for outliers for the same purpose you are looking for. If you are looking for one for more illustrative purposes one data set worth knowing is Anscombe's quartet for demonstrating how misleading some descriptive statistics can be. For your own investigations, many data sets with and without outliers can be found on Google's beta dataset search and are worth exploring if you are curious!


Agglomerative Hierarchial Clustering in python using DTW distance I am new to both data science and python.
I have a dataset of the time-dependent samples, which I want to run agglomerative hierarchical clustering on them. I have found that Dynamic Time Warping (DTW) is a useful method to find alignments between two time series which may vary in time or speed.

I have found dtw_std in mlpy library and scipy.cluster.hierarchy in SciPy in order to cluster my data.

From the scipy docs, I find that I could use my custom distance function:

  metric : str or function, optional The distance metric to use in the
  case that y is a collection of observation vectors; ignored otherwise.
  See the pdist function for a list of valid distance metrics. A custom
  distance function can also be used.

But I am stuck matching this information to implement clustering.

My dataset is in the format of dataframe which each row corresponds to a sample.

Here is my questions:

1- How can I provide distance matrics for the linkage function?

2- How to set my custom distance function?

import pandas as pd
import scipy.cluster.hierarchy as hac
import mlpy

dataset = pd.read_csv ( "dataset.csv",encoding='utf-8' )
X # distance matrics
Z = hac.linkage(X, metrics=mlpy.dtw_std, method='average')
cluster = hac.fcluster(Z, t=100, criterion='maxclust')
leader = hac.leaders(Z, t=100, criterion='maxclust')

fig = plt.figure(figsize=(25, 10))
dn = dendrogram(Z)
plt.show()

edit:

Here is how I compute distance matrix, then I pass it to linkage:

# computing distance matrix
dm = pdist ( dataset ,lambda u,v: mlpy.dtw_std ( pd.Series(u).dropna().values.tolist(),pd.Series(v).dropna().values.tolist(),dist_only=True ))
z = hac.linkage(dm, method='average')
cluster = hac.fcluster(z, t=100, criterion='maxclust')
leader = scipy.cluster.hierarchy.fcluster(z, t=100, criterion='maxclust')

Use a precomputed distance matrix, and distance="precomputed".

HAC will compute a distance matrix anyway.


Is the LSTM share the same model parameter in each block? I learn the LSTM recently, and a little bit confuse about the model parameters about LSTM,

The follow is the LSTM structure

And it is equation as (I slightly ignore the bias in each equation):

$$i_t = \sigma(x_tU^i + h_{t-1}W^i)$$
$$f_t = \sigma(x_tU^f + h_{t-1}W^f)$$
$$o_t = \sigma(x_tU^o + h_{t-1}W^o)$$
$$\tilde{C}_t = \tanh(x_tU^g + h_{t-1}W^g)$$
$$C_t=\sigma(f_t*C_{t-1} + i_{t}*\tilde{C}_t)$$
$$h_t = \tanh(C_t)*o_t$$

I am confused about the parameters: $U^i, U^f, U^o , U^g$ and $W^i, W^f, W^o , W^g$.  Are those parameters trained by data(e.g., back-propagation)?

My second question is that, In the Figure, We have the three block of LSTM 1, 2, and 3 (mark by blue color). Do them share the same $U^i, U^f, U^o , U^g$ and $W^i, W^f, W^o , W^g$. ?  Or each block have their own $U^*$ and $W^*$?

I am confused about the parameters: $U^i, U^f, U^o , U^g$ and $W^i, W^f, W^o , W^g$.  Are those parameters trained by data(e.g., back-propagation)?

Yes. They do get trained by backpropagation. There are caveats here. Variants to backprop too exist when training a sequence-to-sequence to models. Check out BPTT for example.

  My second question is that, In the Figure, We have the three block of LSTM 1, 2, and 3 (mark by blue color). Do them share the same $U^i, U^f, U^o , U^g$ and $W^i, W^f, W^o , W^g$. ?  Or each block have their own $U^*$ and $W^*$?

Each block has the same set of weights shared among them.

An following mini example demo would be like in Keras.

from keras.models import Sequential
from keras.layers import Dense, Dropout
from keras.layers import Embedding
from keras.layers import LSTM
model = Sequential()
model.add(LSTM(128, input_shape=(None, 100)))
model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy',
              optimizer='rmsprop',
              metrics=['accuracy'])

creates an simple LSTM model. If you look at model summary to count the params you got.

model.summary

You will get

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
lstm_3 (LSTM)                (None, 128)               117248
_________________________________________________________________
dropout_2 (Dropout)          (None, 128)               0
_________________________________________________________________
dense_2 (Dense)              (None, 1)                 129
=================================================================
Total params: 117,377
Trainable params: 117,377
Non-trainable params: 0
_________________________________________________________________

And the total of 117,377 params you see is obtained from (((128*128)+(128*100)) * 4) + (4*128). Here W is of size 128*100 and U is 128*128 and there are 4 of them resp. and last 4*128 goes for the biases.


How can ANN handle varied sized inputs? I have a dataset with a message (string) and an associated mood. I am trying to use an ANN to predict one of the 6 moods using the encoded inputs.
This is how my X_train looks like:
array([list([1, 60, 2]),
   list([1, 6278, 14, 9137, 334, 9137, 8549, 1380, 7]),
   list([5, 107, 1, 2, 156]), ..., list([1, 2, 220, 41]),
   list([1, 2, 79, 137, 422, 877, 5, 230, 621, 18]),
   list([1, 11, 66, 1, 2, 9137, 175, 1, 6278, 5624, 1520])],
  dtype=object)

Since every array has a different length, it's not being accepted. What can I do about it?
PS: The encoded values were generated using keras.preprocessing.Tokenizer()

One way is to encode your input in a fixed size dimension. That is, you can use an RNN, like an LSTM, with padding and its output should be the input of an ANN.


What can we understand from max-activation generated images? There are several approaches to generate psychedelic images, providing maximum activations for individual neirons in convolutional neural networks. For example there is a lot of them there https://app.slack.com/client/T040HKJE3/threads/thread/C04655480-1581762147.250800?cdn_fallback=2 or a bit in https://arxiv.org/abs/1311.2901 (M. Zeiler) or in https://www.youtube.com/watch?v=ghEmQSxT6tw But what can we get from them?

E.g. here
does presence of 6 fish’es pieces means there are mainly 6 different positions of fishes which the net saw on 1000 train images? Or this is rather caused by some corner effects or patter size? Like 224 source pixels/100 = 2.2 fishes horizontally.

I also don't understand well how they are obtained, so can't conjecture what aspects they can depict.

There is approach when we find parts of train/validation images which gives strongest activations. This is perfectly understandable for me.

Those are the activation maps of the learned features.

In the case of this specific model, the filters learn that the "fishy" parts of the image:

Head-ish and dorsal fin-ish sections that define convex hulls
Scale-ish textures insides the convex hulls and surrounded by water-ish textures outside
Orange-ish and gray-ish colors insides the convex hulls and blue-ish colors outside
The head-ish section tend not to be in the corners of an image (since people frame pictures and people think faces are interesting)


Avoid iterations while calculating average model accuracy I am fitting a model in R.

use createFolds method to create several k folds from the data set
loop through the folds, repeating the following on each iteration:

train the model on k-1 folds
predict the outcomes for the i-th fold
calculate prediction accuracy

average the accuracy

Does R have a function that makes folds itself, repeats model tuning/predictions and gives the average accuracy back?

Yes, you can do all this using the Caret (http://caret.r-forge.r-project.org/training.html) package in R. For example,

fitControl <- trainControl(## 10-fold CV
                           method = "repeatedcv",
                           number = 10,
                           ## repeated ten times
                           repeats = 10)

gbmFit1 <- train(Class ~ ., data = training,
                 method = "gbm",
                 trControl = fitControl,
                ## This last option is actually one
                ## for gbm() that passes through
                verbose = FALSE)
gbmFit1

which will give the output

Stochastic Gradient Boosting

157 samples
 60 predictors
  2 classes: 'M', 'R'

No pre-processing
Resampling: Cross-Validated (10 fold, repeated 10 times)

Summary of sample sizes: 142, 142, 140, 142, 142, 141, ...

Resampling results across tuning parameters:

  interaction.depth  n.trees  Accuracy  Kappa  Accuracy SD  Kappa SD
  1                  50       0.8       0.5    0.1          0.2
  1                  100      0.8       0.6    0.1          0.2
  1                  200      0.8       0.6    0.09         0.2
  2                  50       0.8       0.6    0.1          0.2
  2                  100      0.8       0.6    0.09         0.2
  2                  200      0.8       0.6    0.1          0.2
  3                  50       0.8       0.6    0.09         0.2
  3                  100      0.8       0.6    0.09         0.2
  3                  200      0.8       0.6    0.08         0.2

Tuning parameter 'shrinkage' was held constant at a value of 0.1
Accuracy was used to select the optimal model using  the largest value.
The final values used for the model were n.trees = 150, interaction.depth = 3
and shrinkage = 0.1.

Caret offers many other options as well so should be able to suit your needs.


Predict items customers would buy in next order I am working on a time series classification problem to identify what items customers would buy in their next order (customers orders different products every week).

Let's say we have a customer who orders every week but different items. Overall, this customer ordered 1k items in the past, and my task would be to identify approximately 80-120 unique items (number of items customer order each week was between 80-120) from these 1k items.

Right now, I am using the xgboost classifier, and my precision is close to 70%, and recall is 65%. I understand xgboost cannot handle seasonality by default. So I created additional features like a week of the month, week of the year, season (Spring/summer/winter/fall), etc.

I wanted to know if anyone in this community worked on a similar kind of problem. I have seen lot of peoples posting same kind of problem but not sure whether they ended up being helped/ moving it to production.

Try association rules, they will help you in finding the right set of rules like what is chance of customer buying X if he purchased A, B, C in the past.

Also from my personal experience, these kinds of tasks are more business need driven. Like sales capture from the item. So even if your model is 60% accurate but captures 90% sales, you should be good to go. Try to give it a thought in this way.


Why in deep learning or another AI technique python is used rather than other languages? I want to know, why python? Instead of C#, Java, C\C++ and so on?

Short answers

Tooling. Python has fantastic math, statistics, and linear algebra libraries.
Less Code, Same Result. Python provides quick and simple ways of achieving programming solutions compared to C#, Java, C++, etc. This means you'll write less code and achieve the same result.


Reduction of feature values I have a data set of 700+ mil records with a feature that should yield good predictive power. The problem is that it has far more unique values than it should. The 10k+ unique values should map to about 150. I have that list of 150 values I want them to map to. Thinking about using a distance algorithm (levenshtein?) to map unique values from data to the desired set of values. What are some other ways to think about this problem?

Ex. 'Table', 'tab', 'tbl' should all map to 'table'. I'm not about to manually build a lookup table for this process given the volume of unique values. The unique values in the data are all derived from the desired values - they are acronyms or abbreviations.

I agree with the idea of using a similarity or distance measure (approximate string matching). I would try a bunch of them and test them on a sample: Levenshtein, Jaro, overlap coefficient or cosine (optionally with TF-IDF) over bi/tri-grams of characters.

I would also try to capture the most common abbreviations and have a lookup table for these common cases because:

Computing similarity/distance measures takes time, so it's inefficient to compute the same result many times for the same string (and it's likely that some of these abbreviations are used many times).
That gives you an opportunity to check that the mapping is correct (or to fix if it's not) at least for the most common cases, thus minimizing the overall amount of noise in the data.


t-SNE: Why equal data values are visually not close? I have 200 data points that have the same values on all features.

After t-SNE dimension reduction they doesn't look so equal anymore, just like this:

Why aren't they on the same point in the visualization and even seems to be distributed in two different clusters?

You're correct that the same values in T-SNE can be distributed across different points, the reason this happens is clear if you take a look at the algorithm that T-SNE runs across.

To address your first concern about the points actually not being the same after the algorithm has been applied to the dataset. I will leave you with an exercise to verify it for yourself, consider a simple array $x_1 =
 [0,1]$ and $x_2 = [0,1]$ and run actual algorithm against it and see for yourself that the resultant points are not actually identical.You can cross reference your answer against this.

import numpy as np
 from sklearn.manifold import TSNE
 m = TSNE(n_components=2, random_state=0)
 m.fit_transform(np.array([[0,1],[0,1]]))

You'd also observe that changing the random_state actually modifies the output co-ordinates of the model. There isn't any real correlation between the actual co-ordinates and their output. Since the first step of TSNE computes the conditional likelihood.

Let us now try to rationalize, using the algorithm the reason this happens, just using math, without any intuition, for the moment. Note, that $x_i$ and $x_j$ are both vectors in this situation. $p_{j | i} = \frac{exp(\frac{-||x_j - x_i||^2}{2\sigma^2})}{\sum_{k \neq i}{exp(\frac{-||x_j - x_i||^2}{2\sigma^2})}}$. Now, if we calculate the $p_{ij} = \frac{p_{i|j} + p_{j | i}}{2N}$, we can see that the value is 1. After applying KL divergence, we obtain the values stated above. Now, let us apply some intuition to this. $p_{ij}$ is, informally, the conditional probability that $x_i$ would pick $x_j$ as it's neighbor. This justifies the result, 1, for two reasons. The first, being that there is no other neighbor, so it must pick the only other vector in the list of co-ordinates. In addition, the points are identical and the chances that they are picked as the others neighbor should be high, as we see.

Now coming to whether the absolute co-ordinates in $\mathbb{R}^2$ have any significance. They really don't. The randomness can redistribute the points wherever you want them to go. However, what is more interesting is the ratios of the distances between the points and these are relative and are relative even when we project it onto higher dimensions which is pretty interesting.

So, the truth is, rather than looking at the two clusters, look at the distances between them, because that conveys more information than the co-ordinates themselves.

Hope this answered your question :)


Slow convergence with rpart I'm building a decision tree in R using the rpart function, available in the library of the same name, but am experiencing some serious performance issues when partitioning data using more than one feature (there is roughly 5000 rows in my data set, and the features in question have around 30 unique values each). Running rpart with the one feature completes in a couple hours (running single thread on a CentOS server with about 250GB ram), but, after 6 days, a model with a second feature added has yet to complete!

I've created an approach to extracting the path through the tree new data takes (see my question/solution here), which is my main motivation here, but the method requires an object that can be type-cast as a party object. A colleague of mine told me that he's seen performance boosts using the ctree function, but the output of this is an inscrutable S4 object that I can't figure out how to convert. My questions are two:

Does anyone know a way to speed up rpart beyond what I've already done here:clf <- rpart(Y ~ X$var1 + X$var2, rpart.control(maxcompete=1, maxsurrogate=0))?
Does anyone know how to convert a ctree object to a party object? Alternatively, does anyone have concurring or contradicting observations on ctree performance?

I have faced similar challenges. You could try to use the RevolutionR rxDTree function which performs reasonably well on large data sets and can be converted to a rpart object.

I found to achieve a reasonable performance boost by specifying the minimum bin size with rpart.control(minbucket=2). You could also tune the complexity parameter. Note that both may lead to poor performance.


Using Classification Algorithms to Interpret Clustering Reuslts Often clustering algorithms only output a bunch of class labels, and do not provide any sort of interpretation of the classes formed by the algorithm. It seems to me not entirely unreasonable to attempt to get some sort of interpretation by using the class labels provided by the clustering algorithm as the target in a supervised classification problem.

For a concrete example, say you cluster using k-means, and use a single decision tree classifier to predict the cluster based off of other features. The decision tree then should be able to give you some way of interpreting the clusters.

However, I can't find any literature (or blog posts...) mentioning this as a technique for interpreting the results of a clustering algorithm, which leads me to believe it is problematic. So in short:

Question: Can supervised classification algorithms be used to interpret the reuslts of unsupervised clustering algorithms? If not, why?

There's nothing wrong with this idea and although I don't have literature on hand, I'm fairly confident I've seen this sort of thing done. I disagree that clustering algorithms often don't provide interpretation though. There are definitely plenty that don't, but I'm not sure k-means is one of them. The centroids of your clusters should provide you with the interpretability you're looking for. Passing the results of k-means into a decision tree is probably just going to exchange the centroid for left and right bounds for your features (although it might actually be interesting if the tree ignores particular dimensions in the decision process). Generative models like GMM and LDA also give a lot of useful information.

Regarding literature, although I don't think I've seen this specifically applied to clustering, there's definitely a fair amount of on-going research into techniques for adding interpretability to "black-box models". Consider for example this article: Interpretable & Explorable Approximations of Black Box Models

  We propose Black Box Explanations through Transparent Approximations (BETA), a novel model agnostic framework for explaining the behavior of any black-box classifier by simultaneously optimizing for fidelity to the original model and interpretability of the explanation. To this end, we develop a novel objective function which allows us to learn (with optimality guarantees), a small number of compact decision sets each of which explains the behavior of the black box model in unambiguous, well-defined regions of feature space. Furthermore, our framework also is capable of accepting user input when generating these approximations, thus allowing users to interactively explore how the black-box model behaves in different subspaces that are of interest to the user. To the best of our knowledge, this is the first approach which can produce global explanations of the behavior of any given black box model through joint optimization of unambiguity, fidelity, and interpretability, while also allowing users to explore model behavior based on their preferences. Experimental evaluation with real-world datasets and user studies demonstrates that our approach can generate highly compact, easy-to-understand, yet accurate approximations of various kinds of predictive models compared to state-of-the-art baselines.

Poke around for "black box model interpretability" and you'll find a ton more recent research.


Image to image translation from image to sketch using GANs, do I need to annotated the face at all? So far I have read 2 papers on GANs and it seems like they are unsupervised networks that only uses supervision for the discriminator. If I want to translate an image from a picture to line art all I need to do is to prepare my ground truth "the unmodified picture" and the corresponding line art correct? Do I have to pre-classify any facial feature for the generator to learn the corresponding line art or is that what Conditional GANs are for? I am trying to brainstorm my own project but the data preprocessing stage is a little hazy.

Transferring an original image to a different style of image is called style transfer. Although you can use a GAN for such a case, I am not convinced this is the best approach. Furthermore, there have been many papers recently published suggesting very interesting ways for which to train models for style transfer. An amazing implementation of a file transfer algorithm which can convert video in real time can be found here.

There is another method that I have come across more recently which uses the same approach but a different loss function. The paper can be found here.

The underlining method

These methods use a combination of two networks. The first network acts as a transformation function $f_W$ which maps your image $x$ to a transformed image $\hat{y}$. The purpose is to train this network such that the output $\hat{y}$ is the original input image in the new style, in your case as line art. To train this network we will use the second network on the right side which learns how to mix in the Style Target and the Content Target. We want the transformed image $\hat{y}$ is still similar in content, but we want the transformed image to have a similar style as the target.

To train this network we will use a massive amount of input images using some online database source. This training process will be very long. However, once it is complete then we can strip the network on the right side and you will only need the network which approximates $f_W$. You will then be able to put a new image into the input of this network and the output will be the line art version of the input image.

You will need to tune the extent at which the input image is line artified by tuning the weight on the style loss during the training.


Gradient Descent Step for word2vec negative sampling For word2vec with negative sampling, the cost function for a single word is the following according to word2vec:
$$
E = - log(\sigma(v_{w_{O}}^{'}.u_{w_{I}})) - \sum_{k=1}^K log(\sigma(-v_{w_{k}}^{'}.u_{w_{I}}))
$$

$v_{w_{O}}^{'}$ = hidden->output word vector of the output word

$u_{w_{I}}$ = input->hidden word vector of the output word

$v_{w_{k}}^{'}$ = hidden->output word vector of the negative sampled word

$\sigma$ is the sigmoid function

And taking the derivative with respect to $v_{w_{O}}^{'}.u_{w_{j}}$ is:

$
\frac{\partial E}{\partial v_{w_{j}}^{'}.u_{w_{I}}} = \sigma(v_{w_{j}}^{'}.u_{w_{I}}) * (\sigma(v_{w_{j}}^{'}.u_{w_{I}}) - 1)
$ $ if w_j = w_O $

$
\frac{\partial E}{\partial v_{w_{j}}^{'}.u_{w_{I}}} = \sigma(v_{w_{j}}^{'}.u_{w_{I}}) * \sigma(-v_{w_{j}}^{'}.u_{w_{I}})
$ $ if w_j = w_k \ for \ k = 1...K$

Then we can use chain rule to get

$ \frac{\partial E}{\partial v_{w_{j}}^{'}} = \frac{\partial E}{\partial v_{w_{j}}^{'}.u_{w_{I}}} * \frac{\partial v_{w_{j}}^{'}.u_{w_{I}}}{\partial v_{w_{j}}^{'}} $

Is my reasoning and derivative correct? I am still new to ML so any help would be great!

Looks good to me. This derivative is also presented in the paper (equations 56-58).

The paper you're linking to is the most advanced attempt - at least to best of my knowledge - to explain how word2vec works, but there is also a lot of other resources on the topic (just search for word2vec on arxiv.org). If you're interested in word2vec, you may find GloVe interesting too (see: Linking GloVe with word2vec).


Generate new features from two columns I have database with three columns, y,x1 and x2:
>>>y      x1       x2
0  0.25   -19.3   -25.1
1  0.24   -18.2   -26.7
2  0.81   -45.2   -31.4
...

I want to create more features based on the x columns.
until now I have just created random functions and tries to check their correlation with the y, but my question is if there is any propeer way/ common functions in order to create thise new features.
I have used PolynomialFeatures of scikit learn but as I understood is not common to do more than 3.
from sklearn.preprocessing import PolynomialFeatures
#split x y....

poly = PolynomialFeatures(3)
poly=pd.DataFrame(poly.fit_transform(X))

My end goal is to use those new columns in random forest algorithm (I have more columns than x1 and x2 but those two that are interesting for me  and would like to investigate them and their relationshop more).

You are going to have to do something - You can try combining them in different ways, multiply them together, divide them by each other, subtract one from another. Without the context around what these features actually relate to its difficult to say what would make sense. Ultimately to make a new derived feature you are going to have to combine them or transform them in some way.


Does REPEATED K-fold cross validation make sense with Random Forest? When using random forest, would using normal cross-validation and just taking the average results from multiple models with different random states give me the same results as using Repeated K-fold cross validation?
Repeated K-fold cross-validation basically repeats cross-validation with multiple different splits of the data and reports the average results.

To the title question, yes, repeated k-fold makes sense with random forests; and to the question body, no, the results will not generally be the same as repeated model fits with one k-fold split.
The reason is that fixing one k-fold split and then repeatedly fitting random forests (with different random seeds) still only gives each forest access to $(k-1)/k$ of the data at a time.  It may be easiest to think about the case when the number of trees is astronomical: the random choices for the bagging get averaged out, to the point where different random seeds don't actually matter: the forests converge to the same result, given a training split.  Then the average of the forests' scores are the same for each of the splits, and so you average just $k$ scores.  Compare that to repeated $k$-fold, where each of the forests converges, but are all on different training sets, and so the average happens with more variety.
Whether that has a sizable impact, or in which direction, is harder to say.  Repeated $k$-fold seems like it should give more stable results, even when the number of trees is something more reasonable, because the forests are less correlated.


Is it possible to pass dataframes between R and Python using Arrow with zero-copy? Arrow is an in-memory format, so I can have a dataframe in Python backed by the arrow format. So the dataframe sits in memory, can I use that dataframe directly from R without making a copy of the dataframe? Imagine if the dataframe is 100G in size so copying is very inefficient.

I read in the documentation of Arrow that there is zero-copy streaming, but there isn't a way to make the whole dataframe available.

Not currently, though hopefully very soon. https://issues.apache.org/jira/browse/ARROW-3750 is in progress and hopefully will resolve in the coming weeks.


Image multi class classifier CNN I have a problem, im designing a multiclass classifier to classify medic images, I have to classify in which grade of desease is it, this are 6 grades , each time the joint deforms a little, so, mi original dataset was imabalanced, each class have like 16 to 200  images that are very very similar with some little deatils of change, ,so i did some contrasts and flips to the images to have like 500 images per class aprox, i used a vgg16 architechture to train the model but, it soesnt work 30% acc, i ddid a simple model with 2 layers of convolution, I have like 70% of accurracy but my recall is very very low like 0.001 to 0.3, and when I predict with an image, all the images goes to the incorrect class, i dont know how to correct my model , use other metrics or some architechture for this type of images?.Here's a sample of the code https://github.com/Franciscogtu/OARSI-IMAGE-CLASSIFIER.git
Thank you for the help

Have you tried a batch normalization layer between the conv2D layers and the dense top? I'd also keep the number of filters low ,maybe 32 or 64. Same for the number of nodes in the top, no point in having a huge dense top with hundreds of nodes.


differences between feature weighting and feature selection what are the differences between feature weighting and feature selection? And is feature importance like feature weighting?

Feature selection is choosing among your features what are the most important ones.

Feature weighting is from the features that you have selected ranking the ones that contribute most to the model.

Just to keep a basic and simple response.


How to prepare/augment images for neural network? I would like to use a neural network for image classification.  I'll start with pre-trained CaffeNet and train it for my application.

How should I prepare the input images?

In this case, all the images are of the same object but with variations (think: quality control).  They are at somewhat different scales/resolutions/distances/lighting conditions (and in many cases I don't know the scale).  Also, in each image there is an area (known) around the object of interest that should be ignored by the network.

I could (for example) crop the center of each image, which is guaranteed to contain a portion of the object of interest and none of the ignored area; but that seems like it would throw away information, and also the results wouldn't be really the same scale (maybe 1.5x variation).

Dataset augmentation

I've heard of creating more training data by random crop/mirror/etc, is there a standard method for this?  Any results on how much improvement it produces to classifier accuracy?

The idea with Neural Networks is that they need little pre-processing since the heavy lifting is done by the algorithm which is the one in charge of learning the features.

The winners of the Data Science Bowl 2015 have a great write-up regarding their approach, so most of this answer's content was taken from:
Classifying plankton with deep neural networks. I suggest you read it, specially the part about Pre-processing and data augmentation.

- Resize Images

As for different sizes, resolutions or distances you can do the following. You can simply rescale the largest side of each image to a fixed length.

Another option is to use openCV or scipy.
and this will resize the image to have 100 cols (width) and 50 rows (height):

resized_image = cv2.resize(image, (100, 50))

Yet another option is to use scipy module, by using:

small = scipy.misc.imresize(image, 0.5)

- Data Augmentation

Data Augmentation always improves performance though the amount depends on the dataset. If you want to augmented the data to artificially increase the size of the dataset you can do the following if the case applies (it wouldn't apply if for example were images of houses or people where if you rotate them 180degrees they would lose all information but not if you flip them like a mirror does):

rotation: random with angle between 0° and 360° (uniform)
translation: random with shift between -10 and 10 pixels (uniform)
rescaling: random with scale factor between 1/1.6 and 1.6
(log-uniform)
flipping: yes or no (bernoulli)
shearing: random with angle between -20° and 20° (uniform)
stretching: random with stretch factor between 1/1.3 and 1.3
(log-uniform)

You can see the results on the Data Science bowl images.

Pre-processed images

augmented versions of the same images

-Other techniques

These will deal with other image properties like lighting and are already related to the main algorithm more like a simple pre-processing step. Check the full list on: UFLDL Tutorial


Can HDF5 be reliably written to and read from simultaneously by separate python processes? I'm writing a script to record live data over time into a single HDF5 file which includes my whole dataset for this project. I'm working with Python 3.6 and decided to create a command line tool using click to gather the data.

My concern is what will happen if the data gathering script is writing to the HDF5 file and the yet-to-be ML application tries to read data from the same file?

I took a look at The HDF Group's documentation about HDF5 parallel I/O, but that didn't really clear things up for me.

HDF5 parallel I/O will not solve this problem.  That technology is primarily intended for performance, not for collision avoidance.
What you want is know as SWMR (single-writer/multiple-reader):

Data acquisition and computer modeling systems often need to analyze and visualize data while it is being written. It is not unusual, for example, for an application to produce results in the middle of a run that suggest some basic parameters be changed, sensors be adjusted, or the run be scrapped entirely.
To enable users to check on such systems, we have been developing a concurrent read/write file access pattern we call SWMR (pronounced swimmer). SWMR is short for single-writer/multiple-reader. SWMR functionality allows a writer process to add data to a file while multiple reader processes read from the file.

SWMR was first included in HDF5 version 1.10.0 released on 2016-03-30

Concurrent Access to HDF5 Files - Single Writer/ Multple Reader (SWMR)
The Single Writer/ Multiple Reader or SWMR feature enables users to
read data concurrently while writing it. Communications between the
processes and file locking are not required. The processes can run
on the same or on different platforms as long as they share a common
file system that is POSIX compliant.


Keras: matching logistic regression performance with sequential neural network? Assume a binary classification problem and a relatively small dataset ($\sim \mathbb{R}^{5000 \times 39}$). By using common ML techniques, starting with logistic regression, I'm able to reach ~0.76-0.79 AUC on the validation sample, depending on the model. So there clearly are some significant signals to be extracted from the model.

However, when trying even a simplest sequential neural network, it does not learn at all.

I.e., using keras from R: (NB: AFAIK the language shouldn't be a problem, since it's only a wrapper for R)

model <- keras_model_sequential() %>%
         layer_dense(units = 48,
                     activation = "sigmoid",
                     input_shape = 39) %>%
         layer_dense(units = 48, activation = "sigmoid") %>%
         layer_dense(units = 2, activation = "sigmoid")

model %>% compile(
  optimizer = "rmsprop",
  loss = "categorical_crossentropy",
  metrics = c("accuracy")
)

history <- model %>% keras::fit(
  x_train,
  y_train,
  epochs = 500,
  batch_size = 16,
  validation_split = 0.7, shuffle = T
)

However, whatever number of epochs or batch sizes, the model doesn't learn at all, the validation accuracy corresponds to the distribution of the target variable (predicts all as 1).

What am I doing wrong? Shouldn't a small one/two layer NN with sigmoid activations at least reach the performance of logistic regression? Any suggestions of how to at least replicate the logistic regression results? Or maybe I've missed some steps in the code?

Would be greateful for any insight!

Some question you might want to think about:

Is your dataset big enough? What kind of data is it? Time series? Should you be shuffling? What learning rate are you using? Can you change it and see the effect on the learning curves: plot(history) output?

With regards to your model, you usually leave the last connected layers joined only by a linear activation (i.e. don't use an activation function, just an identity matrix). You have used the sigmoid all the way through, which is fine, but not for final layer! I have corrected this by not including such a non-linearity, rather the softmax activation, which will squash all values into the range of [0, 1], so they can be interpreted as probabilities.

I have increased the number of layers and neurons in the initial layers, and swapped in the preferred non-linearity: the Rectified Linear Unit (ReLU). I would recommend having a quick read of this intro from Stanford's CS231n course, which covers some of the best practices. Also, have a look here for a sample of a larger network performing classification, to see how R Keras is best used.

Give the following code a test, and also plot the history to get more intuition as to how the training is progressing: how many epochs might be necessary, whether you are over- or underfitting, etc.

model <- keras_model_sequential() %>%
    layer_dense(units = 200, activation = "relu", input_shape = 39) %>%
    layer_dense(units = 100, activation = "relu") %>%
    layer_dense(units = 100, activation = "relu") %>%
    layer_dense(units = 50, activation = "relu") %>%
    layer_dense(units = 2, activation = "softmax")

model %>% compile(
              optimizer = "rmsprop",
              loss = "categorical_crossentropy",
              metrics = c("accuracy")
          )

history <- model %>% keras::fit(
                                x_train,
                                y_train,
                                epochs = 500,
                                batch_size = 16,
                                validation_split = 0.7, shuffle = T
                            )

plot(history)                           # Are we overfitting?

Disclaimer: If you don't have much data, this model might be even worse that your original one - or you will massively overfit and get 100% training accuracy and terrible validation/test accuracy.

It could well be the case that your data is just better modelled with a simpler model, or that you do not have enough data to get a neural network to hone in on a nice optimum in its optimisation.


How to add noise to supervised (binary-classifier)? Note: The question is not about validating/testing a trained model.

Say i have an unlabeled features set, I want to approximate the true labels (for the sake of argument lets assume it's a binary classification prob).

I also have a trained model to predict the labels.
Now I want to use this model (since it is trained on the true labels, it should approx. my distribution)  along with some added noise (maybe gaussian) to generate the labels for this unknown dataset.
This should always result in a fixed effect size.

How can I do that? My features are mix of categorical and continuous values.

Would be awesome to illustrate the additive noise via python numpy package.

Apologies, I am a rookie!

I'm not sure why/where you want to apply the noise, but if you want to add some Gaussian noise to a variable, you can do this:

import numpy as np

target_dims = your_target.shape
noise = np.random.rand(target_dims)
noisy_target = your_target + noise

Now use the noisy_target as input to your model.


Testing Model Performance Once the Model is built we want to check its performance, i did the following

Predicted it on training set.
Compute confusion matrix and ROC curve on training set.
Predicted on test set
Computed Confusion Matrix and ROC Curve on test set.

I want to know is it a right way and what more precise can be done?

I am using Decision Tress  Random Forest SVM Both Linear Radial Naive Bayeian

Logistic regression and Neural Network.  Do all of them have different

Performance Measure

You should provide more information about your problem. How large is your data set? Do you have hyper-parameters? What is your classifier?

Normally, you should split your entire data set into three parts: training, validation and testing sets. The testing set you should never use before the model is finally tuned. If you data set is not large enough, you should you use the cross-validation method to measure the error of your classifier.

There are several metrics to assess a performance of your classifier: accuracy, precision and recall, Cohen's kappa, F1-score, etc.

If you are working with python and scikit-learn, I would recommend to check this book. It is nicely written and explains basics of model assessments.


How is a single element of the training set called? This question is only about the vocabulary.

Do / can you say

data item
data sample
recording
sample
data point
something else

when you talk about elements of the training / test set? For example:

  The figure shows 100 data items of the training set.

  Database A contains the same data items as database B, but in another format.

  The remaining data items were removed from the dataset.

  Those 10 classes have 123456 data items.

Please provide papers with examples.

According to Google n-grams:

The term you are looking for is "Example".
Source: Martin Zinkevich, Research Scientist at Google (http://martin.zinkevich.org/rules_of_ml/rules_of_ml.pdf)

Instance: The thing about which you want to make a prediction. For example, the instance might be a web page that you want to classify as either "about cats" or "not about cats".

Label: An answer for a prediction task  either the answer produced by a machine learning system, or the right answer supplied in training data. For example, the label for a web page might be "about cats".

Feature: A property of an instance used in a prediction task. For example, a web page might have a feature "contains the word 'cat'".

Example: An instance (with its features) and a label.


How does a Decision Tree handle an unknown category in a split? I have in mind a situation where a Decision Tree is trained with a dataset where one category has got just three possible values: A, B and C.

So as I understand the node for this category will have three splits: A, B and C.

What will happen in case an observation with a null for this feature arrives, or if in the testing set appears a value D for this category?

Does Decision Tree leave one of the splits as default to handle this situations?

There are two perspectives to this question, from a mathematical or machine learning perspective and from a technical perspective. From a technical perspective this depends on the implementation of the decision tree. An unseen value is not that different from a missing value, and sklearn for example does not deal with unknown values well and will fail on unseen or unknown values. Other tree implementations might deal better with this.

From a mathematical or ML perspective, without making some assumptions you cannot solve this problem of course. However an assumption you could make is that unknown or unseen values are average values. With that assumption you could follow both paths on the split where the unknown value occurs. Then you collect both results from the split and weigh them by the number of training samples that happened there. That way you can still make a prediction for these unknown values.


What is industry standard for workflow when training models? I'm assuming that each time someone trains a model and wants to tweak it/iterate, that they don't have to wait hours and hours for it to learn and then output. So my question is, how do people manage this workflow?

There are online models and there are offline models. Online models make sense when you need to have your model constantly be updated. Otherwise you can save money by updating your model as frequently as necessary; nightly, weekly, etc. The machines used to train models are resources that cost money, after all, and you do not want to reserve them longer than necessary.

One trick to save time is to use the last run's model parameters as initial values for the next iteration.


Predicting or patron find of a binary variable over time I'm new to ML and trying to find some practical use to it I encountered with the chance of saving the connections and disconnections (the binary variable) of a bunch of users like this:

"User A connected at 10:02:33

User A discconected at 10:05:02"

I was wondering how could I determine whenever the user is going to connect again, analyzing his pasts connections, disconnections and time online.

My concers are the following:

Would this be posible?
What's the best method for this?
How much samples per user would I need?
What would be the best way to structure the data, and besides the user identifier, action(connect/disconnect) and time the action take place what other info would be useful?

An just as an extra question: Would it be possible to add more data in real-time to improve the algorithm prediction? How?

Since you say you are a beginner, I suggest using a simple binary classifier: logistic regression. It takes the input vector, and transforms it by taking its dot product with a weight parameter (estimating which is the goal) before passing it through the logistic function:

$\hat y(x) = \dfrac{1}{1+\exp(-\left< w, x \right>)}$ and $y=\begin{cases}1, \text{user present} \\ 0, \text{user absent} \end{cases}$.

For details read the wikipedia article. For features, I would use the time of day encoded as $\left( \sin (2\pi t/T), \cos (2\pi t/T) \right)$, a hot-encoded categorical variable for the day of the week, a binary variable to indicate weekends, and so on. It's a crude model that leaves a lot of data on table, such as correlations between users, but as a beginner's exercise it is appropriate.


Ensemble model overfitting? I am attempting a classification project. I have split my data, 20000ish, into training and test sets. On the training sets I run a selection of classifieds including knn random forests and gbm . These give me about 20-30% accuracy at best. For each sample I generate probabities of each class and make a new model

Knn proba 1, knn proba 2.... Random forests proba 1 etc

On this I then run a random forest classifier which gives me a 90ish% accuracy against the test set.

Fantastic!... But when I use the model against new data the accuracy is very poor.

In part it feels like a case of overfitting but surely the test set should be poor as well

Why might the test data so good but new data so poor... What have I done wrong?

Thanks
Chris

Yes I suspect you are overfitting. When you build your first stage of models (nearest neighbors, random forest, gradient boosting, etc...) is the process like this?

random_forest.fit(train_data,target)
random_forest_probabilities = random_forest.predict_proba(train_data)
...

If so you are plugging in the same data you used to train the models into your models to get probabilities, which will lead to better probabilities than should be expected in real life and thus much better than should be expected results on your 2nd stage model (i.e.: overfitting). To remedy this you must do everything within a cross validation loop, that looks like this in pseudo code:

for <fold> in <crossvalidation>:
     random_forest.fit(<train_data_fold>,<target_fold>)
     <fold predictions> = random_forest.fit(<test_data_fold>)
     ...

Doing things this way will emulate real life since your predictions will be gathered using models that did not include that data when training them. You can build your final model using all of your prediction variables as input. If you struggle with actually executing on this either edit your question with code or post a new question on how to run everything with a CV loop.


Neural Network - Sparsity of collaborative based filtering and modelling the prediction problem I'm fairly new to machine learning and for that matter, neural networks, but for the past couple of days I decided to take a stab at a fairly classical and practical problem of neural networks/machine learning which is recommendation systems.

Apologies if this is an unnecessarily broad question, but I found it hard to read up on resources answering this particular question. My main question is, how do you even model the problem (or what directions/advice is there on how to model it)?

Let $M$ denote the set of all movies you can recommend, each of which have an associated $id$ to them. What exactly would the input to the model be (and the output) and how would you segregate the training data and the observed results? For example, if I have a row of training data (row of size $|M|$ and each entry is a number between $1$ and $10$ that denotes the user's rating, or $0$ if the user has not seen it yet), would I just remove, say half of the user's ratings that are between $1$ and $10$ (so are meaningful) and replace them with $0$ in the training row and then in my observed row (for testing) it would be the entire row with no missing ratings like illustrated below?

$$\underbrace{[9 \ 0 \ 0 \ 5 \ 7 \ 0 \ ... \ 8 \ 0 \ 10 \ 0]}_{Information} \rightarrow \underbrace{[0 \ 0 \ 0 \ 0 \ 0 \ 0 \ ... \ 8 \ 0 \ 10 \ 0]}_{Training} \rightarrow \underbrace{[9 \ 0 \ 0 \ 5 \ 7 \ 0 \ ... \ 8 \ 0 \ 10 \ 0]}_{Observed} $$

A few naive ways that are obvious are:

Input size = $|M|$, output size = $|M|$

Each input neuron is given the associated user rating for the $i^{th}$ movie. Assume some number of intermediate layers and then we get to the output size, which is $|M|$ output neurons each of which attempt to predict the predicted rating of the user for that movie. This seems weird, because basically almost all of the input neurons will be given $0$ and almost all of the output neurons will try to output $0$ (since it's likely a user has only seen an insignificant portion of all $|M|$ movies). This doesn't seem like the neural network will learn anything useful. And what do I do for movies that the user hasn't seen? Do I just consider this a $0$ loss? So the loss function for neuron $i$ is just:
$$\text{Loss}(y^{(i)}) = 0 \ \text{if the user hasn't seen ith movie, otherwise: } \ \phi{(y^{(i)}, t^{(i)})}$$

Where $\phi$ is some loss function of the neuron predicted output $y^{(i)}$ and the observed value (the actual user rating) $t^{(i)}$. I find it difficult to believe that this will yield any useful ratings though..since the loss has no effect on the nonsensical outputs of my neural network (it doesn't affect the $i^th$ neurons' values if the user hasn't seen it yet). This also poses the problem of how a prediction will be made then. Do I just take all the highest predicted values from the neural network and return the respective movies?

Because of the above problems, I tried to express the model differently, but I don't think there is anyway to overcome the sparsity of the data and also the fact that the output size must be $|M|$ as I can't have it try to predict discrete integer movie ids. Could someone please enlighten me on this topic or provide some possible insight to obvious mistakes I am making?

You don't have to use full input and output vectors to predict the ratings for a user.

Use the embedding idea and make pairs of (user_id,movie_id) for training. So your input could be like this: (112,1456)=7, it means the user with id number 112 watched the movie number 1456 and the rating was 7.So the size of embedding for the User is  |unique users|  and movies are |unique movies| you call it |M|.

With this trick, we changed the structure of the problem to what we already know. It's regression!

Now you can add all the bias(and whatever you want) to movies and users embeddings. Use the known loss functions for regression like
RMSE and you all set. For sure you can add other linear layers, relu, and dropouts on top of this embeddings and make it just like fully connected NN and get even better results.


Problems finding an LSTM model for classification I am doing a study for the classification of musical genres using deep learning techniques. The work consists of making a classification using an LSTM model.

I am using GTZAN as a data set, and preprocessing them using Librosa. This way I obtain the spectral characteristics, specifically: Spectral centroid.

My input data has the following size: X_train (750, 20, 1249).

My model is:

model = Sequential ()
model.add (LSTM (units = 512, return_sequences = True, input_shape = (X_train.shape [1], X_train.shape [2])))
model.add (LSTM (units = 32, return_sequences = False))

model.add (Dense (units = 10, activation = 'softmax'))
model.compile (loss = losses.categorical_crossentropy,
                 optimizer = Adam (),
                 metrics = ['accuracy'])

model.summary ()

I get these results:

I scale also my data using MinMaxScaler.

I appreciate any help. Thank you.

Since you are preprocessing the data using Librosa ( maybe extracting MFCCs ), I assume that the preprocessing is correct and needs no intervention.

On viewing the first two layers,

model.add (LSTM (units = 512, return_sequences = True, input_shape = (X_train.shape [1], X_train.shape [2])))
model.add (LSTM (units = 32, return_sequences = False))

You can notice that the difference between the units present in the layers is quite huge ( 512 and 32 ). Hence the 512-dimensional data is packed in a 32-dimensional vector immediately in the next layer.

  You can try reducing this difference. You can use 64 - 32 units in both the layers.

Also, most sources and blogs, say that for recurrent neural networks, RMS optimizer is the best choice. Since we use the hyperbolic tangent activation function, there is a problem with it called exploding gradients. RMSProp optimizer can handle such problems efficiently since the square root of gradients is utilised here.

  Try using the RMSProp optimizer with a low learning rate.

I haven't tried this myself, but you can use recurrent or input dropout in LSTMs. They may help you to overcome the overfitting problem.

  Also, you can try adding some more Dense layers with Dropout layers in between them.

Tip:

LSTMs commonly face the problem of overfitting, so you can implement any of the other methods which are meant to eliminate overfitting. Also, as a common step you can try reducing the batch sizd during training.


MLP classifier Gridsearch CV parameters to tune? I'm looking to tune the parameters for sklearn's MLP classifier but don't know which to tune/how many  options to give them? Example is learning rate. should i give it[.0001,.001,.01,.1,.2,.3]? or is that too many, too little etc.. i have no basis to know what is a good range for any of the parameters. Processing power is limited so i can't just test the full range. If anyone has a general guide of which are the most important to tune and a general range that would help me a lot moving forward. Thanks!

Usually learning rate would be between 0.0 to 1.0 , please go ahead by specifying values in those range.
This thread would be definitely helpful for your further research and reference
Thanks,
Durga


How is the target variable passed to the final estimator in this pipeline? There is a pipeline like below. X is features and y is the target variable.
I would like to know how y is passed to the estimator, LinearSVC. As far as I know, StandardScalerreturns only transformed X. So, I thought that y was not passed to LinearSVC. However, this code worked and I could make prediction. Thus, I would like to know how y was reached to the final estimator.
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.svm import Linear SVC
svm_clf = Pipeline([
            ("scaler",StandardScaler()),
            ("linear_svc",LinearSVC(C=1,loss="hinge")),
          ])
svm_clf.fit(X,y)

The Pipeline object performs a .fit_transform(X, y) sequentially based on how you structure your pipeline until just before the last step (the estimator). The estimator does not perform a transform because it's purpose is not to transform your input array X, but to make a prediction. The fit_transform method performs a fit, then a transform on that pipeline step when it is called. In your case that means:

Your standard scalar is fit based on X and y
Your standard scalar transforms your input data X
Your LinearSVC is fit based on your previously transformed X and your un-transformed target y

Since your LinearSVC is the last step of the pipeline object and doesn't have a transform method the process stops. Your Pipeline object that contains your StandardScalar and the LinearSVC now contains a fit version of both these objects. Now you can perform a .predict() method with this pipeline which will cause the fit StandardScalar to perform a .transform() on input X, then this transformed X will be passed to the final estimator LinearSVC for a .predict() which will return your predicted y array.


Training dataset decreasing in quality (Google data science blog) I have a complex algorithm that decides when it should show customers of an only shop an ad on our website, after they log in, in hope that they will buy what is in the ad. We have no control what is in the ad, because another department of our company handles that - our model only chooses to whom the ad is shown (we would like to keep the number of people to whom we show the ad as low as possible, because we will just offend customers with the ad otherwise; this means don't show the ad to people who log in with an intent to buy something, only show it to those that would not buy anything if they weren't presented an ad). The model was training using training data where we randomly showed some people ads and some not and recorded their respective response.

An initial implementation of the model is running live, and works decently well in classifying to whom it makes sense to show the ad. But each day new data is acquired and the model is retrained using the whole dataset (which with each retraining gets larger) every few days. But there is the worry that its performance might decrease if we keep doing this, because the current model instance influences the new data that comes in, that is used to train future model instance: The percentage of the customer who will not be shown ads randomly will decrease and there will be more and more data of customer that were shown an ad only if the model believes they will buy what is on it.

Are we right to worry that our data will become biased in the future?

The [unofficial Google data sicence blog][2], section "Using randomization in training", might contain useful information about this problem, but unfortunately it is too technical for me to make much sense of all of it. Other than the message "Yes, worry about this: but keep sending out a small part of ads randomly and you will be fine", which I got, what else could I do to reduce this problem?

Here are some things to consider. You may already be doing some of them, but it's hard to tell.

As new data arrive, add them to your database. Randomly sample, with replacement, your database. All other things being equal, take a 2/3 sample and reserve 1/3 for testing. Record the training error and the prediction error.
At intervals, take past training data sets and measure prediction error on a sample of new data only.
Prepare time series of the training error, contemporary prediction error and post-training prediction error.
Look for trends and temporal autocorrelation.
Consider possible seasonality in demand for the offered product.
Consider market changes (entry of new competitor, substitution goods, price points)

This is just the view from 30,000 feet. It's hard to say more without knowing more about your model.

Good luck.


Why do activation functions have to be monotonic? I am currently preparing for an exam on neural networks. In several protocols from former exams I read that the activation functions of neurons (in multilayer perceptrons) have to be monotonic.

I understand that activation functions should be differentiable, have a derivative which is not 0 on most points, and be non-linear. I do not understand why being monotonic is important/helpful.

I know the following activation functions and that they are monotonic:

ReLU
Sigmoid
Tanh
Softmax: I'm not sure if the definition of monotonicity is applicable for functions $f: \mathbb{R}^n \rightarrow \mathbb{R}^m$ with $n, m > 1$
Softplus
(Identity)

However, I still can't see any reason why for example $\varphi(x) = x^2$.

Why do activation functions have to be monotonic?

(Related side question: is there any reason why the logarithm/exponential function is not used as an activation function?)

The monotonicity criterion helps the neural network to converge easier into an more accurate classifier.
See this stackexchange answer and wikipedia article for further details and reasons.

However, the monotonicity criterion is not mandatory for an activation function - It is also possible to train neural nets with non-monotonic activation functions. It just gets harder to optimize the neural network.
See Yoshua Bengio's answer.


how to predict an image using saved model from keras import optimizers
from keras.models import load_model
from keras.preprocessing import image
import numpy as np
import cv2
import scipy.misc
from keras.wrappers.scikit_learn import KerasClassifier
# dimensions of our images
img_width, img_height = 313, 220
# load the model we saved
model = load_model('model.h5')
sgd = optimizers.SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy','mse'])
test_image= image.load_img('/Images/1.jpg',target_size = (img_width, img_height))
#x= scipy.misc.imread('/Images/1.jpg').shape
test_image = image.img_to_array(test_image)
print test_image
test_image = np.expand_dims(test_image, axis = 0)
test_image = test_image.reshape(img_width, img_height*3)
result = model.predict(test_image)
print result

When I ran the above code I get this error:

  ValueError: Error when checking : expected dense_1_input to have shape (36,) but got array with shape (660,)

Having problems with dimensions is very common. As others have mentioned, the method predict expects to get a batch of images. You should run model.summary() to see what the expected dimensions of the input are. The batch size itself might have been designed to be flexible during training, by specifying None in the first dimension on the model input_shape parameter.

Without knowing more of the dimensions, I cannot say exactly where your problems lies. You could include that information by editing your question.

It will likely help to have a look at the documentation for predict function of the model objects.

There is an argument: batch_size, which defaults to 32 if not fixed by the model itself, which you can see from the model.summary(). If you set this equal to 1, perhaps you will get a prediction.

Below is a modified version of your code that I would expect to return a prediction.

from keras.models import load_model
from keras.preprocessing import image
import numpy as np

# dimensions of our images    -----   are these then grayscale (black and white)?
img_width, img_height = 313, 220

# load the model we saved
model = load_model('model.h5')

# Get test image ready
test_image = image.load_img('/Images/1.jpg', target_size=(img_width, img_height))
test_image = image.img_to_array(test_image)
test_image = np.expand_dims(test_image, axis=0)

test_image = test_image.reshape(img_width, img_height*3)    # Ambiguity!
# Should this instead be: test_image.reshape(img_width, img_height, 3) ??

result = model.predict(test_image, batch_size=1)
print result

Note: there is ambiguity in the dimensions where I highlight it. A colour image will have three dimensions: (height, width, 3). Black and white will have only two dimensions: (height, width).


Where can Batch Normalization be used? CNNs or everywhere? Should BatchNormalization be used only in CNNs or can they be used in Fully Connected Networks, Recurrent networks as well?

A Batch Normalization layer essentially is normalizing outputs of your hidden units, so their outputs would always have a similar scale. By eliminating such internal covariate shift in outputs of hidden layers, your deeper layers become more independent from their previous layers.

Nothing here is CNN specific - Batch Normalization may be applied to Fully Connected and Recurrent neural networks as well, but they are more useful with deep neural networks, which tend to accumulate this shift with each layer activated during Forward Propagation.


What’s the best way to save many pandas dataframes together? I’m looking for a way to save house prices data by city, for example a pandas panel with one dataframe per city. But I need the dataframes to be independent, meaning that if one dataframe is corrupted, the others are untouched.
I tried using pickle and csv, but once a line was corrupted I lost the whole file.

If you want to get quite involved and be able to specify names for each of the panels you create, you could look at the h5 file format.

This allows you to group datasets in named containers. You can then read them from disk later on one by one i.e. you don't need to read the whole dataset into memory.

Here is an example of a function that would save such a dataset:

def save_h5(h5_filename, data, labels, descr=None,
            data_dtype='float32', label_dtype='float32'):
    """Create a compressed .h5 file containing:
    data    : numpy array
    labels  : numpy array
    descr   : text description ofthe data contained (must be a string)
    """

    if os.path.exists(h5_filename):
        # prevent overwriting a file
        sys.exit('File already exists!')

    h5_fout = h5py.File(h5_filename)

    h5_fout.create_dataset(
        name='data',
        data=data,
        compression='gzip', compression_opts=4,
        dtype=data_dtype)

    h5_fout.create_dataset(
        name='labels',
        data=labels,
        compression='gzip', compression_opts=4,
        dtype=label_dtype)

    if descr is not None:
        h5_fout.create_dataset(
            'description', data=descr)

    h5_fout.close()

For the meaning of the parameters, head over the documentation.

You can write a similar function to make accessing the saved h5 file. This really is a flexible way to save data, and it can be compressed with one of the best known (widely-spread) algorithms in the open-source world: gzip! There are also other possibilities implemented.

On a side note, if you want to minimise the possibility of corruption, you could consider saving each panel/DataFrame (whichever method you go for) into separate files, and then make copies/backups.

Additionally, you said:

  I tried using pickle and csv, but once a line was corrupted I lost the whole file.

... the beauty of a simple csv file is that you can actually open it in notepad or a spreadsheet and usually find the line which is "broken" and fix/delete it. Pickle, on the other hand, is a little more complicated to debug.


Is the graphic of deep residual networks wrong? I am currently wondering if the following graphic of deep residual networks is wrong:

I would say the graphic describes

$$\varphi \left (W_2 \varphi(W_1 x) + x \right ) \qquad \text{ with } \varphi = ReLU$$

The $\mathcal{F}(x)$ does not make sense to me. Assuming both weight layers are simple MLPs without bias, where the first one has a weight matrix $W_1$ and the second one has a weight matrix $W_2$, what is $\mathcal{F}$?

In the text, they define

$$\mathcal{F}(x) := \mathcal{H}(x) - x$$

where $\mathcal{H}(x)$ is "the desired underlying mapping" (whatever that exactly means).

Also, equation (1) seems strange to me:
$$y = \mathcal{F}(x, \{W_i\}) + x$$
In figure 5 they have two weight layers and call this a building block. Why is there only one weight matrix in this equation?

My thoughts

I think the authors could mean
$$\mathcal{F}_i = \varphi(W_i x)$$
In that case, in the image where $\mathcal{F}(x)$ is it should be
$$\mathcal{F}_1(x) = \varphi(W_1 x)$$
and where $\mathcal{F}(x) + x$ is should be
$$\mathcal{F}_2(\mathcal{F}_1(x)) + x = \varphi \left (W_2 \varphi(W_1 x) + x \right )$$

(Per the diagram), $F(x)$ here is simply the entire two-layer non-linear chain that is operating on the input $x$. Then, the final output is simply $F(x) + x = H(x)$. That's it!

The thing that may be confusing you is that $F(.)$. In this case, they do not mean for $F$ to simply encompass one operation. Instead, it encompasses any set of operations processing $x$, up until you add $x$ back. Hope that helps!

PS: It is also common to see this type of nomenclature in a lot of DNN literature, whereby one refers to an entire deep non-linear chain as $D(x)$. For example in Generative Adversarial Networks, (GAN)s, $D(x)$ refers to the entire deep net devoted to the discrimination process, while $G(x)$ refers to the entire net devoted to the noise shaping. In both cases, they are composed of entire functions/nets, and do not signify simply one operation.


ValueError while using linear regression I have loaded a dataset and converted into data frame while I am using linear regression I am receiving the following value error as shown in my code below. I am at the moment doing a tutorial and could not figure out how come the arrays are 1-D as the error shows.

from sklearn.datasets import load_boston
boston_dataset=load_boston()

#create a pandas dataframe and store the data
df_boston=pd.DataFrame(boston_dataset.data)
df_boston.columns=boston_dataset.feature_names

#append price, target as a new column in the dataset
df_boston['Price']=boston_dataset.target

#print first five observations
df_boston.head()

CRIM    ZN  INDUS   CHAS    NOX RM  AGE DIS RAD TAX PTRATIO B   LSTAT   Price
0   0.00632 18.0    2.31    0.0 0.538   6.575   65.2    4.0900  1.0 296.0   15.3    396.90  4.98    24.0
1   0.02731 0.0 7.07    0.0 0.469   6.421   78.9    4.9671  2.0 242.0   17.8    396.90  9.14    21.6
2   0.02729 0.0 7.07    0.0 0.469   7.185   61.1    4.9671  2.0 242.0   17.8    392.83  4.03    34.7
3   0.03237 0.0 2.18    0.0 0.458   6.998   45.8    6.0622  3.0 222.0   18.7    394.63  2.94    33.4
4   0.06905 0.0 2.18    0.0 0.458   7.147   54.2    6.0622  3.0 222.0   18.7    396.90  5.33    36.2

#assign features on x-axis
X_features=boston_dataset.data

#assign target on y-axis
Y_target=boston_dataset.target

#import linear model-the estimator
from sklearn.linear_model import LinearRegression
lineReg=LinearRegression()

#fit data into the estimator
lineReg.fit(X_features,Y_target)

ValueError                                Traceback (most recent call last)
<ipython-input-16-4b5b068e587b> in <module>()
      1 #fit data into the estimator
----> 2 lineReg.fit(X_features,Y_target)

/usr/local/lib/python3.4/dist-packages/sklearn/linear_model/base.py in fit(self, X, y, sample_weight)
    480         n_jobs_ = self.n_jobs
    481         X, y = check_X_y(X, y, accept_sparse=['csr', 'csc', 'coo'],
--> 482                          y_numeric=True, multi_output=True)
    483
    484         if sample_weight is not None and np.atleast_1d(sample_weight).ndim > 1:

It seems in your code you build a pandas dataframe but you do not use it. I recreated your code line by line and was unable to get the same error. Try the following

from sklearn.datasets import load_boston
from sklearn.linear_model import LinearRegression
boston = load_boston()
X = boston.data
Y = boston.target

lineReg = LinearRegression()
lineReg.fit(X, Y)
lineReg.score(X, Y)

This results in an error of $0.7406$. Of course, this result is kind of meaningless because you should split your data into a training and testing set in order to accurately test your results.

You should do the following

from sklearn.datasets import load_boston
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

boston = load_boston()
X = boston.data
Y = boston.target
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, shuffle= True)

lineReg = LinearRegression()
lineReg.fit(X_train, y_train)
lineReg.score(X_test, y_test )

This will always give a different score because the data is shuffled before being split into your two sets. You MUST separate your training data are the start of your algorithm and do not pollute your training set with the testing set otherwise you risk biasing your model! This is very important.