Distance measure calculation addresses for record linking At the moment we use different methods for record linking locations in different datasets.

Theoretically given two locations we can give a prediction on how well they match (are the same). This is not just based on address data (street, house number, zip, city, country, latitude, longitude) but also based on the name, type of establishment and other properties like phonenumber.
Since most features are prone to fuzzy errors (different spellings, writing styles, formatting, human entry error, null-values (absent)) this means we cannot do strict comparisons.

Using many business rules, basic distance calculations, etc.  we can clean & reformat the data so we can have a pretty good comparison and high accuracy on matches.

eg.

  Datlinq BV,
  Roer 266,
  2908MC, Capelle aan den IJssel

is the same as

  datlinq,
  266 Roer,
  capelle a/d yssel

but not as

  Ferro BV,
  Roer 266,
  2908MC, Capelle aan den IJssel

Our problem is many a scaling issue:

We need smart ways to reduce the dataset so we don't compare every location in dataset A to every location dataset B.

At this moment we do some pre-filtering (boxing) on latitude / longitude (if present)

However I was wondering if we could somehow generate a vector for each data location and us a basic distance calculation to find matches (or match candidates) in other datasets.

Or maybe another way to find very similar locations across huge datasets.

I'd love to use a machine learning model for this, but I am stumped how to generate numerical features out of a string address, location type & telephone number, without applying to many business rules.

So the actual question is:

Are there smart ways to do (parallel) record linking across big datasets (using Spark or so)
Is the approach of converting locations to vectors using a ml model and then using a distance calculation to find nearby points a good approach?
Is there terminology I should use to find more information about this problem? Are there papers / experiments?

-- Update 20170829 151213

So we are working on prefiltering datasets based on nearby locations, but unfortunately for many datasets we don't have reliable lat/longs. So I'm looking for a more general approach.

So based on the fact that we cannot positively resolve a nearby location without comparing textual addresses. A smart heuristic is something I'm looking for, but I want to incorporate weights to certain (fuzzy) features, eg. zipcodes should match more closely towards the beginning then towards the end, phonenumbers other way around. Streets and cities maybe spelled very differently including abbreviations.

Sure I can extract all these features and create business rules with arbitrary weights, but I'd rather have an unsupervised ML model learn the most important features based on positive and negative test/trainingsets of address matches.
I would also guess that weights will vary differently by country/language so maintaining these by hands seems a suboptimal solution.

Is the problem that you can resolve a new address B to a location reliably, but, don't know if it's the same address as an existing address A? then it seems straightforward to perform the expensive check for near-matches on only addresses that also resolve to a nearby location, and consider them in order of distance. The simple distance calculation shouldn't be expensive.

Or are you saying that you can't resolve B to a location by itself, and have to resolve it by finding a fuzzy match with some A, but you don't know where A might be? Then you need some cheap heuristic match to compute over all records. What about simply the distribution of characters/digits in every string? easy to compute, small, and easy to compare. Presumably near-matches have nearly the same letters and digits, so might correlate well with actual fuzzy matches.

Edited to add:

You could try to build a classifier that can tell when two different strings represent the same address. That's not your ultimate goal, I know. But along the way you might be able to learn a useful intermediate vector representation of the strings that is relevant for determining same/not same. And then use that intermediate representation.

In theory that's what a neural network could do. Inputs are two strings; they share weights/layers that transform to an intermediate rep; then the top of the network learns same/not same.

"Simple as that" but in practice you might find that learning this way takes such a huge network to do well that it's infeasible.

Maybe someone knows an equivalent idea that doesn't need the brute force of deep learning.


How is H2O faster than R or SAS? I am trying to understand the abstract details that explain how h2o is faster than R and SAS for data science computations.

I have used R, SAS Base and H2O. First, I do not think that H2O seeks to be either R or SAS.

H2O provides data mining algorithms that are highly efficient. You can interface with H2O using several APIs such as their R API. The benefit of combining R and H2O is that H2O is very good at exploiting multi-cores or clusters with minimal effort of the user. It is much harder to achieve the same efficiency in R alone.

The reason why H2O is much faster is that they have a very good indexing of their data and their algorithms are written such that they exploit parallelism to the fullest. See http://h2o.ai/blog/2014/03/h2o-architecture/

R with the default matrix dynamic libraries can only use one CPU core. Revolution R community edition ships with the Intel Math Kernel Library. This allows for some matrix computations in parallel but definitely not as efficient as H2O. For SAS it is a bit harder to say anything considering it's closed source but based on my CPU utilization I would assume that they have a similar approach as Revolution R. Their matrix algebra exploits parallelism but they algorithms are not as efficient as H2O. Their data storage is also not as efficient as H2O.

Lastly, H2O with R comes at a very different price tag than SAS.

Hope this clarifies a bit.


How does skewed data affect deep neural networks? I'm playing around with deep neural networks for a regression problem. The dataset I have is skewed right and for a linear regression model, I would typically perform a log transform. Should I be applying the same practice to a DNN?
Specifically, I'm curious how skewed data affects regression with a DNN and, if it's negatively, are the same methods that would be applied to a linear regression model the right way to go about fixing it? I couldn't find any research articles about it but if you know of any feel free to link them in your answer!

Strictly theoretically it makes no difference on DNN, I answered it today here and I said:
Here is why: We already know mathematically that NN can approximate any function. So lets say that we have Input X. X is highly correlated, than we can apply a decorrelation technique out there. Main Thing is, you get X` that has different numerical representation. Most likely more difficult for NN to learn to map to Outputs y. But still in Theory you can Change the architecure, Train for longer and you can still get the same Approximation, i.e. Accuracy.
Now, Theory and Praxis are same in Theory but different in Praxis, and I suspect that this Adjustments of Architecture etc will be much more costly in reality depending on the dataset.
BUT I want to add another point of view: Convergence speed. Strickly theoretically you dont even need [batch normalization] for performance (you can just adjust weights and bias and you should get same results) but we know that making this transformation has big benefits for NN
To conclude for you: Yeah, I had experience where it made difference, and where it didnt. You cant expect theoretical results that say skewed is bad


Understanding Logistic Regression Cost function Linear Regression cost function:

$$J(\theta) = \frac{1}{2 m} \sum_{i=1}^m (h_{\theta}(x^{(i)}) - y^{(i)})^2$$

where:

$$h_{\theta}(x) = \theta_0 + \theta_1 x_1$$

Logistic Regression cost function

$$J(\theta) = \frac{1}{m} \sum_{i=1}^m - (y^{(i)} \times log(h_{\theta}(x{(i)})) + (1-y{(i)}) \times log(1 - h_{\theta}(x{(i)})))$$

where:

$$h_{\theta}(x) = g(\theta_0 + \theta_1 x_1 + \theta_2 x_2)$$

Intuitively linear regression is easy to understand as it optimizes the average squared distance between hypothesis and training data. But in case of logistic regression, I failed to understand the cost function.

What does the logistic regression cost function represent?

The way my intuition works for logistic regression is simple, say you are trying to classify if you have a picture with a dog on it, if that is the case, you output 1 (true), if not, you output 0 (false)

The cost function essentially represents the diff between what your model output (imagine, you have a dog in a picture, and your model outputs 0.75 instead of 1, your cost is 0.25).

Now, this is not a very mathematical way to put it, but at least it has helped me a lot to understand logistic regression.


Outlier/Anomaly Detection History I have been reading about different methods of anomaly detection, their structure and the way they work.

Recently I have been trying to find some scholar articles, writings or books where I can learn more about the history of anomaly detection.

Until now I was not able to find anything, would appreciate if any of you would give me and help or suggestions regarding this.

I highly recommend you take a look at this anomaly detection survey paper. It not only provides with a great review but also categorises the different approaches focusing in different types of data or domains where these techiques have been applied.

If you go through Section 1.3, you will find a series of interesting references which go as far as 1985. So you could have a look at those to make sense of the historic evolution of these approaches from numerical, statistical or ML standpoints.

Also, since the paper is quite dated already, I suggest a reading about new methods such as deep learning methods to complement your knowledge. A nice starting point could be this deep learning survey paper, which I think is another great more recent survey paper.


Including biased data in training model My friend is in the business of getting cats to the top of mountains. He currently uses a set of heuristics to decide which cats are most likely to get to the top.

For the ones he likes, he feeds, at great expense, in the hope that they will be strong enough to get to the top. For the ones he doesn't like, he thinks it not worth his time / money to feed, and he's happy and surprised if they get to the top anyway. From looking at the data, we know that on average, the ones he doesn't feed get to the top less often.

I am training a classifier to help him decide which to feed. Does it make sense to include rows in the training data where he fed them and where he didn't? When I excluded the rows where he did not feed them, the model did worse, which I did not expect.

Forgive the example. It's contrived, but I think retains all of the characteristics of the actual situation.

That's a wonderful question, and beautifully posed.  I love how you capture the essence of the issue in such a clean way.  Unfortunately, the answer is that your data set does not have enough information to help you decide which cats you should feed.  Without a properly controlled experiment, you don't have a way to infer causality; you can't rule out the possibility of confounding factors.

Suppose that 70% of the cats he feeds make it to the top, and 20% of the cats he doesn't feed make it to the top.  Let me make up four alternative explanations for why that might happen:

Hypothesis #1: There's only one kind of cat in the world.  The characteristics of the cat are irrelevant.  If you feed them, they will have a 70% chance of making it to the top; if you don't, they'll have only a 20% chance.  This is true for all cats, regardless of their attributes.
Hypothesis #2: There are two kinds of cats in the world: the strong, and the weak.  The strong have a 70% chance of making it to the top of the mountain, regardless of whether you feed them or not.  The weak have a 20% chance of making it to the top, regardless of whether you feed them.  You can tell the two kinds of cats apart by their characteristics, and your friend happens to like the strong ones and dislike the weak ones.
Hypothesis #3: There are two kinds of cats in the world: the athletes, and the champion nappers.  The former have a 70% chance of making it to the top of the mountain if you feed them, or a 50% chance if you don't.  The latter have a 40% chance of making it to the top if you feed them, or a 20% chance if you don't.  You can tell the two apart by their characteristics, and your friend happens to like the athletes and dislike the nappers.
Hypothesis #4: There are two kinds of cats in the world: the hungry, and the overweight.  The former have a 70% chance of making it to the top of the mountain if you feed them (they're pretty fit, but they already have all the food they need, so why bother exploring?), or a 100% chance if you don't (hunger drives them to greater heights).  The latter have a 21% chance of making it to the top, if you feed them, or a 20% chance, if you don't (either way, they probably won't make it, because they are so overweight and out of shape).  You can tell the two apart by their characteristics, and your friend happens to like the hungry ones and dislike the overweight ones.

All four hypotheses are equally consistent with the data.  The data set is useless at helping you distinguish between them.  Yet each hypothesis leads to a different prediction about which cats to feed: the first two say it doesn't matter, the third says you should feed the ones your friend likes, and the fourth says you should feed the ones your friend dislikes.  No amount of statistics or machine learning is going to enable you to make a decision based solely on the data.

To distinguish between these possibilities, you either need some kind of prior that will give you a basis for choosing among the models that are consistent with the data (perhaps based on your domain knowledge), or you need to conduct a controlled experiment (where the choice of whether to feed a cat is randomized, rather than being selected based on whether your friend likes the cat).

What if we have a prior that says that feeding a cat can only help its chances of making it to the top, and not hurt its chances?  Well, that's an example of the first of two options I suggest in the last paragraph.  However, it's still not informative enough to let us decide which cats to feed.

The problem is that we need to tell whether feeding the cats your friend likes improves their odds more than it improves the odds of feeding the cats your friend doesn't like.  We just don't have any information on that.

The chances could be 70% vs 69% for cats he likes and 90% vs 20% for cats he doesn't (i.e., cats he like have a 70% chance if fed or a 69% chance if not fed, etc.); or it could be 70% vs 0% for cats he likes and 21% vs 20% for cats he doesn't.  In the former case, you should feed the cats he doesn't like, as they see a 70 percentage point improvement from feeding, while the ones he likes see only a 1 p.p. improvement.  In the latter case, you should feed the cats he does like, because they see a 70 p.p. improvement, while the ones he dislikes see only a 1 p.p. improvement.  You just can't distinguish between these two possibilities from the dataset you have.  So your prior would need to be considerably more detailed than that, before you can make any headway.

If you'd like to learn more about the subject and under what circumstances we can draw conclusions about causality despite the lack of a controlled experiment, you might start by reading Judea Pearl's work.


How to preprocess with NLP a big dataset for text classification TL;DR
I've never done nlp before and I feel like I'm not doing it in the good way. I'd like to know if I'm really doing things in a bad way since the beginning or there's still hope to fix those problems mentioned later.
Some basic info
I'm trying to do some binary text classification for a university task and I'm struggling at the classification because the preprocessing with NLP is not being the best.
First of all, it's important to note that I need to have efficiency in mind when designing things because I'm working with very large datasets (>1M texts) that are loaded in memory.
This datasets contains data related to new articles with title, summary, content, published_date, section, tags, authors...
Also, it's important to mention that as this task being part of a learning process I'm trying to create everything by myself instead of using external libraries (only for boring or complex tasks)
Procedure
The basic procedure for the NLP preprocessing is:

Feature extraction -> str variable with title, summary and content attributes joined in the same string
Lemmatization -> same str as input but with lemmatized words
Stopword filtering
Corpus generation -> dict object with lemmatized words as key and the index they're being inserted in the dictionary as value.

After generating the corpus with all those samples, we can finally safely vectorize them (which is basically the same process as above but without the building corpus step).
As you might guess, I'm not strictly following the basic bag of words (BOW) idea since I need to relieve memory consumption so it raises two problems when trying to work with AI algorithms like DecisionTreeClassifier from sci-kit.
Problems
Some of the problems I've observed till the moment are:

Vectors generated from those texts needs to have the same dimension Does padding them with zeroes make any sense?
Vectors for prediction needs also to have the same dimension as those from the training
At prediction phase, those words that hasn't been added to the corpus are ignored
Also, the vectorization doesn't make much sense since they are like [0, 1, 2, 3, 4, 1, 2, 3, 5, 1, 2, 3] and this is different to [1, 0, 2, 3, 4, 1, 2, 3, 5, 1, 2, 3] even though they both contain the same information

Let me first clarify the general principle of classification with text data. Note that I'm assuming that you're using a "traditional" method (like decision trees), as opposed to Deep Learning (DL) method.
As you correctly understand, each individual text document (instance) has to be represented as a vector of features, each feature representing a word. But there is a crucial constraint: every feature/word must be at the same position in the vector for all the documents. This is because that's how the learning algorithm can find patterns across instances. For example the decision tree algorithm might create a condition corresponding to "does the document contains the word 'cat'?", and the only way for the model to correctly detect if this condition is satisfied is if the word 'cat' is consistently represented at index $i$ in the vector for every instance.
For the record this is very similar to one-hot-encoding: the variable "word" has many possible values, each of them must be represented as a different feature.
This means that you cannot use a different index representation for every instance, as you currently do.

Vectors generated from those texts needs to have the same dimension Does padding them with zeroes make any sense?

As you probably understood now, no it doesn't.

Vectors for prediction needs also to have the same dimension as those from the training

Yes, they must not only have the same dimension but also have the same exact features/words in the same order.

At prediction phase, those words that hasn't been added to the corpus are ignored

Absolutely, any out of vocabulary word (word which doesn't appear in the training data) has to be ignored. It would be unusable anyway since the model has no idea which class it is related to.

Also, the vectorization doesn't make much sense since they are like [0, 1, 2, 3, 4, 1, 2, 3, 5, 1, 2, 3] and this is different to [1, 0, 2, 3, 4, 1, 2, 3, 5, 1, 2, 3] even though they both contain the same information

Indeed, you had the right intuition that there was a problem there, it's the same issue as above.
Now of course you go back to solving the problem of fitting these very long vectors in memory. So in theory the vector length is the full vocabulary size, but in practice there are several good reasons not to keep all the words, more precisely to remove the least frequent words:

The least frequent words are difficult to use by the model. A word which appears only once (btw it's called a hapax legomenon, in case you want to impress people with fancy terms ;) ) doesn't help at all, because it might appear by chance with a particular class. Worse, it can cause overfitting: if the model creates a rule that classifies any document containing this word as class C (because in the training 100% of the documents with this word are class C, even though there's only one) and it turns out that the word has nothing specific to class C, the model will make errors. Statistically it's very risky to draw conclusions from a small sample, so the least frequent words are often "bad features".

You're going to like this one: texts in natural language follow a Zipf distribution. This means that in any text there's a small number of distinct words which appear frequently and a high number of distinct words which appear rarely. As a result removing the least frequent words reduces the size of the vocabulary very quickly (because there are many rare words) but it doesn't remove a large proportion of the text (because the most frequent occurrences are frequent words). For example removing the words which appear only once might reduce the vocabulary size by half, while reducing the text size by only 3%.

So practically what you need to do is this:

Calculate the word frequency for every distinct word across all the documents in the training data (only in the training data). Note that you need to store only one dict in memory so it's doable. Sort it by frequency and store it somewhere in a file.
Decide a minimum frequency $N$ in order to obtain your reduced vocabulary by removing all the words which have frequency lower than $N$.
Represent every document as a vector using only this predefined vocabulary (and fixed indexes, of course). Now you can train a model and evaluate it on a test set.

Note that you could try different values of $N$ (2,3,4,...) and observe which one gives the best performance (it's not necessarily the lowest one, for the reasons mentioned above). If you do that you should normally use a validation set distinct from the final test set, because evaluating several times on the test set is like "cheating" (this is called data leakage).


Is is possible to train on Hypergraphs in Keras? I know that vanilla Keras doesn't support operations with graphs.
For example, the Spektral library, based on Keras API, provides some layers to work with simple graphs. However, it doesn't support graphs with multi-node connections (i.e. hypergraphs).
Is there any way to train a deep learning model on the hypergraph input data?

As far as I'm aware, deep learning on hypergraphs is still a relatively new area, so I don't think there's any ready-made solution for hypergraphs. I did find this repo, which implements some models in keras to accompany a recent paper on hypergraph learning, but it is hardly a library.
You may also check out this paper, which cites a pair of techniques for converting a hypergraph to a graph:

Notably, previous methods [for hypergraph learning] typically decompose the hyperedge into pair-wise relationships where the decomposition methods can be divided into two categories: explicit and implicit. For instance, given a hyperedge (v_1,v_2,v_3), the explicit approach would decompose it directly into three edges, $(v_1,v_2), (v_2,v_3), (v_1,v_3)$, while the implicit approach would add a hidden node $e$ representing the hyperedge before decomposition, i.e., $(v_1,e),(v_2,e),(v_3,e)$.

If you're okay with doing this, then there are a few options besides Spektral:
Deep Graph Library provides a Tensorflow.keras compatible API, although their documentation seems to favor PyTorch.
There is also Graph Nets, which is built on top of Tensorflow.keras, although it's quite bare-bones at the moment.


What does big O mean in KNN optimal weights? Wiki gives this definition of KNN

In pattern recognition, the k-nearest neighbors algorithm (k-NN) is a
non-parametric method used for classification and regression. In both
cases, the input consists of the k closest training examples in the
feature space. The output depends on whether k-NN is used for
classification or regression:

In k-NN classification, the output is a class membership. An object    is classified by a plurality vote of its neighbors, with the object
being assigned to the class most common among its k nearest neighbors
(k is a positive integer, typically small). If k = 1, then the object
is simply assigned to the class of that single nearest neighbor.
In k-NN regression, the output is the property value for the object.    This value is the average of the values of k nearest
neighbors.

k-NN is a type of instance-based learning, or lazy learning, where the
function is only approximated locally and all computation is deferred
until classification.
Both for classification and regression, a useful technique can be to
assign weights to the contributions of the neighbors, so that the
nearer neighbors contribute more to the average than the more distant
ones. For example, a common weighting scheme consists in giving each
neighbor a weight of 1/d, where d is the distance to the neighbor.

and this explanation about "The weighted nearest neighbour classifier"

The k-nearest neighbour classifier can be viewed as assigning the k
nearest neighbours a weight 1/k and all others 0 weight. This can be
generalised to weighted nearest neighbour classifiers. That is, where
the ith nearest neighbour is assigned a weight ${\displaystyle
> w_{ni}}$, with ${\displaystyle \sum _{i=1}^{n}w_{ni}=1}$. An analogous
result on the strong consistency of weighted nearest neighbour
classifiers also holds.
Let $C_{n}^{wnn}$ denote the weighted nearest classifier with weights $\{w_{{ni}}\}_{{i=1}}^{n}$.
Subject to regularity conditions on the class distributions the excess risk has the following asymptotic expansion
${\mathcal  {R}}_{{\mathcal  {R}}}(C_{{n}}^{{wnn}})-{\mathcal  {R}}_{{{\mathcal  {R}}}}(C^{{Bayes}})=\left(B_{1}s_{n}^{2}+B_{2}t_{n}^{2}\right)\{1+o(1)\},$

and this formula

With optimal weights the dominant term in the asymptotic expansion of the excess risk is ${\mathcal  {O}}(n^{{-{\frac  4{d+4}}}})$

Does $\mathcal  {O}$ here mean the Big O notation or something else?

From Optimal weighted nearest neighbor classifiers Richard J. Samworth
derived an asymptotic expansion for the excess risk (regret) of a weighted nearest-neighbor classifier. This allows us to find the asymptotically optimal vector of nonnegative weights, which has a rather simple form. He showed that the ratio of the regret of this classifier to that of an unweighted k-nearest neighbor classifier depends asymptotically only on the dimension d of the feature vectors, and not on the underlying populations.
From the paper in (Page 11-12)

In a complementary approach to that taken in most of this paper, Audibert and Tsybakov (2007) study the minimax properties of plug-in classifiers.
They show in particular that a certain classifier obtained by modifying a
the local polynomial estimator of the regression function $η$ attains the minimax
rate over a set of distributions $P$ of random vectors $(X, Y )$ on $\mathbb{R}^d * \{ 1,2 \}$ for which the regression function belongs to a H¨older class, $P$ satisfies a margin condition and the marginal distribution of $X$ satisfies a so-called strong density assumption. This rate is $O(n^{−4/(d+4)})$ when the H¨older smoothness parameter is 2, and the margin power parameter is 1.

From the properties of Hölder condition

Any α–Hölder continuous function f on a metric space $X$ admits a Lipschitz approximation by means of a sequence of functions $(f_k)$ such that $f_k$ is k-Lipschitz and
$ {\displaystyle \|f-f_{k}\|_{\infty ,X}=O\left(k^{-{\frac {\alpha }{1-\alpha }}}\right)}$.

Here we can state that the Big $"O"$ is just an $"O"$ in Holder condition with Lipschitz approximation and not a Big-O-notation


Viterbi-like algorithm suggesting top-N probable state sequences implementation Traditional Viterbi algorithm (say, for hidden Markov models) provides the most probable hidden state sequence given a sequence of observations.

There probably is an algorithm for decoding top-N probable hidden states sequences (k-shortest paths or the like).

But is there a good implementation anywhere?

Thanks.

UPD (copy-pasted from comments): Viterbi algorithm decodes the MOST probable sequence given observations + model. That is, argmax_x p(state_1,...,state_n|obs_1,...,obs_n). What I am asking for is an implementation of how to get the 'next argmax-es' with possibly slightly smaller probabilities GIVEN the observed sequence.

Apparently, I misunderstood your question.

There are several methods for finding the k-best paths with extending versions of the Viterbi algorithm.

My first advice would be to look at this question on SO that is similar to yours and has a good illustrated answer.

Then, I would refer you to two articles/thesis that are publicly available and from where one can extend his/her research. (Disclaimer: these references may not be the "best" one but I've chosen them because they are publicly available and provide a good number of reference to deepen research on the topic)

The k-best paths in Hidden Markov Models. Algorithms and Applications to Transmembrane Protein Topology Recognition. Thesis by Golod, available here. (a thesis like this gives countless references on the topic)
Decoding HMMs using the k-best paths: algorithms and applications by Brown and Dolog, available here (a partial short version of what you will find in the aforementioned thesis)

My previous answer:

For anyone coming across this question looking for a way of computing some of the typical state sequences of an HMM (as I thought first this question was about), just know that such a concept of most probable sequence without specifying data is not really something used in any theory about HMMs, as far as I know. However, one can follow these steps:

As a first try, I would implement something like this:

Get the state at time $t=0$

Draw an initial state $s_0$ from the initial state probability mass function (pmf)

Get the state at time $t+1$

Draw the new state $s_1$ from the pmf defined by the $s_0$-th row of the transition matrix

Repeat this step as many times as needed to draw states up to $s_N$

Then you can repeat the entire procedure $X$ times in order to get as many sample paths as you wish.

This is very fast and easy to implement and it will give you what you want. Many scientific libraries in many languages have a built-in function for drawing a random sample from a pmf.


(CNN+)RNN-HMM hybrid for learning phonemes from a spectogram I am currently working on a speech recognition task, on applying deep learning onto the standard acoustic model (gmm-hmm).

I've currently generated a spectrogram of my utterances, and using simple pattern recognition managed to receive a 40%WER on a yes/no dataset.. Not great but a start.  The CNN is being fed a context window of 40 frames, in which the center frame is being detected for, my question is whether the use of an RNN could benefit here? so that the RNN handles the context, and the CNN does "image analysis" on one frame spectrogram.

and if so will it implementation wise cause some problems, When I was doing it with a CNN, was the context dependency resolved by doing pattern recognition on a bigger part of the spectrogram, depending of the context window size, but introducing RNN, CNN only has to do analysis on one frame (which I see can become to even get a proper results from) at the time, and can the one framed information be piped to the RNN until a certain context size has reached? and if so how?

Have you seen this...

http://ieeexplore.ieee.org/document/7953168/

  ... We propose to use a recently developed deep learning model, recurrent convolutional neural network (RCNN), for speech processing, which inherits some merits of recurrent neural network (RNN) and convolutional neural network (CNN). The core module can be viewed as a convolutional layer embedded with an RNN, which enables the model to capture both temporal and frequency dependance in the spectrogram of the speech in an efficient way.

Sound pretty accurately to what you are searching for, or trying to implement.


Neural Network Classification Probelm How can 2-layer networks be used to classify more than two categories?
Can it be done just by adding more units/nodes into the existing layers?

I think the 2-layer neural network can be used to classify two or more class problem. It does not depend on the number of nodes of each layer. The problem you want to solve (i.e., the dataset you work on) decides how many classes you have.
For more, see here, here, here or here.


Fast way of computing covariance matrix of nonstationary kernel in Python Suppose I have symmetric positive definite covariance function $k:\mathbb{R}\times\mathbb{R}\rightarrow \mathbb{R}$ that is non-stationary (i.e. $k(x,y) \neq g(|x-y|)$ for any function $g$). Is there a fast way in Python given design points $(x_1,\ldots,x_n$) to calculate its covariance matrix $(k(x_i,x_j))_{i,j}$?

If the covariance function is stationary then we can compute the whole matrix at once using numpy's matrix operations and avoid slow Python loops - e.g. in this.

Currently my implementation is:

dim = len(X)
kern_mat = np.zeros((dim,dim))
   for i in range(dim):
      for j in range(i+1):
         kern_mat[i,j] = kern(X[i],X[j])
         kern_mat[j,i] = kern_mat[i,j]

Any help with speedups or otherwise is appreciated!

I still would apply numpy's covaranice function using numpy.apply_along_axis

import numpy as np

x = np.array([[0, 2], [1, 1], [2, 0]]).T
np.apply_along_axis(func1d=np.cov, arr=x, axis=0)


What is a channel in a CNN? I was reading an article about convolutional neural networks, and I found something that I don' understand, which is:

  The filter must have the same number of channels as the input image so that the element-wise multiplication can take place.

now, what I don't understand is : what is a chennel in a convolutional neural network? I have tried looking for the answer, but can't understand what is it yet.

Can someone explain it to me?

Thanks in advance.

Let's assume that we are talking about 2D convolutions applied on images.

In a grayscale image, the data is a matrix of dimensions $w \times h$, where $w$ is the width of the image and $h$ is its height. In a color image, we normally have 3 channels: red, green and blue; this way, a color image can be represented as a matrix of dimensions $w \times h \times c$, where $c$ is the number of channels, that is, 3.

A convolution layer receives the image ($w \times h \times c$) as input, and generates as output an activation map of dimensions $w' \times h' \times c'$. The number of input channels in the convolution is $c$, while the number of output channels is $c'$. The filter for such a convolution is a tensor of dimensions $f \times f \times c \times c'$, where $f$ is the filter size (normally 3 or 5).

This way, the number of channels is the depth of the matrices involved in the convolutions. Also, a convolution operation defines the variation in such depth by specifying input and output channels.

These explanations are directly extrapolable to 1D signals or 3D signals, but the analogy with image channels made it more appropriate to use 2D signals in the example.


Python: How to plot time interval from a Dataframe in Pandas I have the a dataframe(df) which has the data of a Job being executed at different time intervals. It includes the following details about the execution of a job:

Job Start Time (START)
Job End Time (END)
Time Interval (interval) i.e., END - START.

A small part of dataframe is shown below.
Dataframe(df):
  END    |  START   |  interval
1423.0   |  1357.0  |    66.0
33277.0  |  33325.0 |   -48.0
42284.0  |  42250.0 |    34.0
53466.0  |  53218.0 |   248.0
62158.0  |  62073.0 |    85.0

I want to plot a graph with the x-axis as the timestamp and the y-axis with the interval. I tried to do it with the START time but it is not giving the correct result. How can we do this?
Code
fig_dims = (12, 10)
fig, ax = plt.subplots(figsize=fig_dims)

sns.lineplot(x = 'START', y = 'interval', data = df, ax = ax)

Output

Required Output
x-axis - Timestamp
y-axis - Interval

Some suggested alternative plotting methods to visualise this data:

Histogram of the y-axis. Check the distribution of time intervals
df.plot.hist(by='interval', bins=10) #test varying the bin size

Plot smaller subsets of the data if the order is important e.g.
df[:100].plot() furthermore, if there is periodicity in the data, e.g. daily, hourly etc. you could plot each hour on top of each other (in a different colour) to compare the differences between periods.


What quantile is used for the initial DummyRegressor for Gradient Boosting Regressor in scikit-learn? According to the documentation of Scikit-Learn Gradient Boosting Regressor:

init: estimator or ‘zero’, default=None: An estimator object that is used to compute the initial predictions. init has to provide fit and predict. If ‘zero’, the initial raw predictions are set to zero. By default a DummyEstimator is used, predicting either the average target value (for loss=’ls’), or a quantile for the other losses.

So what quantile is used for the DummyRegressor if the loss function is 'huber'? Is it the 50-quantile, ie. median?
I need this information because I am reconstructing the predictor for the Gradient Boosting Regressor for use in another software environment.

Yes, a GBM with Huber loss initializes with the median.  The relevant bit of code is the method init_estimator of the loss class, in the file _gb_losses.py.  For HuberLossFunction:
def init_estimator(self):
    return DummyRegressor(strategy='quantile', quantile=.5)

(source)


Which term is correct Datafication or Datification? I have recently started reading Introduction to Data Science: A python approach to Concepts, Techniques, and Applications  and taking notes on Data Science.
Chapter 1 repeatedly uses the term Datification(the process of rendering into data aspects of the world that have never been quantified before). I could not find the word in the dictionary and the web shows some results but with different spellings Datafication(a technological trend turning many aspects of our life into data).
I am wondering which one is correct?

It's a neologism, there's no standard spelling or even meaning.
For the record there are related terms which are more standard: "data representation" (usually low-level) or "knowledge representation".


Deciding input variables for an ANN I am trying to create an ANN with as many as 50 independent variables. Based on my intuition and business logic, I have tried to create many combination of input variables to feed the ANN, retrain it and try to predict. But the results are not as expected and although the model make very good test set predictions it fails when new data is fed. Can someone plz guide where I am doing wrong? My immediate thought is I need to select the input variables based in a better way. Statistically is there a way to decide that?

What information can I provide about the model or results which would help to solve this?

I feel that is something more fundamentally wrong if your test scores are good but your predictions on new data are bad. I think it sounds like your new prediction data is not coming from the same distribution as your train/test data. That means it is different in some way which will cause you model to perform more poorly on the new data.


Two questions about word2vec and gensim I've written the code below to try word2vec implementation of gensim. I've two questions:

Even though I've removed stop words, the word "the" is listed as one of the most similar words of "friend".
The most similar words of "friend" is not satisfying (at least according to my subjective evaluation). Should I try a larger text (austen-emma.txt file contains 192427 words) or the problem is something else?

Thanks.
import nltk
from nltk.tokenize import sent_tokenize
from nltk.corpus import gutenberg
import gensim
from gensim.models import Word2Vec
from gensim.parsing.preprocessing import remove_stopwords
from nltk.tokenize import RegexpTokenizer

text = gutenberg.raw('austen-emma.txt').
text = remove_stopwords(text).
tokenizer = RegexpTokenizer(r'\w+').
data = [].

for i in sent_tokenize(text):
    temp = [].
    for j in tokenizer.tokenize(i):
       temp.append(j.lower()).
    data.append(temp).

model = gensim.models.Word2Vec(data, min_count = 1,
                          size = 32, window = 2)

model.wv.most_similar(positive='friend', topn=10)

[('mind', 0.9998476505279541),
 ('present', 0.9998302459716797),
 ('till', 0.9998292326927185),
 ('herself', 0.9998183250427246),
 ('highbury', 0.999806821346283),
 ('the', 0.9998062252998352),
 ('place', 0.9998047351837158),
 ('house', 0.999799907207489),
 ('her', 0.9997915029525757),
 ('me', 0.9997879266738892)]

Run the lower casing operation before running stop-word removal since it is case sensitive.
Your window size, currently set to 2 is probably too low. Run it with default setting of 5. It will allow more natural similar pairs to be discovered (words frequently found in similar context but with distance more than 2). It would be also helpful to increase the minimum count threshold to weed out low frequency words which can give rise to spurious correlation.

In general, word2vec needs some manual tuning before the results start subjectively making sense (unless your dataset is very large).


What are the state-of-the-art models for identifying objects in photos? From my observations and little experience it appears that most of the ML project are about classifying stuff. Is there cancer signs on the photo? Does the picture show car, whale or banana? Etc.

I need to implement a model for face identification. Not detection/recognition, but identification: having two different photos of the same person, my model should determine if in the pictures is depicted the same person.

I want to achieve that using Tensorflow with convolutional nets. I've read this paper: http://ydwen.github.io/papers/WenECCV16.pdf and center loss looks promising. What do you think about that? Are there any new ideas/papers/implementations regarding that problem that are worth attention?

I asked this question also on MachineLearning reddit (https://www.reddit.com/r/MachineLearning/comments/8cysrx/d_what_are_the_stateoftheart_models_for/) and got an useful link with FaceNet implementation, trying here also :)

EDIT : Deep Face Recognition: A Survey New on arxiv 4/18/2018 looks like best survey of methods over face related tasks :).

Beyond Facenet, there are a few approaches which may be good to look at how many faces do you intend to have your system know about - aka netowrks that have a network to directly output which face it is (~ <10K), vs a feature map for clustering (~ 10k to 100K), or comparing any 2 faces (~ >100K) - below are examples of approaches to each...

This paper just came out:
Exploring Disentangled Feature Representation Beyond Face Identification- 2018 - Reported Accuracy 99.816 . Use an encoder-decoder like scheme to compute features of a face. Then given all the faces you computed this on, do clustering to find which ones are the same face (TSNE - distance in feature space). This paper is cool since it also gives features of each face like 'smiling', and these are used to augment the search. Along similar line is this

Before that (other than Facenet) - DeepFace - Accuracy 97.35. Its the facebook lib. If its state-of-the-art enough for them, its state of the art enough for me. Approach is given two images, put them into siamese network, for first detect, then 3d model the face, then project to 2d featuremap, which then combine to label saying whether they are same person or not.

Robust Face Recognition via Multimodal Deep
Face Representation - 2016, Reported accuracy 98.43. This is interesting because they trained on a relatively small dataset (CASIA WebFace). However this has the last layer being the number of identities in the dataset - so this could be a llimiting factor if you want to recognize millions of identities like facebook does. Otherwise, this looks easiest to implement/mess with.

Patch-based Face Recognition using a Hierarchical
Multi-label Matcher - not sure what is going on here, but looks interesting.

I would think another limiting factor for you is how many examples do you have per face - eg do you have 4K identities and 4mil images like the facebok dataset, or 10k identities and .5mil images (CASIA WebFace), or LFW with like ~5K identities and ~15k images.


What is the difference between step size and learning rate in machine learning? I am using TensorFlow to implement some basic ML code in Python. I was wondering if anyone could give me a short explanation of the meaning of and difference between step size and learning rate in the following functions.

I used tf.train.GradientDescentOptimizer() to set the parameter learning rate and linear_regressor.train() to set the number of steps. I've been looking through the documentation on tensorflow.org for these functions but I still do not have a complete grasp of the meaning of these parameters.

Thank you and let me know if there is any more info I can provide.

(I posted this on Stack Overflow before I knew there was a Data Science forum board too, sorry)

Both learning rate $\eta$ and step size $\Delta w$ are linked to gradient descent. In the most simple case they are linked by :

$$ \Delta w = w(t+1)-w(t) = -\eta \frac{\partial E(w)}{\partial w}$$

Where $t$ is the epoch and $E$ the error function.

In that simple case, they only differ by $ - \frac{\partial E(w)}{\partial w} $, which sometimes lead to use one term instead of the other. However in a more general case (learning rate depending on weights, learning rate depending on epoch, added momentum, or minibatch learning) the distinction may have a greater importance.

Edit : I answered the title of your question, in the body you seems to refer to the number of step, which is the number of iteration or number of epoch of the aforementionned formula.


How do companies like Amazon track what products are most frequently bought together? What methods could someone use to find out what products are most frequently grouped with each other per order? Are there applications that can make achieving this goal easier?

This is called Association Learning. Quoting Wikpedia:

  Association rule learning is a method for discovering interesting
  relations between variables in large databases. It is intended to
  identify strong rules discovered in databases using some measures of
  interestingness

So, your question asks about how companies like Amazon and Walmart know what products are frequently bought together. So, that is done by the Apriori algorithm, which falls under the class of Association Learning algorithms.

Again, quoting Wikipedia:

  Apriori is an algorithm for frequent item set mining and association
  rule learning over transactional databases. It proceeds by identifying
  the frequent individual items in the database and extending them to
  larger and larger item sets as long as those item sets appear
  sufficiently often in the database.

This algorithm generates rules which show what items are bought along with which items, depending on the support and confidence thresholds. (As you didn't ask about the particulars, I would skip the details. Pl check wikipedia for the finer details and the algorithm itself).

The arules package of R implements the apriori algorithm. Unfortunately, there is no fast implementation of the same in Python.


Scalable open source machine learning library written in python I believe sci kit learn is written in python,however that not scalable.Spark mlib or ml is scalabale but written in scala.I am looking for an ongoing effort where a machine learning library is being built in python (available in github or so) so that I can contribute to that.Is anyone aware of such effort.

Is there a specific reason beside the fact that you would like to contribute? I am asking because there is always pyspark that you can use, the Spark python API that exposes the Spark programming model to Python.

For deep learning specifically, there are a lot of frameworks built on top of Theano -which is a python library for mathematical expressions involving multi-dimensional arrays-, like Lasagne, so they are able to use GPU for intense training. Getting an EC2 instance with GPU on AWS is always an option.


How to prevent a neural network from outputing too clustered probabilities? I have a CNN outputting probabilities using a logistic output. The performances are good on the test set. Yet the probabilities it outputs are very clustered, they are either 0 or 1!

I would like some more granularity in the probabilities outputed by the network. Some features are more obvious than other and the network should be able to learn this. Any idea on how I could solve this problem?

That's usually the most ideal case that you get in classification, but if you really want to, you can add a penalty on the predictions to force them to be away from zero or one, e.g. loss'(x) = loss(x) - param * prob_hat(x) * (1 - prob_hat(x)), or some different penalty.


Choosing the periodicity in a SARIMA model Given the order (P,D,Q,s) of a SARIMA model, s is an integer representing the number of periods in a season. Intuitively, I suppose it would be 12 for monthly data and 4 for quarterly data.

But if I have hourly data (for a whole year) and I'm using only a small number of days as training data (say 10 or 14), what is a proper choice for s?

The choice of periodicity in a ARIMA model is depending on the seasonality in your time-series data. You should plot your time-series and see if you have a "curve"-pattern that occurs regulary.

Based on the "distance" (hours in your data set) you can make a guess what the right choice of periodicity is.


Training LSTM with different sequence lengths in Keras functional api I am trying to train an LSTM model using Keras functional API. My training data is of shape:

>>> data.shape()
(100000,variable_sequence_lengths,295)

where 100000 corresponds to the number of instances (the whole number of sequences) and 295 denotes the number of features in each element of a given sequence.

I am getting errors regarding the shape of the input data. How to define the shape of the data in the input layer and the subsequent one (LSTM) considering variable sequence lengths?

from keras.layers import Input, Dense, concatenate, LSTM
from keras.models import Model
import numpy as np

inputs = Input(shape=(x,y,z))
x=LSTM(128, return_sequences=True, input_shape=(a,b,c))(inputs)
....

What values should x, y, z and a, b, c take?

Model specification is per sample, number of samples should not be included in the dimensions, i.e. (a, b, c) or (sample_size, timesteps, dimension) should be changed to (timesteps, dimension). Also, for variable-length sequences, timesteps should be None. That is, (timesteps, dimension) should be (None, 295). The same goes for (x, y, z).

Please check out this post for feeding variable-length multi-dimensional sequences to Keras LSTM.


sklearn KNN fit throws out error : value too large for dtype('float64') I have cleaned the data from nan values and infinite values, the only feature which has a large float is the column 8  (it's a sum)
I have no Idea how to fix this last error, I tried all previous solutions which are related to my question but nothing worked.
This is the result of : df.isnull().sum():

I am following this tutorial (because I'm just starting to learn about machine learning supervised algorithm KNN):
Tutorial Link
Please help me out!

I'm almost certain that you have missing values, check that and fill them beforehand
Assuming you data frame is called df
df.info() will give you that info
Additionally df.describe() is a good method to validate the maxmimum value is not np.inf


variables selection in regression models I develop price prediction data model using multiple linear regression, ridge, lasso and elastic net regression, initially I had 215 variables. after creating models I ran a python code to check how many variables have used in final models, this is python code which i use for the detect number of variables in ridge regression,

print("Ridge Regression Selected " + str(sum(coef_ridge != 0)) + " Variables and Neglected " +  str(sum(coef_ridge == 0)) + " Variables")

This is a out put which I got

Ridge Regression Selected 209 Variables and Neglected 6 Variables

above code ran for all other modeling methods separably. then Multiple regression   212 variables selected, Ridge 209 variables selected, Lasso 68 variables selected and Elastic net regression 77 variables selected.

my question is according to my knowledge lasso and elastic net regression already use variables selection method because of that, number of selected variables reduced up to 68 and 77 but how multiple linear regression neglect 3 variables and ridge regression reduce 6 variables? that mean multiple and ridge regression also use variable selection method in their code? please someone explain these scenario

Lasso stands for ´least absolute shrinkage and selection operator´. It has a penalty that is the absolute value and makes a lot of variables converge to cero. There is a ton of blogs that explain really well Lasso on the internet, have a look!

Elastic Net is a combination of Ridge and Lasso. So it will also reduce the variables a lot.

Ridge is a quadratic penalty so the convergence is mellower. There are also some posts about why this happens Why will ridge regression not shrink some coefficients to zero like lasso?

In Multiple Linear regression, there is no penalty, so coefficients have not a special reason to be zero. They will only be cero when there is no relation with the target variable. Imagine that you add a feature that is just random noise. You will end up with a cero coefficient for your linear model.

This should give you an intuition of why Lasso shrinks most of the features, Enet a little bit less, Ridge not so many and Linear Regression only some or none.


How does recommendation by matrix factorization deal with new movies / users for which there are ratings? Assume you have the ratings of $n$ users for $m$ movies in a matrix $R \in \mathbb{R}^{n \times m}$. You compute a representation

$$R = U \times \Sigma \times V$$

by initializing $u_i, v_j \forall i \in 1, \dots, n \forall j \in 1, \dots, m$ randomly and optimizing the following expression through gradient descent:

$$\min_{u_i, v_i} \sum_{p_{ij}} \left ( p_{ij} - u_i \cdot v_j \right )^2 \text{ with } u_i \in \mathbb{R}^{1 \times r}, v_j \in \mathbb{R}^{r \times 1}$$

This is how I understand how Simon Funk did it.

But how would you deal with a new user? How would you tell what that user likes?

(Or similarly, with a new movie?)

tl;dr:  It can't!

Why not?

This is one of the main problems of collaborative-filtering recommender systems (i.e. ones that rely solely on the user-item interaction matrix to generate recommendations).

This is referred to as the cold start problem:

  As collaborative filtering methods recommend items based on users' past preferences, new users will need to rate sufficient number of items to enable the system to capture their preferences accurately and thus provides reliable recommendations.
  source

The same is true for new items.

How can we combat this?

The easiest way to combat this issue is to initially identify similar users through demographics and recommending popular items to the user group that best represents him.

Another very popular approach is making a hybrid recommender system, i.e. employs both collaborative-filtering (CF), content-based (CB) and/or even dummy (D) techniques. For instance it could initially recommend the most popular items (D) for new users, then start recommending similar items to the ones he initially liked and finally when the user has rated enough items start looking for users with similar tastes.

In practice most real-world recommender systems are hybrid.

  Netflix is a good example of the use of hybrid recommender systems. The website makes recommendations by comparing the watching and searching habits of similar users (i.e., collaborative filtering) as well as by offering movies that share characteristics with films that a user has rated highly (content-based filtering).
  source


Using deep learning on graphs I have directed graphs with each vertex has some label in "class01_01.gml" format. There are multiple such classes. I need to do multi class classification on these files using deep learning.

Note that each graph in a particular class has different set of vertices.

Is anyone familiar how to apply deep learning in this context?

PS: I'm not that familiar in applying deep learning on graphs.

One of the most popular ways of doing classification on graphs is through graph convolutional networks. By running a convolution over the nodes of a graph, the neural net is able to learn the local neighborhoods of the graph. The seminal example is probably the paper Semi-Supervised Classification with Graph Convolutional Networks  by Kipf and Welling. https://arxiv.org/abs/1609.02907


How to Minimize mean square error using Python I want to minimise mean square error function to find best alpha value (decay rate) for my model.

Here is the description of my model:

time:              1st_month   2nd_month   3rd_month  4th_month  5th_month
Product_shipped     500            600         453      200        789

 If there is delay in products installed after shipping , we multiply by alpha

 Suppose alpha=-0.01
 We create a lower traingular matrix

 month   p1             p2           p3         p4            p5
  1    495.0249169
  2     490.0993367 588.119204
  3     485.2227668 582.2673201 439.6118267
  4     480.3947196 576.4736635 435.2376159 192.1578878
  5     475.6147123 570.7376547 430.9069293 190.2458849 750.5200159

M(1,1) is calculated as 500*(e^-alpha*month(=1))

M(2,1) is calculated as 500*(e^-alpha*month(=2))

M(2,2) is calculated as 600*(e^-alpha*month(=2))

So forth and so on.

Then Predicted Product shipment is sum across row:

Predicted_Installation
  495.0249169
  1078.218541
  1507.101914
 1684.263887
  2418.025197

We have originall Installation:

Original_Installation
   565
   1200
   1677
   1876
   2500

I want to minimise F(sum(Original_Installation-Predicted_Installation)^2) to find alpha which minimise this. How can we frame this or solve this in Python.

For this kind of problem, I would definitely start with scipy.otpimize methods.

I reproduce here an example on how to use it in your context:

import numpy as np
from scipy.optimize import minimize

ALPHA_TRUE = 0.5 # used only to generate some test data

def model(params, X):
    # here you need to implement your real model
    # for Predicted_Installation
    alpha = params[0]
    y_pred = np.exp(-alpha * X)
    return y_pred

def sum_of_squares(params, X, Y):
    y_pred = model(params, X)
    obj = np.sqrt(((y_pred - Y) ** 2).sum())
    return obj

# generate some test data
X = np.random.random(10) # this is "month" if I understood your problem correctly
Y = model([ALPHA_TRUE], X) # Original_Installation

# perform fit to find optimal parameters
# initial value for alpha (guess)
alpha_0 = 0.1

res = minimize(sum_of_squares, [alpha_0, ], args=(X, Y), tol=1e-3, method="Powell")
print(res)

The result looks like this:

   direc: array([[-1.12550246e-12]])
     fun: 0.0
 message: 'Optimization terminated successfully.'
    nfev: 225
     nit: 9
  status: 0
 success: True
       x: array(0.5)

You have to take a deep look at the documentation to find the best fitting method depending on whether alpha is bounded or not or whether you have constraints on your parameters.

I have also played around recently with the same kind of stuff using tensorflow gradient descent optimization (example: https://stellasia.github.io/blog/2020-02-29-custom-model-fitting-using-tensorflow/)


how to classify text based on more than one column I passages of text to classify by topic. I am using scikit learn, e.g. linear svc, but open to other options. Currently, use only the text of each passage (column labeled "En" below). But I feel that using the title of each passage (column labeled "Ref" below) would help.
    Ref               En                                                Topics
3   Gittin            modifier meaning board referred unique name mo... dinei-haget
11  Even HaEzer       hand katafres explanation hand slanted obvious... dinei-haget
67  Rest on Holiday   similar two baskets untithed fruit front first... laws-of-holidays
118 Beitzah           mishna states one ate food prepared festival e... laws-of-holidays
131 Sabbath           one may mix water salt oil dip one bread put c... rabbinically-forbidden-activities-on-shabbat

One excellent idea, which @Erwan mentioned in the comments, is to simply include the title together with the text of the passage. But I have two issues with that:

How do I do that?
Don't I want to make the title (or "Ref") count for more weight than the other words in the passage?

In order to make the model take into account the two columns and distinguish whether a word is from the title or the article, you can generate the two sets of features independently and then concatenate the two vectors of features. This way the learning algorithm can assign different weights to a particular word depending on whether it comes from the title or the article.
However there is a disadvantage in doing that: depending on how many instances, how many words etc. there are, increasing the number of features might make the task harder for the model (for instance cause overfitting). It's probably worth testing the two options:

concatenate the text from the two columns then generate the features, i.e. no distinction between the columns but simpler job for the classifier
generate the features independently then concatenate the two sets of features (as above).


Extract First names from usernames John10 , michaelscott, James.white , Jr.Jones , James-Anderson ,WhiteWalter10 -- These are some of the different cases of usernames possible(there may be more ). I have about 200K such usernames . I need to extract the first name from all of them. If only first name not possible then atleast only the relevant names (for example - WhiteWalter10 should give only Walter or White and Walter ). I already have a dataset of surnames so I can filter the first name

I think your best bet is a gazetteer approach. If you go to this link, you will find plenty of datasets containing lists of existing first names. This should help you detect a big majority of first names using regular expressions.

Now, there are some preprocessing steps you could take:

remove any digits, punctuation, etc.
if you have upper cases, you could split the username such that "MichaelScott" becomes "Michael" and "Scott". Now, in this case, you end up with two things that could be the first name. You could use the assumption that the first one is the first name and the second one is the surname.

Once you've done this, look at usernames where no first names were detected. Now, it could very well be that the username doesn't contain any first names (i.e. user123213543), or the user is bit cheeky and could use special characters or digits to write their first name (i.e. $am, B0b, etc.). If these situations exist, you could create strategies to deal with them.


What is the structure of a MLP that allows for arbitrary decision boundaries? I'm just doing some reacher for my machine learning class, I found out it must be three layers MLP. But why? Is there a proof/paper or I'm wrong?

I understand that the fact that MLP allows for arbitrary decision boundaries derives from the fact that an MLP is a universal approximator.
The theoretical area addressing the capability of neural networks to be able to approximate any function are the different flavors of the universal approximation theorem, for which there are two directions: the arbitrary width case (which you refer to when you assume that the ability comes from the three layers), and the arbitrary depth case.
The wikipedia page contains the mathematical formulation of both directions, as well as reference to their respective proof for different assumptions.


News topic detection and categorization If I want to get how many and what kind of topics are covered by New York Times each week from a bag of words model(All the news covered by NYT in a week) how should I approach? Using traditional unsupervised LDA didn't help much.

I'm not an expert in this field but you should take a look at the work of Bhargav Srinivasa Desikan, a gensim contributor, who works a lot with topic modelling.

He has a couple of notebooks on his github account which could interest you, especially this one (should be pretty much your use case if I understand your problem correctly).

The aforementioned notebook evaluates Latent Semantic Indexing, the Hierarchical Dirichlet Process as well as the Latent Dirichlet Allocation for identifying topics.


Keras Callback example for saving a model after every epoch? Can someone please post a straightforward example of Keras using a callback to save a model after every epoch? I can find examples of saving weights, but I want to be able to save a completely functioning model after every training epoch.

Setting 'save_weights_only' to False in the Keras callback 'ModelCheckpoint' will save the full model; this example taken from the link above will save a full model every epoch, regardless of performance:

keras.callbacks.ModelCheckpoint(filepath, monitor='val_loss', verbose=0, save_best_only=False, save_weights_only=False, mode='auto', period=1)

Some more examples are found here, including saving only improved models and loading the saved models.


Script to convert non-numeric elements that are numbers to numeric I'm self-teaching myself some scikit-learn and tensorflow right now, and trying to get better at cleaning data since the rest seems reasonably straightforward.

I downloaded some Google Play Store data from here, and upon taking a peak, it appears to be primarily numerical, although the numbers stored are not of a recognized type.

dat.info()
returns

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10841 entries, 0 to 10840
Data columns (total 13 columns):
App               10841 non-null object
Category          10841 non-null object
Rating            9367 non-null float64
Reviews           10841 non-null object
Size              10841 non-null object
Installs          10841 non-null object
Type              10840 non-null object
Price             10841 non-null object
Content Rating    10840 non-null object
Genres            10841 non-null object
Last Updated      10841 non-null object
Current Ver       10833 non-null object
Android Ver       10838 non-null object
dtypes: float64(1), object(12)
memory usage: 1.1+ MB

A lot of these are obviously numerical anyway when checking with head(), but some entries have something like "3.0M" in place of 3000000, and so I can't have any fun with them yet (I'm trying to first make histograms of them to explore). I wrote this quick loop to try and address this by converting them to a string, taking off the "M", then making them floats, but it doesn't seem to be working because the output still isn't numeric.

k=0
for i in dat["Reviews"]:
    #make the entry a string:
    j=str(i)
    #remove the Ms- I'm assuming these mean million
    if j[-1]=='M':
        j=j[:-1] #cut off last letter
        j=float(j)
        j=j*10**6
    dat[k,"Reviews"]=j
    k=k+1

There is likely a far easier way to do this that I'm missing, but the fact not even this works is strange. Does anyone have any ideas on how to deal with an unknown object type in a dataframe like this?

Try avoid looping when working pandas. Either create a function and apply it to your dataframe/series or you can use a lambda like I do below. I basically took your if/then logic with x*10**6 and applied it to a new column

import pandas as pd
df = pd.DataFrame({'Number':'10M 20M 10.5M 30M 100M 1000000 5000000'.split()})

    Number
0      10M
1      20M
2    10.5M
3      30M
4     100M
5  1000000
6  5000000

df['Num_Transformed'] = df['Number'].apply(lambda x: float(x[:-1])*10**6 if str(x).upper()[-1]=='M' else float(x))

    Number  Num_Transformed
0      10M       10000000.0
1      20M       20000000.0
2    10.5M       10500000.0
3      30M       30000000.0
4     100M      100000000.0
5  1000000        1000000.0
6  5000000        5000000.0


Data Augmentation for Regression ANN with low Sample Size There is a Dataset of 65 tuples.
I want to Augment new Data from this set and validate my ANN on the original Data.
Is there a possibility, that my ANN already overfits on the augmentet Data. For example that the augmentation Is simulated by the ANN, and how does one prevent this ?

In theory it's possible, everything depends on your data, the model you're training and how you're going to augment your training data.


What is the Value of X in KNN and Why? I have a dataset of 25 instances these instances are divided into 2 classes Green Circles and Blue Squares

data distributed as this graph

I want to predict X's class based on "Likelihood Weighted KNN with k =3"

In normal KNN this is easy

the nearest 3 points are 2 Blue Squares and 1 Green Circle

which means X will be Blue Square

there are more Blue Squares neighbours than Green Circles (2 vs 1)

But What is needed is to find the Likelihood Weighted KNN with k =3

This is my try

In this case we have to calculate the weight (Likelihood) for each instance

Each Green Circle likelihood is $\frac{1}{5} $ , we have 5 Green Circles

While for Blue Squares it is  $\frac{1}{20} $ , we have 20 Blue Squares

Therefore the weights around X will be $\frac{1}{5} $  Green Circle, and $\frac{2}{20} $ Blue Squares.

which means $\frac{1}{5} > \frac{2}{20} $

Then X is Green Circle

Well, this is wrong :(

Can someone help me find the

Green circles and blue squares are samples of two different classes. How does it matter that "X" belongs to which one of either of these "Green circle" or "Blue Square" class?

Likelihood should be done based upon the number of classes and the K value, instead of dataset samples.

  For me likelihood weight will be:

  Likelihood of being blue square ~ 0.66 i.e. 2/3, whereas likelihood of
  being green circle is 1/3.

Prepared another example:
Where I have 2 Green circle and 16 blue square, whereas K=5

If I follow your approach, likelihood of
Green Square is 1/2, Blue square is 4/16 = 1/4, whereas from the image it's clear that "X" belongs to Blue Square class only.

[Update]
Thanks @Ben Reiniger, for correcting me.

I spent more time and what I understood is, it's not always correct to put equal weights(likelihood) across different classes in dataset. My earlier observation was based upon the assumption that all classes have equal weights and I was wrong, even though data is bit skewed.

Consider an another example:

We have a huge dataset of patient reports, where features are created based upon the several tests done so far. Our task is to identify the possibility if a particular person suffering from cancer or not.

In this case, even if a single feature point towards cancer disease, we cann't neglect it and should predict towards rare class, such that patient can go for further analysis.

In such cases weights are not equivalent across different classes.
According to me, wights should be defined based upon the use case.

A very well explained its calculation in How does Weighted KNN works?


How to make edges appear in a graph when adjacency matrix value is beyond a particular threshold? I've created a graph from an adjacency matrix using the igraph package but I want edges to appear in the graph only when the value in the adjacency matrix is beyond a particular threshold. Using the present method I am getting all the edges which have non zero weights.

set.seed(42)
iris_matrix <- as.matrix(iris[,1:4])
image(iris_matrix)
g<-cor(iris_matrix)
library(igraph)
R<-graph.adjacency(g, mode=c("undirected"), weighted=TRUE, diag=FALSE, add.colnames=NULL, add.rownames=NULL)
plot(R)

Line 7 is what you need:

set.seed(42)
iris_matrix <- as.matrix(iris[,1:4])
image(iris_matrix)
g<-cor(iris_matrix)
library(igraph)
R<-graph.adjacency(g, mode=c("undirected"), weighted=TRUE, diag=FALSE, add.colnames=NULL, add.rownames=NULL)
R[R > thresh] <- 0
plot(R)


Logistic regression threshold value How can i set the threshold value for the target variable. For example if a target variable is chance_of_admit and it has values from 0 to 1, how can I pick a value and so that I can convert it to 0's and 1's to perform logistic regression

So there two ways of doing this, IMHO,

By creating a well balanced target variable by choosing the right threshold. As I suggested in the comments above. In doing so we are simply taking care of values which should be treated as positive which would otherwise become negative if we take lower threshold.
By using the mean threshold and that will generate imbalanced target variable and when you perform the modelling, you can see the ROC and PRC curves and decide the threshold based on that. But keep in mind, it also depends what kind of problem you are solving.


Stochastic gradient descent in logistic regression I am very new to machine learning and in my first project have stumbled across a lot of issues which I really want to get through.

I'm using logistic regression with R's glmnet package and alpha = 0 for ridge regression.

I'm using ridge regression actually since lasso deleted all my variables and gave very low area under curve (0.52) but with ridge there isn't much of a difference (0.61).

My dependent variable/output is probability of click, based on if there is a click or not in historical data.

The independent variables are state, city, device, user age, user gender, IP carrier, keyword, mobile manufacturer, ad template, browser version, browser family, OS version and OS family.

Of these, for prediction I'm using state, device, user age, user gender, IP carrier, browser version, browser family, OS version and OS family; I am not using keyword or template since we want to reject a user request before deep diving in our system and selecting a keyword or template. I am not using city because they are too many or mobile manufacturer because they are too few.

Is that okay or should I be using the rejected variables?

To start, I create a sparse matrix from my variables which are mapped against the column of clicks that have yes or no values.

After training the model, I save the coefficients and intercept. These are used for new incoming requests using the formula for logistic regression:

Where a is intercept, k is the ith coefficient and x is the ith variable value.

Is my approach correct so far?

Simple GLM in R (that is where there is no regularized regression, right?) gave me 0.56 AUC. With regularization I get 0.61 but there is no distinct threshold where we could say that above 0.xx its mostly ones and below it most zeros are covered; actually, the max probability that a click didn't happen is almost always greater than the max probability that a click happened.

So basically what should I do?

I have read how stochastic gradient descent is an effective technique in logit so how do I implement stochastic gradient descent in R? If it's not straightforward, is there a way to implement this system in Python? Is SGD implemented after generating a regularized logistic regression model or is it a different process altogether?

Also there is an algorithm called follow the regularized leader (FTRL) that is used in click-through rate prediction. Is there a sample code and use of FTRL that I could go through?

Stochastic gradient descent is a method of setting the parameters of the regressor; since the objective for logistic regression is convex (has only one maximum), this won't be an issue and SGD is generally only needed to improve convergence speed with masses of training data.

What your numbers suggest to me is that your features are not adequate to separate the classes. Consider adding extra features if you can think any any that are useful. You might also consider interactions and quadratic features in your original feature space.


how to sort a multi-level pandas data-frame by a particular column? I want to sort a multi-index pandas dataframe by a column but don't want the entire dataframe to be sorted at once. But rather want to sort by one of the indices. Here is an example of what I mean:
Below is an example of a multi-index dataframe.

first  second
bar    one       0.361041
       two       0.476720
baz    one       0.565781
       two       0.848519
foo    one       0.405524
       two       0.882497
qux    one       0.488229
       two       0.303862

What I want to do is to get the following:

first  second
bar    one       0.476720
       two       0.361041
baz    one       0.848519
       two       0.565781
foo    one       0.882497
       two       0.405524
qux    one       0.488229
       two       0.303862

These are generated by hand to show what I want. Notice that the second dataframe is not completely sorted. But within each multi-index, it is sorted in descending order. I have a large dataframe. Is there a simpler way of do it (such as a function) instead of grouping the dataframe based on the indices, and then concatenate the individually sorted dataframes.

Thanks for your help.

TO BE EDITTED
There would definitely be more optimal ways for addressing this problem. But this is a simple approach.
import pandas as pd
df = pd.DataFrame({'first': ['bar', 'bar', 'baz', 'baz', 'foo', 'foo', 'qux', 'qux'],
                    'second': ['one', 'two', 'one', 'two', 'one', 'two', 'one', 'two'],
                    'value': [0.361041, 0.476720, 0.565781, 0.848519, 0.405524, 0.882497, 0.488229, 0.303862]})

    first   second  value
    0   bar one 0.361041
    1   bar two 0.476720
    2   baz one 0.565781
    3   baz two 0.848519
    4   foo one 0.405524
    5   foo two 0.882497
    6   qux one 0.488229
    7   qux two 0.303862

import itertools
#storing unique values of the first column.
unique = list(df['first'].unique())
#unique
#The following line, gives the row indices of the dataframe with specific value on the index ('first') column.
indices =[df.index[df['first'] == unique[i]].tolist() for i in range(len(unique))]
#Then this line sorts each pair (it can be more than 2) of the elements from the value column.
sorted_values = []
for [i,j] in indices:
    sorted_values.append(sorted([df['value'].iloc[i],df['value'].iloc[j]]))

#The following line returns the row index of a specific value of 'value' column.
#df['value']
[df['value'] == 0.361041].index.tolist()
#This ravels the sorted_list and gives the goal order of the values in 'value' column
values = []
values = list(itertools.chain(*sorted_values))
#values

#output
[0.361041, 0.47672, 0.565781, 0.848519, 0.405524, 0.882497, 0.303862, 0.488229]

Now that we have the actual order of elements in the value column, we can rebuild the whole dataframe based on this column.
However, I've done ascending ordering instead of your desired descending one.


Time Decay Formula for Conversion Attribution Question:

What is the Time Decay formula that web analytics packages use to distribute credit across the multiple sessions associated with a user before a conversion?

Context:

All web analytics packages like Google Analytics rely on the concept of:

USERS visitors, ie unique cookied browsers. Users have many...
SESSIONS visits, ie sets of pageviews within close time proximity.
CONVERSIONS  a success event such as a sign-up which can occur during a session.

These analytics packages allow you to distribute the credit of a single conversion across all the preceding sessions because each played a part in leading the user to conversion. One such model is Time Decay, which gives a diminishing share of credit to sessions further from the conversion.

Time Decay Visual: (1st session) ▁ ▂ ▃  ▅ ▆ (conversion)

more info on attribution models.

Sample Data:

I am assuming the data required would be:

n - the total number of sessions a user has before a conversion
t - the total duration of time from the user's first session to conversion
tbc - the individual time before conversion for each session

So for three different sample users, data may look like this:

UserID  n   t   Session#    Timestamp   TBC Conversion in session?
User1   4   12  1           2015-10-01  12  No
User1   4   12  2           2015-10-06  7   No
User1   4   12  3           2015-10-11  2   No
User1   4   12  4           2015-10-13  0   YES
User2   3   4   1           2015-10-09  4   No
User2   3   4   2           2015-10-09  4   No
User2   3   4   3           2015-10-13  0   YES
User3   1   0   1           2015-10-14  0   YES

Here's one solution:

using the following data:

USER-LEVEL:

fss first session start (timestamp) - time of first pageview of
first session for that user
css converting session start (timestamp) - time of the first pageview of session where user converted
ns number of sessions (int) - total # of sessions
before conversion for user

SESSION-LEVEL:

ss session start (timestamp) - timestamp of first pageview in session

FORMULA: calculated for each session

FOR FIRST SESSION: Attribution Credit is just linear: (or really whatever you want to make it)
ALL OTHER SESSIONS:

First calculate the SUM of all session's "time before conversion":
Then Calculate the credit for each session:

it isn't perfect because it distributes credit based on the relative position of a session in the User's lifetime.  Ideally you have a preset halflife (like 7 days) so that comparison is more consistent.


Replace Values with corresponding numbers in the same dataframe Please help me to replace NR values with corresponding values as in the dataframe.

I'm not sure if this is most efficient way but it may help you.

df = read_csv('filename.csv')
grouped_df = df.groupby('City')
temp_df = []
for key, item in grouped_df:
    if 'NR' in item['Route'].values.tolist():
        values = item['Route'].values.tolist()
        to_change = [x for x in values if x != 'NR'][0]
        item = item.replace('NR', to_change)
        temp_df.append(item)
    else:
        temp_df.append(item)
final_df = pd.concat(temp_df, axis=0)

Output:

        City Route
4          A     3
5          B     4
6          C     5
7          D     7
8          D     7
9          D     7
10         D     7
2   Kolkatta     2
3   Kolkatta     2
0    Manipur    10
1    Manipur    10


How to convert my tensorflow model to pytorch model? I performed transfer learning using ssd + mobilenet as my base model in tensorflow and freezed a new model. Now I want to convert that model into pytorch. Is there any way how I can achieve it? Any help would be really helpful..

You can build the same model in pytorch. Then extract weights from tensorflow and assign them manually to each layer in pytorch. Depending on the amount of layers it could be time consuming. Building the model depends on the model and I think not everything is possible in pytorch that is possible in tensorflow. Examples how to assign weights in pytorch and extract weights from tensorflow are given below.

Getting weights from tensorflow variables W and b:

weights = sess.run(W)
bias = sess.run(b)

where sess is a tf.Session.

Assigning weights to pytorch:

import torch
from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F

class Kernel_Emb(nn.Module):

    def __init__(self,D_in,H,D_out):
        super(Kernel_Emb, self).__init__()
        self.linear1 = nn.Linear(D_in,H)
        self.linear2 = nn.Linear(H,D_out)
        self.linear1.weight = torch.nn.Parameter(weights)
        self.linear1.bias = torch.nn.Parameter(bias)

If the variable is not defined in tensorflow (source: https://stackoverflow.com/questions/36193553/get-the-value-of-some-weights-in-a-model-trained-by-tensorflow):

  If you do not currently have a pointer to the tf.Variable, you can
  get a list of the trainable variables in the current graph by calling
  [tf.trainable_variables()][4]. This function returns a list of all
  trainable tf.Variable objects in the current graph, and you can
  select the one that you want by matching the v.name property. For
  example:

# Desired variable is called "tower_2/filter:0".
var = [v for v in tf.trainable_variables() if v.name == "tower_2/filter:0"][0]


Comparison of machine learning approaches for a topic in a scientific paper As part of my master's thesis, I have made a prediction of data with approaches of machine learning in a topic where are no papers yet.
The topic is a regression problem for which several machine learning approaches have been tested:
SVR, RandomForestRegressor, GradientBoosting, DecisionTrees, Artificial Neural Networks, LSTMs, K-NearestNeighbors..

It has been shown that useful results can be achieved with all approaches. Which of the methods is the best depends heavily on the parameter configuration.

It would be too much work to test any parameter configuration for any approach. Besides, I think it would be uninteresting to show dozens of plots about the parameter tuning tests, because they also depend heavily on the respective data sets.

So how do you typically compare different types of machine learning approaches in scientific papers that are heavily dependent on the parameters? Just play a bit with grid search and randomized search until you are satisfied with the parameter configuration?

If I understand, you could evaluate your approach based on your efforts in using the right preprocessing, picking the right features, building the appropriate architecture (choosing the model that best suits the problem).

Each step will make the loss decrease until you reach a point, you will have to search over the hyperparameters to squeeze every percent of accuracy, which is heavily a computational problem.


Recommended Language/Framework for Building a New Recommendation Engine Next week I'm going to begin prototyping a recommendation engine for work. I've implemented/completed the Netflix Challenge in Java before (for college) but have no real idea what to use for a production/enterprise level recommendation engine. Taking into consideration everything from a standalone programming language to things like Apache Mahout and Neo4j, does anyone have any advice on how to proceed?

I am developing a recommendation engine for stack overflow (personal project). Check it on http://recommender.im .

It is still a working in progress, but I have a quite functional website working. I am putting there most of the code I used through python notebooks.

Basically I used:
- Frontend: angularJS
- Website backend: flask + scikit-learn
- machine learning and data preparation: python, pandas, scikit-learn

I really like python for data science as the community and libraries are really good.


How to import a tsv file to Matlab I am really new to data science. Please don't mark me down as this website is my only hope of progress.
I have set of data I obtained from NASA website. When I saved it, it saved as "tsv' file. (Tab separated values). I want to open it on Matlab as a Matrix as I have a code to run on that matrix.
Basically I want to import that file to matlab and start running the code on it.
Can someone please help me or guide me in the right direction.

I tried various things such as import data, tdfread but so far nothing has worked for me. I was first trying to export the tsv file to MS Excel and then go from Excel to Matlab. That too I don't know how to do. I will give you the link of my data which I want to import on to Matlab.

The link for my data is the following.

Please take a look at http://vizier.u-strasbg.fr/viz-bin/VizieR?-source=J%2FApJS%2F209%2F31

If you just click on submit at the lower right corner, one will see the data.

a = importdata('J_ApJS_209_31_table3-150618.tsv') [This the command I used].
Error message on matlab is

    ??? Error using ==> importdata at 136
    Unable to open file.
    Error in ==> data at 1

Then I wrote a script.

    tdfread(J_ApJS_209_31_table3-150618,'\t')

Error message I get is
        ??? Undefined function or variable 'J_ApJS_209_31_table3'.

    Error in ==> data at 1

tdfread displays the File Open dialog box for interactive selection of a data file, then reads data from the file. The file should have variable names separated by tabs in the first row, and data values separated by tabs in the remaining rows. tdfread creates variables in the workspace, one for each column of the file. The variable names are taken from the first row of the file. If a column of the file contains only numeric data in the second and following rows, tdfread creates a double variable. Otherwise, tdfread creates a char variable. After all values are imported, tdfread displays information about the imported values using the format of the tdfread command.

tdfread(filename) allows command line specification of the name of a file in the current folder, or the complete path name of any file, using the string filename.

tdfread(filename,delimiter) indicates that the character specified by delimiter separates columns in the file. Accepted values for delimiter are:

' ' or 'space'
'\t' or 'tab'
',' or 'comma'
';' or 'semi'
'|' or 'bar'
The default delimiter is 'tab'.

s = tdfread(filename,...) returns a scalar structure s whose fields each contain a variable.


General equation for getting an idea of the scale of a machine learning project I'm writing an application for a project where we intend to teach a model to predict one aspect of an environment (traffic safety) using a database with 10 images (about 300x300px and, say, 256 colors) for each of either 100 000 or 15 million locations.

I must come to grasp with if both, one or none of these projects are feasible with our hardware constraints. What can I expect? Is there some formula or benchmark that one can refer to? Will one be able to do this on a laptop with a decent GPU, a dedicated ML computer or does it require the level of infrastructure that Google and Amazon use?

Interesting but difficult question! It depends on the efficiency of your algorithm, both in terms of training and scoring/predicting, but to get a first idea I would go by the amount of data that we're talking about.

256 colors are 8 bits per pixel, times 300x300 pixels. Uncompressed, you have 720 kB per image. 10 images per location: 7.2 MB data per location. 100k locations: 720 GB of uncompressed input data, 15M locations: 108 TB. If you compress your data, meaning you store them a JPG or something, I don't know, but I would expect you need about a factor 10 less storage (it depends how easily the images can be compressed and how well JPG compression works).

Given unlimited time (and storage), any amount of data can be crunched on a laptop, although my laptop doesn't have 10 TB storage. But I would expect 720 GB worth of uncompressed data to be impractical on a single computer, unless you have an exceptionally efficient algorithm. That would be an algorithm that might be specifically designed so that it can be trained on all the data on a single computer, not a CNN with a bunch of layers that I expect you have in mind.

Using distributed computing in the cloud or on-premise has its cost too, in terms of infrastructure cost, complexity to debug, etc. But I would expect that with this kind of dataset, it's worth it.

That's something different from "the level of infrastructure that Google uses", which is millions of servers. I would start processing with, for example, 100 cores, and see how it goes. Don't forget to turn them off when you're done!


Why does my minimal CNN example show strongly fluctuating validation loss? I'm fairly new at working with neural networks and think I am making some basic mistake. I am trying to assign simulated images to 5 classes to test, what (if any) networks are helpful for a large data problem we have in our group. I am training the ResNet50 CNN included in Keras on 40000 (256 by 256 pixel greyscale) images. While training loss is improving quickly within the first epoch, validation loss fluctuates wildly in (to me) fairly random manner.

I am trying to use as many high level functions as possible and have ended up with the following code:

from keras.preprocessing.image import ImageDataGenerator
from keras.applications.resnet50 import ResNet50
from keras.models import Model
from keras.layers import GlobalAveragePooling2D, Dense

inputShape = (256, 256,1)
targetSize = (256, 256  )
batchSize = 128

# First, load resnet
base_model = ResNet50(include_top =False,
                weights =None,
                input_shape=inputShape)
x = base_model.output

# modified based on the article https://github.com/priya-dwivedi/
# Deep-Learning/blob/master/resnet_keras/Residual_Networks_yourself.ipynb

x = GlobalAveragePooling2D()(x)
predictions = Dense(5, activation= 'softmax')(x)
model = Model(inputs = base_model.input, outputs = predictions)

# Compile model, might want to change loss and metrics
model.compile(loss='categorical_crossentropy',
        optimizer='adam',
        metrics = ["acc"]
        )

# Define the generators to read the data. Training generator performs data argumentation as well
train_datagen = ImageDataGenerator(
        samplewise_center =True,
        samplewise_std_normalization = True,
        shear_range=0.2,
        zoom_range=0.2,
        horizontal_flip=True)

test_datagen = ImageDataGenerator(
        samplewise_center =True,
        samplewise_std_normalization = True,
        )

train_generator = train_datagen.flow_from_directory(
        'trainingData/FolderBasedTrain256',
        target_size=targetSize,
        color_mode = 'grayscale',
        batch_size=batchSize,
        save_to_dir = 'trainingData/argumentedImages',
        class_mode='categorical')

validation_generator = test_datagen.flow_from_directory(
        'trainingData/FolderBasedValidation256',
        target_size=targetSize,
        batch_size=batchSize,
        color_mode = 'grayscale',
        class_mode='categorical')

# Fit the Modell, saving the history for later plotting
history = model.fit_generator(
        train_generator,
        epochs=20,
        validation_data=validation_generator,
        #steps_per_epoch = 62,
        steps_per_epoch = 31,
        #callbacks=[checkpointer],
        validation_steps=9)

I can always create more images or train longer, but to me, this looks as if something fundamental went wrong somewhere. I would be very gratefull for any and all ideas. Thank you!

EDIT: I was told to stress that validation and training set are both created by exaclty the same simulation routine, so they should be relativly easy to classify

EDIT2: Found the error! My batch size did not fit with the amount of data and the steps by epoch, resulting in the CNN not seeing all the training data. Now everything converges nicely and I can evaluate the choice of modell. Thanks to all contributers

There is nothing fundamentally wrong with your code, but maybe your model is not right for your current toy-problem.

In general, this is typical behavior when training in deep learning. Think about it, your target loss is the training loss, so it is directly affected by the training process and as you said "improving quickly". The validation loss is only affected indirectly, so naturally it will be more volatile in comparison.

When you are training, the model is attempting to estimate the real distribution of the data, however all it got is the distribution of the training dataset to rely on (which is similar but not the same).

Suggestions:

I think that your model is an over-kill for a 256x256 grayscale dataset with just 5 classes (Resnet was designed for ImageNet which contains RGB images from 1000 categories). In current state, the model finds its very easy to memorize the training set and overfit. You should look for models that are meant to be used for MNIST, or at most CIFAR10.

You can insist on this model and attempt to increase regularization, but I'm not sure if it will be enough to prevent the model from overfitting in this case.


Using categorical and continuous variables in Deep Learning I would like to apply a MLP to some business seller data. I found that the data is a mix of both categorical and continuous features. For what I read it is not advisable to feed a neural network with both types of data (reference unknown/unavailable) and I remember that I read that one can use the following model:
Categorical variables-->NN model 1
                                        ----->NN model 3---->Output
Continuous variables--->NN model 2

So in this model we have two neural networks that are fed each one with only categorical or continuous variables, and then the outputs (from both models) are feed to the third model.
Side note: For what I can see in this proposed model I can end up with an endless "loop", because maybe the output from the model 1 is categorical and the model 2 output is continuous. (?)
My question is how can I model data with a mixture of categorical and continuous features using deep learning (i.e. Neural Networks)? I do not want to use a random forest or any form of decision tree.
Thanks in advance.

If I understand well your main issue is to transform some Categorical variables into Continuous, or some Continuous into Categorical.
You have many ways to do that:
Continuous to Categorical :

You can yourself set intervals, transforming your continuous value into a category. For example, for a variable Age, you define the following intervals : [0;10[, [10;20[, ... [90;100[. If someone is 36 yo, he'll be on the interval [30;40[. You'll then only have 10 categories. The complicated part is to find what are the best intervals to lose avoid losing to much information

Categorical to Continuous

You can apply some Sklearn Category Encoder, which, knowing your targer on training set, will transform each category to a value between 0 and 1. This value is calculated using statistics (different for each encoder), and illustrates how often your category ends up affecting your target.