How can machine learning algorithms solve this particular problem? Let's think of a case of an e-commerce website which lists products for sale. Now a person can come on a particular product page and decide to add it to the shopping cart or not. If we look at it as a classification problem then, specifications of the product can be its features(e.g. for a cell phone, camera resolution, screen size, price, warranty period) and for a particular product page view by a customer either he clicked on add to cart button or not can be the success criteria(classification label).

An observation is that for the same set of features, X for a product page view, very few customers click on add to cart button (y = 1). That means for the same X, there will be many Y=0 and very few Y=1.

My peer says that machine learning algorithms like XGBoost will be able to classify this. While I am not convinced of the fact that how will the algorithm be able to predict that for an X, Y will be 1 when in reality itself it has a very slight chance of being 1 as many customers will not add it to cart.

I know it will give me a probability score of y being 1 but that will be too low to make use of.

I know it will give me a probability score of y being 1 but that will be too low to make use of.

Given the description you have of input data, which as you explain it is just the specs of a product (and implicitly that you are classifying for a user who has reached the product page somehow), then that's the best you can do. You simply don't have the data about the motivations and history of the user that would enable you to refine further than the mean probability of purchase given the product stats.

If you can access data about the visitor, features such as what other pages they have visited, previous purchases, whether they have clicked through from a particular advert or search result etc, then you may be able to refine the value further. However, even with that kind of backing data, you would still be quite far from predicting a clear "will buy" vs "won't buy" - instead you may be able to refine a generic p=0.01 to a personalised range p=0.0001 to p=0.05 depending on the user details.

These kind of shifts in probabilities can still be used successfully in advertising and sales pipelines. The difference between a 0.01 and 0.05 probability when deciding what to advertise for example is a factor of 5 efficiency improvement on use of advertising sections of a web site. The trick is finding the right way to make use of the information.

In general, you won't be able to make a reliable prediction that a user will buy a product like a mobile phone on a website. Too many hidden variables. So don't try to design systems that rely on a direct yes/no prediction.


OneHotEncoding target variable? I'm working on a multiclass classifier with 6 classes on the target column and I was thinking about Hot Encoding the classes, thus having 6 target columns. Will this improve efficiency? I am using sklearn.
L.E: improve efficiency compared to just label encode

It would be a bad idea to use one hot encoding because this would make the problem multi-label. This would make the problem much more complex for the model and would very likely lead to lower performance, or it would require much more data in order to reach the same performance as regular classification. This is because there are much more combinations possible in the multi-label problem: every instance can have any number of classes, from zero to all of them. Your training data would not have these cases, but the algorithm would assume that they can happen. So in your case this higher level of complexity is pointless since there can be only one class.
With scikit the standard way to encode the target in a multiclass task is to use the LabelEncoder.


Anomaly detection in time series The use case :

Everyday, we have metrics that are established daily to check that our systems are doing fine. From time to times, bugs occur in the workflow building these metrics, and I have to build an algorithm that will alert us when it seems there is a problem, so that we can check where it is coming from.

What I'm trying :

I don't know much about time series. The data I have displays a weekly seasonality, and the volume isn't constant (it's growing). See image : http://www.blabala.com (I don't have enough rep to post multiple images)
Therefore, from what I read, ARIMA seems to be a relevant algorithm.

I stumbled upon the tsoutlier package in R, and its functions locate.outliers.oloop() and discard.outliers(), so I had the following idea: instead of trying to train something I don't master, let's have the automatic parts of the algorithm train on their own, and use that: if it detects outliers, then throw an alert.

First question: does it seem to be a good or a bad idea?

If it's a good approach :

I'm having trouble getting something that works well. I've explored two ways of doing things :

1) train the whole thing, and run it on itself.

cval <- 3
mo2 <- locate.outliers.oloop(TS_train,
                             arima_train,
                             maxit.iloop = 100,
                             maxit.oloop = 100,
                             cval = cval)
d2<-discard.outliers(mo2,TS_train, method = "bottom-up",
                     tsmethod.call = arima_train$call,
                     cval = cval)

OTL <-  TS_train[sort(d2$outliers$ind)]

plot(1:length(TS_train), TS_train, type = 'p')
lines(1:length(TS_train), TS_train, col = "grey")
points(sort(d2$outliers$ind), OTL, col = "red")

This hasn't given very good results. See image 1.

2) have a sliding window: train the algorithm with day 1 to 60 as training data, evaluate day 61, slide one day, train the algorithm with day 2 to 61 as training data, evaluate day 62, slide one day, and so on.

plage <- 60
cval <- 3
ylim <-  range(DT$col1) + c(-1,1) * .05 * diff(range(DT$col1))

for( i in (plage + 1):65 ) {
  TS_tmp <- ts(DT[(i-plage):i]$col1, frequency = 7)
  aa <- auto.arima(TS_tmp)
  mo_tmp <- locate.outliers.oloop(TS_tmp, aa,
                                  maxit.iloop = 100, maxit.oloop = 100,
                                  cval = cval)
  d_tmp <- discard.outliers(mo_tmp, TS_tmp, method = "bottom-up",
                            tsmethod.call = aa$call, cval = cval)

  Outl <- TS_tmp[sort(d_tmp$outliers$ind)]
  print(plot(DT[(i-plage):i]$col1, type = "b", xlim = c(1, nrow(DT)), ylim = ylim))
  print(points(sort(d_tmp$outliers$ind), Outl , col = "red"))
}

It seems to perform worse... (See image 2)

Then again, I don't know much about time series and don't really know how to evaluate how well a model performs.

My questions :

Are any of the two approaches a good idea ? Which one is best ? Any better idea ?
I'm using the 9 months of data I have in approach n1. In approach n2 (if it's relevant), how much data should be included in my training set ? (currently : 60 days)
Any advice on how to best calibrate the model(s) ?

I suggest you try to remove the trend (growing) from your data then try to model it. Then difference with model could be your outlier and alarm

see this:https://anomaly.io/seasonal-trend-decomposition-in-r/


Nutritional image classification task I need a model that is able to receive as input an image of a nutritional information chart and tell the level of sugar that the product has. It would be a 3-class classification problem (low if sugar is below 5g, medium if it's between 5 and 22.5g and high if it has more than 22.5g). I have prepared all the data and I have 16000 images in total. However, I'm not able to train a proper model with the data. I have tried a simple convolutional neural network of 3 convolutional layers, the pretrained inception resnet v2 from keras, and even an attentional model (Github). The result is always the same, an accuracy equal to the proportion of samples from the most common class. So these models are unable to solve the issue and just bet for the most likely.
What kind of network could be able to solve this problem? I have never dealt with networks that have to "read" and classify text.

Another option would be to apply an established Optical Character Recognition (OCR) system to the raw images. After converting the raw images to plain text, put the relevant data into a tabular dataframe. Once in a tabular dataframe, fit a deep learning or traditional machine learning model can be fit.
It can be tempting to build an end-to-end deep learning system. However, some problems are best solved by combination of subsystems. Those subsystems could include human expertise, rules, traditional machine learning, and just a small amount of deep learning.


How to implement Brown Clustering Algorithm in O(|V|k^2) I am trying to implement the Brown Clustering Algorithm.

Paper details: "Class-Based n-gram Models of Natural Language" by Brown et al

The algorithm is supposed to in O(|V|k^2) where |V| is the size of the vocabulary and k is the number of clusters. I am unable to implement it this efficiently. In fact, the best I can manage is O(|V|k^3) which is too slow. My current implementation for the main part of the algorithm is as follows:

for w = number of clusters + 1 to |V|
{
   word = next most frequent word in the corpus

   assign word to a new cluster

   initialize MaxQuality to 0

   initialize ArgMax vector to (0,0)

   for i = 0 to number of clusters - 1
   {
      for j = i to number of clusters
      {
         Quality = Mutual Information if we merge cluster i and cluster j

         if Quality > MaxQuality
         {
            MaxQuality = Quality

            ArgMax = (i,j)
         }
      }
   }
}

I compute quality as follows:

1. Before entering the second loop compute the pre-merge quality i.e. quality before doing any merges.
2. Every time a cluster-pair merge step is considered:
    i. assign quality := pre-merge quality
   ii. quality = quality - any terms in the mutual information equation that contain cluster i or cluster j (pre-merge)
  iii. quality = quality + any terms in the mutual information equation that contain (cluster i U cluster j)  (post-merge)

In my implementation, the first loop has approx |V| iterations, the second and third loop approx k iterations each. To compute quality at each step requires approx a further k iterations. In total it runs in (|V|k^3) time.

How do you get it to run in (|V|k^2)?

I have managed to resolve this. There is an excellent and thorough explanation of the optimization steps in the following thesis: Semi-Supervised Learning for Natural Language by Percy Liang.

My mistake was trying to update the quality for all potential clusters pairs. Instead, you should initialize a table with the quality changes of doing each merge. Use this table to find the best merge, and the update the relevant terms that make up the table entries.


Filtering Pandas Dataframe Results without saving to a variable first Is there a way to more efficiently filter a result on a data frame without having to explicitly save it in a variable and then filter? For instance, in the code below I would like to add something to line 3 to be able to achieve either df1 or df2.

import pandas as pd
df=pd.DataFrame({'A' : [1,2,3,4,5], 'B' : [0,0,0,0,0]})
df=df.var()
df1 = df[df!=0]
df2 = df[df > 3]

If you want to filter Pandas Series "on the fly", you can use .loc[] indexer in conjunction with the callable method (for example using lambda function).

Demo:

In [8]: df.var()
Out[8]:
A    2.5
B    0.0
dtype: float64

In [9]: df.var().loc[lambda ser: ser!=0]
Out[9]:
A    2.5
dtype: float64

In [10]: df.var().loc[lambda ser: ser>3]
Out[10]: Series([], dtype: float64)

If you want to filter a DataFrame, then you can use DF.query(...) method:

In [11]: df
Out[11]:
   A  B
0  1  0
1  2  0
2  3  0
3  4  0
4  5  0

In [12]: df.query("A >= 3")
Out[12]:
   A  B
2  3  0
3  4  0
4  5  0


Identifying trend and seasonality of time series data As a part of a statistical analysis engine, I need to figure out a way to identify the presence or absence of trends and seasonality patterns in a given set of time series data. While most answers and tutorials in the Internet outlines methods to predict or forecast time series data using machine learning models, my objective is simply to identify the presence any such pattern.

Example: daily sales data through a year

This data set could show upward trends with monthly seasonality
or no actual trend with yearly seasonality.

If I am not to manually inspect the scatter plot of the distribution, in what ways can I determine the presence of those patterns?

I have looked at the following approaches so far:

Using moving average or exponential smoothing to smooth the time series curve and then check if the resulting line can be approximated to a linear curve, which is supposed to provide an upwards or downward trend if any.
Using auto correlation to check for seasonality, which I am yet to confirm if it's a possible solution or not (any ideas?)

Another idea could be Fourier Transformation, which takes a time serie as an input (time domain), and converts it into frequency domain.

Consider this example:

When you transform the time series data from time domain into frequency domain, you can observe the repeated patterns (=seasonality). In this case, there is peak at 12 (day/night rhytm), at 24 (day) or 168 (week).

I can imagine you can perform the FFT and then extract the peaks based on certain threshold

The Git repo also contains a nice introduction into FFT.

Source: https://github.com/balzer82/FFT-Python


Can machine learning models treat a vector as a whole feature to learn We know a ML model naturally takes a feature vector with real valued elements as input and learn to predict. But can it treat a fixed-size vector as a whole feature to learn?
For example, when using a 128-bit string as a feature, we can calculate the total number of different strings occurred in datasets and apply one-hot encoding to them. However, the dimension of the encoding can be very large（thousands）in this way. Or we can use binary to represent the 128-bit string, which only contributes to 128 dimensions. I don't know if the traditional ML models (not CNN) such as random forests are able to learn the dependence of the 128 features. Or is there any ways they can treat the 128 features as a whole?

To avoid having too much features produced by a OneHot, you can either :

Transform your data before your OneHot, regrouping different classes in intervals you created yourself (for example, reducing your 128 features to 30 classes you created yourself) then applying OneHot
Use CategoryEncoder (Such as TargetEncoder, James-Stein, LeaveOneOut, WeightOfEvidence) : this will make stats and transform the classes into a number, higher if the class is well known (appearing often) and tend to give more True prediction than the average


SelectKBest for text analytics I have a corpus of data

Class   Tweet
-1  toxic phenol ingredient in vaccines  the detrimental effects of injected phenol  have not been fully evaluated
-1  doctors give flu shots to pregnant women despite evidence of harm to fetus
-1   hearus autism and the mmr vaccine  the most diabolical medical scandal of the century       cdcwhistleblower  vaxxed
-1  rt   how to naturally detox from mandatory vaccine injections     hearus  cdcwhistleblower
-1  rt   the removal of vaccine exemptions forces parents who know that vaccines injure and sicken their children into coerced ha

I'll be doing some supervised learning on the text.  I put the text through a pipeline as follows:

selector = SelectKBest(chi2, k = K)

clf = SVC(kernel = 'linear')

p = Pipeline([('vect', CountVectorizer()),
                    ('tfidf',TfidfTransformer() ),
                    ('feat',selector),
                    ('clf',clf)])

SelectKBest will return K of the best features.  I suppose then that those will be used to train the model.

Do I need to predict new tweets on the same features?  Or can I just leave the pipeline as is and use CV to find the best K?

Your Pipeline is now a model that expects similar input as before. You can use a CV scheme to choose your K inside your Pipeline, after it has evaluated the options and chosen the correct K it will use that to predict. For actual predictions, enter a list of tweets and it will classify them, no need to manually specify which features to use. If you want to know which features your Pipeline has chosen you can look in the properties of your 'p' object.


Identifying repeating sequences of data in byte array Given a sample of hexadecimal data, I would like to identify UNKNOWN sequences of bytes that are repeated throughout the sample. (Not searching for a known string or value) I am attempting to reverse engineer a network protocol, and I am working on determining data structures within the packet. As an example of what I'm trying to do (albeit on a smaller scale):

(af:b6:ea:3d:83:02:00:00):{21:03:00:00}:[b3:49:96:23:01]

{21:03:00:00}:(af:b6:ea:3d:83:02:00:00):01:42:00:00:00:00:01:57

And

(38:64:88:6e:83:02:00:00):{26:03:00:00}:[b3:49:96:23:01]

{26:03:00:00}:(38:64:88:6e:83:02:00:00):01:42:00:00:00:00:00:01

Obviously, these are easy to spot by eye, but patterns that are hundreds of chars into the data are not. I'm not expecting a magic bullet for the solution, just a nudge in the right direction, or even better, a premade tool.

I'm currently needing this for a C# project, but I am open to any and all tools.

I believe the problem that you are referring to, is that of "Motif Discovery in Time Series Data". An appreciable amount of research literature already exists in this domain, so you can look through those. If the data that you handle is not very large, you can find some relatively easy to implement algorithms.

If the data is large , you can look at more recent publications in this domain. As a starting point I would recommend taking a look at how Motif Discovery is done in SAX. SAX takes continuous signals as inputs and discretizes them. These discrete levels are then stored as alphabets. This resulting data looks very much like yours in my opinion. Take a look at what they do in "Mining Motifs in Massive Time Series Databases".


Why use gradient descent on Deep Nets / RNNs when cost function is not convex? Why do we use gradient descent on very non-convex loss functions such as in Deep nets / RNNs rather than a heuristic search (genetic algorithms, simulated annealing, etc)?

Even if your cost function is not convex, any minimization algorithm will do its job - including gradient descent. That said, a standard gradient descent is a local optimization procedure and can only ever guarantee a local minimum, not a global one. For quite a few applications, a good local minimum is good enough - for others, a lot of effort is put into using “global” optimization algorithms (such as genetic/evolutionary, simulated annealing, scatter search, multi start techniques, direct/multilevel searches etc...). Those algorithms are definitely more powerful but they are also significantly slower in converging to a good, potentially global, minimum. And none of them can guarantee global optimality unless you allow them a veeeeeeery large number of functions evaluations.


Class weighting during validation in Keras I would like to know if the class weighting is also used in evaluating the loss in the validation data during the training. If not, is there a way to adjust the fit() function so that it takes into account the class weighting during the validation loss calculation.

As described in Keras.models.py[1], validation_data can be a tuple (inputs, targets, sample_weights).  So, you can use this parameter for class weighting during the validation loss calculation.

[1] https://github.com/fchollet/keras/blob/master/keras/models.py


Handling unbalanced datasets with XG boosting Suppose you want to model (predict) a rare disease, and you use the parameter "pos scale weight" as a hyperparameter in XG boost . For example I have 20 times more positive cases, can I then use pos scale weight = 0.05, even though for example the ratio in het the real world would be not 1/20 but 1/2000?

This is a tricky question because it depends on your objective.

If your objective is to have comparable performance on the two classes (i.e. comparable sensitivity and specificity), and the imbalance in your training data is $1:20$, then yes, it makes sense to give the minority class examples a weight 20 times greater than for examples of the majority class.

However, in most applications, this is not exactly the objective. You should use the real costs of misclassification, or an estimation thereof. Ideally, you would quantify the costs of false negatives and false positives (although this is easier said than done, especially when comparing the cost of illness to, for example, the cost of a medical professional's wasted time).

Then, if your training set had the same class priors (how often a class occurs) as the real data, you would use these costs directly as weights for, respectively, the positive and negative classes.

In your case, it's more complicated, because the priors in your training data are different from those in the real data. You cannot use the misclassification costs directly as weights, you have to adjust for the real distribution.

Using a dataset with a $1:20$ imbalance instead of $1:2000$, you've effectively already given the negative class about 100 times more weight than it naturally has. It's like there's already a weight hidden in there. For the misclassification costs to be used as weights, you would have to cancel out that factor.

Suppose $c_{fp}$ is the cost of a false positive (misclassifying a negative example) and $c_{fn}$ is the cost of a false negative (misclassifying a positive example). If rate of positive data is $1/20 = 0.05$ in the training data and $1/2000 = 0.0005$ in the real distribution of your domain, then you would use:

$$
\frac{\frac{1}{2000}}{\frac{1}{20}}\cdot c_{fp} = \frac{c_{fp}}{100}$$
as the weight for errors on the negative class and
$$\frac{\frac{1999}{2000}}{\frac{19}{20}}\cdot c_{fn} \approx 1.052 \cdot c_{fn}$$

as the weight for errors on the positive class.


Remove unnecessary Neurons from a Neural Network regarding a particular output Suppose we have a Neural Network with a binary output (0 or 1). What I am trying to do is to remove neurons or layers from the NN while maintaining a correct classification for all the instances that were classified as 1 in the original NN, same thing for the output 0. Said differently, is there any way to spot neurons that are paramount to the correct classification of the instances of a particular class ? The aim is to remove all the unnecessary neurons regarding that output. Currently, I am trying to use the back propagation phase to try to attribute a fitness to each neuron regarding its contribution to a certain class.
In the case of Binary Neural Networks (binary weights and activations), a research track could be compiling the NN to a Boolean Formula and reasoning on it to spot the neurons that does not contribute to the chosen output, but it is not always obvious to carry out this compilation.

There is a lot of literature on this topic. For example, have a look at https://arxiv.org/pdf/1710.09282.pdf


SVM regularization - minimizing margin? I'm currently studying from Andrew Ng's Stanford handouts here (I'm at part 8). Now from what I gathered from before, all the time our goal was to minimize ||w||^2 so that we can maximize the margin. However, now as he's writing about the regularization, he's saying:

  The parameter C controls the relative weighting between the twin goals of making the ||w||^2 large (which we saw earlier makes the margin small) and of ensuring that most examples have functional margin at least 1.

Why would making the margin small be currently a "goal"?

You are minimizing the entire loss equation. If it contains regularization, you force the weights to be small too. Having small weights is favorable characteristic because the algorithm is not focusing strongly on one feature, all happen to be important, so the risk of overfitting to some feature is smaller.

So it is really some tradeoff as always, between the big margin but small focus on dataset distinct examples.


What do I need to learn in order to really understand Machine Learning? I've graduated with a math major in undergrad, but mostly focused on algebra (Galois Theory, Knot theory, etc.). I work in something unrelated right now, but now I want to study machine learning. The question is, what kind of knowledge should I have if I want to really understand machine learning?

Say, here are some things I can think of, but obviously I'm missing a lot, I'm assuming.

"Fundamentals" (Calculus, Linear Algebra, Discrete Mathematics, Coding, etc.)
Probability (but what specific areas?)
Statistics (but what kind?)
Algorithms
Differential Equations

But what else? Or what subfields of what I've mentioned above are particularly important (i.e. Bayesian statistics)?

Edit: I am currently considering a graduate program in ML, and wanted to know if this is something I really want to do / know more about it / prepare myself.

From somebody with a PhD in Probability working with AI/ML for a living. Basics of Probability theory, maybe wikipedia/cousera/... for the very basics if you never had a class in it, followed by e.g. “Probability with Martingales” by Williams.
The papers here will also give you a good feel.
As for books, this one on “classical” machine learning is pretty good and free For Deep Learning https://www.deeplearningbook.org/, this one also has the basics of probability theory.
For reinforcement learning http://incompleteideas.net/book/the-book-2nd.html.
For the applied side, slides from https://web.stanford.edu/class/cs224n/ and http://cs231n.stanford.edu/.
As for statistics don’t bother, I’ve never seen any machine learning work reference a theorem in “pure” statistics, if there is such a thing.
Of course standard undergraduate calculus and linear algebra. And learn some Python while you’re at it


Number of units for first layer in Keras Sequential Model I have a huge CSV structured dataset. I'm feeding that dataset to a Keras Sequential Model. My question is, can my Model have number of units greater than the number of input features?
At the moment, my features or csv columns are 118 and Model summary is as:

First layer of model has 256 units with elu activation.
Second layer 128 with elu activation
Third layer 64  with elu activation
Last layer 1 unit with segmoid activation

Now, I'm getting 100% accuracy on both training and testing data, which is surprising to me. For 50 features, the model was giving 94.25% accuracy on training and 95.68% accuracy on testing data. Accuracy can be even 99 or 99.XX% but 100% is something wrong I guess? If it's 100% on test data, it's probably not the overfitting..? Please guide me!

You can try "Border Pairs Method", "Bipropagation" or "One step method" to finding out how many neurons you need in each layer.
https://www.researchgate.net/publication/322617800_New_Deep_Learning_Algorithms_beyond_Backpropagation_IBM_Developers_UnConference_2018_Zurich


Dealing with optional data in an Artificial Neural Network I want to train an Artificial Neural Network on some data however some of the fields are optional, dependent upon the values contained in other fields. The data in these optional fields is missing, but correctly so. I am wondering how I should factor this into my neural network.

I feel like what I might need to do is turn certain inputs on/off dependent upon the values contained in other inputs but then I am wondering how this would affect weighting in the hidden layer. Could I have some kind of conditional hidden layer or something for only when these optional values are 'activated'?

What would be the standard approach to this? It seems surprisingly hard to find any material on this online!

Thanks in advance for any help!

Several options, depending on your algorithms:

Default to 0
Default to 1
Default to Average


How to interpret FPR and TPR in ROC curve? I am playing with ROC and trying to draw some curves. I am using example from this scikit page. I do not understand one thing: when I print out the content of tpr and fpr, I see two arrays of numbers (21 elems in each) - why is that?

TPR (True Positive Ratio) is a proportion of those tuples classified as positives to all real positive tuples. So, it should be one number

I see it as follow: I take classifier (like Decision Tree), train it on some data and finally test it. Then I can calculate TPR and FPR and I should have only two values.

Instead, I receive arrays. Why is that? I tried to change classifier from that example to Decision Tree and then I receive only one point (which is usually [0, 1]). How can I draw ROC curve on decision tree? Or - to be more precise with my question - why do I get an array of values as my TRP/FPR instead of single values?

As explained here in the docs of the function returning the ROC Curve
they return a array because when you interpret a ROC curve what you're looking the performance of the predictor given a threshold. So, it means that, if I'm looking for only classifications that the model is above 80% sure, the ROC curve will have a different TPR/FPR than a threshold of 90%.

If you have further questions, a great link explaining it: http://www.dataschool.io/roc-curves-and-auc-explained/


Why Is Overfitting Bad in Machine Learning? Logic often states that by overfitting a model, its capacity to generalize is limited, though this might only mean that overfitting stops a model from improving after a certain complexity. Does overfitting cause models to become worse regardless of the complexity of data, and if so, why is this the case?

Related: Followup to the question above, "When is a Model Underfitted?"

Overfitting is empirically bad.  Suppose you have a data set which you split in two, test and training.  An overfitted model is one that performs much worse on the test dataset than on training dataset.  It is often observed that models like that also in general perform worse on additional (new) test datasets than models which are not overfitted.

One way to understand that intuitively is that a model may use some relevant parts of the data (signal) and some irrelevant parts (noise).  An overfitted model uses more of the noise, which increases its performance in the case of known noise (training data) and decreases its performance in the case of novel noise (test data).  The difference in performance between training and test data indicates how much noise the model picks up; and picking up noise directly translates into worse performance on test data (including future data).

Summary: overfitting is bad by definition, this has not much to do with either complexity or ability to generalize, but rather has to do with mistaking noise for signal.

P.S. On the "ability to generalize" part of the question, it is very possible to have a model which has inherently limited ability to generalize due to the structure of the model (for example linear SVM, ...) but is still prone to overfitting.  In a sense overfitting is just one way that generalization may fail.


Should one-hot encoded categorical features needs to be scaled when used along with text feature while deriving semantic similarity? My aim is to derive textual similarity using multiple features. Some of the features are textual for which I am using (Tfhub 2.0) Universal Sentence encoder. There are other categorical features which are encoded using one-hot encoder.
For example, for a single record in my dataset, feature vector looks like this:

text feature's embedding is 512 dimension vector - 1 X 512
categorical (non-ordered) feature vector -  1 X 500 (since there are 500 unique values in the feature)
my final feature vector - 1 X 1012

After this, I derive similarity matrix using cosine-similarity to decide if two such records are semantically same or not.
Problem is, there is a difference in the range of values for text feature (real numbers) and one hot encoded feature (0 or 1).
So shall I scale the one hot encoded vector with min-max scalar or using some other technique?

No, do not scale the one hot encoded vector with min-max scaling. That will lose the meaning of the data. One hot encoding means a data point is completely on a dimension or not. There is no meaning for those data points that are only fractional part of a dimension.
A better option to derive textual similarity using multiple features is to embedded all features in the same embedding space (including categorical features). StarSpace is one such embedding method.


Word taxonomies for Facebook likes categories I query the Facebook graph API to get some users' likes, which come with a "category" field, which can be, for instance, Italian Restaurant or Health & Wellness website and so on.

I need to draw a profile of the user so I was thinking of retrieving the top level root in a taxonomy graph of the aforementioned categories in such a way that the first example falls into something like Food and the second one into something like Health. This way I'd know, by a majority rule, who is a sporty person, who likes fashion etc.

Is there a free public API I could use to get word taxonomies? I found Alchemy, an IBM project which is pretty cool but unfortunately limited to 1000 API requests per day and I'd need way more than that given the big amount of users and categories I have.

I know I'm probably asking for too much (a free and unlimited tool!) so the question might become "do you have better ideas about how to solve this"?

I think total number of default categories in Facebook is somewhat static. There is around 140-160 categories.

ref: https://stackoverflow.com/questions/4216648/facebook-pages-authoritative-list-of-categories/8576572#8576572 and http://www.marketinggum.com/types-of-facebook-pages-for-business/

You can cluster (manually) them in to 10-12 generic groups.

eg: Sports Event,
    Sports League,
    Amateur sports team,
    Sports Venue,
    Athlete,
    Sports/Recreation/Activities etc
all belongs to Sports category.

Similarly cluster and form generic groups.

And then you can assign a user to any of these generic group based on the like category.


GridSearchCV vs RandomSearchCV and How it works? GridSearchCV vs RandomSearchCV

Can somebody explain in-detailed differences between GridSearchCV and RandomSearchCV?
And how the algorithms work under the hood?

As per my understanding from the documentation:

RandomSearchCV

This uses a random set of hyperparameters. Useful when there are many hyperparameters, so the search space is large. It can be used if you have a prior belief on what the hyperparameters should be.

GridSearchCV

Creates a grid over the search space and evaluates the model for all of the possible hyperparameters in the space. Good in the sense that it is simple and exhaustive. On the minus side, it may be prohibitively expensive in computation time if the search space is large (e.g. very many hyper parameters).

Imagine the following scenario:

params = {
   epoch = [20, 30, 40, 50], #those numbers are only for example
   dense_layer_size = [20, 30],
   second_danse_layer = [30, 40]
}

In GridSearch you try all the combinations of your parameters, in this case:

(4 * 2 * 2) = 16 #Total of parameters

In RandomSearch as in documentation:

  Not all parameter values are tried out, but rather a fixed number of parameter settings is sampled from the specified distributions. The number of parameter settings that are tried is given by n_iter.

In this case, he select "n_iter" number of combinations and try then. Is good for optimizing less parameters, but, in cases when you don't have sure about multiples parameters, may let aside some combinations that would be better.

Another good approach, is Using an genetic algorithm to optimize your network.

Genetic algorithm generate a few combinations of "individuals" (combination of parameters) and at each step, select the best individuals(called parents) and crossover then, generating more individuals with the characteristic from both parents. This way, in some case you can optimize your network and add random elements when you don't have sure about what parameter would be increase your result.

If you want an easy to integrate genetic algorithm you can check this project.


Why do we need activation function (like ReLU) after an affine layer? In Convolutional Neural Networks, assume the input and the output of the affine layer are $x$ and $y$, respectively. This affine operation $y = W^{\top} x + b$ has already add non-linearity to the system given that $b \neq 0$.

Why do we still need a function like ReLU to add non-linearity to the system?

This affine operation $y = W^{\top} x + b$ has already add nonlinearity to the system given that $b \neq 0$.

This is not considered a non-linearity in the context of data science. Different disciplines define linearity sometimes in subtly different ways. Critically, the $+b$ performs identically in terms of fitting to data, as extending $x$ with a new dimension, always $1$, and moving the values of $b$ into the weights $W$. This simpler multiplication is clearly linear.

Also importantly, the affine transformations form a group such that any two affine transformations combined are just another affine transformation with different parameters. Without a non-linearity in a hidden layer, a 2-layer neural network would be the same as a single layer one, and not able to learn whole classes on non-linear relations.

No matter how many affine transformations you apply to inputs for instance, you will not be able to approximate the XOR function, or any significant portion of $y=\text{sin}(x)$


Machine learning algorithms for forming Homophones from input dataset word https://www.google.com/search?sxsrf=ALeKk01_SgA8G4UfNm4rOqku4yJBFvKhLw%3A1600154854621&source=hp&ei=5mxgX8ztI6KZ4-EPq-mL8Ak&q=homophones+example&oq=Homophones&gs_lcp=ChFtb2JpbGUtZ3dzLXdpei1ocBABGAEyBQgAELEDMgUIABCxAzICCAAyCAgAELEDEIMBMgUIABCxAzICCAAyAggAMgUIABCxAzoHCCMQ6gIQJzoECCMQJzoFCAAQkQI6CAguELEDEIMBOgUILhCxA1DkKliKSGDuUGgBcAB4AIAB6wGIAe8NkgEFMC44LjKYAQCgAQGwAQ8&sclient=mobile-gws-wiz-hp
Are there Machine learning algorithms for forming Homophones from input dataset word?
Homophones examples :
accessary, accessory.
ad, add.
air, heir.
all, awl.
allowed, aloud.
alms, arms.
Input : ad
Output : ad, add
Are there Machine learning algorithms for forming Homophones from input dataset word taking Indian regional languages viz Hindi, Gujarati, Bengali etc and other languages viz French, German, Italian, Spanish, Dutch etc?

I have very limited knowledge about homophones generator.
I feel to make a homophone detector, one should focus more on the phonetics of the word rather than the spellings.

try to make a word-phonetics list dataset and then train a model.
focus on Levenstein/fuzzy/edit distance between the phonetics of words.

eg - two, too and to all have the same phonetics - T UW .
try this website to find phonetics - http://www.speech.cs.cmu.edu/cgi-bin/cmudict?in=to
they have already mapped the words and phonetics.
I think finding homophones in non-english languages would be an uphill task.


Trouble with grouping by and counting in r I have a data.frame called world. This data frame lists countries in column name.y, I have a column for languages named languages.  I need to List the names of countries where more than 4 languages are spoken.

My first take on this is to aggregate country and languages.

language <- data.frame(table(World$name.y,World$language))

when I run head(language) it doesn't work out.

The dplyr package should be useful for this.

library(dplyr)

World %>%
  group_by(name) %>%
  mutate(num_distinct_languages = n_distinct(language)) %>%
  filter(num_distinct_lanagues > 4)


Why isn't leaky ReLU always preferable to ReLU given the zero gradient for x<0? It looks to me like the leaky ReLU should have much better performance since the standard ReLU can’t use half of its space (x < 0 where the gradient is zero). But this doesn't happen and in practice most people use standard ReLU.

One reason that ReL Units have been introduced is to circumvent the problem of vanishing gradients of sigmoidal units at -1 and 1.

Another advantage of ReL Units is that they saturate at exactly 0 allowing for sparse representations, which can be helpful when hidden units are used as input for a classifier. The zero gradient can be problematic in cases where the unit never activates in a gradient based scenario when the unit is initially not activated.

This problem can be alleviated by using leaky ReL Units. On the other hand, leaky ReL Units don't have the ability to create a hard-zero sparse representation which can be useful in certain cases. So, there is a bit of a trade-off and, as in general with NN, it depends on the use cases when which unit performs better. In most cases, if the initial settings can make sure that the ReL Unit is activated (e.g. by setting the biases to small positive values) one would expect ReL and leaky Rel Units to perform very similarly.

Also, leaky RelU (if parametric) introduces another parameter (the slope for $x<0$) that needs to be learned during training and therefore adds more complexity/training time.


What are the elements in a BERT word embedding? As far as I understand, BERT is a word embedding that can be fine-tuned or used directly.

With older word embeddings (word2vec, Glove), each word was only represented once in the embedding (one vector per word). This was a problem because it did not take homonyms into account. As far as I understand, BERT tackles this problem by taking context into condsideration.

What does this mean for the word embedding itself? Is there still one vector for each word token? If so, how is context taken into consideration? If no, what is the format of the embedding?

Some points first:

BERT is a word embedding: BERT is both word and sentence embedding. It needs to be taken into account that BERT is taking the sequence of words in a sentence into account which gives you a richer embedding of words in a context but in classic embeddings (yes, after BERT we can call others "classic"!) you mostly deal with neighborhood i.e. the semantic of the word vector is kind of the average of all semantics it had in the training set.
it did not take homonyms into account: Either a typo from you or I did not understand homonym very well. To be honest had to search it! Google says "two or more words having the same spelling or pronunciation but different meanings like "right" and "write". That is not a problem for word embedding. Maybe you meant something else?!
BERT tackles this problem by taking context into consideration: All embeddings take context into consideration. The difference is in capturing context when the sequence of the words are taken into account e.g. modeling sentence.

About questions:

Is there still one vector for each word token?: It's one vector per token per layer. Means that for finding a single vector for a word you take n layers and sum the values up. As you know, more you go towards later layers (i.e. toward the output layer) information (features) encoded in layers gets richer. One could also concatenate all after each other and get a higher dimensional representation. Be aware that it is completely task dependent. BERT authors tried different things and came up with summing up last four layers for NER task.

The interesting thing in comparison with classic embeddings is that you can now encode the sentence-dependent semantic of a token! Means that in word2vec (trained on general corpus text like Wikipedia) you have 1 vector for the word apple and if you inspect it you probably see that it is in a relation with both iphone and fruit (never tried it. Just made up an example to make my point for you. Let me know if you tried and something else came out!) but with BERT, you can encode the sentences containing similar words in different context and check the encoding of those words in the sentence. You surprisingly see how it captures the semantics!

The last but not the least is this blog post which is the base of my answer.

Hope it helped!


What techniques should I use to compare the similarity between a bunch of texts? If I have a list of job postings stored as raw texts and I want to compare the similarity of all the job postings to a given resume, what techniques or algorithms should I use?

I'm thinking of starting with a model that converts the job postings into a vector space using TF-IDF and then just calculate the matrix cosine similarity. Can I improve on this?

As of now, I can think of two ways to formulate this problem:
1. Search problem
Parse your job listings and index them in some sort of search engine like Solr or ElasticSearch. You can build capabilities like Semantic search using Word2Vec models, etc.
Now write a query engine which takes a resume and queries this Search engine. It will be blazing fast since job listing will be all indexed.
2. Similarity problem
I would have created hybrid similarity function. For example:
a) How many top key words matched between resume and job listing
b) Similarity of resume and job listing using Doc2Vec (Pre compute vector for job listings)
c) Using algorithms like Locality Sensitive Hashing to reduce the lookup space
This approach will be slow but might yield a good result.


Using a K-NN Classification Approach for Time Series Data? I have a dataset which contains time-series data of water flow over time.  I have a flow meter connected to a kitchen faucet, and I am trying to cluster or classify specific water usage events.

The data is collected every second, and in each row I am given a value for the amount of gallons which are flowing through my flow meter.

For example, I am trying to classify someone washing their hands, filling a teapot, cleaning dishes, etc...

Is this something that I can use a k-NN Classification Approach to cluster these events?  If a clustering based approach isn't good, what other method of classified would be good for this type of data?

If I run some experiments, I can classify each event and turn it into a supervised learning problem.  But at the moment, none of the water events are classified.

A very abridged version of my dataset looks like the following:

EDIT

water = pd.DataFrame(shower1)
rng = pd.date_range('2016-09-01 00:00:00', '2016-09-30 23:59:58', freq='S')
water = water.reindex(rng,fill_value=0.0)
water = water['shower1']
df = pd.DataFrame({'time_stamp':rng,'water_amount':water})

starts = (df['water_amount']>0)&(df['water_amount'].shift(1)==0) #find all starts of events
n_events = sum(starts) #total number of events
df.loc[starts,'event_number'] = range(1,n_events+1) #numerate starts from 1 to n
df['event_number'] = df['event_number'].fillna(method='pad').fillna(-1) #forward fill all the values
df.loc[df['water_amount']==0,'event_number']=-1 #set all event numbers to -1 where the water amount is 0

df.groupby('event_number').agg({'time_stamp':'first',
                                    'water_amount':'sum'}) #feature matrix

It seems pretty clear from looking at the data when an event starts and ends(basically whenever there is a sequence of positive values). So, instead of starting with some complicated models, I'd suggest calculating a few simple features (like length of the event, total amount of water, amount/seconds, time to previous event, time of day in seconds from start of recording) for every event and then try some clustering algorithm on that new data. k-NN might even produce something meaningful. But a statistical summary of the features can probably already give you a better idea of how to further approach this.

EDIT1

import pandas as pd
import numpy as np

rng = pd.date_range('2017-01-01 14:00:00', '2017-01-01 14:01:00', freq='S')
water = [0,0,0.2,0.3,0.4,0,0,0.3,0.2,0.5]*6+[0]
df = pd.DataFrame({'time_stamp':rng,'water_amount':water,'event_number':np.zeros(len(water))})

j = 1
for k in range(len(df)):
    if df.ix[k,'water_amount']== 0:
        df.ix[k,'event_number'] = -1
    else:
        if df.ix[k-1,'water_amount'] > 0:
            df.ix[k,'event_number'] = df.loc[k-1,'event_number']
        else:
            df.ix[k,'event_number'] = j
            j = j+1

df.groupby('event_number').agg({'time_stamp':'first',
                                'water_amount':'sum'}) #feature matrix

EDIT2

rng = pd.date_range('2017-01-01 14:00:00', '2017-01-01 14:01:00', freq='S')
water = [0,0,0.2,0.3,0.4,0,0,0.3,0.2,0.5]*6+[0]
df = pd.DataFrame({'time_stamp':rng,'water_amount':water})

starts = (df['water_amount']>0)&(df['water_amount'].shift(1)==0) #find all starts of events
n_events = sum(starts) #total number of events
df.loc[starts,'event_number'] = range(1,n_events+1) #numerate starts from 1 to n
df['event_number'] = df['event_number'].fillna(method='pad').fillna(-1) #forward fill all the values
df.loc[df['water_amount']==0,'event_number']=-1 #set all event numbers to -1 where the water amount is 0

df.groupby('event_number').agg({'time_stamp':'first',
                                    'water_amount':'sum'}) #feature matrix


Initial values of memory and previous block output in LSTM? I am trying to understand LSTM and reading colah blog , As LSTM structure looks like this :

So LSTM takes three inputs:

Input vector
Memory from previous block
Output from previous block

and return :

memory from current block
output from current block

which will become previous memory and previous block output for next state ,

My confusion is what is the initial values of ct-1 ( memory from previous block) and ht-1 ( hidden output from previous block) , since there is no previous block then what are initial values ?

Either you use a zero vector state for both (most commonly used), or you can use other approaches such as described in this article.

I have personally always used zero vectors.


What Does the Normalization Factor Mean in the AdaBoost Algorithm? I am studying the AdaBoost algorithm. The update rule for a weak hypothesis is:

$Dt+1(i) = Dt(i)exp(−αtyiht(xi))/zt $

where $zt$ is a normalization factor chosen so that $Dt+1$ is a distribution.

What does the 'normalization factor' mean? Could I have an explanation with an example, please?

The normalization factor  is used to reduce any probability function to a probability density function with total probability of one. See Wikipedia.

Say your unnormalized value is [0.1, 0.2, 0.3, 0.2]. You normalize it by dividing it by $z_t=0.1+0.2+0.3+0.2=0.8$, therefore get the normalized value [0.125, 0.25, 0.375, 0.25] which sums up to 1..


Improve performances of a convolutional neural network I am doing image classificaition, and to do this I have built the following neural network:

def Network(input_shape, num_classes, regl2 = 0.0001, lr=0.0001):

model = Sequential()

# C1 Convolutional Layer
model.add(Conv2D(filters=96, input_shape=input_shape, kernel_size=(3,3),\
                 strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation before passing it to the next layer
model.add(BatchNormalization())

# C2 Convolutional Layer
model.add(Conv2D(filters=384, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation
model.add(BatchNormalization())

# C3 Convolutional Layer
model.add(Conv2D(filters=768, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Batch Normalisation
model.add(BatchNormalization())

# C4 Convolutional Layer
model.add(Conv2D(filters=384, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Batch Normalisation
model.add(BatchNormalization())

# C5 Convolutional Layer
model.add(Conv2D(filters=256, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation
model.add(BatchNormalization())

# C6 Convolutional Layer
model.add(Conv2D(filters=256, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation
model.add(BatchNormalization())

# C7 Convolutional Layer
model.add(Conv2D(filters=256, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation
model.add(BatchNormalization())

# Flatten
model.add(Flatten())

flatten_shape = (input_shape[0]*input_shape[1]*input_shape[2],)

# D1 Dense Layer
model.add(Dense(4096, input_shape=flatten_shape, kernel_regularizer=regularizers.l2(regl2)))
model.add(Activation('relu'))
# Dropout
model.add(Dropout(0.4))
# Batch Normalisation
model.add(BatchNormalization())

# D2 Dense Layer
model.add(Dense(4096, kernel_regularizer=regularizers.l2(regl2)))
model.add(Activation('relu'))
# Dropout
model.add(Dropout(0.4))
# Batch Normalisation
model.add(BatchNormalization())

# D3 Dense Layer
model.add(Dense(1000,kernel_regularizer=regularizers.l2(regl2)))
model.add(Activation('relu'))
# Dropout
model.add(Dropout(0.4))
# Batch Normalisation
model.add(BatchNormalization())

# Output Layer
model.add(Dense(num_classes))
model.add(Activation('softmax'))

# Compile

adam = optimizers.Adam(lr=lr)
model.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['accuracy'])

return model

  # create the model
  model = Network(input_shape,num_classes)
  model.summary()

it works good enough, but I would like to increase its performances.

How could I modify it to do so?

I was thinking about adding layers, which should give better performances, but I haven' t understand well if I should add convolutional layers or dense layers. Moreover I would like to find other ways to increase accuracy than simply adding layers.

Can somebody please help me?

Thanks in advance.

[EDIT] I am considering a training set of 1200 images, which represent 4 wheater conditions : Haze, Rainy, Snowy, Sunny.

With my model, the Test accuracy is 0.797500, and the Test loss is 1.881952.

I would like to increase more my accuracy, but I don' t have other ideas than adding convolutional layers. I could try to change the size of the kernels and other hyperparameters, but I have other ideas.

It sounds like the model is underfit; to verify this, you would check that training and testing accuracy are similar, but, both lower than desired.

One cause is likely the very low number of training images for a model of this complexity.

You could try augmenting the training set with transformations of those images, for example by rotating the images in the training set? (This was done in the original alexnet paper: http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf see section 4.1 on p.5).


How to evaluate RMSE with standard deviation I have regression model, where target is between 0 to 1. standard deviation of target is 0.817 and RMSE of model on hold out is 0.52. I am wondering if this good model or not.

Any feedback will be useful

Thanks

The residuals of your model seem to vary by a large extent (as denoted by RMSE, which is in the same units as the target). Without any context on the problem you are solving, it would be hard to decide whether the model is good or not. But, it does appear to be on the poorer side.


Supervised learning vs reinforcement learning for a simple self driving rc car I'm building a remote-controlled self driving car for fun. I'm using a Raspberry Pi as the onboard computer; and I'm using various plug-ins, such as a Raspberry Pi camera and distance sensors, for feedback on the car's surroundings. I'm using OpenCV to turn the video frames into tensors, and I'm using Google's TensorFlow to build a convoluted neural network to learn road boundaries and obstacles. My main question is, should I use supervised learning to teach the car to drive or should I provide objectives and penalties and do reinforcement learning (i.e, get to point B as fast as possible while not hitting anything and staying within the road boundaries)? Below is a list of the pros and cons that I've come up with.

Supervised learning pros:

The inputs to the learning algorithm are pretty straightforward. The car learns to associate the video frame tensor and sensor distance readings with forward, backward, and angular wheel displacement
I can more or less teach the car to drive exactly how I want (without overfitting, of course)
I've done tons of supervised learning problems before, and this approach seems to comfortably fit my existing skill set

Supervised learning cons:

It's not clear how to teach speed, and the correct speed is pretty arbitrary as long as the car doesn't go so fast that it veers off the road. I suppose I could drive fast during training, but this seems like a crude approach. Maybe I could manually add in a constant variable during training that corresponds to the speed for that training session, and then when the learning algorithm is deployed, I set this variable according to the speed I want?

Reinforcement learning pros:

If I build my car with the specific purpose of racing other people's self driving cars, reinforcement learning seems to be the natural way to tell my car to "get there as fast as possible"
I've read that RL is sometimes used for autonomous drones, so in theory it should be easier in cars because I don't have to worry about up and down

Reinforcement learning cons:

I feel like reinforcement learning would require a lot of additional sensors, and frankly my foot-long car doesn't have that much space inside considering that it also needs to fit a battery, the Raspberry Pi, and a breadboard
The car will behave very erratically at first, so much so that maybe it destroys itself. It might also take an unreasonably long time to learn (e.g., months or years)
I can't incoporate explicit rules later on, e.g., stop at a toy red-light. With supervised learning, I could incorporate numerous SL algorithms (e.g., a Haar Cascade classifier for identifying stoplights) into a configurable rules engine that gets evaluated between each video frame. The rules engine would thus be able to override the driving SL algorithm if it saw a red stoplight even though the stoplight might not have been part of the training of the driving algorithm. RL seems too continous to do this (ie., stop only at the terminal state)
I don't have a lot experience with applied reinforcement learning, although I definitely want to learn it regardless

I'd suggest you to try a hybrid approach:

First, train your car in supervised fashion by demonstration. Just
control it and use your commands as labels. This will let you get all
the pros of SL.
Then, fine tune your neural net using reinforcement learning. You
don't need extra sensors for that: the rewards may be obtained from
distance sensors (larger distances = better) and from the speed
itself. This will give you the pros of RL and train your NN to the correct goal of driving fast while avoiding obstacles instead of the goal of imitating you.
Combining both approaches will get you the pros of both SL and RL
while avoiding their cons. RL won't start from random behavior, just
small gradual deviations from what you tought the NN. A similar approach was applied successfuly by Google DeepMind with AlphaGo.
You can always include explicit rules on top of this. Implement them with high priority and call the NN only when there is no explicit rule for current situation. This is reminiscent of the Subsumption Architecture.


What dataset was Stanford NER trained on? I would like to re-train the Stanford NER library from scratch as a 1 class model.

Only 3,4 and 7 class models are available out of the box.

Is it possible to obtain the data that the model was originally trained on?

The original paper mentions two corpora: CoNLL 2003 (apparently here now) and the "CMU Seminar Announcements Task". However according to the page linked in the question the actual NER was trained on a larger combination of corpora:

  Our big English NER models were trained on a mixture of CoNLL, MUC-6, MUC-7 and ACE named entity corpora, and as a result the models are fairly robust across domains.

So it might be difficult to obtain the exact original training data. However most of these corpora were compiled for some shared tasks and should be available online. There are probably more recent ones as well: a quick search "named entity recognition shared task" returns many hits.


Transform an Autoencoder to a Variational Autoencoder? I would like to compare the training by an Autoencoder and a variational autoencoder. I have already run the traing using AE. I would like to know if it's possible to transform this AE into a VAE and maintain the same outputs and inputs.

Thank you.

Yes. Two changes are required to convert an AE to VAE, which shed light on their differences too. Note that if an already-trained AE is converted to VAE, it requires re-training, because of the following changes in the structure and loss function.

Network of AE can be represented as $$x \overbrace{\rightarrow .. \rightarrow  y \overset{f}{\rightarrow}}^{\mbox{encoder}} z \overbrace{\rightarrow .. \rightarrow}^{\mbox{decoder}}\hat{x},$$

where

$x$ denotes the input (vector, matrix, etc.) to the network, $\hat{x}$ denotes the output (reconstruction of $x$),
$z$ denotes the latent output that is calculated from its previous layer $y$ as $z=f(y)$.
And $f$, $g$, and $h$ denote non-linear functions such as $f(y) = \mbox{sigmoid}(Wy+B)$, $\mbox{ReLU}$, $\mbox{tanh}$, etc.

These two changes are:

Structure: we need to add a layer between $y$ and $z$. This new layer represents mean $\mu=g(y)$ and standard deviation $\sigma=h(y)$ of Gaussian distributions. Both $\mu$ and $\sigma$ must have the same dimension as $z$. Every dimension $d$ of these vectors corresponds to a Gaussian distribution $N(\mu_d, \sigma_d^2)$, from which $z_d$  is sampled. That is, for each input $x$ to the network, we take the corresponding $\mu$ and $\sigma$, then pick a random $\epsilon_d$ from $N(0, 1)$ for every dimension $d$, and finally compute $z=\mu+\sigma \odot   \epsilon$, where $\odot$ is element-wise product. As a comparison, $z$ in AE was computed deterministically as $z=f(y)$, now it is computed probabilistically as $z=g(y)+h(y)\odot \epsilon$, i.e. $z$ would be different if $x$ is tried again. The rest of network remains unchanged. Network of VAE can be represented as $$x \overbrace{\rightarrow .. \rightarrow  y \overset{g,h}{\rightarrow}(\mu, \sigma) \overset{\mu+\sigma\odot \epsilon}{\rightarrow} }^{\mbox{encoder}} z \overbrace{\rightarrow .. \rightarrow}^{\mbox{decoder}}\hat{x},$$
Objective function: we want to enforce our assumption (prior) that the distribution of factor $z_d$ is centered around $0$ and has a constant variance (this assumption is equivalent to parameter regularization). To this end, we add a penalty per dimension $d$ that punishes any deviation of latent distribution $q(z_d|x) = N(\mu_d, \sigma_d^2)$$= N(g_d(y), h_d(y)^2)$ from unit Gaussian $p(z_d)=N(0, 1)$. In practice, KL-divergence is used for this penalty. At the end, the loss function of VAE becomes: $$L_{VAE}(x,\hat{x},\mu,\sigma) = L_{AE}(x, \hat{x}) + \overbrace{\frac{1}{2} \sum_{d=1}^{D}(\mu_d^2 + \sigma_d^2 - 2\mbox{log}\sigma_d - 1)}^{KL(q \parallel p)}$$
where $D$ is the dimension of $z$.

Side notes

In practice, since $\sigma_d$ can get very close to $0$, $\mbox{log}\sigma_d$ in objective function can explode to large values, so we let the network generate $\sigma'_d = \mbox{log}\sigma_d  = h_d(y)$ instead, and then use $\sigma_d = exp(h_d(y))$. This way, both $\sigma_d=exp(h_d(y))$ and $\mbox{log}\sigma_d=h_d(y)$ would be numerically stable.
The name "variational" comes from the fact that we assumed (1) each latent factor $z_d$ is independent of other factors, i.e. we ignore other $(\mu_{d'}, \sigma_{d'})_{d' \neq d}$ when we sample $z_d$, and (2) $z_d$ follows a Gaussian distribution. In other words, $q(z|x)$ is a simplified variation to the true (and probably a more complex) distribution $p(z|x)$.


Seemingly good results with training a CNN but bad when testing I have an image classification task, and I am using Keras for a network with CNN layers, with what seems like good results in training, translates to poor performance in testing. Upon training, I quickly see an increase in accuracy and validation accuracy to around the following level:

4678/4678 [==============================] - 2s 427us/step - loss: 0.0607 - acc: 0.9795 - val_loss: 0.1605 - val_acc: 0.9590

Now my first reaction was that this might be too good to be true, turns it was. In testing the model, it had very bad performance.

Bashar Haddad suggests that there may be a data imbalance problem, but with the following data specifications of data, before being split into 25% validation, I don't think that's my problem:

(data is being fit consecutively in a for loop)
average number of datapoints in training: 4500
average number of datapoints in testing: 1500
average number of classes: 46 (max 49)

and most of the classes had a good amount of datapoints in each.

Based on this, I see the user found increasing batch size and decreasing learning rate helped, and I fit the model in the following way...

varying batch size from 100 -> 500
varying epochs size from 50 -> 250
cnn_model.fit(X_train, y_train,
                        batch_size=500,
                        epochs=50,
                        verbose=1,
                        shuffle=True,
                        validation_data=(X_test, y_test)
            )

Varying batch_size and epochs had little to no effect.

With both accuracy and validation accuracy high, and also very little difference between them while training, I am confused as to why using this model on test data results in such bad performance.

What could be reasons for this? And possible solutions to solve it?

You model might just overfit. This is a classical case with neural networks. It means that your model is very good on training data but performs poorly on testing data, i.e. it is very bad at generalizing...

You can monitor the performance on a validation set and use dropout to circumvent overfitting...

[EDIT] As @Jan van der Vegt said, overfitting on validation set is unlikely. Something similar happened to me in the past and it turned out that I was using a very small validation set. I'm still not sure why but I had very similar performance on both train and validation sets. As soon as I increased the size of the validation set, this behaviour disappeared. Although data leakage was indeed my first guess, I could not find any error in the way datasets were created. So my advice is: make sure that you don't have data leakage between training and validation set and make sure that your validation scores can be trusted by having a large enough validation size...


Does fine-tuning of transferred layers perform better than frozen transferred layers? I recently learned concepts of transfer learning. Is it necessarily true that fine-tuning of transferred layers perform better than frozen transferred layer? why?

Transfer learning means to apply the knowledge that some machine learning model holds (represented by its learned parameters) to a new (but in some way related) task. Whereas fine-tuning means taking some machine learning model that has already learned something before (i.e. been trained on some data) and then training that model (i.e. training it some more, possibly on different data).

From this we can conclude that if we are to use the learning of one model and concentrate only on some specific part of it, we can use transfer learning and train the network again with only a little amount of data. Whereas if we have sufficient data that we want the model to be trained upon, fine-tuning the model will come to the rescue. Refer to the previous question and this article


How could PageRank be used to rank paragraphs in relevance to keywords? I have a data set with keywords describing paragraphs in a car manual and the actual paragraphs. I want to rank those paragraphs by that keyword using PageRank algorithm.

How would I rank these paragraphs in relevance to the keywords?

The keywords are given an ID number while the paragraphs in the manual are texts. Let's say there is a keyword with ID number 89 and I have about 15 paragraphs that the keyword describes. How would the paragraphs be ranked in relevance to the keyword? Can I even use PageRank for this problem?

I understand for ranking websites, backlinks are used to sort the websites into ranks but for paragraphs of texts, how would I link and rank them?

If I understand, you have a set of paragraphs, and each paragraph is tagged with keywords. This defines a mapping from paragraph to keyword and vice-versa (an index and a reverse index). You can create a graph of paragraphs and keywords, but all of the edges are undirected (or they all have the same direction). Paragraphs only connect with keywords, and keywords only connect with paragraphs.

To establish relationships between paragraphs, you could define a transitive equivalence relation such as "all paragraphs that connect to the same keyword are now connected to each other". This would allow you to rank the paragraphs that contain a given keyword by their number of connections to other paragraphs, but it may not be an appropriate measure of relevance.

For PageRank to be most helpful, I think you need directional relationships (e.g. if "see page 123" was mentioned in a paragraph). To better represent the complexity of your data, creating additional relationships between keywords (e.g. semantic meaning, lexical relationships) and creating additional relationships between paragraphs (e.g. document classification) may help. Natural Language Processing with Python discusses these topics.


What is the meaning of spherical dataset? In the following article, one of the statement is as follows:

  The K-means algorithm is effective only for spherical datasets

What does spherical dataset mean?

In this case, a picture is a worth a thousand words. They literally mean data whose distribution on X,Y is roughly a sphere. Different clustering algorithms work better on different distributions. For example, K means does poorly on the arrangement in the first two rows but OK on the last row.


Microsoft custom vision vs Tensorflow model? I am planning to implement my own image classifier model using TensorFlow instead of a custom vision platform.
what is the biggest difference between custom vision(https://www.customvision.ai/) vs TensorFlow?

There are many differences as these are inherently complete different products with different goals.

customvision

[+] cloud deployment comes out of the box (including a rest API)
[+] labelling tool to add data and label them
[-] you have no control over the learning algorithm
[-] difficult to run your model locally/completely for free

tensorflow (or any framework really)

[-] you need deploy your model yourself
[-] you need to manage your data yourself
[+] you have full control over your network and how you train it
[+] you can embed your model into your code, run it locally, whatever you feel like


Using neural network regression for skewed data I'm using a feed forward network for a regression problem where the response variable is a ratio that can be negative and is very heavily skewed.

As the response can be negative, I can't just log transform the raw data. I've thought about doing log(1+x), rank(x) and standardize(rank(x)). What would be the best way to deal with this while preserving as much feature of the data as possible?

I faced this problem many times, these are the two solutions that worked best based on my experience:

The log(x+1) transformation as you proposed: you take log( x + 1 ). This is the log of the distribution. If your dependent variable has a lower bound (i.e. a minimum value it can possibly take) then I suggest you to use log( x + 1 ) + min( x ). In this way you constrain the distribution to take only positive values, and use ReLU at the output node
One that I tried recently at work and that worked very well is a scaling between 1 and the 95th percentile. This operation is analogous to the more classical min-max scaling, but instead calculating it using min(x) and max(x), you shring min(x) and 95th percentile. I said 95th percentile, but 99th or any other could also work well - you can choose the one that best fits your task. This become very useful when simple min-max scaling doesn't work (i.e. when shrinking everything in the 0-1 range would cause smaller observations to "disappear"). What you'll have in the end is a distribution with almost any observation in the 0-1 range, with very few outliers beyond that, with little influence on the overall quality of the prediction. This technique assume a ReLU activation at the output node. In this way you make sure to always get outputs in the desired, meaningful range.

I tried other techniques (one based on the upper part of Sigmoid function, and many other based on sequences of transformations with logs, square roots, standardization, and other monotonic functions), but they didn't perform better than the two I chose. Of course this is only based on personal experience.

PS:  These custom standardization cannot rely on methods such as sklearn's .fit() method. Remember to manually save the parameters in a specific file, otherwise you won't be able to revert your distribution back to the original!


GradientTape not computing gradient I understand that so long as i am defining a computation in tf.GradientTape() context, the gradient tape would compute the gradient w.r.t all the variables that the output of the computation depends on. However, i think i am not quite grasping the subtelties of the gradient as the following code does not execute as i expect it to:

import tensorflow as tf
x = tf.Variable(2.)
loss_ = x**2-2*x+1
with tf.GradientTape(persistent=True) as g:
    loss = loss_*1
print(g.gradient(loss,x))
output: None

Why is the gradient wrt x not computed?

I am able to compute only the gradients that are wrt to the variables that are getting explicitly used within the context. for example the following code does not compute gradients as well:

import tensorflow as tf
tf.compat.v1.disable_eager_execution()
x = tf.Variable(2.)
t1 = x**2
t2 = -2*x
t3 = 1.
with tf.GradientTape(persistent=True) as g:
    loss = t1+t2+t3
print(g.gradient(loss,x))

Since you are trying to compute $\partial loss \over \partial x$ you need to perform all operations that lead from $x$ to $loss$ inside GradientTape's scope, so that it can monitor them.

x = tf.Variable(2.)

with tf.GradientTape(persistent=True) as g:
    loss_ = x**2-2*x+1
    loss = loss_*1

print(g.gradient(loss,x))
# Will output: tf.Tensor(2.0, shape=(), dtype=float32)


Keras categorical_crossentropy loss (and accuracy) When training a neural network with keras for the categorical_crossentropy loss, how exactly is the loss defined? I expect it to be the average over all samples of
$$\textstyle\text{loss}(p^\text{true}, p^\text{predict}) = -\sum_i p_i^\text{true} \log p_i^\text{predict}$$
but couldn't find a definitive answer in the docs nor in the code. An authoritative reference is desirable.

Looking at the code I'm not sure if the computation is delegated to tensorflow/theano.

(There is an analogous question concerning the accuracy; the code is clearer, but I don't see a call to mean()?)

PS.
From this code, it appears that loss and accuracy are computed as in loss_and_acc(...) but before the last training epoch (keras version 2.0.4, same results for tensorflow and theano backend).

#!/usr/bin/python3

import numpy as np
from numpy.random import randint, seed

from keras import __version__ as keras_version
from keras.models import Sequential
from keras.layers import Dense

N = 4 # Classes
S = 10 # Samples

nn = Sequential()
nn.add(Dense(input_dim=1, units=N, kernel_initializer='normal', activation='softmax'))
nn.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

seed(7)
X = np.random.random((S, 1))
Y = np.vstack([np.eye(1, N, k=randint(0, N)) for _ in range(S)])
#for (x, y) in zip(X, Y) : print(x, y)

def loss_and_acc(NN, X, Y) :
    loss = []
    acc  = []
    for (p, q) in zip(Y, NN.predict(X)) :
        loss += [ -sum(a*np.log(b) for (a, b) in zip(p, q) if (b != 0)) ]
        acc  += [ np.argmax(p) == np.argmax(q) ]
    return (np.mean(loss), np.mean(acc))

print("Keras version: ", keras_version)

for _ in range(10) :
    print("Before:  loss = {}, acc = {}".format(*loss_and_acc(nn, X, Y)))
    H = nn.fit(X, Y, epochs=1, verbose=0).history
    print("History: loss = {}, acc = {}".format(H['loss'][-1], H['acc'][-1]))

Output:

Using Theano backend.
Keras version:  2.0.4
Before:  loss = 1.3843669414520263, acc = 0.2
History: loss = 1.3843669891357422, acc = 0.20000000298023224
Before:  loss = 1.3834303855895995, acc = 0.2
History: loss = 1.3834303617477417, acc = 0.20000000298023224
Before:  loss = 1.3824962615966796, acc = 0.3
History: loss = 1.3824962377548218, acc = 0.30000001192092896
Before:  loss = 1.381564486026764, acc = 0.3
History: loss = 1.3815644979476929, acc = 0.30000001192092896
Before:  loss = 1.380635154247284, acc = 0.3
History: loss = 1.380635142326355, acc = 0.30000001192092896
Before:  loss = 1.3797082901000977, acc = 0.3
History: loss = 1.3797082901000977, acc = 0.30000001192092896
Before:  loss = 1.378783941268921, acc = 0.2
History: loss = 1.378783941268921, acc = 0.20000000298023224
Before:  loss = 1.3778621554374695, acc = 0.2
History: loss = 1.3778622150421143, acc = 0.20000000298023224
Before:  loss = 1.3769428968429565, acc = 0.2
History: loss = 1.3769428730010986, acc = 0.20000000298023224
Before:  loss = 1.3760262489318849, acc = 0.3
History: loss = 1.3760262727737427, acc = 0.30000001192092896

I am using keras with tensorflow backend. I checked and the categorical_crossentropy loss in keras is defined as you have defined. This is the part of code (not the whole function definition)-

def categorical_crossentropy(target, output, from_logits=False, axis=-1):
    if not from_logits:
        # scale preds so that the class probas of each sample sum to 1
        output /= tf.reduce_sum(output, axis, True)
        # manual computation of crossentropy
        _epsilon = _to_tensor(epsilon(), output.dtype.base_dtype)
        output = tf.clip_by_value(output, _epsilon, 1. - _epsilon)
    return - tf.reduce_sum(target * tf.log(output), axis)

As you can see in last line, it is returning the sum of product of true values and log of output values for each observation. You can find complete function definition here in line 3176.

For theano backend, it should be same. You can check here in line 1622.


MSE loss different in Keras and PyToch My problem is that in PyTorch I cannot reproduce the MSE loss that I have achieved in Keras.

I have trained the following model in Keras:

from keras.models import Sequential
from keras.layers import Dense

model = Sequential()
model.add(Dense(10))
model.add(Dense(1))
model.compile(optimizer = "adam", loss = "mean_squared_error")
model.fit(X_train, y_train,
      batch_size = 32,
      epochs = 200
     )

The shape of the training data is:

print(X_train.shape)
>>>(3550, 10)
print(y_train.shape)
>>>(3550,)

After training the MSE is ~0.15:

mse_train = model.evaluate(X_train, y_train)
>>>3550/3550 [==============================] - 0s 18us/step
print("Train MSE: ", mse_train)
>>>Train MSE:  0.1499910642017781

Then I initialize the same model in PyTorch:

import torch
import torch.nn as nn

class NN(nn.Module):
 def __init__(self):
  super(NN, self).__init__()
  self.dense1 = nn.Linear(10, 10)
  self.dense2 = nn.Linear(10, 1)

 def forward(self, x):
  out = self.dense1(x)
  out = self.dense2(out)
  return out

net = NN()
criterion = nn.MSELoss()

And assign the weights I have achieved in Keras:

from keras.models import load_model
keras_model = load_model(MODEL_PATH)

dense_weights = keras_model.layers[0].get_weights()
weights = torch.tensor(dense_weights[0].swapaxes(0,1))
bias = torch.tensor(dense_weights[1])
net.dense1.weight.data = weights
net.dense1.bias.data = bias

dense_weights = keras_model.layers[1].get_weights()
weights = torch.tensor(dense_weights[0].swapaxes(0,1))
bias = torch.tensor(dense_weights[1])
net.dense2.weight.data = weights
net.dense2.bias.data = bias

Now I try to calculate the MSE loss:

X_train_torch = torch.tensor(X_train, dtype=torch.float)
y_train_torch = torch.tensor(y_train, dtype=torch.float)

outputs = net(X_train_torch)
loss = criterion(outputs, y_train_torch)
print("Train loss: ", loss)
>>>Train loss:  0.338391376896338

The MSE is now ~0.34 and is twice as big as calculated in Keras.

What could be the reason? Is there a bug in my calculation?

The problem was that outputs and y_train_torch had different shapes.


How to plot cost versus number of iterations in scikit learn? One of the recommendations in the Coursera Machine Learning course when working with gradient descent based algorithms is:

  Debugging gradient descent. Make a plot with number of iterations on the x-axis. Now plot the cost function, J(θ) over the number of iterations of gradient descent. If J(θ) ever increases, then you probably need to decrease α.

Do gradient descent based models in scikit-learn provide a mechanism for retrieving the cost vs the number of iterations?

Note The Code Is Self Explanatory(It's Hard-coded)..

  Here's the function which we are considering

def f(a,b):
    return a**2 + b**2

fig = plt.figure(figsize=(10, 6))
ax = fig.gca(projection='3d')
plt.hold(True)
a = np.arange(-2, 2, 0.25)
b = np.arange(-2, 2, 0.25)
a, b = np.meshgrid(a, b)
c = f(a,b)
surf = ax.plot_surface(a, b, c, rstride=1, cstride=1, alpha=0.3,
                       linewidth=0, antialiased=False,cmap='rainbow')
ax.set_zlim(-0.01, 8.01)

  Here's a 3D view of Gradient Descent Reaching The Optimum(in case interested, it will not always work, Have a look at the end plot..)

def gradient_descent(theta0, iters, alpha):
    history = [theta0] # to store all thetas
    theta = theta0     # initial values for thetas
    # main loop by iterations:
    for i in range(iters):
        # gradient is [2x, 2y]:
        gradient = [2.0*x for x in theta]
        # update parameters:
        theta = [a - alpha*b for a,b in zip(theta, gradient)]
        history.append(theta)
    return history

history = gradient_descent(theta0 = [-1.8, 1.6], iters = 30, alpha = 0.03)

fig = plt.figure(figsize=(10, 6))
ax = fig.gca(projection='3d')
plt.hold(True)
a = np.arange(-2, 2, 0.25)
b = np.arange(-2, 2, 0.25)
a, b = np.meshgrid(a, b)
c = f(a,b)
surf = ax.plot_surface(a, b, c, rstride=1, cstride=1, alpha=0.3,
                       linewidth=0, antialiased=False)
ax.set_zlim(-0.01, 8.01)

a = np.array([x[0] for x in history])
b = np.array([x[1] for x in history])
c = f(a,b)
ax.scatter(a, b, c, color="r");

plt.show()

And Here's the output we will get

  When Gradient Descent will fail (unfortunately)..

Unfortunately, if the function has many extrema, then the Gradient Descent could find the local minimum instead of global minimum. One trick is to overcome this disadvantage is to run SGD several times with different initial guessed values for $x$.


Decision tree or rule What criteria can be used to decide whether to develop a decision tree or one or more rules using oneR or jRip(contained in RWeka). The similarities between these two algorithms (decision tree and rule) is very high making it a little confusing.

JRip implements a propositional rule learner, “Repeated Incremental Pruning to Produce Error Reduction” (RIPPER), as proposed by Cohen (1995) and OneR builds a simple 1-R classifier, proposed by Holte (1993).

Its hard to say which algorithm works better. Best approach is to compare different classification algorithms performance in terms of precision, recall, accuracy, f1 score, AUC, specificity and sensitivity on your train/test data set and pick the one that gives best results or use ensemble of top performing algorithms to build your final model.

There is a good white paper doing similar exercise of comparing different classification algorithms including OneR and JRIP here.

Hope this helps.


Feature transformation possible at selected features or only at all? I want to cluster. I have different features for that. Some features have a very small value range (from 0 to 0.8) and some have a very large value range (from 0 to 5 million). I want to use the logarithm. Do I have to apply this feature transformation to all features, including the small values (shift the values before by +1) or can I just choose the features I want to do the transformation on? Is there any literature or work on this?

Either can be good or bad. It's not necessary to apply the same transformation to all attributes; but if youe attributes differ that much, you likely have some major problem anyway.

It depends on what you want to achieve what is appropriate. Try to approach it from having to explain your choices later. Can you argue why you transform the data this way? What is the statistical meaning of this transform for your problem? Here you'll quickly realize that with mixed data you can't really argue much - your results are based on guessing, not in a sound model.


Avoiding the zero problem I have a dataset and I'm trying to predict the label for my sample but I couldn't map it since that case never showed here is my sample (I'm using naïve Bayesian method)

X=(Age=middle, has_job=false, own_house=true, credit_rating=good)

and that's it the dataset

What I'm supposed to do to fix the problem?
I know that I should  avoid it but didn't know how

I totally agree with Esmailian.

Naive Bayes is Naive - Assumes Independence.

Steps:

Calculate Independently
Smooth using Laplacian smoothing (to avoid zeroing the whole value)

Additional Tip:

Use log instead of multiplying the probabilities. (this will make sure that your values are not closing to zero, keeping some context.)

Example:

$$p(a) = p(x1)\cdot p(x2)$$

Applying logarithm,

$$\log(p(a)) = \log(p(x_1)) + \log(p(x_2))$$

As you need to finally classify, taking the log won't hurt.


Composing phrases into a grammatically correct sentence? I'm wondering if there exist any models which could take in an ordered list of phrases without punctuation and generate a grammatically correct sentence from it.

For example, for the input: ["My dog", "hates that", "guy named Michael Vick"]

The output could be either "My dog hates that guy named Michael Vick!", "My dog hates that guy named Michael Vick.", "My dog hates that guy named Michael Vick...", or even "My dog hates that guy named... Michael Vick.". Ideally, the probability of the sentence would be returned alongside the output.

Is there a specific name for this problem, so I can do a literature search myself?

Thanks!

There are few modules implemented using TensorFlow which intended to do the same.
Example:
https://github.com/atpaino/deep-text-corrector

This deep-learning models trained to check spellings and correct small grammatical errors.

Thanks!


Decision trees: leaf-wise (best-first) and level-wise tree traverse Issue 1:

I am confused by the description of LightGBM regarding the way the tree is expanded.

They state:

  Most decision tree learning algorithms grow tree by level
  (depth)-wise, like the following image:

Questions 1: Which "most" algorithms are implemented this way?
As far as I know C4.5 and CART use DFS. XGBoost uses BFS. Which other algorithms or packages use BFS for decision trees?

Issue 2:

LightGBM states:

  LightGBM grows tree by leaf-wise (best-first).It will choose the leaf with max delta loss to grow. When growing same
  leaf, leaf-wise algorithm can reduce more loss than level-wise algorithm.

Question 2: Is it correct to say that level-wise growth trees will have equal depth for all leaves?

Questions 3: If Question 2 is not correct, then the trees from level-wise and leaf-wise growth will look the same at the end of the traversal (without pruning etc). Is it a correct statement?

Questions 4: If question 3 is correct, how can "leaf-wise algorithm can reduce more loss than level-wise algorithm"? Does it have to do with the post-pruning algorithm?

If you grow the full tree, best-first (leaf-wise) and depth-first (level-wise) will result in the same tree. The difference is in the order in which the tree is expanded. Since we don't normally grow trees to their full depth, order matters: application of early stopping criteria and pruning methods can result in very different trees. Because leaf-wise chooses splits based on their contribution to the global loss and not just the loss along a particular branch, it often (not always) will learn lower-error trees "faster" than level-wise. I.e. for a small number of nodes, leaf-wise will probably out-perform level-wise. As you add more nodes, without stopping or pruning they will converge to the same performance because they will literally build the same tree eventually.
Reference:
Shi, H. (2007). Best-first Decision Tree Learning (Thesis, Master of Science (MSc)). The University of Waikato, Hamilton, New Zealand. Retrieved from https://hdl.handle.net/10289/2317

EDIT: Regarding your first question, both C4.5 and CART are depth-first examples, not best-first. Here's some relevant content from the reference above:

1.2.1 Standard decision trees
Standard algorithms such as C4.5 (Quinlan, 1993) and CART (Breiman et al., 1984) for the top-down
induction of decision trees expand nodes in depth-first order in each
step using the divide-and-conquer strategy. Normally, at each node of
a decision tree, testing only involves a single attribute and the
attribute value is compared to a constant. The basic idea of standard
decision trees is that, first, select an attribute to place at the
root node and make some branches for this attribute based on some
criteria (e.g. information or Gini index). Then, split training
instances into subsets, one for each branch extending from the root
node. The number of subsets is the same as the number of branches.
Then, this step is repeated for a chosen branch, using only those
instances that actually reach it. A fixed order is used to expand
nodes (normally, left to right). If at any time all instances at a
node have the same class label, which is known as a pure node,
splitting stops and the node is made into a terminal node. This
construction process continues until all nodes are pure. It is then
followed by a pruning process to reduce overfittings (see Section
1.3).
1.2.2 Best-first decision trees
Another possibility, which so far appears to only have been evaluated in the context of boosting
algorithms (Friedman et al., 2000), is to expand nodes in best-first
order instead of a fixed order. This method adds the ”best” split node
to the tree in each step. The ”best” node is the node that maximally
reduces impurity among all nodes available for splitting (i.e. not
labelled as terminal nodes). Although this results in the same
fully-grown tree as standard depth-first expansion, it enables us to
investigate new tree pruning methods that use cross-validation to
select the number of expansions. Both pre-pruning and post-pruning can
be performed in this way, which enables a fair comparison between them
(see Section 1.3).
Best-first decision trees are constructed in a divide-and-conquer
fashion similar to standard depth-first decision trees. The basic idea
of how a best-first tree is built is as follows. First, select an
attribute to place at the root node and make some branches for this
attribute based on some criteria. Then, split training instances into
subsets, one for each branch extending from the root node. In this
thesis only binary decision trees are considered and thus the number
of branches is exactly two. Then, this step is repeated for a chosen
branch, using only those instances that actually reach it. In each
step we choose the ”best” subset among all subsets that are available
for expansions. This constructing process continues until all nodes
are pure or a specific number of expansions is reached. Figure 1.1
shows the difference in split order between a hypothetical binary
best-first tree and a hypothetical binary depth-first tree. Note that
other orderings may be chosen for the best-first tree while the order
is always the same in the depth-first case.


Object detection stops predicting well according to how I collect the images On my problem I have to use 4k resolution (3840x2160) to record images and detect qrcodes at the same time. The images I end resizing it to 1920x1080 to save it on disk.
The problem is that if I train using this resized data the model stops predicting well compared to when I record the video on native 1920x1080.
I can't have good results nor if I predict with resized to 1920x1080 neither native 1920x1080 with this train.
If I train using native recorded at 1920x1080 it predicts well both with native 1920x1080 and resized 1920x1080.
Is there some explanation about why this is occurring?
I always try to save the images as jps with quality=95 and the camera I'm using compresses the image before sending to opencv (directly affects the fps).
I can't go back to 1920x1080 because the qrcode reader stops detecting.
How I record the video without resize (working method):
for z, frame in enumerate(frames):
    try:
        self.out.write(cv2.cvtColor(frame2, cv2.COLOR_BGR2RGB))
    except AttributeError:
        self.out = cv2.VideoWriter(ANNOTATION_RECORD_VIDEO_FILE, cv2.VideoWriter_fourcc('m','p','4','v'), 60, (conf.cameras.phase1[0].resolution.width, conf.cameras.phase1[0].resolution.height))
        self.out.write(cv2.cvtColor(frame2, cv2.COLOR_BGR2RGB))

With resize (frames are coming with 4k res):
for z, frame in enumerate(frames):
    frame = cv2.resize(frame, dsize=(conf.annotation.resolution_to_save['width'],
                                        conf.annotation.resolution_to_save['height']),
                                interpolation=cv2.INTER_CUBIC)

    try:
        self.out.write(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))
    except AttributeError:
        self.out = cv2.VideoWriter(ANNOTATION_RECORD_VIDEO_FILE, cv2.VideoWriter_fourcc('m','p','4','v'), 60, (conf.annotation.resolution_to_save['width'], conf.annotation.resolution_to_save['height']))
        self.out.write(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))

On my problem I have to use 4k resolution (3840x2160) to record images
and detect qrcodes at the same time. The images I end resizing it to
1920x1080 to save it on disk.

Have you tried using a HAAR Cascade classifier to crop out the region with the QR code so that you'll have to deal with only an image at lower dimensions. I guess that will help you with the frame rate.

The problem is that if I train using this resized data the model stops
predicting well compared to when I record the video on native
1920x1080. I can't have good results nor if I predict with resized to
1920x1080 neither native 1920x1080 with this train. If I train using
native recorded at 1920x1080 it predicts well both with native
1920x1080 and resized 1920x1080.

Im not entirely sure what you've meant by this. A bit more clarity would be nice !

Is there some explanation about why this is occurring?

If youre training on the resized 4k image and testing on the 1920x1080 image its natural that this happens. because the sub sampling removes features. It's better to look at a few resizing algorithms and write your own resizer. because qr codes have intricate details that can be missed while sub sampling

I always try to save the images as jps with quality=95 and the camera
I'm using compresses the image before sending to opencv (directly
affects the fps).

Instead of this why can't it be recorded in 1920x1080 and supersampled using interpolations. This would not be as good as the 4k but it will be close. And then you can sharpen the image until it's at par with 4k


NLP algorithm: sentiment with specific guidelines So I have this situation, I have filtered a bunch of single independent sentences that I filtered because they contain the word X (in my case, X = "budget"). if the meaning of the sentence is "budget increases/goes up/etc" my result is 1 and, if not, 0. all sentences hold such meaning (the word budget is not used in different ways)

this is not a traditional "sentiment" problem and I do not know how to approach this. I could run into a variety of cases ("budget did not go down" // "A budget cut was in order" // "A budget cut was not in order")

if required, I can categorize by hand up to 400/500 sentences and say with certainty the result (0, 1). could you point me to any model that would solve this? would the amount of hand-labeled samples be enough?

This seems like a straight forward classification problem: "Increased", "Decreased" (I assume "Neither Increased nor Decreased" can be a third category as well). Something simple like a Support Vector Machine (SVM) could work.

That being said, only 400-500 example cases aren't a lot so you might run into overfitting especially if you try to run whatever model you land on later down the road.


Creating an LSTM NN for Fault Classification with Keras Goal: Classify time series data from a wind turbine as being anomalous or
non-anomalus in real time and from there predict what the anomaly is in a later, more refined, model.

I have a CSV file with 6 columns corresponding to 6 sensors at uniform measurement. I know I need to separate these into separate vectors to vectors/tensors to run them through a 6 input NN.

My question is how do I create an LSTM that classifies data instead of predicting a value? Using a sigmoid?

It is very simple.

Just add a Dense layer (Keras-wise) with one unit after your LSTM network, with sigmoid activation. If you don't use Keras, Dense layer is simply a fully-connected neuron with one unit (in your case), which you can easily implement in the deep learning framework of your preference.

Then you can train your network to directly classify the input sequence using soft-labeling, so that it outputs a probability from 0 to 1 that a sequence is anomalous. Needless to say, you can have an arbitrary number of inputs, not just one.

Bottom line: use a fully-connected layer with one unit (neuron) and sigmoid activation after the LSTM output