Need recommandations of using timing variables for forecasting sales I have a data.table that contains many timing variables.

Date: it gives the date of Sales
Promo2(week, year): It describes the calendar week and the year when the store has started to participate in the Promo2 (reduction of prices).
CompetitionOpenSince(Month, Year): Gives the approximate year and
the month of the moment when the nearest competitor was opened.
PromoInterval: Describes the consecutive intervals Promo2 is launched,
naming the months when the promotion was launched again. For example.
"Feb, May, Aug, Nov" means that each round starts in February, May,
August, November of a given year for this store.

The Goal is to predict sales for next few week. I have other factor and binary variables such as the type of store(a b,c..), the level of Assortment, CompetitionDistance, number of Customers...

In my work I suggest some variable that uses timing variables above:

Compute the number of days between the day of sales and the first
day when the store started the promo2(Day-Promo2).
Compute the number of days between the day of Promo2 and the day
   when the competitor opened the promo2(Promo2-CompetitionOpenSince).

Note: Dont surprised when I wrote (Day-Promo2) and (Promo2-CompetitionOpenSince) because I transform Promo2 and CompetitionOpenSince to date type using some hypothesis to simplify the work.

What can you suggest more than that? I find it really hard to create new other useful explaining variables mostly the  PromoInterval variable!

Thank you for your suggestions!

Generally it works well to include some lagged variables.

For example:

Sales last week
Sales a month ago
Inventory a month ago
etc


can we have different features for different hyperplanes in SVM? is it possible if we have different features for different classes of svm?

For example one of the hyperplane:

$$w_1\cdot \text{age}+ w_2 \cdot \text{ trip duration} +w_3 \cdot \text{ income}$$

and the other hyperplane be

$$w_1\cdot \text{cost}+ w_2 \cdot \text{trip duration} +w_3 \cdot \text{purpose of trip}$$

the other one

$$w_1\cdot \text{distance}+ w_2 \cdot \text{trip duration} +w_3 \cdot 0?$$

Does it make sense?

It is possible.

All of these hyperplanes live in the space of

$$\text{(age, trip duration, income, cost, purpose of trip, distace)}$$

even though some of the coefficients are $0$.

Remark: For the last hyperplane $w_3\cdot 0 = 0$.


Trying to change pandas column dtype from str to float I am trying to convert a pandas column from a str to a float. Before I convert the strings to floats using astype(float), I need to remove characters from the string which cannot be converted into floats, like a comma or a space. In trying to remove these characters, I am using list compression and the str.replace method like this:

data.loc[:,'column_name'] = [char.replace(',', '') for char in
                                data.loc[:,'column_name']]

but I am getting an error saying:

AttributeError: 'float' object has no attribute 'replace'.

The confusing part, though, is that when I look at the dtypes, the dtype for that column shows up as 'o', or object. So I have no idea what is going on.

In pandas the object type is used when there is not a clear distinction between the types stored in the column.

So, I guess that in your column, some objects are float type and some objects are str type. Or maybe, you are also dealing with NaN objects, NaN objects are float objects.

a) Convert the column to string: Are you getting your DataFrame from a CSV or XLS format file? Then at the moment of reading the file, you can specify that that column is an str type or just make the type conversion of the column you are dealing with.

b) After that, you can apply the string changes and/or deal with the NaN objects.

c) Finally, you transform your column into float type`.


Binary Neural Network Classification or Multiclass Neural Network Classification? I am confused about the difference between a binary and multiclass neural network classification. If I am writing an algorithm that has 2 output classes (Obama or Romney), but not yes or no (so not like Obama or not Obama), then is it a binary neural network or a multi class (2 class) neural network classification?

What I do know:
A binary neural network classification outputs 1 unit. If a multi class classification neural network has k classes, then it has k outputs.

What I think:

I think it is a binary neural network classification because I am really only trying to output one thing, whether a county votes for Romney or Obama. I am confused because I thought binary was more like Romney or not Romney classification and I am not sure if that is the same as Romney or Obama. Just wanted to double check and clarify my understanding.

I think you are making things more confusing then they are.

Binary

In this case you have two possible outputs:

Obama = 1.
Not-Obama (who in this case can only be Romney) = 0.

Multi-Class

In this case you have k possible outputs, for example when k = 4:

k = 0: Obama
k = 1: Romney
k = 2: Clinton
k = 3: Bush

There are approaches to tackle multi-class classification as binary classification which are called One-vs-rest classification and One-vs-one classification, other classifiers, such as Random Forests, are able to deal with a multiclass setting in a natural way. For a brief report, see here.


One approach would be to fit a linear regression and look at Cooks Distance to detect outliers.
https://en.m.wikipedia.org/wiki/Cook%27s_distance

Some R background: http://r-statistics.co/Outlier-Treatment-With-R.html

Some Python discussion: https://stackoverflow.com/a/52322232/9524424


What are R's memory constraints? In reviewing “Applied Predictive Modeling" a reviewer states:

  One critique I have of statistical learning (SL) pedagogy is the
  absence of computation performance considerations in the evaluation of
  different modeling techniques. With its emphases on bootstrapping and
  cross-validation to tune/test models, SL is quite compute-intensive.
  Add to that the re-sampling that's embedded in techniques like bagging
  and boosting, and you have the specter of computation hell for
  supervised learning of large data sets. In fact, R's memory
  constraints impose pretty severe limits on the size of models that can
  be fit by top-performing methods like random forests. Though SL does a
  good job calibrating model performance against small data sets, it'd
  sure be nice to understand performance versus computational cost for
  larger data.

What are R's memory constraints, and do they impose severe limits on the size of models that can be fit by top-performing methods like random forests?

As Konstantin has pointed, R performs all its computation in the system's memory i.e. RAM. Hence, RAM capacity is a very important constraint for computation intensive operations in R. Overcoming this constraint, data is being stored these days in HDFS systems, where data isn't loaded onto memory and program is run instead, program goes to the data and performs the operations, thus overcoming the memory constraints.  RHadoop (https://github.com/RevolutionAnalytics/RHadoop/wiki) is the connector you are looking for.

Coming to the impact on algorithms which are computation intensive, Random Forests/Decision Trees/Ensemble methods on a considerable amount of data (minimum 50,000 observations in my experience) take up a lot of memory and are considerably slow. To speed up the process, parallelization is the way to go and parallelization is inherently available in Hadoop! That's where, Hadoop is really efficient.

So, if you are going for ensemble methods which are compute intensive and are slow, you would want to try out on the HDFS system which gives a considerable performance improvement.


Can you explain the difference between SVC and LinearSVC in scikit-learn? I've recently started learning to work with sklearn and have just come across this peculiar result.

I used the digits dataset available in sklearn to try different models and estimation methods.

When I tested a Support Vector Machine model on the data, I found out there are two different classes in sklearn for SVM classification: SVC and LinearSVC, where the former uses one-against-one approach and the other uses one-against-rest approach.

I didn't know what effect that could have on the results, so I tried both. I did a Monte Carlo-style estimation where I ran both models 500 times, each time splitting the sample randomly into 60% training and 40% test and calculating the error of the prediction on the test set.

The regular SVC estimator produced the following histogram of errors:

While the linear SVC estimator produced the following histogram:

What could account for such a stark difference? Why does the linear model have such higher accuracy most of the time?

And, relatedly, what could be causing the stark polarization in the results? Either an accuracy close to 1 or an accuracy close to 0, nothing in between.

For comparison, a decision tree classification produced a much more normally distributed error rate with an accuracy of around .85.

A regular SVM with default values uses a radial basis function as the SVM kernel.  This is basically a Gaussian kernel aka bell-curve.  Meaning that the no man's land between different classes is created with a Gaussian function.  The linear-SVM uses a linear kernel for the basis function, so you can think of this as a ^ shaped function.  It is much less tunable and is basically just a linear interpolation.

People are kind of hammering this question because you didn't provide a lot of information, but looking deeply into what you posted... this hits on some fundamental aspects of really understanding the details of bias and variance and the difference between linear and nonlinear basis functions in SVM.

Check out this image describing the four regions of high and low bias and high and low variance.  Obviously the best place to be is  low variance and low bias.

First lets assess variance -

Now take a look at your plots:

The nonlinear basis function has higher variance.  See how it is noisier than the linear kernel!  The linear kernel has lower variance.  See how it is less noisy!

Now lets assess bias -

Which kernel is more accurate?  We can add the errors that you provided.  The nonlinear kernel has a total error of ~550+325=~875.  The linear kernel has an error of ~690+~50=~740.  So the linear kernel seems to do better overall, but they are pretty close overall.  This is were things get tricky!

Putting it all together

See how the linear kernel did a poor job on 1's and a really great job on 0's.  This is pretty unbalanced.  Where as the nonlinear kernel is more balanced.  It kind of seems like the sweet spot might be to create a balanced model that doesn't have such high variance.  How do we control for high variance?  Bingo - regularization.  We can add regularization to the nonlinear model and we will probably see much better results.  This is the C parameter in scikit learn SVMs, which you will want to increase from the default.  We could also play with the gamma parameter.  Gamma controlls the width of the Gaussian.  Maybe try increasing that one slightly to get less noisy results e.g. a larger no-man's land between classes.

Hope this helps!


What is deconvolution operation used in Fully Convolutional Neural Networks? When I was reading this this paper, Fully Convolutional Networks for Semantic Segmentation, I found that they use an up-sampling layer to classify each pixel in to a class. I have two questions:

How do you understand the mathematics behind the de-convolution operation?
Why do we use an upsampling layer? Is it for extract more global context?

Upsampling layer is used to increase the resolution of the image. In segmentation, we first downsample the image to get the features and then upsample the image to generate the segments.

For deconvolution operation we pad the image with zeroes and then do a convolution operation on that, hence it is upsampled.

For eg: - If after downsampling the images becomes:

[[1, 1]
[1, 1]]

Then if we pad it with zeroes,

[[0, 0, 0, 0, 0, 0]
 [0, 0, 0, 0, 0, 0]
 [0, 0, 1, 1, 0, 0]
 [0, 0, 1, 1, 0, 0]
 [0, 0, 0, 0, 0, 0]
 [0, 0, 0, 0, 0, 0]]

Now, if we do convolution on the above image with a 3 x 3 filter, we will get an image of shape (4, 4). Thus we upsampled an image of shape (2, 2) to (4, 4).


This was a x-post. There is a great solution over here, where I originally asked the question.

  If I understand your problem, you'll want something like this:

bottom <- seq(1750, 2010, 5)
library(dplyr)
new_df <- mlt.mx.info %>%
  arrange(Year) %>%
  mutate(year2 = as.numeric(substr(Year, 6, 9))) %>%
  mutate(new_year = paste0(bottom[findInterval(year2, bottom)], "-",(bottom[findInterval(year2, bottom) + 1] - 1)))
View(new_df)

  So what this does, it creates bins, and outputs a new column
  (new_year) that is the bottom of the bin. So everything from 1750-1754
  will correspond to a new value of 1750-1754 (in string form; the
  original is an integer type, not sure how to fix that). Does this do
  what you want? Double check the results, but it looks right to me.
  @goodtimeslim


Backpropagation Through Time (BPTT) of LSTM I am currently trying to understand the BPTT for Long Short Term Memory (LSTM) in TensorFlow. I get that the parameter num_steps is used for the range that the RNN is rolled out and the Error backpropagated. I got a general question of how this works.

For reference a repitition of the formulas. I'm referring to:
Formulas LSTM (https://arxiv.org/abs/1506.00019)

Question:
What paths are backpropagated that many steps? The constant error carousel is created by the formula 5, and the derivate for backpropagation ($s(t)\rightarrow s(t-1)$) is $1$ for all timesteps. This is why LSTMs capture long range dependencies. I get confused with the dependencies of $g(t)$, $i(t)$, $f(t)$ and $o(t)$ of $h(t-1)$. In words: The current gates do not just depend on the input, but also on the last hidden state.

Doesn't this dependency lead to the exploding/vanishing gradients problem again?

If I backpropagate along these connections I get gradients that are not one. Peephole connections essentially lead to the same problem.

Thanks for your help!

Doesn't this dependency lead to the exploding/vanishing gradients problem again?

Absolutely, and you better have vanishing gradients otherwise you have a training problem.

Vanishing gradient in this case is not a bad thing, it is a good thing (unlike in feedforward). Let $C(t)$ be the cost function evaluated at time $t$ and $ W(t)$ be some weight of the network at time $t$. What vanishing gradient in this case means is that $ dC(t)/dW(t-u)$ becomes smaller and smaller as u becomes bigger and bigger. That is good because $ dC(t)/dW =  \sum_{u=0}^{num\_steps} dC(t)/dW(t-u) $, so if the gradients didn't vanish in time, then the gradients would explode. So $W$ gets a proper non-vanishing gradient even if the gradients in time vanish because the gradient for $W$ is the sum at all times of the gradient for $W$.

In LSTMs the gradients are sure to vanish in time because the activation functions are sigmoids and tanh's so their derivatives are less than or equal to one, so as they get multiplied they slowly become smaller.

This compares to what is normally called the vanishing gradient problem which occurs when gradients vanish while passing from top layers to bottom layers, because that means that $ dC/dW $ for $W $ of the lower layer is vanishing and so the lower layers don't get trained, only the upper layers get trained.

Also, as mentioned in the comments, the above applies to any RNN, not only LSTMs. What sets LSTMs appart from vanilla RNNs in with regards to this question is the gating functions which allows the LSTM to control what it remembers and what it forgets and how much of the new input it takes in. While the above is true in practice for LSTMs (and is true on average also in theory), in theory, one could have a time step $t$ where the output has ignored the last 10 inputs and only depends on the input 11 timestep back ($t-11$), in which case the gradient for the weights 11 timesteps ago will not have decayed. Of course that means that at the next time step ($t+1$) the gradients for 11 steps ago ($t+1 -11 = t-10$) will be zero because the input was totally disregarded at $t-10$. So on average it averages out and you still have the same situation for LSTMs.


Change values of consecutive repeating letters in pandas column I've got one Dataframe like:

id value block
1 a 1
2 a 1
3 a 2
4 a 2
5 b 3
6 c 4

And I want to change the value column to the next value based where
the series changes. Like below. The change must be defined by the
block column.

id value block
1 a 1
2 a 1
3 b 2
4 b 2
5 c 3
6 None 4

I thought about using shift but there are different lengths of continuity in the value column.

I think this should work somehow by using cumsum to detect the change.

For this particular case in which difference between consecutive block elements is 0 or unitary, you could use diff() and something like this:

import pandas as pd, numpy as np
df = pd.DataFrame({'id': [1, 2, 3, 4, 5, 6], 'block': [1, 1, 2, 2, 3, 4]})
values = ['a', 'b', 'c', 'None']
indexs = np.insert(np.cumsum(np.diff(df.block)), 0, 0)
df['value'] = [values[np.asscalar(x)] for x in indexs]

When value is already part of your dataframe, you could simply replace the values list given above by the following:

df = pd.DataFrame({'id': [1, 2, 3, 4, 5, 6], 'value': ['a', 'a', 'a', 'a', 'b', 'c'], 'block': [1, 1, 2, 2, 3, 4]})
values = np.append(df['value'].unique(), 'None')
indexs = np.insert(np.cumsum(np.diff(df.block)), 0, 0) # unchanged
df['value'] = [values[np.asscalar(x)] for x in indexs] # unchanged

which conducts to the same result.


On assiging weights for unbalanced classes Consider a dataset that will be split into train and test. The model will be learned using the train set and evaluated using the unseen test set. Now the dataset is unbalanced -- it contains more examples belonging to a particular kind of class. In that case, one of the ways to balance it (apart form the ones mentioned here: Train, test split of unbalanced dataset classification) is by assigning weights based on the samples. What is the proper way to assign weights? Should I assign weights on the full dataset and then split into train and test?

What is the proper way to assign weights? Should I assign weights on the full dataset and then split into train and test?

No, your test set doesn't have to be weighted and shouldn't be, because it should reflect the real distribution of the data. So you should split first and weight only the instances in the training set, since this is during the training stage that these weights are taken into account.


Sparse Data Interpolation I have a price data set where on some days there are up to five data points and some days none at all. For example:

2.110   2017-04-19
1.910   2017-04-23
1.980   2017-04-24
1.980   2017-04-24
1.980   2017-04-24
1.230   2017-04-24
2.100   2017-05-04
1.920   2017-05-08

The total data set is around 500 points.

What is the best way to go about interpolating this data and making it's frequency daily?

I'm considering using Gaussian Process Modelling using PyMC3 but would really appreciate any ideas or advice. Additionally, the volatility of the price data must be preserved as best as possible as this will be input to a volatility model later on.

If you know that the multiple prices on a single day are chronological (and you don't have the actual timestamps), I would suggest simply taking the last price. This is then as close as you can get to using the Closing Price, which is the most common. Often you have Open, High, Low, Close, for example, but just use the Close prices.

Interpolating, or imputing, the data can be done in many ways. One factor that might help decide on a method, is how many of the points are missing? You have ~500 data points: how many days in total? how many of those require imputing?

For a relatively low number of missing values, scattered more or less randomly over your time-series, I would suggest going for simpler imputation methods, rather than model-based one.

A model-based approach can introduce other biases/problems that would potentially be harder to debug and more difficult to understand, so I would hold off there initially.

Forward-filling

This is my go to method, to just get started. If only a few percent of the entire time-series are missing, something as simple as fill-forward might be acceptable. Even the gaps in data are randomly or evenly spaced along the timeline, this is also a reasonable approach.

A nice property of this method is that it agrees with the theory on stochastic price-paths. These are usually modelled as Gaussian random variables, following the principles or Brownian motion. This all culminates in the idea of a Martingale Process, which states:

  at a particular time in the realized sequence, the expectation of the next value in the sequence is equal to the present observed value even given knowledge of all prior observed values.

Random sampling

One way to fill a seemingly stochastic price-path, would be to compute the mean and variance of the dataset, then draw random samples from that distribution to fill missing values. This should preserve the mean and variance of the dataset's distribution quite well (and fairly).

Here is is important to realise that you may inadvertently be inducing a bias within that dataset, namely that you have use population statistics to alter the data. This means, depending on your model testing method later on, that you violate information flow through time. E.g. when predicting the 100th step, using the 99 preceding steps, it could be the case that the 99th step was one you imputed. Using the mean and variance of the entire dataset to do so has inherently encoded some information from the 100th step (and all following steps) into that 99th step. This is a subtle detail, but something to be aware of.

If you can put the time-series in a Pandas DataFrame, it might give you some more ideas by skimming the DataFrame's built-in methods to handle missing data.

Here is a more in-depth analysis of possibilities with some good explanations.


Feature agglomeration: Is it testing interactions? I have been looking at feature agglomeration in Python's scikit-learn. According to the user guide, feature agglomeration "applies Hierarchical clustering to group together features that behave similarly". Does this mean it is testing for interactions between features and groups features that do interact together? What does "behave similarly" mean in this context?

From the documentation:

  Similar to AgglomerativeClustering, but recursively merges features instead of samples.

In standard agglomerative clustering you receive a matrix $M^{n \times m}$ representing $n$ samples of dimension $m$ that you want to cluster. In feature agglomeration the algorithm clusters the transpose of the matrix, i.e. $M^T$ so it clusters $m$ samples of dimension $n$, these samples represent the features.

The default distance used to cluster the features (samples in the transpose matrix) is the euclidean distance, but you can also use l1, cosine and others.

For example suppose you have 3 samples of dimension 3 (a matrix 3x3 matrix)::

----------------------------------
| feature1 | feature2 | feature3 |
----------------------------------
|     1    |   1.05   |    10    |
----------------------------------
|     1    |   1.05   |    10    |
----------------------------------
|     2    |   2.05   |    20    |
----------------------------------

If you want to reduce the dimension of your dataset to 2 dimensions, the algorithm clusters together feature1 and feature2, and leaves feature3 unchanged, the new matrix is:

----------------------------------
| feature1 - feature2 | feature3 |
----------------------------------
|         1.025       |    10    |
----------------------------------
|         1.025       |    10    |
----------------------------------
|         2.025       |    20    |
----------------------------------

The resulting feature is determined by the pooling function in the case above it computes the arithmetic mean. See this for a more in depth usage example.


How to measure the similarity between two images? I have two group images for cat and dog. And each group contain 2000 images for cat and dog respectively.

My goal is try to cluster the images by using k-means.

Assume image1 is x, and image2 is y.Here we need to measure the similarity between any two images. what is the common way to measure between two images?

Check this handout!

Well, there a few so... lets go:

Given two images $J[x,y]$ and $I[x,y]$ with $(x,y) \in N^{N \times M}$...

A - Used in template matching:

Template Matching is linear and is not invariant to rotation (actually not even robust to it) but it is pretty simple and robust to noise such as the ones in photography taken with low illumination.

You can easily implement these using OpenCV Template Matching. Bellow there are mathematical equations defining some of the similarity measures (adapted for comparing 2 equal sized images) used by cv2.matchTemplate:

1 - Sum Square Difference

$$ S_{sq} = \sum_{(n,m) \in N^{M \times N}}  \bigr(J[n,m] - I[n,m]\bigl)^2$$

This can be normalized as
$$ \frac{S_{sq}}{\sqrt{\sum J[n,m]^2 \times \sum I[n,m]^2}} $$

2 - Cross-Correlation

$$ C_{crr} = \sum_{(n,m) \in N^{M \times N}}  \bigr(J[n,m] \times I[n,m]\bigl)^2$$

This can be normalized as
$$ \frac{C_{crr}}{\sqrt{\sum J[n,m]^2 \times \sum I[n,m]^2}} $$

B - Image descriptors/feature detectors:

Many descriptors were developed for images, their main use is to register images/objects and search for them in other scenes. But, still they offer a lot of information about the image and were used in pupil detection (A joint cascaded framework for simultaneous eye detection and eye state estimation) and even seem it used for lip reading (can't direct you to it since I am not sure it was already published)

They detect points that can be considered as features in images (relevant points) the local texture of these points or even their geometrical position to each other can be used as features.

You can learn more about it in Stanford's Image Processing Classes (check handouts for classes 12,13 and 14, if you want to keep research on Computer vision I recomend you check the whole course and maybe Rich Radke classes on Digital Image Processing and Computer Vision for Visual Effects, there is a lot of information there that can be useful for this hard working computer vision style you're trying to take)

1 - SIFT and SURF:

These are Scale Invariant methods, SURF is a speed-up and open version of SIFT, SIFT is proprietary.

2 - BRIEF, BRISK and FAST:

These are binary descriptors and are really fast (mainly on processors with a pop_count instruction) and can be used in a similar way to SIFT and SURF. Also, I've used BRIEF features as substitutes on template matching for Facial Landmark Detection with high gain on speed and no loss on accuracy for both the IPD and the KIPD classifiers, although I didn't publish any of it yet (and this is just an incremental observation on the future articles so I don't think there is harm in sharing).

3 - Histogram of Oriented Gradients (HoG):

This is rotation invariant and is used for face detection...

C - Convolutional Neural Networks:

I know you don't want to used NN's but I think it is fair to point they are REALLY POWERFULL, training a CNN with Triplet Loss can be really nice for learning a representative feature space for clustering (and classification).

Check Wesley's GitHub for a example of it's power in facial recognition using Triplet Loss to get features and then SVM to classify.

Also, if your problem with Deep Learning is computational cost, you can easily find pre-trained layers with cats and dogs around.

D - Check on previous work:

This cats and dogs fight has been going on for a long time... you can check solutions on Kaggle Competitions (Forum and Kernels), there were 2 on cats and dogs This One and That One

E - Famous Measures:

SSIM Structural similarity Index
L2 Norm (Or Euclidean Distance)
Mahalanobis Distance

F - Check on other kind of features

Cats and dogs can be a easy to identify by their ears and nose... size too but I had cats as big as dogs... so not really that safe to use size.
But you can try segmenting the images into animals and background and then try to do region property analisys...
Also, check on this image similarity metrics toolkit page it is in C but...
Check this paper on image similarity
Take a look on this Stack Overflow question and this Research Gate one
If you have the time, this book here: Feature Extraction & Image Processing for Computer Vision from Mark S. Nixon have much information on this kind of procedure
You can try Fisher Discriminant Analysis and PCA to create a mapping and the evaluate with Mahalanobis Distance or L2 Norm


How to treat sparse categorical features in a Neural Network for multiclass classification with Tensorflow? I am building a Neural Network for multiclass classification.

My dataset has 3 millions of observations. My features are 7 unordered categorical values. My problem is sparse as 4 features among the 7 can take more than 1500 different values. The label is a categorical variable that takes 2100 different values.

My model outputs a vector of 2100 probabilities through the softmax function.

I am using Tensorflow with Python and I wanted to use one-hot encoding to encode each of the features.

Do this method suffer from the sparsity of my data?

For instance, do the One-Hot encoding transform my inputs into a vector of more than 2000*4 = 8000 features? In that case, if I use a mini-batch gradient descent with a batchsize of 50, is the fact that I have a lot more features than samples (8000>>>50) a problem ("curse of dimensionality")?

Would you have a solution, and a way to implement it on Tensorflow with Python?

Having more features than batchsize it's not a problem indeed is the rule in many fields like computer vision. Having a very sparse dataset is also the norm in NLP. Both communities rely on dropout to prevent overfitting.

You may see an example of using drouput with Tensorflow here.


Where can I find documentation or paper mentioning pre-trained distilbert-base-nli-mean-tokens model? I am trying to find more information about pre-trained model distilbert-base-nli-mean-tokens. Can someone please point me to it's paper or documentation? Is it based on DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter paper? This is published in March 2020. I am looking for links between this paper and Sentence-BERT (sentence-transformers). Original sentence-bert paper is published in Aug 2019. I wanted to try pre-trained model using S-BERT model and hence tried distilbert-base-nli-mean-tokens model. After implementation I found out that it's much faster than other pre-trained models available on sentence-transformer website. While studying it's paper I realised original paper do not mention this pre-trained model.
I found Making Monolingual Sentence Embeddings Multilingual using
Knowledge Distillation this paper published by same author which do mention DistilmBERT but not DistilBert Can someone please help me solve this mystery?

The distilbert-base-nli-mean-tokens model is the DistilBERT base model for natural language inference (NLI) using the MEAN pooling strategy for CLS tokens. For more information on the DistilBERT base model, have a look at the original paper. The pooling strategy is simply the way the model combines the different embeddings (and thus information) from the different layers.


Difference between paragraph2vec and doc2vec Is paragraph2vec the same as Doc2vec or is every approach different?

There may be differing implementations, but these two terms refer to the same thing.

Both convert a generic block of text into a vector similarly to how word2vec converts a word to vector. Paragraph vectors don't need to refer to paragraphs as they are traditionally laid out in text. They can theoretically be applied to phrases, sentences, paragraphs, or even larger blocks of text.

Here's one definition of a paragraph vector:

  An unsupervised algorithm that learns fixed-length feature
  representations from variable-length pieces of texts, such as
  sentences, paragraphs, and documents.

And the full paper if you are interested:

https://cs.stanford.edu/~quocle/paragraph_vector.pdf


Spark 1.5.1: Train many binary classifiers, save them, then use them on new data I have a DataFrame representing an annotated dataset with 300 labels.

The DataFrame looks like follow (the first row is just to explain the columns):

binary Vector column for labeling, feature column
[0, 1, 1, 0, 0 , 0.... ], featureVec
[0, 0, 1, 0, 1 , 0.... ], featureVec
......

The labelling column, represent the presence or the absence of each label as an annotation of a feature vector (which we can see in the second column).

Please correct me if I am wrong: I can't train one multi-class classifier because the labels are not exclusives, so I would like to train one binary classifier per label.

As I have 300 labels, then I need to train and optimise 300 classifier (say logistic regression classifier for example).

What is the best way and best practices to train my classifiers with N-cross validation for parameter optimisation? any example code or reference is highly appreciated.

Once the classifiers are optimised, what is the best way and best practices to save the best models and use them to classify the new data?

This is a general ML answer, not about Spark 1.5.1, but perhaps it can help you find a way to the solution.

Your problem is multi-label as well as multi-class (e.g. http://scikit-learn.org/stable/modules/multiclass.html).

Some algorithms have multi-label versions, and will deal with the problem internally so you don't have to. If you don't have a multi-label version of your algorithm, you can either change to an algorithm that does, or roll your own multi-label solution around your algorithm, for example using binary classifiers for each label. (This typically leads to worse performance as there are many complications and possible optimisations to take into account).


Is it possible for a neural net to score as high as a different form of supervised learning? I've been working with the Adult Census Income dataset from UCI
http://archive.ics.uci.edu/ml/datasets/adult

I've created two different models, one using a gradient boosted classifier with sklearn, and one with a neural net using Keras/Tensorflow.

So I'm not interested in code hints or anything, but I have a general question about machine learning-I have a significantly higher accuracy using the gradient boosted classifier than I do with the neural net.

In general, is it possible for my (or any, really) neural net to reach the same accuracy as any other kind of supervised learning? Does it just take a lot of hard work and elbow grease to tune the neural net well enough? I'm working with the same dataset, using the same feature engineering for both the NN and the gradient boosted classifier.

Generally speak, no. Deep learning models struggle to compete when it comes to tabular data.

If we head over to kaggle were people compete to build the best model we find that usually the best performing non-ensemble models for this kind of data are gradient boosting trees. More specifically it tends to be either XGBoost or more often now days it is LightGBM that performs best. Both are highly optimized implementations of gradient boosting trees.

Feature engineering and parameter tuning are both important to get extra performance. But the gap does not shrink since you would get the same improvements if you did it on a gradient boosting model instead.

Actually other models tend to gain more from feature engineering than neural networks since one of the strengths of neural networks is that they perform a sort of automatic feature engineering when they are trained.

Neural networks usually shine in the domain of unstructured data such as text and images.


Сlassification using convolutional layers I have a simple neural network of 2 layers, recognizing 10 classes:

self.layer1 = nn.Sequential(
    nn.Conv2d(1, 32, kernel_size=5, stride=1, padding=2),
    nn.ReLU(),
    nn.MaxPool2d(kernel_size=2, stride=2))
self.fc1 = nn.Linear(14 * 14 * 32, 10)

How can a classification be made using the convolutional layer in this case?

You need Dense() layers after convolution / pooling layers. Conv and max-pool are meant to process pixel data in a format that can be later received from dense layers.
It's the Dense() output layer that, using a softmax activation, can technically perform the classification (by returning a one-hot encoded vector).


Terminology: "flat geometry" in the context of clustering Sklearn's documentation refers to "flat" or "non-flat" geometry of clusters to describe the use-cases of their implemented clustering algorithms. Those terms are not directly defined. However, the following quote states:

  Non-flat geometry clustering is useful when the clusters have a specific shape, i.e. a non-flat manifold, and the standard euclidean distance is not the right metric.

So far, I do understand that a manifold type of shape is poorly represented by a cluster centroïd and that clustering such dataset should rather rely on local density, nearest neighbors, or connectivity constraint. However, I do not understand how it relates to the concept of flatness according to Wikipedia's definition:

  In geometry, a flat is a subset of a Euclidean space that is congruent to a Euclidean space of lower dimension.

It seems contradictory to me. For instance, a hypersphere is not geometrically flat but would be flat using sklearn's terminology.

My questions are then:

Are flat and non-flat geometry a legit terminology in machine learning and statistics?
If yes, what is the mathematical definition?
If no, what is the most appropriate alternative? (e.g. manifold vs convex ???)

Are flat and non-flat geometry a legit terminology in machine learning
  and statistics?

These are terminologies from Mathematics, they are valid in any field.

  What is the mathematical definition?

(Non) flat manifold

In mathematics, a (Riemannian) manifold is said to be flat if its curvature is everywhere zero; otherwise non-flat. This is very different than the definition of flat object in geometry. According to that definition, only points, lines, and hyper-planes are flat (not for example hyperspheres or segments).

For example, set $\left\{(t,t):t\in(-1,1)\right\}$ is a 1D flat manifold in ${\Bbb R}^2$, set $\left\{(t,t^2):t\in(-1,1)\right\}$ is a 1D non-flat (positively curved) manifold in ${\Bbb R}^2$, and a hypersphere is an $n$D non-flat (positively curved) manifold in ${\Bbb R}^{n+1}$.

Accordingly, a cluster with a (non) flat shape corresponds to a (non) flat manifold.

Here are some examples from the document.

Points are concentrated around (A) two 1D non-flat manifolds (circles) which are non-convex, (B) two 1D non-flat manifolds (arcs) which are non-convex, (C) three 1D flat manifolds (segments) which are convex, (D) three 0D flat manifolds (centers as points) which are convex.

(Non) flat geometry vs a Flat

By "flat geometry" the document means "Euclidean geometry" not "a flat object defined in geometry". If we measure distances (consequently lengths, areas, volumes, etc.) via Euclidean distance we are obeying the Euclidean geometry, otherwise, we are obeying a non-Euclidean geometry. For example, if we measure a distance between two points by following a non-flat manifold (a geodesic), we are obeying a non-Euclidean geometry. As an illustration,

In (A), the red line measures a distance obeying a flat geometry, the blue line measures the distance obeying a non-flat geometry (by moving along the non-flat manifold). If an appropriate map of manifold to a lower dimension is possible (B), obeying a flat geometry would be equivalent to obeying the non-flat geometry before mapping.


How to train a supervised sequence classifier like CRF, if we have to extract start date and end date from a user query in python I have to build a chatbot, in python in which a user can apply for a leave. I want to extract start date and end date from a users query. I did some research on couple of algorithms and found CRF Entity Extractor as the best one. I now want to see a similar implemented solution in python, which I can use it as a reference. I would like to see an end to end solution right from training dataset to predicting start and end dates from query. Kindly help.

For example:

1.query: "I want to take leave from 2nd April to 5th April."
Predicted dates: "02-04-2020" and "05-04-2020"

1.query: "I want to take leave on next Monday"
Predicted dates: "09-03-2020"

I'm not sure that building a model using CRF is the best approach here. It's going to require quite a bit of training data and effort to get it working like you want. Dates are pretty structured in most cases so there are more straightforward approaches to extracting them. Stanford's SUTime library, for instance, does exactly what you're describing in your question. Although is is Java based there is a Python wrapper for it. For example, for the following input:

query = 'I need a desk for tomorrow from 2pm to 3pm'

Using the SUTime Python wrapper, this is the parsed result you'd get:

[
    {
        "end": 26,
        "start": 18,
        "text": "tomorrow",
        "type": "DATE",
        "value": "2016-10-14"
    },
    {
        "end": 42,
        "start": 27,
        "text": "from 2pm to 3pm",
        "type": "DURATION",
        "value": {
            "begin": "T14:00",
            "end": "T15:00"
        }
    }
]


How to interpret fit from regression (decision) tree which has used 0 variables I have fit a regression tree to my dataset and the output from summary(tree1) is as follows:

Regression tree:
tree(formula = y ~ X)
Variables actually used in tree construction:
character(0)
Number of terminal nodes:  1
Residual mean deviance:  0.985 = 71680 / 72770
Distribution of residuals:
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
-1.0050 -1.0050 -0.3122  0.0000  0.7842  2.2140

So seemingly the tree has found no variables/splits worth including. The reason this is curious is because other methods have found trends in the data. For example a linear regression line found several significant variables and p-value for F-statistic was ~0 (although this is a very noisy dataset based on human behaviour and R-squared was not much bigger than 0).

How can I interpret this regression tree output? There are clearly trends in the dataset as shown by linear regression. Are regression trees more focused on accurate predictions and so won't fit well to a noisy dataset?

It is hard to tell what is going on without knowing the data and your actual approach/method/model/code.

Just some remarks. Linear regression is parametric and results hinge on the assumptions behind the parameterization. So maybe not too good in a noisy setup.

Single trees are often weak learners. The advantage of trees however is, that there is no parameterization behind. So they are in principle rather flexible. It is state of the art to combine many trees to one ensemble (random forest).

Especially with noisy data, boosting (also tree based) is really good. Boosting makes a lot of very small trees and (by updating weights to specific observations), it focusses on cases which are hard to predict. I guess this would be a thing to try if you work with predictive modeling.

Here are some very basic Python examples for boosting if you want to give it a try: https://github.com/Bixi81/Python-ml


C parameter error in pipeline I'm trying to build a classifier for my dataset and I'm having an issue with using my gridsearchCV and pipeline together. Here is my code:

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import Imputer, StandardScaler
from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, classification_report

imp = Imputer()
scaler = StandardScaler()
clf = LogisticRegression(multi_class='multinomial')

Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, test_size=0.3, random_state=42)

pipeline = make_pipeline(imp, scaler, clf)

param_grid = {'penalty':["l1","l2"], 'C':np.arange(0.001, 1, 0.01),
                  'solver': ['newton-cg', 'lbfgs', 'liblinear', 'sag', 'saga']}

search = GridSearchCV(estimator=pipeline, param_grid=param_grid, cv=5)

Once I call the method on search, I get the following error.

search.fit(Xtrain, ytrain)

ValueError: Invalid parameter C for estimator Pipeline(memory=None,
 steps=[('imputer', Imputer(axis=0, copy=True, missing_values='NaN', strategy='mean', verbose=0)), ('standardscaler', StandardScaler(copy=True, with_mean=True, with_std=True)), ('logisticregression', LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
      intercept_scaling=1, max_iter=100, multi_class='multinomial',
      n_jobs=1, penalty='l2', random_state=None, solver='liblinear',
      tol=0.0001, verbose=0, warm_start=False))]). Check the list of available parameters with `estimator.get_params().keys()`.

I'm not sure how my C parameters are invalid. Does anyone know what I'm doing wrong?

I have already posted it here:

https://stackoverflow.com/questions/43366561/use-sklearns-gridsearchcv-with-a-pipeline-preprocessing-just-once/55401454#55401454

Suppose you have this pipeline:

classifier = Pipeline([
    ('vectorizer', CountVectorizer(max_features=100000, ngram_range=(1, 3))),
    ('clf', RandomForestClassifier(n_estimators=10, random_state=SEED, n_jobs=-1))])

Then, when specifying parameters you need to include this 'clf_' name that you used for your estimator. So the parameters grid is going to be:

params={'clf__max_features':[0.3, 0.5, 0.7],
        'clf__min_samples_leaf':[1, 2, 3],
        'clf__max_depth':[None]
        }


How do you set sigma for the Gaussian similarity kernel? Let's say we have $n$ two-dimensional vectors: $$\mathbf{x}_1,\dots,\mathbf{x}_i,\dots,\mathbf{x}_n=(x_{1_1},x_{1_2})^T,\dots,(x_{i_1},x_{i_2})^T,\dots,(x_{n_1},x_{n_2})^T$$ How do you set $\sigma$ for the Gaussian similarity kernel:
$$s(\mathbf{x}_i,\mathbf{x}_j)=\exp\left(-\frac{||\mathbf{x}_i-\mathbf{x}_j||^2}{2\sigma^2}\right)$$

Updated Answer

According to a reference paper in Spectral Clustering (von Luxburg) the $\sigma$ is simply set to 1. A further tuning can be applied with some visualization inspection but I did not find any discussion regarding setting this parameter.

Using code snippet below you see the effect:

import numpy as np
import matplotlib.pyplot as plt
from scipy.spatial.distance import pdist

def gausker(x1,x2,sigma):
    dist = np.linalg.norm(x1-x2)
    return np.exp(-dist**2/(2.*(sigma**2.)))

data = np.array([[0,0],[1,1],[1,0],[0,1],[10,10],[10,9],[9,10],[9,9]])
plt.figure()
plt.plot(data[:,0],data[:,1],'o',ms=20)
plt.show()
s = np.std(pdist(data))
for sigma in [1, s, 10, 100]:
    gaus = np.zeros((8,8))
    for ii in range(8):
        for jj in range(8):
            gaus[ii,jj] = gausker(data[ii,:],data[jj,:],sigma)
    plt.figure()
    plt.imshow(gaus,extent=[0, 1, 0, 1])
    plt.colorbar()
    plt.title(str(sigma))
    plt.show()

For Machine Learning algorithms is better to have more distinction. The Gaussian similarity kernel cares about local similarities. The image show the kernel for $\sigma=1$

Conceptually is similar to a k-nearest neighbors graph, since it considers local neighborhood and almost disregards the relationship between two nodes far apart.


Is NLP suitable for my legal contract parsing problem? My company has a product that involves the extraction of a variety of fields from legal contract PDFs. The current approach is very time consuming and messy, and I am exploring if NLP is a suitable alternative.
The PDFs that need to be parsed usually follow one of a number of "templates". Within a template, almost all of the documents are the same, except for 20 or so specific fields we are trying to extract. That being said, there are sometimes slight inconsistencies such as missing/added spaces, punctation, etc. Currently there are about 15 templates, but sometimes another gets added. Most of the data that needs to be extracted is in tabular form, but a few values are littered in paragraph text.
With the current, non-ML approach, we use a proprietary PDF parser that extracts out all of the tables into a json format. This parser must be custom configured for every new PDF template that appears, and this is time consuming. After converting to json, a custom regex expression must be made for each value in each template, leading to several hundred regexes total. Usually this approach is pretty accurate, but sometimes the slight inconsistencies can break it, meaning a the regex needs to be revised.
I'm wondering if NLP Named Entity Recognition would be a cleaner solution to this problem. My idea is to label all of the values in a few hundred sample documents and then train a custom NER model in a library like Spacy or Flair. Ideally, we could feed in the raw, extracted text from the PDFs instead of having to configure the custom parser to extract json.
The advantage I see in using the NLP approach is that we wouldn't have to configure the custom parser and write a bunch of regexes every time a new template appears. At worst, we would have to label a few of the new documents every time a template is added, which would presumably be faster and easier than the current approach. I think we could also generate tons of synthetic training data easily by swapping labeled values between different documents.
I am concerned that, using an ML approach, we wouldn't be able to achieve near perfect accuracy, which is a requirement. I'm also not sure how well NLP can perform on raw text from tables as opposed to paragraphs. The nice thing is that the documents tend to be very similar within templates.
I've never done NLP before, so I'm wondering if anyone here thinks this approach would be worthwhile exploring. If this is feasible, does anyone have suggestions on how to get the best results?

Here's what I would have done.

extract raw data from the the pdfs.
use rule based NER from spacy(it's quite fast), but it will require some manual pattern making.
Also apply multiprocessing if you can.
https://spacy.io/usage/rule-based-matching
You can also write a document classification to segregate the templates and send the templates to appropriate code,running parallely.
Also checkout blackstone library for legal texts, it might be of some use https://spacy.io/universe/project/blackstone


Adding new variable to model Let's say I already have a logistic regression model (or other) with N number of explanatory variables and is 70% accurate. Now if there are other variables available, how would I test if the new variables would improve my accuracy without building new model.

I do not think you can estimate the effect of a variable without adding it to the model. This is because the effect of a variable on the model's discriminatory power depends on

the strength of association between the outcome variable and the new variable
whether the new variable are collinear with some of the old variables

You could in principle estimate both strength of association and collinearity, but it probably is bad practice and would result in overfitting.

Also in general I think it is best to not use accuracy to evaluate a logistic regression (see ref) but rather a proper scoring rule like Brier Score.

Further, when comparing two nested models (i.e. where one model contains a subset of the variables of the other model) I believe that best practice is to compare the AIC or BIC, or perform a likelihood ratio test.


InterquartileRange takes up most instances in data set I'm very new to this community, so please overlook my noobness.

I have a data set with 2948 instances and I tried to remove outliers using InterquartileRange filter in Weka. The issue is that the number of 'YES' instances in ExtremeValues and Outliers takes up to 2947 and 2946 respectively. In other words, all my data are considered outliers.

What does this say about my data set? Or am I not meant to perform IQR on this data, if so, is there other algorithms to identify outliers other than IQR?  And how would one perform regression on such a data set?

Thank you.

The InterQuartileRangeFilter from weka library uses an IQR formula to designate some values as outliers/extreme values. Any value outside this range $[Q_1 - k(Q_3-Q_1), Q_3 + k(Q_3-Q_1)]$ is considered some sort of an outlier, where $k$ is some constant, and $ IQR = Q_3 - Q_1$.

By default weka uses $k=3$ to define something as outlier, and $k=3*2$ to define something as extreme value (extreme outlier).

The formula guarantees that at least 50% value are considered non-outliers. Having a single variable (univariate sample of values), it's practically impossible to reproduce your result.

Note however that this filter can be applied to a data frame. When applied like this, it will consider as an outlier any instance of the data frame which has at least one value of the instance considered as outlier for that variable.

Now, supposing that you have a data frame with 2 variables, totally uncorrelated (independent). Considering again that only 10% of the values from each variable are considered outliers, due to independence, one can expect that $(1-0.9)^2$ values will not be outliers. If you have $p$ variables like that in your data frame, you might expect to have only $(1-0.9)^k$ normal values, and is not very hard to arrive in that situation.

There are two things which you will have to consider. One is to increase the factors for outliers if in general too many values are considered outliers (ideally you would like to take a look at each variable graphically and if possible to get some idea about the distribution beneath). The second one is to check if you have many values which are totally independent. The second hint does not solve your problem but might give you a reason why it happens.


how does minibatch for LSTM look like? Minibatch is a collection of examples that are fed into the network, (example after example), and back-prop is done after every single example. We then take average of these gradients and update our weights. This completes processing 1 minibatch.

I read these posts

[1]
[2],
about padding entries in a minibatch so they have same length
and about preserving the cell state but the following is still unclear to me:

Question part a:

How a minibatch entity would look like for LSTM?
Say, I want it to reproduce Shakespeare, letter by letter (30 characters to choose from).

I launch LSTM, let it predict for 200 characters of a poem, then perform back propagation. (hence, my LSTM works with 200 timesteps). Does this mean my minibatch consist of 1 example whose length is 200?

Question part b:

If I wanted to launch 63 other minibatches in parallel, would I just pick 63 extra poems?  (Edit: Original answer doesn't mention this explicitly, but we don't train minibatches in parallel. We train on 1 minibatch, but train its examples in parallel)

Question part C:

If I wanted each minibatch to consist of 10 different examples, what would such examples be, and how would they be different from 'what I perceive as a minibatch'?

I think you need to distinguish between training and execution of the model. During training, you can use batches, which in your case will be different fragments from Shakespeare. So, a batch will be a list of fragments, and the language model will start from the first character on each element of the batch and do the backward and forward pass.

When you execute the model, once it is trained, you would like to see one single example, in which case you can set the batch size to one.

I believe this answers your three questions.


Classifying ultrasound videos with a small dataset I have a small dataset of ~300 ultrasound clips, about evenly divided between 3 classes. Due to the nature of the data (medical) it's difficult to get more.

I need to train a model (or an ensemble) to classify these clips.

Feature reduction is a necessary first step. I've considered using HOGS (for example) on a cropped part of the video to create relatively small set of time-series features.
However, even a "dumbed down" version of HOGS would give at least 100 time series features.

That being said, lets assume for a minute I'm able to reduce the feature space even more while maintaining variability between classes.

Which model(s) would be best suited for this problem?

One option in similar problems is to use transfer learning on raw images. For similar problems (few hundred images for training), solution that worked for me is transfer learning + image pre-processing to generate augmented images (E.g.: Stretched / Rotated versions of images).

High level design is :

Use a model that has been trained on large number of images (millions). This model has already learnt image processing as a domain (such as edge detection, phase / color changes and so on) and might remove need to feature engineering (like HOGS)
Add few more layers to be trained for Ultrasound images
Train these new layers with images (Existing layers are frozen since they are already trained)

Examples of this approach are :

https://www.researchgate.net/publication/311094437_Transfer_Learning_with_Convolutional_Neural_Networks_for_Classification_of_Abdominal_Ultrasound_Images

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5359213/

https://www.sciencedirect.com/science/article/pii/S2095809918301887


train_test_split ValueError: Input contains NaN I tried to do a stratified sampling by way of train_test_split in order to save myself some trouble later. So I wrote the following lines:

from sklearn.model_selection import train_test_split

X=data_df
y=data_df.pop('class')

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.125, stratify=y)

I got the error :

ValueError: Input contains NaN

Any help is welcome!

Check to see if you have any null or nan values:

X[X.isnull() == True]

Then you have to decide what to do with those nan values. Something that is commonly done is to forward fill in place.

X.fillna(method = 'ffill', inplace = True)
y.fillna(method = 'ffill', inplace = True)


Real world performance evaluation on problem similar to malware detection I have a question on how to evaluate the performance of models similar to malware detection in the real world. For example, I have created a malware detection model to predict whether the engine will be down. And the model seems to do a decent job so I push it to production. However, in real-world, when the model predicts the engine is going to be down, I will send someone to fix it. The tricky part is I am doing the prevention by sending someone to fix the engine before it has been down. Are there any good methods to evaluate the performance of the model? Thanks!

Can you measure a proxy measurement that will be measurable even if the system never goes down? For example, does the engineer find “malware” or whatever the analogue is in this situation?

If not, and if it is safe to do so, why not simply take no action in X% of cases and use the data gathered in those cases as evaluation data?


Building a model in two stages This question is regarding a use case related to predictive maintenance. The final model built (based on the steps mentioned below) is used to predict the failure of a particular component for a particular type of device. In the Machine Learning context, it's regression use case for which deep learning model(s) are being built.

The model building process consists of two stages:

Initially based on the data (temporal in nature) from multiple devices, one single model has been built. The model has been cross validated with the data from the same set of devices (expected).
Now based on the model type and hyperparameters decided at #1, models are being built for individual devices (using data from respective devices. The temporal frequency of data at this stage is different from the frequency of the data used at #1). These models (for individual devices) will be deployed in production (i.e. one model per device). The model built at #1 will NOT be used in production.

Is this two stage process a valid approach of model building?

As per my understanding, instead of the two stage process and large number of individual models, a single model can be built with data from all devices (with matching frequency) in one shot. There is a possibility that devices differ from each other based on different characteristics. But those characteristics could be used as a feature while building the single model.

Based on domain knowledge, I am more or less convinced that one single model can server multiple devices. But, my question is more about the validity of two stage approach:

Decide model type and hyperparameters in the first stage.
Build individual models with data from individual devices based on the model type and hyperparameters decided in the first stage.

This is a bad idea. The cross validation score that you obtained from your first model is valid only for that model. If you train a new model there is no guarantee that it will have similar performance, you have to test each and every model.

This could work only if you can guarantee that the data for the second model will be very similar to the data of the first model. You would have to be very sure of that.


Question about AlphaGo Zero's Neural network architecture? The following text is quoted from the AlphaGo Zero Paper 2017 from Nature. My question is regarding the eight features.

The input to the neural network is a 19 × 19 × 17 image stack
comprising 17 binary feature planes. Eight feature planes, Xt, consist
of binary values indicating the presence of the current player’s
stones (Xit=1 if intersection i contains a stone of the player’s
colour at timestep t;

This sounds like the t means time and the eight features are the latest 8 board states. If this is the case, this will be my question.

0 if the intersection is empty, contains an
opponent stone, or if t < 0). A further 8 feature planes, Yt,
represent the corresponding features for the opponent’s stones. The
final feature plane, C, represents the colour to play, and has a
constant value of either 1 if black is to play or 0 if white is to
play. These planes are concatenated together to give input features st
= [Xt, Yt, Xt−1, Yt−1,..., Xt−7, Yt−7, C].

I have the feeling the eight features should be the combination of rotated (4) and reflected (2
) board states of the current board state. Please correct me if I am wrong.

History features Xt, Yt are necessary, because Go is not fully observable solely from the current
stones, as repetitions are forbidden; similarly, the colour feature C
is necessary, because the komi is not observable.

Board repetition can be easily prevented by keeping tracking of all previous board states and when a new move is attempted, just compare against all previous board states. The same is for komi. Keeping 8 previous board states to prevent repetition doesn't sound right to me. What if the repetition happened to the last 9th board state? If this is not a serious paper published to the Nature, I would highly doubt this is wrong.

This sounds like the t means time and the eight features are the latest 8 board states.

That is correct. The description in the article is clear.

I have the feeling the eight features should be the combination of rotated (4) and reflected (2) board states of the current board state. Please correct me if I am wrong.

That is wrong.
This data augmentation can be done by modifying state data stored from each episode, and applied during weight update step (either randomly or exhaustively), because any rotation/reflection of a board should have the same expected value. It is not necessary to provide that augmentation at the feature level.

Board repetition can be easily prevented by keeping tracking of all previous board states and when a new move is attempted, just compare against all previous board states.

A game engine can do that, as could bespoke code added to the agent that was specific to Go. However, a neural network cannot do that unless those previous states are part of the input.

The same is for komi.

If you want the board to correctly predict the value of a state, the nerual network needs to have the current player identity encoded in some way. Having it tracked outside of the neural network's inputs (i.e. outside of state representation) does not help with this.

What if the repetition happened to the last 9th board state?

Most typically in Go, the repetition rule has effects over a relatively short duration, because moves add stones in new positions. The relevant rules are called ko and superko, with superko being optional. When applying the standard ko rule, the repetition that it blocks is only likely to be achievable through play on short timescales (2 steps) that are covered by the AlphaGo Zero representation.
In the unlikely event that a play is blocked due to superko rule and repetition from earlier than 8 steps ago, the game engine would prevent a play that the agent may have assigned a high value to, and the agent could get the value wrong. I suspect that this has little or no impact on performance, as loops in Go board state of this duration are rare and do not occur much in practice.


Preprocessing for finetuned CNN model from pretrained models Is it necessary to preprocess the images the same way as they were during the training of pretrained models in our finetuned model to use it for a different classification task ? Say, I have a pretrained VGG16 model which I am finetuning to use it for a different classification task ? Now, before feeding my dataset to this finetuned model, should I necessarily use vgg16.preprocess_input() for training on this dataset or I can use my own preprocess_input ?

If a network was trained let's say on centered input values (i.e. ranging from -1 to 1) with subtracted mean channel values computed on the ImageNet, it is desirable to keep this preprocessing to make fine-tuning easier.

From other hand, CNN sometimes can be surprisingly robust to this kind of input fluctuations and, unless you freeze the pre-trained part, can adapt to the changes in the input. So, if you just change mean channel values from default [103.939, 116.779, 123.68] to [128, 128, 128] this should work ok. Otherwise, just compare how the accuracy is improving during fine-tuning using the vgg16 preprocessing and your method.


Criterion for Firing a perceptron The criterion for firing a perceptron is as follows

Why is it that when the function $w \cdot x + b = 0$ the output is zero as well. Why couldn't it have been set to 1?

If one were to simulate the behavior of the perceptron with a sigmoid function, $w \cdot x + b$ can be multiplied by any arbitrary constant $C>0$ (as $C$ tends to infinity) as long as $w \cdot x+b$ is not equal to 0. This is because at 0 condition, the firing always gives output as 1 instead of zero it should have giving. However, changing the firing rule overcomes this problem. Then, is there any reason why 0 should be the output when $w \cdot x+b= 0$?

The perceptron algorithm was invented in 1957 at the Cornell Aeronautical Laboratory by Frank Rosenblatt (Wikipedia EN)

It was the first neural network like architecture. It was thought that (a combination of) perceptrons could learn anything. There was a lot of controversy about this and after a while Marvin Minsky showed that a single perceptron could not solve the XOR problem. This was the end of the first neural network hype.

After a couple of decades the multilayer perceptron was invented, and with it the backpropagation algorithm. The backpropagation algorithm can learn weights that are not directly connected to the output unit. However, it needs a differentiable activation function. Hence, the ramp activation function was not sufficient anymore and others like the sigmoid and tanh were used for this.

Besides, the perceptron was first implemented using electronic boards and not with the software available to us today. Binary thresholds made this complicated process a lot easier.

So, in short Rosenblatt could have chosen for any version of the heaviside step function but he chose for the one where $w \cdot x + b = 0$ leads to a zero output. If you think binary values this makes sense, a zero input (i.e. False) leads to a zero output.


Protein interaction prediction- how to input this data structure I know the basics of machine learning and have quite an experience with time series data or data fed in a tabular format. But in the picture, the data is arranged as a graph. Is there a way to input the graph into a ML tool such as Artificial Neural Network or any other? I don't know if there is a theory for handling such data structure. The task is to recreate the graph from the output of the ML algorithm after training. So, whatever input I get, the output should be the same as the input -- quite similar to an auto-associative memory. Can somebody please help?

https://en.wikipedia.org/wiki/Adjacency_matrix

For such problems, you can tabulate these connections as adjacency matrix and create a network to predict weights for the matrix given some properties of nodes (Say for a social graph; given properties of User1 and User2 [for example Zipcode, school ...] output 1, or 0),

Some examples are :

https://www.biorxiv.org/content/biorxiv/early/2018/01/14/247577.full.pdf
http://kawahara.ca/convolutional-neural-networks-for-adjacency-matrices/

Edit : Illustration of Train_X and Train_Y

Columns A through H form Train_X and Column I is Train_Y.


Extracting the feature map in Tensorflow Object Detection API I would like to know how can I extract the feature map of a mobilenet trained on tensorflow object detection API. I want to take that feature map in order to feed another classifier. Thanks!

import tensorflow as tf

# Sample frozen model
MODEL = "frozen_inference_graph.pb"

# An existing operation from the frozen model
OP_NAME = "WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_2/BatchNorm/feature_0/beta/read/_360__cf__363"

# Load the graph from the frozen model
detection_graph = tf.Graph()
with detection_graph.as_default():
    od_graph_def = tf.GraphDef()
    with tf.gfile.GFile(MODEL, 'rb') as fid:
        serialized_graph = fid.read()
        od_graph_def.ParseFromString(serialized_graph)
        tf.import_graph_def(od_graph_def, name='')

# print all operations
for op in detection_graph.get_operations():
    print(op.name)

# print tensor ( without :0 you will get the operation itself )
print(detection_graph.get_tensor_by_name("{}:0".format(OP_NAME)))


input shape of dataset in CNN My dataset is a simple table of 20 columns and 100,000 rows.It is not a image data as commonly used in CNN. What input shape should I provide in this case?

Right now I did-

input_shape = (21,109713,1)

model.add(Conv2D(32, kernel_size=(5, 5), strides=(1, 1),
                 activation='relu',
                 input_shape=input_shape))

which gives the error-

ValueError: rng_mrg cpu-implementation does not support more than (2**31 -1) samples

I have mentioned this in other posts also:

One can use Conv1d of Keras for usual features table data of shape (nrows, ncols). To input features, following 2 steps are needed:

xtrain.reshape(nrows, ncols, 1)
# For conv1d statement:
input_shape = (ncols, 1)

For example, taking first 4 features of iris dataset:

To see usual format and its shape:

iris_array = np.array(irisdf.iloc[:,:4].values)
print(iris_array[:5])
print(iris_array.shape)

The output shows usual format and its shape:

[[5.1 3.5 1.4 0.2]
 [4.9 3.  1.4 0.2]
 [4.7 3.2 1.3 0.2]
 [4.6 3.1 1.5 0.2]
 [5.  3.6 1.4 0.2]]

(150, 4)

Following code alters the format:

nrows, ncols = iris_array.shape
iris_array = iris_array.reshape(nrows, ncols, 1)
print(iris_array[:5])
print(iris_array.shape)

Output of above code data format and its shape:

[[[5.1]
  [3.5]
  [1.4]
  [0.2]]

 [[4.9]
  [3. ]
  [1.4]
  [0.2]]

 [[4.7]
  [3.2]
  [1.3]
  [0.2]]

 [[4.6]
  [3.1]
  [1.5]
  [0.2]]

 [[5. ]
  [3.6]
  [1.4]
  [0.2]]]

(150, 4, 1)

This works well for Conv1d of Keras. For input_shape (4,1) is needed.


Neural network does not converge with negative symbols I've created a simple 2-2-1 feedforward ANN to predict an XOR using Keras.

The activation function I'm using on all layers is a tanh, so in order to make use of the entire range of the function, i.e. [-1, 1], I've decided to use -1 instead of 0 as the symbol.

My input data is, thus, [[-1, -1], [-1, 1], [1, -1], [1, 1]], for an output of [[-1], [1], [1], [-1]].

I thought this would give better results since I'm using the entire range of the function and it supposedly would converge better because of that. Also, since I'm just using a different symbol, it should be the same as with using 0 and 1.

However, my network can not converge (giving a 0.5 accuracy), and what baffles me the most is that using 0 and 1 as symbols converges, and at a much faster rate.

Is there a reason for such a counter-intuitive (at least in my conception) thing to be happening?

I tried this experiment and was able to get some positive results. I will describe what I tried then perhaps you can specify where the differences may lie and we can further explore them. From what I tried I would assume that you are simply not training long enough.

Creating the data

import numpy as np

n = 100000
x_train = np.zeros((n,2))
y_train = np.zeros((n,))
for i in range(n):
    x_train[i,0] = np.random.choice([-1,1])
    x_train[i,1] = np.random.choice([-1,1])
    if x_train[i,0] == 1 and x_train[i,1] == 1 or x_train[i,0] == -1 and x_train[i,1] == -1:
        y_train[i] = -1
    else:
        y_train[i] = 1

x_train = x_train.reshape(n, 2,)

n = 1000
x_test = np.zeros((n,2))
y_test = np.zeros((n,))
for i in range(n):
    x_test[i,0] = np.random.choice([-1,1])
    x_test[i,1] = np.random.choice([-1,1])
    if x_test[i,0] == 1 and x_test[i,1] == 1 or x_test[i,0] == -1 and x_test[i,1] == -1:
        y_test[i] = -1
    else:
        y_test[i] = 1

x_test = x_test.reshape(n, 2,)

print(x_test[0].T)
print(y_test[0])

  [ 1.  1.]
  -1.0

Build the model

As you describe the model is 2 input nodes, 2 hidden nodes and 1 output node. Every node is using tanh as its activation function.

input_shape = (2,)

model = Sequential()
model.add(Dense(2, activation='tanh',
                 input_shape=input_shape))
model.add(Dense(2, activation='tanh'))
model.add(Dense(1, activation='tanh'))

model.compile(loss=keras.losses.mean_squared_error,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

Train the model

Because I generated many instances of the data I am only training for 10 epochs. However, if your input space is only the four possible inputs you may want thousands of epochs. Neural networks do take a long time to converge.

epochs = 10
batch_size = 128
# Fit the model weights.
history = model.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test, y_test))

  Epoch 10/10 100000/100000 [==============================] - 1s
  9us/step - loss: 9.3983e-05 - acc: 1.0000 - val_loss: 7.9096e-05 -
  val_acc: 1.0000


Linear Regression and learning rate Could you please tell me why do we use a learning rate to move into the direction of the derivative to find the minimum? Why is it not good if you simply count it where is it 0?

Learning rate gives the rate of speed where the gradient moves during gradient descent. Setting it too high would make your path instable, too low would make convergence slow. Put it to zero means your model isn't learning anything from the gradients.


K-fold cross-validation with validation and test set For a project I want to perform stratified 5-fold cross-validation, where for each fold the data is split into a test set (20%), validation set (20%) and training set (60%). I want the test sets and validation sets to be non-overlapping (for each of the five folds).
This is how it's more or less described on Wikipedia:

A single k-fold cross-validation is used with both a validation and test set. The total data set is split in k sets. One by one, a set is selected as test set. Then, one by one, one of the remaining sets is used as a validation set and the other k - 2 sets are used as training sets until all possible combinations have been evaluated. The training set is used for model fitting and the validation set is used for model evaluation for each of the hyperparameter sets. Finally, for the selected parameter set, the test set is used to evaluate the model with the best parameter set. Here, two variants are possible: either evaluating the model that was trained on the training set or evaluating a new model that was fit on the combination of the train and the validation set.

Right now, I've implemented something that looks like the following (described here):
kf = KFold(n_splits = 5, shuffle = True, random_state = 2)

for train_index, test_index in kf.split(X):
    X_tr_va, X_test = X.iloc[train_index], X.iloc[test_index]
    y_tr_va, y_test = y[train_index], y[test_index]
    X_train, X_val, y_train, y_val = train_test_split(X_tr_va, y_tr_va, test_size=0.25)
    print("TRAIN:", list(X_train.index), "VALIDATION:", list(X_val.index), "TEST:", test_index)

Although it does quite nicely provide me with 5 folds whereby for each fold I have a validation set, test set and training set, the validation sets have overlap between each of the five folds (so some instances present in fold 1 are also present in fold 2, etc.). For the test sets this is not the case (i.e. they do not have overlap between folds).
Is there a way to prevent this overlap between folds from occurring for the validation set as well?

You can leave a portion of your dataset as a hold-out test set for the final model validation from the beginning, and proceed with the k-fold cross-validation strategy with the rest of the data as follows:

source: https://scikit-learn.org/stable/modules/cross_validation.html
This way, you do not have the overlap you mention and, mainly, you make sure that the model is finally evaluated with a never-seen before data.


Loss function and data distribution for positive / negative examples I have an AlexNet architecture that predicts in a supervised-way, whether a given image belongs into classes 1-10. But it should also recognize if it doesn't fit into any of these classes.

My thought was that I just add an 11th class and label all my negative-data with this class and train the softmax-cross-entropy loss with Gradient Descent Optimizer (Adam) as usual.

Now my question is if this is a valid approach on solving my problem?

I have around 7.000 images per class and could obtain up to 70.000 negative examples. I have the strong feeling that training with 70.000 negative-examples my network would always predict the outcome to belong to class 11 (not matching any other). Am I correct with the assumption? If yes: How many negative examples would do the job?

Your approach of adding negative examples to your data set, replacing the 10-output softmax final layer with an 11-output softmax, and retraining is certainly valid.

You need these additional examples because the network can only learn to classify categories that it has seen before.

Retraining the final layer only is a good first attempt because it is the cheapest, but if you don't get good results you can try fine tuning one or more additional layers at the end of the network.

All these techniques fall under the topic of Transfer Learning.

Regarding the second part of your question about how many negative examples to include - this is known as Class Imbalance Problem in Machine Learning.

Whether your model will always predict the negative class if you include many negative examples depends on the data and the model, but it is certainly a possibility.

The safest approach would be to undersample your negative examples and include only 7000 to match the frequency of other classes in the training set. This should be a good first try, but if it doesn't work you should consider a couple of questions:

What is the expected distribution of my target population?
What is the cost of misclassifying each type of example?

And then make an informed decision about how to proceed, which may involve either sampling negative examples differently, or adjusting your performance metric to something like AUC (which can guide early-stopping, for example).

Additional material:
A systematic study of the class imbalance problem in convolutional neural networks


No module named "" error when loading the pickle file I created a model and I saved it in a pickle file using the Algorithm SVR(Support Vector Regression)

import pickle
pickle.dump(model,open('carb patients data/Pickles/svr.pickle', 'wb'))

In jupyter notebook it gives an error

  can't pickle _thread.RLock objects

So I converted that jupyter file in to a .py file and downloaded it and executed using the Python Idle. Then it got saved in that particular location. But when I try to load my pickle file from another Jupyter Notebook it gives an error called,

  No module named 'sklearn.svm._classes'

The code how i tried to load the pickle file,

from flask import Flask, request, redirect, url_for, flash, jsonify
import numpy as np
import pickle as p
import json
from sklearn import ensemble
app = Flask(__name__)

@app.route('/api/', methods=['POST'])
def makecalc():
  data = request.get_json()
  prediction = np.array2string(model.predict(data))

  return jsonify(prediction)

if __name__ == '__main__':
   modelfile = 'Clustering Patients/carb patients data/Pickles/svr.pickle'
   model = p.load(open(modelfile, 'rb'))
   app.run(debug=True, host='0.0.0.0')

Any thing I have done wrong or is it an issue in the Jupyter Notebook? How can I load my saved pickle file?

According to the scikit-learn model persistence docs, it may be better to use joblib instead:

Save model

from joblib import dump

dump(model, 'filename.joblib')

Load model

from joblib import load

model = load('filename.joblib')


What techniques to use for image matching I have a database with around 30,000 pictures. All of them are a different object. They are all from a certain perspective, the pictures itself are the same size but the objects vary in size. I want to build a system that you can query with a new picture, and that will return it's nearest neighbor, given that it is similar enough. The queried images will look relatively similar to the originals, there might be some horizontal and or vertical translations, a bit different lightning and sometimes there will be a sticker on a different place. Some queried objects will not be in the set, and that needs to be returned as well. What are good techniques to try and what would be the downside? Getting multiple pictures of each object is infeasible. Here's some ideas, I'm wondering if there is more to try:

Euclidean distance on the raw data (very sensitive but fast)
Use traditional keypoint matching, linear matching is very slow unfortunately
Use (denoising) autoencoder for lower dimensional feature representation, linear match on this encoded space (smallest Euclidean distance, at least faster linear search)
Learn siamese network for linear matching (don't know how fast this works but seems slow too)
Learn deep binary autoencoder onto 28 bits which allows for very quick narrowing of the search space to do one of the previous methods, by using these bits as memory mapping to a list of candidate solutions

Any other ideas?

Hashing is the way to go if you want fast -- constant time -- retrieval of nearest neighbors. Here's a recent example using neural networks to learn a binary hash:

Deep Learning of Binary Hash Codes for Fast Image Retrieval (code) (slides)

You want to avoid doing all-pairs computations like correlations.


RAM crashed for XML to DataFrame conversion function I have created the following function which converts an XML File to a DataFrame. This function works good for files smaller than 1 GB, for anything greater than that the RAM(13GB Google Colab RAM) crashes. Same happens if I try it locally on Jupyter Notebook (4GB Laptop RAM). Is there a way to optimize the code?
Code
#Libraries
import pandas as pd
import xml.etree.cElementTree as ET

#Function to convert XML file to Pandas Dataframe
def xml2df(file_path):

  #Parsing XML File and obtaining root
  tree = ET.parse(file_path)
  root = tree.getroot()

  dict_list = []

  for _, elem in ET.iterparse(file_path, events=("end",)):
      if elem.tag == "row":
        dict_list.append(elem.attrib)      # PARSE ALL ATTRIBUTES
        elem.clear()

  df = pd.DataFrame(dict_list)
  return df

Part of an XML File ('Badges.xml')
<badges>
  <row Id="82946" UserId="3718" Name="Teacher" Date="2008-09-15T08:55:03.923" Class="3" TagBased="False" />
  <row Id="82947" UserId="994" Name="Teacher" Date="2008-09-15T08:55:03.957" Class="3" TagBased="False" />
  <row Id="82949" UserId="3893" Name="Teacher" Date="2008-09-15T08:55:03.957" Class="3" TagBased="False" />
  <row Id="82950" UserId="4591" Name="Teacher" Date="2008-09-15T08:55:03.957" Class="3" TagBased="False" />
  <row Id="82951" UserId="5196" Name="Teacher" Date="2008-09-15T08:55:03.957" Class="3" TagBased="False" />
  <row Id="82952" UserId="2635" Name="Teacher" Date="2008-09-15T08:55:03.957" Class="3" TagBased="False" />
  <row Id="82953" UserId="1113" Name="Teacher" Date="2008-09-15T08:55:03.957" Class="3" TagBased="False" />

This conversion in needed so that I can perform furthur Data Analysis.
I have asked this on StackOverflow (Link) but the answers did not solve my query. I hope to find a solution here.

import dask
import dask.bag as db
import dask.dataframe as dd
from dask.dot import dot_graph
from dask.diagnostics import ProgressBar

dask.set_options(get=dask.multiprocessing.get)
tags_xml = db.read_text('data/Tags.xml', encoding='utf-8')
tags_xml.take(10)

Refer this link for complete tutorial Dask with XML


Soft and hardware for manipulating multi-million record data sets If starting from scratch, what hardware and software would be ideal and under \$5k or \$10k for manipulating a data set of 10 to 20 million records without slowing to the crashing point or taking several hours to complete? Manipulation, if it matters, means multivariate regression (simple OLS with less than a dozen variables), basic graphs, and tables.

Use vowpal wabbit for building your regression or classification models. It'll train at the speed at which you can read your data

Use pandas and dask for plotting graphs etc.
You can use an AWS i2.8xlarge or equivalent machine which will cost you ~ USD 6k for an year


"Random Forest" variant of other classifiers I am using SVM with linear kernel for one of my multiclass text classification problem and getting accuracy of 0.78. I also tried Random Forest but the best accuracy I could get was 0.72. This set me thinking what if I could simulate Random Forest but with a SVM classifier and not decision tree. I am not sure whether it will result in improved accuracy or not. I wanted to have something in favor before I implemented this and a quick google search didn't help either.

Any thoughts on this?

The notion of ensembles of models leading to better outcomes is widespread in the Data Science and Machine Learning communities. For example, the widely read text 'The Elements of Statistical Learning' by Hastie, Tibshirani and Friedman, devotes space to the general notion of model averaging (section 8.7-8.8) with further discussion of boosting in relation to trees (e.g. chapter 10). Hence, it is common to see examples of ensembled models in the wild, using any conceivable algorithm as the base, even to the point of effectively seeing ensembles of ensembles.

Specifically with respect to Support Vector Machines, there are at least a few attempts at improving SVM performance by using ensembles. A recent example is EnsemblesSVM (Claesen,De Smet, Suykens, De Moor - see homes.esat.kuleuven.be/~claesenm/ensemblesvm/). The authors promise that this algorithm will lead to better training times by ensembling lightly trained SVMs, and distribute a free implementation via the website above.


Differences between big data, data warehousing, business intelligence and data science? I know they are four different areas, but I would like to know what are the main differences between those disciplines, and how they are related to each other, if some of them depend on each other, and what is the specific objective of each one.

The simplest way to explain:

Big data is essential the data itself. It is "big" because of the size and scale of it. You can think of this as the datasets that are composed up of hundreds of thousands or millions of observations. With more people online, more data is generated and this is what people mean by big data.
Data warehousing is the storage of all of these massive amounts of data. You can think of this as the place where big data is stored. This can be on large servers or large databases or even in the cloud.
Business intelligence is the use of big data (or small data) to draw insights that could be useful for making business decisions or applications. This is essentially leveraging the data to help improve aspects of a company.
Data science is a very broad term that can encompass topics like ML, data analysis, Artificial Intelligence, statistics etc. There is no one specific definition but more generally can mean anything having to do with working with data these days.


unbalanced data classification I used XGBoost to predict company's bankruptcy, which is an extremely unbalanced dataset. Although I tried weighting method as well as parameter tuning, the best result which I could obtain is as follows:

Best Parameters: {'clf__gamma': 0.1, 'clf__scale_pos_weight': 30.736842105263158, 'clf__min_child_weight': 1, 'clf__max_depth': 9}
Best Score: 0.219278428798
Accuracy: 0.966850828729
AUC: 0.850038850039
F1 Measure: 0.4
Cohen Kappa: 0.383129792673
Precision: 0.444444444444
recall: 0.363636363636

Confusion Matrix:
[[346   5]
 [  7   4]]

As the confusion matrix shows my model can not identify bankrupted companies very well, which results in poor performance measures such as precision, recall, Cohen's kappa, F measure. Also, I tried  BlaggingClassifier, which is presented here. Really, it gives the following result:

Best Parameters: {'clf__n_estimators': 64}
Best Score: 0.133676613659
Accuracy: 0.809392265193
AUC: 0.819606319606
F1 Measure: 0.188235294118
Cohen Kappa: 0.142886555487
Precision: 0.108108108108
recall: 0.727272727273
Confusion Matrix: [[285  66]
 [  3   8]]

As it is shown, it predicts positive classes well but it does poor on negative class ( too many false positive).
Could you please let me know how it is possible to combine these two classifier results in order to obtain a better result? A simple way to combine two classifier is to use a convex linear combination for predicted probabilities: t * p1 + (1 - t) * p2, where 0 <= t <= 1 and p1, p2 are predictions of the two classifiers. Then , I should search for an optimal value of t over a grid but I don't know how to do it?

I read that anomaly detection for example,one class svm and Isolation Forest could be used for extremely unbalanced data set, so could you please let me know how to do it i.e., by an example code?
In general, I would appreciate if you could let me know how to deal with this issue.

For starters, I am not confident that combining the results will give you what you expect. Have you checked if the true negative remain the same in both occurrences?

Moreover, have you tried to alternate the hyper parameters on the XGBoost model responsible for class balancing like max delta step or scale pos weight.

You can also try to use Sampling techniques for Over Sampling and Under Sampling. Or try possibly using many trees on  XGBoost on a different random Under Sample.

P.S. I am interested on the anomaly detection paper you mentioned, could you provide a link, thnx.


Present results of the best or the last iteration on dev set? Which is the correct way - presenting the results of the best or the last iteration on the dev set in a paper? In research papers I usually see only one value, is it the best iteration of all?

I'm asking because my model training last iteration wasn't the best one. Instead the best iteration was somewhere in the middle of the training.

Is dev set a validation set here? Your best model occurred where the error on the validation set was lowest. That's the model you choose. However the validation error will be a slightly optimistic estimate of the true generalization error. Ideally you have held out a test set as well, and evaluate and report error on that.


How to approach automated text writing? What are the tools, practices and algorithms used in automated text writing?

For example, lets assume that I have access to wikipedia/wikinews and similar websites API and I would like to produce article about "Data Science with Python".

I believe that this task should be divided into two segments. First would be text mining and second would be text building. I'm more or less aware how text mining is performed and there are lots of materials about it in Internet. However, amount of materials related to automated text building seems to be lower. There are plenty of articles which says that some companies are using it, but there is lack of details. Are there any common ideas about such text building?

You should probably do some reading in the field of "Natural Language Generation", since this seems to relate most directly to your question.

But the way you have described the process -- "text mining...text building" -- leads me to wonder if you are aiming for something much more ambitious.  It seems as though you aim to automate the process of 1) reading natural language texts, 2) understanding the meaning, and then 3) generate new texts based on that semantic knowledge.  I'm not aware of any general-purpose end-to-end systems that can do that, not even specialized systems by the likes of Palantir.  What you are aiming for would probably pass the Turing Test for fully capable Artificial Intelligence.


F1 on test set using Naive Bayes classifier decreases as training set size increases. Why? I've done a simple naive bayes classification task with a very small data set. As the training set size increases from 100 data points to 300 data points, the F1 score on the test set decreases. But why? Is it likely overfitting? What are the first places I should examine to explain this behavior?

Overfitted model is something that shows very less error on your training set and then when you test it on a test set, it fails badly. This might be due to class imbalance and you might be giving something that it has not seen or due to using more number of features than you were supposed to, making the model to fit in every datum. Wiki page is nice overview read for this.

Answering your next question, F1 score as might already be knowing has two components in it, Precision and Recall. Check whether there are reasonable number of TP,TN when compared to FN, FP and if you find those numbers to be fair enough, then F1-score is just a metric for you. There is no benchmark score for an awesome F1 score. The decrease might be just because your model has just more data to predict now.

So overfitting can lead to less F1-score but reduced F1-score does not mean overfitting always.


Equally sized topics in Latent Dirichlet allocation I'm using the topicmodels package for R to cluster a big set of short texts (between 10-75 words) into topics. After manually reviewing a few models it seems like there are 20 realtivly stable topics. However, what I find really weird is that they are all roughly the same size! Each topic catches around 5% of tokens and 5% of texts. In terms of the tokens, the smallest topic is 4.5% the largest 5.5%.

Can anybody suggest if this a 'normal' behaviour? This is the code I'm using:

ldafitted <- LDA(sentences.tm, k = K, method = "Gibbs",
             control = list(alpha = 0.1, # default is 50/k which would be 2.5.  a lower alpha value places more weight on having each document composed of only a few dominant topics
                            delta = 0.1, # default 0.1 is suggested in Griffiths and Steyvers (2004).
                            estimate.beta = TRUE,
                            verbose = 50, # print every 50th draw to screen
                            seed = 5926696,
                            save = 0,    # can save model every xth iteration
                            iter = 5000,
                            burnin = 500,
                            thin = 5000, #  every thin iteration is returned for iter iterations. Standard is same as iter
                            best = TRUE)) #only the best draw is returned

In short: My question is if there are circumstances under which it is reasonable that Latent Dirichlet allocation will cluster text in topics of equal size? Or is it something I should be worried if it happens?

It is normal, the best explantation I found for it, is from physics. Since Gibbs Sampling was known in physics long before LDA and LDA can simply be seen as a kind of matrix factorization.
There is a system which has particles (words) and the particles can be in different states (topics). States with lower energy have a higher probability of being occupied than states with higher energy.
Or simply: Since LDA is a dimensity reduction method that just compresses a words x documents matrix into a topics x documents and a words x topics matrix, the most effective way to do it is by maximizing the entropy which is done if the clusters get equal size.

Just noticed that you can simply derive the above assumption from the dirichlet distribution itself: Take a look at the density (PDF) function and ignore the normalizing factor at the front and only assume symmetric alpha parameter. This results in a product term of X1...XK, that product term gets it's maximum if all X1...XK's have equal size, since the sum X1+X2+...+XK = 1, it's the same case as entropy above.