Finding outliers in Image dataset I have been working on an image classification tasks for which I am extracting the image frames from the video stream collected for different classes.

I have already trained an image classification model (using transfer learning) however due to the outliers (or overlap in the class distribution) accuracy of the model is poor. And not able to generalize the new images / video streams.

Could you please help me with the below queries

How the sample is distributed in each class ? Can I use any visualization techniques (for example : histogram) to see the sample distribution.
And also going through the image one by one is tedious process so is there a technique with which I can find the outliers (outlier images) from the samples. So that I can remove outliers before training the model.

Any updates on this..

Thank you

BiGAN, Bi-directional GAN(variation of GAN(Generative Adversarial Network)) can be explored for anomaly detection.
I have used for my use case. Since I dont have large image dataset, my results are not that good.


Represent outlier days I have hourly power consumption data for 30 days. On representing, each day data using a separate line, I get a plot as

I want to highlight the days with abnormally high consumption (in other words, the outlier days). I think that the current plot is too much congested. Is there any other better representation to show the outlier days?

One idea would be to plot the daily average power consumption in a bar plot:

For a finer visualization of day-hour peaks, you can try to plot it in a matrix format:


Naive Bayes Classifier Could someone please explain to me how and why can we go from equation $4.3$ to equation $4.4$:

$$\hat{c}= \arg\max_{c \in \mathcal{C}}P(c|d) = \arg\max_{c \in \mathcal{C}}\frac{P(d|c)P(c)}{P(d)}\tag{4.3}$$

$$\hat{c}= \arg\max_{c \in \mathcal{C}}P(c|d) = \arg\max_{c \in \mathcal{C}}P(d|c)P(c)\tag{4.4}$$

We are trying to select the optimal $c$, here $d$ is fixed and hence $P(d)$ and $\frac1{P(d)}$ is just a positive constant.

Multiplying an objective function with a positive constant doesn't change the optimal solution, hence we can drop $P(d)$.


Why is rpart not splitting this data even when there is gain in gini? df <- tibble(x1=factor(c("S1", "S1", "S2", "S2")), y=factor(c(1, 1, 0, 1)))
md <- rpart(formula=y~., data=df, method="class", control=rpart.control(minsplit=2, cp=0))
nrow(md$frame) #outputs 1

consider the split
left child node :
"S1", 1
"S1", 1
right child node :
"S2", 0
"S2", 1
Here the gain in gini would be 1/8 = 0.125
why is rpart not doing this split?

It seems that rpart is actually using accuracy rather than gini in the cost complexity pruning, see e.g. https://stats.stackexchange.com/a/223211/232706
Since your split doesn't improve the misclassification rate, rpart doesn't make it even with cp=0.  Setting cp=-1, the split is performed.


TypeError: Expected binary or unicode string, got [ ERROR SUMMARY:

I'm getting the following error:

TypeError: Expected binary or unicode string, got [

BACKGROUND:

I have several features that are histories of user activity. I am trying to predict whether a given user will take an action (represented by a 0 or 1 in my y_train list below) based off of their histories of different types of actions. For example, one feature might be button_A_click_per_day and the history for a given user would be a vector of button A clicks per day over the past 365 days.

The relevant snippets of my code is as follow:

import tensorflow as tf

# Build feature columns for classifier
feature_columns = []
for key in X_train:
    col = tf.feature_column.numeric_column(
        key=key,
        shape=max_width,
    )
    feature_columns.append(col)

# Build classifier
classifier = tf.estimator.DNNClassifier(
    feature_columns=my_feature_columns,
    hidden_units=[10, 10],
    n_classes=2,
)

And X_train is structured as follows:

>>> X_train['<feature_name>']
129     [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
...
1294    [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
860     [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
Name: <feature_name>, Length: 1377, dtype: object

My train input function is as follows:

def train_input_fn(features, labels, batch_size):
    dataset = tf.data.Dataset.from_tensor_slices((dict(features), labels))
    dataset = dataset.shuffle(buffer_size=1000).repeat(count=None).batch(batch_size)
    return dataset.make_one_shot_iterator().get_next()

Finally, when issuing the following command...

classifier.train(
    # input_fn is a fn that takes not arguments and returns an iterator.
    input_fn=lambda: train_input_fn(X_train, y_train, batch_size=100),
    steps=1000,
)

I get the error:

~/.pyenv/versions/anaconda3-5.0.1/lib/python3.6/site-packages/tensorflow/python/util/compat.py in as_bytes(bytes_or_text, encoding)
     63   else:
     64     raise TypeError('Expected binary or unicode string, got %r' %
---> 65                     (bytes_or_text,))
     66
     67

TypeError: Expected binary or unicode string, got [0.0, 0.0, 0...

MY QUESTIONS:

Do you know what I'm doing wrong to get this error?
Is there a better way to model features that are histories?

You should use a built-in function for train_input_fn instead of writing your own. For example:

train_input_fn = tf.estimator.inputs.numpy_input_fn(
    x={"x": np.array(training_set.data)},
    y=np.array(training_set.target),
    num_epochs=None,
    shuffle=True)

There is a complete example in the TensorFlow documentation here.


Bounded ReLU at 1 if x>1? Is there a closed form solution for a ReLU that is bounded with a maximum value at 1?
I am trying to produce output values for pixel intensities 0 <= x <= 1, but my outputs are producing values greater than 1. How can I counteract this?

Being itself a non-linearity, the very advantage of ReLUs is their linear regime, that prevents it from saturating too early.

Having said that, a couple of alternatives for your case are to either clip the output of the ReLUs or to place sigmoid (or tanh, adjusting the output range) after them (or instead of them).


why is MSE of prediction way different from loss over batches I am new to machine learning so forgive me if i ask stupid question. I have a time series data and i split it into training and test set.

This is my code:

from numpy import array
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

# split a univariate sequence into samples
def split_sequence(sequence, n_steps_in, n_steps_out):
    X, y = list(), list()
    for i in range(len(sequence)):
    # find the end of this pattern
        end_ix = i + n_steps_in
        out_end_ix = end_ix + n_steps_out
        # check if we are beyond the sequence
        if out_end_ix > len(sequence):
            break
        # gather input and output parts of the pattern
        seq_x, seq_y = sequence[i:end_ix], sequence[end_ix:out_end_ix]
        X.append(seq_x)
        y.append(seq_y)
    return array(X), array(y)

# choose a number of time steps
n_steps_in, n_steps_out = 10, 5
# split into samples
X, y = split_sequence(trainlist, n_steps_in, n_steps_out)
# define model
model = Sequential()
model.add(Dense(100, activation='relu', input_dim=n_steps_in))
model.add(Dense(n_steps_out))
model.compile(optimizer='adam', loss='mean_squared_error')
# fit model
history = model.fit(X, y, epochs=2000, verbose=0)
# demonstrate prediction
x_input = array(testlist[0:10])
x_input = x_input.reshape((1, n_steps_in))
yhat = model.predict(x_input, verbose=0)
yhat=list(yhat[0])

when i do print(history.history['loss'][-10:-1]) it gives me roughly 0.55 and when i do

from sklearn.metrics import mean_squared_error
mean_squared_error(testlist[11:16],yhat)

it gives me 0.11. Why is it so different?

From my understanding, you are comparing the prediction error on the test set versus the training loss error. When the training error is greater than the test error, that means your model is under-fitted.

Under-fitting occurs when there is still room for improvement on the test data. This can happen for a number of reasons: If the model is not powerful enough, is over-regularized, or has simply not been trained long enough. This means the network has not learned the relevant patterns in the training data. There are number of ways you can resolve under-fitting problem:

Increase the time of training procedure
Increase number of parameters in your model or the complexity of your model


How to visualize categorical data with numeric columns I have data like as shown below

I would like to represent the above tabular data in a visual form.
However, the below graph may not work because my real data as 50K unique drug names.

So, is there any better way to represent this? Can you share some suggestions on how this can be represented?

Clearly there's no way to have the names of the drugs.
Assuming the relation between the two columns is important, a scatter plot with units prescribed as X and number of patients as Y might work. You could even add the name of the drug for a few isolated points. Transparency/opacity can be used to show the dense areas.
In case the relation between the columns is not important, you could just plot the two distributions (histograms) with different colours on the same graph.


Why is r squared lowered when adding polynomial features? I am trying to find a best fit line f(x) = ? for a random set of x,y coordinates.

Linear Regression with polynomial features works well for around 10 different polynomials but beyond 10 the r squared actually starts to drop!

If the new features are not useful to the Linear Regression I would assume that they would be given a coefficient of 0 and therefore adding features should not hurt the overall r squared.

I reproduced this problem when housing price predictions when creating a large amount of interaction features.

I have my python code below:

Create Random Data

import numpy as np
import matplotlib.pyplot as plt

def pol(x):
    return x * np.cos(x)

x = np.linspace(0, 12, 100)
rng = np.random.RandomState(1234)
rng.shuffle(x)
x = np.sort(x[:25])
y = pol(x) + np.random.randn(25)*2

plt.scatter(x, y, color='green', s=50, marker='.')

plt.show()

Regress and Check Each R Squared

from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures

for p in range(1,30):
    plot_range = [i/10 for i in range(0,120)]
    poly = PolynomialFeatures(p)
    X_fin = poly.fit_transform([[samp] for samp in x])
    X_fin_plot = poly.fit_transform([[samp] for samp in plot_range])
    reg = LinearRegression().fit(X_fin, y)

    from sklearn.metrics import mean_squared_error, r2_score
    print(p,r2_score(y, reg.predict(X_fin)))

Display Last Regression Line

plt.scatter(x, y, color='green', s=50, marker='.')
plt.plot(plot_range,reg.predict(X_fin_plot))
plt.show()

I also have two plots to compare. The first is with 10 polynomial features and the second is with 40. Notice how the second misses the majority of the first points.


Problem trying to build my own sklean transformer I build the following sklearn transformer :

class Cat2Rat( BaseEstimator , TransformerMixin ) :
   def __init__( self , col2trans ) :
      self._col2trans = col2trans

   def fit( self, X, y = None ):
      return self

   def transform( self , X , y = None ) :
      for col in self._col2trans :
         p = X[ col ].value_counts() / X.shape[0]
         dic = dict( [ ( i , p[i] ) for i in ( X[ col ].value_counts() ).index ] )
         X.replace( { col : dic } , inplace = True )
      return X

This transformer is replacing catagorical values by their rates.

For example :

myarray = np.array([ [ 1 , 1 , 3 , 'v' , 0 ] ,
                 [ 2 , 2 , 2 , 'v' , 1 ] ,
                 [ 4 , 5 , 1 , 'w' , 1 ] ,
                 [ 2 , 1 , 9 , 'w' , 1 ] ,
                 [ 1 , 0 , 4 , 'w' , 1 ] ] )

colnames = [ 'one', 'two', 'three' , 'four' , 'target' ]

df = pd.DataFrame( myarray , columns = colnames )

Value 'v' ( 'w' ) for column 'four' is replaced by 2/5 ( 3/5 ).

My purpose is to fit the transformer on df and apply it to another dataframe df2 :

myarray2 = np.array([ [ 2 , 7 , 3 , 'v' , 0 ] ,
                    [ 9 , 2 , 2 , 'v' , 0 ] ,
                    [ 4 , 5 , 1 , 'w' , 1 ] ]  )

colnames2 = [ 'one', 'two', 'three' , 'four' , 'target' ]

df2 = pd.DataFrame( myarray2 , columns = colnames2 )

I am doing this that way :

# Transformer instance
trsf = Cat2Rat( [ 'four' ] )

# Fitting
trsf.fit( df )

# Then applying
trsf.transform( df2 )

But the rates are those of column 'four' values of df2 not df ( on which the transformer was fitted ).

I must have been missing something on the way to properly build such a transformer.

Could someone give some clue on how to fix the transformer so that it gives proper result?

Thanks.

The definition of dic should happen inside the fit method; you're wanting the rates to be based on the dataframe passed to fit.  So you'll want to set dic as another class attribute, to be referenced in the transform method.

(This kind of replacement is called target encoding, and there's an existing package to do it in sklearn.)


Intuition behind the PCA algorithm I am trying to understand PCA intuitively. Here it goes: After finding the eigenvectors and eigenvalues of the covariance matrix of the dataset, the eigenvalues will represent how spread out the dataset is, and the eigenvector will represent the direction of the eigenvalue. And what we are trying to do is to order the eigenvector from highest eigenvalue to lowest (highest meaning the features are more spread out hence more independent of each other and less redundant) and project the original dataset onto the new set of "axis" (the eigenvectors). For example, if we have a 3-D dataset, then there will be three eigenvectors each orthogonal to each other (so they will form a 3-D space). Next, we project the original dataset onto this new set of "axis" space, and if we want to reduce to 2-D, we would take out the least significant "axis" (smallest eigenvalue)

I am familiar with the technical steps of performing PCA, but I am struggling to understand the actual intuition behind the algorithm. Were there any mistakes or flaws in my explanation above, especially the part of explaining dimension reduction with the use of the word 'axis'? Thanks a lot in advance!

In simple words, PCA (Principal Component Analysis) is a dimension-reduction method where you try to reduce the number of variables from a larger number to a smaller one without any loss of the important information available in the data.
Mathematically, it is a projection of a higher-dimensional object in a lower-dimensional vector space.

A great example in a real-life scenario is where we see 3D objects in a 2D Television. Reference

Another good explanation on the intuition on PCA

Hope it helps.


Detecting unique icon on video in real time For a project I am working on I'd like to be able to detect unique icons/barcodes in video footage. Suppose you have 10 people in the frame where each person is wearing a t-shirt with a similar but unique identifier (something like a barcode) and the programme is to detect each unique identifier.

Here are three options I'm working with now and associated drawbacks:

Using brute force feature matching (SURF, SIFT or ORB) - not sure how accurate this would be in real-world scenarios as people move and rotate so to if the identifiers are very similar can this still achieve good accuracy?

Training tiny YOLO model. Now this seems like a good option but is not very scalable and it just seems like there's a simpler solution
Simple template matching - is not a very good option due to changing perspective, size and angles

Any advise would be highly appreciated!

EDIT

I'd like to add some images for context. The idea is that each person would have a logo with a unique binary patter around it. See images below:

As you can see the two are similar but not the same due to the stripes around it.

I'm applying an Adoptive Gaussian Blur to it and getting 'barcode' like images that are clearly different

Now, it doesn't seem to me that normal SURF matching would be able to recognise the subtle difference in two images at least not reliably

Well, the first thing you need to consider about bar codes/QR Codes is that they are usually black with white background.

If T-Shirts or Unique tags are been used, they should follow this pattern (or any one). You should be able to easily detect these. Take for example this image processing class at the end there is a bar-code example, which here I reproduce:

0 - Given a image:

1 - You can binarize by converting to grayscale and then performing adaptive threshoulding, which is basically a clustering algorithm for pixels:

2 - Then you can filter blobs by eccentricity

3 - Then by major axis:

4 - And finally by orientation

This is a basic algorithm for segmenting bar-codes on hard surfaces. A T-shirt will render distorted bar-codes, but if you can narrow down the search region by detecting probable bar-codes in binary images you probably will get a large speed-up for then use of more complex algorithms such as the ones you propose.

Suggestions:

1 - Add a FCN to regenerate image distorted by T-Shirt surface before step (3) in the previous algorithm

2 - Train a CNN to detect bar-code blobs on binary images (substituting steps 2,3 and 4)

Update

I will add another answer since you changed a bit the question by giving examples. Your unique identifiers seen to be compose of a fixed center (equal to all identifiers) with concentric arcs of 3 different radius $r_1 < r_2 < r_3$ and the arcs with radius $r_2$ and $r_3$ are punctured with holes that are supposed to make them different from in each identifier.

You can use threshoulding and Hough transform to identify the circles, you can use the center as reference to calculate the angular distance between the holes. Then you can differ an identifier from another by:

Counting holes
and then
Analyzing their angular distance

You can also use the arcs to predict/calculate the transformations suffered by the identifier by image distortions (rotation, shifting, non-linear transformation) and correct them.


How to encode a time series as an image to feed it into CNN? I want to try CNN in the task of stock chart pattern recognition. I suspect that feeding a line chart won't work because the image will have a lot of empty pixels.

What time series encoding options are available to get a lossless, dense image?

You can encode time series to images using image encoding methods like Recurrence Plots (RP), Gramian Angular Field (GAF), Markov Transition Field (MTF).

See the following article:

Estebsari, A.; Rajabi, R. Single Residential Load Forecasting Using Deep Learning and Image Encoding Techniques. Electronics 2020, 9, 68.
https://doi.org/10.3390/electronics9010068


Neural Network Hidden Layer Selection I am trying to build an MLP classifier model on a dataset containing 30000 samples and 23 features.
What are the standards I need to consider while selecting the number of hidden layers and number of nodes in each hidden layer?

First try a simple model: The input layer and the output layers dimension are defined by your data / your problem definition. Then train a model without any hidden layer.
See how good it performs. Is it good enough? If yes, you're done. If no, continue
Add a hidden layer of reasonable size or adjust a hidden layers size. Go to step (2).

The "reasonable" size part might be difficult. As a guidance: If you have only a single node, it is certainly too small for a 1000 class problem. It might be big enough for a 2 - 3 class problem. I would usually suggest to keep the size of the features per layer roughly constant or at most reduce it by 1/10 or triple it. But that is only gut feeling.

The reason for my preference for simple models is Occam's razor, the fact that they are often faster, easier to analyze and to manually improve.

For more information about topology learning and rules how to design neural networks, see:

  Thoma, Martin. "Analysis and Optimization of Convolutional Neural Network Architectures." arXiv preprint arXiv:1707.09725 (2017).

Especially chapter 2.5 and chapter 3.


Using TensorFlow's preexistent Inception labeler I know I can retrain Inception to label images, but can I just provide an image to a non retrained Inception and get a label back?

Yes, but the labels are limited to the ones from ImageNet.


How to pad a batch of documents? Hello PyTorch experts:
Sentences and documents both can be variable length.
Lets say, we have following 2 docs:
doc1=[torch.tensor([1,2,3,4]),torch.tensor([4,5,6]),torch.tensor([7,5])]
doc2=[torch.tensor([1,2,3]),torch.tensor([4,5])]

(Here, each tensor is a sentence and each number in the tensor is an index of embedding matrix)
doc1=pad_sequence(doc1,batch_first=True)
doc2=pad_sequence(doc2,batch_first=True)

batch=pad_sequence([doc1,doc2],batch_first=True)

This will throw an error because the length of the longest sentence in the first document is greater than the length of the longest sentence in the second sentence.
So, we need to pad each sentence to the longest sentence of the batch.
One solution would be, find the length of the longest sentence in each batch and pad each sentence to that fixed length. But the pad_sequence function of PyTorch does not support that.
Am I missing something? Is there any other PyTorch way to do this?

I solved this problem by using a keras function:
torch.tensor(tf.keras.preprocessing.sequence.pad_sequences(doc1,maxlen=longest_sent,padding='post')

torch.tensor(tf.keras.preprocessing.sequence.pad_sequences(doc2,maxlen=longest_sent,padding='post')


Splitting training and test set with financial data I am using trees algorithms (decision tree, random forest and XGBoost) to forecast the sign of the returns in the stock market (classification).

I am using this article as a reference:

  http://rpubs.com/raaraa/412512

When splitting the training and the test set, the author uses the following R code:

index <- sample(1:nrow(stock_indicators))
size=0.2*nrow(stock_indicators)
test <- stock_indicators[index, ]
train <- stock_indicators[-index, ]

Is it correct to use a random set of dates to split the test and the training set with time series financial data?

It looks like look-ahead bias to me.

This would be the alternative:

train <-head(stock_indicators,round(0.70 * nrow(stock_indicators)))
test <- tail(stock_indicators,round(0.30 * nrow(stock_indicators)))

But with this split I cannot get significant accuracies even trying with different datasets.

Do you have any suggestions?

I think what you're doing is correct, in fact it would be even more correct to introduce a gap between your test and train set, i.e.

train <-head(stock_indicators,round(0.65*nrow(stock_indicators)))
test <- tail(stock_indicators,round(0.25*nrow(stock_indicators)))

The reason for this is time series data (usually) exhibits strong serial auto-correlation, so if you put the price for one day in your training set, and the next days price your test set they're a long way from independent and your test error is biased.

The reason you're not getting a good fit is that the data is probably not stationary, i.e. what's driving stock prices is changing over time and it's hard to forecast - if it wasn't everyone would do it perfectly.

Another way of doing the split is to split the data into blocks (say weeks) and then take 80% of the blocks for the training set and 20% for the test set. This is useful if say your data is seasonal (i.e. electricity demand), in which case using the last 20% isn't really representative of the full future you want to forecast. I'd argue stock prices aren't overly seasonal.


Decision Trees change result at every run, how can I trust of my results? Given a database, I split the data in train and test.
I want to use a decision-tree classifier (sklearn) for a binary classification problem.
Considering I already found the best parameters for my model, if I run the model on the test set I obtain at each run (considering the same hyper-parameters) a different result. Why that?  Considering I am using as metric the accuracy score, I have variations from 0.5 to 0.8. Which result should I take as correct, because I am not sure If it's correct to take the best result on the test set, or maybe I should consider an average of the results... I dont know.
Example, scoring for gridsearchCV is accuracy. I obtain
Grid scores on development set:

0.627 (+/-0.129) for {'max_features': 2}
0.558 (+/-0.152) for {'max_features': 3}
--> Best parameters: {'max_features': 2}
    Best score: 0.626876876876877 (this is the accuracy)

Using the best estimator on the test set, I obtain accuracy 0.83.. which I think it is only due to the case. In fact I try again and the result is this

Grid scores on development set:
0.584 (+/-0.126) for {'max_features': 2}
0.572 (+/-0.168) for {'max_features': 3}
--> Best parameters: {'max_features': 2}
    Best score: 0.5840215215215215
On the test set accuracy: 0.62 !!

So, how can I trust my results?
Second, would not it be better to use CV on all data, instead of just splitting at the beginning the dataset in test and train just one time?

I read about random_state but the problem is that the results depend on which value I use.. example
random_state = 2 --> accuracy 0.6 (on test set)
random_state = 6 --> accuracy 0.79 (on test set)
so basically this does not resolve my problem.
How can I validate my model if I dont know which one to use?

how can I trust of my results?

You probably shouldn't trust your results, because the large variation is likely caused by overfitting. Basically your model is not very reliable.

My guess is that you have either too many features or not enough instances, or any combination of these two issues.

  I am not sure If it's correct to take the best result on the test set, or maybe I should consider an average of the results

It is definitely not correct to take the best result. The average result is more representative of the true performance, but you should also provide the standard deviation.

  Second, would not it be better to use CV on all data, instead of just splitting at the beginning the dataset in test and train just one time?

Yes, that's a good idea: as far as I understand, currently you're just manually running the program several times and obtaining different performance each time. You can indeed use Cross-validation instead.

  I read about random_state but the problem is that the results depend on which value I use.. example random_state = 2 --> accuracy 0.6 (on test set) random_state = 6 --> accuracy 0.79 (on test set) so basically this does not resolve my problem. How can I validate my model if I dont know which one to use?

That's normal: the split function separates the training and test set randomly according to a random sequence. Setting the random seed to a particular value guarantees the exact same sequence every time, but you don't want that since it makes the splitting non-random.

Instead you should work on reducing the variation. Usually it's not possible to add instances, so you should probably try to reduce the number of features, or simplify them.


What does it mean when people say a cost function is something you want to minimize? I am having a lot of trouble understanding this. Does it mean you should not use the cost function very often?

No, it means you are trying to find the inputs that make the output of the cost function the smallest. It doesn't mean that you should "minimize" use of it.


Multivariate analysis Can we use weka for multivariate data analysis? When we have more than one variable as the dependent variable... ( without using factor analysis to reduce the number of variables associated with the dependent variable). Thank you

No.  I don't think so. And I don't think that is what "multivariate" means.

I rather think all of Weka's APIs and classifiers assume that there is exactly one attribute which gets assigned the "class" attribute label (that's the column to be predicted/classified).

Although there is a MultiClassClassifier meta-classifier, this is designed to improve
algorithms that can only handle binary class-attribute prediction problems (e.g. Weka's Glass.arff dataset with Logistic regression)


Text-Classification-Problem, what is the right approach? I'm planing to write a classification program that is able to classify unknown text in around 10 different categories and if none of them fits it would be nice to know that. It is also possible that more then one category is right.

My predefined categories are:

c1 = "politics"
c2 = "biology"
c3 = "food"
...

I'm thinking about the right approach in how to represent my training-data or what kind of classification is the right one. The first challenge is about finding the right features. If I only have text (250 words each) what method would you recommend to find the right features? My first approach is to remove all stop-words and use the POS-Tagger (Stanford NLP POS-Tagger) to find nouns, adjective etc. I count them an use all frequently appeared words as features.

e.g. politics, I've around 2.000 text-entities. With the mentioned POS-Tagger I found:

law:           841
capitalism:    412
president:     397
democracy:     1007
executive:     112
...

Would it be right to use only that as features? The trainings-set would then look like:

Training set for politics:
feature law         numeric
feature capitalism  numeric
feature president   numeric
feature democracy   numeric
feature executive   numeric
class politics,all_others

sample data:
politics,5,7,1,9,3
politics,14,4,6,7,9
politics,9,9,9,4,2,1
politics,5,8,0,7,6
...
all_others,0,2,4,1,0
all_others,0,0,1,1,1
all_others,7,4,0,0,0
...

Would this be a right approach for binary-classification? Or how would I define my sets? Or is multi-class classification the right approach? Then it would look like:

Training set for politics:
feature law         numeric
feature capitalism  numeric
feature president   numeric
feature democracy   numeric
feature executive   numeric
feature genetics    numeric
feature muscle      numeric
feature blood       numeric
feature burger      numeric
feature salad       numeric
feature cooking     numeric
class politics,biology,food

sample data:
politics,5,7,1,9,3,0,0,2,1,0,1
politics,14,4,6,7,9,0,0,0,0,0,1
politics,9,9,9,4,2,1,1,1,1,0,3
politics,5,8,0,7,6,2,2,0,1,0,1
...
biology,0,2,4,1,0,4,19,5,0,2,2
biology,0,0,1,1,1,12,9,9,2,1,1
biology,7,4,0,0,0,10,10,3,0,0,7
...

What would you say?

I think perhaps the first thing to decide that will help clarify some of your other questions is whether you want to perform binary classification or multi-class classification. If you're interested in classifying each instance in your dataset into more than one class, then this brings up a set of new concerns regarding setting up your data set, the experiments you want to run, and how you plan to evaluate your classifier(s). My hunch is that you could formulate your task as a binary one where you train and test one classifier for each class you want to predict, and simply set up the data matrix so that there are two classes to predict - (1) the one you're interested in classifying and (2) everything else.

In that case, instead of your training set looking like this (where each row is a document and columns 1-3 contain features for that document, and the class column is the class to be predicted):

1           2           3           class
feature1    feature2    feature3    politics
feature1    feature2    feature3    law
feature1    feature2    feature3    president
feature1    feature2    feature3    politics

it would look like the following in the case where you're interested in detecting the politics class against everything else:

1           2           3           class
feature1    feature2    feature3    politics
feature1    feature2    feature3    non-politics
feature1    feature2    feature3    non-politics
feature1    feature2    feature3    politics

You would need to do this process for each class you're interested in predicting, and then train and test one classifier per class and evaluate each classifier according to your chosen metrics (usually accuracy, precision, or recall or some variation thereof).

As far as choosing features, this requires quite a bit of thinking. Features can be highly dependent on the type of text you're trying to classify, so be sure to explore your dataset and get a sense for how people are writing in each domain. Qualitative investigation isn't enough to decide once and for all what are good features, but it is a good way to get ideas. Also, look into TF-IDF weighting of terms instead of just using their frequency within each instance of your dataset. This will help you pick up on (a) terms that are prevalent within a document (and possibly a target class) and (b) terms that distinguish a given document from other documents. I hope this helps a little.


Why do we move in the negative direction of the gradient in Gradient Descent? It is said that backpropagation, with Gradient Descent, seeks to minimize a cost function using the formula:
$$ W_{new} = W_{old} - learningRate \cdot \frac{\partial E}{\partial W} $$
My question is, if the derivate indicates in which direction the function (the graph of the error with respect to the weights) is decreasing, then why subtract from an already negative gradient?
Why not allow the current direction of the gradient (negative lets say) to be the driving factor for updating the weights:
$$ W_{new} = W_{old} + learningRate \cdot (-gradient) $$

Consider a simple example where the cost function to be a parabola $y=x^2$ which is convex(ideal case) with a one global minima at $x=0$
Here your $y$ is the independent variable and $x$ is the dependent variable, analogus to the weights of model that you are trying to learn.
This is how it would look like.

Let's apply gradient descent to this particular cost function(parabola) to find it's minima.
From calculus it is clear that $dy/dx = 2*x$. So that means that the gradients are positive in the $1^{st}$ quadrant and negative in the $2^{nd}$. That means for every positive small step in x that we take, we move away from origin in the $1^{st}$ quadrant and move towards the origin in the $2^{nd}$ quadrant(step is still positive).
In the update rule of gradient descent the '-' negative sign basically negates the gradient and hence always moves towards the local minima.

$1^{st}$ quadrant -> gradient is positive, but if you use this as it is you move away from origin or minima. So, the negative sign helps here.
$2^{nd}$ quadrant -> gradient is negative, but if you use this as it is you move away from origin or minima(addition of two negative values). So, the negative sign helps here.

Here is a small python code to make things clearer-
import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(-4, 4, 200)
y = x**2

plt.xlabel('x')
plt.ylabel('y = x^2')
plt.plot(x, y)

# learning rate
lr = 0.1
np.random.seed(20)
x_start = np.random.normal(0, 2, 1)
dy_dx_old = 2 * x_start
dy_dx_new = 0

tolerance = 1e-2
# stop once the value has converged
while abs(dy_dx_new - dy_dx_old) > tolerance:
    dy_dx_old = dy_dx_new
    x_start = x_start - lr * dy_dx_old
    dy_dx_new = 2 * x_start
    plt.scatter(x_start, x_start**2)
    plt.pause(0.5)
plt.show()


Who wrote the formula for gini importance/sklearn's feature importance score? I've been looking for a paper where the Gini importance was first proposed, but I am not sure if this is actually how it came to be.

Here's the formula I am familiar with and am looking to find in a paper:

$$\frac{N_s}{N_t} * \left(i - \frac{{N_s}_r}{N_s} i_r - \frac{{N_s}_l}{N_s} i_l \right)$$

where

$N_s$ = number of samples at a particular node

$N_t$ = number of total samples

$i$ = Impurity

$*_r$ = measure of the right child node

$*_l$ = measure of the left child node

If I understand properly, this is the formula sklearn's random forests also use in model.feature_importances_

Tags (since I don't have the rep to create new ones yet): Feature Importance Score, Gini Importance

There is a paper that covers "The origins of the Gini index". Gini index was detailed by Leo Breiman et al. in "Classification and regression trees" book in 1984. Leo Breiman also wrote a seminal paper on Random Forests in 2001 which includes the notion of feature importance.


Macro F1 result higher than accuracy for imbalanced dataset In one of the research papers on fake news detection, the authors collected a fake news binary dataset (fake vs. real news) consists of 16,817 real articles and 5,323 fake ones.

The authors presented the results using accuracy, precision, recall and F1 without specifying which kind of averaging they applied on the F1 metric (macro, micro, weighted, etc.).

Here are the results:

If you can notice for the last system, the accuracy value is 0.689 and the F1 value is 0.717 which is higher than the accuracy.

Thus, give the imbalanced status of the dataset, is it possible that the authors averaged the classes in the F1 metric using macro way?

For me, this "impossible" to happen, and I argue that probably they used weighed F1 score.

They calculated the "standard" F1 score as defined for binary classification tasks:

precision = 0.656
recall = 0.792
f1 = 2 * (precision * recall) / (precision + recall)
f1

gives

0.7176132596685083

Other versions of the F1 score are used for multiple classes as you can see here under "Extension to multi-class classification":

  The F-score is also used for evaluating classification problems with more than two classes (Multiclass classification). In this setup, the final score is obtained by micro-averaging (biased by class frequency) or macro-averaging (taking all classes as equally important). For macro-averaging, two different formulas have been used by applicants: the F-score of (arithmetic) class-wise precision and recall means or the arithmetic mean of class-wise F-scores, where the latter exhibits more desirable properties.

Alternatively, see here for the scikit learn implementation of the F1 score and its parameter description.


Share Jupyter Notebook with a non programmer I would like to share a Jupyter Notebook with somebody who is not a programmer (via a URL).

I understand that I must publish my work to GitHub and then use http://nbviewer.jupyter.org/ to render the notebook. However, I don't know how to correctly publish my .ipynb file into Github.

GitHub has built in support for showing a notebook. You will just need to run the notebook yourself, then upload the file to Github like all other file types.

Your viewer will be able to see your notebook without any installation.


How are the Convolution kernels learned? As I went through the basics of machine learning, I failed to understand how do the Convolutional layers in a CNN learn the convolution kernels. After looking at first few tutorials, I thought the kernels were fixed: for example, there was a 2D kernel which extracts vertical lines, another one extracted horizontal lines etc. Later on I realized that kernels are learned by the network and not fixed.

Is there a good explanation on how the learning goes, how does the backpropagation work (compared to fully connected layers), what is the optimization function, etc.?

p.s.
Let this be an example function to explain: https://keras.io/layers/convolutional/.

Sometimes the kernel function is very difficult to choose in a fixed form even if once found it would be easier. It can be learned to map the data in a features space (Hilbert space) through an Convex Optimisation method such as Semidefinite programming, using Hyperkernels, etc.

Some very usefull papers for your question can be found here:

Learning the Kernel via Convex Optimisation

Learning the Kernel Matrix with Semidefinite Programming

Learning the Kernel with Hyperkernels

Learning the Kernel Function via Regularization

I hope that answers your question


Machine Learning - Where is the difference between one-class, binary-class and multinominal-class classification? Where is the difference between one-class, binary-class and multinominal-class classification?

If I like to classify text in lets say four classes and also want the system to be able to tell me that none of these classes matches the unknown/untrained test-data.

Couldn't I just use all the methods that I mentioned above to reach my goal?
e.g. I could describe C1, C2, C3 and C4 as four different trainings-sets for binary-classification and use the trained models to label an unknow data-set ...

Just by saying, Training-Set for C1 contains class 1 (all good samples for C1) and class 0 (mix of all C2, C3 and C4 as bad samples for C1).

Is unlabeled-data C1 -> 1 or 0

Is unlabeled-data C2 -> 1 or 0
... and so on ...

For multinominal classification I could just define a training-set containing all good sample data for C1, C2, C3 and C4 in one training-set and then use the one resulting model for classification ...

But where is the difference between this two methods? (except of that I have to use different algorithms)

And how would I define a training-set for the described problem of categorizing data in those four classes using one-class classfication (is that even possible)?

Excuse me if I'm completely wrong in my thinking. Would appreciate an answer that makes the methodology a little bit clearer to me =)

The difference between these methods is the assumptions they make about the task.  Multi-class classification assumes that each document has exactly one label.  So a document can either be about sports or weather, not both.
Multi-label classification allows a document to have any combination of labels, including none.  So a document can be about only sports, only weather, sports AND weather, or neither.

You could train a multi-label classifier with data where each document has exactly one label, but there is no guarantee that the predictions made at test time will have only one label.  Also you are forcing the classifier to do more work (and potentially make more errors) by considering more possible labelings than it needs to.
Therefore, if the multi-class assumption makes sense for your problem, you are better off with a multi-class classifier.

The method that you describe for training individual binary classifiers corresponds to multi-label classification.  The binary classifiers that you use could each be trained from one-class data or two-class data.  However, this is only one of the many ways to do multi-label classification (see the wikipedia page above for more).

Unfortunately, the problem that you describe does not cleanly fit into either multi-class or multi-label classification, since you want each document to have at most one label.


Why do we add "αd" to N in Laplace Smoothing? I just started to learn Naive Bayes algorithm. Then I learned to use Laplace smoothing to avoid getting probability of zero.
I understand the purpose of using it, but, in the expression of Laplace smoothing below, I do not really understand why we need to add "αd" to N in the denominator. As far as I researched, one of websites says that it is done so that the division will never be greater than 1. However, I still do not understand why we add "αd". Could someone explain why we have to add "αd" to N in the denominator ?

You're right, the idea is that, if $\alpha d$ is not used, then you might have $\hat{\theta_i}$ bigger than 1.
That being said, I don't think the main reason of Laplace smoothing is to have probabilities greater than 0. I think the important thing is that it acts as a regularization technique (a smoother, indeed).
The smoother works the following way: we are mixing the empirical observations ($\frac{x_i}{N}$) and the theoretical distribution without data ($\frac{1}{d}$), and $\alpha$ is a parameter that controls how much do we care about the theoretical distribution. The idea is: the more we trust the theoretical distribution, the more regularization we are adding. It even has a Bayesian interpretation in terms of priors and posteriors.
So I don't think it has to do with probabilities being between 0 and 1, I think it has to do with performing the mixture of distributions (or update of the posterior, if you want) properly.


Fitting models to 2d data I am reading the documentation on tf.lattice (https://www.tensorflow.org/lattice/overview)

I am wondering how the training data was created/trained. Is it represented by (x,y) intpositions and some intensity value output?

Is the answer already there?

"
The function  can capture non-linear interactions between features. You can think of the lattice parameters as the height of poles set in the ground on a regular grid, and the resulting function is like cloth pulled tight against the four poles.

With  features and 2 vertices along each dimension, a regular lattice will have  parameters. To fit a more flexible function, you can specify a finer-grained lattice over the feature space with more vertices along each dimension. Lattice regression functions are continuous and piecewise infinitely differentiable.

"
EDIT-

Specifically for these pictures, they have given the example of dateset ranking restaurants(links are below). These graphs shows number of reviews(Y-Axis)/Average rating (x-axis).It has source code also.
https://www.tensorflow.org/lattice/tutorials/shape_constraints
https://www.tensorflow.org/lattice


How to construct validation set for time series for NN? I would like to train my model with a validation set. As the data is a time series I have to use timeseriessplit:

import numpy as np
from sklearn.model_selection import TimeSeriesSplit
X = np.array([[1, 2], [3, 4], [1, 2], [3, 4], [1, 2], [3, 4]])
y = np.array([1, 2, 3, 4, 5, 6])
tscv = TimeSeriesSplit(n_splits=5)
print(tscv)
TimeSeriesSplit(max_train_size=None, n_splits=5)
for train_index, test_index in tscv.split(X):
...    print("TRAIN:", train_index, "TEST:", test_index)
...    X_train, X_test = X[train_index], X[test_index]
...    y_train, y_test = y[train_index], y[test_index]
TRAIN: [0] TEST: [1]
TRAIN: [0 1] TEST: [2]
TRAIN: [0 1 2] TEST: [3]
TRAIN: [0 1 2 3] TEST: [4]
TRAIN: [0 1 2 3 4] TEST: [5]

Using this method, I obtain a train and test set.
But how can I generate a validation set now?

Im new to the topic too but I think the Idea is to create a Train/Test-Set and then take the TrainSet and Split it again in 2 Sets (mostly called Train and Development Set) for example with a KFold-CV. Train your model on the Train Set and improve it with the Developement Set. Then take the final model and use it on the whole trainingset.

The picture give you a clearer idea I think.


Question about Knn and split validation I have a big database with 40k recors and 2 classification classes. In this big database the 76% of records belong to the first class.

I've used a 70-30 split partition with stratified sampling, and the K-nn gives the best accuracy on k = 20.

1) Is it too big value for k ?

2) Is it possible that this big value for k is due to the disproportion of the 2 classes in the database , even if i used a stratified sampling ?

There is generally a trade of in k-NN:

k should be large enough to cancel out any noise.
k shouldn't be too large to create big biased boundaries.

One rule of thumb is to select an odd k value to avoid ties in binary class problems.
Small and big value of k is subjective to the underlying structure of the data set itself, that's why we check over a range of k values.
Both of the above gets settled if done a good job with cross validation.
So the most important question you need to focus is:

Have you chosen the right metric for model selection, keeping the imbalance of classes into consideration?

I would recommend you to first identify the class that is more important for you to predict correctly. Then, look at the confusion matrix and ponder upon different metrics (eg. recall, precision etc.) before finalizing a model/ k value.


Choosing the right corpus to build a TF IDF Vectorizer used for comparing the similarity of two strings I am working on evaluating when a pair of string objects can be considered equal (e.g. given that we are talking about journals, is "international journal of air and water pollution" the same of "air and water pollution"?) and I was wondering what is the proper corpus to use to build a TF IDF vectorizer.

I am currently using as corpus all the distinct values of these strings belonging to the same type (in this example all the strings referring to journals). Is this a valid approach? Why? What could be other valid approaches?

Word Mover’s Distance (WMD) is an algorithm for finding the distance between pairs of strings. It is based on word embeddings (e.g., word2vec) which encoded the semantic meaning of words into dense vectors.

  The WMD distance measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to "travel" to reach the embedded words of another document.

For example:

Source: "From Word Embeddings To Document Distances" Paper


Why would batch normalization allows us to use higher learning rate in the neural network? I am doing some study about the BatchNormalization:
https://towardsdatascience.com/batch-normalization-8a2e585775c9

In the article, it says:

Using batch normalization allows us to use much higher learning rates, which further increases the speed at which networks train.

Could anyone please share their thoughts on why would batch normalization allow higher learning rate? Thanks!

Section 3.3 of the original batch normalization paper has a good explanation of why this is the case.

Problem with a higher learning rate

First you need to understand the problem with higher learning rate. Higher learning rate causes exploding or vanishing gradients. In other words, gradients get multiplied by each other, so lower layers experience a compound effect of the gradients that are in higher levels.

How does batch normalization help?

Batch normalization is all about keeping the activations of all layers normalized, preventing them from becoming too large or small. So this directly helps to prevent exploding/vanishing gradient. Due to this reason, batch normalization allows higher learning rates.


Industry Standard Process for Fraud/Outlier/Anomaly Detection I want to write my thesis about Fraud Detection in ERP Databases. I'm looking for a Industry Standard Processs such as CRISP-DM for Data Mining Projects, in order to justify my approach in solving the issue of finding outliers/anomalies in the data set.
Is there any established Standard Process which would be more suitable for such an project?

Thanks.

You should refer this survey paper on Anomaly Detection (from University of Minnesota).

Please let me know if this helps you.


Order of normalization / augmentation for image classification I'm currently working on a common image classification with CNN.

I would like to use both normalization (substract mean / divide by std per channel) and data augmentation (rotation, color, blur, ...) but I don't know how to use them together.

Which order should I use ?

First normalize with parameters based only on original images and then augment it (augment a normalized image is relevant ? should I ban some type of augmentation like color ?)
Augment data and apply normalization based on all image (compute mean/ std with augmented images) which seems to be counterintuitive.
Augment data and apply normalization based on only original image which means that data are not really normalized
Or don't use both methods

You can use both methods, in the vast majority of examples I have seen and worked with the usual order is.

Augmentate your data, rotations, zoomings, flips....
Normalize your images based on the new and augmented dataset.

Doing it like this you can also see the results of the augmentation phase, while if you normalize it first you cannot visualize it


What is the meaning of each element in input_shape of Conv1D in Keras? I have a time-series data for 3 classes (each class is 35 second) as I extract each 1 second for 95 feature extracting so my final data has shape (105,95) (rows for time and column for feature).
I am currently building a 1D-CNN model for classification 3 class. But I got stuck as first layer of Conv1D. I have learnt that the input_shape of Convd1D is (batch_size, new_step, input_dim) but honestly I dont know what exactly what each element mean and how can I modify (reshape) my input data into Conv1D layer shape?
And one more question, I know that CNN required fixed input size. But I split my data into train and validation data as k parameter = 5, mean that my data will become: Train = (84,95) and Validation = (21,95). So with the same model how can we train and validate data with difference size?
I just start to learn about CNN but still not know how Conv1D operate?
I am appreciate your help.
Thank you very much.
Hear is my code (Which still get error because of first Conv1D input_shape):
X = dataset[:, 0:95].astype(float)
Y = dataset[:, 95].astype(int)

# Encode class values as integers
encoder = LabelEncoder()
encoder.fit(Y)
encoder_Y = encoder.transform(Y)

y = np_utils.to_categorical(encoder_Y)

acc_per_fold = []
loss_per_fold = []

fold_no = 1

kfold = KFold(n_splits = 5, shuffle = True)
for train, test in kfold.split(X,y):

    model = Sequential()
    model.add(Conv1D(100, 3, activation = 'relu', input_shape = (I really dont know what to put in here??))
    model.add(Conv1D(100, 3, activation = 'relu'))
    model.add(MaxPooling1D(3))
    model.add(Conv1D(150, 3, activation = 'relu'))
    model.add(Conv1D(150, 3, activation = 'relu'))
    model.add(GlobalAveragePooling1D())
    model.add(Dense(3, activation = 'softmax'))

    model.compile(loss = 'categorical_crossentropy', optimizer = 'adam', metrics = ['accuracy'])

    #model.summary()

    print('------------------------------------------------------------------------------')
    print(f'Training for {fold_no}...')

    history = model.fit(X[train], y[train], batch_size = 7, epochs = 150, verbose = 1)

    scores = model.evaluate(X[test], y[test], verbose = 0)
    print(f'Score for fold {fold_no}: {model.metrics_names[0]} of {scores[0]}; {model.metrics_names[1]} of {scores[1]*100}%')
    acc_per_fold.append(scores[1] * 100)
    loss_per_fold.append(scores[0])

    # Increase fold number
    fold_no = fold_no + 1

X[train].shape[0] - This is the number of instances. Let's say it is M
X[train].shape[1] - This is the shape of each  instance. Each instance is (1 x N)
Since input instances are of 1-D, the  input data become m x N. Had it been 2-D, it would have been m x Nx x Ny

And one more question, I know that CNN required fixed input size. But I split my data into train and validation data as k parameter = 5, mean that my data will become: Train = (84,95) and Validation = (21,95). So with the same model how can we train and validate data with difference size

Size means the size of each instance which is same in this case i.e (1,95). Just the number of instances are different i.e. 21 in Test and 84 in Train.
TimeSeries data -
 In TS learning, our initial data is just a time-based sequence of individual datapoint. We then make it Feature and Target by using a certain points (e.g. M) as Feature and  (M+1)th as Target. M is identified based on TS characteristics.
e.g.
1,2,3,4,5 will be (If I use 2 Feature)
1,2(X) --> 3(Y)
2,3(X) --> 4(Y)
3,4(X) --> 5(Y) etc.
These are 3 train records.
May read this too - Machinelearningmastery


How to calculate $\phi_{i,j}$ in VGG19 network? In the paper Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network by Christian Ledig et al., the distance between images (used in the loss function) is calculated from feature maps $\phi_{i,j}$ extracted from the VGG19 network,
where $\phi_{i,j}$ is defined as "feature map obtained by the j-th convolution (after activation) before the i-th maxpooling layer".

Can you elaborate on how to calculate this feature map, may be for VGG54 mentioned in the paper?

$\phi_{5,4}$  means 4th convolutional layer before 5th max-pooling layer right? But 4th layer has so 512 filters. So we would have 512 feature spaces. Which one to choose from this? Also what does "after activation" mean?

I found this answer related to the same issue, but the answer didn't explain much.

In section 2.2.1 of the paper, they state that they use euclidean distance.  I'm going to take your word that there are 512 filter activations in that layer; if I'm reading this right, there aren't 512 feature spaces, there is a 512-dimensional feature space that they are calculating euclidean distance in. So your distance function between two images $p$ and $q$ is just the standard Euclidean distance formula:

$$ d(\mathbf{p},\mathbf{q}) = \sqrt{\sum_{i=1}^{512}(p_i - q_i)^2}$$

where $\mathbf{p}$ and $\mathbf{q}$ are vectors holding the corresponding filter activations of $p$ and $q$.

Edit: Above the horizontal rule is my original answer which is wrong (or incomplete). What I think is happening is that the authors are taking the euclidean distance as above for each position in the feature maps at the $i,j$ layer, and averaging those distances to generate a scalar loss value. So for a 7x7 feature map, they'd be taking 49 512-dimensional euclidean distances and averaging them to get the VGG19 5,4 loss. This is how I read equation (5) in section 2.2.1 in their paper. I think the missing piece is that the authors don't bother with the square root in the euclidean distance formula. As discussed below, I think the notation is unclear.


What is the lightest browser to launch a jupyter notebook? Currently I am using Firefox as default browser, I have no issue to naviguate with it, but I have a little limited memory (8GB, 6.8 available) and Firefox takes a lot of RAM for nothing big, the more I use it in the time, the more it "eats" the RAM. I want to know which is the lightest browser to use jupyter notebook to save my RAM for computations on my dataset for instance.

Try 32-bit Chrome or Firefox, it may significantly lower memory usage.

Remember, it is surprisingly difficult to estimate physical memory consumption of given app: https://dzone.com/articles/windows-process-memory-usage-demystified

Also - unused RAM is wasted RAM. Do not blame browser for taking and using, what's available. When overall memory usage will raise, browser can drop some internal cache and free that memory.


Why I am getting prediction score 1 i.e. 100% I am reading few parameters and trying to predict target value using Linear regression and GB. Surpicingly I am getting score = 1 on test data. How come? Can anyone tell me whats wrong with this code?

from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

dataset = pd.read_csv("prod_data_for_ML.csv",header = 0)

#Data Pre-processing
data = dataset.drop('organization_id',1)
data = data.drop('status',1)
data = data.drop('city',1)

#Find median for features having NaN
median_zip, median_role_id, median_specialty_id, median_latitude, median_longitude = data['zip'].median(),data['role_id'].median(),data['specialty_id'].median(),data['latitude'].median(),data['longitude'].median()
data['zip'].fillna(median_zip, inplace=True)
data['role_id'].fillna(median_role_id, inplace=True)
data['specialty_id'].fillna(median_specialty_id, inplace=True)
data['latitude'].fillna(median_latitude, inplace=True)
data['longitude'].fillna(median_longitude, inplace=True)

#Fill YearOFExp with 0
data['years_of_experience'].fillna(0, inplace=True)

#Start training

labels = dataset.location_id
train1 = data
reg = LinearRegression()
x_train , x_test , y_train , y_test = train_test_split(train1 , labels , test_size = 0.20,random_state =1)

# x_train.to_csv("x_train.csv", sep=',', encoding='utf-8')
# x_test.to_csv("x_test.csv", sep=',', encoding='utf-8')

reg.fit(x_train,y_train)
reg.score(x_test,y_test)
```

You are using your target variable location_id as a feature. You need to remove it from data/train1/ X variables.

In other words, you are trying to predict location_id by location_id.

If you use reg.feature_importances_ you will see, that location_id affects your prediction 100% and others have no effect on prediction result.


How to cluster and visualize 3D data in python I have a 3D dataset of x,y,z points with 2 categories, category A and B. My end goal is to cluster all points in category B into volumes (spheroids/clouds) and find all points of category A close to edge of those volumes. I assume there won't be any points of category A inside the spheroids. The points of category B are very highly clustered in space, so clusters are probably very evident.
In 2D GIS I have used Kernel Density Estimation and K-Means clustering for similar tasks, but since I am dealing with 3D data, and non-geographic at that (relative to a fictional 0,0,0 origin), and since I am comfortable with the python data science tools, I think matplotlib/scipy/numpy/sklearn/pandas/etc are probably better tools for this. But I am not sure what tools and libraries specifically would be good to look at.
So my question is 2-fold:

what libraries would be well suited to find the 3D clusters and the points of category A "close" to them
what tool would allow me to visualize the clusters, preferably in an interactive plot that allows me to zoom/pan/rotate

The following steps is one method to achieve your result. I used Wolfram Language but the method can be applied by any language with the right libraries.

FindClusters
for category A data (dataA),
calculate the
ConvexHullMesh
for each of these clusters,
for each point in category B (dataB)
calculate the
RegionDistance
to each of the category A hulls,
and
Pick the
dataB points by their nearest category A hull.

We can collect related 3D example data from "AdministrativeDivision" Entity object properties.
dataA =
  Select[FreeQ[_Missing]]@
   EntityValue[
    EntityClass["AdministrativeDivision", {"ParentRegion" -> Entity["Country", "UnitedStates"]}]
    , {"GiniIndex", "TotalVotingRate", "HomeOwnershipRate"}];
First@dataA

{0.4776, 56.3712%, 70.7%}

I used FindClusters with the "MeanShift" method to cluster. Two clusters were found.
clusters = FindClusters[dataA, Method -> "MeanShift"];
Length@clusters

2

The list of ConvexHullMesh for each cluster is obtained by
hulls = ConvexHullMesh /@ clusters

These can be visualised with their internal points by combining aListPointPlot3D of clusters with a Graphics3D of hulls (with low Opacity to make them transparent) with Show.
cp =
 Show[
  ListPointPlot3D[
   clusters
   , PlotStyle -> ColorData[110]
   , PlotTheme -> {"Web", "FrameGrid"}
   , BoxRatios -> Automatic]
  , Graphics3D[
     {Opacity[.1]
     , MapIndexed[
       {ColorData[110] @@ #2, EdgeForm[{Thin, Opacity[.1], ColorData[110] @@ #2}], #1} &
       , hulls]}]
  ]

For category B example data we need points outside of the hulls of the clusters. We can create a Cuboid around the RegionUnion of hulls and hollow out the volume of hulls by taking the RegionDifference. This region can be visualised with RegionPlot3D.
With[
  {ru = RegionUnion[hulls]}
  , rd =
   RegionDifference[
    Cuboid @@ Transpose[
      MapAt[Ceiling[#, 0.01] &, {All, 2}]@
       MapAt[Floor[#, 0.01] &, {All, 1}]@
        RegionBounds@ru]
    , ru]
  ];
RegionPlot3D[rd
 , PlotStyle -> Opacity[.1]
 , Axes -> True]

Then we can generate RandomPoints inside this region for dataB.
SeedRandom[19283745]
dataB = RandomPoint[DiscretizeRegion@rd, 20];

The dataB points can be combined with the dataA cluster plot with Show.  All of the dataB points are outside of the dataA hulls.
Show[
 cp
 , ListPointPlot3D[dataB
  , PlotStyle -> Black]
 ]

Now that we have example category B data (dataB) we can calculate the RegionDistance of each point to each of category A's hulls. Then by Ordering these distances the First entry gives the hull the point is closest to.
nc =
 First /@
  Ordering /@
   Transpose@
    Through[
     Function[r, RegionDistance[r, #] &, Listable][hulls][dataB]
     ]

{2, 1, 2, 1, 2, 2, 1, 1, 2, 2, 2, 2, 1, 1, 1, 1, 1, 2, 2, 1}

Lastly we Pick the dataB points by their closest hull and combine their plot with the dataA cluster plot with Show. dataB points have been coloured to indicate their closest dataA cluster.
pncB = Pick[dataB, nc, #] & /@ Range@Length@hulls;

Show[
 cp
 , ListPointPlot3D[
  pncB
  , PlotStyle -> ColorData[104]
  , BoxRatios -> Automatic
  ]
 ]

Hope this helps.


Dueling Network gradient with respect to Advantage stream Looking at Dueling DQN:

$Q = V + A - mean(A)$

For simplicity, let's assume we are working with 4 neurons. Recall that Value stream only has 1 neuron $(v_0)$

Re-writing the above equation, we get:

$$
 \left[  \begin{array}{c}
 Q_0\\Q_1\\Q_2\\Q_3
\end{array}
\right] =
\left[  \begin{array}{c}
 v_0\\v_0\\v_0\\v_0
\end{array}
\right] +
 \left[  \begin{array}{c}
 a_0\\a_1\\a_2\\a_3
\end{array}
\right] -
 \left[  \begin{array}{c}
 \frac{1}{N}(a_0 + a_1 + a_2 + a_3)
\\\frac{1}{N}(a_0 + a_1 + a_2 + a_3)
\\\frac{1}{N}(a_0 + a_1 + a_2 + a_3)
\\\frac{1}{N}(a_0 + a_1 + a_2 + a_3)
\end{array}
\right]
$$

Question:

V (the Value) is a usual Dense Layer which has 1 neuron at the output.

A (the Advantage) is a usual Dense Layer which has N neurons at the output - what is the correct gradient vector to be passed to the neurons of this Advantage stream?

Thoughts:

From this post:

  Since Q is a simple sum of functions you have:

  $$\nabla_{\theta} Q(s,a) = \nabla_{\theta}V(s) + \nabla{\theta}A(s,a)
 - \frac{1}{numActions} \sum_{a'}\nabla_{\theta}A(s, a')$$

  You get the gradients of the V and A networks as usual by backprop.

To me, the correct gradient vector to be passed to the advantage stream is:

$$gradForA = \frac{dE}{dQ}\frac{dQ}{dA}$$

however, the second fraction of the equation is what makes me puzzled.
Is it as simple as:

$$\frac{dQ}{dA} =  \left[  \begin{array}{c}
1-\frac{1}{N}\\1-\frac{1}{N}\\1-\frac{1}{N}\\1-\frac{1}{N}
\end{array}
\right] $$

It's probably not, especially if we look at the "mean" vector, in the example above.
We can see that its every entry contains contribution of all advantage neurons - because they are summed.

Am I doomed to perform 2 backpropagations for the advantage dense layer, in parallel? - One for $+A$ and one for $-mean(A)$.  I would then add-up (component-wise) the two gradient vectors. Sounds like a clumsy idea..

I verified the following formulas with Gradient Checking, and it's confirming they are correct. (Getting discrepancy less than 0.00002)

Let's draw the tree diagram of what's affecting what:

If we try to find gradient affecting $a_0$, we have to sum-up the pink chains. They make it evident that $a_0$ has affected all $Q$ variables, thus we sum these chains.

So the formula for entire gradient that has to enter $A$ layer is:

$$
gradForA =  \left[  \begin{array}{c}
 \frac{\partial E}{\partial a_0}
\\\frac{\partial E}{\partial a_1}
\\\frac{\partial E}{\partial a_2}
\\\frac{\partial E}{\partial a_3}
\end{array}
\right]  = \left[  \begin{array}{c}
 \\\frac{\partial E}{\partial Q_0}(1 - u) + \frac{\partial E}{\partial Q_1}(-u)+ \frac{\partial E}{\partial Q_2}(-u) + \frac{\partial E}{\partial Q_3}(-u)
\\\frac{\partial E}{\partial Q_0}(- u) + \frac{\partial E}{\partial Q_1}(1-u)+ \frac{\partial E}{\partial Q_2}(-u) + \frac{\partial E}{\partial Q_3}(-u)
\\\frac{\partial E}{\partial Q_0}(- u) + \frac{\partial E}{\partial Q_1}(-u)+ \frac{\partial E}{\partial Q_2}(1-u) + \frac{\partial E}{\partial Q_3}(-u)
\\\frac{\partial E}{\partial Q_0}(- u) + \frac{\partial E}{\partial Q_1}(-u)+ \frac{\partial E}{\partial Q_2}(-u) + \frac{\partial E}{\partial Q_3}(1-u)
\end{array}
\right]
$$

where $u$ is $\frac{1}{N}$

Bonus:

By looking at the above image, we see that the $v_0$ has similarly affected all $Q$ variables, so its gradient is:
$$
gradForV = \frac{\partial E}{\partial Q_0} + \frac{\partial E}{\partial Q_1}+ \frac{\partial E}{\partial Q_2} + \frac{\partial E}{\partial Q_3}
$$

$gradForV$ is a scalar (not a vector) because value stream only has 1 neuron, always.

Bonus #2

to compute $gradForA$ it's obvious to do via 2 for-loops, nested in one another. However, you can get rid of nesting and improve code performance by re-writing the formula as follows:

$$
gradForA =  R + scalarB
$$

where

$scalarB = (-u)(\frac{\partial E}{\partial Q_0} + \frac{\partial E}{\partial Q_1}+ \frac{\partial E}{\partial Q_2} + \frac{\partial E}{\partial Q_3})$

and

$$ R =  \left[  \begin{array}{c}
 \frac{\partial E}{\partial Q_0}
\\\frac{\partial E}{\partial Q_1}
\\\frac{\partial E}{\partial Q_2}
\\\frac{\partial E}{\partial Q_3}
\end{array}
\right]
$$

The cool thing is that $scalarB$ can be computed once (it's a scalar), and re-used when computing the $gradForA$ vector.

Notice that $scalarB$ and $gradForV$ are almost the same thing, just with extra $(-u)$  So you can re-use this value in your code as well.


What are some options for an offline chatbot on Android? I have already created an online chatbot for my Android app using Dialogflow (formerly api.ai). It allows users to add a new item to their inventory. I now need to create an offline version of this chatbot. However, Dialogflow does not allow you to download the agent to work offline.

I have looked into several options, including RiveScript and ALICE (Pandorabots). These seem to be a bit too low-level for my purposes and require quite a bit of customization. I was wondering if there are any options similar to Dialogflow that can be used offline.

EasyNLU project might interest you: https://github.com/kolloldas/EasyNLU

  EasyNLU is a Natural Language Understanding (NLU) library written in Java for mobile apps. Being grammar based, it is a good fit for domains that are narrow but require tight control.


Error while using the pandas_datareader package I am trying to do a basic project where I grab some data from Morningstar or Google Finance, but when I import the package according to the usage instructions on GitHub and run Python in Pycharm, it returns the error:

ImportError: cannot import name 'is_list_like'

What should I do? Obviously I am doing something wrong.

BEFORE you import pandas_datareader, run this:

pd.core.common.is_list_like = pd.api.types.is_list_like

where pd stands for your pandas import.

Let me know if this works :)


How to interpret classification output - Predective model What is the significance of macro avg ? I'm not sure if this report signify a good predictions by the model.
Thanks in advance.

Your model is decent. Predictions are better than random chance.
Macro average is a simple average of the performance measures (precision, recall etc) across all classes/labels.
Micro average is a weighted average of the same , where the weights are based on the number of samples per class/label (support column).


How to extract particular key from the dict()? I have a dict() with 1000 keys. First 4 entries of dict are like.

{
    'aaa': [1,0,6,8,0,5,9,1,1,0],
    'abc': [1,1,1,2,4,0,0,0,9,8],
    'cfg': [0,0,0,4,3,1,0,0,0,1],
    'cghjj': [7,8,9,2,3,0,0,0,0,0]
}

I want to create a dataset using each key one by one. I want to pick key1. Then create a dataset using function1 using the values of key1 and then pick key2. Creating another dataset using values of key2 and appending the result of key2 to key1 row-wise. Then with key3... and then append the result to the result of key1 and key2 and so on.. up to 1000 keys.

dataset=dict()

create_dataset:
    select values of key 1.
    b1=function1()
    b1=np.asarray(b1)

    then select key 2.
    b2=function1()
    b2=np.asarray(b2)

    np.append(b1,b2,axis=0)

...upto 1000 key.

You could iterate over all the keys and apply function1 to them and them append to the full_data_set. At the end you have all them concatenated.

However, Python lists are unordered, so you don't control the order which keys are chosen.

data = {
    'aaa': [1, 0, 6, 8, 0, 5, 9, 1, 1, 0],
    'abc': [1, 1, 1, 2, 4, 0, 0, 0, 9, 8],
    'cfg': [0, 0, 0, 4, 3, 1, 0, 0, 0, 1],
    'cghjj': [7, 8, 9, 2, 3, 0, 0, 0, 0, 0]
}

def function1(l: list):
    # do something
    return l

full_data_set = []

for key in data.keys():
    full_data_set += function1(data[key])

If you do want to control the order of the keys, you need to make the ordered list of the keys somehow.

key_order = ['aaa', 'abc', 'cfg', 'cghjj']

data = {
    'aaa': [1, 0, 6, 8, 0, 5, 9, 1, 1, 0],
    'abc': [1, 1, 1, 2, 4, 0, 0, 0, 9, 8],
    'cfg': [0, 0, 0, 4, 3, 1, 0, 0, 0, 1],
    'cghjj': [7, 8, 9, 2, 3, 0, 0, 0, 0, 0]
}

def function1(l: list):
    # do something
    return l

full_data_set = []

for key in key_order:
    full_data_set += function1(data[key])


GPT-3 API Documentation? Has documentation of the GPT-3 API been made public?
I would be interested in keeping myself up to speed on the API's capability.

No - GPT-3 API is not currently public.
However once you get access, the documentation can be found at https://beta.openai.com/api-ref.


How to predict past test set on time series data using LSTM I'm trying to do a regression on some inventory amounts with the following model using Keras:

model = Sequential()
model.add(LSTM(100, batch_input_shape=(BATCH_SIZE, TIME_STEPS,
X_train_timeseries.shape[2]), dropout=0.0,
           recurrent_dropout=0.0, stateful=True,
kernel_initializer='random_uniform'))
model.add(Dropout(rate=0.5))
model.add(Dense(20, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
optimizer = optimizers.RMSprop(lr=LEARNING_RATE)
model.compile(loss='mean_squared_error', optimizer=optimizer, metrics=['mae'])

I have a time step of 60, how do I predict past my test data to make actual predictions into the future?

The best answer I found is here

Unfortunately I couldn't get it to work for my solution but reworked my solution and just incremented the sliding window, appended that answer to my test data, and iterated like that.


Equation of a Multi-Layer Perceptron Network I'm writing an article about business management of wine companies where I use a Multi-Layer Perceptron Network.
My teacher then asked me to write an equation that lets me calculate the output of the network. My answer was that due to the nature of multi-layer perceptron networks there is no single equation per se. What I have is a table of weights and bias. I can then use this formula:
$$f(x) = (\sum^{m}_{i=1} w_i * x_i) + b$$
Where:

m is the number of neurons in the previous layer,
w is a random weight,
x is the input value,
b is a random bias.

Doing this for each layer/neuron in the hidden layers and the output layer.
She showed me an example of another work she made (image on the bottom), telling me that it should be something like that. Looking at the chart, I suppose that it is a logistic regression.
So, my questions are the following:

Is there any equation to predict the output of a multi-layer perceptron network other than iterating over each neuron with $w*x+b$?
Should I just tell my teacher that a logistic regression is a different case and the same does not apply to this type of neural networks?
Is the first formula correct to show that a value of a neuron is the sum product of the previous layers plus the bias?

Edit 1: I didn't wrote the formula but I do also have activation functions (relu).

You are forgetting one element of the MLP which is the activation function. If your activation function is linear - then you can simply flatten out all the neurons into one single linear equation. The advantage of MLP however is its non-linearities so I suspect in your network you do have some activation (sigmoid? tanh? relu? etc..).
As for your graph - you could simply output predictions from your MLP and plot the exact scatter plot you have above. The only difference would be you wouldn't have a simple way of expressing this network in algebraic notation (as you have done on the existing x-axis).
To describe networks effectively in text you should look into matrix notation describing the weights and inputs of each layer. Maybe take a look at something like this to get started: https://www.jeremyjordan.me/intro-to-neural-networks/


Labeling Dataset Algorithmically I am working in emotion analysis of tweets. I collected close to .6 million tweets. I dont want to label them manually, instead i coded a bunch of complex rules to arrive at their label.

Before I use these algo generated labels for my ML solutions, I want to check/analyze how good is this labeling. Are there any statistical/otherwise tests that I can run to understand the quality of the labeling I get from my code.

Are there any statistical/otherwise tests that I can run to understand the quality of the labeling I get from my code.

Yes. You can treat your automated labelling as a model in its own right (it essentially is an "expert" model, that takes in additional features). Collect ground truth data with known accurate labels, and use a metric such as accuracy, AUROC, F1 score to determine how well your expert model works. Usually you should pick one metric that makes sense to you based on the problem you are trying to solve. Most statistics libraries will allow you to assess these metrics on arbitrary data, so it should be possible to find the relevant part of the API and feed it your synthetic labels to compare with ground truth data you have collected.

But wait! That is exactly what you would need to do in order to train and assess an ML model. About the best you can hope is that you will need less ground truth data to assess your expert than you would to train the ML version from scratch. For complex ML models such as deep neural networks, this could well be true.

Note that you may hit a limitation from this approach. Say that you assess from ground truth data that your expert is 85% accurate. You will not be able to use it to train an ML solution that is better than 85% accurate, because the ML will learn to copy the expert's predictions as best it can.

One thing to watch out for - if you analyse why the expert got some values wrong and then adjust it to give correct answers for those labels, you will need to remove the labels you used to do this from your test data - and ideally collect more, or take more from some unused pool, to replace them. Otherwise you will fit your expert to the test data and get incorrect reporting of how well the expert generalises to unseen data. Same reasoning means you should not use tweets and labels that you analysed when developing your expert model in order to test it.


Interpretation of the data through scatter plot I was exploring the data and had observed the data points are forming a triangle on the lower side.
x-axis: Total items
y-axis: Cancelled items

Can someone help me in interpreting this data? And help me how to proceed further in analyzing and building a model?

It is forming a triangle because you always have Cancelled items < Total items which is expected.
Using this representation of the data is not so informative as many points are clustered and we can't assess the distribution. You might want to consider a plot such as a 2D histogram of (x=Total items, y=Proportion of Cancelled Items) in order to assess if some relation exists. And if you do so you might want to normalize each X slice so as to visualize the distribution for each range [Xmin, Xmax]
If you are looking for a linear regression, you might want to use scikit-learn to fit a linearRegression model with X = total Items and Y=Cancelled Items and check the correlation coefficient.


Extracting sub features from inside a df cell? I have a dataframe containing several features of form:

Id, Acol,                   Bcol,   Ccol,           Dcol,
1,  X:0232,Y:10332,Z:23891, E:1222, F:12912,G:1292, V:1281
2,  X:432,W:2932            R:2392, T:292,U:29203   Q:2392
3,  Y:29320,W:2392          R:2932, G:239,T:2392    Q:2391

...about 10,000 Id's

where 1,2,3 are the Id's.
Acol, Bcol, Ccol, and Dcol are the feature columns,
X, Y, Z, W are sub-features of feature "Acol" and so on...

How can I extract the sub-features/features from this sort of dataframe?

You were not specific where you wanted to end up with the data in this frame, so I will simply show how to break out the features and sub features into a format that can be pivoted into a form as needed:

Code:

Most important element is taking a feature column and breaking out the sub features.  That can be done like:

def get_sub_features(feature_col):
    # split on commas and then colons
    feature_df = feature_col.str.split(',').apply(
        lambda feature: pd.Series(
            dict([sub_feature.split(':') for sub_feature in feature]),
            name=feature_col.name), 1)

    # add a feature name column to use as an index
    feature_df['feature'] = feature_col.name

    # name the columns as sub-feature for later stacking
    feature_df.columns.names = ['sub-feature']

    # return dataframe with id/feature_name index
    new_index = [feature_df.index.name, 'feature']
    return feature_df.reset_index().set_index(new_index)

Test Code:

df = pd.read_fwf(StringIO(u"""
    Id  Acol                    Bcol    Ccol            Dcol
    1   X:0232,Y:10332,Z:23891  E:1222  F:12912,G:1292  V:1281
    2   X:432,W:2932            R:2392  T:292,U:29203   Q:2392
    3   Y:29320,W:2392          R:2932  G:239,T:2392    Q:2391"""
                          ), header=1).set_index(['Id'])
print(df)

feature_cols = ['Acol', 'Bcol', 'Ccol', 'Dcol']
stacked = pd.concat(get_sub_features(df[f]).stack() for f in feature_cols)
print(stacked)

Results:

                      Acol    Bcol            Ccol    Dcol
Id
1   X:0232,Y:10332,Z:23891  E:1222  F:12912,G:1292  V:1281
2             X:432,W:2932  R:2392   T:292,U:29203  Q:2392
3           Y:29320,W:2392  R:2932    G:239,T:2392  Q:2391

Id  feature  sub-feature
1   Acol     X               0232
             Y              10332
             Z              23891
2   Acol     W               2932
             X                432
3   Acol     W               2392
             Y              29320
1   Bcol     E               1222
2   Bcol     R               2392
3   Bcol     R               2932
1   Ccol     F              12912
             G               1292
2   Ccol     T                292
             U              29203
3   Ccol     G                239
             T               2392
1   Dcol     V               1281
2   Dcol     Q               2392
3   Dcol     Q               2391
dtype: object

Accessing Data:

Some examples:

print(stacked.xs('T', level=2))
print(stacked.iloc[stacked.index.get_level_values('sub-feature') == 'T'])

Results:

               0
Id feature
2  Ccol      292
3  Ccol     2392

                           0
Id feature sub-feature
2  Ccol    T             292
3  Ccol    T            2392


How to do multitask learning using Caffe? I wonder how to do multitask learning using Caffe. Should I simply use the output layer SigmoidCrossEntropyLoss or EuclideanLoss, and define more than one outputs?

E.g. is the following architecture valid (3 outputs, i.e. 3 tasks concurrently learnt)?

Corresponding prototxt file:

name: "IrisNet"
layer {
  name: "iris"
  type: "HDF5Data"
  top: "data"
  top: "label"
  include {
    phase: TRAIN
  }
  hdf5_data_param {
    source: "iris_train_data.txt"
    batch_size: 1

  }
}

layer {
  name: "iris"
  type: "HDF5Data"
  top: "data"
  top: "label"
  include {
    phase: TEST
  }
  hdf5_data_param {
    source: "iris_test_data.txt"
    batch_size: 1

  }
}

layer {
  name: "ip1"
  type: "InnerProduct"
  bottom: "data"
  top: "ip1"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 2
  }
  inner_product_param {
    num_output: 50
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}

layer {
  name: "ip2"
  type: "InnerProduct"
  bottom: "ip1"
  top: "ip2"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 2
  }
  inner_product_param {
    num_output: 3
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}

layer {
  name: "loss"
  type: "SigmoidCrossEntropyLoss"
  # type: "EuclideanLoss"
  # type: "HingeLoss"
  bottom: "ip2"
  bottom: "label"
  top: "loss"
}

Every Blob in caffe can be assigned a nonzero loss weight.

And you can have an arbitrary number of outputs.

This means you can just learn n different networks on the same data with different targets and assign every loss function it's own weight. Caffe takes care of adding up all the loss.


Why are ensembles so unreasonably effective It seems to have become axiomatic that an ensemble of learners leads to the best possible model results - and it is becoming far rarer, for example, for single models to win competitions such as Kaggle. Is there a theoretical explanation for why ensembles are so darn effective?

For a specific model you feed it data, choose the features, choose hyperparameters etcetera. Compared to the reality it makes a three types of mistakes:

Bias (due to too low model complexity, a sampling bias in your data)
Variance (due to noise in your data, overfitting of your data)
Randomness of the reality you are trying to predict (or lack of predictive features in your dataset)

Ensembles average out a number of these models. The bias due to sampling bias will not be fixed for obvious reasons, it can fix some of the model complexity bias, however the variance mistakes that are made are very different over your different models. Especially low correlated models make very different mistakes in this areas, certain models perform well in certain parts of your feature space. By averaging out these models you reduce this variance quite a bit. This is why ensembles shine.


Improving the performace of the Naive Bayes classifier by decorrelating the data I was wondering if it is possible to improve the performance of the Naïve Bayes classifier by decorrelating the data. The Naïve Bayes assumes conditional independence of the features given some class $P(a_1, a_2 | c_1) = P(a_1 | c_1)P(a_2 | c_1)$ which is not necessarily true.

What if we applied a transformation into the space using decorrelation which will result in $Cov(a_1, a_2) = 0$ for all features. We have that if $a_1$, $a_2$ independent, then $Cov(a_1, a_2) = 0$. Although the converse does not strictly follow (e.g. consider the two dependent random variables $X \sim Norm(0, 1)$, and $X^2$ whose covariance is 0), the new features will be closer to being independent.

As you pointed out, a null covariance does not guarantee that variables are independent. You can have strongly dependent variables which show a covariance equal to 0. See a famous plot taken from Wikipedia page on Pearson correlation coefficient

A standard way to uncorrelate variables is to perform principal component analysis on the dataset, and use all components which are, by definition, uncorrelated. But please note that the Naive Bayes assumption is conditional independence, so independence for each class. In the general case, PCA projection would not be the same depending on the selected class.

Regarding the performance of Naive Bayes, I see no reason that a simple transformation should improve the model. This is supported by some reasonably well-received answers on similar questions, for instance this one on Stack Overflow, or this other one on Cross Validated.

However, I have read several reports of people getting increased accuracy when performing PCA prior to Naive Bayes. See here for instance, for a few exchanges on the subject, within a community of which trustworthiness is unknown to me; or search "PCA naive bayes" through your web search engine.

So in the end, I have no mathematical evidence to support this, but my feeling would be that the model performance variation before/after transformation could depend on the problem, especially the directions of principal components for each class.

Perhaps, if you perform some tests, you could share the result here. Otherwise, I think you could get answers with more mathematical background on Cross Validated.


What is "oracle" in statistics? When I read several statistical papers, they mention "oracle" property or "oracle" estimator.
What do they mean by "oracle"?
I understand this oracle is not a company name, but have no idea what this means.

For example,
Candes and Tao 2007 is about "oracle" inequality in a context of large dimensional estimation of linear model (p>>n).
https://projecteuclid.org/euclid.aos/1201012958)

Zou 2006 discusses "oracle" property of Lasso.
(http://www.tandfonline.com/doi/abs/10.1198/016214506000000735)

I am not familiar with this field but I have to read one paper which discusses high dimensional test in statistics.

"Oracle" refers to something that has access to the ground truth. It has the perfect information of which in practice, we rarely have some luxury. In such paper, you can typically see that it reflects a comparison with a model with perfect information.

From Candes and Terry Tao's paper:

"Even though $n$ may be much smaller than $p$, our estimator achieves a loss within a logarithmic factor of the ideal mean squared error one would achieve with an oracle which would supply perfect information about which coordinates are nonzero, and which were above the noise level.
To see why this is true, suppose one had available an oracle letting us know in advance the location of the S nonzero entries of the parameter vector

From Hai Zhou's paper:

We show that the adaptive lasso enjoys the oracle properties; namely, it performs as well as if the true underlying model were given in advance

Notice that oracle procedure/properties is also defined in Hai Zhou's paper reflecting the ability to rediscover the ground truth.


Multidimensional regression in Keras I'm trying to implement the One Hidden Layer Model presented in this article using Keras.

This is my code:

from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten, Activation
from keras import optimizers

model = Sequential()
model.add(Dense(100, input_dim=9216))
model.add(Activation('relu'))
model.add(Dense(30))
model.compile(loss='mean_squared_error', optimizer=sgd, metrics=['accuracy'])

sgd = optimizers.SGD(lr=0.01, momentum=0.9, nesterov=True)
hist = model.fit(X_train, y_train, epochs=10, verbose=0, validation_split=0.2)

y_pred = model.predict_classes(X_valid)

  X_train shape is (2140, 9216)

  y_train shape is (2140, 30)

  X_valid shape is (1783, 9216)

But y_valid shape is (1783,). I'm trying to understand why I'm not getting a (1783, 30) output. Am I missing something?

The problem is using predict_classes in:

model.predict_classes(X_valid)

this is designed to select the argmax (index of the maximum output) and choose it as the predicted class, for a classifier.

You have a regression problem, and just want the raw output from the network. So instead you should call:

y_pred = model.predict(X_valid)


How would I implement a model in Keras to generate embeddings based on documents? I would like to create a model which generates embeddings for documents. I can create the model, but I was wondering how I would implement a training scheme where data consisted of two documents, either identical or different, and would be fed in parallel to the model (two instances running at the same time), which would create embeddings for their respective document. I want to use cosine distance to see similarity, and distance would be lower for documents that are the same. How would I train/implement this?

For starters, it depends on why you're trying to create Document Embeddings.
There's the TF-IDF algorithm; it's simple to grasp and implement, and it facilitates using cosine distance as a metric. To train in parallel, you could probably feed both documents to the model and have a synchronized counter doing the calculations such as Term Frequency and Inverse Document Frequency, and then in the end you can use those values to scale the elements of your TFIDF matrix.
Alternatively, you can use the Doc2Vec algorithm as explained here.
It's an extension of the Word2Vec model, but parallelizing the training for this one might be more complicated.
I'm sure there are more ways, but these are the two most popular that I know of.


Understanding minimizing cost correctly I cannot wrap my head around this simple concept.

Suppose we have a linear regression, and there is a single parameter theta to be optimized (for simplicity purposes):

$h(x) = \theta \cdot x$

The error cost function could be defined as $J(\theta) = \frac1m \cdot \sum (h(x) - y(x)) ^ 2$, for each $x$.

Then, theta would be updated as:

$\theta = \theta - \alpha\cdot \frac1m \cdot \sum (h(x) - y(x)) \cdot  x$, for each $x$.

From my understanding the multiplier after the alpha term is the derivative of the error cost function $J$. This term tells us the direction to head in, in order to arrive at the minimum making a small step at a time. I understand the concept of "hill climbing" correctly, at least I think.

Here is where I don't seem to wrap my head around:

If the form of the error function is known (like in our case: we could visually plot the function if we take enough values of theta and plug them in the model), why can't we take the first derivative and set it to zero (partial derivative if the function has multiple thetas). This way we would have all the minimums of the function. Then with the second derivative, we could determine whether it's a min or a max.

I've seen this done in calculus for simple functions like $y = x^2 + 5x + 2$ (may years ago, maybe I am wrong), so what is stopping us from doing the same thing here?

Sorry for asking such a silly question.

Thank you.

Consider differentiating this $$\nabla_\theta\|X\theta -y\|^2=2X^T(X\theta -y)=0$$

Hence solving this, would give us $$X^TX\theta =X^Ty$$

Solving this would give us the optimal solution theoretically. However, numerical stability is an issue and also don't forget computational complexity. The complexity to solve a linear system is cubic.

Also, sometimes, we do not even know even have a closed form, a gradient based approach can be more applicable.


Data Science Podcasts? What are some podcasts which are related to data science?

This is a similar question to the reference request question on CrossValidated.

Details/rules:

The podcasts (the theme and the episodes) should be related to data science. (For example: A podcast which is about some other domain, with an episode which speaks about data science in that domain, is not a good reference/answer.)
Personal opinions/reviews (if any) would be very helpful too.

I strongly suggest Talking Machines. It's a very well put together podcast from a professor at Harvard. They cater to both machine learning experts and enthusiasts.

Their interviews are often done from NIPS, and the guests are usually top tier practitioners.


Database options for JSON storage, queried with Apache Drill I am planning to set up a JSON storage system. It will store tens of millions of JSON records, all in the same format. I'd like to be able to query the data using Apache Drill.

It looks like there is Drill support for MongoDB and Postgres. However, I'm unsure of the pros and cons of each, and how I'd structure the schema if I'd choose Postgres.

Would be beneficial to know the structure of the data and what you are wanting to do with the data. Do you really need a database at all? Drill will read from a filesystem/S3/etc?

But to answer your question with the information you provided.

The pro of MongoDB over PostgreSQL would be in scaling and read performance (after a certain scale). If this is mainly a read only, or analytic database (primarily reads) then MongoDB would probably work.

PostgreSQL has the advantage of ACID compliance and more than likely better performance if you don't go above the ~10's of millions of records (especially with JSONB, which supports indexes on the datatype).

Without knowing much about the data, you could simply have two columns within your postgresql JSON table (id, JSON_field) and then use the functions within PostgreSQL to query your JSON field. It is a little more involved in querying JSON data within PostgreSQL but after you get used to it, it is pretty simple.


Coloring clusters so that nearby clusters have different colors I have clustered a large number of points (~3000) into (~400) clusters.  I want to plot the data and visualize the clusters.  I want to make sure that nearby clusters have different colors.  Can anyone recommend an approach to coloring the clusters?
This is a conceptual question, but I'm most interested in solutions in python or R.

matplotlib already takes care of coloring adjancent clusters with different colors. But, I believe it uses a unique color for each cluster. If that's the case, 400 colors would be too much.
There might be better ways, but worst case, try this:
We want to color with minimum number of colors. Hence, the problem turns to a graph coloring problem in which, we don't want two connected adjacent nodes have the same color. Finding the minimum number of colors is an np-hard problem but we can use approximate algorithms.
One way could be defining your cluster centroids as graph nodes and storing their connections and then using a graph coloring algorithm.
Step 1: Store your clusters as an adjacency matrix and convert this matrix to a graph wherein each centroid is the representative of a cluster and the adjacency matrix contains the connectivity(neighborhood) between each centroid and the others.
Explanation
Code
Step 2: use a coloring algorithm to color the nodes of the graph.
This post contains a coloring algorithm that gets the graph and returns the coloring for that.
Step 3: having the color of each centroid, you can color the whole clusters with minimum number of colors.