Compare between similar and dissimilar couples of instances I label couples of similar and dissimilar instances based on user behavior.
each instance has a lot of features.
I have few ways of labeling the couples.
I know want to evaluate which of the label methods produce the most homogeneous distribution in the groups or to tell if the two groups comes from the same distributions.

I am looking for a statistical measures mostly.
Any suggestions?

You can compute a similarity score between each couple of instances (diff in features) and then you can check if the distribution of the difference for each group (similar and dissimilar) is significantly different using the Kolmogorov-Smirnov test.


lengthy criteria in dataframe selector I'd like to get records with country codes not in a long list , something along the lines of

country_list = ['BR','MX','CO',...]
df = df[df['country_code'] not in country_list]

ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

what's an elegant way to do it?

df = df[~df['country_code'].isin(country_list)]

https://pandas.pydata.org/pandas-docs/stable/gotchas.html#using-the-in-operator


Understand RandomizedSearchCV output I am trying to do hyperparameter tunning for the Randomforest regression model.
I'm using RandomizedSearchCV (scikit-learn) and I defined verbose=10.
For that reason, I'm getting messaged while it's running and I would like to understand them a bit better.
Those are my parameters for RandomizedSearchCV:
from joblib import Parallel, delayed, parallel_backend
# Use the random grid to search for best hyperparameters
# First create the base model to tune
rf = RandomForestRegressor(-1)

with parallel_backend('threading',n_jobs=12):
    rf_random = RandomizedSearchCV(estimator = rf, param_distributions = random_grid, n_iter = 12, cv = 3, verbose=10, random_state=42)#, n_jobs = -2)
    # Fit the random search model
    rf_random.fit(x_train, y_train)

After running a while )and is still running) I have those messages:

my questions are:
1. what are the rows before the first red row? I thought it may be the combinations of parameters but that doesn't make a lot of sense because those are not enough
2. What is the meaning of the row between the red rows? Why those parameters are there? is this after one CV?
3. "Done one task" - what is the task? one CV?
4. Why do I get score only to some of the parameters?

1. what are the rows before the first red row? I thought it may be the combinations of parameters but that doesn't make a lot of sense because those are not enough

Parameters, which are the candidates of the CV are printed

2. What is the meaning of the row between the red rows? Why those parameters are there? is this after one CV?

It is printed after the completion of the 3rd(as defined in your code) fold for a parameter Candidate

3. "Done one task" - what is the task? one CV?

This is from the joblib I believe. It's the completion of one fold i.e. one out of 36.

4. Why do I get a score only for some of the parameters?

The score is printed after the completion of 3(as defined in your code) fold of each parameter candidate
In a multi-thread setup, we can't rely on the sequence of the messages as it will depend on the Thread allocation.
My answer is based on this small setup.
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import RandomizedSearchCV
iris = load_iris()
logistic = LogisticRegression(solver='saga', tol=1e-2, random_state=0)

distributions = dict(C=[0.1,0.5,1.0],
                      penalty=['l2', 'l1'], max_iter=[50,100])
clf = RandomizedSearchCV(logistic, distributions, random_state=0,n_iter = 5, cv = 2, verbose=10)

from joblib import Parallel, delayed, parallel_backend
with parallel_backend('threading',n_jobs=12):
    search = clf.fit(iris.data, iris.target)

Output

Fitting 2 folds for each of 5 candidates, totalling 10 fits
[CV] penalty=l2, max_iter=100, C=0.5 .................................
[CV] penalty=l2, max_iter=100, C=0.5 .................................
[CV] penalty=l1, max_iter=100, C=1.0 .................................
[CV] penalty=l1, max_iter=100, C=1.0 .................................
[CV] penalty=l2, max_iter=50, C=0.5 ..................................
[CV] penalty=l2, max_iter=50, C=0.5 ..................................
[CV] penalty=l2, max_iter=100, C=1.0 .................................
[CV] penalty=l2, max_iter=100, C=1.0 .................................
[CV] penalty=l2, max_iter=100, C=0.1 .................................
[CV] penalty=l2, max_iter=100, C=0.1 .................................
[CV] ..... penalty=l2, max_iter=100, C=1.0, score=0.960, total=   0.0s
[CV] ..... penalty=l2, max_iter=100, C=0.5, score=0.973, total=   0.0s
[CV] ..... penalty=l2, max_iter=100, C=0.1, score=0.947, total=   0.0s
[CV] ..... penalty=l1, max_iter=100, C=1.0, score=0.973, total=   0.0s
[CV] ..... penalty=l2, max_iter=100, C=1.0, score=0.973, total=   0.0s
[CV] ...... penalty=l2, max_iter=50, C=0.5, score=0.973, total=   0.0s
[CV] ..... penalty=l2, max_iter=100, C=0.1, score=0.893, total=   0.0s
[CV] ..... penalty=l2, max_iter=100, C=0.5, score=0.960, total=   0.0s
[CV] ...... penalty=l2, max_iter=50, C=0.5, score=0.960, total=   0.0s
[CV] ..... penalty=l1, max_iter=100, C=1.0, score=0.987, total=   0.0s

[Parallel(n_jobs=12)]: Using backend ThreadingBackend with 12 concurrent workers.
[Parallel(n_jobs=12)]: Done   3 out of  10 | elapsed:    0.0s remaining:    0.1s
[Parallel(n_jobs=12)]: Done   5 out of  10 | elapsed:    0.0s remaining:    0.0s
[Parallel(n_jobs=12)]: Done   7 out of  10 | elapsed:    0.0s remaining:    0.0s
[Parallel(n_jobs=12)]: Done  10 out of  10 | elapsed:    0.0s finished


Anomaly Detection in Time Series: How to label the data How to label time series so that we can train it on machine learning models to classify data point as anomaly or not?

If I have time series, and anomaly occurs at time t, should I label that point 1 and other points as 0, or there are any other methods to label the time series?

Anomaly or Outlier detection are usually used when we have cases that happens rarely. So it is not easy to gather data for those cases. Then what we do is, we just train with our normal data. It is also the same for time series.

It is usually falls under Unsupervised learning, as there is no need to label your data. You feed all the data that you have to the model (For example one Class SVM) and it would try to understand or find clusters/groups in the data. Afterwards for test data, it would give you a number as a result of prediction/detection. This number would represent how close or far this test data is to normal group/clusters.

Of course you should have labeled data for the propose of testing the model. For this case you can either generate anomaly data manually just for the sake of testing or you should identify some anomalies by analysing the data. Then you can test the model with these labeled data and see if it can actually recognise them or not.

Of course you can tackle this issue as two-class classification but then this is not anomaly detection anymore, and it will be a normal classification model.

Hope this helps.

For further info you can take a look at these:

https://scikit-learn.org/stable/modules/outlier_detection.html

https://scikit-learn.org/stable/modules/generated/sklearn.svm.OneClassSVM.html#sklearn-svm-oneclasssvm


Feed forward neural network, output as list of targets and associated probabilities I am working through an FNN tutorial, right now it outputs a sigmoid probability from 0-1 (0.8956 for example). My own data has 3+ possible targets so i need the output to be a list of the targets and the associated probability for testing on new samples. (In this instance it would be: 0 (probability), 1 (probability)). How would i go about this? Here is the code. Thanks.

import numpy as np

# sigmoid function
def nonlin(x,deriv=False):
    if(deriv==True):
    return x*(1-x)
    return 1/(1+np.exp(-x))

# input dataset
X = np.array([  [0,0,1],
                [0,1,1],
                [1,0,1],
                [1,1,1] ])

# output dataset
y = np.array([[0,0,1,1]]).T

# seed random numbers to make calculation
# deterministic (just a good practice)
np.random.seed(1)

# initialize weights randomly with mean 0
syn0 = 2*np.random.random((3,1)) - 1

for iter in xrange(10000):

# forward propagation
l0 = X
l1 = nonlin(np.dot(l0,syn0))

# how much did we miss?
l1_error = y - l1

# multiply how much we missed by the
# slope of the sigmoid at the values in l1
l1_delta = l1_error * nonlin(l1,True)

# update weights
syn0 += np.dot(l0.T,l1_delta)

print "Output After Training:"
print l1

If you have more than two targets, then you should use softmax activation instead of sigmoid. Softmax activation gives you the probability associated with each class in the output. The only thing you need to do before applying softmax is to convert your targets into one-hot encoded vectors. If you want to know more about softmax in detail, you can read about it here


How to do Machine Learning the right way? I have a basic understanding about Machine Learning in general. My question is how it is done in the practical applications of it.

If I take the following definition of ML

  A computer program is said to learn from experience E with respect to
  some class of tasks T and performance measure P, if its performance at
  tasks in T, as measured by P, improves with experience E.

It talks about Experience E. What I understand from the above is the more data you give to the algorithm E increases, and that in turn increases P.

Now let's consider a scenario where you build a decision tree model from 10,000 data rows available. Now I have the model, so can I say that my model has learned and just stop there!(use that model for prediction from that point on-wards forever)?

According to the definition, I need to feed more data so the experience increases, and in turn I get a performance improvement.

So is Machine Learning a continuous process so that You cannot build the model and just stop there. Do we need to feed more data to the algorithm time by time and improve the model so that the model actually LEARNS?

There is not really a "right way" to use machine learning outside of running the algorithms, any more than there is a "right way" to use a sorting algorithm, or any other kind of complex automation.

Some guidelines/thoughts:

You can start using your model to make predictions, once those predictions are useful to you. This may be once they pass some threshold in performance that would make the predictions the best way of achieving some other goal. That may just mean "better than the last ML model", but could also mean "cheaper and faster than having some people doing it, even if it is less accurate".
Depending on how your training data is collected, and how the model works, it may be possible to both use your model for predictions, and to continue training it on new data as it arrives.
However, there is no requirement for continuous learning. Learning systems that continuously adapt to new data are useful in some situations (example might be recommendation systems), but might be inappropriate in others (example might be automatic driver for a vehicle, where you don't want to risk reduction in performance whilst in use).

One thing that is considered very standard practice is to demonstrate how the machine learning system is performing after a period of learning by using a test metric. The simplest variation is to keep some of your precious data to one side, deliberately not training on it, then measure the performance. Often two such sets are kept aside - a cross validation set is used to help decide automatically between variations of your learning algorithm, and a test set is used to assess the end result of that selection.


Built a Heap in Cluster algorithm Does anyone could explain in details how heap structure works in Cluster Algorithm?

I am planning to code CURE in Matlab (Clustering using representatives) but, at first, the paper mentions that all points of datasets are considerered as clusters and heap helps to merge new clusters and I do not know how exactly this data structure works in this algo.

Thanks for your support.

A heap works in clustering the same way it works outside of clustering.

It's purpose is to efficiently find the minimum or maximum of a set, remove it, then find the next.

Efficient implementations of heaps in scripting languages may be impossible. For good performance, you usually need low-level memory access to avoid copying.


Strange plot on a simple dataframe Rstudio I don't usually use R and I'm having trouble with a simple plot from a dataset. I just wanted to make a scatterplot and look at the values, but it keeps ploting a strange "box" thing which I don't really know if it's an error or if I'm missing data manipulation to make it work.
install.packages("remotes")
remotes::install_github("brunoruas2/gujarati", force=TRUE)
data(package = .packages(all.available = TRUE))
data(package = 'gujarati', Table3_6)

#Exercício 3.20

data(package = 'gujarati', Table3_6)
df <- Table3_6

#Nomes das colunas
names(df)[2] = 'Produçãox1'
names(df)[3] = 'Produçãox2'
names(df)[4] = 'Remuneraçãoy1'
names(df)[5] = 'Remuneraçãoy2'

summary(df)
View(df)

# Plot 1
plot(Remuneraçãoy1 ~ Produçãox1, data=df)

Update:
It was only a matter of transform string numbers in numerical numbers.
I did it by transforming each "non numeric" column in character and used the as.numeric().
The code looks like this now:
    #Nomes das colunas
names(df)[2] = 'Produçãox1'
names(df)[3] = 'Produçãox2'
names(df)[4] = 'Remuneraçãoy1'
names(df)[5] = 'Remuneraçãoy2'

df$Produçãox1 <- as.numeric(as.character(df$Produçãox1))
df$Produçãox2 <- as.numeric(as.character(df$Produçãox2))
df$Remuneraçãoy1 <- as.numeric(as.character(df$Remuneraçãoy1))
df$Remuneraçãoy2 <- as.numeric(as.character(df$Remuneraçãoy2))

#Regressão
reg1 <- lm(df$Remuneraçãoy1 ~ df$Produçãox1)
reg2 <- lm(df$Remuneraçãoy2 ~ df$Produçãox2)

#Plot
plot(df$Remuneraçãoy1 ~ df$Produçãox1,xlab='Produção do setor empresarial',
     ylab='Remuneração real')
abline(reg1, col='blue')

plot(df$Remuneraçãoy2 ~ df$Produçãox2,xlab='Produção do setor empresarial não agrícola',
     ylab='Remuneração real')
abline(reg2, col='red')


What methods exist for distance calculation in clustering? when we should use each of them? What methods exist for distance calculation in clustering? like Manhattan, Euclidean, etc.?
Plus, I don't know when I should use them. I always use Euclidean distance.

Well, there is a book called

  Deza, Michel Marie, and Elena Deza. Encyclopedia of distances. Springer Berlin Heidelberg, 2009. ISBN 978-3-642-00233-5

I guess that book answers your question better than I can...

Choose the distance function most appropriate for your data.

For example, on latitude and longitude, use a distance like Haversine. If you have enough CPU, you can use better approximations such as Vincenty's.

On histograms, use a distribution-baes distance. Earth-movers (EMD), divergences, histogram intersection, quadratic form distances, etc.

On binary data, for example Jaccard, Dice, or Hamming make a lot of sense.

On non-binary sparse data, such as text, various variants of tf-idf weights and cosine are popular.

Probably the best tool to experiment with different distance functions and clustering is ELKI. It has many many distances, and many clustering algorithms that can be used with all of these distances (e.g. OPTICS). For example Canberra distance worked very well for me. That is probably what I would choose as "default".


Confusion regarding confusion matrix I am confused on how to represent the confusion matrix -- where to put the FP and FN. Link1 and Link2 show different confusion matrix for binary classification. The rows represent the actual and columns represent the predicted values. Based on my understanding, the correct confusion matrix should have been:
                        | Pred Neg| Pred Pos
                        |___________________
Actual Negative(class0) |  TN     |  FP
------------------      |-------------
Actual Positive(Class1) | FN      | TP

where  TN (class 0) indicates the amount of correctly identified normal patterns. TP (class 1) indicates the amount of correctly identified malignant patterns. On the other hand, FP indicates that the classifier predicted the signature to be malignant when infact it was normal.
$  \text{Precision} = \frac{TP}{TP+FP}$,  $\text{Recall or TPR} = \frac{TP}{TP+FN}$,
Question 1: Is my understanding and construction of the confusion matrix correct?
Question 2: What is the intuitive difference between Precision and recall? What happens if precision < recall?

Question 1: Is my understanding and construction of the confusion
matrix correct?

Yes, you are correct in your definitions and the way you construct the confusion matrix. The links you have provided also agree with each other. They just switch rows and columns, since there is no hard rule regarding the presentation, as long as the correct relations are maintained.
Link 1 shows this matrix:
          | Pos Class | Neg Class
Pos Pred  |    TP     |    FP
Neg Pred  |    FN     |    TN

Link 2 shows the same matrix, but transposed:
          | Pos Pred  | Neg Pred
Pos Class |    TP     |    FN
Neg Class |    FP     |    TN

Question 2: What is the intuitive difference between Precision and
recall?

Precision is the rate at which you are correct when you predict a positive class. It takes into account all of your positive predictions and figures out which proportion of those is actually correct. When your precision is high, this means that once you make a positive prediction, you are likely to be correct about it. This says nothing about how correct your negative predictions are -- you might make 1 positive and 99 negative predictions on 100 actual positives and still get 100% precision, since your only positive prediction just happened to be correct.
Recall is the rate at which you are able to predict the positive class correctly. It takes into account all of the actual positive classes and figures out which proportion of those you have predicted correctly. When your recall is high, this means that very few actual positives slip by your model without being detected as such. This says nothing about how good you are at being actually correct with your positive predictions -- a model that always predicts a positive class easily achieves 100% recall.
One usually strives to optimize both precision and recall by finding the most acceptable balance between the two. You might want to read this article about the Precision-Recall curve to get a fuller understanding of the relationship between these metrics.

What happens if precision < recall?

As you have highlighted in your post, the two formulas differ only in the denominator. It follows that when precision is smaller than recall, then the number of false positives in your predictions is larger than the number of false negatives.


Observation window in Predictive modelling In any predictive modelling exercise we first start with defining observation window and perform window for the product/problem. Just wanted to know if the window is different for different predictors or it's same for all the predictor variables. for ex- variable A can be calculated in 3 year of observation window. Can another variable (Variable B) be calculated in 2 year of window ?

It works fine if you make sure that window remains the same for both seen and unseen data.


Ordering a material science dataset (properties names, properties scalars, formulas) I'm dealing with a materials science dataset and I'm in the following situation,
I have data organized like this:
Chemical_ Formula     Property_name            Property_Scalar

    He                Electrical conduc.          1
    NO_2              Resistance                  50
    CuO3              Hardness
    ...               ...                        ...
    CuO3              Fluorescence                300
    He                Toxicity                    39
    NO2               Hardness                    80
    ...               ...                         ...

As you can understand it is really messy because the same chemical formula appears more than once through the entire dataset, but referred to a different property that is considered. My question is, how can I easily maybe split the dataset in smaller ones, fitting every formula with its descriptors in ORDER? I really need help on this... thank you.  ( I used fiction names and values, just to explain my problem.)
I'm on Jupyter Notebook and I'm using Pandas.
I'm editing my question trying to be more clear:
My goal would be to plot some histograms of (for example) n°materials vs conductivity at different temperatures (100K, 200K, 300K). So I need to have both conductivity and temperature for each material to be clearly comparable. For example, I guess that a more convenient thing to obtain would be:
Chemical formula     Conductivity      Temperature

      He                 5                  10K
      NO_2               7                  59K
      CuO_3              10                 300K
      ...                ...                ...
      He                 14                 100K
      NO_2               5                  70K
      ...                ...                ...

Given that your Dataframe is:
df2 = pd.DataFrame({
    "Chemical_Formula":["He", "NO_2", "CuO3", "CuO3", "He", "NO2"],
    "Property_name":["Electrical conduc.", "Resistance", "Hardness", "Fluorescence", "Toxicity", "Hardness"],
    "Property_Scalar":[1, 50, 10, 300, 39, 80]
})

Chemical_Formula
Property_name
Property_Scalar

0
He
Electrical conduc.
1

1
NO_2
Resistance
50

2
CuO3
Hardness
10

3
CuO3
Fluorescence
300

4
He
Toxicity
39

5
NO2
Hardness
80

You can use pivot to "unmelt" this in a wide format
df3 = df2.pivot(index="Chemical_Formula", columns="Property_name")

Chemical_Formula
('Property_Scalar', 'Electrical conduc.')
('Property_Scalar', 'Fluorescence')
('Property_Scalar', 'Hardness')
('Property_Scalar', 'Resistance')
('Property_Scalar', 'Toxicity')

CuO3
nan
300
10
nan
nan

He
1
nan
nan
nan
39

NO2
nan
nan
80
nan
nan

NO_2
nan
nan
nan
50
nan

From then on you can drop columns you don't need and plot them.


Attention using Context Vector: Hierarchical Attention Networks for Document Classification In the paper, "Hierarchical Attention Networks for Document Classification", the authors use attention to compose words to sentences and then sentences to a document representation. They make use of a context vector $u_w$ to compute the attention weights for the annotation of each word in the sentence.

The paper states in section 2.2, "The context vector $u_w$ can be seen as a high level representation of a fixed query “what is the informative word” over the words. The word context vector $u_w$ is randomly initialized and jointly learned during the training process."

This implies that the context vector is independent of the sentence input. It remains the same for every sentence and is learned as a parameter of the neural network. If this is the case, how will $u_w$ accurately provide attention weights for words in a random sentence, given that sentences are so diverse in meaning.

I do not understand the workings of $u_w$, since it is independent of the sentence input.

Can someone explain?

I can not completely agree with this previous answer.

The context vector $u_w$ is not computed by the Eq. 5 and 6.
As stated in the paper, $u_w$ is randomly initialised and learned during the training.
Instead, it is $u_{it}$ that is computed by Eq. 5 and 6, using the (again) random initialised  $W_w$ and $b_w$.


Space between an object and the ground truth bounding box Which way is better for drawing the ground truth boxes for object detection?

Drawing as tight as the sides of the box touch the border of the object, or

Make a little space between the box and the object?

The first image is a better bounding box.  A perfect bounding box has no space in between the edges of the object and the box, but it also fully contains the object.


What is the definition of Data Scout? I am looking for a definition of Data Scout.

what is the difference between Data Mining and Data Scout?

Copying from the abstract I found here - read the paper for more insights:

In the year 2011, the CMS collaboration introduced the novel concept
of data scouting, allowing to take data that otherwise would be
rejected by the usual trigger filters. This special data flow, based
on event-size reduction, was created to maintain sensitivity to new
light resonances decaying to jets or muons, with very small online and
offline resources allocated to it. The challenges implied by this new
workflow and the solutions developed within the CMS experiment are
shown. This technique is now a standard ingredient for CMS data-taking
strategy. The present status of data scouting in CMS is presented.

I do not come from a Physics background, and hence I could not completely understand the context while reading the paper in the larger scheme of things. But atleast this gives you a heads up about what it is.
Coming to Data Mining, there are plenty of definitions you can find. My definition would be using raw data to find patterns or insights from large datasets. This could be done with the use of many algorithms from math, computer science or statistics.
Some data mining algorithms are k-nearestneighbor, k-means, naive bayes, and so on.
So what is the difference between data scouting and mining? In mining, you consider raw datasets and perform some operations. In scouting, if I am not wrong, you take the data discarded by some filters (do not know what these filters are) while performing some operation, and then considering that data to perform additional operations.


Manipulating multi-indices for a pandas dataframe I have a pandas dataframe with multi-index. I have couple of questions on this.

The indices are week numbers (38 to 42) and for each week, day of the week (DOW). So it looks like

The problem is the 2nd level index, that is, DofWeek, is automatically sorted in lexicographic ordering. But I want to have the usual weekdays ordering, i,e for week 39, I want it to appear as Monday, Tuesday, Wednesday, Thoursday, Friday, Saturday and Sunday and NOT in the order they appear. So when I graph it, it appears this way:

I want it to appear in the usual Monday, Tuesday, Wednesday etc. I have tried reindexing, but it didn't work. This is my first question.

My second question is regarding querying such dataframes. For example when I do dataframe.query('DofWeek == 'Saturday'), it is throwing an error saying Saturday is not defined. However, if I swap levels and then query, it works. For example dataframe.swaplevel().query('weekNum == 39'), it works perfectly. As far as I know, the multi-indices should be sorted in order for certain methods to work. But in my case, they are. Then why isn't it working.

Any help would be hugely appreciated.

Thanks.

Ordering

You can achieve this by changing the datatype to ordered categorical:

df['DofWeek'] = pd.Categorical(
                df['DofWeek'],
                categories=['Monday','Tuesday','Wednesday','Thursday','Friday','Saturday','Sunday'],
                ordered=True)

And then sorting the index:

df = df.sort_index()

Error

The issue with dataframe.query('DofWeek == 'Saturday') is bad quote encapsulation. You could solve it with e.g. 'DofWeek == "Saturday"'


What layer to consider for embeddings when building a encoder-encoder model? I am currently building a encoder-encoder based model with cosine loss function. The dataset is supervised learning.
Here's a basic encoder,
from tensorflow.keras import layers

class LSTMEncoder(layers.Layer):

    def __init__(self,
                 units,  # dimensionality of the output space
                 input_dim,  # vocab of size
                 output_dim,  # embedding dimension
                 name='encoder',
                 **kwargs):
        super(Encoder, self).__init__(name=name, **kwargs)
        self.embedding = layers.Embedding(input_dim=input_dim, output_dim=output_dim)
        self.lstm = layers.LSTM(units=units)

    def call(self, inputs):
        emb = self.embedding(inputs)
        return self.lstm(emb)

Both the encoders are very similar, and has a output layer since it's a supervised learning,
I added a dense layer at the end for probabilities and loss func is consine loss
Now, My goal is not to predict the classes, rather getting the trained embedding for each text sentence from encoder, but encoder has both embedding layer and lstm layer
so I want to know whether to cosider the output of lstm layer as trained embeddings or should I extract embeddings from embedding layer?

You either of them:

If you use the output of the LSTM, you will have contextual embeddings, like BERT or ELMo. This means that each time you want to use your embeddings to encode some input text, you need to pass the input text to your encoder and take the output of the LSTM.
If you use the embedding table, you will have non-contextual embeddings, like word2vec. This means that each time you want to use your embeddings to encode some input text, you just need to look up the appropriate vector in the embedding table.


How does YOLO algorithm detect objects if the grid size is way smaller than the object in the test image? In YOLO algorithm how do these grids output a prediction if some grids only see a small black portion of the car if the model was trained on datasets with full images?

Each grid predictor in YOLO should only have a high score that an object is within it, if it detects the centre of the bounding rectangle is inside itself. So a grid point that contains only the wing mirror of a car should decide it has a low probability of containing the centre of the car.

The predicted bounding rectangles are not constrained in the same way - YOLO can (and often does) predict bounding box dimensions from the centre that are larger than the grid cell dimensions.

Each grid point is independently able to predict whether or not it contains an object's centre, what the bounding box dimensions are for the object, and what the object class is.

If you trace the layer connectivity, you will see that the grid cells are effectively interconnected in lower output layers, so the network as a whole "sees" more of each object and can influence individual object predictions, suppressing some and encouraging others, when objects span multiple grid locations. The grid cells are not isolated into sections, or restricted to only using data from the area that they cover for the prediction. The concept of what part of each image a feature "pixel" can access from the base image in a CNN is called the "receptive field" of the network, and can be calculated based on the architecture as explained in this blog on Medium.


How to understand the return values of scipy.interpolate.splrep Background

Continuation of Spline interpolation - why cube with 2nd derivative
as following Cubic Spline Interpolation in youtube. The example in the youtube is below.

Implemented using scipy.interpolate.splrep and try to understand what the returns of the splrep function are.

  Given the set of data points (x[i], y[i]) determine a smooth spline approximation of degree k on the interval xb <= x <= xe.
  Returns
  tck : tuple
  A tuple (t,c,k) containing the vector of knots, the B-spline coefficients, and the degree of the spline.

import numpy as np
from pylab import plt, mpl

plt.style.use('seaborn')
mpl.rcParams['font.family'] = 'serif'
%matplotlib inline

def create_plot(x, y, styles, labels, axlabels):
    plt.figure(figsize=(10, 6))
    for i in range(len(x)):
        plt.plot(x[i], y[i], styles[i], label=labels[i])
        plt.xlabel(axlabels[0])
        plt.ylabel(axlabels[1])
    plt.legend(loc=0)

x = np.array([3.0, 4.5, 7.0, 9.0])
y = np.array([2.5, 1.0, 2.5, 0.5])
create_plot([x], [y], ['b'], ['y'], ['x', 'y'])

import scipy.interpolate as spi
interpolation = spi.splrep(x, y, k=3)

IX = np.linspace(3, 9, 100)
IY = spi.splev(IX, interpolation)

create_plot(
    [x, IX],
    [y, IY],
    ['b', 'ro'],
    ['x', 'IY:interpolation'],
    ['x', 'y']
)

Questions

How to interpret and understand the return values and which resources to look into to understand?

  A tuple (t,c,k) containing the vector of knots, the B-spline coefficients, and the degree of the spline.

The return value on Knots

interpolation[0]

array([3., 3., 3., 3., 9., 9., 9., 9.])

I thought the first tuple element would be the knots which would be the x, but not. What are these 3., 3. ... values?

The return values on B-spline co-efficient

interpolation[1]

array([ 2.5 , -2.21111111, 6.18888889, 0.5 , 0. , 0. , 0. , 0. ])

Please help or suggest where I should look into and what to understand about "B-spline coefficient" to be able to interpret these values?

The solution of the first interval is (0.186566, 1.6667, 0.24689), hence I thought these values would be in the 2nd element, but not. How the solution values would relate to the return values?

This documentation may work:

BSpline

Univariate spline in the B-spline basis.

$S(x) = \sum_{j=0}^{n-1} c_jB_{j,k;t}(x)$

where $B_{j,k;t}$ are B-spline basis functions of degree k and knots t.


Simple logistic regression wrong predictions In the following example, the prediction obtained for $x=[0,0]$ after fitting the model is $1$, but expected is $0$.

X = np.array([
    [0, 0],
    [1, 0],
    [0, 1]
])

Y = np.array([0,1,1])

model = LogisticRegression()
model.fit(X,Y)

x = [0,0]
y = model.predict([x])[0]

It only works if I duplicate the training row of ($x=[0,0], y=[0]$)

But why?

Because you have a class imbalance and very little data. Your model is essentially working off of a prior probability that disproprtionately favors the positive class, and you didn't give the model enough data to escape this prior in the learning process. Consider the following detailed output from your model:

model.score(X,Y) # 0.6666
model.coef_      # array([[ 0.36566712,  0.36566712]])
model.intercept_ # array([ 0.18517658])

model.predict_proba(X)
# array([[ 0.45383769,  0.54616231],
#        [ 0.36566869,  0.63433131],
#        [ 0.36566869,  0.63433131]])

Your model correctly identifies that [0,0] is more likely to be negative than either of the other observations, but the model isn't able to escape its bias towards the positive class. If you duplicate your dataset a couple of times, the model will eventually have enough "evidence" to escape this bias. Geometrically, two things will happen when we give the model more data:

The discrimination boundary will shift towards the negative class. This will manifest as an increasingly negative intercept term.
The slope of the logistic curve will steepen, having the effect that a unit change in the inputs will have a larger impact on the outcome. If we give the model enough perfectly separable data, the curve will approach a step function and the model will output probabilities of 0 and 1. This manifests in the magnitude of the coefficients. With just three observations, the curve is fairly flat and changing the values of the inputs doesn't have much effect on the outcome probability, so the resulting class assignment is essentially determined by the intercept alone.

Contrast the results from your original model with the what happens when we replicate your dataset 100 times:

model2 = LogisticRegression()
X2 = np.matlib.repmat(X,100,1)
Y2 = np.matlib.repmat(Y,100,1).ravel()
model2.fit(X2, Y2)

model2.score(X,Y) # 1.0
model2.coef_      # array([[ 4.95489068,  4.95489068]])
model2.intercept_ # array([-2.00091433])
model2.predict_proba(X)

# array([[ 0.88089304,  0.11910696],
#        [ 0.04954891,  0.95045109],
#        [ 0.04954891,  0.95045109]])


How can I identify the most predictive factors? I've been playing around with bagged trees and random forests. How can I tell what factors most influenced the categorization? Will scikitlearn just spit it out, or is it trickier than that?

It basically lies within the fitted object.

model.fit(X, y)
importances = model.feature_importances_

The importances are in the same order as the columns of features ($X$).


What is Dropout in FC layer? I know the purpose of dropout is to avoid overfitting by deactivating some neurons.

However, I am interested in how it's done, ie: Math behind it, or intuition about why it works.

How

Adding a dropout layer to a neural network has a few functions. First of all, here is some simple code that implements it:

In [1]: import numpy as np

In [2]: weights = np.random.randint(0, 100, (8, 8))    # example weights

In [3]: weights
Out[3]:
array([[10, 48,  9, 87, 23, 18, 60, 83],
       [29, 48, 88, 43, 34, 56, 52, 82],
       [35,  0, 70,  3,  8, 88,  6, 15],
       [39, 16, 59, 91, 33, 13, 53, 73],
       [20, 56, 70, 35, 16, 12, 80,  6],
       [21, 17, 23, 21, 72, 93, 58, 56],
       [26, 86, 38, 90, 91, 87, 65,  0],
       [67,  6, 10, 94, 19, 25, 49, 61]])

In [4]: dropout_mask = np.random.randint(0, 2, (8, 8))    # either 0 or 1

In [5]: dropout_mask
Out[5]:
array([[1, 1, 0, 0, 0, 1, 0, 1],
       [1, 0, 1, 1, 0, 0, 0, 0],
       [1, 0, 0, 1, 0, 1, 1, 0],
       [0, 0, 0, 0, 1, 0, 1, 0],
       [0, 1, 0, 0, 0, 0, 1, 0],
       [0, 1, 1, 1, 0, 0, 0, 1],
       [0, 1, 1, 0, 1, 1, 1, 1],
       [1, 0, 0, 1, 0, 1, 1, 1]])

        # we make a copy here just to use helpful variable names
In [6]: masked_weights = weights.copy()

        # where the mask is "on" (==1), we set those weights to 0
In [7]: masked_weights[dropout_mask == 1] = 0

In [8]: masked_weights
Out[8]:
array([[ 0,  0,  9, 87, 23,  0, 60,  0],
       [ 0, 48,  0,  0, 34, 56, 52, 82],
       [ 0,  0, 70,  0,  8,  0,  0, 15],
       [39, 16, 59, 91,  0, 13,  0, 73],
       [20,  0, 70, 35, 16, 12,  0,  6],
       [21,  0,  0,  0, 72, 93, 58,  0],
       [26,  0,  0, 90,  0,  0,  0,  0],
       [ 0,  6, 10,  0, 19,  0,  0,  0]])

So we have selected roughly half of weights and turn them off for a pass through the network.

EDIT:

Droput randomly drops neurons on each pass in training, as shown above, but during test time (a.k.a. inference), the dropout layers are deactivated by default. This means that all neurons are available and are used. There is still, however, the issue of scaling - so while all neurons are active during inference, their outputs are scaled to reflect the layers overall output such that the expected overall sum remains unchanged. This can be performed a few different ways - see comments on this answer for related links.

There are of course nice implementations of dropout in all deep learning frameworks (PyTorch, Tensorflow, CNTK, and so on). As is discussed and shown here.

Intuition

One intuitive explanation, is that is ensures each neuron increases its robustness and manages to contribute to the network in its own right. What I mean by this, is that dropout prevent chain or clusters of neurons being strongly dependent on one another. It helps focus the neurons to be able to deliver some useful information in as many different scenarios as possible, e.g. when it only receives signal from a subset of the neurons in the preceding layer.

It can be helpful to think of dropout in terms of regularisation. We add regularisation terms to models in order to penalise them again certain behaviour. An example would be in linear regression via the Bayesian Information Criterion, whereby we increase a model's error if it uses more and more covariates. We also choose the form of error/regularisation when computing values in backpropagation e.g. we select the sum of squared errors (the $L_2$ loss), or just the raw error (the $L_1$ loss). Dropout, on the other hand, applies a change the network itself! We are constricting it by simply removing access to a random selection of neurons during one single pass through the network.

Maths

There actually is not a lot of maths involved, with the original paper explaining the concept in no more than four lines:

The magic really happens in that second line, with the element-wise multiplication of a Bernoulli distributed random variable $\textbf{r}$ and the vector outputs of the preceding layer, $\textbf{y}$. This performs exactly the same operation as IN [7] in my code example above.

I believe the reason is that it was inspired by the intuitive idea, easy to implement and finally proven to work exceptionally well in practice. The above equations can then be applied to various problems - I would suggest reading the paper for some examples.

For further understanding of dropout (and everything around neural network concepts!), I heartily recommend reading Michael Nielsen's book. Exaplantions of dropout with excellently helpful diagrams can be found in Chapter 3.


How to Validate Decision Tree model by using *statistical tests*? I'm reading sklearn Decision Trees reference page.

In the advantages section, it is mentioned that 'Possible to validate a model using statistical tests. That makes it possible to account for the reliability of the model.'

Can someone please explain what statistical tests are performed to validate the DT model?

I am assuming you are using the D.T. for binary classification for a moment. One of the first tests I learned (and still dam good) is the 2x2 contingency table or frequency table or marginal frequencies or 2-way tables (many names means it's been around a while). It is simple to use and can branch off into many other tests and areas. Such as;

Phi Coefficient of Association
Chi-Square Test of Association
Fisher Exact Probability Test
Accuracy and precision
F-score or F-measure, ...(to name a few)

I usually like Kahn Academy way over Wikipedia(hate their explanations).


Mathematical bias and weight vs machine learning bias and weight I am a little confused about the term Bias and Weight with respect to machine learning.
Say we want to predict the heights of people whose weights are given. So plot weights to x-axis and height to yaxis. To find out the linear relationship between height and weight we draw a straight line that shows the relationship between height and weight.

Using the equation for a line, you could write down this relationship
$y= mx+b ...i)$ more specifically in the machine learning terminology it could be $y= b+ w_1x_1...ii)$
So here b is the bias as per machine learning. However, b is the y-intercept as per mathematics. By defination b can be defined in machine learning A value indicating how far apart the average of predictions is from the average of labels in the dataset. Is that mean bias(b) is the distance between some particular point of the red line(as per picture) to the true point(say a blue or green point).
Now another confusion if this is the case then what is loss? As per defination loss is A measure of how far a model's predictions are from its label. Then what is the difference between loss and bias?
Now, for weight, here, weight(m)means slope as per equation i). Mathematically slope can be define as
$m = \frac{rise}{run} =\frac{y_2 - y_1}{x_2 - x_1}$
However, weight$(w_1)$ can be defined in machine learning as A coefficient for a feature in a linear model. So my confusion is that is the procedure of finding weight is same as finding slope in mathematics?

Okay so let's start with the first question:

Is that mean bias(b) is the distance between some particular point of the red line(as per picture) to the true point(say a blue or green point).

You'd be correct had you used the word difference rather than distance. Bias is the difference between the estimated value and the true value. Think of it in this way, if your weighing machine always showed 5kg less for every measurement then you'd be adding 5 to every weight it measures, as a way to offset all measures made by the machine. So if your data is 'biased' towards one side then this bias term offsets this behaviour by adding/subtracting the 'biases of the weights. Here's some more rigorous explanation to it.

what is the difference between loss and bias?

Loss, on the other hand, is a measure of how close your estimator is to true prediction, there are different loss function that one can employ with different type of training, for linear regression we can use MSE. Using loss functions we can objectively optimize our weights and biases. So, having an appropriate value of bias(and weight) causes our loss to be lower, thereby increasing the accuracy of our model.

So my confusion is that is the procedure of finding weight is same as finding slope in mathematics?

The procedure is different, but the end result is the same. To see why you'd need to realize is that you are calculating $\hat y$ as the following: Given $m,b$ and a point $x$, calculate the value of $\hat y$ which is then used to calculate the loss. This value along with $m$ and $b$ is then used to calculate a better value for $m$ and $b$ using Gradient Descent. Notice at no point are we talking about directly calculating $m$ using $\hat y$, We are iteratively calculating better values for weights and biases. The end result? For single-dimension data, the weight that the model has learnt is indeed the slope (when plotted) and bias is the y-intercept. This stands true as the number of dimension increases but then rather than being single values, it becomes a multi-dimensional vector.


Features reduction for the not correlated data set I am working with classification problem on a training data set, which have 100 features. All the features in pairs haven't visible correlation. One can see it in the example pair plot for the some of features:

I am trying to find the right way to decrease the number of features.
All the methods of the important feature selection return different sets of features. For example:

The MARS returns only five important features, but,
Correlation with threshold >1% selects twenty of them.
Lasso selects near twenty but selected features are different from the feature set, returned by correlation method.
RandomForest selects near twenty-five features different from previous methods.
The feature rating in the returned feature sets is different.
The PCA method is not applicable, because there are not any visible linear correlated features.
etc.

My point is the different tools return different important features. One can combine/union them, one can intersect them. One can use N methods of selection and as a result, one will get N feature sets.

The method: "Check them all" is not applicable there. As an example, one can calculate the time effort if N sets multiplied M prediction models, multiplied 3 rounds of the model tuning. The time effort will be even more if one will union or intersect feature sets. It will take forever! There should be some tactics, an algorithm for the filter a final selection.

How to select the best set of the features if the data set is large, non-correlated and noisy?

I believe that what you are looking for is Best Subset, Forward Stepwise, or Lasso?

Here is the R implementation Best Subset

If your dataset is too large, try to subsample your data and then running the feature selection algorithm several times. If you are getting similar results there you go, if the results are different then rank them and select the most frequent ones.

If anyone knows a python implementation for the best subest algorithm, please let me know!


Find the Weight Values using Linear Regression Analysis in Python I have the equation below, which is a noise quality metric of an image:

If the BIQS is 1, it means the image is clean. Else, if it is between 0 and 1, it means the image may contain blur and noise. The smaller the value of BIQS, the dirty is the image. The task is to get the weights value for w1, w2,w3 and w4, given the set of values Blur_mean, Blur_noise, Noise_mean and Noise_ratio, which reflects the image quality. The weights reflects how much each term (Blur_mean, Blur_noise, Noise_mean and Noise_ratio ) contributes to the total image quality score. For Example:

     Blur_mean     Blur_noise    Noise_mean       Noise_ratio
1.   0.8398212     0.8948329     0.9849308        0.89428493
2.   0.8989332     0.8989432     0.7576812        0.35546632
3.   0.4324123     0.3123232     0.2123332        0.33213233
n.   .........     .........     .........        ..........

I don't know the basis for identifying the appropriate values or weight for w1,w2,w3 and w4. Is it possible to use linear regression analysis to find the values of w1, w2,w3 and w4 with almost/ close to 1 R-square or small standard error ? From what I observe, there are 4 independent variables here:

`Blur_mean`
`Blur_noise`
`Noise_mean`
`Noise_ratio`

Dependent Variable is BIQS and the unknowns are w1, w2, w3 and w4. Is there any library that takes care of it in Python?

Try the scikit-learn library.

From the documentation for linear regression:
"LinearRegression fits a linear model with coefficients w = (w1, ..., wp) to minimize the residual sum of squares between the observed targets in the dataset, and the targets predicted by the linear approximation."

https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html


Same validation accuracy, different train accuracy for two neural networks models I'm performing emotion classification over FER2013 dataset. I'm trying to measure different models performance, and when I checked ImageDataGenerator with a model I had already used I came up with the following situation:

Model without data augmentation got:

train_accuracy = 0.76
val_accuracy = 0.70

Model with data augmentation got:

train_accuracy = 0.86
val_accuracy = 0.70

As you can see, validation accuracy is the same in both models, but train accuracy is significantly different. In this case:

Should I go with the model which uses data augmentation, as it's
train accuracy is higher?
Should I expect overfitting from it, and choose the model without
data augmentation as it's accuracy values are closer?
Third option. Should I perform more comprobations? If so, which ones?

Thanks for your time.

Based on this, both models generalize equally well. However both have overfit, the second one more significantly. You simply want to avoid that. For example, I'd expect the best validation loss to be closer to training loss if you're using early stopping. You can turn up things like dropout.

If you do so, I think you'll find the augmented model ends up producing a better model (lower validation loss).


Why do so many functions used in data science have derivatives of the form f(x)*(1-f(x))? The sigmoid function's derivative is of that form, and so is the softmax function's. Is this by design, or some strange coincidence that seems to work for ML models/neural networks?

Sigmoid function is a partial case of softmax, when the number of classes $K=2$. That's why the similarity of their derivatives shouldn't surprise you.

  Why do so many functions used in data science have derivatives of the
  form f(x)*(1-f(x))?

If you consider the following differential equation

$y' = y \cdot (1-y)$

you will find the general solution in the form

$y(x) = \frac{e^x}{c + e^x} = \frac{1}{1 + c e^{-x}}$

So, in some sense, there are not that many functions with this property: they are "very like" sigmoid function.


Difference in labelling and normalizing train/test data I am working on a dataset comprised of almost 17000 data points. Since it's a financial dataset and the components are many different companies, I need necessarily to split it by date. Therefore, supposing I have 10 years of data, I am training over the first 8 years and testing over the remaining 2. This approach I am pretty sure is consistent with the classification problem I need to do.

I am using LSTM network for predicting the direction of financial returns, depending on a bunch of features which are derived from companies' financial statements. Starting from the fact I am obtaining training accuracy greater than test accuracy with almost any architectures and hyperparameter configuration, I suppose there is something wrong in the way I have manipulated the dataset.

Here comes my concerns. I labelled my dataset looking at the median returns and putting 1 if the return for a single data point (company value at a specific date) is above such median, 0 otherwise. Am I correct if I compute two different medians? So that I labelled the training set using its median return and in the same way the test set using its own median return? Should I compute the median over the entire dataset, label it and then splitting?

Moreover, I scaled the training data to be in a range of (0,1). Should I do the same kind of normalization with my test set? I did it, but I wasn't sure about it.

It's kind of my first application of neural networks and I need those clarification about hwo treating the dataset, without influencing the results.

Ideally, there should be no information leakage between your training and test sets. You need to scale your test data using the bounds found for the training data. And, you need to calculate the median with respect to your training data. Think as if you'll have the test samples one by one, not as a batch. Then, how would you calculate your test median, and how would you scale it?


Why infinite sampling is not realisitc assumpition in most real applications I came across the below paragraphs, which I believe are the answers to the question Why infinite sampling is not realistic assumption in most real applications. Still i dont get the below explanation ?. When we draw more samples from the environment, MC brings the approximate value function close to the true value function isn't it ? Then why infinite sampling is not considered as a realistic assumption.

We made two unlikely assumptions above in order to easily obtain this guarantee of convergence for the Monte Carlo method. One was that the episodes have exploring starts, and the other was that policy evaluation could be done with an infinite number of episodes. To obtain a practical algorithm we will have to remove both assumptions. We postpone consideration of the first assumption until later in this chapter.
The second approach to avoiding the infinite number of episodes nominally required for policy evaluation is to forgo trying to complete policy evaluation before returning to policy improvement. On each evaluation step we move the value function toward , but we do not expect to actually get close except over many steps. We used this idea when we first introduced the idea of GPI in Section 4.6. One extreme form of the idea is value iteration, in which only one iteration of iterative policy evaluation is performed between each step of policy improvement. The in-place version of value iteration is even more extreme; there we alternate between improvement and evaluation steps for single states.
For now we focus on the assumption that policy evaluation operates on an infinite number of episodes. This assumption is relatively easy to remove. In fact, the same issue arises even in classical DP methods such as iterative policy evaluation, which also converge only asymptotically to the true value function. In both DP and Monte Carlo cases there are two ways to solve the problem. One is to hold firm to the idea of approximating in each policy evaluation. Measurements and assumptions are made to obtain bounds on the magnitude and probability of error in the estimates, and then sufficient steps are taken during each policy evaluation to assure that these bounds are sufficiently small. This approach can probably be made completely satisfactory in the sense of guaranteeing correct convergence up to some level of approximation. However, it is also likely to require far too many episodes to be useful in practice on any but the smallest problems.

It is not a realistic assumption because you don't have infinite time or decimal precision to find the absolutely correct value function, but you don't need that anyway since a rough estimate of it will be enough to improve the policy.

If your question is why you can't let the agent learn indefinitely in real applications, I'm guessing it is because it may be potentially expensive or dangerous to let it explore randomly in a real scenario, so you want to deploy it with an optimal or near-optimal, deterministic, policy.


Are there any good NLP APIs for comparing strings in terms of semantic similarity? I want to create a chatbot which informs the user about traffic at the streets but not in real-time for the moment. I have created a small database with MySQL which has some data stored regarding traffic and I fetch them with a PHP script whenever this is appropriate depending on the interaction of the user with the chatbot.

I wonder how to deal with the case when the user asks variations of the same question which therefore can be answered with the same answer.
For example:

Why is there traffic at High Street?
What is the cause of traffic at High Street?
Why did I encounter traffic at High Street?
I am stuck in traffic at High Street. Why is this?

Obviously, I can start by removing stopwords (e.g. did), by naming entities (e.g. road -> High Street), by defining synonyms and by applying a text similarity measure (e.g. Levenshtein distance etc).

However, I feel like reinventing the wheel if I do this. Therefore, my question is:

Are there any APIs which can compare strings in terms of semantic similarity (without even requiring training)?

I know that there software platforms such as Dialogflow which are suitable for these tasks but still you must explicitly state all the variations of the same question so that you will get the same answer. Therefore, I look for a API where you will explicitly state only one of these variations of the same question (e.g. Why is there traffic at High Street?) and then the API will figure out by itself which other variations are identical to it in terms of meaning or not.

You can use the Universal Sentence Encoder from Google and calculate the similarity between texts using the cosine similarity or angular distance between their vector representations.


Which predictive model is appropriate? I'm completely lost when trying to choose the type of predictive model for my problem. Is it autoregressive model, nonlinear time series, Markov Chain or other? Can someone please give me some advise?

78,
18,
51,
89,
19,
43,
62,
28,
94,
49

Suppose, everyday I'm given 10 data, and an example was listed above. They're random numbers generated by two devices, namely Device A and Device B. Each of them is capable to generate random numbers from 0 to 9.

The first number in the data is generated by Device A, while the second number is generated by Device B. For instance, for the first data of "78", "7" was generated by Device A and "8" was generated by Device B. Similarly, for the last data of "49", "4" was generated by Device A, and "9" was generated by Device B.

I want to be able to predict the next outcome variable after the last "49".

I have a total of 300 historical data for 30 days.

From my initial investigation for the 300 data, every device tends to produce repeated sequences. For instance, Device A will repeat the sequence "6-2-9-4" (as in the last 4 data). That means this sequence appeared twice within the 300 historical data for Device A. For another example, the sequence "8-1-9-9" (the 2nd to the 5th data) in Device B appeared twice, too. Each of them produce at least three repeated sequences.

I'd like to predict the next outcome variable after the last "49". Which model is more appropriate?

Thank you in advance!

I will assume that by "random" you mean that the numbers don't follow any particular mathematical function. If they were really random then there wouldn't be any pattern to discover so there would be no point trying to predict anything.

From your description I understand the following:

The value of the digit doesn't have any numerical property. In particular the natural order doesn't play any role. This suggests that the digits can be considered as categorical variables.
The data is sequential (the order in the sequence of digits matters) but there is no notion of time involved.
Apparently the two devices produce two independent sequences. You might want to check but if this is the case this calls for two distinct models, one for each device (otherwise you should use a single joint model).

Based on these observations I would use a simple sequential model such as Hidden Markov Model or Conditional Random Fields.


Converting continuous to categorical variable What method must be chosen for converting a continuous variable(socio-economic ratio) into a categorical variable, the quantiles are as follows:

df['ses'].quantile([0,0.25,0.5,0.75,1])

0.00     0.000
0.25     1.070
0.50     1.979
0.75     3.341
1.00    11.889

Whenever I want to convert a continuous feature into categorical with bins, I use one of the following two ways.

Freedman–Diaconis rule (wikipedia source)

Bin width = 2 x IQR(x) x n^(-1/3)

Where IQR(x) is max-min and n is the number of your observations in your sample.

Frequency Bins

You get such a bin width that each of them will have the same number of observations.

I don't have a rule of thumb, where to use each of them. I usually try both and measure the impact on accuracy. Then decide which I will use in the final model.


How can I run Labeled LDA over one textual document? I have 200K tweets and I already a applied the LDA (Latent Dirichlet Allocation) algorithm using Gensim python library. And now I need to apply over them the labeled/supervised LDA. Can any one help me the steps of this task that I should follow. Otherwise, I will be thankful if you can provide me another python library to do so.

A python package that you can check. It's an online max-margin topic model.


Combine multiple classifiers to build a multi-modal classifier Suppose I am interested in classifying a set of instances composed by different content types, e.g.:

a piece of text
an image

as relevant or non-relevant for a specific class C.

In my classification process I perform the following steps:

Given a sample, I subdivide it in text and image
A first SVM binary classifier (SVM-text), trained only on text, classifies the text as relevant/non-relevant for the class C
A second SVM binary classifier (SVM-image), trained only on images, classifies the image as relevant/non-relevant for the class C

Both SVM-text and SVM-image produce an estimate of the probability of the analyzed content (text or image) of being relevant for the class C. Given this, I am able to state whether the text is relevant for C and the image is relevant for C.

However, these estimates are valid for segments of the original sample (either the text or the image), while it is not clear how to obtain a general opinion on the whole original sample (text+image). How can I combine conveniently the opinions of the two classifiers, so as to obtain a classification for the whole original sample?

Basically, you can do one of two things:

Combine features from both classifiers. I.e., instead of SVM-text and SVM-image you may train single SVM that uses both - textual and visual features.
Use ensemble learning. If you already have probabilities from separate classifiers, you can simply use them as weights and compute weighted average. For more sophisticated cases there are Bayesian combiners (each classifier has its prior), boosting algorithms (e.g. see AdaBoost) and others.

Note, that ensembles where initially created for combining different learners, not different sets if features. In this later case ensembles have advantage mostly in cases when different kinds of features just can't be combined in a single vector efficiently. But in general, combing features is simpler and more straightforward.


Problem with escape character in paste-function in R I am trying to paste together a path-string. Like this "C:\testfile.txt"
This doesn't work at all:
filename <- "testfile.txt"

path <- paste("C:\\\",filename,sep="")

This also doesn't produce the right result:

filename <- "testfile.txt"

path <- paste("C:\\\\",filename,sep="")

[1] "C:\\\testfile.txt"

Can you help me to prudece a path-string with only one backslash?

This works:
> paste("C:\\",filename,sep="")
[1] "C:\\testfile.txt"

Although you see TWO backslashes, there's really only one there, because usually when printed a backslash means "the next character is encoding a special character". In this case the special character is literally a backslash.
You can see this by counting the number of characters in two backslashes:
> nchar("\\")
[1] 1

And anyway you should probably be using file.path to construct paths and not fiddling around with backslashes:
> file.path("C:",filename)
[1] "C:/testfile.txt"

(Note forward slashes here also work in Windows and don't suffer from having to be backslash-encoded).


How can we conclude that an optimization algorithm is better than another one When we test a new optimization algorithm, what the process that we need to do?For example, do we need to run the algorithm several times, and pick a best performance,i.e., in terms of accuracy, f1 score .etc, and do the same for an old optimization algorithm, or do we need to compute the average performance,i.e.,the average value of accuracy or f1 scores for these runs, to show that it is better than the old optimization algorithm? Because when I read the papers on a new optimization algorithm, I don't know how they calculate the performance and draw the train-loss vs iters curves, because it has random effects, and for different runs we may get different performance and different curves. So do we compare best performance or average performance?

Whenever possible, you should use the average performance when comparing different methods, and preferably even mention the standard deviation across different runs (see this question for an example why it's important sometimes). It's perfectly fine to also provide the best performance, ideally you can even present a boxplot comparison of the different methods.

What is really unacceptable is to compare a best performance for one method against a mean performance for the other method (it should go without saying but I remember a paper where the authors were happily doing just that in order to make their method look better).


Splitting image dataset with few subjects but many data I'm carrying out training/testing of a convolutional neural network for facial expression recognition with various datasets - all labelled by 7 emotion classes.

For other datasets, there are a large number of mostly unique subjects so I randomly split. In this case, however, there are only 6 subjects but a large number of images for each subject in each class. Randomly splitting seems ineffective because of the similarity in images - think of how an emotion changes per frame.

Is the best method to separate an entire subject for testing? Or something else?

I did run the network with random splitting and achieved 100% validation accuracy so I believe that is unlikely to be the best method. Thanks for your time.

I think you're hitting on the fact that by training and testing on the same subjects your model is not going to be able to generalize to new subjects very well. If you're only interested predicting emotions for these subjects, you are taking the right approach. However, if you want to generalize your model to new, unseen subjects, you should split your training and testing sets such that subjects in the training set are not in the test set, and vice versa to get a more accurate test score. Most likely it will not perform as well and you should consider collecting more data on new subjects if possible.

Here is a similar question on Stack Exchange.


Statistical machine translation word alignment for FR-ENG and ENG-FR: what is p(e) and p(f)? I'm currently trying to implement this paper, but am struggling to understand some of the math here. I'm pretty sure I understand how to implement the E-step, but for the M-step, I'm confused on how to compute the M-step. It says just before section 3.1 that $p_1(x, z; \theta_1) = p(e)p(a, f|e; \theta_1)$, and then the same for $p_2$ but with $e$ and $f$ swapped. The second part of this makes sense to me, but what is $p(e)$ or $p(f)$? From my understanding, $e, f$ are sentences in the bi-text. So how would we compute the probability of a sentence?

It says earlier that $p(e)$ and $p(f)$ are arbitrary distributions that don't affect the optimization problem, but then how do we compute $p_1(x, z; \theta_1)$?

Thanks!

You are right that $p(e)$ is the probability of the English sentence. Estimating the probability of a sentence is achieved by a language model.

This kind of machine translation model is known as the noisy channel model. The noisy channel model says that given a french sentence $f$, its best English translation is

$$e^* = \arg\max_{e\in E} p(e)p(f|e)$$

In this equation the $p(e)$ is the language model. Back in the era of IBM models (which are built upon the noisy channel approach), it is usually an n-gram based language model, calculated as (assuming bigram) $$p(e_1e_2...e_n)=p(e_1|<s>)p(e_2|e_1)p(e_3|e_2)...p(</s>|e_n)$$

And $p(f|e)$ is the translation model where you need to use the EM algorithm to solve. Inside the EM algorithm you do not update the language model parameters, so yes, $p(e)$ and $p(f)$ don't affect the optimization problem.


How do I remove outliers from my data? Should I use RobustScaler? I am aware I can use DecisionTree but I want to use XGBoost How do I remove outliers from my data? Should I use RobustScaler? I am aware I can use DecisionTree but I want to use XGBoost...
Please can you help me, This is a bit urgent, I am not sure how to do it, I have researched and seen previous question but it doesnt work well and was not helpful.
Thank you

Cheers

First of all, you don't need to remove outlier because "Decision family algorithm" like XGBoost can handle it.
Secondly, you can use Tukey method (Tukey JW., 1977):

def detect_outliers(df,n,features):
    outlier_indices = []
    # iterate over features(columns)
    for col in features:
        # 1st quartile (25%)
        Q1 = np.percentile(df[col], 25)
        # 3rd quartile (75%)
        Q3 = np.percentile(df[col],75)
        # Interquartile range (IQR)
        IQR = Q3 - Q1
        # outlier step
        outlier_step = 1.5 * IQR
        # Determine a list of indices of outliers for feature col
        outlier_list_col = df[(df[col] < Q1 - outlier_step) | (df[col] > Q3 + outlier_step )].index
        # append the found outlier indices for col to the list of outlier indices
        outlier_indices.extend(outlier_list_col)
        # select observations containing more than 2 outliers
        outlier_indices = Counter(outlier_indices)
        multiple_outliers = list( k for k, v in outlier_indices.items() if v > n )
        return multiple_outliers
Outliers_to_drop = detect_outliers(data,2,["col1","col2"])
data.loc[Outliers_to_drop] # Show the outliers rows
# Drop outliers
data= data.drop(Outliers_to_drop, axis = 0).reset_index(drop=True)

https://www.kaggle.com/yassineghouzam/titanic-top-4-with-ensemble-modeling

And Thirdly, I suggest you try discrete (binning) continuous variable instead of remove outlier for xgboost.


predict gives the same output value for every image (Keras) I am trying to classify images and assign them label 1 or 0. (Skin cancer or not).

I am aware of the three main issues regarding having the same output in every input.

I did not split the set and I'm just trying to apply the CNN on the train set, I know it doesn't make sense but it's just to verify how it's working. (Predicting on the unlabeled data gives exact same probability)

I have verified the three main points:

1: Scaling the date (both image size and pixel intensity values)
2: Taking a low learning rate
3: I only tried with small epochs 6 at most because of the computation time, is it worth it to let it run one day just to see results with more epochs ?

Anyway I can't understand how a bad training could lead the network to give same class probability every time ?

I tried the on batch options etc.. doesn't change anything.

Accuracy is very low as this kind of classification is not really suited for CNNs but this shouldn't explain the weird result though.

Here is my output :

Here are different parts of the program:
Model :

Resizing :

Thanks for help and sorry for the ugly screenshots.

When all the predictions are giving exact the same value you know that your model is not learning thus something is wrong!

In your case the problem is having the last dense layer with the softmax AND the sigmoid activation.

model.add(keras.layers.Dense(1, activation=tf.nn.softmax))
model.add(keras.layers.Activation('sigmoid'))

This is creating a conflict where the softmax is outputting a 1 (since there is only one node) and the sigmoid takes this 1 and computing its sigmoid value gives:

  1/(1+exp(-1)) = 0.731058

And there is our friend!

To solve this, you just need to remove the last activation layer, and change the softmax for a sigmoid since your output is binary:

model = keras.Sequential()
model.add(keras.layers.Conv2D(16, [3,3], activation='relu', padding='same'))
model.add(keras.layers.Conv2D(32, [3,3], activation='relu', padding='same'))
model.add(keras.layers.Conv2D(64, [3,3], activation='relu', padding='same'))

model.add(keras.layers.BatchNormalization())
model.add(keras.layers.Dropout(0.15))
model.add(keras.layers.Activation('relu'))
model.add(keras.layers.Flatten())
model.add(keras.layers.Dense(50))
model.add(keras.layers.Dense(1, activation=tf.nn.sigmoid))
#model.add(keras.layers.Activation('sigmoid'))

This should work!


What does test score provide in terms of real application? Initially, I wanted to create a classification tool for some program codes in a specific environment based on the application patterns they show. I had 5 different patterns I wanted to look for and eventually my aim was to come up with some kind of statistics about this programming environment that somehow represents the trends (like %40 of the programs are for voting purposes, there has been an increase in the trading programs in the last month, etc). Now, after months of labeling data and training models, I finally obtained 5 different models each one for a different pattern that has varying scores on different metrics (I had quite an unbalanced training data for each class, so just to keep things somehow consisted, always tried to look at different metrics like f1, mcc, roc auc all at once for comparison).

Now, my question is, what do exactly my test scores tell me about my real life data when I apply my models to data that is not possible for me to label or see? Is it possible to derive different statistics as mentioned above using my trained models or am I only limited to my training & testing data? Is it at least possible to mention some kind of probability on a single program, like this is predicted to be a voting program with %X probability?

I would appreciate if you could help me figure out how to interpret the test results from a perspective of final application.

Evaluation is a crucial part of any serious ML project, but when it comes to evaluation choices there is no perfect answer. Generally speaking one evaluates a system in order to know how well it performs (i.e. how reliable are its predictions), usually to know which level of quality to expect when used in production (but not only). However the evaluation results are only useful to some extent:

the evaluation method/measures should be chosen so that it actually represents "quality" for the target task. In general this is imperfect because no evaluation score can fully represent the diversity of a specific task (and that's assuming good evaluation choices)
the test data should be representative of the production data, and in reality it rarely is from the same distribution.
There's almost always a chance factor that one tries to eliminate using cross-validation or other methods, but again that's imperfect.

Besides these unavoidable simplifications, an evaluation score can be interpreted for exactly what it is. For example a precision score of 80% for class X means that one should expect an instance predicted X as truly of class X in 80% of the cases in average. Predicting a specific probability for a particular prediction requires either using a probabilistic model or devising a specific model which predicts a confidence score/probability... But in both cases the model could still can be wrong about the probability it predicts!


Load data from multiple dataframes containing both path and labels in keras I am aware that there exists a function in keras.preprocessing.image.ImageDataGenerator called flow_from_dataframe. But this function assumes that we have one dataframe containing all the paths to the images and labels associated with it. But what if we have multiple dataframes and we want to use these dataframes on a per-epoch basis so that in each epoch we can load a different dataframe and the labels and images associated with it?
Thanks in advance.

I believe the training will continue if you don't recompile and we put dataset only in the last step.
So, something similar to this snippet should work
train_datagen = ImageDataGenerator(rescale = 1./255, shear_range = 0.2, horizontal_flip = True)

training_set_1 = train_datagen.flow_from_dataframe(dataframe=df, color_mode='grayscale',target_size = (28, 28,1), batch_size = 2)
training_set_2 = train_datagen.flow_from_dataframe(dataframe=df, color_mode='grayscale',target_size = (28, 28,1), batch_size = 2)

model = keras.models.Sequential([ ... ])

optimizer = keras.optimizers.SGD(lr=0.2, momentum=0.9, decay=0.01)
model.compile(loss="binary_crossentropy", optimizer=optimizer,
              metrics=["accuracy"])

for i in list(range(10)):
    history = model.fit(training_set_1,epochs=1)
    history = model.fit(training_set_2,epochs=1)

Definitely, learning can be very noisy based on the difference in variance in datagens.
 Also,history will be reset everytime.


Why use both validation set and test set? Consider a neural network:

For a given set of data, we divide it into training, validation and test set. Suppose we do it in the classic 60:20:20 ratio, then we prevent overfitting by validating the network by checking it on validation set. Then what is the need to test it on the test set to check its performance?

Won't the error on the test set be somewhat same as the validation set as for the network it is an unseen data just like the validation set and also both of them are same in number?

Instead can't we increase the training set by merging the test set to it so that we have more training data and the network trains better and then use validation set to prevent overfitting?
Why don't we do this?

Let's assume that you are training a model whose performance depends on a set of hyperparameters. In the case of a neural network, these parameters may be for instance the learning rate or the number of training iterations.

Given a choice of hyperparameter values, you use the training set to train the model. But, how do you set the values for the hyperparameters? That's what the validation set is for. You can use it to evaluate the performance of your model for different combinations of hyperparameter values (e.g. by means of a grid search process) and keep the best trained model.

But, how does your selected model compares to other different models? Is your neural network performing better than, let's say, a random forest trained with the same combination of training/test data? You cannot compare based on the validation set, because that validation set was part of the fitting of your model. You used it to select the hyperparameter values!

The test set allows you to compare different models in an unbiased way, by basing your comparisons in data that were not use in any part of your training/hyperparameter selection process.


Avoiding Overfitting with a large LSTM net on a small amount of data I'm reposting this question from AI.SE here as I think it was maybe off-topic for AI.SE...
1. Context
I'm studying Health-Monitoring techniques, and I practice on the C-MAPSS dataset. The goal is to predict the Remaining Useful Life (RUL) of an engine given sensor measurements series. There's a wide litterature about the C-MAPSS dataset, including both classical (non-DL) ML techniques and DL-based approaches. A few years ago, LSTM-based networks showed promising results (see Long Short-Term Memory Network for Remaining Useful Life estimation, Zheng et al, 2017), and I'm trying to reproduce these results.
The C-MAPSS dataset contains a low amount of data. The FD001 subset has for instance only 100 run-to-failure series. When I pre process it to get fixed-length time series, I can get up to ~20 000 framed series. In the article mentioned above using LSTM, they use two hidden LSTM layers with 64 units each, and two fully-connected layers with 8 neurons each (~55 000 parameters).
2. Problem
LSTMs induce a great number of parameter, so overfitting may be encountered when training such a network. I can use L1 or L2 regularization, dropouts, the net will still be largely oversized regarding to the dataset. Keeping the same architecture, I can't reach the scores and RMSE in the paper in the validation set, and overfitting is always here.
However, one thing that works is reducing the number of units of the LSTM layers. Expectedly, with only 24 units instead of 64 per layer, the net has much less parameters (~9000), and it presents no overfitting. The scores and RMSE are a bit worse than the one in the paper, but it's the best I can get so far. Although these results are fine for me, I'm curious about how it was possible for the authors of the paper to avoid overfitting on their LSTM(64,64) net.
3. Question
LSTM are great, but they induce a lot of parameters that hinder a correct learning on small dataset : I wonder if there is any method to tackle this specific issue. Would you have any advice on how to avoid overfitting with a LSTM-based net on a small dataset ?
4. Infos
I provide here below more infos about my net and results :
Network architecture
model = keras.models.Sequential([
    keras.layers.LSTM(24, return_sequences=True, kernel_regularizer=keras.regularizers.l1(0.01),
                      input_shape=input_shape),
    keras.layers.Dropout(0.2),
    keras.layers.LSTM(24, return_sequences=False, kernel_regularizer=keras.regularizers.l1(0.01)),
    keras.layers.Dropout(0.2),
    keras.layers.Dense(8, activation='relu', kernel_regularizer=keras.regularizers.l2()),
    keras.layers.Dropout(0.2),
    keras.layers.Dense(8, activation='relu', kernel_regularizer=keras.regularizers.l2(), bias_regularizer=keras.regularizers.l2()),
    keras.layers.Dense(1, activation='relu')
])

Scores (Validation set)

Paper:  Score = 16.14 ; RMSE = 338
My LSTM(64, 64): Score = 26.47; RMSE = 3585 (overfits)
My LSTM(24, 24): Score = 16.82; RMSE = 515

Edit : Results for solution proposed by @hH1sG0n3

LSTM(64, 64) with recurrent_dropout=0.3 : Score = 16.36; RMSE = 545

You may want to check a couple of hyperparameters that it appears you are not testing for in your code above:

Gradient clipping: large updates to weights during training can cause a numerical overflow or underflow often referred to as “exploding gradients.”

# configure your optimizer with gradient norm clipping
opt = SGD(lr=0.01, momentum=0.9, clipnorm=1.0)

Reccurent dropout: Dropout that is applied to the recurrent input signal of the units of the LSTM layer.

keras.layers.LSTM(24, kernel_regularizer=keras.regularizers.l1(0.01), ..., recurrent_dropout=0.3)

Stateful: Is it clear from the paper if the model retains its state with each training iteration. You can experiment with this as well.

keras.layers.LSTM(24, kernel_regularizer=keras.regularizers.l1(0.01), ..., stateful=True)


How to make a region of interest proposal from convolutional feature maps? Problem
Keras does not have any direct implementation of region of interest pooling. I am aware of how to perform maxpooling, but I don't know how to get bounding boxes from feature maps passed from convolutional layer.
Is there any way to directly implement a region proposal algorithm?

Example
Let's say there is an architecture like this:

So we have a multi-input neural network architecture that eventually leads to the ROI MaxPool layer. We have three inputs, screenshot, textmaps and candidates, let's take candidates out. Then we would have such code in Keras:
from keras.models import Model
from keras.layers import Input, Dense, Conv2D, ZeroPadding2D, MaxPooling2D, BatchNormalization, concatenate
from keras.activations import relu
from keras.initializers import RandomUniform, Constant, TruncatedNormal

#  Network 1, Layer 1
screenshot = Input(shape=(1280, 1280, 0),
                   dtype='float32',
                   name='screenshot')
# padded1 = ZeroPadding2D(padding=5, data_format=None)(screenshot)
conv1 = Conv2D(filters=96,
               kernel_size=11,
               strides=(4, 4),
               activation=relu,
               padding='same')(screenshot)
# conv1 = Conv2D(filters=96, kernel_size=11, strides=(4, 4), activation=relu, padding='same')(padded1)
pooling1 = MaxPooling2D(pool_size=(3, 3),
                        strides=(2, 2),
                        padding='same')(conv1)
normalized1 = BatchNormalization()(pooling1)  # https://stats.stackexchange.com/questions/145768/importance-of-local-response-normalization-in-cnn

# Network 1, Layer 2

# padded2 = ZeroPadding2D(padding=2, data_format=None)(normalized1)
conv2 = Conv2D(filters=256,
               kernel_size=5,
               activation=relu,
               padding='same')(normalized1)
# conv2 = Conv2D(filters=256, kernel_size=5, activation=relu, padding='same')(padded2)
normalized2 = BatchNormalization()(conv2)
# padded3 = ZeroPadding2D(padding=1, data_format=None)(normalized2)
conv3 = Conv2D(filters=384,
               kernel_size=3,
               activation=relu,
               padding='same',
               kernel_initializer=TruncatedNormal(stddev=0.01),
               bias_initializer=Constant(value=0.1))(normalized2)
# conv3 = Conv2D(filters=384, kernel_size=3, activation=relu, padding='same',
#               kernel_initializer=RandomUniform(stddev=0.1),
#               bias_initializer=Constant(value=0.1))(padded3)

# Network 2, Layer 1

textmaps = Input(shape=(160, 160, 128),
                 dtype='float32',
                 name='textmaps')
txt_conv1 = Conv2D(filters=48,
                   kernel_size=1,
                   activation=relu,
                   padding='same',
                   kernel_initializer=TruncatedNormal(stddev=0.01),
                   bias_initializer=Constant(value=0.1))(textmaps)

# (Network 1 + Network 2), Layer 1

merged = concatenate([conv3, txt_conv1], axis=-1)
merged_padding = ZeroPadding2D(padding=2, data_format=None)(merged)
merged_conv = Conv2D(filters=96,
                     kernel_size=5,
                     activation=relu, padding='same',
                     kernel_initializer=TruncatedNormal(stddev=0.01),
                     bias_initializer=Constant(value=0.1))(merged_padding)

If you look at the end of the code (and architecture itself), we pass concatenated activations from two different Conv+ReLu layers and then pass it to ROI MaxPool layer.

Thank you!

To implement region proposal you need two major parts:

The region proposal network that generates a set of candidate bounding boxes. It can be implemented simply as two convolutional layers to 1) predict the object presense and 2) predict offsets for the default (anchor bounding boxes)
The ROI pooling layer that provides a fixed-size feature vector for an arbitrary sized proposal.

Here is an implementation of Faster R-CNN in Keras, and here is a detailed explanation of the model and the code.

Here is implementations of the RPN, and here is implementation of the ROI pooling.


Support Vector Machine (SVM) for classification problem based on Earth Mover's Distance (EMD) I would like to run SVM for my classification problem using the Earth Mover's Distance (EMD) as a distance measurement. As I understood the documentation for Python scikit-learn (https://scikit-learn.org/stable/modules/svm.html#svm-kernels) it is possible to use custom kernel functions:

import numpy as np
from sklearn import svm

def my_kernel(X, Y):
    return np.dot(X, Y.T)

clf = svm.SVC(kernel=my_kernel)

Also there is a package with EMD implemented (https://pypi.org/project/pyemd/). I tried to run it similar as in example using my own data (below). I have distributions of eigenvalues. But there is an error and can't figure out what does it mean.

from pyemd import emd
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split

# Construct train and test data
X = np.concatenate((eigs_ctrl_fmri, eigs_pat_fmri))
y = np.concatenate(([0] * eigs_ctrl_fmri.shape[0], [1] * eigs_pat_fmri.shape[0]))

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=42) # 70% training and 30% test

model_svm = SVC(kernel=emd)
model_svm.fit(X_train, y_train)

TypeError: emd() takes at least 3 positional arguments (2 given)

So, it means that for emd function we need one more argument. Example in documentation says that this is distance matrix:

from pyemd import emd
import numpy as np
first_histogram = np.array([0.0, 1.0])
second_histogram = np.array([5.0, 3.0])
distance_matrix = np.array([[0.0, 0.5],
                             [0.5, 0.0]])
emd(first_histogram, second_histogram, distance_matrix)

3.5

But how I should construct distance matrix for SVM model - in advance calculate the distance between all samples in my data? If it is, I don't really understand how I should provide an input to SVM model.

And one thing about EMD - why do I need the distance matrix to calculate the earth mover's distance? For me it looks strange that I need to calculate the distance and provide it as an input to the function which calculating the distance. I undertstand that I missing something in basic understanding the EMD principle and how to use it for SVM. Can somebody help me please with that?

Thank you for any info to boost my understanding!

pyemd has a different API than scikit-learn. Per the scikit-learn docs:

"Your kernel must take as arguments two matrices of shape (n_samples_1, n_features), (n_samples_2, n_features) and return a kernel matrix of shape (n_samples_1, n_samples_2)."

As you rightly point out, the EMD implemented by pyemd takes three arguments, but the distance_metric is not the distance matrix between the distributions. It defines the distance between the histogram bin edges, which is independent of the particular distribution.

If you want to use the EMD implemented by pyemd, I'd suggest wrapping it in another function so that you match the function signature expected by scikit-learn. That will require you to compute distance_metric in the wrapper and pass it as an argument to pyemd.emd(). Again, the distance_metric argument only depends on how you have chosen the bin edges for the discretized distribution.


How to use Cross Validation data set? I am new to data science, and the dataset I am working on is divided into train set, test set, and validation set.
However, till now I was splitting the data with train_test_split. I expected model.fit() would have an option like validation =. But, there is none.
How I can use the CV set to improve results?

Well, there are ways to do it within model.fit(). Namely, you can define validation_split parameter, (e.g. 0.2 or 0.3) which represents a percentage of the data that will be used as a validation in training (0.2 = 20%, 0.3 = 30%).
The second way, when you have your validation data, is to use validation_data = (x_val, y_val), where you just pass your validation x and y


Saving images in a non-retraceable way, but still able to train R-CNN's on them For a computer vision project I am working with images that the company only allows me to have on my computer for a maximum of 24 hours due to regulations. Every day a few hundred images come in via the system. I am trying to solve a object detection problem to prioritize workflow. For this I need more than a few hundred images to train RCNN's.

Q: I am wondering if there is a way to save images in such a way that they are NOT re-traceable anymore to the original image, while at the same time it IS possible to train some kind of RCNN on them?

I was thinking about saving saving convolutional features and labeled bounding boxes instead of the images. But that way it might be re-traceable and I want to be compliant with the regulations. Also, this way I can't train the convolutional layers of a network anymore, since I will only be working with it's output.

Wow. It is common for medical images to be heavily regulated (have to be kept on a separate server, cannot be copied, monitored access, etc...), but your situation is even tougher! Anyway, here's a few thoughts:

I don't think you can extract conv features, because a great deal of information (sometimes nearly everything) for reconstruction can be stored in those features (e.g., see Understanding Deep Image Representations by Inverting Them). Presumably this means you cannot simply train some kind of autoencoder and keep the latent representations either.
You could try to run the training in an "online" manner, meaning train on the new set of images each time they come. However, we must fight to avoid overfitting, so some heavy regularization (dropout, weight decay, early stopping, maybe even using a Bayesian neural network) and data augmentation probably would be needed. One methodology very relevant to this is work on catastrophic forgetting and continual learning. For instance, see Overcoming catastrophic forgetting in neural networks. You may be able to adapt these techniques to your case by treating each new set of data as a new task. One approach is to identify the important parameters each time you've trained on a new data subset, and prevent them from being changed to much when training on the next set.
A related approach is to use ensembles (similar to boosting). I imagine something like this (I'm sure you can think of something better): let $B_t$ be the $t$th data subset you receive.
Suppose you already have a set of models
$\mathcal{M}_{t-1} = M_1,\ldots,M_{t-1}$
with fixed weights from the previous time steps
(denote $\Theta_i$ as the parameters of $M_i$).
Let ${f}$ be a model that combines the outputs of $\mathcal{M}_{t-1}$ (i.e., if $M_i:\mathcal{I}\rightarrow{\mathcal{D}}$, then $f:2^\mathcal{D}\rightarrow\mathcal{S}$).
Then you train $f$ and $M_t$ on $B_t$, but your loss focuses on data examples where the outputs of $\mathcal{M}_{t-1}$ are poor.
The idea is to avoid overfitting by training $M_t$ to only care about learning things that the other older models don't already know.
This is a little tough in the detection case (I haven't thought about how to do it honestly), since the output is complicated (maybe an architecture like Deep Sets could help).
If it was simple regression ($\mathcal{S}=\mathbb{R}^n$), you could have $f(I)=\sigma_a(\sum_{\ell=1}^t c_\ell M_\ell(I))$, with a regularized loss function that enforces $c_u,\,u<t$ don't change too much, and $c_t$ isn't too large (note only all the $c_k$'s and $\Theta_t$ are trainable for $B_t$).
One other idea might be to train generative models (like GANs), and use them to generate novel data from the same distribution as these inputs. How to avoid forgetting for the critics is an issue of course.
Usually, you can generate a GAN sample $S$, find its nearest neighbour in the original set $N_S$, and (empirically) $S$ tends to be fairly far from $N_S$. But this may not be sufficient to pass the requirements in your case.
Finally, you may be able to use some work from the privacy and fairness machine learning literature. The idea is that you can destroy or ignore certain "undesirable" information in the model, data, and/or representations, in order to preserve privacy and/or enforce some notion of fairness in the model. In your case, you may be able to train some kind of "fair, minimal autoencoder" that can learn a representation of your data that does not contain problematic properties, yet is still useful for training. For instance, if the images are sensitive because they contain information about some particular property of a given person, you may be able to learn a representation that does not hold information about that property (and then save and keep those). It really depends on the specifics of your situation of course.
I am not an expert in this area, unfortunately, but a place to start might be Learning Adversarially Fair and Transferable Representations.