Reason of use of Product rule of Probability in Predicting Error P(x,y) = P(y|x)P(x)
Why do we use this in estimating expected prediction error?
i.e. E{(y - f(x))^2}
I researched and I came to know that it helps in figuring out noise but How?

The general idea is that there is a true joint probability $p(x,y)$ which can be factorized like
$$p(x,y) = p(y|x)p(x)$$
$p(x)$ basically measures the probability of observing the input $x$, while $p(y|x)$ represents the true relationship between $x$ and $y$.
We would like to learn a model distribution $p_\theta(x, y)$ which is factorized accordingly
$$p_\theta(x,y) = p_\theta(y|x)p(x)$$
$p(x)$ can be considered the same in both cases, so what we are trying to find is $p_\theta(y|x)$ as an approximation to $p(y|x)$ by optimizing the model parameters $\theta$.
In the case of regression, we make the assumption that there is a function $f_\theta(x)$ and the observed $y$ is distributed normally around $f_\theta(x)$ with noise variance $\sigma^2$, i.e.
$$p_\theta(y|x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left[-\frac{(y-f_\theta(x))^2}{2\sigma^2} \right]$$
Since $p_\theta(x,y)$ should approximate $p(x,y)$ as closely as possible, we want to minimize the KL-divergence between the two
$$
\begin{align}
&D_{KL} \left[ p(x,y) | p_\theta(x,y) \right] \\
&= \int \int p(x,y) \log\left[ \frac{p(x,y)}{p_\theta(x,y)} \right] dx dy \\
&= \int \int p(x,y) \log p(x,y) dx dy - \int \int p(x,y) \log p_\theta(x,y) dx dy
\end{align}
$$
Note that the first integral does not depend on $\theta$. The second term can be further simplified using the facotization
$$\int \int p(x,y) \log p_\theta(x,y) dx dy = \int \int p(x,y) \log p_\theta(y|x) dx dy + \int \int p(x,y) \log p(x) dx dy$$
Here the second term does not depend on $\theta$. Using all this, we can write the optimization problem for the KL-divergence as
$$
\begin{align}
\theta^* &= \text{argmin}_\theta D_{KL} \left[ p(x,y) | p_\theta(x,y) \right] \\
&= \text{argmin}_\theta \left[ - \int \int p(x,y) \log p_\theta(y|x) dx dy \right]
\end{align}
$$
Plugging in the model assumption about $p_\theta(y|x)$ this gives
$$
\begin{align}
\theta^* &= \text{argmin}_\theta \left[ \int \int p(x,y) \frac{(y-f_\theta(x))^2}{2\sigma^2} dx dy + \log\sqrt{2\pi\sigma^2}\right] \\
&= \text{argmin}_\theta \left[ \int \int p(x,y) (y-f_\theta(x))^2 dx dy \right]
\end{align}
$$
where we use that neither the constant shift nor the constant scale change the solution of the $\text{argmin}_\theta$.
This way we have derived the mean squared error as the loss for regression from probabilties and their factorizations. The final step would be that we use the empirical risk to approximate the expectaion / integral, i.e.
$$
\int \int p(x,y) (y-f_\theta(x))^2 dx dy \approx \frac{1}{N} \sum_{n=1}^N (y_n - f_\theta(x_n))^2
$$
which repsoduces the mean square error as a loss.


What are the effects of clipping the reward in stability? I am looking for stabilizing my results of DQN, I found clipping is one technique to do it but I did not understand it completely!

1- what are the effects of clipping the reward, clipping the gradient, clipping the error in stability and how makes results more stable?

2- In DQN nature it has written they clipping the reward? Would you please explain this more?

3- which of them are more effective in stability?

You could clip for several reasons.

If you clip the gradient, the stabilizing effect is to force the optimizer to do only small changes in the backward step. Of course, you could also decrease the learning rate but the effect is slightly different. When you decrease the learning rate you basically say "learn more slowly". Instead, with gradient clipping, you say "learn as usual but if you have to change your mind rapidly don't do it" (I'm not sure that this sentence is understandable, English isn't my first language).
If you clip the error, the effect is the same. Maybe it changes a bit in a mathematical point of view but the bigger result is equal to clipping the gradient.
Clipping the reward doesn't give you any direct stabilizing effect. It is only a particular case of the more general reward shaping. In Playing Atari with Deep Reinforcement Learning, it's stated that:

  Since the scale of scores varies greatly from game to game, we fixed all positive rewards to be 1 and all negative rewards to be −1, leaving 0 rewards unchanged. Clipping the rewards in this manner limits the scale of the error derivatives and makes it easier to use the same learning rate across multiple games. At the same time, it could affect the performance of our agent since it cannot differentiate between rewards of different magnitude.

Basically, they've done it to make the environments similar to each other in rewards terms when seen by an agent. If you think of it, it's not difficult for you to play to a game when the score is given as multiple of thousand and then play to another one where even one hundred could be a great score. This doesn't apply to RL agents and so they reshaped the reward.


Statistically Robust Distance Measure/Metric for comparing more than two network data series I have about 30 lists of unequal length (some of which are triplicates of the data), corresponding to metrics relating to nodes of different graphs. I want to compare their similarity using a distance metric, but was unsure which method I can use given the data lists are of unequal length. I was exploring using dynamic time warping, but was wondering if there is any other more basic method.

For example, I was a considering creating histograms with same bin edges and number of bins for each list and using a distance metric on the frequency, but I am not sure how to go about this using python, or if there's a function/package that does this already. Is this even a "good" way of doing it?

I'm also interested in finding a way to measure the statistical significance of the distance measures between the different graphs.

This is a lot in one question, I am new to this and appreciate any help.  Thank you in advance!

Just to clarify the question:
Do the lists describe different graphs, or do you need the similarities to learn which lists refer to the same graph?
Is data tri- or duplicated within single lists or are lists duplicating other lists? Would you consider removing redundant metrics?

Sorry, this was rather a comment than an answer, but I am not yet entitled to comment here, so I am starting the discussion like this.


How reliable are model performance reportings? I had a conceptual doubt about estimating and reporting a classification model's performance. Say my model works with range of depth values and gives out different readings of test errors. We choose the model with depth having lowest test error rate as M1

Now if we want to report our model's performance on a hidden test set, would it wise to say that M1 would perform equally well on this new test set and with the same test error rate?

The question of the measured test error of a classification model is reliable, hence if the test error on unknown set $T_1$ is the same as on $T_2$ is hard two answer. It depends on the following factors:

How many digits of the error are reported?
How many samples have $T_1$ and $T_2$? The more digits are reported, the more samples you need. As a rule of thumb, make sure that any change in the reported error means at least 3 samples have changed. So if you use accuracy and report two decimal places (e.g. 12.34%), then 0.01% must be bigger than 3 => $3 < \frac{0.01}{100} \cdot |T_1| \Leftrightarrow 30000 < |T_1|$
The distribution must be similar. The simplest part is the distribution of classes. The more difficult part is how the features look like.

For other forms of error analysis, you might want to look into my Master's thesis Analysis and Optimization of Convolutional Neural Network Architectures


Keras, re use convoluted layers training for various flattened layers training I want to train a range of models, that are very similar; the convoluted layers are the same but the flattened part changes. Example; different density, initially I have model.add(Dense(512)) but I would also like to test 1000 and 4096; I would also like to test adding another layer.
I have no problem creating this models and running them, but it feels like a waste of time to retrain the convoluted layers every time when I'm not modifying anything in them.
Is it possible to save the training done in the convoluted layers and reuse that when training the flattened layers?
This is the base model I have:
model = Sequential()

model.add(Conv2D(32, (3, 3), activation='relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2), padding='same'))

model.add(Conv2D(32, (3, 3), activation='relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2), padding='same'))

model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2), padding='same'))

model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2), padding='same'))

model.add(Conv2D(128, (3, 3), activation='relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2), padding='same'))

model.add(Flatten())
model.add(Dropout(0.2))
# -------------------- Start of Variable Section --------------
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.2))
# -------------------- End of Variable Section ----------------
model.add(Dense(1))
model.add(Activation('sigmoid'))

Other options I want to try (changes to be made inside the variable section) are:
model.add(Dense(1000, activation='relu'))
model.add(Dropout(0.2))

or
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.2))

etc.
To achieve that, where do I save the model? If I remove the flattened section:
model.add(Flatten())
model.add(Dropout(0.2))
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(1))
model.add(Activation('sigmoid'))

The model accuracy is unsurprisingly ~none, if I leave the section that is not variable, like this:
model.add(Flatten())
model.add(Dropout(0.2))
model.add(Dense(1))
model.add(Activation('sigmoid'))

I can save the model and load it, but how would I add those flattened layers?

You can simply train your convolutional layers, save the model and load weights from specific layers using the get_weights and set_weights methods (see also this previous answer). After loading the weights for you convolutional layers you can freeze those layers using the trainable attribute to make sure the weights are not changed during training.


Is it necessary to standardize your data before clustering? Is it necessary to standardize your data before cluster?  In the example from scikit learn about DBSCAN, here they do this in the line:

X = StandardScaler().fit_transform(X)

But I do not understand why it is necessary.  After all, clustering does not assume any particular distribution of data - it is an unsupervised learning method so its objective is to explore the data.

Why would it be necessary to transform the data?

Normalization is not always required, but it rarely hurts.

Some examples:

K-means:

  K-means clustering is "isotropic" in all directions of space and
  therefore tends to produce more or less round (rather than elongated)
  clusters. In this situation leaving variances unequal is equivalent to
  putting more weight on variables with smaller variance.

Example in Matlab:

X = [randn(100,2)+ones(100,2);...
     randn(100,2)-ones(100,2)];

% Introduce denormalization
% X(:, 2) = X(:, 2) * 1000 + 500;

opts = statset('Display','final');

[idx,ctrs] = kmeans(X,2,...
                    'Distance','city',...
                    'Replicates',5,...
                    'Options',opts);

plot(X(idx==1,1),X(idx==1,2),'r.','MarkerSize',12)
hold on
plot(X(idx==2,1),X(idx==2,2),'b.','MarkerSize',12)
plot(ctrs(:,1),ctrs(:,2),'kx',...
     'MarkerSize',12,'LineWidth',2)
plot(ctrs(:,1),ctrs(:,2),'ko',...
     'MarkerSize',12,'LineWidth',2)
legend('Cluster 1','Cluster 2','Centroids',...
       'Location','NW')
title('K-means with normalization')

(FYI: How can I detect if my dataset is clustered or unclustered (i.e. forming one single cluster)

Distributed clustering:

  The comparative analysis shows that the distributed clustering results
  depend on the type of normalization procedure.

Artificial neural network (inputs):

  If the input variables are combined linearly, as in an MLP, then it is
  rarely strictly necessary to standardize the inputs, at least in
  theory. The reason is that any rescaling of an input vector can be
  effectively undone by changing the corresponding weights and biases,
  leaving you with the exact same outputs as you had before. However,
  there are a variety of practical reasons why standardizing the inputs
  can make training faster and reduce the chances of getting stuck in
  local optima. Also, weight decay and Bayesian estimation can be done
  more conveniently with standardized inputs.

Artificial neural network (inputs/outputs)

  Should you do any of these things to your data? The answer is, it
  depends.

  Standardizing either input or target variables tends to make the training
  process better behaved by improving the numerical condition (see
  ftp://ftp.sas.com/pub/neural/illcond/illcond.html) of the optimization
  problem and ensuring that various default values involved in
  initialization and termination are appropriate. Standardizing targets
  can also affect the objective function.

  Standardization of cases should be approached with caution because it
  discards information. If that information is irrelevant, then
  standardizing cases can be quite helpful. If that information is
  important, then standardizing cases can be disastrous.

Interestingly, changing the measurement units may even lead one to see a very different clustering structure: Kaufman, Leonard, and Peter J. Rousseeuw.. "Finding groups in data: An introduction to cluster analysis." (2005).

  In some applications, changing the measurement units may even lead one
  to see a very different clustering structure. For example, the age (in
  years) and height (in centimeters) of four imaginary people are given
  in Table 3 and plotted in Figure 3. It appears that {A, B ) and { C,
  0) are two well-separated clusters. On the other hand, when height is
  expressed in feet one obtains Table 4 and Figure 4, where the obvious
  clusters are now {A, C} and { B, D}. This partition is completely
  different from the first because each subject has received another
  companion. (Figure 4 would have been flattened even more if age had
  been measured in days.)

  To avoid this dependence on the choice of measurement units, one has
  the option of  standardizing the data. This converts the original
  measurements to unitless variables.

Kaufman et al. continues with some interesting considerations (page 11):

  From a philosophical point of view, standardization does not really
  solve the problem. Indeed, the choice of measurement units gives rise
  to relative weights of the variables. Expressing a variable in smaller
  units will lead to a larger range for that variable, which will then
  have a large effect on the resulting structure. On the other hand, by
  standardizing one attempts to give all variables an equal weight, in
  the hope of achieving objectivity. As such, it may be used by a
  practitioner who possesses no prior knowledge. However, it may well be
  that some variables are intrinsically more important than others in a
  particular application, and then the assignment of weights should be
  based on subject-matter knowledge (see, e.g., Abrahamowicz, 1985). On
  the other hand, there have been attempts to devise clustering
  techniques that are independent of the scale of the variables
  (Friedman and Rubin, 1967). The proposal of Hardy and Rasson (1982) is
  to search for a partition that minimizes the total volume of the
  convex hulls of the clusters. In principle such a method is invariant
  with respect to linear transformations of the data, but unfortunately
  no algorithm exists for its implementation (except for an
  approximation that is restricted to two dimensions). Therefore, the
  dilemma of standardization appears unavoidable at present and the
  programs described in this book leave the choice up to the user.


Convolutional Network for Text Classification I am trying to train a convolutional neural network with Keras at recognizing tags for Stack Exchange questions about cooking.

The i-th question element of my data-set is like this:

id                                                         2
title                    How should I cook bacon in an oven?
content    <p>I've heard of people cooking bacon in an ov...
tags                                 oven cooking-time bacon
Name: 1, dtype: object

I have removed tags with BeautifulSoup and removed punctuation too.
Since questions' content are very big I have decided to focus on titles.
I have used sklearn CountVectorizer to vectorize words in titles. However they were more than 8000 words (excluding stop words). So I decided apply a part of speech tagging and retrieve only Nouns and Gerunds.

from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer(stop_words='english')
titles = dataframes['cooking']['title']
pos_titles = []
for i,title in enumerate(titles):
    pos = []
    pt_titl = nltk.pos_tag(word_tokenize(title))
    for pt in pt_titl:
        if pt[1]=='NN' or pt[1]=='NNS' or pt[1]=='VBG':# or pt[1]=='VBP' or pt[1]=='VBS':
            pos.append(pt[0])
    pos_titles.append(" ".join(pos))

This represents my input vector. I have vectorized tags too and extract dense matrixes for both input and tags.

tags = [" ".join(x) for x in dataframes['cooking']['tags']]
Xd = X.todense()

Y = vectorizer.fit_transform(tags)
Yd = Y.todense()

Split data into train and validation set

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(Xd, Yd, test_size=0.33, random_state=42)

Now I am trying to train a Conv1D network

from keras.models import Sequential
from keras.layers import Dense, Activation,Flatten
from keras.layers import Conv2D, MaxPooling2D,Conv1D, Embedding,GlobalMaxPooling1D,Dropout,MaxPooling1D

model = Sequential()

model.add(Embedding(Xd.shape[1],
                    128,
                    input_length=Xd.shape[1]))
model.add(Conv1D(32,5,activation='relu'))
model.add(MaxPooling1D(100,stride=50))
model.add(Conv1D(32,5,activation='relu'))
model.add(GlobalMaxPooling1D())
model.add(Dense(Yd.shape[1], activation ='softmax'))

model.compile(optimizer='rmsprop',
          loss='categorical_crossentropy',
          metrics=['accuracy'])
model.fit(X_train, y_train, batch_size=32,verbose=1)

But it gets stucked on a very low accuracy and it shows a barely increasing loss along the epochs

Epoch 1/10
10320/10320 [==============================] - 401s - loss: 15.8098 - acc: 0.0604
Epoch 2/10
10320/10320 [==============================] - 339s - loss: 15.5671 - acc: 0.0577
Epoch 3/10
10320/10320 [==============================] - 314s - loss: 15.5509 - acc: 0.0578
Epoch 4/10
10320/10320 [==============================] - 34953s - loss: 15.5493 - acc: 0.0578
Epoch 5/10
10320/10320 [==============================] - 323s - loss: 15.5587 - acc: 0.0578
Epoch 6/10
 6272/10320 [=================>............] - ETA: 133s - loss: 15.6005 - acc: 0.0550

This wildml blog post has a very clear explanation of how to use 1D convolution on text. And Debo, DS at x.ai, provided some example Keras code to classify text using a character-based model (input documents are sequences of one-hot encoded characters rather than words or POS tags):

from keras.models import Model
from keras.layers import Input, Dense, Dropout, Flatten
from keras.layers.convolutional import Convolution1D, MaxPooling1D

inputs = Input(shape=(maxlen, vocab_size), name='input', dtype='float32')
conv = Convolution1D(nb_filter=nb_filter, filter_length=filter_kernels[0],
                     border_mode='valid', activation='relu',
                     input_shape=(maxlen, vocab_size))(inputs)
conv = MaxPooling1D(pool_length=3)(conv)
conv1 = Convolution1D(nb_filter=nb_filter, filter_length=filter_kernels[1],
                      border_mode='valid', activation='relu')(conv)
conv1 = MaxPooling1D(pool_length=3)(conv1)
conv2 = Convolution1D(nb_filter=nb_filter, filter_length=filter_kernels[2],
                      border_mode='valid', activation='relu')(conv1)
conv3 = Convolution1D(nb_filter=nb_filter, filter_length=filter_kernels[3],
                      border_mode='valid', activation='relu')(conv2)
conv4 = Convolution1D(nb_filter=nb_filter, filter_length=filter_kernels[4],
                      border_mode='valid', activation='relu')(conv3)
conv5 = Convolution1D(nb_filter=nb_filter, filter_length=filter_kernels[5],
                      border_mode='valid', activation='relu')(conv4)
conv5 = MaxPooling1D(pool_length=3)(conv5)
conv5 = Flatten()(conv5)
z = Dropout(0.5)(Dense(dense_outputs, activation='relu')(conv5))
z = Dropout(0.5)(Dense(dense_outputs, activation='relu')(z))

pred = Dense(n_out, activation='softmax', name='output')(z)
model = Model(input=inputs, output=pred)
model.compile(loss='categorical_crossentropy', optimizer='rmsprop',
              metrics=['accuracy'])

The last 3 lines are important. You can't use softmax on your output and you can't use 'categorical_crossentropy' for multi-label tagging (your problem). Your text tagging problem should be broken down into multiple binary classification problems, or you need to use a different loss function like 'binary_crossentropy'. And for binary_crossentropy, use a sigmoid activation function rather than softmax on the output. See this SO answer for details on multi-label tagging problems in keras and TF.

If you want a more thorough explanation, check out Chapter 7 in my book, NLP In Action.


How to calculate steady/incremental growth? I have timeseries data for stocks at minute intervals.
What is the best way to calculate incremental growth, for example if I have a stock's price from 9am to 2pm at minute intervals, how can calcule the incremental growth. I don't want to simply calculate the difference between the price at 2pm and 9am, I want to be able to identify stocks which have risen steadily between the two time intervals

What stops you looking at lags?
df = data.frame(c(1,2,3,4,4,3,4,5,6,7))
colnames(df)<-c("price")
df$delta=lag(df$price, 1)
df$inc = df$price-df$delta
df

Would yield:
   price delta inc
1      1    NA  NA
2      2     1   1
3      3     2   1
4      4     3   1
5      4     4   0
6      3     4  -1
7      4     3   1
8      5     4   1
9      6     5   1
10     7     6   1

The condition here would be that each value of inc (except the first one) must be $>0$.


Reduce Categorical Values I'm working on one use case where I have to explore source code repo files. Different files will be a categorical values for me. But with such large number of files, One Hot Encoding comes out to be very large.
Also, all files are divided among unique modules, such that each files belongs to a specific module.
So if I consider unique files associated with specific modules comes out to be small in number and One Hot Encoding matrix is also not large.

So if I define different modules as a  categorical features, I'm not sure how to map the associated unique files.
How to derive new feature with such combination of two categorical features?
Any other better way to handle such use case?

Recently practitioners are representing categorical variables as embeddings for  ML models. I can see a solution to your problem there.
As your problem is having a two-level hierarchy you can consider two embeddings, one set of embeddings for modules, and another set for files. For every document, you can take their combination and pass it as input to the model. In this way, both the module and file-level information will be captured.
References:
https://towardsdatascience.com/deep-embeddings-for-categorical-variables-cat2vec-b05c8ab63ac0


How to choose the model parameters (RandomizedSearchCV, .GridSearchCV) or manually Faced with the task of selecting parameters for the lightgbm model, the question accordingly arises, what is the best way to select them? I used the RandomizedSearchCV method, within 10 hours the parameters were selected, but there was no sense in it, the accuracy was the same as when manually entering the parameters at random. +/- the meaning of the parameters is clear, which ones are responsible for retraining, which ones are for the accuracy and speed of training, but it’s not entirely clear if you select manually one at a time or in pairs, or even more options?

Below is an example of how I implemented the selection of parameters:

SEED = 4
NFOLDS = 2
kf = KFold(n_splits= NFOLDS, shuffle=False)

    parameters = {
          'num_leaves': np.arange(100,500,100),
          'min_child_weight': np.arange(0.01,1,0.01),
          'feature_fraction': np.arange(0.1,0.4,0.01),
          'bagging_fraction':np.arange(0.3,0.5,0.01),
          'min_data_in_leaf': np.arange(100,1500,10),
          'objective': ['binary'],
          'max_depth': [-1],
          'learning_rate':np.arange(0.001,0.02,0.001),
          "boosting_type": ['gbdt'],
          "bagging_seed": np.arange(10,42,5),
          "metric": ['auc'],
          "verbosity": [1],
          'reg_alpha': np.arange(0.3,1,0.2),
          'reg_lambda':  np.arange(0.37,0.39,0.001),
          'random_state': [425],
          'n_estimators': [500]}

model = lightgbm.LGBMClassifier()
RSCV = RandomizedSearchCV(model,parameters,scoring='roc_auc',cv=kf.split(train),n_iter=30,verbose=50)
RSCV.fit(train,label)

Thanks for the clarification. You can configure the parameters once or twice at a time by re-instantiating the RSCV object each time, passing different parameter dictionaries each time. For example:

SEED = 4
NFOLDS = 2
kf = KFold(n_splits= NFOLDS, shuffle=False)

    parameters = {
          'num_leaves': np.arange(100,500,100),
          'min_child_weight': np.arange(0.01,1,0.01),
    }

model = lightgbm.LGBMClassifier()

RSCV = RandomizedSearchCV(model,parameters,scoring='roc_auc',cv=kf.split(train),n_iter=30,verbose=50)
RSCV.fit(train,label)

    parameters = {
          'feature_fraction': np.arange(0.1,0.4,0.01),
          'bagging_fraction':np.arange(0.3,0.5,0.01),
          'min_data_in_leaf': np.arange(100,1500,10),
    }

RSCV = RandomizedSearchCV(RSCV.best_estimator_,parameters,scoring='roc_auc',cv=kf.split(train),n_iter=30,verbose=50)
RSCV.fit(train,label)

By passing the RSCV.best_estimator_ instead of model to the second time, it will automatically use the best values for num_leaves and min_child_weight that it identified in the first run and effectively "freeze" those as they are, finding the best combination of feature_fraction, bagging_fraction and min_data_in_leaf under those constraints.

My understanding, however, is that the best approach is to do as you are currently doing and simply include all the parameters in one search. I generally set n_estimators to some lowish value (200 or so) and learning_rate to some reasonably high value whilst doing the tuning, and then beef them up later. This will generally drastically reduce the time taken to perform the tuning. It does necessitate a refit afterwards to take account of the new values.

model = lightgbm.LGMBClassifier(n_estimators=100, learning_rate=0.2

params = {
    'num_leaves':np.arange(100, 500 ,100)
    ...
}

RSCV.fit(train, label)

model = RSCV.best_estimator_
model.set_params(n_estimators=1000, learning_rate=0.001)
model.fit(train, label)

The literature on Randomised Search is usually quoted as being this paper, which I recommend reading: http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf. One key takeaway is the conclusion that roughly 60 trials are needed to find the optimal parameter space with a reasonably high probability, so I'd recommend increasing the n_iter parameter of RSCV to 60.

Finally, the use of np.arange() results in each of your lists being uniformly distributed. For example, the tuning algorithm will select values for feature_fraction between 0.1 and 0.4 with equal likelihood. You can also use Scipy's distribution functions to define other distributions; normal and log-normal for example that targets the search on a particular area (for example, you could make the optimiser more likely to try values closer to 0 than values close to 1 for a the feature_fraction parameter.

Hope all that helps.


Regression Model for explained model(Details inside) I am kind of a newbie on machine learning and I would like to ask some questions based on a problem I have .

Let's say I have x y z as variable and I have values of these variables as time progresses like :

t0  = x0 y0 z0
t1  = x1 y1 z1
tn  = xn yn zn

Now I want a model that when it's given 3 values of x , y , z I want a prediction of them like:

Input : x_test y_test z_test
Output : x_prediction y_prediction z_prediction

These values are float numbers. What is the best model for this kind of problem?
Thanks in advance for all the answers.

More details:
Ok so let me give some more details about the problems so as to be more specific.

I have run certain benchmarks and taken values of performance counters from the cores of a system per interval.

The performance counters are the x , y , z in the above example.They are dependent to each other.Simple example is x = IPC , y  =  Cache misses , z  = Energy at Core.

So I got this dataset of all these performance counters per interval .What I want to do is create a model that after learning from the training dataset , it will be given a certain state of the core ( the performance counters) and predict the performance counters that the core will have in the next interval.

AFAIK if you want to predict the value of one variable, you need to have one or more variables as predictors; i.e.: you assume the behaviour of one variable can be explained by the behaviour of other variables.
In your case you have three independent variables whose value you want to predict, and since you don't mention any other variables, I assume that each variable depends on the others. In that case you could fit three models (for instance, regression models), each of which would predict the value of one variable, based on the others. As an example, to predict x:

x_prediction=int+cy*y_test+cz*z_test

, where int is the intercept and cy, cz, the coefficients of the linear regression.
Likewise, in order to predict y and z:

y_prediction=int+cx*x_test+cx*z_test
z_prediction=int+cx*x_test+cy*y_test


What is Hellinger Distance and when to use it? I am interested in knowing what really happens in  Hellinger Distance (in simple terms). Furthermore, I am also interested in knowing what are types of problems that we can use  Hellinger Distance? What are the benefits of using  Hellinger Distance?

Hellinger distance is a metric to measure the difference between two probability distributions. It is the probabilistic analog of Euclidean distance.

Given two probability distributions, $P$ and $Q$, Hellinger distance is defined as:

$$h(P,Q) = \frac1{\sqrt2}\cdot \|\sqrt{P}-\sqrt{Q}\|_2$$

It is useful when quantifying the difference between two probability distributions. For example, if you estimate a distribution for users and non-users of a service. If the Hellinger distance is small between those groups for some features, then those features are not statistically useful for segmentation.


Meaning of 'hue" in seaborn barplot Seaborn barplot has three parameters.

  x, y, hue : names of variables in data or vector data, optional

Question

What is hue? It seems the attribute to plot but why it is called "hue" because when I googled, the result is about color?

Google

Hue - Wikipedia

  Hue is one of the main properties (called color appearance parameters) of a color, defined technically (in the CIECAM02 model)

In seaborn, the hue parameter determines which column in the data frame should be used for colour encoding. Using the official document for lmplotprovided an example for this.

import seaborn as sns; sns.set(color_codes=True)
tips = sns.load_dataset("tips")
g = sns.lmplot(x="total_bill", y="tip", data=tips)

Adding `hue="smoker" tells seaborn you want to colour the data points for smoker and non-smoker differently.

>>> g = sns.lmplot(x="total_bill", y="tip", hue="smoker", data=tips)


Negative Value in Covid-19 Dataset I am currently practicing my data visualization skills with ourworldindata dataset. I noticed there is a negative value of new_cases in Benin country. Out of curiosity, what does that mean? I mean, why would the new_cases has a negative value?
Here is where the negative value is found.

Looks like that number is calculated from the total_cases column. Value = "total_cases from day before" - "total_cases of that day". In this case: -209 is 130 - 339. Not sure why that number would decrease, but could be they changed the way they count, or just an error in the data.


Identifying outliers in an unknown distribution I have a sorted sequence of integers, e.g.

1,2,480,1000,1100

representing volumes in some categories. The task is to separate the valid data (high volumes) from the outliers (low volumes).

In the above example I expect following obvious split

1,2   |   480,1000,1100

This is of course trivial, if there is a definition of outliers, such as less than .1% of the total volume, but I'd prefer to have an
algorithm for calculating this threshold as a result.

I don't know anything about the distribution of the values, except for the outliers have low values and the regular categories have high volumes.

Intuitively, I'd say the approach should pass all possible N-1 splits of the sequence and evaluate some criterion, but I fail to
find a meaningful one (e.g. I distinctly failed comparing the WSS of the sequence with the sum of the WSS of both splits).

I'd appreciate hints to the algorithm or to transformation of this problem in other well known unsupervised problem.

If the high volume and low volume data both appear in non-negligible probability, you may view the problem as a clustering problem with two clusters. Gaussian mixture model or K-means algorithm may help you in this case.


Training a LSTM on a time serie containing multiple inputs for each timestep I am trying to train a LSTM in order to use it for forecasting. The problem is basically a multivariate multi-step time series problem.
It is simply an experiment to see how statistical models (ARIMA, Holts-Winters, ...) and neural networks compare for a given problem.
As my dataset is perfectly fit for a statistical model, I am having trouble when trying to format it to train the LSTM as I have multiple entries for one timestep (corresponding to different entities) and I don't really know how to deal with it since the sequence is no longer tied by the time of observation. Let's say my dataset looks like the following example :
time | ent | obs

  1 ---   1 ------      5

  2 ---   1 ------     6

  2 ---   5 ------     1

  3 ---   2 ------     7

  3 ---   5 ------     4

As you can see, not every entity have an entry for any given time, and one timestep can have multiple entries.
I thought of training the LSTM for each entity but I would have too few data for most of them. Some threads gave me the idea to separate each entity into batches but the number of observations is not constant so it wouldn't work for me.
How do you think I am supposed to tackle this problem?

The answer to this question highly depends on what relationship between the variables you are interested in.

If you are interested in the relationship between time and observation-value, treating the entities as different batches could make sense, under the assumption that the role of individual entities doesn't really matter to you. In this case, you would, for example, add the mean of each entity (or the overall mean) to all entities with missing values to get a constant number of observations per entity. But you could also simply average all values in each timestamp and include other features as min & max. This would most probably deliver better results.
If you are interested in the relationship between entities and observation-value, this is a matter of missing data in time series. There are a lot of techniques that can help you with that from simply imputing the mean to more sophisticated methods like a Kalman filter. However, in the end, you will have to ask yourself why these observations are missing and choose the appropriate method. But since you are using time-dependent models in your experiment, I assume, this is not of interest to you.
If you are interested in the interrelationship of all three variables, you are dealing with panel data. In this case, I don't see a reasonable possibility to model this with an LSTM. Maybe another RNN-architecture could work, however, the only paper I found was Tensorial Recurrent Neural Networks for Longitudinal Data Analysis from Mingyuan et.al. But in the end, it would not matter, since an ARIMA-model also isn't appropriate for panel data. Usually, you use a Difference-In-Differences approach for that kind of data. In this case, I would suggest changing the dataset for your experiment.


How to deal with missing data for only some categories Or in other words, data for category A is irrelevant for category B. So it is not present, how can imputing missing data distort/effect learning models broadly.
I can't find any logic how to deal with this relative data. So I am sorry that I don't show any effort.

In the following example, geographical zone is only present for Gaz entries.

Data sample:

There are three types of missing data: Missing Completely At Random (MCAR), Missing At Random (MAR) and Missing Not At Random (MNAR).

Your case is the second, where according to wikipedia it:

  occurs when the missingness is not random, but where missingness can be fully accounted for by variables where there is complete information

This means that the presence or not of entries in zone can be derived from the column Produit.

Because the values aren't missing completely at random, normal imputation techniques (e.g. fill with the most common value) shouldn't be applied. Instead I'd recommend treating the missing values as their own category. Just create a category (let's say not available) and fill the missing with this value. From a data science view this makes more sense.


does accuracy and performance depends on whole dataset or fraction of that data mimics the whole? MNIST dataset is handwritten images which contains 60000 training and 10000 testing samples , can I use only a fraction of dataset for instance 10000 for training and 4000 for testing. I want to know that whether my results will be comparable to other claimed results. Like I used random forest classifier on 10k training and 4k testing and got accuracy of 93 %. would this result be equivalent when I use 60k training and 10k testing .

and in some published papers I read that they only used 40 images per class and evaluated their accuracy and performance using them does it conforms to the standards that use 60k as training and 10k as testing.
Or we can say that the choice of selection of training and testing data is dependent on the method used to classify

This is actually a pretty complex question that has to do with many facets of how information entropy is carried in a dataset and how your learning algorithms will make use of the information.

Machine Learning is Optimization

Machine learning at its core is just finding the ideal parameter values in order for a specific function to be represented. This means you will use the data in your dataset and some framework in order to tune these parameters. Learning is an iterative process, as you can see providing one instance to your algorithm will not result in very good results. However, there comes a point when you have provided sufficient data that the information contained in that subset is representative of the information contained in the entire dataset.

You can see this in practice, when training your model you will notice that performance starts to converge to a specific value. If you had infinite data this value would not get any better. This is caused by limitations in your data and not your model (machine learning algorithm). This is the best the specific model you are using can do with that kind of data.

How to get better performance?

Assuming you still have more data but performance is no longer improving, you will want to consider a more complex model. This model will likely be able to extract information from your dataset that is too subtle for your simpler model. This is why deep learning outperforms most classical machine learning algorithms when big data is available. Deep learning makes use of very high complexity to approximate functions very accurately, however, the tradeoff is that a lot of data is needed.

Actually my dataset is quite limited, can I improve my performance?

You actually can! This goes back to the core of machine learning, the dataset you are using. The dataset you have constructed has a certain information entropy based on the features you selected. By increasing the information entropy in your dataset, the complexity of your model can be reduced and this will lead to better performance.

Intuitively, if you are trying to classify dog/cat, but your dataset contains information about the weather the day the measurements were taken, this will reduce the entropy of the information for classifying dog/cat, thus obscuring your other features and reducing performance.

Furthermore, sometimes you may overlook the importance of a certain feature, or overlooking the potential of performing transformations to your features to better suit your model. For example, if you are using linear regression to determine points on 2 circles (radius=1, radius=4). Performing linear regression on the Cartesian plane will lead to very poor results, essentially 50%. However, transforming your $x, y$ coordinates into polar coordinates will lead to 100% performance as they will be perfectly separable. This is a simple application of the kernel trick.


How to handle such a large class imbalance in text data? I am working on a multi class text classifier. The total number of class that are there is 265 and total number of rows is 20,000. The class with largest number of occurrences  has 6000 samples and there are many classes with exactly 1 sample as well. Preliminary analysis of the data led me to put a cutoff of 10 samples to be their for a class to be recognized and I made a separate miscellaneous class of less than 10 samples. Now I am reduced to 27 classes. And there is still class imbalance as can be seen from the figure with the class having largest number of samples has around 6000 and the lowest has 10.

How do I deal with such large class imbalance? Are their algorithms which are better suited to handle such large class imbalance?

]1ps://imgur.com/a/8hJw9MJ)

For the most part, as you add more classes to your multiclass classification problem, it becomes more difficult to construct a model. All algorithms can run into trouble because there are fewer examples of each class to learn from. In case the 27 class target feature does not work well for you, you can decrease the number of classes even more. To do this:

Continue what you already did by grouping small classes into the "other" category. Perhaps a model with 2 or 3 classes would be useful for you.
Try a semi-supervised learning approach. Try clustering of your different categories and group similar categories into the same cluster. Your model will now predict cluster A, cluster B, etc.


Is there HMM that looks two elements back? The original HMM model looks only one element back (considers only previous state). Sometimes it viable to look two elements back. Is there an extension to HMM that allows that?

I think the keyword you are looking for is Higher Order Hidden Markov Models. Your "ordinary" HMM would be considered a First Order HMM. So yes, there is literature these types of HMMs.


Steps in exploratory methods for mild-sized data with mixed categorical and numerical values? Experienced in signal/image analysis, and new to data science, I recently was challenged with a relatively simple dataset: 100 to 200 items, about 10-20 numerical variables (in the [0-1] or percentage range), with only one variable used at present time for ranking, and 5 to 10 categorical variables, each with few options. A categorical variable takes about 2 to 4 different values.

I would like first to get insight on potential structures in such data. I have browsed Agresti's Analysis of Ordinal Categorical Data, some have advised me to invest on TDA (Topological data Analysis). Yet I do not know where to start from.

Do you have guidelines, best practices on such REAL data to gradually address the aforementioned issues, from visualization to genuine processing/inference?

You can get a reasonably good approximation of steps for exploratory data analysis (EDA) by reviewing the EDA section of the NIST Engineering Statistics Handbook. Additionally, you might find helpful parts of my related answer here on Data Science SE.

Methods, related to EDA, are too diverse that it is not feasible to discuss them in a single answer. I will just mention several approaches. If you are interested in applying classification to your data set, you might find information, mentioned in my other answer helpful. In order to detect structures in a data set, you can try to apply principal component analysis (PCA). If, on the other hand, you are interested in exploring latent structures in data, consider using exploratory factor analysis (EFA).


Why Pair plot is taking up the target variable into consideration? I am working on Haberman's cancer survival data set.

I tried to visualize the pair plot with features as AGE, Op_Year, axil_nodes_det and target variable as Surv_status. I expect 12 plots but getting 16 including plots with Surv_stautus which is my target variable
here is my code:

sns.set_style("whitegrid");
sns.pairplot(patients, hue="Surv_status", size=3);

maybe because Surv_status is number.
there are two solution:

define variables:

sns.set_style("whitegrid"); sns.pairplot(patients, hue="Surv_status",vars=["AGE", "Op_Year", "axil_nodes_det"], size=3);

change target value to a string:

patients['Surv_status'] = patients['Surv_status'].astype(str)

or

    patients['Surv_status'] = patients['Surv_status'].str


GridSearch without CV I create a Random Forest and Gradient Boosting Regressor by using GridSearchCV. For the Gradient Boosting Regressor, it takes too long for me. But I need to know which are the best parameters for the models. So I am thinking if there is a GridSearch without CV because the OOB score is sufficient to evaluate the models. I hope I could explain what I mean.

GridSearchCV is built around cross validation, but if speed is your main concern, you may be able to get better performance using a smaller number of folds.
From the docs:

class sklearn.model_selection.GridSearchCV(estimator, param_grid, scoring=None, n_jobs=None, iid='deprecated', refit=True, cv=None, verbose=0, pre_dispatch='2*n_jobs', error_score=nan, return_train_score=False)
cv: int, cross-validation generator or an iterable, optional
Determines the cross-validation splitting strategy. Possible inputs for cv are:
    None, to use the default 5-fold cross validation,

    integer, to specify the number of folds in a (Stratified)KFold,

    CV splitter,

    An iterable yielding (train, test) splits as arrays of indices.

For integer/None inputs, if the estimator is a classifier and y is either binary or multiclass, StratifiedKFold is used. In all other cases, KFold is used.

cv defaults to 5, so changing it to 2 should provide a significant speedup for you. This will weaken the cross validation significantly.
Alternatively, you may be able to pass in a single test/train split for your value of cv. This would effectively disable cross validation and remove the benefits that it provides.


Pandas - Sum of multiple specific columns I created this script:
        import pandas as pd
        pd.set_option('display.min_rows', None)
        pd.set_option('display.max_columns', None)
        df = pd.read_excel('file.xlsx', sep=';', skiprows=6)
        df = df.drop(['Position', 'Swap'], axis=1)
        df.drop_duplicates(['Time'])
        df = df.groupby(['Symbol'])[['Profit', 'Volume', 'Commission']].sum()

        df['Comm. ratio'] = (df['Commission'] / df['Profit'])
        df['Comm. ratio'] = df['Comm. ratio'].round(2)
        df['Net profit'] = (df['Profit'] + df['Commission'])
        df.loc['Total'] = pd.Series(df['Commission'].sum(), index = ['Commission'])

    Symbol  Profit Volume Commission Comm. ratio Net profit

    BA
    FB
    NVDA
    US500
    USTEC
    Total                     -45

    df.loc['Total'] = pd.Series(df['Commission'].sum(), index = ['Commission'])

works on "commission" column, but I'd like to have a multiple column sum for "Profit, Commission, and Net profit" in the "Total" row.
I couldn't make it work.
Thanks!

df.loc['Total'] = pd.Series([df['Commission'].sum(),df['Profit'].sum(),df['Net profit'].sum()], index = ['Commission','Profit','Net profit'])


How to select the best parameters for GridSearchCV? I've created a couple of models during some assignments and hackathons using algorithms such as Random Forest and XGBoost and used GridSearchCV to find the best combination of parameters. But what I'm not able to understand is how to select those parameters for GridSearchCV. I randomly put the parameters such as
params = {"max_depth" : [5, 7, 10, 15, 20, 25, 30, 40, 50,100],
         "min_samples_leaf" : [5, 10, 15, 20, 40, 50, 100, 200, 500, 1000,10000],
         "criterion": ["gini","entropy"],
         "n_estimators" : [10, 15, 20, 40, 50, 75, 100,1000],
         "max_features" : ["auto", "sqrt","log2"]}

But how do I decide if I could select better parameters which might be computationally better as well? I can't use the same above parameters for a Random Forest Classifier every single time surely?

That is indeed a drawback with grid search strategy, since you must know in advance each one of the possible combinations to try out, and that might be not optimal neither to get the best evaluation metric value nor in computation performance.
You have other interesting strategies, not exhaustive hyperparameter search, for instance random search or based on bayesian tuning, for a more efficient search and being a "more clever" search strategy in the second option.
You can have a look at HyperOpt library with several optimization algorightms (see also this link for a practical use case), and more recently Keras released a nice keras tuner (which I love by the way).
You can also have a look at this answer for a worked out example on a XGB model using Hyperopt, and this one for using keras tuner.
You can also check the keras tuner wrapper for sklearn models: https://keras-team.github.io/keras-tuner/documentation/tuners/#sklearn-class


What is the best deep learning library for scala? Does any one has a recommendation for what libraries to use for deep learning?

Hi (I created deeplearning4j: warning biased source),

It's fairly new but we are open to feedback. We ported keras to scala:
http://github.com/deeplearning4j/ScalNet/

This uses deeplearning4j underneath. We would love contributions or issues.


Poor performance of regression model for imbalanced data I am trying to train a neural network model to solve a regression problem. The specificity of my dataset is that it has something like an exponential distribution of target values (imbalanced). Therefor, the model seems to output just values less than 2 (if the range is [0,6]), for instance, and it absolutely ignores bigger target's values, which have a smaller performance in the dataset. How is it possible to improve the model's results in such a case?
For example, when it comes to a multiclass classification, we can weigh penalties for errors on a smaller class to enhance performance with imbalanced data. Are there any tricks in terms of regression? Which loss-functions could be useful? It seems, that MSE loss function is more preferable than RMSE. Is it more powerful loss functions for this problem?
There is a paper about such a problem of imbalanced regression (http://proceedings.mlr.press/v74/branco17a/branco17a.pdf) which might be helpful to someone. However, I'm more interested in special tricks for a neural network, not pre-processing approaches (I can't generate more data for example).

Neural Networks can in general be interpreted as a regression problem and as such, you could apply well known ways of dealing with this. This paper gives you a good introduction to different approaches. For instance you can upsample the minority class, or you could do loss weight balancing during training.

For instance, consider the data point $x_i$ that can belong to one of two classes $a$ and $b$. For instance, class $a$ is here the minority. Then you would, during training, multiply the calculated loss with the weight. $$L_i=\begin{cases}l_a\cdot x_i,&\text{if datapoint $x_i$ is in the minority class}\\l_b\cdot x_i,&\text{if datapoint $x_i$ is in the majority class}\end{cases}$$ with $l_a$ > $l_b$ which would be natural for this example.

You could also look into continous performance measures that could work nicely with imbalanced dataset. For instance the generalized dice (F1) score.


Probability that ensemble model is correct based on accuracies of its classifiers I'm trying to understand what I did wrong when trying to answer this question.  The exact question is:

Assume that we have 3 trained prediction models, and each model outputs either
-1 or 1. We then tested the accuracies of these models and obtained the following outcomes:

Model
Accuracy

m1
0.60

m2
0.55

m3
0.45

Let M be the ensemble model that outputs a plurality vote of these three models. If we assume that the errors of the models m1, m2, and m3 are independent, what is the probability that M(x) would be correct on a test instance x?

I thought that because this was a plurality vote, and the classification errors are independent of each other, I could simply take the weighted average of the accuracy of the three classifiers:
$
\begin{align*}
P(X) &= \sum_{all\ models\ M_j} P(C_i|x,M_j)P(M_j) \\
 &= \frac{1}{L} \sum_{all\ models\ M_j} P(C_i|x,M_j) \\
 &= \frac{1}{3}(0.60+0.55+0.45)\\
 &= 0.53
\end{align*}
$
But I was told that this is incorrect (with no context as to why).
Can someone explain why this is incorrect?  If this is a plurality vote (which to me assumes that the votes of each classifier are equal), why can I not simply take the weighted average?

It's not the actual data, it's the probabilities. So you should consider all the scenarios of voting.
For the Ensemble to be correct,
Either any two or all the three should be correct
=$[m_1*m_2*(1- m_3) + m_1*(1-m_2)*m_3) + (1-m_1)*m_2*m_3] + [m_1*m_2*m_3]$
= [0.6*0.55*(1-0.45) + 0.6*(1-0.55)*0.45 + (1-0.6)*0.55*0.45] + 0.6*0.55*0.45
= 0.5505


Unbalanced target variable in Orange, how do I balance it? So I want to perform a predictive model to predict churn.

I have 2 datasets, one with churn and the other without (so I can later perform predictions).

The issue is that I think my Confusion matrix is kinda bad since my target variable is highly unbalanced:

which mostly leads to this confussion matrix:

(Similar values for both logistic regression and decision tree).

This is my workflow:

Is there any way to balance the data? I can't find it in the Orange documentation.

For unbalanced classes use the Python Script widget, with imblearn add-on, you will need to code!

Link for thread in github


Is there a paper accomplishing finding physical law from observation without premade perception, using machine learning? For example:
Isaac Newton finds law of universal gravitation just by looking a falling apple, without any premade perception of that phenomenon. Is it possible to accomplish that kind of discovery using machine learning?

You can check this paper

Discovering physical concepts
with neural networks - https://arxiv.org/pdf/1807.10300.pdf

I quote one of the first sentences

  Here, we present a neural network architecture that can be used to discover physical concepts from experimental data without being provided with additional prior knowledge


On the properties of Hyperbolic Tangent Kernel How do Hyperbolic Tangent Kernels work? That is what is the intuition behind them? Can you provide proofs and examples for illustration?

Hyperbolic Tangent Kernels are defined as:
$$ K(x, x^\prime) = tanh\bigg(\alpha (x\cdot x^\prime) + c\bigg)$$

For example, for Gaussian RBF kernel, the intuition is that the support vectors affect the decision surface based on the locality of influence. What is the analog for Hyperbolic Tangent (Sigmoid Kernels)?

Some references on the Hyperbolic Tangent Kernels are:

Hsuan-Tien Lin and Chih-Jen Lin
Sabri Boughorbel, Jean-Philippe Tarel, Nozha Boujemaa

Sigmoid kernels owe their popularity to neural networks, which traditionally used the sigmoid activation function. Sigmoid kernels de-emphasize extreme correlation. In a way they behave a bit like correlation coefficients, which also has a limited range, emphasizing similarity in orientation. $c$ shifts the operating point on the sigmoid, affecting the relative emphasis of the angle between the inputs. Perhaps this visualization (for $c=0$) might help mentally visualize this:

Your first reference states that sigmoid kernels behave like RBFs for certain parameters. This makes them suited to nonlinear classification. You probably know that the sigmoid kernel is only conditionally PSD, and thus sometimes does not correspond to the kernel function of any implicit feature map per Mercer's theorem.

Someone this interested in kernel methods should have a copy of Learning with Kernels!


Training 128x128 autoencoders on 512x512 images, produces strange gridline after recombining So I'm training an autoencoder that can recreate 128x128 images, so it can recreate any images by splitting them into 128x128 patches first, running it through the autoencoder, and having them combined with each other to form the original image.

I should be using other dimensions too but right now I'm testing this with 512x512 images. So :

   x = next_train_batch(25)

    for xx in range(0,4):
        x_index = xx*128
        for y in range(0,4):
            y_index = y*128

            this_image = x[:, x_index:x_index+128, y_index:y_index+128, :]
            sess.run(training, feed_dict={x_in: this_image, step_iter_global:step_iter})

This is how I'm doing it now. But this keeps happening:

True, this is just in the beginning of the training process. But I guess I wonder why the autoencoder is having a hard time learning the edges of the patches. These gridlines actually are still present in visualizations of certain layers in the autoencoder (the reason I'm doing this is because I'm trying new layers that could potentially improve autoencoders) even if the output image doesn't have it.

The problem is that the model has no idea what the image should look like outside of the 128x128 crop you feed it with, so it fails to extrapolate appropriately. One simple fix is to train the model on bigger (and overlapping!) crops, like 144x144, and then crop out 8 pixels from each side.

However, even this approach does not guarantee absence of any visible edges, as it's hard for the model to enforce this "boundary consistency" without direct access to its output from the previous patch. This could be approached with more powerful decoders, like PixelRNN / PixelCNN


Getting mean and covariance matrix for multivariate normal from keras model I have a dataset that has 6 input features and 5 output features. I want to use a keras sequential model to estimate the mean vector and covariance matrix from any row of input features assuming the output features to be following Multivariate Normal Distribution.
That is for my dataset for any row of 6 input features, I want to get a mean vector of 5 values and a 5*5 covariance matrix.
sample=pd.DataFrame({'X1':[1,2,3,4,5,6],
              'X2':[1,3,1,5,2,7],
              'X3':[3,0,0,7,5,0],
              'X4':[0,4,3,2,5,8],
              'X5':[9,7,0,2,4,5],
              'X6':[1,1,8,7,0,0],
              'Y1':[0.5,1.2,6.3,4.5,1.5,6.6],
              'Y2':[6.1,4.3,2.1,1.5,4.2,8.7],
              'Y3':[0,0,3.2,3.7,5.5,0.2],
              'Y4':[0.5,1.4,8.3,5.2,1.5,1.8],
              'Y5':[2.9,1.7,6.3,5.2,9.4,1.5]})
sample

    X1  X2  X3  X4  X5  X6  Y1  Y2  Y3  Y4  Y5
0   1   1   3   0   9   1   0.5 6.1 0.0 0.5 2.9
1   2   3   0   4   7   1   1.2 4.3 0.0 1.4 1.7
2   3   1   0   3   0   8   6.3 2.1 3.2 8.3 6.3
3   4   5   7   2   2   7   4.5 1.5 3.7 5.2 5.2
4   5   2   5   5   4   0   1.5 4.2 5.5 1.5 9.4
5   6   7   0   8   5   0   6.6 8.7 0.2 1.8 1.5

For loss function I am using the following, which maximizes the log probability.
def lossF(y_true, mu, cov):

  dist = tfp.distributions.MultivariateNormalTriL(loc=mu, scale_tril=tf.linalg.cholesky(cov))
  return tf.reduce_mean(-dist.log_prob(y_true))

I am trying something like below, but getting confused in the middle.
#X_train has 6 values in each row
#y_train has 5 values in each row
#y_pred should be either a distribution function or mu & cov for each row

opt = Adam(learning_rate=0.001)
inputs = Input(shape=(6,))
layer1 = Dense(24, activation='relu')(inputs)
layer2 = Dense(12, activation='relu')(layer1)
predictions = ???
model = Model(inputs=???, outputs=???)
model.compile(optimizer=opt, loss=loss_fn)
model.fit(X_train, y_train, epochs=100, batch_size=100)
y_pred=model.predict(X_test)

Note: instead of getting mu and cov separately, if its possible to get distribution function as output that would be helpful too.

Given that the covariance matrix has to be positive definite, the cholesky decomposition is a good way to solve this problem.
So the output of the network will be the mean vector mu and the upper triangular part of the cholesky matrix (denoted T here). The diagonal of this matrix must be positive elements (the diagonal of the covariance matrix are standard deviations):
p = y_train.shape[1] # dimension of the covariance matrix
inputs = Input(shape=(6,))
layer1 = Dense(24, activation='relu')(inputs)
layer2 = Dense(12, activation='relu')(layer1)
mu = Dense(p, activation = "linear")(layer1)
T1 = Dense(p, activation="exponential")(layer1)# diagonal of T
T2 = Dense((p*(p-1)/2), activation="linear")(layer1)
outputs = Concatenate()([mu, T1, T2])

Now let's define the loss function. Firstly, let's define the function that will extract the outputs of the network:
def mu_sigma(output):
    mu = output[0][0:p]
    T1 = output[0][p:2*p]
    T2 = output[0][2*p:]
    ones = tf.ones((p,p), dtype=tf.float32)
    mask_a = tf.linalg.band_part(ones, 0, -1)
    mask_b = tf.linalg.band_part(ones, 0, 0)
    mask = tf.subtract(mask_a, mask_b)
    zero = tf.constant(0, dtype=tf.float32)
    non_zero = tf.not_equal(mask, zero)
    indices = tf.where(non_zero)
    T2 = tf.sparse.SparseTensor(indices,T2,dense_shape=tf.cast((p,p),
         dtype=tf.int64))
    T2 = tf.sparse.to_dense(T2)
    T1 = tf.linalg.diag(T1)
    sigma = T1 + T2
    return mu, sigma

Now for the loss function:
from tensorflow_probability import distributions as tfd
def gnll_loss(y, pred):
    mu, sigma = mu_sigma(pred)
    gm = tfd.MultivariateNormalTriL(loc=mu, scale_tril=sigma)
    log_likelihood = gm.log_prob(y)
    return - tf.math.reduce_sum(log_likelihood)


How to find bias for perceptron algorithm? My question is very basic. I am starting with ML and am working on the perceptron algorithm. I successfully computed the weights for this input data:

X = [[0.8, 0.1], [0.7, 0.2], [0.9, 0.3], [0.3, 0.8], [0.1, 0.7], [0.1, 0.9]]
Y = [-1, -1, -1, 1, 1, 1]

Output_weights = [-0.7, 0.5]

But I didn't take bias into account, i.e. I assumed the discriminator line goes through the origin. Now, let's say I add another point into my training set:

new_X = [4,4]
new_Y = [-1]

How do I proceed if I want to compute the bias as well? In the first iteration for example, I'd set default weights to $[0,0]$, so I find the first point that is incorrectly classified.

Without bias, it is easy. I compute the dot product

0.8*0 + 0.1*0 = 0

should be $-1$, so it is incorrectly classified. I update the weights to:

[-0.8,-0.1]

However, taking bias into account, I get:

0.8*0 + 0.1*0 + bias

Now, how do I update the weights and the bias? What is the procedure?

I have searched several tutorials like this or this but didn't find an answer. A link to some resource would help, too.

Rather than viewing your data as

X = [[0.8, 0.1], [0.7, 0.2], [0.9, 0.3], [0.3, 0.8], [0.1, 0.7], [0.1, 0.9]]
Y = [-1, -1, -1, 1, 1, 1]

You could have treat the problem as

X = [[1, 0.8, 0.1], [1, 0.7, 0.2], [1, 0.9, 0.3], [1, 0.3, 0.8], [1, 0.1, 0.7], [1, 0.1, 0.9]]
Y = [-1, -1, -1, 1, 1, 1]

That is to append a $1$ in every single entry of $X$. The weight corresponding to $1$ would be the bias.

That is $Y=Ax+b$ can be written as $Y=[A, e]\begin{bmatrix}x \\b\end{bmatrix}$ where $e$ is the all one vector.


Algorithm for segmentation of sequence data I have a large sequence of vectors of length N. I need some unsupervised learning algorithm to divide these vectors into M segments.

For example:

K-means is not suitable, because it puts similar elements from different locations into a single cluster.

Update:

The real data looks like this:

Here, I see 3 clusters: [0..50], [50..200], [200..250]

Update 2:

I used modified k-means and got this acceptable result:

Borders of clusters: [0, 38, 195, 246]

Please see my comment above and this is my answer according to what I understood from your question:

As you correctly stated you do not need Clustering but Segmentation. Indeed you are looking for Change Points in your time series. The answer really depends on the complexity of your data. If the data is as simple as above example you can use the difference of vectors which overshoots at changing points and set a threshold detecting those points like bellow:

As you see for instance a threshold of 20 (i.e. $dx<-20$ and $dx>20$) will detect the points. Of course for real data you need to investigate more to find the thresholds.

Pre-processing

Please note that there is a trade-off between accurate location of the change point and the accurate number of segments i.e. if you use the original data you'll find the exact change points but the whole method is to sensitive to noise but if you smooth your signals first you may not find the exact changes but the noise effect will be much less as shown in figures bellow:

Conclusion

My suggestion is to smooth your signals first and go for a simple clustering mthod (e.g. using GMMs) to find an accurate estimation of the number of segments in signals. Given this information you can start finding changing points constrained by the number of segments you found from previous part.

I hope it all helped :)

Good Luck!

UPDATE

Luckily your data is pretty straightforward and clean. I strongly recommend dimensionality reduction algorithms (e.g. simple PCA). I guess it reveals the internal structure of your clusters. Once you apply PCA to the data you can use k-means much much easier and more accurate.

A Serious(!) Solution

According to your data I see the generative distribution of different segments are different which is a great chance for you to segment your time series. See this (original, archive, other source) which is probably the best and most state-of-the-art solution to your problem. The main idea behind this paper is that if different segments of a time series are generated by different underlying distributions you can find those distributions, set tham as ground truth for your clustering approach and find clusters.

For example assume a long video in which the first 10 minutes somebody is biking, in the second 10 mins he is running and in the third he is sitting. you can cluster these three different segments (activities) using this approach.


Use pretrained word vectors over custom trained word2vecs Currently i'm working on a sentiment analysis research project using LSTM networks.

As the input I convert sentences into set of vectors using word2vec.

And there are some well  pretrained word vectors like Google word2vec.

My problem is, is there are any advantages of using custom trained word2vecs(train using a dataset which related to our domain, such as user reviews of electronic items)  over pretrained ones.

Whats the best option

use a pretrained word2vec
Train our own word2vec using a dataset related to the domain

Can any one help me on this
Thanks

The entire philosophy of Distributed Word Representations makes use of the fact that a word is understood by the context it has . When we say context , we mean the words that come in the neighborhood of a particular word . Now context in natural language is a tricky thing . As an example . the words open and create are two words that are not so similar semantically . But think of the scenario of a Bank related corpus , you will frequently see statements like

                     How do I open a new account
                                  &
                    How do I create a new account

In the context of such a document , open and create become similar words. That is why context specific word vectors are needed.

Now , when it comes to which is the better option between the two

Use pre-trained vectors
Use custom vectors ,

it depends on how much data do you have for your custom use case . If you have enough data then it's always safe/better to go for a custom vectorization as it will be very specific to the context the corpus has . But in all other cases , you can use the pre trained embeddings. These generalize quite well on a variety of docs.


How to use a different model to deep neural network with reinforcement learning based on DQN? Is it possible to implement a reinforcement learning algorithm without using a deep neural network (DNN) as used in deep reinforcement learning e.g. Deep Q-Network (DQN)?

How can I replace the DNN in the DQN algorithm with another algorithm? Should it be supervised or unsupervised, and what is this called - is it  "un/supervised reinforcement learning" or "un/supervised DQN"?

In the following DQN pseudo-code, if I want to replace a DNN section with another un/supervised algorithm, is it possible? If so, how?

Well, if you remove the DNN, I would not call that a Deep Q-Network anymore.. but it is definitely possible to remove that and still consider the approach as Reinforcement Learning.

Actually, the function of the deep neural network is just to approximate the Q-value function. The DNN is just a function in the form $Q_\theta(s, a)$ and based on samples, you adjust $\theta$ to minimize the error.

In practice, you could use any function you want. Obviously, some functions will work better than others. There are simpler approaches based on linear regression, least squares, kernel-based approaches, etc.

With respect to unsupervised learning.. mm. I guess you could force it somehow, but keep in mind that it would not totally make sense given that RL already provides you feedback so it makes sense that you use it.

Some interested advantage of neural networks is that they can find common elements in $\phi$ and rapidly abstract away some noise that if is not relevant with respect to predicting the Q-value of certain state/action.

I noticed that you would like to use something like random forests, etc. There is nothing to stop you from doing that. In particular, random forests are not ideal because they don't deal well with non-stationary. Instead they require bit batches of data. However, this does not mean, it is impossible.

In DQN, after you sample from the environment, you obtain feedback from the environment in the form of rewards associated with state/action pairs. RL combines this information with previous predictions using TD (there are some variations like TD($\lambda$), etc.). What TD gives you is a better estimate of $Q(s,a)$ and your predictor (in this case, your random forest) should learn that when the input is $(s,a)$ the correct answer is $Q(s,a)$.

It is likely that you've seen the same inputs before (specially for initial states) but because TD improves the reference value, you would need to drop these previous values and replace them with the most recent ones. Random forest would need to get this whole batch again.

In general, I accept that while both cases are supervised learning, adjusting DQN to work with non-parametric methods such as random forests can be inconvenient.

If I had to guess based on the comments, I think your main question circles around the distinction between parametric and non-parametric supervised learning. In that case, it is fair to say that most non-parametric methods are not easy to combine with DQN.


Train/Test dataset and model I would like to ask you how to work on train and test dataset. I have unlabelled data. They are short text (max 100 characters) and I would need to understand their sentiment.
To do this, I am manually assigning labels (1,0,-1). However I have more than 2000 text and I would like to find a way to do it automatically, after considering a small labelled set.
What I was thinking is to split the dataset into train and test since the beginning and work with the training dataset to label data.
Unfortunately, I have not understood how to assign labels to the remaining texts, I.e. how to predict the sentiment of data in the test dataset.
Could you please tell me what would be next steps and, if you have anything that you think can be useful for a better understanding, suggesting an example to follow? Many thanks

You want to manually label some cases and then extend that "manual labeling" to the rest of the data.
This is a supervised learning excercise with prior manual labeling by you.
Let's suppose you have partitioned a random, suitably sized training data set. Now you need to model a classification algorithm via the classical modeling pipeline and use this model to predict the classifaction/label in the rest of the data.
So yes this is easily possible,however modeling a text classification model is non-trivial and you need to understand basic modeling.
Here are the basic steps, please read upon each step you do not know how to do:

Split your data into a training set (you will model on this data), a test set and a target set.

Manually label your training and test set.

Choose the kind of classification algorithm you want to use. You can use classical ML models but this involves heavily tokenizing and transforming your data to numerical components. You could also use more advanced deep learning techniques for text classification like BERT.

Create a suitable transformer to tidy and transform your data into the right format for your chosen algorithm.

Train the model on your manually labeled training data.

Evaluate and optimize your performance with the test data set.

Use the final model to predict labels in your target set.

Understand that the quality of the automatic labeling will only be as good as your manual labels.


Alternative methods for improved clustering separation? I have the following labeled cluster, which is what an ideal clustering algorithm would generate:

Now, I have applied a basic K-Means clustering algorithm to the data, and the outcome is as follows:

I recognize that this is a tough problem to properly cluster because some of the classes are so similar.

But I was wondering if there are any alternative algorithms that may help me improve the separability of the clusters, and improve how well my unsupervised clustering algorithm would work on new data?

Your data doesn't appear to be easily separable. In general, one could apply some kind of transformation that pulls apart the distributions for each class. Having labels available makes it possible, in principle, to learn such a transformation (as @Emre metnioned in the comments). But, there are a couple issues with your particular data set. 1) You don't appear to have many data points (unless you've only plotted a small subset). This would limit you to very simple transformations (otherwise you'd probably get severe overfitting). 2) The points are simply overlapping. A transformation can only work based on its inputs and, if the coordinates are indistinguishable, there's nothing that can be done. In the best case, you might be able to pull the lower left turquoise cluster and the yellow points further from the main mass, but the rest of the points are pretty much intermingled. Any transformation that could manage to separate them in the training data would be very complicated, and probably just reflect sample noise (i.e. it would probably be completely overfit, and not generalize to new data).

The ideal thing would be to find/measure additional (relevant) variables. In this case, the classes may become separable in the higher dimensional space. For example, imagine additing a third axis, where the red points become 'lifted' above the blue points.


How to make predictions based on correlations? I have correlation values for profit based on three different attributes - attribute1,attribute2,attribute3. Now I want to make predictions for the profit based on these 3 different correlation values.

I wanted suggestions for some efficient algorithms that can help me to make predictions  from these correlation values.

Note: I am new to this so I would appreciate some suggestions if there is some lack of understanding in me.

The correlation coefficient will tell you, how much predictive power the individual attributes posess with respect to profit in a linear sense.
Easier: The correlation coefficient tells you how line-like your data is.
( There might still be a very predictive arc like structure in your data which the correlation coefficient can't capture )

So you can either calculate a regression line for profit with the attribute having the highest correlation coefficient, or you can calculate a hyperplane for all three attributes to get the desired prediction. Regression lines and hyperplanes are usually calculated by least-squares fitting.

But beware to overestimate predictive power, because it is extrapolation what you do. New data might behave different than old data which makes extrapolation potentially less exact the further you are away from known data.


Algorithms to find average distance between data maxima I'm having trouble figuring out a way to analyze some simple data. When graphed, the data I have make a somewhat sinusoidal curve. What I want to do is to find the x-values of the maximum peaks of the sinusoidal curve. I then want to subtract each of these x-values from the last peak found and average these differences to obtain an average distance between peaks. Is there an easy way to do this with Excel, Mathematica, or MatLab (the programs I have available to me?).

Thanks in advance!!!

The first technique that comes to mind is cubic spline interpolation, which is good at approximating sinusoidal functions without suffering from Runge's phenomenon. Here's a visual of cubic splines applied to a similar problem:

Matlab has an implementation documented here.

Once you have a representation for the underlying function you should be able to find the local maxima easily and take the differences you are looking for.


Feature name extraction directly from dataset I would like to visualize my features using the code below. However, I am getting an error that my features are being recognized as "nan" instead of their actual names.

Instead of the below Feature = [and then writing the features that I want], I am assigning  feature = data [0,1:]. This is the first row of my data that has the features in it. I have many features that I don't want to write as string but would just like to extract from the data file directly. How can I do this and get the names of the features instead of 'nan'?

# Load the dataset
data = load_data('credit')

# Specify the features of interest
features = [
        'limit', 'sex', 'edu', 'married', 'age', 'apr_delay', 'may_delay',
        'jun_delay', 'jul_delay', 'aug_delay', 'sep_delay', 'apr_bill', 'may_bill',
        'jun_bill', 'jul_bill', 'aug_bill', 'sep_bill', 'apr_pay', 'may_pay', 'jun_pay',
        'jul_pay', 'aug_pay', 'sep_pay',
    ]

# Extract the instances and target
X = data[features]
y = data.default

from yellowbrick.features import Rank1D

# Instantiate the 1D visualizer with the Sharpiro ranking algorithm
visualizer = Rank1D(features=features, algorithm='shapiro')

visualizer.fit(X, y)                # Fit the data to the visualizer
visualizer.transform(X)             # Transform the data
visualizer.poof()                   # Draw/show/poof the data

If you are reading in a csv, try;

data = pd.read_csv('filepath', header=None)

Then you can do what you were doing to read the features from the first line


Random Forest in R: Error in eval(expr, envir, enclos): object not found I use the randomForest package in R but I'm getting the following error:

Error in eval(expr, envir, enclos) : object 'a' not found

The line that causes the problem is the following:

model = randomForest(class ~ a + b + c, data=train, ntree=100

I looked up the error and apparently it has to do with the fact that train does not contain a, but I manually checked train and it does contain a.

As a matter of fact, when I try to build a decision tree instead, it works just fine! (with the same formula and dataset)

model = rpart(class ~ a + b + c, data=train, method="class")

So I don't understand where this error is coming from. Any ideas?

EDIT: There seems to be a problem with the fact that I use bigrams for my text. If I just use unigrams, it works fine. If I use this code for example:

model = randomForest(class ~ `stack overflow`, data=train, ntree=100)

It returns the error named above: object 'stack overflow' cannot be found.

In general column names in R need to be formatted without spaces for R to recognize them properly. Importing files with spaces, R will drop in a period and remove the space

This limit is is true for variables as well, but it will not auto-format them for you.

The most accepted methods for naming are:

stackOverflow camel case

stack_overflow with the underscore and a combination of lower and upper as you see fit

stack.overflow with period (which many dislike because it is often easy to not see the period)

If you use these conventions you will avoid the name issue and be able to focus on the underlying models.

A caveat, which may or may not matter to you. If you are going to pick a naming style and you work with other languages and pass data around, consider picking a style which works in both or all of them. It is easier for you to have one way, and it makes moving files easier if the names are compatible.


We are using PostgreSQL to store big data and are concerned it may crash the on-board Neural Net. Any thoughts? We have figured out how to write Big Data to our PostgreSQL database. We would like to run this (normalized) data into the on-board Neural Network in Orange. Our lead programmer said he thought Orange may very well choke on this large amount of data if we feed it directly into the Orange Neural Net from Big PostgreSQL Data. Does any one here have any experience with this problem? Should we maybe write a widget that caches the data and feeds it into the neural network in discrete chunks? Has anyone here attempted to feed gigabytes of data into the Orange Neural Net? Is TensorFlow a better solution?

Yes, Orange does not handle very large data sets very well. In fact, if you try to load in something like the file size that you are talking about (gigabytes of data) it will crash. When ever you want to create a large scale neural network, you will need to use packages like Tensorflow and Keras. You can not do this in Orange.


Find suitable locations using Machine Learning Just for fun, I am currently trying to find suitable locations to deploy new stores. So what I did so far is to take the actual sites of current stores and to assign surrounding variables to it. These features include for example: point of interest density, population density, region popularity etc. In total I have 9000, 100 dimensional points. 1000 of these points contain stores already, the remaining 8000 do not.

In the next step I want to perform dim reduction using PCA. However, I am not sure how to proceed afterwards. Should I try to cluster the points? Or how can I „predict“ which of the points are suitable candidates for new stores? Maybe using some kind of skip gram model?

Hoping to get some advise:)

Cheers,
Tom

Are you sure PCA is the correct way to go?
It's an analytical problem and being able to interpret the results are very important.

How about the correlation between the number of stores and nearby features? Find out what makes a good location. What are the most important features? Run forward or backward selection as an example, or use another model/feature selection technique.

It's not a pure machine learning case you have here. It's a typical analytical data science problem.

If you still want to do classification, just train a model. You have POI features and some others. You know if there is a store or not :) I might not fully understand the problem here. You train on a 50% a store exist location, and 50% a store does not exist in this location dataset. Train a classifier, and classify other areas.

I'd still start to visualize and understand the data as I mentioned first. It's much underrated and the way to start solving most problems.

Hope that gave you some hints,

Cheers


How to measure retention statistics? I have a dataset with the ID, name, joining date, leaving date as features. I was asked to measure employee retention and health of it. What can I derive from these?

What are some latest trends and examples which I can find relating these?

Thanks.

I know this is a discussion, but given that I couldn't find this on google search, it would be helpful for someone.

ID and name are probably not predictive of retention.
With joining and leaving date, you can model the distribution of time at the organization.
Unless you have other features, you will not be create a useful model.


What is the difference between (objective / error / criterion / cost / loss) function in the context of neural networks? The title says it all: I have seen three terms for functions so far, that seem to be the same / similar:

error function
criterion function
cost function
objective function
loss function

I was working on classification problems

$$E(W) = \frac{1}{2} \sum_{x \in E}(t_x-o(x))^2$$

where $W$ are the weights, $E$ is the evaluation set, $t_x$ is the desired output (the class) of $x$ and $o(x)$ is the given output. This function
seems to be commonly called "error function".

But while reading about this topic, I've also seen the terms "criterion function" and "objective function". Do they all mean  the same for neural nets?

Geoffrey Hinton called cross-entropy for softmax-neurons and $E(W) = \frac{1}{2} \sum_{x \in E}(t_x-o(x))^2$ a cost function.

The error function is the function representing the difference between the values computed by your model and the real values. In the optimization field often they speak about two phases: a training phase in which the model is set, and a test phase in which the model tests its behaviour against the real values of output. In the training phase the error is necessary to improve the model, while in the test phase the error is useful to check if the model works properly.

The objective function is the function you want to maximize or minimize. When they call it "cost function" (again, it's the objective function) it's because they want to only minimize it. I see the cost function and the objective function as the same thing seen from slightly different perspectives.

The "criterion" is usually the rule for stopping the algorithm you're using. Suppose you want that your model find the minimum of an objective function, in real experiences it is often hard to find the exact minimum and the algorithm could continuing to work for a very long time. In that case you could accept to stop it "near" to the optimum with a particular stopping criterion.

I hope I gave you a correct idea of these topics.


Stacked Model performance? I am currently working with a dataset that seems very easily separable and I have an accuracy of 99% for SVM (NN-98%, RF-98%, DT-96-97% and I have checked for leakage & overfitting). As part of my project, I am also learning how to implement a hybrid model but its accuracy is also 99% (1 record misclassified). It seems to be misclassifying the same record as SVM and no other algorithm seems to get that record correct. Probability wise- RF and Logistic Regression give a probability of 0.5-0.7 but all other models give 0.9-1 probability for the wrong prediction. I was wondering what I should do now. Are there certain techniques I can use to reduce the probability to below 0.5 in the stacked model so that it isn't classified the way it currently is? I am using the StackingCVClassifier for implementation and this is a binary classification project.
I am currently using SVM, LR, RF as base models and NB as the meta-model.
My other question is if there is even value in making a hybrid model if the accuracy is the same as SVM's?
Thank you!

Are there certain techniques I can use to reduce the probability to below 0.5 in the stacked model so that it isn't classified the way it currently is?

It's generally not a good idea to try to bias the classifier in order to deal better with some specific instances, because it's likely to make it weaker in with some (possibly many) other instances. The way to do that would be oversample in the training set the instances which will help predict the target instance correctly, but that's a terrible idea, don't do it! :)
In general it's perfectly normal to have some errors, the data can contain noise or even sometimes annotation errors.

My other question is if there is even value in making a hybrid model if the accuracy is the same as SVM's?

If the data is really easy to classify, it's totally possible that stacking learners doesn't improve performance: if the performance of a single model reaches the maximum obtainable on this data, clearly there's nothing to improve.
However it's also possible that the benefit of stacking is not visible in this particular test: in this hypothesis the meta-model is indeed better in general that a single learner, but the test set just doesn't contain any instance on which this can be seen.
A way to check this is to reduce the size of the training set: by making things harder for the classifiers, it's possible that the weaknesses of the individual learners will show up.


Credit card check digit validation Apart from Luhn algorithm, can we use a machine algorithm to check if last digit of credit card number is valid? I see this a multi-class classification problem but unable to find the right algorithm to solve it.

Am basically looking out for an approach to find check digit for any sequence, where check digit generation algorithm is not known - Credit card is just an example.

There are multiple cases where the number issuing authority does not disclose the check digit algorithm. For e.g. GSTIN in India - cleartax.in/s/know-your-gstin OR PAN in India - en.wikipedia.org/wiki/Permanent_account_number If we can try different ML algorithms for valid set of such sequences, we might be able to create a verification model for such cases

Without knowing what algorithm was used to generate the check digit, it's not possible to say whether machine learning will be effective at reverse-engineering that data.

As an easy case, given sufficient data, it's likely that a recurrent neural network could learn the Luhn algorithm.  It would likely be a particularly inefficient and suboptimal way to do it, and you might need an awful lot of training data, but I expect it'd probably work.

As a hard case, if the check digit is generated using a cryptographically secure hash function and a secret key, then no machine learning algorithm is ever going to reverse-engineer that.

So it's simply not possible to say, in the abstract.  The answer depends on the check digit algorithm that is being used.

In general, machine learning isn't a silver bullet.  It isn't magic fairy dust that can be used to solve all problems.  My sense is that machine learning is probably not going to be particularly well-suited to this kind of situation.  It might be more effective to start by trying some common check digit algorithms (e.g., use linear algebra to check whether it is a linear checksum, and so on).


Training data set for food image recognition There are many excellent posts and answers referencing data sources. However I can't seem to find many for food image/photograph recognition. I would have thought, with so many people taking and uploading pictures of their food, that this should be readily available. Can anyone help? Food101 and UEC are good but limited. Thank you.

You could use the Food 101 dataset or
UEC Food 256, both contains real-world food images.


Detecting outlier with combining two vectors I want to combine the following vectors in a way that just the red point (number 7) becomes inconsistent with other points( become an outlier and become distant from other points) and other points become consistent with each other. Please note that I have tested mahalanobis distance and Kullback-Leibler divergence between two vectors but they were not so good and detects.

a=[1.3269   1.3354  1.3318  1.3282  1.34666 1.3460  1.36084 1.3526  1.3539  1.3510  1.3480  1.3479  1.34893]

b=[0.0352,0.0992,0.1570,0.1431,0.1634,0.1629,0.1046,0.1655,0.1635,0.1642,0.1658,0.1666,0.15735]

Thanks in advance.

Autoencoder Solution

You could try an autoencoder. The autoencoder would take an input vector and it would try to recreate it as an output. So you take your input, and measure the distance between the input and the predicted output variable, using your metric of choice (euclidean should work but try various). Larger distances can be thought of as more abnormal. So you can stack rank your observations from weirdest to most normal.

Make sure that you are only training the autoencoder on normal data though. This would of course assume that you have more than the 13 samples that you are looking at. If not this probably isn't going to work very well, just because of the small sample.

KDE Solution

The idea is to use Kernel Density Estimation to generate a non-parametric joint density of your data set. You then find what the probability of finding a value that extreme would be. Here is some code using python's sklearn package:

from sklearn.neighbors.kde import KernelDensity
import numpy as np
X=np.matrix([[1.3269, 1.3354, 1.3318, 1.3282, 1.34666, 1.3460, 1.36084, 1.3526, 1.3539, 1.3510, 1.3480, 1.3479, 1.34893],[0.0352, 0.0992, 0.1570, 0.1431, 0.1634, 0.1629, 0.1046, 0.1655, 0.1635, 0.1642, 0.1658, 0.1666, 0.15735]])
kde = KernelDensity(kernel='gaussian', bandwidth=.45).fit(X.T)
score=kde.score_samples(X.T)
prob=np.exp(score)
print(prob/prob[6])

This code shows that the observations in the lowest probability density areas are observations 1,2 and 7. Of course this would work better with a larger sample, and you need to fuss with the bandwidth to calibrate it, but that this should do it.