Very slow convergence with CNN I am new to deep learning. I am working on training an SSD model on a set of small objects. I am using Adam gradient descent for optimization and a large input (800x800), but I seem to only get an improvement of 0.010 after every 20 or so epochs(350 steps).

What can I do or look for to speed up convergence on this model?

Implement the below mentioned techniques and check

Add Batch Normalization
Increase the learning rate
Standard/Normalize the inputs if you have not done it already


What is Pruning & Truncation in Decision Trees? Pruning & Truncation
As per my understanding
Truncation: Stop the tree while it is still growing so that it may not end up with leaves containing very low data points. One way to do this is to set a minimum number of training inputs to use on each leaf.
Pruning is a technique in machine learning and search algorithms that reduces the size of decision trees by removing sections of the tree that provide little power to classify instances.

Can somebody explain the in-detailed implementation of these techniques in GBDT frameworks like XGBoost/LightGBM.

Which parameters are used in implementing these techniques?

Your understanding is correct. xgboost has nice explanation in the docs.

Reading the original papers is always great idea. Here's one for LGBM and here's one for XGBoost.

As for dessert, here's catboost paper.

Boosting is by far one of the most important concepts in hard machine learning. It's good to know it by heart.


Train test split ERROR So I was applying the random forest classifier model to a problem,however it showed this error ,even though the columns in X and Y of my dataset are equal. How can I resolve this?

ValueError                                Traceback (most recent call last)
<ipython-input-123-8b3f6408c588> in <module>
      1 from sklearn.metrics import confusion_matrix,accuracy_score
----> 2 cm = confusion_matrix(y_pred4,Y_test)

~\Anaconda3\lib\site-packages\sklearn\metrics\classification.py in confusion_matrix(y_true, y_pred, labels, sample_weight)
    251
    252     """
--> 253     y_type, y_true, y_pred = _check_targets(y_true, y_pred)
    254     if y_type not in ("binary", "multiclass"):
    255         raise ValueError("%s is not supported" % y_type)

~\Anaconda3\lib\site-packages\sklearn\metrics\classification.py in _check_targets(y_true, y_pred)
     69     y_pred : array or indicator matrix
     70     """
---> 71     check_consistent_length(y_true, y_pred)
     72     type_true = type_of_target(y_true)
     73     type_pred = type_of_target(y_pred)

~\Anaconda3\lib\site-packages\sklearn\utils\validation.py in check_consistent_length(*arrays)
    203     if len(uniques) > 1:
    204         raise ValueError("Found input variables with inconsistent numbers of"
--> 205                          " samples: %r" % [int(l) for l in lengths])
    206
    207

ValueError: Found input variables with inconsistent numbers of samples: [7500, 2500]

In your confusion matrix kernel , you passed two arguments. Y_pred and y_true. They are supposed to have same dimensionality. But they are different. That's why the error occured.


Learning rate in the Perceptron Proof and Convergence Every perceptron convergence proof i've looked at implicitly uses a learning rate = 1.

However, the book I'm using ("Machine learning with Python") suggests to use a small learning rate for convergence reason, without giving a proof.

Can someone explain how the learning rate influences the perceptron convergence and what value of learning rate should be used in practice?

(My answer is with regard to the well known variant of the single-layered perceptron, very similar to the first version described in wikipedia, except that for convenience, here the classes are $1$ and $-1$.)

Let's denote:

$d$ is the dimension of a feature vector, including the dummy component for the bias (which is the constant $1$).
$x^r\in\mathbb R^d$ and $y^r\in\{-1,1\}$ are the feature vector (including the dummy component) and class of the $r$ example in the training set, respectively.
$\eta _1,\eta _2>0$ are training steps, and let there be two perceptrons, each trained with one of these training steps, while the iteration over the examples in the training of both is in the same order.
$w_0\in\mathbb R^d$ is the initial weights vector (including a bias) in each training.
for $i\in\{1,2\}$: let $w_k^i\in\mathbb R^d$ be the weights vector after $k$ mistakes by the perceptron trained with training step $\eta _i$.
for $i\in\{1,2\}$: with regard to the $k$-th mistake by the perceptron trained with training step $\eta _i$, let $j_k^i$ be the number of the example that was misclassified.

If $w_0=\bar 0$, then we can prove by induction that for every mistake number $k$, it holds that $j_k^1=j_k^2$ and also $w_k^1=\frac{\eta_1}{\eta_2}w_k^2$:

base ($k=1$):
$w_0^1=w_0=w_0^2$, so both perceptrons would predict the same class for each example until and including the first mistake, so $j_1^1=j_1^2$.
Now: $$w_1^2=w_0+\eta_2y^{j_1^2}x^{j_1^2}=\eta_2y^{j_1^2}x^{j_1^2}=\eta_2y^{j_1^1}x^{j_1^1}$$ and thus:$$w_1^1=w_0+\eta_1y^{j_1^1}x^{j_1^1}=\eta_1y^{j_1^1}x^{j_1^1}=\frac{\eta_1}{\eta_2}\eta_2y^{j_1^1}x^{j_1^1}=\frac{\eta_1}{\eta_2}w_1^2$$
step (let $k$ be a mistake number such that the claim holds for $k$, and for every natural smaller than $k$):
After the $k$-th mistake, perceptron $1$ would predict that an example $r$ is positive iff $(w_k^1)^Tx^r>0$. According to the induction hypothesis, this is true iff $(\frac{\eta_1}{\eta_2}w_k^2)^Tx^r>0$, but we can divide both sides by $\frac{\eta_1}{\eta_2}$, as it is a strictly positive scalar, and thus we get: $$(w_k^1)^Tx^r>0\leftrightarrow (w_k^2)^Tx^r>0$$ i.e. after the $k$-th mistake, both perceptrons would predict the same class for each example. According to the induction hypothesis, $j_k^1=j_k^2$, and so both perceptrons would iterate over the same examples in the same order after the $k$-th mistake. Therefore, $j_{k+1}^1=j_{k+1}^2$.
Now: $$w_{k+1}^2=w_k^2+\eta_2y^{j_{k+1}^2}x^{j_{k+1}^2}=w_k^2+\eta_2y^{j_{k+1}^1}x^{j_{k+1}^1}$$ and thus (according to the induction hypothesis):$$w_{k+1}^1=w_k^1+\eta_1y^{j_{k+1}^1}x^{j_{k+1}^1}=\frac{\eta_1}{\eta_2}w_k^2+\eta_1y^{j_{k+1}^1}x^{j_{k+1}^1}=\frac{\eta_1}{\eta_2}(w_k^2+\eta_2y^{j_{k+1}^1}x^{j_{k+1}^1})=\frac{\eta_1}{\eta_2}w_{k+1}^2$$

We showed that the perceptrons do exactly the same mistakes, so it must be that the amount of mistakes until convergence is the same in both. (You could also deduce from this proof that the hyperplanes defined by $w_k^1$ and $w_k^2$ are equal, for any mistake number $k$.)
Thus, the learning rate doesn't matter in case $w_0=\bar 0$.

In case $w_0\not=\bar 0$, you could prove (in a very similar manner to the proof above) that in case $\frac{w_0^1}{\eta_1}=\frac{w_0^2}{\eta_2}$, both perceptrons would do exactly the same mistakes (assuming that $\eta _1,\eta _2>0$, and the iteration over the examples in the training of both is in the same order).
Thus, for any $w_0^1\in\mathbb R^d$ and $\eta_1>0$, you could instead use $w_0^2=\frac{w_0^1}{\eta_1}$ and $\eta_2=1$, and the learning would be the same.
In other words, even in case $w_0\not=\bar 0$, the learning rate doesn't matter, except for the fact that it determines where in $\mathbb R^d$ the perceptron starts looking for an appropriate $w$.

Finally, I wrote a perceptron for $d=3$ with an animation that shows the hyperplane defined by the current $w$.
I think that visualizing the way it learns from different examples and with different parameters might be illuminating.

import numpy as np
import matplotlib.pyplot as plt
from matplotlib.animation import FuncAnimation
import itertools
import time

EXAMPLES1 = np.array([[9, 0], [8, 0]], dtype='float')
EXAMPLES2 = np.array([[5, 0], [6, 0]], dtype='float') # closer to the origin
EXAMPLES_CLASSES1_2 = np.array([1, -1])

EXAMPLES3 = np.array([[20, 26], [16, -11], [21, 2], [18, 25],
                    [26, -19], [11, 6], [20, -10], [17, 17]], dtype='float')
EXAMPLES4 = EXAMPLES3 - 11 * np.ones(EXAMPLES3.shape) # closer to the origin
EXAMPLES5 = np.array([[20, 26], [16, 27], [21, 24], [15, 25],
                    [22, 23], [14, 27], [20, 20], [13, 24]], dtype='float')
EXAMPLES_CLASSES3_4_5 = np.array([1, -1, 1, -1, 1, -1, 1, -1])

W0_0 = np.zeros(3)
W0_1 = np.array([1., 0, -2])
LEARNING_RATE_1 = 0.00156
W0_2 = np.array([1., 0, -2]) / 0.00156
LEARNING_RATE_2 = 1

EXAMPLES = EXAMPLES1
EXAMPLES_CLASSES = EXAMPLES_CLASSES1_2
LEARNING_RATE = LEARNING_RATE_2
W0 = W0_2
INTERVAL = 3e2
INTERVAL = 1
INTERVAL = 3e1

N = len(EXAMPLES)
assert(EXAMPLES.shape[1] == 2)
assert(N == len(EXAMPLES_CLASSES))
assert(LEARNING_RATE > 0)
assert(len(W0) == 3)

fig, ax = plt.subplots()
XS = EXAMPLES[:, 0]
YS = EXAMPLES[:, 1]
max_abs_x = max(max(XS), abs(min(XS)), 1)
max_abs_y = max(max(YS), abs(min(YS)), 1)
bottom_limit = -2 * max_abs_y
top_limit = 2 * max_abs_y
left_limit = -2 * max_abs_x
right_limit = 2 * max_abs_x
plt.xlim(left_limit, right_limit)
plt.ylim(bottom_limit, top_limit)
x_range_size = right_limit - left_limit
y_range_size = top_limit - bottom_limit

# plot the examples
for x, y in zip(EXAMPLES, EXAMPLES_CLASSES):
    if y == 1:
        plt.plot(x[0], x[1], marker='o', markersize=4, color="green")
    else:
        plt.plot(x[0], x[1], marker='o', markersize=4, color="red")

# add a dummy column of ones for the bias.
EXAMPLES = np.hstack((EXAMPLES, np.ones((N, 1))))

hyperplane, = plt.plot([], [], color='blue')
hyperplane.set_visible(False)
k_text = plt.text(ax.get_xlim()[0] + x_range_size / 70,
                  ax.get_ylim()[1] - y_range_size / 25, 'mistakes: 0')
next_example_idx = 0
k = 0
w = W0

def get_hyperplane(w):
    w_x, w_y, bias = w
    if w_x == w_y == 0:
        print(f'No hyperplane is defined by w')
        return None
    if w_y == 0:
        x1 = x2 = -bias / w_x
        y1 = bottom_limit
        y2 = top_limit
        if not left_limit <= x1 <= right_limit:
            print(f'The hyperplane is out of range of our graph')
            return None
        else:
            print(f'The hyperplane defined by w is `x = {x1}`')
    else:
        # find the hyperplane representation as y = m * x + b.
        m = -w_x / w_y # m = -1 / (w_y / w_x)
        b = -bias / w_y
        # b = -bias / np.linalg.norm(np.array([w_x, w_y]))
        print(f'The hyperplane defined by w is `y = {m} * x + {b}`')

        if m == 0:
            y1 = y2 = m * left_limit + b
            x1 = left_limit
            x2 = bottom_limit
        else:
            potential_points_on_limits = {
                (left_limit, m * left_limit + b),
                (right_limit, m * right_limit + b),
                ((bottom_limit - b) / m, bottom_limit),
                ((top_limit - b) / m, top_limit)}
            for x, y in set(potential_points_on_limits):
                if not left_limit <= x <= right_limit:
                    potential_points_on_limits.remove((x, y))
                elif not bottom_limit <= y <= top_limit:
                    potential_points_on_limits.remove((x, y))
            if len(potential_points_on_limits) < 2:
                print(f'The hyperplane is out of range of our graph')
                return None
            x1, y1 = potential_points_on_limits.pop()
            x2, y2 = potential_points_on_limits.pop()

    return (x1, x2), (y1, y2)

def update(frame):
    global k, w, next_example_idx

    for j in itertools.chain(range(next_example_idx, N),
                             range(next_example_idx)):
        x = EXAMPLES[j]
        y = EXAMPLES_CLASSES[j]
        perdicted_y = 1 if np.dot(w, x) > 0 else -1
        if perdicted_y != y:
            print(f'\nMisclassification of example {x}')
            w += LEARNING_RATE * y * x
            k += 1
            k_text.set_text(f'mistakes: {k}')
            print(f'This is mistake number {k}')
            print(f'Update w to {w}')

            hyperplane_xs_and_ys = get_hyperplane(w)
            if hyperplane_xs_and_ys:
                hyperplane.set_data(hyperplane_xs_and_ys)
                hyperplane.set_visible(True)
            else:
                hyperplane.set_visible(False)

            next_example_idx = j + 1
            return hyperplane, k_text

    # The perceptron successfully predicted all of the examples.
    time.sleep(3)
    exit()

ani = FuncAnimation(fig, update, blit=True, repeat=False, interval=INTERVAL)
plt.show()


PCA for complex-valued data I'm quite shocked for encountering this error on PCA from sklearn

  ValueError: Complex data not supported

After trying to fit complex-valued data. Is this just unimplemented thing? Should I just go ahead and do it 'manually' with SVD or is their a catch for complex-values?

Apparently this functionality is left out intentionally, see here. I'm afraid you have to use SVD, but that should be fairly straightforward:

def pca(X):
    mean = X.mean(axis=0)
    center = X - mean
    _, stds, pcs = np.linalg.svd(center/np.sqrt(X.shape[0]))

    return stds**2, pcs


Missing Values In New Data (Before someone marks this as duplicate - I'm not asking about training data, I'm asking about new data which has come in and needs to be classified)

Suppose I've got a dataset which has 5 predictors - v1 - v5.
Since my dataset contains NA's, I've been imputing these values during training, which is fine.

However, if I get a new item to classify, it may contain 1 or more NA's. My initial thought was to create a model for each predictor variable; so for example, create a model where v2 - v5 predict v1, in the case that v1 is missing.

This would result in 5 additional models. One problem is that if v1 and v2 are missing, I'd need to guess both values somehow. I'm not sure how to do this. It makes me think that this isn't the best approach.

I'm a bit stuck, and any insight into this would be much appreciated :)

No expert here, but predicting missing value using existing variables seems suspicious to me. Does it really work?

Say $v_5$ is missing and you want to impute its value based the other variables available, using some arbitrary function $g$:
$$v_5 = g(v_1, v_2, v_3, v_4)$$
and the target variable T is:
$$T = f(v_1, v_2, v_3, v_4, v_5)$$
which is essentially
$$T = f(v_1, v_2, v_3, v_4, g(v_1, v_2, v_3, v_4))$$
or
$$T = h(v_1, v_2, v_3, v_4)$$
So essentially you are estimating $T$ using available information regardless of the missing value. Why bother to impute $v_5$ using arbitrary function?

I tend to think that imputation only makes sense only if you have a good reason for the value imputed -- often they come from domain knowledge, which means that you need to impose an external restriction on the functional and parametric form of $g$ based on your prior.


Dealing with diverse text data I'm currently working with a dataset with a wide range of document lengths -- anywhere from a single word to a full page of text.  In addition, the grammatical structure and use of punctuation varies wildly from document to document.  The goal is to classify those documents into one of about 10-15 categories.  I'm currently using ridge regression and logistic regression for the task, and CV for the alpha values of ridge.  The feature vectors are tf-idf ngrams.

Recently I've noticed that longer documents are much less likely to be categorized. Why might this be the case, and how can one "normalize" for this kind of variation?  As a more general question, how does one typically deal with diverse data sets?  Should documents be grouped based off of metrics like document length, use of punctuation, grammatical rigor, etc. and then fed through different classifiers?

I'm not sure how you are applying a regression framework for document classification. The way I'd approach the problem is to apply a standard discriminative classification approach such as SVM.

In a discriminative classification approach the notion of similarity or inverse distance between data points (documents in this case) is pivotal. Fortunately for documents, there is a standard way of defining pairwise similarity. This is the standard cosine similarity measure which makes use of document length normalization to take different document lengths into account.

Thus, practically speaking, in cosine similarity you would work with relative term weights normalized by document lengths and hence document length diversity should not be a major issue in the similarity computation.

One also has to be careful when applying idf in term weights. If the number of documents is not significantly large the idf measure may be statistically imprecise thus adding noise to the term weights. It's also a standard practice to ignore stopwords and punctuations.


What kind of research can be done with genomic data? It is well known that science has given us large amounts of free accessible data, such as https://www.1000genomes.org and https://www.ncbi.nlm.nih.gov/genbank. How can we play around with the data and apply data science/machine learning to it? What could be some ideas?

My own ideas:

Biological data visualisation
Gene prediction using hidden-markov-model

Any more?

Determine the function of genes and the elements that regulate genes
throughout the genome.
Find variations in the DNA sequence among people and determine their
significance. The most common type of genetic variation is known as a
single nucleotide polymorphism or SNP (pronounced “snip”).  These
small differences may help predict a person’s risk of particular
diseases and response to certain medications.
Discover the 3-dimensional structures of proteins and identify their
functions.
Explore how DNA and proteins interact with one another and with the
environment to create complex living systems.
Develop and apply genome-based strategies for the early detection,
diagnosis, and treatment of disease.
Sequence the genomes of other organisms, such as the rat, cow, and
chimpanzee, in order to compare similar genes between species.
Develop new technologies to study genes and DNA on a large scale and
store genomic data efficiently.
Continue to explore the ethical, legal, and social issues raised by
genomic research.

Source: https://ghr.nlm.nih.gov/handbook/genomicresearch?show=all


what information can we obtain from t-SNE? I see that t-SNE can help us reduce dimensions and visualize the data. But what information are we gaining from this visualization? As we know that the new axis don't have a meaning in our context.
Moreover, if we have a class labeled data, then what information can we gain from the visualization? We already know that there are some 'n' classes and that we have to classify new examples in one of these classes.
Or am I wrong to understand t-SNE?

The aim of dimensionality reduction is to preserve as much of the significant structure of the high-dimensional data as possible in the low-dimensional map.

Dimensionality reduction yields a more compact, more easily
interpretable representation of the target concept, focusing the
user’s attention on the most relevant variables when you don't know
anything about data.

Having said that Dimension Reduction will not help if you have a less features  and only some are highly correlated as you can just use that correlated features for model so it all depends on data.
Unlike PCA, t-sne uses the local relationships between points to create a low-dimensional mapping. This allows it to capture non-linear structure.
T-sne also solves the Crowding Problem making the optimization "spread-out" the medium distance points to prevent crowding.
To understand more about t-sne plot refer here


Deep learning model performance for different datasets? I have a question that my single deep neural network model gives above 90% accuracy for one data set and the same model gives an accuracy between 70-80% for the other data set. I want to know why this variation in accuracy happens in spite of the fact that i am using the same deep neural network model.

I can think of one reason. Because you're using different data sets, the properties of your data set are likely to be different. So obviously what the model will learn will be different. And maybe the architecture of your network suits one data set more than the other. But also if you try training the model on your other data set(with less accuracy) for more epochs, it's possible that the accuracy will improve. Your model could be just be slow to understand this data set.

It's like when you see a human you can easily recognize a face. But when you see a software code, it's not always easy to find the bug. But if you're trained well enough you'll eventually be faster and better in recognizing the bugs.

I hope it makes sense to you.


Softmax vs Sigmoid in RBM/Auto Encoder final layer I'm creating a deep network with 3 hidden layers for classification of the MNIST dataset. The network architecture is something like:

784 input nodes -> 150 hidden nodes (layer 1) -> 125 hidden nodes (layer 2) -> 100 hidden nodes (layer 3) -> 10 output nodes (1 for each possible digit)

For the output layer, I need to decide on an activation function. I've previously used the sigmoid activation function, but it seems many people are using softmax in this particular implementation. I'd like to know which is best suited for the task and why.

Many thanks

This choice mainly depends on what your output represents. Given a vector $\mathbf{x}$, the sigmoid function is given by
$$ \sigma(x_i) = \frac{\exp(x_i)}{1 + \exp(x_i)} $$
while the softmax is given by
$$ \mathrm{softmax}(x_i) = \frac{\exp(x_i)}{\sum_{j=0}^N \exp(x_j)}$$
A key difference is that the output of the sigmoid function applied to $x_i$ only depends on $x_i$. The output of the softmax function depends on all elements of the vector $\mathbf{x}$.

The sigmoid will squash each $x_i$ into the range $(0, 1)$, which enables you to interpret $\sigma(x_i)$ as the probability of $x_i$.
But, if you have multiple classes, e.g. 0-9 in MNIST, each probability is independent, that means you could have probability $p=0.9$ for the digit 1 and $p=0.5$ for the digit 3. This is undesirable if you want to distinguish between multiple classes. However, if multiple classes can appear at the same time, then sigmoid is well suited.

Now, the softmax is basically a sigmoid function which is normalized such that $\sum_{j=0}^N \mathrm{softmax}(x_j) = 1$. This means that the output of a softmax layer is a valid probability mass function, i.e. the output of your neural network is the probability of the input belonging to a certain class.

So, if you want to classify between e.g. the 10 digits as in MNIST, then softmax is the way to go. If the classes are independent, i.e. it is possible that an image shows a cat and a dog at the same time, then sigmoid is the way to go.


scikit-learn: High / low value for C in SVM I'm playing with scikit-learn. Looking into the user guide and documentation they say:

  A low C makes the decision surface smooth, while a high C aims at classifying all training examples correctly.

The default value is 1.0, which is used by most of the examples. I also found the value C=100.

What is meant by "low" and high for C?

The loss function in SVM consists of both Classification error and Margin error.

Now C is constant that attaches itself to the classification error as given below:

SVM_Error = C*Classfication Error + Margin Error

So, the high value of C means that classification error will be more than the marginal error and therefore model will focus on classifying all the data points accurately. Conversely, if the low value of C means that marginal error will be more than the classification error and therefore model will focus creating the larger margin for the hyperplane


Checking TF-IDF Results I am working with TF-IDF and cosine similarity to do document comparisons and given a document, which document in the data is the most similar. However, sometimes it returns a high similarity between two documents which don't seem very similar when reading through the document pair. Are there conventional ways of evaluating which words contribute to the high similarity score between two documents using TF-IDF and cosine similarity?

Yes, Cosine TF-IDF is quite transparent so it's usually reasonably easy to visualize the words which contribute the most to a score. Cosine is defined as the dot product divided by the product of the norms, so you can isolate the terms:

dotproduct(d_1,d_2) = tfidf(w1,d1) * tfidf(w1,d2) + tfidf(w2,d1) * tfidf(w2,d2) + ... + tfidf(wN,dN)

Ranking the words w_i by descending order of the term tfidf(w_i,d1) * tfidf(w_i,d2) gives the top most contributing words for the similarity score.

Mind that if the documents have big differences in size, this will have an effect since cosine is normalized with their norms.


Chi-Squared test: ok for selecting significant features? I would have a question on the contingency table and its results.
I was performing this analysis on names starting with symbols as a possible feature, getting the following values:
Label          0.0  1.0
with_symb      1584 241
without_symb     16 14

getting a p-value which lets met conclude that variables are associated (since it is less than 0.05).
My question is if this result might be a good result based on the chi-squared test, so if I can include in the model.
I am selecting individually features to enter the model based on the chi-squared. Maybe there is another way to select the most appropriate and significant features for the model.
Any suggestions on this would be great.

I will raise several issues that could arise if you are selecting features based on chi-2 tests

Repeated use of chi-2 test can lead to spurious results unless you correct for the number of times you run it

You can include features that are correlated with each other, i.a. A is correlated with B, and both are correlated with label. Not sure, but I think, this can lead to results where model performs worse with more features.

I would try starting with all the features, remove the ones linearly correlated.  But this is just a suggestion.
Also, mutual information can be used to estimate how well any given feature describes the label.


Analyze additions and subtractions to/from a data set I have a data set in the following form:
Product    |    Date
123        |    2019-01-01
456        |    2019-01-01
123        |    2019-01-02
123        |    2019-01-03
456        |    2019-01-03
123        |    2019-01-04
456        |    2019-01-04
789        |    2019-01-04

This is just a simplified version. The full set has ~300 products and four months of data. I want to understand how the product set changed over time. It's obviously easy to calculate the count per day and see that I lost one product on Jan 2nd and gained one on Jan4th, but then I don't know what product it was.
Is there a more systematic way of going about this? Ideally, the output would show me a list of days and what products dropped out / were added that day. I thought about min(date), max(date) by-product before, but products can drop and be added repeatedly and I wouldn't capture this back and forth this way.
Available environments are Python, SQL, and Excel.

This response is based on your domain. For,e.g. sales, a product may not be sold every day and so there would be no record of it.

According to your dataset in which each day you would expect to see an occurrence of the product in your data set, you could consider the following approaches.

Mathematically:

A = the set of all possibilities (i.e. product occurrence for each date - you could generate this and use )

B = sample data set provided would be provided

C = A - B
  = days that a product was missing.

DPART1 = You could then continue by retrieving the min(date) and max(date) for each product in the dataset to represent the introduction of the new product and possible cease of an existing product.

D= You could then filter the dataset (C) to remove dates less than the min(date) and greater than the maxdate).

In terms of sql:
A - Cartesion/Cross product of all your products and dates
B = Your current sample data set
C = SELECT * FROM A MINUS SELECT * FROM B
DPART1 = SELECT PRODUCT, MIN(date) as INTRODUCED_DATE, MAX(date) as CEASED_DATE from YourSampleDataSET GROUP BY PRODUCT
D = SELECT C.PRODUCT, C.Date FROM C LEFT JOIN DPART1 ON C.PRODUCT = DPART1.PRODUCT
    WHERE DPART1.PRODUCT IS NULL OR (
       C.Date BETWEEN DPART1.INTRODUCED_DATE AND DPART1.CEASED_DATE
    )

Nb. "DPART1.PRODUCT IS NULL" ensures that you do not filter products that may not be in your sample subset.

Operational considerations, I would recommend indexes in your database to assist with the queries, partitioning where possibly.


Balanced Accuracy vs. F1 Score I've read plenty of online posts with clear explanations about the difference between accuracy and F1 score in a binary classification context. However, when I came across the concept of balanced accuracy, explained e.g. in the following image (source) or in this scikit-learn page, I was a bit puzzled as I was trying to compare it with F1 score.

I know that it is probably impossible to establish which is better between balanced accuracy and F1 score as it could be situation-dependent, but I would like to understand some pros/cons of the two performance metrics, as well as some applications in which one could be more suitable and useful than the other (especially in an imbalanced binary classification context).

One major difference is that the F1-score does not care at all about how many negative examples you classified or how many negative examples are in the dataset at all; instead, the balanced accuracy metric gives half its weight to how many positives you labeled correctly and how many negatives you labeled correctly.

When working on problems with heavily imbalanced datasets AND you care more about detecting positives than detecting negatives (outlier detection / anomaly detection) then you would prefer the F1-score more.

Let's say for example you have a validation set than contains 1000 negative samples and 10 positive samples. If a model predicts there are 15 positive examples (5 truly positive and 10 it incorrectly labeled) and predicts the rest as negative, thus

TP=5; FP=10; TN=990; FN=5

Then its F1-score and balanced accuracy will be

$Precision = \frac{5}{15}=0.33...$

$Recall = \frac{5}{10}= 0.5$

$F_1 = 2 * \frac{0.5*0.33}{0.5+0.3} = 0.4$

$Balanced\ Acc = \frac{1}{2}(\frac{5}{10} + \frac{990}{1000}) = 0.745$

You can see that balanced accuracy still cares about the negative datapoints unlike the F1 score.

For even more analysis we can see what the change is when the model gets exactly one extra positive example correctly and one negative sample incorrectly:

TP=6; FP=9; TN=989; FN=4

$Precision = \frac{6}{15}=0.4$

$Recall = \frac{6}{10}= 0.6$

$F_1 = 2 * \frac{0.6*0.4}{0.6+0.4} = 0.48$

$Balanced\ Acc = \frac{1}{2}(\frac{6}{10} + \frac{989}{1000}) = 0.795$

Correctly classifying an extra positive example increased the F1 score a bit more than the balanced accuracy.

Finally let's look at what happens when a model predicts there are still 15 positive examples (5 truly positive and 10 incorrectly labeled); however, this time the dataset is balanced and there are exactly 10 positive and 10 negative examples:

TP=5; FP=10; TN=0; FN=5

$Precision = \frac{5}{15}=0.33...$

$Recall = \frac{5}{10}= 0.5$

$F_1 = 2 * \frac{0.5*0.33}{0.5+0.3} = 0.4$

$Balanced\ Acc = \frac{1}{2}(\frac{5}{10} + \frac{0}{0}) = 0.25$

You can see that the F1-score did not change at all (compared to the first example) while the balanced accuracy took a massive hit (decreased by 50%).

This shows how F1-score only cares about the points the model said are positive, and the points that actually are positive, and doesn't care at all about the plathero points that are negative.


Why are results of cross-validation before splitting is different from results after splitting? All data in my binary classification problem are represented by X and y.  Now, I do stratified cross-validation on this data as follows:
scoring = {'accuracy' : make_scorer(accuracy_score),
           'precision' : make_scorer(precision_score),
           'recall' : make_scorer(recall_score),
           'f1_score' : make_scorer(f1_score)}

model=RandomForestClassifier(n_estimators=50,random_state=10)

results = cross_validate(estimator=model, X=X, y=y, cv=10, scoring=scoring)

If run the code, I will get the following results:
Accuracy : 0.5436815489342804
Precision : 0.020165565854870747
Recall : 0.11013513513513513
F1_score : 0.03315023853741518

Now, I split X and y as follows:
#Test, training data split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.20, stratify = y)

#Split the training data into validation set
X_val_train, X_val_test, y_val_train, y_val_test = train_test_split(X_train, y_train, test_size = 0.1, random_state=0, stratify=y_train )

Now, if I do the same cross-validation procedure like before on X_train and X_train, I will get the following results:
Accuracy : 0.8424393681243558
Precision : 0.47658195862621017
Recall: 0.1964997354963851
F1_score : 0.2773991741912054

I do not understand why results are so different and why this happened.

Without having more information my hypothesis the complexity of your model.
I think you might have a model so complex that it needs a "large" amount of data for having a good performance (First scenario with cross validation on full data)
Then when you split your data in train - test set and repeat the cross validation the amount of data is reduced naturally so it is causing that your model do not have enough samples to generalize.
That might be confirmed via a learning curve:

If the training and cross-validation scores converge together as more data is added (shown in the left figure), then the model will probably not benefit from more data. If the training score is much greater than the validation score then the model probably requires more training examples in order to generalize more effectively.


the models in deep-learning on a little corpus I have a little corpus (around 450 observations with approximately 50 words) with domain specific words. I have to classify each observation with a target variable with 5 classes. I tokenized and did the preprocess on my text and now I want to do the classification.
I tried random forest and perceptron and I get an accuracy around 0.75 which is pretty good!! I saw on the web deep learning was a good tool to work with when we have textual data (that's why I tried perceptron) but I'm pretty new in this field.
Do you have any idea which model in deep learning is good considering my corpus?

With Transfer learning.
There are already some pre trained models that most likely will do fine with you text. You just have to retrain them with some simple API. Try fastai for example. here and here


Simple neural network implementation in keras I have a binary classification problem(benign/malicious) and I have applied simple neural network with one hidden layer for solving the problem. I have 46 features in my dataset and for the hidden layer I am using 46/2. Also my data is not scaled so for it I am using standardscaler. I am getting accuracy of around 99.79% with the code. However the accuracy was unexpected, I was expecting around 93-94%, I am afraid that I am leaking some data or making some silly mistake.

def create_baseline():
    model = Sequential()
    model.add(Dense(23, input_dim=46, kernel_initializer='normal', activation='relu'))
    model.add(Dense(1, kernel_initializer='normal', activation='sigmoid'))
    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
    return model

estimators = []
estimators.append(('standardize', StandardScaler()))
estimators.append(('mlp', KerasClassifier(build_fn=create_baseline, epochs=1000, batch_size=len(X), validation_split=0.15, verbose=0)))
pipeline = Pipeline(estimators)
kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=seed)
scores = cross_val_score(pipeline, X, y, cv=kfold)
print("%.2f%%" % (scores[1]*100))

I am using around 5871 malicious samples and 3488 benign samples.

Just to cross check I have implemented without cross validation, a very simple implementation, and I am getting ~99% accuracy in that also.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=42)

scale = StandardScaler(with_mean=0, with_std=1)
new_X_train = scale.fit_transform(X_train)
new_X_test = scale.transform(X_test)

model = Sequential()

model.add(Dense(23, input_dim=46, kernel_initializer="normal"))
model.add(PReLU(alpha_initializer='zero', weights=None))

model.add(Dense(1, kernel_initializer='normal'))
model.add(Activation('sigmoid'))

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

history = model.fit(new_X_train, y_train, epochs=1000, batch_size=len(X_train), validation_split=0.15)

scores = model.evaluate(new_X_test, y_test)
print("%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))

print(history.history.keys())

plt.plot(history.history['acc'])
plt.plot(history.history['val_acc'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()

plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()

The plots I am getting are:

Considering the nature of your problem (disease classification?), probably you have imbalanced data: more negative/benign w.r.t. positive/malicious. In such cases accuracy is not a good metric to assess the quality of a model.

Consider, as example, a dataset with 99 "negative" cases and only one "positive" case, and consider a classifier which predict "negative" all the times. The accuracy of such model is, actually, 99%:

What you want to do in order to deal with unbalanced data is:

1) just use another metric, as ROC-AUC. This can be sufficient if you have a dataset not so much unbalanced, say up to 1:20. This metric is not officially supported in Keras, but you can find a custom callback which calculate the AUC score every end of epoch here.

2) if the imbalance is very big, say 1:1000, you need to balance the dataset. There are lots of techniques, the simplest one is to make copies of the minority class samples, as many as needed to reach the balance. Other smarter techniques exist, and some of then are included in imbalanced-learn for SKLearn.

If you want to have a better overview of techniques and metrics used in such cases, I suggest you this paper.


Multi target classification I am working on traffic violation data set which contains 36 columns(variable). I have two dependent variables out of this.

Example

id, description, age, sex    alcohol     vehicle     violation level Accident

120   speed      28    M       Yes         car             severe       yes
122   win-tint   35    F       No         truck            Medium       no
123   left-turn  26    M       No          car             light        no
124  failure-veh 28    M       No          car             severe       yes

The dependent variables are "violation level" and "accident".

violation level has 3 classes -- sever medium light

Accident has 2 levels -- Yes No

I want to predict both violation level and accident. I think this can be multitarget prediction.

Can someone help me which algorithm is good for this? I have seen some articles suggesting scikit- multiout put classification and neural network with multiple out put layers.

Or Can I go ahead with two models?

predicting the traffic violation level.

Predicting the accidents using the violation levels.

Basically I am trying to predict violation levels and then predicting how these violations contribute to accidents.

Any help would be appreciated.

At first the way you explained it, it sounded to me it is a multilabel classification (like e.g. CelebA) and approaches I have used for there was coming to my mind. Also I thought it is hierarchical classification because you have subcategories in your dependent variable. But at the end you mention that you want to

  Predicting the accidents using the violation levels.

Aren't then your dependent variables (violation level , and accident) already correlated/related to each other? It seems to me that you do not have two independent dependent variable. I would try to reduce the problem to a simple multiclass classification, in your case 6 as follows (assuming you have these explicit labels for each data point):

Severe-Yes
Severe-No
Medium-Yes
Medium-No
Light-Yes
Light-No

Then you have 6 classes/labels to build a model e.g. Gradient Boosting Tree or Neural Network.


How does the descending gradient know what weights to adjust? I was reading about descending gradient.
How does the descending gradient know what weights to adjust?
Does it adjust to all network weights at the same time?
Does each weight have an associated error?

In general (or at least the basic implementation for the gradient descent), you apply for each iteration the update rule for each weight, using the partial derivative of the loss function with respect to that weight, as follows:


sentiment analysis nltk python I'd like to perform sentiment analysis on stock comment using scikit and nltk. I already have about 100 comments on different stocks like "this stock will rock" which I marked as positive (1) or "this is doomed stock" which I marked as negative(0). So I'd like to train classifier which can tell whether new comments I add are negative or positive. So my question is how to perform it. I've already searched the Net but all I found was movie review sentiment analysis which is quite remote from the topic.

What you're doing right now is a traditional classification using supervised learning. This is a great method for predicting outcomes, but I suspect there are much better ways to complete this sentiment analysis project you're working on.

Without knowing what the goal of your analysis is, I would suggest you look at the NLTK package. A lot of work has been done to idenify how positive or negative a collection of words is, and you could piggyback on the work of some really smart people.

If you're having trouble starting, you can look at NLTK's how-to tutorial, some Kaggle kernels, or find some articles online.

Good luck!


Map a number to a string I have vector c(1,3,4,5,7).
I want convert it in this way:
c('apple','pie','nuts','cookies','some_other_class')

Do you have any short ways to do it? Like a switch method or something like that?

Found the solution:

  y <- factor(levels = c(1,3,4,5,7),
              labels = c('apple','pie','nuts','cookies','some_other_class')
             )


How can Clustering (Unsupervised Learning) be used to improve the accuracy of Linear Regression model (Supervised Learning) I came through this questions and I failed to find the right answer for it.

How can Clustering (Unsupervised Learning) be used to improve the accuracy of Linear Regression model (Supervised Learning)?

a- Creating different models for different cluster groups.
b- Creating an input feature for cluster ids as an ordinal variable.
c- Creating an input feature for cluster centroids as a continuous variable.
d- Creating an input feature for cluster size as a continuous variable.

Just an idea:

You may be able to cluster a continuous variable and add the clusters as indicator in a linear regression jointly with an interaction term.

Say your linear model is income (inc) explained by years of job experience (exp)...

inc = b0 + b1*exp + u.

Now say you have two groups (low/high skilled workers). If you cluster inc in two groups, and add this as a dummy with interaction term, you may be able to increase the fit of the model.

inc = b0 + b1*exp + b2*indicator + b3*exp*indicator + u.

The idea is that low/high skilled workers will increase income contingent on experience in a different way (different slope). However, this approach requires that you can identify clusters well.

Edit: same strategy may work with clustering the X.


How does one individually extract elements from a tensor? Let us assume that I have a tensor in tensorflow with 2 elements.

How do I extract these individual elements and perform calculations on them?

def multiplication(a):
    return a[0]*a[1]

def main():
    sess=tf.Session
    a=tf.constant([[2,3]])
    answer= multiplication(a)
    print(sess.run(answer))
    #answer should be 6

First recognize that your input array has shape (1,2), so a[0]*a[1] will not work.

Following is the code that is working for me-

def mul(a):
    return a[0][0]*a[0][1]

a = tf.constant([[2,3]])
b = tf.contrib.eager.py_func(func = mul, inp=[a], Tout = tf.int32)
with tf.Session() as sess:
    sess.run(b)

If you want to pass custom functions, check this link- https://www.tensorflow.org/api_docs/python/tf/py_function


Why is "next state" kept in RL experience replay? Following this explanation on what is experience replay (and others), I noticed an experience element is defined as

$e_t = (s_t,a_t,r_t,s_{t+1})$

My question is, why do we need the next state in the experience?

To my understanding, our networks learn state to action and action to reward mappings, so I fail to see where the "next state" is used in experience replay?

True, but if you take a look at the equations of Q learning or Advantage functions in Policy Gradients you will see that the expected value of the next state is being used. For this, you need to know in which state you have landed from state $s_t$. For example, you can approximate your future reward from state $s_{t+1}$ by using $Q(s_{t+1})$. To do this you need to give to your network as input the $s_{t+1}$.


1D convolution for uni-variate data Every one. I have EEG dataset with 80 subjects, 3072 data points and 100 trials. This a univariate data, it mean there is only one channel. I am confused how to feed this data to convolution neural network.
 Most of blog and tutorial deals with multivariate data. But i have univariate and i am clueless how to move ahead

I believe that the following image (original link) will be helpful to understand. 1D convolutions work exactly the same way as 2D convolutions, the main difference is how the filter/mask is applied. 1D convolution is intuitively like a sliding window of a certain width.

Many packages like Keras or PyTorch have native 1D convolution function/modules, so I would recommend checking their documentation and perhaps source code for deeper understanding.


Returned loss value is different than the loss printed with verbose Could someone explain why the loss returned is different than the loss printed during the evaluation?
They are the same in the Tensorflow documentation https://www.tensorflow.org/guide/keras/train_and_evaluate
Code:
results = model.evaluate(test_data, test_target, verbose=2)
print("test loss, test acc:", results)

Output:
45/1 - 0s - loss: 1.2592 - mae: 0.7602
test loss, test acc: [1.05335361427731, 0.76020277]

The model.evaluate function predicts the output for the given input and then computes the metrics function specified in the model.compile.Based on y_true and y_pred and returns the computed metric value as the output.
You can use  model.metrics_names  property of your model to find out what each of those values corresponds to.


Clustering with multiple distance measures I'm trying to use clustering to automate a group-finding process with the aim of being able to automatically detect groups in unseen data. The data are html elements within any given webpage, this includes numerous features including spatial and visual information rendered from a web browser. The detected groups are usually semantically meaningful, for example a group of navigation links or a list of items in the main content area.

I've implemented a subtle distance measure that captures how similar or different these elements are and I'm passing these values to DBSCAN for clustering. So far it works quite well but I'm trying to think of ways to improve it.

The distance function is actually a weighted sum of 10 separate distance measures that capture different kinds of similarities and differences. I've weighted the more reliable and decisive measures higher than others. In theory, the measures that are weighted lower should only have an effect when they are congruent and are otherwise overpowered by the higher-weighted measures. Additionally each measurement function can itself amplify or attenuate it's own weight for a given item-pair, for example set its weight to zero if it is unsure and wants to abstain, or set it higher if it has found a similarity/difference that is particularly notable or reliable.

Although it's broadly working quite well, I've noticed that often one or two of the largest clusters should be further split. I've considered two methods for this: (a) attenuating the weights of measures that are mostly giving a low distance value within the cluster (thereby amplifying other measures with higher variance) and then rerunning DBSCAN or (b) searching clusters for subsets with large numbers of co-occurring features (e.g. some css class in the parent element and a particular font-size) and either splitting the cluster based on these, or adjusting the distance measures to reflect this apparent grouping and then rerunning the global DBSCAN with the adjusted values.

Currently it all works unsupervised but I'm open to manually creating a set of ground-truth data for improve it. Maybe then this could be used to optimize the weights in such a way that minimizes within-cluster distances and maximizes between-cluster distances? I also like the idea of running clustering multiple times, repeatedly adjusting the weights based on the variances of each measure within the cluster.

I'm looking for more ideas on how to optimize this, I would appreciate any input. I have seen some comments elsewhere that unsupervised clustering is only suitable for exploring data and not automating decisions, but I haven't been able to find a better way of finding these groups with my limited ML/data-science experience. Given all the time I've spent building this, the ideal thing would be to figure out a clever way of adaptively adjusting these weights in a more intelligent way.

This is just a wild guess, but I was wondering whether your custom distance function is indeed a distance function.
In particular, your problem might occur because your distance function $d(x, y)$ does not separate observations: This is the case if there exist two observations $x_1$ and $x_2$ that are distinct, $x_1 \neq x_2$, but have zero distance, $d(x_1, d_2) = 0$. Then $x_1$ and $x_2 $ would necessarily be put into the same cluster by your algorithm.
If this happens for many observations in your test set, then you will observe large clusters.

You might be able to check on paper that your function satisfies the four axioms of a distance function (see the Wikipedia article above); if that is not possible, you could at least check it by looping over your test set.


Adding anomalies to the Dataset Recently I have been trying different Scikit-Learn anomaly detection clustering methods, like

DBSCAN
Isolation Forest.

Based on how many training data I use, how I tweak on the algorithms

Example in DBSCAN I play around this min_samples and eps distance.

My problem
Now I am getting different results, however this is where the problem is coming. I don't know how many anomalies I am supposed to get, so this leads to me not being sure what is working best.

I have read a it into adding anomalies yourself into the dataset. Is this a thing?
Meaning that I would know if the found anomalies from the algorithm are the same as the one that have been added to the dataset.

I might have gotten this wrong.

The other problem is that the column I want to check for anomalis has numbers that really differ in magnitude so I also wouldn't what numbers I should add as anomalies.

Would appreciate some help.

Without labels for which data points are anomalies, you have no way to know how many anomalies you should get.

The closest thing to adding anomalies to the dataset is using synthetic data for anomaly detection. I have seen some practitioners doing this, and failing. When you create the synthetic data, it matches your inductive bias*. The real world very rarely matches that inductive bias*. Afterwards, you can easily find a clustering algorithm with the same inductive bias* which can flag your synthetic data as anomalies. However, when that algorithm is deployed it will fail miserably.

*in plain language, inductive bias is the assumptions you have of how the world should be.


About data cleansing, to what extent should we do our work? As far as I know, when we deal with 'big data'. It is common we deal with more than 10 years' customers data. Quality issue is always there.

What I am thinking about, to what extend we should deal with the data quality issues? A robust algorithm should tolerate with the data quality issues but if we do better data cleansing work we get outcome confidently.

So how to balance this?

Somebody please give some ideas!

Thanks.

This is a question that is very dependent upon the data in question. Assuming that you can train your models in a reasonable time, I would start by not cleaning the data at all, and seeing how well your model performs, and then cleaning it a bit and redoing the experiment and so on.
This is because it is possible to overclean your data, essentially removing variations in the data which are actually useful to your modelling.
The amount of cleaning that you apply is yet another meta-variable in your model.


How to save and load a Q-Learning Agent I know this may sound nooby, but how do I save a Deep Q-Learning agent's progress? I mean when I close at i.e. episode 500 when my agent is trained and I restart (in my case a pygame) my agent is going to load its data and it will continue with its progress ( does not start again very stupid :) ). How can it be done?

If you need more details/clarification, feel free to ask! :)

Thanks for any answer! I really do appreciate it!

To save a model's state, it is enough to save the model's parameters. If you are using Torch, then you can save it as follows torch.save(model.state_dict(), path_to_save).

When you want to resume training with a saved model, you would have to first create the model instance, and then you can use model.load_state_dict(torch.load(path_of_saved_model)) to update the new instance with the parameter values of the saved model.

You can find more reference here: https://pytorch.org/tutorials/beginner/saving_loading_models.html

There will be something similar for Tensorflow or any other framework that you use.

You can also have an evaluation function that evaluates how good your model is after a certain number of iterations, and save the one which performs the best.

Hope that helps!


Integer encoding and weighing when one feature consists of more names Hello I am trying to make a content based movie recommendation system and one feature is genre of the movie. I will give an integer number to each genre randomly. However, some movies are of more than 1 genre. I will use tf-idf for weigthing these features. However, I am very confused that what to do when a movie is horror and action and comedy movie at the same time. Should I multiply or add these weighted features, I have no idea. Also I do not know why we use weighting in the first place at all. Can you help me get through this? By the way, let me say that each movie will be regarded as a document and then tf-idf calculation is done on them.

That's a good question. This task would be referred to as multi-label encoding. Bascially if a movie belongs to several genres, you one hot-encode each genre and add the vectors.
If there were only 6 genres (horror, romance, action, adventure, comedy, fantasy)
For instance a movie that is horror, action and comedy (The Dead don't Die?):

horror = [1, 0, 0, 0, 0, 0]
action = [0, 0, 1, 0, 0, 0]
comedy = [0, 0, 0, 0, 1, 0]

So a movie that belongs to all three would be encoded as: [1, 0, 1, 0, 1, 0]. A movie that only has a category then its encoding is a one-hot encoded label.
You can perform this task in scikit-learn with a multi label binarizer


Machine Learning - Euclidian Distance Classifier exercise I'm taking part in an elective subject at university which mainly focuses on the foundations of Machine Learning.
Now we got our first exercise - this task should be done practically in any language (I've chosen Python). Our teacher doesn't explain the relations between theory and practice well and so it's hard for all of us to follow along - so I decided to post this question here. I don't want anyone to give me a solution, I just don't understand what he wants and may be a hint how to approach this question.:
Here is the full exercise:

Euclidean distance classifier

Develop an Euclidian distance classifier as below: Generate 1000
random points corresponding to each class out of 3 classes with
feature size 2 for a 3-class classification problem. For the
simplicity consider the classes following N([0 1 2], I), N([0 0 1], I)
and N([1 0 0],I) respectively.

Generate the output an 1000-dimensional vector whose ith component
contains the class where the corresponding vector is assigned,
according to the minimum Euclidean distance classifier.

I understand that I should generate random points with two features which are belonging to one of the three classes - okay. But I don't get the second part of the sentence. The classes are normally distributes with a mean(?)-vector of [0, 1, 2], [0, 0, 1] and [1, 0, 0]?

For what does the second paramter I stand in the normal distribution
Does the vector stand for the position/mean of the multivariate normal distribution?
How would you approach this question? Using a k nearest neighbor algorithm?

Thanks for any helpful answers!
Max

Clearly your exercice is not very clear, at least it is not to me.
I guess you should consider $I$ as the variance or std deviation of the variable, just make sure that is a parameter of your code so you can change it later in case it is not what you assumed it was. Check up in your course if it corresponds to something, but it doesn't seem anything familiar to me.
Not sure what your second question means, but here are the vectors that need to be created according to me :

Vector of points (1st question), each points lives in 2-D space (feature size).
Vector of classes (2nd question), this vector looks something like [1, 0, 2, 2, 1, 1, ...]. For this vector, it would mean that first value of your vector of points belongs to class 1, second value to class 0 , third to class 2, ...

As the exercise is asking about euclidian distance classifier, you just need to create an algorithm that takes a point, computes the euclidian distance with each class center, and classify it in the closest class. (Not sure that kind of classifier can be considered Machine Learning but w/e).
You probably noticed it, but the mean values given are 3-D vectors, which is quite weird as I said that our points live in 2-D space. I guess the last value of each mean vector is the class of the point generated (just my guess, not blaming your teacher, but as I said your exercise isn't very clear). So according to that interpretation, the classes would be distributed according to distribution :

[0, 1], I variance or std deviation for class 2.
[0, 0], I variance or std deviation for class 1.
[1, 0], I variance or std deviation for class 0.

Hope it helps, if you have any remaining questions, feel free to ask.


Extracting email id gives error I am extracting email ids and storing them into a new column variable, but I am getting the issue:

    import re
    def email_extract(comments):
        comments1 =re.findall(r'[\w\.-]+@[\w\.-]+',comments)
       return comments1

   data["email_id"] = data.COMMENTS.apply(lambda x: email_extract(x))

--- TypeError                                 Traceback (most recent call
last) <ipython-input-33-d9b73bdc4f8e> in <module>()
----> 1 data["email_id"] = data.COMMENTS.apply(lambda x: email_extract(x))

C:\ProgramData\Anaconda4\lib\site-packages\pandas\core\series.py in
apply(self, func, convert_dtype, args, **kwds)    3192
else:    3193                 values = self.astype(object).values
-> 3194                 mapped = lib.map_infer(values, f, convert=convert_dtype)
3195     3196         if len(mapped) and
isinstance(mapped[0], Series):

pandas/_libs/src\inference.pyx in pandas._libs.lib.map_infer()

<ipython-input-33-d9b73bdc4f8e> in <lambda>(x)
----> 1 data["email_id"] = data.COMMENTS.apply(lambda x: email_extract(x))

<ipython-input-32-97f3705d1972> in email_extract(comments)
      2 def email_extract(comments):
      3     #re_pattern = re.compile(r'[\w\.-]+@[\w\.-]+')
----> 4     comments1 =re.findall(r'[\w\.-]+@[\w\.-]+',comments)
      5     return comments1

C:\ProgramData\Anaconda4\lib\re.py in findall(pattern, string, flags)
    221
    222     Empty matches are included in the result."""
--> 223     return _compile(pattern, flags).findall(string)
    224
    225 def finditer(pattern, string, flags=0):

TypeError: expected string or bytes-like object

How can I fix this issue?

You may have missing data (e.g. np.nans) in your COMMENTS field. This will throw an error in your email_extract function.
Try to filter the problematic rows out, e.g. with

filtered_data = data[~data.COMMENTS.isna()] #purge problematic comments

and applying the email extraction to the now "clean" column:

filtered_data.COMMENTS.apply(lambda x: email_extract(x))


Multivariate, Multi-step LSTM time series forecast I'm trying to predict the Pollution using a Multivariate and Multi-step LSTM code, I've been following this tutorial.

I've been following the code until the end, but couldn't understand where the code writer determined the Pollution column to be the code output (predicting the Pollution column instead of an other column?) I'm new to python, and it got me confused.

At first, I thought that this is the part of the code where we define it, but I was wrong:

# invert scaling for forecast
pred_scaler = MinMaxScaler(feature_range=(0, 1)).fit(dataset.values[:,0].reshape(-1, 1))
inv_yhat = pred_scaler.inverse_transform(yhat)
print(inv_yhat.shape)
# invert scaling for actual
inv_y = pred_scaler.inverse_transform(test_y)
print(inv_y.shape)

Any explanations on how he determined the pollution column out of the 7 other columns to be the output?

I know this tutorial, it's a good start for RNNs but it contains a lot of passages and transformations that could have been kept shorter.

First, he defines a function called series_to_supervised() to process data to be fed into an RNN. Paragraph 3, line 37:

reframed = series_to_supervised(scaled, 1, 1)

This reframed dataframe contains all data, either y columns and all the X variables to make a prediction.

In the following code block it is turned into a numpy array at line 2:

values = reframed.values

Ok, so now all our information is store in values. Now it's time to separate it in train and test:

train = values[:n_train_hours, :]
test = values[n_train_hours:, :]

And again, each train and test is separated in x and y pieces:

# split into input and outputs
train_X, train_y = train[:, :-1], train[:, -1]
test_X, test_y = test[:, :-1], test[:, -1]

Line 7-8. This is where the dependent variable is separated from the rest. Knowing it was the last column, it was extracted with index -1 (i.e. the last element).

As I said, this tutorial is a good start to learn time series prediction with RNNs. However, I find that sometimes he tried to simplify the steps so much... that he ended up with some messy parts. All those objects: reframed, values, train, test, ... there was no need to make so many of them.

That apart, I'm a fan of the blog. It provided a lot of useful tips on RNNs.


Python sklearn - average classification reports Using Python's sklearn module,

 from sklearn.metrics import classification_report
 y1_predict = [0, 1, 1, 0]
 y1_dev = [0, 1, 1, 0]
 report_1 = classification_report(y1_dev, y1_predict)
 y2_predict = [1, 0, 1, 0]
 y2_dev = [1, 1, 0, 0]
 report_2 = classification_report(y2_dev, y2_predict)

Is there a way to combine (maybe just an average) report_1 and report_2 I'm looking for an implementation like:

 report_average = average(report_1,report_2)

Or does this have to be done manually? I was hoping that printing the report_average would have average values between the two reports.

Here's a MWE of the accepted answer:

    from sklearn.metrics import classification_report
    import pandas as pd
    import numpy as np
    from functools import reduce
    def report_average(*args):
        report_list = list()
        for report in args:
            splited = [' '.join(x.split()) for x in report.split('\n\n')]
            header = [x for x in splited[0].split(' ')]
            data = np.array(splited[1].split(' ')).reshape(-1, len(header) + 1)
            data = np.delete(data, 0, 1).astype(float)
            avg_total = np.array([x for x in splited[2].split(' ')][3:]).astype(float).reshape(-1, len(header))
            df = pd.DataFrame(np.concatenate((data, avg_total)), columns=header)
            report_list.append(df)
        res = reduce(lambda x, y: x.add(y, fill_value=0), report_list) / len(report_list)
        return res.rename(index={res.index[-1]: 'avg / total'})

    y1_predict = [0, 1, 1, 0]
    y1_dev = [0, 1, 1, 0]
    report_1 = classification_report(y1_dev, y1_predict)
    y2_predict = [1, 0, 1, 0]
    y2_dev = [1, 1, 0, 0]
    report_2 = classification_report(y2_dev, y2_predict)

    report_ave = report_average(report_1,report_2)

    print(report_ave)

Which yields

             precision  recall  f1-score  support
0                 0.75    0.75      0.75      2.0
1                 0.75    0.75      0.75      2.0
avg / total       0.75    0.75      0.75      4.0

It maybe a little bit complicated, since I convert the reports to pandas.DataFrame for calculation. But I think it's worth it, because it works well with two or more report as well. Try below:

import pandas as pd
import numpy as np
from functools import reduce
def report_average(*args):
    report_list = list()
    for report in args:
        splited = [' '.join(x.split()) for x in report.split('\n\n')]
        header = [x for x in splited[0].split(' ')]
        data = np.array(splited[1].split(' ')).reshape(-1, len(header) + 1)
        data = np.delete(data, 0, 1).astype(float)
        avg_total = np.array([x for x in splited[2].split(' ')][3:]).astype(float).reshape(-1, len(header))
        df = pd.DataFrame(np.concatenate((data, avg_total)), columns=header)
        report_list.append(df)
    res = reduce(lambda x, y: x.add(y, fill_value=0), report_list) / len(report_list)
    return res.rename(index={res.index[-1]: 'avg / total'})

output:

report_average  = report_average(report_1, report_2)
print(report_average)
             precision  recall  f1-score  support
0                 0.75    0.75      0.75      2.0
1                 0.75    0.75      0.75      2.0
avg / total       0.75    0.75      0.75      4.0

report_3 = report_2
report_average  = report_average(report_1, report_2,report_3)
print(report_average)
             precision    recall  f1-score  support
0             0.666667  0.666667  0.666667      2.0
1             0.666667  0.666667  0.666667      2.0
avg / total   0.666667  0.666667  0.666667      4.0


Train LSTM model with multiple time series I am predicting energy usage for a bedroom within a school residential building with date, temperature, and humidity as input features, using 7 time-steps and predicting for one-day (one-timestep).

I am able to predict for one bedroom. The building has about 100 bedrooms, all more or less the same layout and occupancy, so the data for all these 100 bedrooms have the same date range as the first bedroom and same humidity and temperature levels on the same day. Now I want to train my model with all the 100 bedroom's energy data so that the model captures all 100 rooms energy patterns.

Then, I want to predict what the energy usage is for any given bedroom (not a specific bedroom), using this model that has been trained with all 100 bedrooms. In other words, the business question is "What is the expected energy usage for a bedroom within this residential school building?"

I would like to know how I can train all these 100 rooms into one LSTM model, given that LSTM is a time series model, and 100 rooms mean I have 100 times series. I cannot simply concatenate the datasets of all rooms.

How would I structure my data and run it in the LSTM model?

I would appreciate if someone could show me an example of how it could be done.

As you mention, all rooms have similar properties, thus it makes sense developing a single LSTM model for all of them.

You have to pre-process your data so that you have input vectors $X_i$ of 7 time steps and 1 time step as label vector $Y_i$:

$
X_i = [x_i[t-6 ],x_i[t-5 ],x_i[t-4 ],x_i[t-3 ],x_i[t-2],x_i[t-1 ],x_i[t]]\\
Y_i = [x_i[t+1]]
$

where the subscript $i$ denotes the $i_{th}$ vector.

Let's imagine you can generate 100 vectors for each room, you'll end up with a dataset of $10^4$ input vectors. The number of vector per room will depend on the overlap you use, for instance, if you have data for 100 days per each room and you use an overlap of 99 days (your window slides one day in the future), you'll have 99 vectors for each room.

To train the LSTM you use the typical Mini-batch training. Make sure you don't propagate the state for batch sample $i$ to sample $i+1$ in order to treat them individually (in Keras you set the stateful flag to False)


Trade off between Bias and Variance What are the best ideas or approaches to trade off between bias and variance in Machine Learning models.

You want to decide this based on how well your model performs and generalizes. If your model is underfitting, you want to increase your model's complexity, increasing variance and decreasing bias. If your model is overfitting, you want to regularize the model and/or feed it more training data, decreasing variance and increasing bias.


Depth of a Neural network I was self-teaching myself. I totally understand why depth of a neural network affects the learning and how it differs than its width. But I am looking for some theoretical justification about it. Papers I could come up with, e.g., Benefits of depth in neural networks or
The Power of Depth for Feedforward Neural Networks   are unfortunately is too deep and long. I am not also super good at Mathematics, however, I believe there must be a plain, short and compact math behind it. Can someone point me to some tutorials/articles/papers/reports, where I can easily understand it?

I think the short mathematical intuition would be to show that Deeper offers more flexibility.

Imagine we want to fit an impossibly complicated function, like this:

... but in an n-dimensional space - we obviously cannot visualise such a function. But we can agree is is complicated

Basic math analogy

The goal of the neural network would be to map the raw input data (e.g. images to a convolutional network) to some output, by approximating the complicated function.

So if we have some input, and apply a non-linear function $f$ to it, we transform it into something else:

$$output_1 = f(input)$$

Perhaps that gave us a curvy function, but it doesn't get close to matching the complicated function, so we endow the model with another chance, by applying a second non-linear function:

$$output_2 = g(output_1)$$

Giving a second function is almost like offering another degree of freedom (or flexibility) to the model.

We continue like this until the chain of non-linear functions is able to map out the output space sufficiently well. Perhaps we end up with this:

$$output_6 = k(j(i(h(g(f(input)))))$$

In this framework, imagine every single non-linear function as one of the layers in a deep network. The deeper the network gets, the more functions we are applying and the more we mould and transform the input to something else; perhaps in different ranges, by different magnitudes and so on.

This should (in a very hand-wavey manner) convince you that more functions applied gives more possibilities in the final output space. So more layers gives us more power to express more and more complicated functions.

A practical note: the more layers we add, the more powerful the model and the larger the tendency to either:

be very difficult to train
eventual overfit the training data completely
the longer the model takes to train
the greater the level of regularisation that is likely required to obtain a reasonable validation metric on unseen data

image source


"Hadoop" formats for user database: online advertising I was wondering if someone could point me to suitable database formats for building up a user database:

basically I am collecting logs of impressions data, and I want to compile a user database

which sites user visits, country/gender/..? and other categorisations with the aim of
a) doing searches: give me all users visiting games sites from france...
b) machine learning: eg clustering users by the sites they visit

so I am interested in storing info about 100's of millions of users

with indexes? on user, sites, geo-location

and the idea would be that this data would be continually updated ( eg nightly update to user database of new sites visited etc)

what are suitable database systems. Can someone suggest suitable reading material?
I was imagining Hbase might be suitable...

Storing user profiles

If you just want to store all user profiles... just save them into normal RDBMS. Assuming one user profiles takes 10Kb of storage, you need only ~9.5Gb for every million of users, which is pretty little and gives you all advantages of mature relational databases.

It makes sense to use HBase only when you have really many users (say, > 1B) or when data is very sparse (most columns are empty). But don't expect it to be as convenient as good old SQL databases.

In advertising, and especially in real-time bidding, very fast retrieval of user profiles is needed. Aerospike becomes more and more popular for this task.

Analysing data slices

Common use of business logs is to analyse specific slices of data, e.g. number of users from France that visited sites from "game" category on November 1-14, 2014. Standard way to manage such data efficiently is to organize them into data cubes. You won't get individual records (e.g. users), but you'll get aggregated statistics really fast.

Such cubes may have many different dimensions, but in 99% of cases they have date field that they are partitioned by. It makes great sense, because almost every query includes time period to get data from.

As for software, Vertica is great for such aggregations. Cheaper* solution from Hadoop world is Impala, which is also great.

(* - if you count only license price)

Machine learning

It really depends on concrete tasks and ML toolkit in use. For real-time bidding you would want blazing fast access to user profile vectors and would probably prefer Aerospike. For online learning Spark Streaming may be used as a data source, and no storage used at all. For offline machine learning there's excellent MLlib from the same Spark project, which works with a variety of sources.


Can CNNs detect features of different images? In lecture, we talked about “parameter sharing” as a benefit of using convolutional networks. Which of the following statements about parameter sharing in ConvNets are true? (Check all that apply.)

It allows parameters learned for one task to be shared even for a different task (transfer learning).
It reduces the total number of parameters, thus reducing overfitting.
It allows gradient descent to set many of the parameters to zero, thus making the connections sparse.
It allows a feature detector to be used in multiple locations throughout the whole input image/input volume.

Here are the correct answers:

It reduces the total number of parameters, thus reducing overfitting.
It allows a feature detector to be used in multiple locations throughout the whole input image/input volume.

Why isn't the following answer also correct:

It allows parameters learned for one task to be shared even for a different task (transfer learning).
Doesn't ConvNets allow parameters to be shared, detecting the similar features of different images?

The point is that parameters sharing is not the same as reusing some learnt weights in a task for another task (a.k.a transfer learning); it is, transfer learning can be used with convolutional neural networks, but parameters sharing does not mean transferring knowledge from one task to another.
For a detailed definition of parameters sharing, look at this documentation from Stanford university , section Parameter Sharing
As explained in that page, you can think of it as a tool for reducing the number of weights while learning in a specific task (so forget about traqnsferring these weights to another task, which could be the case, but not the meaning of this concept).
An intuitive way of reasoning the concept could be:
parameter sharing assumption is relatively reasonable: If detecting a horizontal edge is important at some location in the image, it should intuitively be useful at some other location as well due to the translationally-invariant structure of images


Generative network understanding I was going through GAN's notebook by fchallot on Generative Adversarial Networks where, in the Generator Network, he creates a Dense layer with $16*16 * 128$ (where 128 is the number of channels).

How exactly does latent_dim=32  becomes of shape $16 * 16$ in the network.
How are these values decided?

latent_dim does not become of shape 16*16

x = layers.Dense(128 * 16 * 16)(generator_input)

mean: The input of size 32 (the latent_dim) is connected to a layer of size 16*16*128

These value are decided by the data scientist, they are the hyper-parameters.
There is a lot of research on how to adjust hyper-parameters for GAN.
People use generally a size of 100 for the latent_dim but it may depends of the complexity of the image you are trying to generate.


How to make a Python code execute on GPU and not CPU I am doing some image pre-processing using Python 3. As I know the code is executed on CPU for which the process is a little bit slow, its a good idea to make it run on GPU for the faster process because GPU can do graph operation much faster than CPU. Is there a way to make it run on GPU without using a library?
Please note that it is required to make it like a GPU version of Keras or TensorFlow.

You may apply Wolfram Language to your project. There is a free Wolfram Engine for developers and with the Wolfram Client Library for Python you can use these functions in Python.

Either the CUDALink Overview and CUDALink Guide or the OpenCLLink Overview and OpenCLLink Guide will enable you to run code on your GPU.  However, the CUDALink guide image process functions built-in so perhaps best to start there for your project.

from wolframclient.evaluation import WolframLanguageSession
from wolframclient.language import wl, wlexpr

Start the Wolfram session.

wolfSession = WolframLanguageSession()

The CUDA package is included with the Wolfram Engine but has to be explicitly loaded.

wolfSession.evaluate(wl.Needs('CUDALink`'))

Then you can check if the engine has located one or more compatible GPUs with CUDAQ.

print(wolfSession.evaluate(wl.CUDALink.CUDAQ()))

True

There is a one-time download of CUDA binaries when CUDAQ is evaluated. You can make your first call directly in the kernel to track the one-time download instead of first calling through Python. Run woframscript in the terminal and evaluate the Needs and CUDAQ functions at the input prompt.

Detailed information can be returned with CUDAInformation.

print(wolfSession.evaluate(wl.CUDALink.CUDAInformation(1,'Core Count')))

160

Grab an image for some CUDA image function examples.

imgWolfram=wolfSession.evaluate(
    wl.Rasterize(
        wl.Import('https://i2.wp.com/oemscorp.com/wp-content/uploads/2015/07/wolfram-language-logo.png'),
        'Image'
    )
);

The image can be Exported in one of the supported Raster Image Formats or Vector Graphics Formats.

wolfSession.evaluate(
    wl.Export(
        '<path with image filename>',
        imgWolfram
        )
    )
)

A few example CUDA image functions include

CUDAImageConvolve

wolfSession.evaluate(
    wl.Export(
        '<path with image filename>',
        wl.CUDALink.CUDAImageConvolve(imgWolfram,  [[-1,0,1],[-2,0,2],[-1,0,1]])
    )
)

CUDAErosion

wolfSession.evaluate(
    wl.Export(
        '<path with image filename>',
        wl.CUDALink.CUDAErosion(imgWolfram,6)
    )
)

CUDAOpening

wolfSession.evaluate(
    wl.Export(
        '<path with image filename>',
        wl.CUDALink.CUDAOpening(imgWolfram,6)
    )
)

Terminate the Wolfram session.

wolfSession.terminate()

Since you have processing many image files you would get slightly better performance by processing directly in wolfram script with the Wolfram Engine rather than by calling the engine from Python.

Hope this helps.


How to install XGBoost or LightGBM on Windows? I'm a Windows user and would like to use those mentioned algorithms in the title with my Jupyter notebook which is a part of Anaconda installation.

I've tried in anaconda promt window:

pip install xgboost

which retuned:

  Could not find a version that satisfies the requirement xgboost (from
  versions: ) No matching distribution found for xgboost

Likewise, I've tried:

conda install lightgbm

which returned:

  Solving environment: failed

  CondaHTTPError: HTTP 000 CONNECTION FAILED for url
  https://repo.anaconda.com/pkgs/r/noarch/repodata.json.bz2 Elapsed: -

Help would be appreciated.

Follow this XGBoost installation guide: https://xgboost.readthedocs.io/en/latest/build.html

If you are using python3 then make sure that you run: pip3 install xgboost

To fix the problem with lightgbm on windows try installing OpenSSL first, refer this: https://www.cloudinsidr.com/content/how-to-install-the-most-recent-version-of-openssl-on-windows-10-in-64-bit/n


Heuristics, methods to speed up searches over subsets of big set (combinatorially NP hard probably) I have a reasonable-sized set of size N (say 10 000 objects) in which I am searching for groups of compatible elements. Meaning that I have a function y = f(x_1, x_2, x_3, ..., x_n) returning bool 0/1 answer whether n elements are compatible. We are interested in executing this search on each subset smaller than 8 elements of N.
Which is obviously either NP hard or close to this.
Even for pairwise search for n elements set we have n^2 pairs to explore, which is somehow doable yet still grows quite fast for big sets.
Yet then it goes nasty when we are looking for triples (identifying 3-element cliques) and quadriple or greater magnitudes.

Would you suggest any machine learning approach to help with this task? Maybe at efficiently sampling the search-space and identifying which subsets to search on, maybe fast heauristic evaluations of the function to direct the searches.

This is not ML but your setup reminds me of the Covering Array Problem, which is used for Software Testing.
If you consider your configuration arrays [0 0 1 0 1 0 1 0 1 .... ] as certain tests run , a typical question is what is the minimum number of tests I need to run while I can cover most different setups? Important to increase test completeness but also minimize time it takes to run them.
Possible some covering array solutions might help. There are even tables that are pre-computed that you can use for different number of N and coverage strengths https://math.nist.gov/coveringarrays/coveringarray.html
And as you pointed out this is a combinatorial problem that suffers from combinatorial explosion. You might want to divide/conquer (split the N into k manageable groups).


Document Layout Analysis - state-of-the-art? What is the current state-of-the art within document layout analysis?

I.e. detecting columns, separating images from text, distinguishing between page numbering and text and so on.

I am looking for papers and algorithms on the topic.

After some research, I came across ICDAR (International Conference on Document Analysis and Recognition), which is taking place biannually and seems to be the most complete and up-to-date source for state-of-the-art document analysis algorithms.

ICDAR is hosting a competition in different subfields each year. At the time of this writing, the latest results from ICDAR 2015 have been published as Competition on Recognition of Documents with Complex Layouts –
RDCL2015. (Search online for a PDF version.) This paper describes state-of-the-art document layout analysis algorithms, and compares their performance to both open-source and commercial solutions.


ICD-10 codes in Machine Learning Can anyone provide specific techniques with using ICD-10 codes in Machine Learning? I have usually used a simply approach of creating multiple binary column representing ICD-10 codes… which can get extremely long. Or I have used Hashing features. Are there other techniques or ways to use ICD-10 codes in ML? Anyone can provide a useful link to see various approaching to modeling such features?

Most machine learning around ICD codes deals with auto-encoding documents or NLP to extract ICD codes automatically from documents. Once you have them, most applications I have seen use them as is.

One simple alternative would be to assign the codes to relevant categories to reduce the number of levels and make them more interpretable. For example

injury types (e.g. F00–F99 for mental disorders)
body parts (e.g. S10-S19 & .. for neck injuries)
severity of injury etc.

depending on your application. Of course, if you don't want to define the categories beforehand, this can be done using some clustering or embedding method etc.


Why does measuring for Asymmetric Binary Attributes exclude t (Values which are 0 for both)? I'm teaching myself Data mining and now struggling from this problem.

According to this table(http://slidewiki.org/slide/22352),
if the attributes are Asymmetric, dissimilarity is calculated without t (Values which are 0 for both)

Let me give an example.

There are 2 objects, called Products.

They have 2 attributes, COLOR and SIZE.

COLOR is either RED(1) and BLUE(0) (They are binary).
SIZE is either BIG(1) or SMALL(0) (They are binary too).

When there are RED-BIG(1-1) & RED-SMALL(1-0) objects, d is 1/2, regardless of if the attributes are symmetric or asymmetric.

However, if they are BLUE-SMALL(0-0) and BLUE/BIG(0-1), d is 1/2 for symmetric BUT 1/1 for asymmetric.

How does the difference come from? In both cases, there is only one difference, BUT dissimilarity isn't the same.

I know it is the old post, but I also search for the answer to this question. and here what I got:

first of all, binary asymmetric attributes only care the 1 values. for example in measuring the distance of the patient based on their symptom, 1 means the symptom exists and 0 mean it does not exist. Or another example in measuring the distance between 2 documents based on the occurrence of words/term (most of these cases not using binary, but still asymmetric), 0 means the term does not exist in the document.

in those 2 cases, the number of symptom or term can be very huge and may cause a lot of 0 values appear in 1 records. This is because one disease may only have a small number of symptoms or one document may only contain a small number of words/term. If all of these 0 values are used to calculate the distance between two records, the similarity can be huge. example

record 1 :  1 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

record 2 :  0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

assymetric:
d(1,2) = 3/4  --> similarity(1,2) = 1-3/4 = 1/4

IF we consider it as symmertic:
d(1,2) = 3/20 --> similarity(1,2) = 1-3/20 = 17/20 (large similarity)

that is why the attributes with the same 0 values between two records are not included in the distance calculation.


Default value of learning rate in adam optimizer - Keras I am working on a image classification problem using Transfer Learning.

Parameters used given below:
Adam optimizer with learning rate - 0.0001

adamOpti = Adam(lr = 0.0001)
model.compile(optimizer = adamOpti, loss = "categorical_crossentropy, metrics = ["accuracy"])

For testing I used adam optimizer without explicitly specifying any parameter (default value lr = 0.001).

With the default value of learning rate the accuracy of training and validation got stuck at around 50%.
And when I use learning rate = 0.0001 in the first epoch itself I could see that the accuracy is going to 90%.

Could you please help me understand
1. why with lower value of learning rate the accuracy is increasing rapidly?
2. Also which of the above used learning rate is the better?
3. How could I make use of decaying learning rate in Keras?

Thank you

Learning rate is a very important hyperparameter, and often requires some experimentation.  There are some good Related questions here, make sure to check those out.

With too large a learning rate, you might bounce around an optimum, or you might start off by sling-shooting out to a part of the parameter space where the gradients vanish.  (With too small, you might take too long to converge to an optimum, or you might find a poor local optimum.  These effects are lessened by adams momentum effect.)
That said, the best learning rate generally depends on the problem.  Presumably the default was chosen pretty well for general use, but your finding isn't surprising.
There is a parameter decay for simple decay, or you can set up a callback for step decays.


CDF with a massive amount of data I am trying to plot a CDF of 7 files I have. Each file looks like this:

1
2
1.5
2.3

and so on. The problem is about the sizes of the files, which are:

862M
1,8G
2,4G
18G
2,0G
1,8G

I have put up a simple R script which simply loads the files and plot them. When I generate fake files of smaller size, the script works fine. However, it is running since three days and it hasn't produced any output using the full files.

The script is this:

library(ggplot2)
data <- read.table('file1.csv')
data$g = "G1"
data2 <- read.table('file2.csv')
data2$g = "G2"
data3 <- read.table('file3.csv')
data4$g = "G3"
data4 <- read.table('file4.csv')
data4$g = "G4"
data5 <- read.table('file5.csv')
data5$g = "G5"
data6 <- read.table('file56.csv')
data6$g = "G6"
data7 <- read.table('file7.csv')
data7$g = "G7"

dftotal = rbind(data,data2)
dftotal = rbind(dftotal,data3)
dftotal = rbind(dftotal,data4)
dftotal = rbind(dftotal,data5)
dftotal = rbind(dftotal,data6)
dftotal = rbind(dftotal,data7)

gp <- ggplot(data = dftotal, aes(x = V1), group = factor(g)) + stat_ecdf()
ggsave('cdf.eps',gp)

Does anyone knows a more efficient way to do this?

My estimate is that there are around 5.4b numbers in your dataset, all in the range of 0-50 according to your comment. I doubt there are much faster ways to do this on all the data than you are currently doing. However if you would just take every file and get random samples of 0.1-10% of each file and then combine these, you will still get an ecdf that is visually almost exactly the same while reducing memory issues and computing power necessary.


Image Feature Vectors I have downloaded a dataset from Amazon. http://jmcauley.ucsd.edu/data/amazon/ Dataset involves feature vectors of images. There are around 1.5 M feature vectors.

Dataset consists of 10 characters (the product ID), followed by 4096 floats (repeated for every product).

Every product image involves feature vectors with (4096x1) size. Feature vectors involve float numbers.

What do these float numbers mean?

What I understood is, there are at total 4096 features, and each index of feature vectors indicate a specific feature. The values in feature vectors indicate the frequency of regarding feature in all specific image.

Is it so? Or, if it is not, what might be the right explanation?

Thanks,

The same link shows how these features are extracted, with a deep look into the cited article "Image-based recommendations on styles and substitutes":

  Features are calculated from the original images using the
  Caffe deep learning framework [11]. In particular, we used a
  Caffe reference model with 5 convolutional layers followed
  by 3 fully-connected layers, which has been pre-trained on
  1.2 million ImageNet (ILSVRC2010) images. We use the
  output of FC7, the second fully-connected layer, which results
  in a feature vector of length F = 4096.

The reference neural network that was mentioned is the BAIR Reference CaffeNet at the Caffe Model Zoo, which is a slightly modified version of AlexNet.

Since the model was trained over ImageNet, which contains a wide variety of photographs of various categories (1000 of them, if I recall correctly), retrieving the neural codes of one of the layers (obtained just by forward propagation) will give you visual features with a fair representation of the images, even if the network was not specifically trained for Amazon's tasks (such as product recommendation). What these values actually mean is not something that tangible: it is the outcome of multiple 2D convolutions and other normalization and regularization functions, the parameters of which were adjusted specifically for classifying photographs from ImageNet.

The FC7 layer has a rectified linear unit activation (ReLU), which means that they are all non-negative numbers (potentially with several zeros). And since it's a fully connected layer that follows several convolutions, there is no intuitive mapping between a feature index and a certain characteristic of the image. You may picture the network as a highly complex function that yields a high-level representation of the image, under the form of a vector of numbers.

See also the paper Neural Codes for Image Retrieval, where the authors retrieve features from a pre-trained neural network in this fashion, for retrieving images in a different image domain.


How should a decision tree handle an attribute that can be anything? Say I have AttributeA that can take values A1, A2, A3, AttributeB that can take values B1, B2, B3, etc. and I know ahead of time that my classification table looks like

AttributeA | AttributeB | AttributeC | Classification

A1 | B1 | anything | Class 1

anything | B2 | anything | Class 2

A3 | B1 | C2 | Class 3

A2 | anything | C3 | Class 4

...

I'm curious how I would modify a decision tree to handle attributes that can take on any value. One idea I had was to change single rules with "anything" into multiple rules where every possible value of that attribute is explicitly stated. For instance, the rule A1 | B1 | anything | Class 1 could be changed into the three rules:

A1 | B1 | C1 | Class 1

A1 | B1 | C2 | Class 1

A1 | B1 | C3 | Class 1

I'm sure this would work, but I'd like to see if there are any existing decision tree implementations that can handle "anything" or "does not matter" entries.

I think your description of anything matches the idea of missing values. Basically by stating that a value could take any value you say that you do not know the value. In standard Breiman description of CART or random forests there is a way to handle missing values. If you use Python stack, however, the implementations does not allow missing values. If that is the case one surrogate would be a different new value for categorical data. For numerical continuous variables I don’t know something better than imputing with average, but it is not quite the same thing.


How to deal with Nominal categorical with label encoding? So if my dataset looks like this:

  names life_style instrument  times
0   sid   creative      piano    1.5
1  aadi   artistic     guitar    1.4
2  aman  traveller       drum    1.1
3   sid   artistic     guitar    1.5
4  aadi   creative       drum    1.4

Now i want to deal with those Nominal categorical variables , Easy and go to approach is use Label encoding , But suppose if i am using sklearn label encoder then:

from sklearn.preprocessing import LabelEncoder
big_data = dataset_pd.apply(LabelEncoder().fit_transform)

which will output:

   names  life_style  instrument  times
0      2           1           2      2
1      0           0           1      1
2      1           2           0      0
3      2           0           1      2
4      0           1           0      1

Now it is converting each column but each column have same  numeric values range from 0 to 5. The instrument variable is now similar to 'names' variable since both will have similar data points, which is certainly not a right approach. I have few questions :

How should i treat those values without loosing information ? better
approach for this type of data points ?
I am thinking to use random forest for this type of data , Any
suggestion on model also will be helpful for me.
If some variable are appearing only once in huge dataset should i
remove those variables ?

Thank you in advance !

Label encoder is better suited for output classes. You are looking to handle categorical variables. The term is "one hot encoding" which means creating a binary column for each unique value of categorical data. Check get_dummies function in Pandas
It is hard to give any suggestion on model to use without knowing the objective, amount of data available, accuracy requirements. You are better off trying multiple models and checking test set errors to see which one works better.
Variables with high degree of missing values or low fill rate can be removed from the dataset.


Tools for nutritional value tracking and visualization I want to track the nutritional value of different flours over time. What I got so far is a
data table like this:

|------------|------------|------------|------------|
|   Flour    |   Date     |   Energy   |   Sugar    |
|------------|------------|------------|------------|
|   type 1   |   03/20    |   310      |   2        |
|   type 2   |   03/20    |   290      |   1.8      |
|   type 1   |   04/20    |   310      |   2.1      |
|   type 2   |   04/20    |   287      |   1.7      |

Over time, I will add more nutritional values for the flours already in the table and
might add more types of flour. I might also want to add additional columns e.g. for protein.
The nutritional values of the different flour types should then be plotted over time
like this:

Energy
|
|     *         *             * type 1
|     #                       # type 2
|               #
|
|-----------------------Date
    03/20     04/20

What are the best tools to implement such a project?
My constraints are that I only want to use free and open-source software which runs under Arch Linux and that that i can export the plots

I would recommend the Python libraries: matplotlib and seaborn for generating and exporting static plots, or Plotly for interactive plots with hover features that you can export to a web browser.