K-means attempts to group distributions into $k$ similar categories based on some metric of nearness. If your dataset contains repeated values this is not a problem. It simply means that you have 2 distributions from which you are drawing instances, and the variance of these distributions is essentially zero.

For example, if you have a dataset from which you want to cluster two species of fish with 2 features: color and length. However, the fish of each species are all clones (ignore other biology stuff i know nothing about like environmental factors). Thus they will all have exactly the same color and the same length in either group. Each measurement of each individual will be exactly the same. This is fine. We simply have 0 variance in each distribution.

Clustering is very well suited for very tight distributions because they are easily separable. However, distributions with high variance will have significant overlap and will thus cause many instances to be wrongfully classified as being a part of the other cluster.

The ideal number of clusters

This value can be calculated by determining the average distance between each instance and its nearest cluster. You will see that once you surpass 2, the average distance will not decrease in any further in this case, which means a single distribution is being split into 2 groups. Thus 2 is the ideal number of clusters.


Choosing between CPU and GPU for training a neural network I've seen discussions about the 'overhead' of a GPU, and that for 'small' networks, it may actually be faster to train on a CPU (or network of CPUs) than a GPU.

What is meant by 'small'?

For example, would a single-layer MLP with 100 hidden units be 'small'?

Does our definition of 'small' change for recurrent architectures?

Are there any other criteria that should be considered when deciding whether to train on CPU or GPU?

EDIT 1:

I just found a blog post (possibly outdated? It's from 2014):

"...Most network card[s] only work with memory that is registered with the CPU and so the GPU to GPU transfer between two nodes would be like this: GPU 1 to CPU 1 to Network Card 1 to Network Card 2 to CPU 2 to GPU 2. What this means is, if one chooses a slow network card then there might be no speedups over a single computer. Even with fast network cards, if the cluster is large, one does not even get speedups from GPUs when compared to CPUs as the GPUs just work too fast for the network cards to keep up with them.

This is the reason why many big companies like Google and Microsoft are using CPU rather than GPU clusters to train their big neural networks. "

So at some point, according to this post, it could have been faster to use CPUs. Is this still the case?

EDIT 2: Yes, that blog post may very well be outdated because:

Now it seems that GPUs within a node are connected via PCIe bus, so communication can happen at about 6GiB/s. (For example: https://www.youtube.com/watch?v=el1iSlP1uOs, about 35 minutes in). The speaker implies that this is faster than going from GPU1 to CPU to GPU2. It would mean the network card is no longer the bottleneck.

Unlike some of the other answers, I would highly advice against always training on GPUs without any second thought. This is driven by the usage of deep learning methods on images and texts, where the data is very rich (e.g. a lot of pixels = a lot of variables) and the model similarly has many millions of parameters. For other domains, this might not be the case.

  What is meant by 'small'? For example, would a single-layer MLP with 100 hidden units be 'small'?

Yes, that is definitely very small by modern standards. Unless you have a GPU suited perfectly for training (e.g. NVIDIA 1080 or NVIDIA Titan), I wouldn't be surprised to find that your CPU was faster.

Note that the complexity of your neural network also depends on your number of input features, not just the number of units in your hidden layer. If your hidden layer has 100 units and each observation in your dataset has 4 input features, then your network is tiny (~400 parameters). If each observation instead has 1M input features as in some medical/biotech contexts, then your network is pretty big in terms of number of parameters. For the remainder of my answer I'm assuming you have quite few input features pr. observation.

One good example I've found of comparing CPU vs. GPU performance was when I trained a poker bot using reinforcement learning. For reinforcement learning you often don't want that many layers in your neural network and we found that we only needed a few layers with few parameters. Moreover, the number of input features was quite low. Initially I trained on a GPU (NVIDIA Titan), but it was taking a long time as reinforcement learning requires a lot of iterations. Luckily, I found that training on my CPU instead made my training go 10x as fast! This is just to say that CPU's can sometimes be better for training.

  Are there any other criteria that should be considered when deciding whether to train on CPU or GPU?

It's important to note that while on a GPU you will always want to fill up the entire GPU memory by increasing your batch size, that is not the case on the CPU. On the CPU an increase in batch size will increase the time pr. batch. Therefore, if it's important for you to have a very large batch size (e.g. due to a very noisy signal), it can be beneficial to use a GPU. I haven't experienced this in practice though and normally small batch sizes are preferred.


Visualization Tools that can show proper branch and merge in Graphs I want help finding the visualization tool can draw similar architecture as given in the image below

Keras visualization produces something similar to this graph. But I'm working in Pytorch. I tried using Netron and Graphviz, both produce graphs that do not show the branching and merging properly.

This is the plot I rendered in Graphviz for another branching and merging architecture.

Here's the python code I used to render the plot

graph {
    graph [rankdir=RL]
    input [label=Input]
    conv1 [label="Conv 1"]
    conv2 [label="Conv 2"]
    conv3 [label="Conv 3"]
    conv4 [label="Conv 4"]
    conv5 [label="Conv 5"]
    conv6 [label="Conv 6"]
    conv7 [label="Conv 7"]
    conv8 [label="Conv 8"]
    conv9 [label="Conv 9"]
    batch1 [label="Batch Norm1"]
    batch2 [label="Batch Norm2"]
    batch3 [label="Batch Norm3"]
    caps1_a [label="Caps 1(a)"]
    caps1_b [label="Caps 1(b)"]
    caps2_a [label="Caps 2(a)"]
    caps2_b [label="Caps 2(b)"]
    caps3_a [label="Caps 3(a)"]
    caps3_b [label="Caps 3(b)"]
    sum1 [label=Sum]
    sum2 [label=Sum]
    sum3 [label=Sum]
    stack [label=Stack]
    sum4 [label=Sum]
    softmax [label=Softmax]
    input -- conv1 [constraint=true]
    conv1 -- conv2 [constraint=false]
    conv2 -- conv3 [constraint=false]
    conv3 -- conv4 [constraint=false]
    conv4 -- conv5 [constraint=false]
    conv5 -- conv6 [constraint=false]
    conv6 -- conv7 [constraint=false]
    conv7 -- conv8 [constraint=false]
    conv8 -- conv9 [constraint=false]
    conv3 -- batch1 [constraint=false]
    conv6 -- batch2 [constraint=false]
    conv9 -- batch3 [constraint=false]
    batch1 -- caps1_a [constraint=false]
    batch2 -- caps2_a [constraint=false]
    batch3 -- caps3_a [constraint=false]
    caps1_a -- caps1_b [constraint=false]
    caps2_a -- caps2_b [constraint=false]
    caps3_a -- caps3_b [constraint=false]
    caps1_b -- sum1 [constraint=false]
    caps2_b -- sum2 [constraint=false]
    caps3_b -- sum3 [constraint=false]
    sum1 -- stack [label=w1 constraint=false]
    sum2 -- stack [label=w2 constraint=false]
    sum3 -- stack [label=w3 constraint=false]
    stack -- sum4 [constraint=false]
    sum4 -- softmax [constraint=false]
    input -- conv2 [label=k1 constraint=false]
    conv2 -- conv4 [label=k2 constraint=false]
    conv4 -- conv6 [label=k3 constraint=false]
    conv6 -- conv8 [label=k4 constraint=false]
}

You could use Tensorboard, via the tensorboardX interface.

That allows you to load models from PyTorch (and Chainer, MXNet etc.) into Tensorboard. This will then show you the full graph interactively.

From the homepage:


How to run XGBoostregressor using reg: tweedie as objective? I installed XGBoost for anaconda on windows 10 based on the instructions provided here. It seems that xgboost 0.6 is already installed. It performs well using "reg:linear". However, if I use "reg:tweedie", this error is reported:

xgboost.core.XGBoostError: b'[23:08:27] src/objective/objective.cc:21: Unknown objective function reg:tweedie'

As I understood, it is a versioning issue. However, it should be noted that before installing the latest version, the old version should be manually removed. Then, try to install the new version.


Approximating multi-variable function with neural network in python I'm trying to use 2-5-1 neural network to approximate function $$x_1 \in[-3,3], x_2\in[-1,3],  f(x_1,x_2)=\sin(2x_1+x_2)$$

I used code from Implementing a flexible neural network with backpropagation from scratch, to avoid using any complex libraries and tried teaching my network to approximate with following data:

# Define dataset
n = 40
np.random.seed(4)
x1_low, x1_up = -3, 3
x2_low, x2_up = -1, 3
x1s = np.random.uniform(x1_low, x1_up, size=n)
x2s = np.random.uniform(x2_low, x2_up, size=n)

Xs = []
for _x1 in x1s:
    for _x2 in x2s:
        Xs.append([_x1, _x2])

Zs = [my_func(_x1, _x2) for _x1, _x2 in Xs]

# Define test data
x1_pred = np.random.uniform(x1_low, x1_up, size=n)
x2_pred = np.random.uniform(x2_low, x2_up, size=n)
Xs_pred = []
for _x1, _x2 in zip(x1_pred, x2_pred):
    Xs_pred.append([_x1, _x2])
actual_ys = [my_func(_x1, _x2) for _x1, _x2 in Xs_pred]

# Train and test neural network
for ee in range(0, 4):
    for e in range(1, 4):
        alpha = e / 10 ** ee
        nn = NeuralNetwork()
        nn.add_layer(Layer(2, 5, 'tanh'))
        nn.add_layer(Layer(5, 1, 'sigmoid'))
        errors = nn.train(Xs, Zs, alpha, 300)
        print('Accuracy: %.2f%%' % (nn.accuracy(nn.predict(Xs_pred), actual_ys) * 100))
        # Plot changes in mse
        plt.plot(errors)
        plt.ylim([0, 1])
        plt.title(str('Changes in MSE - alpha ' + str(alpha)))
        plt.xlabel('Epoch (every 10th)')
        plt.ylabel('MSE')
        plt.show()

But I can't seem to have MSE lower than $0.4$. What can I do here to make it more accurate?

One problem that I see is that notice that sine is a function that takes value from $-1$ to $1$ but sigmoid function takes value from $0$ to $1$.

Hence you are being penalized whenever the sine value takes negative value.

You might like to try to change your last layer to a tanh layer or alternatively, rather than predicting sine directly, predict $\frac{\sin(2x_1+x_2)+1}{2}$ first.

I manage to achieve MSE of $0.228686$ using the tanh modification. Of course, you can still try to tune other parameters and try other stuff to improve the model.

import numpy as np
import matplotlib.pyplot as plt
import math

class Layer:
    """
    Represents a layer (hidden or output) in our neural network.
    """

    def __init__(self, n_input, n_neurons, activation=None, weights=None, bias=None):
        """
        :param int n_input: The input size (coming from the input layer or a previous hidden layer)
        :param int n_neurons: The number of neurons in this layer.
        :param str activation: The activation function to use (if any).
        :param weights: The layer's weights.
        :param bias: The layer's bias.
        """

        self.weights = weights if weights is not None else np.random.rand(n_input, n_neurons)
        self.activation = activation
        self.bias = bias if bias is not None else np.random.rand(n_neurons)
        self.last_activation = None
        self.error = None
        self.delta = None

    def activate(self, x):
        """
        Calculates the dot product of this layer.
        :param x: The input.
        :return: The result.
        """

        r = np.dot(x, self.weights) + self.bias
        self.last_activation = self._apply_activation(r)
        return self.last_activation

    def _apply_activation(self, r):
        """
        Applies the chosen activation function (if any).
        :param r: The normal value.
        :return: The "activated" value.
        """

        # In case no activation function was chosen
        if self.activation is None:
            return r

        # tanh
        if self.activation == 'tanh':
            return np.tanh(r)

        # sigmoid
        if self.activation == 'sigmoid':
            return 1 / (1 + np.exp(-r))

        return r

    def apply_activation_derivative(self, r):
        """
        Applies the derivative of the activation function (if any).
        :param r: The normal value.
        :return: The "derived" value.
        """

        # We use 'r' directly here because its already activated, the only values that
        # are used in this function are the last activations that were saved.

        if self.activation is None:
            return r

        if self.activation == 'tanh':
            return 1 - r ** 2

        if self.activation == 'sigmoid':
            return r * (1 - r)

        return r

class NeuralNetwork:
    """
    Represents a neural network.
    """

    def __init__(self):
        self._layers = []

    def add_layer(self, layer):
        """
        Adds a layer to the neural network.
        :param Layer layer: The layer to add.
        """

        self._layers.append(layer)

    def feed_forward(self, X):
        """
        Feed forward the input through the layers.
        :param X: The input values.
        :return: The result.
        """

        for layer in self._layers:
            X = layer.activate(X)

        return X

    def predict(self, X):
        """
        Predicts a class (or classes).
        :param X: The input values.
        :return: The predictions.
        """

        ff = self.feed_forward(X)

        # One row
        if ff.ndim == 1:
            return np.argmax(ff)

        # Multiple rows
        return np.argmax(ff, axis=1)

    def backpropagation(self, X, y, learning_rate):
        """
        Performs the backward propagation algorithm and updates the layers weights.
        :param X: The input values.
        :param y: The target values.
        :param float learning_rate: The learning rate (between 0 and 1).
        """

        # Feed forward for the output
        output = self.feed_forward(X)

        # Loop over the layers backward
        for i in reversed(range(len(self._layers))):
            layer = self._layers[i]

            # If this is the output layer
            if layer == self._layers[-1]:
                layer.error = y - output
                # The output = layer.last_activation in this case
                layer.delta = layer.error * layer.apply_activation_derivative(output)
            else:
                next_layer = self._layers[i + 1]
                layer.error = np.dot(next_layer.weights, next_layer.delta)
                layer.delta = layer.error * layer.apply_activation_derivative(layer.last_activation)

        # Update the weights
        for i in range(len(self._layers)):
            layer = self._layers[i]
            # The input is either the previous layers output or X itself (for the first hidden layer)
            input_to_use = np.atleast_2d(X if i == 0 else self._layers[i - 1].last_activation)
            layer.weights += layer.delta * input_to_use.T * learning_rate

    def train(self, X, y, learning_rate, max_epochs):
        """
        Trains the neural network using backpropagation.
        :param X: The input values.
        :param y: The target values.
        :param float learning_rate: The learning rate (between 0 and 1).
        :param int max_epochs: The maximum number of epochs (cycles).
        :return: The list of calculated MSE errors.
        """

        mses = []

        for i in range(max_epochs):
            for j in range(len(X)):
                self.backpropagation(X[j], y[j], learning_rate)
            if i % 10 == 0:
                mse = np.mean(np.square(y - nn.feed_forward(X)))
                mses.append(mse)
                print('Epoch: #%s, MSE: %f' % (i, float(mse)))

        return mses

    @staticmethod
    def accuracy(y_pred, y_true):
        """
        Calculates the accuracy between the predicted labels and true labels.
        :param y_pred: The predicted labels.
        :param y_true: The true labels.
        :return: The calculated accuracy.
        """

        return (y_pred == y_true).mean()

def my_func(x1, x2):
    return [math.sin(2*x1+x2)]

n = 40
np.random.seed(4)
x1_low, x1_up = -3, 3
x2_low, x2_up = -1, 3
x1s = np.random.uniform(x1_low, x1_up, size=n)
x2s = np.random.uniform(x2_low, x2_up, size=n)

Xs = []
for _x1 in x1s:
    for _x2 in x2s:
        Xs.append([_x1, _x2])

Zs = [my_func(_x1, _x2) for _x1, _x2 in Xs]

# Define test data
x1_pred = np.random.uniform(x1_low, x1_up, size=n)
x2_pred = np.random.uniform(x2_low, x2_up, size=n)
Xs_pred = []
for _x1, _x2 in zip(x1_pred, x2_pred):
    Xs_pred.append([_x1, _x2])
actual_ys = [my_func(_x1, _x2) for _x1, _x2 in Xs_pred]

# Train and test neural network
alpha = 0.001
nn = NeuralNetwork()
nn.add_layer(Layer(2, 5, 'tanh'))
nn.add_layer(Layer(5, 1, 'tanh'))
errors = nn.train(Xs, Zs, alpha, 300)
print('Accuracy: %.2f%%' % (nn.accuracy(nn.predict(Xs_pred), actual_ys) * 100))


Are Intersection over Union (IoU) scores preferring larger objects? According to the fcn model I implemented and the PASCAL VOC benchmark (here)
I find that objects with larger sizes in an image receive better IoU or AP score in the test set. Why do the IoU scores have a bias towards larger scale objects? Can anyone give an explanation why this could happen？

There are a lot of reasons for this behaviour:

training data: In your training data, there could be more larger objects than smaller objects, so the model didn't have enough information about smaller objects to learn to segment them well.
training procedure: Depending on your training procedure (e.g. loss function), your model will focus more on some type of objects (in your case, larger objects ...)
model architecture: Every model has some advantages, so some models are better in recognizing objects of various sizes. E.g.  convolutional and pooling layers can increase a receptive field of neurons so that they could recognize objects of various sizes more easily. Also, some model architectures are specialized in recognizing models of varying sizes.

Possible directions:

training data: Know your data! :) Inspect the dataset to see what's in. The things to look for: are all classes represented well, are labels (ground truth) good, sizes of objects, ...
training procedure: Try with various loss functions (cross entropy, dice loss, cross entropy + dice loss, focal loss, ...). Also, dataset augmentation techniques are important (e.g. with dataset augmentation you can create objects of various sizes ...).
model architecture: Resnet, Inception, InceptionResnet - those are the backbones that you might want to use for your backbone as they have proved to be good for semantic segmentation.


Can someone explain what batch size is doing in convolutional NNs? I've noticed that the performance of my models vary quite a bit as a function of the batch size, both in terms of the time to converge and (possibly) the amount of overfitting.I thought batch size was simply controlling the number of images sent to the gpu / cpu at any given time, but I can't match that up with this behavior. Can someone explain what batch size controls in the model?  Why the variation in performance? Any good resources?

Sorry for the naive question. This is my first time using these sorts of models, and in the background reading I've done, I've still not fully grasped what this parameter is doing.

Batch size determines how often the weights are updated during training. The smaller the batch size, the more frequent the updates. On the other hand, when the batch size is small, updates are made without evaluating a large portion of the data at hand so at times, the gradient may be moving in a direction other than minimum if that particular batch is not a good representative of the entire dataset. So in a sense, a large batch size helps you get to a lower loss faster compared to a smaller one - it takes relatively larger steps with little noise. In practice, one rarely uses the entire dataset because it takes too much time per iteration. Of course for smaller dataset, this is not a problem.

Here you can find a good discussion of this issue by Andrew Ng.

Having said that, I also vaguely remember a paper discouraging from using too large batches (more than 1024). Smaller batches might wiggle the loss left and right too much before reaching a minimum, but they will get closer to it. And also some discussion about training with very large batches not performing as well on validation sets. So there is a trade-off and the optimal value should be picked by trial and error.


What's the best way to select features independent of the model being used? I am using tensorflow's DNNRegressor to model a multivariate regression problem. I want to form an optimal feature subset from a mixed bag of categorical and continuous features. What would be the best way to proceed? The reason I want this approach to be independent of the model is because I couldn't find much about feature selection/evaluation in direct context of tensorflow.

There is no best way, but if there was, Tensorflow would definitely not be it. There are three ways I can think of, and each of them is worse than the others in some cases (no free lunch):

Measure correlation of variables with the output variable, and take the variables with the highest correlation. This is a rather poor method as it considers linear correlation, and this is not a very good measure of dependence. Instead of this, although a bit more computationally intensive, is to measure the distance correlation between the output variable and all the variables, and select the features with highest distance correlation.
Fit a linear model with L1 penalization (Lasso). Lasso will automatically select variables for you setting the weights of the non important variables to 0. The variables with the nonzero weights are the ones you can select.
Fit a random forest or a gradient boosting, and take the variables with the highest feature importances. This has worked very well for me in practice.

I hope this helps.


What does the number after a machine learning model name mean? I'm not sure if this is off-topic, but I'm posting here anyway.

So I saw lots of machine learning models have like an ID after their names, for example, resnet101, resnet152, densenet201 etc. What exactly do those numbers 101, 152 and 201 mean? And how it's determined?

Sometimes it refers to a version (like windows 10), sometimes it refers to a size of the parameters, like the size of a layer, the number of parameters (like for GPT models), or the size of parameters in memory. They are just names so we can know what we are talking about. I don't think there is a general convention.


What does Prec@1 in fastText mean? In Bag of Tricks for Efficient Text Classification paper which is popular right now, he calculates prec@1 for the datasets in the experimentation segment. What does that mean?

Its "Precision at 1", or how often the highest ranked document is relevant:

http://ir-ratio.blogspot.co.uk/2012/03/precision-at-1-and-reciprocal-rank.html

Suppose you are looking for items about monkeys. Your query engine queries documents for "monkeys" and ranks by relevance. If the highest ranked document is indeed about monkeys, then that's a win for your query algorithm. But if the highest ranked document is ranked 1 because it has the text "Enough of your monkey business" then its a loss, because that's not really about monkeys.

Repeat over a bunch of search terms. The Precision-at-one is then the number of wins over the total number of search terms tried.


Best way to format data for supervised machine learning ranking predictions I'm fairly new to machine learning, but I'm doing my best to learn as much as possible.

I am curious about how predicting athlete performance (runners in particular) in a race of a specific starting lineup. For instance, if RunnerA, RunnerB, RunnerC, and RunnerD are all racing a 400 meter race, I want to best predict whether RunnerA will beat RunnerB based on past race result information (which I have at my disposal). However, I have many cases where RunnerA has never raced against RunnerB; yet I do have data showing RunnerA has beat RunnerC in the past, and RunnerC has beat RunnerB in the past. This logic extends deeper as well. So, it would seem that RunnerA should beat RunnerB, given this information. My real concern is when it gets more complicated than this as I add more features (multiple runners, different distances, etc), and so I'm turing to ML algorithms to help my predictions.

However, I am having difficulty figuring out how to include this in my row data that I can train (after all, correctly formatting data is 99% of proper machine learning), and I am hoping that someone here might have thought along the same lines in the past and might be able to shed some light.

Example:

I am currently trying to include RunnerX-RunnerY past race data by counting all the races that RunnerX and RunnerY have run together and normalizing them on a scale from -1 to 1; -1 indicating RunnerX lost all past races against RunnerY; and +1 indicating that RunnerX has  won all past races against RunnerY; and +1 indicating. And 0 indicating an equal number of wins and losses (or no past races against each other).

For instance, if RunnerA is racing RunnerB, and RunnerA has beat RunnerB in the past, then I want the algorithm to know that (denoted by a +1 on the RunnerB column of row RunnerA); same for vice versa. Taking it another step further, If RunnerA is racing RunnerC (but the two have never raced each other in the past), and RunnerA has beat RunnerD in a past race, and RunnerD has beat RunnerC in a past race, then I want the algorithm to learn that RunnerA should beat RunnerC. I say beat here, but I mean an "average beat" for any RunnerX-RunnerY combinations when data for more than 1 past race is available.

I have set my data up as:

name     track   surface  distance  age    RunnerA   RunnerB   RunnerC   RunnerD
RunnerA  Home    2        400       11     0         1         0         1
RunnerC  Away    2        400       12     0         0         0         -1
RunnerD  Home    2        400       10     0         0         1         0

which shows that RunnerA has beat RunnerB and RunnerD in the past. RunnerC has lost to RunnerD. And RunnerD has beat RunnerC.

The problem:

The problem is that I don't really think this is a correct display of the information for an ML algorithm.

From what I understand, ML data should be row independent. And this data isn't because row 1 (RunnerA) has beat RunnerD, yet the data indicating RunnerD has beat RunnerC is in row 3.

Does anyone have any ideas how I might be able to incorporate this past win-percentage-for-runner-pair-combination data??? I'm totally stuck here. I've read a lot about some algorithms that estimate the win loss by simply totaling win statistics, but those don't say anything about the actual probability of a particular runner to beat another particular runner.

Any pointers would be super helpful.

Thanks!!!

This problem looks like a lot with the problem of ranking college football teams. I have never worked on this ranking problem, but I believe you can borrow some tools used there to build your model.

Here goes a couple of references:

Colley Matrix Rankings - This was one of computer rankings used by the BCS. It is also the only one that shared their methodology.

An example of the Colley Matrix Rankings - An easy example to follow.

The Perron-Frobenius Theorem and the Ranking of Football Teams - A well known reference that presents some ranking methods. This paper also shows how to assess the probability of winning a game based on the rankings of the teams.


Using RNN (LSTM) for predicting one future value of a time series I have been reading several papers, articles and blog posts about RNNs (LSTM specifically) and how we can use them to do time series prediction. In almost all examples and codes I have found, the problem is defined as finding the next x values of a time series based on previous data. What I am trying to solve is the following:

Assuming We have t values of a time series, what would be its value at time t+1?

So using different LSTM packages (deeplearning4j, keras, ...) that are out there, here is what I am doing right now:

Create a LSTM network and fit it to t samples. My network has one input and one output. So as for input I will have the following patterns and I call them train data:

  t_1,t_2

  t_2,t_3

  t_3,t_4

The next step is to use for example t_4 as input and expect t_5 as output then use t_5 as input and expect t_6 as output and so on.
When done with prediction, I use t_5,t_6 to update my model.

My question: Is this the correct way of doing it? If yes, then I have no idea what does batch_size mean and why it is useful.

Note: An alternative that comes to my mind is something similar to examples which generate a sequence of characters, one character at a time. In that case, batch_size would be a series of numbers and I am expecting the next series with the same size and the one value that I'm looking for would be the last number in that series. I am not sure which of the above mentioned approaches are correct and would really appreciate any help in this regard.

Thanks

I recently did a lot of reading and code writing on LSTMs so I'll try to pitch in and answer the question, though I'm not yet familiar with Theano Keras or deeplearning4j. From a quick scour of the Internet it seems that the meaning of batch_size is program-dependent, so it might be a good idea to check the online Help files. From what I gather though, it can refer to a couple of different things:
1) the count of cases fed into the training algorithm for processing before the next examination of inputs and/or outputs.
2) the number of iterations of the training algorithm over the last set of input before the next examination of inputs and/or outputs.
3) the size of sequences fed to the LSTM algorithms.

With that in mind it seems almost certain that Theano and deeplearning4j are referring to #1 and/or #2, given the following links. On the other hand, I've run across journal articles recently, where batch size referred to sequence size, so watch out for a potential mismatch between the software and academic terminology:

☻ This post at CrossValidated indicates that in deeplearning4j and Keras the batch_size should be set to 1 for online learning. So in other words, batch_size controls the number of training cycles before the next input/output check.

☻ In Keras, the batch_size apparently sets the number of training cycles to execute or # of cases to train on before checking the inputs and/or outputs again, according to this Github page.

☻ This Github package also seems to use a different variable to denote sequence length.

☻ From Tony Kanh's answer at this StackOverflow page, in TensorFlow the batch_size determines the size of the output vector. This is probably identical to the number of items processed in each training cycles, since multiplying it by the num_steps and size (probably the sequence size?) determines the dimensions of the output.

☻ For closely related neural nets like Bidirectionals, the batch size is apparently equivalent to the sequence size, not the number of cases fed into the training algorithm. See p. 5, Berglund, et al., "Bidirectional Recurrent Neural Networks as Generative Models," available at the Cornell University Library website without a paywall.

As I said I haven't used Theano and deeplearning4j yet, but I hope that at least provides a starting point to find the correct answer.


Homemade deep learning library: numerical issue with relu activation For the sake of learning the finer details of a deep learning neural network, I have coded my own library with everything (optimizer, layers, activations, cost function) homemade.

It seems to work fine when benchmarking in on the MNIST dataset, and using only sigmoid activation functions.

Unfortunately I seem to get issues when replacing these with relus.

This is what my learning curve looks like for 50 epochs on a training dataset of ~500 examples:

Everything is fine for the first ~8 epochs and then I get a complete collapse on the score of a dummy classifier (~0.1 accuracy). I checked the code of the relu and it seems fine. Here are my forward and backward passes:

def fprop(self, inputs):
    return np.maximum( inputs, 0.)

def bprop(self, inputs, outputs, grads_wrt_outputs):
    derivative = (outputs > 0).astype( float)
    return derivative * grads_wrt_outputs

The culprit seems to be in the numerical stability of the relu. I tried different learning rates and many parameter initializers for the same result. Tanh and sigmoid work properly. Is this a known issue? Is it a consequence of non-continuous derivative of the relu function?

I don't know exactly what the problem is, but maybe you could try checking the value of your gradients and see if they change a lot around the 8th epoch.


What's the complexity of HDBSCAN? I can't find any complexity information about HDBSCAN by google or wiki.
And how about compare to OPTICS?

By googling: HDBSCAN is order of n squared whereas optics is order of n times log(n).


How to Monitor ML Classification Models in production? I've often heard of measures like Population Stability Index and Characteristic Stability Index. I might be mistaken, but these seem to be more applicable towards looking at the changes in univariate distributions and are more linear.

Are there any methods that are more robust in capturing multivariate relationships and interactions?

Yes, PSI and CSI are useful when we want to understand if there is any change/shift in the population type.
But to understand multi collinearity, VIF is what you should look into.
Ref: https://etav.github.io/python/vif_factor_python.html
and if you think you are left with only few variables, then you need to create few additional variables (interaction variables) and where there is a chance that they may not be correlated with others.


Categorical, nominal or continuous variable? I'm trying to understand what level of measurement is best for describing the 'number of rooms in a flat' feature.

First of all, I think it's not a continuous feature, because rational values like 1.42, for example, don't make sense.

To decide whether a feature is categorical or nominal, we should try to find an ordering between values. And here is my question: should we look for an order with respect to the response feature (in my case 'Price of a property')?

We can say a '1-room flat' is cheaper than a '2-room flat', and so on. But that is not always the case. In general, maybe it's true, but there are cases when a '1-room flat' in the city centre is way more expensive than the one further away.

So, a can't decide which representation to choose - categorical or nominal.

I don’t know where you’re getting this particular classification scheme from, (continuous, categorical, nominal), but it’s worth mentioning that this isn’t a great scheme for classifying data types. You’ve already encountered an example of a data type (count data/natural numbers) that doesn’t fit neatly into them.

It is also worth mentioning that a lot of people will use “nominal” and “categorical” interchangeably to describe a discrete data type with no natural ordering. Probably the most widely-accepted term to describe discrete data with a natural ordering is ordinal data.

In this case, "number of rooms in a flat" has an unambiguous ordering. In a very real and intuitive sense, a five-room flat has more rooms than a two-room flat. A one-room flat has fewer rooms than a three-room flat. It makes sense to talk about one flat having a greater or lesser number of rooms than another, and if we want to know if that ordering matters in some way, we need to preserve the ordering in our analysis.


How do I allow user to run a R script without making a website or web application? I have data & a R script that creates a report from the data. I can't expose the data to internet. Also, I cant expose my script to internet / user. But I would like to eliminate myself from the work, and allow couple of users (yeah, only three users that use this script to generate reports, but they do it weekly) to run the script and generate report for them.
I would like the community to suggest me how I can achieve this. Any information helps.

If you work on Linux, you may create a script that loads data and runs R inside and give the users the access to execute the main script. I do not know the details but this may be a possible solution.


some algorithms to identify of a pattern in text data I have few sentences like below in my project(around 25000),

sentence1 = 'Must be able to multi-task in a fast-paced, deadline-driven environment'
sentence2 = 'Strong organisational skills and proactive approach'
sentence3 = 'in this role you will design develop and revise application simulations in alignment with product implementation timelines'
sentence4= 'ensures appropriate cross-referencing between documents within and across qms chapters/topics'

From these sentences, I am trying to determine what functionality is being expected from the person or like what person has to do. I need to identify those keywords. I can use any ML techniques or any other method to determine the keywords,

for example in

sentence1 = [multi-task, fast paced, deadline-driven environment]
senetence3 = [design develop and revise application simulations]

I tried to identify those words with help of POS but that wasn't very useful. Are there any algorithms or ideas or methods that can detect such keywords ?

Since you don't seem to have any annotated data, the best you can do is probably this:

Optional first step: remove stop words (there are many such lists available, for example https://pythonspot.com/nltk-stop-words/)
Calculate the Inverse Document Frequency for every word in the vocabulary. This is intended to measure the importance of a word based on how often it's used, in the sense that a word used in fewer sentences makes it more important for these sentences.
Calculate TF-IDF for every word in a sentence, then rank the words by their TF-IDF weights. In general the top ones are supposed to be the most relevant. Optionally you can decide to select the top N words as keywords.


When should I use normal Q learning over a DQN? From this article here, it says that using a tabular Q function is less scalable than a deep Q network. I assume that this means that the Q table approach works for some environments, but once they become more complex, they aren't as efficient.
For example, the Frozen Lake environment used in the article states that the deep Q network is slower than the Q table. The Frozen Lake environment has a relatively simple environment with 16 states and 4 actions per state. However, in an environment such as a game of snake, there are many more states, making the Q table larger. How should I decide between a Q table and a Deep Q Network?

I haven't worked with reinforcement learning but in general one should judge based on the data (or in this case states). The shift of algorithms from normal ones to deep ones is made when we require more power. So it boils down to your need!
If you can get results with simple algorithms always choose that but if simple algorithms don't cut it you will have to go for the deeper ones.
Not going on the true computational meaning but generally deeper models are more complex and one should always avoid complex models where ever possible.
You also mention "slower". That has a lot of perspective into it so you shouldn't make a final judgement on the models in that aspect!


Word embeddings and punctuation symbols I have a decent understanding of word embeddings (at its core, one can think of a word being converted into a vector of, say, 100 dimensions, and each dimension given a particular value... this allows to do math with the words, also it makes the training sets to be non-sparse...)

But today something came to my mind, what about punctuation symbols such as , . () ? ! ... ?

They do have a huge impact on the meaning of sentences and, like words, the position and context in which they are used is relevant.

So the question is, how should this be modeled? are pretrained sets like GloVe including punctuation symbols? should I simply remove punctuation symbols from text?

Punctuation such as , . () ? ! ... ? are all included in pretrained word vectors such as Glove. However, it is common to find phrases, abbreviations, and misspellings that are not contained within pretrained vectors. What to do with those phrases is highly dependent on the goals of the analysis and context of the text.


Is there any way to get samples in under each leaf of a decision tree in Sklearn ? I have trained decision tree . I also have a graph of the tree (
) . Now i want to see which samples (red circled ones ) are under which leafs . I am using sklearn's implantation .  Is there any way to get the samples under a leaf ? ?

Very good and analytical answers are provided here and here.

For me, the easiest way would be to find the leaves where each sample belongs and then split the dataframe into clusters using iloc, to further analyse the clusters. This can be done using:

clf.apply(iris.data)

or

clf.predict(iris.data)


How to improve accuracy further for forest cover prediction I am doing Forest-Cover-Type-prediction on Kaggle

this is the train and test data

> dim(train)
[1] 15120    56
> dim(test)
[1] 565892     56

So far I have done this :

combined this data comb <- rbind(train,test)

then mapped 40 mutually exclusive Soil_Type columns into one and 4 mutually exclusive Wilderness_Area column into one.

comb$Soil <- apply(comb[grep("Soil_Ty+",colnames(comb))], 1 , function(x){which(x == 1)})
comb$Wilderness <- apply(comb[grep("Wilderness_Are+",colnames(comb))], 1 , function(x){which(x == 1)})

after deleting these 44 columns I got it down to 14 columns.

> dim(comb)
[1] 581012     14
> colnames(comb)
 [1] "Id"                                 "Elevation"                          "Aspect"
 [4] "Slope"                              "Horizontal_Distance_To_Hydrology"   "Vertical_Distance_To_Hydrology"
 [7] "Horizontal_Distance_To_Roadways"    "Hillshade_9am"                      "Hillshade_Noon"
[10] "Hillshade_3pm"                      "Horizontal_Distance_To_Fire_Points" "Cover_Type"
[13] "Soil"                               "Wilderness"

then I separated the data

train <- comb[1:15120,]
test <- comb[15121:581012,]

and ran randomForest on this data

set.seed(415)
fit <- randomForest(Cover_Type ~ .,data=train[-1], importance=TRUE, ntree=2000, na.action = na.omit)
varImpPlot(fit)
predi <- predict(fit,test)

and got an accuracy of 0.70372 on Kaggle. Now I have hit the wall. I tried plotting various variables against the Cover_Type(variable to be predicted) but couldn't do figure out what to do with those.

How do I improve accuracy from here? what is the general approach in such cases?

Couple of suggestions:
Your training set is smaller than your test set? It should be the other way around.

You should also tune the RF hypermeters using the held out set or Cross Validation. The two parameter people tune on RF are ntrees and mtry.


What is the intuitive meaning of "leaf weight" in xgboost I looked through Tianqi Chen's presentation, but I'm struggling to understand the details of what the leaf weights are,  and I would appreciate if someone could help clarify my understanding.

To put the equations into words on the slide "Put into context: Model and Parameters", the predicted value/score (denoted as yhat) is equal to a sum of the K trees of the model, which each maps the attributes to scores. So far so good, I think.

Then in this slide shown below it gives a mock example of a decision tree that calculates the amount that someone likes a computer game X.  (Aside: Isn't this a weird example? Who likes a computer game X by an amount 2? What does that even mean? Why not choose an example that has a more concrete, relatable meaning?)

Now this is where I start to get lost. I can tabulate mock data for the mock example, assuming that the model is perfect so that the weights (w1, w2, w3) are equal to the true value. But even that seems weird: what's the difference between the predicted value/score and the weights?

       x_i: "attributes"          y_i, the true score (not yhat_i, which is the predicted score)
                |                                 |
|¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯|¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯|
| Age <15 (x_0) | is male (x_1) | Amount likes the computer game X  |
| 1             | 1             | 2                                 | (Young boy)
| 1             | 0             | 0.1                               | (Young girl)
| 0             | 1             | -1                                | (Old man)
| 0             | 0             | -1                                | (Old woman)
| 0             | 0             | -1                                | (Young woman, older than 15)

My question is: Can someone please share what would be the ACTUAL function f? I assume it's a vector/matrix, but what are the actual numbers? Then my bonus follow up is how would you compute the f for this mock example? I feel it's such a simple question, but I can't figure out the answer. If someone could break this down for me in nitty gritty detail it would be a huge help. Thanks!

I would read the paper (section 2.2 specifically) for a more concise presentation and formal definition of the terms.

You need to define the loss function $\mathcal{L}$ for your problem. Then $g_i$ and $h_i$ are the first and second derivatives, respectively, of $\mathcal{L}$. The optimal weights are then (equation (5) or slide 27).
$$
w_j^* = -\dfrac{\sum_{i \in I_j}g_i}{\sum_{i \in I_j}h_i + \lambda}
$$
where $\lambda$ is a regularization parameter.

So those weights give you the minimum of (the second order approximation to) $\mathcal{L}$.

Edit:

Since the weights are given in the example the hard work is done (see Figure 1 in the paper). Following the definitions, $f =w_{q(x)}$ and $q(x)$ maps an instance to a leaf node. Putting this together gives:
$$
f = \begin{cases}
      2 & \mbox{age} < 15 \mbox{ and } \mbox{male} \\
      0.1 & \mbox{age} < 15 \mbox{ and } \mbox{not male} \\
      -1 & \mbox{age} \geq 15
   \end{cases}
$$


Mass convert categorical columns in Pandas (not one-hot encoding) I have pandas dataframe with tons of categorical columns, which I am planning to use in decision tree with scikit-learn. I need to convert them to numerical values (not one hot vectors). I can do it with LabelEncoder from scikit-learn. The problem is there are too many of them, and I do not want to convert them manually.

What would be an easy way to automate this process.

If your categorical columns are currently character/object you can use something like this to do each one:

char_cols = df.dtypes.pipe(lambda x: x[x == 'object']).index

for c in char_cols:
    df[c] = pd.factorize(df[c])[0]

If you need to be able to get back to the categories I'd create a dictionary to save the encoding; something like:

char_cols = df.dtypes.pipe(lambda x: x[x == 'object']).index
label_mapping = {}

for c in char_cols:
    df[c], label_mapping[c] = pd.factorize(df[c])

Using Julien's mcve will output:

In [3]: print(df)
Out[3]:
    a   b   c   d
0   0   0   0   0.155463
1   1   1   1   0.496427
2   0   0   2   0.168625
3   2   0   1   0.209681
4   0   2   1   0.661857

In [4]: print(label_mapping)
Out[4]:
{'a': Index(['Var2', 'Var3', 'Var1'], dtype='object'),
 'b': Index(['Var2', 'Var1', 'Var3'], dtype='object'),
 'c': Index(['Var3', 'Var2', 'Var1'], dtype='object')}


SMOTE vs SMOTE-NC for binary classifier with categorical and numeric data I am using Xgboost for classification. My y is 0 or 1 (true or false). I have categorical and numeric features, so theoretically, I need to use SMOTE-NC instead of SMOTE. However, I get better results with SMOTE.
Could anyone explain why this is happening?
Also, if I use some encoder (BinaryEncoder, one hot, etc.) for categorical data, do I need to use SMOTE-NC after encoding, or before?
I copied my example code (x and y is after cleaning, include BinaryEncoder).
_train, X_val, y_train, y_val = train_test_split(x, y, test_size=0.2, random_state=1)

smt = SMOTE()
X_resampled, y_resampled = smt.fit_resample(X_train, y_train)

params_model1 = {
    'booster': ['dart', 'gbtree', 'gblinear'],
    'learning_rate': [0.001, 0.01, 0.05, 0.1],
    'min_child_weight': [1, 5, 10, 15, 20],
    'gamma': [0, 0.5, 1, 1.5, 2, 5],
    'subsample': [0.6, 0.8, 1.0],
    'colsample_bytree': [0.6, 0.8, 1.0],
    'max_depth': [3, 4, 5, 6, 7, 8],
    'max_delta_step': [0, 1, 2, 3, 5, 10],
    'base_score': [0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65],
    'reg_alpha': [0, 0.5, 1, 1.5, 2],
    'reg_lambda': [0, 0.5, 1, 1.5, 2],
    'n_estimators': [100, 200, 500]
}

skf = StratifiedKFold(n_splits=3, shuffle=True, random_state=1001)

xgb = XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
                    colsample_bynode=1, colsample_bytree=0.3, gamma=1,
                    learning_rate=0.1, max_delta_step=0, max_depth=10,
                    min_child_weight=5, missing=None, n_estimators=1000, n_jobs=1,
                    nthread=None, objective='binary:logistic', random_state=0,
                    reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=None,
                    silent=None, subsample=0.8, verbosity=1)

scoring = 'f1'
rs_xgb = RandomizedSearchCV(xgb, param_distributions=params_model1, n_iter=1,
                            scoring=scoring,  n_jobs=4, cv=skf.split(X_resampled, y_resampled), verbose=3,
                            random_state=1001)

rs_xgb.fit(X_resampled, y_resampled)

refit = rs_xgb.best_estimator_

joblib.dump(refit, 'validator1.pkl')

loaded_xgb = joblib.load('validator1.pkl')

y_predict = loaded_xgb.predict(X_val.as_matrix())

print(confusion_matrix(y_val, y_predict))

print("Final result " + str(f1_score(y_val, y_predict)))

You have to keep in mind that machine learning is still largely an empirical field, full of ad-hoc approaches that, while they happen to work well in most cases, they lack a theoretical explanation as to why they do so.

SMOTE arguably falls under this category; there is absolutely no guarantee (theoretical or otherwise) that SMOTE-NC will work better for your data compared to SMOTE, or even that SMOTE will perform better compared with much simpler approaches, like oversampling/undersampling. Quoting from section 6.1 on SMOTE-NC of the original SMOTE paper (emphasis added):

  SMOTE-NC with the Adult dataset differs from our typical result: it performs worse than plain under-sampling based on AUC. [...] even SMOTE with only continuous features applied to the Adult dataset, does not achieve any better performance than plain under-sampling.

The authors proceed to offer some possible explanations as to why they see such not-typical performance with SMOTE/SMOTE-NC on the said dataset, but as you will see this has to do with a deep focus on the dataset itself and its characteristics, and it is itself rather empirical in nature and hardly "theoretical".

Bottom line: there is not really much to be explained here regarding your question; any further detail will require going deep with the specific characteristics of your dataset, which of course is not possible here. But I would suggest to not bother, and continue guided by your experimental results, rather than by any (practically non-existent) theory on the subject...


How to extract the positions of employee from raw text I have raw text like "Mr John Fullerton is Chief Executive Officer
and Managing Director of Australian Rail Track
Corporation Ltd, and was appointed to the
position in February 2011."

I easily identified name of PERSON(John Fullerton) by using Stanford NER, now i want to extract the positions(Chief Executive Officer). Stanford NER treat Chief Executive Officer as 'O', so which method should i use here?

Have a look at this to see how a Knowledge Base can be built. I would say the best way is to build your own Knowledge Base based on your corpus if you have enough data.

The idea (simplified) for starting is to have patterns such as "NAME is POSITION" by seeing some of data. Through this you find many names and positions. Then you extract new patterns from them and scan whole data again. Do it until no new pattern shows up. Then you end up with lots of is-a relations which is what you need.

For extracting positions you may need sentence segmentation. Have a look at slide 56 onward to get an overview.

PS: If you do not have enough data, you may crawl web and get texts and try to make your lexical databases. Target websites with more probability of having such a context.

Hope it helps. Good Luck!


Confusion regarding the Working mechanism of Activation function For binary classification irrespective of the model used, the sigmoid function is a good choice for output layer because the actual output value ‘Y’ is either 0 or 1 so it makes sense for predicted output value to be a number between 0 and 1.

My confusion is that is there a binary step function in the output layer which squashes the values of the linear combination of weights and and inputs to 0 or 1? Does classification means always applying a thresholding function on top of a linear or non-linear function which is in the hidden layer?

Say the predicted output value is 0.75 and actual Y is 0. Then, how is 0.75 converted to 1? The loss function would calculate the error as actual - predicted = 0-0.75 = -0.75

Can somebody please explain the math or point out some links where the working steps are shown? Thank you.

I think following picture by Rubens Zimbres can express how this process will be done except loss function part:

  Is there a binary step function in the output layer which squashes the values of the linear combination of weights and and inputs to 0 or 1?

Answer : Yes if you consider following example shows how Sigmoid squishes values to be between 0 and 1:

  Does classification means always applying a thresholding function on top of a linear or non-linear function which is in the hidden layer?

Answer: It does.

  Say the predicted output value is 0.75 and actual Y is 0. Then, how is 0.75 converted to 1? The loss function would calculate the error as actual - predicted = 0-0.75 = -0.75

Answer: Loss function and calculation of error for classification-based NN is not just by computation of  $Y_Actual - Y_prediction$

but It is used normally by combination of Softmax activation function and cross-entropy as follow:

See the whole training process Multinomial Logistic Classification $D(S(wx+b),L)$


Can we use machine learning to generate a text output based on the input strings Problem : Generate a text output based on input strings which will be combined using a number of rules.

Example :

    Feature1               Feature2                    O/P

Rule 1  Enum_Domain           Priority          /Enum_Domain/Priority

Rule 2 Enum_Domain.EnumData   Name              /Enum_Domain/EnumData/Name

Rule 1  Trunkgroup            Gateway            /Trunkgroup/Gateway

Rule 2  GatewayGrp.Gateway    IP                /GatewayGrp/Gateway/IP

This is a simple programming problem, but is there any machine learning algorithm that can learn these rules and generate the output based on the two inputs.

Yes, sequence 2 sequence models attempt to do this. This can be used in a number of domains, from typo fixing to machine translation. They are encoder -> decoder based, which means you have a part that encodes your input and then a decoder that generates a new sequence based on this encoding (and usually some attention). In this case your encoder would likely be two recurrent neural networks of which the output would be concatenated and then a decoder that takes this concatenated output and turns this into a new sequence. If you want to use attention you need to adapt the standard attention a bit because you have two textual inputs, but if you understand how it works this would not be too difficult to adapt.


Justification for values used in backpropagation I'm learning the method for backpropagation in adjusting weights. A generalization of a formula used to determine the change made to a respective weight is

where  is the rate the total error changes as the i-th weight changes. I get why this value is interesting in the context of what we're trying to accomplish with backpropagation, but I don't totally understand why this is a good value to use when determining how to adjust the new weight, particularly when there's multiple weights influencing the total error. Why is this a "good" number to use to when changing our weight?

The objective of back-propagation is to isolate the effect of each weight on the total error. Once the effect is isolated, each weight can be changed individually such the total error is minimized.

During back-propagation, the effects of other weights are automatically isolated. Take, for example, the function:

$$
E_{total}(x) = w_1x + w_2x^2 \\
$$

Then

$$
dE_{total}(x)/dw_1 = d(w_1x)/dw_1 + d(w_2x^2)/dw_1 \\
dE_{total}(x)/dw_1 =x + 0
$$

So, you see, back-propagation ignores the effects of other weights by definition of differentiation. When you back-propagate with respect to $w_1$, all other $w$s are considered constant.

$dE_{total}/dw_i$ is the change in error as you change a particular weight $w_i$ and only that weight. So it makes sense that you would want to change $w_i$ in the opposite direction of that change i.e. $-dE_{total}/dw_i$.


Efficacy of model depends on scaling? I'm trying to train a model to distinguish between two kinds of time signals, those with RTS noise, and those with only white noise.

I have a simple 1D CNN that works well (92% accuracy) with one training set, but turns into a complete coin flip with another. To the eye these sets are very similar. One was created using real signals, the other with simulated signals. The only real difference to me is the mean magnitude. Is there a reason the model fails so reliably with this second set? Do I need to normalize the data somehow?

from keras.models import Sequential
from keras.layers import Dense, Dropout,Activation
from keras.layers import Embedding
from keras.layers import Conv1D, GlobalAveragePooling1D, MaxPooling1D,
Flatten, LSTM
import numpy as np

x_test = np.load('C:/Users/Ben WORK ONLY/Desktop/GH repos/RTS ML detect
beta/x_test.npy')
x_train = np.load('C:/Users/Ben WORK ONLY/Desktop/GH repos/RTS ML detect
beta/x_train.npy')
y_test = np.load('C:/Users/Ben WORK ONLY/Desktop/GH repos/RTS ML detect
beta/y_test.npy')
y_train = np.load('C:/Users/Ben WORK ONLY/Desktop/GH repos/RTS ML detect
beta/y_train.npy')
X_train = np.expand_dims(x_train, axis=2)
X_test = np.expand_dims(x_test, axis=2)

model = Sequential()
model.add(Conv1D(32, 12, activation='relu', input_shape=(1500, 1)))
model.add(MaxPooling1D(3))
model.add(Conv1D(64, 12, activation='relu'))

model.add(MaxPooling1D(3))
model.add(Conv1D(128, 12, activation='relu'))
model.add(GlobalAveragePooling1D())
model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))

model.compile(loss='binary_crossentropy',
          optimizer='rmsprop',
          metrics=['accuracy'])

model.fit(X_train, y_train, batch_size=16, epochs=5)
score = model.evaluate(X_test, y_test, batch_size=16)

#model.save('C:/Users/Ben WORK ONLY/Desktop/GH repos/RTS ML detect
beta/CNNlin_model.h5')

The shapes of the two train/test sets:

The answer to your question is: yes, efficacy of a model depends on scaling. It's very important to scale your variables in the right range and combine them with the right activation function.

The reason is the following: the power of Neural Networks is due to the fact that they can learn any non-linear regularity of your data. This depends on the use of non-linear activation functions (tanh, ReLU, ELU, you name it). However, most of activation functions tend to behave in a non-linear way only around zero. Take the plot of a ReLU, for example. If you move further away from zero (in both directions) the function becomes very "linear" (i.e. its derivative is a constant).

All common activation functions tend to behave like this: non-linear (i.e. very powerful) in the locality of zero, and very linear (or flat) further away from zero. That is why all data are usually scaled in the [0, 1] or in the [-1, 1] range. In this way, activation functions can give their best, and Neural Networks can learn all the most complex patterns in your data.

When you work with CNNs, for example, most of pixel data come in the [0, 255] range. This is very bad for all activation functions, since between 0 and 255 pretty much any of them will look almost completely linear. In this way, your CNN wouldn't be able to learn much.


Why does my random forest classifier predicts one class more often? I have a random forest classifier that predicts 0 class about twice as often as class 1. It also predicts class 0 with higher probabilities than class 1.
It is not a imbalanced dataset.
I tried setting class 1 weight to 100 and it seems to solve the problem, though I suppose it's not a correct solution :D
K-NN  gives the same problem.
Since I changed y from 0 and 1 to B and A it started to predict second class more frequently. So can the problem be somehow connected to data type?
Code:
dataset = pd.read_csv('regtraining.csv')
X = dataset.iloc[:, :-5].values
y = dataset.iloc[:, 50].values

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2)
classifier = RandomForestClassifier()
classifier.fit(X_train, y_train)

y_pred = classifier.predict(X_test)

print(confusion_matrix(y_test, y_pred))

When I try multiple random states of train test split, one of predicted classes is always predicted much more frequently.
Edit: After some research I think the random forest splits so that the classes are predicted in an alphabetic order (A, B values for y give more of A but B, A values for y give more of B).
Thanks for your answers, I am new to machine learning :D

While prediction, If you have more date relevant to class 0 obviously this is the expected one.
But if your observation is like, the model predicts more data with class 0 as incorrect, then you model is kind of over-fitted.
To avoid these,

57% and 43% is not kind of class imbalanced, but still you use
class_weight with model.

Feature Engineering helps a lot with these kind of situations.


How does XGBoost use softmax as an objective function? I'm quite used to seeing functions like log-loss, RMSE, cross entropy as objective functions and it's easy to imagine why minimizing these would give us the best model. What's difficult to imagine is how XGBoost uses softmax, a function used to normalize the logits, as a cost function. As mentioned in the docs here.

How can a softmax function be minimized?

It's not softmax to be minimized, but crossentropy loss function, which is based on softmax. Crossentropy is calculated on a softmax output, that's why they are a standard couple in ML. Tree-based classifiers find "cuts", or portions of the variables' space in a way that minimizes the entropy of a dataset.

If you want to explore the relationship between softmax and crossentropy further, you can start with the nice explanations provided here. If you want to dig deeper, you can find a very detailed and technical explanation here.


Why is 10000 used as the denominator in Positional Encodings in the Transformer Model? I was working through the All you need is Attention paper, and while the motivation of positional encodings makes sense and the other stackexchange answers filled me in on the motivations of the structure of it, I still don't understand why $1/10000$ was used as the scaling factor for the $pos$ of a word. Why was this number chosen?

Amirhossein's blog post explains the intuition for positional encoding very well.
My takeaway from the blog is that: Consider just a pair of sinusoids (sine and cosine). Suppose we are within 1 full cycle (e.g. 0 to 2pi), the resulting encoding is basically guaranteed to be unique. I.e. there is a 1-to-1 mapping from real numbers (1, 1.5, 2, 2.34,etc.), x, to the pair of (sin(x), cos(x)). Thus, encoded to this 2-tuple vector.
Therefore, the purpose of the 10000 is probably just to make sure the full cycle is extremely large.
If you plot sin(1/10000*x), you will observe that to complete 1 full cycle, x >50k. This is more than sufficient to encode the words which would probably have <1k words.
Note that according to the original equations, the 1st pair of sinusoids is just sin(pos) and cos(pos). In this case, the positional encoding is arguably not unique. i.e. There may be two positions (note: they are integers) that has the same encoding, though this may only occur if we have a very long sentence such that the positions have exactly the same encoding. Though one may argue that since pi is irrational, it is unlikely any integer positions will have same encoding..


Distribution of Regression Residuals: Is this a normal distribution? I've created a histogram as well as a QQPlot from the residuals of my Regression Model:
Mean: 0.35
Standard Deviation: 18.14

Judging from these plots, is it okay to say that my residuals are normally distributed?
Or what else can I draw from these plots?
Update: Created the Histogram using
ns.distplot(x, hist=True)

Here's the result:

You can perform a statistical test to confirm your data is normally distributed
Try:
from scipy import stats

np.random.seed(42)
x = np.random.normal(2, 1, size=1000)
k2, p = stats.normaltest(x)
alpha = 0.001
print("p = {:g}".format(p))

if p < alpha:  # null hypothesis: x comes from a normal distribution
    print("The null hypothesis can be rejected")
else:
    print("The null hypothesis cannot be rejected")

This function tests the null hypothesis that a sample comes from a normal distribution. It is based on D’Agostino and Pearson’s test that combines skew and kurtosis to produce an omnibus test of normality.


contextual bandits for online learning Which of the algorithms in the current literature for contextual bandits can be implemented for online learning and which ones can't? I'd really appreciate it if someone could provide a link to papers too! Thanks for the help!

My answer can only be considered partial, I've not compiled a list, but I believe all algorithms implemented here, are, well, implemented for both offline and online mode. This one, can also be implemented for online mode.

Not trying to imply you should use that implementation, but this is kind of a living proof transcending deducing very analytically from articles. The thing to understand, is that certain CB algorithms are paired with rather benign algorithms for using offline-accumulated data for training them, in turn paired with mathematical proofs that the loss they incur in that offline training is a good predictor for the loss they'll incur in online mode (if the real world is still 'sufficiently similar' to the one logged from).

Some (other than mentioned above) algorithms may be only applicable to offline training, or at least I'm not aware of a theoretical refutation, that an algorithm may offline-train better in a way precluding direct use for online learning with the same algorithm. But many algorithms are encoded in software only for offline evaluations, as a lot of research dwells on offline. So I think it's a good question!

I think you should certainly email an author of any article that seems really helpful to you, to specifically ask them, if the article doesn't make that 100% clear, and they might even point you in rare cases at a solid online implementation! Do note online usage entails more production-readiness software considerations, and might be an extra mile in terms of the quality of the software expected as such ...


Calculating the Number of Parameters of a 2D CNN Layer How can I calculate the number of parameters for a 2D CNN layer?
I usually use the equation: $output \ width= ((W-F+2*P )/S)+1 = (x)$
The same answer will be valid for the output height considering that they have the same dimension. So the output dimension will be (x,x).
However, what if I want to calculate the number of parameters for a 2D CNN layer with filter size 5x5, input channel/depth = 3, and output channel or depth of 8?

When we say the filter size if 5x5 it is for an image with 1 input channel, for three the filter size is 5x5x3 (but at a lot of places this additional info is skipped to make things easy to understand). When you apply a kernel of 5x5x3 to an image the output is just one channel. To get an output of 8 channels you need 8 such kernels. In that case, the number of parameters would be 5x5x3x8.


K-means clustering to separate temperature vertical profiles I have temperature measurements from weather stations in a mountainous region and I want to obtain a vertical profile from these data at any given time.
In a simple case one can just plot all values in a temperature vs height chart and apply a linear/polynomial/fancy regression to obtain a curve and therefore an estimate of the lapse rate dT/dz.
In some cases, though, the weather changes abruptly and temperature will differ widely between, say, the eastern and western parts of the region. In such case the cloud of values in the above-mentioned plot will be quite messy and a regression won't be of help - R^2 will be quite low.
I was thinking of using a clustering algorithm in these cases, to split values into two groups (e.g. west and east), each having its own vertical profile, which would then (ideally) be suitable for applying regression. The algorithm could use information about each value's geographical position (lat-lon), which is hidden in the temp vs height plot, to figure out to which group each value belongs.
Can I achieve something like this using k-means? Any suggestion is welcome.

here is my 2 cents: k-means is an unsupervised method and therefore it's best suited if you wanted to explore the optimal number of clusters the temperature can be split, this could be useful to see that maybe there is more regions other than east or west were the temperature is different like north and south and you could use a silhouette score or elbow curve to find the optimal number of clusters.
But, what you describe here is a supervised classification problem where you already have the labels(east and west) and want to separate the temperature accordingly, depending on the type of data I would recommend a classification algorithm such as k-nearest neighbors
you could use k-means with a fixed number of clusters, in this case 2, but it would not be as effective as a supervised algorithm


How can I perform stratified sampling for multi-label multi-class classification? I am asking this question for few reasons:

The dataset in hand is imbalanced
I used below code

x = dataset[['Message']]
y = dataset[['Label1', 'Label2']]
train_data, test_data = train_test_split(x, test_size = 0.1, stratify=y, random_state = 73)

but the error message that I am getting is The least populated class in y has only 1 member, which is too few. The minimum number of labels for any class cannot be less than 2. I removed those classes where the class count is < 2 in each individual label.

I am not sure why this error is popping.

So, I am thinking to go to implement the stratified sampling myself. Here,  I need help in deciphering the cause of the problem here and the implementation of the stratified sampling in multi-label classification so that it works well for the individual batches too while training.

The error here seems to be because you want train and test data (so two data sets), meaning that each class must be present in each of the data sets. This would mean that each class must have at least two samples. It is a design choice of whoever implemented train_test_split. I guess it might not technically be stratified otherwise.

You can see where it is implemented in the SciKit Learn source code, within class StratifiedShuffleSplit:

classes, y_indices = np.unique(y, return_inverse=True)
n_classes = classes.shape[0]

class_counts = np.bincount(y_indices)

if np.min(class_counts) < 2:
    raise ValueError("The least populated class in y has only 1"
                     " member, which is too few. The minimum"
                     " number of groups for any class cannot"
                     " be less than 2.")

np.unique finds indices of the each the classes in y. Because the option return_inverse=True is passed, it returns an array of indices that will allow full reconstruction of the input array, y. This means, to get the total number of classes that are present, you need to use np.bincount; creating class_counts.

The final check is whether or not class_counts is less than the number of data sets you want to create. If it is, then you cannot create a properly stratified split of your data - so you get an error.

As to how you might create your own version: one way I implemented stratified sampling was to use histograms, more specifically NumPy's histogram function. It worked well for continuous labels (i.e. not discrete classes) - and I was not looking at a multi-label problem, so you might have to adjust my suggestion to allow it to accomodate your needs.

The main idea is to split the labels into bins of a histogram and then randomly sample from those bins, with the option to allow for duplicates. That is really the part that will solve your specific problem of < 2 labels in a class. I realise this doesn't specifically answer your problem, but perhaps it will give you some new ideas.

If duplicates don't make sense or are strictly not allowed in your experiment, then you could think about merging the smaller classes toether in some way, so they will have > 2 labels per class. This might be more useful than deleting them, but whether or not it is feasible  will depend on your data.


Why is it necessary to clean/preprocess data if the unseen real data is bound to consist missing values, outliers etc.? Can it really be called Generalization if we remove/modify data points to suit our model?

It depends. Actually there are research papers finding that neural network can sometimes cope very well with sparse or noisy labels. But basically when you apply any machine learning approach, you want your model to pick up patterns in your data. In order to be a pattern something needs to be repetitive and predictable. Outliers and errors in your data might not adhere to that. They rather break the pattern. It can still work but it is harder to recognize it.

Making a model "learn" outliers/missing values etc. will only work, if these incidences also appear in a pattern which makes them unlikely to be outliers. As a rule of thumb it is therefore a good idea to remove this noise.

Finding patterns in exceptions & errors

As I mentioned some researches try to address the noise issue for example by classifying if there is random ("white") noise or structured noise. They not only predict the target value but also the type of noise and then try to make another prediction based on both pieces of information. So there are some active approaches to address noise but usually this is due to the fact that it would be to cumbersome or even impossible to remove the noise.


Classification of Colors using Machine Learning from an Array of Pixels From a very high level of possible machine learning methodologies, what method of classification would be good to identify objects or situations based on the color sensor pixel array (i.e., 16x16 array) being fed into Python?

For example, if I want to identify certain fruits based on the shape and color detected in the array:

banana is curved and yellow
apple is circle and red
orange is circle and orange

If I am reading in a constant stream of data in this array form, are there any similar machine learning and classification applications that might be best suited for this type of application?

This is a classical problem for a convolutional neural network. Typically, these work with series of layers using 2d-convolution to find features (e.g., "curved" or "yellow"), then pooling (to borrow from adjacent convolutions), then some combination.

I think the main question is how you are planning to get a good, labeled, dataset (that is, one where a banana is already labeled as "banana", for training your NN).


Speaker clustering/diarization I am working on the problem of speaker clustering. I am using k-means clustering. The ground truth cluster values and k-mean cluster values do not correspond due to different methods of labelling (manually labelled for ground truth and via scikit's k-means function). So I need to find all permutations of matching values and choose the matching values with the highest percentage of match/correctness. I have a ground truth npy array and a labels_by_kmeans npy array as shown below:

Ground truth

Labels by Kmeans

As you can see, arrays 0 and 2 in both tables have 0s and 2s,3s that correspond to each other. However, in array 1 5s seems to corresponds to 0s. I am trying to find the best fit for 0s in ground truth in Labels by k-means.

How can I do this via code?

Implement the

Hungarian matching algorithm

It will find the best mapping.


Propensity Modeling for Retail Marketing: Model Adjustments Based on Consumer Life Changes. I built a predictive model using logistic regression for direct marking creatives. I built a “children’s” model which predicts buying behavior for those that purchase boys, girls and baby apparel.  But one may have a high propensity right now to buy baby clothes but in 12 months, when ones  kid is a toddler, their propensity drops for baby and increases for Girls/Boys.  I’ve been trying to brainstorm ways to take into account that scenario.  My question is, can anyone suggest options for adjustments to the model to based on that additional criteria?

If you have the purchase data for all these customers, this could be one way to approach the problem:

You could either cluster the customers into natural groups based on their most recent purchases or find association rules (the likelihood of the customer purchasing in a particular category based on their most recent purchases).

I'm not sure how well this approach is going to work for you, but it worked for me in my scenario of propensity scoring. So, if it makes sense, try to relate your data to mine.

I had a list of customers and their purchase behaviors. From those purchase behaviors I deduced association rules to determine which customer is likely to purchase in what product category (based on their previous purchases- associated with the previous purchases of the group).

You can also include recency and frequency of the product purchases into the model to decide whether or not to recommend a particular product/offer to the customer for a specific time.

Based on the confidence, support and lift metrics of those rules for each customer, I mapped them to highly likely to purchase in this category with confidence being their propensity score.

Let me know if this works with your data.


How valuable is a categorical feature that has a predominant category over all other ones? Is a categorical feature that has almost equally distributed in it's category more important or the one which one of it's category is predominant over all other ones?
In data prepossessing step for "House Price" competition, I want to decide if Street feature is important or I can drop it from data set in order to avoid over-fitting. So I have plotted a swarm-plot as follows:

How should I interpret this plot? Does it show that the Street plot can be dropped, or it says it is valuable for creating a model?

Feature importance is an empirical question. Train a model with the feature in it. Train another model without the feature in it. Then see which model does better when predicting new data, aka the test dataset. The difference between model performance on train and test datasets is one way to measure overfitting.


is it possible to decide model without any data? Today I just faced a very unique demand from my superior. He asked me whether I can make a model first before we gather the data for training because we don't have any data yet.
I was utterly confused about what to do with this. Did anyone have any suggestion how should I approach modelling without any data at all? Thanks

This is not a very strange situation in real world companies who nowadays want to build data science applications and other data related stuff, but without enough historical data (or none at all).
In this case, defining what a model is might help you, so:

are you/your superior considering only machine learning models? In that case, you need the data to train with
are you also considering a less sophisticated approach like rules-based models first? In that case, you can generate (i.e. programm it) such rules based on business knowledge before going directly into the machine learning pipeline, for which you need data

Another option, which I used once to check some ideas in advance (before having data) is to simulate some data based on the known data distributions which you know you might have in a near future; for instance, you might want to simulate clients ages, clientes accounts amounts... from other banks stored in some open-data platform.
In this case, you can model your data with for instance a kernel density estimator, to generate afterwards some synthetic samples. Below you can find what I onced made with a similar situation, where the orange bars are the open data retrieved with variables similar to what I would eventually have in my company (in this case I needed ages for each marital status, and I found it for a bank of a similar country) and used to generate the kernel density data generator (blue line):


Does it make sense to flatten two different features into a single array When I have N samples of 2 different features (different scale, different meaning etc.), but whose values are scalar numbers of the same format.

Does it make sense to flatten the data into a 1D array of length [N*2] and feed it in this 1D shape into the neural network. Or must I keep them separate (as columns or otherwise)? I will standardize each feature before combining them into the single array.

(I want to do this for convenience reasons i.e. not deal with multidimensional arrays).

The data is input for a classification problem.

I believe it does, for example, if you flatten both features into 1-D array and then concatenate them, they should still preserve most of the information. Because total number of feature vector still remains the same.


About A technique mentioned on ESL to transform an unsupervised into supervised I'm reading page 495 of Elements of Statistical Learning.

Here we discuss a technique for transforming the density estimation problem
into one of supervised function approximation. This forms the basis
for the generalized association rules described in the next section.

Let $g(x)$ be the unknown data probability density to be estimated, and
$g_0(x)$ be a specified probability density function used for reference. For example,
$g_0(x)$ might be the uniform density over the range of the variables.
Other possibilities are discussed below. The data set $x_1, x_2, . . . , x_N$ is presumed
to be an i.i.d. random sample drawn from $g(x)$. A sample of size $N_0$
can be drawn from $g_0(x)$ using Monte Carlo methods. Pooling these two
data sets, and assigning mass $w = N_0/(N + N_0)$ to those drawn from $g(x)$,
and $w_0 = N/(N + N_0)$ to those drawn from $g_0(x)$, results in a random
sample drawn from the mixture density $(g(x) + g_0(x)) /2$. If one assigns
the value $Y = 1$ to each sample point drawn from $g(x)$ and $Y = 0$ those
drawn from $g_0(x)$, then
$$μ(x) = E(Y | x) =\frac{g(x)}
{g(x) + g_0(x)}$$

can be estimated by supervised learning using the combined sample
$(y_1, x_1), (y_2, x_2), . . . , (y_{N+N_0}, x_{N+N_0})$
as training data.

I don't understand what are mass and mixture density. And I want to know
why $μ(x) = E(Y | x) = \frac{g(x)}{g(x) + g_0(x)}$.

Fascinating problem, welcome to the site! In the second edition, this is covered in section 14.2.4 Unsupervised as Supervised Learning. The idea here is to design a classifier (that's supervised learning) between the desired but unknown distribution, and a known reference distribution (14.10), then to use this classifier along with the reference distribution to estimate the unknown distribution at any point (14.12). You can see that when the classifier deems a point to have come from $g$ ($\hat \mu \to 1$), the density $g_0 \hat \mu /(1-\hat \mu)$ increases. The converse is also true. This is what you would expect.

Mass refers to the probability measure of the sample. The idea is to redress by difference in sample sizes by weighting the probability of each sample inversely by its size (or equivalently, the size of the other sample). That's why the density of the mixture is the average of the components. A mixture density is simply a distribution formed by a mixture or sum (of two distributions in this case). Finally, $\mu$ is commonly used in statistics to denote the mean. The conditional mean of $Y$ is simply the probability that $x$ is drawn from $g$ (because the $Y=0$ term does not contribute to the expectation), thus

$$\mu(x) \equiv \mathbb E(Y|x) = \mathrm P(Y=1|x) = \frac{g}{g+g_0}(x)$$

This equation comes from Bayes' rule, after using $g = P(x|Y=1)$ and $g_0 = P(x|Y=0)$. This approach shows the power of using density ratios. In fact, there's a whole book devoted it.


Which colour channel from a TIFF image do I have to use? I'm going to use the following dataset to do semantic segmentation with U-Net network.
LGG Segmentation Dataset
This dataset contains brain MR images together with manual FLAIR abnormality segmentation masks. The images were obtained from The Cancer Imaging Archive (TCIA). They correspond to 110 patients included in The Cancer Genome Atlas (TCGA) lower-grade glioma collection with at least fluid-attenuated inversion recovery (FLAIR) sequence and genomic cluster data available. Tumor genomic clusters and patient data is provided in data.csv file.

I have found that brain images are in TIFF format, and they are in RGB (with three channels). I have opened one with Gimp:

I don't know if there is any special information on each colour channel because this question is strange (I don't understand it).
Which channel do I have to use if I want to use greyscale images? Or maybe I can convert them into greyscale.

From the readme of your kaggle link;

All images are provided in .tif format with 3 channels per image.
For 101 cases, 3 sequences are available, i.e. pre-contrast, FLAIR, post-contrast (in this order of channels).
For 9 cases, post-contrast sequence is missing and for 6 cases, pre-contrast sequence is missing.
Missing sequences are replaced with FLAIR sequence to make all images 3-channel.
Masks are binary, 1-channel images.
They segment FLAIR abnormality present in the FLAIR sequence (available for all cases).

So you could use all three channel.


principles of time series analysis by neural network models I can understand for speech signals, words are correlated and therefore one should have a reason to believe that recurring NNs or LSTMs could predict by running some complex algorithm with weights and an activation function.

But for random digital signals like stock prices, why can the prices be predicted in any way? Tomorrow's price has nothing to do with those in the past.

According to the Efficient-market-hypothesis, you are right: the price movement of today does not affect the price movements of yesterday. However, this hypothesis describes an abstract assumption, which can be very useful for macroeconomic modeling but should not be considered as a comprehensive description of reality (The same applies to other economic assumptions, like that of the Homo economicus for example.).

In fact, there is a lot of criticism regarding the efficient market hypothesis. The Cryptocurrency-Bubbles and the Dot-com bubble are most probably the best examples that price movements in financial markets are not pure random walks in many cases, but interfere with behavioral psychological effects.

From my experience, I can assure you that many indicators are quite consistent: if you're trading in a growing market, the probability of an increase in prices is slightly higher than that of a decline (even though there will be a tipping point sooner or later). In addition, there are other indicators, such as volatility, which are relatively stable. Of course, you could try to manually determine all these indicators and include them as features in an ordinary DNN, but in practice, using an LSTM usually turns out to be easier and more accurate.


Train Test Split for overlapping samples I'm doing a classification of time sequenced sensor data (in Python), where I'm segmenting the sensordata into samples, with a certain window-length (e.g. 3 seconds). However, the samples are also overlapping each other. For example, the first sample is 0s $\rightarrow$ 3s, second is 2.7s $\rightarrow$ 5.7s...

I'm wondering now, how I can do a proper train test split for this samples. Right now, I see two ways to do this:

Split the samples without shuffling them first and drop the sample "at the border", to avoid overlap between train and test data. However, this is not optimal because I would like to have samples from all over the dataset for both the training and the test set. Otherwise I would just test the classifier on a sequence of the data, that might be very different from the sequence used for training.
Shuffle the samples first and then split them into training in test. This would result in an overlap between training and test data and would therefore produce overly optimistic results.

Does anyone have another idea how this could be done?

You can shuffle the samples first with a serial number attached to all the samples. Now, randomly select p% of the samples as test data and leave the (100-p)% samples as the training data. During this random selection, make sure that if a sample of serial #n has been selected as test, then #n-1 & #n+1 are not picked.

In that way,

You are not testing the classifier on a sequence of the data.
Avoid overlap between train & test samples.


How do I read the cord_19_embeddings_2020-07-16.csv from the COVID-19 Open Research Dataset Challenge (CORD-19) on Kaggle? What I'm trying to do
I wanted to use the CORD19 word embeddings csv to map them to certain findings from the rest of the dataset, but as we can see there are no stings in the first column.
The way I know word or sentence embeddings, is what they map a word or a sentence to multiple vectors.
The values in the first column look somewhat like hashes, and they are the main problem I can't use the dataset.
Can somebody give me a pointer on what I'm looking at and how to use them?
I have not found documentation, usage examples or submissions on kaggle that have explained or outlined how this file is supposed to be used.

So, after a lot of digging, I found something in the comment section.
They are document embeddings.

There is a github repo that specifies an API.

Paper on arxiv

Example usage of a similar approach

Relevant Comments from the Kaggle Comment section on the Data Update Log for the CORD19 Dataset:

Comment 1

Comment 2

Examples how to visualize the embeddings in a Jupyter Notebook:
import pandas as pd
from whatlies import Embedding, EmbeddingSet
#Docs: https://rasahq.github.io/whatlies/api/embeddingset/

#transponse dataframe
sample_df = pd.read_csv('data/cord_embeddings_sample.csv', header=None, delimiter=',', index_col=0).T

def to_ems(df):
    ems_dict = {}
    for columnName, columnData in df.iteritems():
        ems_dict.update({str(columnName): Embedding(columnName, columnData)})
    return EmbeddingSet(ems_dict)

ems = to_ems(sample_df.head(10))

ems.plot_correlation()

ems.plot_interactive("ug7v899j","02tnwd4m")

You can even do NLP with the jsons from the dataset and link them to the embeddings via the UUID and SHA from metadata.csv.
Example:
Find words that relate to smoking and color the respective papers:
I created 2 EmbeddingSets where I filtered the embeddings for papers that have smoking-related words in their text body and subtracted their UUIDs from the list.
Both EmbeddingSets can be displayed in the plot.
from whatlies.transformers import Umap

# add 2 embedding sets

emb1 = non_smoking_ems.add_property('set', lambda d: 'non smoking papers')
emb2 = smoking_ems.add_property('set', lambda d: 'smoking papers')

both = emb1.merge(emb2)

#add a clustering transformer that reduces dimensionality (like umap) and visualise them
both.transform(Umap(2)).plot_interactive('umap_0', 'umap_1',color='set', annot=False)


Normalization of possibly not fully representative data I am trying to train a classification RNN model on a sequence of table medical data, but I stuck with the normalization problem. I realized that I cannot simply use MinMaxScaler, because of 3 problems:

outliers, but I could fight them or use RobustScaler instead.
I am not sure that some features in my dataset include all possible ranges. Like I have max(feature_A) == 10, but with the data update, it could become 20. And if I'll preprocess data the same way I will get bad prediction results.
Some features do not have a limit at all and will only increase with time, like how many years patients were treated, for example. I could suppose that this value is !>100years, for example, but if my mean value is 10 years, it will squeeze feature values a lot.

My dataset is pretty large, like millions of observations, thus there is a pretty good chance that it is representative, though. But I am concerned with the small-time range, like all those observations are for the 2 years only, thus, some feature values (like how many years patients were treated) could still grow their bounds.
How should I handle this?
My concerns example:
import pandas as pd
from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()

#### like, initial state
df1 = pd.DataFrame({'A': [1, 2, 3, 4, 5], 'B': [10, 40, 60, 80, 100]})
""" output:
   A    B
0  1   10
1  2   40
2  3   60
3  4   80
4  5  100
"""

scaler.fit_transform(df1)
""" output:
array([[0.        , 0.        ],
       [0.25      , 0.33333333],
       [0.5       , 0.55555556],
       [0.75      , 0.77777778],
       [1.        , 1.        ]])
"""

#### new data arrived, while preprocessing is the same
df2 = pd.DataFrame({'A': [1, 2, 3, 4, 5, 10, 10], 'B': [10, 40, 60, 80, 100, 120, 140]})
""" output:
    A    B
0   1   10
1   2   40
2   3   60
3   4   80
4   5  100
5  10  120
6  10  140
"""

# now 5 in "A" scaled to 0.4 instead of 1, same in "B"
scaler.fit_transform(df2)
""" output:
array([[0.        , 0.        ],
       [0.11111111, 0.23076923],
       [0.22222222, 0.38461538],
       [0.33333333, 0.53846154],
       [0.44444444, 0.69230769],
       [1.        , 0.84615385],
       [1.        , 1.        ]])
"""

PS: I've duplicated this question in different communities (question in ai got most of views):

https://stats.stackexchange.com/questions/523097/normalization-of-possibly-not-fully-representative-data
https://ai.stackexchange.com/questions/27678/normalization-of-possibly-not-fully-representative-data

A friend of mine answered on this question in different social media on different language, I'll post his answer here:

1. scaler should be saved in this case.
You do fit_transform in the example the second time you run it, but you should just transform. Scaler should be "fit" once on data train and not change it afterwards. Then you will get 5 in 1 in both cases. And when 10 appears, scaler will convert it to 2.
2. It's not a fact that this will break the model.
I think everything is clear here, but just in case:
Let us consider the simplest case. Suppose there is a linear dependence y = 3 * x, we don't know about it, but there is a dataset:
X Y
1 3
2 6
3 9

If we assume that the model has the form y = a * x, in the process of learning it turns out that
a = 3 gives the best score and when suddenly x = 100 appears, it has no problem mapping to 300, even though 100 is strongly out of dataset (1,2,3).
3. You can check if this breaks the model now.
Suppose there is a chip that takes values 0, 1, 2, 3, 4, 5, 6, 7, 8 ...
You are afraid that after some time samples with values greater than the current maximum (9, 10, 11, ...) will appear and the model will behave strangely on them. You can try to "simulate" this situation.
Split dataset into train and test so samples with values from 0 to 6 will go into train, and samples with values from 7 to 8 will go into test. If accuracy doesn't fail at the test stage, then we can assume that when all the 0-8 ranks get into train, then the appearance of 9-11 won't break anything.

Also, there is answer for the second question: https://stackoverflow.com/questions/46744076/machine-learning-normalizing-features-with-no-theoretical-maximum-value


NLP Feature creation from phrase matching I'm building a model to classify email content, to decide whether the email should lead to a JIRA ticket being "Raised" or "Not Raised". The problem I am having is the data is highly imbalanced with only around 11% being classed as "Raised". So far, the Random Forest classifier is providing the highest level of accuracy but the True Positive Rate/Recall is sitting at around 40% and I can't seem to increase upon this. I have been provided with a list of phrases that should they be contained in the email content, then in all likelihood a ticket needs raising. Looking for some tips as to the best method to create a new feature based on phrase matching? Has anyone any experience in the best methods for doing this?

The problem with imbalance is that the optimizer can get a very good score by declaring everything 'not raised'. You need to cheat with your training data by removing that incentive. I would suggest a training set that is balanced 50/50 between the classes. Your evaluation set can still be representative, which will give you a sense of how it'll generalize.