XLNET how to deal with text with more than 512 tokens? From what I searched online, XLNET model is pre-trained with 512 tokens, and https://github.com/zihangdai/xlnet/issues/80 , I didn't find too much useful information on that either.
How does XLnet outperform BERT on long text when the max_sequence_length hyperparameter is less than 1024 tokens ?

BERT also has the same limit of 512 tokens.
Normally, for longer sequences, you just truncate to 512 tokens.
The limit is derived from the positional embeddings in the Transformer architecture, for which a maximum length needs to be imposed. The magnitude of such a size is related to the amount of memory needed to handle texts: attention layers scale quadratically with the sequence length, which poses a problem with long texts.
There are new architectures that specifically deal with longer sequences, like Longformer, Reformer, Big Bird and Linformer.


Implementation of GANs I'm working on a particle physics dataset and want to know what libraries I would need to implement GANs and other generative algorithms like  in Python.

You can use Keras, docs found here, to train a GAN. I have provided a GAN implementation in a previous answer here. Here is an alternative version that I have used more recently (found online).

import numpy as np
import time
from tensorflow.examples.tutorials.mnist import input_data

from keras.models import Sequential
from keras.layers import Dense, Activation, Flatten, Reshape
from keras.layers import Conv2D, Conv2DTranspose, UpSampling2D
from keras.layers import LeakyReLU, Dropout
from keras.layers import BatchNormalization
from keras.optimizers import Adam, RMSprop

import matplotlib.pyplot as plt

class ElapsedTimer(object):
    def __init__(self):
        self.start_time = time.time()
    def elapsed(self,sec):
        if sec < 60:
            return str(sec) + " sec"
        elif sec < (60 * 60):
            return str(sec / 60) + " min"
        else:
            return str(sec / (60 * 60)) + " hr"
    def elapsed_time(self):
        print("Elapsed: %s " % self.elapsed(time.time() - self.start_time) )

class DCGAN(object):
    def __init__(self, img_rows=28, img_cols=28, channel=1):

        self.img_rows = img_rows
        self.img_cols = img_cols
        self.channel = channel
        self.D = None   # discriminator
        self.G = None   # generator
        self.AM = None  # adversarial model
        self.DM = None  # discriminator model

    # (W−F+2P)/S+1
    def discriminator(self):
        if self.D:
            return self.D
        self.D = Sequential()
        depth = 64
        dropout = 0.4
        # In: 28 x 28 x 1, depth = 1
        # Out: 14 x 14 x 1, depth=64
        input_shape = (self.img_rows, self.img_cols, self.channel)
        self.D.add(Conv2D(depth*1, 5, strides=2, input_shape=input_shape,\
            padding='same'))
        self.D.add(LeakyReLU(alpha=0.2))
        self.D.add(Dropout(dropout))

        self.D.add(Conv2D(depth*2, 5, strides=2, padding='same'))
        self.D.add(LeakyReLU(alpha=0.2))
        self.D.add(Dropout(dropout))

        self.D.add(Conv2D(depth*4, 5, strides=2, padding='same'))
        self.D.add(LeakyReLU(alpha=0.2))
        self.D.add(Dropout(dropout))

        self.D.add(Conv2D(depth*8, 5, strides=1, padding='same'))
        self.D.add(LeakyReLU(alpha=0.2))
        self.D.add(Dropout(dropout))

        # Out: 1-dim probability
        self.D.add(Flatten())
        self.D.add(Dense(1))
        self.D.add(Activation('sigmoid'))
        self.D.summary()
        return self.D

    def generator(self):
        if self.G:
            return self.G
        self.G = Sequential()
        dropout = 0.4
        depth = 64+64+64+64
        dim = 7
        # In: 100
        # Out: dim x dim x depth
        self.G.add(Dense(dim*dim*depth, input_dim=100))
        self.G.add(BatchNormalization(momentum=0.9))
        self.G.add(Activation('relu'))
        self.G.add(Reshape((dim, dim, depth)))
        self.G.add(Dropout(dropout))

        # In: dim x dim x depth
        # Out: 2*dim x 2*dim x depth/2
        self.G.add(UpSampling2D())
        self.G.add(Conv2DTranspose(int(depth/2), 5, padding='same'))
        self.G.add(BatchNormalization(momentum=0.9))
        self.G.add(Activation('relu'))

        self.G.add(UpSampling2D())
        self.G.add(Conv2DTranspose(int(depth/4), 5, padding='same'))
        self.G.add(BatchNormalization(momentum=0.9))
        self.G.add(Activation('relu'))

        self.G.add(Conv2DTranspose(int(depth/8), 5, padding='same'))
        self.G.add(BatchNormalization(momentum=0.9))
        self.G.add(Activation('relu'))

        # Out: 28 x 28 x 1 grayscale image [0.0,1.0] per pix
        self.G.add(Conv2DTranspose(1, 5, padding='same'))
        self.G.add(Activation('sigmoid'))
        self.G.summary()
        return self.G

    def discriminator_model(self):
        if self.DM:
            return self.DM
        optimizer = RMSprop(lr=0.0002, decay=6e-8)
        self.DM = Sequential()
        self.DM.add(self.discriminator())
        self.DM.compile(loss='binary_crossentropy', optimizer=optimizer,\
            metrics=['accuracy'])
        return self.DM

    def adversarial_model(self):
        if self.AM:
            return self.AM
        optimizer = RMSprop(lr=0.0001, decay=3e-8)
        self.AM = Sequential()
        self.AM.add(self.generator())
        self.AM.add(self.discriminator())
        self.AM.compile(loss='binary_crossentropy', optimizer=optimizer,\
            metrics=['accuracy'])
        return self.AM

class MNIST_DCGAN(object):
    def __init__(self):
        self.img_rows = 28
        self.img_cols = 28
        self.channel = 1

        self.x_train = input_data.read_data_sets("mnist",\
            one_hot=True).train.images
        self.x_train = self.x_train.reshape(-1, self.img_rows,\
            self.img_cols, 1).astype(np.float32)

        self.DCGAN = DCGAN()
        self.discriminator =  self.DCGAN.discriminator_model()
        self.adversarial = self.DCGAN.adversarial_model()
        self.generator = self.DCGAN.generator()

    def train(self, train_steps=2000, batch_size=256, save_interval=0):
        noise_input = None
        if save_interval>0:
            noise_input = np.random.uniform(-1.0, 1.0, size=[16, 100])
        for i in range(train_steps):
            images_train = self.x_train[np.random.randint(0,
                self.x_train.shape[0], size=batch_size), :, :, :]
            noise = np.random.uniform(-1.0, 1.0, size=[batch_size, 100])
            images_fake = self.generator.predict(noise)
            x = np.concatenate((images_train, images_fake))
            y = np.ones([2*batch_size, 1])
            y[batch_size:, :] = 0
            d_loss = self.discriminator.train_on_batch(x, y)

            y = np.ones([batch_size, 1])
            noise = np.random.uniform(-1.0, 1.0, size=[batch_size, 100])
            a_loss = self.adversarial.train_on_batch(noise, y)
            log_mesg = "%d: [D loss: %f, acc: %f]" % (i, d_loss[0], d_loss[1])
            log_mesg = "%s  [A loss: %f, acc: %f]" % (log_mesg, a_loss[0], a_loss[1])
            print(log_mesg)
            if save_interval>0:
                if (i+1)%save_interval==0:
                    self.plot_images(save2file=True, samples=noise_input.shape[0],\
                        noise=noise_input, step=(i+1))

    def plot_images(self, save2file=False, fake=True, samples=16, noise=None, step=0):
        filename = 'mnist.png'
        if fake:
            if noise is None:
                noise = np.random.uniform(-1.0, 1.0, size=[samples, 100])
            else:
                filename = "mnist_%d.png" % step
            images = self.generator.predict(noise)
        else:
            i = np.random.randint(0, self.x_train.shape[0], samples)
            images = self.x_train[i, :, :, :]

        plt.figure(figsize=(10,10))
        for i in range(images.shape[0]):
            plt.subplot(4, 4, i+1)
            image = images[i, :, :, :]
            image = np.reshape(image, [self.img_rows, self.img_cols])
            plt.imshow(image, cmap='gray')
            plt.axis('off')
        plt.tight_layout()
        if save2file:
            plt.savefig(filename)
            plt.close('all')
        else:
            plt.show()

if __name__ == '__main__':
    mnist_dcgan = MNIST_DCGAN()
    timer = ElapsedTimer()
    mnist_dcgan.train(train_steps=10000, batch_size=256, save_interval=500)
    timer.elapsed_time()
    mnist_dcgan.plot_images(fake=True)
    mnist_dcgan.plot_images(fake=False, save2file=True)


How to transform predicted results when doing cross-validation in sklearn? I want to do cross-validation in sklearn like below, but the predicted result of X still need to be transformed to reduce the distance from y. How to do that with adding a custom function?

model = XGBRegressor(n_estimator = 500,
                     learn_rate = 0.05,
                     random_state = 0)

pipeline = Pipeline( steps = [('preprocessor', preprocessor),
                            ('model', model)
                            ])

scores = -1 * cross_val_score(pipeline, X, y,
                            cv = 3,
                            scoring = 'neg_mean_absolute_error',
                            verbose = 0)

There are only 0 and 1 in y, so I want to round and cut decimals off in the result of X, just after the step for predicting of X.

I maybe need other operations on y in the future. For example, fitting the correct result only includes the numbers like 0.5, 1.5, 2.5 ...

Example:

X - The input

ID Column_1 Column_2 Column_3
0    'A'       10     True
1    'A'       20     False
2    'B'       30     True

y - The correct result

ID Result
0  1
1  0
2  1

The current output

ID Result
0   0.899
1  -0.001
2   1.102

The expected output

ID Result
0    1
1    0
2    1

I have posted this question on Stack Overflow but also get no answers.

If your target values are just 0 and 1, you should probably be treating it as classification, and use e.g. XGBClassifier instead of XGBRegressor.
You brought up (originally in the comments on your SO post, now edited into your questions) a scenario when your true values might be limited to 0.5, 1.5, 2.5.  That's unusual enough that I suspect the best answer depends on context, but:

Consider ordinal regression, which treats the target values as ordered but non-numeric.

If the values really are to be treated numerically, then you're left with (I think) doing it manually, which is closest to your original question.  (I'd like to stress one more time that this is odd, and optimizing your loss function may do something unintuitive.)  Since the predictions out of sklearn will be numpy arrays, maybe try numpy.digitize.
Notes: This is useful for post-processing, but won't be easy to incorporate into the model fitting nor a pipeline (since sklearn's predict doesn't behave the same way as transform).  If you really need to process the continuous predictions before scoring inside a model fit, I think you're left with hacking the model's code a bit; and with xgboost, that'd be quite a task.


Advantages of one shot learning over image classification This is a rather conceptual question. From what I've read I gather that one shot learning is useful for use cases in which you don't have datasets of millions of images of employees etc. By a one shot operation, the network or networks will do a similarity measure to determine whether the given image belongs to a bunch of images or not.

However, some of these siamese networks are trained on millions or hundreds of millions of images. So doesn't that defeat the purpose of one shot learning ? Why can't plain image classification + transfer learning be used instead ? For instance, in face recognition systems, for every image - classify, doesn't fall into any of the classes - reject. If this is a legitimate new employee image, add the image to the database and retrain the network using transfer learning.

Does one shot learning remove the need for this retraining ? Can someone help me with these basic intuitions please

In the case of face detection in particular it can get very use case specific:

1a) Let's say I am facebook, and I have a billion possible faces of users to tag.  With the one-shot approach, I am storing a representation of each new face as you suggest, I assume it is something like One-shot Learning with Memory-Augmented Neural Networks. Now I have a problem with a 'memory' that there are billion faces, and that is a lot of computation to do for lookup. The representations learned must increase in size w.r.t to the dataset in order to be able to distinguish between instances, so I get hit again.

1b) With Deepface, which uses the siamese architecture you mention I have the advantage that to classify a new user, I just have to run it on maybe 1k images of friends around the network where that image comes from. I do not have to retrain the network if a new human is found, since after the initial training, there is not such a problem with doing a comparison between images, as the network is doing a one-vs-one, not a one-vs-all query. The network is not learning classes, but representations and the differences between representations that are most indicative of two examples being different classes.

2a) Now let's say I am a modeling agency with 800 employees, and I have 1000 examples per person. In that case, the one shot learning approach is feasible, especially if  you have downloaded a pretrained network, and do the fine-tuning as you suggested. There is close to no cost for any projected growth (the one-shot learning paper trained on a 4million example dataset).

2b) In this case it may be feasible to finetune a network with just a final layer with 1000 logits (800 for employees, and a 200 for growth). This would be totally infeasible for the billion or even million classes (individual humans) - but still probably not advisable.

In short, it depends on how many examples per class you have and the number of classes you have. Many architectures out there only work when you have millions of images for relatively few classes (this applies to all the imagenet entries). Others are designed to learn from 10k examples or less, with 50 examples per class, and yet others are designed for not explicitly using the notion of classes, when even that becomes too big.

For a comprehensive review of face detection check out Deep Face Recognition: A Survey. New on arxiv 4/18/2018


why should i do target encoding within cv loop? i wish to use target encoding, using the category encoders sklearn library. I don't really understand why it is necessary to include this as a step in a sklearn pipeline WITHIN the cross validation loop?
e.g. this example here does so Target encoding with KFold cross-validation - how to transform test set?
my methodoloigy is similar to the one in the link except i do not use any smoothing. my dataset is quite large around 300-500k.
however looking at my result below does this not indicate over fitting?

From the performance point of view, you can one-hot encode the whole dataset before the model fitting and you will get identical results.
Why people do it in the CV loop?
Ideological perspective
In the CV, we set the part of the data aside and run the model on the given dataset from scratch. For example, you may want to normalize the database, but minimum, maximum, mean, and std are dependent on a particular fold, and, therefore, should be done inside the CV loop. Therefore, preprocessing is inside the CV loop, and so one hot encoding belongs here.
Sparse categorical variables
Suppose that the output of your model is a categorical random variable with tons of categories. For example, your output is a single word that described the image, and there are thousands of classes. Instead of one-hot encoding and getting tons of columns of almost all zeros, you may want to use sparse categorical variables, i.e., to map the target or a feature to integer values. For simplicity of the argument, suppose that a categorical feature is encoded as a sparse variable. Then, the values of the encoded column are 0,1,2...,n and for the linear regression, the order matters. In other words, 2 is closer to 0 than to 100. To make sure that your model performs well regardless of encoding order, you need to check cross-validation.


one hot encoding target variable in tree and non tree (knn) methods I am learning about label encoders, one hot encoding etc applied to datasets for classification via KNN and XGBoost type trees. However, I am a bit confused as to whether the target variable should be one hot encoded or not.
In my case, I have a dataframe which has 500x10 size and the last column is the target (for prediction) country column while the rest of the 499 columns contain floating point values. So, an example row:
feature1 ..... fetaure499 target
1.1  ......... 40.1       Germany
6.1  ......... 265.7       USA
8.1  ......... 98,7      Singapore

In this case, what is the optimal way to encode the target for:
[1] KNN
[2] tree based methods.
My relatively naive understanding says that one hot encoding is useful for the feature space while for the target such as this, i.e list of countries labelencoding should be enough. Is this true for this case? Are there any situations I need to aware of when the target has to be label encoded?
Thank you.

The answer depends on the library you are using. If you are using scikit - you wouldn't need to one hot encode the targets. Scikit handles it automatically.
If you were using keras to build a neural network, you might want to use one hot encoded labels because the built in loss function in keras (e.g categorical crossentropy) expects labels to be one hot encoded.
So both approaches work and ate a function of the library or tools you use.


Why do the arrange of clusters change when change the unit I did k-means clustering with 2d data. The x axis represent depth (km). And result was :.
But when I converted km to metr, I got it

Could you give explanation why do this happens?

When you change the scale or range of your features the euclidean distance between pairs of observations in your dataset changes significantly and one feature with the larger range dominates over all the others.
So, in your case, k-means was tricked into believing that the depth in metres is more important than the other feature with the lower scale.
You can fix this by normalizing all features in your dataset to a similar scale so as to reduce the bias towards specific features.


How to obtain original feature names after using one-hot encoding This question is on an implementation aspect of sklearn DecisionTreeClassifier

How do I get the feature names ranked in descending order, from the feature_importances_ returned by the sklearn DecisionTreeClassifier?

The problem is that the input features to the classifier are not the original ones - they are numerical encoded one from pandas DataFrame get_dummies.

For example I take the mushroom dataset from the UCI repository.
Features in the dataset include - cap_shape, cap_surface, cap_color, odor etc.

pandas dataframe getdummies encodes these into multiple features based on values of the original features.
say cap_shape has values b,c,f,k .. after encoding new columns are cap_shape_b, cap_shape_c, cap_shape_f. Similar transformation happens for other features.

After training, the classifier tells me that the top two features are:
cap_shape_b, cap_shape_c, cap_shape_f, odor_a,odor_c,odor_f,odor_l.
From this result thrown by the classifier, I want my function to return the original features, that is, cap_shape and odor.

Consider using the one-hot encoder in category_encoders module for your encoding. It has an inverse_transform method which I believe will transform your one-hot encoded data back to its original form.


Batch Normalization and Input Normalization in CNN I build my CNN on Keras, normally in the ImageDataGenerator I saw the rescale = 1. / 255 used to normalize input data (pixel value) from [0-255] to [0-1]. Then I read about Batch Normalization Layer, I wonder if they are mutually-exclusive or can they be used together in the same network?

I have tried to adapted the BN Layer to my network (BN after every activation Layer) but the loss fluctuates more than without using BN.

Its is basically not really important to rescale your input to [0,1]. Your input data should simply be in the same range. So [0,255] would be also a legit range.
BN should be different to interpret. It helps to keep your activations through the whole network on the same level. It is especially helpfull when we talk about deeper archtitectures.
Actually when you spend some extra time/iterations, your network should also converge without BN.


Train MLP Neural Network on time series data? Newbie question here but I was curious to ask if an MLP Neural type network can be trained on time series data?
The dataset that I have is an electricity type data set from a building power meter and I can find I can train a decent NN model with including a lot of weather data and also a lot of one hot encoding dummy variables for time-of-week. (day, hour, month number, etc.)
I am experimenting in Python with the Tensorflow Keras library and I know the default during the training process randomly shuffles the data. Is this a No-No for a time series type problem where the random shuffle will take out the seasonality from the data? (stationary/non-stationary) The results shuffling the data really aren't that bad at a glance but not-randomly shuffling the data the results for MLP NN are poor, like the model doesn't train well.
I know some other times series forecast methods can include ARIMA, LSTM, etc. but I was curious to inquire if MLP can be used for these purposes too? What I ultimately need is a short term forecast method that can incorporate hourly weather forecast (from a web API) to forecast future hourly building electricity. Any tips greatly appreciated.
Thanks

So I posted this same question on machine learning mastery post about removing trends and seasonality difference transform for time series data. And Jason Brownlee responded to my questions:

Yes, you can use MLP, CNN and LSTM. It requires first converting the
data to a supervised learning problem using a sliding window:

https://machinelearningmastery.com/time-series-forecasting-supervised-learning/

Then evaluating models in a way that respects the temporal ordering of
obs, called walk-forward validation:

https://machinelearningmastery.com/backtest-machine-learning-models-time-series-forecasting/

You can see tens of tutorials and my book on this here:

https://machinelearningmastery.com/start-here/#deep_learning_time_series
MY REPLY TO THIS POST
Ben February 12, 2021 at 8:09 am #

Cool thanks for all the info. So if I used multivariate sliding window
for MLP NN, is Ok when training the model that shuffle_data == True?
or should I not shuffle training data…? Thanks so much!

Jason's REPLY TO MY QUESTION ABOVE
Jason Brownlee February 12, 2021 at 1:36 pm #

Yes, as long as all data in the training dataset is in the past
compared to the test set.


Regression and Classification in one Neural network For example consider object localization problem. Here NN will have 5 ouputs. output[0] will tell probability of object present in image, other 4 will tell bounding box coordinates.
As we see that output[0] has to use classification loss like cross entropy and output[1] to output[4] will have to use regression loss like Mean-squared-error.
So Total loss is something like this:
loss=Cross_entropy(output[0],Y[0])+MSE(output[1:5],Y[1:5]) #Y is true value

Are loss like that backprogationable in vectorised form?
Can I implement that kind of loss in tensorflow?
If yes, how does tensorflow do that? Does it perform differentiation on each element of vector or  matrix instead of whole thing at once?

Yes, these type of loss functions can be optimized using backpropagation, also in Tensorflow.
The value of the loss is a scalar (same as just the cross entropy, or the MSE, otherwise you wouldn't be able to add them), which means that it doesn't really work any different from just optimizing for any other scalar loss function. As long as the operations involved in calculating the loss function are differentiable ("smooth") enough, Tensorflow (or any other framework that does automatic differentation) doesn't care.
Think of it this way: calculating any interesting loss function involves summing up a bunch of terms living in a higher-dimensional vector space. In this case, for some of the directions in this vector space, you apply a different function on them than for others in that vector space. Doesn't really matter to Tensorflow.
What does matter though is how you sum them: since the cross entropy and the MSE are not working on the same type of units (think dimensional analysis), you have to determine some sort of scale between them. You can view this as a hyperparameter, where you can chose how important it is to have the classification correct vs how correct your bounding boxes are.


How to update the posterior belief when we are observing a stream of correlated data from a fixed but unknown data source I want to build a [probabilistic] model that aims to infer the true value of an unknown categorical variable, $y \in \{1,2,..., K\}$.

We have a dataset $(X,y): \mathbb{R}^d\rightarrow \{1,2,..., K\}$ and we can train a classifier that gives $d$-dimensional data, $X$, and estimates the output $y$.

Now, suppose that $X$s are correlated and all coming from a fixed $y$. I mean, we are observing $X^1, X^2,...., X^T,...$ over time and we know that $y$ is fixed for all of them.

For example:

We receive $X^1$ (at time $t=1$) and our previously trained classifier produces a guess about $\hat{y}^1$.
Then, we receive $X^2$, and we again use the classifier to guess $\hat{y}^2$.
Then, we receive $X^3$, and so on.

So, at time $t=T$ we have $\hat{y}^1, \hat{y}^2, ..., \hat{y}^T$.

Now, the question is: How can I make a model to use these estimations ($\hat{y}^1, \hat{y}^2, ..., \hat{y}^T$) and improve my belief about the true $y$ over time, considering that:

dimension $d$ is not small. e.g. $d >50$
data samples, $X$s, are not i.i.d. but all coming from a fixed unknown $y$.
classifier is not optimal (just trained on some available data) and at each round gives an estimate about the $\hat{y}^t$ for the current $X^t$.

I have been reading some materials and came across the following but I am not sure which one is better to investigate more into:

Sequential Hypothesis Testing
Optimal stopping
Sequential probability ratio test
HDI+ROPE decision rule: highest density interval (HDI) region of practical equivalence (ROPE)

Or is there any specific Bayesian framework for it?

I think all of the options you give will yield results for the problem you described. It seems you have a y like a die that yields results which are rolls of the die all unpredictable but still linked from the y and you want to infer the y based on the x values. Like a
 Hidden Markov model.

The part about wanting to accurately make an estimate of y at each x and the x's being correlated make me believe that LSTM technology may be of benefit. If you want to use a NN.

Optimal stopping and Sequential Hypothesis Testing and Sequential probability ratio test and HDI+ROPE will all work for the abstract problem you describe. Until you give us more details about your problem like what specifically you will be working with it is hard to give you concise direction.

If you create a LSTM That takes in an X and makes a guess about which y and train it on x data you have then you would have the predictive model you seek.


Identify patterns leading to error in structured log files I am currently working on parsed log data from IoT devices (log entries are batched every 15 minutes). All info_* / err_* are counts of the related log events:
| machine_id | timestamp           | os_ver | days_since_reboot | cpu_load | info_featA | info_featB | err_badrequests | err_timeoutrequests |
|------------|---------------------|--------|-------------------|----------|------------|------------|-----------------|---------------------|
| 1          | 2020-09-18 12:25:00 | 0.01   | 12                | 0.59     | 12         | 543        | 0               | 0                   |
| 1          | 2020-09-18 12:40:00 | 0.01   | 12                | 0.83     | 1          | 23         | 1               | 0                   |
| ...        | ...                 | ...    | ...               | ...      | ...        | ...        | ...             | ...                 |
| 2          | 2020-10-28 11:40:00 | 0.02   | 2                 | 0.12     | 43         | 54         | 0               | 1                   |
| 2          | 2020-10-28 11:55:00 | 0.02   | 2                 | 0.43     | 32         | 54         | 3               | 5                   |
| 2          | 2020-10-28 12:10:00 | 0.02   | 2                 | 0.23     | 0          | 4          | 0               | 23                  |

I am tasked to find some common patterns that might lead to an error event (e.g an increase in info_featA entries precedes a positive count in err_badrequests). For this reason I've thought to re-encode error columns into a boolean (True if err_* > 0) but I am fairly lost on the right approach to go about sequence analysis or finding the conditions that can lead to an error.
Also, error type is not very important for my problem as I just care about the presence of a positive error count (regardless of its type):
| machine_id | timestamp           | os_ver | days_since_reboot | cpu_load | info_featA | info_featB | has_err |
|------------|---------------------|--------|-------------------|----------|------------|------------|---------|
| 1          | 2020-09-18 12:25:00 | 0.01   | 12                | 0.59     | 12         | 543        | 0       |
| 1          | 2020-09-18 12:40:00 | 0.01   | 12                | 0.83     | 1          | 23         | 1       |
| ...        | ...                 | ...    | ...               | ...      | ...        | ...        | ...     |
| 2          | 2020-10-28 11:40:00 | 0.02   | 2                 | 0.12     | 43         | 54         | 0       |
| 2          | 2020-10-28 11:55:00 | 0.02   | 2                 | 0.43     | 32         | 54         | 1       |
| 2          | 2020-10-28 12:10:00 | 0.02   | 2                 | 0.23     | 0          | 4          | 0       |

At the moment I have created lagged variables (up to 2 hours before) to check if any lagged variable is somewhat correlated with error counts through a correlation matrix. I am pretty sure this is not the way to approach such problem.

The main goal is to identify (if any) some common patterns in the observed metrics that lead to an error.

There are several different ways to model this.
One option is to simplify the problem ignoring time. Fit a machine learning model that find the features that are associated with presence of errors.
Another option is treat this as event-prediction in time series data. In this case, it would be multivariate time series.


Error when checking target: expected Output to have 2 dimensions, but got array with shape (631, 80, 2641) I'm making a Text Chunking program using Bi-LSTM from the model I of the paper "Neural Models for Sequence Chunking". The inputs are sequences of words and the outputs are "B-NP", "I-NP", and "O".

The problem I have is the output. The expected output is 2 dimensions, not 3 dimensions.

The X Train shape is (631, 80) while the Y Train shape is (631, 80, 2641).

I get the problem, but I don't know how to solve it. I've used np.array and reshape on Y Training, but no luck.

I don't have experience using Tensorflow and Keras and this is my first time, so I was hoping you could help me.

# Token Indexing
word2idx = {"PAD": 0, "UNK": 1}
word2idx.update({w: i+2 for i, w in enumerate(vocab_words) if w in vocab_words})
tag2idx = {t: i for i, t in enumerate(vocab_tags)}
idx2tag = {v: k for k, v in iteritems(tag2idx)}

# Padding
X = [[word2idx[w[0]] for w in s] for s in quran_sentences]
X = pad_sequences(maxlen=max_length, sequences=X, padding="post",value=word2idx["PAD"])

y = [[tag2idx[w[1]] for w in s] for s in quran_sentences]
y = pad_sequences(maxlen=max_length, sequences=y, padding="post", value=tag2idx["O"])
y = [to_categorical(i, num_classes=n_tags) for i in y]

# Split Data
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, shuffle=False)

# Parameters
hidden_state_encoder_size = 100

batch_size = 64

training_epoch = 200

embedding_size = 80

# Input
inputs = Input(shape=(max_length,), name="Input")

# Embedding
embed = Embedding(input_dim=n_words+2, output_dim=embedding_size, input_length=max_length, name="Embedding")(inputs)

# Bi-LSTM
encoder = Bidirectional(LSTM(units=hidden_state_encoder_size, return_state=True, name="LSTM"), name="Bi-LSTM")
encoder_outputs, forward_h, forward_c, backward_h, backward_c = encoder(embed)

# Concatenated
state_h = Concatenate(name="Concat_1")([forward_h, backward_h])
state_c = Concatenate(name="Concat_2")([forward_c, backward_c])

# Average
average = Average()([state_h, state_c])

# Outputs
outputs = Dense(num_decoder_tokens, activation="softmax", name="Output")(average)

# Build Model
model = Model(inputs, outputs, name="Seq2Seq Chunking")

# Compile & Training
# Compile
model.compile(optimizer='rmsprop', loss='categorical_crossentropy')

# run training
model.fit(X_tr, np.array(y_tr),
          batch_size=batch_size,
          epochs=training_epoch,
          validation_split=0.1)

Error:

ValueError: Error when checking target: expected Output to have 2 dimensions, but got array with shape (631, 80, 2641)

Alright, so... I got help from someone. He mentions changing LSTM to return_sequences.

# Input
inputs = Input(shape=(max_length,), name="Input")

# Embedding
embed = Embedding(input_dim=n_words+1,
                  output_dim=embedding_size,
                  input_length=max_length,
                  name="Embedding")(inputs)

# Bi-LSTM
encoder = Bidirectional(LSTM(units=hidden_state_encoder_size,
                             return_sequences=True,
                             dropout=dropout_rate,
                             name="LSTM"),
                        name="Bi-LSTM")

encoder_outputs = encoder(embed)

# # Average
# average = Average()([state_h, state_c])

# Outputs
outputs = Dense(n_tags,
                activation="softmax",
                name="Output")(encoder_outputs)

The average layer is still wrong, but it's the close thing. I might have to edit it later.


how to update the pre-trained word2vec model with new train data using genism Hi I have used the genism to load the Spanish  fasttext word2vec model with following code:

Word2Vec = KeyedVectors.load_word2vec_format('../data/raw/wiki.es.vec',
                                             binary=False)

and now I want to train the word2vec with new training sentences.

I know that in the genism can use the following code to train new sentences if the model is loaded:

 model.build_vocab(sentences)  # prepare the model vocabulary
 model.train(sentences, )

But how to update the model if I used the KeyedVectors.load_word2vec to load the model.

Thanks!

You cannot use keyedvectors for resuming training as per Gensim documentation. You will need to save the full model in previous training run to allow resuming training. Keyedvectors remove training related information and other metadata to reduce memory footprint.


Why CNN input images are often square shaped? Why it's so that in convolutional neural networks we generally take the image dimensions of input image to be generally a square? We even do padding to make it happen. Why not different dimensions?

What I understood is that computer computes multiplication and division (by 2) much faster than the rest.

Can someone shed some light on this?

Any link or reference will be appreciated. I already have CS231n notes and lecture slides.

I'm not sure, (also studied the CS231n class), but I'm guessing it's something to do with:

matrix operations,
convention, and/or
control size of image out.

Not a lot of evidence points towards (1), cause we have insane libraries to do this (ATLAS, MKL, or OpenBLAS). (2) makes more sense practically. (3), because you mentioned zero padding (hyperparameter), the CS231n course says this is done to control the size of output from the current CNN layer and control the spatial size so that input (h x w) = output (h x w). Also, I guess it also reduces errors in matrix operations when you have the same image size (h = w) instead of (h > w or h < w).

Most likely, (3) is the predominant reason. This is just my thoughts. Someone please confirm.


How to validate a clustering model without a ground truth? Im dealing with a dataset (text messages about source code comments) that are not labeled. I don't have a assumption about the implicits classes in this dataset. I want to discovery (by clustering) the common hidden patterns shared by the groups of messages. This is a unsupervised learning problem. I was asked how i will  validate this method (patterns discovery, clusters) without a dataset of correct answers to measure the output of the model with the "reality". Im neither a specialist in the field of the messages dataset to manualy inspect and label the data. So, how to approach this question or provide a scientific explanation about the model output? How to prove that the clusters generated by the model are reasonable or correct?

In my opinion there are two ways:

Ask a few experts to assess the quality of the clusters based on a sample (after the clustering has been done, much easier than pre-annotating the whole data especially in the case of clustering)
If the clustering is done in the perspective of using the result in another task, the performance of this other task will reflect the quality of the clustering.

Imho any measure based on the distance between clusters or other technical measure would be a flawed evaluation, because it would depend on the quality of the representation. Such measures might provide some useful indications though, just not a proper evaluation for the task.


Model assessment for EXTREMELY imbalanced dataset I am dealing with an extremely imbalanced dataset, with about 10,000 negative samples for each positive sample.
I am now trying to come up with an adequate measurement of model accuracy but none seem to fit.
Many places recommend the PR curve over the AUC curve for imbalanced datasets (e.g. The most informative curve for imbalance datasets) but it looks like all these recommendations are good for less imbalanced sets.

Since the denominator for precision takes into account the number of false positives, more negative samples in the dataset means smaller precision vale.

I noticed that my PR curve tends to look nice and informative as long as I keep the positive/negative ratio to around 1/10-20, but as more negative samples are taken into account the curve looks less and less like it should.

My question is whether there's a better way to assess model performance for super imbalanced datasets, or maybe I am missing something with my interpretation of the PR curve and its purpose.

I would only look at the precision/recall scores of the undersampled class (the positive class in your case).

Checking the performance on the oversampled class seems quite meaningless, since it is quite easy to achieve very high scores.

Then, balancing the precision/recall with a F-beta score will depend on your specific use case.


State of the art for Object detection/image recognition I was asked to verify the feasibility for solving a particular problem: recognizing for a fashion brand the model of its products.

I have little experience with image recognition in general, I always used Google Vision Api or some pre trained nets from Google/tensorflow like the VGG-16.

I am wondering: Do you think this level of granularity can be achieved?

I guess I should for example train the last layer of a pre-trained network, but I need many images to do that.

Maybe.

Do they have a unique feature on the item  (e.g. a polo player, a red sole) the model can be trained on? I have seen people try identifying clothing model from images via Google's Vision API and it didn't work well. If you are in a fashion category where the business model is to copy designs like H&M or Zara do, you're likely going to have trouble.


Daily new data for my neural network, and I want transfer(?) learning I made my neural network, it is pre-trained for 180 days of data.

It filters the fraud data of credit cards everyday and 1-days new data is comming in.

And I also want after the filtering,

  I want to re-train my AI-model but I just want to use new 1-day
  data only(because training neural network is really time-consuming work).

My AI model is 0(not fraud)/1(fraud) classification model. I want to change my neural net by 1/181.... because the amount of data is just a one day

How I should train the neural network? If I use just 1 day data and run many epoch(time), It will over-fitted... By early stopping, It seems the one-day data train is not.... sufficient......

I think memory like LSTM may need my neural net... What design of neural net is best for my situation??

So, I believe the question you are asking is how to re-train the model on a single day's worth of data, rather than training on all 180 day's worth of data.

It definitely seems reasonable that you have seen overfitting, this can be due to the complexity of the model (related to depth of network).

So changing the model's architecture to suit this classification task is definitely a good idea. For this, without knowing your model architecture, I would look at the overall depth (i.e. number of layers in neural network).

To clarify, LSTMs (like RNNs [Recurrent Neural Networks]) are typically used for sequential data as data  $x_i \in \textbf{x}$, where $0 < i < N$ ($N$ is number of data examples in sequence) is fed sequentially in the model to either output at each time step $i$ or later in the input sequence.

One suggestion from the top of my mind is, since a RNN is simply an normal NN with a recurring hidden layer, we can use transfer learning (which you spoke about). Transfer learning essentially when we pre-train a model on usually a large quantity of (generic) data and then we take this pre-trained model and alter the output section of the network's architecture to suit the classification task (in your case, add a softmax layer to output a probability distribution over the 2 classes). We then train this altered model with different data (in your case your single day's worth of data).
To prevent the model from losing generalisation performance, we typically freeze parameter updates within the pre-trained model's hidden layers and only allow the altered section of the model to update their parameters. This also has the added benefit of reducing time taken to train the model.
So, that would be my suggestion, firstly pre-train your model with the 180 day's worth of data, then alter your pre-trained model to suit this task to classify for a day's worth of day, train the model but only update parameters in the altered section of the model.


What's the difference between transfer learning and feature extraction in CNN? So from what i understand, transfer learning is the fact of training a model on a dataset where you have a lot of data, then keeping most of trained coefficients, and only re-training the last layer on a dataset for a problem where you have a little amount of data
And feature extraction for CNN means that you have a generic model which extracts features from the image, and then you only need to train the last layer to solve a particular problem on images.
So is feature extraction just a particular case, or is there some subtlety i did not understand ?

Transfer learning is, as you said, retraining the final layer of a deep network. Not only is this useful for solving problems with limited training examples, but also when you don't have adequate computing resources to train a network from scratch. Some models have hundreds of millions of parameters, which could take weeks to train if you only have modest equipment. However, if you have sufficient data, adapting weights via transfer learning is not preferable because the features that were extracted from the original training process are unlikely to be ideal for another application.
Feature extraction, in the context of CNNs, is not necessarily an explicit process, rather a sort of high-level product of the training process. Feature extraction refers to the portion of the training process by which a CNN learns to map input space to a latent space that can subsequently be used for classification via the final layer. In other words, the hidden layers learn discriminatory features in the form of weight-adjusted (usually by backpropagating the error) convolutional filters. Thus the term "feature extraction" generally refers to the portion of the training process that occurs before the final layer. So it is not part of transfer learning in which only the last layer is trained.


Auto-scheduling with Machine Learning I am very new to Data Science, but I have an use case which I want to solve.

I want to build a data synchronization scheduler which keeps track of the amount of data sync after every scheduled triggers and auto-adjusts the next schedule.

For example :

Let us suppose I have 3 jobs to execute. Currently, we keep each of
them at 5 minutes interval (say) but this needs to be auto-scheduled.

So let at 10 AM Job 1 got executed and got 10 entries.
At 10 AM job 2 got executed and got 100 entries.
At 10 AM job 3 got executed and got 200 entries.

For such a scenario, job 1 got less stream of data than job 2 and job 3. The auto-scheduler in such a case will auto-adjust the interval and recommend to change the next execution at :

Job 1 - may be 10 min interval
Job 2 - may be 5 min interval
Job 3 - may be 2 min interval.

The scheduler will train itself based on time based historical data as well, for instance if stream of data is more at 10 AM for job 3, it might be less at 1 PM, when Job 1 might have more data. The scheduler would automatically adjust the time and make next schedule of 1 PM at lesser time interval for job 1 than job 3.

Can you suggest me any algorithm which I can follow to support this case ? Or even if you can help me how to proceed in ML, it would help me a lot.

I want to build a data synchronization scheduler which keeps track of the amount of data sync after every scheduled triggers and auto-adjusts the next schedule.

This doesn't immediately strike me as a "use machine learning" problem to be honest. If you just want the scheduler to schedule the next run to be after an amount of time that's determined by the number of records processed in the current batch then that's quite a simple and deterministic formula. You could have the default gap be 20 mins and then do something like:

next gap = 10 * (10 / n)

Where n is the number of records processed in the last run. That would mean in this case:

  So let at 10 AM Job 1 got executed and got 10 entries.

  At 10 AM job 2 got executed and got 100 entries.
  At 10 AM job 3 got executed and got 200 entries.

The next Job 1 would be scheduled at 20 * (20 / 20) = 20 minutes, the next job 2 would be at 20 * (20 / 100) = 4 minutes and the next job 3 would be at 20 * (20 / 200) = 2 minutes.

If you really wanted to use ML for it, I guess I'd suggest using a time series forecasting algorithm like ARIMA or Prophet to predict the number of samples each job will have to process over the next hour or something, and set the next run-time appropriately based on that.


Anomaly detection in multiple parameters I am a newbie to data science with a typical problem. I have a data set with metric1, metric2 and metric3. All these metrics are interdependent on each other. I want to detect anomalies in metric3. Currently, I am using Nupic from numenta.org for my analysis and it doesn't seem to be effective. Is there any ML library which can detect anomalies in multiple parameters?

One way to use both metric1 and metric2 in order to find anomalies in metric3 is to consider residual analysis.

In your case, this would require, creating a predictive model with metric1 and metric2 as the predictors and metric3 as the response variable.

Then, calculate the residuals for metric3 as its predicted value subtracted from its true value. Now, you can report the all members of the lowest decile [or any other percentile] as one kind of an anomaly and all the members of the highest decile [or any other percentile]   as another kind of an anomaly.


How to get used to matricial/vectorial operations? I came to data science/machine learning from another background in computer science and i feel that i'm lacking of experience with matricial/vectorial operations.
Python or Matlab, for instance, provide awesome features like numpy to easily manipulate tabular data. However, the first solution that comes to my mind when coding is for or while loops. I've faced situations where my code could be easily reduced to 1 line or  so with numpy.
I'd like to know if there are good lectures/books you can recommend to change this way of thinking when it comes to problem solving/coding approaches.
Thanks in advance.

tl;dr: refresh your linear algebra and check for built-in functions.
In addition to brevity/elegance, vectorization often dramatically improves performance. Tools like MATLAB and numpy use high-level vectorized wrappers for low-level optimized implementations, which in the end are composed of your traditional loop operations. Where MATLAB and Python are interpreted (nuances exist, but are irrelevant here), the low-level subroutines that they wrap are obviously much faster. Also, tools that they wrap, like BLAS (Basic Linear Algebra Subprograms), are so heavily optimized that it'd be difficult for someone write a low-level subroutine that matches or exceeds the performance achieved with such libraries; many people over many decades worked to optimize them!
Considering problems from a math-oriented perspective helps with vectorization. For example, say you wanted to find the $L_p$ norm of a vector $\vec{x}$. A traditional CS program would look something like:
norm = 0
for i in vec_x:
    norm = norm + pow(i,p)
norm = pow(norm,1/p)

but a mathematician would simply write $||\vec{x}||_p$. First, you should check to see if a built-in function exists to perform your task, as a mathematician might. Then you should look to vectorize using math operators such as matrix multiplication, element-wise operations, etc., which these high-level tools implement. Going back to the $L_p$ example, you could write vectorized code (in MATLAB): (sum(x.^p))^(1/p), but it'd be even better to write norm(x,p). Knowing that such functions exist is a combination of familiarity with the tool and Googling.


Does Tensorflow uses vectorization in its operators I'm kinda new to tensorflow and just wanted to know if it already performs vectorization in its operators like multiplying matrices and so on.
And if not if its doable.

As an example let's say we have 2 matrices and we want to multiply them. Instead of for loops to do the operation I'm looking for a vectorized way to do this.

Thanks in advance.

Yes, tensorflow works with tensors, which are like vectors but more generic, so dont worry about loops. Just make sure the matrices you are multiplying have compatible dimension. Its completely tensor arithmetic, which for your case will be vector algebra.


sort occurrence matrix to minimize its spatial variance Is there any algorithm or an approach that sort occurrence matrix to reduce its spatial variance. I mean by spatial variance moving window variance (n*n moving box).

In case someone got the same concern, https://gmarti.gitlab.io/ml/2017/09/07/how-to-sort-distance-matrix.html


Adding recommendations to the output of a classification model I have built a binary classification model using:

logit
decision trees
random forest
bagging classifier
gradientboost
xgboost
adaboost

I have evaluated the above models and chose xgboost based on training/test and validation metrics (accuracy, prediction, recall, f1 and AUC).

I want to now productionalize it and share the output with the business. The output would basically have a list of items with the predicted class and that could be filtered based on business needs.

However, Instead of simply giving the business the predicted classes, I want to add insights/recommendations as to why a specific item was predicted with class X and how you could go about working on the item to change its class from say X to Y.

How do I go about this? I thought of using feature importance, but my input data shape is [800,000 * 1,050] and I am not sure if it would the best way to proceed.

Are there any existing industry standard methodologies that can add interpretability to such models and convert them from a black box models to prescriptive models?

Decision tress can be plotted in python, this is the most visualizing machine learning algorithm, you can see a link here: https://medium.com/@rnbrown/creating-and-visualizing-decision-trees-with-python-f8e8fa394176.

For everything else you need to do data exploration again. Try to visualize in bar plots, for example you can show, if you have class 'X' and 'Y', what your confidence intervall of your future 1 if it is class X and what if it is class 'Y'.


Clustering pair-wise distance dataset I have generated a dataset of pairwise distances as follows:

id_1 id_2 dist_12
id_2 id_3 dist_23

I want to cluster this data so as to identify the pattern. I have been looking at Spectral clustering and DBSCAN, but I haven't been able to come to a conclusion and have been ambiguous on how to make use of the existing implementations of these algorithms. I have been looking at Python and Java implementations so far.

Could anyone point me to a tutorial or demo on how to make use of these clustering algorithms to handle the situation in hand?

In the scikit-learn implementation of Spectral clustering and DBSCAN you do not need to precompute the distances, you should input the sample coordinates for all id_1 ... id_n.  Here is a simplification of the documented example comparison of clustering algorithms:

import numpy as np
from sklearn import cluster
from sklearn.preprocessing import StandardScaler

## Prepare the data
X = np.random.rand(1500, 2)
# When reading from a file of the form: `id_n coord_x coord_y`
# you will need this call instead:
# X = np.loadtxt('coords.csv', usecols=(1, 2))
X = StandardScaler().fit_transform(X)

## Instantiate the algorithms
spectral = cluster.SpectralClustering(n_clusters=2,
                                      eigen_solver='arpack',
                                      affinity="nearest_neighbors")
dbscan = cluster.DBSCAN(eps=.2)

## Use the algorithms
spectral_labels = spectral.fit_predict(X)
dbscan_labels = dbscan.fit_predict(X)


Group small values in a pie chart When plotting a Pie chart from a dictionary using Python's matplotlib, I get some classes that are too mashed together because of their small size. Is there a way to group the smallest values together and maybe plot them in a separate pie chart (or graduated bar) using python?

Here is the code I used:

import matplotlib.pyplot as plt

plt.style.use('ggplot')

dic = {'class a': 26.9,
       'Class b': 18,
       'Class c': 16.8,
       'Class d': 13,
       'Class e': 8.83,
       'Class f': 5.97,
       'Class g': 3.59,
       'Class h': 2.01,
       'Class i': 1.42,
       'Class j': 1.09,
       'Class k': 0.903,
       'Class l': 0.873,
       'Class m': 0.28,
       'Class n': 0.24,
       'Class o': 0.112}

labels = dic.keys()
sizes = dic.values()

fig, ax = plt.subplots()

ax.pie(sizes, labels=labels, autopct='%1.1f%%')
ax.axis('equal')
plt.tight_layout()
plt.show()
'''

This might not be the best solution:

I'm attaching the new code in which I split the data using a threshold and then plot them separately into 2 pie charts:

import matplotlib.pyplot as plt
import pandas as pd

plt.style.use('ggplot')

dic = [{'class a': 26.9,
       'Class b': 18,
       'Class c': 16.8,
       'Class d': 13,
       'Class e': 8.83,
       'Class f': 5.97,
       'Class g': 3.59,
       'Class h': 2.01,
       'Class i': 1.42,
       'Class j': 1.09,
       'Class k': 0.903,
       'Class l': 0.873,
       'Class m': 0.28,
       'Class n': 0.24,
       'Class o': 0.112}]

labels = dic.keys()
sizes = dic.values()

#converting the data into a dataframe
df = pd.DataFrame(dic)
dft = df.transpose()

dft.columns = ['Value']
dft['Class'] = dft.index

#defining the threshold
threshold = 5
dft_subset1 =  dft[dft['Value'] <= threshold]
dft_subset2 =  dft[dft['Value'] > threshold]

plot1 = dft_subset1.plot.pie(y='Value', figsize=(5, 5))
plot2 = dft_subset2.plot.pie(y='Value', figsize=(5, 5))

Output:


Opencv kmeans predict equivalent I'm moving my project from Python's libraries to opencv, and I have one big problem.

In Python skylearn, I have kmeans object, which has two useful methods: fit_predict and fit, which both are nescessery for my project.

The problem is I can't find any equivalent in opencv.
There is only function cv::kmeans, which is not enough for me.

When I'm using Python skylearn the pseudo code of my program looks like:

kmeans.fit_predict(train_data) // it gaves me labels, and compute centers

kmeans.predict(test_sample) // it gaves me labels for test sample computed from centers which were computed above

Is there any equivalent or way to do the same in opencv?

The OpenCV kmeans function is equivalent to fit(). It returns the cluster centers, and you can use this to implement your own predict()function. For a new sample, calculate the distance to each of the cluster centers. The cluster with the lowest distance is the predicted class.


np.unique() explanation? What happends in this numpy function:
https://numpy.org/doc/stable/reference/generated/numpy.unique.html
a = np.array([1, 2, 5, 3, 2])
u, indices = np.unique(a, return_inverse=True)

The results are:
u

array([1, 2, 3, 5)]
indices

array([0,1,3,2,1), dtype=int64)
u[indices]

array([1, 2, 5, 3, 1])
u are the single values inside this array. What is indices? why is it a 3 and not a 5? and what is going on in u[indices]?

Firstly the function returns the variables u and indices:

u contains the unique elements sorted. In other words no element is repeated (the number 2 does not appear twice) and the elements will be listed from the smallest to the largest value
indices is the same size as a and basically it contains the index in u you should used to recover a. So when they give you [0,1,3,2,1], in this case the number 3 refers to the 3rd index of u = [1, 2, 3, 5] which in this case is 5

You can see how u and indices can be used to recover a by running the following code:
for element in indices:
      print(u[element])

This also explains your last question: u[indices] it basically gives you back a. Note that in your question u[indices] should return array([1, 2, 5, 3, 2]). I assume you made a typo


How do I calculate the maximum likelihood (machine learning statistics) of this table of data? I’m taking a machine learning course and I’m stuck. I've already tried to ask around in the forums of the class but I don't understand how to implement the formulas they give me. Can you please help me understand how to solve this problem from the ground up?

  “Calculate the likelihood of this data.”

  X  | Y

  2.5 | +1

  0.3 | -1

  2.8 | +1

  0.5 | +1

  the current estimates of the weights are $w_0 = 0$ and $w_1 = 1$. ($w_0$: the intercept, $w_1$: the weight for $x$).

Now Here's what I've already done in case that matters:

When I went on to the forums to try to understand I was told this:

  You can use the sigmoid function to find the likelihood of any given
  one, and then once you have the probabilities of any given one
  occurring you should be able to find the probability of the whole
  thing

Which is nice; now I know it has something to do with the sigmoid function: $$\frac{1} {1 + e^{-x}}$$

sigmoid(2.5) = 0.9241418

I hope I'm on the right track, I really have no way of knowing. So then I calculated the sigmoid for all of them and multiplied them together hoping that would give me the probability of the whole thing because that's how you combine probabilities, right? (The probability of a dice role hitting 2 three times is $1/6 \times 1/6 \times 1/6$).

But that apparently isn't the right answer, and I didn't take into account the $y$ column or the weights because I don't know how. How sad for me.

Can you please explain it to me, I feel like I'm a blind man looking for the right colored paper.

You might have mistaken the equation for maximum likelihood , and it should be like this :

  $p^y*(1-p)^{1-y}$

while y is your label , it should be 0 or 1 , 0 for negative and 1 for positive , and p is your sigmoid function value.

so , in your case it should be :

  p(2.5)*(1 - p(0.3))*p(2.8)*p(0.5)

Hopes this will help you , good luck !


Validation vs. test vs. training accuracy. Which one should I compare for claiming overfit? I have read on the several answers here and on the Internet that cross-validation helps to indicate that if the model will generalize well or not and about overfitting.

But I am confused that which two accuracies/errors amoung test/training/validation should I compare to be able to see if the model is overfitting or not?

For example:

I divide my data for 70% training and 30% test.

When I get to run 10 fold cross-validation, I get 10 accuracies that I can take the average/mean of. should I call this mean as validation accuracy?

Afterward, I test the model on 30% test data and get Test Accuracy.

In this case, what will be training accuracy? And which
two accuracies should I compare to see if the model is overfitting or not?

Which two accuracies I compare to see if the model is overfitting or not?

You should compare the training and test accuracies to identify over-fitting. A training accuracy that is subjectively far higher than test accuracy indicates over-fitting.

Here, "accuracy" is used in a broad sense, it can be replaced with F1, AUC, error (increase becomes decrease, higher becomes lower), etc.

I suggest "Bias and Variance" and "Learning curves" parts of "Machine Learning Yearning - Andrew Ng". It presents plots and interpretations for all the cases with a clear narration.

  When I get to run 10 fold cross-validation, I get 10 accuracies that I
  can take the average/mean of. should I call this mean as validation
  accuracy?

No. It is a [estimate of] test accuracy.
The difference between validation and test sets (and their corresponding accuracies) is that validation set is used to build/select a better model, meaning it affects the final model. However, since 10-fold CV always tests an already-built model on its 10% held-out, and it is not used here to select between models, its 10% held-out is a test set not a validation set.

  Afterward, I test the model on 30% test data and get Test Accuracy.

If you don't use the K-fold to select between multiple models, this part is not needed, run K-fold on 100% of data to get the test accuracy. Otherwise, you should keep this test set, since the result of K-fold would be a validation accuracy.

  In this case, what will be training accuracy?

From each of 10 folds you can get a test accuracy on 10% of data, and a training accuracy on 90% of data. In python, method cross_val_score only calculates the test accuracies. Here is how to calculate both:

from  sklearn import model_selection
from sklearn import datasets
from sklearn import svm

iris = datasets.load_iris()
clf = svm.SVC(kernel='linear', C=1)
scores = model_selection.cross_validate(clf, iris.data, iris.target, cv=5, return_train_score=True)
print('Train scores:')
print(scores['train_score'])
print('Test scores:')
print(scores['test_score'])

Set return_estimator = True to get the trained models too.

More on validation set

Validation set shows up in two general cases: (1) building a model, and (2) selecting between multiple models,

Two examples for building a model: we (a) stop training a neural network, or (b) stop pruning a decision tree when accuracy of model on validation set starts to decrease. Then, we test the final model on a held-out set, to get the test accuracy.
Two examples for selecting between multiple models:

a. We do K-fold CV on one neural network with 3 layers, and one with 5 layers (to get K models for each), then we select the NN with the highest validation accuracy averaged over K models; suppose the 5 layer NN. Finally, we train the 5 layer NN on a 80% train, 20% validation split of combined K folds, and then test it  on a held out set to get the test accuracy.

b. We apply two already-built SVM and decision tree models on a validation set, then we select the one with the highest validation accuracy. Finally, we test the selected model on a held-out set to get the test accuracy.


I need to force my random forest model to learn one crucial relationship between features. How can I achieve that? Say, given 10 independent features as input to my RF model, when feature 1 and feature 3 are 100 (or less), my model output must be 5 despite the values of the other features. How can I teach that relationship to my RF model - so it will always obey that?

I have tried adding repeated artificial data points to the training set where features 1 and 3 are less or equal to 100 and the output is always zero, but the RF model does not seem to understand/learn that relation just based on that.

I also tried to play with (1) number of trees (2) number of nodes and (3) max nodes without any success.

I am using the mlr library in R to create my RF model. Thanks.

Machine learning algorithms are not to know, they predict. A tree-based ML algorithm will give you probabilities of being in each class, not whether it belongs to a particular class. By preparing your labels carefully; if your data has enough examples of which “feature 1 and 3 are less and equal to 100” and the result is 0, it is highly probable that your algorithm will predict its class with a strong accuracy. Obviously,  the strength of the result depends on what your ML task is, what is your data, how it is structured and what your hyperparameter settings are. I do not know if there is another way to bend the forest in such a manner.

I know that is not the kind of an answer you expected, but I’ll be glad if I could help.


Unable to access saved code I am new to jupyter notebook and i can't find my saved .ipynb code after logging out of jupyter notebook accidentally.
Thanks in anticipation

I guess you have made a file in your default document folder. A probable location for your notebook is where it starts if you have not changed the default path or maybe the location in which your files is located. Try to rerun your jupyter from your terminal or command prompt. It will open the contents of the folder you are looking for your notebook.


MATLAB @ symbol for fminunc / fmincg function I am following Andrew Ng's Coursera class on machine learning, and I came across this syntax for the fminunc and fmincg functions:

fmincg (@(t)(lrCostFunction(t, X, (y == c), lambda)),initial_theta, options);

Specifically, it is the @ that's confusing me. I read up on it and I still don't understand what we're doing with it here. Can someone clarify what this is doing, as well as how the t in the function call to lrCostFunction is related to the @(t) in the beginning.

@(something) is used to call a function in MATLAB or octave.

Suppose you create a function within a code. And you set a keyword for that function.
we have a sum(x,y) function which takes two inputs and returns the sum.
Now you fix the value of y, say y = 3; And you want to change the value of x every time. You can design the inline function by following:

y=3
temp = @(p) sum(p,y);

ret = temp(4);  % What this will do is it will add 4 with 3 and return the answer.

This is used where we have to call a function multiple times and we don't want to write the arguments every time. It's a shortcut method.

In your case

fmincg (@(t)(lrCostFunction(t, X, (y == c), lambda)),initial_theta, options);

the value of t will be replaced every time with initial_theta. That is how the fmincg or fminunc functions work.


When to use different Word2Vec training approaches? So I am learning Word2Vec for the first time and my question is quite basic: How to know what approach to use? I.e, Word2Vec in Tensorflow or Word2Vec trained with Gensim ?

In what cases would implementing it through the more manual first approach be useful vs. the second one? If there is already an easier way to train a word2vec model using gensim, why is that not used always?

Furthermore, what is the benefit in using a pre-trained model like the Google News dataset? What happens when there are words that are not included in the news dataset?

Sorry if this question is basic, I just want to get a clearer grasp of the overall picture.

The advantage of using pre-trained vectors is being able to inject knowledge from a larger corpus than you might have access to: word2vec has a vocabulary of 3 million words and phrases trained on the google news dataset comprising ~100 billion tokens, and there's no cost to you in training time.

In addition, they are fast and easy to use, just load the embeddings and look them up. It's straightforward to substitute different sets of pre-trained vectors (fastText, GloVe etc) as one might be more suited to a particular use case.

However, when your vocabulary does not have an entry in word2vec, by default you'll end up with a null entry in your embedding layer (depending on how you handle it). You'll need to consider the scale/impact and how to address it (keep/discard/consider online training). As yazhi says a decision must be made about how to handle out of vocabulary words.

The advantage of learning word vectors from your own corpus is that they would be derived from your dataset, so if you have reason to believe that the composition of your data is significantly different from the corpus used for the pre-trained vectors then that may result in better downstream performance. However, that comes at a cost in time taken to train your own vector representations.


Is it ok to interprete PCA plot this way? I have several samples (C2, C4, C5) and want to check if they are at a certain stage. I included some known samples (D0 - D77) which were generated at different stages by another lab. In the PCA plot, my samples cluster together on the left and the known samples are dispersed on the right. I think the major difference among all the samples is different experimental protocols (PC1) and the second is different stages (PC2). So my samples are at the same stage. Is that right? And can we say my samples are at a stage between D12 and D19 (when projected to known samples, my samples are located between D12 and D19)? I have no strong mathematical background. Hope someone with math background can give some explanation. Thanks!

[ UPDATE1 ]
I did this analysis using prcomp in R. The input is a 2D numeric matrix with 17436 rows and 42 columns. Each row represents a gene and each column represents a sample. The number is the gene expression level for a gene in a sample. The gene expression level is normalized using DESeq2 and thus the numbers are comparable across genes (rows) and samples (columns). For the 42 columns, 18 are from my experiments and the rest from published datasets. Besides different protocols, it is possible there are other differences. Generally, the table is a combination of two sources of data. In my data , C2, C4 and C5 are three cell lines which we processed in parallel in experiments. In published datasets, they sampled the cells at different time points (Day 0 to Day 77).

[ UPDATE2 ] R Code for PCA and plotting

# normCounts: normalized count from DESeq2, with 17436 rows and 42 columns
normCounts0 <- normCounts[ rowSums(normCounts) > 0, ]
tab <- t(normCounts0)
pca <- prcomp(tab, scale = TRUE)
tmp.x <- as.data.frame(pca$x)
tmp.x$sample<-c(rep("C2_Con",3),rep("C2_KD",3),rep("C5_Con",3),rep("C5_KD",3),rep("C4_Con",3),rep("C4_KD",3), rep("D0",4),rep("D7",4),rep("D12",2),rep("D19",4),rep("D26",2),rep("D33",2),rep("D49",2),rep("D63",2),rep("D77",2))

require("ggplot2")
p <- ggplot(tmp.x, aes(x=PC1, y=PC2, color=sample))
p + geom_point() + scale_color_discrete(breaks=c("C2_Con","C2_KD","C5_Con","C5_KD","C4_Con","C4_KD", "D0","D7", "D12","D19","D26","D33","D49","D63","D77"))

Given the information provided, I do not think it's enough to say that your samples are between stages D12 and D19. I admit knowing nothing about your domain, but with what you're showing it is very conceivable that other factors are causing the spread of your samples from the others on PC1 and it's too much of a stretch to say that your samples "project" into the others in between D12 and D19. Why does it have to be a projection orthogonal to the axis described by PC1? Remeber that PC1 is a linear combination of all your inputs. That is, it describes an arbitrary line through your input space that is probably not aligned with any one thing in particular. For example, all three of the following projections could be envisioned for your problem:

Another red flag for me is that you say PC1 is related to differences in experimental protocols. You haven't posted a scree plot or listed the eigenvalues for the different PCs but obviously PC1 soaks up the most variance so it's a pretty big deal. If you're saying that PC1 is correlated with experimental protocol, then that would seem good because hopefully the impact of experimental protocol is minimized in other PCs. However, that's never completely the case, really.

Again, not knowing your domain it's hard to tell what the impact of experimental protocol is on the distribution of your data in the PC plot, but I'd be VERY wary of comparing these data in this way if your data was produced using a different experimental design and, as you noted, "there may be other differences."

My hunch is that you would need to continue minimizing differences in the protocols between your experiments and the published experiments before you could really use this type of analysis. Others might disagree but if you posted the sources of your data, the eigenvectors, and possibly a subset of the data, experts in your field may be able to chime in on this methodology.

The short answer is: you could be right in your interpretation but at this point you don't have enough evidence to suggest your samples are "in between" D12 and D19 in terms of the stage. And I worry about differences in methodology completely swamping your ability to interpret things this way.


SVM hyperplane margin so that $H_0$ is equidistant from $H_1$ and $H_2$.
However, here the variable $\delta$ is not necessary. So we can set $\delta=1$ to simplify the problem.
$$w\cdot x+b=1 $$
and
$$w\cdot x+b=−1$$

Why is this assumption is taken? If it is taken, we can get the distance between two planes as $2$ directly because both are parallel and differ by $2$. how it is $\frac{2}{\left\|w\right\|}$ instead.
got equations from this https://www.svm-tutorial.com/2015/06/svm-understanding-math-part-3/

After we have

$$w^Tx + b = \pm \delta$$

We can always divide everything by $\delta$,

$$\left( \frac{w}{\delta}\right)^Tx + \left( \frac{b}{\delta}\right)=\pm1$$

Now, we can set $\tilde{w}=\frac{w}{\delta}$ and $\tilde{b}=\frac{b}{\delta}$.

$$\tilde{w}^Tx+\tilde{b}=\pm1$$

This is as if we have set $\delta=1$ from the beginning.

The derivation of the distance formula has been given in equation $(19)$ in the article that you linked to and you might like to be more specific if you can't understand it. The distance should be $\frac{2\delta}{\|w\|}$ if $\delta$ is not set to be $1$.


Optimal projection for data visualization I have 90k points in $\mathbb{R}^{32}$ (i.e., a 90k by 32 real matrix) which I want to visualize. I know I can cluster my points (k-means &c), but I want to select a few "interesting" 2-planes in $\mathbb{R}^{32}$, project the points there and scatter-plot them.

How do I select the interesting 2-planes to project to?

There are a few interesting plots and transformations you could start with, each dependent upon the purpose of your analysis. Below are some first steps I might take.

If simply visually looking for clusters:
If clusters are what you are after, then I would recommend applying a Principal Component Analysis to the dataset and then plotting the dataset with the first 2 principal components as the axes. However, the major downside to PCA is that you will have to go "unpack" the principal components to find the original variables. In other words, you'll be may be able to identify cool clusters, but it will be a little harder to tie the findings back to your 32 variables.
If looking for quick relationships between variables to build on:
With "only" 32 variables you could do a pairwise plot. However, a smarter way may be to first identify the relationships mathematically (e.g. correlation) and then plotting those variables.
If looking for quick relationships in each variable to build on:
Or, look at 32 histograms to start off with. Look out for clear bi-modal (or more) to begin piecing together an understanding of how your variables may contribute to an unsupervised model. If you end up look at 32 unimodal histograms then you can conclude early that no matter how you cluster, you will simply end up with a blob.

Actually, in usual analytics workflows I would go 3 > 2 > 1. But if I was fishing for clusters and just wanted to see if clusters will appear or not, PCA would be a good shortcut.

Also, feel free to sample your dataset. 90k points on your screen will likely do more harm than good.


what does `a factor of K-fold` mean in GPU-based training? From the book "Deep Learning and Convolutional Neural Networks for Medical Image Computing"

  As we learned about the current state of research on deep learning, I
  was surprised to find that other investigators had used convolutional
  neural networks, one type of deep learning, in the past [22, 23]. But
  there seemed to be something different about the most recent crop of
  deep learning algorithms. They routinely used GPU processing to
  accelerate training by as much as a factor of 40-fold. They also used
  multiple convolution layers and multiple data reduction layers.

does 40-fold mean "K-Fold Cross-Validation"?

The author here refers to a comparison between older and most recent implementations of deep learning algorithms regarding execution time (accelerating processing speed). It just means that "they used GPUs and achieved x40 better execution time". It has nothing to do with K-Folds.


Time-series grouped cross-validation I have data with the following structure:
created_at | customer_id | features | target
2019-01-01             2   xxxxxxxx       y
2019-01-02             3   xxxxxxxx       y
2019-01-03             3   xxxxxxxx       y
...

That is, a session timestamp, a customer id, some features, and a target. I want to build an ML model to predict this target, and I'm having issues to do cross-validation properly.
The idea is that this model is deployed and used to model new customers. For this reason, I need the cross-validation setting to satisfy the following properties:

It has to be done in a time-series way: that is, for every train-validation split in cross-validation, we need all created_at of the validation set to be higher than all created_at of the training set.
It has to split customers: that is, for every train-validation split in cross-validation, we cannot have any customer both in train and validation.

Can you think of a way of doing this? Is there an implementation in python or in the scikit-learn ecosystem?

As @NoahWeber mentioned, one solution is to:

split by customer ids (A)
do the time series split on all dataset (B)
keep in the training (resp. testing) dataset only the data from training (resp. testing) customers split (A) and from training (resp. testing) time series split (B).

Below is a code sample I was writing at the same time he answered.
import pandas as pd
import numpy as np
from sklearn.model_selection import RepeatedKFold
from sklearn.model_selection import TimeSeriesSplit

# Generating dates
def pp(start, end, n):
    start_u = start.value//10**9
    end_u = end.value//10**9

    return pd.DatetimeIndex((10**9*np.random.randint(start_u, end_u, n, dtype=np.int64)).view('M8[ns]'))

start = pd.to_datetime('2015-01-01')
end = pd.to_datetime('2018-01-01')
fake_date = pp(start, end, 500)

# Fake dataframe
df = pd.DataFrame(data=np.random.random((500,5)), index=fake_date, columns=['feat'+str(i) for i in range(5)])
df['customer_id'] = np.random.randint(0, 5, 500)
df['label'] = np.random.randint(0, 3, 500)

# First split by customer
rkf = RepeatedKFold(n_splits=2, n_repeats=5, random_state=42)
for train_cust, test_cust in rkf.split(df['customer_id'].unique()):
    print("training/testing with customers : " + str(train_cust)+"/"+str(test_cust))

    # Then sort all the data (if not already sorted)
    sorted_df = df.sort_index()

    # Then do the time series split
    tscv = TimeSeriesSplit(max_train_size=None, n_splits=5)
    for train_index, test_index in tscv.split(sorted_df.values):
        df_train, df_test = sorted_df.iloc[train_index], sorted_df.iloc[test_index]

        # Keep the right customers for training/testing
        df_train_final = pd.concat( [ df_train.groupby('customer_id').get_group(i) for i in train_cust ])
        df_test_final = pd.concat( [ df_test.groupby('customer_id').get_group(i) for i in test_cust ])

Note: Generating random dates is based on this post
Note bis: I tested the generated training/testing dataframes ready for cross-val with this sample code that you can add right after the line df_test_final:
# Test condition 1: temporality
for i in range(len(df_test_final)):
    for j in range(len(df_train_final)):
        if df_test_final.index[i] < df_train_final.index[j]:
            print("Error with " + str(i) + "/" + str(j))

# Test condition 2: training customers are not in testing final df
for i in train_cust:
    if i in df_test_final['customer_id'].values:
        print("Error in df_train with " + str(i) + "th customer")

# Test condition 2: testing customers are not in training final df
for i in test_cust:
    if i in df_train_final['customer_id'].values:
        print("Error in df_train with " + str(i) + "th customer")

Here is a pseudo-code implementation:
function keep_customer_ids( data, ids ):
    goal: this function returns a subset of data with only the events that have
          an id tag that is in ids
    data: labeled events containing features, date and a customer id tag
    ids: the customer ids you want to keep
    for event in data:
        if event has a customer id tag that is in ids, keep it
        else, drop it
    return data

algorithm:
    for the number of cross-val you want:
        customer_train_ids, customer_test_ids = split_by_customers( customer_ids )
        train_data, test_data = split_data_in_time_series_way( data )
        final_train_data = keep_customer_ids( train_data, customer_train_ids )
        final_test_data = keep_customer_ids( test_data, customer_test_ids )
        do_the_fit_predict_things( final_train_data, final_test_data )


what is BIO Tags for creating custom NER Named entity recognization? I want to create custom NER Named entity recognition but im confused with this part of what is BIO Tags. Can anyone please explain the steps for creating NER and about this B, I, O tag.

It is easy. You need to tag a phrase using B (Begin), I (Interior), and E (End). For example, you want to tag "United States of America" as the name of a country. You will tag likes:

United(B_Country) States(I_Country) of(I_Country) America(E_Country)

In the same text if you find "Islamic Republic of Iran", you will tag likes:

Islamic(B_Country) Republic(I_Country) of(I_Country) Iran(E_Country)

Also, you will tag "United Kingdom" likes:

United(B_Country) Kingdom(E_Country)

Therefore, for every label that you have in the label set, you will have three labels in the tagging that is B_LabelName, I_LabelName, and E_LabelName.

Notice that In some tagging systems, also uses from I_LabelName instead of E_LabelName.

Now, what is "O"? In some cases the all part of the phrase is not in the label but comming in the middle of the phrase and we need to say it is out ("O") of the label. For example, in the text analysis we want to take out "of" from country names. Hence we will tag "United States of America" like the following:

United(B_Country) States(I_Country) of(O_Country) America(E_Country/I_Country)


What database should I use? I am a high-school students who is learning about data science in his free time. I have gotten a neural network to work which is able to solve xor problems. My neural network uses sigmoid as the activation function for both the hidden and output layers. It also has only one hidden layer. I am wondering about what would be the best simple problem which I could solve with my neural net. I would like a database in which there is a probability output or something similar since I've had problems converting sigmoided output to normal values. I have looked on the UCI machine learning repository but have found nothing witch has caught my eye. I would appreciate any help! :)

Search "dataset" instead of "database".
see these:

A Neural Network from Scratch in just a few Lines of Python code
Multi Layer Neural Networks with Sigmoid Function Deep Learning for Rookies


How to see/change learning rate in Keras LSTM? I see in some question/answers that ask to decrease the learning rate. But I don't know how can I see and change the learning rate of LSTM model in Keras library?

In Keras, you can set the learning rate as a parameter for the optimization method, the piece of code below is an example from Keras documentation:
    from keras import optimizers

   model = Sequential()
   model.add(Dense(64, kernel_initializer='uniform', input_shape=(10,)))
   model.add(Activation('softmax'))

   sgd = optimizers.SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
   model.compile(loss='mean_squared_error', optimizer=sgd)

In the above code, the fifth line lr is learning rate, which in this code is set to 0.01. You change it to whatever you want.
For further detail, see this link.
Please tell me if this helped to solve your problem.


Automatically generating images for an art show with deep learning software My pursuit is to generate something like a grottesque(a kind of painting producing human-animals and plants hybrids).  I need to do something like this painting in order to create an art exhibition. I don't need that produced images are ordinate or beautiful, just monstrous images produced automatically by a computer. Can someone help me? And is it possible to do it in a month?

You can try to generate morphed images and then scale any size. You can use this or similar libraries.


Error on multitask neural nets where all outputs not observed for every example Let's say I have 2 datasets, each from a set of experiments.  Dataset A measures a set of properties X for set S, while dataset B measures properties Y for set T.  X and Y are highly correlated, and S and T have (not perfect) overlap.  To give an example, I might have

A   |   ID   |   pKa    |  log P
    |  cid12 |   3.51   |   1.2
    |  cid51 |   2.32   |   .9

B   |   ID   | Pol Srf A|  cLogP
    |  cid12 |   48.5   |   1.5
    |  cid88 |   61.2   |   .9

I would like to use the shared information here (since all of these are physical properties) in a multitask neural net setup.  For examples in both S and T, I have a target vector as long as X union Y.  But for members exclusive to S or T, I am missing values.

In the examples, I have

 C   |   ID   |   pKa    |  log P  | Pol Srf A|  cLogP
     |  cid12 |   3.51   |   1.2   |   48.5   |   1.5
     |  cid51 |   2.32   |   .9    |    na    |   na
     |  cid88 |    na    |   na    |   61.2   |   .9

When I train, do I simply zero fill these outputs, then not backpropagate error coming from these nodes?  So for  cid12, error from all nodes backpropagates, but for cid51, only error from pKa and logP?

If this is the case, is that a standard procedure implemented in most NN libraries (Theano, for example) and what is that called?

You have already found the answer by yourself: if there are missing values, you should not backpropagate on them. I do not know of any particular name for this action.

I do not think that there are already implemented functions to do so, but it seems quite straightforward to embed your loss function (e.g. mean squared error) into a personalized one.

For example, let's say that you filled all missing values with $-1$:

(Disclaimer: I did not test this code but got inspiration in this answer)

import theano
from keras import backend as K

def custom_objective(y_true, y_pred):
    indices = y_true==-1
    y_pred2 = y_pred[indices]
    y_true2 = y_true[indices]
    # then you could use any already implemented loss function, or
    mse = K.mean(K.square(y_pred2-y_true2))
    return (mse)

Note that with nan coded as 0 you could use flatnonzero().


What is the feature matrix in word2vec? I'm a beginner in neural networks and currently I'm exploring the word2vec model. However I'm having a tough time to understand what the feature matrix exactly is.

I can understand that the first matrix is a one-hot encoding vector for a given word, but what does the second matrix signify? More specifically, what does each of those values (ie. 17, 24, 1 etc) mean?

The idea behind word2vec is to represent words by a vector of real numbers of dimension d. Therefore the second matrix is the representation of those words.

The i-th line of this matrix is the vector representation of the i-th word.

Let's say that in your example you have 5 words : ["Lion", "Cat", "Dog", "Horse", "Mouse"], then the first vector [0,0,0,1,0] means you're considering the word "Horse" and so the representation of "Horse" is [10, 12, 19]. Similarly, [17, 24, 1] is the representation of the word "Lion".

To my knowledege, there are no "human meaning" specifically to each of the numbers in these representations. One number is not representing if the word is a verb or not, an adjective or not... It's just the weights that you change to solve your optimization problem to learn the representation of your words.

This tutorial may help : http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/ even though I think the image you put was from this link.

You can also check this, which may help you get started with word vectors with TensorFlow : https://www.tensorflow.org/tutorials/word2vec


How to perform model selection for One-Class Classification? Consistency based model selection does not perform well for many datasets for One-Class Classification (OCC). So I am looking for some other model selection criteria.

Since, only one class of data (i.e. the positive class) is available, classification error is no longer a valid criterion. It is very challenging task to decide on the optimal parameters in OCC. Here, model selection means the optimal parameter selection for the one-class classifier.

Please suggest any additional methods except consistency based model selection method.

Have you read Janssens' dissertation "Outlier Selection and One-Class Classification"? He has a chapter on evaluation which may be of use. Have you thought about artificial generation of negative instances?

I had to deal with OCC evaluation awhile back and never came upon an entirely satisfactory solution. As I recall, the basic problem boils down to the fact that you don't really have labelled instances of the negative class. Without these instances there is no way of computing precision = TP/(TP+FP) which relies upon the count of incorrectly accepted negative instances (FP). This just leaves you with recall = TP/(TP + FN) as a computable metric - which by itself is next to useless in evaluating classifier performance i.e. it's trivial to write a classifier that get's 100% recall, just predict the positive class for every instance in your test data... you will correctly identify every true positive instance, but you will also incorrectly predict every negative instance as a positive instance. What is typically required is a balance or trade-off between precision and recall.

You didn't mention a particular problem domain in which you are working, but perhaps you can leverage this domain knowledge as a work-around to the problem of OCC evaluation. This particular problem I will relate is technically cast as a PU Learning problem (learning from Positive and Unlabelled instances). I'll try to describe some solutions that I explored while tackling this challenge - I'll give you fair warning that they are not entirely satisfactory solution, but I believe they are logically sound and somewhat defensible. From a purely pragmatic perspective, they are definitely better than the alternative.

Implicit Negative Class

Scenario: imagine a membership-based incentive program, in which people can pay an annual fee to become a member of Foobar Inc. Members, but not non-members, are entitled buy widgets at Foobar's retail outlets.

You are in charge of a marketing drive to increase Foobar's membership. You have limited funds and reach, so you need to carefully select people from the general population that you believe will most likely become members if you target them with a promotional campaign. In this scenario you have a database of known Foobar member's (the labelled, positive class data, which we label POS) and a database of known non-members from the general public. This is your unlabelled data that contains people who may or may not become members if targeted. All we have is POS instances and UL instances -- there are no labelled NEG instances.

Depending upon your particular problem, context, and domain, you may be able to transform the OCC/PU problem into a more classical problem with standard performance metrics. In the situation I just described, there are at least three ways of creating an implicit negative class:

Use the spy technique to generate "reliable" negative instances
that can be placed in your test set;
If historical campaign data is available you could identify those
individuals who were targeted multiple times for campaign
membership, but who rejected it every time -- these could be
"consistently negative" instances;
If your known membership pool consists of members of varying
value to the business, you could divide members into good/bad, and
use the bad instances as a proxy for NEG instances.

One final area you may want to explore is evaluation in collaborative filtering research. Collaborative filtering on unary responses (e.g., Facebook "like")

  can be considered the matrix completion analog of the
  positive-unlabelled (PU) learning problem in classification and
  regression modelling. (Aggarwal, 2016)

Perhaps this field has developed some novel approaches to OCC evaluation that you can leverage. Without knowing the details of your particular case, however, its hard to provide a more detailed solution. Please update this thread with your eventual solution -- there is so much more research that needs to be done in this particular problem domain.


Segmentation model to predict face forward and profile parts of the face I am developing a model for feature counting on a person's face that consumes three photos (one face forward and two profile pictures). My model can already detect features, but it counts some features twice.
Consider the image below. There are only four labels (1, 2, 3, 4 and 5). Model detects four labels (1, 2, 3, 4) on the profile photo and four labels on the face forward photo (1, 2, 3, 5).
Labels (1, 2, 3) are counted twice. A straightforward approach would be to segment hundreds of faces into "face forward region" and "profile region". This is tedious and accurate labeling is hard. Is there a better way to segment the face in regions or to detect duplication of the detected features?

There is a library called facemesh for Python. It can detect face landmark points in Python. Connecting facemesh points to crop image to a desired polygon is trivial.