Deep neural networks for mostly zero valued data I have some data with around 1000 features. Problem is most of the features are 0. In each row usually around 100 features have values and rest of the features are 0. A sample example data is given below

ID  f1  f2  f3 ........ f1000  label
 1  45  0    0            32     1
 2  23  0    0            0      2
 3  0   40   0            0      1
 4  0   0    23           24     3
 5  78  0    64           0      2

As you can see there are lot of 0 in each row. Which deep neural networks method should I use? I was thinking about RNN or CNN but I am not sure if these are the best choices. I was thinking about some methods which will ignore zero values i.e does not consider the zero values in the calculation. I think RNN can do this. But I am not exactly sure.

There's this misunderstanding that deep learning is generally suitable if you have loads of data, which, judging from your comment, is your case. This is generally an inaccurate belief.

Deep learning (including CNN and RNN), are complex models with thousands of parameters that are able to model complex relationships. Such relationships are generally "hidden" in vast amounts of data, but this is not always the case. You may have at hand data that are generated from a simple distribution, and as such it will not need a complex model to approximate, even if your sample size is huge.

Here's a fabricated example: Let's say that you have the function y=a*x1 + b*x2 + c*x3 + d*x4. This function entails linear relationships between all independent variables and the dependent variable, y. You could sample this function a million times and still you would have at hand data that could be approximated through linear regression.

Coming to your case: to identify what kind of algorithm you would need, you first need to look at your data, perhaps through performing a statistical analysis. Then I would suggest to start simple. Try logistic regression, for starters. Is the model satisfactory in validation? If not, try perhaps decision trees. Or a shallow neural net. Always validate (you have loads of data so validation should be easy!). My (admittedly wild) guess is that your classification problem could be addressed with much simpler algorithms than DNN. But YMMV, of course.

FWIW, here's another answer similar to your case. And another one.


Length of document in doc2vec I have 100 sentences that I want to cluster based on similarity. I've used doc2vec to vectorize the sentences into 20 dimensional vectors and applied kmeans to cluster them. I haven't got the desired results yet.

I've read that doc2vec performs well only on large datasets. I want to know if increasing the length of each data sample, would compensate for the low number of samples, and help the model train better?

For example, if my sentences are originally "making coffee", "making tea", "playing with dogs", would changing them to "making coffee requires a cup of milk and some coffee powder", "making tea requires boiling water and some tea leaves" (supplement each document with more information) help in getting better results? Would the model understand the context better?

Increasing the sentence length might help, but not much. You can try n-grams(2 or 3) while vectorizing the data. Generally more number of sentences will help.


Named Entity Recognition: NLTK using Regular Expression Many times Named Entity Recognition (NER) doesn't tag consecutive NNPs as one NE. I think editing the NER to use RegexpTagger also can improve the NER.

For example, consider the following input:

  "Barack Obama is a great person."

And the output:

Tree('S', [Tree('PERSON', [('Barack', 'NNP')]), Tree('ORGANIZATION', [('Obama', 'NNP')]),
    ('is', 'VBZ'), ('a', 'DT'), ('great', 'JJ'), ('person', 'NN'), ('.', '.')])

where as for the input:

  'Former Vice President Dick Cheney told conservative radio host Laura Ingraham that he "was honored" to be compared to Darth Vader while in office.'

the output is:

Tree('S', [('Former', 'JJ'), ('Vice', 'NNP'), ('President', 'NNP'),
    Tree('NE', [('Dick', 'NNP'), ('Cheney', 'NNP')]), ('told', 'VBD'), ('conservative', 'JJ'),
    ('radio', 'NN'), ('host', 'NN'), Tree('NE', [('Laura', 'NNP'), ('Ingraham', 'NNP')]),
    ('that', 'IN'), ('he', 'PRP'), ('``', '``'), ('was', 'VBD'), ('honored', 'VBN'),
    ("''", "''"), ('to', 'TO'), ('be', 'VB'), ('compared', 'VBN'), ('to', 'TO'),
    Tree('NE', [('Darth', 'NNP'), ('Vader', 'NNP')]), ('while', 'IN'), ('in', 'IN'),
    ('office', 'NN'), ('.', '.')])

Here Vice/NNP, President/NNP, (Dick/NNP, Cheney/NNP) is correctly extracted. So, I think if nltk.ne_chunk is used first, and then if two consecutive trees are NNP, there are higher chances that both refer to one entity.

I have been playing with NLTK toolkit, and I came across this problem a lot, but couldn't find a satisfying answer. Any suggestion will be really appreciated. I'm looking for flaws in my approach.

You have a great idea going, and it might work for your specific project. However there are a few considerations you should take into account:

In your first sentence, Obama in incorrectly classified as an organization, instead of a person. This is because the training model used my NLTK probably does not have enough data to recognize Obama as a PERSON. So, one way would be to update this model by training a new model with a lot of labeled training data. Generating labeled training data is one of the most expensive tasks in NLP - because of all the man hours it takes to tag sentences with the correct Part of Speech as well as semantic role.
In sentence 2, there are 2 concepts - "Former Vice President", and "Dick Cheney". You can use co-reference to identify the relation between the 2 NNPs. Both the NNP are refering to the same entity, and the same entity could be referenced as - "former vice president" as well as "Dick Cheney". Co-reference is often used to identify the Named entity that pronouns refer to. e.g. "Dick Cheney is the former vice president of USA. He is a Republican". Here the pronoun "he" refers to "Dick Cheney", and it should be identified by a co-reference identification tool.


Why is my CNN not training Hi I am trying to train a CNN to differentiate between pictures of dogs and pictures of cats. It does not seem to learn anything no matter how I change the architecture. I have used the following code to create the datasets:
train_generator = tf.keras.preprocessing.image_dataset_from_directory(
    G:/Dogs_cats/Train',
    labels="inferred",
    label_mode="binary",
    class_names=None,
    color_mode="rgb",
    batch_size=150,
    image_size=(350, 350),
    shuffle=True,
    seed=1)
   # validation_split=0.2,
   # subset='validation',
   # interpolation="bilinear",
  #  follow_links=False,)

validation_generator = tf.keras.preprocessing.image_dataset_from_directory(
   'G:/Dogs_cats/Validation',
    labels="inferred",
    label_mode="binary",
    class_names=None,
    color_mode="rgb",
    batch_size=150,
    image_size=(350, 350),
    shuffle=True,
    seed=1)

I then created my CNN model:
model = tf.keras.models.Sequential([
        tf.keras.layers.Conv2D(60, (3,3), activation='relu', input_shape=(350, 350, 3),padding='same',kernel_initializer='he_uniform'),
        tf.keras.layers.MaxPooling2D(2, 2),
        tf.keras.layers.Dropout(0.3),
        tf.keras.layers.Conv2D(5, (1,1), activation='relu', padding='same',kernel_initializer='he_uniform'),
        tf.keras.layers.Conv2D(60, (3,3), activation='relu',padding='same',kernel_initializer='he_uniform'),
        tf.keras.layers.MaxPooling2D(2, 2),
        tf.keras.layers.Dropout(0.3),
        tf.keras.layers.Conv2D(60, (3,3), activation='relu',padding='same',kernel_initializer='he_uniform'),
        tf.keras.layers.MaxPooling2D(2, 2),
        tf.keras.layers.Dropout(0.3),
        tf.keras.layers.Flatten(),
        tf.keras.layers.Dense(412, activation='relu'),
        tf.keras.layers.Dense(1, activation='sigmoid')
])

from tensorflow.keras.optimizers import Adam

model.compile(loss='binary_crossentropy',
        optimizer=Adam(),
        metrics=['BinaryAccuracy'])

and trained it:
history = model.fit_generator(train_generator,
                              epochs=450,
                              callbacks=[early_stopping],
                              verbose=1,
                              validation_data=validation_generator)

However the model doesn't improve at all or improve its accuracy on either the training or validation datasets.
Epoch 1/450
134/134 [==============================] - 116s 846ms/step - loss: 3993.2059 - binary_accuracy: 0.5017 - val_loss: 0.7324 - val_binary_accuracy: 0.4792
Epoch 2/450
134/134 [==============================] - 117s 845ms/step - loss: 0.8022 - binary_accuracy: 0.5158 - val_loss: 0.7285 - val_binary_accuracy: 0.4784
Epoch 3/450
134/134 [==============================] - 99s 719ms/step - loss: 0.7572 - binary_accuracy: 0.5128 - val_loss: 0.7252 - val_binary_accuracy: 0.4762
Epoch 4/450
134/134 [==============================] - 92s 666ms/step - loss: 0.7219 - binary_accuracy: 0.5165 - val_loss: 0.7219 - val_binary_accuracy: 0.4750
Epoch 5/450
134/134 [==============================] - 93s 671ms/step - loss: 0.7132 - binary_accuracy: 0.5177 - val_loss: 0.7134 - val_binary_accuracy: 0.4734
Epoch 6/450
134/134 [==============================] - 93s 677ms/step - loss: 0.7080 - binary_accuracy: 0.5160 - val_loss: 0.7009 - val_binary_accuracy: 0.4826
Epoch 7/450
134/134 [==============================] - 93s 675ms/step - loss: 0.6989 - binary_accuracy: 0.5205 - val_loss: 0.7013 - val_binary_accuracy: 0.4856
Epoch 8/450
134/134 [==============================] - 93s 676ms/step - loss: 0.6947 - binary_accuracy: 0.5162 - val_loss: 0.7015 - val_binary_accuracy: 0.4852

What can I do to make the CNN learn? I am not sure if there is a problem with the way I am loading my dataset or with the way I am running the model.

A rule of thumb, as you go deeper, number of filters increase and the size of filter remains same or increases. You don't follow both of them. This will help your network learn.
Then, consider increasing the number of filters in proper fashion if still your network is not learning.


Manual install vs conda install tensorflow-gpu Asking mainly for Ubuntu 16.04/18.04.
What's the difference between installing CUDA and cuDNN together with tensorflow-gpu in conda (conda install tensorflow-gpu), and installing it all by hand and then using pip?
Does it mean CUDA and cuDNN are only available in the environment where conda install tensorflow-gpu was invoked?
Also, why do some people use the -c flag when installing it?

Apparently all Cuda and CuDnn libraries are included with packages from Anaconda Cloud according to this guy:  https://www.pugetsystems.com/labs/hpc/How-To-Install-CUDA-10-together-with-9-2-on-Ubuntu-18-04-with-support-for-NVIDIA-20XX-Turing-GPUs-1236/

  You do NOT need to install CUDA 10 if; You are a developer working
  with CUDA 9.x or earlier in your code base and there are no compelling
  new features that you want to risk breaking your application to try.
  See the release notes. You are working with Python machine learning
  frameworks AND using Anaconda Python. Most of the frameworks have
  up-to-date builds available from Anaconda Cloud that include the
  needed CUDA libraries i.e. you don't have to install anything outside
  of the package you want. You are using GPU accelerated software
  packages that only depend on CUDA run-time support from the display
  driver. That includes most "ready to use" or commercial software. You
  don't know what CUDA is. In that case then, well, thanks for reading
  my post. I hope you enjoy it and learn something from it.

Check out this advice:  https://towardsdatascience.com/tensorflow-gpu-installation-made-easy-use-conda-instead-of-pip-52e5249374bc

He says just create a conda env like conda create --name tf_gpu tensorflow-gpu where tf_gpu is the env name and tensorflow-gpu is to include cuda and cudnn in that env.  Breaking it down into separate commands, it looks like:

conda create --name tf_gpu
activate tf_gpu
conda install tensorflow-gpu


Stanford NER is not properly extracting percentages I'm trying to extract percentages using Stanford NER. But it is not extracting percentage properly.

import os
from nltk.tag import StanfordNERTagger
os.environ['CLASSPATH'] = 'stanford-ner-2015-12-09/stanford-ner.jar'
os.environ['STANFORD_MODELS'] = 'stanford-ner-2015-12-09/classifiers'

inp_str = 'total revenue received was one hundred and twenty five percent 125% for last financial year'
split_inp_str = inp_str.split()
st = StanfordNERTagger('english.muc.7class.distsim.crf.ser.gz')
print(st.tag(split_inp_str))

This gives following output

[('total', 'O'), ('revenue', 'O'), ('received', 'O'), ('was', 'O'), ('one', 'O'), ('hundred', 'O'), ('and', 'O'), ('twenty', 'O'), ('five', 'PERCENT'), ('percent', 'PERCENT'), ('125%', 'O'), ('for', 'O'), ('last', 'O'), ('financial', 'O'), ('year', 'O')]

Expected ouput:

[('total', 'O'), ('revenue', 'O'), ('received', 'O'), ('was', 'O'), ('one', 'PERCENT'), ('hundred', 'PERCENT'), ('and', 'PERCENT'), ('twenty', 'PERCENT'), ('five', 'PERCENT'), ('percent', 'PERCENT'),('125%', 'PERCENT'), ('for', 'O'), ('last', 'O'), ('financial', 'O'), ('year', 'O')]

Why is it not extracting 125% or one hundred and twenty five percent?

Looking at your code, I dont believe there is any mistake in it. The Stanford NER tagger is modelled by a Conditional Random Field Classifier and is hence bound to make mistakes. If you are able to check for a large number of samples and calculate an accuracy metric on the basis of that you might get a good idea if the pre-trained model is useful for you. Else, I suggest that you train your own model using. The stanford NLP group provide an implementation for training, you just have to give it the data in the required format. Look at this blog to get a good idea on how to do it.


Bounding Boxes in YOLO Model The YOLO model splits the image into smaller boxes and each box is responsible for predicting 5 bounding boxes.

My question is how does the model make these bounding boxes for every grid cell ? Does each box have a predefined offset with respect to say the center of the grid cell.

I AM NOT TALKING ABOUT THE FINAL BOUNDING BOX THAT ENCLOSES THE OBJECT
I am talking about the 5 predicting bounding boxes that are present for each grid cell.

Like for example if the smaller grid cell is located at say 50x50 (the center of it) then the bounding boxes should be at (50+5)x(50+5) or something like that

If not then how do the bounding boxes come to be ?

Paper - https://arxiv.org/pdf/1506.02640.pdf

Andrew Ng's explanation actually covers the YOLOv2 which uses anchor boxes. YOLOv1, which is the paper you linked, does not use anchor boxes so its not exactly the same.

They key to understanding how the bounding boxes are formed is to first understand how the output is encoded. To which, I'll recommend this link: https://hackernoon.com/understanding-yolo-f5a74bbc7967

Briefly speaking, and I'll be using the example from the paper, for S=7, B=2 and C=20, our output is a 7x7x30 tensor that encodes where (bounding box coordinates) and what the objects (probability of class) are. To achieve this, we construct a fully-connected layer at the end of our CNN that will give us 7x7x30 (rather forcefully). Hence on our first forward pass, each cell will have 2 random bounding boxes. A loss is calculated. The weights of the CNN will then be adjusted according to reduce that loss (opitimisation). Then the following passes will produce bounding boxes closer to the ground truth.


Why does discriminiator accuracy falls to 0%, and is there a fix around this? I am training a Vanilla-GAN(or original GAN 2016) on a pokemon dataset https://www.kaggle.com/kvpratama/pokemon-images-dataset, for few epochs the discriminator has 100% accuracy over the real examples, but as the epochs pass it falls to 0% around 4-5 epochs.

One interesting effect however is that when I increase my batch size from 32 to 64, this effect is seemed to have a delayed effect. i.e It kicks in around 400-500 epochs.

Please help.

I figured out my answer a while back. SO here it goes...

Ensure that your training data is normalized. This was what causing the discriminator's accuracy to fall.
You need to decrease your dropouts as it can cause heavy bias if done ineffectively.
Remove any unnecessary layers in your discriminator as they can cause overfitting.
Use batch normalization.
Introduce dropouts in your generator.


How to build an encoder using a distance matrix I have a similarity/distance matrix:

   a | b | c
 a 0 | 1 | 2
 b 1 | 0 | 3
 c 2 | 3 | 0

I want to build an encoder/model that learns an n-dimensional representation of each of the points in the dataset s.t. the euclidean-difference between the representations produces the difference provided in the matrix, e.g. distance(a,b) = 1 etc.

Any ideas?

This is essentially what t-SNE does. It supports transforming data from any input dimension to any output dimension, and it tries to preserve distances between all pairs of points.

From the author's website:

  Can I use a pairwise Euclidean distance matrix as input into t-SNE?

  Yes you can! Download the Matlab implementation, and use your pairwise
  Euclidean distance matrix as input into the tsne_d.m function.


Data Science Pipelines vs Common CD/CL What is the advantage of Data Science Specific CI/CD (kubeflow, Algo, TFX, mlflow, sagemaker pipelines) vs the already baked flavors that are more generic: Jenkins, Bamboo, Airflow, Google Cloud Build, ...

My guess is the Data Science ones give more structure around the common ML operations and are better optimized for compute and memory needed to train, deploy, doing things in parallel and run inference on models?

TData science pipelines are designed to manage the end-to-end data lifecycle (e.g., clean data, fit model, and serve model).
CI/CD pipelines are more general purpose software engineering tools around the automation of common tasks (e.g., running a testing suite).
The advantage of using data science pipelines is that the pipelines have already primitives for common data science tasks. The disadvantages are that the tools are often immature and require a user to adhere to the tools established workflow.
In general, both systems do not optimize for compute or memory. They are just code runners. The code or platform would have to optimize for compute or memory.
These system can sometimes parallelize the parts of the code that is easy to parallelize. For example, serving models is often independent for different users.


How to proceed 2 executions in 1 step in hive? I am wondering if there is a way to proceed 2 exectuions in 1 step in hive.
For example:

SELECT * FROM TABLE1
SELECT * FROM TABLE2
;

Do this in one window, and do not have to open 2 hive windows to execute each line separetly.

Can it be done on HUE?

You can use HiveCLI Tool to run HiveQL with a given sql file.

  $HIVE_HOME/bin/hive -f /home/my/hive-script.sql

Please see official document: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli

What you need to do is to

Put your HiveQLs in a file as below
SELECT * FROM TABLE1;
SELECT * FROM TABLE2;
Use HiveCLI and run with above file


What are the possible ways to handle class unbalance in a large scale image recognition problem with Deep Neural Nets? I have 22 classes of objects but they have very skewed distributions where max class has 100.000 images and the min class has 1600 images. In that setting I would like to hear some possible solutions to this balance problem.

I have tried followings so far;

multiply number of instances in the lower classes up to the max class by replicating the instances, possibly adding some noise as well.
changing the learning regarding the class distribution in the given minibatch of the next epoch. (no implemented but in my mind)

What are your suggestions?

You can also modify your cost function so that instances in the smaller class have more weight. This makes training computationally less demanding, when compared to your option 1.


Architecture for linear regression with variable input where each input is n-sized one-hot encoded I am relatively new to deep learning (got some experience with CNNs in PyTorch), and I am not sure how to tackle the following idea. I want to parse a sentence, e.g. I like trees., one-hot encoded the parse output of each word, and feed that into a ML system. The output of each sentence is a floating-point number. As an example, the sentence I like trees. could be pre-processed and encoded as fixed-size feature vectors per token:

[[0 1 0 0 1] [1 0 0 0 0] [1 0 1 1 0] [0 0 0 0 1]]

or flattened

[0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 0 0 1]

However, the length of sentences can differ of course. From what I know there are some fixes to this. Using padding (when shorter than your defined cut-off) or cutting off the length to that cut-off (when longer). Another solution I often see mentioned is the use of an RNN/LSTM but I have no experience with them (only the basic theoretical notion).

The expected output (label) of this sentence could be something like 50.2378.

My question, then, is which architecture is best suited for this task? RNNs are popular in NLP so I am leaning towards them, but I am not sure whether a regression task fits well (or how it fits) with the architecture of an RNN. I have looked for RNNs and regression but I can only find use cases involving time series, not NLP or one-hot encoded features.

In essence, what I have is a dataset of sentences that are preprocessed to get some features per token. Let's assume for brevity sake that these features are syntactic, e.g. the word's POS tag and its dependency information (e.g. subj, obj, and so on). These features would then get one-hot encoded (I assume) to get an easy-to-use dataset. The input, thus, are sentences encoded in such a way that the information of tokens is shown, such as the flattened example above.

The output for every sentence is some floating-point number, representing some value that has been calculated beforehand in theory these will indicate the sentence's equivalence with its translation, but that is not at all important for the system. The important part is that every input is a sentence mapped to a number.

Some dummy data. The actual sentence, it's vector representation, and the output.

I like trees.                   [0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 0 0 1]                           50.2378
What is that thing?             [1 1 0 1 0 0 0 0 1 1 1 0 0 0 0 1 1 0 1 0 0 1 1 0 1]               20.1237
Who are you?                    [0 1 0 0 0 0 0 1 0 0 1 0 1 0 0 0 1 1 0 1]                           1.6589
The cookies smell good today.   [0 0 1 0 1 1 0 1 1 0 1 1 1 0 1 1 1 0 1 0 1 0 1 1 1 0 0 0 0 1]     18.6295
I do!                           [0 1 0 0 1 1 0 1 1 0 0 0 1 0 1]                                     24.5489

The goal, then, is that an unseen given sentence can be pre-processed and one-hot encoded and given as input, and that an output (floating-point number) is predicted based on the trained model/function.

The output for every sentence is some floating-point number, representing some value that has been calculated beforehand (...) the important part is that every input is a sentence mapped to a number.

In essense this is a regression problem, i.e. given some input predict the (most likely) numeric output. However you could also see this as a classification problem if you are not interested in real numbers but in some range (=class), or a probability for each class.

  RNNs are popular in NLP so I am leaning towards them,

RNNs are well suited for sequence processing. The way you present the problem does not seem like there is much sequence processing involved - while you can get this to work with RNNs, I propose to try other less computationally involved approaches first.

  I have looked for RNNs and regression but I can only find use cases involving time series, not NLP or one-hot encoded features.

Unless there is some specific requirement to use RNNs or in extension LSTMs, I would try random forests, e.g. the RandomForestRegression regressor from scikit-learn. This should work reasonably well using the one-hot encoded sentences (as outlined in the question) as X and the pre-calculated number as Y. Should you decide to instead make this a classificastion problem, you can easily switch to the RandomForestClassification classifier from the same package.

  an unseen given sentence can be pre-processed and one-hot encoded and given as input, and that an output (floating-point number) is predicted

I suggest to use a Pipeline for both training and prediction, then serve this Pipeline to your application.


Plot a training/validation curve in Pytorch Training I have the following training method and I'm confused how may I modify the code to plot a training and validation curve history graph with matplotlib
def train(n_epochs, loaders, model, optimizer, criterion, use_cuda, save_path):
    """returns trained model"""
    since = time.time()

    # initialize tracker for minimum validation loss
    valid_loss_min = np.Inf
    train_loss_min = np.Inf

    for epoch in range(1, n_epochs + 1):
        print('\n')
        print('Running Epoch ', epoch, ' of ', n_epochs)
        print('\n')
        time_elapsed = time.time() - since
        print('Training for {:.0f}m {:.0f}s'.format(time_elapsed // 60, time_elapsed % 60))
        # initialize variables to monitor training and validation loss
        train_loss = 0.0
        valid_loss = 0.0

        # train the model
        model.train()
        for batch_idx, (data, target) in enumerate(loaders['train']):
            # move to GPU
            if use_cuda:
                data, target = data.cuda(), target.cuda()
            ## find the loss and update the model parameters accordingly
            optimizer.zero_grad()
            output = model(data)
            loss = criterion(output, target)
            loss.backward()
            optimizer.step()
            ## record the average training loss
            train_loss = train_loss + ((1 / (batch_idx + 1)) * (loss.data - train_loss))
            if batch_idx % 10 == 0:
                    print('Epoch ', epoch, ' Training batch ', batch_idx)
                    print('Train Loss ', train_loss)

        # validate the model
        model.eval()
        print('\n')
        print('             Evaluating the model')
        print('\n')

        for batch_idx, (data, target) in enumerate(loaders['valid']):
            # move to GPU
            if use_cuda:
                data, target = data.cuda(), target.cuda()
            ## update the average validation loss
            output = model(data)
            loss = criterion(output, target)
            valid_loss = valid_loss + ((1 / (batch_idx + 1)) * (loss.data - valid_loss))
            print('Valid Loss ', valid_loss)
        # print training/validation statistics
        print('\n')
        print('Epoch: {} \tTraining Loss: {:.6f} \tValidation Loss: {:.6f}'.format(
            epoch,
            train_loss,
            valid_loss
        ))
        print('\n')
        ## save the model if validation loss has decreased
        if valid_loss <= valid_loss_min:
            print('Validation loss decreased ({:.6f} --> {:.6f}). Saving model..'.format(valid_loss_min, valid_loss))
            torch.save(model.state_dict(), save_path)
            print('Model Saved')
            valid_loss_min = valid_loss

    time_elapsed = time.time() - since
    print('Training completed in {:.0f}m {:.0f}s'.format(time_elapsed // 60, time_elapsed % 60))

    return model

You should use Tensorboard. It has been integrated with PyTorch. See this.


Can we add other features along with Text in sentiment analysis Can we add other features along with Text to a ML model . Like giving text and other features as one input combining them and predict the output value. As model can learn some more better if given some extra features along with text from tf-idf vector matrix. Can we input other features along with tf-idf vector matrix to a ML model or DL model.

Yes you definitely can.
Here's an example: Using Convolutional Neural Networks to Classify Hate-Speech

The authors used classic embeddings concatenated with a vector of size 28 representing the presence or not (in a tweet) of each letter of the alphabet (26) plus any digit (1) plus any other symbol (1).
So basically for a tweet like 'I love NLP!' the representation would be:

(embeddings)(00001000100110110000000)(0)(1)

where you have the classic embeddings for each word, then a one hot vector for the alphabet letters, another hot vector (just a binary variable actually) for digits and one last that check for the presence of any other symbol (! in this case). The resulting concatenated vector is then passed to a LSTM, convolution layer or whatever architecture you want to use.

Consider that there are infinite other variables that you can concatenate to the embeddings. For example you could concatenate counters for each alphabet letter or any other symbol in a sentence (in this case I suggest to separately normalise these variables between 0 and 1 before concatenating them to the embeddings), one-hot representation of n-grams or dependencies, the total number of words, presence or not and counters of capital letters, presence of entities, sentiment scores obtained through external resources like Vader (see nltk) and so on.


Here is a solution using data.table package.

DF3 <- data.table(merge(DF1, DF2, by = "CATEGORY")) # merge tables by category
DF3$TIME_OK <- (DF3$TIME >= DF3$START & DF3$TIME <= DF3$END) # create a column to check time
DF3 <- DF3[DF3$TIME_OK == T,] # subset table
DF3 <- DF3[,.(NO_OF_OBJECT = .N), by = c("CATEGORY", "DATE", "TIME")] # aggregate results

Basically, it merges both data frames and subsets only relevant time period before aggregating results.


vgg16 needs less epochs than resnet ,why? Recently i Have been comparing the vgg16 with resnetv1 with 20 layers.I have found out that although each epoch on vgg takes more time to complete,it generally needs less epoch to reach a certain training accuracy than resnet20.Why vgg learns faster ? is my experiments correct ? I have tried it on Cifar100 and a proportion of imagenet(tiny image net from stanford cv course)

the vgg has nearly 14m parameters but resnet has only 0.3m.
here is my implementation of the resent:

def resnet_layer1(inputs,initializer,
             num_filters=16,
             kernel_size=3,
             strides=1,
             activation='relu',
             conv_first=True,batch_normalization=True):

conv = Conv2D(num_filters,
              kernel_size=kernel_size,
              strides=strides,
              padding='same',
              kernel_initializer=initializer
             )

x = inputs
if conv_first:
    x = conv(x)
    if batch_normalization:
        x = BatchNormalization()(x)
    x = Activation(activation)(x)
else:
    if batch_normalization:
        x = BatchNormalization()(x)
    x = Activation(activation)(x)
    x = conv(x)
return x
def resnet_1(model_number,x_train,y_train,x_test,y_test,datagen,initializer,epochs=20,bs=512,output_nodes=10,optim='adam',padding='same',dout=True,callbacks=None):

depth=20
# Start model definition.
num_filters = 16
num_res_blocks = int((depth - 2) / 6)

if model_number == 1:
    resnet_layer=resnet_layer1
elif model_number == 2:
    resnet_layer=resnet_layer2
elif model_number == 3:
    resnet_layer=resnet_layer3
elif model_number == 4:
    resnet_layer=resnet_layer4

inputs = Input(shape=x_train.shape[1:])
x = resnet_layer(inputs=inputs,initializer=initializer)
# Instantiate the stack of residual units
for stack in range(3):
    for res_block in range(num_res_blocks):
        strides = 1
        if stack > 0 and res_block == 0:  # first layer but not first stack
            strides = 2  # downsample
        y = resnet_layer(inputs=x,initializer=initializer,
                         num_filters=num_filters,
                         strides=strides)
        y = resnet_layer(inputs=y,initializer=initializer,
                         num_filters=num_filters,
                         activation=None)
        if stack > 0 and res_block == 0:  # first layer but not first stack
            # linear projection residual shortcut connection to match
            # changed dims
            x = resnet_layer(inputs=x,initializer=initializer,
                             num_filters=num_filters,
                             kernel_size=1,
                             strides=strides,
                             activation=None,
                             batch_normalization=False)
        x = layers.add([x, y])
        x = Activation('relu')(x)
    num_filters *= 2

# Add classifier on top.
# v1 does not use BN after last shortcut connection-ReLU
x = AveragePooling2D(pool_size=8)(x)
y = Flatten()(x)
outputs = Dense(output_nodes,
                activation='softmax',
                kernel_initializer=initializer)(y)

# Instantiate model.
model = Model(inputs=inputs, outputs=outputs)
model.summary()
# Compile the model
model.compile(loss=['categorical_crossentropy'], optimizer=optim, metrics=["accuracy"])

checkpointer = ModelCheckpoint(filepath=str(model_number)+'_weights.hdf5', verbose=1,save_weights_only=True,save_freq=2000000)

callbacks.append(checkpointer)

if x_test.all() == None:
    history=model.fit_generator(datagen.flow(x_train, y_train,batch_size=bs),callbacks=callbacks,epochs=epochs, steps_per_epoch=x_train.shape[0]//bs)
else:
    history=model.fit_generator(datagen.flow(x_train, y_train,batch_size=bs),callbacks=callbacks,epochs=epochs,validation_data=(x_test, y_test), steps_per_epoch=x_train.shape[0]//bs)

return history,model

For some reason VGG might be better suited for cifar10 (maybe kernel sizes etc.). Generally speaking, however, this isn't the case. I've trained VGGs much slower than even the largest resnets (i.e. 150 layers).


Recommend tags for documents I'm working on a unusual issue (for me) and I need some advice.

My goal is to have a recommendation algorithm that propose tags for a document, based on all the previously tagged documents. For example, if I upload each month an invoice from "stackoverflow", and I apply each time the same tags ("stackoverflow" and "invoice" for instance), the algorithm should recognize the document as similar to previous stackoverflow invoices and suggest the same tags to the user.

I was thinking about extracting some features of the document ( text, sender's email...), compare the similarity with previous documents and suggest the tags of the most similar documents.

The tags are user-defined, and thus not a finite set, so the algorithm should learn continuously. This is my issue, because I will have to store all the documents in a database and evaluating the similarity between a new document and all the previous ones will eventually be very expensive as the number of documents in my database grow.

On the other side, incremental learning algorithms can learn continuously without a massive database, but don't support an non-finite set of output classes.

So, as I see it, I have 2 options :

massive database with all documents, and evaluating similarity
between a new document and each document in the database : not very
scalable, possibly time-expensive
re-train from scratch a classification algorithm every time a new document is uploaded : I will still have a massive database, but training can be done in the background. This is the best of the 2 options, but I'm wondering if there could be a better way.

Any advice about more appropriate algorithm would be greatly appreciated. If it matters, I work with Python.

This is where Word Embedding play an amazing role. If you want to solve the problem you have at hand with Natural Language Processing approach, what you can do is learn vector representations for these documents and store them with the labels i.e tags for that text.

You can then use the similarity of vector representations between documents, to predict the tags.

This way, your computation is not intensive at all and you can keep retraining as and when you get more data.


Neural network accuracy in Torch depends on compute power? I am new to machine learning and have quite good understanding of basic concepts.

I was implementing a 3 Layer neural network on MNIST dataset with 784, 100, 10 neurons in input, hidden, output layer respectively. I did not use any regularization here.

Firstly I trained the network on Intel i5 4th generation quad-core CPU with 4GB of ram which gave me 64% of accuracy.
Then I trained the exact same network, with exact same code on Intel i5 7th generation quad core CPU with 8gb of ram which gave accuracy of about 89%
This is the link to the implementation.

My question is in Torch, does the compute power effect the accuracy of the network? or is it something else that I am missing which has resulted in this huge change.

I did not use any different weight initialization method than the default provided in the torch libraries, so that is ruled out.
I also did not use anything else which may effect the network to change its accuracy to this extent.

Available compute power does not directly affect the accuracy of a neural network. If your different runs of the network have:

identical architecture and meta-params
identical code (including library code)
all training data is identical
all stochastic parts of training use the same random seed and generator
all data types are identical precision (e.g. all vectors and matrices are 32-bit or 64-bit floats)

then the behaviour of neural network training in each run is fully deterministic and repeatable. Having a faster processor will just get you to the result faster*.

The most likely difference between your tests is due to not seeding the random number generators used in the training the process. For you this includes weight initialisation, possibly train/test split and possibly shuffling training data in each epoch. As you did not use any regularisation, then accuracy of the trained network can vary quite a bit due to over-fitting.

To verify this, you can train a second or third time on each CPU. I expect you will see a lot of variation in final accuracy, regardless of which machine you run it on.

* This does mean that having a faster machine can result in you having a more accurate final network in practice when you are tuning the parameters, because you can try more variations of meta-params with multiple training sessions.


Classify pdf files - image approach vs. text approach I'm about to start a project with the objective to classify PDF-documents. I'm wondering if there's a best practice approach to tackle this problem.

Concretely I'm wondering if one of the following two approaches performs usually better:

use an OCR reader to convert the files to text and train the classifier model on the text data
convert the files to images and train a CNN classifier model

I'm planning to classify different testimonials and certificates mostly.
Since most of these files share a similar layout and text within the classes the ideas should both work out. I'm wondering if anybody has already made experience with this and could tell me about some advantages/disadvantages when using a specific approach.

I'd highly appreciate any kind of help.

Both methods will be beneficial for different cases.

If you think that dependencies in the text are more discriminative of the classes, then NLP apporach. Image approach in this case will need to be super complex to catch this kind of information.

On the other hand layout and position could be very informative and this it could happen is not encoded in text and only image approach can catch this information.

Conclusion. Maybe think about hard-encoding layout and position features and pass them along to NLP model.


Solving classification task with deep network I am currently working on a classification problem:

The dataset (2000 features, 25 labels) I am using is seperable by using a 2-layered Multi-Layer-Perceptron (1 Input + 1 OutputLayer = 1 Weight Matrix) with achieving great accuracy.

Note: The Data I am using are spectrograms.

Now what I am curious about is the following: Can I use a deep network (more than 2 layers) to even increase the accuracy?

What I think about: Increasing the network structure would make the network interpret the data in a more complex way and therefore should be able to even correctly classify little "outbrakers" and "noisy elements" of the dataset.

If you are using two hidden layers, it may mean that your data is not linearly separable. If you just use one unit in a single hidden layer then you can claim that your data is linearly separable, which in your case you may say that there exist a hyper-plane that separates your data linearly.

As the answer to your question, yes. It is possible to reduce the amount of error by adding layers or adding extra units to the current architecture but there are some points that are necessary to be considered.

Whenever you add more layers, there will be vanishing and exploding gradients which may cause your network not to learn, or learning may happen so slowly. To avoid, you should use ReLU activation in order to avoid saturation of gradients. Moreover you have to use He or Xavier initialization techniques to avoid having bad random weights. There are other techniques for solving this problem which are called skip connections but at least I've never seen the use of them in MLPs Although they are really helpful for solving the mentioned problem.
Covariat shift is the problem of learning for deeper layers. As a solution you have to use Batch Normalization to somehow normalize the activations of the deeper layers.
Overfitting is a problem that happens for large architectures that are not fed with enough training data. You should use regularization techniques. There are so many papers about this but in your case which is using MLP I highly recommend you using Dropout technique which is invented by the so called God Father of deep learning.


How does R-CNN and AlexNet compare? I know AlexNet does object classification in images [categories] and R-CNN does object localization [category and bounding box].
How does R-CNN and AlexNet compare?

Are they used for the same purpose or R-CNN does more? Does R-CNN use Alexnet as a sub-module?

I think you got most of it from the way you wrote your question.

How does R-CNN and AlexNet compare?
Are they used for the same purpose or R-CNN does more?

They are different things. AlexNet is a CNN architecture, i.e a neural network with a specific set of layers. R-CNN is multistep method that does object localization and classification using a search algorithm with a CNN (of any architecture) and a SVM.

Does R-CNN use Alexnet as a sub-module?

R-CNN can use AlexNet, or other CNN architectures, as its' feature extractor.


How to set a newtwork with two objectives? Suppose I have a x_train, y1_train and y2_train.

I want to construct a network (such as simple MLP) to fit y1_train and to be low correlated with y2_train (or to fit -y2_train) simultaneously.

How could I achieve this goal? Is the custom loss function a good solution?

I use keras as my tool.

So this problem is less of a Deep learning problem and more of a logical problem. Your y1_train and y2_train can be modelled along with a pointer label that points what output to be considered in the output. Lets say we create the concatenated output as follows:

[[0/1],[y1_train], [y2_train]]

Where 0 could represent weather the label to be selected is y1 or y2 and so on.

But if you are planning to create a little more complex output and train different outputs on different loss functions, here is an article you should refer to Multiple output tutorial/examples and Custom Loss Function for Unequal Weighted Multiple-Output Node Regression


What is the reason behind taking log transformation of few continuous variables? I have been doing a classification problem and I have read many people's code and tutorials. One thing I've noticed is that many people take np.log or log of continuous variable like loan_amount or applicant_income etc.

I just want to understand the reason behind it. Does it help improve our model prediction accuracy. Is it mandatory? or Is there any logic behind it?

Please provide some explanation if possible. Thank you.

This is done when the variables span several orders of magnitude. Income is a typical example: its distribution is "power law", meaning that the vast majority of incomes are small and very few are big.

This type of "fat tailed" distribution is studied in logarithmic scale because of the mathematical properties of the logarithm:

$$log(x^n)= n log(x)$$

which implies

$$log(10^4) = 4 * log(10)$$

and

$$log(10^3) = 3 * log(10)$$

which transforms a huge difference $$ 10^4 - 10^3 $$ in a smaller one $$ 4 - 3 $$
Making the values comparable.


Advise resources on un-supervised learning I have seen that people coming into data science will rush into scikit-learn or other libraries without trying to learn the knowledge behind. Its good to follow a top-down approach but most of times people don't go into depth, even if they do its in supervised-learning area. That't why I think un-supervised is neglected (or is it heavy influence of statistical inference). I would like to learn the un-supervised learning in-depth I tried some books so far (Sugiyama, Masashi - Introduction to Statistical Machine Learning-Morgan Kaufmann (2016) book) was better among others to build intuition around concepts.
If anyone have gone through same path and know other good reads or video tutorials kindly share in this thread.

I would recommend going through Towardsdatascience and Medium articles. They can explain the methods in a very simple and light way by summarizing the theory and also inlcuding the practical side of things.
for example: https://towardsdatascience.com/introduction-to-unsupervised-learning-8f1b189e9050
If you want to dive more into the maths the famous ESLII (Elements of Statiscal Learning) https://web.stanford.edu/~hastie/Papers/ESLII.pdf and Pattern Recognition  of Bishop (http://users.isr.ist.utl.pt/~wurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf) are the way to go. These two are the "bible" of Machine Learning


Is a good shuffle random state for training data really good for the model? I'm using keras to train a binary classifier neural network. To shuffle the training data I am using shuffle function from scikit-learn.
I observe that for some shuffle_random_state (seed for shuffle()), the network gives really good results (~86% accuracy) while on others not so much (~75% accuracy). So i run the model for 1-20 shuffle_random_states and choose the random_state which gives the best accuracy for production model.
I was wondering if this is a good approach and with those good shuffle_random_state the network is actually learning better?

If this is split is a train/validation split (not a hold out test set) then you should be doing cross-validation. You are going to be overly optimistic about the performance of your model for this set of features and hyperparameters if you try to split it "just right". Cross-validation will give you a more accurate portrayal regardless of your split. If this is for a train/test split (test being a hold out test set), this is a very bad practice, since you are informing your decision on how to make the split based on the performance of the test set.


Using PCA as features for production I struggle with figuring out how to proceed with taking PCA into production in order to test my Models with unknown samples.
I'm using both an One-Hot-Encoding an an TF-IDF in order to classify my elements with various models, mainly KNN.
I know i can use the pretrained One-Hot-Encoder and the TF-IDF encoder in order to encode the new elements in order to match the final feature Vector.
Since these feature vectors become very large i use an PCA in order to reduce the dimension of them.
This is an example of how i currently pre process the features:
if method == "pca_tfidf":
        df['tfidf'], featurenames= hero.clean(df['OpcodeString']).pipe(hero.tfidf, return_feature_names=True)
        X = df['tfidf']
        X = pd.DataFrame(item for item in X)
        if maxFeatures >= X.shape[1]:
            maxFeatures = X.shape[1]
        y = df["%n class"] # Split label from data
        ActualY = df["actual class"]

        Final_PCA = PCA(n_components= maxFeatures ,random_state=42)
        Final_PCA.fit(X)
        X_pca = Final_PCA.transform(X)

        total_var = Final_PCA.explained_variance_ratio_.sum() * 100
        printPCA(X_pca, total_var)

        X = pd.DataFrame(item for item in X_pca)

Now what i don't understand is how to proceed with new unknown samples. Since PCA is calculated in relation to every element in the train dataset, would I have to recalculate every component new with all new samples? Wouldnt that falsify with what values the model has been trained?
Can i just use Final_PCA.fit(X_new) with the TFIDF Values of the unknown samples in order to produce a feature vector which the model can then classify?
Additionally im questioning myself weather my approach is good at all. I have read a few times that PCA seems to not be great for categorical feature values. However i was able to achieve great results for my approach when i calculate the TFIDF and PCA first and then do the Test/Train split afterwards.
Thanks for any help!

I think you made a mistake here:

i was able to achieve great results for my approach when i calculate the TFIDF and PCA first and then do the Test/Train split afterwards.

This evaluation was flawed due to data leakage: both the TFIDF and the PCA should be calculated using only the training set.
This is more or less the source of your issue with how to proceed in production: normally in production one should just apply the same process as with your test set, but here you were able to prepare your test set differently (and wrongly).

For the TFIDF step, the IDF should be computed only using the training set. Note that in the test set you might have to deal with out of vocabulary words.
For the PCA step, I'm not too sure how to proceed but I found this explanation.


Euclidean distance between time series in Python While thinking about similarity between two time series, one can use DTW to approach the issue. There is a Python package for that mlpy.

It is also said to compare time series via simple euclidean distance.

Is there a Python package to this?

This only makes sense if you look at the time series as vectors with same length. In that case, there is no need for a package. Just use, for example,
numpy.linalg.norm(a-b).


Is it possible to generate syllogisms using an NLP algorithm? I want to build a tool that generates sensible syllogisms. An example of a syllogisms is: all A are B. all C are A. all C are B). I want the triplet (A, B, C) to be semantically related to each other in the way described by the syllogisms. That is, I would like my tool to generate 'a=humans; b=mortal; c=greeks' and not 'a=chickens; b=burgers; c=frogs'.

There's a syllogism generator online (http://krypton.mnsu.edu/~jp5985fj/courses/609/Logic/Silly%20Syllogisms.htm) but it doesn't generate syllogisms that are semantically related, it generates random terms for A, B, and C which may or may not be related.

My question is, in NLP, are there any research papers for generating semantically valid syllogisms? What topics would I need to research to build this tool?

That would probably be related to textual entailment and also relation extraction.

I'm not aware of any specific work but I would check in the biomedical domain, because there are resources such as SemRep and I wouldn't be surprised if people tried to use it for similar purposes.


Kmeans: Between class intertia I'm using KMeans of scikit-learn for clustering of forums.

The attribute inertia_ of model gives the within class inertia but how can i get the between class inertia? In other word how can I verify the quality of my clustering?

Thanks

Total variance = within-class variance + between-class variance.

i.e. if you compute the total variance once, you can get the between class inertia simply by

between-class variance = total variance - within-class variance


Dealing with events that have not yet happened when building a model I was building a model that predicts user churn for a website, where I have data on all users, both past and present.

I can build a model that only uses those users that have left, but then I'm leaving 2/3 of the total user population unused.

Is there a good way to incorporate data from these users into a model from a conceptual standpoint?

This setting is common in reliability, health care, and mortality. The statistical analysis method is called Survival Analysis. All users are coded according to their start date (or week or month).  You use the empirical data to estimate the survival function, which is the probability that the time of defection is later than some specified time t.

Your baseline model will estimate survival function for all users.  Then you can do more sophisticated modeling to estimate what factors or behaviors might predict defection (churn), given your baseline survival function.  Basically, any model that is predictive will yield a survival probability that is significantly lower than the baseline.

There's another approach which involves attempting to identify precursor events patterns or user behavior pattern that foreshadow defection. Any given event/behavior pattern might occur for users that defect, or for users that stay. For this analysis, you may need to censor your data to only include users that have been members for some minimum period of time. The minimum time period can be estimated using your estimate of survival function, or even simple histogram analysis of the distribution of membership period for users who have defected.


How to print a Confusion matrix from Random Forests in Python I applied this random forest algorithm to predict a specific crime type. The example I took from this article here.

import pandas as pd
import numpy as np
from sklearn.preprocessing import LabelEncoder
import random
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import GradientBoostingClassifier
import matplotlib
import matplotlib.pyplot as plt
import sklearn
from scipy import stats
from sklearn.cluster import KMeans
import seaborn as sns
# Using Skicit-learn to split data into training and testing sets
from sklearn.model_selection import train_test_split
# Import the model we are using
from sklearn.ensemble import RandomForestRegressor
import os
os.environ["PATH"] += os.pathsep + 'C:/Program Files (x86)/Graphviz2.38/bin/'

features = pd.read_csv('prueba2.csv',sep=';')
print (features.head(5))

# Labels are the values we want to predict
labels = np.array(features['target'])
# Remove the labels from the features
# axis 1 refers to the columns
features= features.drop('target', axis = 1)
# Saving feature names for later use
feature_list = list(features.columns)
# Convert to numpy array
features = np.array(features)

# Split the data into training and testing sets
train_features, test_features, train_labels, test_labels = train_test_split(features, labels, test_size = 0.25, random_state = 42)

baseline_preds = test_features[:, feature_list.index('Violent crime')]
# Baseline errors, and display average baseline error
baseline_errors = abs(baseline_preds - test_labels)
print('Error: ', round(np.mean(baseline_errors), 2))

# Instantiate model with 1000 decision trees
rf = RandomForestRegressor(n_estimators = 1000, random_state = 42)
# Train the model on training data
rf.fit(train_features, train_labels);

# Use the forest's predict method on the test data
predictions = rf.predict(test_features)
# Calculate the absolute errors
errors = abs(predictions - test_labels)
# Print out the mean absolute error (mae)
print('Promedio del error absoluto:', round(np.mean(errors), 2), ' Porcentaje.')

# Calculate mean absolute percentage error (MAPE)
mape = 100 * (errors / test_labels)
# Calculate and display accuracy
accuracy = 100 - np.mean(mape)
print('Precision:', round(accuracy, 2), '%.')

# Get numerical feature importances
importances = list(rf.feature_importances_)
# List of tuples with variable and importance
feature_importances = [(feature, round(importance, 2)) for feature, importance in zip(feature_list, importances)]
# Sort the feature importances by most important first
feature_importances = sorted(feature_importances, key = lambda x: x[1], reverse = True)
# Print out the feature and importances
[print('Variable: {:20} Importance: {}'.format(*pair)) for pair in feature_importances];

# Import tools needed for visualization
from sklearn.tree import export_graphviz
import pydot
# Pull out one tree from the forest
tree = rf.estimators_[5]
# Import tools needed for visualization
from sklearn.tree import export_graphviz
import pydot
# Pull out one tree from the forest
tree = rf.estimators_[5]
# Export the image to a dot file
export_graphviz(tree, out_file = 'tree.dot', feature_names = feature_list, rounded = True, precision = 1)
# Use dot file to create a graph
(graph, ) = pydot.graph_from_dot_file('tree.dot')
# Write graph to a png file
graph.write_png('tree.png')

So my question is: how can I add a Confusion matrix to measure accuracy? I tried this example from this here, but it doesn't work. The following error appears:

Any advice?

From the code and task as your present it, a confusion matrix wouldn't make sense. This is because it shows how well a model is classifying samples i.e. saying which category they belong to. Your problem (as the author in your link states) is a regression problem, because you are predicting a continuous variable (temperature). Have a look here for more information.

In general, if you do have a classification task, printing the confusion matrix is a simple as using the sklearn.metrics.confusion_matrix function.

As input it takes your predictions and the correct values:

from sklearn.metrics import confusion_matrix

conf_mat = confusion_matrix(labels, predictions)
print(conf_mat)

You could consider altering your task to make it be a classification problem, for example by grouping the temperatures in to classes of a given range.

You could say transform the target temperature to be a new_target_class, then change your code to use the [RandomForestClassifier][3].

I have done a quick and dirty conversion on the same data linked in that article, check it out here. I basically use the minimum and maximum values of the target variable to set a range, then aim for 10 different classes of temperature and create a new column in the table which assign that class to each row. The top it looks like this (click on picture to enlarge):

If you can get those predictions going using the RandomForestClassifier, you can then run the confusion matrix code above on the results.


Classification accuracy based on top 3 most likely classifications My goal is to recommend jobs to job seekers based on their skill set.

Currently I'm using an SVM for this, which is outputting one prediction, e.g. "software engineer at Microsoft". However, consider this: how significantly different are the skill sets of a software engineer at Microsoft and a software engineer at IBM? Probably not significantly different. Indeed, by inspection of my data set I can confirm this. Hence, the SVM struggles to discriminate in situations like this, of which there are many in my data set, and my classification accuracy is about 50%.

So I had an idea.

In SK Learn, once you've trained some model, you can compute the probability a particular input X belongs to each class.

So for each input X in my test set, I took the the top 3 most likely classifications. Then I tested whether or not the correct label was in the top 3 predictions. If it was, then I considered the prediction to be correct. In doing so, the classification accuracy increased to over 80%.

So my question is: is this a valid approach to measuring classification accuracy? If it is, then does it have a name?

In my mind, it is valid given my intended application, which is to recommend a selection of jobs to a job seeker, which are relevant to their skill set.

Cross posted from CS SE:
https://cs.stackexchange.com/questions/117695/classification-accuracy-based-on-top-3-most-likely-classifications

I'm interested to know what perspective data scientists have on this.

Yes, it's common to consider that the prediction is made of multiple answers (typically top N most relevant answers) and use a performance measure based on that.

Currently you're treating the problem as a classification problem but logically this is more like a recommendation problem or an information retrieval problem (like results from a search engine). Usually for this kind of problem the gold answer would also consists of a list of several items, but apparently your dataset contains a single answer for every instance.

Answer to comment: a couple of papers using some top N performance measures (note: it's just a quick selection based on the keyword "information retrieval")

https://www.microsoft.com/en-us/research/publication/letor-benchmark-collection-research-learning-rank-information-retrieval/
http://informationr.net/ir/18-2/paper582.html

The CLEF series of Shared Tasks have proposed many datasets and evaluation measures across the years, it's probably a good source for resources and papers... if you have a bit of time to explore it ;)


implementing conv2d in fourier domain using np.einsum --> ValueError: einstein sum subscripts string contains too many subscripts for operand 0 According to the convolution theorem, convolution operation changes to pointwise multiplication in fourier domain - here I have 'fft_x' of shape (batchsize, height, width, in_channels) which is the fft of input data and similarly 'fft_kernel' of shape (height, width, in_channels, out_channels) which is fft of the kernel after being padded to image size. To get pointwise multiplication of these in efficient way, I was using einsum in the following way -

...
    print(fft_x)
    print(fft_kernel)
    output = 0
    n=int(self.no_of_kernels/2)+1      # n = out_channels here
    for i in range(n):
        output += np.einsum('ijkl,jkl->ijk', fft_x, fft_kernel[i])
    return output
...

It gives the following output and error-

Tensor("input_11:0", shape=(?, 28, 28, 1), dtype=complex64)
Tensor("fourier__conv2d_11/transpose:0", shape=(28, 28, 1, 17), dtype=complex64)
...
...
ValueError: einstein sum subscripts string contains too many subscripts for operand 0

Could anyone please explain why this error is arising? Thanks in advance for any help.

import tensorflow as tf
output  = tf.einsum('ijkl,jklo->ijko', fft_x, fft_kernel)

I used tf.einsum instead of np.einsum, and it worked.
Also as seen in above, I removed the loop and changed the equation in einsum.


Ignoring symbols and select only numerical values with pandas In one field I have entries like 'U$ 192,0'. Working on pandas, how I ignore non numerical data and get only the numerical part?

Use str.strip if the prefix is fixed or str.replace if not:

data = pandas.Series(["U$ 192.0"])
data.str.replace('^[^\d]*', '').astype(float)

This removes all the non-numeric characters to the left of the number, and casts to float.


RNN using multiple time series I am trying to create a neural network using time series as input, in order to train it based on the type of each series.
I read that using RNNs you can split the input into batches and use every point of the time series into individual neurons and eventually train the network.

What I am trying to do though is use multiple time series as an input. So for example you might receive input from two sensors. (So two time series), but I want to use both of them in order to get a final result.

Also I am not trying to predict future values of the time series, I am trying to a get a classification based on all of them.

How should I approach this problem?

Is there a way to use multiple time series as an input to an RNN?
Should I try to aggregate the time series into one?
Or should i just use two different neural networks? And if this last approach is correct, if the number of time series increases wouldn't that be too computer intensive?

Multivariate time series is an active research topic you will find a lot of recent paper tackling the subject.

To answer your questions, you can use a single RNN. You can input one value for each time step. Nothing keeps you from adding another value at each time step (if your sensor are synchronized). Your model will then learn how to classify with a two dimensional time series.

You check this blog. In your case, only the output is different.

As for the two last points, aggregating the time series into one is risky in the sense that you might lose important information during the process. Finally the main disadvantage of your last point is that you won't be able to use a potential correlation between the two time series for the final classification.


What does the pair of vertical lines in empirical entropy formula mean? I am learning from this post.

$\alpha$ is the ratio of the first subset,

$$\alpha=\frac{\left|D_{1}\right|}{\left|D\right|}$$

according to the context and code of the post, $\left|D\right|$ means the number of samples?

What are the pair of vertical lines called?

Is it the L1 norm symbol? It does not seem to be the absolute value symbol.

$D$ is a set which is further divided into sets $D_{1}$ and $D_{2}$. In this context the vertical lines represent the cardinality of the set i.e. the number of elements in the set. Hence, $\alpha$ represents the ratio of number of samples of $D_{1}$ to $D$.


Avoiding overfitting in unsupervised ML I am using a unsupervised pattern matching approach to create a trade strategy. I use the output of the pattern matched results to decide whether to enter a trade or not. For deciding the best pattern parameters I run several combinations over the entire data set and choose the best parameters that yield the best profits. My question is whether this would be considered overfitting. If so, how may I avoid the same? I looked at several posts on StackOverflow but did not find anything that relates to my particular use case.

You could partition your dataset between train and test sets. Then arrive at the pattern parameters from the train set. After that apply your learnt model on the test set to evaluate its performance.
If you figure out that your model is performing poorly then you know you have overfit.
Also since this is a trading strategy and possibly involves timeseries, you want to ensure there is no data leakage and the test set is of a time frame later than that of the test set.


Decision tree vs. KNN In which cases is it better to use a Decision tree and other cases a KNN?

Why use one of them in certain cases? And the other in different cases? (By looking at its functionality, not at the algorithm)

Anyone have some explanations or references about this?

They serve different purposes.

KNN is unsupervised, Decision Tree (DT) supervised.
(KNN is supervised learning while K-means is unsupervised, I think this answer causes some confusion.)
KNN is used for clustering, DT for classification. (Both are used for classification.)

KNN determines neighborhoods, so there must be a distance metric. This implies that all the features must be numeric. Distance metrics may be affected by varying scales between attributes and also high-dimensional space.

DT, on the other hand, predicts a class for a given input vector. The attributes may be numeric or nominal.

So, if you want to find similar examples you could use KNN. If you want to classify examples you could use DT.


Are deep learning models way over the required capacity for their datasets' estimated entropies? this question might seem a bit odd. I was doing some self-studies into information theory and decided to do some more formal investigations into deep learning. Please bear with me as I try to explain.
I took a large "training" subset of MNIST as my guinea pig.

1) Converted every image in MNIST into "black-and-white" (pixels values only 0 or 1)

2) Summed over all data images to build a histogram over the pixels - I counted the number of times each pixel gets a 1 value in the dataset

3) Normalized histogram to get an estimate of the "true" probability distribution

4) From this, I got the following probability distribution (shown as a heatmap with matplotlib):

[

5) Now I calculated the entropy and got: $191$ bits

6) According to David MacKay in his Information Theory book, we could interpret a neural network as a noisy channel and consider each neuron as having a 2 bit capacity. Although he does state to use this idea with care. Chapter 40 of his book http://www.inference.org.uk/itila/book.html)

7) So, as a rough estimate (and with care) we could say we would need a neural network of 95 neurons in order to be able to encode the labeling of this MNIST training set (190/2)
8) Now we can get to my question:

Even if this is a very "back-of-the-envelope" calculation, shouldn't a neural network capable of learning the labelling be at least in the ballpark of 95 neurons?
Why do we need, for instance, a neural network with 21840 parameters to get 99% accuracy? (considering the one in PyTorch's example for MNIST: https://github.com/pytorch/examples/blob/master/mnist/main.py)

The current thinking is that it is easier to fit an overparameterized neural network, since the local extrema are different ways of expressing the same thing, whereas in a minimal neural network you have to worry about getting to the global extremum:

  The subtle reason behind this is that smaller networks are harder to
  train with local methods such as Gradient Descent: It’s clear that
  their loss functions have relatively few local minima, but it turns
  out that many of these minima are easier to converge to, and that they
  are bad (i.e. with high loss). Conversely, bigger neural networks
  contain significantly more local minima, but these minima turn out to
  be much better in terms of their actual loss. Since Neural Networks
  are non-convex, it is hard to study these properties mathematically,
  but some attempts to understand these objective functions have been
  made, e.g. in a recent paper The Loss Surfaces of Multilayer Networks.
  In practice, what you find is that if you train a small network the
  final loss can display a good amount of variance - in some cases you
  get lucky and converge to a good place but in some cases you get
  trapped in one of the bad minima. On the other hand, if you train a
  large network you’ll start to find many different solutions, but the
  variance in the final achieved loss will be much smaller. In other
  words, all solutions are about equally as good, and rely less on the
  luck of random initialization.

CS231n Convolutional Neural Networks for Visual Recognition


Machine Learning: Single input to variable number of outputs Is there a machine learning algorithm that maps a single input to an output list of variable length? If so, are there any implementations of the algorithm for public use? If not, what do you recommend as a workaround?

In my case, the input is a single scalar and the output is a list of scalars with variable length. For example, suppose I wanted to output a list of ones given the length of the list as input. Then <input, output> could be <1, [1]>, <2, [1, 1]>, etc. A small tweak would be providing the square root of the length in which case <2, [1, 1, 1, 1]> would be an answer. Note: the input need not be tied directly to the output.

For a more complex example, suppose I want to learn the look-and-say sequence. Valid <input, output> pairs would be: <1, [1]>, <2, [1, 1]>, <3, [2, 1]>, <4, [1, 2, 1, 1]>, <5, [1, 1, 1, 2, 2, 1]>, etc. My problem is also similar in that I can generate more examples; I am not restricted to a finite set of examples.

I would try to set a multilabel classification algorithm and make the output standard by adding zeros. So if your data is like this: <1, 1>, <2, [1, 1]>, <3, [2, 1]>, <4, [1, 2, 1, 1]>, <5, [1, 1, 1, 2, 2, 1]>. The maximum number of output is 6. So you could transform your data into something like:
<1, [1,0,0,0,0,0]>, <2, [1, 1,0,0,0,0]>, <3, [2, 1,0,0,0,0]>, <4, [1, 2, 1, 1,0,0]>, <5, [1, 1, 1, 2, 2, 1]>

Another option that occurs to me is to add the limit dynamically. Let say you have your training and test set. You can search for the biggest length and create an algorithm that adds the zeros to both datasets. Then let's say a new data you want to predict has a bigger length, then you'll need to recompute all training and test with for this new prediction. You can even check how extending the limit affects your model.


Should I update my regularisation L1 and L2 regularisation parameters in online setting? I have been working on online learning for a few weeks now, especially with Vowpal Wabbit and logistic regression. My understanding of the online learning algorithms and the problem is alright but I can't get my head straight about the regularisation issue.

In a standard machine learning problem, one uses a validation dataset in order to tune regularisations parameters for L1 and L2. But how do you choose those regularisation parameters in an online setting? Do you just fix them from start, or should I update them while training occurs?

As you say @Alexis, cross-validation is the standard way of selecting a model (or its parameters) in offline learning (especially when previous knowledge in not available). For online training, one usually incorporates some sort of (automatic) compensation for the model parameters that adjust them at the same time that training takes place (the alternative being to cross-validate on some gather data or equivalently to pre-optimize your parameters). Check also this answer as well as, for example, this paper (it shows that this issue is a very active research area)!


Ad click prediction: what are the negative examples? I am analysing the log of a website and I would like to build a classifier to predict the users that are likely to click on an Ad.

The Ad can be displayed to the visitor several times.

To build any classifier I need positive and negative examples:

The positives are the visitors who clicked on the Ad (easy).
The negatives are the visitors who saw the Ad but didn't click (not very obvious).

Question

Is there a convention about how/when to consider a user as a negative example?

I presume that I should define a threshold of impressions (views) per user, if the user reaches this threshold without clicking on the Ad, I consider him/her as as negative example?

Any reference or guidance is highly appreciated.

You're overthinking it.  You might not need a threshold.  Start with the simplest approach you possibly can: If you showed the ad to a visitor, then that's a negative example.  Each time you show an ad to a visitor, you end up with an instance, whether positive or negative.  If you've showed the ad three times to the same visitor, you end up with three examples.  (Maybe all negative; or maybe one is positive and two are negative; but that's fine.)

Bonus tip: Do research methods for handling class imbalance.


Predict probability when model was trained in balanced dataset I have a dataset of about 1M observation and I had to predict a response that occurs only about 10.000 times (1%).

I decided to train a random forest, but this takes a lot of time to train because the data is too large for my hardware. So I decided to take a sample, but an aleatory sample would be already too large to have a minimum quantity of response. (If  I take 10% aleatory, i would have only 1000 response)

Then I took a stratified sample. All responses and 10.000 aleatory non-responses, and trained my model in this dataset.

But now I need to rescale the probability so I have the real probability of the observation to be response.

I tried to simulate this problem with this code in R. Training a model in a balanced dataset and another one in the unbalanced data. But those models are not very correlated and I didn't find a good way to tranform the probabilities to the original unbalanced scale.

Found this is a good reference for future readers

I found that for logistic regression I can do this by just changing the intercept this way:

$$ \hat{\beta_0}  =  \hat{\beta_0^*} - log(\frac{\gamma_1}{\gamma_2})$$

Where $ \gamma_1 = Pr(Z=1|Y=1)$ and $ \gamma_2 = Pr(Z=1|Y=0)$. $Z$ is the an aleatory variable indicating if the observation is in the reduced dataset.

This was found in this book (in portuguese) page 216

simulate_data <- function(n){
  X <- data.frame(matrix(runif(n*20), ncol = 20))
  list(
    X = X,
    Y = rbinom(n, size = 1, prob = apply(X, 1, sum) %>% pnorm(mean = 13)) %>% as.factor()
  )
}

balance <- function(X, Y){
  X <- rbind(
    X[Y == "1",],
    X[Y == "0",] %>% sample_n(length(Y[Y == "1"]))
    )
  return(list(
    X = X,
    Y = as.factor(c(rep(c(1,0), each = length(Y[Y == "1"]))))
  ))
}

train <- simulate_data(100000)

library(randomForest)
m_desb <- mean(train$Y == "1")
modelo_desb <- randomForest(train$X,train$Y, ntree = 200, cutoff = c(1-m_desb, m_desb), nodesize = 30, mtry = 8)
bal <- balance(train$X, train$Y)
m_bal <- mean(bal$Y == "1")
modelo_bal <- randomForest(bal$X,bal$Y, ntree = 100, cutoff = c(1-m_bal, m_bal), nodesize = 50)

Code for plot

library(ggplot2)
data.frame(
  unbalanced = predict(modelo_desb, newdata = test$X, type = "prob")[,2],
  balanced = predict(modelo_bal, newdata = test$X, type = "prob")[,2]
) %>%
  ggplot(aes(x = balanced, y = unbalanced)) +
  geom_point(size = 0.3) +
  xlim(0,1) +
  geom_smooth() +
  geom_hline(yintercept = m_desb, linetype = "dashed") +
  geom_vline(xintercept = m_bal, linetype = "dashed")

Daniel,
What you did goes under the name of "oversampling." There is a sample of some "real" population, and you replace it with a sample from a "manufactured" population. The problem that makes sense in application is the estimation of

$$P_r(Y=1|X) = \text{probability of response=1 in the $\mathbf{real}$ population given the predictor $X$} $$

but by using an oversample you are estimating
$$P_m(Y=1|X) = \text{probability of response=1 in the $\mathbf{manufactured}$ population given the predictor $X$}$$

The two probabilities are related. I'll worked the details. I'll pretend the predicor $X$ is discrete. If $X$ takes numerical values one has to replace some probabilities by probability densities.
$$\dots\dots\dots$$

To simplify the notation, let $\pi_1 = P_r(Y=1)$ and $\mu_1 = P_m(Y=1)$ be the probabilities of response in the real and manufactured populations, let
$$
L_r =
\frac{P_r(X=x|Y=1)}{P_r(X=x|Y=0)} = \frac{\frac{P_r(Y=1|X=x)}{P_r(Y=0|X=x)}}{\frac{\pi_1}{1-\pi_1} }$$
be the odds ratio of $Y=1$, i.e.: the ratio of the odds among cases with $X=x$ and the odds in the general $\mathbf{real}$ population. Finally, let $L_m$ be the corresponding ratio in the $\mathbf{manufactured}$ population.

By Bayes' Theorem:
$$ P_r(Y=1|X=x) = \frac{P_r(Y=1,X=x)}{P_r(X=x)} =   \\
=\frac{P_r(X=x|Y=1)\space \pi_1}{P_r(X=x|Y=1)\space\pi_1 + P_r(X=x|Y=0)\space (1 - \pi_1)} = \\
=\frac{L_r\space \pi_1}{L_r\space\pi_1 + \space (1 - \pi_1)} \tag{1}
$$
In a similar way, we get an analogous result for the manufactured population:
$$
P_m(Y=1|X=x) = \frac{L_m\space \mu_1}{L_m\space\mu_1 + \space (1 - \mu_1)} \tag{2}
$$
Since the manufactured sample is a random sample, stratified by $Y$, the conditional distribution of X within responders is the same as in the real population. Same as for non responders, i.e.:
$$
 P_r(X=x|Y=j) = P_r(X=x|Y=j) $$
for $j=0,1.$ If the sample stratified by values of Y was anything other than random sample these would not be true.
It follows that $\boxed{ L_r = L_m }$. Next we solve for $L_m$ in terms of $P_m(Y=1|X)$ from (2) and replace in (1).

$$\dots\dots\dots$$
$\mathbf{Digression}$:
Here is my easy way to carry the steps, without mess. Two non-zero vectors $\mathbf{v_1}$, $\mathbf{v_2}$ are parallel iff there is $\lambda \ne 0$ such that $\mathbf{v_1}=\lambda \mathbf{v_2}.$ Below I will use this idea, and I will not care about the exact value of $\lambda$, so I will be using "$\lambda$" as a short-hand for "$\mathbf{\text{some non-zero mumber}}$."
$\mathbf{\text{End Digression}}$:
$$\dots\dots\dots$$

The easy way to solve is to observed that for non-zero messy values of $\lambda$ (not the same in each occurrence!) one has:

$$
\begin{bmatrix}
 P_r(Y=1|X) \\
 1 \\
\end{bmatrix} = \lambda
\begin{bmatrix}
       \pi_1 &0\\
      \pi_1      &1-\pi_1
\end{bmatrix}
\begin{bmatrix}
L_r\\
1
\end{bmatrix} ,
$$
and
$$
\begin{bmatrix}
 P_m(Y=1|X) \\
 1 \\
\end{bmatrix} = \lambda
\begin{bmatrix}
      \mu_1 &0 \\
      \mu_1      &1-\mu_1
\end{bmatrix}
\begin{bmatrix}
L_m\\
1
\end{bmatrix}
. $$

Therefore,

$$
\begin{bmatrix}
L_m\\
1
\end{bmatrix}
 = \lambda
\begin{bmatrix}
      \mu_1  &0 \\
      \mu_1      &1-\mu_1
\end{bmatrix}^{-1}
\begin{bmatrix}
 P_m(Y=1|X) \\
 1 \\
\end{bmatrix} ,
$$

and so (remember that here $\lambda$ stands for "some non-zero number")

$ \space
\begin{bmatrix}
  P_r \\
   1
\end{bmatrix} = \lambda
\begin{bmatrix}
       \pi_1 &0 \\
      \pi_1      &1-\pi_1
\end{bmatrix}
\begin{bmatrix}
L_r \\
1
\end{bmatrix} = \\
\text{              }=\lambda
\begin{bmatrix}
      \pi_1      &0    \\
      \pi_1      &1-\pi_1
\end{bmatrix}
\begin{bmatrix}
       \mu_1     &0     \\
      \mu_1      &1-\mu_1
\end{bmatrix}^{-1}
\begin{bmatrix}
 P_m \\
 1 \\
\end{bmatrix} = \\
\text{              }=\lambda
\begin{bmatrix}
      \pi_1 (1- \mu_1)        &0 \\
      \pi_1  - \mu_1     & \mu_1 (1- \pi_1)
\end{bmatrix}
\begin{bmatrix}
 P_m \\
 1 \\
\end{bmatrix} =
\lambda
\begin{bmatrix}
      \pi_1  (1-\mu_1)  P_m \\
      (\pi_1  - \mu_1) \; P_m   +  \mu_1 (1- \pi_1)
\end{bmatrix}.
$
Thus,
$$ P_r = \frac{\pi_1 (1- \mu_1) P_m}{(\pi_1 - \mu_1) \; P_m   +  \mu_1 (1- \pi_1) } $$

$$\dots\dots\dots$$
Example: Let's work the details of a Binomial model,
$$P_m(Y=1|X) = \frac{e^{\beta_0 + \beta X}}{1+e^{\beta_0 + \beta X}} $$
or in the "where $\lambda$ is some non-zero scalar" notation (I would not had digressed before if I did not had ulterior motive.. :) ):

$$
 \begin{bmatrix}
 P_m \\
 1 \\
\end{bmatrix} =
\lambda
\begin{bmatrix}
e^{\beta_0 + \beta X} \\
1 + e^{\beta_0 + \beta X}
\end{bmatrix}
$$

What is the implied model in the real population?

$ \space
\begin{bmatrix}
     P_r \\
      1
\end{bmatrix} =
\lambda
\begin{bmatrix}
      \pi_1 (1- \mu_1)   &0 \\
      \pi_1  - \mu_1     &\mu_1 (1- \pi_1)
\end{bmatrix}
\begin{bmatrix}
  P_m \\
   1
\end{bmatrix} =
$

$\;$
$
= \lambda
\begin{bmatrix}
      \pi_1 (1- \mu_1)   &0 \\
      \pi_1  - \mu_1     &\mu_1 (1- \pi_1)
\end{bmatrix}
\begin{bmatrix}
e^{\beta_0 + \beta X} \\
1 + e^{\beta_0 + \beta X}
\end{bmatrix}=
$

$\;$
$
= \lambda
\begin{bmatrix}
      \pi_1 (1- \mu_1)   e^{\beta_0 + \beta X} \\
      \pi_1 (1- \mu_1)   e^{\beta_0 + \beta X} + \mu_1 (1- \pi_1)
\end{bmatrix} =
\lambda
\begin{bmatrix}
\frac{\pi_1 (1- \mu_1)}{\mu_1 (1- \pi_1)} e^{\beta_0 + \beta X} \\
1 + \frac{\pi_1 (1- \mu_1)}{\mu_1 (1- \pi_1)} e^{\beta_0 + \beta X}
\end{bmatrix}
.$

If we let $\tau = \ln(\frac{\pi_1 (1- \mu_1)}{\mu_1 (1- \pi_1)})$, we can absorve this constant in the exponents to get:
$$
\begin{bmatrix}
     P_r \\
      1
\end{bmatrix} =
\lambda
\begin{bmatrix}
e^{\tau + \beta_0 + \beta X} \\
1 + e^{\tau + \beta_0 + \beta X}
\end{bmatrix}
.$$
Taking the ratio and simplifying the non-zero constant in numerator and denominator we get that fitting a logistic model to the manufactured population results in an implied logistic model for the real population, $\mathbf{\text{with the same coefficients for X}}$ and with a difference in the constant (in the logistic model) given by:
$$ \beta_{real} = \tau + \beta_0 $$

$$\dots$$
Note that, according to your reference, the ratio of $\gamma_1 = Pr(Z=1|Y=1)$ and $\gamma_0 = Pr(Z=1|Y=0)$ should come up. Indeed:
$$ \gamma_1 = Pr(Z=1|Y=1) = \frac{P(Z=1,Y=1)}{P(Y=1)} = \frac{P_r(Y=1|Z=1)P_r(Z=1)}{P_r(Y=1)} = \frac{P_m(Y=1)}{P_r(Y=1)} P_r(Z=1)=
\frac{\mu_1}{\pi_1}P_r(Z=1) $$
likewise (i.e. change Y to 1-Y),
$$ \gamma_0 = \frac{1-\mu_1}{1-\pi_1}P_r(Z=1) $$
so
$$ \ln(\frac{\gamma_1}{\gamma_0}) = - \ln(\frac{\pi_1 (1-\mu_1)}{\mu_1 (1-\pi_1}) = \tau $$

$$\dots\dots\dots$$
Notes for full disclosure: I worked with the probability model. When one works with finite samples the example above suggests two ways of estimating the coefficients:
* estimate coefficients using the sample from the real population
* estimate coefficients using the manufactored populations

It terns out that this two estimators are not the same (it is obvious if one consideres one estimator is based on more cases than the other). Both estimators are asymtopically consistent, but it can be shown the one based on the manufactored population is more biased (forgot the reference :( ).

In the data science space we are more concern with the quality of the predictions than the parameters of the parameters used to make those predictions, so as long as you check results properly (e.g.: using a testing set to build models and another to validate them), the bias in the parameters should not deter us from using oversampling.

$$\dots\dots\dots$$


where does the downloaded file gets stored in the R-Studio Server on Ubuntu EC2 Instance when using download.file() Studio Server on an Ubuntu EC2 instance for the first time and successfully started r-studio server in my browser. I also have putty ssh client.For the Purpose of this Analysis i first want to download a file from the Website called UCI Machine Learning Repository and then access the file for downstream Analysis, so i used the Below shown R Script in the r-studio server in my browser

download.file("https://archive.ics.uci.edu/ml/machine-learning-databases/haberman/haberman.data", destfile = "Haberman.txt")

This works exactly fine but since i am new to work in the Cloud Platform i couldn't understand where RStudio stores the Haberman.txt file, because when i work on my PC i used to specify the Path to store the Haberman.txt file in the destfile argument as shown below

 download.file("https://archive.ics.uci.edu/ml/machine-learning-databases/haberman/haberman.data", destfile = "E:/Couresera_Datasets/Haberman.txt")

Now when i Navigate to the specified Path in my PC i will be able to find the Haberman.txt file but where can i find the Haberman.txt file in the Ubuntu EC2 instance when i use the download.file()?

Any Help Will be Appreciated.

Thank you in Advance

It'll put it exactly where you asked for it:

 destfile = "Haberman.txt"

that is, a file called Haberman.txt in the current working directory at the time you ran download.file. You can get the current working directory with getwd() in R.

So for example if its something you can read with read.csv, you can do:

hdata = read.csv("Haberman.txt")

as long as you haven't changed the current working directory (like with setwd).


Job title similarity I'm trying to define a metric between job titles in IT field. For this I need some metric between words of job titles that are not appearing together in the same job title, e.g. metric between the words

  senior, primary, lead, head, vp, director, stuff, principal, chief,

or the words

  analyst, expert, modeler, researcher, scientist, developer, engineer, architect.

How can I get all such possible words with their distance ?

That's an interesting problem, thanks for bring out here on stack.

I think this problem is similar to when we apply LSA(Latent Semantic Analysis) in sentiment analysis to find list of positive and negative words with polarity with respect to some predefined positive and negative words.

Good reads:

Learning Word Vectors for Sentiment Analysis
Unsupervised Learning of Semantic Orientation from a Hundred-Billion-Word Corpus

So, according to me LSA is your best approach to begin with in this situation as it learns the underlying relation between the words from the corpus and probably that's what you are looking for.


Two parallel models for semantic segmentation in Keras I want to build two parallel models for image semantic segmentation in Keras.

    input1 = Input(shape=(480,480,3))
    input2 = Input(shape=(480,480,1))

    c1_1 = Conv2D(filters=64, kernel_size=(3,3), activation='relu',  padding='same')(input1)
    c1_1 = MaxPool2D(strides=(2,2))(c1_1)

    c2_1 = Conv2D(filters=16, kernel_size=(3,3), activation='relu',  padding='same')(input2)
    c2_1 = MaxPool2D(strides=(2,2))(c2_1)

   (...)

    # Merge to models:
    c = concatenate([c1_n, c2_n], axis=3)
    c = UpSampling2D(size=(2,2))(c)
    c = Conv2D(filters=512, kernel_size=(3,3), activation='relu', padding='same')(c)

   (...)

   output_layer = Conv2D(6, kernel_size=(1,1), activation='softmax')(c)

   model = Model([input1, input2], output_layer)

   model.compile(optimizer=Adam(2e-4), loss='categorical_crossentropy', metrics=['categorical_accuracy'])

   def my_generator(x_train, y_train, batch_size):
     data_generator = ImageDataGenerator(
        (...).flow(x_train, x_train, batch_size, seed=42)
     mask_generator = ImageDataGenerator(
        (...).flow(y_train, y_train, batch_size, seed=42)

     while True:
        x_batch, _ = data_generator.next()
        y_batch, _ = mask_generator.next()
        yield [x_batch[:,:,:,:3], x_batch[:,:,:,3]], y_batch

     #X = [72, 480, 480, 4]
     #Y = [72, 480, 480, 5]
     model.fit_generator(my_generator(X, Y, 7),
                       steps_per_epoch = 60,
                       validation_data = (X_test, Y_test),
                       epochs=150, verbose=2)

But i have an error:

ValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 2 array(s), but instead got the following list of 1 arrays: [array([[[[ 0.33891213,  0.37238494,  0.33054393,  0.        ],
     [ 0.34728033,  0.38493724,  0.33472803,  0.        ],
     [ 0.35146444,  0.39330544,  0.35983264,  0.        ],
     ....

Can you help me with my error or tell me whether this assumption is correct?

This is probably laye but incase someone else is stuck, check out this link.

Basically, your model expects two inputs as defined here:

model = Model([input1, input2], output_layer)

Therefore you will need to pass a list of two inputs with the same shape as you defined here:

input1 = Input(shape=(480,480,3))
input2 = Input(shape=(480,480,1))


Improving misclassification for one class in a multi-class classification task Here I am trying to use 3 convolution layer neural network to classify a set of images (train data: (3249) , validation data: (487), test data: (326))
I have one class which is misclassified and I cannot understand what to do next. I have tried to reduce the value of dropout layer, but results got worst.
I know that the next solutions could be useful if I had bad classification for all classes :

Get more data

Try New model architecture, try something better.

Decrease number of features (you may need to do this manually)

Introduce regularization such as the L2 regularization

Make your network shallower (less layers)

Use less number of hidden units

What do you thing could be a good choice if I have only misclassifcation of one class?
Number of  total samples per class :
Black rot: 1180
Esca: 1383
healthy: 423
leaf blight: 1076

I had split the two datasets as follow:
x_train, _x, y_train, _y = train_test_split(x,y,test_size=0.2, stratify = y, random_state = 1)
x_valid,x_test, y_valid, y_test = train_test_split(_x,_y,test_size=0.4, stratify = _y, random_state = 1)

The main problem is that there are too many Escas in your dataset. If you look at the confusion matrix, the Esca column gets predicted (wrongly and correctly) much more that the others. This is clearly a symptom of a skewed data set.
Try augmenting your images to generate a larger 'super-dataset'. Then sample a 'sub-dataset' from that 'super-dataset', such that it has an equal distribution across all classes. Train on the 'sub-dataset'.
If you can't augment your data to have a better distribution across the 4 classes, here are some ideas:

Modify the loss function to more aggressively penalize Black Rots classified as Escas.
Split into two networks; the first differentiates between
Leaf Blight-Healthy-Black Rot/Esca, and the second differentiates between
Black Rot-Esca.


Export weights (formula) from Random Forest Regressor in Scikit-Learn I trained a prediction model with Scikit Learn in Python (Random Forest Regressor) and I want to extract somehow the weights of each feature to create an excel tool for manual prediction.

The only thing that I found is the model.feature_importances_ but it doesn't help.

Is there any way to achieve it?

def performRandomForest(X_train, y_train, X_test, y_test):

    '''Perform Random Forest Regression'''

    from sklearn.ensemble  import  RandomForestRegressor

    model  =  RandomForestRegressor()
    model.fit( X_train , y_train )

    #make predictions
    expected  = y_test
    predicted  = model.predict( X_test )

    #summarize the fit of the model
    mse  = np.mean(( predicted - expected )** 2)
    accuracy = ( model.score ( X_train , y_train ))

    return model, mse, accuracy

At the moment, I use the model.predict([features]) to do it, but I need it in an excel file.

The SKompiler library might help:

from skompiler import skompile
skompile(rf.predict_proba).to('excel')

Check out this video.


Creating new columns by iterating over rows in pandas dataframe I have a pandas data frame (X11) like this: In actual I have 99 columns up to dx99

    dx1      dx2    dx3    dx4
0   25041   40391   5856    0
1   25041   40391   25081   5856
2   25041   40391   42822   0
3   25061   40391   0       0
4   25041   40391   0       5856
5   40391   25002   5856    3569

I want to create additional column(s) for cell values like 25041,40391,5856 etc. So there will be a column 25041 with value as 1 or 0 if 25041 occurs in that particular row in any dxs columns. I am using this code and it works when number of rows are less.

mat = X11.as_matrix(columns=None)
values, counts = np.unique(mat.astype(str), return_counts=True)

for x in values:
    X11[x] = X11.isin([x]).any(1).astype(int)

I am getting result like this:

dx1     dx2     dx3    dx4  0   25002   25041   25061   25081   3569    40391   42822   5856
25041   40391   5856    0   0   0       1       0       0       0          1        0       1
25041   40391   25081  5856 0   0       1       0       1       0            1      0       1
25041   40391   42822   0   0   0       1       0       0       0           1       1       0
25061   40391   0       0   0   0       0       1       0       0          1        0       0
25041   40391   0    5856   0   0       1       0       0       0          1        0       1
40391   25002 5856   3569   0   1       0       0       0       1          1        0       1

When number of rows are many thousands or in millions, it hangs and takes forever and I am not getting any result. Please see that cell values are not unique to column, instead repeating in multi columns. For ex, 40391 is occurring in dx1 as well as in dx2 and so on for 0 and 5856 etc. Any idea how to improve the logic mentioned above?

There's a much more pythonic solution in pandas...

This takes less than a second on 10 Million rows on my laptop:

for x in X11.E.unique():
    X11[x]=(X11.E==x).astype(int)
X11

Here are the details laid out:

Simple small dataframe -

import numpy as np
import pandas as pd

X11 = pd.DataFrame(np.random.randn(6,4), columns=list('ABCD'))
X11['E'] = [25223, 112233,25223,14333,14333,112233]
X11

Binarization method -

for x in X11.E.unique():
    X11[x]=(X11.E==x).astype(int)
X11

Dataframe with 10 Million rows -

pd.set_option("display.max_rows",20)
X12 = pd.DataFrame(np.random.randn(10000000,4), columns=list('ABCD'))
foo = [25223, 112233,25223,14333,14333,112233]
bar=[]
import random
for x in range(10000000):
    bar.append(random.choice(foo))
X12['E'] = bar
X12

Timed binarization (aka one-hot encoding) on 10 million row dataframe -

import time
start = time.clock()

for x in X12.E.unique():
    X12[x]=(X12.E==x).astype(int)
elapsed = (time.clock() - start)

print "This is the time that this took in seconds: ",elapsed

X12

Hope this helps!


Model selection in active learning I am dabbling in active learning and was wondering how to combine this in seeking out the best architecture for the network.
In my understanding, active learning uses a heuristic for selecting the best instances to label in order to learn as quickly as possible. However, the way these instances are chosen are dependent on the model itself.
Is there a way to handle this model dependency?
It seems to me that models architecture is dependent on the train size, for example. If that's the case, wouldn't it be beneficial to allow the model to change its architecture during active learning?
Is there a way to do this or do we have to be very careful when selecting a model before performing active learning.
Another possibility I see is to perform a grid search on the networks architecture on all the labeled data after every so-often queries. But then again, the model that comes out on top is still dependent on the initially chosen model...

There are nested loops in training machine learning models. The outer loop is selecting the hyperparameters for the model, including the architecture. The inner loop trains the model's parameters while holding the hyperparameters constant.
It is possible to try many possible options on outer loop and make those options dependent on the results of the inner loop.