How to decide which method to use TFIDF, or BOW In a huge dataset for NLP it is taking very long time to classify my dataset
therefore, trying each feature extraction method separetly is time consuming and not effecient.
Is there a way that can tell me which method (TFIDF or Bag Of Words) is more likely to give the highest F1 score.
I tried test them on smaller subset (1000 records) it was fast but best method in smaller subset does not mean it is the best in complete dataset.
any other way to decide which method to use?

There is no specific way to deal with these kinds of experimentation. Below are some important points to remember before doing experimentation

If you are using NN to do the work, dense vectors like word2vec or fasttext may give better results than BoW/TfIdf

If you have more OOV words then fasttext may give better output than basic Word2Vec

If you are using linear algorithms like Logistic Regression/Linear SVM, BoW/TfIdf may have some advantage over averaging all the word vectors in the sentence. But it's not always true.

For the tree-based algorithms, training time may increase if we use BoW/TfIdf features because of the huge feature-length.


Why are policy gradients on-policy? I'm not entirely sure why policy gradients have to be on-policy and have to update using trajectories sampled from the current behaviour. In REINFORCE, the loss function is determined by the log probability of an action times the reward (or discounted reward).

For a state $s$, if I take action $a$ and arrive in state $s'$ I will always see reward $r$. So if I have keep all these values, run an $s$ from the past through my current actor then why can't I update my actor with the new log probability of $a$ and the known reward $r$. I don't need to actually have played and seen a new result.

Can someone correct my understanding?

Thanks

The Policy Gradient theorem states that the gradient of the expected reward is equal to the expectation of the log probability of the current policy multiplied by the reward. Notice that to compute the integral of the expectation we can use Monte Carlo method. For this you would need sampled trajectories. Have you looked at a derivation? The whole theorem comes from mathematical derivations. Check in that page the Assumptions and the Likelihood Ratios (REINFORCE) sections. Hope it helps!


How pre-trained BERT model generates word embeddings for out of vocabulary words? Currently, I am reading BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I want to understand how pre-trained BERT generates word embeddings for out of vocabulary words? Models like ELMo process inputs at character-level and can generate word embeddings for out of vocabulary words. Can BERT do something similar?

BERT does not provide word-level representations, but subword representations. You may want to combine the vectors of all subwords of the same word (e.g. by averaging them), but that is up to you, BERT only gives you the subword vectors.
Subwords are used for representing both the input text and the output tokens. When an unseen word is presented to BERT, it will be sliced into multiple subwords, even reaching character subwords if needed. That is how it deals with unseen words.
ELMo is very different: it ingests characters and generate word-level representations. The fact that it ingests the characters of each word instead of a single token for representing the whole word is what grants ELMo the ability to handle unseen words.


How to choose the best parameter values for TfidfVectorizer in sklearn library? Recently, I used  TfidfVectorizer in scikit-learn library to calculate a matrix of TF-IDF features. However, I do not know how to set some parameters such as max_features, min_df, max_df, etc.

I think these parameters are mostly used when you combine the vectorizer and a machine learning model in a pipeline. Therefore, you should tune these parameters based on the outcome of your model training.

For example, if your task is to classify input texts, you may want to tune the max_features parameter such that the number of features is not too large, but your model can still perform reasonably well. In general, these parameters are used to control the size of the vector to be fed into your machine learning model, so as to avoid overfitting and improve the efficiency of model training.


What is the difference between different batch_sizes in Keras Sequential models? I am interested to know, what happens when I choose batch_size=1 or batch_size=1000 or any other numbers in Keras.lstm.mode.fit() function for example when I am configuring batch_input_shape? Does this make effect on my final result and changing them?

I need for future prediction using batch_size=1 but affraid of getting bad/wrong result!

May someone explains the important effects of choosing batch_size from 1 to N numbers?

I need for future prediction using batch_size=1 but affraid of getting bad/wrong result!

Batch size is not connected to the future prediction - it's just the number of samples which will be used in one forward pass to calculate the loss and then in a backward pass to calculate the gradients and parameter updates.

In order to regulate the future prediction (as far as I understand from your question), you just need to define the sample size (or width, or time steps, whatever name you use ...).

Regarding the "important effects of choosing batch_size from 1 to N numbers?" - with a larger batch size your training process might finish faster, but with smaller you might avoid local minimum (but that - local minimum, generally, shouldn't be a problem with deep neural networks).


Effective Methods for Background Removal on Images I'm interested in learning about how background removal works on images taken of clothing items. Do we need a specific color difference between the background and the clothing item in order to be able to determine object edges? How are edges determined?

Can somebody link me to algorithms/models/methods used effectively?

This problem used to be solved by analytic methods as you can find here, but these are extremely outdated and boring. However, for simpler cases they work very well and are much faster to implement.

Using deep learning is much more powerful for this particular case and you dont actually need that much data, although you will need to take some time to prepare your data.

The concept

We will train the data on a set of smaller image patches extracted from the original images where the target labels will be foreground and background.

Preparing the data

First, you will need to take your images and draw a mask over them in order to identify your labels. You can do this using paint I suppose. You will take your original images and color the foreground in white and the background in black. This will be labels 1 and 0 respectively.

In Python you will load the original images and their respective label images. You will then split the image and labels into patches of size $k \times k$. You can pick whatever patch size you think is best suited for your kind of data. This is a hyper-parameter you will need to tune using cross-validation. Each patch will have its associated label which is the label of the center of the patch.

Build the model

Then you will build a standard convolutional neural network model where the inputs are going to be the images patches and the output will be the label.

Segmenting new images

To segment new images, split that image into patches and predict the label. All the patches which result in $label=1$ is the foreground.

Alternative method

Alternatively, you can predict the values for the entire patch at once, that means the outputs of your model network will be the same size as your input. You will thus have a label for each pixel in the patch.


Exception: Data must be 1-dimensional appears when trying to make a submission I am doing image classification, and until now I have built my network and evaluated it. The only thing that remains to do is to do the submission, so I have a blind test set which contains images with no labels, and using the model I have created, I have to make predictions on this blind test set.

I have 4 classes: HAZE, RAINY, SUNNY, SNOWY. My code is the following:

blind_testSet = '/content/drive/My Drive/Colab Notebooks/submission'

test_datagen_blind = ImageDataGenerator(
    featurewise_center=True,
    featurewise_std_normalization=True,
    rescale = 1. / 255
   )

test_generator_blind = test_datagen.flow_from_directory(
    directory=blind_testSet,
    target_size=(256, 256),
    color_mode="rgb",
    batch_size=batch_size,
    class_mode="categorical",
    shuffle=False
)

preds =transfer_model.predict_generator(test_generator_blind,verbose=1,steps=val_steps)

import pandas as pd
import numpy as np

predicted_class_indices=np.argmax(preds,axis=1)
labels = (train_generator.class_indices)
labels = dict((v,k) for k,v in labels.items())
print(labels)
predictions = [labels[k] for k in predicted_class_indices]
filenames=test_generator_blind.filenames
results=pd.DataFrame({"image ID":filenames,
                  "Predictions":preds})
results.to_csv("submission.csv",index=False)

the problem is that I get the error:

    Exception                                 Traceback (most recent call last)
<ipython-input-22-37b4cd72c225> in <module>()
      9 filenames=test_generator_blind.filenames
     10 results=pd.DataFrame({"image ID":filenames,
---> 11                       "Predictions":preds})
     12 results.to_csv("submission.csv",index=False)

4 frames
/usr/local/lib/python3.6/dist-packages/pandas/core/internals/construction.py in
sanitize_array(data, index, dtype, copy, raise_cast_failure)
    727     elif subarr.ndim > 1:
    728         if isinstance(data, np.ndarray):
--> 729             raise Exception("Data must be 1-dimensional")
    730         else:
    731             subarr = com.asarray_tuplesafe(data, dtype=dtype)

Exception: Data must be 1-dimensional

Can someone help me? Thanks.

[EDIT]I have also tried to to :

    preds =transfer_model.predict_generator(test_generator_blind,verbose=1,steps=val_steps).reshape(-1,1)

but it does not work and I get the same error.

Python tells you that the data you give for the column "predictions" is not 1-dimensional (i.e. it's not a flat list). And indeed preds is not a 1-dimensional array, what you want to give is the corresponding labels that you collected in predictions.

So Instead of this:

results=pd.DataFrame({"image ID":filenames,
                  "Predictions":preds})

you probably want:

results=pd.DataFrame({"image ID":filenames,
                  "Predictions":predictions})


How do subsequent convolution layers work? This question boils down to "how do convolution layers exactly work.

Suppose I have an $n \times m$ greyscale image. So the image has one channel.
In the first layer, I apply a $3\times 3$ convolution with $k_1$ filters and padding. Then I have another convolution layer with $5 \times 5$ convolutions and $k_2$ filters. How many feature maps do I have?

Type 1 convolution

The first layer gets executed. After that, I have $k_1$ feature maps (one for each filter). Each of those has the size $n \times m$. Every single pixel was created by taking $3 \cdot 3 = 9$ pixels from the padded input image.

Then the second layer gets applied. Every single filter gets applied separately to each of the feature maps. This results in $k_2$ feature maps for every of the $k_1$ feature maps. So there are $k_1 \times k_2$ feature maps after the second layer. Every single pixel of each of the new feature maps got created by taking $5 \cdot 5 = 25$ "pixels" of the padded feature map from before.

The system has to learn $k_1 \cdot 3 \cdot 3 + k_2 \cdot 5 \cdot 5$ parameters.

Type 2.1 convolution

Like before: The first layer gets executed. After that, I have $k_1$ feature maps (one for each filter). Each of those has the size $n \times m$. Every single pixel was created by taking $3 \cdot 3 = 9$ pixels from the padded input image.

Unlike before: Then the second layer gets applied. Every single filter gets applied to the same region, but all feature maps from before. This results in $k_2$ feature maps in total after the second layer got executed. Every single pixel of each of the new feature maps got created by taking $k_2 \cdot 5 \cdot 5 = 25 \cdot k_2$ "pixels" of the padded feature maps from before.

The system has to learn $k_1 \cdot 3 \cdot 3 + k_2 \cdot 5 \cdot 5$ parameters.

Type 2.2 convolution

Like above, but instead of having $5 \cdot 5 = 25$ parameters per filter which have to be learned and get simply copied for the other input feature maps, you have $k_1 \cdot 3 \cdot 3 + k_2 \cdot k_1 \cdot 5 \cdot 5$ paramters which have to be learned.

Question

Is type 1 or type 2 typically used?
Which type is used in Alexnet?
Which type is used in GoogLeNet?

If you say type 1: Why do $1 \times 1$ convolutions make any sense? Don't they only multiply the data with a constant?
If you say type 2: Please explain the quadratic cost ("For example, in a deep vision network, if two convolutional layers are chained,
any uniform increase in the number of their filters results in a quadratic increase of computation")

For all answers, please give some evidence (papers, textbooks, documentation of frameworks) that your answer is correct.

Bonus question 1

Is the pooling applied always only per feature map or is it also done over multiple feature maps?

Bonus question 2

I'm relatively sure that type 1 is correct and I got something wrong with the GoogLe paper. But there a 3D convolutions, too. Lets say you have 1337 feature maps of size $42 \times 314$ and you apply a $3 \times 4 \times 5$ filter. How do you slide the filter over the feature maps? (Left to right, top to bottom, first feature map to last feature map?) Does it matter as long as you do it consistantly?

My research

I've read the two papers from above, but I'm still not sure what is used.
I've read the lasagne documentation
I've read the theano documentation
I've read the answers on Understanding convolutional neural networks (without following all links)
I've read Convolutional Neural Networks (LeNet). Especially figure 1 makes me relatively sure that Type 2.1 is the right one. This would also fit to the "quadratic cost" comment in GoogLe Net and to some practical experience I had with Caffee.

I am not sure about the alternatives described above, but the commonly used methodology is:

Before the application of the non-linearity, each filter output depends linearly on all of the feature maps before within the patch, so you end up with $k_2$ filters after the second layers. The overall number of parameters is $3 \dot{} 3\dot{}k_1 + k_1\dot{} 5 \dot{} 5 \dot{} k_2$.

Bonus 1: Pooling is done per feature map, separately.

Bonus 2: The order of "sliding" does not matter. In fact, each output is computed based on the previous layer, so the output filter responses do not depend on each other. They can be computed in parallel.


Sound Classification for Multiple Classes for English Letters I have recorded audio files for the English letters, each file includes 26 letters. I have split each letter into a separate audio file. Now I want to put similar audio letters into one folder. I can do it manually but it will take time. Is there a classifier method to this?

If in each file the letters are spoken separately, with silence in between, and always in the same order (A,B...,Z) then one can try to automate finding each section and its label.
Use a Voice Activity Detection (VAD) module to detect each spoken character. Then assign A to the first voiced area, B to the next etc. If you have very clean and uniform audio, then it may work to use volume instead of voice activity detector.
Note that this process is a bit vulnerable to missed sections, because then all the subsequent labels will be shifted. It is also vulnerable to speakers saying characters out of order, or repeated.
An alternative would be to use a Speech Recognition system. That should be able to handle arbitrary order of utterances.


How can I build a seq2seq model , which is topic aware I have developed a chatbot, which is basically a seq2seq LSTM network. Which can generate text based on input text. But the problem I am having right now is it is not topic aware.
As an example :

Input : Wanna go to the stadium ? Topic: Football

Answer: No , I don't like football.

What I am planning to do , is to tokenize each possible topic

As an example {1: "Football",2: "Food" ..........}

But don't know how to embed this tokenized topic in training .

Try this approach (not embedded but filtering your model) with Topic modeling, that could be used to filter the results of your model by topic:

Separate each doc in several sentences
Obtain the best topics ( occur > 3, chars > 3 ) from sentences if you do not have an idea of topics to declare (LatentDirichlet Allocation)
Discard some topics out of interest

Finally assign a dominant topic to every sentence and filter results by topic.

I have found a reference from steps 1-3:
https://medium.com/@actsusanli/when-topic-modeling-is-part-of-the-text-pre-processing-294b58d35514


Why doesn't the binary classification log loss formula make it explicit that natural log is being used? I'm completing a DataCamp course where we are introduced to the log loss formula for binary classification:

Two scenarios are given to show how the formula is used. One with p=0.1 and one with p=0.5.  The answers the instructor displayed were 2.3 and .69, respectively.  However, using a calculator, the answers for log(0.1) and log(0.5) are -1 and -0.30, respectively. I later tried using natural log instead and got the same answers as the instructor, except negative.  Specifically, the calculator returned -2.3 for ln(0.1) and -0.69 for ln(0.5).

Is it common in math for log to implied to be "ln" or "log e" without stating it explicitly in the formula?  Also, is there something about the log loss binary classification formula that suggest that the absolute value of the result should be taken?

This comes down to the change-of-base formula. For any two numbers $a$ and $b$, the following equation is true.

$$
\log_a(x) = \frac{\log_b(x)}{\log_b(a)}.
$$

What this means is that the errors are proportional. So if you wanted to change to using $\log_{10}$, you would simply end up multiplying by a constant factor, and model selection would be the same.

Explicitly,

$$logloss(N=1) = \frac{y \log_{10}(p) + (1 - y) \log_{10}(p)}{\log_{10}(e)}$$

Or, equivalently

$$
logloss(N=1) \cdot \log_{10}(e) = y \log_{10}(p) + (1 - y) \log_{10}(p)
$$

In other words: The base of the logarithm doesn't matter because everything ends up being proportional.


How to estimate missing values when calculating NDCG I would like to compare recommendations methods using NDCG metric on MovieLens dataset.
In ranking problem, the goal is to rank items based on their relevance for user. Ranking models can be learned based on ratings matrix, where each user rates small subset of all items. Ratings for other items are unknown.
Collaborative Filtering methods can be used to create model which generalize training datasets and predict ratings for unrated items.
Let's consider following example on dataset consisted of 5 movies.
User A rated only 3 movies:

movie 1 - 5 stars
movie 3 - 3 stars
movie 4 - 2 stars

Model predict following results

movie 1 - 5 stars
movie 2 - 4 stars
movie 3 - 3 stars
movie 4 - 2 stars
movie 5 - 1 stars

How NDCG@3 should be calculated in this example ? Movie 2 get second best score but it has not been rated by user although it's highly relevant for user A based on other user ratings. Giving movie 2 1 star rating as ground true penalized model because it predicted highly relevant movie which was not rated by user.
Many papers measure model performance on MovieLens using NDCG, but I have not found details how NDCG is calculated. What is the best practice for solving this problem?
Is it good idea to estimate unknown rating value based on movie ratings median or average ?

Movie 2 get second best score but it has not been rated by user although it's highly relevant for user A based on other user ratings.

The issue with this is that you're injecting your opinion into what items are relevant to user A.

How NDCG@3 should be calculated in this example ?

You cannot evaluate your model on items the user has not rated. Generally with explicit feedback data you want to ignore missing values. Imputation may be possible, but it is more challenging and not commonly used. The correct modeling pipeline is to first make a training & testing set. Train your model on the training set and then for each user rank their known items that are present in the testing set (source: p.g. 24). Then you can compute various cutoff-sensitive metrics.
In your example, you should not be predicting ratings for movie 2 or 5. If your training set was movie 1 & movie 3 then you would make predictions for movie 4 -- obviously in a real application you should have more data so that there is more than one example in your test set.


CNN model for more than 3 channels input Are there any special cnn architectures for data, which has more than 3 channels? For example, satellite imagery, where we can consider different wavelenghts. It's clear, that we can readily use unet or something like that, but when there are 20 channels, for instance, it seems, that approach should differ. Not a neural network model, which can get feature-vector consisted of pixels from each channel, isn't sufficient perhaps, because spatial characteristics are important too.

A CNN does not care about the starting number of channels. It can be one, three, 20, or anything really. Here's a simple example in the python library Keras for how you might start out a CNN with 20 channels, assuming your images are 100x100. Obviously these numbers can be changed to whatever your problem needs.

from keras.layers import Input, Conv2D

# Assumes images are 100x100 with 20 channels
inputs = Input(shape=(100, 100, 20))

# First convolution will take in the 20 channels and output 32 channels
x = Conv2D(32, strides=(2, 2), activation='relu')(inputs)

# Second convolution will take in the 32 channels and output 64 channels
x = Conv2D(64, strides=(2, 2), activation='relu')(x)

...


What is the type of each variable i.e. Categorical or Continuous? I have following attributes in my data-set.
Name, Date of Birth, Annual Income, Tax Paid, Gender.
What is the type of each variable i.e. Categorical or Continuous? and why ?

PS. I believe categorical attributes are the ones having fixed set of Values. Please let me know if I am wrong with this assumption. Also the "Name" attribute is just a label. but what category does it fall in ?

Name attribute cant be continuous, if you really want to use it then take it as a categorical. It depends on what is your use case.
Gender is categorical.
Date of Birth/Annual Income/Tax Paid are in general continuous variable. But you may change them in categorical depending on your use case.


Convolutional neural network block notation The paper by He et al. "Deep Residual Learning for Image Recognition" illustrates their residual network in Figure 3 as follows:

I am not a neural network expert, so could somebody please explain to me what the highlighted notation above "3x3 conv, 256, /2" means? The first part is clear (convolutional neural network with a 3x3 pixel window), but what is the "256" and the "/2"?

3x3 conv, 256, /2

stands for:

3x3 Kernel
256 filters
a stride of 2 halving the spatial dimensions

The latter is explained on page 3 where the authors state

  (ii) if the feature map size is halved, the number of filters is doubled so as to preserve the time complexity per layer. We perform downsampling directly by
  convolutional layers that have a stride of 2.

This means ResNet does, except for the beginning and end of the network, not use pooling layers to reduce spatial dimensions but conv. layers.

Also, table 1 shows what is happening:

The part you have highlighted in your screenshot is the transition from conv3_x to the conv4_x layer of the 34-layer network. As you can see in the table the output size is reduced from 28x28 to 14x14 (that is what /2 does) while the filters are doubled from 128 to 256.


Adding hour data to Date feature in time series prediction I'm a noobie with regards to working with time series data. I have a time series data which has date column in YYYY-MM-DD and a hour column containing 0 to 23 representing hour of the day and the value containing a value between 70 to 100 which is what we need to predict.

Models I tried:
I checked for Stationarity of the data and I split the dataset based on time(oldest 80% train and newest 20% as a test) and tried two models: AR and ARIMA. AIC score for ARIMA was around 2500.0

My questions are: Earlier I discarded the hour feature and built models with only Date feature. So Will Adding hour data improve my model? If I'm adding hours, Should I keep it as a separate column or should I include the hour in the YYYY-MM-DD date feature?

Thanks in advance

I am a bit confused by the setup of this question but I'll try my best.

First, if your time series has YYYY-MM-DD, and then you also have an hour column with ranges 0:23, then what you have is a hourly time series. That is, the natural index of the time series is by the hour, not by any other frequency of time, and therefore, you would have no need to include either of these two columns in your "dataset".

If the hour of the day is a strong "predictor", then this is just another way of saying that your time series exhibits hourly seasonality and therefore, including hour of the day may help if you see that the residuals of the time series are not consistent with white noise (an assumption of ARIMA models) and perhaps have a superimposed pattern on the ACF plot. A seasonal ARIMA model might be of interest to you, or perhaps an ARIMAX model with an exogenous hour of the day variable. Furthermore, certain particular days might have noticeable and consistent effects, or even months. There may also be one time effects, etc. which might be useful to include as exogenous variables.


How to JUST represent words as embeddings by pretrained BERT? I don't have enough data (i.e. I don't have enough texts) --- have only around 4k words in my dictionary. I need to compare given words, then I need to representate it as embedding.
After the representation of words I want to clusterize it, find similar vectors (i.e. words). Maybe even then make a classification to a given classes (classification there unsupervised --- since I don't have labeled data to train on).
I know that almost any task can be solved "inside" BERT, i.e. using fine-tuning in final layer.
Since all described above, I have two QUESTIONS; answers/hints/anything really appreciated since i'm stuck on that:

How to just extract embeddings from BERT using some dictionary of words and use word representations for futher work?
Can we solve inside BERT using fine-tuning the next problem: a). Load dictionary of words into BERT b). Load given classes (words representing each class. E.g. "fashion", "nature"). c) Make an unsupervised classification task?

BERT does not give word representations, but subword representations (see this). Nevertheless, it is common to average the representations of the subwords in a word to obtain a "word-level" representation.

You may try to handle this as a normal tagging problem, where the tag of each word is the class associated with the word, much like part-of-speech (POS) (e.g. this) tagging or named entity recognition (NER) (e.g. this). Normally, you associate the tag to either the first or the last subword token in the word. If you prepare a dataset that way, you could fine-tune BERT to perform word tagging with the classes you need. If you only have the words, you could find some text corpus (ideally of the intended domains) and apply the described data preparation process.


One hot encoding of target variable containing classes 1 to 9 not including zero While predicting a solution for a sudoku puzzle using CNN, the target variable should predict values from 1 to 9 for all the 81(9*9) values in the puzzle. Hence the target value shape is (81,9).
Using keras.to_categorical to convert target variable from (81,1) to (81,9) shows error.
to_categorical takes argument y which must contain integers from 0 to num_classes whereas 0 is not included in the target variable considered here. It works fine when num_classes is taken as 10 but results in (81*10) shaped variable. Is there any other way to get the target variable as (81,9) without altering the target variable values?

I don't know it this is an optimal approach. The natural way to solve a SUDOKU with computer science is Linear Programming. I am curious if a CNN will solve the problem.
Can you add +1 or -1 in the post processing / pre processing of the predictions? It will be the easier way to solve it.


What is one hot encoding in tensorflow? I am currently doing a course in tensorflow in which they used tf.one_hot(indices, depth). Now I don't understand how these indices change into that binary sequence.

Can somebody please explain to me the exact process???

Suppose you have a categorical feature in your dataset (e.g. color). And your samples can be either red, yellow or blue. In order to pass this argument to a ML algorithm, you first need to encode it so that instead of strings you have numbers.

The easiest way to do such a thing is to create a mapping where:

red --> 1
yellow --> 2
blue --> 3

and replace each string with its mapped value.

However this might create unwanted side effects in our ML model as when dealing with numbers it might think that blue > yellow (because 3 > 2) or that red + yellow = blue (because 1 + 2 = 3). The model has no way of knowing that these data were categorical and then were mapped as integers.

The solution to this problem is one-hot encoding where we create N new features, where N is the number of unique values in the original feature. In our exampel N would be equal to 3, because we have 3 unique colors (red, yellow and blue).

Each of these features be binary and would correspond to one of these unique values. In our example the first feature would be a binary feature telling us if that sample is red or not, the second would be the same thing for yellow and the third for blue.

An example of such a transformation is illustrated below:

Note, that because this approach increases the dimensionality of the dataset, if we have a feature that takes many unique values, we may want to use a more sparse encoding (like the one I presented above).


What to do when we can't trust our human classifiers? Suppose we want to design a neural network that can diagnose skin cancer. We want this neural network to consider the possibility that the doctor we hired misclassified some of our images while labeling. How can we design our neural network?

First, neural networks are good in dealing with"label noise". I'm currently on mobile/vacation, so remind me to search the paper on Friday.

Second, the more important question is how to get a good ground truth. Without a good ground truth you can't evaluate your models, no matter how good they might be.

I see the ways:

(1) have multiple experts label the stuff. Then you can make the ground truth a probability, not a simple label. If 9 experts say it is cancer and 1 says it is not, you would label it with 90%

(2) wait. If you can access the patients data, it will likely be more obvious in a year (especially if it was not treated)

(3) other diagnostic methods: I'm not a medical doctor, but I'm pretty sure there are invasive methods to diagnose cancer which are reliable


Decision tree in sentiment analysis Creating a classifier to do "Sentiment Analysis" can be done with several algorithms like SVM, KNN, Neural networks,Decision tree...

and lately i have read about Decision tree and how it works and i would like to ask if someone can provide me with an explanation and a conceptual representation of how the decision tree will be created to tackle this problem, and what/how attributes will placed in the nodes of the tree
to classify a sentiment of a text as positive|negative|neutral ?

any help would be appreciated, and thanx.

Decision trees are supervised methods, so they need to be trained on some annotated data. Thus the general idea is the same as for any text classification: given a set of documents (for instance represented as TFIDF vectors) together with their labels, the algorithm will calculate which how much each word correlates with a particular label.

For instance it might find that the word "excellent" often appears in documents labeled as positive, whereas the word "terrible" mostly appears in negative documents. By combining all such observations it builds a model able to assign a label to any document.


redundancy or functional dependency of two columns Being a beginner in Data Analysis I ask you not to be much judgemental about this question.

I was tring to find if there is a standard way (function?) in Pandas module to identify redundancy of a column. For example, notably for categorical data types,  if a column is a function of another it could be considered as redundant and thereof could be disregarded without any loss of prediction power. I would prefer not to implement any heavy machinery myself, but use common knowledge methods, therefore is my question.

I guess correlation is a good metric, but it has a different meaning and it is not a good fit for two categorical columns.

Any answers would be much appreciated, cheers.

Well, it always depends, for example, on what model you might be training (i.e. some are robust to multicollinearity). I am pretty sure you are aware, but to have it said as a rule of thumb it is always helpful if you know what you are looking for, rather than hoping naively one function or method would give all the answers.

Said that, there are good progress and in fact a powerful Python package pandas-profiling that can simply takes a pandas dataframe and returns lots of useful information in a blick of an eye (well if it is not super large dataset). And yes, for redundancy, correlation is a good start, and pandas-profiling highlights of highly correlated variables based on Spearman, Pearson and Kendall matrices. Take a look at this post for quick read about pandas-profiling.

For categorical columns, often things get trickier. In past, pandas-profiling did not offer anything. I just double checked, and interestingly they implemented Cramer’s V at this Closed Issue at their repo (see this nice post 'The Search for Categorical Correlation' if you want to learn about Cramer’s V4). I cannot confirm its functionality, you gotta test it on your own, but I believe it should be reliable as many contribute to this project. Unfortunately their documentations isn't thorough, but it seems it is there, see correlations.html page.

Update: Just figured that they have an advanced tutorial under Examples i.e.  link to the Colab notebook Tutorial: report structure using Kaggle data (advanced), where the dataset is a combination of many data types including numerical and categorical, and they in fact show all the correlations elegantly, see a screenshot:

Good luck!


How to handle multi-label feature for binary classification problem? I have dataset like :

   profile     category  target
0        1      [5, 10]       1
1        2          [1]       0
2        3   [23, 5000]       1
3        4  [700, 4500]       0

How to handle category feature, this table may have others additional features too. One hot encoding lead to consume too much space.because number of rows is around 10 million. Any suggestion would be helpful.

I recommend using scikit-learn's MultiLabelBinarizer to encode your category feature.  This is the multi-label analog of one-hot encoding, but it has a sparse_output mode, which should solve your problem of limited memory.


Why training model give great result but real data gives very bad result: Azure ML Studio I am using Two-Class Boosted Decision Tree to train model.

Evaluation result I'd say really good.

But when I am using real dataset - the result is very bad.

What can possibly go wrong that makes such huge difference?

Below is the screenshot of my model:

Two Class Boosted Decision Tree parameters (default):

Your question is not clear. There's 2 ways to understand it. Which dataset did you use to train your model?

You trained and tested on a premade dataset. The result  is great. Then you applied this model to real dataset and the result is really bad.

If this is the case, you should retrain on your real dataset or apply some Transfer Learning techniques to your current model.

You trained and tested on a premade dataset. The result is great. Using the same model, you trained and tested on real dataset but the result is much worse.

I can't tell exactly the reason for this. Normally, real data is much more noisy. Did you handle missing data and do some feature engineering before training?


Conditional entropy calculation in python, H(Y|X) Input
X: A numpy array whose size gives the number of instances. X contains each instance's attribute value.
Y: A numpy array which contains each instance's corresponding target label.

Output : Conditional Entropy

Can you please help me code the conditional entropy calculation dynamically which will further be subracted from total entropy of the given population to find the information gain.

I tried something like the below code example. But the only input data I have are the two numpy arrays. can you please help me correct this ?                              [code]

def gain(data, attr, target_attr):

    val_freq = {}
    subset_entropy = 0.0
for record in data:
        if (val_freq.has_key(record[attr])):
            val_freq[record[attr]] += 1.0
        else:
            val_freq[record[attr]]  = 1.0
for val in val_freq.keys():
        val_prob = val_freq[val] / sum(val_freq.values())
        data_subset = [record for record in data if record[attr] == val]
        conditional_entropy += val_prob * entropy(data_subset, target_attr)

##Entropy
def entropy(Y):
    """
    Also known as Shanon Entropy
    Reference: https://en.wikipedia.org/wiki/Entropy_(information_theory)
    """
    unique, count = np.unique(Y, return_counts=True, axis=0)
    prob = count/len(Y)
    en = np.sum((-1)*prob*np.log2(prob))
    return en

#Joint Entropy
def jEntropy(Y,X):
    """
    H(Y;X)
    Reference: https://en.wikipedia.org/wiki/Joint_entropy
    """
    YX = np.c_[Y,X]
    return entropy(YX)

#Conditional Entropy
def cEntropy(Y, X):
    """
    conditional entropy = Joint Entropy - Entropy of X
    H(Y|X) = H(Y;X) - H(X)
    Reference: https://en.wikipedia.org/wiki/Conditional_entropy
    """
    return jEntropy(Y, X) - entropy(X)

#Information Gain
def gain(Y, X):
    """
    Information Gain, I(Y;X) = H(Y) - H(Y|X)
    Reference: https://en.wikipedia.org/wiki/Information_gain_in_decision_trees#Formal_definition
    """
    return entropy(Y) - cEntropy(Y,X)


Changing the shape of the input layer in tensorflow I have a convolutional network which consists of multiple CNN layers (which could work with any size) and a fully connected layer at the end which can only work with fixed input size.

I have trained the network for certain image size and the fully connected layer can only work for those input sizes.

I don't want to retrain the entire network when operating on different input sizes. Is there any way to achieve training of only the newly added fully connected layer.

My code for inference looks like this

def inference(images, reuse=False, trainable=True):
    coarse1_conv = conv2d('coarse1', images, [11, 11, 3, 96], [96], [1, 4, 4, 1], padding='VALID', reuse=reuse, trainable=trainable)
    coarse1 = tf.nn.max_pool(coarse1_conv, ksize=[1, 3, 3, 1], strides=[1, 2, 2, 1], padding='VALID', name='pool1')
    coarse2_conv = conv2d('coarse2', coarse1, [5, 5, 96, 256], [256], [1, 1, 1, 1], padding='VALID', reuse=reuse, trainable=trainable)
    coarse2 = tf.nn.max_pool(coarse2_conv, ksize=[1, 3, 3, 1], strides=[1, 2, 2, 1], padding='SAME', name='pool1')
    coarse3 = conv2d('coarse3', coarse2, [3, 3, 256, 384], [384], [1, 1, 1, 1], padding='VALID', reuse=reuse, trainable=trainable)
    coarse4 = conv2d('coarse4', coarse3, [3, 3, 384, 384], [384], [1, 1, 1, 1], padding='VALID', reuse=reuse, trainable=trainable)
    coarse5 = conv2d('coarse5', coarse4, [3, 3, 384, 256], [256], [1, 1, 1, 1], padding='VALID', reuse=reuse, trainable=trainable)
    coarse6 = fc('coarse6', coarse5, [6*10*256, 4096], [4096], reuse=reuse, trainable=trainable)
    coarse7 = fc('coarse7', coarse6, [4096, 4070], [4070], reuse=reuse, trainable=trainable)
    coarse7_output = tf.reshape(coarse7, [-1, 55, 74, 1])
    return coarse7_output

Actually, if you change the input the input size, nothing goes wrong with the convolutional layers but the outputs of these layers increases and that will cause to the increased number of inputs to the dense layers 1. Consequently, you will have to have extra weights and you have to train them. That is why it is better to have a fixed size input. But there are other solutions that can be expanded to your task. Take a look at the contents of the third week of this course. As you will see, you have to input patches of the image to the network but what is done there is for detection tasks, you have to expand it to your task, but maybe resizing all inputs to a predefined size is simpler.


Time series analysis model evaluation performance metrics integration in time series application After study in time series analysis, I recognized RMSE and MAPE  are the best evaluation metrics for used model in real time series application. But my queries are below as this is my first practice application project in time series analysis:
If I use past 1 year data for training and forecast for next 30 days but we will get actual data after 30days in real time. How could we integrate  these RMSE and MAPE with time-series application?
As I heard from some data scientist if I skip first training cycle and calculate these RMSE and MAPE in next retraining cycle for past training cycle where I have actual data and forecast data, Is this approach useful ?
I concerned about calculated RMSE and MAPE could help to improve model performance in real time.
Could the evaluation metrics calculation every time on retaining useful from production point of view?
My thanks in advance

I am assuming you have a model running in production and it retrains periodically (example: every month) and it forecasts for the next X days (example: 30 days) and you are trying to evaluate the model using RMSE and MAPE (If this assumption is wrong, please clarify it in your question)

If I use past 1 year data for training and forecast for next 30 days but we will get actual data after 30days in real time. How could we integrate these RMSE and MAPE with time-series application?

you will have to back test your model, i.e. predict with your model on a day in the past from which you will have 30 forecast-ed values and 30 actual values. Then the aggregation happens in mean part of the Root Mean Square Error or Mean Average Percentage Error. Ideally, your model should not have seen the evaluation data during the training phase.

As I heard from some data scientist if I skip first training cycle and calculate these RMSE and MAPE in next retraining cycle for past training cycle where I have actual data and forecast data, Is this approach useful ?

Yes, this is one of the ways to evaluate the model.

Could the evaluation metrics calculation every time on retraining useful from production point of view?

It is a good practice (unless it costs too much) to monitor the model in production regularly, for this you will have to estimate the evaluation metric in the frequency that is most suitable for you.


Feature importance for random forest classification of a sample Using a random forest is it possible to determine which features were the driving features to classify a specific sample as class A?

I know I can ask which features are more important to perform classification of ANY sample, but can I ask this for a specific sample?  E.g. Why was sample 1 classified as A?  Which of its features were much more like class A than class B?

Does it even make sense to ask this question of a random forest?

Bonus points on how to do it with sklearn in python :)

EDIT

Question has been answered in a crosspost here:  https://stats.stackexchange.com/questions/174229/feature-importance-for-random-forest-classification-of-a-sample

Python implementation here:  http://blog.datadive.net/random-forest-interpretation-with-scikit-learn/

So let me clarify your query: you have trained a random forest model to classify a dataset into multiple classes, e.g. A, B, C, or D. Now, you want to understand for each different class label (e.g. A), which features contributed to it being classified as class A vs. not class A, and so on.

To find the importance of a variable in random forest, each variable is permuted among all trees and the difference in out of sample error of before and after permutation is calculated. The variables with highest difference are considered most important, and ones with lower values are less important.
this method gives importance for the entire multinomial classification, i.e. A vs. B/C/D, B vs. A/C/D, etc. and not just for one label. So you can't claim that feature1 is more important to identify class A, but feature 2 is more important to identify Class B.

I've been using the random forest package implemented in R and it does not seem to provide any way to decipher the internal details on the one-vs-all categorization.

If you're really interested in doing just that, then you should fit a set of random forest models each for classifying class A vs. the rest, class B vs. the rest, etc. The variable importance of these separate models would give you the feature importance for classifying each label.

You can use the same library to fit these models using scikit-learn.  You may refer to their help page here - http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html


Convolutional neural networks for non image dataset Can we use Convolutional Neural networks for a non image dataset for prediction?

The dataset is a record of student academic details

I know that CNN is mostly used in  computer vision and image processing for analyzing visual imagery.
And it is also used in natural language processing for text classification.

I am new to this please guide me through

Thank you!!

Short answer:

Yes, absolutely

Longer answer:

As a rule-of-thumb, you can always use any technique on any problem, as long as you define it properly. CNNs are meant to work on signal data. 1D signals can be sound waves, financial records, etc. 2D signals are usually images. 3D signals can be videos (with time being the 3rd dimension). And obviously, you can have as many dimensions as you'd like.

So as long as you can turn your input data (let's say a single training instance) into a signal, then be my guest. However, I fail to see how this dataset can be viewed as a signal.


Random Forest Regression. How to represent really long list of categories for processing I'm trying to build a model to solve a regression task.

Simplified, the data look like:

7.16, A 1, B 4, C 15, D 3
8.21, C 3, D 2, G  7, M 4, Y 9
1.85, D 3, N 1, L  1

The first column is a label, consequent columns are a class and respective number.

For example, the first row demonstrates that there is one of A, four of B, fifteen of C, and three of D - such a configuration gives us 7.16.

My Primary Question:

How do I represent my data so that I can fit a Random Forest Regression model?

My Thoughts:

Figure out the whole list of classes (there are 26 letters, but I have way more classes) and put zeros for all of the unknown classes into each record. Vector.sparse() might help here as it can reduce the amount of required memory.
Figure out the entire list of classes, correspond a number starting from 0 and convert each record into a vector (i.e. the first one would look): 0 1 1 4 2 15 3 3. Then each column is treated as a separate feature, but in this case I have to specify that some of my features are categorical and that's not possible with Spark since different records are of different length (In Spark I have to provide columns of features, which are categorical with number of classes. In the example that would be a map(0 -> 26, 2 -> 26, 4 -> 26) - like every second column is a class with 26 categories).
It's possible to build features like A:1, B:4, C:15, D:3 and treat all of them as categorical. Or move decision burden (categorical vs continuous) to Spark itself.

If the problem is that you don't know all the categories in advance and different records have different categories, then the hashing trick might help.


Where to find list of Tensorflow pretrained models available in download.tensorflow.org/models I am trying the find the pretrained models (graph.pd and labels.txt) files for Tensorflow (for all of the Inception versions and MobileNet)

After much searching I found some models in, https://storage.googleapis.com/download.tensorflow.org/models

like, https://storage.googleapis.com/download.tensorflow.org/models/inception5h.zip

https://storage.googleapis.com/download.tensorflow.org/models/inception_dec_2015.zip

But I found this as links from various blogs/etc. But is there a way to list all the models/zips in this dir? https://storage.googleapis.com/download.tensorflow.org/models

I want the latest of Inceptionv3 and MobileNet, does anyone know the file names in models for these?

It seems the models hosted on this are not listed anywhere publically (or at least my research failed!). However, there are official and several nonofficial GitHub repositories with high-level TensorFlow model definitions and pretrained weights. For example:

Nonofficial: that includes NASNet, ResNeXt, ResNet, InceptionV4, InceptionResnetV2, Xception, DPN (need PyTorch).
Official: contains a wide range of official and research models such as resnet, wide-deep, inception, delf, and tcn.

Hope it helps.


Python Seaborn: How can one obtain a box plot (box and whisker plot) for the non-diagonal variable pairs in a pairplot()? I tried using kind="box" but am not sure if that works. For the example I tried, I got blank subfigures in the non-diagonal parts of the PairGrid.

Edit: Details.

The function I am trying to use is seaborn.pairplot. As mentioned in the link, by default, this function will create a grid of Axes such that each variable in the data parameter will be shared in the y-axis across a single row and in the x-axis across a single column. The diagonal Axes are treated differently, drawing a plot to show the univariate distribution of the data for the variable in that column.

I am looking for a way to make sure that the kind parameter leads to a box plot (seaborn.boxplot) for the non-diagonal parts of the PairGrid.

This wouldn't be possible in seaborn without changing the source code.
pairplot is designed with only two options for the non-diagonal plots: scatter or reg.  This can be seen in the source code:
if kind == "scatter":
    plot_kws.setdefault("edgecolor", "white")
    plotter(plt.scatter, **plot_kws)
elif kind == "reg":
    plotter(regplot, **plot_kws)

One option would be copy the source code and create a function similar to pairplot, but with the ability to plot boxplots.


How to write formula inside the loop to run this code in every hour continously in every day in panda python I have a csv file with five column data value.

five columns are ['A','B','C','D','time']

Here my 'D' column is the output column and 'A'is my first column.

Here I upload my coda and csv file.

data = pd.read_csv('data20.csv')
data = pd.DataFrame(data,columns=['time', 'A','B','C','D'])
for i in range(1, len(data)):
   data.loc[i+1,'A'] = data.loc[i, 'D'] + data.loc[i, 'B'] - data.loc[i, 'C']

my csv file ;

In my csv first row in A column 63 is represented and my output is 63 in D column. D column is my desired outputs. then in second row in A column its represented as 0. but I want to apply previous output value(63) into second row as A column input. Then that value will calculate by column B and reduce by C column then my desired output value is 60 represent in D column. So like this process will continue. So in A column 0 values replace by previous output value in D column.

Here I want to call that every hour this process continue. In my csv file till to 2 p.m output value is 72.

So then this 72 is will be the input of A column at 3p.m.then B and C column values will calculate or reduce using this formula will give the output for at 3.00 p.m.

So like this I want to find the value of D column at 4p.m, 5 p.m, 6 p.m till to 12 midnight and it will continous to next day. So I wrote the code and I ddn't have any idea to give time for this process. Can anyone help me to solve this problem?

you could do something like

import time
while True:
    <your code>
    time.sleep(3600)

this would execute your code every hour.
I hope this answer your question.


What are the equations involved in calculation of the parameters of embedding layer? I'm trying to perform sentiment analysis on some data using keras.I'm using embedding layer and then LSTM. I know that embedding layer decreases the sparsity of the one hot encodings of the words and its parameters are trained while back-propagation, but I don't know the mathematics of its implementation.

Thanks in advance.

Like all hidden layers in a neural network, an embedding layer can be thought of as a feature extractor with parameters that are automatically learned during training.
So, like any other layer, the parameters are adjusted during training by the backpropagation algorithm.

The specific equations used to compute error gradients and update weights are likely to depend on the optimizer used to train the network.  The classic optimizer is stochastic gradient descent, but more sophisticated optimizers like Adagrad, RMSProp, and ADAM are commonly used nowadays.


Why Does XGBoost Keep One Feature at High Importance? I am training an XGboost model for binary classification on around 60 sparse numeric features. After training, the feature importance distribution has one feature with importance > 0.6, and all the rest with importance <0.05.
I remove the most important feature, and retrain. The same distribution forms; the most important feature has importance > 0.6, and the rest have < 0.05. I continued to remove the most important feature and retrain, remove and retrain, remove and retrain, etc. My f1-score started to drop, but every time there was one feature more important than the rest.
Also worth noting, when I removed the most important feature and retrained, the new most important feature was not the second most important feature from the previous training.
I cannot explain this behaviour intuitively. Does anyone know why this pattern arises?

According to the docs "gain" is the default, which is calculated by this formula, where I don't think intuition would help. General consensus is, that feature importance is a tricky concept, as long as your model performs fine, you shouldn't worry about it. Especially with the linked formula for gain you can see that the first split may be the most important by far, depending on gamma, so that every other feature importance is low by default.
If, on the other hand you chose "weight", that would be just be the number of times the feature is chosen to split. That may mean that one feature contains a lot of unique values and can be chosen a lot of times and still improve accuracy.
All of this depends heavily on the data and parameters you are using. There isn't any general answer to this pattern.


What technologies are fastest at performing joins on large datasets? By "large", I mean in the range of 100m to 10b rows.

I'm currently using both Hadoop MapReduce and Amazon RedShift. MapReduce has been a little disappointing here. Redshift works very well if the data is distributed well for the given query.

Are there other technologies that I should be looking at here? If so, what are the trade offs?

More importantly than the technology is the type of join you are using. For instance if the join keys are sorted, you can use sort merge joins and use join orders to get a better performance.

That being said, you can use in memory solutions for fastest joins if the size of your intermediate results will not blow up your cluster memory. Look at Spark SQL or Mem-SQL for instance.


Generating image embedding using CNN I have a CNN model using cifar -10 dataset. The model was built using Keras (Tensorflow).

Now based on this model, I have to generate an image embedding (vector). That means - an input image comes and I have to output the embedding vector of that image.

I am not sure how to do that. This is not a straight forward prediction/classification output. Rather I have to output the embedding of the input image (which is off couse the predicted embedding but an embedding vector nonetheless).

Any suggestion?

You should use something like an autoencoder.
Basically. you pass your images through a CNN (the encoder) with decreasing layer size.
The last layer of this network is the one that produce the embeddings (that is, a lower dimensional representation of your input), and the number of neurons you use here is the length of your vector embedding for the input images.

Now, your embeddings are useful only if they actual encode the data in your images.
To achieve this, you need another network (the decoder) that takes as input the image embedding and outputs an image with the same dimension of the input.
Here, you try to minimize a loss function that tells you the distance between the image you generate from the embedding and the initial image (it might be Euclidean distance between pixel values).

Finally, if you need to output the embedding of an image, you just need to pass the image through the encoder network and collect the output.

Here's a simple tutorial.


To which category does this algorithm belongs? I have came across the Catboost package. Among the classes in categories in Sklearn, Catboost seems to belong to Ensemble methods.
What are then the advantages of Catboost over AdaBoost, Bagging etc.?

Ensemble Methods as defined in Wikipedia:

  In statistics and machine learning, ensemble methods use multiple
  learning algorithms to obtain better predictive performance than could
  be obtained from any of the constituent learning algorithms alone.

All those methods you mentioned are tree-based ensemble models:

Bagging (Breiman, 1996): Fit many large trees to bootstrap-resampled versions of the training data, and classify by majority vote.
Random Forests (Breiman 1999): Fancier version of bagging (only a subset of features are selected at random, unlike bagging where all features are considered for splitting a node).
Boosting (Freund & Shapire, 1996): Fit many large or small trees to reweighted versions of the training data. Classify by weighted majority vote. There is a nice article explaining gradient boosting trees.

In general (in terms of prediction capability, boosting to be the best):

Boosting > Random Forests > Bagging > Single Tree

You might be wondering where AdaBoost fits?

Adaptive Boosting (or in short AdaBoost, was the first really successful boosting algorithm) works on improving the base learner esp. where it fails on predictions. Please note that the base learner can be any machine learning algorithms upon which the boosting is applied to obtain a strong learner. When Decision stumps are used as base learner, AdaBoost is comparable to the above-mentioned boosting trees. You might be asking, again, what are their differences, see below (taken from this book):

Modern Boosting Trees

Due to the success of gradient boosting trees, there are types of boosting algorithms, namely: Gradient Boosting, XGBoost, and Catboost. They are conceptually very similar, yet they differ in e.g. sampling methods, regularizations, handling categorical features, performance etc. Strongly recommend checking this article out if interested to learn more.

Personal Note: About 1.5 year ago I was a fan of XGBoost (for many reasons), till I experimented Catboost. Now I really like Catboost. First of all, it easily handles a mixture of numerical and categorical features EVEN without coding the categorical ones. And the default hyperparameters give comparable results to fine-tuned hyperparameters in XGBoost, thus less hassle. At present Catboost community is smaller than let's say XGboost, kind of makes it less attractive, but it is growing. Last note: I am not affiliated to any of these methods/implementations.

Hope, it is now better! ;-)


ML technique to predict next year output based on text quantities I have a random data that I would like to predict how much a quantity will be in 2020.
The data looks like this:

 year        components           total_components
 2019        [Pen, Pencil, Books       4
             Paper]
 2018        [Pen, Pencil, Books       5
             Paper, Eraser]
 2017        [Pen, Pencil, Books       6
             Paper, Eraser, Napkin]
 2016        [Pen, Pencil, Books       7
             Paper, Eraser, Napkin,
             laptop]

In my head, I thought of time series forecasting or RNNs but the data is a bit strange to apply the techniques.

Which ML technique would you suggest here? Thanks :-)

I'm not sure machine learning is the right method to count the number of arguments in a list. You can just write a function that does that and it's all you need...

Now if you want to predict the number of total components, assuming it holds some kind of temporal relation, there are a few techniques possible. But remember: Garbage in, Garbage out...

The easiest is to build a regression model by using past numbers as features, this gives an equation like this:

$$Y_t = a_1 \times Y_{t-1} + a_2 \times Y_{t-2} + ... + b$$

Next you can look at time series (ARIMA, SARIMA), which are a bit more advanced: additional to the time steps, they can take moving average and seasonal components into account.

And finally, you can also look at neural networks. If you have enough, and more importantly meaningful data, you can look at multilayer perceptron with a time window, 1d-convolutions and RNN/LSTM... Plenty resources exist if you want to go with this.


Error after merging two Deep Learning models VGG16 and ResNet50 I have merged two different models namely VGG16 and ResNet50 and given the outputs of the two models as input to another model. I have checked the Layers graph is correct. Before merging the code was running perfectly fine giving correct outputs. I am getting an error:
"ValueError: Shapes (None, None) and (None, 7, 7, 3) are incompatible" on the line 6

ValueError
Traceback (most recent call last)
<ipython-input-36-620554d0106f> in <module>()

4     epochs = 200,

5     validation_data = validation_generator,

----> 6     validation_steps=2

my code is:
inputs_2 = keras.Input(shape=(224, 224, 3), name="img")
vgg = VGG16(input_tensor=inputs_2, weights='imagenet', include_top=False)
for layer in vgg.layers:
    layer.trainable = False

resnet = ResNet50(input_tensor=inputs_2, weights='imagenet', include_top=False)
for layer in resnet.layers:
    layer.trainable = False

mergedOutput = Concatenate()([vgg.output, resnet.output])
x = layers.Dense(256, activation="relu")(mergedOutput)
prediction = Dense(3, activation='softmax')(x)
model = Model(inputs=vgg.input, outputs=prediction)

model.compile(loss="categorical_crossentropy",optimizer='adam',metrics=['accuracy'])

keras.utils.plot_model(model, "mini_resnet.png", show_shapes=True)
train_datagen = image.ImageDataGenerator(
rescale = 1./255,
shear_range = 0.2,
zoom_range = 0.2,
horizontal_flip = True,
)

test_dataset = image.ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
TRAIN_PATH,
color_mode = "rgb",
target_size = (224,224),
batch_size = 32,
class_mode = 'categorical'
)
print(train_generator.class_indices)

validation_generator = test_dataset.flow_from_directory(
VAL_PATH,
color_mode = "rgb",
target_size = (224,224),
batch_size = 32,
class_mode = 'categorical')

history = model.fit_generator(
train_generator,
steps_per_epoch=8,
epochs = 200,
validation_data = validation_generator,
validation_steps=2
)

The 2D feature map is extended to the output layer and hence throwing errors at the Softmax Layer.
 Add Flatten Or GlobalAveragePool after Concatenate.
mergedOutput = keras.layers.Concatenate()([vgg.output, resnet.output])

mergedOutput = keras.layers.GlobalAveragePooling2D()(mergedOutput)  # This line

x = keras.layers.Dense(256, activation="relu")(mergedOutput)


What's the best graphic in R to study air pollution? I have a Python dataframe (in addition to an excel file) with data on air pollution for countries, (one line every day between 2014 and 2020). I would like to see what's the polluant the most related to industrial production, fort that I want to compare the data during quarantine and after.

These are the first lines of my excel file (also a python dataframe)

I am new to Python/R as well as statistics and I do not really know how to handle this data and what type of graphic I should use ?

Any information is welcome

Thank you very much

Well you can just start with simple line graph. Each line in graph will represent Specie. The X axis will be years from your data and Y axis will be the median. I am not sure why you have median and not actual data but this is how you can chart with visualization. Use this link to start.


h2o, different stopping metric leads to different optimal for hyperparameters I want to choose the "optimal" hyperparameters for gbm. So I run the following code using the h2o package

# create hyperparameter grid
hyper_params = list(ntrees = c(10,20,50,100,200,500),
                    max_depth = c(5,10,20,30),
                    min_rows = c(5,10,20,30),
                    learn_rate = c(0.01,0.05,0.08,0.1),
                    balance_classes=c(T,F))

# random subset of hyperparameters
search_criteria = list(strategy = "RandomDiscrete", max_models = 192, seed = 42,
                       stopping_rounds = 15, stopping_tolerance = 1e-3, stopping_metric = "mean_per_class_error")

# run grid
gbm.grid <- h2o.grid("gbm", grid_id = "gbm.grid", x = names(td.train.h2o)[!names(td.train.h2o)%like%"s_sd_segment"], y = "s_sd_segment",
                     seed = 42, distribution = "multinomial",
                     training_frame = td.train.hyper.h2o, nfolds = 3,
                     hyper_params = hyper_params, search_criteria = search_criteria)

# sort results
gbm.sorted.grid <- h2o.getGrid(grid_id = "gbm.grid", sort_by = "mean_per_class_error", decreasing=FALSE)

hyperparameters_gbm <- as.data.table(gbm.sorted.grid@summary_table)

This gives as optimal combination for the hyperparameters

learn_rate  max_depth   min_rows    ntrees
0.08    10  5   200

Then I am trying to do the same but with different stopping_metric. So in the above i use mean_per_class_error, and in the following i use logloss, so i run the following code:

# create hyperparameter grid
hyper_params = list(ntrees = c(10,20,50,100,200,500),
                    max_depth = c(5,10,20,30),
                    min_rows = c(5,10,20,30),
                    learn_rate = c(0.01,0.05,0.08,0.1),
                    balance_classes=c(T,F))

# random subset of hyperparameters
search_criteria = list(strategy = "RandomDiscrete", max_models = 192, seed = 42,
                       stopping_rounds = 15, stopping_tolerance = 1e-3, stopping_metric = "logloss")

# run grid
gbm.grid <- h2o.grid("gbm", grid_id = "gbm.grid", x = names(td.train.h2o)[!names(td.train.h2o)%like%"s_sd_segment"], y = "s_sd_segment",
                     seed = 42, distribution = "multinomial",
                     training_frame = td.train.hyper.h2o, nfolds = 3,
                     hyper_params = hyper_params, search_criteria = search_criteria)

# sort results
gbm.sorted.grid <- h2o.getGrid(grid_id = "gbm.grid", sort_by = "logloss", decreasing=FALSE)

hyperparameters_gbm <- as.data.table(gbm.sorted.grid@summary_table)

which gives as optimal combination for hyperparameters

learn_rate  max_depth   min_rows    ntrees
0.1 20  5   500

I know that i use as argument strategy = "RandomDiscrete", but still for instance the optimal combination for gbm using stopping_metric = "mean_per_class_error" is the "50th optimal combination" for the gbm using stopping_metric = "logloss", and the "2nd optimal combination" for gbm using stopping_metric = "logloss", is the "14th optimal combination" for gbm using
stopping_metric = "mean_per_class_error"

Why could that happen ?

First of all, you are using different metrics to determine how well you are doing, that means it's not weird that different metrics find different hyperparameter settings that work better. Second of all, some hyperparameters might not matter for the problem you are solving, which means all the signal you are getting from those hyperparameters is noise. Third of all, most of the machine learning algorithms are stochastic, meaning there is randomness involved in training them and sometimes in evaluating them, this means even starting the same grid or random search could lead to different hyperparameters. That said the probability of that is only high if the real performances are close to each other.


Using nlp to analyze accident report I want to use Natural Language Processing to analyze traffic accident reports and from the text determine two things:

Direction of vehicle travel (just compass directions like north, southeast, etc.)
Vehicle movement descriptions (e.g. backing, turning left on a red, stopped in traffic, turning right, parked).

A fragment of an accident report narrative looks something like this:

  TU2 was parked and attended on the west side of Main St, facing west,
  engine turned off. TU1 was parked on the east side of Main St, facing
  east and exited the parking spot, driving backwards. TU1's rear
  collided with TU2's rear. TU2 was still parked with the engine off
  when rear ended by TU1.

The accident reports are in thousands of files, with all accident reports for one year in one city in a single file.  The "answers" (labels) to vehicle travel direction and movement descriptions are provided, along with the narrative of the accident report.  So I have a decent training dataset.

I was thinking about starting with an approach like an n-gram bag of words and a simple classifier for vehicle direction (north, southwest, etc.).  Would that be a good start?

Firstly, I hope that the label is either a short summary or words of varying length, not just one word direction. Because moving cars involved in an accident may have multiple directions, or one car could be just parked like the example.
Secondly, given that you are planning to predict varying length label, and given the example text, I am pretty sure that bag of words will not work well. You need the context. And you dont have million sized training set to train a transformer deep neural net. So try to utilize  pre-trained embeddings like USE, GloVe etc. If you are using the embedding, it provides you awesome feature engineering done. Just train some low complexity model like random forest/ xgboost on (label, embeddings). You can also explore pretrained summary generators like bertseq2seq, however the code for this didn't work for me yet.


Autoencoder anamoly detection I recently  learnt about the anamoly detection using  autoencoders(specifically denoisinng autoencoders).To train the autoencoders do we need  a data  having some pattern? or is there some way to train in abnormal data ?Also how we decide that the data is suitable for training autoencoder model.

You need normal data to train on. If you have abnormal instances also, those should be excluded from the training set. Having access to labeled abnormal/normal data is very useful for the validation and testset.
Anything that differs from the normal data (as learned by the autoencoder) is considered an anomaly.

If you have a lot of labeled abnormal and normal data, then you can consider using binary classification instead.


Can i use chi square test to remove a particular variable from the model? I have 5 variables - $v_{1}$, $v_{2}$, $v_{3}$, $v_{4}$, and $v_{5}$. All are categorical variables.

I conducted a $\chi^{2}$ (chi-square) test to understand the relationship between these variables. I could see that $v_{3}$ and $v_{4}$ have p-values of 0.  I understand that these two variables are dependent on each other. Should I remove one of the variable from further analysis?

What if the same $v_{3}$ and $v_{4}$ are independent of other variables, like $v_{1}$, $v_{2}$, and $v_{5}$?

What you described is the Filter Method. One of the three methods of Feature Selection. The most common mistake that one could do is to make the feature selection as an independent step on your whole process and then decide which model he will use. The most appropriate way to do it is to include your Feature Selection process in your accuracy test and try different kind of feature selections. The best way for the feature selection is the domain knowledge. If you don't have it, then you start working with one of the three methods, like the Filter Method.

What you need to do is to test the chisquare or mutual information on your features with your label column. This will give you which of the features have low effect on your prediction and you can remove it.

You can find a briefly detailed article about Feature Selection here and here.


when DataFrame has mutliIndex,how to visualize it? I'm doing the "video games sales" project downloaded from kaggle, the data is like this:

and I want to know on each Platform, which are the most three popular Genres? but I don't know how to visualize it, I use pandas to filter the data, here is the code:

so how could I use pandas(or seaborn,etc.) to finish this target? I'm appreciate for your answers...

So your question is how to plot the top 3 game genres for each game platform (console). You are definitely on the right track in using the groupby method.
It is a bit long winded, but this would definitely do the job:
c = df.groupby(["Platform", "Genre"]).size().reset_index()
c.columns = ["Platform", "Genre", "Frequency"]

for p in df["Platform"].unique():
    """
    This line will take a subset of the c dataframe such that it only contains
    entries for a particular platform.
    The .sort_values() method will then arrange the frequency in descending order.
    The .head(n) will then take the top n entries and therefore, the top three genres
    for a particular platform.
    """
    df_ = c[c["Platform"] == p].sort_values("Frequency")
    sns.barplot(x = "Platform", y = "Frequency", hue = "Genre", data = df_)
    plt.show()

Here, you can edit the code so that for each platform, each graph appears on a separate axis.


Machine Learning/Text Analysis/Algorithm Pointers I'm looking for a reference or point in the right direction since I'm not too familiar with machine learning or algorithms. I'd prefer to work in R, but I could also do Python. Any reference to a specific package I could use would be great.

So I have a list of Senate bills and their descriptions that are one or two sentences long. I have a subset of bills that also have accompanied bill types (ie "budget", "immigration", etc). I want to create an algorithm that will assign a bill type(s) to the other subset of bills that don't have bill types.

I would want to be able to do this by using the underlying relationship between descriptions and bill types for the first subset and apply it to the second so that I can predict bill type from bill description. Does this make sense? Any advice would be welcome!

Yes, This absolutely makes sense. This is a common NLP (Natural Language Processing) problem. You should use word embedding models alingside LSTM (Long Short Term Memory) and deep neural networks. Actually, first you should represent every word as a vector of fixed dimension (e.g. 100) using word2vec. Then you should build a deep neural network architecture. The inputs of this network are your word vectors for each bill that are concatenated to form a sequence of vectors. The output of this network is a label that indicate which type of bill you are considering. I highly recommend that you use Keras which is a great python package for dealing with deep learning and NLP. Also, you can use existing word2vec sets that contain vector representations of most of words of a language (e.g. English). For example you can use GoogleNews word2vec set.


What is the difference between observation and variable? I have a matrix with size m×n that is built from n number of individuals for person identification. So, n is the number of person and m is the number of feature's value for the person.

It makes me confused about observation and variables. What will I call n and m? Which one represents observation and which one represents variable? My confusion will be cleared to you if you visit the following link: How to do SVD and PCA with big data?

In your case, 'n' is the number of observations, and 'm' is the number of variables.  Think of a table in a database - each row is a 'record', and each record has various properties.  A record corresponds to an observation.  Each property is a variable or property of that observation.

In some contexts people make a distinction between data (i.e. observations, things observed) and variables (parameters to learn - things that vary in a mathematical sense).  But in the context of the link you sent, a variable is a feature.


How to dual encode two sentences to show similarity score I've been trying to grasp the concept of Google's semantic experiences. By using it, I'm planning to implement a semantic query tool.

With universal sentence encoder I can first pre-encode all sentences and put them in the database. When user wants to perform query, input will too be converted in 512-dimensional vector, and we will perform sequential search on whole database by comparing cosine similarity (highest similarity vector is picked). But this is extremely slow...

Fortunately, on their semantic experiences page, they wrote the following:

  The Universal Sentence Encoder model is very similar to what we're
  using in Talk to Books and Semantris, although those applications are
  using a dual-encoder approach that maximizes for response relevance,
  while the Universal Sentence Encoder is a single encoder that returns
  an embedding for the input, instead of a score on an input pair.

One of the simpler methods they use for transforming a sentence into embedding vector is DAN (deep averaging neural network).

From what I understand, sentence is split in words, which are converted into vectors (word2vec), then we get average of all vectors that we have obtained. Average value is finally passed to one or more hidden feedforward layers, and eventually output layer that has softmax activation functions and has 512 neurons.

How can I create a dual encoder though? Do I use two different neural networks? Or does output just contain one neuron that outputs similarity?

Thank you!

P.S

My apologies if anything ignorant has been mentioned here, I'm quite confused with the concept of projecting lower dimensional space to 512 dimensional vector space.

How can I create a dual encoder though? Do I use two different neural
  networks? Or does output just contain one neuron that outputs
  similarity?

You don't need to create two different neural networks (or two different encoders) for it. Whenever you have a symmetric pieces problem, like in your case, its two sentence similarity, you should encode both the sentences using the same encoder and then use cosine similarity. (It can be the similarity of two faces also - just the way FaceNet is expected to be used.)

  we will perform sequential search on whole database by comparing
  cosine similarity (highest similarity vector is picked). But this is
  extremely slow...

You are right, I am assuming you would be using a for/while loop for computing the similarity. In computing the vector similarity of one sentence against a database of thousands or millions, you should make use of a high-performance library like BLAS which would do the same calculations sub-second.

Facebook three years ago came out with a faster matrix library called FAISS. This would reduce your speed drastically.


On what threshold we should resample the data? When working with churn datasets, we usually find imbalanced datasets. My question is how to decide on what basis we should resample the data.
For example: while splitting the data before training we split in train and test on the threshold (70-30 or 70-25), likewise if I have 62% of 0 class, 38% of 1 class in this case do we need to resample the data?

First of all, resampling or artificial sampling is not necessary nor panacea in many cases of imbalanced learning. Other methods may yield good results (eg threshold tuning, class weights adjustment and so on..).
That being said, resampling is indeed helpful in many cases of imbalanced learning.
But when do we have imbalance really? Is a 60-40 split imbalanced or not?
This is the heart of your question and cannot be answered fully (except for extreme cases) unless other methods are tried first and then resampling.
In many cases whether imbalance indeed is a problem depends on the case at hand and there are no hard limits (except of course extreme cases as mentioned above).
So first try other methods to adjust balance and then try resampling, is my advice.


How to build a database of image data for machine learning? I want to build a database of image data for machine learning. But how should this be done? I'm assuming people don't just dump all of their image data into a folder? Do they use a relational database management system, like MySQL? Or do they use a NoSQL database, like MongoDB? Is there a textbook that explores this part of machine learning in particular? Is this what "data warehouse" refers to?

There are several approaches to this as you need both the input (images) and if your problem is a classification one, you need to reliably store the labels. You might also have some additional information about the images that could be useful for your problem:

you can store the images in such a way that all information is contained in the permanent store (for instance folder names with the labels that you want to learn and all the images of a given class within that folder). Keras has a method that allows you to create a dataset from a directory tf.keras.preprocessing.image_dataset_from_directory.

another way (which I prefer) is to store in a (SQL) database all of the metadata (label, image url in a table for instance). This is more flexible because you can easily change a label, add a new category without having to move images around. This also allows you to change the format and add additional data related to each image.


I am working on a project, and I've been trying to come up with a custom loss/score function. Would really appreciate some help So, the loss function needs to be a function of (say) two variables A and B. It needs to be continous and differentiable in terms of A. Basically, I have a set of precomputed values (B), and during training, after every forward pass, I get a value A. A can vary from 0 to 1. And B can vary from 0 to (say) a bit large value, 10.

Now, this is how I want the loss function to work :

A - high, B - high
=> Ideal situation. Loss should be a small value i.e. No change in A required.

A - high, B - low
=> Loss should be such that it moves towards decreasing the value of A.

A - low, B - high
=> Loss should be such that it moves towards increasing the value of A.

A - low, B - low
=> Ideal situation. Loss should be a small value i.e. No change in A required.

Instead of computing $A$, make your algorithm compute $10\times A.$
Then use your favourite loss function for continuous variables. The most common one is the squared $L^2$-norm $l(10\times A, B) = (10\times A-B)^2.$

Here is a plot of $l$ that I made with Wolfram Alpha:.


What is the best way (cheapest / fastest option) to train an model on massive dataset (400GB+, 100m rows x 200 columns)? I have a 400GB data set that I want to train a model on. What is the cheapest method to train this model? The options I can think of so far are:

AWS instance with massive RAM and train CPU (slow, but instances are cheap).
AWS instance with many GPUs and using Dask + XGBoost to distribute (fast, but expensive instances, and  I don't even think there would be an instance large enough to handle).

I have just assumed XGBoost is the best package since its tabular data, but if another gradient boosted tree package would be better at handling this, that would be acceptable as well.
Any help would be greatly appreciated.

I don't know many boosting packages but I've been using XGboost for a while now and the biggest tabular dataset I've had was more than 40 times smaller than yours. The training took 2-3 days.
In my experience training time is worse than linear with the size of the data even though it highly depends on the data itself and the hyperparameters you chose. My guess is your training would be very (too) long.
If you really want to use XGboost, you should train on GPU, it seems to me that you are looking at cloud providers, I know google offers managed training of XGboost on GPU, others surely do as well.
With that amount of data I think you should consider using deep learning. You could maybe use tabnet which is a great model developed for tabular data by Google AI. It is easy to try using pytorch for instance.


one-hot-encoding categorical data gives error I am currently working on the Boston problem hosted on Kaggle.  The dataset is nothing like the Titanic dataset.  There are many categorical columns and I'm trying to one-hot-encode these columns.  I've decided to go with the column MSZoning to get the approach working and work out a strategy to apply it to other categorical columns.  This is a small snippet of the dataset:

Here are the different types of values present in MSZoning, so obviously integer encoding only would be a bad idea:

['RL' 'RM' 'C (all)' 'FV' 'RH']

Here is my attempt on Python to assign MSZoning with the new one-hot-encoded data.  I do know that one-hot-encoding turns each value into a column of its own and assigns binary values to each of them so I realize that this isn't exactly a good idea.  I wanted to try it anyways:

import pandas as pd
from sklearn.preprocessing import LabelEncoder, OneHotEncoder

train = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/train.csv")
test = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/train.csv")

labelEncoder = LabelEncoder()

train['MSZoning'] = labelEncoder.fit_transform(train['MSZoning'])
train_OHE = OneHotEncoder(categorical_features=train['MSZoning'])
train['MSZoning'] = train_OHE.fit_transform(train['MSZoning']).toarray()

print(train['MSZoning'])

Which is giving me the following (obvious) error:

C:\Users\security\Anaconda3\lib\site-packages\sklearn\preprocessing\_encoders.py:392: DeprecationWarning: The 'categorical_features' keyword is deprecated in version 0.20 and will be removed in 0.22. You can use the ColumnTransformer instead.
  "use the ColumnTransformer instead.", DeprecationWarning)
Traceback (most recent call last):
  File "C:/Users/security/Downloads/AP/Boston-Kaggle/Boston.py", line 11, in <module>
    train['MSZoning'] = train_OHE.fit_transform(train['MSZoning']).toarray()
  File "C:\Users\security\Anaconda3\lib\site-packages\sklearn\preprocessing\_encoders.py", line 511, in fit_transform
    self._handle_deprecations(X)
  File "C:\Users\security\Anaconda3\lib\site-packages\sklearn\preprocessing\_encoders.py", line 394, in _handle_deprecations
    n_features = X.shape[1]
IndexError: tuple index out of range

I did read through some Medium posts on this but they didn't exactly relate to what I was trying to do with my dataset as they were working with dummy data with a couple of categorical columns.  What I want to know is, how do I make use of one-hot-encoding after the (attempted) step?

First of all, I noticed you have loaded the same dataframe for both train and test. Change the code like this:

import numpy as np
import pandas as pd

train = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/train.csv")
test = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/test.csv")

At this point, one-hot encode each variable you want with pandas' get_dummies() function:

# Onhe-hot encode a given variable
OHE_MSZoning = pd.get_dummies(train['MSZoning'])

It will be returned as a pandas dataframe. In my Jupyter Notebook it looks like this:

OHE_MSZoning.head()

You can repeat the same command for all the variables you want to one-hot encode.

Hope this helps, otherwise let me know.


What is the correct way to apply KNN to a time-series using a rolling window? I have a time-series. The index is weekly dates and the values are a certain indicator that I made. I think I understand how to apply KNN in this situation but I'm not sure how exactly to do it.

This is what I currently have:

1. Set lookback period to 200 rows (which is 200 weeks)
2. Set the KNN value to 10 Nearest Neighbors
3. Start with the 201 st row
4. Get the previous 200 days
5. Do Knn.fit(x = prev_200_row, y = profit_after_each_row, neighbors = 10)
6. Do Knn.predict(current_row)
7. Save the prediction to a list
8. Go to the next row and repeat steps 4, 5, 6, 7, 8 until no more rows.

Is this is a correct use of the process? Eventually, I would like to compare different values for K (10 nearest neighbors or maybe 50 nearest neighbors).
I am going to build on this so I want to make sure that I am starting correctly. I am wondering if it is correct that the Knn.fit is being called after every row but it seems correct to me.

I never used KNN on time series. I didn't know it was possible before reading your question. But by googling it found this tutorial that feel pretty clear.

And if understand you correctly what you do with KNeighborsRegressor, by repeating 4 - 5 - 6 you will based the later predictions on what you predicted for the previous row.

If this is not intended you can just predict every row following the 200 used for training.

Hope it helps :)


Markov Chains for sequential data I am new to Markov chains and HMM and I am looking for help in developing a program (in python) that predicts the next state based on 20 previous states (lets say 20 states in last 20 months).  I have a sequential dataset with 50 customers i.e. the rows contain sequence of 20 states for each of the 50 customers (dataset has 50 rows and 20 columns excluding the headers). I am trying to determine the next state using markov chains and all the literature in the web is focused around examples of text strings. I am looking something specific to the kind of example I have.
Can somebody please help me come up with the initial probability matrix and then consider the 20 states to predict the next state?

If you know what the state history is, you don't need a 'hidden' Markov model, you just need a Markov model (or some other mechanism).  The 'hidden' part implies a distinction between some sequence of unobservable states, and some observations that are related to them.  In your case, you say you have observed the past states for each customer, so you don't necessarily need to infer anything 'hidden'.

The simplest way to proceed in your case would be to calculate a transition matrix, i.e. probability of state given previous state.  That's a very simple model but it might do what you want.  To do this, just look at all state pairs, and count to get p(s2 | s1) = p(s1 & s2)/p(s1).  This is equivalent to a 1-gram model that you've probably read about. Each state is akin to a word.

You could also make a more complex model, like a 2-gram model or even an RNN.  Honestly, since you have a fixed amount of history, you can just throw your data into an scikit-learn model or xgboost or something, where each customer's history is the vector of predictors and the next state is the outcome.  It won't know the sequential dependencies, but you are essentially indexing the past states by time, so it may work pretty well.

If you need more clarification about part of this, just ask.


Gradient descent with vector-valued loss My understanding of gradient descent as an optimizer for a neural network is as follows:

Let $w$ be a vector of weights encoding a configuration of the network, and $l : w \mapsto \textrm{network loss}$ a function which calculates the loss over some batch of data given this configuration. Then, the weight update is $-\alpha \nabla l(w)$, where $\alpha$ is the learning rate, since the vector $\nabla l(w)$ represents the direction of greatest increase in the neighborhood around $w$.

I see clearly that this works for $l(w) \in \mathbb{R}$, but am wondering how it generalizes to vector-valued loss functions, i.e. $l(w) \in \mathbb{R}^n$ for $n > 1$.

I see clearly that this works for $l(w) \in \mathbb{R}$, but am wondering how it generalizes to vector-valued loss functions, i.e. $l(w) \in \mathbb{R}^n$ for $n > 1$.

Generally in neural network optimisers it does not*, because it is not possible to define what optimising a multi-value function means whilst keeping the values separate. If you have a multi-valued loss function, you will need to reduce it to a single value in order to optimise.

When a neural network has multiple outputs, then typically the loss function that is optimised is a (possibly weighted) sum of the individual loss functions calculated from each prediction/ground truth pair in the output vector.

If your loss function is naturally a vector, then you must choose some reduction of it to scalar value e.g. you can minimise the magnitude or maximise some dot-product of a vector, but you cannot "minimise a vector".

* There is a useful definition of multi-objective optimisation, which effectively finds multiple sets of parameters that cannot be improved upon (for a very specific definition of optimality called Pareto optimality). I do not think it is commonly used in neural network frameworks such as TensorFlow. Instead I suspect that passing a vector loss function into TensorFlow optimiser will cause it to optimise a simple sum of vector components.


How to transform stock data for LSTM-based neural network I am trying to classify stock returns using an LSTM-based neural network.

I would like to use closing price and volume as features (see below), but am unsure of whether I need to transform these (e.g., by differencing) before feeding them into the network?

If anybody has done this sort of thing before and could give me some advice, or could refer me to any papers, I'd be very grateful.

Thanks,

Jack

Yes, you should calculate the log return for each stock, then feed them to the neural network.

I did a similar project before, but with only the close price, not the volume. I follow the method in this paper: Deep learning with long short-term memory networks for financial market predictions

Here's the code: https://github.com/tqa236/LSTM_algo_trading

Please take it with a grain of salt, the code quality is not very high. Also, the result is not quite impressive, both in my project and the original paper for these several recent years. We reach the same conclusion that it's really difficult to use vanilla LSTM to predict the stock market nowadays.


How do I perform Leave One Out Cross Validation For Top n Recommendation Sytems? I am new in making recommendation systems . I am using the  surpriselib library to evaluate my recommendations. All the Accuracy Metrics are well supported in this library. But I also want to compute the Hit Rate of my top n recommender system.

I know the formula for hit rate is:

(no items users have already purchased)/(no of users)

But this does not makes sense to me because to train and test the user vs item ratings I have only those users in my dataset which have already rated all the items that are in the dataset. (**After all this is what enables me to measure accuracy of my rating predictions **)

This means I will get a HIT RATE OF 100% everytime !!!

Is my theory wrong if not what can I do to properly measure hit rate
I know there is one method called Leave one out cross validation
which can help me in this case . If that's the answer how do they work with top n recommender systems?

Here is my procedure for computation of the "Hit Rate with leave-one-out cross validation":

leave only one actual user-item interaction (this can be rating, purchase or whatever you are dealing with in your data) out of the train dataset (this gives you a total of U "unseen" interactions provided that you have at least one interaction per user);
Generate top n recommendations for all user with whatever recommendation system you are using;
if a user actually interacted with the recommended item (is the left-out" item was among the top n recommendations for that particular user) => that's a hit
Hit rate = sum of all the hits across the users (this can be at most U since you can have at most 1 hit per user) DIVIDED BY the number of users (U).