Can I use unsupervised learning followed by supervised learning? I have a question about classifying documents using supervised learning and unsupervised learning.

For example: - I have a bunch of documents talking about football. As we know, football has a different meaning in UK, USA, and Australia. Therefore, it is difficult to classify these documents to three different categorizations (which are soccer, American football and Australian football).

My approach tries to use cosine similarity terms based on unsupervised learning. After we use the cluster learning, we are able to create a number of clusters based on cosine similarity, where each cluster will contain similar documents terms. After we create the clusters, we can use a semantic feature to identify these clusters depending on a supervised model like SVM to make accurate categorizations.

My goal is to create more accurate categorizations, because if I would like to test a new document, I would like know if this document can be related to these categorizations or not.

You can definitely try to first cluster your data, and then try to see if the cluster information helps your classification task.

For example if your data looked like this (in 1D):

AA A AA A A      BBB B B B BB BB BB      AA AA A A AAA

then it may be reasonable to run a clustering algorithm on each class, to obtain two different kinds of A, and learn two separate classifiers for A1 and A2, and just drop the cluster distinction for the final output.

Other common unsupervised techniques used include PCA.

As for your football example, the problem is that the unsupervised algorithm does not know what it should be looking for. Instead of learning to separate american football and soccer, it may just as well decide to cluster on international vs. national games. Or Europe vs. U.S.; which may look like it learned about american football and soccer at first, but it put american soccer into the same cluster as american football, and american football teams in Europe into the Europe cluster... because it does not have guidance on what structure you are interested in; and the continents are a valid structure, too!

So usually, I would not blindly assume that unsupervised techniques yield a distrinction that matches your desired result. They can yield any kind of structure, and you will want to carefully inspect what they found before using it. If you use it blindly, make sure you spend enough time on evaluation (e.g. if the clustering improves your classifier performance, then it probably worked as intended ...)


Simple example of Parzen window (kernel density estimation) I am confused about the Parzen Window question.

Suppose we have two training data points located at 0.5 and 0.7, and we use 0.3 as its rectangle window width. How do we estimate its probability density?

According to the definition, the probability density is $\frac{k/n}{V}$, where $k$ is the number of patterns, $n$ is the total number of points, and $V$ is the window region.

Therefore, is the density for this question $(1/2)/1$?

Then what if we use a triangle instead of a rectangle?

In kernel density estimation, rectangle, triangle, or Gaussian  kernels assign weight to positions around query point $x$. For rectangle and square, the weight is steady, for triangle, weight drops linearly with distance, and for Gaussian, weight drops exponentially with distance. Here is an image for square, rectangle, and triangle kernels (one dimensional).

Here is the formula for kernel density estimation in one dimension:

$$\hat{f}_{h}(x) =\frac{1}{nh}\sum_{i=1}^{n}K\left(\frac{x-x_i}{h}\right)=\frac{1}{n}\sum_{i=1}^{n}K_h\left(x-x_i\right)$$

where scaled kernel $K_h$ is defined as:
$$K_h(x-x_i) := \frac{1}{h}K(\frac{x-x_i}{h})$$

Example

We have two training points located at $0.5$, and $0.7$ in one dimension.

If we assume query point is $x=0.5$, and $h$ is 0.3 (for rectangle and triangle, the scaled width is 0.6 from -0.3 to 0.3), for rectangle kernel we have:
$$\begin{align*}
\hat{f}_{0.3}(0.5) &=\frac{1}{2*0.3}\left(rect \left(\frac{0.5-\color{blue}{0.5}}{0.3} \right)+rect \left(\frac{0.5-\color{blue}{0.7}}{0.3} \right)\right) \\
&= \frac{1}{0.6}(0.5+0.5) = 1.67
\end{align*}$$
and for triangle kernel we have:
$$\begin{align*}
\hat{f}_{0.3}(0.5) &=\frac{1}{2*0.3}\left(tri \left(\frac{0.5-\color{blue}{0.5}}{0.3} \right)+tri \left(\frac{0.5-\color{blue}{0.7}}{0.3} \right)\right) \\
& = \frac{1}{0.6}(1+0.33) = 2.22
\end{align*}$$

For square kernel $sqr$, kernel outputs 1 for each point inside the hyper-cube, and $h$ would be the width of scaled hyper-cube (from $-h/2$ to $h/2$). For the square kernel, estimation in $d$ dimension is:
$$\hat{f}_{h}(x) =\frac{1}{nV}\sum_{i=1}^{n}sqr\left(\frac{x-x_i}{h}\right)=\frac{1}{nV}k=\frac{\frac{k}{n}}{V}$$

where $k$ is the number of points inside a hyper-cube centered at $x$ with width $h$, and $V=h^d$ is the size of hyper-cube in $d$ dimensions (length of a segment in one dimension).

The previous example for $h=0.3$ would be:
$$\begin{align*}
\hat{f}_{0.3}(0.5) &=\frac{1}{2*0.3}\left(sqr \left(\frac{0.5-\color{blue}{0.5}}{0.3} \right)+sqr \left(\frac{0.5-\color{blue}{0.7}}{0.3} \right)\right) \\
& = \frac{1}{0.6}(1+0) = \frac{\frac{1}{2}}{0.3} = 1.67
\end{align*}$$
Note that these are estimations for density not probability, thus, they are allowed to be larger than one (imagine a rectangle with width 0.5 and height 2 as pdf of a continuous random variable).


How to find from which row and column the value belong? Suppose I created the below data frame

data = {'Height_1': [4.3,6.7,5.4,6.2],
        'Height_2': [5.1, 6.9, 5.1, 5.2],
        'Height_3': [4.9,6.2,6.5,6.4]}

df = pd.DataFrame(data)

Suppose someone comes and asks me

Find the row and column of height 6.9 ?
Find in how many rows and columns height 6.2 is present?

Please help me with what will be the code for this?

You can use pandas .isin() function to compare all the values of your dataframe to a given value:

In [10]: df[df.isin([6.9])]
Out[10]:
   Height_1  Height_2  Height_3
0       NaN       NaN       NaN
1       NaN       6.9       NaN
2       NaN       NaN       NaN
3       NaN       NaN       NaN

Now, if you want to get rows and column directly from it use .stack() on it. So, it will be like:

In [11]: df[df.isin([6.9])].stack()
Out[11]:
1  Height_2    6.9
dtype: float64

The output is a series. This will work in case of multiple matches too where the output will be a dataframe.


what is the best to do with highly correlated features? In my data set 2 features C1 and C2 are highly correlated. I did the following steps. Could you please let me know if it is correct and make sense? do you have a better approach?
First I used the linear model to find the  fitted line:
C1=a*C2+b
from sklearn import linear_model
reg=linear_model.LinearRegression()
y_reg=data1['C1']
x_reg=data1['C2']
reg.fit(x_reg2,y_reg2)
a=reg.coef_
b=reg.intercept_
print(a,b)

After finding a and b I removed C1 and C2 from the dataset and added a new Variable: new=a*C1+b
My next question is how I can understand if this line is good?

No I don't think this is good: this means that you replace both variables with only an estimated value of C2 based on C1 (or of C1 based on C2, it's not totally clear to me).
Assuming you really want to avoid having two highly correlated features, you could instead:

just arbitrarily pick one of them (which is similar to what you're doing but "cleaner")
simply use the average of the two values (which might be similar to what you intended to do?)

Over a whole set of features you could use more advanced methods such as feature extraction.
But it's important to make sure you actually need to do this: correlated features are a problem for some ML methods, but not all of them. For example having correlated features with Decision Trees wouldn't cause any problem. In any case it's worth experimenting with/without the feature in order to observe what actually happens with your data.


Predictive modeling on big data set that can't fit into memory I am trying to build a Decision-Tree model on top of a dataset that is about 10G in size on my local computer. However, I only have 8G memory. What I am doing now is just random sampling certain subset of the data, trying different model parameters and checking the prediction performance with the scikit-learn package. Still I need a final model based on the model parameters from those random sampling results, but obviously I cannot build a model on top of the full dataset. What can I do with this problem? I am a newbie of machine learning, any suggestions are welcome. Thanks!

Whether you're using this for a hobby or for your job, I would recommend EMR on Amazon Web Services. The name is anachronistic (it's no longer just for map reduce), but EMR enables you to quickly stand up an Apache Spark cluster of several machines. Spark comes with machine learning libraries for building trees, and can scale to however many machines you need (to the point that you no longer face memory constraints). EMR is cheap because you're charged hourly and don't have to buy your own hardware. Apache Spark is great for your resume, and it's also a practical skill because it's generally the first tool most data scientists turn to when working with data sets that are too large for a single machine.


Spacy Text classification (Binary Classification) I have a dataset of two folders. One of them contains the documents(text, pdfs) related to personal information (like name,email,address etc), the  other contains non-personal information.

I have to train a model using Spacy, based on these two folders. So, when we predict a given document, it should predict among these two folders.

I have tried writing many codes taking reference from Github, but nothing seem to be worked.

So, can anyone  give me a code sample to train a model based on the information given above and predict ?

I have done some hands on, on the below code

import spacy
from spacy import displacy
from spacy.util import minibatch, compounding

train_data = [("This has names, emails, addresses ", {'cats': {'POSITIVE': 1}} ), ("This has games, food, etc", {'cats': {'POSITIVE': 0}})]

nlp = spacy.load('en_core_web_sm')

if 'textcat' not in nlp.pipe_names:
    textcat = nlp.create_pipe("textcat")
    nlp.add_pipe(textcat, last=True)
else:
    textcat = nlp.get_pipe("textcat")

textcat.add_label('POSITIVE')

other_pipes = [pipe for pipe in nlp.pipe_names if pipe != 'textcat']

n_iter = 1

with nlp.disable_pipes(*other_pipes):
    optimizer = nlp.begin_training()
    print("Training model...")
    for i in range(n_iter):
        losses = {}
        batches = minibatch(train_data, size=compounding(4,32,1.001))
        for batch in batches:
            texts, annotations = zip(*batch)
            nlp.update(texts, annotations, sgd=optimizer,
                      drop=0.2, losses=losses)

Here in the above code, I have trained the model using two simple sentences. I need to train on two folders, as mentioned in the question.
This code just says model has trained.
And also how can i save this model and test it for documents to predict ??

You're very close to having a working script. The textcat training example in the spacy repository shows how to save the model, reload it, and run it on a new text: https://github.com/explosion/spaCy/blob/master/examples/training/train_textcat.py

From around line 103:

    # test the trained model
    test_text = "This movie sucked"
    doc = nlp(test_text)
    print(test_text, doc.cats)

    if output_dir is not None:
        with nlp.use_params(optimizer.averages):
            nlp.to_disk(output_dir)
        print("Saved model to", output_dir)

        # test the saved model
        print("Loading from", output_dir)
        nlp2 = spacy.load(output_dir)
        doc2 = nlp2(test_text)
        print(test_text, doc2.cats)
```


First order vs Second order online machine learning algorithms In this git repo the online learning algorithms are classified as first order and second order. I tried searching what it means, yet I'm unable to understand the different between such first oder and second order online learning algorithms.

What is the difference between first order and second order online learning?

First order and second order algorithms are algorithms which use respectively at least first derivative / second derivative.

This post explains well the definition.


Fill the null values in the dataframe with condition traindf[traindf['Gender'] == 'female']['Age'].fillna(value=femage,inplace=True)
I've tried to update the null values in the age column in the dataframe with the mean values.Here I tried to replace the null values in the age column of female gender with the female mean age.But the column doesn't get updated.why?

traindf.loc[traindf['Gender'] == 'female','Age'] = traindf.loc[traindf['Gender'] == 'female','Age'].fillna(value=femage)


How to predict an outcome within a specific time window? I have a dataset which has around 10K records.

My objective is to predict whether the customer will churn or not. Binary classification problem with each class representing around 55:45 proportion and 20 features.

I understand when it's just about prediction, I can apply some binary classification algorithm and find out whether the customer churns or not

But how do I incorporate the objective of finding whether the customer will churn in 30 days or not?

Another example is find whether patient will be dead within 30 days from the date of discharge. I have his date of discharge along with other features like Blood pressure, Cholesterol etc.

Rather than just predicting whether he will be dead or not anytime in future, I would like to restrict it to 30 days from date of discharge.

Hope I gave the details to help you understand the question better.

Beeing X in the future and beeing X in specific time in the future is just a subset of the first one.

So what one really needs to do is just determine the probabilities (or parameters that give us these probabilities)

P(X|t>30)

Where you can model t, also as your feature. So just fit a model on this data, where you have mutliclassification of:

dead within 30 days
dead after 30 days
alive after 30 days


How Transformer is Bidirectional - Machine Learning Asking question in datascience forum, as this forum seems well suited for data science related questions: https://stackoverflow.com/questions/55158554/how-transformer-is-bidirectional-machine-learning/55158766?noredirect=1#comment97066160_55158766

I am coming from Google BERT context (Bidirectional Encoder representations from Transformers). I have gone through architecture and codes. People say this is bidirectional by nature. To make it unidirectional attention some mask is to be applied.

Basically a transformer takes key, values and queries as input; uses encoder decoder architecture; and applies attention to these keys, queries and values. What I understood is we need to pass tokens explicitly rather than transformer understanding this by nature.

Can someone please explain what makes transformer bidirectional by nature

Answer received so far:
1. People confirmed that Transformer has Bidirectional nature, rather than an external code making it bidirectional.
2.
My doubt: We are passing Q K V embeddings to transformer, to which it applies N layers of self attention using ScaledDotMatrix attention. Same thing can be done by unidirection approach as well. May I know what part I am missing in my understanding. If someone can point to code where it is getting bidirectional, it would be a great help.

It is the encoder part of the Transformer model that is bidirectional in nature, not the whole model.
The full Transformer model has two parts: encoder and decoder. This encoder-decoder model is used for sequence-to-sequence tasks, like machine translation.
There are other tasks, however, that do not need the full model, but only one of its parts. For instance, for causal language modeling (e.g. GPT-2) we need the decoder. For masked language modeling (e.g. BERT), we need the encoder.
The decoder is designed so that each predicted token can only depend on the previous tokens. This is achieved with self-attention masking, and it is what makes the decoder unidirectional.
The encoder does not have self-attention masking. Therefore is designed not to have any dependency limitation: the token representation obtained at one position depends on all the tokens in the input. This is what makes the Transformer encoder bidirectional.


Managing NaN in target variables (testing) Please can someone advise me on how to handle NaN in my target variables set?

I've tried a variety of things but none is working. Here's what I've tried:

Imputing zeros (0) in Y_test
Replacing NaN with whitespace in the Y_test
Emptying the Y_test

The dataset came already separated as training and test set. When I apply my logit classifier or any other classifier for that matter, it keeps returning a TypeError.

  TypeError: '<' not supported between instances of 'int' and 'str'

Please help!

Well, you can't do supervised learning on unlabeled data. Either you discard those records or you have to go semi-supervised way.


Can I load my own weights? Full code source:
#Download COCO pre-trained weights
!wget --quiet https://github.com/matterport/Mask_RCNN/releases/download/v2.0/mask_rcnn_coco.h5
!ls -lh mask_rcnn_coco.h5

COCO_WEIGHTS_PATH = "mask_rcnn_coco.h5"

model.load_weights(COCO_MODEL_PATH, by_name=True,
                       exclude=["mrcnn_class_logits", "mrcnn_bbox_fc",
                                "mrcnn_bbox", "mrcnn_mask"])
elif init_with == "last":
    # Load the last model you trained and continue training
    model.load_weights(model.find_last()[1], by_name=True)

Can I load my own "*.h5" file?
For example: I interrupted my kernel after 5 epochs. Can I load my last epoch?
Can You explain it for me?
It will be continue a process learning?

Can I load my own "*.h5" file?

Yes, you can load your own .h5 file. But to load weights into a model, you need to have a model architecture defined for it. If the weight dimensions or layer sizes does not match, it will throw an error, of course.

But instead of saving weights, if you save the model by using model.save_model, it does have architecture of the model saved allowing you to re-create the model, weights of the model, training configuration (loss, optimizer), state of the optimizer, allowing you to resume training. But this does not load hyperparameters that are present for that epoch like decay in learning rate etc. You have to write a custom callback for this.

  Can I load my last epoch?

Yes, if specified. Keras does not automatically save the model for every epoch or anything until you ask it to. So if you had interrupted something, you cant resume it from there. You have to write a custom callback to make it happen. For details on what are callbacks. For details on how to save the model after every epoch, check out this answer.


Amount of data needed for deep learning vs support vector machine I often read about the fact, that the amount of data to train and get a generalizing model for a deep learning algorithm is much higher in comparison, e.g. to a support vector machine. It makes sense, because of the huge amount of parameters in a deep learning approach, which potentially leads to overfitting.

However: Are there any systematic studies on this? Do deep learning approaches really need more data?

Best regards,
Gesetzt

When it comes to how much data model M needs to accurately model problem P, there are 3 factors to consider:

What is the dimensionality of model M?

The power of neural networks is that they can model functions up to to a high number of dimensions, which means that they can, therefore, model any function with fewer dimensions. Indeed, a quadratic function can always be estimated by a cubic function, whereas the converse isn't true.

Support vector machines are an interesting type of models because the kernel trick allows them to model arbitrarily high-dimensional relationships. This allows you to eventually tweak your kernel based on the amount of data at your disposal.

What is the dimensionality of the space represented by the data at your disposal?

Not all data are created equal, and the amount of data is only a proxy to estimate the dimensionality of the problem space. Always consider the possibility that your data might be an unrepresentative sample of your space. The amount and type of noise present in the data is also a huge factor. A factor which gets mitigated the more data you have.

What is the true dimensionality of problem P?

Easily the most important factor in terms of how much data you need, and sadly, the only factor that we can never truly know. In a perfect world, the dimensionality of the model should be greater or equal to the dimensionality of the problem. Equal being, of course, the ideal scenario.

Do deep learning approaches really need more data?

Theoretically, the need the amount of data that will appropriately match the true dimensionality of the problem. Because they have a higher number of parameters, more data generally help reduce the chance of overfitting. However, as previously stated, a model with a dimensionality much higher than the problem's is ill-suited, to begin with.

The power behind neural networks is that they have such a large solution space that the chance to converge in a satisfying spot is quite high.


Tips to improve Linear Regression model I have just run a Linear regression model on the Dataset having 7 independent variable and 1 target variable. Below is the R squared and MSE values.

Mean squared error for training set : 36530921.0123
$R^2$ value for training set : 0.7477

Can anybody please give me some tips to increase the efficiency of this model.

Edit: I have just implemented the same problem using Linear regression with Normalization of the features.
I got the below output:
Mean squared error for training set : 5.468490570335696e-10
R2 value for training set : 0.9275088299658416
Mean squared error for training set : 4.111793316375822e-10
R2 value for training set : 0.9342888671422529

So can we consider normalizing the dataset to get better accuracy ?

You can build more complex models to try to capture the remaining variance. Here are several options:

Add interaction terms to model how two or more
independent variables together impact the target variable
Add polynomial terms to model the nonlinear relationship between an independent variable and the target variable
Add spines to approximate piecewise linear models
Fit isotonic regression to remove any assumption of the target function form
Fit non-parametric models, such as MARS


Look for previous date in dataframe that has certain column category in R I have the following data frame:

      Date.POSIXct       Date      WeekDay DayCategory Hour Holidays   value
1 2018-05-01 00:00:00 2018-05-01      MA    MA-MI-JU    0        0      30
2 2018-05-01 01:00:00 2018-05-01      MA    MA-MI-JU    1        0      80
3 2018-05-01 02:00:00 2018-05-01      MA    MA-MI-JU    2        0      42
4 2018-05-01 03:00:00 2018-05-01      MA    MA-MI-JU    3        0      90
5 2018-05-01 04:00:00 2018-05-01      MA    MA-MI-JU    4        0      95
6 2018-05-01 05:00:00 2018-05-01      MA    MA-MI-JU    5        0       5

DayCategory groups days of the week in the following way: Mondays goes to LU DayCategory. Tuesday, Wednesday and Thursdays go to MA-MI-JU DayCategory.
Friday goes to VI, Saturdays to SA and Sundays to DO Categories respectively.

I would like to find the value for the same hour in the previous day (Date) with the same DayCategory, while Holidays remains within the same group (e.g. if one instance has holiday 0 but previous day with same DayCategory has 1, we should lookv for the previous one, etc.)

As an intermediate step and to understand the process I would like to add a column PreviousDaySameDayCategory with the Date of the previous day that has the same DayCategory that the corresponding row. Some times it will be just the same date minus seven days ("LU","VI","SA","DO") but other days it will be just one day.

Reproducible data:

library(lubridate)
Date.POSIXct <- seq(as.POSIXct("2018-05-01"), as.POSIXct("2018-05-31"), "hour")
mydf <- as.data.frame(Date.POSIXct)
mydf$Date <- as.Date(substr(as.character(mydf$Date.POSIXct),1,10))
mydf$WeekDay <- substr(toupper((weekdays(mydf$Date))),1,2)
mydf$DayCategory <-as.factor(ifelse(mydf$WeekDay == "MA" | mydf$WeekDay == "MI" | mydf$WeekDay == "JU",
                                    "MA-MI-JU", mydf$WeekDay))
mydf$Hour <- hour(mydf$Date.POSIXct)
mydf$Holidays <- c(rep(0, 24*7),rep(1, 24*7), rep(0, 24*16+1))
set.seed(123)
mydf$myvalue <- sample.int(101,size=nrow(mydf),replace=TRUE)

I have manually started the first days and craeted a vector of how the solution should look like:

  a <- rep(NA, 24)
  b <- mydf$value[1:24]
  c <- mydf$value[25:48]
  d <- rep(NA, 24)
  e <- rep(NA,24)
  f <- rep(NA,24)
  g <- rep(NA,24)
  h <- rep(NA,24)
  i <- mydf$value[169:192]
  solution <- c(a,b,c,d,e,f,g,h,i)
  solution

I would appreciate any hint in the thinking process to solve this kind of problems that I face with relative frequency.

I also published this question in Stack Over Flow, I guess this kind of question fits more in this community than in StackOverflow, is this correct? I would like to delete the one that less fit the commnunity.

If I understand you correctly, you want to retrieve the date of the previous same day category, same holiday class, of the same hour.

If it is the case, perhaps you can try:

1) Segment rows within the same hour, day category, and holiday group into the same cluster (or filter, etc.)

2) For each cluster, Rank the Date (first day in the group is 0, second = 1, ... nth day = n - 1, etc.) from the lowest, etc.

3) For each Rank r, assign the PreviousDaySameDayCategory by looking up the Date for row r - 1 for r - 1 >= 0 within the same cluster

It is rather straightforward to achieve with dplyr. Please let me know if the following makes sense to you.

(For some reasons, I am not able to run your solution)

library(dplyr)
rankedDf <- mydf %>%
  group_by(DayCategory, Hour, Holidays) %>%
  arrange(DayCategory, Hour, Holidays, Date) %>%
  mutate(rowRank = order(Date), previousRowRank = order(Date) - 1) %>%
  left_join(., ., by = c("previousRowRank" = "rowRank", "DayCategory", "Hour", "Holidays")) %>%
  select(
    Date.POSIXct = Date.POSIXct.x,
    Date = Date.x,
    WeekDay = WeekDay.x,
    DayCategory,
    Hour,
    Holidays,
    myvalue = myvalue.x,
    PreviousDaySameDayCategory = Date.y,
    PreviousValueSameDayCategory = myvalue.y
  )
print.data.frame(rankedDf, nrow = 500)

Note: I also include DayCategory, etc. in the arrange so that it's easier to understand how it works; the code works just fine with only Date as the parameter.


Number of internal nodes in a Decision Tree The number of internal nodes in a decision tree is not equal to the size (n) of the input feature vector? It is always less than n or the upper bound on the number of decision nodes will be n? How to draw a relation?

Building a decision tree is a process where the algorithm picks the first feature to split on $i_1$ from the set of features $n$ that it can split on $i_1={1,...,n}$.

After splitting, the algorithm has to decide whether to stop building the tree or split on another feature $i_2={1,...,n} \ $ AND $i_2 \ne i_1$, and the process repeats where the algorithm keeps deciding whether to stop or split on a new feature $i_3={1,...,n} \ $ AND $i_3 \ne i_2 \ne i_1$. This keeps going until the algorithm decided to stop building the tree or it has split on all $n$ features and there exists no feature $i_{n+1}={1,...,n}$ that can satisfy the condition $i_{n+1} \ne i_n \ne ... \ne i_2 \ne i_1$. Thus there are no more features to split on.

Thus the maximum number of nodes has to be less than or equal to the size ($n$) of the input feature.


Why i am getting t statistics different import scipy.stats as st
st.t.ppf(0.95, df=9)

Output: 1.8331129326536335

As per this, I should get 2.262 with dof = 9.

t.ppf is calculating a 1-tail inverse cdf. It looks like you're trying to look up the t-value for p=.95, but the value you are referring to in the table is a 2-tailed value of .95, meaning the one tailed value is 1 - (1-.95)/2 = .975. So

In [104]: st.t.ppf(0.975, df=9)
Out[104]: 2.2621571627409915


More output neurons than labels? When we train a neural network model for a classification problem, we usually have a dense output layer of size equal to the number of labels we have.

If the layer size was greater, the model can still be trained and have output

How can we interpret such output? are there any applications for this?

The output layer is usually the same size as the last dense layer because we apply a loss function to train the model by comparing the last layer to what the output should be. If your output layer was bigger, it's less intuitive what your loss function should be, but the interpretation of your output would likely come from how you define this loss.


Why is not AUC or other metrics used for splitting nodes in decision trees? There are common ways to split a tree in decision trees and all their variants:

Gini Index
Entropy
Misclassification

Why there is not a method which uses directly AUC or accuracy (or whichever the modeler need) to split the nodes.

Is it because of common use, or there is a mathematical explanation for it?

On accuracy:
Why we use information gain over accuracy as splitting criterion in decision tree?

AUC has been explored; it seems to work well, but is slower:
https://www.semanticscholar.org/paper/Learning-Decision-Trees-Using-the-Area-Under-the-Ferri-Flach/46e40f487e555277033f188778d6c5c05df8daa4
http://proceedings.mlr.press/v7/doetsch09.html


Time series forecasting dilemma. Could feature engineering overcome time dependency? I keep reading articles about time series forecasting.

They all start from the same assumption: time series forecasting can't be treated as a regression/classification problem. It is time dependent, which means our target y at time t depends on what the value y was at t-1.

The motivation often comes with some example data showing some trend/seasonality. Also, other supporting points are:

The data distribution (mean, variance) changes over time.
Traditional train/test splits won't make sense since what is the point of forecasting January data with September data?

Fair enough. But let me try to point out this example. Let's say we have a simple timestamp, variable dataset and we are trying to predict the value at t+1

  |  timestamp |  value |
  |  01/01/2019|   10   |
  |  01/02/2019|   12   |
  ...
  |  31/12/2019|   ???   |

What we know is that there is no trend it's very weekly-cyclic instead, which means the value at t will probably depend on its value at t-7 days.
We also know that depending on whether it's a day during the week or during the weekend, data will change accordingly.

What prevents me to use some basic feature engineering and transform the example data as follow?

  |  timestamp | value_at_t_minus_7 | day_of_week |  value |
  |  01/01/2019|   11   |   02   |   10   |
  |  01/02/2019|   12   |   03   |   12   |
  ...
  |  31/12/2019|   10   |   02   |   ???   |

It is not time dependant from a formal perspective but the correlation between its lagged values and the information about the day of the week should take me where I want to, being able to use now classic and flexible methods such as Random Forest, XGB and also splitting train and test (of course leaving the holdout set for validation) to get a good sense of how my model is performing.

Could anyone offer their input supporting with some proper motivation?

Thanks!

They all start from the same assumption: time series forecasting can't be treated as a regression/classification problem. It is time dependent, which means our target y at time t depends on what the value y was at t-1.

Time series forecasting must take into account time dependency, but it doesn't have to be the only source of information. Many complex forecasting tasks take into account external variables and time dependency altogether.

What prevents me to use some basic feature engineering and transform
the example data as follow?
| timestamp | value_at_t_minus_7 | day_of_week | value |

Nothing prevents you from doing that. Moreover, that value_at_t_minus_7 is time dependency. day_of_week can also be thought as a simplification, or summarization of time dependency information - because studying seasonality we concluded that x can be explained with the value of one week ago. It's another way to make time explain the output.
When working with regressions (such as ARIMA or Tree-based models) you have to manually specify what is the shape of time dependency. Therefore the method you proposed is perfectly "legal" and actually very common. Good luck with your task.


Algorithm or function or model that encourages clustered classfication? I have a soft classfication problem, i.e., the correct label for a certain instance is not just one class with 100% probability, but rather bunch of classes with probabilities that sum up to one.

What I know as apriori information (I know it because I understand the underlying physical phenomenon) that the classes are close together, like a cluster. E.g, class one and five and eight always come together cause they're adjacent (PS: I can create adjacency matrix if required).

What do I want?

I want some way to tell the NN about this fact. Currently it is vanilla classfication neural net. Any suggestions or readings or guidannce are appreciated.

Why not create another set of classes which corresponds to the groups you want? This works if you care more about groups. Later on you can subsequently classify within groups. Another idea would be to devise a loss function which penalize less within group misclassification and more outside group classification but this involves going into implementation details of your NN. As an idea it would involve writing something like softmax multiplied with misclassification cost matrix


Starting my career as Data Scientist, is Software Engineering experience required? I am an MSc student at the University of Edinburgh, specialized in machine learning and natural language processing. I had some practical courses focused on data mining, and others dealing with machine learning, bayesian statistics and graphical models. My background is a BSc in Computer Science.

I did some software engineering and I learnt the basic concepts, such as design patterns, but I have never been involved in a large software development project. However, I had a data mining project in my MSc. My question is, if I want to go for a career as Data Scientist, should I apply for a graduate data scientist position first, or should I get a position as graduate software engineer first, maybe something related to data science, such as big data infrastructure or machine learning software development?

My concern is that I might need good software engineering skills for data science, and I am not sure if these can be obtained by working as a graduate data scientist directly.

Moreover, at the moment I like Data Mining, but what if I want to change my career to software engineering in the future? It might be difficult if I specialised so much in data science.

I have not been employed yet, so my knowledge is still limited. Any clarification or advice are welcome, as I am about to finish my MSc and I want to start applying for graduate positions in early October.

1) I think that there's no need to question whether your background is adequate for a career in data science. CS degree IMHO is more than enough for data scientist from software engineering point of view. Having said that, theoretical knowledge is not very helpful without matching practical experience, so I would definitely try to enrich my experience through participating in additional school projects, internships or open source projects (maybe ones, focused on data science / machine learning / artificial intelligence).

2) I believe your concern about focusing on data science too early is unfounded, as long as you will be practicing software engineering either as a part of your data science job, or additionally in your spare time.

3) I find the following definition of a data scientist rather accurate and hope it will be helpful in your future career success:

  A data scientist is someone who is better at statistics than any
  software engineer and better at software engineering than any
  statistician.

P.S. Today's enormous number of various resources on data science topics is mind-blowing, but this open source curriculum for learning data science might fill some gaps between your BSc/MSc respective curricula and reality of the data science career (or, at least, provide some direction for further research and maybe answer some of your concerns): http://datasciencemasters.org, or on GitHub: https://github.com/datasciencemasters/go.


Turning R scripts into a "data science tool" I have R scripts that process a client's datasets (make figures, summary tables, etc). I'd like to package these scripts into a standalone program, aka data science tool, that I can give to the client and that requires no interaction with code on their part. Ideally they will open the tool, load data, click run, and be given reports. How can I accomplish this?

You can refer below the answer to the similar question asked.

https://stackoverflow.com/questions/14096520/compile-r-script-into-standalone-exe-file

Alternatively,
You can use R-shiny app for it. You can build application which can be deployed on website also. See below link for the examples of application made by R-shiny.

https://shiny.rstudio.com/gallery/

This way client will not be seeing your code at all and will be working on it just as a tool.


Word2Vec Implementation In word2vec why is the implementation of likelihood function multiplication of probabilities of finding a neighbouring word given a word? I didnt get why the probabilities should be multiplied.Is there a reason/intuition behind it ?

"probabilities of finding a neighboring word given a word"

here you refer to the Skip-Gram architecture, where given the center word you predict the surrounding words.

This extract from these notes might clarify your question.   Note that by assuming the conditional independence the total probability factors into a product.

  "As in CBOW, we need to generate an objective function for us to evaluate the model. A key difference here is that we invoke a Naive Bayes assumption to break out the probabilities. If you have not
  seen this before, then simply put, it is a strong (naive) conditional independence assumption. In other words, given the center word, all output words are completely independent."

Maybe this article can also help, though it is about negative sampling it is a very clear exposition.


How can i test the performance of a model when the test data contains seen and unseen data To test the performance of my model based on some selected features, i try to use unseen and seen data. However, when choosing the accuracy  based on all data, the model is almost overfitting since the size of seen data is larger than the other. When i take only the unseen data. The choosen features do not guarantee that the seen data are properly predicted. Is there any way to surpass this issue.

Thanks

The test of your model should always be done exclusively on unseen data. That is fundamental to asses your model's capacity to generalize and predict observations it has never seen.
If you test it using already seen data, that is like cheating, and overfitting will be 100% guaranteed.

Repeat the test only on unseen data, and check the difference between train and test performances. If differences are big, then you have an overfitting problem. There are several ways to fight overfitting, but they depend on the nature of your model. Could you please provide some information on it?


Skin Detection Classifier I have a small data set containing around 80 images for people, and the corresponding ground truth for skin regions. I want to train a classifier to be able to detect a skin, and use it later on my own generated data, but I'm not sure what is the right way to go. I believe it would be hard to train a CNN due to the small number of images. So can someone suggest a direction?

I suggest to give transfer learning a try for two reasons:

Your dataset is relatively small
Skin, faces and similar human body features have been successfully classified or segmented by existing neural networks

Specifically, you might want to have a look at Skin Lesion Analysis Toward Melanoma Detection. While the tasks were slightly different it did include a segmentation task of Melanomas. Here is an example of what they did:

And this is a snapshot of how some of the models performed:

Source: both images are from the paper linked above

It might be close enough to your task to re-use their networks or some part of it. Once you have found a transfer network, you can vary the degree to which you replace or retrain it`s layers to see what works. Given your relatively small dataset I would not retrain too much but start with the final layers only.

However, depending on how exactly your dataset looks like, there might be other tasks (e.g. from security applications where one needs to detect, classify or mask an object (e.g. a person or a face)) which resemble yours even better. Therefore, before digging into a specific model and apply transfer learning I'd do some more research.


What is the difference between regplot and lmplot in seaborn? Seaborn library in python suggests to use either lmplot or regplot to visualise a regression between two variables. What is the difference between the two plots ? The result I was able to get are slightly different but I have no idea why !

regplot() performs a simple linear regression model fit and plot. lmplot() combines regplot() and FacetGrid.

The FacetGrid class helps in visualizing the distribution of one variable as well as the relationship between multiple variables separately within subsets of your dataset using multiple panels.

lmplot() is more computationally intensive and is intended as a convenient interface to fit regression models across conditional subsets of a dataset.

You can research this comparison for yourself in the seaborn docs for regplot(), lmplot(), and FacetGrid to see which function will meet your needs.


How to add additional information in a neural network other than putting as input data? I am trying to use a neural network to predict final concrete strength. I have used material properties of concrete as input data. Now if I want to add some additional information like early age strength, is there any way to put them in the hidden layer? I mean other than using as input data?

If your inputting data, it will always be an input :). But it is definitely possible to add in certain inptus after the first hidden layer. But what is your reasoning behind this? Why would you want this?

You should just input all your data in the first layer, let the neural network figure out how to use the data effectively itself. The neural network is like a black box, you put something in and something comes out - don't interfere too much by adding seperate inputs to the hidden layers.


Time-series decomposition to a base level and an effect of another feature I've got a time-series data (let's denote it as y) and some feature (let's denote it as x). y is dependent on x, but x is often equal to 0. Even then, y is not 0, so we can assume that there's a base level in y which is independent of x. Additionally, we can observe some seasonality in y. I need to decompose y into base level and an effect of x. And I need some hint about methodology. I have googled and found plenty of methods to decompose time-series data into trend, seasonality and random noise. However, my case is different, because I have an additional feature x and I would like just to extract its effect, and leave trend, seasonality and noise alltogether. What I have in mind can be represented on a plot below, where turquoise area represents base level of y and red area represents an effect of x. What method would allow to make such a split? I would also appreciate any links or materials.

You are describing multivariate time series analysis, modeling the interactions and comovements among a group of time series variables.

You can start with a vector autoregression (VAR) model, one of the most successful, flexible, and easy to use models for
the analysis of multivariate time series. Then work through the decomposition of time series.

You'll have to decide if you want assume if all components time series are stationary and ergodic. Each component will have autocovariances and autocorrelations but there are also cross lead-lag covariances and correlations between all possible pairs of components.


Pytorch LSTM not training So I am currently trying to implement an LSTM on Pytorch, but for some reason the loss is not decreasing. Here is my network:
class MyNN(nn.Module):
    def __init__(self, input_size=3, seq_len=107, pred_len=68, hidden_size=50, num_layers=1, dropout=0.2):
        super().__init__()

        self.pred_len = pred_len

        self.rnn = nn.LSTM(
            input_size=input_size,
            hidden_size=hidden_size,
            num_layers=num_layers,
            dropout=dropout,
            bidirectional=True,
            batch_first=True
        )

        self.linear = nn.Linear(hidden_size*2, 5)

    def forward(self, X):
        lstm_output, (hidden_state, cell_state) = self.rnn(X)

        labels = self.linear(lstm_output[:, :self.pred_len, :])

        return lstm_output, labels

And my training loop
LEARNING_RATE = 1e-2

net = MyNN(num_layers=1, dropout=0)

compute_loss = nn.MSELoss()

optimizer = optim.Adam(net.parameters(), lr=LEARNING_RATE)

all_loss = []
for data in tqdm(list(train_loader)):
    X, y = data

    optimizer.zero_grad()

    lstm_output, output = net(X.float())

    # Computing the loss
    loss = compute_loss(y, output)
    all_loss.append(loss)
    loss.backward()

    optimizer.step()

# Plot
plt.plot(all_loss, marker=".")
plt.xlabel("Epoch")
plt.xlabel("Loss")
plt.show()

And this is what I got

I have been trying to look for what the hell I am doing wrong but I have no idea. Also, before I used a keras LSTM and it worked well on the dataset.
Any help?
Thanks!

You look at loss at every batch. You should average your loss over all batches. When you look at different batches your loss may increase simply because one batch is harder to predict than the other one. That's why it's not really interpretable. So start with that. If the problem persists it's probably exploding gradients. In that case lower your learning rate to 1e-3 or 1e-4 or even less if it continues.


Poor Precision-Recall curve for binary classifier trained on balanced data, with imbalanced test data I have an very imbalanced dataset (9:1), for which I have performed under-sampling and achieved a balanced training set (~130k samples total post balancing).

I am performing classification using RandomForest. My test set comprises of unbalanced data (since this is how it is expected to be in the real world), and my ROC curve is pretty good, with AUC=0.873. However, my P-R curve suffers, with AUC=0.547.

If I balance the test data, I see a better AUC; however, this is not how the ground truth has behaved in the past, so quoting performance on balanced test data doesn't seem too rational.

What am I doing wrong? Should I train on imbalanced data? Or are there any ways where I can improve the P-R curve?

The precision-recall curve and the ROC curve respond very differently to class imbalance. The precision-recall curve is very sensitive to imbalance while the ROC curve can mask the effect of imbalance.

For the TPR (=recall) and FPR you divide by the true number of samples in each class, while for precision you divide by the number of positive predictions.

If you have few positive samples the ratio between false and true positives becomes more sensitive (compared to false positives [few] and all true negatives [many]).

Depending on your use case, this might be a problem (or not) and should be part of your evaluation.


Reinforcement (Q) learning: does it learn while in production? I have a question for which I could not find the answer to it:

While training reinforcement learning (using DQN), I get a model for the best reward for the next action. Now, if I deploy this model (i.e., use this model to make predictions), does it keep learning (i.e., updating the Q values)?

Now, if I deploy this model (i.e., use this model to make predictions), does it keep learning (i.e., updating the Q values)?

If you want it to (and understand how to code it) then yes a reinforcement agent - including a DQN-based one - can do this. This is online learning, and is possible also with many supervised learning techniques.

Because there is risk of the agent learning incorrectly, you may choose to limit online learning, or disable it in production.  It could also happen by mistake if you are not sure how to stop it, and simply deploy your existing agent code from the training scripts into production. So make sure that you, or whoever you get to implement production version of the agent, understands how to control whether learning is still occurring.

You may choose to enable online learning if your initial training was in simulation, and you would like the agent to learn more from real-world interactions. Or you might choose to do so if the problem was non-stationary because the environment description changes over time. Many problems involving interacting with a population of people change over time with demographic changes.

If the environment is stochastic, then in theory you could also switch off exploration - typically by setting $\epsilon = 0$ in an $\epsilon$-greedy behaviour policy. That would allow the agent a (limited) ability to refine its Q value estimates with a relatively low risk. The agent would continue to attempt to act optimally, but may learn enough to decide that different actions were optimal. Note this is still not without risk, because the learning process could fail in some way, leading the agent to predict a wildly incorrect optimal action.

Allowing Q learning to explore non-optimal actions - typically by setting $\epsilon > 0$ in an $\epsilon$-greedy behaviour policy - is more risky in production, because the agent will occasionally pick a non-optimal action in order to refine its estimates of that action. It may result in improved learning in the longer term though, so you might decide to do that if the consequences of non-optimal actions were mild.


What's a good book (or other resource) to learn Imitation Learning? I must learn and apply imitation learning on a robot for my thesis. I'm looking for decent sources of information on this topic.

I found a great book, it's just what I was looking for. In case someone finds it useful:

An Algorithmic Perspective on Imitation Learning: T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbel, & J Peters.


Hi, im currently working for a company that has some inventory control problems First, I was asked by the manager to make a plot showing produced vs received items, its a multistage process so we are only in charge of one of the steps which is designing, I made the plot comparing Received cases vs produced here in my country, produced out of the country, total produced and % of advancement.
Later on in a meeting she asked me to show the graph and table I made to the production supervisors, and she explained that somehow they could predict inventory with it, after the meeting I've been thinking a lot about it, I made another 2 graphs one dependant on the day of the week vs received items, and another dependant on time of the day vs received cases, but now im thinking of doing a regression analysis to be able to predict inventory.
This is not my field, I am a chemist, but I know that if I somehow do a regression analysis and while doing so I succesfully take into account all the variables that affects inventory I should be able to come up with some equation that gives me the answer I need right?
What do you think of my approach? any advice ?

Okay, so since the visuals are done in excel, I think you can start with simple linear regression in excel. Im not an excel expert in this regard, but I think this article will be a good starting point
https://www.ablebits.com/office-addins-blog/2018/08/01/linear-regression-analysis-excel/
So, what you can do:

Get your data and fit linear regression on it.
You need to choose the dependent variable y (this you want to predict) - this will be your inventory variable. All other features (basically other data in your disposal) will act as independent Xi variables.
You fit linear model of form y = bx + a (simplified)
After you have fit the model, you can access its coefficients. One of them will be an Intercept other ones will be coefficients for your input X features.
This coefficient will be X affect on your y (inventory). So, for example, if coefficient is 0.1 for X1, then increasing X1 for 1 will lead to 0.1 increase in y.
At this point you have dependency between input data and inventory change.

Thats a lot more to consider - how good your model is (for this you use metrics, for example MAE and test your model on unseen data), do you have a data-leak, and so on and so on.  You will also need to pre-process your data in viable format, for example extract day/week number from date to try capturing seasonality. Also you should avoid correlation between variable, thats really a lot to think of.
But most importantly, I think you should try to implement it even if you dont know all details and all the rules just to get a fealing if this task fits you, after it you can go on and lear more about all this stuff.
(I would definitely recommend going through this series one day if you are interested https://www.youtube.com/playlist?list=PLOg0ngHtcqbPTlZzRHA2ocQZqB1D_qZ5V )
Hope this helps.


How can generative models be used in machine learning classification applications? My understanding of generative models is that they generate data to match certain statistical properties. Intuitively, I find it hard how generative models can be used for classification purposes. On the other hand, discriminative models being used for classification is self-explanatory.

You will find an explanation on Wikipedia. But let us sum up:

Given your Data $D$ you are interested in target values $Y$, your classifications. Discriminative models are, like your said, the straight forward way of modelling the correspondence of your target given your data, $P(Y|D)$. The generative model, on the other hand, calculates $P(D|Y)$.

Let us consider the example of e-mail spam filtering. We have a set of reference e-mails and a label for each mail, which indicates whether it is spam. If we now, for example, look at Naive Bayes, we can see it utilizes the Bayes Formula to calculate the posterior estimate $P(Y|X) = \frac{P(X|Y)P(X)}{P(Y)}$. Opposed to other Bayesian Inference methods, Naive Bayes does not hold $P(X)$ as a model, but $P(X|Y)$, which can be modelled by our reference e-mail set. In this domain, all e-mails can be modelled as equally distributed, thus making $P(X)$ uninteresting. $P(Y)$ can also be approximated by the reference set (only looking at the labels).

This way, we can generate probabilities for our label $Y$, which is binary in our case. Now we can apply the maximum a posteriori approach: We calculate $P(Y=true|X)$ and $P(Y=false|X)$ and see which of both is more likely. The most likely option will be selected as our classification.


Should I consider feature scaling for all gradient descent based algorithms? In the Coursera course machine learning in the section on Multivariate Linear Regression, Andrew Ng provides the following tips on gradient descent:

Use Feature Scaling to converge quicker

Get feature into an approx -1 < x < 1 range
Mean normalization

Andrew Ng also provides some other tips:

Plot cost vs iterations

to ensure cost decreases on every iteration (try smaller alpha)
to identify if convergence is too slow (try larger alpha)
to identify approximately the number of iterations to converge

Are these tips applicable to all problems involving gradient descent using different machine/deep learning algorithms or just to Multivariate Linear Regression?

About the tips regarding plot cost vs. iteration, they are generally applicable to gradient descent approaches, including deep learning, where hyperparameter tuning (e.g. learning rate) is crucially important.

About the proper input scaling, it is not only related to the machine learning approach, but to the specific problem under consideration. Sometimes machine learning algorithms rely on distances to compute the similarity between individuals. Scaling changes some of these distances. In these cases, the resulting distance after scaling should be assessed to check whether it is more appropriate than without scaling. Here you can find examples for clustering. For some machine learning algorithms, you need standardized features, e.g. regularized linear/logistic regression. For most optimization-based machine learning algorithms, it makes sense to have feature scaling. On the other hand, there are problems where scaling doesn't even make sense (e.g. discrete input problems, like token-based natural language processing).


Detecting grammatical errors with BERT We fine-tuned BERT (bert-base-uncased) model with CoLA dataset for sentence classification task. The dataset is a mix of sentences with and without grammatical errors. The retrained model is then used to identify sentences with or without errors. Are there any other approaches we could make use of using BERT, other than building a classifier?

Absolutely, you could use bert pre trained model to suggest corrected sentences without annotated data as long as you are using a pre trained model. Here is a good repo that shows it
https://github.com/sunilchomal/GECwBERT
Hope it helps


Advice on transitioning from Andrew Ng's Stanford Coursera Machine Learning (in MATLAB) into Python? I'm currently finishing up Andrew Ng's Coursera course, taught in MATLAB/Octave, but I'm looking to code in Python. The course is an introduction into how some algorithms work from scratch. These include: Linear/Logistic Regression, gradient descent, Support Vector Machines, Neural Networks, K-means clustering, anomaly detection, Gaussian distribution. (Decision trees/random forests aren't covered so I have to learn those).

Any advice on implementing this theoretical knowledge into Python? Is it necessary/recommended to recode the MATLAB algorithms from scratch in Python, or instead should I jump straight into learning Python libraries (sk-learn for now with pandas, matplotlib, etc.) and start projects?
Any project recommendations for my level? Certain concepts I should focus on now? Where to next? I know Python relatively well and have completed tutorials on numPy, Pandas, matplotlib, and sk-learn (very basic).

Next I plan on studying Harvard's CS109 data science course and taking Andrew Ng's Deep Learning Specialization on Coursera.

My ideal goal (a long journey) is to become a machine learning engineer and get into AI and deep learning.

Any advice is welcome! Thanks a lot.

There are so many ways to expand your horizons on ML/DL. Currently I share the same conundrum, so to speak.

In my opinion it is not necessary. Since you have completed the course, if you are confident about the material and know how each algorithm works, re-implementing them on another tool would not be that helpful since there are existing frameworks that you can use. On the other hand if you are not that confident, re-implementing everything on a lower level  (not via APIs) can serve as a second learning experience, making you more fluent in python and more familiar with tensorflow. So, pick a scenario in regards to your case. For me a combination seems to work. I find papers along with other theoretical material and then try to apply it via the frameworks on projects that I like.
The greatest source of projects on any level is Kaggle. You can start with the Titanic competition, choose an algorithm that you like and explore what can be done. Make your submission and then head to another kernel that is more to your liking. You can also study certain pipelines on existing kernels that worked for other people. Considering concepts and methodologies, I would recommend an approach where you study each and every tool that you use and really understand the math behind it. There is no one-pipeline/framework solves all problems (No Free Lunch Theorem).

You also mention the deeplearning.ai specialization, it's a great source of information and Andrew Ng gives you intuition on aspects that are not very obvious when studying on your own. Another thing that helped me a lot is talking with people in the industry. Finding a mentor who inspires you and can give you his/her input on such questions is the best way to move forward. Go to AI/ML meetups and share your interest with other people, get to know companies in your area or even start an internship!

Good luck with your journey!


How do I represent SURF Features into Bag of Words to determine Nearest Neighbors? I'm trying to use Speeded Up Robust Features (SURF) to get the $k$ most similar images from a set of images in my directory. I'm planning to use $k$-Nearest Neighbours ($k$-NN) for this. As far as I know, the size of SURF descriptors are $n \times 64$ or $n \times 128$ depending on how many descriptors I want. There are suggestions and I've read about the Bag of Visual Words method, where these patches are converted to bags of words, similar to the common Natural Language Processing technique. I've also read that Bag of Visual Words are generated by clustering, whereas the feature patches are clustered together.

What I don't understand is, how can I use these Bag of Visual Words to train my $k$-NN such that I can get similar images? I really can't grasp how those clusters can generate BoWs. Say I have 1000 images, if I convert them, what will they look like? Will they be 1000 BoWs that still represent the same images?

The Bags of Visual Words (BoWs) approach to image retrieval, described in works such as Sivic et al. "Video Google: a text retrieval approach to object matching in videos" (2003) and Csurka et al. "Visual categorization with bags of keypoints" (2004), is composed of multiple phases:

First, a visual vocabulary, often called a codebook, is generated. This is usually done by applying k-means clustering over the keypoint descriptors of a data set, or a sufficiently descriptive fraction of it. The vector $\mathcal{V}$ of size $k$ containing the centroids $\mathcal{V_i}$ of each cluster is your visual vocabulary.

In this case, each image $x$ should yield a variable number $x_n$ of SURF keypoint descriptors, usually of size 64 each. One can aggregate all keypoint descriptors from multiple images and perform k-means clustering over all of them.
The choice of the $k$ hyperparameter in clustering depends on the image domain. One may try multiple values of $k$ (10, 100, 1000) to understand which is more suitable for the intended task.

Afterwards, each image is "tested" against the codebook, by determining the closest visual vocabulary points and incrementing the corresponding positions in the BoW for each keypoint descriptor in the image. In other words, considering an image's BoW $B = \{ o_i \}$: for each image keypoint descriptor $d_j$, $o_i$ is incremented when the smallest distance (often the Euclidean distance) from $d_j$ to all other visual vocabulary points in $\mathcal{V}$ is the distance to $\mathcal{V}_i$.

The result is a histogram of visual descriptor occurrences of size $k$, which can be used for representing the visual content. As similar images will yield similar bags of words, one can compare images through their BoWs. The Euclidean distance between them is a commonly used metric here.

Therefore, the bag of words of each image makes a global representation of  that image. When performing content-based image retrieval, the $n$ most similar images are retrieved by fetching the $n$ closest (or approximately closest) bags of words to the query image's. No training process is required at this point (we can picture visual vocabulary generation as an offline training phase).


Random forest implementation with probability of choosing column or guarantee of choosing set of columns Are there random forest implementations which allows for choosing set of columns which will be always selected for every tree in the forest ? or implementations allowing for specifying probabilities of choosing each column ?

Both cases could be simulated by choosing variables (partially or fully randomly) and building tree containing all variables in new temporal dataset, then repeating this procedure until we obtain assumed number of trees and then merging trees into forest,

but there are drawbacks like
moving large number of data in scripting language until they are sent to low level implementation of forest for building tree or lack of merging procedures, which makes work with set of trees harder.

Ranger in R can do the first request. The always.split.variables argument defines which columns should always be included.


Spark Scala alternative Machine Learning Library? I’m using Spark Scala for multiclass classification,  and features are continuous. MLlib seems to be limited to Decision Tree and Random Forest for this type of classification – for Naïve Bayes, Multinomial and Bernoulli are supported, where I would need to use Gaussian, and LogisticRegressionWithLBFGS would not be suitable either.

I know that in Python, you can integrate sci-kit learn with Spark, but are there any options when using Scala?

Interested to hear people’s thoughts on this.

The way you can use scikit-learn is basically broadcasting your data to the workers and then do different folds of crossvalidation or different parameter settings in your grid_search on different workers. That is all that the scikit-learn package in pySpark does as far as I know. This is similar to a normal mapping. Implementing this should be relatively easy for any Machine Learning library in Scala (I do not use Scala, so I cannot help you with suggestions in that regard)


Policy gradient vs cost function I was working with continuous system RL and obviously stumbled across this Policy Gradient.

I want to know is this something like cost function for RL? It kinda gives that impression considering we are finding out how efficient the system is as a whole (weighted sum of rewards multiplied by all the policies).

Let's take the example of vanilla PG:

$$g = \mathbb E\Big[\sum R_t*\frac{(\partial)}{(\partial\theta)}ln\pi_\theta(a_t|s_t)\Big]$$

Here, the gradient is nothing but the expected value of the Return (which is nothing but the discounted sum of all the reward) multiplied by the how the policy (the network output) needs to change according to the network weights.

This seems similar to a cost function where we use the total error using cross-entropy (something similar to the information given by return) and then we use this to see how the weights of the neural network can be changed through backpropagation.

Let me know if I've got this right.

Yes there is a cost (score/utility) function and your intuition is correct. In vanilla PG we optimize the expected return $J$ of a trajectory $\tau$ under policy $\pi$ parametrized by $\theta$:

$$\nabla_{\theta} J\left(\tau\right)
\approx \frac{1}{N} \sum_{i=1}^{N} \nabla_{\theta} \log \pi_{\theta}\left(\tau_{i}\right) r\left(\tau_{i}\right)$$

(You can find lots of information here: https://spinningup.openai.com/en/latest/algorithms/vpg.html)

The vanilla PG is very closely related to maximum likelihood:

$$\nabla_{\theta} J_{\mathrm{ML}}(\theta) \approx \frac{1}{N} \sum_{i=1}^{N} \nabla_{\theta} \log \pi_{\theta}\left(\tau_{i}\right)$$

The two gradients are almost the same except the reward multiplication. This reward is the learning signal. Thinking in terms of backpropagation your gradients are being multiplied by the reward signal. You can think of the vanilla PG as a cost-sensitive classification - with the broad sense.

I will give you a very simple example with Neural Networks in order to demonstrate you the result of the PG learning mechanism. Assume a very simple architecture:

( input --> CNN --> CNN --> fully connected (fc) layer --> out1: V(input), out2: $\pi(a|input)$ )

which can be trained asynchronously or synchronously. The output has two heads one for the expected reward and one for the policy.
Let's assume a task in which an agent has to learn to select between two rewarding targets (orange r=10, green r=1). The targets are presented either both of them or one at a time randomly upon completion of an episode.

Assume now that the agent is fully trained. Taking the fc representations from various episodes and running tsne (clustering) we get the following picture:

This 2D representation tells us that the network has a "clear understanding" when there is only the green target, only the orange target and both of them together. We could colored differently the cluster to get a sense of the expected reward or the action preference (as the fc encodes spatial information, reward information and action preference).

Policy Gradients are mapping states to action distributions (and/or reward predictions). This means that the learned function (the Neural Network) should have the appropriate representations to do that mapping. And this is essentially what is learned by the network: a decision boundary (given input state). Please note that this is a very simplistic example to get an insight of what the network has learned so you can easily draw parallels to the classification case.


Test Binary Classifier on a Test-set that includes only one class I'm working on a disease binary classification problem.
0 = healthy , 1 = not healthy
The disease is a movement disorder that appears on the patient while moving a specific movement.
I applied leave-one-out cross-validation to train on all patients except one to test on and so on.
The problem is, some patients don't show class not healthy at all in their datasets. Thus, the evaluation metrics f1-score, precision & recall went down sharply because of those patients.
I tried to use SMOTE oversampling, but it didn't work because it will generate new samples from the data of other patients. Additionally, I tried class-weights, but it didn't work as well, because there is not a minor class in the test set to give it a higher priority.
How can I solve this problem?

Give cross-validation or bootstrapping a shot!  Also, for metrics, look at per class performance and macro averaged.


How to identify recurring patterns in this set of transactional data I'm working on a dataset of banking transactions and would like to find recurrent transactions.
I've been mapping transactions per merchant in timeseries, and tried to use acf from statsmodels.tsa.stattools to calculate the autocorrelation function but i'm not getting the expected results:

r = acf(ts, fft=False)

For example this set of transaction (ASSURANCE DESJ) is getting an acf score of 0.3159 when it's obviously a recurring transaction (same amount, same frequency).

Another example of recurring transactions with acf=0.22775:

But this one should not be found as a recurring transactions, and get a score not too far from the previous set (0.26919):

I've been checking a lot of different methods, I acutally came up with a combination of auto-correlation on the regular timeserie, auto-correlation on a timeserie with amount=1, with stationnary checking and other rules to have a not so perfect results. I've also checked at ARIMA and other methodology without luck.

Would you have a better way to detect recurring transactions from timeseries ?

Link to Datasets

I don't know the model you use, but I would suggest the perhaps old-fashioned feature engineering approach:

In each set of transactions, calculate the duration between two consecutive transactions for every pair of consecutive transactions (i.e. difference between the dates). Then add the standard deviation of these durations as a a feature, so that a very regular transaction set should get a near zero value.
Same idea with the amount of the transaction: the standard deviation will be zero if the amount never changes.
Additional ideas:

number of distinct amounts across all transactions
number of distinct duration values between transactions in days, weeks, months, years.
standard deviation of the sum by day/week/month/year


How can I calculate a rolling window sum in pandas across this MultiIndex dataframe? I can't work out how to get the moving annual sum from this data:

>                          revenue
> txdate   2014-01-31     2014-02-28      2014-03-31     2014-04-30     ....
> user_id
> 1            0              10             165             0
> 2          265             265             200           250
> 3          770             985            1235           900
> ....

Previously I would have tried something like this and adjusted until it worked:

df.groupby(level='practice_id').apply(lambda x: pd.rolling_sum(x, 12))

but it's deprecated and I'm not getting my head around the 0.18 changes to rolling despite reading the docs, and I'm not sure that the shape of the data is helpful (it's close to what needs to be inserted in a db table). The original data format is as follows:

> txdate          user_id        tx_amount
> 2014-01-01         2               5
> 2014-01-02         2               5
> 2014-01-02         3              30
> 2014-01-03         3              15
> 2014-01-02         2              10

I reshaped with the following cmd:

> df.set_index('txdate').groupby('user_id').resample('M').agg({'revenue': np.sum})

I'm thinking I might need to reverse the order of operations.

If anyone else comes looking, this was my solution:

# find last column
last_column = df.shape[1]-1

# grab the previous 11 columns (also works if there aren't that many in the df)
wanted = df.iloc[:, (last_column-11):last_column]

# calculate the rounded moving annual total
mat_calc = round(wanted.sum(axis=1)/len(wanted.columns), 2)

Probably not the most pandastic solution, but it works well.


DataFrame.dtypes error when training model using xgboost I am getting a DataFrame.dtypes error while following the last steps of this tutorial.

Here is my code:

import xgboost as xgb
regr = xgb.XGBRegressor(colsample_bytree=0.2,
                       gamma=0.0,
                       learning_rate=0.05,
                       max_depth=6,
                       min_child_weight=1.5,
                       n_estimators=7200,
                       reg_alpha=0.9,
                       reg_lambda=0.6,
                       subsample=0.2,
                       seed=42,
                       silent=1)

regr.fit(train_new, label_df)

And this is the error I am receiving:

ValueErrorTraceback (most recent call last)
<ipython-input-64-010296d611e9> in <module>()
     12                        silent=1)
     13
---> 14 regr.fit(train_new, label_df)

D:\anacodna\lib\site-packages\xgboost\sklearn.pyc in fit(self, X, y, sample_weight, eval_set, eval_metric, early_stopping_rounds, verbose, xgb_model, sample_weight_eval_set, callbacks)
    358                                    missing=self.missing, nthread=self.n_jobs)
    359         else:
--> 360             trainDmatrix = DMatrix(X, label=y, missing=self.missing, nthread=self.n_jobs)
    361
    362         evals_result = {}

D:\anacodna\lib\site-packages\xgboost\core.pyc in __init__(self, data, label, missing, weight, silent, feature_names, feature_types, nthread)
    378         data, feature_names, feature_types = _maybe_pandas_data(data,
    379                                                                 feature_names,
--> 380                                                                 feature_types)
    381
    382         data, feature_names, feature_types = _maybe_dt_data(data,

D:\anacodna\lib\site-packages\xgboost\core.pyc in _maybe_pandas_data(data, feature_names, feature_types)
    237         msg = """DataFrame.dtypes for data must be int, float or bool.
    238                 Did not expect the data types in fields """
--> 239         raise ValueError(msg + ', '.join(bad_fields))
    240
    241     if feature_names is None:

ValueError: DataFrame.dtypes for data must be int, float or bool.
                Did not expect the data types in fields Alley, Condition2, Electrical, GarageType, GarageYrBlt, Heating, LandContour, LandSlope, LotShape, MiscFeature, PavedDrive, RoofMatl, Street, Utilities

Check the values of train_new. You'll see that the columns mentionned are not of the expected types.

Another suggestion, i'm not sure of xgboost's handling of nulls. Might be that in those columns aswell.


Naive Bayes vs Full Bayes model classifiers I have a hard time to understand when Naive Bayes works better than Full Bayes.
In general, i know that naive bayes does the assumption that features are independent given the class. However, if features indeed independent, does it mean that assuming that they are dependent yield worst result?

e.g. i got this data points for two features, each class is colored in different color.

Now my intuition is that Naive bayes will work well here, given a specific class we have two different distributions of the class and both are "unstructured".
However, i did ran Naive bayes (with normal pdf) and full bayes (with multivariate pdf) classifiers on that data (using multivariate) and got the same accuracy.

There's no clear definition of "Full Bayes" as a classifier. Most "real world" non-Naive Bayesian classifiers take into account some but not all dependencies between features. That is, they make independence assumptions based on the meaning of the features.

If by "full Bayesian" you mean a joint model (as your example suggests), then one of the problems is that such a model doesn't generalize: it just describes the probabilities in the training set, and that implies that it's likely to overfit badly. This is actually why NB works quite well in most cases: yes it makes unrealistic independence assumptions, but this simplification allows the model to capture basic patterns from the data. In other words, the ability of the model to generalize comes from its excessively simplified assumptions.

Note: as far as I can tell, your example is well chosen and you should see a big difference between NB and a joint model: NB should perform no better than a random baseline while the joint model should obtain near perfect accuracy. There's probably a mistake somewhere if you don't obtain these results. But while this is a good toy example, it cannot help you understand the advantage of the NB assumptions.


Obtain a model where each feature vector is past few samples and labels are future few samples? So, I have this data set where each instance is made of past 20 samples of 2 variables. Labels are next five samples. So every instance looks like this:

Instance1_feature = [s1_var1 s2_var1 ... s20_var1 s1_var2 s2_var2 ... s20_var2]
Instance1_label = [s21_var1 s22_var2 ... s25_var5 s21_var2 s22_var2 ... s25_var2]

Instance2_feature = [s2_var1 s3_var1 ... s21_var1 s2_var2 s3_var2... s21_var2]
Instance2_label = [s22_var1 s23_var2 ... s26_var5 s22_var2 s23_var2 ... s26_var2]

All the data are categorical. var1 takes value in {1,2,3,4,5}; var2 takes value in  {6,7,8,9,10}
I am familiar with machine learning. Hence I am looking for a way to develop a model to predict labels for a new instance. I have reasons to believe that by looking at the past samples, it is possible to predict future samples. Its like stock prices.

Any help, links to code samples will be highly appreciated.

EDIT This is a dataset I collected while driving. I want to see if I can predict next 5 seconds of my driving, given I have past 20 seconds of data. That's why I created the feature vectors and labels like this. The values  {1,2,3,4,5} and {6,7,8,9,10} are categorical, as in the variables are split into segments.

This looks like almost an ideal use case for a Recurrent Neural Network (RNN) model. I say almost, as your output variable is 2 separate class vectors, that part is slightly fiddly.

The main advantage of using a RNN is that the model would be built on the assumption that s1_var1 is of the same type as s2_var1 etc. Whilst with other models you could flatten the time series into a 10 x 20 = 200 long vector, and the output likewise into a 50 long vector, and treat the whole problem as a single step supervised learning, such a model would not include the same assumption.

For your problem, the RNN would have 10 inputs (corresponding to the 5 classes of each of 2 variables) and 10 outputs. You would train it by presenting your input sequence one sample at a time in the correct order 1 to 20, then continue running the network 5 more steps, comparing the outputs against expected. When predicting, you do similar, but then just read off the outputs as predictions, one at a time.

There are a few different Python libraries that support RNN models, and there are different internal choices (such as using LSTM layer versus GRU) which you might want to explore. I am currently learning the Keras framework, which supports some options for RNNs. Probably the example code which learns to predict an exponentially-decaying sine wave would be a good place to start trying to understand the Keras model - although that's a regression, it should give you some insight into how RNNs can be built for this kind of series-based data.

An alternative to RNN here might be a hidden markov model - if you have some insight into the internal state of the system, that could be a better option.


Is the probabilistic cutoff in random forest flexible? I fit the random forest to my dataset with a binary target class. I reset the probabilistic cutoff to a much lower value rather than the default 0.5 according to the ROC curve. Then I can improve the sensitivity (recall) but meanwhile sacrificed the precision.

Just wanna confirm that the default 0.5 is not much meaningful and a practical probabilistic cutoff was often derived from ROC curve in practice. Am I on the right track on the application of random forest and other tree based models.

Yes, you are exactly right. 0.5 is just a heuristic, ROC curve and precision-recall curve give a much better idea of what the cut off should be. You can then use predict_proba, extract the probabilities and do the classification based on the cut-off you have inferred from ROC curve and.  precision-recall curve


RandomForestClassifier OOB scoring method Does the random forest implementation in scikit-learn use mean accuracy as its scoring method to estimate generalization error with out-of-bag samples? This is not mentioned in the documentation, but the score() method reports the mean accuracy.
I have a highly unbalanced dataset, and I am using AUC of ROC as my scoring metric in the grid search. Is there a way to tell the classifier to use the same scoring method on the OOB samples as well?

In general, the performance of classifiers are compared using accuracy, this is a measure of the number of correctly classified instances divided by the total number of instances. However, from the training data we can get a better approximation of the expected error from our classifier when we are using ensemble learning or bagging techniques.

Out-of-bag error

This metric is the accuracy of examples $x_i$ using all the trees in the random forest ensemble for which it was omitted during training. Thus it kind of acts as a semi-testing instance. You can get a sense of how well your classifier can generalize using this metric.

To implement oob in sklearn you need to specify it when creating your Random Forests object as

from sklearn.ensemble import RandomForestClassifier
forest = RandomForestClassifier(n_estimators = 100, oob_score = True)

Then we can train the model

forest.fit(X_train, y_train)
print('Score: ', forest.score(X_train, y_train))

  Score:  0.979921928817

As expected the accuracy of the model when evaluating the training set is very high. However, this is meaningless because you can very well be overfitting your data and thus your model is rubbish. However, we can use the out-of-bag score as

print(forest.oob_score_)

  0.86453272101

This is the accuracy whilst evaluating our instances in the training set using only the trees for which they were omitted. Now let's calculate the score on the testing set as

print('Score: ', forest.score(X_test, y_test))

  Score:  0.86517733935

We see that the accuracy measured by oob is very similar to that obtained with the testing set. It thus follows through the theory that the oob accuracy is a better metric by which to evaluate the performance of your model rather than just the score. This is a consequence of bagging models and cannot be done with other types of classifiers.

Calculating oob using different metrics

Yes, you can do this! However, it depends how exactly your code is structured. I am not sure how you can include the oob and AUC all together with the cross_val_score function. However, if you are doing the cross validation folds manually you can do the following, the random forests algorithm in sklearn provides you the decision function of the oob as

print(forest.oob_decision_function_)

The class can then be obtained using

from sklearn import metrics
pred_train = np.argmax(forest.oob_decision_function_,axis=1)

Then we can calculate the AUC by using the following

metrics.roc_auc_score(y_train, pred_train)

  0.86217157846471204


When should we choose agglomerative clustering over K-means clustering? I was working on a clustering based model and I read about hierarchical clustering and K-Means clustering.
Under what conditions should I choose agglomerative over K-means clustering?

You can see this comparison table in sklearn, which gives some intuition about where and when each algorithm is successful:

It might be a good idea to try both and evaluate their accuracy, with an unsupervised clustering metric, like the silhouette score, to get an objective measure of their performance on a specific dataset.
Some other major differences are:

K-means performs better for 2D & 3D spheres
Hierarchical clustering can have reduced performance on larger datasets
Hierarchical clustering is sensitive to outliers


"Results do not have equal lengths" using ldply in R package plyr I've found a few similar questions, but I am new to R and can't figure out how it applies to my specific problem. Here is my code:

library(rvest)
library(plyr)
library(stringr)

#function passes in letter and extracts bold text from each page
fetch_current_players<-function(letter){
  url<-paste0("http://www.baseball-reference.com/players/", letter, "/")
  urlHTML<-read_html(url)
  playerData<-html_nodes(urlHTML, "b a")
  player<-html_text(playerData)
  player
}

#list of letters to pass into function
atoz<-c("a","b","c","d","e","f","g","h",
        "i","j","k","l","m","n","o","p","q","r",
        "s","t","u","v","w","x","y","z")
player_list<-ldply(atoz, fetch_current_players, .progress="text")

So what this code is trying to do is use the URL structure of this website to pass a list of the letter A through Z into my function to produce a list of names that are in bold. I think the problem is that each list of players it returns is of different lengths and that is producing an error as when I manually type in each letter into the function the function appears to work.

Any help is appreciated, thanks!

Here's a slightly modified version using some newer "tidyverse" packages:

library(rvest)
library(purrr) # flatten/map/safely
library(dplyr) # progress bar

# just in case there isn't a valid page
safe_read <- safely(read_html)

fetch_current_players <- function(letter){

  URL <- sprintf("http://www.baseball-reference.com/players/%s/", letter)
  pg <- safe_read(URL)

  if (is.null(pg$result)) return(NULL)

  player_data <- html_nodes(pg$result, "b a")

  html_text(player_data)

}

pb <- progress_estimated(length(letters))
player_list <- flatten_chr(map(letters, function(x) {
  pb$tick()$print()
  fetch_current_players(x)
}))


How can I load file from my local drive in keras using get_file? I am using google colab and I need to load data from my local drive. How I can do that using get_file? what should be the origin?

get_file() is used for loading files from a URL,hence it can not load local files. If you have mounted you gdrive and can access you files stored in drive through colab, you can access the files using the path '/gdrive/My Drive/your_file'.
for eg:

base_dir ='/gdrive/My Drive/weld_data'

train_generator = datagen.flow_from_directory(
base_dir,
target_size=(IMAGE_SIZE, IMAGE_SIZE),
batch_size=BATCH_SIZE,
subset='training')

If you need to unzip the file, you can use
    import zipfile
    with zipfile.ZipFile(your_file, 'r') as zip_ref:
        zip_ref.extractall(directory_to_extract_to)

I used '/content' as the directory_to_extract_to.

Then you can access the data the usual way.

base_dir = '/content/my_folder'

train_generator = datagen.flow_from_directory(
base_dir,
target_size=(IMAGE_SIZE, IMAGE_SIZE),
batch_size=BATCH_SIZE,
subset='training')


Autoregressive Moving Average ARMA with statsmodels Can I please get direction on what is wrong in the code? All forecasts provide output except the ones listed bellow.

so basically doesn't pick ['UserDefinedData4'] and ['ItemId'] separate ... only when both are  in group_level ...

this link is with the program:
program location

elif forecast_type == 4:
    # forecast weekly order quantity at item level with the dummy variable 'Quarter' using AR(8)
    mape = forecast(dataset, 'W', ['ItemId'], 'Quarter', 8, output_file)
elif forecast_type == 5:
    # forecast weekly order quantity at item level with the dummy variable 'Month' using AR(8)
    mape = forecast(dataset, 'W', ['ItemId'], 'Month', 8, output_file)
elif forecast_type == 6:
    # forecast monthly order quantity at item level with the dummy variable 'Quarter' using AR(2)
    mape = forecast(dataset, 'M', ['ItemId'], 'Quarter', 2, output_file)
elif forecast_type == 7:
    # forecast weekly order quantity at customer level with the dummy variable 'Month' using AR(8)
    mape = forecast(dataset, 'W', ['UserDefinedData4'], 'Month', 8, output_file)
elif forecast_type == 8:
    # forecast weekly order quantity at customer level with the dummy variable 'Quarter' using AR(8)
    mape = forecast(dataset, 'W', ['UserDefinedData4'], 'Quarter', 8, output_file)
elif forecast_type == 9:
    # forecast monthly order quantity at customer level with the dummy variable 'Quarter' using AR(2)
    mape = forecast(dataset, 'M', ['UserDefinedData4'], 'Quarter', 2, output_file)

Thank you for help.

Found the issue and I will post the answer in case anybody will get this issue with ARMA.

model = sm.tsa.ARMA(endog=dataset_period[col], order=(nlag,0), exog= dataset_period[dummy_variable_period]).fit()
dataset_period['predict_'+str(col[1])+'_'+str(col[2])] = model.predict()
dataset_period['ape_'+str(col[1])+'_'+str(col[2])] = abs(  (dataset_period[col] - dataset_period['predict_'+str(col[1])+'_'+str(col[2])]) / dataset_period[col])
valid_ape = dataset_period['ape_'+str(col[1])+'_'+str(col[2])][dataset_period['ape_'+str(col[1])+'_'+str(col[2])] < 50]

  issue was with the parameters. the model accepted 2 parameters for group_level and I was basically providing 1 so in this case automatically become zero and raise error.

  solved with creating a new def and the model will use single column as the parameter for group_level

model = sm.tsa.ARMA(endog=dataset_period[col], order=(nlag,0),exog= dataset_period[dummy_variable_period]).fit()
dataset_period['predict_'+str(col[1])] = model.predict()
dataset_period['ape_'+str(col[1])] = abs((dataset_period[col] - dataset_period['predict_'+str(col[1])]) / dataset_period[col])
valid_ape = dataset_period['ape_'+str(col[1])][dataset_period['ape_'+str(col[1])] < 50]`


Data sources problem I’m doing an “analytics” project in which I have to collect data from small businesses (SMEs) in South America and try to come up with some insights about what are their needs in general. The project’s coach basically wants to collect data from social media and business association websites and do some text analytic on it to see if we can gain some insights.

Ideally, the idea of the project is that given a country and a sector, find what the companies in those sectors are up to and see what are their problems.

This is what I've thought:

Find keywords in the twitter API and facebook API.
Find News about SMEs in a given country and crawl the content.
Find associations web sites. I dis this but the content here is useless most of the time.

The goal of this project is to gain some insights and see if some new products can be customized to these companies.

Thus, my question is: how would you solve a problem like this if you do not have any data? I was also thinking about getting some contact information from a sample of this companies and send them all a survey or something.

The World Bank runs Enterprise Surveys that 'provide the world's most comprehensive company-level data in emerging markets and developing economies. Business data are available on 130,000 firms in 135 countries'. You can get access by requesting on the site subject to some confidentiality terms.

The data are fairly comprehensive and would give you a good starting point to form hypotheses as which areas (access to credit, regulation, corruption, etc) matter most that you could then try to test by mining news or social media.

The World Bank also collects data on Doing Business that, while not specific to small companies, highlight country-specific challenges.

Plenty of research has been published around the two programs above.


Build a sentiment model from scratch I would like to know how I can create a sentiment model from scratch.
I have my data, list of texts, with no labels about sentiment.
Author     Quotes
Dan Brown  “Everything is possible. The impossible just takes longer.”
Dan Brown  “Great minds are always feared by lesser minds.”
Dan Brown   “Men go to far greater lengths to avoid what they fear than to obtain what they desire.”
Dan Brown   “Google' is not a synonym for 'research'.

etc...
I have 20000 of quotes like the above, from other authors too.
My dataset is the only set of data that I have, so I would need to split into training (80%) and test set (20%).
I should work on my training set to build the vocabulary of 'sentiment'.
What I have thought is to clean the text removing stopwords, except negative words (like no, not, ...). Then look for adjectives and assign them a score manually (?).
Once done this, I should have also consider n-grams, especially with negative stopwords, in order to create a small dictionary which could take into account also this.
However I do not know if it could be the right approach, if it is something insane...
Is there any way to build from scratch a sentiment model? How could the 'machine' learn from this?
I would need to develop a model from scratch as I would like to analyse texts in other languages (like Italian or Spanish) and there are no models (not good model at least) in Python to do that.

What you're describing is indeed the traditional approach for building a sentiment analysis system, so I'd say it looks like a reasonable approach to me.
I'm not up to date with the sentiment analysis task at all, but I think it would be worth studying the state of the art for several reasons:

There might be more recent, better approaches
There might be datasets in the languages you're interested in, and if there is that could save you a lot of time. Check if there are any shared tasks about this, they often provide annotated datasets.


What is the correct way to compute Mean F1 score? I have a set of 10 experiments that compute precision, recall and f1-score for each experiment.
Now, average precision & average recall is easy to compute. I have some confusion regarding average f1-score.

There are 2 ways on how i can compute mean f1-score:

Take f1 scores for each of the 10 experiments and compute their average.
Take average precision & average recall and then compute f1-score using the formula f1 = 2*p*r/(p+r)

I could not find any strong reference to support any of the arguments. The closest document i could find is this: https://www.kaggle.com/wiki/MeanFScore

Can anyone explain with some reference (if possible) which of the methods is correct and why?

EDIT: One of the members suggested this source. Though, i still suspect the reliability of the source. I have seen people not using the method explained above in their research publications. (Even i would be using it in one of my publications) I would expect some more opinions from the community to verify this idea.

This paper Apples-to-apples in cross-validation studies: pitfalls in classifier performance measurement by Forman and Scholz discuss the different methods for computing the average F-score in cross validation. The paper shows that under very high class imbalance some of the computation methods (average of individual folds F-scores or F-score based on average of individual folds precision and recall) can lead to biased results. The paper recommends computing the F-score by adding the TP, FP, FN from each fold, computing the precision and recall and finally the F-score.