Predicting a numeric variable from many numeric variables, how to choose a proper structure? I need to estimate the value of a one numeric variable from the values 8 numeric variables

y = f(x_1, ..., x_8).

I have an historical dataset where I can relate thousands of y values to the corresponding x, that will grow with time potentially to millions of records.

Of course I don't know the relationship between x and y, for sure I know it is not linear with x (I'm guessing y depends quadratically on some xs) and in principles it could be that there is no analytical expression at all for this relationship.

For these reasons I guessed to solve the problem with a Neural Network but I'm pretty new to this topic so I'm trying to figure out (with very little success) which model should be suitable for my problem, i.e. how many hidden layers to choose and how many neurons for each layer? Should I put ReLUs or other functions between layers?

Of course I'm not interested just in solving this particular situation but in understanding the possible choice criterias!

Also I would be interested in this: if I know that y probably depends on combinations or functions of the x_i such as y = f(x_1 - x_2, x_3**2, ..., x_8) should I put directly those guesses as inputs in the net or plain x_8s?

If it's useful in any way, I'm pretty familiar with python and I would like to implement this solution with PyTorch for my personal learning benefit. An hint for the Model class structure would be most appreciated.

Thank you

How much data do you have? this is a regression problem that doesn't require using neural networks, you can solve this by using much simpler algorithms ( SVM, Linear Regression , Tree-based regressors ) ( I'm assuming you don't have millions of rows of data ). As in for the inputs, you can either use the numeric variables x1 to x8 or create some other variables based on those depending on which is more correlated with y. It is the algorithm's mission to go through your data and come up with a function that gives you a y for your X .


Which between random forest or extra tree is best in a unbalance dataset? I have an unbalanced dataset, with 3 classes, with 60% of class 1, 38% of class 2, and 2% of class 3.
I don't want to generate more examples of class 3, and I cannot get more examples of class 3.
The problem is that I need to choose between RandomForest, and ExtraTree (this is homework), and explain why I choose one of these.
So I choose the Random Forest classifier, but I am not sure if my assumptions are right or no.
I choose that, because, the split of extra tree is random, so the probabilities of the pick some examples of class 3 are low, and because I think (this is the real question) that because Random is more high-variance than Extra tree, can be more useful because the high variance can help with the dataset is unbalance.
So are this two assumption especially the last one, correct? I choose correctly random forest over extra tree?
Thanks

Both Random Forest Classifier and Extra Trees randomly sample the features at each split point, but because Random Forest is greedy it will try to find the optimal split point at each node whereas Extra trees selects the split point randomly.
I would choose Random Forest because it's more likely to create a split point that accounts for the imbalanced class, whereas Extra Trees might keep splitting over and over again on a subset of the data without separating out class 3 due to the random split point.


Similarity Measure between two feature vectors I have face identification system with following details:

VGG16 model for feature extraction
512 dimensional feature vector (normalized)

I need to calculate similarity measure between two feature vectors. So far I have tried as difference measure:

Pairwise cosine, euclidean distance
Dot product (both vectors are normalize, so their dot product should be in range [-1, 1])

These methods are working fine when I want find closest feature vector from set of Feature Vectors. By sorting difference(distance) measure. But this method is relative measurement.
I want to have percentage of similarity. So I can say given image is Person XXX with degree of certainty of x percentage
Lets say that I have 2 feature vectors ( of person a and b).

I want to calculate accuracy level (in a percentage) that person a is
indeed person b, by probability of some percentage.

So, thank you for clarifying the question. Just to confirm that the question is asking how to set an appropriate threshold for face feature vectors (represented a a and b, for example).
What I would recommend is to look at either cosine similarity or euclidean distance, which you have implemented. From here, I would then look the distribution fo the similarity metric over all face vector pairings. Here, this can give you an idea of general level of similarity over all face vector pairings (e.g. if most of the distribution is skewed towards lowers facial similarities, then this could indicate overall lack of similarity between face vector pairings).
If the distribution is skewed to higher similarity values, then you could take, say, the 90th percentile over this distribution as the threshold for determining whether a == b.
Another idea would be to use the idea of softmax in the following way. This method would allow for others to replicate your methodology precisely:
For each face $f_i \in F$, you compute the softmax over the face vector pairings between $f_i$ and $f_j \in F, i \neq j$. Then, from this, you select the index $j$ which has the highest value, which can then be assumed to be such that $f_i$ == $f_j$ (a == b)


Is there such thing as linear and non-linear data? While doing machine learning projects we've heard that logistic regression works well with "Linear data" and decision tree works well with "non-linear data"
However concept of linear and non-linear data does not make sense. To me only linearly separable data and non-linearly separable data makes sense to me, it only makes sense to say logistic regression works well with "Linearly separable data" since it is a linear function. In mathematics linear functions are polynomials with degree one and all other functions that are not linear are considered non-linear function.
What exactly is linear data and non-linear data?

Naming linear data or non-linear data is a bit misleading and wrong I would say. Instead, there is linear relation and non-linear relation between variables would be better and correct naming. It can easily be explained by example from real life.

Non-linear relation: Any kind of relationship that is not linear can be put forward as an example (e.g. quadratic relation: y = intercept + x^2, square root relation: y = intercept + 6 * sqrt(x) etc.) Let's imagine a world where we are not aware of the free fall equation and we use ML to learn and predict it :). In that case (specific and simplified case), the algorithm would need to learn an equation in the following structures = v₀t + (1/2)gt², where t is our independent variable, s is our target or dependent variable and v₀, g would be our coefficients to be learned. In other words, you would want to predict free fall (distance or height) using time. In that fictional world, Newton’s Law of Universal Gravitation (F = G * (M1 * M2) / r^2), Area of a circle (A = π * r^2) would also be a quadratic problem for ML. Linear functions do not meet the requirements of such cases unless you use some kind of transformation, let's say take the square root of r^2 from Newton's law or the square root of t^2 from free fall then use them in a linear function. The most known example of non-linear real-life relation is a prediction of height using age. Height changes continuously, however, in different rates: until 13 gradually increases, between 13-18 significantly increases, between 18-25 slightly changes, and after 25 it does not change. Thus, it is impossible to fit it into a linear equation, because it is not dependent on a single coefficient or i.o.w. it does not fit into the formula height = intercept + b * age because coefficient (b) is not constant over time (age). Moreover, Graph, Tree, and other similar structures are also fitted into a non-linear relationship. In cases where the result is highly dependent on ifs, linear algorithms are useless because you cannot fit the relationship into y = a + b * x. Let's say, you want to predict who will win a chess game basing on the moves they make, you would use a tree-like (e.g.: alpha-beta pruning) algorithm to predict it.
Linear relation: It is simply the relations where those relations can be explained by the y = a + b * x formula or even y = a + b1* x1 + b2*x2 + .... The simplest example is predicting the cost that is spent in the bar (or any kind of entertainment venue). cost = intercept (let's imagine you pay money for entering to the bar)  + (the price of a drink) * (the number of drinks bought) + (the price of appetizer) * (the number of appetizers bought) would be our formula.


Display a subset of classes in axes in ggplot I have the following plot, is there any way in ggplot to display just the numbers 1 to 10 instead of all of them?
Numbers from 10 and after are not so important, but I need to display the ones before.
Thank you

I used this kind of wor around, but if you have another solution, please enlighten me :)

df %>%
  ggplot() +
  geom_bar(mapping = aes(x = N%>% as.factor())) +
  xlab(label = 'N') +
  theme(axis.text.x = element_text(angle = 45, size = 8))


What fields offer most data science job opporunities? I'm now transitioning to data scientist as bioinformatics PhD.

What fields need lots of data scientists? Or offer more opportunities?
I guess business/finance and internet?

What difference does it make? Nothing stops you from applying to all of them. The real question is, can you and do you want to write software or do you just want to analyze data? Pay attention to the job descriptions in this regard. Data analysts are typically expected to be strong in classical statistics, while machine engineers are expected to know ... machine learning. Then every field has certain problems that you need to be familiar with before taking the technical interview, such as recommender systems for media companies, and time series forecasting for business/finance. You can find out what they are by reading around and actually applying to companies. Expect several rounds of interviews, including mathematical and programming challenges (real-time or off-line), and questions about your research.

Once again, every field needs data scientists, so this is not a criterion.


Data cleaning: Relationships between columns I have a training data set distributed in two files.

File 1: This contains actual classification for each X1. X1 is unique in this file. X1 has one-to-one relationship with X2, i.e. X2 is also unique. Y is binary.

| X1 | X2 | Y  |
| 1  | 4  | 0  |
| 3  | 5  | 1  |
...
| 8  | 9  | 1  |

File 2: This contains the real 'observations' of the experiment. X1 can appear multiple times.

| X1 | X3 | X4 |
| 3  | 4  | 5  |
| 3  | 1  | 2  |
...
| 1  | 4  | 8  |

Here I can combine the two tables to have a structure like below and use them as observations:

| X1 | X2 | X3 | X4 | Y |
| 3  | 5  | 4  | 5  | 1 |
| 3  | 5  | 1  | 2  | 1 |
...
| 1  | 4  | 4  | 8  | 0 |

For test data I have similar structure, just the Y column is missing in File 1.

I have multiple concerns here:

X1 and X2 has one-to-one dependency in the data, i.e. X1 = f(X2) and X2 = f(X1)
Y = f'(X1) or f'(X2)
Frequency distribution of X1,X2 and Y changes dramatically in the new joined data set.

Questions:

Does this kind of transformation of data leads to any insights?
Does regression and ensemble learning techniques are capable of capturing these internal relationships?

I see several issues in your data.

First of all, if there is a one-to-one relationship between X1 and X2, you should remove one of the two columns, because they are redundant. Redundant data may have a negative impact on your classification performance.

Secondly, the fields X3 and X4 also seem to be totally redundant, since the value of the class label Y only depends on X1/X2. So unless the columns X3 and X4 may be interesting on their own, I don't see the point of including them into the data.

Having dealt with these issues, and in order to obtain Y from X1/X2, there are two possibilities. If file1 contains the value of Y for any possible value of X1 in your domain, you don't need any machine learning technique. You have a perfect mapping. Otherwise, you will need to apply machine learning to find a function that "fills the gaps". Depending on the nature of the Y variable, you will need to use a regression (if Y is a real number) or classification (if Y is a discrete variable).


When I would use a specific similarity coefficient over another? Like using Jaccards over Dice.
I want real examples, of when I would prefer to use Jaccards, Dice, Cosine or any other similarity coefficient.

Jaccard - measures similarity of assymetric, binary attributes. For example, if you have insurance claims with binary attributes ("poor driving record", "premium paid in cash") you can compare claims with those attributes.
Cosine - measures similarity between vectors, like feature vectors. Could be used in a recommender system where a user asks to see items similar to some selected item.
Dice - equivalent to F1-score. Often used in image segmentation, comparing a model's output with reference masks.


OOB decision function doesn't match prediction in scikit-learn RandomForest I am using a RandomForest for multiclass classification. I would like to use the oob_decision_function to explore precision/recall, but I don't understand the OOB results.

I am using 25,000 trees (n_estimators=25000) and the possible class values are 0,1 or 2.

My results are (truncated to the first 10 rows):

y_train = [1 2 1 1 0 0 2 2 2 2]
y_pred  = [2 2 2 1 0 2 2 2 2 2]

oob_decision_function:
[[ 0.25377529  0.28080796  0.46541674]
 [ 0.32162915  0.3250808   0.35329005]
 [ 0.34463485  0.27709584  0.3782693 ]
 [ 0.31091392  0.2982096   0.39087648]
 [ 0.34932553  0.28632762  0.36434685]
 [ 0.31535905  0.19570567  0.48893528]
 [ 0.25472683  0.35845451  0.38681866]
 [ 0.32521156  0.31721116  0.35757728]
 [ 0.30706625  0.32703203  0.36590172]
 [ 0.29785305  0.22490485  0.4772421 ]]

The dataset is not uniform:

class 0:  32% of samples
class 1:  27% of samples
class 2:  41% of samples

The predictions don't seem to agree with the decision function. See for example the 4th sample: the prediction (y_pred[3]) is class 1, but the OOB values are 0.3109 (class 0), 0.2982 (class 1) and 0.3908 (class 2). Why is the prediction class 1 and not class 2? I thought after 25,000 trees the prediction should closely match the OOB probabilities, or am I not understanding OOB properly?

A few questions:

how are the OOB values calculated? I thought it was as follows: for
each of the 25,000 trees it creates a new bagged training set. Let's
say a given sample was not included (ie. out-of-bag) in 6000 of those
trees. When the sample is out-of-bag, the current tree is used to predict the
class for that sample. So for our given sample, it was estimated 6000
times and let's say the predictions were class 0 (1000 times), class
1 (2000 times) and class 2 (3000 times). Therefore the OOB values
would be 1000/6000 (class 0), 2000/6000 (class 1) and 3000/6000
(class 2). Is this correct?
the OOB values are calculated using the results of the predictions
from individual trees (as explained above), but the output
predictions (ie. y_pred) are from the final ensemble of trees. Is
this correct?
should the OOB values converge to the cross validation values as the
number of trees increases? AFAIK k-fold cross validation splits the
data into k subsets and uses the entire forest on each subset.
is there some sort of weighting being applied (e.g. if my data are
not uniformly distributed)?

From your example I understand that $y_{pred}$ are the predictions of the random forest on the training set. If this is the case, $y_{pred}$ would give you training performance whereas the oob values will give you validation performance. Because you are overfitting your training predictions are close to the true values whereas your validation/oob predictions are not.

When bagging, on average each bag contains 2/3 of the samples leaving the remaining 1/3 in the oob. When you use the random forest to predict on the training set, for each sample there will be about 17 000 (~2/3*25000) trees which have seen that datapoint already since it's in their bag and results will not be reliable in terms of generalisation. In other words, when you predict on the training set, 2/3 of the trees will give you training predictions and 1/3 (oob) will give you validation predictions so overall it's like getting training performance.

Answering your last four questions:

Random forest in sklearn gets the probability of each class for each tree and then averages them across all bags to get the final probabilities. This is usually called 'soft voting', 'hard voting' being the method you describe.
See 1.11.2.1. Random Forests
I haven't seen it stated clearly in the documentation but I would expect that the same method that is used to make general predictions ('soft voting') will be used to get the oob values.
Yes, cross validation and oob scores should be rather similar since both use data that the classifier hasn't seen yet to make predictions.
Most sklearn classifiers have a hyperparameter called class_weight which you can use when you have imbalanced data but by default in random forest each sample gets equal weight.


Python script/GUI to generate positive/negative images for CascadeClassifier? For the first time, I am playing around with a Cascade Classifier with the OpenCV package (also new to the latter). I realized that it would probably be faster to write my own GUI/script to generate the needed positive and negative images from the set of images I have than to open each file in Photoshop or Paint, but I also suspect this has been done many times before.

In particular, I am looking for a GUI that lets users page through files in a directory and then use mouse clicks to draw rectangles on a particular image and have the coordinates of the rectangle recorded for later purposes. Any suggestions? If not, I'll  be sure to post a link when/if I finish this. It seems like something of general enough utility I am surprised I can't find it in the OpenCV package itself.

So I wrote the script. It gave me an excuse to learn Tkinter. It's pasted below. Note this is a one-off, not a model of good programming practice! If anyone uses this and has bugs or suggestions, let me know. Here's the git link and model code is pasted below:

import Tkinter
import Image, ImageTk
from Tkinter import Tk, BOTH
from ttk import Frame, Button, Style
import cv2
import os
import time
import itertools

IMAGE_DIRECTORY =            # Directory of existing files
POSITIVE_DIRECTORY =         # Where to store 'positive' cropped images
NEGATIVE_DIRECTORY =         # Where to store 'negative' cropped images (generated automatically based on 'positive' image cropping
IMAGE_RESIZE_FACTOR =        # How much to scale images for display purposes. Images are not scaled when saved.

# Everything stuffed into one class, not exactly model programming but it works for now
class Example(Frame):

    def __init__(self, parent, list_of_files, write_file):
        Frame.__init__(self, parent)
        self.parent = parent
        self.list_of_files = list_of_files
        self.write_file = write_file
        self.image = None
        self.canvas = None
        self.corners = []
        self.index = -1
        self.loadImage()
        self.initUI()
        self.resetCanvas()

    def loadImage(self):
        self.index += 1
        img = cv2.imread(self.list_of_files[self.index])
        print(self.list_of_files[self.index])
        while not img.shape[0]:
            self.index += 1
            img = cv2.imread(self.list_of_files[self.index])
        self.cv_img = img
        img_small = cv2.resize(img, (0,0), fx = IMAGE_RESIZE_FACTOR, fy = IMAGE_RESIZE_FACTOR)
        b, g, r = cv2.split(img_small)
        img_small = cv2.merge((r,g,b))
        im = Image.fromarray(img_small)
        self.image = ImageTk.PhotoImage(image=im)

    def resetCanvas(self):
        self.canvas.create_image(0, 0, image=self.image, anchor="nw")
        self.canvas.configure(height = self.image.height(), width = self.image.width())
        self.canvas.place(x = 0, y = 0, height = self.image.height(), width = self.image.width())

    def initUI(self):
        self.style = Style()
        self.style.theme_use("default")
        self.pack(fill=BOTH, expand=1)

        print "width and height of image should be ", self.image.width(), self.image.height()
        self.canvas = Tkinter.Canvas(self, width = self.image.width(), height = self.image.height())
        self.canvas.bind("<Button-1>", self.OnMouseDown)
        self.canvas.pack()

        nextButton = Button(self, text="Next", command=self.nextButton)
        nextButton.place(x=0, y=0)

        restartButton = Button(self, text="Restart", command=self.restart)
        restartButton.place(x=0, y=22)

    def nextButton(self):
        new_img = self.cv_img[self.corners[0][1]/IMAGE_RESIZE_FACTOR:self.corners[1][1]/IMAGE_RESIZE_FACTOR, self.corners[0][0]/IMAGE_RESIZE_FACTOR:self.corners[1][0]/IMAGE_RESIZE_FACTOR]
        files = self.list_of_files[self.index].split("/")
        try:
            os.stat(POSITIVE_DIRECTORY+files[-2])
        except:
            os.mkdir(POSITIVE_DIRECTORY+files[-2])
        print("saving to ", "{}{}/{}".format(POSITIVE_DIRECTORY, files[-2], files[-1]))
        cv2.imwrite("{}{}/{}".format(POSITIVE_DIRECTORY, files[-2], files[-1]), new_img)
        self.saveNegatives(files)
        self.restart()
        self.loadImage()
        self.resetCanvas()

    def saveNegatives(self, files):
        low_x = min(self.corners[0][0], self.corners[1][0])/IMAGE_RESIZE_FACTOR
        high_x = max(self.corners[0][0], self.corners[1][0])/IMAGE_RESIZE_FACTOR
        low_y = min(self.corners[0][1], self.corners[1][1])/IMAGE_RESIZE_FACTOR
        high_y = max(self.corners[0][1], self.corners[1][1])/IMAGE_RESIZE_FACTOR

        try:
            os.stat(NEGATIVE_DIRECTORY+files[-2])
        except:
            os.mkdir(NEGATIVE_DIRECTORY+files[-2])

        new_img = self.cv_img[ :low_y, :]
        cv2.imwrite("{}{}/{}{}".format(NEGATIVE_DIRECTORY, files[-2], "LY", files[-1]), new_img)
        new_img = self.cv_img[ high_y: , :]
        cv2.imwrite("{}{}/{}{}".format(NEGATIVE_DIRECTORY, files[-2], "HY", files[-1]), new_img)

        new_img = self.cv_img[ :, :low_x ]
        cv2.imwrite("{}{}/{}{}".format(NEGATIVE_DIRECTORY, files[-2], "LX", files[-1]), new_img)
        new_img = self.cv_img[:,  high_x: ]
        cv2.imwrite("{}{}/{}{}".format(NEGATIVE_DIRECTORY, files[-2], "HX", files[-1]), new_img)

    def restart(self):
        self.corners = []
        self.index -=1
        self.canvas.delete("all")
        self.loadImage()
        self.resetCanvas()

    def OnMouseDown(self, event):
        print(event.x, event.y)
        self.corners.append([event.x, event.y])
        if len(self.corners) == 2:
            self.canvas.create_rectangle(self.corners[0][0], self.corners[0][1], self.corners[1][0], self.corners[1][1], outline ='cyan', width = 2)

def main():

    root = Tk()
    root.geometry("250x150+300+300")

    list_of_files = []
    file_names = []
    walker = iter(os.walk(IMAGE_DIRECTORY))
    next(walker)
    for dir, _, _ in walker:
        files = [dir + "/" +  file for file in os.listdir(dir)]
        list_of_files.extend(files)
        file_names.extend(os.listdir(dir))

    list_of_processed_files = []
    processed_file_names = []
    walker = iter(os.walk(POSITIVE_DIRECTORY))
    next(walker)
    for dir, _, _ in walker:
        files = [dir + "/" +  file for file in os.listdir(dir)]
        list_of_processed_files.extend(files)
        processed_file_names.extend(os.listdir(dir))

    good_names = set(file_names) - set(processed_file_names)
    list_of_files = [f for i, f in enumerate(list_of_files) if file_names[i] in good_names]

    app = Example(root, list_of_files, IMAGE_DIRECTORY+"positives")
    root.mainloop()

if __name__ == '__main__':
    main()


How will a rotation matrix affect contestants in machine learning contests? Machine Learning contests like Kaggle usually layout the machine learning task in a human-understandable way. E.g. they might tell you the meaning of the input (features). But what if a Machine Learning contest doesn't want to expose the meaning of its input data? One approach I can think of is to apply a (random) rotation matrix to the features, so that each resulting feature doesn't have obvious meaning.

A rotation on the input space shouldn't change a model's ability to separate the positives from the negatives (using binary classification as an example) -- after all the same hyper plan (when applied the same rotation) can be used to separate the examples. What could be changed by the rotation is the distribution of each feature (i.e. when looking at a single feature's values across all examples) if a contestant cares about them. However, rotation is PCA-invariant, so if a contestant decides to work on the PCA-ed version of the input then the rotation doesn't change anything there.

How much do contestants reply on statistical analysis on the (raw, i.e. non-PCA-ed) input features? Is there any (other) thing that I should be aware of that a rotation can change for a contestant during such a contest?

Kaggle competitions with clean, anonymised and opaque numerical features are often popular. My opinion is they are popular because they are more universally accessible - all you need is to have studied at least one ML supervised learning approach, and maybe have a starter script that loads the data, and it is very easy to make a submission. The competitions become very focused on optimising parameters, picking best model implementations and ensembling techniques. The more advanced competitors will also refine and check their CV approaches very carefully, trying to squeeze the last iota of confidence out of them in order to beat the crowd climbing the public leaderboard.

Examples of historic Kaggle competitions with obfuscated data might be Otto Group Product Classification or BNP Paribas Cardif CLaims Management. For some of these competitions the data is adjusted for anonymity of the users who might otherwise be identified from the records. In other cases it is less clear what the sponsor's motivation is.

However, there are negative consequences (you will find these complained about in the same competitions):

Use of insight from domain knowledge, or exploration/study of the underlying principles from the subject being predicted are effectively blocked. It is hard to assess the impact of this, but it is possible that the sponsors miss out on potentially better models.
Doing "just" the machine learning side can be a bit too mechanical and boring for some competitors, who may not not try as hard.

  How much do contestants reply on statistical analysis on the (raw, i.e. non-PCA-ed) input features?

There are always data explorations and views of data published in forums (and Kaggle's scripts - called kernels), and many people view, upvote and presumably use the insights from them. I recall at least one competition forum thread where there was a lot of discussion about weird patterns appearing in data, which were probably an artefact of obfuscation (sorry I cannot find the thread now).

With obfuscated data, there can be attempts to de-obfuscate, and they have sometimes been partially successful.


Caret "train" or special package I see that there is an XGBoost package as well as XGBoost option in the "train" function in Caret. What is the difference between these two options, is it that they provide different settings? More generally, how would one choose between using the "train" function and a standalone package for the method?

Caret is designed to:

  streamline the model training process for complex regression and classification problems. The package utilizes a number of R packages...

(https://cran.r-project.org/web/packages/caret/vignettes/caret.html)

Caret's train has two draws: a common API for many different models, and performing hyperparameter tuning by default.

Specifically, caret calls the xgboost package. It does some checking of your data before calling xgb; most notably, it uses that to select the objective function (which may cause some confusion). And it provides some hyperparameter distributions to search over (which you can modify if desired). However, it doesn't seem possible to search over the hyperparameters of xgboost that aren't built into the caret list of parameters.

So, it seems to me: use caret for a convenient wrapper and hyperparameter tuner, and use the original package for finer control.


Properly using activation functions of neural network I'm trying to understand the hidden layers of neural networks. Input layer section covers the steps that I use before passing information to hidden layer where concerns appear.

Input Layer:

From my understanding the first step in the neural network is to weight
inputs by picking the "best" linear function.

Methods used for finding optimal weights (may not be relevant to problem):

   For finding optimal weights solving quadratic minimization problem is what I usually do, perhaps by finding global minimum of the convex (quadratic error $||Ax-b||^2$ in this case, $A$ being basis matrix and $b$ being ideal vector) function (since every critical point is global minimum there) or orthogonal projection equations (since error is orthogonal to the column space of basis, and their dot product gives us the equation - $A^T(Ax-b$), equating it to $x$: $x = (A^TA)^{-1}A^Tb$.

Once best linear coefficients are found (say $m$ as slope and $c$ as bias), next step is to weight the inputs by evaluating them into the function $f(x)=mx+c$.

Hidden Layer:

After all inputs are weighted, they must be summed up which gives us a constant number.

I understand that this constant number must then be inputted to some activation function (perhaps sigmoid: $f(x)=\frac{1}{(1+e^{-x})}$ or reLU: $f(x) = max(0, x)$.

Representation of simple neural network with single hidden layer having sigmoid as activation function:

Picture reference

Problem:

But constants that are evaluated in these activation functions obviously output constants again, so how can they be utilized for data prediction?

For example, say sum of all weighted inputs is $15$, then its evaluation in sigmoid function will be $\frac{1}{(1-e^{-15})} = 0.99999969409$. That constant number can't be utilized for data prediction (classification/regression) so what are next steps to take? If activation function returns a constant number, how can the data be predicted for different input variables?

Am I having incorrect perspective for activation functions?

How is data predicted from activation functions? (considering that it returns constants on weighted input sum).

You should consider the fact that the label of your input data is going to be predicted by the network. Moreover, the outputs of the network usually represent the probability of belonging to each class. The label is not predicted by the activation function, it is predicted by the network. Take a network as a mapper which can map the inputs to outputs. The activations are used to add non-linearity in order to approximate complicated non-linear mappings.

  What's the point of multiple hidden layers?

As you can read here, the purpose of adding those is that you can learn different complicated regions. Another interpretation can be this that by adding more layers you can learn more complicated mappings which have non-linear behaviour.

About the global minimum, because you can not see the cost function which is the error based on the weights, due to the fact that it is not possible to visualise it because it has so many dimensions, you should use gradient-descent based algorithms.


How to scrape imdb webpage? I am trying to learn web scraping using Python by myself as part of an effort to learn data analysis. I am trying to scrape imdb webpage.
I am using BeautifulSoup module. Following is the code I am using:
r = requests.get(url) # where url is the above url
bs = BeautifulSoup(r.text)
for movie in bs.findAll('td','title'):
    title = movie.find('a').contents[0]
    genres = movie.find('span','genre').findAll('a')
    genres = [g.contents[0] for g in genres]
    runtime = movie.find('span','runtime').contents[0]
    year = movie.find('span','year_type').contents[0]
    print title, genres,runtime, rating, year

I am getting the following outputs:
The Shawshank Redemption [u'Crime', u'Drama'] 142 mins. (1994)

Using this code, I could scrape title, genre, runtime,and year but I couldn't scrape the imdb movie id,nor the rating. After inspecting the elements (in chrome browser), I am not being able to find a pattern which will let me use similar code as above.
Can anybody help me write the piece of code that will let me scrape the movie id and ratings ?

Instead of scraping,  you might try to get the data directly here.  It looks like they have data available via ftp for movies, actors, etc.


Machine Learning or Survival Analysis? I am working on building prediction model for disk failures (time taken to occur a disk failure and what parameters could strongly affect disk failures). I am bit confused on-

What data preprocessing steps should I perform. The dataset is
highly imbalanced (500 failures and ~40000 non-failures)
What type of Machine Learning models should I take into consideration as data is highly imbalanced?
Few days back, I read about Survival Analysis
and now I am in conundrum whether the problem would be of Survival
Analysis or Machine Learning?

I am currently working with dataset provided by BackBlaze(https://www.backblaze.com/b2/hard-drive-test-data.html).

It would be really great it I could get some direction :)

Some algorithms, such as SVM or Logistical regression, have possibility to add a weight to certain class, therefore fix the unbalanced issue.

This really sounds like a job for Survival analysis, which is especially designed to answer questions like "When machine X fail" or "Which attribute influence the most the failure". You can simply start by plotting the Kaplan-Meier curve and then further stratify it by some attribute. Then you can try Cox regression model - it is useful to see the influence of an attribute on survival - the hazard ratio.
But don't forget to verify the assumptions (functional form and proportional hazard).

In R the Survival analysis is implemented very well, so don't be affraid.
There is simple and short tutorial which might help.


Mastering NLP: Reading List I've searched the web and there are hundreds of recommendations on what to read. The time moves on and new better quality techniques are published, so I would like to know what is relevant in 2018?

My background is 4 years of BSc in Maths & Stats (top uni) + 1 year of role in Data Science (building predictive models, no NLP).

If possible, please separate it into sections/readings, e.g.

Background (History, e.g. philosophical)
Theoretical (Mathematics)
Practical (Using Tensorflow and other NLP libraries to build algorithms)

I have a few side projects that I would like to do:

Build an algorithm which answers multiple choice questions

E.g. given a question:

Which is not a fruit?
1) Apple 2) Cucumber

I would like NLP to understand negation, and find that the topic of the question is fruit. Then I'd probably incorporate Google Search API or something.

Categorise a list of 'keyword' searches into categories.

Let's take google which probably has something like this, it categorises every keyword and gives recommendations. Given a list of 10,000 searches, I would like to categorise them into N categories, based on similarity (not just how similar the words are, but including synonyms).

I would suggest instead of trying to get many sources, get one source that goes through concepts first (fairly robustly), then seek out sources to refine or deepen your knowledge. One source comes from Stanford's NLP group, and is Introduction to Information Retrieval.  The only thing I don't like about this books is that it tends to orient documents as columns (where data science has more or less agreed that they are rows), but that's a pretty trivial concern (as long as you can take a matrix transpose).  Aside from that, this book has excellent explanations, and the proper depth and breadth to be considered an exhaustive base for NLP.


Anomaly detection on time series I've just started working on an anomaly detection development in Python.

My data sets are a collection of timeseries. More in details, data are coming from some sensors/meters which record and collect data on boilers or other equipments.

As I said before, the data which I have to work with, are timeseries, so a timestamp and the relative value detected by sensor; a value is anomalous when it's bigger or smaller than the others near it; basically a peak.

I need to develop an unsupervised classification model, because I haven't labels for all data.

Another important aspect, is that this data are "season dependent"; in fact a boiler should be has a higher consumptions in winter than summer. Those values must not be considered as anomalies.

Since I've no experince on this topic, I'm here to ask you, what is the best algorithm/approace to solve this problem.

Furthermore, do you know some books or links to suggest?

You can have a look here, where many open-source algorithms specifically for anomaly detection on time-series data (e.g. metrics) are collected, both for online of offline settings.

Almost all of them are unsupervised approaches that require no labels to detect the anomalies.

They also automatically handle some of the issues you mentioned, like seasonality.


Extension of NMF to 3D AFAIK, Non-Negative Matrix Factorization (NMF) is the procedure of looking for matrices $A$ and $B$ such that

$$Data_{ik} = \sum_j A_{ij} B_{jk}$$

My data matrix is in fact 3D. I would like to fit the following model to my data

$$Data_{ikl} = \sum_j A_{ij} B_{jk} C_{jl}$$

It would be great to know if this model has a name and is already implemented somewhere (preferably Python)

Are you looking for something like this?

"Introduction to Tensor Decompositions and their
Applications in Machine Learning"
https://arxiv.org/pdf/1711.10781.pdf

I'm not sure if there are better papers for it, but from what I can tell tensor decomposition seems like what you're describing.


Just think of it as another feature transformation that grasp a set of sequential data group into a convoluted, compressed data. The text is a sequence of words(tokens), and each focus word would have different relationship to another words back and front of it. Conv1D help us to get nice and neat set of features that represent such relationship.

https://realpython.com/python-keras-text-classification/


Error rate of AdaBoost weak learner always bigger than 0.5? As far as i understand, weak learners of AdaBoost should never yield a error rate > 0.5

After training one, i only receive error rates above 0.5. How is that even possible? The AdaBoost Tree still gives quite good results, but all learners weights should be zero, so it should fail. Also the trees get worse from iteration to iteration

is it possible that my threshhold for the error rate instead is 0.9 (accuracy 0.1), as i got 10 classes and literature mostly focusses on binary cases?

from sklearn.ensemble import AdaBoostClassifier

adaboost_tree =  AdaBoostClassifier(
        DecisionTreeClassifier(max_depth=max_d),
        n_estimators=estimators,
        learning_rate=1, algorithm='SAMME')
adaboost_tree.fit(data_train, labels_train)

As far as I understand, weak learners of AdaBoost should never yield a error rate > 0.5

This is only true for binary classification problems.  The simplest possible classifier would pick the majority class, ensuring that accuracy is always >= 0.5
For a problem with $N$ classes, the worst error rate for a weak learner will be $\frac{N-1}{N}$.

  Is it possible that my threshold for the error rate instead is 0.9 (accuracy 0.1), as I have 10 classes and literature mostly focuses on binary cases?

Yep that's exactly right.  Your weak learners will have, at worst, an error rate of 0.9.  Unsurprisingly, most of your learners have a better error rate better than random guessing, which explains the good performance of your ensemble.

Edit:

  Also the trees get worse from iteration to iteration

This is a common feature in boosting algorithms.  Each iteration, training samples are reweighted such that the mistakes from previous iterations are given more importance.  The net effect is that the learners in later iterations are specialized for classifying the examples that earlier learners got wrong.  Since later learners are specialized for a few hard-to-classify examples, they will have worse performance on the entire dataset.


Regression equation for ordinal data I'm doing research where a part of the collected data is of Ordinal type. I will implement ANN with Logistic Regression function in the Activation function. What I have learnt from documents of other websites as well as an answer in https://datascience.stackexchange.com/, the target value is of ordinal type while the independent variables are of ratio or interval type. But my independent data is of Ordinal type and the target data will be label (say Like or Unlike).
How should I build a function for ANN if I'm not wrong in my understanding?

I think that what is influenced by the type of your independent variables is not the structure of your neural network but the encoding of your data. If your independent variables are ordinal, then you can map them to integer numbers or other ordered sets (and maybe normalize them to make them between 0 and 1).

If your target variable is a label, it is a classification problem. Therefore, the neurons in the last layer in your neural network should have an activation function optimized for classification problems, like sigmoid.

You also need a specific loss function, like binary crossentropy. There was a case study, however, that showed that for ordinal classification (when your target labels are ordinal), mean squared error works well.

You can take any activation functions for all hidden layers in your network as long as they are non-linear. The choice of a particular activation function will influence the rate of convergence of your learning algorithm but I don't see how a particular activation function in hidden layers may be preferable for a certain type of input variables, like if they are ordinal or not.


Is Excel sufficient for data science? I'm in the process of preparing to teach
an introductory course on data science using the R programming language.
My audience is undergraduate students majoring in business subjects.
A typical business undergrad does not have any computer programming experience,
but has taken a few classes which use Excel.

Personally, I am very comfortable with R (or other programming languages)
because I majored in computer science.
However, I have the feeling that many of my students
will feel wary of learning a programming language
because it may seem difficult to them.

I do have some familiarity with Excel,
and it is my belief that while Excel can be useful for simple data science,
it is necessary for students to learn
a serious programming language for data science (e.g., R or Python).
How do I convince myself and the students
that Excel is insufficient
for a serious business student studying data science,
and that it is necessary for them to learn some programming?

Edited in response to comment

Here are some of the topics that I will be covering:

Data processing and data cleaning
How to manipulate a data table,
e.g., select a subset of rows (filter),
add new variables (mutate),
sort rows by columns
SQL joins using the dplyr package
How to draw plots (scatter plots, bar plots, histograms etc.)
using the ggplot2 package
How to estimate and interpret statistical models such as
linear regression, logistic regression,
classification trees, and k-nearest neighbors

Because I don't know Excel very well,
I don't know whether all of these tasks can be done easily in Excel.

First of all check out this post. It has many reasons why Excel is inferior to other solutions, regarding data science tasks. Excel also can't handle large datasets (hundreds of thousands of records - not to mention anything in the vicinity of Big Data), image and sound data.

Excel is good for simple tasks concerning spreadsheets; it emphasizes more on presentation and ease of use, while having minimal support for actually analysing the data. Unless all you want to do is to calculate simple statistical measures (mean, average, etc) or building a very simple model (e.g. linear regression), Excel is inefficient. That being said, 99% the work a company has to deal with concerning data is simple enough to be manageable through Excel.

However Data Science mainly deals with regression, classification and complex models that excel isn't equipped to handle! If your students want to have a look at data science you need to teach them a tool that will be useful to them (R, python, etc.). These languages also have libraries with tons of built in models to "play with".

Another really huge reason I would go with the latter options is that they are open source. I personally feel that open source software should be preferred from an educational standpoint to proprietary solutions (this is also why I suggest python and R over Matlab)!


Track underlying observation when using GridSearchCV and make_scorer I'm doing a GridSearchCV, and I've defined a custom function (called custom_scorer below) to optimize for.  So the setup is like this:

gs = GridSearchCV(estimator=some_classifier,
                  param_grid=some_grid,
                  cv=5,  # for concreteness
                  scoring=make_scorer(custom_scorer))

gs.fit(training_data, training_y)

This is a binary classification.  So during the grid search, for each permutation of hyperparameters, the custom score value is computed on each of the 5 left-out folds after training on the other 4 folds.

custom_scorer is a scaler-valued function with 2 inputs: an array $y$ containing ground truths (i.e., 0's and 1's), and an array $y_{pred}$ containing predicted probabilities (of being 1, the "positive" class):

def custom_scorer(y, y_pred):
    """
    (1) y contains ground truths, but only for the left-out fold
    (2) Similarly, y_pred contains predicted probabilities, but only for the left-out fold
    (3) So y, y_pred is each of length ~len(training_y)/5
    """

    return scaler_value

But suppose the scaler_value returned by custom_scorer depends not only on $y$ and $y_{pred}$, but also knowledge of which observations were assigned to the left-out fold.  If I have only $y$ and $y_{pred}$ (again: the ground truths and predicted probabilities for the left-out fold, respectively) when the custom_scorer method is called, I don't know which rows belong to this fold.  I need a way to track which rows of training_data get assigned to the left-out fold at the point when custom_scorer is called, e.g. the indices of the rows.

Any ideas on the easiest way to do this?  Please let me know if clarification is needed.  Thank you!

Firstly; this is a really clear, well written question. Kudos!

I think the answer is to take the folding out of the CV and do this manually. You can generate the indices of the training and testing data using KFold().split(), and iterate over them in this manner:

from sklearn.model_selection import KFold, GridSearchCV
import pandas as pd
from sklearn.ensemble import RandomForestClassifier

iris = pd.read_csv('https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv')

kf = KFold(n_splits=3)

for train_idx, test_idx in kf.split(iris):
    print(train_idx, test_idx)

And what you'll get is three sets of 2 arrays, the first being the indices of the training samples for this fold and the second being the indices of the testing samples for this fold. Using that, you could manually cross-validate like this:

kf = KFold(n_splits=3)

x = iris.drop('species', axis=1)
y = iris.species

max_depths = [5, 10, 15]

scores = []

for i in range(len(max_depths)):
    rfc = RandomForestClassifier(max_depth=max_depths[i])
    scores.append({'max_depth':max_depths[i], 'scores':[]})
    for train_idx, test_idx in kf.split(iris):
        rfc.fit(x.iloc[train_idx], y.iloc[train_idx])
        scores[i]['scores'].append(custom_scorer(y.iloc[test_idx], rfc.predict(x.iloc[test_idx]), train_idx, test_idx)

So that's running once per value in max_depths, setting that parameter to the appropriate value in a RandomForestClassifier. It's then fitting 3 times, once per fold defined in KFold() and passing several things to the call to custom_scorer()...

y.iloc[test_idx] which is our y_true
rfc.predict(x.iloc[test_idx]) which is our y_pred
train_idx which is the indices of the samples of our training data
test_idx which is the indices of the samples of our testing data

Hope that helps. Out of interest: why do you need to know which observations are left out?


can you suggest best method for background / ambient noise removal from cough audio and keep only cough signal I'm performing background/ambient noise removal form cough signal and keep only cough audio signals using python.Can you suggest best noise filters to achieve this and keep only cough audio signal from .wav file.

First let's see some data: typical cough data for someone with pneumonia:

image source. This source paper also contains more information on processing cough data.

The simplest way I can think of would be to analyse and extract the average/base background noise of an many samples as you can, then simply subtract the background noise from the entire sound wave. This background noise is essentially the area outside of the start and end lines indicated on the image above.

Outline of steps will need to:

define the start and end of the cough itself (as per the image)
measure the average magnitude of the background: e.g. np.mean(np.abs(background))
subtract the average background noise from the sound clip

If you can plot the data again, you should notice that the lines outside of start and end are almost flat (but not exactly flat, we only subtracted the average background noise)

Links:

Have a look at this answer, which goes through similar steps in more detail
Here is a great list of libraries you can use to work with sound data in Python
Audacity (software package) has a lot of tools and is scriptable, using Python


Is it normal that a classifier always wrongly predicts the same samples? I'm trying to improve the accuracy of a classifier, a random forest one.
I built different models with the same hyperparameters but with different random seeds, trained them with the same training data, used the same test daata to make the predictions and compared the results. I discovered that 50% of the errors were always made on the same samples. Therefore, do these samples which are always wrongly predicted deserve a particular attention or is it kind of logic ?

I hope the question is clear enough.

What you are experiencing is pretty normal given your approach. A random forest is an ensemble of decision trees where models are trained in parallel using bootstrapped samples (a technique called bagging). Even though the decision trees are randomized (a thorough explanation of random_state can be found in this question), they still rely on an internal criterion (such as Gini index by default in RandomForestClassifier) to split the nodes and define decision paths. The fact that some of your samples are consistently being misclassified regardless of the random state is an indication of their objective difficulty when using this specific criterion.

  Therefore, do these samples which are always wrongly predicted deserve a particular attention or is it kind of logic ?

You are absolutely correct with your first thought. Paying particular attention to wrongly predicted samples in ensembles is the goal of a technique called boosting. The main idea is to train ensemble models in sequence, with new learners focusing on data points that the rest of the ensemble has previously failed on. A great overview of ensemble approaches is presented in this answer, which I highly recommend.

As far as boosting algorithms go, there are different flavors as well: you might want to try sklearn's AdaBoost and gradient tree boosting implementations, or XGBoost. These may help you finally defeat those pesky hard-to-classify samples, but be aware that bagging (your current model) has its own perks that boosting lacks.


Folds in Cross validation I am performing 10-folds cross-validation to evaluate the performances of a series of models (variable selection + regression) with R. I created manually the folds with this code.

At the moment I'm performing first variable selection, then hyperparameters tuning through cv, and finally testing the performance with RMSE and MAE for all the models, but I have a doubt.

Is it correct to "use" the same fold for all the models? Or should I do a separate cv for each model?

I recommend trying both (more than once), and exploring any differences. In my experience, using the same set of folds for all models or using a new set of folds for each model doesn't make any material difference. Post if you find different!

Regarding "I'm performing first variable selection, then hyperparameters tuning through cv", maybe watch https://www.youtube.com/watch?reload=9&v=S06JpVoNaA0 to be sure you are not introducing any bias.


Why is the reported loss different from the mean squared error calculated on the train data? Why the loss in this code is not equal to the mean squared error in the training data?
It should be equal because I set alpha =0 , therefore there is no regularization.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.neural_network import MLPRegressor
from sklearn.metrics import mean_squared_error

#
i = 1 #difficult index

X_train = np.arange(-2,2,0.1/i).reshape(-1,1)
y_train = 1+ np.sin(i*np.pi*X_train/4)

fig = plt.figure(figsize=(8,8))
ax = fig.add_axes([0,0,1,1])
ax.plot(X_train,y_train,'b*-')
ax.set_xlabel('X_train')
ax.set_ylabel('y_train')
ax.set_title('Function')
nn = MLPRegressor(
    hidden_layer_sizes=(1,),  activation='tanh', solver='sgd', alpha=0.000, batch_size='auto',
    learning_rate='constant', learning_rate_init=0.01, power_t=0.5, max_iter=1000, shuffle=True,
    random_state=0, tol=0.0001, verbose=True, warm_start=False, momentum=0.0, nesterovs_momentum=False,
    early_stopping=False, validation_fraction=0.1, beta_1=0.9, beta_2=0.999, epsilon=1e-08)

nn = nn.fit(X_train, y_train)

predict_train=nn.predict(X_train)

print('MSE training : {:.3f}'.format(mean_squared_error(y_train, predict_train)))

When I ran this code I found loss = 0.02061828 and the MSE in the training (MSE training) = 0.041

That's because the square loss is defined as 0.5*MSE.

See definition here:


How multi-scale CNN selects final output map I read a few days ago about multi-scale CNN (OverFeat method) which you can access to presentation via this link. They performed CNN on different scales of an image and then combine all output maps. They said inside of that presentation:

Classification    performed   at  6   scales  at  test time,  but only    1 scale at  run time    .

So my question is: If we use 6 different scales of CNN architecture, then we have different convolution layers in every scale (I guess so). So how in OverFeat, they use 1 scale in run time? If we use a specific scale, then how can we access other feature extractors of different scales? And I see in the article, they combine feature maps of different scales but I can't figure out how this process performed.

Think of this as varied filter size and varied filter values. It will extract different representation (or say capture different part of the image), and then stack them to get a bigger feature vector. Then, you do the featurisation .Also, check for dilated CNNs used for NLP. They are based on somewhat similar concepts.


can I use t-sne or PCA to reduce number of classes? I wanted to know if I can use t-sne or PCA to reduce the number of classes depending on the similarity between them. For example, if I have 100  classes of 100 different animals and would like to put all the cats in a group and all the dogs in a group etc. (to get few groups of these 100 classes).

No. t-Distributed Stochastic Neighbor Embedding (t-SNE) and Principal Component Analysis (PCA) are dimension reduction techniques, aka fewer columns of a tidy dataframe.

Clustering will reduce the number of observations, aka fewer rows of a tidy dataframe. In particular, you might be looking for hierarchical clustering.


How to Maximize recall for Minority class? I have a dataset with 4.7k records and 60 features. 1558 records of indication label 1 and 3554 records indicating label 0. Am solving a binary classification problem. I used SMOTE on training data to balance the data

I applied various feature selection algorithms

1) RFECV - returns optimal 30-40 features based on model that I use

2) SelectFromModel - returns top n features based on what I keyed in. I tried n=15 and n=10

3) Kbest - returns top n features based on what I keyed in. I tried n=15 and n=10

My questions

1) The output from Feature selection algorithm 1,2,3 provides feature which are little bi different. I mean from algo 2 and algo 3, around 10 features are same but remaining 5 are different. What does this imply? Is it normal to see such differences? Or am I making any mistakes here?

2) I tried resampling the minority class using SMOTE and applied Gridsearchcv but still I don't see a good recall value for minority class(which is oversampled now). Please find the screenshot below for metrics

3) Does Gradientboosting, xtreamegradientboosting,catboost algorithms doesn't require us to select features? Are they intelligent enough to pick features and help us increase the accuracy? Should I pass the data (all features) as is

Can you help me as to how can I increase the metric values above 90 for both classes?

Currently am trying only with 15-20 features because certain features from feature selection output were correlated, so I removed them and using only 15-20 features.

Can you offer some suggestions please? CATboost is running for a long time.

Here are my two cents.

1) While there are some things that you may want to do differently (see part 2), what you observe is normal. If you had a trivial problem with a simple landscape, all of the methods would converge on exactly one subset of features. Feature selection in the real world, however, is rarely a trivial problem (as long as you are looking for "true" optimality and not a "good enough" solution). You are facing a search space of size $2^{60}$, which is impossible to assess in its entirety, so navigation methods will vary. RFECV is a top-down approach that attempts to shed as many weak features as possible based on the CV scores, SelectFromModel relies on estimator's internal objectives to do the selection, while SelectKBest in its default form is a simple filter that computes ANOVA F-values without considering feature interactions. Since the criteria are so different, you are bound to end up with different feature subsets -- each method operates on a landscape that is "distorted" by its objective function. The fact that the subsets you have found intersect suggests that the common features possess some sort of optimality (when all used approaches are considered). While you are at it, consider sequential feature selection methods, especially the floating approaches (both bottom-up and top-down), as they can be a great deal more effective than plain RFE / KBest.

2) You are using SMOTE to balance the training data. From your problem statement it seems that you are introducing synthetic data points before feature selection. If this is the case, then please consider the implications: for RFECV (if you do something like rfecv.fit(X_train, y_train)), the validation sample is not drawn from the original distribution, but from the one already affected by synthetic data, so the CV scores that drive the selection are not based on the true qualities of the features. This discrepancy may be too small to actually affect anything, but the fact remains. In your case, the class balance is 70/30, which is not that bad. Do you actually observe any increase in performance after balancing? As far as metrics go, the area under Precision-Recall curve is stable irrespective of the balance.

3) Decision trees (extensively used in many flavors, including XGBoost and CatBoost that you mentioned) have feature selection embedded into the process of learning. This means you can just plug in full featured data into the learner and have it figure out the most important features completely on its own. It's very effective and usually sufficient. However, in order to have variety of opinions within the ensemble, forest methods (most of them, anyway) will give all features an equal chance of being used, so it's very likely that you will end up with a situation where all features have some importance -- you will still have to decide which ones to keep. Weak features with noise that correlates with your labels by pure chance can affect your model's ability to generalize and they can even trick you into keeping them as useful. Also, when your sample size is limited (and it always is), increasing the number of features does not help the confidence of your assessments, either. Without enough observations, how can you be sure that the label is what it is because of this one feature and not ten others? So delegating it all to the forest without any preliminary or followup steps is not necessarily the best course of action.

The answer to the general question of the post: How to Maximize recall for Minority class?

This is quite straightforward. Define recall on minority class as your objective, i.e. let it be the output of the scoring function that you supply to your feature selection algorithm. For sklearn's feature selection methods it's the score_func parameter, for mlextend's SFS it's scoring. The resulting feature subsets are likely to contain something of interest.

Only after performing a thorough search for optimal features, selecting the best kind of model for your problem and having tuned its hyperparameters, if you still for whatever reason want to push your precision/recall further, you may want to look into stacking -- this will squeeze out the last bits of performance juice at the expense of model complexity. But the chances are that it's really not worth it.


Measuring Value in Data Science? Why does it seem that it's difficult to find out how people in data science create measurable value?  All I find on the internet are buzzwords like data cleaning, visualization, and writing about the data.  This is equivalent of describing a landscaper as trimming trees and grass or saying an investment banker spreads M&A.  That is insufficient.  Let my type out an insufficient answer:

  A data scientist uses database querying languages like SQL as well as statistical programming languages like R to design experiments with data and visualizing such experimental conclusions to decision-makers in the hopes of making better decisions and thus creating value for a company/cause.

How is this insufficient?  Well, how do we know that data scientists' recommendations actually create value?  If we can't measure the value of our analyses, then how can we determine if hundreds of hours of learning/working in the subject is meaningful?  Extreme example:

  George is a data scientist working for Company X.  After hundreds of hours of data cleaning/experimentation, he concludes Decision A will benefit Company X.  George convinces a Product Manager to apply Decision A that ultimately increases the revenues of Company X by $0.01

How do we know that several years of work may not amount to basically nothing?  Let me give you a depressing and real/plausible example:

  Instead of becoming a data scientist, George becomes a portfolio manager, managing investment portfolios for clients.  After 20 years of management, George has a track record of -1% per annum, compared to an S&P 500 return of 8%.

In the above example, George is a useless person and has lost value for people.  How do data scientists know that they aren't destroying value, and if we can't figure that out, what are the steps to avoid value destruction and create the most value for companies?

I read your question as:

  tl;dr. How do you know that a data scientist adds value to a company?

The job of a data scientist is to pick out the best data set, scrub it, fit a statistical model. And most importantly, a data scientist needs to know the problem statement and why is the analysis being done.

For example, consider a DS team in an e-commerce company. Their primary (let's assume) is to build a highly effective recommender system, which needs to adapt itself to a lot of variables, like the user behaviour, the session-based behaviour, user segments, etc.

So, if the recommender system is still not being able to get the company a considerable CTR or upsells/cross-sells, then maybe the team is not adding proper value to the company.

So, a data scientist needs to have a keen knowledge of the business value which an experiment is expected to add to the company. The decisions made from the experiments should ultimately add value to a company (monetary or otherwise). If not, then just like any other employee, it means that the person have failed in the job.


What's the best way to plot a bar graph with large numeric difference in values? I'm graphing the value of the Ruble against the US Dollar in the 1990s. There was hyper-inflation from 1992-1997 (where the "y" values expanded from 125 to 6000). On Jan 1st 1998, the government revalued the money to 5. Since then the number has trended upward but has only reached 70 or so.

This creates a problem in my bar or line graph. You can't understand the low values after the stabilization. Any suggestions to help with this discrepancy when viewing the whole dataset in one graph?

Plotting using matplotlib

A log scale will make the difference before and after revaluation look less extreme, and a log transformation is usually pretty defensible for things measured in units of currency (especially when there are no 0-value entries). Other transformations exist, but since this is mostly for aesthetics I don't think that you'd need much that a log transformation couldn't offer.

Fiddling with multiple vertical axes and continuity breaks may help, but at an almost-guaranteed cost of making the graph very difficult to view and interpret correctly. If you need to show both periods, I would probably log transform.

You will know your needs far better than I do, but I do question a little bit how valuable seeing the before and after this way is. The revaluation was an arbitrary decision introduced at an arbitrary time, and behavior of the currency before the revaluation is probably not very informative of anything that happened afterwards (and vice-versa). It's far from clear to me that there is much value in showing both at once like this, at least from my naïve look at a single graph.

The main reason I can think of to show both at once this way is to display how extreme the inflation was, in which case the scale differences would be a feature and not a bug.


In Tensorflow, What kind of neural network should I use? I am doing Tensorflow tutorial, getting what TF is. But I am confused about what neural network should I use in my work. I am looking at Single Layer Neural Network, CNN, RNN, and LSTM RNN.

-----------------------What I'm going to do is...-----------------------

There is a sensor which measures something and represents the result in 2 boolean ways. Here, they are Blue and Red, like this:

the sensor gives result values every 5minutes. If we pile up the values for each color, we can see some patterns:

number inside each circle represents the sequence of result values given from sensor.
(for example, 107 was given right after 106) when you see from 122 to 138, you can see decalcomanie-like pattern.

I want to predict the next result value, before the sensor imparts the result, with probability.
Machine has to know what the next will be, based on patterns from past results.

I may do supervised learning using past results. But I'm not sure which neural network or method is suitable. Thinking that this work needs pattern using past results (have to see context), and memorize past results, maybe LSTM RNN (long-short term memory recurrent neural network) would be suitable one.

Could you tell me which one is suitable for this work?

Sure, you can use an RNN. I would create two features for the past $k$ run lengths, as well as the length of the current run; e.g., just before t=150, the current run would be length 2 (red), and previous three runs would be (1,1,1) for red and (1,1,5) for blue. The intuition here is that the run lengths seem to follow some sort of exponential distribution, and you want to help the model estimate the scale parameter by feeding it samples of the length. You could additionally encode the past k events as a bit string, with 1 representing red and 0 representing blue. You have a classification problem, so you should use classification loss like the cross-entropy, and a softmax output layer to get your probabilities.


How can discard "peaks" values from a chart? I have this chart

where you can see that some points are far away from the "trend" of the curve. The chart represents $5,000$ values where each value corresponds to a specific hour of a day, month and year. In this case, it goes from  01/01/18 until 05/08/18 (non-american format) and as you can see at the first lower peak, it was caused because at 12.00 the cost was $20$ USD but at 1.00 AM it went up to $51$ USD.

I've read that some people just delete those 'peaks' values but I don't know if there's a better solution. This graph shows almost a year but I have in mind on graphing $10$ years of this prices so the situation could be worse. Do you know a better strategy? I'm thinking maybe in calculating average costs for every $6$ hours but it feels like a cheap solution to me.

I would say you have two options to choose from:

Aggregate your data
Use a low-pass-filter

Aggregating

Aggregating your data over long time periods is not cheap (whatever that means). It is a common way of displaying trends over long periods of time.

Take a look at how Bloomberg displays the exchange rate of EUR over USD.

You can filter by time period, and if you select 1 day, you see the hourly rate:

or you can filter by month, and see the rate aggregated by day. For them, the last rate is the important one, so that is the value you see. For you, however, will depend on your use case. You can choose to pick one of the following: mean, median, mode, max, min, first, last or anything else that makes sense for you.

Signal processing filtering

If you want to take a more mathematical approach to your problem, you could use a signal filtering technique to reduce high-frequencies from your signal.

What you need is a low-pass filter.

If you are using Excel, you certainly can do this.
However, I would recommend you do that in some programing language like Matlab or Python, that contain easier to use signal processing packages.


best approach for CNN training with multiple subcategories and one category I need to classify pictures into 2 categories: approved and rejected.
Rejected category has different type of images which are not allowed (subcategories), for example nude or gore or anime etc.

What approach in training the CNN will be better or they are equal for the CNN:

To have output final layer with 2 categories (neurons) -
approved and rejected
To have a lot of neurons in the final layer, one for each "subcategory" (and feed NN with corresponding labels), and later when doing actual inference just manually
aggregate them to rejected category?

I'm not sure if using the first approach CNN can easily apply OR operation for such a different subcategories, thus that training will be effective. Just feel that multi-categories approach is easier for NN. Is there any approved science behind it?

Okay, so I used the custom loss function to optimize both ways simultaneously:
 - binary rejected vs approved
 - multiple categories
And using the coefficient argument to balance between two losses.
Thats it


How to remove background (watermark) logo from image I have been scratching my head for a while. What I have is a scanned PDF document with text and water marked logo at the back as in the below image.

I want to do OCR over this, which becomes very difficult because of the logo. All the ratchet I've done so far is for coloured images where they can find contrast difference. I've hit a wall when solving the same for an B&W image as shown. Would love any help/suggestions for an approach/method to achieve what I am looking for.

Take a look at the python cv2 module. It has functions that should enable you to remove the watermark. If you have a separate image of the watermark and it is always in the same coordinate location in each image you should be able to subtract it from the images


Should kernel size always be a prime number? Should kernel size always be a prime number? E.g. (3,3) (5,5) (7,7).
While tinkering with sklearn.preprocessing.KernelCenterer(), I noticed that I could only get it to work on a 3,3 array not 2,3 not 6,1.
Then I realized that in a square that is prime-by-prime, there is always a center - hence Centerer.
[ ][ ][ ]
[ ][X][ ]
[ ][ ][ ]

[ ][ ][ ][ ][ ]
[ ][ ][ ][ ][ ]
[ ][ ][X][ ][ ]
[ ][ ][ ][ ][ ]
[ ][ ][ ][ ][ ]

Odd numbers, usually, but not necessarily. This can depend on the problem at hand. The odd length of the side of the kernel is used in order to emphasize the center of the kernel. For a more detailed discussion of the geometry of kernels and convnets have a look at this thesis.


Collinearity between continuous and categorical variable I have a medical dataset with features age, bmi, sex, gender, # of children, region, charges, smoker. Here smoker, gender, sex and region are categorical variables and others are numerical features. How do I check for collinearity between these in my dataset?

You could just regress against any given variable.
You could also generate a matrix of correlation metrics. Depending on the variable type, you would need a different metric. Here are some common ones:

Numeric-Numeric: Pearson's Correlation Coefficient

For rank correlation, you can use Spearman's Rank Correlation Coefficient

Nominal-Nominal: Cramér's V

For two categories, this is equivalent to the Pearson Phi Coefficient

Numeric-Nominal: t-test for 2 variables or ANOVA for more than 2

You can also just regress against them. For example, logistic regression between age and sex could suffice.


Skewed multi-class data I have a dataset which contains ~100,000 samples of 50 classes. I have been using SVM with an RBF kernel to train and predict new data. The problem though is the dataset is skewed towards different classes.

For example, Class 1 - 30 (~3% each), Class 31 - 45 (~0.6% each), Class 46 - 50 (~0.2% each)

I see that the model tends to very rarely predict the classes which occur less frequent in the training set, even though the test set has the same class distribution as the training set.

I am aware that there are technique such as 'undersampling' where the majority class is scaled down to the minor class. However, is this applicable here where there are so many different classes? Are there other methods to help handle this case?

I would suggest you to use libsvm, which already has adjustable class weights implemented in it. Rather than replicating the training samples, one modifies the C parameter for different classes in the SVM optimization. For example if your data has 2 classes, and the first class is only 10% of the data, you would choose class weights to be 10 and 1 for class 1 and 2 respectively. Therefore, margin violations of the first class would cost 10 times more than the margin violations for second class, and per-class accuracies would be more balanced.


Gas consumption outliers detection - Neural network project. Bad results I tried to detect outliers in the energy gas consumption of some dutch buildings, building a neural network model. I have very bad results, but I can't find the reason.

I am not an expert so I would like to ask you what I can improve and what I'm doing wrong. This is the complete description: https://github.com/denadai2/Gas-consumption-outliers.

The neural network is a FeedFoward Network with Back Propagation. As described here I splitted the dataset in a "small" dataset of 41'000 rows, 9 features and I tried to add more features.

I trained the networks but the results have 14.14 RMSE, so it can't predict so well the gas consumptions, consecutively I can't run a good outlier detection mechanism. I see that in some papers that even if they predict daily or hourly consumption in the electric power, they have errors like MSE = 0.01.

What can I improve? What am I doing wrong? Can you have a look of my description?

Just an idea - your data is highly seasonal: daily and weekly cycles are quite perceptible. So first of all, try to decompose your variables (gas and electricity consumption, temperature, and solar radiation). Here is a nice tutorial on time series decomposition for R.

After obtaining trend and seasonal components, the most interesting part begins. It's just an assumption, but I think, gas and electricity consumption variables would be quite predictable by means of time series analysis (e.g., ARIMA model). From my point of view, the most exiting part here is to try to predict residuals after decomposition, using available data (temperature anomalies, solar radiation, wind speed). I suppose, these residuals would be outliers, you are looking for. Hope, you will find this useful.


How many features does (Random Forest) need for the trees? How can I determine how many features Random Forest needs to create a tree?

The (maximum) number of features is a hyperparameter, i.e. it depends on what you set it to or what the default of the implementation you are using is. In "Introduction to Machine Learning with Python" by Mueller and Guido the authors recommend the following:

  As described earlier, max_features determines how random each tree is,
  and a smaller max_features reduces overfitting. In general, it’s a
  good rule of thumb to use the default values:
  max_features=sqrt(n_features) for classification and max_features=log2(n_features) for regression.

Which is, for example, in line with the default in scikit learn. This what the parameter looks like for classification in scikit learn:

  max_features: int, float, string or None, optional (default=”auto”)

  The number of features to consider when looking for the best split:

  If int, then consider max_features features at each split.
  If float, then max_features is a fraction and int(max_features *
  n_features) features are considered at each split.
  If “auto”, then max_features=sqrt(n_features).
  If “sqrt”, then max_features=sqrt(n_features) (same as “auto”).
  If “log2”, then max_features=log2(n_features).
  If None, then max_features=n_features.

  Note: the search for a split does not stop until at least one valid
  partition of the node samples is found, even if it requires to
  effectively inspect more than max_features features.

Source: scikit learn documentation


Identifying and Accounting for trend/seasonality in Predictor Variables I'm currently working with a dataset that has been collected over several years, and I suspect my predictor variables are changing over time for their predictive power.

I could go back year by year and run the data the same way each time to see how efficient each predictor is, then trend the predictive power over time manually. There has to be a better way.

Can anyone point me towards the technique I should cram on?

Using time as an explanatory variable may be a good start. Then, depending on your model you may want to add interaction effects between time and your other variables. This will take into account change over time.

However, it will be a bit difficult to link evolution of the model to the evolution of some explanatory variable. If you know what variable evolve over time, you may just want to add time differentiated variables ($X_n - X_{n-1}$). This will give you a clearer view on what evolution is impacting the model output. Depending on your problem you might want to consider different time lags.


Why does my Keras model learn to recognize the background? I'm trying to train this Keras implementation of Deeplabv3+ on Pascal VOC2012, using the pretrained model (which was also trained on that dataset).

I got weird results with the accuracy quickly converging to 1.0:

5/5 [==============================] - 182s 36s/step - loss: 26864.4418 - acc: 0.7669 - val_loss: 19385.8555 - val_acc: 0.4818
Epoch 2/3
5/5 [==============================] - 77s 15s/step - loss: 42117.3555 - acc: 0.9815 - val_loss: 69088.5469 - val_acc: 0.9948
Epoch 3/3
5/5 [==============================] - 78s 16s/step - loss: 45300.6992 - acc: 1.0000 - val_loss: 44569.9414 - val_acc: 1.0000

Testing the model also gives 100% accuracy.

I decided to plot predictions on the same set of random images before and after training, and found that the model is encouraged to say everything is just background (that's the 1st class in Pascal VOC2012).

I'm quite new to deep learning and would need help to figure out where this could come from.

I thought that perhaps it could be my loss function, which I defined as:

def image_categorical_cross_entropy(y_true, y_pred):
    """
    :param y_true: tensor of shape (batch_size, height, width) representing the ground truth.
    :param y_pred: tensor of shape (batch_size, height, width) representing the prediction.
    :return: The mean cross-entropy on softmaxed tensors.
    """
    return tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(logits=y_pred, labels=y_true))

I am a bit uncertain on whether my tensors have the right shape. I'm using TF's dataset API to load .tfrecord files, and my annotation tensor is of shape (batch_size, height, width). Would (batch_size, height, width, 21) be what's needed? Other errors from inside the model arise when I try to separate the annotation image into a tensor containing 21 images (one for each class):

tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [12,512,512,21] vs. [12,512,512]
         [[Node: metrics/acc/Equal = Equal[T=DT_INT64, _device="/job:localhost/replica:0/task:0/device:GPU:0"](metrics/acc/ArgMax, metrics/acc/ArgMax_1)]]
         [[Node: training/Adam/gradients/bilinear_upsampling_2_1/concat_grad/Slice_1/_13277 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:1", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_62151_training/Adam/gradients/bilinear_upsampling_2_1/concat_grad/Slice_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:1"]()]]

Thank you for your help!

Your model is overfitting. Each epoch only has 5 images. The model is "memorizing" the answer for each image.

In order to minimize the chance of overfitting, increase the number of images. There should be several thousand example images for each category of object.


To detect unauthorized access using outlier detection I am working on project where my task is to find unauthorized access using any machine learning technique.

Let me clear my problem definition.

UserA access website using chrome browser from windows PC.
UserB access website using Internet Explorer from windows PC.
UserA never use safari to access website but all of a sudden it uses safari to access website from mac system.

Point 1 and 2 are normal logins but point 3 may be used by attackers because it is not matching with UserA behavior.

Thus i am am trying to implement an unsupervised model in python to first learn normal behaviors of all users for website access with other features such as -

username
country
service_key (unique identifier)
system
app_type (client, server, enterprise etc.)

Below I am sharing sample data-set of mine.

+==========+=========+=============+=========+==========+
| username | country | service_key | system  | app_type |
+==========+=========+=============+=========+==========+
| userA    | india   | e08fe2d     | windows | 2        |
| userA    | india   | e08fe2d     | android | 2        |
| usreB    | china   | bb15d36     | windows | 3        |
| userB    | india   | bb15d36     | windows | 3        |
| userB    | russia  | bb15d36     | mac     | 3        |

| userA    | usa     | e08fxxx     | mac     | 3        |
| userB    | china   | bb1xxxx     | ios     | 1        |
+==========+=========+=============+=========+==========+

Above 5 records are manually analyzed and marked as normal logs and collected from different users system. But the last 2 records are abnormal and should be detected as outlier because their is no match in features. Here to be note is that logins can have different country and system but it cannot have different username, service_key and app_type.

Can anyone suggest me to model this or can share blog or samples in python?

This question is quite broad. I'll try to set you on the right path, more so than providing a truly complete answer.

Theoretical background

As others have mentioned, the task you're trying to do is usually known as anomaly detection, also known as novelty detection.

There's many possible ways to approach this kind of task, depending on the assumptions you are willing to make about your data.

Before we start exploring alternatives, it's important to mention here that all your variables  are categorical (country, service_key, system, app_type). This is important, since usually algorithms usually handle either continuous variables or discrete variables, and categorical ones fall into the latter type.

There's some tricks you can do, however, to handle them interchangeably. One-hot encoding is simple and popular, and enables you to transform a categorical variable into N binary variables. For some classes of algorithms, you can then use these binary variables as if they were continuous. You can even mix binary and truly continuous variables.

Now, there's usually two families of methods to approach this sort of problem: parametric vs non-parametric.

Parametric methods require that you do assumptions on the underlying probability distributions of your data. You can then estimate these parameters using your data.

Non-parametric methods, on the other hand, are simpler to use, as they don't require assuming a underlying distribution. They might also be less powerful / exact, especially if you don't have a lot of data and your data very closely follows a known probability distribution.

Practical advise

If you're willing to do some statistical work, look into bayesian inference.

For a easier non-parametric method, look into into One-Class SVM. Here is a code example using python and scikit-learn and more practical information on anomaly detection

But, honestly, you're giving us a important clue here:

  Here to be note is that logins can have different country and system
  but it cannot have different username, service_key and app_type.

If you already know all this, you might want to completely skip the complicated parts. Just compare those values directly, summarize them into a distance measure (you can give different weights to service_key and app_type, for example) and see if that's good enough


How to create feature representation? Let's say I've a dataset with 800 rows(40 entries for each of 20 users). The entries are user session log ( columns are - browser, os, time, date etc for a specific session).

Now each user has unique id (1-20). Let's say user_id=1 is special one and I need to detect it whenever new data comes.

So for new data I need to predict whether that session is of user_id=1 or not.

My question is How to do that ?

One way I thought is make a feature representation for each of the 20 users, and whenever new data comes, take the distance of the data from each of feature and see the minimum distance.

But the problem is when I make a unique feature representation for each user, how to deal with columns like browser, os - because a user can have used multiple browsers in all those 40 sessions ?

If I understand correctly, you want to create features? there are a few ways to do this. I will first talk about Label Encoding and Hot Label Encoding(link is the first I found on Google). This depends on the model, I will explain below.

Since we want to use categorical data in a model, most models behave better with numerical data. So let's convert categorical -> numeric:

import pandas as pd
from sklearn.preprocessing import LabelEncoder

df = pd.DataFrame({'user':['adam','adam','susie','jane'],
                   'browser':['chrome','firefox','chrome','opera']})

   browser   user
0   chrome   adam
1  firefox   adam
2   chrome  susie
3    opera   jane

label_encoder = LabelEncoder()
# convert browser to numeric
df['browser_encoded'] = label_encoder.fit_transform(df['browser'])

   browser   user  browser_encoded
0   chrome   adam                0
1  firefox   adam                1
2   chrome  susie                0
3    opera   jane                2

In the "real world" you would drop browser column as its now encoded. I left it to show you what's happening.

Now notice how our data is now numeric. But now there is a problem. If you're using something like multiple linear regression, you've just assigned a higher weight to the Opera browser since it's a higher number. (2>1 and 2>0). But since this is categorical, and Opera isn't "weightier" than other browsers in our model, we need to solve this. This is where One Hot Encoding comes in to play.

pd.concat([df,pd.get_dummies(df['browser'])], axis=1)

   browser   user  browser_encoded  chrome  firefox  opera
0   chrome   adam                0       1        0      0
1  firefox   adam                1       0        1      0
2   chrome  susie                0       1        0      0
3    opera   jane                2       0        0      1

Now we have our data in columns. the 1 value appears when someone has used that browser for that record. See the first record? where chrome column is 1, because adam used 'chrome' in the browser column?.

There is one last thing when it comes to label encoding! You don't want to fall in the dummy variable trap with certain models. The basic logic is: If you know the browser is not Chrome or Firefox, it must be Opera. If you know the browser is Chrome, you know that it is not Firefox or Opera. So you only need N-1. If you use all the columns, your model may not perform well. You need to drop only one of the columns. you can do that with drop_first=True

pd.concat([df,pd.get_dummies(df['browser'], drop_first=True)], axis=1)
   browser   user  browser_encoded  firefox  opera
0   chrome   adam                0        0      0
1  firefox   adam                1        1      0
2   chrome  susie                0        0      0
3    opera   jane                2        0      1

Another way to create features is to make them yourself. You mentioned that a user can use more than one browser. I'm making something up for the sake of a simple example - but maybe the number of browsers they use can be a feature? You can create one like so:

df['num_of_browsers_used'] = df.groupby('user', as_index=False).transform('nunique')['browser']
    browser user    browser_encoded num_of_browsers_used
0   chrome  adam    0   2
1   firefox adam    1   2
2   chrome  susie   0   1
3   opera   jane    2   1


Problem in Recommendation for categorical data? I have been building a recommendation model to recommend certain questions in an interaction platform to users to help each other.
I have calculated an affinity score between categories to find which top categories are to be recommended. But each category has questions by users in itself. The amount of the questions increases with every new post a user posts in a certain category. Now how can I choose which of these questions to recommend when I have chosen the category through my affinity score ? Do I make it random ? Do I display the questions which come first in the data base ? Or is there any better alternative ?

Welcome to the site, I'll propose few alternatives below in increasing order of complexities.

Simple sorting criteria: Think what the user wants to see and create a score for sorting questions. Ordering of question based on some criteria like number of answers or number of views it has already received. Show top N based on score.
Derived sorting criteria: Use a combination of factors like number of answers, freshness of question, popularity of question based on number of views to create a derived score. Show top N based on score.
Adding Discovery to (1-2): The options above will penalize new questions and run into typical explore Vs. exploit trade-off in any recommendation system. You can alleviate that by adding a random bump to score of new questions so that they get a chance to participate.
Jointly learning categories and questions: You can setup your recommendation algorithm to actually work at question ID level and use categories as side information. Check this worked out example of LightFM library in Python. Coincidentally, it uses stackoverflow questions and categories as examples.

The first 3 options are easy to implement but biased. Fourth one will need some data transformations and I'm not sure if your objective is to model directly at question ID level.


Decision tree with final decision being a linear regression Question:
I want to implement a decision tree with each leaf being a linear regression, does such a model exist (preferable in sklearn)?

Example case 1:

Mockup data is generated using the formula:

y = int(x) + x * 1.5

Which looks like:

I want to solve this using a decision tree where the final decision results in a linear formula. Something like:

0 <= x < 1 -> y = 0 + 1.5 * x
1 <= x < 2 -> y = 1 + 1.5 * x
2 <= x < 3 -> y = 2 + 1.5 * x
etc.

Which I figured could best be done using a decision tree. I've done some googling and I thought the DecisionTreeRegressor from sklearn.tree could work, but that results in points being assigned a constant value in a range, as shown below:

Code:

import matplotlib.pyplot as plt
import numpy as np
from sklearn.tree import DecisionTreeRegressor

x = np.linspace(0, 5, 100)
y = np.array([int(i) for i in x]) + x * 1.5

x_train = np.linspace(0, 5, 10)
y_train = np.array([int(i) for i in x_train]) + x_train * 1.5

clf = DecisionTreeRegressor()
clf.fit(x_train.reshape((len(x_train), 1)), y_train.reshape((len(x_train), 1)))

y_result = clf.predict(x.reshape(len(x), 1))
plt.plot(x, y, label='actual results')
plt.plot(x, y_result, label='model predicts')
plt.legend()
plt.show()

Example case 2:
Instead of one input, there are two inputs: x1 and x2, output is computed by:

x1 = 0 -> y = 1 * x2
x1 = 1 -> y = 3 * x2 + 5
x1 = 6 -> y = -1 * x2 -4
else -> y = x2 * 20 - 100

Code:

import matplotlib.pyplot as plt
import random

def get_y(x1, x2):
    if x1 == 0:
        return x2
    if x1 == 1:
        return 3 * x2 + 5
    if x1 == 6:
        return - x2 - 4
    return x2 * 20 - 100

X_0 = [(0, random.random()) for _ in range(100)]
x2_0 = [i[1] for i in X_0]
y_0 = [get_y(i[0], i[1]) for i in X_0]
X_1 = [(1, random.random()) for _ in range(100)]
x2_1 = [i[1] for i in X_1]
y_1 = [get_y(i[0], i[1]) for i in X_1]
X_2 = [(6, random.random()) for _ in range(100)]
x2_2 = [i[1] for i in X_2]
y_2 = [get_y(i[0], i[1]) for i in X_2]
X_3 = [(random.randint(10, 100), random.random()) for _ in range(100)]
x2_3 = [i[1] for i in X_3]
y_3 = [get_y(i[0], i[1]) for i in X_3]
plt.scatter(x2_0, y_0, label='x1 = 0')
plt.scatter(x2_1, y_1, label='x1 = 1')
plt.scatter(x2_2, y_2, label='x1 = 6')
plt.scatter(x2_3, y_3, label='x1 not 0, 1 or 6')
plt.grid()
plt.xlabel('x2')
plt.ylabel('y')
plt.legend()
plt.show()

So my question is: does a decision tree with each leaf being a linear regression, exist?

I think the easiest way to do this would be to have a decision tree where the final decision results in a linear formula.

Setting aside whether this is actually easiest/best, this type of model does exist, usually called "model-based recursive partitioning".  See e.g. https://stats.stackexchange.com/q/78563/232706
There are several packages in R for this: partyfit (and the older simpler party), mob, Cubist; unfortunately, there don't seem to be any in Python yet.  Here's a discussion on including it in sklearn, from mid-2018.


How to calculate joint feature contribution for XGBoost Classifier in python? I referred to this beautiful document to research about joint feature contibutions. But this works only for RandomForest algorithms because of treeinterpreter (does not work with xgboost). Is there a similar way out for XGBoost as well?
Basically what I want to achieve is to find out the joint contributions of all the combination of features towards the prediction. For example if I have a, b and c as my features, I want to know what is the effect of ab, bc and ca towards the prediction result. It is very similar to shap and lime, but for combinations of features.

I did some research and learnt about xgbfir package. It gives the joint contributions into an excel file. You can set the level of interaction with this. I wrote some code around it to generate a plot that solves the purpose.

If the package is not installed

pip install xgbfir

After the installation:

import xgbfir
from matplotlib import pyplot as plt

xgbfir.saveXgbFI(model, feature_names=X.columns, OutputXlsxFile='FI.xlsx')

joint_contrib = pd.read_excel('FI.xlsx')

xls = pd.ExcelFile('FI.xlsx')
df1 = pd.read_excel(xls, 'Interaction Depth 0')
df2 = pd.read_excel(xls, 'Interaction Depth 1')
df3 = pd.read_excel(xls, 'Interaction Depth 2')

frames = [df1, df2, df3]
joint_contrib = pd.concat(frames)

joint_contrib=joint_contrib.sort_values(by='Gain', ascending=True)
joint_contrib=joint_contrib.head(20)

height = joint_contrib['Gain']
bars = joint_contrib['Interaction']
y_pos = np.arange(len(bars))

plt.barh(y_pos, height)
plt.yticks(y_pos, bars)
plt.show()

This will give the top 20 feature interactions in terms of gain.

Thanks to Philip Cho who introduced me to xgbfir.

Follow the link for more information regarding xgbfir