Best place for Bengin files for training a classifier? Trying to train a Deep neural network classifier, and I got a big data set from virusshare.com, however I also need some normal (non-malware) files. Format must be a PE (x64 or x86).

Does anyone know where to get such a dataset? Or would I have to create my own?

You can use VirusTotal. VirusTotal got a Private API where you can query files (for example only those that has 0 positives) and then download the files, you need to contact them to understand how much it cost.

If you don't need the binary itself and you are using different information from the files structure you can use endgame's dataset


Is there any way to create column based on some previous column in PANDAS dataframe? Given: I have Pandas Dataframe as shown below

| Employee_ID | Manager_ID |
|:-----------:|:----------:|
| E068        | E067       |
| E071        | E067       |
| E229        | E069       |
| E248        | E144       |
| E226        | E223       |
| E236        | E241       |
| E066        | E001       |
| E067        | E001       |
| E144        | E001       |
| E223        | E001       |
| E069        | E066       |

Problem Statement:

This problem is to identify the Head of Manager by using Employee and their Manager data.

About:

We have an Employee ID and their Manager ID. Please note that Manager ID are from Employee ID. Since each manager has one Manager above their level.

STEPS:

First, we'll take all UNIQUE ID in Manager ID column.
Then for each ID from Manager ID column, we will look for their respective Manager ID(Manager)
Then we will create a new column say Level 1 we will put manager for each Manager ID on their respective cell.
Similarly, we will repeat the above 3 processes again till there is no Manager ID for that particular ID.
This way we can identify the Head of Manager.

I am able to solve the problem in EXCEL.
By using =IFERROR(VLOOKUP(C2,$A:$B,2,FALSE),"")

But this approach lead me to create new column in excel for each level hierarchy. And putting the formula on first cell of that particular column and then dragging the result for each manager

But incase of big companies there would be n no. of level of hierarchy. So creating the new column in excel for each level of hierarchy would be time consuming task. Hence, I am looking for an optimal solution.

Expected Output:

| Employee ID | Manager ID | Level 1 | Level 2 | Head of Manager |
|:-----------:|:----------:|---------|---------|-----------------|
| E068        | E067       | E001    |         | E001            |
| E071        | E067       | E001    |         | E001            |
| E229        | E069       | E066    | E001    | E001            |
| E248        | E144       | E001    |         | E001            |
| E226        | E223       | E001    |         | E001            |
| E236        | E241       |         |         | E241            |
| E066        | E001       |         |         | E001            |
| E067        | E001       |         |         | E001            |
| E144        | E001       |         |         | E001            |
| E223        | E001       |         |         | E001            |

  The Employee ID column contain UNIQUE ID while Manager ID contain DUPLICATES ID.

Thank you for your time and consideration.

Let's call the dataframe "manager" with column names "Emp" and "Man". First you get the people who are always head mananagers:

head_man = set(manager["Man"]) - set(manager["Emp"])

This returns

{'E001', 'E241'}

and then use the following function to find the head managers of the rest:

def manager_of(emp):
  if emp in head_man:
    return emp
  else:
    return manager_of(manager[manager["Emp"] == emp]["Man"].values[0])

Complete Code:

import pandas as pd

head_man = set(hrms_data["Manager ID"]) - set(hrms_data["Employee ID"])

def manager_of(emp):
  if emp in head_man:
    return emp
  else:
    return manager_of(hrms_data[hrms_data["Employee ID"] == emp]["Manager ID"].values[0])

for index, row in hrms_data.iterrows():
  hrms_data.loc[index,"Head of Manager"] = manager_of(row["Employee ID"])

Edit: hrms_data["Head of Manager"] = hrms_data["Employee ID"].map(manager_of) instead of the for loop is better.


Multi-class classification configuration 1) What are the appropriate activation and loss functions for multi-class classification problem?

Is it so that:

Up to 2 classes $\rightarrow$ Binary classification $\rightarrow$ Activation: Sigmoid $\rightarrow$ Loss: binary_crossentropy
From 3 classes $\rightarrow$ Multi-class classification $\rightarrow$ Activation: Softmax $\rightarrow$ Loss: categorical_crossentropy

If so then...

'Dogs and cats' are binary classification but 'dogs, cats and horses' multi-class classification?
If we have 2 classes then we can't output probability ranges for prediction results?

2) Is it so that for multi-class classification, the last dense layer must have a number of nodes equal to the number of classes?

In the following example there are two classes (dogs and cats) and last dense layer has one node (why not 2 nodes?). If we have 3 or 10 classes then last node should have 3 or 10 nodes?

In my experience the activations are irrelevant to whether the classes are binary or multi-class. Their effectiveness depends on the kind of data you have.

Furthermore, there is one node in the layer because you are doing a binary classification. In other words, when the value of the last node is low or below a threshold the class is 0 and 1 if otherwise(value of last node is high or above a thresold).

And yes, if there are more classes you need as many nodes in the last Dense Layer. You can also you 2 nodes when you only have two classes in your data and predicting the one with the highest score/output.


When to use bayesian linear regression instead of linear regression? When does it make sense to use a bayesian approach, maybe in context to linear regression?
To be more concrete: Assume you measure a certain number of devices and you wanna' check the linear relationship between, e.g., voltage and current.
Why bayesian and/or why not? How would it be different?

The Bayesian approach should be used in the case of:

Strong priors - You have preexisting data and / or domain knowledge that you want to incorporate into the analysis.

Distributional estimates - Instead of point estimates, the result of Bayesian analysis will yield distributions. Those distributions will better quantify the uncertainty of predictions.

In the case of voltage and current for devices, Bayesian linear regression will predict a distribution of current for a given voltage. That is useful because many devices are noisy and possibly the degree of noise changes over voltages.


How do I predict continuous value from time series data? I have a glove that have 2 IMUs (Inertial Measurement Unit) attached to it. It can give the rotation value as Quaternion (x,y,z,w) and acceleration of the hand (x,y,z).
I put it on my left hand and I want to predict the position of the hand relative to some fixed point on my body (e.g. my head).

When I collect the data, I just use the Vive controller to track my hand.

Right now I just feed the value into a simple model like SVM to predict (x,y,z) of the hand relative to my head.
The problem is that the output is jumpy as the sensors are not stable enough. And because it doesn't take into account data in the previous timestep.

I want to utilize time series data as my data is time series in nature.

Which algorithms I should use if I'm treating my data as time series?
You can suggest me things I should learn about too.

I will first list some examples of models you could look into.

  You can suggest me things I should learn about too.

You can look at more classical regression type models, or go deeper in to deep neural networks, utilising Long Short Term Memory cells, which model relationships over time.

Classical

The main term that will help you search for related tutorials/documentation will be autoregressive, which is a fancy name for saying a target value is predicted from its recent history. How recent, is a parameter you can tune your model for, called lags.

Have a look at the options in the statsmodels module for Python. More specifically, you might want to try out the ARIMA model class. For more details on ARIMA, check out this thread.

There are many traditional models that you could use for a time-series problem. Terms you might consider searching for include:

Here is an introduction article about these Generalised Linear Model (GLM) -
Here is a variant, called General Additive Models (GAM) with a good walkthtough using R

Deep Learning

This is a much newer topic and is overall a fair bit complexer than the classical models discussed above. If you go this route, you would probably make progress quickest by using a Deep Learning framework such as Keras, which allows you to build complex models without too much time investment. It is a wrapper library that uses either Tensorflow, Theano or CNTK in the background (these are the available backends).

In a deep learning approach involving time-series analysis, you almost certainly want to start with an LSTM model. This stands for Long Short Term Memory, and uses a complex cell to monitor various states as you pass in your time-series data. It is a really big topic, so I will just provide:

an introduction to the logic behind LSTM, and
a nice tutorial on how to get things working

If you decide to go this way (as opposed to classical models), then yes, you should understand backprop.

This is a really broad problem that you have described and there are many many approaches. It is also likey very active in research within the fields of (self-)localisation.


Data Mining - Intent matching and classification of text PROBLEM

Suppose you have a list of 100,000+ google queries related to travel bookings. For example:

hotels in london
barcelona flight
city breaks to berlin
khao san road hostel
luxury holiday to paris
new york business class flight price
disneyland trip...

How can I extract the location i.e., London
Classify the Line of Business i.e., flight, hotel, package, etc
Classify the affinity i.e., luxury, family, city break, beach etc
Use this info to record the frequency of various pattern that exists in the keywords

i.e. **keyword pattern**                **frequency**
(destination) hotel                          xxx
flight to (destination)                      yyy
(theme) (destination) hotel                  zzz

POTENTIAL SOLUTIONS

Manual - get an exhaustive as possible list of locations (most of
the queries will be for tourist destinations) and look for a match
against the keywords. Again, compare the keywords against possible
Line of Business Identifiers and Affinity Identifiers
Google Cloud Natural Language API - This can be used to analyse
Entities and Sentiment of text. e.g.
     hotels in london -> entity(hotels), entity(london)
     barcelona flight -> entity(flight), entity(barcelona)
This isn't very powerful and only supports english.
Machine learning - seems difficult as I haven't got any descriptors
for the keywords. Would Naive Bayes be applicable or SVM?

I'd preferably like to run any solution in R too.

Could someone please suggest a direction/potential solution?

The questions you have listed are more or less independent from each other.
To extract location, you should be looking into the class of solutions called Named Entity Recognition. It is widely supported, and, for example, NLTK should have location extraction support for languages but English.

The second task, the line of business, seems cryptic to me, since you are solving the task, you have complete information on the full list of those lines, so probably writing any kind of detector (custom NER would work) is a way to go, however, if all the queries are formed as LOB + location, by solving the first one, you can subtract the location out of query to get the line of business.

From the examples of queries you have provided, I don't see any way for a human being to extract any affinity, so you should investigate it to understand if those queries actually carry such kind of information.

With (1) and (2) given, the fourth task should be trivial to count.


MDP - RL, Multiple rewards for the same state possible? This question is from  An introduction to RL  Pages 48 and 49. This question may also be related to below question, although I am not sure:
Cannot see what the "notation abuse" is, mentioned by author of book

On page 48, it is mentioned that p:S * R * S * A -> [0,1] is a deterministic function:

  The dynamics function $p : \mathcal{S} \times \mathcal{R} \times \mathcal{S} \times \mathcal{A} \rightarrow [0, 1]$ is an ordinary deterministic function of four arguments.

However, on page 49, in equation 3.4, there is summation over r:

  $$\sum_{s' \in \mathcal{S}}\sum_{r \in \mathcal{R}} p(s',r|s,a) = 1  ,\text{for all } s \in \mathcal{S}, a \in \mathcal{A}(s)$$

My question is, does this mean, it is possible that performing an action $a$ that takes us to state $s'$, could result in multiple rewards?

it is possible that performing an action $a$ that takes us to state $s′$, could result in multiple rewards?

Yes, that is true the general case that any $(s,a)$ pair can result in a range of results for both $s'$ and $r$. Also $s'$ and $r$ can vary independently, provided each ones distribution only depends on $(s,a)$. In practice $r$ often depends strongly on one or more of $s$, $a$ or $s'$ (if it depends on the latter, then it still depends in absolute terms on just $s$ and $a$ because $s'$ does - it is just that values of $s'$ and $r$ are allowed to correlate).

This does not affect the statement about $p(s',r|s,a)$ being deterministic. It is the probability of specific $s',r$ results occurring that should be deterministic and depend on $(s,a)$ in a Markov Decision Process.


How to print out to a file using Standford Classifier I am using Standford Classifier for my project.
This project takes training data to tune the algorithm then test data to classify text inputs into categories.

So the format for test and training data is tab-delimited text which means predictor -TAB- input text

The software prints out the output to stdout (command line).
Is there anyway to output to a text file?

I searched the javadoc of the project site, and I found:

But I don't know how to use this property.

I tried -csvoutput=%1%n%c on command line
But it gives me java null pointer exception error when I try to run it.
Any ideas?
Thank you so much.

I don't have experience with Stanford Classifier, but if it is sending output to stdout that you would like to capture, you can simply pipe the output to a file. You didn't mention an OS, but on most the > operator is used to redirect stdout to a file.

For example, if you are running Standard Classifier like this:

java -cp stanford-classifier.jar [option...]

You can instead run it like this to output to a file:

java -cp stanford-classifier.jar [option...] > my-output.tsv


Weighted Sum with restricted weights Given a value $y$ and some values $x_1, ..., x_n$.
How do I find weights $w_1, ... w_n$ so that the $error = y - w_1*x_1 + ... w_n*x_n$
is minimal, where the weights have to sum up to 1 or a different value?

The name of the model you are looking for is: Constrained Estimation of Ordinary Least Squares, this model allows you to introduce linear constrains to your estimation by just modifying the matrix expression of the OLS estimation.

The resulting model should satisfy the expression

$$Q^T\beta = c$$

It is solved with the equation:
$$\hat{\beta^c} = \hat{\beta}-(X^TX)^{-1}Q(Q^T(X^TX)^{-1}Q)^{-1}(Q^T\hat{\beta}-c)$$

For your problem, $Q=[1,1,1]$, $c=1$, $X$ and $Y$ are the matrixes for your data.


Training accuracy with SGD So I am computing the training accuracy of my CNN on a batch training data and it almost always gives 100% accuracy. But I found from somewhere that in order to get the training accuracy, it is tested against the entire training data and not the specific batch that you used for your training.

Is training accuracy even computed? I wonder if you don't use batches for training. Then it will be 100% accuracy always?

I'm suspecting that this is due to using a small batch size.

Training accuracy is not considered because it can give falsified evaluation value for over-fitting 100% in your case. Hence it is always computed against the batch not entertained for training. This corresponds to the generalized error.
For more information see K-Fold cross validation.

This might help: https://thebayesianobserver.files.wordpress.com/2012/02/debugging-ml.png


Mirror column plot with different y-axis in ggplot I am trying to convert this plot

Into a mirror column plot

I have two datasets:

school_purchases
recent_school_shootings

I am not very experienced with visualisation but this is my approach

ggplot(school_purchases_month, aes(x = date, y = total)) +
  geom_col(fill = "black", color = "white") #this plots one column chart of spending

I also want recent_school_shootings$killed on the other side

recent_school_shootings$killed=mutate(recent_school_shootings, killed= -killed)

ggplot(school_purchases_month, aes(x = date, y = total)) +
  geom_col(fill = "black", color = "white") +
  geom_col(data = subset(recent_school_shootings, killed > 0),
             aes(x = date, y = killed))

but it's throwing all kinds of error. Can you point me in right direction?

You can either reshape the data into long format (after joining the two datasets) and use geom_col once, or have two geoms as you have in your question. However, the latter needs some variation to have the aes in geom_col and not in ggplot.

Below, I have an example with mpg dataset, illustrating the two options (except the joining). I also included an example with your datasets but since you don't have a reproducible example, I cannot be sure of the outcome.

library(dplyr)
library(ggplot2)
library(tidyr)

## Calling geom_col twice
mpg %>%
  group_by(manufacturer) %>%
  summarise_if(is.numeric, mean) %>%
  select(manufacturer, cty, hwy) %>%
 ggplot() +
  geom_col(aes(x=manufacturer, y=hwy, fill="hwy")) +
  geom_col(aes(x=manufacturer, y=-cty, fill="cty")) +
  theme(axis.text.x = element_text(angle = 90, vjust = 0.4))

## Converting data to long format
mpg %>%
  group_by(manufacturer) %>%
  summarise_if(is.numeric, mean) %>%
  mutate(cty = -cty) %>%
  select(manufacturer, cty, hwy) %>%
  pivot_longer(., -manufacturer) %>%
 ggplot() +
  geom_col(aes(x=manufacturer, y=value, fill=name)) +
  theme(axis.text.x = element_text(angle = 90, vjust = 0.4))

For your datasets, first we join them on date column and then we can use geom_col on each of the desired columns. Note that I have aes in geom_col and not in the ggplot. You need to make sure that your date column is not class of factor (but joining may resolve that issue). If you shared output of dput(head(dataset)) for each of your two datasets, I could further investigate your problem.

recent_school_shootings %>%
  full_join(., school_purchases_month, by = "date", suffix = c("s", "p")) %>%
 ggplot() +
  geom_col(aes(x=date, y=total, fill="Purchase")) +
  geom_col(aes(x=date, y=-killed, fill="Shooting"))

Created on 2020-05-20 by the reprex package (v0.3.0)


Why does averaging over vector values cause errors? As a way to improve my model, I want to average GloVe vectors over a sentence. However, I can't get np.mean to work. The following code works when not averaging over words. (copied from other code)

embeddings_dict = {}
with open("glove.6B.50d.txt", 'r') as f:
    for line in f:
        values = line.split()
        word = values[0]
        vector = np.asarray(values[1:], "float32")
        embeddings_dict[word] = vector

g = open("input.txt", 'r')

vector_lines = []

for line in g:
    clean_line = line.translate(str.maketrans('', '', string.punctuation))
    array_line = clean_line.lower().split()
    vec_line = []
    i = 0
    for word in array_line:
        i += 1
        try:
            vec_line.append(embeddings_dict[word])
        except:
            vec_line.append(embeddings_dict["unk"])
    while i < 30: //pad up to thirthy words with zero vectors
        vec_line.append(np.zeros(50))
        i += 1
    vector_lines.append(np.asarray(vec_line))

X = np.asarray(vector_lines)

To average over words, I'm modifying a small part of the code

        try:
            vec_line.append(embeddings_dict[word])
        except:
            vec_line.append(embeddings_dict["unk"])

    #padding is not necessary anymore
    #while i < 30:
    #    vec_line.append(np.zeros(50))
    #    i += 1

    vec_mean = np.mean(vec_line, axis=0, keepdims=True)[:,None]
    vector_lines.append(vec_mean)

X = np.asarray(vector_lines)

This gives me the error "ValueError: could not broadcast input array from shape (1,50) into shape (1,1)". It feels as if I have tried every possible modification to this code as I could, but I keep on getting shape issues. What is causing all of these issues?

Change this:

import bumpy as np
a = np.array([1,2,3])
b = np.array([4,5,6])
vec_line = [a,b]

print(np.mean(vec_line, axis=0, keepdims=True)[:,None])
>>[[[2.5 3.5 4.5]]]
np.mean(vec_line, axis=0, keepdims=True)[:,None].shape
>>(1,1,3)

To this:

print(np.mean(vec_line, axis=0))
>>[2.5 3.5 4.5]
np.mean(vec_line, axis=0).shape
>>(3,)


How to deal with missing data for Bernoulli Naive Bayes? I am dealing with a dataset of categorical data that looks like this:

    content_1   content_2   content_4   content_5   content_6
0         NaN         0.0         0.0         0.0         NaN
1         NaN         0.0         0.0         0.0         NaN
2         NaN         NaN         NaN         NaN         NaN
3         0.0         NaN         0.0         NaN         0.0

These represent user downloads from an intranet, where a user is shown the opportunity to download a particular piece of content. 1 indicates a user seeing content and downloading it, 0 indicates a user seeing content and not downloading it, and NaN means the user did not see/was not shown that piece of content.

I am trying to use the scikit-learn Bernoulli Naive Bayes model to predict the probability of a user downloading content_1, given if they have seen downloaded / not downloaded content_2-7.

I have removed all data where content_1 is equal to NaN as I'm obviously only interested in data points where a decision was actively made by the user. This gives data as:

    content_1   content_2   content_3   content_4   content_5   content_6
0         1.0         NaN         1.0         NaN         NaN         1.0
1         0.0         NaN         NaN         0.0         1.0         0.0
2         1.0         0.0         NaN         NaN         NaN         1.0

In the above framework, NaN, is a missing value. For data points where a Nan is present, I want the algorithm to ignore that category, and use only those categories present in the calculation.

I know from these questions: 1, that there are essentially 3 options when dealing with missing values:

ignore the data point if any categories contain a NaN (I.e. remove the row)
Impute some other placeholder value (e.g. -1 etc.) or
Impute some average value corresponding to the overall dataset
distribution.

However, these are not the best option for the following reason:

Every single row contains at least 1 NaN. This means, under this
arrangement I would discard the entire dataset. Obviously a no go.
I do not want the missing value to add to the probability
calculation, which will happen if I replace Nan with say -1. I'm also using a Bernoulli Naive Bayes, so as I understand, this requires singly 0 or 1 values.
As this is categorical data, it does not make sense for me to do this,
in this way (it was either seen or not, and if not, it is not needed).

The answer here indicated that the best way to do this, is, when calculating probabilities, to ignore that category if it is a missing value (essentially you are saying: only compute a probability based on the specific categories I have provided with non missing values).

I do not know how to encode this when using the scikit-learn Naive Bayes model, whether to do this as a missing value.

Here's what I have so far:

df=pd.read_clipboard()
from sklearn import datasets
from sklearn.naive_bayes import BernoulliNB
# Create train input / output data
y_train = df['content_1'].values
X_train = df.drop('content_1', axis=1).values
# Loud Bernoulli Naive Bayes model
clf = BernoulliNB()
clf.fit(X_train, y_train)

Obviously, this returns an error because of the present NaNs. So how can I adjust the scikit-learn Bernoulli model to automatically ignore the columns with NaNs, and instead take only those with 0 or 1?

I am aware this may not be possible with the stock model, and reviewing the documentation seems to suggest this. As such, this may require significant coding, so I'll say this: I am not asking for someone to go and code this (nor do I expect it); I'm looking to be pointed in the right direction, for instance if someone has faced this problem / how they approach it / relevant blog or tutorial posts (my searches have turned up nothing).

Thanks in advance - appreciate you reading.

Your search results are on point: without dropping or imputing data, there's no built-in way to do what you want with BernoulliNB.

There is, however, a way out: train separate Bayesian models on filtered samples from your data, and then combine their predictions by stacking them.

Filtering

Filtering here means:

Isolating samples from your original df, each having only a subset of df.columns. That way, you'd have a DataFrame only for content_2, one for content_2, content_3, in a sort of a factorial combination of columns.
Making sure each sample is made only of rows that have no NaNs for any of the columns in the subset.

This part is somewhat straightforward in your case, yet a bit lengthy: you'd have $n!$ (n factorial) combinations of columns, each of which would result in a separate sample. For example, you could have a sample named df_c2 containing only content_2 rows valued 0 or 1, df_c2_c3 with only content_2 and content_3 columns filled, and so on.

These samples would make NaN values non-existent to every model you'd train. Implementing this in a smart way can be cumbersome, so I advise starting with the simplest of scenarios - e.g. two samples, two models; you'll improve gradually and reach a solid solution in code.

Stacking Bayesian Models

This is called Bayesian Model Averaging (BMA), and as a concept it's thoroughly addressed in this paper. There, weight attributed to a Bayesian model's predictions is its posterior probability.

The content can be overwhelming to absorb in one go, be at ease if some of it doesn't stick with you. The main point here is that you'll multiply each model's predicted probabilities by a weight 0 < w < 1 and then sum (sum results shall be in $[0, 1]$). You can attribute weights empirically at first and see where it gets you.

Edit:

Due to the added complexity of my proposed solution, as stated in this (also useful) answer, you could opt to implement Naive Bayes in pure Python, since it's not complicated (and there are plenty tutorials to base upon). That'd make it a lot easier to bend the algorithm to your needs.


Which classification algorithms are negatively affected by class imbalances? I've seen a few posts and papers floating around the web (mostly those related to over/undersampling, SMOTE, and cost-sensitive training) that, when discussing class imbalance, specify that certain algorithms are negatively impacted by class imbalance.

Which algorithms are those? Which are not? How can we figure out whether or not an algorithm or approach will be negatively affected by class imbalance?

[This answer is based on my limited knowledge, please don't hesitate to edit or propose improvements in the comments]

Actually I think it's a bit misleading to say that algorithms can be affected by class imbalance, because it's not exactly the algorithm which is affected it's the evaluation method (I mean "evaluation" in a broad sense including the loss function used by the algorithm during training). Some algorithms may be closely related to a particular loss function or internal optimization strategy, so by association such algorithms have the same weaknesses.

A simple way to see that class imbalance issues completely depend on the evaluation method is to compare micro-average performance with macro-average over the classes in a case where 99% of the instances belong to the same class:

micro-average gives the same weight to every instance, so a model which assigns the majority class will look as if it performs very well.
macro-average gives the same weight to every class, so assigning the majority class  won't work better than random.

So technically the problem of class imbalance could (should?) be seen as a design choice between maximizing the number of instances correctly classified (default evaluation) and any other alternative, for example giving equal weight to every class.  But of course it's not practical nor common to design a specific evaluation measure or loss function specific to every problem.


How to interpret KDE distribution graph? I would like to know how to interpret this distribution graph. I have been doing an exercise from the book called 'Python for Finance Cookbook' by Eryk Lewinson. It does not give an in-depth explanation on this graph and i have watched videos on YouTube and it does not explain the comparison of distribution.
I would also like to know if this is positively skewed.
This is the distribution graph.

Thank you for your help!

Please find beautiful,  explanation about KDE,  In your graph on X Coordinateif the tail is stretching long towards right side then its positively skewed, it means most of your data points were distributed to left side and vise versa for negative skewness.

Always we needs to ensure that data points on the graph needs to be equally distributed to form Gaussian Normal Curvefor better outcome.
There are various techniques to get rid of skewness , please find top 3 methods to play around with skewed data .

Some prerequisites references for any Distribution Charts

Percentiles, Quartiles, and Interquartile
Range

Percentiles examples

Quartiles examples


What is a good interpretation of this 'learning curve' plot? I read about the validation_curve and how interpret it to know if there are over-fitting or underfitting, but how can interpret the plot when the data is the error like this:

The X-axis is "Nº of examples of training"
Redline is train error
Green line is validation error

Thanks

The X axis is the number of instances in the training set, so this plot is a data ablation study: it shows what happens for different amount of training data.
The Y axis is an error score, so lower value means better performance.
In the leftmost part of the graph, the fact that the error is zero on the training set until around 6000 instances points to overfitting, and the very large difference of the error between the training and validation confirms this.
In the right half of the graph the difference in performance starts to decrease and the performance on the validation set seems to be come stable. The fact that the training error becomes higher than zero is good: it means that the model starts generalizing instead of just recording every detail of the data. Yet the difference is still important, so there is still a high amount of overfitting.


How do I group similar type of skills together? Suppose that I have a file which has thousands of skills starting from A-Z. Now, I would like to create a model that can group similar skills together (example neural network and SVM can group together). I know that I can use NLP for this problem, but I'm not sure about the algorithm that I can use to get the best result.

I'm new to NLP so any help is greatly appreciated.

If all you have is the name/word, then I see only two ways:

You can look at word similarity, basically by measuring how similar the spelling is. There are a number of possible metrics (e.g. Levenshtein distance). This will probably not get you very far.
You can look at semantic similarity. You can use pre-trained word embeddings to map the words to a new vector space where you can calculate the distance between the word embeddings, e.g. with word2vec or other implementations. Alternatively, you can use a dictionary/corpus (e.g. https://www.nltk.org/index.html) that provides you a word taxonomy that allows you to compute semantic similarity between words via relationships like synonyms etc. See here for different approaches.


Weighted samples in Tensorflow for convolutional neural networks For my binary classification problem (A vs B),  each image in either class has its individual weight. This means, for example, if I have 10000 images for A, not all of the images are equally important. Weight of "0.9" would mean it is very important, and a weight of "0.01" means that it is not that important.
Can this information be communicated to the training in Tensorflow?
Thanks for the help.

Same as in case of the weighted classes, you have to account on sample weights in your loss function.

If you use Tensorflow in combination with Keras, you can simply use the sample_weight argument of the model.fit function.

As an alternative, you can implement your own loss function. However,
depending on your setup/framework it can get tricky to determine sample indices from inside the loss function.


X Second samples taken on unevenly spaced intervals I have dataset of following specification:

512 samples taken at unevenly spaced intervals over the year
Each sample is an 8 second data from sensors with 4ms resolution
Samples are not labeled

For example, I have 5 samples taken on first day, then more then 10 samples take on 5th day and so on.
I want to cluster data to check if I can infer the mode of operation for the machine from single 8 second sample. Also, I want to measure the performance of the component over the year for predictive maintenance.
Currently I want to use self organizing maps for clustering purposes. I am new to this data science and am currently learning. The usual methods use evenly spaced samples. Also each samples in these cases is single input (Like stock value at the end of a day) instead of X second data taken at time Y.
My question is: How do I input such data into any model?

It depends how you want to cluster the data, but here are some options....

FEATURE ENGINEERING
You could, for example, completely ignore the timestamps and just seek
to cluster the different modes of operation are based on the
magnitude of the feature alone. Here, simply extract the values into
a new feature list.
Otherwise, you have to consider what is important about the samples. For example, does the fact that one snapshot containing 5 samples vs 8 samples distinguish it between modes? If so they, build your feature vector based off counting the number of samples in a day.
In doing so, you will create single features for each day from various attributes in the day (e.g. magnitude, number of samples), enabling you to cluster them based off these features.

RESAMPLING
Conversely, you could resample all the data into uniform intervals, but this would probably need lots of assumptions, and would therefore introduce noise.


Policy gradient/REINFORCE algorithm with RNN: why does this converge with SGM but not Adam? I am working on training RNN model on caption generation with REINFORCE algorithm. I adopt self-critic strategy (see paper Self-critical Sequence Training for Image Captioning) to reduce the variance. I initialize the model with a pre-trained RNN model (a.k.a. warm start). This pre-trained model (trained with log-likelihood objective) got 0.6 F1 score in my task.

When I use adam optimizer to train this policy gradient objective, the performance of my model drops to 0 after a few epochs. However, if I switch to gradientdescent optimizer and keep everything else the same, the performance looks reasonable and slightly better than the pre-trained model. Is there any idea why is that?

I use tensorflow to implement my model.

Without the code there's not much we can do but, I'd guess you need to significantly lower the learning rate. From my experience Adam requires a significantly lower learning rate compared to SGD.


The meaning of random word dropout in NLP I have been reading the early paper on pre-training in NLP (https://arxiv.org/abs/1511.01432) and I can't understand what random word dropout means. The authors completely ignore explaining this method as if it was a standard thing. Can someone explain what they really do and what is the purpose of that?

It is not uncommon that we can make sense of a sentence without reading it completely. Or when you are having a quick look at a document, you tend to oversee some words and still understand the main point. This is the intuition behind the word dropout.

Generally this is done by randomly dropping each word in a sequence following for example a Bernoulli distribution:

  $X \leftarrow X \odot \vec{e}, \vec{e} ∼ B(n, p)$

where X is the index of the word token, n
is the lenth of the sequence, and
 $\vec{e}$ is a vector with each word dropout state.

This is usually done after calculating the word embeddings, and the words selected to be left out are normally changed to the <UNK> equivalent embedding.

By doing this, we allow out model to learn more flexible ways of writing/convey meaning.


What does "zero-meaned vector" mean I'm trying to reproduce an algorithm designed in a paper. And everything is going well except one thing:

It says we considered the lengths zero-meaned accelerometer vectors and created a feature for the mean and standard deviation of this value. and I do not understand what is it zero-meaned vectors?

Example dataset:

-0.6946377  12.680544  0.50395286
5.012288    11.264028  0.95342433
4.903325    10.882658  -0.08172209
-0.61291564 18.496431  3.0237172
-1.1849703  12.108489  7.205164
1.3756552   -2.4925237 -6.510526
-0.61291564 10.56939   5.706926
-0.50395286 13.947236  7.0553403

Can any body help me?

I found only this information https://www.quora.com/What-does-it-mean-when-a-vector-is-zero-mean but I'm not sure about it.

Thank you.

"Zero-meaned" means the vector has been transformed so that its mean is 0.

Typically, you would do this by subtracting the mean of each column from that column. (This is for dimensional as well as algorithmic reasons; you don't want to subtract a person's weight from their height.)

It sounds like here they're actually talking about the row mean--that is, $(-0.6946377, 12.680544,  0.50395286)$ would be transformed to $(-4.857924, 8.5172577, -3.65933344, 4.1632863, 7.40047)$, where the first three are the original features minus the row mean, the fourth is the row mean, and the fifth is the standard deviation of the original features.

This would make sense if the three have the same units (if they're all accelerations at the same scale, this works), and so you want a separate measure of how much it's being accelerated at all and how much it's being accelerated in a particular direction.


How to implement a POS tagger using Word co-occurence and Clustering concepts? I am thinking of implementing a POS tagger by myself. a POS tagger, extracts the syntax role of a word in a sentence.
According to my studies, word co-occurrence is a technique to analyze word occurrence which can be used to construct a graph where nodes are words and the weights between them is their co-occurrence weight.
I am wondering if there is a way to apply clustering algorithms on this graph to group words based on their syntax roles? i mean i want to do the same as a POS tagger does. the main idea is that after constructing the graph of co-occurrence, how can i apply a clustering or community detection algorithm on this graph to group nodes based on their roles in a sentence? for example, to group nodes which are verbs, or group nodes that are nouns. in clustering i don't need to know this is verb or not, i only want to group nodes which have a similar syntax in graph. then i will analyze them to find out they are weather a verb or noun or ...
I would be so thankful if you give mind on starting this project.

First, you are right that word co-occurrence graphs have been used in various applications of NLP. More precisely in applications related to the meaning of the words, typically for topic modelling or word sense induction. These applications follow the linguistic principle that words which occur in similar contexts tend to have a similar meaning. This is the basis of distributional semantics.
However this principle is unlikely to help finding words which have a similar syntactic function, simply because the co-occurrences in a sentence don't correspond to their syntactic role: a sentence usually contains many distinct POS tags, it doesn't have a specific POS category. Basically, word co-occurrences are useful for semantics, not syntax.
If the goal is to implement a POS tagger, why not use state of the art methods known to work very well for POS tagging? There is a lot of training data available, for instance the Universal Dependencies corpora.
Note that POS tagging is normally a supervised task (based on annotated data), whereas your idea appears to be about finding groups of similar POS in an unsupervised way. This would be another story entirely, probably involving grammatical inference.


Apache Spark ML vs Flink ML I'm sorry, if I ask my question by the wrong community.
What is a difference between Spark ML and Flink ML and between Spark and Flink in general? The both projects are the projects of Apache, I would like to know why Foundation has two similar projects.

PS I have found some interesting article Fast Big Data: Apache Flink vs Apache Spark for Streaming Data
It has answers on my question.

PPS This question has been already discussed by Stack Overflow community. The topic is What is the difference between Apache Spark and Apache Flink

Both Spark and Flink are designed to process data in batch or stream over distributed environment.

Flink primarily being defined as its ability to process streaming data in real time and being considered as good option for processing low data latency data and high fault tolerance on distributed systems on a large scale.
However, Spark is utilized predominantly in the case of batch processing over distributed environment.
Spark do have streaming API on the other hand but Flink is getting more traction for executing this need.
Flink can be utilized on local JVM,  standalone, Yarn and cloud similarly Spark can be deployed over local, standalone, Yarn, Mesos.

There are many more difference between these two. I found some really interesting answers for your question on Quora, have a look-

Differences between Apache Spark and Apache Flink - 1
Difference between Apache Flink and Apache Spark - 2
Difference between Hadoop, spark, flink

Hope it helps!


Structuring CIFAR100 for resnet18 This will probably be a basic question since I am starting with computer vision. I am trying to use resnet18 from pytorch and work with CIFAR-100 dataset. Single image has size 3x32x32 and the model cannot forward this throwing error. It can process arrays of size 224, 128, 64. How should I structure my images to process them with resnet18 specifically (not other arch).

PyTorch offers a range of transformations and you can simply apply a transformation to resize your images when loading your data. For example by using these transforms (using CIFAR10 as an example here which comes with 3x32x32 images too):

# transform for train data
transform_train = transforms.Compose([
    transforms.RandomResizedCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                 std=[0.229, 0.224, 0.225])
    ])

# transform for test data
transform_test = transforms.Compose([
    transforms.Resize(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                 std=[0.229, 0.224, 0.225])
    ])

# load data and apply transforms
train_data = datasets.CIFAR10('data', train=True,
                              download=True, transform=transform_train)
test_data = datasets.CIFAR10('data', train=False,
                             download=True, transform=transform_test)

Both, transforms.RandomResizedCrop(224) and transforms.Resize(224) give the desired size. (the random cropping is applied as data augmentation to further diversify training data) You will find more details on available transforms in the PyTorch documentation.

As a side note: the size requirement is the same for all pre-trained models in PyTorch - not just Resnet18:

  All pre-trained models expect input images normalized in the same way, i.e. mini-batches of 3-channel RGB images of shape (3 x H x W), where H and W are expected to be at least 224. The images have to be loaded in to a range of [0, 1] and then normalized using mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225].


Difference between explicit and implicit density with and without the relation to neural network I have a lack of understanding about this issue. Could anybody explain it or give an advice to a good literature regarding it? I don't understand what is a explicit density model and how it differs from an implicit density one. Could anyone exemplify the difference between these two kinds of models?

Here is the picture source.

This paragraph in this paper may be of help, methinks.


Changing Timestamp format for Date-Time in Excel/Pandas/Python? I have a excel data with time stamp format like this "2019-06-10T14:05:00+05:30"
[YYYY-MM-DDTHH:MM:SS], I want it to be converted into "2019-06-10 14:05" [YYYY-MM-DD HH:MM],

Is it possible to convert into required format using Excel or Python Libraries?
or any other way?

Thank you!

Using python's pandas library
strftime() helps obtain specific format
to_datetime() of pandas helps convert string to datetime object

import pandas as pd
dt = "2019-06-10T14:05:00+05:30"
dt = pd.to_datetime(dt, errors='ignore')
transformed_date = dt.strftime("%Y-%m-%d %H:%M")
print(transformed_date)
print(type(transformed_date))
dateType = pd.to_datetime(transformed_date)
print(type(dateType))


How do I build a DQN which selects the correct objects in an environment based on the environment state? I have an environment with 4 objects in it. All of these objects can either be selected or not selected. So the actions taken by my DQN should look like - [1,0,1,1],[0,0,0,1],[1,1,0,0]...etc

Where 1 denotes that the object was selected and 0 denotes that the object was not selected.
The environment state being given as input to the DQN consists of attributes for each of the object and other factors of the environment. The DQN would get rewards based on the selection it made.
I'm new to reinforcement learning and I've only built DQNs that had to select one action out of the entire action space. But how do I build a DQN or a Reinforcement learning network for this particular environment?

The DQN agent does not need to care what the actions represent, in your case it only needs to make a discrete choice, and it is simple to enumerate the action space. Ignoring the meaning of them for a moment, you have 16 discrete actions. The simplest way to model that is to have a single index discrete action space of 16 actions for the agent which you then map to the selections that you need in order to assess the results. As long as you do this consistently (e.g. take the binary representation of the action index number), this is fine.

It is also possible that using a more direct representation would help, depending on the true nature of the value function. In which case you could use it, provided you model the neural network for $\hat{q}(s,a,\theta)$ with action vector concatenated to state vector in the input, and a single output of the estimated action value for that specific combination. To assess which action to take, you would create a minibatch of 16 inputs, all of which have the same state component, and cover the 16 possible input variations. Then you would pick the combination with the highest estimate and look at the action part of the input vector to discover which action was estimated to be best.

If you are not sure which approach would suit the problem best, you could try both.


understanding linear algebra of a forget gate This blog covers the basics of LSTMs.

A forget gate is defined as :

$$f_t = \sigma(W_f \cdot [h_{t-1}, x_t]+ b_f)$$

At this point the linear algebra confuses me more than it should. The syntax of $W\cdot [h,x]$ is confusing in this context. I think a vector should go into the activation function since the output $f$ is a vector, but the syntax of the forget gate above implies that the input has $2$ columns because $[h,x]$ will be an $n\times 2$ matrix

For the sake of example  lets say ...

\begin{align} W &= \begin{bmatrix} 0 & 1 \\
          2 &3 \end{bmatrix}\\
    h &= \begin{bmatrix} -1 \\
          2 \end{bmatrix}\\
    x &= \begin{bmatrix} 3 \\
          0 \end{bmatrix}\\
    b &= \begin{bmatrix} 1 \\
          -2 \end{bmatrix}\end{align}

Can anyone give the final vector that goes into the sigmoid function ?

I think the math is

$$  \begin{bmatrix} 0 & 1 \\ 2 & 3 \end{bmatrix}\begin{bmatrix} -3 & 3 \\ 2 & 0 \end{bmatrix} + \begin{bmatrix} 1 \\ -2\end{bmatrix} = \begin{bmatrix} 2 & 0 \\ 4 & 6\end{bmatrix}+ \begin{bmatrix} 1 \\ -2\end{bmatrix} = \text{ Something wrong}$$

Note that $$[h_{t-1}, x_t]$$
is the concatenation of two vectors.
In your example, it would be: $$[h_{t-1}, x_t] = [-1, 2 , 3, 0]$$
and then the dimensions of $W_f$ would be $2 \times 4$, where $2$ is the dimension of the output of the LSTM cell, i.e. the activation $h_t$, that you defined to be of dimension $2$.

Hence, $$W_f \cdot [h_{t-1}, x_t] $$ is a multiplication of a matrix of dimension $2\times4$ by a vector of $4$, which will return a vector of dimesion $2$. And then the sigmoid function will be applied point wise on each of the two elements of the result.

Hope it makes sense.


significance test and sample size estimation for classifiers What is the test to tell if e.g. an F1 score of 0.69 for classifier A and 0.72 for classifier B is truly different and not just by chance? (for mean-values one would use a "t-test" and obtain a "p-value"). I have access to the underlying data and not only to the F1 scores.
... and how can one estimate the sample size needed in the test-set in order not to miss a true difference between the F1 scores? (as in the example above) (for mean-values one would use a "power analysis"). Or in other words, if I want to know which classifier (A or B) is truly better (to a certain significance level): how many test cases do I need?
Google just returns some research papers but I would need some type of established standard methods for the sample size and significance test (ideally implemented as a python package).
--- EDIT ---
Thanks for pointing out this post in the comments - it points in the right direction but unfortunately does not solve my two related problems as stated above.

I think there are two primary possibilities for answers, so I'll try to address both of them:
(Recommended) If you have multiple independent trials per classifier:
This is a lot simpler than if you have only one trial per classifer (and much better). If you have multiple $F_1$ scores per classifier, then you can simply conduct a paired t-test or a Wilcoxon signed rank test. You'd probably go with Wilcoxon if you don't have many trials to approximate a normal distribution via the central limit theorem or if for whatever reason your $F_1$ scores are not normally distributed.
If you only have one trial per classifier:
In this situation, we are going to find a way to express $F_1$ in terms of $F_1\pm \delta_{F_1}$ to see if the two scores overlap given some confidence level.
Step 1: Decomposition
We can first decompose the metric we are trying to calculate:
$$F_1\equiv2\frac{precision\cdot recall}{precision+recall}\equiv2\frac{PPV\cdot TPR}{PPV+TPR}$$
We are looking to find the uncertainty in the $F_1$ score, so we first need to find the uncertainties in the values composing it. To do this, we can consider the precision and recall metrics in terms of Bernoulli trials by partitioning the instances into $P$ and $N$, positives and negatives. Let's also assume $P+N\gg1$.

Edit to answer the question about determining sample sizes:
First, choose a maximum width $w$ for your confidence interval in terms of percentage points (i.e. choose a margin of error MoE and double it to account for both sides of the mean). Then, using our notion of Bernoulli trials, we can determine your minimum sample size based on the z-score for your confidence interval as well as the width (MoE):
$$n\ge \frac{4z^2(p(1-p))}{w^2} \because z\sqrt{\frac{p(1-p)}{n}}\le MoE$$
The safest estimate would be to choose $p=0.5$ which would maximize $n$ for a given $z$ and $MoE$. Realistically though, this would be an overestimate for the required $n$ because you probably don't have a random classifier.
As an example, if you wanted an MoE of 10% for a CI of 95%, you'd need:
$n\ge 97$ assuming worst case $p=0.5$.

When we consider Bernoulli trials, we can talk in terms of success/failure. So the number of successful $P$, which would be $TP$, can be considered a percentage of the total number of $P$ instances, which is simply the true positive rate $TPR$. The same goes for $TN$ and $N$, being $TNR$. In other words:
$$TPR=\frac{TP}{P}=\frac{TP}{TP+FN}\enspace \enspace, \enspace\enspace TNR=\frac{TN}{N}=\frac{TN}{TN+FP}$$
Step 2: Binomial Proportion Confidence Intervals
Now, we can develop binomial proportion confidence intervals per $P$ and $N$. There are many formulas to do this, but I am going to stick with probably the most basic one. That way, we make sure that our result assumes the form $TPR\pm \delta_{TPR}$.
Now let's pick a confidence level:

If you want a $99\%$, then $z_{0.01/2}\approx2.576$
If you want $95\%$, then $z_{0.05/2}\approx 1.96$ confidence level

We can calculate the individual confidence intervals:
$$TPR\pm z\sqrt{\frac{TPR(1-TPR)}{P}}\enspace \enspace, \enspace\enspace TNR\pm z\sqrt{\frac{TNR(1-TNR)}{N}}$$
To get $PPV$, we can first define $TP=P(TPR)$ and $FP=N(1-TNR)$. Then:
$$ PPV=\frac{TP}{TP+FP}=\frac{P(TPR)}{P(TPR)+N(1-TNR)} $$
Step 3: Propagate PPV Uncertainties
We want to find $PPV\pm \delta_{PPV}$, so we need to propagate the uncertainties from $TPR$ and $TNR$. Because $P$ and $N$ are determined by the dataset rather than your model (and they are exact "measurements"), they are without uncertainties:
$$ PPV\pm \frac{P(TPR\pm \delta_{TPR})}{P(TPR\pm \delta_{TPR})+N-N(TNR\pm \delta_{TNR})}=\frac{X}{Y} $$
Let's first find the individual components $X$ and $Y$, keeping in mind that when uncertain values add, their uncertainties add as well:
$$ X=P(TPR)\pm P(\delta_{TPR}) $$
$$ Y=(N+P(TPR)+N(TNR))\pm (P(\delta_{TPR})+N(\delta_{TNR})) $$
$$ = \frac{P(TPR)\pm P(\delta_{TPR})}{P(TPR\pm \delta_{TPR})+N-N(TNR\pm \delta_{TNR})}$$
Now we can divide $X$ by $Y$. To do so, we need to keep in mind that the resulting uncertainty is the sum of their fractional uncertainties, like so:
$$ PPV=\frac{X}{Y}\enspace \enspace , \enspace \enspace \frac{\delta_{PPV}}{|PPV|}=\frac{\delta_X}{|X|}+\frac{\delta_Y}{|Y|}\Rightarrow \delta{PPV}=\frac{X}{Y}\left(\frac{\delta_X}{|X|}+\frac{\delta_Y}{|Y|}\right) $$
For simplicity, I'll write $PPV$ because we know that $$PPV\pm \delta_{PPV}\equiv \frac{X}{Y}\pm\frac{X}{Y}\left(\frac{\delta_X}{|X|}+\frac{\delta_Y}{|Y|}\right)$$
(Finally) Step 4: Propagate TPR and PPV Uncertainties
Just as we did with $PPV$, we need to propagate the uncertainties for the $F_1$ score calculation. Fortunately, it's the same process as the uncertainties for multiplication combine in the same manner as division:
$$ F_1 = 2\frac{PPV\cdot TPR}{PPV+TPR} = 2\frac{A}{B}$$
$$ A=(PPV\cdot TPR)\enspace \enspace, \enspace \enspace B=(PPV+TPR) $$
$$ \frac{\delta_A}{|A|}=\frac{\delta_{PPV}}{|PPV|} + \frac{\delta_{TPR}}{|TPR|}\Rightarrow \delta_A=|A|\left(\frac{\delta_{PPV}}{|PPV|} + \frac{\delta_{TPR}}{|TPR|}\right) $$
$$ \delta_B = \delta_{PPV} + \delta_{TPR} $$
Then we also have to combine the uncertainties for the division of $A$ by $B$:
$$ \delta_{\frac{A}{B}} = \left|\frac{A}{B}\right|\left(\frac{\delta_A}{|A|}+\frac{\delta_B}{|B|}\right) $$
This yields our final answer because we know that:
$$ F_1 \pm \delta_{F_1} \equiv 2\left(\frac{A}{B} \pm \delta_\frac{A}{B}\right) $$
Now you can see whether your $F_1$ scores overlap given some confidence interval! You could also probably see how much they overlap in terms of a ratio, but it'd probably be more understandable if you just stuck with a binary yes/no overlap.


Column vector for training example I have a question about the training example for machine learning. I know most people make training examples to a column vector. However, most of the time when we get a training example, like from a panda data frame object and transfer it to a NumPy array, each example will be a row vector. I know I can use transpose to change it to a column vector. But I want to ask why do we do that? What makes a column vector better suited for training examples?

We use column vectors because it offers efficient memory access in column-major format.
For example: CUDA, which is commonly used to accelerate libraries such as tensorflow,  operates most efficiently when you coalesce memory. Strided memory access in parallelized applications cause significant slow-downs; threads (which are themselves sequentially indexed) should operate on sequentially indexed memory, rather than striding by a number of elements equal to the matrix row size. This is true even for cpu applications, mostly because of CPU caching. You want to make as few transactions as possible, which of course only column-major column-vectors or row-major row-vectors can offer.
Neither column-major nor row-major is universally "superior" and often they can be arbitrarily decided upon. However, there are a few reasons row-major is used in some applications but isn't used in others. In particular, relational databases are inherently row-major when you consider that each entry must be a single row. The C programming language is also inherently row-major because a 2D matrix is an array of arrays that stack row upon row when you consider the first dimension is the row index, as you do in math. The reasons for column-major format are to reflect mathematical interpretations, which also ties into some legacy integration; Fortran, for example, uses column-major where C is row-major. Fortran is still used in scientific computing applications, and although most have moved away from it, some of the most popular libraries originally written in Fortran have maintained a consistent API regardless of the language. BLAS, for example, was initially written in Fortran, but its many implementions in other language (including C/C++ for cuBLAS) have typically maintained its API. Many other popular mathematical programming tools, such as MATLAB, use column major format. It makes sense coming from a math-perspective where you consider the first dimension as the row index.


What is the difference between handcrafted and learned features I am having difficulty understanding what the differences are between handcrafted and learned features.

Is it just the case that the handcrafted features are the input variables, and that the learned features would refer to the output variable? Or are they (handcrafted and learned) both types of input variables?

Can anyone help with this?

Thanks.

In short

The classification is the following:

handcrafted are features that are manually engineered by the data scientist.
learned features are ones that are automatically obtained from a machine learning algorithm

Let me give you an example:

Suppose you are doing an image classification task, where you wanted to classify cats from dogs. You want to build a classifier but you are faced with the dilemma, how do I input my data to the classifier?. You have two options:

Use the raw pixel data. The issue with this is that you have a vast feature space, which makes it hard for models to generalize.
Attempt to extract features from the image so that you can reduce your feature space.

Now if you choose the second you have two more options:

Manually define a set of features and extract them. Some examples include edge detection, corner detection, histrograms, etc. The problem with this approach is that nothing guarantees that the number of corners is a good descriptor for classifying cat and dog images.
The alternative is to train a ML model to identify and extract useful features for this specific classification task. This is exactly what a Convolutional Neural Network does. It searches for what features are best to classify the images?.

Traditionally the first approach was used extensively in Machine Learning. That changed however with the arrival of Deep Learning.

An example of what features CNNs identify when classifying different types of images:

Note that these features aren't general, low-level features like edges or corners; instead they are tailored for each class. That is the power of training a model to extract features.

Comparison

Because learned features are extracted automatically to solve a specific task, they are extremely effective at it. In fact deep learning models that perform feature extraction and classification outperform models that classify manually extracted features by a large margin. This is one of the reasons why deep learning is so popular.

On the other hand, we have no control on what features the model will extract from the data. In many cases these features are only good for classifying the data and have no real-world interpretation. They are only good for the task that they were trained for.


Stateful LSTM : Using different training window Would it make sense for stateful LSTM (or LSTM in general) if in one epoch I feed [0-9],[10-19],[20-29],[30-39]...[990-999] (with corresponding labels/Y data) from my dataset. When I've presented all of the data for that epoch I then call model.reset_states()

After that epoch, could I then move the window forward by some arbitrary amount e.g. 2, so I then feed in [2-11],[12-21],[22-31],[32-41]...[982-991]

I would do that for 99 batches (the final sequence in the batch is now length 8 so I can't make a complete sequence).

Would doing that make sense?  That way the network learns sequences from different starting/end points, and has differing output/Y values to train against.

I don’t see anything wrong with that. In fact it sounds like a good form of data augmentation.

It does sound like you are training with a batch size of 1, which may be slow. You could think about creating batches in a similar way, where the first elements of the batches are the sequences 1-10, 11-20, ..., the second elements of the batches are 2-12, 13-23, ..., all the way to 9-19, 10-29, ...


Why Heaps' Law Equation looks so different in this NLP course? I'm actually not sure if this question is allowed on this community since it's more of a linguistics question than it is a data science question. I've searched extensively on the Web and have failed to find an answer and also the Linguistics Beta Stack Exchange community also doesn't seem to be able to help. If it's not allowed here please close it.

Heaps' Law basically is an empirical function that says the number of distinct words you'll find in a document grows as a function to the length of the document. The equation given in the Wikipedia link is

$$V_R(n) = Kn^\beta$$

where $V_R$ is the number of distinct words in a document of size $n$, and $K$ and $\beta$ are free parameters that are chosen empirically (usually $0 \le K \le 100$ and $0.4 \le \beta \le 0.6$).

I'm currently following a course on Youtube called Deep Learning for NLP by Oxford University and DeepMind. There is a slide in a lecture that demonstrates Heaps' Law in a rather different way:

The equation given with the logarithms apparently is also Heaps' Law. The fastest growing curve is a corpus for Twitter data and the slowest is for the Wall Street Journal. Tweets usually have less structure and more spelling errors, etc. compared to the WSJ which would explain the faster-growing curve.

The main question that I had is how Heaps' Law seems to have taken on the form that the author has given? It's a bit of a reach but the author didn't specify what any of these parameters are (i.e. $C$, $\alpha$, $r(w)$, $b$) and I was wondering if anybody might be familiar with Heaps' Law to give me some advise on how to solve my question.

The plot shows Heaps' Law but the formula is something different, it is Zipf's Law.

$f(w)$ is the relative frequency (or probability) of word $w$. That is, given a random word, it will be $w$ with probability $f(w)$. Therefore, if a document has $n$ words, it has on average $n\times f(w)$ occurrences of word $w$.

The formula can be re-written as follows:
$$f(w)=C(r(w)-b)^{-\alpha}$$
which is a power-law distribution that shows Zipf's Law, however with a slightly different parameterization by introducing cut-off $b$.

$r(w)$ denotes the rank of word $w$. For example, if we sort all the words in a  news corpus based on their frequency, $r(\text{'the'})$ would be 1, $r(\text{'be'})$ would be 2, and so on,
Cut-off $b$ ignores highly frequent words $r(w) \le b$, effectively shifting up the rank of remaining words,
$C$ is the normalizing constant, i.e. $C=\sum_{r=\left \lfloor  b \right \rfloor + 1}^{\infty}(r-b)^{-\alpha}$, which gives $\sum_{w,r(w)>b} f(w) = 1$, and
Exponent $\alpha$ denotes the rate of drop in probability when rank increases. Higher $\alpha$, faster drop.

Exponent $\alpha$ is determined by fitting the formula to some corpus, as shown in the table. Generally, lower $\alpha$ (in the case of twitter), thus slower drop,  means corpus has more word diversity.


Tips for a new data scientist I am about to start a job in which I will be working with large datasets and will be expected to find trends, etc...  I have found lots of resources on where to learn ML and other hard skills and feel that I am (semi) competent on this end.

I am interested in knowing if there are specific soft skills that are helpful as a data scientist.  What are things you wish you knew starting out?

While Kaggle is very useful when learning, it also presents clear objectives.  How do you handle being given a dataset, but no clear objective?

Let me know if this is too broad, I can think of more specific questions.

I think there are a lot of important soft skills to consider in the Data Science domain.

Here are some of them:

Know for a fact what the goal is, spending a lot of time on data wrangling, models, visualization and reports when it was not all for the specific goal in mind is a waste. Communicating with less technical people is a skill in itself.
Iterate repeatedly with the product owner. Keep making sure you are on the right path.
If the data doesn't tell the story they thought/want tell them it is not the case, be clear in why this is happening, what biases might be playing a role etcetera. Do not apply all kinds of filters or keep changing parameters to get the desired results.

Regarding your second question:

The objective has to be either gotten from the product owner explicitly or derived from a less mathematical objective. An example could be where you need to predict train arrivals based on some features. They want the model to predict as many times as possible within a 10-minute error range. This is relatively explicit.

Sometimes it is less clear than that, they might say we need it as accurate as possible. Then you will have to decide what to optimize, in some cases, this will just be minimizing the MSE but in other cases, other things might make more sense for your case. Usually, this will be clear from the implicit objective and something that you will get better at with more experience. Both implicit and explicit objectives derive from clear communication with the product owner.


Where can I find a dataset that contains criminal case sentencing data? I would like to study a dataset where each record represents a criminals case in the US and contains attributes such as:

Type of crime
Defendant Age/Sex/Race
Plea
Verdict
Sentence

Is there a dataset like this?

Probably you checked this already, kaggle.com crime tag:
kaggle crime tag
Where yo have some interesting datasets, like:
Executions in the United States, 1976-2016


Dimensionality reduction based on value of a variable I have a dataset including 100k high dimensional data (e.g. houses in LA) (dim=100, e.g. house parameters like area, distance to downtown, etc.). Below is the 2-component PCA representation of the data, colored by target value (e.g. housing price).

As you can see, price function has a stochastic behavior and changes rapidly everywhere.

Is there a dimensionality reduction method that takes the target value into account and as a result, it aggregates low priced homes on one side and we can see a gradient of housing prices in the map?
Can manifold learning help in anyway with this task?

I tried Backward feature extraction based on ridge regression to reduce the dimensionality based on their effect on regression results on target value, and then applied PCA and normalized it, but the output was not as desired. Here is the output:

Thanks

Actually the first plot is still better! The first point to mention is that in 2d you lose a vast amount of information i.e. none of these plots are necessarily telling you something. If the 2d plot is good you can be happy but if not, you should not necessarily get disappointed! I strongly recommend you to see the performance of your dimensionality reduction by evaluating your regression model.

For 100 dimensions I would first try to find correlation/dependence of features and target (try not to use linear correlation without visualization!). Then, out of all features remove non informative ones. On the remaining features try different dimensionality reduction algorithms (not only PCA, for example NMF if values are all non-negative and/or LLE) with more dimensions. Evaluate your results on your validation set.


Reporting test result for cross-validation with Neural Network I have a small dataset, so I have to use cross validation to report the test result to get a better estimate of the classification result. For some reason, I have to use neural networks to do this.

Because neural networks have their unique quirks e.g finding hyper-parameters, I am using a nested cross-validation. I am dividing up my dataset into 10 folds for cross-validation. Then I am dividing the 9 folds that are for training, again in 10 folds. From those 10 folds, I am using 9 folds to train with different hyper-parameters(in my case it is number of hidden units and dropout rate), and using the other fold to to get the accuracy with different hyper-parameters (kind of like a validation set in the deep learning literature).

Then I am training my model again on all of the 9 folds of the first division of data with the best hyper-parameters I found. Because I am missing out on some data from the initial 9 folds for using as a validation set. Now when I am reporting the test result, I set the epoch number for training on my training data a fixed number of times, and when my network is doing the best on the test set, I stopped the training, saved that model for future use, and report that result. My question is about this last part. Am I doing something wrong on reporting this result? Just to make it clear, I am not tuning any hyper-parameters at this stage. I am just setting the network to stop training on the training data when it reaches the best test result. I think this is a really subtle problem, if it is a problem at all. That is why I am confused.

I am doing this whole thing 5-6 times with different seeds for different divisions of data, and I am only reporting the mean of all of these runs.

Now when I am reporting the test result, I set the epoch number for training on my training data a fixed number of times, and when my network is doing the best on the test set, I stopped the training, saved that model for future use, and report that result. My question is about this last part. Am I doing something wrong on reporting this result?

Technically yes this is incorrect process for reporting an unbiased test metric. This could be a bad over-estimation of performance if cv results are noisy and vary randomly epoch-to-epoch. You should in theory treat the early stopping epoch number same as your other hyper-parameters, discover a good value from the lower-level cross-validation and stick with it.

There is a problem with my suggestion though - the early stopping epoch number is sensitive to training data set size, and you just increased the size when you changed from lower level cross-validation to the higher level one. So you might get an unbiased measure at the expense of significantly worse results.

First, take a look at your learning curves. Just how sensitive are the cv results to epoch number? If they are not sensitive - no obvious over-fit over a reasonable range of epoch numbers - then just pick a mid-range fixed number of epochs that applies to all folds. That way you will have your unbiased estimate, and probably not compromised on model quality.

Alternatively, you may just have to be happy knowing that your test estimate is biased, but you have reduced the variance significantly by using 10-fold CV, and have only searched one hyper-parameter at the top level (not all three). It may only be slightly biased, and still a good estimate. The smoother and less jittery the learning curves are, the more you can get away with this - but sadly you won't get a measure of the bias, so you'll never be 100% sure.


What does it mean to take the "average" of two decision trees by 'voting' I have heard, in relation to random forest algorithm, that the algorithm will fit many decision trees and take the average of them by votes. (This is related to bagging as well)

I understand what the average means for something example such as $\vec{x}=[1,2,3], \; \bar{x} =2 $. But I don't know what it would mean if I had two decision trees.

Could anyone please provide a simple example / explanation of this averaging process for a couple of decision trees?

I think that you are mixing together two different things - random forests
for regression and for classification. Regression means to predict a
continuous value (number). Random forest can construct multiple regression
trees, each of which makes a prediction about the number.  In that case,
it is simple to understand. The numerical predictions are averaged to give
a robust prediction of the true number value.

However, I think that you are asking about classification - predicting a
nominal value (also called categorical or factor). In this case, each
decision tree predicts a category.  Usually, it does not make sense to
talk about averaging categories. Instead, the multiple decision trees
"vote" - that is one counts how many times each category was predicted
and takes the category that received the most votes as the prediction.
There is no averaging, only counting.

Here is a simple example.

Data

V1   V2    V3    Class
A     C     E      X
A     C     F      X
B     C     F      Y
B     D     F      Y
B     D     E      X

Decision Tree 1 uses only feature V1:
If V1 = A, predict X, otherwise predict Y

Decision Tree 2 uses only feature V2:
If V2 = C, predict X, otherwise predict Y

Decision Tree 3 uses only feature V3:
If V3 = E, predict X, otherwise predict Y

Now we want to predict the class of a new point (A, C, F):
 - Decision Tree 1 sees V1 = A and predicts Class=X
 - Decision Tree 2 sees V2 = C and predicts Class=X
 - Decision Tree 3 sees V3 = F and predicts Class=Y
There were two votes for X and one vote for Y,  so the forest predicts X,
the class that received the  majority of the votes.


Is there a way to force a transformer to return a pandas dataframe? I am having issues with scikit-learn converting dataframes to numpy arrays.

For instance, the following code

from sklearn.impute import SimpleImputer
import pandas as pd

df = pd.DataFrame(dict(
    x=[1, 2, np.nan],
    y=[2, np.nan, 0]
))

SimpleImputer().fit_transform(df)

Returns

array([[1. , 2. ],
       [2. , 1. ],
       [1.5, 0. ]])

Is there a way to use an imputer that returns a pandas dataframe instead of a numpy array? Is there a scikit-learn implementation for that? I am aware of sklearn-pandas, but the interface is kind of different.

Might be late but for anyone with the same question the answers (as almost everything with Scikit-lear) is the usage of Pipelines
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import FunctionTransformer
from sklearn.pipeline import Pipeline
import pandas as pd

df = pd.DataFrame(dict(
    x=[1, 2, np.nan],
    y=[2, np.nan, 0]
))

imputer = Pipeline([("imputer", SimpleImputer()),
                    ("pandarizer",FunctionTransformer(lambda x: pd.DataFrame(x, columns = ["x", "y"])))])

imputer.fit_transform(df)


Classification when variables are in ranges I want to classify my data and some of my variables are ranges.

I classify location so for example, school, the hours that people are at school are from 7:00 to 14:00, some of my variables are categorical (working day) and some of them numerics(frequency of visiting in a month).

I thought using LDA but how can I declare a range as one variable?

btw, I use Python

Example:

place         visiting time    frequency    workday
school         8:00-14:00        18-20           1
restaurant    13:00-21:00        0-3            both
bank          8:00-17:00         0-4             1
night club     21:00-2:00        0-4            both

The algorithm should be supervised because I insert the ground truth (the place) by myself.

TIA


How to define the number of features to select in RFECV? I am trying to work on feature selection stage for my dataset.

I am a newbie to ML. I have around 60 columns and am trying to select top 15 features. I came to know about RFECV for which I wrote a code like as shown below. I am aware that n_features is present for RFE but it is missing for RFECV. Is there anyother way to assign the number of features to select?

model  = RandomForestClassifier(n_estimators=100, random_state=0)
# create the RFE model and select 15 attributes
rfe = RFECV(model,step=5, cv=5,min_features_to_select = 15,max_features_to_select = 15) # this doesn't work. `n_features=15` also doesn't work
rfe = rfe.fit(X_train_std, y_train)
# summarize the selection of the attributes
feat = rfe.support_
fret = rfe.ranking_
features = X.columns
print(features[feat].tolist())

Can someone help me to get top 15 features only? Where can I configure the n_features paramter?

Currently it displays more than 30 features. I don't really know how or from where does it get its number (30)?

That's the point of RFECV over RFE: the former selects the best number of features through cross-validation.  If you want 15 features, use RFE instead (or some other feature selection method).

From the API docs,

  cross-validated selection of the best number of features

and from the User Guide

  RFECV performs RFE in a cross-validation loop to find the optimal number of features.

To respond to the comments (my response is a bit too long for comments):

Yes, RFECV is meant to produce the optimal number of features.  RFE is run from the full feature set down to 1 feature on each of the cross-validation splits, then those models are scored on the test folds and averaged; then the best-scoring number of features can be taken and then RFE is run again down to that number of features.  The thought is that (barring some special domain knowledge) RFECV is better than RFE, except that it may take rather longer to run.  By definition, the RFECV's accuracy (on the CV-splitted training set) will be better than RFE with any other fixed number of features.

Now, the usual caveats apply.  The number of features returned by RFECV may not necessarily generalize the best to unseen data, especially if that data isn't perfectly iid with the training data.  Especially with small datasets, the splits made in the CV may affect the results, and perhaps going from the smaller CV train sets to the whole train set should allow more (or fewer?) features.  And the top features on each of the splits may not be the same, so again when going to the full training set the question of increasing (decreasing?) number of features is reasonable.


Any idea about application of deep dream? Recently Google publicized interesting deep dream. Besides art generation such as http://deepdreamgenerator.com/, do you see any potential applications of deep dream in computer vision or machine learning?

There's already at least one application out, if you interpret 'application' broadly enough: Decoupled Deep Neural Network for Semi-supervised Semantic Segmentation by Hong, Noh and Han. They use it for image segmentation. Standard image recognition networks can only give you a bounding box for each object recognized on an image. If you want to know which pixels constitute that object, you have to do image segmentation.

Basically, after finding a dog on an image, Hong et al's architecture back-propagates the dog-ness through the neural network down to the pixel level, to find the pixels that were the most responsible for the dog appearing. (They then use this heatmap as input for a supervised segmentation network, there's no deep dreaming in that part.)

This is already kind of an existence proof that the Deep Dream idea can be useful outside image manipulation. But I wouldn't downplay image manipulation itself either. I mention two things that are not immediate applications of Deep Dreaming, and we don't have them currently, but I can kinda see a plausible road from the original Deep Dream algorithm towards these:

Beautifying pictures and human faces and bodies. (Automating what a Photoshop retouch artist does.)
CSI-style image upscaling with fake but believable interpolated detail.


How to predict customer's next purchase Suppose we want to predict what customer will buy during his next visit to the Electronic Shop based on his past purchase history. I know that it is a very broad question, but I am new to machine learning and don't have much idea about how to approach this problem.

The simplest thing that comes to my mind is to find the most frequent items that customer has bought and suggest it. However, I don't think that this is a very robust approach as it doesn't consider this scenario:

Computer (1st Purchase) -> Mouse (2nd Purchase) -> Mouse Pad (3rd Purchase)

I am looking for a simple model to get started and scale in terms of features and training data. I would love to hear suggestions of experienced Data Scientist as it is a most common problem.

Thank you.

Take a look at association rule learning (https://en.wikipedia.org/wiki/Association_rule_learning). A really common algorithm is the Apriori agorithm. You could use the package apyori, it works great: https://pypi.python.org/pypi/apyori/1.1.1


What is the underlying pattern, or algorithm(s) used in these numbers? I have 11 lottery tickets (used) and I have discovered that in each ticket, the 3rd digit's value is +1 of the value of the 6th digit. I have 11 tickets, each ticket is composed of 16 digits. Would someone (anyone) help me find the logic used here, in creating the other digits and their interrelation? I am a complete noob in data analysis, and any help would be greatly appreciated. For those who would like to know, I incidentally discovered the certain pattern, and made me realize these numbers are not totally random and if I could find the underlying pattern I'd be able to predict. The excel file is here Excel file

11 instances of data is insufficient for discovering the behavior (even finding patterns) for such presumably complex systems. You need at least thousands of them.
These numbers are encoded versions of numbers generated by Serial Key Generation Algorithms, which in my experience are usually implemented by Python. So it's not at all strange that there may be a correlation between some digits of them. As it is necessary for the system to recognize its created numbers from the numbers created by other lotteries.
Even if there was a correlation or correlation between all the instances of lottery numbers, correlation doesn't imply causation and these patterns, if found any, couldn't really give any information about the key generation system itself.
If you are really serious about it. you could go to this page and enter your data as a sequence it would show you the pattern, if there is any:
The On-Line Encyclopedia of Integer Sequences


Logistic Regression Model for categorical features with multiple values in each category I am working on an insurance use case to build a logistic regression classifier to predict if a policy will lapse or not. The dataset has more than 20 categorical features for a policy. Each categorical feature in itself can have multiple values out of which only one will be applicable to a policy at a time. e.g., there could be multiple ways of premium payment (like annually, half-yearly, quarterly or monthly). The policy will have either of these value attached to it. As part of feature reduction, I used chi-square test against my target feature (if a policy will lapse or not) and removed a few features. After this, I created dummy variables (0 or 1) for all of them and tried to find a correlation using .corr() function in pandas. This operation helped to identify some dummy features have a high correlation with target feature e.g a specific value of product type, renewal type, source of business. Using these dummy features variable I trained my logistic regression classifier which is giving an accuracy of 94%, precision of 81%, recall of 98% and area under AUC value of 0.98, which I believe is a case of model overfitting.

I believe, if I use this model on a policy which does not have categorical feature values on which it is trained, it will fail miserably.

Please guide me if it is correct to use individual values of categorical feature to train the model? And if this value is not present/applicable to a new data then the model will not be able to predict with a high level of accuracy. Please guide how is feature selection done in such a case and how should the model be trained. Will appreciate if link to similar implementations could be provided.

Looks like this is a great problem for catboost. Try it out

I believe, if I use this model on a policy which does not have categorical feature values on which it is trained, it will fail miserably. ANY data set that has big covariate shift will fail miserably.

NOTE read through docs of catboost there is part for feature selection.


LSTM for prediction of next location step - help with standardization I have a few questions regarding the topic and I hope someone might have experience with any of them.

What I am trying to do is train an LSTM network, whose input is a sequence of N steps in a XYZ space (i.e 3 features over N point per sample, each point is part of some coordinate in space) and i want to predict the next point in the same XYZ space.

Note: Not all samples are from the same DB, meaning the XYZ space varies between some samples/

My questions are:

Say I want to use min-max scaling, do I scale across ALL samples at once or scale per XYZ space? say i have 100 samples from XYZ_A and 50 from XYZ_B, do i take my min/max from 150 point of both space or not?
should I also scale my output labels? and if so should I use them in the initial scale or should I only scale on my train inputs and use that scaler on my outputs?
When I want to make a prediction after training, should I scale the data according to my training data?

OR since my features are relatively from the same domain should I skip scaling all together?

Say I want to use min-max scaling, do I scale across ALL samples at once or scale per XYZ space? say i have 100 samples from XYZ_A and 50 from XYZ_B, do i take my min/max from 150 point of both space or not?

Even though your samples come from different databases, they represent the same physical quantity, i.e. 3D coordinates. Since the units of measurement are the same for the samples of both databases, you should apply the scaling over all 150 samples, not separately.

  should I also scale my output labels? and if so should I use them in the initial scale or should I only scale on my train inputs and use that scaler on my outputs?

I wouldn't use the word labels, since you are predicting the next XYZ coordinates, therefore it is a regression problem, not classification. However, you should also apply the scaling function on those values too, since they have the same units of measurement as the input.

  When I want to make a prediction after training, should I scale the data according to my training data?

Exactly, usually you fit a scaling function to your training dataset and you apply the same (already fit) function on the testing dataset.


Visualization of an Order Verified by Wilcoxon The background: After I've conducted a survey about preferences of users in ridesharing, I now want to construct an order of importance of these preferences (this is to answer one of my research questions). For instance price might be more important than comfort of the vehicle might be more important than the played music. I collected the data based on a five-point likert-scale.

My procedure so far:

Of course clearing the data
Generate an order via a simple approach: very important gets $4$, important gets $3$ and so on. Then I build a sum over all answers and I sort the preferences based on their sums.

The second step provides me with an order, which is not statistically verified. Therefore,

I perform a one-sided Wilcoxon signed-rank test for each pair of preferences.

The resulting p-values are in the following matrix:

The visualization can be read as follows: preference a is statistically proven to be not less important than preference d to j for all common significance levels. (The order of preferences comes from the second step.)

The problem is, I've collected $42$ preferences and the corresponding visualization has the following problems:

$42$ is quite a lot and the visualization gets confusing.
Moreover, to understand what is shown in the graphic a lot of brain work is needed, especially for an unexperienced user.

Do you have an idea how to visualize the result better?

I do not have a good way to answer your question since I do not know who your audience is, what actions/understandings are you trying to drive from your analysis, etc.

I think that is where you need to start. If your audience is executives in charge of revenue, perhaps showing the difference in revenue where the top n preferences are in a car vs n-1, n-2, ....  Or frequency of the top preferences.

Given the audience and the action/understanding you may not need to show all 42 preferences in the mainline of your presentation.

You can also look for inspiration on your audience/action from traditional statistical methods for this type of analysis.  Namely conjoint part of choice modeling. Take a look at some of those public analysis or classes and maybe an idea will take hold that is similar to your audience/action.

This is more of a path forward rather than an answer but I do not see the question as answerable right now.


help understanding nested cross validation From what I read online, nested CV works as follows:

I divide my whole data in k folds, where k-1 folds are the train set and one fold is the test set.
There is the inner CV loop, where we may conduct a grid search using the train set to find the best hyperparameters for a model.
There is the outer CV loop, where we measure the performance of the model that won in the inner fold, on the test set.
We repeat the above procedure for different test and train sets until, at some point, all folds got his place as a test set once.

what I cannot understand is that because we find the hyperparameters in each outer loop we run, we might have models with different hyperparameters being tested in the test set in each loop, so can we use this nested cross validation to find the best hyperparameters for a model? or the goal of this evaluation is to find the best algorithm, like SVM or Naive Bayes, for a data set? because, if we are getting different hyperparameters in each loop, we cannot say which one is the best one.

English is not my first language, if it my text is hard to understand, please tell me, so I can fix it.

Generally, you consider the outer CV as just estimating the performance of the method of fitting.  You don't expect a final model out at this point.  You can then apply the same method of fitting a model (what was happening in the inner CV) to the entire dataset to produce a model, and you say that you expect the performance estimate to apply to this new model.

https://stats.stackexchange.com/a/65158/232706
https://datascience.stackexchange.com/a/16856/55122
https://stats.stackexchange.com/q/11602/232706
https://stats.stackexchange.com/q/52274/232706


Multidimensional scaling producing different results for different seeds I took the data from here and wanted to play around with multidimensional scaling with this data. The data looks like this:

In particular, I want to plot the cities in a 2D space, and see how much it matches their real locations in a geographic map from just the information about how far they are from each other, without any explicit latitude and longitude information. This is my code:

import pandas as pd
import numpy as np
from sklearn import manifold
import matplotlib.pyplot as plt

data = pd.read_csv("european_city_distances.csv", index_col='Cities')

mds = manifold.MDS(n_components=2, dissimilarity="precomputed", random_state=6)
results = mds.fit(data.values)

cities = data.columns
coords = results.embedding_

fig = plt.figure(figsize=(12,10))

plt.subplots_adjust(bottom = 0.1)
plt.scatter(coords[:, 0], coords[:, 1])

for label, x, y in zip(cities, coords[:, 0], coords[:, 1]):
    plt.annotate(
        label,
        xy = (x, y),
        xytext = (-20, 20),
        textcoords = 'offset points'
    )
plt.show()

Most of the cities seem to be around the correct general location relative to each other, except a few infractions - Dublin is too far away from London, Istanbul is in the wrong location, etc. However, if I give a different random_state value, it produces a different "map". For example, random_state=1 produces the following map, where many of the cities do not seem to be around the correct general location relative to other cities:

What I don't understand is, dimensionality reduction methods are not supposed to have randomness associated with them, and thus should not give different results for different seeds. But it does here; so what does it mean?

The documentation of the sklearn.manifold.MDS function states that random_state is "the generator used to initialize the centers". So, in particular, I guess what I'm asking is, whatever initialization of the centres we choose, shouldn't all of them lead to one unique result?

I get a much more "accurate" map (to my eyes at least) by giving the following hyperparameter values:

mds = manifold.MDS(n_components=2, dissimilarity="euclidean", n_init=100, max_iter=1000, random_state=1)

I think the answer to your problem is to understand that euclidian distance matrices are too much to uncover the information. Basically there is a lot of underlying constraints to verify (n*(n-1)/2 constraints for 2n variables).

One way to solve that kind of problem in human acceptable time is to consider 'physical' approaches. Basically you put tiny springs and other forces between your data points and solve a physical system problem. Thats why you usually see graphs 'giggle' when plotting them. As you can guess, these kind of approaches depend heavily on the initialisation of points in space. That's why there is a 'random' component in usual technics.


How to deal with count data in random forest I am working on a classification model where my target class is a biased class with the class shape as

    0     1
 20694   101

Most of my features are the count of number of times a certain event was triggered. While exploring these features I found that my target variable is only associated with certain values of features. For example as below

                 0         1
Feature V1
0                12014    75
1                6490     16
2                1177     6
3                402      2
4                176      2
5                100
6                84
7                61
8                39
9                23
10               26
11               14

As we can see that 1 only occurs when V1 has value of 0 to 4.Thus for any unseen data my model would always predict 0 whenever V1 has value greater than 4.

I thought of using bestNormalize package, however the transformations it is suggesting looses correlation when applied to the data.

Any suggestion would be of great help.

Thanks a lot in advance!!

P.S. Happy to share the data if required.

Some thoughts:

Your data is highly imbalanced. This is a critical issue which should be dealt with. Possible solutions include simple under-/over-sampling to more complicated synthetic approaches like SMOTE.
Decision trees and random forests do not require feature scaling - this means that normalisation is not needed (unless perhaps you plan on using some other modelling technique which uses regularisation).
Just because this data set only shows a particular relationship between values of V1 and target does not mean it is always the case - especially if your model is to be deployed for a period of time. The relationship may change over time so do not rush to artificially curtail your model.