What is a good method to generate the KS-statistic in R? Kolmogorov-Smirnov (KS) statistics is one of the commonly used measures to assess predictive power for marketing or credit risk models.

The KS statistic is usually published for logistic regression problems to give an indication of the quality of the model.

A Wikipedia page (https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test) gives this explanation:

The KS statistic calculation below is from page 5 on https://cran.r-project.org/doc/contrib/Sharma-CreditScoring.pdf

require(ROCR)
set.seed(7)
prd=runif(1000)
act=round(prd)
prd[sample(1000,500)]=runif(500) #noise

pred<-prediction(prd,act)
perf <- performance(pred,"tpr","fpr")
#this code builds on ROCR library by taking the max delt
#between cumulative bad and good rates being plotted by
#ROCR see https://cran.r-project.org/doc/contrib/Sharma-CreditScoring.pdf
ks=max(attr(perf,'y.values')[[1]]-attr(perf,'x.values')[[1]])
plot(perf,main=paste0(' KS=',round(ks*100,1),'%'))
lines(x = c(0,1),y=c(0,1))
print(ks);
auc <- performance(pred, measure = "auc")
auc <- auc@y.values[[1]]
print(auc)

It gives a KS of 0.4939511 and an AUC of 0.7398465. Area under the curve is obviously correct as one can see.

Is the R code for the KS statistic correct?
If not, what should it be?

Update

Warning: From a sample decision tree output which is very clumpy, the Sharma method gives a better estimate of the KS score while the ks.test gives a very bad one.

     auc         ks.score.D^+ sharma.ks
    0.6153846    0.7045455    0.2307692

The ks.test function gives this error: 1: In ks.test(prd, act, alternative = "greater") :  cannot compute exact p-value with ties

act=c(1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1,
  0, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0,
  0, 0, 1)
prd=c(0.914893617021277, 0.914893617021277, 0.914893617021277,
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277,
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277,
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277,
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.352941176470588,
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277,
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277,
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277,
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277,
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277,
  0.914893617021277, 0.352941176470588, 0.352941176470588, 0.914893617021277,
  0.914893617021277) ##this only has two values
ks.test(prd, act, alternative='greater')$statistic
#gives 0.7045455 which clearly isn't correct

Also consider using the builtin ks.test:

> ks.test(prd, act, alternative='greater')

        Two-sample Kolmogorov-Smirnov test

data:  prd and act
D^+ = 0.494, p-value < 2.2204e-16
alternative hypothesis: the CDF of x lies above that of y


Python & Pandas : (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'localhost' (timed out)") I'm writing a Python Script to store JSON data into MySQL Database. I used pandas to store into MySQL Database.

I used two different modules (MySQLdb and sqlalchemy) to connect to MySQL dtaabase.

Python Code:

user@sys:~$ cat pandas_to_mysql.py
import pandas
from pandas.io import sql
from sqlalchemy import create_engine
import MySQLdb

json_string_final='[{"id": 772, "name": "abcd"}]'
df=pandas.DataFrame(eval(json_string_final))
print(df)

#----Using MySQLdb---
con=MySQLdb.connect(host='localhost',user='user1',passwd='pwd',db='myDB')
try:
    df.to_sql(con=con, name='myTable1', if_exists='replace')
except Exception as error1:
    print("Error with connection 1 ------> ")
    print(error1)

#----Using Engine----
engine = create_engine("mysql+pymysql://{user}:{pw}@localhost/{db}".format(user="user1",pw="pwd",db="myDB"))
try:
    df.to_sql(con=engine, name='myTable2', if_exists='replace')
except Exception as error1:
    print("Error with connection 2 ----->")
    print(error1)

I used two diffrent connections to store the data into MySQL dtabase. But connection was not established. Below is the error message.

Output:

user@sys:~$ python3 pandas_to_mysql.py
    id  name
0  772  abcd
Error with connection 1 ------>
Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': not all arguments converted during bytes formatting
Error with connection 2 ----->
(pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'localhost' (timed out)")
(Background on this error at: http://sqlalche.me/e/e3q8)

But i'm able to connect to MySQL database manually from the same terminal using the following command.

MySQL Connection:

user@sys:~$ mysql -u user1 -h localhost -D myDB -p

It means, there is no issues with host, user, password, database name. Then what is the issue?

The following works for me:

df.to_sql(con="mysql://user:password@localhost/database_name, name='myTable2', if_exists='replace')


When using Scikit Learn Grid Search, why are my train and cv scores high, but my test score is a lot lower? I'm using scikit learn to run some models, and am very confused as to why my test score is so much lower than my cv score and my train score.
At the start, I do a 80-20 train-test split. On the train set, I run a gridsearch with 5-fold cross validation to choose hyperparameters. refit is set to true, so after picking hyperparameters the model is refit onto the whole training set, and used to predict the test set.
When I look into cv_results_ I find that my mean_train_score (what I'm interpreting to be the train score for each k-fold cross validation loop) is really high. When I look at the mean_test_score (what I'm calling cv score), it is also really high. But then when I use my external test score, the scores are really low. This is true for all models I'm using (I'm testing 10 models). The numbers can be seen in the following picture.
Note: I'm using F1 Macro Score as measure of model performance.

(LR) Logistic Regression, (QDA) Quadratic Discriminant Analysis, (NN) Nearest Neighbors, (LSVM) Linear Support Vector Machine, (RBFSVM) Radial Basis Function Support Vector Machine, (NB) Naive Bayes, (ANN) Artificial Neural Network, (RF) Random Forests, (AB) AdaBoost Random Forests, (GB) Gradient Boosted Random Forests
So since my test set performance is a lot lower than my training score, I'm sure I'm overfitting. But I don't know why my CV score would do so well then? If my setup is prone to overfitting, wouldn't I see overfitting with the 4/5 of my train set when I did the 5-fold cross validation, meaning my CV score would be low too? I don't see why I wouldn't overfit leading to high CV scores in the 5-fold CV step, but overfit for low performance in the testing set.

Here are some ideas of things to try:

I would try to investigate the problem by not using the 'refit' option. Run the Grid Search CV yourself, obtain the best parameters and train a new model on all of the training data after you input those best parameters. This is just to  make sure there is nothing funky going on with SKlearn.
Try using another metric for classification like AUC and see if there is any difference.
Also, it might be helpful if you provide more information about your data, such as the type of outcome and the type of predictors you are using


How to run a model in an application using gpu (without CUDA) I am looking for a way to utilize a computer’s gpu without using cuda (or any installable software). The reason for this is I have an application where the neural network runs on a user’s computer and I can’t assume they will have the knowledge/ spend the time and effort to install CUDA. I see that it is possible to utilize the gpu using WebGL but from my reading, it sounds like due to other limitations with this approach it is not actually faster than using cpu. Does anyone know how to do this maybe using openCL or some other way to do it?

I'm pretty sure that you will need CUDA to use the GPU, given you have included the tag tensorflow. All of the ops in tensorflow are written in C++, which the uses the CUDA API to speak to the GPU. Perhaps there are libraries out there for performing matrix multiplication on the GPU without CUDA, but I haven't heard of a deep learning framework that doesn't use CUDA, when executing on the GPU.

What you can perhaps do it find a solution that works using Docker. The latest version doesn't require any additional Nvidia software to be installed. It would mean your customer doesn't need to mess around with CUDA themselves, they just need docker installed, which is generally less than 5 minutes work. Here is the Tensorflow documentation for a starter.

In addition, using Docker is perhaps the most professional and isolated way to deliver code to somebody else's machine. There are many benefits and it works on Windows, Mac, Linux. Any changes you make can simply be pushed to their machine, without them having to change anything. It can also scale arbitrarily to multiple machines, running locally or in the cloud, the list goes on and on...


How to justify the usage of 200 dimensions in word vectors instead of the 300 dimensions? When employing machine learning methods in NLP, most of studies use 200 or 300 dimensional vectors. 300 dimensional embeddings carry more information and this, therefore, is considered to produce better performance results in general.

If you have unlimited computational resources and the training time is not a problem for you, when does it make sense to use 200 dimensional embeddings instead of 300 dimensional vectors in a classification problem (e.g. in sentiment analysis) and why?

I am assuming that you are using off-the-shelf word2vec, GloVe, or other pretrained vectors. That is, the vectors are not being learnt from scratch in your classification task.

Dimensionality around word vectors is already answered here : https://stackoverflow.com/questions/45394949/what-is-dimensionality-in-word-embeddings

The reasons 200-300 dimensions are chosen normally, is that they have seen it produce the results that are very close to or equivalent to when they have chosen higher dimensions.

When it comes to training time, the time taken to generate word embeddings for 200-300 dimensions are not significantly different for most of the experiments I have personally come across.


avoiding premature convergence with neural networks (EA's) I am currently writing a program that would be able to play snake on an 25*25 grid. It works by optimizing a set of weights of 300 different solutions (each solution would be a different neural network) with the aid of an evolutionary strategy, thus by random mutation and parent selection. I have decided not to apply crossover on the parents pool due to the black box problem concerning MLPs and other neural networks. My population size is 300 and 10 parents are selected every generations (200 gen. total). Every other solution which is not a parent gets deleted to make place for 290 new mutated solutions based on the parents.
My network structure (MLP) is the following : 6 input nodes, (10,10) hidden nodes, 3 output nodes.
With my current version of the programme, I am able to reach a steady score of 60 apples (or points) eaten (around 15/200 have a score above 60). And this only after 50 generations! But now, it seems like i have bumped into a local minima that I am not able to overcome. Does anyone have any idea if further progress is still possible or that i might have hit the ceiling? The quickly elevating performance might be a sign that my exploration/eploitation balance might not be ideal, but with each solution containing 190 weights, more exploration would seem too computationally expensive and would take forever on my i7 10gen laptop :/.
Can i call this a premature convergence and is there a way to deal with this, or have i hit the limits of my idea?

You can take a look at dropout, it will get you out of the local minima you are stuck in.


Segmenting pandas dataframe with lists as elements I am working with a pandas DataFrame that contains a combination of numerical and categorical data on customer demographics and preferences. Each row contains information about an individual customer. df has 5000 rows and 30 columns.
df = pd.DataFrame({'col1':list('mno'),
                    'col2':[['a','b'],['d'],['a','b','c','d']],
                    'col3':[1,2,3],
                    'col4':[[1,3,4],[5,6,7],[8,3]]})
df

   col1     col2              col3        col4
0   m    ['a', 'b']            1        [1, 3, 4]
1   n    ['d']                 2        [5, 6, 7]
2   o    ['a', 'b', 'c', 'd']  3        [8, 3]

I am hoping to use k-prototypes to segment customers into distinct groupings since there is no ground truth on class labels and I have mixed data. However, I am unsure how to effectively analyze the columns that have lists of multiple values, e.g. col2 and col4. I looked into one-hot encoding to break up those columns into multiple dummy columns, e.g. pd.get_dummies, but I'm concerned that creating a sparse matrix may negatively affect my results. Can anyone suggest a better way of handling this problem?

Are you concerned that a column might have too many values and that one hot encoding will produce features with less predictive power? In that case you could select a handful of the features for more prevalent class values and keep them as is, while combining the other features for rarer class values into one feature.


(pre-trained) python package for semantic word similarity I am searching for a python package that calculates the semantic similarity between words. I do not want to train a model (what most packages seem to offer) - the package should have been pre-trained on ideally thousands of natural language books and documents (e.g. on how often do words occur in close proximity to each other in the training material) and be simple to install/use. As for example in pseudo code below:

import XYZ

assessor = XYZ.loadPreTrainedModel("standard_text")
assessor.scoreWords("pilot", "airplane")  # returns 0.94 (I made up these numbers)
assessor.scoreWords("student", "university")  # returns 0.91
assessor.scoreWords("cat", "dog")  # returns 0.82
assessor.scoreWords("cat", "airplane")  # returns 0.13
assessor.scoreWords("student", "apple")  # returns 0.25
...

The spaCy Python package might work for you.  It allows you to easily "install" large pre-trained language models, and it provides a nice high-level interface for comparing word vectors.

To install spaCy:

pip install spacy

Then you need to download a language model.  I believe these models are trained on Common Crawl, which is a massive dataset.  You should choose one of the medium or large models, because the small models do not ship with word vectors.

python -m spacy download en_core_web_md

Using spacy models to compute word similarity is a breeze:

import spacy

# load the language model
nlp = spacy.load('en_core_web_md')

word1 = 'cat'
word2 = 'dog'

# convert the strings to spaCy Token objects
token1 = nlp(word1)[0]
token2 = nlp(word2)[0]

# compute word similarity
token1.similarity(token2)  # returns 0.80168

Here's an example that's more similar to the one in your question:

import spacy

nlp = spacy.load('en_core_web_md')
token = lambda word: nlp(word)[0]  # shortcut to convert string to spacy.Token
score_words = lambda w1, w2: token(w1).similarity(token(w2))

score_words("pilot", "airplane")      # 0.5998
score_words("student", "university")  # 0.7238
score_words("cat", "dog")             # 0.8017
score_words("cat", "airplane")        # 0.2654
score_words("student", "apple")       # 0.0928


How to deploy ML models in production? Have been learning Machine Learning concepts and doing hands-on past few months. Now got a bit different doubt than implementation of code.

This is regarding measures to be considered while deploying in a Prod environment, say something like "A newbie's checklist to ML production" example -

Are Feedback loops always considered?
Is cross-validation done in final prod model or only during train/test the model?
How the auto-suggestion in Amazon or any ecommerce website works? - Meaning does the model get trained on every page load or already pre-trained and just predicts (suggests) based on user selections?

I know the questions are too general and newbie, but will help many to understand the concepts relating to real-world. Please advise.

Let me know for any further information and I will be happy to furnish/respond.

I got some pointers (below links) to my doubts. Hope it helps other future readers as well.

https://www.quora.com/How-do-you-take-a-machine-learning-model-to-production/answer/H%C3%A5kon-Hapnes-Strand
https://www.quora.com/Do-most-machine-learning-algorithms-run-in-batch-or-do-they-run-every-time-they-get-a-new-bit-of-data/answer/H%C3%A5kon-Hapnes-Strand

Will keep updating this space as and when I get some good answer(s) regarding this.


Converting paragraphs into sentences I'm looking for ways to extract sentences from paragraphs of text containing different types of punctuations and all. I used SpaCy's Sentencizer to begin with.
Sample input python list abstracts:
["A total of 2337 articles were found, and, according to the inclusion and exclusion criteria used, 22 articles were included in the study. Inhibitory activity against 96% (200/208) and 95% (312/328) of the pathogenic fungi tested was described for Eb and [(PhSe)2], respectively. Including in these 536 fungal isolates tested, organoselenium activity was highlighted against Candida spp., Cryptococcus ssp., Trichosporon spp., Aspergillus spp., Fusarium spp., Pythium spp., and Sporothrix spp., with MIC values lower than 64 mug/mL. In conclusion, Eb and [(PhSe)2] have a broad spectrum of in vitro inhibitory antifungal activity."]

Code:
from spacy.lang.en import English

nlp = English()
sentencizer = nlp.create_pipe("sentencizer")
nlp.add_pipe(sentencizer)

# read the sentences into a list
for doc in abstracts[:5]:
    do = nlp(doc)
    for sent in list(do.sents):
        print(sent)

Output:
A total of 2337 articles were found, and, according to the inclusion and exclusion criteria used, 22 articles were included in the study.
Inhibitory activity against 96% (200/208) and 95% (312/328) of the pathogenic fungi tested was described for Eb and [(PhSe)2], respectively.
Including in these 536 fungal isolates tested, organoselenium activity was highlighted against Candida spp.,
Cryptococcus ssp.,
Trichosporon spp.,
Aspergillus spp.,
Fusarium spp.,
Pythium spp.,
and Sporothrix spp.,
with MIC values lower than 64 mug/mL. In conclusion, Eb and [(PhSe)2] have a broad spectrum of in vitro inhibitory antifungal activity.

It works fine for normal text but fails when there are dots (.) present in the sentence elsewhere other than at the end, which breaks the whole sentence as shown in the above output. How can we address this? Are there any other proven methods or libraries to perform this task?

There is nothing in SpaCy that you can use out-of-the-box. However, they allow you to use custom components
To solve your problem, I see at least three ways to do it.

NTLK

NLTK allows you to add known abbreviations as exceptions. See this StackOverflow post.

Use a regular expression

Since your problem is that you have some example of dots that shouldn't mean a sentence starts, you could customize a basic regular expression to include that behavior. Here is a stackoverflow answer that could get you started.

Post-processing

You could also use SpaCy's default segmentation, and then merge sentences if they end with a known abbreviation. It's not incredibly elegant, but it will work.


What can be the best dropout value and the FC layer for good accurate predictions? I am retraining the pre-trained model  VGG16 in the last FC layers.
I used the below function .

what can be the best combination of FC layers and dropout values for the best predictions. ?

In the area of Neural Networks there isn't one right value that works for all kinds of situations. You just need to repeat experiments, till you reach a satisfactory value. Ideally Dropout values range from 0 to 0.5 . I would use a dropout value of .3 or .2 in my neural networks. Also, the number of neurons in FC layer, impact the degree of fit in your network. Too small numbers, result in underfit. Too large values, result in overfit. Keep on increasing the complexity of the model and evaluate  it against a validation loss, that will inform you about increasing or decreasing the number of FC layers.


Optimizing co-occurrence matrix computation I am computing co-occurence matrix for a fixed windows size in python using scipy's lil_matrix for storing the counts and computing the counts by sliding the context window over each word and then counting in the window.

Now the code is taking too much time for relatively small corpus size also (100 MB Wikipedia dump). The code is :

def gen_coocur(window_size=5):
        '''
        Generates coocurrence matrix
        '''
        # vocab is precomputed.
        coocur_matrix = lil_matrix((len(vocab)+1, len(vocab)+1), dtype=np.float64)
        for page in self.wiki_extract.get_page():
            # word_tokenize is tokenizer from nltk
            doc_tokens = word_tokenize(page.decode('utf-8'))
            N = len(doc_tokens)
            for token in self.vocab:
                for i in xrange(0,window_size):
                    if (token in doc_tokens[0:i] or token in doc_tokens[i:(i+window_size+1)]) and token != doc_tokens[i]:
                        coocur_matrix[self.vocab[doc_tokens[i]],self.vocab[token]] +=1
                for i in xrange(window_size, (N-window_size)):
                    if token in doc_tokens[(i-window_size):(i+window_size+1)] and token != doc_tokens[i]:
                        coocur_matrix[self.vocab[doc_tokens[i]],self.vocab[token]] +=1
                for i in xrange(N-window_size, N):
                    if (token in doc_tokens[i:N] or token in doc_tokens[i-window_size:N]) and token != doc_tokens[i]:
                        coocur_matrix[self.vocab[doc_tokens[i]],self.vocab[token]] +=1

vocab is a dictionary which maps words -> wordId. How can I optimize this code to run faster?

From easiest to hardest:

Try running it in pypy or numba
Find a faster implementation. Unfortunately I can not recommend one.
Parallelize the loop over the documents. Not so hard since your vocabulary is precomputed. (Even if it weren't you could get away with it using the hashing trick.) Combine this with the first bullet.
Rewrite the inner loop in Cython.
Rewrite the whole thing in a faster language like C++ or Scala.


Why does my Multilayer Perceptron only classify linearly? I wrote a multilayer perceptron in Python. I'm trying to get it to do nonlinear classification. It has two hidden layers, so it should be perfectly capable. Unfortunately, it only seems to be able to classify linearly. I am hesitant to call this "underfitting", because I am yet to properly achieve non-linear classification, even a non-linear classification that is an underfit. In the below graph, I attempted to classify points above and below the sine function, but it just drew a straight line through the dataset.

Points of uncertainty (outputs around 0.5) are in yellow.

I am using the sigmoid as the activation function, which is simply 1/(1+((math.e) ** (-sum))).

I use two input nodes, twenty hidden nodes in two hidden layers (ten each) and one output node. I have a bias on each hidden layer and one on the output node. My input is not normalized.

I feedforward hundreds of random points and calculate the difference between the output and desired. I calculate the mean-squared error and preserve the lowest mean-square error in each generation of my evolutionary algorithm. I won't put that on display here, since it seems to work pretty well in getting the error low, although my MSE function might be incorrect.

d=0
error = 0
while d < (tPoints): # while all the points in the list have not been feedforwarded
    output1 = feedForward(n, [xValsAbove[d], yValsAbove[d]]) # feedforwards a point above the function, returns output (0,1)
    output2 = feedForward(n, [xValsBelow[d], yValsBelow[d]]) # feedforwards a point below the function, returns output (0,1)

    if output1 < 0.99:
        error += (output1)**2
    if output2 > 0.01:
        error += (1-output2)**2
    d+=1
error = error/tPoints
return error # return mean-squared error

My neurons do linear regression lines just fine. Since I have an MLP with a few layers, I wrote an algorithm for feedforwarding. It just takes the outputs of the previous layer as inputs to the next layer.

def feedForward(n, inputs): # (inputs, inputList, hidden1, hidden2, outputs, bias):

    inputList = n.getNeuronList()[0]
    hidden1 = n.getNeuronList()[1]
    hidden2 = n.getNeuronList()[2]
    outputs = n.getNeuronList()[3]
    bias = n.getNeuronList()[4]

    FFInputs = []
    FFHidden1 = []
    FFHidden2 = []
    FFOutputs = []

    for inp in inputList:
        FFInputs.append(inp.feedForward(inp.getWeights(), inp.getInputs()))
    for hidden in hidden1:
        FFHidden1.append(hidden.feedForward(hidden.getWeights(), FFInputs, bias[0]))
    for hidden in hidden2:
        FFHidden2.append(hidden.feedForward(hidden.getWeights(), FFHidden1, bias[1]))
    for out in outputs:
        FFOutputs.append(out.feedForward(out.getWeights(), FFHidden2, bias[2]))

    return FFOutputs

Graph done with matplotlib.

Does anyone experienced in ANNs have any idea why it would do this? Should I only use one layer? I am evolving the weights with an evolutionary algorithm, so should I just do more generations? Should I use a different activation function?

Your error calculation looks wrong:

if output1 < 0.99:
    error += (output1)**2
if output2 > 0.01:
    error += (1-output2)**2

If I am reading this correctly, output1 = 1.0 is a correct classification, since you treat any value above 0.99 as no error at all. However, you then measure the squared error as being output1 ** 2 which means the largest error is at 0.99. This is going to confuse the network, it will treat classifications above the line as perfect 1.0 if they score > 0.99, or otherwise want to adjust weights to make the classification 0.0. The opposite is true for your scoring of points under the line.

You need to either

Swap the conditionals (change < 0.99 to > 0.01 and vice-versa)

or

Replace (output1) with (1-output1) and (1-output2) with (output2)

in order to make the logic self-consistent. Which you choose depends on whether above the line points are positive class, or it is the negative ones. If above the line is positive you should use option 2.

In addition, as comments have pointed out, you might be a little ambitious with your approach. I spot the following things that make your job harder:

Using genetic algorithm search. Whilst it is possible to train small NNs like yours using a GA, it is less efficient. GAs might be a good choice for control systems or for a-life scenarios, but are far behind cutting edge for supervised learning.
Using mean squared error metric for classification. You should use logloss, which more heavily penalises bad guesses. With a mean squared error, there is a higher chance the network will settle for a few bad misclassifications if it also gets some other reasonable ones.
Ambitious function to separate. A neural network should be able to classify your function, but that's a lot of curvature to learn for a starting problem. Reduce it to just a couple of cycles to start with should help you debug practical issues.
No input normalisation. This may lead to saturated values in the network - close to 1.0 or 0.0, making it very hard to discriminate between good and bad weights. You don't show your weight initialisations - large initial weights can also cause this effect (although using a GA may actually help here, depending on how you are mutating weights).


For a linear model without intercept, why does the redundent term in one-hot encoding function as intercept? In this question Elias Strehle pointed out that if we keep all the levels during one hot encoding on a linear model without an intercept, the redundant feature will function as an intercept. Why is this the case?
Isn't that in a linear model, the intercept term ($x_0$, not $\beta_0$) will always have a value of 1? Suppose the feature to encode is gender, then some of the rows of the redundant term will be 0 while others will be 1s.

Let's have a look how dummies work:
R Example:
# Some data
df = data.frame(y=c(30,32,28,10,11,9),gender=c(1,1,1,0,0,0), gender2=c(0,0,0,1,1,1))

# 1) Regression with constant and dummy
summary(lm(y~gender,data=df))

# 2) Regression without constant and dummy
summary(lm(y~gender-1,data=df))

# 3) Regression without constant and two dummies
summary(lm(y~gender+gender2-1,data=df))

Results:
Case 1:
Since dummies generally work as "contrasts" to some base category (1 vs. 0 / "on" vs. "off") and since the base category has a mean of 10, the intercept term equals 10 and for gender = 1, the difference to the base category is identified (here 20) because the mean of category gender = 1 is 30. (Remember that a regression with only an intercept or with dummies simply gives the arithmetic mean).
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  10.0000     0.9129   10.95 0.000394 ***
gender       20.0000     1.2910   15.49 0.000101 ***

Case 2:
Without a constant only gender = 1 is considered, since for gender = 0 we have $0 + 0 * \beta$ so that gender = 0 is dropped. The coefficient now is the mean of gender = 1.
Coefficients:
       Estimate Std. Error t value Pr(>|t|)
gender   30.000      4.546   6.599   0.0012 **

Case 3:
Including a dummy for both groups (denote gender = 0 from above as gender2) without adding an intercept now gives the mean for each group directly. Note that the interpretation of the coefficients is different here compared to case 1.
Coefficients:
        Estimate Std. Error t value Pr(>|t|)
gender   30.0000     0.9129   32.86 5.11e-06 ***
gender2  10.0000     0.9129   10.95 0.000394 ***

The interesting bit is when you add some additional $x$:
Some new data, now including $x$:
df = data.frame(y=c(30,32,28,10,11,9),gender=c(1,1,1,0,0,0), gender2=c(0,0,0,1,1,1), x=c(20,22,25,28,30,29))

Regression with both dummies, no intercept:
summary(lm(y~gender2+gender-1+x,data=df))

Coefficients:
        Estimate Std. Error t value Pr(>|t|)
gender2  19.8864    12.6285   1.575   0.2134
gender   37.6136     9.7446   3.860   0.0307 *
x        -0.3409     0.4342  -0.785   0.4897

is the same as...
Regression with one dummy and intercept (apart of the dummy interpretation discused above):
summary(lm(y~gender+x,data=df))

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  19.8864    12.6285   1.575   0.2134
gender       17.7273     3.1973   5.544   0.0116 *
x            -0.3409     0.4342  -0.785   0.4897

... so the marginal effect of $x$ is the same. This is in contrast to...
Regression with one dummy, no intercept:
summary(lm(y~gender+x-1,data=df))

Coefficients:
       Estimate Std. Error t value Pr(>|t|)
gender 22.38736    1.41677  15.802 9.37e-05 ***
x       0.34086    0.03864   8.822 0.000911 ***

Here the marginal effect of $x$ is entirely different.
Why is this?
When you fit some new data, you will see that the fitted line for $x$ goes through (0,0) "no intercept in the model".
newdata = data.frame(gender=c(0,0,0,0,0,0), x=c(-1,0,1,2,3,4))
predict(lm(y~gender+x-1,data=df), newdata=newdata)

         1          2          3          4          5          6
-0.3408643  0.0000000  0.3408643  0.6817286  1.0225929  1.3634572

This happens because there are cases in which you have $0 + \beta x$ (which is 0 for $x=0$). Or as $x$-matrix (first row would be the intercept, for illustration = 0):
\begin{pmatrix}
0 & 1 & x_1\\
0 & 1 & x_2\\
0 & 0 & x_3\\
0 & 0 & x_4
\end{pmatrix}
However, when you have the two dummies included, you have:
\begin{pmatrix}
0 & 1 & x_1\\
0 & 1 & x_2\\
1 & 0 & x_3\\
1 & 0 & x_4
\end{pmatrix}
So there is no case in which you force $\beta x$ to be zero.
See this post for further discussion on regression without constant term.


Regression vs. Expected value for campaign results Let's say that I have 100k customers, and I can only send 10k letters for a loan, and I do have information from past campaigns and I know I can expect about 100 responders.

If I want to get in my next campaign the best possible outcome in terms of money (i.e. the responders who request the highest loans), what would be the difference between:

1) Creating a regression model using 0 for those who have not responded in past campaigins, and the loan amount for those who did, and select the 10k highest, or
2) Same as above, but also create a classification model and multiply the regression versus the probability of respond, and select the 10k highest

I'm confused as to what would be the proper procedure here
Any help will be greatly appreciated!

There are special types of regression models for these kinds of problems.

The expected loan amount for a given individual is a function of both the probability that they respond and take out some type of loan and the loan amount they take out conditional on them responding.  In mathematical terms, we can think of it as expected value of the loan amount and it can be expressed like this:

$E[loan$ $amount] = Prob(response) * E[amount | responded] + Prob(no$ $response) * E[amount | not$ $responded]$

(It seems like in your case you may be assuming that without receiving a letter an individual has a zero probability of taking out a loan, so you can drop everything after the $+$ operator, if that's the case.)

So you have to model both variables - probability and amount.  The question is how.  If you just go with the first approach you outlined, i.e., a basic regression for modeling counts like Poisson or Negative Binomial and include 0's in the model, then you will be assuming that the zeros and the nonzeros (positive values) come from the same data-generating process, governed by the same data distribution, and are influenced in the same manner by the same set of predictors.  But models like hurdle models and zero-inflated models (e.g., zero-inflated Poisson) separately model the binary outcome of whether a count variable is zero-valued or positively-valued and the conditional distribution of positive values.

(If you want to know more hurdle and zero-inflated models, here is a couple of links to Q&A's on the Cross Validated site:
https://stats.stackexchange.com/questions/81457/what-is-the-difference-between-zero-inflated-and-hurdle-models, https://stats.stackexchange.com/questions/279273/zero-inflated-distributions-what-are-they-really.)

You could also manually construct those separate binomial (logistic) and count models (using separate sets of predictors if you wanted) and try to combine the results.  One issue though is that you would need to model amounts only for those individuals who have previously responded (as you are looking for expected amount conditional on responding) and it would be difficult for you to calculate/estimate expected amounts for individuals who have not previously responded and for any new customers.


DecisionTreeRegressor under the hood of GradientBoostingClassifier I'm inspecting the weak estimators of my GradientBoostingClassifier model. This model was fit on a binary class dataset.
I noticed that all the weak estimators under this ensemble classifier are decision tree regressor objects. This seems strange to me intuitively.
I took the first decision tree in the ensemble and used it to predict independently on my entire dataset. The unique answers from the dataset were the following:
array([-2.74, -1.94, -1.69, ...])

My question is: why and how does the gradient boosting classifier turn the weak estimators into regressor tasks (instead of classification tasks) that are not bound by 0 and 1? Ultimately the GradientBoostingClassifier outputs a pseudo-probability between 0 and 1: why aren't the ensemble of weak estimators doing the same?

After reading through more of the documentation I found the section that covers the classification case. It can be found here. Additionally this statquest was very useful.

1.11.4.5.2. Classification
Gradient boosting for classification is very similar to the regression case. However, the sum of the trees
is not homogeneous to a prediction: it cannot be a class, since the trees predict continuous values.
The mapping from the value
to a class or a probability is loss-dependent. For the deviance (or log-loss), the probability that
belongs to the positive class is modeled as
where  is the sigmoid function.
For multiclass classification, K trees (for K classes) are built at each of the  iterations. The probability that
belongs to class k is modeled as a softmax of the
values.
Note that even for a classification task, the
sub-estimator is still a regressor, not a classifier. This is because the sub-estimators are trained to predict (negative) gradients, which are always continuous quantities.


Weight pruning of CNN I was confused when i was reading about weight pruning on CNN. Is it applied for all the layers including convolutional layers or only it is done for dense layers?

The short answer is both.

Filter pruning

One approach is to remove filters that are "less contributing" to the overall learning every epoch or so. Defining "less contributing", that's the interesting bit about this research. For example it could be based on the running mean average of each filter, and remove filters of CNNs that have smaller mean. Some other research out there,

Using L1 norm to prune filters paper
This article list several other approaches (https://jacobgil.github.io/deeplearning/pruning-deep-learning)

Dense layer pruning

Obviously the parameter bottleneck of pretty much all CNNs happens at the first dense layer connected to the output of the last convolution layer. Therefore, doing proper pruning on this layer will be quite effective.

I couldn't find any good papers that introduces parameter compression for dense layers. But I'll update my solution if I come across any.

Knowledge distillation

Another really cool approach is known as knowledge distillation. Knowledge distillation is when you train a smaller network to mimic a larger network. One popular paper is found here.


Evaluation methods for multi-class classification I am looking for single-number evaluation method that can be used in multi-class classification tasks that take into account imbalanced data-sets. For instance, ROC-AUC defined by binary classifiers, is a single-number and takes into account imbalanced data-sets. On the other hand, accuracy is single-number, defined for multi-class classifiers and does not take into account imbalanced data-sets. Finally, the confusion matrix is defined for multi-class, takes that into account but is not single-number. Is there any evaluation method that satisfies the three conditions?

In https://www.sciencedirect.com/science/article/pii/S004896971831163X?via%3Dihub, we have used the product of the sensitivity of class i (i.e., the ratio of data correctly classified for class i) obtained for each class. This summarizes these values into a single index, which ranges between 0 and 1 and to some extent is irrespective to data imbalance. This approach induces a good balance between the error comitted to each category because any value appreciably below 1 reduces significantly the performance. To relax this allowing a greater error in 1 o more categories you can simply use (in increasing order): the min, the geometric mean or the arithmetic mean. In https://ieeexplore.ieee.org/document/6940273/ and https://ieeexplore.ieee.org/abstract/document/5428802/ you can find these alternatives.
Good luck.
Rafael.


Understanding time series anomaly detection using Autoencoder I'm studying how to detect anomalies in the time series using an Autoeconder. In particular, I'm following the guide posted in the Keras website, but I don't understand why they are creating and how can I adapt it to my dataset. In their guide they load the dataset and create a sequence:
TIME_STEPS = 288

# Generated training sequences for use in the model.
def create_sequences(values, time_steps=TIME_STEPS):
    output = []
    for i in range(len(values) - time_steps):
        output.append(values[i : (i + time_steps)])
    return np.stack(output)

x_train = create_sequences(df_training_value.values)
print("Training input shape: ", x_train.shape)

The reason why they are using 288 as TIME_STEPS is because they have a value for every 5 mins for 14 days. My questions are:

Does this method split the data or is he just creating a 3D variable in the correct format for the Convolutional Autoencoder?

I have a dataset where there are stored the measurements of 30 devices. Each device has about 4000 values and it is structured as well:
 device01 0.02;0.13;1.15;0.10;8.30;........;4.20
 device02 0.06;0.13;1.40;0.03;7.40;........;6.30
 ........
 device30 0.03;0.24;1.10;0.43;4.40;........;2.30

Since I do not have a timestamp reference in my dataset, how can I define the TIME_STEPS variable?

The length of TIME_STEPS in one sequence is more of a Hyperparameter. That you should try to optimize.

Does this method split the data or is he just creating a 3D variable in the correct
format for the Convolutional Autoencoder?

It simply create dataset for a 1-dimsnional convolutional network.Something like this,
$\hspace{3cm}$
Each row is an instance. Convolution will happen across the row.

I have a dataset where there are stored the measurements of 30 devices

You may simply assume each data point as a Time-step assuming it was collected simultaneously.
But your data has an additional dimension i.e. Devices.

So you should either build one model for each device or take the average of each device and consider it as the single time-step.
Or you can apply a 1-D CNN with a 2-D kernel in your first layer, assuming each device as a channel. Check an example here.
In case, Keras doesn't allow a 2-D kernel, then use a 2D-CNN with kernel size "30xM". With that, the convolution will happen in only one direction.
 Beware that if you use the last 2 options, then your pre-processing function will have to change i.e each time-step will have "30x1" as a dimension.


How to generate data if algo itself is involved in the process with a feedback loop? I have an algorithm which would be a rather easy classification task with a set of features and a class output which I would like to solve with a machine learning algo.

But I am having issues and doubts about the data generation. The features my algo uses as inputs are processed beforehand by other algorithms and more importantly, also have a feedback loop to my the algorithm I want to change.

Basically, the better my algo is getting, the less false positives there should be. But with less false positives, I have more and more imbalanced data to work with, which would mean, it is harder to train the algorithm.
I could reduce the performance of my algo on purpose and generate data, but then I am not sure if the data I am getting is any meaningful as there is a feedback loop.

To me this seems like a chicken, egg problem.

This does sounds like a bad idea, since you are selecting your data beforehand and hence likely to cause sample bias. Have you looked at anomaly detection approaches?


ResourceExhaustedError when building Sequential model i have a big problem when trying to build my model,
input shape: (1447, 224, 224, 3)
output shape: (1447, 154457)

model = Sequential([
        Conv2D(filters=64, kernel_size=(3, 3), activation='relu', padding='valid', input_shape=(224,224,3)),
        BatchNormalization(axis=-1),
        Conv2D(filters=64, kernel_size=(3, 3), activation='relu', padding='valid'),
        BatchNormalization(axis=-1),
        Conv2D(filters=128, kernel_size=(3, 3), activation='relu', padding='valid'),
        BatchNormalization(axis=-1),
        MaxPool2D(pool_size=(2, 2), strides=(2, 1)),
        Conv2D(filters=128, kernel_size=(3, 3), activation='relu', padding='valid'),
        BatchNormalization(axis=-1),
        MaxPool2D(pool_size=(2, 2), strides=(2, 1)),
        Conv2D(filters=128, kernel_size=(3, 3), activation='relu', padding='valid'),
        BatchNormalization(axis=-1),
        MaxPool2D(pool_size=(2, 2), strides=(2, 1)),
        Conv2D(filters=256, kernel_size=(3, 3), activation='relu', padding='valid'),
        BatchNormalization(axis=-1),
        MaxPool2D(pool_size=(2, 2), strides=(2, 1)),
        Conv2D(filters=512, kernel_size=(3, 3), activation='relu', padding='valid'),
        BatchNormalization(axis=-1),
        Conv2D(filters=512, kernel_size=(3, 3), activation='relu', padding='valid', strides=3),
        BatchNormalization(axis=-1),
        Conv2D(filters=512, kernel_size=(3, 3), activation='relu', padding='valid', strides=3),
        AveragePooling2D(pool_size=(6, 1), strides=1, padding='same'),
        ReLU(),
        BatchNormalization(axis=-1),
        Flatten(),
        Dense(154457, activation='relu'),
        Dense(154457)
    ])

i got below error,
ResourceExhaustedError: OOM when allocating tensor with shape[11264,154457] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:RandomUniform]

what is the problem here as i am new to neural network thing.

As described in the error message, the problem is that your model needs more memory than your GPU has. Note that OOM stands for "out of memory".
The specific layer that is demanding too much memory is Dense(154457, activation='relu'). Nevertheless, the last layer is even bigger.
You should think if you really need an output of dimensionality 154457. If you really need it, you maybe should go for approaches that are specifically designed for that scenario, like adaptive softmax.
Another option is to use a GPU with more memory that can fit this model.


How to compare two regression models? Which measurement(s) should one choose to compare two regression models?

After modifying a learning algorithm(specifically, a regression algorithm, let's call it M1) to generate another learning algorithm M2, how to validate if the above modification is efficient?

here is what I did(with 10-fold cross-validation)

I choose MSE as the only measurement, at each run, for M1 and M2, calculate the MSE of both the training and testing set.

And the result shows that:

average MSE of the training set of 10 runs: M2 < M1
average MSE of the testing set of 10 runs: M2 < M1

Question:

according to the above list, can we draw a conclusion that M2 is better than M1? thus, the modification of algorithm M1 is efficient(at least on this dataset)?

Or:

Did I miss some other important measurements? Is there a rule of thumb of comparing two regression models?

There are two things to consider:

Sampling bias
Metric

The sampling bias problem is that your test set is likely not the complete set of things you're interested in. So, no, you can't simply check MSE_1 < MSE_2 and conclude it is always the case when it's "just" for your dataset the case. This is what significance tests are for. (Although this kind of reasoning is super common in machine learning and I did it myself already )

Then the question if the metric is the correct one for your application. Typical choices are: MSE, mean absolute error, custom cost functions


How to Proceed with Tokenized text content which is given in number? I have one data set of customer review, but the text data is given is tokenized text number. I am unable to proceed thinking about how to proceed?

As I am encountering such data set the first time, so just need guide how to proceed.

As you can see text field is given in number, so how to proceed please guide?.
it will predict the 0/1 +ve or -ve category.

Using numbers for words is a very common way of representing words in a corpus. The thing that is missing here is how you have arrived at these numbers. Generally, if you have a corpus, you get these numbers by getting the index of these words in the vocabulary. For example, if you have two sentences :
1. John went to London
2. John went to London with Mary

Based on the order in which these numbers come, you can assign the following representation :

John : 1 , went : 2, to : 3, London : 4, with : 5, Mary : 6
Further, your sentence will be
1. 1 2 3 4
2. 1 2 3 4 5 6


Using dates from predicting loan My goal is:

Calculate the probability of a client take a loan.

My problem is:

How to take in account the date of the loan in this process.

Certainly the date is important here since, let's assume my clients usually take loans closer to Christmas, for example.

My initial thought was to use the day of the year, since this would absorb those "special dates" like between days 340~365 they are more likely to take the loan since this is close to Christmas and this way I don't have to "manually consider" those special dates and can even discover new special periods.

Question:

Is this a sound approach?

You are going in the right direction. Dates can contain a lot of information depending on the task you want to learn. A problem with your suggestion of using the day of the year straight up is that the last day of the year and the first day of the following year are very close to each other, while in your representation they are the furthest away. An alternative to this could be using the fact that this is cyclical and map the day_of_year feature onto a circle and use the coordinates on this circle as features to represent this. A lot of the important features regarding dates are cyclical, like the day of the week, the day of the month, the hour of the day etcetera. An alternative which allows for easier non-linear relationships would be to one-hot encode them as classes. This has the advantage of a more free representation, but the downside that it will cost more features and it cannot generalize based on the fact that two options might be next to each other.

With regards to additional domain knowledge, like knowing that being close to christmas, or close to salary day might be relevant, it's almost always beneficial to add these features to your data. This could be done by calculating absolute distance to christmas, or days before pay day etcetera. Some of these features can be work intensive to create, for example localized holiday features, but they can significantly influence people's decisions and if that is what you are trying to predict, adding these will benefit your performance.


How to preprocess Acoustic Data I am dealing with acoustic data with very high sampling frequency of 2MHz and want to build a classifier.

I was wondering if there are any rules of thumb for preprocessing acoustic data. Is it better to directly use raw data (timesignal) or first to construct spectrograms, and to use these?

There are papers, which say raw is better, and there are papers saying spectrograms are better. It somehow seems to me, that authors already had a preferred method, even before writing the paper. I think a real comparison is difficult.

I read the paper "Deep Learning and Its Applications to Machine Health Monitoring: A survey", in which a study of different methods was done. I looked up his references, but authors seemed just to pick raw or spectrograms without explaining. For example, in the paper "End-to-end learning for music audio" from Dieleman spectrograms are preferred. In "Sample-Level deep convolutional neural networks for music auto-tagging using raw waveforms", they claim their 1D structure is better or at least comparable to 2D architectures.

Personally I had better experience with spectrograms.

As far as the paper "Sample-Level deep convolutional neural networks for music auto-tagging using raw waveforms", I can give you some of my intuitions about the question since I and my colleague proceeded the experiments.

To summarize, I suggest you to use spectrogram based approaches in your situations.

There are two reasons I would like to point out,

First, training raw waveform based architecture takes about 4 times longer than spectrogram based model when the sampling rate is ranging from 16kHz to 22kHz. In your case, sampling rate is even 22Mhz. I think it will take a lot more time than spectrogram based model with similar performances.

Second, to obtain well trained raw waveform based model, we need more than 50 hours audio since the model has more parameters and deeper layers. In my opinion, the benefit of using a raw waveform-based model is not the performance improvement, but on the generative model. If we use well performing raw waveform based model, we would not need to reconstruct audio signal from spectrogram when the case is generative model. This is the main reason why we performed reported experiments.

If computing power and memory improve with current trends, we expect that the raw waveform-based model will be the mainstream in the near future. But now I think the spectrogram-based model is more convenient, especially for industrial applications.


Which, if any, machine learning algorithms are accepted as being a good tradeoff between explainability and prediction? Machine learning texts describing algorithms such as gradient boosting machines or neural networks often comment that these models are good at prediction, but this comes at the price of a loss of explainability or interpretability. Conversely, single decision trees and classical regression models are labelled as good at explanation, but giving a (relatively) poor prediction accuracy compared to more sophisticated models such as random forests or SVMs. Are there machine learning models commonly accepted as representing a good tradeoff between the two? Is there are any literature enumerating the characteristics of algorithms which allow them to be explainable? (This question was previously asked on cross-validated)

Are there machine learning models commonly accepted as representing a
  good tradeoff between the two?

I assume that by being good at prediction you mean being able to fit nonlinearities present in the data while being fairly robust to overfitting.
The tradeoff between interpretability and being able to predict those nonlinearities depends on the data and question asked. There really is no free lunch in data science and no single algorithm can be considered to be the best for any set of data (and the same applies for interpretability).

The general rule should be that the more algorithms you know the better it is for you as you can adopt to your specific needs more easily.

If I had to pick my favorite for classification task that I often use in business environment I would pick elastic-net for logistic regression. Despite strong assumption about the process that generates the data it can easily adopt to data thanks to the regularization term maintaining its interpretability from basic logistic regression.

  Is there are any literature enumerating the characteristics of
  algorithms which allow them to be explainable?

I would suggest you to pick a well written book that describes the commonly used machine learning algorithms and thier pros and cons in diffrent scenarios.
An example of such book can be The Elements of Statistical Learning by  T. Hastie, R. Tibshirani and J. Friedman


How to visualise multidimensional categorical data with additional time dimension I am trying to visualize a dataset that registers the career of people in an organisation and their backgrounds before starting in that particular organisation. I 'd like to show how individuals are positioned in the space based on their background areas and show the evolution through time in the beginning of each decade. I have seen the answers here but my data are much more categorical and also, ideally, I 'd like to display all three background categories (see below) at the same time

A sample of my data follows:

Person  BkgArea1    BkgArea2    BkgArea3    TotalSize   YearFrom    YearTo
P1  3   2   0   5   1959    1965
P2  0   2   1   3   1959    1967
P3  3   0   2   5   1991    2008
P4  3   0   0   3   1959    1980
P5  3   0   0   3   1981    1998
P6  0   2   1   3   2006    2014
P7  2   2   2   6   2008    2014
P8  0   0   3   3   1991    1996
P9  3   3   1   7   1966    1972
P10 3   0   2   5   1975    1991
P11 3   2   0   5   1998    2013
P12 3   1   0   4   2004    2013
P13 0   2   0   2   1998    2010
P14 0   3   0   3   2003    2008
P15 1   2   3   6   1998    2008
P16 3   0   2   5   1973    1977
P17 1   3   0   4   1998    2012
P18 3   0   2   5   1996    2008
P19 0   3   1   4   1998    2011
P20 3   2   0   5   1998    2006
P21 3   0   0   3   1986    1989
P22 0   3   2   5   1996    2014
P23 3   2   3   8   1959    1976
P24 3   2   0   5   1998    2001
P25 0   0   3   3   1998    2011
P26 3   3   2   8   1965    1992
P27 2   3   2   7   2010    2014
P28 3   2   2   7   1986    1998
P29 3   3   0   6   2013    2014
P30 0   3   2   5   1976    1977
P31 3   3   0   6   1986    1987
P32 0   2   2   4   1980    1990
P33 3   0   2   5   1975    1986
P34 0   3   3   6   1977    1991
P35 3   2   2   7   1963    1974
P36 3   3   0   6   1998    2001
P37 0   2   3   5   1998    2004
P38 0   3   2   5   1974    1980
P39 3   2   3   8   1989    1998
P40 0   3   3   6   1991    1998
P41 0   1   2   3   1998    2001
P42 0   2   0   2   2003    2012
P43 3   3   3   9   1973    1986
P44 3   2   1   6   1978    1986
P45 0   2   0   2   2002    2012
P46 0   3   3   6   1982    1988
P47 0   3   0   3   1992    1998
P48 3   2   2   7   1998    2004
P49 2   2   0   4   2012    2014
P50 3   2   2   7   2008    2014

The values in columns BkgArea1-3 are essentially categorical values, that indicate the strength of an individual in a specific category. It is thus
0: no background, 1: weak background, 2: average background, 3: strong background. The TotalSize column is the sum of the scores of the individuals

The initial idea was to assign basic colors to BkgAreas 1-3, namely Red, Green, Blue, and then colour the individuals according to the mixture of their backgrounds, weighing each color with the strength of each area, i.e. 2/3 * Red if an individual has score 2 in BkgArea1, etc. I then tried to display the area as 3d scatter plot but it looks very complicated and hard to interpret.

I also tried to apply Multiple Correspondence Analysis but I haven't got very satisfying results. Do you have any ideas as of how to represent this dataset?

UPDATE

After people having posted various useful answers but not 100% covering the representation needs, I summarize what I would like to achieve:

Group/cluster the individuals according to their backgrounds and relative to the three background areas. if someone has scores in more than one of them, they should be displayed somewhere in the middle.
Show, visually if possible, the number of individuals and the TotalSize of the individuals. This could be a cloud
Show the development of the organization. I think this might be easy to do with stacked column charts or histograms

RadViz can be a method to solve 1 and 2, or at least I am inclined toward that representation. The problem in my data is that individuals under RadViz will be placed at very specific points so we lose somewhat the track of how many individuals have specific background characteristics

This is something you can get in Tableau in 10 minutes.

What you need is year binning, skills weights and length(year_end - year_start) for Gannt chart:


Poisson regression options in python I want to predict count data. In my understanding both standard classification and regression are not well suited for this. A poisson or binomial regression algorithm seems to do the trick.

I am used to doing most of my ML tasks in sklearn. But on this topic I could not find an implementation. Are there any suitable options within the python universe for this?

Not quite sklearn but have you tried xgboost?

The XGBRegressor in xgboost accepts many different objective functions including poisson count:poisson for count data.

It also plays nicely with sklearn so can be used with grid search, pipelines etc.


Fix for NDCG Limitation One limitation of NDCG and way to overcome the limitation (as mentioned in
https://en.wikipedia.org/wiki/Discounted_cumulative_gain#Limitations) is

"Normalized DCG does not penalize for missing documents in the result. For
example, if a query returns two results with scores 1,1,1 and 1,1,1,1,1
respectively, both would be considered equally good, assuming ideal DCG is
computed to rank 3 for the former and rank 5 for the latter. One way to take
into account this limitation is to enforce fixed set size for the result set
and use minimum scores for the missing documents. In previous example, we
would use the scores 1,1,1,0,0 and 1,1,1,1,1 and quote nDCG as nDCG@5"
In the mentioned scenario, scores are 1,1,1 and 1,1,1,1,1 and both have NDCG =
1
After applying fix scores are 1,1,1,0,0 and 1,1,1,1,1. NDCG is still 1 for
both, so it seems the fix is not working
Could you explain where im wrong and if possible could you elaborate the fix
for NDCG limitation of not penalizing missing documents or point out to some
resource. Thanks

the idea is that when reporting nDCG@5 for the two results (associated to the same query) you have to use the same ideal discounted cumulative gain (in your example it would be the one from 1,1,1,1,1 which is the highest). Then the result 1,1,1,0,0
ends up penalized (nDCG@5 < 1) for the missing documents.


How to calculate similarity between 2 users based on the images they share? Say there are 2 users, A and B, and they each shared 10 images (in some social media site), which I have collected in 2 folders separately.
I want to calculate the similarity between the 2 users based on all the images they shared. For example, there is a sense of the 2 users being more similar if they both share pictures of food; and the 2 users would be dissimilar if one of them shares pictures of food while the other posts selfies. And the 2 users should be even more similar if they both share pictures of food containing chicken dishes only. And so on.
How do I calculate this similarity? I thought of adding up the similarities of each pair of images, and then averaging them. But this clearly doesn't work, because take only 1 user for instance. If you calculate this user's similarity to itself, the similarity will not be 1. But we expect the similarity of something with itself to be 1.
So, how do we measure this similarity of users based on their images shared?
EDIT: I am getting the following similarities in my actual use case:
import time

img_path = 'Images/All_Users/'
user_list = ['User '+str(i) for i in range(1,11)]
df = pd.DataFrame(columns=user_list, index=user_list)
for user1 in user_list:
    for user2 in user_list:
        t = time.time()
        sum_img_sim = 0
        user1_files = [x for x in os.listdir(img_path+user1+'/') if "jpg" in x]
        user2_files = [x for x in os.listdir(img_path+user2+'/') if "jpg" in x]

        for image1 in user1_files:
            for image2 in user2_files:
                sim = image_similarity(img_path+user1+'/'+image1, img_path+user2+'/'+image2)
                sum_img_sim += sim

        df[user1][user2] = sum_img_sim/(len(user1_files)*len(user2_files))
        print(user1 +' and '+ user2 + ' done in ' + str(time.time()-t) + ' secs. Similarity: ' + str(df[user1][user2]))

So, as you can see, there's not much difference in the similarities, and I'm getting anomalies like User 2 has a similarity of 0.47 with itself, while User 1 and User 10 have a similarity of 0.49

Use a similarity vector with the dimension of all possible similarities, which initially are set to zero everywhere. Go over the shared images of one user and for each image set the vector to 1 at the positions of the similarities from the image.
Do the same for another user.
Use the scalar product as similarity metric between the two users.
You can add weights for similarities which appear more than 1 time on the shared images.


what is the difference between euclidean distance and RMSE? I'm searching for a loss function that fits my Project. Actually I have two question but they are in the same direction. I take a look at the definition of the root mean squared error and the euclidean distance and they look the same to me! that's why I want to know what's the difference between the two. what would be the difference if I use rmse as a loss function or the euclidean distance??

the second question is how to search for a loss function. I mean I know it depends on the problem and commun things are MSE for Regression and Cross entropy for Classification but let's say I have a specific problem, how do I search for a loss function? I also saw that some people use a custom loss function and most of the deep learning frameworks allows us to define a custom loss function but why would I want to use a custom one? how I get the intuition that I need a custom loss function?

now to explain my problem. I'm doing a project where I need to reduce the GPS Error of a vehicle (I have some vehicle data and my neural network will try to predict the longitude and latitude so it's a regression problem) that's why I get the Idea of maybe the euclidean distance would make sense as a loss function, right? now somehow MSE also make sense to me because it is getting the difference between prediction and ground truth. does this make sense to you as a professional ML Engineer or Data scientist ? and if there would be a custom loss function that you can use, what would you suggest and why?

Let's say that RMSE and euclidean distance are not of the same type, even though their formulations are close. RMSE is a loss function, while euclidean distance is a metric. See this question on Cros Validated to better understand the difference between a loss function and a metric: a loss function is generally based on a reference metric.

Euclidean distance is a metric, so it quantifies the distance between two observations.
RMSE is, as the name suggests, the root of the mean of the squared error between a true value and a predicted value, over a range of observations. RMSE is generally intended for model performance assessment.

Your confusion probably comes from the fact that, if you apply the RMSE formula to your set of coordinates (longitude and latitude), it gives the euclidean distance. But this would not be called RMSE in that case, rather euclidean distance.

In your case, you may use the RMSE of euclidean distance as loss function. The error made by your predictor is the euclidean distance, and your loss function would be the RMSE of these errors.

Defining a loss function is strongly problem-specific. First, you need to determine which metrics to use as error function. In your case, the euclidean distance between the actual position and the predicted one is an obvious metric, but it is not the only possible one. For instance, you could use the squared or cubed euclidean distance in order to give more weight to cases that are not well predicted. You could also design an ad-hoc metric to consider:

assymmetry, e.g. to be more tolerant on errors towards the east side than the west side
anisotropy, e.g. to be more tolerant on errors in the latitude axis than the longitude
uncertainty tolerance, e.g. error is null if the euclidean distance is less than a tolerance, and is equal to the euclidean distance otherwise
etc. (depending on your problem)

The subsequent metric will allow you to evaluate the performance of the trained model over a test set, and will therefore give you a disitrbution of errors. You need to shrink this distribution to a single scalar value used as global model performance: you can pretty much choose any scalar that represents something about the distribution:

Mean squared error (MSE) or its root (RMSE) - MSE is faster computed but RMSE has the advantage or having the same dimension as the error function (a distance in your case)
Other types of $\mathcal{L}_p$ norms (RMSE is the $\mathcal{L}_2$ norm)
Any quantile of the distribution (median, 75%, 95%, maximum value)
Whatever you think is relevant!


Extracting Names using NER | Spacy I'm new to NER and I've been trying to extract names using Spacy. Here's my code:
import spacy
spacy_nlp = spacy.load('en_core_web_sm')

doc = spacy_nlp(text.strip())

# create sets to hold words
named_entities = set()
money_entities = set()
organization_entities = set()
location_entities = set()
time_indicator_entities = set()

for i in doc.ents:
    entry = str(i.lemma_).lower()
    text = text.replace(str(i).lower(), "")
    # Time indicator entities detection
    if i.label_ in ["TIM", "DATE"]:
        time_indicator_entities.add(entry)
    # money value entities detection
    elif i.label_ in ["MONEY"]:
        money_entities.add(entry)
    # organization entities detection
    elif i.label_ in ["ORG"]:
        organization_entities.add(entry)
    # Geographical and Geographical entities detection
    elif i.label_ in ["GPE", "GEO"]:
        location_entities.add(entry)
    # extract artifacts, events and natural phenomenon from text
    elif i.label_ in ["ART", "EVE", "NAT", "PERSON"]:
        named_entities.add(entry.title())

The model seems to have a decent accuracy with certain kinds of names. However it is unaware of how people’s names can differ around the world (not adapted to suit cultural differences). Is there a possible workaround to avoid this bias?

The NER model performance on a particular text depends on which data it was trained with originally, and naturally the standard models (like en_core_web_sm) are trained with English data which doesn't contain a lot of names from non-US/UK origin (same for other kinds of entities like organizations or locations).
Better performance can be achieved by training your own model with your own labelled data, but that requires you (or somebody) to annotate a reasonably large sample of data manually.


Make embedding more Gaussian-like I am trying to train a neural network to find a mapping(embedding) to a lower dimensional space. I would like for my dataset, once mapped to the lower dimensional space, to appear gaussian-like distributed.

Does anyone have any idea what form of regularization I could use. I have been reading-up on KL-divergence but that seems to require a parametrized gaussian (rather than samples). Any help would be tremendously appreciated!

One easy approach is to use a modified VAE loss with a stochastic encoder. In other words, let $z=E(x)\in\mathbb{R}^n$ and $y = D(z)$ be your pseudo-decoder (not a decoder, since you said your output is not meant to perform reconstruction; in fact it's not strictly necessary at all). Then, as in VAEs, define the probabilistic encoder $E(x)$ via
$$
\mu(x) = f_{\theta_1}(x), \;\; \Sigma(x) = g_{\theta_2}(x), \;\; z\sim \mathcal{N}(\mu,\Sigma)
$$
where $\Sigma$ is a diagonal covariance matrix. Use the reparametrization trick for backprop. Commonly this is written $z\sim q_\theta(z|x)$.

Now, let your current loss (not necessarily reconstruction) be written $\mathcal{L}(x,z,y)$. Our "pseudo-VAE" regularized loss for one data point is then just:
$$
\mathfrak{L}(x)=\mathcal{L}(x,z,y) + \beta\, \mathcal{L}_\text{KL}(x,z)
$$
where $\beta\in\mathbb{R}^+$ and the KL loss is written as
$$
\mathcal{L}_\text{KL}(x,z) = \frac{1}{2}\left[
-\log|\Sigma(x)| - n + \text{tr}(\Sigma) + \mu^T\mu
\right]
$$
assuming you want to be $\mathcal{N}(0,1)$ distributed.

Alternatively, you can treat each minibatch as approximating the marginal distribution, and then do moment matching. Concretely, let $X=(x_1,\ldots,x_n)$ be a minibatch of inputs, and $E(X)=Z=(z_1,\ldots,z_n)$ be the corresponding batch of encodings (embeddings). Suppose you want to make it "look like" $\mathcal{N}(\mu,\Sigma)$. Then let
$$ \widehat{\mu}(Z)=\frac{1}{n}\sum_i z_i \;\;\;\;\;\&\;\;\;\;\; \widehat{\Sigma}(Z)  =\frac{1}{n-1}\sum_i (z_i - \widehat{\mu}(Z))(z_i - \widehat{\mu}(Z))^T $$
so that our moment matching penalty (which you can add to your current loss function) is just
$$
\mathcal{L}_\text{M}(X) = \alpha |\mu - \widehat{\mu}|^2 + \gamma || \Sigma - \widehat{\Sigma}(Z) ||_F^2
$$

Another, slightly more complex approach, is to use an adversarial method, as in adversarial auto-encoders, adversarially learned inference (ALI), etc...
Basically, you train another network $C(z)$, which you train like the discriminator of a GAN (e.g., LS-GAN, WGAN-GP), but your generator is just $E(x)$. You train your critic $C$ to differentiate $ z\sim\mathcal{N}(0,I_n) $ and $z=E(x)$, and maximize its loss when training the "generator". This is equivalent to a conditional GAN, where the noise input is fixed (a Dirac Delta) or non-existent, the output is just the embedding, and the true samples are simply normally distributed random vectors.


values passed to user-defined distance function by KNeighborsClassifier is wrong I have a data-set in which all features are binary and the class of each data-point is also binary. I am trying to use KNearestClassifier with a user-defined distance function as follows:

KNN = KNeighborsClassifier(n_neighbors=3,
                           algorithm='ball_tree',
                           metric='pyfunc',
                           metric_params={"func": lev_metric})
x_train, x_test, y_train, y_test = train_test_split(df_sum,
                                                    y,
                                                    test_size=0.1,
                                                    random_state=0)
KNN.fit(x_train, y_train)

and my custom metric function is as follows:

def lev_metric(a, b):
    print(a)
    print(b)
    return levenshtein(a, b)

the metric function expects two ndarrays of binary values of 0s and 1s. When knn.fit calls the metric function, "b" looks as expected (e.g. [0 1 1 0 0 1 0 1...) but "a" looks like gibberish and is an ndarray with real valued elements between 0 and 1, for example :

[0.32222222 0.42222222 0.34444444 0.47777778 0.41111111 0.38888889
 0.4        0.31111111 0.35555556 0.35555556 0.42222222 0.46666667
 0.36666667 0.32222222 0.41111111 0.32222222 0.36666667 0.35555556
 0.41111111 0.33333333 0.4        0.42222222 0.3        0.37777778
 0.38888889 0.48888889 0.41111111 0.43333333 0.34444444 0.35555556
 0.43333333 0.38888889 0.43333333 0.32222222 0.47777778 0.34444444...

what am I missing? I have also checked that the "x_train" is correct. Also, isn't knn an instance based learner? why is it calling the distance function in fit anyways? is it not supposed to just memorise the training examples? Thanks.

Neighbors-based methods are known as non-generalizing machine learning methods, since they simply “remember” all of its training data (possibly transformed into a fast indexing structure such as a Ball Tree or KD Tree).

(emphasis mine.  Source: https://scikit-learn.org/stable/modules/neighbors.html#nearest-neighbors )

You are using algorithm='ball_tree', which roughly speaking uses Euclidean balls to cluster the points, making use of the ball distances for bounds on the distances between actual datapoints (wiki).  So, I suspect the "a" value you're seeing is actually one of the balls' centers.

For the Levenshtein distance, you probably should use the 'brute' algorithm.


References/tutorials about data mining and machine learning I am learning data analytics and I wonder if there are some good references and tutorials about machine learning, data analytics and data mining?
What I'm searching for is an understandable reference/tutorial, which isn't very technical and isn't very basic either, in other words the material begins with the basic steps towards advanced steps.
Thank you.

I recommend "An Introduction to Statistical Learning" (ISL) by Gareth James, Daniela Witten, Trevor Hastie and Rob Tibshirani. The book is available online.
The book covers a number of topics in ML and for each topic there is a "lab" in R (code is also available for Python, in fact there are several pages on Github covering the labs).
ISL is not too technical but gives a sound introduction. In case you want to advance on some topics, you can switch to the advanced book "Elements of Statistical Learning" (ELS).


Using Tensorflow object detection API vs Keras I am new to machine learning. I am curious to know what is the difference between using Keras instead of TensorFlow object detection API. We need to manually configure hidden layers and input layer in Keras so what is the advantage to use Keras and how to know how many layers should configure to achieve object detection using Keras.

Please check two different types of implementation 1) Using Keras 2) Using Tensorflow Object detection API without Keras

Thanks !!!

Keras provides you high level api or can say wrapper written on top of multiple backends.
These back ends have the core implementation of DNN.
List of Keras supported backends are:

Tensorflow
Theano
CNTK

**Source: Keras documentation for supported backends

  Keras hides a bit complexity of DNN implementation, but again restrict
  your freedom. In case if you write a code in Tensorflow, you have
  explicitly specifies and calculate optimizer, cost function and other
  things, but it provides you flexibility.

So for me writing in Keras just a convenience.

As far my knowledge is concern, so far we dont have any fix formula to identify number of layers sufficient for Object detection :).


ScikitLearn - RandomForestRegressor score different in and out of grid search I am using RandomForestRegressor (scikit-learn python package). I am looking for the best values for hyperparameters n_estimators and min_samples_split for fitting my regressor on a train dataset ( X_train, y_train ) :

  param_grid = { 'n_estimators' : range( 10 , 201 ) ,
                 'min_samples_split' : range( 2 , 11 ) }

  rfr = RandomForestRegressor()

  best_score = 0
  best_param = {}

  for param in ParameterGrid( param_grid ):
      rfr.set_params( **g )
      rfr.fit( X_train , y_train )
      score = rfr.score( X_train , y_train )

  if score > best_score :
      best_score = score
      best_param = param

  print( 'best_score : {0}'.format( best_score ) )
  print( 'best_parameters : {0}'.format( best_param ) )

I am not using cross validation on purpose.

The best hyper parameters are in the best_param variable , the corresponding score in the best_score variable.

After that, I set the hyper parameters of my regressor using the best ones :

rfr.set_params( **best_param )

I fit the regressor on my train dataset:

rfr.fit( X_train , y_train )

Finally just to check I obtain the score of the regressor with the best hyper parameters:

rfr.score( X_train , y_train )

The score is not the same as the one calculated in the grid : 0.806 out of grid versus 0.963 in the grid.

I do not understand why. The hyper parameters are the best ones, and the dataset used to calculate these best parameters is the same. Any hints?

I have found the answer to my question : there is randomness in RandomForestRegressor().

Using the random_state hyper parameter fix the problem :

rfr = RandomForestRegressor( random_state = 123 )

What is surprising is the difference of score obtained from different values of the random_state hyper parameter. It means my best hyper parameters n_estimators and min_samples_split are strongly depend on the random_state hyper parameter.

I guess it is something I will have to deal with.


How to use NLP to determine the normal words in the text I have a lot of .txt files. I want to train the model which remove bad words from the document(bad word - random sequence of characters).
For example: bad words - hsA , ubj8 ,t+h ,SSSh ,_^[ ,SVWh ,SSSSSS
normal words : *.exe , copy , write ,write ,ftp.exe

You can probably train a classification model to classify normal words from bad words, using character subsequences as features in the words. For example, exe can be a feature for distinguishing a normal word from a bad word in your case.


PCA on Neural Networks dimensions reduction? The dataset which was extracted from the database consists of more than 50         columns, I call these columns dimensions, can I call them dimensions?
Obviously, I have to do dimension reduction on them. But since PCA like algorithms often do axis rotating to generate some new axises. I don't think I will PCA algorithm in dimensions reduction. So I calculated the correlations between these columns(parameters), and filtered these who has a high value and some other rules. So can I still call it dimensions reduction? Since I only did some parameters filtering
The reason I don't use PCA like algorithms is because I want to implement Neural Network classification, and I need the real parameters.

Please comment on these, anything even criticizing is welcomed.

The approach you use to do dimensions reduction is agnostic to the method you use for classification. You can use PCA to preprocess your data before to train any type of classifier, including artificial neural networks if that's what you want to use.


How to improve accuracy of deep neural networks I am using Tensorflow to predict whether the given sentence is positive and negative. I have take 5000 samples of positive sentences and 5000 samples of negative sentences. 90% of the data I used it for training the neural network and rest 10% for testing.

Below is the parameter initialisation.
   Batch size = 100
   Epochs = 30
   Number of hidden layers = 3
   Nodes in each hidden layer = 100
   epochs = 30

I could see in each epoch the cost function is getting reduced reasonably. However the accuracy of the model on test set is poor (only 56%)

 Epoch  1 completed out of  30 loss :  22611.10902404785
 Epoch  2 completed out of  30 loss :  12377.467597961426
 Epoch  3 completed out of  30 loss :  8659.753067016602
 Epoch  4 completed out of  30 loss :  6678.618850708008
 Epoch  5 completed out of  30 loss :  5391.995906829834
 Epoch  6 completed out of  30 loss :  4476.406986236572
 Epoch  7 completed out of  30 loss :  3776.497922897339
 -------------------------------------------------------
 Epoch  25 completed out of  30 loss :  478.93606185913086
 Epoch  26 completed out of  30 loss :  450.8017848730087
 Epoch  27 completed out of  30 loss :  435.0913710594177
 Epoch  28 completed out of  30 loss :  452.10553523898125
 Epoch  29 completed out of  30 loss :  539.5199084281921
 Epoch  30 completed out of  30 loss :  685.9198244810104
 Accuracy of Train : 0.88155556
 Accuracy of Test : 0.524

Is there any parameter that can be tuned to increase the accuracy of the model considering the same number of data set.

You need much more data.

Deep NN shines when you have excessive amounts of data.

With only a little bit if data it can easily overfit. The big difference between training and test performance shows that your network is overfitting badly. This is likely also because your network model has too much capacity (variables, nodes) compared to the amount of training data. A smaller network (fewer nodes) may overfit less.


Can data science be applied to establishing the author of a text? There are many instances of quotes and texts where the true author may be disputed. Are there data science or machine learning techniques which may be useful to establish the authorship?

Yes - this is an active area of study, especially famously with respect to disputed plays of Shakespeare. Features commonly extracted include frequency of key words, novel vocabularly, word length, and these are well adapted to being used as data mining features.

For example, here is a thesis that data mined Shakespeare's extant plays to test whether they could have been the work of a number of authors often touted as being the "true Shakespeare" -> http://www.cs.dartmouth.edu/~datamining/Final.pdf


How does exactly class_weight in Keras work? I'm working on a multi-label problem in Keras, using binary-cross-entropy loss function with sigmoid activation. Let's say I have 4 classes, so a response might look like this:

[1, 0, 0, 1]

Though a vast majority of the responses will actually be zero's all around

[0, 0, 0, 0]

The data is also very imbalanced between the different classes, and if i did a sum on y, the total number of positives in each class would look something like this

[2000, 500, 1000, 250]

So to give equal weight to each class I would think that I should construct a dictionary-like so

weights = {0: 2.5, 1: 10., 2: 5., 3: 20}

And feed that to the class_weight parameter in Keras.

Here is my question though:

If my response variable was binary (only first class for example), I would need to feed a dictionary that defines both a factor for the 0 an 1

weights = {0: 1, 1: 2.5}

Which I take to mean that positive samples are weighted higher than negatives. But, how do weighted samples work with the multi-label (or multi-class for that matter)? since in that case, I've only given weights to the classes in general, not specifically positives and negatives

This should be the loss function I'm working with

−(ylog(p)+(1−y)log(1−p))

And with binary-cross-entropy and sigmoid, this is applied to each class and the total loss is then the sum of all class losses for all samples.

But where exactly is the class weight applied here? Is it multiplied on the entire function? If that's the case, then it's not really fixing the class imbalance as I see it, since it also weighs the negatives higher, not just the positives? I would imagine that is desirable is to only multiply positive responses?

If anyone can clear up exactly how this all works so I can understand a bit more what's going on under the hood I would be grateful.

A weighted classification layer computes the weighted cross entropy loss for classification problems. Weighted cross entropy is an error measure between two continuous random variables. For prediction scores Y and training targets T, the weighted cross entropy loss between Y and T is given by

$$ F = -\frac{1}{N}\sum_{i=1}^{N}\sum_{k=1}^{M} w_i \cdot T_{ni} \cdot log(p_{ij}))$$

where $N$ is the number of instances, K is the number of classes, and w is a vector of weights for each class.

The cost function per label is measuring how good our classifier is doing per label, but it is affected by the number of instances $N$ that don't contain this label.


QGtkStyle could not resolve GTK I have installed Orange 3 in Ubuntu 18.04 using Anaconda. It runs just fine, but the menus appear as blank.

I obtain the following error when I execute it:

QGtkStyle could not resolve GTK. Make sure you have installed the proper libraries.

I have been trying to sort it for days without success.

Any idea of how to fix this?

A quick fix has been published at https://stackoverflow.com/a/50583925/3866828

It works adding

export QT_STYLE_OVERRIDE=gtk2

at the end of the ~/.bashrc file. Then it's a matter of executing
source ~/.bashrc to see the content of the menus the next time that Orange 3 is loaded.


Best way to model a problem where multiple rows contribute to the result This is my first question here and I am very computer savvy but a ML novice. I have done lots of learning algo's where the question fits on a single row so I can train the model with a lot of rows and then give it a value and a clear result is found.
However I have come up with a problem that doesn't fit this pattern and I am stuck on where to start.
I have a formula (simplified representative is)
Z/X = Y
Where:

X can be one of 6 number (1, 2, 3, 4, 5, 6)
Y is an integer between 0 and 20 (inclusive)
Z is the result

Y is the number I am trying to find
The problem is X is invisible to me, the only number I can see is Z. I have multiple rows in the dataset ranging from a single row to 100 rows.
What I want to do is take the list of Z and back calculate the certainty of working out Y. As an example:
Z=

30
10
10
15
5

If you look at these numbers above and apply the 6 possible version of X you get:

30 / 1 = 30 (outside range), 30 / 2 = 15(valid), 30 / 3 = 10(valid), 30 / 4 = 7.5(invalid), 30 / 5 = 6(valid), 30 / 6 = 5(valid)
10 / 1 = 10(valid), 10 / 2 = 5(valid), 10 / 3 = 3 1/3(invalid), 10 / 4 = 5/2(invalid), 10 / 5 = 2(valid)  10 / 6 = 5/3(invalid)
10 / 1 = 10(valid), 10 / 2 = 5(valid), 10 / 3 = 3 1/3(invalid), 10 / 4 = 5/2(invalid), 10 / 5 = 2(valid)  10 / 6 = 5/3(invalid)
15 / 1 = 15(valid), 15 / 2 = 7.5(invalid), 15/ 3 = 5(valid), 15 / 4 = 3.75(invalid), 15 / 5 = 3(valid)  15 / 6 = 2.5(invalid)
5 / 1 = 5(valid), 5 / 2 = 2.5(invalid), 5 / 3 = 5/3(invalid), 5 / 4 = 5/4(invalid), 5 / 5 = 1(valid)  5 / 6 = 5/6(invalid)

What all of them have in common is 5 is a possible value of Y and therefore there is a good chance that Y is 5
So far it looks like a simultaneous equation but the problem exists that the Y can change over time (but very slowly), so it isn't always an exact science. Also the formula isn't exact so the observed Z has a wander error in it.
If I had one row the certainty would be lower, if I have a mix (as above) then there really is only one so I would expect certainty to be higher.
My question is:
Does anyone have any advice on how to tackle this problem, algorithm's to google etc?
Thanks
Dave

You could try the following.

Filter the results the way you describe; for all the valid options of X in the row assign equal probabilities
Apply the law of total probability. I guess in this case it's just summing all the probabilities for all possible X and divide by number of rows.

I.e. in you case

for 30 it's either 15, 10, 6 or 5 with probability 0.25 each
for both 10 it's 10, 5 or 2 with prob 0.33
for 15 it's 15, 5, 3 with prob 0.33
for 5 it's 5 or 1 with prob 0.5

That's mean in total Y is 5 with probability ~0.35 or 10 with prob ~0.18
If the X is integer, then some level of noise in Y should not be that big of a deal. Should be solved by small allowance of discrepancy on filter stage.
Slowly changing is trickier, but you probably could just make window smoothing, like say the probability of 10 is the sum of probs for 9, 10, 11.
I would not bother with estimation by number of rows - you could work out something based on CLT, but in fact the implicit sampling on step 2 already takes this into account.


What is the difference between a Categorical Column and a Dense Column? In Tensorflow, there are 9 different feature columns, arranged into three groups: categorical, dense and hybrid.

From reading the guide, I understand categorical columns are used to represent discrete input data with a numerical value. It gives the example of a categorical column called categorical identity column:

ID   Represented using one-hot encoding
0    [1, 0, 0, 0]
1    [0, 1, 0, 0]
2    [0, 0, 1, 0]
3    [0, 0, 0, 1]

But you also have a dense column called indicator column, which 'wraps'(?) a categorical column to produce something that looks almost identical:

Category (from category column)   Represented as...
0                                 [1, 0, 0, 0]
1                                 [0, 1, 0, 0]
2                                 [0, 0, 1, 0]
3                                 [0, 0, 0, 1]

So both 'categorical' and 'dense' columns seems to be able to represent discrete data, so that's not what distinguishes one from another.

My question is: In principle, what are the difference between a 'categorical column' and a 'dense column'?

I have read this answer that explains the difference between indicator columns and categorical identity columns, but I am looking for a more generic answer distinguishing categorical and dense columns.

Sparse vs Dense

The 'categorical column' is a sparse column. Sparse and dense columns (or matrices) are in a way opposites of each other.

Sparse columns usually contain a lot of zeros. Whereas, dense columns have more non-zero entries. This matters, because the way they are stored and processed can differ.

Sparse

If we take your sparse example:

ID   Represented using one-hot encoding
0    [1, 0, 0, 0]
1    [0, 1, 0, 0]
2    [0, 0, 1, 0]
3    [0, 0, 0, 1]

What information needs to be stored to reconstruct this matrix? All you need is:

The number of columns and rows
Value for every non-zero entry

So, by having:

n_columns = 4
n_rows = 4
non_zero_entries = {
    (0, 0): 1,
    (1, 1): 1,
    (2, 2): 1,
    (3, 3): 1
}

you can reconstruct it. This means you don't need to store all the 16 integers to reconstruct the matrix.

In Tensorflow, the 'categorical column' matrix, has the special property that it is a one-hot encoding, meaning that one and only one entry in the columns is non-zero. However, the sparse principle doesn't need to be one-hot.

For example:

matrix = [
    [0, 0,   0, 0],
    [0, 2,   0, 1],
    [0, 0, 0.5, 0]
]

Can be reconstructed with:

n_columns = 4
n_rows = 3
non_zero_entries = {
    (1, 1): 2,
    (1, 3): 1,
    (2, 2): 0.5
}

Imagine a matrix of size $100 \times 100$. There are $10,000$ values in it. Now, suppose only $1$ entry is non-zero (for whatever reason). Then, this large $10,000$ values matrix, can be represented by:

n_columns = 100
n_rows = 100
non_zero_entries = {
    (x, y): 1
}

Dense

One the other hand, dense columns are not stored by defining only the non-zero entries. The 0s are also being stored.

While more memory intensive, they are less computationally expensive because they don't need to be reconstructed. Also, often the 0s are likely to be changed over time, and if there is already an int being stored, no new memory locations need to be assigned.

Usability

Not all algorithms can process a sparse matrix. Often, matrix operations require the whole vector be present. Depending on the implementation, the assumptions that non-present elements are zero is not there, therefore an explicit zero needs be stored and passed along.

Like you mentioned in your questions:

  But you also have a dense column called indicator column, which
  'wraps'(?) a categorical column to produce something that looks almost
  identical

This is true. What they represent can be exactly the same. However, how they are stored and the efficiency of the implementations differ.


How to approach model reporting task I have been tasked to report on an ensemble model that was created in h2o which includes several model subtypes such as Random Forest, GBM, linear models etc.  The end goal is to predict churn rates for products in a large telco company, but the approach we use could apply to any similar problem.

The models produced in this way contain a few potentially useful performance measures such as variable importance, precision, recall and some others.  Each model has roughly 150 input variables.

The model scores have been used to group the customers by decile and measure the churn rate of each group.

The present situation is that the scores appear to be too good which suggests we may have a data leakage problem.  For instance, for one of the models the 1st decile captures 84% of the churn, with 99% of the churn captured by the 4th decile.

My task is to understand and report on potential issues with the model performance so we can improve the models and recommend action to the business.  What I would like to know is:

What are some basic analyses that I can perform to address the data
leakage issue.
How can I leverage the model metadata to better
understand model performance?
What other important questions should I
know to ask in order to fully address this task?

Remove input data to test for leakage

This is very generalized question, so without knowing the types and provenance of the input data, this can be hard to answer.

But, in general, to check for leakage, you can use the model on some subsets of the input variables while removing other input variables. If you get data from multiple sources, then try removing all input variables from a single source, then re-run your models. You may be able to identify the source of the data leakage. Alternately, if computational power allows, you can brute force it by running the model with each of the 150 input variables removed, or all sets of two variables, etc.

Use customer-centered time data

Regarding model meta-data, again I would investigate data provenance. Are you predicting churn using the complete patterns of customers who stopped using the service? What I mean to say is, instead of looking backwards from a fixed real-time period, like today, to all customers who did or did not stop using the service, try looking from a fixed customer-time. Use only data from the first year that each customer used the service, and attempt to predict whether each customer will remain with the service for another year.

The warning signs of a customer dropping the service may be obvious in the months leading up to that customer dropping the service, but by then, the predictive power of your model may be too late to stop that customer from leaving. Instead, index the time component of each customer's history to zero when the first start using the service, and run your model on this data.


Using predicted probabilities and bayesian inference to update beliefs I'm currently working on a project to predict the likelihood of an outcome. I'd like to implement a system where the belief the event will happen is updated after running the model on new data. However, I'm having trouble figuring out exactly how to implement that. Any explanations of updating beliefs using Bayes theorem would be helpful.

Anything that can be done in the classical, frequentist statistical approach, has a Bayesian counterpart.

Beliefs can be described, under certain assumptions, as probability distributions. You can take a given PDF as your prior belief, combining it with the Likelihood function of your data through Bayes theorem, and obtain a posterior distribution describing your updated belief in light of the empirical evidence provided.

Unfortunately I have no information of your problem at hand, on what data you are working with, and their nature.

If you are interested in Bayesian statistical models I suggest you to take a look at an R package called JAGS, that you can use to implement pretty much any Bayesian model, and with ready-to-go MCMC algorithms.

I wish I could be more specific, please provide more details about your problem and your data.


Use regression instead of classification for hard labeled ranking datasets Let's imagine I have a dataset of movie reviews with annotated sentiment:

-1 means negative
 0 means neutral
+1 means positive

I see a lot of people trying to do classification to try to answer those types of problems, but shouldn't regression be used instead? To me using regression would allow the system to model that there is a transition between labels, e.g. 0 is in between. Any thoughts on this?

An issue with regression is that the distance between negative, neutral and positive are not necessarily the same. This transition/uncertainty you mention can be modeled by using a probability distribution over the classes as opposed to a hard argmax. If you use Neural Networks for example the last softmax layer gives you this information for free (For example: 0.1 on Negative, 0.6 Neutral and 0.3 Positive).