Both train and test scores high. Could my model be overfitting? all. Currently I am working in a regressor to predict the median house value using 1990 census as a dataset (the same on Aurélien Géron used on his book).

My best result so far has been a score of ~0.96 for the training set and ~0.80 and for the test set with a bagging regressor. Here's how I had been doing:

precisao_treino = clf.score(dados_treino_feat,dados_treino_targ) #training score
precisao_valid = clf.score(dados_valid_feat,dados_valid_targ) #test score

(0.9698880465469455, 0.8087685894971024)

But here is where I get confused. That very high training score indicate that there is a potential overfit on my model. However, the test score is maintained a relative good score. So I don't know if my model is overfitting or if doing well at predicting. Could someone explain to me whether this a bad or good model?

Here's how my final dataset looks like, with one-hot-encoding:

['longitude',
 'latitude',
 'housing_median_age',
 'total_rooms',
 'total_bedrooms',
 'population',
 'households',
 'median_income',
 'median_house_value',
 '<1H OCEAN',
 'INLAND',
 'ISLAND',
 'NEAR BAY',
 'NEAR OCEAN']

My solution to removing outliers:

df = df[(np.abs(zscore(df)) < 6).all(axis=1)]

My model:

clf = ensemble.BaggingRegressor(n_estimators=20)

Overfitting is nothing but when the model tries to rot learn the data given to it during the training process.

In such cases, the validation accuracy goes down as the model cannot interpret and classify unseen data.

  Reviewing the accuracies of the model, the validation accuracy soars to 0.8 which is quite good.

The training accuracy is 0.96 which is a good sign and also it shows that model has generalised so well that it can classify all the data ( train and validation ) very well.


Is there a quantitative way to determine if a class of algorithms tends produce low bias or low variance models? I understand that some machine learning models tend to be low bias, whereas others tend to be low variance (source).  As an example, a linear regression will tend to have low variance error and high bias error.  In contrast, a decision tree will tend to have high variance error and low bias error.  Intuitively this makes sense because a decision tree is prone to overfitting the data, whereas a linear regression is not.  However, is there a more quantitative way to determine if a class of algorithms tends to produce low bias or low variance models?

It's more a matter of complexity of the model than of the class of algorithms. Of course some classes of algorithm produce more complex models than others by construction, but this is not always the case. For example the complexity of a Decision Tree usually depends on the options/hyper-parameters: maximum depth, pruning, minimum number of instances in a branch. If these parameters are set to produce a small tree then the risk is bias (underfitting), but if they are set to produce a large tree then the risk is variance (overfitting).
The complexity of a model depends mostly on the number and nature of its parameters, so as a first approximation the number of parameters is a reasonable quantitative measure of complexity (see this closely related question). Also keep in mind that most models try to use all the features they are provided with, so the number of features also has a high impact on the model complexity.


Imbalance classes problem I have a dataset of 4 classes with the following number of instances:

Class 0:  13175
Class 1:  82
Class 2:  75
Class 3:  121

Have have applied several subsampling and oversampling methods from the Python imbalance-learn API but none of them had a good performance for all classes. I have applied:

Undersampling: CondensedNearestNeighbour, EditedNearestNeighbours, NeighbourhoodCleaningRule, RandomUnderSampler.
Oversampling: SMOTE, ADASYN
class_weight:['balanced'] parameter option in my grid search
costcla library, but it does not work with more than two classes.

And I was not successful. Could you suggest a solution for this problem?

I think you're in a pretty difficult state. I think the class imbalance techniques you're using are fine. I would advise you to try framing your problem in a hierarchical classification manner.

Level 1 Classifier

classifier_A between class 0 & class (1 + 2 + 3)

Note: Apply all sort of class imbalance methods to get good accuracy here

Level 2 Classifier

classifier_B between class 1, class 2, class 3

Final pseudo model

if predict_classifier_A(x_test) == "class 0":
   result = "class 0"
 else:
   result = predict_classifier_B(x_test)

I'm not sure will it work or not but its worth a try. Let us know if it worked better.


Generating Image data sets for training CNNs I want to build a system that recognises (with a given uncertainty) the make and model of a car from an image. I have decided to use Convolution Neural Networks, specifically the Caffe framework.

My next decision is how best to build my data set. According to this book, I need around 5000 data points for each class (so lets say ~500k images).

I've done a little bit of reading on here and other resources, and it seems that the Google Custom Search API is a potential option; but that limits me to at most 100 searches per day (for free). I thought about building a script to scrape sites like Autotrader; but my experience with scraping the web is zero.

Does anyone have experience generating image data sets of this size? Any pearls of wisdom that you could share with me? I am happy to invest time and effort learning for example Beautiful Soup or this Google API, but I don't want to waste time going down the wrong rabbit hole.

I decided turning my comment into an answer.

If you want to go pro, use a framework such as scrapy.

Personally, I find them overly-cumbersome and I have been successful using the following approach. I think your use case is simple enough for it to be of use to you as well.

Assuming you are using Python3 as well, you can grab a webpage easily, and then get what you want using XPath notation.

from lxml import html
import urllib.request

# keep running until there are no "next" pages
for page in range(999):
    url = 'http://blablabla.com/?page=%d' % page
    text = urllib.request.urlopen(url).read()
    tree = html.fromstring(text)
    images = tree.xpath('//img[@class="car"]/href()')
    types = tree.xpath('//div[@class="type"]/text()')
    if not images:
        break
    for i, (cartype, image) in enumerate(zip(types, images)):
        urllib.request.urlretrieve(image, '%s-page%d-img%d.png' % (cartype, page, i))

(Purely illustrative example.)

Now adjust as you may. XPath is an incredibly powerful notation of accessing XML nodes. Much more is possible than I am writing here. Take this tutorial to learn the full XPath syntax.

Some web designers make it much harder to access whatever you want because they do not properly class-ify their HTML objects. In those cases, you may have to access a parent node and ask for their child. Or access a sibling and then get the siblings. Anyway, XPath and Python's lxml package make all this incredibly easy.

Any modern browser like Chrome and Firefox also lets you easily explore the DOM of any webpage. Just right-click and press Inspect or go to Developer Tools in the Tools menu or some such.

Note: some website like scholar.google.com disallow scrapers and are very good at detecting if that's what you're doing. You can specify an user-agent for urllib, but it might be futile. Even advanced frameworks may not be able to help you there.

EDIT: I have made a blog post where I elaborate a little more.


How should clickstream data be prepared before user segmentation can be performed? I'm interested in doing segmentation/clustering of users in clickstream data and am looking for some good suggestions about how to go about it.

Lets say my data consists of observations made up of visitors to a website. The data is in clickstream/weblog format and so is made up of users cookie data. Lets say I can identify unique users via their IP address (as a basic example). How should I go about preparing my data so that it can be segmented to find users with similar behaviours? One of my thoughts around this, is that because the data is event-driven, the same user can obviously appear multiple times in the data even though it may all be related to the same session of that user. How are these types of problems tackled so that you can perform segmentation based on user behaviour?

Thanks for your suggestions!

This sounds like a time-related predictive task...i.e. forecasting. The data you have available is event-driven, e.g. data is recorded when a user clicks on a link/refreshes a page/clicks on a Google ad etc. This means that you need to treat your data as if it were a time-series for each user. This would be easier if you had access to ESP (Event Stream Processing) but let's say that you do not. Therefore, you need to identify each user, aggregate their info/features so that you have features identifying the number of unique pages they visited, the number of sessions they had etc. etc. Then you can segment these users and you may or may not derive useful segments from them. You could even go a step further and do session analysis whereby you identify unique sessions and cluster/segment user sessions looking for nice visual patterns or anomalies.


What GPU size do I need to fine tune BERT base cased? I want to fine tune BERT Multilingual but I'm not aware about the GPU requirements to train BERT Multilingual. I have GTX 1050ti 4GB on my local machine. I want to know what size of GPU is needed and what type of GPU is needed to train BERT. I have access to server resources. Could anyone tell me what size of GPU should I request for on server.

Speaking about vanilla BERT.
It is currently not possible to fine-tune BERT-Large using a GPU with 12GB - 16GB of RAM, because the maximum batch size that can fit in memory is too small (even with batch size = 1).
The fine-tuning examples which use BERT-Base should be able to run on a GPU that has at least 12GB of RAM using the hyperparameters given on this page.
However, GPU training is single-GPU only.


How is the Schwarz Criterion defined? I am currently reading slides about the $k$-means algorithm. In the analysis, the professor writes

  Minimize Schwarz Criterion: $W(C) + \lambda m k \log R$

$W(C)$ is Within-class scatter. I guess $\lambda$ is a weighting factor which has to be chosen by the developer and $k$ is the number of clusters. But what is $m$ and what is $R$?

See also this StackExchange post. The Schwarz criterion, or BIC, is very general, so the question is, "how does this formula apply the BIC to $k$-means?"

The Wikipedia page on Bayesian Information Criterion (BIC) uses this definition of BIC:

$$BIC = -2 \ln \hat{L} + \kappa \ln {n}$$

where:

$n$ is the #observations
$\kappa$ is the #free parameters (I use Greek $\kappa$ to distinguish from your $k$)
$\hat{L}$ is the maximized likelihood after fitting parameters

Pelleg and Moore (2000) use $R$ for the #observations.  Assuming your professor does to, that means:

$\kappa = \lambda m k$, the # free parameters.
$W(C) = -2 \ln \hat{L}$

So, what are $\lambda$ and $m$?  I can't tell which is which, but it has to come out something like this:

$k$ is the number of clusters [as you say]
$\lambda$ is the number of dimensions (columns) in the data
$m$ is the number of parameters to estimate for each dimension (2 for the usual Gaussian: mean and variance).

It makes sense that $W(C)$, the "within-class scatter" would be measured by log likelihood.  Note, $W(C)$ has to sum the log likelihood for all clusters.


What is the purpose of setting an initial weight on deep learning model? I'm now learning about deep learning with Keras, and to implement a deep learning model at Keras, you set the initializer to set its initial weights on.

from keras.models import Sequential
from keras.layers import Dense
model = Sequential()
model.add(Dense(12, input_dim=8, kernel_initializer="random_uniform"))

The kernel_initializer can take something others, such as random_normal, which uses Gaussian, not uniform distribution, and zero, which literally sets all weights to 0.

However, I don't understand why you like to set different weights at the initializer. Specifically, what advantages does it have over setting all initial weights to 0, which sounds more natural for novices like me?

Also, should the initial weights, if needed, be always set a tiny value (e.g. 0.05) to?

This is greatly addressed in the Stanford CS class CS231n:

  Pitfall: all zero initialization. Lets start with what we should not do. Note that we do not know what the final value of every weight should be in the trained network, but with proper data normalization it is reasonable to assume that approximately half of the weights will be positive and half of them will be negative. A reasonable-sounding idea then might be to set all the initial weights to zero, which we expect to be the “best guess” in expectation. This turns out to be a mistake, because if every neuron in the network computes the same output, then they will also all compute the same gradients during backpropagation and undergo the exact same parameter updates. In other words, there is no source of asymmetry between neurons if their weights are initialized to be the same.

There are several weight initialization strategies; each one is best suited for a type of activation function. For instance, Glorot's initialization aims at not saturating sigmoid activations, while He's initialization is meant for Rectified Linear Units (ReLUs).


Intuitive Explanation of R-squared Here is a nice definition of R-squared that I have found on the internet.

  R-squared is a statistical measure of how close the data are to the
  fitted regression line. It is also known as the coefficient of
  determination, or the coefficient of multiple determination for
  multiple regression.

  The definition of R-squared is fairly straight-forward; it is the
  percentage of the response variable variation that is explained by a
  linear model. Or:

  R-squared = Explained variation / Total variation

  R-squared is always between 0 and 100%:

  0% indicates that the model explains none of the variability of the
  response data around its mean. 100% indicates that the model explains
  all the variability of the response data around its mean. In general,
  the higher the R-squared, the better the model fits your data.
  However, there are important conditions for this guideline that I’ll
  talk about both in this post and my next post.

Could someone please explain to me what does "variability of the response data around its mean" stand for, and also "it is the percentage of the response variable variation that is explained by linear model"? I am having trouble understanding these concepts.

--variability of the response data around its mean---
It means the model You trained tells you how much Accuracy it can provide when predicting. Higher the R_squared ,More accurate the model.

it is the percentage of the response variable variation that is explained by linear
This is literally the same question as above.


How to merge monthly, daily and weekly data? Google Trends returns weekly data so I have to find a way to merge them with my daily/monthly data.

What I have done so far is to break each serie into daily data, for exemple:

from:

2013-03-03 - 2013-03-09 37

to:

2013-03-03 37
2013-03-04 37
2013-03-05 37
2013-03-06 37
2013-03-07 37
2013-03-08 37
2013-03-09 37

But this is adding a lot of complexity to my problem. I was trying to predict google searchs from the last 6 months values, or 6 values in monthly data. Daily data would imply a work on 180 past values. (I have 10 years of data so 120 points in monthly data / 500+ in weekly data/ 3500+ in daily data)

The other approach would be to "merge" daily data in weekly/monthly data. But some questions arise from this process. Some data can be averaged because their sum represent something. Rainfall for example, the amount of rain in a given week will be the sum of the amounts for each days composing the weeks.

In my case I am dealing with prices, financial rates and other things. For the prices it is common in my field to take volume exchanged into account, so the weekly data would be a weighted average. For financial rates it is a bit more complex a some formulas are involved to build weekly rates from daily rates.  For the other things i don't know the underlying properties. I think those properties are important to avoid meaningless indicators (an average of fiancial rates would be a non-sense for example).

So three questions:

For known and unknown properties, how should I proceed to go from daily to weekly/monthly data ?

I feel like breaking weekly/monthly data into daily data like i've done is somewhat wrong because I am introducing quantities that have no sense in real life. So almost the same question:

For known and unknown properties, how should I proceed to go from weekly/monthly to daily data ?

Last but not least: when given two time series with different time steps, what is better: Using the Lowest or the biggest time step ? I think this is a compromise between the number of data and the complexity of the model but I can't see any strong argument to choose between those options.

Edit: if you know a tool (in R Python even Excel) to do it easily it would be very appreciated.

when given two time series with different time steps, what is better: Using the Lowest or the biggest time step ?

For your timeseries analysis you should do both: get to the highest granularity possible with the daily dataset, and also repeat the analysis with the monthly dataset. With the monthly dataset you have 120 data points, which is sufficient to get a timeseries model even with seasonality in your data.

  For known and unknown properties, how should I proceed to go from daily to weekly/monthly data ?

To obtain say weekly or monthly data from daily data, you can use smoothing functions. For financial data, you can use moving average or exponential smoothing, but if those do not work for your data, then you can use the spline smoothing function "smooth.spline" in R: https://stat.ethz.ch/R-manual/R-patched/library/stats/html/smooth.spline.html

The model returned will have less noise than the original daily dataset, and you can get values for the desired time points. Finally, these data points can be used in your timeseries analysis.

  For known and unknown properties, how should I proceed to go from weekly/monthly to daily data ?

To obtain daily data when you have monthly or weekly data, you can use interpolation. First, you should find an equation to describe the data. In order to do this you should plot the data (e.g. price over time). When factors are known to you, this equation should be influenced by those factors. When factors are unknown, you can use a best fit equation. The simplest would be a linear function or piecewise linear function, but for financial data this won't work well. In that case, you should consider piecewise cubic spline interpolation. This link goes into more detail on possible interpolation functions: http://people.math.gatech.edu/~meyer/MA6635/chap2.pdf.

In R, there is a method for doing interpolation of timeseries data. Here you would create a vector with say weekly values and NAs in the gaps for the daily values, and then use the "interpNA" function to get the interpolated values for the NAs. However, this function uses the "approx" function to get the interpolated values, which applies either a linear or constant interpolation. To perform cubic spline interpolation in R, you should use the "splinefun" function instead.

Something to be aware of is that timeseries models typically do some sort of averaging to forecast future values whether you are looking at exponential smoothing or Auto-Regressive Integrated Moving Average (ARIMA) methods amongst others. So a timeseries model to forecast daily values may not be the best choice, but the weekly or monthly models may be better.


Extracting events with attributes from unstructured text I am scraping websites of organisations (mostly retailers) and I want to use NLP to extract information from the websites’ unstructured text. The first thing I want to do is to identify covid-related events in the text, for example “The shop will be closed from the 3rd of March” or “Unfortunately we have to close permanently.” The lexicon is rather limited, involving perhaps a few dozen (or hundreds at most) phrases/expressions.
I am very familiar with regular expressions, and I think it is possible to use a rule-based approach to extract some events and their attributes (e.g., dates), particularly with a small lexicon. However, the limitations of the rules are obvious (it is easy to miss expressions with small variations), and I would like to use also some ML approaches. I am familiar with ML approaches like sentiment analysis and topic modelling, but they seem to be designed for classification problems, rather than on this kind of extraction of specific attributes and data points from text. I also know NER that would work well to get dates and place names, for example, but not for events (e.g., closure of a shop x at date y).
Are there smarter ways to do this kind of NLP, going beyond the manual definition of several RegEx? Perhaps a lexical pattern learning from annotated examples?

I think the closest standard NLP task would be relationship extraction. In general it's a quite complex task which involves NER, syntactic analysis and semantic role labeling.
Note that there are various works using the term "event extraction" (for example this), but as far as I know there is no clear definition of the task. It's often related to putting events on a timeline, this would be quite different from your goal but possibly related.
A basic approach would be to treat the problem as a sequence labeling task like NER: given some annotated "events" in a training corpus, the model might be able to learn the patterns and detect any new "event" in a text.


How would you use a series of events as a factor for a NN? I am working with an LSTM nn built with Keras. I have the need to pass in a history of events as a single factor and I'm thinking of doing it with as a list. But I've never passed in a list in this manner - is that something that is possible with Keras? Or do I need to go through some sort of categorization process for this?

Your input matrix should have dimensions (# of instances, length of history of events, 1). For example, I have text data that is sequential and I will cut it up into sequences which are sequentially dependent. Each row in the matrix is the sequence, the history of events. My label for each of these is the next sequential letter, or event in your case.

The input $X$ has the following shape

print(X.shape)

  (163717, 100, 1)

The LSTM model in Keras.

# define the LSTM model
model = Sequential()
model.add(LSTM(256, input_shape=(X.shape[1], X.shape[2])))
model.add(Dropout(0.2))
model.add(Dense(y.shape[1], activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam')
model.fit(X, y, epochs=20, batch_size=128, callbacks=callbacks_list)

$y$ is my targets. Can be any kind of classification.

If you have 1000 instances of these sequences of events which have 50 historical values then your matrix should have shape (1000, 50, 1)

If you want to add structured data you can always do this after the LSTM layers into the densely connected layer. You can have two models whos outputs are concatenated and then added together. This is a lot easier to do with the functional API of Keras as it is much more flexible and allows you to interact with the tensors more directly. For example

from keras.models import Sequential, Model
from keras.layers import Concatenate, Dense, LSTM, Input, concatenate
from keras.optimizers import Adagrad

seq_input = Input(shape=(X.shape[1], X.shape[2]))
seq_model = LSTM(256)(seq_input)
seq_model = Dropout(0.2)(seq_model)
seq_model = Dense(100, activation = 'softmax')(seq_model)

metadata_input = Input(shape=(8, ))
metamodel = Dense(10, )(metadata_input)

merged = concatenate([seq_model, metamodel])
merged = Dense(64, activation = 'softmax')(merged)
merged = Dense(y.shape[1], activation = 'softmax')(merged)

model = Model(inputs=[seq_input, metadata_input], outputs=merged)
model.compile(loss='categorical_crossentropy', optimizer='adam')

This example is as above. However, I add a metadata input of size 8 to the model after the LSTM layers. This will contribute in the output as it will embed the information from those features in the last few layers of the model.


predicting loan default based on transaction history i have a training dataset with 25,000 different customers, each with a transaction history of 50-500 bank transactions (both deposits and withdrawals, the exact # varies across customers). each customer was given a loan (from 500-2000$), and they either defaulted or did not default on the loan (this information is given in the training set). the test set has around 15,000 different customers (not the same people as above), and i need to predict whether or not they will default based on their transaction history. there is also a string associated to each transaction giving the type of transaction (ie, were they buying beer, coffee, etc).

i'm wondering what is the best way to approach this problem from a machine learning perspective? ie, what model would be best to predict the probability of a customer defaulting on a loan based on their transaction history?

i'm thinking a neural network in keras would be the easiest approach, but i'm not sure how to train it due to the variable input size (different # of transactions per customer). also, i'm wondering if i should make use of the transaction type string, because that adds an entirely new element of difficulty (parsing the string and classifying it into high/low risk type transactions, etc)...unless there are libraries that can do this automatically..

thanks

Some context:
The problem you are trying to solve can be defined as a classification problem.
Where based on features $x^1...x^n$ we are trying to predict if $y$ is class $default$ or class $no$ $default$.
There are several algorithms that are able to learn the different functions between the features and the class, functions that can be used for prediction purposes.
Neural network is one such function.

Answering the question(s):

1) handling variable input size: this is common when looking at time based related data, where each instance might have different history length and or frequency.
One simple approach is deducing the feature space to the biggest common space(I.E, truncate the history lengths to shortest existing in all). Be aware that in this approach you wont use all of data you have.
Another approach is aggregating the history into one or more features:
'total deals', 'average deal amount', 'average time between deals', etc..

2) Handling the 'string' columns: This data can be probably treated as a categorical feature.
Encoding it into several different variables(each representing one(or more) possible value(s) this feature have) is a common way including it as a feature in the learning algorithms.

There are other methods handling each question, but these should be a good starting point.


Predict the first observations of a time series when order of the model is higher Suppose you have you have a time series with 365 observations, one for each day of the year, and you split the first 183 rows in training set and the latest 182 in test set.
Suppose you create an AR (autoregressive model), and you set the order of the model to 4. So you will have:
$y(t) = a1y(t-1)+a2y(t-2)+a3y(t-3)+a4y(t-4)$
In a situation like this, is it possible to do predictions on the first observation of the test set? That basically is the 184th row
I think no, because we do not have y(t-1),...,y(t-4) but we have only y(t)=value of 184th observation.
So, the first row we can predict is 188th, right? Because:
$y(t-1)=value$ of $187^{th}$ row
$y(t-2)=value$ of $186^{th}$ row
$y(t-3)=value$ of $185^{th}$ row
$y(t-4)=value$ of $184^{th}$ row
I think that until now is right. Correct me if I am wrong.
But if I want to predict the obs. 184, the first one of test set, is there any way? I mean, without decreasing the order of the model from 4 to 1 for example.

The obvious answer is to use the last 4 data points of your training dataset. Note that there is no harm or bias in doing it. The purpose of breaking the data set into training and test datasets is to estimate and forecast on different datasets.
In your model, the dependent variable is $y_5$ to $y_{183}$. On the other hand, the four explanatory variables are: $y_1-y_{179}$; $y_2-y_{180}$; $y_3-y_{181}$; and $y_4-y_{182}$.
So the test data set for each of the explanatory variable actually starts from $y_{180}$, $y_{181}$, $y_{182}$ and $y_{183}$ respectively.
Therefore, by using the last four data points of your train dataset, you are not introducing any bias in terms of revalidating the model on fitted data.


Is there any model agnostic way to calculate the weight importance for neural networks given a set of inputs? I was curious if it's possible to calculate what weights are important and what weights are redundant (or have high redundancy) for separate tasks in neural networks?
And if this is doable in a model agnostic way (maybe also differentiable..)?

You should look into fisher matrices. Elastic weight consolidation finds weights which are important for dataset 1 when training dataset 2 by finding the fisher matrix and slows the tuning of them down by adding a regularization term to the loss function. It's non trivial though and I got a lot of trouble implementing it in PyTorch.
The paper which proposed elastic weight consolidation is here: https://arxiv.org/pdf/1612.00796.pdf


Subsequence parttern matching for time series I have a set of time series data (just like voice sequence data) with the pattern as shown in the first figure (theoratical data). The measured data is given as presented in the second figure. What I want to do is localizing/finding the subsequece pattern as shown in the red squares. Is there any algorithm to solve this problem? It looks like the classification/regression problem in machine learning, but I have no idea on how to start it.

The real state of the art here is the Matrix Profile suite, developed by Eamonn Keogh and his team in University of California at Riverside (UCR). Here are some links to get you started:

Matrix Profile Foundation homepage
The UCR Matrix Profile Page
MPA: a novel cross-language API for time series analysis paper (2020) with links to Python, R, and Go implementations
Introducing matrixprofile-ts, a Python Library for Detecting Patterns and Anomalies in Massive Datasets
Towards a Near Universal Time Series Data Mining Tool: Introducing the Matrix Profile (PhD thesis)
100 Time Series Data Mining Questions (with answers!) (highly recommended high-level view)

You'll find links to implementations in the above stuff.
Apart from the team at UCR, there is another (and possibly more efficient) implementation of the relevant algorithms in the STUMPY Python package:

STUMPY documentation & Github repo
STUMPY: A Powerful and Scalable Python Library for Time Series Data Mining paper (2019)


class weights formula for imbalanced dataset I am trying to make some semantic segmentation. I have 7 imbalanced classes in my case.
I found several methods for handling Class Imbalance in a dataset is to perform Undersampling for the Majority Classes or Oversampling for the minority classes. but the most used one is introducing weights in the Loss Function. And I found several formula to calculate weights such us:
wj=n_samples / (n_classes * n_samplesj)
or wj=1/n_samplesj
which is the best one?

I really don't suggest Under/Oversampling as it would change the distribution of dataset. we should consider distribution as a useful feature of dataset. so I think the weighted loss would have better performance in most cases. if you're using TF/Keras, this link would be useful. you can use a variety of loss functions, like the below one, to apply the weight.
tf.nn.weighted_cross_entropy_with_logits(
    labels, logits, pos_weight, name=None
)

A value pos_weight > 1 decreases the false negative count, hence increasing the recall. Conversely setting pos_weight < 1 decreases the false positive count and increases the precision.


Pearson correlation with data sets that have values on different scales I have two datasets with which I want to do a Pearson correlation analysis. I have carried out the analysis which makes sense, however I want to be sure it is valid given that both datasets have values on different scales.
The features in both datasets are exactly the same (the actual samples are of course different). The range of values are as follows:
dataset1 = 3-20
dataset2 = 10-30

Now my understanding is that pearson correlation coefficient is not dependent on the change or difference in scale of the different datasets, however for all examples I have seen, although the scales are different, the lower value is always 0. As you can see though, the lower value on one dataset is 3 and the lower value on the other is 10. (in either case though they mean they same level of 'low', it is just how they are measured.
Can anyone confirm this is still fine?

For quantitative-quantitative comparisons, scales generally do not matter in spearmean, pearson, and kendall correlation.


What is the advantage of positional encoding over one hot encoding in a transformer model? I'm trying to read and understand the paper Attention is all you need and in it, they used positional encoding with sin for even indices and cos for odd indices.

In the paper (Section 3.5), they mentioned

  Since our model contains no recurrence and no convolution, in order for the model to make use of the order of the sequence, we must inject some information about the relative or absolute position of the tokens in the sequence. To this end, we add "positional encodings" to the input embeddings at the bottoms of the encoder and decoder stacks.

My question is that if there is no recurrence, why not use One Hot Encoding. What is the advantage of using a sinusoidal positional encoding?

The theoretical advantage should be that the network should be able to grasp the pattern from the encoding and thus generalize better for longer sentences. With one-hot position encoding, you would learn embeddings of earlier positions much more reliably than embeddings of later positions.

On the other hand paper on Convolutional Sequence to Sequence Learning published shortly before the Transformer uses one-hot encoding and learned embeddings for positions and it seems it does not make any harm there.


Why do cost functions use the square error? I'm just getting started with some machine learning, and until now I have been dealing with linear regression over one variable.

I have learnt that there is a hypothesis, which is:

$h_\theta(x)=\theta_0+\theta_1x$

To find out good values for the parameters $\theta_0$ and $\theta_1$ we want to minimize the difference between the calculated result and the actual result of our test data. So we subtract

$h_\theta(x^{(i)})-y^{(i)}$

for all $i$ from $1$ to $m$. Hence we calculate the sum over this difference and then calculate the average by multiplying the sum by $\frac{1}{m}$. So far, so good. This would result in:

$\frac{1}{m}\sum_{i=1}^mh_\theta(x^{(i)})-y^{(i)}$

But this is not what has been suggested. Instead the course suggests to take the square value of the difference, and to multiply by $\frac{1}{2m}$. So the formula is:

$\frac{1}{2m}\sum_{i=1}^m(h_\theta(x^{(i)})-y^{(i)})^2$

Why is that? Why do we use the square function here, and why do we multiply by $\frac{1}{2m}$ instead of $\frac{1}{m}$?

Your loss function would not work because it incentivizes setting $\theta_1$ to any finite value and $\theta_0$ to $-\infty$.

Let's call $r(x,y)=\frac{1}{m}\sum_{i=1}^m {h_\theta\left(x^{(i)}\right)} -y$ the residual for $h$.

Your goal is to make $r$ as close to zero as possible, not just minimize it. A high negative value is just as bad as a high positive value.

EDIT: You can counter this by artificially limiting the parameter space $\mathbf{\Theta} $(e.g. you want $|\theta_0| < 10$). In this case, the optimal parameters would lie on certain points on the boundary of the parameter space. See https://math.stackexchange.com/q/896388/12467. This is not what you want.

Why do we use the square loss

The squared error forces $h(x)$ and $y$ to match. It's minimized at $u=v$, if possible, and is always $\ge 0$, because it's a square of the real number $u-v$.

$|u-v|$ would also work for the above purpose, as would $(u-v)^{2n}$, with $n$ some positive integer. The first of these is actually used (it's called the $\ell_1$ loss; you might also come across the $\ell_2$ loss, which is another name for squared error).

So, why is the squared loss better than these? This is a deep question related to the link between Frequentist and Bayesian inference. In short, the squared error relates to Gaussian Noise.

If your data does not fit all points exactly, i.e. $h(x)-y$ is not zero for some point no matter what $\theta$ you choose (as will always happen in practice), that might be because of noise. In any complex system there will be many small independent causes for the difference between your model $h$ and reality $y$: measurement error, environmental factors etc. By the Central Limit Theorem(CLT), the total noise would be distributed Normally, i.e. according to the Gaussian distribution. We want to pick the best fit $\theta$ taking this noise distribution into account. Assume $R = h(X)-Y$, the part of $\mathbf{y}$ that your model cannot explain, follows the Gaussian distribution $\mathcal{N}(\mu,\sigma)$. We're using capitals because we're talking about random variables now.

The Gaussian distribution has two parameters, mean $\mu = \mathbb{E}[R] = \frac{1}{m} \sum_i h_\theta(X^{(i)})-Y^{(i))}$ and variance $\sigma^2 = E[R^2] = \frac{1}{m} \sum_i \left(h_\theta(X^{(i)})-Y^{(i))}\right)^2$. See here to understand these terms better.

Consider $\mu$, it is the systematic error of our measurements. Use $h'(x) = h(x) - \mu$ to correct for systematic error, so that $\mu' = \mathbb{E}[R']=0$ (exercise for the reader). Nothing else to do here.
$\sigma$ represents the random error, also called noise. Once we've taken care of the systematic noise component as in the previous point, the best predictor is obtained when $\sigma^2 = \frac{1}{m} \sum_i \left(h_\theta(X^{(i)})-Y^{(i))}\right)^2$ is minimized. Put another way, the best predictor is the one with the tightest distribution (smallest variance) around the predicted value, i.e. smallest variance. Minimizing the the least squared loss is the same thing as minimizing the variance! That explains why the least squared loss works for a wide range of problems. The underlying noise is very often Gaussian, because of the CLT, and minimizing the squared error turns out to be the right thing to do!

To simultaneously take both the mean and variance into account, we include a bias term in our classifier (to handle systematic error $\mu$), then minimize the square loss.

Followup questions:

Least squares loss = Gaussian error. Does every other loss function also correspond to some noise distribution? Yes. For example, the $\ell_1$ loss (minimizing absolute value instead of squared error) corresponds to the Laplace distribution (Look at the formula for the PDF in the infobox -- it's just the Gaussian with $|x-\mu|$ instead of $(x-\mu)^2$). A popular loss for probability distributions is the KL-divergence.
-The Gaussian distribution is very well motivated because of the Central Limit Theorem, which we discussed earlier. When is the Laplace distribution the right noise model? There are some circumstances where it comes about naturally, but it's more commonly as a regularizer to enforce sparsity: the $\ell_1$ loss is the least convex among all convex losses.

As Jan mentions in the comments, the minimizer of squared deviations is the mean and the minimizer of the sum of absolute deviations is the median. Why would we want to find the median of the residuals instead of the mean? Unlike the mean, the median isn't thrown off by one very large outlier. So, the $\ell_1$ loss is used for increased robustness. Sometimes a combination of the two is used.

Are there situations where we minimize both the Mean and Variance? Yes. Look up Bias-Variance Trade-off. Here, we are looking at a set of classifiers $h_\theta \in H$ and asking which among them is best. If we ask which set of classifiers is the best for a problem, minimizing both the bias and variance becomes important. It turns out that there is always a trade-off between them and we use regularization to achieve a compromise.

Regarding the $\frac{1}{2}$ term

The 1/2 does not matter and actually, neither does the $m$ - they're both constants. The optimal value of $\theta$ would remain the same in both cases.

The expression for the gradient becomes prettier with the $\frac{1}{2}$, because the 2 from the square term cancels out.

When writing code or algorithms, we're usually concerned more with the gradient, so it helps to keep it concise. You can check progress just by checking the norm of the gradient. The loss function itself is sometimes omitted from code because it is used only for validation of the final answer.

The $m$ is useful if you solve this problem with gradient descent. Then your gradient becomes the average of $m$ terms instead of a sum, so its' scale does not change when you add more data points.

I've run into this problem before: I test code with a small number of points and it works fine, but when you test it with the entire dataset there is loss of precision and sometimes over/under-flows, i.e. your gradient becomes nan or inf. To avoid that, just normalize w.r.t. number of data points.

These aesthetic decisions are used here to maintain consistency with future equations where you'll add regularization terms. If you include the $m$, the regularization parameter $\lambda$ will not depend on the dataset size $m$ and it will be more interpretable across problems.


How to use build_analyzer in sklearn feature extraction I'm trying to get list of n-gram tokens for text

Ex: 'How to use build_analyzer in sklearn feature extraction '

output :['How', 'use', 'build_analyzer', 'sklearn', 'feature', 'extraction', 'How use', 'use build_analyzer', 'build_analzer sklearn', 'sklearn feature', 'feature extraction']

from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer

vectorizer = TfidfVectorizer(stop_words = 'english',ngram_range=(1, 2), token_pattern=r'\b\w+\b', min_df=1)

 df['Text'].apply(lambda x : vectorizer.build_analyzer(x))

TypeError: build_analyzer() takes 1 positional argument but 2 were given

build_analyzer() returns a callable that let's you extract the tokenizing step from the transformation pipeline wrapped in the CountVectorizer or TfidfVectorizer. You can do something like this:

analyze = vectorizer.build_analyzer()
df['Text'].apply(lambda x: analyze(x)) #or df['Text'].apply(analyze)


Should output data scaling correspond to the activation function's output? I am building an LSTM with keras which have an activation parameter in the layer. I have read that scaling on the output data should match the activation function's output values.

Ex: tanh activation outputs values between -1 and 1, therefore the output training (and testing) data should be scaled to values between -1 and 1. So if the activation function is asigmoid the output data should be scaled to values between 0 and 1.

Does this hold for all activation functions? If I use ReLu as activation in my layers what should the output data be rescaled to?

What you read hold true for the neurons of the output layers and not for the hidden layers!

Hence, its true that if you are using tanh in output layers then you need the data labels to be within [-1, 1] where as between [0, 1] for sigmoid.

As for your concern with relu, use it on output layers if you know that the range of the labels of the data is positive only. If you are using relu in hidden layers then the scaling doesn't depend on relu but rather on the type of activation function used in output layers.


recommendation system for eCommerce healthcare portal suggestion I am trying to build a recommendation system. My system is basically a ecommerce application where our customers answers a bunch of questions related to healthcare (their basic health related question). Based on their ansers, we recommend some product. This process of recommendationis based of conventional rule base aapproach. Think of it as bunch of if-else condition. Now I am playing around with some machine learning technique nd want to see if this way will add any value in our healthcare system. I am at the very starting point and can use any suggestion from you guys. The suggestion could aim towards following:

Any product that you feel which leverages ML technique with respect to health(considering HIPPA constraints)
Any product that you feel which leverages ML technique with respect to health
What could be the first step towards building such a system.

Your recommendation system will be designed to tell the customer what product they should choose, however, this doesn't account for what products the customer likes.  A ML method could take all the input parameters from the recommendation system and provide a recommended product based on what similar users liked.

There's no specific ML technique for considering HIPPA constraints.  This would come into play more in the pre-processing stage.  For example, you might not be able to acquire specific birth dates and addresses, but maybe you can get an age range and a zip code (first 3 digits only).

Logistic regression is frequently used in healthcare for binary classification.

There are lots of tutorials online for building ML models.  Kaggle has a good tutorial for Python and R using titanic survival data.


Independence/correlation test between features (not feautre-label) I'm doing a Naive Bayes prediction model where I've 666 features to select from.

I've tried the SelectKBest chi2 test but it's a features-label test whereas what I'm trying to do is to find out the independency and also correlation feature-features. As the assumption to use Naive Bayes is that variables are all independent to each other, I have to remove features that are dependent/correlated to each other.

What are the other independence tests that I should look into apart from $chi^2$? (I have continuous features)

This package is very useful, this does the feature engg. for you, gives you the important variables. sample code would be something like this.

    set.seed(123)
    boruta.input <- Boruta(target variable~., data = training_data, doTrace = 2)
    print(boruta.input)

    #plot a graph for better understanding
    plot(boruta.input, xlab = "", xaxt = "n")
    lz<-lapply(1:ncol(boruta.input$ImpHistory),function(i)
    boruta.input$ImpHistory[is.finite(boruta.input$ImpHistory[,i]),i])
    names(lz) <- colnames(boruta.input$ImpHistory)
    Labels <- sort(sapply(lz,median))
    axis(side = 1,las=2,labels = names(Labels),
    at = 1:ncol(boruta.input$ImpHistory), cex.axis = 0.7)

By using this you can get all the important features but the downfall is it takes time if you have more data
In my case my data consists 40 features and 200,000 records it took almost 2 hours but the results were good.

For better understanding you can go through this link

--

I think you can use this test too, in the description it tells that it can be applied on numerical data too.

If you have data which is Nominal data, you can use G-test

Attaching one more link along with this, which consists of tests for different kinds of features. Might be helpful in the future Link

I have a question, why aren't you applying PCA(or any DR Techniques) on the data-set? (assuming that you din't try any DR Techniques)

If you apply DR Techniques, I think you might get better Components which would aid you in getting better results/better understanding of data.

As you mentioned that you were using NB Classifier, why did you choose that, is there any specific reason?
As you know that there are many other Classifiers which might outrun NB Classifier.
In my case I used NB Classifier for twitter sentiment mining, as those are short sentences and NB Classifier best suited for my analysis.

Hope my answer is helpful!

NB: Naive Bayes

DR: Dimensionality Reduction


Scaling thousands of automated forecasts in R Hello and thanks in advance!  I'd like some advice on a scalability issue and the best way to resolve.  I'm writing an algorithm in R to produce forecasts for several thousand entities.  One entity takes about 43 seconds to generate a forecast and upload the data to my database.  That equates to about 80+ hours for the entire set of entities and that's much too long.

I thought about running several R processes in parallel, possibly many on a few different servers, each performing forecasts for a portion of total entities.  Though that would work, is there a better way?  Can Hadoop help at all?  I have little experience with Hadoop so don't really know if it can apply.  Thanks again!

If you're working with R language, I would suggest first to try use R ecosystem's abilities to parallelize the processing, if possible. For example, take a look at packages, mentioned in this CRAN Task View.

Alternatively, if you're not comfortable or satisfied with the approaches, implemented by the above-referred packages, you can try some other approaches, such as Hadoop or something else. I think that a Hadoop solution would be an overkill for such problem, considering the learning curve, associated with it, as well as the fact that, as far as I understand, Hadoop or other MapReduce frameworks/architectures target long-running processes (an average task is ~ 2 hours, I read somewhere recently). Hope this helps.


What could make a set of the train data more predictive than the whole train data I took a sample of my training data and balanced it and then trained my model. The results obtained are more accurate than using the whole set of train data (balanced or imbalanced). My question is: what could explain this result? (the entire set of data does not contain any noise).

Thanks in advance.

In absence of noise and if the difference of accuracy you observe is significant, the only reason I see is that, by luck, the distribution of the training data subset happens to be closer to the distribution of your validation data than the distribution of the whole training data.
This should not happen with truly representative validation data (i.e. with training and validation data built by random split of a given dataset).


Explainable AI and unsupervised algorithms There are several packages that allow explaining ML algorithms (Lime, Shap and so on). However, it is not clear how we can explain unsupervised algorithms for example, if we use PCA for dimensionality reductions first and random forest after, how it is possible to explain the whole process?

PCA isn't a great example, as it's inherently explainable, in the same way that linear regression is inherently explainable.

In supervised learning, at a high level, the Shapley value for each feature tells you the extent to which including that feature in an ML model allows you to make better predictions about the target variable than if you did not have access to that feature. One way of implementing this is to calculate the difference in test cross entropy between a model which includes that feature and one which does not (and it's much more complicated than that in reality, as there are many models which do not include that feature, you have to average over all coalitions of other features, but that's another story).

So in short, you need some sort of cost function, which including features tends to improve.

Now let's turn our example to the simplest of all clustering algorithms, KMeans. In KMeans, the most sensible cost function is probably to split your data into train and test, fit on a training set, and then take the sum over your test set, of each point's euclidean distance from its nearest centroid.

Now, what happens if you remove a feature? Well, in general, post fitting, the sum described above will run over fewer features, so the cost will be lower. This in turn will mean that every feature has a negative Shapley value...not good

More generally, in unsupervised learning, the cost function tends to be related to the probability of seeing a datapoint, according to the model, i.e. the likelihood. The key however, is that this is not just the likelihood of the target, it's the entire, multidimensional likelihood of all features (indeed, there is no target in unsupervised learning).

What you would need, in order to calculate Shapley values for unsupervised learning problems, is some way to describe "how good the clustering is" with one number. The "goodness of clustering" would need to be comparable when you changed the dimensionality of the problem...and standard cost functions used in inferential clustering (which KMeans is kind of a simplified form), do not satisfy this.

So I've only answered a subset of your problem, I've said that Shapley values don't really make sense for inferential clustering problems, rather than "explainability doesn't make sense for unsupervised algorithms". I don't know enough about all unsupervised algorithms to claim Shapley values don't make sense for any unsupervised algorithm, but it certainly seems to me to be the case, that you'll need an algorithm which lends itself to determining for any coalition of features included in the clustering, "how good the clustering is", in a way which somehow "objectively measures whether similar instances are grouped", rather than which calculates full data likelihood in a way which is inversely proportional to the number of features in the model. This seems to go against the idea of unsupervised learning, namely that there's no objective truth.


What is Extreme Learning Machine? Why tuning of weights is not required? The wikipedia says:

  Extreme learning machines are feedforward neural network for classification, regression,
      clustering, sparse approximation, compression and feature learning with a single layer or
      multi layers of hidden nodes, where the parameters of hidden nodes (not just the weights
      connecting inputs to hidden nodes) need not be tuned.

How is it possible to assign values to links without actually seeing the Training set?

Till now from my research to understand this, I've made a lead, but not satisfactory because I can't anything from this below paragraph.

  For any nonconstant piecewise continuous function
  that is used as the activation function, if the parameters of
  the hidden neurons are tuned, then the function can make
  the SLFNs approximate any target continuous function f (x).
  Then, according to any continuous distribution probability, the
  function sequence {h subscript i (x)} where i goes from 1 to L   can be randomly generated, and
  it has the universal approximation capability, which means that
  $$\lim_{L→ \infty} ||L
i=1 βi hi (x) − f (x)|| = 0$$
  holds with probability
  of one with appropriate output weights $β$.

SOURCE: A Parallel Multiclassification Algorithm for Big
Data Using an Extreme Learning Machine, IEEE transactions on neural networks 2017

Can anyone prove/explain this with an example please?

Also any extra knowledge on extreme learning machine is very welcomed.

ELM is actually solving a generalized linear problem :

\begin{equation}
H\beta = Y
\end{equation}

where $H$ is the matrix of the hidden layer output, $\beta$ is the weight vector (or matrix) connecting the hidden layer and the output layer and $Y$ symbolizes the target of your training.

The goal is to find $\beta$ using the pseudo-inverse of H.

Solving this problem is equivalent of finding the $\beta$ achieving the minimum of the Least-square problem $||H\beta - T ||$ :
\begin{equation}
\beta = \underset{\beta}{\text{min}} \left \vert\left \vert H\beta - T \right\vert\right\vert
\end{equation}
To answer your question

  How is it possible to assign values to links without actually seeing the Training set?

Since $H$ depends on the activation functions, on the initialized weight matrix linking Input layer and hidden layer and on each hidden node bias, it encompasses all the input information and ensures a learning by tuning $\beta$ coherently.

You can find lot more details in ELM Theory and Application or in this ELM presentation for example.


Statistical comparison of model performance when training and validation data is always the same I have a labelled dataset of text in 4 languages (1000 samples per languages makes a total of 4000 samples). In one experiment I would like to assess the performance of a classification algorithm (neural network) on an unseen language. That means using the 3000 tweets from three languages as training data and 1000 samples from the fourth language as validation data:

training data:
- 1000 English samples
- 1000 German samples
- 1000 Italian samples

validation data:
- 1000 Dutch samples

Now, I am trying to find a statistic to find if there is a significant difference between 2 versions of the classification algorithm. I have been reading and find that 2x5 cross-validation or 10x10 cross validation using a modified student t-test is (one of) the best options. While I can apply this to most other experiments, I don't think I can apply this directly to this particular experiment since I am not using folds.

My question is, if I can still use the corrected student t-test training the network a number of times on the same training and validation data? If so, what should be the number of training and validation folds? If not, what would be a better approach?

For reference, below is my Python implementation of the Nadeau and Bengio correction using the equation stated here.

def corrected_dependent_ttest(data1, data2, n_training_folds, n_test_folds, alpha):
    n = len(data1)
    differences = [(data1[i]-data2[i]) for i in range(n)]
    sd = stdev(differences)
    divisor = 1 / n * sum(differences)
    test_training_ratio = n_test_folds / n_training_folds
    denominator = sqrt(1 / n + test_training_ratio) * sd
    t_stat = divisor / denominator
    # degrees of freedom
    df = n - 1
    # calculate the critical value
    cv = t.ppf(1.0 - alpha, df)
    # calculate the p-value
    p = (1.0 - t.cdf(abs(t_stat), df)) * 2.0
    # return everything
    return t_stat, df, cv, p

If your model is deterministic (no randomness), then repeating the training/testing on the exact same set of data is pointless - you will get the exact same answer every time. The benefit of cross-validation is that it provides an unbiased estimate of your model performance, and does so by using different perturbations of the train/test data. You can still do something similar, selecting 80% of your training data and testing on some subset of the test data, and repeatedly doing a resampling. There's a slight difference from traditional CV, where your train/test set are mutually exclusive and essentially define one another, whereas in this case, your training and test datasets can be defined totally independently (but that shouldn't be a problem).

Incidentally, what model are you using that you expect can accurately predict data that's completely unlike what it's been trained on? Usually the point of the training data is to provide the model with examples of the data you expect to see, along with the correct output. It's not clear to me why you'd want to train a model to classify tweets in English, but then test how it performs at classifying Dutch tweets - the model has never seen Dutch, so I don't expect it would perform well. It seems this evaluation would test how similar English and Dutch are, rather than testing how good your classification model is.


Machine learning tasks classification I am trying to be precise in definitions.

We can solve regression, classification, clusterisation, dimensionality reduction, visualization, feature extraction tasks.

But also there are supervised, unsupervised, ..., tasks of ML.

I see that regression is a part of the supervised learning.

Are there any other particular tasks?
How to make the word "tasks" more clear?

Some of the "tasks" (there are many more, for example anomaly detection; and people just love to invent their own tasks marginally different from before hoping to be the first to spawn a new subdomain...) can both be supervised or unsupervised.

Example:

PCA is unsupervised dimensionality reduction
Fishers' LDA is supervised dimensionality reduction

... and both predate any notion of machine learning, data science or data mining. These things are just rebranded into ML now, squeezed into a very narrow view that never was able to capture the essence of even clustering (beyond k-means, that seems to be the only method machine learners still understand and get taught, as the L2 nature of the method fits their world view well enough). I'd rather avoid making all this "machine learning" - it was not, and it doesn't fit well.


Alternatives for categorical prediction Upfront question:

What are some alternative methods for predicting categorical data?

Details:
I routinely process data that is 100% categorical. Almost always, this data is nominal (while, occasionally ordinal data such as 'quarters' may be included). When the time comes to run predictive modeling, I feel boxed-in to utilize only one method: multinomial regression models.
I have yet to have success employing randomForest or decision trees due to the amount of levels for the data (e.g., some variables have upwards of 111 levels in a set of 12000 observations).
I understand that, given the restricted data type, my options will be limited. However, I am wondering if anyone else has created predictive models with (only) nominal data and what method was chosen.

Note: I utilize R [3.6.1] for processing and modeling.

There are many supervised machine learning algorithms for classfication.
Here is a list to get you started:

Tree-based

Decision trees
Random Forest
Boosted Trees

Naive Bayes (NB)
K-nearest neighbors algorithm (k-NN)
Support vector machines (SVM)
Neural Networks / Deep Learning (DL)


Geo Add-on Choropleth Widget not found I am trying to create a Choropleth map using the Chorolpeth widget from the Geo Add-on in Orange. However, this widget is not appearing? Any ideas?
My current version: Orange version 3.24.1 for Windows

Two things:

Try reinstalling/updating Orange to the latest version.

Try reinstalling/updating the Geo package in Orange, from the Add-ons menu, to the latest version.

I am currently running Orange version 3.27.1 and Geo version 0.2.8 and the Choropleth widget appears.


focal loss function help I am working on a relation extraction and classification problem. The data is in the form of text files. The data is imbalanced. I want to use focal loss function to address class imbalance problem in the data. My question is: Can focal loss  be utilized for extraction and classification task to increase the accuracy? Focal loss has been applied on object detection task and for image classification task. The link is below. I want to use this on text classification task.

https://shaoanlu.wordpress.com/2017/08/16/applying-focal-loss-on-cats-vs-dogs-classification-task/

http://openaccess.thecvf.com/content_ICCV_2017/papers/Lin_Focal_Loss_for_ICCV_2017_paper.pdf

From Focal Loss for Dense Object Detection paper, focal loss is defined as:

$$FL(p_t) = −(1 − p_t)^γ log(p_t)$$

It is a generalized formation of a loss function that can be directly implemented for any classification problem, including text classification.

Focal loss function is designed for problems with 1:1000 training imbalance. Does your data actually have that much imbalance? If not, you can use much simpler methods (i.e., resampling).


Intuition of "Head" in Attention models (Transformer)? I keep seeing the "head" in attention models (transformers). Aside from the mathematical formula, could anyone please share the intuition behinds the idea "head" ? Thanks a lot!

I like to think about it in the context of progression of attention mechanisms in neural networks.
Early attention mechanisms were implemented as explicit sliding windows over the encoding sequence.  For example, Graves 2013 implemented it as the average of several sliding Gaussians.  So, it was a local attention mechanism to attend to a learnable window over the data.
Then you have Luong and Bahdanau attention, which implemented similar mechanisms, but separate from the network type and as a softmax function.  This is essentially single-head attention.  Now it's not a window, but a score and weighted average over all of the encoding input.  The idea of the K, Q, V flows naturally from that, but in that case the Q is a hidden state of the decoder, and K/V the hidden state of the encoder.  This is more flexible and can attend to more arbitrary windows over the sequence.
Transformers flow naturally from these mechanisms, but now K, Q, V all come from the same sequence, except for the second layer of the decoder, which acts the same as Luong or Bahdanau attention.  In addition, they first split the K, Q, V along their last dimension (the model parameter dimension) to create several heads that can attend to different parts of the sequence.  This is much better at attending to long term and varied parts of the sequence to concatenate an overall context for each Query.  This, of course, is very advantageous for natural language translation, but as we are seeing, it is also great for covering the image space in vision problems.  Intuitively, this makes sense as I view the Transformer as almost a superset of a convolution network, where the filters for each partition of an image attend to various parts of the image and this is learned.
Obviously, there are many papers I'm sure I haven't read and steps in between these.


If my data looks like this, is linear regression just never gonna be useful? Clearly you can see that there is no linear relationship. If there was one it would be a vertical line which is not that useful. I need some type of regression rather than classification as the output is continuous. The only options I can think of are neural networks and SVR (support vector regressors). Is linear regression a waste of time here? What would you use?

The use case is - I am trying to predict the rgb color of a polygon based on its area, length, index, complexity, #of lines, # of curves, and intensity (black and white).

In this first step I am merely trying to predict the amount of red from 0 to 1. (scaled down from 255).

Edit:

here are the labels - rgb values: http://www.heypasteit.com/clip/2HZX

here is the data with the 7 features: http://www.heypasteit.com/clip/2HZY

I tried running a RANSAC model on your data, but got worse results than a straight linear regressor. The ten-fold cross-validated mean absolute error over all three response variables (r,g,b) for the linear model was about 0.37. I also ran a random forest model for comparison, and got about the same score. This suggests that the linear model is not shabby, but it is up to you to decide if either is good enough.

import sklearn.linear_model, sklearn.cross_validation, sklearn.ensemble, pandas

labels = pandas.read_csv('labels.csv', header=None, names=['r', 'g', 'b'])
features = pandas.read_csv('features.csv', header=None, names=['area', 'length', 'index', 'complexity', 'lines', 'curves', 'intensity'])

sklearn.cross_validation.cross_val_score(sklearn.linear_model.LinearRegression(), features, labels[:len(features)], 'mean_absolute_error', 10, -1).mean()

I also tried visualizing the data with the outliers clipped, but did not find it very enlightening, so I did not include it here.


Why do people use CrossEntropyLoss and not just a softmax probability as the loss? I don't understand why one would add additional complexity to log, probabilities for the loss function of a classification Neural Network. What benefit does that have, as opposed to just using the 0-1.0 values(probabilities of a class) you get from the softmax function at the final layer?

Does this add extra non-linearity that we don't understand why it does good, but just happens to do good a lot of times since we give the Neural Net some more complexity?

Let me explain it with logistic regression first:

The output of logistic regression is $\hat{y}= \dfrac{1}{1 + e^{-\theta x}}$
Consider the minimum square loss function $\mathcal{L} = (1 -\hat{y})^2$
substitute $\hat{y}$ in the second equation $\mathcal{L} = (\dfrac{e^\theta x}{1 + e^{-\theta x}})^2$

Tha above formula is not convex, so optimization is much more difficult since the gradient may point to local optima.

Whereas cross entropy looks like.$$\mathcal{L} = -y\log \hat{y} - (1-y)\log (1-\hat{y}) $$ which has two advantages.

It is convex when subtitute $\hat{y}$ for its value (considering that $y$'s value is either 1 or 0)
For each row in your dataset you are taking on count both labels at the same time

When we generalize from binari logistic regression to multiclass classification, we use the same ideas of using convex loss functions with respect to the output and also loss functions which minimize at the same time for all labels.


Is Gradient Descent central to every optimizer? I want to know whether Gradient descent is the main algorithm used in optimizers like Adam, Adagrad, RMSProp and several other optimizers.

No. Gradient descent is used in optimization algorithms that use the gradient as the basis of its step movement. Adam, Adagrad, and RMSProp all use some form of gradient descent, however they do not make up every optimizer. Evolutionary algorithms such as Particle Swarm Optimization and Genetic Algorithms are inspired by natural phenomena do not use gradients. Other algorithms, such as Bayesian Optimization, draw inspiration from statistics.

Check out this visualization of Bayesian Optimization in action:

There are also a few algorithms that combine concepts from evolutionary and gradient-based optimization.

Non-derivative based optimization algorithms can be especially useful in irregular non-convex cost functions, non-differentiable cost functions, or cost functions that have a different left or right derivative.

To understand why one may choose a non-derivative based optimization algorithm. Take a look at the Rastrigin benchmark function. Gradient based optimization is not well suited for optimizing functions with so many local minima.


Differences between normalization and standarization in multiple regression Consider the following question regarding multiple regression

1) Can someone explain why we have to transform dependent variable using log-transformation (Normalization) when appear positive skewed y variable in regression model?

2) After log-transformation whether do I need to standardize that y variable using min max scale or StandardScaler methods?

3) If independent variables also have skewed data then do I need to Normalization for that variables or standardizing is enough?

I'll go through your question one by one.

  1) Can someone explain why we have to transform dependent variable using log-transformation (Normalization) when appear positive skewed y variable in regression model?

Not necessarily log transformations, any kind of transformation (square, square-root, log, Z-scores, you name it) necessary to make the distribution of your data look more "Normal" (i.e. Gaussian). That is because all mainstream (frequentist) statistical models rely on the normality assumption of data (and residuals). When data are not Normal enough, the computation of parameters such as confidence intervals, standard errors, and p values will be unreliable.

  2) After log-transformation whether do I need to standardize that y variable using min max scale or StandardScaler methods?

That is not mandatory either. Sometimes it is useful to scale your dependent variable in a range such that all its likely values are "easy to reach" by the parameters of your predictive model.

  3) If independent variables also have skewed data then do I need to Normalization for that variables or standardizing is enough?

Normalization is a technique that can work. There is no general rule though. Sometimes you can use min-max scaling, some other times Z-scores work better. Other times again you'd achieve better performance with some custom scaling technique that invented ad hoc.


Privacy through moving averages? I am considering the following hypothetical situation: I have a time series of data. In general, 'the public' should have access to features of this data. However, making the time series available would constitute a privacy leak. I am considering making a moving average available instead.

Can anyone recommend either some literature on this, or some alternative methods?

I understand that this is a case by case question. However, I think there should be a general answer available along the following lines:

1) Privacy leaks occur because you can match up the time stamp to an individual, by using outside information.

2) Therefore, you want to make it so that each window aggregates the data of several individuals. (The data is of a form where the mean is a meaningful quantity.)

There are of course ways to break this privacy, if one is sufficiently determined. I think in this case no one is. So I'm looking for literature that deals with some real world case studies, if possible.

(This situation is hypothetical. I do not have access to the data. I am 'the public' that wants the data, and I want to suggest a reasonable approach for aggregation.)

I also asked this question over on cross validated: https://stats.stackexchange.com/questions/324204/privacy-through-moving-averages

The key question is how we can allow the public to make useful queries to a dataset without revealing private information.

The field of Differential Privacy deals with answering just that.

The key concept is to track a user's queries and the information revealed and add uncertainty to the answers to guarantee privacy.

For example, a broad query like "what is the rate of cancer in Europe?" could have an answer of with "1% +/- .1%", whereas "what is the rate of cancer for the population of men aged 72 living at my neighbours's address?" would give an answer of "50% +/- 50%": we do not want to reveal whether a specific individual has cancer or not.

Differential privacy gives users an information budget and tracks the queries, results and information revealed. The more you ask, or the closer you get to personal information the less specific the answer. At one point the information budget is exhausted and can query no more. You end up with nothing more than "your neighbour has cancer with probability p=0.5 +/- 0.5".

This also guards against constructing a clever sequence of queries to pin down the answer like in the children's game of "20 questions". For example, you might try to ask questions like "what is the rate of men of cancer for men of age 72 in this city?", "what is the rate of cancer for men of height 182 cm at age 72?", "what is the height distribution of men with my cancer in my street?" etc. until you can deduce whether your 182 cm tall, 72 year old, male neighbour has cancer or not.

Differential Privacy guarantees that such strategies will never work (under certain assumptions).

Moving Averages

With this background, your example of only publishing the moving average seems quite reasonable. Differential Privacy can be used to calculate how long or short the moving average needs to be on your dataset in order to provide the best possible information while preserving privacy.


What feature engineering is necessary with tree based algorithms? I understand data hygiene, which is probably the most basic feature engineering. That is making sure all your data is properly loaded, making sure N/As are treated as a special value rather than a number between -1 and 1, and tagging your categorical values properly.

In the past I've done plenty of linear regression analysis. So feature engineering mainly concerned with:

Getting features into the correct scale using log, exponent, power transformations
Multiplying features: if you have height and width, multiply to make area
Selecting features: remove features based on P value

But, for LightGBM (and Random Forest) it seems like the scale of the features doesn't matter because orderable items are ordered and then randomly bisected. Interactions of features don't matter because one of the weak classifiers should find it if it is important. And feature selection isn't important because if the effect is weak then those classifiers will be attenuated.

So, assuming you can't find more data to bring in, what feature engineering should be done with decision tree models?

Feature engineering that I would consider essential for even tree based algorithms are:

Modular arithmetic calculations: e.g. converting a timestamp into day of the week, or time of day. If your model needs to know that something happens on the third Monday of every month, it will be nearly impossible to determine this from timestamps.
On a similar vein, creating new features from the data you have available can drastically improve your predictive power. This is where domain knowledge is extremely important - if you know of, or think you know of a relationship then you can include variables that describe that relationship. This is because tree based methods can only create splits that are horizontal or vertical (i.e. orthogonal to your data).
Dimension reduction is typically performed by either feature selection or feature transformation. Reducing the dimension through feature selection will likely not help much with the models you mention, but an algorithm may or may not benefit from feature transformation (for example principal component analysis) depending on how much information is lost in the process. The only way to know for sure is to explore whether feature transformation provides better performance.


Measuring performance of different classifiers without class type in data To measure the performance of a classification algorithms on a dataset that has an attribute for class type, I divide my dataset to training and test samples and then create a confusion matrix for False Positive, False Negative, True Positive and True Negative samples. Hence there is a class type attribute(e.gYes or No ), the confusion matrix is pretty easy to calculate.

Now suppose that I have a dataset that lacks a class type attribute and all samples are of the class Yes .

How can I measure the performance of different classification algorithms using these kind of datasets?

Normal classification will not work in this case, classifiers learn functions that are able to seperate the training examples. In your case you only have one class which will not work for all the classical examples. There is a machine learning task called One-class classification, see this Wikipedia page. In your case PU Learning (Positive and Unlabeled) seems most appropriate. Evaluating the performance is extremely difficult without having some negative examples as well however. See this question and answer on the Stats stackexchange.


Model threshold for classification model How do I extract a model score based on desired precision recall for a classification model? Is there a command to extract it from precision_recall function in sklearn?

p, r, thresholds = precision_recall_curve(y_test, y_scores)

For more clear understanding refer to this article:
https://towardsdatascience.com/fine-tuning-a-classifier-in-scikit-learn-66e048c21e65


Is it possible to change the input columns of a trained ML model while making predictions from it without affecting the accuracy? Consider the following scenario.
I have trained a K-Means model on some input features, say, (A, B, C, D and E).
Now at the time of making predictions I want to make the model predict using only fewer number of features e.g. (A, D, E) as opposed to the number of features it was trained initially i.e. (A, B, C, D and E).
With the understanding of above facts I have some questions which are as follows :-

Is this approach a correct approach, or logical with respect to machine learning principles ?
Will it affect the model accuracy and if yes then how ?
If I have to provide features B and C, then can I populate them with zeros and then provide it to trained model for making predictions.
Will the action taken in step 3) affect the model accuracy, if yes and then why and how ?

Is this approach a correct approach, or logical with respect to machine learning principles?

It will affect the performance of the model in the sense that your algorithm learned to separate the clusters based upon distance according to all the features. I have read discussions about how to calculate feature importance on unsupervised problems like yours, so you could do some research on that and figure out a way to measure the importance of your features so that you have an idea of how influential is a feature on your model and therefore the impact on removing one. In this case removing will imply filling the features you are not using with NaNs so your model must be prepared for such scenario (sklearn pipelines are the best way of doing this)

Will it affect the model accuracy and if yes then how?

First of all, you are referring to an unsupervised model (K-means) so the metrics you mention does not apply, rather there must be metrics on the separateness of the clusters you formed (silhouette score, etc) and according to my first answers, you may adapt a version of Permutation importance using a metric according to your problem to see how it impacts to remove a feature the general performance.

If I have to provide features B and C, then can I populate them with zeros and then provide it to trained model for making predictions

Remember you are using an algorithm that is based on euclidian distance so imputing with zeros may have an undesired result

Will the action taken in step 3) affect the model accuracy, if yes and then why and how?

Sure it will, imputing with zeros will take the features not present to the origin on the euclidean space, so be careful with that


Time series prediction without sliding window I've got a collection of historic log files for a process that repeats daily, roughly the same each day.
I want an NN that, from the moment that the daily process starts, estimates what time the whole process will finally complete.
As the daily process progresses, the estimate should become more accurate and converge on the actual completion time as soon as possible.
A degree of confidence in the prediction would be useful too.

The daily process is a collection of mostly serial subprocesses. Mostly serial because some subprocesses have no impact on the end time ( critical path vs non ).
There are other pattens hidden in the data. Weekends take longer than weekdays. In the real world, maybe others too.
There is a small amount of noise in the data, subprocess duration can be affected by completely external factors.
The performance of one given day has little / no expected correlation with the next day, although there may be long term trends.

I've had a little success with a simple RNN but results have been underwhelming, even after trying a lot of hyperparameters.
Best results so far have been from one-hot encoding and "cross joining" to expand each day into multiple samples and padding them to the same maximum dimensions...

Xsample1.1 = [Event1, ]              Y = EventN
Xsample1.2 = [Event1, Event2, ]      Y = EventN
Xsample1.N = [Event1, ..., EventN ]  Y = EventN

I've encoded weekday as one-hot variables and normalized datetimes to one universal day in preprocessing.

All the examples I find are for predicting the next N samples, sliding forward, not a fixed end sample.
I'm using ipython3, pandas and keras and I've generated a test population  using some simple logic I'd expect an NN to easily handle. YHat for each file is the last datetime. The .debug files capture how the prediction should evolve but that information is not available from the training data and so is not used in the NN.

Is this a time series or regression problem, what type of NN should I use and how should i structure the input data for best results.

Implementing this with a recurrent neural network is not that difficult. My suggestion is close to what you already suggested but instead of sampling different points in time I would just broadcast your target over all time steps. If there are a total of N time steps, and we want to predict on steps t=1 to t=N-1, our X is a matrix of shape [N-1, k] with k the number of features. Y is a sequence of shape [N-1] with all the values being equal to EventN.

This is similar to what you actually want to achieve, namely at every time step you want to predict the final value. Your loss function will be calculated over all time steps and backpropagated through time. It's possible that you might want to weigh your function as a function of the time step, maybe later predictions are more or less important, you'll need to do some analysis on this. Another recommendation would be to add some feature of time in your feature explicitly, that might help the network.


Is it cheating to stratified sample the whole dataset based on a previous evaluation result? I trained a model using a small mri dataset(57 patients).
The model's performance was so low(Train set 0.7, Val set 0.7, Test set 0.45).

I found the model segment tumor in upper part of brain well, couldn't segment tumor which is in middle part of brain.

So i stratified sampled the whole dataset with a position stratum(upper, not upper), the model performance was improved.

In this case, is it cheating to do that?

By having seen the results on the held-out test set, learning something and applying that to the model, you have essentially added some bias to your model. So if we are being strict, yes... You cheated a little bit  ;)

Without having a solid (meaning unbiased) validation metric, you cannot objectively compare the two results. You used prior information to make an improved modelling decision, which is great :)   However, that information flowed directly from your test data, which introduced the bias.

It can be difficult when you have limited data, I know. You could maybe look into data augmentation to synthetically increase your dataset. In your case, adding random rotations and mirrored images could help a great deal!


What does it mean to "warm-start" XGBoost? In the project I am currently working on (predicting whether or not someone will click on some item from the mailing list that I send), each day data about users is extracted and the models are learned from scratch. Since users from yesterday don't really differ that much from users today I decided to try to somehow use the old models.

For the Deep Net and SVM models it's easy - I just make the algorithms start with the weights that were obtained last time and make them work through the new dataset using gradient descent.

What about the XGBoost though? I see that I have two options to use previous data:

Make predictions using the old model and feed them as a parameter to the new model
Use the old booster

The first one looks like it will make the new model focus too much on the old results, whereas the second one also feels a bit weird: I built 1000 trees and now I boost them with new data - so here it looks like I will end up mostly disregarding the old data and just rebuilding the forest to match the new data.

My first thought was to simply use half of trees from the old model and half from the new one. This way I would still use old knowledge but balance it out with new data. After some time I would get something like a funny pyramid-shaped forest with very few "old" trees and very many "young" ones (another way to think about it is like a power series model where data from period t-1 is given half the importance that data from period t gets). The idea looks neat, but I haven't seen anyone do that anywhere, so I remain skeptical.

Are there any options that I missed? I would be grateful for both comments on the possibilities that I outlined above and new ideas on treating this problem.

Why not just keep training your old model?
just pass the previous xgb model (path + filename) as an argument in the train function.

bst = xgb.train(param0, dtrain, num_round, evals=[(dtrain, "training")], xgb_model=prev_xgb_model)

xgb_model=prev_xgb_model


(Pattern-Matching) what is the best measurement can be used to match drug names from different countries I need to compare the price of the local drug-list of my country with prices of drug-lists of other countries (CSV format), I have a manually-matched table whic is my target and I use python via Jupyter on windows.

In the following, there are two records of two different drug-lists

one drug in list 'A'

Name:   40MG ABSEAMED*

Form:   injectable solution

Dosage: 0.9%

matched drug in list 'B'

Name:   ABSEAMED SOLUTION FOR INJECTION 40MG/0.4ML

Form:   SOLUTION FOR INJECTION

Dosage: 25mg/ml

My simple process is to clean data from undesired symbols and spaces, then to count occurances of features.

Problems:

using simple distribution function (intersected words in both) / (words of list 'A') the name of drug (ABSEAMED) is considered as priority as other features ('FOR', 'SOLUTION', ...)
By removing the Form words from Name words, some names becomes null
after cleaning data, I get noisy features (25mg/ml ===> 25 mg ml) and this dropped the accuracy of matching from 56% to 24%

  What are the basic recommended techniques/tools (Automates, POS_tagger, ...) of Natural Language Processing that may give a high-accuracy results ?

I thought about creating my own POS tag to match tags before contents

(e.x. the drug name '0.225% W/V SODIUM CHLORIDE (1/4 NORMAL SALINE)

can be tagged as \dosage \symbol \name \quote*

but I'm not sure enough that it can improve my result

Any help is appreciated.

One of the workable approaches:
If you have sufficient data, and, you need to so this often, you can train your own custom NER model. Write a pipeline to match drugs and compare price and deliver a small app that solves the problem once and for all!
Custom Named Entity Recognition models helps you identify a named entity from a given chunk of text. There are plenty of models available that provide out of the box support of healthcare related named entities, like, drug!
Once you have extracted named entities like drug, dosage etc. (btw, dosage can also be solved by writing a regex), you can compare the cosine similarities to find the right match. You can try different word embeddings to have more accurate match.


Resolving time in NLP "I want to go swimming next Tuesday"
I want to machine to learn the date I want to go swimming. Is there any approaches or libraries that can

Extract "next Tuesday"
And calculate the exact date?

There is a Python library called dateparser that will accept a wide variety of formats, including relative dates like "next Tuesday", and return exact datetime representation.


Training set, validation set, and test set with Orange Is it possible with Orange (only using its widgets, without writing Python code) to implement the following typical machine learning processes?

Train a training set,
Validating a validation set (e.g. using k-fold cross validation), and
Testing the model with a test set.

Using the 'Test & Score" widget, a single dataset is used. Does this mean that only validation is performed, and there is no actual test? Can in this case classification results (accuracy, AUC, etc.) considered reliable?

The answer is yes. If Test & Score is given only one data set, then all it can do is show results of cross-validation.

To test the models on a separate data set, use separate File widgets to load training and test data. Connect File widget with training data to Test & Score, and the connect File widget with Test data to Test & Score. The connect whatever learner to test (in the workflow on the figure I've used logistic regression and random forest). Make sure "Test on train data" is selected in the Test & Score.


PyTorch vs. Tensorflow eager Google recently included in tensorflow's nightly builds its Eager mode, an imperative API to access tensorflow computation capabilities.

How do tensorflow eager compare to PyTorch?

Some aspects that could affect the comparison could be:

Advantages and disadvantages of eager due to its static graph legacy (e.g. names in nodes).
Intrinsic limitations of either of them that the other does not have.
Areas in which one of them needs improvement (e.g. feature completeness, computational optimizations).
Ecosystem differences (e.g. tensorboard?).

Note1: Yaroslav Bulatov wrote a review about eager's nice features.

Note2: In a previous question, I requested a comparison between PyTorch and Tensorflow Fold. At that time, it seemed to me that Fold could face PyTorch thanks to Google backing it. I was very very wrong: in the end, Google itself abandoned Fold in favour of Eager. I understand that this was due to intrinsic limitations in the normal tensorflow API that led Fold not to be very friendly, which constrained its adoption.

One the key advantages that is I use a lot is that is compatible with pdb, so that

pdb.set_trace # To the rescue

Permits the use of python data structures

and let us to use pythonic control flow instead of using the main tf equivalents.

Also it allows to avoid metaprogramming issues such as "lazy loading" and adding a bunch of operations to my graph. Also autograd similarities


How to input a 3d model into ML algorithm? I have a machine learning model that uses csv with measured data about buildings: width, length, height etc. I use it to predict some features and it works properly.

I would like to drop csv with length, height and width, and I would like to use some kind of algorithm to parse 3d model into the ML algorithm.

The second reason is to try this approach with nonrectangular buildings, which are hard to describe in simple csv generated by humans.

I do not necessarily need the algorithm to read the width, length etc. but to be able to predict some values based on training set of 3d models acompanied by csv with results.

What should I look into? Where can I find information about this approach?

Convolutional Neural Networks can work with 3 dimensional data, and they don't require much feature engineering. Provided you have enough training data, that seems like a way to go. As long as your data is in the form of a grid it is actually easy-peasy. Just imagine an image-like pipeline, but of the form [x,y,z,n]. Where n is the number of examples. As opposed to [x,y,n] (black and white) or [x,y,c,n] (color-channels).

It might be the case however that your data is in a vector format (as tends to happen with building models). I do believe there are experiments with straight up graphs and deep learning, but my feeling is that you might be better rasterising your model in that case.

You will probably need to add some layers to compress the data concepts (conv and pools etc) and end with a fully connected layer of 1 node (since you are looking for a scalar).


How many minima does the residual sum of squares have for the logistic curve? Suppose we had some data points $\{(x_i,y_i)\}$ where $x_i$ is a real number and $y_i$ is zero or one. We want to fit the logistic function to the data where the logistic function is $\hat{y}=\frac{1}{1+e^{-(\beta x +\beta_0)}}$. To do this we would like to choose $\beta_1$ and $\beta_0$ in order to minimize the residual sum of squares, $RSS = \sum_i (y_i-\hat{y}_i)^2$.[nevermind that cross-entropy is a better cost function.] I believe there is only a single minimum. How can I know? Does anyone know of a proof for this or a counter example?

Sigmoid itself is not a convex function (see this) and square loss based on sigmoid, such as $ \left( A - \frac{1}{1 + e^{-z}} \right)^2 $ is not convex.

Simply plotting the square loss function with A = 5 shows that it's convex for x > 0 and concave, otherwise

If you want mathematical proof, take the second derivative and you'll see that it not strictly positive nor negative.


Why does using Gradient descent over Stochatic gradient descent improve performance? Currently, I'm running two types of logistic regression.

logistic regression with SGD
logistic regression with GD

implemented as follows
SGD= SGDClassifier(loss="log",max_iter=1000,penalty='l1',alpha=0.001)
logreg = LogisticRegression(solver='liblinear', max_iter=100, penalty='l1', C=0.1)

nevermind the hyperparameters as I've used GridsearchCV and tried multiple combinations.
When calculating accuracy logistic with GD performs better than SGD. I want to understand why this is the case, is using GD instead SGD one way to mitigate underfitting model?

SGD has a regularization effect and finds the solution faster. GD on the other hand takes a look at whole data and finds the next best step.
SGD may be come to optimal global minima but GD can. But GD is not practical with large data.


Risk score from Neural Network classifier (more than 2 categories) I am trying to use a Neural Network to perform multiclass classification. The classes represent Insurance Risk Level. The most risky level is Level 1, the least risk corresponds to Level 10. The labels came from Unsupervised clustering unannotated insurance data. The architecture is:

Input Layer 43 features

Dense, 1000 neurons, ReLu activation

Dense 500 neurons,  ReLu activation

Output 10 neurons, Softmax activation

In particular, for binary classification we can use single neuron as last layer. Then, we define the score to be the input value to this final layer. But for more than 2 classes I don't know what to do.

  My question: Is there a way to obtain a Risk score that is consistent, that is, the greater the Level is, the smaller the score is  ?  I tried to use logits, not normalized probabilites, etc. But not sure if there is a way to get a single consistent score from logits ? . Thanks for any hint on this.

How about doing in an easier way?

Perform the classification in normal way, the greater the score the greater the level. Then, for the final output, just reverse the level.

For example, from the classification, the lowest score comes to level 1 and the highest score comes to level 10. Then, make level 1 as level 10 and level 10 as level 1 for the final output