how to build word2vec content based recommendation? I am building a content-based recommendation system for hotel accommodation. I have a hotel name, hotel description and location. I combined hotel name, description and location. Then, applied NLP and converted into vectors using TF-IDF.

I am taking hotel name and location as an input and recommend the hotel based on cosine similarity. I wanted to use word2vec instead of TF-IDF and make a recommendation. Anyone can suggest how to do this?

You can find a number of blogs teaching the use of Word2Vec,

Introduction to Word Embedding and Word2Vec
A Beginner's Guide to Word2Vec and Neural Word Embeddings

If you are building a hotel recommendation system, I think the name of the hotel is not a good feature. Users are less concerned with the name of the hotel than its location or description provided.

You can take a small description from the user and the location in which they would like to search for the hotel. Using Doc2Vec, convert the description ( text ) into a fixed-sized vector ( embedding ). Find a hotel whose description is the most similar to the one provided by the user ( maybe using Cosine Similarity or any other measures ).
Before this find a hotel that is close to the location provided by the user ( say a hotel which lies in a radius of 2 km from the given location ).

Using these two factors, you can prepare a list of hotels whose location and description match the user's needs.


How can both generator and discriminator losses decrease? In this paper there is a plot of how the loss of gans looks through epochs.

Figure 2:

These are of course averaged losses.

How can both the discriminator loss and generator loss decrease?

real_y = discriminator(real_sample)
fake_y = discriminator(generator(noise))
discriminator_loss = real_y-fake_y+1 # nicer, it is between 0 and 2

fake_y = discriminator(generator(noise))
generator_loss = fake_y

I would expect one of the losses to increase as the other one decreases. Since they use the same calculation of fake_y and one decreases -fake_y and the other fake_y. One optimizer is making fake_y less and the other optimizer is making it more.

Maybe the loss functions aren't calculated like I said.

In the widely used analogy:

  In simple terms the generator is like a forger trying to produce some counterfeit material, and the discriminator is like the police trying to detect the forged items.

We can measure how good the police is by how many times out of 100 fakes he can identify the real and fake ones and the forger as deceiving the policeman out of 100 fakes.

Wouldn't that mean that if the police get better, the forger gets worse? (if using the aforementioned measure of how good they are)

Therefore we wouldn't be able to see both of them being good at the same time! But the graph from the paper indicates otherwise.

What am I missing?

In the subsection of the figure, they compute the average loss across 100 iterations, which is why the loss is monotonically decreasing because on average the loss does decrease with the training.
You are correct in inferring that if this was reported on an iteration to iteration basis, the loss would be a zig zag curve, which is less nice to look at than a smooth curve.
As in the comments, the loss can monotonically decrease to 0, if it is the Wasserstein distance. The Wasserstein distance is computed as $\sup_{f \in L_{k}} E_{x \sim P_X}[f] - E_{x \sim P_{\theta}} [f]$ where the first term is the expectation of the function, $f$ (discriminator) over the batch and the second term is the generated data from the generator computed on the discriminator. Note that by this formulation, it returns the $k$-Wasserstein distance.


Anomaly Detection/Novelty detection I have a data-set that has over 6 million normal data and around 50 anomaly data. Those anomaly data is identified manually (by monitoring the user`s activity over camera and identify). I need to develop a model to detect these anomalies.
My problem is that the anomaly data looks like normal data, which means they are not outliers or has a certain pattern. If I plot the normal data over anomaly data they are in the same distribution.
I tried several anomaly detection approaches:

Multivariate Gaussian Distribution Approach to identify anomalies

I tried to create new features that anomaly data will be outliers and then I can use Multivariate Gaussian Distribution Approach, but could not able to find any combination to isolate the anomalies.

I guess there is no point of using a classification algorithm since dataset is highly imbalanced.

I tried OneClassSVM, DecisionTree, RandomForest but AUC is 0.5 (as good as random).

How to implement a model for this kind of scenario?
Other methods which I can think about:

Develop a NN with AutoEncoders
Try Generate Synthetic Samples and resample the dataset

I believe you can use a classification algorithm where you manually overrepresent the "anomalies" class. By how much, depends on the cost induced by the anomalies.

Just to illustrate what I mean:

Anomalies cover a continuum between two extremes: 1) those which can be safely ignored, because they induce no costs, and 2) those whose costs is unbearable. As an example close to the extreme (1), imagine a flower shop getting an order for Iris virginica. The customer will hardly object if, among 6 million flowers shipped, 50 actually belong to the species Iris versicolor.

On the other hand, if you are shipping airplane parts, and a faulty part would cause a disaster with 300+ dead, then you are closer to the extreme (2). You will likely want to avoid shipping even one faulty part.

In practice, it will have to be a trade-off: You can be on the safe side if you ship no parts at all, but then you go bankrupt. On the other hand, if you do no quality assurance at all, you ship enough faulty parts to cause a disaster in a foreseeable future, have to pay damages, and go bankrupt again.


How to make tool that's robust to user-generated typos? Background: A client is generating datasets (excel files). They asked I make an app to analyze the datasets, e.g. summary tables and figures. I'm doing this in R shiny.

Problem: There are many typos in the user-generated data. For example, Alice is sometimes entered as Alce, alice, alice., Aalice, or Jennifer (middle name). Some errors are easy to correct for, like trailing whitespace and upper/lowercase. Others are nearly impossible, like knowing Jennifer (middle name) is really Alice.

How can I make my tool robust to errors in the data when I don't have control over data entry?

TL;DR

Use fuzzy string matching to account for spelling mistakes.
Solving Jennifer for Alice will require you to know where to look in your DB for these cases, or talk to make a better Excel file to force people to only input first names (e.g. make a an entry to a cell restricted to a given list).

Fuzzy String Matching

In R, you can use adist or the stringdist package. These can be used to measure the distance from an entry (e.g. Aalice to a list of potential matches [Alice, Bianca, Chris].

Here is an articles explaining how to use both.

An extract from the article:

source1.devices<-read.csv('[path_to_your_source1.csv]')
source2.devices<-read.csv('[path_to_your_source2.csv]')
# To make sure we are dealing with charts
source1.devices$name<-as.character(source1.devices$name)
source2.devices$name<-as.character(source2.devices$name)

# It creates a matrix with the Standard Levenshtein distance between the name fields of both sources
dist.name<-adist(source1.devices$name,source2.devices$name, partial = TRUE, ignore.case = TRUE)

# We now take the pairs with the minimum distance
min.name<-apply(dist.name, 1, min)

match.s1.s2<-NULL
for(i in 1:nrow(dist.name))
{
    s2.i<-match(min.name[i],dist.name[i,])
    s1.i<-i
    match.s1.s2<-rbind(data.frame(s2.i=s2.i,s1.i=s1.i,s2name=source2.devices[s2.i,]$name, s1name=source1.devices[s1.i,]$name, adist=min.name[i]),match.s1.s2)
}
# and we then can have a look at the results
View(match.s1.s2)

All this assumes that you have a list of names that are actually valid.


How to utilize measurement accuracy metadata in classifier Given that one wants to ascribe a class to groups of measurements using a classifier model, in what way can one include information about measurement accuracy?
More specifically, is there a feature engineering strategy for including information with the intent to allow the model to weight a subset of the sample more than other subsets?
As an example, say I have have coordinate and accuracy information (lat, lon, measurement_accuracy) in a time-series. I want to make a classifier which takes a group of 2 coordinate pairs, and output a class from this group. Each sample therefore consists of 2 coordinate pairs.
The measurement accuracy could be used to indicate to the model which of these measurements should be trusted most, but does not by itself give any hint towards the class the sample belongs to.
given the 3 samples
[ # learn to trust the first coordinate set more in making a prediction
    {'lat': 11, 'lon': 40, 'acc': 5},
    {'lat': 9, 'lon': 40, 'acc': 2}
],
[ # learn to trust the second coordinate set more in making a prediction
    {'lat': 74, 'lon': 131, 'acc': 1},
    {'lat': 78, 'lon': 140, 'acc': 4}
],
[ # learn to treat these coordinates approx. equally
    {'lat': 74, 'lon': 131, 'acc': 1},
    {'lat': 78, 'lon': 140, 'acc': 1}
]

and assuming acc is an indication of the coordinate pair measurement accuracy I would like the model to trust the first coordinate pair more in the first sample, trust the second one more in the second sample, and in the third sample trust both coordinate pairs equally.
What kind of a representation of the data could enforce this logic, given that the data will be fed into the model in a tabular way?

You could use a multi label classifier so that your model could use the extra feature to decrease its confidence score. If there is indeed a correlation with inaccuracy of the measurement and "acc" it should be able to account for that in its prediction through its probability output


Which deep learning framework have support for gtx580 GPU? I would like to train convolutional neural networks using a gtx580 gpu. I tried setting up TensorFlow but it did not work (wrong cuda compute compability).

Which deep learning framework can best support a gtx580 gpu in terms of functionality and speed?

Unfortunately you can't use GTX 580+Tensorflow.

TensorFlow GPU support requires having a GPU card with NVidia Compute Capability >= 3.0, while GTX 580 has only 2.0 compute capability.

However, caffe supports CUDA compute capability 2.0. You can have a try.

References

https://www.tensorflow.org/versions/r0.10/get_started/os_setup#prepare_environment_for_linux

https://developer.nvidia.com/cuda-gpus


General way to reduce features Let's say I have a giant dataset (600+ columns) and I have no idea what insights I might get from it or what model I want to run.

What are some of the best ways to find the most influential columns/features? I know it's important to know what models to run first, but is there a general way to find columns that's more important than others?

So far I've already removed the columns that are 80%+ empty, but I still have over 600 columns and I have no idea where to start. Please let me know your thoughts! I'm kind of new so any pointers will help.

Another way of looking at 600 columns is having a vector with 600 dimensions. So, the answer to reduce the 600 dimensional vectors to fewer dimensions without losing lot of information will be dimensionality reduction.

Principal component analysis(PCA) is a commonly used one. T-distributed stochastic neighbour embedding (TSNE) has been the state of the art for reducing higher dimensional vectors. Data visualisation using TSNE

To get the quick intuition about the vectors, the best thing is to reduce it to 2 or 3 dimensional vectors and plot them in a graph. Data visualisation always helps to get an overview of the data you are dealing with.


Classification task - how to handle large data file? I'm planning to construct a classification model for predicting New York taxi trip fare. The CSV-datafile for this is very large, containing 112 234 626 rows (ca 10 GB). I have managed to download this file to my computer's local disc but
the further memory problems arise when I'm starting to use this file for parsing and model construction.

So, my questions are:

What is the optimal number of rows that I should take from this datafile for use in my task?
How should I take these rows to prevent low memory problems?
How should I take these rows so that it contains optimal sample of all the months. The file contains data of year 2018 and if I take the rows so that it contains only pre-specified number of rows then for example the dataframe may contain only data for January (but I want all months).

2018 Yellow Taxi Trip Data

With pandas:

import pandas as pd

dataset = pd.read_csv(path, chunksize=1000) #number of rows per chunk

for chunk in dataset:
    #process chunk etc


why does my calibration curve for platts and isotonic have less points than my uncalibrated model? i train a model using grid search then i use the best parameters from this to define my chosen model.
model = XGBClassifier()
pipeline = make_pipeline(model)

kfolds = StratifiedKFold(3)
clf = GridSearchCV(pipeline, parameters, cv=kfolds.split(x_train, y_train),
                           scoring='roc_auc', return_train_score=True)

clf.fit(x, y)

model = clf.best_estimator_

using this model from gridsearch i then calibrate it and plot uncalibrated vs calibrated..
y_test_uncalibrated = model.predict_proba(x_test)[:, 1]
fraction_of_positives, mean_predicted_value=calibration_curve(y_test,y_test_uncalibrated,n_bins=10)

plt.plot(mean_predicted_value, fraction_of_positives, 's-', label='Uncalibrated')

clf_isotonic = CalibratedClassifierCV(model, cv='prefit', method='isotonic')
clf_isotonic.fit(x_train, y_train)
y_test_iso = clf_isotonic.predict_proba(x_test)[:, 1]
fraction_of_positives, mean_predicted_value = calibration_curve(y_test, y_test_iso, n_bins=10)

plt.plot(mean_predicted_value, fraction_of_positives, 's-', color='red', label='Calibrated (Isotonic)')

i do the same for platts above.. however i get the following results:

i don't understand why there are less points now for isotonic and platts? i don't feel as though i am doing anything wrong in my code. am i making any mistakes?

The default strategy for calibration_curve is 'uniform', i.e. each of the bins has equal width.  If, after calibration, your model makes no predictions inside a bin, there will be no point plotted for that range.
You could change to strategy='quantile', which would guarantee 10 points plotted for each curve; you'll get many more of the red/yellow dots further to the left.

Also, ideally you should not be fitting the calibration on the same data that you trained the original model.  From the User Guide,

An already fitted classifier can be calibrated by setting cv="prefit". In this case, the data is only used to fit the regressor. It is up to the user make sure that the data used for fitting the classifier is disjoint from the data used for fitting the regressor.


Dense Layer vs convolutional layer - when to use them and how I know what's the difference between the two, however I am a little confused on to use them. I have also seen some models that have a mix of both. what's the logic behind it? or is it only random things?

As known, the main difference between the Convolutional layer and the Dense layer is that Convolutional Layer uses fewer parameters by forcing input values to share the parameters. The Dense Layer uses a linear operation meaning every output is formed by the function based on every input. In other words, we "force" every input to the function and let the NN learn its relation to the output. As a result, there appear n*m connections (or weights) where n denotes the number of inputs and m denotes the number of outputs. On the other hand, the Convolutional layer uses a filter to operate the convolution operation which has a small size most of the time. An output of the convolution layers is formed by just a small size of inputs which depends on the filter's size and the weights are shared for all the pixels. That is, the output is constructed by using the same coefficients for all pixels by using the neighboring pixels as an input.
In the convolutional layer, as known, the filter's center is located in the pixel and then the linear operation is processed using the neighboring pixels of that pixel. That is, we, in advance, know that there is a strong relationship between the neighboring pixels. If we would not be sure that the neighboring pixels have a strong relationship with neighboring pixels it would not be logical to use Convolutional Layer, instead we would force all the pixels (inputs) to the function to learn the relationship by using Dense Layer. So, by using the Convolutional Layer we assume that the neighboring pixels are the main representative of the center pixel and as we go far away from the pixel, that is, the pixels that are away from the center pixel do not really possess the same characteristics as the center pixel. They might even be a different object thus may lead to a spurious result or would cause your function to learn redundant information which in reality has no relationship with.
In short, since we have prior knowledge about our data and information in it, we not only take the heavy load from our model by using a convolutional layer but also show it the exact location of the data that might be useful for it to learn while keeping it away from redundant data. However, frequently we use both in the same model, it is often simply due to the fact that we do not know what is going on in layer 10 (for example). Namely, we do not have that prior information about data anymore, because we do not know what it learns in those deep layers. Thus, we use Dense Layer, by giving all the inputs, we give the "full responsibility" to learn. In other words, we say to a Dense Layer that "Here are my features (pixels maybe), I don't know the true relationship between them, please find it yourself". We say to a Convolutional Layer that "Here are my features (pixels maybe), I am sure that it is enough just to look the few pixels around the center pixel" and that relationship is preserved through all the pixels.
So knowing, how it works, in different contexts, you can apply them to different scenarios. If you have a good knowledge of your data and its structure then you can choose the relevant layer while designing your model.


Significant testing - repeated observation over multiple days I work in mobile gaming, and want to analyze A/B test groups, but I believe I'm introducing errors in my calculations.

The metric I'm looking at is:
num of unique players who engaged in battle that day/ num of unique players who were active that day.

I currently have my data that with each row as active players for the day:
date, player_id, group A/B, boolean 0,1 if engaged in battle that day
I split the groups A/B and take the array of booleans and do a t-test.
I feel that I'm overstating the sample size.

For example, I might have 1,000 people in Group A, and each day 300 are active from that group, of those 300 most of them are the same 200 people who are active everyday.
I'm feeding 300 active players x 7 days = 2,100 rows of data into a t-test, but most of this is data from the same 200 people.

What is the correct way to do this?

Thank you!

It depends on what you are testing and what your assumptions are.
Here are couple of scenarios:

If you are looking at single day, then it does not matter.

If you are comparing multiple days but looking at each day independently, they you might want to control for the multiple comparisons problem.

If you are looking for changes over days, then you might want to treat it as a time series problem.


Dataframe looks the same but the structure is different when loop I am generating a dataframe from a JSON file, this JSON file can come from 2 different sources, so the internal structure is slightly different, so what I am doing is first detecting the source and from there I do a set of operations that gives me a Dataframe

Everything is good until here (I thought), as when I print it in jupyter it shows me the way I wanted they look the same (structure), the problem goes when I loop through them,

I get completely different results (this df have each same number of columns, 7 columns)

When I loop:

In 1 I have only 2 columns in the other one I get all the columns.

I am looping:

for i, (index, row) in enumerate(df_trans.iterrows()):
    print(row)

Is there a way to see how is structure, I am quite confused of why the print of the df loops the same but when looping is not

EDIT

I notice that when I print the dataframe after a grouping I get the followin

df_summary_trans_cs.groupby(['Date'])['sale', 'refund','Balance'].agg('sum')

I get all the columns

but when I add the column

df_summary_trans_cs.groupby(['Date'])['sale', 'refund','Balance', 'Trans'].agg('sum')

I only get that column, the other 3 dissapears

I finally got what was wrong, it has to do with the type in the series, so when i printed the whole dataframe the values where there, but the grouping was not been apply as it seems they were not number

I apply

df.MissingColumn = df.MissingColumn.astype(np.float64)

And this fixed the issue


why we need data labelling tool for computer vision? Before start training images with tensorflow object detection api we need to use labelling tool to annotate our images and converted to XML format.
What happens when we convert our annoted image to XML file? Why we need that?

You need to label data in order to train classifier. Labelled data are at the basis of all supervised ML. XML files are just one of the many format you can use to store information, people usually use them since they can be easily read by many tools.


Why I am having ValueError in this Linear Regression? from sklearn.linear_model import LinearRegression

ClosePrices = data['Close'].tolist()
OpenPrices = data['Open'].tolist()

OpenPrices = np.reshape(OpenPrices, (len(OpenPrices), 1))
ClosePrices = np.reshape(ClosePrices, (len(ClosePrices), 1))

regressor = LinearRegression()
regressor.fit(OpenPrices, ClosePrices)

I am having the error

ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

What is the solution?

There are most likely missing values in your data, those missing values are encoded as NaN.

The most common solutions for missing data are:

Drop the instances/rows that have any missing values.
Impute the missing values.


Predict Best Branch Locations For a Company I have a dataset of existing 156 branches of my company with the longitude and latitude of each branch. Now we want to open 10 more branches. How can we predict the best locations for opening the new branches using machine learning? As much as I have searched on google, I have gone through by using ARCGIS geo-spatial data or using openstreetmap but couldn't find all the steps properly. Therefore, Kindly guide me as I am a junior data scientist in my company and its my task to deliver the best locations.

This is a broad question, what I offer is one of many possible approaches; I will use R workflow (sf package to be exact).

The first step is to generate a grid covering your area of interest - city, country, continent, whatever :) Consider the approach described in this answer on StackOverflow. The size of the grid cells will depend on your company product - the catchment area of a corner shop is different from a catchment area of a strip mall.

The next step is link your store locations to grid cells - sf::st_intersects will be your friend here, and aggregate your desired metric (sales volume, new customer acquisition, or what not) per grid cell.

The third step is to find as much demographic and economic data as possible for administrative regions of your country (counties, districts, census areas...) - this will be country specific, so I cannot be your guide you here - and again link them to your grid.

The last step is creating a regression for desired output in a grid cell depending on the economic and demographic indicators.

Once you have a formula you can apply it to all the empty grid cells and pick the 10 most promising.

Edit [2019-01-22]: I have extended the suggested workflow into a blog post, you can view it at
http://www.jla-data.net/eng/branch-network-optimization-in-r/


can I use z-score normalization even if it doesn't make sense for my data to be negative? I'm planning to use z-score as a Normalization Method for a Project but I noticed if I do that then I ll have a data in the range [-1, 1] which is wierd because I have data that doesn't make sense for it to have negative values. Let's say for example speed or distance, it doesn't make sense that speed would have a negative value after normalization! is it logical to think like this or am I wrong and it is perfectly fine to use z-score even if the data will be negative and it doesn't make sense?

ps: I know velocity can be negative if we are talking about Vectors but I meant to say that I have a discrete Values for Speed or Distance a.k.a Length of something which cannot be negative.

I think it doesn't really matter that there can be negative values after as long as you rescale your data correctly at the end.

Think about what positive/negative values mean for z-score. This has nothing to do with whether your use case (for example speed) can realistically have negative values or not. With z-score positive values simply mean that the value is above the group mean while negative values can be interpreted as the opposite.

As long as you are able to rescale after your model (to get your data back to the original interpretation) using z-score should be fine.


Meaning of axes in Linear Discrimination Analysis Looking at the LDA of the Iris Dataset, which looks like this:

It's understandable that the 3 types of flowers have succesfuly been seperated into categories, with versicolor and virginica slightly overlapping. My question is, how does this help us in a practical way? What can we do with the information, now that we've seperated the 3 categories like this. It's great to know that Setosa is apparent for values between 5 and 10, but what do those values mean?

You could be asking yourself the same questions using PCA! Usually these linear transformations (or even non-linear for that matter, PCA, LDA, ICA, UMAP etc.) are used to reduce the high dimensional features space to serve one or multiple purposes. For example in PCA one could use it to compress the high-dimensional feature space to lower (usually 2 or 3) by finding the directions of maximal variance, either for better separation or a simple compression or better explainability via projected visualization for human eyes. Sometimes they are served for better class separation like the case of LDA, in a supervised fashion (in contrast to PCA). In case you missed it check out Sebastian Raschka's post on the difference between LDA and PCA for dimensionality reduction (be careful with LDA assumptions).

In order to answer the question about the axes, we better touch briefly on what LDA is aiming to do, borrowing the explanation from this blog post:

  LDA aims to find a new feature space to project the data in order to
  maximize classes separability. The first step is to find a way to
  measure this capacity of separation of each new feature space
  candidate. In 1988, a statistician called Ronald Fisher proposed the
  following solution: Maximize the function that represents the
  difference between the means, normalized by a measure of the
  within-class variability. The Fisher’s propose is basically to
  maximize the distance between the mean of each class and minimize the
  spreading within the class itself. Thus, we come up with two measures:
  the within-class and the between-class.

[The Answer]: As in the case of PCA: the first axis (by convention PCA 1) represents the axis holding the maximum variance in the data, PCA 2 the second in the rank and so forth. The same analogy holds true to LDA. Here though, LDA 1 (your x-axis in the graph, again by convention and the visual judgement) accounts for the maximum variation among the classes, and LDA 2 the second and so forth (still confused, you might one to check this video). You can define a simple linear model to separate them, that is it, that perhaps you couldn't using the original features.

[Bonus]: Although I am finished with the answer, and the following it is not super relevant, but I would like to bring a super nice example what these techniques might be able to answer from data. In the case of, again PCA, Tableau team is showing such capabilities, see this video. In real world problems, not a simple IRIS, these will come handy, and surely these possible even outside Tableau, but requires more efforts.


Statsmodel logit with sample weights Using sklearn I can consider sample weights in my model, like this:
from sklearn.linear_model import LogisticRegression
logreg = LogisticRegression(solver='liblinear')
logreg.fit(X_train, y_train, sample_weight=w_train)

Is there some clever way to consider sample weights also in the Logit method of statsmodel.api?
import statsmodels.api as sm
logit = sm.Logit(y, X)

It seems that there is a way of using sample weights which requires a little more work than just using a single argument, see this stackoverflow answer.


Re-sampling of a Histograms Bins I would like to be able to resample a histograms bins without having access tot he raw data.  And just to be clear, by resample, I mean to change the number of bins and still provide a good estimate of the original probabilities of those bins.

I can think of many ways to do this, but having trouble figuring out which is the best method which maintains the same probability in the resulting histogram.  The easy one would be if the input histogram X had x bins and the desired result histogram Y has y bins where x = y.  This is a simple 1 - 1 sampling of the original bins.  The problem forms to me as I decrease y lower than x or increase it above x.

For example: If x = 10 bins and y = 20 bins, it seems like you could simply double each of x's bins so you have Y = { x1, x1, x2, x2, x3, x3, ..., x10, x10 } but this seems like a naive approach as it seems like the 2nd copy of the previous bin should be influenced also by the next bin value  y2 = (x1 + x2)/2 for example.

If x = 20 bins and y = 7 bins I can see it isn't fair to simply sample a value based upon a linear interpolation between data point as there might be 3 or 4 points on either side of the sample that should be a part of the probability for the resampled data.

I would also like to consider the possibility that the histogram is contained on the ends, so in the case of measuring water temperature below freezing isn't a likely temp for water nor above boiling for the standard cases.  I would like to be able to consider the probability beyond one or both extreme bins to be 0.

Is there a standard algorithm which can be coded in C++/C# or something in pseudo code that I can convert to code for the above re-sampling / re-sizing?

You may use Interpolation on the the bins and heights of the HistogramList to produce a smooth PDF for use in ProbabilityDistribution.  With a smooth distribution then you need not worry about increasing or decreasing bin sizes.
For example, with data
SeedRandom[9123]
data = RandomVariate[WeibullDistribution[2, 2], 10^6];

has Histogram
hist = Histogram[data, Automatic, "PDF", PlotRange -> Full]

You have the histogram bins and heights so I'll just use HistogramList to collect them and display with ListStepPlot.
hlData = HistogramList[data, Automatic, "PDF"];
lstep = ListStepPlot[Transpose@{First@hlData, Append[Last@hlData, 0]}, Mesh -> Full]

A smooth InterpolatingFunction can be constructed from the bins and heights with Interpolation.
ifData = Interpolation[Transpose@{First@hlData, Append[Last@hlData, 0]}];
ifPlot = Plot[ifData[x], {x, Sequence @@ First@ifData["Domain"]}, PlotStyle -> Purple]

It can be seen that it matches the bins and heights.
Show[hist, ifPlot, lstep]

We want to use this function as the PDF of a ProbabilityDistribution so that NProbability can be calculated and RandomVariates can be generated.
The function needs to be non-negative over its domain.
NMinimize[ifData[x], {x} ∈ Interval @@ ifData["Domain"]]

{0., {x -> 7.4}}

Minimum value is zero at x equal to 7.4.
Must Integrate to 1 over its domain.
NIntegrate[ifData[x], {x} ∈ Interval @@ ifData["Domain"]] // N

0.996821

The integral is just shy of 1 but we can ask ProbabilityDistribution to "Normalize" the PDF.
dist = ProbabilityDistribution[ifData[x], {x, Sequence @@ First@ifData["Domain"]},
   Method -> "Normalize"];

This distribution can be used in RandomVariate, Probability and other functions of the Random Variables guide.
Probabilities can be calculated
NProbability[0.9 < x < 2.2, x \[Distributed] dist]

0.514336

Pseudo-random numbers generated
RandomVariate[dist, 5]

{0.494373, 1.16545, 2.94366, 4.06116, 1.72519}

Properties like the SurvivalFunction, CDF, and others can be calculated.
Plot[SurvivalFunction[dist, x], {x, Sequence @@ First@ifData["Domain"]},
 PlotStyle -> Purple]

Update
Widening the bins will result in a loss of information so I would not recommend it.  However, if you must then Subdivide the InterpolatingFunction "Domain" into the number of bins you required.
downsamples =
  Function[numbins,
    With[{bins = Subdivide[##, numbins] & @@ First@ifData["Domain"]},
     {
      bins,
      NProbability[#1 < x < #2, x \[Distributed] dist] & @@@
       Partition[bins, 2, 1]
      }]
    ] /@ Range[36, 12, -6];

Addressing your question in the comment; The total probability in each of the 5 cases is still one.
Total[downsamples[[All, 2]], {2}]

{1., 1., 1., 1., 1.}

However, notice the differences in the PDF histograms.
ListStepPlot[
 Transpose[{First@#, Append[Last@#, 0]}] & /@ downsamples,
 Mesh -> Full,
 PlotRange -> All,
 PlotLegends -> StringTemplate["`` bins"] /@ Range[36, 12, -6]]

The initial histogram had 37 bins. The plot above shows how the information is lost as the bin width is widened (number of bins decreased).  I would recommend working with dist instead.
Hope this helps.


Some robust perhaps more complex as neural networks to measure the distance between data? Now after certain preprocess we can repeatedly get two sets of data points, exhaustively enumerate one from the first set and one from the second set to form pairs. Each pair of these two data points have the same dimensionality, say two dimensional or one dimensional. The target is to measure the top n nearest ones among the distances of these pairs each time and perhaps finally the average of them. Euclidean distance is the simplest way to measure, but is there any more robust perhaps more complex way even applying neural networks to measure the distance between the pair? For instance, add a linear neural network layer after each data point, and measure the distance between the two outputs of the layer.

It all depends how you frame your problem.

Whats complex? more parameters, more time complexity, more exotic looking etc...

Here are 2 cool ideas how to measure similarity:

Mahalanobis distance measure of the distance between a point P and a distribution D. Cool but more importantly its robust because you are using parameters of your distribution to measure the distance.

Neural networks. If we relax the definition of "measuring distance between data" then Check siamesse networks. The gist of it is, can you group similiar sequences of data together. Why I like it? cause of one-shot learning


Should I use keras or sklearn for PCA? Recentl I saw that there is some basic overlapping of functionality between keras and sklearn regarding data preprocessing.
So I am a bit confused that whether should I introduce a dependency on another library like sklearn for basic data preprocessing or should I stick with only keras as I am using keras for building my models.
I would like to know the difference for scenarios like

which is good for production
which will give me better and faster response
is there any issue with introducing dependency over other libraries for just 1 or 2 functionality
which has a better compatibility with other tools like tensorboard or libraries like matplotlib, seaborn etc.

Thanks in advance.

What I would suggest is to build a sklearn pipeline in which one step will be the sklearn PCA and the last step will be your Keras model.
Sklearn pipelines are easy to put into production and can handle a lot more of transformations.


I got the following error : 'DataFrame' object has no attribute 'data' I am trying to get the 'data' and the 'target' of the iris setosa database, but I can't. For example, when I load the iris setosa directly from sklearn datasets I get a good result:

Program:

from sklearn import datasets
import numpy as np
iris = datasets.load_iris()
X = iris.data[:, [2, 3]]
y = iris.target
print('Class labels:', np.unique(y))

output:

Class labels: [0 1 2]

But if I try to load it directly from extension '.csv' I get the following error:

Program:

import pandas as pd

iris = pd.read_csv('iris.csv', header=None).iloc[:,2:4]

x = iris.data
y = iris.target

output:

'DataFrame' object has no attribute 'data'

Why does this happen?

"sklearn.datasets" is a scikit package, where it contains a method
  load_iris().

load_iris(), by default return an object which holds data, target and other members in it. In order to get actual values you have to read the data and target content itself.

Whereas 'iris.csv', holds feature and target together.

  FYI: If you set return_X_y as True in load_iris(), then you will directly get
  features and target.

from sklearn import datasets
data,target = datasets.load_iris(return_X_y=True)


Are there any actual, commercial uses of GANs already? Doing research on the internet, I found many scientific papers, ideas, and experiments concerning GANs. But I was unable to find a single example of it being already used commercially.

Q1 can you give examples of companies already using GANs in their product?

Q2 if you're unable to give examples, what's the reason for it? Are GANs too young to be already commercialized? Or maybe companies just have no reason to reveal that they're using this framework to train their AI systems?

PS: I'm aware of some "real world" uses:

Some people use it to create fake news, and other nasty things and probably benefit from it.
Neural photo editor available on Github

But it's hard to call then "commercial" uses.

I have seen a showcase of a commercial company that used a version of GAN-based super resolution to reconstruct a more detailed image of the seabed out of low resolution depth gauge profiles.

I could imagine that you could use that to visualize a seabed for VR presentations (I'm not sure whether that is done). However I very much doubt the added value over more traditional interpolation, when applied for deciding whether a fairway should be dredged.


Image classification if rotated version same I asked this question on stackoverflow but was advised to come here.

I have some images to classify. I see that Convolutional neural network may be best for this, e.g. here. However, for my images, their rotated versions (to any degree) is effectively same image. What should I do so that this aspect is also taken into account.

One method I thought is to create many rotated versions (say, every 30 degree rotation) of each training image and add them to train set with same label. Is there any other way? Also, will it help if I get each image in a circular area?

Two example images are shown below:

Just encode your images first and do the classification on the encoded data (Wavelet transformation). Parts of similar image regions will result in very similar encoded matrix. Further denoising the encoded matrix by zeroing the absolute values that are less or equal some small constant should reduce overfitting and improve your accuracy.

Here an example for a simple Haar-Wavelet transform encoder:

First we take the R values of the RGB-image and divide it into 8x8 matrices such as this one:
   $\begin{pmatrix}{97,95,99,94,97,88,90,95}\\{88,86,89,90,95,96,99,98}\\{95,99,94,91,88,85,90,94}\\{78,75,84,86,80,74,77,99}\\{71,75,78,70,71,84,91,90}\\{67,74,69,70,65,68,74,70}\\{59,55,64,68,70,84,86,90}\\{79,82,88,90,85,82,88,90}\end{pmatrix}$

Now we begin with the first row: $\begin{pmatrix}{97,95,99,94,97,88,90,95}\end{pmatrix}$

Grouping two adjacent values we obtain the 4 groups:

{97,95},{99,94},{97,88},{90,95}

Next step is to calculate the average of each group:

(96, 96.5, 92.5, 92.5)

And the difference of the second value of each group and the average:

95-96 = -1, 94-96.5 = -2.5, 88-92.5 = -4.5, and 95-92.5 = 2.5

Putting them together: (-1, -2.5, -4.5, 2.5)

Attaching them to the list of our average values we get:

(96, 96.5, 92.5, 92.5, -1, -2.5, -4.5, 2.5)

Now we repeat that process, but only using the first 4 values:

Grouping: {96, 96.5}, {92.5, 92.5}

Averages and differences: (96.25, 92.5, 0.25, 0)

One last time the same process but now only using the first 2 values:

Grouping: {96.25, 92.5}

Average and difference: (94.375, -1.875)

Putting it all together we obtain the new values of the first row, where we can see, that one value was already zeroed out:

(94.375, -1.875, 0.25, 0, -1, -2.5, -4.5, 2.5)

The next steps would be just applying the above process on the other 7 rows and then doing the same for every column of the resulting 8x8 matrix. Then doing the same on every other 8x8 part of our RGB-image's R-matrix and then doing the same for the G-matrix and B-matrix. The resulting RGB-matrices will be very sparse because of all the zeroed values. We got rid of all low entropy pixels what is left are the core features of the objects from the image. Using some simple similarity metrics will result in a high accuracy classifier.

The whole process can be very easy implemented using a simple recursive function, or by using linear algebra and matrix multiplications.
This is a lossless compression, meaning the resulted matrix still have all needed information to decode it and getting the original image.
However it is possible to apply some denoising routines to get more zeroes, but doing it, information will be lost.


Forecast Model recognize future trend I set up a forecasting model that predicts call data. The forecast model uses a random forest regression model.

Data:
I have call data about every day in 15 minutes intervals of a year since 2013.

Here is a plot of the accumulated values over months:

It can be clearly seen that call data has almost doubled in 2017 over 2016. This trend should also be observable for the next few years.

features:

First, the format of my data:

 DATE                     CALL
 ....
 2017-10-23 10:15:00.000    259
 2017-10-23 10:30:00.000    292
 2017-10-23 10:45:00.000    309
 ....

From this I extracted the following features:
I have extracted the following features to predict my target variable Y (call data):

-Weekday
-Month
-Holiday (yes / no)
-Interval of the day

So I ask my model:

What is the call volume of a day and interval with the following features?

I have used the years 2015-2016-2017 to train the model. However, the model does not give the desired prognosis.

He even predicts the days for 2017 wrong. Although I gave him the data as training data.

Questions:

- Should I work on my features?

- How do I show my forecasting model that the data will double year by year
  as observable since 2016?

Before going into modelling, I guess you can do a bit more of exploratory analysis(month by month, year by year). If you find any trend or seasonality and so on.

Why did you go to RF directly without using basing Techniques like ARIMA, ARMA, Exponential Smoothening and AR so on.

Sometime RF might not give you as good results as Base models, I think you don't have trend, this is from your graph(but not certain). If you can try doing some research and see if there are some external factors which are effecting your demand. Why did that happen and what is the root case for it.

To your model to understand, it needs some feature which explains its spikes somehow, it can achieved by doing feature engg/ research


Autoencoders for feature creation When using an auto encoder to create non-linear dimensional reduced featires, is it more common to use the output of the network (the prediction of the input features) or to use the weights from the (or 1 of the if there are multiple) hidden layers? If the hidden layer is used, do you use the hidden layer activation as features or weights from the hidden layer to the output?

When you want to use Auto-Encoders (AEs) for dimensionality reduction, you usally add a bottleneck layer. This means, for example, you have 1234-dimensional data. You feed this into your AE, and - as it is an AE - you have an output of dimension 1234. However, you might have many layers in that network and one of them has significantly less dimensions. Lets say you have the topology 1234:1024:784:1024:1234. You train it like this, but you only use the weights from the 1234:1024:784 part.

When you get new input, you just feed it into this network. You can see it as a kind of preprocessing. For the later stages, this is a black box.

This is manly useful when you have a lot of unlabeled data. It is called Semi Supervised Learning (SSL).


Any useful tips on transfer learning for a text classification task I am doing a supervised binary text classification task.
I want to classify the texts from site A, site B, and site C.
The in-domain performance looks OK for texts of each site. (92%-94% accuracy).
However, if I applied the model trained on texts of one site directly onto texts of another site(without fine-tuning), the performance downgrades a lot. (7%-16% downgrade for accuracy).
Approaches I already tried:

Doc2vec embedding(trained on texts from one site) + logistic regression.

Bert embedding + logistic regression. (Using bert-as-a-service to generate the embeddings based on google pre-trained bert models).

TF-IDF + logistic regression.

Pre-trained Word2vec embedding(average word embedding for text) + logistic regression.

All of those approaches don't work very well.
I knew that the performance downgrade is unavoidable, but I would like to get a maybe 3% - 5% downgrade.

Generally the task of recognizing one type of text against "anything else" is a quite difficult problem, since there is so much diversity in text that there cannot be any good representative sample of "anything else".
Typically this problem is treated as a one-class classification problem: the idea is for the learning algorithm to capture what represents the positive class only, considering anything else as negative. To my knowledge this is used mostly for author identification and related stylometry tasks. The PAN workshop series offer a good deal of state of the arts methods and datasets around these tasks.
It is also possible to frame the problem as binary classification, but then one must be very creative with the negative instances in the training set. Probably the main problem with your current approach is this: your negative instances are only "randomly selected among all other topics of the site". This means that the classifier knows only texts from the site on which it is trained, so it has no idea what to do with any new text which doesn't look like anything seen in the training data. A method which has been used to increase the diversity of the negative instances is to automatically generate google queries with a few random words which appear in one of the positive instances, then download whatever text Google retrieves as negative instance.
Another issue with binary classification is the distribution of positive/negative instances: if you train a model with 50/50 positive/negative, the model expects that by default there is 50% chance for each. This can cause a huge bias when applied to a test set which contains mostly negative instances, especially if these don't look like the negative instances seen during training.
Finally be careful about the distinction semantic topic vs. writing style, because the features for these two are usually very different: in the former case the stop words are usually removed, the content words (nouns, verbs, adjectives) are important (hence one uses things like TFIDF). In the latter it's the opposite: stop words and punctuation should be kept (because they are good indicators of writing style) whereas content words are removed because they tend to bias the model the topic instead of the style. In stylometry features based on characters n-grams have been shown to perform well... even though it's not very clear why it works!


What is local-NMF? How is it better than original NMF? I am reading this paper, but don't really understand.

Do the words "part-based" or "local" for non-negative matrix factorization (NMF) mean that the algorithm aims to factorize some specific parts more than other unimportant parts?

Like the algorithm tries to seek some max signals in matrix and focus on that part? Is there a simple explanation about local-NMF?

The "parts-based" is explicitly stated in the text to be a synonym to "non-subtractive". It is a natural feature of NMF. Its underlying assumption is that all elements of a source matrix are non-negative, as well as that the source matrix is a product of two other matrices, all elements of which are non-negative. Thus the source matrix is a sum of independent "parts". If the parts were allowed to be negative, the problem would lose capacity for unique solutions, since then, for example one could generate out of thin air fake components that are equal and opposite.

The "local" refers to spatial locality. Especially in image processing, frequently each component is expected to be concentrated within a small region, such that all related pixels are not too far from each other. This is not achieved by naive implementation of NMF, which results in components "smeared" across the entire domain, being obviously non-local.


exercise 9.3.2 from mmds book I am reading this book http://infolab.stanford.edu/~ullman/mmds/ch9.pdf
there is an exercise 9.3.2 a) it says

  Exercise 9.3.2 : In this exercise, we cluster items in the matrix of
  Fig. 9.8. Do the following steps. (a) Cluster the eight items
  hierarchically into four clusters. The following method should be used
  to cluster. Replace all 3’s, 4’s, and 5’s by 1 and replace 1’s, 2’s,
  and blanks by 0. use the Jaccard distance to measure the distance
  between the resulting column vectors. For clusters of more than one
  element, take the distance between clusters to be the minimum distance
  between pairs of elements, one from each cluster.

I am not able to understand how clustering is done?If I replace all 3,4,5 with 1 and 1,2, with 0 I am left with a matrix

  a b c d e f g h
A 1 1   1 0   1 0
B   1 1 1 0 0 0
C 0   0 1   1 1 1

What is meant by cluster formation here? How do I solve this exercise or any solution available online please give a link.

Replacing the non-zero values is meant to simplify the matrix, i.e. we only care whether the value is zero or not.

Then you're supposed to calculate the Jaccard distance between pairs of columns. For instance between the first two columns you have $|a\cap b|=1$, $|a\cup b|=2$ so the distance is 1/2.

After that you're supposed to apply hierarchical clustering using the distance. This means progressively grouping elements (here columns) which are close to each other according to the distance.


Why is the variance going down so much in this weight initialization problem(using pytorch)? first look at this example

>>> x = t.randn(512)
>>> w = t.randn(512, 500000)
>>> (x @ w).var()
tensor(513.9548)

it makes sense that the variance is close to 512 because each one of 500000, is a dot product of a 512 vector and a 512 vector, that is sampled from a distribution with a standard deviation of 1 and mean of 0

However, I wanted the variance to go down to 1, and consequently the std to be 1 since standard deviation is square root of variance, where 1 is the variance.

To do this I tried the below

>>> x = t.randn(512)
>>> w = t.randn(512, 500000) * (1/512)
>>> (x @ w).var()
tensor(0.0021)

However the variance is actually now 512 / 512 / 512 instead of 512/ 512

In order to do this correctly, I needed to try

>>> x = t.randn(512)
>>> w = t.randn(512, 500000) * (1 / (512 ** .5))
>>> (x @ w).var()
tensor(1.0216)

Why is that the case?

Basically, because the variance is quadratic in w.

You can consider your problem as computing the variance over scalar products $z = \vec{x}\vec{w}$ by drawing 500000 samples of $\vec{w}$ from $p(\vec{w})$. Let's compute the mean first

$$ E_{p(\vec{w})} [ \vec{w}^T\vec{x} ] =  \vec{\mu}_w^T \vec{x} = 0$$

because you draw the $\vec{w}$ from a zero-centered Gaussian. With the expectation value as zero, the variance is given by

$$ E_{p(\vec{w})} [ (\vec{w}^T\vec{x})^2 ] \\
= E_{p(\vec{w})} [ \vec{x}^T\vec{w} \vec{w}^T \vec{x} ] \\
= \vec{x}^T E_{p(\vec{w})} [ \vec{w} \vec{w}^T ] \vec{x}
$$

$E_{p(\vec{w})} [ \vec{w} \vec{w}^T ]$ is the covariance matrix of the $\vec{w}$ and since they stem from independent standard normal distributions is diagonal with $\sigma_w^2 = 1$ on the diagonal. So that gives

$$
E_{p(\vec{w})} [ (\vec{w}^T\vec{x})^2 ] = \vec{x}^T \vec{x} \approx 512
$$

as you are approximating the sum of variances of 512 independent samples of standard Gaussian. If you now scale the $\vec{w}$ with a constant $k$, you can do this calculation again and it will give you

$$
E_{p(\vec{w})} [ (\frac{1}{k}\vec{w}^T\vec{x})^2 ] \\
= \frac{1}{k^2} E_{p(\vec{w})} [ (\vec{w}^T\vec{x})^2 ] \\
\approx \frac{1}{k^2} 512
$$

So in order to end up with 1 in the very end, you need to scale with $k=\sqrt{512}$.

I hope this answers your question.


Python Neural Network library/program with simple installation? (Solved?) Edit: I found a solution to my problem; there's a scipy installer that doesn't require openBLAS, and that seems to have worked.

I'm trying to find a neural network library for Python, but nothing I've tried has been what I've wanted; the libraries a lot of people seem to be recommending (e.g Keras, scikit-learn) require scipy, which won't actually install on my computer.  (I think it requires openBLAS, which doesn't seem to have a tutorial for installation.)  I did find a program that looks like it will work, but it has a maximum of one hidden layer.  Can someone point me to a library that actually works, or show me how to install scipy?

If you are trying out neural networks, I would recommend keras, it is a wrapper around TensorFlow. With just a few lines of code, you can build a neural net.

Scikit learn is really good for ml algorithms like logistic or linear regression, naive Bayes, svm and others. I am not sure if it is good enough for neural networks.

I would still think, you would require some blas library.


Hypothesis vs Hyperplane in Machine Learning I am finding it hard to understand the clear difference between Hypothesis and Hyperplane.
I know that Hypothesis is a candidate model that maps inputs to outputs after training . And , Hyperplane is the decision boundary in a classification algorithm .
But , I can't seem to understand how the two are differentiated in equations .
Can someone help me understand their differences in  equations with some visualizations ?

Let's say you want to learn a specific mapping from $\mathbb{R}^{n}$ to $\mathbb{R}^{m}$.
The following elaboration assumes that you are referring to supervised learning.
Hyperplane
Hyperplanes play a key role in neural networks.
Consider the set $H_{v,d} := \{x \in \mathbb{R}^{n} \mid  \langle x,v \rangle = d \} $ for $v \in \mathbb{R}^{n}$ and $d \in \mathbb{R}$.
For $v = 0$, we have $H_{v,d} = \begin{cases} \emptyset & d \neq 0 \\ \mathbb{R}^{n} & d = 0 \end{cases}$
If $v \neq 0$, $\dim(H_{v,d}) = n-1$ and $H_{v,d}$ is a hyperplane.
If $d= 0 $, $H_{v,d}$ if a vector space (going through the origin), otherwise it is an affine space.
In general, a hyperplane is an affine subspace with co-dimension 1, which is of the form $H = v + U := \{v+u \mid u \in U \}$, where $U$ is a subspace of dimension $\dim(U) = n-1$.
For example in a binary classification task, a hyperplane can be used to seperate the two classes. A geometric explanation of the role of hyperplane in neural networks can be found here.
In short, a neural network uses for each neuron a hyperplane (in the hidden or output layer) to define the output value of the neuron. All points within the same side (with respect to the hyperplane) are either mapped the same value (using a heavy-side function), or it depends on the distance to the hyperplane (e.g. using the sigmoid function). This understanding can be used to interprete the mapping of an input vector to a layer $l>1$. Essentially a neural network learns an arrangement of hyperplanes, which define regions. Each region is mapped to the same value (in case of the heavy-side function) or again, it further depends on the position within such a region (see here)

Model Hypothesis
A model hypothesis usually refers to something else.
We consider a set of model hypothesis $\mathcal{H}$, and each model $h \in \mathcal{H}$ may be used as the desired mapping. For example, we can define the set $$P[n] := \left \{ \begin{align*}
f: \mathbb{R}^{n} &\rightarrow \mathbb{R} \\
 x &\mapsto \sum_{r = 0}^{n} \sum_{\substack{b_{1}+\ldots+b_{n} = r \\ b_{k} \in \mathbb{N}, \forall k} }  a_{b_{1},\ldots,b_{r}} \prod_{s = 1}^{r} x_{s}^{b_{s}}
\end{align*} \middle \rvert  a_{b_{1},\ldots,b_{r}} \in \mathbb{R}
 \right \}. $$
The set collects all multivariate polynomials of degree at most $n$.
Then we can consider the hypothesis set
$\mathcal{H} := \left \{
\begin{align*}
f:\mathbb{R}^{n} &\rightarrow \mathbb{R}^{m} \\ x  &\mapsto \begin{pmatrix} p_{1}(x) \\ \vdots \\ p_{m}(x) \end{pmatrix} \end{align*} \middle \lvert p_{1},\ldots,p_{m} \in P[n]
\right \}$.
Then, $h \in \mathcal{H}$ maps an input vector $x$ to $h(x) = y$, where each component $y_{u}$ is given by a multivariate polynomial function in $x$ of degree at most $n$.
The task of the training is then to find the best coefficients which results in an  optimal hypothesis $h^{*} \in \mathcal{H}$, which is the model that is used to perform inference. Here, optimal means that we find the best model within the set of all considered models in $\mathcal{H}$.
Likewise for neural networks, if the "architecture" of the neural network is fixed, this defines the hypothesis set $\mathcal{H}$, where each $h \in \mathcal{H}$ uses the same neural network architecture, but with a specific choise of weights.
Training a neural network then delivers a model $h \in \mathcal{H}$, with "optimal" weights (in practice the weights are often not optimal).
So in short, the set of model hypothesis $\mathcal{H}$ defines which mappings can be used. The training within supervised learning then selects the best hypothesis $h \in \mathcal{H}$.


Ordinal features to decision tree in Python I have a data set with ordinal features.Each feature might have 6 to 7 levels.
Based on my search for R if you have ordinal data, rpart treats ordinal and nominal differently.
https://stats.stackexchange.com/questions/94502/decision-tree-splitting-factor-variables

But now I'm implementing the decision tree with Python and there nothing comparable to rpart to handle ordinal data. It seems Python sklearn does not handle categorical data well and I have to use one hot encoding. In this case, the order of level like level 1  to level 2 to level3......to level 6 will just disappear.

https://stackoverflow.com/questions/38108832/passing-categorical-data-to-sklearn-decision-tree

Any suggestions?
Thanks.

Ordinal variables are treated exactly the same as numerical variables by decision trees.  (And so, you might as well encode them as consecutive integers.)

As for (unordered) categorical variables, LightGBM (and maybe H2O's GBM?) supports the optimal rpart-style splits [using the response-ordering trick when suitable, else trying all splits when not too expensive].  If you want a single decision tree, just set hyperparameters accordingly.

See also:
Why decision tree needs categorical variable to be encoded?
Ordinal Attributes in a Decision Tree


What are some methods for clustering individuals into distinct groups based on Features? I started with a dataset that contained many dimensions for individuals (each id is a separate individual), and extracted three Features/Attributes columns for each individual Id.

My goal is to bucket these individuals into two or three distinct groups, based on these features, to see if I can identify distinct separations between certain groups.

I was wondering if anyone has any recommendations, or algorithms (preferably in Python) that would cluster these individuals into distinct groups?  I do not have these Individuals classified, so it's an unsupervised clustering problem.  I was thinking that K-Means might be a good option, or something similar to PCA which could reduce my dimensionality and provide insight into the Features that seem to separate the groups of individuals into distinct groups.

Thanks for looking!

Note: The data shown below was artificially generated to illustrate my question.

Appendix: Reference Data:

Pattern,Feature A,Feature B,Feature C

6,2.18,0.13,8.00
9,9.31,3.67,6.58
11,0.89,1.83,4.33
13,9.73,9.50,1.59
23,0.51,6.49,0.09
26,9.04,3.42,3.90
27,8.35,9.18,3.40
28,3.04,5.63,6.88
32,9.78,6.52,8.50
43,4.11,3.36,1.83
49,9.57,1.52,7.09
51,1.13,9.98,9.42
53,6.22,1.37,7.07
62,8.79,3.03,7.52
63,6.27,7.29,0.98
71,4.64,0.06,6.55
73,1.34,9.32,5.15
83,4.53,3.85,2.04
84,9.48,9.71,3.23
86,3.80,3.00,0.76
88,1.73,0.64,9.96

Sounds like you are on the right track.  There are many ways to attack this problem.  If you want to visualize the clustering, it would help to reduce the data down to two components.  This could be done via PCA or manifold learning if you want to go non-linear (http://scikit-learn.org/stable/modules/manifold.html).

In terms of clustering, there are many different methods to do this.  Here are some comparisons of different methods: http://scikit-learn.org/stable/modules/clustering.html


Can an output class be defaulted? In my use-case of multi-class classification, my data distribution is like below:

It might be too silly to ask this (and possibly could be gravely wrong), but is there a provision to default an o/p class to a value which is safe to be defaulted than to be predicted a completely wrong outcome.

Ex. Suppose the case, where an incoming email meant to be for "hardware" department but is predicted as for "Company Leadership" department and hence routed to all senior members of the company. In such case, since the prediction accuracy of the entire output class is below say 20% accuracy, I would like to default that entire class to "service desk" group and let them manually sort it.

Hope I made my question clear (Might be confusing as well!). Please let me know if any clarifications required. I would be happy to amend the wordings.

Thanks. :)

I don't think there is a standard method to do this. But if you use a probabilistic model you can use the predicted probability together with thresholds on each class to only allow classifications that you deem certain enough. Then if the class with the highest probability does not meet the threshold you can set it to the default class.

I tested it out with sklearn and numpy, this could be an approach:

# Train probabilistic classifier
clf.fit(X_train, y_train)

# Get probabilities
probas = clf.predict_proba(X_test)

# Get the class with highest probability
highest_proba_class = np.argmax(probas, axis=1)

# Set different thresholds per class
thresholds = np.array([0.9, 0.2, 0.5])

# Init our prediction array
predictions = np.zeros_like(highest_proba_class)

# Set a default class to set if we don't reach threshold
default_class = 2

# Loop over predictions
for idx, highest_class in enumerate(highest_proba_class):

    # Threshold check if threshold was met, otherwise set default
    if probas[idx][highest_class] >= thresholds[highest_class]:
        predictions[idx] = highest_class
    else:
        predictions[idx] = default_class


Survival analysis to estimate kanban tasks completion times I am working on a problem to estimate task completion time in kanban (project management tool).
While doing EDA, I looked at tasks that are either done or cancelled. In this case, I defined the completion time as the time taken from task creation to done/cancelled.
I noticed I am running into an issue with that definition. I am disregarding tasks that have not been done yet. If we think of "task = done" as "event = 1", this is like throwing away observations with "event = 0" in survival analysis, giving us a biased result.

How should I handle this?
I would also like to get some inputs on how should I approach "done" vs "cancelled"?

It's a matter of defining exactly which problem you want to solve, and there might be many variants:

If the goal is really to estimate "time completion", then imho you should use only completed tasks, since the other tasks haven't been "completed". Note that in this case you're counting time actually spent on the task.
If the goal is to estimate "time of solving the task", whether by completing it or cancelling it, then you're counting the duration between the time the task was initialized and the time it was either completed or cancelled. Note that in this case the duration may include time spent on other tasks.

In both cases above, I don't see any proper way to include tasks which are still pending. My idea for these cases would be to calculate a different statistic, something like "rate of completed tasks after X days" for instance.


What is the procedure to create a bag of visual words model with SIFT? I have more than 1500 black and white classified images in a training set and I want to create a probabilistic model to classified new images. To be more explicit, given a new black and white image, my model has to predict:

animal: 84%
vegetal: 12%
mineral: 4%

I watched this lecture and I still have questions about the procedure to create this model.

1. Extraction of the keypoints of each image

If I correctly understood the video, the first step is to extract all the keypoints with SIFT from all the images to create a kind of dictionary of visual words. Each word is described by a multidimensional vector.
Next, I have to use the $k$-means method to create groups of visual words.

Question 1: how many groups should I create? Does it exist a rule-of-thumb to determine the $k$ number I have to use?

2. Creation of histograms

Now, for each image, I have to create an histogram/a vector where each features corresponds to one group defined in part 1. The value associated to each feature corresponds to the frequency of the word in the image.

Question 2: how can I create this vector? Indeed, each image is unique and will never match perfectly with the words in my dictionary (words who is, by the way, mean of different words). Who can I bypass this problem?

3. Creation of the model

Finally, I have to create my model. In the video, the lecturer used SVM and has to create one classifier by category (binary classifier). In my case, I have 100 different categories (my introduction was a simplification) and I prefer to have only one classifier. Also, I want to get the probabilities, given an image, to be part of a category.

Question 3: is it possible to create only one classifier who gives probabilistic data?

To finish, don't hesitate to make suggestions or corrections about the procedure I described if you know a better way to classified my data.

Question 1: how many groups should I create?
There is no rule of thumb but typically if you have n categories of data you may want to set k as 10*n. This has known to work well, however you can always change this number by looking at the accuracy on your validation data set.

Question 2: Creation of histograms
Let us say you extract m features from each image and your dictionary contains k words, for each of these m features you would find the closest match to the k words and assign this word to that particular feature. This way you will have m number of visual words taking one of the k values. Now you count the occurrence of these visual words and creating a normalized histogram out of it. So each image will have a normalized histogram of the k visual words.

Question 3: Classification
Now that we know how to extract a histogram for a given image, we would compute the histograms on the training data and given a test image we extract its histogram. Now, how we classify this is entirely up to you. You could just find the closest match in the histograms of training data using euclidian distance or do a weighted sum etc.

A simple way of obtaining probabilistic data is to train a neural network with softmax at the output layer and given a test image pass its histogram through the network and obtain the probabilities.


Working with inaccurate (incorrect) dataset This is my problem description:

"According to the Survey on Household Income and Wealth, we need to find out the top 10% households with the most income and expenditures. However, we know that these collected data is not reliable due to many misstatements. Despite these misstatements, we have some features in the dataset which are certainly reliable. But these certain features are just a little part of information for each household wealth."

Unreliable data means that households tell lies to government. These households misstate their income and wealth in order to unfairly get more governmental services. Therefore, these fraudulent statements in original data will lead to incorrect results and patterns.

Now, I have below questions:

How should we deal with unreliable data in data science?
Is there any way to figure out these misstatements and then report the top 10% rich people with better accuracy using Machine Learning algorithms? -How can we evaluate our errors in this study? Since we have unlabeled dataset, should I look for labeling techniques? Or, should I use unsupervised methods? Or, should I work with semi-supervised learning methods?
Is there any idea or application in Machine Learning which tries to improve the quality of collected data?
Please introduce me any ideas or references which can help me in this issue.

Thanks in advance.

I have read @MaximHaytovich's answer and it is a good one. I would just like to suggest some further options, which were generalized in that answer as feature engineering. I would suggest trying to do the obvious first and analyze the data before transforming it to become ready for any machine learning algorithm. First look at the data and learn yourself and hypothesize about what patterns could be indicators of fraud. What is the nature of the data that you are 100% sure is reliable (it is mentioned in the question you posted that there is available data that is reliable) and how informative are they for the goal? The more familiar you are with the data, the better interpretation you can provide to any output you get from the methods you use. Try doing some outlier analysis on the incomes or any features that may be fraudulent that may make sense to do outlier detection on. Are there any suspicious missing values? Try doing clustering (one of the unsupervised algorithms you were talking about) to see if there are any in-betweens or irregularities.


How can we identify potential customers for a new list of customers? I have two data sets:

Customer demographic data;
Transaction data of the customers.

Now, if I have to identify potential customers to develop a marketing strategy, I would make use of clustering to identify the similar groups and develop a marketing strategy accordingly.
How could I proceed to identify potential customers in a new list of customers not previously seem by the clustering algorithm? Is it like we apply the interpretations we observed from the clusters formed to the new customers??
Need some guidance on this.

You are absolutely right. This method is called customer segmentation. Here we cluster customers based on many features like their demographic and income.
Suppose we get to know a particular set of people who belong to high income group/rich demographic are not spending much then we can do promotions to increase sales to get potential new customers hence it help drive business decision.
Having said that it's not best to run the algo once on past data and make decision as we know that customer behaviour changes frequently .So,companies usually run clustering algorithm daily to take advantage of the new customer behaviour.
So, to make our approach robust we need to find optimal no of clusters, choose best features hence it is a iterative process.
Refer this to know more.


How to perform tokenization for tweets in xlnet? X_train has only one column that contains all tweets.
xlnet_model = 'xlnet-large-cased'
xlnet_tokenizer = XLNetTokenizer.from_pretrained(xlnet_model)

def get_inputs(tweets, tokenizer, max_len=120):
    """ Gets tensors from text using the tokenizer provided"""
    inps = [tokenizer.encode_plus(t, max_length=max_len, pad_to_max_length=True, add_special_tokens=True) for t in tweets]
    inp_tok = np.array([a['input_ids'] for a in inps])
    ids = np.array([a['attention_mask'] for a in inps])
    segments = np.array([a['token_type_ids'] for a in inps])
    return inp_tok, ids, segments

inp_tok, ids, segments = get_inputs(X_train, xlnet_tokenizer)

AttributeError: 'NoneType' object has no attribute 'encode_plus'

You need to do pip install sentencepiece for it to work.
By the way, you can also give the tweets as a list to the tokenizer. You don't need to tokenize them one by one.
tokenizer(tweets, max_length=max_len, padding='max_length', add_special_tokens=True)


conv2d function in pytorch I'm trying to use the function torch.conv2d from Pytorch but can't get a result I understand...

Here is a simple example where the kernel (filt) is the same size as the input (im) to explain what I'm looking for.

import pytorch

filt = torch.rand(3, 3)
im = torch.rand(3, 3)

I want to compute a simple convolution with no padding, so the result should be a scalar (i.e. a 1x1 tensor).

I tried this with conv2d:

# I have to convert image and kernel to 4 dimensions tensors to use conv2d
im_torch = im.reshape((im_height, filt_height, 1, 1))
filt_torch = filt.reshape((filt_height, im_height, 1, 1))
out = torch.nn.functional.conv2d(im_torch, filt_torch, stride=1, padding=0)
print(out)

But the result is not what I expected:

tensor([[[[0.6067]], [[0.3564]], [[0.5397]]],
    [[[0.2557]], [[0.0493]], [[0.2562]]],
    [[[0.6067]], [[0.3564]], [[0.5397]]]])

To give an idea of what I'd like, I want to reproduce scipy convolve2d behavior:

import scipy.signal
out_scipy = scipy.signal.convolve2d(im.detach().numpy(), filt.detach().numpy(), 'valid')
print(out_scipy)

which prints:

array([[1.195723]], dtype=float32)

To get equivalent results :
im_torch = im.reshape((1, 1, 3, 3))
filt_torch = filt.reshape((1, 1, 3, 3))

out = torch.nn.functional.conv2d(filt_torch, im_torch, stride=1, padding=0)
print(out)

import scipy.signal

out_scipy = scipy.signal.convolve2d(im.detach().numpy(), filt.detach().numpy()[::-1, ::-1], 'valid')
print(out_scipy)

There was a shaping issue (https://pytorch.org/docs/stable/nn.functional.html) and the application of the filter in torch differs a bit with how it is handled in scipy : https://discuss.pytorch.org/t/functional-conv2d-produces-different-results-vs-scipy-convolved2d/17762


Data Reshaping for CNN using Keras I'm a beginner in Keras. I've loaded MNIST dataset in Keras and checked it's dimension.
The code is

from keras.datasets import mnist

# load data into train and test sets
(X_train, y_train), (X_test, y_test) = mnist.load_data()

print(X_train.shape)
print(y_train.shape)
print(X_test.shape)
print(y_test.shape)
print("Shape: ", X_train[0].shape)

And the output is

(60000, 28, 28, 1)
(60000, 10, 2, 2, 2, 2)
(10000, 28, 28, 1)
(10000, 10, 2, 2)
Shape:  (28, 28, 1)

As X_train and X_test are already in the shape (#sample, width, height, #channel). Do we still need reshaping? Why?
The tutorial I'm following use the following reshaping code:

X_train = X_train.reshape(X_train.shape[0], 28, 28, 1).astype('float32')
X_test = X_test.reshape(X_test.shape[0], 28, 28, 1).astype('float32')

My second question is that why is .astype('float32') is used in code?

Lastly, I could not understand the output of print(y_train.shape) and print(y_test.shape).

Please suggest.
I've already read Reshaping of data for deep learning using Keras however still my doubts are unclear.

Answer 1
The reason for reshaping is to ensure that the input data to the model is in the correct shape. But you can say it using reshape is a replication of effort.

Answer 2
The reason for converting to float so that later we could normalize image between the range of 0-1 without loss of information.


NER vs Text classification for very short sentences Given a large set of short sentences (around 20-30 words) and multi label task (around 100 labels , can be to 3 labels per sentences ).

The location of each annotation is not impotent (i.e i only need to know if the annotation is included in the sentence)

Which method will be more beneficial ? using NER models with labels attached to the tokens from each sentence, or text classification where the sample is the whole sentence .

The labels are action that physician  is doing (i.e "clean wound" , "remove skin" etc)

Using NER (more generally sequence labeling) means classifying every token in the sentence, so if the goal is only to label every sentence there's no strong need for it in your case.

However NER might be more appropriate in case the order of the words is important, because sequence labeling models take it into account whereas traditional text classification methods often use a "bag of words" representation (order doesn't matter).

To some extent it also depends whether the labels are always related to a particular term in the sentence: if yes, then NER might be better at locating these terms (this is related to the point about order). If no, then classifying at the level of sentences is likely to perform better.


the relationship between the number of filters/kernels and the number of feature maps The above image is from "Deep Learning Tutorial" by Yann LeCun and Marc'Aurelio Ranzato (see pages 73 and 81).

I don't understand why 64 kernels (from the input to layer 1) produce 64 feature maps while 4096 kernels (layer 2 to layer 3) give 256 feature maps.

In the linked tutorial, each kernel is 2-dimensional, and applied to a single channel/feature map from its input. To construct a feature map, the output from $N_{input\_channels}$ convolutions are added together - this is important as it allows to build feature maps that have interactions between feature maps in the previous layer.

From the diagram, the first input layer has 1 channel (a greyscale image), so each kernel in layer 1 will generate a feature map. However, once you have 64 channels in layer 2, then to produce each feature map in layer 3 will require 64 kernels added together. If you want 256 feature maps in layer 3, and you expect all 64 inputs to affect each one, then you usually need 64 * 256 = 16384 kernels. The value 4096 is coming from some other aspect of the architecture not shown in the diagram, such as dividing the feature map into groups so that each output layer only processes a fraction of the input layers.

There are a few notation and presentation differences between presentations and tutorials on convolutional networks, depending on the source. This is one of them. Other sources may arrange the kernels into a 4D structure: $N_{input\_channels} \times N_{output\_channels} \times K_{width} \times K_{height}$ to make the relationship between kernels and input/output more explicit.

Another way to show the same thing is to treat the kernels as 3D, and ensure that the kernel depth $K_{depth}$ is the same as the number of input channels $N_{input\_channels}$ - this is same thing mathematically as summing up multiple separate convolutions, and dimensions might be $N_{output\_channels} \times K_{width} \times K_{height} \times K_{depth}$


Is there any PageRank-like method on weighted graph? I found that PageRank algorithm depend heavily on the existence of edges, but the weights of edges give small effect.
For example, PageRank on triangle graph(A -> B,  B -> C,  C -> A) is always (1/3, 1/3, 1/3) even if one of the weight is 0.0000001.
Is there any PageRank-like method work well on the weighted graph?

Thanks in advance!

Weighted versions of Pagerank do exist, and it is easy to incorporate edge weights into the PageRank algorithm (just multiply each edge's probability by its weight vector, and then normalize to make the edge probabilities to add up to 1)

The difficult question is how to set these weights. Backstrom and Leskovec propose an algorithm to learn these weights in their WSDM paper on Supervised Random Walks which they propose for use in link prediction and link recommendation.


Is there any data set containing list of windows software and the category each one of them belongs to? I wanted a dataset which lists many windows softwares according to their categories  for a project. eg: anaconda: software development, roblox: gaming, VLC: video player etc. I searched but could not find anything similar to this. Please link me to one of them if you are familiar to any such dataset.

What kind of categories are you looking for? Could you give more insight into why you need such a dataset? I did a prima facie search and your question seems true. I can create the dataset based on web scraping based on your requirements.


Dataset image size and inference speed Does training/fine tuning a pre-trained model on a the same dataset but with sizes scaled down (e.g., by 70%) improve inference speed? More generally, does training a CNN on smaller images improve inference speed?

As you have phrased it the answer is no.

Models typically have a fixed input and output size (except RNNs, which are not really used for images).
This means that you can not input smaller images to the network.
If it appears that you are doing so it is either because the network is automatically resizing or padding your small image up to the model input size.
In these cases, the time performance of the model will be fairly constant.

If you were to train the same model with a smaller image input size, you would be correct in saying that inference (and train) time would be faster than the larger image input size model.
You could also load the pre-trained weights from the larger model into the smaller model, in most cases.

Note that what you are considering will change the input resolution and as such the features will change, decreasing the loss-performance of the model.


A robust metric in the presence of class imbalance When evaluating the performance of a multiclass classification problem, on a highly imbalanced dataset, what is the most robust metric for this purpose?

I read a paper that states:

  "Average precision is a robust metric in the presence of class
  imbalance since it excludes the ‘true negatives’ constituent in
  specificity, focusing instead on precision, or positive predictive
  value."

The confusing part, for me, is the applied methodology in this paper. They used balancing techniques (SMOTE, class_weigh, random sampling) to oversample minority classes, but they are still concerned about the evaluation metrics and 'true negatives'.

For evaluating the classification of a highly imbalanced, there are several measures that you may consider. Remember, that in such a problem we would prefer a measure that is not biased towards one of the classes but gives similar importance to both classes.

Methods for evaluating classifiers could be divided into two types: singular assessment measures where the evaluation is given for a specific threshold (i.e. trade-off) and area-under-curve (AUC) analyses where a range of thresholds is considered.

The F-measure combines precision and recall. Precision is sensitive to data distribution as it combines values from the two rows of the confusion-matrix hence, as the difference between the numbers of negative and positive instances gets larger, the influence of the positive class (minority) gets smaller. Recall, on the other hand, is not sensitive to data distribution since it does not refer to the negative class at all. Affected by the precision, the F-measure must be sensitive to class distribution as well. The F-measure gives higher importance to the correct classification of positive instances, a change in the TNR value will influence the F-measure value less than a change in TPR. This is explained by the impact of the recall, which considers only the correct classification of positive class instances.

The Geometric Mean (G-measure, GM) combines TPR and TNR, denoting the accuracy rates measured on both classes separately. The GM is independent of the class sizes, indicating the relative part of correctly classified instances in each class regardless of the degree of imbalance. That is, the GM is not influenced by the imbalance between the classes, The GM relates to the performance on both classes with the same importance since it is a harmonic mean of the TPR & TNR and their effect is symmetrical. Higher values of GM are achieved when the accuracies on both classes are high and similar. For example, a higher GM value is accomplished when accuracies on both classes are equal to 0.5 than when one is 0.6 and the other 0.4.

The ROC-AUC is based on the TPR and FPR, which is 1-TNR, and hence have the same reasoning for suiting such problems as the GM.

The PR-AUC, precision-recall curve, are also useful for highly imbalanced data and also is robust for changes in the balance of classes (addition of samples). I've found this article very useful for understanding AUC measures and the benefits of each one of them.

The AUC analysis provides a more generic evaluation however, in most cases it will not give the best solution when a specific trade-off is to be considered. The AUC is only relevant if your classifier output is not discrete but continues.

For more reading and comparisons you may use Prof. Haibo He lectures which are very comprehensive and useful.


Batch data before feed into CNN network I am working on a project to classify CT scan images using the CNN model, the image size is huge and I want to feed it into the network using the idea of batches, tried doing that with this pieces of code:

    # train_data size = 5460
    num_epochs = 14
    batch_size = 390
    batch = 0
    print("Starting training...")
    for epoch in range(num_epochs):
        train_batch = train_data[batch:batch_size]
        batch += batch_size
        batch_size += batch_size
        ep_loss = 0
        for data in train_batch:
            X = data[0]
            Y = data[1]
            _, c = sess.run([optimizer, cost], feed_dict={x_img: X, y_label: Y})

my questions are:

1- Is this a right way to do batching? or there is a better way?

2- with the above code I am using 'AdamOptimizer' is it a good optimization technique for the idea of batching or I should use another one?

It's is a good practice to use batches to train neural networks. As Yann LeCun said:

  Training with large minibatches is bad for your health.  More
  importantly, it's bad for your test error.  Friends dont let friends
  use minibatches larger than 32.

Although, it will not help you deal with large images. This is what convolutions are for.

If you are using Keras, then the batch implementation is entirely made for you. If you use Tensorflow, then you can use the tf.data api or do your own implementation. The code you provided seem to work for your particular training size, but you may want to adapt it for all kinds of training size (namely using num_iterations = training_size // batch_size) and making sure that the case where not all examples where shown to the network, the last remaining examples are included as well before ending the epoch...

I have had good results with the Adam Optimizer, although you might want to tweak the hyperparameters to get better results.


clarification on splitting individual trees in extra trees? So I am a beginner in machine learning and just started learning about random trees in this article here. When it talks about tuning the hyperparameter K, I'm a bit confused as to how it works. It says:

  The parameter K denotes the number of random splits screened at each node to develop
  Extra-Trees. It may be chosen in the interval [1, ... , n], where n is the number of attributes.

So K would be the number that determines how many attributes to consider for a random split. Then to split, a random attribute from that set will be chosen? But what I'm wondering is that:

If K > 1, in a given set of attributes [1,2,3,4,...,n], is it always a contiguous subset of size K? Or is it K random attributes chosen from those n attributes? And once you choose a random attribute from that subset, it is replaced or left out?

It also says:

  For a given problem, the smaller K is, the stronger the randomization of the trees

I'm confused as to why this is.

So K would be the number that determines how many attributes to consider for a random split. Then to split, a random attribute from that set will be chosen?

No, a random split is made for each of the randomly selected features, and these $K$ potential splits are scored and the highest scoring one is made.

  If K > 1, in a given set of attributes [1,2,3,4,...,n], is it always a contiguous subset of size K? Or is it K random attributes chosen from those n attributes? And once you choose a random attribute from that subset, it is replaced or left out?

We choose $K$ random attributes, not necessarily (and probably not) contiguous.  I think the original algorithm selects without replacement, and the implementation in sklearn definitely does it without replacement.  I could imagine another implementation deciding to do it with replacement though: still the split point for that feature could be different between the two selections.  (One more caveat is when a selected feature is constant for the population in the current node: these should be left out, and probably it is implementation-specific whether to select a new feature or just have fewer than $K$ in the end.)

  It also says:

  For a given problem, the smaller K is, the stronger the randomization of the trees

  I'm confused as to why this is.

The smaller $K$ is, the fewer features you consider for a split, and so the more the algorithm depends on these random choices (and is less likely to latch on to noisy, spurious correlations).  Compare to a full decision tree model, in which every split point of every feature is considered.


Explanation of how DeepExplainer works to obtain SHAP values in simple terms I have been using DeepExplainer (DE) to obtain the approximate SHAP values for my MLP model. I am following the SHAP Python library.
Now I'd like learn the logic behind DE more. From the relevant paper it is not clear to me how SHAP values are gotten. I see that a background sample set is given and an expected model output is calculated based on this data and the difference is calculated with the current model's output. This difference is the sum of the SHAP values. However, I don't understand how each contribution is obtained? Could you give an explanation with simple terms?

From https://en.wikipedia.org/wiki/Shapley_value, it is possible to understand that direct computation of Shapley values is difficult with their general formula :

$$ \varphi_i(v) = \frac{1}{\text{number of players}} \sum_{\text{coalitions excluding }i} \frac{\text{marginal contribution of }i\text{ to coalition}}{\text{number of coalitions excluding } i \text{ of this size}}$$

Basically beacuse the number of coalitions excluding i grows in complexity with $\sum_{k=1}^{n-1} k!$, where n is the number of variables. Some progresses has been made in the direction of evaluating this sum with Monte-Carlo techniques (as mentionned in https://christophm.github.io/interpretable-ml-book/), but those calculations are still intensive.

In their article (http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions) Lundberg and Lee proposes two new approaches, relying on SHAP - (these are the Shapley values of a conditional expectation function of the original model) :

A model agnostic approach basically rewriting the problem as a linear regression problem, which is intuitively less expensive to compute. Basically saying that SHAP is a function of the weight of the model and trying to approximate it.
A model specific approach. Assuming input independence (which is rarely true ...) they show how to compute SHAP value directly from model weights. Starting with linear models they devise similar relations for NN with usual propagation techniques.

As to which method is used exactly for MLP, I don't know exactly, but the second one seems more appropriate (model specific, exact method).


Data Science oriented dataset/research question for Statistics MSc thesis I'd like to explore 'data science'. The term seems a little vague to me, but I expect it to require:

machine learning (rather than traditional statistics);
a large enough dataset that you have to run analyses on clusters.

What are some good datasets and problems, accessible to a statistician with some programming background, that I can use to explore the field of data science?

To keep this as narrow as possible, I'd ideally like links to open, well used datasets and example problems.

Just head to kaggle.com; it'll keep you busy for a long time. For open data there's the UC Irvine Machine Learning Repository. In fact, there's a whole Stackexchange site devoted to this; look there.


Why does "classification+regression" work better than "classification" in my NN? I am working on a research problem where I got stuck on something which doesn't really make sense to me. To explain the issue I will give an example of a similar problem (face age detection).

Suppose that I have many face images that I precisely know how many **days** old the person in each image. (I need precision in my problem).

I train my model using CNN in three ways.

In the first training, I create a neural network that gives an output of 75 classes (say the ages are 0-74) which tells me the correct age with 30-40% accuracy. Below is the final code piece after the CNN layers:

net1 = tf.reduce_mean(net, axis=(1, 2), name='gap')

# Fully-connected classifier
net1 = slim.fully_connected(net1, 512, activation_fn=utils.lrelu, scope='cmi_ff1')
net1 = slim.dropout(net1, prob, scope='dropout1_net1')
net1 = slim.fully_connected(net1, 256, activation_fn=utils.lrelu, scope='cmi_ff2_net1')
net1 = slim.dropout(net1, prob, scope='dropout2_net1')

y_ = slim.fully_connected(net1, n_classes, activation_fn=tf.nn.softmax, scope='cmi_ffo_net1')
loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits( logits=y_, labels=y))

n_classes=75 as the number of ages is 75.

I keep everything the same and change the output layer from classification to a regression which results in significantly lower accuracy that each person approx 37 years old. This tells me that my neural network is trying to minimize the error and doesn't learn anything. I keep everything the same except for the output layer and loss function.

y2_ = slim.fully_connected(net1, 1, activation_fn=None, scope='cmi_ffo_net2')
loss2 = tf.reduce_mean(tf.abs(y2_ - tf.cast(y,tf.float32)))

Up to now, nothing sounds weird to me (although I am a newbie in DL). The weird thing is happening when I decide to use two heads instead of a single head. I split the network into two right after the last convolutional layer.

net1 = tf.reduce_mean(net, axis=(1, 2), name='gap')
net2 = tf.reduce_mean(net, axis=(1, 2), name='gap')

# Fully-connected classifier
net1 = slim.fully_connected(net1, 512, activation_fn=activation, scope='cmi_ff1')
net1 = slim.dropout(net1, prob, scope='dropout_net1')
net1 = slim.fully_connected(net1, 256, activation_fn=activation, scope='cmi_ff2_net1')
net1 = slim.dropout(net1, prob, scope='dropout_net1')

y_ = slim.fully_connected(net1, n_classes, activation_fn=tf.nn.softmax, scope='cmi_ffo_net1')
loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(logits=y_, labels=y))

# Fully-connected regressor
net2 = slim.fully_connected(net2, first_fc, activation_fn=activation, scope='cmi_ff1')
net2 = slim.dropout(net2, prob, scope='dropout_net1')
net2 = slim.fully_connected(net2, second_fc, activation_fn=activation, scope='cmi_ff2_net2')

y2_ = slim.fully_connected(net2, 1, activation_fn=None, scope='cmi_ffo_net2')
loss2 = tf.reduce_mean(tf.abs(y2_ - tf.cast(y,tf.float32)))

Here, the accuracy of classification goes up to 65-70% and regression remains to the same (converges to the middle value). While regression seems to be useless in the entire process, somehow it gives an amazing push to the classification to improve accuracy that I can't even speculate about the reason.

I am adding a figure that shows the accuracy and loss of the two-headed network. (I was trying to optimize some hyperparameters.)

Explanation of the figure:

subplot 1,1: Training accuracy for classification head
subplot 2,1: Validation accuracy for classification head
subplot 1,2: Loss for classification head
subplot 1,3: Loss for regression head
subplot 2,2: Total loss: 0.2* regression + classification

I wonder what could be the reason for this or is there something that I do wrong? Also, I know the code is not complete but I can't share the entire code. I just need some directions about what to think and where to look.

You are using accuracy to estimate the performance of your model, which is a bad idea. You can't understand the limitations and advantages of your model using accuracy as a evaluation metric. Consider the following situations

Your model predicts 36 instead of 37
Your model predicts 106 instead of 37

Accuracy as a evaluation metric, treats both of them in the same way. I'd suggest you to go with MAE or RMSE for evaluation. You can train the network either using classification or regression, but make sure that you evaluate it using the right metric.

Btw, the multihead idea is really good.