Number of filters in Convolutional Neural Networks How do I know how many filters to choose from? And how does the neural network learn and adjust the filters?

Number of filters depends on the problem you are solving and the complexity of objects, which have to be recognized by your model. During a training step model receives images, implements transformation to them using current states of convolutional filter's weights to extract some significant information and then compute the error between output and groundtruth labels. Then using back propagation weights are recomputed to optimize the model's performance. You can find here more information, how it works https://towardsdatascience.com/how-does-back-propagation-in-artificial-neural-networks-work-c7cad873ea7 First layers usually extract some simple features of objects on images (for example, it could be horizontal or vertical lines). Going deeper the complexity of features are growing. There is link to the original unet paper with description of its architecture, which might clarify usage of filters https://arxiv.org/pdf/1505.04597.pdf


Which animal has similar brain complexity to VGGNet I know that it sounds weird, but I read somewhere that VGGnet has roughly 14.5k neurons and 138/144M parameters depending on the exact architecture as it's one of the biggest.

Also I read here, that the biggest neural networks trained on super computers have roughly 80M neurons, which is similar to frog's brain.

If we had to compare the complexity of VGGNet to brain of some kind of animal (even simple one) what would it be?

According to this wiki: List of animals by number of neurons

The closest ones you got are:

Pond snail with 11,000 neurons.
Sea slug with 18,000 neurons.

I think this was more of a biology question since you covered the data science part.


Forecasting one time series with missing data with help of other time series I have time series $R$, which shows, how something changes at the regional level.

I have several time series $U_i$, which show, how something changes at a special unit $I$ level.

There are many units in the region. $R$ has no missing data. Different $U_i$ have their own missing periods.

I want to forecast $U$ after a missing period using information of $R$ and information of $U$  when it was availible.

My thoghts till now:

Suppose $R$ is known on interval $[0, 365]$. Suppose $U_i$ is known on interval $[0,300]$. Let's take $R$ and $U_i$ both on interval $[0,300]$, take difference between them and trying to predict that difference with linear regression. So for interval $[301,365]$, I will have differences and to restore $U_i$, I will just have to take out my differences from $R$.

I don't like my solution, because:

We need a model for each $U_i$.
Because, sometimes data is more sparse and I don't even have a $[0,300]$ known interval, so not able to train regression properly.

Not a fully complete answer, but some inputs.

Your time series are correlated.
I assume that the measure you want to forecast for a region is an aggregation of units forecasts.

To address the first point, I usually use Vector Autoregressive Model (VAR) that forecast all time-series at once (each one being expressed as a regression using the others)
The second point involves the concept of hierarchical forecasting and reconciliation. You can exploit the fact that the regional forecast should/must equal the unit- forecasts. There can be a process to adjust forecasts to take that into account.

There are both packages for VaR and hierarchical reconciliation in R but as far as I know no direct code to handle both at the same time...

You may find this paper providing some details on the proposed approach:
https://mpra.ub.uni-muenchen.de/76556/1/MPRA_paper_76556.pdf


How to create correlation matrix but with only part of the rows? I would like to have correlation matrix like this one, but with only 3 bottom rows but all the columns. How can I generate it?

corr = corrdata.corr()
sns.heatmap(corr, mask=np.zeros_like(corr), annot=True, cmap=sns.diverging_palette(220, 10, as_cmap=True),
            square=True, ax=ax, fmt='.2f' ,  )

You can use a slice of the correlation matrix dataframe.

import pandas as pd
import seaborn as sns

df = pd.DataFrame({'A':[0,1,2,3,],'B':[4,5,6,7],'C':[8,9,10,11]})
sns.heatmap(df.iloc[2:3, 1:3], annot=True, cmap=sns.diverging_palette(220, 10, as_cmap=True),square=True, )


AttributeError: 'str' object has no attribute 'keys' While converting json file to csv,I got the above error
My code is shown below:

import json
import csv
data = open("SRInFlow.json",encoding="utf8")
rows = json.loads(data.read())
r=rows.values()
fieldnames = [b'dateTimeValues', b'timeSeries']
with open('C:/DeepLearning/tensorflowpoc/Codes/SRInFlow.csv', 'w') as f:
    dict_writer = csv.DictWriter(f, fieldnames=fieldnames)
    dict_writer.writeheader()
    for d in r:
  --->    dict_writer.writerow(d)

The error displays in the last line. How to rectify the error? Thanks in advance

This error is because you are initialising a dictionary writer dict_writer = csv.DictWriter(f, fieldnames=fieldnames) but then you are passing a normal row containing only the values r=rows.values() -> for d in r: ->dict_writer.writerow(d), when the writer expects a dictionary including the keys. That is what the error means.

I don't know how your JSON looks but you can try something like this:

import json
import csv

data = '{"dateTimeValues": [1,3,4], "timeSeries": [0,0,0]}'
rows = json.loads(data)

r=zip(*rows.values())

fieldnames = ['dateTimeValues', 'timeSeries']
with open('c:/Users/jsz.P3-GROUP/Desktop/shit.csv', 'w') as f:
    dict_writer = csv.writer(f)
    dict_writer.writerow(fieldnames)
    for d in r:
        dict_writer.writerow(d)

This initialises a simple csv writer dict_writer = csv.writer(f)(not dictionary), then inputs the fieldnames as a normal row for the header dict_writer.writerow(fieldnames) and finally inserts only the values as in your example.

Note that for my json string I had to transpose the values first as in r=zip(*rows.values()).

Hope this helps.


How distribution of data effects model performance? I am working on House Prices: Advanced Regression Techniques dataset. I was going through some kernels noticed many people converted SalePrice to log(SalePrice) see below:

I can see that taking a log transform reduced the skewness of data and made it more normal-like. But, i was wondering will it improve my model's performance or is use-full in any way. If it is then how a normal distribution of target variable was a catalyst?

Good question. Your interpretation is adequate. Using a logarithmic function reduces the skewness of the target variable. Why does that matter?

Transforming your target via a logarithmic function linearizes your target. Which is useful for many models which expect linear targets. Scikit-Learn has a page describing this phenomenon: https://scikit-learn.org/stable/auto_examples/compose/plot_transformed_target.html

Important to note

If you modify your targets before training, you should apply the inverse transform at the end of your model to compute your "final" prediction. That way, your performance metrics can be comparable.

Intuitively, imagine that you have a very naive model which returns the average target regardless of the input. If your targets are skewed, it means that you will under-/over-shoot for a majority of the predictions. Because of this, the range of your error will be greater, which worsens scores such as the Mean Absolute or Relative Error (MAE/MSE). By normalizing your targets, you reduce the range of your error, which ultimately should improve your model directly.


Postitive event but unsure when it occurred in time Morning,

I have a lot of data of which I am positive an event (target data / the event I want to predict going forward) occurred within a two week time frame, however I am unsure when it happened within this time frame.

I can get daily or more frequent feature data, but the target data only appears in 2 week gaps or in some cases 4 week gaps.

Currently I use an average of any feature during the timeframe or gather the feature data on the same day as when the target data is available but is there a more appropriate / better way?

I am using the data for machine learning purposes.

This is referred to as "interval censoring": that is to say, you know the event occurred within an interval, but not exactly where in that interval.

I don't think there has been a lot of attention in the ML community for interval censored data (or any, really). However, there has been a reasonable amount in the statistical community, myself included. As such, I've written the R package icenReg for interval censored regression models. Regression models certainly can be used as ML tool, although these don't yet have all the bells and whistles that are more typical for ML problems (i.e. no penalized regression a la elastic net etc.).

However, icenReg at least contains a tool for generalized cross validation, although it is hidden from the public. It can be extracted by icenReg:::icenReg_cv.


Step-by-step construction of an RBF neural network I have started to look at Radial-Basis-Function Neural Networks (RBF-NN), and would like to solve the following task as an exercise: Given is the data in the image below. Each output $Y_j$ of the network is defined as:

$(*)$  $Y_j = \sum_{i}^N w_{ij}* exp(-\frac{||x-\mu_i||^2}{2\sigma_i^2})$

for the -ith neuron. The task is to draw an RBF network that perfectly classifies the data, with suitable means, covariances and weights. In a second step a point has to be taken and classified in the worked-out model.

My ideas We have two nodes in the input layer, one for each dimension. The hidden-layer has as many neurons as there are training samples. Each of these calculates the activation given by the exponential above. The output layer has three output nodes as there are three classes. I would determine $\Sigma$ and $\mu$ as follows: For each class and training-sample $x_i$ of this class, $\mu_j$ is just the centroid of the training samples, $\sigma = \frac{1}{m}\sum_i ||x_i-\mu||$, and $\Sigma = \sigma*I_d$.

Questions That does not seem to be correct however - if I have the same $\Sigma$ and $\mu$ for all the hidden nodes of the same class, each new test-input would result in the same activation for all these nodes..so what exactly are the particular $u_i$ and $\sigma_i$? Also, for perfect classification, I would set all the weights which belong to the correct class to 1, and all the others to zero - would that make sense?

The hidden-layer has as many neurons as there are training samples.

This already sounds a bit fishy to me. The hidden layer of an RBF network resembles the Radial Basis functions. Thus, you have to think about how many centers shall your network have. This should not correspond to the number of training samples your have, since this would force the network to learn by heart. Let's look at the example you gave in the question:

Here I propose to use a network with four hidden neurons, corresponding to four RBF centers, marked with stars. This is, in the end, how your network should learn to generalize over the data you feed into it. Note that you do not have to set the class of an RBF network in prior, the network is supposed to learn the classes of each neuron by itself. How this is done is properly explained on the wikipedia page. A nice thing about RBF networks is that there is an analytical solution (over the Moore-Penrose pseudoinverse) for your weight vector.

  That does not seem to be correct however - if I have the same $\Sigma$ and $\mu$ for all the hidden nodes of the same class, each new test-input would result in the same activation for all these nodes..so what exactly are the particular $u_i$ and $σ_i$?

I suppose you are talking about the centers $\mu_i$ and their width $\sigma_i$. These values, like described in the link above, can be set via a clustering over the data you have. I've also seen applications that adapt the centers online, but this is usually not necessary. A simple unsupervised learning / clustering approach is typically sufficient if you have all training data available.

  Also, for perfect classification, I would set all the weights which belong to the correct class to 1, and all the others to zero - would that make sense?

If your designed network is able to solve your task, the analytical solution will do the job. Wikipedia describes how it is calculated.


Graph & Network Mining: clustering/community detection/ classification I am working on graphs/networks where nodes and edges have some attributes.

I want to know what algorithm exist for:

1) clustering a graph to k groups: depend only on the structure (edge attribute only)

2) Community Detection: ( same as graph clustering) but the number of communities is unknown.

3) Classification: a supervised method where I have labels and I want to classify the nodes based on their attributes and their connections (edges).

4) Page Rank: detecting the most important nodes in a group (community, cluster) based on their connection

thank you very much.

Well ... Some points. Networked data is modeled with graphs. When you have different attributes you have Property Graph.

For clustering, you can extract the topology of the subgraph based on desired attributes and then use any Modularity-based algorithm (most recommended is Blondel algorithm). In Blondel algorithm you don't need to know the number of communities in advance.

Have a look at Network Science book by Barabasi to get more insight to networks.

For classification you may extract features from graphs and use common classification algorithms or use graph kernels and feed it to kernel methods such as SVM. Follow this.

Page rank is one of the methods for ranking but you have simpler choices according to your problem. See Centrality measures from the book above. There you can also see details of different ranking algorithms.

If you need more info you may drop a comment here.

Hope it helped. Good Luck!


Combining time-series data from different devices I am using four different sensor models (four different devices) to gather  some biometric data(eg. heart rate) for the same user, for the same time period. In an ideal scenario, all the devices are properly positioned, so for the time being we are not concerned about null values and bad readings.

The data follows this format:

timestamp: XXXXXXXXXX //unix epoch timestamp
duration: XXX //duration in seconds
value: XX
state: <state> //string that characterizes the state of the user based on the value

The devices have quite varied levels of sensitivity, and their individual graphs may contain different "state" sequences, both in terms of the number of appearances of each "state" and their individual timestamps.

I am now tasked with presenting the data into a single graph that combines the data from all devices, and clearly presents the "state" of the user over time. Is there a way to do this without making arbitrary decisions regarding the accuraccy of the devices, or the significance of small duration readings? (I am doing this in Python if that is of any importance)

Assuming that duration is the same across devices, or that it is possible to resample the data such that it is (ex: median across 1 minute intervals).

You can present the numeric data (value) as a line plot, with each sensor values as its own series. You may want to (or not!) normalize the values within each series to make them more comparable.

For the state, map each distinct state to a numerical value. Ex: { "resting": 1, "running": 2, ... }. Then use a point plot with time as X axis, and this numeric state value as Y. Mark the y ticks with the state labels. Each sensor should have its own color.

Compare and contrast

When working with multiple related data series, often the the similarities and dissimilarities between dataseries can be just as informative (or more) than each series in itself.

If the data is supposed to represent the same thing. Then consider computing the median of the value across all sensors, and then for each sensor the difference from group-mean. Plot in a line plot like above.

If you expect agreement in "state", then you can compute the mode across the sensors. And then for each sensor compute when the mode differs. Plot this in a state plot similar to above, but now have "mode" as a series - and for each sensor series only plot values that differ from mode. This can also be represented time-independent as a confusion matrix, either overall or for each sensor.


model selection in clustering I am working on a mall customer segmentation dataset (5 features, 200 rows) using clustering. This dataset does not have any ground truth labels. I had a few doubts regarding clustering:

Can I use model selection in clustering using the silhouette score? - Since my dataset does not have any ground truth labels, I read on the sklearn documentation that you can use Silhouette score to evaluate the performance of the model. Can I use different clustering techniques (like K Means, DBSCAN, Mean shift, etc.) and select the model with the highest silhouette score? The idea is sort of similar to how we do model selection in supervised learning except in the latter we use cross validation.

How do I detect overfitting in clustering? Since the dataset has no labels, I cannot think of a way to identify if the model is overfitting the data.

How do I plot the final clusters when my dataset has more than 2 dimensions? I have seen a lot of visualizations around clustering (like the one below):

Should I use PCA to reduce the features to 2 and then plot the clusters? or is there another way to do this?

To answer your initial question, yes you can use silhouette score with different clustering methods. You could also use the Davies-Bouldin Index or the Dunn Index.
Regarding over-fitting, (this is my personal suggestion) but you could train the model n times on different types of the same data to see if there clustering is the same even though the values are changed. Short example: If you have to cluster 5 apples and 6 oranges, the cluster should be the same for 10 apples and 12 oranges. You can find a bit more detail on this here: https://datascience.stackexchange.com/a/20292/103857
For your third query:
Calculate distances between data points, as appropriate to your problem.
Then plot your data points in two dimensions instead of fifteen, preserving distances as far as possible. This is probably the key aspect of your question. Read up on multidimensional scaling (MDS) for this. Finally, color your points according to cluster membership.
(source for third query: https://stats.stackexchange.com/a/173823)
Regarding pca, its subjective. PCA works well with high correlation. If your dimensions are like apples and oranges then your directly effecting your models performance, so do keep that in check. A bit of eda would help before you dive into that.


Using sensor data and a know reference point infer the position of a moving robot Say, the robot is starting at a known position and I've data coming off of the robot as it traverses the grid layout. Exploiting the nuances captured in the data - like the implication of unequal rpm in the wheels indicating the robot is turning off to the side of slower wheels- I would like to dynamically infer the location of the robot relative to its starting position.

The ideal solution would be as the data is streaming in my model would dynamically infer its location from the data. I'd like to know if there are any existing solutions for this problem or any ideas that can help me get started are appreciated.

I see two classes of solutions to your problem:

Solutions without machine learning

The way we did this when I studied realtime systems was to analyze the data coming from the wheel sensors by hand:

First, plot the rate of rotation of the robot w.r.t. the wheel speeds. Is the rate of rotation linear with the difference in speed of the two wheels? The ratio of speeds? Try to find a relation between these variables so that you can later calculate the change in the orientation of the robot from its wheel speeds over a period of time.

Second, do the same with forward and backward movement: when the wheels are turning at rpm X, how fast does the robot move? Is it a linear relation? Find an equation for it.

Once you have these two relations defined, you can dynamically update the position of the robot by determining its direction (based on previous direction and current rate of rotation) and speed (based on wheel rpm), and adding those to the previous position.

This approach is a lot of work, but will help you understand how the wheel speeds influence your robot's movement.

Solutions with machine learning

Given the popularity of machine learning in data science, I presume it might be what you were looking for when asking your question here.

What you have is time series data with two variables: speed for two wheels.

You could drive your robot around the grid for a little while, accelerating, decelerating, turning, stopping, etc. while recording the robots position as well as its wheel speeds at regular, short intervals. This is how you would build a training dataset. You will need to find a way to record the position: perhaps you could use triangulation to locate the robot in the grid, or an overhead camera. You will end up with a sequence of data points (or multiple sequences), each with two wheel speeds (the input variables) and two position coordinates (the output variables).

You would then need to train a model on the time series data, with the position at each step of the time series as your model output (a regression with two output variables).

What kind of model would suit this purpose? It would be very interesting to see what a RNN, with LSTM or GRU units, could do in this situation. The only way to find out if it'll work, however, is to give it a try.

This approach is also quite a bit of work, and will not help you understand the impact of the wheel speeds on movement much (because the model will most likely not be interpretable).


How to model segmentation of a sequence to similar parts? I guess LSTM is good for sequence modeling but how would you model "clustering" with it? Meaning, the input is a sequence and the output is labels with similar properties (I have labeled data). For example:

input:  1 2 1 1 2 1 2 5 6 5 4 5 1 1 2 1 1 2 1
output: 1 1 1 1 1 1 1 2 2 2 2 2 1 1 1 1 1 1 1

The problem is how to tell the model to learn differences and not the particular example, as a new example could have totally different values. I could have taught it to predict the N*(N-1)/2 pairwise differences (similar non similar) but it will not learn the dynamics.

The problem can be reframed as a binary classification by ignoring the order, and then a decision tree algorithm can learn to separate the groups with perfect accuracy.

import numpy as np
from sklearn.tree import DecisionTreeClassifier

# Define training data
training_data    = np.array([1, 2, 1, 1, 2, 1, 2, 5, 6, 5, 4, 5, 1, 1, 2, 1, 1, 2, 1]).reshape(-1, 1)
training_targets = np.array([1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1]).reshape(-1, 1)

# Train a descision tree with scikit-learn
clf = DecisionTreeClassifier(max_depth=1)
clf.fit(training_data, training_targets)

# Let's if the model can predict unseen data
test_data_low = np.array([0]).reshape(1, -1)
prediction = clf.predict(test_data_low)
assert prediction == 1

test_data_high = np.array([7]).reshape(1, -1)
prediction = clf.predict(test_data_high)
assert prediction == 2

The ability of a model to learn to generalize (i.e., predict total new values) can be increased by increasing regularization. In the above example, one method to increase regularization is limiting the depth of the decision tree. I have set the depth to be a single split, aka decision stump. A decision stump is the most regularized version of a decision tree.


Cold start recommender system with features I have to develop a recommender system where most of the users only buy 1 item, so I have a cold-start problem. For this reason, I'm discarding matrix factorization techniques and content-based systems in general.
Luckily, I have lots of features for each user. I have the location, age, gender, the device they use to log in, the web browser, mobile carrier, answers to a small login form, among others.
What approaches can I follow given that I have this cold start problem with features?
I understand I can treat it as a multi-class classification problem, where I predict the item that the user is most likely to buy. This has two inconveniences I can think of:

What happens when a new item is added? Do I need to retrain the whole model?
How do I prevent the system from recommending only the most common item? The multi-class problem might give higher probabilities to very common items and my recommendations might get dummy (like recommending bread on a supermarket, doesn't add any value since the user is going to buy it anyway).

Can these issues be solved with a classification approach? Are the other approaches to address these?

Since you have access to user level features, try a k nearest neighbors recommender algo.
When a user signs into your site, find the k most similar looking users (based on a similarity metric such as cosine similarity), and recommend products that those users have purchased.


Data scientist vs machine learning engineer What are the differences, if any, between a "data scientist" and a "machine learning engineer"?

Over the past year or so "machine learning engineer" has started to show up a lot in job postings. This is particularly noticeable in San Francisco, which is arguably where the term "data scientist" originated. At one point "data scientist" overtook "statistician", and I'm wondering if the same is now slowly beginning to happen to "data scientist".

Career advice is listed as off-topic on this site, but I view my question as highly relevant since I'm asking about definitions; I'm not asking about recommendations given my own career trajectory or personal circumstances like other off-topic questions have.

This question is on-topic because it might someday have significant implications for many users of this site. In fact, this stack-exchange site might not exist if the "statistician" vs "data scientist" evolution had not occurred. In that sense, this is a rather pertinent, potentially existential question.

Good question. Actually there is a lot of confusion on this subject, mainly because both are quite new jobs. But if we focus on the semantics, the real meaning of the jobs become clear.

Beforehand is better to compare apples with apples, talking about a single subject, the Data. Machine Learning and its sub-genre (Deep Learning, etc.) are just one aspect of the Data World, together with the statistic theories, the data acquisition (DAQ), the processing (which can be non-machine learning driven), the interpretation of the results, etc.

So, for my explanation, I will broad the Machine Learning Engineer role to the one of Data Engineer.

Science is about experiment, trials and fails, theory building, phenomenological understanding.
Engineering is about work on what science already knows, perfecting it and carry to the "real world".

Think about a proxy: what is the difference between a nuclear scientist and a nuclear engineer?

The nuclear scientist is the one which know the science behind the atom, the interaction between them, the one which wrote the recipe which allow to get energy from the atoms.

The nuclear engineer is the guy charged to take the recipe of the scientist, and carry it to the real world. So it's knowledge about the atomic physics is quite limited, but he also know about materials, buildings, economics, and whatever else useful to build a proper nuclear plant.

Coming back to the Data world, here another example: the guys which developed Convolutional Neural Networks (Yann LeCun) is a Data Scientist, the guy which deploy the model to recognize faces in pictures is a Machine Learning Engineer. The guy responsible of the whole process, from the data acquisition to the registration of the .JPG image, is a Data Engineer.

So, basically, 90% of the Data Scientist today are actually Data Engineers or Machine Learning Engineers, and 90% of the positions opened as Data Scientist actually need Engineers. An easy check: in the interview, you will be asked about how many ML models you deployed in production, not on how many papers on new methods you published.

Instead, when you see announces about "Machine Learning Engineer", that means that the recruiters are well aware of the difference, and they really need someone able to put some model in production.


Decision trees, categorizacion and oversampling I want to create a model to predict the propensity to buy a certain product. As my proportion of 1's is very low, I decided to apply oversampling (to get a 10% of 1's and a 90% of 0's).
Now, I want to discretize some of the variables. To do so I run a tree for each variable against the target.
Should I define the prior probabilities when I do this (run the trees), or it doesn't matter and I can use the over-sampled dataset just like that?

Do you use Python?
Python class DecisionTreeClassifier has an attribute class_weight for this purpose.
So you do not need to adjust it manually. Check here
https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html
Regarding discretizing or encoding - hard to say what is better without knowing your data.
Unless you are really sure one is the best choice, you can check the model by using encoding instead of discretizing and compare the quality.


How to deal with categorical variables I am new to Data Science and have a problem with categorical variables.
My data set has 2 columns of strings - Departure City and Arrival City. It looks like this:

ID  | DepCity| ArrCity |Price|Time|SomeColumn   |ColToPredict|
1   |London  | Berlin  | 300 | 95 | 220         | 4          |
2   |Dublin  | Nice    | 420 |115 | 59          | 1          |
3   |Milan   | Brussels| 150 |108 | 154         | 3          |
4   |Paris   | Rome    | 160 |120 | 200         | 4          |
250 |Madrid  | Oslo    | 290 |300 | 110         | 2          |

So there are a lot of categorical variables in both columns and these columns are important (their values depend on other columns).

I use Python and sklearn. And it's not possible just to eliminate them as suggested in some tutorials.

I know there is a way to deal with categorical variables by creating new columns with zeros and ones for each variable. But I'm not sure that it's my case, because I have about 30 unique cities in each column. What will be the best way to convert categorical variables into numerical?

In your specific case you are working with cities, which can be modelled as geographic location. I've never used it be you could look at the geohash method.

Or could also use an identifiant for each city. An algorithm like a random forest can directly use it this way.


How to choose most appropriate Machine Learning model? I need some help from anyone who are familiar with machine learning area.

ID        | Mach_1  | Mach_2 | Mach_3  | Mach_4 | Mach_5 | Rejected Unit (%)
127189.11     1         0        1         1        1           0.23
178390.11     0         0        0         1        0           0.10
902817.11     1         0        1         0        1           0.60

Above is the example of my data, for each ID there are several mach that are available and if that particular mach is used for that ID , the value will be 1 and if the mach is not used it is 0. And the rejected unit is the percentage value of rejected unit for that ID.

What I want to know is which mach is the most affected to the rejected unit. And what is the percentage for each mach that is affected to rejected unit.

Can anyone help me to advise on what machine learning algorithm/model that I can use to analyse this case study ?

EDIT:

I have done the linear regression and below is my code, however the output has show two warnings as shown in the screenshot below.

import statsmodels.api as sm
#create model
mod = sm.OLS(y_train,X_train)
res = mod.fit()
print(res.summary())

Fit a linear model. Using rejected unit % as target. Then see the coefficients of the linear regression to see how much contributes each to the result.

You can adapt your code to this example of sklearn:

https://scikit-learn.org/stable/auto_examples/linear_model/plot_ols.html#sphx-glr-auto-examples-linear-model-plot-ols-py

If you are feeling confortable with this, you can move to higher performance/non parametric algorithms as decission trees. Or just use a LASSO regression.


I want to extract name from CV I want to extract name from CV.
I need high level if accuracy more than 95 %. I have started with taking assumptions that it is highly likely to be found in 10% top lines or if not there then in some section similar to Personal details.
Can u plz  provide any help/guidance ?

Since CV data is mostly structured you can use Stanford NER for named entity recognition or use CRF to train and model your own named entity recognizer.


Tagging documents for doc2vec I am working on resume parsing script. I am trying to tag documents sentences with TaggedDocument function, provided by gensim.

What I have managed for now is to divide every text into sentence, put into one flat array and give every sentence an i (its order, basically) tag.

tagged_data = [TaggedDocument(words=word_tokenize(_d.lower()), tags=[str(i)]) for i, _d in enumerate(texts_flat)]

For the reason of possible improvement I want to tag every sentence not only with its order but with the name/order of text it is from. For that, I have made a list of lists, where every text is a list and every text contain list of sentences. i.e

texts = [text1 = [sent1, sent2, ...], text2, text3 ...]

How to iterate over this kind of document?
I came up with smth like

tagged_data = [TaggedDocument(words=word_tokenize(_d.lower()), tags=[str(i) + '.'+ str(j)]) for i, j, _d in enumerate(texts)]

but i get ValueError

  ValueError: not enough values to unpack (expected 3, got 2)

Is it even going to do anything?

I've managed to write a function that iterates over it for me:

class TaggedDocumentIterator(object):
def __init__(self, doc_list, labels_list):
    self.labels_list = labels_list
    self.doc_list = doc_list
def __iter__(self):
    for idx, doc in enumerate(self.doc_list):
        for idy in enumerate(doc):
            yield TaggedDocument(words=idy[1].split(), tags=[str(self.labels_list[idx]) + '.' + str(doc.index(idy[1]))])

docLabels = list(range(len(texts)))
data = list(texts)
sentences = TaggedDocumentIterator(data, docLabels)


Convolutional neural network for sparse one-hot representation I have some basic features which I encoded in a one-hot vector.

Length of the feature vector equals to 400.
It is sparse.

I saw that conv nets is applied to a dense feature vectors.

Is there any problems to apply conv nets to a sparse feature vectors?

I would not apply convolutional neural networks to your problem (at least from what I can gather from the description).

Convolutional nets' strengths and weaknesses are related to a core assumption in the model class: Translating patterns of features in a regular way either has a minor impact on the outcome, or has a specific useful meaning. So a pattern 1 0 1 seen in features 9,10,11 is similar in some way to the same pattern seen in features 15,16,17. Having this assumption built in to the model allows you to train a network with far fewer free parameters when dealing with e.g. image data, where this is a key property of data captured by scanners and cameras.

With one-hot encoding of features, you assign a feature vector index from a value or category essentially at random (via some hashing function). There is no meaning to translations between indices of the feature vectors. The patterns 0 0 1 0 1 0 0  and 0 0 0 1 0 1 0 can represent entirely different things, and any associations between them are purely by chance. You can treat a sparse one-hot encoding as an image if you wish, but there is no good reason to do so, and models that assume translations can be made whilst preserving meaning will not do well.

For such a small sparse feature vector, assuming you want to try a neural network model, use a simple fully-connected network.


Is the search for a specific n-gram the same like a string search? Is the result of a search for a specific n-gram like sherlock+holmes equal to the result of a regex search for "sherlock holmes" in the same document corpus?

So if i read about n-grams for certain words, that's the same like normal string search?

Example:

https://books.google.com/ngrams/

https://books.google.com/ngrams/info

Well, as you state the problem, it is true that the search for a certain sequence of strings/words is the same as looking for the corresponding n-gram.

However, keep in mind that an n-gram, when you use it for ML, (often) is represented as a factor. So a certain sequence of words or strings, is thought of carrying valuable information. Like „John Holmes“ is different from „Sherlock Holmes“, when it comes to, e.g. identifying book titles or covers.

In general, an n-gram is just a certain sequence of words/strings, which carry more information than only one word/string.


What knowledge do I need in order to write a simple AI program to play a game? I'm a B.Sc graduate. One of my courses was 'Introduction to Machine Learning', and I always wanted to do a personal project in this subject.

I recently heard about different AI training to play games such as Mario, Go, etc.

What knowledge do I need to acquire in order to train a simple AI program to play a game? And what game do you recommend for a beginner?

This is what I know in Machine Learning so far -

Introduction to the course and to machine learning. K-Nearest Neighbor algorithm, and K-means algorithm
Statistical Inference
Gaussian Mixture Model (GMM) and Expectation Maximization (EM)
Probably Approximately Correct (PAC) model, including generalization bounds and model selection
Basic hyperplane algorithms: Perceptron and Winnow.
Support Vector Machines (SVM)
Kernels
Boosting weak learners to strong learners: AdaBoost
Margin-Perceptron
Regression
PCA
Decision Trees
Decision Trees pruning and random forests

There are multiple ways to approach solving game playing problems. Some games can be solved by search algorithms for example. This works well for card and board games up to some level of complexity. For instance, IBM's Deep Blue was essentially a fast heuristic-driven search for optimal moves.

However, probably the most generic machine learning algorithm for training an agent to perform a task optimally is reinforcement learning. Technically it is not one algorithm, but an extended family of related algorithms that all solve a specific formalisation of the learning problem.

Informally, Reinforcement Learning (RL) is about finding optimal solutions to problems defined in terms of an agent that can observe the state of an environment, take actions in that environment and experience rewards which are somehow related to the state and action. RL solvers need to be designed to cope with situations where rewards are received later than when important actions were taken, and this is usually achieved by the algorithm learning an internal expectation of later rewards associated with state and/or state-action pairs.

Here are some resources for studying Reinforcement Learning:

Reinforcement Learning: An Introduction (Second Edition)
Algorithms for Reinforcement Learning (PDF)
Udacity Reinforcement Learning course
David Silver UCL lectures on Reinforcement Learning

You will find the subject itself is quite large as more and more sophisticated variations of the algorithms are necessary as the problem to solve becomes harder.

Starting games for studying reinforcement learning might include:

Tik-tac-toe (aka Noughts and crosses) - this can be solved easily using search, but it makes for a simple toy problem to solve using basic RL techniques.
Mazes - in the reinforcement learning literature, there are many examples of "grid world" games where an agent moves in single N,E,S,W steps on a small board that can be populated with hazards and goals.
Blackjack (aka 21)

If you want to work with agents for playing video games, you will also want to learn about neural networks and probably in some detail - you will need deep, convolutional neural networks to process screen graphics.

A relatively new resource for RL is OpenAI Universe. They have done a lot of work to package up environments ready to train agents against, meaning you can concentrate on studying the learning algorithms, as opposed to the effort of setting up the environment.

Regarding your list of current skills: None of them are directly relevant to reinforcement learning. However:

If you can understand the maths and theory from your previous course, then you should also be able to understand reinforcement learning theory.
If you have studied any online or batch supervised learning techniques, then these can be used as components inside a RL framework. Typically they can be used to approximate a value function of the game state, based on feedback from successes and failures so far.


How to predict probabilities from a new data set from an already built and validated model in Python? I have built a classification model using the following steps (and in the mentioned order) in Python -

Data cleaning - Removing unwanted variables and separating Predictor variables from response variable
Label Encoding
Standardization( StandardScaler)
Train Test Split
Smote
Model Building
Model Testing using Test data

a) Now I have a new dataset, and I want to predict using the above built model. How do I do it ? Which of the above steps should I follow and which ones should I skip ?
b) Also, is the arrangement of any of the steps aforementioned very, very wrong so that it needs to be changed ?

Welcome to Data Science on Stack Exchange,
You should apply the same steps 1-3 to the new dataset as you did to the original train and test datasets. Smote is a special case and couldn't be applied anyway as you don't have the response in the new dataset. In general, your train, test, validate, and any new datasets should go through the same data pre-processing. The new dataset will replace your Test dataset in your prediction.
There may be more steps you can add, but the current arrangement is fine.


AUC is high but not able to represent other class properly I wanted to know if it makes sense to make 2 ROC curves for each of the 2 classes?
I am doing a binary classification problem but AUC is good at 82%.
But the F Score of the class labelled 1 is very poor (around 0.4).
So the AUC is very good but the F Score is very poor.
What does the ROC curve actually mean in this case?
I always thought, better the AUC score, better the classifier.
But in my case it is not able to capture the class 1 properly.
What should be done in such cases?
Any better metric to evaluate how good the classifier is?

It does not make sense to make two separate AUC / AUROC curves for each class, these are aggregating functions, same as F1-score. If AUC is high but F-score is not then it's possible you have an imbalanced data set (classes are not equally represented in the data), since the AUC will not be able to measure this, while the F-score does. So, if you do have an imbalanced data set you should use something other then AU(RO)C, such as the F-score.


Local Development for Apache Spark I'm wondering how other developers are setting up their local environments for working on Spark projects. Do you configure a 'local' cluster using a tool like Vagrant? Or, is it most common to SSH into a cloud environment, such as a cluster on AWS? Perhaps there are many tasks where a single-node cluster is adequate, and can be run locally more easily.

Spark is intended to be pointed at large distributed data sets, so as you suggest, the most typical use cases will involve connecting to some sort of Cloud system like AWS.

In fact, if the data set you aim to analyze can fit on your local system, you'll usually find that you can analyze it just as simply using pure python.  If you're trying to leverage a series of local VMs, you're going to run out of memory pretty quickly and jobs will either fail or grind to a halt.

With that said, a local instance of spark is very useful for the purpose of development.

One way that I've found that works is if I have a directory in HDFS with many files, I'll pull over a single file, develop locally, then port my spark script to my cloud system for execution.  If you're using AWS, this is really helpful for avoiding big fees while you're developing.


Doubt to use accuracy or macro f1 measure in an unbalanced classification task I have a multi-class classification task where the organizers said that the final results will be using the Accuracy measure.

The provided data is unbalanced, and I don't have an idea about the test set (is it balanced or not), but I think it will be balanced since they use accuracy.

Anyway ..

My question: Is it a good idea to tune my system using F1-macro rather than Accuracy? since the training data is unbalanced.

or it's better to use the Accuracy?

using accuracy for unbalance data means that correct classification for the most populous class members is more important than others. If the importance of correct classifying for all data records is equal in your problem accuracy is one of the worst choices.

There are some other good choices beside F1-macro which can be more helpful. Some of these metrics are as follows:

"Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN",
"MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC",
"AUCI", "G", "DP", "DPI", "GI"

Disclaimer:

If you use python, PyCM module can help you to find out these metrics.

Here is a simple code to get the recommended parameters from this module:

>>> from pycm import *

>>> cm = ConfusionMatrix(matrix={"Class1": {"Class1": 1, "Class2":2}, "Class2": {"Class1": 0, "Class2": 5}})

>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]

After that, each of these parameters you want to use as the loss function can be used as follows:

>>> y_pred = model.predict      #the prediction of the implemented model

>>> y_actu = data.target        #data labels

>>> cm = ConfusionMatrix(y_actu, y_pred)

>>> loss = cm.Kappa             #or any other parameter (Example: cm.SOA1)


When using KNN, how do I know which order of the Minkowski distance to use? I am learning about KNN and ML in general. I know that KNN usually uses second order Minkowski distance (Eucledian Distance), but I assume it cal also use other orders. But what is the benefit to choosing a higher order with respect to the general performance of the model? Is it faster? More accurate? Does the level of noise influence my decision for the order? How do I know that because of this and this condition, I have to use a higher order Minkowski distance?

Unfortunately, there is no rule (either general or rule of thumb) for using a given distance rather than another. Minkowski distance of n-th order could work greatly for some task and some datasets, and extremely bad for others.

The reason Euclidean distance is used as default is due to its simplicity and the fact that there is no reason to consider more complex formulae as better a priori.

I suggest you to try multiple measures of distance (or multiple Minkowski orders) and check which works best for your current task. Please consider the risk of overfitting. I would run models more than once, for example using techniques for keeping overfitting under control, such as k-fold Cross Validation, and/or triple train-validation-test splits.


keras validation mean squared error always similar to 1 is there any reason why the validation mean squared error output from Keras is always very similar to 1? Thank you. All of my training results looks like:

155/155 [==============================] - 0s - loss: 6062.6136 - mean_absolute_error: 0.8344 - mean_squared_error: 1.0271 - val_loss: 0.8252 - val_mean_absolute_error: 0.8252 - val_mean_squared_error: 1.0164
Epoch 29/1000
155/155 [==============================] - 0s - loss: 5870.5280 - mean_absolute_error: 0.8324 - mean_squared_error: 1.0211 - val_loss: 0.8246 - val_mean_absolute_error: 0.8246 - val_mean_squared_error: 1.0130
Epoch 30/1000
155/155 [==============================] - 0s - loss: 5668.5083 - mean_absolute_error: 0.8311 - mean_squared_error: 1.0134 - val_loss: 0.8244 - val_mean_absolute_error: 0.8244 - val_mean_squared_error: 1.0106
Epoch 31/1000
155/155 [==============================] - 0s - loss: 5530.8119 - mean_absolute_error: 0.8288 - mean_squared_error: 1.0115 - val_loss: 0.8243 - val_mean_absolute_error: 0.8243 - val_mean_squared_error: 1.0089
Epoch 32/1000
155/155 [==============================] - 0s - loss: 5222.6773 - mean_absolute_error: 0.8283 - mean_squared_error: 1.0119 - val_loss: 0.8245 - val_mean_absolute_error: 0.8245 - val_mean_squared_error: 1.0071
Epoch 33/1000
155/155 [==============================] - 0s - loss: 5090.0273 - mean_absolute_error: 0.8273 - mean_squared_error: 1.0078 - val_loss: 0.8247 - val_mean_absolute_error: 0.8247 - val_mean_squared_error: 1.0060
Epoch 34/1000
155/155 [==============================] - 0s - loss: 4878.2420 - mean_absolute_error: 0.8272 - mean_squared_error: 1.0093 - val_loss: 0.8245 - val_mean_absolute_error: 0.8245 - val_mean_squared_error: 1.0046

note: I have standardized my input and output with sklearn standardization:

from sklearn import preprocessing
    X_scaler = preprocessing.StandardScaler().fit(X_list_total)
    X_list_total_standardized = X_scaler.transform(X_list_total)
    Y_scaler = preprocessing.StandardScaler().fit(Y_list_total)
    Y_list_total_standardized = Y_scaler.transform(Y_list_total)

Does it just mean that there is nothing to learn from the data at all?

It could be very close to 1 (or for that matter, changing very slowly) for many reasons:

Maybe your learning rate is very small.
Maybe the network has reached its learning capacity. Since we do not know anything about the network architecture, we can't rule this out.

This does not mean there is nothing to learn from the data. In the worst case, there is nothing to learn from data, with the given architecture,  hyperparameters and the time you are willing to let the network learn.


Which supervised ML model to use for exam/grade prediction? So I plan on making a mobile app that will let students predict their final grades based on their mock exam results. I can train my model with previous years results.
X: 5 mock results
Y: Final grade obtained
However, I have the issue that sometimes, or most the times, the user may be using the app whilst not having taken ALL the mock exams yet, they may want to see if they are on track and use it once they have done maybe 3 mocks. How should I go about this for choosing my supervised algorithm/model?

First you need to figure out what type of variable the response is. Is it categorical (eg. A, B, C, etc) or continuous (75%, 83%, 92%, etc)? If it's categorical, it'll definitely be ordinal. This means you can explore methods like ordered logistic regression. If it's continuous, look at something like beta regression.
For the app - if the student only keys in 3 of the 5 mock exam results, you could just get them to 'guess' their results for the other 2 mock exams. In my opinion, this is better than inputing the remaining 2 mock exam outcomes. This is since the student should have a good handle on they're going to perform.


Good approach to increase accuracy for a continuous value that is highly variable/sensitive to the inputs? I am trying to predict a continuous 'Y' variable using a variety of algorithms and feature engineering techniques. My issue is that Y is extremely variable and I reached a asymptote in accuracy.
This is the structure of my feature variables (with swapped variable names)

Screw Width (mm)
Screw Height (mm)
Screw Angle (degrees)
Screw Type
Screw Material
Car Model Id
Car Age (Years)

0.53
0.24
43
Eye Bolt
Carbon
1
3

My target variables go as followed:

Speed without Screw
Speed with Screw

24
29

Which I merge into a single variable Y:

Speed Delta

5

The range of the Delta can go from -2000. to 165,000. Please note I swapped the variable names again so I'm not in reality predicting speed.
Currently my R2 score is 0.9 and the mean prediction error rate is 25%. I want to get it down to 15%
What I have tried so far:

Constraining the Input and Sensitivity Analysis on the variables for current Regressor Choice

For some time I tried:

Splitting the data (e.g. run on Screw Width < 0.5 mm)
Build a model on the split data
Record the new R2 score/Mean Error

To my surprise it wasn't one variable or range of that variable that was causing huge variance as repeating this experiment gave me similar results. For example filtering heavily on the variables and repeating steps 1-3 gave me results like as followed:

Using different regressors to predict a single continuous variable

I tried using a simple Linear Regressor, MLP Regressor, Gradient Boosting and Random Forest. I haven't played too much with the parameters of each regressor and am not sure if that is my downfall. Currently a RandomForestRegressor is giving me best results but each regressor converges around a similar range.

Simple Feature Engineering:

I tried OHE for non numerical variables, Log Transforming the Data. The difference was noticeable but still small.

Running more data

I have close to 250,000 simulations (or rows) and reached a bit of an asymptote at 25% accuracy after 150,000 simulations with the strategies/algorithms I deployed.

Summary: I need advise on how to tackle this problem. Do I just keep on randomly trying methods/manipulating the data until something sticks. Or is there a better path moving forward to get a better accuracy?
What have I not tried?

Taking the difference (ie speed1-speed2) as the target variable effectively dismisses any low-frequency variablitiy and targets only high-frequency variability, even noise.
One approach would be to bin the (highly-variable) target variable into fixed range bins and take the mid point (or any other fixed point) of each bin as the new target (stabilised) variable. This will decrease variability (and noise) a bit, or even a lot. Then any regression algorithm can be used on the new stabilised target variable with most probably better results (depending of course on the number of bins, more bins more variability, less bins less variability but there can be some balance point).


Product of dot products in neural network In a neural network, it is common to compute a dot product of the form

$$\langle w, x \rangle = w_1 x_1 + w_2 x_2 + \ldots + w_n x_n$$

and use it as argument to some activation function. This is done for several different weights $w$ and activation functions (one per neuron). The results are used as input to the next layer, where we compute again several dot products and use them as argument to more activation functions.

Using many layers leads do several theoretical difficulties of deep learning and one has to make some effort to make everything works. But at the end, hopefully it will end up working.

I was thinking in a way of "cheating" the use of many layers. The idea is to use just one layer in a way that the argument to the activation function is complex enough so the program will learn properly. Instead of a dot product I was thinking in using products of dot products:

$$\prod_{k=1}^L\langle w^{(i)}, x \rangle = (w_1^{(1)} x_1 + w_2^{(1)} x_2 + \ldots + w_n^{(1)} x_n) \cdot \ldots \cdot (w_1^{(L)} x_1 + w_2^{(L)} x_2 + \ldots + w_n^{(L)} x_n)$$

The result is them used as argument to some activation function. Since this argument already has several weights, my hope is that the need of several layers of activation functions wouldn't be necessary anymore. I even though about using sum of these products and tested it. I'm getting $\sim 80\%$ of accuracy in a classification problem where I know it is possible to get more than $90\%$. Does my approach has a fundamental limitation I'm not aware of?

Thanks.

Extra: Above I gave a simplified description of my idea (which I think is enough), but it may be helpful to put the whole thing here. My final model of learning is given by the following formula

$$ = \textbf{f}\left( \sum_{r=1}^R \left( \prod_{i=1}^{L-1} \langle \textbf{w}^{(i,r)}, \textbf{x} \rangle \right) \left[
\begin{array}{c}
    w_1^{(L,r)} \\
    \vdots\\
    w_m^{(L,r)} \\
\end{array}
\right] \right) = $$

$$ = \left[
\begin{array}{c}
    f\left( \sum_{r=1}^R \sum_{i_1, \ldots, i_{L-1}=1}^n w_{i_1}^{(1,r)} \ldots w_{i_{L-1}}^{(L-1,r)} x_{i_1} \ldots x_{i_{L-1}} w_1^{(L,r)} \right) \\
    \vdots\\
    f\left( \sum_{r=1}^R \sum_{i_1, \ldots, i_{L-1}=1}^n w_{i_1}^{(1,r)} \ldots w_{i_{L-1}}^{(L-1,r)} x_{i_1} \ldots x_{i_{\ell-1}} w_m^{(L,r)} \right) \\
\end{array}
\right]$$

$f:\mathbb{R} \to \mathbb{R}$ is the activation function and $\textbf{f} = (f, \ldots, f)$ is the "vectorized" version of this function. Note that I tried to come up with a very flexible formulation before the activation function. The idea is that this would avoid the need of more layers. I hope everything is clear at this point. Thanks again.

Well, you are basically just creating a big nth degree polynomial with no small degrees.

So yes, fundamentally flawed because you don't have a bias in your weights, and then the optimization will be very complex because each parameter/weight depends on all the others entries in the computation.


How to train a keras model on both original and augmented data from ImageDataGenerator? I have a dataset that contains about 87000 images in a directory, with each class in a separate subfolder. I've tried the class ImageDataGenerator() and the function flow_from_directory() for generating the images, it worked completely fine but I have a question..
Does flow_from_directory() only yield the augmented images? and if this is the case, how can I train my model "which has overfit the training set" on both original and augmented data?
Thanks

ImageDataGenerator do augmentation on the fly base on the setting you give , so it did not separate “original” or “augmented “ data , just the possibility of data been augmented.
so your question actually is “ how to tune model when overfitting happens ”? I think you can start from adding dropout or reduce number of parameters. Hope this helps, cheers.


What tasks you train with one set of features and predict with another? The most common scenario in supervised learning is to have data points with a set of features and train a model to make classification predictions afterward.

Usually, for predictions to make sense with new data points, these new data points need to have the same features and come from the same distributions.

My questions is: what are scenarios in which the data points you want to predict do not necessarily have the same set of features from the data points you trained your model with?

If you mean completely new set of features for prediction: it will not be helpful. Your model 'learns' something on the training feature space and you hope to apply the learning to new data-points in the same feature space. If the feature space for prediction is completely new, the learning will be useless!
If you mean somewhat new set of features: that often happens in text applications where new tokens might show up in the data which were never seen in training data. Generally handled by some strategy like zero vector or random vector.
Same features but different distribution: this can happen in practice but model performance will go down.


How to more simply see overlapping data for dozens of variables? I'm trying to think of the best way to see how multiple variables (about 40) related to a very large userbase can be seen to interact with one another.

As an analogy, imagine I had survey data of superheros liked by thousands of school children, such that Adam Agner only likes Batman, Brian Agosto likes Superman and Batman, Cian Ailiff likes Wonderwoman, Superman and the Flash, etc. The index would be the list of children's names (tens of thousands), and the variables would be each superhero (from a list of 50) each with a True or False value.

Rather than just the amount of children who like each superhero, if I can see the overlap information in some easy way, I might find that an unusually large number only like Batman, or that most of those who like Superman are likely to also like the Flash.

The easiest would be to do it visually, but Venn diagrams wouldn't be practical with a large number (50 here) of variables. So it would seem like a correlation matrix with a heat-map base would be the way to go, with something like this:

I can imagine a heat map with superheros plotted on both axes could work for seeing interesting matches of one variable against all other ones, but it would still require some less ideal extra steps to see when three or more superheros matched, like seeing the colour overlap of superhero1 with superhero2 & superhero3 both be high, and that's kind of guesswork, because the actual children who like all 3 could still be low.

At the moment, the best solution I can think of is to try to reduce my variables down into categories, so the example could be Marvel, DC, male, female, but that loses some potential data that could be helpful from overlaps within those categories.

Maybe if I did something like the image above, the circle size would be the number for that overlap, and the colour could be number matches with other variables (without listing them, but I could further investigate). That kind of coding is a little out of my comfort zone, but I could try.

Ideas appreciated! I'd ideally do this in matplotlib or some other means within python, but if I have to use Matlab or another tool then I will consider that. Ideas or suggestions appreciated! Hopefully this was clear, thank you!

For this example specifically, I would suggest visualizing the data using a Chord Diagram.

Chord diagram from Delimited.io:

A chord diagram would allow you to see interactions and co-occurrences of likes between each of the heroes directly, including relative magnitude of the effects intuitively and immediately. You could also include other properties of the character (universe of origin, gender, time period of introduction etc) by use of color and/or positioning of the hero in question on the circumference of the diagram.

As chord diagrams are a graph-based approach, you'd want to transform the individual observations you have into what would be a (this has to be a first for this term) hero-like-cofrequency matrix formatted as follows:

+-----------+--------+------------+----------+
|           | Batman | Spiderrman | Superman |
+-----------+--------+------------+----------+
| Batman    | 0      | 2          | 3        |
+-----------+--------+------------+----------+
| Spiderman | 2      | 0          | 1        |
+-----------+--------+------------+----------+
| Superman  | 3      | 1          | 0        |
+-----------+--------+------------+----------+

Note that this represents an undirected graph, and that it is symmetric about the identity column.

As you're using Python, you may want to look into using Bokeh to create the chord diagram. One tutorial for doing so is available here.

Best of luck, true believer!


Variable Importance in unsupervised anomaly detection algorithms I am working on an anomaly detection problem to detect fraud in insurance claims. I have used the PyOD package and used algorithms like ABOD, CBLOF, Isolation Forest, and AutoEncoder. I couldn't find any way to identify the important features which make the data points anomalies ( like variable Importance in Random Forest). Is there a way to identify the important features in unsupervised anomaly detection?

There have been workshops dedicated to "outlier detection and description" (ODD), but there came out nothing from them that convinced me, unfortunately. but YMMV.
It definitely won't be enough to just use some library! You'll need to read and implement papers. There are subspace outlier detectors and correlation outliers, for example, that will tell you which features were relevant for a particular outlier.

But in general I believe you'll quickly run into multiple testing problems: in real data, every point is anomalous if you just try attribute combinations hard enough. Donald Trump is anomalous because of his orange skin, fake hair, and small hands, for example. If you only look at his xenophobia, he probably is pretty normal, unfortunately.


Help to choose algorithm for computing difference between 2 texts? I have a task to create a tool, which will be able to find articles-duplicates of a given reference article.
I know word vectorization (tf-idf,word2vec), RNN methods, but i can not choose something suitable for my situation.

My requirments:

data are being collected on the fly  (program parses articles from web sites, so i don't have regular DB with collection of texts)
there is a reference text, whose copies need to be found
copies could be copypasted, partially copypasted (by paragraphs) or paraphrased
reference-vs-copy comparison algorithm is preferable, but not required (instead of reference-vs-corps)
algorithm shouldn't do deep semantic analyzis, only kind of word counting, word vectorization, substring search
instead one algorithm, i can use a set of herurisitcs
algorithms can do false positive dicisions

I come up with such ideas:

download pretrained word2vec and compare means of word-vectors
Build a dictionary word->count from every text and compare it to reference dictionary
collect about 100 texts, vectorize them according to tf-idf and find closest to the reference

I will apreciate, if you will point specific algorithms, libs, examples based on key-word extractions, dummy substring search, line difference comparison for python or CLI.

algorithm shouldn't do deep semantic analyzis, only kind of word counting, word vectorization, substring search

For such tasks, test if built-in Elastic Search / SOLR models work well. These products have multiple models that can be tuned with configuration. With these models, you provide a document and find other documents that are "similar".

Some of the available algorithms are :

Cosine Similarity
Okapi_BM25
DFR
LM Jelinek Mercer

Docs :

https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-similarity.html

https://lucene.apache.org/solr/guide/6_6/morelikethis.html


Neural networks: which cost function to use? I am using TensorFlow for experiments mainly with neural networks. Although I have done quite some experiments (XOR-Problem, MNIST, some Regression stuff, ...) now, I struggle with choosing the "correct" cost function for specific problems because overall I could be considered a beginner.

Before coming to TensorFlow I coded some fully-connected MLPs and some recurrent networks on my own with Python and NumPy but mostly I had problems where a simple squared error and a simple gradient descient was sufficient.

However, since TensorFlow offers quite a lot of cost functions itself as well as building custom cost functions, I would like to know if there is some kind of tutorial maybe specifically for cost functions on neural networks? (I've already done like half of the official TensorFlow tutorials but they're not really explaining why specific cost functions or learners are used for specific problems - at least not for beginners)

To give some examples:

cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(y_output, y_train))

I guess it applies the softmax function on both inputs so that the sum of one vector equals 1. But what exactly is cross entropy with logits? I thought it sums up the values and calculates the cross entropy...so some metric measurement?! Wouldn't this be very much the same if I normalize the output, sum it up and take the squared error?
Additionally, why is this used e.g. for MNIST (or even much harder problems)? When I want to classify like 10 or maybe even 1000 classes, doesn't summing up the values completely destroy any information about which class actually was the output?

cost = tf.nn.l2_loss(vector)

What is this for? I thought l2 loss is pretty much the squared error but TensorFlow's API tells that it's input is just one tensor. Doesn't get the idea at all?!

Besides I saw this for cross entropy pretty often:

cross_entropy = -tf.reduce_sum(y_train * tf.log(y_output))

...but why is this used? Isn't the loss in cross entropy mathematically this:

-1/n * sum(y_train * log(y_output) + (1 - y_train) * log(1 - y_output))

Where is the (1 - y_train) * log(1 - y_output) part in most TensorFlow examples? Isn't it missing?

Answers: I know this question is quite open, but I do not expect to get like 10 pages with every single problem/cost function listed in detail. I just need a short summary about when to use which cost function (in general or in TensorFlow, doesn't matter much to me) and some explanation about this topic. And/or some source(s) for beginners ;)

This answer is on the general side of cost functions, not related to TensorFlow, and will mostly address the "some explanation about this topic" part of your question.

In most examples/tutorial I followed, the cost function used was somewhat arbitrary. The point was more to introduce the reader to a specific method, not to the cost function specifically. It should not stop you to follow the tutorial to be familiar with the tools, but my answer should help you on how to choose the cost function for your own problems.

If you want answers regarding Cross-Entropy, Logit, L2 norms, or anything specific, I advise you to post multiple, more specific questions. This will increase the probability that someone with specific knowledge will see your question.

Choosing the right cost function for achieving the desired result is a critical point of machine learning problems. The basic approach, if you do not know exactly what you want out of your method, is to use Mean Square Error (Wikipedia) for regression problems and Percentage of error for classification problems. However, if you want good results out of your method, you need to define good, and thus define the adequate cost function. This comes from both domain knowledge (what is your data, what are you trying to achieve), and knowledge of the tools at your disposal.

I do not believe I can guide you through the cost functions already implemented in TensorFlow, as I have very little knowledge of the tool, but I can give you an example on how to write and assess different cost functions.

To illustrate the various differences between cost functions, let us use the example of the binary classification problem, where we want, for each sample $x_n$, the class $f(x_n) \in \{0,1\}$.

Starting with computational properties; how two functions measuring the "same thing" could lead to different results. Take the following, simple cost function; the percentage of error. If you have $N$ samples, $f(y_n)$ is the predicted class and $y_n$ the true class, you want to minimize

$\frac{1}{N} \sum_n \left\{
\begin{array}{ll}
1 & \text{ if } f(x_n) \not= y_n\\
0 & \text{ otherwise}\\
\end{array} \right. = \sum_n y_n[1-f(x_n)] + [1-y_n]f(x_n)$.

This cost function has the benefit of being easily interpretable. However, it is not smooth; if you have only two samples, the function "jumps" from 0, to 0.5, to 1. This will lead to inconsistencies if you try to use gradient descent on this function. One way to avoid it is to change the cost function to use probabilities of assignment; $p(y_n = 1 | x_n)$. The function becomes

$\frac{1}{N} \sum_n y_n p(y_n = 0 | x_n) + (1 - y_n) p(y_n = 1 | x_n)$.

This function is smoother, and will work better with a gradient descent approach. You will get a 'finer' model. However, it has other problem; if you have a sample that is ambiguous, let say that you do not have enough information to say anything better than $p(y_n = 1 | x_n) = 0.5$. Then, using gradient descent on this cost function will lead to a model which increases this probability as much as possible, and thus, maybe, overfit.

Another problem of this function is that if $p(y_n = 1 | x_n) = 1$ while $y_n = 0$, you are certain to be right, but you are wrong. In order to avoid this issue, you can take the log of the probability, $\log p(y_n | x_n)$. As $\log(0) = \infty$ and $\log(1) = 0$, the following function does not have the problem described in the previous paragraph:

$\frac{1}{N} \sum_n y_n \log p(y_n = 0 | x_n) + (1 - y_n) \log p(y_n = 1 | x_n)$.

This should illustrate that in order to optimize the same thing, the percentage of error, different definitions might yield different results if they are easier to make sense of, computationally.

It is possible for cost functions $A$ and $B$ to measure the same concept, but $A$ might lead your method to better results than $B$.

Now let see how different costs function can measure different concepts. In the context of information retrieval, as in google search (if we ignore ranking), we want the returned results to

have high precision, not return irrelevant information
have high recall, return as much relevant results as possible
Precision and Recall (Wikipedia)

Note that if your algorithm returns everything, it will return every relevant result possible, and thus have high recall, but have very poor precision. On the other hand, if it returns only one element, the one that it is the most certain is relevant, it will have high precision but low recall.

In order to judge such algorithms, the common cost function is the $F$-score (Wikipedia). The common case is the $F_1$-score, which gives equal weight to precision and recall, but the general case it the $F_\beta$-score, and you can tweak $\beta$ to get

Higher recall, if you use $\beta > 1$
Higher precision, if you use $\beta < 1$.

In such scenario, choosing the cost function is choosing what trade-off your algorithm should do.

Another example that is often brought up is the case of medical diagnosis, you can choose a cost function that punishes more false negatives or false positives depending on what is preferable:

More healthy people being classified as sick (But then, we might treat healthy people, which is costly and might hurt them if they are actually not sick)
More sick people being classified as healthy (But then, they might die without treatment)

In conclusion, defining the cost function is defining the goal of your algorithm. The algorithm defines how to get there.

Side note: Some cost functions have nice algorithm ways to get to their goals. For example, a nice way to the minimum of the Hinge loss (Wikipedia) exists, by solving the dual problem in SVM (Wikipedia)


Why does "Depth = Semantic representation" in convolutional neural networks? I was watching some videos online about convolutional networks, and the speaker was discussing the concept of running a filter over an image.

He said, and it is also shown in the image below, that "Depth = Semantic representation".

This sounds nice, but I'm not sure exactly what is so special about reducing the spatial dimension of the image in favor of increased depth? What makes depth more important than any other dimension? A dimension is just dimension, right?

EDIT: I have a hunch that the reason is going to have something to do with translation independence...

The point of the layer depth and gradual pyramidal reduction is to build up a hierarchy of spatially invariant representations, each more complex than those of the prior levels. For example, at the lowest level, a convolutional may be able to pick out noteworthy arrangements of pixels; at the next level, it can condense these into particular spots, basic shapes, edges, etc.; then at higher levels it can recognize increasingly large and complex objects.  I'll borrow an example from Gerod M. Bonhoff's thesis1 on Hawkins' Hierarchical Temporal Memory (HTM), which is a closely related concept, one that also makes use of receptive regions to build invariant representations. At higher levels, the filtering process allows a convolutional or HTM to assemble individual lines and shapes into objects like "dog's tail" or "dog's head"; at the next stage they can be recognized as a "dog" or perhaps a particular variant, like "German shepherd."

This is made possible not only by the stacking of multiple layers, but the divisions of neurons within them into separate receptive regions. The receptive regions mimic actual neuronal "cell assemblies" and cortical columns that learn to fire together in groups; this enables clustering around particular types of objects, while the additional layers allow them to be related together into objects of increasing sophistication. The decrease in spatial dimensions in the example you cited reflects the narrowing of the receptive regions as we move up the pyramid; the third dimension (i.e. depth within a layer, as opposed to the depth of the layers) increases in tandem so that we can provide a wider choice of spatially invariant representations to select from at each stage i.e., each filter in the output volume depth-dimension learns to look at something different. If we simply narrowed the pyramid at each stage along every dimension, eventually we'd be left with just a narrow range of objects to choose from; taken far enough, it might just leave us with a single node at the top reflecting a single yes-no choice between "is this a dog or not?" This more flexible design allows us to choose more combinations of the previous layer's spatially invariant representations. I believe that this also enables a convolutional net to take into account various orientation problems, including translation independence, by adding more cell assemblies/columns to deal with each reorientation of an invariant representation.

As this excellent tutorial at github explains,

  First, the depth of the output volume is a hyperparameter: it corresponds to the number of filters we would like to use, each
  learning to look for something different in the input. For example, if
  the first Convolutional Layer takes as input the raw image, then
  different neurons along the depth dimension may activate in presence
  of various oriented edged, or blobs of color. We will refer to a set
  of neurons that are all looking at the same region of the input as a
  depth column (some people also prefer the term fibre).

This type of design is inspired by various biologically plausible structures found in actual organisms, such as the eyes of cats. If what I've said here isn't clear enough to answer your question, I can add a lot of further detail, including more examples, some based on actual organs of that kind.

1See pp. 26-27, 36 76 Bonhoff, Gerod M., Using Hierarchical Temporal Memory for Detecting Anomalous Network Activity. Thesis delivered March, 2008 to the faculty of the Air Force Institute of Technology at Wright-Patterson Air Force Base, Ohio.


Weka class attribute suggestion We are trying to run J48 on a classified data set. Our class attribute has two possible values ( 0,1) when running J48 the tree terminates at the very first node and doesnt process any further.

Instead of considering (0- false) as the starting point of J48. How can we consider running J48 by selecting (1-true) as the starting point of the tree?

Any suggestion will be greatly appreciated.

I am assuming that you have only 1 attribute (numeric).
What you can do is

modify the .arff file and set 0s to 1 and 1s to 0.
Then you can again run j48 classifier and visualise the decision tree.

It would give you inverted result.
Also, try watching this video.WEKA on MOOC


Question About Discriminator of CycleGan The Discriminator of CycleGan outputs not just a single value to say that the image is real or fake.... But It outputs a grid of numbers (like 8X8 or 7x7), where each number says whether one patch of the input image is fake or real.
So, my question is that, why do we do this.. What benefits does it gives us, and what was the problem in the approach where we only outputted a single value?

This is explained in the original CycleGAN paper:

For the discriminator networks we use 70 × 70 PatchGANs, which aim to classify whether 70 × 70 overlapping image patches are real or fake. Such a patch-level discriminator architecture has fewer parameters than a full-image discriminator and can work on arbitrarily sized images in a fully convolutional fashion.

Therefore, the arguments for using patch-based discriminators are that:

It has fewer parameters.
It enables using them in arbitrarily sized images.


How to extract information from plot images? Are there any free tools or libraries that can understand a plot image file automatically? Things like type detection (line, bar, scatter), as well as label and axis scaling detection.

In the tools I have found so far, one has to do manual configuration. I'm asking if it is possible in our AI driven world to let a machine do this.

If there are none, I will start a project for myself as it seems like a challenging but instructive AI task. Is this effort doomed from start? Are there any helpful libraries in C++ or Java and theoretical algorithms one could consider?

Thanks,
Tomas

In the general sense this is most definitely a non-trivial problem and any full featured solution will require a considerable degree of work.

Success will probably be dependent on how much you constrain your problem.
Monochrome will be easier than color.
Limited font, font size, line and marker styles will help.

The more constrained, the more chance of success.

For line detection, such as axes, you might want to consider the Hough and Radon transforms.

Here are some stackexchange questions covering some of the topics you might want to address, sadly no bargraphs, some have a manual component which you might think how to automate. They are a good starting point to review some of the difficulties involved in the problem.

Selecting a curve:
https://mathematica.stackexchange.com/questions/44355/how-to-make-a-curve-selectable-from-a-scaned-image-and-convert-it-to-a-list-of-c

Datapoints and Axes:
https://mathematica.stackexchange.com/questions/1524/recovering-data-points-from-an-image

Scatter:
https://mathematica.stackexchange.com/questions/26356/how-do-i-find-the-coordinates-of-points-in-this-image

Datapoints:
https://mathematica.stackexchange.com/questions/3831/how-can-i-extract-data-points-from-a-black-and-white-image

https://mathematica.stackexchange.com/questions/14444/obtain-data-points-from-a-graph-on-an-image-without-axes


Difference between Dueling DQN and Double DQN? I have read some articles, but still can not figure out the difference between the Dueling DQN and Double DQN? What exactly is the difference between them? Also, Does Dueling DQN need to be built on top of a Double DQN? Thanks!

They are separate techniques.
Double uses two networks to avoid over optimistic q-values.
Dueling separates the estimator using two new streams, value and advantage. The two streams are then aggregated.


sklearn stratifying dataset with more than one label I have come across sklearn.model_selection.train_test_split as a method to split up the train and testing dataset.

Furthermore they have a stratify which you can set to the labels to ensure your train and test data set have approximately the same proportion of labels of each type.

My question is what if you have more than one type of label for your data, so that your labels are not a size len(dataset) x 1 but a size of len(dataset) x m where m is the number of your label types. What tool can you use to stratify (and split) your dataset in the multilabel case.

Looks like you have a set of "dummy" variables which were previously the same variable.

What you could do is to return to the original variable, get the stratified data, and then rerun the one-hot encoding to have the variables again.

Remember: If your dataset is very balanced, your results with and without stratification will be almost the same.


How does MITIE perform named entity recognition? I'm trying to use MITIE to extract named entities from short text. I'm interested in entities such as dates, times, names, and locations. Out of the box, MITIE only recognises names, locations, and organisations. I'd like to train it to recognise dates, times and other categories as well. From looking at the structure of MITIE's directories and from the dlib website, I gather that this is done via an SVM. Is this correct?

With regards to adding new categories to the named entity recogniser, I have several questions:

Can this be done in an augmentative fashion? That is, given an existing NER system, can I just add categories examples and train it to recognise those as well? Or do I need to train models from scratch?
If I do need train models from scratch, what dataset can I use to do this?
Related to adding new examples, is there an online method that I can use, feeding the system new examples and categories as and when I need to?

After having used MITIE for a few weeks, I feel like I at least have enough to answer my basic questions:

(and 3.) All models need to be trained from scratch - there is no online method to add new samples to the model as they come in. This is unfortunate because MITIE takes at least 45 minutes to an hour to train on a ~20k-sized dataset.
The datasets I used were ATIS, CoNLL 2003, and DBpedia

I've found MITIE to be quite good as far as classification accuracy goes, although it takes a bit of work to prepare datasets for it.


What are the alternatives to Python + Spark (pyspark)? I like Python, and I like Spark, but they don't fit very well together.  In particular,

it is very hard to use python functions in spark (have to create JVM binding for function in python)
it is hard to debug pyspark, with py4j in the middle

So I wonder if there are any alternatives to pyspark that supports python natively instead of via an adapter layer?
Reference

Try checking dask. It's a distributed library which is native with Python and it uses pandas and numpy. So its like using pandas with some wrapper for distribution computation.


Gaussian Mixture Models Clustering When using the EM algorithm in Gaussian Mixture Models (GMM), in the E-step, we take each x set in the training dataset to calculate and update the "weight" and parameters of each Gaussian distribution of the clusters (M-step). I have read that we do this until it converges. I am a little confused here. Does that mean it loops through the whole training dataset X every time in "one step" of the EM algorithm? Or is "one step" corresponding to calculating ONE particular x set in the whole dataset and using it to update the parameters and weights of each distribution?

To make the question more clear, does each "step" in the EM algorithm in GMMs involve the whole training dataset X or does each step mean one particular x set in the training set, and we loop over the dataset one at a time to improve one at a time?

Any help will be appreciated! Thanks.

At the beginning of the algorithm, you are taking an initial guess for the relevant parameters. Then in each iteration, in the E-step you are computing the responsibilities for all the given datapoints and in the M-step you are computing the weighted means and variances.

Since you are clustering the datapoints there is not training or test set. You just want to divide your dataset to two (or more) clusters.


Dataset to be used to learn NN regressions in classroom experiments Can you suggest me datasets with intermediate size of about 1000-10000 to work with neural network regressions in classroom experiments?
It seems that there are much more datasets for classification tasks than for regression tasks.
I know two good datasets:

Boston Housing dataset

Auto MPG Data Set

But they actually seem to be very small. On the other hand, I know other datasets such as Zillow price but they seem to be very large to explore different types of NN regressions in classroom experiments.

May try these two Kaggle datasets -

camnugent/california-housing-prices
kumarajarshi/life-expectancy-who

Try your question here - https://opendata.stackexchange.com/


How to measure a relationship between interests of topics between two users? I was trying to find a way to calculate the relationship between two Users and their interests. A user can have multiples interests, and an interest can be duplicated in a user's interests. Example:

User1: {
   interests: {"golf": 3, "baseball: 1", "tenis: 2"}
}
 User2: {
   interests: {"golf": 5, "baseball": 1, "football": 4}
}
 User3: {
   interests: {"golf": 1, "baseball": 1, "tenis": 1}
}

Comparing User1 & User2:

They have in common 2 interests, but User2 seems to be a very fan of golf.

Comparing User1 & User2:

They have in common 3 interests, while User3 seems to have a slightly interest in all common interests, not being a very fan of any interests listed.

What I want do to is

The relationship between User1 and User2 should be a higher number than the relationship between User1 and User3, knowing that User3 has only a superficial interest in the topics, and User2 is more likely to be a fan like User1 to golf, for example, meaning that they may want to know each other and etc.

e.g. User1&User2  = 2.5; User1&User3 = 1.8

I tried to use the mean of the intersection of all interests between User-k and User-y and it seemed a reasonable approach, but I think it can have a better way to do this. Suggestions?

Thanks!

You can create a sparse user-interest matrix, impute missing values with some reasonable value, and apply a similarity function. E.g., center all non-missing values on 0 and mutate each value to adjust for the user's and interest's mean. Then impute 0s for all missing data, and use a traditional similarity function like cosine.

You might also use Bayesian methods. You could model the relationship between users and interests. In this way you wouldn't have to impute missing data and it would account for confidence in inference and prediction.


clipping the reward for adam optimizer in keras I would like to clip the reward in keras. I saw it is possible to clip the norm and clip the value is sgd as follows:

sgd = optimizers.SGD(lr=0.01, clipnorm=1.)
sgd = optimizers.SGD(lr=0.01, clipvalue=0.5)

What are clipping the norm and clipping the value?

Also, How it is possible to implement the clipping the reward for Adam? Would you please let me know how I can do this?

Gradient clipping takes two main forms in Keras: gradient norm scaling (clipnorm) and gradient value clipping (clipvalue).

1. Gradient Norm Scaling

  Gradient norm scaling involves changing the derivatives of the loss function to have a given vector norm when the L2 vector norm (sum of the squared values) of the gradient vector exceeds a threshold value.
  For example, we could specify a norm of 1.0, meaning that if the vector norm for a gradient exceeds 1.0, then the values in the vector will be rescaled so that the norm of the vector equals 1.0.

2. Gradient Value Clipping

  Gradient value clipping involves clipping the derivatives of the loss function to have a given value if a gradient value is less than a negative threshold or more than the positive threshold. For example, we could specify a norm of 0.5, meaning that if a gradient value was less than -0.5, it is set to -0.5 and if it is more than 0.5, then it will be set to 0.5.

3. Keras Syntax

The gradient clipping syntax for Adaptive Moment Estimation (Adam) is very simple and follows the same syntax as for Stochastic Gradient Descent (SGD) shown above:

opt_adam = optimizers.adam(clipnorm=1.)
opt_adam = optimizers.adam(clipvalue=0.5)


How is the property in eq 15 obtained for Xavier initialization I am new in this field so please be gentle with terminology.
In the original paper; "Understanding the difficulty of training deep feedforward neural networks", I dont understand how equation 15 is obtained, it states that giving eq 1 :

$$
W_{ij} \sim U\left[−\frac{1}{\sqrt{n}},\frac{1}{\sqrt{n}}\right]
$$

it  gives rise to variance with the following property:

$$
n*Var[W]=1/3
$$

where $n$ is the size of the layer.

How is this last equation(15) obtained?

Thanks!!

The fixed answer of $1/3$ is a result of their decision to use the uniform distribution along with the parameterised arguments, namely $1/\sqrt{n}$.

For a uniform distribution, denoted with lower and upper bounds $a$ and $b$:

$$
U(a, b)
$$

the variance is defined as:

$$
\frac{1}{12} (b - a)^2
$$

So in the case of the authors, Glorot and Bengio, the two bounds are simply the square root of the number of neurons in the layer of interest (generally referring to the preceding layer, as they put it). This size is called $n$, and they set the bounds on the uniform distribution as:

$$
a = - \frac{1}{\sqrt{n}}
$$
$$
b = \frac{1}{\sqrt{n}}
$$

So if we plug these values into equation 15, we get:

$$
Var = \frac{1}{12}(\frac{1}{\sqrt{n}} - -\frac{1}{\sqrt{n}})^2
$$

$$
Var = \frac{1}{12}(\frac{2}{\sqrt{n}})^2
$$

$$
Var = \frac{1}{12} * \frac{4}{n}
$$

And so finally:

$$
n * Var = \frac{1}{3}
$$


What is the form of data used for prediction with generalized stacking ensemble? I am very confused as to how training data is split and on what data level 0 predictions are made when using generalized stacking. This question is similar to mine, but the answer is not sufficiently clear:

How predictions of level 1 models become training set of a new model in stacked generalization.

My understanding is that the training set is split, base models trained on one split, and predictions are made on the other. These predictions now become features of a new dataset. One column for each model's prediction, plus a column that contains the ground truth for these predictions.

Split training data into train/test.
Train base models on training split.
Make predictions on test split (according to linked answer, use k-fold CV for this).
Create a feature for each model, filling it with that model's predictions
Create a feature for the ground truth of those predictions.
Create a new model and train it on these predictions and ground truth features.

Question 1: Are these the only features used to train the "meta" model? In other words, are none of the actual features of the original data included? The linked answer says it is common to include the original data, but I have not read about it elsewhere.

Question 2: If the above algorithm is correct, What is the form of the data when making predictions? It seems like it would also have to have predictions as independent variables. If so, that means running all new incoming data through all base models again, right?

Question 3: I keep seeing an "out-of-fold" requirement for the first level predictions. It seems that doing a simple train/test split as mentioned above would fulfill this. However, would you not want a 3rd split to test the combined model's generalization? Or is this type of ensemble bulletproof enough not to worry about it?

Q1.  This can be done either way.  You may use only the base model predictions, or those and all the original features, or anywhere in between.  Passing along the original features may be known as "feature-weighted stacking," the idea being that the meta-estimator can learn that some of the base models are better on certain subsets of the original data (but AIUI the original approach there is to select/engineer more useful subsetting features, not to simply pass along all the original features; I assume this depends on context).

For example, the parameter passthrough in sklearn's StackingClassifier passes the original dataset along, as does use.feat in mlr's makeStackedLearner.

Q2.  Absolutely, making predictions requires each of the base models making their predictions (this can be done in parallel), then passing those off to the meta-estimator.

Q3.  Yes, a simple split suffices to prevent the data leak that occurs in more naive stacking, where the meta-estimator is fitted using the "predictions" of the base models on their own train set.  And yes, you will need another test set for evaluating the performance of the ensemble.  k-fold cross-validation is often recommended for the base models so that more data is available for all the models to train on.


Feature Interactions vs Feature Importances Can somebody explain in-detailed differences between Feature Interactions and Feature Importances? And their applications?

The most intuitive way to explain it I found it on this Nature: Explainable AI for Trees: From Local Explanations to Global Understanding:
Feature Importance: The contribution of a feature to the final prediction. If it was a linear regression, this will be the coefficient. For a decision tree you have mean decreased accuracy or as the paper says, shapley values. From the shapley github

Feature Interaction:  A model makes a prediction based on two features, we can decompose the prediction into four terms: a constant term, a term for the first feature, a term for the second feature and a term for the interaction between the two features. The interaction between two features is the change in the prediction that occurs by varying the features after considering the individual feature effects.
Lets say our model is like:
y = a + b·x_1 + c·x_2 + d·x_1·x_2

The feature importance terms will be b, and c.
The feature interaction will be the product x_1 · x_2
So one is the coefficient and the other one the relationship between the two. In the following graph, you can see the relationship between Age and Sex changes when the Age increases value.


Which courses should I take in order to learn ML and AI? I want to learn ML and AI so I want to know which courses(It would be nice if the courses were free) should I take in order to do that and in what order should I learn them. I want the courses to provide a deep understanding of the concepts used.

In my opinion, start with:

Coursera's Andrew Ng's classes

Machine Learning (very good for the maths)
then go to his deep learning class (I haven't done it but I only hear good things from it)

Udacity had a very good Data Science Specialisation divided in 4 classes. I don't know if it's still available (I did it two years ago) but it is probably either still there or an improved version has replaced it. The 4 courses where:

Intro to Data science
Visualisation
Machine Learning
Text Mining

For the rest, I suggest you to do a quick google search...