Get long answers from BERT We are using Google BERT for question and answering. We are using vanialla bert-base-uncased as well as squad trained checkpoints.

The answers from BERT are very short and crisp. For example, if we ask describe a chatbot, then it will simply return, takes input from user and replies ...  Though the answer actually is complete paragraph.

Will training BERT on long paragraphs can solve this problem? If yes, any idea from where we can get such a QnA dataset, as it is not possible to create a huge QnA dataset manually.
Is there any other tweak which can be done at some BERT layer, so that it starts understanding a long answer?
Or is there any other framework or system already have solved this kind of problem, by maybe integrating with some other neural network technique, or by using a pipeline of multiple components?

You can check quora for your QnA dataset.

As far as long answers are concerned it may not be because of BERT as  SQuADhas very crisp and short answers for most of its questions.

BERT already does understand long term dependencies in your data(one of reasons for using deeply bidirectional tranformer).


How to do batch inner product in Tensorflow? I have two tensor a:[batch_size, dim]  b:[batch_size, dim].
I want to do inner product for every pair in the batch, generating c:[batch_size, 1], where c[i,0]=a[i,:].T*b[i,:]. How?

There is no native .dot_product method. However, a dot product between two vectors is just element-wise multiply summed, so the following example works:

import tensorflow as tf

# Arbitrarity, we'll use placeholders and allow batch size to vary,
# but fix vector dimensions.
# You can change this as you see fit
a = tf.placeholder(tf.float32, shape=(None, 3))
b = tf.placeholder(tf.float32, shape=(None, 3))

c = tf.reduce_sum( tf.multiply( a, b ), 1, keep_dims=True )

with tf.Session() as session:
    print( c.eval(
        feed_dict={ a: [[1,2,3],[4,5,6]], b: [[2,3,4],[5,6,7]] }
    ) )

The output is:

[[ 20.]
 [ 92.]]


What ML algorithm can I use for building a "recommended" list for players? We have a large touchscreen kiosk in local malls where people can go up to it and play a game under different categories. We want to implement ML to build a recommended list of games for each player. For now, let's assume there are 5 different games under each of the 4 categories and the games can be either available or unavailable at any given time. Also, there is a table containing a log of each play and a table of all the games.

Ideally, we would want to pass 2 different data sets; example: a list of all the plays in the last 30 days with any necessary data, and a list of all the currently available games to play with the necessary "relatable" data. Out of this, we would then end up with a list of the top 10 recommended games for each player based on all that data. There will be more data than just the game's category to tie it together to the player/person, but I want to get a simple demo working first.

I am guessing I need to look into some type of recommendation system but I'm not sure which would best to try out. For now, we are looking at using Python; AWS's ML tool will be too costly. I am open to recommendations on other languages/tools.

Before jumping into machine learning solutions, it would be good to think more about the problem you're solving. If there are only 20 games and some are unavailable at any given time, then a well laid-out menu with good navigation is superior to a recommender system. Recommender systems are only appropriate when people cannot adequately parse all of the available options.

If you do want personalized recommendations, you don't even have to start with machine learning models. You can simply recommend that players keep playing the same games or the most popular games.

And if it turns out that machine learned models are best, I suggest looking at association rule mining based on unary data (which gives you shopping-basket recommendations: people who played games A, B, and C also played games D and E) or some variety of collaborative filtering based on ratings data (which gives you a user-item preference space). That totally depends on what sort of feedback you get from users about their individual game experiences.


How to improve LSTM accuracy on multiclass text classification? So, I'm trying to build a LSTM model to classify multiclass text label. The goal is to make a prediction about user rating (1, 2, 3, 4, 5) based on their review.
My hyperparameter is like this:
# set hyperparameter
vocab_size = 5000 # make the top list of words (common words)
embedding_dim = 32
max_length = 354
oov_tok = '<OOV>' # OOV = Out of Vocabulary

Model:
model = Sequential()
model.add(Embedding(vocab_size, embedding_dim))
model.add(Dropout(0.5))
model.add(Bidirectional(LSTM(embedding_dim)))
model.add(Dense(1, activation='softmax'))
model.summary()

Compile:
model.compile(
    loss='categorical_crossentropy',
    optimizer='Adam',
    metrics=['accuracy']
)

Then, the result I get:

I only get 50% accuracy and I think it's not a good result. Any idea how can I improve the accuracy? Here's the collab: Collab's

50% is quite decent because you have five labels and random guessing model would have achieved only 20% accuracy. So you know your model is learning something.
The other thing you want to check out is whether this is suited to be a regression problem more than classification. For e.g, misclassifying a 5 (ground truth) into a 4 is better than misclassifying into a 2. Your current loss function treats both situations as equally bad. So, it's possible that your current accuracy measure is underestimating the models true performance.
You should also try out simple naive bayes model on top of bag of words. It will be interesting to compare how that compares with the more complex lstm.


Without knowing more about the training data it's impossible to explain the prediction results properly. One way to interpret the results is that the model is producing a probability distribution, over 3 classes, for each image you feed. This would be the case if your training data consists of one-hot vectors where exactly one element is 1 and the other two are 0.

The output of the model is your final layer after the final activation function is applied, so the prediction has the same shape of your dense_2 final layer. The None in the first dimension indicates that the size of that dimension at runtime is dynamic, and in this case the first dimension represents batch elements. Hence the output is an array of shape [batch_size, 3] and you have one 3-element vector for each of the 4 example images fed into your network.

In your comment you indicated that your final activation is the softmax function, which means that each 3-element vector can represent a probability distribution over 3 possible outcomes (notice that each vector sums to one: by definition softmax normalizes its output).

If you're trying to predict classes for each image, which appears to be the case, the predicted class label integer would be the index of the 3-element vector that has the highest probability. Typically these integers represent something semantically relevant to your application.

With a softmax output you still have to apply a decision rule on the output to actually retrieve the predicted class. For example, if your possible classes are ["dog", "cat", "mouse"] and your example image outputs a vector [.06, .9, .04], then you could choose the "predicted class" to be "cat". Or you could incorporate the predicted probability distribution into a more elaborate probabilistic model.


training neural net with multiple sets of time-series data I have the following data

($x^1_i$, $y^1_i$) for $i=1,2,...N_1$

($x^2_i$, $y^2_i$) for $i=1,2,...N_2$

...

($x^m_i$, $y^m_i$) for $i=1,2,...N_m$

Is it possible to train a neural net to produce some $y_k$ where $k<=min(N)$ given a input ${x_1, x_2, ..., x_{k-1}}$?

If so any suggestion of documentation/ library I can look at (preferably python)?

Yes, this is a straightforward application for neural networks.  In this case yk are the outputs of the last layer ("classifier"); xk is a feature vector and yk is what it gets classified into.  For simplicity prepare your data so that N is the same for all.  The problem you have is perhaps that in the case of time series you won't have enough data: you need (ideally) many 1000's of examples to train a network, which in this case means time series, not points.  Look at the specialized literature on neural networks for time series prediction for ideas on network architecture.

Library: try Pylearn2 at http://deeplearning.net/software/pylearn2/  It's not the only good option but it should serve you well.


Computing confidence interval for average from individual predictions In the attached image, I am plotting actual vs predicted along with confidence intervals for each of the predicted points. The black star represents the average of all the predicted points. What formula can I use to combine the confidence intervals of all the predicted data points to get a confidence interval for the average? Also, please assume that the average could be a weighted average.

I am using python, numpy.

OK, we have two dozen predictions-with-uncertainty, and actual point values. I'm still not sure we know enough to answer, so I'm going to just outline some points and hope someone can improve upon this. A quick search found three similar StackExchange questions:

Combining 13 independent regressions seems the most similar -- Note comment warning this might be meaningless. The best-answer suggests a sampling approach because the result has no closed form. Using a Normal approximation would have a closed form, but I think your plot suggests the errors are not Normal, or your 95% CI is much too narrow.
Combining 2 -- See esp. the reply discussing meta-analysis and whether you have a fixed effects or a random effects model. This may be what you are doing, in which case re-formulating the question for that community may be best.
Delta Method -- Seems to me the same question, but takes a direct maximization approach, assuming approximations hold.

A couple of observations suggesting any simple average is not going to give you what you want, and you need to be very clear what is being combined, and what question the result is going to answer:

If these are 95% Confidence Intervals, then your error model is much too conservative. Nearly all of them should include the true value (gray dashed diagonal) but only about 5 do. Therefore any parametric combination will be at least as unreliable. But suppose you have a black box with uncalibrated estimates, and you just want to know the "average" from that black box, however uncalibrated.
If these are two-dozen unrelated estimates from a crowdsourcing platform, the "average" is not meaningful. On the other hand, if they are performance measures of the crowd, then the average is something like the average performance of the crowd across all question types. In that case:

1st approximation would just average the two-dozen points and use the standard deviation of that. (Each point can be weighted.)
2nd approximation would treat each estimate and CI as Normal, and use one of the formulas for combining Normal CIs. Note however that it seems error bars are wider towards the edges. That may require special handling.
3rd approximation: sample say 100 forecasts from the two-dozen distributions and then combine those ~2400 samples. (Weighting can be done by varying sample size for each forecast.) Works even if the distributions are not assumed to be Normal.

If instead you have two-dozen forecasts from a single statistical model on different input data, and you want to know the average forecast across all input combinations, don't combine them. Just run the model with no factors specified, and use that result.

Hopefully a bona fide statistician can supply a more definitive answer.


Why the performance of VGG-16 is better than Inception V3? I’ve initialised VGG16 and InceptionV3 with ImageNet weights and fine tuned using very small learning rate from the first layer on a chest X-ray dataset of 20K images (since the medical images are different from ImageNet classes). I reportedly find the VGG16 shows no overfitting and is more accurate than InceptionV3 that overfits and gives less accurate results. What is the reason?

What are the cons?
It has so many weight parameters, the models are very heavy, 550 MB + of weight size.
Which also means long inference time
Why not just make the model deeper?
More heavier model
More training time
Vanishing gradient problem

With a given receptive field(the effective area size of input image on which output depends), multiple stacked smaller size kernel is better than the one with a larger size kernel because multiple non-linear layers increases the depth of the network which enables it to learn more complex features, and that too at a lower cost. 3X3 kernels help in retaining finer level properties of the image.


Time Series - Prediction I have time series for product usage over an year on daily basis. Product usage exhibits seasonality i.e. it usage increase/decreases by more than the normal usage during that time. When i get the current day's data and its more than normal, i want to check if its exhibits seasonality or not and take some actions accordingly. I am thinking of using Double Exponential Smoothing to achieve this. Is this an right approach or is there an better approach to do it.

A useful introductory reference is Forecasting: Principles and Practice by Hyndman and Athanasopoulos. Depending on your data, you might be able to build a simple additive or multiplicative model with seasonal, trend, and cyclic components.

Besides exponential smoothing, you can look at ARIMA models. Exponential smoothing is based on trend and seasonality while ARIMA is based on autocorrelations in your data.


Dividing a dataset to parallelize machine learning training on the cloud I'm very new to machine learning. I am doing a project for a subject called parallel and distributed computing, in which we have to speed up a heavy computation using parallelism or distributed computing. My idea was to have a dataset divided in equal parts, and for each subset to have a neural network to be trained on a separate machine in the cloud. Once the models are trained, they would be returned back to me and somehow combined into a single model. I am aware of federated learning but it doesn't quite fit my scenario of actually sending and dividing the dataset into the cloud. Does someone know any feasible approaches (maybe a variant of federated learning) of how one would do this?

There are many ways to parallelism machine learning. It is often better to distribute the model parameters, not the data.
Training models only a subset of data will result in worse parameter estimates than training a model on random samples of the data.
Additionally, moving data around is more expensive than moving parameters.


Practical problems in anomaly detection where the number of normal data is extremely high compared to abnormal data If the ratio of abnormal data is about 1 to 10,000 normal data, even if the true negative rate is 99%, there will be 100 false positive data, and the precision( TP/(TP+FP) ) will be low.
If this kind of anomaly detection is to be put to practical use, I think it is necessary to create a model with a fairly high prediction accuracy.
How do the actual examples of anomaly detection in the world deal with this problem?
Is it difficult to put anomaly detection to practical use for difficult problems that do not allow for high prediction accuracy, or for problems where the number of anomalies is too small?

The presumption I feel you might be making is that a Machine Learning model is required for this task, but to be honest statistical approaches and less fashionable algorithms are available.
The reality is training a ML model on  your data is going to be a struggle because of the significant class imbalance and low volume of anomalies. That is why pretrained models are attractive, in particular Neural Networks.
To get a feel of the range of options, I find the Python library pyOD a great resource. It has about 30 different ways to detect outliers.


sklearn ColumnTransformer creates new columns in output when there are overlapping columns between steps I need to process some dataframe columns in different steps using ColumnTransformer.
The first step process the date columns (timestamp) imputing missing values and the second step applies scaling to all the numeric columns (including the dates columns). In output I get a number of columns which is the sum of the numeric columns and the dates columns, but the dates columns are a subset of the numeric columns so this is not correct.

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

dates_columns = ['ts_1', 'ts_2']
numeric_columns = ['ts_1', 'ts_2', 'n_1', 'n_2']

column_transformer = ColumnTransformer([
    ('imputer_dates', SimpleImputer(strategy='median', missing_values=0), date_columns),
    ('scaler', StandardScaler(), numeric_columns)
])

X_transformed = column_transformer.fit_transform(X)
print(X_transformed.shape) # Got 6 columns, but it should be 4

How to fix this?

The description says-

".................the features generated by each transformer will be concatenated to form a single feature space"

Based on this I would not expect it to "reduce" the number of columns.

On top of my mind, another pipeline which computes on dates column and feeds its output to numeric column transformation in columnTransformation.


Converting nominal data to numeric - is using dictionaries the right approach? Currently I have a data-set with roughly 7 types of nominal data; these are things like "workclass" or "marital status". It is my understanding it is best to convert nominal data like this into numeric data. So, what I've done so far, as a simple experiment, is make up a dictionary for each nominal data type, so in our example of "marital status", it would be "Divorced":0, "Married":1, "Widowed":3, "Separated":4.

However, as I understand, there are some problems with this approach, as it implies that Divorced->Married are more related than Divorced->Widowed. I tried to arrange this data into a logical pattern, but for a lot of it its impossible. So here are my questions:

Is there a good approach to do this without running into the above problem, specifically in Python? I'm using Pandas DF's at the moment, and I saw an example online of something called Binary Vectorizers, but I'm not sure if that won't introduce too many attributes/columns to my data set.
When I convert a nominal value to an interval one, do I replace the  original (For example Workclass) with the Workclass_Number data? Do I only enter the number data into my models?
Are there any guidelines on the numbers that should be used? For example I used simple intervals of 1 for all of my data, starting at 0.

Usually, when dealing with nominal attributes, you want to use the binary vectoriser approach. Unless you have a very large number of nominal values, it typically doesn't matter that you're adding a few attributes. If you do have a huge amount of nominal values, you can try things like dimensionality reduction on the binary attributes (although this reduces interpretability) or joining some nominal values together.
Yes.
If the nominal attribute is discrete but ordered (called an ordinal attribute) -- for example, the severity of a cancer (stage 1, 2, 3 or 4) --
then you can use whatever numbers you feel are appropriate. It really depends on your domain knowledge, and the choice of model. For some models like decision trees, it doesn't matter, but for others like logistic regression the choices of the numbers can be influential. I would recommend just continuing as you have been by using intervals of 1, unless your knowledge of the domain says that particular adjacent nominal values should be further or closer than the others.


Using already trained classifiers in Orange I know that with Orange it is possible, using the 'Test & Score" widget, to test the performance of a classifier trained on a specific training set using a different test set ('Test on test data' option).

My question is: is it possible to use a training set to train a classifier, tune its hyperparameters through cross validation on a separate validation set, and, finally, test the classification performance on a third, separate set using the already trained classifier?

How this can be achieved?

A scheme like this should work. Namely, you pass training data directly into the the learner widget and pass its trained Model into Predictions widget along with labeled test data.


How to preprocess tensorflow imdb_review dataset I am using tensorflow imdb_review dataset, and I want to preprocess it using Tokenizer and pad_sequences
When I am using the Tokenizer instance and using the following code:
tokenizer=Tokenizer(num_words=100)
tokenizer.fit_on_texts(df['text'])
word_index = tokenizer.word_index
sequences=tokenizer.texts_to_sequences(df['text'])

print(word_index)
print(sequences)

I am getting the error
TypeError: a bytes-like object is required, not 'dict'

What I've tried
store dataset as dataframe and then iterate over the text column, and store it in a list, and then tokenize it.
df = tfds.as_dataframe(ds.take(4), info)
# list to store corpus
corpus = []
for sentences in df['text'].iteritems():
  corpus.append(sentences)

tokenizer=Tokenizer(num_words=100)
tokenizer.fit_on_texts(corpus)
word_index=tokenizer.word_index
print(word_index)

But i'm getting the error
AttributeError: 'tuple' object has no attribute 'lower'

How can I use the text column and preprocess it to feed it to my neural network?

For TypeErrors it is always good to check the exact type of the variable causing issues, especially when troubleshooting TensorFlow and Keras:
print(type(df['text']))

fit_on_texts expects a list of string or similar, but you are providing a dictionary, so you'll want to convert accordingly. Example:
from typing import List, Dict

foo_dict: Dict[str, str] = {"one": "first sentence", "two": "second sentence"}
foo_list: List[str] = list(foo_dict.values())

print(foo_dict)
print(type(foo_dict))

print(foo_list)
print(type(foo_list))

Output:
{'one': 'first sentence', 'two': 'second sentence'}
<class 'dict'>
['first sentence', 'second sentence']
<class 'list'>

API docs will help here:
https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/text/Tokenizer#fit_on_texts
I'm using type annotations in the example above which I find helps troubleshooting and avoiding bugs.


YOLOv3 Predicting bounding boxes for grid containing multiple centers I recently started learning about YOLO and object detection, and I am kind of stuck on something. I was wondering if someone could explain to me what happens when a grid cell contains the centers of more than one bounding box. Does the grid cell still predict a single bounding box?
In YOLOv3 the image is divided into several grid cells of size $13\times13$, $26\times26$ and $52\times52$ Each of these grid cells are responsible for predicting a bounding box if the center of the ground truth bounding box is in the grid cell.
I know that each grid cell predicts $B$ boxes, where $B$ is the number of anchor boxes. The dimensions of each of these predictions is $4+1+n$, where $n$ is the number of classes. $4$ for $b_x,b_y,b_w,b_h$, and 1 for the objectness score. $b_x,b_y,b_w,b_h$ is defined below:
$$
b_x = \sigma(t_x)+c_x \\
b_y = \sigma(t_y)+c_y \\
b_h = e^{t_h}p_h \\
b_w = e^{t_w}p_w
$$
$p_h$ and $p_w$ is the height and width of the anchor boxes. $t_x, t_y, t_h, t_w$ are predictions made from a single anchor box of a grid cell. My confusion is when a grid cell contains the centers of more than one bounding box, there are more than one values for $t_x, t_y, t_h, t_w$, but the description of the YOLOv3 training process or the above equations does not seem to take this into account.

It depends on the combination of thresholds of the objectness score and non-max suppresion (NMS):
Objectness score $C$
Apart from the location and dimensions predictions ($t_x, t_y, t_h, t_w$) for each box, an objectness score $C$ is also predicted for each of the boxes. It tells us how likely an object is inside  of a certain predicted bounding box (BB).
But How much it needs to be to ensure an object is inside?
This has not a precise answer, instead we use a certain threshold. For example, we could set that a BB contains an object if and only if the objectness $C>0.5$ (threshold = $0.5$). This value depends on the implmentation and it can be asked to the user as in this repo
Non-max suppresion (NMS)
Now that we know that there is an objectness score threshold, we can rephrase the question  in the following way:
What if two BBs of the same grid cell have an objectness score $>$ threshold? Would one of these two BBs be eliminated?
This depends on the threshold used in non-max suppression. If this concept is new to you, I recommend you this tutorial about it.
Broadly speaking it's used to discard bounding boxes that have a lower objectness score than the bounding box assigned to the same class that has the highest objectness score, and that have an overlap with it bigger than the threshold we are talking about now.
This means that higher thresholds used in NMS will lead to discarding a fewer number of boxes (we will be allowing a bigger overlapping between the boxes).
Conclusion
So, to sum up, In the situation of the question (more than one object per grid cell). Only boxes that have values of $C$ higher than its threshold will be considered. And out of these boxes, if there are some that are assigned to the same class then the box that have the highest $C$ will be considered and also the ones that don't get eliminated after applying non-max suppresion.


ggvis vs. ggplot2+Shiny; which one to choose for interactive visualization? There is a similar question here in CrossValidated, and I have read the answers. My question is a bit different. I don't want to merely visualize my data, and indeed what I want to visualize is not easy to visualize with either package.

I have two sets of points ($x,y$ coordinates) on my plot. I want to add edges and make it similar to a graph. If I have $n$ points, then based on the nature of my problem, I need to add $(n/2)^2$ edges to the plot.

Then I want to add some interactive features to my plot. For instance, by clicking on one point (vertex), all other vertices which are not connected to this clicked vertex must disappear to have a more lucid and focused plot.

I tried to find a network visualization package suited to my project, but it seems all of them use the data about connection of edges or nodes and provide a network based on some specific layout algorithm. My data is different. I have the coordinates of nodes and I want to establish some edges between some of them. In my data the nodes have their own fixed locations and must not be moved around.

Finally, I narrowed down the packages to ggvis and ggplot2. ggvis is new and fresh in addition to having intentional design to generate interactive plots. But I found out that ggplot2+Shiny can also produce interactive plots. (Is that correct?) Besides, I figured that ggplot2's capabilities are more varied and matured than ggvis: for instance ggplot2 has a zooming feature while ggvis does not have it. More important, as ggvis is growing and changing, I don't want to invest a lot of time on a work that becomes technically obsolete or buggy in a year or so. I thought this is the case if I code based on ggvis.

Now may you please evaluate my choice (ggplot2+shiny) and advise me about possibility to do what I want to do with it?

As I did not know any of the mentioned packages, I have started learning ggplot2 from Wickham's book and I love it! But I am afraid of spending a few weeks and finding out that this amazing package cannot do what I want to do even with the aid of Shiny.

P.S: I want to put the final code on github so probable users can download and run it. Therefore, I do not need to make a web application, there is no need to any website or putting the interactive map online. I just need to have an interactive plot which can be run from inside of Rstudio.

Well, many months have been passed from this help request. I write this answer to my own request in order to share my experience. I learned ggplot2 and then ggvis as well as Shiny. Shiny can work with both of them, but I found ggvis more structured and lucid comparing to ggplot2. Something which is expectable as the former is being developed based on the development experience of the latter. Moreover, ggvis seems much faster than ggplot2 in quick graph/chart generation, it seems much more suitable for 'Shiny' and creating an applet. However, the negative point is the lack of many features in ggvis since the package is under development and has not become matured yet.

And many thanks to comments here. I have checked Gephi, it wouldn't help. I created my applet which works really fine, however I finally reached to this point that I need to use more powerful tools such as d3js, as one has already suggested.


Is there a maximum limit to the number of features in a Neural Network? I have created a dataset which has rather large number of features for example-100,000. Is it too large for a decent computer to handle ( I have a 1080ti )?

It highly depends on your data. If it's image, I guess it is somehow logical but if not I recommend you constructing covariance matrix and tracking whether features have correlation or not. If you see many features are correlated, it is better to discard correlated features. You also can employ PCA to do this. Correlated features cause larger number of parameters for neural network.

Also I have to say that maybe you can reduce the number of parameters if your inputs are images by resizing them. In popular nets the length and height of input images are usually less than three hundred which makes the number of input features 90000. Also you can employ max-pooling after some convolution layers, if you are using convolutional nets, to reduce the number of parameters. Refer here which maybe helpful.


Influence of a data point on the regression result? Let's say I perform multiple regression where y = income, x1 = educaiton, x2 = sex, and x3 = religion from 2003 to 2018, where the data is measured daily.

Is there any way to quantify an impact of a single day data (e.x. 2005-07-01) on the regression result?

It is hard to quantify exactly what a single point is doing, but you would benefit from plotting data to look for suspicious points and then measuring differences in statistics if you remove it.

Don't forget about the value of plotting data! Anscombe's quartet is a great reminder of how important this is. https://en.wikipedia.org/wiki/Anscombe%27s_quartet


Certainity of a classifier How to build a classifier that by default will predict that it is for class 1, but if the classifier believes with 80 certainity that it belongs to 0, it will be classed as 0. How to check how certain a classifier is on it's prediction.

Many classifiers will give the option to get predicted probability. Then you can just put a threshold. Here is how it can be done in with sklearn:

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification

# Make a dataset
X, y = make_classification(n_samples=1000, n_features=4,
                           n_informative=2, n_redundant=0,
                           random_state=0, shuffle=False)

clf = RandomForestClassifier(n_estimators=100, max_depth=2,
                              random_state=0)
clf.fit(X, y)

# 1 if proba is less than 0.8, otherwise 0
predictions = 1 - (clf.predict_proba(X)[:, 0] > 0.80)


What is the output polytree after aplying the Ramex algorithm to this graph? I've been trying to understand the way this algorithm works, but I can't get a consistent result.

It has two phases: the first one coverts a table of events into a graph, and the second where the graph is tranformed into a polytree.

The question is about the second phase, using a back-and-forward heuristic:

This is the heuristic algorithm:

Algorithm 3. Back-and-Forward Heuristic
Input: Network G;
Output: Poly-tree S;
Initialize S;
For each vertex in G
  For each edge in G
    x = arg_max(weighted forward-vertex not visited in G and connected with S; weighted back-vertex not visited in G and connected with S)
  End-for;
  Update solution S with x;
End-for;

The example shown in the paper is this:

Fig. 2. (a) Original cyclic network, (b) Forward Heuristic provides a tree solution, (c) Back-and-forward Heuristic provides a poly-tree solution

Right now I'm having trouble applying the back-and-forward algorithm to the vertex b. I'll describe my thought process:

Initialize S -> Take 'a' and put it in the tree (S).
Look for the edges that connect 'a' (the tree S) to any unexplored node and select the one with higher weight -> 'c'.
Add 'c' to the tree. a--674-->c
'a' is now explored/visited.

Move to next vertex - 'b'.
Look for the edges that connect the tree ('a' or 'c') to any unexplored node and select the one with higher weight -> 'd'.
Add 'd' to the tree. a--674-->c--684-->d
'b' is now explored/visited.

Move to next vertex - 'c'.
Look for the edges that connect the tree ('a', 'c' or 'd') to any unexplored node and select the one with higher weight -> 'e'.
Add 'e' to the tree. a--674-->c--684-->d--1080-->e
'c' is now explored/visited.

Move to next vertex - 'd'.
Look for the edges that connect the tree ('a', 'c', 'd' or 'e') to any unexplored node and select the one with higher weight -> 'h'.
Add 'h' to the tree. a--674-->c--684-->d--1080-->e--930-->h
'd' is now explored/visited.

And now 'b' is left hanging because 'b' and 'd' are visited and the algorithm says the connection must be to a node not visited in G and the polytree shows b-->d.

Is my interpretation wrong?

Here's my interpretation of the algorithm, not sure it is completely right but that could help you:

Initialize S: null (a is added after that)
Enter in first loop on vertices (assume we start with a)
Select the edge with highest weight connected to a: a -> c, and add it
Same with updated S (a or c) c -> d, and add it
Same with updated S: d -> e
Same with updated S: e -> h
Same with updated S: e -> g
Same with updated S: this is where forward and forward & backward algorithms diverge. Forward algorithm only looks at weights for edges starting from S (this is a -> b), while forward and backward algorithm also screens backward links, so the selected link is b -> d which has a higher weight (396) than a -> b (335)
Last update of S with e -> f

This is what seems right to explain the figure, and works to build a polytree. I am a bit confused because it does not exactly correspond to what is written in Algorithm 3, but it is not so rare that pseudo-code algorithm make no actual sense, and would yield wrong results if written directly in any programming language.


Predict Ethereum price up or down by given Bitcoin price How would you proceed, if you have 2 parameters, for example, the price of the Bitcoin and the price of Ethereum for each day and you want to predict based on the Bitcoin price if Etherium will go “up” or “down”?  Some ideas or some tutorials in this area?

So, the question asks how to predict the price Etherium given the price of Bitcoin.
It is worth doing some Explanatory Data Analysis (EDA) to see if there is a correlation between these two variables.
You might find that the relationship between the two variables is non-linear, which indicates that maybe the model that you would make with Bitcoin price feature is going to be too simple and will lead to undercutting. Therefore, it would be worth adding more complexity by increasing the number of features, which could also be correlated with Etherium prices.


Higher dimension data visualization in Matlab/Octave I am working on sparse recovery for a classification task. I use Pine hyperspectral dataset which is a freely available dataset and this image contains 200 Dimension (Depth/channels/bands). In addition, this dataset contains labels (Ground truth information). Thus I extract the corresponding pixels for each class and now I want to visualize them to see really how they look like. But I do know now how to visualize them for all 200 channels. I did this visualization for 3D but I want to know how can I expand this for 3D.

Here is my code for 3D, I would appreciate any help for visualizing my data in 3D
Not: I simulate the dataset.

clear all
% Number of classes = 5
B = [1 1 2; 2 3 4; 2 1 3; 4 3 4; 5 5 3];
B_vec = reshape(B, 5*3, 1);
% Image with 200 depth
A = rand(5,3, 200)*100;
A_vec = reshape(A, 5*3, 200);
%Indexing the desire classes.
indx = find(or(B==1, B==4, B==3));
class_num = B_vec(indx);
%Extracting the corresponding pixels (base on indx)from 200D image.
for k = 1: length(indx)
img_value(k,:) = A_vec(indx(k),:);
end
G_idx_1 = (find(class_num==1));
for k = 1:length(G_idx_1)
C_1(k,:) = img_value(G_idx_1(k),:);
end
G_idx_3 = (find(class_num==3));
for k = 1:length(G_idx_3)
C_3(k,:) = img_value(G_idx_3(k),:);
end
G_idx_4 = (find(class_num==4));
for k = 1:length(G_idx_4)
C_4(k,:) = img_value(G_idx_4(k),:);
end

X1 = [C_1'];
X2 = [C_3'];
X3 = [C_4'];
plot3(X1(1,:), X1(2,:), X1(3,:),'bo'); axis on; grid on; hold on
plot3(X2(1,:), X2(2,:), X2(3,:),'go');
plot3(X3(1,:), X3(2,:), X3(3,:),'ro'); hold off;
title('class1, class2, class3')

You can use t-SNE to perform dimensionality reduction to create a 2-D or 3-D embedding space for data visualization purposes.

For Matlab you can check the implementation and the example.


How would you categorize email subjects to find similar emails? I have a list of email subjects like:

<XYZ> commented on <ABC>
Weekly review for <Company>
Your account is ready

And I want to find patterns in them so I can group them.

Is there a well known algorithm I can use?

Preferably with wide language implementations or easy re-implementation.
The algorithm should be unsupervised.
The number of different emails is not known.

Update:

I think I can break this down into two problems:

Group subjects by the similar words they use, resulting in the following. Each group should be very distinct from the rest (they should be almost perfectly exclusive) and the algorithm should give relatively small number of groups with good length of the common words.

[commented, on]
[weekly, review]
[your, account, is, ready]

Once grouped, it should be easy to find a state automaton that accepts only the group's subject and thus eliminates variable
Then I can go back and check if there are any intersections and tweak the variables.

Having said that, is it better to use a completley different approach like neural nets maybe? I have zero experience with those, but if it makes more sense, I am open to learning.

As per your problem and the description, I'll suggest you to try Text Mining - Bag of Words approach. I did something similar using the same approach and it was really helpful.

More details about the approach are here.

Give it a try. Cheers!


Structuring experiment/training data with months in mind We're using a whole year's data to predict a certain target variable.The model works like data - OneHot encoding the categorical variables - MinMaxScaler - PCA (to choose a subset of 2000 components out of the 15k) - MLPRegressor. When we're doing a ShuffleSplit cross-validation and everything is hunky-dory (r^2 scores above 0.9 and low error rates), however in real life, they're not going to use the data in the same format (e.g. a whole year's data), but rather a subset that is due that month (e.g. in March they'll need the algorithm to predict rows that are due in March) and at this point, the algorithm fails.

The expectation is that we have some data (for example, 2018 December - 2019 December) that we're training on, and then we'll need to predict items due in January 2020 (that's not yet available). Again, if we do a random train-test-val split, the r^2 scores are great, but the team lead is not okay with this, as he said the real-life use case is that we train on 2019's data and then predict on a monthly basis, so he wants us to train on not all the data, but only 2018 December to 2019 November, and then validate on 2019 December data.

I see how this is methodically incorrect, as we're training on data that has different properties (whole year's data vs a certain month's due data), and this reflects on the validation scores. If I isolate a certain month to simulate the real life scenario (e.g. train on 11 months of data and test on one specific month), depending on the month we're getting r^scores between 0.3 and 0.7 depending on the month isolated (which is a far cry from the 0.9 if randomly sampled from the whole dataset).

What I can't figure out is, how can we structure this modelling so that the test data (items due in a specific month) has the same properties than the training set (the rest of the months, or maybe 12 months including 1 month from last year), while preserving information for the whole year? Or should we be doing 12 models for each month?

Sounds like your problem is predicting a value for a specific month given historical data. That could be modeled as a time series problem.
Also, you also say the data strategy changed over time. That means you can not make a stationary process assumption.


CrossMapLRN2d in pytorch I had to convert a code written in pytorch to keras (with tensorflow backend). But there was this layer called CrossMapLRN2d which had no direct counterpart in Keras. So wanted to know what does this layer do and how to implement it in keras.
The exact line of code was

nn.CrossMapLRN2d(size=5, alpha=0.0001, beta=0.75, k=1.0)

Local (contrast) normalization

Local Response Normalization (LRN) layer implements the lateral inhibition we were talking about in the previous section. This layer is useful when we are dealing with ReLU neurons. Why is that? Because ReLU neurons have unbounded activations and we need LRN to normalize that. We want to detect high frequency features with a large response. If we normalize around the local neighborhood of the excited neuron, it becomes even more sensitive as compared to its neighbor.

2 options in keras. First it is already implemented check here but thats "old" keras. New, Google implementation can be found here


Support vector regression and paremeters I am doing load forecasting using SVR(kernel='rbf').How can I understand which is the best value for parameters C, epsilon and gamma?Thanks.

It looks like that you are using scikit-learn. In this case use Grid Search Cross Validation  or Randomized Search Cross Validation to find the best parameters.

sklearn.grid_search.GridSearchCV(estimator, param_grid, scoring=None, cv=None, ...)

In these approaches you basically loop over possible sets of your parameters, specified via param_grid. For each set you perform a cross-validation (default is 3-fold, but you can specify it via cv parameter). Cross-validation gives you the mean value and deviation of your 'scoring' parameter (e.g. 'accuracy' for classification, 'r2' for regression). The set of parameters with the best 'scoring' is the winner.

See example here. It also show how to output not only the best set of parameters, but also "top-N". I find it very useful in building intuition about my model.


Query similarity: how much data is used in practice? I recently read Similarity Measures for Short Segments of Text (Metzler et al.).  It describes basic methods for measuring query similarity, and in the paper, the data consists of queries and their top results. Results are lists of page urls, page titles, and short page snippets.  In the paper, the authors collect 200 results per query.

When using the public Google APIs to retrieve results, I was only able to collect 4-10 results per query.  There's a substantial difference between 10 and 200.  Hence, how much data is commonly used in practice to measure query similarity (e.g., how many results per query)?

References are a plus!

When using the public Google APIs to retrieve results, I was only able to collect 4-10 results per query.

Here's how to get more than 10 results per query: https://support.google.com/customsearch/answer/1361951?hl=en

  Google Custom Search and Google Site Search return up to 10 results per query. If you want to display more than 10 results to the user, you can issue multiple requests (using the start=0, start=11 ... parameters) and display the results on a single page. In this case, Google will consider each request as a separate query, and if you are using Google Site Search, each query will count towards your limit.

There are other search engine APIs as well (e.g., Bing)


How should continuous outputs be in convolutional neural network? I have labeled faces images (label is the age -continuous value-) dataset and I want to construct a Convolutional Neural Network model to predict the age of a person.

I have the following questions.

How the label i.e. the age must be encoded ?
Which activation function to use ?

Convolutional networks can be used for regression tasks too. The difference corresponds to the output layers of the dense networks. In classification tasks you use sigmoid or softmax depending on your task. In regression tasks you can simply use linear activation function as the non-linearity of the last layers. Consequently, if you have the ages as the outputs of your network and if they are in a same scale as the inputs you can simply use them as they are right now.


Can I conduct independent t-test when data is infested with outliers ? and how to interpret the t-statistics? I am working on 2 sample independent t-test. I have conducted analysis on test group vs control group and I have to write a report but I have few questions.

Do we have to take out the outliers and then perform t-test?

Once I perform t-test- can anybody explain the t-test output? The explanation should not be in terms of statistical terms but in such a way that non business person can also understand. I need simple explanation for confidence intervals and difference in means of the two samples.

What kind of charts can we draw to represent our results?

It's fine to do a t-test on unequal sample size, however, the power wouldn't be as good as equal sample size.

1:) Yes or no. Impossible to say without plotting the outliers. What's more important, can you assume your data be normally distributed? Have you checked the QQ-plot? Have you checked the histogram? Do they look like close to a normal distribution? While the t-test is robust against non-normal data as long as the sample size is sufficient large, your data shouldn't behave too far away from a normal.

When you think about outliers, ask yourself the following questions:

How many outliers? If you have many, t-test is probably not appropriate.
Why the outliers? If it's a random error (you're just unlucky), you could include it in the t-test. If it's a systematic error, stop the test, go back and check your data.
How do you define the outliers?
Do those outliers look symmetry? If so, you might assume your sample come from a normal population. You can check the skewness of your data.

You have to try to understand those outliers to come with up a decision.

2:) You can just explain like "the probability of the difference in means is (or isn't) significant".

3:) You should draw a box-plot for each group.


Train, test and submission files - what am I supposed to do with all of them? this might be very beginner's question.
I'm working on Kaggle's HomeCredit Default Risk problem which has among others dataset train, test and submission files as can be seen in the link provided. The test dataset does not contain TARGET feature and submission has ID and TARGET columns where TARGET has fixed value 0.5.

I'd normaly split the train dataset and used it to train and score a model and than created a submission file based on the test dataset.
Thus, I don't understand what I am supposed to do with the already existing submission file.

Futher, when I score a model based on splitted train data, I'm getting unexpected high values which can't be right. I did before OHE-encoding, standardization and PCA, it that might have some impact on the model accuracy result.

Insights would be apprecited.

The submission file is just there as a reference. You are not supposed to do anything with it, but when you make your own submission it be on the same format.

Regarding your high accuracy. Try making your own submission file and submit it to Kaggle and it will show you if your score is real or not. :)

link for submission


Neural Networks - Strategies for problems with high Bayes error rate I am building a Neural Network for a binary classification problem where the Bayes error (lowest possible error rate) is probably close to 50%.

What makes the task easier is that I don't need to make a prediction for each observation of the test sample. I only want to make a prediction for the observations where the model has a fairly high confidence. However a high rate at which  predictions are made is better than a low one.

So far, I have used a standard neural network (feed-forward, cross-entropy loss, L2 regularization and sigmoid activation on final node). In the testing sample, I only take into account the observations for which the final node's value $(\hat{Y}_i)$ is outside of an interval of low confidence:
$$\text{predicted class}_i = \begin{cases}
 1 &\text{ if } \hat{Y}_i > 0.5 + a \\
 0 &\text{ if } \hat{Y}_i < 0.5 - a \\
 \text{NA} &\text{else}
\end{cases} \\
\text{where } a\in [0, 0.5] \text{ indicates the level of confidence required}$$

To tune the hyperparameters (including $a$), I have designed a metric that depends positively on:

Test-sample accuracy (only counting predictions different from NA)
Percentage of predictions that are different from NA.

I am not yet satisfied with the performance achieved with this approach, and I am sure that there are smarter ways to approach this, for example a custom loss function. Advices, links to articles, or even related search keywords are welcome.

You have a high Bayes error rate and it means that you almost can not learn anything. You have to add extra features and investigate whether your data has a small Bayes error or not. Currently, it is worse than a disaster. This large Bayes error illustrates that you have patterns, input vectors, that have completely same components as each feature but different labels. Take a look at here.


How to choose between Tensorflow and Pytorch? Recently I've been working on a pretty vanilla ANN model in Python with sklearn (and its preprocessing pipeline), mostly in jupyterhub notebooks if that matters.
I am considering changing the framework I work with as I need some more tools (drop out at least). but I am not so familiar with Python frameworks. The 2020 state of AI report point out (p. 14) that the two main frameworks are PyTorch and Tensorflow.
What are the practical differences between those two frameworks ?

As of October 2020...
In terms of basic neural network functionality, they are pretty equivalent.
Some differences:

Stability: tensorflow 2.0 underwent a lot of changes from tensorflow 1.x, specifically in the very way it worked (they changed from a computational graph paradigm to an imperative paradigm). This caused a lot of friction and left many underlying problems in tensorflow.
API coherence: tensorflow API has evolved and changed a lot and, through its evolution, many pieces of functionality were duplicated, and later where deprecated and removed. Also, there is a framework called Keras, which, at the beginning was a separate piece of code, but now is integrated in tensorflow. The co-existence of tensorflow and keras brought tension (see here). The Pytorch API has been more consistent over time.
Documentation: the API of tensorflow has changed a lot over the time that many tutorials and stackoverflow questions are outdated. Pytorch has changed less and has kept good backward compatibility so, while there are some tutorials that may include outated practices, most of them should work.
Deployment: tensorflow is known to be better suited for "production scenarios", e.g. it has tensorflow serving for exposing trained models through a service. This, however, is changing and now there are official alternatives for model deployment for pytorch.
Mobile support: tensorflow lite is the official way of having deep learning models use the GPU of Android mobile devices, so it's easier to deploy on Android if you use tensorflow. Note that, while there is a mobile version of Pytorch, it does not support GPU yet.

So, how to choose? This is my opinion:

Use tensorflow if you have a specific reason for using it, e.g. you want to deploy on Android easily, you want to use tensorflow serving, you want to reuse models that are only available for tensorflow.
Use pytorch in any other case.


Getting value of 2 in normalized confusion matrix I have created confusion matrix with python and I have gotten weird results that made me worried:

as you can see, I have value of two, and if I understand correct, normalized matrix cannot have value higher than 1. and also, some columns' sum is not 1.
this is how I created this matrix:
df_confusion = pd.crosstab(df['labels'], df['prediction'])
df_conf_norm = df_confusion / df_confusion.sum(axis=1)

and then I just plot it.
my question is - is it possible to get value of 2? does the sum of column make sense not to be 1? do I have mistake? and if correct, what can explain that?
EDIT

Switch the axes:
# Build the table
confusion_table = pd.crosstab(df["prediction"], df["labels"])

# Normalise
confusion_table = (confusion_table / consufion_table.sum(0))

# Verify
print(confusion_table.sum())

Alternatively, you can play with confusion_table.divide and the axis argument: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.divide.html


Math behind L2 Regularization for Logistic Regression I read that L2 regularization in logistic regression creates a sort of sphere that limits the choice of $w$ weight, but why does this happen?

Your question is really about the method of Lagrange multipliers in constrained optimization, not logistic regression per se. The gist of it is that a constrained optimization problem can be recast as an unconstrained optimization problem by adding a term, called the regularizer, and vice versa. The sphere comes from recasting the unconstrained problem into a constrained one; recall that a constant $L_2$ norm defines a hypersphere.


'RandomizedSearchCV' object has no attribute 'best_estimator_' from sklearn.model_selection import RandomizedSearchCV,GridSearchCV
import xgboost

classifier=xgboost.XGBClassifier()
random_search=RandomizedSearchCV(classifier,param_distributions=params,n_iter=5,scoring='roc_auc',n_jobs=-1,cv=5,verbose=3)
random_search.best_estimator_

  .AttributeError: 'RandomizedSearchCV' object has no attribute 'best_estimator_'

You have to fit the RandomizedSearchCV first in order to access this attribute.

random_search.fit(X_train, y_train)
print(random_search.best_estimator_)


What is "lag" in time series forecasting? I'm studying machine learning (e.i. time series analysis). I encountered an Azure tutorial, Retail Forecasting.

https://gallery.azure.ai/Experiment/Retail-Forecasting-Step-2-of-6-train-time-series-models-3

Here, they introduce new features called "lag", which I don't understand what it means. In 3.4, they said "The module selects the best lag of this index based on maximum correlation." What does it mean? Could you please let me know some references to learn background knowledge?

Lag features are target values from previous periods.

For example, if you would like to forecast the sales of a retail outlet in period $t$ you can use the sales of the previous month $t-1$ as a feature. That would be a lag of 1 and you could say it models some kind of momentum. But you could also apply a lag of 12 to model the sales of the same month a year ago (since retail sales are often seasonal depending on the format, category and SKU). Accordingly, it depends on the dataset which lags work best and looking at correlation is one way to select the lag values.

Have a look at this blog post for a simple explanation of lag features.


Newton's method optimization for Deep Learning I'm reading this paper "Deep learning via Hessian-free optimization" by J. Martens, I am having difficulty figure out the following statement:

  In the standard Newton's method, $q_{\theta}(p)$ is optimized by computing the $N\times N$ matrix $B$ and then solving the system $Bp = −\nabla f(\theta)$.

(section 3 of the paper)

Is there any theorem, or statement anywhere regarding why the above system needs to be solved to optimize the local approximation? I came across another paper that has a reference to J. Martens and has used the same statement.

If you take a look at section 2, it says

  The central idea motivating Newton’s method is that $f$ can be locally
  approximated around each $\theta$, up to 2nd-order, by the quadratic: $$ f(\theta + p) \approx q_\theta(p) \equiv f(\theta) + \nabla f(\theta)^Tp + \frac{1}{2} p^TBp \, \, (1) $$  where $B = H(\theta)$ is the
  Hessian matrix of $f$ at $\theta$. Finding a good search direction then reduces
  to minimizing this quadratic with respect to $p$.

To minimize, you need to take the derivative of (1) with respect to $p$ and set it to zero:

$$\Rightarrow \nabla f(\theta) + Bp = 0$$

which is equivalent to $Bp = -\nabla f(\theta)$.


how come accuracy_score recognizes the positive label and precision_score does not? I am executing this code which works perfectly for me:
(I only have 'positive' and 'negative' sentiments):
from sklearn import metrics
print('Accuracy:',metrics.accuracy_score(test_sentiments, predicted_sentiments))
print('Precision:',metrics.precision_score(test_sentiments, predicted_sentiments, pos_label='positive'))

My question is: how come accuracy_score recognizes the positive label and precision_score does not?
ps: if I execute:
print('Precision:',metrics.precision_score(test_sentiments, predicted_sentiments))

or
print('Accuracy:',metrics.accuracy_score(test_sentiments, predicted_sentiments, pos_label='positive'))

They both fail.

Accuracy is symmetric in the naming of positive/negative classes, but precision is not: for accuracy, it doesn't matter which class is "positive."  So accuracy_score doesn't have a parameter pos_label, and will error if you try to pass that parameter; meanwhile precision_score has default pos_label=1, so if your labels don't include 1 and you leave the parameter to the default, you'll get an error.


How to correctly interpret the table function to determine logistic regression accuracy? I have a set of data, that gives the length of a species of abalone, and its corresponding type (male, M, female, F, or infant, I). (https://archive.ics.uci.edu/ml/datasets/abalone)

I have constructed a logistic regression to create a model that will determine whether the abalone is M/F or I, given the length. (M and F are classed as the same.)

So I write the following in R to generate and test the model on data points:

g <- glm(sex ~ length, family="binomial")
pairs <- paste(round(predict(g, type="response")), sex)
table(pairs)

The output table is:

pairs
0 F  0 I  0 M  1 F  1 I  1 M
218    6  210 1089 1336 1318

How can I correctly interpret this?

These are the options I have come up with:

1089 females correctly identified, 218 females incorrectly identified; 1318 males correctly identified, 210 males incorrectly identified; 1336 infants correctly identified, 6 incorrectly identified.
218 females correctly identified, 6 infants and 210 males incorrectly identified; 1089 females incorrectly identified, 1336 infants and 1318 males incorrectly identified.

The outcome of a multi-nomial or binomial is confusion matrix (2*2 for binomial, n*n for multinomial),

Interpretation of the confusion matrix is for example:

Accuracy of the model : ((TP + TN) / (TP + FN + FP + TN) ) * 100
It means you model could exactly pick this much percent of data was classified correctly.

Precision: (TP / (TP + FP))*100
That measure of correctness achieved in positive prediction i.e. of observations labeled as positive, how many are actually labeled positive

Recall: (TP / (TP + FN)) * 100
It is a measure of actual observations which are labeled (predicted) correctly i.e. how many observations of positive class are labeled correctly. It is also known as ‘Sensitivity’.

F-Measure: ((1 + β)² × Recall × Precision) / ( β² × Recall + Precision )
It combines precision and recall as a measure of effectiveness of classification in terms of ratio of weighted importance on either recall or precision as determined by β coefficient.
generally β is 1.

If you want read them in just normal words in you example:

        Predicted
          M   F
       M 100 150
actual F 100 300

It means your model could predict 100+300 correctly out of 100+150+100+300

Actual number of males where 250 but you could classify 100 correctly and 50 wrongly.

Similarly, Actual number of females where 400 but you could classify 300 correctly and 100 wrongly.

If you any need more detail let me know.

Go through this Link, you will get better idea.
This Link is an interactive chart, which will give you better understanding.


How to fill missing enteries in column A, and add respective corresponding enteries to column-B with value of previous cell I am facing an issue with an excel file. I have an excel sheet with 2 columns

Column A :  Time Increment with per second

Column B :  A particular value of a machine sensor

The problem i am facing is when the machine is stopped (not in motion), the depth increment stop for that particular time and do not make entries in the excel sheet, and once the machine start moving it again add the entries for the starting point. Let me explain in better way as below.

04:23:12    2709.082597
04:23:13    2708.747333
04:23:14    2708.442548
04:23:15    2708.229198
**04:23:16  2708.229198
04:24:01    2708.137763**
04:24:02    2707.924413

The machine is stopped at 04:23:16  and it resumed its activity at 04:24:01 with a gap of 44 seconds.So the excel sheet missed the time entries for those 44 seconds. There are more than 1000 entries like this in the file. What i want that Column A (Time) should be incremented from Cell A  to Cell Z, without missing any entry of the second (even when the machine is stopped), and corresponding value at column B should be added to all 44 cells which should be the value of sensor corresponding to the time 04:23:16 (when the machine stopped) i.e 2708.229198.

Like for example the machine is stopped at 04:23:16, Sensor value at Column B is 2708.229198, i want that file should fill all the missing entries for the time between 04:23:16 to 04:24:01 (44 seconds), and for all those 44 seconds the value at Column B should be 2708.229198 (A value of column B corresponding to 04:23:16), and this can be done for the whole sheet. As i explained its a very lengthy file and to do the option manually is not suitable, that may take hours or even days, I want some practical solution to sort out this issue.

Any help or guidance will be appreciated in this regard.

Jawad

try to make a sheet derived from the source. first, put all possible time in the new sheet in an ordered way. second, do vlookup from source for value. third, put vlookup inside a iferror to find missing value and for second part of iferror use previouse value.


When do I use Multiply and Add I want to know the effect of Add and Multiply in keras by functionality. The dumb way of thinking is that they are meant to add and multiply keras tensors. I want to know when are they to be used. For example, look at the code below from here. Why use Multiply in spatial_squeeze_excite_block and why use Add in channel_spatial_squeeze_excite? Can we switch Add and Multiply in these functions? Why not?
def spatial_squeeze_excite_block(input):
    ''' Create a spatial squeeze-excite block
    Args:
        input: input tensor
    Returns: a keras tensor
    References
    -   [Concurrent Spatial and Channel Squeeze & Excitation in Fully Convolutional Networks](https://arxiv.org/abs/1803.02579)
    '''

    se = Conv2D(1, (1, 1), activation='sigmoid', use_bias=False,
                kernel_initializer='he_normal')(input)

    x = Multiply([input, se])
    return x

def channel_spatial_squeeze_excite(input, ratio=16):
    ''' Create a spatial squeeze-excite block
    Args:
        input: input tensor
        filters: number of output filters
    Returns: a keras tensor
    References
    -   [Squeeze and Excitation Networks](https://arxiv.org/abs/1709.01507)
    -   [Concurrent Spatial and Channel Squeeze & Excitation in Fully Convolutional Networks](https://arxiv.org/abs/1803.02579)
    '''

    cse = squeeze_excite_block(input, ratio)
    sse = spatial_squeeze_excite_block(input)

    x = Add([cse, sse])
    return x

They are defined that way. Have a look a the paper cited in the docstring. The sse is defined in eq. 3, and the scSE is defined in the text in 2.3.


Detecting abundance of a certain periodic pattern in a time series? I am really stumped at the moment about how to solve a particular problem. I have many time series like this:

This represents the number of hours a person spends on a website each day throughout the year. Any days where they are not seen to be using the website have zero values, rather than missing values.

What I really want to do is to calculate a metric telling me to what extent there is a consistent "1 hour per day, once per week" pattern, or, say a "10 minutes a day 6 times per week" pattern and I have written some code that produces this pattern. I don't have any background really in machine learning or deep learning, but I have looked into some techniques from spectral analysis and time series. I also found a technique called "Dynamic Time Warping" which I could in theory use to find out how much of a cost there is in mapping this time series onto the "ideal" one (representing the usage patterns I just mentioned).

I tried plotting the periodogram and correlogram, but there seems to be very little correlation indicating periodicity of any kind:

Can anyone shed some light onto how I could use certain techniques to get the information I need, or any introductory texts on periodicity detection or data mining for those who have never really used it before?

Time warping is used to align different time series, is more useful for data preprocessing rather than for what you want.
"What I really want to do is to calculate a metric telling me to what extent there is a consistent '1 hour per day, once per week' pattern"
I don't know if you have a specific reason to use a 'metric', if what you need if to find periodical patterns across people what you could do is apply different moving averages to every time series to extract different periodical patterns, then you could evaluate the cross-correlation between different people time series to evaluate which patterns are subjective and which can be instead considered true across people.


What is the theoretical differences of Multitask learning vs Fine tuning based transfer learning? Suppose, I have the following scenarios.

I have a bunch of fruits, i.e., apple, orange, and banana. I simply made a Multitask model, where my network first tell me which fruit it is, and then telling me the color of it. Suppose, if I give my network an apple, it tells me, (a) it is apple, (b) it is red. By doing some theoretical study, I have understood that it is one type of inductive transfer learning (TL) (correct me, if I am wrong). So here, the network is learning 2 task simultaneously.

I have a bunch of objects, i.e., cube, ball, and triangle. Here also I want my network will do the same thing like scenario 1. So it will tell me, (a) whether it is a cube or not, and (b) then tell me the color. So what I did is, I transferred the learned weight and parameters from the network of scenario 1 to this scenario. Thus I performed the fine-tuning based TL in here.

So , from theoretical point of view, I have few confusions. I need to clarity my idea, and need some ideas from experts.

If I consider the scenario 2, by definition of fine-tuning based TL, the task of scenario 1 (apple, and red) is my source task, and the task of scenario 2 (cube, and red) is the target task. From my understanding, I think that every inductive TL approach has a source task and target task. So, for scenario 2, thus it satisfies my understanding.

[REAL QUESTIONS]
2. Now the confusion starts for my theoretical understanding. For scenario 1, it has also 2 tasks - (a) identify the fruit, (b) identify the color. So here, what will be my source task, and what will be my target task. For clarifying my theoretical description or explaining my thinking into words, I need to know this.
3. As I am doing 2 TL tasks here, how to define the whole scenario ?

As per my understanding,, in the both scenerio above when learning the object type and color what you are doing is a multitask learning. That is you are teaching your model to do two task simultaneously, 1. Predicting Object type(Which fruit/shape) and 2. Predicing color. So it is not like TL rather Multi task learning.
1.
TL does have some source task.But all you do in TL is that you freeze the whole model except the last few layers and retrain them on your specific dataset.(or add fews layers at the end and train them). TL doesn't train your model from beginning fro new task. But the aboe scenerios are not TL. So it is ambiguous to name them source and target task.
2.As said above, both of them are multitask learning rather than TL.So t is ambiguous to name them source and target task.
3.Both of the scenerio mentioned is multitask learning. Although you can use any of them as your source task and do a TL on it to work on the other. For example, you can train your Fruits and Color model in scenerio 1 and do a transfer learning on it for the second scenerio(to learn shape and color). As they have some same properties underneath, so transfer learning is a valid option here.


If there are no missing values in our training set, should we accommodate missing values in an unseen test set? My training data has no missing values. I'm unsure whether or not I should fit say, imputation, on the training set so that I can accommodate possible missing values on the test set, because the test set is 'locked away' during training time. Should I peek at the test data to see if there are missing values, or should I fit imputation for all features on a complete training set?

First, we must understand about a common statistical term called population. Given a population say X, a random sample is drawn (in the ideal conditions). Now suppose you are asked to build a predictive model based on this random sample. So, you split the sample into train, test and validation sets. And you start to build the model on the train set. You begin with the initial data cleaning activities, and you find there is no missing data in it.
With this brief background, now to answer your Q literally forces me to rather ask you a question;
Q. Did you collect the data or was it given to you?

If your response to the former part of my Q is yes then by all means do whatever you want with the data. And if your response to the latter part of my Q is yes then you should not make any attempt to pollute it. Nor should you make any attempt to peek into the holdout dataset, because then subconsciously the predictive model you have built will be sabotaged.
Finally, remember the concept of population and sample. It holds the key. Build your model such that it accounts for all initial data preprocessing activities. If a particular step does not fulfill the dataset, then code can be written to avoid it
Edit 1
Basis of the OP suggesting the dataset was given to them, I'm revising the answer further;
i think your getting confused between the population data and new data. As per my answer, models are built basis of the population data. The new data should have the same attributes and properties as the population data. However, if for some reason unknown to you, the new data has missing values but the original data does not, then there are a couple of options, namely;
a. Determine the missing data pattern,
i. if its missing completely at random (MCAR), means there is no relationship between the missingness of the data and any values, observed or missing data. In other words, no systematic differences exist between participants with missing data and those with complete data. In these instances, the missing data reduce the analyzable population of the study and consequently, the statistical power, but do not introduce bias: when data are MCAR, the data which remain can be considered a simple random sample of the full data set of interest.
ii. Missing at random (MAR). When data are MAR, the fact that the data are missing is systematically related to the observed but not the unobserved data.
iii. Missing not at random (MNAR). When data are MNAR, the fact that the data are missing is systematically related to the unobserved data, that is, the missingness is related to events or factors which are not measured by the researcher.
Once you have determined the nature of missingness, then you should determine if the missing data can be imputed or not.
In conclusion, if the new data has missing values then you can treat them or else remove them. The choice is yours. My suggestion will be to follow the middle path, in which you can build 2 models; first model without the missing data and second model with the imputed missing data. Evaluate and validate both the model performance and choose the one which closely matches the desired business/research outcome.


ML Project - Achieve 2 Objectives I have a dataset with 5K records focused on binary classification. I am posting it here to seek your suggestions on project methodology

Currently what is my objective is

1) Run statsmodel logistic regression to find risk factors that influence the outcome

2) Then build a predictive model based on best features (may or may not include risk factors). because as you may know not all significant variables are good predictors.

Though I can use scikit-learn logistic regression to build a predictive model but I am planning to go with Xgboost because it provides better performance in my dataset (non-linear data slightly imbalanced)

I do step one because I have to find what are the risk factors that influence the outcome, so I am doing it. (ex: risk factors that influence customer to default in loan repayment) You know where we get p-values and find significant risk factors.

In 2nd step, I build predictive model because I realized through running the built model that not all risk factors are good predictors. So in the end, I include new set of features that help in better prediction along with risk factors

Do you think that I am right in having/approaching this as two objectives problem?

Do you think what I am doing is redundant or am proceeding in the right direction?

Do you think there is no reason to use 2 algorithms separately?

Do you have any suggestions or tips to make it easy to achieve my objective?

Xgboost does the feature selection for you. If you want to report how valuable certain features are for prediction, print the feature importances. Those will, however, just tell you "feature $x_1$is very important for predicting the outcome, feature $x_2$ is nearly useless for predicting the outcome, etc.".

To get risk factors with p-values, you need a more interpretable model. You have to use linear classification, for example. Then you can make statements like "high $x_1$ correlates with a positive outcome".

If you really want to only use a subset of features, train an Xgboost model on a validation dataset and drop features that have low importance. Then run an Xgboost model with the remaining features on the remaining training set.

You need to think about what you want: Do you want to explain observations and extract explicit knowledge from data and try to find possible causal links? Use linear methods. Do you want to predict an outcome for new patient data? Use gradient boosting. You can obviously do both.


Weight decay in neural network I have been reading through this book and am trying to do the exercises.

The problem is "Connecting regularization and the improved method of weight initialization" part 3. We have to use a heuristic argument to prove "the weight decay will tail off when the weights are down to a size around $1/\sqrt{n},$ where $n$ is the total number of weights in the network."

For more context on the same problem (Parts 1 and 2):

Heuristic argument for Weight decay and regularization

Speed decay proof for L2 regularization and non-normalizied weight initiation

The relevant equation appears to be this:

$$C= -\frac{1}{n_t} \sum_{xj} [y_j \ln{a^L_j}+(1−y_j)\ln{(1−a^L_j)}]+\frac{\lambda}{2n_t}\sum_{w}w^2.$$

(The book seems to use the $n$ notation to represent two different things. I've changed it so that $n_t$ refers to the size of the training set, and $n_w$ refers to the number of weights in the network)

I think it has something to do with the last term, $\frac{\lambda}{2n}\sum_{w}w^2$, because if we substitute $w=\frac{1}{\sqrt{n_w}}$ then the whole term simplifies to $\frac{\lambda}{2n}$. This would mean that the partial derivative with respect to the weight becomes $\frac{\partial C}{\partial w}$ instead of $\frac{\partial C}{\partial w}+\frac{\lambda}{n}w$. Is this enough of an explanation to conclude that weight decay falls off once the weights are around $\frac{1}{\sqrt{n}}?$

According to the book, the problem of initializing weights with too big of a standard deviation is that it is very likely to cause neurons to saturate.

But with L2 regularization, when saturation occurred only the L2 term will affect the gradient, and cause weight decay.

And when weights get small enough not to cause saturation (for example around $1 / \sqrt n$), the other term comes to affect the gradeint.

So the relative influence of L2 term decreases.

(And of course, the absolute effect of L2 term will decrease by decaying the weight.)

Why $1 / \sqrt n$?

If all of the $n$ input neurons are 1 and the standard deviations of weights are $\sigma$, the standard deviation of the input to hidden neurons will be $\sqrt{n}\sigma$.

If you want $\sqrt{n}\sigma$ to be 1 to avoid saturation, $\sigma$ should be $1 / \sqrt n$.


Improving initial Cross Entropy in CNN I have a Convolutional Neural Network, that is structured as binary classifier. I have two relatively standard convolutional/relu/pooling layers followed by 2 layer fully connected network outputting to a softmax with loss layer for the binary classification. However I observed something unusual:

In Version 1 of the network I had both convolutional layers derive 10 features, upon initialisation I had a Cross Entropy Error of around 28.

In Version 2 I upped the number of features of the convolutional layers to 64 features. Despite still having the same fully connected layers and the same softmax, my Cross Entropy Error jumped to 340.0

My question is, why would this happen. Surely the randomness is the same and the softmax with loss function should normalise so that both outputs add up to 1. So why would Cross Entropy suddenly jump so high

My understanding of the cross entropy effects for outputting large numbers was helped by this answer

By increasing number of features in convolutional layers, you are actually increasing the number of learnable parameters which were initialized to random values. So, randomness increases and that's why the cross entropy loss increases.Hope this is satisfiable justification to your question.

Ex, Suppose convolution layer with :

filter_size = (3, 3) and number_feature = 10 => number_parameters = 3*3*10 = 90
filter_size = (3, 3) and number_feature = 64 => number_parameters = 3*3*64 = 576


Some algorithms and approach for identification of specific patterns? I am working on a hobby project where I have a data related to financial trades(such as stock trades). Now I want to analyse this data and extract possible suspicious patterns and trades. Since I am new to Data Science so I would love to get few suggestions regarding what are some effective algorithms(machine learning) that can be relevant to such a scenario. One category that I think is using a possible outlier detection algorithm but I am unable to determine which one to use. I would also appreciate few pointers related to what my approach should be.

I suggest , go for Anomaly detection:

Anomaly Detection is done assuming our data has a probability distribution(gaussian). We can plot data to see if thats the case, if not we can make it gaussian using log transforms. Gaussian distribution specifies the regions and probabilities of our data lying in those regions.

For example : replace original feature x -> Log(x) or feature x -> (x)^4/3 etc..

Also regarding the threshold value which decides outliers you can play with it and see that with Higher threshold you will be rejecting more entries and this might be required where doctors are trying to isolate cancer patients amongst many normal ones without taking any risk/chances. Again outliers here doesn't mean cancer patient but definitely worth a medical test. And you can set it to lower value if you are getting too many normal data flagged as outliers.

We have skewed data sets since we have more examples of one kind than the other. For example when we get air craft engine data we might just have data for few bad ones and mostly for good ones.Use of cross validation data is suggested.F1-Score is a pretty good metric to evaluate the performance of the algorithm.

To get a proper hold on this topic , I also suggest go through  anomaly detection course videos by Andrew NG in machine learning on Coursera. Free course and very nicely made.

The well known techniques for the same are:
Density-based techniques (k-nearest neighbor),Support vector Machines,Cluster Analysis Anomaly detection. People also use Fuzzy logic based techniques.
And for python I suggest use libraries like scikit-learn and TensorFlow , they have superb implementations of usable machine learning algorithms.Heres a video link from a googler,giving general overview for the same :https://www.youtube.com/channel/UCwUtgW0JCPFjLbKx4VVr3bw


Issue while predicting multiple values which possess different order of magnitude (regression) I have a system that take electric signal and output parameters (regression). However I run into an issue. The parameters I want to predict are not on the same magnitude. I would like also to use only one neural networks if possible.

Parameter 1:
Mean    = 12.53673
Minimum = 10.00461
Maximum = 14.98899

Parameter 2:
Mean    = 148656955394038
Minimum = 75029133522564
Maximum = 224934092847235

Parameter 3:
Mean    = 1.475720134278e+17
Minimum = 7.506184799345e+16
Maximum = 2.249190781380e+17

...

If I use any loss function such as RMSE, MSE, MAE, MSLE the losses of the big parameters takeover all the rest. The network therefore randomly guess all the smaller parameters

I am searching a good way to find those parameters using one network. I tried using "mean absolute percentage error". But the learning is exceptionally slow, and it often get stuck into extremely low value and extremely high value.

I am maybe thinking about creating one loss per target. Freeze all the network except the output layer and each epoch let one of the loss train the whole network. But I am not sure it is a good idea.

Currently I am normalizing the target before predicting it, but I don't like this solution very much. I would prefer having a slightly more complicated architecture than normalizing and denormalizing (afraid of data leakage) and it seems more robust to directly output the right value.

Moreover In the future I will have an issue similar except I will need to predict series that possesses different order of magnitude (and not only some distinct value). So I hope being able to reuse the technique I am learning

Any help would be very appreciated!
Thanks :)

EDIT:
For those who happens to have the same issue, this is the code I went with (keras)

class Model:
    def __init__(self, shape_features, shape_targets, mean_targets, std_targets):
        self.shape_features = shape_features
        self.shape_targets = shape_targets
        self.mean_targets = mean_targets
        self.std_targets = std_targets

    def build(self):
        inputs = Input(shape=self.shape_features, name='Input')
        x = Conv1D(filters=16, kernel_size=3, activation='relu', name='Convolution1D_1')(inputs)
        x = MaxPooling1D(pool_size=2, strides=2, padding='valid', name='MaxPooling1D_1')(x)
        x = Conv1D(filters=32, kernel_size=3, activation='relu', name='Convolution1D_2')(x)
        x = MaxPooling1D(pool_size=2, strides=2, padding='valid', name='MaxPooling1D_2')(x)
        x = Conv1D(filters=64, kernel_size=3, activation='relu', name='Convolution1D_3')(x)
        x = MaxPooling1D(pool_size=2, strides=2, padding='valid', name='MaxPooling1D_3')(x)
        x = Flatten(name='Flatten')(x)
        x = Dense(128, activation='relu', name='Dense_1')(x)
        x = Dense(64, activation='relu', name='Dense_2')(x)
        x = Dropout(0.2, name='Dropout')(x)
        outputs = Dense(self.shape_targets, activation='linear')(x)
        model = tf.keras.Model(inputs=inputs, outputs=outputs)

        def de_normalizing(tensor):
            return tensor * self.std_targets + self.mean_targets

        predictions = Lambda(de_normalizing)(outputs)
        model_prediction = tf.keras.Model(inputs=inputs, outputs=predictions)

        def custom_loss(y_true, y_pred):
            y_true = (y_true-self.mean_targets)/self.std_targets
            return K.mean(K.square(y_pred - y_true), axis=-1)

        return model, model_prediction, custom_loss

One model is used for training, the other is used for predictions

I am maybe thinking about creating one loss per target. Freeze all the
  network except the output layer and each epoch let one of the loss
  train the whole network. But I am not sure it is a good idea.

This is one option. It should work but it may be slower to converge as you only make use of only a third of your losses each epoch. Also, it is not completely unlikely that your losses may work "against each other". Meaning that they could somewhat cancel each other out, which can make convergence take longer.

  Currently I am normalizing the target before predicting it, but I don't like this solution very much. I would prefer having a slightly more complicated architecture than normalizing and denormalizing (afraid of data leakage) and it seems more robust to directly output the right value.

That's what I would do. I don't think you should be afraid of data leakage. Computers have enough precision in them that normalizing/denormalizing shouldn't create any issues. Also, keep in mind that computers have greater precision for floats close to 0. So normalizing your data may actually be good.

And when you say, I'd rather directly output the right value. You can do both! Your network could directly output the denormalized value, but the loss used to train the network can be computed on any layer before that.


Is it right to maintain the train distribution in test set for unbalanced data? If the training set was unbalanced the chances are the model will be biased. But if the data distribution in the test set is the same distribution as the train set, this kind of bias is not going to affect validation accuracy. But my question is if this is the right thing to do? It's not cheating? What if we want to use the model for commercial business that we have no idea how the distribution of data would be? In this case, what is the right thing to do?

If the training set was unbalanced the chances are the model will be biased.

Not really. Depending on the loss function you use. Also, note that for data to be unbalanced at least it has to be in a proportion of 1/100.
The rest of the questions:
ML is based on the hypothesis that train and test look alike. Oversampling methods can help in training time, still in validation and test, you should not use the oversample and validate with the real data.
Use your evaluation metrics on the real test, with test distribution and don't oversample there.

we want to use the model for commercial business that we have no idea how the distribution of data would be?

If you have no idea of how the distribution will be you have a problem. The hypothesis that people make is that the future distribution resembles something of the current real distribution (last week,last month, one year ago....)


Outlier detection for unbalanced classes I have to make a predictive model for predicting a boolean Won/Lost variable based on some other numeric data; and further find out the features of observations that have 'Won'.

However, the number of 'Won's' in my dataset is 0.05%. I've tried both oversampling and downsampling, but it hasn't worked. Even if I take an equal amount of 'Won's and 'Lost's', the model is not accurate for the rest of the 'Lost' values. I've also tried out weights, but it's not working well. Ideally I think I have to put a very high weight for 'Won'.

PS: Using RandomForestClassifier, with a confusion matrix to verify.

I'm not keen on trying out SMOTE, as I've heard it's tough in Python.

So now I'm trying to look at it in a different way, and do anomaly detection for the 'Won' case, as it natural for the data to have so few 'Won' cases.  So, two questions

Is this a correct approach?
How to go about it using Python?

you need to distinguish between these cases:

Data Imbalance
Data Imbalance + Very few number of samples (minority class)
Severe Data Imbalance + Very few number of samples (minority class)

20:60   vs.  10:20    vs. 100:1000  vs. 10:100

and these cases:

similarities between different classes.
wide variations within the same class.

You need to understand to which of these cases your problem belong.

if you have very severe data imbalance + very few number of samples + wide variation within the majority class and similarities between different classes. regular oversampling or down sampling techniques will not help you as well as most of the synthetic oversampling techniques designed specifically to deal with the data imbalance but the assumption is to have enough number of samples.

Try to focus more on ensemble techniques that designed mainly to deal with data imbalance.
SMOTE-Boost
RUSBoost
SMOTEBagging
IIIVote
EasyEnsemble


Reconstructing original data points from t-SNE output I have been trying to understand t-SNE for a while now and I have this very basic question on the comparison of PCA and t-SNE, on which I would really appreciate some help. In case of PCA suppose the Eigen vectors (say first $m$ PCs are retained) are $U_{n\times m}$ and your data matrix is $X_{n\times p}$ with covariance matrix $C_{n\times n}=XX^T$, then the reduced data set is obtained by performing a projection $X_r=U^TX$. From this $X_r$ one can get back $X$ by simply multiplying $X_r$ with $U$. Thus $U$ acts as the basis functions on which the data is projected. I was wondering what would be the basis function for t-SNE type dimensionality reduction algorithm. I know it's non-linear so such a linear projection would not be available, but is there a way to get back the original data set from the reduced data set as obtained as the output of t-SNE algorithm? Apologies if this is too intuitive or fundamental and is simply because of my lack of understanding t-SNE. Any links to material related to this would be appreciated

There is no closed form.

It is a local embedding, and you really cannot expect to find a good inverse mapping.

It's doable, but not very good. Use gradient descent in the input domain. Just like tSNE but with the t-distribution and the Gaussian reversed. However, the original Gaussian has a sigma parameter that you also need to find. So you need to also optimize this to have the desired perplexity. You'll need to do the math for this optimization problem yourself.


Does regularisation make the loss noisy? I implemented dropout and got a loss plot like this

Before implementing regularisation the loss the was not noisy at all

I understand why implementing dropout would increase the noise as different neurons would be active in successive epochs, but does L2 regularisation add noise? Why or why not

I think your best bet to answer that question is to continue to test it on other datasets to confirm that your observation holds true in multiple situations.

Here is my intuition (although I'm by no means an expert).

The regular loss function is (wx + b - y)^2
The loss function for L2 Regularization is L2 = (wx + b -y)^2 + lamdba * w^2

In the first equation, you only have to worry about w as one term. As a simple example, If you update it by .1 and it improves, you'd be safe to update it again in the next round by .1

In the L2 equation, you have an additional instance of w to deal with and it's squared and multiplied by another number. It would be easy to imagine that the same circumstance may lead you to jump the gap and skip over where the gradient is equal to zero.


What is the best reference for multi-class SVM? Can someone suggest some papers about the multi-classification methods by SVM? One against all? A good survey or paper which clearly describes all of the steps.

The best resource for multi-classification support vector machines (SVM) is the book "Pattern Recognition and Machine Learning" by Bishop.

The paper "A comparison of methods for multiclass support vector machines"  by Hsu and Lin also covers the topic.


Regression for forecasting I am trying to fit a regression model to predict the revenue generated from the sales of a particular product. I have found out seasonality both in the outcome(sales revenue) as well as in the input variables.
Sales Revenue

The 'Seasonally Adjusted Annual Rate' and the 'Non Seasonally Adjusted' version of each of the input variables are available.

What version should be used to training the model ?
Should 'month' be included as a categorical variable as a means to capture
seasonality ?
Should the seasonality be removed from revenue if the Seasonally Adjusted version of inputs are used?

If the goal is to predict actual revenue, you want to predict the unadjusted rate. You should include month as a variable so your model has a chance of getting the seasonality right.

If the goal was more performance monitoring (e.g. "We did better in March than February"), then you would want to be analyzing the adjusted rate. You see this all the time in the US news when they forecast adjusted home sales, car sales, or unemployment.


Beginner in programming and data science with 100 hours to spend learning the basics I started using the below link to teach myself data science, with some mathematical knowledge (including calculus, linear algebra, stats/probability) but very little programming experience:
Quora - Roman Trusov's answer on how to learn Data Science in 100 hours.
The above link has the following starting advice:

You will need an RDBMS to handle the data, so the first day would look like this:

Install and configure MySQL. Import the dump into the database
Read SQL basics. Spend some time doing simple exercises to get the hang of manipulating the data...

After a few days, I'm still stuck at this stage. I'd like to learn how to do data science, and was hoping that following the plan in the above link would achieve that quickly. But I'm starting to feel it is not working. Is debugging MySQL installations an important part of becoming a data scientist - so I should just get advice on that and try harder? If not, what can I do differently to achieve my goals of learning data science in similar time frame?

Being good at a subject does not automatically make someone a good teacher, and it looks like Roman's answer on Quora has fallen into a trap of thinking everything he knows is simple and could be picked up quickly. Also making the potential student attempt things outside of a beginner level - effectively just by research on the web following a 2 paragraph pointer - is going to make progress slow and frustrating. Despite no doubt good intentions, the advice there is likely to give a very poor learning experience.

There are full structured tutorials available on free Massive Online Open Courses (MOOCs) available for learning data science topics. These have been put together by professionals who know how to teach, and the effort put into designing any of these courses, providing materials etc, totally dwarfs the effort that went into the advice on Quora.

Ignore the Quora answer, and sign up for one or more of these MOOCs. You can find them at Coursera, Udacity, edX and other similar places. If you want to cram a lot of study into a short time, look for "at your own pace" courses where all the materials are available to you immediately.

The almost canonical start point to test the waters would be Andrew Ng's Machine Learning course on Coursera. Machine Learning is one of the more immediately accessible and fun parts of data science, and the course goes into theory and practice with additional sections for beginners at the start covering necessary maths and programming. Following that entire course is probably around 100 hours total effort. You won't come out of it knowing data science, but you will gain useful practical skills, and get a real taster for the machine learning side of the subject.


There could be a problem with the linear layer after the attention inside a transformer? My question regards this image:

It seems that after the multi head attention there is a linear layer as they mention also from here:

the linearity is given by the weights W^{o}. my quesion is: for the decoder, doesn't this linear layer mess up with the masking of the attention? I mean, if each token from the attention layer should not depend from the next tokens then these weights seem to mess up the whole thing or they don't? Indeed these linear weights will learn the dependencies among all the tokens and during inference there could be a problem, maybe(?). Thanks for any clarification.

No, this is not a problem.
If we zoom into the scaled dot product attention blocks, which happen before the projection with $W^O$ we see this:

There, you can see how the masking of the current and future positions happens inside the scaled dot product attention, which happens before the multiplication by $W^O$. Therefore, the values learned for $W^O$ are trained only with the information from the previous tokens in the decoder.