Machine Learning Design Document I was trying to create a design document for a machine learning project.

From a flow perspective, I know that I would have a training flow, a validation/test flow followed by a production flow.

However, from a design document perspective, I was no able to find any sample template that details out the different sections / subsections that a design document for a Machine Learning project should contain. I believe that the design doc for a ML project should be different from a normal project.

Also, what are the different documents that a ML project should have, from a process perspective?

Any ideas in this direction would be helpful.

By ‘design document’ I assume you are authoring a technical doc in an industry job.

I treat this similarly to an academic journal publication but with less formatting and page restrictions. Ultimately, you want to provide sufficient information such that if Apple hired you tomorrow your replacement could reproduce the work. Further, your colleagues should be able to grasp the problem attacked, solution proposed, and evidence of success. I usually put the yawn-inducing deep stuff in appendix,  but suggested sections might include:
0) executive summary
1) background/motivation
2) data sources
3) analysis/algorithm/model
4) validation performance
5) [optional] real world test results/pilot
6) [optional] estimated return on investment
7) [appendix] SQL source code, modeling heuristic, outlier treatment, etc.


Apache Spark alternatives for local compute I am creating a relatively large hobby project in Scala that needs a few ml algorithms for text classification into topics. My dataset is not huge, it is < 500,000 items with dimensionality 5 (2 dimensions are free form text).

From what I've started with on Spark, it is heavily geared toward distributed computing and other production related concerns but it has a nice ml library. Is it worth using Spark if I only plan to run my project on a local dev machine level or is there a more appropriate library out there for me in Scala? Something like scikit-learn, but in Scala.

You're right, Spark is intended to scale in a distributed computing environment, but it is absolutely performs well locally.

When running locally Spark will still 'distribute' processing across local Executors, you have options to control the number of CPU cores the Spark job can use, and how many cores each Executor can use. You can get a lot out of using Spark locally, and Spark makes it very easy to scale up to a cluster of machines (e.g. on AWS or Google Cloud)  when your single machine can't handle the tasks and it becomes necessary.

I'd also check out SMILE as an alternative to Spark MLlib in Scala.


Algorithm to find Unique users from their transactions Scenario
An eCommerce site allow users to purchase items without creating an Account. However, it captures following attributes from the Purchases based on the type of payment and type of delivery:

Name
Address
Email Address
Card/Account Number

Note: It is possible one type of transaction only get one subset of attributes (e.g. Name and Address) and another type of transaction get another subset of attributes (e.g. Name and Card Number)
Expectation
To offer customized products or to prevent Fraud, it want to find individual users.
E.g. It should identify following three transactions are done by single user:

name: Alice, email: alice@wonder.land
email: alice@wonder.land, Card Number: 123456
name: Bob, Card Number: 123456 (Alice made purchase for Bob).

As you can see car-number/email is a stronger link than Name. So different attributes have different weights.
I have looked at couple classification algorithms (k-mean, k-mode..), but none of them is addressing this use case as I don't want to classify users but identify individual users.
Is this a standard data science problem and which algorithm will best fit here? Thank you!

You don't need any ML here: in the question you just gave the full algorithm to find the target output ;) At best a ML system would find just the same method after using a lot of annotated data.
The method is essentially what you said:

Group the users by their card number or email (easily done with a map for each), this gives you the cases of "sure" identical users.
Group the users by their name, this gives you the cases of "possible" identical users. For this part you might want to use string similarity measures.


Build a Generalised Regression Model Containing Linear and Non Linear Predictor Variables with a Target Variable in Python Imagine a dataset having five predictor variables and a target variable, through scatter plot I observed three predictor variables having a linear relationship with the target variable and the other two having a nonlinear relationship.

How can I build a Generalized Linear Regression model in such a way that the non-linearity of the two variables is explained along with linear relationship of the other three variables?

I suggest using "Generalised Additive Models". These type of models are linear but can treat wild non-linearity. The idea is - e.g. with regression splines - that a number of linear regressions are "stacked", so that they can jointly account for highly non-linear effects.

Here is a Python implementation: https://pygam.readthedocs.io/en/latest/

When you are bound to linear regression (OLS), you can add polynomials to the regression. In this case you simply generate a new "column" in your data frame, containing e.g. $x^2$. You can add this variable to the regression directly because linear regression is additive:

Example:

$$ y = \beta_0 + \beta_1 x + u $$

...can be augmented by a squared term for $x$...

$$ y = \beta_0 + \beta_1 x + \beta_2 x^2 + u $$

... and this also works for $x^3$ (and so on) or you can take $log()$ etc.

With GAM you don't have to decide on how to model non-linearity. That is the great advantage of GAMs. When you stick to OLS, you need to check if the non-linearity (imposed by you) really helps to improve fit and/or prediction.

GAM are very well explained in "Introduction to Statistical Learning" have a look at Chapter 7. There also is Python code for the Labs in the book.


Merging three different customer segmentation systems into one I have been given a task where I have three existing customer segmentation systems (rule based e.g. if customer spends X in Y amount of time AND whatever then put in top spender segment is one segment, another is measured on metrics such as time spent on the site, etc and the last one is the lifetime loyalty segmentation system).
I have been asked to merge into one customer segmentation system.
I would like to ask your opinion or sources of information to decide whether I keep this rule based system (maybe a point based system) or use another method such as clustering. They want it to be flexible, thus be able to add additional layers as time goes by. They must be able to cater for different markets, which have different 'rules'

DO you believe I should keep the rule based system? Use clustering? What are the disadvantages of merging all three current customer segmentation systems?

You are supposedly supposed to merge the existing three systems.

I think it is a fairly safe bet they don't want you to add a fourth, that is unstable in the sense that as data changes, segments will change.


Why are there over 600 open pull-requests in Scikit-learn github repo? I considered submitting a function that I deem missing to scikit-learn repo, but as of June 28th 2019 there are over 600 pending Pull Requests: https://github.com/scikit-learn/scikit-learn/pulls.

Can you explain the situation with maintainers? I mean, reviewing a pull-request is not that big of a deal compared to writing commits and also this normally has higher priority than writing code, so why I see new commits in the project, but PRs are still not reviewed?

By the way, I also found a scikit-learn-contrib project that seems to be contain many additional functions: https://github.com/scikit-learn-contrib.

mean, reviewing a pull-request is not that big of a deal compared to writing commits and also this normally has higher priority than writing code, so why I see new commits in the project, but PRs are still not reviewed?

I assume that it's not that simple if they want to maintain the stability and the homogeneity of their code base. If they were just quickly accepting PRs the whole software would become an unmanageable mess in no time, causing problems for a large community of users. It's almost unavoidable for any significant software project to become harder and slower to maintain and develop when it reaches a certain size, especially if many users depend on it.

Also it's maintained by a community of volunteers so there might not be that many contributors who are both knowledgeable enough about the code and have enough free time to study all the PRs coming regularly.


What should be the value of non-rated field when finding cosine similarity I am working on a very basic book recommender system. I want to know what to do with the fields which aren't rated by the user when finding cosine similarity, should we ignore them and calculate only with the rated fields or should we mark them 0.

The book I am following says to exclude the fields since it will give wrong interpretation in case of Euclidean and Pearson Correlation but in case of Cosine Similarity, it makes all the non-rated fields to 0.

Can someone explain why is it needed to make the non-rated fields to 0 only for Cosine and not others or is there a different way to do it.
(I know how making fields 0 in Euclidean and Pearson affect the output but not sure of cosine)

Implementation-side, there is a good reason to make 0 correspond to not rated.  Since most users haven't rated most books, 0 will be the most common value and the cosine similarity function can use sparse matrices internally to speed up the computation.  The sparse matrix shortcut is the main reason why people use cosine similarity in the first place.

On the other hand, it will throw off your recommendations if the algorithm interprets a 0 to be closer to a 1 than a 2. You do need to intentionally ignore those, but you can easily do so inside your cosine similarity implementation rather than filtering the data before applying it.


Point of dropping weights in mini batch for purpose of regularization I have been reading "drop" is a method to regularize model better. It's purpose is to update only some % of weights in backprop and it helps you to not over fit the model. But I am wondering, is this method useful in minibatch at all?

Yes dropout is useful this case too. The purpose of dropout is to make the model generalize better by causing it to use all available weights from a given layer. It is supposed to make the subsequent layer not rely on a small number of activations from the previous layer, but to use them all.


Scikit MLPClassifier vs. Tensorflow DNNClassifier I am trying to understand the differences between Scikit MLPClassifier and Tensorflow DNNClassifier for classification task and hoping that some experts can share a light. as far as I understand, both supports backpropagation, activation functions (inc. relu), optmizer (sgd/adam). MLPClassifier can also have deep neural networks by specifying the # of hidden layers and nodes.  The only difference between two that I can see is DNNClassifier supports GPU training while MLPClassifer does not. Outside of GPU support, are there other differences between them? Why do one want to use Tensorflow's DNNClassifier for basic deep neural network training (I am talking basic feed-forard NN here not the CNN, RNN, LSTM, etc.) than Scikit's MLPClassifier?

Thanks!

Maybe you want to use a more complicated activation function - i.e. leakyReLU or you want to add batch normalization. From a brief look at the API there is no option for batchnorm in SKlearn. Also it appears that you can't use dropout with MLPClassifier.

You might also feel more comfortable with tensorflow features - i.e. tensorboard, or you might feel more comfortable with SKlearn. To each his own.


Can I use xgboost on a dataset with 1000 rows for classification problem? I have used all types of classification algorithms on my dataset yet I couldn't improve my score no matter how I try.

So I've read about Xgboost classifier. So I was wondering is it practical to use xgboost on a dataset with around 1000 rows.

Please let me know.

Yes, XGBoost is famous for having been demonstrated to attain very good results using small datasets often with less than 1000 instances.

Of course when choosing a machine learning model to fit your data, the number of instances is important and is related to the number of model parameters you will need to fit. The greater the number of parameters in the model the more data you will need to reduce the bias of your final model. If you do get good results using a complex model on very few instances then there is a high probability that you are overfitting. For example 1000 instances is hardly enough to fit a deep neural network.

That being said, the distribution of your classes and the noise in the data is always going to be a limiting factor to how well any model you select will fit your data.


Clustering or classifing n-gram-based text categories I have large set of data records looking like this:

"text", "category"

I extract n-grams from text (2-, 3- and 4-grams) and store count of each n-gram per category, like so:

"ngram1", "category1", 1000
"ngram1", "category2", 20
"ngram1", "category3", 15
"ngram2", "category1", 25
"ngram2", "category2", 550
"ngram2", "category3", 600

Is there a clustering or classification algorithm that could help with finding groups within categories, based on counts of same n-grams within categories?

In the above example that algorithm should be able to show that "category2" and "category3" are part of the same group.

If your ultimate goal is to cluster similar categories and assuming that you have labels of each text as category1, category2,...,categoryN from 1 to N, a bag of words method would be sufficient in order to create features so that you can run multiple desired clustering algorithms.

K-means can be a good starting point for getting similar groups of text document categories, you can try out different k for more sensible outcomes.

Methodology that I would adapt here, forgive me if understood your problem wrong, would be to concatenate all the textual data for a given category and create 1,2,3,..n gram vectors. The values of these vectors can be TF-IDF of a given n-gram, and so on. Still different feature matrices can be formed to compare category vectors but TF-IDF is a pretty common way to do it in.

After acquiring n-gram feature vectors for each text category, you can apply k-means algorithm with several distance metrics to find similar categories.

Suppose we have 5 categories of text as given:

Soccer Games
Basketball Games
Politics
Movies

The proposed method can capture the similarity between soccer and basketball games as they are both sports.

Hope this can help


Is there a process flow to follow for text analytics? I am trying to draw a process flow (like a template) to be followed while on text analysis projects. So far, I've come up with this.
Text Analytics Steps

Data Collection

Acquire data
Convert data into plain text

Remove Duplicate Entries
Text Parsing and Extracting Features

Tokenization
Parsing

Remove HTML characters
Decode complex symbols to UTF-8
Spell check
Apostrophe look-up
Remove punctuation marks
Remove expressions / emojis
Split attached words
Slangs look-up
Remove URLs

Lemmatization / Stemming (Normalization of Tokens)
Parts-of-Speech Tagging

Text Filtering

Remove start-words
Remove stop-words
Remove irrelevant words based on frequency

Text Transformation

Bag of Words Representation
TF-IDF

Text Mining / Analysis (whichever analysis needed)

Text Categorization
Text Classification (supervised)
Topic Modeling (unsupervised)
Text Clustering
Similarity Analysis
Sentiment Analysis

Is this flow in the right order of steps?
What are the steps/sub-steps that I am missing?
Does the process flow look like a template or go-to flow chart when undertaking any text analytics project?
Edit: Updated process flow

This is a great place to start! While not catalogued in a "process flow", Daniel Jurafsky's book, "Speech and Language Processing" talks through the various calculations and steps related to analyzing text that you will find useful.

The reason I say that a process flow is not provided is because Jurafsky - in great detail - explains the pros and cons of particular methods applied throughout a pipeline, and how this could change results. As an example, when calculating perplexity (an inverse metric that quantifies how well a language model can predict the next word in a statement), you should capture beginnings, ends, and stop words of statements - as opposed to other methods that require the removal of stop words.


Which Outlier Detection Method? Why? For detecting an outlier in a vector I have tested different well known outlier detection methods. Finally, I used combination of different methods and an agreement between those methods. Now, a person asks why did you choose this combination and algorithms!? You can reach different combinations and use other algorithms and they may yield better results. What should I answer? I cannot just say based on tests as there are many other algorithms that I haven't tested (cannot test all algorithms). It is not a logical response, I think.

I'm looking for tests to justify my selected methods and combination and say why I have selected these methods.

Please let me know your suggestions.

You can justify your choices by using data.

Treat the anomaly detection like a supervised learning problem where the concept is being anomaly.
Then you'll be able to present - for each method - its confusion matrix.
Not only it will be a good justification, it will enable to understand the expected results.

Many times, we have models and we wonder which confidence threshold we should use for alerting. In the supervised learning framework you'll be able to do trade-off like "increasing the confidence to X will lead to a better precision Y yet a decrease of the recall to Z".


How to do time series regression without scikit and numpy in Python? On a recent Hackerrank interview I was faced with the following problem:

Given a set of timestamps (format 2019-11-26 11:00) and their corresponding stock prices (single float value), approximate the missing stock prices for a set of timestamps.

I tried solving it with an SVR model at first, but I had to realize that none of the usual data science libraries were available for this test.

How would you solve this problem without data science libraries in a limited amount of time?

I ended up doing the following: Took the nearest available measurements and solved the y = mx + b equation based on them, so my prediction would be the corresponding y value for my x.

This seemed to solve the problem quite well for the given test case. My solution in Python

In case you want to perform a simple time-series regression without using any packages such as Numpy etc, you need to write and solve the model yourself. You can either use gradient descent or least squares to solve the model.

Let's look at a least squares solution. A simple model (omitting sibscripts) would look like:

$$ y = \beta_0 + \beta_1 y_{t-1} + u,$$

where $y$ is explained by a lag $y_{t-1}$ and $u$ is the error term (you can also use more lags if you want). You can "combine" this into a matrix with first column equal to one (the intercept) and the second column containing the lagged $y_{t-1}$. In matrix notation, the model looks like:

$$ y = \beta X + u .$$

Now you can solve by:

$$ \hat{\beta}=(X'X)^{-1} X'y.$$

Now it should be relatively easy (but still some work) to solve the problem without using packages such as numpy.

As an alternative to matrix notation and gradient descent, you can also solve a linear regression by other means, e.g. as demonstrated in this post.


How to release datasets with fingerprinting I intend on monetising some large datasets. These datasets are anonymised and released to (paying) clients via a web api. Are there any standard algorithms such that if the datasets are intentionally leaked publicly, the data can be altered such that the responsible party can be identified, while at the same time the data remains practically useful?

There are certain approaches which come to mind, such as every client's data being very slightly different with known changes. For example in spatial data, every lon/lat pair is altered by the same very small vector. My worry is that if the data is anonymised again by the client before being leaked, a naive attempt might easily be circumvented.

(I am not a data scientist so I'm not really sure what the correct jargon is for what I am looking for)

"Digital watermarking" is a set of techniques that might be useful in this context.

From the wikipedia page:

  "Watermarking" is the process of hiding digital information in a carrier signal... Digital watermarks may be used to verify the authenticity or integrity of the carrier signal or to show the identity of its owners. [Emphasis added]

To address your requirement, you would insert a unique watermark for each client that receives your data. Watermarking techniques address requirements such as robustness to modification and imperceptibility.

As an example, this paper talks about watermarking numerical data: "Watermarking Numerical Data in the Presence of Noise", pdf


How to determine what algorithm to apply in the following dataset (included) I have a dataset that I will be using to build a classifier on. Below I have plotted the First and the Second Principal Component of the data using sklearn.preprocessing.PCA. Since the two different classes are not well separated a linear classifier will not work here.

My question is which classifier would be best for this scenario.

My research brought be to KNN. But My intuition says the class ratio is highly imbalanced a large value of k in KNN would always tend towards the larger class count. It will be a nightmare to train it on SVM since therw are to many observations in the dataset and it will take too long.

Note that doing a dimensionality reduction with the target can lead you to the manifold problem. You can see in the image. What ends up happening is that the target information is lost.

The information that you provide is not enough to make a guess of what algorithm will be better.

Normally reducing the dimensionality of the problem to a lower dimension space in order to plot where you want to have the decision boundaries is not a good idea.

It's really hard to have a human understanding of how do these algorithms do the decision boundaries in a large scale dimension, that is why it is just better to have an empirical approach. Try a few of them, select a metric, and choose the one that has a higher score.


How can I get probabilities of next word with ELMO? ELMO is a language model, build to to compute the probability of a word, given some prior history of words seen. How can I get this probability from pretained ELMO model?

ELMO is not a language model. As specified in the paper:

  We introduce a new type of deep contextualized word representation that models both (1)
  complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses
  vary across linguistic contexts (i.e., to model
  polysemy).

ELMO representations (embeddings) are indeed calculated by looking at the context of the word. But the output is a fixed size vector that contains/represents the contextualised meaning of that word. This can be compared to other word embeddings such as word2vec, glove or fasttext but with the novelty of having different representations of the same word based on their context within a sentence.

I suggest you do some reading about the differences about language models and word representations.


How to get feature importance from a keras deep learning model? In case of scikit-learn's models, we can get feature importance using the relevant attributes of the model.
I've been working on a RNN, using LSTMs for text embedding.
Is there any way to get feature importance of various features from the finalized model?

There is plenty of methods to calculate feature importance. I recommend trying two of them LIME and SHAP.

I don't want to copy-paste material and tutorial provided by the author so please refer to these two repositories.


Is there any measure to find how much classifiers are confident? Assume that we have two classifiers (C1, C2) and two classes (A, B). These classifiers give us the belonging probability for each class for each instance. Suppose that we have an instance X which is actually should be classified as A. C1 classification result is (1, 0) and C2 classification result is (0.9, 0.1) which means they both classified X correctly as A. Obviously C1 is more confident. Is there any measure that I can use to compare my classifiers based on that?

There are many measures which implicitly take into account the confidence of a prediction. One very common one is Log Loss (also called Cross Entropy).

$-log \space P(y_t|y_p) = -(y_t \space log(y_p) + (1-y_t)\space log(1-y_p))$

Using this metric, confident correct classifications are rewarded more than relatively less confident correct classifications, and confident misclassifications are heavily punished.

Any proper scoring rule meets this criteria when applied to a classification problem.

Some others examples are:

Surprisal
Brier score (as mentioned by darXider in his comment)
Spherical Scoring Rule
Logarithmic Scoring Rule

And an infinite number of other, related functions.


Designing a ConvNet to facilitate game playing For fun I want to design a convolutional neural net to recognize enemy NPCs in a first person shooter. I have captured 100 jpegs of the npcs as well as 100 jpegs of not-NPCs. I have successfully trained a really simple convNEt to identify NPCs. This was really easy because the game actually highlights the NPCs with a red marker to let humans identify them. Makes it SUPER easy for a machine learning algorthm to find them.

Great , so now I can classify a screenshot of an NPC. The next step is to identify these in a data stream at 60 frames per second. We all know that the stupid little processors inside most cameras have a face detection algorithm that operates in real time. So my i7 with 2 NVIDIA gpus can do this no sweat. So now I have to grab the screen buffer, capture a screen shot, feed it to my conVnet, get the location of the NPC, and then move the mouse cursor to the center of that NPC.

Are there any easy to follow tutorials of running a convolutional neural net on a data stream like this?

I've recently started using OpenCV's python implementation, and I found some good OpenCV tutorials on this website: http://www.pyimagesearch.com/ that I really liked. OpenCV allows you to do Haar Cascades for fast facial recognition (by default it doesn't use a convoluted neural network but an optimized implementation of Ada Boosting that evaluates frames in stages for faster processing). OpenCV converts each frame into a multidimensional numpy tensor/matrix that you can then feed into into your ML algorithm (e.g., in TensorFlow or some other library), although I think most people just use the built-in OpenCV face classifiers. In any case, I believe OpenCV can process up to 70 frames per second, so it should be fast enough for you.

The original paper that invented Haar Cascades:
https://www.cs.cmu.edu/~efros/courses/LBMV07/Papers/viola-cvpr-01.pdf

The OpenCV documentation that further explains Haar Cascades:
http://docs.opencv.org/3.1.0/d7/d8b/tutorial_py_face_detection.html#gsc.tab=0


Bandwidth selection Kernel Density Estimation I want to do KDE on data that are not necessarily normal using Gaussian kernels. In KDE in wikipedia an expression for the bandwidth is given when the underlying distribution of the data is gaussian. I am looking for a similar bandwidth selection for non-normal data.

There is vast literature for bandwidths. Different bandwidth perform good in different situations. Its not necessary to use Rule of Thumb bandwidth with normal data. For instance, see Silverman 1986. "Density Estimation".


Metrics for Evaluating Performance of Logistic Regression I built a Logistic Regression model and I would like to evaluate the performance of the model. I would like to understand its evaluation metrics.

What do the metrics Sensitivity, Specificity, False Positives Rate, Precision, Recall, and Accuracy tell us about this model?

Since Logistic regression is not same as Linear regression , predicting just accuracy will mislead.
** Confusion Matrix** is one way to evaluate the performance of your model. Checking the values of True Positives, False Negatives ( Type II Error) are really important.

** ROC Curve** Receiver Operating Characteristic(ROC) summarizes the model’s performance by evaluating the trade offs between true positive rate (sensitivity) and false positive rate(1- specificity)

The below link will give you more information:

https://www.analyticsvidhya.com/blog/2015/11/beginners-guide-on-logistic-regression-in-r/


Ways to reconstruct shuffled pixels of a video file? Suppose that you have a video file which pixel order has been shuffled once. That is, a random order have been defined once and applied to all frames.

Does it exist some known approach for retrieving the initial order of pixels?

I have some ideas around retrieving the initial topology by placing pixels which values are correlated in space and time closer together. I wonder if this has been studied and if efficient algorithms were published.

Also this problem can be thought of as a way to project to a 2D matrix a set of values varying in time in order to be able to apply computer vision techniques (like CNN), with the assumption that these values are indeed somehow correlated.

A general solution to this does not exist, even if we add some assumptions about the distribution of e.g. colours and shapes in the images or temporal coupling such as consecutive frames being similar.

Problem

Let $F_1,\dots,F_i$ be the $n$ original frames, each with $m$ pixels. Let $P$ be the permutation that is applied to the pixels of each frame before we get them. You can think of $P$ as the enemy's code-book.

Now, as input we are receiving $P(F_1),\dots,P(F_n)$. The goal is to find the inverse permutation $Q$ to restore the images. Thus $QP=I$ is the identity map and for example $Q(P(F_1))=F_1$. Note that we do not know any of the correct frames $F_i$.

Let $Q_1,...,Q_{m!}$ be the $m!$ possible permutation functions of the $m$ pixels.

The goal is to select the unique $j\in\{1,\dots,m!\}$ so that $Q_jP=I$.

No General Solution

Under our statistical model this means selecting the $Q_j$ which maximises the likelihood that $Q_j(P(F_i))$ is drawn from the same distribution as the reference statistics for images and the temporal statistics between consecutive frames $Q_j(P(F_{i})$ and $Q_j(P(F_{i+1})$ which is our prior knowledge.

There is a canonical counter-example where the enemy gives you a scrambled movie with two frames where all the pixels are the same colour, so $n=2$, $F_1=F_2$ and $Q_j(F_1)=Q_j(F_2)=F1=F2$ for every $j$. Thus, for all $j$, the in-frame and inter-frame statistics are equiprobable for each $j$ and give us no information to select the maximum likelihood permutation $Q_j$ (except in the degenerate case where $m!=1$).

Thus, we cannot guarantee uniqueness and the problem is unsolvable without further assumptions.

Further Assumptions

It is interesting to see if we can solve the problem by adding more constraints.

If we restrict the enemy to only sending us "real" movies and assuming there are enough different pixels and frames to so that a unique $Q_j$ with maximum likelihood exists, we would still have to calculate statistics for $O(m! \times n)$ permuted frames to find the maximum.

This is brute force code-breaking.

In order to benefit from neural networks, and back-propagation in particular, we would need a differentiable loss function with respect to the input (which is an encoding of $j$ or our permutation $Q_j$). The question then, would be to see if such a function can be found.

Otherwise the problem is more similar to cryptanalysis in the special case where we know that the enemy's code book is a permutation of the clear-text (or clear-image).


What kind of "vector" is a feature vector in machine learning? I'm having trouble understanding the use of Vector in machine learning to represent a group of features.

If one looks up the definition of a Vector, then, according to wikipedia, a Vector is an entity with a magnitude and direction.

This can be understood when applying Vectors to for example physics to represent force, velocity, acceleration, etc...: the components of the Vector represent the components of the physical property along the axes in space. For example, the components of a velocity vector represent the velocity along the x, y and z axes

However, when applying Vectors to machine learning to represent features, then those features can be totally unrelated entities. They can have totally different units: one feature can be the length in meters of a person and another can be the age in years of the person.

But then what is the meaning of the Magnitude of such a Vector, which would then be formed by a summation of meters and years? And the Direction?

I do know about normalization of features to make them have similar ranges, but my question is more fundamental.

I'm having trouble understanding the use of Vector in machine learning
  to represent a group of features.

In short, I would say that "Features Vector" is just a convenient way to speak about a set of features.

Indeed, for each label 'y' (to be predicted), you need a set of values 'X'. And a very convenient way of representing this is to put the values in a vector, such that when you consider multiple labels, you end up with a matrix, containing one row per label and one column per feature.

In an abstract way, you can definitely think of those vectors belonging to a multiple dimensions space, but (usually) not an Euclidean one. Hence all the math apply, only the interpretation differs !

Hope that helps you.


Effect of adding extra unrelated features to linear perceptron Suppose that we are training a linear regressor (perceptron). Adding extra features that are not related to the target (e.g. randomly generated values) before training will typically ____ our training error.

Would it typically increase, decrease or stay the same?

Listed Answer: decrease

The answer to this question is listed as decrease, but wouldn't adding extra unrelated features not change the training error, because the model would just learn to ignore those features (add 0 weights to them). Even if the model didn't learn to ignore the extra features, if anything wouldn't the training error increase as it doesn't seem like the model would be able to learn a relationship that would benefit the training error from a completely random unrelated feature?

If you add a new feature to the perceptron, the perceptron actually gets one more parameter. So in some sense it is not the same model anymore. But lets ignore that.

Correlation is the key problem. Assume the new feature is a random number which happens to have positive correlation with the desired output for the training dataset, but negative correlation with the test dataset. Then you would expect this new feature to influence the test error negatively, right? At least I would.

But your question was about the training error. In that case I actually don't see a reason why it should be the case. I'd suggest to try it.

And anyways, ask the person/organisation, where you found this question. (And post the answer back)


Use Feature Constructor in Orange to extract number from string? I would like to use the Feature Constructor to extract numbers from a column of strings.

 Input          Output
S.c 3224   ---> 3224
South      ---> 0
322523     ---> 322523
S-1c 4     ---> 14
-1.37      ---> 137

Internal Orange 3.7.0 Documentation says For continuous variables you only have to construct an expression in Python.  Great!

I found information about allowed functions in the feature constructor code:

#Only expressions with no list,set,dict,generator comprehensions
#are accepted.
<...>
__ALLOWED = [
    "Ellipsis", "False", "None", "True", "abs", "all", "any", "acsii",
    "bin", "bool", "bytearray", "bytes", "chr", "complex", "dict",
    "divmod", "enumerate", "filter", "float", "format", "frozenset",
    "getattr", "hasattr", "hash", "hex", "id", "int", "iter", "len",
    "list", "map", "memoryview", "next", "object",
    "oct", "ord", "pow", "range", "repr", "reversed", "round",
    "set", "slice", "sorted", "str", "tuple", "type",
    "zip"
]

Based on the list, I was hoping to use an expression like this:
int(filter(str.isdigit, inputStringColumn) or 0)

...but that is an "Invalid Expression" (no further feedback)

How can I extract a number from a string?

Just to highlight Adam's timesaving answer
int(''.join(filter(str.isdigit, "0"+str(inputColumn))))


Xgboost multiple class predictive performance beats one versus rest I have an NLP task I'm tackling with xgboost (R implementation).

Before describing my doubt I'll give you some background:

I have a corpus of documents for which I did topic discovery, using a term x term matrix clustering approach. For each document, I get a topic score computed using the terms in the document (with a TfIdf score). Then for each document, I pick up the topic with the highest score.

The following step is to create a model that given the term x document score matrix and the best topic per document, predicts the best topic.

I tried two different approaches:

a multiple class model, where a topic is associated with each document;
a one versus rest series of models, one per topic, where each document is labeled as belonging or not to a topic.

Here are the results of the two approaches, using AUC:

    i                 topic    single     multi
1   1             Topic.nv1 0.9564445 0.9880821
2   2  Topic.nv10_Topic.wv9 0.9848492 0.9969546
3   3            Topic.nv11 0.9174293 0.9741100
4   4 Topic.nv12_Topic.wv11 0.9874073 0.9967725
5   5 Topic.nv13_Topic.wv10 0.9509909 0.9916768
6   6 Topic.nv14_Topic.wv12 0.9864622 0.9959161
7   7            Topic.nv15 0.7333333 0.9333333
8   8   Topic.nv2_Topic.wv3 0.9590279 0.9877953
9   9   Topic.nv3_Topic.wv5 0.9448966 0.9879057
10 10   Topic.nv4_Topic.wv2 0.9521490 0.9908656
11 11   Topic.nv5_Topic.wv6 0.9761665 0.9946294
12 12             Topic.nv6 0.9439377 0.9889028
13 13   Topic.nv7_Topic.wv4 0.9656248 0.9926163
14 14             Topic.nv8 0.9673726 0.9944970
15 15   Topic.nv9_Topic.wv8 0.9716538 0.9929586
16 16             Topic.wv1 0.9610704 0.9925414
17 17             Topic.wv7 0.9765398 0.9904255

It is visible that the multiclass approach systematically outperforms the one vs rest one. NB: These are training set performances.

Is there a clear theoretical reason for this?

Multiclass models in XGBoost consist of n_classes separate forests, one for each one-vs-rest binary problem.  At each iteration, an extra tree is added to each forest.  But it isn't actually a one-vs-rest approach (as I thought in the first version of this answer), because these trees are built to minimize a single loss function, the cross-entropy of the softmax probabilities.
https://discuss.xgboost.ai/t/multiclassification-training-process/29
https://github.com/dmlc/xgboost/issues/806
https://github.com/dmlc/xgboost/issues/3655

In general, the one-vs-rest models are very good at identifying the single class, whereas the multiclass model has to balance performance on all of them.  More specifically, I think that the softmax may be responsible for the phenomenon you're displaying.  (I'm still thinking about it, but I thought I should post the above for now.)

Suppose one of your documents is reasonably likely to be in either of two topics: the probability scores given by the forests are 0.9, 0.85, then <0.1 in all the rest.  In your topic-1 model, you make a fairly confident judgement that this document is of topic 1 (score of 0.9).  But in the multiclass ensemble, you see things as much more uncertain; maybe the model applies softmax, so that the probability of topic 1 is only ~0.5.

More extreme, suppose the individual topic model scores are all 0.9.  Now the multiclass ensemble applies softmax and produces equal 1/17 probabilities for each topic!

In the other direction, suppose one of your documents is judged unlikely to fit any of the topics: all the individual topic model probability scores are 0.01.  In the multiclass ensemble, that gets scaled up to 1/17 (OK, 17 topics makes this a harder sell).

Hrm, except how likely is it to get the 0.9 and 0.85, since a training sample in one of these two topics will be pushed toward 0 by the other model... ?  Especially when your scores are fairly high, so it's not like the models have huge blind spots.
(This part still causes a problem with the correct understanding of how XGBoost works; the log-loss of the softmax probabilities still penalizes being confident about belonging to two different classes...)


How does the number of trees effect the prediction time in gradient boost classification trees? After tuning hyper-parameters for a gradient boosted model, I have found that the best tree count (iterations) is a few thousand.
I'm worried that such a high count might impact prediction performance.
Can someone explain the relation between trees count and prediction time?

The time it takes to get a prediction from a model of gradient boosted classification trees should be linear in the number of trees. So getting predictions from a model with 1000 trees should take about twice as long as 500 trees, and about half as long as 2000 trees.

You'll need to test it yourself and check if it's fast enough for your use case. Modern libraries like xgboost can handle high numbers of trees with remarkable efficiency. One thing to keep in mind is that it's generally faster to get predictions for a whole block of test examples at once than it is to get the predictions one at a time.


How to use correct weights in linear regression model I'm trying to understand can we implement a simple linear regression model.

Let's say we are predicting price currencies. We want to know whether the currency will raise or not.

As i understand, we need to define two vectors for this:

$x=[1,2,3,4,5,6,7,8,9,10,11,12]$ - months

$y=[2.30,2.33,2.29,2.30,2.36,2.40,2.46,2.50,2.48, 2.43,2.38,2.35]$ - average prices.

Let's plot this:

Before sigmoid separator, i will try a simple linear separator.

I'm guessing that at first, I need to choose some random slope (and bias will be the smallest scalar in vector).

$f(x) = 0.026x+2.3$

As we see, the separator is inaccurate, let's try the quadratic cost function at f(1):

$C = \frac{1}{N} \sum_{i=0}^{N}(\hat{y} - y)^2$

$C = \frac{1}{1} \sum_{i=0}^{1}(2.3259999999999996-2.30)^2=0.0006759999999999897$

It seems accurate in the beginning, but it gets worse as it progresses, so somehow i need to improve it.

From my knowledge, the next step is to find the derivative of the function.

Normally, gradient descent algorithm is used for this, but finding a slope of the tangent line is very easy here:

$\frac{dy}{dx} = 0.026$

What is the next step? How can i use this derivative to use proper weights to minimize the cost function?

You are correct for a linear separator line the derivative seems trivial. In this case gradient descent is not necessary because a closed form solution exists for the weights.

$w = (X^TX)^{-1}X^Ty$

We only use optimization techniques such as gradient descent for models where a closed form solution does not exist. However, if you perform gradient descent on the weights using the derivative and doing the following equation iteratively

$w^{new} = w^{old} - \nu \frac{dy}{dx}$

would yield the same result as the closed form solution. This is however very unnecessary when not necessary.


I'm not getting the no of parameters concept in CNN Hi guys i've attached two images of question please help me on solution.

Thank you


What is the difference between "Adding more LSTM layers" or "Adding more units on existence layers"? What is the difference between adding more LSTM layers and just increasing the units of existing layers? Which one is preferred and in which situation?

With the layers - you are trying to generate higher and higher level features as you proceed from layer to layer. Based on the problem at hand, anything from 2 and above layers could serve the purpose. Given that LSTMs operate on sequence data, it means that the addition of layers adds levels of abstraction of input observations over time. In effect, chunking observations over time or representing the problem at different time scales. This works!
With the units - you are trying to capture the function in a better way. Too less and you could underfit. Too many and there could be overfitting.
By adding more layers, you can reduce the number of units in each layer and bring about a greater 'depth' instead of 'width'. What this simply means is that you can approximate the function better using a 'right' combination of width and depth. To arrive at 'right' there is no option other than to play around for hours with different combinations.
One option is the 'stretch pant' approach. Take much higher sizes than when you actually need. Then use regularisation techniques to fit better.


Filling missing values for important features So I have this dataset from census the census bureau database which contains 40 attributes and a target column specifying if the total income is >50k.
P.S : this is not the known ADULT dataset (which is a lot cleaner), the difference is that mine wants to predict the total person income rather than the adjusted gross income.

My problem is that some of the attributes such as : class of worker (meaning if he's self employed, local government...),  major occupation and and industry codes have 100 000 missing values knowing that the total length of my dataset is 150 000.

Now it bugs me to simply drop those column because it seems to me like an important information to know if we want to predict the total income.

What should I do in such a position ?

Thanks !

I would first build some models without those predictors. If 2/3 of your training set is missing so you can't have your basic model based on them. If your accuracy is too low I would try to use a smaller data set and not use the missing values. Those kind of missing values are not somthing that you can easily define (like a default job) unless you have a prior on your data (e.g most people in the data are data scientists). If you do have any kind of such prior you can use missing data methods more efficiently.


Simple Imputer cannot impute by column I have X_train that shapes (14599, 13), i'm trying to impute NaN with column's median but somehow it imputes with row resulting error because in a row there are date, and other than integer values. I already lookup if SimpleImputer has axis parameter but could not find that it exists. How to solve this?

import pandas as pd
import seaborn as sns
import numpy as np
import matplotlib.pyplot as plt
from sklearn.impute import SimpleImputer
from sklearn.model_selection import train_test_split

plt.close('all')
avo_sales = pd.read_csv('avocados.csv')
avo_sales.rename(columns = {'4046':'small PLU sold',
                            '4225':'large PLU sold',
                            '4770':'xlarge PLU sold'},
                 inplace= True)
avo_sales.columns = avo_sales.columns.str.replace(' ','')

plt.scatter(avo_sales.Date,avo_sales.TotalBags)

x = np.array(avo_sales.drop(['TotalBags'],1))
y = np.array(avo_sales.TotalBags)

X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)

imp = SimpleImputer(strategy='median')
X_train = imp.fit_transform(X_train)

the output

ValueError: Cannot use median strategy with non-numeric data:
could not convert string to float: '12/31/2017'
```

I don't think it's trying to impute across rows; rather, it's trying to impute in the dates column.  You may want to use ColumnTransformer to select which columns to impute.


Data discrimination after clustering My task consists of two points:

1) Make data clustering;

2) Assign new data to the resulting clusters;

I wanted to highlight the boundaries of clusters as min/max values for each coordinate of an observation belonging to the cluster, then assign observations from the new data to a particular cluster in accordance with its boundaries. However, the problem is that cluster boundaries intersect and each observation can belong to several clusters.
What adequate methods can be used to discriminate data in this case?

If your definition of points belonging to a cluster is simply the points closest the cluster centroid, then the boundaries cannot overlap. The point assignments are a Voronoi map like:

(Source: https://www.quora.com/What-is-the-difference-between-K-Means-and-Voronoi/answer/Ethan-Brooks-3)

The closest centroid, and thus assignment, is unambiguous.


How to clone Python working environment on another machine? I developed a machine learning model with Python (Anaconda + Flask) on my workstation and all goes well. Later, I tried to ship this program onto another machine where of course I tried to set up the same environment, but the program fails to run. I copied the program to other machines where it also runs smoothly.

I cannot figure out what the problem is in the failed case (both the program code and the error message are copious so I am not able to present them here) but I'm almost certain that it is something with the different versions of the dependencies.

So, my question is that given an environment where a certain program runs well, how can I clone it to another where it should run well also? Of course, without the cloning of the full system ;)

First of all this is a Python/Anaconda question and should probably be asked in a different stack exchange subsite.

As for the question itself - you can export your Anaconda environment using:

conda env export > environment.yml

And recreate it using:

conda env create -f environment.yml

Please note that as others suggested - you should use virtual environments which allows you to create a certain environment that is separated from that of your machine and manage it more easily.

To create a virtual environment in Anaconda you can use:

conda create -n yourenvname python=x.x anaconda

which you activate using:

source activate yourenvname


Input data of variable length - two scenarios I'm trying to figure out how I could train a neural network with inputs that have variable length. This issue comes up in the following 2 scenarios I'm trying to solve.

Scenario 1:
I have a long list of running distances for various runners which looks something like has 3 columns: runner, date, distance.
Obviously some runners have a lot of entries and others don't. I'm trying to make predictions on the number of miles a given runner will run next. So I'm guessing i need to transform my data to have one line per runner, which gives me variable length features. How can I deal with this in a ML application?

Scenario 2:
I'd like to take various strings ("teststring", "P@ssword", "NotAPassword123", etc...) and classify it as a password or not. I guess i'm trying to figure out how to a) convert strings to numbers to train on and b) how to deal with the fact that they have variable length.

Thanks for reading this far...

For scenario 1, if you have enough data for each runner, you could build separate models for them otherwise you can add runner as a categorical variable by one hot encoding your runners and then trying out your model.
For scenario 2, you can create fixed sized vectors for each string by handcrafting features, such as count of consonants, count of vowels, presence of special characters, etc.


Can a neural network recognize a letter B as an A if your trained it so? You have a neural network. And you have, say, pictures of $100,000$ hand-written letters (A-Z). Now you make a typical Training and the neural network will recognize an A as an A, a B as a B, ...

Now you repeat Training but with a change: you tell the neural network for all pictures with the letter B that its an A too.

Now the questions after training:

Does the network recognize A as an A as good as before?
Does the network recognize B as an A too?

Does the network recognize A as an A as good as before?

Yes, it can. The reason is that if you try to illustrate the confusion matrix you can see that the FP, FN, TP and TN can be kept at a good level. What you have done is actually changing the distribution of your input data. Suppose that you train a network and it should recognize cats and dogs. You can train it just using white cats or with cats of different colours. The point here is that what you've done is not telling that $A$ and $B$ are of the same class. What you have done is classifying the first two letters as the same entity. Consider that what your network learns depends on the kind of network you use. Avoid using just Dense nets. Having image inputs, Dense nets try to find something like a mask while convolutional layers find appropriate features and those features which are better for classification would then passed to the dense layers.

  Does the network recognize B as an A too?

Again yes! But it depends on the distribution of your data. It can recognize $B$s, because you have trained it in that way. But should consider that the performance depends on the data you have. How your data is obtained to be good for generalisation.


Is DQN limited to working with only image frames? I have few questions about Deep Q Network.

Does DQN only accept image frames as input? I have never hear (read) a paperwork where it doesn't use image frames.
If the first is a No, then does image frames as input for training faster than any other options? For example, in any atari game, is frames input faster to train while having the same perfomance quality?

I know an image frames is but numbers too, like other datas, but from what I know even a low quality frame is considered huge.

DQNs don't only accept image frames as input. For instance, this DQN for the CartPole environment takes in a state with only 4 elements (the position and velocity of the cart and the angle and velocity of the pole.)

Images aren't necessarily more efficient representations than other options.  For instance, here is another DQN for the same CartPole environment which uses an image representation and is much less efficient.  A couple reasons people often use image representations include:

It's the only available representation for the task.
Other models  uses image representations and you want to compare.
You want to pass the network the same information available to humans.

If none of these apply, though, if you have a representation which gives you more insight into the position or trajectory of elements in the screen it's probably best to use that rather than a raw image.


When should one use L1, L2 regularization instead of dropout layer, given that both serve same purpose of reducing overfitting? In Keras, there are 2 methods to reduce over-fitting. L1,L2 regularization or dropout layer.
What are some situations to use L1,L2 regularization instead of dropout layer? What are some situations when dropout layer is better?

I am unsure there will be a formal way to show which is best in which situations - simply trying out different combinations is likely best!

It is worth noting that Dropout actually does a little bit more than just provide a form of regularisation, in that it is really adding robustness to the network, allowing it to try out many many different networks. This is true because the randomly deactivated neurons are essentially removed for that forward/backward pass, thereby giving the same effect as if you had used a totally different network! Have a look at this post for a few more pointers regarding the beauty of dropout layers.

$L_1$ versus $L_2$ is easier to explain, simply by noting that $L_2$ treats outliers a little more thoroughly - returning a larger error for those points. Have a look here for more detailed comparisons.


how to do the imputation for categorical feature with a missing rate? I have a dataset containing a categorical feature with a missing rate 95%. What value can replace the missing cells? Or drop this feature?

You can turn it into a one-hot encoded feature with an added class of 'Missing', depending on the cardinality (how many categories are there). If the cardinality is too high, you will need to use other techniques for high cardinality features but you can still have 'Missing' as an additional category.


Dimensionality reduction with PCA limitations What are the cases when we should not use PCA for dimensionality reduction and what to use in such cases?

You should not use PCA if you only have categorical variables, and thus the distance function in PCA is invalid.

Correspondence analysis is a common alternative.


How do I get the latitude and longitude information contained csv file? I have already done a project on "Segmenting and Clustering Neighborhoods in Toronto" using the latitude and longitude information contained CSV file link attached with the question[http://cocl.us/Geospatial_data]. How do I obtain the latitude and longitude information of a given territory?

If what you have are places names and you want to get its coordinates, you can use some library, for example, geopandas. It has the geocoding feature, which converts place names to locations on Earth.

You can also use the Google maps Geocoding API.

There are many more API services which solve this problem, but those two should get you started.

Hope this helps!


Why choose TensorFlow? I have noticed that most of the deep learning developers use TensorFlow. So why choose TensorFlow? What is the advantage of TensorFlow over Theano and CNTK?

I will outline some points about the libraries and point you to some good comparisons that I have read. The GitHub star counts are just another reference point to help compare popularity. While I don't condone simply following the masses, those stars do help you see what a lot of other people think about the frameworks.

Tensorflow

is very well documented
scales up into production, being able to use many GPUs or Google TPUs
allows flexible creation of DL architectures, using basic building blocks
is backed by Google
has a very large following - just look at how many stars it has received over on GitHub!

For reference, the awesome NumPy only has 7059 stars!

CNTK

is gaining popularity (they started a bit later than Google)
has very efficient implementations for specific use cases, such as the usage of CNNs and LSTMs in the text domain
is nicely linked to other Micosoft products, like their Azure based toolkits
scales well in production
is backed by Microsoft
has a good following on Github:

Theano

While still open source (and so able to be further developed by the community), the team behind Theano announced that they will no longer actively develop it. This means it will now likely fall even further behind other leading frameworks, and new functionality coming from ongoing research are not likely to make it into the library.

We can see that many people admire Theano, but given that it is basically the oldest DL framework, the star count tells a tale:

Additional points

Have a look for some overviews and comparisons of the deep learning frameworks in general to get a better understanding of when you might use one over another. Here are some I found useful:

Comparing top deep learning frameworks
Battle of the deep learning frameworks
Choosing a machine learning framework   - has a good conclusion

I would suggest reading about the difference between static and dynamic computational graphs. Tensorflow e.g. builds and compiles a static graph of your model and then pushes data through. PyTorch on the other hand allows you to create your model dynamically, giving more freedom during the development of new architectures.

A common workflow would be to develop and do research with PyTorch, then try to write the final production code in Tensorflow for deployment.

One last point - you should be aware of the library Keras. This is a wrapper around the base libraries, such as Tensorflow, Theano and CNTK - maybe more in the future). It is highger-level and easier to use that the others, but behind the curtains it is really just using one of the libraries you are asking about. It is just change on flag in a config file to swap between them!

Just for completeness, the Keras GitHub star banner:

[All gitHub star counts as of April 2018]

I realise some of these points may become outdated and untrue over time. In which case, feel free to leave a comment and I can revise my post.


Python - accessing dictionary values for math operations I have this dictionary:
stocks = {'FB': 255, 'AAPL': 431, 'TSLA': 1700}

and this script:
shares = input('How many shares? ')
stock = input('What\'s the stock? ')

for name in stocks.keys():
    ticker = (stocks[name])
    if name == stock:
        print('The price for', stock, 'is', ticker)

amount = int(ticker) * int(shares)
print(amount)

I wanted to access the value of the key when it is selected in the "stock" input and multiply the ticker chosen in the input, but it always multiplies 'TSLA' value 1700 by the number of shares selected, even if I choose FB as stock input.
The right output should be:
How many shares? 10
What's the stock? FB
2550
Instead I get 17000
I'd like to understand why.. thank you!

The reason is that ticker = (stocks[name]) is outside the if-statement. Since the loop does not stop when name == stock it continues to loop until the end of stocks and name will always be assigned to the last key in the dict.
One of multiple solutions is to move it within the if-statement:
shares = input('How many shares? ')
stock = input('What\'s the stock? ')

for name in stocks.keys():
    if name == stock:
        ticker = (stocks[name])
        print('The price for', stock, 'is', ticker)

amount = int(ticker) * int(shares)

print(amount)

This gives:
How many shares? 10
What's the stock? FB
The price for FB is 255
2550

And you get the same output even without a loop:
shares = input('How many shares? ')
stock = input('What\'s the stock? ')

ticker = (stocks[stock])
print('The price for', stock, 'is', ticker)

amount = int(ticker) * int(shares)

print(amount)

This gives:
How many shares? 10
What's the stock? FB
The price for FB is 255
2550


I did not really feel clear on what you were trying to accomplish. But here is how you load the format you showed into a numpy array:

import numpy as np

# Read file from disk
with open('path/to/file.txt') as f:
    txt = f.read()

# Make a list of values in string format
vals = txt.replace('{','').replace('}','').strip().split()

# Cast values to floats
vals = [np.float32(val) for val in vals]

# Store in numpy array
vals = np.array(vals)


Custom objective function in xgboost for Regression I am working on a regression problem, where I want to modify the loss function in xgboost library such that my predictions should never be lesser than the actual value. I have written this code:

def custom_loss(preds, dtrain):
    labels = dtrain.get_label()
    df = preds - labels
    df = pd.DataFrame(df, columns=['val'])
    df['valg'] = df['val'].apply(lambda x: 10*abs(x) if x<0 else x)
    grad = df['valg'].as_matrix()
    return preds-labels, grad

This essentially means that I want to penalize those predictions which are lesser than my actual value more. However, this isn't working and there is no improvement in my predictions. Can anyone help me figure out where I am going wrong? Thanks.

Whole Python script:

params = {"booster" : "gbtree",
  "eta": 0.20,
  "max_depth": 4,
  "subsample": 0.75,
  "colsample_bytree": 0.65,
  "silent": 1,
  "eval_metric": "rmse",
  }
num_round = 400

def custom_loss(preds, dtrain):
    labels = dtrain.get_label()
    df = preds - labels
    df = pd.DataFrame(df, columns=['val'])
    df['valg'] = df['val'].apply(lambda x: 5*abs(x) if x<0 else x)
    grad = df['valg'].as_matrix()
    return preds-labels, grad

dtrain = xgb.DMatrix(X_train.drop('price_act', axis=1),
label=X_train['price_act'])
dtest = xgb.DMatrix(X_test.drop('price_act',axis=1),
label=X_test['price_act'])

watchlist = [(dtrain,'train'), (dtest,'eval')]

bst = xgb.train(params, dtrain, num_round, watchlist, custom_loss)

Looking at the documentation example here, a xgboost custom loss function needs to return the gradient and second-order gradient. Your function does not return those values for the stated goal.


new and fresh ideas with stack exchange dataset I'm a new bee for data analysis . i need to work on a research project in big data analysis. first of all i search for a dataset and i found interest in stack exchange  data dump. however i browse for researches i found a lot .And whatever i thought about a idea based on this dataset , its already done by someone else .
please help me out with a new and useful idea for my research .

It might help to have a bit more information on what you want to do, but have a look at Kaggle - it's a regular big data science competition, all of the data sets are huge and available for download.

There's also some smaller datasets made available by Tableau.

Hope that give you some inspiration!


How do i perform feature selection for clustering? I am new to Data science and trying to learn clustering? I have to partition the given dataset into different clusters into customer clusters based on their purchasing habits? How do I select the features that important for clustering?

Heirarchy_1, Herarchy_2 etc describe the product.

The data described in tables above is for purchases. You, however, want to segment users. Therefore, the first thing is to derive a dataset of user data.
As a first approach, I suggest that you create a table with one row per customer ID, where you have information about how many products the customer has purchased of each of the possible hierarchy values (e.g. customer ID: 312425, tableware products purchased: 5, cookware products purchased: 10, bath scale-manual products purchased: 0), and then do clustering on that.


Explain output of logistic classifier [Note : There is some serious problem with logic used to get the best banner. I got it late. Directly read the answer to get general info, or you can also try to the find the mistake.]

Problem: Given a set of user features, select an ad with the highest probability to be clicked.

Dataset - https://www.kaggle.com/c/avazu-ctr-prediction/data (First 100000 tuples from training set and split that it 80:20 training:testing)

Tutorial followed - https://turi.com/learn/gallery/notebooks/click_through_rate_prediction_intro.html

C14 is my ad id.
Problem :- Given ('device_type', 'C1', 'C15') return ad id.

Training:-

I have taken 'click' as my target and ('device_type', 'C1', 'C15', 'C14') as my input features.

I used logistic regression classifier in graphlab library to train the model.

I am doing ad selection in the following way:-
Given a set of features ('device_type', 'C1', 'C15', X) Iterate X over all possible values of C14 and pass the features to predictor to get the probability that X ad will be clicked. Return the ad with maximum click probability.

MY PROBLEM IS LOGISTIC CLASSIFIER IS ONLY RETURNING ONE AD for every test tuple, though with different click probability, it means only one ad is getting the highest probability to be clicked.

Can anyone explain this observation?

When using boosted tree classifier instead of logistic classifier for the above prediction, I am able to get different ads as my prediction and hence getting better results.

It means your logistic classifier is biased towards one class, this could be because of below reasons that I can think of.

Class Imbalance:  This article explains how to identify and overcome the class imbalance problem.
Overfitting: This article explains how to tackle over fitting.

Logistic classifier works better if data is linearly related, if you find non-linear relationships in data I would suggest use better algorithms like GBM/SVM/Random forest, which will give you much better and accurate results.


Apply multivariable linear regression to a dataset in pandas with sklearn I'm trying to predict the population for states and the country in 2050. My current dataset has values for each state from 1951,1961...2011 in the same table. Here is a sample view:

Row States/Union Territories         1951   1961    1971    1981    1991    2001    2011
0   Andaman and Nicobar Islands   31    64       115     189     281     356    381
1   Andhra Pradesh               31115  35983   43503   53551   66508   76210   84581
2   Arunachal Pradesh             307   337     468     632     865     1098    1384
.
.
.

So i ran a simple regression for values in 2011 and the model works very well. My question here is, how do i run it for an entirely new column(2050), which has no data to compare with in order to test for accuracy or any other metrics? One thing i intuitively tried was to add a new column  as 2050 and put all the values as 0, but then even the predicted values were 0 so that's of no help. I'm new to this so thanks for any help!

Associated code:

data = pd.read_csv("final_doc.csv", encoding = "latin-1")
data.head()
data['2050'] = 0
X = data[['1951', '1961', '1971', '1981', '1991', '2001']].values
Y = data[['2011']].values
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.2)
regressor = LinearRegression()
regressor.fit(X_train, Y_train)
y_pred = regressor.predict(X_test)
fin = pd.DataFrame(y_pred, columns = ['2011'])
fin

The form of your model is incorrectly specified. You want to model your data so that your independent variables are year (a numeric variable) and state/union (a categorical variable). Your dependent variable will be the population size.

In order to develop such a model you will need to restructure your data so that it is in a long format (see below)

States/Union Territories      Year                Pop
-----------------------------------------------------
Andaman and Nicobar Islands   1951                 31
Andaman and Nicobar Islands   1961                 64
           ...                 ...                ...
Andhra Pradesh                1951              31115
Andhra Pradesh                1961              35983
           ...                 ...                ...
Arunachal Pradesh             1951                307
           ...                 ...                ...

Once the data is transformed into a proper format, you can then fit your model.
I should mention you will need to recode your categorical variable - you can use a method called one hot-encoding to achieve this.

Once your model has been fit, you will be able to predict the expected population size for a given year and Union/State.

The following post Multiple Linear Regression Model in 7 Steps with Python provides a great guide to developing a model.

Hope this helps!


Visualizing a graph with a million vertices What is the best tool to use to visualize (draw the vertices and edges) a graph with 1000000 vertices? There are about 50000 edges in the graph. And I can compute the location of individual vertices and edges.

I am thinking about writing a program to generate a svg. Any other suggestions?

I also suggest Gephi software (https://gephi.github.io), which seems to be quite powerful. Some additional information on using Gephi with large networks can be found here and, more generally, here. Cytoscape (http://www.cytoscape.org) is an alternative to Gephi, being an another popular platform for complex network analysis and visualization.

If you'd like to work with networks programmatically (including visualization) in R, Python or C/C++, you can check igraph collection of libraries. Speaking of R, you may find interesting the following blog posts: on using R with Cytoscape (http://www.vesnam.com/Rblog/viznets1) and on using R with Gephi (http://www.vesnam.com/Rblog/viznets2).

For extensive lists of network analysis and visualization software, including some comparison and reviews, you might want to check the following pages: 1) http://wiki.cytoscape.org/Network_analysis_links; 2) http://www.kdnuggets.com/software/social-network-analysis.html; 3) http://www.activatenetworks.net/social-network-analysis-sna-software-review.


What is the best architecture for Auto-Encoder for image reconstruction? I am trying to use Convultional Auto-Encoder for its latent space (embedding layer), specifically, I want to use the embedding for K-nearest neighbor search in the latent space (similar idea to word2vec).

My input is 3x224x224 (ImageNet), I could not find any article that elaborates a specific architecture (in terms of number of filters, number of conv layers, etc.)
I tried some arbitrary architectures like:

Encoder:

Conv(channels=3,filters=16,kernel=3)
Conv(channels=16,filters=32,kernel=3)
Conv(channels=32,filters=64,kernel=3)

Decoder:

Deconv(channels=64,filters=32,kernel=3)
Deconv(channels=32,filters=16,kernel=3)
Deconv(channels=16,filters=3,kernel=3)

But I'd like to start my hyper-parameters search from a set up that proved itself on a similar task.
Can you refer me to a source or suggest an architecture that worked for you for this purpose?

I don't know about an architecture being definitively the best, but there are some best practices you can follow. Check out these papers:

Learning to Generate Images with Perceptual Similarity Metrics
Push it to the Limit: Discover Edge-Cases in Image Data with Autoencoders
Walking the Tightrope: An Investigation of the Convolutional Autoencoder Bottleneck

To sum it up, residual blocks in between downsampling, SSIM as a loss function, and larger feature map sizes in the bottleneck seem to improve reconstruction quality significantly. How that translates to the latent space is not entirely clear yet.


Relationship between KS, AUROC, and Gini Common model validation statistics like the Kolmogorov–Smirnov test (KS), AUROC, and Gini coefficient are all functionally related. However, my question has to do with proving how these are all related. I am curious if anyone can help me prove these relationships. I haven't been able to find anything online, but I am just genuinely interested how the proofs work. For example, I know Gini=2AUROC-1, but my best proof involves pointing at a graph. I am interested in formal proofs. Any help would be greatly appreciated!

The Wikipedia entry for Receiver operating characteristic references this paper for the Gini=2AUROC-1 result: Hand, David J.; and Till, Robert J. (2001); A simple generalization of the area under the ROC curve for multiple class classification problems, Machine Learning, 45, 171–186.  But I'm afraid I don't have easy access to it to see how close it comes to what you want.


How do we ensure that a model generalize well in the real world When we have trained a model on a training data set as well as the test set, how can we ensure that a model will generalize in the real world.I mean both the test set and the training set are a very small subset compared to what exist in the real world and there might be inherent bias in the datasets as well.

As a beginner in machine learning , it will help me and others a lot if you share the techniques to avoid the common pitfalls

You should try out the model that you built in real world by using test data that reflect real data.

Machine learning is an iterative process, you have to try it out and see the performance.

Your test data should be extended to reflect real life scenario that you expect it to work well.
You might want to change your objective function sometimes, your existing objective function might not have capture some real life factor. For example, if you see your model keep making mistakes on certain type of data, you might want to penalize it more.
Your model might not last forever, real life data might change over time and you have to tune your model to adapt to it. Your model might not have forseen some scenarios and might not know what should be the correct response to be.


Why is there a $2$ at the denominator of the mean squared error function? In the famous Deep Learning Book, in chapter 1, equation 6, the Quadratic Cost (or Mean Squared Error) in a neural network is defined as

$ C(w, b) = \frac{1}{2n}\sum_{x}||y(x)-a||^2 $

where $w$ is the set of all weights and $b$ the set of all biases, $n$ is the number of training inputs, x is the set of all training inputs, y(x) is the expected output of the network for input x, and $a$ is the actual output of the network for input $x$, with respect to $w$ and $b$.

Most of this formula seems very clear to be, except the $2$ in the denominator. If I understand it correctly, we are summing up the squared vector length of (the actual output minus its expected output), for each training input (giving us the total squared error for the training set) and then divide this by the number of training samples, to get the mean squared error of all training samples. Why do we divide this by $2$ as well then?

In other places I've seen that Andrew Ng's lecture defines the Mean Square cost in a similar way, also with the $2$ in the denominator, so this seems to be a common definition.

This is just for mathematical convenience. When you differentiate $C(w,b)$, you will get an extra $2$. To eliminate that, $2$ is kept beforehand in denominator.

You can also watch this video on SVM lecture by Patrick Winston, where he uses a similar formula and then tells that he is using $2$ in denominator just for mathematical convenience.


[R]Fuzzy C-Means, different between ppclust vs e1071? no. of cluster = 10, data points = 6000

library(ppclust)
cm <- fcm(x,centers = cen)

takes ~ 10 minutes

library(e1071)
cm <- cmeans(x,cen,1000)

takes ~ 1 minute

the only reason I prefer ppclust is because it allows nstart, meaning I am not stuck with local optima.

Given both are doing same thing, why is one taking 10x time then other?

Look at the source code. The guts of e1071::cmeans are written in C, but ppclust::fcm looks like all R and has some ugly looking eval stuff which might make it dead slow.


How to test accuracy of an unsupervised clustering model output? I am trying to test how well my unsupervised K-Means clustering properly clusters my data.  I have an unsupervised K-Means clustering model output (as shown in the first photo below) and then I clustered my data using the actual classifications.

The photo below are the actual classifications.  I am trying to test, in Python, how well my K-Means classification (above) did against the actual classification.

For my K-Means code, I am using a simple model, as follows:

kmeans = KMeans(n_clusters=4, random_state=0).fit(myData)
labels = kmeans.labels_

What would be the best way for me to compare how well my unsupervised KMeans clustering model did against the actual classifications?

Since you have the actual labels, you can compare them with the obtained labels and evaluate performance. Typically purity and nmi (normalized mutual information) are used. Read this (Evaluation of Clustering) document for detailed explanation.

If you don't have actual labels then you can use modularity or [Silhouette] for measuring clustering performance.


Class Imbalence Problem even after Balancing Data So I am training a neural network on a binary classification problem and my Case (1) and Controls (0) were imbalanced so I oversampled my cases so that that the training set was 0.5053 made up of controls. I did not balance my test set which was 0.562 controls. In the beginning my train and test accuracy rises (it is not very accurate still but I expect this to be the case) but then the training accuracy steeply drops while the test accuracy plateaus.

They end up with accuracies of 0.5053 and 0.562 respectively so the network is just classifying everything the same. I do not understand how this behavior arises as I thought that balancing my training set would avoid the problem of classifying everything the same? Also, the training set begins to learn initially upwards from 50/50 but I cannot understand its reversion. Is there anything I can do to prevent this? Or should I just employ early stopping when the training accuracy begins to decrease?

Any insight would be appreciated!

opt = tf.keras.optimizers.SGD(lr=0.000001, momentum=0.9, decay=0, nesterov=True)

model = keras.Sequential([keras.layers.Dense(100,kernel_initializer='he_uniform',bias_initializer=keras.initializers.Constant(value=0.01),activation=tf.nn.relu,kernel_regularizer=regularizers.l2(0.1)), keras.layers.Dense(100,kernel_initializer='he_uniform',bias_initializer=keras.initializers.Constant(value=0.01),activation=tf.nn.relu, kernel_regularizer=regularizers.l2(0.1)), keras.layers.Dense(1, activation=tf.nn.sigmoid)])

model.compile(optimizer=opt, loss='binary_crossentropy', metrics=['accuracy'])

The fact tha balancing the dataset Will prevent the overfitting and thus, good results in the test set, Works under the assumption that the model does not underfit.

What it may be happening is that the model is too bad that it can't perform the classification task in either the un-balanced and the balanced dataset.

It's difficult to analyse the problema without more information, you may have facing convergence issues (too large learning rate, vanishing gradients), you may also be computing the accuracy in a bad way...


What is a good approach for a lifespan? Let's say I wan't to predict the lifespan of an ad in a listing.

I know a bunch of thing from the ad like:

the title
the price
the location
etc

The target value is the duration of the ad in the listing before it's being removed (item has been sold).

What would be the best approach for engineering the target?

I've tried categorizing the log of the duration, but it's not leveraging the cyclic pattern you can see in the histogram of the lifespan (in hours):

x-axis : lifespan in hours

I think you need to come up with a way to treat the data such that you're thinking in days, not hours, right? The peaks look they're just at 24, 48, 72, 96, (1 day, 2 days, 3 days, 4 days) etc, and are pretty much normally distributed around those peaks.

I think a good test might be to try a categorical approach to start, to see how well you can predict which 'peak' the ad belongs to (is this an ad in the 24-hour normal distribution? the 48-hour?). If you can figure out which peak that ad belongs to then you might be able to identify features that indicate whether it's more likely to be on the short- or long-side of the hump. If you have bad results putting the ads into categories that might tell you something too.

If you do try this, be careful of how you measure performance as your dataset will be unbalanced by the higher occurrence rate of quickly-pulled ads.


How to combine the features extracted from different CNN architectures? I want to combine the features extracted from different CNN architectures into my fully connected layer. How to proceed?

You can tf.keras.layers.concatenate your extracted feature before flatten them in order to pass it to your fully connected layer. Note than you can actually use other operation such as multiply or add to merge your feature.


How to work with Sensitive (personal) Data, without actually seeing it? I'm currently working on a project that has sensitive data. I'm allowed to see the data on a single platform (company PC), but outside of that platform I cannot process it. I'd like to train a ML algorithm with the data but to do so I'll have to use a different PC. I cannot process the data on that PC though.
Is there anyway to translate and preserve the data in a format that can still be used for ML, but isn't in it's explicit original form?

You can use the following techniques to mask sensitive data:

Substitution cipher -  any character of plain text from the given fixed set of characters is substituted by some other character from the same set depending on a key.

Tokenization masking - mask source string data based on criteria that you specify in an algorithm.

Principal Components Analysis (PCA) or other dimension-reducing techniques - combine several features and then carry out ML training only on the resulting PCA vectors.

Data Coarsening - decrease the precision or granularity of data to make it more difficult to identify sensitive data within the dataset.

Also, read  Data Masking on Wikipedia.


two autoencdoers learned by two similar vectors (each one with its own). Similarity of hidden layer vectors will be the same? If I will train one autoencoder with one vector only and second autoencoder with second vector only, does it mean if vectors were similar, that the hidden layer vectors of both autoencoders will be similar as well. Autoencoders structure is identical. The quantity of hidden layer neurons is smaller that input. So i want to reduce dimentionality. I understand that it will be overfitting. But for me it looks like you not learn features but just mimick your input function with the smaller dimensional vector while loosing information. So you receive kinda low dimentional approximation of learned vector. So if vector V1 was similar to V2 and I will use V1 to train Autoencoder A1 and V2 to train Autoencoder A2, does it mean that hidden layer vector of A1 and hidden Layer vector of A2 won't be that similar like V1 and V2 but still aproximately quite similar?

I'd say if you are training with a single vector, then no they wouldn't be the same.

It seems unintuitive at first but one has to realize it doesn't make sense to learn the features of a single vector, only the features of a set of vectors. The goal of an auto-encoder it to use its parameters to compress its dataset into a lower-dimension representation (A.K.A latent space) as efficiently as possible.

By efficiently, I mean that the most amount of useful information is encoded into the latent space. This means the vectors close in input space will also be close in the latent space.  The auto-encoder achieves this by learning to utilize groups of commonly occurring pixels (e.g. features). For images, an A.E. might learn to describe a picture in terms of these-

Now instead of describing an image as a with all its pixels (i.e. the original image) we can instead describe it with lines of blobs of colors, effectively reducing the amount of information needed. Lines and blobs are learned instead of random pixels because lines and lines and blobs can describe any picture, but a random group of pixels can not.

As you can see, it is necessary when we have multiple vectors that we learn features that can approximate all the vectors in the dataset. And hence when we have only one vector we can use any combination of features because any combination of features will be able to approximate the original vector.


How to predict when an appointment will be scheduled? I have a dataset of tens of thousands of appointments. Appointments have a created date and scheduled date. Something like this:
ID   Created      Scheduled
1    08/01/2020   08/05/2020
2    08/01/2020   08/07/2020
3    08/02/2020   08/04/2020
...

I'm trying to predict the probability of all possible schedule dates based on a created date in the future. So basically, if a customer created an appointment tomorrow (August 20th), what's the probability they would schedule their appointment for August 21th, August 22nd, August 23rd, etc. In theory, customers can create appointments into perpetuity but practically no one makes an appointment more than ~2 months in advance.
Some observations I've made are:

Customers prefer making appointments on weekends
Customers prefer making appointments during the last or first few days of the month
Most appointments are scheduled within 2 weeks of the day the customer creates the appointment

I've been struggling with this problem. I first tried to just look at how many days out a customer schedules their appointment. Something like 10% of the time it's the next day, 15% of the time it's 2 days out, etc. But that didn't account for how customers preferred making appointments on weekends and the start/end of the month. So it was incredibly inaccurate.
I'm frankly stumped about how to approach this problem. I would appreciate it if people have ideas on how I can go about this. Thank you! Please let me know if anything here is unclear.

Create new features specifying days of the week.
df['Created'] = pd.to_datetime(df['Created'])
df['Scheduled'] = pd.to_datetime(df['Scheduled'])
df['Scheduled_day_of_week'] = df['Scheduled'].dt.day_name()
df['Created_day_of_week'] = df['Created'].dt.day_name()

Create a new column day difference which will be the difference between the two days.
Then you can visualize for each day what would be the day difference.
Similarly you can try for the date.
This process is called Feature Engineering.


Why maximize ELBO in the variational autoencoder? For a variational autoencoder, we have that:

$$\mathcal{L}(x,\theta,\phi) := \mathbb{E}_{z \sim q_\phi(z|x)}[\log p_{\theta}(x|z)] -KL[q_{\phi}(z|x) ||p(z)] $$

This is called the variational lower bound or evidence lower bound (ELBO). But I think what we're actually trying to maximize is the log-likelihood of our data:

$$\log p_{\theta}(x) = \mathcal{L}(x,\theta,\phi) + KL[q_{\phi}(z|x)||p_{\theta}(z|x)]$$

There are a few things I'm unsure about, in increasing order of difficulty.

For the actual loss function of a VAE, we use $-\mathcal{L}$, more or less. Of course, it's expensive to actually calculate the expectation, which is why we use a single $z$ sample each time, yes?
We are told to treat $p(z)$ as being $\mathcal{N}(0,1)$, but I don't see what causes it to become normal. We're just told to plug in $\mathcal{N}(0,1)$ for $p(z)$ when calculating the loss, and all this seems to do is ensure that $q_\phi(z|x)$ gets closer to it. We know nothing about $p(z)$, right (besides the fact that if all $p_{\theta}(z|x)$ are close to some distribution, then so is $p_{\theta}(z)$)?
How should we think about that KL divergence in the second formula? It's usually pointed out that for fixed $p_{\theta}$, maximizing $\mathcal{L}$ is equivalent to minimizing the KL div. But $p_{\theta}$ is not fixed; it's being learned. We could also improve $\log p_{\theta}(x)$ (our ultimate goal) by making that divergence worse, couldn't we? What justifies optimizing $\mathcal{L}$ alone (other than tractability)?

Basically, I'm confused by the common explanation that what we really want is to minimize that second KL divergence, and also that the best way to do that is to maximize the ELBO.

For the actual loss function of a VAE, we use $−\mathcal{L}$, more or less. Of course, it's expensive to actually calculate the expectation, which is why we use a single  sample each time, yes?

Yes. It turns out a single sample MC estimate has fairly low variance in this case. However, the Importance Weighted Autoencoder shows that taking multiple samples can be useful.

  It's usually explained that we treat () as being $\mathcal{N}(0,1)$. But simply plugging in $\mathcal{N}(0,1)$ to the KL divergence in the loss (as we do) just ensures that $_(|)$ gets close to it. We know nothing about $()$, right (besides the fact that if all $_(|)$ are close, then so is $_()$?

I prefer to think of it this way. There are in fact two models: a stochastic encoder (inference model) $q_\phi(x,z)=q_\phi(z|x) q(x)$ and a probabilistic decoder (generative model) $p_\theta(x,z)=p_\theta(x|z) p(z)$. Our goal is to enforce that $ q_\phi(x,z) $ and $ p_\theta(x,z) $ are close (small KL divergence).

Note that $q(x)$ and $p(z)$ are the empirical distribution and Bayesian prior, and are fixed ahead of time (not learned). This is for the vanilla VAE. However, these are very different from the aggregate marginals:
$$
q_\phi(z) = \int q_\phi(z|x) q(x) \, dx
\;\;\;\;\&\;\;\;\;
p_\theta(x) = \int p_\theta(x|z) p(z) \, dz
$$
In other words, $p(z)$ is something we choose as part of the model. Our job is then to ensure that the inference marginal $q_\phi(z)$ is close to it. In other words, we know $p(z)$, but we learn $q_\phi(z)$ (also called the inferred prior or aggregate posterior), which we would like to match it.

  How should we think about that KL divergence in the second formula? It's usually pointed out that for fixed $_$, maximizing $\mathcal{L}$ is equivalent to minimizing the KL div. But $_$ is not fixed; it's being learned. We could also improve $\log _()$ (our ultimate goal) by making that divergence worse, couldn't we? What justifies optimizing $\mathcal{L}$ alone (other than tractability)?

It's important to note that while $p_\theta(x)$ is being learned (well, an approximation is being learned), there is actually an "correct" one that can be computed via Bayes Rule:
$$ p_\theta(z|x) = \frac{p_\theta(x|z) p(z) }{p_\theta(x)} $$
when $ p_\theta(x|z) $ is fixed.
In other words, we don't have to learn an inference model; there is an optimal one given by Bayes rule. In theory we can just learn the generative model and use this rule to create the encoder.
But this is of course intractable to compute.

  How should we think about that KL divergence in the second formula?

We are doing variational inference. We cannot compute the true posterior $p_\theta(z|x)$. So instead we compute an approximation $q_\phi(z|x)$. The second KL says the approximation should be good. Notice something important: when the variational posterior approximation is perfect, then the ELBO equals the marginal likelihood. In other words, when our $q_\phi$ is perfect, then we are indeed optimizing the true marginal likelihood, as you noted we want to do.

  We could also improve $\log _()$ (our ultimate goal) by making that divergence worse, couldn't we?

No, decreasing it should always make the marginal likelihood worse. I don't have a proof, but I conjecture it is so.

  What justifies optimizing $\mathcal{L}$ alone (other than tractability)?

Well, it is a lower bound, so maximizing it guarantees we are "pushing up" the true marginal likelihood. As noted above, when the variational approximation is good, we are guaranteed to be doing the right thing.
There is a lot of literature on variational inference theory; I suspect one can say more under more constrained conditions.

  Basically, I'm confused by the common explanation that what we really want is to minimize that second KL divergence, and also that the best way to do that is to maximize the ELBO.

I don't know if it is the best way. It is certainly an efficient one. But in some cases, it may be better to perform inference more directly, e.g. Hamiltonian Monte Carlo methods.


Help on designing data science curriculum from "traditional" offerings So I'm hoping to ask professionals for their advice here. My university doesn't offer a "data science" major and I'm cobbling one together from different departments. So far I have a math major with stat emphasis but I'm having trouble finding technology use in this program so I was thinking computer science for science credits. I also have a choice of an Information Management minor or a Business Analytics minor, both from the business college.

So basically, I'm wondering:

Which areas of computer science should I focus on? (I can only take 2 aside from the intro CSI and CSII course).
Which minor should I go for? Analytics consists of applied econ analytics, marketing research, and quantitative analysis while the info minor is management, programming, busn intelligence (advanced excel), and "big data". Right now I'm leaning towards the info one because it seems to have more software in it, including SQL.

(I should also mention that I'm located in Las Vegas and my university even offers probability and gambling statistics but that's in the hospitality college and not sure that would be worth the trouble. It sounds interesting though, and might be good experience?).

My advisors have no idea what I'm going for, partly b/c they're all in different departments, so I hoped someone from the real world job market might know a thing or two :-)

Of the traditional majors, I think computer science + statistics is the most obvious. You can substitute applied mathematics or even physics for the statistics major, if you want. Always keep the computer science courses.

If the choice is between information management and business analytics, given that you're already studying maths, it's hard to say. They're both pretty light-weight, and will add little to your math major. I would not consider them a plus per se as a prospective employer reading your resume. Unless they're cheap, I'd take online classes and get some practice instead.


Help with algorithm approach for computer vision I hope this is the right forum to ask.

I had a client approach me with a demand for a vision system for their assembly line.

The problem they are facing is that the operator sometimes forgets to put all three parts of the product together. They want a vision system that can spot if any of three main components are missing (only missing nothing else).

I have the knowledge to build a covnet in keras but I think that would be a way to big hammer for this nail and I have to go to the client any produce images myself (ergo I can sit there for a day or two and take photos the products). So I might end up with a data set of maybe 200 images. I will try and make some artificially bad images, taking of the different components and photoing that. I don't think I'm allowed to show what the product looks like but I made a sketch.

What algorithm can I read up on that would solve this problem in a good and time efficient manner? Demands on the camera and pc for the any suggested algo would also be very much appreciated (resolution etc).

this problem set is in the realm of object detection and image classification,you need have need a good dataset from your client or scrape one yourself,you will need to perform real time object detection in the assembly line so for that case you will need a have a really good convnet that has high accuracy.One way to have such a high accurate model is to perform transfer learning using your preferred models such as inception,caffe,lenet etc. you will need to delve more into learning about anchor boxes,for this kind of problem look at the keras implementation of Yolo algorithm https://towardsdatascience.com/object-detection-using-yolov3-using-keras-80bf35e61ce1