Extracting words belonging to a key from the text I have a description of a product say mobile in this form -

"5.5" (13.97cm) full hd 1920 x 1080 pixels ips screen gorilla glass display with 401ppi resolution - 450 nits brightness & contrast ratio of 1000:1first phone with theatermax technology and dolby atmos surround sound over speakers13mp f2.2 primary camera with 5p lens element, dual led dual tone flash, 5mp front camera with 4p lens elementandroid v5.1 os with 1.3 ghz mediatek 6753 64-bit octa core processor, arm mali t720 gpu, 3 gb ddr3 ram, 16gb internal memory (expandable up to 128gb) and dual micro sim dual standby (4g+4g)3300mah lithium-ion battery with fast charger 2.0 providing talk-time of 29 hours"

I have to extract the important elements of the description and map it to keys like -

Display size - 5.5
Display Resolution - 1920 x 1080
Sim type - Dual Sim
Ram - 3gb
Storage - 16gb.

I have a dictionary that maps key values pairs.

Example - {Ram:["3gb","3 GB","4 gb"],"Sim type":["Dual","Micro","Nano"]}

Can anyone suggest how to do this. I am coding in python. How to proceed it with NLTK. Should I use ngrams??? Any useful examples that uses self tagging will be helpful. All examples i see uses already defined tags like noun,person,organisation etc. I want to give my own dictionary as corpus and train the model so that it extracts the relevant key values pairs from the text.

You have two options:

Use regular expressions to extract features of interest based on your dictionary. You may have mixed results, since, for example, dual sim cards will be micro or nano, so you will get two types from single description. However, that just says, that the dictionary isn't well structured.
Train your own NER (named entity recognition) system. For that, first, you will need a data set which will consist of your descriptions and marked entities. This would require manual labour, either yours or paid one one some crowd sourcing platform. When you will have a marked-up dataset, you could use NLTK to train a NER. Markup may be done in any way familiar to you, but, generally speaking, xml-like markup is the best choice since there are many ways to train a NER, and most of frameworks know how to parse xml-like tags:

Example markup:

<screen_size>5.5"</screen_size> (13.97cm) full hd <resolution>1920 x 1080</resolution> pixels ips screen gorilla glass display etc.


Intuitive explanation of difference between PCA and SVD Can someone explain the difference between SVD and PCA with real life example?

PCA.
Suppose that you have n data points comprised of d numbers (or dimensions) each. If you center this data (subtract the mean data point $\mu$ from each data vector $x_i$) you can stack the data to make a matrix

$$
X =  \left(
 \begin{array}{ccccc}
  && x_1^T - \mu^T && \\
  \hline
  && x_2^T - \mu^T && \\
  \hline
  && \vdots && \\
  \hline
  && x_n^T - \mu^T &&
 \end{array}
\right)\,.
$$

The covariance matrix

$$
S = \frac{1}{n-1} \sum_{i=1}^n (x_i-\mu)(x_i-\mu)^T = \frac{1}{n-1} X^T X
$$

measures to which degree the different coordinates in which your data is given vary together. So, it's maybe not surprising that PCA -- which is designed to capture the variation of your data -- can be given in terms of the covariance matrix. In particular, the eigenvalue decomposition of $S$ turns out to be

$$
S = V \Lambda V^T = \sum_{i = 1}^r \lambda_i v_i v_i^T \,,
$$

where $v_i$ is the $i$-th Principal Component, or PC, and $\lambda_i$ is the $i$-th eigenvalue of $S$ and is also equal to the variance of the data along the $i$-th PC. This decomposition comes from a general theorem in linear algebra, and some work does have to be done to motivate the relatino to PCA.

SVD is a general way to understand a matrix in terms of its column-space and row-space. (It's a way to rewrite any matrix in terms of other matrices with an intuitive relation to the row and column space.) For example, for the matrix $A = \left( \begin{array}{cc}1&2\\0&1\end{array} \right)$ we can find directions $u_i$ and $v_i$ in the domain and range so that

You can find these by considering how $A$ as a linear transformation morphs a unit sphere $\mathbb S$ in its domain to an ellipse: the principal semi-axes of the ellipse align with the $u_i$ and the $v_i$ are their preimages.

In any case, for the data matrix $X$ above (really, just set $A = X$), SVD lets us write

$$
X = \sum_{i=1}^r \sigma_i u_i v_j^T\,,
$$

where $\{ u_i \}$ and $\{ v_i \}$ are orthonormal sets of vectors.A comparison with the eigenvalue decomposition of $S$ reveals that the "right singular vectors" $v_i$ are equal to the PCs, the "right singular vectors" are

$$
u_i = \frac{1}{\sqrt{(n-1)\lambda_i}} Xv_i\,,
$$

and the "singular values" $\sigma_i$ are related to the data matrix via

$$
\sigma_i^2 = (n-1) \lambda_i\,.
$$

It's a general fact that the right singular vectors $u_i$ span the column space of $X$. In this specific case, $u_i$ give us a scaled projection of the data $X$ onto the direction of the $i$-th principal component. The left singular vectors $v_i$ in general span the row space of $X$, which gives us a set of orthonormal vectors that spans the data much like PCs.

I go into some more details and benefits of the relationship between PCA and SVD in this longer article.

Original post on crossvalid


Agent Collapse / Overfitting during Training I'm new to reinforcement learning so please bear with me. I'm training an agent to play ms-Pacman using the actor-critic method.
Below are the results of a couple of runs, in both graphs the orange line in the average of the previous 100 values. The left graph is the episode duration and the right is the reward earnt in each episode.
In both the runs, the agent steadily learns and then collapses, and is unable to recover. Has the agent reached it's maximum potential with this method or is this a case of overfitting?
Hyperparameters for both the runs:

Batch Size: 128
Replay Memory Buffer: 500,000
Epsilon Minimum: 0.1
Epsilon Maximum: 1.0
Epsilon minimum is reached at around 2,000,000 steps (approx episode 3,000)

There are lots of things that might lead to this performance. With the very limited information provided I can only guess and suggest the reasons.
I have personally used lots of Pacman variations (including this one) with RL models successfully. My first initial guess is that you are stopping exploration too early. Pacman environments are very hard to solve (up to the last level). You can get a reasonable performance though within a couple of days training (depending on your system). I highly doubt that the agent underfits/overfits as it will take time for the agent to reach  a good score. Also underfitting/overfitting would be better observable when you test your agent (you do not update weights). I would let exploration to become minimum once the agent reaches the last episode. Then I would test the current state of the agent with very little or none amount of exploration (full exploitation).


Ellipses detecting at the image I have the following image:

I need to get two ellipses which will 'describe' my 'ring'.

I have the following code at the moment:

import cv2
import numpy as np
import imutils

image = cv2.imread('front2.png', cv2.IMREAD_COLOR)

# Convert to grayscale
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

_, gray = cv2.threshold(gray, 50, 255, cv2.THRESH_BINARY)

# Downsize image (by factor 4) to speed up morphological operations
gray = cv2.resize(gray, dsize=(0, 0), fx=0.25, fy=0.25)
cv2.imshow("Gray resized", gray)
cv2.imwrite('Gray_resized.png', gray)

# Morphological opening: Get rid of the stuff at the top of the ellipse
gray = cv2.morphologyEx(gray, cv2.MORPH_OPEN, cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (5, 5)))
cv2.imshow("Gray removed noise", gray)
cv2.imwrite('Gray_removed_noise.png', gray)

# Resize image to original size
gray = cv2.resize(gray, dsize=(image.shape[1], image.shape[0]))

# Find contours
cnts, _ = cv2.findContours(gray, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)

# Draw found contours in input image
image = cv2.drawContours(image, cnts, -1, (0, 0, 255), 2)

if len(cnts) != 0:
    for cont in cnts:
        if cont.size < 10 or cv2.contourArea(cont) < 100:
            continue
        elps = cv2.fitEllipse(cont)
        cv2.ellipse(image,elps,(0,255,0),2)

# Downsize image
out_image = cv2.resize(image, dsize=(0, 0), fx=0.25, fy=0.25)
cv2.imshow("Output image", out_image)
cv2.imwrite('Output_image.png', out_image)

I've got images like:

In this way it detected one ellipse well enough but it hasn't detected the second one.

As i read it'd be more logical to use 'cv2.THRESH_BINARY_INV' in threshold but in this way it doesn't detect ellipse in the center:

Also if i try to lower the threshold my ring starts to fade:

The main questions: How to get rid of this noise at the top of my ring for detecting contours? And how to detect these ellipses?

I've looked at these answers:
1st
2nd
3rd
 but i haven't managed to achive my goal.

P.S. I can't use Hough-Circle method here. Because i have few similar images and radius for 'x' and 'y' axis may different and i need this precision(it means i need only ellipses and their radiuses for two axises).

I would recommend to use adaptive threshold rather than global threshold. Adaptive threshold deals better with uneven illumination. I have modified your code accordingly. You may want to experiment further with the value of block size and C value.

I have also used RETR_CCOMP and only show contours which either have a parent (index 3 != -1) or don't have any children (index 2 == -1).

I don't guarantee that this will work for other images but it may be a good starting point for your further experimentation.

import cv2
import numpy as np
import imutils

image = cv2.imread('front2.png', cv2.IMREAD_COLOR)

# Convert to grayscale
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
h, w = gray.shape
s = int(w / 8)
gray = cv2.adaptiveThreshold(gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY_INV, s, 7.0)

# Downsize image (by factor 4) to speed up morphological operations
gray = cv2.resize(gray, dsize=(0, 0), fx=0.25, fy=0.25)
cv2.imshow("Gray resized", gray)
cv2.imwrite('Gray_resized.png', gray)

# Morphological opening: Get rid of the stuff at the top of the ellipse
gray = cv2.morphologyEx(gray, cv2.MORPH_OPEN, cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (5, 5)))
cv2.imshow("Gray removed noise", gray)
cv2.imwrite('Gray_removed_noise.png', gray)

# Resize image to original size
gray = cv2.resize(gray, dsize=(image.shape[1], image.shape[0]))

# Find contours
cnts, hier = cv2.findContours(gray, cv2.RETR_CCOMP, cv2.CHAIN_APPROX_NONE)

# Draw found contours in input image
image = cv2.drawContours(image, cnts, -1, (0, 0, 255), 2)

for i, cont in enumerate(cnts):
    h = hier[0, i, :]
    print(h)
    if h[3] != -1:
        elps = cv2.fitEllipse(cnts[i])
    elif h[2] == -1:
        elps = cv2.fitEllipse(cnts[i])
    cv2.ellipse(image, elps, (0, 255, 0), 2)

# Downsize image
out_image = cv2.resize(image, dsize=(0, 0), fx=0.25, fy=0.25)
cv2.imshow("Output image", out_image)
cv2.imwrite('Output_image.png', out_image)
cv2.waitKey(0)
cv2.destroyAllWindows()


Learning time of arrival (ETA) from historical location data of vehicle I have location data of taxis moving around the city sourced from: Microsoft Research
Overall it has around 17million data points.

I have converted the data to JSON and filled up mongo. A sample looks like this:

{'lat': 39.84349, 'timestamp': '2008-02-08 17:38:10', 'lon': 116.33986, 'ID': 1131}
{'lat': 39.84441, 'timestamp': '2008-02-08 17:38:15', 'lon': 116.33995, 'ID': 1131}
{'lat': 39.8453, 'timestamp': '2008-02-08 17:38:20', 'lon': 116.34004, 'ID': 1131}
{'lat': 39.84615, 'timestamp': '2008-02-08 17:38:25', 'lon': 116.34012, 'ID': 1131}
{'lat': 39.84705, 'timestamp': '2008-02-08 17:38:30', 'lon': 116.34022, 'ID': 1131}
`{'lat': 39.84891, 'timestamp': '2008-02-08 17:38:40', 'lon': 116.34039, 'ID': 1131}
{'lat': 39.85083, 'timestamp': '2008-02-08 17:38:50', 'lon': 116.3406, 'ID': 1131}

It consists of a taxiID - ID field, timestamp of its latitude and longitude combination.

My question is: I want to use this data to calculate estimated time of arrival(ETA)

So far, I am doing it a crude way by querying mongoDB with aggregation. It is totally inefficient.

I am looking at some sort of learning algorithm where the historical data can be used to train it. In the end, given two points, the algorithm should traverse the possible route by referring historical data and give an estimate of time.
Calculating time estimate is not a problem at all if I get the array of JSON documents between the points. But, getting those right arrays is.

Any pointers in this direction will be very helpful.

Based on what I figured out from your problem:

1

You can easily convert your data to a graph using Networkx, igraph or any other tool/library/software. Then what you need is a Shortest Path Algorithm (Dijkstra is widely used and implemented in all graph/network analysis softwares). Once you created the graph you can simply calculate the average estimated time.
For turning the problem into a Learning Problem, you can use historical time estimations for different paths and assign a weight to an edge proportional to the property of that edge (e.g. traffic jam probability, time conditions) and try to predict the ETA for a new query.

2

You can also turn it into a Network Science Problem and use Graph Theoretc approaches to approach the question. You can start with statistical analysis of nodes and edges attributes e.g. passing time distribution, shortest path length distribution, probabilistic modeling of traffic jam and so on to see if some meaningful insight leads you the next step.
Another idea is to use graph clustering algorithms to extract most connected parts of the town and go through the analysis of them i.e. calculate the ETA for different parts instead of whole the data and assign the estimated time to the members of corresponding cluster and reduce the computational complexty if your algorithm.

3

The last but not least is having a look at ArangoDB. It's a new database model which is based on graphs and you can run queries on millions of edges in an amazing speed! all what you need is a bit of javascript knowledge and even if you don't have it you can use AQL language designed for arangoDB. The interesting point is that it uses JSON files as the standard data format so you are already half way through ;)

Hope i could help :)
Good Luck!


How to convert binary classifier to multiclass classifier? I am a biggener student in Machine learning, and I want to ask if is it possible to convert a binary classifier label (y) by applying some condition on column1 to get a third situation.

I.e. Instead of having just 2 cases "Easy" and "Difficult" as output, I need to apply a condition on an additional feature in order to get as output "Easy", "Normal", "Difficult".

and I need also some keyword that can I use on Google as a request to look for solution.

Yes it is.

For multiclass classification problems, you can use 2 strategies: transformation to binary and extension from binary.

In approaches based on transformation to binary, you have:

OVA (one versus all), which is based on training k binary classifiers (k = #classes), where the i-th classifier is specialized on distinguishing the i-th class from all the other k-1 classes.
OVO (ove versus one), which is based on training k * (k-1) / 2 classifiers, where each classifier learns to distinguish 2 classes only. When a prediction is required, each clasisfier votes on the class it thinks it's correct, and the class with more votes is selected as the output.

On the other hand, you have extension from binary approaches: some classification algorithms are already capable of dealing with these multiclass problems. Some examples: kNN, decision trees, naive bayes...

You can find a bunch of resources on this.
For more practical purposes, please check out the following resource: https://scikit-learn.org/stable/modules/multiclass.html


Suggestion on tag clustering visualization I have a database of tags given by users to the product. For example

user; product; tag
1; A; Tag1
1; A; Tag2
2; A; Tag1
2; B; Tag1
..
..

I am trying to cluster tags which are given together to any product. At the end I want to visualize them like PCA plot where I can see clusters of tags which are 'closer' (tendency of user to assign these tags together) to each other. So far I can think of applying t-SNE or simply PCA to get some kind of clustering with available tags.

For this purpose, I made frequency matrix like following

first_tag; second_tag; occurrence
Tag1; Tag2; 2393
Tag1; Tag3; 38
Tag2; Tag3; 8393
..
..

I am stuck here. I don't know how to proceed with clustering. I simply tried visualizing it with networkx library of python where I made edges with 'first_tag' and 'second_tag' and 'occurrence' as its weight. But it was futile exercise, I couldn't get anything out of it. I thought of using sklearn.decomposition PCA but I am struggling to convert this data-set into proper matrix which can be fed to such algorithms. I can always make n x n matrix by making rows and columns equal to number of tags but it was causing very slow and sometimes out of memory errors. Any other elegant solution for this?

Instead of clustering, association rules is likely to be much more appropriate on such data.

Nevertheless it's obviously possible to treat cooccurrences as a similarity measure and cluster this for example using a variant of hierarchical clustering (for similarities, not distances!) Or by transforming the similarities to distances. You can also try to treat these as affinities for tSNE or MDS, for example (again, your usual library will likely not just work, but you will need to write some code yourself), but it's probably much easier to simply visualize the cooccurence matrix as a "heat map", after arranging similar tags close to each other with hierarchical clustering.


Can all data be represented tabularly? ... assuming that tables allowed for nested non-primitive values.
I'm trying to think of any instance where data cannot be represented as tables (or a series of tables)... and I can't think of any. However, I'm skeptical of myself on this one... its seems too simple to defined a pseudo-law which state "all data can be represented as tables".
Can anybody think of instances where this isn't true?

One way or another any data can be represented in a table or even in a big binary string, since after all the physical memory of a computer is just one big binary sequence. But the question is whether the table format adequately represents the semantics of any data, and the answer is definitely no: while there are tabular representations for graphs, text, images or videos, these representations simplify and/or make assumptions about the nature of the data. Virtually any representation is a simplification of the data it represents, but some representations are more faithful to the original data than others.


How much data is needed for a GBM to be more reliable than logistic regression for binary classification? When comparing a GBM to a logistic regression for a binary classification, there a pros and cons to each. I'm interested in understanding the general tradeoff between the length of the data set (number of rows) vs the reliability of the fit out-of-sample.

Obviously the more data, the more reliable the predictions will be out of sample (all else being equal). The more rows of data, the more likely that GBM will be more predictive on a real data set. So i'm wondering is there a rough rule of thumb when you would say the data is too small to use GBM, better of using a logistic regression. I know that logistic regressions can be more predictive in some cases, but my understanding is that GBM usually outperforms.

I'm particularly interested in the case that the number of predictive variables is far less than the total number of variables available. For example, we have 100 variables, 10 of which might be highly predictive of the target variable.

I know about all the benefits of train/test sets and CV. But that doesn't work all the time either. For example, your colleague has just run the train/test loop too many times, has added too many unreliable variables into the mix to improve the test AUC, and now you have a GBM model that is crazy good. Too good.

In my experience GBM does at least as well as LR on small datasets too. The main advantages of GBM over LR occur when you have

correlated variables
nonlinearity
interactions

If you want to simplify your GBM you can use feature importances to drop unimportant features. You might also want to try PCA to see if you can reduce the number of features. After this you could try a LR model on the resulting features to see the model performance.

If you're worried that you've overfit your model to the test set, you'll need to find a new test set!


Hot Encode vs Binary Encoding for Binary attribute when clustering I am planning to use data for a clustering problem that contains a column with a binary value BUY/SELL.

Should I be converting this attribute and assign it binary values (BUY=1, SELL=0), and keep it on the same column, thus reducing the number dimensions

OR

Hot encode the attribute (adding two columns BUY and SELL and putting 1 on the appropriate column)?

How do these two methods of nominal to numeric conversion affect the final model for popular clustering algorithms (K means, Hierarchical, etc...)

Not much of difference in your case. The difference is in just 1 dimension which does not affect much. The only point I can add is that if the number of BUY and SELL values are not the same, you can replace them with their frequencies i.e. if 40% BUY and 60% SELL, then replace BUY with 0.4 and SELL with 0.6


Coursera ML - Does the choice of optimization algorithm affect the accuracy of multiclass logistic regression? I recently completed exercise 3 of Andrew Ng's Machine Learning on Coursera using Python.

When initially completing parts 1.4 to 1.4.1 of the exercise, I ran into difficulties ensuring that my trained model has the accuracy that matches the expected 94.9%. Even after debugging and ensuring that my cost and gradient functions were bug free, and that my predictor code was working correctly, I was still getting only 90.3% accuracy. I was using the conjugate gradient (CG) algorithm in scipy.optimize.minimize.

Out of curiosity, I decided to try another algorithm, and used Broyden–Fletcher–Goldfarb–Shannon (BFGS). To my surprise, the accuracy improved drastically to 96.5% and thus exceeded the expectation. The comparison of these two different results between CG and BFGS can be viewed in my notebook under the header Difference in accuracy due to different optimization algorithms.

Is the reason for this difference in accuracy due to the different choice of optimization algorithm? If yes, then could someone explain why?

Also, I would greatly appreciate any review of my code just to make sure that there isn't a bug in any of my functions that is causing this.

Thank you.

EDIT: Here below I added the code involved in the question, on the request in the comments that I do so in this page rather than refer readers to the links to my Jupyter notebooks.

Model cost functions:

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def compute_cost_regularized(theta, X, y, lda):
    reg =lda/(2*len(y)) * np.sum(theta[1:]**2)
    return 1/len(y) * np.sum(-y @ np.log(sigmoid(X@theta))
                             - (1-y) @ np.log(1-sigmoid(X@theta))) + reg

def compute_gradient_regularized(theta, X, y, lda):
    gradient = np.zeros(len(theta))
    XT = X.T
    beta = sigmoid(X@theta) - y
    regterm = lda/len(y) * theta
    # theta_0 does not get regularized, so a 0 is substituted in its place
    regterm[0] = 0
    gradient = (1/len(y) * XT@beta).T + regterm
    return gradient

Function that implements one-vs-all classification training:

from scipy.optimize import minimize

def train_one_vs_all(X, y, opt_method):
    theta_all = np.zeros((y.max()-y.min()+1, X.shape[1]))
    for k in range(y.min(),y.max()+1):
        grdtruth = np.where(y==k, 1,0)
        results = minimize(compute_cost_regularized, theta_all[k-1,:],
                           args = (X,grdtruth,0.1),
                           method = opt_method,
                           jac = compute_gradient_regularized)
        # optimized parameters are accessible through the x attribute
        theta_optimized = results.x
        # Assign thetheta_optimized vector to the appropriate row in the
        # theta_all matrix
        theta_all[k-1,:] = theta_optimized
    return theta_all

Called the function to train the model with different optimization methods:

theta_all_optimized_cg = train_one_vs_all(X_bias, y, 'CG')  # Optimization performed using Conjugate Gradient
theta_all_optimized_bfgs = train_one_vs_all(X_bias, y, 'BFGS') # optimization performed using Broyden–Fletcher–Goldfarb–Shanno

We see that prediction results differ based on the algorithm used:

def predict_one_vs_all(X, theta):
    return np.mean(np.argmax(sigmoid(X@theta.T), axis=1)+1 == y)*100

In[16]: predict_one_vs_all(X_bias, theta_all_optimized_cg)
Out[16]: 90.319999999999993

In[17]: predict_one_vs_all(X_bias, theta_all_optimized_bfgs)
Out[17]: 96.480000000000004

For anyone wanting to get any data to try the code, they can find it in my Github as linked in this post.

Limits of numerical accuracy and stability are causing the optimisation routines to struggle.

You can see this most easily by changing the regularisation term to 0.0 - there is no reason why this should not work in principle, and you are not using any feature engineering that particularly needs it. With regularisation set to 0.0, then you will see limits of precision reached and attempts to take log of 0 when calculating the cost function. The two different optimisation routines are affected differently, due to taking different sample points on route to the minimum.

I think that with regularisation term set high, you remove the numerical instability, but at the expense of not seeing what is really going on with the calculations - in effect the regularisation terms become dominant for the difficult training examples.

You can offset some of the accuracy problems by modifying the cost function:

def compute_cost_regularized(theta, X, y, lda):
    reg =lda/(2*len(y)) * np.sum(theta[1:]**2)
    return reg - 1/len(y) * np.sum(
      y @ np.log( np.maximum(sigmoid(X@theta), 1e-10) )
      + (1-y) @ np.log( np.maximum(1-sigmoid(X@theta), 1e-10) ) )

Also to get some feedback during the training, you can add

                       options = {
                           'disp': True
                       }

To the call to minimize.

With this change, you can try with regularisation term set to zero. When I do this, I get:

predict_one_vs_all(X_bias, theta_all_optimized_cg)
Out[156]:
94.760000000000005
In [157]:

predict_one_vs_all(X_bias, theta_all_optimized_bfgs)
/usr/local/lib/python3.6/site-packages/ipykernel/__main__.py:2: RuntimeWarning: overflow encountered in exp
  from ipykernel import kernelapp as app
Out[157]:
98.839999999999989

The CG value of 94.76 seems to match the expected result nicely - so I wonder if this was done without regularisation. The BFGS value is still "better" although I am not sure how much I trust it given the warning messages during training and evaluation. To tell if this apparently better training result really translates into better digit detection, you would need to measure results on a hold-out test set.


Autenocoder and anomaly detection task I'm trying to create an autoencoder for the anomaly detection task, but I'm noticing that even if it performs very well on the training set, it starts to stop recreating half of the test set. I tried with more than 10 models, (LSTM, ConvAE, ConvLSTM) and all of them fails to reconstruct the time series in the same point.
These are the performances on the training set. The blue part is the original time series and the red one is the time series reconstructed by the AE.

These are the performance on the training set. I don't understand why all the models stop performing from that point. Could that means that there are anomalies in that part?

EDIT: I'm updating the question with some details about my dataset and my code: I have a dataset with 30 devices, and for each one I have about 9000 values. The dataset is structured as well:
device1   device2   device3   ....   device30
 0.20      0.35      0.12              0.56
 1.20      2.10      5.75              0.16
 3.20      9.21      1.94              5.12
 5.20      4.32      0.42              9.56
 ....      ....      ....              ....
 7.20      6.21      0.20              -9.56

Since I'm following this guide, I started creating a sequence method to prepare my data for the Conv1D layer:
TIME_STEPS = 10
# Generated training sequences for use in the model.
def create_sequences(values, time_steps=TIME_STEPS):
    output = []
    for i in range(len(values) - time_steps):
        output.append(values[i: (i + time_steps)])
    return np.stack(output)

This is where I create the sequences and normalize the dataset:
# split the train/val/test set
n_features = dataset_sequences.shape[1]
X_train = dataset_sequences[0:3000, :]
X_val = dataset_sequences[3000:6000, :]
X_test = dataset_sequences[6000:9000, :]

# normalize the data
train_mean = X_train.mean()
train_std = X_train.std()
X_train = (X_train - train_mean) / train_std
X_val = (X_val - train_mean) / train_std
X_test = (X_test - train_mean) / train_std

Then, I feed my X_train with shape (3000, 10, 30) to my Conv1D autoencoder:
model = tf.keras.Sequential(
    [
        tf.keras.layers.Input(shape=(X_train.shape[1], X_train.shape[2])),
        tf.keras.layers.Conv1D(
            filters=64, kernel_size=5, padding="same", strides=1, activation="relu"),
        tf.keras.layers.Dropout(rate=0.2),
        tf.keras.layers.Conv1D(
            filters=32, kernel_size=5, padding="same", strides=1, activation="relu"),
        tf.keras.layers.MaxPooling1D(pool_size=2),
        tf.keras.layers.Conv1DTranspose(
            filters=32, kernel_size=5, padding="same", strides=1, activation="relu"),
        tf.keras.layers.Dropout(rate=0.2),
        tf.keras.layers.Conv1DTranspose(
            filters=64, kernel_size=5, padding="same", strides=1, activation="relu"),
        tf.keras.layers.UpSampling1D(size=2),
        tf.keras.layers.Conv1DTranspose(filters=30, kernel_size=5, padding="same"),
    ]
)

Looking at your graphic there seems to be a change in the series behaviour right about when your models start predicting 0. Since you have had the same issue with several models I would say that this is a data issue.

Have you checked looking at these values?
Try plotting the 1-period return?
Try skipping the parts in the validation set that work and make prediction directly from when the model is starting to fail - or even a bit after i.e X_test[1200:] or X_test[2000:]


Is there a rule of thumb for a sufficient number of trials for hyperparameter search I am implementing a quite complicated Bayesian hyperparameter search in hyperopt library on a CNN.
Is there a rule of thumb for a "sufficient" number of trials? Perhaps based on the number of hyperparameters that constitute the search space?

In a pure random search, 60 points is often given as a rule of thumb, because provably with probability 95% such a search finds a hyperparameter combination in the top 5%.
However, that 5% is as a percent of the volume of the search space, so giving much-too-broad a search space, the best 5% might not be a fantastic score for the model.  So it does seem to depend, not exactly on the number of hyperparameters, but on the volume of good combinations, which in turn indirectly depends on the number of hyperparameters.
Bayesian optimization should take fewer iterations (but not necessarily less time!) to reach an optimum than a pure random search, and in particular should rule out swaths of bad volume relatively quickly.
So, to conclude...no, I don't actually have a rule of thumb.  60 is probably a decent bet, given the tradeoff of the last two paragraphs; I'd go up to 100 if the training isn't too expensive.  Also consider whether your package allows you to continue a Bayesian search: you could analyze the results so far (mostly, see if the last few iterations have clustered around some point, and how widely the objective function varies) to see if you want to proceed.  Finally, note that scikit-optimize sets a default of 50, but there doesn't seem to be much behind that.


How to match up categorical labels in training and evaluation I am creating a CNN to categorise a sentence into one of N possible labels.

I have used the tutorial from WildML to start the code, and I have modified it to allow multiple outputs instead of just true/false.

I am using VocabularyProcessor to convert the words into numbers and then using OneHotEncoder to transform the labels for evaluation. I am currently using the same VocabularyProcessor for converting both X and y

Where x_text is the loaded sentences and y_text is the correct labels

max_document_length = max([len(x.split(" ")) for x in x_text])
vocab_processor = learn.preprocessing.VocabularyProcessor(max_document_length)
X = np.array(list(vocab_processor.fit_transform(x_text)))

y_val = np.array(list(vocab_processor.transform(y_text)))

from sklearn.preprocessing import OneHotEncoder
onehotencoder = OneHotEncoder(categorical_features = [0])
y = onehotencoder.fit_transform(y_val).toarray()
y = y[:, 1:]

This then goes through the training.

The problem that I am finding is that during evaluation, if the labels are not exactly the same then the indexes do not match the prediction that is coming out.

Basically what my question is, is how do I match up the output of the evaluation with the text label so that I can use it after training without hard coding the labels up front

You are going about it all wrong. You shouldn't transform your labels into numerical values and match them to the output; that's not how classification is done. Instead, for each class (category) you will add a node in the output layer; the predicted class is then the argmax of the output layer.

For example, let's say you have three classes: simple, compound and complex. We'll assign an integer for each class starting from $0$.
Your output layer will have three nodes, one for each class, and for each training example $x$, the true label $y$ is a three dimensional "hot-encoded" vector. (i.e $y = (y_0, y_1, y_2)$ where $y_i = 1$ if $x$ belongs to class $i$ otherwise $y_i = 0$ for $i \in \{0,1,2\}$).

Now, let's suppose you feed your (trained) network an input value x and it outputs the following vector:
$y' = (0.4, 1.4, 2.5)$. The predicted label in this case is $\arg\max(y') = 2$. So the prediction for the sentence x is: complex.

Note: since argmax is not differentiable, we use softmax during training to convert the output vector y' into a vector of probabilities that sums up to $1$ before applying the loss function. The prediction in this case is the class with the highest probability. For example, if we apply softmax to the same y' from the example above, we get: $\text{softmax}(y') = (0.08, 0.23, 0.69)$. It's clear that the highest probability ($69\%$) belongs to the class $2$ (complex) and that these probabilities sum up to $1$.


Understanding the XLNet model for a concrete case I'm a data science student, recently I reviewed the XLNet paper and I have a doubt about it:
Imagine we are using many categories, let's say 200, can this model has problems reaching a good accuracy (only applying what the paper says), if not, why? I thought that if we combine a hierarchical classification and then apply the XLNet model it can lead to better results for many categories but I'm not sure because many examples in Kaggle and other websites use XLNet directly. Has some sense what am I saying or I didn't properly understand what XLNet does since I didn't see anyone applying this proposal for many categories?

There's no way to do that, XLNet is a pretrained model so the only thing we can do is tune it.


Can we calculate AUC for deep learning based regression task There is  a paper [ref.attaced], where they used a deep  learning based regression and evaluated using mse and AUC. The targets are continuous values such as 1,2,3 up to 16 and has been normalized before training.
Is it possible to calculate  the AUC for regression task.?

AUC is for binary classification tasks only, it's not possible to calculate it on a regression model. However it's plenty of other metrics one can use to check a regressor performance, such as MAE and MSE.


Is adding the embedded words of a sentence to represent the sentence a good approach? I have a dataset of sentences in a non english language like :

word1 word2 word3 word62

word5 word1 word2

Now i want to turn each variable length sentence to a fixed size vector to give it to my model, and i want all the words in the sentences to have effect on the output
I thought maybe i can use an algorithm like word2vec and turn each word into a fixed size vector, and add all of them to represent the sentence, is this a meaningful approach? is this better than adding the hot one vectors of the words to represent the sentence?  is there a better approach than these two?
EDIT1: basically i have a dataset of random variable length sentences and i want to embed them the best way possible, meaning keeping as much  information as possible in the resulting embedded vectors (which all have the same size)

So the question asks how to represent a series of words a uniform vector representation, which is not dependent on sequence.
The idea you suggested is definitely not a bad idea, you should try that out.
You should also try Doc2Vec, which works on the same principle as Word2Vec but this time it will output a vector which represents the meaning of a section of text that is longer than 1 word.
The main problem with this sort of representation is that you lose sequential information of the text and you treat words in sentences as a “bag of words”. If you happy to make that assumption, then continue using your approach of Doc2Vec.
Otherwise, you might better off using a sequential model architecture, such as RNN/LSTM, etc. Here you can input each word at a given time step initially as a one hot encoded vector and then you add an embedding layer before it goes into the sequential model to transform the one hot encoding into a word embedding.


feature engineering in test and train sets (on combined data or separately on train and test) Background: As part of prediction analysis, I am given a train and test dataset. Both train and test data have numerical and categorical predictor variables and additionally, train data has a numerical target variable. The objective is to predict target in the test.

train = [c1,c2,x3,x4, y] = [Xc,X, y]
test = [c1,c2,x3,x4] = [Xc,X]

Xc, X denotes categorical and numerical predictor variables. I am trying to generate additional features from categorical variables Xc such as count features, count_mean, count_variance and similar features from a combination of categorical variable and a numerical variable (mean, variance etc).

Problem: Is it better to generate features on a combined dataset train+test or is it better to generate features separately on train and test datasets?

What are the implications when the distribution of a categorical variable are different in train and test and what happens when they are similar?

The idea is to build features based on your training set. In a real-world application, all you will have is a training set to build and deploy a model, and that model will predict the test examples that the application needs. In this setting, it is not realistic to assume that you will have the test data to generate features, as you will already have your model delivered. So, the mechanism to generate the features can only be fed with training information.

About this:

  What are the implications when the distribution of a categorical variable are different in train and test and what happens when they are similar?

The implications of having different distributions of the data in the training and test sets are very bad. Your learning algorithm will only learn about things that are on the training set, and for that reason, if your test set has a different distribution you should consider changing your training set. However, if they follow the same distribution, then your algorithms will generalize properly if you do things right.


Devices behavior in one continuous variable vs events rate I have devices on which I have time series data of one continuous variable. I have to evaluate the relation between the profile of that variable on those devices and "events".

Those events are given in terms of occurrences on a time period.

My first intention is to make clusters of similar behavior of that variable and compare those clusters with the low/middle/high events rates.

I was thinking about doing a K-means with the min, max, quartilles, mean, normal q-q p value, Kurstosis, etc. as dimensions, but I don't think it's a good idea because:

Those dimensions are not independant
It's "losing" data and so potentially losing classification potential

Do you have some suggestions to group similar devices together?

Also, do you have other ideas to establish that relationship?

Context:

python3 with the scipy stack
~ 3000 devices and hundreds of thousands of data per day; 5 months to consider

Done with K-means clustering with descriptive statistics as features:

In short, I've tried the idea described in the question, even if I was thinking it won't work. Let the experience talk...

I initially had a list of devices data. Each element of the list were 2 columns, R rows matrix, and R was different for each device. So, per device:

[
    [mesureValue, timestamp],
    ...,
    [mesureValue, timestamp],
]

Since I'm only interested in the measureValue distribution, I've transformed the inital data to a 8 columns, N rows matrix, where N = number of devices.

The columns are, computed on the correponding device's measure value:

Arithmetic mean
Median
First quartille
Third quartille
Minimum
Maximum
Range
Standard deviation

With this matrix, I've applied K-means clustering using scikit learn (python).

I made the link between the matrix line and the physical device by using pandas Data Frames (python) who's line index are in fact the serial number of the device.

I've tried with 5 clusters, and it works.

Just in case of, if I need improvements in the future, I'm planning to add other statistics in the columns, especially for deviation vs. normality. So, for example Kurstosis and normal q-q plot p value.

Best regards.


Gaussian Mixture Models EM algorithm use average log likelihood to test convergence I was investigating scikit-learn's implementation of the EM algorithm for fitting Gaussian Mixture Models and I was wondering how they did come up with using the average log likelihood instead of the sum of the log likelihoods to test convergence.

I see that it should cause the algorithm to converge faster (given their default parameters), but where does that idea come from ?

Does anyone know if they based this part of the implementation on a specific paper or if they just came up with it and used it ?

In most explanations of the EM algorithm I have come across, they would have used log_likelihoods.sum() instead of log_likelihoods.mean().

It makes unit testing easier; invariant to the size of the sample.

Reference: the github discussion that led to the change.


PCA: projection of positive data on negative side of plane I did PCA on my data and projected the data on first two eigen vectors. After projection I see that the scatter plot of the data starts from [-1,-1]. My data is all positive. Is it correct for the data to be negative in the projected space.

Yes, it is, there is no reason why a component cannot have negative values.

Let's take this matrix:

[[1, 1],[0.5, 1]]

The eigenvectors have negative values:

[[ 0.81649658, -0.81649658],
    [ 0.57735027,  0.57735027]]


Semantic Segmentation using Keras: loss function and mask I am about to start a project on semantic segmentation with a grayscale mask. The thing is, we have to detect for each pixel of the image if its an object or the background (binary class problem). I struggle to relate this pixel binary classification task with a mask labelling, since each pixel will be in a range [0,255]. I have started implementing an U-net with Keras according to this methodology (being fairly new with keras).

What kind of loss would you use? - I was thinking of binary_crossentropy
What kind of labelling would you use? And therefore, what would be the output shape of the CNN if I do this binary classification pixelwise?

Sorry if I do not use the proper technical terms.

let me see if I can help.

(1) I would definitely recommend binary crossentropy for your loss function.

(2) Your labels should be "masks", which are images (the same size as your input images) where your "0-class" pixels are 0's and your "1-class" pixels are 1's.  This is basically a black and white image where black and white represent the 2 different classes. The output of the UNET network will be a single channel image (the same dimensions as the input) where your "0-class" is colored black and your "1-class" is colored white.

Here is an example:

The output mask in this image is what your label ("mask") should look like.
If you train your network right, it should also be the output of UNET.  Train for about 40 epochs and you will start to see results.


What exactly is BatchNormalization() in Keras? A month or two straight away into building image classifiers, I just sandwiched the BatchNormalization layer between conv2d. I wonder what it does, but I have seen my model learn faster in presence of these layers.
But I'm worried if there's any catch? I read somewhere that I don't need dropout layer if I'm using batch normalization! Is it true?
And also tell me in which manner should I use this layer. In which kind of problems I should and shouldn't use this layer?
Just write down anything you know about the layer that you think will help me!

Batch Normalization is a layer that is put in between convolution and activation layers or sometimes after activation layers. It is used to normalize layer’s input to reduce the internal covariate shift problem.
This problem occurs by changing in distribution of the input data in the early layers, and because every layer depends on the input of the former layers, it becomes a problem for this layer since it requires repeatedly adjusting to new input distributions.
Batch Normalization is a technique for training very deep neural networks that standardizes the inputs to a layer for each mini-batch. This has the effect of stabilizing the learning process and dramatically reducing the number of training epochs required to train deep networks.
It can be implemented during training by calculating the mean and standard deviation of each input variable to a layer per mini-batch and using these statistics to perform the standardization.
You read more here, here or here.
Also, I can tell you that this layer has a good impact on the performance and the achieved scores of your model.
Hope this helps you understand the importance of BN.


Supervised clustering I'm working on a clustering problem. I have a training set composed of sets of points where the clusters are known and I want to find the good clusters on a testing dataset.
It's a kind of supervised clustering.

I looked for articles about supervised clustering but I didn't find a lot of informations. There is "semi-supervised clustering" which consists of using informations on couples of points (must-link or don't-link relations) but, in my task, I don't have this kind of information. There are also some kind of "metric learning supervised clustering" which uses the labelized clusters to estimate a metric that would produce the given clusters using k-means. That kind of technique could help me but there is not much articles about it and I wonder if I'm not finding the good keywords or something.

What are the techniques/algorithms to cluster data points using labelized data (training points with known clusters) ?

What you are looking for is called KNN algorithm, also knows as k-nearest neighbours. It’s a supervised algorithm where you have points and their clusters given and you use these to learn a pattern for test points.


classification imbalance data - bias and class weight This page shows a classification problem. They have used bias as well as bias along with class weights.

What is difference between bias and weights? In some other techniques such as Random Forest we could adjust cut off from 50% to say 90% so that if the edge node has 90% examples of high density class then only it will predict that class. Out of bias and weights, which one is similar to cut off probablity?
When should I use just one of them or both? I can always run 3 models 1 with bias, 1 with class weight and 1 with both and compare the results. But is there any better refrence?

keras.layers.Dense(1, activation='sigmoid',
                         bias_initializer=output_bias)

and

weighted_history = weighted_model.fit(
    train_features,
    train_labels,
    batch_size=BATCH_SIZE,
    epochs=EPOCHS,
    callbacks = [early_stopping],
    validation_data=(val_features, val_labels),
    # The class weights go here
    class_weight=class_weight)

Normally the bias is "part" of the weights, and for this particular algorithm or model, the bias is a way to use to help solve unbalanced data issues. For fraud detection, the rate of event data is really rare, here event is the transaction is a fraud.

In this so-called machine learning model, the whole tutorial was doing logistic regression, a century-old statistical learning. We mostly using it for classification.

keras.layers.Dense(1, activation='sigmoid',
                         bias_initializer=output_bias),

At the last layer of the model, it has a sigmoid activation, here it's a logistic function (https://en.wikipedia.org/wiki/Sigmoid_function).

For fraud detection, the data in a tabular format, aka, the data is one dimension, not like the image. For one dimension data, statistics will shine over deep learning. Since the target event is rare, a representative sample is unlikely to have enough target events to build a good predictive model. Fortunately, the amount of information in a data set with a categorical outcome  (such as a response to a marketing campaign and fraud detection) is determined not by the total number of cases in the data set, but by the number of cases in the rarest outcome category.

One way the tutorial borrowed from the statistics is oversampling. While oversampling improve the model performance with less time, it also introduces some biases. Yo need to correct these biases so that the results or the model learned are applicable to the population. For example, one widespread strategy for predicting rare events is to build a model on a sample that disproportionally over-represents the event cases. Here you choose a sampling of data that contains all of the events (fraud) and only a subset of the nonevents (normal transaction).

Again, it will introduce biases, and later you need to correct them, this is why you need to calculate the correct bias.

Don't confuse with another bias from the coefficient, it is from the same rationale, but for now and here, you should think the bias we are talking now is not $\beta_0$ or the intercept in the linear equation but the bias of the model.
$$y=\beta_0+\beta_1 x$$
"The linear equation assigns one scale factor to each input value called a coefficient, and it is commonly represented by the Greek letter Beta (β). One more coefficient is added, giving the line an additional degree of freedom (moving the line upwards or downwards) and is called intercept or the bias coefficient."

The effect of the oversampling is like the figure showing below.

This graph of the response surface for a logistic regression model, which is the machine learning model in the tutorial. As you can see, oversampling does not affect the slops, but it does make the intercept too high, which is why we call it bias also. The difference between the intercepts is called offset, whose equation is $ln(\dfrac{\pi_{0}\rho_{1}}{\pi_{1}\rho_{0}})$.

$\pi_0$ is the proportion of non-events in the population
$\pi_1$ is the proportion of events in the population
$\rho_1$ is the proportion of events in the sample
$\pi_0$ is the proportion of non-events in the sample

Another method for adjusting for oversampling is to incorporate sampling weights, by correcting via sample weights. These weights are not the $\beta_1$ from the linear regression function, nor from the machine learning weight, it is a ration for the event and non-event classes.

$ weights_{i} = \dfrac{\pi_{1}}{\rho_{1}}$ if $y_i = 1$
$ weights_{i} = \dfrac{\pi_{0}}{\rho_{0}}$ if $y_i = 0$

The offset method and the weighted method are not statistically equivalent. So the parameter estimates are not exactly the same, but they have the same large-sample statistical properties. So here the offset, using bias approach is better and considered superior.

The bottom line is there are two sets of biases and weights:
- model parameter: you can think bias and weight as $\beta_0$ and $\beta_1$.
- oversampling: bias is talking the sampling method is non-traditional, and weights are the "proportional rate" between classes.

Hope this helps.


Address parsing using spaCy I am trying to parse addresses from various documents using spaCy using NER but the results are not so accurate.
I know this is bit generic question but it would be a great help if I could get reference of any past work or good articles or techniques to apply to this.

Please look at my comment to add more information to your post. Based on the information you provided, here are my remarks:

SpaCy is trained to find locations, not addresses per se

If you use a "common" language, SpaCy is trained using WikiNER data, where locations aren't addresses but more like geographical places like city names, country names etc. So it's quite normal to not be able to detect full addresses.

You likely need to train your own entity recognizer. They detail how to do this on their website, including code samples: https://spacy.io/usage/training#ner

Don't underestimate SpaCy's rule-based matching

Is it a fancy neural network? No. Does it matter? Also no. SpaCy allows you to create rules to find entities and in cases like addresses which are generally following a pattern across entities.


How to work with different Encoding for Foreign Languages I've got a Word Embedding File called model.txt. This contains 100 Dimensional vectors for over a million French words. These words contain accented characters such as é, â, î or ô.
Let me explain my problem with the following example:
Consider these two words and their respective vectors, both of which are taken from model.txt:
etait -0.100460 -0.127720 ...

était 0.094601 -0.266495 ...

Both words signify the same meaning but the former is without the accents while the later has accents.
Now I'm trying to load this word embedding using the gensim.models.KeyedVectors in the following way:
model = KeyedVectors.load_word2vec_format(open(model_location, 'r',
                                              encoding='utf8'),
                                          binary=False)
word_vectors = model.wv

To which I get the following error:
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-82-e17c33c552da> in <module>
     10 model = KeyedVectors.load_word2vec_format(open(model_location, 'r',
     11                                               encoding='utf8'),
---> 12                                           binary=False)
     13
     14 word_vectors = model.wv

D:\Anaconda\lib\site-packages\gensim\models\keyedvectors.py in load_word2vec_format(cls, fname, fvocab, binary, encoding, unicode_errors, limit, datatype)
   1547         return _load_word2vec_format(
   1548             cls, fname, fvocab=fvocab, binary=binary, encoding=encoding, unicode_errors=unicode_errors,
-> 1549             limit=limit, datatype=datatype)
   1550
   1551     @classmethod

D:\Anaconda\lib\site-packages\gensim\models\utils_any2vec.py in _load_word2vec_format(cls, fname, fvocab, binary, encoding, unicode_errors, limit, datatype, binary_chunk_size)
    286                 vocab_size, vector_size, datatype, unicode_errors, binary_chunk_size)
    287         else:
--> 288             _word2vec_read_text(fin, result, counts, vocab_size, vector_size, datatype, unicode_errors, encoding)
    289     if result.vectors.shape[0] != len(result.vocab):
    290         logger.info(

D:\Anaconda\lib\site-packages\gensim\models\utils_any2vec.py in _word2vec_read_text(fin, result, counts, vocab_size, vector_size, datatype, unicode_errors, encoding)
    213 def _word2vec_read_text(fin, result, counts, vocab_size, vector_size, datatype, unicode_errors, encoding):
    214     for line_no in range(vocab_size):
--> 215         line = fin.readline()
    216         if line == b'':
    217             raise EOFError("unexpected end of input; is count incorrect or file otherwise damaged?")

D:\Anaconda\lib\codecs.py in decode(self, input, final)
    320         # decode input (taking the buffer into account)
    321         data = self.buffer + input
--> 322         (result, consumed) = self._buffer_decode(data, self.errors, final)
    323         # keep undecoded input until the next call
    324         self.buffer = data[consumed:]

UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 7110-7111: invalid continuation byte

which I thought made sense if my file was encoded in a different format. However, using git I tried checking the encoding of the file using file * and got:
model.txt: UTF-8 Unicode text, with very long lines

Now, if I try to write the above code and have the encoding set to latin1, there isn't any problem to load this document but at the cost of not being able to access any of the words which contains an accent. Essentially throwing an out-of-vocab error upon executing:
word_vectors.word_vec('était')
How am I supposed to approach the problem? I've also got the .bin file of the model, should I try to use that to load my words and their corresponding vectors?

Nevermind, the solution was trivial. Since I had the .bin file I could just open it in binary form. If somebody doesn't really have the .bin file, they could consider converting the .txt file to .bin and solve further.


Decision Trees - how does split for categorical features happen? A decision tree, while performing recursive binary splitting, selects an independent variable (say $X_j$) and a threshold (say $t$) such that the predictor space is split into regions {$X|X_j < t$} and {$X|X_j >= t$}, and which leads to greatest reduction in cost function.

Now let us suppose that we have a variable with categorical values in {$X$}. Suppose we have label-encoded it and its values are in the range 0 to 9 (10 categories).

If DT splits a node with the above algorithm and treat those 10 values are true numeric values, will it not lead to wrong/misinterpreted splits?
Should it rather perform the split based on == and != for this variable? But then, how will the algorithm know that it is a categorical feature?
Also, will one-hot encoded values make more sense in this case?

You are right on all counts:

  If DT splits a node with the above algorithm and treat those 10 values are true numeric values, will it not lead to wrong/misinterpreted splits?

Yes absolutely, for exactly the reason you mention below:

  Should it rather perform the split based on == and != for this variable? But then, how will the algorithm know that it is a categorical feature?

Yes, as you correctly assume a (true) categorical variable should be compared only for equality, not order.

In general the algorithm cannot guess the nature of the feature, there has to be some parameters in the implementation which provide it with this information. Some implementations allow this, for example with Weka the features are typed with either a "numeric" or "nominal" (categorical) type.

  Also, will one-hot encoded values make more sense in this case?

Correct again, that's what should be done for a categorical feature in case the implementation treats all the features as numeric values.


Series prediction for any given time I have a time series of data points. Then I am given a future timestamp and  I have to predict the value for the data point. For simplicity, you can assume that the timestamp is bounded i.e. for e.g. query can be of at most 1 hr in the future.

This is different from tradition train and predict models. Here you will be given the time as query input apart from past data.

Currently, I am training different models for each minute(yeah, 60 models: lots of waiting time ).

I am wondering if there is something available for this specific task?

EDIT: To get the view about the data, you can assume a simple time series of real numbers, and I have to use history to predict for any general time(within next 1 hr).

Considering ocam's razor I would recommend to use the simplest model first and increase the complexity if the simple models fail:

Exponentially Weighted Moving Average, allows only for auto correlation with lag one
ARIMAX, allows for several lagged autocorrelation, seasonal adjustments and external regressor.
Fourier transformation, allows to fit more shapes time series but they are often more complex to explain to users

All these models can predict one step ahead and you can repeat this up to 60. Of course the confidence intervals become much wider these many steps ahead. Base R has many time series available.

These two models work well for many time series data. I would only start to use neural nets when these are not working or if you are looking for say edge detection.

Neural nets

If neural nets are the only option to go you can check this post or if you have matlab available this post.

I recently found this very interesting article on medium that explains how to fit a Neural Net to time series data. A more detailed implementation guide can be found here. I have not tried either approaches but I thought that the may be useful for future reference.

Time series modus operandi

Note that in any case, before staring off modeling your time series it is highly recommended to take some preparatory steps:

Make your series stationary as explained here
Understand what elements you can decompose in your series as explained here

Hope this helps


Why joint probability in generative models? I have been reading about generative models for past few days and there's one question that's bugging me. Many sources(Wiki, Google dev article) mention that generative models try to model the joint probability distribution p(x, y) and that to generate new samples we sample from this distribution. My question is why model and sample from the joint distribution and not the conditional distribution p(x|y)?

Its because joint distribution takes all of the values variable Y can take and conditional only one. Additionally it could be easier to compute/use approximative methods for joint.


How many model parameters did Microsoft's 2015 ImageNet-winner network have? In 2015 a team of researchers from Microsoft won the ImageNet contest with a network of 152 layers.

How many parameters did their model have?

The number of parameters is 60,344,232, according to:

https://resources.wolframcloud.com/NeuralNetRepository/resources/ResNet-152-Trained-on-ImageNet-Competition-Data

Although, from https://arxiv.org/pdf/1512.03385.pdf (page 6), it looks like an ensemble was used for ILSVRC 2015:

"We combine six models of different depth to form an ensemble (only with two 152-layer ones at the time of submitting). This leads to 3.57%
top-5 error on the test set (Table 5). This entry won the 1st place in ILSVRC 2015."


Multiclass Classification with large number of categories I am making a recommendation system (kind of) and I have to recommend the item a user is most likely to buy in his next purchase. Doesn't matter if he already bought this item.

Given this, I'm treating this problem as a multiclass-classification problem with 4000 categories (number of different items users can buy).

Searching in Wikipedia I found this link and decided to use the One vs -rest method. So I decided to train one random forest for each item using as covariates flags if the user bought each item before (so I have around 4000 covariates). Then I will decide a rule to decide the recommended item (something like the one which has the largest probability to be bought or the largest lift.)

My problem is that it's taking too long to train (5 to 10 min per item):

> 5*4000
[1] 20000
> 20000/60
[1] 333.3333
> 333.3333/24
[1] 13.88889

So in the best case it would take 2 weeks to train.

I would like to know if the method i'm using is right, and if there's another faster method to achieve this.

You might have more luck with a Naive Bayes Classifier.  It can handle a large number of target classes, and is relatively fast to train, since you largely just calculate a bunch of univariate stats to plug into at prediction time.  It won't capture fancy interactions as much as a random forest though, so if you are concerned about "they only buy shoelaces if they bought shoes but NOT shoeshine" vs "they often buy shoelaces if they've bought shoes" then it may disappoint.  You may also want to incorporate a time component, but I'm not sure what you're doing.

https://en.wikipedia.org/wiki/Association_rule_learning may also be relevant.


Vertical lines of overlapping points in Seaborn Swarmplot When I plot my data to see its distribution via Swarmplot, these two vertical lines that seems to even overlap occur in the figure.

It seems to occur even around values that are not that dense, such as around -2.5.
I feel like I might be missing how Swarmplot works but anyone knows the explanation?

As per the Sea born documentation, -

https://seaborn.pydata.org/generated/seaborn.swarmplot.html

"This function is similar to stripplot(), but the points are adjusted (only along the categorical axis) so that they don’t overlap. This gives a better representation of the distribution of values,

  but it does not scale well to large numbers of observations.

This style of plot is sometimes called a “beeswarm”."

A workaround may be adjusting the figsize.


Issue with Custom object detection using tensorflow when Training on a single type of object I am training a pre built tensorflow based model for custom object detection. I want to detect only 1 type of object. I have taken lot of images from different angles and in different light conditions. I am training on K80 Nvidia GPU. Everything is working and when I train I can see the loss function falling to 0.3. But the loss values drops very quickly to under 1 when I start training. I am using SSD mobile Net as the base configuration for the model. When I try to test the model, it just draws a big square on the input image, rather than detecting the desired object in the image. Basically, it fails to detect the object.

I tried to train the model with a different set of images of mac n chesse which had lot of variations. Then the model worked fine and detected images of mac n chesse in the input image. But when I have pictures of single object then the model fails to detect. Please help me understand what I am doing wrong here

How many images are you using to train?  You might need more... let's say  at least 1000 for a single category - depending on how big your MobileNet is (there are different variants).

Are you using a pretrained MobileNet, or are you training it from scratch. If Mac'n Cheese was not a category in the original data used for a pre-trained model, then you might need to train from scratch, or only keep the first few layers of the model frozen, as they learn very general features from images that will liekly be useful for your target class (and many other). If you are unsure, search for transfer learning, to see what is possible.

The nominal value of the loss is quite hard to interpret, so values of 0.3 or 1.0 don't mean a lot without context. You can use them for comparison - so if you change something and loss values then start dropping to 0.005, you know that you are likely onto a good idea.

You can check out this tutorial notebook on MobileNet, which might give you more ideas about how to play with the network


Classifier performance evaluation I have an unbalanced dataset which has 920 samples in total, 689 belong to the first class, and 222 to second class. and both classes are significant for me.
so when building a classifier model such as SVM or KNN. what measurement should I consider to evaluate the performance of the classifier? usually people use accuracy. but in my case some times I get high accuracy but zero specificity which clearly indicates that the class is biased towards the majority class (class one in my case).  I've been advised to use the F-score which combines both specificity and sensitivity. Also, there is the AUC.
so what do you suggest?

Useful metrics in such scenario are:

F1 Score (and precision / recall)
ROC Curves  (Metric is : Area Under the ROC Curve (AUC))

Few articles on how to choose metrics for a specific project are:

Evaluation Metrics, ROC-Curves and imbalanced datasets by David S. Batista,
What metrics should be used for evaluating a model on an imbalanced data set? by Shir Meir Lador,
Choosing the Right Metric for Evaluating Machine Learning Models — Part 2 by Alvira Swalin.


CUDA compatibility of GTX 1650ti versus 1650 I am confused about CUDA compatibility. I am studying deep learning and looking for a laptop to buy. One laptop has GTX 1650ti and another has GTX 1650. Will both be able to use GPU for model training, or only second one?
I checked for gpu compatibility. On the nvidia website only gtx 1650 is mentioned. But on some other forums I read that both can work.

Both can run CUDA to accelerate deep learning. Cards that end in "ti" are slightly better versions than their non-ti counterparts.
NVIDIA, the corporation that makes the 1650 and 1650ti cards, also develops CUDA. Only (and all) NVIDIA graphics cards currently support CUDA because it is proprietary.
However, because deep learning models are often limited by memory capacity, I recommend buying a card such as the GTX 1070, which is comparably priced but has 8GB of RAM.


How to visualise a large correlation matrix? I have a dataset with 24 variables, 21 of them numeric. As part of model building I decided to look into the correlation between features and so what I get is a large correlation matrix (21 * 21).
Now visualising such large matrices becomes a very messy task and you end up hurting your eyes. So what I have done is set a threshold and to slice out those rows that have greater than this value (say 0.60). However, I'm getting a matrix that has now several NaNs. When I try to drop these null values, the matrix loses all data and what I'm left is a 0*0 matrix.
corr_matrix = data.corr()

threshold = 0.60

high_corr = corr_matrix.loc[corr_matrix >= 0.60]

high_corr.dropna(inplace=True)

print(high_corr)
Empty DataFrame
Columns = []

Visualising the matrix with nans is a good idea but it also results in empty squares. I'm looking for a way where only those rows that have values >= threshold are retained, with no nans. That would make a much smaller matrix which is much less messier when plotted in matplotlib. However I haven't been able to code it that way; can anyone suggest some strategies to deal with such large matrices?

Try this (note that I didn't add error checking, so it'll crash if your threshold removes all values). Also, I made it an absolute high pass rather than a normal high pass because I assume you'd be interested in strong negative correlation as well? If you're not, just remove the abs() in the filter function.
from numpy.random      import randn
from pandas            import DataFrame
from seaborn           import heatmap
from matplotlib.pyplot import show
from itertools         import combinations

def absHighPass(df, absThresh):
    passed = set()
    for (r,c) in combinations(df.columns, 2):
        if (abs(df.loc[r,c]) >= absThresh):
            passed.add(r)
            passed.add(c)
    passed = sorted(passed)
    return df.loc[passed,passed]

labels = [chr(x) for x in range(65,91)]
corrDf = DataFrame(randn(26,26), index=labels, columns=labels).corr()

#heatmap(corrDf,cmap="YlGnBu")
heatmap(absHighPass(corrDf,0.5),cmap="YlGnBu")
show()

This is the filtered heatmap:

And this is another run but with the unfiltered heatmap:


Is it possible to automate generating reproducibility documentation? First, think it's worth me stating what I mean by replication & reproducibility:

Replication of analysis A results in an exact copy of all inputs and processes that are supply and result in incidental outputs in analysis B.
Reproducibility of analysis A results in inputs, processes, and outputs that are semantically incidental to analysis A, without access to the exact inputs and processes.

Putting aside how easy it might be to replicate a given build, especially an ad-hoc one, to me replication always possible if it's planned for and worth doing. That said, it is unclear to me is how to execute a data science workflow that allows for reproducibility.

The closet comparison I'm able to think of is documentation generators that generates software documentation intended for programmers - though the main difference I see is that in theory, if two sets of analysis ran the "reproducibility documentation generators" the documentation should match.

Another issue, is that while I get the concept of reproducibility documentation, I am having a hard time imagining what it would look like in usable form without just being a guide to replicating the analysis.

Lastly, whole intent of this is to understand if it's possible to "bake-in" reproducibility documentation as you build out a stack, not after the stack is built.

So, Is it possible to automate generating reproducibility documentation, and if so how, and what would it look like?

UPDATE: Please note that this is the second draft of this question and that Christopher Louden was kind enough to let me edit the question after I realized it was likely the first draft was unclear. Thanks!

To be reproducible without being just a replication, you would need to redo the experiment with new data, following the same technique as before.  The work flow is not as important as the techniques used.  Sample data in the same way, use the same type of models.  It doesn't matter if you switch from one language to another, so long as the models and the data manipulations are the same.

This type of replication will show that the results you got in the first experiment are less likely to be a fluke than they were earlier.


why does making the target variable normally distributed helps? while working on some regression problems I have found that if the target variable is skewed, making it normally distributed(using transformations) almost always helps. Why is that?

Should we also transform independent variables to have near normal distributions?

In some cases, it may actually help getting better results (depending on the model type), but it is also likely that the improvement comes from the fact that the performance metric is computed differently. For instance, a skewed distribution will lead to high MSE values due to cases located on the other side of the distribution, while the MSE is limited if the data is transformed to a normal distribution. So when comparing the cases, make sure you evaluate the performance on the back-transformed target.

Cases where the model will actually perform better with a normally distributed target include, among others, Gaussian process regression, because of the underlying assumption of a Gaussian random variable. There should be quite a few other model types which somehow have similar assumptions, and thus perform better with transformed data.


How can I make a face verification system? I'm building a face verification system using face embeddings (of 512 dimensions) from a pretrained Facenet model. For this, if I have some 4 to 5 images for a person, how can I successfully verify a new unseen images as either the same or different person?
I think this of one class classification (because of having single class) task, and googled it, but unable to find any reliable sources which suits well for this task.
Then, I've trained a SVM classifier with embeddings of a person and also with dummy embeddings of some 4 to 5 persons (so that having positive and negative classes). But it doesn't seem to work well.
Please suggest me with a ML algorithm/technique for this task, thank you.

I am assuming Facenet is an image classifier and it will give embedding for a face similar to other CNNs. If that is the case you don't need to train different classifiers, you can just remove the head of Facenet and initialize a small network on top of it and train an end to end network for better accuracies.


Non differentiable loss function I have a loss function that minimizes the error according to what I want the neural network to do. The problem is, that it is a nondifferentiable function. How can I handle this?

the loss function: $(1-y) \cdot log(1-p) + min((1-y)-(y \cdot log(p)))$

$y$: target
$p$: prediction
len((1-y)-(y*log(p))) = len(y) = len(p)

I have tried to smooth the minimum, but I am not sure this is good enough. As you can see, the min operator is nondifferentiable

How to handle a nondifferentiable loss function with Neural Networks?

You can optimize with non-gradient based methods. The field is called derivative-free optimization.
Local Search is one common approach for derivative-free optimization of neural networks.


Metrics values are equal while training and testing a model I'm working on a neural network model with python using Keras with TensorFlow backend. Dataset contains two sequences with a result which can be 1 or 0 and positives to negatives ratio in dataset is 1 to 9. Model gets the two sequences as input and outputs a probability. At first my model had a Dense layer with one hidden unit and sigmoid activation function as output but then I changed my models last layer to a Dense with two hidden unit and softmax activation function and changed my dataset's result using Keras to_categorical function. After these changes the model metrics which contains Accuracy, Precision, Recall, F1, AUC are all equal and has a high and wrong value. Here are the implementation I used for those metrics
def recall(y_true, y_pred):
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
    recall = true_positives / (possible_positives + K.epsilon())
    return recall

def precision(y_true, y_pred):
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
    precision = true_positives / (predicted_positives + K.epsilon())
    return precision

def f1(y_true, y_pred):
    precisionValue = precision(y_true, y_pred)
    recallValue = recall(y_true, y_pred)
    return 2*((precisionValue*recallValue)/(precisionValue+recallValue+K.epsilon()))

def auc(y_true, y_pred):
    auc = tf.metrics.auc(y_true, y_pred)[1]
    K.get_session().run(tf.local_variables_initializer())
    return auc

and here is the training result
Epoch 1/5
4026/4026 [==============================] - 17s 4ms/step - loss: 1.4511 - acc: 0.9044 - f1: 0.9044 - auc: 0.8999 - precision: 0.9044 - recall: 0.9044
Epoch 2/5
4026/4026 [==============================] - 15s 4ms/step - loss: 1.4573 - acc: 0.9091 - f1: 0.9091 - auc: 0.9087 - precision: 0.9091 - recall: 0.9091
Epoch 3/5
4026/4026 [==============================] - 15s 4ms/step - loss: 1.4573 - acc: 0.9091 - f1: 0.9091 - auc: 0.9083 - precision: 0.9091 - recall: 0.9091
Epoch 4/5
4026/4026 [==============================] - 15s 4ms/step - loss: 1.4573 - acc: 0.9091 - f1: 0.9091 - auc: 0.9090 - precision: 0.9091 - recall: 0.9091
Epoch 5/5
4026/4026 [==============================] - 15s 4ms/step - loss: 1.4573 - acc: 0.9091 - f1: 0.9091 - auc: 0.9085 - precision: 0.9091 - recall: 0.9091

after that I tested my model using predict and calculated metrics using sklearn's precision_recall_fscore_support function and I got the same result again. metrics are all equal and has high value (0.93) which is wrong based on the confusion matrix I generated

What am I doing wrong?

This may happen by different reasons:

You're model learned to fast (less than one epoch). In this case, u need to increase dataset (by adding some data or augmentations)
You're loss function doesn't work correctly return zero gradients every call
You 'train' model not in train mode
You data load incorrect. Test what is input for u model


How to predict based on multiple samples? I am relatively new to ML so I apologies in advance if my question shows lack of understating of the field.

The problem

A particular study course has a high drop-out rate and we want to reduce it.

The dataset

We have two data sets:

Data about the student themselves (e.g. scored of initial aptitude tests, demographic). This dataset includes the feature to predict. E.g.

ID, AGE, RESULT_TEST1, RESULT_TEST2,DROPPED_OUT
AAAA, 21, 0.6, 0.4, TRUE
BBBB, 20, 0.3, 0.9, FALSE

Exams taken by the students during the course. This dataset contains data about the type of the exam (e.g. subject, level of complexity) and the results obtained by the student. Some of the exams are required, some are mandatory and some can be taken on voluntary bases. That is, each student can take an arbitrary list of exams. E.g.

ID, EXAM_ID, EXAM_COMPLEXITY, EXAM_SCORE
AAAA, XXXXX, 0.8, 0.4
BBBB, YYYYY, 0.2, 0.8

The goal

The idea is to use ML to calculate the likelihood of a particular student to drop-out during the course using historical data. The system should be able to predict based on the two datasets, how likely is a student to at drop-out so that we can give him/her more support.

The challenge

How do I combine the two datasets to train a model? I could create a superset joining the two but what would I then use to predict a result?

E.g. If I train the model on:

ID, AGE, RESULT_TEST1, RESULT_TEST2, EXAM_ID, EXAM_COMPLEXITY, EXAM_SCORE
AAAA, 21, 0.6, 0.4,XXXXX, 0.8, 0.4
BBBB, 20, 0.3, 0.9,YYYYY, 0.2, 0.8

How do I predict the likelihood of DROPPED_OUT passing all the data I have for a new student (multiple samples)?

Any explanation or pointers to documentation would be greatly appreciated.

Thank you in advance.

Alex

This is a reasonably standard problem for supervised ML:

The class is the variable "dropped_out"
Given the goal to predict a variable which is specific to a particular student, an instance must represent a student, not an exam.

This definition of what an instance should consist of seems to be the part that you didn't reach yet: you correctly saw that you need to join the two datasets but in your example you join them by exam id. As a result you obtain "instances" which each represent a particular exam by a particular student, and of course the same student might appear several times in the data. The solution is to join your datasets by student id in order to make a single instance contain all the information for one student, i.e. something like this:

AGE, RESULT_TEST1, RESULT_TEST2, SCORE_EXAM1, SCORE_EXAM2, SCORE_EXAM3,...., DROPPED_OUT

However it seems that the exams are not normalized, so I see two options:

Simplification: for each student, give only some summary statistics about their performance at exams, for example min, max, avg, std dev for both the score and the complexity. This gives a fixed number of features (8 in my example), each with a specific role so that the ML method can "make sense" of it.
Refactor the data: if possible, rearrange the exam data so that a column corresponds to the same exam for different students. This would mean that the exam complexity is not needed anymore, because the distribution of the grades is the only thing which matters. It's ok to have some missing/undefined values for the students who didn't take a particular exam, most ML methods can deal with that.

The second option is very likely to give better results than the first, but it might be impractical to transform the data this way.


Input with variable length Classification problem I have a dataset with patient information with discrete labels (labels are stages of a particular disease) which needs to be predicted (Basically a classification problem).

The dataset looks something like below:

Patient#|Visit#|Other medical features related to patient and visit|label (disease stage)

So, I am interested in using patient's past visit data inorder to predict the current disease stage. But, the problem is that all the patients don't have equal number of visits. So, I can't just append the past visit information to predict the future visit label like below:

concat(Patient #n 1st visit (X = all input features)|label of this visit| Patient #n 2nd visit (X = all input features)) and then try to predict the label for 2nd visit using previous visit information.

In the above problem, the number of visit =1, but I have a varying number of visit for each patient. How can I tackle this problem?

The goal is to predict the current stage for one patient, so for the classification part you must have a single instance for every patient. So the question is how to transform your features repeated for every visit into a fixed number of features.

The standard option is to use expert knowledge to engineer the features. For example there might be some features which are only relevant at a certain point in time and no longer relevant later, so for these you can just keep the latest value. There might be some features for which the evolution across time is significant, so for example the average increase in the past N months might make sense. The idea is to "summarize" any number of visits into a fixed number of features.

A more advanced option would be to "vectorialize" your features, i.e. generate some kind of embedding using some unsupervised deep learning method (but that's not my area of expertise).


How to distort data in a clever way? For example, I have some time series. How can I change my data, so it will not be obvious to understand what was original values?

Ideally transformation would allow to revert and reconstruct original data with some noise(or to save the relationship between time series values with no option to revert to original values)

I thought of: multiplying each value of series by some number, which will do shift and scale operation or I can compute some continuous function(like log), where argument is my time series data.

Nevertheless, I believe there should be more advanced method.
Thanks!

Update:
After careful considerations, and answer of @D.W. below and trying to transform data, I found that I spend more time on this than on main problem I have.

Initial problem was twofold:

1) to send real data to home computer to play at home any time I want.

2) to send real data to home computer, transform it a bit, to share with people in the web, to get some advise on my main problem.

After thinking, stressing, askying, googling I decided to drop the idea to do anything.
Few good links to people who have the same problem and are intended to transfer, transform data:

For problem 1) I believe stream cipher is a an option to read more about ( as it transforms your data to a random string). Result is bad, as if one will look at your working computer logs, you are busted.

For problem 2) read about laplace noise. Also, believe to be bad.

Regards.

The short answer is: don't do that; anything you come up with on your own is likely to leak information in subtle ways.

The longer answer is: talk to your data protection authority.  Only use a method if they consider it acceptable.  (Hint: it's very likely that the answer you will get back is "no, don't do that".)

The technical details are: there is an entire body of work on how to anonymize or de-identify data, so it can be revealed in a way that can't be linked to the people whom the data is about.  However, there's a long history of attempts to do this that failed; where it was actually possible to re-identify the people, violating their privacy.  See the following research paper:

Myths and Fallacies of "Personally Identifiable Information".  Arvind Narayanan and Vitaly Shmatikov. Communications of the ACM, 53:6, June 2010.

If you wanted to consider a scheme to do what you want, you'd need to tell us a lot about the nature of the data, what it reflects, what the privacy interests are, how the data is distributed, what other external sources might be available that might enable re-identification, and many other details.  Since you haven't provided any of that information, your specific question isn't answerable in its current form.

But ultimately, I hope I've convinced that what you're trying to do is a bad idea.  Don't endanger people's data by going "cowboy" with some homebrew scheme to obfuscate the data; instead, talk to your data protection authority about what their requirements are.


Where exactly does $\geq 1$ come from in SVMs optimization problem constraint? I've understood that SVMs are binary, linear classifiers (without the kernel trick). They have training data $(x_i, y_i)$ where $x_i$ is a vector and $y_i \in \{-1, 1\}$ is the class. As they are binary, linear classifiers the task is to find a hyperplane which separates the data points with the label $-1$ from the data points with the label $+1$.

Assume for now, that the data points are linearly separable and we don't need slack variables.

Now I've read that the training problem is now the following optimization problem:

${\min_{w, b} \frac{1}{2} \|w\|^2}$
s.t. $y_i ( \langle w, x_i \rangle + b) \geq 1$

I think I got that minizmizing $\|w\|^2$ means maximizing the margin (however, I don't understand why it is the square here. Would anything change if one would try to minimize $\|w\|$?).

I also understood that $y_i ( \langle w, x_i \rangle + b) \geq 0$ means that the model has to be correct on the training data. However, there is a $1$ and not a $0$. Why?

First problem: Minimizing $\|w\|$ or $\|w\|^2$:

It is correct that one wants to maximize the margin. This is actually done by maximizing $\frac{2}{\|w\|}$. This would be the "correct" way of doing it, but it is rather inconvenient. Let's first drop the $2$, as it is just a constant. Now if $\frac{1}{\|w\|}$ is maximal, $\|w\|$ will have to be as small as possible. We can thus find the identical solution by minimizing $\|w\|$.

$\|w\|$ can be calculated by $\sqrt{w^T w}$. As the square root is a monotonic function, any point $x$ which maximizes $\sqrt{f(x)}$ will also maximize $f(x)$. To find this point $x$ we thus don't have to calculate the square root and can minimize $w^T w = \|w\|^2$.

Finally, as we often have to calculate derivatives, we multiply the whole expression by a factor $\frac{1}{2}$. This is done very often, because if we derive $\frac{d}{dx} x^2 = 2 x$ and thus $\frac{d}{dx} \frac{1}{2} x^2 = x$.
This is how we end up with the problem: minimize $\frac{1}{2} \|w\|^2$.

tl;dr: yes, minimizing $\|w\|$ instead of $\frac{1}{2} \|w\|^2$ would work.

Second problem: $\geq 0$ or $\geq 1$:

As already stated in the question, $y_i \left( \langle w,x_i \rangle + b \right) \geq 0$ means that the point has to be on the correct side of the hyperplane. However this isn't enough: we want the point to be at least as far away as the margin (then the point is a support vector), or even further away.

Remember the definition of the hyperplane,

$\mathcal{H} = \{ x \mid \langle w,x \rangle + b = 0\}$.

This description however is not unique: if we scale $w$ and $b$ by a constant $c$, then we get an equivalent description of this hyperplane. To make sure our optimization algorithm doesn't just scale $w$ and $b$ by constant factors to get a higher margin, we define that the distance of a support vector from the hyperplane is always $1$, i.e. the margin is $\frac{1}{\|w\|}$. A support vector is thus characterized by $y_i \left( \langle w,x_i \rangle + b \right) = 1 $.

As already mentioned earlier, we want all points to be either a support vector, or even further away from the hyperplane. In training, we thus add the constraint $y_i \left( \langle w,x_i \rangle + b \right) \geq 1$, which ensures exactly that.

tl;dr: Training points don't only need to be correct, they have to be on the margin or further away.


Andrew Ngs Class - Why Did He Change up the Cost Function? I am taking Andrew Ng's Machine Learning Intro class. Looks like he changed the cost function without any explanation in the second week. Specifically:

He no longer squares each deviation between the estimated and the actual value
He no longer divides the sum of deviations by 2 (I guess because he doesn't square it, but that leaves the question of why for both
He multiplies the whole thing by the first number in the sample for which he is finding the delta between estimated and actual. Why?

This image might explain it better for those familiar with the class:

Also, which of the two is "correct"? Or if there is no "correct" answer, what is the use case for each?

The difference is the "original" is the derivative of a cost function and "new" is the result of substituting in an actual cost function.

When the derivative of a function raised to a power, the exponent becomes a constant. In this case, the derivative of the squared loss function becomes a constant 2, and the 2 in the numerator then cancels with the 2 in the denominator. The result is just dividing by $m$.


Finding the appropriate CNN Model Architecture and Parameters I am currently creating a CNN model that classifies whether the font is Arial, Verdana, Times New Roman and Georgia. All in all there are 16 classes since I considered also detecting whether the font is regular, bold, italics or bold italics. So 4 fonts * 4 styles = 16 classes.

The data that I have used in my training are the following:

 Training data set : 800 image patches of 256 * 256 dimension (50 for each class)
 Validation data set : 320 image patches of 256 * 256 dimension (20 for each class)
 Testing data set : 160 image patches of 256 * 256 dimension (10 for each class)

Below is the sample screenshot of my data:

Below is my initial code:

import numpy as np
import keras
from keras import backend as K
from keras.models import Sequential
from keras.layers import Activation
from keras.layers.core import Dense, Flatten
from keras.optimizers import Adam
from keras.metrics import categorical_crossentropy
from keras.preprocessing.image import ImageDataGenerator
from keras.layers.normalization import BatchNormalization
from keras.layers.convolutional import *
from matplotlib import pyplot as plt
import itertools
import matplotlib.pyplot as plt
import pickle

image_width = 256
image_height = 256

train_path = 'font_model_data/train'
valid_path =  'font_model_data/valid'
test_path = 'font_model_data/test'

train_batches = ImageDataGenerator().flow_from_directory(train_path, target_size=(image_width, image_height), classes=['1','2','3','4', '5', '6', '7', '8', '9', '10', '11', '12','13', '14', '15', '16'], batch_size = 16)
valid_batches = ImageDataGenerator().flow_from_directory(valid_path, target_size=(image_width, image_height), classes=['1','2','3','4', '5', '6', '7', '8', '9', '10', '11', '12','13', '14', '15', '16'], batch_size = 16)
test_batches = ImageDataGenerator().flow_from_directory(test_path, target_size=(image_width, image_height), classes=['1','2','3','4', '5', '6', '7', '8', '9', '10', '11', '12','13', '14', '15', '16'], batch_size = 160)

 imgs, labels = next(train_batches)

 #CNN model
 model = Sequential([
     Conv2D(32, (3,3), activation='relu', input_shape=(image_width, image_height, 3)),
     Flatten(),
     Dense(16, activation='softmax'),
 ])

 print(model.summary())

 model.compile(Adam(lr=.0001),loss='categorical_crossentropy', metrics=['accuracy'])
 model.fit_generator(train_batches, steps_per_epoch = 50, validation_data= valid_batches, validation_steps = 20, epochs = 1, verbose = 2)

 model_pickle = open('cnn_font_model.pickle', 'wb')
 pickle.dump(model, model_pickle)
 model_pickle.close()
 print('Training Done.')

 test_imgs, test_labels = next(test_batches)

 predictions = model.predict_generator(test_batches, steps = 160, verbose = 2)
 print(predictions)

Can anyone suggest how will I know the right network architecture and parameters to get the optimal accuracy? How should I start tweaking my network?

In many cases in deep learning it works well  to start off with a model which has a very high capacity and potentially overfits. From thereon you can reduce the model capacity to narrow the gap between train and validation error. In this chapter of the Deep Learning Book by Goodwell you find a good description of manual hyperparameter selection and how they influence model capacity.

Moreover, for many tasks well engineered solutions already exist. So check what worked for similar tasks and try out these architectures. For example, MNIST handwriting recognition is somewhat similar to your task. Wikipedia gives several architectures which work well on MNIST.

The article "Handwritten Digit Recognition using Convolutional Neural Networks in Python with Keras" contains architectures for MNIST, too. Again, this might be close enough to your tasks so I suggest to give these a try. The article includes an architecture very similar to yours but also a more complex one.

In other cases you could also check out pre-trained models. But here it might not even be required here.


quest for online tutorials on CNN backprop I'm looking for video or text tutorial materials on CNN backprop that show how weights (convolutional kernels and last layer weights) are trained. Anyone know?

Here is one good material: https://becominghuman.ai/back-propagation-in-convolutional-neural-networks-intuition-and-code-714ef1c38199

This one also might help: https://leonardoaraujosantos.gitbooks.io/artificial-inteligence/convolution_layer.html (it also has other useful materials).