Does it make sense that datetime encodes one-hot-vector like one-hot-encoding or something else like I'm new to machine learning and deep learning. I've wanted to solve time series problem, which has data every single second. Plus, I've been doing research on word2vector and time series data lately. And one day, I've come up with an idea transforming sequence data like date time into one-hot-encoding?

    time
2017-11-01 00:00:01
2017-11-01 00:00:02
2017-11-01 00:00:03
2017-11-01 00:00:04
.
.
.

My idea has some limitations like below,

too high dimensions for learning (1day = 60* 60 * 24 = 86400(s))
unlimited time - time will be generated every moment even right now as well
difference between seconds is too small to learn

I want you to determine what I'm saying above about limitations. Plus, I'd like you to give me some idea to develop time series data into one-hot-vector for machine learning and deep learning? + what do you think about this idea?

If you are trying to predict future values then it doesn't make sense to treat them as categorical features. There is nothing you will learn that can predict future data, since you won't see those times again.

The same holds even if you are trying to predict unseen past data but each time appears only once or a small number of times.

Instead the time values should determine the order of your training data. This way you can avoid leaking future data, and models with state can learn features from the sequential nature of the data.

However, in some cases you may want to extract additional features from the time values. Here are some examples:

Time since last interesting event
Number of interesting events in last time window of size n
Time of day (morning / afternoon / etc)
Day of week
Holiday
Season

Let's make up an example. Here is a dataset of times that users visited a website:

time               user
2017-11-01 00:00   Alice
2017-11-01 00:00   Bob
2017-11-02 00:00   Chris
2017-11-03 00:00   Alice
2017-11-04 00:00   Alice
2017-11-04 00:00   Bob
2017-11-07 00:00   Chris
2017-11-10 00:00   Alice

And here is the same dataset with additional features we have added:

time               user    last_visit   weekend?   time_of_day
2017-11-01 16:22   Alice   N/A          No         afternoon
2017-11-01 11:13   Bob     N/A          No         morning
2017-11-02 20:35   Chris   N/A          No         evening
2017-11-03 16:07   Alice   2 days       No         afternoon
2017-11-04 17:20   Alice   1 day        Yes        afternoon
2017-11-04 10:44   Bob     3 days       Yes        morning
2017-11-07 08:06   Chris   5 days       No         morning
2017-11-10 17:11   Alice   6 days       No         afternoon

If we are trying to predict when a certain user might visit next, then these features might help us a lot. For example we might decide that Alice is more likely to visit in the afternoon, or Bob is unlikely to visit two days in a row.


how does word variety depend on total words? I want to compare the word variety of several books. But some are short, while others are long. So how can I correct for the fact that longer books will generally contain a larger number of unique words. I tried simply dividing the number of unique words in each book by the number of total words in each book. But I think that was overdoing it, because now the shorter books appear to have the most variety. What is the best approach to this?

The simple answer is to use a sample of equal size from every book, or even better to randomly extract several samples of equal size from every book and then use the mean across samples.

  I tried simply dividing the number of unique words in each book by the number of total words in each book

This is known as the type/token ratio, the simplest way to measure lexical density. I think it makes perfect sense in the case you describe, as far as I know usually it's not too biased.


Which input to use when generating a new sequence I want to use sequence-to-sequence architecture to generate sequences.

My data has such structure

[0, 0, 1, 0, ..., 0, 1] --> [12.34, 0.78, 1.54, 6.90, ..., 5.32]

I follow this tutorial to achieve it.

After forwarding through Encoder network encoder_hidden is used as a decoder_hidden. But what should I use as a first decoder_input to Decoder network?

The original tutorial uses a Start Of the Sequence token, but I can't use it because it is encoded as 0. Probably 0 as a number will give some additional information for decoder.

As you have seen, normally you need a "special token" to be given to the decoder as the first element in its input to start the autoregressive generation.

However, given that your output are real (floating point) numbers, it is a bit trickier, as you are not dealing with a discrete token vocabulary where you could simply reserve a token for that.

I would suggest using a specific value, like $0.0$. Your model should be able to figure out the no influence of the $0.0$ in the first position.

Another option would be to learn the value to be used as first token. You would have an extra trainable parameter that you use as value for the first position.


GridSearch mean_test_score vs mean_train_score I am working with scikit learn and GridSearch in order to find the best parameters in my classifiers.

I have a map of different hyperparameters and I want to print out GridSearch results, but I do not understand one thing - what is the difference between mean_test_score and mean_train_score?

As I understand, GridSearch performs cross-validation in order to find the best classifier, but how do these 2 params differ from one another? I always thought that cross-validation gives only one mean, which is a mean of the performance from trained models using N subsets of given data. For example, if I perform a cross-validation with X subsets, I will have X different accuracy scores and then I will have only one mean value.

So, how can I interpret these 2 parameters and what is the difference between them?

When you do k-fold cross-validation, you train k models, each one of them leaving the proportion $1/k$ of the data out. For each of the models, you can compute its train error and validation error. The train error will be the error on the data selected to train the model, and the validation error will be the data left out of the training.

For this reason, you have k training errors and k validation/test errors, and computing their averages will give you the quantities you are talking about.


Cluster a categorical variable without breaking the existing categories Let’s say I am modelling data on flights and delays. I have 14 airlines (carrier variable) in the data below. I would like to know if I can create fewer categories or clusters of this categorical data, so instead of having 14 airlines maybe end up with 3 or 4 clusters of airlines.

require(data.table)
# Data takes ~30 seconds to load
flights <- as.data.table(read.csv(url("https://raw.githubusercontent.com/wiki/arunsrinivasan/flights/NYCflights14/flights14.csv")))
length(unique(flights$carrier))

km <- kmeans(flights[,.(dep_delay,carrier)], centers = 4, nstart = 1)

Is there any methodology that could be used? I tried k-means but it seems it is not the most appropriate.

Without looking at the data, can't you just use your Domain Knowledge to create meaningful groups/attributes for the airlines data frame? Such as "low-cost carrier/premium", "foreign (non-USA) airline/US-Airline" , "long-distance carrier/domestic only", "founded in decade" ,  or whatever? With only 14 airlines you can assign appropriate values yourself, manually, without relying on some cryptic algorithm.

Besides, is your dataset identical to the dataset in the R package "nycflights13"?

If so, you could easily load it with library(nycflights13), after installing the package of course.


Strong overfitting accompanying strong class imbalance I'm training an xgboost binary classification model. The data I have is around 600k and positive is only 0.1% of it. I tried to use all overfitting prevention techniques xgboost has to offer (tune eta, gamma, min_child_weight, subsample feature/data, early stopping etc). However, my model either strongly overfits but with decent metric on test set or does not overfit too strong but general performances are bad. I've listed two training results (metric is auc-pr) here,

AUC-PR
130 trees
3800 trees

train
0.18
0.98

test
0.05
0.45

validation(early stopping)
0.06
0.46

I checked for data leakage but doesn't seem to have any. My question is:

Can I use the model with 3800 trees, even though it's overfitting, it's getting decent results on test set?
Do I absolutely have to be strict in preventing overfitting for this strong class imbalance scenario? If so, what are other techniques?
Anything I did wrong?

With complex data it's rare not to have any overfitting (or underfitting). Ideally one wants to avoid strong overfitting, and if given a choice between two models it's clearly safer to use one which isn't overfit. But if it's impossible to avoid, from a practical perspective there's no reason not to use a model for this reason, in my opinion. The problem with an overfit model is simply that it's going to perform poorly on fresh instances, so the worst that can happen is the performance that you observe on the test set.
Just an idea: I don't know if it makes sense with your data but it might be worth trying one-class classification on the minority class. This could be a way to avoid the imbalance issue, and could be followed by a second stage of regular binary classification.


Clustering geo location coordinates (lat,long pairs) What is the right approach and clustering algorithm for geolocation clustering?

I'm using the following code to cluster geolocation coordinates:

import numpy as np
import matplotlib.pyplot as plt
from scipy.cluster.vq import kmeans2, whiten

coordinates= np.array([
           [lat, long],
           [lat, long],
            ...
           [lat, long]
           ])
x, y = kmeans2(whiten(coordinates), 3, iter = 20)
plt.scatter(coordinates[:,0], coordinates[:,1], c=y);
plt.show()

Is it right to use K-means for geolocation clustering, as it uses Euclidean distance, and not Haversine formula as a distance function?

K-means is not the most appropriate algorithm here.
The reason is that k-means is designed to minimize variance. This is, of course, appearling from a statistical and signal procssing point of view, but your data is not "linear".
Since your data is in latitude, longitude format, you should use an algorithm that can handle arbitrary distance functions, in particular geodetic distance functions. Hierarchical clustering, PAM, CLARA, and DBSCAN are popular examples of this.
This recommends OPTICS clustering.
The problems of k-means are easy to see when you consider points close to the +-180 degrees wrap-around. Even if you hacked k-means to use Haversine distance, in the update step when it recomputes the mean the result will be badly screwed. Worst case is, k-means will never converge!


Clustering after PCA: Use the standardized data, or take into account the variation explained at each PC? I am interested in clustering daily gridded data.

Because of the many dimensions (gridpoints), I first perform PCA to reduce the dimensionality and keep the n-first PCs that account for at least 85% of the variation of the actual data. Then I use these n PCs as inputs to k-Means clustering.

My question is if I should use the standardized (mean=0, std=1) PCs as inputs to the k_Means clustering, or adjust the PCs based on the % of total variation that each PC accounts for. The adjustment can be something like PC[i] * Variation[i] /sum(Variation)

The 1st option results in PCs with the same variance, thus the clustering is unbiased. The 2nd option will end up with a bias towards the PCs that explain most of the variation.

Based on my understanding it is better to work with unbiased data. Nevertheless, in the case of PCA, the PCs are by default of varying importance. Would this support the use of the adjusted PCs?

I had a more careful check, and I will try to answer my question based on the additional insights. I would appreciate further answers/comments regarding the accuracy of the provided answer.

The actual results of the PCs are not standardized, and their variance is proportional to the square root of the total variance (variation) explained by each PC. These values are the ones that should be used for the subsequent clustering. In this way, the weighting of the PCs is indeed proportional to the (square root of the) explained variance.

If the standardized PCs (mean=0, std=1) are used, then the values should be adjusted with the square root of the % of variance explained by each PC. In this way, the adjusted PCs will have the same weighting as the actual PCs, and the only difference will be on the range of the values.

I run the analysis for clustering based on 1) Actual PCs, 2) Standardized PCs, 3) Standardized PCs adjusted with the % of the variance, and 4) Standardized PCs adjusted with the square root of the % of the variance. 1) and 4) were practically equal (I assume some small differences are due to the fact that the range of the values affects the instances very close to 0) and the best ones, followed by 3). Option 2) was by far giving the worst performance. The performance was tested by the Sum of Squared Distances between each field and the allocated centroid on the original dataset (daily gridded data).


Conditional Multivariate Gaussian Distribution - Section 2.3, equation 2.74 Please refer section 2.3 (page 86-87) in Pattern Recognition and Machine Learning - Bishop

$$\mathit{N}(\mathbf{x}|\mathbf{\mu}, \Sigma)$$
where

$
\mathbf{x} =
\begin{bmatrix}
x_{a} \\
x_{b}
\end{bmatrix}
$,
$
\mathbf{\mu} =
\begin{bmatrix}
\mu_{a} \\
\mu_{b}
\end{bmatrix}
$,
$
\mathbf{\Sigma} =
\begin{bmatrix}
\Sigma_{aa} && \Sigma_{ab}\\
\Sigma_{ba} && \Sigma_{bb}
\end{bmatrix}
$

The equation below expresses the Quadratic term of exponent in the Bivariate Gaussian probability distribution (ref. book eq. 2.70; ignoring $-\frac{1}{2}$):

$$(x - \mu)^{T}\Sigma^{-1}(x - \mu) = (x_a - \mu_a)^{T}\Lambda_{aa}(x_a - \mu_a)
+(x_a - \mu_a)^{T}\Lambda_{ab}(x_b - \mu_b)
+(x_b - \mu_b)^{T}\Lambda_{ba}(x_a - \mu_a)
+(x_b - \mu_b)^{T}\Lambda_{aa}(x_b - \mu_b)$$

  Author states "..conditional distribution $p(x_{a} | x_{b})$ can be
  evaluated from the the joint distribution $p(x) = p(x_{a}, x_{b})$ by
  fixing $x_b$ to the observed value and normalizing the resulting
  expression to obtain the valid probability distribution over $x_{a}$
  .." and then filters out only the linear terms with $x_{a}$ and $x_{a}^{T}$
  component. The result is presented in equation 2.74 as:

$$x_{a}^{T}\{\Lambda_{aa}\mu_{a} - \Lambda_{ab}(x_b - \mu_b)\}$$

I follow the steps, and see that above is due to $x_{a}^{T}$, $x_{a}$ in the first 2 terms of equation 2.70 - where "term(s)" refer to the components in 2.70 added using $+$ operator.

  However, I seem to be getting two additions elements in $x_a$, in
  excess of the those present in 2.74 - due to the linear $x_a$ factor
  in $3^{rd}$ term (of equation 2.70). Excess term(s):  $$ (x_{b} -
 \mu_{b})^{T}\Lambda_{ba}x_a $$

Where am I going wrong? Please guide.

  Note: I am taking the $x_a$ too to be linear component, whereas 2.74 is (seemingly)
  considering only the $x_{a}^{T}$ as linear term. Seems like I am losing track
  somewhere in the middle (perhaps in equation 2.71 - related extract,
  below):

$$ -\frac{1}{2}(x - \mu)^{T}\Sigma^{-1}(x - \mu) = -\frac{1}{2}{x^{T}\Sigma^{-1}x} + {x^{T}\Sigma^{-1}\mu} + constant \ldots eq. 2.71$$

...where 'const' denotes terms which are independent of ${x}$, and we have made use of the symmetry of $\Sigma$.

\begin{align}&-\frac12(x - \mu)^{T}\Sigma^{-1}(x - \mu) \\&= -\frac12(x_a - \mu_a)^{T}\Lambda_{aa}(x_a - \mu_a) \tag{1}
\\&-\frac12(x_a - \mu_a)^{T}\Lambda_{ab}(x_b - \mu_b) \tag{2}
\\&-\frac12(x_b - \mu_b)^{T}\Lambda_{ba}(x_a - \mu_a) \tag{3}
\\&-\frac12(x_b - \mu_b)^{T}\Lambda_{aa}(x_b - \mu_b) \tag{4}\end{align}

Now let's focus on $(1)$,  the linear part in $x_a$ are

$$-\frac12 (-x_a^T\Lambda_{aa}\mu_a - \mu_a^T\Lambda_{aa}x_a)=x_a^T\Lambda_{aa}\mu_a \tag{5}$$

Now focus on $(2)$, the linear part in $x_a$ is

$$-\frac12x_a^T\Lambda_{ab}(x_b-\mu_b) \tag{6}$$

Now focus on $(3)$,  the linear part in $x_a$ is

$$-\frac12(x_b-\mu_b)^T\Lambda_{ba}x_a=-\frac12x_a^T\Lambda_{ab}(x_b-\mu_b) \tag{7}$$

There is no linear term of $x_a$ in $(4)$,

Adding $(5)$ to $(7)$, we have

\begin{align}x_a^T\Lambda_{aa}\mu_a-\frac12x_a^T\Lambda_{ab}(x_b - \mu_b)-\frac12x_a^T\Lambda_{ab}(x_b - \mu_b)&=x_a^T\Lambda_{aa}\mu_a-x_a^T\Lambda_{ab}(x_b - \mu_b)\\
&=x_a^T(\Lambda_{aa}\mu_a -\Lambda_{ab}(x_b-\mu_b))\end{align}


How to deal with ternary Output neurons in the Output classification layer of a simple feedforward Neural Net? I was looking into the multi-label classification on the output layer of a Neural Network.
I have 5 Output Neurons where each Neuron can be 1, 0, or -1. independent of other Neurons.
So for example an Output would look like :

Output

1

0

-1

0

1

I used to take the tanh- activation function and partition the neuron into 3 ( y<-0.5, -0.5<y<0.5, y>0.5) to decide the class in each of those neurons after the prediction. See Figure below.

Question:
Are there any better alternatives on how to approach this ternary Output activation?
I stumbled upon this blog post regarding n-ary activation functions which I found very interesting.

I think the newly suggested would make the partitioning mentioned above much more meaningful!
Or do you think I should one-hot-encode my whole system such that a neuron can only have a value of 0 or 1 and then just use sigmoids activation with a threshold of 0.5?

The typical approach for this kind of scenario is to handle the output as a discrete element and have each output be a probability distribution over the discrete output space, that is, for each output you generate 3 numbers between 0 and 1 where the sum of them adds up to 1. You normally obtain this by generating for each position a vector of 3 elements in $\mathbb{R}$ instead of a single element and then passing it through a softmax function.
The typical loss function for that is the categorical cross-entropy, which receives the probability vector (although depending on the implementation and the parameters, it may receive the vector before the softmax, which are called the "logits"), and the index of the expected discrete output.
This is the usual approach for multi-class classification that you will find in most tutorials, like this one.


How can I restart model fit after x epochs if loss is high? I am having trouble sometimes getting a fit on my data, and when I restart fit (with shuffle=true) then I sometimes get a good fit.

See my previous question:

Why does my model sometimes not learn well from same data?

As a work around, I want to automatically restart the fitting process, if loss is high after x epochs. How can I achieve this?

Here is a simplified structure:

def train_till_good():
    # Some loop around train()

def train():
    load_data()
    model = VerySimpleNet2();

    checkpoint = keras.callbacks.ModelCheckpoint(filepath=images_root + dataset_name + '\\CheckPoint.hdf5')

    myOpt = keras.optimizers.Adam(lr=0.001,decay=0.01)

    model.compile(optimizer=myOpt, loss='categorical_crossentropy',  metrics=['accuracy'])

   LRS = CyclicLR(base_lr=0.000005, max_lr=0.0003, step_size=200.)

    tensorboard = keras.callbacks.TensorBoard(log_dir='C:\\Tensorflow', histogram_freq=0,write_graph=True, write_images=False)

    model.fit(train_images, train_labels, shuffle=True, epochs=num_epochs,
              callbacks=[checkpoint,
                         tensorboard,
                         LRS],
              validation_data = (test_images, test_labels)
              )

def VerySimpleNet2():
    model = keras.Sequential([
        keras.layers.Dense(112, activation=tf.nn.relu, input_shape=(224, 224, 3)),
        keras.layers.Dropout(0.4),
        keras.layers.Flatten(),
        keras.layers.Dense(3, activation=tf.nn.softmax)
    ])
    return model

  EDIT 1

Here is another question that describes the data:

Why does adding random pixels stop my model learning in cnn?

While you should be able to do that one way or another, it won't be a good solution to your problem described here and the other question. If the difference between runs are very high - ie it depends how you split your dataset, that probably means that there is something inherently wrong with the dataset or the way it is split into training and validation sets.

Somehow you might be able to get a good result on training and validation data and take that as your base model and use for making predictions, but that does not mean that it is the best result or even an accurate description of your model. For this purpose, you need to spare a third dataset to check the accuracy: splitting the entire dataset as (training/validation/test)

I know this does not actually answer your question, but I thought this would help your process.


how to assign back categorical variables to train and test data after training and testing using inverse_transform? how to assign back categorical variables to train and test data after training and testing using inverse_transform? Like training and testing, data will have encoded numerical values. So, how to assign back categorical values to those variables to train and test dataset after training and testing?
Please help me with this.

If you are using ordinal data
Use this to encode

from sklearn.preprocessing import OrdinalEncoder
oe=OrdinalEncoder()
social['Gender']=oe.fit_transform(social[['Gender']])

Use this to decode
social['Gender']=oe.inverse_transform(social[['Gender']])

If you are using nominal variables
Use this to encode
from sklearn.preprocessing import OneHotEncoder

ohe = OneHotEncoder()

temp_var= ohe.fit_transform(social[['Gender']])

Use this to decode
temp_var2=ohe.inverse_transform(temp_var)

Hope it helps


Classification of medical dataset I have medical data set with different types of attributes and I have to do the classification based on attributes: there are no class labels in this data set.  Suggest to me which algorithm works better for this dataset.

First of all, your question is not that clear. Next time try adding an example of your dataset.

As you probably know, Machine learning algorithms are described as learning a target function (f) that best maps input variables (X) to an output variable (Y): Y = f(X)  .

As @ankit-seth mentioned,
If you want to classify the data you need to have the labels, and then you can use ML algorithms to do that.
I'm guessing you need to cluster(group) your data - try reading about K-Means and go from there ...

good luck :)


High accuracy on test-set, what could go wrong? You are given a pre-trained binary ML classification model with 99% accuracy on the test-set (assume the customer required 95% and that the test-set is balanced). We would like to deploy our model in production. What could go wrong? How would you check it?

My answer was that the model could be biased towards our test set and could fail to generalize on yet to be seen data. We can check this by running the model on multiple unrelated test-set that it haven't seen before.
Is this the right angle?

There are many possible ways to make errors giving you a really huge score on test in Data Science. Here are a few examples :

Your test set is also in your train set : Imagine you have data from 2010 to 2020, and use 2019 as test. If you trained on all 2010-2020 without excluding 2019, then you'll test on a data well known by the model, since it was used to train it. Moreover, if the model tends to overfit (so fits "too perfectly and precisely" with training set), you could achieve a 99% accuracy

Data leakage : This is a phenomenom in which your test set contains info that you shouldn't have in real new cases. Example : you're using Titanic dataset, predicting who dies and who survives. Imagine now the dataset has an attribute called "Hour of death", empty if the person survived, and filled with an hour if he died. Then your model will just learn "if this attribute is empty then the person survive, else he died". On your test set, you'll apply your model, knowing this info that you shouldn't know if you had to predict true new cases

Wathever happens, a 99% accuracy have to make you wonder and look for errors, this is almost impossible to achieve unless your problem is REALLY easy (and might not need a Data Science model at all)


What is the best machine learning algorithm for large, noisy datasets with interaction between variables? My initial thought was a neural network but I don't see how a neural network can properly predict interaction between variables (ie. x1 * x2) since each node is just a sum of previous inputs?
Would a decision tree be better suited at capturing the interaction between variables?
My dataset is large, with 400 features and 5,000,000 instances. All data is in percentile and the label is also a percentile. The dataset is quite noisy as well, (customer data, predicting likelihood of becoming a return customer).

Probabilistic Random Forest tends to work better then other algorithms on noisy datasets. But the data you are using also plays a major role on whether a algorithm will work or not.
Check this paper Probabilistic Random Forest for more details. Happy Learning


Account for unknown error in time series data Given:
Time series data collected from sensors.

There is an unexpected gradual drop in the initial data when sensors are idle.
However, this drop is not so visible when sensors are active because the drop is masked by the actual measurements.

Main question: How do I eliminate this drop from the sensor output?

Not sure if this is an accurate assumption, but let's assume that the drop is a convolution of an unknown system X (leak, sensor drift etc.) with the system being measured G.

I am feeding this into a neural network.

How do I get an approximation to the unknown influence X (from initial data)?

My goal: Apply (-X) to all other data collected from G before training the neural net and expect an increase in performance.
Should I expect an increase in performance?

I would set up a sensor as idle, purposefully to determine the decay in the signal. Then try an ARIMA or similar model on the data collected to find out what an approximation of the decay function is.

Further to this you should determine if the model still fits if they decay is put into a "stop/start" mode.

Afterwards you can apply a corrective function to remove or take into account the decay on your other data.


Database missing extensions:tym_system_time,quantile orange widget error I'm working with the orange data mining tool. In most of the examples data  taken from files or google sheets links but in the blog link they given that we can take data from postgresql database tables also but when follow the steps of blog I'm getting error.I attached the screen shots of error.

I need help in solving this error.

This is not really a proper place to ask questions about software issues, but rather about data mining concepts.

However, I'll answer your question.
Your database is missing both extensions as specified in the error message. Here are some brief instructions.


When to apply binning? I have read quite a lot about discretization techinques, measuring WoE and IV etc., but the basic question - when is it worth to use binning and when isn't - is not entirely clear. On the one hand, the result of discretization seems quite "pretty" and more easy to handle for me, but on the other hand it is always emphasized that binning always causes information loss. Could you provide some clues when should one use this technique?

Mostly this is avoided if reasonably possible in my experience, but:

One scenario where you may want to consider binning is if you are doing modeling with a logistic model.  The coefficients indicate that per unit increase of the IV, the log odds of the dependent variable will increase by ___.  The important distinction is that we are assuming a linear relationship in the logit by saying "per unit increase, we see ___ increase", when in fact it may not be linear.  To fix this, one possible solution is two bin/discretize your IV to make the relationship more linear, and this interpretable.  This is only important if you are interested in interpretability, not just prediction.  There are also other techniques to account for this that may be preferable (like generalized additive models)

Another scenario is for certain kinds of "big" data where the signals can shift a little bit on an axis due to measurement/machine error, and binning techniques allow you to make sure that the correct signal is matched up closely to other runs where the signal may be in a slightly different position. like below:
you may bin 65,000 x-axis values into 3200 x-axis values and average their appropriate y-values.  This reduces dimensionality and accounts for the minor shifts in x-axis position that can occur.  e.g. bin 1324 will likely capture the same signal across all other runs, where if you specificed bin 23,950, it might correspond to bin 24,000 on another run of the machine/technique, and your columns wouldn't match up correctly between samples.  In the example below, by binning you are making sure each respective compound is correctly captured, considering slight x-axis variation.

Last but not least, among several other factors seen here and here, you are purposefully dropping information for the model, which means you will lose power, as you mentioned.  Also, when you are deciding a cutpoint to (for example) categorize a continuous variable as high vs low, how do you choose the cutpoint?  Can you defend your choice, or is it somewhat arbitrary?  Did you choose multiple cutpoints to see which one favored your results and thus committed the tempting garden of forking paths/multiple testing?


How does Xgboost learn what are the inputs for missing values? So from Algorithm 3 of https://arxiv.org/pdf/1603.02754v3.pdf, it says that an optimum default direction is determined and the missing values will go in that direction. However, or perhaps I have misunderstood/missed the explanation from the article, it doesn't say what exactly is the input.

For example, I have (parent) node A (with 50 inputs) splitting into node B and node C. Now, say of the 50 inputs there are 7 missing values.

The other 43 inputs are split into B and C accordingly. What I seem to be understanding is that, it will allocate the remaining 7 into B and C and determine which one gives a higher gain score; that will be the optimal direction.

However, given the 7 values are missing (Which means I don't know what are these 7 values), how does allocating missing values into any of the child nodes change the gain score, or rather minimize the loss function? This seems to suggest that Xgboost is inputting something for the missing values. I can't seem to find out what is Xgboost inputting for these missing values. I hope this question isn't too vague/general and easy.

Edit: I think "Missing values" may be a vague term. What I meant here is (From wiki) "In statistics, missing data, or missing values, occur when no data value is stored for the variable in an observation."

From the author himself (https://github.com/dmlc/xgboost/issues/21), he said
" tqchen commented on Aug 13, 2014
xgboost naturally accepts sparse feature format, you can directly feed data in as sparse matrix, and only contains non-missing value.

i.e. features that are not presented in the sparse feature matrix are treated as 'missing'. XGBoost will handle it internally and you do not need to do anything on it."

And,

" tqchen commented on Aug 13, 2014
Internally, XGBoost will automatically learn what is the best direction to go when a value is missing. Equivalently, this can be viewed as automatically "learn" what is the best imputation value for missing values based on reduction on training loss."

The procedure is described in their paper, section 3.4: Sparsity aware split-finding.

Assume you're at your node with 50 observations and, for the sake of simplicity, that there's only one split point possible.

For example, you have only one binary feature $x$, and your data can be split in three groups:

Group $B$: 20 observations such that $x=B$,
Group $C$: 20 observations such that $x=C$,
Group $M$: 10 observations such that $x=?$

The algorithm will split based on $x$, but does not know where to send the group $M$. It will try both assignments,

$$(B,  M), C \qquad \text{and} \qquad B, (C, M),$$

compute the value to be assigned to the prediction at each node using all the data, and chose the assignment that minimizes the loss.

For example, if the split $(B,  M), C$ is chosen, the value of the left node will have been computed from all the $B$ and $M$ samples.

This is what is meant by

  Automatically "learn" what is the best imputation value for missing values based on reduction on training loss.

Comment question:

  how can group M be calculated if I don't know what are the values for them?

The value we compute is not based on the feature $x$, which we don't know, but based on the known label/target of the sample. Let say we are doing a regression and trying to minimize the mean square error, and let say we have the following:

For group $B$, the mean of the target is $5$.
For group $C$, the mean of the target is $10$.
For the missing value group $M$, the mean of the target is $0$.

Note that even if the value of $x$ is missing, we know the value of target of the sample, otherwise we could not use it to train our model.

In this case, the split would be made on $(B,M), C$, the value assigned to the right node containing samples $C$ would be $10$, and the value assigned to the left node containing samples in $(B, M)$ would be the mean of the target for the whole group. In our example, the mean would be

$$\frac{|M|}{|B| + |M|}\text{mean}(M) + \frac{|B|}{|M|+|B|}\text{mean}(B) = \frac{10}{30}0 + \frac{20}{30}5 = 3.\overline{3}$$


When would you choose certain CNN pretrained models? I'm totally new to the field of deep learning so I'm just trying to get a sense of some of the decisions made, especially with convolutional neural networks. After reading a few blogs/articles about the different pretrained CNN architectures, I'm wondering:

When would you choose a certain architecture over another? I've seen a lot of examples using VGG-16 and I'm curious why this one is the go-to when it seems like such a large (and slow?) way to train? Are there certain examples of applications where one architecture would be better than another?

There is a list of models and their performance and parameter count here. I prefer to use the MobileNet model initially. It is about as accurate as VGG but has about 4 million parameters versus the 140 million for VGG. Thus from a computational standpoint and thus training time it is far more efficient. If you modify it to use the callbacks ReduceLROnPlateau and ModelCheckpoint (documentation is here) you can in most cases achieve very good performance. I have found using the Adamax optimizer works very well, documentation for that is here. The code to use MobileNet is shown below

mobile = tf.keras.applications.mobilenet.MobileNet( include_top=False,
                                                           input_shape=(image_size,image_size,3),
                                                           pooling='avg', weights='imagenet',
                                                           alpha=1, depth_multiplier=1)
x=mobile.layers[-1].output
x=keras.layers.BatchNormalization(axis=-1, momentum=0.99, epsilon=0.001 )(x)
predictions=Dense (len(classes), activation='softmax')(x)
model = Model(inputs=mobile.input, outputs=predictions)
for layer in model.layers:
    layer.trainable=True
model.compile(Adamax(lr=lr_rate), loss='categorical_crossentropy', metrics=['accuracy'])

If the model does not train well on your data set you can add more dense layers but it is always best to keep the model as simple as possible initially and try to optimize the hyper-parameters before making your model more complex.


Should Feature Selection processes be apply on training data or on all data? I've realized that on examples and guides, sometimes feature selection processes (correlation elimination, backward/stepwise) are applied on the train data after splitting all data but on the other hand, sometimes its applied on all data.

So is there any clear answer for that? Which is more logical?

Like any preprocessing step, feature selection must be carried out using the training data, i.e. the process of selecting which features to include can only depend on the instances of the training set.

Once the selection has been made, i.e. the set of features is fixed, the test data has to be formatted with the exact same features. This step is sometimes called "applying feature selection" but it's an abuse of language: it's only about preparing the test data with the features which were previously selected during the training stage.

Applying feature selection on the test data is a mistake because the training depends on it, so that would mean that the model "has seen" the instances of the test set and therefore invalidate the results on the test set.


What is the right way to store datasets for a CNN project Our image classification project has thousands of raw photos, masks and reshaped images. We store source code in git. But datasets don't belong to source code version control. How should we store thee sets of images?

You can use google drive to save these and import the drive on your code.


F - measure derivation (harmonic mean of precision and recall) We can define the F - measure as follows:

  $F_{\alpha}=\frac{1}{\alpha \frac{1}{P}+(1-\alpha)\frac{1}{R}} $

Now we might be interested in choosen a good $\alpha$. In the article  The truth of the F-measure the author states that one can choose the conditions:

  $\beta=R/P$, where $\frac{\partial F_{\alpha}}{\partial P}=\frac{\partial F_{\alpha}}{\partial R}$

and then we obtain $\alpha=1/(\beta^2+1)$ and

  $F_{\beta}=\frac{(1+\beta^2)PR}{\beta^2 P+R} $

It is said that The motivation behind this condition is that at the point where the gradients of E w.r.t. P and R are equal, the ratio of R against P should be
a desired ratio $\beta$. I understand that the condition will garantify that the user is willing to trade an increment in precision for an equal loss in recall. But I do not get why the equality of both partial derivatives correspond to these hypothesis. I would rather understand when one partial derivative equals the other partial derivative multiplied by minus one. Could anyone explain me why the desired condition (condition in words) correspond to this equality (condition in math terms)?

EDIT:

Well, we could do the following:

  $\partial F=\frac{\partial F_{\alpha}}{\partial P}\partial P+\frac{\partial F_{\alpha}}{\partial R}\partial R$.

And since we want for $\partial P=-\partial R$ that $\partial F=0$, we obtain easily the condition.

But I have one problem with this: Since $\frac{\partial F_{\alpha}}{\partial P}/\frac{\partial F_{\alpha}}{\partial R}=1$, and the fact that the gradient is perpendicular to each level curve (https://ocw.mit.edu/courses/mathematics/18-02sc-multivariable-calculus-fall-2010/2.-partial-derivatives/part-b-chain-rule-gradient-and-directional-derivatives/session-36-proof/MIT18_02SC_pb_32_comb.pdf) we would have that the level curve must have $m=-1$. Nonetheless, when I calculate the level curve for some constant $c$ I get the result,

$R(P)=\frac{c(1-\alpha ) P}{P-c\alpha}$,

which clearly is not a linear funcion with $m=-1$. What am I missing?

why the equality of both partial derivatives correspond to these
  hypothesis. I would rather understand when one partial derivative
  equals the other partial derivative multiplied by minus one.

Your intuition of "trading off" by "subtracting" a value is correct when you speak in terms of $\Delta R$ and $\Delta P$ (as you yourself noticed in the edit), but is not valid if you speak in terms of derivatives.

Think of it this way: a partial derivative with respect to a parameter tells you by how much your metric changes if you improve that variable. By the very design, $F$ will always increase if you improve either $R$ or $P$ - its partial derivatives are positive everywhere.

They are not equal everywhere, however. At some configurations $(R,P)$ you would gain more by improving one parameter rather than the other. Say, if your $R$ is much lower than $P$, your $F$-score would increase more if you improved the $R$ a bit rather than $P$.

In the case of an $F_1$ metric, $R$ and $P$ enter the formula symmetrically and it is kind-of easy to see that whenever $R=P$, either of the components contributes equally to the increase of $F$. In fact, the set of points where $R$ and $P$ are "equal in importance" is exactly the line $R=P$ (which can also be written as $R/P=1$, if you prefer).

Now consider the equation for $F_\beta$:

$$\frac{\text{constant}}{\beta^2\frac{1}{R} + \frac{1}{P}}.$$

Let us see how much it changes in response to increasing $R$ or $P$. As both parameters are in the denominator, the change in value of $F_\beta$ is proportional to the change in the value of its denominator, hence we may simplify a bit and only consider the change in

$$\beta^2\frac{1}{R} + \frac{1}{P}$$

You can now compute that a $dx$ increase in $R$ would decrease the value of the expression above by $\beta^2\frac{dx}{R^2}$. A similar increase in $P$ would decrease the value of the above expression by $\frac{dx}{P^2}$. There is a set of points $(R,P)$ where these effects are exactly equal. These are the points where
$$\beta^2\frac{dx}{R^2} = \frac{dx}{P^2} \quad\Rightarrow\quad \frac{R}{P}=\beta$$

When $\beta > 1$ you are effectively putting more marginal importance on recall, in the sense that until your recall is at least $\beta$ times the precision, it is always beneficial to work on it, rather than improving precision.

  What am I missing?

You are missing the fact that the condition $\frac{\partial F_\alpha}{\partial P} = \frac{\partial F_\alpha}{\partial R}$ does not hold everywhere, but just on the set of points $R=\beta P = \sqrt{\frac{1-\alpha}{\alpha}}P$. This line intersects the level curve at just a single point.

The line with slope -1 would be the tangent to the level line at that point - the green dashed line on the illustration below.


How to start with receipt OCR text detection I'm thinking about an OCR system for digitalizing receipts. On the input system would take a picture of receipt and then return classified data (total_sum = Y, date = X, etc.). My question is regarding how to start. My initial thought was that I should start with detecting classes (name of the shop, receipt id etc.) on image and splitting it, then I would send parts of image OCR. My second idea is more NLP based. I would normally pass an image to OCR system and then try to do some classification on text data. Which approach would be better?

Not sure if you tried Tesseract or pytesseract yet. I never worked on receipts but I had a different use case where I needed to segment images before I applied the tesseract for text recognition. For simple use case and especially in English, tesseract is quite powerful so I would start with the 2nd option first and see if it's good enough. Probably you may look at this answer on stackoverflow for idea.  https://stackoverflow.com/questions/55140090/pytesseract-reading-receipt


What to do if my target variable is column of lists? How I can transform my target variable(Y)?

As it is list, I cann`t use it for fitting model, because I must use integers for fitting.

you can have multiple Y. Each item will be a binary variable which means that item exists or not.


Reinforcement learning: decreasing loss without increasing reward I'm trying to solve OpenAI Gym's LunarLander-v2.

I'm using the Deep Q-Learning algorithm. I have tried various hyperparameters, but I can't get a good score.

Generally the loss decreases over many episodes but the reward doesn't improve much.

How should I interpret this? If a lower loss means more accurate predictions of value, naively I would have expected the agent to take more high-reward actions.

Could this be a sign of the agent not having explored enough, of being stuck in a local minimum?

How should I interpret this? If a lower loss means more accurate predictions of value, naively I would have expected the agent to take more high-reward actions.

A lower loss means more accurate predictions of value for the current policy (technically it is more complicated for Q-learning off-policy estimates, but the covergence will still be limited by experience reachable in the current policy). Unfortunately a loss metric in RL cannot capture how good that policy is.

So what it means is that your policy has settled into a pattern where values can be estimated well by the neural network that you are using for Q. For some reason it is not finding improvements to that policy - typically it should be doing that before the loss metric drops, as each improvement in value estimates should reveal better possible actions, and once those start being taken by a new policy, the value estimates become out of date, and the loss increases again.

  Could this be a sign of the agent not having explored enough, of being stuck in a local minimum?

Exploration could be an issue. The "local minimum" in that case is probably not an issue with the neural network, but that small variations in policy are all worse than the current policy. As you are learning off-policy, then increasing the exploration rate may help find the better states, at the expense of slower overall learning. Also, methods that explore more widely than randomly on each action could be better - e.g. action selection methods that consistently pick unexplored state/action pairs such as Upper Confidence Bound.

Also a possibility is that the structure of your network generalises well under the current policy, but is not able to cover better policies. In that case, whenever the exploration suggests a better policy, the network will also increase estimates of unrelated action choices - so it would try them, notice they are better, then back off as the new values also cause unwanted policy changes in other situations.

If you know a better policy than the one that is being found, then you could plot a learning curve with the policy fixed, see if the network can learn it. However, usually you will not know this, so you may be stuck with trying some variations of neural network architecture or other hyperparameters.

There are other methods than DQN (e.g. A3C, DDPG), as well as many add-ons and adjustments to DQN that you could try (e.g. eligibility traces, double learning).


Continuous Estimated Time of Arrival I'm trying to create a model for when a shipped product will arrive at its destination. There are several stages the delivery goes through, so it's not just drive time from point A to point B. My first model looks at the status of the product on the first stage and uses average time deliveries for that product time to predict the # of minutes after the first stage that the product will be delivered.
I want to make another model that gives more of a continuous prediction, taking into account how long the delivery has been in a certain stage. For example, if most deliveries are done with the first stage after 15 minutes and it's been 10 minutes, the model should account for that in the eta.
How would I approach this? I can feed in the # of minutes it's been in the stage as an input but it seems like I can come up with a huge amount of data for each example.
Sorry if this isn't a clear question.

If I understand correctly (not sure), it looks to me like you don't need a model which can predict at any time, you just need:

a model which predicts the ETA at any stage given information about the past stages. The easiest way to do that is probably to just train a different model for each stage, since the number of stages is fixed.
Then between two stages the ETA can be updated in a deterministic way: if the last stage was passed at time $t$ and the predicted ETA was say 10mn, then at time $t'$ the ETA is just 10mn - $(t'-t)$.


Alternatives to doc2vec? What are some alternatives to the doc2vec embedding model? I.e models that convert paragraphs/documents into vectors, not just models that take the mean/sum of the word embeddings of each word in the document.

Depending on your target task. If you are to classify documents, then e.g. fastText has it's own approach and there are other classification techniques, not strictly generating embeddings, like LSA / LDA (using topic modelling) or word mover distance.


Image classification using cnn I did image classification using CNN and it successfully classified the images  but How to save predicted images to separate folder for example i have two classes cat and dog after prediction how to save predicted cats images to cat folder and dog images to dogs folder

I haven't seen a library or module that will do this for you, but it should be simple enough to handle. If you saved the image names in your original dataset while loading, you can add the predictions to that same dataset and then loop through row by row, renaming the image based on the classification category.


How to check and correct misspelling in the data of pairs of words? I have user generated text containing names of ports often containing typos and the actual port names. I would like to correct the misspelling of user generated text containing the names of ports. Can anyone help me think of some ideas or steps to take?
Perhaps using word2vec? RNN?

thank you in advance.

Another way to reframe the problem is approximate/fuzzy string matching. FuzzyWuzzy is an approximate/fuzzy string matching package for Python. The package can find the nearest element in a set to a user's inputted value with a level of confidence. Something like this:

from fuzzywuzzy import process

ports = {"Houston", "New York", "Newark", "Long Beach"}
user_input = "Strong Beach"
best_guess = process.extractOne(user_input, ports)
print(f"The best match for '{user_input}' is '{best_guess[0]}' which is a {best_guess[1]}% match.")

  The best match for 'Strong Beach' is 'Long Beach' which is a 82% match.


Cluster Analysis - Comparing Same Individuals Clustered Across Different Datasets with different features I have an interesting problem, and I think my Google is failing me since I can't find the same problem anywhere.

I have a set of individuals. I have 4 different datasets, with (some) to (all) of those individuals in any given set.  So if I had 10 people, n=10, then maybe in DS1 I have all 10, DS2 has 7 of them, DS3 8, and DS4 5.

Each dataset DS# has it's own features.  So maybe DS1 has 100 features, DS2 has 10, DS3 has 30, and DS4 also has 30.

I want to cluster the individuals who exist in any given dataset.  So, if I did 3 clusters total, I'd have 3 clusters for DS1, 3 clusters for DS2, ... , 3 clusters for DS4.

I want a metric that captures how similar the results are.  I'm doing this part for exploratory purposes.  I'm also considering later incorporating this into a similarity metric for optimization purposes, but that's a further-out idea.

I however cannot find existing work that does something like this. Is there such work someone can point me to?

Much appreciated.

Literature

The closest line of work to your problem is Multi-View Clustering. Each data set DS is considered as a view, and views share a central entity (e.g. an individual).

2004
Multi-View Clustering is a survey on the topic. Also, here are two git projects from two papers (git 1, git 2) on the topic (they are implemented in Matlab and unfortunately are not production-ready).

A shortcut

Since most mainstream clustering methods (e.g. K-means, DBSCAN) accept a custom distance matrix $d(x, y)$ instead of explicit data vectors $x$ and $y$, partiality of features can be handled outside the method.

For example, suppose $x$ and $y$ are both present in views $1$ and $3$ but not view $2$. You may first build their vector by fusing their features from views $1$ and $3$, then computing their distance $d(x, y)$. Do the same for all pairs of individuals based on their common features, and finally feed the distance matrix $d(X, Y)$ to a clustering algorithm. This way, based on your domain knowledge, you can manually assign different weights to features of different views too. For example, you can define the distance as
$$d(x, y) := \left \| x[f_1:f_3] - y[f_1:f_3] \right \| + 2\left \| x[f_4:f_5] - y[f_4:f_5] \right \|$$
which implies that the difference in view $3$ with features $f_4$ and $f_5$ is twice as important as the difference in view $1$ with features $f_1$ to $f_3$.

Comparing two partitions

If views are clustered separately, two partitions $p_1$ and $p_2$ respectively
 over views $1$ and $2$ can be compared using metric Normalized Variation of Information (NVI). If partitions are the same, it outputs $0$, and if they are statistically independent, it outputs $1$. A similar measure (that is not a metric) is Normalized Mutual Information (NMI).

If all individuals are not present in both partitions, either (1) remove non-shared individuals from both partitions, or (2) individuals in $p_1$ that are not present in $p_2$ can be assigned to an imaginary cluster $misc_2$ in $p_2$, the same for those in $p_2$ that are not in $p_1$ using $misc_1$ in $p_1$.


Random Forests with complementary features In my dataset, I have 2 features that are not only correlated but that makes sense only in the presence of each other. For instance, one would be the number of times a task was attempted and the other one would be the number of successes.

As mentioned, it seems to be me that taken one of the 2 individually does not give any information. Should I do a scheme where if I pick one of them in a tree of my RF, I automatically include the other one?

And if so, is it possible to do so using the RF class from scikit-learn?

Thanks!

Notice that random forests (and decision trees in general) do not assume that the given features are independent. On the contrary, a typical classification path from the root to a leaf in one particular tree/classifier in the random forest would be e.g. to apply a different rule on the successes feature, based on the value of the attempts feature. So as one comment suggests, the algorithm will be able to identify certain dependencies.

However, you need to keep in mind that decision trees (and in certain sense random forests as a consequence) define only linear separations between classes. Thus, to enhance the domain space, you might want to try to "hint" to the algorithm some additional meta-features, and possible semantics among the features. For example, have you considered also introducing success ratio (successes divided by attempts) as an additional feature?

Notice that additional features are not guaranteed to help, even if the easiest way to know is to try them. The reason for this is that the algorithm might already be able to learn the additional semantics you give it. Having said that, to me it is not obvious that a random forest would be able to "learn" a feature like success ratio.


Pylearn2 vs TensorFlow I am about to dive into a long NN research project and wanted a push in the direction of Pylearn2 or TensorFlow?  As of Dec 2015 has the community started to lean one direction or another?

This link has given me concern about getting tied to TenserFlow.

You might want to take into consideration that Pylearn2 has no more developer, and now points to other Theano-based libraries:

  There are other machine learning frameworks built on top of Theano that could interest you, such as: Blocks, Keras and Lasagne.

As Dawny33 says, TensorFlow is just getting started, but it is interesting to note that the number of questions on TensorFlow (244) on Stack Overflow already surpasses Torch (166) and will probably catch up with Theano (672) in  2016.


What do neural networks learn first? I'm running some experiments with NNs (actually I'm running an LSTM classifier), and I stumbled across a question I haven't found the answer so far.

What do NNs learn first? When we train a network for classifying objects, for example, the accuracy usually starts low on the training set and increases throughout the epochs. My question is, in these early epochs, which examples of my training set the network is correctly classifying? Are they the "easiest" examples or simply the examples that come first? It seems to me that they are the easiest examples, i.e., those that are easily distinguishable from each other.

I'd be happy if there's a mathematical explanation for that. Any help is appreciated!

Potentially your looking for a description of the “vanishing or exploding gradient problem” in neural networks.

This post does a deep dive into this problem.  The author states the issue briefly as:

  an important observation: in at least some deep neural networks, the
  gradient tends to get smaller as we move backward through the hidden
  layers. This means that neurons in the earlier layers learn much more
  slowly than neurons in later layers.”

On a high level, we can look at the output of the neural network as a composition of functions $\hat{y} = f(g(h(x)))$.

After obtaining the loss, $y - \hat{y}$, we use gradient descent to begin to updating the weights.  The weights in the deepest layer depend only on the derivative of the deepest function in the earlier composition, while shallow layers’ weights depend on the outer functions in the composition.

The authors explain that this results in a derivative for the shallow layers that includes the multiplication of the deeper weights, which are usually < 1 (in the vanishing gradient problem).  Multiplying many numbers < 1 results in a very small number and a vanishing update to the weights in the shallow layers of the network.


Understanding python XGBoost model dump output of a very simple tree I am trying to understand the model dump output from XGBoost. I would like to step through and see exactly how the model arrived at it's prediction. To simplify I trained a model with 1 tree and 1 max depth, and as expected all records get one of two predictions as it has a single split - the values are {0.5386398434638977, 0.5011891722679138}. However, when I look at the model dump I see the following

booster[0]:
0:[f40<70.5] yes=1,no=2,missing=1
    1:leaf=0.00475667231
    2:leaf=0.154868156

I have no idea how to interpret this in a way that makes sense with the prediction. What am I missing? Thanks!

The scores at leaves measure log-odds, not probabilities.  (With more trees, these scores get summed to give a final log-odds approximation, then goes through a sigmoid to get probability approximations.)

And indeed, $1/(1+e^{-0.00475667})\approx 0.501189$.


Interpretation of variable or feature importance in Random Forest I'm currently using Random Forest to train some models and interpret the obtained results.

One of the features I want to analyze further, is variable importance. The thing is I am not familiar on how to do a proper analysis of the results I got.
Let's say I have this table:

| Predictor  | Importance |
----------------------------
|   var_1    |   num_1    |
...
|   var_n    |   num_n    |

What is a proper analysis that can be conducted on the values obtained from the table, in addition to saying which variable is more important than another?

I was suggested something like variable ranking or using cumulative density function, but I am not sure how to begin with that.

I would be reluctant to do too much analysis on the table alone as variable importances can be misleading, but there is something you can do.  The idea is to learn the statistical properties of the feature importances through simulation, and then determine how "significant" the observed importances are for each feature.  That is, could a large importance for a feature have arisen purely by chance, or is that feature legitimately predictive?

To do this you take the target of your algorithm $y$ and shuffle its values, so that there is no way to do genuine prediction and all of your features are effectively noise.  Then fit your chosen model $m$ times, observe the importances of your features for every iteration, and record the "null distribution" for each.  This is the distribution of the feature's importance when that feature has no predictive power.

Having obtained these distributions you can compare the importances that you actually observed without shuffling $y$ and start to make meaningful statements about which features are genuinely predictive and which are not.  That is, did the importance for a given feature fall into a large quantile (say the 99th percentile) of its null distribution?  In that case you can conclude that it contains genuine information about $y$.  If on the other hand the importance was somewhere in the middle of the distribution, then you can start to assume that the feature is not useful and perhaps start to do feature selection on these grounds.

Here is a simulation you can do in Python to try this idea out.  First we generate data under a linear regression model where only 3 of the 50 features are predictive, and then fit a random forest model to the data.  Now that we have our feature importances we fit 100 more models on permutations of $y$ and record the results.  Then all we have to do is compare the actual importances we saw to their null distributions using the helper function dist_func, which calculates what proportion of the null importances are less than the observed.  These numbers are essentially $p$-values in the classical statistical sense (only inverted so higher means better) and are much easier to interpret than the importance metrics reported by RandomForestRegressor.  Or, you can simply plot the null distributions and see where the actual importance values fall.  In this case it becomes very obvious that only the first three features matter where it may not have been by looking at the raw importances themselves.

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestRegressor

# number of samples
n = 100
# number of features
p = 50
# monte carlo sample size
m = 100

# simulate data under a linear regression model
# the first three coefficients are one and the rest zero
beta = np.ones(p)
beta[3:] = 0
X = pd.DataFrame(np.random.normal(size=(n, p)),
                 columns=["x" + str(i+1) for i in range(p)])
y = np.dot(X, beta) + np.random.randn(n)

# fit a random forest regression to the data
reg = RandomForestRegressor()
reg.fit(X, y)

# get the importances
var_imp = (pd.DataFrame({"feature": X.columns,
                        "beta": beta,
                       "importance": reg.feature_importances_}).
           sort_values(by="importance", ascending=False).
           reset_index(drop=True))

# fit many regressions on shuffled versions of y
sim_imp = pd.DataFrame({c: np.empty(m) for c in X.columns})

for i in range(m):
    reg.fit(X, np.random.permutation(y))
    sim_imp.iloc[i] = reg.feature_importances_

# null distribution function
def dist_func(var, x):
    return np.mean(sim_imp[var] < x)


User-product positive (click data) available. How to generate negative (no-click data)? Its very common in recommender that we have user product data which have label as an e.g. "click". In order to learn the model, I need click and no-click data.

Simplest approach to generate is to take user-products pairs which are not found in click data. However, that may be misleading. Example:

user1, product1 (click)
 user2, product2 (click)
 user2, product3 (click)
 user3, product2 (click)

I can take user1 with all products except product1 and label them as "no_click" and so on. But this may not be true. Maybe user1 would have clicked product2 if he was shown product2. But just because he was shown other sets of products - he had no opportunity to decide to click/no-click product2.

So how to address unary data problem?

So, there are two problems.

Recording impressions (shows)
How to deal with non-impressions

For (1) you should be recording this information. If it is not currently being recorded you should start recording this information. Given that you do not have this information you want to provide recommendations. Fortunately, with just click data you can still create a utility matrix, see 9.1.1.
You could then use user- or item-based collaborative filtering as described in the paper. This is basically an exercise in populating the utility matrix and trying to find "scores" for unclicked items. Your recommendation would be an unclicked item with the highest score.
For (2) you will still make recommendations on unclicked items. So, that alone is not an issue. You will want to optimize your impressions however. You also cannot have full knowledge where a user can see all possible options. You need to record impressions and understand a number of things.

show rate of an item
click rate of an item
how to incorporate new items
how to optimize which items to show

This is a huge topic and basically this is the problem domain of online advertising. However, a recommendation engine tries to find items of interest in the long tail, which is a bit different from ad optimization. This is a feedback loop to evaluate your recommendation. A/B tests are common. You will want to test click rates and recommendation errors between your current system and new system.
Also see here and here.


Opensource Speech Recognition Library that is secure and trained on large data For all those who are working on developing a chatbot/assistant and care about the privacy of users consuming the speech recognition library, can you suggest an open souce library which is trained on a large data. Big concern is the privacy that's why not going for Google Diagflow or IBM Watson or Amazon Lexa or Wit.

Would appreciate a lot if someone can suggest a good library.

I'd recommend having a look at DeepSpeech.

You can find a maintained implementation of the project in here.

You may clone it, and train a model in order to have your very own model for speech recognition.


How can one can aim for a career in machine learning? I'm interested in various machine learning methods and wonder how can I aim for a career in the field.
What are most basic skills to have?

I mean the matters strictly connected to the machine learning - not programming or libraries.

Machine Learning is a really big field. Depending on what exactly you want to do, there might be huge differences.

Having said that, the following skills are helpful:

Programming, especially Python / C++
Frameworks like TensorFlow, sklearn, Torch, ...
Algorithms like neural networks (especially gradient descent), SVM, decision trees, clustering algorithms, Q-Learning, ...
Mathematics (e.g. probability theory / statistics, linear algebra, analysis for calculating gradients

Knowing how to program the GPU directly (CUDA) is probably not always necessary, but certainly a big bonus.

In case you're interested what you learn at KIT (Karlsruhe, Germany) in an introductory machine learning course, see my blog post for exam preparation (I'm sorry, it is mostly in German).

Online Courses

There are lots of online material for machine learning. For example an Coursera ML course


Formatting multi-level variable width data for model I would like to calculate a value based on the number of lines on outage at a given time.

An outage consists of the two endpoints and the voltage of the line:

outage = ["node1", "node2", "120"]

The historical values I have are based on per hour time stamps, so
the format I believe I will need to feed the model is like so:

[timestamp, [outage1, outage2, etc...], value]

Do most machine learning libraries accept data in a variable width format like this? Alternatively, the only other thing I can think of is to have a super-wide data set and do a one hot encoding where each column is a node and its value is the voltage (scaled perhaps).

Example:

  Columns:
    [timestamp, node1, node2, node3, etc..., value]
  Values:
    ["1/1/2019 14:00", 120, 120, 0, etc..., 5]

There are thousands of nodes on this grid. What is the best approach for formatting this data in a way that most models will accept?

Looking back at this, I've ended up finding an answer to my own question.

What I am trying to do here is a pivot, which converts categorical data into multiple columns, and then sets the value to that of another column.

Additionally, I've found that it is very common for models to have thousands of columns, if not more.


Are the raw probabilities obtained from XGBoost, representative of the true underlying probabilties? 1) Is it feasible to use the raw probabilities obtained from XGBoost, e.g. probabilities obtained within the range of 0.4-0.5, as a true representation of approximately 40%-50% chance of an event occurring? (assuming we have an accurate model)

2) The same question as in 1), but for other models that can output raw probabilities.

It depends on the definition of accurate model, but in general the answer to your question 1) is No.

Regarding your second question (based on results in the paper of Niculescu-Mizil & Caruana linked below):

boosted trees and stumps - NO
Naive Bayes - NO
SVM - NO
bagged trees - YES
neural nets - YES

You can test whether it is the case for your particular model and dataset by looking at the so called reliability plot:

Create N bins based on the model output (e.g. 10-20)
Create a scatter plot with average model output for each bin along X axis and average true probability for each bin along Y axis

Ideally, your X-Y points should lie near the diagonal Y=X, otherwise the output of your classifier cannot be interpreted as a probability of an event.

However, not all is lost and if needed, one can try to modify (calibrate) the output of the model in a such way that it better reflects the observed probability. In order to assess whether the calibration exercise was successful one might look at the reliability plot based on the calibrated model output (instead of using raw model output).

Two most widely used techniques for the calibration of classifier outputs are Platt scaling and isotonic regression, see the links below.

Note, that it is not advisable to cailbrate the classifier using the training dataset (you might need to reserve a separate subset of your data for the purpose of calibration).

Some relevant links

Predicting Good Probabilities With Supervised Learning

CALIBRATING CLASSIFIER PROBABILTIES

Classifier calibration with Platt's scaling and isotonic regression


Text categorization: combining different kind of features The problem I am tackling is categorizing short texts into multiple classes. My current approach is to use tf-idf weighted term frequencies and learn a simple linear classifier (logistic regression). This works reasonably well (around 90% macro F-1 on test set, nearly 100% on training set). A big problem are unseen words/n-grams.

I am trying to improve the classifier by adding other features, e.g. a fixed sized vector computed using distributional similarities (as computed by word2vec) or other categorical features of the examples. My idea was to just add the features to the sparse input features from the bag of words. However, this results in worse performance on the test and training set. The additional features by themselves give about 80% F-1 on the test set, so they aren't garbage. Scaling the features didn't help as well. My current thinking is that these kind of features don't mix well with the (sparse) bag of words features.

So the question is: assuming the additional features provide additional information, what is the best way to incorporate them? Could training separate classifiers and combining them in some kind of ensemble work (this would probably have the drawback that no interaction between the features of the different classifiers could be captured)? Are there other more complex models I should consider?

If I understand correctly, you essentially have two forms of features for your models. (1) Text data that you have represented as a sparse bag of words and (2) more traditional dense features. If that is the case then there are 3 common approaches:

Perform dimensionality reduction (such as LSA via TruncatedSVD) on your sparse data to make it dense and combine the features into a single dense matrix to train your model(s).
Add your few dense features to your sparse matrix using something like scipy's hstack into a single sparse matrix to train your model(s).
Create a model using only your sparse text data and then combine its predictions (probabilities if it's classification) as a dense feature with your other dense features to create a model (ie: ensembling via stacking). If you go this route remember to only use CV predictions as features to train your model otherwise you'll likely overfit quite badly (you can make a quite class to do this all within a single Pipeline if desired).

All three approaches are valid and have their own pros and cons. Personally, I find (1) to typically be the worst because it is, relatively speaking, extremely slow. I also find (3) to usually be the best, being both sufficiently fast and resulting in very good predictions. You can obviously do a combination of them as well if you're willing to do some more extensive ensembling.

As for the algorithms you use, they can essentially all fit within that framework. Logistic regression performs surprisingly well most of the time, but others may do better depending on the problem at hand and how well you tune them. I'm partial to GBMs myself, but the bottom line is that you can try as many algorithms as you would like and even doing simple weighted ensembles of their predictions will almost always lead to a better overall solution.


Why must a CNN have a fixed input size? Right now I'm studying Convolutional Neural Networks.
Why must a CNN have a fixed input size?
I know that it is possible to overcome this problem (with fully convolutional neural networks etc...), and I also know that it is due to the fully connected layers placed at the end of the network.
But why? I can not understand what the presence of the fully connected layers implies and why we are forced to have a fixed input size.

I think the answer to this question is weight sharing in convolutional layers, which you don't have in fully-connected ones. In convolutional layers you only train the kernel, which is then convolved with the input of that layer. If you make the input larger, you still would use the same kernel, only the size of the output would also increase accordingly. The same is true for pooling layers.

So, for convolutional layers the number of trainable weights is (mostly) independent of input and output size, but output size is determined by input size and vice versa.

In fully-connected layers you train weight to connect every dimension of the input with every dimension of the output, so if you made the input larger, you would require more weights. But you cannot just make up new weights, they would need to be trained.

So, for fully-connected layers the weight matrix determines both input and output size.

Since CNN often have one or more fully-connected layers in the end, there is a constraint on what the input dimension to the fully-connected layers has to be, which in turn determines the input size of the highest convolutional layer, which in turn determines the input size of the second highest convolutional layer and so on and so on, until you reach the input layer.


Accuracy graph of binary classification by CNN Why in binary classification of images with CNN the loss and accuracy graph are so unstable? I mean accuracy of validation test does not increase smoothly, it goes to 80%, then comes to 60%, then again goes to 84% and so on. Same is the case with train accuracy. Now how do I know that how many epochs is the optimal number?

There is no way to definitely note how many epochs is required. Your accuracy graphs may be unstable for several reasons such as -

Irregular data distribution
Faulty model
Large batch size (as mentioned in the comments)
Overfitting and underfitting (in this case im not too sure which one)

You can fix it through the following methods -

Analyse your data well and see if one class is heavily grater than other.
Make sure all duplicate images are removed.
Try different models, start with pre-trained. Shouldn't take much compute power.
Approach transfer learning (neural networks)
Adjust learning rate (as mentioned above)
General hyperparameter tuning.

if the dataset is from somewhere public, see the models and the approaches others have used. You might see where your model actually lacks.


Should a beginner in ML use libraries like TF and Keras? As a beginner in ML field, should I in beginning implement various algorithms from scratch (using only NumPy for example) and then start using ML libraries like TensorFlow, Keras or vice-versa? For example, I implemented multilayer feed-forward neural network in Python using only NumPy but it took me few weeks to understand NNs at the level that I can implement them from scratch, would it be better that I spent that time to cover theory less thoroughly and use libraries to make more advanced models easily?

Start of by reading up on math/theory. For neural networks I can really recommend Michael Jensens free online book. All along he gives good python examples using only numpy.
For DNNs, TF and Keras are IMO the easiest approach. For linear models, scikit learn would probably cover most.


How to do postal addresses fuzzy matching? I would like to know how to match postal addresses when their format differ or when one of them is mispelled.

So far I've found different solutions but I think that they are quite old and not very efficient. I'm sure some better methods exist, so if you have references for me to read, I'm sure it is a subject that may interest several persons.

The solution I found (examples are in R) :

Levenshtein distance, which equals the number of characters you have to insert, delete or change to transform one word into another.

agrep("acusait", c("accusait", "abusait"), max = 2, value = TRUE)
## [1] "accusait" "abusait"
The comparison of phonemes

library(RecordLinkage)
soundex(x<-c('accusait','acusait','abusait'))
## [1] "A223" "A223" "A123"
The use of a spelling corrector (eventually a bayesian one like Peter Norvig's), but not very efficient on address I guess.
I thought about using the suggestions of Google suggest, but likewise, it is not very efficient on personal postal addresses.
You can imagine using a machine learning supervised approach but you need to have stored the mispelled requests of users to do so which is not an option for me.

As you are using R you might want to look into the stringdist package and the Jaro-Winkler distance metric that can be used in the calculations. This was developed at the U.S. Census Bureau for linking .

See for more information on the Jaro and Jaro-Winkler distance in this journal.

For a comparison of different matching techniques, read this paper


How to predict next visit date based on this data I have a dataset shown below. Here, status is if visit has been done or not and schedule is if next_action_scheduled.

df =

visit_date |status |scheduled_visit

2018-09-13|1|0000-00-00

2018-11-13|1|0000-00-00

2019-01-13|1|0000-00-00

2019-03-13|0|0000-00-00

2019-05-13|0|0000-00-00

So consider this case like user is visiting in every 2 month.

I would like to find this pattern for all users based on the historical visits with the customer. I would like to find the next predicted visit for that customer and if we missed that visit, then I ask "Do you want to schedule a visit" Or "You missed a visit with this customer."

I'm new in Ml. I tried to use RNN for time series forecasting but I'm getting a really high error rate. Prediciton is nowhere close. What model/algorithm I should use and learn about to make a model for this? I want to create a solution like if I feed in the user_id it will check all these values and send notification within app.

Thank you

If you plot days since the first visit vs the visit number, this ends up being a simple regression problem.

import pandas as pd

data = pd.read_csv('data.txt', '|')
data['visit_date'] = pd.to_datetime(data['visit_date'])
data['days_since_first_visit'] = (data['visit_date'] - data['visit_date'][0]).map(lambda delta: delta.days)
data = data.reset_index()

data.plot('index', 'days_since_first_visit')

I would recommend using a library like scikit-learn or statsmodels to do the regression.


training when Multiple labels per image I have multiple labels per image. is it better to train taking each each label separately or should i mark all the labels present as 1 in the same image? which method is better? i will be using CNN architecture

Assuming you want to classify the images (and not use bounding boxes to locate classes within each image), a common way it to create a target vector for each image, which holds the information regarding all classes and is what the model would eventually predict.

If you have a dataset with, say 5 classes, and your first example image contains classes 1 and 4, you would create your target vector for that image to be:

example_sample = ... # your image array
example_sample_y = [1, 0, 0, 1, 0]

This is a kind of one-hot encoding, as the vector has a placeholder for each of the 5 classes, but only a 1 when the class is present.

Have a look at this high-level walkthrough.

I think your other suggestion of training an image

You want to learn some kind of joint probability between the classes, and in my opinion, training one the same image with different outcomes (e.g. the sample image above twice, producing either a 1 or a 4) will not only be very inefficient during training, but will also be mathematically confusing. The same input can give 2 possible outputs! This implies your underlying function that maps images to classes is not well-defined. That isn't usually a good thing!


ML model deployment architecture? I came from a software development background and we have separate servers of the same database (dev, test, prod). The reason for this is because we develop our apps against the dev DB, run tests against the Test DB, and prod is prod. This is so we create a clear separation and won't bring down prod trying to build our app.

Do you guys train your models the same way? Have 3 environments of the same database and as your model goes from dev to test to prod, it trains against the corresponding environment?

Example:

Data scientist plays around with a 3 different algos for classification. Creates 3 models (A, B, C) using dev env's database.
Data scientist evaluates 3 models and selects Model A after testing/validation.
Data scientist deploys the code to TESTING/STAGING env (same hyperparameters). This time, however, the model trains using TESTING/STAGING env's databases. A version of Model A is created using data from the TESTING/STAGING databases.
Data scientist deploys the code to PROD env (same hyperparameters). This time, however, the model trains using PROD env's databases. A version of Model A is created using data from the PROD databases.

No, you don't train against 3 separate databases, because for each dataset, you would need to do the whole work from scratch ( exploratory data analysis , feature engineering etc..). You need a valid training dataset, from which you create your validation dataset and test dataset ( a hold-out set as a final verification step ). Yeah sure, you deploy your model for example as a webservice on a test server to make sure is working as expected. But model verification, including avoiding overfitting or underfitting, having a good accuracy and all that is done prior to any deployment.


Adaboost vs Gradient Boosting How is AdaBoost different from a Gradient Boosting algorithm since both of them use a Boosting technique?
I could not figure out actual difference between these both algorithms from a theory point of view.

Both AdaBoost and Gradient Boosting build weak learners in a sequential fashion.
Originally, AdaBoost was designed in such a way that at every step the sample distribution was adapted to put more weight on misclassified samples and less weight on correctly classified samples. The final prediction is a weighted average of all the weak learners, where more weight is placed on stronger learners.
Later, it was discovered that AdaBoost can also be expressed in terms of the more general framework of additive models with a particular loss function (the exponential loss). See e.g. Chapter 10 in (Hastie) ESL.
Additive modeling tries to solve the following problem for a given loss function $L$:
$ \min_{\alpha_{n=1:N},\beta_{n=1:N}} L\left(y, \sum_{n=1}^N \alpha_n f(x,\beta_n) \right)$
where $f$ could be decision tree stumps. Since the sum inside the loss function makes life difficult, the expression can be approximated in a linear fashion, effectively allowing to move the sum in front of the loss function iteratively minimizing one subproblem at a time:
$ \min_{\alpha_n,\beta_n} L\left(y, f_{n-1}((x) + \alpha_n f_n(x,\beta_n) \right)$
For arbitrary loss functions this is still a tricky problem, so we can further approximate this by applying a steepest descent with line search, i.e. we update $f_n$ by taking a step into the direction of the negative gradient.
In order to avoid overfitting on the gradient, the gradient is approximated with a new weak learner. This gives you the gradient boosting algorithm:

Start with a constant model $f_0$
Fit a weak learner $h_n$ to the negative gradient of the loss function w.r.t. $f_{n-1}$
Take a step $\gamma$ so that $f_n= f_{n-1} + \gamma h_n$  minimizes the loss $L\left(y, f_n(x) \right)$

The main differences, therefore, are that Gradient Boosting is a generic algorithm to find approximate solutions to the additive modeling problem, while AdaBoost can be seen as a special case with a particular loss function. Hence, Gradient Boosting is much more flexible.
On the other hand, AdaBoost can be interpreted from a much more intuitive perspective and can be implemented without the reference to gradients by reweighting the training samples based on classifications from previous learners.
See also this question for some further references (quote):

In Gradient Boosting, ‘shortcomings’ (of existing weak learners) are identified by gradients.
In Adaboost, ‘shortcomings’ are identified by high-weight data points.


How do i use the Gaussian function with a Naive Bayes Classifier? I would like some help with the maths and to check I have understood the algorithm correctly. So I’ve been learning off of a video and I tried my own example. At the end of it I got dodgy results; I think they are “dodgy” because my outputs are not between 0 and 1. If I went with the higher output then yes it has correctly classified my new data but I’m just not sure since the numbers are completely different. I tried by doing the math by hand with a calculator instead of with python but I still got a weird result, so it makes me think that my method is wrong.Here’s what I did, please have a look and tell me what you think:

# data

x = [[65,50],

[62,49],

[130,78],

[124,79],

[54,47],            # the heights/weights of a child

[169,76],              (0), or an adult (1)

[59,58],

[169,79],

[52,52]]

y = [0,0,1,1,0,1,0,1,0]

new = [58, 54]

Okay so here’s what I think I should do:

Work out the prior probability (in this case 5/9 children, 4/9 adults)
Work out the means and variances for each feature and each class:

x̄(C,H) = 58.4        x̄(C,W) = 51.2

x̄(A,H) = 148          x̄(A,W) = 78

σ²(C,H) = 23.44    σ²(C,W) = 14.16

σ²(A,H) = 445.5    σ²(A,W) = 1.5

Compute the Gaussians f(x,x̄, σ²):

f(58, 58.4, 23.44) = 0.08212001659 (child height)

f(54, 51.2, 14.16) = 0.08038037412 (child weight)

f(58, 148, 445.5) = 2.129877168 x10(-6) (adult height)

f(54, 78, 1.5) = 1.343766812 x10(-84) (adult weight)
Multiply the Gaussians for the adult and child classes:

0.08212001659 x 0.08038037412 = 6.600837656 x10(-3) (child Gauss)
2.129877168 x10(-6) x 1.343766812 x10(-84) = 2.862058252 x10(-90) (adult Gauss)

Calculate the probability using bayes theorem:

(child gauss x child prob)/(child gauss x child prob x adult gauss x adult prob)
(adult gauss x adult prob)/(child gauss x child prob x adult gauss x adult prob)

(6.600837656 x10(-3) x 5/9)/(6.600837656 x10(-3) x 5/9 x 2.862058252 x10(-90) x 4/9) = 9.841633571 x10(89)
(2.862058252 x10(-90) x 4/9)/(6.600837656 x10(-3) x 5/9 x 2.862058252 x10(-90) x 4/9) = 341.3788376

...so those were the final probabilities. If i went with the higher one which you are supposed to then the algorithm would've correctly classed the data as a child, but the probabilities were not between 0 and 1 which makes me think this method may not work for other data. I also ran the data through Scikit learn which classified the new data as a child also. Please let me know what you think and if i have been unclear about something, let me know in the comments.

You got the Bayes theorem wrong. There should be a plus in the denominator:

(child gauss x child prob)/(child gauss x child prob + adult gauss x adult prob)

(adult gauss x adult prob)/(child gauss x child prob + adult gauss x adult prob)


How to animate growth of a social network? I am seeking for a library/tool to visualize how social network changes when new nodes/edges are added to it.

One of the existing solutions is SoNIA: Social Network Image Animator. It let's you make movies like this one.

SoNIA's documentation says that it's broken at the moment, and besides this I would prefer JavaScript-based solution instead. So, my question is: are you familiar with any tools or are you able to point me to some libraries which would make this task as easy as possible?

Right after posting this question I'll dig into sigma.js, so please consider this library covered.

In general, my input data would be something like this:

time_elapsed; node1; node2
1; A; B
2; A; C
3; B; C

So, here we have three points in time (1, 2, 3), three nodes (A, B, C), and three edges, which represent a triadic closure between the three considered nodes.

Moreover, every node will have two attributes (age and gender), so I would like to be able to change the shape/colour of the nodes.

Also, after adding a new node, it would be perfect to have some ForceAtlas2 or similar algorithm to adjust the layout of the graph.

Fancy animations are cool
I was very impressed when I saw this animation of the discourse git repository. They used Gourse which is specifically for git. But it may give ideas about how to represent the dynamics of growth.
You can create animations with matplotlib
This stackoverflow answer seems to point at a python/networkx/matplotlib solution.
But D3.js provides interaction
If you're looking for a web-based solution then d3.js is excellent. See this, this and this for example.
See also this stackoverflow question, the accepted answer points to D3.js again.
Conclusion
I would be drawn towards the python/networkx options for network analysis (possibly to add attributes to your raw data file for example). Then, for visualisation and dissemination D3.js is perfect. You might be surprised how easy it can be to write d3.js once you get into it. I believe it even works within an ipython notebook!


How are Lagrangian multipliers zero except for support vectors in dual representation of SVM? How can we conclude that the Lagrangian multipliers are zero, except support vectors, in a dual problem? I cannot seem to see it.

$$L(\alpha)=-\frac{1}{2}\sum_i \sum_j \alpha_i \alpha_j y_i y_j x_i' x_j + \sum_i{\alpha_i}
$$

In optimization, we have something called complementary slackness condition, it is part of the KKT conditions.

Every constraint, $g_i(x^*)\le 0$ in the primal corresponds to a dual variable $\mu_i$ (Lagrange multiplier).  The condition state that $$g_i(x^*)\mu_i=0$$

For points that are not support vectors, we have $g_i(x^*)<0$, hence we must have $\mu_i=0$.


What algorithms should I choose for a recommender system and why? To my knowledge, recommender systems are broadly classified into collaborative and content. Collaborative in turn is divided into 1) Memory (uses similarity metrics) and 2) Model (well known Matrix/Tensor factorization). Content based involves constructing a user profile and then an algorithm like SVM to classify and recommend items. Now here are my questions:

What other algorithms can I use for recommendation and why?
Can I use neural networks? (understanding them has been a bit difficult for me)
Is it true that neural networks (NN) are only suited for text and image processing and numerical data doesn't need complex algorithms like NN?

Short answer:

It depends on your data. What do you want to do ?

Longer answer:

Use content based approaches if you have data on your items. Use collaborative approaches is you have data on your users. Use both if you have both. I would say content based approaches are general machine learning problems (how do I extract meaningful information from data) whereas collaborative filtering is really a recommender system specific work (how user's behavior can suggest users/items similarity/connections).
Well you can. Neural nets are just a kind of algorithm, you surely can use them for content based analysis, and it might be possible to use them to enhance your collaborative algorithm.
NN use texts and images as numerical data, so I don't understand your question.

If you want a good insight in today's recommender systems, take a look at this article.


How can I get semantic word embneddings for compound terms? I need to build semantic word embeddings representation of compound terms like "electronic engineer" or "microsoft excel". One approach would be to use a standard pretrained model an average the words but, since I have a corpus  of my domain, is there a possible better approach?

To be more precise:

The data I have is a corpus of millions of documents. Each document is ~ half a page and contains these compound terms. However there may be compound terms not included in the corpus.

Thanks

If you want an exact answer, please provide a precise question i.e. define what data you have, and what you exactly wants.

This said, in a general manner, you need a dataset of texts that contain these compound terms. How to treat compound terms is a whole scientific field in itself, but since you're talking about semantic word embeddings, I suggest you take a look at the article Distributed Representations of Words and Phrases
and their Compositionality. The same guys who introduced word2vec describe here a simple method to go from word representation to phrase representation, giving btw a way to merge compound terms in single terms. The words "microsoft excel" become "microsoft_excel" and get their own unique embedding.

If you want a python implementation for that, take a look at the gensim.models.phrase class. This does the same work as presented in the previous article.


How is the "loss" calculated which is supplied by the callback log in Keras? I.e. categorical cross entropy? binary cross entropy? Something else?

Or is it perhaps the loss function which you pass into the model.compile method?

Yes, it is the loss function you pass to model.compile.

See here for loss metric usage examples. You can also define your own metric (see "Custom metrics" at the bottom of the page from the last link).

You can read about the Model class documentation here. The first method is the compile method with argument descriptions.

If you want to dig deeper into Keras callbacks and the built-in loss functions, check out these sources:

losses source code

Keras callbacks


Generating Similar Words (or Synonyms) with Word Embeddings (Word2Vec) We have a search engine, and when users type in Tacos, we also want to search for similar words, such as Chilis or Burritos.

However, it is also possible that the user search with multiple keywords. Such as Tacos Mexican Restaurants, and we also want to find similar word such as Chilis or Burritos.

What we do is to add all the vectors together for each word. This sometimes works, but with more keywords the vectors tend to be in a place where there are no neighbors.

Is there an approach where we can use not only one word, but multiple word, and still gives us similar results? We are using pre-trained glove vectors from Stanford, would it help if we train on articles that are food related, and specifically use that type of word embeddings for this task?

Word Mover’s Distance (WMD) is an algorithm for finding the minimum distance between multiple embedded words.

  The WMD distance measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to "travel" to reach the embedded words of another document.

For example:

Source: "From Word Embeddings To Document Distances" Paper

In your problem, it will allow you to find that "Tacos Mexican Restaurants" is similar to "Burritos Taqueria" even though they share no common string literals.