Feeding XLM-R embeddings to neural machine translation? I’m very new to the field of deep learning. My aim is to make a translation between Catalan to Catalan Sign Language. The grammar of the two languages is different

  Input: He sells food.
  Output (sign language sentence): Food he sells.

I've been playing around with XLM-R and go the token id like this

input Ids: [200, 100, 2003, 1037, 3835, 3351, 5012, 300, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

I don't know how to use the embeddings in Sequence to Sequence NMT model. or any other means to do machine translation with a very small data set. The language is low resource language

import torch
from transformers import XLMRobertaModel, XLMRobertaTokenizer

tokenizer = XLMRobertaTokenizer.from_pretrained('xlm-roberta-large')
model = XLMRobertaModel.from_pretrained('xlm-roberta-large')

def get_ids(tokens, tokenizer, max_seq_length):
token_ids = tokenizer.convert_tokens_to_ids(tokens)
input_ids = token_ids + [0] * (max_seq_length - len(token_ids))
return input_ids

s = "test sentence"
stokens = tokenizer.tokenize(s)
print(stokens)
stokens = ["[CLS]"] + stokens + ["[SEP]"]
input_ids = get_ids(stokens, tokenizer, 15)

print(tokenizer.convert_tokens_to_ids(['test']))
print(tokenizer.convert_tokens_to_ids(['▁test']))
print(tokenizer.convert_ids_to_tokens([26130]))
print(tokenizer.convert_ids_to_tokens([30521]))
tokens_tensor = torch.tensor([input_ids])
print(input_ids)
print(tokens_tensor)
```

You only got the indices in the XLM-R's vocabulary. This is the input to XLM-R, you need to actually run the model. By calling

model(tokens_tensor)

you get a tuple of tensors that are outputs of the model. Check the documentation for the outputs are.


Does nearest neighbour make any sense with t-SNE? Answers on here have stated that the dimensions in t-SNE are meaningless, and that the distances between points are not a measure of similarity.

However, can we say anything about a point based on it's nearest neighbours in t-SNE space? This answer to why points that are exactly the same are not clustered suggests the ratio of distances between points is similar between lower and higher dimensional representations.

For example, the image below shows t-SNE on one of my datasets (15 classes).

Can I say that cro 479 (top right) is an outlier? Is fra 1353 (bottom left) is more similar to cir 375 than the other images in the fra class, etc? Or could these just be artefacts, e.g. fra 1353 got stuck on the other side of a few clusters and couldn't force its way through to the other fra class?

No, it is not necessary that this is the case, however, this is, in a convoluted way, the goal of T-SNE.

Before getting into the meat of the answer, let's take a look at some basic definitions, both mathematically and intuitively.

Nearest Neighbors: Consider a metric space $\mathbb{R}^d$ and a set of vectors $X_1, ..., X_n \in \mathbb{R}^d$, given a new vector $x \in \mathbb{R}^d$, we want to find the points such that $|| X_1 - x || \le ... \le ||X_n - x ||$. Intuitively, it's just the minimum of the distances using a suitable definition of norm in $\mathbb{R}^d$.

Now coming to whether the nearest neighbors actually matter while applying dimensionality reduction. Usually in my answers, I intend to rationalize something with mathematics, code and intuition. Let us first consider the intuitive aspect of things. If you have a point that is a distance $d$ away from another point, from our understanding of the t-sne algorithm we know that this distance is preserved as we transition into higher dimensions. Let us further assume that a point $y$ is the nearest neighbor of $x$ in some dimension $d$. By definition, there is a relationship between the distance in $d$ and $d + k$. So, we have our intuition which is that the distance is maintained across different dimensions, or at least, that is what we aim for. Let's try to justify it with some mathematics.

In this answer I talk about the math involved in t-sne, albeit not in detail (t-SNE: Why equal data values are visually not close?). What the math here is, is basically maximizing the probability that two points remain close in a projected space as they are in the original space assuming that the distribution of the points is exponential. So, looking at this equation $p_{j | i} = \frac{exp(\frac{-||x_j - x_i||^2}{2\sigma^2})}{\sum_{k \neq i}{exp(\frac{-||x_j - x_i||^2}{2\sigma^2})}}$. Notice that the probability is dependent on the distance between the two points, so the further apart they are, the further apart they get as they get projected to lower dimensions. Notice that if they are far apart in $\mathbb{R}^k$, there is a good chance they will not be close in the projected dimension. So now, we have a mathematical justification as to why the points "should" remain close. But again, since this is an exponential distribution, if these points are significantly far apart, there is no guarantee that the Nearest Neighbors property is maintained, although, this is the aim.

Now finally a neat coding example which demonstrates this concept too.

from sklearn.manifold import TSNE
from sklearn.neighbors import KNeighborsClassifier
X = [[0],[1],[2],[3],[4],[5],[6],[7],[8],[9]]
y = [0,1,2,3,4,5,6,7,8,9]
neighs = KNeighborsClassifier(n_neighbors=3)
neighs.fit(X, y)
X_embedded = TSNE(n_components=1).fit_transform(X)
neighs_tsne = KNeighborsClassifier(n_neighbors=3)
neighs_tsne.fit(X_embedded, y)
print(neighs.predict([[1.1]]))
>>>[0]
print(neighs_tsne.predict([[1.1]]))
>>>[0]

Although this is a very naive example and doesn't reflect the complexity, it does work by experiment for some simple examples.

EDIT: Also, adding some points with respect to the question itself, so it is not necessary that this is the case, it might be, however, rationalizing it through mathematics would prove that you have no concrete outcome (no definitive yes or no).

I hope this cleared up some of your concerns with TSNE.


How Linear SVM Regression and Multiple Linear Regression different in terms of the regression result? They starts from the same equation as below.

y = w*x + b

But they solve it differently. MLR specified the w and b by minimizing the square error whereas SVM specified w and b by minimizing the loss function defined by C and epsilon.

I am wondering if the result of regression is significantly different. I guess that if the given data set is clean and well-explained by input features, the resultant w and b between SVM and MLR will be close. Putting my original question differently, I don't find any reasons to use linear SVM regression over multiple linear regression.

They could potentially be very different, because linear regression penalizes using squared error while SVRs penalize absolute error. You might have learned that "outliers" can have an outsized influence on the regression line because of this--this won't be the case with SVR, however.

Furthermore, if the SVR you're talking about is like the one described here (great resource there, by the way) then choosing a larger $\epsilon$ will yield a flatter regression line/hyperplane, whereas you don't have such a tuning parameter for regular ol' linear regression. This parameter enables you to try to fight overfitting by flattening the regression line.


Saving and loading keras.callbacks.History object with np.save and np.load I have been saving my training history in keras as follows:

history = model.fit(X_train, Y_train, epochs=700, batch_size=128,validation_data=(X_cv, Y_cv))

np.save('./history_sim#', history)

I am then trying to load the training history from the various simulations in order to print figures like loss vs. epoch, etc. as follows:

history = np.load('history_sim#.npy')

When I try to load the training history, I am receiving the following error message:

"ValueError: Unknown metric function:precision"

I am very worried that I have lost all of the training history now. Training takes several days and I am on a bit of a time crunch. Is the data lost or is there some way to get the data from the .npy file where I have saved it?

I already figured out that it works perfectly if I save/load as follows:

np.save('./history_sim#', history.history)
np.load('history_sim#').item()

I will do that for the rest of the simulations, but there are already a few that were saved the first way and I do not have time to re-run them.

Thanks!

Additional Info:

import keras_metrics

# Calculate precision for the second label.
precision = keras_metrics.precision(label=1)

# Calculate recall for the first label.
recall = keras_metrics.recall(label=0)

opt = Adam(lr=0.05, beta_1=0.9, beta_2=0.999, decay=0.0)
model.compile(loss='binary_crossentropy', optimizer=opt, metrics=[precision, recall])

use allow_pickle=True in np.load

>>>np.save("a.npy",his)
>>>item = np.load("a.npy",allow_pickle=True).item()


What is "experience replay" and what are its benefits? I've been reading Google's DeepMind Atari paper and I'm trying to understand the concept of "experience replay". Experience replay comes up in a lot of other reinforcement learning papers (particularly, the AlphaGo paper), so I want to understand how it works. Below are some excerpts.

  First, we used a biologically inspired mechanism termed experience replay that randomizes over the data, thereby removing correlations in the observation sequence and smoothing over changes in the data distribution.

The paper then elaborates as follows:

  While other stable methods exist for training neural networks in the reinforcement learning setting, such as neural fitted Q-iteration, these
  methods involve the repeated training of networks de novo hundreds
  of iterations. Consequently, these methods, unlike our algorithm, are
  too inefficient to be used successfully with large neural networks. We
  parameterize an approximate value function $Q(s, a; \theta_i)$ using the deep
  convolutional neural network shown in Fig. 1, in which $\theta_i$ are the parameters (that is, weights) of the Q-network at iteration $i$. To perform
  experience replay, we store the agent's experiences $e_t = (s_t, a_t, r_t, s_{t+1})$ at each time-step $t$ in a data set $D_t = \{e_1, \dots, e_t \}$. During learning, we apply Q-learning updates, on samples (or mini-batches) of experience $(s, a, r, s') \sim U(D)$, drawn uniformly at random from the pool of stored samples. The Q-learning update at iteration $i$ uses the following loss
  function:

  $$
L_i(\theta_i) = \mathbb{E}_{(s, a, r, s') \sim U(D)} \left[ \left(r + \gamma \max_{a'} Q(s', a'; \theta_i^-) - Q(s, a; \theta_i)\right)^2 \right]
$$

What is experience replay, and what are its benefits, in laymen's terms?

The key part of the quoted text is:

  To perform experience replay we store the agent's experiences $e_t = (s_t,a_t,r_t,s_{t+1})$

This means instead of running Q-learning on state/action pairs as they occur during simulation or actual experience, the system stores the data discovered for [state, action, reward, next_state] - typically in a large table. Note this does not store associated values - this is the raw data to feed into action-value calculations later.

The learning phase is then logically separate from gaining experience, and based on taking random samples from this table. You still want to interleave the two processes - acting and learning - because improving the policy will lead to different behaviour that should explore actions closer to optimal ones, and you want to learn from those. However, you can split this how you like - e.g. take one step, learn from three random prior steps etc. The Q-Learning targets when using experience replay use the same targets as the online version, so there is no new formula for that. The loss formula given is also the one you would use for DQN without experience replay. The difference is only which s, a, r, s', a' you feed into it.

In DQN, the DeepMind team also maintained two networks and switched which one was learning and which one feeding in current action-value estimates as "bootstraps". This helped with stability of the algorithm when using a non-linear function approximator. That's what the bar stands for in ${\theta}^{\overline{\space}}_i$ - it denotes the alternate frozen version of the weights.

Advantages of experience replay:

More efficient use of previous experience, by learning with it multiple times. This is key when gaining real-world experience is costly, you can get full use of it. The Q-learning updates are incremental and do not converge quickly, so multiple passes with the same data is beneficial, especially when there is low variance in immediate outcomes (reward, next state) given the same state, action pair.
Better convergence behaviour when training a function approximator. Partly this is because the data is more like i.i.d. data assumed in most supervised learning convergence proofs.

Disadvantage of experience replay:

It is harder to use multi-step learning algorithms, such as Q($\lambda$), which can be tuned to give better learning curves by balancing between bias (due to bootstrapping) and variance (due to delays and randomness in long-term outcomes). Multi-step DQN with experience-replay DQN is one of the extensions explored in the paper Rainbow: Combining Improvements in Deep Reinforcement Learning.

The approach used in DQN is briefly outlined by David Silver in parts of this video lecture (around 01:17:00, but worth seeing sections before it). I recommend watching the whole series, which is a graduate level course on reinforcement learning, if you have time.


will k-means clustering converge to the same results given the same data set? I did some study on the k-means clustering algorithm. It seems that the only non-deterministic part is the centroid - initialization.

Assume I have 10k data points, and a given k. I then initialize the initial centroids randomly in my each try:

Try_1:
   Initial k-centroids randomly with seed_1. Then keep updating the centroids until converge (assuming we can use the 10k data points multiple times)

Try_2:
   Initial k-centroids randomly with seed_2. Then keep updating the centroids until converge (assuming we can use the 10k data points multiple times)

Try_3:
   Initial k-centroids randomly with seed_3. Then keep updating the centroids until converge (assuming we can use the 10k data points multiple times)

Try_4:
   Initial k-centroids randomly with seed_4. Then keep updating the centroids until converge (assuming we can use the 10k data points multiple times)

Try_5:
   Initial k-centroids randomly with seed_5. Then keep updating the centroids until converge (assuming we can use the 10k data points multiple times)

In these 5 tries, will the final cluster results be the same?

They won't necessarily be the same. Consider observations equally distributed over a circle (radius = 1). Depending on the initial centroids, the algorithm will converge on different solutions. For instance, consider the case where two centroids are initially located on each side of one of the circle's diameters. Those can be any pair of points, and the algorithm will already be converged with different solutions.

However, there exist cases where the algorithm will necessarily converge to the same solution. For instance, consider points equally distributed over a segment, in a 2 cluster problem. It is quite clear (although a bit harder to explain) that any initialization will eventually converge to the same solution (this actually need sadditional assumptions, such as no point being on the edge of both clusters, at least).

In your example case, with a more complex structure, the problem is more difficult to analyze. There are some problems which will likely give the same results every time, other that will yield different results. But anyway you can't be sure, in the general case, that it will fall back to a single solution.


Machine learning algorithm to classify matrices I want to know what could be first choice for a machine learning algorithm to classify matrices.

Each matrix belongs to either class A or class B.
The classification problem is: To classify each matrix into either class A or class B (say signal matrix or noise matrix).

Each matrix is arranged in a fashion where each row contains certain features of an object associated with the matrix.

I obviously have thousands of such class A and class B matrices to train on.

I will be really obliged if you could help. Thanks.

You can indeed use a simple neural network for this task, flatten your matrix and feed the matrix entries as inputs to your network. However, I would suggest using a convolutional neural network, these have had very good performance on image data, which is essentially a matrix. This class of NN is powerful because it uses a neighborhood around each point as an input and thus significantly reduces the number of inputs to the network, which in turn reduces the model complexity.

I would also suggest using some matrix transformations to get better features. For example you can perform eigenvector decomposition of the matrix to ensure that the features you are feeding to your network are orthogonal. This may increase performance of your classifier.

If you give us some examples of your dataset we can help you build a network to start testing ideas.


Weights not converging while cost function has converged in neural networks My cost/loss function drops drastically and approaches 0, which looks a sign of convergence. But the weights are still changing in a visible way, a lot faster than the cost function. Should I ensure the weights converge too?

Some details: I just calculated for 1 epoch. My loss function is mean square difference. I use no optimizer. I tried several experiments and different initial weights but all got converged loss function with ever-changing weights.

The weights in a model do not need to converge to stop training.

One possible explanation is that the model error surface has a big, wide valley. If that is the case, the loss function would be low throughout the valley but there would be many weight combinations that would all yield similar performance on the training dataset. Once a model has reached an acceptable loss function value there is no reason to continue training, just take any set of weight values.


Improve test accuracy for TensorFlow CNN I'm trying to use Tensorflow for signal classification. The signals are either normal or high-risk signals. For this purpose, I used convolutional neural networks. The length of signals are 685 and the architecture is:

Convolution layer with 27 channels and 1 by 16 window size and stride 1.
Max pooling layer with 1 by 2 window size and stride 2.
Convolution layer with 14 channels and 1 by 32 window size and stride 1.
Max pooling layer with 1 by 2 window size and stride 2.
Convolution layer with 4 channels and 1 by 32 window size and stride 1.
Max pooling layer with 1 by 2 window size and stride 2.
Convolution layer with 3 channels and 1 by 10 window size and stride 1.
Max pooling layer with 1 by 2 window size and stride 2.
Fully connected layer with 20 neurons and dropout layer.
Fully connected layer with 10 neurons and dropout layer.
And finally Soft max layer.

After training the network using AdamOptimizer with learning rate 0.001 with 150000 signals the training accuracy is near 95 percent (batch training with 16 batch size is used), however testing accuracy using 20000 new signals is almost 50 percent. Since there is just 2 classes, this accuracy is no better than random guess.

model="conv1d-27-16-1,maxpool-2,conv1d-14-32-1,maxpool-2,conv1d-4-32-1,maxpool-2,conv1d-3-10-1,maxpool-2,full-20,full-10,softmax"

cnn=CNN_1D(model,input_size=685,n_classes=2,num_epochs=20,batch_size=16,dropout=0.75)
cnn.read_data('train_input','train_targets','test_input','test_targets')
cnn.build_model()
cnn.training(validation_set='all')

How can I improve the testing accuracy in my network?

This is clearly a case of overfitting, since your validation error is way bigger than your training error. Having understood that overfitting is your problem, there are many ways you can address this problem:

Dropout (keras)
Weight decay (L2 regularization)
Early stopping (keras)

I can see that you are already doing dropout, so setting the dropout higher should help. Moreover, you can try using dropout in more layers, not just the two last ones. If setting the dropout rate higher and setting more layers with dropout does not work, then this probably means that your training and validation data do not come from the same distribution. You can study statistical properties of your training data and validation and study if they come from the same distribution.

Extra comment: if there are just two classes, why don't you use a sigmoid activation function in the last layer?


Workflow for Iteratively Capturing and Improving Machine Learning Models I am working with multiple machine learning methods (support vector regression, random forest regression, and knn regression through scikit-learn) and want to know how to determine which method is most appropriate to pursue given incrementally increasing my training sample by 10,000 records.

My training data size is about 80k records, and my test dataset is approximately 30k records. I start with 10k records to train each of my models because the system I am working in has very limited cores (and runs on a VM on a different continent) so I can't simply start a training session and walk away for a few hours.

As I increase the training size, I expect that my training validation scores (RMSE, MAPE) will decrease on the full training dataset, and my testing validation scores will increase.

What are the metrics that I should capture about each run in order to make informed decisions about the complexity, speed, and long term viability of these machine learning models?

(This post does a great job of explaining the general steps associated with the *entire* data science workflow. However, I am particularly interested in exploring what would be considered the ideal set of metrics that I should capture after having trained, tested, and validated a machine learning model.)

For Modelling , it's probably best to go with Partial learning models. Explore DASK.

If you can, getting cloud support to run your models is also excellent idea. For examples Google colab.

For all models keep log of model accuracy and cross validation accuracy. It's good idea to keep these details in small table and build a line graph which shows accuracy details for each version.

For latency, use Pytest and build differential testing code so that after each iteration of testing it provide tesrit results.


Translating a business problem into a machine learning solution: job-adds website I have the following problem:

A company, let's call it X, has a job-adds website. It works as a marketplace where job-seekers and companies that have job vacancies can meet.

Their business model is such that job-seekers pay a subscription to get access to the website after a trial period. Companies, on the other hand, don't pay anything to post job-adds.

In order to attract companies to post jobs on the website they call them and ask if they're interested. They have a database with N companies that did business with them in the past. Out of those "N" they can only call 20% due to budget and personnel constraints.

They select those 20% out of simple heuristic: companies that posted job adds in the last six months.

What they want is to call companies that have high probability of posting a job add that will generate a match. They consider that a match has happened based on a proxy: the company contacted n different candidates for a particular job-add.

I want to find a data-driven approach to select the best companies for them to call. In other words, provide them with a list of companies that have a high probability of generating adds that match with their current database of users.

The problem I'm having is how to transform this business problem into a machine learning solution.

What I have thought so far is to build a classification model. Each row is a job-add. The features are variables related to the company (size, location), to the user (age, CV, last time logged in), and the job-add itself (salary,education required, position, industry). The target variable is a binary of "the job had a mach or not."

The problem with my approach is that the model is not actually learning anything since my target variable is already known for each job-post (I know which job-adds had a match and which didn't).

I'm looking for some ideas on how I can structure this problem in a data-driven framework. Should I continue with my classification model? Maybe some type of collaborative filtering? I'm sure machine learning can provide better results than their simple heuristic. Thanks!

The data I have:

User data, including CV;
Who applied to what job and when;
Data from the companies that posted job-adds;
Contacts made by companies to job-seekers (This just tells me if the company clicked a button to look at the user's contact info);

With this kind of general problem there are many possible approaches, and I can't list them all.  In general, you want to go with the simplest model that creates value over a more naive approach.  So we can look at the different approaches, ranked by complexity, and see what you need for each.  This is the order I would try for this analysis:

(1) The simplest approach is just contact a flat 20%, drawn randomly.  This is very easy to do, but hopefully it can be improved upon.

(2) Next, you can try an approach based on the basic statistics of the problem at hand.  For example, you can calculate for those employers which previously placed ads, how many applications* did they get.  And then start with the ones that got the most, and work your way down.

(3) Now that you have baselines sets from doing a blind and then a basic descriptive analytics approach, you can try something a bit more complicated.  Assuming you have the text from the past ads, you can mine that text for features which might be descriptive of what they are looking for.  If the employers placed multiple ads, you can combine those and look for features in the aggregate.  Once you do the same with the resume data, you can create a recommendation engine to match potential employers with employees.  You can then use that data to determine which possible employer is most likely to have potential matches in your client pool.  As a side effect, this engine can also guide your clients to which jobs might be the best matches.  There are many variations which you can use here.  For example, for the recommendation engine, you may want to weight your client sample towards clients which have logged in the most recently, or most frequently, so or you can just filter out the "stale" clients who haven't really been looking recently.

*An alarm bell went off when you said you know who applied.  Are you sure?  Or do you just have clickthrough statistics to know whether they started an application?  How do you know that they applied?  The first rule of real world data analytics is that you need to be skeptical of your data and especially of what you think your data represents.


How to restore deleted objects in R? Suppose if I delete all the objects from current session:

rm(list=ls())

Is there any way in base R or using a function from a package which lets me restore the deleted objects from the current session?

The answer is unfortunately no. There is no handy ctrl-z method.

A tip to avoid these situations: I suggest you always save either the 'environment' or, as I prefer to do, the scripts with the codes for the desired objects, and save them regularly. I never type any commands directly into a work space, but always in a script which I save. This is so I can always look back at which steps I took in my coding. Personally, I save my scripts with date notations in the file name - so when I change a script I'll still have access to the old version of the scripts. It has happened quite a few times that I've changed a code and later realized that the old code was better. It's quite a hassle when you don't have the old file saved.


Why we use an activation function for introducing nonlinearity instead of a polynomial Perceptron implementation? I perceive a single perceptron as a single linear function $y = a_1x_1 + a_2x_2 + ... + a_nx_n + b_0$ with a goal to calculate the best weights combination $ w_1, w_2, ..., w_n $ that minimizes the given loss function.
The problem with this type of network is that it would not be able to perform well on a non linear dataset, thus an activation function would be used in order to tackle this. I am wandering what could happen if instead of a linear Perceptron, we introduce a polynomial Perceptron in the form of $ y = a_1x_1^k + a_2x_2^r + ...+ a_nx_n + b_0$ and how this will compare with the original perceptron.

Well, thanks to the universal approximation theorem from a purely theoretical point of view, absolutely nothing.
The main issue is with computation. You can find more information here. Mainly, you want functions easy to calculate (polynomials are ok) but with specific regions where derivatives are monotonic (here polynomials are not good) and approximating the identity near the origin (again, polynomials not good).


What are the differences between Reinforcement Learning (RL) and Supervised Learning? What is the difference between Reinforcement Learning (RL) and Supervised Learning?

Does RL hava more difficulty in finding a stable solution?

Does Q-learning have more difficulty in finding a stable solution?

Does getting stuck in a local minimum happen more in supervised learning?

Is this figure correct saying that Supervised Learning is part of RL?

What are difference between Reinforcement Learning (RL) and Supervised Learning?

The main difference is to do with how "correct" or optimal results are learned:

In Supervised Learning, the learning model is presented with an input and desired output. It learns by example.
In Reinforcement Learning, the learning agent is presented with an environment and must guess correct output. Whilst it receives feedback on how good its guess was, it is never told the correct output (and in addition the feedback may be delayed). It learns by exploration, or trial and error.

  Could we say RL has more difficulty in [finding] a stable solution?

No, because the types of problems it solves are usually different, you cannot compare like with like.

However:

You could use RL as a framework to produce a predictive model for a dataset (using RL to solve a Supervised Learning problem). This would be inefficient, but it would work and there is no reason to expect it to be unstable.
In general, RL problems present additional challenges to supervised learning problems with the same degree of complexity in the relationship between input and output. They are "harder" in that sense, as more details need to be managed, there are more hyperparameters to tune.
RL can become unstable in ways that don't apply in supervised learning. For instance Q learning with neural network approximation tends to diverge, and needs special care (often experience replay is enough)

  Could we say that [getting stuck] in [a] local minimum is seen more in supervised learning?

No. Internally, a RL agent will often use one kind of supervised learning algorithm to predict value functions. RL does not include any special features that can avoid or work around local minima. In addition, a RL agent can get stuck in ways that don't apply to supervised learning, for instance if an agent never discovers a high reward, it will create a policy that ignores entirely the possibility of getting to that reward.

  Is this figure is correct saying that Supervised Learning is part of RL?

No. The figure is at best an over-simplified view of one of the ways you could describe relationships between the Supervised Learning, Contextual Bandits and Reinforcement Learning.

The figure is broadly correct in that you could use a Contextual Bandit solver as a framework to solve a Supervised Learning problem, and a RL solver as a framework to solve the other two. However, this is not necessary, and does not capture the relationships or differences between the three types of algorithm.

Otherwise the figure contains so much over-simplification and things that I would consider as errors (although this might just be missing context that explains the diagram), that I recommend you simply forget about trying to interpret it.


is it possible get a overfit underfit comparation between models, with this chart? (homework) I am trying to interpret this chart.
I am not sure how to interpret this, because, I think that the fact of the for examples LGBM Validation error, is wide and similar to train boxplot, there arent problem of overfitting, but when I see another type of charts of the execution of LGBM, I can see that really the LGBM is overfitted, so really I don't know how to interpret this of the correct way.

But I don't know how could interpret beyond this:
LightGBM is maybe the best option because it is faster and finally you can get enough accuracy with that, and in comparison with the other two, bagging have less overfit because of the differences between the error is less.
Any idea?
Thanks

Your chart seems to show that light GBM models are very inconsistent in terms of F1 score. The other two types of model tend to have lower validation accuracy than training accuracy, suggesting overfitting is occurring to some extent (but this is ubiquitous in machine learning so it’s not a deal breaker by any means). The best median validation performance is by RandomForest, however some outliers underperformed the models using bagging. Possibly a good approach would be to have an ensemble of RandomForest models.


Difference between bagging and pasting? I found the definition:
Bagging is to use the same training for every predictor, but to train them on
different random subsets of the training set.
When sampling is performed with replacement, this method is
called bagging (short for bootstrap aggregating).
When sampling is performed without replacement, it is called pasting.

What is "replacement" in this context?

When a sampling unit is drawn from a finite population and is returned
to that population, after its characteristic(s) have been recorded,
before the next unit is drawn, the sampling is said to be “with
replacement”. In the contrary case the sampling is “without
replacement”.

Source: OECD Glossary of Statistical Terms


Plot RDD data using a pyspark dataframe from csv file I have just started working with pyspark on very large csv file. I am using Spark version 2.1.0. I want to read data from a .csv file and load it into a spark dataframe and then after filtering specific rows, I would like to visualize it by plotting 2 columns (latitude and longitude) using matplotlib.

This is what I've done so far:

import packages:

from pyspark.sql import SparkSession
from pyspark.conf import SparkConf
from pyspark.sql.functions import *
import matplotlib.pyplot as plt

build spark session:

spark = SparkSession.builder \
     .master("master:IP") \
     .appName("plot track") \
     .config(conf=SparkConf()) \
     .getOrCreate()

read .csv file:

df = spark.read.csv('data.csv')

filter rows according to values in a specific column ("_c1"):

one_ID = df.filter("_c1 == 1234")

save column data? (Latitude data is in column "_c3" and longitude data is in column "_c4")

latitude = one_ID.rdd.map(lambda y: y._c3)
longitude = one_ID.rdd.map(lambda x: x._c4)

plot:

plt.plot(longitude,latitude, 'o-')
plt.show()

I thought this would work but it does not. I get the error -
- 'TypeError: float() argument must be a string or a number'

Any help would be greatly appreciated. Thanks.

There is no need of two rdd values,

plotdf=df.select(_c3,_c4);
plotdf.plot( style=[])
plt.show()

If you still face the error, please follow the link and set index.


Bi-directionality in BERT model I am reading the paper BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding that can be found here.

It looks to me that the crux of the paper is using masked inputs to achieve bidirectionally.

This is an excerpt from the Google AI blog here which states:

  "However, it is not possible to train bidirectional models by simply
  conditioning each word on its previous and next words, since this
  would allow the word that’s being predicted to indirectly “see itself”
  in a multi-layer model.  To solve this problem, we use the
  straightforward technique of masking out some of the words in the
  input and then condition each word bidirectionally to predict the
  masked words."

Can someone please help me understand how does bidirectionally allow the words to see themselves and how masking solves this problem?

Thanks.

Let's take an example : "I went to the shop."

Let's say you want to predict "to" and "the". With bidirectionality, you will predict :

p(to | I, went, the, shop) : No problem here.

p(the | I, went, to, shop) : Here we have a problem, because we already saw the word 'the' while predicting 'to'. It's trivial for the model to predict 'the'.

With MLM, if we take the same example :

p(to | I, went, [MASK], [MASK], shop)
p(the | I, went, [MASK], [MASK], shop)

There is no more problem because the word cannot see itself in other predictions. It's more difficult to predict, but anyway only 15% of the time the word is masked.

Edit

In MLM, BERT will see all the words of the sentence, including the word to predict itself.
This is because BERT will create representations not only for the [MASK] token, but also for the other tokens of the sentence (but in this case we are interested only in [MASK]).

Note : It is also because BERT will create representations of all tokens that we need to replace [MASK] sometimes by the original token, sometimes by a random token.

You can find more detailed information in this blog in the "Pretraining" section.


Accessing Flask WS APIs over intranet - I have 2 scripts - A.py and B.py, and both are Flask apps.
A.py renders a web page and acts as my UI taking inputs from user.
B.py is hold the main logic and has a web service API being called by A.py.

Both run as flask app services in localhost from different ports - 5001 and 5002, respectively.

My doubts -

1) Every time I make any changes to my B.py (or A.py), do i need to stop and restart the services?

2) I'm currently running both the services from local machine for proof-of-concept purposes. If I use WSGI (thinking of Bottle), will it be accessible from other machines over intranet?

I'm on Windows 10 (64-bit), Python 3.6 and Flask 1.1.1

Please advise.

Edit 1) Added OS and Python version details.

1) YES. the updates won't be visible not until your restart your webservice. ( Both by the way ).

2) Gunicorn works well with Flask as a wsgi app. I'm not sure how to answer your question. All i can say is that i have webservices deployed to production with both Gunicorn+ Flask and Flask only, and they are both accessible over internet. Only difference is that Flask is not meant for production deployment, only for testing purposes.


Algorithms for aggregating duplicate identities based on non-numerical data? I have a large dataset (2M entries) of people, but many people have multiple entries in the database with slightly (or significantly) different identifying information. For example, I may have J. Doe and John Doe, or I may have John Doe with an accompanying email address and John Doe without an accompanying email address.

I've been looking at different clustering algorithms but nothing seems well-suited to what I'm doing, which is to aggregate the entries based on rules like the following:

guess first and last names based on whether one of the names is written in all capitals
aggregate "J. Doe" and "J. Doe" if email addresses match
aggregate "J. Doe" into "John Doe" if no other people have first name starting with "J" and last name "Doe"

With a smaller dataset this would be a relatively straightforward task to do just with some simple rules, but with the number of entries I have the aggregation tasks can get really slow and the logic gets pretty convoluted. My current solution (based around using the fulltext search function within my database to find similar entries, adding hashes based on those results, and then aggregating based on a mix of hashes and types of ambiguity) works, but every time I try to run it or update it it just screams that it's the kind of problem somebody else has already solved. But I haven't been able to find a solution.

Are there algorithms that will do what I want based on rules like this? Or specific packages or software that might be helpful? Or am I approaching this problem completely wrong?

Thanks!

(Please note though that I am well aware that there are many different ways to incorrectly aggregate identities (e.g. that J. Doe could mean John Doe or James Doe), so I don't need warnings against trying to aggregate things.)

I haven't yet successfully solved my record linkage problem, but I wanted to share some of the stuff I've found in the process in case it's of use to anyone else. This is a work in progress based here on GitHub.

Record Linkage Resources

(also known as deduplication, data matching, entity resolution)

Background

Documents

Wikipedia pages on Record Linkage and Data Deduplication
Overview slideshttps://www.umiacs.umd.edu/~getoor/Tutorials/ER_VLDB2012.pdf
[Dedupe]'s explanation of how their software works: https://dedupe.io/developers/library/en/latest/How-it-works.html

Talks

Peter Christen, Record Linkage lectures at ADRC-Scotland: 1 2 3 4 (2015)
Mike Mull, The Art and Science of Data Matching: https://www.youtube.com/watch?v=Y-nYEOgq3YE (2015)
Rhydwyn Mcguire, Join for real life: https://www.youtube.com/watch?v=cEcVIjyHfiQ (2013)
Andrew Rowe, Big Data Deduplication and Data Matching using Python: https://www.youtube.com/watch?v=Z6mlvrYEYnk (2013)

Books

Peter Christen, Data Matching: http://www.springer.com/us/book/9783642311635
Thomas N. Herzog, Fritz J. Scheuren and William E. Winkler, Data Quality and Record Linkage Techniques: http://www.springer.com/us/book/9780387695020

Free software

(last updated, github stars as of Nov 2017)

Python

Dedupe: https://github.com/dedupeio/dedupe (2017, 1,413)
RLTK: https://github.com/usc-isi-i2/rltk/ (2017, 13)
Febrl: https://sourceforge.net/projects/febrl/ (2013)

Java

Duke: https://github.com/larsga/Duke (2016, 469)
JedAI: https://github.com/scify/JedAIToolkit (2017, 12)
LSHDB: https://github.com/dimkar121/LSHDB (2017, 9)
FRIL: http://fril.sourceforge.net/ (2011) (Windows binary available)

R

RecordLinkage: https://r-forge.r-project.org/projects/recordlinkage/ https://cran.r-project.org/web/packages/RecordLinkage/index.html
fastLink: https://github.com/kosukeimai/fastLink

Other

OpenRecLink: http://reclink.sourceforge.net/ (2016)

C++ with GUI

Registry Plus™ Link Plus: https://www.cdc.gov/cancer/npcr/tools/registryplus/lp_tech_info.htm (2007)

Commercial software and solutions

Data Ladder DataMatch: https://dataladder.com/
Dedupe: https://dedupe.io/ (freemium frontend for Dedupe Python library)
LinkageWiz: http://www.linkagewiz.net/
WinPure Clean and Match: http://www.winpure.com/cleanmatch.html
Reifier: http://nubetech.co/

For SAS

(free but requires SAS) The Link King: http://www.the-link-king.com/

Data Cleaning

Name Parsers

Python

probablepeople: https://github.com/datamade/probablepeople (2017, 204)
Name Parser: https://github.com/derek73/python-nameparser (2017, 232)

JavaScript

parse-full-name: https://github.com/dschnelldavis/parse-full-name (2017, 18)

Papers

US Census Bureau Data Linkage research: https://www.census.gov/srd/csrm/RecordLinkage.html
Learnable Similarity Functions and their Application to Record Linkage and Clustering (via)
[https://www.microsoft.com/en-us/research/publication/improving-entity-resolution-with-global-constraints/ (via)
https://arxiv.org/abs/1312.4645 (via)
http://people.cs.umass.edu/~mwick/MikeWeb/Publications_files/wick09entity.pdf (via)
http://homes.cs.washington.edu/~pedrod/papers/mrdm04.pdf (via)
https://www.researchgate.net/publication/318874102_Clink_-_A_Novel_Record_Linkage_Methodology_based_on_Graph_Interactions

Organizations

US Census Burea - Center for Statistical Research and Methodolgy - Record Linkage: https://www.census.gov/srd/csrm/RecordLinkage.html
Stanford Entity Resolution Framework: http://infolab.stanford.edu/serf/
ANU Data Mining and Matching Group https://dmm.anu.edu.au/ https://web.archive.org/web/20160515215747/datamining.anu.edu.au/projects/linkage.html (Archive.org link)

Misc

DuDe (framework for comparing record linkage results): https://hpi.de/naumann/projects/data-quality-and-cleansing/dude-duplicate-detection.html
Datasets to use for evaluating deduplication software: https://hpi.de/naumann/projects/repeatability/datasets.html
https://www2.vrdc.cornell.edu/news/3/20050420-Record%20Linkage%20Software.pdf


How do we know a neural network test accuracy is good enough when results vary with different runs? In every paper I read about prediction models, the training accuracy and the test accuracy (sometimes also the validation accuracy) is stated as a discrete number. However, in experience, depending on how the weights are initialized, different training runs result in different testing results.
How does a standard data science researcher pinpoint the accuracy metric to be written in a paper, and how does ze get certain about its validness?

So, the question is about how to report test accuracy, etc, when you see variation over executions.
As @Nikos M. has eluded to, you typically train and test the model at least 3 times and then report the average test evaluation metrics (along with the standard deviation to show the level of variation). This is why you are seeing 'discrete' values reported in papers, this is because they are averaged over x train/test runs.


Estimate the location of an object in a field using computer vision and math I'm trying to see how to detect the location of a soccer ball in the field using the live camera. What are some ways to achieve this? At this point I'm more interested in ideas and thoughts and not really code.
1- Assuming we have a fixed camera with a wide shot. How to find the ball location on the actuall field? It's probably using a computer vision approach.
2- A camera is zooming into the ball. But we know the location of the camera and maybe it's turning angle. Can we estimate the ball location on the field using this info? Or maybe we need additional info? Can we do it with two cameras as reference points? Maybe some math or triangulation sort of approaches would help here.
Any thoughts would be helpful.

Your problem is focused on the object detection perspective. For a intro into the topic with some code you could look into this - https://medium.com/analytics-vidhya/detecting-objects-on-ip-camera-video-with-tensorflow-and-opencv-e2c25297a75a.
General object detection is quite easy for the second picture but for the first it would be a bit difficult. You could use dlib and CNN networks such as YOLO and RCNN (family of models).
Yes you could do it with two cameras. The configuration depends on you whether you use selected area for both cameras or both cameras cover the entire field.
https://www.geeksforgeeks.org/detect-an-object-with-opencv-python/ - The code here detects stop signs. You could find a dataset which deals with footballs/sports and do the exact same.
This link provides your objective for close range (zoomed in) photos - https://towardsdatascience.com/yolov2-to-detect-your-own-objects-soccer-ball-using-darkflow-a4f98d5ce5bf


Any non Deep Learning python packages for sequence classification.? Stats model or any other machine learning python packages for doing sequence classification(that can be multi class) and sequence prediction (Both next step and regression).

PS : Input data will be n number of sequences with their respective target (class/regression)

Also each sequences having finite number of events like E1,E2 ..

Have you thought of using Conditional Random Fields? I had used it for my thesis some years ago and papers from Prof. Sunita Sarawagi were a great resource for me. You can use the CRFsuite library which also offers a python wrapper.


What visualization I should choose for Monte Carlo simulations in timeline events? I wasn't sure if I should open this question in Cross Validated or here. But since the question belongs to a bigger project related with Data Science, I chose this one.

I will present a simplified version of my working project, since the original is too complicated and domain specific.

Let's say that we have a timeline of 1 hour (60 minutes). During this period a job starts running and create user notifications in random points. I have written a Monte Carlo simulation to study this process.

My main questions:

How well those notifications are spread on the period of 60 minutes?
Do we have parts of this timeline that don't contains notifications and are the rest clustered in a specific time?
Changing the random functions in the implementation, how the above answers will be affected?

A pseudo code for the Monte Carlo, which mimic the actual code is:

Repeat one million times
    number_of_notifications  = get_random_number_of_notifications ()
    previous_point = 0
    for i in range(0 to number_of_notifications):
        interval = get_random_interval()
        new_point = previous_point + interval
        previous_point = new_point

Note: In the current implementation, two or more notifications can be at the same minute.

First, I thought that creating a histogram of the specific minutes during the simulations would help me to answer the first question. But then, I realize that I could have one simulation with all the events in the first half and another one in the second half and the histogram misleading that it is well spread.

Then I thought that it might be nice to plot also the min, max, average and std of the intervals in each simulation. But then, would be enough to answer them?

What kind of visualizations should I try to give me insights about the notifications in the Monte Carlo simulation?

Instead of  (or additionally to) histogram you may check the density plot of the distribution of the intervals between the
events. In the r snipped below the process id identified with  ID and the interval length is INT_LEN .

densityplot(~INT_LEN,  groups = ID,
data =  df,
scales=list(x=list(rot=90, cex= .9),y=list(cex=.9)),par.strip.text=list(cex=.8), plot.points = F,
ylab="density", xlab="Interval between Events", main=paste("Interval Length Density per Process"  ) )

This show the distribution (in my simple case there are two different distribution). One group of processes has a normal distribution of event with a mean of 1,2. So you may argue that there are 50 events per minute on average.

But this may not be true as the time sequence is lost in this view.

To get the time series information I select a basic interval length, where the distribution of the event doesn't play a role any more.
I use a minute, but if this is to coarse go down. The point is there is no difference to get all events at the beginning of the interval (minute)
or uniformly distribute within the interval.

This selection allows to aggregate the data on the interval level and present it as a time series. The graph below shows the count of event per minute.

xy <- xyplot(CNT ~ MINUTE_NO , groups = ID,
data =  df ,    type=c("o","g"),
scales=list(rot=90, x=list( cex= .9 ),y=list(rot=0, cex=.8, alternating = c(1,1), tck=c(1,0), relation = "free")),par.strip.text=list(cex=.8),
ylab="event count per minute", xlab="time", main=paste("Time Series - Event Count per Minute"  ) )
print (xy)

Here you can spot a possible trend in the distribution of the events.


Predict using a saved regression model I have trained an ANN model for a regression problem which takes 10 parameters as input and gives 1 output. After training, I saved the model as json and weights as a .h5 file using keras. Now I want to make predictions on the new data. I have loaded the model and my question here is how do I scale this single row of input values, before feeding it to the model? Some of the machine learning algorithms did not require scaling(Standardization/Normalization) So I could load those models and used for making predictions. How do I do in an ANN model, since we scale the data for training?

Use sklearn.preprocessing.OneHotEncoder for example and transfer the one-hot encoding to your web-service ( i'm guessing that's how you're using the model for inference ) via sklearn.pipeline.Pipeline. The pipeline will save the state of your fit on your training data and apply the same function on your production data.

Example :

pipeline1 = Pipeline([
                ('OneHotEncoder', OneHotEncoder())
            ])
pipeline1.fit(trainingdata.column1.values.reshape(-1,1))

This is how you create a pipeline containing the onehotencoder , fit your data on the pipeline. All is left is dumping your pipeline in a file, loading it later in your production environment, and call the transform method on your loaded pipeline :

joblib.dump(pipeline1,"pipeline1.joblib")
# Production environment
pipeline1 = joblib.load('pipeline1.joblib')
momo = pipeline1.transform(productiondata.column1.values.reshape(-1,1)).toarray()

And here , the variable momo contains your production data with the pipeline ( containing the one-hot encoding operation ) applied to it.


White noise seasonality Is the above graph a white noise?
I'm confused by the spikes at certain places.
The above plot has been obtained after doing a first order differencing on a time series. How can I justify whether there is seasonality present/absent in the data after differencing?

How can I justify whether there is seasonality present/absent in the
  data after differencing?

Plot the amplitude of the Fourier transform of the signal.

If there is seasonality, you will see a peak at the appropriate frequency on the Fourier plot. This should be close to the plot's origin, because seasonality means slow changes and thus low frequencies.


How to make prediction using tensorflow models? As a newbie to tensorflow, I am using this tutorial from google for binary classification using a simple dense neural network.

The slightly annoying thing about this (and a few other) tutorials is they completely gloss over the part of how to actually make a prediction from a dataframe of features, and directly move to model evaluation using some method of the trained model which hides the actual prediction procedure. So basically, I finished the model training, but even after that, I see no way mentioned on how to actually use the model to predict classes of unknown samples.

To put concretely, I have a trained model, a pandas dataframe called test, and a list of columnames which correspond to feature names. Based on the variables declared in the tutorial, I tried

feature_names=['age', 'sex', 'cp', 'trestbps', 'chol', 'fbs', 'restecg', 'thalach', 'exang', 'oldpeak', 'slope', 'ca', 'thal']

model.predict_proba(x=test[feature_names])

But it is throwing a type error. Basically, I need a function which will give me the classes, or ideally, the softmax probabilities of the classes from the feature frames without the label, because that is how we use any model.

First, you need to convert the dataframe in numpy array or tf.data dataset that the model understands. For this purpose, the tutorial provides you with a function:

# A utility method to create a tf.data dataset from a Pandas Dataframe
def df_to_dataset(dataframe, shuffle=True, batch_size=32):
  dataframe = dataframe.copy()
  labels = dataframe.pop('target')
  ds = tf.data.Dataset.from_tensor_slices((dict(dataframe), labels))
  if shuffle:
    ds = ds.shuffle(buffer_size=len(dataframe))
  ds = ds.batch(batch_size)
  return ds

test_ds = df_to_dataset(test, shuffle=False, batch_size=batch_size)

Then you can get the predictions like this:

model.predict_proba(x=test_ds)

Take into account that the test-input format should be the same as training-input format, so if you have done any preprocessing (on_hot_encode, standardize, bucketize, etc) to the training dataset, you should do it also to the test dataset

Another way, if your dataset is ready for prediction, you can just transform it to a numpy array:

model.predict_proba(x=test[feature_names].values)


Scikit-learn: Getting SGDClassifier to predict as well as a Logistic Regression A way to train a Logistic Regression is by using stochastic gradient descent, which scikit-learn offers an interface to.

What I would like to do is take a scikit-learn's SGDClassifier and have it score the same as a Logistic Regression here. However, I must be missing some machine learning enhancements, since my scores are not equivalent.

This is my current code. What am I missing on the SGDClassifier which would have it produce the same results as a Logistic Regression?

from sklearn import datasets
from sklearn.linear_model import LogisticRegression
from sklearn.linear_model import SGDClassifier
import numpy as np
import pandas as pd
from sklearn.cross_validation import KFold
from sklearn.metrics import accuracy_score

# Note that the iris dataset is available in sklearn by default.
# This data is also conveniently preprocessed.
iris = datasets.load_iris()
X = iris["data"]
Y = iris["target"]

numFolds = 10
kf = KFold(len(X), numFolds, shuffle=True)

# These are "Class objects". For each Class, find the AUC through
# 10 fold cross validation.
Models = [LogisticRegression, SGDClassifier]
params = [{}, {"loss": "log", "penalty": "l2"}]
for param, Model in zip(params, Models):
    total = 0
    for train_indices, test_indices in kf:

        train_X = X[train_indices, :]; train_Y = Y[train_indices]
        test_X = X[test_indices, :]; test_Y = Y[test_indices]

        reg = Model(**param)
        reg.fit(train_X, train_Y)
        predictions = reg.predict(test_X)
        total += accuracy_score(test_Y, predictions)
    accuracy = total / numFolds
    print "Accuracy score of {0}: {1}".format(Model.__name__, accuracy)

My output:

Accuracy score of LogisticRegression: 0.946666666667
Accuracy score of SGDClassifier: 0.76

The comments about iteration number are spot on. The default SGDClassifier n_iter is 5 meaning you do 5 * num_rows steps in weight space. The sklearn rule of thumb is ~ 1 million steps for typical data. For your example, just set it to 1000 and it might reach tolerance first. Your accuracy is lower with SGDClassifier because it's hitting iteration limit before tolerance so you are "early stopping"

Modifying your code quick and dirty I get:

# Added n_iter here
params = [{}, {"loss": "log", "penalty": "l2", 'n_iter':1000}]

for param, Model in zip(params, Models):
    total = 0
    for train_indices, test_indices in kf:
        train_X = X[train_indices, :]; train_Y = Y[train_indices]
        test_X = X[test_indices, :]; test_Y = Y[test_indices]
        reg = Model(**param)
        reg.fit(train_X, train_Y)
        predictions = reg.predict(test_X)
        total += accuracy_score(test_Y, predictions)

    accuracy = total / numFolds
    print "Accuracy score of {0}: {1}".format(Model.__name__, accuracy)

Accuracy score of LogisticRegression: 0.96
Accuracy score of SGDClassifier: 0.96


Transfer learning with many small datasets Context
I am working on a NLP-model that can classify documents into one of N categories. I have document data from a number of different customers. The document topics are similar across customers but they classify them into different categories.
For simplicity, assume that the documents can contain six different topics: A,B,...,F. Each customer classify the documents differently from the topics, i.e. N mentioned above is customer specific and the mix of topics is (somewhat) different: Customer 1 have three classes (ABC, D, EF), customer 2 have four (A, BC, DE, F), customer 3 have four (AB, CD, E, F), and so on.
The goal is to use the data from the current customers to develop a model that can transfered and retrained on a new customer (with a different mix of topics A,B...,F and potentially only a small amount of data)
What I have done
I have been searching here and in papers for strategies on transfer learning with multiple smaller datasets with different output categories but have not been able to find anything. I have only been able to find papers that deals with larger datasets.
What I am considering
My current thought is to use a deep learning (keras/tf) model with this layout:
document(from customer X) -> preprocessing(shared among customers) -> feature extraction (shared among customers) -> classification (specific to customer X)
With this approach, I would during training switch the customer specific classification layer for each customer. This would train the preprocessing and feature extraction layers on data from all customers but only train the classification layers on individual customer data.
Questions

Do you have any references to papers or any other material that describes a similar situation?

Is this the best strategy for training this model? More specific, is it possible to make a layout of the model where the all customers classification layers are present in the model at the same time and the switching between layers can be handled by a cleverly engineered loss function?

I ended up doing what I proposed in the question: document(from customer X) -> preprocessing(shared among customers) -> feature extraction (shared among customers) -> classification (specific to customer X). I had a loss for each customer X.
I trained the network with a custom training loop: https://keras.io/guides/writing_a_training_loop_from_scratch/
The training converged, which is what I was most uncertain about. The testing accuracy was a bit better compared to training the model with a single customer.
I still did not manage to find any papers on this topic though.


Can you l2 normalize word2vec vectors for density clustering? I have a situation where I have to cluster word2vec vectors (200 length dimension vectors on a very large corpus). I decided to use Density based clustering (DBSCAN, HDBSCAN) because my dataset is very high in noise, and I do not want it to be part of my clusters. My knowledge on Cosine distance is limited but I find that Density based clustering algorithms do not have a direct implementation using cosine distance (pairwise_distance calculation is too memory intensive).

My question here is can I normalize the Word2vec vectors using L2 normalization using: norm_data = normalize(vector_array, norm='l2') from the sklearn library to normalize these vectors, and then use euclidean distance over the normalized vectors?

Can someone suggest any other better technique to cluster word vectors when there is noise in the dataset?

You can do DBSCAN, OPTICS, HDBSCAN with cosine similarity - they do not expect or require Euclidean distances.
Yes, one version of cosine distance corresponds nicely to Euclidean distance of L2 normalized vectors. So there is some theoretical support to L2 normalize the vectors and try Euclidean then.
Given how word2vec is trained, I don't think Euclidean nor cosine distance is the right thing to do. Instead, use the raw dot product as a similarity measure (beware that it is not limited to [0;1] but allows negative similarity - values may be too spread out to use them intuitively).
t-SNE, which was discussed in the comments and another answer, also relies on distances to the nearest neighbors. So it does suffer from the same problem - cosine, Euclidean, dot product? The fast ELKI implementation does support cosine distance as well as Euclidean, but you appear to be set on Python already. Also, if you have a large vocabulary, t-SNE may give you some scalability trouble. The standard algorithm will need O(n²) memory and time.


Regression: What defines Linear and non-linear models or functions Linear regression is used when there is a linear relationship between the input and output variables. Does this linear relationship mean that there is no power over the variables or the parameters? In m understanding, by linear we mean linear with respect to the parameters (no power). Please correct me if wrong. Basically, I am a bit confused looking at this tutorial: https://www.gaussianwaves.com/2013/03/how-to-estimate-unknown-parameters-using-ordinary-least-squares-ols/

where linear regression is used to fit the straight line but if the function is a cubic or higher order polynomial (power over the variables), can we still use linear regression? I have seen that for fitting higher order polynomials, we use Regularized Linear Regression in order to penalize the higher order termed variables. Based on these confusions, can somebody please help clarify the following points?

MAIN QUESTION FOR WHICH BOUNTY IS OPENED:

1) Linearity is with respect to the variables or the parameters? I saw a similar question asked here Is a "curve" considered "linear"? but the second answer says that linearity is in terms of the parameters such as weights or other hyperparameters and polynomial regression is a special type of linear regression. So, if the parameters of the model have a power then that function is a nonlinear function and hence we get a nonlinear model. Is that correct?

Other questions that have been answered and understood by me

2) Do higher order termed polynomials fall under linear or non-linear models?

3)The sigmoid function does not have any power and is commonly used in the hidden layers of multi layer perceptron thereby making MLP non-linear machine learning model? How come sigmoid curve is a non-linear function?

4) Is there a check whether to go for linear or non-linear functions ie., whether to fit linear regression or non-linear models such as SVM with kernel?

I think it is helpful to distinguish between linear functions (representing the relationship between independent and dependent variables) and linear models (representing the relationship between the model parameters and the outcome).

A linear model can be represented by a non-linear function. A linear regression model is any model that is represented by a linear function in the parameters. A polynomial can be represented as such.

Likewise a nonlinear regression model is any model that is represented by a nonlinear combination of parameters.

To quote from Wikipedia:

  Although polynomial regression fits a nonlinear model to the data, as
  a statistical estimation problem it is linear, in the sense that the
  regression function E(y | x) is linear in the unknown parameters that
  are estimated from the data.

Let the model be represented by the function $f(x,w)$ with inputs $x$ and parameter vector $w$. Polynomial regression has the form $f(x,w) = w_0 + w_1*x + w_2*x^2 \ldots$ It is linear in the parameters because $f(x,a+b) = (a_0+b_0) + (a_1+b_1)*x + (a_2+b_2)*x^2 \ldots = a_0 + a_1*x + a_2*x^2 \ldots + b_0 + b_1*x + b_2*x^2 \ldots = f(x,a) + f(x,b)$. But it is not linear in the inputs because $f(x+y,w) \neq f(x,w)+f(y,w)$.

Often people think about the relationship $y=f(x)$ between $x$ and $y$, but linear in linear regression refers to linearity in $w$.


get_dump() leaf value and AUC I have used Xgboost fitted a model with AUC around 0.73 and I printed out my last booster:

booster[599]:
0:[userkn_hometypecnt<22] yes=1,no=2,missing=1
    1:[userkn_60d_opencardniu_days<40] yes=3,no=4,missing=3
        3:[userkn_30d_opencardniu_days<13] yes=7,no=8,missing=7
            7:[userkn_60d_opencardniu_days<24] yes=15,no=16,missing=15
                15:[userkn_timeminperiod_firstday<1029] yes=29,no=30,missing=29
                    29:leaf=0.000352735
                    30:leaf=-0.0100666
                16:[userkn_rate_aopencardniusum_actiondaycnt<0.972506] yes=31,no=32,missing=31
                    31:leaf=0.000398097
                    32:leaf=-0.0129448
            8:[userkn_hometyperate<0.0977183] yes=17,no=18,missing=17
                17:leaf=0.0239075
                18:[userkn_rate_aopencardniusum_actiondaycnt<0.957994] yes=35,no=36,missing=35
                    35:leaf=-0.00201536
                    36:leaf=0.00858442
        4:[userkn_newacitoncntactiondayavg<8.82511] yes=9,no=10,missing=9
            9:[userkn_mingap_importcard_open<297306] yes=19,no=20,missing=19
                19:[userkn_rate_aopencardniusum_actiondaycnt<0.974763] yes=37,no=38,missing=37
                    37:leaf=-0.0138254
                    38:leaf=0.00521038
                20:[userkn_onlinetime_firstday<1961.5] yes=39,no=40,missing=39
                    39:leaf=0.0247849
                    40:leaf=-0.00297016
            10:[userkn_60d_opencardniu_days<59] yes=21,no=22,missing=21
                21:[userkn_rate_repeatcntmaxactionrepeatcnt_actioncnt<0.124787] yes=41,no=42,missing=41
                    41:leaf=0.0101992
                    42:leaf=-0.0222082
                22:leaf=0.0145614
    2:[userkn_hometyperate_firstday<0.25266] yes=5,no=6,missing=5
        5:[userkn_aenterapplyloanpagecntactiondayavg<0.787338] yes=11,no=12,missing=11
            11:[userkn_newacitoncntactiondayavg<8.48678] yes=23,no=24,missing=23
                23:[userkn_worktimeactionrate<0.36514] yes=43,no=44,missing=43
                    43:leaf=-0.0178327
                    44:leaf=0.0168168
                24:leaf=0.0254048
            12:[userkn_newacitontyperate_firstday<0.794737] yes=25,no=26,missing=25
                25:[userkn_newacitoncntactiondayavg<7.14581] yes=47,no=48,missing=47
                    47:leaf=0.0175715
                    48:leaf=-0.00748876
                26:leaf=0.0174804
        6:[userkn_aopencardniurate_firstday<0.0458042] yes=13,no=14,missing=13
            13:[userkn_avgperday_opencardniu_cnt<7.44167] yes=27,no=28,missing=27
                27:leaf=0.00171541
                28:leaf=-0.0229204
            14:leaf=0.00968641

If I am right, the leaf value is the value of logodds and it can be changed into a probability with the sigmoid function. However in the last booster all the leaf values changed to around 0.5 probability.

Which means all the samples will be marked as good/bad cases half and half? So it's no difference with a random guess at a binary classification?

Am I right or any other opinions are quite appreciated!

Could you clarify what you mean by "However in the last booster all the leaf values changed to around 0.5 probability"?

My understanding is when computing predicted probabilities, you'd need to add base score (default = 0.5) to estimated weight parameter (leaf score), like so:

$\hat{p} = \frac{\text{exp(0.5 + w)}}{\text{1 + exp(0.5 + w)}}$

where $\text{w}$ is the estimated leaf score.

Below, is the link to the default xgboost parameters in python API: https://xgboost.readthedocs.io/en/latest/python/python_api.html

class xgboost.XGBClassifier(max_depth=3,
      learning_rate=0.1, n_estimators=100, silent=True,
      objective='binary:logistic',
      booster='gbtree', n_jobs=1, nthread=None, gamma=0, min_child_weight=1,
      max_delta_step=0, subsample=1, colsample_bytree=1,
      colsample_bylevel=1,
      reg_alpha=0, reg_lambda=1, scale_pos_weight=1, base_score=0.5,
      random_state=0,
      seed=None, missing=None, **kwargs)

base_score:
The initial prediction score of all instances, global bias.

Does this answer your question?


Dense output from neural network I would like to create a loss function that encourages the output of the embedding of an autoencoder to be dense. I don't have an explicit condition for how density is defined, but one option would be that the average distance between a point and its k nearest neighbors should be minimized, where k is some parameter. This is balanced against the reconstruction error, so the embedded points won't all converge to the same value. Is there a way to construct a loss function of this, or something similar, in a way that allows the average density around each point to be backprogated through the autoencoder?

Very good question!
An approach towards your goal is Center loss. The loss rewards if points of a class gather around a center. You can adapt this approach for your Autoencoder.


How are 1x1 convolutions the same as a fully connected layer? I've recently read Yan LeCuns comment on 1x1 convolutions:

  In Convolutional Nets, there is no such thing as "fully-connected layers". There are only convolution layers with 1x1 convolution kernels and a full connection table.

  It's a too-rarely-understood fact that ConvNets don't need to have a fixed-size input. You can train them on inputs that happen to produce a single output vector (with no spatial extent), and then apply them to larger images. Instead of a single output vector, you then get a spatial map of output vectors. Each vector sees input windows at different locations on the input.
  In that scenario, the "fully connected layers" really act as 1x1 convolutions.

I would like to see a simple example for this.

Example

Assume you have a fully connected network. It has only an input layer and an
output layer. The input layer has 3 nodes, the output layer has 2 nodes. This
network has $3 \cdot 2 = 6$ parameters. To make it even more concrete, lets say you have a ReLU activation function in the output layer and the weight matrix

$$
\begin{align}
W &=
\begin{pmatrix}
 0 & 1 & 1\\
 2 & 3 & 5\\
\end{pmatrix} \in \mathbb{R}^{2 \times 3}\\
b &= \begin{pmatrix}8\\ 13\end{pmatrix} \in \mathbb{R}^2
\end{align}
$$

So the network is $f(x) = ReLU(W \cdot x + b)$ with $x \in \mathbb{R}^3$.

How would the convolutional layer have to look like to be the same? What does LeCun mean with "full connection table"?

I guess to get an equivalent CNN it would have to have exactly the same number of parameters. The MLP from above has $2 \cdot 3 + 2 = 8$ parameters.

Your Example

In your example we have 3 input and 2 output units. To apply convolutions, think of those units having shape: [1,1,3] and [1,1,2], respectively. In CNN terms, we have 3 input and 2 output feature maps, each having spatial dimensions 1 x 1.

Applying an n x n convolution to a layer with k feature maps, requires you to have a kernel of shape [n,n,k]. Hence the kernel of you 1x1 convolutions have shape [1, 1, 3]. You need 2 of those kernels (or filters) to produce the 2 output feature maps. Please Note: $1 \times 1$ convolutions really are  $1 \times 1 \times \text{number of channels of the input}$ convolutions. The last one is only rarely mentioned.

Indeed if you choose as kernels and bias:
$$
\begin{align}
w_1 &=
\begin{pmatrix}
 0 & 1 & 1\\
\end{pmatrix} \in \mathbb{R}^{3}\\
w_2 &=
\begin{pmatrix}
2 & 3 & 5\\
\end{pmatrix} \in \mathbb{R}^{3}\\
b &= \begin{pmatrix}8\\ 13\end{pmatrix} \in \mathbb{R}^2
\end{align}
$$

The conv-layer will then compute $f(x) = ReLU\left(\begin{pmatrix}w_1 \cdot x\\ w_2 \cdot x\end{pmatrix} + \begin{pmatrix}b_1\\ b_2\end{pmatrix}\right)$ with $x \in \mathbb{R}^3$.

Transformation in real Code

For a real-life example, also have a look at my vgg-fcn implementation. The Code provided in this file takes the VGG weights, but transforms every fully-connected layer into a convolutional layers. The resulting network yields the same output as vgg when applied to input image of shape [244,244,3]. (When applying both networks without padding).

The transformed convolutional layers are introduced in the function _fc_layer (line 145). They have kernel size 7x7 for FC6 (which is maximal, as pool5 of VGG outputs a feature map of shape [7,7, 512]. Layer FC7 and FC8 are implemented as 1x1 convolution.

"Full Connection Table"

I am not 100% sure, but he might refer to a filter/kernel which has the same dimension as the input feature map. In both cases (Code and your Example) the spatial dimensions are maximal in the sense, that the spatial dimension of the filter is the same as the spatial dimension as the input.


How to learn from multiple data sources with different input variables but the same underlying pattern? I will explain with an example:

Let's say you have 2 factories that produce pulp paper. Each have similar processes where the laws of physics give the same outcome. Now let's say this 2 factories have equipment and sensors from different manufacturers, so the output of those sensors is not comparable in any way (different number of variables, different metric system etc.). Although for both factories I can caculate the output easily and determine the learning target in a comparable way (eg. metric tonnes of paper).

Is there a way of using deep learning to learn from both datasets at the same time? I mean increase the predictive power upon a sample from factory 1 due to insights on factory 2?

What about having 3 DNN, 2 for reducing feature representation and standardizing output representation and the third one for learning the general pattern common to both and predicting the final output?

What you are referring to is multi-view learning. Multi-view learning basically tells us how multiple data sources or multiple feature subsets can be combined to create a more robust learning curve for the algorithm.

In recent years, starting from 2013 a lot of research has been carried in this rapidly growing field. A good introduction to the topic can be found in the link below. It contains a more theoretical and mathematical approach to understanding the method.

http://research.ics.aalto.fi/airc/reports/R1011/msml.pdf


issue with oneHotEncoding So i have a PandasDataFrame with categorical variables in a column which i want to one hot encode i've used the following code from an ML udemy course

from sklearn.preprocessing import OneHotEncoder
onehotencoder=OneHotEncoder(categorical_features=[10])
Y= onehotencoder.fit_transform(X).toarray()

However i get the following error

ValueError: could not convert string to float:

A bit of information Y is converted to an object from a df using

Y=df.iloc[:,:].values

I want to oneHotencode the 10th column of y which contains string values. The type of Y in variable explorer is object and if execute

type(Y)

i get numpy.ndarray

I'm new to Pandas and sklearn and would really appreciate any help.

There is an easy way to use one hot encoding in pandas and you can read about it in the following link:

https://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html

The get_dummies method is easy and optimized for use in pandas Data Frames.

Good luck!


Deep Neural Network - Backpropogation with ReLU I'm having some difficulty in deriving back propagation with ReLU, and I did some work, but I'm not sure if I'm on the right track.

Cost Function: $\frac{1}{2}(y-\hat y)^2$ where $y$ is the real value, and $\hat y$ is a predicted value. Also assume that $x$ > 0 always.

1 Layer ReLU, where the weight at the 1st layer is $w_1$

$\frac{dC}{dw_1}=\frac{dC}{dR}\frac{dR}{dw_1}$

$\frac{dC}{w_1}=(y-ReLU(w_1x))(x)$

2 Layer ReLU, where the weights at the 1st layer is $w_2$, and the 2nd layer is $w_1$ And I wanted to updated the 1st layer $w_2$

$\frac{dC}{dw_2}=\frac{dC}{dR}\frac{dR}{dw_2}$

$\frac{dC}{w_2}=(y-ReLU(w_1*ReLU(w_2x))(w_1x)$

Since $ReLU(w_1*ReLU(w_2x))=w_1w_2x$

3 Layer ReLU, where the weights at the 1st layer is $w_3$, 2nd layer $w_2$ and 3rd layer $w_1$

$\frac{dC}{dw_3}=\frac{dC}{dR}\frac{dR}{dw_3}$

$\frac{dC}{w_3}=(y-ReLU(w_1*ReLU(w_2(*ReLU(w_3)))(w_1w_2x)$

Since $ReLU(w_1*ReLU(w_2(*ReLU(w_3))=w_1w_2w_3x$

Since the chain rule only lasts with 2 derivatives, compared to a sigmoid, which could be as long as $n$ number of layers.

Say I wanted to updated all 3 layer weights, where $w_1$ is the 3rd layer, $w_2$ is the 2nd layer, $w_1$ is the 3rd layer

$\frac{dC}{w_1}=(y-ReLU(w_1x))(x)$

$\frac{dC}{w_2}=(y-ReLU(w_1*ReLU(w_2x))(w_1x)$

$\frac{dC}{w_3}=(y-ReLU(w_1*ReLU(w_2(*ReLU(w_3)))(w_1w_2x)$

If this derivation is correct, how does this prevent vanishing? Compared to sigmoid, where we have a lot of multiply by 0.25 in the equation, whereas ReLU does not have any constant value multiplication. If there's thousands of layers, there would be a lot of multiplication due to weights, then wouldn't this cause vanishing or exploding gradient?

Working definitions of ReLU function and its derivative:

$ReLU(x) = \begin{cases}
  0, & \text{if } x < 0, \\
  x, & \text{otherwise}.
\end{cases}$

$\frac{d}{dx} ReLU(x) = \begin{cases}
  0, & \text{if } x < 0, \\
  1, & \text{otherwise}.
\end{cases}$

The derivative is the unit step function. This does ignore a problem at $x=0$, where the gradient is not strictly defined, but that is not a practical concern for neural networks. With the above formula, the derivative at 0 is 1, but you could equally treat it as 0, or 0.5 with no real impact to neural network performance.

Simplified network

With those definitions, let's take a look at your example networks.

You are running regression with cost function $C = \frac{1}{2}(y-\hat{y})^2$. You have defined $R$ as the output of the artificial neuron, but you have not defined an input value. I'll add that for completeness - call it $z$, add some indexing by layer, and I prefer lower-case for the vectors and upper case for matrices, so $r^{(1)}$ output of the first layer, $z^{(1)}$ for its input and $W^{(0)}$ for the weight connecting the neuron to its input $x$ (in a larger network, that might connect to a deeper $r$ value instead). I have also adjusted the index number for the weight matrix - why that is will become clearer for the larger network. NB I am ignoring having more than neuron in each layer for now.

Looking at your simple 1 layer, 1 neuron network, the feed-forward equations are:

$z^{(1)} = W^{(0)}x$

$\hat{y} = r^{(1)} = ReLU(z^{(1)})$

The derivative of the cost function w.r.t. an example estimate is:

$\frac{\partial C}{\partial \hat{y}} = \frac{\partial C}{\partial r^{(1)}} = \frac{\partial}{\partial r^{(1)}}\frac{1}{2}(y-r^{(1)})^2 = \frac{1}{2}\frac{\partial}{\partial r^{(1)}}(y^2 - 2yr^{(1)} + (r^{(1)})^2) = r^{(1)} - y$

Using the chain rule for back propagation to the pre-transform ($z$) value:

$\frac{\partial C}{\partial z^{(1)}} = \frac{\partial C}{\partial r^{(1)}} \frac{\partial r^{(1)}}{\partial z^{(1)}} = (r^{(1)} - y)Step(z^{(1)}) = (ReLU(z^{(1)}) - y)Step(z^{(1)})$

This $\frac{\partial C}{\partial z^{(1)}}$ is an interim stage and critical part of backprop linking steps together. Derivations often skip this part because clever combinations of cost function and output layer mean that it is simplified. Here it is not.

To get the gradient with respect to the weight $W^{(0)}$, then it is another iteration of the chain rule:

$\frac{\partial C}{\partial W^{(0)}} = \frac{\partial C}{\partial z^{(1)}} \frac{\partial z^{(1)}}{\partial W^{(0)}} = (ReLU(z^{(1)}) - y)Step(z^{(1)})x = (ReLU(W^{(0)}x) - y)Step(W^{(0)}x)x$

. . . because $z^{(1)} = W^{(0)}x$ therefore $\frac{\partial z^{(1)}}{\partial W^{(0)}} = x$

That is the full solution for your simplest network.

However, in a layered network, you also need to carry the same logic down to the next layer. Also, you typically have more than one neuron in a layer.

More general ReLU network

If we add in more generic terms, then we can work with two arbitrary layers. Call them Layer $(k)$ indexed by $i$, and Layer $(k+1)$ indexed by $j$. The weights are now a matrix. So our feed-forward equations look like this:

$z^{(k+1)}_j = \sum_{\forall i} W^{(k)}_{ij}r^{(k)}_i$

$r^{(k+1)}_j = ReLU(z^{(k+1)}_j)$

In the output layer, then the initial gradient w.r.t. $r^{output}_j$ is still  $r^{output}_j - y_j$. However, ignore that for now, and look at the generic way to back propagate, assuming we have already found $\frac{\partial C}{\partial r^{(k+1)}_j}$ - just note that this is ultimately where we get the output cost function gradients from. Then there are 3 equations we can write out following the chain rule:

First we need to get to the neuron input before applying ReLU:

$\frac{\partial C}{\partial z^{(k+1)}_j} = \frac{\partial C}{\partial r^{(k+1)}_j} \frac{\partial r^{(k+1)}_j}{\partial z^{(k+1)}_j} = \frac{\partial C}{\partial r^{(k+1)}_j}Step(z^{(k+1)}_j)$

We also need to propagate the gradient to previous layers, which involves summing up all connected influences to each neuron:

$\frac{\partial C}{\partial r^{(k)}_i} = \sum_{\forall j} \frac{\partial C}{\partial z^{(k+1)}_j} \frac{\partial z^{(k+1)}_j}{\partial r^{(k)}_i} = \sum_{\forall j} \frac{\partial C}{\partial z^{(k+1)}_j} W^{(k)}_{ij}$

And we need to connect this to the weights matrix in order to make adjustments later:

$\frac{\partial C}{\partial W^{(k)}_{ij}} = \frac{\partial C}{\partial z^{(k+1)}_j} \frac{\partial z^{(k+1)}_j}{\partial W^{(k)}_{ij}} = \frac{\partial C}{\partial z^{(k+1)}_j} r^{(k)}_{i}$

You can resolve these further (by substituting in previous values), or combine them (often steps 1 and 2 are combined to relate pre-transform gradients layer by layer). However the above is the most general form. You can also substitute the $Step(z^{(k+1)}_j)$ in equation 1 for whatever the derivative function is of your current activation function - this is the only place where it affects the calculations.

Back to your questions:

  If this derivation is correct, how does this prevent vanishing?

Your derivation was not correct. However, that does not completely address your concerns.

The difference between using sigmoid versus ReLU is just in the step function compared to e.g. sigmoid's $y(1-y)$, applied once per layer. As you can see from the generic layer-by-layer equations above, the gradient of the transfer function appears in one place only. The sigmoid's best case derivative adds a factor of 0.25 (when $x = 0, y = 0.5$), and it gets worse than that and saturates quickly to near zero derivative away from $x=0$. The ReLU's gradient is either 0 or 1, and in a healthy network will be 1 often enough to have less gradient loss during backpropagation. This is not guaranteed, but experiments show that ReLU has good performance in deep networks.

  If there's thousands of layers, there would be a lot of multiplication due to weights, then wouldn't this cause vanishing or exploding gradient?

Yes this can have an impact too. This can be a problem regardless of transfer function choice. In some combinations, ReLU may help keep exploding gradients under control too, because it does not saturate (so large weight norms will tend to be poor direct solutions and an optimiser is unlikely to move towards them). However, this is not guaranteed.


How to find similar points to a positive set when you don't have any negative set? The task I'm used to do is the following. A client comes to see me with a set of clients (called positive companies) and he wants me to find other similar prospects. Usually, he also gives me a set of negatives companies and I have a big set of potential companies (that I call the basket).

I perform this task by doing a Adaboost classifier that I train with the positives and negatives. I then run this classifier on the basket. Each company in the basket receives a score and the highest score shows the most promising prospects for the client.

Now, a new client doesn't have any set of negatives to give and I'm a little bit lost. I can not do a supervised learning anymore, obviously. I first thought of performing a k-nearest neighbours on each positive and I would receive a list of "close" prospects. The problem with that is that I don't have a score anymore.
Furthermore, with the k-nearest method, I should define a distance which I don't like because I don't want to give subjective weights to features. Indeed, the Adaboost classifier would learn some weights and would itself predict which features are important.

Could someone indicate me how I could tackle this problem?

To summarize, you have labeled data in one class (positive) and unlabeled data. You want to find the positive examples in the unlabeled data. The general name for this setting in machine learing is one-class classification, which is a fairly broad field.

A sub-area that is particularly relevant is positive-unlabeled learning, which is the problem of training a classifier when one has just positive and unlabeled data.

Also note that you have all the examples that need to be predicted at training time. Therefore you can use a transductive learning algorithm. Particularly, if you have a notion of which companies are similar, you could construct a graph by connecting similar companies by edges. You could then run a graph propagation algorithm that would assign scores to the unlabeled items.

Finally, here is a similar question where the answer suggests a method of positive-unlabeled learning.


Why don't we find the analytical function of the cost function? Then we could derive it and find minimum(s). e.g. in small networks the cost function has not so many variables.

The power of networks come from hidden layers with non-linear activation functions. Said non-linear activation function make the calculation of an analytical solution impossible (except maybe for some very specific cases that are not really usefull).
If you need more convincing look at logistic regression : a one-neuron network with sigmoid activation which does not have an analytical solution.
As to why there is no closed form solution, you need to understand that the linear regression you gave in exemple is a very simple problem of projecting a 'point' on a 'plane'. Using an activation function on one neuron is equivalent to projecting a point on a complex manifold, which do not yield an analytical solution (except, again for some trivial cases). For neurons in succesions I am not even sure there is a geometrical interpretation.


CV Pipeline for Time Series with Differing shape for X and y I am playing around with ML models to forecast a time series. I'd like to generate a sklearn pipeline like

from sklearn.preprocessing import FunctionTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler

pipe = Pipeline(
    steps=[
        ("gen", FunctionTransformer(generate_features)),
        ("imp_nan", SimpleImputer(strategy="median")),
        ("scale", StandardScaler()),
        ("reg", keras.wrappers.scikit_learn.KerasRegressor(build_model, epochs=35))
    ]
)

Where generate_features takes in my data X and produces a collection of features for a given timestamp. For example a 5 day rolling standard deviation. However, my target y could have a different shape.

For example, I may want to forecast 5 hours in the future (and so my target will have a date time index with period 5 hours), but my input data could have a frequency of 1 min (and so initially an index with period 1 min that gets transformed into a period of 5 hours). This leads to issues when trying to cross validate. For example, if I use

splitter = TimeSeriesSplit(n_splits=3)
grid = GridSearchCV(pipe, {"gen__kw_args": [{"windows": (2,4,6)},]}, cv=splitter)
grid.fit(X_train, y_train)

I immediately get an error telling me the shape of X_train and y_train are not the same. This makes sense because the splitter needs to be used on both X and y.

I want to do this because I'd like to perform a grid cv on the parameters of my generate_features function (as well as the parameters of my model...)

My Question

Is there a nice way to create a pipeline and perform cross validation on time series feature generation when our input shape of X could be different to our input shape of y?

Edit: Here is the traceback

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-11-f0c6e65ecbb8> in <module>
      1 splitter = TimeSeriesSplit(n_splits=3)
      2 grid = GridSearchCV(pipe, {"gen__kw_args": [{"windows": (2,4,6)},]}, cv=splitter)
----> 3 grid.fit(X_train, y_train)

~/.local/lib/python3.8/site-packages/sklearn/model_selection/_search.py in fit(self, X, y, groups, **fit_params)
    648             refit_metric = 'score'
    649
--> 650         X, y, groups = indexable(X, y, groups)
    651         fit_params = _check_fit_params(X, fit_params)
    652

~/.local/lib/python3.8/site-packages/sklearn/utils/validation.py in indexable(*iterables)
    246     """
    247     result = [_make_indexable(X) for X in iterables]
--> 248     check_consistent_length(*result)
    249     return result
    250

~/.local/lib/python3.8/site-packages/sklearn/utils/validation.py in check_consistent_length(*arrays)
    209     uniques = np.unique(lengths)
    210     if len(uniques) > 1:
--> 211         raise ValueError("Found input variables with inconsistent numbers of"
    212                          " samples: %r" % [int(l) for l in lengths])
    213

ValueError: Found input variables with inconsistent numbers of samples: [522323, 52417]

I've done a bit of searching and have actually found a solution to this using tsfresh.

You can find the sklearn transformers here:
https://tsfresh.readthedocs.io/en/latest/text/sklearn_transformers.html

Here is the code snippet from the example.

pipeline = Pipeline(
    [
        ('augmenter', RelevantFeatureAugmenter(column_id='id', column_sort='time')),
        ('classifier', RandomForestClassifier())
    ]
)

df_ts, y = load_robot_execution_failures()
X = pd.DataFrame(index=y.index)

pipeline.set_params(augmenter__timeseries_container=df_ts)
pipeline.fit(X, y)

We pass an empty dataframe to pipeline.fit and specify the input data as a parameter to the tsfresh transformer. df_ts and y have a totally different shape!

:-)


Why do train, test, validation datasets need to have the same distribution? I've found a lot of martial on how to deal with differently distributed train/test/validation data sets, however I'm struggling to find out why they they need to have the same distribution.

Can anyone explain to me in detail why these data sets need to have the same/similar distributions? or point me toward resources that explain it?

You can think of the data distribution as the world that the model lives in. You want to train it to perform well in this world and to do this you have to train it on examples representing this world. Further you have to estimate its performance in this world by testing it on more examples from this world. If you train your model in one world, test and validate on another and deploy it into a third, you have no idea how well the model was trained for the real world or if it was trained for the real world at all.

Let's look at a little example. I created a little toy data set and ran regression with the same network, same optimizer and everything the same, except the initialization.

All 5 models give the same prediction in the domain where there is data (they even over-fit a little bit in the same way). But for x-values far away from the training data, we get widely different predictions from different models.

So what does this tell us? It tells us that our models are basically worthless for inputs that are "far" from the training data. "Far" means "from a different distribution" in this sense. But if our models are worthless for inputs that do not come from the original distribution, then any quality measures based on test and validation datasets are also worthless.

I used the word "far" a bit loosely here, basically as if I was talking about a distance. I think this is somewhat correct, but it's not the whole story. Even if the support of two distributions is the same, they can be "far" from each other if one distribution puts a lot of weight on events that the other only weighs little.

This is also a problem, because this would mean you get many samples from one distribution where the other gives only a few. The amount of training data you have for a particular input domain will influence how well your model generalizes to this domain. If now in the test or validation set you shift the bulk of the distribution, you basically over-emphasize input values that the model has hardly been trained for, while at the same time hardly testing it in domains that it has been trained for well. So again your measure of model performance does not really tell you how well your training worked.

This is why training, test and validation dataset should stem from the same distribution, but also the inputs once the model is deployed need to be from the same distribution, otherwise your quality measures and your predictions will be highly inaccurate.


Where to find state-of-the-art time series data deep learning approaches? Just like this (which stopped updating) that shows the newest deep learning methods utilized in image recognition, is there some similar lists/summaries (in blog posts/paper/competitions etc.) about newest deep learning methods used in time series analysis?

There is an archive of papers related to time series analysis at Arxiv Sanity Preserver which you may find useful. Among the many papers listed there are some which deal with deep learning approaches to time series.


Transformer architecture question I am hand-coding a transformer (https://arxiv.org/pdf/1706.03762.pdf) based primarily on the instructions I found at this blog: http://jalammar.github.io/illustrated-transformer/.
The first attention block takes matrix input of the shape [words, input dimension] and multiplies by the attention weight matrices of shape [input dimension, model dimension]. The model dimension is chosen to be less than the input dimension and is the dimension used as output in all subsequent steps.
There is a residual connection around the attention block and the input is meant to be added to the output of the attention block. However the output of the attention block is shape [words, model dimension] and the input is form [words, input dimension]. Should I interpolate the input down to the model dimension as is done in ResNet? Or maybe add another weight matrix to transform the input?

The input dimensionality is the embedding size, which is the same as the model dimensionality, as explained in section 3.4 of the article:

3.4 Embeddings and Softmax
Similarly to other sequence transduction models, we use learned embeddings to convert the input tokens and output tokens to vectors of dimension $d_{model}$.

Therefore, the input dimensionality and the model dimensionality are the same, which makes them suitable for the residual connection.


How to select between models when AUC scores are similar? I use two machine learning algorithms for binary classification and I get this result :

Algo 1 :

 AUC- Train : 0.75      AUC- Test: 0.65          big Train / overfitting

Algo 2 :

 AUC- Train : 0.72      AUC- Test: 0.65          small train / small overfitting

Which one is better?

Based on the AUC score they are the same. It does not really matter if the model is overfitting or not. What matters is how well it performs on new data (test score).

Overfitting is just an indication that there might be room for improvement by making your model more general. But until the test score has increased the model has not improved even if it is overfitting less.