Multilingual Bert sentence vector captures language used more than meaning - working as interned? Playing around with BERT, I downloaded the Huggingface Multilingual Bert and entered three sentences, saving their sentence vectors (the embedding of [CLS]), then translated them via Google Translate, passed them through the model and saved their sentence vectors.

I then compared the results using cosine similarity.

I was surprised to see that each sentence vector was pretty far from the one generated from the sentence translated from it (0.15-0.27 cosine distance) while different sentences from the same language were quite close indeed (0.02-0.04 cosine distance).

So instead of having sentences of similar meaning (but different languages) grouped together (in 768 dimensional space ;) ), dissimilar sentences of the same language are closer.

To my understanding the whole point of Multilingual Bert is inter-language transfer learning - for example training a model (say, and FC net) on representations in one language and having that model be readily used in other languages.

How can that work if sentences (of different languages) of the exact meaning are mapped to be more apart than dissimilar sentences of the same language?

My code:

import torch

import transformers
from transformers import AutoModel,AutoTokenizer

bert_name="bert-base-multilingual-cased"
tokenizer = AutoTokenizer.from_pretrained(bert_name)
MBERT = AutoModel.from_pretrained(bert_name)

#Some silly sentences
eng1='A cat jumped from the trees and startled the tourists'
e=tokenizer.encode(eng1, add_special_tokens=True)
ans_eng1=MBERT(torch.tensor([e]))

eng2='A small snake whispered secrets to large cats'
t=tokenizer.tokenize(eng2)
e=tokenizer.encode(eng2, add_special_tokens=True)
ans_eng2=MBERT(torch.tensor([e]))

eng3='A tiger sprinted from the bushes and frightened the guests'
e=tokenizer.encode(eng3, add_special_tokens=True)
ans_eng3=MBERT(torch.tensor([e]))

# Translated to Hebrew with Google Translate
heb1='חתול קפץ מהעץ והבהיל את התיירים'
e=tokenizer.encode(heb1, add_special_tokens=True)
ans_heb1=MBERT(torch.tensor([e]))

heb2='נחש קטן לחש סודות לחתולים גדולים'
e=tokenizer.encode(heb2, add_special_tokens=True)
ans_heb2=MBERT(torch.tensor([e]))

heb3='נמר רץ מהשיחים והפחיד את האורחים'
e=tokenizer.encode(heb3, add_special_tokens=True)
ans_heb3=MBERT(torch.tensor([e]))

from scipy import spatial
import numpy as np

# Compare Sentence Embeddings

result = spatial.distance.cosine(ans_eng1[1].data.numpy(), ans_heb1[1].data.numpy())

print ('Eng1-Heb1 - Translated sentences',result)

result = spatial.distance.cosine(ans_eng2[1].data.numpy(), ans_heb2[1].data.numpy())

print ('Eng2-Heb2 - Translated sentences',result)

result = spatial.distance.cosine(ans_eng3[1].data.numpy(), ans_heb3[1].data.numpy())

print ('Eng3-Heb3 - Translated sentences',result)

print ("\n---\n")

result = spatial.distance.cosine(ans_heb1[1].data.numpy(), ans_heb2[1].data.numpy())

print ('Heb1-Heb2 - Different sentences',result)

result = spatial.distance.cosine(ans_eng1[1].data.numpy(), ans_eng2[1].data.numpy())

print ('Heb1-Heb3 - Similiar sentences',result)

print ("\n---\n")

result = spatial.distance.cosine(ans_eng1[1].data.numpy(), ans_eng2[1].data.numpy())

print ('Eng1-Eng2 - Different sentences',result)

result = spatial.distance.cosine(ans_eng1[1].data.numpy(), ans_eng3[1].data.numpy())

print ('Eng1-Eng3 - Similiar sentences',result)

#Output:
"""
Eng1-Heb1 - Translated sentences 0.2074061632156372
Eng2-Heb2 - Translated sentences 0.15557605028152466
Eng3-Heb3 - Translated sentences 0.275478720664978

---

Heb1-Heb2 - Different sentences 0.044616520404815674
Heb1-Heb3 - Similar sentences 0.027982771396636963

---

Eng1-Eng2 - Different sentences 0.027982771396636963
Eng1-Eng3 - Similar sentences 0.024596810340881348
"""

P.S.

At least the Heb1 was closer to Heb3 than to Heb2.
This was also observed for the English equivalents, but less so.

N.B.
Originally asked on Stack Overflow, here

Sadly, I don't think that Multilingual BERT is the magic bullet that you hoped for.
As you can see in Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT (Wu and Dredze, 2019), the mBERT was not trained with any explicit cross-lingual task (for example, predicting a sentence from one language given a sentence from another language).
Rather, it was trained using sentences from Wikipedia in multiple languages, forcing the network to account for multiple languages but not to make the connections between them.
In other words, the model is trained with predicting a masked instance of 'cat' as 'cat' given the rest of the (unmasked) sentence, and predicting a foreign word meaning 'cat' in a masked space given a sentence in that language. This setup is does not push the model towards making the connection.
You might want to have a look in Facebook's LASER, which was explicitly trained to match sentences from different languages.
P.S
The fact that the sentences do not have the same representation does not mean that the mBERT cannot be used for zero-shot transfer learning across languages. Again, please see Wu and Dredze


No gradients provided for any variable I have composed a customized loss function (kl_loss):

def tensor_pValue(pnls,pnl):
    vec=tf.contrib.framework.sort(pnls,axis=-1,direction='ASCENDING')
    rank_p=tf.divide(tf.range(0,264.5,1),264.0)
    return tf.gather(rank_p, tf.searchsorted(vec,pnl,side='left'))

def kl_divergence(p, q):
    epsilon = 0.00001
    p=p+epsilon
    q=q+epsilon
    return tf.reduce_sum(p * tf.log(p/q))

def kl_loss(predicted_pnL,actual_pnl_tensor):
    p_dist=tf.squeeze(tf.map_fn(lambda inp:tensor_pValue(inp[0],inp[1]),(predicted_pnL,actual_pnl_tensor),dtype=tf.float32))
    u_dist=tf.random.uniform([264],0,1,dtype=tf.float32)
    return kl_divergence(p_dist,u_dist)

And then i constructed a simple net work using Keras:

optimizer = tf.train.AdamOptimizer(0.001)
input_dim = X_train.shape[1]
model = keras.Sequential([
keras.layers.Dense(UNITS, activation=tf.nn.relu,
             input_dim=input_dim),
keras.layers.Dense(UNITS, activation=tf.nn.relu),
keras.layers.Dense(264)
])
model.compile(loss=lambda y, f: kl_loss(f,y), optimizer=optimizer)
model.fit(X_train, train_y, epochs=EPOCHS, batch_size=BATCH_SIZE,verbose=0)

And got following errors:

ValueError: No gradients provided for any variable: ["", "", "", "", "", ""].

Can anyone help to take a look on where might be wrong on this? Thank you very much!

If you are using a default KL divergence loss, I recommend using an implemented one: tf.keras.losses.KLDivergence.

If it is the problem to use Keras from TF, implement it as they do: https://github.com/tensorflow/tensorflow/blob/3699977134badccb7032fa6921d70e01ba8fdf7d/tensorflow/python/keras/losses.py#L978


Kmeans cluster validation when I have labeled test data I'm trying to implement the unsupervised k-means algorithm for sentiment analysis of imdb movie dataset created by stanford.
The steps that I followed is :
1) Load the comments
2) Apply tokenization and stemmetion ,use tf-idf algo to create tfidf matrix.
3) Use k-means algo to divide the data into 2 clusters.

My problem is how do I validate the the clusters
I have labeled test data. I want to check if all the negative examples go in one cluster and all the positive examples go in another cluster.

For this I used the scikits predict() command. But the parameters that need to be provided to the predict() command should be same as the parameters of fit() command. But as tfidf matrix that is passed has the dimensions dependent on the data passed.

In order to keep the same dimension for both training & testing set, you need apply your fit on the training set with you tokenization, tf-idf etc..
You need to apply same treatments as you do on training, on the testing set and then this will be okay :

from sklearn.feature_extraction.text import CountVectorizer

corpus_train = [
    'This is the first document.',
    'This document is the second document.'
]
corpus_test = [
    'And this is the third one.',
    'Is this the first document?',
]

vectorizer = CountVectorizer()
X_train = vectorizer.fit_transform(corpus_train)
print(vectorizer.get_feature_names())

# Returns
['document', 'first', 'is', 'second', 'the', 'this']

So your training set looks like this :

print(X_train.toarray())
# Returns
[[1 1 1 0 1 1]
 [2 0 1 1 1 1]]

print(X_train.shape)
# Returns
(2, 6)

Then just transform your testing set with the same vectorizer :

X_test = vectorizer.transform(corpus_test)
print(X_test.toarray())
# Returns
[[0 0 1 0 1 1]
[1 1 1 0 1 1]]

print(X_test.shape)
# Returns
(2, 6)

So you get your dimension of 6 (the number of features) for both training & testing set :)


Manual feature engineering based on the output So, I'm working on a ML model that would have as potential predictors : age , a code for his city , his social status ( married / single and so on ) , number of his children and the output signed which is binary ( 0 or 1 ). Thats the initial dataset I have.

My prediction would be based on those features to predict the value of signed for that person.

I already generated a prediction on unseen data. Upon validation of the results with the predicted result vs the real data , I have 25% accuracy. While cross-validation gives me 65% accuracy. So I thought : Over-fitting

Here is my question, I went back to the early stages of the whole process, and started creating new features. Example : Instead of the code for the city which makes no sense to have that as input to a ML model, I created classes based on the percentage of signed, the city ,with the higher percentage of 'signed' ( the ouput ) , gets assigned to a higher value of class_city, which improved a lot in my Correlation Matrix the relationship signed-class_city which makes sense. Is what I'm doing correct? or shouldn't I create features based on the ouput?
Here is my CM
:

After re-modelling with 3 features only ( department_class , age and situation ) i tested my model on unseen data made of 148 rows compared to 60k rows in the training file.

First model with the old feature ( the ID of the departement ) gave 25% accuracy while the second model with the new feature class_department gave 71% ( Again on unseen data )

Note : First model with 25% has some other features as ID's ( they might be causing the model to have such a weak accuracy with the deparment_ID )

No you should not do this, it is causing a data leak. Data leaks happen when the data you are using to train a machine learning algorithm happens to have the information you are trying to predict.

It will give your model information about your test data during training. It will cause your test scores to be overly optimistic and will make the model worse at generalizing to totally unseen data.

Good resource on data leakage


Classification Threshold Tuning with GridSearchCV In Scikit-learn, GridSearchCV can be used to validate a model against a grid of parameters. A short example for grid-search cv against some of DecisionTreeClassifier parameters is given as follows:
model = DecisionTreeClassifier()
params = [{'criterion':["gini","entropy"],"max_depth":[1,2,3,4,5,6,7,8,9,10],"class_weight":["balanced"]}]
GSCV = GridSearchCV(model,params,scoring="f1_micro")
GSCV.fit(X_train,y_train)
GSCV.best_params_

Now, I am only concerned with binary classification. It is the case for many algorithms that they compute a probability score, and set the decision threshold at 0.5. My question is the following: If I want to consider the decision threshold as another parameter of the grid search (along with the existing parameters), is there a standard way to do this with GridSearchCV? For instance, something like the last parameter "decision_threshold" in the following tunegrid would be ideal:
params = [{'criterion':["gini","entropy"],"max_depth":[1,2,3,4,5,6,7,8,9,10],"class_weight":["balanced"], "decision_threshold": [0.1,0.2,...,0.9]}]

Needless to say, I am not interested in a particular solution working only for DecisionTreeClassifier. Instead,  a general solution  for any classifier that uses a probability decision threshold that can be tuned. Preferably, I would like to keep my current GridSearchCV  if possible.

As far as I know you cannot add the model's threshold as hyperparameter but in order to find the optimal threshold you can do as follows:

make a the standard GridSearchCV but use the roc_auc as metric as per step 2
 model = DecisionTreeClassifier()
 params = [{'criterion':["gini","entropy"],"max_depth":[1,2,3,4,5,6,7,8,9,10],"class_weight":["balanced"]}]
 GSCV = GridSearchCV(model,params,scoring="roc_auc")
 GSCV.fit(X_train,y_train)
 GSCV.best_params_
 best_model = GSCV.best_estimator_

Once you have the best hyper parameters set you can obtain the threshold that maximizes the roc curve as follows:
 from sklearn.metrics import roc_curve
 preds = best_model.predict_proba(X_train)[:,1]

 fpr, tpr, thresholds = roc_curve(y_train, preds)
 optimal_idx = np.argmax(tpr - fpr)
 optimal_threshold = thresholds[optimal_idx]

This threshold will give you the lowest false positive rate and the highest true positive rate


Can I use the Softmax function with a binary classification in deep learning? I want to create a deep learning model (CNN) for binary classification, can I used the softmax function instead of the sigmoid function in binary classification?

Adding the classification layer to the model, will be like this

model.add(Dense(2, activation='softmax'))

For binary classification, it should give the same results, because softmax is a generalization of sigmoid for a larger number of classes.


Hierarchical clustering with the consensus matrix as similarity matrix I'm following this article on consensus clustering in Python programming. On page 7 the authors state that "The consensus matrix lends itself naturally to be used as a visualization tool to help assess the clusters’ composition andnumber. In particular, if we associate a color gradient to the 0–1 range of real numbers, so that white corresponds to 0, and dark red corresponds to 1, and if we assume the matrix is arranged so that items belonging to the same cluster are adjacent to each other (with the same item order used to index both the rows and the columns of the matrix), a matrix corresponding to perfect consensus will be displayed as a color-coded heat map characterized by red blocks along the diagonal, on a white background.", see the image below.

The consensus matrix itself is a (N × N) matrix that stores, for each pair of items, the proportion of clustering runs in which two items are clustered together. The consensus matrix is obtained by taking the average over the connectivity matrices of every perturbed dataset. To go from a consensus matrix to the visualization above, the authors say the following: "We can use the consensus matrix itself to determine the optimal item order. In particular, if we carry out hierarchical clustering with the consensus matrix as similarity matrix, the induced dendogram will have its leaves arranged so as to have items with highest consensus index adjacent to each other, thus maximizing the block-diagonal nature of the heat map ordered accordingly (it is important to emphasize again that the same item order is used to index both the rows and the columns of the matrix)."
I have my consensus matrix after 10 times sampling and clustering my dataset with 500 items. The consensus matrix looks like:

Where a value of 1 or 0 indicates perfect agreement/consensus on the clustering results of the 10 times sampling. However, I fail to understand how we end up with the visualization above. I have a few questions:

The authors state that we use the consensus matrix as a similarity matrix, but doesn't hierarchical clustering need a distance matrix? For example in the Python sklearn implementation of AgglomerativeClustering it says: "If “precomputed”, a distance matrix (instead of a similarity matrix) is needed as input for the fit method."
What do the authors mean with "it is important to emphasize again that the same item order is used to index both the rows and the columns of the matrix"?

To address your two questions:

Agglomerative clustering requires a distance metric, but you can compute this from your consensus-similarity matrix. The most basic way, is to do this:
distance_matrix = 1 / similarity matrix
Although, they may explicitly state in the paper what function they use for this transformation.

I think this is just to say that the matrix is symmetric. The x-axis of the heatmap will be n=0,1,2,3,4 and the y-axis of the heatmap will be n=0,1,2,3,4. This is the same procedure as for a correlation matrix. Just keep your matrix as is, and it will keep that order.


Class leaking on validation set I am quite new in the ML field. I think I correctly understood the information leaking problem during the testing/validation phases but I am struggling to understand some François Chollet statements that I will now report. This discussion will be based on Keras only for practical reasons.

First of all: Keras initially supported various metrics now removed, like for example precision, recall, f1, fbeta. As stated here they were removed only for a clarity reason: they were calculated as local metrics and so they were (almost) meaningless.
Those metrics clearly used the labels/classes to be computed. However in that post nobody complained about this problem
Now comes the hard part: Keras allows to specify both class and sample weights during the training (with the class_weight and sample_weight arguments). These both influence the training loss and the metrics listed in weighted_metrics.
However, class_weight does not influence the metrics listed in weighted_metrics during the validation nor can be used in the evaluate method, where it is only possibile to specify the sample_weight argument but not the class_weight one.

François Chollet clearly explained this design choice (quote):

  You'll notice that evaluate has a sample_weight argument but no class_weight argument. Validation evaluation in fit follows the same pattern. The reason is that class weighting is meant to help training (by modulating contributions of different classes), but not to affect evaluation. Evaluation, because it reflects production constraints, implies that labels are not available when making predictions, and thus can't be used to modulate evaluation metrics. (link)

  Additionally I have noticed that in the case where validation_split is used, class_weight is applied to the validation data, which is not the expected behavior (classes are of course supposed unknown for the validation data). I am also fixing this. (link)

  I mean that weighting by classes in a test setup is a form of label information leak, because it cannot be reproduced in a real test situation (when you don't have the labels, such as when using model.predict()). It's acceptable during training, by definition, but should not be done at testing time. (link)

I cannot understand why this is the case during the validation phase:

A class leaking like this is not such a serious problem: there is no backprop during the validation phase and so the potential leaking is quite limited. In addition I think this is not more serious than the standard leaking due to the continuous model tuning on the same validation set
Without using the classes, during the validation we could not compute interesting quantities such as the precision, the recall, the confusion matrix and so on. Computing them on the training set would produce a biased estimate.
Using the classes info would be thus extremely useful in order to obtain good estimates.
In addition, as stated before, those metrics were available and nobody complained about them.

I know we could obtain a similar effect by playing with the sample_weight argument but I am more interested in understanding this problem.
I know that the classes will not be available (theoretically) during the testing phase but I think that, during the development and tuning, these information and statistics could help better understanding the errors done by the model.

You could calculate these metrics and then attempt to clarify how your model is failing during validation, but then you're effectively turning your simulated test environment into a second round of training. Any further tuning decisions made from calculating these metrics defeat the purpose of using the validation set as a chance to prove your model's ability to generalize.


Dealing with a dataset having target values on different scales? I am currently working on a dataset having 10 features and one continuous target variable. One of the features is 'Country' , in which there are seven unique values [Argentina ,Denmark , France...etc].

Now , the continuous target variable is sales of a given product in that particular country for a given month in a given year.

It has been given in the problem statement , that the Sales have been taken in the local currency of that country , so now I have values on different scales and I am not sure how to deal with them.

When I grouped the data as per the different countries , (using pandas's groupby function) , I got at least 1000 observations for each country. So maybe I could train a model separately for each country ? All kind of approaches will be appreciated.

TL:DR The most common and straight-forward approach would be to scale all the numerical data to be within a given range. This makes the currency differences irrelevant, as all fluctation remains constant, relative to the original scale.

Rescaling

Here is an example function to scale the numerical columns of your dataset:

def rescale(data, new_min=-1, new_max=1):
    """Rescale the columns of Dataframe to be in the range [new_min, new_max].

    Parameters
    ----------
    data    : a Pandas DataFrame
    new_min : the target minimum for each of the columns, optional
    new_max : the target maximum for each of the columns, optional

    Returns
    -------
    out : the rescaled input data, with each column now in the range [new_min, new_max]

    """
    return (data - data.min()) / (data.max() - data.min()) * (new_max - new_min) + new_min

You can then do the following to see nice descriptions of each column in your dataframe, ensuring the min/max values are as desired:

your_dataframe.describe()

One thing to perhaps try out, would be whether or not to scale **all* data to be within a fixed range, say [0, 1], or whether to scale each individual currency to the range [0, 1]. That equates to applying the above function on either single columns of your dataframe, or the entire dataframe.

Considerations

Things to keep in mind/check for would be:

if some currencies are nominal huge compared to others (e.g. 1 euro = 17,000 Indonesian Rupiah), the smaller currencies will have time-series values all very close to zero - this makes computation/optimisation more difficult - especially if you use methods such as gradient descent.
Scaling the currencies individually to a range would lose the relativity of their nominal values. To see this, plot the currencies before and after scaling.

Unify the Currencies

If you have (or could get) exchange rate the data, another alternative would be to convert all of the currencies to the one one your choice.

As the exchange rate changes over time, this would also inherently introduce new information into the dataset, namely the relative economic health of each of the countries. Depending on your use case, this could be something that really helps, or you might really want to avoid. If you have to time, to do it, it could definitely make for some interestinf research!

You can get currency data from Quandl - they have a Python API too, which is simple enough to use.


ETL and Data Engineering - is it purely the knowledge of tools or is there theory behind it? I would like to better understand what a good Data Englineer must know or what he does. Job descriptions mostly list tools that are required, such as Python.
If it is possible to separate Data Engineering from Data Science, on what principles is Data Engineering based, what is the result of the Data Engineering? Is it creating some data structures? If so, what these structures might be? Are there standards or best practices?

First of all I just want to say that I am not a data engineer and there is definitely someone out there that can answer this better than me.
I do think that there is a lot of theory behind data engineering. It is also very interesting. I too thought that it was boring and I was more interested in just data science/ machine learning. I am not sure if I can say exactly what principles data engineering is based on but it is about how to best store data, access data and creating underlying systems for more efficient computing. The first paper I read that really got me interested in this stuff was the original paper for Spark.
I also just did a quick google for data engineering PhD and came across this. There is a lot of interesting new research going on with how to store data using "nano-structures".  There's also an area of research in quantum databases, which seems like a really interesting database abstraction.
I would be interested in hearing a more informed and complete answer from someone else who is in this field! In fact it might be useful to post this question on another stack exchange site.


Can we use two independently measured features in a same ML model? Two features are measured at different times but belong to the same target. In which conditions or form these features can be modeled together? Or they shouldn't be used in the same model but modeled separately? E.g. I am trying to identify users by looking at their mouse dynamics data and keyboard dynamics data, however, these two are measured independently but they belong to the same person (I have this scenario for many users). To explain it better, imagine it like I gave a text to users to repeatedly write them using the keyboard (and I recorded their keyboard usage) and similarly I gave them a different task to use mouse repeatedly (and I recorded that too). The sample size is also different for each feature.

From the example I assume that an instance corresponds to a user, and you have both full sequences of the mouse and keyboard as features for predicting the user. I can think of two options for using these features in the same model:

With feature engineering, find a way to represent both sequences as a fixed array of features. For example you might have features such as average typing speed average mouse speed, number of mouse movements, number of times each key is pressed, etc.
Similar idea but in a more DL approach: find a way to represent both sequences as embeddings (there are methods for word embeddings, sentence embeddings, graph embeddings...)

In my opinion the main issue is the variable length of the sequences, not the fact that they are not aligned (the alignment would matter if the target variable was for one element in the sequence).


Optimization motion planning by using Bellman equation From the Montana article "Kinematics of Contact and Grasp", if I have a ball roll on the plane without sliding, the motion equation is described below:
\begin{equation*}
\begin{bmatrix}
\dot{u}_{2} \\
\psi
\end{bmatrix}
=
\begin{bmatrix}
M^{-1}_{2}R_{\psi}\\
T_{1}+T_{2}R_{\psi}
\end{bmatrix}
M_{1}\dot{u}_{1}
\end{equation*}

${(1)}$ for Ball, ${(2)}$ for Plane.  ${\psi}$ is the angle of contact between ${x}$ axis of coordinates of ball and plane. ${K, T, M}$ are the curvature form, torsion and metric tensor at time $t$ relative to coordinates of ball and plane.

We have \begin{equation*}
R_{\psi} =
\begin{bmatrix}
\cos_{\psi} -\sin_{\psi}\\
-\sin_{\psi} -\cos_{\psi}
\end{bmatrix}
\end{equation*}

How can I use the Bellman equation by Sutton and Barto to discrete path planning of the motion of the ball?

\begin{align}
v_{\pi}(s) = \sum_a \pi(a|s) \sum_{s'} P_{ss'}^a(R_{ss'}^a + \gamma v_{\pi}(s'))
\end{align}

From my understanding that agent is a ball, environment is the plane, action is rolling ball without sliding and the achieved goal is motion planning from one point to another point.

The goal is how to optimize the path planning of the point in the motion.

I do not know how to determine the policy $\pi,$ and how to build up a function from Bellman equation. Please leave comments.

From my understanding that agent is a ball, environment is the plane, action is rolling ball without sliding and the achieved goal is motion planning from 1 point to another point.

That's already a good start. Still, I would recommend to be a bit more concrete to clear everything up. The action is not only the rolling ball, but has to be some kind of instructive control, e.g. move the ball 1cm forward, steer 10° clockwise, accellerate by a - just to name a few possibilities. Given this Action set $A$ with $a\in A$. $P_{ss\prime}^a$ models the state transition probabilities - if you are in a deterministic environment, it is constant 1 for one state and 0 for all others. $R$ is the reward, and here is where it becomes tricky. You want to find the best trajectory to the goal, thus you want to penalize if the agent takes unneccessary actions, leading to the agent reaching the goal as fast as possible. I would recommend a constant -1 for that purpose, if you end the episode when the ball reaches the goal. If this is not the case, it has to be more sophisticated, e.g. the negative distance of the ball towards the goal. This should be everything you need to get your hands dirty.


Partial Derivative - Pattern Recognition Bishop section 2.3 Please refer to page $93$, Pattern Recognition book by Bishop.
This question is related to partial derivative of a term, the proof refers to appendix C.19 which states:
$$\frac{\partial({a^{T}x})}{\partial{x}} = \frac{\partial({x^{T}a})}{\partial{x}} = a$$
and then moves on to take partial of normal probability distribution w.r.t. $\mu$ to get (eqn 2.120):
$$\frac{\partial(x_{n} - \mu)^{T}\Sigma^{-1}(x_{n} - \mu)}{\partial{\mu}}  = (x_{n} - \mu)$$

  I am unable to see how we get the term. I tried checking elsewhere,
  but the explanation "consider $(x_{n}-\mu)$ to be scalar and
  $(x_{n}-\mu)^{T}$ as vector"  does not resonate, as per my understanding both are
  vectors (as $x_{n}$ is vector) - it's their dot product that results in scalar.

I tried solving it step-by-step:
Expand the term, ignoring $\Sigma^{-1}$ (contains no $\mu$ term) which gives:
$$\frac{\partial{(x_{n}^{T}x_{n} - x_{n}^{T}\mu - \mu^{T}x_{n} + \mu^{T}\mu)}}{\partial{\mu}}$$
As per my understanding $\frac{\partial}{\partial{\mu}}$ of: $x_{n}^{T}x_{n} = 0$; of $-x_{n}^{T}\mu = -\mu^{T}x_{n} = -\mu$  but, not sure about $\mu^{T}\mu$

  Given the context, I request help on partials :

  Is $\frac{\partial}{\partial{\mu}}$[ $x_{n}^{T}x_{n} = 0$,  $(-x_{n}^{T}\mu = -\mu^{T}x_{n}) = -\mu$]
  $\frac{\partial}{\partial{\mu}}(\mu^{T}\mu)$
  I have some understanding of calculus, so guide to minimum that'll help me derive equation 2.120.

Magnus & Neudecker will take time (450+ pages) and so I may have to gallop over  another partial on page 94 $$ \frac{\partial{(x_{n}-\mu)^{T}\Sigma^{-1}(x_{n}-\mu)}}{{\partial{\Sigma}}}$$

$$\frac{\partial}{\partial \mu} (x_n^Tx_n)=0, $$$$\frac{\partial}{\partial \mu}( -\mu^Tx_n)=-x_n$$
$$\frac{\partial }{\partial \mu}(\mu^T\mu) = 2\mu$$
Note that if $A$ is symmetric, then $$\frac{\partial }{\partial y}(y^TAy)=2Ay.$$

\begin{align}\frac{\partial }{\partial \mu}\left(-\frac12 (x_n-\mu)^T\Sigma^{-1}(x_n-\mu) \right)&=-\frac12\frac{\partial }{\partial \mu}\left( (\mu-x_n)^T\Sigma^{-1}(\mu-x_n) \right)\\
&=-\frac12\left(2\Sigma^{-1}(\mu-x_n) \right)\\
&= \Sigma^{-1}(x_n-\mu)\end{align}


One-hot & interaction one-hot on multiple categorical I was wondering if there is any value to creating combined features out of multiple categorical variables when the individual categorical variables are already one-hot encoded?
Simple example: there is a variable P with categories {X,Y} and and a variable Q with categories {Z,W}. After one-hot we would have 4 variables, P.X, P.Y, Q.Z and Q.W.
In this scenario I'm wondering if the algorithm (Xgboost or a deep neural network) would sufficiently learn interaction effects between these or is there further value to creating variables: X.Z, X.W, Y.Z, Y.W which would be the unique combinations of P and Q.
The reason I am asking is to try to asses whether to embark on creating these interaction variables in my real-world scenario, where I have 7 such categorical features and 6-15 categories each which would mean thousands of new variables to account for all possible levels of permutations.

Tree models will be able to figure out the Interaction whether Label Encoded or OHE.
If you separately create all the combinations of Category values as new Features, then there is nothing left for interaction (At least 1st level).
e.g. let's take the example Features
P -  {X,Y}
Q  - {Z,W}
Let's assume, P and Q have interaction on X and W.
 If we Label encode - We will observe multiple consecutive splits on P==X and Q==W.
 If you OHE - you will observe multiple consecutive splits on Is_P_X and Is_Q_W (since both are separate feature now).
In these cases, you will have to observe the splits to see the interaction or can use a partial dependence plot or similar approaches.
 If you Create features on all combinations -
In this case, you will observe very high Feature importance for feature X_W that is actually because of interaction but you will not observe it as consecutive splits on two feature but a split on single feature i.e X_W
In these cases, you can observe the interaction with Feature importance.
On a dummy data where I added the interaction effect on output for Sunday and Breezy weather.
Image 01 - Showing Interaction in the splits

Image 02 - Simply splitting on Sun_Breezy when every combination is a separate feature


How to handle non consistent time series( using LSTM ) The time series dataset I am working on has missing samples.
I am trying to use keras and LSTM for prediction.
How should I handle the missing timestamp samples ( sometimes there are missing weeks even months)?

Does this affect the overall performance of the LSTM predictor?

If you have missing data you can add a Masking layer to your model which will prevent contributions from those time steps to be included.  You can see the documentation for it here: https://keras.io/layers/core/#masking

Example taken from the keras documentation:

  Consider a Numpy data array x of shape (samples, timesteps, features),
  to be fed to an LSTM layer. You want to mask timestep #3 and #5
  because you lack data for these timesteps. You can:

  set x[:, 3, :] = 0. and x[:, 5, :] = 0. insert a Masking layer with
  mask_value=0. before the LSTM layer:

With example code:

model = Sequential()
model.add(Masking(mask_value=0., input_shape=(timesteps, features)))
model.add(LSTM(32))


Machine Learning - Choice of features for determining hypothesis I am a newbie to machine learning and have the following elementary questions.

Given a labeled dataset with multiple features, is it for a ML algorithm determine on its own what features are to be considered for determining the hypothesis function? Or is it for a user who is training the algorithm to explicitly indicate what features are to be considered?
Is there a mapping between problem space versus the hypothesis function type? As an example based on past experience, can we confidently state that for a housing price prediction problem, linear equation is appropriate. While for determining diabetes based on a given dataset, a quadratic hypothesis equation has proven reasonable.

Or is it usual for a designer to specify a custom function and allow the ML algo to determine the coefficients of this custom function (e.g. square root of sum of squares of 9 of 15 features divided by e^n) by reducing the error using gradient descent?

There are algorithms that use Feature Selection to utilize only the "best" features for the given dataset (variants of adaboost for example). There are also algorithms randomly use subsets of the features (such as random forest). Finally there are algorithms that learn weigh your inputs and use them accordingly (Neural networks and the like)
There is "common knowledge", but it is more related to the type of features and classification problem that you are facing (i.e. regression vs. classification, continuous vs discrete, binary vs. multi-class, etc.).

If you have a specific problem that you can model using such a function then it is a completely reasonable approach to design such a function.
Once you have modeled your problem with such a function and defined your loss function - you could use any optimization method that seems reasonable.
However it is beneficial to try and find similar problems that others have solved before and understand how their solution relates to your problem.


How to compute time-lagged correlation between two variables with many examples at each time t? I have a dictionary of following form:
datetimes = {year : {name : (score1, score2)}} #there are 50+ names/year

So, essentially, I'm trying to get an aggregate picture of how score1 at year_n is correlated with score2 at year_n.

An application of a specific correlation formula depends on the data-type (continuous or rank data etc). Given that your data is continuous, you can apply  Karl Pearson formula. However, if you're interested  interested in cause and effect relationship, you may prefer to use simple regression  model.


Creating a image classification model I am working on a dataset to classify facial expressions.
Dataset has 7 classes, training images 28000 and test images 7000. I created 2 models
Model1:
this model has 11 layers. Initially model was working fine but now accuracy is increasing very slowly after 60% and drooping sometimes. model 1 link
then I thought that due large dataset my model is over fitting and I reduced the images by randomly deleting in each class and reducing train set images to 15000 and test set images to 3900. and created another model.
Model 2:
This model has 9 layers. The graph for first 40 epochs was fine but after then when I resumed training its seems to overfit.model 2 link
first 40 epoch

this seems fine i guess
after resuming training

I have also tried model with fewer layers nut cant figure out why my accuracy is stuck at and model is not improving
dataset link, I don't understand is there a problem with dataset, can anyone provide me link to another dataset. Figure out what changes are required in model.

1.) You should not change your test set, even if you downsample the training set.
2.) At least in the plots you show there is no clear indication of overfitting. Did you see what happens if you train for longer epochs, and tried different learning rates ?
3.) If you have some overfitting, normally you would simplify the model, and not reduce the size of the dataset. Also you can think about data augmentation to prevent overfitting.
4.) You can compare your solution to the solutions available at kaggle.
5.) With such a black-box solution (vanilla CNN) you will not get super high results.  Though I do not know the upper bound. You can compare to kaggle for that..
6.) To further improve you have to integrate some prior knowledge (e.g. facial key points).


Spatially constrained geospatial similarity What's the current methodology for clustering geospatial data by features?

Example: I have some demographic dataset. Let's say this contains average home price and population density.

So, an example correlation here would be home price vs population density. But, the trick is how the clustering gets pulled. For example, an affluent area with high population density isn't the same as one with low population density. Applying a basic distance metric wouldn't take this into account since low vs highs could offset each other giving similar distances. This leads me to possibly some form of weighted clustering to pull centroids.

Not sure what methodology takes this into account.

I assume you are trying to find a suitable distance metric based on features of different areas (although spatial distances might also easily be plugged in). In that case, I would first try to make sure the different features are correctly scaled, for example, to zero mean and unit variance.

If the result does not seem right, I would also try looking at different distance metrics. A simple alternative example is the L1 norm:

L1(a, b) = sum_x |x_a - x_b|


Using SVD for clustering The dataset that I am experimenting with is in the form of a table with columns userid and itemid. If there is a row for a given user and a given item, that means the user accessed the item (like in an online store). I am trying to cluster similar items based on this data. If a pair of items is accessed together often, then the items are similar.

Because this is a case of a high dimensionality (# of users and items will be in 10,000's) I think I am justified in trying to use SVD as a pre-clustering step and then do some classical clustering. When I tried doing this I got poor clustering results when compared with simple hierarchical clustering. Items that weren't very similar were being bucketed together in one dimension, while there were available dimensions that weren't used. The results weren't completely random, but they were definitely worse than the output from the hierarchical clustering. I attempted the SVD step with Mahaut and Octave and the results were similar. For the hierarchical clustering I used the Jaccard measure.

At this point I am starting to doubt the notion of SVD as a way to reduce dimensionality. Do you think that SVD cannot be used effectively in this case (and why?) or do you think that I made some mistake along the way?

We are using Singular Value Decomposition in the much same manner as you, except rather than clustering similar items, we are using a reduced-rank matrix to power a recommendation engine based on a term-document matrix in Latent Semantic Indexing.

From your brief description, your approach seems sound enough. However, I highly recommend reading Berry, Dumais & O'Brien's Using Linear Algebra for Intelligent Information Retrieval.

Key to using SVD is selecting an acceptable rank-k approximation to the original sparse matrix. You should carry out some exploratory analysis to see how much variance can be explained, using the singular values in the diagonal matrix Sigma. This question was brought up in this question on CrossValidated.

A lot of the papers I've read suggest anywhere a rank k from 200 to 300 singular values. In a proof-of-concept implementation, we had original sparse matrix of about 10000 rows (unique terms) to about 1000 columns (unique documents), and we were capturing just under 85% of the variance with only 300 singular values.

However, that really hinges upon the nature of your data, so your mileage may vary.


What are the advantages of HDF compared to alternative formats? What are the advantages of HDF compared to alternative formats? What are the main data science tasks where HDF is really suitable and useful?

Perhaps a good way to paraphrase the question is, what are the advantages compared to alternative formats?

The main alternatives are, I think: a database, text files, or another packed/binary format.

The database options to consider are probably a columnar store or NoSQL, or for small self-contained datasets SQLite.  The main advantage of the database is the ability to work with data much larger than memory, to have random or indexed access, and to add/append/modify data quickly.  The main *dis*advantage is that it is much slower than HDF, for problems in which the entire dataset needs to be read in and processed.  Another disadvantage is that, with the exception of embedded-style databases like SQLite, a database is a system (requiring admnistration, setup, maintenance, etc) rather than a simple self-contained data store.

The text file format options are XML/JSON/CSV.  They are cross-platform/language/toolkit, and are a good archival format due to the ability to be self-describing (or obvious :).  If uncompressed, they are huge (10x-100x HDF), but if compressed, they can be fairly space-efficient (compressed XML is about the same as HDF).  The main disadvantage here is again speed: parsing text is much, much slower than HDF.

The other binary formats (npy/npz numpy files, blz blaze files, protocol buffers, Avro, ...) have very similar properties to HDF, except they are less widely supported (may be limited to just one platform: numpy) and may have specific other limitations.  They typically do not offer a compelling advantage.

HDF is a good complement to databases, it may make sense to run a query to produce a roughly memory-sized dataset and then cache it in HDF if the same data would be used more than once.  If you have a dataset which is fixed, and usually processed as a whole, storing it as a collection of appropriately sized HDF files is not a bad option.  If you have a dataset which is updated often, staging some of it as HDF files periodically might still be helpful.

To summarize, HDF is a good format for data which is read (or written) typically as a whole; it is the lingua franca or common/preferred interchange format for many applications due to wide support and compatibility, decent as an archival format, and very fast.

P.S. To give this some practical context, my most recent experience comparing HDF to alternatives, a certain small (much less than memory-sized) dataset took 2 seconds to read as HDF (and most of this is probably overhead from Pandas); ~1 minute to read from JSON; and 1 hour to write to database.  Certainly the database write could be sped up, but you'd better have a good DBA!  This is how it works out of the box.


Visualizing word embeddings I am working on a text-classification problem, and trying to understand how to work with the tensorboard projector for an embeddings layer in Keras. Borrowing an example from the Deep Learning with R book, I have a model set up like this:

library(keras)

max_features <- 2000
max_len <- 500

imdb <- dataset_imdb(num_words = max_features)
c(c(x_train, y_train), c(x_test, y_test)) %<-% imdb
x_train <- pad_sequences(x_train, maxlen = max_len)
x_test = pad_sequences(x_test, maxlen = max_len)

model <- keras_model_sequential() %>%
  layer_embedding(input_dim = max_features + 1, output_dim = 128,
                  input_length = max_len, name = "embed") %>%
  layer_conv_1d(filters = 32, kernel_size = 7, activation = "relu") %>%
  layer_max_pooling_1d(pool_size = 5) %>%
  layer_conv_1d(filters = 32, kernel_size = 7, activation = "relu") %>%
  layer_global_max_pooling_1d() %>%
  layer_dense(units = 1)

summary(model)

model %>% compile(
  optimizer = "rmsprop",
  loss = "binary_crossentropy",
  metrics = c("acc")
)

When it comes time to configure the callback_tensorboard object, however, I'm a little lost. It seems the API has changed since the book was written, and I haven't found a good working example yet. The embeddings_data property is apparently required if the embeddings_freq parameter is set, and it needs to match the shape of the model inputs c(?, 500). I can satisfy this by simply passing all the word tokens as a matrix:

tensorboard("my_log_dir")

callbacks = list(
  callback_tensorboard(
    log_dir = "my_log_dir",
    histogram_freq = 1,
    embeddings_freq = 1,
    embeddings_data = matrix(1:max_features, nrow = 4)
  )
)

history <- model %>% fit(
  x_train, y_train,
  epochs = 5,
  batch_size = 128,
  validation_split = 0.2,
  callbacks = callbacks
)

The model trains OK, but when trying to look at the result in projector, it seems that the embeddings are being processed as 64,000 dimensions (2,000 words * 128d layer output?) with a single point for each 500 element vector in embeddings_data. This makes PCA computation stall out.

I expect a final result closer to the example at https://projector.tensorflow.org/, where Dimensions are equal to the layer output dimensions, and there is a single point for each word.

What am I missing? Are there any good working examples of visualizing a standard word embedding layer with current versions of Keras and Tensorflow?

I am using R 3.3.4, Keras 2.2.0, and Tensorflow 1.11.

It seems there are a couple issues at work here.

First, apparently the PCA computation wasn't stalling out because of the absurd dimensions being reported - updating to tensorflow 1.12 and Keras 2.2.4 has eliminated that problem.

Second, it seems that Keras bases the generation of embedding data for tensorboard on the layer inputs rather than the actual layer weights. Maybe this is important for some other usage, but it makes things very complicated in a standard word/token embedding scenario. To work around this, I ended up skipping embeddings during training, exporting the embedding weights at the end, and then training a new throwaway model with a much simpler topology to generate the projections I'm interested in. Given an object model trained from the samples above:

w <- model %>% get_layer(index = 1) %>% get_weights()

emodel <- model <- keras_model_sequential() %>%
  layer_embedding(input_dim = max_features, output_dim = 128, input_length = 1) %>%
  layer_global_max_pooling_1d() %>%
  layer_dense(units = 1)

emodel %>% get_layer(index = 1) %>% set_weights(w) %>% freeze_weights()

emodel %>% compile(
  optimizer = "rmsprop",
  loss = "binary_crossentropy",
  metrics = c("acc")
)

callbacks = list(
  callback_tensorboard(
    log_dir = "my_log_dir",
    embeddings_freq = 1,
    embeddings_data = c(1:max_features)
  )
)

history <- emodel %>% fit(
  1:max_features, 1:max_features,
  epochs = 1,
  batch_size = 256,
  callbacks = callbacks
)

tensorboard("my_log_dir")

Ideally there would be some simpler/cleaner way to achieve this, but this works for now.

edit

Another user asked in a now-deleted question about adding token labels to Projector. For posterity, that is as simple as adding a value for embeddings_metadata in the callback_tensorboard. This should be a path (relative to your log_dir) to a tab delimited file with the desired labels. Details are here: https://www.tensorflow.org/guide/embedding#metadata


inconsistency occurred when using cross_val_score in python import pandas as pd
from sklearn.datasets import load_iris
from sklearn.svm import SVC

#import data
iris = load_iris()
X = pd.DataFrame(iris.data, columns=iris.feature_names)
y = iris.target

#split data
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y,
 test_size = 0.2, shuffle=False)

#modeling using SVM
model = SVC(kernel='linear', C = 1)
model.fit(X_train, y_train)
model.score(X_test, y_test)

# here I get score: 0.8666666

# Now I use cross_validation with cv = 5,

from sklearn.model_selection import cross_val_score
model = SVC(kernel='linear', C = 1)
scores = cross_val_score(model, X, y, cv = 5, scoring = "accuracy")
scores

#here I got array([0.96666667, 1.        , 0.96666667, 0.96666667, 1.        ])

None of the numbers in the above arrays equal to 0.866666, I wonder why the inconsistency happens(since "cv = 5" matches the condition "test_size = 0.2").

One reason might be your train_test_split() setup. Using shuffle=False means you just use the first 80% examples of your data to train with no randomness. Have a look at your labels:

>>> y
Out:
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
       2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
       2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])

As you can see the iris dataset is sorted by its classes from 0 to 2. This means your train_test_split() will be unbalanced since class 2 will be underrepresented:

>>> y_train
Out:
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
       2, 2, 2, 2, 2, 2, 2, 2, 2, 2])

And even worse your test data only has one class:

>>> y_test
Out:
array([2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
       2, 2, 2, 2, 2, 2, 2, 2])

On the other hand cross-validation will use a stratified split preserving the class distribution for all folds as described in the sklearn user guide:

  For integer/None inputs, if the estimator is a classifier and y is either binary or multiclass, StratifiedKFold is used. In all other cases, KFold is used.

Which I'd expect to be leading to better results. Now, also giving the train_test_split() this advantage of balanced classes (stratified labels) gives me a different result:

>>> X_train, X_test, y_train, y_test = train_test_split(X, y,
 test_size = 0.2, stratify=y)

runfile(...)

>>> model.score(X_test, y_test)
Out: 1.0


Algorithm for campaign optimization (Digital Advertising) Suppose i am running an Ad thru an Ad exchange A, and i have a set of campaigns running on it. I have

The spend of the campaign.
The budget allocated to it.
The number of hours it took to exhaust it's budget.
Total installs.
Total Revenue.

I want to write an algorithm to divide the total budget x to n campaigns, allocating higher budgets to a campaign delivering higher revenue.

sounds like a great place for a weighted average and a little algebra.

budget_n = (revenue_n / total_revenue) * total_budget

This is a simple solution you may have already tried, but in principle, the revenue fraction can be as complex as you need it to be as all the fractions across all campaigns add up to 1.


Is it valid to compare SHAP values across models? Let's say I have three models:

a random forest with 100 trees
a random forest with 1000 trees
an xgboost model.

I can rank the importance of my features on my dataset for each model using SHAP, and compare relative importance across models. What is not clear is whether I can meaningfully compare the actual numerical SHAP values across models.
I am using the same features for all models.

Shapley values were designed in the context of game theory (source), to share value created by a coalition of player in a game. It has multiple properties, including linearity. The linearity ensure that if you were to average your models, the resulting Shapley value would be the average of Shapley values for individual models. Shapley values  are comparable in that sense of considering an average model.
I think the general answer would be the opposite. Inutitively, because looking at an individual Shapley value you can't know if the value is due to an 'individual performance' or 'overall performance' of the coalition. So looking at two values in the context of ML, the difference might be both explained by a different contribution of the individual value but also by a difference in the overall performance of the model. So I would avoid doing that in the general case. (But I often do so - along with comparing individual predictions -  to check if two models with similar overall performance have learned the same things or not)
Overall I would suggest you use a more suited criteria (like an information criteria) for your model selection, then use Shapley value to explain the model you selected. Not using Shapley Values to do some sort of model selection.
Note that I am mainly talking about Shapley values and not SHAP, which is an approximation. You need to be cautious with SHAP as the approximation rely on the absence of correlation between your features, which rarely happen in practice.


What is a batch in machine learning? Karpathy's' LSTM batch network LSTM batch network operates with batches

def checkSequentialMatchesBatch():
    """ check LSTM I/O forward/backward interactions """
    n,b,d = (5, 3, 4) # sequence length, batch size, hidden size
    input_size = 10
    WLSTM = LSTM.init(input_size, d) # input size, hidden size
    X = np.random.randn(n,b,input_size)
    #...

def checkBatchGradient():
    """ check that the batch gradient is correct """
    # lets gradient check this beast
    n,b,d = (5, 3, 4) # sequence length, batch size, hidden size
    input_size = 10
    WLSTM = LSTM.init(input_size, d) # input size, hidden size
    X = np.random.randn(n,b,input_size)
    #...

What does batch applied for? I'm only familiar with feeding one-hot word representation vector and can't understand LSTM learning process with batches. Please explain in terms of text processing.

Thanks in advance.

The accepted answer is correct, but it may also be helpful to think of a batch from a classification standpoint.

Suppose you have a binary classification problem that you are trying to solve using a multilayer perceptron, with 1000 examples of each class.

When training the model, you don't want to have to wait until the model has seen all of the data before a weight update is performed. That's computationally inefficient. Instead, you take, for example, 100 random examples of each class and call it a 'batch'. You train the model on that batch, perform a weight update, and move to the next batch, until you have seen all of the examples in the training set. One pass through the training set in this manner is called an 'epoch'.


How to use LeakyRelu as activation function in sequence DNN in keras?When it perfoms better than Relu? How do you use LeakyRelu as an activation function in sequence DNN in keras?
If I want to write something similar to:

 model = Sequential()
    model.add(Dense(90, activation='LeakyRelu'))

What is the solution? Put LeakyRelu similar to Relu?

Second question is: what are the best general setting for tuning the parameters of LeakyRelu? When is its performance significantly better than Relu?

You can use the LeakyRelu layer, as in the python class, instead of just specifying the string name like in your example. It works similarly to a normal layer.

Import the LeakyReLU and instantiate a model

from keras.layers import LeakyReLU
model = Sequential()

# here change your line to leave out an activation
model.add(Dense(90))

# now add a ReLU layer explicitly:
model.add(LeakyReLU(alpha=0.05))

Being able to simply write e.g. activation='relu' is made possible because of simple aliases that are created in the source code.

For your second question:

  what are the best general setting for tuning the parameters of LeakyRelu? And when its performance is significantly better than Relu?

I can't give you optimal settings for the LeakyReLU, I'm afraid - they will be model/data dependent.

The difference between the ReLU and the LeakyReLU is the ability of the latter to retain some degree of the negative values that flow into it, whilst the former simply sets all values less than 0 to be 0.  In theory, this extended output range offers a slightly higher flexibility to the model using it. I'm sure the inventors thought it to be useful and perhaps proved that to be the case for a few benchmarks. In practice, however, people generally just stick to the ReLU, as the benefits of the LeakyReLU are not consistent and the ReLU is cheaper to compute and therefore models train slightly faster.


Is statistics a pre-requisite to learn deep learning? I am very interested in deep learning, and I was recommended to read The Elements of Statistical Learning, but it is very hard to get into. So my question is :

How important is learning statistics and reading books like elements of statistical learning, if my end goal is to learn deep learning?

I should have mentioned that I have already done Andrew Ng's Machine Learning course.

I would not call it a pre-requisite, but I would say reading books the that is necessary for knowing it all very well.

I would recommend you to start with Andrew Ng's Machine Learning course on coursera.

Once you are familiar with that, move on to deep learning videos:

Nando de Freitas
Hugo Larochelle

While you are going through topics one after the other, go back to the books and read what they have to offer. Before you know, you would have done it all.

Good Luck!


Removing irrelevant content in columns I have a dataset with a column named Package Size and it has data in two units inches and meters mentioned in words. How do I preprocess the data such that I remove the unnecessary word and also convert them into a single unit?

Example:

Package Size 13 inches4 meters

Package Size(in meters) 0.3302 4

Since you had not provided the tag on which language you are using. I am providing a sample solution in python with pandas for loading the data and transforming.

>> import pandas as pd
>> sample = pd.DataFrame({'a': ['1 inch', '2 meters', '1 metre', '2.5 inches', '2.8 meters']})
            a
0      1 inch
1    2 meters
2     1 metre
3  2.5 inches
4  2.8 meters
>> def get_number(a):
       for _ in a.split(' '):
           try:
               return float(_.strip())
           except ValueError:
               pass
>> m['b'] = m['a'].apply(lambda a: get_number(a) * 0.0254 if 'inch' in a else get_number(a))
>> m
              a       b
  0      1 inch  0.0254
  1    2 meters  2.0000
  2     1 metre  1.0000
  3  2.5 inches  0.0635
  4  2.8 meters  2.8000

Hope this gives you a general idea of how to do it.


How can I rank paths through an HMM? I have a profile hidden Markov model that I use to identify all instances of a user-defined pattern of symbols in a long sequence of symbols. I use the Viterbi algorithm to find the most probable path that generates that long sequence of symbols, and it all works very well to identify the user's pattern. But I'm now interested in extending this, and identifying the k-most probable paths through my model that would generate some long sequence. Is there an existing algorithm to do this? I've come across something called the "1-best" or "k-best" algorithm, but I can't find much of anything describing it.

I've also considered what I'm sure would be a heuristic, by finding the most probable path and then returning the neighbourhood around it. This would be found by iterating over each move between states in the most probable path, setting that particular step to 0, and resolving using Viterbi. This would obviously increase the run-time by a factor of (length of the path ~= length of pattern), which I suspect will be reasonable for my purposes.

Has anybody come across something like this, or can anybody see anything obviously wrong in my neighbourhood idea? I'd appreciate any feedback. As a note, this has been cross-posted to the Theoretical Computer Science forum.

Use log-probabilities, then use k-shortest paths.

The probability of a path is the product of the probabilities on its edges.  If you log-transform everything (so that each edge is annotated with the log of the probability rather than the probability itself), the log-probability of a path becomes the sum of the log-probabilities on the edges.

Now you want to rank the paths by decreasing log-probability, and find the $k$ paths with the highest log-probability.  Note that the log-probability of a path is just the length of the path (the sum of the lengths of the edges).

Therefore, the problem becomes: given a graph with a length on each edge, find the $k$ shortest paths.

There are efficient algorithms for finding the $k$ shortest paths.  See, e.g., https://cstheory.stackexchange.com/a/32501/5038.


Graph modularity measure There are several metrics for the quality of a graph clustering, e.g. Newman modularity. These enable you to compare two candidate clusterings of the same graph.

Does anyone know a metric that will answer the question "how modular is this graph"? For example the first of these two graphs is more modular than the second:
    o===o-----o====o    o----o===o-----o

It would be possible to choose a clustering algorithm, run it, and compute your preferred modularity metric for the best clustering found. But this is only a lower bound, so it doesn't seem very satisfactory.

The question matters. For example, the work of life scientists will be easier if the molecular organisation of life is modular than if it is not. It would be good to have a robust test - some of the discussion so far seems to involve wishful thinking.

My best attempt at this is:
- a tree is more modular if the edges near leaves are higher weight
- the modularity of a graph is the modularity of its min cut spanning tree
Does anyone know of an established answer to this question?

I am not sure there is a clear answer to this, especially as the problem does not seem to be well-defined right now - your "figure" seems to indicate edge weights but you then mention node weights, something significantly different.

If the question is whether you can find a way to split a graph into two smaller modules, then you might want to look into applying Sparsest Cut techniques - a cut with low cost would imply (?) high modularity. I believe these can be easily modified to account for either unlabeled, edge-labeled or node-labeled graphs.


Using Amazon Personalize to build a Recommendation System I would like to build a recommendation system based only in the items metadata.

I have an input vector with some desirable topics that the user want to read about, for example:  (self-help, yoga, sports)

On the other hand I have a dataset with books with Title, Description, among other fields.

Up to now I am building this model locally with Python, using clustering to group books by similarity.

However, I would like to build a Recommendation System using Amazon Personalize for this case using only item metadata. I do not want to add other info as ratings for now.

Is it possible?
Do you know some example?

As far as I have researched, Amazon Personalize is not useful for this project.
This is because Amazon Personalize requires User-Item Interaction data to train the model.
To use Amazon Personalize, you need a dataset with historical data that contains at least the fields:

User
Item
Date

where the Date field represent the time when the user interacted with the Item.


Memory error - Hierarchical Dirichlet Process, HDP gensim I am running Hierarchical Dirichlet Process, HDP using gensim in Python but as my corpus is too large it is throwing me following error:

model = gensim.models.HdpModel(corpus, id2word=corpus.id2word, chunksize=50000)

 File "/usr/cluster/contrib/Enthought/Canopy_64/User/lib/python2.7/site-packages/gensim/models/hdpmodel.py", line 210, in __init__
    self.update(corpus)
  File "/usr/cluster/contrib/Enthought/Canopy_64/User/lib/python2.7/site-packages/gensim/models/hdpmodel.py", line 245, in update
    self.update_chunk(chunk)
  File "/usr/cluster/contrib/Enthought/Canopy_64/User/lib/python2.7/site-packages/gensim/models/hdpmodel.py", line 313, in update_chunk
    self.update_lambda(ss, word_list, opt_o)
  File "/usr/cluster/contrib/Enthought/Canopy_64/User/lib/python2.7/site-packages/gensim/models/hdpmodel.py", line 415, in update_lambda
    rhot * self.m_D * sstats.m_var_beta_ss / sstats.m_chunksize
MemoryError

I have loaded my corpus using following statement:

corpus = gensim.corpora.MalletCorpus('chunk5000K_records.mallet')

And the data which I used to load corpus has 5 million records. And this is working for me when I am loading only 50K records. So I have added chunksize option HdpModel but it is still giving me an error.

Please let me know how I can solve this issue. And I am running this on High Performance Computing so I think there should be a solution to resolve this issue as this cluster has really big size memory and disk capacity.

you can use an alternative of HDP that is LDA. HDP won't give hierarchical output. HDP and LDA are both creating a flat hierarchy. The only difference is that HDP is generating topics based on topic generated in a pervious iteration. Online LDA is quite memory efficient as well as good at capturing topics.


Trying to figure out which book to tackle first I'm currently at my 4th and last year of my Applied Maths degree and just recently got an internship at a startup. During this period I've been reading and getting deeper and deeper into studying machine learning. And so I've stumbled upon two interesting books:
1- "The elements of statistical learning", Jerome H. Friedman, Robert Tibshirani, and Trevor Hastie
2-"Understanding machine learning (From theory to algorithms)", Shai Ben-David, Shai Shalev-Shwartz
I intend to read both books but just wanted to hear from someone more experienced in this field which one i should focus on first, and each books strenghts and weaknesses.

I assume it will probably be more beneficial for you to focus on getting a sample of the different techniques in use rather than learning the theoretical frameworks, probability bounds, and so on. In that case, I'd say the following sections are most useful to you:

Elements of Statistical Learning. Chapters 1–4 should give you a good flavour of 'traditional' classification and regression techniques such as nearest neighbours, linear regression, logistic regression, etc. Chapter 7 is also recommended by the authors as essential as it goes through the principles of selecting a model: these concepts will apply no matter what method you use.
From there you could sample what you were interested in: chapter 11 on neural networks might be interesting, although you might find a more modern treatment preferable. I like Deep Learning by Goodfellow, Bengio and Courville, which you could browse through and choose relevant chapters.
Support vector machines (chapter 12) are also pretty popular and worth being familiar with, and in the second edition you might read chapter 15 on random forests, too. Chapter 10 on boosting might be fun, too: XGBoost and similar often win machine learning competitions, so knowing the background would be helpful.

Understanding Machine Learning. Part I has a rather theoretical slant; I suspect many practitioners don't know much about PAC learning and VC dimension, although they are interesting if you have the time. In Part II you could review any methods you don't see in ESL, and take a look at the unsupervised learning techniques in Part III. Part IV is again theoretical and probably not as relevant.


Training and cross validation error curves I have a graph which plots training datasize on X axis and accuracy on y axis. I plotted the curves using sklearn's learning_curve.

It is observed that the accuracy of training dataset decreases but the accuracy of validation dataset increases. I am not able to justify this behavior.

Normally, as training dataset increases, the training accuracy is supposed to increase right?
Also, assuming that the dataset is very noisy and hence training accuracy is decreasing as dataset size increases. But this doesn't explain why validation accuracy increases because noise is supposed to affect that too.

I think what you're seeing is normal behaviour:

With only few samples (like 2000) it's easy for a model to (over)fit the data - but it doesn't generalize well. So you get high training accuracy, but the model might not work well with new data (i.e. low validation/test accuracy).

As you add more samples (like 9000) it becomes harder for the model to fit the data - so you get a lower training accuracy, but the model will work better with new data (i.e. validation/test accuracy starts to rise).

So:

As the training dataset increases, the training accuracy is supposed to decrease because more data is harder to fit well.
As the training dataset increases, the validation/test accuracy is supposed to increase as well since less overfitting means better generalization.

Andrew Ng has a video about learning curves. Note that he plots the error on the y axis, you have the accuracy on the y axis.. so the y axis is flipped.

Also take a look at the second half of the video. It explains high bias and high variance problems.

Your model seems to have high variance (due to the big "gap" between the two curves) - it's still too complex for the small amount of data you've got. Either getting more data or using a simpler model (or more regularization on the same model) might improve the results.


Text classification and clustering with complete date imbalance I have a set of scientific papers of authors who have common research interests from PUBMED and I would like to:

Clustering papers and extracting features from them in order to find other authors with similar research interests in PUBMED
Develop a classification algorithm to determine if a given author
has the similar research interest to those in my original set.
Really, I am trying find new authors and be able to tell if an
author that I discovered, is similar to my original set of authors.

Unfortunately because my data is only one class, I have an extreme data imbalance problem.  How should I fix this?

Thus as far as I have looked at the frequency of terms in the papers with mixed results.  I have also been using unsupervised learning to extract features with mixed results.

My set of papers and authors are assumed to share similar research interests. Do I need papers and authors that I know aren’t in the set in order to build a classification algorithm?

Should I look at a paper as a whole or should I look at the papers written by each author for classification, instead of looking at the papers as a group?

I suppose I could randomly pick other papers and use those as the examples of other fields. Of course, picking paper at random there is the chance of false negatives.

I am using R with the tm package to do the mining.

The question is not very clear but I will give a try anyways.

First of all please note that the major approach to tackle the analysis of co-authorship networks is not what you mentioned, but Network Science techniques as co-authorship is a Social Network.

After this necessary tip let's back to the question itself which is tackling the problem from NLP point of view.

  Unfortunately because my data is all of one class already I have an extreme data imbalance problem. How should go about fixing this?

You don't have a class imbalanced problem. Class imbalanced problem means that you have more than one class in which one or more classes are highly larger than others. But you have only one class! This is not a class imbalanced problem. You only need to define your question precisely.

  My set of papers and authors are assumed to share similar research interests. To build a classification algorithm do I need papers and authors that I know aren’t in the set?

Depends on what you want to do! Classification of what? You can set your question like classification of authors in the same field then you do not need additional papers. You extract text features according to different papers of all individuals and train your classifier:

Concatenate all sentences of the papers of one individual and call it data. the name of individual is the label. Of course if a paper has two authors, the text will be assigned to both of them.
Extract features from your text data. Unsupervised feature extraction does not care about the labels (TF-IDF is an example) and supervised methods do care.
Choose a classifier and start your learning phase as always.

but if you set your question like classification of authors from different field then sure you need data from other fields! In this case instead of each individual being a class, each field (including all authors in that field) becomes a single label.

  Should I look at the paper as a whole or should I look at the papers written by each author for classification and instead of looking at the papers as a group?

It's super difficult to understand what you mean but I guess I answered it above. Depends on your question! If you are going to classify authors inside a community, then all papers of that author is one text data object.

  I suppose I could randomly pick other papers and use those at the examples to one that don’t share research interests. Of course, picking paper at random there is the chance of false negatives.

Again difficult to understand but anyways. In this case you better gather data from another field. Randomly mine some papers from other fields and construct a new class of NOT THIS FIELD. Then go for a binary classification(Logistic Regression works well on TF-IDF features). It should be pointed that as your data is text you can not cover every text which is not in your main class so limit your question to a specific border in domain.

Hope I could help :)

PS: The question needs a surgery. I will edit it soon and please check if my edit is conceptually correct.


Using deep learning or random forest I am very new to machine learning, and I am trying to build a model to classify this data set (UCI heart disease). I have built a simple model using random forest and got an 80% accuracy. The size of the set is about 900. To get the best value possible, should I continue with random forest or go with deep learning (such as TensorFlow)?

Thanks!

The given dataset has 303 observations and 76 features. This is usually considered very small. Deep learning needs plenty of data to be stable since it is highly nonlinear. Random forest is one option but simpler structured models can be considered. One important thing which have to be investigated in those situations is regularization. Fighting with over fitting looks like the main goal.


Cross validation for C5.0 algorithm I want to try K-fold cross validation in R for C5.0 algorithm,

The following is the code i use. Can someone suggest me how can i include k-fold as well?

Classifi_C5.0 <- C5.0(TARGET ~., , data = training_data_SMOTED, trails = 500,
                      control = C5.0Control(minCases = mincases_count,
                                            noGlobalPruning = FALSE))

Is it required to do k-fold cross validation for Random forest?

I would say cross validation is unnecessary here since the multiple partitioning of the data and variables is already implicit in Random Forests.   But it's still a good practice to hold out a testing set that is distinct from the training set.  This is mostly because you may introduce changes in your random forest to improve the performance on the test sets overall, thereby introducing the bias that the random forests are trying to overcome.   So if you withheld a portion of your data and judged the final performance of the RF on that withheld set only in the predict step, then it's fine.


Membership ratio graph Does anyone know how the what the kind of graph in this image is called?

Each color represents a class and the height of the color, in a particular instant, represents the ratio of elements that belong to that class.

How can I produce such a graph, as an example, in R or Python?

This looks like a "proportional stacked area chart." The R Graph Gallery gallery has a similar one here:

They have great sample sample to provide guidance on how to re-create it (https://www.r-graph-gallery.com/136-stacked-area-chart.html).


Should you turn off label smoothing when validating? As the subject says. On one hand, the answer should be yes because label smoothing is a regularization feature and how can you know if it improves performance without turning it off? On the other hand, I haven't seen any authoritative source claiming that it should be turned off during validation, not even the article that introduced the technique mentions it. And afaict, as the feature is implemented in Keras it is not turned off during validation.

The way most people gain an initial understanding of label smoothing (and what most common explanations have to say on the subject) plays a great role in how one would approach this question.
At first glance, label smoothing is exactly what the name suggests: we modify the labels or some portion of them in order to get a better, more general, more robust model. It makes sense that we don't want the model to learn from (and later predict with) extreme confidence levels, especially when we know that some of the labels are wrong, since this hurts the model's abilities to perform on unseen data. The intuitive explanation of LS mechanics, then, is that we are not feeding the model with pure 1's and 0's, but with less confident values instead, resulting in a more reserved decision function that doesn't extrapolate in an extreme manner. Now, we know that smoothed labels are not the true labels, so at this point the main concern pops up - once we have trained on smoothed labels, do we also use smoothed labels for validation?
If you think about LS purely as a data manipulation technique, the answer to the question above is not obvious, as it may go both ways depending on the argument. However, one must remember that LS is almost always regarded as a regularization technique -- you mention this in the question yourself -- and there is a good reason for that. Regularization, by definition, is when the loss function is extended with an additional regularization term, which usually has to do with penalization. In LS, this penalty term is responsible for punishing high confidence predictions. Even though it may not appear as such, LS, once applied, becomes an essential part of the loss function, which should persist between training and validation if we aim to take advantage of the technique. When we apply LS during training, we are effectively trying to minimize a loss function with the regularization term added in. Throwing it out during validation would defeat the very purpose of including it in the first place: if we decide not to apply LS to the validation set as well, we are making the mistake of expecting extreme confidence (1/0 labels) from a learner that was just a moment ago specifically trained against making overly confident predictions on previously unseen data. The correct thing to do would be to validate that the predictions are of moderate confidence, just as desired. This is why the validation set should also have the regularizer present in the loss function, i.e. have smoothed labels as well.


Multiple output for multi step ahead prediction using LSTM with keras I am new to deep learning and LSTM (with keras). I am trying to solve a multi-step ahead time series prediction. I have 3 time series: A, B and C and I want to predict the values of C. I am training an LSTM feeding 3 steps back data points to predict the next 3 steps in the future.
The input data looks like:

X = [[[A0, B0, C0],[A1, B1, C1],[A2, B2, C2]],[[ ...]]]

with dimensions: (1000, 3, 3). The output is:

y = [[C3, C4, C5],[C4, C5, C6],...]

with dimensions: (1000, 3).

I am using a simple LSTM with 1 hidden layer (50 neurons).
I set up an LSTM with keras as:

n_features = 3
neurons = 50
ahead = 3
model = Sequential()
model.add(LSTM(input_dim=n_features, output_dim=neurons))
model.add(Dropout(.2))
model.add(Dense(input_dim=neurons, output_dim=ahead))
model.add(Activation('linear'))
model.compile(loss='mae', optimizer='adam')
model.fit(X, y, epochs=50)

This model works fine. Now, I'd like to predict the values of B as well (using the same input). So I tried to reshape the output in a similar way as I did for the training that has multiple features:

 y = [[[B3, C3],[B4, C4],[B5, C5]],[[ ...]]]

so that it has dimensions: (1000, 3, 2). However, this gives me an error:

Error when checking target: expected activation_5 to have 2 dimensions,
but got array with shape (1000, 3, 2)

I guess the structure of the network needs to change. I tried to modify model.add(Dense(input_dim=neurons, output_dim=ahead)) with no success. Should I reshape the y differently? Is the structure of the network wrong?

model = Sequential()
model.add(LSTM(input_dim=n_features, output_dim=neurons))
model.add(Dropout(.2))
model.add(Flatten())
model.add(Dense(ahead))
model.compile(loss='mae', optimizer='adam')
model.fit(X, y, epochs=50)

The differences include:

I'm flattening the output from LSTM, cause it's going to give a 3D tensor and a Dense layer for this scenario I believe is going to expect a 2D tensor.
Made the change the change suggested in another answer.
Removed the activation layer, since default activation of Keras dense layer is linear.


What is the role of $W_{ax}, W_{aa}, W_{ay}$ in forward propagation in RNN? Are they hyperparameters? Why are they needed? In RNN introduction in Coursera sequence model course, the following formula for forward propagation in RNN was introduced.  What exactly is the role of $W_{ax}, W_{aa}, W_{ay}$? What do they do?

In the lecture, it was told that:

$W_{ax}$: parameter governing connection from $x$ to hidden layer (not sure
what exactly does that mean: what happens if it is not governed?)
$W_{aa}$: governing activations (Why govern activations? What happens
if it is not governed?)
$W_{ay}$ governs output prediction (What is the point of governing that? What happens if it is not governed?)

In standard neural network these were the formula of forward propogation

$$
z_1 = w_1 X_1+b_1 \\
A_1 = g(Z_1)
$$
Consider there were only 4 layers then last layer
$$
z_4 = w_4X_4 + b_4 \\
A_4 \text{ or } \hat{y} = g(z4)
$$
I'm able to correlate these equation with  $a^{<1>}$ in RNN
but I am unable to correlate the role of $W_{ya}$ present in  $\hat{y}^{<1>}$  highlighted in yellow.
Please explain in very simple terms with an example on what is the job of $W_{ax}, W_{aa}, W_{ay}$ in forward pass in standard RNN.

Don't get hung up on the word "govern" here. $W_{ax}$, $W_{ay}$ and $W_{aa}$ are simply the weights and they play in principle the same role weights play in feed forward network (except that feedforward networks do not have $W_{aa}$):

$W_{ax}$ are the weights from your input layer to the first hidden layer (just as they are in feedforward networks)
$W_{ay}$ are the weights from your last hidden layer to the output layer (just as they are in feedforward networks)
$W_{aa}$ are the weights applied to the hidden state when it is fed from $t$ to $t+1$ (this is what you do not have in feedforward networks since they do not propagate through time)

You can also read more about this in the respective chapter of the deep learning book which, I think, provides a good explanation of RNNs.

EDIT (based on your comment):

In a feedforward network you would have $\hat{y} = g(W_{ay}\text{ }a+b_y)$ while your RNN has $\hat{y}^{<i>} = g(W_{ay}\text{ }a^{<i>}+b_y)$ (with $i$ being an index for the time step). There is really nothing new here with regards to the output layer in an RNN. In the below image you see to which connections the weights are applied in a feedforward net and in an RNN (I omitted the bias for simplicity):


Extract date/duration from text The text and output to be extracted are similar to the following :
"Check it every two weeks" - two weeks
"Check it on day 1 and day 14" - day 1 and day 14
"day 19 and day fourteen are important" - day 19, day fourteen
"The game is in 6 wks" - 6 weeks
"Check it in 6mo" - 6 months
"days 1 and d14 are important" - day 1, day 14
"Check it on days 11 and 14" - day 11, day 14
"check it on day one and twelve" - day one, day twelve
I have tried using SUTime library to extract the necessary information, but it just works in the case of first example and fails to properly extract information from the most of the other sentences. Considering the many ways in which the same text can be written, it is not very feasible to use ReGex.
I was wondering if there are any feasible NLP/ML solutions to extract this information. If NLP/ML is the best way to solve this, I have two questions: 1. I am not sure if I can use classification models and my first preference was to use a regression model, is it right to do that? 2. In either of the cases, I don't have a lot of labeled data to train. So, it'd be helpful if anyone can let me know if there's any open source labeled data for training+.

What you are trying to do here is named-entity recognition. Namely, the task consists of classifying substrings into a set of named entities (i.e. person, location, etc.). From a more formal perspective, this is a sequence labeling task that classifies parts of a sequence.
This task can be approached in different ways:

gazetteers/string matching
regular expressions
machine learning

I would highly recommend using SpaCy. They allow you to customize and retrain a model with your own data and labels, and typically, for such a use case, the model will perform well without requiring massive amounts of data.


Using partially defined features in an unified deep learning model Suppose we have two types of feature A and B. A is defined for all kinds of samples while B is only defined for some of the samples. Here, B is partially defined does not mean B is missing value (such as NaN) for the rest of the samples. In fact, we can not define a well-defined feature B in such samples.
For example, for stock market and commodity market, price feature (A) do exist in both of them. However, only commodity market has the concept of inventory feature (B).
My question is that how we can use feature A and B simultaneously in an unified deep learning model (instead of using them separately in two different models and combine their outputs thereafter)? Is there any frames or neural network structures can handle such problem ?

You can fill the B features with zeroes where they are not available, and add an extra 0/1 feature indicating whether the B features are present or not.
Then, you can fit whatever neural architecture is more suitable for the nature of the input data, e.g. dense layers with ReLU activations, LSTMs.


Confusion matrix - classes get confused with each other I got an example from this book on page 318: Chemometrics for Pattern Recognition By Richard G. Brereton. The example explained that; classes A and B are more often mistaken for each other than they are for classes C and D,..., as they may be more closely related or even can be better viewed as one single group.

Given a confusion matrix and assign some values to each class. How do you know which classes get confused with each other the most? Also, what it means by closely related or eve can be better viewed as one single group? How are A and B closely related and viewed as one single group in this example? Based on the value in the confusion matrix?

Example:

Chemometrics for Pattern Recognition
By Richard G. Brereton - Google Book

The confusion matrix as the name suggest helps to identify how many of the predicted classes are being confused with true classes. For instance, in this case, the rows indicate the predicted classes and the columns indicates the correct classes.

So if we consider the diagonal values, the predicted and the true class is the same. Hence, the diagonal values do not indicate any confusion. When we check the upper triangular part of this matrix,  the value 11 in your example indicates that 11 examples have been predicted as A when the correct values are actually B. Also, looking at (2,1)(the first value of second row) the value is 10. This implies that 10 values predicted as B actually are A. This shows that A and B are confused.

When two classes are closely related, it becomes harder to predict. For instance, an example I come across in my work is this. When I want to classify customer reviews, Speed of App and Speed of delivery classes seem to get confused because they have almost similar keywords in many of their examples. Since they both refer to speed, they are closely related and this leads to the confusion


How to make ML model publicly available? When I have finished building my machine learning model and want to make it publicly available for other users then what could be the best solution for that?
I mean I'm looking for a solution where there is a UI interface for an input/output and the ML code get executed through this UI.

Simple Jupyter Notebook - The code is not executable.
Jupyter Notebook + MyBinder + ipywidgets - nice UI can be build by the ipywidgets and the ML code is executable but still the user needs to execute all the cell boxes before he/she can use the ML model.
plotly/Dash - So far I have no experience by this. It is said that it can be used for building web applications with highly custom UI's, deploying apps to servers and then share them through URLs. Is it so that when using this then I need to have my own server where the ML code runs?
Is there any solution where I don't need to have my own server and can deploy and run my ML project?

The easier solution in my opinion consist to build your application using a framework like Streamlit and deploy it using docker.
Docker let you encapsulate your code and all needed package in a container, which you could think as an virtual machine (but its not); it will run on all OS since it use your configuration.
Of course there is plenty of other way but its my personnal favorite.


How to build a unbiased predictive ML model when the record of the event is less compared to the total number of records? I am trying to build a model that will predict the communication loss of a wireless device. For now I am using RandomForestClassifier along with Device and Location as the features. I am getting both the train score and test score as 99%. So I am pretty sure the model is giving biased result. One of the reason might be because the record of communication loss events are very less compared to the the record with no communication loss Some people advised me that it might not be possible to build a prediction model based on the situation. But I would like to have more suggestion or advice if there is anything I can do about it.

1) It seems that your data are unbalanced, you should look into that. Common techniques include oversampling the minority class, but you might have a bigger problem here.

2) It is unclear that you have enough information for what you are trying to achieve (type of device and location doesn't seems to be enough).

3) Based on the two preceding remarks, you have to acknoledge it is unlikely you will be able to get the date of an event. Would a human be able to guess that is a good question to ask yourself if you try to apply ML on a problem. If the answer is no, then a ML solution is unlikely to work. You can do statistical analysises to try to know if there are devices types or locations that are more susceptible of loss, but you won't be able to get exact time and location of a loss.


Constructing an image from a dense layer output I am trying to reconstruct an image from a dense layer with is a concatenation of outputs from a
1) convolutional network with image inputs; and
2) dense layer with numerical inputs

The concatenated 1D tensor is fed to a dense layer which I need to reconstruct as an image.

The code I am using right now is as so:

merge_output = tf.keras.layers.concatenate([convolutional_model_output, numerical_model_output])
densem1 = Dense(8092, activation='relu')(merge_output)
#densem2 = Dense(512, activation='relu')(densem1)
densem2 = Dense(self.image_width*self.image_height*3, activation='relu')(densem1)
reshapem1 = Reshape(target_shape=(self.image_height, self.image_width, 3))(densem2)

convm1 = Conv2D(filters=32, kernel_size=3, padding="same" , activation='relu')(reshapem1)
convm2 = Conv2D(filters=3, kernel_size=3, padding="same" , activation='relu')(convm1)

However, the training is not able to converge to the true output and looks like a kaleidoscope of weird colors.

Is there something wrong with my approach? I do realize that concatenating convolutional and numerical features and reshaping them as an image might be a possible pain point, is there a better way to represent the layer or the specific problem?

Thanks in advance.

Concatenating the two output layers is not such a pain as you might think. For starters, there are a number of things that can possibly go wrong with the model. Let's look at them one by one:

1) If the loss curve is unstable or is not coverging, try using batchnorm in the intermediate dense layers. Since there a lot of dense layers, it can lead to overfitting as well so try using dropout too. Experiment with batchnorm and dropout separately as well as together to see which one helps.

2) Check the range of the input and output of your decoder network: Normalise the data to be within 0 and 1 by simply multiplying the RGB values with 255. Use this normalised value as the true target value if your output image is same as input image or else use the normalised target image as your target output.  Similarly, to obtain the output value between 0 and 1, simply clip the output from relu to 1 and store it in a float variable as its a fraction. Use this float variable for computing the loss. Multiply it with 255 and store it in integer variable for visualisation purpose.

If these don't work you might want to look at other upsampling methods out there like: Deconvolution layers and Pixel Shuffle layers. You might also have a look at the decoder of generative models like GANs, specifically the CGAN (conditional GAN) which do a similar thing of merging information from an image with some known prior.


can I say that my variable is "approximately" normally distributed? I am studying a variable on 50000 observations, I have applied a cox box transformation to make it normal, but even with transformation the kolmogorov smirnof and the anderson darling test tells that my distribution is not normal,
given the qqplot that I have, and the number of observations can I say that my variable is approximately normal? I want to use parametric test but all the test rely on the normality assumption

Your data is already binned. You should do a Chi-Squared test to see if the hypothesis that your data is normally distributed is correct. You can refer to this question to see how you can go about this.

You will set your baseline as the Normal distribution with the mean and variance from your dataset. Then you will test the difference in expected value between the Normal distribution and yours.