Comparing dataframe object with string value in django I'm implementing machine learning model and using training dataset from MySQL table and all this is built upon Django. So basically all the calculations are done by converting entire data from MySQL table to dataframe.

df = pd.read_sql("select * from naivebayes_player",connection)

However, I'm facing problem in comparing dataframe column value with a string.

So I have a column named classification in MySQL table which has 2 fixed values 'RS' or 'NRS' stored in varchar(10) format. Since I've converted an entire table into dataframe whenever I calculate the count of 'RS' values in classification column in dataframe it always returns 0. But actually, there are 63 entries of 'RS'.

total_RS = df['classification'][df['classification']=='RS'].count()

In above line of code I'm trying to find out all records where classification is 'RS' which should be 63 but I'm getting 0. What am I doing wrong?

I have tried above code when reading data from CSV instead of MySQL table and everything worked fine.

I don't see anything glaringly wrong with your code, but you can try this instead:

df['classification'].value_counts()['RS']

If you omit the indexing at the end there, the call to value_counts() will concretely show you what values appear with what frequency in the classification column, which might help you debug what's going on if this still doesn't work.


Insignificant input variable but high R2 Can a linear regression model with a single input variable x with x not significant but the model has a high R2?

Can this exist? If yes, what are the reasons?

This can happen, but only with very small amounts of data. An example (in R) with three data points:

x <- c(0,1,2)
y <- c(-0.1,0.7,1.2)
summary(lm(y ~ x))

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  -0.0500     0.1118  -0.447   0.7323
x             0.6500     0.0866   7.506   0.0843 .
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.1225 on 1 degrees of freedom
Multiple R-squared:  0.9826,    Adjusted R-squared:  0.9651
F-statistic: 56.33 on 1 and 1 DF,  p-value: 0.08432

Notice the R-squared is very high, but the x parameter is not 95%-significant.

With lots of data, one cannot have a very high R-squared parameter without the corresponding parameter being significant.


Combining data sets with different IDs? I'm learning R and trying to do a comparison of MLB player salaries vs. WAR

One dataset contains all the salaries and the other contains WAR information, and they have their variables labelled differently. For example, each of the team names in one set is an abbreviation (ATL, NYY, HOU vs. Braves, Yankees, Astros). With that example, I'm sure I could find a way to make ATL == Braves, but with all the players that doesn't seem so easy.

The players are labelled in one dataset as the first 5 letters of the last name, first two letters of the first name, and then two numbers to differentiate matches. The other set simply has the names. An example would be Bryce Harper vs. harpebr03.

There's no apparent way to sort via alphabetical order and matching up each player's variable ID would take way too much time. Is there an easier way to link these differently named variables in R?

Here is some code that deals with the second part of your question regarding matching names. (BTW including sample data always helps)

require(dplyr)  ## for join

# create two example dataframes
d1 <- data.frame(player_first_last=c("Joe Smith", "Jack Johnson", "A Jones", "Slow Mo"), d1_value=c(1,2,3,4))
d2 <- data.frame(player_last_first2=c("SmithJo","JohnsJa", "MoSl"), d2_value=c(11,12,13))

# first two letters of first name using substr, and trimming whitespace
d1$player_first2 <- trimws(substr(d1$player_first, 1, 2 ))

# last name - regex capturing everything after the first whitespace
# this is a "regular expressions"
d1$player_last <- gsub(".+\\s(.+)", "\\1", d1$player_first_last)

# concatenate last 5 and first 2 using paste0
d1$player_last_first2 <- paste0(substr(d1$player_last,1,5), d1$player_first2)

# join on constructed name -- full_join in case there are differences
d3 <- full_join(d1, d2, by="player_last_first2")

Results in  d3

  player_first_last d1_value player_first2 player_last player_last_first2 d2_value

1         Joe Smith        1            Jo       Smith            SmithJo        1
2      Jack Johnson        2            Ja     Johnson            JohnsJa        2
3           A Jones        3             A       Jones             JonesA       NA
4           Slow Mo        4            Sl          Mo               MoSl        3


Can LSTM have a confidence score for each word predicted? LSTM networks can be used to generate new text. Given a sequence, I can predict the next word. Is there a way to get a score associated to each word predicted?

In particular, if the new word has never been seen by the LSTM network, can we train the LSTM to output a score of "no confidence"?

For example, this article gives the following example:

For example, let us consider the following three text segments:

1) Sir Ahmed Salman Rushdie is a British Indian novelist
and essayist. He is said to combine magical realism with
historical fiction.

2) Calvin Harris & HAIM combine their powers for a magical
music video.

3) Herbs have enormous magical power, as they hold the
earth’s energy within them.

Consider an LM that is trained on a dataset having the
example sentences given above — given the word “magical”,
what should be the most likely next word: realism, music, or power?

Say, that, in fact, the next word is neither one, but "power", but my LSTM has never seen that word before. So, the LSTM is going to predict one of the three words it has seen, but I would like it to output a low confidence score. Is this possible?

Couldn't you require your softmax output to exceed a threshold for the prediction or you call it "not confident"?

I have not tried word prediction, but the softmax gives you a value for each output, where all of those values sum to 1. So pick the column with the highest value as your predicted word and use that value to determine your confidence. (It may be fairly low, since there might be a small chance of lots of words and then larger chances of your most-common three.)


Unsupervised feature reduction for anomaly detection with autoencoders I am collecting a big number of generated numeric features for the task of unsupervised anomaly detection.

I can assume that all training data is considered normal.

I expect some of the generated features to be characterised with low standard deviation, for example, some features might be always 0 in the train examples. In contrast, I expect that some of these features will deviate in anomaly instances.

As I have a lot of feature, I strive to perform feature reduction / selection. However, using simple feature selection methods, will completely remove the non deviating features, affecting the upcoming detection for the worse.

I was thinking about using stacked auto-encoders for the sake of feature reduction, so that whenever a feature deviates a lot from the stdv it will affect all resulted features - causing a noticeable anomaly.

Will this technique work? if not, why? and what other technique could work for that.

Also, if it does, and I am planning of using deep auto-encoders for the sake of anomaly detection as well, is the first step of feature reduction redundent?

I have used stacked auto-encoders to reduce our 40 features step by step to 5 features and then output back to 40 features (some of my features were all zeros/ non deviating features). Training this on original (assumed to have no outliers) gives you a network which has learnt an abstract representation of the 40 features with 5 features.

When outliers show up these values are different and the network was unable to construct these particular data points back and the error scores for these values should be typically higher.

I used this with Deeplearning4j with SparkDl4jMultiLayer with KLDivergence loss functions on all layers except output layer with MSE loss functions. with Sigmoid activation function.


Tools and Techniques for Analyzing German Automotive Discussion Forum Posts I work for a German online disussion forum around all things automotive, a bit like a “StackOverflow for cars”, if you will.

We would like to train a model using TensorFlow with our high quality content, to be able to evaluate the quality of new content our users post on our platform.

Our ultimate goal is to be able to put a link to the best answer to a question on our discussion forums.

We (two backend Java developers and myself, a JavaScript frontend web developer) are very new to the field of data science and machine learning, currently going through the tutorials from Google and trying to figure out where to start.

Which tools and technologies would you recommend to use for this project?

How can we train a model to suit our needs?

Are there any tutorials that demonstrate how to train a model to use German text as input?

You could go with different approaches:

Lemme point few,

You could just extract the keywords or tokenize out of the query using libraries like spacy or nltk, both support german languages. Then go for like page ranking approach, based on query optimization: Uber.
You could go for Attention based seq2seq model, where you feed the inputs as questions and the answers as output. More like how you train chatbots and language translation models. This is popularly known as Neural Machine translation. Tensorflow has an open source implementation. nmt

The second one is feasible because it has tonnes of examples out there. But try to use spacy and nltk for tokenizing according to German language and also try to use German word embeddings: example, which contains pre-trained weights on huge corpora representing each German word.

Hope this helps.


How to read Several JSON files to a dataframe in R? I have a folder with 30,000 plus JSON file. A sample file with contents is posted below.

{
"name": null, "release_date_local": null, "title": "3 (2011)",
"opening_weekend_take": 1234, "year": 2011,
"release_date_wide": "2011-09-16", "gross": 59954
}

However, I need the data in a df in a structure as given below:

name relase_date_local title    opening_weekend_take year release_date gross
NA   NA                3 (2011) 1234                 2011 2011-09-16   5994

Here is my code snippet to get all the files as a list:

path = "./Week1/jsonfiles"
temp = list.files(path, pattern = "*.json")
filename = paste(path, temp, sep = "/")
movies = c()
for (i in filename){
  movie = fromJSON(file = i)
  movies = c(movies, movie)
}

Please advise, how can I read all 30,000 files as rows of a df?

First, you can use the full.names parameter to list.files() to get
the full path added to each file.

temp <- list.files(path, pattern="*.json", full.names=TRUE)

Next, there are issues with the data since they contain NULL values
which throws off a quick-and-dirty solution. So, we have to take each
list element and convert any NULL to NA.

Finally, we can use the handy purrr::map_df() to take the whole list
of lists and turn them into a data.frame:

movies <- purrr::map_df(temp, function(x) {
  purrr::map(jsonlite::fromJSON(x), function(y) ifelse(is.null(y), NA, y))
})


Best approach for this unsupervised clustering problem with categorical data? I'm a software engineer new to Machine Learning. I've read about basic non-supervised techniques like k-means and hierarchical clustering and now I'm trying to put them into practice with a basic problem.

Say I've got a lot of rows of data with each row looking something like this:

"employeesRange": "11-50",
"category": "Lobbying",
"categoryGroup": "Polication Action",
"sector": "Industrials",
"subCategory": "Direct",
"tags": [ "Progressive", "Libertarian", "PAC" ]

I want to analyze this data and look for patterns, i.e. clusters of information that frequently appear together. For example maybe it's a common pattern to find smaller groups of lobbyists 11-50 people for "Progressive", "Libertarian" causes but "Presidential" or "Local Politics" groups tend to skew larger.

Or maybe there's another link to be found between categories and sector, that kind of thing.

This seems a little harder then the examples I've read about because some of my data is a tag cloud (the "Tags" field) which is unstructured can contain multiple entries and the more structured fields of categories and employees range.

It seems a good place to start is using hierarchical clustering (since I don't know k a-priori) maybe following the recommendations here:

https://stackoverflow.com/questions/23943391/how-to-cluster-users-based-on-tags

I have a few questions about this. First, is this a good approach? And is Mahout the best tool for this? Any obvious ways to simplify? And finally, how can I combine the tag clustering approach with the other more structured data in the rows?

K-means is a reasonable approach and a sensible way to understand the data.
I've never used mahout, but I would use R or Python for this sort of analysis because of the nice libraries available to quickly implement K-means.
The clustering approach with the tags is fairly straightforward. You can essentially encode this using an indicator variable (also known as a binary encoding). You can set this variable/feature to 1 if the tag appeared in the list of tags and 0 otherwise. Then you only need to allocate space for the total number of tags that exist. If you have a large set of tags, you can limit them by taking tags with at least some frequency or some other "sensible" way.
You can choose $k$ in a number of ways. Typically people choose K arbitrarily because they want, e.g., 10 groups to segment their customers or data into. In the simulation I've provided, it'll give you a lame way to optimize for K using an incremental improvement.

I've made a notebook with a simulation walking you through how to "tokenize" your tags and represent them with a binary/one-hot encoding. It's worth noting that this tokenization ignores the order of the tags, which may be okay for your use case.

It's also worth noting that K-means certainly isn't the only way to measure the similarity of your data but I think it's a nice intuitive start.

Again, for choosing $k$ the approach outlined in the notebook is exhaustive, since it starts from 1 and goes until each observation is a cluster. This means you'll have to run K-means $n$ times, which is silly in practice to do but useful from a learning perspective here. In general, this isn't ideal because it's expensive and typically you don't want to set $k = n$ but this simulation gives nice intuition about what's happening.

In practice, you can just do it in gaps for a large number of clusters (e.g., 5, 10, 15, 20, .., 100) or something like that and choose the one that has the biggest drop-off by eye-balling it. This is a very arbitrary and unsatisfying way to choose $k$, but it seems to work okay for many people.


Price prediction based on historic data Im new to ML. I'm trying to predict if a new Music Album will exceed X amount of dollars in Sales.
I'm looking to build a model to go only after potential best sellers. I do have historic data for Music Sales from 2010 till 2016. I have many signals:

Music Genre
Music Band/Artist name
Label
Year released
Country of origin
Part of a Series/Volume... etc.
Sales per month

What type of ML problem is this one?

There are two broad classes of problems in machine learning, classification and regression. As in this answer, Regression involves estimating or predicting a response (the dependent variable is continuous). Classification is identifying group membership (the dependent variable is discrete).

Your problem is a regression problem, you must try to estimate the real number of sales. You can look here for a similar problem and techniques to solve it.


Why does gradient descent gives me much better Relative Squared Error then the Least Squares approach? Am I doing regression task with 7 dependent variables and 10000 data points. The SGD gives me 22% of mean absolute percentage error on test and train dataset. And Least Squares method using numpy scipy.optimize.least_squares give me only 58% (I have tried different settings.). I thought that Least Squares should give the same or better performance with such size dataset. What can be the reasons?

The reason is that the two metrics, mean absolute percentage error (MAPE) and mean square error (MSE) are optimising to different targets. Improving one can be done at the expense of the other.

As a simple example, consider this data:

x = [ 0,  1,  2,  3,  4,  5]
y = [ 3,  5, 10, 10, 11, 15]

The best fit mean squared error (MSE) for a line on this data is $\hat{y} = 2.23x + 3.43$, which has MSE of $1.18$, and a mean absolute percentage error (MAPE) of $11.0$%.

The best fit mean absolute percentage loss for a line on this data is $\hat{y} = 2.35x + 2.99$, which has a MSE of $1.24$, and a MAPE of $8.34$%.

You can see that optimising for MAPE gives a worse MSE, and vice-versa.

The difference can get extreme when there is a large range for y values (in terms of orders of magnitude covered), because optimising for MAPE will favour being more accurate on small values at the expense of larger ones. So if we change y to be:

y = [ 1,  2, 10, 10, 11, 20]

Then optimising mean abs percentage gives the line $\hat{y} = 3.78x + 1.05$ with MSE $7.09$ and MAPE $21.9$%. But optimising for mean square error gives $\hat{y} = 3.49x + 0.286$ with MSE  $4.56$ and MAPE $62.9$% - this is a larger difference, and I suspect that your data has a large range of target variable causing a similar effect.

You can potentially get closer results using the Least Squares regressor by using a transformed target variable $z = log(y)$ and transforming back at the end. This still won't be quite the same, but it does reduce the difference significantly - in my last example if I try this, I get MAPE $24.2$% - compared to $21.9$% for optimising MAPE directly.


Not sure if over-fitting I trained the data this way :
There are four classes, the data distributed evenly (same amount of labels).

Used min_max_scaler
Used train_test_split(X,y,test_size=0.3,random_state=42,stratify=y)
Ran GradientBoostingClassifier - once with n_estimators=32, and once with n_estimators=500 on the training data
Used predict on the test data
Got accuracy=0.94 on n_estimators=32 and accuracy=1 on n_estimators=500.
Precision and recall from classification report is also 1 for all class

Seems fishy but I can't figure out why... what am I doing wrong ?

Depending on your data, you may be overfitting, however that isn't necessarily the definitive answer.

Gradient boosted trees are a powerful algorithm and for a while performed as state-of-the-art. If your data happen to represent the target value in a systematic way that you haven't uncovered yet, it's likely that with 500 estimating trees the algorithm found a perfect solution. It's not unheard of.

On the other hand, I don't know much about your data. How many samples do you have? 100? 100,000? The former will be much easier to perfectly model. The latter may also be predictable (albeit less likely) if the variance between classes is predictable. The number of features may also play a role, and the significance of each feature.

As suggested in the comments, Cross Validation may help you discover what's going on here. I highly suggest reading the paper I linked above to see an example of rigorous CV. Carefully read what they did to see how you can model your own CV setup.

You might consider checking out the feature importance returned by your classifier. If one feature is significantly important, it might indicate a close correlation between that feature and the target variable (which should indicate that you need to take a close look at that feature).


Is there an implementation of pocket algorithm in Keras? As explained in Wikipedia, the pocket algorithm is a very simple variant/addition of/to ANN which keeps a copy of the best model seen so far and returns that one as the trained model (instead of the actual final state of the model). Implementing it is very simple and straightforward.

I was wondering if this algorithm is implemented in Keras.

Yes this is usually part of the early stopping algorithm, where you supply a cross-validation data set, and a limit on number of epochs since best result so far.

In Keras, you can use an instance of the EarlyStopping class, choosing the metric that you want the best model for, and setting the patience parameter to limit the number of epochs to test after any best so far result. The instance is supplied to the fit method as a callback.

See http://parneetk.github.io/blog/neural-networks-in-keras/ for an example (last section)


Wrong Graph Plot using K-Means in Python This is my first time implementing a Machine Learning Algorithm in Python. I tried implementing K-Means using Python and Sklearn for this dataset.

from sklearn.cluster import KMeans
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt

# Importing the dataset
data = pd.read_csv('dataset.csv')
print("Input Data and Shape")
print(data.shape)
data.head()

# Getting the values and plotting it
f1 = data['Area'].values
f2 = data['perimeter'].values
f3 = data['Compactness'].values
f4 = data['length_kernel'].values
f5 = data['width_kernel'].values
f6 = data['asymmetry'].values
f7 = data['length_kernel_groove'].values

X = np.array(list(zip(f1,f2,f3,f4,f5,f6,f7)))
# Number of clusters
kmeans = KMeans(n_clusters=7)
kmeans = kmeans.fit(X)
# Getting the cluster labels
labels = kmeans.predict(X)
# Centroid values
centroids = kmeans.cluster_centers_

plt.scatter(X[:,0], X[:,1],cmap='rainbow')
plt.scatter(centroids[:,0], centroids[:1], color="black", marker='*')
plt.show()

The graph doesn't seem to plot the data correctly. How can I debug this issue?

Well, there are some issues:

Dimension vs K: Before talking about visualization I would like to address some clustering concept. Your data is in 7 dimensions but it does not mean that you have 7 clusters! Be careful here. For instance I have two features of people let's say salary and number of years they have working experience. Here I have two features but does it mean that there necessary two categories inside the data? sure not!
Visualization: Your data is in 7 dimension which is not visualizable. So you decided to reduce this to two which is a correct approach but you did a wrong thing for this correct approach. You can not take the first two features to visualize 7 dimensions, you need to REDUCE it to two features using Dimensionality Reduction algorithms like PCA, NMF, etc. What you did is actually IGNORING 5 dimensions of the points which are extremely informative for placing them in a 7-dimensional space.

Solution

Everything is right. Just add a PCA to your code like this:

From sklearn.decomposition import PCA
Model = PCA(n_components=2)
X_new = Model.fit_transform(X)
... Use X_new instead of X for K-means procedure

Please note that I wrote this relying on my memory so better to check the documentation if I had a typo or smth. In case you have more question you can comment here.

Good Luck!


Predicted and true values distributions comparison Is this alarming when a distribution of predicted values differs from a distribution of true values? I use xgbregressor and get the following plots

Usage of boxcox doesn't improve the case.
My data is spatial-temporal. I make a cash-flow forecasting for some city and time is treated like 12 features corresponding to months that I feed to xgb.
 The figure shows data for one year.

I don't know the method you're using but I suspect that what you observe here is a common problem with supervised learning: models tend to favour predictions close to the mean, that is avoid extreme predictions because these are usually more risky (higher loss if it's a mistake). As a consequence the std deviation of the predictions is often significantly smaller than the s.d. of the ground truth.

Afaik there's no perfect solution. Typically you could try to encourage risky predictions a bit more in the loss function, if that's an option with your method. But in most applications it's safer to learn to live with this issue.


How to handle Overfitting I am working on machine learning classification problem with two classes (0/1). I would like to build a prediction model. The problem is that I have a small dataSet of shape=(89, 21) which may caused over-fitting. problem (20 independent variables). I notice that the results are highly affected by the train data and test data sizes( i.e. how the split was done ). the best results with LR was 0.90 and the worst I got 0.74.
Algo
I Split data using this instruction :
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.37, random_state=2)

then I did an upsimpling of training set since it was unbalanced {0}=62 {1}=27
the  table below  contain the best results on the test-set I got (test error).
$$\begin{array}{c|c}
 & \text{Accuracy} & \text{ROC } \\ \hline
\text{LogisticRegression } & 0.909091 & 0.928571 &  \\ \hline
\text{DecisionTree } & 0.939394 & 0.934524 &  \\ \hline
\text{adaBoost } & 0.848485 & 0.845238 &  \\ \hline
\text{RandomForest } & 0.878788 & 0.928571 &  \\ \hline
\text{GradientBoosting } & 0.848485 & 0.827381 &  \\ \hline
\end{array}$$
the next table contain the training error
$$\begin{array}{c|c}
 & \text{Accuracy} & \text{ROC } \\ \hline
\text{LogisticRegression } & 0.963415 & 0.963415 &  \\ \hline
\text{DecisionTree } & 1.00 & 1.00 &  \\ \hline
\text{adaBoost } & 1.00 & 1.00 &  \\ \hline
\text{RandomForest } & 1.00 & 1.00 &  \\ \hline
\text{GradientBoosting } & 1.00 & 1.00 &  \\ \hline
\end{array}$$
I need some guideline (ideas, tutorials,...) about how to manage the over-fitting problem .
Thanks

This is a very general question, however, there are many different solutions as explained below. For your case, probably, item 2 is not the case because you can not gather a large number of data points. I would recommend using solutions 1, 3, 5, and 6 (I see you used this method but try to combine it with other solutions such as cross-validation, regularization, and feature selection).

Cross-validation: Use the initial training dataset to produce multiple mini train-test splits. Use these splits to tune your model.

For example, in k-fold cross-validation, partition the data into k subsets. Then, train the model on k-1 folds iteratively while using the remaining fold as the test set. In this way, you can use the cross-validation to tune hyperparameters with only the original training set.

Train with more data: Try to use more data points if possible.

Perform feature selection: There are many algorithms that you can use to perform feature selection and prevent from overfitting

Early stopping: When you’re training a learning algorithm iteratively, you can measure how well each iteration of the model performs.

Up until a certain number of iterations, new iterations improve the model. After that point, however, the model’s ability to generalize can weaken as it begins to overfit the training data.

Use regularization. As also will be discussed in item 7, the higher the complexity of the model the higher the chance of overfitting. For example, in the case of logistic regression, when the weights are large, the model gets complicated and it probably won't work on the unseen test dataset. Regularization helps to decrease the weights and so the complexity of the model.

Use ensembling methods such as Random Forest and Gradient Boosting. One of the main issues with decision trees is that they are prone to overfitting; i.e., high variance, it means that they work very well on training data but not on the unseen test dataset. One solution to prevent overfitting in the decision tree is to use ensembling methods such as Random Forest, which uses the majority votes for a large number of decision trees trained on different random subsets of the data.

Simplifying the model: very complex models are prone to overfitting. Decrease the complexity of the model to avoid overfitting. For example, in deep neural networks, the chance of overfitting is very high when the data is not large. Therefore, decreasing the complexity of the neural networks (e.g., reducing the number of hidden layers) could help to prevent overfitting.

Drop out method. In deep neural networks, randomly dropping some of the connections between layers by multiplying noise sampled from a Bernoulli distribution could help to prevent overfitting.


How to train a model on a data where there are multiple data inside a data point? I'm trying to do prediction on capacity column, however each data point consist of more data.

Each data point represent a cycle data. Each cycle has a capacity. Each cycle runs for some time duration, and in that duration some data is collected over which capacity is dependant

I tried exploding the dataset and copying the capacity values to each row, but that shouldn't be the case because each row will get different capacity predicted. Is there a way to train such kind of dataset?

If your list are of fixed size you can separate them in different columns.e.g. load1,load2...
If thats not the case, you need to define some statistics for the cycles like average load, max_load, min_load..


Spacy - Ignored (essential) entities during labeling - how important is this aspect? I'm training a custom Spacy model. Does it affect the results whether I leave entities unlabeled? For example, Germany is LOC. In one example (1) I label this. In the other example (2) I ignore it and label another entity:
Example 1: Germany (LOC) is a country in Central and Western Europe.
Example 2: Germany borders Denmark (LOC) to the north.
It's not about one entity that is occasionally ignored, but about several.

Yes, it matters. A lot. You need to label every entity you encounter in each sentence. As long as they are non-overlapping, you can add as many entity types and entities per document as you'd like.
First of all, it matters because, in your example, your model receives one example of "Germany" that is a LOC and one that isn't. So, it must mean that the context around "Germany" determines whether it is an entity. Not labeling your entities in some sentences works against the very task you are trying to learn.
In their documentation, they show how to define training data with multiple entities of the same type per document:
TRAIN_DATA = [
    ("Who is Shaka Khan?", {"entities": [(7, 17, "PERSON")]}),
    ("I like London and Berlin.", {"entities": [(7, 13, "LOC"), (18, 24, "LOC")]}),
]


How do I deal with missing values in a data set? I am trying to build a binary classification model which predicts whether a patient would me infected with a certain disease at the the end of his hospital stay or not. The features that I have are results of different standard medical tests. But the issue is almost all of these results have around 60% - 80% missing values as not all the tests are relevant for all the patients. So how do I deal with the missing values as dropping them is not an option here.
Also since the medical test results lie on scale ranging from low to high, should i converted them to categorical variable with High, Low, Medium, Null (for missing data), based on the standard medical test ranges?

If the features are categorical, just fill the NAs with "Missing" as a new category. If they're continuous there are a number of approaches you could try. As a starting point, you could simply fill them with the mean value (or median if you have outliers significantly skewing things) and add a new binary feature that flags the value as being missing in the original feature. So...

| Feature A |
| --------- |
|    14     |
|    nan    |
|    23     |

becomes

| Feature A | Feature A Missing |
| --------- | ----------------- |
|    14     |          0        |
|   18.5    |          1        |
|    23     |          0        |

I've never found a hard and fast rule for the "best" way of doing things; I suggest trying the above and cross-validating your model to make sure its performance is within acceptable limits.

Discretizing continuous variables into High/Medium/Low/Missing may also be helpful; again, try it and cross-validate to see if it improves performance. You should try it both with and without the original continuous variables too.


How do I assign specific values to categorical variables I have a Pandas data frame with columns within a survey with the following categorical values -  "Increased, Decreased, Neutral". My question is how can I assign specific numerical values to these categorical values, namely +1 for Increased, -1 for Decreased and 0 for Neutral.

One possible way to map from string values to specific numerical values is by using a Python dictionary as a lookup table. The lookup table can be used for each value in the column with .apply(func) on the column.
import pandas as pd
l = [{'col1':'Increased'},{'col1':'Decreased'},{'col1':'Neutral'}]
df = pd.DataFrame(l)
print(df)

Output:
        col1
0  Increased
1  Decreased
2    Neutral

Create mapping and apply:
value_map_d = {'Increased':1,'Neutral':0,'Decreased':-1}
df['col1_numerical'] = df['col1'].apply(lambda x: value_map_d.get(x))
print(df)

Output:
        col1  col1_numerical
0  Increased               1
1  Decreased              -1
2    Neutral               0

In the example above, I read the values column col1 and write the numerical codes to a new column in the data frame called col1_numerical. The original column is dtype object and the output column is dtype int64 because all the values retrieved in the value map are integers.


Q learning and Neural Network for Tic Tac Toe I have been working on a tic-tac-toe assignment for my Robot Learning class. We were asked to program a tic-tac-toe game and assign; +1 if X wins, -1 if O wins and 0 it the game results with a draw. In part 1, we were told to use Q table and in part 2 we were told to replace the Q table with a Neural network as the functional approximator.

It is my understanding that both methods should be achieving the optimal policy, can you confirm or deny my understanding?

It is my understanding that both methods should be achieving the optimal policy, can you confirm or deny my understanding?

Yes, I would expect a neural network for Q Learning to find the optimal policy, provided it remains stable*. The value estimates might be slightly inaccurate, but the resulting policy should be completely optimal. That is because in tic tac toe, all the value estimates should be -1, 0 or +1, and the data is cleanly separated.

You should be able to get a neural network to learn the optimal Q table from the first experiment using supervised learning. In fact that would be a good test of whether your NN has capacity to learn that table.

* Neural networks added naively to Q-learning agents are often not stable. In fact that is so common a problem in scaling up RL agents that it has a name: "the deadly triad". This is generally not solved by elegant mathematical changes to the agent, but by some engineering tricks:

Experience replay. Save observations (S, A, R, S') and sample from this memory table later to train in mini-batches.
Alternating networks. Use an old frozen copy of the neural network to calculate $\text{max}_{a'} Q(S',a')$ for the TD target $R + \gamma\text{max}_{a'} Q(S',a')$


Multicollinearity in Decision Tree Can anybody please explain the affect of multicollinearity on Decision Tree algorithms (Classification and regression).
I have done some searching but was not able to find the right answer as some say it affects it and others say it doesn't.

Desicion trees make no assumptions on relationships between features. It just constructs splits on single features that improves classification, based on an impurity measure like Gini or entropy. If features A, B are heavily correlated, no /little information can be gained from splitting on B after having split on A. So it would typically get ignored in favor of C.

Of course a single decision tree is very vulnerable to overfitting, so one must either limit depth, prune heavily or preferly average many using an ensemble. Such problems get worse with many features and possibly also with co-variance but this problem occurs independently from multicolinearity.


Why dont we use 2d cnn filters for Nlp tasks? CNNs are used in NLP for various tasks. But I cannot find a clear understanding of why do we only use 1d filters in these networks?

Text is a 1D sequence, but is typically treated as a sequence of embedding vectors. So yes it is in some sense 2D input. But the embedding dimension doesn't really have any spatial meaning; adjacent dimensions aren't any more related than any others. There is no invariance across the embedding dimension either; the same values in one part of the embedding don't mean the same thing. So the assumptions a 2D convolution don't make sense for this type of input.


n_jobs = -1 equivalent in keras I have recently started learning deep learning. In machine learning using sklearn library with n_jobs = -1 all my cpu cores are used and this speeds the grid search. Now I am trying to fit an rnn model on training data, which is taking a lot of time. Is there a way I can speed up the training?

# Initialising the RNN
regressor = Sequential()

# Adding the first LSTM layer and some Dropout regularisation
regressor.add(LSTM(units = 50, return_sequences = True, input_shape = (X_train.shape[1], 7)))
regressor.add(Dropout(0.2))

# Adding a second LSTM layer and some Dropout regularisation
regressor.add(LSTM(units = 50, return_sequences = True))
regressor.add(Dropout(0.2))

# Adding a third LSTM layer and some Dropout regularisation
regressor.add(LSTM(units = 50, return_sequences = True))
regressor.add(Dropout(0.2))

# Adding a fourth LSTM layer and some Dropout regularisation
regressor.add(LSTM(units = 50))
regressor.add(Dropout(0.2))

# Adding the output layer
regressor.add(Dense(units = 1))

# Compiling the RNN
regressor.compile(optimizer = 'adam', loss = 'mean_squared_error')

# Fitting the RNN to the Training set
regressor.fit(X_train, y_train, epochs = 100, batch_size = 32,shuffle=False)

Here you do a single fit() of the model whose name tells for itself - Sequential.

Unless you are doing cross-validation or some kind of distributed learning with multiple models, there is no benefit of running several fits in parallel.

However, you can have significant speed-up on iteration level, depending on how Keras backend is configured to work with your hardware.

Assuming you are using Keras with the tensorflow backend (default):

If you are running on CPU, tensorflow should already pick up all available cores by default (if not, it might help checking How to run Keras on multiple cores?);
If you are running on GPU:

(1) make sure Keras backend can see your GPU

from keras import backend
print(backend.tensorflow_backend._get_available_gpus())

If your GPU is not showing up here, some configuration might be necessary
on your system (it deserves another topic);

(2) take a look at the CuDNNLSTM - it is the Keras LSTM implementation
optimized for GPU.


CNN 1D for time series. Value Error. I did read all the possible answers in StackOverflow regardiing this problem I been fighting with this problem for 2 weeks now. And I extensively research for solutions here and in other sites.
I have a dataset of 4 dimensions device_id, time and longitude and latitude.
After preprocessing it look like this i got this shape

data1.shape
(95751, 4)

data1cnn2 = data1.values.reshape(1,95751, 4)

data1cnn2.shape

(1, 95751, 4)

y.shape

(95751,)

Then this is the code for the CNN1d . using Tensor-flow

modelcnn = Sequential()
modelcnn.add(Conv1D(filters=64, kernel_size=2, activation='relu', input_shape=(95751, 4)))
modelcnn.add(MaxPooling1D(pool_size=2))
modelcnn.add(Flatten())
modelcnn.add(Dense(50, activation='relu'))
modelcnn.add(Dense(1))
modelcnn.compile(optimizer='adam', loss='mse')

then I fit the model

modelcnn.fit(data2cnn, y, epochs=1000, verbose=0)

and I get this mistake

ValueError: Input arrays should have the same number of samples as target arrays. Found 1 input samples and 95751 target samples.

modelcnn.summary()

Layer (type)                 Output Shape              Param #
=================================================================
conv1d_2 (Conv1D)            (None, 95750, 64)         576
_________________________________________________________________
max_pooling1d_2 (MaxPooling1 (None, 47875, 64)         0
_________________________________________________________________
flatten_2 (Flatten)          (None, 3064000)           0
_________________________________________________________________
dense_3 (Dense)              (None, 50)                153200050
_________________________________________________________________
dense_4 (Dense)              (None, 1)                 51
=================================================================
Total params: 153,200,677
Trainable params: 153,200,677
Non-trainable params: 0
_

I also try with different solution but got different type of mistakes

I did

data1cnn = data1.values.reshape(95751, 4,1)

data1cnn.shape
(95751, 4, 1)

but i got this other mistake

ValueError: Error when checking input: expected conv1d_2_input to have shape (95751, 4) but got array with shape (4, 1)

modelcnn.summary()
Layer (type)                 Output Shape              Param #
=================================================================
conv1d_2 (Conv1D)            (None, 95750, 64)         576
_________________________________________________________________
max_pooling1d_2 (MaxPooling1 (None, 47875, 64)         0
_________________________________________________________________
flatten_2 (Flatten)          (None, 3064000)           0
_________________________________________________________________
dense_3 (Dense)              (None, 50)                153200050
_________________________________________________________________
dense_4 (Dense)              (None, 1)                 51
=================================================================
Total params: 153,200,677
Trainable params: 153,200,677
Non-trainable params: 0

I am completely stuck and I have read all the question and answers that are here . With no success

Your first error is because Keras wants the batch size as first dimension. Given that your Y has 95751 observations, your X must be reshaped in order to have 95751 records in the first dimension, as you did in the second part.

The code doesn't work because Keras has already put the first dimension (None = batch_size) for you (see model.summary()), so you don't need to specify that in input_shape. You need to change this

input_shape=(95751, 4)

into this

input_shape=(4, 1)

Update after comment - working example

Here's a working example of your code.

First I define X and Y following your shapes. Here I use random numbers given that I don't have your data

data1cnn = np.random.rand(95751, 4, 1)
y = np.random.rand(95751,)

I check the shapes to be sure that they are the same as the ones you have provided in your second part of the question

print('X and Y shape')
print(data1cnn.shape, y.shape)

Output

X and Y shape
(95751, 4, 1) (95751,)

Looks correct. Now I define the model: everything is the same as in your code but input_shape=(4, 1)

modelcnn = Sequential()
modelcnn.add(Conv1D(filters=64, kernel_size=2, activation='relu',
input_shape=(4, 1)))
modelcnn.add(MaxPooling1D(pool_size=2))
modelcnn.add(Flatten())
modelcnn.add(Dense(50, activation='relu'))
modelcnn.add(Dense(1))
modelcnn.compile(optimizer='adam', loss='mse')

I print the summary

print('Model summary')
print(modelcnn.summary())

Output

Model summary
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv1d_1 (Conv1D)           (None, 3, 64)             192
_________________________________________________________________
max_pooling1d_1 (MaxPooling (None, 1, 64)             0
_________________________________________________________________
flatten_1 (Flatten)         (None, 64)                0
_________________________________________________________________
dense_1 (Dense)             (None, 50)                3250
_________________________________________________________________
dense_2 (Dense)             (None, 1)                 51
=================================================================
Total params: 3,493
Trainable params: 3,493
Non-trainable params: 0

Then I train it using the same command you wrote in your question but verbose=1 to print some information during training

modelcnn.fit(data1cnn, y, epochs=1000, verbose=1)

and it trains without a problem given that I get as output

Epoch 1/1000
95751/95751 [==============================] - 6s 60us/step - loss: 0.0844
Epoch 2/1000
95751/95751 [==============================] - 7s 76us/step - loss: 0.0833
Epoch 3/1000
95751/95751 [==============================] - 6s 59us/step - loss: 0.0832
Epoch 4/1000
95751/95751 [==============================] - 6s 66us/step - loss: 0.0831
Epoch 5/1000
95751/95751 [==============================] - 5s 54us/step - loss: 0.0831
Epoch 6/1000
95751/95751 [==============================] - 5s 54us/step - loss: 0.0830


How often do we see normally distributed data I am having difficulty in exactly understanding several statistical tests, such as the t-test and ANOVA test. These tests require that the data we use be normally distributed.

However, whilst sharing my experience in analyzing data a bit, I have analyzed several data sets from numerous sources online (web scraping, open-accessed data sources online, etc.), with the considerably high number of samples (hundreds, thousands). An example of the data in question is the amount of donation given to certain campaigns in a fixed periods of time (day 1 at 1pm, day 2 at 1pm, etc.).

And when I tested whether the distribution of the data was normal, using visual aids (histograms, Q-Q plots) and Shapiro-Wilks test, they all showed me that the data is not normal. For example, Shapiro-Wilk test gave a p-value of so small (less than 0.00000000000000022), of course, the null hypothesis has to be rejected, i.e. the data is not normally distributed.

Because I read in articles like in this link, it says

  However, even if the distribution of the individual observations is not normal, the distribution of the sample means will be normally distributed if your sample size is about 30 or larger

So naturally, I am confused, is my data normally distributed or not? How often do you encounter normal and not-normal distribution, in real-life data?

EDIT
Following the response by @hssay in his answer and comments, my main objective here is that I want to do ANOVA test to determine the relationship between my numerical and categorical data. But ANOVA needs the data to be normally distributed. So now I am confused as to how to conduct it since I have a "sample" consisting of thousands of rows of data that I took only once.

Your confusion is apt. Normally distributed data doesn't show up that often. Most of the real world datasets are more complex than normal. Many of natural occurring phenomenon (think: height of people in certain population) might be normal. But most of the cases where human behaviour plays a strong role (donations as you mentioned, incomes, peoples preferences) will show other distributions like fat tailed distributions or power law distributions.

But the result that you highlighted talks about a result in statistics called central limit theorem, which states that the mean that you infer from averaging will be normally distributed (irrespective of distribution of data). I'll explain below with an example.

Imagine you want to talk about heights of all males in United States. The first question that you might ask about such data is what's the central tendency (mean). But you may not have data about every male in the United States (getting that data will be too expensive). So you take a sample of let us say 100 people from each state. There's no way for you too know the distribution or shape of heights the entire population (you haven't collected that data!). You take the sample that you collected and calculate average of the heights. Can you make some statements about the how close is the average to the real mean of entire population? The computed average after sampling is a random variable since you will get different answer for every different sample. Central limit theorem says that this random variable is normally distributed with its mean same as the population mean and your estimate will get pretty closer as you have bigger samples. (Central limit theorem works in case of i.i.d. samples, that is samples are independent of each other and they are picked from same population.)

So in summary, normal distribution shows up more often with statistical tests because you are talking about distribution of average of sampled data, not the actual distribution of data (which can be non-normal). In a way, central limit theorem is why most of real world applications of statistics from simulations to election studies (psephology) works!

All the theory above answers the initial part of your answer. But for specifically running ANOVA test, you need the data to be (close) to normal distribution. Look at histogram of the data (yes, I know you have done sampling only once) and you can run tests for normality assumptions.


Yarn service parameter for pseudodistributed I'm trying to set up Hadoop in the pseudodistributed form. So I edited the suggested xml files. In the file yarn-site.xml, I had to insert:

<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>

What is the "mapreduce_shuffle" option?

The mapreduce_shuffle in this config file is part of Plugable Shuffle and Sort.  Shuffle and Sort are what connect the mappers to the reducers. A nice graphical representation of this is the following (shuffle is called "copy" in this figure):

The Hadoop arcitecture alllows these steps to be customized:

  The pluggable shuffle and pluggable sort capabilities allow replacing the built in shuffle and sort logic with alternate implementations. Example use cases for this are: using a different application protocol other than HTTP such as RDMA for shuffling data from the Map nodes to the Reducer nodes; or replacing the sort logic with custom algorithms that enable Hash aggregation and Limit-N query.

The mapreduce_shuffle parameter is mentioned in the Hadoop Docs, and is the default value for this configuration.


Temporal Aspects in Machine Learning Concept drift means that the statistical properties of the target variable, which the model is trying to predict, change over time in unforeseen ways.

With reference to the classic house price prediction use case:

House prices change over time thus the model I use today could make no sense in the future.

What is the best approach to address concept drift?

Do we keep updating the input replacing older house prices of yesteryear?
Do we add an extra feature for Date of Sale - by including a temporal aspect as a feature with larger data sets?
Do we eventually change model hyperparameters during training to build a model that fits better the new data?

It's not really possible to adress concept drift in general. But I can bring two similar answers for drift of houses prices :

As other prices the drift is usually well measured and studied. As one would correct price for inflation, one can correct past house prices with a housing index (typically this index for the US). It will help your model having prices that are comparable over years.

Another way to tackle drift is to consider a ratio with a relevant variable that has a similar drift. For housing price, that might be median income of the neighborhood. This will give you a variable that is less sensitive to the overall drift.

As you can see those two methods are pretty much equivalent here in practice, as it mainly consist in correcting features and eventually, targets. The main difference is that in the first case you talk about dollars directly which is often more business oriented. Application of those methods can get a bit difficult if you try to use your model to predict the future and need to project housing index or median wage.


400 positive and 13000 negative: how to split dataset up (train, test, validation) Working on a medical diagnostic convolutional neural networking problem, and it's not obvious (to me) how the dataset should be split up.

Do I have enough data to split it in 3, or should I just have train and validation?

What proportion of images should I put in each?

(research article links appreciated :])

For the problem of an imbalanced dataset, you can look into stratified sampling, or stratified-cross-validation (as mentioned here). One idea might be to create stratified batches from the data.

I would probably make all attempts to get train/val/test splits, because you will otherwise face issues when claiming a final test accuracy, as the model might well have seen your entire dataset.

One could imagine splitting to have e.g. 300/9750 (pos/neg) in the training dataset, and during training, you create stratified batches from those 1050 images, so each batch e.g. of 50 images, might contain, 10 positives and 40 negatives. This is still somewhat imbalanced, but you are pushing the balance into a more favourable direction in that the model should be able to learn more efficiently.

In medical research it is often the case that there are too few samples (in addition to class imabalnces), and so there is usually a huge effort that goes into data augmentation, which you might also be able to make use of it. Here is some related literature  (extremely fresh - edited one week ago!).

Here is another approach, whereby the authors (William Fithian & Trevor Hastie) devise a subsampling method, which uses the features of the samples to accept/reject them. They design it for the simplist case (logistic regression), but perhaps it might give you ideas:

  ... using a pilot estimate to preferentially select examples whose responses are conditionally rare given their features.

Something to be especially aware of when using the ideas I mentioned above is overfitting. Cross-validation is probably what can best help you out in this respect.


Are word embeddings further updated during training for document classification? I am relatively new to the area of using word embeddings in NLP tasks. From a large corpus of documents, I train word2vec word embedding vectors and afterwards I am going to use these for document classification, combined with RNN based classifiers (LSTM, GRU), which is a pretty standart pipeline nowadays.

There is one issue; that should we update the word embeddings during the document classification training as well.

I am used to tasks like image classification/object detection in the past. You get an image input and the convolution features extracted from that image are updated during the numerical optimization of a CNN. But the image itself is never updated, naturally, since it is the original data.

How do we treat the embedding vectors in the world of text documents? They are not "natural" exactly like the images, we learn them from an unsupervised method first (word2vec, GloVe or any other tool) so I think they can be further fine-tuned during the supervised training. Is it the common practice to update the embedding vectors as well as the RNN parameters during the training of the sequence classifier or should we leave them as constant (in order to avoid overfitting) ?

Word embeddings are generally used as input features, which like you noticed for image based models, do not get modified during training.

It is in fact quite difficult to update embeddings during training (or at all after they have been computed!), because they are often in some latent space that holds information regarding their relationships to one another. It is for this reason that adding new vocabulary to existing embeddings is extremely challenging, and updating computed embeddings is also difficult without using all the data that was originally used to create them.

I am afraid I don't have an example I can point to where embedding are used during training and simultaneously updated/improved upon. I can imagine that there are ways to store them, however, such that it is technically feasible (although perhaps memory intensive!)


How to predict churn events that may happen within a period of time? I am trying to build a model that predicts churn events in the future. The business wants to be able to identify which customers are likely to terminate the services within a month. "Within a month" can mean the next day or the 30th day. The problem is some of the features are time-based, for example how many months into the current term, the number of tickets created in the last two weeks, etc. If the event date is floating, how do I calculate the values of these features? Should I make 30 copies of the same churned service and calculated these time-based features for each of them? Is there a better way to approach this?

The problem is similar to modeling defaults of a subset of companies in a specific basket, big or small. The problem is very well-researched, drawing from methods developed in asset pricing, actuarial science and survival analysis. For a good reference, you may look into
Duffie & Singleton (2003). Credit Risk: Pricing, Measurement, and Management.
The most popular and flexible solution is modeling the churn evens as correlated counting processes with intensities that depend on the predictors that you have mentioned above + some other predictors. Counting process is an extension of Poisson process.


Layer notation for feed forward neural networks Apologies in advance, for I have a fairly rudimentary question on the notations for studying Feed-Forward Neural Networks. Here is a nice schematic taken from this blog-post.

Here $x_i = f_i(W_i \cdot x_{i-1})$ where $f_i$ is the activation function. Let us denote the number of nodes in the $i^{\text{th}}$ layer by $n_i$ and each example of the training set being $d-$dimensional (i.e., having $d$ features).

Which of the following do the nodes in the above graph represent?

Each one of the $d$ features in every example in the training set. In this case, $n_0 = d$ and $x_0$ is $(d \times 1)$.
Each example of the training set, which is $d-$dimensional. In this case, $n_0$ is the number of examples and $x_0$ is $(d \times n_0)$.

In both cases, the weight matrix $W_i$ is $(n_i \times n_{i-1})$.

On the one hand, most references like this blog-post claim it is (1), while on the other, I can also find few references such as this video which seem to claim it is (2). Which one of them carries the right interpretation?

Although seems like the back propagation algorithm can be executed in both representations, I'm quite sure it only makes sense in one of them. Any help will be greatly appreciated.

Well, the image you sent is not nicely denominated.

The first layer in the image is $x_0$ which is the input consisting of d dimensions, it is actually the first sample of the training set. Here its dimensions are $x_{01}, x_{02}, x_{03}, x_{04}$ (the green nodes in the left, hence, d equals $4$). Then the next layer which is called $x_1$ is the first hidden layer and subsequently $x_2$ is the second hidden layer and $x_3$ is the output of this feed-forward network.

By this definition, $x_0$ is the input with d dimensions $x_{01}, x_{02}, x_{03}, x_{04}$ and for calculating each node in the proceding hidden layer which is called $x_1$ here, we should do:

consider the most up node in hidden layer $x_1$ as the node we want to figure its value. We call it $x_{11}$, first we compute a linear computation of weights and inputs and then we apply some activation function $\sigma$ to it:
$$x_{11} = \sigma(x_{01} \cdot w_{11} + x_{02} \cdot w_{12} + x_{03} \cdot w_{13} + x_{04} \cdot w_{14})$$

also, an offset might be added to this expression.
consider that hidden layers can be of any size.

  Each one of the $d$ features in every example in the training set. In this case, $n_0 = d$ and $x_0$ is $(d \times 1)$.

$n_0 = d$ and $x_0$ is $(d \times 1)$ is right and in the first layer, yes each node is depicting a single one of d features of the input. but not for the hidden layers.

  Each example of the training set, which is $d-$dimensional. In this case, $n_0$ is the number of examples and $x_0$ is $(d \times n_0)$.

No as I mentioned, this is an architecture that depicts the process for a single training set. Hence, each node is not a sample of the training set.
You set $n_0$ for the number of nodes in the first layer which is input. so $n_0$ here equals d and $x_0$ which is the input equals $(x_{00}, x_{01}, x_{02}, ...x_{0d})$, $0$ showing that this is the first sample of training set.

In the backpropagation process, we have the same architecture. Then by calculating the gradient of each node, we update each weight. this process is done so many times to find the most optimal weights. there are various approaches for this weight updating thing like batch, etc updates.


Predicting similarity between nouns like university names and tech companies? I am trying to extract entities like university studied at and tech companies from resumes , I have a list of popular universities and companies and I want to find out which university best matches the extracted entity

Example

1) University in the list : IIT Bombay

Extracted entity : Education : Indian Institute of technology Bombay Btech

2)University in the list : Infosys

Extracted entity : Infosys India Ltd.

As you can see , there are extra unwanted words , short forms , expanded forms etc recognized in the extracted entity , is there any sentence similarity algorithms best suited for this purpose ?

Using SpaCy for entity extraction.

This is a case of entity resolution for which a standard method is not available. You will have to write your own method also using abbreviation resolution. The python Dedupe package has some distance metrics which you could use to calculate the similarities.


Changing a list of tensors to a tensor of lists! I have a list of tensors created by
[some_function(x) for x in something]

and this list becomes like this
[<tf.Tensor: shape=(), dtype=string, numpy=b'I have gone and you will go'>, <tf.Tensor: shape=(), dtype=string, numpy=b'we will go'>]

But I want to have an object like this
<tf.Tensor: shape=(3,), dtype=string, numpy=array([b'I have gone and you will go', b'we will go'], dtype=object)>

Which is a tensor of list of strings (Opposite to the previous one). Is there any way to this, assuming that I have access to all these tensors, i.e. some_function(x) for x in something separately.

In order to stack list of tf tensors you could use the tf function stack hence the name
for ur case,

tf.stack([some_function(x) for x in something],axis=0)

or you could stack them in numpy and then convert the array to a tensor, to do so using numpy np.stack inputting a list and axis


Which Amazon EC2 instance for Deep Learning tasks? I have discovered that Amazon has a dedicated Deep Learning AMI with TensorFlow, Keras etc. preinstalled (not to mention other prebuilt custom AMIs). I tried out this with a typical job on several GPU-based instances to see the performances. There are five such in the Ireland region (maybe in other regions exist even more, I don't know, this variance is a bit confusing):

g2.2xlarge
g2.8xlarge
p2.xlarge
p2.8xlarge
p2.16xlarge

My first question is, what is the difference between the two groups (g-something and p-something)? Both group mentions "GPUs along with CPU", but no further clue for Deep Learning usability.

My second problem is that I have been running my job on g2.2 and g2.8 as well, and while the task processing toke quite long time to run, the workload of the GPUs was relatively low (20-40%). Why didn't the framework increase the workload if there is spare processor capacity? Is is necessary/possible to paramter/set anything to optimize the work?

I think the differences and use cases are well pointed here. As far the workload, there are features which help you optimise it. According to the official documentation, you can try:

For persistency,

  sudo nvidia-smi -pm 1

Disabling the autoboost feature

  sudo nvidia-smi --auto-boost-default=0

Set all GPU clock speeds to their maximum frequency.

  sudo nvidia-smi -ac 2505,875


Does CNN take care of zoom in images? Suppose a convolution neural network is trained on small images of an object, say flower, as in following 3 training images:

Will this CNN correctly classify if the same object is present in zoomed form in a test image? As in following example:

What if the situation is reverse, i.e. training on large sized objects and testing with small sized object?

What is the best way to take care of different size of object that may be present for correct classification of images?

(The images are from: https://www.shutterstock.com/video/search/flowers )

I think more information on the project can help. For instance in your example, you just need to detect the flower frame in the whole image and crop it. Then you simply resize your images to be the same size before feeding your NN. In this case, detecting the target frame and scaling it is a part of pre-processing and the learning phase stays the same.

Another approach is to learn this scaling. It means that the learning algorithm needs scale-invariant features but NNs are supposed to actually create features in their hidden layers through learning. So if you have sufficiently big dataset in which all variants are presented enough, your NN will learn it (most probably with poor results). The process is intuitive ML-wise:

NN extract features from input images i.e. when the label of same object with different sizes are the same, it captures those features which discriminate these objects in a scale-invariant manner.
This was simplified! The trick is that CNN can learn resizing using a technique called Pooling. But you are the one who plays with parameters to make your NN working!

Probably the best is the situation in which your dataset is not that huge and rich or you do not want to play with many parameters or you want to increase the quality of results. Then focus on finding scale-invariant features. Either you extract them from your images and feed it to the NN, or you design NN in a way that it finds them itself. A more intuitive way is to train different models on different scales and ensemble them.

Hope it helps :) Good Luck!

PS: SIFT is patented. Be careful not to use it for commercial purposes.


TypeError: object of type 'Bidirectional' has no len() I'm running a bidirectional LSTM. But this error is appearing:

TypeError: object of type 'Bidirectional' has no len()

What's wrong in this code? Please help.

 import pandas as pd
 import numpy as np
 from tqdm import tqdm, trange
 import unicodedata

 from keras.preprocessing.sequence import pad_sequences
 from keras.utils import to_categorical
 from keras.models import Model
 from tensorflow.keras.layers import Input
 from tensorflow.keras.layers import LSTM, Embedding, Dense, TimeDistributed, Dropout, Bidirectional

 # Defining Constants

 # Maximum length of text sentences
 MAXLEN = 180
 # Number of LSTM units
 LSTM_N = 150
 # batch size
 BS=48

 input = Input(shape=(MAXLEN,))

 model = Embedding(input_dim=n_words, output_dim=MAXLEN, input_length=MAXLEN)(input)

 model = Dropout(0.2)(model)

 model = Bidirectional(LSTM(units=LSTM_N, return_sequences=True, recurrent_dropout=0.1))(model)

 out = TimeDistributed(Dense(n_tags, activation="softmax"))(model)  # softmax output layer

 model = Model(input, out)

 model.compile(optimizer="adam", loss="categorical_crossentropy", metrics=["accuracy"])
 history = model.fit(X, np.array(y), batch_size=BS, epochs=2, validation_split=0.05, verbose=1)

That code instantiates model over and over. model should be instantiated once and all other layers are added to that instance. Something like this:

from tensorflow.keras.models import Sequential

model = Sequential()
model.add(Embedding(input_dim=n_words, output_dim=MAXLEN, input_length=MAXLEN))
model.add(Dropout(0.2))
model.add(Bidirectional(LSTM(units=LSTM_N, return_sequences=True, recurrent_dropout=0.1)))
model.add(TimeDistributed(Dense(n_tags, activation="softmax")))


Neural Network for predict a set in a certain time stamps I have a market transactions dataset including time stamps and goods as follow.

John always buy milk and bread in Super Market. Besides that, he also buys some goods like the following:

On Monday, John bought milk, bread {beer, chocolate}.
On Tuesday, John bought milk, bread {potato}.
On Wednesday, John bought milk, bread {chocolate, avocado, peanuts}.

Can we answer the question: "What will he buy on Thursdays?".

For example: He will buy {beer, avocado} besides milk and bread on Thursdays.

Which one should I use? Supervised or unsupervised.

Which model can I use to predict a set of goods in this case?

Since you have a transaction data set on your hands, we are talking clear-cut supervised learning here. And while you could certainly try to use neural networks, I think you should start simpler and switch to more complex and costly algorithms once you see the simple ones failing (this is generally a good idea).

Now form what you describe, you have some sort of time-series data, with a categorical quantity that you want to predict, for which you will want some sort of modified ARMA model. You could set up a logistic regression for each good you are interested in (i.e. beer, chocolate, potato, etc.) that will predict whether or not he will buy the product. The variables you feed into the model are a set of binary observations describing if he bought the product a day ago, a week ago, four weeks ago, etc.. Play around with those lags. The nice thing about logistic regression here is that you'll see right away what is the period of the purchases. If you see a large coefficient on the one-week-ago-variable, you know that he eats avocados weekly and so forth. Note that if something is purchased weekly, it's also purchased every four weeks, so you will have some correlation issues.

If you don't get decent predictions, you could try different models, e.g. trees or ensemble methods such as random forests or boosted trees. If those don't work either, you probably have a time series without clear periodicity. In that case you could try setting up your variables differently, e.g. using the number of days since he bought the product.

One last word of advice: You might want to check correlation between purchases. Maybe he buys avocados, tomato, and onion together to make delicious guacamole. If he then has some onions leftover, he might be inclined to buy more avocado and so on.


Dendrogram: ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all() I am trying to plot a Dendrogram to cluster data but this error is stopping me. My datea is here.
I first chose columns to work with:
    df_euro = pd.read_csv('https://assets.datacamp.com/production/repositories/655/datasets/2a1f3ab7bcc76eef1b8e1eb29afbd54c4ebf86f2/eurovision-2016.csv')
   samples = df_euro.iloc[:, 2:7].values[:42]
   country_names = df_euro.iloc[:, 1].values[:42]

        # Calculate the linkage: mergings
            mergings = linkage(samples , method = 'complete')
            # Plot the dendrogram
            dendrogram(
                mergings,
                labels = y,
                leaf_rotation = 90,
                leaf_font_size = 6
            )
            plt.show()

But I'm getting this error which I can't understand. I googled it and checked that both have the same shape (1066,5) and (1066,). There is no NA in both features and country_names.
I know the issue is with labels because it works well when I remove it but I couldn't find away to solve it. Any help will be really appreciated. Alternately, how can I add the labels to the Dendrogram?
The error I am getting:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-113-7fffdc847e5e> in <module>
      4 mergings = linkage(feat, method = 'complete')
      5 # Plot the dendrogram
----> 6 dendrogram(
      7     mergings,
      8     labels = target_col,

C:\ProgramData\Anaconda3\lib\site-packages\scipy\cluster\hierarchy.py in dendrogram(Z, p, truncate_mode, color_threshold, get_leaves, orientation, labels, count_sort, distance_sort, show_leaf_counts, no_plot, no_labels, leaf_font_size, leaf_rotation, leaf_label_func, show_contracted, link_color_func, ax, above_threshold_color)
   3275                          "'bottom', or 'right'")
   3276
-> 3277     if labels and Z.shape[0] + 1 != len(labels):
   3278         raise ValueError("Dimensions of Z and labels must be consistent.")
   3279

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\generic.py in __nonzero__(self)
   1476
   1477     def __nonzero__(self):
-> 1478         raise ValueError(
   1479             f"The truth value of a {type(self).__name__} is ambiguous. "
   1480             "Use a.empty, a.bool(), a.item(), a.any() or a.all()."

ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

Just in case someone else is searching for the same issue, by converting the labels to list, it will work.
samples= df_euro.iloc[:, 2:7].values[:42]
country_names= list(df_euro.iloc[:, 1].values[:42])

mergings = linkage(samples, method='single')

# Plot the dendrogram
fig, ax = plt.subplots(figsize=(15, 10))
fig  = dendrogram(mergings, labels=country_names)
plt.show()


Analysis of clustering results Suppose that i have multidimensional dataset and performed some partitioning clustering on it. Is there any way to find out what objects in a particular cluster have in common (except the fact that clustering algorithm decided to put them together)?

I've read many times that clustering is not a well-posed problem in general and that one should not overinterpret its results but still people are trying to cluster multidimensional data and make some practical sense of the results. I just can't find any good source on how the interpretation is done in practice.

Any tips and resources will be highly appreciated.

You can write code to analyze this. But this is very much data specific.

In some cases like k-mrans, you already get cluster averages. So you can easily compare the averages.

If you don't have major correlations in your data, then histograms can be insightful. Make stacked histograms, colored per cluster, to see if there is a different distribution in each attribute. Etc.
There are so many ways to do this, and they are not specific to clustering.


Drop NA Values with SAS I have an xlsx file that I wish to delete the NA values in SAS. I'm new to SAS, how do I do this?

For numeric data(observations) in order to drop you can use missing and delete:

data olddata;
set cleandata;
if missing(coalesce(of _numeric_)) then delete;run;

Note: untested code


How can I look up classes of ImageNet? After downloading the imagenet urls (link), I see that it is a single 1.1 GB text file which starts like this:

n00004475_6590    http://farm4.static.flickr.com/3175/2737866473_7958dc8760.jpg
n00004475_15899    http://farm4.static.flickr.com/3276/2875184020_9944005d0d.jpg
n00004475_32312    http://farm3.static.flickr.com/2531/4094333885_e8462a8338.jpg
n00004475_35466    http://farm4.static.flickr.com/3289/2809605169_8efe2b8f27.jpg
n00004475_39382    http://2.bp.blogspot.com/_SrRTF97Kbfo/SUqT9y-qTVI/AAAAAAAABmg/saRXhruwS6M/s400/bARADEI.jpg
n00004475_41022    http://fortunaweb.com.ar/wp-content/uploads/2009/10/Caroline-Atkinson-FMI.jpg
n00004475_42770    http://farm4.static.flickr.com/3488/4051378654_238ca94313.jpg
n00004475_54295    http://farm4.static.flickr.com/3368/3198142470_6eb0be5f32.jpg
n00005787_13    http://www.powercai.net/Photo/UploadPhotos/200503/20050307172201492.jpg
n00005787_32    http://www.web07.cn/uploads/Photo/c101122/12Z3Y54RZ-22027.jpg

I'm pretty sure n00004475_6590 is the class of the image. How can I look up what this means (in natural language)?

The first part before the underscore can be entered in this URL:

http://www.image-net.org/api/text/wordnet.synset.getwords?wnid=n00004475


Anomaly Detection In Univariate Time Series Data Using ARIMA In Python With Updating I have trained an ARIMA model on some 15 minute incremented time series data by using the statsmodels library. I would like to determine how anomalous the next 15 minute increment's data I observe is. I would then like to update the model with that data.

I can predict the range of future outcomes within an accepted error margin like so:

model = arima_model.ARIMA(data.COUNTS,(p,d,q)).fit()
print model.forecast(alpha=.001)[2]

That outputs:

  array([[ 16.13395152,  48.47024783]]))

So you could say any number outside that range is anomalous. However, if my next observation were, for example, 50 that would presumably be less anomalous than an observation of 51. Which way do you recommend I use to determine exactly how anomalous an observation is using an ARIMA model?

Also, how should I go about updating my model with the new observation?

Have you considered using a quantile forecast as your ARIMA?

Quantile forecast is when, instead of predicting the average, you predict quantiles. In your case, you could predict for a given day the probability that an observation is below the 0.5%-percentile and the 99.5%-percentile. That range would define the 1% of "abnormal" values for the next days.

Recapitulating, usually in regression you find the average value by minimizing $\min_\beta \sum_i||X_i\cdot\beta -y_i||^2$.

You could find the median value by minimizing $\min_\beta \sum_i|X_i\cdot\beta -y_i|$.

You can also find any given quantile $\tau$ by minimizing $\min_\beta \sum_{i|y_i\geq X_i\cdot\beta}\tau|X_i\cdot\beta-y_i| + \sum_{i|y_i< X_i\cdot\beta}(1-\tau)|X_i\cdot\beta-y_i|$.

Of course, these minimizations problems become more and more complicated. You can only estimate your $\beta$ for this last one using ADMM or gradient descent.

I don't think you'll find python tools for this. You'll find plenty of packages in R though. For instance, see this page:

http://robjhyndman.com/hyndsight/quantile-forecasts-in-r/

One last piece of advise: if you need this forecasting to interoperate with other python code, my recommendation is to do it via CSV files. Just have your python code do the pre-processing and generate a CSV file, then R doing its thing and generating another CSV file, then use this file for the post-processing in python, etc. Some people try to use things like rpy2, but I find it to be overengineering.


The Bias-Variance Trade-Off I'm reading "An Introduction to Statistical Learning: With Applications in R".
In the Paragraph 2.2.2 The Bias-Variance Trade-Off, the authors say:

I'm not able to understand why the bias tends to initially decrease faster than the variance increases. Can you help me ?
Both rigorous and intuitive explanations are greatly appreciated

One way to look at this is through the idea of under-/overfitting

First off, here is a sketch of the generally observed relationship between bias and variance, in the context of model size/comlpexity:

Say you have a model which is learning quite well, but your test accuracy seems to be pretty low: 80%. The model is essentially not doing a great job of mapping input features to outputs. We have a high bias. But for a wide variety of input (assuming a good test set), we consistently obtain this 20% error; we have a low variance. We are underfitting

Now we decide to use a bigger model (e.g. a deep neural network), which is able to capture more details of the feature space and so maps inputs to outputs more accurately. We now have an improved test accuracy: 95%. At the same time, we notice that several runs of the model produce different results; sometime we have 4% error, and sometimes 6%. We have introduced a higher amount of variance. We are perhaps somewhere around the optimum model complexity shown on the graph above.

You say ok... let's create a monolithic neural network. It totally nails training and ends with a perfect accuracy: 100%. However, the test accuracy now drops to 90%! So we have zero bias, but a large variance.
We are overfitting. The model is almost as good as a look-up table for training data, but doesn't generalise at all when it sees new samples. Intuitively, that 10% error corresponds to a difference in distribution between the training and test sets used $\rightarrow$ the model knows the training distribution in extreme detail, some of which do not apply to the test set (i.e. the reality).

In summary: The bias tends to decrease faster than the variance increases, because you can likely still make a more competitive model for your dataset; the model is underfitting. It is like the low-hanging fruit that you can easily get - so an incremental improvement on the red curve above gives a big decrease in bias (increase in performance). Obviously that pattern cannot go on indefinitely, with each increment in model complexity, you get a lower increase in performance; i.e. you have diminishing returns. Furthermore, as you begin to overfit, the model becomes less able to generalise and so exhibits larger errors on unseen data; variance is creeping in.

For some more intuition between bias/variance in machine learning, I'd recommend this talk by Andrew Ng. There is also a text summary of the talk, for a quicker overview.

For a brief but more mathematical explaination, head over to this post of Cross-Validated. The second answer there is very recent and is perhaps better than the (old) accepted answer.


Understanding SVM Kernels Following Andrew Ng's machine learning course, he explains SVM kernels by manually selecting 3 landmarks and defining 3 gaussian function based on them. Then he says that we are actually defining 3 new features which are $f_1$, $f_2$ and $f_3$.

$\hskip0.9in$

And by applying these 3 gaussian functions on every input data:
$$x=(x_1,x_2)\to \hat{x}=(f_1(x), f_2(x), f_3(x))$$

it seems that we are mapping our data from $\mathbb R^2$ space to a $\mathbb R^3$ space. Now our goal is to find a hyperplane in the 3 dimensional space, where our transformed data is linearly separable. Is my understanding correct? If not, how these 3 new features should be interpreted?

$\hskip1in$

In some blog posts, i have read that by using a gaussian kernel, we are mapping our data to an infinite dimensional space (where gaussian kernel computes the dot product of transformed input data) which contradicts with my above understandings.

Yep, this is the correct interpretation. The kernels make a difficult classification problem into a much simpler one by making the data linearly separable by transforming it into a higher dimension. I think this image does a good job of illustrating that.


How does generalised advantage estimation work? I've been trying to add GAE to my A2C implementation for a while now, but I' can't quite seem to grok how it works.

My understanding of it, is that it reduces the variance of the advantage estimation function by kind of 'averaging out' (or generalising) the advantages based off the values in the rollout.

I tried to run through the maths on my own, and in the end I just had one advantage for the whole rollout, is this right? Normally, we'd have one advantage for each timestep in the rollout.

Can anyone provide an explanation on the intuition of GAE?

I found very intuitive the explanation of the GAE in the Supplementary material of this paper: DeepMimic. You do not need to read the paper. Just go straight to the Supplementary material section on page 143:15. For the λ-return you can find lots of information in the Reinforcement Learning book of Sutton and Barto. Hope it helps!


Public cloud GPU support for TensorFlow I'm building convoluted neural networks with TensorFlow. I've got the latest Mac Pro, use all of my cores, and yet it still takes me several hours to train a single network. I need to do grid search on many different tuning parameters, so this is not ideal.

This is for a personal project, so I want to save money by using the cloud. I have used AWS extensively, and I like the idea of using their spot market, however I've heard that 1) Amazon's GPU instances aren't compatible out of the box with TensorFlow (AWS's GPUs have CUDA 3.0 and TensorFlow requires 3.5) and 2) Amazon's GPUs are a bit slow.

Does Microsoft's Azure or Google's Google Compute Engine have easy-to-use GPUs when using TensorFlow? How do their GPUs compare to those of Amazon in terms of the relevant neural-net-training tech specs, prices, and ease of use? Are there smaller cloud providers that you would recommend instead?

Also, roughly how much faster can I expect training to take on one of these virtual GPUs compared to a top of the line Mac Pro?

Microsoft announced a couple of weeks ago virtual machines in Azure with GPUs. They use K-80 NVIDIA cards. The biggest machine has 4 GPUs and 224 GB of ram.

Good to play with deep learning :-)


Deep learning model gives random results First I am new to machine learning if it is an obvious question, I am sorry.

dataset_coefficients = loadtxt(
'in.csv', delimiter=',')
dataset_answers = loadtxt(
    'out.csv')
X = dataset_coefficients[:, 0:4]
y = dataset_answers

model = Sequential()
model.add(Dense(32, input_dim=4, activation='relu'))
model.add(Dense(16, activation='sigmoid'))
model.add(Dense(8, activation='sigmoid'))
model.add(Dense(4, activation='sigmoid'))
model.add(Dense(2, activation='sigmoid'))
model.add(Dense(1, activation='softmax'))

model.compile(loss="mean_squared_error",
              optimizer='adam', metrics=['accuracy'])

model.fit(X, y, epochs=250, batch_size=100, verbose=0)
_, accuracy = model.evaluate(X, y, verbose=0)
print('Accuracy: {}%'.format(accuracy * 100))

in.csv looks like this

     1,       -21,       147,      -343
     1,        19,       115,       225
     1,         1,       -64,       -64
     1,        30,       300,      1000
     1,        16,        64,         0
     1,         3,       -81,      -243
     1,         3,         3,         1
     1,       -13,        16,       192

out.csv

     1
     2
     3
     1
     2
     3
     1
     2

When I test the accuracy with the data which was previously used for training

Output is

Accuracy: 33.33333432674408%

No matter how big my dataset is, the result has the exact same accuracy rate of 33% which seems like the prediction is literally random. What am I doing wrong? Thanks

I identified several problems when looking through your codes :

The first column for in.csv is a dummy variable for bias term (all 1 for the whole column). For NN layers in Keras unless you put include_bias=False, weights for bias will be included automatically hence this column would not be necessary.
Your variable is not scaled/standardized, although sometimes it will work without normalization but it is always recommended to do this for NN since input like that may lead to unstable training. Consider using StandardScaler or MinMaxScaler for starter.
You mentioned on the tags that you are working on classification so this is what you should do. First, identify the number of unique classes, this should be the number of hidden unit on the last layer. Next, you have two choices, if you want to use OneHotEncoder then when compiling your model your loss should be 'categorical_crossentropy'. The other choice, you first make sure that your label index starts from 0 and then use 'sparse_categorical_crossentropy' as your loss.
Softmax activation function applies the function AFTER the weights and biases operation. So the output of your model will first be a set of probability of the sample being some class and to get one output simply call .argmax(axis=1) which will return the index with the most probability on each row. You also need to do + 1 since your class labels starts from 1 instead of 0.


Model parameters & Hyper parameters of Neural Network & their tuning in training & validation stage I am new to Machine learning in that Artificial Neural Network. I am using nnet package in R for training of Neural Network. What is the difference between model parameters and hyperparameters? I have heard that the hyper parameters are set before we set the model parameters. What are the hyper parameters and model parameters of a Artificial Neural Network and when they are actually tuned? i.e. Are hyper parameters tuned in Training stage of Neural Network and model parameters in validation stage?

The parameters of a neural network are typically the weights of the connections. In this case, these parameters are learned during the training stage. So, the algorithm itself (and the input data) tunes these parameters.

The hyper parameters are typically the learning rate, the batch size or the number of epochs. The are so called "hyper" because they influence how your parameters will be learned. You optimize these hyper parameters as you want (depends on your possibilities): grid search, random search, by hand, using visualisations... The validation stage help you to both know if your parameters have been learned enough and know if your hyper parameters are good.

If you want to know more about hyper parameters and parameters in general in machine learning, look for "deep learning versus shallow learning".


.h5 file size is same before and after training? learner = ConvLearner.pretrained(arch, md, ps=0.5) #dropout 50%
learner.load('ResNet34_256_1-2')
learner.fit(lr,1)
learner.save('ResNet34_256_1')

h5 file in load and save is having same size. Should it increase after training?
How do I know that saved model is better than loaded on?

This is not surpising.
h5 is the save file of the model's weights. The number of weights does not change before and after training (they are modified, though), therefore, your file should have the same size.

To know if your model after training is better, you have to measure it : you have to make predictions on a test set, and measure it, with usual metrics like accuracy or F score and so on. look at this to see usual metrics used in machine learning.


Document matching with more priority to certain features than others I am working on recommendation systems wherein I need to match the similarity of 2 users. Now, I know that I can use Tfidf vectorizer to calculate the the cosine similarity score between them. But, now suppose I have some features where I have different priorities for those features. So, for each feature there will be a different priority and the one with with higher priority will be checked first. So, when I get cosine similarity based on that feature, I will move on to the next feature and so on. How can I achieve this?

If you don't want all features to have the same weight in the cosine similarity you could just use weighted cosine similarity.

Cosine similarity is calculated as

$similarity = \frac{\vec A\cdot \vec B}{\Vert A\Vert\cdot\Vert B\Vert}$

Now, if you multiply each component $a_i$ of $\vec A$ and $b_i$ of $\vec B$ with a weight $w_i$ you will get a weighted cosine similarity.

Which weights you should use depend on the application. If you have a ranking of your features $\vec R$ you could give $\frac{1}{r_i}$ or $\frac{1}{r^2_i}$ a try. The feature with rank 1 will have a weight of 1 in both cases. In principle any measure of feature importance will do as weights.


Details on soft decision trees In Distilling a Neural Network Into a Soft Decision Tree the authors mention

  exponentially decaying running average of the actual probabilities with a time window

but provide no formula for it.

What does it mean speaking formal?

This is likely the Exponential Moving Average function.
In simplest form it is
ma_new = alpha * new_sample + (1-alpha) * ma_old

Where alpha is the parameter that controls rate of decay, between 0.0-1.0.


What is the best approach for detecting the defect? Assuming that we have 10 same objects, they are lined up and equidistant. If any of them is rotated a very small angle(5 - 10deg), what is best method to detect them? I am using a camera to capture the image. I want to divide the image into 10 parts and then feed each part into CNN model to classify the defect. Is that a possible way? What is the recommendation for this problem?

It is a possible way, but this class of problems falls under outlier detection in machine learning.
The defining feature is that one can only train for usual events not rare events, there are few or even no data at all, so a different approach needs to be taken.
Since usual neural network training requires adequate samples from each class, a workaround is needed. For example autoencoder networks are an option for outlier detection


Using an unsupervised Isolation Forest, how does one identify the optimal number of outliers from the anomaly scores? I am using an unsupervised isolation forest algorithm and computing anomaly scores to detect outliers from a 2 dimensional toy dataset. From a scatter plot, I am able to detect/visualize the data points with the highest anomaly scores (example: top 10 or top 15 outliers from the data)as my outliers. Is the number of outliers subjective to user decision (for example:Anomaly score of 0.5>S>1 is an outlier and everything less than 0.5 is not) or is there a way to detect the optimal number of outliers based on the anomaly scores?

The way Isolation Forests seem to be used in most of the cases involve having some kind of prior "guess" about what proportion of outliers you expect (if you want to be on the safe side, you might be encline to increase it for instance). Based on the proportion you set (it is the contamination parameter in scikit-learn), the observations are labeled according to their anomaly score.

That being said, nothing prevents you from coming up with your own rules after you ran the Isolation Forest. You can plot the anomaly scores and then see for yourself if another threshold could make sense based on the distribution of the scores. This would be a more empirical rule.