Identify significant features in clustering results I'm a student in Data Analysis, working on a data clustering exercise.
Two clusters have been identified based on a dataset with 40 features. To interpret and label these clusters, I'm wondering if there is a way to determine which features are the most determinant in the clustering output. For instance, I would imagine that I could take out one feature from the clustering and see how much it affects the output. However there are probably smarter ways.
I would greatly appreciate if someone could point me in the right direction.
Thanks!

A similar post appears on Cross-Validated, "Estimating the most important features in a k-means cluster partition".
Quoting from that post:

One way to quantify the usefulness of each feature (= variable =
dimension), from the book Burns, Robert P., and Richard Burns.
Business research methods and statistics using SPSS. Sage, 2008.
(mirror), usefulness being defined by the features'
discriminative power to tell clusters apart.

We usually examine the means for each cluster on each dimension using
ANOVA to assess how distinct our clusters are. Ideally, we would
obtain significantly different means for most, if not all dimensions,
used in the analysis. The magnitude of the F values performed on each
dimension is an indication of how well the respective dimension
discriminates between clusters.

Another way would be to remove a specific feature and see how this
impact internal quality indices. Unlike the first solution, you
would have to redo the clustering for each feature (or set of
features) you want to analyze.
FYI:

Can a useless feature negatively impact the clustering?
Can the choice of the measurement units of the features impact the clustering?
Why vector normalization can improve the accuracy of clustering and classification?
What are the most commonly used ways to perform feature selection for k-means clustering?

Furthermore, there is a paper on Feature Selection in Clustering Problems

A novel approach to combining clustering and feature selection is
presented.   It  implements  a wrapper strategy  for  feature
selection,  in  the sense that the features are directly selected by
optimizing the discriminative power of the used partitioning
algorithm.  On the technical side, we present  an  efficient
optimization  algorithm  with  guaranteed  local  convergence
property.  The only free parameter of this method is selected by a
resampling-based stability analysis.  Experiments with real-world
datasets demonstrate that our method is able to infer both meaningful
partitions and meaningful subsets of features.


Softmax cost function In this post, https://cs231n.github.io/linear-classify/#webdemo it mentions "Softmax classifier interprets the scores inside the output vector f as the unnormalized log probabilities." Aren't we taking the log afterward and f is equal to just Wx+b?

No, that is not what is happening, the definition of the Softmax classifier is: $$z \in \mathbb{R}^K; \sigma(z)_j = \frac{e^{z_j}}{\sum_{k = 1}^{K} e^{z_k}} $$ Let's talk first about what the softmax function aims to do, but before we can understand that, we need to understand the aim of classification in general. Classifiers are functions which take a vector in $\mathbb{R^n}$ and output one of $k$ classes. The softmax function takes in a vector $z$ and outputs a value in $\mathbb{R^+}$ whose values sum to one.

Now let's get to your question, if I were to just take $\log(Wx + b)$. It is actually almost exactly that, we just take the exponent so that our values range from $0$ to $1$. However, these values are not normalized and in order to normalize it we need to divide by the sum and we take the exponents to maintain a positive value.


How to summarize(add) a column according to same year and plot in R? I have following type of data. But I would like to have graph with X-axis as Year and add all the sales added up in that year on Y- axis.

The Data frame name is salesdata. I have nearly 8000 rows of data. and the column headers are in the picture below.

Thanks!

Step 1: Changing dates to years

First you can change your column of dates (Order_Date) to simply years by using this formula:

> salesdata$Order_Date = format(salesdata$Order_Date, "%Y")

This will give your Order Dates just the year the sales was done in.

Step 2: Summarizing your sales data

After this you can create a vector with the summarizations of your sales data per year. This can be done using tapply:

> tapply(salesdata$Sales, salesdata$Order_Date, FUN=sum)

This will give you an overview of the total sales per year.

Step 3: Plotting your graph

You can use barplot() to plot your bar chart using the values you got in your tapply function.

One-liner

The function as a oneliner:

> barplot(tapply(salesdata$Sales, format(salesdata$Order_Date, "%Y"), FUN=sum))

Of course you can edit your graph to your liking.


One hot encoding with too many features (~ 10,000) I am building a model to predict time off and sick leave for a specific employee.

Each of the employees has one row per day from 01/01/2013 to 31/12/2018 in the dataset flagged with 0 or 1 (if that day was taken as a time off or sick day). I am using information like location, age, job position, etc.

I was thinking to use employee id as a feature to train the model. Some users have obvious patterns, e.g, taking Monday off for some consecutive weeks. On the other hand, I feel like I do not have enough information as to have one model per employee, specially if they have being hired recently.

After researching a bit, seems that the only encoding that would make sense for employee id would be 'One Hot Encoding'. However, this would generate up to 10,000 features which doesn't seem very optimal.

Do you have any ideas that can help me build this model in a better way?

Thanks in advance

I think you are on the right track. What you are looking for is what is known as fixed effects in econometrics/statistics. Your have a model in which features X determine outcome y (so y = bX + u). On top of that you have unobserved heterogeneity (each individual employee) in your model. For each employee, the relation y <-> X may be a little different (fixed effect per employee), so that your model becomes y = b1X + b2Z + u, where Z denotes employees.

See here for an overview and here for an R implementation.

You can solve this by adding one dummy/indicator/one-hot per employee (known as least squares dummy variable fixed effects). So technically speaking, each employee gets his/her own intercept in the model. Thus the Z above is a massive matrix of one-hot encoded employee labels.

There are solutions in Stata (areg) and R (felm) which "absorb" the massive dummy variable set, so that it does not bother you when looking at the regression coefficients. If you are not interested in statistical inference (but only in making predictions), you will not look at the coefficients anyway.

I don't know the datails of your data (i.e. observations per employee), but from what you say, adding one dummy (aka one-hot feature) per employee would be the right way to go to control for unobserved heterogeneity of the employees.


Questions about CNN: weights and biases I have a question regarding CNN.
I do understand how they work on the surface. Very simply put, they are D-NN for images.
I'll use this example as a reference for this question. In the example, they are not initializing weights and biases anywhere. They are using tf.layers.conv2d() function in the example.
Let's focus only on 2 of the function's arguments:

filters: specify the number of filters to apply
kernel_size: specifies the size of each filter

Questions:

We are not defining these filters. Does tensorflow define them for us or in general if I were to use this for a different set of images how does this work?

Does this filter include both weights and biases? If not, then what exactly does the GradientDescentOptimizer() defined in the above example update after each training step?

I understood the code and understand how the entire process works. I also understand convolution and how it works. But, I'm trying to apply these concepts and implement my own code using tensorflow and CNN and I'm kind of stuck here.

If you do not specify them, as it is clear in the signature of the functions you are referring to, the function will use the default value for them. For instance, you can see that the default value for stride is (2, 2) which means if you don't define the value of stride the method will use the mentioned value as the default value for the stride. Consequently, if you don't specify it, it does not mean there isn't such thing. In programming this approach helps programmers not to define many different overloads of a typical function.

As the response for the second question, again, if you don't specify the initial weights, TensorFlow itself will use Glorot method for initialization. So, definitely the filters will have values called weights in order to operations like convolution, actually cross correlation, be applicable.

As a recommendation, I highly suggest you taking a look at here and here.


Cannot get the prediction right using Stochastic Gradient Descent: Always predicts 1 I have a CSV file with 20 columns and 785 rows. The 785th row for each column is a label describing the encoded image. The encoded image is either 3 or 5. So 1-784 row is the encoded image and 785th row is the label that names the image.

I loaded the CSV file 3_5_small.csv and segregated the labels from the encoded data.

which as you see is the image of number 3.

Now, I decided to use logistic regression to predict the images from the encoded data. I used Stochastic Gradient Descent as explained in the machine learning course by Andrew NG. But I do not think, I got it right. Before following the code, here are the steps I did:

Transformed train_labels_3_5 which contained only 3 and 5 to 1 and 0 respectively. So I want to predict the image of 3. If the output probability is < 0.5, it will be 5 and > 0.5 will be 3.
Randomly shuffled the train_data_3_5 and train_labels_3_5 to the same degree.
Randomly generated the theta vector
Passed the theta vector and X vector into the hypothesis function
Updated the theta vector.

This is all I did. Here is the code to what I have done.

train <- function(data, labels, alpha = 0.001) {
  #browser()

  #Initialize the theta vector
  theta <- seq(from = 0, to = 1, length.out = nrow(data))

  number_of_iterations = 10
  for(noi in 1:number_of_iterations) {
    for(i in seq(1:ncol(data))) {
      x = as.vector(data[,i]) #Create a x vector
      h = hypothesis(x, theta) #Call the hypothesis function to get the probability
      y = labels[1,i]
      theta <- theta - (alpha * ((h - y) * x))
    }
  }
  return(theta)
}

But on the test data and even on the training data, this does not predict correct at all. I do not know where have I gone wrong. I have revisited the algorithm, the lecture but cannot figure out, what am I doing incorrectly. It always predicts 1 no matter I pass the vector for 3 or 5!

My network does always predict the same class. What is the problem?

I had this a couple of times. Although I'm currently too lazy to go through your code, I think I can give some general hints which might also help others who have the same symptom but probably different underlying problems.

Debugging Neural Networks

Fitting one item datasets

For every class i the network should be able to predict, try the following:

Create a dataset of only one data point of class i.
Fit the network to this dataset.
Does the network learn to predict "class i"?

If this doesn't work, there are four possible error sources:

Buggy training algorithm: Try a smaller model, print a lot of values which are calculated in between and see if those match your expectation.

Dividing by 0: Add a small number to the denominator
Logarithm of 0 / negativ number: Like dividing by 0

Data: It is possible that your data has the wrong type. For example, it might be necessary that your data is of type float32 but actually is an integer.
Model: It is also possible that you just created a model which cannot possibly predict what you want. This should be revealed when you try simpler models.
Initialization / Optimization: Depending on the model, your initialization and your optimization algorithm might play a crucial role. For beginners who use standard stochastic gradient descent, I would say it is mainly important to initialize the weights randomly (each weight a different value). - see also: this question / answer

Learning Curve

See sklearn for details.

The idea is to start with a tiny training dataset (probably only one item). Then the model should be able to fit the data perfectly. If this works, you make a slightly larger dataset. Your training error should slightly go up at some point. This reveals your models capacity to model the data.

Data analysis

Check how often the other class(es) appear. If one class dominates the others (e.g. one class is 99.9% of the data), this is a problem. Look for "outlier detection" techniques.

More

Learning rate: If your network doesn't improve and get only slightly better than random chance, try reducing the learning rate. For computer vision, a learning rate of 0.001 is often used / working. This is also relevant if you use Adam as an optimizer.
Preprocessing: Make sure you use the same preprocessing for training and testing. You might see differences in the confusion matrix (see this question)

Common Mistakes

This is inspired by reddit:

You forgot to apply preprocessing
Dying ReLU
To small / to big learning rate
Wrong activation function in final layer:

Your targets are not in sum one? -> Don't use softmax
Single elements of your targets are negative -> Don't use Softmax, ReLU, Sigmoid. tanh might be an option

Too deep network: You fail to train. Try a simpler neural network first.


Euclidean vs. cosine similarity I have a text dataset which I vectorize using a tfidf technique and now in order to make a cluster analysis I am measuring distances between these vector representations. I have found that a common technique is to measure distance using cosine similarity, and when I ask why euclidean distance is not used, the common answer is that cosine similarity works better when vectors have different magnitude.

Since my text vectorized representation is normalized I wonder which is the advantage of using cosine similarity over euclidean distance in order to cluster my data?

On L2 normalized data it is an easy and good exercise to prove that they are equivalent.

So you should try to solve the math yourself.

Hint: use squared Euclidean.

Note that it is common with tfidf to not have normalized data because of various technical reasons, e.g., when using inverted indexes in text search. Furthermore, cosine is faster on very sparse data.


Is a neural network suitable for this application Apologies if this question is not a suitable format. I am a novice in data science.

I have a database of species observation data consisting of ~16 million records.
Each record consists of:

latitude
longitude
date
time
species observed (that's species singular, not plural)

This data has been manually vetted by experts, so there is an additional field for each species in a record that classifies the observation as either valid or invalid (or more accurately speaking, likely correct/likely incorrect)

I am exploring the idea of training a neural network on this data to automatically classify new records as being valid or invalid ("invalid" data will be flagged for manual expert review.)

The vast majority of records are classified as 'valid', so my worry is that there isn't much information to train the model on what constitutes 'invalid'.

However, a good predictor of whether a record is valid is, informally speaking, "are there other records of this species close by (spatially and/or temporally)"

I'm not sure where to start with formulating a neural network for this problem. E.g.

Inputs: latitude, longitude, date, time, species

Output: validity

OR

Inputs: latitude, longitude, date, time

Outputs: one output for each known species indicating validity

I like the idea of this second model as I can input a time and location and get out a list of likely species.

So my concrete questions are:

Does this sounds like an application suitable for a neural network?
If so, where might I start with formulating a model for my problem? Or can someone point me in a good direction to learn more about this topic.

You may or may not apply neural network to your problem. Neural network will give predictions for sure but it can't be said how much efficient will it be for this problem. You have to code and check it for yourself. Also, if you have all the above data in tabular form and you know the labels(which I guess is valid/inavlid), I will say try it with xgboost. Neural networks are exceptional for unsupervised learning but in the case of supervised learning, there may be a model which may outperform a neural network.


LabelEncoding selected columns in a Dataframe using for loop I have certain columns in my dataset that are "object" type, so first I found them and now I want to transform from categorical to numerical data.
How can I do it in multiple columns using a for loop? I've been struggling with it. I don't want to repeat code.

I've been trying this, but it doesn't work

for column in object_columns:
    df.loc[:,column] = labelencoder.fit_transform(df.loc[:,column])

LabelEncoding your features is a bad practice

You should avoid using LabelEncoder to encode your input features! Don't believe me? Here's what scikit-learn's official documentation for LabelEncoder says:

  This transformer should be used to encode target values, i.e. y, and not the input X.

That's why it's called LabelEncoding.

Why you shouldn't use LabelEncoder to encode features.

This encoder simply makes a mapping of a feature's unique values to integers. For example, let's say we want to encode a feature called shirt color, which represents the color of the shirt someone's wearing. This feature has values ['red', 'green', 'blue', ...]. If you encode these into integers, i.e. [1, 2, 3, ...], you might confuse your model by because you have now given relationships to these values that don't exist in the real world, e.g. red < greed < blue or red + green = blue. This type of feature is called nominal and preferably should be one-hot encoded.

There are features however, where you might want to map their values to integers. These are called ordinal. For example, the feature rating, which has values ['bad', 'good', 'excellent', ...]. By mapping these to integers you actually preserve the relationsips these values hold in the real world, e.g. bad < good < excellent. There is a catch to this however, in order to do the above, you need to map each value with a specific integer (e.g. we can't map 'good' -> 1, 'bad' -> 2, 'excellent' -> 3, because that doesn't preserve the real-world relationship of these values). The computer doesn't know which number to map to each value, though, so if you use LabelEncoder even on ordinal variables, it most likely won't generate the correct encoding.

How to properly encode ordinal features

A more proper way of encoding ordinal variables is manually choosing the mapping. This requires more work and isn't as elegant as a one-liner that encodes all values, but is the only correct way. Let's see how we can do this in pandas.

custom_mapping = {'bad': 1, 'good': 2, 'excellent': 3}

df['rating'] = df['rating'].map(custom_mapping)

Obviously this needs to be done for each ordinal feature.

At this point I think it's clear that I strongly recommend against using LabelEncoder, but if you still want to do it at least do it correctly.

If you still want to use LabelEncoding

While both answers by @ggordon and @Anan Srivastava will do what you want, they don't have much value in practice. The problem isthat by not bounding the fitted LabelEncoder to a variable, you are loosing the mapping from categories to numbers. If you want to predict on future data, you won't know which number to encode each category with.

Expanding upon @ggordon's answer

columns_to_be_encoded = [...]  # list of column names you want encoded

# Instantiate the encoders
encoders = {column: LabelEncoder() for column in columns_to_be_encoded}

for column in columns_to_be_encoded:
    df[column] = encoders[column].fit_transform(df[column])

This way you have a dictionary of fitted encoders so that you can reuse the same encoding if you wish.


Reducing the effect of down voters with rating system I  have a site in which users rate things in a 1-5 star system. Once an item reaches the top of the charts, some users tend to start rating it 1 star even though it got a majority of 4-5 stars to get where it's at. It's not rampant, I would say 10-20% of the new votes are 1's. Clearly they are trying to manipulate the rating system, and I want to prevent that.

The current way I am doing that is by having a "reasonable window" of what I consider to be a legitimate vote.

For items with less than 10 votes;
I currently do nothing and take the mean as it's rating.

Once an item starts getting more than 10 votes I tie them to a window of their mean. This window is defined as

Window = 4.5 - Log(TotalVotes, 10);

So a reasonable vote range is then (Mean - Window) thru (Mean + Window)

Once the reasonable vote range is found, the "Rating" is just the mean of all the reasonable votes (those who fall in the reasonable range).

This means an item with a real mean of 4.2 with 100 votes would have a window of 4.5-Log(100,10) = 2.5, so if that item gets a 1 star vote, it will be ignored in the rating. However, the 1 star will still effect the underlying mean.

This has worked good in general but the issue is when an item's Mean - Window is just at the brink of 1.0, as soon as it dips below 1.0 every 1 star vote now is included in the rating and the rating drops significantly even the difference before and after may have just been one more 1 star rating.

I need a better system/way of accomplishing to filter out these 1 star ratings, and not just them but handle the situation in where someone may get their friends to upvote an item 10 votes and all 5 stars, where its true rating may be more 3 star.

Looking for any recommendations of how to handle user-driven rating systems and normalizing outlier votes.

You should look into other estimators of location.

What you want is a robust estimator, with a high break-down point.

The extreme approach would be the median.

But you may get more numerically interesting results with a trimmed mean.

You define a threshold, say 2%. Then you remove the top 2% of votes, and the bottom 2% of votes, and take the mean only of the remaining entries. An app with 98% 5 stars will still get a 5.0

But to prevent manipulation, I would look into other signals. Such as clustered votes from a single region, for example.


How to bin Colour values (categorical- high cardinality) to primary colours? I have a colour variable with values for different e-commerce products like - olive, teal, blue, green etc.

I want to bin these colours to the primary colour values.

eg -

olive ---> green
teal ----> blue
blue and green will remain the same.

I also have images available of these products, therefore, I am wondering if there is any solution using some clustering technique or using a pre-trained model for classification in some way.

Since there are small, finite number of color options, it would be straightforward to manually create a dictionary where the key is the primary color and values are a set of other color values.
In Python, it would be something like:
colors = {"blue": {"blue", "teal"},
          "green": {"green", "olive"},
         }


Multiclass ROC Curve using DecisionTreeClassifier I built a DecisionTreeClassifier with custom parameters to try to understand what happens modifying them and how the final model classifies the instances of the iris dataset. Now My task is to create a ROC curve taking by turn each classes as positive (this means I need to create 3 curves in my final graph). To do this I need to instantiate a OnevsRestClassifier and passing the previous classifier as parameter, so it automatically recognize the parameters I modified (such as the weights of the class). This is my current code:
from sklearn.datasets import load_iris
from sklearn.metrics import accuracy_score
from sklearn.metrics import f1_score
from sklearn.metrics import roc_curve, auc
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import label_binarize
from sklearn.model_selection import train_test_split
from sklearn import tree
from sklearn.multiclass import OneVsRestClassifier
from sklearn.metrics import roc_auc_score
from sklearn.metrics import confusion_matrix
import numpy as np
import graphviz

iris = load_iris()
X = iris.data
y = iris.target

# Binarize the output
y = label_binarize(y, classes=[0, 1, 2])
n_classes = y.shape[1]

# My DecisionTreeClassifier
clf = tree.DecisionTreeClassifier(criterion="entropy",random_state=300,min_samples_leaf=5,
                                  class_weight={0:1,1:10,2:10})

np.random.seed(0)

indices = np.random.permutation(len(iris.data))
indices_training=indices[:-10]
indices_test=indices[-10:]

iris_X_train = iris.data[indices_training]
iris_y_train = iris.target[indices_training]
iris_X_test  = iris.data[indices_test]
iris_y_test  = iris.target[indices_test]

# Training
clf = clf.fit(iris_X_train, iris_y_train)

# Test
predicted_y_test = clf.predict(iris_X_test)

print(confusion_matrix(iris_y_test, predicted_y_test))

print("Predictions:")
print(predicted_y_test)
print("True classes:")
print(iris_y_test)

# Learn to predict each class against the other
classifier = OneVsRestClassifier(clf)

# Train
classifier = classifier.fit(iris_X_train, iris_y_train)

# Test
y_score = classifier.predict_proba(iris_X_test)

# Compute ROC curve and ROC area for each class
fpr = dict()
tpr = dict()
roc_auc = dict()
for i in range(n_classes):
    fpr[i], tpr[i], _ = roc_curve(iris_y_test, y_score)
    roc_auc[i] = auc(fpr[i], tpr[i])

My problem is that I get the error:

Class label 2 not present.

At the line: classifier = classifier.fit(iris_X_train, iris_y_train)
This is when I train the new classifier and I do not understand why. I checked on the iris dataset and there are three classes, so the label 2 should correspond to virginica, right?

The combination of class_weight and indices_test only have 10 data points results in Class label 2 not present.
Since iris dataset is perfectly balanced there is no reason to specify class_weight. Additionally, scikit-learn has train_test_split which automatically makes a split that maintains equal proportions of every class in both training and testing.


How is the concept of data different for different disciplines? How is the concept of data different for different disciplines? Obviously, for physicists and sociologists, "data" is something different.

Data is, at it's most basic reduction, a raw element of something. Data is a raw "thing" that exists in any form from which we can analyze it and construct intelligence. When I was an Intelligence Analyst, we used to define data as "anything and everything that could be used to construct a hypothesis."

Thus, data for any discipline is interchangeable; as a sociologist, I have a vector of discrete variables indicating ethnicity, as an economist I have a vector with housing prices, and as an anthropologist I have a vector of tablet names used in some long-gone civilization.

Data is data.


Role of a known predictor in a disease prediction I am working on finding out whether the patient will develop the disease or not in a hospital.

Might be a basic info but I am just sharing it anyway.

Usually through historic data, I was able to see that patient who was in hospital or the icu for a longer time (more days) did develop the disease.

Similarly if he was in ventilation, it is also an indication of his health status

The distribution of hospital stay and ventilation hours between patients who developed disease and didn't develop disease are different and I verified them visually.

Now my question is

1) Do we need to include these as a predictor (input variable) in our model? I ask because am thinking by using these two variables, model might miss some other input variables which has an influence on outcome. For ex: I may not know whether his urea reading is an indication of whether he will develop disease or not.

Basically what I am trying to know is should we feed variables which we are confident that they will impact the outcome into the model?

Or is the model to help us know something which we aren't aware of .

Can you help me with this?

Basically what I am trying to know is should we feed variables which we are confident that they will impact the outcome into the model?

  Or is the model to help us know something which we aren't aware of .

It depends, there's no general rule. The goal is of course to predict the target variable as well as possible, i.e. to maximize the performance. So the question is to find the right balance between:

Giving only a few features which are known to help predict the target variables. In this case we make things as easy as possible for the learning algorithm, so that it can perform optimally with the features provided. Disadvantage: maybe there are other more subtle indications that would have helped the algorithm make better predictions.
Feeding the learning algorithm with a very high number of features. In this case we let the algorithm find the relevant indications itself, hoping that it will catch even subtle clues that a human expert would easily miss. Disadvantage: too many features are susceptible of causing two problems: overfitting, when the model relies on very specific patterns which happened by chance in the training set; and redundancy, which can also decrease the performance with some algorithms.

The data-driven answer is to try various combinations of features on the spectrum between these two options, and evaluate the corresponding models on a validation set (preferably with cross-validation in order minimize the effect of chance). This can be done manually or using feature selection/extraction methods.

[added for clarity:] when selecting the best option between different sets of feature (or different algorithms as well), one needs to use a validation set which is distinct from the test set (and also from the training set of course), otherwise there is again a risk of overestimating the performance.


Xgboost quantile regression via custom objective This is my first time posting, so please bare with me if I miss giving necessary info...

I'm new to GBM and xgboost, and I'm currently using xgboost_0.6-2 in R. The  modeling runs well with the standard objective function "objective" = "reg:linear" and after reading this NIH paper I wanted to run a quantile regression using a custom objective function, but it iterates exactly 11 times and the metric does not change.

I just simply switched out the 'pred' statement following the GitHub xgboost demo, but am afraid it is more complicated than that and I can't find any other examples on using the custom objective function. Do I need to take it a step further and take derivatives for the 'grad' and 'hess' part?

Or could it be a problem with xgboost (doubtful)?

qntregobj <- function(preds, dtrain) {
  qr_alpha = .5
  labels <- getinfo(dtrain, "label")
  preds <- ifelse( preds - labels >= 0
                 , (1-qr_alpha)*abs(preds - labels)
                 , qr_alpha*abs(preds - labels)
                 )
  grad <- preds - labels
  hess <- preds * (1 - preds)
  return(list(grad = grad, hess = hess))
}

step1.param <- list( "objective" = qntregobj
                   , "booster" = "gbtree"
                   , "eval.metric" = "rmse"
                   , 'nthread' = 16
                   )
set.seed(123)
step1.xgbTreeCV <- xgb.cv(param = step1.param
              , data = xgb.train
              , nrounds  = nrounds
              , nfold = 10
              , scale_pos_weight = 1

              , stratified = T
              , watchlist = watchlist

              , verbose = F
              , early_stopping_rounds = 10
              , maximize = FALSE

              ## set default parameters here - baseline
              , max_depth = 6
              , min_child_weight = 1
              , gamma = 0
              , subsample = 1
              , colsample_bytree = 1
              , lambda = 1
              , alpha = 0
              , eta = 0.3
  )
  print(Sys.time() - start.time)

  step1.dat <- step1.xgbTreeCV$evaluation_log
  step1.dat

Which produces:

iter train_rmse_mean train_rmse_std test_rmse_mean test_rmse_std nround
 1:    1        122.6362     0.04268346       122.6354     0.3849658      1
 2:    2        122.6362     0.04268346       122.6354     0.3849658      2
 3:    3        122.6362     0.04268346       122.6354     0.3849658      3
 4:    4        122.6362     0.04268346       122.6354     0.3849658      4
 5:    5        122.6362     0.04268346       122.6354     0.3849658      5
 6:    6        122.6362     0.04268346       122.6354     0.3849658      6
 7:    7        122.6362     0.04268346       122.6354     0.3849658      7
 8:    8        122.6362     0.04268346       122.6354     0.3849658      8
 9:    9        122.6362     0.04268346       122.6354     0.3849658      9
10:   10        122.6362     0.04268346       122.6354     0.3849658     10
11:   11        122.6362     0.04268346       122.6354     0.3849658     11

Perhaps the blog below provides an answer to your question.

https://www.bigdatarepublic.nl/regression-prediction-intervals-with-xgboost/

Without go through code in much detail, probably, your problem can be described as followed (from the blog):

  In the case that the quantile value q  is relatively far apart from the observed values within the partition, then because of the Gradient and Hessian both being constant for large difference x_i-q, the score stays zero and no split occurs.

Then the following solution is suggested:

  An interesting solution is to force a split by adding randomization to the Gradient. When the differences between the observations x_i and the old quantile estimates q within partition are large, this randomization will force a random split of this volume.


R random forest on Amazon ec2 Error: cannot allocate vector of size 5.4 Gb I am training random forest models in R using randomForest() with 1000 trees and data frames with about 20 predictors and 600K rows. On my laptop everything works fine, but when I move to amazon ec2, to run the same thing, I get the error:

Error: cannot allocate vector of size 5.4 Gb
Execution halted

I'm using the c3.4xlarge instance type so it's pretty beefy. Does anyone know a workaround for this to get it to run on this instance? I would love to know the memory nuances that causes this problem only on the ec2 instance and not on my laptop (OS X 10.9.5 Processor  2.7 GHz Intel Core i7; Memory  16 GB 1600 MHz DDR3)

Thanks.

Here's some advice (use at your own risk!):

make sure that your R environment on EC2 is identical to the one on your laptop
make sure you're using 64-bit images for your virtual machine instance
try to create/enable swap space: https://stackoverflow.com/a/22247782/2872891
see this discussion: https://stackoverflow.com/q/5171593/2872891
see this discussion: https://stackoverflow.com/q/12322959/2872891
see this discussion: https://stackoverflow.com/q/1358003/2872891

If the above-mentioned simpler measures don't help OR you want to achieve more scalability and/or performance, including an ability to parallelize the process on a single machine or across multiple machines, consider using bigrf R package: http://cran.r-project.org/web/packages/bigrf. Also see this discussion: https://stackoverflow.com/q/1358003/2872891.


Keras Applications - using images larger than the default size I would like to use eg Xception network with default input size 299x299, but my images are 450x600. Are there any other options besides cropping and subsampling ?

Have a look at where the reshaping happens. Just before that, you can insert a global average pooling layer in. This way you can handle any size.

However, I recommend cropping and scamming. Create multiple crops if necessary and average the results. That is likely still faster than using a bigger image.

How to use Xception

#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""See https://martin-thoma.com/image-classification/ for details."""
from __future__ import print_function

import numpy as np
import json
import os
import time

from keras import backend as K
from keras.preprocessing import image
from keras.applications.xception import Xception
from keras.utils.data_utils import get_file

CLASS_INDEX = None
CLASS_INDEX_PATH = ('https://s3.amazonaws.com/deep-learning-models/'
                    'image-models/imagenet_class_index.json')

def preprocess_input(x, dim_ordering='default'):
    """
    Standard preprocessing of image data.

    1. Make sure the order of the channels is correct (RGB, BGR, depending on
       the backend)
    2. Mean subtraction by channel.

    Parameters
    ----------
    x : numpy array
        The image
    dim_ordering : string, optional (default: 'default')
        Either 'th' for Theano or 'tf' for Tensorflow

    Returns
    -------
    numpy array
        The preprocessed image
    """
    if dim_ordering == 'default':
        dim_ordering = K.image_dim_ordering()
    assert dim_ordering in {'tf', 'th'}

    if dim_ordering == 'th':
        x[:, 0, :, :] -= 103.939
        x[:, 1, :, :] -= 116.779
        x[:, 2, :, :] -= 123.68
        # 'RGB'->'BGR'
        x = x[:, ::-1, :, :]
    else:
        x[:, :, :, 0] -= 103.939
        x[:, :, :, 1] -= 116.779
        x[:, :, :, 2] -= 123.68
        # 'RGB'->'BGR'
        x = x[:, :, :, ::-1]
    return x

def decode_predictions(preds, top=5):
    """
    Decode the predictionso of the ImageNet trained network.

    Parameters
    ----------
    preds : numpy array
    top : int
        How many predictions to return

    Returns
    -------
    list of tuples
        e.g. (u'n02206856', u'bee', 0.71072823) for the WordNet identifier,
        the class name and the probability.
    """
    global CLASS_INDEX
    if len(preds.shape) != 2 or preds.shape[1] != 1000:
        raise ValueError('`decode_predictions` expects '
                         'a batch of predictions '
                         '(i.e. a 2D array of shape (samples, 1000)). '
                         'Found array with shape: ' + str(preds.shape))
    if CLASS_INDEX is None:
        fpath = get_file('imagenet_class_index.json',
                         CLASS_INDEX_PATH,
                         cache_subdir='models')
        CLASS_INDEX = json.load(open(fpath))
    results = []
    for pred in preds:
        top_indices = pred.argsort()[-top:][::-1]
        result = [tuple(CLASS_INDEX[str(i)]) + (pred[i],) for i in top_indices]
        results.append(result)
    return results

def is_valid_file(parser, arg):
    """
    Check if arg is a valid file that already exists on the file system.

    Parameters
    ----------
    parser : argparse object
    arg : str

    Returns
    -------
    arg
    """
    arg = os.path.abspath(arg)
    if not os.path.exists(arg):
        parser.error("The file %s does not exist!" % arg)
    else:
        return arg

def get_parser():
    """Get parser object."""
    from argparse import ArgumentParser, ArgumentDefaultsHelpFormatter
    parser = ArgumentParser(description=__doc__,
                            formatter_class=ArgumentDefaultsHelpFormatter)
    parser.add_argument("-f", "--file",
                        dest="filename",
                        type=lambda x: is_valid_file(parser, x),
                        help="Classify image",
                        metavar="IMAGE",
                        required=True)
    return parser

if __name__ == "__main__":
    args = get_parser().parse_args()

    # Load model
    model = Xception(include_top=True, weights='imagenet')

    img_path = args.filename
    img = image.load_img(img_path, target_size=(224, 224))
    x = image.img_to_array(img)
    x = np.expand_dims(x, axis=0)
    x = preprocess_input(x)
    print('Input image shape:', x.shape)
    t0 = time.time()
    preds = model.predict(x)
    t1 = time.time()
    print("Prediction time: {:0.3f}s".format(t1 - t0))
    for wordnet_id, class_name, prob in decode_predictions(preds)[0]:
        print("{wid}\t{prob:>6}%\t{name}".format(wid=wordnet_id,
                                                 name=class_name,
                                                 prob="%0.2f" % (prob * 100)))

Why it works with any size

Look at the model.summary() of Xception, especially the output shape. Notice the global average pooling layer? Before that, the shape is determined by the input. Meaning until that point, it can be anything.

  Global pooling is another type of transition layer. It applies pooling over the complete feature map size to shrink the input to a constant 1 × 1 feature map and hence allows one network to have different input sizes.

Source: Me: Analysis and Optimization of Convolutional Neural Network Architectures


How to use the output of GridSearch? I'm currently working with Python and Scikit learn for classification purposes, and doing some reading around GridSearch I thought this was a great way for optimising my estimator parameters to get the best results.

My methodology is this:

Split my data into training/test.
Use GridSearch with 5Fold Cross validation to train and test my estimators(Random Forest, Gradient Boost, SVC amongst others) to get the best estimators with the optimal combination of hyper parameters.
I then calculate metrics on each of my estimators such as Precision, Recall, FMeasure and Matthews Correlation Coefficient, using my test set to predict the classifications and compare them to actual class labels.

It is at this stage that I see strange behaviour and I'm unsure how to proceed. Do I take the .best_estimator_ from the GridSearch and use this as the 'optimal' output from the grid search, and perform prediction using this estimator? If I do this I find that the stage 3 metrics are usually much lower than if I simply train on all training data and test on the test set. Or, do I simply take the output GridSearchCV object as the new estimator? If I do this I get better scores for my stage 3 metrics, but it seems odd using a GridSearchCV object instead of the intended classifier (E.g. a random Forest) ...

EDIT:
So my question is what is the difference between the returned GridSearchCV object and the .best_estimator_ attribute? Which one of these should I use for calculating further metrics? Can I use this output like a regular classifier (e.g. using predict), or else how should I use it?

Decided to go away and find the answers that would satisfy my question, and write them up here for anyone else wondering.
The .best_estimator_ attribute is an instance of the specified model type, which has the 'best' combination of given parameters from the param_grid. Whether or not this instance is useful depends on whether the refit parameter is set to True (it is by default). For example:
clf = GridSearchCV(estimator=RandomForestClassifier(),
                    param_grid=parameter_candidates,
                    cv=5,
                    refit=True,
                    error_score=0,
                    n_jobs=-1)

clf.fit(training_set, training_classifications)
optimised_random_forest = clf.best_estimator_
return optimised_random_forest

Will return a RandomForestClassifier. This is all pretty clear from the [documentation][1]. What isn't clear from the documentation is why most examples don't specifically use the .best_estimator_ and instead do this:
clf = GridSearchCV(estimator=RandomForestClassifier(),
                    param_grid=parameter_candidates,
                    cv=5,
                    refit=True,
                    error_score=0,
                    n_jobs=-1)

clf.fit(training_set, training_classifications)
return clf

This second approach returns a GridSearchCV instance, with all the bells and whistles of the GridSearchCV such as .best_estimator_, .best_params, etc, which itself can be used like a trained classifier because:
Optimised Random Forest Accuracy:  0.916970802919708
[[139  47]
 [ 44 866]]
GridSearchCV Accuracy:  0.916970802919708
[[139  47]
 [ 44 866]]

It just uses the same best estimator instance when making predictions. So in practise there's no difference between these two unless you specifically only want the estimator instance itself. As a side note, my differences in metrics were unrelated and down to a buggy class weighting function.
[1]: http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV.fit


Does it make sense to randomly select features as a baseline? In my paper, I am saying that the accuracy of classification is $x\%$ when using the top N features.

My supervisor thinks that we should capture the classification accuracy when using N randomly selected features to show that the initial feature selection technique makes an actual difference.

Does this make sense?

I've argued that no one cares about randomly selected features so this addition doesn't make sense. It's quite obvious that randomly selecting features will provide a worse classification accuracy so there's no need to show that using any sort of feature ranking metric will be superior.

Your supervisor is asking to use random feature selection as a baseline. Random performance is a common baseline in machine learning. For example in binary classification with equal numbers of samples from each group, a trained classification should perform better than 50% (random guessing).


How can I fix regression model interpretation of feature? I'm building a regression model to predict the values of a feature $Y$ given a set of other features $X_{1}, X_{2}, X_{3}..X_{n}$.

Onde of these other features, let's say $X_1$, is known to be inversely proportional to $Y$ based on domain's knowledge. The problem is my model is interpreting his coefficient as positive, letting it directly proportional to $Y$. I've tried plenty of different models to verify if I could get better interpretation, such as OLS, Linear Regression, and Logistic Regression, but every model I tried failed to interpret the $X_1$ coefficient.

What can I do to get a regression that better reflects the real-world behavior of this coefficient?

Unless there's a mistake in your code, or the coefficient on $X_1$ is not significant, I'd be inclined to trust the model output.

It's not unusual for data to behave this way. Just because $X_1$ and $Y$ are inversely related with respect to the marginal distribution of $(X_1, Y)$, as can be concluded from a scatterplot of the two variables, does not mean this relationship holds conditional on other variables.

Here is an example where $(X_1, Y)$ are inversely related, but are positively related conditional on another value, $X_2$. (The example is generated using R -- you've tagged python, but this concept is language-agnostic):

library(tidyverse)
library(broom)
set.seed(1)
N <- 100
dat <- tibble(
    x2 = sample(1:4, size = N, replace = TRUE),
    x1 = x2 + rnorm(N) / 3,
    y = x1 - 2 * x2 + rnorm(N) / 5
)
ggplot(dat, aes(x1, y)) +
    geom_point(aes(colour = factor(x2))) +
    theme_bw() +
    scale_colour_discrete("x2")

Here are the outputs of a linear regression model. You'll notice that the coefficient on $X_1$ is negative when $X_2$ is not involved, as anticipated, but is positive when $X_2$ is involved. That's because the interpretation of a regression coefficient is the relationship given the other covariates.

lm(y ~ x1, data = dat) %>%
    tidy()
#> # A tibble: 2 x 5
#>   term        estimate std.error statistic  p.value
#>   <chr>          <dbl>     <dbl>     <dbl>    <dbl>
#> 1 (Intercept)   -0.492    0.154      -3.20 1.83e- 3
#> 2 x1            -0.809    0.0549    -14.7  1.33e-26
lm(y ~ x1 + x2, data = dat) %>%
    tidy()
#> # A tibble: 3 x 5
#>   term        estimate std.error statistic  p.value
#>   <chr>          <dbl>     <dbl>     <dbl>    <dbl>
#> 1 (Intercept)   0.0189    0.0540     0.349 7.28e- 1
#> 2 x1            1.04      0.0681    15.3   1.42e-27
#> 3 x2           -2.05      0.0726   -28.2   1.60e-48

Created on 2020-04-27 by the reprex package (v0.3.0)

This concept extends to more than two covariates, as well as continuous covariates.


Feature reduction convenience In the field of machine learning, I'm wondering about the interest of applying feature selection techniques.

I mean, I often read articles or lectures speaking about how to reduce the number of feature (dimensionality reduction, PCA), how to select the best features (feature selection etc).

I'm not sure of the main purpose of this:

Does feature reduction techniques always improve accuracy of the learned model?
Or is it just a computational cost purpose?

I would like to understand when it is necessary to reduce the number of features and when it is not, in order to improve interpretability or accuracy.
Thanks!

Feature Selection (FS) methods are focused on specializing the data as much as possible to find accurate models for your problem. Some of the main issues that drive the need for FS are:

Curse of dimensionality: Most algorithms suffer to grasp relevant chacteristics of data for a specific prediction task, when the number of dimensions (features) of the data is high, and  the number of examples is not sufficiently big. Check here some more detailed explanation
Correlation between variables: Typically, the presence of highly correlated pair of variables can cause ML algorithms to pay to much attention to a particular effect that is "over-represented". For this reason many FS methods address the reduction of this correlation. Reducing the number of correlated variables oftenly increases the model predictive power.
Latent features: Although specific variables might be highly expressive for your problem, a lot of power can be achieved when finding "latent features", such as linear and non-linear combinations of the original variables. Here there are hundreds of approaches, from PCA to Neural Networks. Independently from the approach (and statistical assumptions) the idea is to create new features that condense the information of a bigger set of features into a smaller one. Hopefuly the new set of features is more representative, and being smaller could be more easily learnt.

Feature selection does not necessarily improve the predictive quality of the model. Reducing or transforming the features might lead you to loss of information and then a less accurate model. Is an open and complex field of research. However in many cases it becomes quite useful. It will depend on how good and different are your original features for describing the target variable.
If you look into bionformatics, you'll see people dealing with thousands or even millions of features, while having only hundred of examples. Here feature selection becomes increasingly relevant.

PS: Most commonly I've seen the term "feature selection" used for the creation of compound features, as most examples I've mentioned, while Feature Extraction term is used for actually removing specific features from the dataset without taking into account is relation with the target variable.


Is window based sequencing a good idea to obtain more training data for LSTMs? I am trying to do an unsupervised autoencoder based outlier detection for time series using LSTMs. Here, there are multiple time series, and an entire series is to be considered as an outlier.
However, I only have around 25-30 time series instances to work on (though each series comprises ~10k points).
I wanted to know whether creating sliding windows for each time series to generate more data is an okay idea for training the autoencoder more accurately. Also, in that case, how does one merge the results to selectively identify which of the original 25-30 time series is an outlier?
I am relatively new at working with LSTMs and would really appreciate suggestions on whether this idea is feasible.

Splitting a time-series into analysis windows, usually with overlap, is quite common practice. In anomaly detection, but also in classification or forecasting. It works great as long as your anomalies can be detected by analyzing such windows independently.
In such a setup, the length of the analysis window becomes a critical hyper-parameter - and will be data/task-dependent.
To get an overall anomaly score you can use any standard statistics to merge scores for individual windows. A simple mean would be the first thing to try, but there are more approaches.


Semi Supervised Learning without label propagation I am trying to cluster some words by affinity.
Using Word2Vec I obtained vector representation of every word that I can cluster with a normal unsupervised method.

Of these words, though, I know the classification of some of them, for example I know:

Colors group ==> red, blue, yellow
Gender group ==> man, woman

UNKNOWN ==> shoes, brown, beautiful.

As the example shows, brown should be categorized as a Color, but shoes and beautiful should be different clusters.

How could I use this information to create a semi-supervised model in order to cluster every word?

While the question is generic, I actually tried to create a program in Python, I tried the scikit methods sklearn.semi_supervised.LabelSpreading and sklearn.semi_supervised.LabelSpreading.
These, though, are not what I need since they only assign the known labels to the remaining objects, meaning that I would end up only with my Colors and Gender group.

One strategy that seems good here is instance-level constrained clustering. These methods are semi-supervised algorithm that have "must-link" and "cannot-link" constraints between instances of known labels. So in your example, you would bind the 4 pairs (red, blue), (red, yellow), (blue, yellow), and (man, woman) as "must-link", and the 6 pairs (red, man), (red, woman), ..., (yellow, woman) as "cannot-link".

The results are similar to unsupervised clustering. For example, if you were to use DBSCAN (ignoring the labels/constraints), you would not need to specify the number of clusters/groups you're trying to achieve, and the algorithm would even find "outliers".

In fact, there is a version of DBSCAN that supports instance-level constraints, called C-DBSCAN. It is described in the work "Density-based semi-supervised clustering" by Ruiz et al (2010).

I do not know of any out-of-the-box implementations available, but I have a working version of C-DBSCAN I implemented for an experiment. However it is not documented nor is it performant/production level. You can find it at my lab's repository if you're interested (also contains the C-DenStream, which is the data streams version of it, but it does not seem to fit your problem).


validation/training accuracy and overfitting If we randomly split the data into training data and validation data, and assume the training data and validation data have similar "distributions", i.e. they are both good representations of the whole data set.

In this case, should the validation accuracy always be roughly the same as the training accuracy if there is no overfitting? Or is it possible that, for some cases, there could exist an "intrinsic" gap between the training and validation accuracy that is not due to overfitting or bad representation of the validation data?

If such "intrinsic" gap exists, how to tell if the gap between the training and validation accuracy is "intrinsic" or caused by overfitting?

I'm not sure what you mean by 'intrinsic'.

A higher training accuracy means you have found a model of the training data that doesn't describe the validation data equally well.  Because of this, any statistically significant difference in training-validation accuracy is due to overfitting.   Figuring out what 'statistically significant' means in this context is not totally trivial, unfortunately, but the general point still stands.


Fastest way for 1 vs all lookup on embeddings I have a dataset with about 1 000 000 texts where I have computed their sentence embeddings with a language model and stored them in a numpy array.

I wish to compare a new unseen text to all the 1 000 000 pre-computed embeddings and perform cosine similarity to retrieve the most semantic similar document in the corpus.

What is the most efficient to perform this 1-vs-all comparison?

I would thankful for any pointers and feedback!

There are libraries that are specialized in exactly that task, for instance FAISS by Facebook AI Research:

  Faiss is a library for efficient similarity search and clustering of dense vectors. It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM. It also contains supporting code for evaluation and parameter tuning. Faiss is written in C++ with complete wrappers for Python/numpy. Some of the most useful algorithms are implemented on the GPU. It is developed by Facebook AI Research.

For cosine similarity, you can use FAISS class IndexFlatIP having normalized the vectors first, as specified in FAISS documentation.

If you want to know more about the techniques behind FAISS, you can have a look at their research paper, and if you want to know more about the fundamentals of similarity search in general, you can have a look at this blog post by Flickr .

Some alternatives to FAISS are Annoy, NMSLib and Yahoo's NGT. You can find a couple of comparisons of these and other libraries here and here.


How to implement a basic query management and recommendation system I'm trying to prototype a system where given a textual query (e.g. a question), I get a list of most relevant documents/questions among a pool of available documents/questions (similar to what we see here in Stack Exchange when submitting a question).

One simple idea coming to my mind is to extract important keywords from query, and do matching based on keywords. Or any other ways, maybe through machine learning or data mining approaches.

I am very new to data science, and I believe this is a very basic system. Is there any available light-weight open source project on this? If not, any hints how to do this?

You might want to take a look at the solr project to see if it is a good fit.


Q Learning Neural network for tic tac toe Input implementation problem I've recently become interested in machine learning, specifically neural networks, and after creating ones to solve basic problems such as XOR and Sin and Cos graphs, however i am now looking into reinforcement learning and specifically q learning with neural networks, to try this out and see if i could implement this, i used a q learning neural network for tic tac toe. However I am unsure as to why my program does not learn correctly. One idea I have is that i have implemented the input neurons incorrectly.I am using visual basic console application so just let me know if you need me to post any code

My view on how to implement this:

Use 18 input neurons and have the first 9 being the state before placing a move, and the next 9 being the state after placing a move.

another question I have in terms of when you teach the neural network, would you feed it the old state and then the state after both you and the opponent have made a move?al networks, so they can play against each other and teach themselves

Second Problem

This next potential problem i have is how I use my threshold and exploration values to choose whether to pick a random action or one the neural network chooses. I increase my threshold linearly from 0 to 1 throughout the iterations. while exploration is a random value between 0 and 1. is this correct? or is there a better way of doing it?

Any feedback would be greatly appreciated, and if anyone has any problems with my question such as it being unclear, not making sense or anything else please let me know so i can fix it

thank you for all who take the time out to try and help.

One idea I have is that i have implemented the input neurons incorrectly.

There is no single correct way to implement the solution to this problem. There are less efficient and more efficient ways for specific problems.

  Use 18 input neurons and have the first 9 being the state before placing a move, and the next 9 being the state after placing a move.

This should work. The first set of 9 neurons will receive a representation of the current game state. The second set of 9 neurons will receive a representation of the intended action.

For predicting Q values from a state, action pair, this is a reasonable approach. Staying with the same architecture of 18 inputs, you could also have the second set of neurons be one-hot encoded to where the agent will to place the 'X' or 'O'.

In the game Tic Tac Toe however, you can look for more compact representations if you like. You have already noticed that because the game is simple and deterministic, that you can represent the agent's action as simply "desired next state". And in fact, the current state does not actually matter to a player, other than to enforce the rules of what are valid next states.

You won't be implementing the rules of the game into the agent - they are part of the environment. Therefore, you can do away with the initial state altogether in your estimate, and work with the end state of each move - this is called the afterstate in the RL literature, and you will find it used a fair bit in deterministic games, or even in non-deterministic games where the randomness happens before the action choice (e.g. in Backgammon). An afterstate representation is more efficient because it encodes the fact that you don't care what route a player took to get to certain board position, you just care about the value of that position as the game continues.

Having said all that, if your goal is to learn basic RL, then you don't need to be looking for the most efficient solution, just one that works. Don't expect your NN-based learner with state-action logic to be the most efficient learner however.

  Another question I have in terms of when you teach the neural network, would you feed it the old state and then the state after both you and the opponent have made a move?

Not as inputs to the network at the same time, no. In your (state, action [=next_state]) representation, your action representation should be the board state for the current player's move and before the other player takes any action. The resulting next state however, will be after the other player takes their action.

If you want to train two separate bots against each other, then each would see the current state, then it would choose an action, then it would either get the reward for winning, or the opponent would take a turn. If the opponent won, then the first bot should receive the (negative) reward. If the opponent's move was not final, then the first bot should see the state after the opponent's move and get to choose its next action.

Again, this is inefficient. For a win/draw/lose game like Tic Tac Toe, you don't need two separate agents, each with their own learning algorithm, in order to train through self-play. You can instead alter Q-Learning slightly to work with the minimax algorithm. In brief this means alternating between the agent selecting actions that maximise the expected reward (for player 1) or minimise it (for player 2).

However, like before, your 2 networks set up should be able to work, and is quite interesting dynamic - you could try different learning parameters, different NNs etc, and see which learns to win quickly (but don't forget starting player has an advantage for early random play, so you'd want to switch which learning algorithm was used for which player to get a fair assessment).

The difference again is in terms of efficiency - a single network inside a modified RL with minimax will typically learn faster that two separate networks.

I have implemented a simple tabular (i.e. no neural networks) Q-Learning based agent for Tic Tac Toe in Python. It uses afterstate representation and modified Q learning with minimax, and learns purely online through repeated self-play.


Preserving explained variance while reducing dimensionality We have a function $f:R^N \rightarrow R$ and a set of points $D=\{ x\in R^N\}$. How is it possible to linearly lower the dimension of points to $M \ll N$ such that the fraction of explained variance* will be preserved as much as possible?

we refer to the fraction of explained variance in prediction of output of $f$ using dataset $D$

$f$ is not defined for $\mathbb{R}^M$, so as well as reducing $x \in \mathbb{R}^N$ to $y \in \mathbb{R}^M$ you are creating a function approximator $g(y) \approx f(x)$.

I suggest using a neural network autoencoder with N dimension inputs and M dimension "bottleneck" layer. You will need to scale inputs and outputs. Using a standard auto-encoder, you will have to measure the fraction of preserved variance in $f(x)$ afterwards. The auto-encoder is not directly going to do that for you, instead it will, like PCA, attempt to use y to encode x (unlike PCA, it may do so nonlinearly).

You could take this further if $f(x)$ is differentiable. Instead of usual mean-squared error as the loss function, you could use $\frac{1}{k}\sum_{i=1}^{k} (f(x_{i}) - f(\hat{x_i}))^2$ - that will encourage the NN to preserve variance in $x$ that matters to $f(x)$. You will have to figure out the gradient function for this analytically. Also bear in mind I have not tried this, just attempted to match your requirement to some theory.

In addition, you will now have $g(y)$ as it will be $f(\hat{x})$. You can generate any $y$ from $x$ by running the first half of the autoencoder, and can generate any $\hat{x}$ from $y$ by running second half of the autoencoder.

You might also be able to adapt t-SNE by using your function $f(x)$ to generate distances that need to be preserved when reducing dimensions.


High, constant training loss with CNN I try to solve a multi-character handwriting problem with CNN and I encounter with the problem that both training loss (~125.0) and validation loss (~130.0) are high and don't decrease. I use the following architecture with Keras:

x = Convolution2D(32, (3, 3), padding ='same', kernel_initializer='he_normal')(model_input)
x = Activation('relu')(x)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Convolution2D(32, (3, 3), kernel_initializer='he_normal')(x)
x = Activation('relu')(x)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Dropout(0.25)(x)
x = Flatten()(x)
conv_out = (Dense(512, activation='relu', kernel_constraint=maxnorm(3)))(x)

lst = [x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11, x12, x13]

sgd = SGD(lr=lrate, momentum=0.9, decay=lrate/nb_epoch, nesterov=False)
model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])

At human check I see that as a result, all predicted labels are a constant sequence.

What are the possible ways of improvement? The training data seems OK (at least as far as any pile of handwriting could be).

I suspect you are facing a dying ReLU problem.  Check the gradients for each layer and see if they are starting to become 0.  Due to the way backpropagation works and a simple application of the chain rule, once a gradient is 0, it ceases to contribute to the model.  Therefore, if you're model is stuck then it's likely that a significant number of your neurons are now dead.

Try using a LeakyRelu or Maxout and see if that helps.


Does the Koalas library allow to use all Pandas machine learning libraries like Scikit-Learn, XGBoost, and TensorFlow? I would like to implement a model based on some cleaned and prepared data set. I already have a bit of experience with PySpark, but from a data scientist's perspective it can be cumbersome to work with it. Therefore I would like to try Koalas. I have a lot of experience with Pandas and hope this API will help me to leverage my skills. My question is: does the Koalas library allow to use all Pandas machine learning libraries like Scikit-Learn, XGBoost, and TensorFlow ? As the Koalas API is relatively new, what draw-backs can be expected besides the lack of documentation ?

I think you have misunderstood the koalas library. You can say its Pandas on Distributed System. You can use Koalas similar to pandas. There are few drawbacks with respect to APIs which is documented in their docs and few articles already written on medium.

You can do your EDA and straight away use them in all the libraries you have mentioned.

Recent pandas 1.0 is faster compared to older versions. It also uses Numba Behind the scene.

Vaex is another library which is available and you can use for EDA but the api names are different from pandas.

Also you can use Modin and dask and use it like pandas with few limitations again.

Clearly these libraries have no dependencies on Sklearn, XGBOOST or TF. You can split train and run your model.


How can I do web scrapping of all the articles containing certain keywords mentioned in a dictionary, using Python? I would like to pull out the title and year of the all articles published in BBC since 2010 based on some keyword search for instance [MBA, LBS, Harvard...... and so on]. How can I do this using python? BeautifulSoup takes a link and extract some text from the link rather than searching the relevant links by itself based on some keywords.

if you want to just pull news articles from BBC news, then you could use their API.

they provide client libraries in different languages (including python)

quoting documentation:

You can search for articles with any combination of the following criteria:

Keyword or phrase. Eg: find all articles containing the word 'Microsoft'.
Date published. Eg: find all articles published yesterday.
Source name. Eg: find all articles by 'TechCrunch'.
Source domain name. Eg: find all articles published on nytimes.com.
Language. Eg: find all articles written in English.

I believe the first one is what you are looking for.


How to calculate the distance between two locations using Haversine Formula? I have the columns of Latitude and Longitude of city like shown below :
             City                        Latitude                 Longitude

1) Vauxhall Food & Beer Garden           -0.123684                  51.485020

2) 14 Hills                              -0.129212                  51.507426

3) Cardiby                               -0.123234                  52.476264

Now I want to calculate distance between the coordinates of specific place with all cities as shown in  data frame . Like
Lon = 0.1245
Lat = 51.685
These above two coordinates should be subtracted with Lat/Lon cities of DataFrame . Output should look like:
      City                        Latitude                 Longitude            Distance

1) Vauxhall Food & Beer Garden           -0.123684                  51.485020    275km

2) 14 Hills                              -0.129212                  51.507426    856km

3) Cardiby                               -0.123234                  52.476264    584km

My code :
from haversine import haversine

from math import sin, cos, sqrt, atan2, radians

df['Latitude'] = [radians(i) for i in df['Latitude']]
df['Longitude'] = [radians(j) for j in df['Longitude']]

lat2 = radians(50.0863)
lon2 = radians(14.4139)

dlon = [i-lat2 for i in df['Latitude']]
dlat = [j-lon2 for j in df['Longitude']]

df['distance']=haversine(df['Latitude'],df['Longitude'],dlat,dlon)

print(df['distance'])

It is not working ,

I am not an expert in geometrics but I found that this function has already been supported by sklearn. So I decided to help you define a generic function.
from sklearn.metrics.pairwise import haversine_distances
from math import radians
import pandas as pd

def distance(location1, lat, lon):
  location1_radian = [radians(_) for _ in location1]
  location2 = [lat, lon]
  location2_radian = [radians(_) for _ in location2]
  result = haversine_distances([location1_radian, location2_radian])
  result = result * 6371000/1000  # multiply by Earth radius to get kilometers

  return result[0][1]

df = pd.DataFrame({
    'Latitude': [-0.123684, -0.129212, -0.123234],
    'Longitude': [51.485020, 51.507426, 52.476264]
})

df.apply(lambda x: distance([0.1245, 51.685], x.Latitude, x.Longitude),axis=1)
# 0    35.440884
# 1    34.434944
# 2    92.195943
# dtype: float64


Keras Model Predict is not predicting all images flowing from directory? I have the following code where I have done all the training and passed the testing set as a flow from directory. After that when I pass that object into the model.predict option, the array received is not of the same length as the test set length. Code:

PATH = '/content/testing'
testGen.reset()
testGen = valAug.flow_from_directory(
    PATH,
    class_mode="categorical",
    target_size=(75, 75),
    color_mode="rgb",
    shuffle=False,
    batch_size=BS)

predIdxs = model1.predict_generator(testGen,
    steps=(totalTest // 32))
print(len(predIdxs))
# for each image in the testing set we need to find the index of the
# label with corresponding largest predicted probability
predIdxs = np.argmax(predIdxs, axis=1)
print(len(predIdxs))
import pandas as pd
from glob import glob
test_df = pd.DataFrame()
id = []
for x in glob('/content/testing/0/*'):
    id.append(x)
for x in glob('/content/testing/1/*'):
    id.append(x)
test_df['id'] = id
test_df['category'] = predIdxs
print(test_df)
test_df.to_csv('submission.csv', index=False)

After that the output I got is as follows:

Found 55505 images belonging to 2 classes.
55488
55488
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-81-09c1488e5ac6> in <module>()
     25     id.append(x)
     26 test_df['id'] = id
---> 27 test_df['category'] = predIdxs
     28 print(test_df)
     29 test_df.to_csv('submission.csv', index=False)

3 frames
/usr/local/lib/python3.6/dist-packages/pandas/core/internals/construction.py in sanitize_index(data, index, copy)
    609
    610     if len(data) != len(index):
--> 611         raise ValueError("Length of values does not match length of index")
    612
    613     if isinstance(data, ABCIndexClass) and not copy:

ValueError: Length of values does not match length of index

Lenght of test set(totalTest = 55505) but only 55488 data is predicted. Why is data lost here?
P.S: The model I have used is a pretrained Inception V3 model where I have downloaded the weights beforehand and run the model. I got about 85% accuracy. And I have tried the same method using Resnet block also and I have received the results without error. Why am I getting an error here? Any help would be appreciated.

I presume that your generator is at fault here :

predIdxs = model1.predict_generator(testGen,
    steps=(totalTest // 32))

You do an integer division on the size of your test set, but the result is not an integer and thus truncated (floored). Later on you (presumably) use original size for one column, and want to assign the data from your predict_generator (which is shorter) to another column of that same dataframe, creating a mismatch.


How to find the filename associated with a prediction in Keras? My question is really simple, how to find the filename associated with a prediction in Keras? That is, if I have a set of 100 test samples named  and I get a numpy array which contains the estimated class probabilities, how do I map the filenames to the probabilities?

import cv2
import os
import glob

def load_test():
    X_test = []
    y_test = []
    os.chdir(testing_path)
    file_list = glob.glob('*.png')
    for test_image in file_list:
        img = cv2.imread(test_image,1)
        X_test.append(img)
        y_test.append(1)
   return X_test,y_test

if __name__ == '__main__':
   X_test = np.array(X_test, dtype = np.uint8)
   X_test = X_test.reshape(X_test.shape[0],3,100,100)
   X_test = X_test.astype('float32')
   X_test /= 255

The order of the files that populate file_list, is the same order X_test appears in, by row.

So just match the indices to correlate filename with prediction.

X_test[0] ~ prediction[0] ~ file_list[0]


How do you work with Latent Dirichlet Allocation in practice One need to provide LDA with a predefined number of latent topics. Let say I have a text corpus in which I hypothesize there are 10 major topics, all composed of 10 minor subtopics. My objective is to be able to define proximity between documents.

1) How do you estimate the number of topics in practice ? Empirically ? With another method like Hierarchical Dirichlet Process (HDP) ?

2) Do you build several models ? For major and minor topics ? Is there a way to capture the hierarchical structure of the topics ?

There are many methods of performing this optimization- namely, choosing the optimal number of topics to supply for LDA and many papers have been authored on the topic.

Several of note, which each define metrics by which to evaluate LDA models for quality of topics are:

Rajkumar Arun, V. Suresh, C. E. Veni Madhavan, and M. N. Narasimha Murthy. 2010. On finding the natural number of topics with latent dirichlet allocation: Some observations. In Advances in knowledge discovery and data mining, Mohammed J. Zaki, Jeffrey Xu Yu, Balaraman Ravindran and Vikram Pudi (eds.). Springer Berlin Heidelberg, 391–402. http://doi.org/10.1007/978-3-642-13657-3_43
Cao Juan, Xia Tian, Li Jintao, Zhang Yongdong, and Tang Sheng. 2009. A density-based method for adaptive lDA model selection. Neurocomputing — 16th European Symposium on Artificial Neural Networks 2008 72, 7–9: 1775–1781. http://doi.org/10.1016/j.neucom.2008.06.011
Romain Deveaud, Éric SanJuan, and Patrice Bellot. 2014. Accurate and effective latent concept modeling for ad hoc information retrieval. Document numérique 17, 1: 61–84. http://doi.org/10.3166/dn.17.1.61-84
Thomas L. Griffiths and Mark Steyvers. 2004. Finding scientific topics. Proceedings of the National Academy of Sciences 101, suppl 1: 5228–5235. http://doi.org/10.1073/pnas.0307752101

As luck would have it, if you're using R, these metrics have already been compiled for you in a convenient package called ldatuning which provides a  set of utilities and metrics to help tune the correct number of topics within LDA models.

Alternatively, if you're using Python, the gensim package can provide you with a lot of utilities to assist. For example, the package implements a metric that they call "topic coherence" that they claim corresponds roughly to clarity of  topic distinction for a human among many other utilities for tuning.


How to visualize time-intervals I have a pandas table with a start and end time for each entry. They don't intersect, but range from seconds to days. I want to visualize them, ideally the x-Axis being the date on which they occus, and the y-axis the time. However, I haven't found a good way to achieve this.

One thought I had were using Boxplots (one Box with one start and one endtime per entry) and grouping by [year, month, day], however this idea doesn't work when the interval starts and ends on a different day.

Wolfram recently released a free Wolfram Engine that can be called in Python (and other languages) so you can use its functions like TimelinePlot.

First get some dateIntervals by using the start and end dates of 2017 "TropicalStorm" hurricanes.

ec = EntityClass["TropicalStorm", "Hurricanes2017"];
timeIntervals = Interval /@ EntityValue[ec, {"StartDate", "EndDate"}];

timeIntervals contains Intervals of DateObjects. The first 3 of these are

timeIntervals[[;; 3]]

A basic TimelinePlot of timeIntervals gives

TimelinePlot[timeIntervals]

However, TimelinePlot has many options to customise it.  For example, take timeIntervals, GatherBy "Month", SortBy the first date of first interval in each set (see Part), and add an Automatic LineLegend using the "MonthShortName"s of each set as legend keys.

With[{
  byMonth =
   SortBy[#[[1, 1, 1]] &]@GatherBy[timeIntervals, DateValue[#[[1, 1]], "Month"] &]
  },
 TimelinePlot[
  byMonth,
  PlotLegends ->
   LineLegend[Automatic, DateValue[#[[1, 1, 1]], "MonthNameShort"] & /@ byMonth]
  ]]

Hope this helps.


Efficient way to search list of items in a text document I have a list of items (size ~50K) and several documents( average page per document ~10).
I am trying to find what all items are listed in each document as follows :

for document in documents_folder:
    document_text = extract_content(document)
    for item in item_list:
        if item is found in document_text:
             save(doument,item)

This takes ~5 sec per document to execute.

How can this be optimized ?
Is this the best approach ? Is there a better way to do this ?

Any suggestion is appreciated

I have not tried this before but I will suggest:

Get all unique items from all documents:
Create a dictionary where key is that unique item and values is list of document ids (document names) where this item is present.

Once you are done. Just search your item in the dictionary key, it will return all the documents ids where this item is present.

This algorithm is called Inverted Indexing in Information retrieval.


Explainable anomaly detection There are plenty of working for explaining prediction in supervised learning (e.g. SHAP values, LIME).

What about for anomaly detection in unsupervised learning? Is there any model for which there are libraries that can give you justifications, such as "row x is an anomaly because feature 1 is higher than 5.3 and feature 5 is equal to 'No'"?

The LIME framework can probably be used to do this as well.

Outlier detection sets a specific label to outliers (say 1), and another one to inliers (say 0). From then on, you can train interpretable models (decision trees for instance) to predict the labels set by your unsupervised model.

I don't know much about SHAP values, but I guess, with this approach, you could do the same.


Why training and validation similar loss curves lead to poor performance I am training a binary classification neural network model using matlab the graph that I got using 20 neurons in hidden layer is given below.
the confusion matrix and graph between cross entropy vs epochs.

to prevent overfitting in a model the training curve in a loss graph should be similar to the validation curve.

but in the current situation the third graph shows curve where validation curve is similar to training although the overall accuracy is low as compared to the curve where the two curve diverges in the above plot.

WHy this is happening and what I am doing wrong in understanding these curves?

to prevent overfitting in a model the training curve in a loss graph
  should be similar to the validation curve.

That's not always the case, maybe the train set has slightly a different distribution from the validation set, therefore, our focus is just when training loss decreases while validation loss slightly or never decreases.

In the former graph, you notice there was a slight increase in the validation error while the training error was decreasing, while the latter graph has the same behavior due to a less complex model, the model was not able to fit the training data well.


Where does the random in Random Forests come from? As the title says:

Where does the random in Random Forests come from?

For each tree you randomly select from the variables that you can use to split tree nodes. Generally you randomly select 1/3 of the variables per tree.


Should I accept a model with good validation loss & accuracy but bad training one? I am training a model for 3D CT scan classification (is there a nodule or not in the image).
I'm using a 3D CNN and I obtaint the following curve;

(blue curve is the validation data (10% of dataset))
as you can see, I obtain good result on both validation loss (categorical cross entropy, up to 0.16 for val) and metric (categorical accuracy) (up to 1.0 for val) but not on the training data.
I'm using a grid deformation on the training data and several dropout layer since my dataset is very small (120 samples), so it could explain the lower result on training set.
Still, I dont know if I can accept this type of result ?

The validation set having just 12 samples is too small to consider deploying a model which performs well on the validation set.
A model that performs well on the training set with 110 samples and then performs better on the small validation set would be preferable.
Since your set is too small, all though it is not prudent, you should consider running the model on the entire 120 samples to see how it performs on the entire dataset.
I presume you can't get more samples, if you can, that would be a good step.


Can I create pretrain model with tensorflow? I takes a long time for train neural network model. It have to train every time when I run code. If I get high accuracy from training , Can I use same training model with another code without new training anymore ? How to do?

You don't need to train your model each time form the beginning. After training, you can store you model and next time you can load it, train it from that point or use it for your recall phase. You can find good tips here and here which have discussed storing and loading models in Tensorflow.


cross validation issues I have come here from this great answer. I have come across many approaches for using cross validation and the answer to the attached question is by far explaining it the best to me. My dilemma is that now that I m not able to figure out what to use Kfold cross validation for:-

Testing overfitting?
Hyperparam tuning?
Any other use case?

and that too how? I am unable to figure out what to do with the average score that comes after kfold cross val, what to do with the folds and what to do with a model trained on k-1 folds of train data?

Cross validation is basically for hyper-parameter tuning.

You train a set of model hyper-parameter setting on kfold cross validation and take the average score from kfold cross validation as the approximate performance of each model hyper-parameter setting. Then, the model hyper-parameter setting which has the highest score will be the choice of your model setting. This hyper-parameter setting can be treat as the best you can get from this kind of model.

Later, you can use this model setting to test the general performance on your test dataset.


Tensorflow's .shuffle(BUFFER_SIZE) I came across the following function in Tensorflow's tutorial on Machine Translation:
BUFFER_SIZE = 32000
BATCH_SIZE = 64
data_size = 30000
train_dataset = train_dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True)

I went through several blogs to understand .shuffle(BUFFER_SIZE), but what puzzles me is the fact that a BUFFER_SIZE > DATA_SIZE results in a perfectly uniform shuffling. Neither do I understand what they mean by 'uniform shuffling', nor do I understand how a BUFFER_SIZE> DATA_SIZE is even possible.
From what I understand, tensorflow keeps a BUFFER_SIZE of elements, selects a random element and adds the next input element into the buffer. This makes sense if the BUFFER_SIZE is <= DATA_SIZE. But, what happens to the buffer in case we have more number of elements than the size of the dataset? Do we not have some NULL elements? How does it result in uniform shuffling?
Could anyone please explain to me with an example of how BUFFER_SIZE > DATA_SIZE results in a uniform shuffling? And, what exactly do we mean by uniform shuffling?

Shuffling begins by making a buffer of size BUFFER_SIZE (which starts empty but has enough room to store that many elements). The buffer is then filled until it has no more capacity with elements from the dataset, then an element is chosen uniformly at random. This means that each example in the buffer is equally likely to be chosen, with probability 1/BUFFER_SIZE. Then, a new example is loaded to fill the slot in the buffer that was emptied. This continues until there is nothing left to load.
A uniform shuffle would be what you would think of as truly random: any sequence of examples is equally likely. If the buffer is smaller than the size of the dataset, this is not possible. Here is an example of why: imagine I have ten examples, labelled 1 to 10, and a buffer of size 2. First I load examples 1 and 2 into the buffer, then I have no more space so I must randomly select something from the buffer. Therefore, my random shuffle always begins with example 1 or 2: not uniformly random!
If you have a buffer as big as the dataset, you can obtain a uniform shuffle (think the same process through as above). For a buffer larger than the dataset, as you observe there will be spare capacity in the buffer, but you will still obtain a uniform shuffle.


Why the validation accuracy is constant for all the pre-trained models while applying on medical datasets with .czi format? #import all the libraries
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix,classification_report
import os
import sklearn.utils

path='/content/drive/My Drive/Drug/spheroid_imageset/untreated/training_set'

X=np.load(path+'/X.npy')
Y=np.load(path+'/Y.npy')

X=X.reshape(-1,224,224,1)

X,Y = sklearn.utils.shuffle(X,Y)

tf.keras.callbacks.EarlyStopping(
    # monitor="val_loss",
    # min_delta=0,
    patience=3,
    verbose=0,
    mode="auto",
    baseline=None,
    restore_best_weights=False,
)

image_size=224

base_model = tf.keras.applications.MobileNet(weights=None,include_top=False,input_shape=(image_size,image_size,1))
base_model.trainable = False

x = base_model.output
x = tf.keras.layers.GlobalAveragePooling2D()(x)
x = tf.keras.layers.Dropout(0.50)(x)
x = tf.keras.layers.Dense(16,activation='relu')(x)
x = tf.keras.layers.Dense(3,activation="softmax")(x)
model = tf.keras.Model(inputs=base_model.input, outputs=x)
model.compile(optimizer='adam',loss='sparse_categorical_crossentropy',metrics=['accuracy'])

history=model.fit(X,Y,epochs=50,validation_split=0.25,callbacks=[callback])

weights: One of None (random initialization), 'imagenet' (pre-training on ImageNet), or the path to the weights file to be loaded. Default to imagenet.

With weights=None, it is randomly initialized and
With base_model.trainable = False, it is not being trained.
So, basically your Conv. layer is not getting trained.


Android: NLP library for date recognition in string I am currently working on an android app which should make appointments automatically by reading the incoming messages from your mobile phone. I've managed to create a service which monitors the incoming messages, but now I need an Natural Language Processing algorithm in order to find the date for the appointment.

I've tried DialogFlow, but I found out it cannot be used offline and that is not the purpose of the app. It should work offline too!

Does anyone have a suggestion for a library I can use?

Maybe too late, but you could have a look at HeidelTime. It is a Java library that can detect dates, times, durations and time sets in texts for many languages. The downside of HeidelTime is that it seems to be not maintained since 2018. Also, you need the TreeTagger tool in order to use it. But the README of HeidelTime explains the necessary steps quite detailed.

Another alternative or addition to HeidelTime would be defining several regular expressions of common date and time formats and find these in your input text.


Can points not being passed to the loss function influence it? If I were to construct a model where only a subset of the training data is passed to the loss function, can the other parts of the dataset influence the fit?
More concretely, in this first example:
model.fit(x_train_subset, y_train_subset, val=[x_val, y_val])

Here, I am passing the model, and then the loss function, only the subset to be trained on.
In another example:
model.compile(loss=CustomLoss) #this loss will be based on x_train_subset
model.fit(x_train, y_train, val=[x_val, y_val])

Here, I have passed the entire training set to the fit, but using only a subset of them to find the loss.
My questions are:

Conceptually, are these the same?
If a point is not being passed to the loss function is it effectively invisible to the model training procedure?

1.Technically they are the same.They will compute the same loss. Although there will be a subtle difference in performance. In the first case gradient will only be calculated for the subset but in the second case gradient will be calculated for the whole training set but only the gradient of subset will be used to compute cost and update.
2.No, the points won't be invisible to the training procedure. the gradient will be calculated for all of the points but they won't be used.


How to Setup Multivariate Time Series Dataset for Classification First post on StackExchange. I’m fairly new to ML, with about 1 year of experience so please pardon any ignorance or misuse of terms.

I have a multivariate time series dataset where I would like to predict the likelihood of an outcome of 1/0 (think of this as a Conversion) in the next 3 time periods for unique ObjectID’s. I have a table that takes snapshots of ObjectID over time, and I want to use current data to predict if the ObjectID will convert (1 or 0) in the coming 3 time periods. Once an ObjectID reaches conversion, it will stay as such.

There’s a twist: some of the X variables are static, they do not change with time. I call these attributes, there are 8 in my dataset. These are essentially characteristics ObjectID in question.
I have 2 variables that change with time, Age (in months) and a categorical variable with 7 levels through which the ObjectID progresses. Here’s how the data looks:

ObjID   Age  Time      Attr1    Attr2   Att3    CurrCat Conversion
id1234  0   1/1/2019    ABC      XYZ    HIJ        A       0
id1234  1   1/2/2019    ABC      XYZ    HIJ        B       0
id1234  2   1/3/2019    ABC      XYZ    HIJ        A       0
id1234  3   1/4/2019    ABC      XYZ    HIJ        D       0 <-- current time
id6789  0   1/1/2019    CBA      ZYX    JIH        C       0
id6789  1   1/2/2019    CBA      ZYX    JIH        C       0
id6789  2   1/3/2019    CBA      ZYX    JIH        D       1
id6789  3   1/4/2019    CBA      ZYX    JIH        A       1

How can I setup this dataset for a classification or decision tree model?

I'll be building the model in Python, so any suggested packages would be helpful too.

I have good experiences with Keras LSTM. Think it should also work with the time constant features. Here is a helpful tutorial. https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/


Why would removing a variable in adaboost decrease error rate? I was trying to classify an outcome on some data using adaboost (the ada package in R) and I was playing around with the training data set of descriptors when I realized that removing a column in the descriptor matrix increased the accuracy of the output on the training data. Specifically, the number of false negatives dropped/true positives increased.

Aside from removing a single column in the descriptors, I left everything else the same, including number of iterations.

Imagine that one of the column is just random data -- then it's not informative at all, so no classifier will be improved by including it.

However, ada's stochastic boosting implementations will always have some chance of including that variable in the classifier it generates. As a result, removing it has the potential to improve the classifiers generated.

(In your case, you might check whether that variable is part of the final model generated.)


How to get an aggregate confusion matrix from n different classifications I want to test the accuracy of a methodology. I ran it ~400 times, and I got a different classification for each run. I also have the ground truth, i.e., the real classification to test against.

For each classification I computed a confusion matrix. Now I want to aggregate these results in order to get the overall confusion matrix. How can I achieve it?

May I sum all confusion matrices in order to obtain the overall one?

There are a few ways to achieve your "master confusion matrix".

Sum all the confusion matrices together:  Like you suggested, summing this results in a confusion matrix.  The problem with this is you can not interpret totals.
Average the entries.  This method is the same as number one, but you divide each entry by the number of trials (~400 in your case).  This would be my preferred method because then you can actually translate each category to a (mean) +- (an error measurement) and actually see which categories are the most volatile or stable.  Careful with interpreting this 'error measurement' though.
Report a problem specific measurement of the confusion numbers.  For example, if your numbers have outliers, medians would preferred over means.

There are other statistics that are possible to report as well.  You can redo the method to keep track of individual classifications.  Then we can say other important stats like '% of classifications that stay the same and are accurate', etc...