Are there any reports on Hardware errors affecting experiments? I recently wrote this as a list of what I have seen / can think of as problem sources that make it hard to reproduce (replicate?) an experiment. I think I have seen most of them, except for hardware errors.

My thought was this: A typical computer vision experiment might run a training on a GPU for many hours / several days. Even if a bit-flip happens only once every $10^9$ FLOPs, that would be 11500 errors per second with an Nvidia Titan GTX 1080 Ti. I don't know how this error would affect later calculations (how the problem is numerically conditioned).

So: Are there any reports on Hardware errors affecting experiments?

(Blog posts, journal articles, posters?)

Memory corruption seems to be an issue that is important enough for companies to buy expensive ECC memory for their clusters.
The Wikipedia article on ECC memory lists some causes for memory corruption, including (to my surprise and delight) cosmic rays and ingenious hackers.


Can the performance of a CNN be dependent on the train-test-val split random seed? I am doing multi-class classification and comparing the effects of 2 image enhancement techniques (IET).
IET 1 performs better than IET 2 at random seed x (for train-test-val split)
IET 2 performs better than IET 1 at random seed y (for train-test-val split)
Is this normal, or am I doing something wrong?

Yes, it can depend on it because it changes the data distribution with the network is trained.
You shouldn't consider random seed as a hyperparameter. Keep the same random seed and run comparison. Do this for at least 5 or 10 random seeds. You will definitely have a winner. If you don't go for more than 10 unless you get a winner. That will give you sufficient proof about the efficacy of the method. Otherwise since one is good in random seed or other one in other, just choose one of those random seeds, choose the method that gives best in it and continue your work.


Is there a clustering algorithm which accepts some clusters as input and outputs some more clusters? Heres the task: I have data I don't know much about. The final task is to build a classifier to classify the samples into a few categories. Some of the categories are pretty clear, we can easily use these as labels for a classifier. But I guess there are more useful categories possible, because right now most of my samples don't belong to any category. As I am no expert in the specific field, I would like to use some clustering algorithm to show possible label ideas.
When using traditional clustering algorithms, they find all sorts of patterns in the data I am not interested in.
So I am looking for a way to tell the algorithm: "Hey, find some clusters in my data, but please take the existing clusters (or labeled data) into account." This should tell the clustering algorithm what I am interested in, and in what not.
Does something like this exists? Or any other idea how to solve the problem of finding additional labels?
BTW: in my case, I am doing NLP.

You are describing semi-supervised learning where the training dataset is only partially labeled.
One common set of techniques to solve that problem is active learning. In active learning, there is a learning loop where the algorithm makes predictions and a human corrects those predictions.
Pre-clustering is a specific active learning technique to address the problem you describe. The goal is to repetitively select the most representative training examples to add new labels and as well as avoiding repeatedly labeling samples in same cluster. "Active Learning Using Pre-clustering" by Nguyen and Smeulders goes into greater detail.


Use liblinear on big data for semantic analysis I use Libsvm to train data and predict classification on semantic analysis problem. But it has a performance issue on large-scale data, because semantic analysis concerns n-dimension problem.

Last year, Liblinear was release, and it can solve performance bottleneck.
But it cost too much memory. Is MapReduce the only way to solve semantic analysis problem on big data? Or are there any other methods that can improve memory bottleneck on Liblinear?

Note that there is an early version of LIBLINEAR ported to Apache Spark. See mailing list comments for some early details, and the project site.


predict an array like [1,2,3] increasing I want to generate a simple model and classify it with decision tree.

The idea is if numbers in an array are increasing then what I need is that.

eg.

[1,2,3]
[2,4.2,5.6]

these paterns like increasing will be predicted as true

and other then those will be predicted as false

How can I achieve this with decision trees?

Machine learning is not needed to solve that problem. This deterministic code tests if numbers in an array are increasing:

def strictly_increasing(nums):
    "Deterministically check for only increasing values"
    return all(x2 > x1 for x1, x2 in zip(nums, nums[1:]))

assert strictly_increasing([1, 2, 3])
assert strictly_increasing([2, 4.2, 5.6])
assert not strictly_increasing([1, 1, 1])
assert not strictly_increasing([3, 2, 1])


improving performance for a limited dataset with noisy images, pattern recognition I am trying to recognize doodles in noisy images like in this one below. My dataset consists of only 10 000 images and 30 categories I've implemented a CNN but it is giving me a 6% accuracy. I am thinking about removing the noise before feeding the images to my CNN, but I have no idea which methods to use to remove this type of noise and I am not even sure that removing noise improves the NNs performance. Do you have any suggestions about methods to remove the noise or other tips to improve the performance of the model?

There are some techniques that you can research and apply. My first advice would be decrease the number of the neurons that you implemented already on your hidden layers. You can use some regularization and dropout functions. There are some denoising techniques as well if the problem is exactly this denoising data. I would recommend you to detect the real problem because usually there is a method calls "data augmentation" to overcome overfitting problems which increase the dataset with noisy data in a logical way.
As I mentioned above there are some techniques for denoising. Here is the link that you can look for more details about them. Also there are further techniques to approach this denoising but I highly recommend to detect your exact problem and improve your question.
Maybe it is related to your network modelling because 6% is very low. You should get more useful data at the end with this much dataset.


What is the best metric to evaluate highly imbalanaced binary classifiction? (such as fraud detection in credit card) What is the best metric to evaluate highly imbalanaced binary classifiction? (such as fraud detection in credit card?

I have examining several metrics precision recall F1 lassification Report (macro avg,weighted avg), ROC, AUC,.. but I do not know what is more acceptable to evaulate highly imbalanced binary classification such as credit card fraud detection https://www.kaggle.com/mlg-ulb/creditcardfraud

Simple precision/recall/F-score is perfectly suitable for imbalanced data. It should be computed on the minority class of course.

precision says how often the system is correct when predicting an instance in the minority class.
recall says how often the system detects an instance which belongs to the minority class (this is usually the hardest part with imbalanced data)
As usual F-score can be used as a "summary metric" of both precision and recall.

ROC and AUC (which are also based on precision/recall) can be used only with a soft classifier, i.e. when the system predicts a numerical value instead of a binary label.

Macro/micro/weighted average (usually over F-score) are irrelevant: averaging over the two classes would take into account the majority class which is very easy to predict, so it would mask the useful information about the minority class.


What is the difference between a data-driven model and an empirical model? Are they the same?
Empirical models, per Wikipedia, are

  any kind of (computer) modelling based on empirical observations rather than on mathematically describable relationships of the system modelled.

But since data-driven models are also based on observations, what is the difference?

They are the same thing, but it is important you understand differences in process between the statistics school of thought and the data mining mining school of thought. In data mining we impute and approximate relationships, where as some relationships are certain and have precise computations. Obviously, it is always better to compute something precisely than to approximate a relationship.

An example in finance: Bond prices have precise computations - they follow a certain relationship. The are simply discounted cashflows. Other types of securities might require inference because the absolute relationship is unknown, likely because either the cashflow or the interest rate is not known.

Empirical means based on evidence or observations. You are testing something. It might be possible to generate data per a simulation - this is common with genetic models - that is not observed in real life. In that instance, your data driven model would not be empirical.

  em·pir·i·cal
  /əmˈpirik(ə)l/
  | adjective |
  based on, concerned with, or verifiable by observation or experience rather > than theory or pure logic.
  "they provided considerable empirical evidence to support their argument"

Data mining versus statistics:

Normally what you try to do in statistics (as opposed to data science / data mining) is you come up with the definate mathematical model first, the you test it empirically. If the relationship is statistically significant, than it is thought to be a good model.

In data mining you seek out the inferred relationships first and have no manner of testing them.


How to create interactive plot of thousands of images as output of t-SNE? I have many images that I want to plot as a result of running t-SNE and I want to be able to interactively explore them.  matplotlib does not allow enough interactivity to explore, and plotly is too slow.

Is there an easy way to plot thousands of images in a way that different patterns can be explored?

It is certainly possible to create interactive plots of many thousands of images, as Google has done in their interactive art t-SNE Map.

However, as far as I have found, there is not any canned way to do this.  https://github.com/YaleDHLab/pix-plot is an open source application that also does an impressive job at plotting thousands of images, and if you are using python (e.g. with scikit-learn's t-SNE), it does not look too bad to modify to your use case.


Denormalise data in Neural Networks I wrote a Neural Networks prediction model in Python.

My data has a few inputs and two outputs. In order to make it work, I have to normalise every column on data for good prediction results.

However, I had an issue. I run several times the prediction model with the same inputs to get the average and standard deviation of it. But obviously, those are also normalised to 0-1. On the test data I knew the min and max, so I could denormalised them. On the prediction values, I cannot the min and max real value.

How do you solve this kind of problem and if you cannot, are there any other decent prediction techniques without the need of normalisation?

Normalise using your training data statistics. Save the values used (e.g. mean and sd per feature), treating them as part of your model. Once you have used these values to transform input, they become fixed translate/scale factors in the model.

Use the same values to normalise test data or new inputs as required. You do not need to calculate new normalisation constants for new data. In fact doing so will most likely reduce the effectiveness of your model.

The same principle applies to interpreting output values if you need to scale those into range that your model produces. Scale according to your training data.


How to incorporate keyboard positions on character level embeddings? I am working with NLP and have character level embeddings.

I have embeddings learned from Wikipedia text.

Now, I want to learn embeddings from chat data (where misspellings and abbreviations are way more common). Usually, the character n doesn't follow from character b, however, during texting, this can be common because they are close together on the keyboard, and a misspelling occurs.

So, my questions is: what are strategies to incorporate character keyboard position information to a traditional character level embedding?

Note: it can be assumed that only QWERTY keyboards exist.

Character keyboard position information is an example of noisy channel model information, an error that depends on how a word is transmitted. It is very common to add noisy channel model information to spell checkers, including spell checkers that use character-level embeddings.

Most character-level embedding models would automatically learn to model common transmission mistakes. Characters that are frequently confused in the dataset would be embedded closer to each other because they frequently co-occur. There would be minimal gain by explicitly adding channel information to a character-level embedding model during training.


How to ensemble classifier incorporating all features in python? I am doing a text classification task(5000 essays evenly distributed by 10 labels). I explored LinearSVC and got an accuracy of 80%. Now I guess whether accuracy could be raised by using ensemble classifier with SVM as base estimator?

However, I do not know how to employ an ensemble classifier incorporating all the features? Please note that I do not want to combine the different features directly in a single vector.

Therefore, My first question: in order to improve the current accuracy, is it possible to use ensemble classifier with svm as base estimator?
My second question How to employ an ensemble classifier incorporating all features?

Boosting - is the ensemble which tries to add new models that do well where previous models lack. Bagging in scikit lets you send the base classifier as the parameter. Go through 1.11 of the link to understand more!

But since, you already have in mind that SVM performs better Voting Classifier which is present in sklearn.ensemble lets you give weights to the classifiers which you seem to be performing well. For instance in your question, SVM can be given more weight. It also has another parameter 'voting'. If hard, uses predicted class labels for majority rule voting else if soft, predicts the class label based on the argmax of the sums of the predicted probabilities.


Two-class classification model with multi-type input data As a first introduction to machine learning and keras, I just finished reading Deep Learning with R by François Chollet with J.J. Allaire. I would like to extend the book's IMDB example of two-class classification to a multi-input version using functional API.

library(keras)

## Keras IMDB example

#load data
imdb <- dataset_imdb(num_words = 10000)
train_data <- imdb$train$x
train_labels <- imdb$train$y
test_data <- imdb$test$x
test_labels <- imdb$test$y

#Encoding the integer sequences into a binary matrix
vectorize_sequences <- function(sequences, dimension = 10000) {
results <- matrix(0, nrow = length(sequences), ncol = dimension)
for (i in 1:length(sequences))
results[i, sequences[[i]]] <- 1
results
}

x_train <- vectorize_sequences(train_data)
x_test <- vectorize_sequences(test_data)
y_train <- as.numeric(train_labels)
y_test <- as.numeric(test_labels)

#Defining the model
model <- keras_model_sequential() %>%
layer_dense(units = 16, activation = "relu", input_shape = c(10000)) %>%
layer_dense(units = 16, activation = "relu") %>%
layer_dense(units = 1, activation = "sigmoid")

#Compiling the model
model %>% compile(
optimizer = "rmsprop",
loss = "binary_crossentropy",
metrics = c("accuracy")
)

#Training the model
history <- model %>% fit(
partial_x_train,
partial_y_train,
epochs = 20,
batch_size = 512,
validation_data = list(x_val, y_val)
)

#Defining the model using functional API
model <- keras_model_sequential() %>%
layer_embedding(input_dim = 10000, output_dim = 8,
input_length = maxlen) %>%
layer_flatten() %>%
layer_dense(units = 1, activation = "sigmoid")

So, I would like to create a similar model where we predict class A and B based on 1 continuous and 3 categorical input data. In this dummy example, continuous1, categorical1 and categorical2 are 1D tensors while similarly to the IMDB example, categorical3 is a 2D tensor of shape (samples, indices) with length num_index=20 and are one-hot encoded.

#Dummy data
dat <- data.frame(samples=c(rep(paste0("A_",1:9800)),rep(paste0("B_",9801:10000))))
dat$label <- c(rep("A",9800),rep("B",200))
dat$continuous1 <- c(rnorm(9800, 100, 45),rnorm(200, 0, 25)) #may    normalize to unit variance
dat$continuous1[dat$continuous1<0] <- 0
dat$categorical1 <- c(rep(1:100,98),rep(1:100,2))
dat$categorical2 <- c(rep(1:98,100),rep(99:100,100))

pool_A <- factor(1:15,levels=1:20)
pool_B <- factor(16:20,levels=1:20)
dat_categorical3 <- vector("list",10000)
names(dat_categorical3) <- c(as.character(dat[dat$label=="A",]$samples),as.character(dat[dat$label=="B",]$samples))
for(i in 1:9800){
  dat_categorical3[[i]] <- (as.numeric(table(sample(pool_A, 20, replace=T))) > 0) + 0L
}
for(i in 9801:10000){
  dat_categorical3[[i]] <- (as.numeric(table(sample(pool_B, 20, replace=T))) > 0) + 0L
}

As I've never built a Keras model before, I'm a bit confused on how to set up the layers representing the different data inputs using functional API. Any suggestions and feedback are highly appreciated, thanks!

With the help from a colleague, we came up with this:

set.seed(666)

#Dummy data
dat <- data.frame(samples=c(rep(paste0("A_",1:9800)),rep(paste0("B_",9801:10000))))
dat$label <- c(rep("A",9800),rep("B",200))
dat$continuous1 <- c(rnorm(9800, 100, 45),rnorm(200, 0, 25))
dat$continuous1[dat$continuous1<0] <- 0
dat$categorical1 <- c(rep(1:100,98),rep(1:100,2))
dat$categorical2 <- c(rep(1:98,100),rep(99:100,100))

pool_A <- factor(1:15,levels=1:20)
pool_B <- factor(16:20,levels=1:20)
dat_categorical3 <- vector("list",10000)
names(dat_categorical3) <- c(as.character(dat[dat$label=="A",]$samples),as.character(dat[dat$label=="B",]$samples))
for(i in 1:9800){
  dat_categorical3[[i]] <- (as.numeric(table(sample(pool_A, 20, replace=T))) > 0) + 0L
}
for(i in 9801:10000){
  dat_categorical3[[i]] <- (as.numeric(table(sample(pool_B, 20, replace=T))) > 0) + 0L
}
dat_categorical3_tensor <- do.call(rbind,dat_categorical3)

## Split data for training-validating-testing

# As we have much fewer Bs than As, each partition has to have a more or less equal number of Bs(?)
training_Bs <- sample(x=as.character(dat[dat$label=="B",]$samples), size=67, replace=F)
validating_Bs <- sample(x=setdiff(as.character(dat[dat$label=="B",]$samples),training_Bs), size=67, replace=F)
testing_Bs <- setdiff(as.character(dat[dat$label=="B",]$samples),c(training_Bs,validating_Bs))

# We also parition the data containing As equally, though this could probably be done in e.g., 80-10-10
training_As <- sample(x=as.character(dat[dat$label=="A",]$samples), size=3267, replace=F)
validating_As <- sample(x=setdiff(as.character(dat[dat$label=="A",]$samples),training_As), size=3267, replace=F)
testing_As <- setdiff(as.character(dat[dat$label=="A",]$samples),c(training_As,validating_As))

# Put together
training_dat <- dat[dat$samples %in% c(training_As,training_Bs),]
training_categorical3_tensor <- dat_categorical3_tensor[rownames(dat_categorical3_tensor) %in% c(training_As,training_Bs),]
training_labels <- ifelse(training_dat$label=="A",0,1)
training_dat2 <- as.matrix(training_dat[,c(3:5)]) #use this

validating_dat <- dat[dat$samples %in% c(validating_As,validating_Bs),]
validating_categorical3_tensor <- dat_categorical3_tensor[rownames(dat_categorical3_tensor) %in% c(validating_As,validating_Bs),]
validating_labels <- ifelse(validating_dat$label=="A",0,1)
validating_dat2 <- as.matrix(validating_dat[,c(3:5)]) #use this

testing_dat <- dat[dat$samples %in% c(testing_As,testing_Bs),]
testing_categorical3_tensor <- dat_categorical3_tensor[rownames(dat_categorical3_tensor) %in% c(testing_As,testing_Bs),]
testing_labels <- ifelse(testing_dat$label=="A",0,1)
testing_dat2 <- as.matrix(testing_dat[,c(3:5)]) #use this

## Keras model

# Input layers
all_dat_input <- layer_input(shape = 3, dtype = 'float32', name = 'all_dat_input')
categorical3_indices_input <- layer_input(shape = 20, dtype = 'float32', name = 'categorical3_input')
input_tensor <- c(all_dat_input, categorical3_indices_input)

# Output layers
all_dat_out <- all_dat_input %>%
  layer_dense(units = 128, activation = 'relu') %>%
  layer_dropout(rate = 0.5) %>%
  layer_dense(units = 128, activation = 'relu') %>%
  layer_dropout(rate = 0.5)

categorical3_indices_out <- categorical3_indices_input %>%
  layer_dense(units = 128, activation = 'relu') %>%
  layer_dropout(rate = 0.5) %>%
  layer_dense(units = 128, activation = 'relu') %>%
  layer_dropout(rate = 0.5)

output_tensor <- layer_concatenate(c(all_dat_out, categorical3_indices_out)) %>%
  layer_dense(units = 128, activation = 'relu') %>%
  layer_dropout(rate = 0.5) %>%
  layer_dense(units = 128, activation = 'relu') %>%
  layer_dropout(rate = 0.5) %>%
  layer_dense(units=1, activation="sigmoid")

model <- keras_model(inputs=input_tensor, outputs=output_tensor)

# Compile
model %>% compile(
  optimizer = "rmsprop",
  loss = "binary_crossentropy",
  metrics = "accuracy"
)

# Fit
history <- model %>% fit(
  x=list(training_dat2,training_categorical3_tensor),
  y=training_labels,
  batch_size=256,
  epochs=20,
  validation_data=list(list(validating_dat2,validating_categorical3_tensor),validating_labels)
)

# Compare with put aside testing data
results <- model %>% evaluate(list(testing_dat2,testing_categorical3_tensor),testing_labels)

results
$loss
[1] 0.0000001584833
$acc
[1] 1

# Using the trained network to generate predictions on new data
predictions <- model %>% predict(list(testing_dat2,testing_categorical3_tensor))

head(predictions[3267:3332])
[1] 0.9999921 0.9999992 0.9999999 0.9999996 0.9999999 0.9999542
# the network correctly identifies Bs as Bs with a confidence between >99%

(Remember that this is dummy data!)

I'm currently looking into feature importance to see which data inputs contributed more than others to the prediction. lime seems to be made for this!


Why activation functions used in neural networks generally have limited range? Why do we generally use activation functions with only limited range in neural networks? for e.g.

$sigmoid$ activation function has range $[0, 1]$
$tanh$ activation function has range $[-1, 1]$

Q1) Suppose I use some other non-linear activation function like $f(x)=x^2$, that don't have any such limited range then what can be potential problems in training such a neural network?

Q2) Suppose I use $f(x)=x^2$ as activation function in neural network and I am Normalizing the layers (to avoid values keep multiplying to higher values) then would such a Neural Network work? (This is again in reference to the question I posted in heading that "Why do we generally use activation functions with only limited range in neural networks?")

The main goal of an Activation function is to add non-linearity but at the same time

It must not explode the large inputs else time to reach minima will become very long
It should have a smooth gradient, this will help the training keep going towards minima without being stuck

With your example function -

We will achieve non-linearity but gradient will be very large for big value and it will slow down the training

I believe you guessed this, so you asked the next question. What if I normalize the data -

Normalization helps in bringing two different features at the same level. But the larger value of the same feature will still be larger compared to the smaller one

But such function stops learning at both the extreme due to flattening gradient. Also, they require high computation
We have RELU without an upper limit. It's a good trade-off on all the mentioned points


Computationally Inexpensive Imputation Techniques in R I have a large data-frame (155257 x 21 to be specific) with only a few missing values. Say, some 2.16% of the values need to be imputed. The values are floating point numbers.

I'd like to use a method that is much faster than it is accurate, because of the size of the data-set and the fact that I don't have much to lose in a speed-accuracy tradeoff.

Running missForest() takes several hours while Hmisc's impute() function gives unsatisfactory results.

What functions in R might be useful in such (or similar) case?

Take a look at the h20 package https://cran.r-project.org/web/packages/h2o/h2o.pdf.

Everything is designed with parallelization in mind. I've had great success with many of their implementations, in R and Scala.

If you have to do it in R and are going for pure speed I doubt you'll find something faster.


Are there any methods for finding the value of variable which has significant influence on response? I have a dataset which has 5 variables and 1 response. The variables are discrete. I want to find the key variable and its value which leads to a significant increase or decrease to the response.

It is therefore a problem of detecting the importance of the variables.
You have many solutions based on the analysis of a regression model, Stepwise regression algorithms (backwards, forwards), random forest...

However, the simplest solution is probably to use a decision tree: the results obtained are presented under a graphical form that is simple to interpret, and is an effective aid to decision support.

Once the graph is obtained, we can already have an idea of the importance of the variables by looking at the first nodes (it often happens that we use the conditions of these first nodes to create synthetic features that we reinject in a logistic regression).
Then to obtain more precise results, we can compute the total reduction of the criterion brought by that feature in the splits.

In python, with scitkit-learn, you could use the feature_importances_ attribute of a  DecisionTreeClassifier to get the details of the importance of the variables:

import matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeClassifier, plot_tree

# init the classifier
clf = DecisionTreeClassifier()

# don't forget that you need to one-hot encode categorical
# variables before you fit a tree with sklearn!
clf.fit(X, Y)

# plot the graph
plot_tree(clf, filled=True)
plt.show()

print(clf.feature_importances_)

Explanation : The objective of each split is to find the division, or more precisely the variable and the division rule, which will contribute to the strongest decrease of the heterogeneity of the son nodes on the left $\kappa_l$ and on the right $\kappa_r$.
In the case where Y is a qualitative variable, several heterogeneity functions can be defined for a node: a criterion defined from the notion of Entropy or from the Gini concentration (actually there is also the CHAID criterion which is based on the statistical test of $\chi^2$).
In practice, it turns out that the choice of the criterion has little influence, and it is often Gini that is chosen by default.

In the node $\kappa$, where $p_\kappa^l$ is the number of element of the class $l$ in the node $\kappa$, and $m$ the number of classes.

\begin{align*}
\textit{Entropy} :
     &S_\kappa = - \sum_{l=1}^{m} p_\kappa^l log(p_\kappa^l) \\
\textit{Gini} :
    &G_\kappa = \sum_{l=1}^{m}p_\kappa^l(1 - p_\kappa^l) = 1 -
                \sum_{l=1}^{m}(p_{\kappa}^{l})^2
\end{align*}


Validation loss increases and validation accuracy decreases I have an issue with my model. I'm trying to use the most basic Conv1D model to analyze review data and output a rating of 1-5 class, therefore the loss is categorical_crossentropy. Model structure is as below
# define model
model = Sequential()
model.add(Embedding(vocab_size, 100, input_length=max_length))
model.add(Conv1D(filters=32, kernel_size=8, activation='relu'))
model.add(MaxPooling1D(pool_size=2))
model.add(Flatten())
model.add(Dense(10, activation='relu'))
model.add(Dense(5, activation='softmax'))

# compile network
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

# fit network
model.fit(final_X_train, final_Y_train, epochs=5, batch_size=50, validation_data=(final_X_val, final_Y_val), callbacks=callback)

Total params: 14,977,717
Trainable params: 14,977,717
Non-trainable params: 0
Train on 212135 samples, validate on 69472 samples
Epoch 1/5
loss: 0.9452 - acc: 0.5781 - val_loss: 0.8440 - val_acc: 0.6309

Epoch 2/5
loss: 0.7707 - acc: 0.6711 - val_loss: 0.8234 - val_acc: 0.6433

Epoch 3/5
loss: 0.5807 - acc: 0.7657 - val_loss: 0.9144 - val_acc: 0.6345

Epoch 4/5
loss: 0.3736 - acc: 0.8575 - val_loss: 1.1982 - val_acc: 0.6194

Epoch 5/5
loss: 0.2285 - acc: 0.9173 - val_loss: 1.5770 - val_acc: 0.6073

Training acc increases and loss decreases as expected. But validation loss and validation acc decrease straight after the 2nd epoch itself. The overall testing after training gives an accuracy around 60s.
The total accuracy is :  0.6046845041714888

I've already cleaned, shuffled, down-sampled (all classes have 42427 number of data samples) and split the data properly to training(70%) / validation(10%) / testing(20%).

If you see any improvements to fix this problem, please let me know. :)

What you are experiencing is known as overfitting, and it’s a common problem in machine learning and data science.
Overfitting happens when a model begins to focus on the noise in the training data set and extracts features based on it. This helps the model to improve its performance on the training set but hurts its ability to generalize so the accuracy on the validation set decreases.
To deal with overfitting, you need to use regularization during the training. You can try adding dropout layers or batch-normalization layers, adding weight regularization, or you can artificially increase the size of your training set by performing some data augmentation.
You can read more about it in the following post:
What are the possible approaches to fixing Overfitting on a CNN?


Merge two models - Keras I was reading through many blogs and understood the relevance and scenario of having merging two model. But, how to decide the Merge mode for two models. ie. concat, sum, dot, etc.

For eg. I am working on the task of Auto Image captioning. So, captions and Images are 2 kinds of input that I need to handle and merge them at certain point for model to let know which caption is for which image. I am learning the text representation and Image representation by 2 different network designs. Now, at the state after learning representation for both the inputs, how do i decide what ways(concat, add, etc) to use to join/merge two representations.

Generally, I'd say its the same way as finding your "best" feature list, pre-processing methods and algorithm - you'd find most answers starting with "give it a try and compare", "check which combination gives you a better cross-validation?" etc..

More specifically, when there is no farther documentation (and its an open source project) I usually take a look at the code for more intuitions.

In this case concat, dot, average and the like where strait forward implemented as it sounds (concatenating date, averaging, etc..), but it does imply about the amount of processing needed: some of the merging functions adds to the input needed to be processed (concat, add) and some reduces it like an aggregation function (average, min, max)

Beyond that the resulted tensor object will perform a bit different for different datasets. Please give it a try and let us know what merging combination worked best for you, and the dataset type :)


What common/simple problem would work well as a web app? Context
I'm currently writing a simple tutorial to demonstrate a tool to data scientists and analysts that turns Jupyter Notebooks into web apps. Basically, it discusses setting up the web app as a front end, running some code in the notebook and then returning data to the web app.
Question
My question is, what is a small interesting problem in data science that I could solve in the notebook?
I'm looking for something more interesting than doubling an input but smaller/simpler than building a computer vision model.
Additional information
As you can probably tell, I am new to data science. Apologies if this is the wrong forum for this type of question.
Here is the version with a non-interesting problem being solved in the notebook, it may provide more context if needed.
Thanks in advance for any help.

Maybe you could try solving easy classification problems like with Iris Dataset or Titanic Dataset. You'll find many tutorials dealing with those subjects, and they are basic and famous exercices for someone starting in Data Science.


Chinese restaurant process vs Dirichlet Process On Wikipedia Dirichlet Process page, regarding the connection between the Chinese restaurant process and the Dirichlet process it's state the following

  If one associates draws from the base measure
  H with every table, the resulting distribution over the sample space S is a random sample of a Dirichlet process.

What does it mean to:

  Associate draws from the base measure H with every table?

It doesn't make any sense to me.

A sample from a Dirichlet Process, or DP, is a distribution over a sample space $S$. Here the DP is defined based on a base distribution $H$ over $S$. For instance, in the Wikipedia example from your page, the sample space is all real numbers $\mathbb{R}$ and the base distribution is the standard Normal.

The Chinese restaurant process, or CRP, defines a partition over integers $1,2,...,n$ at each time $n$, and $n$ can go to $\infty$. In this metaphor each block in the partition is called a table. Notice that the CRP itself has nothing to do with the original sample space $S$ or the base distribution $H$.

To associate draws from the base measure H with every table, it means for each table $i$, you independently draw one sample $s_i$ from $S$ according to the base distribution $H$. You repeat the sample $s_i$ for $b_i$ times, where $b_i$ is the number of "customers" seated in the $i$'th table in that Chinese restaurant. The distribution of all those repeated samples for all tables, is your sample distribution from the DP.


Where can I get a comprehensive criminal dataset? I want to create a machine learning model to predict the probability of Person committing crime in future. Given his/her past crime history, age, gender, race, employment history, family information (Married, Having young kids, etc.), financial status etc.

Please suggest some data sources having such information to train the model.

Take a look at the Bureau of Justice Statistics site. Other sites that you might find interesting would be U.S. City Census Crimes Data. Of course, Knoema, will provide more of an international flare that you might find useful.


distribution difference between image and text Once for the task of image captioning I've read that, the features extracted from image and text by deep networks are from two different worlds and got different distribution. My question is how is the distribution in two of them and how are they different?

Suppose you trained two identical neural nets on different datasets.  Network A is trained using a dataset of cat pictures.  Network B is trained using a dataset of traffic sign images.  Because the two networks are identical, they will obviously produce a feature map in the same space, right?  But the distribution of features in that space will be different for the two networks, because they were trained on different datasets, and you need different feature extractors to recognize cats vs. traffic signs.
This is analogous to what you have read about the text/image features.  Suppose we train a network to embed image data in some N-dimensional space, and then we train another network to embed text data in the same N-dimensional space.  Although the resulting feature vectors are in the same space, they are almost certain to have different distributions, because they were trained using different datasets.
Unfortunately, we cannot give a general answer about how the distributions are shaped and what exactly the differences are.  These details will vary from case to case.  Although we may not know exactly how the distributions are different until we have them in hand, we can be confident that they are in fact going to have substantial differences.


What could be the problem leading to the result that a model can never perfectly overfit? I tried to fit my model on a small batch of 128 samples for binary classification. The model should be powerful enough as it has hundreds of thousands of parameters. It should be able to overfit to 100% accuracy. However, it only fts to 96% for the best. It is about the same as when I train it on 30,000 samples. So, I tried the following but all failed:
use a smaller batch of 16 samples, it still cannot overfit
use different optimizers, including Adam, SGD, Adagrad, even reset the optimizer every 1,000 epochs, not working
every epoch, only train on the samples that are misclassified, not working.
The problem should be with this network since another more basic neural network can 100% fit. This one can only 99.2% fit. The top layer is indeed sigmoid.
Anyone got any idea what could be the problem?

I would guess there are a few samples that aren't necessarily from the same distribution as the rest. I would try identifying outliers and removing them.


Replace values in dataframe with another dataframes values based on condition I have 2 dataframes with the same columns names and rows number but the elements' values are different. I need to change the value of each element in the first dataframe to 1 if its value in the the other dataframe is 1. I am stuck on this task. Any help pls?

The easiest way to do it for your case is something like this:

In [1]: import pandas as pd

In [2]: df1 = pd.DataFrame([0, 1, 2, 3, 1, 2, 3, 1, 2], columns=["A"])

In [3]: df2 = pd.DataFrame([9] * len(df1), columns=["A"])

In [4]: df1
Out[4]:
   A
0  0
1  1
2  2
3  3
4  1
5  2
6  3
7  1
8  2

In [5]: df2
Out[5]:
   A
0  9
1  9
2  9
3  9
4  9
5  9
6  9
7  9
8  9

Wherever the row of column A in df1 is equal to 1, insert 1 into the same row of column A in df2
In [6]: df2.A[df1.A == 1] = 1

In [7]: df2
Out[7]:
   A
0  9
1  1
2  9
3  9
4  1
5  9
6  9
7  1
8  9

You could also use the pd.DataFrame.where method, to replace the part df1.A == 1.
There are also many other ways to get the same thing done, but my example is the most straighforward :)


Choosing the right parameters for SARSA and Q-Learning & Comparing Models What is the correct way to fine-tune a model's (SARSA(0), SARSA(lambda), Q(0), Q(lambda)) parameters, and how can one compare the models?

I read that typically one compares the number of actions needed to achieve the goal, after X episodes. However, my model is slightly different than maze/grid worlds, so I'm not sure on how to do it.

Essentially its a time model, with X-days, and each day an action is selected, the reward on each day depends on the action selected, and the end-state reward depends on the actions selected in the other 249 days. It is an optimization model for a regulatory situation, which implies revising the whole reward every 250 days, based on the performance (which is tied to actions) of the other 249 days.

With that being said, I cannot measure the performance based on the number of actions needed to reach the terminal state, as it will always be reached with the same number of actions.

The models are being trained in a simulation, the optimal policy will be achieved once per model. The state-space is quite large actually, which is one of my concerns about how many epochs (episodes) I should have, its a 250*8*7*3000 space-action (3-state combination, 250,8,7 and 3000 possible actions per state).

Ideally one would compare all possible models - as there are quite a few - with a smaller number of episodes, and the selected ones would then run with the full amount of episodes, for time-reasons.

Any suggestions on how to compare the models? Based on average rewards with the optimal policy's actions?

As you are building policies in simulation, and can avoid the need to use approximate methods (the state space is small enough to fit in a table in memory), then your goal is to converge on the optimal policy. With correct setup, all the methods you are comparing are guaranteed to converge for your problem, provided the assumptions of reinforcement learning are not broken.

  With that being said, I cannot measure the performance based on the number of actions needed to reach the terminal state, as it will always be reached with the same number of actions.

This is just the reward scheme for maze solvers. For those the goal is to solve the maze quickly.

You have a reward scheme. The best policy maximises the expected reward. That is the only measurement that a completed RL output (your policy) cares about. As all your chosen learners should learn the optimal policy, you cannot compare them on this matter. So you maybe need to look at efficiency - how quickly the algorithm converges to optimal policy.

  The state-space is quite large actually, which is one of my concerns about how many epochs (episodes) I should have, its a 250*8*7*3000 space-action (3-state combination, 250,8,7 and 3000 possible actions per state).

I would characterise this as a small state space. It will fit into memory as a table. Looping through the 40 million state/action pairs should be pretty fast unless the simulation step is expensive for some reason.

In fact, if you have a simple model for transitions and rewards at each step, I would encourage you to use Value Iteration (a dynamic programming method) instead of the sampling based approaches in SARSA or Q-Learning. To do so, you would need to be able to define code for state transitions and rewards given current state and action. Sometimes though, the simulation is much easier to code than the transitions/rewards model. If that's the case for you, stick with SARSA or Q-Learning.

As there are no consequences to you for bad decisions and low rewards during training stages - learning offline in simulations - then Q-Learning may be preferable as it learns the optimal policy whilst exploring. Compared to  SARSA you have to be concerned about how to reduce $\epsilon$ so as to converge on the optimal policy. For Q-Learning you can leave $epsilon$ at a relatively high value (e.g. 0.1) and still learn whilst refining estimates of alternative actions.

However, I am a little concerned about this:

  the end-state reward depends on the actions selected in the other 249 days

This is a problem, since it breaks the assumptions of the Markov Decision Process that all the algorithms are based upon. The reward at any step should only be based on current state/action. By doing this, you may invalidate the policy. It definitely would defeat dynamic programming (may prevent it converging) unless you included action history in the state (which would definitely make your state space large). However, it may still work for sampling methods with a high value of $\lambda$, because these are more robust when faced with environments which are not strictly MDPs.

  Any suggestions on how to compare the models?

I suggest on CPU time required to learn the optimal model, assuming you intend to repeat this process on different data routinely. The way to find out is to experiment, as convergence will depend on properties of the problem, how fast your simulation code runs, and in your case you are stretching one of the assumptions behind the design of the RL solvers which could make strongly bootstrapping methods (like SARSA(0)) either fail to converge or converge very slowly.

If convergence takes too long, set a cutoff at some time limit (e.g. an hour) and find out expected reward from initial conditions when following the policy learned so far. You could Monte Carlo sample that for a fair comparison (and if results are stochastic even with a deterministic policy, then give plenty of time for 1000s of runs to get a good estimate).


Amplifying a Locality Sensitive Hash I'm trying to build a cosine locality sensitive hash so I can find candidate similar pairs of items without having to compare every possible pair. I have it basically working, but most of the pairs in my data seem to have cosine similarity in the -0.2 to +0.2 range so I'm trying to dice it quite finely and pick things with cosine similarity 0.1 and above.

I've been reading Mining Massive Datasets chapter 3. This talks about increasing the accuracy of candidate pair selection by Amplifying a Locality-Sensitive Family. I think I just about understand the mathematical explanation, but I'm struggling to see how I implement this practically.

What I have so far is as follows

I have say 1000 movies each with ratings from some selection of 1M users. Each movie is represented by a sparse vector of user scores (row number = user ID, value = user's score)
I build N random vectors. The vector length matches the length of the movie vectors (i.e. the number of users). The vector values are +1 or -1. I actually encode these vectors as binary to save space, with +1 mapped to 1 and -1 mapped to 0
I build sketch vectors for each movie by taking the dot product of the movie and each of the N random vectors (or rather, if I create a matrix R by laying the N random vectors horizontally and layering them on top of each other then the sketch for movie m is R*m), then taking the sign of each element in the resulting vector, so I end with a sketch vector for each movie of +1s and -1s, which again I encode as binary. Each vector is length N bits.
Next I look for similar sketches by doing the following

I split the sketch vector into b bands of r bits
Each band of r bits is a number. I combine that number with the band number and add the movie to a hash bucket under that number. Each movie can be added to more than one bucket.
I then look in each bucket. Any movies that are in the same bucket are candidate pairs.

Comparing this to 3.6.3 of mmds, my AND step is when I look at bands of r bits - a pair of movies pass the AND step if the r bits have the same value. My OR step happens in the buckets: movies are candidate pairs if they are both in any of the buckets.

The book suggests I can "amplify" my results by adding more AND and OR steps, but I'm at a loss for how to do this practically as the explanation of the construction process for further layers is in terms of checking pairwise equality rather than coming up with bucket numbers.

Can anyone help me understand how to do this?

I think I've worked something out. Basically I'm looking for an approach that works in a map/reduce type environment and I think this approach does it.

So,

suppose I have b bands of r rows and I want to add another AND stage, say another c ANDs.
so instead of b * r bits I need hashes of b * r * c bits
and I run my previous procedure c times, each time on b * r bits
If x and y are found to be a candidate pair by any of these procedures it emits a key value pair ((x, y), 1), with the tuple of IDs (x,y) as the key and the value 1
At the end of the c procedures I group these pairs by key and sum
Any pair (x,y) with a sum equal to c was a candidate pair in each of the c rounds, and so is a candidate pair of the entire procedure.

So now I have a workable solution, and all I need to do is work out whether using 3 steps like this will actually help me get a better result with fewer overall hash bits or better overall performance...


K-Nearest neighbor in transformed space When googling "weighted KNN", the results appear to be focused on weighting the nearest neighbor values after those neighbors have been determined.  I'm looking for something that assigns a level of importance to various dimensions that could possibly change the neighbors that are considered nearest with the goal of maximizing the accuracy of predictions made using the resulting model.

For example, if I have a new observation defined as [1, 2], #1 and #2 below would be considered the nearest neighbors assuming K=2 under a normal KNN (as I understand it):

[1.5, 2.5] d=0.707107
[1.8, 1.5] d=0.943398
[4.0, 2.2] d=3.006659

However, if the dimensions were weighted by importance, such as [0.1, 1], the items above would have respective distances of 0.502494, 0.50636, 0.360555 based on sqrt((d1*w1)^2 + (d2*w2)^2), which would make #3 the overall nearest.

Given the goal of maximizing prediction accuracy, I'm wondering what methods are available to accomplish this?  I'd like to potentially use this for both classification and regression - are there methods other than KNN that I should consider?

You can modify the matrix used to compute the distances between points. By default, KNN with quantitative features is used with Euclidean distances, this is, the identity matrix in the equation (x-y)' I (x-y) (it returns the squared Euclidean distance between point X and Y).

Instead of the identity matrix, you can use other positive definite matrices and compute Mahalanobis distances.

Other modifications you can do is manipulate each feature. This also affects the output of KNN, aside of standardizing, you can try non linear modifications of the features, reducing kurtosis or skewness, for example.


High probabilities of success for wrong predictions I'm studying the behavior of machine failures in a production scenario. For this, I generated random data to form my imbalanced training set, consisting of categorical data, which indicate whether or not there was a failure in each subperiod. The failures were generated according to a exponential distribution. I have 24 features (Period_1 to Period_24), each containing information about the historical failures for 448 subperiods. Furthermore, I have three more features consisting of Temperature, Moisture, and Pressure (generated using the Normal distribution). My intention is to predict the behavior of the failures for the next period based on these features.
I used the ROC metric and considered several strategies to deal with unbalanced data, such as oversampling, undersampling, ROSE, and ADASYN. Furthermore, I tried to use ensemble to improve performance. I tested all of the following models: gradient boosting algorithm, random forest, Classification and Regression Trees, neural networks, Bagged CART, SVM, C5.0, eXtreme Gradient Boosting, and k-Nearest Neighbors. I also tried to use regularized models but none of these strategies worked. The best result obtained was using the model "SVMRadial" considering resampling with the ROSE package. In this case, ROC = 0.7614, Sensitivity = 0.7639, and Specificity = 0.6065 for the training set and Sensitivity = 0.75, and Specificity = 0.6914 for the test set (the latter obtained through the Confusion Matrix). However, when making predictions, the trained model is resulting in high probabilities for wrong predictions. So, I would like to know if this is a problem of the training model. Also, would anyone have any idea how to improve these results?
Any help will be appreciated.
A sample of the data:

The code for the chosen model:
set.seed(123);

partition <- createDataPartition(data_failures$Period_24, p = 0.8, list = F)
trainingSet <- data_failures[partition,]
testingSet <- data_failures[-partition,]

train.control <- trainControl(method = "repeatedcv", number = 10, repeats = 3, sampling = "rose", classProbs = TRUE, summaryFunction = twoClassSummary)

model_24 <- train(Period_24 ~., data = trainingSet, method = "svmRadial", preProc = "zv", metric = "ROC", trControl = train.control)

print(model_24)

predictions <- predict(model_24, newdata = testingSet)
print(confusionMatrix(predictions, testingSet$Period_24))

I'm using R considering the caret package.

It is difficult to diagnose without having more info. But based upon what you mention I point:

Since you have a class imbalance, you MUST use regularizarided models that penalize the cost function  made by the minority class, more strictly than those errors for the majority class (Boosting models, Linear models)

Try using another metric like ROC AUC or Balanced Accuracy to measure the performance of your model so that you can see an adequate point of view of how your model is performing.
An additional advantage of using ROC Score is that you can find the "optimal" threshold being the one that maximizes Area under the roc curve when you make predictions with your model

This is the most important part. Use a technique to debug your model like SHAP values,Partial dependence plot in order to understand how your model is learning and the relationships between the features and the target

If you are using ensemble methods, you can set something named Monotonicity constraints That will allow you to add business knowledge to the model in order to correct potential non-sense relationships that the model may find


Finding the duplicate values between all columns and sort in new column with Pandas? I have this DataFrame:
    CL1 CL2 CL3 CL4
0   a   a   b   f
1   b   y   c   d
2   c   x   d   s
3   x   s   x   a
4   s   dx  s   s
5   a   c   d   d
6   s   dx  f   d
7   d   dc  g   g
8   f   x   s   t
9   c   x   a   d
10  x   y   y   a
11  c   a   x   y
12  f   s   d   s
13  d   d   w   a

Intention:

With help Pandas

1- I wanna search and find the similar values between all columns (CL1-CL4) and sort in a new column (SIM).
2- I wanna find the non-similar values between columns and sort in another column (NON-SIM).
What i want

How can i do that? With df.pivot_table i was not successful.

Given your input data is saved in a variable df, I count the values which occur in all 4 unique columns as follows:
import pandas as pd
import numpy as np

output = (
    df
    .melt()
    .drop_duplicates()
    .groupby("value")
    .agg(count=("value", "count"))
    .reset_index()
)
output["SIM"] = np.where(output["count"] == 4, "SIM", "NON-SIM")
output = output.pivot(columns="similarity", values="value")

print(output)

similarity  NON-SIM SIM
0           NaN     a
1           b       NaN
2           c       NaN
3           NaN     d
4           dc      NaN
5           dx      NaN
6           f       NaN
7           g       NaN
8           NaN     s
9           t       NaN
10          w       NaN
11          x       NaN
12          y       NaN


choosing model based on last or best iteration on validation set This is a very basic question, however I haven't found a satisfying answer until now.
When training a neural network we must choose the number of epochs. The usual advice is to train as long as the loss on the validation set goes down. But it happens that the validation loss goes down, then a little bit up, then down again. So for me the most reliable way to find the point to stop training and get the best model would be to take a very high number of epochs and just choose the best validation loss out there (be it the last iteration or, more probable, something in between). One drawback is of course the long training time due to high number of epochs.
To avoid long training time and still get a good validation score, people use "early stopping" techniques (as described in the often cited paper Early Stopping — But When? from Lutz Prechelt). In the paper mentioned, the author points out that indeed the validation curve can go up and down and you can never be sure that you reached some kind of absolute minimum. In consequence, if you use "early stopping" methods, you introduce some kind of bias in your model training.
That is why I thought to train as many epochs as I can afford and choose the best epoch to sotp training afterwards. I thought this would be less biased and the most "correct" way to choose the number of epochs for training.
In stackexchange I found the following related questions:

Present results of the best or the last iteration on dev set? where the answer is, that we should choose the best iteration and not the last one.

Reporting test result for cross-validation with Neural Network where the answer is that if is not correct to choose the best iteration since this introduces also a bias. Instead, one should opt for a apriori fixed number of epochs and choose the last iteration.

If I put the theory for a moment aside, I think it is unwise to stop training too soon if the validation loss still goes down. Also it is unwise to choose a validation loss that is higher because I trained until overfitting. So clearly, I am inclined to take some minimum.

To sum it up, at the bottom is a plot of my validation loss of a 5-Fold-CV. Using early stopping could have stopped training at those strange plateaus. Using the best loss score seems also a bit "biased" in those highly jittering regions.

What is the correct way to stop training here??

Too much noise in the curve may indicate a too-high learning rate. You could try reducing the learning rate with the number of epochs, when the curve becomes flat.

You can also use early stopping with a "window". That means, stop training if the loss is not getting better after N epochs. Where getting better means a % or a fixed number at your choice too.

I would use the best epoch. Using the last epoch sounds weird to me and could introduce some bias too if you wait to see some initial results before choosing a maximum number of epochs (and you are probably going to do that).

Also, you may want to spend some time trying to improve your training set if you have not done so already. Maybe you can find more data or augment it.

If you are unable to overfit, you may as well want to try with a more complex model. Regularization/dropout could help with your leaning curves too.


What are the output shape of LSTM with "return_sequences" equal to "True" or "False"? What are the output shape of LSTM with Keras implementation when "return_sequences" equals to "True" or "False" ?

Assuming your output state size is h. You will generate nxh for n data points if return_sequences is false. On the other hand you would be having nxtxh dimensional array as output if return_sequences is set to true. Here t is the number of timestamps in each data point.


How to implement PCA color augmentation as discussed in AlexNet I read through "ImageNet Classification with Deep Convolutional Neural Networks" again specifically for details on how to implement PCA color augmentation.
I am unsure if I have it right. Here is how I did this in numpy:

# original image is 224x224x3 of dtype uint8

renorm_image = np.reshape(original_image,(original_image.shape[0]*original_image.shape[1],3))

renorm_image = renorm_image.astype('float32')
renorm_image -= np.mean(renorm_image, axis=0)
renorm_image /= np.std(renorm_image, axis=0)

cov = np.cov(renorm_image, rowvar=False)

lambdas, p = np.linalg.eig(cov)
alphas = np.random.normal(0, 0.1, 3)

delta = np.dot(p, alphas*lambdas)

delta = (delta*255.).astype('int8')

pca_color_image = np.maximum(np.minimum(original_image + delta, 255), 0).astype('uint8')

One serious doubt is the line "delta = (delta*255.)". I have to do this to rescale things such that the numbers make sense. I hope someone can give me feedback if this is right.

You should not apply *255.

delta was supposed to be added to renorm_image, because you calculated this delta using cov, which was based on renorm_image.

Then how would you restore renorm_image to your original image? *std + mean or *255?

Obviously you should apply *std + mean.

Therefore,

delta = (delta*255.).astype('int8')
pca_color_image = np.maximum(np.minimum(original_image + delta, 255), 0).astype('uint8')

should be changed to:

mean = np.mean(renorm_image, axis=0)
std = np.std(renorm_image, axis=0)
pca_augmentation_version_renorm_image = renorm_image + delta
pca_color_image = pca_augmentation_version_renorm_image * std + mean
pca_color_image = np.maximum(np.minimum(pca_color_image, 255), 0).astype('uint8')


How to improve accuracy in the following code? I have the about 43 different categories of traffic signs images data. If I am using the small data of 3 categories the maximum accuracy I am getting is around 65% and I have tried a lot of different layer values as well as losses and optimizers. When I am using the complete dataset I am only getting accuracy of about 5%. Please give some pointers on how to improve my accuracy. Please tell me should I add more layers and if somebody has any good example somewhere, please provide the link.

import cv2
import numpy as np
import os
import sys
import tensorflow as tf
import glob

from sklearn.model_selection import train_test_split

EPOCHS = 10
IMG_WIDTH = 30
IMG_HEIGHT = 30
NUM_CATEGORIES = 10
TEST_SIZE = 0.4
path = []
data = []
label = []

def main():

    # Check command-line arguments
    if len(sys.argv) not in [2, 3]:
        sys.exit("Usage: python traffic.py data_directory [model.h5]")

    # Get image arrays and labels for all image files
    images, labels = load_data(sys.argv[1])

    # Split data into training and testing sets
    labels = tf.keras.utils.to_categorical(labels)
    x_train, x_test, y_train, y_test = train_test_split(
        np.array(images), np.array(labels), test_size=0.2
    )
    # Get a compiled neural network
    model = get_model()

    # Fit model on training data
    model.fit(x_train, y_train, epochs=EPOCHS)

    # Evaluate neural network performance
    model.evaluate(x_test,  y_test, verbose=2)

    # Save model to file
    if len(sys.argv) == 3:
        filename = sys.argv[2]
        model.save(filename)
        print(f"Model saved to {filename}.")

def load_data(data_dir):
    """
    Load image data from directory `data_dir`.

    Assume `data_dir` has one directory named after each category, numbered
    0 through NUM_CATEGORIES - 1. Inside each category directory will be some
    number of image files.

    Return tuple `(images, labels)`. `images` should be a list of all
    of the images in the data directory, where each image is formatted as a
    numpy ndarray with dimensions IMG_WIDTH x IMG_HEIGHT x 3. `labels` should
    be a list of integer labels, representing the categories for each of the
    corresponding `images`.
    """
    subdirs = [x[0] for x in os.walk('C:\\Users\\LENOVO\\traffic\\gtsrb')]
    for i in range(1,NUM_CATEGORIES+1):
        subdirs[i] = subdirs[i] + "\*.*"
    for i in range(1,NUM_CATEGORIES+1):
        for file in glob.glob(subdirs[i]):
            a = cv2.imread(file)
            a = cv2.resize(a,(IMG_WIDTH,IMG_HEIGHT))
            data.append(a)
            label.append(i-1)

    return(data,label)

def get_model():
    """
    Returns a compiled convolutional neural network model. Assume that the
    `input_shape` of the first layer is `(IMG_WIDTH, IMG_HEIGHT, 3)`.
    The output layer should have `NUM_CATEGORIES` units, one for each category.
    """
    model = tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(
        32, (3, 3), activation="relu", input_shape=(30,30,3)
    ),
    tf.keras.layers.MaxPooling2D(pool_size=(2, 2)),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dropout(0.5),

    # Add an output layer with output units for all 10 digits
    tf.keras.layers.Dense(NUM_CATEGORIES, activation="softmax")])
    model.compile(
    optimizer="Nadam",
    loss="categorical_crossentropy",
    metrics=["accuracy"])
    return (model)
    raise NotImplementedError

if __name__ == "__main__":
    main()

Adding more images in the training set is one way to increase accuracy. You can also do transfer learning, i.e. using layers trained by a larger dataset if you are classifying common things such as animals which have networks that have already been trained such as the ResNet50.

When you have enough images that the accuracy does not increase anymore, you can try adding more layers, with convolutional and ReLu layers alternating. If you train for many epochs and the model starts to overfit i.e. training set accuracy still trends up but test set accuracy trends down, you can add layers that reduce overfitting, such as the dropout layer. You should first get the model to overfit and then try reducing overfitting.

If you don't have enough training data you could also do augmentation, e.g. adding rotated, shifted and horizontally flipped images to the training set.

The thing with traffic signs is that if you are classifying images with also the landscape instead of just the cropped traffic signs, you are going to have problems. You should then also detect the traffic signs (YOLO v3 could be used as transfer learning) instead of just classifying them.

Here's also a good example of traffic sign classification using transfer learning on Kaggle: https://www.kaggle.com/valentynsichkar/traffic-signs-classification-with-cnn


convert predict_proba results using class_weight in training As my dataset is unbalanced(class 1: 5%, class 0: 95%) I have used class_weight="balanced" parameter to train a random forest classification model. In this way I  penalize the  misclassification of a rare positive cases.

rf = RandomForestClassifier(max_depth=m, n_estimators=n_estimator,class_weight = "balanced")
rf.fit(X_train, y_train)

The “balanced” mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as n_samples / (n_classes * np.bincount(y))

In my case the classes frequencies are:

fc = len(y_train)/(len(np.unique(y_train))*np.bincount(y_train))

10000/(2*np.array([9500,500]))
array([ 0.52631579, 10.        ])

I apply my model over a test dataset using predict proba function:

y_predicted_proba = rf.predict_proba(X_test)

The second column presents the probabilities of being 1 to the input samples. However I understand this probability must be corrected to be real.

If I divide them by class_weight values these new probabilities don't sum one...

How can this correction be achieved?

The correction that you are talking about is called probability calibration -- you want the "real" probability of an observation being in each class, right?

The two most common approaches to probability calibration are Platt Scaling and Isotonic regression. Since you are training on a balanced training set (which is the right thing to do in your case because the original dataset is unbalanced), you can apply these techniques afterwards on your test set.

Sorry I can't explain the techniques fully here, but hopefully knowing the names of these terms gives you a starting point.


How to read the output of Binary cross entropy? Suppose for a single training example, the true label is [1 0 0 0 0] while the predictions be [0.1 0.5 0.1 0.1 0.2]. How to calculate its binary cross entropy? And how output value decides whether a sample belongs to class a or b? I also want to know that that in the formula of binary cross entropy what is 1-p?

Binary cross-entropy is a simplification of the cross-entropy loss function applied to cases where there are only two output classes. Essentially it can be boiled down to the negative log of the probability associated with your true class label.

Cross entropy

Cross entropy is defined as

$L = -\sum ylog(p)$

where $y$ is the binary class label, 1 if the correct class 0 otherwise. And $p$ is the probability of each class.

Let's look at an example, if for an instance $X$ the output label is 0 and your model output was $[0.7, 0.3]$. Then we can see that the loss function using binary cross entropy is

$L = -(1*log(0.7) + 0*log(0.3)) = 0.155$

Notice how the loss is only affected by the true label's prediction probability.

Binary cross-entrop

Binary cross entropy is for an example with 2 output classes and can be simplified as

$L = (ylog(p) + (1-y)log(1-p)) = (1*log(0.7) + (1-1)log(1-0.7)) = 0.155$

Your example

For your example, $[1,0,0,0,0]$ and predictions $[0.1 0.5 0.1 0.1 0.2]$ the loss is

$L = -(1*log(0.1) + 0*log(0.5) + 0*log(0.1) + 0*log(0.1) + 0*log(0.2)) = 1$


What is the accuracy majority class classifier? I have an SFrame and a model:

train_data,test_data = products.random_split(.8, seed=0)
selected_words_model = graphlab.logistic_classifier.create(train_data,
                                                     target='sentiment',
                                                     features=selected_words,
                                                     validation_set=test_data)

After computing the accuracy of the model with `selected_words_model.evaluate(test_data) I'm asked "What is the accuracy majority class classifier on this task?" Yet I don't even know what this "means accuracy majority class classifier", shouldn't it be "accuracy of the majority class classifier" ?

Here is my attempt.

All these materials come from this coursera ML fundations course exercise.

I suspect you are right that there is a missing "of the," and that the "majority class classifier" is the classifier that predicts the majority class for every input. Such a classifier is useful as a baseline model, and is particularly important when using accuracy as your metric. This matches what your notebook comments in the next bullet, so that's likely what was intended.


Machine learning with partially defined labelling using existential quantifiers I apologize for a vague title, but I know only the basic machine learning jargon. Let's assume that we have domain set $X$ and label set $L = \left\{0, 1\right\}$, but the labelling function $f$ is only partially defined by function $g:P→L$, where:

(1)  $P$ is a partition of $X$

(2)  $\it g(p) = 0 ⟹ \forall x \in p: f(x) = 0$

(3)  $\it g(p) = 1 ⟹ \exists x \in p: f(x) = 1$

The meaning of function $g$ is that for some elements of $P$ all members have label $0$ and for remaining elements of $P$ at least one member has label $1$, but we don't know which is $1$ and which is $0$.

For example: we have boxes of apples with possibly different number of apples in each box. We know which boxes have all good apples (apple with label $0$) and remaining boxes have at least one bad apple (apple with label $1$), but we don't know which ones are bad or good.

Is there a machine learning technique to deal with such partially defined labelling?

EDIT

Let's assume that in boxes of apples example above, a classifier we looking for is: good apples are green and hard, bad apples are brown and soft. As a model of an apple, my lucky guess is a 4-d feature vector with color, hardness, weight and size components. All of the apples are observable, so we can calculate the feature vector for each of them. We know labels for all apples in "good" boxes, but have only an existential statement about labels of apples in "bad" boxes.

After measuring all apples, I look for clusters of feature vectors or run some unsupervised learning algorithm. I will discover 2 distinct classes: (green, hard, _, _) and (brown, soft, _, _). These classes will also satisfy conditions (2) and (3) above, so I will accept (green, hard) vs. (brown, soft) as a correct classifier.

With this procedure in mind, let me rephrase my question. Assuming my guess of feature vector is correct, so in about 10-d vector there are enough significant features for correct classification, the partition set $P$ has 1K elements and the domain set $X$ has 100K elements, what is the correct machine learning technique to use?

It's probably possible.  I'll suggest a few plausibly-practical methods, starting from very crude (but probably not super-effective) to more complex (might be more effective but might be computationally expensive).  I expect that many of these approaches might be relatively slow to train the classifier but doable and worth a try.  But first, let me try to formalize the  problem more precisely.

Theory

In machine learning it's often helpful to formulate your problem by identifying a loss function.  Then you can formulate your problem as finding $f$ that minimizes the loss function.  In your case, I think a natural loss function would be that $L(f,g)$ counts the number of partitions $p$ such that conditions (2) or (3) are violated.  You could of course add a regularization term as well.

Once you've defined a loss function, your problem becomes: given $g$, find $f \in \mathcal{H}$ that minimizes $L(f,g)$, where $\mathcal{H}$ is the hypothesis space (the space of valid/legal models).

Let's start with a theoretical thought experiment.  Suppose you didn't care about the computational time to train a classifier, and just want to find a valid classifier $f$.  Let $\mathcal{H}$ be the hypothesis space, i.e., the set of allowable functions.  (For instance, for a linear SVM, $\mathcal{H}$ would be the set of SVM weights -- i.e., the set of linear decision boundaries.)  If you didn't care about computational time, you could just enumerate all $f$ to find one that minimizes $L(f,g)$ -- so this proves that the problem is well-defined.

More generally, suppose we have a space of classifiers $f$ that output probabilities instead of hard labels (think: logistic regression).  Then we can define a loss function based on the cross-entropy loss:

$$L(f,g) = \sum_{p : g(p)=0} \max \{-\log(1-f(x)) : x \in p\} + \sum_{p : g(p)=1} \min \{-\log(f(x)) : x \in p\}$$

You could add a regularization term here, too.

Of course, other loss functions are possible as well; but the point is that once you have a loss function, then classification becomes well-defined as an optimization problem.

Plausible method: one-class classifier

One crude approach is to use a one-class classifier.  For each partition $p$ such that $g(p)=0$, you obtain many instances $x$ where you want to have $f(x)=0$.  So, you could add each one of those to a training set, and then train a one-class classifier on it.

This is simple and easy to implement.  However, I suspect its accuracy might be low.

Plausible method: active learning

Suppose you have a way to obtain labelled instances if you need, through manual training.  In particular: let's assume that given a partition $p$, if you really needed to, you could construct hard labels for instances $x \in p$, and specifically, if $g(p)=1$, you could find an instance $x \in p$ that should receive the label $1$.  This manual-labelling process might be difficult and tedious and time-consuming/expensive, so not feasible to do on a large scale, but let's suppose you could do it on a small scale if needed.

Then you could use active learning:

Let $f$ be an arbitrary classifier (maybe trained using a one-class classifier).  Let $T$ be an initial training set, initially empty.
Repeat until convergence:

a. Pick a partition $p$ such that $f(x)$ is wrong for some $x \in p$.

b. If $g(p)=0$, add find $x \in p$ such that $f(x)=1$, and add $(x,0)$ to $T$.

c. If $g(p)=1$, use the manual-labelling process to find some $x \in p$ that should be labelled $1$ and add $(x,1)$ to $T$.

d. Train a new boolean classifier $f$ using standard supervised learning methods on the training set $T$.

There are various optimizations you could use to break ties in step 2c (in hopes of reducing the amount of manual labelling); e.g., if you're using the cross-entropy loss defined above, you could choose the $p$ that contributes the most to the total loss for $f$.  Or, you could choose the $p$ that is most "different" from all previously considered $p$ with $g(p)=1$.  Or, you might be able to adapt other active learning methods.

This might work.  However, it does require the ability to do some manual labelling of individual instances, which is more than you promised us in the original problem statement.

Plausible approach: direct optimization

Another possible approach would be to choose a differentiable classifier, and the optimize the (cross-entropy) loss $L(f,g)$ defined above directly using gradient descent.  For instance, you could choose $f$ to be a logistic regression or neural network classifier, as these are differentiable.  Now, you can compute the gradient of $L(f_\theta,g)$ with respect to the parameters $\theta$ of the model (e.g., for logistic regression, $\theta$ are the weights) and then apply gradient descent to find the model parameters $\theta$ that minimize $L(f_\theta,g)$.

This may get a little messy, as loss function $L$ contains $\min$ and $\max$ functions, which have points where they are non-differentiable.  If you don't do anything special, this can cause oscillation around the non-differentiable boundary.  One possible trick is to replace each $\min$ by a softened version of the min, and replace each $\max$ by a softened version of the max, so that the loss function becomes differentiable.

Alternatively, you could adjust ordinary gradient descent to behave well around them.  One standard technique is to reduce the step size when you get near the boundary to avoid stepping across the boundary, or to use projected gradient descent.  Thus, if we have a term

$$\ell(\theta) = \min(\ell_1(\theta),\ell_2(\theta))$$

and we compute the gradient $\nabla \ell(\theta)$, the gradient will be based on whichever of $\ell_1(\theta),\ell_2(\theta)$ is smaller for this particular value of $\theta$.  Suppose the gradient is $g = \nabla \ell(\theta)$.  Ordinary gradient descent would take a step in the direction $g$, say adjusting $\theta$ to $\theta - \lambda \cdot g$ for some small constant $\lambda$.  You could additionally require that the step avoid crossing the boundary line $\ell_1(\theta)=\ell_2(\theta)$.  For instance, if $\ell_1(\theta)<\ell_2(\theta)$ for this particular value of $\theta$, we'll have $g = \nabla \ell(\theta) = \nabla \ell_1(\theta)$.  So, take a step $- \lambda' \cdot g$, but constrain $\lambda'$ to be in the range $0 \le \lambda' \le \lambda$ and to satisfy $\ell_1(\theta - \lambda' \cdot g) \le \ell_2(\theta - \lambda' \cdot g)$ -- i.e., don't step across the other side of the discontinuity.

Note that using ordinary gradient descent will probably be very slow, as each evaluation of the loss function requires evaluating $f$ on every point in $X$: $O(|X|)$ time.  You'll probably do better by using stochastic gradient descent: in each iteration, pick a single partition $p$, compute the gradient of the term in $L(f,g)$ for $p$, and move in the direction of that gradient.  Or, pick a minibatch of 20 partitions $p$ and compute the gradient of the sum of those 20 terms.

Plausible method: boosting

Another reasonable method would be to use boosting with a very simple classifier, such as a decision stump.

A decision stump is a decision tree with a single level: you pick one feature, compare it to some threshold, and make a binary decision based on whether the value of that feature is above or below the threshold.  Decision stumps are convenient because you can enumerate all candidate stumps -- all possible combinations of a feature and a threshold -- and evaluate the loss function for each.  (In practice, as an optimization, we don't enumerate all possible stumps.  Rather, for a particular feature, we randomly pick 100 candidate thresholds by sampling 100 times from the training set, and we compute the loss for each candidate threshold; we do this for each feature, and take the best overall combination.)

You could use AdaBoost directly with decision stumps and minimize the loss function defined above, and you might get a reasonable classifier.

For even better results, you could use gradient boosting.  This is a slight generalization where each leaf of each stump outputs a continuous value rather than a boolean; the classifier's output is obtained by summing the value from each stump, and then checking whether the sum is positive or negative.  The general methodology can be applied with any differentiable loss function, so you could apply that methodology to the cross-entropy loss function defined above.

Plausible method: incremental optimization

Here's another approach that can be applied to classifiers that aren't differentiable.
In my algorithm, I will maintain a tentative training set $T \subseteq X \times \{0,1\}$ that assigns a tentative label to some of the elements of $X$.  The training set $T$ will evolve throughout the algorithm, but it will always remain consistent with $g$, defined as follows:

We say that $T$ is consistent with $g$ if, (a) for every $p$ such that $g(p)=0$, we have $(x,0) \in T$ for all $x \in p$, and (b) for every $p$ such that $g(p)=1$, there exists at least one $x \in p$ such that $(x,1) \in T$.

The training set $T$ specifies one possible labelling that would be consistent with $g$.  It's not necessarily the right one; it's just one possibility.

At each step, the algorithm uses $T$ as labels to train a classifier $f$ using standard supervised learning methods and computes $f$'s loss on $T$, then adjusts $T$ to reduce the loss of $f$ on $T$.  Define $\text{train}(T)$ to be the classifier $f$ obtained by training on the training set $T$, using some boolean classifier of your choice.  Here is the algorithm:

Set $T$ to be any training set that is consistent with $g$.  Compute $f := \text{train}(T)$ and $\ell = L(f,g)$.
Repeat until convergence:

a. Enumerate all training sets $T'$ that differ from $T$ in only one element yet remains consistent with $g$, and for each train a classifier and compute the loss.  Pick the $T'$ that minimizes $L(\text{train}(T'),g)$.

b. Set $T := T'$.

Step 2a can be done by enumerating all partitions $p$ such that $g(p)=1$. Suppose $T$ includes $(x,1)$ where $x \in p$.  Then enumerate all $x' \in p$ and consider $T' = T \setminus \{(x,1)\} \cup \{(x',1)\}$.  There are at most $|X|$ such possibilities, and you have to train the classifier on each, so each iteration of the loop might be pretty slow... but one could hope that it will eventually converge on a reasonable solution.

Basically, this tried to maintain some tentative labels that are consistent with your $g$, and that also are consistent with the regularity conditions imposed by the classifier $f$ (e.g., linear separability, or whatever), incrementally adjusting it at each step of the iteration.

For your particular parameter values ($|X|=10^5$, $|P|=10^3$), I suspect this will be too slow.  But you could try experimenting with the approach for scaled-down versions of your problem, with a smaller feature space ($|X|$ is smaller) and see if it seems to lead to useful results.

If this algorithm gives useful results, there many heuristics and optimizations you could use to speed up convergence:

You could probably make a smarter choice for the initial value of $T$ and speed up convergence, by applying a one-class classifier to the set of 0-labelled instances (since the instances of the form $(x,0) \in T$ never change throughout the algorithm) and using that to choose tentative initial label.
You could use something akin to stochastic gradient descent to speed convergence: in step 2a, pick a single $p$ (or a small minibatch of $p$'s) and  consider just training sets $T'$ that differ from $T$ only in $p$ (but not in other partitions.
You could try using an online/incremental training method for training $f$.  Since $T'$ is so similar to $T$, rather than training a new classifier from scratch, you might be able to reuse much of the effort from training on $T$ to more quickly compute $\text{train}(T')$ given $\text{train}(T)$.
Depending on the nature of the feature space and the partition function, you might be able to solve the following problem more efficiently than brute force: given a training set $T$ and a partition $p$ such that $g(p)=1$ and an $x \in p$ such that $(x,1) \in T$, find some $x' \in p$ such that minimizes the loss when training on $T' = T \setminus \{(x,1)\} \cup \{(x',1)\}$.  For instance, with logistic regression, you might be able to train a classifier $f_0$ on $T \setminus \{(x,1)\}$, then classify each $x' \in p$ using $f_0$ and rank them by the probability score that logistic regression outputs.  This is a heuristic and not guaranteed to find the optimal $x'$, but it's also a lot more efficient than enumerating all candidates for $T'$ and training a classifier on each.

A useful subroutine

Many of the above schemes require solving the following subproblem:

Given a partition $p$ and a classifier $f$, find $x \in p$ such that $f(x)=1$, or report that none exists.

If the classifier outputs confidence/probability scores, then the subproblem is:

Given a partition $p$ and a classifier $f$, find $x \in p$ that maximizes $f(x)$.

The naive way to solve this subproblem is to enumerate all $x \in p$.  Depending on the particular classifier you choose and the structure of the partitions, there may be more efficient algorithms.

For instance, if we have a linear classifier, maximizing $f(x)$ becomes equivalent to maximizing $x \cdot w$ for some known vector $w$.  If $p$ has a simple structure, there might be faster ways to solve this optimization problem.  For instance, if $p$ is a convex region defined by linear inequalities, you can maximize $x \cdot w$ subject to $x \in p$ using linear programming.

And you can use a fast algorithm for this subproblem to speed up the training process, for many of the methods sketched above.  Whether that is possible will depend on the specific type of classifier you choose and the structure of the partitions, but if you're trying to do this in practice, I recommend you ask a separate question about how to solve this subproblem for the particular type of classifier and partition structure that arises in your application -- that might enable significant computational optimizations.


How to find a similarity value between cars So I have a database of web-scraped cars, and I want to find the similarity between cars based on the km driven (e.g 69000), the model year (e.g 2012), and the trim of the car which will be one of three categories: base, mid, top.

What similarity measure can I use that will give me a decently accurate similarity ratio

I suggest you use a weighted per-attribute similarity, for instance let a and b a pair of tuples representing the attributes of some car A and another car B. For example:

a = (69000, 2012, base)
b = (71000, 2013, base)

For the km driven I would use Skm = 1 / (1 + |km(a) - km(b)|), so:

Skm = 1 / (1 + |69000 - 70000|) = 1 / 1001 = 0.0009

For the case of the year the same function could be use, then:

Syear = 1 / (1 + |2012 - 2013|) = 1 / 2 = 0.5

Last, for the trim of the car I would use a function that returns 0 if the categories were different 1 otherwise.

Strim = (base == base) = 1

Finally,

S = Wkm * Skm + Wyear * Syear + Wtrim * Strim

Where Wkm + Wyear + Wtrim = 1.0 and I would adjust the values according to what you think is decent accurate similarity ratio. Setting the values to be Wkm = 0.6, Wyear = 0.2, Wtrim = 0.2  the similarity S(a, b) = 0.30054. This similarity has the nice property that tuples with the same attributes yield a value of 1.

A good practice is to normalize the values of each attribute so you don't assign low values to any attribute for instance you could divide the columns of the km and year attributes for the maximum value, therefore for the pair of tuples in the example you will have the following transformed tuples:

a' = (0.985, 0.99, base)
b' = (1, 1, base)

Now the similarity using the same weights is S(a, b) = 0.9. Alternatively you could use a similarity matrix for the trim of the car, this is base could be more similar to mid thant to top. For example:

cat  | base | mid | top |
-------------------------
base |  1   | 0.5 | 0.2 |
-------------------------
mid  |  0.5 |  1  | 0.7 |
-------------------------
top  |  0.2 | 0.7 |  1  |

The values of the matrix have to be adjusted to fit your needs. For a more in depth comparison between similarity functions between categorical attributes (trim) see this for a comparison of different similarity measures for continuous attributes (year, km) see this.

Again to see which similarity is the best will depend on the data and on your perception, I would select a random set of pairs of cars and manually assign a similarity. Then split this set into a train set and test set. Use the train set to adjust the parameters of the different possible similarities, in the case of my proposal Wkm, Wyear and Wtrim. Then once your satisfied evaluate on the test set.


2D Pytorch tensor doesn't have independent random values I've written some Python to create a pytorch tensor of random values, sampled from a Student's t distribution with 10 degrees of freedom:

t = torch.Tensor(())
def random_from(shape):
    return torch.distributions.StudentT(10, t.new_zeros(shape), t.new_ones(shape)).sample()

If shape is of the form $(n, m)$, all values in the resulting 2D tensor are identical. I don't understand why. I did try reading Pytorch's documentation, but I couldn't find anything that helped me understand what would be a better syntax. I suppose I could create separate samples then concatenate them, but apart from the speed implications I'd like to know where my existing syntax goes wrong.

You don't use the parameters "loc" and "scale" the right way. They are not suppose to be tensors. Bellow the right syntax :

dist = torch.distributions.StudentT(10, 0, 1)
dist = torch.distributions.StudentT(10) # 0 and 1 are the default parameters

Then you can sample multiple values like that :

t = dist.rsample(torch.Size([n,m]))
t = dist.rsample(torch.Size([n]))


Are there pitfalls in using the output of machine learning model, such as a neural net as the input to a traditional GLM or similar? To be more specific, loss reserving models in actuarial science, such as the chain ladder method, can be expressed as GLMs. I have developed a predictive model using neural nets which takes into account some aspects of the insured (it is an individual risk model). Can the output of this model be safely used as an input to the insurance company's existing loss reserve model?

Theoretically there is no problem. I've seen tree models put as predictors in logistic models. NN as input into a GLM moodel makes sense. The ultimate decition should be made based on the predictability of the NN.

You have to mind a few issues:

model mantenance and deployment. The NN model would probably have more parameters than your vanilla GLM deployment might be more involved. You could live it frozen and never update it, and let the GLM model use the NN score as long as it adds to the GLM prediction.
interpretability. There might be managers whose acountability is to review risk, and they may not be comfortable with a NN. In that case feeding the NN results into a GLM might make it more acceptable, much as credit scores are used in some risk models.


Dimensions For Matrix Multiplication Can anyone explain why the following code produces input_t with a shape of (32,) instead of (,32), given the fact that inputs has a shape (100, 32)? Shouldn't input_t produce a vector with 32 attributes/columns?

import numpy as np

timesteps = 100
input_features = 32
output_features = 64

inputs = np.random.random((timesteps, input_features))

state_t = np.zeros((output_features,))

W = np.random.random((output_features, input_features))
U = np.random.random((output_features, output_features))
b = np.random.random((output_features,))

successive_outputs = [ ]

for input_t in inputs:
    output_t = np.tanh(np.dot(W, input_t) + np.dot(U, state_t) + b)
    successive_outputs.append(output_t)
    state_t = output_t

Imagine the matrix inputs as a 2D table. You have 100 rows and 32 columns. Then the for loop acts as an iterator which will return values along the first dimensional axis. This dimension thus disappears and returns the remaining dimensions. When there is a single dimension the default in Python is $(n,)$.

A matrix $(100, 32)$ iterates through $(32,)$

A matrix $(100,28,28)$ iterated through $(28,28)$

A matrix $(100,2,2,2)$ iterated through $(2,2,2)$


Confused AUC ROC score I am working on binary classification problem, I try to evaluate the performance of some classification algorithms (LR,Decission Tree , Random forest ...).
I am using  a cross validation technique (to avoid over-fitting) with AUC ROC as scoring function to compare the performance of the algorithms, but I am getting a weird results with Random forest and AdbBoost, I have a perfect AUC_ROC score (i.e. =1) despite the fact that the recall(TPR) and FPR of this algorithms are different from 1 and 0 respectively .

def FPR(y_true, y_pred):
    tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
    result = fp / (fp+tn)
    return result
def FNR(y_true, y_pred):
    tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
    result = fn / (tp+fn)
    return result
FPR_scorer = make_scorer(FPR)
FNR_scorer = make_scorer(FNR)

def get_CrossValResults2(model,cv_rst,bestIndx):
    best=pd.DataFrame.from_dict(cv_rst).iloc[bestIndx]
    roc="{:.12f}".format(best['mean_test_roc_auc'])
    acc ="{:.0%}".format(best['mean_test_accuracy'])
    prec ="{:.0%}".format(best['mean_test_precision'])
    rec ="{:.0%}".format( best['mean_test_recall'])
    f1 ="{:.0%}".format(best['mean_test_f1'])
    r2="{:.2f}".format(best['mean_test_r2'])
    g_mean="{:.2f}".format(best['mean_test_gmean'])
    pr_auc="{:.8f}".format(best['mean_test_pr'])
    fnr="{:.0%}".format(best['mean_test_fnr'])
    fpr="{:.0%}".format(best['mean_test_fpr'])
    rst = pd.DataFrame([[ model, acc,prec,rec,fpr,fnr,f1,roc,pr_auc,g_mean,r2]],columns = ['Model', 'Accuracy', 'Precision', 'Recall','FPR','FNR', 'F1-Score','ROC_auc','PR_auc','gmean','r2'])
    return rst

cross_val_rst = pd.DataFrame(columns = ['Model', 'Accuracy', 'Precision', 'Recall','FPR','FNR', 'F1-Score','ROC_auc','PR_auc','gmean','r2'])

scoring = {'accuracy':'accuracy','recall':'recall','precision':'precision','fpr':FPR_scorer,'fnr':FNR_scorer,'f1':'f1' ,'roc_auc':'roc_auc','pr':'average_precision','gmean':Gmean_scorer,'r2':'r2'}
param_grid = {'n_estimators': [200],
             'max_depth': [80,90],
              'min_samples_leaf': [2,3, 4],
            'min_samples_split': [2,5,12],
            'criterion': [ 'gini'],
             'class_weight' : [class_weights], 'n_jobs' : [-1]}
clf = GridSearchCV(RandomForestClassifier(class_weight=class_weights), param_grid, cv=kfold,scoring=scoring,refit=refit)#Fit the model
bestmodel = clf.fit(X,Y)

cross_val_rst = cross_val_rst.append(get_CrossValResults2(model='Random Forrest',bestIndx=bestmodel.best_index_,cv_rst=bestmodel.cv_results_),ignore_index=True)

Oh, I think I've finally got it.  It's just an averaging problem: for each fold in your k-fold cross-validation, you get perfect auROC, but at the default threshold of 0.5 your hard classifiers (for each fold) sometimes have $FPR=0$ and $TPR<1$, but some other times $FPR>0$ and $TPR=1$.  Then averaging you are able to get both $\operatorname{mean}(FPR)>0$ and $\operatorname{mean}(TPR)<1$.
To check, have a look at the cv_results_ table, particularly at each test fold scores (split<i>_test_<xyz>), rather than just the mean_test_<xyz> scores.


How can one use unstructured data for forecasting? How can one use unstructured data for forecasting purposes? (by unstructured, I mean unstructured in the database sense).

I have a forecasting system that uses historical data and a set of additional explanatory features to forecast demand. In terms of databases, this is as structured as data can get.

I've heard mentioned in several places that big data and unstructured data can be used to improve demand forecasts.

But when I dig into the details, it seems that they always end up reducing the unstructured data to some structured format first, then feed it to whatever forecasting method they are using. They aren't using "true unstructured data".

How can one use true unstructured data in a forecasting algorithm? Any publications, references on this?

How you use data depends mostly on the domain problem you're trying to address.

In the case you mention, it would depend on what exactly you're trying to forecast. For example, if you're trying to forecast sales or renewalls, there are some immediate use cases where you can leverage "unstructured" data to increase the number and/or the quality of the signals (features) that you are feeding into your model: 1) process audio data from contact centre interactions to determine the overall sentiment of a customer (i.e. "is he up for a sales call?"); 2) process text from customer reviews and use sentiment/keywords as a preditor for a churn model; etc!

  it seems that they always end up reducing the unstructured data to
  some structured format first, then feed it to whatever forecasting
  method they are using.

Probably because their forecasting algorithm or method requires that data is inputted in a specific format, like most forecasting or classification algorithms do; different algorithms will have different tolerances to data issues like missing values or outliers. So you process a piece of data that is unstructured with the goal of using it, usually by adding it to a data model or inputting it as data for a report or as a feature for a predictive model.

PS: I'm curious if anyone knows a case where you just throw unstructured data at a predictive algorithm (e.g. a neural network) and get something meaningful or useful out of it.


Does MNIST generalise to european handwriting? I want to use the MNIST dataset to teach my neural network to recognise numbers. My problem is: The data I am working with is "european". What i mean by that is: the seven always has a dash (it often has a dash in MNIST too so that is more unlikely to cause a problem) and more importantly: the 1 is always written with this hook in Europe, whereas Americans tend to use a straight line (almost 100% of MNIST 1s are a straight line). So it seems to me that training on MNIST can cause problems. Have you ever ran into that problem? And if so, how did you solve it? I haven't found good solutions online.

for sure this is too late for you but as i used mnist some days ago and could not get good results for my own handwriting... maybe someone else is in search for a solution:

  MNIST does represent american handwriting and will not achive good
  results in recognition of european writing.