What is/are the default filters used by Keras Convolution2d()? I am pretty new to neural networks, but I understand linear algebra and the mathematics of convolution pretty decently.

I am trying to understand the example code I find in various places on the net for training a Keras convolutional NN with MNIST data to recognize digits. My expectation would be that when I create a convolutional layer, I would have to specify a filter or set of filters to apply to the input. But the three samples I have found all create a convolutional layer like this:

model.add(Convolution2D(nb_filter = 32, nb_row = 3, nb_col = 3,
                        border_mode='valid',
                        input_shape=input_shape))

This seems to be applying a total of 32 3x3 filters to the images processed by the CNN. But what are those filters? How would I describe them mathematically? The keras documentation is no help.

Thanks in advance,

By default, the filters $W$ are initialised randomly using the glorot_uniform method, which draws values from a uniform distribution with positive and negative bounds described as so:
$$W \sim \mathcal{U}\left(\frac{6}{n_{in} + n_{out}}, \frac{-6}{n_{in} + n_{out}}\right),$$

where $n_{in}$ is the number of units that feed into this unit, and $n_{out}$ is the number of units this result is fed to.

When you are using the network to make a prediction, these filters are applied at each layer of the network. That is, a discrete convolution is performed for each filter on each input image, and the results of these convolutions are fed to the next layer of convolutions (or fully connected layer, or whatever else you might have).

During training, the values in the filters are optimised with backpropogation with respect to a loss function. For classification tasks such as recognising digits, usually the cross entropy loss is used.
Here's a visualisation of some filters learned in the first layer (top) and the filters learned in the second layer (bottom) of a convolutional network:

As you can see, the first layer filters basically all act as simple edge detectors, while the second layer filters are more complex. As you go deeper into a network, the filters are able to detect more complex shapes. It gets a little tricky to visualise though, as these filters act on images that have been convolved many times already, and probably don't look much like the original natural image.


Deep Neural Network and Activation Function I want to know why we need Activation function in DNN hidden layers.
I know a bit, like it will help us in,

Increasing model complexity and introduce non-linearity
Avoiding Gradient Vanishing  problem.

Still I'm not able to map its requirement.
An illustrative example will be definitely helpful.

Activation function is an essential part of hidden layers in DNN. Without it the network won't learn any other than a linear equation as a result and usage of DNN is meant to find more complex patterns than linear. There is a mathematical proof that the network reduces to include one weight instead of a weight matrix when no Activation function is present, thus the importance is obvious and huge.
Activation function sort of makes the DNN to test combinations of hidden layer values and that way it helps the network to converge towards the dataset form when compared to validation data and updated accordingly.
The meaning is the same as a steering wheel for a car: without it the brightest car is useless, it would apply for straight roads only if any (self-driving cars excluded).
That is a simplification of the concept, hope you got something out of it.
Very much more complex and thorough answers included in What is the purpose of an activation function in neural networks? like given by Fatemeh in comments. There you get more points on why and how the Activation function is needed and used.


Can `k=1` be a good choice for K neighbors classification? Running sklearn.KNeighborsClassifier() on Kaggle's Leaf Classification sample (set of 99 species, 10 specimen each), with defaults kNN parameters and a grid search optimization (using sklearn.model_selection.GridSearchCV), I find the best accuracy is achieved for k=1.

Although this is a technically valid choice of k, I feel using a single neighbor means the model wasn't able to find a strong relationship between neighbors.

Am I right, or is k=1 sometimes a good choice, and for which kind of problems ?

Note that using neg_log_loss as scoring parameter leads to an optimal number of k=5 neighbors.  I am still struggling to understand how this is an improvement since the accuracy is smaller then.

Update : I was able to find more information using the "1NN" and "1-NN" keywords, which suggest this is indeed a good choice in some situations.

Having k = 1 is not inherently unreasonable. This just means that all new observations will be predicted to be the class label of the first closest neighbour you have in your training set. Predicted class probabilities should just be 1 or 0 since the the estimator will take on the form 1/1 or 0/1. This also makes it clearer as to why log loss will probably not favor a k = 1 model; the predicted probabilities will be far too extreme in the cases where the model favors the wrong class label. Consider the case where the the true label is 0, i.e. $y_i = 0$. If the closest neighbour in your training set is of class label 1, then your model will incorrectly predict $Pr[Y = 1|X] = 1.$ Hence, $ (1-y_i) * ln(1-Pr[Y = 1|X])$ will be infinity. On the other hand, if the true label is 1, i.e., $y_i = 1$, and the closest neighbour to this observation ends up being of label 0, now $Pr[Y = 1|X] = 0$ and we have $y_i * ln(Pr[Y = 1|X])$ infinity as well. Thus, log loss will either be 0 (where your model predicts correctly) or -infinity for all of your observations; clearly very extreme.

However, I would just like to discuss the validity of even using accuracy as a scoring measure in the first place. Accuracy is simply not a proper scoring rule and often leads to unoptimal, bogus models. The problem is due to the arbitrary nature of the threshold you use to classify a "1" or a "0", and also that a correct classification of "1" with predicted probability 51% is not the same as a correct classification of "1" with predicted probability 90% (for instance).

Logloss is a proper scoring rule that is minimized when the predicted probabilities are close to the true, population probabilities. Accuracy is clearly influenced by what threshold you choose to use and I would not be surprised at all to see the accuracy seemingly "higher" if you choose the right threshold that maximizes accuracy for the model fit on log loss (assuming you are just using the very arbitrary 50% cutoff). Thresholds chosen should depend on the context of how your model is being used and perhaps one's own personal belief in risk; in particular, the costs of making incorrect predictions.

If you are absolutely forced in classifying objects as 1 or 0, then I suggest tuning the threshold as part of your model validation process and then assessing the accuracy on a held out test set (using the threshold found in your validation). See this thread here for more discussion on this topic.


How can I improve the accuracy of my model? (Cab Cancellation Prediction) I am trying to predict based on several parameters like trip type, car type, source of booking, start time, lead time (start- book) and a few other params whether or not a customer will cancel. From the code below the accuracy of default.ct the 1st classification I do is giving me an accuracy of 75%. deeper.ct the deeper tree that I am generating is giving me an accuracy of 70%. Progressively the accuracy of the pruned tree also is remaining the same. Boosting with adabag package is taking way too long because I've nearly 5,00,000 observations across 19 variables. xgboost is giving me the best mlogloss value at about 0.43.

What can I do to improve the accuracy of the model?

    # Generate classification tree
    default.ct <- rpart(tag ~ ., data = train.df, method = "class",
    control=rpart.control(minsplit=2, minbucket=1, cp=0.001))
    summary(default.ct)$used
    printcp(default.ct)

   # generate confusion matrix for training data
   prp(default.ct, type = 1, extra = 1, under = TRUE, split.font = 1, varlen =
  -10)
  default.ct.point.pred.train <- predict(default.ct,train.df,type = "class")
  confusionMatrix(default.ct.point.pred.train, train.df$tag)

    deeper.ct <- rpart(tag ~ ., data = train.df, method = "class", cp = 0,
   minsplit = 1)
   # count number of leaves
   length(deeper.ct$frame$var[deeper.ct$frame$var == "<leaf>"])

   ## Use cross-validation to prune the tree
   cv.ct <- rpart(tag ~ ., data = train.df, method = "class", cp = 0, minsplit =
   5, xval = 5)
   # use printcp() to print the table.

   printcp(cv.ct)
   # Use variable c to store accuracy data for different cp and print it out
   c <- list()

   for (i in 1:nrow(cv.ct$cptable)){
   pruned.ct <- prune(cv.ct,
                  cp = cv.ct$cptable[i])
   pruned.ct.point.pred.train <- predict(pruned.ct,valid.df,type = "class")
   c[i] <- confusionMatrix(pruned.ct.point.pred.train, valid.df$tag)$overall[1]
   }

    # prune the tree with second large cp and use it to predict validation data
    pruned.ct <- prune(cv.ct, cp = cv.ct$cptable[2])
length(pruned.ct$frame$var[pruned.ct$frame$var == "<leaf>"])

Create addition variable: Eg: lead_time-start_time can be time to book.
Reduce variables with many classes if present (part of EDA)
standardize numeric variables - (val-mean)/sigma
Tree is a very weak classifier, you will have to do bagging or boosting (like ada boost or gbm or random forest
try parameter tuning - I am not pasting any links since I don;t know the policy for advertising over here, but just search GBM parameter tuning in google - you'll get multiple links

You have mentioned time issues when you tried ensemble methods. To solve that:

take a subset of data and try ensembling on that (once you finalize the model, run it on whole dataset)
if you are using XGBoost, you have an option of taking in another model as an input (you can run this in batches)


Machine Learning library in Python, list or numpy or pandas Background: We are trying to build a customized ML library in Python 3 to tackle analysis we often repeat, in a general fashion. But it would not be nearly as general as sklearn. In fact, we are prepared to break some interfaces if that give us enough performance boost in return.

The basic starting point would be constructing a Learner by feeding it X and y, and predict on new input X0

learner = Learner(X, y)
y0 = learner.predict(X0)

One of the design decision is what data type to use for X and y, here are 3 choices, along with some rudimentary brainstorm advantages for each respectively:

native Python list: X being a list of lists, y being a list. Would this have better performance for being 'closer to metal'?
numpy: X being an ndarray of (n, p), y being an ndarray of (n, ). This would benefit from the richer functionalities in numpy/scipy. This is also the data type choice of sklearn.
pandas: X being a DataFrame, y being a Series. This can utilize more Data Analysis (read: dirty work) functionalities from pandas. This way we can also refer to variables with their names instead of just integer indices. But performance would be the worst?

Please share your thinking of pros and cons for each choice from both tech and math perspectives. Thanks in advance!

PS: I thought about whether this should be a StackOverflow question, but still feel this is more Data Science.

Pandas does normally a decent job allowing dataframes to behave as numpy arrays.

My recommendation is to use numpy types, the reason is that, for consistency with pretty much what the industry is doing, you are much safer with numpy.

I love pandas, and I love the dataframes, but they provide extra functionality that the model does NOT need, the same way in general programming you will not use a String to represent a boolean (even though tou could do it with a String), simply because you should use whatever data types provides you the functionality you need... and nothing else.

So, numpy is the way to go. As for python lists, you do not get the mathematical operations that you get with numpy, so do not consider them.


What is the difference between ensemble methods and hybrid methods, or is there none? I have the feeling that these terms often are used as synonyms for one another, however they have the same goal, namely increasing prediction accuracy by combining different algorithms.

My question thus is, is there a difference between them? And if so is there some book/paper that explains the difference?

Here is a para that I found by searching What are hybrid methods in Machine Learning, on google.

  "In general, it is based on combining two different machine learning techniques. For example, a hybrid classification model can be composed of one unsupervised learner (or cluster) to pre-process the training data and one supervised learner (or classifier) to learn the clustering result or vice versa." Along with that example, let's consider an example of Ensemble Learning that is Random Forest. *

In classical ensemble learning, you have different or similar algorithms, working on different or the same data-sets (for example Random Forest Stratifies the data set and builds different Decision Trees for those data-sets, while at the same time you can build different models on the same unstratified data-set and create an ensemble method). So in essence, you have different machine learning models, working independently of each other to give a prediction and then there is a voting system (hard or soft voting) which determines the final prediction.

According to the example of the hybrid machine learning model that we saw, the models in hybrid machine learning models essentially feed their output to one another (one-way) in an effort to create an efficient and accurate machine learning model. So the difference in both is that ensemble methods work independently to vote on an outcome while hybrid methods work together to predict one single outcome, which no voting element present in it.

*https://www.sciencedirect.com/science/article/pii/S1568494609001215


Cosine similarity or logistic regression for spam filtering I'm trying to choose an algorithm for filtering spam.
I found two options:

Create word dictionary for spam and not spam data.
Calculate average TF-IDF for each word and use cosine similarity for filtering.
Or use word dictionary for training logistic regression model.

Could you suggest what fit the best for my goal.
Maybe I should use some another algorithm.

Use logistic regression which allows the weights to be learnt. By using cosine similarity you are forcing the weights to be the same for all features (assuming that you normalize the features first). This is putting unnecessary restrictions on the model.


How to appropriately set weights for weighted KNN I'm currently trying to predict a continuous variable using KNN. Instead of treating each neighbor equally I would like to use the weights to create a weighted average. The weights by themselves are not ideal, as the closer a neighbor the more I would like that neighbor to influence the final results.

This lead me to consider the inverse of each of the distances, but this doesn't handle the case where an instance is the exact same -> with a distance of 0.

Any recommendations on how to properly set the weights of each neighbor relative to their distance? Similar to how the inverse would handle this, but one that allows for 0 values.

Technically speaking the inverse of the distance should not pose any problem. Considering the case when you have to classify an observation (instance) whih is identical with one observation from the 'learned' data set, than the inverse distance is not defined by a direct formula since you have to divide by zero. But considering the math behind, inverse of the zero distance is positive infinity. Which means that, the weight for the identical point would dominate all other weights of the data instances. Thus the classification for that specific point could be taken as the class of the identical observation from the data set.

On the other hand the inverse of the euclidean distance is only one type of distance. You can always apply a kernel function on that to weight non-uniformly the contributions. For example you can choose a normal density with mean equals zero and variance equals 1. Than, your weight would be the value of density function for the euclidean distance.


pandas - under a column, count the total number of a specific value, instead of using value_counts() value_counts()

function outputs the number of all unique values in a column, for example

 apple   3
 orange  2
 banana  1

I want to search the total number of (value = 'apple') only, which function can replace value_counts()?

You have plenty of ways to do it. You won't see a big difference in performance. My suggestion is to use whatever feels more convenient for you or your team.

import pandas as pd
import numpy as np

#Let's create a dataframe with 10 million integers from 0 to 100
df = pd.DataFrame(np.random.randint(0,100,size=(10000000, 1)), columns=list('A'))

#And now count the value 5 with 4 different ways

%timeit df[df.A == 5].shape[0]
10 loops, best of 3: 25.4 ms per loop

%timeit len(df[df.A == 5])
10 loops, best of 3: 25.4 ms per loop

%timeit len(df[df.A == 5].index)
10 loops, best of 3: 25.6 ms per loop

%timeit df['A'].value_counts()[5]
10 loops, best of 3: 149 ms per loop

As you can see, only the last one takes more time to run.

EDIT: Addition to your comment, you could try this

df = data.groupby('a_1').get_group(a_2)['suffix']
len(df[df.suffix == 'a_3'])


Ensemble Techniques - Boosting I understand boosting is a sequential learning technique and it use the prediction from previous model as a dataset for new model ,after adding weight to the misclassified data points.
The point which was not clear how the weights are added for misclassified ones and diminished for the correctly classified ones. It would be great if veterans can help me to understand this

Thanks in advance

Actually, it depends on the boosting algorithm you used.

In the original boosting algorithm (Schapire 1990), three classifiers are used (say $C_1$, $C_2$ and $C_3$). The training dataset is randomly divided into three subsets, $D_1$, $D_2$ and $D_3$. The training process is as follow:

Train $C_1$ using $D_1$, and use $D_2$ to test $C_1$
The instances in $D_2$ which misclassified by $C_1$ (say m instances) are taken as training data for $C_2$ along with m instances that are correctly classified by $C_1$
Train $C_2$ using the 2*m instances taken from step 2
use $D_3$ to test both $C_1$ and $C_2$
Train $C_3$ on the instances that $C_1$ and $C_2$ disagree with each other (namely, the instances that $C_1$ and $C_2$ predicted differently)

For Adaboost, the training subsets are selected by sampling from the whole training set with probabilities.

At the beginning of running the algorithm, the probabilities are uniform. After a classifier is trained and tested, the probabilities of each successfully classified instances are decreased. Then, all the probabilities are normalized after adjustment. The next classifier is trained using dataset sampled with the adjusted probabilities.

reference: Ethem Alpaydin: Machine Learning


How can I fix this "convex" problem ? Is it just a matter of overfitting? I get some metrics on validation data while training a model , and in my case the they are :

  (0.25, 0.31, 0.46, 0.57, 0.65, 0.75, 0.77, 0.78, 0.84, 0.84, 0.85,
  0.84, 0.84, 0.84, 0.82, 0.8, 0.8, 0.79, 0.78, 0.77, 0.77, 0.77, 0.75, 0.74, 0.73, 0.73, 0.73, 0.73, 0.73, 0.73)

They can described like this :

In my view , the ideal result should be like :

Is it a matter of overfitting ?

Unfortunately , I tried few times to change the regular coefficients to avoid overfitting , and adjust learning rate coefficients to slow down , but it was still "convex" .

How can I achieve the ideal result showed above ?

Much appreciated if anyone would give me some constructive tips ?

Yes, what you are seeing is a classic case of overfitting.

You stated that you use a linear model such as logistic regression. To regularize these types of models, usually L1 and/or L2 regularization is applied. L1 regularization is simply $||W||_1$ and L2 is $||W||_2^2$ usually.

Another method is to alter the labels of the model in a specific way, which is a method of regularization I created (shameless plug). Here is the link to the paper: https://arxiv.org/abs/1609.06693

Hope this helps.


Question classification I have 10 classes and 10-15 questions in each class .
Given a new question, I want to find the class to which the question is most similar?

This is called sequence classification. An approach to attack this problem is to deploy an LSTM-RNN. The key is the preprocessing of the dataset; how you encode a written question into appropriate input for the network.

The answer to this is called word embedding and detailed instructions of how to implement it can be found in this tutorial, hope it helps :)


99% validation accuracy but 0% prediction results (UNET Architecture) I am debugging results from the UNET architecture that I am using for identifying corneal reflection in eye images. While I am getting over 99% training accuracy and also very high (over 99%) validation accuracy, when I run the validation images myself, I am getting nothing but blank images from model prediction. When I used the same architecture with exact same parameters for training with a mask set for pupil, I get again high accuracy numbers but running the validation set again in the prediction gave great results. Here is the sample output for the data set where I am having trouble, mismatch of validation and prediction results:

Train on 326 samples, validate on 140 samples
Epoch 1/1
277s - loss: 0.1961 - dice_coef: 0.0012 - acc: 0.9834 - val_loss: 0.0338 - val_dice_coef: 4.8418e-11 - val_acc: 0.9979
326/326 [==============================] - 79s

Here is my code:

#-------------------------------------------------------
# Define UNET model
#-------------------------------------------------------
print("Compiling UNET Model.....")

def dice_coef(y_true, y_pred):
    y_true_f = K.flatten(y_true)
    y_pred_f = K.flatten(y_pred)
    intersection = K.sum(y_true_f * y_pred_f)
    coef = (2. * intersection + K.epsilon()) / (K.sum(y_true_f) + K.sum(y_pred_f) + K.epsilon())
    return coef

x_data = x_data[:,:,:,np.newaxis]
y_data = y_data[:,:,:,np.newaxis]
x_train, x_val, y_train, y_val = train_test_split(x_data, y_data, test_size = 0.3)

input_layer = Input(shape=x_train.shape[1:])
c1 = Conv2D(filters=8, kernel_size=(3,3), activation='relu', padding='same')(input_layer)

l = MaxPool2D(strides=(2,2))(c1)
c2 = Conv2D(filters=16, kernel_size=(3,3), activation='relu', padding='same')(l)

l = MaxPool2D(strides=(2,2))(c2)
c3 = Conv2D(filters=32, kernel_size=(3,3), activation='relu', padding='same')(l)

l = MaxPool2D(strides=(2,2))(c3)
c4 = Conv2D(filters=32, kernel_size=(1,1), activation='relu', padding='same')(l)

l = concatenate([UpSampling2D(size=(2,2))(c4), c3], axis=-1)
l = Conv2D(filters=32, kernel_size=(2,2), activation='relu', padding='same')(l)

l = concatenate([UpSampling2D(size=(2,2))(l), c2], axis=-1)
l = Conv2D(filters=24, kernel_size=(2,2), activation='relu', padding='same')(l)

l = concatenate([UpSampling2D(size=(2,2))(l), c1], axis=-1)
l = Conv2D(filters=16, kernel_size=(2,2), activation='relu', padding='same')(l)

l = Conv2D(filters=64, kernel_size=(1,1), activation='relu')(l)
l = Dropout(0.5)(l)

output_layer = Conv2D(filters=1, kernel_size=(1,1), activation='sigmoid')(l)

model = Model(input_layer, output_layer)

model.compile(optimizer=Adam(1e-4), loss='binary_crossentropy', metrics=[dice_coef, 'acc'])

#-------------------------------------------------------
# Train UNET MOdel
#-------------------------------------------------------
if train_opt:
    print("Training UNET Model.....")
    weight_saver = ModelCheckpoint(weights_file, monitor='val_dice_coef', save_best_only=True, save_weights_only=True)
    annealer = LearningRateScheduler(lambda x: 1e-3 * 0.8 ** x)
    hist = model.fit(x_train, y_train, batch_size = 8, validation_data = (x_val, y_val), epochs=1, verbose=2, callbacks = [weight_saver, annealer])
    model.evaluate(x_train, y_train)

Thanks a lot for your help!

There is no "mismatch" of accuracy. Your problem is that you have an image segmentation problem where 99% of the pixels should be zero. So getting 99% accuracy is trivially easy. A model that predicts just blank output images would score roughly the same as your network has so far. Your accuracy metric is not meaningful.

The low Dice coefficient score gives you a better idea of what is going on. That covers positive case matches only, and a good score would be close to 1.0. The low score shows that the network is not focussing on getting positive pixels correct. Instead the network has learned to predict close to zero everywhere, because that minimises the loss metric to a first approximation very well.

How to fix things?

First, stop reporting accuracy. This metric is misleading you, and you need to find another so that you can assess your model fairly. As you already have Dice, you could just drop accuracy and use Dice instead. Alternatively, you could use a weighted accuracy, inversely proportional to mean number of pixels in each class of your training data. Looking at Keras metrics options, you will probably want to add another custom metric for this. You have already done this with the dice_coef function, so that should not be a problem.

Second, train your network as it is for longer. Your example has only one epoch. Depending on how many training examples you have, this is probably too low for an image processing network. Try increasing the number in a geometric advancement - 3, 10, 30, 100, 300, 1000 - using the new metric to see if you get any improvement during training.

You could also try altering the cost function to weight it in favour of positive pixels. Keras' fit function has a class_weight parameter for this purpose, you could set it e.g. to class_weight = {0:1, 1:20}


anomaly detection - conceptual help on machine learning I am working on Anomaly detection model problem for a finance data set -  set of card transactions. My team member suggested an idea that
" First train the model with normal instances of data and to the trained model pass the anomalous data during testing to see whether the model will be able to detect the anomaly or not."
Is it possible like this with any Machine Learning algorithms. If so, can anybody suggest what algorithms can be used for this idea?

An additional question: How about the situation when there is no properly labelled fraudulent data and we have just transactional information. Does semi-supervised learning works? I do not have experience in this type of ML approach. Can anybody suggest good resources for this and also suggest can Semi-supervised learning or hybrid supervised learning works for anomaly/ fraud detection?

There is two options:

You have a label

Then what you probabibly have is a binary classification problem, probably unbalanced. If this happens follow this https://towardsdatascience.com/practical-tips-for-class-imbalance-in-binary-classification-6ee29bcdb8a7

You have no label

Then you have an unsupervised learning problem, in finance Isolation Forest is really used. If this is the case have a look at this https://towardsdatascience.com/outlier-detection-with-isolation-forest-3d190448d45e


How word2vec can handle unseen / new words to bypass this for new classifications? In simple terms, if my classification is based on word2vec as features, what I am supposed to do, if a new word comes, which does not have a word2vec?
I am trying to used word2vec or word vectors for classification based on entity.
For example:
I have to classify the following words in a sentence as:
"Google gives information about Nigeria"

Here, I would like to classify Nigeria as location.
Suppose I have good word2vec vectors for each of the words, based on some readings I came to know that, recurrent neural networks can be used for this. So, word2vec will capture most locations with a kind of similar word vectors.
But my questions are:
a) Suppose a new location is there. lets say, Russia . So, do I need to assign a new word vector for this location ?
b) If my input for training does not have grammatical sense. For example,
" Google information Nigeria " . Everything else Nigeria is associated with a non-location label. Will this condition work for find new location in non-grammatical sentences.

One way to do this is to use context information to represent each word along with the w2v vector. You can choose to represent this information in any way you like: add another 600 dimensions (100D w2v vectors for 3 left and 3 right context words), just another 100D as the sum of context vectors or any other fixed length representation for your context.

When you're training, you can use a version of 'word dropout' that will utilize this information. 20% of the time, set your w2v vectors to zero, forcing your classifier to use the context dimensions to represent the word.

When you encounter a new word, the hope is, the classifier learned to use the context information as well as it learned to use the w2v information.


Metrics to determine K in K-cross fold validation Consider a scenario where the dataset in hand is quite large, let's assume 50000 samples (quite well balanced between two classes).  What metrics can be used to decide the K value in a K-fold cross-validation? In other words, can a 5-fold CV be enough or should I go for a 10-fold CV?

The rule of thumb is the higher K, the better. But, putting aside the computational costs, what can be used to decide the value of K? Should we look at the overall performance, e.g. average accuracy? That is, if accuracy (5CV) ~ accuracy(10CV), we can opt for 5-fold CV?. Is the standard deviation between the performance of different folds important? That is, the lower the better?

The rule of thumb is the higher K, the better.

I think a better rule of thumb is: The larger your dataset, the less important is $k$.

However, it is useful to have a general understanding of the impact of $k$ on the performance estimator (leaving aside computational costs):

Increasing $k$ decreases the bias because the training set better represents the data
Increasing $k$ increases the variance of the estimator because the training data sets are becoming more similar

Also note that there is no unbiased estimator for the variance of the $k$-fold CV. Together this means that there is no metric that can tell you the best $k$ if you leave computational costs aside. Some empirical studies suggest that 10 is a reasonable default.

And to be clear, $k$ is not a hyper-parameter you want to tune to find the best accuracy. If you start performing $k_2$-fold CV to find the best $k_1$ something hopefully feels wrong.


How to create ROC - AUC curves for multi class text classification problem in Python I am working on a multiclass text classification problem and trying to plot ROC Curve but no success so far. Tried many solutions available but didn't work. Kindly please someone help me out with the following piece of code to plot the ROC curve. There are actually five different classes for which I am performing text classification.
categories = ['Philonthropists', 'Politcians', 'Showbiz', 'Sportsmen', 'Writers']
train = dt.load_files(r'C:\Users\...\Learning\Train', categories=categories, encoding='ISO-8859-1')
test = dt.load_files(r'C:\Users\...\Learning\Test', categories=categories, encoding='ISO-8859-1')
count_vector = CountVectorizer()
x_trian_tf = count_vector.fit_transform(train.data)
tfidf_transformer = TfidfTransformer()
x_train_tfidf = tfidf_transformer.fit_transform(x_trian_tf)
learn = MultinomialNB().fit(x_train_tfidf, train.target)
x_test_tf = count_vector.transform(test.data)
x_test_tfidf = tfidf_transformer.transform(x_test_tf)
prediction = learn.predict(x_test_tfidf)
print("Accuracy is of Multinomial Naive Bayes Classifier", accuracy_score(test.target, prediction) * 100)

First check out the binary classification example in the scikit-learn documentation. It's as easy as that:
from sklearn.metrics import roc_curve
from sklearn.metrics import RocCurveDisplay
y_score = clf.decision_function(X_test)

fpr, tpr, _ = roc_curve(y_test, y_score, pos_label=clf.classes_[1])
roc_display = RocCurveDisplay(fpr=fpr, tpr=tpr).plot()

In the case of multi-class classification this is not so simple. If you have 3 classes you could do ROC-AUC-curve in 3D. Have a look at the resources here.
What you can do and what is much simpler is to make 4 one-vs-all curves. You basically have a binary setting for each class.
In your case:
import matplotlib.pyplot as plt
# all the same up until now
prediction = learn.predict(x_test_tfidf)
proba = learn.predict_proba(x_test_tfidf)
print("Accuracy is of Multinomial Naive Bayes Classifier", accuracy_score(test.target, prediction) * 100)

for i in range(len(categories)):
    y_test_bin = np.int32(test.target == i)
    y_score = proba[:,i]
    fpr, tpr, _ = roc_curve(y_test_bin, y_score, pos_label=0)
    plt.subplot(2,2,i)
    roc_display = RocCurveDisplay(fpr=fpr, tpr=tpr).plot()


Markov switching models What are some reference sources for understanding Markov switching models?

Firstly, for understanding the Markov switching models, a nice knowledge of Markov models and the way they work. Most importantly, an idea of time series models and how they work, is very important.

I found this tutorial good enough for getting up to speed with the concept.

This is another tutorial on a similar application of the switching model, which is the regime switching model.

The statsmodels library has a nice support for building the Morkov switching models.

Here is one simple and quick Python tutorial which uses the statsmodels library.


normal conda install vs forge install In order to install some libraries, we just use:

conda install pandas

While in other cases we use:

conda install -c conda-forge html5lib

What is the difference between them?

conda-forge is just an alternative channel where you can upload to or download packages from. Usually it makes no difference where you download the package from but sometimes conda-forge has the latest version.


Oscilations in loss curve I saw a similar question, but I think my problem is something different.
While training, the training loss and the validation loss move around one number, not decreasing significantly.
I have 122707 training observations and 52589 test observations with 55 explanatory variables and one dependent, One CONN1D with 24 filters, 2 Lstm years with 24 units and one dense layer. I've added a dropout rate of 0.2 between the layers. Total parameters 13417.
Seems like my model is not learning at all. Does it mean that the dataset is not a good representation of the specific problem? Should I increase the number of epochs? I use Adam optimizer with default learning rate.

Adding additional info:
I am trying to predict next hour air pollution based on previous value air pollution concentration, previous hour meteorological data as temperature, windspeed etc. Day, Hour and Month are also included and encoded with One Hot Encoding. Additionaly the wind degree is decomposed according to its sin and cos components. Previously I tried normalization of data but it didn`t seem to give any difference. Haven't try any other models. Here is the model:
model = tf.keras.models.Sequential([
tf.keras.layers.Conv1D(filters=24, kernel_size=3,
                  strides=1, padding="causal",
                  activation="relu",
                  input_shape=[None, 55]),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.LSTM(24, return_sequences=True),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.LSTM(24, return_sequences=True),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(1),
tf.keras.layers.Lambda(lambda x: x * 1000)
])

Somewhere I saw Labda layer after Dense layer in regression. I noticed that adding Lambda layer at the end speeds up the learning. I multiply the output with 1000 because it is the maximum value for the variable I want to predict.

Rather than oscillations, it looks like white noise, like a random walk. In other words, as you said, your model is not learning anything.
Unfortunately it's impossible to say what's wrong, since we can't see any code. We need more information about dataset, how you processed it, model implementation, all the hyperparams you chose, what other versions you tried before that one, ... the list is countless. But most importantly it's really hard to help you without code.
If the dataset is a good one the problem must be some error you made along the way.

EDIT:
Here's what I think:

You don't need Conv layers followed by RNN layers. This doesn't really make sense. Let the LSTM receive raw input.

Don't use Dropout with RNNs, they don't go along very well together. Dropout makes sense with Dense and Conv data, but in RNNs, where sequence is everythin, they can actually make things worse. Somebody uses recurrent dropout as an alternative but it's not necessary.

Don't use return_sequences=True between an LSTM and a Dense layer. That must be used between LSTM's only.

That Lambda layer at the end is probably causing most of the error. If you multiply all your predictions by 1000, what you get is by definition a prediction that on a completely different scale than your target value.

The Network overall is too deep and has too many parameters. I assume you are working with the famous Beijing air quality dataset. In this case, It is enought to work with one LSTM layer, followed by a Dense node to make the prediction. Everything else is overkill and not necessary for a simple dataset like that.

Something much more simple, like:
model = tf.keras.models.Sequential([
    tf.keras.layers.LSTM(24, input_shape=(seq_len, n_vars)),
    tf.keras.layers.Dense(1),
])

has higher chance to work. (Please specify the input shape correctly). Try playing with its hyperparameters, after you made sure all variables are properly scaled between train and test data.
Good luck!


Gaussian process regression: sudden increase of the prediction's variance I am trying to do a variance-based uncertainty sampling like in the book "Active Learning" by Burr Settles, page 18, Figure 2.6. Link to the book: http://active-learning.net/

I use Gaussian process regressor from scikit-learn to fit a 1-D Gaussian function. The parameters of the regressor are the same as in this example (a noise-free case) http://scikit-learn.org/stable/auto_examples/gaussian_process/plot_gpr_noisy_targets.html

kernel = C(1.0, (1e-3, 1e3)) * RBF(10, (1e-2, 1e2))
gp = GaussianProcessRegressor(kernel=kernel, n_restarts_optimizer=9)

Initially, I randomly choose two training data points and fit the curve based on them. The Figure below shows the fit (top) and the variance (bottom)

Then I add one more training data point taken at the place where the variance is the largest, and re-fit the data.

I repeat this process for a few iterations and the interpolation improves with each iteration, as expected. For example, this is the iteration 7:

However, at the next iteration, the interpolation is worse. The variance suddenly increases:

The next few iterations are fine again. For example, iteration 9:

But at the iteration 11 I get a large variance again:

Iterations 12 through 15 show very small variance again - the curve is interpolated very well at those iterations. I stopped at 15.

Why does this sudden increase of variance happen at certain steps? How to avoid it?

When I change the parameter n_restarts_optimizer, this behavior is still observed but at different iteration steps. For example, when n_restarts_optimizer=8, the increase in variance is at iteration 9. When n_restarts_optimizer=7, the increase in variance is at iteration 4, 10, 11. When n_restarts_optimizer=4, the increase in variance is at iteration 7, 14.

When I change the parameter length_scale in the RBF kernel from 10 to 1.0 I don't see this increase in variance anymore. However, when I make it even smaller (length_scale = 0.1) it appears again.

Hmm, I've seen this before. It seemed to be some kind of overfitting on the time component where only very nearby samples were taken into account, before defaulting to the prior (which has zero mean). That would explain why increasing the length scale would help.

Do you see large changes in the hyper parameters of the kernel between iterations (gp.kernel_.get_params())? You could consider using the hyper parameters from a previous iteration as initial guess for the next iteration.


What is the difference between feature generation and feature extraction? Can anybody tell me what the purpose of feature generation is? And why feature space enrichment is needed before classifying an image? Is it a necessary step?
Is there any method to enrich feature space?

Feature Generation -- This is the process of taking raw, unstructured data and defining features (i.e. variables) for potential use in your statistical analysis. For instance, in the case of text mining you may begin with a raw log of thousands of text messages (e.g. SMS, email, social network messages, etc) and generate features by removing low-value words (i.e. stopwords), using certain size blocks of words (i.e. n-grams) or applying other rules.
Feature Extraction -- After generating features, it is often necessary to test transformations of the original features and select a subset of this pool of potential original and derived features for use in your model (i.e. feature extraction and selection). Testing derived values is a common step because the data may contain important information which has a non-linear pattern or relationship with your outcome, thus the importance of the data element may only be apparent in its transformed state (e.g. higher order derivatives). Using too many features can result in multicollinearity or otherwise confound statistical models, whereas extracting the minimum number of features to suit the purpose of your analysis follows the principle of parsimony.
Enhancing your feature space in this way is often a necessary step in classification of images or other data objects because the raw feature space is typically filled with an overwhelming amount of unstructured and irrelevant data that comprises what's often referred to as "noise" in the paradigm of a "signal" and "noise" (which is to say that some data has predictive value and other data does not). By enhancing the feature space you can better identify the important data which has predictive or other value in your analysis (i.e. the "signal") while removing confounding information (i.e. "noise").


Multiple time-series predictions with Random Forests (in Python) I am interested in time-series forecasting with RandomForest. The basic approach is to use a rolling window and use the data points within the window as features for the RandomForest regression, where we regress the next values after the window on the values within the window. Just plain autoregressive model (with lags), but with Random Forest instead of linear regression.
Problem:
If I have more than one time-series (multiple time-series), how to pass them in RF regression?
For example: given two time series $y_1(t)$ and $y_2(t)$, the outcome time series is $z(t)$ and I am interested in predicting the values of $z(t)$ based on the combination of $y_1$ and $y_2$.
What I need, is to use rolling window for each $y_1$ and $y_2$, and then feed these values within the window from both time-series into RF regression, to predict the value of $z(t)$.
Question:
How do I incorporate the data from both rolling windows into the input for RF regression?

Random forest (as well as most of supervised learning models) accepts a vector $x=(x_1,...x_k)$ for each observation and tries to correctly predict output $y$. So you need to convert your training data to this format. The following pandas-based function will help:

import pandas as pd

def table2lags(table, max_lag, min_lag=0, separator='_'):
    """ Given a dataframe, return a dataframe with different lags of all its columns """
    values=[]
    for i in range(min_lag, max_lag + 1):
        values.append(table.shift(i).copy())
        values[-1].columns = [c + separator + str(i) for c in table.columns]
    return pd.concat(values, axis=1)

For example, the following code:

df = pd.DataFrame({'y1':[1,2,3,4,5], 'y2':[10,20,40,50,30], 'z': [1,4,9,16,25]})
x = table2lags(df[['y1', 'y2']], 2)
print(x)

will produce output

   y1_0  y1_1  y1_2  y2_0  y2_1  y2_2
0   1.0   NaN   NaN  10.0   NaN   NaN
1   2.0   1.0   NaN  20.0  10.0   NaN
2   3.0   2.0   1.0  40.0  20.0  10.0
3   4.0   3.0   2.0  50.0  40.0  20.0
4   5.0   4.0   3.0  30.0  50.0  40.0

The first two rows have missing values, because lags 1 and 2 are undefined on them. You can fill them with what you find appropriate, or simply omit them.

When you have matrix of $x$ values, you can feed it, for example, to a scikit-learn regressor:

from sklearn.ensemble import RandomForestRegressor
rf = RandomForestRegressor().fit(x[2:], df['z'][2:])

Finally, a piece of advice. Your model could much improve if you used not only raw lagged values as features, but also their different aggregations: mean, other linear combinations (e.g. ewm), quantiles, etc. Including additional linear combinations into a linear model is useless, but for tree-based models it can be of much help.


two different attention methods for seq2seq I see two different ways of applying attention in seq2seq:

(a) the context vector (the weighted sum of encoder hidden states) fed into the output softmax, as shown in the diagram below. The diagram is from here.

(b) the context vector fed into the decoder input as shown the diagram below. The diagram is from here.

What are the pros and the cons of the two different approaches? Is there any paper comparing the two?

(a) is Luong's attention mechanism (link) while (b) is  Bahdanau's mechanism (link)


Source of Arthur Samuel's definition of machine learning Many people seem to agree that Arthur Samuel wrote or said in 1959 that machine learning is the "Field of study that gives computers the ability to learn without being explicitly programmed".

For example the quote is contained in this page, that one and in Andrew Ng's ML course. Several articles also contain this quote, and the reference is always the following article, which doesn't actually contain the quote.

Samuel, A. L. (1959). Some studies in machine learning using the game of checkers. IBM Journal of research and development, 3(3), 210-229.

Is there a reliable source? Or is this actually not a quote, but rather an interpretation of Samuel's article?

The exact quote exists in neither the 1959 paper nor the 1967 paper (second version).

These are the closest quotes from the 1959 paper:

  A computer can be programmed so that it will learn to play a better
  game of checkers than can be played by the person who wrote the
  program.

And

  Programming computers to learn from experience should eventually
  eliminate the need for much of this detailed programming effort.

Also, Wiki page of Arthur Samuel states that:

  He coined the term "machine learning" in 1959

and references the 1959 paper.

Either the quote is created as a gist of Arthur Samuel's 1959 paper, or it is said but not written by him. In my opinion, the former is more probable, since it is not even remotely mentioned in the 1967 paper.


Simple prediction with Keras I want to make simple predictions with Keras and I'm not really sure if I am doing it right. My data looks like this:

col1,col2
1.68,237537
1.69,240104
1.70,244885
1.71,246196
1.72,246527
1.73,254588
1.74,255112
1.75,259035
1.76,267229
1.77,267314
1.78,268931
1.79,273497
1.80,273900
1.81,277132
1.82,278066

Now, I want to predict col2 by col1 and this is how I'm doing it:

df = pandas.read_csv('data.csv', usecols=[0, 1], header=None)
X = df.iloc[:, :-1].values.astype(np.float64)
y = df.iloc[:, -1:].values.astype(np.float64)
scalarX, scalarY = MinMaxScaler(), MinMaxScaler()
scalarX.fit(X)
scalarY.fit(y.reshape(len(y),1))
X = scalarX.transform(X)
y = scalarY.transform(y.reshape(len(y),1))

model = Sequential()
model.add(Dense(4, input_dim=1, activation='relu'))
model.add(Dense(4, activation='relu'))
model.add(Dense(1, activation='linear'))
model.compile(loss='mse', optimizer='adam')
model.fit(x=X, y=y, epochs=3, verbose=1)
for num in range(1, 21):
    Xnew = np.array([[float(Decimal('2.{}'.format(num)))]])
    ynew = model.predict(Xnew)
    print("X=%s, Predicted=%s" % (Xnew[0], ynew[0]))

What you are trying to do here is forecast the future values of a time series. This is a predictive problem and the future values will depend on a number of latent factors. I will assume all we have access to is historical data from the series as your question indicates.

If you want to predict a future value for the time series, you should not only use the current value as an input, but rather you should use a chunk of the historical data. Since you have 18,000,000 instances, this is a lot, you can make your network quite complex in order to capture some latent trends hidden inside your data which can help predict the future value. To predict a value at time $t$ we will use the $k$ previous values. This hyper-parameter needs to be effectively tuned.

Restructure the data

We will structure the data such that the features $X$ are the $k$ previous time measurements, and the output target $Y$ is the current time measurement. The one that is being estimated by the model.

k = 3
X, Y = [], []
for i in range(len(col1) - k):
    X.append(col2[i:i+k])
    Y.append(col2[i+k])

X = np.asarray(X)
Y = np.asarray(Y)

Split your data

from sklearn.cross_validation import train_test_split
x_train, x_test, y_train, y_test = train_test_split(X, Y, test_size=0.33)

Using the data in a Keras model

This is a simple Keras model which should work as a first iteration step. However, due to the small amount of data you provided us I cannot get any meaningful results after training.

import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv1D, MaxPooling1D, Reshape
from keras.callbacks import ModelCheckpoint
from keras.models import model_from_json
from keras import backend as K

x_train = x_train.reshape(len(x_train), k, )
x_test = x_test.reshape(len(x_test), k, )

input_shape  = (k,)

model = Sequential()
model.add(Dense(32, activation='tanh',
                 input_shape=input_shape))
model.add(Dense(32, activation='tanh'))
model.add(Dense(1, activation='linear'))

model.compile(loss=keras.losses.mean_squared_error,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

model.summary()

epochs = 10
batch_size = 128
# Fit the model weights.
history = model.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test, y_test))


Azure Data Studio opens Jupyter Notebook / Lab HTML When I start Jupyter Lab or Notebook from the terminal, Azure Data Studio opens the auto-redirect HTML file instead of Firefox.

Is there a way to correct this behavior?  I'm experiencing this issue on Ubuntu 18.04 on my personal computer, but do not see this behavior on my work computer, which is a Macbook Pro.  Currently running Azure Data Studio 1.5.4.

jupyter                   1.0.0
jupyter_client            5.2.4
jupyter_console           6.0.0
jupyter_core              4.4.0
jupyterlab                0.35.3
jupyterlab_launcher       0.13.1
jupyterlab_server         0.2.0

EDIT: After starting the HTML in ADS, this is the output I see in the terminal:

[main 11:17:33 PM] update#setState idle
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

Changing the file ~/.config/mimeapps.list with settings to
text/html=firefox.desktop
fixed the issue.
This issue is related to this one where other solutions are also provided (set a default browser for Jupyter).


What is weight and bias in deep learning? I'm starting to learn Machine learning from Tensorflow website. I have developed a very very rudimentary understanding of the flow a deep learning program follows (this method makes me learn fast instead of reading books and big articles).

There are a few confusing things that I have come across, 2 of them are:

Bias
Weight

In the MNIST tutorial on tensorflow website, they have mentioned that we need bias and weight to find the evidence of the existence of a particular pattern in an image. What I don't understand is, where and how the values for Bias and Weight are determined?

Do we have to provide these values or does the TensorFlow library calculates these values automatically based on the training data set?

Also if you could provide some suggestions on how to accelerate my pace in deep learning, that would be great!

Tensorflow Beginners Tutorial

Mathematically speaking, Imagine you are a model (No! not that kind, figure 8 ones).
Bias is simply how biased you are. If you are a Nigerian, and you are asked "Which nationality have the most beautiful women" you might say Nigerian Ladies. We can say it's because you are biased. So your formula is $Y = WX + nigerian$.
So what do you understand? Biased is that pre-assumption in a model like you have.
As for weight, logically speaking, Weight is your Gradient (as in linear algebra).
What is Gradient? it's the steepness of the Linear function.
What makes the linear gradient very steep (High positive value)?
It's because little changes in X(input) causes Large differences in Y axis(output). So you (Not as a Model anymore, but a brilliant Mathematician (your alter ego)) or your Computer tries to find this gradient, which you can call weight. The difference is that you use a pencil and graph book to find this, but the black box does its electronic Magic with registers.
In the Machine Learning Process, computers, or you, try to draw many straight lines or Linear functions across the data points.
Why do you try to draw many straight lines?
Because in your graph book/Computer memory, you are trying the see the line that fit appropriately.
How do I or Computer know the line that fits appropriately?
In my secondary school, I was taught to draw a line across the data points, visually checking the line that cuts through perfectly in the middle of all the data point.(Forget those A.I hype, our brains can calculate by just staring at things). But as for the computer, it tries the standard deviation and variance of each line towards the data points. The line with the least deviation (sometimes we call it the error function) is chosen.
Cool! so and what happens
The gradient of that line is calculated, let's say the Weight of the Learning problem is Calculated.
That's Machine Learning at its basic understand and a high school student plotting graph in his/her Graphbook.


Regression loss function is nan I'm a beginner with ANN and DL in general. I have a regression task with a target of 2-dimensions, my dataset only have 46 samples (small dataset I think).
I tried the code below that does a regression with only one output which works normally.

When I change to a two dimensional regression, my loss function becomes equal to NaN.
I tried to change the optimizer and fix the dropout rate, but nothing changed, any solution?

import numpy
import pandas
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

#load data
dataframe = pandas.read_excel("data1.xlsx")
dataframe.isnull().any()

dataset = dataframe.values

X = dataset[:, 0:5]
Y = dataset[:,5:7]

def baseline_model():
    #create model
    model = Sequential()
    model.add(Dense(10, input_dim=5, kernel_initializer='normal', activation
='relu'))
    model.add(Dropout(0.5))
    model.add(Dense(2, kernel_initializer='normal'))

    #compile model
    #model.compile(loss='sparse_categorical_crossentropy', optimizer='adam')
    model.compile(loss='mean_squared_error', optimizer='adam')
    return model

#fix random seed for reproducibility
seed = 7
numpy.random.seed(seed)

estimators=[]
estimators.append(('standardize', StandardScaler()))
estimators.append(('mlp', KerasRegressor(build_fn=baseline_model, epochs=50, batch_size=32, verbose=0)))
pipeline = Pipeline(estimators)

kfold = KFold(n_splits=10, random_state = seed)
results = cross_val_score(pipeline, X, Y, cv=kfold)

print ("Result: %.2f (%.2f) MSE" %(results.mean(), results.std()))

I faced same issues and searched solution in many places. But apparently, it's because I put wrong label in my dataset. It should be 0,1,2,... n but I made 1,2,3,....n. It should be started from zero. After I changed the label, it works well now.


Problem faced when collect data randomly from cluster I have a semi structured data set. I need to collect some data (unlabeled) randomly for labeling.

As initiative at first I separated labeled and unlabeled data. Then I convert those data from string to numeric as all data were string.

After that I cluster unlabeled data into 2(cluster 0 and cluster 1) by applying k-means clustering algorithm.

Now when I collect some data randomly from both cluster it takes indexes from both labeled and unlabeled data. But my expectation was, indexes will come just from unlabeled data as cluster was created by using unlabeled data.

I couldn't figured out the problem. I am giving my code below.

train_label.head()

The indexes of labeled data stats from 0 to 343 (344 data) and unlabeled data starts from 344 to 863 (519 data)

train_unlabeled.head()

# define X and y
feature_cols = ['TopLeft', 'TopMiddle', 'TopRight', 'MiddleLeft','MiddleMiddle', 'MiddleRight', 'BottomLeft', 'BottomMiddle', 'BottomRight']

# X is a matrix, hence we use [] to access the features we want in feature_cols
X =train_unlabeled[feature_cols]`

# y is a vector, hence we use dot to access 'label'
y =train_unlabeled['Class']

X_scale = scale(X)
reduced_data = PCA(n_components=None).fit_transform(X_scale)
kmeans = KMeans(init='k-means++', n_clusters=2, n_init=10)
model=kmeans.fit(reduced_data)
reduced_data.shape

labels=kmeans.labels_

Now I was trying to collect 95% data from each cluster (cluster was created by train_unlabeld data)

x = 0.95
i=0
C_i = np.where(labels == i)[0].tolist()
n_i = len(C_i)
sample_i = np.random.choice(C_i, int(x * n_i))
print (i, sample_i)
list1=(sample_i)
x = 0.95
i=1
C_i = np.where(labels == i)[0].tolist()
n_i = len(C_i)
sample_i = np.random.choice(C_i, int(x * n_i))
print (i, sample_i)
list2=(sample_i)

When I print the index while collecting data randomly depending on cluster(0 and 1) I found these indexes

As the index of unlabeled data is started from 344, so how the index of labeled data comes in cluster 0 and ? I didn't understand it.

The problem was understanding the index between train_unlabeled data and the cluster. There are no connection between index of train_unlabeled data set and the data that belongs to cluster. As I am using jupyter notebook, so the indexes given to itself. Now when I clustered those unlabelled data (index start from 344 to 863) into 2, then which data belongs which cluster it already takes its new index for cluster which is not same as previous index.

For better understanding I'm giving an example:

suppose for unlabelled data (train_unlabeled) the data those indexes are 344,345,346,347 and 348. After clustering 344 and 346 took their place at cluster 0 and 345,347,348 at cluster 1. While clustering the new index of 344 will be 0, for 345 will be 1, for 346 will be 2 and so on.

That's why when I print the indexes of data, belongs to cluster 0 and 1 (by using print (i, sample_i) this command) it just shows the index of data of its own cluster.

My mistake was I though that both indexes are same.

Now for keep similarity of indexes before and after cluster, I reset the the index of train_unlabeled data.

The command is: train_unlabeled = train_unlabeled.reset_index()

Here comes another problem, that is a new column named:index is added.

So, after resetting index (now, index starts from 0), I drop the extra index column by using this command : train_unlabeled =train_unlabeled.drop('index', axis=1, inplace=True)

And at last I cluster that unlabeled data (train_unlabeled) and would be able to take 95% data randomly from cluster by using cluster index.


Gini index as a labeling strategy for leaf nodes Can we use the gini index to assign a class to a leaf node? If yes how? As far as I understand the gini index can only be used as a splitting metric.

Short answer: No

Long answer:

What do you mean by 'assign a class to a leaf node'?  The question itself is strange. Gini index is used as splitting criteria in the building process of decision tree and the classes in leaf nodes are the final result of a building process.

The building of a decision tree is an automatic process based on the principle of greedy maximization of some quality criteria/splitting rule (Gini uncertainty (Gini impurity/Gini index), information gain, Misclassification error): at each step, the algorithm chooses the variable that gives the greatest value upon splitting. Then the procedure is repeated recursively until the entropy is zero (or some small value to account for overfitting).

Maximizing the Gini uncertainty can be interpreted as the maximization of the number of pairs of objects of the same class that are in the same subtree.

I will leave an example of tree below.

Also, if you want, you can check out this wonderful notebook on the theme that gives a full understanding of decision trees and the theory behind them:

https://mlcourse.ai/articles/topic3-dt-knn/#2.-Decision-Tree


Is data subsampling appropriate for hyperparameter optimisation? Fundamentally, under what circumstance is it reasonable to do HPO only on a subsample of the training set?
I am using Population Based Training to optimise hparameters for a sequence model. My dataset consists of 20M sequences and was wondering if it would make sense to optimise on a subsample due to restricted budget.

Your subsample has to be representative of your original dataset.
To do so, as you are in a supervised case, I would get a random subsample that keeps the classes distribution (for instance getting randomly 40% of each class).
Note
If you have classes with too few examples, I would also try not to sample them. Risk is even with random sampling you could loose information when a cluster is too small. Plus, if your problem is computation time, that won't be a problem to keep the too small clusters while sampling the bigs.


What is the Best NoSQL backend for a mobile game What is the best noSQL backend to use for a mobile game? Users can make a lot of servers requests, it needs also to retrieve users' historical records (like app purchasing) and analytics of usage behavior.

Some factors you might consider:

Developer familiarity: go with whatever you or your developers are familiar with.  Mongo, Couch, Riak, DynamoDB etc all have their strengths but all should do ok here, so rather than going for an unfamiliar solution that might be slightly better go for familiar and save a bunch of development time.

Ease of cloud deployment:  for example, if you are using Amazon AWS, then DynamoDB is likely an excellent choice.  Sure, you could use Mongo on AWS, but why bother?  Other cloud providers have their own preferred db, for example if you are using Google AppEngine, it makes sense to use BigTable or Cloud Datastore.

Your use case seems both well suited to NoSQL and not very challenging since your data has a natural partition by user.  I think you'd be technically ok with anything, which is why I'm mainly covering other factors.


How to encode an array of categories to feed into sklearn I'm working on a recommendation problem, broadly following the Youtube paper on theirs. Their surrogate problem is to recommend the next video a user will watch. One feature they include in their model is a representation of watch history (I.E. which videos a user has watched), which they achieve by embedding the Video ID's and then averaging that embedding.

So for data, they might have...

[
    [1234, 5678, 9012],
    [1234],
    [9012, 1234, 8245],
]

and each of those IDs is embedded into say 8 features, those 8 features are averaged for every observation in the training data and those 8 average features are input to the deep layers of the network.

Now I can replicate that in keras using a combination of tf.keras.preprocessing.sequence.pad_sequences plus Embedding() and GlobalAveragePooling1D(), and that works reasonably well. My problem is that I'd like to evaluate other algorithms on this same problem (after all, I have WAY less data than Youtube does so I'm not sure I need an NN at all). I can't really conceptualise how to deal with this encoding of historical "watches" though in a way that would work in a more traditional algorithm like XGBoost or RandomForests in sklearn. Ordinal Encoding and taking the average seems plain silly, I obviously can't OneHot encode the 1st to nth video watch and so on. Any ideas of how to tackle this would be greatly appreciated.

EDIT:

I guess possible FeatureHasher fits the bill? Rather than an average, it produces a sum of each of the hashed input categories. That's still (be default) gonna lead to 10s of thousands of features though so I'm not sure that it's going to be a significant improvement over OneHotEncoding!

EDIT 2:

Alternatively, perhaps I could treat this as a text embedding problem by concatenating the array of categories into a single string and use CountVectorizer to create an array of (n_samples, n_categories), then use TruncatedSVD to reduce that to a more manageable number of components.

I guess I've a couple of ideas there, anyone know if they're worth trying?

If the sequence of watches is meaningful, then you do need some kind of classifier that creates user/item embeddings from the sequence of watches, so you're probably looking at GRUs and LSTMs. Neural nets aren't overkill here per se. This is a pretty good treatment of the topic: https://towardsdatascience.com/introduction-to-recommender-system-part-2-adoption-of-neural-network-831972c4cbf7

However this simple approach above does mean you have a multi-label class with a lot of dimensions. This can be hard to scale and may not train well without much data. Feature hashing doesn't help so much in the output because you need to map each back to the right video

If you're willing to forego the sequence, which often doesn't carry that much information beyond simple recency (recent watches are more important). Yes the usual way of dealing with this is as a huge matrix factorization problem. Sparse SVD is possible but maybe overkill; ALS is simpler and about as effective, and works well for implicit input like you have here. You can readily add weights to watches with ALS, which can be used to deemphasize older watches.


A deep learning approach that can calculate distance to obstacles After reading about the most famous object detection CNN based methods: YOLO, YOLO 9000, r-cnn, faster r-cnn, etc., I was wondering if there is an architecture that can calculate the distance to the obstacle after detection?
Thank you.

Actually, I don't think this is a problem that could be solved just from the image because it could be a small object that is really close or and big object really far. As far as I know, self driving cars use sensors for this. From what I found here, you would need the actual height of the object to calculate it from one image.

I strongly suspect you could also calculate the distance by having two images and then using the distance between the two cameras, and the Pythagorean theorem to calculate it. But that's just conjecture.


Best network structure for unsupervised learning Currently using networkX to build a probability based network for unsupervised learning, it is a basic reader understanding relational connectivity through probabilities, network slows significantly after about 40 pages (8000 words) any suggestions for a better network platform would be much appreciated.

You are describing a problem that can be solved with Probabilistic Graphical Models (PGMs). PGMs use graphs to expresses the conditional dependence structure between random variables.

pgmpy is a Python package for PGMs.


What is the bleu score of professional human translators? Machine translation models are usually evaluated using bleu score. I want to get some intuition for this score. What is the bleu score of professional human translator?

I know it depends on the languages, the translator ect. I just want to get the scale.

edit: I want to make it clear - I talk about the expected bleu. It's not a theoretical question, it is an experimental one.

The original paper "BLEU: a Method for Automatic Evaluation of Machine Translation" contains a couple of numbers on this:

  The BLEU metric ranges from 0 to 1. Few translations will attain a score of 1 unless they are identical to a reference translation. For this reason, even
  a human translator will not necessarily score 1. It
  is important to note that the more reference translations per sentence there are, the higher the score is. Thus, one must be cautious making even “rough”
  comparisons on evaluations with different numbers
  of reference translations: on a test corpus of about
  500 sentences (40 general news stories), a human
  translator scored 0.3468 against four references and
  scored 0.2571 against two references.

But as their table 1 (providing the numbers compared to two references, H2 is the one mentioned in the text above) shows there is variance among human BLEU scores:

Unfortunately, the paper does not qualify the skill level of the translators.


What is the intuition behind Ridge Regression and Adapting Gradient Descent algorithms? So I was going through Adaptive Gradient Descent, and learning the intuition behind it: optimizing the learning algorithm, and getting the model to converge faster. The way AdaGrad does this, is by dividing the weight vector by the $L_2$ norm. The equation (copied from here), for it, is:

$\theta_{t+1} = \theta_{t} - \dfrac{\eta}{\sqrt{G_t}} \odot g_{t}$

where, if i understand correctly, $\sqrt{G_t}$, is the root of the sum of the square of the gradients, which is the $L_2$ norm. Now if I understand correctly, $g_t$ in this case would be considered the "loss function"

Then I got introduced to Ridge Regression, which basically adds the $L_2$ norm (scaled by a lambda) to the loss function. The intuition behind this is smoothing the loss function so that it doesn't overfit the data. The equation, (reworked from here) is:

$Loss_{L_2} = Loss + \lambda L_2$

So my question, assuming all my understanding so far is correct, is: What is the intuition behind adding as opposed to dividing by the $L_2$ norm?

I understand that you can regularize the loss function in AdaGrad and that they are two separate things, but I still am having trouble wrapping my head around how the math works the way it does. Like why does dividing by the $L_2$ norm make it converge faster? Why doesn't this just smooth the weight vector as well? What is going on here intuitively?

Adagrad is completely unrelated to ridge regression.  There is no reason to expect that there should be any similarity in their methods.

Adagrad divides by the L2 norm of the gradient.  Ridge regression modifies the objective function by adding the L2 norm of the parameters.  So even though it looks like the same thing, it's really not; those are two unrelated values.

There's no principle in mathematics that the L2 norm always has to be added or divided.  It's just a number.  It's like seeing one equation where you multiply by 2 and another equation where you add 2 and asking "why does the second one add 2 rather than multiplying by 2?" -- the answer is "why not? there's no reason to expect them to have to use the number 2 in the same way".


what is the difference between positional vector and attention vector used in transformer model? what is the difference between positional vector and attention vector used in transformer model ? , i saw a video in youtue and the defintion for positional vector was give as :* "vector that gives context based on postion of word in sentence "*
defintion for attention vector was give as "For ever word we can have attention vector generated which captures contextual relationship between words in sentence"
Capturing context information based on distance(postional vector) and attention (attention vector ) sounds same right? or is it different ?

So the question asks between the difference between an attention vector and a positional vector.
To answer this question, will give some context into how the transformer differs from a sequential model, such as RNNs and LSTMs.
In the case of RNNs and LSTMs, data is fed sequentially "one-by-one" into the model to predict the output (whether that is at each time step or after observing the whole sequence. This is irrelevant though in the context of the question.)
In a transformer model, the whole sequence is fed into the model, just like as you would with a conventional neural network. However, the problem is that, unlike with RNNs/LSTMs, there is no way for the transformer model to understand the ordering of the instances in the sequence as the whole sequence is fed into the model. Therefore, we need positional embeddings (positional vector, in your terminology) to add information to the individual instances which tells the model the ordering of the instances in the sequence.
Attention, in the context of transformers, works on the basis that it assigns higher coefficients to instances in the sequences which are most relevant to decoding the hidden representation from the encoder. Unlike a basic encoder-decoder model, with attention, we are able to flexibly assign which input instances in the sequences have the most "say" in predicting the next output instance in the output sequence.
I hope this clarifies some understanding. If not, there is a great article on transformers here: http://www.peterbloem.nl/blog/transformers


Low precision on classification model I am working since some months on a prediction from lead to a sale. Someone makes a lead on my website and I want to predict if this user will make a sale. I have these metrics on the test data.

Now when I check for example for the month May_2020 all the leads I had predicted as 1 (probability >=0.5) and compare with the real sales I receive a conversion rate of 22%. That means, only 22% of all the predicted with 1 lead becomes sale. What do you think?

Is this caused because of low precision? The precision is similar to the conversion rate or is this only luck or the precision implicates the conversion rate.
Any idea of how to reduce this big error of 78%?

Precision gives the percentage of correctly predicted positives out of total positives predicted by the model. Therefore, low conversion rate of your model is due to low precision.
Looking at the support, this looks like a class imbalance problem. 92% of your training examples belong to class '0'.Therefore, while training, your model has more examples of class '0' to learn from.
There are a few ways to deal with this problem:

Over-sampling - This when you randomly draw samples from the minority class with replacement and use it in the training set
Under-sampling - This is when you draw random samples from the majority class with or without replacement and decrease the number of examples of the majority class given to the model while learning.
Synthetic sampling - There are also techniques like SMOTE that generate samples of minority class synthetically.

These are only few techniques I have highlighted. You can check this link for further reading https://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/


Drawing Neural Network diagram for academic papers Is there any tool that one can use to draw neural network architecture diagram for research papers?
Example diagram:

There are few online tools available that give you ability to draw "canned" CNN  diagrams like NN-SVG
EXAMPLE :

Another popular choice seems to be InkScape

There is always PowerPoint


How to predict class label from class probability given by predict_generator for testdata? While using Keras' flow_from_directory method to train my model on a multi-class image classification problem, the predict_generator function gives the class probabilities.

So, my query is how to get the corresponding class-labels for those class probabilities?

You just take the class with the maximum probability. This can be done using numpy argmax function.


Identifying features which drives recommendations I'm want to a recommendation engine for one of the clients to recommend products to his customers. One of the requirement is that recommendations should be explainable. i.e. Why does the recommendation engine recommend certain products to a given customer, based on which features? I'm thinking of using either hybrid models (content based and collaborative filtering) or deep and wide models. However, I'm not sure whether there is any way to identify features which drive recommendations. Are there any other models in which results are more interpretable?

This is original paper by Yehuda Koren et.al. on recommenders for implicit feedback datasets. It implements collaborative filtering method in a novel way for implicit feedback case. Section 5 of this paper gives a method for implementing explanations.

Though you will have to build the implementation yourself from their explanation. To the best of my knowledge no library implementation has explanation part inbuilt.


Technical question: how to build exe from python code? I have a predictive model in IPython notebook. How can I create dll or exe from it? And is it possible to save in executable file already trained model?

I want a program with very simple syntax(X - file with objects, out -- file with answers):

./model X out

I don't know if is appropriate community for this question. If not, I'll delete it.

You'll need to copy, or convert*, your IPython code (I think) into a Python script. After that you can use software such as py2exe to create a standalone application. Other hypothesis are:

cx_freeze
pyInstaller
py2App (Mac OsX)

It's possible to save data inside the scripts (it will be compiled the same way as code as long as it is valid). Other possibility is to save in an external file and just load the data on execution. If the data is in a class check pickle to save it to a file.

as @dmb suggested.

EDIT: Ah I forgot to mention the arguments. That is easily achievable. Check this Question in SO.


Pandas - read CSV with spanish characters I have DataFrame which has column with spanish regions, when I import the .csv as following

df = pd.read_csv('...',delimiter=';',decimal=',')

And print the content of column, I see this: Castilla y Le�n.

I could not find a way how to fix it, but I am assuming there is smart way how to do it.

read_csv has an optional argument called encoding that deals with the way your characters are encoded.

You can give a try to:

df = pandas.read_csv('...', delimiter = ';', decimal = ',', encoding = 'utf-8')

Otherwise, you have to check how your characters are encoded (It is one of them).

You can read the doc of read_csv here


How to add a column for descending row numbers into dataset in R I am new to R and would like to insert a new column that numbers the row to a large dataset.
I have no idea how to use 'mutate()' to insert this. Would appreciate any help.
Thanks.

Use mutate in combination with row_number as follows:
df %>% mutate(row = row_number())


Regression coefficients vs feature_importances_ vs none On looking at various machine learning methods at the scikit-learn, it appears that some modules such as LinearRegression provide coefficients (coef_), others such as AdaBoostRegressor provide feature_importances_ while some e.g. BaggingRegressor and BernoulliNB do not provide either of these.
Why this difference? Are coef_ similar to feature_importances_ to assess the contribution of a variable in prediction? How to assess the feature importance for modules where neither of these is available?

The main difference between Linear Regression and Tree-based methods is that Linear Regression is parametric: it can be writen with a mathematical closed expression depending on some parameters. Therefore, the coefficients are the parameters of the model, and should not be taken as any kind of importances unless the data is normalized.

On the other hand, non-parametric methods have very different ways of measure the importance. In layman terms, measuring the importance of variables on a tree can be done by checking how close they appear to the root node. In ensembling methods, like bagging, one can compute the importance of a variable as the average among the ensemble, like in this stackoverflow answer.

The main difference is, then, the fact that parametric models have, through their parameters, a way of showing the importance of the variables, while non parametric models need some extra work.


@RISK Vs R. Monte Carlo simulation in R? Are there any cheaper or open source alternatives to @RISK or are there packages for R that would be able to perform the same tasks?

http://www.palisade.com/risk/

@RISK (pronounced “at risk”) performs risk analysisusing Monte Carlo simulation to show you many possible outcomes in your spreadsheet model—and tells you how likely they are to occur. It mathematically and objectively computes and tracks many different possible future scenarios, then tells you the probabilities and risks associated with each different one. This means you can judge which risks to take and which ones to avoid, allowing for the best decision making under uncertainty. (Taken from their website)

My main purposes for it would be for the construction company I currently work for:

 Able to provide more accurate models for job costing, e.g. time to construct certain parts, associated costs, etc.

 Provide more accurate models in regard to advertising and marketing, allowing us to better understand and predict what provides the better return.

If this isn't the right forum for this question, please tell me where it needs to be asked.

Thank you,

Monte Carlo simulations are very easy in R. The simplest approach is to write your own scripts that carry out the steps you need for your simulations. To construct these scripts you will need to understand what you are simulating, that is what is the distribution of outcomes, and what are you measuring about those outcomes. If you understand what you are simulating than writing these scripts is easy, but will require programming in R. If you are new to R and/or to programming then it will take some effort to get up to speed. In the end, you will probably understand your data, simulations, and results better. There are packages for Monte Carlo in R, but these are not packages that will do the simulations for you and would require as significant learning of R to be able to use (as much as writing your own scripts).

I suggest reading more about conducting Monte Carlo simulations in R. There are many resources online and offline, such as this book or this tutorial.


k-means classifies one data point as a group I have 1000 sets of one dimensional data (360 each in length), and I want k means to classify what is a small/medium/large value (n_clusters=3) for each set of data, but I'm getting a lot of instances where the large group only has 1 data point because that value is so far away from the rest, but the rest look like they can clearly create 3 clusters.

In some other cases, it does seem to make sense to use 1 data point as the large group since the rest are so close together. It's not clear if there can be 3 distinctive clusters.

What would be an efficient way to deal with this?

Two ideas come to mind, which could be combined or not.

Try to identify the single point as an outlier, and remove it from consideration for the clustering.
Allow $k$ to vary a little.

Using both and allowing $k\in\{2,3\}$ allows you to find only two groups in the main set of points, plus the outlier.  Using just (2) with $k\in\{3,4\}$ could find clusters Low/Med/Large/Outlier...that has the nicety that outlier detection is done by the k-means algorithm rather than another preprocessing step, but runs the risk of finding four honest clusters when you only wanted three.


Classification of DNA Sequences I have a database of 3190 instances of DNA consisting of 60 sequential DNA nucleotide positions classified according to 3 types: EI, IE, Other.

I want to formulate a supervised classifier.

My present approach is to formulate a 2nd order Markov Transition Matrix for each instance and apply the resulting data to a Neural Network.

How best to approach this classification problem, given that the Sequence of the data should be relevant? Is there a better approach than the one I came up with?

One way would be to create 20 features (each feature representing a codon). In this way, you would have a dataset with 3190 instances and 20 categorical features. There is no need to treat the sequence as a Markov chain.

Once the dataset has been featurized as suggested above, any supervised classifier can work well. I would suggest using a gradient boosting machine as it might be better suited to handle categorical features.


Handling a feature containing multiple values I have a dataset in following format:

Movie ID |  Actors | Director | language | ReleaseYear | Genre

1  |    Anil Kapoor;Manisha Koirala;Jackie Shroff;Anupam Kher;Danny Denzongpa;Pran  | Vidhu Vinod Chopra  | hin | 1994  |  Drama;Romance;Patriotic

As you can see the columns Actors and Genre has multiple values. I need to perform cluster analysis on this dataset. I don't know what would the best way to deal with such data. I am thinking of two possible solution(don't know if it is a correct approach for this type of problem)--

Split the column, say, 'Actors' into multiple columns e.g. Actors 1, Actor2,..., and then perform the cluster anylysis.
Row split i.e. convert a single row of data into multiple rows each row for one value of columns like 'Actors' and 'Genre'.

Please suggest me the best way to deal with such data for cluster analysis.

You need to rethink your approach. Rather than "what can I code to make things work", you need to ask "what is the right thing to do, and how can I implement this".

Clustering is hard (easy to run, hard to get good results).

Clustering non-continuous data is even harder.

The reason why people create a lot of false results here is because you have infinitely many way to weight different values and attributes, and essentially you can get almost any result you want just by playing around with parameters. Don't let yourself be lured by the common hack of "one hot encoding" everything; it just shows that people don't (want to) know what they are doing.

To get a reliable result, you need to be very clear on your assumptions, such as "I assume all actors are equally representative, and the overlap as measured by Jaccard is a good indicator of similarity" (I would disagree here, actors should be weighted).
Then you need to do the same thing with genres. Here it is even more questionable hos to compute similarity of genres.
And after that, you need to combine all these different similarities into one. That is probably the hardest step, and will involve even more weighting parameters.

All in all, I'd say: whatever you do, the clustering is not statistically sound. It's too many parameters chosen without even a good reason (yet a proof) that this way is better than another way of choosing them.
In particular, you will never know if there is an even better clustering with different parameters.


How to Get Live Data Updates & Graphing with Tableau Public I'm a big fan of Tableau's easy-to-use data visualization tools. I tried building dashboards with Python's dash package, but it was too complex for me and Tableau offered an easy way to building beautiful data visualizations.

The only downside to Tableau is that it's free version (Tableau Public) does not allow you to connect to a live data source and thus make live graphs and dashboards (only exception is Google Sheets once every 24 hours), for this you'd need to fork out USD 70 a month at least.

Hence my question: Has anyone found a workaround? Specifically, has anyone figured out a way to get live data updates using just Tableau Public?

I wrote the question above, worked on the problem a little longer, and came to the following solution.

Warning: it is anything but ideal, but it works.

Put simply I wrote a small Python script using only one external library - pyautogui - the others are all native (time json requests) which retrieves my data online, updates a local file and executes the actions required to manually update the workbook in Tableau. You just need to create your workbook and the visualization you want and then run the script.

Here's the script:

import pyautogui, time
import json
import requests
def auto_tab (url_name, file_name, workbook_name):
    url = url_name
    r = requests.get(url)
    data = r.json()
    with open(file_name, 'w') as f:
        json.dump(data, f)

        pyautogui.press('win')
    pyautogui.typewrite('Tableau')
    pyautogui.press('enter')
    time.sleep(5)
    pyautogui.hotkey('alt', 'o')
    time.sleep(2)
    pyautogui.typewrite(workbook_name)
    pyautogui.press('enter')
    time.sleep(5)
    pyautogui.press('f5')
    time.sleep(2)
    pyautogui.press('enter')
    time.sleep(5)
    pyautogui.press(['alt', 'space', 'c'])
    time.sleep(2)
    pyautogui.press('enter')
    time.sleep(10)
    return print('done!')

times_to_repeat = 3

while times_to_repeat >= 0:
    auto_tab('your_url', 'your_file', 'your_workbook')
    times_to_repeat -= 1

So, like I said: it's not ideal, but it works.

Downsides:

You need to have your machine running to run the script (perhaps you can run it from a server or a VM)

Upsides:

You can live graph even if you have embedded the graph in html
It's fast
It's free


How to use keras.utils.Sequence with multiple files I am looking at the documentation and this guide

https://www.tensorflow.org/api_docs/python/tf/keras/utils/Sequence

https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly

And I need to write in

def __len__(self):

and

def __getitem__(self, idx):

Which gets the length of the dataset, and the getting a data sample using a particular index. So I am wondering if this is appropriate to use for multiple files, and if so, are there any best practices for this?

The Sequence class from keras works great with multiple files. I've personally created a custom data generator using the Sequence class to load and pre-process multiple files. You should be able to figure it out using this post, I've found it to be very easy to follow the process of creating a custom data generator step by step.


What are the challenging benchmarks for self driving cars object detection beside Kitti I am working on 3D object detection in the context of self driving cars and I was wondering if there are other challenging benchmarks for this beside Kitti.

You may find Apolloscape dataset interesting. It was recently released by Baidu. From the article describing the dataset:

  In this paper, we present a large-scale open dataset,
  ApolloScape, that
  consists of RGB videos and corresponding dense 3D point
  clouds.

Apart from the article,
there have been some blog posts with summarized information on the dataset.


Can feedback loops occur in machine learning that cause the model to become less precise? In discussions about ML algorithms, in for instance crime prediction, it is often claimed by non-experts that there are problems with feedback loops causing the model to become biased and give the wrong results.
Basically saying that the model's predictions give more attention to that type of data, and when retraining with the results, the predictions become skewed so even more attention is given to the same data type, and so on.

Is this true?

I would think that retraining the model with new data would make it more precise, regardless of how that data originated.

Yes, this is a real problem that manifests once system is used by real users.

Most prominent example is News Echo Chamber (accentuated by ML based recommendation systems)

ML algo sees that you like news / videos related to certain point of view, you watch more of such videos and model becomes more convinced of your choice. So it suggests even more content with similar views.

https://en.wikipedia.org/wiki/Echo_chamber_(media)

http://theconversation.com/explainer-how-facebook-has-become-the-worlds-largest-echo-chamber-91024

https://www.theguardian.com/science/blog/2017/dec/04/echo-chambers-are-dangerous-we-must-try-to-break-free-of-our-online-bubbles

https://www.quora.com/Would-you-say-that-Quoras-generated-news-feed-suffers-from-an-echo-chamber-dilemma


What is the difference between data leakage and endogeneity? I have the impression the former is used in ML whereas the latter is used in econometrics. They both carry the idea that information from the target is "leaking" in explanatory variables.

Is there any difference between those two notions?

Endogeneity refers to explanatory variables correlated with error term because of a missing variable or measurement error. Data Leakage is introduction of spurious explainability in the model because of bringing in new data than ground truth like synthetic data (say we used SMOTE). In the former case, we will see that the model is biased too much for some feature coefficients whose explainability seeped into the error term. In the latter case we will see that the model will have high variance and lacks generalization which makes it useless for production/test data. Typical example to introduce data leakage is by SMOTING (yeah using the word as a verb) before you do validation split.


Under what conditions should an autoencoder be chosen over kernel PCA? I've recently been looking at autoencoders and kernel PCA for unsupervised feature extraction.

Lets consider just for a moment linear PCA. Its my understanding that if a autoencoder (with a single hidden layer) has linear activation functions then the sum-of-squares error function (chosen for convenience rather than statistical relevance) will have a unique solution for the weights. This can be easily seen, consider the sum-of-squares error function for the autoencoder input $\boldsymbol{x}$ and the autoencoder output $\hat{\boldsymbol{x}}$, that we wish to solve:

$$
\begin{aligned}
& \min_{\boldsymbol{W},\boldsymbol{V}} {1\over 2N}\sum^N_{n=1} \|\boldsymbol{x}^{(n)} - \hat{\boldsymbol{x}}^{(n)}\|^2\\
\end{aligned}
$$
An autoencoder is a neural network whose input is its output. Defining the following: Let $\boldsymbol{W}$ be the weights matrix between the input layer and the hidden layer and $\boldsymbol{V}$ be the weights matrix between the hidden layer and the output layer. I'm ignoring the bias terms for now. Let $\psi$ be the activation function for each unit on the hidden layer and let $\phi$ be the activation function for each unit on the output layer. Then we have two mappings in an autoencoder:

$$
\boldsymbol{z} = \psi\left(\boldsymbol{W}\boldsymbol{x}\right), \:\:\:\:\:\:\: \hat{\boldsymbol{x}} = \phi\left(\boldsymbol{V}\boldsymbol{z}\right)
$$

$$
\begin{aligned}
& \min_{\boldsymbol{W},\boldsymbol{V}} {1\over 2N}\sum^N_{n=1} \|\boldsymbol{x}^{(n)} - \phi\left(\boldsymbol{V}\psi\left(\boldsymbol{W}\boldsymbol{x}^{(n)}\right)\right)\|^2\\
\end{aligned}
\label{eqn: 1}
$$
If $\psi$ and $\phi$ are linear, i.e  $\phi(\boldsymbol{s})=\boldsymbol{s}$ and $\psi(\boldsymbol{s})=\boldsymbol{s}$, then $\hat{\boldsymbol{x}} = \boldsymbol{V}\boldsymbol{W} \boldsymbol{x} = \boldsymbol{A} \boldsymbol{x}$ (i.e rotation) and
$$
\begin{aligned}
& \min_{\boldsymbol{W},\boldsymbol{V}} {1\over 2N}\sum^N_{n=1} \|\boldsymbol{x}^{(n)} - \boldsymbol{V}\boldsymbol{W}\boldsymbol{x}^{(n)}\|^2\\
& \min_{\boldsymbol{W},\boldsymbol{V}} {1\over 2N}\sum^N_{n=1} \|\boldsymbol{x}^{(n)} - \boldsymbol{A}\boldsymbol{x}^{(n)}\|^2\\
\end{aligned}
$$

This is recognised as principle component analysis. This solution for the weights "forms a basis set which spans the [linear] principle component space" (Bishop, Pattern Recognition and Machine Learning). Bishop then goes on to say that these weight vectors aren't necessarily orthogonal or normalised. I assume that this is because we have not proposed any constraints on $\boldsymbol{A}$ as we would have done in PCA by way of proposing an orthogonal basis?

So am I right in thinking that the autoencoder I've described above does not in general give the linear PCA solution, but in some very special cases they may match?

Now bishop goes on to say that the only way to handle non-linear inputs is to add more hidden layers to the autoencoder and have some layers use non-linear activation functions. However one would assume that the same problem as above persists? i.e. that one is not guaranteed to get the same solution as PCA.

There seem to be additional issues [Bishop]:

Training requires solving a non-linear optimisation problem.
One may end up with a solution that corresponds to a local minima, not a global one.
One must specify the number of subspace dimensions as part of the network architecture.

Therefore my question is, why use autoencoders at all for unsupervised feature extraction? Why not just use kernal PCA? Which will reduce to linear PCA when the data permits it. Is there some other advantage that I'm missing? Under what conditions should an autoencoder be chosen over kernel PCA?

I think you can interpret autoencoders as essentially performing kernel PCA, but with an extremely complex kernel. Like, really, stupidly complex.

The kinds of things you can accomplish with autoencoders are spectacular. For example, consider BicycleGAN.

If kernel PCA is a step above PCA, autoencoders are miles away. If you have an application where PCA makes sense but isn't working out, try kernel PCA. If you have an application where condensing your input into an embedding of some kind would be valuable but PCA is unlikely to be very helpful, use an autoencoder.