In Weka, how to draw learning curve evaluated on both test and training set? This is just for finding overfitting gap.
After initial research, I can only find method to draw learning curve using evaluation of test set.  However, I could not evaluate on training set and over the two learning curves.

This is only possible with KnowledgeFlow. In WekaManual.pdf (which is included in Weka package) for version 3.7.12 there is an example in Chapter 7.4.2 "Plotting multiple ROC curves" with picture and step-by-step instructions. For other Weka versions it is the same, just find the appropriate chapter.

To give an impression on how it goes, I extracted the picture from the manual. It will draw two curves for two classifiers. For your question it is very similar. You use one classifier and then connect trainingSet to one ClassifierPerformanceEvaluator and testSet to another.


How to train a machine learning model if there is a relationship between two different data points? How to train a machine learning model if there is a relationship between two different data points? I mean not relationship between two variables among the point but that between two different data examples or points, so that i want to classify similar such set of datapoints into one single class . Which algorithm is most suited in this case . I want to write the algorithm on my own.

This is a very broad question and really depends on the type of data you have and how it is distributed.

There are many types of classifiers that you can use.
This link, from scikit-learn shows you a comparison between many algorithms that could help you choose which one to pick.

I understand you want to write the algorithm yourself, but I would recommend you looking at scikit-learn.org and trying out different algorithms. Once you have tried and wanted to implement yourself, you can have a look at blogs, like this one, which explains in details the intrinsic works of models, like k-nearest-neighbours for example.


Pytorch Luong global attention: what is the shape of the alignment vector supposed to be? I am looking at the Luong paper on Attention models and global attention. I understand how the alignment vector is computed from a dot product of the encoder hidden state and the decoder hidden state. So that all makes sense.
My question is, what should the dimensions of the alignment score tensor be? If I have my data in batches, I basically compute a single score for each timestep in the hidden state, right. So should the alignment vector be of dimension [sequence length, 1], or something like that? I would then softmax this alignment vector and multiply it by every element in the batch to compute the context vector, right.
Again, my key question is what the dimensions of the alignment score vector or tensor should be. Thanks.

You answered yourself [sequence length, 1] is correct assuming you work with a single sentence. (Or actually, the 1 dimension depends on implementation.)
In practice, the data is typically batched, so it will be [batch, sequence length 1]. This can be element-wise multiplied with the encoder states of dimension [batch, sequence length, hidden size] and summed in the middle dimension to get a context vector of dimension [batch, hidden_size].
Note also that if the sentences do not have the same length, you need to deal with padding, i.e., set the padded positions to -inf before softmax.


What's the difference between finding the average Euclidean distance and using inertia_ in KMeans in sklearn? I've found two different approaches online when using the Elbow Method to determine the optimal number of clusters for K-Means.

One approach is to use the following code:

distortions_2.append(sum(np.min(cdist(data,
                                      kmeanModel.cluster_centers_,
                                      'euclidean'),
                                axis = 1)) / data.shape[0])

Another is to use inertia_ from sklearn.cluster.KMeans:

distortions_3.append(kmeanModel.inertia_)

When I plot the results (using the same random states) both give different results but I'm not sure what the differences are, can anyone help?

Edit: If I replace the normalisation factor / data.shape[0] with squared **2 as suggested below, then I still don't get the same as for the inertia plot:

distortions_2.append(sum(np.min(cdist(data,
                                      kmeanModel.cluster_centers_,
                                      'euclidean'),
                                axis = 1)) ** 2)

Using squared just makes the plot a little smoother, but definitely not the same as using intertia_, I'm just not quite sure how inertia_ is calculated and what it means.

You will notice that the inertia is the sum of squared distance between each point and its nearest cluster center. Thus by removing the normalization term / data.shape[0] and instead replacing that by a squared **2 these two expressions will be equivalent

distortions_2.append(sum(np.min(cdist(data
                              , kmeanModel.cluster_centers_
                              , 'euclidean')
                              , axis = 1)) **2)

and

distortions_2.append(kmeanModel.inertia_)

For example using this artificial data source, where we generate 5 Gaussian distributions with 300 points in each.

params = [[[ 0,1],  [ 0,1]],
          [[ 5,1],  [ 5,1]],
          [[-2,5],  [ 2,5]],
          [[ 2,1],  [ 2,1]],
          [[-5,1],  [-5,1]]]

n = 300
dims = len(params[0])

data = []
y = []
for ix, i in enumerate(params):
    inst = np.random.randn(n, dims)
    for dim in range(dims):
        inst[:,dim] = params[ix][dim][0]+params[ix][dim][1]*inst[:,dim]
        label = ix + np.zeros(n)

    if len(data) == 0: data = inst
    else: data = np.append( data, inst, axis= 0)
    if len(y) == 0: y = label
    else: y = np.append(y, label)

num_clusters = len(params)

We will then apply KMeans using different number of clusters

k = [1,2,3,4,5,6,7,8,9,10]

inertias = []
dists = []

for i in k:
    kmeans = KMeans(i)
    kmeans.fit(data)
    inertias.append(kmeans.inertia_)
    dists.append(sum(np.min(spatial.distance.cdist(data, kmeans.cluster_centers_, 'euclidean'), axis=1)**2))

plt.plot(range(1, len(inertias)+1), inertias, label = 'Inertia')
plt.plot(range(1, len(dists)+1), dists, label = 'Distance')
plt.legend()
plt.show()


What are the reasons for drawing initial neural network weights from the Gaussian distribution? Are there theoretical or empirical reasons for drawing initial weights of a multilayer perceptron from a Gaussian rather than from, say, a Cauchy distribution?

The goal of initial weight distribution is get some amount of variance to allow learning to take place. Many distributions could work. Gaussian or uniform are the most commonly used.
Cauchy distribution would not be a useful choice for several reasons. The primary reason is the Cauchy distribution is not as common as Gaussian or uniform distribution, thus is not as well supported in programming environments. Another reason could be Cauchy distribution's variance is undefined.


How to extract features from the encoded layer of an autoencoder? I have done some research on autoencoders, and I have come to understand that they can also be used for feature extraction (see this question on this site as an example). Most of the examples out there seem to focus on autoencoders applied to image data, but I would like to apply them to a more general data set.

Therefore, I have implemented an autoencoder using the keras framework in Python. For simplicity, and to test my program, I have tested it against the Iris Data Set, telling it to compress my original data from 4 features down to 2, to see how it would behave.

The encoder seems to be doing its job in compressing the data (the output of the encoder layer does indeed show only two columns). However, the values of these two columns do not appear in the original dataset, which makes me think that the autoencoder is doing something in the background, selecting/combining the features in order to get to the compressed representation.

Here is the complete working example:

from pandas import read_csv
from numpy.random import seed
from sklearn.model_selection import train_test_split
from keras.layers import Input, Dense
from keras.models import Model

# Get input data and separate features from labels
df = read_csv("iris.data")
Y = df.iloc[:,4]
X = df.iloc[:, : 4]

# Split data set in train and test data
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, train_size=0.5, random_state=seed(1234))

# Input information
col_num = X.shape[1]
input_dim = Input(shape=(col_num,))

# Encoding information
encoding_dim = 2
encoded = Dense(encoding_dim, activation='relu')(input_dim)
# Decoding information
decoded = Dense(col_num, activation='sigmoid')(encoded)
# Autoencoder information (encoder + decoder)
autoencoder = Model(input=input_dim, output=decoded)

# Train the autoencoder
autoencoder.compile(optimizer='adadelta', loss='mean_squared_error')
autoencoder.fit(X_train, X_train, nb_epoch=50, batch_size=100, shuffle=True, validation_data=(X_test, X_test))

# Encoder information for feature extraction
encoder = Model(input=input_dim, output=encoded)
encoded_input = Input(shape=(encoding_dim,))
encoded_output = encoder.predict(X_test)

# Show the encoded values
print(encoded_output[:5])

This is the output from this example:

[[ 0.28065908  6.151131  ]
 [ 0.8104178   5.042427  ]
 [-0.          6.4602194 ]
 [ 3.0278277   2.7351477 ]
 [ 0.06134868  5.064625  ]]

Basically, my idea was to use the autoencoder to extract the most relevant features from the original data set. However, so far I have only managed to get the autoencoder to compress the data, without really understanding what the most important features are though.

My question is therefore this: is there any way to understand which features are being considered by the autoencoder to compress the data, and how exactly they are used to get to the 2-column compressed representation?

You are using a dense neural network layer to do encoding. This layer does a linear combination of the input layers + specified non-linearity operation on the input.

Important to note that auto-encoders can be used for feature extraction and not feature selection. It will take information represented in the original space and transform it to another space. The compression happens because there's some redundancy in the input representation for this specific task, the transformation removes that redundancy. Original features are lost, you have features in the new space.

Which input features are being used by the encoder? Answer is all of them. For how exactly are they used? Answer is you can check the weights assigned by the neural network for the input to Dense layer transformation to give you some idea. You can probably build some intuition based on the weights assigned (example: output feature 1 is built by giving high weight to input feature 2 & 3. So encoder combined feature 2 and 3 into single feature) . But there's a non-linearity (ReLu) involved so there's no simple linear combination of inputs.

If your aim is to get qualitative understanding of how features can be combined, you can use a simpler method like Principal Component Analysis. The factor loadings given in PCA method's output tell you how the input features are combined. If the aim is to find most efficient feature transformation for accuracy, neural network based encoder is useful. But you loose interpretability of the feature extraction/transformation somewhat.


How to get a confidence score for predictions? In a regression problem, is it possible to calculate a confidence/reliability score for a certain prediction given models like XGBoost or Neural Networks?

No matter the model, you can always use the non-parametric bootstrap to construct a confidence interval for any parameter, including predictions (which are actually random variables themselves but are reported as expectations). Here's the general procedure:

Let $N$ denote the number of observations in your training data $X$, and $x_j$ denote the specific observation whose prediction, $\hat{y}_j$, you want a CI for.
Let $K$ denote some number of resampling iterations (Must be $\ge 20$ for a CI with coverage $\ge 95\%$)
For $i$ in $K$, draw a $N$ random samples from $X$ with replacement. Denote this $X_i^{*}$
Train a model on $X_i^{*}$ and use this model to form a prediction on $x_j$. Call this $\hat{y}^{*}_{ji}$
Estimate distributional parameters for $\hat{y}_j$ from your sample. A $100 -
 \alpha$ CI is given by the $\frac{\alpha}{2}$ and $100 - \frac{\alpha}{2}$ percentiles of $\hat{y}^{*}_{j}$.


Keras - error when adding layers to loaded model I want to use ResNet50 as a feature extractor. For this purpose, I have loaded the pre-trained model, deleted a few layers and added my layers to the model. For adding my layers, I have used the Sequential API. The code is the following:

resnet_model = ResNet50(weights='imagenet')

# Delete layers
for i in range(12):
  resnet_model.layers.pop()

# Fix weights
for layer in resnet_model.layers:
  layer.trainable = False

base_model = Model(inputs=resnet_model.inputs, outputs=resnet_model.layers[-1].output)

model = Sequential()
model.add(base_model)

model.add(Conv2D(2048, kernel_size=(3, 3), input_shape=(7, 7, 2048)))
model.add(BatchNormalization(momentum=bn_momentum))
model.add(Activation('relu'))

model.add(Conv2D(2048, kernel_size=(3, 3)))
model.add(BatchNormalization(momentum=bn_momentum))
model.add(Activation('relu'))

model.add(Flatten())
model.add(Dense(1000, activation='relu'))
model.add(Dense(200, activation='softmax'))

This code works. However, when I run print(model.summary()) I obtain the following:

Model: "sequential_26"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
model_16 (Model)             (None, 7, 7, 2048)        19115904
_________________________________________________________________
conv2d_31 (Conv2D)           (None, 5, 5, 2048)        37750784
_________________________________________________________________
batch_normalization_23 (Batc (None, 5, 5, 2048)        8192
_________________________________________________________________
activation_917 (Activation)  (None, 5, 5, 2048)        0
_________________________________________________________________
conv2d_32 (Conv2D)           (None, 3, 3, 2048)        37750784
_________________________________________________________________
batch_normalization_24 (Batc (None, 3, 3, 2048)        8192
_________________________________________________________________
activation_918 (Activation)  (None, 3, 3, 2048)        0
_________________________________________________________________
flatten_10 (Flatten)         (None, 18432)             0
_________________________________________________________________
dense_17 (Dense)             (None, 1000)              18433000
_________________________________________________________________
dense_18 (Dense)             (None, 200)               200200
=================================================================
Total params: 113,267,056
Trainable params: 94,142,960
Non-trainable params: 19,124,096
_________________________________________________________________
None

As you can see, the conv layer adds 37750784 new parameters to the network!!! Obviously, this isn't right, but I can't see why this happens...

Actually, the number of parameters for the second layer is correct, the number of parameters in a convolutional layer is calculated as follows:

n_parameters = n_filters_l * (n_filters_l-1 * (kernel_size_h * kernel_size_w) + 1)
             = 2048 * (2048 * (3 * 3) + 1) = 37750784

If you want to decrease the number of parameters, you therefore have multiple options:

Decrease the number of filters in the layer before it
Decrease the number of filters in the layer itself
Use a lower kernel height of width


How to correctly resize input images in a CNN? I have a training set composed of images having different width and height, I need to resize them to a fix dimension nxn or nxm before passing them as the input of a CNN.

I would like to know which parameters I need to take into account for correctly choose the new scaled dimensions n and m.

Two solutions:

Crop images to a fixed size
Pad images to a fixed size with 0s.

Keep in mind if you are doing segmentation, gt masks have to be cropped/padded too.


Interpreting confusion matrix and validation results in convolutional networks I need some help in the assessment of the training results of a convolutional neural network. Here is my setup:

  Architecture: InceptionV3
  Pre-trained InceptionV3 with weights from image net
  replaced last layer, retrained the second half of the network
  Classification: Softmax with cross entropy ploss
  Adam optimizer with keras default parameters
  Hyper parameters: Learning rate 0.0001, batch size 64
  Goal: Multi-label classification with exclusive labels
  Data: 17 classes ranging from 0-16 with 48.600 images in total
  Train, Test, Validation: 80%, 10%, 10%
  Training: 2 epochs training only last layer + 2 epochs training second half of network

Results

Based on my validation set I created a confusion matrix with the following results.

Here are also some more per-class performance metrics:

As you can see, performance is not very good. The test-accuracy was at 62.2% so it was also not very good. Looking at the confusion matrix it is noticeable that the classes 0, 2, 3, 7, 10, 11, 13 and 14 are never predicted. These classes have also the smallest number of samples. I wonder why this happens? Those classes are more rare than the others but nver predicting them seems odd. What would you recommend going forward?

These are my recommendations:

Try to train your model more, with different learning rates. Your current learning rate is so much small. At first steps put it to a big value then reduce it smoothly.
Try to change the architecture. If your task does not have the same scale as image-net surely you would need a network with simpler network architecture, consequently the number of parameters diminishes.
Changing the distribution of data is dangerous, but you can augment your entire data-set by some extent to let your network see different and more examples.
For those classes that have not been learnt you can set the class-weight to a bigger number to construct a cost function which cares more about those classes.
Because your data-set is not balanced, use other metrics to evaluate your model, like F1 score.


Decision Stumps with same value leaf nodes I'm doing some ADA boosting with Decision stumps and in inducing a binary classifying decision stump, i'm finding both leaf nodes to have a positive value. Can this be the case? Is this possible?

What is the overall response rate? If it's low (even 15-20%) it may be difficult to find decision stumps that contain one leaf with > 50% response!

You could consider oversampling or changing cutoff probability, but I think if your using only 2 leaf trees, your model is bound to struggle.


How to start prediction from dataset? I have event dataset in a factless table. It has list of events

timestamp-> event name -> node ( In network)

There is always a sequence of event happening. So how do I start predicting future event based on past one and discover list of nodes that will be affected from past experience.

I am programmer without knowledge of machine learning. I have installed spark,R and have dataset in oracle database.Is there any tutorial/algorithm that I can use to get started. I taught myself scala/R but have no idea on getting started. My dataset are huge i.e. more than 9billion rows for 3 months.

Node            Eventtime       alarmname
192.168.1.112   6/14/2016 19:41 Main power supply has a fault alarm
192.168.1.113   6/14/2016 19:23 Association path broken
192.168.1.113   6/14/2016 19:23 NA
192.168.1.113   6/14/2016 19:23 Association broken
192.168.1.112   6/14/2016 19:23 Mains Failure
192.168.1.112   6/14/2016 19:23 Mains Failure

Additional Information:

I have 98 nodes. I would like to predict:

i. No of node that has alarm or goes down when a single node goes down
e.g. if node A has alarm if in 1 month list of node that has alarm at the same period

ii. The sequence of event occurrence i.e. if one node has mains failure then the next event would be node down.

The problem you are facing is a time series problem.
Your events are categorial which is a specific case (so most common techniques like arima and Fourier transform are irrelevant).

Before getting into the analysis, try to find out whether the events among nodes are independent. If they are independent, you can break them into sequences per node and analyze them. If they are not (e.g., "Main power supply has a fault alarm" on node x indicates the same event on node y) you should the combined sequence. Sometime even when the the sequence are dependent you can gain from using the per node sequence as extra data.

You data set is quite large, which means that computation will take time. You data is probably noisy, so you will probably have some mistakes. Therefore, I recommend advancing in small steps from simple models to more complex ones.

Start with descriptive statistics, just to explore the data. How many events do you have? How common are they? What is the probability of the events that you try to predict? Can you remove some of the events as meaningless (e.g., by using domain knowledge)?

In case you have domain knowledge that indicates that recent events are the important ones, I would have try predicting based on the n last events. Start with 1 and grow slowly since the number of combinations will grow very fast and the number of samples you will have for each combination will become small and might introduce errors.

Incase that the important event are not recent, try to condition on these events in the past.

In most cases such simple model will help you get a bit above the baseline but not too much. Then you will need more complex models. I recommend using association rules that fit your case and have plenty of implementations.

You can further advance more but try these technique first.

The techniques mentioned before will give you a model the will predict the probability that a node will be down, answering your question (ii). Running it on the sequence of the nodes will enable you to predict the number of nodes that will fail answering question (i).


Is over fitting okay if test accuracy is high enough? I am trying to build a binary classifier. I have tried deep neural networks with various different structures and parameters and I was not able to get anything better than

Train set accuracy : 0.70102
Test set accuracy : 0.70001

Then I tried machine learning algorithms such as KNN and Decision Trees etc. And I found that Random forest Classifier from Scikit-learn with n_estimators=100 gave me

Train set accuracy : 1.0
Test set accuracy : 0.924068

I tried adjusting other parameters such as max_depth, criterion But the decrease in training set accuracy also caused the test set accuracy to drop. Like

Train set accuracy : 0.82002
Test set accuracy : 0.75222

My question is, is this

Train set accuracy : 1.0
Test set accuracy : 0.924068

acceptable ? Even thought the model is over fitting, the test set accuracy is better.

If you properly isolate your test set such that it doesn't affect training, you should only look at the test set accuracy. Here are some of my remarks:

Having your model being really good on the train set is not a bad thing in itself. On the contrary, if the test accuracy is identical, you want to pick the model with the better train accuracy.
You want to look at the test accuracy. That is your primary concern. So pick the model that provides the best performance on the test set.

Overfitting is not when your train accuracy is really high (or even 100%). It is when your train accuracy is high and your test accuracy is low.
it is not abnormal that your train accuracy is higher than your test accuracy. After all, your model has an advantage with the train set since it's been given the correct answer already.

At the end of the day, training a machine learning model is like studying for a test. You (the model) use learning resources such as books, past exams, flash cards etc. (train set) to perform well on a test/exam (test set). Knowing your learning resources perfectly doesn't mean you are overfitting. You would be overfitting if this is all you knew and couldn't perform well on the exam at all.


Illustrating the dimensionality reduction done by a classification or regression model Tl;DR: You can predict something, but how do you explain the prediction?

Your usual classification/regression setup
Lets say the data is a classic regression/classification problem: several numerical columns, several nominal columns, and an event which we are trying to predict:

user1, age:18, wealth:20000, likes:tomatoes, isInBigCity:yes, hasClicked:yes
user2, age:25, wealth:24000, likes:carrots , isInBigCity:no , hasClicked:no
...

With the help of Random Forests, SVM, Logistic Regression, Deep Neural Network, or some other method we export a model that can output a probability of the event hasClicked:yes for a new user faced with the choice of clicking.

Extracting the inner topology surfaced by a model
Now, those algorithm do some dimensionality reduction, reducing those inputs to a single probability. My question is: how would you extract what those models are doing and show the dimensionality reduction steps to a human? How would you illustrate the inner topology of the dataset with regards to the predicted class?
I am looking for either:

Visualizations of what a model produced by Random Forests, SVM, Logistic Regression, Deep Neural Network is doing.

Clusterers being extracted from regression/classification models (Surely a single decision tree can be viewed as a hierarchical clusterer)

A model-specific way to project the input data in a space where the Euclidian distance of T-SNE makes sense.

A way to learn a T-SNE-compatible distance out of a regression/classification model.

Clustering methods that optimise the separation of one variable while not using it to cluster.

Clusterers built out of regression/classification models

The goal is to extract some sort market segmentation based on the behaviour of users. And give a high level visualization of it. Something that would expose clearly the reasons why some users transform better than others.
EDIT: Let's reduce the question to one estimator: Random Forests; what would be your answer?

(note: this answer is mid-edit)
There are a number of Machine Learner explainers and diagnostics.
Disclaimers:  (these should increase over time)

I'm not making it exactly reproducible because it would be 2x as long, and its working on being book-like anyway.
This is more about showing the method than going into crazy details.  If you want a deep dive into a nuance, that is a different question.

Let's set up a sample problem.
Mnist is a fair dataset, so let’s first use a random forest to describe it, and then vivisect the learner to understand what, why, and how it works.
Here is my preferred "startup" because if I don't have it, it makes sure I get it.
#list of packages
listOfPackages <- c('dplyr',      #data munging
                    'ggplot2',    #nice plotting
                    'data.table',  #fast read/write
                    'matrixStats', #column standard deviances
                    'h2o',         #decent ML toolbox
                    'keras')       #has mnist data

#if not installed, then install and import
for (i in 1:length(listOfPackages)){
  if(!listOfPackages[i] %in% installed.packages(fields = "Package")){
    install.packages(listOfPackages[i] , dependencies = TRUE)
  }
  require(package = listOfPackages[i], character.only = T, quietly = T)
}
rm(i, listOfPackages)

Here is code for reading mnist using keras:
library(keras)
mnist <- dataset_mnist()
x_train <- mnist$train$x
y_train <- mnist$train$y
x_test <- mnist$test$x
y_test <- mnist$test$y

One has to do some housekeeping:
# Redefine  dimension of train/test inputs
x_train <- array_reshape(x_train, c(nrow(x_train), img_rows*img_cols))
x_test <- array_reshape(x_test, c(nrow(x_test), img_rows*img_cols))

df_train <- data.frame(y=y_train, x_train)
df_test  <- data.frame(y=y_test , x_test)

here is how to process it with a random forest using h2o.ai, assuming its already installed:
# Input image dimensions
img_rows <- 28
img_cols <- 28

#spin up h2o
h2o.init(nthreads = -1)

#move data to h2o
train.hex <- as.h2o(df_train, "train.hex")
test.hex <- as.h2o(df_test, "test.hex")

#prep for random forest
x <- 1:ncol(x_train)
y <- 1
x <- x[-y]

#spin up random forest
myrf <- h2o.randomForest(x, y,
                         training_frame = train.hex,
                         validation_frame = test.hex,
                         ntrees = 150, model_id = "myrf.hex")

Here is how it did.
Confusion matrices:

Here are the train/valid metrics

So what?  So what now?
We have a decent model, and it is crudely compatible with (this) benchmark that says there are things that have less error than it.  Where does it go wrong?
There are about 320 misclassifications in the test dataset, and it is beyond scope to go into each and every one of them.  It looks to be worst at 9, 8, 2, and 3.  Let's look at 8 and 3.
There is a warning given by h2o.ai:
Warning message:
In .h2o.processResponseWarnings(res) :
  Dropping bad and constant columns: [X701, X702, X309, X672, X61, X673, X60, X674, X63, X62, X65, X64, X66, X393, X778, X779, X781, X782, X420, X783, X421, X700, X78, X79, X141, X142, X780, X448, X449, X725, X726, X727, X728, X729, X81, X80, X83, X82, X85, X730, X84, X698, X731, X87, X699, X732, X86, X337, X88, X559, X561, X169, X10, X12, X11, X14, X13, X281, X16, X15, X18, X17, X560, X19, X504, X505, X111, X112, X113, X476, X477, X753, X21, X754, X20, X755, X23, X22, X25, X24, X27, X26, X29, X28, X197, X616, X617, X30, X225, X588, X32, X589, X31, X34, X33, X36, X35, X38, X37, X39, X646, X767, X768, X769, X253, X770, X771, X772, X773, X41, X532, X774, X40, X533, X775, X43, X776, X42, X777, X45, X1, X44, X2, X47, X3, X46, X4, X49, X5, X48, X6, X7, X8, X9, X756, X757, X758, X759, X760, X50, X365, X761, X366, X762, X52, X367, X763, X51, X764, X54, X644, X765, X53, X645, X766, X56, X55, X58, X57, X59].

Sherlock Holmes says (approximately):
Once you eliminate the impossible, whatever remains, no matter how improbable, must [contain] the truth.
Let's remove the impossible from our pixels.
bad_cols <- which(colSds(x_train)==0)
str(bad_cols)

There are 161 of the 784 pixel (columns), or about 21%, that have no information.
Let's see where the RF thinks most of the importance lives.
#pop importance
myimp <- h2o.varimp(myrf)

#sort by relative importance
myimp <- myimp %>% arrange(-relative_importance)
myimp$index <- 1:nrow(myimp)

#plot importances
ggplot(myimp,aes(x=log10(index), y=log10(scaled_importance))) +
  geom_point()

yields this as a plot:

This tells us that, unsurprisingly enough, the majority of the importance is in just a few pixel intensity values.  I like to think of Paretto, the 80/20 rule, Prices law, or Lotka's law.  It also shows that around $10^{2.5}$ steps in, the relationship falls-down, and stops behaving consistent to a single Lotka-style rule.  There could be a "transition in 'physics'" but it is more likely that it is a "cliff" where the data stops living.
Next I want to look at the "kings".  In particular, I want to know how many elements are required to get the same.  I could brute force it, but I don't want to.  When I look at the plot, I see clusters with jumps, and I use those to determine test-sizes to consider.  I see transitions at index values of 10 and 19.
Let's subset and then start with LIME.
(note: this answer is mid-edit)


Understanding LSTM Training and Validation Graph and their metrics (LSTM Keras) I have trained a RNN/LSTM model. I would like to interpret my model results, after plotting the graph for Loss and accuracy (b/w training and Validation data set).

My objective is to classify the labels (either 0 or 1) if i provide only a partial input to the model. In such a way I have performed training.

Train_Validate_Test_Split

Train 80% ; Validate 10 % ; Test 10%

X_train_shape : (243, 100, 5)
Y_train_shape : (243,)

X_validate_shape : (31, 100, 5)
Y_validate_shape : (31,)

X_test_shape : (28, 100, 5)
Y_test_shape : (28,)

Model Summary

Model Graph

Model Metrics

Question or Interpretation from the model results

  Q 1 : What can I understand/interpret from loss and Accuracy graph ?
  How can i confirm whether the model trained properly for my data set
  or not ?

  Q 2 : Whether oscillations in both loss and accuracy, have some effect in >model training ? (Or it is a normal behavior) If not, how
  can I regularize my model without oscillations ?

  Q 3 : What can I interpret or understand from my metrics tabular column ? My > Y_test accuracy is more when compared with Train &
  Validation accuracy, What can i interpret from this behavior ?

From visually inspecting the graph, we see that the validation loss and accuracy has improved with each epoch - with the training loss and accuracy higher than that of validation. This indicates that accuracy has improved with training.
As suggested in another post, one potential solution is to calculate the exponential moving average of the validation loss to remove the oscillations and better determine the improvement in this metric.
If you are finding that the test accuracy is higher than that of training, this might suggest underfitting. This could imply that more training on your model is required, or has been over-regularized.


How to populate pandas series w/ values from another df? I need help figuring out how to populate a series of one dataframe w/ specific values from another dataframe.

Here's a sample of what I'm working with:

df1 = pd.DataFrame({'Year':[1910, 1911, 1912],
                    'CA':[2.406, 2.534, 2.668],
                    'HI':[0.804, 0.821, 0.832]})

df2 = pd.DataFrame({'State':['CA', 'CA', 'CA', 'HI', 'HI'],
                    'Year':[1910, 1910, 1911, 1911, 1911]})

df2['Population'] = pd.Series()

*I'm trying to populate df2['Population'] w/ the corresponding populations from df1 (i.e. the population of a specific state from a specific year)

How can I do this?

Here is a one solution:

df2['Population'] = df2.apply(lambda x: df1.loc[x['Year'] == df1['Year'], x['State']].reset_index(drop=True), axis=1)

The idea is for each row of df2 we use the Year column to tell us which row of df1 to access, and then State to select the column.  Afterwards we reset the index of the result to prevent pandas from keeping the columns separate.


bias variance decomposition for classification problem It is given that:

MSE = bias$^2$ + variance

I can see the mathematical relationship between MSE, bias, and variance. However, how do we understand the mathematical intuition of bias and variance for classification problems (we can't have MSE for classification tasks)?

I would like some help with the intuition and in understanding the mathematical basis for bias and variance for classification problems.

Any formula or derivation would be helpful.

My opinion is that the bias variance trade off is rooted in the Uncertainty principle. It behaves absolutely the same.


Keras input shape error I am trying to load a Keras model and make predictions with it and run into a strange error. An minimal example is the following:

from keras import models
import numpy as np

model = models.load_model('model_4hiddenLayers_16unitsPerLayer_relu_learningRate0p0001.h5')
x = np.ones(36, dtype=float)
prediction = model.predict(x )

The model expects an input shape of (36,), which should be the shape of x, which I verified:

print('x.shape={}'.format(x.shape) )

gives :

  x.shape=(36,)

However when running this code I get the following error message:

  ValueError: Error when checking : expected batch_normalization_1_input to have shape (36,) but got array with shape (1,)

What am I missing here? Thanks for the help.

Some additional info : I am using keras version 2.1.4 with TensorFlow as backend.

I figured out the issue. The "predict" function expects a batch of input arrays, so it expects x to have shape (n, 36) where n is the number of examples. After adding :

x = x.reshape( (1,36) )

the code works fine


nnet in caret. Bootstrapping or cross-validation? I want to train shallow neural network with one hidden layer using nnet in caret. In trainControl, I used method = "cv" to perform 3-fold cross-validation. The snipped the code and results summary are below.
  myControl <- trainControl(## 3-fold CV
    method = "cv",
    number = 3)

  nnGrid <-  expand.grid(size = seq(1, 10, 3),
                         decay = c(0, 0.2, 0.4))

  set.seed(1234)

  nnetFit <- train(choice ~ .,
                   data = db,
                   method = "nnet",
                   maxit = 1000,
                   tuneGrid = nnGrid,
                   trainControl = myControl)

I have few doubts -

The results (attached below) suggests that it performed bootstrapping (25 reps) not cross validation (I was accepting model to be trained only three times for 3-cv for each set of hyperparameter).

I just want to 100% sure on whether model used the original data for training without any pre-processing such as centering and scaling.

I used verboseIter = FALSE in trainControl, but it still prints all the iterations

Are other libraries such as neuralnet, mxnet better than nnet and I can replace them here.

I wan to be sure whether nnet use sigmoid activation function in hidden layer.

Can someone please advise?
No pre-processing
Resampling: Bootstrapped (25 reps)
Summary of sample sizes: 3492, 3492, 3492, 3492, 3492, 3492, ...
Resampling results across tuning parameters:

  size  decay  Accuracy   Kappa
   1    0.0    0.4947424  -0.002382083
   1    0.2    0.5686601   0.141749447
   1    0.4    0.5711497   0.143637446
   4    0.0    0.5076199   0.022765002
   4    0.2    0.7333516   0.468625768
   4    0.4    0.7253675   0.452584882
   7    0.0    0.5002912   0.006079340
   7    0.2    0.7440360   0.488933678
   7    0.4    0.7676500   0.536547080
  10    0.0    0.5064281   0.013648966
  10    0.2    0.7668795   0.535370693
  10    0.4    0.7566465   0.513652332

Accuracy was used to select the optimal model using the largest value.
The final values used for the model were size = 7 and decay = 0.4.

There is a mistake in your train function.  It should be trControl
not trainControl.

By default the train function doesn't do any pre-processing
(preProcess = NULL)

Have trace = FALSE in the train function to stop printing the
trace to the screen.

You can find available model for caret at https://topepo.github.io/caret/available-models.html

Looks like nnet uses sigmoidal activation functions for the hidden layer; see here https://stats.stackexchange.com/questions/78252/whats-the-activation-function-used-in-the-nodes-of-hidden-layer-from-nnet-libra

Additionally, you might want to set linout parameter TRUE for regression and FALSE for classification.


How can I use my own dataset for Image segmentation using Tensorflow I have a huge problem using my own created dataset for image segmentation using Tensorflow. The dataset that I've build contain images like the one shown below:

The problem that I have is: How do I use my own dataset specifically for image segmentation? I've looked at the documentation on how to create datasets but all the examples either only use object detection with a single class or classify the entire image. I want to assign every pixel to a class (image segmentation) and use it to train my model.
I've also found examples of image segmentation but they all use existing datasets such as Cityscapes, ADE20k etc. How can I use my own images and data and transform it to a tensorflow dataset so that I can use it for training?

Just having segmented images is probably not enough. The training data for segmentation needs to be in a specific format.
Have a look at the coco dataset for image segmentation. Sometimes we need to convert the dataset into that format.
I'd suggest reading up a bit on how to train a mask rcnn model on your own dataset. There are a number of articles available online.


Rescale parameter in data augmentation I'm a little bit lost about the rescale parameter in the ImageDatagenerator function.
I know that the rescale argument by itself does not augment my data and that by doing rescale=1./255 it will convert the pixels in range [0,255] to range [0,1].
Currently, I'm only using random flips(vertical and horizontal) and 90º degrees rotations but I'm pondering whether or not I should add the rescale parameter.
My question is what is the advantage in doing that?

As rightly pointed out by you the rescale=1./255 will convert the pixels in range [0,255] to range [0,1].
This process is also called Normalizing the input. Scaling every images to the same range [0,1] will make images contributes more evenly to the total loss.
Without scaling, the high pixel range images will have a large say to determine how to update weights. For example, black/white cat image could be higher pixel range than pure black cat image, but it just doesn't mean black/white cat image is more important for training.
Also the neural network has a higher chance of converging as it make the coefficients in range of [0,1] as opposed to [0,255] so helps the model process input faster.


Should the depth on convolutional layers be set to a figure divisible by 2? I'm reading a book titled Python Deep Learning, and in Convolutional layers in deep learning on the chapter 5, the following is written:

  One more important point to make is that convolutional networks should generally have a depth equal to a number which is iteratively divisible by 2, such as 32, 64, 96, 128, and so on. This is important when using pooling layers, such as the max-pool layer, since the pooling layer (if it has size (2,2)) will divide the size of the input layer, similarly to how we should define "stride" and "padding" so that the output image will have integer dimensions. In addition, padding can be added to ensure that the output image size is the same as the input.

As far as I know, the width and the height should be a figure divisible by 2, in order to use a pooling layer. However, I don't understand why the depth must be a figure divisible by 2. The pooling layer just operates on a 2-dimensional screen based on width and height, and it operates on each filter (depth) separately, right?

Why should the depth also be set to a figure divisible by 2?

You are right, the way it says it, it means that the height and width should be divisible by 2.

While the argument he makes is invalid, it is a good practice for the sakes of memory efficiency to make some parameters like the batch size divisible by 2. I'm guessing you want to make more efficient use of memory blocks. The same logic could apply to the depth of convolutional layers.

This is no way as important as the author of the book says!


Help in understanding the maths behind Logistic Regression I am following the lecture notes available https://www.stat.cmu.edu/~cshalizi/uADA/12/lectures/ch12.pdf

I cannot understand how Eqs 12.4 and 12.5 come,

why the Bernoulli probability has $1-p(x)$ in the denominator,
how come $p(x) = \exp(\beta + \beta^Tx)$
and how $log \frac{p(x)}{1-p(x)}$ evaluates to $\beta + \beta^Tx$.

In general $\beta$ is the parameter of the model but I don't quite follow how come the log expression evaluates to it. Is there some mathematical formula which is skipped that is used to evaluate the log expression? This is crucial for me to know as these values are substituted in eq 12.10 where $p(x) = \exp(\beta + \beta^Tx)$

This feels like a bit of a convoluted way to introduce the concept, but alright :D Let me start at a slightly different point.

Maybe in Machine Learning or in other places you have encountered the $sigmoid$ function:
$$ sigmoid(S) = \frac{e^S}{1+e^S} = \frac{1}{1+e^{-S}} $$
The sigmoid has the nice property to map any real number $S$ to a number between 0 and 1. This is super when dealing with models that have to represent probabilities.
As in the slides they are looking for a (conditional) probability $p(x)$ they go ahead and implicitly set
$$ p(S) = sigmoid(S) $$
They just do it around an extra corner. In order to be consistent with the slides (and to save some tedious writing) I'll now keep talking about the probability $p(x)$ and think of the $sigmoid$-function.
$p(S)$ (i.e. the $sigmoid$) has an inverse function, called the $logit$. Let's try to find the inverse of $p(S)$:
$$ p = \frac{1}{1+e^{-S}} $$
and from there we move some stuff around
$$ 1+e^{-S} = \frac{1}{p} $$
$$ e^{-S} = \frac{1}{p} - 1 $$
$$ -S = \log \left[\frac{1}{p} - 1 \right] $$
$$ S = \log \left[\frac{1}{\frac{1}{p} - 1} \right] $$
$$ S = \log \left[\frac{p}{1 - p} \right] $$
This is where they start in your notes. We can take any number $S$, punch it into the logit, solve for $p$ and Boom! we've got ourselves a nice conditional probability.
So the next question is: what is $S$? $S$ is supposed to be some function of $x$. There is a wide variety of functions you could use. You could even put a massive Neural Network, but for now, let's stick to linear regression
$$ S(x) = \beta_0 + \beta x $$
If you decide to go for $S(x)$ being linear, you can now go step 3. backwards and end up at step 1. with the expression they also show in the notes
$$ p(x) = \frac{e^{\beta_0 + \beta x}}{1+e^{\beta_0 + \beta x}} = \frac{1}{1+e^{-(\beta_0 + \beta x)}} $$

So to answer your questions:

$1-p(x)$ is not in the denominator of the distribution, it seems to me they just introduce it in the $\log\frac{p}{1-p}$ in order to end up with the $sigmoid$.

$p(x)$ is not equal to $e^{\beta_0 + \beta x}$, but equal to the $sigmoid$.

$\log \frac{p}{1-p}$ is declared to be equal to $\beta_0 + \beta x$ in order to end up with the $sigmoid$.

I think it would have been more straight-forward to say that you want to map a linear function to something between 0 and 1, which you can do with the sigmoid and then you would have been done in one step. It seems a bit weird to introduce the inverse of the sigmoid, claiming that this was some property you want and then solve for $p$, but that might be a matter of taste.


How does the Transformer predict n steps into the future? I have barely been able to find an implementation of the Transformer (that is not bloated nor confusing), and the one that I've used as reference was the PyTorch implementation. However, the Pytorch implementation requires you to pass the input (src) and the target (tgt) tensors for every step, rather than encoding the input once and keep on iterating for n steps to generate the full output. Am I missing something here?
My first guesses were that the Transformer isn't technically a seq2seq model, that I have not understood how I'm supposed to implement it, or that I've just been implementing seq2seq models incorrectly for the last few years :)

The Transformer is a seq2seq model.
At training time, you pass to the Transformer model both the source and target tokens, just like what you do with LSTMs or GRUs with teacher forcing, which is the default way of training them. Note that, in the Transformer decoder, we need to apply masking to avoid the predictions depending on the current and future tokens.
At inference time, we don't have the target tokens (because that is what we are trying to predict). In this case, the decoder input in the first step would just be the sequence [], and we would predict the first token. Then, we would prepare the input for the next timestep appending the prediction to the previous timestep input (i.e. []), and then we would obtain the prediction for the second token. And so on. Note that, at each timestep, we are repeating the computations for the past positions; in real implementations, these states are cached instead of re-computed each timestep.
About some piece of Python code illustrating how the Transformer works, I suggest The annotated Transformer, which is a nice guide through a real implementation. You may be most interested in the function run_epoch for the training and in the function greedy_decode for the inference.
def greedy_decode(model, src, src_mask, max_len, start_symbol):
    memory = model.encode(src, src_mask)
    ys = torch.ones(1, 1).fill_(start_symbol).type_as(src.data)
    for i in range(max_len-1):
        out = model.decode(memory, src_mask,
                           Variable(ys),
                           Variable(subsequent_mask(ys.size(1))
                                    .type_as(src.data)))
        prob = model.generator(out[:, -1])
        _, next_word = torch.max(prob, dim = 1)
        next_word = next_word.data[0]
        ys = torch.cat([ys,
                        torch.ones(1, 1).type_as(src.data).fill_(next_word)], dim=1)
    return ys

In greedy_decode you can see how the predictions of the current timestep are concatenated to the input to create the input for the following timestep.


Is there t-SNE in WEKA? I want to use t-SNE in WEKA just for visualization purposes. I tried to look at the package manager but could not find any implementation of it.

Is there anything that I can do to achieve it?

Sadly no, there is not a T-SNE implementation for WEKA.

If you can install python packages in your environment, then you can use the wekaPython package (in WEKA's package manager) to run scikit-learn's T-SNE implementation on data you have loaded into WEKA. Use this code in the 'CPython Scripting' panel (which appears after successfully installing wekaPython):

X = py_data.iloc[:, :-1]
y = py_data.iloc[:, -1]

from sklearn.manifold import TSNE
import matplotlib.pyplot as plt

tX = TSNE().fit_transform(X)

plt.scatter(tX[:, 0], tX[:, 1])
plt.show()


sklearn - How to create a sequential pipeline Update: The examples in this post were updated
I am reposting this question here after not getting a clear answer in a previous SO post
I am looking for a help building a data preprocessing pipleline using sklearn's ColumnTransformer functions where the some features are preprocesses sequentially. I am well aware of how to build separate pipelines for different subsets of features. For example, my pipleline may look something like this:
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import Normalizer

ColumnTransformer(remainder='passthrough',
                  transformers=[
                              ('num_impute', SimpleImputer(strategy='median'), ['feat_1', 'feat_2'])
                                ('Std', StandardScaler(), ['feat_3', 'feat_4']),
                                ('Norm', Normalizer(), ['feat_5', 'feat_6']),
                                ])

Notice that each transformer is provided a unique set of features.
The issue I am encountering is how to apply sequential processing for the same features (different combinations of transformations and features). For example,
ColumnTransformer(remainder='passthrough',
                  transformers=[
                              ('num_impute', SimpleImputer(strategy='median'), ['feat_1', 'feat_2', , 'feat_5'])
                              ('Std', StandardScaler(), ['feat_1', 'feat_2','feat_3', 'feat_4', 'feat_6']),
                              ('Norm', Normalizer(), ['feat_1', 'feat_6'])

                                ])

Notice that feat_1 was provided to three transformations, feat_2 was provided to two transformers (impute and Std), and feat_6 was provided to two transformers (Std and Norm)
A pipeline like this will two duplicate columns for feat_2 and feat_3, and three duplicate columns for feat_1. Building a separate pipeline for each transformation/feature combination is not scalable.

When you want to do sequential transformations, you should use Pipeline.
imp_std = Pipeline(
    steps=[
        ('impute', SimpleImputer(strategy='median')),
        ('scale', StandardScaler()),
    ]
)

ColumnTransformer(
    remainder='passthrough',
    transformers=[
        ('imp_std', imp_std, ['feat_1', 'feat_2']),
        ('std', StandardScaler(), ['feat_3']),
    ]
)

or
imp = ColumnTransformer(
    remainder='passthrough',
    transformers=[
        ('imp', SimpleImputer(strategy='median'), ['feat_1', 'feat_2']),
    ]
)

Pipeline(
    steps=[
        ('imp', imp),
        ('std', StandardScaler()),
    ]
)


Machine Learning for hedging/ portfolio optimization? With increasingly sophisticated methods that work on large scale datasets, financial applications are obvious. I am aware of machine learning being employed on financial services to detect fraud and flag fraudulent activities but I have a lesser understanding of how it helps to predict the price of the stock the next day and how many stocks of a particular company to buy.

Do the hedge funds still employ portfolio optimization techniques that are right out of the mathematical finance literature or have they started to use machine learning to hedge their bets? More importantly, what are the features that are used by these hedge funds and what is a representative problem set up?

That is a rather broad question, and there is tons of literature about quantitative analysis and stock market prediction using machine learning.

The most classical example of predicting the stock market is employing neural networks; you can use whatever feature you think might be relevant for your prediction, for example the unemployment rate, the oil price, the gold price, the interest rates, and the timeseries itself, i. e. the volatility, the change in the last 2,3,7,..., days etc. - a more classical approach is the input-output-analysis in econometrics, or the autoregression analysis, but all of it can be modeled using neural networks or any other function approximator / regression in a very natural way.

But, as said, there are tons of other possibilities to model the market, to name a few: Ant Colony Optimization (ACO), Classical regression analysis, genetic algorithms, decision trees, reinforcement learning etc. you name it, almost EVERYTHING has probably been applied to the stock market prediction problem.

There are different fond manager types on the markets. There are still the Quants which are doing a quantitative analysis using classical financial maths and maths borrowed from the physics to describe the market movements. There are still the most conservative ones which do a long-term, fundamental analysis of the corporation, that is, looking in how the corporation earns money and where it spends money. Or the tactical analysts who just look for immediate signals to buy / sell a stock in the short term. And those quantitative guys who employ machine learning amongst other methods.


How does LSTM fights vanishing gradient? Vanishing gtadient causes too small values which are come from non-linear functions like sigmoid, tanh. LSTM architecture was developed to fight these issues.
But how can it avoid a problem if there are 3 sigmoid, 2 tanh functions and, moreover, they are combinated via multiplication?

There are two factors that affect the magnitude of gradients - the weights and the activation functions (or more precisely, their derivatives) that the gradient passes through.

If either of these factors is smaller than 1, then the gradients may vanish in time; if larger than 1, then exploding might happen. For example, the tanh derivative is <1 for all inputs except 0; sigmoid is even worse and is always ≤0.25.

In the recurrency of the LSTM the activation function is the identity function with a derivative of 1.0. So, the backpropagated gradient neither vanishes or explodes when passing through, but remains constant.
The effective weight of the recurrency is equal to the forget gate activation. So, if the forget gate is on (activation close to 1.0), then the gradient does not vanish. Since the forget gate activation is never >1.0, the gradient can't explode either.
So that's why LSTM is so good at learning long range dependencies.


How does Excel obtain the R² determination coefficient in an exponetial regression? How does Excel obtain the R² determination coefficient in an exponetial regression? This may seem a silly question but Excel actually shows a R² coefficient for non-linear regressions. How does Excel calculate it? Does it have any sense?

Please bear in mind that excel does not handle missing values well. If your data column has some blank cells in it, if you filter them out, the r2 can change (subtly). Excel is fine for an overview but use a more robust method in case you want to get formal figures for publication.


What are the sparse and dense vector ? I cant undestand ,can you explain to me?please.Why do we use for? I am new to neural networks, embeddings, etc. I am struggling understanding things like sparse representation, embeddings, and especially sparse vectors. Could you explain these to me? Why do we need this vector? What is it?

A sparse matrix

A sparse matrix is a matrix that is comprised of mostly zero values.
e.g

     1, 0, 0, 1, 0, 0
A =  0, 0, 2, 0, 0, 1
     0, 0, 0, 2, 0, 0

A Dense matrix

Dense matrices with mostly non-zero values, which are referred to as dense matrices.

e.g.

     1, 1, 3, 1, 4, 5
A =  0, 1, 2, 0, 1, 1
     1, 0, 0, 2, 1, 1


Question Related Numpy With Numpy, what’s the best way to compute the inner product of a vector of size 10 with each row in a matrix of size (5, 10)?

Here are a few ways, using some dummy data:

In [1]: import numpy as np

In [2]: a = np.random.randint(0, 10, (10,))

In [3]: b = np.random.randint(0, 10, (5, 10))

In [4]: a
Out[4]: array([4, 1, 0, 6, 3, 3, 6, 6, 1, 8])

In [5]: b
Out[5]:
array([[9, 0, 6, 1, 1, 1, 4, 7, 4, 7],
       [5, 8, 8, 3, 4, 8, 7, 3, 0, 4],
       [2, 2, 5, 3, 9, 6, 1, 5, 8, 3],
       [2, 0, 4, 3, 5, 3, 3, 4, 3, 3],
       [3, 3, 6, 4, 7, 5, 8, 6, 7, 3]])

Because of the dimensions you asked for, in order to compute inner products (a.k.a. scalar products and dot products), we need to transpose the matrix b so that the dimensions work out.
With a vector of length 10, numpy gives it shape (10,). So it seems 10 rows and no columns, however it is kind of ambiguous. Numpy will essentially do what it has to in order to make dimensions work. We could force it into a (10, 1) vector by using a.reshape((10, 1)), but it isn't necessary. The matrix has a defined second dimensions, so we have a shape (5, 10). In order to multiply these two shapes together, we need to make the same dimensions match in the middle. This means making (10,) * (10, 5). Performing the transpose on matrix reverses the dimensions to give us that (10, 5). Those inner 10s will then disappear and leave us with a (1, 5) vector.

That all being said, we can use any of the following to get equivalent answers:

The standard standard dot-product:

In [7]: a.dot(b.T)
Out[7]: array([174, 174, 141, 119, 190])

The convenient numpy notation:

In [6]: a @ b.T
Out[6]: array([174, 174, 141, 119, 190])

The efficient "Einstein notation", a subset of Ricci calculus (I leave the interested reader to search online for more information):

In [8]: np.einsum('i,ij->j', a, b.T)
Out[8]: array([174, 174, 141, 119, 190])

Here as in the comments from shadowstalker:

In [9]: np.array([np.dot(a, r) for r in b])
Out[9]: array([174, 174, 141, 119, 190])

If your matrices are of dimensions (100, 100) or smaller, then the @ method is probably the fastest and most elegant. However, once you start getting into matrices that make you wonder if you laptop will handle it (e.g. with shape (10000, 10000)) - then it is time to read the documentation and this blog about Einstein notation and the amazing einsum module within numpy!


Cross Validation how to determine when to Early Stop? When using "K-Fold Cross Validtion" for Neural Net, do we:

Pick and save initial weights of the network randomly (let's call it $W_0$)
Split data into $N$ equal chunks
Train model on $N-1$ chunks, validating against the left-out chunk (the $K$'th chunk)
Get validation error and revert the weights back to $W_0$
shift $K$ by 1 and repeat from 3.
Average-out the validtion errors, to get a much better understanding of how network will generalize using this data.
Revert back to $W_0$ one last time, and train the network using the ENTIRE dataset

Question 1:

I realize 7 is possible, because we have a very good understanding of how network will generalize with the help of step 6. - Is this assumption correct?

Question 2:

Is reverting back to the initial $W_0$ a necessity, else we would overfit? (revert like we do in step 4. and 7.)

Question 3, most important:

Assume we've made it to step 7, and will train the model using ENTIRE data. By now, we don't intend to validate it after we will finish. In that case how do we know when to stop training the model during step 7?

Sure, we can train with same number of epochs as we did during Cross validation. But then how can we be sure that Cross Validation was trained with an appropriate number of epochs in the first place?

Please notice - during steps 3, 4, 5 we only had $K$'th chunk to evaluate Training vs Validation loss. $K$'th chunk is very small, so during the actual Cross-Validation it was unclear when to Early-Stop... To make things worse, it will be even more difficult in case of Leave-One-Out (also know as All-But-One), where K is simply made from a single training example

First, I would like to emphasize, that cross-validation on itself does not give you any insights about overfitting. This would require comparing training and validation errors over the epochs. Typically you make such comparison with your eye and you can start with one train/validation split.

Question 1: By getting validation error N times, you develop a reasonable (whether it is very or not very good is a question) understanding of how your network will perform (= what error it will give) on the new unseen data.

Often you do cross validation as a part of grid search of hyper-parameters. Then averaging errors at step 6 is mainly for choosing the best hyper-parameters: you believe that the hyper-parameters are best if the corresponding network produces the smallest average validation error. Simultaneously this error is your estimation on what error the final model will give you on the new data.

If you want, you can proceed with your exploration and compare validation errors (for one and the same hyper-parameter set) with each other, calculate standard deviation in order to get further insights.

The concept "model generalizes well" is more related to the absence of overfitting. To make this conclusion, you need to compare train and validation errors. If they are close to each other, then the model generalizes well. This is not directly related to cross validation.

Question 2:  The only purpose to take the whole dataset is to train the model on more data. And more data is always good. On the other side, if you are happy with one of the models produced during cross-validation, you can take it.

Question 3: You can take the number of epochs as one of the parameters in grid-search of hyper-parameters. This search usually goes with cross-validation inside. When at step 7 you take the whole data set, you take more data. Thus, overfitting, if at all, at this stage can only be reduced. If it bothers you, that each chunk is small, replace K-fold cross validation with, for example, K times 50/50 train/test splits. And I would never worry about leave-one-out. It was developed for small (very small) datasets. For Neural Net to be good, you typically need large or very large dataset.


Using Linear Regression to Learn Polynomial Regression Let's start by considering one-dimensional data, i.e., $d=1$. In OLS regression, we would learn the function
$$
f(x)=w_{0}+w_{1} x,
$$
where $x$ is the data point and $\mathbf{w}=\left(w_{0}, w_{1}\right)$ is the weight vector. To achieve a polynomial fit of degree $p$, we will modify the previous expression into
$$
f(x)=\sum_{j=0}^{p} w_{j} x^{j}
$$
where $p$ is the degree of the polynomial. We will rewrite this expression using a set of basis functions as
$$
f(x)=\sum_{j=0}^{p} w_{j} \phi_{j}(x)=\mathbf{w}^{\top} \boldsymbol{\phi}
$$
where $\phi_{j}(x)=x^{j}$ and $\phi=\left(\phi_{0}(x), \phi_{1}(x), \ldots, \phi_{p}(x)\right)$. We simply apply this transformation to every data point $x_{i}$ to get a new dataset $\left\{\left(\phi\left(x_{i}\right), y_{i}\right)\right\}$. Then we use linear regression on this dataset, to get the weights $\mathbf{w}$ and the nonlinear predictor $f(x)=\sum_{j=0}^{p} w_{j} \phi_{j}(x),$ which is a polynomial (nonlinear) function in the original observation space. Notes
How does this work? Could anyone give me an example and explain it to me in simple terms? How would I go about implementing this in Numpy?
Thank you in Adance

It is quite simple to understand (and to implement using matrices).
Consider a specific example (to generalise later). You have a polynomial function of a single feature $x$):
$$ f(x) = \omega_0 x^0 + \omega_1 x^1 + \ldots \omega_n x^n $$
You can organise coefficients and features in vectors and get $f$ by a scalar product:
$$ \mathbf{\omega} = \begin{pmatrix} \omega_0, \\ \vdots \\ \omega_n \end{pmatrix}, \qquad \mathbf{x} =  \begin{pmatrix} 1, \\ x \\ x^2 \\ \vdots \\ x^n \end{pmatrix}$$
Hence $$ f(x) = \omega^T\mathbf{x}$$.
This is nothing else than a multi-feature linear regression where the $i$-th feature is now the $i$-th power of $x$.
In numpy, imagine you have an array of data x.
To create the vector $\mathbf{x}$ above, you can do (for $n=3$, for instance)
X = np.ones((len(x), 4))
X[:,1] = x
X[:,2] = np.power(x,2)
X[:,3] = np.power(x,3)

And then using sklearn LinearRegression,
model = LinearRegression()
model.fit(X, y)

UPDATE: In sklearn has been recently introduced PolynomialFeatures that precisely performs the transformation I described in numpy (you asked in numpy, but this might be useful as well).


imbalanced dataset in text classififaction I have a data set collected from Facebook consists of 10 class, each class have 2500 posts, but when count number of unique words in each class, they has different count as shown in the figure

Is this an imbalanced problem due to word count , or balanced according number of posts. and what is the best solution if it imbalanced?

update
My python code:

data = pd.read_csv('E:\cluster data\One_File_nonnormalizenew2norm.txt', sep="*")

data.columns = ["text", "class1"]
data.dropna(inplace=True)
data['class1'] = data.class1.astype('category').cat.codes
text = data['text']

y = (data['class1'])
sentences_train, sentences_test, y_train, y_test = train_test_split(text, y, test_size=0.25, random_state=1000)
from sklearn.feature_extraction.text import CountVectorizer
num_class = len(np.unique(data.class1.values))

vectorizer = CountVectorizer()
vectorizer.fit(sentences_train)

X_train = vectorizer.transform(sentences_train)
X_test  = vectorizer.transform(sentences_test)

model = Sequential()
max_words=5000
model.add(Dense(512, input_shape=(60874,)))
model.add(Dense(20,activation='softmax'))####
model.summary()
model.compile(loss='sparse_categorical_crossentropy',
  optimizer='rmsprop',
  metrics=['accuracy'])

model.fit(X_train, y_train,batch_size=150,epochs=10,verbose=2,validation_data=(X_test,y_test),shuffle=True)
predicted = model.predict(X_test)
predicted = np.argmax(predicted, axis=1)
accuracy_score(y_test, predicted)
predicted = model.predict(X_test)
predicted = np.argmax(predicted, axis=1)
accuracy_score(y_test, predicted)

0.9592031872509961

I don't now wether I got your question right. But if you count all words within a class, for example, the word "the" is counted everytime it appears. However, if you count the unique words the word "the" is counted once. This is why your counts differ from your plot. Each class can have a different number of unique words.


Linear machine learning algorithms "often" have high bias/low variance? In this blog, which explains the meaning of bias and variance in machine learning, there's a line under the heading "Bias-Variance Trade-Off"
which says:

  Parametric or linear machine learning algorithms often have a high bias but a low variance.

I know that there's an "often" in the first sentence, but how can it be true?
If a linear ML algorithm has high bias how can we expect it to have low variance?

The "often" is the key here - the way that linear models are built, especially compared to other types of models, are more likely to favor certain types of errors.... in this case, they are more likely to produce bias-type errors rather than variance-type. Another way of thinking about it is that the way that most linear models will give you broadly correct predictions overall, instead of specifically accurate or wildly inaccurate predictions. Obviously that's a super simplistic reading, but I've found an intuitive way to approach. Essentially - the way that linear models are constructed will lead them to predict more generally, instead of reflecting every type of twist and turn in the data. This is partially a function of the models themselves, and partially due to linear models being the first line of attack with smaller datasets.

If you think about the formula, one way to interpret is that the intercept generalizes the model's predictions. That being said, with enough variables and certain types of data, it's very possible to overtrain a linear model (which will tip it towards variance-type errors).

I like to think of the bias/variance tradeoff question as being one with three factors, instead of just the two in the title. I'll use the short definitions below (for more information, I've found wikipedia's article on the topic to be relatively clear):

Bias - causing the types of errors frequently thought of as
"underfitting," high bias might generalize well at the cost of
not-capturing (losing out on) important nuances in the data

Variance - causing the types of errors frequently thought of as "overfitting," high variance will capture most nuances in the
training data including noise that will reduce its likelihood of
generalizing well to future like datasets
Data/Data Quality - While not usually included in the classic understanding of the tradeoff, data type and quality is integral to understanding why a certain model is tipping towards either the bias or variance sides of the tradeoff. This also includes any kinds of time effects or other quirks that in essence mean that a training/test split will not properly represent the model's future performance, especially as regards linear models.


Trying to understand Logistic Regression Implementation I'm currently using the following code as a starting point to deepen my understanding of regularized logistic regression.  As a first pass I'm just trying to do a binary classification on part of the iris data set.

One problem I think I have encountered is that the negative log-loss (computed with loss and stored in loss_vec) doesn't change much from one iteration to the next.

Another challenge I am facing is trying to figure out how to plot the decision boundary once I have learned the logistic regression coefficients. Using the coefficients to plot the 0.5 decision boundary is way off.  This makes me think I have made a mistake somewhere

http://fa.bianp.net/blog/2013/numerical-optimizers-for-logistic-regression/

import numpy as np
import matplotlib.pyplot as plt
from sklearn import linear_model, datasets

iris = datasets.load_iris()
X = iris.data[:, :2]  # we only take the first two features.
Y = iris.target

X = X[:100,:]
Y = Y[:100]

def phi(t):
    # logistic function, returns 1 / (1 + exp(-t))
    idx = t > 0
    out = np.empty(t.size, dtype=np.float)
    out[idx] = 1. / (1 + np.exp(-t[idx]))
    exp_t = np.exp(t[~idx])
    out[~idx] = exp_t / (1. + exp_t)
    return out

def loss(x0, X, y, alpha):
    # logistic loss function, returns Sum{-log(phi(t))}
    #x0 is the weight vector w are the paramaters, c is the bias term
    w, c = x0[:X.shape[1]], x0[-1]
    z = X.dot(w) + c
    yz = y * z
    idx = yz > 0
    out = np.zeros_like(yz)
    out[idx] = np.log(1 + np.exp(-yz[idx]))
    out[~idx] = (-yz[~idx] + np.log(1 + np.exp(yz[~idx])))
    out = out.sum() / X.shape[0] + .5 * alpha * w.dot(w)
    return out

def gradient(x0, X, y, alpha):
    # gradient of the logistic loss
    w, c = x0[:X.shape[1]], x0[-1]
    z = X.dot(w) + c
    z = phi(y * z)
    z0 = (z - 1) * y
    grad_w = X.T.dot(z0) / X.shape[0] + alpha * w
    grad_c = z0.sum() / X.shape[0]
    return np.concatenate((grad_w, [grad_c]))

def bgd(X, y, alpha, max_iter):
    step_sizes = np.array([100,10,1,.1,.01,.001,.0001,.00001])
    iter_no = 0
    x0 = np.random.random(X.shape[1] + 1) #initialize weight vector

    #placeholder for coefficients to test against the loss function
    next_thetas = np.zeros((step_sizes.shape[0],X.shape[1]+1) )
    J = loss(x0,X,y,alpha)
    running_loss = []
    while iter_no < max_iter:
        grad = gradient(x0,X,y,alpha)
        next_thetas = -(np.outer(step_sizes,grad)-x0)
        loss_vec = []
        for i in range(np.shape(next_thetas)[0]):
            loss_vec.append(loss(next_thetas[i],X,y,alpha))
        ind = np.argmin(loss_vec)
        x0 = next_thetas[ind]
        if iter_no % 500 == 0:
            running_loss.append(loss(x0,X,y,alpha))
        iter_no += 1
    return next_thetas

There are several issues I see with the implementation. Some are just unnecessarily complicated ways of doing it, but some are genuine errors.

Primary takeaways

A: Try to start from the math behind the model. The logistic regression is a relatively simple one. Find the two equations you need and stick to them, replicate them letter by letter.

B: Vectorize. It will save you a lot of unnecessary steps and computations, if you step back for a bit and think of the best vectorized implementation.

C: Write more comments in the code. It will help those trying to help you. It will also help you understand each part better and maybe uncover errors yourself.

Now let's go over the code step by step.

1. The sigmoid function

Is there a reason for such a complicate implementation in phi(t)? Assuming that t is a vector (a numpy array), then all you really need is:

def phi(t):
   1. / (1. + np.exp(-t))

As np.exp() operates element-wise over arrays. Ideally, I'd implement it as a function that can also return its derivative (not necessary here, but might be handy if you try to implement a basic neural net with sigmoid activations):

def phi(t, dt = False):
   if dt:
      phi(t) * (1. - phi(t))
   else:
      1. / (1. + np.exp(-t))

2. Cost function

Usually, the logistic cost function is defined as a log cost in the following way (vectorized): $ \frac{1}{m} (-(y^T \log{(\phi(X\theta))})-(1-y^T)(\log{(1 - \phi(X\theta))}) + \frac{\lambda}{2m} \theta^{1T}\theta $  where $\phi(z)$ is the logistic (sigmoid) function, $\theta$ is the full parameter vector (including bias weight), $\theta^1$ is parameter vector with $\theta_1=0$ (by convention, bias is not regularized) and $\lambda$ is the regularization parameter.

What I really don't understand is the part where you multiply y * z. Assuming y is your label vector $y$, why are you multiplying it with your z before applying the sigmoid function? And why do you need to split the cost function into zeros and ones and calculate losses for either sample separately?
I think the problem in your code really lies in this part: you must be erroneously multiplying $y$ with $X\theta$ before applying $\phi(.)$.

Also, this bit here: X.dot(w) + c. So c is your bias parameter, right? Why are you adding it to every element of $X\theta$? It shouldn't be added - it should be the first element of the vector $X\theta$. Yes, you don't regularize it, but you need to use in the "prediction" part of the loss function.

In your code, I also see the cost function as being overly complicated. Here's what I would try:

def loss(X,y,w,lam):
   #calculate "prediction"
   Z = np.dot(X,w)
   #calculate cost without regularization
   #shape of X is (m,n), shape of w is (n,1)
   J = (1./len(X)) * (-np.dot(y.T, np.log(phi(Z))) * np.dot((1-y.T),np.log(1 - phi(Z))))
   #add regularization
   #temporary weight vector
   w1 = copy.copy(w) #import copy to create a true copy
   w1[0] = 0
   J += (lam/(2.*len(X))) * np.dot(w1.T, w)
   return J

3. Gradient

Again, let's first go over the formula for the gradient of the logistic loss, again, vectorized: $\frac{1}{m} ((\phi(X\theta) - y)^TX)^T + \frac{\lambda}{m}\theta^1$.
This will return a vector of derivatives (i.e. gradient) for all parameters, regularized properly (without the bias term regularized).

Here again, you've multiplied by $y$ way too soon: phi(y * z). In fact, you shouldn't have multiplied by $y$ in gradient at all.

This is what I would do for the gradient:

def gradient(X, y, w, lam):
   #calculate the prediction
   Z = np.dot(X,w)
   #temporary weight vector
   w1 = copy.copy(w) #import copy to create a true copy
   w1[0] = 0
   #calc gradient
   grad = (1./len(X)) * (np.dot((phi(Z) - y).T,X).T) + (lam/len(X)) * w1
   return grad

The actual gradient descent implementation seems ok to me, but because there are errors in the gradient and cost function implementations, it fails to deliver :/

Hope this will help you get on track.


Model for Differing Number of Rows per Observation Looking to build a response model (click or no click) on marketing data which displays varying number of offers to a person. I don't want to model which offer they click but do they click any of the offers presented to them. My issue is how to deal with the differing number and types of offers?

Example data could be one table of id's:

id   clicked
001       1
002       0
003       1

And varying number of offers per id:

id  discount_rate  on_amt
001     0.05       100
001     0.10       500
002     0.03        50
003     0.05       100
003     0.10       300
003     0.15       500

Do I create features from the offer data set such as average discount_rate, max on_amt etc.? Or create a very large binary sparse matrix of binned offer types such as rate_5-10_amt_0-50 1/0 and rate_5-10_amt_50-100 1/0 ...?

Or is there a good model that handles variable data like this?

You need to create a tidy version of the data with the on_amt and discount_rat encoded as a categorical variable (e.g., one-hot encoded). If they are continuous, they need to binned into categorical variables then encoded.


Please list some well tested api's for arima model I am looking for a good python api for timeseries models such as ARIMA. Please list some well tested apis and few more advance models possible for financial time-series analysis.

Statsmodels:

Statsmodels is your best bet for a python library that includes ARIMA.  I have used it fairly extensively and am quite happy with it. But, its certainly not as well tested as R based ARIMA models.

R:

If you want something "well-tested" then your best bet is likely to use Rpy2 to call an R based ARIMA library from python. Rpy2 can be a bit tricky based on version reconciliation between python, R and Rpy2.

Here's a turorial on calling R from python using Rpy2

Hope this helps!


What is the minimum number of data to create a function? I gonna introduce the problematic like this:

Let's say there are individuals with different capacities/skills. These capacities/skills are depending of the environment: nature of the floor, weather temperature, wind speed. The capacity/skill I want to study is the speed of the individuals considering the environment.
As you know, speed is depending of the distance, not only because of the formula distance/time unit, but because even Usain Bolt cannot reproduce the same speed for the 100m and 200m discipline, actually his mean speed decrease with the distance, with environment's variables as constants. This is why I cannot just look at the mean speed itself to conclude if it is more adapted or not
So, because factors are variable/do change, I want to isolate with marginal effect, and conclude what nature of the floor is better for this individual to perform, what distance is better to perform too (Usain Bolt is not done for endurance...). To do it, I want to use a linear regression, this is more simple to derivate.

The main issue is my data has not always a lot of historic. This is often that there are no more than 5-6 previous experiences.

So, when I was younger in high school, there were a rule of thumb telling we need a least 5 points to make a function (with OLS). Know that I am older and have more experience, I doubt of this small threshold.
But I have not enough experience to know what really could be this threshold. Do you have any idea ?

There are two issues:

Linear regression might be simple to calculate, but it's not sure that the relation between your variables is actually linear. If not, sometimes this simplification is ok because there's not too much variation, but sometime it's completely wrong.
There's no general minimum number of instances valid for every case. Generally the more complex the relation one wants to represent, the more instances one needs.

I'd suggest you start by plotting your data: visualizing the relation between the variables should tell you whether linear regression is a good option, and seeing how scattered your points are should tell you whether you have enough instances.


The key here is dropout. When a model trains with dropout, only a percentage of the total weights (in your case 50%) are used in predictions, which tends to lower the prediction accuracy. When the validation data is tested, dropout is turned off, and the validation loss tends to be a lot lower. see this answer to a similar question for more info.


How to build an overfitted network in order to increase performances I am learning how to implement CNN, and searching on the internet I have found that a trick to design a good network is to first build it in such a way that it overfits, and then use regularization to elimnate overfitting and have a good performing network.

But how do I do this? I don't understand how do I build a network that overfits on purpose? And also in which way do I use regularization after?

Can someone helo me?

Thanks in advance.

The reason of overfitting is generally because of the high(unnecessary) complexity model. So if you train a too complex(large) model, you will get a high training accuracy and low test set accuracy. Then you can start to fight with overfitting  with getting more data, dropout, regularization, early stopping, global average pooling, feature scale clipping or dropping some of layers from your network.


Is Stratification applicable to both Classification and Regression? It is usually better if you have a not so large but balanced dataset and you are performing classification to apply stratification in order to split it in a training and testing datasets which are both balanced as well.

So there is a notion of having the training and testing datasets represent adequately the overall dataset.

Could you expand this notion to regression?

So the idea is that instead of just shuffling and splitting the dataset, you could group the targets in bins for example [0, 10] [10, 20] etc.
And then have the training and testing dataset be also an adequate representation of the whole dataset by having targets with all kinds of values. (Otherwise, you could end up leaving out some part of the range)

Makes sense? :)

Yes that makes sense. Your training and testing datasets should both be similar distributions to the whole dataset in both classification and regression, and in the regression case, binning the target variable is one way to achieve that. You need to make sure that you choose good bin sizes -- you can completely skew how the distribution looks in a histogram based on the bin sizes. Too small and you only have one or two examples per bin, resulting in a very erratic histogram. Too large and you lose a lot of information about the shape of the distribution.

Binning is just one way to approximate the density function of the distribution. Depending on how your data looks, you might be able to fit a Gaussian (or any other distribution) curve to your target variable and use that instead to sample your train/test split.


parquet format: advise on log content I'm using a python script to log IO of a grid job.
the log is formatted like this:

timestamp;fullpath;event;size
1526994189.49;/tmp/folder/;IN_ISDIR;6
1526994189.49;/tmp/folder2/File;IN_ACCESS;36

Those files are millions of line long. I'm using Spark to generate graphs and detect anomaly in job IO.
But before doing that I need to insert the job ID and the jobname to the column making the file :

timestamp;fullpath;event;size;jobid;jobname
1526994189.49;/tmp/folder/;IN_ISDIR;6;123456;afakejobname
1526994189.49;/tmp/folder2/File;IN_ACCESS;36;123456;afakejobname

The thing is I'm new to big Data technologies and I would like to know if using parquet format is it better to put both jobname and jobid
or knowing that I have only 15 different jobname and jobid in the same log is it better to convert it on the fly using SparkSQL and make a join to a very small table with just jobname;jobid and put only the jobid in my log.

Your second solution sounds great, and to use it in the most effective way, look for broadcast variables in Spark.

Then, you're using the features available in spark to optimize it.

Reference: https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-broadcast.html


How to add epoch in sklearn LinearSVC? I have a model that i need to train multiple times using epochs, i tried adding this code clf_svm.fit(train_features, train_labels, epochs=10, batch_size=64) and it didn't work. how to add epoches in skleran linear svc? training that model several times and save the model here is the code am working with
# empty list to hold feature vectors and train labels
train_features = []
train_labels   = []
# loop over the training dataset
print ("[STATUS] Started extracting haralick textures..")

for train_name in train_names:
    cur_path = train_path + "/" + train_name
    cur_label = train_name
    i = 1
    for file in glob.glob(cur_path + "/*.jpg"):

        print ("Processing Image - {} in {}".format(i, cur_label))

        # read the training image
        image = cv2.imread(file)
        #resize = ResizeWithAspectRatio(image, width=1250, height=1000) # Resize by width OR
        # convert the image to grayscale
        gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
        # extract haralick texture from the image
        features = extract_features(gray)

        # append the feature vector and label
        train_features.append(features)
        train_labels.append(cur_label)

        # otherwise create the model, train the model and save the model
if os.path.exists("D:/ai training/aphids/joblib_model.sav"):
    print("Loading Trained Model")
    clf_svm = joblib.load("D:/ai training/aphids/joblib_model.sav")
else:
        # have a look at the size of our feature vector and labels
        print ("Training features: {}".format(np.array(train_features).shape))
        print ("Training labels: {}".format(np.array(train_labels).shape))

        # create the classifier
        print ("[STATUS] Creating the classifier..")
        clf_svm = LinearSVC(random_state=9, dual=False, max_iter=1000)

        # fit the training data and labels
        print ("[STATUS] Fitting data/label to model..")
        clf_svm.fit(train_features, train_labels)

        #savemodel
        joblib_file = 'D:/ai training/aphids/joblib_model.sav'
        joblib.dump(clf_svm, joblib_file)

Support vector machine model in sklearn support adding max iterations parameter which you can change to a higher value. But they don't have epochs parameters nor do they support batch sizes.
To go into more depth, support vectors use an exact convex optimization algorithm, not stochastic gradient descent (like Neural nets). They work on the entire dataset at the same time. The algorithm will run for max iterations or till minimum value is achieved and report one single accuracy number.
From the question it is not clear why you want to train multiple times using epochs. Comparing with neural networks, the equivalent here will be to just set high max iterations parameter and training once.


Data Augmentation in videos I know some techniques for augmenting data when images are used, but I don't know if there are any such techniques specifically catered to videos. Since videos have an additional temporal dimension compared to images I was wondering if there were any techniques that augments the video data using the view points and the motion information etc

You can augment videos in the temporal dimension through clipping, or taking random sequences of consecutive frames. You can also augment in the spatial dimension by cropping frames randomly to simulate camera jitter. Unfortunately, augmenting video data by extrapolating motion information or by producing a video taken from a different viewpoint seems extremely computationally intensive.


How can I train my CNN to learn (numerically) smaller values better? I'm using a CNN to model a problem that involves precise numerical values from a physical simulation.

After months of design/redesign and optimization, I've noticed that the majority of the "error" in this network is accumulated because of lower accuracy for smaller valued cases, whereas it seems to work fine when the values are numerically larger.

The input values are always normalized to $(0,1)$ when fed into the network.

The error is measured on the denormalized output values.

How can I make sure that the network learns these "small" values better?

I believe that the key problem lies in the fact that the network is "blinded" to the actual size of the I/O pairs because of the normalization process.

Is this an inherent problem with the normalization process, or is this a problem in how the network is designed?

The error is quantified using MAPE, in this case.

It seems you are working on a regression problem with a strictly positive target and use MAPE as the loss function to train the neural network. This loss function could be the culprit. As illustrated nicely in this answer, minimizing MAPE leads to predictions that are biased towards smaller values. This is a consequence of the asymmetry between errors in the positive and negative directions (negative errors are 100% at worst, while positive errors are unbounded).

A possible alternative is to minimize the mean absolute error computed with the logarithm of the target. It will converge to the logarithm of the target's median.


Fitting lines through large point clouds I have a large set of points (order of 10k points) formed by particle tracks (movement in the xy plane in time filmed by a camera, so 3d - 256x256px and ca 3k frames in my example set) and noise. These particles travel on approximately straight lines roughly (but only roughly) in the same direction, and so for the analysis of their trajectories I am trying to fit lines through the points. I tried to use Sequential RANSAC, but can't find a criterion to reliably single out false positives, as well as T- and J-Linkage, which were too slow and also not reliable enough.

Here is an image of a part of the dataset with good and bad fits I got with sequential Ransac:
I'm using the centroids of the particle blobs here, blob sizes vary between 1 and about 20 pixels.

I found that subsamples using for example only every 10th frame worked quite well too, so the data size to be processed can be reduced this way.

I read a blog post about all the things neural networks can accomplish, and would like to ask you if this would be a feasible application for one before I start reading (I come from a non-maths background, so I would have to do quite a bit of reading)?

Or could you suggest a different method?

Thanks!

Addendum: Here is code for a Matlab function to generate a sample point cloud containing 30 parallel noisy lines, which I can't distinguish yet:

function coords = generateSampleData()
coords = [];
for i = 1:30
    randOffset = i*2;
    coords = vertcat(coords, makeLine([100+randOffset 100 100], [200+randOffset 200 200], 150, 0.2));
end

figure
scatter3(coords(:,1),coords(:,2),coords(:,3),'.')

function linepts = makeLine(startpt, endpt, numpts, noiseOffset)
    dirvec = endpt - startpt;
    linepts = bsxfun( @plus, startpt, rand(numpts,1)*dirvec); % random points on line
    linepts = linepts + noiseOffset*randn(numpts,3); % add random offsets to points
end

end

Based on the feedback and trying to find more effective approach I developed the following algorithm using a dedicated distance measure.

Following steps are performed:

1) Define a distance metric returning:

zero - if the points do not belong to a line

Euclidian distance of the points - if the points constitute a line according to the defined parameters, i.e.

their distance is higher or equal than the min_line_length and
their distance is lower or equal than the max_line_length and
the line consists of at least min_line_points points with a distance lower that line_width/2 from the line

2) Calculate distance matrix using this distance measure (use sample of the data for large data sets; adjust the line parameters accordingly)

3) Find the points A and B with maximum distance - break to step 5) if the distance is zero

Note that if the distance is higher than zero the points A and B are building a line based on our definition

4) Get all points belonging to the line AB and remove them from the distance matrix. Repeat the step 3) to find another line

5) Check the coverage of the point with the selected lines, if substantial number of points remains uncovered, repeat the whole algorithm with adjusted line parameters.

6) In case that data sample was used - reassign all points to the lines and recalculate the boundary points.

Following parameters are used:

line width - line_width/2 is the allowed distance of the point from the ideal line =   r line_width

minimum line length  - points with shorter distance are not considered to belong to the same line =  r min_line_length

maximum line length  - points with longer distance are not considered to belong to the same line =  r max_line_length

minimum points on a line - lines with less points are ignored = r min_line_points

With your data (after some fiddling with parameters) I got a good result covering all 30 lines.

More details can be found in the knitr script


Predicting the likelihood that a prediction from a linear regression model is accurate So to set up the problem:
I have a data set that had labeled data like colour, brand and quality as independent variables and the dependent is RRP (price).
I have made a linear regression model using this data and can predict the dependent variable using the independent variables (i am using scikitlearn so just using model.predict. This is causing me siginificant problems and I'm not sure if this is the right way to deal with this and I'm not sure if this will hinder my goal of getting accurate values for predicted variables.
Is there a way to calculate the potential accuracy of the price that is predicted? It seems to me if I ask the model to predict on brand x and quality y and the model knows brand x and the quality always produces a tight range of prices the accuracy is potentially higher?

In case you are talking about providing certain interval to your predictions, what you might need is adding some confidence interval to your linear regression predictor, something which you can make via a resampling method like bootstrapping as a robust way to find predictions intervals. One key advantage is that it does not assume any kind of distribution, being a distribution-free method to find your predictions and, if needed, to your regression coefficients estimates.
The steps would be:

Draw n random samples (with replacement) from your dataset, where n is the bootstrap sample size
Fit a linear regression on the bootstrap sample from step 1 and predict a value
Take a single residual at random from the original regression fit, add it to the predicted value and save the result.
Repeat 1 to 3 steps several times (1k times for instance)
Find the desired percentiles of your interval (2.5th to 97.5th for instance)
Source of info in this book

On the other hand, if you mean providing a generic confidence metric value for your model, you should find for instance a MSE or MSA on a test set.


Amalgamating multiple datasets with different variables coding I have several datasets with thousands of variables. This different datasets have different variables for the same thing. Is there a way to automatically/semi-automatically check compatible variables and make them consistent?

If there is such thing, that would save me months of tedious work. The data is stored in SPSS format.

I don't know of an automated tool for this, but as a suggestion, you could look to using feature selection or correlation of all variables in a particular dataset against all other variables.

It depends on the direction that you are wanting to take your analysis later, but the second method will enable you to see variables that are highly correlated, and so might be good candidates for checking that they are duplicates?

Also, not sure if you are aware, but there are a few R packages (memisc is particularly popular) that enable you to read SPSS files directly and then work on the data directly.

Sorry wont save you months of work though ... but might save you days ...