Why would GradientBoostClassifier do better than XGBoostClassifier? I am working on the Kaggle home loan model and interestingly enough, the GradientBoostClassifier has a considerably better score than XGBClassifier. At the same time it seems to not overfit as much. (note, I am running both algos with default settings).
From what I've been reading XGBClassifier is the same as GradientBoostClassifier, just much faster and more robust. Therefore I am now confused on why would XGB overfit so much more than GradientBoostClassifier, when it should do the contrary? What would be a good reason why this is happening?

GradientBoostClassifier is slower but more precise. In your case, it could be finding a better model without suffering from overfitting.
Here are some of the main differences you are looking for.
XGBClassifier was designed to be faster. However, XGBClassifier takes a few shortcuts in order to run faster. For example, to save time, XGBClassifier will use an approximation on the splits, not spend so much time on calculating and evaluating the best splits. Usually XGB results will be close and you won't see much difference, but your case could be an exception.
If you are still curious, you can try changing some parameters in the XGBClassifier model, such as the tree_method='exact' or you can modify the sketch_eps parameter in XGBClassifier to more closely match the GradientBoostClassifier results. Of course, this will slow down the XGBClassifier model as a result.
sketch_eps [default=0.03]
Only used for tree_method=approx.
This roughly translates into O(1 / sketch_eps) number of bins. Compared to directly select number of bins, this comes with theoretical guarantee with sketch accuracy.
Usually user does not have to tune this. But consider setting to a lower number for more accurate enumeration of split candidates.
range: (0, 1)


How does keras handle subfolders I am going to train a network to label a given image whether it belongs to category A or category B (a binary classifier). I use flow_from_directory to generate train, validation and test data. I keep all the images belong to category A in one folder. However, category B has sub folders. Lets say we have a folder which contains ship images (category A) and another folder which contains all the other images (cats, dogs, buildings, etc.) which are non ships (category B). In my application I keep all the non-ship images in different folders, according to their content, under non_ship folder. At the end I want a binary result (ship or non-ship). In this case I am actually not sure about how keras handles this. Does not it consider all the sub folders and just consider all the non-ship images are in one folder or consider that they belong to different subclasses?

The answer is it does not consider the sub folders. It assumes all the images available in the sub folders are belong to the same category. In this case all the non ship images are assumed they belong to category B.


How can I use my laptop to implement analysis in Python but use a different fast server for computation of analysis? I am working in a company but I am new in the field. We have a central server which is faster than my laptop (of course). So my goal is that I still use my laptop to do python analysis for machine learning but actual computation is happening in the central server so that the computing power is stronger. I am not considering any cloud service as we have a fast server computer.

But I am clueless in this yet. Can anyone advise me how to do this?

Thank you very much in advance.

I have a similar set-up; I run both local and remote installations of Jupyter. On the server I have JupyterHub running, into which i can ssh. Locally I have Anaconda set up etc.

If you are meticulous in mirroring any custom libraries across the system (or even better package them for pip installs) this works quite well.


What are the possible values of a filter in a CNN? I am a trying to write a CNN from scratch in python but I am bit new to CNNs specifically the convolution layers as I am comfortable with the dense layers. I was reading Do filters have different weights for each input channel but I didn't completely understand and had a few questions. I wanted to confirm that if the input layer had 3 channels then for there to be 4 output channels you would need a total of 12 filters.

Also, can the weights(actually I am not completely sure that I know what the weights are, they are the values of the kernels right?) of a filter be a floating number as  I commonly see them being described as -1, 0 ,1.

And finally, are the weights of the kernels. initialised once or rather are they changed during back propagation, as I used to have the notion that only the bias and dense layer would be changed.

Thank you so much for you time, any help is greatly appreciated.

EDIT: I have 1 more question, but I didn't feel it appropriate to make a whole new question. Basically, if I have a filter that looks like [0,1]. The stride is 2. Now if it has to run over [0,1,2,3,4]. First it will match [0,1] with [0,1] then with [2,3]. But won't it stop at [2,3] and never look at 4. How do CNNs solve this problem. Do they add an extra run of [0,1] and [3,4] but this would count 3 two times. Again thanks for the help.

Weights are learned with backpropogation, that's how model learns to identify different patterns with data. The point of a supervised model is to learn weights.

They can be floating value, they are usually represented as [[101][101][101]], because it identifies vertical line, and it's more intuitive to understand.

The filter usually do not contain info on depth, they are square matrix with depth equal to number of channels in input layer, with each filter layer spewing one output layer, so to get 4 output layers, you need to have 4 filters

Stride=2 means it will skip 2 every time, that is, it will go from 0,1 to 3,4


Overfitting with sklearn pipeline - reasons why? So....

I've been playing around with this for FAR TOOO LONG now and I really need some advice.

Most people on kaggle concat training and testing set TOGETHER and then pre scale the data, this seems to provide good results when scoring but i believe this to be data leakage and an incorrect process to perform in real world solutions.

I've kept apart 15% of data as a holdoutset and the rest for CV.

The issue I have is that if I put the scaler into a pipeline, the scale changes on each fold of the CV and gives me wildly different results to if I prescaled the whole dataset prior to hyper parameter tuning.

I'm even getting worse results once I fit to the whole training data (including the holdout set) which suggests its overfitting to the smaller data set.

Is there anything I can do to combat this?

Am I doing something wrong? (should I be putting a scaler into a pipeline?)

Or is this just what it is in the real world?

Any other help regarding overfitting etc would be amazing.

If your dataset is giving really different results for different folds and you just have a scaler and a model in your pipeline could be for different reasons:

The splits of your validation have different difficulties for some reason. Some are just easy to predicto of you haven't choosen the proper validation and crossfold. An example of this would be that there is a temporal dependency and you are not getting it. If this was the case my reccomendation will be to choose a proper validation and crossfold.
Other possible problem would be that the features are not equally distributed, meaning that there are some outliers. If there is outliers when you scale you will have different intervals. If this was the case I would say that before doing anything more complicated doing some pre processing for the whole dataset.

Note: Doing what Kagglers do of concateneting train and test can be dangerous when you put your model into production.


Assign words to various topics Say I have a document and I want to assign all (or most) of the words to pre-assigned topics. So I could take a random selection of words and manually assign them to the appropriate topics, then I want an algorithm to assign the remaining words to the appropriate topics based on how closely they occur to the manually assigned phrases.

Is there an algorithm or technique that does this? I've looked into topic modelling and word2vec and they seem to assign 'topics' in some arbitrary space without any room for manual training and designation of topics.

Latent Dirichlet allocation (LDA) is a generative statistical model that allows sets of observations to be explained by unobserved groups that explain why some parts of the data are similar. LDA does exactly the same thing you wanted to do.


Bagging vs Boosting, Bias vs Variance, Depth of trees I understand the main principle of bagging and boosting for classification and regression trees. My doubts are about the optimization of the hyperparameters, especially the depth of the trees

First question:  why we are supposed to use weak learners for boosting (high bias) whereas we have to use deep trees for bagging (high variance) ?
 - Honestly, I'm not sure about the second one, just heard it once and never seen any documentation about it.

Second question : why and how can it happen that we get better results in the grid searches for gradient boosting with deeper trees than weak learners (and similarly with weak learners than deeper trees in random forest)?

why we are supposed to use weak learners for boosting (high bias) whereas we have to use deep trees for bagging (very high variance)

Clearly it wouldn't make sense to bag a bunch of shallow trees/weak learners. The average of many bad predictions will still be pretty bad. For many problems decision stumps (a tree with a single split node) will produce results close to random. Combining many random predictions will generally not produce good results.

On the other hand, the depth of the trees in boosting limits the interaction effects between features, e.g. if you have 3 levels, you can only approximate second-order effects. For many ("most") applications low-level interaction effects are the most important ones. Hastie et al. in ESL (pdf) suggest that trees with more than 6 levels rarely show improvements over shallower trees. Selecting trees deeper than necessary will only introduce unnecessary variance into the model!

That should also partly explain the second question. If there are strong and higher-order interaction-effects in the data, deeper trees can perform better. However, trees that are too deep will underperform by increasing variance without additional benefits.


How to deal with annotation errors? I know my annotators are not perfect, sometimes making mistakes. What would be the best way to deal with the annotation errors for my training data?

It's very common to have some amount of errors or inconsistencies in a dataset. Sometimes these inconsistencies are not even errors, in some subjective tasks (e.g. translation), annotators may simply not agree on what is the best answer.

What to do with this kind of noise completely depends on the case at hand. If the noise caused by these errors represents a reasonably small proportion of the data, it can safely be ignored: in this case it's up to the learning algorithm to distinguish the relevant patterns from the noise. Otherwise there can be ad-hoc pre-processing implemented to clean up the data. In cases where the subjectivity of the annotator plays an important role, it's useful to have several annotators annotated the same data and check the inter-annotator agreement. This might in turn be used to filter out the least consensual instances, or aggregate the annotations in some way (e.g. majority voting).


How to split data frame into groups, combine rows I have a large data set with 405 columns, many rows, and data from 15 sites.  Each site has 27 columns, each one one quadrats data. Rows are species data.

I would like to split the data into the 15 sites and be able to use functions such as adding or averaging together all 27 columns to get an idea of the species presence at each site. I tried creating a vector of sites and using this to split the data.

Example:

"n<-rep(27,15)                    #repeats 27 15 times (15 sites, 27 quadrats per site in 2018)
names<-c("BB", "BEP","BKP", "BP","BY",'DB','DP','H1P','LTP','NB','NP','NRP','OP','ZB','ZP')
sites18<-as.factor(rep(names[rep(1:15, n)]) )     #site name replicated 27 times

When I use the split function, it loses the species data and makes one long column for each site.

split18<-as.data.frame(split(t(p18),sites18))

I need another solution, perhaps something with an apply function, but I have been unable to find a good solution.

If I understood you correctly, you want to vertically split your data frame and horizontally union the resulting data frames. Here is an example for what I would do in this case:

#assuming exemplary dataset
dataset <- data.frame(matrix(rnorm(n = 27 * 15 * 10), nrow = 10));
colnames(dataset) <- paste(
  as.character(sites18)
  ,rep(1:27, length.out = 27 * 15)
  ,sep = "_");
str(dataset);

#create list of data frames using a vertical split
(list_df_by_group <- lapply(
  names
  ,function(name) dataset[, paste(rep(name, 27), 1:27, sep = "_")]));
names(list_df_by_group) <- names;
str(list_df_by_group);

#horizontally union data frames
final_dataset <- data.frame(matrix(ncol = 28, nrow = 0));
colnames(final_dataset) <- c("group", as.character(1:27));
for(name in names){
  colnames(list_df_by_group[[name]]) <- as.character(1:27);
  final_dataset <- rbind(
    final_dataset,
    cbind(
      data.frame(group = rep(name, nrow(list_df_by_group[[name]])))
      ,list_df_by_group[[name]]));
}
str(final_dataset);


group similar subjects and train only using them I have a dataset with 5k subjects. It's a binary classification problem where I have 3000 positive and 2000 negative subjects.
Now to build a model, I don't like to train the usual way (where we build a generalised model).
Instead I would like to group similar patients together like group1, group2, group3, group4 etc.
And when I new subject to predict, I would like to know which group does he belong to and find out the important predictors for each group?
I know this sounds like K-means clustering but am I right to understand that?
Can anybody provide me the list of steps (pseudocode type) on how to do this? Sorry, am new to ML and exploring various ways to do a classification problem

Clustering would indeed give you something like group1, group2, etc., i.e. it would assign every instance to a group meant to represent similar instances together. With the model you could also assign any new instance to one of the groups in the model, i.e. find the group that this instance is the most similar to. In general clustering doesn't give you which predictors are the most important for a group (there might be ways to calculate this from the model, but afaik it's not standard).
Note that with clustering alone you can't predict the target variable for a new subject. However you could obtain the proportion of positive/negative for every group, i.e. the chances of being positive/negative knowing that the instance belongs to a particular group. You could also do the opposite: chances of belonging to a particular group knowing whether the instance is positive/negative.
An important question is whether you want to include the target variable in the features used for clustering:

If it's included, the model uses it like any other feature to group instances by their similarity.
If it's not included, the model itself doesn't know it so the proportion of positive/negative in a cluster doesn't depend on it. This might be more insightful, depending on what you want to obtain. Another advantage of not including it is that you can still use the outcome of the clustering for training a binary classification model afterwards, typically by adding the cluster an instance belongs to as a feature.

As you can see, there are a few important design decisions to make depending on the goal.
[edit: answer to comment]
In order to use the clustering as a basis for classifying any new instance, here are a few general options:

Basic option: predict the cluster the instance belongs to, then just output the probability of positive of the cluster.
Clustering model + classification model: train a classifier with all the training instances, with one additional categorical feature corresponding to the cluster the instance belongs to. The classification might ge improved compared to only the original features.

Variant: add one feature for every possible cluster, where the value represents how similar the instance is to each cluster (most clustering methods can calculate a distance or similarity against any cluster).

Multiple independent classification models: based on the clustering, train a classification model for every cluster independently, using only the instances from this cluster. Advantage: this allows the classification model to take into account more fine-grained patterns within this cluster. Disadvantage: the model has less instances to be trained from, especially for small clusters.


How to find learning rate decay? Given the number of epochs, batch size and learning rate, is there a formula by which I can calculate the learning rate decay in mini batch SGD?

even though there are some learning rate decay schemes for Standard Gradient descent, convergence of these methods are not justified beyond convex functions or functions whose gradients are globally Lipschitz. Even for those latter maps, when you do not have a good lower bound for the Lipschitz constant L, basically you have no way to say which learning rate is good.

In my joint paper, we proposed a method to modify learning rate using the backtracking version of Gradient Descent, which works quite stably and effectively. More details about the theoretical background and the accompanying source codes can be found in my answer in this link:

Does gradient descent always converge to an optimum?


How to generate a normal distribution with variance $\sigma^2= 0.08$ I have a problem to solve.  I'm lost at generating a random data set for binary classification where each class correspond to the $2D$ region depicted in the figure below. The random data I must generate should have normal distribution with variance $\sigma^2 = 0.08$. I will use $200$ points ($100$ in each region) to train my nural network and report the results in terms of the loss function and the training epochs. New guy here Any tips will be welcomed!!

To generate normal random variable, you can use numpy.random.normal where the scale parameter is the standard deviation. To generate multivariate normal random variable, you can use numpy.random.multivariate_normal.

If you are an R user, you might like to consider mvrnorm

However, I doubt that is what you really want. After all, it is possible to generate points that are beyond the square.

You might like to consider some bounded distribution instead. Consider truncated normal distribution or uniform distribution.


How to deal with large training data? Currently, I use image files and transform them into a *.npy file(saved as a numpy array) as training data. At present this training data set is nearly 3GB.
Now I have more image files, so the training data set will become even larger, maybe up to 40 GB. And I am afraid that the *.npy files can not save this big of a data file.

Is there another possible way to store such a large file?

Since I use Keras to build a neural network model, is it acceptable to split the training data into small parts so as to train the model without having to use all the training data?

Does this sound like a reasonable approach?

I do something similar with keras and GPU training, where i also have only a small memory amount available. The idea would be split the numpy files into smaller ones, let's say 64 samples per file and then load each file and call train_on_batch on those images. You can use keras' train_on_batch function to achieve this:

train_on_batch

train_on_batch(self, x, y, class_weight=None, sample_weight=None)

It's designed to perform a single gradient update over one batch of samples.

Custom Generator

Another idea is to use generators which provide you with data given a directory. They can also be used for data augmentation, i.e. randomly generating new training data from your data. Here is an example from the keras documentation:

train_datagen = ImageDataGenerator(
        rescale=1./255,
        shear_range=0.2,
        zoom_range=0.2,
        horizontal_flip=True)

# just rescale test data
test_datagen = ImageDataGenerator(rescale=1./255)

# this generator loads data from the given directory and 32 images
# chunks called batches. you can set this as you like
train_generator = train_datagen.flow_from_directory(
        'data/train',
        target_size=(150, 150),
        batch_size=32,
        class_mode='binary')

# same es the train_generator
validation_generator = test_datagen.flow_from_directory(
        'data/validation',
        target_size=(150, 150),
        batch_size=32,
        class_mode='binary')

# loads sequentially images and feeds them to the model.
# the batch size is set in the constructor
model.fit_generator(
        train_generator,
        samples_per_epoch=2000,
        nb_epoch=50,
        validation_data=validation_generator,
        nb_val_samples=800)


Negative learning implementation in pytorch I have read a paper on Negative Learning: https://arxiv.org/abs/1908.07387.
The idea is that you can train a network not only by telling what label of the sample is, but by telling what it surely is not.
Let's call the latter a "negative" label.
An excerpt from the paper says (top formula is for usual "positive" label loss (PL), bottom - for "negative" label loss (NL):

I have a problem where "negative" labels collection is much easier than labeling each sample. So it is tempting to use it.
Is there some implementation of such a loss function in pytorch? Or should I write a custom loss layer code? If so, how should I do it?

One implementation is
( (loss+loss_neg) / (float((labels>=0).sum())+float((labels_neg[:,0]>=0).sum())) ).backward()

from NLNL-Negative-Learning-for-Noisy-Labels GitHub repo.


How to interpret the output of XGBoost importance? I ran a xgboost model. I don't exactly know how to interpret the output of xgb.importance.

What is the meaning of Gain, Cover, and Frequency and how do we interpret them?

Also, what does Split, RealCover, and RealCover% mean? I have some extra parameters here

Are there any other parameters that can tell me more about feature importances?

From the R documentation, I have some understanding that Gain is something similar to Information gain and Frequency is number of times a feature is used across all the trees. I have no idea what Cover is.

I ran the example code given in the link (and also tried doing the same on the problem that I am working on), but the split definition given there did not match with the numbers that I calculated.

importance_matrix

Output:

           Feature         Gain        Cover    Frequence
  1:            xxx 2.276101e-01 0.0618490331 1.913283e-02
  2:           xxxx 2.047495e-01 0.1337406946 1.373710e-01
  3:           xxxx 1.239551e-01 0.1032614896 1.319798e-01
  4:           xxxx 6.269780e-02 0.0431682707 1.098646e-01
  5:          xxxxx 6.004842e-02 0.0305611830 1.709108e-02

214:     xxxxxxxxxx 4.599139e-06 0.0001551098 1.147052e-05
215:     xxxxxxxxxx 4.500927e-06 0.0001665320 1.147052e-05
216:   xxxxxxxxxxxx 3.899363e-06 0.0001536857 1.147052e-05
217: xxxxxxxxxxxxxx 3.619348e-06 0.0001808504 1.147052e-05
218:  xxxxxxxxxxxxx 3.429679e-06 0.0001792233 1.147052e-05

From your question, I'm assuming that you're using xgboost to fit boosted trees for binary classification. The importance matrix is actually a data.table object with the first column listing the names of all the features actually used in the boosted trees.

The meaning of the importance data table is as follows:

The Gain implies the relative contribution of the corresponding feature to the model calculated by taking each feature's contribution for each tree in the model. A higher value of this metric when compared to another feature implies it is more important for generating a prediction.
The Cover metric means the relative number of observations related to this feature. For example, if you have 100 observations, 4 features and 3 trees, and suppose feature1 is used to decide the leaf node for 10, 5, and 2 observations in tree1, tree2 and tree3 respectively; then the metric will count cover for this feature as 10+5+2 = 17 observations. This will be calculated for all the 4 features and the cover will be 17 expressed as a percentage for all features' cover metrics.
The Frequency (/'Frequence') is the percentage representing the relative number of times a particular feature occurs in the trees of the model. In the above example, if feature1 occurred in 2 splits, 1 split and 3 splits in each of tree1, tree2 and tree3; then the weightage for feature1 will be 2+1+3 = 6. The frequency for feature1 is calculated as its percentage weight over weights of all features.

The Gain is the most relevant attribute to interpret the relative importance of each feature.

The measures are all relative and hence all sum up to one, an example from a fitted xgboost model in R is:

> sum(importance$Frequence)
[1] 1
> sum(importance$Cover)
[1] 1
> sum(importance$Gain)
[1] 1


What is "data scaling" regarding StandardScaler()? I'm trying to figure out the purpose of StandardScaler() in sklearn.

The tutorial I am following says

  "Remember that you also need to perform the scaling again because you had a lot of differences in some of the values for your red and white [wines]"

So I looked up the function in the sklearn docs.

  "Standardize features by removing the mean and scaling to unit variance"

  https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html

What good would removing the mean do?
What is scaling the data? Hard to google that.

# Scale the data with `StandardScaler`
X = StandardScaler().fit_transform(X)

I will use k-Nearest Neighbor algorithm to explain why we must do scaling as a preprocessing step in most machine learning algorithms.

Let's say you are trying to predict if I transaction is fraudulent or not, that is, you have a classification problem and you only have two features: value of transaction and time of the day. Both variables have different magnitudes, while transactions can vary from 0 to 100000000 (It is just an example), and time of the day between 0 to 24 (let's use only hours).

So, while we are computing the nearest neighbor, using euclidean distance, we will do

distance_class = sqrt(
    (new_value_transaction - old_value_transaction)**2) +
    (new_time_of_day - old_time_of_day)**2)
)

Where old is the reference to our train data and new is related to a new transactions we want to predict the class.

So now you can see that transactions will have a huge impact, for example,

new_value_transaction = $100
new_time_of_day = 10

old_value_transaction = $150
new_time_of_day = 11

class_distance = sqrt(($50)**2) + (1)**2)

Now, you have no indication that transaction value is more important than time of the day, that is why we will scale our data.

Between the alternatives, we can have a lot of different, such as MinMaxScaler, StandardScaler, RobustScaler, etc. Each of them will treat the problem different. To be honest? Always try to use at least two of them to compare results.

You can read more about in the sklearn documentation: https://scikit-learn.org/stable/modules/preprocessing.html#preprocessing

I hope you got the feeling why we should use standardization techniques. Let me know if you have any further questions.

To complement, here is a visual explanation of what I explained above.

Credit: https://www.youtube.com/watch?v=d80UD99d4-M&list=PLpQWTe-45nxL3bhyAJMEs90KF_gZmuqtm&index=10

In the video they give a better explanation. Also, I highly recommend this course, the guys are amazing.


Visualizing software metrics I have the below sets of data per application, you can call them as software metrics. These metrics vary depending on the size of an application.

Bugs
CodeSmells
Vulnerability

The size of the application is determined by LOC (Lines of code), how can i showcase the complexity of each app relative to the lines of code if i visualize each of these parameters.

  Example

           Bugs   LOC
SweetApp   10     10000
SourApp    120    5660000
SaltyApp   55     1500

How do i visualize Bugs per app in relation to the LOC, because LOC determines their complexity a higher number in bugs doesn't necessarily be bad considering a higher number in LOC.

You could use the ratios of bugs (or any other variable) divided by lines of code (LOC). Since ratios may vary a lot for instance bugs vs. vulnerabilities the resulting plot oftentimes doesn't look as good so you could use a normalization procedure, in fact, is the recommended procedure.

In your case Bugs and a few vulnerabilities divided by LOC doesn't look too bad so I included the plot of raw ratios too in this example. I'm using a count overlapping points plot which is a variant of a barplot that doesn't fill bars and only cares about the value as a point. You could use any normalization that you feel gives you the best result. Here I'm only centering the data. In both plots the highest value means higher complexity according to the ratio metric.

Raw Ratios of variables divided by LOC

Normalized Ratios of variables divided by LOC

Code in R needed to replicate these plots

require("reshape2")
require("ggplot2")

df1 <- data.frame(App = c("SweetApp", "SourApp", "SaltyApp"),
                  LOC = c(10000, 5660000, 1500),
                  Bugs = c(10, 120, 55),
                  CodeSmells = c(50, 30, 20),
                  Vulnerabilities = c(2, 3, 10))

#Define ratios
df1$RatiosBugs <- df1$Bugs / df1$LOC
df1$RatiosCodeSmells <- df1$CodeSmells / df1$LOC
df1$RatiosVulnerabilities <- df1$Vulnerabilities / df1$LOC

#Normalize Ratios
df1$NormRatiosBugs <- scale(df1$RatiosBugs, scale = FALSE)
df1$NormRatiosCodeSmells <- scale(df1$RatiosCodeSmells, scale = FALSE)
df1$NormRatiosVulnerabilities <- scale(df1$RatiosVulnerabilities, scale = FALSE)

dfRaw <- df1[, c("App", "RatiosBugs", "RatiosCodeSmells", "RatiosVulnerabilities")]
dfNorm <- df1[, c("App", "NormRatiosBugs",
              "RatiosCodeSmells", "RatiosVulnerabilities")]
dfRaw.m <- melt(dfRaw, id.vars = c("App"))
dfNorm.m <- melt(dfNorm, id.vars = c("App"))

#Plot Raw Ratios
ggplot(dfRaw.m, aes(App, value)) +
  geom_count(aes(color = variable), position = position_dodge(width =  0.4), stat = "identity")

#Plot Normalized Ratios
ggplot(dfNorm.m, aes(App, value)) +
  geom_count(aes(color = variable), position = position_dodge(width = 0.4), stat = "identity")


when x is a vector, derivative of vector diag(f'(x)) is formal notation? https://web.stanford.edu/class/cs224n/readings/gradient-notes.pdf

(4)

this note says this

$$
\frac{\partial \textbf{z}}{\partial \textbf{x}} = \text{diag}(f'(\textbf{x}))
$$

I know this means make a vector to square matrix which has an original vector as diagonal component.

but, since $\textbf{x}$ is a vector, $f'(\textbf{x})$ can't be a vector, derivative of a vector should be a matrix, right? Is this a formal notation for element-wise derivative of vector?

You are partially right and partially wrong: $f'(\textbf{x})$ is a matrix, but $\text{diag}(f'(\textbf{x}))$ means taking the diagonal of that matrix and making a vector out of it.


How curvature information in second order optimization methods helps It is said that second order optimization methods in neural networks work better than first order because they contain information about rate of change of gradient or the curvature. This information helps to choose a better step size for moving forward on the error surface. It is not clear as to how the rate of change of gradient is controlling the step size and leading to better optimization. For simplicity, consider only one weight update iteration.

You need to consider two steps of your first order optimisation process to see why a second order method can be usefull. (For more clarity we'll work in one dimension).
First step : calculate the derivative and move your evaluation point accordingly.
Then, second step : calculate the derivative and move your evaluation point accordingly.
If the derivative in the second step is bigger / lower than in the first step, you'll move less/more in the second step than in the first step. So the rate of change of the derivative along your path impact the change of your step size. So if you had information on the rate of change of the derivative in the frist step it may have been better to incorporate it directly in your first step calculation. In the end it is rather intuitive that the second derivative may able to find better suited step sizes.
In practice it is a bit more complex as the actual gain in performance depends on how the second order derivative is calculated / approximated. (See here)


How to evaluate the "betterness" of competitive good models? How to evaluate the "betterness" of competitive good models?

Lets say I could get good models (> 90% prediction rate) with e.g.:

LinearSVC
F-test based sklearn.feature_selection.f_regression
mutual info -based sklearn.feature_selection.mutual_info_regression

But since these treat e.g. the independence/dependence of features differently, particularly e.g. LinearSVC assumes "relatively independent" features, where as mutual info particularly measure dependence between variables, then

How can I compare these models to each other?

Tests? Knowledge of the data a priori? Something else?

The typical approach would be to compare (cross-)validation performance of the models, at least if accuracy is the selection criterion.

Other criteria could be

simplicity/ease to interpret
how easy the models could be implemented and/or updated in a productive environment


What is a good hardware setup for using Python across multiple users This question is likely somewhat naive.  I know I (and my colleagues) can install and use Python on local machines. But is that really a best practice?  I have no idea.

Is there value in setting up a Python "server"?  A box on the network where we develop our data science related Python code.  If so, what are the hardware requirements for such a box?  Do I need to be concerned about any specific packages or conflicts between projects?

Is installing Python locally a good practice?  Yes, if you are going to develop in Python, it is always a good idea to have a local environment where you can break things safely.

Is there value in setting up a Python "server"?  Yes, but before doing so, be sure to be able to share your code with your colleagues using a version control system.  My reasoning would be that, before you move things to a server, you can move a great deal forward by being able to test several different versions in the local environment mentioned above.  Examples of VCS are git, svn, and for the deep nerds, darcs.

Furthermore, a "Python server" where you can deploy your software once it is integrated into a releasable version is something usually called "staging server".  There is a whole philosophy in software engineering — Continuous Integration — that advocates staging whatever you have in VCS daily or even on each change.  In the end, this means that some automated program, running on the staging server, checks out your code, sees that it compiles, runs all defined tests and maybe outputs a package with a version number.  Examples of such programs are Jenkins, Buildbot (this one is Python-specific), and Travis (for cloud-hosted projects).

What are the hardware requirements for such a box? None, as far as I can tell.  Whenever it runs out of disk space, you will have to clean up.  Having more CPU speed and memory will make concurrent builds easier, but there is no real minimum.

Do I need to be concerned about any specific packages or conflicts between projects?  Yes, this has been identified as a problem, not only in Python, but in many other systems (see Dependency hell).  The established practice is to keep projects isolated from each other as far as their dependencies are concerned.  This means, avoid installing dependencies on the system Python interpreter, even locally; always define a virtual environment and install dependencies there.  Many of the aforementioned CI servers will do that for you anyway.


What's a good F1-score in (not) extremely imbalanced dataset? I have a dataset with around 4.7K focused on binary classification. Class proportion is 33:67. meaning Label 1 is 1558 (33%) and Label 0 is 3154 (67%) of my dataset.

Is my dataset imbalanced? some people say it is not bad

My objective is to increase the F1-score only. I set the class_weight=balanced in my parameters and scoring=f1 during CV as shown below.

svm=SVC(random_state=42)
svm_cv=GridSearchCV(svm,param_grid,cv=5,scoring='f1')
svm_cv.fit(X_train_std,y_train)

Can you let me know through a code sample on how I can increase the weightage to minority class? if that is any different from choosing balanced paramter

Currently my results are like as follows

I understand AUC for few algo is above 80 but I believe F1-score is more important for a imbalanced class problems like mine.

Can you help? I tried Oversampling minority class but not much improvement.

Increasing features doesn't take me to 80% F1-score

I would say you data in not imbalanced. 33:67 is not a bad ratio but Try using under sampling of majority class.
As another option you can try difference algorithms like randomforest. You can also try boosting.


Generation/Synthesis of Data with CNNs Is it possible to use a trained CNN to generate data? After training on data of X and corresponding Ys, given a new Y to generate X?
Or do I have to use GANs or RNNs? I'm still pretty new to the subject so I would be happy to have some leads, if I have to go in this direction.

Currently I am learning tensorflow  and trying to implement the "Convolutional Sequence to Sequence Learning" paper

Clarification: I want to use a CNN to generate sequences of text. I have a reallyb big database of descriptions and also one of 1000 classes, and wanted to try it out with CNNs (since im coming from image classification)

In general, if you want to generate data in X, or X, Y pairs, then you should start by training a generative model - as opposed to a discriminative model, which is what most NN classifiers are. There are many types of generative model.

Variational Autoencoders (VAE)and Generative Adversarial Networks (GAN) have been demonstrated recently with interesting results on images - although both take a lot of training data and time, and are limited to relatively small image dimensions (e.g. 128x128). There are many sub-types of these two designs, including a combined VAEGAN which attempts to combine strengths of both.

With a trained CNN, you can generate data - sort of. What you can do is start with some arbitrary value of X and Y, then use back propagation to calculate gradients of a cost function. But instead of using the gradients to update weights, you back propagate all the way to the input, and use the gradients at the input to alter X, repeating the process multiple times.

This is essentially how Deep Dream and Style Transfer work (although in general these don't use a Y value, but selected activation values within layers). There is a major caveat to this approach - your generated X will not be sampled evenly from any distribution of X that the network has been trained with. Instead you will generate a "super stimulus" X for the given Y.

You mention RNNs. One way these can be used to generate X is by sampling from their output and feeding this back into the input. For text sequences this tends to generate grammatically correct nonsense. I am not sure if this would be considered a strictly generative model, since it is not clear to me whether the input X is being sampled evenly. It is likely that you could use the approach to generate images too, although you would have to take care defining what the sequence is (just a sequence of pixels line-by-line will probably not produce any recognisable image).


Getting unexpected result while using CountVectorizer() I am trying to use CountVectorizer() in a loop, But I am getting an unexpected result. On the other hand, if I use it outside the loop then it works fine. I believe there is some small problem with the logic.

from sklearn.feature_extraction.text import CountVectorizer

keys_1 = ['funny', 'amusing', 'humorous', 'hilarious', 'jolly']
keys_2 = ['horror', 'fear', 'shock', 'panic', 'scream']
keys_3 = ['romantic', 'intimate', 'passionate', 'love', 'fond']

text = ('funny amusing fear passionate')

for i in range(3):
    keys = 'keys_' + str(i+1)
    cv = CountVectorizer(vocabulary = keys)
    data = cv.fit_transform([text]).toarray()
    print(data)

cv1 = CountVectorizer(vocabulary = keys_1)
data = cv1.fit_transform([text]).toarray()
print(data)

cv2 = CountVectorizer(vocabulary = keys_2)
data = cv2.fit_transform([text]).toarray()
print(data)

cv3 = CountVectorizer(vocabulary = keys_3)
data = cv3.fit_transform([text]).toarray()
print(data)

Output

  [[0 0 0 0 0 0]]

  [[0 0 0 0 0 0]]

  [[0 0 0 0 0 0]]

  [[1 1 0 0 0]]

  [[0 1 0 0 0]]

  [[0 0 1 0 0]]

The problem lies in the line:

keys = 'keys_' + str(i+1)

Here, keys becomes a string variable while I guess you would expect it to take the value of the list you defined in the first lines..

Try instead with a dictionary:

my_keys= {
"keys_1" : ['funny', 'amusing', 'humorous', 'hilarious', 'jolly'],
"keys_2" : ['horror', 'fear', 'shock', 'panic', 'scream'],
"keys_3" : ['romantic', 'intimate', 'passionate', 'love', 'fond']
}
text = ('funny amusing fear passionate')

for i in range(3):
    keys = my_keys['keys_' + str(i+1)]
    cv = CountVectorizer(vocabulary = keys)
    data = cv.fit_transform([text]).toarray()
    print(data)


How would I chi-squared test these simple results from A/B experiment? I have results from an A/B experiment where users could do one of three things: Watch, Interact, or Nothing

My data is like this:

   Watch  | Nothing | Interact
A: 327445 | 271602  | 744702
B: 376455 | 140737  | 818204

I tried to use the chisquared test bundled with scipy. I'm completely new to data science, but I believe this is the evaluation metric I would want.

scipy.stats.chisquare([ [327445, 271602, 744702], [376455, 140737, 818204] ])
Power_divergenceResult(statistic=array([  3412.38826538,  41532.93339946,   3456.72996585]), pvalue=array([ 0.,  0.,  0.]))

This does not look like a valid result...

I even tried adding an expected frequencies options, but without success. Maybe I'm missing something, either about evaluating this type of data or just not using scipy correctly. Can anyone help me?

If what you're trying to answer is if the action taken by a user (watch, interact, nothing) is influenced by the group they are in (A or B) you can use the chi2 test of independence.

For that you can use the scipy.stats.chi2_contingency function:

a = [327445, 271602, 744702]
b = [376455, 140737, 818204]

chi2, pvalue, _, _ = scipy.stats.chi2_contingency([a, b])

In this case it returns a chi2 test statistic of 48376.48 and a p-value of 0.0, so the null hypothesis ("action is independent of group") is rejected.

You can also use the scipy.stats.chisquare function to arrive at the same results, but other than with the chi2_contingency function you'll have to calculate the "expected frequencies" yourself. The data you have recorded are your observed frequencies:

obs = np.array([a, b]).astype(float)

(Note that I converted the numbers to float, as the chisquare function will run into some weird integer overflows otherwise...!?)

The expected frequencies are calculated like that:

exp = np.outer(obs.sum(axis=1), obs.sum(axis=0)) / obs.sum()

Finally, calling

chi2, pvalue = scipy.stats.chisquare(obs.ravel(), exp.ravel(), ddof=sum(obs.shape)-2)

returns the same chi2 test statistic and p-value as before.


Is not having overfitting more important than overall score (F1: 80-60-40% or 43-40-40)? I've been trying to model a dataset using various classifiers. The response is highly imbalanced (binary) and I have both numerical and categorical variables, so I applied SMOTENC and Random oversampling methods on Training set. In addition, I used a Validation set to tune the models parameters by GridSearchCV(). As both precision and recall were important for me, I used f1 to find the best model.

I should note that I selected these three subsets by cluster analysis and extracting samples by stratified train_test_split() from each cluster; so I have more confident that the subsets have more similarity.

Due to complex nature of Decision Tree and Random Forest or boosting techniques, I usually get high fitting (high f1 score) on Training set, relatively high on Validation set, but moderate to low on Test set.

The general sign for overfitting is the high difference between Training and Test sets (or between Validation and Test sets in my problem); but I am confused how to select the best model in following cases:

Case A: Training fit is very high; but Validation and Test sets fit are low but close to each other

Case B: Training, Validation, and Test fits are similar; but much lower than Case A.

                            F1 Score
   Model             Train     Val    Test
----------------------------------------
A: SVC                80.1     60.3    37.5
B: MLPClassifier:     43.2     40.0    39.1

I know that Case A might be the best model,however there is no guarantee that is produces similar result for new data, but which model do you pick with regards to overfitting?
(assume that precision and recalls for both models are similar)

The best model in your case is B not just because it scored higher on the test set, but because it showed very little sign of overfitting.
By not overfitting as much and being more consistent in its scores (train/val/test), you know what you're getting from the model. If you try it tomorrow on a secondary test set, you'd expect similar results.

Model A on the other hand is very inconsistent. If you evaluated this model on the secondary test set, you can't tell how much to expect. It could be higher, it could be lower... Generally speaking, if you've overfit as much on the validation set, it's a good indication to start over (re-split the dataset into train/val/test randomly, preprocess, fit the model). This time try less hyperparameter options though.

From the numbers I see even though model B currently is better than A, A might have the capacity to be better than B, if properly trained. I'd suggest re-doing model A's training from scratch, but with more regularization and less hyperparameter-tuning.


How to further improve the kaggle titanic submission accuracy? I am working on the Titanic dataset. So far my submission has 0.78 score using soft majority voting with logistic regression and random forest. As for the features, I used Pclass, Age, SibSp, Parch, Fare, Sex, Embarked.

My question is how to further boost the score for this classification problem?

One thing I tried is to add more classifiers for the majority voting, but it does not help, it even worthens the result. How do I understand this worthening effect?

Thanks for your insight.

big question.

Ok so here's a few things I'd look at if I was you..

Have you tried any feature engineering ?(it sounds like you've just used the features in the training set but I can't be 100%)
Random Forests should do pretty well, but maybe try xgboost too? It's quite good at everything on Kaggle. SVM's could be worth a go also if you're thinking of stacking/ensembling.
Check out some of the tutorials around this competition. There's hundreds of them and most of them are great.

Links:

R #1 (my favourite)

R #2

Python #1

Python #2

...Hopefully this helps


Does Fasttext use One Hot Encoding? In the original Skipgram/CBOW both context word and target word are represented as one-hot encoding.

Does fasttext also use one-hot encoding for each subword when training the skip-gram/CBOW model (so the length of the one-hot encoding vector is |Vocab| + |all subwords|)? If they use it, do they use it in both context and target words?

A quick answer would be No.

Let's walk through how FastText works internally:

For representation purposes, FastText internally initializes a dictionary. Dictionary contains all the collection of words. Besides words, it also maintains the count of all the words in the dictionary (and other information). Every time a new word is added to the dictionary its size is increased and word2int_ is updated to size_(which is increased after assignment).

The code below adds a word into the dictionary.

// adding of new word
void Dictionary::add(const std::string& w) {
  int32_t h = find(w);
  ntokens_++;
  if (word2int_[h] == -1) {
    entry e;
    e.word = w;
    e.count = 1;
    e.type = getType(w);
    words_.push_back(e);
    word2int_[h] = size_++; //  word2int_[h] is assigned a uniuqe value
  } else {
    words_[word2int_[h]].count++; // word's count is being updated here
  }
}

// funciton used to access word ID (which is the representation used)
int32_t Dictionary::getId(const std::string& w) const {
  int32_t h = find(w);
  return word2int_[h];
}

As mentioned in this medium article, word2int_ is indexed on the hash of the word string, and stores a sequential int index to the words_ array. The maximum size of word2int_ vector can be 30000000.

For embeddings matrix of M x N is created where M = MAX_VOCAB_SIZE + bucket_size. Where, M is total vocabulary size including bucket_size corresponds to the total size of array allocated for all the n-gram tokens and N is the dimension of the embedding vector, which means the representation for one word requires the size of 1.

The code below shows how to compute the hash and calculates the ID of the subword. A similar logic is used to access the subword vector. Note here h is an integer value which is calculated using dict_->hash(). This function returns the same h value which is used when adding a word in the dictionary. This makes the process of accessing word IDs only dependent on the value of h.

int32_t FastText::getSubwordId(const std::string& subword) const {
  int32_t h = dict_->hash(subword) % args_->bucket;
  return dict_->nwords() + h;
}

void FastText::getSubwordVector(Vector& vec, const std::string& subword) const {
  vec.zero();
  int32_t h = dict_->hash(subword) % args_->bucket;
  h = h + dict_->nwords();
  addInputVector(vec, h);
}

Long story short, FastText makes use of integer IDs assigned at the beginning and use those IDs to access the embeddings.

I hope this helps. All the code samples are taken from FastText repo. Feel free to dive in to understand more.


Why should I care about seasonal data when I forecast? I have a timeseries with hourly gas consumption. I want to use ARMA/ARIMA to forecast the consumption on the next hour, basing on the previous. Why should I analyze/find the seasonality (with Seasonal and Trend decomposition using Loess (STL)?)?

"Because its there".

The data has a seasonal pattern. So you model it. The data has a trend. So you model it. Maybe the data is correlated with the number of sunspots. So you model that. Eventually you hope to get nothing left to model than uncorrelated random noise.

But I think you've screwed up your STL computation here. Your residuals are clearly not serially uncorrelated. I rather suspect you've not told the function that your "seasonality" is a 24-hour cycle rather than an annual one. But hey you haven't given us any code or data so we don't really have a clue what you've done, do we? What do you think "seasonality" even means here? Do you have any idea?

Your data seems the have three peaks every 24 hours. Really? Is this 'gas'='gasoline'='petrol' or gas in some heating/electric generating system? Either way if you know a priori there's an 8 hour cycle, or an 8 hour cycle on top of a 24 hour cycle on top of what looks like a very high frequency one or two hour cycle you put that in your model.

Actually you don't even say what your x-axis is so maybe its days and then I'd fit a daily cycle, a weekly cycle, and then an annual cycle. But given how it all changes at time=85 or so I'd not expect a model to do well on both sides of that.

With statistics (which is what this is, sorry to disappoint you but you're not a data scientist yet) you don't just robotically go "And.. Now.. I.. Fit.. An... S TL model....". You look at your data, try and get some understanding, then propose a model, fit it, test it, and use the parameters it make inferences about the data. Fitting cyclic seasonal patterns is part of that.


Very Fast Training After First Epoch I trained an InceptionV3 model using plant images. I used Keras library. When training was started, first epoch took 29s per step and then other steps took approximately 530ms per step. So that made me doubt whether there is a bug in my code. I checked my code several times, but its logic seems right to me. I trained my model on Google Colab. I wonder whether there is a memoization mechanism or my code contains bugs. Here my code:

# Yields one image-target pair when called
def image_target_generator(files, labels):
assert len(files) == len(labels), 'Files and labels sizes don\'t match!'

for step in range(len(files)):
    img = cv2.imread(dataset_path + files[step])
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    item = (img, labels[step])
    yield item

# Generating batch
 def batch_generator(gen):

    batch_images = []
    batch_targets = []

    for item in gen:

      if len(batch_images) == BATCH_SIZE:
        yield batch_images, batch_targets
        batch_images = []
        batch_targets = []

      preprocessed_img = preprocess_image(item[0])
      batch_images.append(preprocessed_img)
      batch_targets.append(item[1])

    yield batch_images, batch_targets

# Training generator
def training_generator(files, labels):

  # So that Keras can loop it as long as required
  while True:

    for batch in batch_generator(image_target_generator(files, labels)):
      batch_images = np.stack(batch[0], axis=0)
      batch_targets = keras.utils.np_utils.to_categorical(batch[1], NUM_CLASSES)
      yield batch_images, batch_targets

# Create model
def create_model():
  model = keras.applications.InceptionV3(include_top=False, input_shape= IMG_SIZE, IMG_SIZE, 3), weights='imagenet')

  new_output = keras.layers.GlobalAveragePooling2D()(model.output)
  new_output = keras.layers.Dense(NUM_CLASSES, activation='softmax') (new_output)
  model = keras.engine.training.Model(model.inputs, new_output)

  for layer in model.layers:
    layer.Trainable = True

    if isinstance(layer, keras.layers.BatchNormalization):
      layer.momentum = 0.9

  for layer in model.layers[:-50]:
    if not isinstance(layer, keras.layers.BatchNormalization):
      layer.trainable = False

  return model

# Compiling model
model = create_model()

model.compile(loss='categorical_crossentropy',optimizer=keras.optimizers.adamax(lr=1e-2), metrics=['accuracy'])

# Fitting model
model.fit_generator(
  training_generator(train_x, train_y),
  steps_per_epoch=len(train_x) // BATCH_SIZE,
  epochs = 30,
  validation_data=training_generator(test_x, test_y),
  validation_steps=len(test_x) // BATCH_SIZE
  )

Keras supports lazy execution. The create_model and model.compile code are not executed until it is absolutely required which is right before the first training epoch. That increased time for the first epoch includes building the TensorFlow computational graph based on the plan in your create_model function. All remaining epochs re-use the same computational graph which is why they are significantly faster.


Deep learning: Training from COCO/imagenet weights or from a class-specific trained weights when classify 2 similar objects I have a question about the possible outcome of a trained model. Imagine that I would like to classify 2 different models of Ferrari and the dataset of these 2 models is small (for example, a few hundred images per model).

In the Keras blog, the issue was discussed but in the example, they are classifying dogs and cats, 2 class that are very general, distinct between them, and there are many cats and dogs already included in the original imagenet model (lets say the car/bus/truck are not included in the imagenet output classes).

Will it be better start training the model from Coco/Imagenet weights or start training using the weights of a model previously trained to classify cars only?

If you have a previously trained model to classify only cars, you can use it considering the fact that you have less data. The trained model for cars classification that you have already, would have been fine-tuned on the Imagenet or COCO in the first place. This way I think you would be able to train the model in less amount of time compared to fine tuning on Imagenet weights. Try both the methods once and try to find out the amount of performance increase in terms of time and accuracy over each other.


Is too much or very few training sample of a specific feature hamper the neural network model? I am analysing a technique "Sherlock" - a semantic type of column detecting technique wherein training dataset too many samples of a specific type are limited up to 15K and too few occurring samples exist less than 1K per class also excluded. What is the reason behind this? What are the disadvantages having too much or very few samples of a specific type in the input of a neural network?

Theoretically speaking, there aren't any disadvantages to having too much or too few data. It will only reflect in the overall performance of your model. Based on the Sherlock paper, it seems that it's a choice they made for their preprocessing. This is their explanation:

Certain types occur more frequently in the VizNet corpus than
others. For example, description and city are more common
than collection and continent. To address this heterogeneity,
we limited the number of columns to at most 15K per class and
excluded the 10% types containing less than 1K columns

They did this to reduce the overall imbalance of their dataset.


Alternatives to Heat Maps for Communicating Daily Data I am currently using the heat map below to communicate the number of messages sent on a messing platform over time. I don't think this is the most effective way to communicate the information as it's difficult to read and see trends. Although, I really like the ability to compare by week day (most of the messages are not send on weekends and I'd like to communicate this). What is a good alternative to heat maps that maintain the ability to communicate the weekday? Thanks for your help!

Thanks macaw_9227, that's a great idea. Below are two other ideas I'm considering.


Problems in building model predicting probablity of student's admission I'm trying to build a model predicting the probability of a student's admission in Russian educational system. There are exams for every school subject. The student can take several of them. Based on the results of the exams, the student could be admitted to some faculty (for example, Chemistry, Computer Science, Economics, Politics).

I use scikit-learn to build the model. I tried Naive Bayes, Random Forest and another classification estimators, but there was no much difference. I used method predict_proba to get probabilities, and a dataset with scores for every taken exam as features and faculty as target (faculty is treated as class)

The main problem is that when adding more taken exams, the probability of faculty decreases, while it should increase or not change.

My guess is that the model gives the most popular faculties among students with such scores, not probability of admission. Is there something I can do to improve the model?

Make sure you model the $Y$ data in the correct way.

Some reasons I can think on why do you have such strange results:

You are modelling binary Yes/No answers for all the students vs all the faculties, make sure you only take into account students which presented to the given faculty (some of them passed and some didn't), not all the students.
Sometimes more exams add noise to your answer, because the faculties look for specialized individuals who outperformed in what they are looking for, and that means you will find that the other results are "noise" for the answer.


Label A records B times or label A*B records This question concerns pre-training data sourcing.

Suppose you have a human workforce of B individuals and a potentially unlimited source of data.

The task is labeling images with classes. These classes are somewhat subjective (emotions). This means one individual might label the same image with a different class than another individual.

For then using these labeled records as training data on a neural network that predicts classes on images, is it better to

1) have a number of records (A) labeled redundantly by all B individuals.

2) have every individual label A different records each, yielding A x B labeled records.

Intuition behind 1) is that the mean of subjective labeling would be somewhat objective. Thus training data would be mostly objective.
In addition, probabilities (50% happy, 50% surprised) could be used as input.

Intuition behind 2) is that subjectiveness in labeling of individuals is natural and the NN is trained on that, becoming somewhat "general"/"objective" in it's predictions. Also, more data is always better.

Please excuse the use of subjective and objective in combination with Machine Learning. I know this might not be correct at all.

In my opinion there's no correct answer between the two choices, and you are right about the arguments.

I would argue for a balanced compromise between the two: have a proportion of A (say 20%) labelled by multiple annotators (say at least 3), but the rest of A can be labelled by a single annotator. You could refine the proportion/number of annotators progressively based on the results: for instance if after the first batch one observes that annotators disagreement is very high, then it might be useful to increase the two parameters. This way you can still evaluate the level of disagreement and it effect on performance, while also maximizing the amount of data annotated.

An alternative is to use some form of active learning. For example, first use method 2 (each instance labelled once), then use cross-validation to tran a model and apply it to the corresponding test set. At the end of the process, the misclassified instances are the "hard ones" which need to be re-annotated by multiple annotators. This kind of process can be used iteratively to identify the most "ambiguous" instances.


What is the best algorithm/solution for predicting the following? I have a dataset that comprises 76 countries, and 6 columns of distinct quantitative variables, which are the mean values of that variable relative to each country:

If I were to take a random sample of the 6 variables - an individual within one of the countries - how would I best go about predicting to which country that individual belongs?

I have a whole separate dataset, with thousands of data points, so I know which country the data belongs to and can thus know for certain whether the algorithm is predicting accurately.

Thus far I have been looking at decision trees and random forest options, but none of the example use cases I've seen translate very well to what I am trying to do. Perhaps I'm looking in the wrong place.

It's not predicting the behavior, as the behavior is already known...it's more about predicting to which country-classification the behavior belongs. Ideas? Comments?

Very interesting question indeed.

I will dare to give you an answer, but I could be changing it if I find a better reasoning about the problem after watching the data.

Your problem is perhaps a problem of multi(many)-label classification, where your label is the country and your data is what you have as "patience", "risktaking", etc.

You could work with random forest or decision trees but on the dataset containing all the individuals (not the summarised by country).

Why on this dataset and not on the summarised? Because the summarised is summarised (!), and only has the data for the mean of "risktaking" (for example) and does not take into account the internal variance of risktakingness inside the countries.

Is important to take this into account, because is possible that risktaking has a variance so large inside a country that is makes the variable meaningless to select a nationality for a person and this is not seen in the summarised database.

If is mandatory that you run the model in the summarised dataset first, then the solution is this:

You should run a tree where your node_size parameter is 1, because your summarised dataset has "one individual (country) per label", this is a extreme case where you need a node leaf for every country.


how to shuffle the data for model.fit with custom data generator? So trainfiles is a list that contains the files' directory and name e.g. ['../train/1.npy' ,  '../train/2.npy']
and then I create a dataset as shown in the middle of the code
then I apply it to model fit function

just adding shuffle=True in the argument for model fit function doesn't do anything it seems.
How do I go about shuffling the trainfiles correctly? i.e. when does the trainGenerator function gets executed? i.e. does it gets executed 516 times  when generating 1 batch?

can I just add random.shuffle(trainfiles) right above the for-loop in trainGenerator?

The big issue here is that your generator yields after each file is loaded. This means that your batch size is always the number of examples stored in each file and that the training examples in each batch are always the same.

You can absolutely shuffle the order of the files, but that's just shuffling the batch order around and not shuffling data examples among batches, which is very important for the optimizer. (Each batch yields an estimate of the gradient of the loss function with respect to the model parameters given the particular batch data. The more combinations of examples you're able to give the network, the better you're able to estimate the gradient with respect to the training set.) There are lots of ways to try to fix this, the Tensorflow docs have tutorials for some of them. (You could load a random set of files and deliver batches that contain their data intermixed, then load more, etc)

Also, you have 75,000 files that appear to be (40, 40, 15). That means the whole dataset is (3 000 000, 40, 15) if all the files are the same size, which is ~14.4 GB if you're storing the data as np.float64.  That's big, but not ridiculous. You could probably get a significant chunk of that in memory at once, which is probably better.

Is this float data? If so, do you need that level of precision (np.float64, the default)? I'm guessing probably not.  If you can store it as np.float32 or even np.float16, you might be able to just get the whole thing in memory. If not, you could load it as bigger chunks (half the dataset at once, maybe) and intermix examples among them. Just something to think about.

EDIT:

Also, if you're going to concatenate them all, don't do it in a loop with a big array and iteratively concatenate the smaller arrays:

# NO

big_array = np.load("first_train_file.npy")

for train_file in rest_of_train_files:
    tmp_file = np.load(train_file)
    np.concat((big_array, tmp_file), axis=0)

# YES

list_of_little_arrays = []

for train_file in all_train_files:
    tmp_file = np.load(train_file)
    list_of_little_arrays.append(tmp_file)

all_data = np.concatenate(list_of_little_arrays)

You're going to have loads of memory problems using the first way. I don't know that that's what you tried or not, but just in case.


Lazy vs Eager Learning I wish to better understand the difference between lazy and eager learning. I am having difficulty conceptualising what the "abstraction" refers to between the two.

According to the text book I am reading it says, "The distinction between easy learners and lazy learners is based on when the algorithm abstracts from the data."

A lazy learner delays abstracting from the data until it is asked to make a prediction while an eager learner abstracts away from the data during training and uses this abstraction to make predictions rather than directly compare queries with instances in the dataset.

I understand that KNN algorithm loads all the data into memory so depending on the size of the dataset, the computing requirements could be enormous as well as the fact it's a non-parametric algorithm so there is no training to find parameters, the classification is done on a per query requirement??

Lazy learner:

Just store Data set without learning from it

Start classifying data when it receive Test data

So it takes less time learning and more time classifying data

Eager learner:

When it receive data set it starts classifying (learning)

Then it does not wait for test data to learn

So it takes long time learning and less time classifying data

Hint : In supervised learning

Some examples are :
Lazy : K - Nearest Neighbour, Case - Based Reasoning
Eager : Decision Tree, Naive Bayes, Artificial Neural Networks


An abstract idea for the performance diffs between SLP and MLP Recently I am working on some predictive analytic which based on neural network.

When I tried some tests on MLP with one hidden layer or multiple hidden layers, the results showed that:

one hidden layer performance on prediction is always better than multiple hidden layers
the tests were executed both on Knime and R models, they gave me the same trends

Based on my knowledge, more hidden layers with more neurons should perform better?

Is there a principle on this about for which kind of dataset turns out this kind of result?

Or I may need some book / article / paper to read?

Do you have any abstract idea (not mathematical algorithms) for me, thanks!

UPDATE

I am working on a fraud detection dataset, which includes 1000 observations. and all the dimensions are numericed and normalized...

But I am actually asking for a general idea, for the general idea to choose algorithm for different dataset

More layers/more neurons does not necessarily mean you will get better performance. If your data is too simple or the number of observations is not that high, then adding more parameters (more layers/neurons) may result in overfitting the data. During training, the network will try to represent the training data as closely as possible. When there are a good number of units in the network, then the representation that is learned by the network will mainly model the general trend of the data faithfully. But, if there are too many neurons, then it is possible that some of the neurons will simply model noise in the training data which does not generalise to unseen data, resulting in worse performance.

Also if you are using a saturating nonlinearity as an activation function (i.e. sigmoids or tanh) then adding too many layers may result in vanishing gradients which will cause your network to train very slowly, or perhaps not at all.

Designing neural networks that work well is not very easy, and the best way to get an intuition for what will work is to experiment. One thing to try is to evaluate a range of sizes for your hidden layer(s). If you are using sigmoid or tanh as your activation function, I suggest you try using rectified linear units as well.


Is feature engineering still useful when using XGBoost? I was reading the material related to XGBoost. It seems that this method does not require any variable scaling since it is based on trees and this one can capture complex non-linearity pattern, interactions. And it can handle both numerical and categorical variables and it also seems that redundant variables does not affect this method too much.

Usually, in predictive modeling, you may do some selection among all the features you have and you may also create some new features from the set of features you have. So select a subset of features means you think there are some redundancy in your set of features; create some new features from the current feature set means you do some functional transformations on your current features. Then, both of these two points should be covered in XGBoost. Then, does it mean that to use XGBoost, you only need to choose those tunning parameters wisely? What is the value of doing feature engineering using XGBoost?

Let's define first Feature Engineering:

Feature selection
Feature extraction
Adding features through domain expertise

XGBoost does (1) for you.
XGBoost does not do (2)/(3) for you.

So you still have to do feature engineering yourself. Only a deep learning model could replace feature extraction for you.


How to collect info about unseen bugs given user's comments/feedbacks? I have a dataframe which looks like:
user_id, comment
0, 'Functional but Horrible UI'
1, 'Great everything works well'
2, 'I struggled finding plus button because of theme colors in dark mode'
3, 'Keeps stopping on Android 10'
4, 'I like the functionaity but color theme could be better'
5, 'Consistently crashing. Uninstalled'
6, 'Good overall'
7, 'sfdfsdlfksd'
8, 'I lost in complex settings'
9, 'Configuring app is really a headache'
10, 'aaaaaaaaaaaaa'

And I want to figure out some data science approach to pluck out information about what users are struggling with and which issues appeared how much and stuff like this. Even some simple output would be good for me so that we know which parts of app to focus on more. Like for sample above I am aiming for an output as simple as:
problems = {
'color_theme': 3,
'app_settings': 2,
'crashing' : 2}

So I kinda wants labeling and how much time a label is occured based on to which label a review belongs. But the problem is I cannot train a model with predefined labels because:

I do not have labels for reviews. If we have to go through each review to know what problem is it talking about (i.e. to label it), we would just have filed it as well and would know what we have to work on.

I do not know in advance what problems are gonna come in future so even if we somehow label all at some point in time, it wouldn't be enough as some unseen problem may come and we have to do again.

Even if we have a system of labeling somehow, how would we update model, like do we define a new model with a different architecture for ever changing labels?

So under these circumstances, I was trying to figure out an AI approach to ease in my situation. I am pretty good at python and do have working knowledge of keras/tensorflow and other libraries but none of them seem to have such flexible model approach. I was going through Google Cloud Platform's AI platform as well but it could do sentiment analysis to an extent but not understand in an app context that e.g. button is a part of UI and color as well. So how could I approach this problem in a more elegant way?

It sounds like you are looking for an unsupervised learning approach (meaning you don't need to manually label your data).
Something like k-means clustering could work well. This would allow you to group you comments into k distinct clusters. You could then view counts of comments in those clusters and explore the clusters to determine their meaning.
In order to perform the clustering, you need to transform your data from text to a numerical vector space. A common approach would be tf-idf, but you may find that something else works better.
Since you mentioned Python, both k-means and tf-idf can be accomplished using sklearn:

sklearn.cluster.KMeans
sklearn.feature_extraction.text.TfidfVectorizer

There's a pretty nice example of k-means clustering using tf-idf on Kaggle.


Tensorflow RNN: Batching data of different length I'm starting to experiment with neural networks in Google's tensorflow framework, and in particular am working on a model that has an LSTM layer. I am using the RNN tutorial to get me started.

To train my model, I have sequences of data that are several thousand data points long. In the tutorial, the author is processing small sequences of data (short sentences of paragraphs) and concatenates those sequences together, separated by <end of stream>, divides the entire data set equally into batch_size number of chunks and processes those chunks in parallel. I would like to process each of my sequences of data separately from start to finish, but would like to be able to process several of them in parallel, as the rnn_cell objects are set up to do (they process batch_size sequences at onces).

This would be straight forward to do if the sequences were of the same length, but they are not. An rnn_cell.LSTMCell expects the data to come in with a fixed dimensionaltiy, batch_size x input_size, so there's a problem once some of the sequences reach their end and my batch_size wants to decrease.

How can I work around this? I don't think I can 'left' or 'right' justify my data so that they line up without affecting the training.

I'd suggest padding your sequences to all be the same length. Obviously this isn't ideal as you end up wasting time processing fake data but it does solve the problem.


Does this correlation make sense? I'm trying to work on a ML project and I have a dataset and I'm trying to see if there is a correlation between some of the features in my dataset.

The dataset contains inspection notes for car parts in an industrial process along with some information about the car parts themselves.

I have posted a Jupyter Notebook with all of my code on GitHub. The data used is in the same folder. But to get back to my question, I've plotted a simple correlation matrix to show how the features correlate.

I've done some data augmentation so that I have all numerical values and be able to calculate and plot the matrix.

But after giving this a thought and reading about correlation, I'm not sure that my the correlation makes much sense (in the real world).

If my understanding is correct, correlated variables would be useful in a

The below plot is a representation of correlation between variables for all of the car parts that have been thrown away after inspection.

However, I'm not 100% sure that my following statement is correct, based on the correlation matrix below:

"There seems to be some correlation between the MONTH and the MINUTE features in my dataset."

But although I believe the numbers show this, I don't think this is valuable information and after reading I believe that correlation would work best with measurement values, not inspection values.

I'm seeking help to confirm or help me improve my understanding of what correlation is and how it would apply in my scenario.

I have a hunch that it could be applicable, but I need to tweak my dataset. Like I need to do a count aggregate on all features and build the correlation matrix and plot that, but I'm not sure yet...

I see you have done encoding of feature line using the below code:

line_dict = {'INSPECTION':'0', 'REPAIR':'1' }

if we see the type of the value as below:

   >>> line_dict['INSPECTION']
       '0'
   >>> type(line_dict['INSPECTION'])
   <class 'str'>

it seems it is of object type. This could be the problem. Suggest you do to one hot encoding technique for this feature

Also it is good to an info on your dataset to identify the object columns

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 153381 entries, 0 to 153380
Data columns (total 12 columns):
bodyId        153381 non-null int64
bodyTypeId    153381 non-null int64
color         153381 non-null int64
line          153381 non-null object
shiftNo       153381 non-null int64
Inspection    153381 non-null object
mYear         153381 non-null int64
mMonth        153381 non-null int64
mDay          153381 non-null int64
mHour         153380 non-null float64
mMinute       153380 non-null float64
mSecond       153380 non-null float64
dtypes: float64(3), int64(7), object(2)
memory usage: 14.0+ MB

if you check the data type line  it says object which means the line feature is String


Jupyter Notebook not importing pandas module I am running Jupyter on a server on a virtual environment. I then tunnel my connection so I can access Jupyter on my browser.

When I SSH into the server, I can use the Panda module in both Ipython and Python3.

I ran this code in Ipython

 import pandas as pd

In [2]: print(pd.__file__)
/home/ubuntu/.local/lib/python3.6/site-packages/pandas/__init__.py

Then I tried adding it to my path in Jupyter with the code below, still no luck.

import os
os.getcwd()
import sys
sys.path.append('/home/ubuntu/.local/lib/python3.6/site-packages/pandas/__init__.py')

import pandas as pd

I used these instructions to setup the Virtual Environment https://www.digitalocean.com/community/tutorials/how-to-set-up-jupyter-notebook-with-python-3-on-ubuntu-18-04

Thank you very much in advance, this will help me with my school studies a lot.

I had to start the virtual environment, then install pandas module

source virtual_enviornment_folder/bin/activate
pip install pandas


SMOTE on training data The SMOTE could only be performed on the training data, so how can we do it using Weka? It means we have to put the training and test data in two separate files and run the SMOTE on the training file, so how can we load two datasets to Weka and perform these steps?

Even I'm also not sure about Weka, but so far I read, it provides functionality for data mining, data pre-processing, model training.

Generally what I do,

In case of pre-processing: model generated after pre-processing of training data(it includes SMOTE data), I save this model. Saved model is used to pre-process test data.

Let me know in case, you are expecting something different in answer.


correlation coefficient between weather data and power consumption I have power consumption and the corresponding temperature data for one year, that is, for several weeks (week1, week2,..., week48) as one month has 4 weeks. I want to forecast power consumption for a month.

Forecasting power consumption means forecasting 4 weeks(a month has 4 weeks). I am using Ms excel to obtain the pearson correlation coefficient. The power consumption data and temperature data have a strong correlation of 0.8 for week1 to week48, that is, they have a good correlation for one year. One year means over a large time interval.

But over a small time interval, for eg, correlation b/w power consump and temperature for 12 weeks (3 months) is weak (0.2). I want to use a neural network with window size of 3 to represent 12 weeks(3 months) to forecast the next 4 weeks(next month).

My question is: since the power data and temperature data has good correlation(0.8) for 48 weeks(1 year) but weak correlation(0.2) for 12 weeks(3 months), is it possible to use a neural network with window size of 3 as mentioned above to predict the power consumption for the next month with input variables power data and temperature data?

I am not very sure how effective NN would be for this problem. The way I see it is that you have 48 entries in a time series and now you are trying to predict the next 4. Keeping aside the correlations for a second, it seems like there is very little data to "learn" from. I mean the power consumption is a seasonal thing (much like temperature) but you have only 1 years data so it should be difficult for NN to capture this seasonality.

I would like to raise another point about the way you are using correlations. If you are taking correlations using the actual temperature value then I do not think that to be a good idea. Think of it in this way, I would be using a lot of power at both very high and very low temperature. So instead of using the raw temperature values, I should normalize them with respect to some "comfortable" temperature value by taking the absolute difference between the actual temperature and the "comfortable" temperature.

For instance, let us say that "comfortable" temperature is 25C, then both 5C and 45C would be normalized to 20 and this looks more plausible as my power consumption should be high in both the cases. This might also explain the poor correlation that you observe for 3 months of data. What should be the "comfortable" temperature is an entirely different story altogether.

Do let me know if this line of thought makes sense.


Multidimensional regression in Scala I have a queuing model written in Scala where different categories of people end up a different queues. We have a dataset providing a map of features to the numbers of people ending up at each queue, ie multiple inputs to multiple outputs (continuous values)

I have some experience using mllib for single value predictions in Scala but I can't see that multiple outputs are supported. It doesn't even look to me that mllib has continuous value output support as I can't see how to get a layer without an activation function.

Does anyone know of an ml library that can both support multidimensional regression (please correct me if I have the terminology wrong) and integrate with Scala?

Going forward I'd also love to try an RNN as our data has a time sequence element to it, but I believe this is even rarer in ml libraries.

A regression technique that allow to predict a multidimensional output can be the PLS ( partial least square ). I implemented it in scala and it will be soon available on Clustering4Ever repo. In fact we went a bit further by applying it with the clusterwise pattern which generate k-clusters driving by PLS regression which result with one regression model per cluster which can be much more accurate but also most costly due to approximatively 1000.n regressions. You can look on it with, A new micro batch approach for partial least square clusterwise regression.


How to replace values in a numpy array? I'm learning how to implement and evaluate a Logistic Regression Model, for this I need to change the values of my array from strings to 0 & 1.
I have the following numpy ndarray as a result of a DataFrame.values call
['PAIDOFF', 'COLLECTION', 'COLLECTION', 'PAIDOFF', 'PAIDOFF', 'PAIDOFF', ...]
I would like to know how can I change the values like:
'PAIDOFF' to 0 and 'COLLECTION' to 1
Any help will be much appreciated

import numpy as np

a = np.array(['PAIDOFF', 'COLLECTION', 'COLLECTION', 'PAIDOFF'])

f = lambda x: 1 if x == "COLLECTION" else 0

np.fromiter(map(f,a),dtype=np.int)

Alternative:
np.where(a == "COLLECTION",1,0)


Matrix Factorisation Improvement I am using SGD matrix factorisation (python) using the movielens dataset to make recommendations. I have a website which allows users to give feedback which is positive or negative to whether an item is a good recommendation for a particular movie.

I was wondering if I could use this feedback in my matrix factorisation. I wasn't 100% sure how I would proceed.

So for example I would have a vector like

  m1 m2 m3
m1 0 0  0
m2 5 0 -4
m3 0 0  0

Where m2-m1 is a score of 5 so with an example case where noone rated it negative, 5 people think its good. With the other for m2-m3 being -4 so a poor recommendation.

Any help would be greatly appreciated.

Edit: response from answer

I am currently using a sparse matrix for user item ratings and am using bias.

I have been trying to add additional input sources and am using the following to create a attribute matrix containing a genre representation:

for genre in item_genres:
       genres[genre] = 1
       self.attribute_item_matrix[i] = sum(list(genres.values()))

 pred = self.global_mean + self.bias_user[u] + self.bias_item[i] +np.dot(self.P[u, :],(self.Q[:, i]+self.attribute_item_matrix[i]))

However, I don't think this is correct as it eventually fails and causes a nan error.

I have been following this paper Matrix Factorisation Techniques for recommender systems

Thanks

In my view, you should better go for user-movie matrix, to create a sparse matrix that can be more meaningful. By doing that, if you already know polarity of choices i.e. positive, negative or neutral, you may also have some intensity score of like or dislike like in case of rating. Now, if you want to just classify them as likes, neutral, and dislike, you may assign 1,0 and -1 respectively. This method can have more meaning when you have many movies and many users. Further this can help you establish user-user, movie-user and movie-movie relationship as well. These relationship can potentially help you understand and explore more about recommendation engines.

     m1 m2 m3 m4
u1 -1   0  -1  1
u2  1   1  -1  0
u3  0   0   1  0
u4  -1  1   1   0

If you want to add rating instead of just polarity, just add numbers scaled 0-1. I personally worked on them to write hybrid recommendation engines which have both collaborative filtering and context based filtering models embedded. I would suggest you to work more on relationships and try to explore more about next steps like possibility of like or dislike bias, association of movie info/ actors-directors influence and similarity in movies etc. I hope this explanation helps.


Information Gain & Gini Index for NLP I know how Information Gain and Gini Index work in General.

I have problem figuring out how to apply these techniques in NLP and text feature extraction.

Can someone show me an example of how to implement these techniques on NLP.

Thanks

I will cite explanations from "A survey of Text Classification Algorithms" by Aggarwal and Zhai. The following and more can be read here in Chapters 2.1 (Gini Index) and 2.2 (Information Gain).
Gini Index

Let $p_1(w)...p_k(w)$ be the fraction of class-label presence of the $k$ different classes for the word $w$. In other words, $p_i(w)$ is the conditional probability that a document belongs to class $i$, given the fact that it contains the word $w$. Therefore, we have:
$\sum_{i=1}^{k} p_i(w)=1$
Then, the gini-index for the word $w$, denoted by $G(w)$ is defined as follows
$G(w)= \sum_{i=1}^{k} p_i(w)^2$
The value of the gini-index $G(w)$ always lies in the range $(1/k,1)$. Highervalues of the gini-index $G(w)$ indicate a greater discriminative power of the word $w$.

Information Gain

Let $P_i$ be the global probability of class $i$,and $p_i(w)$ be the probability of class $i$, given that the document contains the word $w$.  Let $F(w)$ be the fraction of the documents containing the word $w$. The information gain measure $I(w)$ for a given word $w$ is defined as follows:
$I(w) = - \sum_{i=1}^{k} P_i * log(P_i) + F(w) * \sum_{i=1}^{k} p_i(w) * log(p_i(w)) +$
$+ (1-F(w)) * \sum_{i=1}^{k} (1 - p_i(w)) * log(1-p_i(w))$
The greater the value of the information gain $I(w)$, the greater the discriminatory power of the word $w$.


How to build a recurrent neural net in Keras where each input goes through a layer first? I'm trying to build an neural net in Keras that would look like this:

Where $x_1$, $x_2$, ... are input vectors that undergo the same transformation $f$. $f$ is itself a layer whose parameters must be learned. The sequence length $n$ is variable across instances.

I'm having trouble understanding two things here:

What should the input look like?
I'm thinking of a 2D tensor with shape (number_of_x_inputs, x_dimension), where x_dimension is the length of a single vector $x$. Can such 2D tensor have a variable shape? I know tensors can have variable shapes for batch processing, but I don't know if that helps me here.
How do I pass each input vector through the same transformation before feeding it to the RNN layer?
Is there a way to sort of extend for example a GRU so that an $f$ layer is added before going through the actual GRU cell?

What should the input look like?

You are right to think a 2D tensor, but usually we add one more dimension for the batch. You can indeed have a variable length number_of_x_inputs, but to train during batch processing all inputs in a single batch will need to have the same shape. (Setting batch size to 1 will get around this.) During inference, you can have whatever length you want. See below code sample.

How do I pass each input vector through the same transformation before feeding it to the RNN layer?

Use TimeDistributed. The example below passes all vectors $x_i$ through the same feed forward network (Dense(5, ...)), but you should be able to swap that out for whatever f you were thinking.

from keras.models import Sequential
from keras.layers import LSTM, Dense, TimeDistributed

x_dimension = 16
num_classes = 2

model = Sequential()

model.add(TimeDistributed(Dense(5, activation='relu'),
          input_shape=(None, x_dimension)))
model.add(LSTM(32, return_sequences=True))
model.add(LSTM(8))
model.add(Dense(num_classes, activation='softmax'))

print(model.summary(90))

This prints the following model:

Layer (type)                            Output Shape                        Param #
==========================================================================================
time_distributed_1 (TimeDistributed)    (None, None, 5)                     85
__________________________________________________________________________________________
lstm_1 (LSTM)                           (None, None, 32)                    4864
__________________________________________________________________________________________
lstm_2 (LSTM)                           (None, 8)                           1312
__________________________________________________________________________________________
dense_2 (Dense)                         (None, 2)                           18
==========================================================================================
Total params: 6,279
Trainable params: 6,279
Non-trainable params: 0
__________________________________________________________________________________________


how to split available data into training and testing (Information security) I was advised to ask my question here.

Recently, I made a post about finding suitable dataset for SIEM (Security Information and Event Management) systems. The goal was to work on classification and correlation to detect security attacks.

I decided to use the dataset from the Honeynet Project Challenges. The problem is this: I don't know if I should use the whole dataset set for my project, because if you look, for example, at the KDD99 dataset, it is devised into two parts: 10% for training and 90% for testing.

I have seen some researcher use dataset A for training and dataset B for testing, do you have any other ideas? I am really stuck at the part of training vs. testing

If my question is too broad. I don't mind some reading materials that will help me deal with my dataset.

Bests,

You should check out k-fold cross validation techique. Its a quality control technique which makes sure, the prediction/classification done by a model would generalize well to unseen test data. The idea is to split the dataset into k times and use it recursively to train and cross validate your network which at end can be used on the test data to report the actual accuracy.

Here is a link for reference :
https://machinelearningmastery.com/k-fold-cross-validation/


Multi target regression when some targets have correct NA labels I'm working on a Keras neural net that does key point prediction of body parts (left foot, left knee, left hip, etc.). For each image (X), the target (Y) is a list of coordinates for the keypoints (left_foot_x, left_foot_y, left_knee_x, left_knee_y), etc.). Whenever a keypoint isn't visible the X and Y coordinate equal = 0.

I suspect that leaving the data in this state will lead to bad results because 0 really means NA (i.e. the keypoint isn't visible). Leaving the X and Y coordinates 0 would suggest the keypoint is actually in the corner which is false. What is the right way to structure data to do multi target regressions when the correct output is sometimes NA?

One option is to modify your network to output a binary label for each body part, signalling whether the part is visible or not.

You would need the following changes:

Modify your network architecture to return a real number in the range $[0, 1]$ for each body part. A sigmoid would be an appropriate activation for the last layer. Its semantics would be the probability that the $i^{th}$ body part is visible.
Extend your training and test data with appropriate values for the new labels.
Change your loss to only take into account the difference between body part positions (expected vs predicted) if the part was expected to be found (e.g. by multiplying). I'm not very sure whether you should use the real labels or the predicted ones here (or even an average of both).
Add a new term (added to the original loss) to your loss with the cross entropy of the body part presence flag. You should probably add a weighting factor $\alpha$ to this new loss term to have a proper scale with respect to the position distance loss term.


Unsupervised clustering of unstructured text by document type I have 100,000+ PDF healthcare documents from which I have extracted text. I would like to cluster these documents by type (e.g. pathology report, doctor visit notes, prescription orders, etc.)

The format and structure of the documents are unknown ahead of time. However, we can assume that the name of the document type occurs explicitly somewhere within the first 50 words of each document. E.g. A pathology report will contain the words "pathology report" somewhere near the top.

The contents of the documents are otherwise irrelevant. I simply want to cluster documents together by document type.

I do not know the names or number of document types ahead of time. I would like the algorithm to automatically determine the best number of clusters.

I'm not very experienced in machine learning, but I am a competent programmer. What's the best way to attack this problem? Thanks.

"pathology report, doctor visit notes, prescription orders" these are classes, not clusters. Clustering may as well find "people with headache", "patients on Thursdays", ... you do not get to control this unsupervised.

So I'd rather suggest to do a classification.

For example, put all documents that contain "patient report" in the first lines into a separate set. Inspect the remainder for another keyword. Add another rule. If the data is as explained, 10 such rules may get the leftover documents down to a manageable scale, and they could eventually just be treated as 'others'.

There is nothing wrong with inspecting documents and adding rules if that solves your problem. You probably don't need to make this work on different document collections (say, law texts) ever.