LSTM Time series prediction for multiple multivariate series I have to predict next min traffic for multiple cities (100+). I am thinking of using LSTM. My main concern is how do I scale the number of cities. How does LSTM learn different amount of traffic and other related features of all cities to predict the next state. What should be the network architecture for such cases.

I was thinking of the following process:

Normalisation of the data with city specific Min,Max scalar
Feed sliding window data(t_1 to t_60) to LSTM and predict (t+1) value
Take the output value and get the actuals values from step 1.

I have read multiple papers and blogs online but mostly the deal with one multivariate time-series. But, in my case its multiple multivariate time series but one generalised model. Can someone suggest what are common industry practises for these and related papers/blogs. Do I need to model for each city (Its not possible in my case because of scalability issues?

I think one question you need to answer is how the traffic is correlated across cities.

If they are not correlated (which is likely) then the city can be a categorical input variable. The network can learn what is common across all cities and what is specific to different cities.

Here is a reference that may be helpful. In this example, the inputs are clearly related, so that may be a difference in your case.

You also may consider developing a model for one city, and then improving it by adding other cities.


Using AWS ML to recommend products I have millions of user ratings on about 2k products. I want to use Machine Learning to analyse these ratings and recommend products to users based on other users ratings of the same and different products.

i.e. user A likes 1,2 & 3, user B likes 1&2 so probably likes 3 based on A's ratings.

Is this possible to do through AWS ML? I'm looking at AWS ML as I'm a developer, not a data scientist and looking to keep this simple.

I've been playing with this all day and think I have figured it out, but just looking for some further advise / guidance.

Based on my experimenting so far, the following format of dataset is what AWS needs

CustomerID, Prod1Rating, Prod2Rating, Prod3Rating,.....

And then I would create a model for each product with the target on each being the row containing that products ratings. To make a recommendations engine from this I then just need to loop through every product for a user and ask each model the score.

This is a classical example of collaborative filtering, in particular you're talking about user-based collaborative filtering.

I don't think Amazon ML supports collaborative filtering (https://forums.aws.amazon.com/thread.jspa?messageID=712158), but you can do it in R and run your R code on the cloud.

It's easy in R, take a look at the recommenderlab package. The vignette has an example with the MovieLens movie rating data set.

You may also use the Microsoft Azure ML framework as it has better support for recommender systems. Google "Azure recommender systems" will give you useful links.


How to process the hyphenated english words for any nlp problem? Im doing preprocessing on english text dataset. I encounter hyphenated words like 'well-known'. Will it be useful

if I remove the hyphen as special character and treat it as a single word 'wellknown' or
separate the word into 2 'well' and 'known' or
use all 3 words 'well' , 'known', 'wellknown' in vector creation(BOW/TF-IDF) process for model input.

Any quick help on this would be more appreciated. Thank you.

They all sound like interesting approaches. The first one is better I think because it allows for unseen hyphenated words to be somewhat understood (as e.g. well + known ~= well-known).
For a tfidf BOW model, you might get good performance from any of the above.
For a model that is sensitive to word order I would certainly go with the first option and might tokenise the text so that I had a token to represent the hyphen too.


How do I implement Dual-encoder model in Pytorch? I am trying to implement the paper titled Learning Cross-lingual Sentence Representations via a Multi-task Dual-Encoder Model.

Here the encoder and decoder share the same weights but I am unable to put it in code. Any links ?

There isnt any as of yet, but scout this website and as soon as it comes it will be there.


How to find patterns in a series of timestamps I have a series of timestamps that represent the time a user clicked a certain button.

My goal is to detect the automated clicks, so I need to find recurring patterns in the data that may point to an automated script. The majority of the data is by regular users. I don't need to detect them online, I just need to detect them in historical data. I also have user_id to make a distinction between users.

Is there a default approach to detecting recurring patterns in a largely unstructured dataset?

Most things I've found are about anomaly detection on a certain value paired with a timestamp but my problem is not really anomaly-detection as I need to find patterns amongst unpatterned data

The question can be super difficult if you only have the number of clicks per time-stamp. The reason is that you might find many different recurrent patterns!
For instance certain time periods in year, month or day (according to the functionality of website) may attract a certain temporal pattern of clicks.
But if you have more relevant features the story is different.

After all I will point out some directions and hope it helps:

Autocorrelation may initially give you a good insight about what is happening there.

Then go for Time-Warping methods to define similarity function and seek it along time-series.

Embedding in high-dimensional spaces disclose the temporal recurrent pattern of a time-series. It's mostly from physics literature and is called State-Space Recunstruction. This method transfers your time-series into a m-dimensional space using this and there you can see the time-series as a dynamic system which may help in detecting recurrent dynamics.


How to statistically prove that a column in a dataframe is not needed I have a pandas dataframe consisting of dimensions and features of a different fabric materials. I have several rows per product material type causing the dataset to seem very huge. From basic logic and domain knowledge I can judge that the columns that show dimensions of the fabric is not necessary to analyse the data I am looking for.

I tried to run a correlation and print a heatmap but it doesnt show any output for most of the columns. Only white color is printed out for most of the column correlation. This makes it difficult for me to judge.

Please advice if there is a better way to statistically prove the relationship between the columns and justify the reason to drop irrelevant columns

With respect to the other answers i observed your point about having features which are strings, you must first find a way to encode them as numerical features which can then help you with PCA and the problem with the heat-map will be resolved too. Also if your problem is of regression then you can use methods such as L1-Regularization which help you in feature selection and you don't have to remove the features while pre-processing.


multipying negated gradients by actions for the loss in actor nn of DDPG In this Udacity project code that I have been combing through line by line to understand the implementation, I have stumbled on a part in class Actor where this appears on line 55 here:  https://github.com/nyck33/autonomous_quadcopter/blob/master/actorSolution.py

# Define loss function using action value (Q value) gradients
        action_gradients = layers.Input(shape=(self.action_size,))
        loss = K.mean(-action_gradients * actions)

The above snippet seems to be creating an Input layer for action gradients to calculate loss for the Adam optimizer (in the following snippet) that gets used in the optimizer but where and how does anything get passed to this action_gradients layer? (https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adam)  get_updates function for optimizer where the above loss is used:

# Define optimizer and training function
optimizer = optimizers.Adam(lr=self.lr)
updates_op = optimizer.get_updates(params=self.model.trainable_weights, loss=loss)

Next we get to train_n a K.function type function (also in class Actor):

self.train_fn = K.function(
            inputs=[self.model.input, action_gradients, K.learning_phase()],
            outputs=[],
            updates=updates_op)

of which action_gradients are the (gradient of the Q-value w.r.t. actions from the local critic network, not target critic network).

The following are the arguments when train_fn is called:

    """
    inputs = [states, action_gradients from critic, K.learning_phase = 1 (training mode)],
    (note: test mode = 0)
    outputs = [] are blank because the gradients are given to us from critic and don't need to be calculated using a loss function for predicted and target actions.
    update_op = Adam optimizer(lr=0.001) using the action gradients from critic to update actor_local weights
"""

So now I began to think that the formula for deterministic policy gradient here:  https://pemami4911.github.io/blog/2016/08/21/ddpg-rl.html
would be realized once the gradient of the Q-values w.r.t. actions were passed from the critic to the actor.

I think that the input layer for action_gradients mentioned at the top is trying to find the gradient of the output of actor w.r.t. to parameters so that it can do the multiplication pictured in the photo. However, to reiterate, how is anything passed to this layer and why is the loss calculated this way?

Edit:  I missed a comment on line 55

# Define loss function using action value (Q value) gradients

So now I know that the action_gradients input layer receive the action_gradients from the critic.
Apparently it is a trick used by some implementations such as Openai Baselines:  https://stats.stackexchange.com/questions/258472/computing-the-actor-gradient-update-in-the-deep-deterministic-policy-gradient-d

But still, why is the loss calculated as -action_gradients * actions ?

This answer is not mine but a reply I received on Quora here:  https://www.quora.com/Why-is-the-loss-for-DDPG-Actor-the-product-of-gradients-of-Q-values-actions/answer/James-MacGlashan?filter=&nsrc=2&snid3=4129828389

Hopefully it helps someone rather than me just deleting the question and I am sorry for my lack of math jax knowledge.

Let’s forget about DDPG, Q-functions, actors, and actions for a minute. I know, know, blasphemy, but bear with me for a moment and we’ll get back to that.

Let’s just talk about loss functions. We’re all good with loss functions, right? They’re the backbone of our standard supervised deep learning literature. That is, in supervised learning, what we do is we define a loss function that defines how bad a prediction is from what it should be.

As a grounding example, let’s revisit the L2 loss we use for a regression problem. We define this loss as:

L
2
(
p
,
l

)

1
2
(
p
−
l
)
2

Where
p
 is a prediction from our neural net and
l
 is the “label:” the value the prediction should have been. If we can tune our neural net parameters so that this loss is always zero, we win! In other words, we want to minimize this loss function.

And how do we minimize it? The standard way these days is to use some flavor of stochastic gradient descent (SGD). And to do that, what we want to do is differentiate our loss in terms of our neural-net parameters
w
. If we used vanilla SGD, we’d do:

w
t
+

1

w
t
−
α
∇
w
L
2

Okay, but there’s a catch here, our loss function was in terms of
p
, not
w
. However, we describe our neural net predictions as a function of some input and the neural-net parameters w:
p
(
x
,
w
)
 for which we know how to compute
∇
w
p
. Since we know that, we can use the chain rule of differentiation, which says:

∇
w
L

2

∇
w
p
∇
p
L
2

Where since w is a vector,
∇
w
p
 will refer to its (transposed) Jacobian matrix.

In our modern era of deep learning libraries, you typically don’t do that yourself. Instead, your library will use the autograd/backpropagation algorithm to compute
∇
w
L
2
 for you, and it will do it by decomposing it slightly differently than we did above.

Despite that fact that modern autograd libraries will do things slightly differently, it will be important to see that we can express the gradient of our loss in terms of neural net parameters in that way.

Back to Actor Critic

Okay, now that we’ve refreshed our memory about loss functions and what applying the chain rule to a loss function looks like, let’s get back to Actor Critic and DDPG.

Let’s ask our selves: What do we really want actor critic to do?

Suppose someone handed you the actual Q-values for some policy. How can you improve that policy? Simple: by making a new policy that selects actions that maximize the Q-values you were handed. This is the foundation of RL. It’s why in Q-learning you set your policy to greedily select the action with the max Q-value (and maybe add some noise for exploration reasons).

Okay, lets assume I have a Q-function: how do I find a policy that maximizes the Q-function if the actions are continuous?? I can’t do what Q-learning does by simply evaluating each action and choosing the max, because there are an infinite number of actions to evaluate!

Wait a second though. We know how to maximize a function over continuous values. We’ve been doing it for ages in supervised learning! Note that maximizing a function is the same as minimizing the negative value of that function. That is:

arg
max
x
f
(
x

)

arg
min
x
−
f
(
x
)

Okay cool, so if I want to maximize some function, I can just use regular SGD on the negative value of that function.

From that, we now have an insight: what if we think of
−
Q
(
s
,
a
)
 as a loss function, where the actions
a
 are “predictions” from some neural net: the actor. In that case, we can use good ol’ SGD to train our actor to choose actions that maximize the Q-values.

To do so of course requires that we have a representation of Q that we can differentiate. But if we simultaneously train a neural net to estimate Q, and we believe its estimates are good**, then we can easily use it as the “loss” function for our actor!

Indeed, if -Q is our loss function, with actions
a
 acting as our “predictions” from our neural-net actor model
μ
 lets substitute that back into chain rule expression for loss functions that we wrote earlier and with which we’re familiar:

∇
w
−

Q

∇
w
μ
∇
a
−
Q

And there it is!

(You may note that the DDPG paper reverses the order of the multiplication from the above. The order really just depends on what convention the paper takes the gradients and Jacobian matrices to mean. If you go back to the original deterministic policy gradients paper, you’ll see they write it out as we did above.)

So why doesn’t DDPG just say to use autograd on -Q like we do with loss functions ?

Okay, so now you’re probably asking yourself:

Why didn’t the paper just say “use autograd on -Q”???

After all, in supervised learning papers we never write out the chain rule in that way, we just assume practitioners have an autograd library and only write the forward version of the loss.

The reason is if you’re not careful, running pure autograd will burn you!

In DDPG, recall that we’re training two neural networks: Q and
μ
. If you simply feed
μ
 into Q and run autograd, it’s going to produce gradients on the parameters of Q in addition to the parameters on
μ
! But when optimizing the actor, we don’t want to change the Q function! In fact, if we did, it would be really easy to make any actor maximize Q by making Q always output huge values!

To run autograd through Q would be like letting a supervised learning algorithm change the values of labels, which clearly is not right!

You can’t simply “block gradients” on Q either, because then no gradients will flow back onto the actor.

So instead, you simply have to make sure you apply the chain rule through Q to
μ
, without adjusting any of the parameters of Q. One way to do that is to compute the gradients of each manually and multiply them, as written in the paper.

There are other ways to do it in autograd libraries too, but now you’re starting to get really specific about which library you’re using. So for an academic paper, it’s better to simply write out precisely what gradient should be computed for the actor, and then let the practitioner decide the best way to compute it in their library of choice.

** This assumption that a neural net’s estimates of the Q-function are good is really important. Because it’s an estimate, it will have errors, and a limitation of the DDPG algorithm is that your actor will exploit whatever errors exist in your neural net’s estimate of Q. Consequently, finding ways to ensure the Q-estimate is good is a very important area of work.


GMM in speech recoginition using HMM-GMM I am trying to solve/understand ASR using HMM-GMM.

At the abstract level i do understand what's happening but I did not
understand how GMM fits into it.

My data has 5K hours of speech from single user. I took the above picture from this article.

I do know what is GMM but i am unable to wrap my head around it.
Can somebody explain with a simple example.

The previous answer was wrong so I removed it.

Here goes my second attempt after reading Speech and Language processing by daniel Jurafsky and James H Martin(good book to read).

The 39 features associated with an observation/acoustic is considered to have come from mixtures of multivariate gaussian.

Why Mixture of MV gaussian ? Assuming a single MV gaussian for each state(phones) is a strong assumption which might not be true.

How does HMM comes into picture with GMM in ASR: Consider an uni-variate case where a single cepstral feature(usually it is 39) is represented by a single gaussian and HMM state has a mean value and variance which generate the particular observation. To get which observation was produced by which state is a part of decoding problem.

Let me know if this is right ?


How can I augment my image data? What are the correct and common ways to normalize image for CNN?
I used to work with text and it was pretty straightforward. Removing stop words, clean text from noise, tokenization, stemming etc. There is no problem with length we can easly add padding. I am not sure how to treat images.
Grey scale is much better, less data as input.
Should I rotate each image for random angle? How to pick right size of image in my traing dataset? What about adding additional filters?

(my goal is to classify 2 classes with tensorflow)

You begin by asking about image normalisation, but then refer to other techniques, which I believe all fall under "image augmentation". So I will answer the more general question:

  how can I perform image augmentation to improve my model?

I would generally say that the more augmentation you can apply, the better. A caveat to that statement is that the augmentations must make sense for your target case. I will explain at the end what I mean by that.

Let's list some ways to augment an image, along with some common values:

Augmentations

1. Flipping

Starting with perhaps the easiest to implement, you can (easily) flip images:

horizontally: makes an image of an arrow pointing left point in the opposite direction - to the right, and vice-versa
vertically: makes an arrow pointing down point up, and vice-versa.

It is more difficult, due the general rectangular shape of images, but you could flip an image diagonally, but I haven't seen anyone do that really. It is getting close to the idea of rotation, I suppose.

2. Rotation

In this case, we simply rotate the image about its center by a random number of degrees, usually within a range of [-5, +5] degrees. Even tiny small angles, which we can maybe hardly perceive, this is a simple trick that can add a lot of rubustness to a model.

3. Crop

We take "chunks" of an image, either at random or using a pre-defined pattern. This helps the model perhaps focus on certain areas of images and not be overwhelmed by perhaps unimportant features. It also simply creates more data.

You might also incorporate this augmentation method as a means to resize your input images to fit into a pre-trained model, which took input images of a different size to your data.

4. Normalisation

This is more of a numerical optimisation trick that one which can really be interpreted from a visual perspective. Many algorithms will be more stable whilst working with smaller numbers, as values are less likely to explode. Numbers closer together (in the sense of continuous space with a linear scale) are also more liekly to produce a smoother optimisation path.

One way of performing normalisation is:

first computing a mean and standard deviation of the images, then
(from all images) we subtract the mean and divide by the standard deviation.

This will produce images whose pixel values that lie over some range, but have a mean value of zero. Have a look here for some other methods and more discussion. You can normalise e.g. over whole images or just over the separate colour channels (RedGreenBlue).

One thing to keep in mind, is that the values used for optimisation must come from the training dataset itself, and must not be computed over the entire dataset including the validation/test datasets. This is because that information should not be passed to the model in any way - it is kind of cheating.

5. Translations

SImply move the image up, down, left or right by some amount, such as 10 pixels, or again within a range of [-5, +5] % of the image size. This will produce pixels along either one or two axes, which must be filled by another colour because the picture has moved out of the frame. It is common to use black or white for these "empty" parts, but you could also use the average pixel value or even crop the shifted image to remove them.

6. Rescaling

This is a less obvious augmentation step, but can add value because the model will be able to extract different features (more or less general) from larger and smaller images. A small image without much detail would only allow the model to learn higher level, or hazier features and not be able to focus on specific details that are available in high resolution images.

Methods such as progressive resizing, commonly used for training GAN architectures, start with smaller images and slowly work up to larger images. This can be done to keep training as stable as possible (in GANs it can increase robustness against mode collapse), but also coincides with the notion that early layers in a network learn high-level features, and layers deeper into e.g. a convolutional network learn representations that are more detailed. Why do the early layers require high resolution images? Let's give them low resolution images, train quicker and hopefully generalise slightly better!

7. Be creative!!

I once made a model for a self-driving car. I knew that the track where the model would be tested had many trees and quite a few high walls, which oth created shadows over the road. These shadows looked a lot like straight lines representing the edges of the road! So I added shadows to my images, at random intensities and random angles with random sizes. The car learnt that a change in brightness over a straight line did not necessarily mean the edge of the road and hence stopped freaking out when it reached one!

Think about the artefacts of your training data and the situations your model will face in the validation data, and try to incorporate them. This is really like feature engineering, incorporated into a model via augmentation... and it can help a lot!

8. Examples

Just to add a nice picture, here is a great display of how interesting variations can be made from a single image, taken from the Keras tutorial, linked below. The original image is top left and the remaing 7 are the output of combinations of a set of random augmentations:

You can have a look at an article like this, which explains augmentation quite well, with examples.

Another great example here, part of a series of articles, goes through augmentation possibilities in some detail.

Implementations

1. Keras: ImageDataGenerator

This is a single class, where you can say which augmentation steps to apply and (if relevant) how much. It allows very easy integration of augmentation into a model.

Check out the official Keras tutorial for some great examples with images.

2. PyTorch: Transforms

You can build a small pipeline of transformations, gathering them together into a Compose object, to have fine-grained control over each augmentation step, its parameters and with which probability it is applied.

Have a look at this tutorial for an example of combining Rescale and RandomCrop.

3. Augmentor: Augmentation library:

This standalone library allows you build up great augmentation pipelines, which then hand off the images into a model. There are a great number of possibilities implemented, such as random warping/distortion, which looks really funky:

It also look as easy to use as the Keras solution shown above!

Going back to the start

Now going back to my opening statement, the reason we want to apply as many augmentations as possible, is because it is synthetically creating more data for our model to learn from. We are trying to teach the model a distribution of possible data inputs and make it learn how to produce the correct output. We usually do this on a training set and then ask for its predictions on a validation set, which contains unseen data. The more data that the model has seen from the distribution/function that generates the data, the less surprised and better it will perform on the "unseen" data. We want our model to be as robust as possible to make kinds of alterations to data, and exposing it to more of that helps.

The caveat, that we only use relevant augmentations, is necessary because as soon as we start using augmentations that skew the input distribution somehow (e.g. showing the model images that make no sense1).

So use as much augmentation as possible, within the constraints of reality for your problem; as well as the time it takes to train the model  ;-)

1 - Imagine training a model to detect cars on a road. If we perform a vertical flip, images shown to the model will contain cars that are essentially driving upside down; "on a grey the sky". This really doesn't make sense and will never occur in a realistic validation set or in the real world.


Is there any way of ordering/sorting vectors? I am working on KD-Tree for nearest neighbor algorithm, where at each level of the tree we arbitrarily choose a dimension to cut upon and sort the points based on that chosen dimension value, after which we can choose mid-point to split upon such that dividing the points from that middle point will give us a balanced tree (has same number of points in either of the subtrees). At each level a single dimension is chosen and process is repeated.

For example: for data having dimensions (x,y,z) let's say first level of the tree 'X' is chosen at cut dimension, 'Y' at second level, 'Z' at third and then repeated again with 'X' again at fourth level and so on. So for first level all points are sorted according to dimension 'X' and split into left and right subtree based on the middle value of the sorted data.

I am sure you got the gist of it. Otherwise please read more here. KD-Tree (Wiki)

Now what I want to try something where rather than using each dimension at different level, I use all at the same level. i.e. I take points p(x,y,z) as vectors. But as you can guess if I do this, I have no way of ordering (not any I know of) the vectors (points) such that I can find a vector to divide upon hence still keeping my tree somewhat balanced.

PS: I am not much concerned about if it is actually a better or worse way of doing thing. Right now it's more about if it is possible to do it this way and then find out why it is better or worse.

Please ask if I should provide any more information or if I mentioned something wrong. I appreciate your help.

Please add more tags if appropriate, I couldn't find any.

Edit: Just to make it clear. Basically I want to be able to have an ordering which won't destroy the order based on distances. i.e. if a point/vector A has less distance from another point P, it comes first  in the order than some farther point, let's say B.
My end goal is to use this to try and create some kind of nearest neighbor algorithm.

If I understand your question correctly, you're looking for something that another (beyond KD-trees) standard space partitioning algorithm does. This one is called BSP-trees (for Binary Space Partition) and it uses vectors instead of dimensions for its space subdivision/partitioning. You can see it contrasted to KD-trees in slide 29 (previewed below) and beyond of this presentation.

The criterion that is used in the algorithm is intended to balance the data in the branches of the tree. This guides the definition of an order on vectors for what seems to be your purpose as well.

Otherwise, more generally speaking, one way to define a total order on a vector space is by identifying an order vector. Then your ordering is defined by the dot product with this vector. This represents projecting the entire space on the line defined by the vector and thus using the total order of the one-dimensional space it defines. For a more detailed explanation of the dot product, consider this video.


In XGBoost, how is a leaf index corresponding to the particular leaf node in actual base learner trees? I've trained a XGBoost model for regression, where the max depth is 2.
# Create the ensemble
ensemble_size = 200
ensemble = xgb.XGBRegressor(n_estimators=ensemble_size, n_jobs=4, max_depth=2, learning_rate=0.1,
                            objective='reg:squarederror')

ensemble.fit(train_x, train_y)

I've plotted the first tree in the ensemble:
# Plot single tree
plot_tree(ensemble, rankdir='LR')

Now I retrieve the leaf indices of the first training sample in the XGBoost ensemble model:
ensemble.apply(train_x[:1]) # leaf indices in all 200 base learner trees

array([[6, 6, 4, 6, 4, 6, 5, 5, 4, 5, 4, 3, 5, 4, 5, 3, 6, 3, 5, 5, 3,
3,
3, 5, 4, 4, 3, 4, 3, 6, 6, 6, 4, 6, 6, 3, 5, 3, 5, 4, 6, 4, 4, 6,
3, 3, 6, 3, 6, 3, 4, 3, 6, 6, 3, 6, 5, 3, 6, 6, 3, 4, 6, 5, 3, 3,
3, 6, 3, 4, 3, 6, 3, 6, 3, 3, 3, 4, 6, 3, 4, 4, 6, 3, 3, 6, 3, 6,
6, 3, 3, 4, 4, 4, 3, 3, 6, 6, 3, 3, 6, 3, 3, 3, 6, 6, 6, 4, 4, 3,
5, 3, 3, 3, 4, 5, 3, 3, 6, 3, 3, 6, 3, 4, 5, 3, 6, 3, 5, 3, 4, 4,
3, 3, 4, 6, 6, 6, 6, 3, 4, 4, 3, 5, 6, 6, 3, 5, 3, 3, 6, 6, 3, 3,
6, 3, 3, 4, 4, 3, 4, 3, 5, 3, 3, 3, 3, 3, 4, 4, 6, 3, 6, 4, 4, 5,
6, 3, 4, 5, 6, 3, 4, 3, 4, 5, 6, 6, 5, 4, 3, 3, 6, 6, 3, 6, 5, 4,
3, 3]], dtype=int32)

Here is my question:

Since there are four leaf nodes in the first tree, how come there is
index 6 for the first training sample?

In the official doc for apply(), it says "Leaves are numbered within [0; 2**(self.max_depth+1)), possibly with gaps in the numbering." So if max_depth is 2, the leaves are numbered between 0 and 7. Since there are only four leaves in a binary tree of depth 2, shouldn't the leaves numbered within [0, 4)? What is the reason behind the design $[0; 2^{(self.max\_depth+1)})$?

Related question: https://stackoverflow.com/questions/58585537/how-to-interpret-the-leaf-index-in-xgboost-tree

I think what you are seeing is the fact that all nodes in the tree are indexed because a priori the model doesn't know where splits will happen (i.e. any node could be a leaf). My guess is that the nodes follow an ordering similar to:

In your case all of the leaf nodes are at the max depth of the tree, so nodes 3-6 show up in your list. By contrast, if your data was all the same value I would expect all labels to be node 0 (because the split criteria was not met). And then you could have intermediate situations where after 1 split there is a node which does not meet the split criteria (in this case you could see either node 1 or 2 show up in your list).
Hope this helps!


Can we use BERT for only word embedding and then use SVM/RNN to do intent classification? According to this article, "Systems used for intent classification contain the following two components:
Word embedding, and a classifier." This article also evaluated BERT+SVM and Word2Vec+SVM.
I'm trying to do the opposite, comparing two different classifiers (RNN and SVM) using BERT's word embedding.
Most Python codes that I found use BERT for the whole intent classification problem which made me confused. Example
I only want to map the words into vectors with BERT and feed the result into a classifier (SVM/RNN).
Does BERT support word embedding and text classification at the same time? Does someone has an explanation? Is what I'm trying to test feasible with Python?
I have a dataframe that has two columns: intent and questions. It's a small dataset.
Thank you!

BERT is a transformer based model. The pre-trained BERT can be used for two purposes:

Fine-tuning: This involves, fine-tuning with the new data to a specific task such as classification or question-answering, etc. Here, the BERT itself acts like a classifier.
Extracting embeddings: Here, you can extract the pretrained embeddings. The difference between Word2Vec (or other word embeddings) and BERT is that BERT provides contextual embeddings, meaning, the embeddings of each word depends on its neighbouring words. However, since it's contextual embeddings, we can make an assumption that the fist token which is '[CLS]' captures the context can be treated as sentence embeddings as be used as input to 'SVM' or other classifer. But, in the case of RNN, you may want to take the pretrained embeddings of each token to form a sequence.

So, how you want to use BERT still remains a choice. But if you can fine-tune the BERT model, it would generally yield higher performance. But you'll have to validate it based on the experiments.


XGBRegressor hyperparameter optimization using xgb cv function I am trying to optimize hyper parameters of XGBRegressor using xgb's cv function and bayesian optimization (using hyperopt package). Here is the piece of code I am using for the cv part.

dtrain = xgb.DMatrix(X_train, label=y_train)
cv_results = xgb.cv(params,dtrain,num_boost_round = 1000, folds= cv_folds,
                        stratified = False, early_stopping_rounds = 100, metrics="rmse", seed = 44)

However, I am getting the following error within the xgb.cv function (part of the Trace):

414     cvfolds = mknfold(dtrain, nfold, params, seed, metrics, fpreproc,
--> 415                       stratified, folds, shuffle)
    416
    417     # setup callbacks

/anaconda3/envs/py36/lib/python3.6/site-packages/xgboost/training.py in mknfold(dall, nfold, param, seed, evals, fpreproc, stratified, folds, shuffle)
    261         except TypeError:
    262             # Custom stratification using Sklearn KFoldSplit object
--> 263             splits = list(folds.split(X=dall.get_label(), y=dall.get_label()))
    264             in_idset = [x[0] for x in splits]
    265             out_idset = [x[1] for x in splits]

AttributeError: 'int' object has no attribute 'split'

I can't figure out why am i getting this error. The documentation for xgboost is also not very clear and sparse.
So any help would be greatly appreciated.

Thanks

It seems to be complaining about the folds.split part, indicating that folds is an integer.  If your cv_folds object is just the number of folds, pass that as nfolds instead; folds expects the actual sklearn KFold object rather than a number.


Model's loss weights I've model with two output layers, age and gender prediction layers. I want to assign different  weight values for each output layer's loss. I've the following line of code to do so.

model.compile(loss=[losses.mean_squared_error,losses.categorical_crossentropy], optimizer='sgd',loss_weights=[1,10])

My question is what is the effect of loss weights on performance of a model? How can I configure the loss weights so that the model can perform better on age prediction?

This will affect how the backpropagation for each of these outputs will cause the intermediate nodes within the network to be updated. If the output nodes' error were equal then the gradient descent process of the intermediate nodes will favor the resulting error of each of these outputs equally. By more heavily weighting an output node it means that the gradient descent will favor a set of parameters that perform better on that output node.


Which are the appropriate prameters for lda modeling? I try to implement in R test for appropriate metrics for lda.

Here the way I try to use LDA

require(quanteda)
    require(quanteda.corpora)
    require(lubridate)
    require(topicmodels)
dtext <- data.frame(id = c(1,2,3), text = c("This dataset contains movie reviews along with their associated binary sentiment polarity labels. It is intended to serve as a benchmark for sentiment classification. This document outlines how the dataset was gathered, and how to use the files provided.", "The core dataset contains 50,000 reviews split evenly into 25k train and 25k test sets. The overall distribution of labels is balanced (25k pos and 25k neg). We also include an additional 50,000 unlabeled documents for unsupervised learning.", "There are two top-level directories [train/, test/] corresponding to the training and test sets. Each contains [pos/, neg/] directories for the reviews with binary labels positive and negative. Within these directories, reviews are stored in text files named following the convention [[id]_[rating].txt] where [id] is a unique id and [rating] is the star rating for that review on a 1-10 scale. For example, the file [test/pos/200_8.txt] is the text for a positive-labeled test set example with unique id 200 and star rating 8/10 from IMDb. The [train/unsup/] directory has 0 for all ratings because the ratings are omitted for this portion of the dataset."),stringsAsFactors = F)

    corp_news <- corpus(dtext)

    dfmat_news <- dfm(corp_news, remove_punct = TRUE, remove = stopwords('en')) %>%
        dfm_remove(c('*-time', '*-timeUpdated', 'GMT', 'BST')) %>%
        dfm_trim(min_termfreq = 0.95, termfreq_type = "quantile",
                 max_docfreq = 0.1, docfreq_type = "prop")

    dfmat_news <- dfmat_news[ntoken(dfmat_news) > 0,]
    dtm <- convert(dfmat_news, to = "topicmodels")
    lda <- LDA(dtm, k = 10)

Which are the parameters which I have to test to make the perfomance of lda better?
example there methods for the topic selection number
algorithms like gibbs, recover and recoverl2
what other tunning it needs?

The main and by far most impactful parameter that you have to search for is k - the number of topics you assume. Conceptually, this is similar to K Nearest Neighbors, in which you have to pre-specify the number of clusters you are searching for and then determine the best fit clusters by properties of the clustering results.

The other parameters, including the method of solving the mathematical problem may produce slightly different results, but none will be nearly as impactful as k.

On a more general note, you can almost always find the answer to this question for an implemented algorithm you're trying to use from the API documentation of the package in question- the topicmodels documentation containing the LDA interface can be found here.

Often, if you don't well understand the properties of the algorithm that you're trying to use, the documentation will also point you to resources which thoroughly describe the algorithm in question. It is this understanding of the algorithm and what it is doing that will enable you to answer the questions like "What parameters should I tune, over what values, and what is likely to change about the outputs when I do so?"


Does the bias of an artificial neuron adjust or remain constant during training? Forum contributor David Waterworth wrote: "we train $w_{0j}$", here $w_{0j}$ is the bias of an artificial neuron.

However, this Wikipedia article

https://en.wikipedia.org/wiki/Artificial_neural_network

(in section "Optimization", under subtitle "Algorithm")

seems to indicate that the bias does not adjust during training:

(I only quoted relevant sentences.)

"Let $N$ be a neural network with $e$ connections, $m$ inputs, and $n$ outputs.

$w_0, w_1, w_2,$ ... denote vectors in $R^e$. These are called weights.

The output of the algorithm is then $w_p$."

In other words, the training/optimization algorithm starts with $w_0$, then produces $w_1$, then produces $w_2$, etc.

Please note that $w_1, w_2, ...$, are $e$-dimensional vectors, and $e$ is the total number of connections.

So, does this Wikipedia article say that only the weights of the connections adjust, but the bias (weight of the neuron) does not adjust?

If the bias also adjusts, then the output vector of the training/optimization algorithm would have more dimensions - The number of dimensions would have to be:

[the total number of connections $e$] plus [the total number of neurons which have biases].

Perhaps the training/optimization algorithm has evolved since this Wikipedia article was written?

Neurons values are something like this

$N_i = w_{i1}x_{i1} + w_{i2}x_{i2} + ... + w_{in}x_{in} + b_i$

the bias is $b$ here

but it also can be written like

$N_i = w_{i0} + w_{i1}x_{i1} + w_{i2}x_{i2} + ... + w_{in}x_{in}$

where bias is $w_{i0}$

it is the weight of index 0

when mentioned weights in optimization it is by default include bias


Does bagging create iid trees? As the title suggests, I have a question regarding the trees produced through the bagging procedure.

Namely, since the bootstrap samples created to fit trees on are independent and identically distributed (iid), are the resulting trees also iid?

In other words, is there any reason there may be correlation between the "bootstrap trees"?

This question may be more suited to CrossValidated. However:

"Bagging" is short for "bootstrap aggregating", meaning that a random sample with replacement is taken from the overall set. The key here is "with". For example, suppose this is your dataset:

{1, 2, 3, 4, 5}

and you are interested in obtaining 4 samples of size 3. You could end up with the following:

{1, 4, 4}

{1, 4, 5}

{2, 3, 4}

{3, 3, 4}

That is, you can have repeated elements within the same bootstrap sample (e.g. the first result, {1, 4, 4}), and each bootstrap sample could contain the same element (e.g. notice the value '4' in each sample). You can see more on this here.

Yes, it is possible for there to be a correlation among the bootstrapped samples, but if you're interested in, for example, a random forest, then the trees are necessarily NOT correlated. If you were to train 100 trees on the same data set, then you could of course end up with correlated trees. However, in a random forest, the trees are uncorrelated. You can see more on this here.

Hope that helps!


Interpret User Interfaces with Machine Learning I am currently working on a prototype of an application that should be able to interact with user interfaces.

Now every user interface has some common elements, like buttons, scrollbars, input fields etc.

I would like to use Machine Learning to "interpret" such user interfaces in a way, in which I later can input a user interface as an image for example, and let the prototype then "try out" the interface, meaning, clicking on buttons, using scrollbars inputting some text into input fields etc.

I know that this would have to be done using Image Recognition, since there are many different UIs.

I am specifically interested in Websites, Adobe Reader with an opened PDF (that in turn can be a form etc.), and Word with an opened Document (again this can contain forms etc.).

Now my main question is if there is already some research going on in this field that I can use, or even an existing tool for parts of the process.

Any help is appreciated :)

I would try experimenting with recurrent neural networks: http://karpathy.github.io/2015/05/21/rnn-effectiveness/. Recurrent neural networks can output sequences of variable length given inputs of variable length. In your case a recurrent neural network might output a sequence like the following when given a user interface: click a button, select a field, type some text, hit enter. For another interface, the network might output only: click one button, click another button, and that's it. This would be useful for you because the sequence and length of actions from interface to interface might change a lot.

You could also experiment with reinforcement learning and build an algorithm that has an objective (reach some final page in as few actions as possible). The algorithm would start by doing random things (like clicking the same button a bunch of times), and then gradually learn over time to take appropriate actions. If you go that route you could use deep learning and Monte Carlo Tree Search (MCTS) like what Alpha Go did.

In either case you're going to need a framework that can train an algorithm quickly because you're likely to have to go through a lot of iterations. TensorFlow (https://www.tensorflow.org/) is one option (I've started using it recently, and I like it a lot because of its easy of use). TensorFlow is capable of building both recurrent neural nets and deep neural nets.


Prioritized Replay, what does Importance Sampling really do? I can't understand the purpose of importance-sampling weights (IS) in Prioritized Replay (page 5).
A transition is more likely to be sampled from experience replay the larger its "cost" is.  My understanding is that 'IS' helps with smoothely abandoning the use of prioritized replay after we've trained for long enough.  But what do we use instead, uniform sampling?
I guess I can't realize how each component in such a coefficient is affecting the outcome. Could someone explain it in words?
$$w_i = \left( \frac{1}{N}\cdot \frac{1}{P(i)} \right) ^\beta$$
It's then used to dampen the gradient, which we try to get from transitions.
Where:

$w_i$ is "IS"
N is the size of Experience Replay buffer
P(i) is the chance to select transition $i$, depending on "how fat its cost is".
$\beta$ starts from 0.4 and is dragged closer and closer to 1 with each new epoch.

Is my understanding of these parameters also correct?
Edit Sometime after the answer was accepted I found an additional source, a video which might be helpful for beginners - MC Simmulations: 3.5 Importance Sampling

Edit As @avejidah said in the comment to his answer "$1/N$ is used to average the samples by the probability that they will be sampled".
To realise why it's important, assume $\beta$ is fixed to 1, we have 4 samples, each has $P(i)$ as follows:
0.1  0.2   0.3     0.4

That is, first entry has 10% of being chosen, second is 20% etc.
Now, inverting them, we get:
 10   5    3.333   2.5

Averaging via $1/N$ (which in our case is $1/4$) we get:
2.5  1.25  0.8325  0.625     ...which would add up to '5.21'

As we can see they are much closer to zero than the simply inverted versions ($10, 5, 3.333, 2.5$). This means the gradient for our network won't be magnified as much, resulting in a lot less variance as we train our network.
So, without this $\frac{1}{N}$were we lucky to select the least likely sample ($0.1$), the gradient would be scaled 10 times.  It would be even worse with smaller values, say $0.00001$ chance, if our experience replay has many thousands entries, which is quite usual.
In other words, $\frac{1}{N}$ is just to make your hyperparameters (such as learning-rate) not require adjustment, when there you change the size of your experience replay buffer.

DQN suffers intrinsically from instability.  In the original implementation, multiple techniques are employed to improve stability:

a target network is used with parameters that lag behind the trained model;
rewards are clipped to the range [-1, 1];
gradients are clipped to the range [-1, 1] (using something like Huber Loss or gradient clipping);
and most relevant to your question, a large replay buffer is used to store transitions.

Continuing on point 4, using fully random samples from a large replay buffer helps to decorrelate the samples, because it's equally likely to sample transitions from hundreds of thousands of episodes in the past as it is to sample new ones.  But when priority sampling is added into the mix, purely random sampling is abandoned: there's obviously a bias toward high-priority samples.  To correct for this bias, the weights corresponding to high-priority samples are adjusted very little, whereas those corresponding to low-priority samples are relativity unchanged.
Intuitively this should make sense.  Samples that have high priority are likely to be used in training many times.  Reducing the weights on these oft-seen samples basically tells the network, "Train on these samples, but without much emphasis; they'll be seen again soon."  Conversely, when a low-priority sample is seen, the IS weights basically tell the network, "This sample will likely never be seen again, so fully update."  Keep in mind that these low-priority samples have a low TD-error anyway, and so there's probably not much to be learned from them; however, they're still valuable for stability purposes.
In practice, the beta parameter is annealed up to 1 over the duration of training.  The alpha parameter can be annealed simultaneously, thereby making prioritized sampling more aggressive while at the same time more strongly correcting the weights.  And in practice, from the paper you linked, keeping a fixed alpha (.6) while annealing the beta from .4 to 1 seems to be the sweet-spot for priority-based sampling (page 14).
As a side note, from my own personal experience, simply ignoring the IS weights (i.e. not correcting at all) results in a network that trains well at first, but then the network appears to overfit, forgets what it's learned (a.k.a. catastrophic forgetting), and tanks.  On Atari Breakout, for example, the averages increase during the first 50 million or so frames, then the averages completely tank.  The paper you linked discusses this a bit, and provides some charts.


Anonymizing Datasets I would like to know what are the best practices for anonymizing datasets?
Ideally I should be able to get the original data back after performing analysis on the anonymized dataset. Should I be using some encryption functions? Hashing maybe?

You should make the difference between the following concepts:

encryption = encoding information in such a way that only authorized parties can read it.
data anonymization = removing personally identifiable information from data sets, so that the people whom the data describe remain anonymous.
de-identification = preventing a person’s identity from being connected with information, while preserving identifying information which could only be re-linked by a trusted party in certain situations (unlike data anonymization which aims to be irreversible).

Your choice depends on your use case.


How can I make seaborn do PairPlot such that it wraps around? I'm trying to do some PairPlots using seaborn, so I can compare a bunch of features against the label I'm trying to model. There are a bunch of features though (~50). So I'm basically doing:

g = sns.pairplot(train_df,x_vars=["MoSold","GarageArea","ScreenPorch","OverallQual"],y_vars="SalePrice",size=4)

Where "SalePrice" is the label that I want to compare the others against. For now I don't care about comparing the features to each other.

The problem I'm having, though, is that when I plot this in my Jupyter notebook, it formats it as one long row.

I'm only doing it with 4 features here, but when I add as many as I want to, it basically makes the row thinner and longer, and impossible to read (because it's trying to make them all share the same y axis).

What I'd like is for it to automatically wrap like panda's hist does:

I can think of a few little hacks. I could do several individual PairPlot commands to manually do it, using a for loop or something, but that would be kind of a pain. Is there some more elegant way to do this? Something where I'd specify the number of columns it'll display with, and then it'll make a grid, where the y axis is always SalePrice?

You should not use sns.pairplot which is for plotting all-by-all comparisons.

You should use sns.regplot which can specify particular comparisons to make. Then use subplots can control the tiling.


Why deep feedforward neural networks are restricted to solve classification problems? I want to use deep neural networks for regression problems, but as far as I've read through, it's mainly for classification. I was wondering why can't a regular convolutional network, or a multilayer perception work as a linear regression, just with multiple layers.
I tried connecting two convolutional layers with max pooling on each, and a fully connected layer together, without activation function, with 10000 MNIST images, 100 epochs, and with 100 batch size. Used mean squared error as the loss function, and regular stochastic gradient descent as the optimizer. Instead of one hot encoding, I used the actual expected numbers as labels.
The results are very promising even though I haven't played optimizing the parameters. The loss after the training is under 1, and I've got R2 values over 0.9 at testing.
So the question is, why neural networks are mainly used for classification while it could handle regression as well?

There is regression by neural networks. For example, have a look at this analysis (Deep Regression), RBF neural networks, and General regression neural networks. Also, linear regression can be implemented using a neural network.
If using neural networks for interpolation, this may work very good. However, if regression is used to extrapolate, results may be unsatisfactory.
Note also that the universal approximation theorems say only that a contineous function can be approximated within a compact set (thus in particular bounded set!).


very low variance explained after applying pca I applied PCA on MNIST data and found that the first 64 components are able to retain 86% of variance.

Is there any problem while applying pca to a big dataset like MNIST. Because in most of the papers I have read to take the components which can explain upto 99% of variance. But it would be pointless to apply pca if such variance comes at 120 pca components.

It is entirely correct to apply PCA to a dataset like MNIST. Intuitively, corner pixels should almost never contain any information as to what digit is contained in the center of the image. So we should disregard them. You should expect similar results as with other datasets. PCA lowers the dimensionality of your data, thus allowing for a less complex model, however, this is at the cost of some information that is rejected when retaining only $n$ components.

When data is limited a less complex model will result in a lower bias and thus better accuracy when applied to novel data. However, the MNIST dataset is plentiful, thus the benefit of removing some features and losing even minimal information may cause your performance to degrade.


Applying and Visualizing k means clustering on a data set that has 9 features I had a data set of images that I have extracted 9 numerical features that I want to apply k means clustering or hierarchical clustering to. I'm just not sure how to go about it. The tutorials I have read all only have 2 or 3 features to them, so it's easy to apply and graph. Any help would be appreciated

Here's the code I have so far:

numpyArr = df.values

ms = MeanShift()
ms.fit(numpyArr)
labels = ms.labels_
cluster_centers = ms.cluster_centers_
n_clusters_ = len(np.unique(labels))
print("Number of estimated clusters:", n_clusters_)

You may use a dimensionality reduction algorithm like PCA method after performing clustering, to reduce the dimension of your clustered data into two dimensions and then visualize the clusters.


Problem importing dataset in zip format I am trying to import a image dataset in Google Colab in order to do image classification. The dataset is in my Google drive, so I do the following:

from google.colab import drive
drive.mount('/content/drive')

now, i want to import my images and divide them into trainig set and test set, so I do the following:

    trainingset = '/content/drive/My Drive/Colab Notebooks/Train'
testset = '/content/drive/My Drive/Colab Notebooks/Test'

batch_size = 32
train_datagen = ImageDataGenerator(
    rescale = 1. / 255,\
    zoom_range=0.1,\
    rotation_range=10,\
    width_shift_range=0.1,\
    height_shift_range=0.1,\
    horizontal_flip=True,\
    vertical_flip=False)

train_generator = train_datagen.flow_from_directory(
    directory=trainingset,
    target_size=(118, 224),
    color_mode="rgb",
    batch_size=batch_size,
    class_mode="categorical",
    shuffle=True
)

test_datagen = ImageDataGenerator(
    rescale = 1. / 255)

test_generator = test_datagen.flow_from_directory(
    directory=testset,
    target_size=(118, 224),
    color_mode="rgb",
    batch_size=batch_size,
    class_mode="categorical",
    shuffle=False
)

num_samples = train_generator.n
num_classes = train_generator.num_classes
input_shape = train_generator.image_shape

classnames = [k for k,v in train_generator.class_indices.items()]

print("Image input %s" %str(input_shape))
print("Classes: %r" %classnames)

    print('Loaded %d training samples from %d classes.' %
   (num_samples,num_classes))
     print('Loaded %d test samples from %d classes.' %
    (test_generator.n,test_generator.num_classes))

but as output I get:

Found 0 images belonging to 0 classes.
Found 0 images belonging to 4 classes.
Image input (118, 224, 3)
Classes: []
Loaded 0 training samples from 0 classes.
Loaded 0 test samples from 4 classes.

so, it seems the dataset has not been uploaded. I think the problem is that the dataset is composed by zip files, in particular I have a dataset composed by moder zip files, and each of these contains images. The images are of four classes: Haze, Sunny, Snowy, Rainy. So are images of these four weathers.

To be more clear I can show the dyrectory:

I am really having troubles understanding how to operate. Can somebody please help me?

Thanks in advance.

Based on Keras documentation here, flow_from_directory should contain one subdirectory per class. Any PNG, JPG, BMP, PPM or TIF images inside each of the subdirectories directory tree will be included in the generator. See this script for more details. That means you should probably unzip your zip files.


What does depth mean in the SqueezeNet architectural dimensions table? First time reading the SqueezeNet paper. Based on my understanding, a fire module contains a squeeze layer of 1x1 filters and a expand layer of 1x1 and 3x3 filters. If we take fire2 for instance, the input dimension is 55x55x96 and we take 16 1x1 filters to convolve over it. This returns a 55 x55x16 output. We then take the output and apply two convolutions, one with 64 1x1 filters and the other with 64 3x3 filters. We then concatenate the two results to create a final output of 55x55x128. In this case, what does the depth of 2 mean? Also, how do I calculate the # of parameters for each layer?

In this paper the depth is defined as the number of layers. The table shows a depth=2 for the fire layers because each is comprised of 2 layers. First is the squeeze layer and then followed by the expand layer.

To calculate the number of parameters of a CNN we can do as follows for a single layer. Assume an input that is 28*28*64. This is the size of the MNIST dataset with 64 channels. Now assuming we want to convolve this input with 32 filters of a 3*3 kernel.

Let us give the following variable names, $l=64$ is the number of channels in the input, $m=n=3$ is the size of the kernel and $k = 32$ is the number of filters. The number of parameters is then calculated as

$((n*m*l)+1)*k = ((3*3*64)+1)*32 = 18,464$.

The $+1$ is used to add the biases.

To calculate the number of parameters in fire2 we first note that this module is comprised of 2 distinct layers.

Layer 1 - The squeeze layer - $s_{1x1}$

$l = 96$, $m = n = 1$ and $k = 16$. Thus giving a total of 1,552.

Layer 2 - The expansion layer

Kernel size 1 - $e_{1x1}$

$l = 16$, $m = n = 1$ and $k = 64$. Thus giving a total of 1,088.

Kernel size 3 - $e_{3x3}$

$l = 16$, $m = n = 3$ and $k = 64$. Thus giving a total of 9,280.

Total

Thus, a total of $1,552 + 1,088 + 9,280 = 11,920$.


Which probability distribution will you use to model outliers? I was asked this question in a recent interview for the position of a Data Scientist:  Which probability distribution will you use to model outliers ?
I told him outliers are like rare events which can be modelled by a Poisson distribution. I pretty sure I'm wrong and the interviewer seemed to think the same. But I don't know the answer to this.
Please advise.

I think the answer is Gaussian distribution. This is a famous approach that is used in Anomaly Detection. What you do is to fit your feature to the Gaussian distribution and the samples which have the probability below the specific threshold are labeled as an outlier.
Quoting from the paper Modeling Outlier Score Distributions:

Many existing unsupervised outlier detection algorithms calculate some kind
of score per data object which serves as a measure of the degree of outlier. Scores are used in ranking data points such that the top n points are considered
as outliers. For example, the statistical-based approach proposed in [4], uses
a Gaussian mixture model to represent normal behavior and each datum is
given a score on the basis of changes in the model. A high score indicates a high
possibility of being an outlier.

Example of usage - 1.
Example of usage - 2


Predicting when a partition in Oracle database will be archived We have an Oracle database in which the main table "A1" is partitioned by the hour of insertion of the row. A row once inserted, may be updated depending upon data in 6 other tables (B,C,D,E,F,G) in the same database.

A partition in table A1 typically gets updated for a few hours after it starts receiving data. After the updates to the partition are finished, the partition gets archived, i.e, it is moved from production to an archive database. The time of creation of a partition and the time of its archival are logged.

Now, my question is given the past history of creation and archival times of partitions, how can I predict when a partition created now in A1 will be archived? What kind of model (regression, etc.) is the best to answer this question? How can I set up a model that tracks the dependence of the archival time of a partition on the attributes in other tables (B,C,D,E,F,G)?

I'd suggest first visualize your data.

The time to archive a partition could be defined as a difference between the partition start (first possible insert) and archiving time (last possible update).

Plot the total distribution of this data for some time interval - check if it follows some distribution. Is the average, median or sd meaningful?

Make a boxplot for each hour to see inter quartile interval and the outliers.

Plot it as a time series - does it have a trend or daily/weekly pattern?

In your simple set up (with one predictor and one response) this will give you impression about the accuracy that you can expect from a model. The choose of the model depends on the result of the visualization.

I personally would start with the simplest models:

all partition archived after $N$ hours;
independent model for each hour predicting say $.9$ quantile of some
history interval;
more advanced 2.: consider the weekday and/or trend and/or
ignoring outliers.

Always check the accuracy and stop if expectations are met.


Time series forecasting: prediction and forecast far from the reality Apologies for the awkward title, but I hope to be able to regain your confidence.

Let's start with the final output I got, so at least you can understand why I'm not happy/concerned about the outcome.

The graph shows the past 2 years website traffic. I was interested to understand whether my learnings could have been applicable to my main profession. I truly believe this is the case, but the results I got aren't very encouraging.

The plot has been obtained running the data past an ARIMA model after different normalisations occurred (e.g. differencing, log, etc.)
The best ARIMA model was found to be 11, 1, 11. This is supported from the below:

You may be tempted to say P = 15, but in fact, after several computations, the best RSME was given by the above. See below (though I'm not sure that RSME can be really applied here)

I understand that almost 69% of confidence is far from being perfect, but I would have in some case expected to return something better than the above output.

Do you have any ideas of what I could have done wrong, what I may improve and how?
Bear in mind I'm learning, so please excuse my limited understanding on complex terminology. The plainer you go, the best chance for me to understand.

Thanks for your support.

EDIT:

Thanks to all comments below, I made some progress. I was also able to get through the SARIMA and counter verify the data with the new data acquired since I started this learning project.

See below the new plot.

Although a perfectly matching fit with the "training" data, when it comes to the forecast, the data hardly correspond.
Given my in-expertise, is the above plot something that you would deem normal? Or Do I need to re-work a lot more to get anything better?

I’m by no means an expert in ARIMA models but I have experimented a bit with them so I can give you a list of some things that I found to improve my models.

I did a grid search of different ARIMA model parameters to find the best model
I also tried some methods to ensemble ARIMA models. For example, averaging the best three ARIMA models gave a better overall score than just one.
Depending on the problem you can play with what level the data is aggregated. I was interested in predicting weekly sales and so I made a model to predict daily sales and then aggregated the predictions to weekly sales. This improved the accuracy dramatically.

Looking at your autocorrelation there appear to be regular weekly patterns. Trying a SARIMA model with a 7 day cycle may result in better scores, I.e. there is a 7 day seasonality.

Finally, I also tried other model types, like facebook’s prophet package or converting it to a regression problem and using weekday, week number, month and holiday days as features in a random forest.


Jupyter notebook - Converted Python dataset to R Dataframe. Got stuck in converting back from R to Python Dataframe I am able to Install Anaconda and able to write Python and R code without any issues based on below help.
https://gist.github.com/simecek/019d87c55fec3839d95bbf8489dde61d

I extracted my data from MS-SQLServer via pypyodbc
Converted the Python dataset to R and maniputed the data (as per my usecase)
Now I am ready to convert my R dataframe back to Python -> Got stuck

The question is..is it possible to convert R dataframe to Python dataframe (as a new dataframe)

Here, my final preparations should be done via Python.

I search for above issue for 3 hours and no luck. Please help.

I find the answer for my own question. I hope this helps to others. Thanks

https://pandas.pydata.org/pandas-docs/stable/r_interface.html


How to match a user with other users with similar interests based on their attributes? Information Available

Consider, there are 'n' users and they have these attributes and values

User A:
Row   | Attribute a | Attribute b | Attribute c
Item 1|    0.593    |    0.7852   |   0.484
Item 2|    0.18     |    0.96     |   0.05
Item 3|    0.423    |    0.886    |   0.156

User B:
Row   | Attribute a | Attribute b | Attribute c
Item 7|    0.228    |    0.148    |   0.658
Item 8|    0.785    |    0.33     |   0.887
Item 9|    0.569    |    0.994    |   0.374

Items in this dataset can be described using the attributes a, b, and, c. So, the items might or might not be the same for different users but the attributes explain the taste of the user.

Currently, I have data for about 1000 users in this format and I can create a classifier for one user that says whether the user will like the given item or not.

Goal

What I want to do is to match users who have similar tastes using the info available above. I don't know much about Recommendation Systems and I'd really appreciate if someone could help me out.

One possible approach would be to create N classifiers (one per each user) and then pick M random items, and run those into the N classifiers.
The outcome would be something like:

        User 1 | User 2 | ... | User N
Item 1:    1   |    0   | ... |   1    --> User 1 and N both like item 1
Item 2:    1   |    1   | ... |   1    --> All users like item 2
...      ...   |   ...  | ... |  ...
Item M:    0   |    0   | ... |   0    --> No user likes item M

where the i-th row contains the result of running i-th item in all N classifiers, and the j-th column contains the results of running all the M items in the j-th classifier.

Then you could see each user as a M-dimentional point, and use a simple classifier such as KNN with haming distance for the distance metric.

With a larger M, you'd get more accurate results, since you're using more variables to compare each user.
The only caveat here is that you'd need those N classifiers to be very accurate, in order to minimize error propagation.


Liner regression and feature scaling Below are few questions where I unable to find out where I am wrong. I added screen shot of image and explanations of the each options that I am understanding. Questions are purely discussion based and short. Please help me out.

In below question, I checked

A. We need to predict the author gender and it can be either male or female. I think  it is  classification problem, so it is supervised learning.

B.  We have given group of spam emails and need to predict does there sub-types are spam or not. I think  it is  classification problem, so it is supervised learning.

C. We need to predict data based on height and age. It is a linear  regression problem because we create graph height vs age will find out the test case. It is supervised learning.

D. Grouping data is a cluster problem, so it unsupervised learning.

In below question, I had checked C and D options because feature scaling creates our dataset in same range which helps to predict the best theta in less iterations and contour graph will be more cleared and symmetric. Ref : https://medium.com/greyatom/why-how-and-when-to-scale-your-features-4b30ab09db5e

For the 1st question, I think B is not a supervised learning problem. You are already given emails which are spam. You need to analyse them to find if there are subtypes (Clustering.)

For the 2nd question, it is just B. Scaling doesn't make each step less computationally expensive. You do same amount of computation.


What approaches are there to not display a search result that a user has no permission to see? Question

What approaches are there to not display a search result that a user is not supposed to see?

Example

Suppose we have the following situation.

High-level view

In an enterprise, there are different repositories like websites, databases, filesystem folder etc. And there is a search engine that crawls all those repositories and creates an index. The user can then use the search engine UI to perform searches.

Let's say the search engine was Apache Lucene and the encompassing indexing and retrieval system Apache Solr.

User permissions are managed via Microsoft Active Directory.

The content in the different repositories are managed with all kinds of applications like MS Word, custom-made applications, database management tools etc.

Search-index and user permissions

The index contains the documents A1, A2, B1 and B2.
User A has only permission to see documents of type A*.

Performing a search

Now, user A performs a search for A* AND B*.

Displaying the results

A system administrator with permission to see everything should see all four documents in the search results: A1, A2, B1 and B2.

User A however, who only has permissions to see documents of type A*, should/will only see two documents in the search results: A1 and A2. So user A shouldn't even know that documents B1 and B2 exist.

Elaboration on the question

What ways are there to implement that requirement that users only see documents they are allowed to see?

I suppose it would be a bad idea to add the information to the index whether a user may see a document. I think that because of this use case: suppose all of a sudden user A may only see document A1 and not A2 anymore; those permissions are now set for example on the filesystem. Now it takes a while until the permissions in the index are updated and during that time user A would still be able to see document A2 in the search results even if he can't click on the result and access it anymore.

Also, the above approach probably would break down when there are hundreds of millions of documents and thousands of employees.

Where and how would the security aspect be implemented in a content indexing and retrieval system in the above scenario?

Since you aren't sure which software you would like to use, I can explain this concept only in general, by example. The clearest example is a content management system like Drupal or Wordpress (not necessarily a data science tool but the idea is the same).

In a content management system, there are different types of content which can be accessed. In your case, there would be two types - document type A and document type B. There can be an unlimited number of documents, each assigned one or more types.

The software has a permission system that gives access to each document type. This is built with roles. In your example, there would be a system administrator role, and a general user role.

Each role has permissions associated with it. You would configure system administrator to be allowed to view document types A and B, but you would configure general users to only be able to see document type A.

Each user registers an account in the software system. A site administrator assigns each account one or more roles. So a user with the system administrator role would be able to see all documents of type A and B. However, if a user only has the general user role, they would only be able to see documents of type A. Also noteworthy is the fact that each user can have multiple roles and will get all permissions associated with every role they have.

USER -has-> ROLE(s)

ROLE -has-> PERMISSION(s)

CONTENT TYPE -requires-> PERMISSION(s)

Every time the content is requested, the software system ensures that the user making the request has the permission to access the content.

When a search is run, the software system has to validate every potential result individually before displaying the search results to the user. The user must have the permission to see each piece of content that is be delivered back from the search.


how to see the correlation graph when there is lot of variables? my variable has more then 4000

my corr() graph looks like this

is there better way to look at this graph more clearly?

You are trying to get a look at ~8 millions coefficients at once, I am afraid there isn't a single technique that would really help that.

However you might want to investigate your correlation structure. The first step to do that would be to do some correlation based clustering. In Python, you might want to plot your dendograms : https://docs.scipy.org/doc/scipy/reference/generated/scipy.cluster.hierarchy.dendrogram.html This will allow you to observe sub-clusters of correlation. Further investigations will depend on what you are trying to achieve.


How neural networks handle large variations in the input? I am reading the paper with the title "Classification of indoor actions through deep neural networks". And I came across this statement:

  With this mechanism very complex functions can be learned combining
  these modules: the resulting networks are often very sensitive to
  minute details and insensitive to large irrelevant variations.

I have read a lot about deep neural network. However, I might have missed the logic behind why deep neural networks are insensitive to large irrelevant features.

They can handle large variations of inputs because the neurons have weights and those weights get optimized as part of learning a good model. So even though the model might take in a value that ranges from 0 to 100,000, if it isn't relevant to predicting the outcome, it will have a very small effect.


Adding more emphasis on most recent data in CNNs I am using a CNN for multivariate time series analysis. The input size is (batch_size, 500, 30) i.e 30 variables and 500 time steps. I would like to put more emphasis on recent data and less on the past data.

What is the best way to achieve this?

One approach is to use Keras - sample_weight parameter in fit() method.

That parameter specifies which steps are more important.


Data augmentation in deep training I'm trying to understand the role of data augmentation and how it can affect the performance/accuracy of a deep model. My target application is a fire classification (fire or not, on video frames), with almost 15K positive and negative samples, and I was using the following data augmentation techniques. Does using ALL the followings always increase the performance? Or we have to choose them somehow smartly given our target application?

rotation_range=20, width_shift_range=0.2, height_shift_range=0.2,zoom_range=0.2, horizontal_flip=True

When I think a bit more, fire is always straight up, so I think rotation or shift might in fact worsen the results, given that it makes the image sides stretch like this, which is irrelevant to fires in video frames. Same with rotation. So I think maybe I should only keep zoom_range=0.2, horizontal_flip=True and remove the first three. Because I see some false positives when we have a scene transition effect in videos.

Is my argument correct? Should I keep them or remove them?

Your reasoning is perfectly correct. Augmentation is just a process, which helps you cover your domain better. You should only pick operators that help you. Abusing augmentation can definitely mess up your model. It's always good idea to print data at those limits, to check yourself. Try also to think, how data will be acquired on production. Albumentations is a nice repo with a lot of available methods.


Maximize the margin formula in support vector machines algorithm I was recently reading about support vector machines and how they work and I stumbled on an article and came across Maximize the distance margin.

Can anyone tell me what do we have to minimize here? I wasn't able to understand this part I pasted below. What is $w$ and $m$ here in the formulas given below?

Maximize the margin

For the sake of simplicity, we will skip the derivation of the formula for calculating the margin, m which is
$$m = \frac{2}{\|\vec{w}\|}$$

The only variable in this formula is w, which is indirectly proportional to m, hence to maximize the margin we will have to minimize ||w||. This leads to the following optimization problem:

$$\min_{(\vec{w}, b)} \frac{\|\vec{w}\|^2}{2}$$

subject to $$y_i(\vec{w}\cdot\vec{x}+b) \ge 1, \forall i = 1, \ldots, n$$

The above is the case when our data is linearly separable. There are many cases where the data can not be perfectly classified through linear separation. In such cases, Support Vector Machine looks for the hyperplane that maximizes the margin and minimizes the misclassifications.

Link to the article : Hackerearth

The separating hyperplane that we are looking for is $w^Tx+b=0$. That is $w$ is the coefficient or the slope of the separating hyperplane and $m$ is the margin.

By scaling, for the positive class, we want the closest data point (support vectors) to satisfies $w^Tx+b=1$ and for the negative data class, we want the closest data point to satisfies $w^Tx+b=-1$.

Recall from geometry class, (a $3$ dimensional case is discussed here) that the distance from a point $y$ from a plane $w^Tx+b=0$ is $$\frac{|w^Ty+b|}{\|w\|}.$$

Hence as a result, the distance of the closest point to the hyperplane would be $\frac{|\pm1|}{\|w\|}=\frac{1}{\|w\|}$ and hence, the margin, that is the closest distance between the two classes would be $\frac2{\|w\|}$.

We want to maximize the margin, $\frac2{\|w\|}$ which is equivalent to minimize $\frac{\|w\|}2$ which is equivalent to minimize $\frac{\|w\|^2}2$ subject to the constraints that we classify the points correctly.


When to stop collecting data? I have developed an iterative process via which I can collect data in batches. The data are points in a predefined 3D space. I am trying to explore and locate clusters in that 3D space based on my data. After some batches have been collected I can locate and create the clusters. As I collect even more data though, small refinements are being made and the new results are not really worth the effort. How can determine whether a new batch is "worth" collecting? Is there some metric I can use to measure how much the batches I have are "similar" or whether a new random batch will affect the overall process?

Quantify how much your clusters change with each batch.

Then stop if the change becomes smaller than a threshold.


XGBoost, binary classification: uneven number of observations per user I'm working on a binary classification problem with XGBoost and I have a dataset, which has uneven number of observations per user. For some users there are over 100 observations, whereas for some users there are only a few. The "USER_ID" feature is not used as an input for XGBoost.

More specifically, I'm trying to model user physical activity (data collected from wearable trackers) in respect to sleep quality, and some of the variables are demographical features such as as age and sex, alongside steps, heart rate etc. Considering the differing amount of data collected from users, some user behaviours (such night-shift work) are represented more in the data than others due to the number of observations.

How should I take this into account when working with XGBoost?

USER_ID  AGE  SEX  X1  X2    ...  y
1        20   M    65  3000  ...  1
1        ...  ...  ... ...   ...  0
1        ...  ...  ... ...   ...  1
2        30   F    80  2500  ...  0
2        ...  ...  ... ...   ...  1
3        40   M    77  8000  ...  0

The classes are otherwise balanced and I'm able to get good performance for the classifier.

You may want to consider a stratified cross-validation approach where you specifically undersample the data from users that have lots of observations or oversample from uses with fewer observations.

Another approach which may also work is to duplicate the observations of some of the users with less observations, this would even out the weighting of the different kinds of observations.

-- Adding on to answer the additional questions in the comments

There are a number of ways to undersample the majority or oversample the minority.  Here's an overview of how to handle the situation (in general).

To undersample the majority, simply remove some % of the training data that is in the majority for each run.

To oversample the minority, duplicate the values that you want to have a higher weight.

Additionally, there is a parameter for XGBoost (scale_pos_weight) which allows you to set the weight of the different samples, see here.  Additionally you could leverage SMOTE from the imblearn library.


Wrapping legacy scripts for reproducibility I'm setting up a system that transforms data through several steps using code written by many people with many levels of coding experience. I'm aiming to achieve reproducibility of the whole process by fully defining the inputs and outputs of all the executing steps.

The problem I'm having is that many of the script that need to run have side effects, which can break this determinism. I want to minimise the amount of rewriting of people's code that I have to do, since this system needs to remain usable by those people.

How can I effectively sandbox scripts so that their inputs and outputs are guaranteed to be well defined? My first thought was through containerisation using eg Docker, but our HPC cluster doesn't yet support this so I'd prefer lighter weight options. Feel free to assume python scripts, although the ideal answer would be language agnostic.

For HPC systems, wrapping scripts in a Singularity container https://sylabs.io/docs/ could be an option.


A faster multidimensional scaling? I've been playing around with some data, and I've created a decently large similarity matrix (each column value for a row represents how similar the document corresponding to the column is to the document corresponding to the row) and I'm trying to scale this data down to 2 dimensions so I can visualize it on a scatter plot, where each point represents a document, and the closer two documents are, the closer they would be on the scatter plot. I've tried using Sammon Mapping to accomplish this, but the problem is it takes a significant amount of time to run. Any ideas?

You may use a sampling based method, in which you get an approximation of the actual result by combining the partial results for samples extracted from your data. There are several methods based on sampling, but you could take a look to the following link as an example, so you can understand what I am talking about:

A Fast Approximation to Multidimensional Scaling


Multi-Head attention mechanism in transformer and need of feed forward neural network After reading the paper, Attention is all you need, I have two questions:

1. What is the need of a multi-head attention mechanism?

The paper says that:

  "Multi-head attention allows the model to jointly attend to
  information from different representation subspaces at different
  positions"

My understanding is that it helps in anaphora resolution. For example:- "The animal didn't cross the street because it was too ..... (tired/wide)". Here "it" can refer to animal or street based on the last word. My doubt is why can't a single attention head learn this link over some time?

2. I also don't understand the need of feed-forward neural network in the encoder module of the transformer.

Thanks for your help.

The basic reasoning, I think, is just to increase capacity. While it is possible in theory for a single head, using multiple simply makes it easier. More specifically though, the paper says (pg 4):

  Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this.

In other words, for a given layer, with only one attention head, the weighted averaging performed by the mechanism prevents you from being able access (differently transformed) information from multiple areas of the input within that single layer.
I think this can be mitigated by using multiple layers, but adding this capability simply increases the capacity of a single layer, and therefore you can perform more useful calculations with less layers. Often this is good because stacking layers (function composition) can lead to more issues than parallelizing layers (as the multi-head does), especially when it comes to gradient computations. (This is partly why residual/skip connections are so useful).

In general,  using multi-head is a common way to increase representational power in attention-based models, for instance see Graph Attention Networks by Velickovic et al. The basic idea is that, although attention allows you to "focus" on more relevant information and ignore useless noise, it can also eliminate useful information since usually the amount of information that can make it through the attention mechanism is quite limited. Using multiple heads gives you space to let more through.
Presumably you are asking about the "Position-wise Feed-Forward Networks". Again, I think this is merely a question of model capacity: the model could function without these layers, but presumably not as well. These layers are interesting because they are essentially 1D convolutions, which usually appear near the end of a network or block in computer vision architectures (which I am more familiar with). In other words, they are localized transforms that operate identically across the input (spatially). The reasoning is often that the previous layers (here, the attention layers; in vision, the conv layers with larger kernel sizes) were reasonable for passing or mixing information spatially across the input. E.g., after an attention layer, the latent representation at each position contains information from other positions. After this, however, we want to consolidate a "unique" representation for each position (which has been informed of the other positions, of course). We do this via processing by a localized layer, which does not consider neighbours or other positions, and simply transforms the local representation on its own. One thing to keep in mind is that, while we want information to mix across "space" (e.g., across an image or sentence), for many tasks we often still need each position to maintain some resemblance/connection to its original identity (rather than say just averaging all information over all positions). So not every layer need be spatially aware. In vision models, 1D convolutions (i.e., kernel width 1) are often used to perform dimensionality reductions; in this case, the dimensionality does not appear to change however.


Tuning a classifier for high precision, with no regard for recall I understand this falls under the decision making aspect, rather than the probabilistic, but for the purposes of some work I am doing, I need the classifier to have very high precision, as I can't afford a false positive. I do not care about false negatives, and consequently, do not care about recall. Since it is currently a binary classifier, some might say to play with the decision probability threshold from its current 0.5 value, but I will eventually need to add a third class, and will therefore need to switch to 3 outputs with softmax. I am unaware of traditional methods for shifting my pipeline towards a high precision outcome, and am looking for ways to achieve this.

If it is any help, the problem is classification of 256x256 grayscale images in a domain that is very difficult to classify, according to current whitepapers in the computer vision research area.

To make your learner cost sensitive, you can increase your training data by more "no" instances. If there are 10 times more "no" instance in your training data, the errors on "no's" will hurt much more and your learner will come up with a dicision scheme which will be biased that way. The variance on the "yes'es" will be lowered on the other hand (pay attention on overfitting here).
After the training, you use your original data for testing and you should get good results here.


Should I normalise image pixel wise for pretrained VGG16 model My goal is to use pretrained VGG16 to compute the feature vectors excluding the top layer. I want to compute embedding(no training involved) per image one by one rather than feeding batches to the network as this network is just to compute embedding and not classification. In batch training I understand the importance of batch normalisation but for a single image should I normalise it pixel wise? All I can think is maybe its useful to reduce the importance of illumination in an image. Am I missing something?

It all depends on how the original pretrained model was trained. If it was trained with normalized data, you should also normalize your data before giving it to the model. Otherwise, the input data distribution will not match what the network was trained with, and then you probably won't obtain good results.
VGG16 pretrained weights expect normalized data as input, so the answer is yes, you should normalize the data (subtracting the mean and dividing by the standard deviation).
If your image domain is similar to ImageNet, you may use the same mean and standard deviation statistics. If your images are from a very different domain, you should compute your own statistics (source).


Random Forest variable Importance Z Score I am really worried How does RandomForest Algorithm Works internally,

Out of bag error
Variable Importance
Z Scores

Let me know all these three ? How to calculate it? What it controls?

Please allow me to cite the site of creators of the RandomForest algorithm
Out of bag error

In random forests, there is no need for cross-validation or a separate test set to get an unbiased estimate of the test set error. It is estimated internally, during the run, as follows:
Each tree is constructed using a different bootstrap sample from the original data. About one-third of the cases are left out of the bootstrap sample and not used in the construction of the kth tree.
Put each case left out in the construction of the kth tree down the kth tree to get a classification. In this way, a test set classification is obtained for each case in about one-third of the trees. At the end of the run, take j to be the class that got most of the votes every time case n was oob. The proportion of times that j is not equal to the true class of n averaged over all cases is the oob error estimate. This has proven to be unbiased in many tests.

Variable importance and Z-score

In every tree grown in the forest, put down the oob cases and count the number of votes cast for the correct class. Now randomly permute the values of variable m in the oob cases and put these cases down the tree. Subtract the number of votes for the correct class in the variable-m-permuted oob data from the number of votes for the correct class in the untouched oob data. The average of this number over all trees in the forest is the raw importance score for variable m.
If the values of this score from tree to tree are independent, then the standard error can be computed by a standard computation. The correlations of these scores between trees have been computed for a number of data sets and proved to be quite low, therefore we compute standard errors in the classical way, divide the raw score by its standard error to get a z-score, ands assign a significance level to the z-score assuming normality.


Time Series Forecasting with Neural Network (mixed data types) I have a dataset with the following format:

TimeStamp  | Action  | UserId
2015-02-05 | Action1 | XXX
2015-02-06 | Action2 | YYY
2015-02-07 | Action2 | XXX
...

I try to forecast future Actions for specific users based on the Users history in the dataset. I started with ntstool in MATLAB, but it can't handle mixed data types or non-numerical values. Now I am looking for other methods to predict future Actions and find periodic patterns in the records.

Is it possible to convert the values for Actions to numeric values or is there a possibility to create a neural network with mixed datatypes on the input? Maybe in R?

I guess I answered the question in the comments, so here goes.

Most ML models cannot deal with categorical values. A common way to solve this is to use one-hot encoding, also known as dummy variables. For very possible value of your categorical variable you create a column which is 0 unless this row has this category, then it is 1. It is possible to remove one of the categories since it is a linear combination of the other dummy variables (if all are 0, the last one must be 1).

The downside of this method is that it increases the dimensionality of your feature space. If you have enough data to support this or not that many categories that is not a problem. There are other alternatives, like taking the average feature of every category and adding that to your features as opposed to the categorical feature.


Why is the curve for alternative hypothesis shifted compared to that of null hypothesis? In hypothesis testing the concept of type 1 and type 2 error is very important and very often we come across the graph shown below. I have trouble understanding why is the curve for alternative hypothesis shifted compared to the null hypothsis?

The idea is that the population has the mean of the null hypotesis, say mean height of Sweden is 181cm. Then you do a sample of the population and find that the sample height is 185. Is this enough for you to discard the null and accept the alternative?

This is what the curves are trying to illustrate. The reason for the shifted curve for the alternative is that the mean is located at a different point on the x-axis (185 in my example).

So, given that h0 is true (mean 181), what are the probability to draw a sample mean of 185? That would be the right tail of the null curve straight under the alternatives mean and all to the right of it.


Improve Precision of a binary classifier - Decision Tree in Python Currently, I am working on a project. The dataset is balanced roughly in the ratio of 50:50. I created a decision tree classifier. I am achieving decent accuracy (~75%) on validation data but the precision for the target variable is biased. For class=0 it is approx. 98% while for the class = 1 it is just 17%.

I have tried scaling the data using MinMaxScaler still no luck.

model = tree.DecisionTreeClassifier(class_weight={1:30}, min_samples_leaf=160, max_depth=10)

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.3, random_state=10)

min_max_scaler = preprocessing.MinMaxScaler()
X_train_scaled = min_max_scaler.fit_transform(X_train)
X_test_scaled = min_max_scaler.fit_transform(X_test)

model = model.fit(X_train_scaled, y_train)

prediction = model.predict(X_test_scaled)

print metrics.accuracy_score(y_test, prediction)
print classification_report(y_test, prediction)

Size of x_train_scaled = 12600 and x_test_scaled = 5400
Accuracy: 75%
Precision: {0:100%, 1:17%}
Recall: {0:74%, 1:100%}
F1-Score: {0:85%, 1:29%}

How can I improve the precision of class=1 while still maintaining the overall precision and accuracy?

I guess differences in accuracies between class 0 and class 1 come from the class_weight parameter you have used. Class 1 will benefit from this overweighting towards class 0. You could try to play on this parameter to re-balance your results in class 0 and class 1.

An other idea could be to play on probabilities outputs and decision boundary threshold. Remember than when calling for method .predict(), sklearn decision tree will compare outputed probability with threshold 0.5. If it is greater than 0.5, then it assign class 1. On the contrary, if it is less than 0.5, it will assign class 0. You could try to play on this threshold by outputing probabilities first with .predict_proba()[:,1] and then test results for different thresholds decision boundaries. You can operate such as below :

model = clf.fit(df[features], df[label])
df["proba"] = model.predict_proba(df[features])[:,1]
threshold = 0.4 # You can play on this value (default is 0.5)
df["pred"] = df["proba"].apply(lambda el: 1.0 if el >= threshold else 0.0)